mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-22 12:43:32 +09:00
f83db514b0eb88cefc2923d17aad59daccb4e358
7777
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
44b95e747d |
fix(cdi): update CDI to v1.66.1
Brings two upstream fixes CDI v1.64.0 users hit in the field: - qemu-img 10.1.0-24.el9 restores HTTP imports to block volumes on 4Kn devices (DRBD/LINSTOR), the regression that kept this repo from following v1.65.0. The earlier -21 build shipped in v1.66.0 did not carry the fix, which is why this targets v1.66.1. - import-controller no longer clears event-based bound-condition annotations on every pod sync, removing the resourceVersion storm on prime PVCs during WFFC imports that starved the scheduler with FailedScheduling retries. The fix is commit e547586e9, a one-line guard in updatePvcFromPod that rides inside a squash titled after its kubevirtci changes, so neither the release notes nor the pull request title mention it. The operator manifest is refetched with make update. Two local customizations survive as patches now: the startupProbe patch gains the correct -p4 strip level (it never applied from the package directory as written), and the control-plane tolerations that were hand-edited into the manifest since v0.0.1 move into patches/tolerations.diff so the next refetch cannot drop them. Behavioral notes for this bump: CDI /metrics now requires a bearer token (nothing in this repo scrapes it); scratch PVCs inherit the target StorageClass instead of the cluster default; the PVC-rendering webhook is always registered but only matches PVCs labeled cdi.kubevirt.io/applyStorageProfile=true, which nothing here sets. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <3811295@gmail.com> |
||
|
|
86d62bbb29 |
fix(harbor): give the nginx proxy explicit resources so it survives the tenant LimitRange (#4095)
## What this PR does With `expose.type: clusterIP` the goharbor chart deploys an nginx proxy pod, but the `harbor` chart set no resources for it. In a tenant namespace the `tenant-quota` LimitRange defaults any container without resources to a 128Mi memory limit, so the proxy is OOMKilled (exit 137) and Harbor never becomes ready. This gives the nginx proxy `t1.small` (512Mi), and bumps the portal (also an nginx container) from `t1.nano` (128Mi) to `t1.micro` (256Mi), which sat at the same limit. Reported from a tenant where the Harbor nginx pod was OOMKilled at a 128Mi limit. ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map against the diff. The change is limited to `packages/apps/harbor/templates/harbor.yaml` and only sets container resource presets already used elsewhere in the same file. No `values.schema.json`, `values.yaml` default, `ApplicationDefinition`, version enum, or `release.prefix` is touched, so nothing downstream (provider schemas, website reference pages, ansible role) is affected. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(harbor): set resources for the nginx proxy and raise the portal preset so neither is OOMKilled by the tenant LimitRange default (128Mi) ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Performance** - Updated Harbor’s web portal resource allocation to improve runtime capacity. - Added default resource settings for the NGINX component. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
5a7e2d5546 |
fix(harbor): give the nginx proxy explicit resources so it survives the tenant LimitRange
With expose.type=clusterIP the goharbor chart deploys an nginx proxy pod, but the chart set no resources for it. In a tenant namespace the LimitRange defaults any container without resources to a 128Mi memory limit, so the proxy is OOMKilled (exit 137) and Harbor never comes up. Set the nginx proxy to t1.small (512Mi) and bump the portal (also nginx-based) from t1.nano (128Mi) to t1.micro (256Mi) for the same reason. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
41f92ec27a |
fix(release): compare host-less image refs on digest alone (#3908)
Ports #3905 to `main`. Same fix, same tests, two cherry-picks, no conflicts. `main`'s copy of `hack/verify-promoted-packages.sh` was byte-identical to the buggy one, so the next promotion cut from here fails the way v1.6.2 just did: `normalized_refs()` gives a host-less ref a repository equal to its own tag, because `repo="${image%:*}"` is a no-op on a bare `tag@sha256:...` scalar with neither `/` nor `:` in it. `hack/lib/image-refs.sh` emits one of those per shape-3 split map, promotion rewrites exactly those tags, and the check then reports changed repositories at identical digests. It cost several rounds on v1.6.2 before the cause was clear, so it is worth not paying twice. Checked against `main`'s own tree rather than assumed from the 1.6 port. 13 host-less refs, same count as `release-1.6`: 12 pair one-to-one with a host-bearing ref at an identical digest, and the 13th is `packages/system/kuberture/values.yaml` (`tag:` only, no `repository:`, no `registry:`), which is genuinely digest-only and is the case the comment says must not be dropped. The narrowing the 1.6 review left open still does not exist here. Of 14 repository-bearing digest maps, exactly one has a slash-free `repository:` — kube-ovn's `global.images.kubeovn` — and `global.registry.address` gives it a shape-4 join that keeps its repository in the compared set. No map anywhere has an empty or non-string `registry:` sibling. `hack/lib/image-refs.sh` differs from `release-1.6`'s in comments only: 47 code lines either side and an empty diff once comments are stripped, same globs and same selectors, so the refs reaching the verifier are unchanged. Not modified here. Red phase checked both directions rather than inherited. Reverting the one line reddens the new split-map case on the digest-set assertion. Replacing `repo=""` with an outright `continue` for host-less refs flips which case fails: the split-map one goes green and `rejects a changed digest carried only by a repository-less image map` reddens at its exit-status assertion, because the verifier then exits 0 on a genuinely changed digest. That second direction is the one worth having a test for, since dropping host-less refs is the obvious-looking fix. Gate: the four promote bats suites individually green, `shellcheck -x --shell=sh` clean, `pre-commit` green. No Go, charts or generated files, so build, vet, test and `make generate` do not apply. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved promoted package validation for image references without explicit registries or repositories. * Promotion-only tag changes are now accepted when the image digest remains unchanged. * Digest changes continue to be detected and rejected correctly. * **Tests** * Added coverage for split image reference formats and repository-less image mappings. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
00c6eb18bf |
docs(e2e): name the container lane's one-per-host limit
SANDBOX_NAME is per-checkout and these are not, which reads like an oversight until you notice that hack/e2e-compose.yaml's container names and the data-srvN zpools are global too. Two checkouts on one host therefore share nodes and pools, and the second one's teardown destroys the first one's storage. Recorded rather than fixed. CI cannot reach it -- the lane already asks for 24 vCPU and 72 GiB against a 32 vCPU / 128 GiB runner, so a second one does not fit beside it -- and isolating it properly means renaming the pools, which the LINSTOR storage-pool registration reads. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Assisted-By: LLM |
||
|
|
c748c5b707 |
docs(e2e): drop the references the removed suites and carve-out left behind
Convention 7 still sent readers to the node-join carve-out, while section 1 of the same file -- rewritten on this branch -- says the script, the marker and the soft_red output no longer exist. Restated as what the deadline now is. hack/e2e-chainsaw/README.md still listed kubernetes-oidc-system and kubernetes-oidc-customconfig among the suites, and select-install.sh's suite_to_source() still mapped both. The mapping is harmless today because the round-trip pin walks existing directories, but select-e2e.sh says the two functions have to stay in step, so a stale half is a trap for whoever reads that promise. sandbox-runner-headroom.bats keeps reading the QEMU sandbox file for every lane, container lanes included: `prepare-env` also matches `prepare-env-container`, and the compose file gives its nodes the same 8 CPU / 24576 MiB by design. The figures are therefore right for both substrates, but only because the two agree -- and that they agree is pinned in container-lane-capacity_test.bats rather than here. Said so in the test, so the next reader knows what would have to change if the container lane is ever sized differently. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Assisted-By: LLM |
||
|
|
b131bd56a4 |
fix(e2e): restore the default StorageClass when the fallback render fails
verify_storageclass_fallback_default deletes the `replicated` class, renders with --dry-run=server, and restores it inline "before any assertion exit" -- the comment's words. Under the caller's `set -eu` the render is a plain assignment, so a failed render exits the function there: before rc is read, and before the restore runs. The management cluster is then left with no default StorageClass for every suite after it, which is the opposite of what the comment promises. Captured with `|| rc=$?`. Pre-existing shape, but the restore block is edited on this branch anyway, and the QEMU lanes are where it bites. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Assisted-By: LLM |
||
|
|
b0e0f38fbe |
test(e2e): keep the chainsaw ops above the deadlines they contain
.chainsaw.yaml states the rule and says why: an op that expires before its inner bounds do is a SIGKILL on the process group, which takes out the diagnostics the failure branch exists to print, while an inner timeout leaves a partial capture and lets the remaining steps run. refresh-and-assert-full-health, added on this branch, breaks it. The poll deadline is 270s, the last in-loop read can start just under it and run 25s, and the failure branch then reads again for another 25s: 320s of inner bounds under a 5m op. Raised to 6m. verify-processes-and-security-context breaks it harder and did so before this branch: 240s for the process counts, then 120s for connectionString and 120s for generations.reconciled, sequentially, is 480s under the same 5m. Fixed here rather than filed, because it is the same defect one step away, found by applying the rule to the neighbour. Raised to 9m. The kubernetes-latest ceiling gets the measurement instead of a change. The OIDC lifecycle folded onto that cluster adds up to 1m + 600s + 1m of passing-path waits that the 67m op does not price, and raising the op is not available: a guard holds COZY_OP_CEILING equal to both suites' timeout, and two suites at 79m plus their teardowns exceed the job's 215-minute cap. On the green run of this branch the whole Test took 569.72s against the 4020s op, with kubernetes-previous at 470.19s. The ceilings only bind once something is already hung, which is what the existing residual note (#3666) says; now it says it with a number. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Assisted-By: LLM |
||
|
|
ec7f17299b |
test(e2e): make the HelmRelease readiness gate hold its own ceiling
Two ways the gate did not do what its arguments promise. The list read carried no bound. kubectl defaults to no request timeout, and the read sits above the deadline test at the bottom of the loop, so against a wedged apiserver it blocks there and timeout_seconds stops being a ceiling: what arrives is the job timeout as a kill, instead of the not_ready dump this function exists to print. Bounded with kubectl's own --request-timeout rather than a `timeout` wrapper, because the unit suite substitutes a kubectl shell function and `timeout` execs a binary. The final diagnostic read is bounded the same way. `now` was sampled before the read, so a read costing more than the poll interval was priced at zero against the deadline and bought the loop another whole iteration past it. Sampled after, on both branches. And the count >= minimum_count half of the gate was pinned by nothing: every fixture returned 11 or 12 items against a minimum of 11, so the condition could be deleted outright and all five tests stayed green. That threshold is the direct replacement for the `wc -l` existence backstop this branch removes from hack/e2e-install-cozystack.bats -- the only thing separating "everything is Ready" from "almost nothing got created", since three HelmReleases that exist are all Ready. Both new cases were checked by reverting the code they cover. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Assisted-By: LLM |
||
|
|
ad00f35f2d |
test(e2e): correct two comments that describe the opposite of the code
pull-requests.yaml said the QEMU substrate "is not wired into any workflow now" and that prepare-cluster and hack/e2e-prepare-cluster.bats stay in the tree for a nightly lane that will carry the coverage. Both halves are wrong in the direction that costs the most: nightly.yaml and e2e-tag.yaml already run `make prepare-env`, which calls prepare-cluster, so anyone trusting that comment deletes live code and takes nightly and every release-candidate run with it. docs/agents/e2e-testing.md states it correctly, so the tree disagreed with itself. hack/e2e-platform-packages.sh said the lane replaces "only LINSTOR with an otherwise-identical Package". It replaces two, and the kubevirt-cdi one is not identical: it raises the importer memory limit from CDI's 600M to 4Gi and the request from 60M to 256Mi. The consequence belongs next to the override -- the merge-gating lane does not exercise the shipped CDI default, and whether a tenant worker disk imports at 600M is answered by the QEMU lanes, which take no override. Also drops the dead half of `resolve_assets`. It resolved nocloud-amd64.raw.xz and exported disk_id; the container lane boots no disk and nothing in this workflow reads that output any more, so all the lookup could still do was fail a release-labelled PR over an asset it does not use. The job stays -- e2e's release arm gates on its result -- and now proves what is still worth proving, that the draft release exists. e2e-tag.yaml keeps resolving the asset, because it consumes it. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Assisted-By: LLM |
||
|
|
f73729937c |
docs(vm-disk): record what immediate binding costs on a node-pinned class
Requesting immediate binding for every source, not just `upload`, is deliberate: a VMDisk is a standalone object populated before any VM consumes it, so on a WaitForFirstConsumer StorageClass a disk with no VM attached would never populate at all. What that costs was recorded nowhere an operator reads. On a node-pinned class the CDI worker pod chooses the node, so the volume binds where the importer landed rather than where the VM will run, and a VM constrained elsewhere -- a nodeSelector, a request only one node satisfies, or the Windows affinity from _cluster.scheduling.dedicatedNodesForWindowsVMs -- stays unschedulable against a node-affinity conflict. One disk is enough for that; it is not a multi-disk-only case. storageClass is immutable, so the exit is delete and recreate. The chart default `replicated` binds Immediate regardless and does not move. Also pinned in tests, in both directions the annotation is emitted: per source type (blank, image, disk) so a regression to the old upload-only condition reddens, and on the existing-DataVolume path, where `helm upgrade` retrofits the annotation onto disks created before it was unconditional -- inert for a volume that has already bound, and now a recorded decision rather than an unasserted side effect. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Assisted-By: LLM |
||
|
|
be55c68fbc |
docs(linstor): keep the strict-topology rationale out of the applied object
A `#` comment in a rendered template is output, not source: the eight lines explaining --strict-topology were applied to the live LinstorCluster and came back on every `kubectl get`. Moved into a template comment, which is stripped at render. The test comment beside the argument-count assertion claimed the count was "a tripwire that forces a re-check" on a piraeus-operator bump. It cannot be. The assertion reads the chart's own rendered args, and those do not move when upstream adds a flag -- the count stays 12 and the suite stays green. It catches an edit to this chart, which is the case that needs it least. Say what it does catch, and say where the upstream half actually has to be checked: the tree vendors the piraeus chart but not the operator resources carrying those defaults, so there is nothing in-tree to diff against and the re-check on a bump stays a human obligation. The list is recorded as verified against v2.10.2. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Assisted-By: LLM |
||
|
|
f34a7f5ea1 |
fix(linstor): refuse drbd.enabled=false on a non-Talos substrate
drbd.enabled=false removes the drbd-logger sidecar, and that is all it removes. The satellite DaemonSet also carries two DRBD-only initContainers that piraeus-operator contributes unconditionally -- drbd-module-loader, which cannot load a module the substrate does not have, and drbd-shutdown-guard -- and the only configuration that deletes them is satellites-talos.yaml, gated on talos.enabled. So with talos.enabled=false and drbd.enabled=false the chart rendered a satellite that can never reach Ready: exactly the state the flag exists to prevent, reached by setting the flag. Not hypothetical -- the isp-full-generic bundle sets talos.enabled=false (packages/core/platform/templates/bundles/system.yaml). Fail at render time and name both flags. The alternative, moving the two initContainer deletions and their volumes out from under talos.enabled, is the real fix for generic Linux and is a larger change than this lane needs; it is tracked in cozystack/cozystack#4092, and the message points there. Refusing is what keeps an operator from getting no signal at all in the meantime. values.yaml states the requirement beside the flag. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Assisted-By: LLM |
||
|
|
12157a2f93 |
docs(agents): describe the container lane as the merged standard
The container lane moves out of "In-flight direction (not yet the merged standard)" and into the body, because it is now what both merge-gating lanes run. The node-join section is rewritten the other way round: it described a deadline that failed the suite without blocking the lane, and there is no longer any such carve-out to describe. What the section gains is the part that was missing while the lane was a spike -- what moves off the per-PR path with the substrate, and where it still gets covered. DRBD and the replicated StorageClass were already named; Immediate binding, tenant StorageClass propagation and the Cozystack Talos node image were not, and all three now say nightly and release-candidate rather than reading as lost. The host CPU and memory visibility gap is stated too, since it is what a reader meets first on a red run and it looks like a product bug. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
186547d835 |
test(e2e): run the merge-gating lanes on Talos containers
The srv1-srv3 nodes of both lanes that gate a merge -- the same-repo one in pull-requests.yaml and the fork one in e2e-fork.yaml -- run as Talos containers instead of QEMU guests, so a tenant worker sits at L2 rather than L3. Measured in CI on commits where both substrates ran the same tree: whole-job wall clock down 24-49 minutes, essentially all of it in Chainsaw, while the two tenant-Kubernetes suites go from 1143-2040s to 466-809s and stop depending on the runner's vgif flag. The node-join soft-red gate goes with the substrate, which is what it was for. It existed because nested virtualisation on the shared runners degraded far enough that a worker registering in minutes elsewhere could miss any deadline this test could afford; #3513 established that the discriminator was the host's kvm_amd vgif flag, and removing a nesting level removes what it acted on. So hack/e2e-node-join-soft-red.sh, the SOFT-RED-node-join.txt marker and the soft_red job outputs are gone from all four workflows, and a missed deadline is an ordinary red again. The ::warning annotation survives, because the reason a run went red should be legible without opening the log, and so does the 124-versus-everything -else test, because "the workers were slow" and "the wait never ran" send a reader to different places. Done by swapping the substrate inside the existing e2e jobs rather than by promoting a separate container job. Those jobs carry more than a substrate -- the GitHub App token, the report-overrun guard, the images list, the SSH breakpoint -- and promoting a job that never had them would have dropped four features without saying so. Three properties of container mode drive the design, and each fails quietly rather than loudly. machine.kernel.modules is a silent no-op -- kernel_module_spec.go returns early on ModeContainer with no error and no event -- so the host loads openvswitch and zfs before the nodes start and the workflow step asserts both rather than trusting them; a missing module surfaces an hour later looking like a CNI or a storage regression. Docker gives a privileged container its own tmpfs /dev seeded once at creation, so /dev/kvm works but ZFS zvols created later never appear and linstor-csi fails ControllerPublishVolume; the compose file binds the host devtmpfs, which is also why talosctl cluster create docker can never carry LINSTOR. And there is no ARP VIP, so 192.168.123.11 stands in for the QEMU lane's .10. The QEMU machinery stays in the tree and keeps running: nightly.yaml and e2e-tag.yaml still call `make prepare-env`, so DRBD, the replicated StorageClass, its Immediate binding mode, the tenant StorageClass propagation that only applies to remotely-accessible classes, and the Cozystack Talos node image with its extensions all keep nightly and release-candidate coverage. None of them is exercised per pull request any more. That is the trade. Nothing on the PR path builds or downloads a nocloud disk now, so build-talos drops the talos-nocloud step and the talos-image artifact. build-talos itself stays: finalize consumes its digest fragment. One property this does not fix, recorded beside the runner class because it will be read off a red run eventually: a container node reports the HOST's CPU and memory as capacity, and hack/e2e-container-up.sh counters that with kubelet systemReserved. That corrects the scheduler and not what a pod reads from /proc, so a workload sizing itself from /proc/cpuinfo or /proc/meminfo is configured for the whole runner. Also carried here because the same files carry them: the ghcr.io pull-through mirror removal (see #4007 -- whichever lands first makes the other a no-op), the authoritative HelmRelease readiness gate, the backup-credential preflight, the prepull guard against an image-less container fragment, and folding the two standalone OIDC suites into the latest-version tenant cluster. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
e49633208c |
ci(tests): parallelize unit test targets
The BATS half of `make unit-tests` ran as one shell loop over ~60 files, which is a nine-minute serial tail no amount of runner CPU could touch. Each file becomes its own Make prerequisite instead, so `-j4` schedules them, and the workflow asks for `make unit-tests test-controllers -j4 --output-sync=target` in a single step: one make process owns all four runner CPUs, the Helm and Go targets share the same bounded pool rather than waiting for each other, and output stays grouped by target instead of four logs interleaving. The existence check moves into its own prerequisite rather than disappearing. It guards against the wildcard matching nothing, which would otherwise make the whole target succeed by having no work. COZYTEST_TRACE rides per target. It arrived with the serial loop it now replaces, and a target that dropped it would go back to streaming full xtrace for every suite -- so the parallelism guard requires the prefix rather than tolerating it. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
5eaf44b8ca |
fix(vm-disk): bind standalone disks immediately
A VMDisk is a standalone object that is populated before any VM consumes it, so on a WaitForFirstConsumer StorageClass CDI needs an explicit signal to schedule its worker and start provisioning. Without it the DataVolume sits in PendingPopulation indefinitely and the disk is never filled -- and nothing says so. This is a consistency fix rather than a new behaviour. The chart default `replicated` is volumeBindingMode: Immediate, so a disk on it already bound and populated at creation; `local`, the cluster's default class, is WaitForFirstConsumer, so the same VMDisk silently did nothing there. The annotation was previously emitted only for source: upload, which is the one source that could not work without it, leaving http, image, disk and blank to differ by class. Two consequences worth knowing. CDI checks the annotation by presence and ignores its value, so "" and "true" are equivalent and this is not a behaviour change on the upload path. And on a node-pinned class each disk now binds at its own creation rather than at VM scheduling, so a multi-disk VM on `local` can find its disks on different nodes -- that is the existing limitation of a node-pinned class, which is why the chart default is not one, but it is reached earlier now. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
0418577f55 |
fix(kubevirt-cdi): make importer resources configurable
podResourceRequirements applies to every worker pod CDI creates -- importer, uploader and host-assisted cloner alike, not only the importer the parameter is named for -- so a value set here reaches the console's disk-upload path as well as the imports. It has to be set on the CDI CR and not on CDIConfig: the operator reconciles CDIConfig.spec from this resource, so a direct patch is silently reverted and status.defaultPodResourceRequirements stays where it was. The default restates CDI's own built-in values exactly -- limits 750m/600M, requests 100m/60M, read off a live cluster -- so an install that sets nothing behaves as it did before the parameter existed. That is deliberate. A 4Gi ceiling was measured as necessary on one hand-built stand, where the decompress+convert of a tenant worker disk was OOM-killed at 99.8-100% and CDI then retried from scratch forever, but the same 20Gi import completes at stock 600M on a QEMU-node cluster and no CI run on either lane has recorded an OOMKilled importer. What has been shown to need the headroom is that substrate, so the override lives with the substrate in hack/e2e-platform-packages.sh rather than here. The failure being answered is memory, and only memory. CDI already ships a CPU limit of its own at 750m, nothing measured has asked for more, and the default here restates that figure rather than raising it. The package had no `test` target, and hack/helm-unit-tests.sh only runs a package whose `make -n test` resolves, so a tests/ directory here was collected by nothing and skipped in silence. Added with the suite. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Assisted-By: LLM |
||
|
|
499034e960 |
fix(linstor): constrain CSI provisioning to the scheduler's chosen node
Without --strict-topology, external-provisioner passes EVERY topology segment as `requisite` and merely *prefers* the node the scheduler picked, so when that node is out of space linstor-csi may legally provision the volume somewhere else. For a node-pinned volume (allowRemoteVolumeAccess: false, the `local` class that is the cluster default) the result is a PV whose node affinity contradicts the pod's other volumes, and the pod is then unschedulable forever against pv "pvc-..." node affinity doesn't match node "srv1": no matching NodeSelectorTerms with nothing erroring anywhere. It bites hardest on CDI imports, because an importer pod mounts two volumes: the disk, and a ~21 GiB scratch volume CDI creates only AFTER the pod is scheduled. The scheduler therefore sizes the node against the disk alone, can place two importers where both disks plus both scratch volumes do not fit, and the second scratch lands elsewhere. Capacity tracking cannot help -- the scratch volume does not exist when placement is decided, and CSIStorageCapacity agreed with LINSTOR to within 5% when this was measured. Reproduced on 2026-08-25 by cordoning two of three nodes so both importers landed on one constrained to 45 GiB: enough for two 20 GiB disks, not for their scratch. With the flag the scratch PVC stays Pending and the pod reports `1 node(s) did not have enough free storage`, then binds and completes once space frees. A silent permanent deadlock becomes a legible, self-correcting wait. The full argument list is restated because `args` has no patchMergeKey, so a strategic merge replaces rather than appends. That makes the list a drift hazard on every piraeus-operator bump, and the note beside it says so -- the count assertion in the test guards this chart's own list and cannot see a flag the operator adds upstream. Verified for node-pinned volumes. For `replicated` the PV is remotely accessible so the constraint is satisfiable wherever the pod lands, and that case is reasoned rather than measured -- hence the switch. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
06ee08f636 |
feat(linstor): gate the DRBD sidecar behind drbd.enabled
On a substrate where the DRBD kernel module is unavailable the drbd-logger sidecar exits immediately, the satellite pod is therefore never Ready, and piraeus consequently registers zero nodes: LINSTOR reports none, linstor-csi-node hangs in Init polling for a node that does not exist, and every PVC stays Pending behind a cluster that looks healthy. So this flag is the difference between working storage and no storage at all there, not a cosmetic reduction in logging. The concrete case is Talos nodes running as containers. DRBD's resource registry is global per kernel, so three containers sharing one kernel cannot represent three DRBD nodes -- the second new-resource for a given name silently aliases the first instead of failing, which is worse than an error because LINSTOR then records a replica it does not have. It applies to kind equally; it is a property of one kernel, not of Talos. Omitting the sidecar from Cozystack's own pod template is not enough, because piraeus strategically merges satellite configurations and a container another configuration contributes survives the omission. So a LinstorSatelliteConfiguration deletes it from the effective DaemonSet as well. What the flag does NOT do is touch a StorageClass, and the comment says so explicitly. Cozystack does not own them -- an operator writes them by hand from the install guide -- so a class carrying layerList: "drbd storage" still exists after setting this false and volumes on it will never provision. The operator-facing half of that warning goes beside the examples they copy, in cozystack/website#675. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
f8abaa6249 |
docs(agents): correct the labeler's trigger list
The auto-labeling section still lists `edited` among the events the labeler runs on. That trigger was removed in #3211, and the workflow carries a note explaining why: it fires on every body edit, so long-lived bot PRs re-ran the job dozens of times on an unchanged commit. The gap is not cosmetic for the reader this document is written for. Acting on it means correcting a wrong PR title and expecting the labels to follow, which they do not until the next push — and since labels are only added, never removed, the `area/uncategorized` the first title produced stays behind either way. Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
|
7cdfe7a197 |
feat(keda): add the quota-blocked alert and restrict operator egress
Close two follow-ups the design proposal names for this PR: - DatabaseAutoscalerScaleBlocked: the Pending-independent companion to DatabaseAutoscalerScaleStuck. A ResourceQuota (or any admission webhook) refuses the new standby pod at admission, so no pod object is created and ScaleStuck's Pending intersection misses it. Key instead on desired_replicas > count(CNPG pods in the namespace): a healthy basebackup's pod exists (Running) so the count already equals desired, while an un-admitted pod leaves the count short. Correct the stale ScaleStuck comment that claimed it covered the quota case. - Enable the vendored KEDA egress NetworkPolicy (flavor cilium): egress is toEntities [kube-apiserver, cluster] only — apiserver and in-cluster endpoints (CoreDNS, vmselect, CNPG) stay reachable, world (off-cluster) is denied — so the operator cannot be used as an SSRF relay off-cluster. The kube-apiserver entity covers the control plane however it is addressed, so it does not starve the operator's apiserver access the way a raw-CIDR policy could. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
6083a9fa1d |
feat(postgres): guard replicas>maxReplicas and pin serverAddress to in-cluster
Close two render-guard follow-ups the design proposal names for this PR: - Fail closed when replicas exceeds autoscaling.maxReplicas. The stock HPA's currentReplicas > maxReplicaCount branch writes /scale = maxReplicaCount unconditionally on the first reconcile — even under the both-direction pause of dryRun/transition — so a resting/seed count above the ceiling would shed the surplus standbys and their PVCs in one step, past the scale-down pacing. Mirrors the existing effectiveMin > maxReplicas guard. - Refuse an autoscaling.serverAddress whose host is not an in-cluster Service DNS name (…svc[.<domain>]). The cluster-privileged KEDA operator fetches it every poll, so an external/IP address is an SSRF surface (cloud metadata endpoint, another tenant); the query body is already namespace-scoped, this restricts the destination. Defense-in-depth pairs with the operator egress policy (keda package). Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
6935df8c7f |
chore(kubevirt): update KubeVirt to v1.9.0 (#4015)
## What this PR does KubeVirt operator manifest goes from v1.8.4 to v1.9.0. The 1.9 release turns every Beta feature gate on by default and adds `disabledFeatureGates` to the CR for opting out, so the CR side needed a few decisions. `Template` is disabled. With it on, virt-operator deploys virt-template: two Deployments, two APIServices, a webhook and a ValidatingAdmissionPolicy that nothing here uses. One more aggregated API server is one more way to break discovery. This one is not a knob. `- Template` is written into the CR unconditionally and there is no value that takes it back out, because nothing on the platform uses VirtualMachineTemplate; if something starts to, the static entry becomes a value then. Asking for it through `extraFeatureGates` is refused at render time with an error that says so, rather than producing a CR KubeVirt would reject. `ImageVolume` is the host-dependent one: off below Kubernetes 1.35, left on from 1.35, with a platform value for runtimes that cannot serve image volumes even there (#3190 has the k3s case). Nothing the platform ships uses containerDisk. The value is `kubevirt.disabledFeatureGates` on the platform values, which the iaas bundle forwards into the emitted `cozystack.kubevirt` Package as `components.kubevirt.values.disabledFeatureGates`: Closes #3190. ```yaml kubevirt: disabledFeatureGates: - ImageVolume ``` The chart refuses to render a CR that names one gate in both lists. KubeVirt's `kubevirt-update-validator.kubevirt.io` webhook rejects that combination and runs with `failurePolicy: Fail`, so on a cluster that already has the CR the write is refused and the `kubevirt` HelmRelease stops converging with everything else the CR carries. The create admitter runs no validation at all (its own comment says the update validator is where that happens, and it only gets installed after the CR exists), and `IsEnabled` in virt-config checks `FeatureGates` before `DisabledFeatureGates`, so on a fresh install the same CR is admitted and the gate silently stays on. Render-time is the only place left where an operator gets told. A name repeated inside one list is not an error: upstream compares the two lists against each other, and both fields are `x-kubernetes-list-type: atomic` in the CRD, so that CR is accepted and the guard stays quiet. The gates the chart pins off can arrive a second time from a supported values file, since the chart writes them itself and an operator listing the gates their cluster declines has no reason to know that, so the range skips them and the CR states each opt-out once. The guard walks its own copy of the seven gates the chart enables. That copy exists because the rendered list has to stay a literal list for `make update` to find its `- VMExport` anchor, and a gate quietly dropped from the copy would keep rendering under `featureGates` while leaving the guard. The suite carries one case per gate, so removing any single name turns exactly one of the seven red. `HotplugVolumes` stays in the list even though 1.9 deprecates it. A deprecated gate listed in the CR is still enabled, and while it is enabled virt-controller skips the declarative hotplug path. Dropping it would switch on declarative hotplug, but then virt-api rejects the VMI-level `removevolume` subresource on VM-owned VMIs, and kubevirt-csi uses that as a fallback during teardown. The CSI driver has to move off that subresource first. Upstream names no removal release for it: the `Deprecated` state's own doc comment says "discontinued in the following release", but `MultiArchitecture` and `DisableMDEVConfiguration` have been deprecated since 1.8.0 and are still registered at 1.9.0, so that is a direction rather than a date. What makes that opt-out list matter is a semantics change: on 1.8 `isFeatureGateEnabled` ended in `return false`, so a Beta gate nobody listed was off; on 1.9 `IsEnabled` ends in `return fg.State == Beta`. Sixteen gates are Beta at v1.9.0 and none of them is in this chart's `featureGates`, so all sixteen arrive on with the bump. Two are switched back off here, one is switched off below Kubernetes 1.35, and I went through the rest: - `Template`: off. Reason above. - `ExternalNetResourceInjection`: off. With it on, virt-controller stops calling `multus.NetworkToResource`, so a NAD's `k8s.v1.cni.cncf.io/resourceName` no longer becomes a resource request on the launcher pod. That is correct for clusters running a network-resources-injector; this platform ships none, so a VM on an SR-IOV NAD would schedule without its device. - `ImageVolume`: off below Kubernetes 1.35, left alone from 1.35 on. KubeVirt renders containerDisk images as `VolumeSource.Image` volumes when the gate is on, and it does not check the host: `dropImageVolumes` in the apiserver strips every image volume and the mounts referencing it while Kubernetes' own `ImageVolume` gate is off, which is the default until 1.35 (`kube_features.go` has it `Default: false` at 1.31 and 1.33). The pod is admitted without the disk and virt-launcher fails looking for it. The management cluster is supported from 1.33, so on 1.33 and 1.34 the chart keeps the KubeVirt gate off and the containerDisk sidecar path in use; from 1.35 the gate stands and `extraFeatureGates` can name it. The apiserver version is a proxy for a runtime capability, not the capability itself, so a cluster at 1.35 or newer whose container runtime cannot serve image volumes (the k3s case in #3190) switches the gate off through `kubevirt.disabledFeatureGates`; that is what the value is for. - `Snapshot`: on. `VirtualMachineSnapshot` needed the gate listed on 1.8; snapshot and export were exercised on the test cluster. - `DeclarativeHotplugVolumes`: on and inert. `handleDeclarativeVolumeHotplug` returns early while `HotplugVolumes` is enabled, which this chart still does. - `GPUsWithDRA`, `HostDevicesWithDRA`: on; each only lets a VMI request a DRA-provisioned device, and nothing here creates one. - `WorkloadEncryptionSEV`: on; needs SEV hardware and a VM that asks for it. - `PasstBinding`: on; needs a VM using the passt binding. - `RebootPolicy`: on; adds a field on the VMI domain spec, unset means the old behaviour. - `VmiMemoryOverheadReport`: on; fills `status.memory.memoryOverhead`, reporting only. - `OptOutRoleAggregation`: on; it enables the `RoleAggregationStrategy` field, and aggregation stops only for a CR that sets it to `Manual`. This one does not. - `KubevirtSeccompProfile`: on; installs the profile, and a workload uses it only where `seccompConfiguration` says so. - `NodeRestriction`: on; virt-handler node authorization, needs the Kubernetes `ServiceAccountTokenPodNodeInfo` gate, GA since 1.32. - `LibvirtHooksServerAndClient`: on; pre-migration hooks on the target launcher, and no hooks ship here. - `PodSecondaryInterfaceNamingUpgrade`: flips on; it adds `--upgrade-ordinal-ifaces` to virt-launcher and a pre-migration hook that renames the ordinal secondary interfaces of VMs still carrying them. The test cluster's migrations exercised that path. Five of the seven gates the chart enables (`ExpandDisks`, `LiveMigration`, `AutoResourceLimitsGate`, `GPU`, `VMExport`) reached GA by 1.9 and are inert: `IsEnabled` returns true for a GA gate whether or not the CR names it. I left the list as it is; dropping the five is a follow-up, because it changes what the CR says without changing what the cluster does and it belongs in its own commit. QEMU is 10.1.0 on both sides of this jump; the el9 build revision moves from -10 to -20 and libvirt goes 11.9 to 11.10. A moving build revision is the thing to check rather than a major: [kubevirt/kubevirt#16386](https://github.com/kubevirt/kubevirt/issues/16386), where every existing VM lost live migration, was 1.6.2 to 1.7.0 on qemu-kvm 9.1.0-19 to 9.1.0-20, one build apart within the same QEMU version. So I tested it: VMs running on the 1.8.4 launcher live-migrated onto the 1.9.0 one on a three-node cluster. `make update` in `kubevirt-operator` used to `rm -rf templates` before the fetch and took the local PrometheusRule template with it. Now it only overwrites the file it downloads. `make update` in `kubevirt` now greps for every line of the new block before it passes: the range directive, the `disabledFeatureGates:` key, the three static entries, and the guard header that decides which static entries render. Every insert in that target is guarded by one directive it also inserts, so a hand-merge that keeps the directive and drops a sibling line leaves the guard satisfied and the insert skipped. That shape covers the directive and nothing else. What makes it worth a grep per line here is the static entries: they are the only lines those inserts add that are CR content in their own right rather than a wrapper around a value, so they render on every install and their absence cannot surface as a value that stopped taking effect. The CR just goes back to deploying virt-template and to letting KubeVirt skip NAD resource injection. The target cannot rebuild a partial block, so it names the missing line and stops. The payload lines of the other four blocks are exposed the same way; I checked, and both dropping `- {{ . }}` from the `extraFeatureGates` range and dropping the `permittedHostDevices:` key from its `with` render green and lose the operator's values. That is a wider change than this PR and I left it out. Two sharp edges I know about and left. `kubevirt.disabledFeatureGates: [HostDevices]` on a cluster that also has `cozystack.gpu-operator` in `enabledPackages` makes the bundle forward the same gate into both lists, and the platform renders fine while the child kubevirt HelmRelease then stops on the guard: fail-closed, with the gate named, but diagnosed one level below where it was configured. And the version-conditional opt-out is evaluated at render time, so a cluster upgraded from 1.34 to 1.35 keeps `ImageVolume` disabled until something re-renders the chart; the direction is the safe one, since the gate stays off and containerDisk keeps the path it had on 1.8.4. Two things I decided against doing here. The guard's copy of the gate list is pinned against a name being dropped, not against a name being added to the rendered list only; the copy exists because `make update` needs the literal `- VMExport` anchor, so closing the other direction means changing the self-heal contract. And `hack/run-kubernetes-block-io_test.bats` still names a `virt-launcher:v1.8.4` image in a fixture: inert, the test matches on the container name, and the file is outside this change. Tested on a three-node Talos v1.13.6 / Kubernetes v1.36.2 cluster with Flux suspended. The upgrade took about ten minutes. virt-operator had already deployed virt-template before the CR change landed and removed it afterwards, CRDs included. Two VMs started on the 1.8.4 launcher, one DataVolume-backed and one containerDisk-backed, live-migrated to the 1.9.0 launcher. VMDisk plus VMInstance through the apps API boots. `addvolume` on the VM subresource reaches Ready, `removevolume` on the VMI subresource is accepted. VirtualMachineSnapshot and VirtualMachineExport from it reach Ready. Restore from snapshot and PVC clone failed on that cluster inside the LINSTOR API layer, unrelated to this change. CDI v1.66.0 from #3920 on top of it: HTTP imports to Block and Filesystem, blank population and hotplug all work. `helm unittest` is 26/26 for the kubevirt chart and 142/142 for the platform chart, and the update idempotency script passes its eleven cases. ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map against the diff. No app added or renamed, no variant, release asset, `ApplicationDefinition` or tooling contract touched, and none of the `packages/core/platform/values.yaml` keys the ansible role sets are in the diff. One repository is reached. `packages/core/platform/values.yaml` gains a top-level key, `kubevirt.disabledFeatureGates`, defaulting to the empty list, and `operations/configuration/platform-package.md` on the website is a hand-written table of `spec.components.platform.values.*` that nothing generates. The row is in cozystack/website#678, which lands after this PR merges. - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/pull/678 - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note chore(kubevirt): update KubeVirt to v1.9.0. Beta feature gates are on by default upstream now. The platform keeps `Template` off, which would deploy virt-template, and `ExternalNetResourceInjection`, which would hand `resourceName` injection for NetworkAttachmentDefinitions to an injector the platform does not run. `ImageVolume` is disabled on management clusters below Kubernetes 1.35: containerDisk-backed VMs are rendered with native image volumes when it is on, and an apiserver whose own ImageVolume gate is off, the default before 1.35, drops those volumes silently, leaving the VM without its disk. On 1.35 and newer the gate stands, and that is the first release where containerDisk-backed VMs are rendered with image volumes instead of sidecars; the bundled Talos ships Kubernetes 1.36, so this is the default there. A `kubevirt.disabledFeatureGates` list in the platform values opts out of any other gate a cluster cannot carry. ``` |
||
|
|
dde7c5099b |
docs(postgres): correct the apply-strategy annotation comment to "two-way"
The ApplicationDefinition annotation comment called client-side apply a "three-way-merge", contradicting the corrected wording everywhere else in the change (config.go, _autoscaling.tpl, db.yaml, values.yaml all say the client-side patch diffs the previous vs new render and does not consult live state). For a CRD, helm-controller v1.5.0 leaves threeWayMergeForUnstructured at its default, so Helm takes the two-way jsonpatch.CreateMergePatch(original, modified) path. This comment ships on the live ApplicationDefinition, so a maintainer reading it in isolation would otherwise get the wrong mental model of the safety property. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
daf733ad9c |
fix(keda): narrow autoscaler RBAC and scope the operator-down alert
Restrict the KEDA operator ClusterRole: set permissions.operator.restrict.secret so it drops cluster-wide secret list/watch, and pin rbac.scaledRefKinds to postgresql.cnpg.io/clusters so it no longer holds a wildcard */scale across every group. Gate DatabaseAutoscalerKedaOperatorDown on the presence of keda-hpa-* HPA series so a tenant running its own Monitoring app (which never scrapes keda_build_info, root-only) does not fire it forever about a healthy platform component. Make the two negative containsDocument matchers in the bundles-default test non-vacuous (any: true) so "KEDA is not installed by default" is actually asserted. Drop the unused pollingInterval/cooldownPeriod params from the cozy-lib keda helper. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
c693ed36fb |
fix(postgres): force client-side apply so the autoscaler seed survives
Under active read-replica autoscaling the chart seeds spec.instances at a constant max(replicas, effectiveMin) rather than omitting it (an omitted field defaults to 1, which the CNPG quorum webhook rejects, and — under server-side apply — the phase-2 omit prunes the field whenever KEDA has not yet written /scale, collapsing an idle-at-floor cluster to 1 and shedding its replica PVCs). Keeping the constant only works if Flux does not force-own and revert it. The postgres ApplicationDefinition therefore carries release.cozystack.io/helm-server-side-apply: "false", so the tenant HelmRelease applies client-side: helm-controller patches the CNPG Cluster from the previous-vs-new rendered manifest without consulting live state, the unchanged constant yields no patch, and KEDA's higher live /scale count is preserved. The constant also equals the disabled/dryRun/transition render (instances: replicas) whenever replicas sits at/above the floor, so those flips are patch-free too. Correct the merge-semantics comments across db.yaml, _autoscaling.tpl and values.yaml to describe the two-way previous-vs-new patch precisely (not a "three-way no-op"), and document the residual floor-change hazard: editing replicas / minReplicas / quorum.maxSyncReplicas moves the seed, so that one reconcile does patch spec.instances to the new seed. Add autoscaling.serverAddress so a monitoring:true tenant can point the ScaledObject at a vmselect that carries both the CNPG metrics and kube_pod_labels. Regenerated README/schema/types/cozyrds. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
27d7dbb140 |
feat(api): add per-app helm apply-strategy override annotation
Add release.cozystack.io/helm-server-side-apply on the ApplicationDefinition, threaded through ReleaseConfig into every emitted HelmRelease's Install/Upgrade.ServerSideApply. Empty leaves the helm-controller default (server-side apply on v1.5.0); "false"/"disabled" forces client-side apply. The postgres read-replica autoscaler needs this: KEDA's HPA writes the CNPG Cluster's /scale subresource (spec.instances) while the chart renders a constant seed. Under helm-controller's default server-side apply the rendered field is force-owned and reverts KEDA's live value on every reconcile. Under client-side apply helm-controller patches the CRD from the previous-vs-new rendered manifest without consulting live state, so the unchanged constant yields no patch and the /scale value survives (verified against Helm v4 createPatch: unstructured/CRD objects take the two-way jsonpatch.CreateMergePatch(original, modified) path unless threeWayMergeForUnstructured is set, which helm-controller v1.5.0 leaves at its default). Guarded by TestConvertApplicationToHelmRelease_ServerSideApplyOverride on the tenant-release builder (convertApplicationToHelmRelease), and by TestParseHelmServerSideApplyAnnotation. Removes a stale package_reconciler_test guard that asserted a false invariant on the wrong (platform Package) builder. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
826f08972f |
fix(apps/postgres): make autoscaler fail-safe, dip-free, and observable
Address review findings on the KEDA read-replica autoscaler where the
implementation diverged from the merged design proposal.
Fail-safe read load. Gate the `or vector(0)` floor on the existence of ANY CNPG
instance pod (`count(kube_pod_labels{...instance_role=~".+"}) > 0`) — primary or
replica — so the query is empty ONLY when no CNPG pod labels exist at all (a
broken kube-state-metrics pipeline), not during initial provisioning when the
sole replica-side pod is CNPG's join Job (jobRole, not instanceRole). Also pin
`ignoreNullValues: "false"` on the trigger: KEDA's prometheus scaler defaults it
true, turning an EMPTY result into metric 0 (not an error) — the HPA would then
desire 0 and clamp to the floor, i.e. silently scale a broken-pipeline cluster
DOWN. With it false an empty result is a scaler error, so the HPA HOLDS and
DatabaseAutoscalerScalerErrors fires; gating on any instance role keeps a healthy
cluster (including a long basebackup) from tripping that alert.
Constant seed = max(replicas, effectiveMin), so enable/dryRun/transition flips
are a merge no-op with no dip and never undercut the quorum floor.
Fail closed when the quorum floor exceeds maxReplicas (was: effectiveMax silently
raised both bounds, unbounded for a large maxSyncReplicas).
Dry-run / transition keep the HPA (unidirectional pause pair, not the HPA-deleting
full pause) but both SelectPolicy=Disabled, so desired normalizes to current — the
recommendation is read from the read-load metric, not a diverging desired count.
Observability. DatabaseAutoscalerAtMax keys on desired >= maxReplicas, NOT the
ScalingLimited condition (which is also true at the floor, so it fired on every
idle cluster with wrong "raise maxReplicas" advice). Dashboard read-load panels
are per-cluster (label_cnpg_io_cluster) with a CPU panel for ReadCPUUtilization
and the exact clamp in descriptions; ScaleStuck's Pending intersection is scoped
to CNPG pods; alerts.yaml narrative comments moved to Go-template comments so they
do not ship into the PrometheusRule.
Guard the seeding invariant: TestBuildHelmReleaseSpecNoForceOrDriftDetection.
Regenerated README/schema/types.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
|
||
|
|
8769232997 |
docs(apps/postgres): correct autoscaling merge-safety notes
The disable case (3) note claimed disabling snaps instances back to .Values.replicas, but the client-side three-way merge compares the newly rendered value against the previously applied one, not the live count. When replicas == effectiveMin (the default) it is a no-op and KEDA's live count is preserved (silent over-provisioning), so it does not snap back. Reword case (3) and the "differs from the live one" framing to match the actual merge semantics. Re-point the no-Force/no-SSA invariant citation from internal/controller/applicationdefinition_helmreconciler.go (which holds no apply-strategy code) to internal/operator/package_reconciler.go buildHelmReleaseSpec, where the HelmRelease spec is actually built. Comment/doc only; helm unittest 52/52 unchanged. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
b40cf82642 |
fix(postgres): seed instances on both paths, add input guards, and harden alerts
Address IvanHunters' deep review round on #3954: - [MAJOR] Enabling autoscaling directly on an existing non-quorum database no longer collapses it to 1 instance and deletes every replica PVC. The chart now SEEDS spec.instances at the quorum floor (effectiveMin) on BOTH the quorum and non-quorum active paths instead of omitting the field in the non-quorum case; the constant seed is a three-way-merge no-op so KEDA still owns the runtime count via /scale, and the enable dip is capped at the floor (>=2) rather than collapsing to 1. Documents the client-side-3-way-merge (not SSA) assumption that the no-op relies on. - [MINOR] Fail closed on autoscaling.minReplicas > autoscaling.maxReplicas instead of silently resolving the ceiling upward (which defeated maxReplicas as a cost knob). The stray maxReplicas: 0 case now rejects rather than clamps. - [MINOR] The disable-wedge guard, its values note and its test no longer emit the invalid `kubectl scale --subresource=scale` (scale has no --subresource); use `kubectl scale cluster.postgresql.cnpg.io/<name> --replicas=N`. - [MINOR] Correct the enablement note: raising minReplicas/maxSyncReplicas on a live cluster is NOT harmless when KEDA's live count exceeds the new floor (it decommissions surplus replicas + PVCs); prescribe pin-first for raises too. - [MINOR] Add DatabaseAutoscalerKedaOperatorDown (absent(keda_build_info)): when the operator is unhealthy a ScaledObject never becomes an HPA and no other alert can fire. Intersect DatabaseAutoscalerScaleStuck with a Pending pod so a healthy long basebackup (Running pod, desired>current >15m) no longer false-fires. - [MINOR] Keep the KEDA CRDs on package uninstall (crds.additionalAnnotations resource-policy: keep via the shim) so removing the keda package doesn't cascade-delete every tenant ScaledObject. - [MINOR] cozy-lib helper: reject a non-numeric bound (int "two" -> 0 -> a scale-to-zero ScaledObject) and render an explicit pollingInterval/cooldownPeriod of 0 (`if not kindIs invalid`, not `with`). - [MINOR] Note the read-load fail-open: a broken metrics join reads as idle and silently pins autoscaling at the floor under real load. New/updated tests cover the non-quorum seed, the min>max guard, the corrected kubectl message, the operator-down + Pending-intersected alerts, the CRD keep policy, and the non-numeric-bound helper guard. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
1ebc021491 |
style(postgres): strip the autoscaling db.yaml comments from the rendered manifest
The autoscaling instance-count explanations were plain `#` YAML comments, so they
shipped verbatim into every rendered CNPG Cluster manifest on every customer
cluster (unlike the {{/* */}} comments elsewhere in the chart). Convert them to
Go-template comments so the rationale stays in-source but is stripped at render,
matching the treatment already applied to the apps.yaml clusterDomain comment.
Also document (as a stripped comment) that WorkloadMonitor.spec.replicas tracks
the quorum floor, not KEDA's live autoscaled target, under active autoscaling.
Addresses the deep cozy-review notes on #3954.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
|
||
|
|
5a5047ac23 |
fix(dashboards): bind the autoscaling dashboard to a datasource variable
The database-autoscaling dashboard hardcoded "datasource": "VictoriaMetrics" on
its namespace template variable, but cozystack provisions VictoriaMetrics
datasources as vm-shortterm / vm-longterm — there is no datasource named
"VictoriaMetrics", so on import the variable query failed ("datasource not
found"), the namespace variable never populated, and every panel (which filters
on namespace=~"$namespace") showed no data. Switch to the repo-standard pattern
used by db/cloudnativepg and the other dashboards: a `datasource`-typed template
variable (ds_prometheus) referenced as { type: prometheus, uid: ${ds_prometheus} }
from the namespace variable and every panel/target (the panels previously carried
no datasource at all and only worked by falling back to the default).
Also reframe the disable-wedge guard message to lead with the invariant (replicas
must exceed quorum.maxSyncReplicas) so a pure-quorum user who never enabled
autoscaling isn't told to "disable autoscaling"; the disable path is kept as a
parenthetical cause.
Addresses the adversarial branch-review findings on #3954.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
|
||
|
|
6e3da63586 |
fix(postgres): guard the disable-wedge, role-scope the freeze count, and harden edges
Address the remaining IvanHunters review points on #3954: - [MAJOR] Disabling autoscaling wedged the HelmRelease under synchronous quorum. An autoscaled + maxSyncReplicas>=1 cluster runs with instances seeded from effectiveMin while replicas stays at its default, so flipping autoscaling off rendered instances: <replicas> next to the quorum and CNPG's webhook rejected it ("maxSyncReplicas must be lower than the number of instances"). Add a symmetric render-time guard on the static-count path (covers off/dryRun/ transition): fail with an actionable message when replicas <= maxSyncReplicas. - [MINOR] Role-scope the freeze brake's currentInstances count (label_cnpg_io_instance_role=~".+") so lingering CNPG join/bootstrap Job pods (cnpg.io/cluster but jobRole, not instanceRole) don't inflate it and nudge desired up while the brake is meant to hold it. - [MINOR] Default maxReplicationLagSeconds to 0 (brake opt-in): only the non-braking path is validated live, so ship the validated default and let the freeze be enabled explicitly until its branch is exercised under load. - [MINOR] Document the floor-change and non-quorum-disable dips (same three-way- merge hazard) and the pin-replicas-first mitigation. - [NIT] Coerce the cozy-lib helper's `paused` (eq "true") so a caller passing the string "false" does not pause the ScaledObject. - [NIT] Trim the keda.clusterDomain injection comment so it no longer ships ~8 verbatim lines into the cozystack-values Secret on every cluster. Regenerate types/schema/README/postgres-rd. New tests cover the disable-wedge guard, the role-scoped freeze count, and the string-"false" paused coercion. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
e8609ac00d |
fix(keda): ship the autoscaling dashboard via grafana-operator, not a dead ConfigMap
The database-autoscaling dashboard shipped from the keda package as a ConfigMap labeled grafana_dashboard: "1" for a Grafana sidecar that cozystack does not run (no k8s-sidecar anywhere; every dashboard is provisioned by grafana-operator from the grafana-dashboards image via GrafanaDashboard CRs). It was enabled by default and never appeared in any Grafana. Move the JSON to dashboards/db/database-autoscaling.json and register db/database-autoscaling in packages/system/monitoring/dashboards.list, exactly how db/cloudnativepg and the other managed-db dashboards ship, and drop the keda ConfigMap, its default-on toggle, and its wrong-mechanism test. Addresses IvanHunters [MAJOR] review feedback on #3954. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
70e4087432 |
fix(postgres): guard empty monitoring namespace; clarify alert scope and helper doc
Address the branch-review nits: - Fail closed when the monitoring namespace is unset. serverAddress builds vmselect-shortterm.<ns-monitoring>.svc, and an empty ns-monitoring rendered the malformed vmselect-shortterm..svc:8481, surfacing late as a KEDA scaler error. The platform always injects it, but a hand-rendered/misconfigured tenant now gets an actionable render-time failure mirroring the KEDA-CRD and target guards. Tested. - Document that the database-autoscaling PrometheusRule matches every KEDA HPA / ScaledObject error (the underlying series carry no owning-app label), which is correct only while database autoscaling is KEDA's sole consumer in cozystack; note to scope the rules if a non-database ScaledObject is ever added. - Correct the cozy-lib.keda.scaledObject usage example: `paused` should reflect dry-run OR migration transition, not dry-run alone, so a second engine copying the example does not drop the transition pause. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
9cf4a171f0 |
fix(postgres): stop coercing numeric-zero autoscaling bounds and floor each lag gate
Two review notes from IvanHunters: - effectiveMin/effectiveMax wrapped minReplicas/maxReplicas/maxSyncReplicas/ maxReplicationLagSeconds in Sprig `default`, which treats a numeric 0 as empty and silently substitutes the fallback (e.g. maxReplicas: 0 became 6, discarding the tenant's value). values.yaml/schema already default these when omitted, so read them directly; an explicit out-of-range value now flows into max() and is clamped to the quorum floor rather than swapped for a different number. Add a test that maxReplicas: 0 clamps to effectiveMin instead of becoming 6. - The replication-lag brake only floored the product $braking with `or vector(0)`, so if a CNPG build stops emitting cnpg_pg_replication_lag or cnpg_collector_wal_records the individual gate resolved to an empty vector. Floor each gate ($lagHigh, $writing) individually so the fail-open decision is explicit and per-gate: the brake engages only on positively-observed lag AND writing, and a missing signal degrades to "scale normally" (safer than freezing on absent data) rather than relying on the emptiness of the product. Assert both gates carry the floor. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
24fe1f802a |
test(cozy-lib): cover the non-map scaleTargetRef guard
Close the last uncovered rejection path of cozy-lib.keda.scaledObject: a non-map scaleTargetRef now has a failedTemplate assertion, so all twelve fail-closed guards are executable-documented. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
a92cb5b5f8 |
test(cozy-lib): cover the min/max required guards of keda.scaledObject
Complete the helper's rejection-path coverage: add the two remaining fail-closed assertions (minReplicaCount / maxReplicaCount required) so all eleven guards are executable-documented. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
2f294c65cd |
test(cozy-lib): cover every fail-closed guard of the keda.scaledObject helper
The helper validates ~10 required inputs but the suite only exercised three rejection paths (non-dict arg, missing scaleTargetRef.name, inverted bounds). Add a parameterised fixture that builds a fully-valid argument dict and drops exactly one required field per case, and assert the fail message for name, scaleTargetRef.kind, scaleTargetRef.apiVersion, query, threshold and serverAddress. The helper's rejection contract is now executable-documented in full rather than sampled. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
42a533fc50 |
fix(platform): inject networking.clusterDomain into the KEDA package
KEDA's clusterDomain is a vendored-subchart value baked into the metrics-apiserver gRPC target (keda-operator.<ns>.svc.<clusterDomain>:9666) and the operator's serving-cert SANs at render time. A chart template cannot derive a subchart value from _cluster, so the keda package hardcoded cozy.local — which silently breaks (NXDOMAIN, every DB HPA FailedGetExternalMetric) on any cluster whose domain is not the default. Inject the authoritative networking.clusterDomain into the shared cozystack-values Secret under the keda subchart key so the value tracks the platform config; the keda package keeps cozy.local as a standalone fallback. The key is ignored by packages without a keda subchart (no app/extra schema sets root additionalProperties:false). Add platform unittests asserting the Secret carries keda.clusterDomain and that it tracks a non-default networking.clusterDomain. Addresses the hardcoded-clusterDomain portability gap found in branch review. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
a7b5077da2 |
fix(postgres): floor the lag-brake frozen term so a KSM gap can't empty the query
The read-load query's base term is floored with `or vector(0)`, but the replication-lag brake's frozen additive term (count(kube_pod_labels) * target) was not. PromQL `+` is a set intersection, so if count(kube_pod_labels) is momentarily empty — a kube-state-metrics scrape gap or KSM rollout — the frozen term is empty and the whole `base + frozen*braking` sum collapses to empty, defeating the base term's own graceful degradation. KEDA's prometheus scaler (ignoreNullValues: true by default) then reads that as 0 and pins desired to minReplicaCount, ignoring real read load exactly when metrics are flaky. This hits the default config (maxReplicationLagSeconds: 30, brake active). Floor the frozen summand with `or vector(0)` so base + 0 = base, and assert the floor in the lag-brake unittest. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
573213ef8b |
test(postgres): assert the standby-only read-load selector in the query
Cover the load-bearing replica selector that both read-load queries depend on: the join must restrict to standby pods via label_cnpg_io_instance_role="replica", the exact key kube-state-metrics exposes for CNPG's cnpg.io/instance_role. A wrong key (e.g. a camelCase label_cnpg_io_instanceRole) would match no series, floor the read term to 0, and silently pin desired to the minimum - the suite previously never asserted the selector at all. Verified live on dev9 (CNPG 1.30.0) that the snake_case key is the one present in kube_pod_labels and the camelCase form does not exist. Adds the assertion for both the ReadConnections and ReadCPUUtilization paths. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
842a59a957 |
fix(postgres): correct the migration-dip docs and pin the KEDA metricType
Two review findings: - The enablement note claimed the two-phase `transition` flow avoids the count dip when enabling autoscaling on an existing cluster. It does not: helm-controller upgrades with Helm's three-way merge (not SSA), which keys off the old-vs-new manifest diff, not KEDA's live /scale value. At phase 2 the chart stops rendering the staged static count, so the field is deleted (no quorum -> CNPG defaults to 1, a brief collapse) or changed (quorum -> rebased to effectiveMin), then the warm KEDA restores it. Rewrite the docs to state the flow reduces (KEDA is pre-warmed, no cold-start) but does not eliminate the dip, and spell out the suspend + scale-subresource pin that actually makes the handoff dip-free. This is the SSA-vs-three-way-merge handoff the design proposal left for the PoC to settle. - Pin metricType: AverageValue on the Prometheus trigger in cozy-lib.keda.scaledObject. The desired = ceil(query / threshold) identity holds only for an AverageValue external metric; it was relying on KEDA's default. Make it explicit and assert it so a re-vendor or default change cannot silently break the arithmetic. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
87b26d6595 |
fix(postgres): only seed spec.instances when synchronous quorum needs it
Narrow the previous unconditional seed to a conditional one, keeping design
proposal §3 (chart omits the field, HPA is the sole writer) intact for the
default path. Without synchronous quorum (maxSyncReplicas=0) CNPG's defaulted
instances=1 is admissible, so the field is omitted exactly as §3 specifies and
KEDA owns it via /scale with nothing contested. Only when maxSyncReplicas>=1 -
where an omitted/defaulted instances=1 fails the admission webhook
("maxSyncReplicas must be lower than the number of instances") - does the chart
seed the quorum floor (effectiveMin) so a fresh Cluster is admitted; the constant
re-render keeps three-way merge from reverting KEDA's higher live value.
This confines the deviation to the case §3 did not consider (sync quorum), so no
design-proposal amendment is needed for the documented path. Verified live on a
dev cluster: the quorum render (instances=2, maxSyncReplicas=1) and the
non-quorum render (instances omitted) are both admitted by the CNPG webhook. Adds
a render test for the non-quorum omit path alongside the quorum seed tests.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
|
||
|
|
610bcb5df9 |
fix(postgres): seed spec.instances at the quorum floor under autoscaling
Under active autoscaling the chart previously omitted .spec.instances entirely,
letting the KEDA HPA own it via /scale. But CNPG defaults an absent
.spec.instances to 1, and its admission webhook rejects a Cluster where
maxSyncReplicas >= instances ("maxSyncReplicas must be lower than the number of
instances"). So a fresh database created with autoscaling.enabled: true AND
synchronous quorum (maxSyncReplicas >= 1) was rejected at admission before KEDA
could ever scale it to effectiveMin (= maxSyncReplicas+1) — the whole tenant
HelmRelease failed. The default maxSyncReplicas: 0 path was unaffected, which is
why it went unnoticed. Verified live: a rendered Cluster with maxSyncReplicas: 1
and an omitted instances is rejected by the dev-cluster webhook; with the seeded
floor it is admitted.
Render instances: effectiveMin under active autoscaling instead of omitting it,
so a fresh Cluster starts at a valid, quorum-satisfying count. Because the chart
emits the same constant floor on every reconcile, Helm's three-way merge is a
no-op on the field and never reverts KEDA's higher live value, so the HPA still
owns the runtime count. Update the enablement docs (a new DB can enable directly
even with synchronous replication; existing DBs above the floor rebase to it
unless the two-phase transition flow is used) and add render tests for the fresh
create + maxSyncReplicas>=1 case.
Addresses the fresh-create-with-quorum admission gap found in branch review.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
|
||
|
|
284f5e5543 |
fix(postgres): fail closed when autoscaling is enabled without KEDA, guard target
autoscaling.enabled is a tenant-facing knob, but KEDA is an optional platform package that is not installed by default. A tenant enabling autoscaling on a cluster without KEDA would render a ScaledObject against an unknown kind and the whole database HelmRelease would fail with an opaque `no matches for kind "ScaledObject"`. Guard the ScaledObject template on `.Capabilities.APIVersions.Has "keda.sh/v1alpha1"` and stop with an actionable message instead. Skipping the render is deliberately NOT an option: db.yaml already drops the static .spec.instances under active autoscaling, so a missing ScaledObject would let CNPG collapse the cluster to 1 instance. Document the KEDA prerequisite on the `enabled` field (README/schema). Also reject a non-positive autoscaling.target at render time: it is a plain integer now, but the desired-replica formula divides the read load by it, so 0 is undefined and negatives invert scaling. cozyvalues-gen has no minimum directive, so the bound is a template guard. Add unittests for both guards (a dedicated suite renders without the KEDA CRD so the fail-closed path is exercised; the main suite now advertises keda.sh/v1alpha1 via capabilities). Addresses IvanHunters (KEDA-absent) and CodeRabbit (non-positive target) review feedback on #3954. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
ec123649bf |
fix(keda): use the KEDA 2.20 metric name in the scaler-errors alert
The DatabaseAutoscalerScalerErrors rule queried keda_scaler_errors_total, which KEDA 2.20 no longer exposes (verified against a live 2.20.2 operator: it emits keda_scaled_object_errors_total for per-ScaledObject errors and keda_scaler_detail_errors_total for per-scaler errors). The alert would have silently never fired. Switch to keda_scaled_object_errors_total and add a unittest assertion pinning the metric name so a rename regresses loudly. Addresses CodeRabbit review feedback on #3954. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
f8fc2d99bf |
fix(postgres): make autoscaling.target an integer and drop the dead root metric
Two review findings on the generated API: - autoscaling.target was a resource.Quantity, which accepts suffixed values like `150m`. The chart splices target straight into the PromQL query and the KEDA threshold, where `150m` means 150 minutes to PromQL and is not a usable KEDA float threshold. Retype it as a dimensionless integer so the schema rejects suffixed input - helm unittest's schema validation now enforces it, and the existing autoscaling suite exercises the integer contract. - ConfigSpec carried a root `metric` field (and spec.metric in the postgres-rd schema) that nothing reads - the chart consumes only autoscaling.metric - so setting it silently did nothing. Remove the dead root param, keeping the ReadMetric enum that autoscaling.metric references. Regenerate types.go, zz_generated.deepcopy.go, values.schema.json, README.md and the postgres-rd schema; update the autoscaling tests to the integer target. Addresses CodeRabbit review feedback on #3954. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
ec06dafd25 |
feat(postgres): fold the replication-lag brake into the autoscaling query
Wire autoscaling.maxReplicationLagSeconds into the read-load query: while the max standby lag has exceeded the threshold within the cooldown window and the primary is actively writing WAL, the query returns currentInstances*target, pinning desired to the current count and freezing scaling in both directions (safer than pinning maxReplicas, which would still allow scale-down under lag). The lag term uses max_over_time as a hysteresis band; the write gate (WAL-record rate > 0) keeps an idle primary from tripping it. Metrics verified live on CNPG 1.30.0: cnpg_pg_replication_lag and cnpg_collector_wal_records exist and carry namespace/pod (the proposal's cnpg_pg_stat_replication_sent_diff_bytes does not). Also floor the read load at zero via `or vector(0)` so an idle cluster returns target rather than an empty result. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |