mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 20:23:39 +09:00
f83db514b0eb88cefc2923d17aad59daccb4e358
7777
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
61238500dd |
fix(e2e): keep the Proxmox tenant kubeconfig per run and remove it
run-proxmox.sh wrote the tenant's super-admin kubeconfig to $TMPDIR with the process umask and never removed it, so it outlived the run on the self-hosted runner, world-readable under the usual umask. dump_diagnostics then read whatever sat at that path, so a run that failed before the fetch reported the previous run's tenant as its own. The file now lives under COZY_PROXMOX_WORKDIR, which the workflow points at runner.temp. It is written under umask 077, removed at the start of apply-tenant so this run never sees a leftover, removed again in the suite's finally, and once more in the workflow's credential cleanup step for a run killed before finally runs. Closes #4806 Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
|
cac152048e |
fix(e2e): read the KubernetesNodes pool release in the HelmRelease guard
cozy_guard_all_helmreleases reads the parent and the kubernetes-<test>- addons. The worker pool is kubernetes-nodes-<test>-md0, an Application HelmRelease the apiserver stamps with RetryOnFailure like the parent, on the node-join path through the talos-reconcile Job it renders. Its name does not start with the addon prefix, so a failed action on it was never read, on the passing path or in the failure-path report. Read it on both paths. The pool name is derived from the parent, the way the addon prefix is, so the two cannot drift. The failure-path report checks that the release exists first: a run that failed between the parent and the pool apply has none, and that is a fact about where the run stopped rather than a failed read. Closes #4737 Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
|
98d0402096 |
test(kubernetes): pin RetryOnFailure on the Proxmox csi release
The KubeVirt csi and cilium releases carry a helm-unittest pin on strategy.name: RetryOnFailure for both actions, and the e2e HelmRelease guard names those two tests as load-bearing. helmreleases/csi-proxmox.yaml sets the same strategy for the same reason and had no pin. Add the mirror test and update the guard comment that pointed at the gap. No template change. Closes #4731 Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
|
bc24b588b2 |
feat(release): sign the release assets and attach a source SBOM
No release carried a signature or an SBOM, so users could verify none of the ISO, disk images, kernel, initramfs, manifests and cozypkg binaries they download, and the OpenSSF Scorecard Signed-Releases check and CLOMonitor's signed_releases and sbom checks failed. hack/upload-assets.sh now hands its asset list to the new hack/sign-release-assets.sh before the first upload. That script writes cozystack-source-sbom.spdx.json with syft over the committed tree and the cozypkg binaries, cozystack-checksums.txt with the SHA-256 of every asset and the SBOM, and cozystack-checksums.txt.sigstore.json with a keyless cosign signature over the checksums. The SBOM does not describe the container images or the Talos boot images, which is what its name says; image SBOMs are #2937. The script verifies the bundle against the identities of tags.yaml on a v* tag and promote-rc.yaml on main before returning, so a bad signature fails the job with nothing uploaded, and hack/upload-assets.bats pins the upload list and that order. The two jobs that call make upload_assets get id-token: write, cosign v3.1.3 from the SHA-pinned cosign-installer, and syft from its release tarball checked against a committed SHA-256, because anchore/sbom-action/download-syft runs an install script fetched at run time; hack/ci-tool-installs-contract.bats now covers both. docs/release.md describes the files, what the SBOM covers, the minimum cosign version to verify a download with, what the certificate identity proves, and that promote-rc.yaml is dispatched from main only. Refs #4759 Assisted-by: LLM Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> |
||
|
|
b62f1caf07 |
fix(keycloak): replace only the old static KEK form, refuse the rest
Any stored KEK that did not decode to exactly 44 characters was overwritten. The proxy skips CR and LF when it decodes KKP_KEK, so a key restored from a file ending in a newline works, and the chart replaced it silently. A 44-character value that is not base64 of 32 bytes was kept, and the proxy could not start with it. Only the 32-character alphanumeric form earlier chart versions wrote is replaced now. A key the proxy accepts is kept, and anything else fails the render, so key material is never overwritten. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
|
9754a05315 |
fix(keycloak): keep the static KEK Secret when the release drops it
The chart now requires a persistent DEK set wrapped with the static KEK, so the KEK Secret has to outlive the release. Disabling encryption or reinstalling the package deleted it, and the next enable minted a fresh KEK that cannot unwrap the DEK set. The lookup already reuses a kept Secret on the next install. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
|
078e9f0daa |
fix(keycloak): require a persistent DEK set for the KMS proxy
Without encryption.deksetSecretName the proxy mints a new DEK set on every start and keeps it only in memory. After any restart of the proxy pod it can no longer decrypt the rows it wrote before, so the default configuration loses access to all encrypted user data on the first rollout, drain or OOM. Refuse to render the proxy without a DEK set unless encryption.fields is "disabled". Passthrough keeps a way for the static backend to get its KEK Secret created before the DEK set is minted with it. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
|
a9ed6f238f |
fix(keycloak): generate a static KEK the KMS proxy can decode
The proxy reads KKP_KEK as base64 of 32 random bytes and refuses to start otherwise. The chart stored 32 alphanumeric characters, which decode to 24 bytes, so the default static backend never started: config: KKP_KEK must decode to 32 bytes (got 24) Generate the key with randBytes and replace a stored key of the old form. The proxy never ran with such a key, so nothing was encrypted under it. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
|
cc7f534f5e |
docs(release): point changelog check B at the Talos installer profile
packages/core/talos/values.yaml is an empty file, so the first path of the check B loop never produces a diff and a Talos bump goes unnoticed by the release verification. The Talos version is the version field of packages/core/talos/images/talos/profiles/installer.yaml, which the package Makefile reads as TALOS_VERSION. Between v1.5.4 and v1.6.0 it moved from v1.13.0 to v1.13.6 and check B printed nothing for it. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
b4f6fb67db |
fix(vm-disk): let CDI adopt the PVC a Velero restore recreates
A Velero restore of a VMDisk or a VMInstance deletes the disk's
DataVolume and restores the PVC under its name, with no owner. The
restored release is in sync, so nothing recreates the DataVolume until
the next Helm upgrade, typically a platform update. That upgrade is
refused by CDI's admission webhook ("Destination PVC ... already
exists") and the release stays failed, followed by the VMInstance
release that looks the disk up.
CDI admits a DataVolume over an existing claim only when the
DataVolume carries cdi.kubevirt.io/allowClaimAdoption or the
DataVolumeClaimAdoption feature gate is on. The restore controller
puts the annotation on the restored PVC, where CDI never reads it, and
the CDI CR does not enable the gate. Setting it on the DataVolume
keeps the scope to vm-disk instead of every DataVolume in the cluster,
and does not depend on a gate CDI intends to remove. A claim CDI
created for this DataVolume carries its uid in createdForDataVolume
and is never treated as adoptable, so a fresh import is unaffected.
The VMDisk backup round-trip now forces an upgrade after the restore,
which is what reaches the failure, and checks that the DataVolume
adopted the restored PVC instead of replacing it.
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Assisted-by: LLM
|
||
|
|
7ad7d39349 |
test(hack): keep every chart's stored Helm release within budget
A chart can grow past the 1 MiB Helm release Secret without any test noticing: it renders, lints and installs, and only the upgrade whose manifest is large enough fails, on a live cluster, with a release that stays failed. backupstrategy-controller sat under 1 KB from the limit before it crossed it. Estimate the stored size the way Helm encodes it (release JSON with the chart's templates and files, gzip, base64) and fail any chart under packages/ above 90 percent of the limit, plus the largest revision of backupstrategy-controller with every default Strategy rendered. The estimate counts files a .helmignore names, because the production path stores them. A size measured without its manifest is a lower bound that passes for the wrong reason, so every chart must render offline except an explicit list of charts that need cluster state, and the worst-case render must succeed and contain the default strategies before it is measured. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
513b5088a8 |
test(backups): fail on a strategy CRD that inlines a pod schema
The release-size budget only reports that the backupstrategy-controller release grew too large, not which field did it, and three strategies once inlined the upstream pod schema before anyone noticed. Check the generated strategy CRDs for an inlined PodSpec and name the property path, with the markers that fix it. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
2f6c6a27b8 |
fix(backups): store the Altinity, Job and FoundationDB pod templates schemaless
backupstrategy-controller renders its strategy CRDs from templates, so Helm stores each of them twice in every release revision, and a revision is one Secret capped at 1 MiB. The Altinity, Job and FoundationDB CRDs each inline the full upstream PodTemplateSpec schema, about 600 KB apiece; once the default Strategy CRs render as well, the stored release no longer fits and the forced upgrade that creates them fails with "Secret ... is invalid: data: Too long". The Kafka, RabbitMQ and Redis strategies already carry their pod template schemaless for this reason. Give the three older fields the same markers. The Go types stay typed, the drivers decode the template as before, and the apiserver still validates the pod when the driver creates the workload; every description is kept, and the largest revision drops to about a quarter of the limit. The rationale moves to the package doc, so the six field docs only tell the reader what schemaless means for them. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
aed7fa896e |
docs(release): add changelog for v1.6.5 (#4940)
This PR adds the changelog for release `v1.6.5`.
✅ Changelog has been automatically generated in
`docs/changelogs/v1.6.5.md`.
|
||
|
|
4814ccfed9 |
docs(release): note the PostgreSQL restart on upgrade in the v1.6.5 changelog
Clusters with a memory limit below 512Mi and no explicit shared_buffers get the new default on upgrade, which needs a PostgreSQL restart. The changelog described the fix but not the restart. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Assisted-by: LLM |
||
|
|
13f612d55e |
docs: add changelog for v1.6.5
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> |
||
|
|
c2dab12a2e |
fix(cluster-api): resync capi-operator every minute
A restarted capi-operator v0.19.0 drops the create events for the provider Deployments it finds already running (kubernetes-sigs/cluster-api-operator#1031). A provider whose Ready condition went False while the operator was down therefore keeps it until that Deployment changes or the informer resyncs, and the first resync after a restart lands two periods later because the handler registers after the informer has listed. With the 10m default that is about 20 minutes, during which capi-providers-core times out on the CoreProvider and everything depending on it waits; every further operator restart starts the clock again. With a one-minute period the same repro recovered 2m02s after the operator took its lease, against no recovery within 4m26s with the default on the same cluster. A resync replays the cache and, with providers in sync, ends in "No changes detected" and an unchanged condition, so it writes nothing to the API server; the operator logged about 15 lines a minute and used 2m CPU and 26Mi of memory. The upstream chart has no value for the flag, so a package patch that make update reapplies adds one, and a bats test holds the rendered argument. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
512ffa7ccd |
ci(release): reject truncated entries and docs/next links
Two defects in the v1.7.0 draft were mechanical and passed the validator anyway. Three entries end in an ellipsis right before their attribution: the PR text was pasted up to a length limit, and the tail it lost is where PR authors put what the operator has to do, so the entry is wrong rather than short. And Feature Highlights linked cozystack.io/docs/next/, which the live site answers with 404. Neither appears in any shipped changelog, so rejecting them costs no good output. The link check is anchored on the host because v1.3.0 names the website's content/en/docs/next/ tree in prose, and the ellipsis check only fires before an attribution because v1.4.0 has one inside a sentence and every compare link has three dots. The offending line numbers go into the one-line reason, since the workflows quote the reason into an annotation that ends at the first newline. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
e2ca1b7bc5 |
ci(release): overlay the release checks the changelog agent runs
The changelog job checks out the rc tag detached, so every file the agent reads by its working-tree path is the rc tag's copy. Only the agent instructions were refreshed from the dispatch ref. The instructions now tell the agent to run the checks in docs/release.md and hack/validate-changelog.sh on its own output before it exits, and the rc tag's copies of those predate any check added since, or for an old enough tag do not exist at all. Refresh both alongside the instructions. docs/release.md is under neither sparse-checkout path and reaches the tooling checkout only because cone mode also carries the files of every parent directory of docs/agents. The contract test pins both copies and that dependency, so a switch to non-cone patterns fails here rather than as a hard failure of the refresh step at promote time. The AI step keeps its read-only token and its tool flags. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
6395ca1759 |
docs(release): check changelog claims against the code
The generator writes each entry from the opening of the PR text, one PR at a time, and nothing in its instructions asks it to check a statement against the tree it ships. The v1.7.0 draft had every author and PR number right and still carried 22 wrong or missing statements, several in the breaking-changes section, where a wrong instruction breaks an upgrade: a companion-repository fix credited to a tag that does not contain it, worker pools said to survive a migration that never pins their template, migrations said to run on a variant that disables them. The checks that would have caught them live in docs/release.md, which the model is never pointed at, and the detailed rules it does read are about bookkeeping. Add the semantic half: read the whole release note, describe the final state at the upper-bound tag, compare with the previous release, take identifiers, migration behaviour and variant scope from the code, collect required actions beyond `!` commits, pin companion changes to a tag, and run the pre-publish checks and the validator on the written file before exiting. The pre-publish checks name the file after $NEW, which in the promotion configuration is the rc tag, so they are told to read the stable file name; otherwise the reference and link checks read a file that does not exist and report nothing. A minor release's versioned docs are not live until it publishes, so those links are checked against the website tree rather than expected to answer 200. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
8bd6d91dbb |
fix(backups): gate every default Strategy CR on its CRD (#4929)
## What this PR does Upgrade from v1.6.x to v1.7 leaves `backupstrategy-controller` HelmRelease in `UpgradeFailed`: `no matches for kind "Kafka" in version "strategy.backups.cozystack.io/v1alpha1"`, same for Rabbitmq and Redis, and none of the new strategy CRDs get created. CRDs of this chart are ordinary templates, and Helm maps every object of a release through the RESTMapper before applying any of them. Default Strategy CRs for kafka, rabbitmq and redis were gated only on the bucket name, so they render next to their not yet served CRDs, Build() fails, nothing is applied and every retry renders the same manifest. Same happens on a first install with external S3 (`provisionBucket: false`) for all older default strategies, bucket name comes from values there. MongoDB default strategy already has the `crdEstablished` gate, this PR adds it to all other default strategies. On an installed cluster nothing changes. Otherwise first revision creates the CRDs and skips the CRs, then DefaultObjectsGate forces the follow-up upgrade that renders them. CRDs stay templates so later CRD changes still reach the cluster on upgrade. Short explanation is added to `docs/operations/backup-classes.md`. To keep this from coming back there is `hack/backupstrategy-crd-gate.bats`: it renders the chart with the bucket gate open and no CRD served and fails on any Strategy CR from any template, so a new `strategy-*-default.yaml` without the gate goes red without anyone updating a list. It also checks that every CR gates on the CRD this chart ships for its kind. helm-unittest covers both halves of the gate for all 11 templates. Checked on a throwaway kind cluster: - v1.6.4 installed, then upgraded: current main fails with the error above, this branch passes with all 11 CRDs Established and the follow-up upgrade creates the rest - modified CRD is applied on a later upgrade - fresh install converges, and fresh install with external S3 converges here while main fails with "no matches for kind" for eight kinds Fixes #4925. Needs backport to release-1.7, both commits apply there cleanly. ### Downstream repositories - [x] No downstream repository is affected by this change ### Release note ```release-note fix(backups): backupstrategy-controller release no longer gets stuck in UpgradeFailed on upgrade from v1.6 and no longer fails a fresh install with external S3, strategy CRDs are created first and default strategies follow on the next reconcile ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Default backup strategies are now created only after their required resource types are available and the backup bucket is resolved. This helps prevent installation and upgrade issues, including when using externally managed S3 storage. * **Documentation** * Clarified when default backup strategies are created during installation and upgrades. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
fe053ce93a |
fix(kubernetes-nodes): hold factory-CA downloads to https redirects
With COZY_TALOS_FACTORY_CA set, curl -L still followed a redirect to plain http, so the image could arrive without the TLS verification the CA was given for. The CA path now passes --proto-redir =https, in line with the refusal of a CA for an http factory. The no-CA program is unchanged. Assisted-by: LLM Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> |
||
|
|
000479a0cc |
feat(kubernetes-nodes): trust a factory CA in the Proxmox template script
A pool can now point CDI at a self-hosted Image Factory served by an internal CA through talos.imageFactoryCA. hack/proxmox-talos-template.sh downloads the same image with curl on the hypervisor, which trusts only the system store and fails there with the same unknown-authority error. COZY_TALOS_FACTORY_CA names a PEM file with those certificates. The program writes them after the hypervisor's system bundle and hands the result to curl --cacert, so they are trusted on top of the system store, as CDI does, and the hypervisor's own trust store is not changed. The file is pasted into the root program, so it must be complete certificate blocks and nothing else; a private key sharing the file, trailing text or an unclosed block is refused, as is a CA given for a plain-http factory. Without the variable the program is byte-for-byte what it was. Assisted-by: LLM Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> |
||
|
|
f47bab7280 |
test(backups): fail CI on any default Strategy CR rendered without its CRD gate
The CRD gate has now been missed twice: MongoDB shipped without it, and Kafka, RabbitMQ and Redis followed after it existed. helm-unittest only checks the templates a case lists, so a new strategy-*-default.yaml that forgets the gate stays green until an upgrade wedges. Render the chart with the bucket gate open and no CRD served, and fail on any strategy.backups.cozystack.io object from any template. Also check that every rendered Strategy CR gates on the CRD this chart ships for its kind, because a gate naming the wrong CRD passes the first check and keeps the CR out of every live render. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
7204992452 |
fix(backups): gate every default Strategy CR on its CRD
The chart ships its strategy CRDs as ordinary templates, and Helm maps every document of a release through the RESTMapper before it applies any. A default Strategy CR gated only on the bucket name therefore fails Build() with "no matches for kind" whenever the bucket name resolves before the CRD is served, the release applies nothing, the CRDs are never created, and every retry renders the same manifest. That happens on the upgrade that first ships a kind: the Kafka, Rabbitmq and Redis CRs leave a v1.6 cluster's HelmRelease stuck in UpgradeFailed on the way to v1.7. It also happens on the first render of an external-S3 install (provisionBucket=false), where the bucket name comes from values and every older default CR renders next to its absent CRD. MongoDB already gates on crdEstablished for this reason. Give every default Strategy the same gate. On an installed cluster the CRDs are established and the render is unchanged; otherwise the first revision creates the CRDs and skips the CRs, and DefaultObjectsGate, which reads the cozy-default BackupClass routes, forces the follow-up upgrade that renders them once the kinds are served. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
11b19635e0 |
fix(monitoring): send alerts to every Alertmanager replica (#4889)
## What this PR does vmalert sends every alert to every Alertmanager replica instead of to the shared Service. The chart gave vmalert a single notifier, `http://vmalertmanager-alertmanager.<namespace>.svc:9093`. That Service is headless and resolves to the three Alertmanager pods, but each alert reached one replica at a time and moved to another one whenever vmalert reconnected. The Alertmanager README asks for the opposite. It says "Do not load balance traffic between Prometheus and its Alertmanagers" and to point the sender at every instance, because the cluster deduplicates notifications only when each replica receives each alert. What this looked like on three clusters running 1.6.4, read through the Alertmanager API of each pod. - On all three, every active alert sat on one replica and the other two had none. The counts were 0, 0 and 10 on the first, 0, 0 and 23 on the second, and 1, 0 and 9 on the third. - On the third one, which also has an email receiver, the active alert count of each replica changed 163 to 180 times in 24 hours. The three replicas sent 63, 67 and 62 emails in that window. Over the same week only three alert names with a severity routed to email fired there. An alert that leaves a replica expires there and goes out as resolved, and the replica that receives it next sends it as a new firing notification. So a single long-running alert keeps producing firing and resolved notifications, and a receiver that rejects concurrent duplicates sees them as conflicts. The change. - `vm/vmalert.yaml` renders one notifier per Alertmanager pod, `http://vmalertmanager-alertmanager-<i>.vmalertmanager-alertmanager.<namespace>.svc:9093`. That is the address the operator builds for StatefulSet pods (`VMAlertmanager.asPodFQDN`), and the replicas already reach each other through the same per-pod names in their `--cluster.peer` arguments. - The replica count comes from a new `monitoring.alertmanager.replicas` helper, which now also sets `spec.replicaCount` of the VMAlertmanager. The notifier list cannot drift from the number of pods. Why not the operator's notifier selector. `VMAlertNotifierSpec.selector` expands matched VMAlertmanagers into the same per-pod addresses, but only when the VMAlert is reconciled. The VMAlert controller does not watch VMAlertmanager objects, so a VMAlert reconciled before its VMAlertmanager exists, which can happen on a fresh install, would get no notifier until something else triggers a reconcile. A static list renders the same addresses without depending on that order. Upgrade impact. The VMAlertmanager is unchanged, it renders `replicaCount: 3` as before. The VMAlert gets new notifier arguments, so the operator restarts the vmalert pod once. Testing. - `helm unittest` passes for the monitoring chart, 24 suites and 304 tests. The new suite `tests/vmalert_notifiers_test.yaml` checks the three notifier addresses, the replica count of the VMAlertmanager and the release namespace in the addresses. - Each of these breakages turns at least one case of the new suite red. The notifier back on the Service address, the helper set to 2, a hardcoded namespace in the address, and a hardcoded `replicaCount` that no longer follows the helper. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: Walked the trigger map in `docs/agents/contributing.md` against the diff. It changes two templates and the helpers of `packages/system/monitoring` and adds a test file. No values, schema, README, package name, make target or `ApplicationDefinition` changes, so no entry applies. ### Release note ```release-note fix(monitoring): vmalert now sends alerts to every Alertmanager replica instead of the shared Service, so the Alertmanager cluster deduplicates them and a long-running alert no longer produces repeated firing and resolved notifications. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Improvements** * vmalert now configures a notifier URL for each of the three Alertmanager replicas, rather than using a single shared URL. * Alertmanager’s replica count is managed consistently with the notifier configuration. * **Tests** * Added checks for the replica count, all three notifier URLs, and the configured namespace. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
f3486f4e3a |
docs: move the operator guides out of docs/ to where readers find them
Five guides in docs/ were written next to the feature code that needed them and left in the source tree, where no operator looks. They are on the website, so a second copy here only drifts: edits kept landing in docs/ after the website copy was made. docs/vm-external-vlan.md is split rather than moved. Its section on staging the reference CNI plugins is a contract that hack/cni-plugins-staging-contract.bats asserts against the multus Dockerfile, the daemonset manifest and the platform values, so that half is package documentation and goes to packages/system/multus/README.md, which the suite now reads. The operator half goes to the website. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
acf02ef5cf |
docs(agents): define where a new document belongs
The `docs/` tree had accumulated component guides that belong on the website: standalone OIDC, GPU/vGPU and VM-networking documents written as part of a feature PR and never routed anywhere. Nothing said where documentation should go, so `docs/` was the default, and nothing said that the website imports a package README only for the packages on its Makefile lists, so prose written for any other package reached no reader. The destination table turns on one test: whether a non-documentation file has to link to the document by path. A merged document records current behaviour and its limitations rather than the development process around it, because a claim about a phase or an unmerged change turns false once the work ships and nothing re-checks it. backup-classes.md still described the `image.tag` pin as one that an in-flight change removes. The pin has since been removed, so the note now says that. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
e05c56524e |
fix(backups): prepare standalone VMDisk restore targets and add VM backup/restore e2e round-trips (#3427)
## What this PR does Closes the uncovered VM backup path from #3236. `hack/e2e-chainsaw/backup/` held only a disabled Chainsaw port of the old `backup.bats`, written against the retired API (`kind: VirtualMachine`, `BackupJob.storageRef`/`strategyRef`, `velero-strategy-default`), so it guarded nothing. This removes it, fixes the in-place restore of a standalone VMDisk in the backup controller, and adds two real round-trips in the `vminstance` suite that drive the `examples/backups/{vmi,vmdisk}` scripts, so the documented demo and the automated test stay in step. ### Controller: standalone VMDisk in-place restore `collectUnderlyingResources` in `internal/backupcontroller/velerostrategy_controller.go` returned nothing for any kind but VMInstance, so `prepareForRestore` did nothing for a standalone VMDisk: no HelmRelease suspend, no DataVolume delete, no `keepOriginalPVC` rename. An in-place VMDisk restore then raced Flux and CDI over the PVC name the data mover was repopulating. A standalone VMDisk now reports its own disk as its underlying resource, so the restore preparation already used for VMInstance disks suspends the `vm-disk-<name>` HelmRelease, deletes its DataVolume and renames the live PVC to `<name>-orig-<hash>` before Velero recreates it. Unit tests cover both the VMDisk and the VMInstance branch. A VMInstance is halted before its disks are touched, but a VMDisk target is not, and every VMInstance disk is a VMDisk. Renaming the PVC of a disk that a running VM mounts would leave the original `Terminating` behind `pvc-protection`; Velero's PVC restore action skips a PVC that still exists, so the RestoreJob would report `Succeeded` having moved nothing, and the VM would lose its disk on the next restart. An in-place VMDisk restore therefore fails before changing anything when a Pod that has not terminated mounts the disk's PVC. The message names the Pod and tells the user to stop the VM or restore the VMInstance instead. `TestPrepareForRestore_VMDiskInUse` covers a running mounting Pod (refused, PVC not renamed, HelmRelease not suspended), a finished Pod, a Pod on another disk, and a VMInstance target whose own launcher Pod must not refuse it. A VMDisk Backup taken before the disk was recorded as its underlying resource has none, so the disk is derived from the application name (`vm-disk-<name>`): such a restore is prepared and checked for a mounting Pod too. The per-disk `orLabelSelectors` stay a VMInstance concern: a VMDisk strategy already selects its disk, and Velero rejects a Backup or Restore carrying both `labelSelector` and `orLabelSelectors`, so appending them for a VMDisk would break a strategy written with `labelSelector`. When an in-place restore fails after preparation, nothing is rolled back. Resuming the disk's HelmRelease while its PVC is renamed and its DataVolume deleted would have CDI import a fresh disk under the original name, which looks healthy and hides that the data sits in the `-orig-` PVC. The failure message and a Warning event instead name the HelmReleases preparation suspended and the `-orig-` PVCs holding the original data, read back from the cluster so that a failure before a step ran does not name what it would have left, and say that a HelmRelease Velero already restored is resumed, so the disk has to be checked; `examples/backups/vmdisk/README.md` describes the recovery. Tests cover both selector paths, the pre-upgrade Backup, a failed Pod list (the restore requeues and suspends nothing), and all three failures after preparation: a `PartiallyFailed` Velero Restore, several Restores owning the job, and a Restore that cannot be created. Because preparation takes the target apart before Velero validates the Restore, the RestoreJob is refused before preparation, with nothing changed, when the Velero Backup is gone (expired or deleted) or not `Completed`. This is the same single check #4672 added, now run on every pass behind the preparation failure path, so a Backup that expires after preparation started names the leftovers too. A Velero Restore that ends in `FailedValidation` fails the RestoreJob like `Failed`, with Velero's validation errors and the leftovers in the message, instead of polling forever. A HelmRelease suspend or a VM halt that does not happen now stops preparation (requeue) rather than letting the rename and DataVolume delete run under a live release or VM, and a VMInstance's disks are renamed only once no Pod mounts them. A VMDisk held only by a Pod already being deleted is waited for rather than refused, for up to 30 minutes after the RestoreJob started, so a Pod stuck terminating on a lost node fails the job instead of keeping it `Running`. Each of these is pinned by a test that turns red when the guard is removed. Between renaming a kept disk's PVC and deleting its DataVolume, CDI can recreate an empty PVC under the original name for the still-present DataVolume. Velero then skips that PVC as already present ("PVC already exists. Skip restore this PVC."), the RestoreJob reports `Succeeded`, and the VM boots a blank disk. The round-trip's new DataDownload check caught this on a VMInstance restore, where the uid, label and `-orig-` checks had all passed. Once the DataVolume is gone (the wait #4672 added), preparation now deletes a PVC found under the original name (read uncached) before the Velero Restore is created; `TestPrepareForRestore_ClearsPVCRecreatedAfterRename` reproduces the recreation inside the DataVolume delete. Preparation spans several reconciles, so the RestoreJob records a `PreparationStarted` condition before it first changes the target; if recording it fails, preparation retries without changing anything, so a later failure cannot report leftovers that were never made. A refusal on a later pass (the Velero Backup expired, a VM started on the disk) then names what preparation left behind instead of saying nothing was changed, and a preparation still waiting 30 minutes after it started fails the same way instead of polling forever. The vmdisk script starts its consumer Pod only on a PVC that carries the restore label, so it cannot pin a PVC CDI recreated in the rename window. Now that a VMDisk target is prepared, `keepOriginalPVC: false` reaches it as well. Its DataVolume is deleted with orphan propagation, and the bound PVC is deleted and waited for before the Restore (the path #4672 added for VMInstance disks). Left in place, Velero would skip the PVC and the CDI webhook would refuse the DataVolume the resumed HelmRelease creates, leaving the disk's HelmRelease stuck. `TestPrepareForRestore_VMDiskWithoutKeepDeletesOriginalPVC` pins it. ### `vminstance-2-backup-roundtrip` (un-gated in CI) A tiny cirros guest (`u1.nano`, 2Gi, in-place restore only) runs the current API end to end: Velero strategy, BackupClass, a booted VMInstance, an ad-hoc `BackupJob` and an in-place `RestoreJob`, each waited to `Succeeded` and failing fast on `Failed`. The VM must come back `Running`, and its disk PVC must have been created by the restore: a new `metadata.uid`, Velero's `velero.io/restore-name` label, and the `vm-disk-<name>-orig-<hash>` PVC that the rename leaves behind. The label alone does not prove the data moved, because with `existingResourcePolicy: update` Velero also stamps it on a PVC it only patched, and the patch path never reaches the data mover. An in-guest signal is not available on this platform: the KubeVirt CR disables serial-console logging cluster-wide and cirros ships no qemu-guest-agent. ### `vmdisk-2-backup-roundtrip` (un-gated in CI) The disk-only companion: import a 5Gi Ubuntu disk, back it up, restore it in place relying on the controller fix above, and require the same three signals (new uid, restore label, `-orig-` PVC), a `Completed` DataDownload for the restore that wrote bytes, and a Ready HelmRelease. On a `WaitForFirstConsumer` StorageClass, such as the `local` class the CI lane uses, the Velero data mover waits until a scheduled Pod has selected a node for the restored PVC (`WaitPVCConsumed` in Velero's generic restore exposer). A VMInstance gets that from its virt-launcher Pod; a detached disk has none, so its `RestoreJob` stays `Running` until the 30m data-mover prepare timeout fails it. The script therefore starts a restricted-profile `pause` Pod on the new PVC once Velero has created it, standing in for the VM that would attach the disk, and removes it after the restore. `examples/backups/vmdisk/README.md` documents this for operators restoring a detached disk. ### Harness details - The disk `storageClass` is an override (default `replicated`). The Chainsaw steps pass the lane's `COZY_E2E_STORAGE_CLASS` (`local` on the DRBD-off PR gate). - Each harness stands up a throwaway in-cluster BSL (`cozy-e2e-vmi`/`cozy-e2e-vmdisk`) at `seaweedfs-s3.tenant-root:8333` with `insecureSkipTLSVerify` and `checksumAlgorithm: ""` (the #3236 fixes), reusing the platform bucket and `cozy-backups-creds` under a distinct prefix, because the platform `cozy-default` BSL derives the external S3 ingress endpoint, unroutable in the CI cluster. - The script waits are bounded so that their worst case stays under the 40m op timeout (38m for the VMDisk step), and a hung wait trips the script's own named timeout rather than a bare `signal: killed`. - On failure, `catch` also describes the Velero `Restore`, `DataDownload` and `DataUpload` objects in `cozy-velero`. - `skipDelete` plus a `finally` that resumes the suspended HelmReleases, reclaims the `-orig-` PVC and its LINSTOR volume (the PV is flipped back to `Delete` first), and drains the Velero backups on the throwaway BSL, branching on kubectl's exit status, before deleting it. When the drain is not confirmed, the BSL is kept and the step fails, so the Velero Backups are not orphaned behind a green step. - HelmRelease waits use the shared `wait_hr_ready`, which fails fast on `Stalled`. After the VMDisk restore the script first waits for `observedGeneration` to reach the restored spec's `generation`, because the `Ready=True` from before the suspend would otherwise satisfy the wait at once. The `vmi` example is CI-drivable: `run-all.sh`/`cleanup.sh` are non-interactive, the VM is waited to boot before the backup, `volumeSnapshotLocations` is dropped from the demo strategies (`snapshotMoveData` routes data through the BSL), and `BACKUP_STORAGE_LOCATION` defaults to `cozy-default`. `select-e2e.sh` maps `examples/backups/{vmi,vmdisk}` edits to the `vminstance` suite, except `07-restore-to-copy.sh`, which the suite skips (`SKIP_RESTORE_TO_COPY=1`) and so stays docs-only. A `packages/system/velero` change selects the `vminstance` suite as well, since these are the only Tests that run a Velero backup and restore. ### Screenshots N/A — no UI changes. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff (backup controller restore preparation for VMDisk, shell scripts under `examples/backups/`, Chainsaw tests, `hack/select-e2e.sh` routing, a deleted stale suite, one doc line): no `packages/apps|extra` add/rename, no `values.schema.json`/enum/default changes, no CRD/ApplicationDefinition/label/annotation/namespace/metric changes. The controller change alters what a VMDisk `RestoreJob` does in the cluster, not any API surface a downstream repository consumes. `cozystack/examples` holds Terraform for bare nodes, unrelated to the in-repo `examples/` tree. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(backups): an in-place RestoreJob for a standalone VMDisk now suspends the disk's HelmRelease, deletes its DataVolume and (with keepOriginalPVC) renames the original PVC before Velero restores it, instead of racing Flux and CDI over the same PVC; without keepOriginalPVC the original PVC is deleted so Velero recreates it from the backup. Such a restore fails before changing anything when a running Pod, such as the launcher of a VM that uses the disk, mounts the disk's PVC: stop the VM or restore the VMInstance instead. If such a restore fails after the disk was prepared, its message and a Warning event name the HelmRelease preparation suspended and the PVC holding the original data. On a WaitForFirstConsumer StorageClass the data mover still starts only once a Pod consumes the restored PVC, for example when the disk is attached to a VMInstance. An in-place restore with keepOriginalPVC no longer reports Succeeded over an empty PVC that CDI recreated under the disk's name while the original was being kept aside. Adds Chainsaw backup/restore round-trips for VMInstance and standalone VMDisk that run a real Velero backup and restore in CI. ``` |
||
|
|
588d8292e7 |
fix(seaweedfs): build SeaweedFS in-tree with a bounded S3 read-ahead (#4819)
## What this PR does Large objects served through a Bucket's S3 endpoint stop tens of megabytes into the download (cozystack/cozystack#3782). The cause is not TLS: the SeaweedFS S3 gateway is OOM-killed mid-response. Its read-ahead for a sequential GET is meant to stay four chunks in front of the reader, but SeaweedFS counts only newly started chunk downloads toward that window and re-runs the prefetch after every 256 KiB read, so the gateway pulls up to about a gigabyte of the object into memory and exceeds the 512Mi limit the S3 deployment gets from its resources preset. The client sees `unexpected eof` over HTTP/1.1 and `INTERNAL_ERROR` over HTTP/2, which is what the CDI importer reported. Upgrading does not help yet: 4.31, 4.45 and the latest release 4.48 all carry the defect. The defect is reported upstream in seaweedfs/seaweedfs#11672, and the fix with its regression test is merged there in seaweedfs/seaweedfs#11674, after 4.48. Until a SeaweedFS release carrying it is vendored, this PR builds SeaweedFS in-tree the way #3343 builds cilium-operator: `images/seaweedfs` downloads the 4.45 source archive and checks it against a pinned checksum, applies `patches/reader-cache-prefetch-window.diff`, runs the tests of the packages the patch touches (including the regression test the patch adds) once, compiles `weed` with the Go toolchain upstream's own builder uses for the tag, and lays it over the upstream `chrislusf/seaweedfs:4.45` image pinned by digest, so everything else in the image stays the upstream build. `weed version` reports `cozystack-patched` in the commit field. All SeaweedFS components run the in-tree image. `make update` re-pins the Dockerfile from the vendored chart: the version, the source checksum, the upstream image digest and the builder toolchain. It fails when the patch no longer applies, which is how the first release carrying the upstream fix will show that the patch is ready to drop. Renovate is disabled for that Dockerfile, since it would move those pins one at a time, and whitespace checks are off for the patch. Tests: - `tests/image_test.yaml` pins every component (master, volume, filer, S3) to the in-tree image with a digest. - `make test-image-version` fails when the Dockerfile's version drifts from the vendored chart's appVersion. - `make update-image`, which `make update` runs, was checked to be idempotent on the current chart, to restore deliberately corrupted pins, and to fail with a clear message when the patch context no longer matches. - `hack/build-matrix_test.bats` now expects seaweedfs to be a build unit next to objectstorage-controller. - The Go regression test inside the patch fails on unpatched 4.45 (read-ahead reaches 60 chunks past the reader) and passes with the patch; the image build runs it. Verified on a test cluster against an 840 MiB object, with every SeaweedFS component running the image this package builds and the S3 gateway at the production limits (512Mi, 1 CPU). Unpatched 4.31, 4.45 and 4.48 gateways were OOM-killed on every full GET. The patched gateway served nine full GETs (six sequential over HTTP/1.1 and HTTP/2, three in parallel) with matching checksums and no restarts; it read from the volume servers about as many bytes as it sent, its live heap after a GET was about 40 MiB, and three parallel GETs peaked at about 140 MiB. The image this PR pins reports `4.45 cozystack-patched` and was rerun through the same checks. Drop the patch, and the in-tree build with it, once a SeaweedFS release carrying the upstream fix is vendored. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(seaweedfs): stop the S3 gateway from being OOM-killed while serving large objects, which cut downloads (including CDI imports from a Bucket URL) off tens of megabytes in. SeaweedFS is now built in-tree with a patch that bounds the gateway's read-ahead. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * SeaweedFS now runs a Cozystack-built image based on version 4.45, pinned to a specific image digest. * **Bug Fixes** * Sequential small reads now keep read-ahead within the configured prefetch window, preventing it from extending unexpectedly. * **Build & Maintenance** * SeaweedFS image updates are aligned with the chart version and verify the source archive, builder image, and patches before building. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
58b4343e5a |
ci(hack): route examples/backups/{vmi,vmdisk} edits to the vminstance suite
The two backup round-trips run the examples/backups/{vmi,vmdisk} scripts as their
harness, but select-e2e.sh mapped an examples/backups/<dir> edit to a suite only
when <dir> was itself a suite name. The dirs are vmi/vmdisk while the suite is
vminstance, so a harness-only edit selected no suite and the round-trip that runs
it never ran. Translate both dirs to vminstance, matching the vm-disk-application
entry in src_to_suites, and pin it with a bats case. The suite runs the vmi demo
with SKIP_RESTORE_TO_COPY=1, so 07-restore-to-copy.sh stays docs-only: selecting
the suite for it would pass without executing it. Both round-trips write to
the in-cluster seaweedfs-s3 endpoint, so a seaweedfs change selects the
vminstance suite as well. Also correct the controller-
owned-teardown note in docs/agents/e2e-testing.md (only Backup/RestoreJob
finalizers reach Velero; a BackupJob has none).
The two round-trips are also the only Tests that run a Velero backup and
restore, so the velero PackageSource maps to the vminstance suite; it
otherwise maps to a velero suite that does not exist, and a Velero change
ran neither round-trip.
Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
|
||
|
|
9cc3a9e459 |
test(vmi): add VMInstance and standalone VMDisk backup/restore e2e round-trips
The disabled hack/e2e-chainsaw/backup/ suite was a Chainsaw port of the retired
backup.bats, written against the old backup API, so the VMInstance and standalone
VMDisk Velero paths had no e2e coverage. Replace it with two round-trips in the
vminstance suite that drive the validated examples/backups/{vmi,vmdisk} scripts,
so the documented demo and the automated test stay in step.
Both run un-gated in CI:
- vminstance-2-backup-roundtrip boots a tiny cirros VM (u1.nano, 2Gi), backs it
up via BackupClass + BackupJob, restores in place, and asserts the VM returns
Running off a disk PVC the restore created: a new uid, Velero's restore-name
label and the -orig- PVC keepOriginalPVC leaves behind. The label alone is not
enough, because existingResourcePolicy: update also stamps it on a PVC Velero
only patched, which moves no data. A guest-side boot proof is unavailable: the
KubeVirt CR disables serial-console logging and cirros ships no guest-agent.
- vmdisk-2-backup-roundtrip imports a 5Gi disk, backs it up, and restores it in
place with the same three assertions. The controller prepares the target
(suspend HelmRelease, delete DataVolume, rename PVC). On a WaitForFirstConsumer
class the Velero data mover waits for a Pod to select a node for the restored
PVC, which a detached disk never gets, so the script starts a pause Pod on the
new PVC, standing in for the VM that would attach it.
The disk storageClass is an override (default replicated; the CI steps pass the
lane's COZY_E2E_STORAGE_CLASS, 'local' on the DRBD-off gate) and the script waits
are bounded so their worst case stays under the 40m op timeout, sized to keep the
whole parallel:1 suite inside the 215m job cap. On failure the catch also dumps
the Velero Restore, DataDownload and DataUpload objects. Each harness stands up a
throwaway in-cluster BSL (the platform cozy-default BSL derives the external S3
endpoint, unroutable in kind); its finally resumes the suspended HelmReleases,
reclaims the keepOriginalPVC orig PVC and its LINSTOR volume, and drains the
Velero backups before deleting the BSL. The example scripts are non-interactive
so the harness can drive them.
Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
|
||
|
|
9af1c0f902 |
fix(backups): prepare standalone VMDisk restore targets before the data mover runs
collectUnderlyingResources returned nothing for a non-VMInstance kind, so prepareForRestore did nothing for a standalone VMDisk: no HelmRelease suspend and no DataVolume delete. An in-place VMDisk restore then raced Flux/CDI — the restored HelmRelease reinstalled and CDI recreated the DataVolume over the same PVC name the data mover was repopulating, wedging the RestoreJob. Report a standalone VMDisk's own disk as its underlying resource, so the existing (VMInstance-proven) restore preparation suspends the vm-disk-<name> HelmRelease, deletes its DataVolume and — with keepOriginalPVC — renames the live PVC before the data mover recreates and repopulates it. No VM, so no IP/MAC is collected. A VMInstance is halted before its disks are touched; a VMDisk target is not, and every VMInstance disk is a VMDisk. Renaming the PVC of a disk a running VM mounts leaves the original Terminating behind pvc-protection, Velero skips a PVC that still exists, and the RestoreJob would report Succeeded having moved nothing while the VM loses its disk on the next restart. An in-place VMDisk restore now fails before changing anything when a Pod that has not terminated mounts the disk's PVC, naming the Pod and telling the user to stop the VM or restore the VMInstance instead. A VMDisk Backup taken before the disk was recorded as its underlying resource has none, so its disk is now derived from the application name and such a restore is prepared and checked for a mounting Pod as well. The per-disk orLabelSelectors stay a VMInstance concern. A VMDisk strategy already selects its disk, and Velero rejects a Backup or Restore that carries both labelSelector and orLabelSelectors, so appending them for a VMDisk would break a strategy written with labelSelector. When an in-place restore fails after preparation, nothing is rolled back: resuming the disk's HelmRelease while its PVC is renamed and its DataVolume deleted would have CDI import a fresh disk under the original name, which looks healthy and hides that the data sits in the -orig- PVC. The failure message and a Warning event now name the suspended HelmReleases and the -orig- PVCs so the user can recover them, read back from the cluster, so a failure before a step ran does not name what it would have left. Preparation takes the target apart before Velero validates the Restore, so it is refused up front when the Velero Backup is gone or not Completed. That check now runs on every pass behind the same failure path as the rest of preparation, and a FailedValidation Restore fails the RestoreJob like a Failed one instead of leaving it Running. A HelmRelease suspend or a VM halt that did not happen now stops preparation instead of renaming and deleting under a running release or VM, and a VMInstance's disks are renamed only once its launcher Pod has let go of them. A VMDisk held only by a Pod already being deleted is waited for rather than refused, for up to 30 minutes after the RestoreJob started. Between renaming a kept disk's PVC and deleting its DataVolume, CDI can recreate an empty PVC under the original name. Velero then skips the PVC as already present, the RestoreJob reports Succeeded and the VM boots a blank disk. Once the DataVolume is gone, preparation now deletes a PVC found under the original name, read uncached, before the Velero Restore is created. Preparation spans several reconciles, so the RestoreJob records a PreparationStarted condition before it first changes the target, and retries without changing anything when recording it fails. A refusal on a later pass (the Velero Backup expired, a VM started on the disk) then names what preparation left behind instead of claiming that nothing changed, and a preparation still waiting 30 minutes after it started fails the same way instead of polling forever. Now that a VMDisk target is prepared, keepOriginalPVC=false reaches it too: its DataVolume is deleted with orphan propagation and the bound PVC is deleted before the Restore, as for a VMInstance disk. Left in place, Velero would skip it and CDI would refuse the DataVolume the resumed HelmRelease creates. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
e8635f6e4b |
build(seaweedfs): stamp the re-pinned in-tree image
The builder, the compile flags and the version stamp changed, so the image built from them has a new digest, and values.yaml has to name it for the components to run that build. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
ecdc043e63 |
build(seaweedfs): keep Renovate and git whitespace off the in-tree build
Renovate would bump the builder, the base image or their digests one at a time into a combination upstream never shipped, and the next `make update` would silently put them back, so it leaves that Dockerfile to the update recipe. The patches are unified diffs of Go sources, whose context lines start with a marker before tabs and blank lines, which git's whitespace checks would otherwise flag or rewrite. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
5592a3f65d |
build(seaweedfs): re-pin the in-tree build from make update
The in-tree build pinned its source, base image and toolchain by hand, and nothing moved them when `make update` vendored a new chart, so a bump would build one release's weed into another release's image, or keep patching a release that already carries the fix. Pin the source archive by checksum and the upstream image by digest as build arguments, compile with the toolchain upstream's own builder uses for the tag, and let `make update` re-pin all of them from the vendored chart and fail when the patches no longer apply. The tests of the patched packages now run once rather than once per target platform, and `weed version` names the build. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
d2c6b85cac |
fix(seaweedfs): run the in-tree SeaweedFS image in every component
With the image keys unset the vendored chart renders chrislusf/seaweedfs at its appVersion, so the patched build from images/seaweedfs would be built and never run, and the S3 gateway would keep being OOM-killed on large GETs. Point every component at the in-tree image, pinned by digest like every first-party reference, and hold that in a unit test so a values change cannot quietly fall back to the upstream binary. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
1913104bd3 |
build(seaweedfs): build SeaweedFS in-tree with a bounded read-ahead
The S3 gateway is OOM-killed while serving a large object. SeaweedFS starts read-ahead downloads after every read of a sequential stream and counts only the downloads it newly starts toward its prefetch window, so a GET reading 256 KiB at a time out of 4 MiB chunks pulls up to about a gigabyte of the object into memory, past the 512Mi limit of the S3 deployment. No SeaweedFS release carries a fix yet, so there is no release to move to instead. Build weed from the 4.45 sources with a patch that keeps the window at the next prefetchCount chunks, and lay it over the upstream image of the same tag so the rest of the image stays upstream's own. The image build runs the tests of the packages the patch touches, including the regression test the patch adds, because go build never compiles them. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
c0449eeabb |
feat(backups): back up Valkey with the Redis strategy (#4898)
## What this PR does
Adds backup and restore for the managed Valkey app.
Valkey runs on the same spotahome RedisFailover operator as Redis
(`spec.engine: Valkey`), with the same sentinel topology, replication
protocol and RDB format. So instead of adding a new strategy kind, the
existing Redis strategy now serves both apps:
- **Driver.** `applicationRef.kind` may be `Redis` or `Valkey`. The
template context gets `.Release.Prefix`, which the driver derives from
the app's kind (`redis-` or `valkey-`). The chart uses that prefix for
every name it builds: the sentinel Service `rfs-<prefix><app>`, the auth
Secret `<prefix><app>-auth` and the CA Secret. In cleanup mode the
prefix comes from the Backup's `applicationRef`, so it works after the
app is gone. A kind with no known prefix fails the render instead of
silently producing an empty prefix. Error messages name the real kind.
- **Upgrade window.** The shipped template reads `{{ .Release.Prefix |
default "redis-" }}`. The strategy CR and the controller ship in one
release, and until the new controller takes the lease, the previous one
(which has no `Prefix` and serves only Redis) renders the new CR; a bare
reference would print `<no value>` into the Secret and sentinel names.
- **Same-kind restore only.** A restore whose target kind differs from
the backup's source kind fails before any Job is created. Replaying a
Redis dump into Valkey, or the reverse, is not something the strategy
can promise across engine releases.
- **Chart.** `cozy-default-redis` builds names from `.Release.Prefix`.
`cozy-default` routes `apps.cozystack.io/Valkey` to
`cozy-default-redis`; the DefaultObjectsGate already dedupes repeated
strategy refs. No new image: the `redis:8` base loads the RDB v11 that
both shipped Valkey versions write. The Valkey chart has no TLS option,
so the optional CA mount stays empty and the Job connects in plaintext.
- **CRD.** Only the `.Release.Prefix` description in the Redis strategy
CRD changes (regenerated with controller-gen v0.16.4; every other
strategy CRD regenerates byte-identical).
- **Docs.** `docs/operations/backup-classes.md` has the Valkey route and
a paragraph on how the shared strategy behaves.
- **E2E.** The Redis backup demo takes `APP_KIND=Valkey`, which switches
the app kind and the name prefix. Its BackupClass routes Valkey too. The
valkey suite gains `valkey-backup-roundtrip`, which seeds a marker,
backs it up, restores it in place and into a copy. In
`hack/select-e2e.sh`, a seaweedfs change now selects the valkey suite
(enforced by `select-e2e_test.bats`), an edit to the shared
`examples/backups/redis/` harness selects both `redis` and `valkey`
(pinned by a new test), and `hack/select-install.sh` keeps the backup
stack in the reduced install for the valkey suite (pinned in
`select-install_test.bats`).
### Validation
- `go test ./internal/backupcontroller/...` passes. New tests: a Valkey
BackupJob renders `--prefix=valkey-`; a Redis backup restored into a
Valkey target fails without creating a Job; the ref validator accepts
Valkey; a cleanup for a kind with no prefix is released without a Job.
The existing Redis render test now also asserts `redis-`.
- The `default "redis-"` fallback, run through the controller's own
template engine with no `Prefix` in the context, renders
`rfs-redis-cache`; the bare reference renders `rfs-<no value>cache`.
- `helm unittest` on backupstrategy-controller: 26/26, with
`cozy-default` at eleven routes. `helm lint` is clean.
- `hack/redis-backup-strategy.bats` executes the shipped script and adds
a Valkey case: it must query `rfs-valkey-<app>` and never `rfs-redis-`.
The fixture now also fails if any `{{ }}` placeholder survives
substitution. All 62 tests in `hack/select-e2e_test.bats` pass.
- Engine compatibility, checked locally with the strategy image
`redis:8-alpine@sha256:ba6e…` against `valkey/valkey` 8.1.8 and 7.2.13
using the shipped script's exact steps: `redis-cli --rdb` from a
password-protected Valkey master, load into the throwaway
`redis-server`, emit the RESP base AOF, `FLUSHALL`, then `--pipe` into a
fresh Valkey. On both versions every check held: the db0/db1/db3
boundaries, a stream with its consumer group, a key's TTL and the marker
all survived, and FLUSHALL wiped the stale target key.
- I have not run this on a live cluster. The new chainsaw round-trip is
what exercises it end to end.
Out of scope, and already true before this PR: the website's backup
pages in `docs/next` don't list Redis, RabbitMQ or Kafka yet, so none of
the triggers in the downstream map fire here.
### Screenshots
N/A — no UI changes.
### Downstream repositories
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
feat(backups): Valkey apps can now be backed up and restored. The platform `cozy-default` BackupClass routes `apps.cozystack.io/Valkey` to the existing `cozy-default-redis` strategy, which dumps an RDB snapshot to the system bucket and restores it in place or into another Valkey app. A Valkey backup restores only into a Valkey app.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added support for backing up and restoring Valkey applications using
the existing Redis backup strategy.
* Updated the backup walkthrough to support both Redis and Valkey, with
resource names selected for the chosen application type.
* **Bug Fixes**
* Prevented restores when the backup and target application types
differ.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
7011510456 |
fix(fluxcd): keep the hook waiting on a release it cannot read
The pre-delete hook read a release's state once, so a single failed read was taken as the release being handled: its fence came off before helm-controller's uninstall, and the report counted it as released cleanly. A state read is now retried within the release's share of the window, a release whose state still cannot be read keeps its fence for the wait like one helm-controller has not handled, and the end of the wait reads each again before naming what the release was left to. A release still suspended after the sweep is named as suspended but not by flux-plunger, which also holds when no manager is recorded for the field at all. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
568e6f5e18 |
docs(fluxcd): correct flux-plunger's recovery order and name its limits
The README gave the recovery steps in the order flux-plunger used before the processed-version annotation moved ahead of the Secret delete. An operator who finds the annotation on a release whose revision Secret is still there would read that state as impossible, when it is what a failed delete leaves and the next pass retries. Two boundaries were left unsaid. The pre-delete hook stops, waits for and lists again only for the flux-plunger of its own release; another one, under a second release name or with changed labels or out of cluster, keeps the fences it sets and gives them up itself. And a backup restore that starts while flux-plunger holds its own suspension does not take the field over, since the restore steps write it only when it differs, so the end of the recovery unsuspends the release under the restore. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
699555c7cb |
fix(fluxcd): give each release its own share of the hook's retries
The pre-delete hook's retry budget was one window spent in listing order, so a release that could never be read or unfenced, listed first, took all of it and every release after it got a single attempt. The same transient failure then released a fence or not depending on where its release sorted. Each release now gets its own share of the window, capped at the window's end, and the steps that cover every release keep the whole of it. Two comments said more than holds. The retry window is measured from the script's start, not the Job's, so scheduling and the image pull come out of its one-minute reserve rather than being covered by it. And the five-minute conflict retry inside kubectl scale never runs here, since without a precondition it sends a merge patch with no resourceVersion to conflict on. helm-controller's finalizer, which the hook reads to tell whether a release being deleted is still held, is now tied to the constant the controller uses, as the fence already was. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
af50c4a6b4 |
fix(fluxcd): let helm-controller uninstall before the hook unfences
The pre-delete hook lifted flux-plunger's suspension of a release being deleted and removed its fence in the same step. helm-controller runs the uninstall of a release being deleted once it is no longer suspended, but when its own finalizer is already gone the fence is all that keeps the object, so removing the fence at once deleted the release before helm-controller could reconcile it, without an uninstall. The hook now keeps such a fence while helm-controller handles the generation the lifted suspension wrote, until shortly before its retry window closes, and names a release it still had to let go of as before. A release being deleted under a suspension another manager also holds was reported as cleanly released, although helm-controller skips the uninstall of a suspended release; it is now named as deleted without one and counted with the others. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
7b84919db3 |
fix(fluxcd): warn about a foreign suspension only on a stuck release
flux-plunger raised its SuspensionNotOwned and TerminatingWhileSuspended Warnings on a suspension it does not own when the release carried its processed-version annotation. That annotation is never removed, so every later suspension of such a release, by a migration, a backup restore or an operator, drew a Warning: one telling the reader to resume it, which mid-restore is the one thing nobody wants, and one calling a migration's deliberate suspend-then-delete an accident. Both also missed the leftover they are for when the first recovery stopped before writing the annotation. A suspension an earlier flux-plunger left mid-recovery freezes the release in the failure recovery started from, since helm-controller rewrites no condition of a suspended release. Both Warnings now key on that state instead, and the README's candidate query with them. Two comments are qualified where they claimed more than holds: a stop that overran the manager's grace period can still let a reconcile set a fence after the drain's last listing, and a process stopped before controller-runtime closes Elected holds the Lease yet does not drain. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
caeaa6b5e5 |
fix(fluxcd): make flux-plunger's shutdown drain report what it left
The drain's only report was an Info line, so an alert on errors never saw fences left behind; it named no release, and printed held: -1 when no listing had answered. A flux-plunger that never led logged that it had released every fence next to the line saying it drained nothing. Fences left behind are now an error naming each release, an unanswered listing says so, and a process that never led says only that. The drain writes its Events itself, since the manager's recorder stops with the manager, but it created one Event object per pass for a release it revisits every second, and each write had a deadline of its own outside the drain's. Repeats now fold into one counted Event, as the manager's recorder folds them, and every write is bounded by the drain. Tests now hold the outcome line, the retry of a release whose fence cannot be given up yet, and the pause between passes, none of which any suite noticed when removed. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
8bcb709ce8 |
fix(fluxcd): refuse a flux-plunger drain deadline that leaves no time
A drain deadline of zero or less is already spent when the drain starts, so flux-plunger would exit holding every uninstall fence without listing one. The chart could produce one: its 40-second floor was a literal, so raising the manager's stop bound or the margin before SIGKILL, which the drain subtraction is already tied to, would leave the floor rendering a non-positive deadline. The binary now refuses such a flag at startup, before it reaches for a cluster, and the chart's floor and the timings the README and the values comment state are held to the same constants. A whole number of a million seconds or more printed in exponent form and was refused as a fraction; the range is now checked first, so it gets the range message. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
7f454d810c |
chore: link the FOSSA license scan for CLOMonitor (#4899)
## What this PR does Adds a root `.clomonitor.yml` that points CLOMonitor's `license_scanning` check at the FOSSA project for this repository. This is step 2 of #4758. ```yaml licenseScanning: url: https://app.fossa.com/projects/git%2Bgithub.com%2Fcozystack%2Fcozystack ``` - **Why a metadata file rather than a badge.** The check passes on a FOSSA or Snyk link in the README or on a URL in `.clomonitor.yml` ([check docs](https://github.com/cncf/clomonitor/blob/main/docs/checks.md#license-scanning)). #4222 trimmed the README badge row, so this uses the file. - **A correction to #4758.** The key is `licenseScanning`, in camelCase. CLOMonitor deserialises the file with serde `rename_all = "camelCase"` ([`metadata.rs`](https://github.com/cncf/clomonitor/blob/main/clomonitor-core/src/linter/metadata.rs), [template](https://github.com/cncf/clomonitor/blob/main/docs/metadata/.clomonitor.yml)). The snake_case `license_scanning` I proposed in the issue would have been ignored silently. - **Precondition from #4758 is met.** The FOSSA project is now analysed at a current `main` revision (`last_analyzed_revision` = `3a463609e`, 2026-10-12, moved past `bdc3525c9`). It has 0 unresolved licensing issues, and its update hook is the GitHub integration. Checked: the file parses to exactly that one key, the URL matches CLOMonitor's FOSSA regexp, and the project page is public (HTTP 200). Expected effect: `license_scanning` passes on the next CLOMonitor scan, worth about +4 to the project score. This does not close #4758. Per @tym83, licensing findings from the fresh scans stay to be triaged there. ### Downstream repositories <!-- Walked the trigger map in docs/agents/contributing.md against the diff: one new root metadata file read only by CLOMonitor; it touches no package, values file, CRD, asset name, hack/ script or namespace. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Added repository metadata describing license scanning and linking to its results. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
43b02bda30 |
test(e2e): run the Redis backup round-trip against Valkey
The Redis backup demo now takes APP_KIND=Valkey, which switches the app kind and the chart prefix the helpers derive sentinel and Secret names from. The demo BackupClass routes Valkey to the same demo strategy. The valkey suite gains valkey-backup-roundtrip, which drives that demo end to end: seed a marker, back it up, restore it in place and into a copy. It writes to tenant-root's seaweedfs through seaweedfs-s3, so a seaweedfs change now selects the valkey suite too, and since the round-trip runs examples/backups/redis as its harness, an edit there selects both the redis and the valkey suites. The reduced E2E install keeps the backup stack for the valkey suite, as it does for the other backup round-trip suites. Signed-off-by: Andrey Kolkov <androndo@gmail.com> Assisted-By: LLM |
||
|
|
0c2f131192 |
feat(backups): back up Valkey with the Redis strategy
Valkey runs on the same spotahome RedisFailover operator as Redis and speaks the same replication protocol and RDB format, so it needs no strategy kind of its own. The only difference the backup Job sees is the chart's resource-name prefix: valkey-<app> instead of redis-<app> for the sentinel Service and the auth Secret. The Redis driver now accepts applicationRef kind Valkey as well as Redis and hands the template .Release.Prefix, derived from the app's kind, which cozy-default-redis uses for every name it builds. A kind with no known prefix fails the render instead of producing an empty one. cozy-default routes apps.cozystack.io/Valkey to cozy-default-redis. A restore must target an app of the same kind as the backup's source: replaying a Redis dump into Valkey, or the reverse, is not something the strategy can promise across engine releases. The shipped template reads the prefix as `.Release.Prefix | default "redis-"`: during an upgrade the previous controller, which has no Prefix in its render context and serves only Redis, renders the new CR, and a bare reference would print "<no value>" into the names. The redis:8 image already loads RDB v11, which both shipped Valkey versions (7.2, 8.1) write. Signed-off-by: Andrey Kolkov <androndo@gmail.com> Assisted-By: LLM |
||
|
|
d5c59aef22 |
chore: link the FOSSA license scan for CLOMonitor
CLOMonitor's license_scanning check passes when the README links a FOSSA
or Snyk project, or when a root .clomonitor.yml names the results URL.
#4222 trimmed the README badge row, so use the metadata file.
The key is `licenseScanning` (camelCase). CLOMonitor deserialises the
file with serde `rename_all = "camelCase"`, so the snake_case
`license_scanning` proposed in #4758 would be ignored without an error.
The FOSSA project is analysed at a current main revision again
(
|