mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-03 18:47:04 +09:00
feat/gpudevice-api-server
7616
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Aleksei Sviridkin
|
06413db809 |
feat(api): add gpuclass and gpudevice option sources
The dashboard orders GPUs by class and attaches them by device name, so it needs the catalog and the tenant's own devices without reading GPUClass or GPUDevice objects itself. The gpuclass source lists every class that still takes orders, a deleted one excluded, with its display name, a short description of the offer, the free units and the most free units on one node from the class status, the per-VM maximum, and the requesting tenant's remaining budget. That budget comes from the shared model over the tenant and each ancestor's subtree, from their declared resourceQuotas and every device, and is 0 for a vGPU class without the entitlement. Pods are read only when an ancestor limits a class's resource name with a raw key, one list per namespace of the outermost such ancestor's subtree, and a virt-launcher pod does not count the units its attached devices already cover. No field names a node. The gpudevice source lists the devices in the request namespace. cozystack-api gets read access to both kinds, and get and list on pods for the raw-key case. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
231f3703f9 |
feat(api): hide raw GPU options when raw names are forbidden
Once the hoster sets gpu.rawResourceNames to forbidden, a VM gets GPUs only through device references, and admission refuses a GPU entry anywhere else. The gpu source would still offer raw resource names, and the instancetype source would still offer instance types such as the gn1 family that bring their own GPU entries, so the form would lead to a VM that admission then refuses. Read the setting and the GPU resource-name set from the controller's settings ConfigMap. Under forbidden the gpu source is empty, and an instance type with any gpus entry, or a hostDevices entry on a GPU resource name, is left out; a NIC passed as a host device keeps its instance type listed. An unreadable setting serves the full lists: the pickers are advisory and admission enforces the policy. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
0fee13a25a |
feat(api): add GPU order bounds to OptionItem
A GPU class picker needs more than a name: the portal offers the smallest of the class's most free units on one node, its per-VM maximum and the tenant's remaining budget as the largest order, and shows the class's free units. Carry them as optional integers on the option item, so an absent field keeps meaning that nothing limits the order there and every other option source stays unchanged on the wire. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
331b9039b8 |
feat(api): refuse raw GPU quota keys when raw names are forbidden
With gpu.rawResourceNames set to forbidden, GPUs reach tenants only as GPUDevices, budgeted by gpu.cozystack.io keys. A hoster sets it only once no quota key on a GPU resource name itself is left, and tenant validation now keeps it that way: a raw GPU key, bare or as requests.* or limits.*, is refused, so nobody can bring the old budget form back. The setting and the effective GPU resource-name set are read from the settings ConfigMap the controller writes, through the uncached client the API already uses for quota reads, so the Deployment needs no new flags. Without the ConfigMap raw names stay allowed. The read happens only when the tenant declares a key that can be a resource name. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
93e04af76d |
feat(api): keep GPU budget keys out of the tenant quota gate
A gpu.cozystack.io/<class> or gpu.cozystack.io/total key caps the whole subtree of the tenant that declares it and is enforced when a GPUDevice is ordered. The tenant chart does not render these keys into the namespace ResourceQuota, so a tenant declaring only GPU keys gets no quota objects and draws from its parent's pool like a tenant that declares nothing. The declaration gate read the same values verbatim. It would carve a child's GPU key out of its parent's and siblings' as if it were a pod resource, and it treated a GPU-only tenant as bounded, leaving that tenant's usage out of its parent's pool. Drop the GPU keys where the declared quotas are parsed, so the child, the parent, the siblings and the pool membership all see the same keys the chart renders. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
036e5fbc41 |
fix(api): read pool usage under the quota key the controller counts
The tenant declaration gate reads the parent pool's current usage from ResourceQuota status.used under the key that carries a resource's limit-side budget, limits.<name>. The quota controller fills limits.* usage only for cpu, memory and ephemeral-storage. For an extended resource, a GPU name included, and for hugepages it counts only requests.<name>, so the limits.<name> key the chart also renders always reports zero. The gate therefore saw such a resource as unused and could grant a child tenant quota the parent pool was already consuming. Read the usage of every other resource the chart renders with a prefix under requests.<name>, where the chart applies no ratio, so that key holds the same declared amount, and read a key that is already a usage key as declared. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
f5d8249d3a |
test(gpu): pin comparison tolerations, sidecar order and hashed names
The rule that Gt and Lt tolerations never tolerate a taint had no case, nor did an init container that finishes before a sidecar starts, and the long-name check compared a pure function with itself. Each now has a case that fails when the behaviour changes, the long name against a hash computed outside the code under test. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
33143a3e01 |
fix(gpu): count sidecar GPUs the way the scheduler does
An init container that always restarts runs beside the main containers
and every later init container, so its GPUs add to theirs. Counting it
as an ordinary init container booked fewer units than the pod holds,
which let the binder overbook a node and undercounted raw quota keys.
Gt and Lt tolerations no longer count as tolerating a taint: where
their feature gate is off the scheduler ignores them, and skipping a
node is safer than binding to one no pod can land on.
The webhook contract now states that every refusal check, template
budgets and entitlement included, runs before anything is written.
Fixes:
|
||
|
Aleksei Sviridkin
|
68ba964f07 |
fix(api): refuse GPU class resource names a device plugin cannot have
A resource name in gpu.cozystack.io would read as a GPU budget key, so
a raw quota key on it would be dropped from the namespace quota and
counted as a budget at once. A name part over 63 characters is not a
valid extended resource, so such a class could never match a node.
Fixes:
|
||
|
Aleksei Sviridkin
|
77992514f9 |
test(gpu): pin template device names cut at a dot or a dash
A long VMI name is cut before the hash is appended; when the cut lands on a dot the result is not a DNS subdomain unless the trailing dot or dash is dropped, and no input exercised that boundary. The parser's doc no longer claims to validate names it leaves to the lookup. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
7aa381b27f |
fix(gpu): refuse any data after the device reference list
A decoder's More reports false at a closing bracket or brace, so
a stray "]" or "}" after the list still parsed as valid. Reading one
more token and requiring end of input refuses every trailing form.
With the VMI name gone from the template-device label, a template
device whose webhook died before attaching it carries no link to a
VMI; the contract now says the controller deletes such a device after
the attachment grace, and that the VMI is read from the attachment.
Fixes:
|
||
|
Aleksei Sviridkin
|
ef53ba9070 |
fix(gpu): keep template device names and marks within API limits
A VMI name can be a 253-character DNS subdomain, and worker VMIs
named after Cluster API machines do grow long. Carried as a label
value it fails the 63-character limit, and with a suffix appended it
fails the name limit, so the webhook could not create the template
device at all. The mark is now a plain "true", and a name too long
for the suffix is shortened with a hash of the full VMI name.
The device-reference parser also refused nothing after the JSON list,
so a malformed annotation could pass as valid.
Fixes:
|
||
|
Aleksei Sviridkin
|
d283ca701b |
test(gpu): name the subtree budget case after the rule it pins
The case checks that a GPU key on an ancestor limits every tenant below it; its name now says that and nothing about what the child declares. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
5d90af051b |
feat(gpu): add the shared GPU placement and budget model
The binder re-checks what admission checked, and cozystack-api shows the remaining budget a later order is checked against; computing these in three places would let them disagree. The model is pure, so each caller builds a snapshot from its own cache. GPU budgets cap the whole subtree of the tenant that declares them, walked over the namespace ancestry and independent of the tenant quota pools. A raw GPU key blocks a class until the same tenant also declares a GPU key, and then caps devices and raw-name pods together. A node takes binds only when it is schedulable, Ready and carries no NoSchedule or NoExecute taint the pod does not tolerate; the node.kubernetes.io taints are left to the first two checks, and a taint never counts as a cordon. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
a11737a2dd |
feat(gpu): fix the shared GPU keys, settings and binding contract
The binder, the admission webhooks, the tenant chart, the policies and cozystack-api all have to agree on the same label, annotation, finalizer and quota keys and on the settings ConfigMap the policies take as a parameter, so they are defined once here. The binder in the leader is the only writer of placement; webhooks running in every replica record an attachment and a bind request in the device status and wait for the binder's answer, so capacity and budgets are decided in one serialized loop. Status is the carrier because no tenant role can write it. Register and Setup exist as no-ops so the controller can be wired to them before the webhooks and reconcilers land. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
0d58dedfe9 |
feat(api): add gpu.cozystack.io GPUClass, GPUDevice and GPUNode
GPUs become a catalog of hoster offers, tenant reservations against them and a per-node inventory, so a tenant can hold a GPU while its VM is off and an order that does not fit is refused at create time. The CRDs enforce what the apiserver alone can: a class name is a DNS label other than total because it becomes part of a quota key, the source is a union with dra reserved for a later member, a device's spec is immutable and orders at least one GPU, and bound devices are listable per node through a field selector on status.nodeName. The CRDs are kept on chart uninstall, since they ship as plain templates and deleting GPUDevice would drop every hold and usage record. maxCount carries no schema default: it defaults to 1 for vgpu classes and to no limit for passthrough, which a structural default cannot express, so readers resolve it through the shared model. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
myasnikovdaniil
|
d1d4e92acb |
feat(release): sign the release assets and attach a source SBOM (#4861)
## What this PR does Implements the release-asset part of #4759: a release cut from a tree that has this change gets an SBOM of its source and one keyless signature that covers all of its assets. These are the two CLOMonitor checks still failing for this repository, `sbom` and `signed_releases`, after #4757 and #4899 brought the project score to 96.69. `hack/upload-assets.sh` now collects its asset list into an array and hands it to the new `hack/sign-release-assets.sh` before the first upload. The script writes three files into `_out/assets`, and they are uploaded with the rest: | File | What it is | |---|---| | `cozystack-source-sbom.spdx.json` | SPDX 2.3 SBOM made with syft, over `git archive HEAD` (the committed release tree, without `_out/`, `.release-tooling` or runner leftovers) and the unpacked `cozypkg-*.tar.gz` binaries. It does **not** describe the container images that `packages/` pins by digest (their SBOMs are #2937, A2 in #3278) or the Talos ISO, disk, kernel and initramfs images. It is named by what it covers, so `cozystack-sbom` stays free for an SBOM of the release as a whole. CLOMonitor's `sbom` check matches `sbom` in an asset name. | | `cozystack-checksums.txt` | SHA-256 of every uploaded asset and of the SBOM. | | `cozystack-checksums.txt.sigstore.json` | Sigstore bundle of `cosign sign-blob` over the checksums, keyless, signed with the job's GitHub OIDC identity. | Before it returns, the script runs the same `cosign verify-blob` that `docs/release.md` gives users, with `--certificate-identity-regexp '^https://github\.com/cozystack/cozystack/\.github/workflows/(tags\.yaml@refs/tags/v|promote-rc\.yaml@refs/heads/main$)'`: `tags.yaml` on a `v*` tag push, `promote-rc.yaml` dispatched from `main`. A signature that check would reject therefore fails the job before anything is uploaded. It also refuses its own three outputs as inputs: `promote-rc.yaml` downloads every rc asset, and a stable release has to get freshly computed checksums and a fresh signature, since promotion restamps several assets. Workflows: `tags.yaml::prepare-release` and `promote-rc.yaml::promote`, the two jobs that call `make upload_assets`, get `id-token: write` and two setup steps: - syft v1.54.1 from its release tarball, checked against a committed SHA-256 (`c069905b…`, the release's checksums entry; the checksums file verifies against anchore/syft's keyless release signature). Not `anchore/sbom-action/download-syft`: pinned by commit, it still runs syft's `install.sh` fetched from a tag at run time, in a job that holds the app token, the GHCR login and `id-token: write`. v1.54.1 is past GHSA-mw2c-m758-9v5q (fixed in v1.52.0). `hack/ci-tool-installs-contract.bats` now fails on `anchore/sbom-action`, on a syft install script, on a syft download without `sha256sum -c` and on version or checksum drift between the two workflows. - `sigstore/cosign-installer` v4.1.2, pinned by SHA, with `cosign-release: v3.1.3`: the version `docs/release.md` asks users for, so the job checks its own signature the way they will. The action verifies that binary against cosign's keyless and KMS bundles with its digest-pinned bootstrap cosign. `hack/ci-tool-installs-contract.bats` holds the action commit and the release equal in both workflows and fails on a step without `cosign-release`. `docs/release.md` gets a "Release signatures and SBOM" section with the user-side verification: cosign v3.1.3 or later (v2.6.5 on the v2 line), the two commands joined with `&&`, what the SBOM covers and what the certificate identity proves. It now also says, at each place `promote-rc.yaml` is run, that it is dispatched from `main` only. ### Decisions for the release owners - **Fail closed.** If Fulcio or Rekor is down, the release job fails and nothing is uploaded. I chose that so that no release goes out unsigned. If you would rather publish unsigned than block, the change is a `|| echo ::warning::` around the call in `upload-assets.sh`. - **Promotion runs the rc tree's script.** `promote-rc.yaml` calls `make upload_assets` from the rc checkout, not from `.release-tooling`. The first signed stable release is therefore the promotion of the first rc tagged after this merges. `release-1.6` gets signatures only if this is backported. CLOMonitor reads the latest release, so the score moves only when that release carries the files. - **Certificate identity pins the refs** (decided in review): `tags.yaml@refs/tags/v…` and `promote-rc.yaml@refs/heads/main`. A `promote-rc.yaml` dispatched from another branch fails the job's own check at the upload step, after the staging branch, the `promotion-*` candidate and the empty draft release exist; nothing in `parse` refuses a non-`main` dispatch up front. Until the `v*` tag rulesets exist, a hand-pushed `v*` tag whose tree has an edited `tags.yaml` still signs, which `docs/release.md` says. - **Follow-ups after the first signed release** (#4759 stays open for them): - `project.documentation.signature-verification` in `security-insights.yml` (#4757); - the SBOM row in `docs/security/self-assessment.md`, which says "Not published at this time"; the image caveat above stays in it. I left both alone so that they do not describe releases that don't exist yet. ### How it was checked - Bats 1.14.0 (the version CI installs), with gh, syft and cosign stubbed: - `hack/sign-release-assets.bats`, 8 tests: what the checksums cover; that syft sees the committed tree and the cozypkg binaries but not untracked files; that the verify identity accepts `tags.yaml` on a `v*` tag and `promote-rc.yaml` on `main`, and rejects a fork, another workflow, `tags.yaml.evil`, `tags.yaml` on a branch, and `promote-rc.yaml` on `release-1.6`, on `main-evil` and on a tag; that `docs/release.md` shows the same regexp as the script; that a failed verification fails the script; that it refuses a missing asset, assets split over two directories (with a same-named file in both, so the test fails if the directory check is removed) and its own outputs as input. - `hack/upload-assets.bats`, 4 tests, new: the exact upload list and order (the assets, then the SBOM, the checksums and the bundle); that the checksums name every upload but the last two; zero `gh` calls when verification fails or an asset is missing. The old `upload-assets.sh` fails all four, a version that signs after uploading fails the last two. - `hack/ci-tool-installs-contract.bats`, 8 tests. Putting `download-syft` back, dropping the `sha256sum -c` line, letting `SYFT_VERSION` drift between the workflows, dropping `cosign-release` from one step, letting it drift, unpinning the installer or giving one step another installer commit each fails it. - Putting the old any-ref regexp back in the script fails the identity test and the docs tie; putting it back in the docs alone fails the tie. - The meta-suites `bats-strict-setup`, `bats-no-negated-assert`, `bats-no-exit-trap`, `bats-runner-coverage` and `bats-runner-contract` pass: 90 tests across these eight files. - The pinned regexp gives the same accept/reject results under `grep -E` and Go's `regexp` (what cosign uses). cosign-installer's install script at the pinned commit, run as written with `cosign-release: v3.1.3`, verifies the binary's keyless and KMS bundles and installs v3.1.3 (its SHA-256 matches `cosign_checksums.txt`). `--yes`, `--bundle`, `--certificate-identity-regexp` and `--certificate-oidc-issuer` are listed, not deprecated, in cosign v3.1.3's help. - The `Set up syft` step's script, run as written with `RUNNER_TEMP`/`GITHUB_PATH` set: the checksum passes and `syft version` reports 1.54.1. A real syft v1.54.1 run through `hack/sign-release-assets.sh` (cosign stubbed) over this tree and the v1.6.5 `cozypkg-linux-amd64.tar.gz` took 10 s and wrote an SPDX 2.3 document with 920 packages: 440 golang, 373 npm, 103 github, 4 without a purl; 60 come from the cozypkg binary. - cosign behaviour behind the docs, with a key-signed legacy bundle whose `cert` holds the bare public key, against the documented identity regexp (tlog ignored, since an attacker can upload their own entry): v2.6.4 and v3.1.2 print "Verified OK", v2.6.5 and v3.1.3 reject it. `--new-bundle-format` makes v2.6.4 reject it but not v3.1.2, so the docs give the minimum version and not the flag. A real keyless protobuf bundle (syft's) verifies on v2.6.5 and v3.1.3. - zizmor 1.26.1 `--offline` reports no findings. actionlint 1.7.12 (with `.github/actionlint.yaml`) reports the same 13 pre-existing info-level shellcheck notes in `tags.yaml`/`promote-rc.yaml` as before, nothing new. shellcheck passes on both scripts. `hack/check-commit-trailers.sh` passes. - Not checked: a real keyless signature. It needs the OIDC token of a job in this repository, so the first rc after the merge is the end-to-end test. Cutting an `-alpha.N` first would exercise it without touching an rc. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: I checked the trigger map file by file. The only entry `hack/upload-assets.sh` can trigger is renaming a release asset, because the website install pages hardcode asset names. This PR adds three assets and renames none of the released ones, `cozypkg-checksums.txt` included (the SBOM was renamed to `cozystack-source-sbom.spdx.json` in review, before any release carried it). The workflow and docs changes match no entry. ### Release note ```release-note feat(release): releases carry cozystack-source-sbom.spdx.json, an SPDX SBOM of the source tree and the cozypkg binaries (not of the container or Talos images), cozystack-checksums.txt with the SHA-256 of each asset, and cozystack-checksums.txt.sigstore.json, a keyless Sigstore signature over the checksums made by the release workflow; docs/release.md shows how to verify a download with cosign v3.1.3 or later. ``` Refs #4759 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Releases now include an SPDX software bill of materials, SHA-256 checksums, and a keyless Sigstore signature bundle. * Release assets are verified against the expected workflow identity before upload; uploads stop if verification fails. * **Documentation** * Added guidance on release signatures, SBOM coverage, asset verification, and which branch dispatches release workflows. * **Tests** * Added coverage for generated release files, upload behavior, and signing and verification failures. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Marian Koreniuk
|
bc24b588b2 |
feat(release): sign the release assets and attach a source SBOM
No release carried a signature or an SBOM, so users could verify none of the ISO, disk images, kernel, initramfs, manifests and cozypkg binaries they download, and the OpenSSF Scorecard Signed-Releases check and CLOMonitor's signed_releases and sbom checks failed. hack/upload-assets.sh now hands its asset list to the new hack/sign-release-assets.sh before the first upload. That script writes cozystack-source-sbom.spdx.json with syft over the committed tree and the cozypkg binaries, cozystack-checksums.txt with the SHA-256 of every asset and the SBOM, and cozystack-checksums.txt.sigstore.json with a keyless cosign signature over the checksums. The SBOM does not describe the container images or the Talos boot images, which is what its name says; image SBOMs are #2937. The script verifies the bundle against the identities of tags.yaml on a v* tag and promote-rc.yaml on main before returning, so a bad signature fails the job with nothing uploaded, and hack/upload-assets.bats pins the upload list and that order. The two jobs that call make upload_assets get id-token: write, cosign v3.1.3 from the SHA-pinned cosign-installer, and syft from its release tarball checked against a committed SHA-256, because anchore/sbom-action/download-syft runs an install script fetched at run time; hack/ci-tool-installs-contract.bats now covers both. docs/release.md describes the files, what the SBOM covers, the minimum cosign version to verify a download with, what the certificate identity proves, and that promote-rc.yaml is dispatched from main only. Refs #4759 Assisted-by: LLM Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> |
||
|
myasnikovdaniil
|
aed7fa896e |
docs(release): add changelog for v1.6.5 (#4940)
This PR adds the changelog for release `v1.6.5`.
✅ Changelog has been automatically generated in
`docs/changelogs/v1.6.5.md`.
|
||
|
Myasnikov Daniil
|
4814ccfed9 |
docs(release): note the PostgreSQL restart on upgrade in the v1.6.5 changelog
Clusters with a memory limit below 512Mi and no explicit shared_buffers get the new default on upgrade, which needs a PostgreSQL restart. The changelog described the fix but not the restart. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Assisted-by: LLM |
||
|
cozystack-ci[bot]
|
13f612d55e |
docs: add changelog for v1.6.5
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> |
||
|
myasnikovdaniil
|
8bd6d91dbb |
fix(backups): gate every default Strategy CR on its CRD (#4929)
## What this PR does Upgrade from v1.6.x to v1.7 leaves `backupstrategy-controller` HelmRelease in `UpgradeFailed`: `no matches for kind "Kafka" in version "strategy.backups.cozystack.io/v1alpha1"`, same for Rabbitmq and Redis, and none of the new strategy CRDs get created. CRDs of this chart are ordinary templates, and Helm maps every object of a release through the RESTMapper before applying any of them. Default Strategy CRs for kafka, rabbitmq and redis were gated only on the bucket name, so they render next to their not yet served CRDs, Build() fails, nothing is applied and every retry renders the same manifest. Same happens on a first install with external S3 (`provisionBucket: false`) for all older default strategies, bucket name comes from values there. MongoDB default strategy already has the `crdEstablished` gate, this PR adds it to all other default strategies. On an installed cluster nothing changes. Otherwise first revision creates the CRDs and skips the CRs, then DefaultObjectsGate forces the follow-up upgrade that renders them. CRDs stay templates so later CRD changes still reach the cluster on upgrade. Short explanation is added to `docs/operations/backup-classes.md`. To keep this from coming back there is `hack/backupstrategy-crd-gate.bats`: it renders the chart with the bucket gate open and no CRD served and fails on any Strategy CR from any template, so a new `strategy-*-default.yaml` without the gate goes red without anyone updating a list. It also checks that every CR gates on the CRD this chart ships for its kind. helm-unittest covers both halves of the gate for all 11 templates. Checked on a throwaway kind cluster: - v1.6.4 installed, then upgraded: current main fails with the error above, this branch passes with all 11 CRDs Established and the follow-up upgrade creates the rest - modified CRD is applied on a later upgrade - fresh install converges, and fresh install with external S3 converges here while main fails with "no matches for kind" for eight kinds Fixes #4925. Needs backport to release-1.7, both commits apply there cleanly. ### Downstream repositories - [x] No downstream repository is affected by this change ### Release note ```release-note fix(backups): backupstrategy-controller release no longer gets stuck in UpgradeFailed on upgrade from v1.6 and no longer fails a fresh install with external S3, strategy CRDs are created first and default strategies follow on the next reconcile ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Default backup strategies are now created only after their required resource types are available and the backup bucket is resolved. This helps prevent installation and upgrade issues, including when using externally managed S3 storage. * **Documentation** * Clarified when default backup strategies are created during installation and upgrades. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
f47bab7280 |
test(backups): fail CI on any default Strategy CR rendered without its CRD gate
The CRD gate has now been missed twice: MongoDB shipped without it, and Kafka, RabbitMQ and Redis followed after it existed. helm-unittest only checks the templates a case lists, so a new strategy-*-default.yaml that forgets the gate stays green until an upgrade wedges. Render the chart with the bucket gate open and no CRD served, and fail on any strategy.backups.cozystack.io object from any template. Also check that every rendered Strategy CR gates on the CRD this chart ships for its kind, because a gate naming the wrong CRD passes the first check and keeps the CR out of every live render. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
7204992452 |
fix(backups): gate every default Strategy CR on its CRD
The chart ships its strategy CRDs as ordinary templates, and Helm maps every document of a release through the RESTMapper before it applies any. A default Strategy CR gated only on the bucket name therefore fails Build() with "no matches for kind" whenever the bucket name resolves before the CRD is served, the release applies nothing, the CRDs are never created, and every retry renders the same manifest. That happens on the upgrade that first ships a kind: the Kafka, Rabbitmq and Redis CRs leave a v1.6 cluster's HelmRelease stuck in UpgradeFailed on the way to v1.7. It also happens on the first render of an external-S3 install (provisionBucket=false), where the bucket name comes from values and every older default CR renders next to its absent CRD. MongoDB already gates on crdEstablished for this reason. Give every default Strategy the same gate. On an installed cluster the CRDs are established and the render is unchanged; otherwise the first revision creates the CRDs and skips the CRs, and DefaultObjectsGate, which reads the cozy-default BackupClass routes, forces the follow-up upgrade that renders them once the kinds are served. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
myasnikovdaniil
|
11b19635e0 |
fix(monitoring): send alerts to every Alertmanager replica (#4889)
## What this PR does vmalert sends every alert to every Alertmanager replica instead of to the shared Service. The chart gave vmalert a single notifier, `http://vmalertmanager-alertmanager.<namespace>.svc:9093`. That Service is headless and resolves to the three Alertmanager pods, but each alert reached one replica at a time and moved to another one whenever vmalert reconnected. The Alertmanager README asks for the opposite. It says "Do not load balance traffic between Prometheus and its Alertmanagers" and to point the sender at every instance, because the cluster deduplicates notifications only when each replica receives each alert. What this looked like on three clusters running 1.6.4, read through the Alertmanager API of each pod. - On all three, every active alert sat on one replica and the other two had none. The counts were 0, 0 and 10 on the first, 0, 0 and 23 on the second, and 1, 0 and 9 on the third. - On the third one, which also has an email receiver, the active alert count of each replica changed 163 to 180 times in 24 hours. The three replicas sent 63, 67 and 62 emails in that window. Over the same week only three alert names with a severity routed to email fired there. An alert that leaves a replica expires there and goes out as resolved, and the replica that receives it next sends it as a new firing notification. So a single long-running alert keeps producing firing and resolved notifications, and a receiver that rejects concurrent duplicates sees them as conflicts. The change. - `vm/vmalert.yaml` renders one notifier per Alertmanager pod, `http://vmalertmanager-alertmanager-<i>.vmalertmanager-alertmanager.<namespace>.svc:9093`. That is the address the operator builds for StatefulSet pods (`VMAlertmanager.asPodFQDN`), and the replicas already reach each other through the same per-pod names in their `--cluster.peer` arguments. - The replica count comes from a new `monitoring.alertmanager.replicas` helper, which now also sets `spec.replicaCount` of the VMAlertmanager. The notifier list cannot drift from the number of pods. Why not the operator's notifier selector. `VMAlertNotifierSpec.selector` expands matched VMAlertmanagers into the same per-pod addresses, but only when the VMAlert is reconciled. The VMAlert controller does not watch VMAlertmanager objects, so a VMAlert reconciled before its VMAlertmanager exists, which can happen on a fresh install, would get no notifier until something else triggers a reconcile. A static list renders the same addresses without depending on that order. Upgrade impact. The VMAlertmanager is unchanged, it renders `replicaCount: 3` as before. The VMAlert gets new notifier arguments, so the operator restarts the vmalert pod once. Testing. - `helm unittest` passes for the monitoring chart, 24 suites and 304 tests. The new suite `tests/vmalert_notifiers_test.yaml` checks the three notifier addresses, the replica count of the VMAlertmanager and the release namespace in the addresses. - Each of these breakages turns at least one case of the new suite red. The notifier back on the Service address, the helper set to 2, a hardcoded namespace in the address, and a hardcoded `replicaCount` that no longer follows the helper. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: Walked the trigger map in `docs/agents/contributing.md` against the diff. It changes two templates and the helpers of `packages/system/monitoring` and adds a test file. No values, schema, README, package name, make target or `ApplicationDefinition` changes, so no entry applies. ### Release note ```release-note fix(monitoring): vmalert now sends alerts to every Alertmanager replica instead of the shared Service, so the Alertmanager cluster deduplicates them and a long-running alert no longer produces repeated firing and resolved notifications. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Improvements** * vmalert now configures a notifier URL for each of the three Alertmanager replicas, rather than using a single shared URL. * Alertmanager’s replica count is managed consistently with the notifier configuration. * **Tests** * Added checks for the replica count, all three notifier URLs, and the configured namespace. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
e05c56524e |
fix(backups): prepare standalone VMDisk restore targets and add VM backup/restore e2e round-trips (#3427)
## What this PR does Closes the uncovered VM backup path from #3236. `hack/e2e-chainsaw/backup/` held only a disabled Chainsaw port of the old `backup.bats`, written against the retired API (`kind: VirtualMachine`, `BackupJob.storageRef`/`strategyRef`, `velero-strategy-default`), so it guarded nothing. This removes it, fixes the in-place restore of a standalone VMDisk in the backup controller, and adds two real round-trips in the `vminstance` suite that drive the `examples/backups/{vmi,vmdisk}` scripts, so the documented demo and the automated test stay in step. ### Controller: standalone VMDisk in-place restore `collectUnderlyingResources` in `internal/backupcontroller/velerostrategy_controller.go` returned nothing for any kind but VMInstance, so `prepareForRestore` did nothing for a standalone VMDisk: no HelmRelease suspend, no DataVolume delete, no `keepOriginalPVC` rename. An in-place VMDisk restore then raced Flux and CDI over the PVC name the data mover was repopulating. A standalone VMDisk now reports its own disk as its underlying resource, so the restore preparation already used for VMInstance disks suspends the `vm-disk-<name>` HelmRelease, deletes its DataVolume and renames the live PVC to `<name>-orig-<hash>` before Velero recreates it. Unit tests cover both the VMDisk and the VMInstance branch. A VMInstance is halted before its disks are touched, but a VMDisk target is not, and every VMInstance disk is a VMDisk. Renaming the PVC of a disk that a running VM mounts would leave the original `Terminating` behind `pvc-protection`; Velero's PVC restore action skips a PVC that still exists, so the RestoreJob would report `Succeeded` having moved nothing, and the VM would lose its disk on the next restart. An in-place VMDisk restore therefore fails before changing anything when a Pod that has not terminated mounts the disk's PVC. The message names the Pod and tells the user to stop the VM or restore the VMInstance instead. `TestPrepareForRestore_VMDiskInUse` covers a running mounting Pod (refused, PVC not renamed, HelmRelease not suspended), a finished Pod, a Pod on another disk, and a VMInstance target whose own launcher Pod must not refuse it. A VMDisk Backup taken before the disk was recorded as its underlying resource has none, so the disk is derived from the application name (`vm-disk-<name>`): such a restore is prepared and checked for a mounting Pod too. The per-disk `orLabelSelectors` stay a VMInstance concern: a VMDisk strategy already selects its disk, and Velero rejects a Backup or Restore carrying both `labelSelector` and `orLabelSelectors`, so appending them for a VMDisk would break a strategy written with `labelSelector`. When an in-place restore fails after preparation, nothing is rolled back. Resuming the disk's HelmRelease while its PVC is renamed and its DataVolume deleted would have CDI import a fresh disk under the original name, which looks healthy and hides that the data sits in the `-orig-` PVC. The failure message and a Warning event instead name the HelmReleases preparation suspended and the `-orig-` PVCs holding the original data, read back from the cluster so that a failure before a step ran does not name what it would have left, and say that a HelmRelease Velero already restored is resumed, so the disk has to be checked; `examples/backups/vmdisk/README.md` describes the recovery. Tests cover both selector paths, the pre-upgrade Backup, a failed Pod list (the restore requeues and suspends nothing), and all three failures after preparation: a `PartiallyFailed` Velero Restore, several Restores owning the job, and a Restore that cannot be created. Because preparation takes the target apart before Velero validates the Restore, the RestoreJob is refused before preparation, with nothing changed, when the Velero Backup is gone (expired or deleted) or not `Completed`. This is the same single check #4672 added, now run on every pass behind the preparation failure path, so a Backup that expires after preparation started names the leftovers too. A Velero Restore that ends in `FailedValidation` fails the RestoreJob like `Failed`, with Velero's validation errors and the leftovers in the message, instead of polling forever. A HelmRelease suspend or a VM halt that does not happen now stops preparation (requeue) rather than letting the rename and DataVolume delete run under a live release or VM, and a VMInstance's disks are renamed only once no Pod mounts them. A VMDisk held only by a Pod already being deleted is waited for rather than refused, for up to 30 minutes after the RestoreJob started, so a Pod stuck terminating on a lost node fails the job instead of keeping it `Running`. Each of these is pinned by a test that turns red when the guard is removed. Between renaming a kept disk's PVC and deleting its DataVolume, CDI can recreate an empty PVC under the original name for the still-present DataVolume. Velero then skips that PVC as already present ("PVC already exists. Skip restore this PVC."), the RestoreJob reports `Succeeded`, and the VM boots a blank disk. The round-trip's new DataDownload check caught this on a VMInstance restore, where the uid, label and `-orig-` checks had all passed. Once the DataVolume is gone (the wait #4672 added), preparation now deletes a PVC found under the original name (read uncached) before the Velero Restore is created; `TestPrepareForRestore_ClearsPVCRecreatedAfterRename` reproduces the recreation inside the DataVolume delete. Preparation spans several reconciles, so the RestoreJob records a `PreparationStarted` condition before it first changes the target; if recording it fails, preparation retries without changing anything, so a later failure cannot report leftovers that were never made. A refusal on a later pass (the Velero Backup expired, a VM started on the disk) then names what preparation left behind instead of saying nothing was changed, and a preparation still waiting 30 minutes after it started fails the same way instead of polling forever. The vmdisk script starts its consumer Pod only on a PVC that carries the restore label, so it cannot pin a PVC CDI recreated in the rename window. Now that a VMDisk target is prepared, `keepOriginalPVC: false` reaches it as well. Its DataVolume is deleted with orphan propagation, and the bound PVC is deleted and waited for before the Restore (the path #4672 added for VMInstance disks). Left in place, Velero would skip the PVC and the CDI webhook would refuse the DataVolume the resumed HelmRelease creates, leaving the disk's HelmRelease stuck. `TestPrepareForRestore_VMDiskWithoutKeepDeletesOriginalPVC` pins it. ### `vminstance-2-backup-roundtrip` (un-gated in CI) A tiny cirros guest (`u1.nano`, 2Gi, in-place restore only) runs the current API end to end: Velero strategy, BackupClass, a booted VMInstance, an ad-hoc `BackupJob` and an in-place `RestoreJob`, each waited to `Succeeded` and failing fast on `Failed`. The VM must come back `Running`, and its disk PVC must have been created by the restore: a new `metadata.uid`, Velero's `velero.io/restore-name` label, and the `vm-disk-<name>-orig-<hash>` PVC that the rename leaves behind. The label alone does not prove the data moved, because with `existingResourcePolicy: update` Velero also stamps it on a PVC it only patched, and the patch path never reaches the data mover. An in-guest signal is not available on this platform: the KubeVirt CR disables serial-console logging cluster-wide and cirros ships no qemu-guest-agent. ### `vmdisk-2-backup-roundtrip` (un-gated in CI) The disk-only companion: import a 5Gi Ubuntu disk, back it up, restore it in place relying on the controller fix above, and require the same three signals (new uid, restore label, `-orig-` PVC), a `Completed` DataDownload for the restore that wrote bytes, and a Ready HelmRelease. On a `WaitForFirstConsumer` StorageClass, such as the `local` class the CI lane uses, the Velero data mover waits until a scheduled Pod has selected a node for the restored PVC (`WaitPVCConsumed` in Velero's generic restore exposer). A VMInstance gets that from its virt-launcher Pod; a detached disk has none, so its `RestoreJob` stays `Running` until the 30m data-mover prepare timeout fails it. The script therefore starts a restricted-profile `pause` Pod on the new PVC once Velero has created it, standing in for the VM that would attach the disk, and removes it after the restore. `examples/backups/vmdisk/README.md` documents this for operators restoring a detached disk. ### Harness details - The disk `storageClass` is an override (default `replicated`). The Chainsaw steps pass the lane's `COZY_E2E_STORAGE_CLASS` (`local` on the DRBD-off PR gate). - Each harness stands up a throwaway in-cluster BSL (`cozy-e2e-vmi`/`cozy-e2e-vmdisk`) at `seaweedfs-s3.tenant-root:8333` with `insecureSkipTLSVerify` and `checksumAlgorithm: ""` (the #3236 fixes), reusing the platform bucket and `cozy-backups-creds` under a distinct prefix, because the platform `cozy-default` BSL derives the external S3 ingress endpoint, unroutable in the CI cluster. - The script waits are bounded so that their worst case stays under the 40m op timeout (38m for the VMDisk step), and a hung wait trips the script's own named timeout rather than a bare `signal: killed`. - On failure, `catch` also describes the Velero `Restore`, `DataDownload` and `DataUpload` objects in `cozy-velero`. - `skipDelete` plus a `finally` that resumes the suspended HelmReleases, reclaims the `-orig-` PVC and its LINSTOR volume (the PV is flipped back to `Delete` first), and drains the Velero backups on the throwaway BSL, branching on kubectl's exit status, before deleting it. When the drain is not confirmed, the BSL is kept and the step fails, so the Velero Backups are not orphaned behind a green step. - HelmRelease waits use the shared `wait_hr_ready`, which fails fast on `Stalled`. After the VMDisk restore the script first waits for `observedGeneration` to reach the restored spec's `generation`, because the `Ready=True` from before the suspend would otherwise satisfy the wait at once. The `vmi` example is CI-drivable: `run-all.sh`/`cleanup.sh` are non-interactive, the VM is waited to boot before the backup, `volumeSnapshotLocations` is dropped from the demo strategies (`snapshotMoveData` routes data through the BSL), and `BACKUP_STORAGE_LOCATION` defaults to `cozy-default`. `select-e2e.sh` maps `examples/backups/{vmi,vmdisk}` edits to the `vminstance` suite, except `07-restore-to-copy.sh`, which the suite skips (`SKIP_RESTORE_TO_COPY=1`) and so stays docs-only. A `packages/system/velero` change selects the `vminstance` suite as well, since these are the only Tests that run a Velero backup and restore. ### Screenshots N/A — no UI changes. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff (backup controller restore preparation for VMDisk, shell scripts under `examples/backups/`, Chainsaw tests, `hack/select-e2e.sh` routing, a deleted stale suite, one doc line): no `packages/apps|extra` add/rename, no `values.schema.json`/enum/default changes, no CRD/ApplicationDefinition/label/annotation/namespace/metric changes. The controller change alters what a VMDisk `RestoreJob` does in the cluster, not any API surface a downstream repository consumes. `cozystack/examples` holds Terraform for bare nodes, unrelated to the in-repo `examples/` tree. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(backups): an in-place RestoreJob for a standalone VMDisk now suspends the disk's HelmRelease, deletes its DataVolume and (with keepOriginalPVC) renames the original PVC before Velero restores it, instead of racing Flux and CDI over the same PVC; without keepOriginalPVC the original PVC is deleted so Velero recreates it from the backup. Such a restore fails before changing anything when a running Pod, such as the launcher of a VM that uses the disk, mounts the disk's PVC: stop the VM or restore the VMInstance instead. If such a restore fails after the disk was prepared, its message and a Warning event name the HelmRelease preparation suspended and the PVC holding the original data. On a WaitForFirstConsumer StorageClass the data mover still starts only once a Pod consumes the restored PVC, for example when the disk is attached to a VMInstance. An in-place restore with keepOriginalPVC no longer reports Succeeded over an empty PVC that CDI recreated under the disk's name while the original was being kept aside. Adds Chainsaw backup/restore round-trips for VMInstance and standalone VMDisk that run a real Velero backup and restore in CI. ``` |
||
|
Aleksei Sviridkin
|
588d8292e7 |
fix(seaweedfs): build SeaweedFS in-tree with a bounded S3 read-ahead (#4819)
## What this PR does Large objects served through a Bucket's S3 endpoint stop tens of megabytes into the download (cozystack/cozystack#3782). The cause is not TLS: the SeaweedFS S3 gateway is OOM-killed mid-response. Its read-ahead for a sequential GET is meant to stay four chunks in front of the reader, but SeaweedFS counts only newly started chunk downloads toward that window and re-runs the prefetch after every 256 KiB read, so the gateway pulls up to about a gigabyte of the object into memory and exceeds the 512Mi limit the S3 deployment gets from its resources preset. The client sees `unexpected eof` over HTTP/1.1 and `INTERNAL_ERROR` over HTTP/2, which is what the CDI importer reported. Upgrading does not help yet: 4.31, 4.45 and the latest release 4.48 all carry the defect. The defect is reported upstream in seaweedfs/seaweedfs#11672, and the fix with its regression test is merged there in seaweedfs/seaweedfs#11674, after 4.48. Until a SeaweedFS release carrying it is vendored, this PR builds SeaweedFS in-tree the way #3343 builds cilium-operator: `images/seaweedfs` downloads the 4.45 source archive and checks it against a pinned checksum, applies `patches/reader-cache-prefetch-window.diff`, runs the tests of the packages the patch touches (including the regression test the patch adds) once, compiles `weed` with the Go toolchain upstream's own builder uses for the tag, and lays it over the upstream `chrislusf/seaweedfs:4.45` image pinned by digest, so everything else in the image stays the upstream build. `weed version` reports `cozystack-patched` in the commit field. All SeaweedFS components run the in-tree image. `make update` re-pins the Dockerfile from the vendored chart: the version, the source checksum, the upstream image digest and the builder toolchain. It fails when the patch no longer applies, which is how the first release carrying the upstream fix will show that the patch is ready to drop. Renovate is disabled for that Dockerfile, since it would move those pins one at a time, and whitespace checks are off for the patch. Tests: - `tests/image_test.yaml` pins every component (master, volume, filer, S3) to the in-tree image with a digest. - `make test-image-version` fails when the Dockerfile's version drifts from the vendored chart's appVersion. - `make update-image`, which `make update` runs, was checked to be idempotent on the current chart, to restore deliberately corrupted pins, and to fail with a clear message when the patch context no longer matches. - `hack/build-matrix_test.bats` now expects seaweedfs to be a build unit next to objectstorage-controller. - The Go regression test inside the patch fails on unpatched 4.45 (read-ahead reaches 60 chunks past the reader) and passes with the patch; the image build runs it. Verified on a test cluster against an 840 MiB object, with every SeaweedFS component running the image this package builds and the S3 gateway at the production limits (512Mi, 1 CPU). Unpatched 4.31, 4.45 and 4.48 gateways were OOM-killed on every full GET. The patched gateway served nine full GETs (six sequential over HTTP/1.1 and HTTP/2, three in parallel) with matching checksums and no restarts; it read from the volume servers about as many bytes as it sent, its live heap after a GET was about 40 MiB, and three parallel GETs peaked at about 140 MiB. The image this PR pins reports `4.45 cozystack-patched` and was rerun through the same checks. Drop the patch, and the in-tree build with it, once a SeaweedFS release carrying the upstream fix is vendored. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(seaweedfs): stop the S3 gateway from being OOM-killed while serving large objects, which cut downloads (including CDI imports from a Bucket URL) off tens of megabytes in. SeaweedFS is now built in-tree with a patch that bounds the gateway's read-ahead. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * SeaweedFS now runs a Cozystack-built image based on version 4.45, pinned to a specific image digest. * **Bug Fixes** * Sequential small reads now keep read-ahead within the configured prefetch window, preventing it from extending unexpectedly. * **Build & Maintenance** * SeaweedFS image updates are aligned with the chart version and verify the source archive, builder image, and patches before building. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrey Kolkov
|
58b4343e5a |
ci(hack): route examples/backups/{vmi,vmdisk} edits to the vminstance suite
The two backup round-trips run the examples/backups/{vmi,vmdisk} scripts as their
harness, but select-e2e.sh mapped an examples/backups/<dir> edit to a suite only
when <dir> was itself a suite name. The dirs are vmi/vmdisk while the suite is
vminstance, so a harness-only edit selected no suite and the round-trip that runs
it never ran. Translate both dirs to vminstance, matching the vm-disk-application
entry in src_to_suites, and pin it with a bats case. The suite runs the vmi demo
with SKIP_RESTORE_TO_COPY=1, so 07-restore-to-copy.sh stays docs-only: selecting
the suite for it would pass without executing it. Both round-trips write to
the in-cluster seaweedfs-s3 endpoint, so a seaweedfs change selects the
vminstance suite as well. Also correct the controller-
owned-teardown note in docs/agents/e2e-testing.md (only Backup/RestoreJob
finalizers reach Velero; a BackupJob has none).
The two round-trips are also the only Tests that run a Velero backup and
restore, so the velero PackageSource maps to the vminstance suite; it
otherwise maps to a velero suite that does not exist, and a Velero change
ran neither round-trip.
Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
|
||
|
Andrey Kolkov
|
9cc3a9e459 |
test(vmi): add VMInstance and standalone VMDisk backup/restore e2e round-trips
The disabled hack/e2e-chainsaw/backup/ suite was a Chainsaw port of the retired
backup.bats, written against the old backup API, so the VMInstance and standalone
VMDisk Velero paths had no e2e coverage. Replace it with two round-trips in the
vminstance suite that drive the validated examples/backups/{vmi,vmdisk} scripts,
so the documented demo and the automated test stay in step.
Both run un-gated in CI:
- vminstance-2-backup-roundtrip boots a tiny cirros VM (u1.nano, 2Gi), backs it
up via BackupClass + BackupJob, restores in place, and asserts the VM returns
Running off a disk PVC the restore created: a new uid, Velero's restore-name
label and the -orig- PVC keepOriginalPVC leaves behind. The label alone is not
enough, because existingResourcePolicy: update also stamps it on a PVC Velero
only patched, which moves no data. A guest-side boot proof is unavailable: the
KubeVirt CR disables serial-console logging and cirros ships no guest-agent.
- vmdisk-2-backup-roundtrip imports a 5Gi disk, backs it up, and restores it in
place with the same three assertions. The controller prepares the target
(suspend HelmRelease, delete DataVolume, rename PVC). On a WaitForFirstConsumer
class the Velero data mover waits for a Pod to select a node for the restored
PVC, which a detached disk never gets, so the script starts a pause Pod on the
new PVC, standing in for the VM that would attach it.
The disk storageClass is an override (default replicated; the CI steps pass the
lane's COZY_E2E_STORAGE_CLASS, 'local' on the DRBD-off gate) and the script waits
are bounded so their worst case stays under the 40m op timeout, sized to keep the
whole parallel:1 suite inside the 215m job cap. On failure the catch also dumps
the Velero Restore, DataDownload and DataUpload objects. Each harness stands up a
throwaway in-cluster BSL (the platform cozy-default BSL derives the external S3
endpoint, unroutable in kind); its finally resumes the suspended HelmReleases,
reclaims the keepOriginalPVC orig PVC and its LINSTOR volume, and drains the
Velero backups before deleting the BSL. The example scripts are non-interactive
so the harness can drive them.
Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
|
||
|
Andrey Kolkov
|
9af1c0f902 |
fix(backups): prepare standalone VMDisk restore targets before the data mover runs
collectUnderlyingResources returned nothing for a non-VMInstance kind, so prepareForRestore did nothing for a standalone VMDisk: no HelmRelease suspend and no DataVolume delete. An in-place VMDisk restore then raced Flux/CDI — the restored HelmRelease reinstalled and CDI recreated the DataVolume over the same PVC name the data mover was repopulating, wedging the RestoreJob. Report a standalone VMDisk's own disk as its underlying resource, so the existing (VMInstance-proven) restore preparation suspends the vm-disk-<name> HelmRelease, deletes its DataVolume and — with keepOriginalPVC — renames the live PVC before the data mover recreates and repopulates it. No VM, so no IP/MAC is collected. A VMInstance is halted before its disks are touched; a VMDisk target is not, and every VMInstance disk is a VMDisk. Renaming the PVC of a disk a running VM mounts leaves the original Terminating behind pvc-protection, Velero skips a PVC that still exists, and the RestoreJob would report Succeeded having moved nothing while the VM loses its disk on the next restart. An in-place VMDisk restore now fails before changing anything when a Pod that has not terminated mounts the disk's PVC, naming the Pod and telling the user to stop the VM or restore the VMInstance instead. A VMDisk Backup taken before the disk was recorded as its underlying resource has none, so its disk is now derived from the application name and such a restore is prepared and checked for a mounting Pod as well. The per-disk orLabelSelectors stay a VMInstance concern. A VMDisk strategy already selects its disk, and Velero rejects a Backup or Restore that carries both labelSelector and orLabelSelectors, so appending them for a VMDisk would break a strategy written with labelSelector. When an in-place restore fails after preparation, nothing is rolled back: resuming the disk's HelmRelease while its PVC is renamed and its DataVolume deleted would have CDI import a fresh disk under the original name, which looks healthy and hides that the data sits in the -orig- PVC. The failure message and a Warning event now name the suspended HelmReleases and the -orig- PVCs so the user can recover them, read back from the cluster, so a failure before a step ran does not name what it would have left. Preparation takes the target apart before Velero validates the Restore, so it is refused up front when the Velero Backup is gone or not Completed. That check now runs on every pass behind the same failure path as the rest of preparation, and a FailedValidation Restore fails the RestoreJob like a Failed one instead of leaving it Running. A HelmRelease suspend or a VM halt that did not happen now stops preparation instead of renaming and deleting under a running release or VM, and a VMInstance's disks are renamed only once its launcher Pod has let go of them. A VMDisk held only by a Pod already being deleted is waited for rather than refused, for up to 30 minutes after the RestoreJob started. Between renaming a kept disk's PVC and deleting its DataVolume, CDI can recreate an empty PVC under the original name. Velero then skips the PVC as already present, the RestoreJob reports Succeeded and the VM boots a blank disk. Once the DataVolume is gone, preparation now deletes a PVC found under the original name, read uncached, before the Velero Restore is created. Preparation spans several reconciles, so the RestoreJob records a PreparationStarted condition before it first changes the target, and retries without changing anything when recording it fails. A refusal on a later pass (the Velero Backup expired, a VM started on the disk) then names what preparation left behind instead of claiming that nothing changed, and a preparation still waiting 30 minutes after it started fails the same way instead of polling forever. Now that a VMDisk target is prepared, keepOriginalPVC=false reaches it too: its DataVolume is deleted with orphan propagation and the bound PVC is deleted before the Restore, as for a VMInstance disk. Left in place, Velero would skip it and CDI would refuse the DataVolume the resumed HelmRelease creates. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Alexey Artamonov
|
e8635f6e4b |
build(seaweedfs): stamp the re-pinned in-tree image
The builder, the compile flags and the version stamp changed, so the image built from them has a new digest, and values.yaml has to name it for the components to run that build. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Alexey Artamonov
|
ecdc043e63 |
build(seaweedfs): keep Renovate and git whitespace off the in-tree build
Renovate would bump the builder, the base image or their digests one at a time into a combination upstream never shipped, and the next `make update` would silently put them back, so it leaves that Dockerfile to the update recipe. The patches are unified diffs of Go sources, whose context lines start with a marker before tabs and blank lines, which git's whitespace checks would otherwise flag or rewrite. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Alexey Artamonov
|
5592a3f65d |
build(seaweedfs): re-pin the in-tree build from make update
The in-tree build pinned its source, base image and toolchain by hand, and nothing moved them when `make update` vendored a new chart, so a bump would build one release's weed into another release's image, or keep patching a release that already carries the fix. Pin the source archive by checksum and the upstream image by digest as build arguments, compile with the toolchain upstream's own builder uses for the tag, and let `make update` re-pin all of them from the vendored chart and fail when the patches no longer apply. The tests of the patched packages now run once rather than once per target platform, and `weed version` names the build. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Alexey Artamonov
|
d2c6b85cac |
fix(seaweedfs): run the in-tree SeaweedFS image in every component
With the image keys unset the vendored chart renders chrislusf/seaweedfs at its appVersion, so the patched build from images/seaweedfs would be built and never run, and the S3 gateway would keep being OOM-killed on large GETs. Point every component at the in-tree image, pinned by digest like every first-party reference, and hold that in a unit test so a values change cannot quietly fall back to the upstream binary. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Alexey Artamonov
|
1913104bd3 |
build(seaweedfs): build SeaweedFS in-tree with a bounded read-ahead
The S3 gateway is OOM-killed while serving a large object. SeaweedFS starts read-ahead downloads after every read of a sequential stream and counts only the downloads it newly starts toward its prefetch window, so a GET reading 256 KiB at a time out of 4 MiB chunks pulls up to about a gigabyte of the object into memory, past the 512Mi limit of the S3 deployment. No SeaweedFS release carries a fix yet, so there is no release to move to instead. Build weed from the 4.45 sources with a patch that keeps the window at the next prefetchCount chunks, and lay it over the upstream image of the same tag so the rest of the image stays upstream's own. The image build runs the tests of the packages the patch touches, including the regression test the patch adds, because go build never compiles them. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Aleksei Sviridkin
|
c0449eeabb |
feat(backups): back up Valkey with the Redis strategy (#4898)
## What this PR does
Adds backup and restore for the managed Valkey app.
Valkey runs on the same spotahome RedisFailover operator as Redis
(`spec.engine: Valkey`), with the same sentinel topology, replication
protocol and RDB format. So instead of adding a new strategy kind, the
existing Redis strategy now serves both apps:
- **Driver.** `applicationRef.kind` may be `Redis` or `Valkey`. The
template context gets `.Release.Prefix`, which the driver derives from
the app's kind (`redis-` or `valkey-`). The chart uses that prefix for
every name it builds: the sentinel Service `rfs-<prefix><app>`, the auth
Secret `<prefix><app>-auth` and the CA Secret. In cleanup mode the
prefix comes from the Backup's `applicationRef`, so it works after the
app is gone. A kind with no known prefix fails the render instead of
silently producing an empty prefix. Error messages name the real kind.
- **Upgrade window.** The shipped template reads `{{ .Release.Prefix |
default "redis-" }}`. The strategy CR and the controller ship in one
release, and until the new controller takes the lease, the previous one
(which has no `Prefix` and serves only Redis) renders the new CR; a bare
reference would print `<no value>` into the Secret and sentinel names.
- **Same-kind restore only.** A restore whose target kind differs from
the backup's source kind fails before any Job is created. Replaying a
Redis dump into Valkey, or the reverse, is not something the strategy
can promise across engine releases.
- **Chart.** `cozy-default-redis` builds names from `.Release.Prefix`.
`cozy-default` routes `apps.cozystack.io/Valkey` to
`cozy-default-redis`; the DefaultObjectsGate already dedupes repeated
strategy refs. No new image: the `redis:8` base loads the RDB v11 that
both shipped Valkey versions write. The Valkey chart has no TLS option,
so the optional CA mount stays empty and the Job connects in plaintext.
- **CRD.** Only the `.Release.Prefix` description in the Redis strategy
CRD changes (regenerated with controller-gen v0.16.4; every other
strategy CRD regenerates byte-identical).
- **Docs.** `docs/operations/backup-classes.md` has the Valkey route and
a paragraph on how the shared strategy behaves.
- **E2E.** The Redis backup demo takes `APP_KIND=Valkey`, which switches
the app kind and the name prefix. Its BackupClass routes Valkey too. The
valkey suite gains `valkey-backup-roundtrip`, which seeds a marker,
backs it up, restores it in place and into a copy. In
`hack/select-e2e.sh`, a seaweedfs change now selects the valkey suite
(enforced by `select-e2e_test.bats`), an edit to the shared
`examples/backups/redis/` harness selects both `redis` and `valkey`
(pinned by a new test), and `hack/select-install.sh` keeps the backup
stack in the reduced install for the valkey suite (pinned in
`select-install_test.bats`).
### Validation
- `go test ./internal/backupcontroller/...` passes. New tests: a Valkey
BackupJob renders `--prefix=valkey-`; a Redis backup restored into a
Valkey target fails without creating a Job; the ref validator accepts
Valkey; a cleanup for a kind with no prefix is released without a Job.
The existing Redis render test now also asserts `redis-`.
- The `default "redis-"` fallback, run through the controller's own
template engine with no `Prefix` in the context, renders
`rfs-redis-cache`; the bare reference renders `rfs-<no value>cache`.
- `helm unittest` on backupstrategy-controller: 26/26, with
`cozy-default` at eleven routes. `helm lint` is clean.
- `hack/redis-backup-strategy.bats` executes the shipped script and adds
a Valkey case: it must query `rfs-valkey-<app>` and never `rfs-redis-`.
The fixture now also fails if any `{{ }}` placeholder survives
substitution. All 62 tests in `hack/select-e2e_test.bats` pass.
- Engine compatibility, checked locally with the strategy image
`redis:8-alpine@sha256:ba6e…` against `valkey/valkey` 8.1.8 and 7.2.13
using the shipped script's exact steps: `redis-cli --rdb` from a
password-protected Valkey master, load into the throwaway
`redis-server`, emit the RESP base AOF, `FLUSHALL`, then `--pipe` into a
fresh Valkey. On both versions every check held: the db0/db1/db3
boundaries, a stream with its consumer group, a key's TTL and the marker
all survived, and FLUSHALL wiped the stale target key.
- I have not run this on a live cluster. The new chainsaw round-trip is
what exercises it end to end.
Out of scope, and already true before this PR: the website's backup
pages in `docs/next` don't list Redis, RabbitMQ or Kafka yet, so none of
the triggers in the downstream map fire here.
### Screenshots
N/A — no UI changes.
### Downstream repositories
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
feat(backups): Valkey apps can now be backed up and restored. The platform `cozy-default` BackupClass routes `apps.cozystack.io/Valkey` to the existing `cozy-default-redis` strategy, which dumps an RDB snapshot to the system bucket and restores it in place or into another Valkey app. A Valkey backup restores only into a Valkey app.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added support for backing up and restoring Valkey applications using
the existing Redis backup strategy.
* Updated the backup walkthrough to support both Redis and Valkey, with
resource names selected for the chosen application type.
* **Bug Fixes**
* Prevented restores when the backup and target application types
differ.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Aleksei Sviridkin
|
7f454d810c |
chore: link the FOSSA license scan for CLOMonitor (#4899)
## What this PR does Adds a root `.clomonitor.yml` that points CLOMonitor's `license_scanning` check at the FOSSA project for this repository. This is step 2 of #4758. ```yaml licenseScanning: url: https://app.fossa.com/projects/git%2Bgithub.com%2Fcozystack%2Fcozystack ``` - **Why a metadata file rather than a badge.** The check passes on a FOSSA or Snyk link in the README or on a URL in `.clomonitor.yml` ([check docs](https://github.com/cncf/clomonitor/blob/main/docs/checks.md#license-scanning)). #4222 trimmed the README badge row, so this uses the file. - **A correction to #4758.** The key is `licenseScanning`, in camelCase. CLOMonitor deserialises the file with serde `rename_all = "camelCase"` ([`metadata.rs`](https://github.com/cncf/clomonitor/blob/main/clomonitor-core/src/linter/metadata.rs), [template](https://github.com/cncf/clomonitor/blob/main/docs/metadata/.clomonitor.yml)). The snake_case `license_scanning` I proposed in the issue would have been ignored silently. - **Precondition from #4758 is met.** The FOSSA project is now analysed at a current `main` revision (`last_analyzed_revision` = `3a463609e`, 2026-10-12, moved past `bdc3525c9`). It has 0 unresolved licensing issues, and its update hook is the GitHub integration. Checked: the file parses to exactly that one key, the URL matches CLOMonitor's FOSSA regexp, and the project page is public (HTTP 200). Expected effect: `license_scanning` passes on the next CLOMonitor scan, worth about +4 to the project score. This does not close #4758. Per @tym83, licensing findings from the fresh scans stay to be triaged there. ### Downstream repositories <!-- Walked the trigger map in docs/agents/contributing.md against the diff: one new root metadata file read only by CLOMonitor; it touches no package, values file, CRD, asset name, hack/ script or namespace. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Added repository metadata describing license scanning and linking to its results. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrey Kolkov
|
43b02bda30 |
test(e2e): run the Redis backup round-trip against Valkey
The Redis backup demo now takes APP_KIND=Valkey, which switches the app kind and the chart prefix the helpers derive sentinel and Secret names from. The demo BackupClass routes Valkey to the same demo strategy. The valkey suite gains valkey-backup-roundtrip, which drives that demo end to end: seed a marker, back it up, restore it in place and into a copy. It writes to tenant-root's seaweedfs through seaweedfs-s3, so a seaweedfs change now selects the valkey suite too, and since the round-trip runs examples/backups/redis as its harness, an edit there selects both the redis and the valkey suites. The reduced E2E install keeps the backup stack for the valkey suite, as it does for the other backup round-trip suites. Signed-off-by: Andrey Kolkov <androndo@gmail.com> Assisted-By: LLM |
||
|
Andrey Kolkov
|
0c2f131192 |
feat(backups): back up Valkey with the Redis strategy
Valkey runs on the same spotahome RedisFailover operator as Redis and speaks the same replication protocol and RDB format, so it needs no strategy kind of its own. The only difference the backup Job sees is the chart's resource-name prefix: valkey-<app> instead of redis-<app> for the sentinel Service and the auth Secret. The Redis driver now accepts applicationRef kind Valkey as well as Redis and hands the template .Release.Prefix, derived from the app's kind, which cozy-default-redis uses for every name it builds. A kind with no known prefix fails the render instead of producing an empty one. cozy-default routes apps.cozystack.io/Valkey to cozy-default-redis. A restore must target an app of the same kind as the backup's source: replaying a Redis dump into Valkey, or the reverse, is not something the strategy can promise across engine releases. The shipped template reads the prefix as `.Release.Prefix | default "redis-"`: during an upgrade the previous controller, which has no Prefix in its render context and serves only Redis, renders the new CR, and a bare reference would print "<no value>" into the names. The redis:8 image already loads RDB v11, which both shipped Valkey versions (7.2, 8.1) write. Signed-off-by: Andrey Kolkov <androndo@gmail.com> Assisted-By: LLM |
||
|
Marian Koreniuk
|
d5c59aef22 |
chore: link the FOSSA license scan for CLOMonitor
CLOMonitor's license_scanning check passes when the README links a FOSSA
or Snyk project, or when a root .clomonitor.yml names the results URL.
#4222 trimmed the README badge row, so use the metadata file.
The key is `licenseScanning` (camelCase). CLOMonitor deserialises the
file with serde `rename_all = "camelCase"`, so the snake_case
`license_scanning` proposed in #4758 would be ignored without an error.
The FOSSA project is analysed at a current main revision again
(
|
||
|
Yan Bondarenko
|
9d4221b890 |
fix(monitoring): send alerts to every Alertmanager replica
vmalert had a single notifier, the headless vmalertmanager-alertmanager Service. It resolves to all three pods, but each alert reached one replica at a time and moved to another when vmalert reconnected. On clusters running 1.6.4 all active alerts sat on one replica while the other two had none, and on one of them the alert count of each replica changed about 170 times a day. The Alertmanager cluster deduplicates notifications only when every replica receives every alert. The replica an alert left let it expire and sent a resolved notification, and the replica it reached sent a new firing one. vmalert now lists each Alertmanager pod as its own notifier, using the <name>-<i>.<service>.<namespace>.svc address the operator itself gives StatefulSet pods. The replica count comes from one helper that also sets spec.replicaCount of the VMAlertmanager, so the two cannot drift. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> |
||
|
Aleksei Sviridkin
|
3a463609e4 |
feat(bucket): add a ui flag to skip the s3manager web UI (#4881)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does Adds a `ui` flag to the Bucket application. Every bucket currently ships an s3manager Deployment, its Service, and an Ingress or HTTPRoute on `<bucket>.<tenant host>`. Tenants that only use the S3 API pay for a pod per bucket and for one DNS name and TLS certificate per bucket, none of which they need. - `packages/apps/bucket`: new `ui` value (default `true`), passed through to the `bucket-system` HelmRelease. The dashboard Role drops its Ingress rule when the UI is off, since an empty `resourceNames` would widen it to every Ingress in the namespace. - `packages/system/bucket`: the Deployment, Service, Ingress and HTTPRoute render only when `ui` is true. With the UI off, the chart no longer requires the tenant apex, as the UI hostname was its only consumer there. - Regenerated `values.schema.json`, `README.md`, `types.go` and the `bucket-rd` ApplicationDefinition. The default keeps existing buckets unchanged. Setting `ui: false` on an existing bucket makes Helm delete the UI objects on the next upgrade; the cert-manager Certificate for the UI host is owned by the Ingress through ingress-shim and goes with it. Tested with `helm unittest` on both charts: new suites cover the default render, `ui: false` behind an Ingress and behind a Gateway, the empty-apex case, the value passthrough and the dashboard Role. The system-chart suite fails against the previous templates. ### Screenshots Not applicable: no dashboard change, only a new field in the Bucket form generated from the schema. ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> Walked the trigger map against the diff. A new field in `packages/apps/bucket/values.schema.json` reaches the Terraform provider (schema, model, expand/flatten) and the website (the Bucket page is hand-written). Both follow-ups are issues rather than PRs, since they apply only once this PR is merged. - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: cozystack/website#750 - [x] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: cozystack/terraform-provider-cozystack#71 - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note feat(bucket): add a `ui` flag to the Bucket application. Set it to `false` to skip the s3manager web UI and its Service, Ingress or HTTPRoute, DNS name and TLS certificate. Defaults to `true`, so existing buckets are unchanged. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a `ui` option for bucket deployments. It is enabled by default and provides the S3 manager web UI at the tenant host. * Set `ui` to `false` to run a bucket without its web UI or the associated Service and Ingress or HTTPRoute. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
cfa9c5a5c7 |
fix(e2e): keep chainsaw asserts polling through transient API errors (#4867)
## What this PR does Chainsaw asserts in the e2e suites can fail long before their timeout. Chainsaw v0.2.15 keeps polling only on NotFound and stops on any other API error. When etcd changes its leader, kube-apiserver answers the GET with a 500, and whichever suite happens to be polling goes red. Between 2026-09-27 and 10-11 this hit 6 of the 397 e2e attempts that reached chainsaw, which is 6 of the 23 assert failures in that window. Each one has `etcdserver: leader changed` in the apiserver logs within 0.2 s of the failure. Five of the six came after 10-09, and four of them were the computeplane ingress-nginx assert. The fix is kyverno/chainsaw#2761, which is still open and not in any release. Until it is released, the e2e sandbox image builds chainsaw v0.2.15 from source with that PR's commit applied unchanged, using the same Go minor and build flags as the v0.2.15 release binary. The build runs the tests of the patched packages. It also runs chainsaw against a fake apiserver that answers 500: a short run of 500s must not fail the assert, and an endless one must fail it at its timeout. Plain v0.2.15 fails both cases after the first GET. The Dockerfile header lists what to remove once a chainsaw release carries the fix. The README sentence added here has to go too, and so does the Renovate rule that keeps the builder on Go 1.26. The upstream PR also breaks one chainsaw test outside the packages it touches, `Test_runE` in `pkg/commands/assert`. Its expected string lacks the new `; operation terminated with error: context deadline exceeded` suffix. I found it with a full `go test ./...` on v0.2.15 with and without the patch. The image build tests only the patched packages, so that test does not run here. One trade-off: when the API stays broken, an assert or `error` step now runs to its timeout instead of failing within seconds, so a red run caused by a dead API gets longer. The proxmox e2e lane runs the chainsaw installed on its runner, not the sandbox image, so it does not get this fix. Known gaps, left for later: the new test covers only an assert on a named object, not one that lists objects or an `error` operation, and its skip guard does not notice if no test runs at all. The patch file has no `-whitespace` entry in `.gitattributes`, so `git diff --check` reports whitespace inside the upstream patch. A multi-platform build of the image would run the builder tests once per platform; the image is built for amd64 only. Closes #4720 ### Screenshots Not a UI change. ### Downstream repositories None of them builds or uses the e2e sandbox image; the change stays inside this repo's CI. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(e2e): a chainsaw assert in the e2e suites keeps polling through a transient API error such as an etcd leader change, instead of failing the suite long before its timeout. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * End-to-end sandbox assertions now retry after transient Kubernetes API errors instead of failing immediately. Persistent errors still cause assertions to fail when their timeout expires. * **Testing** * Added checks that verify assertions recover after temporary API errors and fail within the expected timeout when errors persist. * **Maintenance** * Updated sandbox image build and dependency update settings for the Chainsaw testing tool. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
a3470391e2 |
fix(linstor): mark volumes restored from a snapshot DRBD-initialized (#4866)
## What this PR does Restoring a PVC from a LINSTOR VolumeSnapshot can leave one replica with an empty volume that DRBD marks UpToDate. A pod that lands on that replica sees an empty filesystem. linstor-csi restores the snapshot on one node and autoplaces the other replicas. LINSTOR v1.33.3 copies `Linstor/Drbd/InitialUptoDateOn` from the source volume into the restored volume definition and leaves the definition without `DRBD_INITIALIZED`. When autoplace puts a replica on the node that name points to, that empty new volume wins the initial UpToDate race, gets a random current UUID and a fresh filesystem, and goes StandAlone against the restored replica. LINBIT fixed this in v1.34.0. This PR backports that commit, LINBIT/linstor-server@9d8cdf5, to our v1.33.3 build as a patch. The README entry says to drop the patch with the move to v1.34.0 or later. Tested on a three-node cluster with a thick ZFS pool and `placementCount: 3`. Before the change the reproduction failed every time. With the patch I ran 5 restores, 4 of them with the source volume's winner on a different node than the restore. I also restored a snapshot taken before the change, and a 10Gi volume. Every time all replicas came up connected and UpToDate, a pod on each node read the same checksum, and the backing zvols were byte-identical when read with `O_DIRECT`. The resync after a restore moves about the written data, around 40 s for 1Gi and 134 s for 7.5 GiB on 10Gi. CSI clone PVC-to-PVC was checked too and is not affected. There is no e2e test. Every e2e lane runs without DRBD (`COZY_LINSTOR_DRBD_ENABLED=false`, StorageClass `local`), so a suite for replicated restores would never run in CI. Volumes restored before this fix are not repaired by it. How to find and fix them is in [this comment](https://github.com/cozystack/cozystack/issues/4862#issuecomment-6106661441). Known limitation, shared by upstream v1.34 since that code did not change: the autoplaced replicas skip the initial sync and rely on the source volume's day0 UUID still being in the restored replica's history. If that ever fails, DRBD goes StandAlone with "unrelated data" instead of syncing, which is visible, not silent. I could not reproduce it. Closes #4862 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(linstor): a PVC restored from a VolumeSnapshot no longer gets a replica with an empty volume marked UpToDate. Volumes restored before this release may still have such a replica; https://github.com/cozystack/cozystack/issues/4862#issuecomment-6106661441 explains how to check and repair them. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Restored snapshots with diskful DRBD resources are now recognized as initialized by default, helping them resume without unnecessary metadata recreation. * When force-initial-sync is enabled, restored volumes continue through the existing metadata-recreation path. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
d060c1dbb0 |
fix(kafka): pass the container limits to the Kafka JVMs (#4865)
## What this PR does The Kafka JVMs we ship size themselves from host memory instead of their container limits, and get OOMKilled. The Strimzi 0.45.1-rc1 images carry JDK 17.0.15, which reads `/proc/cgroups` to decide whether it runs in a container. Kernels 6.12 and newer built without `CONFIG_MEMCG_V1` no longer list the memory controller there, so the JVM turns container support off. The JDK fix is JDK-8347811, backported to 17.0.20 and 21.0.10. In CI this shows up in two places. The cluster operator gets a 25 GiB max heap inside its 512Mi limit. In 107 of the last 150 e2e reports the kernel OOM-killed a Java process at about 500M, the limit the cluster operator shares with the topic and user operators. Brokers and controllers start with a 5 GiB heap inside a 1Gi limit, because Strimzi asks the same JDK for the container memory and takes half of what it reports. This PR passes the limits to every Kafka JVM explicitly: - the cluster operator gets `-XX:MaxRAM` and `-XX:ActiveProcessorCount` from its own limits through the downward API; - brokers, controllers, the topic and user operators and the drain-time Cruise Control get `JAVA_TOOL_OPTIONS` with `-XX:MaxRAM` set to their memory limit, plus `-XX:ActiveProcessorCount` where the chart sets a CPU limit. The chart sets one only for brokers and controllers. A LimitRange default on the entity operators or Cruise Control is invisible to the chart, and Strimzi template env takes no `resourceFieldRef`, so those JVMs still count host CPUs. Every java in a container reads `JAVA_TOOL_OPTIONS`, including the heap probe Strimzi runs before starting Kafka. So Strimzi's own heap rules and Kafka's G1 defaults apply to the real memory limit, and the chart does no heap arithmetic. `jvmOptions` with `-XX` would not work here: Strimzi turns those into `KAFKA_JVM_PERFORMANCE_OPTS`, and Kafka skips its G1 settings when that is set. Each JVM now prints a "Picked up JAVA_TOOL_OPTIONS" line on start. In a kind cluster on such a kernel the operator then runs Serial GC with a heap of about 100 MiB, and peaks at 249Mi while creating a Kafka, against 440Mi before. A broker with a 1Gi limit and one CPU gets the same `-Xmx518982205` that Strimzi computes when the JDK sees the limit, with G1 and one concurrent GC thread. Existing Kafka clusters roll once on upgrade, Talos ones included, because the pod environment changes. Their heap does not change on a kernel where the JDK already sees the limit. The chart has no values key for JVM options or heap, so there is no user setting this could override. The Kafka migration hook now applies with `--openapi-patch=false`. Its container runs under a tenant LimitRange default of 128Mi, and a plain `kubectl apply` of an existing pool downloads the cluster's whole OpenAPI document, about 300Mi with the platform's CRDs, so every upgrade in such a tenant failed on an OOM-killed hook. With the flag kubectl builds the same merge patch from the last-applied annotation and peaks at about 35Mi. A larger memory limit would count against the tenant quota, and a server-side apply would stop removing fields that `kubectl create` set on the `kafka` pool. Other Jobs without resources are tracked in #4868. The operator OOMKill is what triggered the e2e failure in #4855. The other half is a Strimzi race. After a restart, the PodSet controller drops the ADD event of a PodSet that has no pods yet and only picks it up at the 5-minute resync. That one is upstream and is described in the issue. The other JVMs we ship are tracked in #4864. Talos nodes are not affected, the Talos kernel keeps `CONFIG_MEMCG_V1=y`. Generic-variant clusters on newer distribution kernels are. CI is affected even though it runs Talos, because there the Talos nodes are containers on the runner's Ubuntu 6.17 kernel. Closes #4855 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(kafka): the Strimzi cluster operator, Kafka brokers and controllers, the entity operators and the drain-time Cruise Control no longer size their JVM heap from host memory on kernels built without cgroup v1 memory support, which got them OOMKilled. The Kafka migration hook no longer runs out of memory on upgrade in a tenant with resource quotas. Existing Kafka clusters roll once on upgrade. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Kafka components now size JVM memory and processor settings based on their configured resource limits, including during migrations. - The Kafka cluster operator now receives container resource limits for JVM sizing. - **Bug Fixes** - Kafka migration operations avoid OpenAPI patch lookups that could use excessive memory. - **Tests** - Added coverage for JVM sizing, resource limits, and migration behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
1f74ad4ee7 |
fix(multus): drop the CPU limit and make resources configurable (#4830)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does The multus daemon was too small to recover a node after a CNI outage, and its limits could not be raised from values. The 100m CPU limit and the 1Gi memory limit of `kube-multus` were hardcoded in the patched upstream template. On a node with about 600 dead sandboxes and 220 pods waiting, multus was throttled in every CFS period. Its calls took longer than kubelet's two-minute CRI deadline, so every call was retried and the queue only grew. With the CPU limit lifted it OOM-looped at 1Gi. With 2 CPU and 4Gi it cleared the backlog in about three minutes. Now the resources come from a new `resources` value. The default keeps the same requests (100m CPU, 100Mi memory), has no CPU limit, and raises the memory limit to 4Gi. The memory limit stays on purpose. Without it the pod cgroup has no `memory.max`, and multus becomes a candidate for the Talos userspace OOM handler again. The chart now refuses to render when an override removes the memory limit or the memory request. Without a request Kubernetes uses the limit as the request, so a node with less allocatable memory than the limit would be left without multus. The change is in `patches/customize-deployment.patch`, so `make update` keeps it. I applied the patch to upstream v4.3.0 and got the committed template, except the image references. The platform reverts a direct edit of the child `cozystack.multus` Package, so a running cluster sets the value through the new `networking.multus.resources` key of the platform Package. The key is passed on only when set, and a value that is not a map stops the render. The system-defaults LimitRange in the namespace sets memory only, so it does not bring a CPU limit back. Closes #4829 ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> Not a UI change. ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/pull/747 - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The diff adds a platform value, so the hand-written key table on the website needs it. The follow-up PR there is a draft until this one merges. ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(multus): the kube-multus container no longer has a CPU limit, and its memory limit is raised from 1Gi to 4Gi, so a node with hundreds of pods can recover after a CNI outage. Its resources can be overridden through the new `networking.multus.resources` platform value. An override that removes the memory limit or the memory request now fails the render. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added optional CNI plugin staging during Multus startup, configurable through chart values. Staging skips when the plugin directory is unavailable and reports individual copy failures while continuing. * Made Multus resource settings configurable, with defaults of 100m CPU and 100Mi memory requests and a 4Gi memory limit. Rendering requires memory requests and limits. * Updated the Multus deployment to use the `cozy-multus` namespace and DaemonSet name, and Cilium as the master CNI configuration. * **Bug Fixes** * Added a 128Mi memory limit to the CNI installation step. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
fa9f09e3d8 |
fix(vm-disk)!: refuse a golden-image clone CDI would leave unbound (#4796)
## What this PR does A VM disk created from a golden image into a StorageClass other than the image's one can hang forever: its temporary clone PVC stays unbound, and the user sees nothing. With the default `csi-clone` strategy, CDI checks the source and target classes for a common provisioner, a matching volume mode and room to expand. It never compares class names. So a target class backed by the same provisioner gets no host-assisted fallback, and the clone is left to the CSI driver. LINSTOR creates it next to the source data, so on a target class that does not allow remote volume access from every node (a node-local class, for example) the PVC may never bind. The vm-disk chart now refuses such a disk at render time and tells the user which StorageClass to set. The check follows CDI's own clone planner. The effective strategy comes from the CDI override, or else from the target StorageProfile. It has to be `csi-clone`, or `snapshot` when CDI gets a VolumeSnapshotClass of the provisioner. Once the snapshot path has such a class it runs the same checks as csi-clone, and LINSTOR ships one. Both classes have to share a driver (the golden's is read from its PV, as CDI does), the volume modes have to match, and the target class has to allow expansion. On top of that the chart refuses only a LINSTOR target whose `allowRemoteVolumeAccess`, read the way linstor-csi reads it, is not true; without the parameter that means a layer list with no DRBD or NVMe layer. A remote-accessible LINSTOR class and other drivers render. When the provisioner or the volume mode differs, CDI falls back to a host-assisted copy, which completes, so the disk renders as before. A target class without expansion is left alone too, see the follow-ups. The class check compares nothing when a lookup it needs returns nothing: under `helm template`, before the golden PVC exists, or without the CDI resource. An empty `storageClass` is compared as the class CDI would pick for a kubevirt DataVolume, which is the default virt class, otherwise the Kubernetes default class. Existing disks are exempt, because their DataVolume spec is immutable. A disk counts as existing when its DataVolume or its PVC exists, since CDI can garbage-collect a completed DataVolume. Under `snapshot`, a snapshot class that both StorageProfiles name but that does not exist is refused too, for any driver, because CDI then fails to plan the clone instead of falling back. This is a breaking change: a new disk in a LINSTOR class without remote access from every node rendered before and is refused now, and so is one whose StorageProfiles name a snapshot class that does not exist. The first case can include a clone that would happen to work: on a class with `Immediate` binding, which I have not verified, or when the node the clone is placed on already holds a replica of the golden image. The way out is the golden image's class or the `copy` clone strategy. How this covers the three fix directions in the issue: 1. Omitting `cloneStrategyOverride` when it is empty already landed on main in #4138: the template skips the field for an empty or null value, the values schema accepts the empty string, and the chart tests pin both. This PR does not touch it. 2. I kept `csi-clone` as the default on purpose. Same-class clones are the common case, and `csi-clone` is the fast path for them. Changing the default is a separate decision, tracked in #4850. 3. vm-disk refuses the mismatch where the clone can hang, as described above. That is narrower than the issue proposed, which was to refuse every mismatch: every case CDI falls back on completes, and a clone into a remote-accessible LINSTOR class should complete too. The last part comes from reading the CDI and linstor-csi sources, I have not run it on a cluster. Follow-ups this PR does not cover: - When several classes claim the default, CDI picks the newest one. The chart does not reproduce that tie-break and compares nothing in that case. - The chart does not compare sizes. A target class without volume expansion is never refused for remote access, even when the disk is no larger than the golden image and CDI would attempt the clone. - Under `snapshot` on a cluster that does not serve the snapshot API, StorageProfiles naming a snapshot class still make CDI fail, and the chart does not look that class up. - A disk cloned from another disk (`source.disk`) into another class of the same provisioner can hang the same way, and nothing refuses it yet. - Whether `csi-clone` should remain the platform default now that it can be turned off: #4850. Closes #2908 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: No values, schema or ApplicationDefinition change. Re-check notes for the next CDI and linstor-csi bumps sit next to the version pins in the kubevirt-cdi-operator and linstor Makefiles. The new note in the vm-disk README reaches the website through the regular reference-page regeneration. ### Release note ```release-note fix(vm-disk)!: a new disk from a golden image is now refused with a clear error when CDI would CSI-clone it into another LINSTOR StorageClass that does not allow remote volume access from every node. Before, such a disk could stay pending forever with no error. A clone that would have landed on a node already holding a replica of the golden image is refused as well; use the golden image's StorageClass or the copy clone strategy. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Prevented image-backed disks from being rendered when CDI’s selected clone strategy cannot safely clone across differing storage classes, including applicable LINSTOR remote-access and snapshot-class checks. * Existing DataVolumes and same-namespace PVCs are not blocked by these checks. Disk creation can proceed when required information is unavailable or the selected strategy supports the operation. * **Documentation** * Clarified when image-backed disks may be refused and recommended using the golden image’s storage class or the copy strategy as alternatives. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
938abdc4d2 |
fix(platform)!: refuse a networking variant change on a live cluster (#4790)
## What this PR does Changing the platform variant on a running cluster can delete its CNI, and nothing stopped that before. A new platform variant (`spec.variant` of the `cozystack.cozystack-platform` Package) can give the `cozystack.networking` Package a different variant, and the operator then deletes every HelmRelease of that Package the new variant does not install. Moving from `isp-full` to `isp-hosted` deletes Cilium and Kube-OVN, so every pod loses networking. Moving from `isp-full` to `isp-slim` deletes Kube-OVN. With this PR the platform render looks up the live `cozystack.networking` Package. If its variant differs from the one about to be rendered, the render fails, and the error names the value that lets it through: `networking.acknowledgeVariantChange: "<new variant>"`. I made the acknowledgement a variant name instead of a boolean like `publishing.proxyProtocolAcknowledgeUnclean`. A leftover `true` would let every later switch through unchecked, while a variant name covers only the switch it names. The guard does nothing when there is no live Package to compare against: `helm template`, a client-side dry-run, or a first install. It is also skipped when `cozystack.networking` is listed in `bundles.disabledPackages`, because the platform does not render the Package then. It lives in the shared Package helper and not in `bundles/system.yaml`, so it covers both places that emit the networking Package, including the `noop` call in the `isp-hosted` branch. The other Packages whose variant follows the bundle, `cozystack-scheduler` and `tenant-application`, keep the same release names in every variant, so changing their variant deletes nothing and they need no guard. The new helm-unittest cases cover the refused switches, an unchanged variant, a matching and a mismatching acknowledgement, a bare boolean, `disabledPackages`, an offline render, and a live cluster with no networking Package yet. A cluster where someone set the Helm-owned networking Package to `cilium-kilo` by hand now fails its platform upgrade, and the error points it at `bundles.disabledPackages`. Acknowledging does not help there, because the render writes the bundle's variant over `cilium-kilo`. helm-controller v1.5.0 installs a release with server-side apply by default, which takes over every rendered field. On a release it installed, the upgrade replaces the variant and the operator deletes Kilo. A release first installed with Helm 3 keeps the apply method recorded in its history, so there the outcome depends on that history. Known gap, left for a follow-up: the guard fires on any variant change, including Talos to generic (`kubeovn-cilium` to `kubeovn-cilium-generic`). Those pairs share release names, so nothing is deleted and only values change. Refusing is safe, and the error does not claim a deletion for such a switch. Closes #4596 ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories This PR adds a key to `packages/core/platform/values.yaml`, and the website keeps a hand-written table of those keys in `operations/configuration/platform-package.md`. - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/issues/746 - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(platform): the platform render now refuses to change the variant of a live cozystack.networking Package, because the operator would then delete the CNI HelmReleases the new variant does not install (isp-full to isp-hosted removes Cilium and Kube-OVN). To go ahead with such a change, set networking.acknowledgeVariantChange to the new networking variant named in the error. A cluster whose cozystack.networking variant was set by hand (for example cilium-kilo) needs cozystack.networking in bundles.disabledPackages before the upgrade, otherwise the platform render refuses. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Changing the networking variant on a running cluster now requires explicit confirmation naming the requested variant. A change without a matching confirmation is blocked. * The confirmation applies only to the specified switch and must be updated for a later variant change. If the running package has no recorded variant, it is treated as using the default variant. * New installations without an existing networking package are unaffected, and networking can still be disabled through its package setting. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
953b965e66 |
fix(kubernetes): keep NFS publish from hanging on a dead server (#4791)
## What this PR does When the kubevirt-csi node plugin publishes an NFS volume, it first mounts the export root into an `nfs-init-*` temp dir to create `/data` and move old files into it. That temp mount was hard. If the NFS server stopped answering at that point, the publish call never returned. kubelet gives up after 2m and retries, each retry blocked the same way, and goroutines and temp dirs piled up until the server came back or the plugin restarted. The fix does three things: 1. The temp root is now mounted `soft` with `retry=0`. It only does metadata work that a retried publish redoes, so an error there is harmless, and a stuck call now ends on its own once the connection to the server is declared dead. The mount the pod gets stays hard, because a soft mount can report a failed write that the server did apply. 2. A publish to a target path that already has one running now returns `Aborted` instead of starting another. This is what stops the pile-up: at most one call per target path is stuck at a time. 3. Moving old files into `/data` now checks the destination with `lstat` and fails the publish on any error other than a missing file. With the soft mount, the check can fail with EIO on a file that `/data` already holds, and a rename issued after the server comes back would replace that file. `lstat` keeps a user's symlink in `/data` that loops or dangles from failing every publish. Such a symlink counts as present and is left alone, while before this change the rename replaced a dangling one. I kept the default retry timeouts on purpose, although the issue suggested lowering `timeo` and `retrans`. The Linux client shares one connection per server for all of that server's mounts on the node, and the first mount sets the connection's TCP timeouts. Lowering them on the temp mount would have changed the timeouts of the pods' hard mounts too. Over NFSv4, a soft RPC fails only after the connection is declared dead, so a stuck call takes minutes to end, longer than kubelet's 2m deadline. The guard keeps kubelet's retries from adding new calls during that time. The guard works per target path, not per volume. Two pods on one node that use the same RWX volume still publish in parallel, and each pod can have at most one stuck call. The unit tests pin the mount options on both mounts, the guard behaviour including its release after a failed publish, and the destination check: a failed check stops the move, and a symlink in `/data` stays in place. They use the fake mounter. I did not test against a real NFS server that went away. Not covered here, and tracked in #4814: - If the server's TCP connection stays up while the NFS service hangs, a soft NFSv4 RPC does not time out. The guard still limits it to one stuck call per target. - `Mount` from mount-utils runs `mount` with no deadline. This predates this change, and the guard bounds it the same way. - If the target is already hard-mounted from the dead server, the check of the target at the start of a publish blocks for good. The guard keeps this to one call per target but does not end it. Unpublish is not coordinated with a publish that is still running either. Both predate this change. - A follow-up comment on the issue reported two leftovers of the bounded unmount: an orphaned `umount.nfs` after a timeout, and a possible `stat` before `umount2`. Both may end sooner now that the temp mount is soft, but I have not verified either. Refs #4581 ### Screenshots Not a UI change. ### Downstream repositories The change is limited to the kubevirt-csi node plugin's Go code. None of the triggers in the contributing guide's map apply: no package added or renamed, no platform values, variants, release assets, ApplicationDefinition semantics or developer tooling touched. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(kubernetes): an NFS volume publish in tenant Kubernetes clusters no longer hangs forever when the NFS server goes away during the publish, and kubelet retries no longer pile up while a publish is stuck. The plugin's temporary mount of the export root is now soft, and a second publish to the same target path is refused while the first is still running. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Concurrent NFS publish requests for the same target now return an `Aborted` error. Different targets remain available, and a target can be used again after a request finishes, including after a failed attempt. * Migration now stops if checking a destination encounters an unexpected error. Existing destinations, including symlinks, are left in place. * Temporary-directory creation and mount errors now identify the affected volume. * **Updates** * Temporary NFS root mounts now use soft behavior with retries disabled. The final data mount remains hard. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
b86513800f |
docs(governance): welcome new contributors and link LF sanctions guidance (#4845)
## What - Adds a "Welcoming new contributors" section to `GOVERNANCE.md`: response expectations, help with DCO, a safety read of the diff before every CI approval for external contributors, and how to review and follow up. - Adds one line to the maintainer expectations pointing to the Linux Foundation guidance on OFAC sanctions and open source, with questions raised privately among maintainers. ## Why Maintainers had no written guidance for first contributions. For sanctions, the LF guidance applies to all LF projects, so the document links to it instead of restating it. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Expanded maintainer guidance for welcoming new contributors, including responding within two working days, helping with contribution and DCO requirements, and suggesting a next step. * Clarified expectations for consistent reviews, explaining decisions, and checking external contributors’ CI changes for workflow, secret, or release-tool risks before approval. * Added guidance on Linux Foundation advice concerning OFAC sanctions and raising related questions privately with other maintainers. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |