## What this PR does
Implements the immutable-tag + rc-promotion flow from #2677. A **stable
release becomes a renamed release-candidate**: the bytes shipped as
`vX.Y.Z` are bit-for-bit the bytes built and e2e-tested as
`vX.Y.Z-rc.N`. No tag is ever force-moved, and stable is never rebuilt —
it is *promoted* by retagging the rc's existing images.
Layered onto the build matrix from #2937/#2983 (this PR is **stacked on
`refactor/build-matrix-2937`** and must merge after it).
## The five force-retag sites, removed
| Site | Before | After |
|---|---|---|
| `tags.yaml` api/apps/v1alpha1 tag | `git tag -f` / `push -f` |
write-once (create-if-absent, fail if it would move) |
| `tags.yaml` release-X.Y.Z branch | `git branch -f` / `push -f` |
compare-before-force (no-op if unchanged; staging branch only) |
| `pull-requests-release.yaml` stable tag | `git tag -f` / `push -f` |
write-once at the PR merge commit (force impossible by construction) |
| `pull-requests-release.yaml` maintenance branch | `updateRef
force:true` | fast-forward-only |
| `auto-release.yaml` patch tags | delete-recreate (cron) | **workflow
deleted** — stable only via explicit promote |
## Version decoupling (the enabler)
The operator baked its version into the image at build time, so an rc
image self-reported the rc string — blocking retag-promotion. It now
reads `COZYSTACK_VERSION` from the environment (threaded via
`cozystackOperator.platformVersion` → Deployment env, stamped by `make
manifests`), falling back to the build-time value when unset. The same
image bits can report any release name. The only runtime reader is the
telemetry metric `cozy_cluster_info{cozystack_version=...}`.
## Promotion flow
`promote-rc.yaml` (`workflow_dispatch`, `rc_tag=vX.Y.Z-rc.N`):
1. Validate the rc release exists and the stable tag does not.
2. `hack/promote-retag.sh` reads the rc's digest-pinned image refs from
`packages/*/*/values.yaml` and `skopeo copy`s each — **by digest** — to
`:vX.Y.Z` and `:latest`, verifying with `skopeo inspect`.
3. Rewrite the cosmetic `-rc.N` substring in vendored tags to the stable
version (the `@sha256` wins regardless), restamp the version-stamped
assets (operator manifests, cozypkg, openapi; the heavy Talos assets are
copied verbatim from the rc draft), open a `release-X.Y.Z` PR.
4. Merging the PR reuses the existing release-PR e2e and
`pull-requests-release.yaml` finalize to cut the **write-once** stable
tag at the merge commit and publish the release. Squash is disallowed
(the tag needs a real merge commit).
`nightly.yaml` cuts write-once `*-nightly.<date>` tags (gated by
`NIGHTLY_ENABLED`); `retention.yaml` keeps the newest 14 per line
(dry-run by default).
## Validation
**Local end-to-end — the core claim is proven, not asserted.** The whole
design rests on "copy-by-digest to a new tag preserves the digest, so
stable == the e2e-tested rc, bit-for-bit." This was exercised against a
real registry (`ttl.sh`) with real `skopeo`:
- Pushed a multi-arch image under a `:v1.4.0-rc.2` tag, built a temp
tree exercising all three `values.yaml` digest shapes (single `image:`
string, split `repository`/`tag`/`digest` map, and the
`platformSourceRef` OCI artifact), then ran the actual
`hack/promote-retag.sh v1.4.0`.
- **Result: `:v1.4.0-rc.2`, `:v1.4.0`, and `:latest` all resolve to the
identical digest** (`sha256:fd8d9aa6…`), across all 17 platform
manifests. The script's own `skopeo inspect` post-check passed, and an
independent re-inspection confirmed it.
Local validation caught (and this PR fixes) two bugs that only surface
against a real registry — never in static lint:
1. `skopeo copy --multi-arch all --all` — mutually exclusive flags,
fatal error; every retag would have failed on CI. Now `--multi-arch
all`.
2. The same digest was retagged twice when an image appeared in two
value shapes; the ref set is now deduped on the canonical `repo@digest`.
**Other local checks:** `go test ./pkg/version` + `go build`/`vet`;
`helm template` renders `COZYSTACK_VERSION` in all 3 operator variants
(omitted when unset); `make manifests` stamps the version into the
install assets; `shellcheck` clean on `promote-retag.sh`; `actionlint`
clean and `act -l` resolves the job graph on all workflows; the rc-tag
parser and nightly version-math unit-tested (accept/reject +
minor-vs-patch bumps).
**Still CI-only (cannot be exercised offline):** the full workflow
runtime — github-script API calls, OCIR auth, the rc-release /
staging-branch preconditions, the nightly `base_ref` push mechanics, and
self-hosted runners. The *logic* inside the steps is unit-tested; the
*orchestration* is not. Treat it as unproven until the workflows run.
## Out of scope / required follow-up (needs repo admin — not doable from
a PR)
- Tag-protection rules: `v*` and `api/apps/v1alpha1/*` create-only by
the CI app, no delete/update; "limit branches/tags updated in a single
push" = 1.
- Enable `NIGHTLY_ENABLED` (and `RETENTION_APPLY` when ready) repo
variables.
- Registry-side pruning of `*-nightly.*` image tags (no OCIR delete
parity yet — tracked TODO in `retention.yaml`).
## Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added nightly publishing (mirror-by-digest, disk build, e2e
validation) and nightly retention pruning.
* Added RC-to-stable promotion via digest retagging (no rebuild).
* Operator/installer and console now expose version/platform metadata,
including runtime override via `COZYSTACK_VERSION`.
* **Bug Fixes**
* Enforced write-once tag behavior and fast-forward-only maintenance
updates across release workflows.
* Made nightly mirroring/selection and retention pruning more selective
and safer.
* **Tests**
* Added Bats coverage for nightly mirroring and RC retagging, plus new
Helm/unit checks.
* **Documentation**
* Updated the release model to center RC promotion and tag immutability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Bring the security documentation in line with controls that are already
in place, and add the two coordinated-disclosure items flagged during the
incubation due-diligence review.
- self-assessment: DCO is enforced by an automated check; private intake
is GitHub Private Vulnerability Reporting; OpenSSF passing badge achieved
2026-04-01 (working toward silver); GHSA/PVR framing for advisory history;
incident-response now summarized publicly.
- SECURITY.md: route reports to Private Vulnerability Reporting with a
MAINTAINERS.md fallback; add a 90-day coordinated-disclosure window and a
CVSS-based remediation-target table.
- CODEOWNERS: document that it encodes the area-scoped Reviewer role, which
is intentionally distinct from the MAINTAINERS.md governance roster, and
that security reporting is not routed through this file.
- Add docs/security/incident-response.md as the public incident runbook.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: tym83 <6355522@gmail.com>
Add the CNCF security self-assessment required for the Incubation due
diligence review, alongside the existing design-level threat model. It
documents the trust boundaries, actors/actions, security functions,
secure development practices, and known gaps.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: tym83 <6355522@gmail.com>
Document the platform's actors, boundary-crossing actions, and security
goals/non-goals, organized by trust boundary (T0-T4) per the CNCF
TAG-Security Actors/Actions model. Records the primary trust-boundary
invariant (tenant RBAC grants no HelmRelease/core writes; the aggregated
API server fixes the chart reference server-side) and the two admission
enforcement points (core kube-apiserver vs cozystack-api). Every claim
cites a source file so the model can be re-verified as the code evolves.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Promotion mutated the registry at dispatch time — retagging ~37 images to
vX.Y.Z (+:latest), publishing the stable cozy-installer chart — before the
promote PR's e2e ran and before any merge. An abandoned promotion then left
stable-named bytes for a version that never shipped, and the (correct) image
write-once check wedged any later re-promotion to the same version.
Move every irreversible registry side effect into pull-requests-release.yaml
finalize, which runs only after the promote PR merged (= after its e2e passed),
making promotion atomic. promote-rc.yaml now only stages the release-X.Y.Z
branch, drafts the stable release, and opens the PR; an abandoned promotion
leaves nothing but an overwritable draft + branch.
Folded into the same restructure:
- Cut the api/apps/v1alpha1/vX.Y.Z Go-module tag in finalize, at the stable
tag's commit. It was gated in tags.yaml on `release_exists==false &&
is_stable`, unsatisfiable under promotion (the draft always pre-exists), so
promoted stables never got their submodule tag — a regression for Go
consumers. Finalize is where the stable tag is born, so it cuts the matching
submodule tag write-once (stable only).
- Gate :latest by max-semver. promote-retag.sh now moves :latest only when
MOVE_LATEST=1 (default off); finalize passes it the same make_latest decision
the release gets, so a patch on an older line moves neither the release's
`latest` nor the images' :latest. Added MOVE_LATEST bats coverage.
- Stamp platformVersion into the published stable chart's default values
(uncommitted, at package time) so `helm --install --version X.Y.Z` reports the
stable version in telemetry instead of the rc version baked into the operator
image. Not committed to the branch — that would leak platformVersion=vX.Y.Z
onto main and mis-stamp the next rc.
- promote-rc: serialize all promotions (static concurrency group, not per-rc);
label the PR full-e2e so the full suite runs; scope the rc->stable sed to the
depth-2 values.yaml glob plus the kubernetes .tag files (the build's own write
set); tolerate a leftover draft so re-promotion is not wedged.
- finalize: assert the PR author is cozystack-ci[bot], and run on an ephemeral
oracle-vm runner (needs skopeo/helm).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Prior wording overstated Phase 1: the rendered
AuthenticationConfiguration carries no claimValidationRules, so the
chart itself does not gate on email_verified. Rewrite the failure-mode
bullet to describe the layered guarantees operators actually rely on:
1. provisioning KeycloakRealmUsers with emailVerified: true,
2. Keycloak's realm-level duplicateEmails: false as an
impersonation-through-second-account guard,
3. the k8s upstream side-effect that rejects tokens explicitly
carrying email_verified: false.
Also flag CEL claimValidationRules as a reasonable follow-up hardening
path, but explicitly out of scope for Phase 1 per the design doc.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
The tenant kube-apiserver auto-applies
`claims.?email_verified.orValue(true) == true` when username.claim is
email, so a Keycloak user with `emailVerified: false` fails login with
`oidc: email not verified` even though token issuance succeeds.
Document the requirement in the operator guide alongside the other
System-mode failure modes.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Adversarial review flag: the previous rewording implied the cozy realm
has `email` in its client-scope defaults, which is misleading — the
per-cluster client my chart creates lists only `groups` and the
per-cluster audience scope as defaults. The `email` claim reaches the
token because the chart-generated `kubectl oidc-login` block passes
`--oidc-extra-scope=email` explicitly, not because of realm-side
default-scope hygiene.
State the actual mechanic: `email` is a built-in OIDC scope available
in every conformant issuer, requested by the client at auth time via
the kubeconfig's exec block. No realm-side configuration required.
For `CustomConfig`, note that the operator should verify their BYO
issuer emits `email` when the client requests the `email` scope
(every conformant provider does).
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Two follow-ups from the final adversarial review:
- The chart-generated OIDC kubeconfig exec block requested only
`--oidc-extra-scope=groups`. On a Keycloak realm where `email` is
not a default client scope, kubectl-oidc-login would fetch a token
that lacks the `email` claim — apiserver would then fail to
identify the user, causing a silent auth failure. Add
`--oidc-extra-scope=email` explicitly so the flow does not depend
on realm-level default-scope hygiene.
- `docs/oidc-tenant.md` still said the `User:` subject "must match
the `preferred_username` claim emitted by the issuer" — obsolete
after the claim switch. Rewrite to say `email`, and add a note
that the chart requests the `email` scope explicitly.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Self-review fixes for two factual errors:
- The "How a user logs in" example named the Secret
`prod-oidc-kubeconfig`, but the cozystack-api prefixes the
cluster name with `kubernetes-` when materialising the
`HelmRelease`, so the actual Secret is
`kubernetes-prod-oidc-kubeconfig`. The NOTES.txt template was
already correct; only the doc example was off.
- The CRB hash formula was described as `<release>:<email>` but
the code uses `<release>-<email>` as the SHA input.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Add the operator-facing guide for the new selector — what the three
modes mean, what the chart provisions, how a user logs in, what the
failure modes are, and what is intentionally out of scope (per-tenant
realms, federation, cross-cluster SSO). The architectural rationale
points back to cozystack/community#24.
Update the chart NOTES.txt to print the OIDC kubeconfig retrieval
snippet, the per-cluster client/issuer values, and the listed
`users[]` roster — only when OIDC is on, otherwise the previous
admin-kubeconfig hint is unchanged.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
nightly.yaml publishes each build under 0.0.0-nightly.<date>.<run-id>.<attempt>
(the run-id + attempt suffix makes every nightly tag immutable), but the docs
still showed the old date-only 0.0.0-nightly.<YYYYMMDD> form in the mirror-stage
description and both the helm-install and oras-pull pin examples. Update them to
the full form; the floating `nightly` example is unchanged.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
"What CI does on a tag push" only described one of the four triggers;
promote-rc.yaml uses manual dispatch and pull-requests-release.yaml fires
on PR merge. Rename to "What CI does during the release process" so
maintainers find the right entry point regardless of which workflow they
are debugging.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from coderabbitai on docs/release.md: the Regular and
Patch sections still told maintainers to push vX.Y.Z to start the build, while
the rest of the doc said the stable tag is created write-once at the merge
commit and stable comes only from rc promotion (with a 'can also push directly'
escape hatch). Rewrite both sections to the single promotion model — push an rc,
promote it via promote-rc.yaml, merge the promote PR, finalize creates the
write-once tag — drop the direct-push escape hatch, and add promote-rc.yaml to
the workflow list.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Replace the nightly-rc-tag description with the new model: a nightly is
an installable copy of main on GHCR (mirror OCIR->GHCR by digest +
re-published packages artifact, cozy-installer chart and nocloud disk),
validated by the full e2e suite, with no GitHub release and no Go-module
tag. Document the helm install path and the dated/floating tag scheme,
and update the Tag-immutability section and workflow link list.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The release doc still described the removed force-retag flow and auto-release.yaml
cron. Update it to the model this PR ships: stable tags are created write-once at
the PR merge commit (never force-moved), the maintenance branch is
fast-forward-only, stable patches come only from rc promotion (nightly-rc.yaml +
promote-rc.yaml + retention.yaml), and the api/apps/v1alpha1 Go-module tag is cut
only for stable. Replaces the "Automated patch tag" and "Force-retagging"
sections (the latter becomes "Tag immutability") and refreshes the See-also list.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Two post-vendor cleanups from the #2963 review:
- **Pin the console image builder to pnpm 11.1.2.** The `Containerfile`
ran `corepack prepare pnpm@latest`, so the container build could drift
to a newer pnpm major than the `pnpm@11.1.2` pinned in `package.json`
and used by CI. `--frozen-lockfile` guards the lockfile, but pinning the
builder makes the image build reproducible and consistent with CI.
- **Repoint a stale doc link.** `docs/storage-immutability.md`
referenced the now-vendored `cozystack/cozystack-ui#6`; it now points at
the in-tree console path.
## Verification
- No source change; `pnpm typecheck` + `pnpm test` unaffected.
- Builder pnpm version now matches `package.json`'s `packageManager`
field.
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Clarified the storage immutability documentation by updating the UI
reference to an in-repo path.
* **Chores**
* Standardized the build environment to use a specific pnpm version,
helping keep installs and builds consistent.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The console Containerfile ran `corepack prepare pnpm@latest`, so the image build could drift to a newer pnpm major than the `pnpm@11.1.2` pinned in package.json and used by CI. `--frozen-lockfile` guards the lockfile, but pinning the builder makes the image build reproducible and consistent with CI.
Also repoints docs/storage-immutability.md off the now-vendored cozystack-ui#6 link to the in-tree console path (follow-up from #2963).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
- scope create-github-app-token to the current repo with read-only
contents permission (was org-wide, inheriting the app's blanket
installation permissions); both token steps only read releases/assets
- fail-fast the Talos image upload when the disk is missing
(if-no-files-found: error) so drift fails in build-talos, not later
as a confusing e2e download error
- download the flux install script to a tempfile before executing it,
so a failed curl is not masked by the piped `sudo bash` exiting 0
- fix the regular-PR artifact-source comment (Talos disk comes from
build-talos, pr.patch from finalize) and the TIA scope wording in
docs/agents/e2e-testing.md
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Reconcile the parallel-matrix PR build (#2983) with main, which since this branch
diverged landed a competing single-VM build restructure (#2939) and
test-impact-analysis E2E (#2559). Per maintainer decision, #2983's matrix build
supersedes #2939:
- .github/workflows/pull-requests.yaml: keep #2983's
plan/checks/build(matrix)/build-talos/finalize jobs; keep main's newer
resolve_assets/e2e (carrying #2559 test-impact E2E). The build->e2e artifact
interface (pr-patch + talos-image) is identical on both sides, so e2e depends
on finalize/build-talos with no input rewiring. main's top-level permissions
and SHA-pinned action refs applied across all jobs.
- All other files merged cleanly (only pull-requests.yaml conflicted).
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Removes the `Release E2E` workflow
(`.github/workflows/release-e2e.yaml`), which ran the full bats suite on
every release tag.
**Why it's redundant.** `release-e2e.yaml` was a non-blocking alarm —
its own header states *"a failure here is an alarm, not a release
blocker."* The coverage it provided is already delivered by the
release-cut PR: the "Prepare release" commit bakes image digests into
`packages/core/` (installer, platform, testing) and other packages, and
`select-e2e.sh`'s full-suite pattern escalates any `packages/core/`
change to the **full** suite. So the release PR already runs the whole
suite against the commit the tag will point at.
**Why now.** The workflow has never been green since it was added
(2026-05-02) — every tag run (`v1.5.0-rc.2`, `-rc.3`, `v1.5.0`) failed
at the build step. Even after #2973 fixed the registry login earlier
today, the next run still failed at *Build Talos image*. A
perpetually-red, non-gating duplicate is pure noise.
**Changes**
- Delete `.github/workflows/release-e2e.yaml`.
- Drop `release-e2e` from the full-suite pattern in
`hack/select-e2e.sh`.
- Repoint the workflow-trigger unit test in `hack/select-e2e_test.bats`
to `pull-requests.yaml` (the remaining E2E workflow), preserving
"workflow edit → full suite" coverage. All 11 `select-e2e_test.bats`
cases pass.
- Update `docs/agents/e2e-testing.md` to describe the release-cut PR as
where the full suite runs.
Refs #2397.
### Release note
```release-note
ci(tests): remove the redundant Release E2E workflow; the full E2E suite already runs on the release-cut PR
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Removed automated release end-to-end testing workflow
* Updated E2E test selection logic to reflect testing infrastructure
changes
* **Documentation**
* Updated E2E testing procedures documentation to reflect current
testing approach
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The "Pin the e2e management cluster Kubernetes version" item is no longer
in-flight — it is merged and live in hack/e2e-prepare-cluster.bats
(--kubernetes-version v1.33.12). Remove it from the "not yet the merged
standard" section; its rationale lives in the bats comment.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Merge the AI-generated v1.5.0 changelog with a fuller pass:
- Add an "Upgrade Notes and Required Actions" section (K8s 1.33+, removal
of upgrade.force, GPU permittedHostDevices ownership, MetalLB FRR-K8s /
HTTPS metrics, automatic TLS on externally-published services).
- Expand "Platform Components" with per-component upstream highlights and
CVE references (Flux v2.8 CVE-2026-23990, MetalLB v0.16 CVE-2025-22874),
plus the new frr-k8s subchart and kuberture package.
- Scope the storageClass-immutability entry (#2639) to UI-only enforcement,
with apiserver enforcement tracked in #2657.
- Broaden Development/Testing/CI coverage and add the v1.4.x website docs
backports (#565, #566, #567, #580).
- Correct the loop-device LVM-filter attribution to talm#215 (the prior
ansible-cozystack#51 reference was a dependency bump, not this change).
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The "What TIA does and does not do" section described the old single
Build job and the `detect-changes` paths-filter. Update it for the new
`plan` job (docs gate + build matrix), note that image builds are now
diff-scoped by hack/build-matrix.sh (distinct from TIA's app-loop trim),
and point the "widen the skip" advice at the plan docs-gate.
Refs #2937
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The Release E2E workflow (`release-e2e.yaml`) ran the full bats suite on
every release tag as a non-blocking alarm — its own header notes "a
failure here is an alarm, not a release blocker." It duplicates coverage
the release-cut PR already provides: the "Prepare release" commit bakes
image digests into `packages/core/` (and other packages), which TIA
escalates to the full suite, so the release PR exercises the whole suite
against the commit the tag will point at.
It has also never been green since it was added (2026-05-02): every tag
run failed at the build step. Even after #2973 fixed the registry login
today, the next run still failed (Build Talos image). Removing a
perpetually-red, non-gating duplicate removes noise without losing
coverage.
Removes the workflow, drops `release-e2e` from `select-e2e.sh`'s
full-suite pattern, repoints its unit test to `pull-requests.yaml`, and
updates the e2e-testing doc.
Refs #2397
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
The **Auto Patch Release** cron (`.github/workflows/auto-release.yaml`)
walked
*every* `release-X.Y` branch — 21 of them, back to `release-0.9` — and
cut a
`vX.Y.(Z+1)` tag whenever a branch had commits after its latest
published
release. This fires patch releases on long-EOL lines (e.g. `v1.3.6`),
whose
release PRs then sit blocked on unrelated CI failures and accumulate
stray
draft releases, tags and changelog PRs.
This restricts the cron to the **2 newest minor lines**
(`SUPPORTED_LINE_COUNT`).
The window is derived from the live branch list, so it slides
automatically as
new `release-X.Y` branches are cut — once `release-1.5` exists the
window
becomes `{1.5, 1.4}` and the trailing line retires with no further
edits.
Currently active: `release-1.4`, `release-1.3`. Skipped: 19 EOL lines.
### Release note
```release-note
ci(release): the automated patch-release job now only cuts releases for the two newest release branches; older EOL lines no longer receive automatic patch tags.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated the automated release workflow to focus patch automation on
only the two newest minor `release-X.Y` maintenance lines.
* Enhanced workflow logging to report the selected branches and any
older/EOL branches skipped.
* **Documentation**
* Refined release documentation to explain the sliding-window selection
for automated patch tagging, including how next patch versions are
determined and when tags are created.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Promotes universal project guidance that previously lived only in a
local
(gitignored) `CLAUDE.md` into the committed agent docs, so every
contributor and
every tool (Codex/Cursor/etc., not just Claude) gets it from a fresh
clone.
- `overview.md`: Build & Development Commands (root + per-package make
targets,
build env vars), Values schema generation, a Testing section, Domain
references
- `contributing.md`: a "Verify Before Considering Done" matrix + Review
Expectations
- Dedup: `overview.md`'s Git Commits block now points to
`contributing.md`
> Note: `overview.md` links to `docs/agents/e2e-testing.md` added in
#2932 — merge
> that one first or together.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Expanded contribution guidance with a “Verify Before Considering Done”
pre-submission checklist and clearer expectations for including
behavioral regression tests in bug-fix PRs.
* Broadened build/development documentation with a “Build and
Development Commands” guide, including required environment variables,
values-schema generation, and test entry points.
* Updated contribution workflow notes by consolidating commit-convention
guidance and adding pointers to task-specific design references.
* Added a Claude entry-point document that directs to the main agents
guide and explains local override placement.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Adds `docs/agents/e2e-testing.md` — a routed agent doc capturing the
e2e/CI
stabilization conventions built up over the last few months:
- No retries on deterministic CI steps (only pure-infra `Prepare
environment` retries)
- Event-driven backstops instead of fixed `sleep`; no `EXIT`/`RETURN`
traps
- Fail-fast HelmRelease-readiness gating + the `status.history`
remediation guard
- First-class failure diagnostics and a reviewer checklist
- Test-Impact Analysis (TIA) and its real scope: it trims only the
per-app loop,
runs after install, and most PRs legitimately escalate to the full suite
Wired into `AGENTS.md` so every agent reads it before e2e/CI work.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Introduced comprehensive E2E testing conventions document with
standards for writing, modifying, and reviewing end-to-end tests and CI
processes.
* Documented best practices for test reliability, including guidelines
on retry behavior, event-driven test validation, and cleanup discipline.
* Updated process documentation to reference new E2E testing
conventions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Address review feedback from lexfrei on docs/release.md: the Patch Releases section described only the manual process, so add an "Automated Patch Releases" subsection covering the nightly cron, the new-commits trigger, the 2-newest-lines support window, and the manual tag-push escape hatch for EOL lines.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from coderabbitai on docs/agents/overview.md:145:
internal/controller/dashboard/README.md does not exist (no such directory),
and there is no dashboard reference doc to point to. Remove the dead bullet;
the backups DESIGN.md reference is valid and kept.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from gemini-code-assist on docs/agents/contributing.md:145:
reword the awkward "on the template" phrasing to "aligned with the template".
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from gemini-code-assist on docs/agents/overview.md:103:
the Helm-only target is make helm-unit-tests; make unit-tests runs the full
unit suite. Point the bullet at the precise target and note the difference.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from gemini-code-assist on docs/agents/overview.md:72:
make unit-tests runs the full suite (Helm, BATS, Go, presets, readiness),
not only Helm unit tests.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Promote the universal project guidance that previously lived only in a
local CLAUDE.md into the committed agent docs, so every contributor and
every tool (Codex/Cursor/etc., not just Claude) gets it from a fresh
clone.
overview.md gains Build and Development Commands (root + per-package
make targets, build env vars), Values schema generation, a Testing
section, and Domain references. contributing.md gains a "Verify Before
Considering Done" matrix and Review Expectations. Dedup: overview's Git
Commits block now points to contributing.md instead of restating it.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Capture the e2e stabilization conventions established over the last few
months as a routed agent doc: no retries on deterministic CI steps,
event-driven backstops over fixed sleeps, no EXIT/RETURN traps,
fail-fast HelmRelease-readiness gating, the status.history remediation
guard, first-class failure diagnostics, and a reviewer checklist. Also
documents Test-Impact Analysis (TIA) and its real scope — it trims only
the per-app loop, runs after install, and most PRs escalate to the full
suite. Wire it into AGENTS.md so every agent reads it before e2e/CI work.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The vfio-manager DaemonSet's k8s-driver-manager init container exits non-zero and labels the node pre-installed whenever it finds a host NVIDIA driver, leaving vfio-pci unbound and the sandbox validator in CrashLoopBackOff. Upstream offers no override knob, so the only working path today is a clean host. Document the symptom (including the Init:CrashLoopBackOff pod state), a build-from-installed purge plus module-blacklist workaround (with the autoremove-vs-DOCA warning and the apt zero-match-abort caveat), the post-reboot verification, and clearing the stale pre-installed label that the operator's success path never resets.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Closes#2765.
## What this PR does
Makes the passthrough and vGPU variants of `cozystack.gpu-operator`
produce a KubeVirt CR that is immediately usable, without an out-of-band
`kubectl patch`. Today the gpu-operator stack reports green,
`nvidia.com/<model>` shows up under node `.status.allocatable`, but
every `VirtualMachine` referencing the resource via
`domain.devices.hostDevices` is rejected by the admission webhook
because `HostDevices` is not in `developerConfiguration.featureGates`
and the resource is not listed under `permittedHostDevices`. That manual
reconciliation is now driven by the platform.
The change has three parts.
`packages/system/kubevirt` now parameterizes the KubeVirt CR template.
`developerConfiguration.featureGates` keeps its existing static defaults
and appends `.Values.extraFeatureGates`;
`spec.configuration.permittedHostDevices` renders verbatim from
`.Values.permittedHostDevices` when set;
`spec.configuration.mediatedDevicesConfiguration` does the same.
Defaults stay empty so the rendered CR is byte-identical to the previous
template when no overrides are supplied. The chart's `make update`
target re-applies the four sed transforms after fetching the upstream
CR, so refreshing the KubeVirt release version stays a single command.
`packages/core/platform` adds two defaults tables under `files/`.
`gpu-passthrough-defaults.yaml` covers Hopper (H100/H200), Ada Lovelace
(L4/L40/L40S), Ampere (A100 PCIe/SXM, A40, A30, A10), Turing (T4) and
Volta (V100/V100S); PCI vendor:device pairs are stable, `resourceName`
slugs follow the `<arch>_<model>_<form>_<mem>` convention
`nvidia-sandbox-device-plugin` generates as of v25.x, and
`externalResourceProvider: true` is set everywhere because those
resources are advertised by the sandbox plugin rather than KubeVirt's
in-tree device plugin. `gpu-vgpu-defaults.yaml` covers the Pascal–Ampere
mdev profile range (A100-40C/80C, A40-24Q/48Q, A30-24C, A10-24Q,
V100D-32C, T4-16Q) — the same family span the upstream
`vgpu-device-manager` walks `/sys/class/mdev_bus/` for. Ada Lovelace and
Blackwell SR-IOV vGPU is intentionally out of scope for the default
table and is expected to be advertised via the user-override hook
described below.
`templates/bundles/iaas.yaml` now selects between the two tables via a
new `bundles.iaas.gpuOperatorVariant` knob (`default` for passthrough,
`vgpu` for mediated; default is passthrough). The same variant flows
through to the gpu-operator Package CR, replacing the previous
always-default call. When the package is enabled, the platform injects
`HostDevices` into `extraFeatureGates` plus the chosen defaults into
`permittedHostDevices` for the kubevirt component's values. Operators
extend or replace via top-level platform values:
```yaml
gpu:
replaceDefaults: false # append (default) vs. wipe-and-replace
permittedHostDevices:
pciHostDevices: [] # cluster-specific PCI passthrough entries
mediatedDevices: [] # cluster-specific vGPU entries
mediatedDevicesConfiguration: {} # extra mediated-device types / per-node mappings
```
`docs/gpu-vgpu.md` is updated to drop the manual `kubectl patch` step
and describe the auto-wiring contract, the per-variant defaults, the
escape hatches, and the sandbox-device-plugin `resourceName` slug
convention so an operator whose node publishes a different name knows
where to override.
## Test coverage
`helm-unittest` covers eight branches in
`packages/core/platform/tests/bundles_gpu_kubevirt_wiring_test.yaml`:
gpu-operator disabled emits no extras; gpu-operator enabled at default
variant emits the HostDevices gate plus `pciHostDevices`; vgpu variant
switches to `mediatedDevices` plus `mediatedDevicesConfiguration`; user
overrides append by default; `replaceDefaults: true` wipes the platform
table; an unknown variant fails fast with a descriptive error.
`packages/system/kubevirt/tests/kubevirt_cr_test.yaml` pins the
empty-values render and the three populated shapes at the CR template
level. `make helm-unit-tests` passes locally on the full repo.
### Release note
```release-note
feat(platform): the KubeVirt CR is now auto-configured for GPU host-device passthrough (HostDevices feature gate plus permittedHostDevices populated from NVIDIA defaults) whenever cozystack.gpu-operator is in bundles.enabledPackages. Operators no longer need a manual kubectl patch to make VMs schedule against a GPU. The variant is selected via bundles.iaas.gpuOperatorVariant (default: passthrough; vgpu: mediated). Extend or replace the NVIDIA defaults via .gpu.permittedHostDevices and .gpu.replaceDefaults.
**Upgrade note:** the bundle now OWNS spec.configuration.permittedHostDevices on the KubeVirt CR. If you previously hand-edited it (kubectl edit kubevirt), the first reconcile after upgrade overwrites your manual entries with the rendered NVIDIA default table. Before upgrading, move any custom entries into .gpu.permittedHostDevices (set .gpu.replaceDefaults: true to keep only your list) and verify each resourceName against what your nodes advertise (kubectl describe node <node> | grep nvidia.com/) — the default table now carries the exact slug the sandbox device plugin generates, which upper-cases the card's PCI-IDs name and keeps the Tesla/GL tokens (e.g. a Tesla T4 is nvidia.com/TU104GL_TESLA_T4), but a different plugin build can publish a different string; a mismatch makes the admission webhook reject GPU VMs on their next restart or migration.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Bundle-managed GPU Operator with selectable passthrough vs vGPU
variants; platform now injects GPU settings into KubeVirt and forwards
vGPU manager overrides when enabled.
* Default platform mappings for PCI passthrough and vGPU mediated-device
resources added.
* **Documentation**
* Expanded GPU/vGPU deployment guide with bundle-managed vs manual
guidance, wiring/precedence rules, resource-name conventions, and
verification steps.
* **Tests**
* New and expanded tests for bundle→KubeVirt wiring, variant behaviors,
default/override merging, and idempotent template parameterization.
* **Chores**
* Added Helm test target and hardened KubeVirt template
update/idempotency checks.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Add a third architectural variant to `cozystack.gpu-operator` for hosts
where the NVIDIA driver and `nvidia-container-toolkit` are already
installed by the distro package manager (the typical Ubuntu/Debian GPU
node shape: apt-installed driver + apt-installed CT).
The existing variants do not produce a working setup in that shape.
`default` (passthrough) runs `vfioManager`, which unbinds the host
driver to bind `vfio-pci` — with the host driver active,
`k8s-driver-manager` early-exits and the bind never happens, so the GPU
never reaches a pod. `vgpu` requires the proprietary NVIDIA vGPU host
driver and a license server; the user-installed open or proprietary
driver does not satisfy it.
The new `container` variant maps onto upstream gpu-operator's third mode
(`sandboxWorkloads.enabled=false` + `devicePlugin.enabled=true`) with
every component that would conflict with the host install switched off:
- `driver.enabled=false` — use the host driver.
- `toolkit.enabled=false` — host already has `nvidia-container-toolkit`.
Enabling the operator's toolkit DaemonSet would overwrite the
`/etc/containerd/config.toml` the host package configured (via
`nvidia-ctk runtime configure`), breaking the host runtime wiring.
Matches NVIDIA's documented guidance and the existing
`examples/values-native-talos.yaml` reference, which pins the same flag
for the same reason.
- `devicePlugin.enabled=true` — publish `nvidia.com/gpu` to the kubelet.
- `vfioManager.enabled=false` — do not unbind the host driver. Upstream
renders this `true` unconditionally; it only stays at zero pods because
no node carries a `vm-passthrough` sandbox assignment when
`sandboxWorkloads` is off — pinned off so the rendered policy matches
intent.
- `cdi.enabled=false` — upstream defaults this `true`, but CDI spec
generation is serviced by the operator-managed toolkit DaemonSet this
variant disables. An apt host doesn't auto-generate CDI specs
(`nvidia-ctk cdi generate` is a manual step) and uses the classic
`nvidia-container-runtime` hook, so leaving CDI on would point the
device plugin at CDI injection with no specs on the host and silently
break allocation.
- `sandboxWorkloads.enabled=false` — regular pod workloads, not VMs.
With this off, upstream `defaultWorkload` is already `container`, so no
override is needed.
Host prerequisite: because the operator's toolkit component is disabled,
the node's containerd must already have the `nvidia` runtime registered
(`nvidia-ctk runtime configure --runtime=containerd`); `apt install
nvidia-container-toolkit` installs binaries only and does not wire
containerd. With `driver.enabled=false` the operator uses the
pre-installed host driver at its standard location, so a stock apt
install needs no `hostPaths.driverInstallDir` override (Talos lands the
driver under a non-standard prefix — see
`examples/values-native-talos.yaml`).
The bundle wiring (`packages/core/platform/templates/bundles/iaas.yaml`)
already enables `cozystack.gpu-operator` as an optional package — no
bundle changes required. Users opt into this variant by setting
`variant: container` in their `Package` CR.
Verified locally with `helm template` against `values.yaml` +
`values-container.yaml`: ClusterPolicy renders with
`driver.enabled=false`, `toolkit.enabled=false`,
`devicePlugin.enabled=true`, `vfioManager.enabled=false`,
`cdi.enabled=false`, `sandboxWorkloads.enabled=false`. Platform `helm
unittest` and the `gpu-operator` variant bats suite are green. Not yet
smoke-tested on a real Ubuntu GPU node — the config matches the
documented apt-host wiring (legacy `nvidia-container-runtime` hook, CDI
off), but an end-to-end CUDA-pod run on hardware is still owed.
Closes#2764
### Release note
```release-note
feat(gpu-operator): add `container` variant for hosts with a pre-installed NVIDIA driver and nvidia-container-toolkit, exposing containerized GPU workloads via the standard NVIDIA device plugin without VFIO passthrough or vGPU.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a "container" GPU operator variant to support containerized GPU
workloads on hosts with the NVIDIA driver and nvidia-container-toolkit.
* **Documentation**
* Updated variants documentation and OS support table to include the
container variant and per-host requirements (including Ubuntu and Talos
notes).
* Clarified Talos native pod example wording.
* **Tests**
* Added regression tests to verify container-variant configuration and
component enablement/disablement.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Expands `docs/release.md` from a ~165-line process-design doc into a
~573-line single source covering both the process design (kept verbatim
— mermaid diagrams and the RC/regular/patch numbered walkthroughs) and
the operational playbook for cutting a release.
New sections added under the existing diagrams:
- **What CI does on a tag push** — phase-by-phase walk through
`tags.yaml` (`prepare-release`, `generate-changelog`,
`update-website-docs`), `pull-requests-release.yaml` (finalize), and
`update-releasenotes.yaml` (sync release body).
- **Automated patch tag** — `auto-release.yaml`'s nightly behavior.
- **Backports** — bot mechanism, recursive-backport regression history
(fix#2584), manual fallback, cherry-pick triage include/skip rule,
monotonic-counter trap for migration backports.
- **Changelogs** — pointer to `agents/changelog.md` plus the recurring
failure modes that have shipped in real changelogs, with a verification
snippet (chart-bump coverage, range-vs-changelog PR sweep, URL liveness,
website PR-author check).
- **Release-time fires** — three customer-impacting regressions (v1.3.0
cert-manager ingressClassName, v1.4.0 kubevirt-instancetypes upgrade,
v1.4.0 Flux 2.7 → 2.8 readiness deadlock) with "what would have caught
this" notes.
- **Pre-release verification checklist**, **CI failures release
engineers commonly hit**, **Force-retagging** (with open RFC #2677
context), **Splitting a release-blocking bundle PR**, **Cleanup after
release**.
Example version numbers across the walkthrough switched from the
upstream `v0.42.x` legacy placeholders to simple semver-style (`v1.2.0`,
`v1.2.1`, `release-1.2`) consistent with semver.org examples and
applicable at any point in v1.x.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Enhanced release process documentation with clarified CI workflows,
comprehensive verification checklists, expanded troubleshooting
guidance, and case studies to ensure more reliable releases.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2732?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The vgpu defaults shipped a mediatedDevicesConfiguration block whose
mediatedDeviceTypes were hardcoded nvidia-NNN ids. Those ids are not
stable identifiers — they are the per-(SKU, driver) indices under which a
host enumerates mdev types in /sys/.../mdev_supported_types/, so the same
profile gets a different number on a different card or driver branch.
Auto-injecting them made KubeVirt try to create mdev types that usually
do not exist on the operator's nodes, and duplicated work the GPU
Operator's vGPU Device Manager already does.
Delegate mdev creation to the vGPU Device Manager, which is name-based and
stable, and keep only the name-based permittedHostDevices.mediatedDevices
exposure list (mdevNameSelector) in the defaults. KubeVirt-driven creation
stays available through the .gpu.mediatedDevicesConfiguration override for
operators who supply host-verified type ids; document the lookup recipe.
Update the bundle wiring, the gpu/vgpu values comments, the vGPU guide and
the tests accordingly (the default vgpu path now emits no
mediatedDevicesConfiguration; the override path is covered explicitly).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The platform now wires permittedHostDevices and the HostDevices feature
gate into the KubeVirt CR automatically whenever cozystack.gpu-operator
is in bundles.enabledPackages. The KubeVirt configuration section now
describes the auto-wiring contract, the per-variant defaults, the
.gpu.permittedHostDevices / .gpu.replaceDefaults escape hatches, and
calls out the sandbox-device-plugin resourceName slug convention so
operators know where to look when a node's allocatable name does not
match the platform default.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The two existing variants assume the host has neither the NVIDIA driver
nor the container toolkit: default unbinds the host driver to bind
vfio-pci for KubeVirt VMs, vgpu requires the proprietary vGPU host
driver and a license server. Neither path produces a working setup on a
node that already ships the driver and nvidia-container-toolkit via the
distro package manager — the most common shape of a Linux GPU node.
The new container variant surfaces the third architectural mode that
upstream NVIDIA gpu-operator already supports: sandboxWorkloads
disabled, devicePlugin enabled, with every component that would
conflict with the host install switched off.
- driver.enabled=false use the host driver
- toolkit.enabled=false host already has nvidia-container-toolkit;
enabling overwrites the containerd
config.toml the host configured via
nvidia-ctk runtime configure
- devicePlugin.enabled=true publish nvidia.com/gpu to the kubelet
- vfioManager.enabled=false do not unbind the host driver
- cdi.enabled=false CDI spec generation is serviced by the
disabled toolkit; an apt host uses the
legacy nvidia-container-runtime hook, so
pin off to match the host runtime wiring
- sandboxWorkloads.enabled=false regular pod workloads, not VMs
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
myasnikovdaniil
