The "Pin the e2e management cluster Kubernetes version" item is no longer
in-flight — it is merged and live in hack/e2e-prepare-cluster.bats
(--kubernetes-version v1.33.12). Remove it from the "not yet the merged
standard" section; its rationale lives in the bats comment.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Merge the AI-generated v1.5.0 changelog with a fuller pass:
- Add an "Upgrade Notes and Required Actions" section (K8s 1.33+, removal
of upgrade.force, GPU permittedHostDevices ownership, MetalLB FRR-K8s /
HTTPS metrics, automatic TLS on externally-published services).
- Expand "Platform Components" with per-component upstream highlights and
CVE references (Flux v2.8 CVE-2026-23990, MetalLB v0.16 CVE-2025-22874),
plus the new frr-k8s subchart and kuberture package.
- Scope the storageClass-immutability entry (#2639) to UI-only enforcement,
with apiserver enforcement tracked in #2657.
- Broaden Development/Testing/CI coverage and add the v1.4.x website docs
backports (#565, #566, #567, #580).
- Correct the loop-device LVM-filter attribution to talm#215 (the prior
ansible-cozystack#51 reference was a dependency bump, not this change).
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
The **Auto Patch Release** cron (`.github/workflows/auto-release.yaml`)
walked
*every* `release-X.Y` branch — 21 of them, back to `release-0.9` — and
cut a
`vX.Y.(Z+1)` tag whenever a branch had commits after its latest
published
release. This fires patch releases on long-EOL lines (e.g. `v1.3.6`),
whose
release PRs then sit blocked on unrelated CI failures and accumulate
stray
draft releases, tags and changelog PRs.
This restricts the cron to the **2 newest minor lines**
(`SUPPORTED_LINE_COUNT`).
The window is derived from the live branch list, so it slides
automatically as
new `release-X.Y` branches are cut — once `release-1.5` exists the
window
becomes `{1.5, 1.4}` and the trailing line retires with no further
edits.
Currently active: `release-1.4`, `release-1.3`. Skipped: 19 EOL lines.
### Release note
```release-note
ci(release): the automated patch-release job now only cuts releases for the two newest release branches; older EOL lines no longer receive automatic patch tags.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated the automated release workflow to focus patch automation on
only the two newest minor `release-X.Y` maintenance lines.
* Enhanced workflow logging to report the selected branches and any
older/EOL branches skipped.
* **Documentation**
* Refined release documentation to explain the sliding-window selection
for automated patch tagging, including how next patch versions are
determined and when tags are created.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Promotes universal project guidance that previously lived only in a
local
(gitignored) `CLAUDE.md` into the committed agent docs, so every
contributor and
every tool (Codex/Cursor/etc., not just Claude) gets it from a fresh
clone.
- `overview.md`: Build & Development Commands (root + per-package make
targets,
build env vars), Values schema generation, a Testing section, Domain
references
- `contributing.md`: a "Verify Before Considering Done" matrix + Review
Expectations
- Dedup: `overview.md`'s Git Commits block now points to
`contributing.md`
> Note: `overview.md` links to `docs/agents/e2e-testing.md` added in
#2932 — merge
> that one first or together.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Expanded contribution guidance with a “Verify Before Considering Done”
pre-submission checklist and clearer expectations for including
behavioral regression tests in bug-fix PRs.
* Broadened build/development documentation with a “Build and
Development Commands” guide, including required environment variables,
values-schema generation, and test entry points.
* Updated contribution workflow notes by consolidating commit-convention
guidance and adding pointers to task-specific design references.
* Added a Claude entry-point document that directs to the main agents
guide and explains local override placement.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Adds `docs/agents/e2e-testing.md` — a routed agent doc capturing the
e2e/CI
stabilization conventions built up over the last few months:
- No retries on deterministic CI steps (only pure-infra `Prepare
environment` retries)
- Event-driven backstops instead of fixed `sleep`; no `EXIT`/`RETURN`
traps
- Fail-fast HelmRelease-readiness gating + the `status.history`
remediation guard
- First-class failure diagnostics and a reviewer checklist
- Test-Impact Analysis (TIA) and its real scope: it trims only the
per-app loop,
runs after install, and most PRs legitimately escalate to the full suite
Wired into `AGENTS.md` so every agent reads it before e2e/CI work.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Introduced comprehensive E2E testing conventions document with
standards for writing, modifying, and reviewing end-to-end tests and CI
processes.
* Documented best practices for test reliability, including guidelines
on retry behavior, event-driven test validation, and cleanup discipline.
* Updated process documentation to reference new E2E testing
conventions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Address review feedback from lexfrei on docs/release.md: the Patch Releases section described only the manual process, so add an "Automated Patch Releases" subsection covering the nightly cron, the new-commits trigger, the 2-newest-lines support window, and the manual tag-push escape hatch for EOL lines.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from coderabbitai on docs/agents/overview.md:145:
internal/controller/dashboard/README.md does not exist (no such directory),
and there is no dashboard reference doc to point to. Remove the dead bullet;
the backups DESIGN.md reference is valid and kept.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from gemini-code-assist on docs/agents/contributing.md:145:
reword the awkward "on the template" phrasing to "aligned with the template".
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from gemini-code-assist on docs/agents/overview.md:103:
the Helm-only target is make helm-unit-tests; make unit-tests runs the full
unit suite. Point the bullet at the precise target and note the difference.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from gemini-code-assist on docs/agents/overview.md:72:
make unit-tests runs the full suite (Helm, BATS, Go, presets, readiness),
not only Helm unit tests.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Promote the universal project guidance that previously lived only in a
local CLAUDE.md into the committed agent docs, so every contributor and
every tool (Codex/Cursor/etc., not just Claude) gets it from a fresh
clone.
overview.md gains Build and Development Commands (root + per-package
make targets, build env vars), Values schema generation, a Testing
section, and Domain references. contributing.md gains a "Verify Before
Considering Done" matrix and Review Expectations. Dedup: overview's Git
Commits block now points to contributing.md instead of restating it.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Capture the e2e stabilization conventions established over the last few
months as a routed agent doc: no retries on deterministic CI steps,
event-driven backstops over fixed sleeps, no EXIT/RETURN traps,
fail-fast HelmRelease-readiness gating, the status.history remediation
guard, first-class failure diagnostics, and a reviewer checklist. Also
documents Test-Impact Analysis (TIA) and its real scope — it trims only
the per-app loop, runs after install, and most PRs escalate to the full
suite. Wire it into AGENTS.md so every agent reads it before e2e/CI work.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The vfio-manager DaemonSet's k8s-driver-manager init container exits non-zero and labels the node pre-installed whenever it finds a host NVIDIA driver, leaving vfio-pci unbound and the sandbox validator in CrashLoopBackOff. Upstream offers no override knob, so the only working path today is a clean host. Document the symptom (including the Init:CrashLoopBackOff pod state), a build-from-installed purge plus module-blacklist workaround (with the autoremove-vs-DOCA warning and the apt zero-match-abort caveat), the post-reboot verification, and clearing the stale pre-installed label that the operator's success path never resets.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Closes#2765.
## What this PR does
Makes the passthrough and vGPU variants of `cozystack.gpu-operator`
produce a KubeVirt CR that is immediately usable, without an out-of-band
`kubectl patch`. Today the gpu-operator stack reports green,
`nvidia.com/<model>` shows up under node `.status.allocatable`, but
every `VirtualMachine` referencing the resource via
`domain.devices.hostDevices` is rejected by the admission webhook
because `HostDevices` is not in `developerConfiguration.featureGates`
and the resource is not listed under `permittedHostDevices`. That manual
reconciliation is now driven by the platform.
The change has three parts.
`packages/system/kubevirt` now parameterizes the KubeVirt CR template.
`developerConfiguration.featureGates` keeps its existing static defaults
and appends `.Values.extraFeatureGates`;
`spec.configuration.permittedHostDevices` renders verbatim from
`.Values.permittedHostDevices` when set;
`spec.configuration.mediatedDevicesConfiguration` does the same.
Defaults stay empty so the rendered CR is byte-identical to the previous
template when no overrides are supplied. The chart's `make update`
target re-applies the four sed transforms after fetching the upstream
CR, so refreshing the KubeVirt release version stays a single command.
`packages/core/platform` adds two defaults tables under `files/`.
`gpu-passthrough-defaults.yaml` covers Hopper (H100/H200), Ada Lovelace
(L4/L40/L40S), Ampere (A100 PCIe/SXM, A40, A30, A10), Turing (T4) and
Volta (V100/V100S); PCI vendor:device pairs are stable, `resourceName`
slugs follow the `<arch>_<model>_<form>_<mem>` convention
`nvidia-sandbox-device-plugin` generates as of v25.x, and
`externalResourceProvider: true` is set everywhere because those
resources are advertised by the sandbox plugin rather than KubeVirt's
in-tree device plugin. `gpu-vgpu-defaults.yaml` covers the Pascal–Ampere
mdev profile range (A100-40C/80C, A40-24Q/48Q, A30-24C, A10-24Q,
V100D-32C, T4-16Q) — the same family span the upstream
`vgpu-device-manager` walks `/sys/class/mdev_bus/` for. Ada Lovelace and
Blackwell SR-IOV vGPU is intentionally out of scope for the default
table and is expected to be advertised via the user-override hook
described below.
`templates/bundles/iaas.yaml` now selects between the two tables via a
new `bundles.iaas.gpuOperatorVariant` knob (`default` for passthrough,
`vgpu` for mediated; default is passthrough). The same variant flows
through to the gpu-operator Package CR, replacing the previous
always-default call. When the package is enabled, the platform injects
`HostDevices` into `extraFeatureGates` plus the chosen defaults into
`permittedHostDevices` for the kubevirt component's values. Operators
extend or replace via top-level platform values:
```yaml
gpu:
replaceDefaults: false # append (default) vs. wipe-and-replace
permittedHostDevices:
pciHostDevices: [] # cluster-specific PCI passthrough entries
mediatedDevices: [] # cluster-specific vGPU entries
mediatedDevicesConfiguration: {} # extra mediated-device types / per-node mappings
```
`docs/gpu-vgpu.md` is updated to drop the manual `kubectl patch` step
and describe the auto-wiring contract, the per-variant defaults, the
escape hatches, and the sandbox-device-plugin `resourceName` slug
convention so an operator whose node publishes a different name knows
where to override.
## Test coverage
`helm-unittest` covers eight branches in
`packages/core/platform/tests/bundles_gpu_kubevirt_wiring_test.yaml`:
gpu-operator disabled emits no extras; gpu-operator enabled at default
variant emits the HostDevices gate plus `pciHostDevices`; vgpu variant
switches to `mediatedDevices` plus `mediatedDevicesConfiguration`; user
overrides append by default; `replaceDefaults: true` wipes the platform
table; an unknown variant fails fast with a descriptive error.
`packages/system/kubevirt/tests/kubevirt_cr_test.yaml` pins the
empty-values render and the three populated shapes at the CR template
level. `make helm-unit-tests` passes locally on the full repo.
### Release note
```release-note
feat(platform): the KubeVirt CR is now auto-configured for GPU host-device passthrough (HostDevices feature gate plus permittedHostDevices populated from NVIDIA defaults) whenever cozystack.gpu-operator is in bundles.enabledPackages. Operators no longer need a manual kubectl patch to make VMs schedule against a GPU. The variant is selected via bundles.iaas.gpuOperatorVariant (default: passthrough; vgpu: mediated). Extend or replace the NVIDIA defaults via .gpu.permittedHostDevices and .gpu.replaceDefaults.
**Upgrade note:** the bundle now OWNS spec.configuration.permittedHostDevices on the KubeVirt CR. If you previously hand-edited it (kubectl edit kubevirt), the first reconcile after upgrade overwrites your manual entries with the rendered NVIDIA default table. Before upgrading, move any custom entries into .gpu.permittedHostDevices (set .gpu.replaceDefaults: true to keep only your list) and verify each resourceName against what your nodes advertise (kubectl describe node <node> | grep nvidia.com/) — the default table now carries the exact slug the sandbox device plugin generates, which upper-cases the card's PCI-IDs name and keeps the Tesla/GL tokens (e.g. a Tesla T4 is nvidia.com/TU104GL_TESLA_T4), but a different plugin build can publish a different string; a mismatch makes the admission webhook reject GPU VMs on their next restart or migration.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Bundle-managed GPU Operator with selectable passthrough vs vGPU
variants; platform now injects GPU settings into KubeVirt and forwards
vGPU manager overrides when enabled.
* Default platform mappings for PCI passthrough and vGPU mediated-device
resources added.
* **Documentation**
* Expanded GPU/vGPU deployment guide with bundle-managed vs manual
guidance, wiring/precedence rules, resource-name conventions, and
verification steps.
* **Tests**
* New and expanded tests for bundle→KubeVirt wiring, variant behaviors,
default/override merging, and idempotent template parameterization.
* **Chores**
* Added Helm test target and hardened KubeVirt template
update/idempotency checks.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Add a third architectural variant to `cozystack.gpu-operator` for hosts
where the NVIDIA driver and `nvidia-container-toolkit` are already
installed by the distro package manager (the typical Ubuntu/Debian GPU
node shape: apt-installed driver + apt-installed CT).
The existing variants do not produce a working setup in that shape.
`default` (passthrough) runs `vfioManager`, which unbinds the host
driver to bind `vfio-pci` — with the host driver active,
`k8s-driver-manager` early-exits and the bind never happens, so the GPU
never reaches a pod. `vgpu` requires the proprietary NVIDIA vGPU host
driver and a license server; the user-installed open or proprietary
driver does not satisfy it.
The new `container` variant maps onto upstream gpu-operator's third mode
(`sandboxWorkloads.enabled=false` + `devicePlugin.enabled=true`) with
every component that would conflict with the host install switched off:
- `driver.enabled=false` — use the host driver.
- `toolkit.enabled=false` — host already has `nvidia-container-toolkit`.
Enabling the operator's toolkit DaemonSet would overwrite the
`/etc/containerd/config.toml` the host package configured (via
`nvidia-ctk runtime configure`), breaking the host runtime wiring.
Matches NVIDIA's documented guidance and the existing
`examples/values-native-talos.yaml` reference, which pins the same flag
for the same reason.
- `devicePlugin.enabled=true` — publish `nvidia.com/gpu` to the kubelet.
- `vfioManager.enabled=false` — do not unbind the host driver. Upstream
renders this `true` unconditionally; it only stays at zero pods because
no node carries a `vm-passthrough` sandbox assignment when
`sandboxWorkloads` is off — pinned off so the rendered policy matches
intent.
- `cdi.enabled=false` — upstream defaults this `true`, but CDI spec
generation is serviced by the operator-managed toolkit DaemonSet this
variant disables. An apt host doesn't auto-generate CDI specs
(`nvidia-ctk cdi generate` is a manual step) and uses the classic
`nvidia-container-runtime` hook, so leaving CDI on would point the
device plugin at CDI injection with no specs on the host and silently
break allocation.
- `sandboxWorkloads.enabled=false` — regular pod workloads, not VMs.
With this off, upstream `defaultWorkload` is already `container`, so no
override is needed.
Host prerequisite: because the operator's toolkit component is disabled,
the node's containerd must already have the `nvidia` runtime registered
(`nvidia-ctk runtime configure --runtime=containerd`); `apt install
nvidia-container-toolkit` installs binaries only and does not wire
containerd. With `driver.enabled=false` the operator uses the
pre-installed host driver at its standard location, so a stock apt
install needs no `hostPaths.driverInstallDir` override (Talos lands the
driver under a non-standard prefix — see
`examples/values-native-talos.yaml`).
The bundle wiring (`packages/core/platform/templates/bundles/iaas.yaml`)
already enables `cozystack.gpu-operator` as an optional package — no
bundle changes required. Users opt into this variant by setting
`variant: container` in their `Package` CR.
Verified locally with `helm template` against `values.yaml` +
`values-container.yaml`: ClusterPolicy renders with
`driver.enabled=false`, `toolkit.enabled=false`,
`devicePlugin.enabled=true`, `vfioManager.enabled=false`,
`cdi.enabled=false`, `sandboxWorkloads.enabled=false`. Platform `helm
unittest` and the `gpu-operator` variant bats suite are green. Not yet
smoke-tested on a real Ubuntu GPU node — the config matches the
documented apt-host wiring (legacy `nvidia-container-runtime` hook, CDI
off), but an end-to-end CUDA-pod run on hardware is still owed.
Closes#2764
### Release note
```release-note
feat(gpu-operator): add `container` variant for hosts with a pre-installed NVIDIA driver and nvidia-container-toolkit, exposing containerized GPU workloads via the standard NVIDIA device plugin without VFIO passthrough or vGPU.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a "container" GPU operator variant to support containerized GPU
workloads on hosts with the NVIDIA driver and nvidia-container-toolkit.
* **Documentation**
* Updated variants documentation and OS support table to include the
container variant and per-host requirements (including Ubuntu and Talos
notes).
* Clarified Talos native pod example wording.
* **Tests**
* Added regression tests to verify container-variant configuration and
component enablement/disablement.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Expands `docs/release.md` from a ~165-line process-design doc into a
~573-line single source covering both the process design (kept verbatim
— mermaid diagrams and the RC/regular/patch numbered walkthroughs) and
the operational playbook for cutting a release.
New sections added under the existing diagrams:
- **What CI does on a tag push** — phase-by-phase walk through
`tags.yaml` (`prepare-release`, `generate-changelog`,
`update-website-docs`), `pull-requests-release.yaml` (finalize), and
`update-releasenotes.yaml` (sync release body).
- **Automated patch tag** — `auto-release.yaml`'s nightly behavior.
- **Backports** — bot mechanism, recursive-backport regression history
(fix#2584), manual fallback, cherry-pick triage include/skip rule,
monotonic-counter trap for migration backports.
- **Changelogs** — pointer to `agents/changelog.md` plus the recurring
failure modes that have shipped in real changelogs, with a verification
snippet (chart-bump coverage, range-vs-changelog PR sweep, URL liveness,
website PR-author check).
- **Release-time fires** — three customer-impacting regressions (v1.3.0
cert-manager ingressClassName, v1.4.0 kubevirt-instancetypes upgrade,
v1.4.0 Flux 2.7 → 2.8 readiness deadlock) with "what would have caught
this" notes.
- **Pre-release verification checklist**, **CI failures release
engineers commonly hit**, **Force-retagging** (with open RFC #2677
context), **Splitting a release-blocking bundle PR**, **Cleanup after
release**.
Example version numbers across the walkthrough switched from the
upstream `v0.42.x` legacy placeholders to simple semver-style (`v1.2.0`,
`v1.2.1`, `release-1.2`) consistent with semver.org examples and
applicable at any point in v1.x.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Enhanced release process documentation with clarified CI workflows,
comprehensive verification checklists, expanded troubleshooting
guidance, and case studies to ensure more reliable releases.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2732?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The vgpu defaults shipped a mediatedDevicesConfiguration block whose
mediatedDeviceTypes were hardcoded nvidia-NNN ids. Those ids are not
stable identifiers — they are the per-(SKU, driver) indices under which a
host enumerates mdev types in /sys/.../mdev_supported_types/, so the same
profile gets a different number on a different card or driver branch.
Auto-injecting them made KubeVirt try to create mdev types that usually
do not exist on the operator's nodes, and duplicated work the GPU
Operator's vGPU Device Manager already does.
Delegate mdev creation to the vGPU Device Manager, which is name-based and
stable, and keep only the name-based permittedHostDevices.mediatedDevices
exposure list (mdevNameSelector) in the defaults. KubeVirt-driven creation
stays available through the .gpu.mediatedDevicesConfiguration override for
operators who supply host-verified type ids; document the lookup recipe.
Update the bundle wiring, the gpu/vgpu values comments, the vGPU guide and
the tests accordingly (the default vgpu path now emits no
mediatedDevicesConfiguration; the override path is covered explicitly).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The platform now wires permittedHostDevices and the HostDevices feature
gate into the KubeVirt CR automatically whenever cozystack.gpu-operator
is in bundles.enabledPackages. The KubeVirt configuration section now
describes the auto-wiring contract, the per-variant defaults, the
.gpu.permittedHostDevices / .gpu.replaceDefaults escape hatches, and
calls out the sandbox-device-plugin resourceName slug convention so
operators know where to look when a node's allocatable name does not
match the platform default.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The two existing variants assume the host has neither the NVIDIA driver
nor the container toolkit: default unbinds the host driver to bind
vfio-pci for KubeVirt VMs, vgpu requires the proprietary vGPU host
driver and a license server. Neither path produces a working setup on a
node that already ships the driver and nvidia-container-toolkit via the
distro package manager — the most common shape of a Linux GPU node.
The new container variant surfaces the third architectural mode that
upstream NVIDIA gpu-operator already supports: sandboxWorkloads
disabled, devicePlugin enabled, with every component that would
conflict with the host install switched off.
- driver.enabled=false use the host driver
- toolkit.enabled=false host already has nvidia-container-toolkit;
enabling overwrites the containerd
config.toml the host configured via
nvidia-ctk runtime configure
- devicePlugin.enabled=true publish nvidia.com/gpu to the kubelet
- vfioManager.enabled=false do not unbind the host driver
- cdi.enabled=false CDI spec generation is serviced by the
disabled toolkit; an apt host uses the
legacy nvidia-container-runtime hook, so
pin off to match the host runtime wiring
- sandboxWorkloads.enabled=false regular pod workloads, not VMs
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Annotate storageClass as immutable across every stateful app whose
chart exposes one — clickhouse, foundationdb, harbor, http-cache,
kafka (Kafka + ZooKeeper), kubernetes (top-level only), mariadb,
mongodb, nats, openbao, opensearch, postgres, qdrant, rabbitmq,
redis, vm-disk. Without coverage across the catalogue the UI would
render storageClass read-only on some apps and editable on others —
a worse mental model than uniform behaviour.
The annotation lands at the schema layer (CEL XValidation rule
'self == oldSelf' on values.schema.json / types.go / embedded
openAPISchema in cozyrds/<app>.yaml). The cozystack aggregated
apiserver does not yet evaluate CEL on application-backed resources;
that is tracked at cozystack/cozystack#2657. The UI
(cozystack/cozystack-ui#6) consumes the marker today.
The PVC-pinning argument is identical for every chart: Kubernetes
fixes a PVC's storageClassName at creation time, so editing this
field on an existing resource never migrated data. The schema
annotation makes that contract explicit.
kubernetes.nodeGroups[name].storageClass is intentionally NOT
annotated: the field is optional and undefaulted, so a strict
'self == oldSelf' rule would block any future attempt to set it on
an existing node group. A richer ratcheting form is needed there —
see docs/storage-immutability.md.
A refactor in pkg/registry/apps/application/rest.go extracts the
inline schema-build path into buildSpecSchema so it can be
unit-tested. Three regression tests in rest_spec_schema_test.go pin
the contract that x-kubernetes-validations survives the
v1->internal->structural conversion (loaded from the actual MariaDB
ApplicationDefinition fixture on disk, not a hand-edited mock).
Each per-app README gains a one-line blockquote pointing to a new
docs/storage-immutability.md that explains the contract and which
consumers enforce it today.
Requires cozystack/cozyvalues-gen with @immutable support — bump the
pre-commit pin in .github/workflows/pre-commit.yml once the release
exists.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
This PR adds the changelog for release `v1.3.4`.
✅ Changelog has been automatically generated in
`docs/changelogs/v1.3.4.md`.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* OpenSearch is now integrated into the 1.3 PaaS bundle so operator and
dashboard components deploy when PaaS is enabled.
* Storage class input now renders as a dropdown including OpenSearch as
an option.
* **Bug Fixes**
* Fixed Harbor reconciliation regression related to bucket secret
handling and render-path coverage.
* **Documentation**
* v1.3.4 release notes published, expanded docs, summary of related
project changes, contributor list and compare link.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2676?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Adds the missing entry for #2649 (OpenSearch wired into PaaS bundle, plus
StorageClass dropdown override for the legacy 1.3 openapi-ui dashboard),
corrects six website-PR author attributions against gh pr view, picks up
two website docs PRs merged just before the v1.3.4 cut (cozystack/website#536
and #538), updates the contributors list, and refreshes the date to today.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from gemini-code-assist on
docs/changelogs/v1.4.0.md:326: the in-prose mention of dislogical was
rendered as inline code rather than as a contributor link. Switch to
the [**@user**](https://github.com/user) format used elsewhere in the
changelog.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from gemini-code-assist on
docs/changelogs/v1.4.0.md:102 and coderabbitai on :103: the AMD
SEV-SNP feature gate in upstream KubeVirt source is WorkloadEncryptionSEV
(see pkg/virt-config/featuregate/active.go), not WorkloadEncruptionSEV
as quoted from the v1.7.0 release notes.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Hand-written patch changelog for v1.2.4 (Copilot quota exhausted on the
automatic Generate Changelog step). Single fix: minimal backport of the
MariaDB backup CronJob duplicate-key removal from #2279 via #2652.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Document the five instance-type series (t1/c1/s1/u1/m1) introduced
alongside the legacy preset deprecation, the per-size CPU and memory
matrix, and the 1:1 legacy-to-instance-type mapping table that powers
cozy-lib aliases, migration 39, and the operator-side deprecation
warning.
Single source of truth referenced from:
- packages/library/cozy-lib/templates/_resourcepresets.tpl
- packages/core/platform/images/migrations/migrations/39
- pkg/apis/apps/presets/legacy.go
Includes an explicit callout that legacy `medium` (1 CPU / 1Gi) and
instance-type `*.medium` (2 CPU) are not the same — same word,
different resources.
Replace inline preset tables in packages/apps/postgres/README.md and
packages/apps/kafka/README.md with a short paragraph that names the
five series and points at the canonical doc, since the per-app
inline tables listed only the legacy seven and would drift further
as the canonical matrix evolves.
Co-authored-by: Matthieu ROBIN <matthieu@hidora.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Updates the NVIDIA GPU Operator from v25.3.0 to v26.3.1 and adds an
**experimental** `vgpu` variant for SR-IOV vGPU on Ada Lovelace and
newer GPUs.
Changes:
- Update vendored upstream chart to v26.3.1 (CDI enabled by default,
component version bumps).
- Pin chart version in Makefile for reproducibility.
- Add `values-vgpu.yaml` for the vGPU variant with `defaultWorkload:
vm-vgpu` so the chart activates the vGPU Manager DaemonSet without
per-node labelling.
- Add `vgpu` variant to PackageSource alongside the existing `default`
(passthrough).
- Rename `values-talos.yaml` → `values-passthrough.yaml` and pin
`defaultWorkload: vm-passthrough` for symmetry with the vgpu variant.
The overrides apply to any host OS using vfio-pci, not Talos
specifically.
- Refresh `hack/dcgm-default-counters.csv` to match the new
dcgm-exporter image tag (`4.5.1-4.8.0`) — adds `DCGM_FI_DEV_FB_RESERVED`
and other fields introduced upstream.
`hack/check-gpu-recording-rules.bats` passes against the new fixture.
- Add `hack/check-gpu-operator-variants.bats` that renders both variants
via `helm template` and asserts the pinned ClusterPolicy values, so the
next chart bump cannot silently un-pin them.
- Add internal documentation for the vGPU bring-up.
The `vgpu` variant assumes the user supplies a proprietary vGPU Manager
image via Package CR component values — the .run cannot be
redistributed.
### `vgpu` variant is experimental
NVIDIA's `vgpu-device-manager` walks `/sys/class/mdev_bus/`, which does
not exist on Ada Lovelace and Blackwell — the DaemonSet errors with "no
parent devices found" and is disabled by default. Until an SR-IOV-aware
controller is shipped, profile assignment is an out-of-band step (`echo
<id> > /sys/.../current_vgpu_type` per VF) that must be re-applied after
every node reboot. Without it, `permittedHostDevices.pciHostDevices`
reports zero allocatable resources. The variant ships so the chart bump
and the underlying skeleton (DaemonSet + image plumbing + docs) land
together; production deployments need an automated profile-assignment
mechanism in addition to this PR.
### KubeVirt requirement
KubeVirt advertises VFs via `permittedHostDevices.pciHostDevices` rather
than `mediatedDevices`. This requires kubevirt/kubevirt#16890 ("vGPU:
SRIOV support", merged 2026-04-10) which is `main`-only — released tags
`v1.6.x` / `v1.7.x` / `v1.8.x` do not include it. The patch is targeted
at the next minor KubeVirt release (v1.9.0); track the PR for the actual
release tag. Backports to released branches are not planned.
### Behavior changes inherited from upstream chart v25 → v26
The upstream chart flipped two defaults that are pinned back to `false`
here, so `default` (passthrough) deployments do not silently change
footprint on upgrade:
- `ccManager.enabled` was flipped to `true` upstream — would auto-enable
Confidential Computing on Hopper hardware (H100/H200/B100) after
upgrade.
- `vgpuDeviceManager.enabled` was flipped to `true` upstream — DaemonSet
would render even in passthrough mode.
Both stay opt-in via Package CR overrides. The new bats test guards the
pin.
Other inherited changes that are user-visible and worth pinning down
before upgrade:
- `driver.licensingConfig.configMapName` is deprecated upstream in
favour of `driver.licensingConfig.secretName`. The deprecation marker
lives in the CRD's `description` field (visible via `kubectl explain`) —
there is no helm-render-time warning, the chart template silently
prefers `secretName` and falls back to `configMapName`. Operators with
existing licensing overrides should migrate the key on the next chart
upgrade.
- `driver.version` upstream default jumped from `570.x` to `580.x`.
Invisible on the `default` variant (which sets `driver.enabled: false`),
user-visible for anyone running the upstream Ubuntu driver path.
- `vfioManager.image` switched from
`nvcr.io/nvidia/cuda:12.8.1-base-ubi9` to
`nvcr.io/nvidia/cloud-native/k8s-driver-manager:v0.10.0`. Affects every
`default` (passthrough) deployment on upgrade — different image,
different layer, different image-pull surface. Worth pre-mirroring for
air-gapped clusters.
- `dcgmExporter.image` switched to the `-distroless` flavour. Smaller
surface, fewer base-image CVEs, but no shell — operators who used to
`kubectl exec` into the exporter for ad-hoc debugging will need to
switch to `kubectl debug` or container ephemeral images.
- `CDI` (Container Device Interface) is enabled by default in v26.3.1;
this is upstream's recommended setting and is left as-is.
- `node-feature-discovery` subchart source switched from the GitHub
Pages Helm repo to `oci://registry.k8s.io/nfd/charts`. Air-gapped
installations need to mirror the new OCI URL.
### Talos
Talos is **not recommended** for the vGPU variant. NVIDIA does not
publicly distribute the vGPU guest driver — it requires NVIDIA
Enterprise Portal access — and Sidero closedsiderolabs/extensions#461
noting that they cannot support vGPU "unless NVIDIA changes their
licensing terms or provides us a way to obtain, test, and distribute the
software". Building a Talos system extension that bakes in the vGPU
driver is therefore not feasible without a private fork that violates
the EULA. Passthrough (`default` variant) on Talos is unaffected.
### Release note
```release-note
[gpu-operator] Update NVIDIA GPU Operator to v26.3.1 and add experimental `vgpu` variant for SR-IOV vGPU on Ada Lovelace / Blackwell (L4, L40, L40S, B100; Ubuntu host; requires KubeVirt v1.9.0+ for `pciHostDevices` advertising). The variant ships skeleton support — profile assignment (`current_vgpu_type` per VF) is currently out of band and must be automated by the operator before production use. On the existing `default` variant: `ccManager` and `vgpuDeviceManager` upstream defaults are pinned back to `false` to preserve prior behavior on chart upgrade; `defaultWorkload` is now pinned to `vm-passthrough` (was unset, falling back to `container`); `driver.licensingConfig.configMapName` is deprecated by upstream in favour of `secretName`; `driver.version` upstream default moves 570 → 580; `vfioManager.image` switches from CUDA UBI9 to k8s-driver-manager (relevant for air-gapped mirrors); `dcgmExporter` image switched to the `-distroless` flavour. NFD subchart source switched to `oci://registry.k8s.io/nfd/charts`.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Introduced vGPU support variant with end-to-end configuration guide,
validation procedures, and troubleshooting documentation
* **Updates**
* Upgraded GPU Operator to v26.3.1 and Node Feature Discovery to v0.18.3
* Improved component health checks and added PodDisruptionBudget
availability controls
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
The `generate-changelog` job in `.github/workflows/tags.yaml` previously
checked out `main` and ran the AI agent there. The agent followed
`docs/agents/changelog.md`, which instructed it to compute the release
range with `git log <previous_version>..HEAD`. That works for minor
releases (cut from `main`), but it breaks for any patch release cut from
a `release-X.Y` branch — `HEAD-on-main` is a strict superset of the tag
and contains commits that were merged to `main` both before and after
the tag.
The v1.3.1 changelog generated by this workflow (#2480) demonstrated the
failure mode: 8 PRs that were merged to `main` but never shipped in
v1.3.1, 6 backport PRs that landed on `release-1.3` *after* v1.3.1 was
tagged, both originals and their backports as separate entries, a
hallucinated 2024 PR (#435), and the `cozystack-ci` bot in the
contributors list. The corrected v1.3.1 changelog is in #2480.
This PR fixes the root cause and tightens the agent guardrails:
* **`tags.yaml`** — check out the release tag commit (`ref: ${{
steps.tag.outputs.tag }}`) instead of `main`, so `HEAD == release
commit` and `git log v<prev>..HEAD` corresponds to what the tag actually
contains. The "Create changelog branch" step still creates the PR branch
from `origin/main`, so PRs continue to merge cleanly.
* **`tags.yaml`** — the AI prompt now states explicitly that `HEAD` is
the release commit and that the upper bound of the range is the new tag,
never `main`.
* **`docs/agents/changelog.md`** — every example replaces `..HEAD` with
`..v<new_version>`, so the instruction is unambiguous regardless of
which branch is checked out.
* **`docs/agents/changelog.md`** — hard rule: backport PRs MUST be
combined with the original into a single entry (`#1606, backport
#1609`), never listed as a second entry. Documented edge case: if the
original isn't in the range, drop the entry entirely (it shipped in a
previous release).
* **`docs/agents/changelog.md`** — forbid using the brief description
verbatim as the detailed description (the failure was `* **fix(foo):
X**: fix(foo): X (...)`), and forbid inventing entries for PRs outside
the release range.
* **`docs/agents/changelog.md`** — filter bot/CI accounts (`app/*`,
`*[bot]`, `cozystack-ci`, `github-actions`, `dependabot`, `renovate`)
out of the human Contributors list. Bot attribution on individual
entries is still valid.
The v1.3.1 changelog correction itself lives in #2480 (separate scope).
### Release note
```release-note
fix(ci): tags.yaml now generates the release changelog from the tag commit instead of `main`, fixing patch-release changelogs (e.g. v1.3.1) that previously included unrelated commits from `main`. The AI prompt and `docs/agents/changelog.md` were tightened to forbid duplicate backport entries, title-as-description, fabricated PRs, and bot accounts in the Contributors list.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Changelog generation now checks out the release tag commit and
verifies whether the release changelog already exists on main before
generating.
* Updated prompts and tooling to treat the release tag as the upper
bound and avoid using HEAD when computing diffs.
* Enforced combining original and backport PRs into single changelog
entries with explicit “backport #…”.
* **Documentation**
* Strengthened authoring rules (distinct brief/detail, forbid invented
entries, PRs must fall within tag-to-tag ranges).
* Updated examples and guidance to use tag-to-tag comparisons and
improved contributor extraction to use bold @user attribution while
excluding automation accounts and the current release file.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Free-floating 'NVIDIA recommends' claims are flagged as unverifiable
in review. Add the doc URL inline so future readers can verify the
permission model decision.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Pin a concrete reference set so future readers can tell whether the
guide is still accurate against the cluster they're running:
KubeVirt main nightly 20260429_74d7c52588, this PR's vgpu variant,
NVIDIA vGPU 20.0 host driver 595.58.02, GRID guest driver 595.58.03.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
NVIDIA's gpu-driver-container repository owns the build path —
keeping a parallel set of build instructions in our docs duplicates
upstream documentation and goes stale every time the upstream
Dockerfile or build args change. Replace the docker build / docker
push snippet with a paragraph pointing at the upstream repo and the
NVIDIA Licensing Portal as the sources of truth, keep only the
items that are Cozystack-specific (private-registry / EULA note).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Apple Silicon / arm64 build hosts produce arm64 images native; GPU
nodes are amd64 and the kubelet pull fails with 'no matching
manifest' when the image is built without --platform. Document the
flag and add a one-line docker login note for private registries.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
NVIDIA's License System User Guide recommends 744 on the
ClientConfigToken file so nvidia-gridd (which does not necessarily
run as the file owner) can read it. Previous example used 0600,
which can silently break license activation when nvidia-gridd
runs as a service user.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Mirroring the corresponding cleanup in cozystack/website#467 review:
* KubeVirt v1.9.0 ETA: drop the 'July 2026' date — KubeVirt does not
publish hard release dates and the in-page wording was inconsistent
across the file. Replace with 'Targeted at the next minor release
(v1.9.0); track the PR for the actual release tag', matching the
website doc.
* Kernel-headers timing: 'The build downloads kernel headers at
container start time' conflated docker build with pod start; the
Dockerfile's entrypoint downloads them at runtime. Reword.
* Replace guillemets with ASCII quotes for consistency with the rest
of the file and the upstream chart docs.
* nvidia-smi -q license check: drop the unnecessary -A 1 — License
Status value is on the same line as the field name.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
cozystack-ci[bot]
