Switch from non-existent latest digest to the available
feat-ui-overhaul image in the official cozystack registry.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Replace max_over_time([10m]) == 1 with an instant vector selector
and a top-level for: 10m clause. The previous expression fired as
long as the metric was non-running at any point in the last 10 minutes;
the new expression fires only when the condition is sustained continuously
for the full 10-minute window, matching the alert name semantics.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
- Use grep -Fxq instead of -w when intersecting with all_apps so apps
with hyphenated names cannot prefix-match each other.
- Collapse build_owners_index and build_reverse_deps into single yq
invocations with -N. Output is identical to the per-file loop and
about 30x faster on the current 91 PackageSource files.
- Use paste -sd ' ' - on the two full-suite emit paths for consistency
with the post-walk join at line 147.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Three changes that compose into a single E2E policy:
1. **Test-impact analysis selector** (`hack/select-e2e.sh`).
Reads `packages/core/platform/sources/*.yaml` (the PackageSource
dependency graph also rendered by `cozypkg dot`) and emits the bats
files affected by a PR diff. Library/core/networking/CI changes,
shared `hack/*.sh` helpers, the install bats, and the
`pull-requests` / `release-e2e` workflows trigger the full suite.
Doc/dashboard/`*.md` diffs select nothing. Per-app bats edits
select only that app. Anything inside `packages/` with no graph
entry, or a system source whose graph has no `*-application`
descendants, falls back to the full suite (conservative).
2. **Default-on TIA in `pull-requests.yaml`** with `full-e2e`
opt-out. Drops the previous `e2e-tia` opt-in shape — the
selector's behaviour has been validated, so making it the default
gives PR runtime savings out of the box. PRs that need the full
suite (e.g. cross-cutting refactors that the selector can't
reason about) carry the `full-e2e` label and get the unchanged
pre-TIA flow. Docs-only PRs (selector emits nothing) skip the
`Run E2E tests` step entirely.
3. **`release-e2e.yaml`** runs the full suite on every tag push
(`v*`). Independent of `tags.yaml`'s release publish flow — a
failure here is an alarm, not a release blocker. Closes the
coverage gap left by making PR runs use TIA: the full suite is
now exercised against shipped code at tag-cut time.
Bug fixes vs. the original `feat/select-e2e-script` draft:
- `dashboard/` -> `dashboards/` (path is plural in this repo).
- `hack/e2e-apps/<name>.sh` and `hack/<name>.bats` were silently
ignored; they now trigger the full suite (test infrastructure
affecting all apps).
- A path matching the graph but yielding zero `*-application`
descendants now falls back to the full suite instead of silently
emitting nothing (system-only changes are no longer silently
skipped).
- Per-app bats edits are matched before the full-suite pattern, so
editing one bats file no longer escalates to the full suite.
- Removed the dead `skip_pattern` variable (PCRE syntax that
`grep -qE` doesn't support; it was never matched).
Tests: `hack/select-e2e_test.bats` covers all 11 cases (the original
6 plus 5 for the bug fixes above).
Workflow inputs are passed via env vars rather than direct `${{ }}`
expansion into shell commands, per workflow injection guidance.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
Sweep across `hack/*.bats` and `hack/*.sh` replacing fragile
`sleep N` / `timeout N until …; do sleep …; done` patterns with
`kubectl wait --for=condition=…` plus a small existence backstop:
until kubectl get <resource> >/dev/null 2>&1; do sleep 2; done
kubectl wait <resource> --for=condition=Ready --timeout=…
Existence backstop added to 33 `kubectl wait` sites across 12 bats
files (redis, kafka, openbao, harbor, mongodb, qdrant, vminstance,
clickhouse, mariadb, postgres, bucket, install). Without it,
`kubectl wait` against a not-yet-created resource sits silently for
the full timeout; with it, the test fails fast with a clear
"resource never appeared" error.
OpenAPI tests' fixed `sleep 5` after starting `kubectl proxy` is now
a port-readiness wait. The 3 remaining genuine sleeps (LINSTOR node
membership propagation, MetalLB end-to-end HTTP path, HR-count
emission heuristic) are annotated with TODO + rationale so a future
reader knows they are intentional.
Same green-path semantics; failures now surface in seconds instead
of after a 5-minute timeout.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
Three fixes for the vminstance bats, surfaced once the 3x retry was
dropped from `Run E2E tests`:
1. Disk delete-recreate race. The teardown used
`kubectl delete vmdisk … || true` and immediately created a new
disk; sometimes the new disk reconciled before the old finalizer
completed, leaving a duplicate PVC and the new VM stuck. Drop the
`|| true`, block until the old disk is gone, then create.
2. VM IP wait timeout 20s -> 120s. Under nested-virt + concurrent
load, a fresh VM sometimes takes 60-90s to acquire its IP via
DHCP. The 20s bound was tripping deterministically.
3. VM ready timeout 20s -> 60s. Same root cause; the readiness
probe needs the IP to be reachable, so the lower bound has to
grow alongside the IP wait.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
`hack/cozyreport.sh` now collects materially more signal so a triage
artifact can show what is broken rather than just dumping pod logs:
- Flux controller logs (helm-controller, source-controller,
notification-controller, kustomize-controller, last 2000 lines)
- Flux source resources (HelmRepository, OCIRepository, GitRepository,
ExternalArtifact)
- Decoded Helm storage secrets for non-Ready HRs
(`base64 -d | base64 -d | gzip -d`)
- Cluster events (all + warning-only filter)
- cert-manager Certificate / CertificateRequest / Order / Challenge
resources + cert-manager logs
- `cozystack-operator` deployment logs (current + previous)
- `Application` / `ApplicationDefinition` / `Tenant` resources
- Sandbox host context per node: df, free, ps, dmesg,
talosctl logs/dmesg/kubelet/containerd
New `hack/cozyreport-summary.sh` writes a `summary.txt` at the archive
root listing what is broken right now — first thing to read when
triaging an artifact.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
The chart used to ship `Namespace cozy-system` directly, which trips a
helm v3 chicken-and-egg whenever the install command uses
`--create-namespace`:
- WITH `--create-namespace`, helm pre-creates the namespace via plain
kubectl-create (no helm meta annotations); the chart's own Namespace
apply then fails with `already exists`.
- WITHOUT `--create-namespace`, helm fails immediately because it
cannot write its release-secret to a non-existent namespace.
The bug was hidden by the 3x retry on the install step in
`.github/workflows/pull-requests.yaml`: attempt 1 fails with the
conflict, attempt 2 sees the failed release and takes the upgrade code
path (patch-merge instead of strict-create). Surfaces cleanly when the
retry is dropped; reproducible on every cold install.
Fix has two pieces:
1. Drop the chart-side `Namespace cozy-system` resource. Install
commands now use the standard `helm upgrade --install --namespace
cozy-system --create-namespace` (matches kube-prometheus-stack /
argo-cd / cert-manager).
2. Add a pre-install/pre-upgrade Job hook (`cozy-system-labeler`) that
patches `cozystack.io/system=true` and
`pod-security.kubernetes.io/enforce=privileged` onto the namespace
so the cozystack-operator pod (hostNetwork=true) is admitted.
The hook runs in `kube-system`, not `cozy-system`. Reason: the operator
pod needs hostNetwork=true (no CNI yet at install time), which violates
PodSecurity baseline. cozy-system therefore needs enforce=privileged
*before* the operator pod is admitted. A labeler pod inside cozy-system
would itself need hostNetwork=true and hit the same admission denial:
pods "cozy-system-labeler-..." is forbidden: violates PodSecurity
"baseline:latest": host namespaces (hostNetwork=true)
Job admission denial means no pod is ever created, .status.failed never
increments and backoffLimit never trips. helm's --wait waits for
Complete=True forever and surfaces only the generic "timed out waiting
for the condition" at its budget.
`kube-system` is PSA-exempt on Talos
(`defaults.exemptions.namespaces` in the apiserver
PodSecurityConfiguration) and unlabeled on vanilla kubeadm/kind/k3s, so
hostNetwork pods are admitted there. The hook's ServiceAccount is bound
to a ClusterRole scoped via `resourceNames: ["cozy-system"]` to the one
target namespace, keeping the cluster permission grant minimal.
`hook-delete-policy: before-hook-creation,hook-succeeded` ensures the
SA, RBAC, and Job are cleaned up after a successful install/upgrade.
Verified end-to-end in CI on the parallel consolidation branch
feat/e2e-optimization (PR #2500): cold install passes admission, the
operator pod comes up, all downstream E2E tests pass.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
**What this PR does**
Adds configurable kubelet resource reservations and eviction thresholds
for tenant cluster worker nodes. Without explicit reservations and
cgroup enforcement, the scheduler treats nearly all node memory and CPU
as allocatable. Under pressure, the OOM killer can target kubelet itself
because no cgroup boundaries protect it, leading to "Kubelet stopped
posting node status" failures and unrecoverable NotReady nodes.
This PR auto-computes per-node-group kubelet reservations from the
effective node capacity (instanceType or explicit `resources`):
- **Memory**: `systemReservedMemory` and `kubeReservedMemory` each
default to 5% of effective memory, clamped 256Mi-1Gi
- **CPU**: `systemReservedCpu` and `kubeReservedCpu` each default to 5%
of effective CPU, clamped 50m-500m
- **Eviction**: configurable `evictionHardMemory` (default 7%) and
`evictionSoftMemory` (default 10%) with validation that hard < soft
- **Disk eviction**: `nodefs.available`, `imagefs.available`, and
`nodefs.inodesFree` thresholds are preserved alongside memory signals
- **Capacity annotations**: cluster-autoscaler annotations now report
allocatable memory and CPU (total minus all reservations and eviction
threshold) instead of raw totals
- **Cgroup enforcement**: `enforce-node-allocatable=pods` configures
advisory-only reservations — the scheduler accounts for reserved
resources when computing allocatable capacity, but no cgroup boundaries
are enforced for system or kube reserved resources
- **Validation**: format checks, zero-value rejection, eviction ordering
(hard < soft), same-unit-type requirement for eviction thresholds, and
total reservation sanity check against effective node capacity
All reservation fields are optional and overridable per node group via
`.nodeGroups[].kubelet`.
**Default rendered kubeletExtraArgs:**
```yaml
kubeletExtraArgs:
system-reserved: "memory=256Mi,cpu=50m"
kube-reserved: "memory=256Mi,cpu=50m"
eviction-hard: "memory.available<7%,nodefs.available<10%,imagefs.available<15%,nodefs.inodesFree<5%"
eviction-soft: "memory.available<10%,nodefs.available<15%,imagefs.available<20%"
eviction-soft-grace-period: "memory.available=1m30s,nodefs.available=1m30s,imagefs.available=1m30s"
eviction-minimum-reclaim: "memory.available=256Mi"
enforce-node-allocatable: "pods"
```
**New values schema:**
```yaml
nodeGroups:
md0:
kubelet:
systemReservedMemory: "" # auto-computed (5% of effective memory, 256Mi-1Gi)
kubeReservedMemory: "" # auto-computed (5% of effective memory, 256Mi-1Gi)
systemReservedCpu: "" # auto-computed (5% of effective CPU, 50m-500m)
kubeReservedCpu: "" # auto-computed (5% of effective CPU, 50m-500m)
evictionHardMemory: "7%" # absolute (e.g. 200Mi) or percentage
evictionSoftMemory: "10%" # must be > evictionHardMemory, same unit type
```
**Release note:**
```
feat(kubernetes): add per-node-group kubelet resource reservations with auto-computed system/kube reserved memory (5% of effective memory, clamped 256Mi-1Gi) and CPU (5% of effective CPU, clamped 50m-500m), configurable memory eviction thresholds, preserved disk eviction signals (nodefs, imagefs, inodesFree), pods-only cgroup enforcement, and reservation-aware allocatable capacity annotations for cluster-autoscaler.
```
## What this PR does
Bumps the vendored `etcd-operator` chart from `v0.4.2` to `v0.4.3` and
wires the new backup machinery through the cozystack `Etcd` app API.
Upstream v0.4.3 ships two new CRDs — `EtcdBackup` (one-shot snapshot)
and `EtcdBackupSchedule` (periodic snapshots) — along with the
`EtcdCluster.spec.bootstrap` restore field, a new `OPERATOR_IMAGE` env
on the operator deployment, and a replacement of the deprecated
`gcr.io/kubebuilder/kube-rbac-proxy` image. The chart portion of this PR
is a clean re-vendor via `make update` — no local patches.
On top of the bump, `packages/extra/etcd` exposes a new optional
`backup.*` values block. When `backup.enabled=true`, the chart renders
an `EtcdBackupSchedule` referencing the platform `EtcdCluster/etcd` and
a `Secret/etcd-s3-creds` with the S3 credentials expected by the
backup-agent. The values field names (`destinationPath`, `endpointURL`,
`s3AccessKey`, `s3SecretKey`, `schedule`) mirror the existing `postgres`
backup pattern for consistency. The chart `fail`s at template render if
any of the four required S3 fields are empty or if `destinationPath` is
malformed (missing `s3://` prefix, `//` segments, no bucket).
Test coverage: 15 `helm-unittest` cases across
`tests/backup-secret_test.yaml` and
`tests/etcd-backup-schedule_test.yaml` (wired into `make test`, picked
up by `hack/helm-unit-tests.sh`), plus three bats cases in
`hack/e2e-apps/etcd.bats` covering the baseline `Etcd` app, the `backup:
{}` default-off path, and the full `backup.enabled=true` case that
verifies the `EtcdBackupSchedule` materializes and the operator
generates the downstream `CronJob`.
`packages/core/platform/sources/etcd-operator.yaml` now sets
`upgradeCRDs: CreateReplace` on the etcd-operator component, so Flux
applies the new CRDs on upgrade of the chart instead of silently
skipping them.
Depends on #2427.
### Scope deferred to follow-ups
- The one-shot `EtcdBackup` CRD is vendored but not exposed through this
chart — users can still hand-apply the CRD directly.
- `EtcdCluster.spec.bootstrap` (restore from an existing snapshot) is
likewise vendored but not exposed. Both gaps are called out in
`packages/extra/etcd/README.md`.
### Release note
```release-note
feat(etcd): update etcd-operator to v0.4.3 and expose S3 backup schedules on the Etcd app. Set `backup.enabled=true` along with `backup.s3AccessKey`, `backup.s3SecretKey`, `backup.destinationPath`, and `backup.endpointURL` to schedule periodic snapshots.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Scheduled S3 backups for etcd clusters (configurable cron, generated
S3 credentials) and restore/init from S3 or PVC via new backup
resources.
* **Tests**
* New end-to-end tests for etcd lifecycle and backup scenarios; added
Helm unit tests for backup templates and validations.
* **Documentation**
* Backup configuration guide and parameter reference added.
* **Chores**
* Operator chart version bump; RBAC, webhooks, and deployment wiring
updated; added chart test target.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Closes the 25 open Dependabot Go-ecosystem alerts on this repository (2
critical, 11 high, 12 moderate) by bumping the affected dependencies in
five Go modules to their latest stable versions. Splits the work into
six signed commits, one per logical block, so the diff in each commit
can be reviewed in isolation.
### Full advisory coverage
| # | Module | Severity | Package | Advisory | Title | Vulnerable |
Shipped |
|---|---|---|---|---|---|---|---|
| 29 | `go.mod` | critical | `google.golang.org/grpc` |
[GHSA-p77j-4mvh-x3m3](https://github.com/advisories/GHSA-p77j-4mvh-x3m3)
| gRPC-Go authorization bypass via missing leading slash in `:path` | `<
1.79.3` | `1.80.0` |
| 28 | `kubevirt-csi-driver/go.mod` | critical |
`google.golang.org/grpc` |
[GHSA-p77j-4mvh-x3m3](https://github.com/advisories/GHSA-p77j-4mvh-x3m3)
| gRPC-Go authorization bypass via missing leading slash in `:path` | `<
1.79.3` | `1.80.0` |
| 32 | `go.mod` | high | `go.opentelemetry.io/otel` |
[GHSA-mh2q-q3fh-2475](https://github.com/advisories/GHSA-mh2q-q3fh-2475)
| Multi-value `baggage` header extraction causes excessive allocations
(remote DoS amplification) | `>= 1.36.0, <= 1.40.0` | `1.43.0` |
| 31 | `go.mod` | high | `github.com/moby/spdystream` |
[GHSA-pc3f-x583-g7j2](https://github.com/advisories/GHSA-pc3f-x583-g7j2)
| SpdyStream: DoS on CRI | `<= 0.5.0` | `0.5.1` |
| 30 | `go.mod` | high | `go.opentelemetry.io/otel/sdk` |
[GHSA-hfvc-g4fc-pqhx](https://github.com/advisories/GHSA-hfvc-g4fc-pqhx)
| BSD `kenv` command not using absolute path enables PATH hijacking |
`>= 1.15.0, <= 1.42.0` | `1.43.0` |
| 27 | `go.mod` | high | `go.opentelemetry.io/otel/sdk` |
[GHSA-9h8m-3fm2-qjrq](https://github.com/advisories/GHSA-9h8m-3fm2-qjrq)
| OpenTelemetry Go SDK vulnerable to arbitrary code execution via PATH
hijacking | `>= 1.21.0, < 1.40.0` | `1.43.0` |
| 26 | `kubevirt-csi-driver/go.mod` | high |
`github.com/opencontainers/runc` |
[GHSA-cgrx-mc8f-2prm](https://github.com/advisories/GHSA-cgrx-mc8f-2prm)
| runc container escape and DoS due to arbitrary write gadgets and
procfs write redirects | `<= 1.2.7` | `1.4.2` |
| 25 | `kubevirt-csi-driver/go.mod` | high |
`github.com/opencontainers/runc` |
[GHSA-qw9x-cqr3-wc7r](https://github.com/advisories/GHSA-qw9x-cqr3-wc7r)
| runc container escape with malicious config due to `/dev/console`
mount and related races | `>= 1.0.0-rc3, <= 1.2.7` | `1.4.2` |
| 24 | `kubevirt-csi-driver/go.mod` | high |
`github.com/opencontainers/runc` |
[GHSA-9493-h29p-rfm2](https://github.com/advisories/GHSA-9493-h29p-rfm2)
| runc container escape via "masked path" abuse due to mount race
conditions | `<= 1.2.7` | `1.4.2` |
| 23 | `kubevirt-csi-driver/go.mod` | high | `golang.org/x/oauth2` |
[GHSA-6v2p-p543-phr9](https://github.com/advisories/GHSA-6v2p-p543-phr9)
| Improper Validation of Syntactic Correctness of Input | `< 0.27.0` |
`0.36.0` |
| 16 | `kubeovn-webhook/go.mod` | high | `golang.org/x/oauth2` |
[GHSA-6v2p-p543-phr9](https://github.com/advisories/GHSA-6v2p-p543-phr9)
| Improper Validation of Syntactic Correctness of Input | `< 0.27.0` |
`0.36.0` |
| 4 | `kubeovn-webhook/go.mod` | high | `golang.org/x/net` |
[GHSA-4374-p667-p6c8](https://github.com/advisories/GHSA-4374-p667-p6c8)
| HTTP/2 rapid reset can cause excessive work in `net/http` | `< 0.17.0`
| `0.53.0` |
| 3 | `kubeovn-webhook/go.mod` | high | `golang.org/x/net` |
[GHSA-vvpx-j8f3-3w6h](https://github.com/advisories/GHSA-vvpx-j8f3-3w6h)
| `golang.org/x/net` vulnerable to Uncontrolled Resource Consumption |
`< 0.7.0` | `0.53.0` |
| 22 | `kubevirt-csi-driver/go.mod` | medium | `golang.org/x/net` |
[GHSA-vvgc-356p-c3xw](https://github.com/advisories/GHSA-vvgc-356p-c3xw)
| `golang.org/x/net` vulnerable to Cross-site Scripting | `< 0.38.0` |
`0.53.0` |
| 21 | `kubevirt-csi-driver/go.mod` | medium | `golang.org/x/net` |
[GHSA-qxp5-gwg8-xv66](https://github.com/advisories/GHSA-qxp5-gwg8-xv66)
| HTTP Proxy bypass using IPv6 Zone IDs | `< 0.36.0` | `0.53.0` |
| 20 | `kubevirt-csi-driver/go.mod` | medium |
`github.com/opencontainers/runc` |
[GHSA-jfvp-7x6p-h2pv](https://github.com/advisories/GHSA-jfvp-7x6p-h2pv)
| runc can be confused to create empty files/directories on the host |
`< 1.1.14` | `1.4.2` |
| 19 | `go.mod` | medium | `golang.org/x/crypto` |
[GHSA-f6x5-jh6r-wrfv](https://github.com/advisories/GHSA-f6x5-jh6r-wrfv)
| `golang.org/x/crypto/ssh/agent` vulnerable to panic if message is
malformed (OOB read) | `< 0.45.0` | `0.50.0` |
| 18 | `go.mod` | medium | `golang.org/x/crypto` |
[GHSA-j5w8-q4qc-rx2x](https://github.com/advisories/GHSA-j5w8-q4qc-rx2x)
| `golang.org/x/crypto/ssh` allows an attacker to cause unbounded memory
consumption | `< 0.45.0` | `0.50.0` |
| 14 | `kubeovn-webhook/go.mod` | medium | `golang.org/x/net` |
[GHSA-vvgc-356p-c3xw](https://github.com/advisories/GHSA-vvgc-356p-c3xw)
| `golang.org/x/net` vulnerable to Cross-site Scripting | `< 0.38.0` |
`0.53.0` |
| 11 | `kubeovn-webhook/go.mod` | medium | `golang.org/x/net` |
[GHSA-qxp5-gwg8-xv66](https://github.com/advisories/GHSA-qxp5-gwg8-xv66)
| HTTP Proxy bypass using IPv6 Zone IDs | `< 0.36.0` | `0.53.0` |
| 9 | `kubeovn-webhook/go.mod` | medium | `golang.org/x/net` |
[GHSA-xrjj-mj9h-534m](https://github.com/advisories/GHSA-xrjj-mj9h-534m)
| `golang.org/x/net/http2` vulnerable to possible excessive memory
growth | `< 0.4.0` | `0.53.0` |
| 8 | `kubeovn-webhook/go.mod` | medium | `golang.org/x/net` |
[GHSA-4v7x-pqxf-cx7m](https://github.com/advisories/GHSA-4v7x-pqxf-cx7m)
| `net/http`, `x/net/http2`: close connections when receiving too many
headers | `< 0.23.0` | `0.53.0` |
| 7 | `kubeovn-webhook/go.mod` | medium | `google.golang.org/protobuf` |
[GHSA-8r3f-844c-mc37](https://github.com/advisories/GHSA-8r3f-844c-mc37)
| `protojson.Unmarshal` infinite loop on certain forms of invalid JSON |
`< 1.33.0` | `1.36.11` |
| 6 | `kubeovn-webhook/go.mod` | medium | `golang.org/x/net` |
[GHSA-qppj-fm5r-hxr3](https://github.com/advisories/GHSA-qppj-fm5r-hxr3)
| HTTP/2 Stream Cancellation Attack | `< 0.17.0` | `0.53.0` |
| 5 | `kubeovn-webhook/go.mod` | medium | `golang.org/x/net` |
[GHSA-2wrh-6pvc-2jm9](https://github.com/advisories/GHSA-2wrh-6pvc-2jm9)
| Improper rendering of text nodes in `golang.org/x/net/html` | `<
0.13.0` | `0.53.0` |
Sorted by severity (critical → high → medium), then by alert number. The
`Module` column lists the manifest path inside the repo (`go.mod` is the
root); `Vulnerable` is what GitHub flagged in the original advisory;
`Shipped` is the version this PR pins to.
### Modules bumped
1. Root `go.mod` (`github.com/cozystack/cozystack`) — purely transitive
bumps for `grpc`, `otel`, `otel/sdk`, `x/crypto`, `spdystream`.
2. `api/apps/v1alpha1/go.mod` — bumps `k8s.io/apimachinery` and
`golang.org/x/net`/`x/text` for consistency with the root module's `go
1.26.2` directive.
3. `packages/apps/kubernetes/images/kubevirt-csi-driver/go.mod` — bumps
the upstream `kubevirt.io/csi-driver` pin to its latest commit, raises
the `k8s.io/*` replace block from `v0.31.4` to `v0.36.0`, bumps `runc`
to `v1.4.2`, and adapts `main.go` to the upstream's new builder-style
API for `service.NewKubevirtCSIDriver` (the constructor was split into
`WithIdentityService` / `WithControllerService` / `WithNodeService`).
4. `packages/system/kubeovn-webhook/images/kubeovn-webhook/go.mod` —
direct bumps to `golang.org/x/net`, `golang.org/x/oauth2`,
`google.golang.org/protobuf`.
5. `packages/system/dashboard/images/token-proxy/go.mod` — bumps `jwx`,
`httprc`, `x/crypto`, `x/sys` for consistency with the root module's
directive.
All five modules now declare `go 1.26.2`. The three vendored-image
Dockerfiles (`kubevirt-csi-driver`, `kubeovn-webhook`, `token-proxy`)
move to `golang:1.26{,-alpine}`.
### First-party Dockerfile bumps
The root `go.mod` directive jump to `1.26.2` means every Dockerfile that
copies the root `go.mod` and builds a cozystack-owned binary needs a
matching base image, otherwise the in-container toolchain has to pull
`go 1.26` on the fly via `GOTOOLCHAIN=auto` at every build. Eight images
are bumped from `golang:1.25-alpine` to `golang:1.26-alpine`:
`cozystack-operator`, `cozystack-api`, `cozystack-controller`,
`backup-controller`, `backupstrategy-controller`, `flux-plunger`,
`kubeovn-plunger`, `lineage-controller-webhook`.
Vendored Dockerfiles (`kubevirt-cloud-provider`, `redis-operator`,
`keycloak-operator`, `linstor-csi`, `multus-cni`, `objectstorage`,
`s3manager`, `kamaji`) are intentionally untouched: each one fetches its
source from upstream at build time and pins its `golang:` base image to
whatever the upstream commit was tested against.
### Why bump past the advisory minimums
Bumping each affected dependency to its latest stable rather than the
lowest patched version means we pick up subsequent (non-CVE) bug fixes
for free and reduce the number of follow-up Dependabot bumps the next
reviewer has to triage. Where this required a cascading bump (e.g.
`runc` v1.4.2 only compiles together with `mount-utils` v0.32+, which
dropped the deprecated `libcontainer/userns` import path), the cascade
was followed through.
### Build verification
`go build ./...` and `go vet ./...` are clean across all five modules
locally. Two `go vet` warnings in the root module
(`pkg/lineage/lineage_test.go:76` unkeyed struct fields,
`cmd/cozypkg/cmd/add.go:521` non-constant format string) and three
`randfill.Continue` test panics in `pkg/apis/{apps,core}/install` and
`pkg/apiserver` also reproduce on `main` and are out of scope for this
PR.
### Out of scope
The `k8s.io/*` libraries in `kubeovn-webhook/go.mod` are still pinned at
`v0.26.0`; they are not flagged by Dependabot and bumping them would be
a separate, much larger maintenance change.
### Release note
```release-note
fix(security): bump Go dependencies in root, api/apps/v1alpha1, kubevirt-csi-driver, kubeovn-webhook, and token-proxy modules to clear all open Dependabot CVE alerts
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Upgraded Go toolchain across components and build stages to 1.26.x.
* Updated numerous dependencies for compatibility, stability and
security (Kubernetes APIs, gRPC, OpenTelemetry, golang.org/x/*, and
related libs).
* Several containers now run as a non-root user (UID/GID 65532) to
improve runtime security.
* **Refactor**
* CSI driver initialization refactored to a builder-style pattern
enabling conditional service wiring.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Replace Dosu's stale-management and PR-size labeling with native GitHub
Actions ahead of disabling the Dosu app on this repo, and migrate the
six `size:*` labels to the canonical `size/*` namespace while we're
rewriting the sizing path.
**Background**: Dosu (the SaaS) currently writes `lifecycle/stale`
lifecycle and `size:*` labels via its bot. Audit of `.github/workflows/`
(full grep) confirmed zero hard dependencies on Dosu's outputs —
`backport*` is human-applied, `release` is auto-applied by `tags.yaml`
itself, and nothing else consumes Dosu-applied labels in any workflow.
### `.github/workflows/stale.yaml` (new)
Daily cron at 04:37 UTC plus manual dispatch. Uses `actions/stale@v10`:
- Stale after 60 days idle: applies `lifecycle/stale`.
- Closes 14 days later, applying `lifecycle/rotten`.
- `remove-stale-when-updated: true` — activity un-stales automatically.
- 100 ops/run.
Issue and PR exempt lists differ on purpose: an accepted issue
(`triage/accepted`) is a known long-tail task, not stale; a held PR
(`do-not-merge/hold`) is paused intentionally.
Priority/security/regression labels protect issues someone explicitly
decided to keep visible.
### `.github/workflows/pr-size.yaml` (new)
Triggers on `opened`, `synchronize`, `reopened` (`pull_request_target`).
Inline `actions/github-script@v7` — no third-party labeler dependency.
Diff-line count with thresholds matching the descriptions in
`.github/labels.yml`:
| label | lines |
|---|---|
| `size/XS` | 0–9 |
| `size/S` | 10–29 |
| `size/M` | 30–99 |
| `size/L` | 100–499 |
| `size/XL` | 500–999 |
| `size/XXL` | 1000+ |
Inline ignore patterns skip:
- vendored Go deps (`vendor/**`)
- kubebuilder/openapi inline generated files
(`zz_generated.{deepcopy,conversion,defaults,openapi}.go`)
- whole code-generated trees: any path under `generated/`, plus
`*.pb.go`
- vendored Helm charts (`packages/system/*/charts/**`)
- lockfiles (`go.sum`, `*.lock`, `*.lockb`)
Concurrency group keyed on PR number with `cancel-in-progress: true`
(latest push wins). `removeLabel` is wrapped in `try/catch` tolerating
404 for races between event payload and execution.
### `size:` → `size/` namespace migration
The legacy `size:X` (colon) form was an artefact of the previous sizing
bot; with that bot being replaced and every other label namespace in the
repo using slashes (`kind/`, `area/`, `priority/`, `triage/`,
`lifecycle/`, `do-not-merge/`, `security/`), `size/` is the consistent
form. Migration via aliases in `labels.yml` — IDs preserved, every
existing labelled PR follows the rename automatically.
The new `pr-size.yaml` emits `size/*` from day 1 and accepts both
prefixes when looking up the previous size label, to handle the brief
window between merge and the next labels-sync run.
### Out of scope (intentional)
- **`lgtm` replacement** — deferred. 33/33 of `lgtm` events in the last
100 PRs come from Dosu; humans never apply it manually. Whether
reviewers actually rely on it is unclear — observe for 7 days
post-Dosu-disable before adding a `pull_request_review`-on-approve
workflow.
- **LLM-based issue auto-classification** (`kind/bug`/`kind/feature` on
inbound issues) — accepted loss; manual triage replaces.
- **Cleanup of `wontfix`/`invalid`** in `labels.yml` — separate cleanup
PR.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
- Pull requests now automatically receive size labels (XS, S, M, L, XL,
XXL) based on lines changed.
- Issues and pull requests are automatically marked as stale after 60
days of inactivity and closed 14 days later if still inactive.
* **Chores**
- PR size label namespace updated to a slash-based format (size/...)
while preserving the old names as aliases for backward compatibility.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
The `Bootstrap Talos cluster` step in `hack/e2e-prepare-cluster.bats`
wraps `talosctl bootstrap` in `timeout 10 sh -ec 'until talosctl
bootstrap ...; do sleep 1; done'`. On a freshly booted Talos node NTP
has not yet converged, so `talosctl bootstrap` returns
`FailedPrecondition: time is not in sync yet` after ~5-6s. With the 10s
outer timeout the embedded `until/sleep 1` loop only completes a single
attempt before it is killed, and the E2E job fails during `Prepare
environment`.
On slower self-hosted runners NTP typically converges before the first
attempt. On fast ephemeral runners (Oracle VMs) it does not, producing a
consistent flake that only the outer workflow-level 3-attempt retry can
paper over.
Bump the outer timeout to 60s and the inter-attempt sleep to 2s. This
aligns with the other bootstrap gates in the same test (etcd members
check at 180s, drain RPC errors at 60s, node registration at 60s) and
gives NTP plenty of time to converge.
### Release note
```release-note
fix(tests): increase `Bootstrap Talos cluster` timeout in E2E from 10s to 60s to accommodate NTP convergence on fast ephemeral runners.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Enhanced cluster bootstrap process resilience by extending retry
mechanisms and adjusting timing intervals to better tolerate transient
failures during system initialization.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
The `filer` and `s3` blocks in the seaweedfs HelmRelease values omitted
the `replicas:` field, so user-supplied `.Values.filer.replicas` and
`.Values.s3.replicas` were silently ignored and the upstream chart
defaults took over. As a second-order effect, the `WorkloadMonitor`
resources for `filer` and `s3` (which read the same values) diverged
from the actual replica count and could trigger spurious alerts when
users set non-default values.
This PR mirrors the existing `master` / `volume` / `db` pattern and
interpolates `replicas:` into both blocks.
Fixes#2531
### Release note
```release-note
fix(seaweedfs): pass filer.replicas and s3.replicas to HelmRelease so user-supplied values take effect
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **Chores**
* SeaweedFS deployment configuration now supports configurable replica
counts for the filer and s3 components, providing flexible cluster
scaling through deployment parameters.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Bumps Talos Linux from v1.12.7 to v1.13.0. The v1.13 release line also
contains the kernel fix for CVE-2026-31431 ("Copy Fail").
Refreshes the bundled system extension digests (amdgpu, i915, drbd, zfs)
to their 1.13.0-aligned builds.
Generated via `make update` in `packages/core/talos`.
Stacked on top of #2545. Targets `fix/talos-1.12.7` so the diff shown
here is just the v1.12.7 → v1.13.0 delta.
References:
- https://copy.fail/
-
https://www.siderolabs.com/blog/exploit-fail-cve-2026-31431-copy-fail-barely-scratches-talos-linux
### Release note
```release-note
feat(talos): bump Talos Linux to v1.13.0
```
## What this PR does
Bumps Talos Linux from v1.12.6 to v1.12.7, which includes the kernel fix
for CVE-2026-31431 ("Copy Fail") — an unprivileged-to-root local
privilege escalation in the Linux kernel crypto subsystem
(`algif_aead`).
This also refreshes the bundled system extension digests (amd-ucode,
amdgpu, bnx2-bnx2x, i915, intel-ice-firmware, qlogic-firmware, drbd,
zfs) to their 1.12.7-aligned builds.
Generated via `make update` in `packages/core/talos`.
References:
- https://copy.fail/
-
https://www.siderolabs.com/blog/exploit-fail-cve-2026-31431-copy-fail-barely-scratches-talos-linux
### Release note
```release-note
fix(talos): bump Talos Linux to v1.12.7, picking up the kernel fix for CVE-2026-31431 ("Copy Fail")
```
Bumps Talos Linux from v1.12.7 to v1.13.0. The v1.13 release line also
contains the kernel fix for CVE-2026-31431 ("Copy Fail").
Refreshes the bundled system extension digests (amdgpu, i915, drbd, zfs)
to their 1.13.0-aligned builds.
Generated via `make update` in packages/core/talos.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Bumps Talos Linux from v1.12.6 to v1.12.7, which includes the kernel
fix for CVE-2026-31431 ("Copy Fail") — an unprivileged-to-root local
privilege escalation in the Linux kernel crypto subsystem (algif_aead).
Also refreshes the bundled system extension digests (amd-ucode, amdgpu,
bnx2-bnx2x, i915, intel-ice-firmware, qlogic-firmware, drbd, zfs) to
their 1.12.7-aligned builds.
Generated via `make update` in packages/core/talos.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Describe the in-place password rotation performed by the Percona operator
when upgrading to releases that scope the system users secret per CR.
Clarify that consumers of the credentials secret need a pod restart to
pick up the new password, custom user secrets are unaffected, and the
legacy namespace-shared secret is left orphaned and must be cleaned up
manually.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
## What this PR does
Adds a fifth piraeus-server patch that retries `drbdadm secondary` after
`mkfs` when DRBD reports the device as held open by an external process.
### Problem
On Talos (and any system with active block-device probing — udev,
multipathd, etc.) the kernel uevent for a freshly-promoted DRBD device
causes an external probe to `open()` the device for superblock
inspection. If LINSTOR satellite issues `drbdadm secondary` while the
probe still holds the fd, DRBD returns `(-12) Device is held open by
someone` and aborts initialization with `StorageException: Failed to
become secondary again after creating filesystem`.
The satellite never reports the final `UpToDate` event to the
controller, so the controller view stays in `SyncTarget(N%) /
Inconsistent / DELETING` permanently — even though the DRBD kernel state
on the satellite is `Established/UpToDate`. This blocks cleanup of
orphan PVs in `Released` phase and accumulates with every PVC creation.
Observed in production on Talos 1.10 / LINSTOR 1.33.2: 4 occurrences in
24 hours across 3 nodes for different PVCs. Manual recovery requires
`drbdadm down/up` on the Primary to force the satellite to re-report
state.
### Fix
New `secondaryAfterMkfs()` wrapper in `DrbdAdm.java`:
- 7 attempts of `drbdadm secondary` with exponential backoff (250 ms → 2
s, capped)
- Retry only triggers when stderr contains `Device is held open by
someone`; any other failure fails fast as before
- Existing `secondary()` contract is unchanged — only
`DrbdPrimary.close()` (the post-mkfs demotion path) is rerouted to the
new wrapper
The retry window covers ~10 s in the worst case, which comfortably
exceeds typical probe durations (probes release the fd within
milliseconds).
### Related upstream
-
[LINBIT/linstor-server#268](https://github.com/LINBIT/linstor-server/issues/268)
— open since 2022, identical symptom, no upstream fix
- [LINBIT/drbd#74](https://github.com/LINBIT/drbd/issues/74) — same
EBUSY pattern with multipathd holding the fd
### Validation
- `git apply *.diff` on a fresh `v1.33.2` checkout applies all five
patches cleanly in alphabetical order
- Java brace/paren/bracket balance verified on the patched file
- `helm template packages/system/linstor` renders without errors
### Release note
```release-note
fix(linstor): retry `drbdadm secondary` after mkfs on transient `Device is held open by someone` errors. Prevents orphan PVs in `Released` phase from accumulating when an external block-device probe (Talos block-controller, udev, multipathd) wins a race against the post-mkfs demotion. Fail-fast behavior preserved for any other secondary failure.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Implemented automatic retry logic for device secondary state
transitions after filesystem creation. System now gracefully handles
scenarios where devices are temporarily held open by other processes.
* **Documentation**
* Added documentation for the new device state transition patch.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Adds a per-satellite plunger handler that detects DRBD resources
stuck in `Primary` on the local node with no consumer and demotes
them to `Secondary`.
### Background
DRBD auto-promotes a resource to `Primary` during the provisioning
path (`mkfs.ext4` against a freshly created replica, for example),
but auto-demote sometimes fails to fire on close. The resource then
stays `Primary` on a node that has no consumer pod, and CSI on the
legitimate target node fails `NodePublishVolume` with:
```
failed to set source device readwrite: exit status 1
```
The only recovery is an explicit `drbdadm secondary` on the stuck
node — there is no upstream automation for it.
This was hit deterministically in Cozystack e2e CI on freshly
provisioned PVCs without any pod migration or node failure. Upstream
tracker: piraeusdatastore/piraeus-operator#942 (and the related
piraeusdatastore/linstor-csi#172).
### Implementation
In `plunger-satellite.sh` (already running as a privileged sidecar
on every linstor-satellite pod) add `drbd_fix_idle_primary`:
```jq
.[]?
| select(.role == "Primary")
| select(all(.devices[]?; .open == false))
| .name
```
For every match, run `drbdadm secondary <res>`.
### Why this is safe
- `open == false` on **every** device of the resource means no reader
or writer holds the block device — the demote cannot interrupt I/O.
- Both detection and action are local to the satellite, no peer
coordination needed.
- If a real consumer is about to arrive, DRBD auto-promote will redo
the promotion when CSI opens the device — so even a misfire is a
no-op.
- The jq filter was validated against a live multi-node cluster
(3 nodes, 22 legitimate Primaries, 0 false positives — every
legitimate Primary has at least one `open: true` volume).
### Release note
```release-note
fix(linstor): plunger now demotes DRBD resources stuck in Primary
without a consumer, fixing transient `failed to set source device
readwrite` mount errors during PVC provisioning.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Enhanced DRBD resource management with automatic remediation for idle
Primary resources. The system now detects resources lacking active
device connections and safely demotes them to Secondary state during
routine maintenance cycles. Robust error handling preserves existing
recovery procedures while improving overall system stability and
preventing potential resource conflicts.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
The `generate-changelog` job in `.github/workflows/tags.yaml` previously
checked out `main` and ran the AI agent there. The agent followed
`docs/agents/changelog.md`, which instructed it to compute the release
range with `git log <previous_version>..HEAD`. That works for minor
releases (cut from `main`), but it breaks for any patch release cut from
a `release-X.Y` branch — `HEAD-on-main` is a strict superset of the tag
and contains commits that were merged to `main` both before and after
the tag.
The v1.3.1 changelog generated by this workflow (#2480) demonstrated the
failure mode: 8 PRs that were merged to `main` but never shipped in
v1.3.1, 6 backport PRs that landed on `release-1.3` *after* v1.3.1 was
tagged, both originals and their backports as separate entries, a
hallucinated 2024 PR (#435), and the `cozystack-ci` bot in the
contributors list. The corrected v1.3.1 changelog is in #2480.
This PR fixes the root cause and tightens the agent guardrails:
* **`tags.yaml`** — check out the release tag commit (`ref: ${{
steps.tag.outputs.tag }}`) instead of `main`, so `HEAD == release
commit` and `git log v<prev>..HEAD` corresponds to what the tag actually
contains. The "Create changelog branch" step still creates the PR branch
from `origin/main`, so PRs continue to merge cleanly.
* **`tags.yaml`** — the AI prompt now states explicitly that `HEAD` is
the release commit and that the upper bound of the range is the new tag,
never `main`.
* **`docs/agents/changelog.md`** — every example replaces `..HEAD` with
`..v<new_version>`, so the instruction is unambiguous regardless of
which branch is checked out.
* **`docs/agents/changelog.md`** — hard rule: backport PRs MUST be
combined with the original into a single entry (`#1606, backport
#1609`), never listed as a second entry. Documented edge case: if the
original isn't in the range, drop the entry entirely (it shipped in a
previous release).
* **`docs/agents/changelog.md`** — forbid using the brief description
verbatim as the detailed description (the failure was `* **fix(foo):
X**: fix(foo): X (...)`), and forbid inventing entries for PRs outside
the release range.
* **`docs/agents/changelog.md`** — filter bot/CI accounts (`app/*`,
`*[bot]`, `cozystack-ci`, `github-actions`, `dependabot`, `renovate`)
out of the human Contributors list. Bot attribution on individual
entries is still valid.
The v1.3.1 changelog correction itself lives in #2480 (separate scope).
### Release note
```release-note
fix(ci): tags.yaml now generates the release changelog from the tag commit instead of `main`, fixing patch-release changelogs (e.g. v1.3.1) that previously included unrelated commits from `main`. The AI prompt and `docs/agents/changelog.md` were tightened to forbid duplicate backport entries, title-as-description, fabricated PRs, and bot accounts in the Contributors list.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Changelog generation now checks out the release tag commit and
verifies whether the release changelog already exists on main before
generating.
* Updated prompts and tooling to treat the release tag as the upper
bound and avoid using HEAD when computing diffs.
* Enforced combining original and backport PRs into single changelog
entries with explicit “backport #…”.
* **Documentation**
* Strengthened authoring rules (distinct brief/detail, forbid invented
entries, PRs must fall within tag-to-tag ranges).
* Updated examples and guidance to use tag-to-tag comparisons and
improved contributor extraction to use bold @user attribution while
excluding automation accounts and the current release file.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Prevents "already exists" errors on repeated hook runs
by deleting hook resources before creating new ones.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
rancher/kubectl:v1.29.10 lacks /bin/sh on Talos clusters,
causing the pre-upgrade hook Job to fail. Switch to
docker.io/clastix/kubectl:v1.32 which is already used
across other dashboard hooks and has /bin/sh available.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Build job in CI failed because the test relied on bats-specific
machinery the in-repo cozytest.sh runner does not provide:
* BATS_TEST_DIRNAME is undefined when the file is sourced as a plain
shell script (cozytest.sh's mode); switch to BATS_TEST_FILENAME
with a $0 fallback, matching hack/check-gpu-recording-rules.bats
(REPO_ROOT pattern).
* setup()/teardown() are bats lifecycle hooks; cozytest.sh runs each
@test in a fresh subshell with set -u and never invokes them. With
TMP only assigned in setup(), every test exited immediately on
'TMP: unbound variable'. Inline the mktemp + trap inside each
@test so both runners work the same.
Verified locally that bash hack/cozytest.sh hack/check-gpu-operator-variants.bats
now reports all six tests OK.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Free-floating 'NVIDIA recommends' claims are flagged as unverifiable
in review. Add the doc URL inline so future readers can verify the
permission model decision.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Pin a concrete reference set so future readers can tell whether the
guide is still accurate against the cluster they're running:
KubeVirt main nightly 20260429_74d7c52588, this PR's vgpu variant,
NVIDIA vGPU 20.0 host driver 595.58.02, GRID guest driver 595.58.03.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
NVIDIA's gpu-driver-container repository owns the build path —
keeping a parallel set of build instructions in our docs duplicates
upstream documentation and goes stale every time the upstream
Dockerfile or build args change. Replace the docker build / docker
push snippet with a paragraph pointing at the upstream repo and the
NVIDIA Licensing Portal as the sources of truth, keep only the
items that are Cozystack-specific (private-registry / EULA note).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Apple Silicon / arm64 build hosts produce arm64 images native; GPU
nodes are amd64 and the kubelet pull fails with 'no matching
manifest' when the image is built without --platform. Document the
flag and add a one-line docker login note for private registries.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
NVIDIA's License System User Guide recommends 744 on the
ClientConfigToken file so nvidia-gridd (which does not necessarily
run as the file owner) can read it. Previous example used 0600,
which can silently break license activation when nvidia-gridd
runs as a service user.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Mirroring the corresponding cleanup in cozystack/website#467 review:
* KubeVirt v1.9.0 ETA: drop the 'July 2026' date — KubeVirt does not
publish hard release dates and the in-page wording was inconsistent
across the file. Replace with 'Targeted at the next minor release
(v1.9.0); track the PR for the actual release tag', matching the
website doc.
* Kernel-headers timing: 'The build downloads kernel headers at
container start time' conflated docker build with pod start; the
Dockerfile's entrypoint downloads them at runtime. Reword.
* Replace guillemets with ASCII quotes for consistency with the rest
of the file and the upstream chart docs.
* nvidia-smi -q license check: drop the unnecessary -A 1 — License
Status value is on the same line as the field name.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Two technical inaccuracies caught during review:
* The claim 'gpus: expects an mdev resource and will not match a
pciHostDevices entry' is wrong. Upstream KubeVirt
(pkg/virt-launcher/virtwrap/device/hostdevice/gpu/hostdev.go) iterates
both the PCI pool (v1.PCIResourcePrefix) and the mdev pool
(v1.MDevResourcePrefix) and accepts whichever resolves. Reword to
describe gpus: vs hostDevices: as a style convention rather than a
hard technical constraint.
* siderolabs/extensions#461 was closed with stateReason=COMPLETED, not
not_planned. Quoting the closure as 'won't fix' is editorial
recasting that does not match the actual closing comment. Replace
with the substantive reason from rothgar's closing comment: NVIDIA
does not publicly distribute the vGPU guest driver, requires
Enterprise Portal access, and Sidero cannot support vGPU until
NVIDIA changes its licensing terms.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Adds hack/check-gpu-operator-variants.bats. The test renders both
package variants with helm template and asserts the resulting
ClusterPolicy spec for the values that this PR cares about:
* ccManager.enabled=false (pinned in values.yaml; upstream v26 default
was flipped to true)
* vgpuDeviceManager.enabled=false (same — and on Ada+/Blackwell the
DaemonSet would crashloop because /sys/class/mdev_bus is empty)
* sandboxWorkloads.defaultWorkload — vm-passthrough on the default
variant, vm-vgpu on the vgpu variant; without these the upstream
'container' default makes either variant a silent no-op on
unlabelled GPU nodes
* vgpuManager.enabled=true on the vgpu variant; driver / devicePlugin
enabled=false on both variants (proprietary host driver path is
out of scope for both passthrough and SR-IOV vGPU)
Negative-tested locally: deleting any of these settings makes the
corresponding test fail, so it actually guards regressions on the
next chart bump.
Also addresses last-pass review comments:
* values-passthrough.yaml: set sandboxWorkloads.defaultWorkload=
vm-passthrough so the default variant is symmetric with the vgpu
variant — neither relies on per-node labels for the variant to
activate. Without this the upstream default 'container' would
apply on unlabelled GPU nodes and vfio-manager would never start.
* values.yaml: rewrite the vgpuDeviceManager comment to lead with
the load-bearing reason (mdev path missing on Ada+/Blackwell)
instead of the secondary 'stays at zero pods on passthrough'
rationalisation.
* docs/gpu-vgpu.md: add a 'Migrating from chart v25.x' subsection
in the Licensing section pointing at the configMapName ->
secretName rename. Old key still works upstream but emits a
deprecation warning; PR body release note already calls this out
but the docs were missing the migration breadcrumb.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
* examples/README.md and examples/values-native-talos.yaml header
comment still pointed at values-talos.yaml after the rename to
values-passthrough.yaml. Update both to the new name.
* values-vgpu.yaml: drop the leftover vgpuDeviceManager.config block.
The chart only renders the config when vgpuDeviceManager.enabled is
true, and we keep it false on Ada+; the orphan config.default key
was scaffolding without effect.
* docs/gpu-vgpu.md and values.yaml comment: stop grouping B100 under
Ada Lovelace. B100 is Blackwell; the SR-IOV vGPU mechanism applies
to both architectures but they should be named correctly.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Addresses second review pass. Three blockers + one polish:
* values-vgpu.yaml: set sandboxWorkloads.defaultWorkload=vm-vgpu.
Without this the upstream default 'container' applies on any node
without the per-node nvidia.com/gpu.workload.config label, the
operator selects the container workload (driver/devicePlugin
DaemonSets), and those are disabled in this variant — so nothing
runs on the node and vgpu-manager never starts. The cluster-wide
default makes the variant functional out of the box; per-node
labels still override it for mixed clusters.
* docs/gpu-vgpu.md: imagePullSecrets is per-component on this chart
(vgpuManager.imagePullSecrets, driver.imagePullSecrets, …). The
Package CR example previously placed the field at gpu-operator: root
where it would be silently ignored at render. Move it under
vgpuManager: where the chart actually reads it.
* docs/gpu-vgpu.md, release note: mark the vgpu variant as
experimental and call out the missing profile-assignment loop
explicitly. Until an SR-IOV-aware controller writes
current_vgpu_type per VF (and re-applies after every reboot —
PCIe re-enumeration resets the value), the variant is not safe
for production. The docs now lead with that warning instead of
burying it in a caveat.
* Rename packages/system/gpu-operator/values-talos.yaml to
values-passthrough.yaml. The overrides (sandboxWorkloads.enabled,
driver and devicePlugin disabled) apply to any host OS using the
vfio-pci passthrough path — not Talos specifically. Update
PackageSource ref accordingly. Comment in the file documents the
rename so future readers understand the prior name.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Addresses follow-up review of the vGPU variant + chart bump.
* values.yaml: pin ccManager.enabled=false and vgpuDeviceManager.enabled=false.
Upstream chart v26.x flipped both to true. ccManager auto-enables CC
mode on Hopper-class hardware after upgrade — a non-trivial behavior
change that should be opt-in. vgpuDeviceManager walks
/sys/class/mdev_bus/, which does not exist on Ada+ (L4/L40/L40S/B100
with vGPU 17/20). Pinning both off keeps the default footprint as it
was on v25.x; users explicitly opt in.
* hack/dcgm-default-counters.csv: refresh from dcgm-exporter tag
4.5.1-4.8.0 (matches the image bumped earlier in this PR). The
previous snapshot was pinned to 4.1.1-4.0.4. Adds DCGM_FI_DEV_FB_RESERVED
and other fields introduced upstream. Header comment updated.
hack/check-gpu-recording-rules.bats passes against the refreshed
fixture.
* docs/gpu-vgpu.md (Licensing section):
- Drop the wrong recommendation to wire the licensing Secret through
gpu-operator's driver.licensingConfig.secretName. That mounts into
the host driver pod, where SR-IOV vGPU does not consume a license.
Document guest-side delivery (cloud-init write_files for
/etc/nvidia/ClientConfigToken/ and /etc/nvidia/gridd.conf) end to
end.
- Correct FeatureType comment. Per the NVIDIA Virtual GPU Software
Licensing User Guide, FeatureType=0 is the unlicensed state, not
auto-detect; FeatureType=1 is the value that asks the driver to
auto-select the license type from the configured vGPU profile.
Switch the example to FeatureType=1 and list all four valid values
with their meaning.
- Add nvidia-smi verification snippet inside the guest.
* docs/gpu-vgpu.md (KubeVirt section): note that L40S SR-IOV VFs share
the PF device ID (10de:26b9), so a single pciVendorSelector matches
both. Recommend lspci -nn -d 10de: as a sanity check on other
generations where PF/VF IDs may differ.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Rewrites docs/gpu-vgpu.md to match the actual NVIDIA driver model on
current data-centre GPUs. Key changes:
- Drop the implicit assumption that vGPU uses mediated devices. Ada
Lovelace and newer (L4, L40, L40S, B100) on the vGPU 17/20 branch
use SR-IOV with per-VF NVIDIA sysfs (current_vgpu_type); KubeVirt
advertises them via permittedHostDevices.pciHostDevices.
- Document KubeVirt PR #16890 as a hard dependency for the SR-IOV
path. The PR is in main since 2026-04-10 and will ship in v1.9.0
(ETA July 2026); released tags v1.6.x/1.7.x/1.8.x do not include
it and backports are not planned.
- Replace the legacy NLS configuration block (ServerAddress=,
ServerPort=7070, FeatureType=1) with the current DLS flow:
ClientConfigToken file in /etc/nvidia/ClientConfigToken/, gridd.conf
with FeatureType=0 (auto-detect).
- Switch the build instructions from the archived
gitlab.com/nvidia/container-images/driver to the upstream
github.com/NVIDIA/gpu-driver-container.
- Switch sample VirtualMachine spec from gpus: (mdev resource) to
hostDevices: (PCI resource).
- Note explicitly that Talos is not recommended for the vGPU variant
— the proprietary .run cannot be redistributed in a Talos system
extension, and siderolabs/extensions#461 was closed won't-fix.
Passthrough on Talos is unaffected.
Rename packages/system/gpu-operator/values-talos-vgpu.yaml to
values-vgpu.yaml — the overrides (sandbox workloads, pod-side driver
and device plugin disabled) are not Talos-specific, they apply to
any VM-only GPU node. Update PackageSource accordingly.
Default vgpuDeviceManager.enabled: false. The DaemonSet walks
/sys/class/mdev_bus/, which does not exist on Ada+ — keeping it on
by default would crashloop modern hardware. Operators of older GPUs
(Pascal/Volta/Turing/Ampere) can flip the flag back on.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Pull latest patch release: GDRCopy v2.5.2, Kata Sandbox Device Plugin
v0.0.3, multi-arch digest fix for KubeVirt GPU Device Plugin (no longer
amd64-only), and OS release name detection fix. NFD remains v0.18.3.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
ConfigMap-based licensing configuration (configMapName) is deprecated
in GPU Operator v26.x. Switch to Secret-based configuration (secretName)
as recommended by the updated CRD schema.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Replace simplified Containerfile with NVIDIA's Makefile-based build
system from gitlab.com/nvidia/container-images/driver. The GPU Operator
expects pre-compiled kernel modules, not a raw .run file.
Add EULA notice about public redistribution of vGPU driver images.
Add note about NLS ServerPort being deployment-dependent.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add vGPU mode as a new variant alongside the existing passthrough (default) variant.
This enables sharing a single physical GPU across multiple VMs using NVIDIA mediated devices.
Changes:
- Add values-talos-vgpu.yaml with vGPU Manager and vGPU Device Manager enabled
- Add 'vgpu' variant to PackageSource
- Add documentation for vGPU setup (driver image, NLS licensing, profiles, KubeVirt)
Users select the vgpu variant via Package CR and provide their proprietary
vGPU Manager image coordinates through component values.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
`hack/helm-unit-tests.sh` auto-discovers any package whose Makefile has
a `test` target. `packages/system/harbor` had none, so the conditional
rendering of `bucket-secret.yaml` (the path this PR actively reshapes)
was un-asserted in CI and the bucket-secret guard would not be pinned
against future refactors.
Add the canonical `test: helm unittest .` target, matching every other
`packages/system/*` package that ships a `tests/` directory
(linstor-scheduler, linstor-gui, cozystack-scheduler, nfs-driver).
Add `tests/bucket_secret_test.yaml` with seven cases covering: spec
absent, spec empty, secretS3 absent, secretS3 empty, secretS3 partial
(only one inner field), bucketName missing, and fully populated. The
first six assert no Secret is rendered; the last asserts the four
expected `stringData` entries plus the static region. 7/7 pass locally
(`bash hack/helm-unit-tests.sh`).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The COSI BucketAccess controller is expected to populate `BucketInfo`
atomically, but the prior guard (`with .Values.spec` / `with .secretS3`)
only short-circuits on missing or empty inputs. A partially populated
`secretS3` would still emit the registry-s3 Secret with empty
`accessSecretKey`/`endpoint` values, and a missing `bucketName` was
not guarded at all — silently breaking Harbor's S3 storage backend.
Tighten the inner guard to require all four required fields
(accessKeyID, accessSecretKey, endpoint, bucketName) before the Secret
is emitted, so partial input falls into the no-render path alongside
unset/empty. The body now uses dot-access on `.secretS3` for
consistency with the new gate. Behaviour for fully populated input is
unchanged.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
PR #2454 renamed the nodeGroups field ephemeralStorage to diskSize and
added a migration guard that fails template rendering when the old name
is used. Update kubelet reservation tests to use the new field name.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Merge origin/main into feat/kubernetes-kubelet-reserved-resources.
PR #2454 (persistent storage for worker nodes) landed in main and
touched the same nodeGroups schema. Conflicts resolved by keeping
both feature sets (kubelet reservations + persistent storage):
- api/apps/v1alpha1/kubernetes/types.go: merged kubebuilder default
marker with all keys from both features
- packages/apps/kubernetes/README.md: merged Parameters table with
diskSize/storageClass + kubelet* rows
- packages/system/kubernetes-rd/cozyrds/kubernetes.yaml: merged
openAPISchema and keysOrder, dropped stale ephemeralStorage path
Signed-off-by: Arsolitt <arsolitt@gmail.com>
GitHub does not allow fetching arbitrary commits by SHA without
uploadpack.allowReachableSHA1InWants. Replace the git init + fetch
approach with a simple shallow clone of the main branch.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
IvanHunters