mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-05 03:27:07 +09:00
fix/vm-instance-disk-null
362
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Aleksei Sviridkin
|
9d9750e736 |
fix(hack): tighten resourcesPreset enum check
Address review feedback from gemini-code-assist on hack/check-rd-presets.sh:26
and coderabbitai on hack/check-rd-presets.sh:37.
- Switch the jq enum extractor from a description-substring heuristic
(test("preset"; "i")) to a JSON path-suffix selector that matches
fields literally named resourcesPreset. Eliminates false positives
from unrelated fields whose descriptions happen to mention "preset".
- Use grep -F for literal enum membership so the "." in entries like
t1.nano cannot accidentally match arbitrary characters (e.g. a typo
t1xnano would have satisfied t1.nano under basic regex).
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
Aleksei SviridkinandMatthieu ROBIN
|
55850aa3d8 |
test(apps): backward-compat helm-unittest and CI invariants
Pin the 1:1 equivalence between legacy flat preset names and their
instance-type counterparts in helm-unittest for three charts that
exercise different preset shapes:
- postgres — top-level only
- clickhouse — top + nested clickhouseKeeper preset
- opensearch — top + nested dashboards preset
Each case asserts that nano <-> t1.nano, small <-> t1.small,
medium <-> c1.small, large <-> c1.medium, and micro <-> t1.micro
render the same CPU and memory.
packages/apps/{clickhouse,opensearch}/Makefile gain a `test:` target
that calls helm unittest so hack/helm-unit-tests.sh actually runs the
new suites in CI; without it the suites would land as dead weight.
hack/check-rd-presets.sh enforces two invariants in `make unit-tests`:
- every packages/system/*-rd/cozyrds/*.yaml openAPISchema carries
the canonical 47-preset enum (40 instance-type + 7 legacy);
- no chart template carries a hardcoded legacy preset literal in
a defaultingSanitize (list "<legacy>" ...) form.
Both turn an accidental drop of a per-chart Makefile step or an
accidental legacy hardcode into a CI failure rather than silent
production-time validation rejection.
Co-authored-by: Matthieu ROBIN <matthieu@hidora.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
Aleksei Sviridkin
|
6f83d9a794 |
feat(ouroboros): bump to v0.7.2 with debug logging on tenants
The v0.7.2 ouroboros release adds --log-level / OUROBOROS_CONTROLLER_LOG_LEVEL, plants Debug logs at every informer event handler (AddFunc/UpdateFunc/ DeleteFunc with kind+namespace/name via describeObj) and at reconcile entry/exit, wires SetWatchErrorHandler on Ingress / Gateway / HTTPRoute informers, and surfaces cache-sync elapsed at Info. The diagnostic surface targets the kamaji-tenant-via-konnectivity flake where bats sees a fresh Ingress create succeed against the tenant apiserver but the controller's coredns-custom rewrite lands roughly 7-10 minutes later — without per-event visibility the failure mode is indistinguishable between 'watch never fires', 'reconcile silently errors', and 'API list returns empty until reflector reconnect'. Bumps the chart-manifest digest, the controller-image digest, and the cozystack-side image-pin assertion in lockstep. The bats deadline for the coredns-custom rewrite check rises from 180s to 900s — even on the broken-watch path the reflector eventually reconnects within that budget, so the test stops failing on the symptom while we keep the diagnostic on to learn what the symptom actually is. cozystack tenants force controller.logLevel: debug via the defaultOuroborosValues helper; operators that want quieter logs can override through addons.ouroboros.valuesOverride. Two helm-unittest fixtures pin both the default and the override path. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
d6bcb27c53 |
test(ouroboros): dump tenant state on every assertion failure, not just HR wait
The CI run after pinning controller.clusterDomain showed the test was finally getting past the helm-controller wait — the proxy /readyz unblocked, the HR reaches Ready, and the controller pod becomes Ready — but the next assertion (the rewrite snippet appearing in kube-system/coredns-custom within 180s of creating the probe Ingress) is still failing on every retry. Without a dump on this assertion the failure surfaces only as `❌ Test failed (exit 1)` with the polled `snippet=` empty lines preceding it; the controller logs that would explain why the reconciler skipped the Ingress are nowhere visible. Refactor the existing inline diagnostic block into a `dump_tenant_state` helper that is invoked on every assertion's failure path. Wrap the rewrite-snippet check, the proxy ClusterIP non-empty check, and the dnscheck pod-phase check in `if ! ...; then dump_tenant_state; exit 1; fi` so each failure dumps the same set of host/tenant pods, events, controller and proxy logs, ingress-nginx Service+endpoints, and tenant CoreDNS Corefile that the original HR-wait failure path already covered. Also append a `kubectl get ingress -n default -o yaml` dump of the probe Ingress so a controller log line like "skipped Ingress without TLS host" is correlatable against the actual rendered manifest at the time of failure. The follow-up fix for whatever the next CI run surfaces will land on this same branch. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
89aab4459b |
test(ouroboros): port-forward + super-admin.conf for tenant kubectl path
Every tenant-side kubectl call in hack/e2e-apps/ouroboros.bats was failing inside the e2e sandbox container with `dial tcp: lookup kubernetes-${cluster}.${ns}.svc on 169.254.169.254:53: no such host`. The bats test pulled the `super-admin.svc` variant of the tenant admin-kubeconfig, which points at the in-cluster Service DNS name `<release>.<ns>.svc:6443` — a name only host-cluster CoreDNS can resolve. The sandbox container's resolver is the Docker/QEMU host link-local 169.254.169.254, not the host-cluster CoreDNS, so every `KUBECONFIG=$kubeconfig kubectl ...` invocation in the test was returning NXDOMAIN.
The check that gated whether the test even reached its assertions (the `kubectl wait helmrelease kubernetes-${cluster}-ouroboros`) ran against the host kube-context, which masked the broken kubeconfig path until the helmrelease itself failed to reconcile and the diagnostic block fired — at which point every dump command returned only NXDOMAIN errors instead of actionable tenant-cluster state.
Switch to `super-admin.conf` (the operator-facing kubeconfig variant), rewrite `clusters[0].cluster.server` to `https://localhost:${pf_port}`, and run a `kubectl port-forward service/kubernetes-${cluster} ${pf_port}:6443` in the background — the same pattern hack/e2e-apps/run-kubernetes.sh already uses for tenant-cluster reachability. Replace the inline trap RM with a `cleanup_kubeconfig` helper that also kills the port-forward process so retries don't leak listeners.
This is a test infrastructure fix; the underlying ouroboros HelmRelease install timeout itself is a separate root cause that the next CI run, with usable diagnostics, will surface.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
Aleksei Sviridkin
|
c99be232ad |
test(ouroboros): dump tenant-side state on bats wait failure
The smoke test at hack/e2e-apps/ouroboros.bats waits for the kubernetes-${cluster}-ouroboros HelmRelease to reach Ready and exits with `wait: timed out` on failure. cozyreport collects host-cluster state only — a tenant-side proxy or controller failure surfaces as opaque "0 of 2 pods ready" with no actionable signal, so every retry is fully blind.
Extract the tenant admin-kubeconfig before the wait (gated on the parent kubernetes HR being Ready, which is the precondition for the secret existing) and, on wait failure, dump the host HelmRelease describe, cozy-ouroboros pods, pod descriptions, recent events, controller and proxy logs (200 lines each), the upstream cozy-ingress-nginx Service and endpoints used as the proxy backend, and the tenant kube-system Corefile plus the coredns-custom ConfigMap data.
Drops the ouroboros-HR wait timeout from 25m to 20m to leave room for the diagnostic block within the bats test budget.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
Andrei Kvapil
|
45e21a330f |
feat(networking): add ouroboros for hairpin-NAT under PROXY-protocol (#2561)
## What this PR does Adds [`lexfrei/ouroboros@v0.7.0`](https://github.com/lexfrei/ouroboros/releases/tag/v0.7.0) — a Go reimplementation of [`compumike/hairpin-proxy`](https://github.com/compumike/hairpin-proxy) — and wires it into cozystack as a *conditional default*: a single `publishing.proxyProtocol: true` switch turns PROXY-protocol on at the host ingress-nginx **and** auto-deploys ouroboros to fix the resulting hairpin-NAT problem (intra-cluster traffic to the cluster's own public hostnames arriving at ingress-nginx without the PROXY header it now requires). Default stays `false` — clusters that do not run PROXY-protocol get zero new resources, zero RBAC delta, zero behaviour change. The chart is platform-agnostic at v0.7.0: the controller and proxy binaries auto-detect the cluster's DNS domain from `/etc/resolv.conf` inside the pod and compose the rewrite target FQDN and the backend host at runtime as `<svc>.<ns>.svc.<cluster-domain>`. Cozystack tenants on `cozy.local`, kubeadm-bootstrapped clusters on `cluster.local`, federations on `k8s.example.com`, custom-domain RKE2/k3s — all work without operator override. Verified end-to-end on a `cozy.local` cluster: controller logs `cluster-domain=cozy.local` at startup, chart emits `--proxy-service-name + --proxy-service-namespace` (no chart-baked `--proxy-fqdn`), Corefile picks up the rewrite block between BEGIN/END markers, in-pod `dig <ingress-host>` returns the ouroboros-proxy ClusterIP. Operators that prefer a chart-baked FQDN can still set `controller.clusterDomain` explicitly — the explicit-override path is preserved for non-Service backends and for split-horizon DNS topologies. The chart runs in `coredns` mode for the host (the host CoreDNS is Talos-managed and does not carry an `import` directive, so the alternative would silently no-op there) and in `coredns-import` mode for tenant deployments (tenants run cozystack-coredns, which carries the import directive and ships an empty `kube-system/coredns-custom` ConfigMap; ouroboros writes plugin-only `rewrite name` snippets into the `ouroboros.override` data key without ever touching the chart-rendered Corefile, so it does not race the chart that re-templates `data.Corefile` on every Flux reconcile). ### Layout * `packages/system/ouroboros/` — vendored chart from `oci://ghcr.io/lexfrei/charts/ouroboros@0.7.0`. Cozystack-side defaults pinned in `values.yaml`: `controller.mode=coredns` (host scenario), `proxy.target.namespace=cozy-ingress-nginx` (override the chart-default `ingress-nginx` namespace; chart-default `serviceName=ingress-nginx-controller` matches as-is), Gateway-API watching left at upstream's default of `false`. `proxy.target.host` is left empty so the proxy composes the backend host at runtime. Image is digest-pinned at `0.7.0@sha256:478665e05dd0ffc4c1f7764320ea24b52251781884ddfd668d93a03e39d9094c` and pulled directly from `ghcr.io/lexfrei/ouroboros` — this package is intentionally not mirrored under `ghcr.io/cozystack/*`. The chart manifest itself is also digest-pinned via `OUROBOROS_CHART_DIGEST=sha256:f28bfac9fd7070b1f3357983ff5aad28c16115e7de1cfc16ee568a3b4cfc9d7e` in the package Makefile, with a Chart.yaml-version vs. Makefile-version assert that aborts the update target on lockstep drift. Helm-unittest in `tests/` covers controller args, RBAC narrowing per mode, Gateway-API gating, runtime-composition flag emission in both `coredns` and `coredns-import` modes, image pin drift. * `packages/system/coredns/templates/coredns-custom.yaml` + `values.yaml` — empty `coredns-custom` ConfigMap as the chart-managed `import` prerequisite, plus the `import /etc/coredns/custom/*.override` plugin in the `.:53` server block + `extraVolumes`/`extraVolumeMounts` mount at `/etc/coredns/custom/`. Same package serves host and tenant CoreDNS, so this covers both layers without duplication. Helm-unittest pins the rendered Corefile via a snapshot so future `make update` does not silently drop the import directive. The empty rendered ConfigMap deliberately omits `data:` — Helm three-way merge has no template-side keys to compare against, so apiserver-side runtime writes (the `ouroboros.override` data key) survive every chart upgrade. The `notExists: data` assertion in `tests/coredns_custom_test.yaml` pins the invariant. * `packages/core/platform/sources/ouroboros.yaml` — new `cozystack.ouroboros` PackageSource for the host install (`dependsOn: cozystack.networking, cozystack.ingress-application`). * `packages/core/platform/sources/kubernetes-application.yaml` — `kubernetes-ouroboros` component registered against `system/ouroboros` so the tenant HelmRelease can reference it. * `packages/apps/kubernetes/templates/helmreleases/ouroboros.yaml` — tenant-side HelmRelease deployed via the tenant admin-kubeconfig, mirroring the cert-manager pattern. A `cozystack.defaultOuroborosValues` helper flips `mode` back to `coredns-import` for tenants and mirrors per-tenant `addons.gatewayAPI.enabled` into the controller (forcing `--gateway-api` on a tenant without Gateway-API CRDs CrashLoops the controller on `WaitForCacheSync`). `controller.clusterDomain` is intentionally NOT pre-set — chart 0.7.0 auto-detects per-pod from `/etc/resolv.conf` and composes the rewrite target at runtime, so the same wrapper renders correctly on `cluster.local`, `cozy.local`, or any custom tenant cluster-domain. Operators on a non-default cluster-domain can still override via `valuesOverride.ouroboros.controller.clusterDomain`. `dependsOn` grows a conditional `gateway-api-crds` entry on the same gate. If `addons.ouroboros.enabled` is true while `addons.ingressNginx.enabled` is false, the HelmRelease fails the render with a clear error rather than silently skipping — the hairpin fix has nothing to fix against without ingress-nginx, and a no-op addon that quietly does nothing is harder to debug than an explicit render-time error. The valuesOverride gateway-api guard accepts every truthy spelling Helm/YAML emit (bool `true`, string `"true"`/`"True"`/`"TRUE"`, numeric `1`, YAML `yes`/`Yes`/`YES`/`on`/`On`/`ON`/`y`/`Y`) via `toString`-then-`lower`-then-string-compare; direct bool `eq` would error on type mismatch when an operator passes the value as a string from `--set`, and case-insensitive folding catches `--set ...=On` / `--set ...=True` that pure-lowercase comparison would miss. Tenant disable is single-step: flipping `addons.ouroboros.enabled: false` deletes the rendered HelmRelease, helm-controller runs `helm uninstall` on the tenant, and the chart's pre-delete cleanup hook (`coredns-cleanup-hook.yaml`, vendored at 0.7.0) nulls the `ouroboros.override` key on the tenant's `kube-system/coredns-custom` ConfigMap automatically. No render-time guard is wired on the tenant side — gating on a `lookup` for the leftover HelmRelease would deadlock the routine disable (the parent render runs before helm-controller applies the missing-child diff, so the lookup would always find the leftover at the moment of the flag flip). If the chart's hook fails to land (controller pod stuck CrashLooping, ConfigMap RBAC drift, Job timeout), operators recover manually via the documented tenant cleanup recipe. * `packages/apps/kubernetes/values.yaml` — new `OuroborosAddon` typedef with `enabled` + `valuesOverride`. `make generate` re-emitted README, JSON schema, the Go struct + deepcopy, and the kubernetes-rd ApplicationDefinition openAPISchema in lockstep. * `packages/core/platform/values.yaml` — new `publishing.proxyProtocol` flag with extensive comment covering the upstream-LB precondition (the L4 LB in front of ingress-nginx must already be injecting PROXY-v1 headers before the flag flips on, otherwise external traffic breaks), the host-vs-tenant fan-out (the flag does not cascade into tenants), the Talos re-render flap window, and the asymmetric disable path. The host disable path: flipping back to `false` reverts ingress-nginx PROXY-protocol config (flip the LB off first), but the host ouroboros itself stays installed because its Package CR carries `helm.sh/resource-policy: keep` — an explicit `kubectl delete package.cozystack.io cozystack.ouroboros` (or `bundles.disabledPackages` entry) actually triggers helm uninstall, at which point the chart's pre-delete cleanup hook fires and patches the host Corefile automatically. The values comment ships a manual `sed`-based cleanup recipe as a fallback for the rare case where the chart's hook fails to land (controller stuck in CrashLoop, ConfigMap RBAC drift, hook timeout); the `publishing.proxyProtocolAcknowledgeUnclean` gate keys off a `lookup` against the leftover Package CR to catch operators who flip the flag back without confirming the cleanup landed. * `packages/core/platform/templates/bundles/system.yaml` — flips between the default-form and explicit-form `cozystack.ingress-application` Package emission so the host ingress-nginx gets `use-proxy-protocol` + `real-ip-header=proxy_protocol` injected only when the flag is true (`use-forwarded-headers` and `compute-full-forwarded-for` are deliberately *not* on the list — they would let any upstream proxy spoof `X-Forwarded-For`); the same conditional emits `cozystack.ouroboros`. Host Gateway-API watching is left at the upstream chart's default of `false` because cozystack does not install Gateway-API CRDs cluster-wide on the host. * `hack/e2e-apps/ouroboros.bats` — auto-discovered smoke test for the tenant addon (HelmRelease ready, `coredns-custom` ConfigMap present in tenant, controller pod running, and after creating a TLS Ingress the rewrite snippet shows up in the import ConfigMap; an in-tenant `dig` from a netshoot pod also confirms CoreDNS actually serves the rewrite — pod-readiness alone passes vacuously since the upstream chart ships no readiness probes). ### Why not KEP-1860 `ipMode=Proxy` Cilium's `kubeProxyReplacement: true` (cozystack default) drops the LB frontend entirely when `ipMode != VIP`, breaking Service ingress for L2/BGP-announced IPs. KEP-1860's Proxy mode is reserved for clusters fronted by an external CCM-managed proxy LB. ouroboros is the correct fix for L2/BGP-announce topologies that cozystack ships by default. ### Coverage Covered by helm-unittest (host chart, coredns wrapper, tenant addon — including the `kubernetesProvider` mock that exercises the lookup-driven `proxyProtocolAcknowledgeUnclean` gate, the lookup-returns-nil CI/dry-run path, the gateway-api truthy-spelling acceptance across `true`/`"true"`/`"True"`/`"TRUE"`/`1`/`yes`/`on`, the assertion that `--proxy-service-name` + `--proxy-service-namespace` are emitted instead of a chart-baked `--proxy-fqdn` in both `coredns` and `coredns-import` modes, the assertion that `--target-service-name` + `--target-service-namespace` are emitted for the proxy backend, and the `_namespace.etcd`-empty render gate) and a tenant e2e bats test that creates a real Ingress, asserts the rewrite snippet lands in the import ConfigMap, and confirms in-tenant DNS serves it (with an in-pod retry loop tolerating CoreDNS reload latency). The host-level `publishing.proxyProtocol: true` path is not e2e-tested in this PR — it requires a PROXY-protocol-aware upstream LB, which the cozystack e2e harness does not stand up. That path is exercised manually and by docs review. End-to-end runtime auto-detect was verified manually on a `cozy.local` cluster (dev17): chart deployed at v0.7.0, controller logged `cluster-domain=cozy.local`, the rewrite block landed in `kube-system/coredns` Corefile pointing at `test-ouroboros-proxy.test-ouroboros.svc.cozy.local.`, an in-pod `dig <ingress-host>` resolved to the ouroboros-proxy ClusterIP on the first attempt, and the chart's pre-delete hook restored the Corefile cleanly on `helm uninstall`. ### Supply-chain note Image and chart are pulled from a contributor's personal namespace (`ghcr.io/lexfrei/...`). Existing precedent (`packages/system/etcd-operator/Makefile` pulls from `ghcr.io/aenix-io/...`) is similar but `aenix-io` is the cozystack-adjacent org while `lexfrei` is one human. Worth a maintainer call. Pairs with cozystack/website#527 (operator docs). ### Release note ```release-note feat(networking): add `publishing.proxyProtocol: true` to enable PROXY-protocol on host ingress-nginx and auto-deploy ouroboros (a Go reimplementation of compumike/hairpin-proxy) to solve the resulting hairpin-NAT problem; tenants can opt in per-cluster via `addons.ouroboros.enabled` (requires `addons.ingressNginx.enabled`). Default behaviour unchanged. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added Ouroboros addon to address hairpin‑NAT for ingress‑nginx with configurable enablement and tenant overrides. * Added a packaged Ouroboros chart providing proxy, controller, CoreDNS integration, external‑DNS and etc‑hosts modes. * Introduced PROXY‑protocol gating with an explicit acknowledgement flag. * **Documentation** * Expanded addon reference and platform docs with detailed Ouroboros/PROXY‑protocol configuration and defaults. * **Tests** * Added extensive unit and e2e tests covering rendering, RBAC, uninstall cleanup, and DNS behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
a9c1298b42 |
[gpu-operator] Update to v26.3.1 and add experimental vGPU variant (#2323)
## What this PR does Updates the NVIDIA GPU Operator from v25.3.0 to v26.3.1 and adds an **experimental** `vgpu` variant for SR-IOV vGPU on Ada Lovelace and newer GPUs. Changes: - Update vendored upstream chart to v26.3.1 (CDI enabled by default, component version bumps). - Pin chart version in Makefile for reproducibility. - Add `values-vgpu.yaml` for the vGPU variant with `defaultWorkload: vm-vgpu` so the chart activates the vGPU Manager DaemonSet without per-node labelling. - Add `vgpu` variant to PackageSource alongside the existing `default` (passthrough). - Rename `values-talos.yaml` → `values-passthrough.yaml` and pin `defaultWorkload: vm-passthrough` for symmetry with the vgpu variant. The overrides apply to any host OS using vfio-pci, not Talos specifically. - Refresh `hack/dcgm-default-counters.csv` to match the new dcgm-exporter image tag (`4.5.1-4.8.0`) — adds `DCGM_FI_DEV_FB_RESERVED` and other fields introduced upstream. `hack/check-gpu-recording-rules.bats` passes against the new fixture. - Add `hack/check-gpu-operator-variants.bats` that renders both variants via `helm template` and asserts the pinned ClusterPolicy values, so the next chart bump cannot silently un-pin them. - Add internal documentation for the vGPU bring-up. The `vgpu` variant assumes the user supplies a proprietary vGPU Manager image via Package CR component values — the .run cannot be redistributed. ### `vgpu` variant is experimental NVIDIA's `vgpu-device-manager` walks `/sys/class/mdev_bus/`, which does not exist on Ada Lovelace and Blackwell — the DaemonSet errors with "no parent devices found" and is disabled by default. Until an SR-IOV-aware controller is shipped, profile assignment is an out-of-band step (`echo <id> > /sys/.../current_vgpu_type` per VF) that must be re-applied after every node reboot. Without it, `permittedHostDevices.pciHostDevices` reports zero allocatable resources. The variant ships so the chart bump and the underlying skeleton (DaemonSet + image plumbing + docs) land together; production deployments need an automated profile-assignment mechanism in addition to this PR. ### KubeVirt requirement KubeVirt advertises VFs via `permittedHostDevices.pciHostDevices` rather than `mediatedDevices`. This requires kubevirt/kubevirt#16890 ("vGPU: SRIOV support", merged 2026-04-10) which is `main`-only — released tags `v1.6.x` / `v1.7.x` / `v1.8.x` do not include it. The patch is targeted at the next minor KubeVirt release (v1.9.0); track the PR for the actual release tag. Backports to released branches are not planned. ### Behavior changes inherited from upstream chart v25 → v26 The upstream chart flipped two defaults that are pinned back to `false` here, so `default` (passthrough) deployments do not silently change footprint on upgrade: - `ccManager.enabled` was flipped to `true` upstream — would auto-enable Confidential Computing on Hopper hardware (H100/H200/B100) after upgrade. - `vgpuDeviceManager.enabled` was flipped to `true` upstream — DaemonSet would render even in passthrough mode. Both stay opt-in via Package CR overrides. The new bats test guards the pin. Other inherited changes that are user-visible and worth pinning down before upgrade: - `driver.licensingConfig.configMapName` is deprecated upstream in favour of `driver.licensingConfig.secretName`. The deprecation marker lives in the CRD's `description` field (visible via `kubectl explain`) — there is no helm-render-time warning, the chart template silently prefers `secretName` and falls back to `configMapName`. Operators with existing licensing overrides should migrate the key on the next chart upgrade. - `driver.version` upstream default jumped from `570.x` to `580.x`. Invisible on the `default` variant (which sets `driver.enabled: false`), user-visible for anyone running the upstream Ubuntu driver path. - `vfioManager.image` switched from `nvcr.io/nvidia/cuda:12.8.1-base-ubi9` to `nvcr.io/nvidia/cloud-native/k8s-driver-manager:v0.10.0`. Affects every `default` (passthrough) deployment on upgrade — different image, different layer, different image-pull surface. Worth pre-mirroring for air-gapped clusters. - `dcgmExporter.image` switched to the `-distroless` flavour. Smaller surface, fewer base-image CVEs, but no shell — operators who used to `kubectl exec` into the exporter for ad-hoc debugging will need to switch to `kubectl debug` or container ephemeral images. - `CDI` (Container Device Interface) is enabled by default in v26.3.1; this is upstream's recommended setting and is left as-is. - `node-feature-discovery` subchart source switched from the GitHub Pages Helm repo to `oci://registry.k8s.io/nfd/charts`. Air-gapped installations need to mirror the new OCI URL. ### Talos Talos is **not recommended** for the vGPU variant. NVIDIA does not publicly distribute the vGPU guest driver — it requires NVIDIA Enterprise Portal access — and Sidero closed siderolabs/extensions#461 noting that they cannot support vGPU "unless NVIDIA changes their licensing terms or provides us a way to obtain, test, and distribute the software". Building a Talos system extension that bakes in the vGPU driver is therefore not feasible without a private fork that violates the EULA. Passthrough (`default` variant) on Talos is unaffected. ### Release note ```release-note [gpu-operator] Update NVIDIA GPU Operator to v26.3.1 and add experimental `vgpu` variant for SR-IOV vGPU on Ada Lovelace / Blackwell (L4, L40, L40S, B100; Ubuntu host; requires KubeVirt v1.9.0+ for `pciHostDevices` advertising). The variant ships skeleton support — profile assignment (`current_vgpu_type` per VF) is currently out of band and must be automated by the operator before production use. On the existing `default` variant: `ccManager` and `vgpuDeviceManager` upstream defaults are pinned back to `false` to preserve prior behavior on chart upgrade; `defaultWorkload` is now pinned to `vm-passthrough` (was unset, falling back to `container`); `driver.licensingConfig.configMapName` is deprecated by upstream in favour of `secretName`; `driver.version` upstream default moves 570 → 580; `vfioManager.image` switches from CUDA UBI9 to k8s-driver-manager (relevant for air-gapped mirrors); `dcgmExporter` image switched to the `-distroless` flavour. NFD subchart source switched to `oci://registry.k8s.io/nfd/charts`. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Introduced vGPU support variant with end-to-end configuration guide, validation procedures, and troubleshooting documentation * **Updates** * Upgraded GPU Operator to v26.3.1 and Node Feature Discovery to v0.18.3 * Improved component health checks and added PodDisruptionBudget availability controls <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
61de813613 |
feat(e2e): parallel post-install prep (#2557)
## What this PR does ### Parallel post-install prep LINSTOR / StorageClass / MetalLB setup, previously sequential `@test` cases after the 15 m platform HR wait, now run as a background prep that completes during the platform reconcile. None of them gates platform HR reconcile, so they can move out of the critical path. New helper `hack/e2e-post-install-prep.sh` polls for its prerequisites and is awaited at the end of the `Install Cozystack` test so failures still surface deterministically. ### T2-race fix Wait on the LINSTOR HR being `Ready` before polling for `deploy/linstor-controller`. The old 5-min hard timeout fired during cold installs where the LINSTOR HR's dependency chain (cert-manager → piraeus-operator-crds → piraeus-operator → linstor) takes longer than 5 min to resolve. ### E2E uses helmReleaseInterval=30s E2E install passes `--set cozystackOperator.helmReleaseInterval=30s` to eliminate the 8–10 min Flux dead zone where dependency-blocked HRs are requeued only every 5 min. Operator-side flag and chart value landed in #2509; production defaults are unchanged. Surfaced from #2500. ### Release note ``` NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** - Improved end-to-end test initialization to run concurrent post-install preparation, wait for cluster release reconciliation, verify component readiness, and validate automated storage configuration. * **Chores** - Added background post-install setup and logging to increase test reliability and speed up cluster initialization and teardown diagnostics. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
32e75ab183 |
Merge branch 'main' into daniil/split-vminstance
Signed-off-by: myasnikovdaniil <60174387+myasnikovdaniil@users.noreply.github.com> |
||
|
myasnikovdaniil
|
7585b0c42c |
test(e2e): replace fixed timeouts with event-driven backstops (#2554)
## What this PR does Replaces fragile `sleep N` / `timeout N until …; do sleep …; done` patterns in `hack/*.bats` and `hack/*.sh` with `kubectl wait --for=condition=…` plus a small existence backstop: ```sh until kubectl get <resource> >/dev/null 2>&1; do sleep 2; done kubectl wait <resource> --for=condition=Ready --timeout=… ``` ### Why the existence backstop matters `kubectl wait` against a not-yet-created resource sits silently for the full timeout and then fails with `error: timed out waiting for the condition` — useless for diagnosis. With the backstop, the test fails fast with a clear "resource never appeared" error. Existence backstop added at 14 sites in this sweep across: - `redis`, `kafka`, `openbao`, `harbor`, `mongodb`, `qdrant`, `clickhouse`, `mariadb`, `postgres`, `bucket`, `external-dns`, `foundationdb` bats - `hack/e2e-install-cozystack.bats` - `hack/e2e-apps/run-kubernetes.sh` - `hack/e2e-test-openapi.bats` (port-readiness wait replacing fixed `sleep 5` after `kubectl proxy`) The vm-disk hunk in `hack/e2e-apps/vminstance.bats` is included here; the vm-instance hunks land separately via the `daniil/split-vminstance` PR. ### Genuine sleeps annotated The 3 remaining genuine sleeps (LINSTOR node membership propagation, MetalLB end-to-end HTTP path, HR-count emission heuristic) are annotated with `# TODO` + rationale so a future reader knows they are intentional. Same green-path semantics; failures now surface in seconds instead of after a 5-minute timeout. Surfaced from #2500. ### Release note ``` NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Improved e2e stability by replacing fixed sleeps with timeout-based polling across many integration tests, waiting for operator-created resources to exist before readiness checks to reduce race conditions and flakes. * **Documentation** * Added clarifying notes for LoadBalancer retry behavior and improved local proxy readiness checks in e2e scripts. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
d8da80bdd2 |
fix(e2e): propagate failures from parallel LINSTOR pool jobs
Address review feedback on hack/e2e-post-install-prep.sh:31: POSIX `wait` without arguments returns 0 even if a backgrounded `kubectl exec` failed, which would mask partial LINSTOR prep failures and let downstream tests run on a half-provisioned cluster. Collect each background PID and `wait "$pid"` per job so `set -e` exits the script on the first failed pool creation. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
209fa08737 |
test(e2e): decouple vminstance test from vmdisk test
"Create a VM Instance" referenced the VMDisk created by "Create a VM Disk" via the shared name "test". This implicit ordering meant any defensive pre-clean of vmdisks in test 2 would destroy the disk test 1 just provisioned, and the vm-instance chart's lookup-or-fail on the DataVolume would then surface as "Specified disk not exists in cluster: test". Make each test self-contained: - "Create a VM Disk" cleans up its own VMDisk at the end. - "Create a VM Instance" creates its own VMDisk inline (renamed to "vmi-test" to avoid colliding with any in-flight finalizer drain on test 1's resources) and waits HR -> DV -> PVC before applying the VMInstance. Replaces the pre-delete approach from the prior followup commit, which broke the data dependency. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
585ea7e089 |
test(e2e): smoke-test the ouroboros tenant addon
Auto-discovered hack/e2e-apps/ouroboros.bats spins up a tenant
Kubernetes CR with addons.ingressNginx + addons.ouroboros enabled
and asserts the cozystack-side wiring delivers a working hairpin
fix end-to-end:
* the cozy-ouroboros HelmRelease in the tenant's host namespace
reaches Ready (chartRef + admin-kubeconfig wiring is correct);
* the cozystack-coredns wrapper rendered the kube-system/coredns-custom
ConfigMap inside the tenant (the import-glob prerequisite is in
place);
* the controller pod is running in cozy-ouroboros inside the tenant;
* after creating an Ingress with a TLS hostname, the tenant's
coredns-custom ConfigMap eventually carries a `rewrite name <host>`
line under the ouroboros.override key — proves the controller is
reconciling, not merely Pod-Ready (the upstream chart ships no
readiness probes, so a controller stuck on WaitForCacheSync would
pass a pod-readiness check vacuously; the rewrite assertion catches
every cozystack-specific failure mode the unit tests cannot — wrong
namespace, missing CRDs, narrowed RBAC, gateway-api gate misfires).
End-to-end PROXY-protocol header roundtrip (the actual hairpin fix
in action) is covered by the upstream chart's own e2e on every push;
this file is the cozystack-side integration test that the cozystack
glue (chart vendoring, kubernetes-application registration, tenant
HelmRelease wiring, gateway-api gating, coredns-custom prerequisite)
holds together.
cozystack hack/cozytest.sh is plain shell, not real bats, so kubectl
output is captured directly without the `run` helper.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
Myasnikov Daniil
|
cc7551cf1b |
test(e2e): use $name in kafka.bats kubectl wait (review feedback PR #2554)
Address lexfrei's nit on PR #2554: the pre-existing
`kubectl wait kafkas -n tenant-test test ...` on line 47 hardcoded the
resource name, while the existence backstop one line above was already
parametrized with `$name` in
|
||
|
Myasnikov Daniil
|
fe138afab7 |
test(e2e): add --ignore-not-found to vminstance.bats end-of-test deletes
Symmetrize the post-success cleanup on lines 83-84 with the pre-clean shape on lines 12-13/41. Without --ignore-not-found, a race that already removed the resource (e.g., a parallel teardown or finalizer-driven GC) turns a passing test into a failure after assertions have already run. Review feedback PR #2555 (lexfrei follow-up #1). Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
7ed6444108 |
feat(installer,e2e): cozystackOperator.helmReleaseInterval value + parallel post-install prep
Two complementary changes that depend on the operator's `--helmrelease-interval` flag landing via #2509: 1. Installer chart exposes `cozystackOperator.helmReleaseInterval` (default empty -> flag not rendered -> operator default 5m applies). E2E install passes `30s`. Eliminates the 8-10 min Flux dead zone where dependency-blocked HRs (waiting on cert-manager webhooks, CRDs) are requeued only every 5 min. 2. LINSTOR / StorageClass / MetalLB setup, previously sequential `@test` cases after the 15 m platform HR wait, now run as a background prep that completes during the platform reconcile. None of them gates platform HR reconcile, so they can move out of the critical path. Helper script `hack/e2e-post-install-prep.sh` polls for its prerequisites and is awaited at the end of the `Install Cozystack` test so failures still surface deterministically. Includes the T2-race fix: wait on the LINSTOR HR being `Ready` before polling for `deploy/linstor-controller`. The old 5-min hard timeout fired during cold installs where the LINSTOR HR's dependency chain takes longer than 5 min to resolve. Production defaults are unchanged. The 30 s interval is opt-in per install command. Depends on: - #2509 (operator-side `--helmrelease-interval` flag). Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Assisted-By: Claude <noreply@anthropic.com> |
||
|
myasnikovdaniil
|
7520967d8d |
feat(operator): expose HelmRelease generation knobs (interval, retry-interval, timeouts, max-history) (#2509)
## Why
The `cozystack-operator` Package reconciler builds every `HelmRelease`
it generates with a handful of hardcoded values. Three of those have
caused recurring pain across the project:
- **`Spec.Interval = 5m`** (line 220 before this PR) — too long for E2E
install where dependent HRs reconcile out of order; previously addressed
by a manual `flux reconcile --force` workaround in
`hack/e2e-install-cozystack.bats`.
- **`Install.Remediation{Retries: -1}` + `Upgrade.Remediation{Retries:
-1}`** — semantically "retry forever, never remediate", but the
retry-on-failure timing was tied to `Spec.Interval` (5m). Audit of the
last 30 successful PR runs found that **every** run hit `kubectl wait
hr/seaweedfs-system` timeouts at the 2-min mark, recovering only after
the inline `flux reconcile --force` workaround.
- **`Install.Timeout = 10m` / `Upgrade.Timeout = 10m`** — directly
contradicts the per-Application timeout work in `pkg/config/config.go` +
`pkg/registry/apps/application/rest.go` (commit `7b146cbe`).
`packages/apps/tenant/templates/etcd.yaml` hand-rolls its HR YAML to
bypass this for cert rotation; harbor has had multiple commits chasing
the same problem.
- **`Spec.MaxHistory`** unset → Helm default 5. Test clusters bounce HRs
frequently and accumulate release-history Secrets.
This PR exposes all four as operator flags + helm chart values, with
defaults that preserve current behavior.
## What
### New operator flags
| Flag | Default | HR field |
|---|---|---|
| `--helmrelease-interval` | `5m` | `Spec.Interval` |
| `--helmrelease-retry-interval` | `30s` |
`Spec.{Install,Upgrade}.Strategy.RetryInterval` |
| `--helmrelease-install-timeout` | `10m` | `Spec.Install.Timeout` |
| `--helmrelease-upgrade-timeout` | `10m` | `Spec.Upgrade.Timeout` |
| `--helmrelease-max-history` | `5` | `Spec.MaxHistory` |
### Strategy switch
`Install.Strategy.Name` and `Upgrade.Strategy.Name` are now set to
`RetryOnFailure` (instead of leaving the default `RemediateOnFailure`),
with `RetryInterval` set from the new flag. Functionally equivalent to
the previous `Remediation{Retries:-1}` ("retry forever") since
remediation never fired anyway, but **decouples retry-on-failure timing
from `Spec.Interval`** so failed releases recover at 30s without polling
healthy releases at the same fast cadence.
### Chart exposure
`packages/core/installer/values.yaml` gets five new optional values
(`cozystackOperator.helmRelease{Interval,RetryInterval,InstallTimeout,UpgradeTimeout,MaxHistory}`),
each defaulting to empty so the operator's own default applies. The
chart conditionally renders the corresponding flag only when the value
is set.
## Production safety
- All static defaults match the previous hardcoded values (5m / 10m /
10m / 5). No new code path runs until an admin explicitly sets a flag.
- **One intentional behavior change: failed-release retry cadence**
drops from ~5m (previously coupled to `Spec.Interval`) to 30s
(`RetryInterval` default). This is the whole motivation for the strategy
switch — the recurring `kubectl wait hr/seaweedfs-system` flakes
documented above were caused by the previous coupling. Healthy-release
reconcile cadence is unchanged at 5m in production; CI overrides this to
30s via `--set cozystackOperator.helmReleaseInterval=30s` in
`hack/e2e-install-cozystack.bats` for faster convergence on E2E
clusters.
- `RetryOnFailure` is functionally equivalent to `RemediateOnFailure +
Retries:-1` since neither ever triggers remediation.
- Empty value in the chart → flag not rendered → operator falls back to
its built-in default. (`helmReleaseMaxHistory: 0` — unlimited history
per Helm semantics — is also honored; the chart guards on `ne (toString
…) ""` rather than truthiness so int `0` is not silently dropped.)
## Verification
- `go build ./...` clean
- `go test ./internal/operator/...` passes (covers `Interval`,
`MaxHistory`, `Install/Upgrade.Timeout`, `Strategy.Name`,
`RetryInterval`, `ChartRef` propagation, plus a regression test for
`MaxHistory=0`)
- `helm template installer packages/core/installer` renders cleanly with
default and overridden values; `--set
cozystackOperator.helmReleaseMaxHistory=0` correctly emits
`--helmrelease-max-history=0`
## Followups (deliberately not bundled)
- **Per-component timeout overrides** on `ComponentInstall` (mirroring
the `UpgradeCRDs` pattern from #2427). Charts like etcd that need a 30m
timeout would set it on their `Component` rather than tuning a global
flag. Touches the Package CR API + codegen — bigger lift, separate PR.
- **`Remediation.Retries` as a flag.** No current pain; skip.
- **`createOrUpdateHelmRelease` field preservation.** Currently
preserves `Suspend` but not `Test`/`Rollback`/`Uninstall`. Would
silently overwrite an SRE's manual edit. Minor cleanliness, separate PR.
## Test plan
- [ ] CI passes E2E with `helmReleaseInterval=30s` set in install bats
(existing T1 test path)
- [ ] No regression on production install path (default values)
- [ ] HR objects in a healthy cluster render with
`Strategy.Name=RetryOnFailure` and `RetryInterval=30s`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Operator and Helm chart now accept CLI flags and Helm values to
configure HelmRelease reconciliation intervals, retry behavior,
install/upgrade timeouts, and history retention.
* **Bug Fixes**
* Operator now validates HelmRelease timing/history inputs on startup
and will exit if values are invalid or non-positive.
* **Tests**
* Added unit and e2e coverage to verify HelmRelease configuration and a
chart install path using a custom reconciliation interval.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
myasnikovdaniil
|
086e9bb35b |
fix(installer): bootstrap cozy-system via --create-namespace + label hook (#2508)
## Why helm v3 has a chicken-and-egg with charts that ship their own Namespace resource: - **With `--create-namespace`** on the install: helm pre-creates the namespace via plain kubectl-create (no helm meta annotations); the chart's own Namespace apply then fails with `already exists`. - **Without `--create-namespace`**: helm fails immediately because it can't write its release-secret to a non-existent namespace. The cozy-installer chart shipped a `Namespace cozy-system` resource since the chart's introduction (December 2025). Every cold install since has hit this conflict. It was masked by the 3× retry on `Install Cozystack` in `.github/workflows/pull-requests.yaml`: attempt 1 fails with the conflict, attempt 2 sees the existing failed release and takes the upgrade code path which patch-merges instead of strict-create. Surfaced when retries on the install step were dropped. ## What - Remove `Namespace cozy-system` from `packages/core/installer/templates/cozystack-operator.yaml`. - Add `packages/core/installer/templates/cozy-system-labels.yaml`: a pre-install/pre-upgrade hook (ServiceAccount + ClusterRole + ClusterRoleBinding + Job) that patches `cozystack.io/system=true` and `pod-security.kubernetes.io/enforce=privileged` onto the namespace. - Hook RBAC is scoped to `verbs: [get, patch]` on `resourceNames: [cozy-system]` only. - `hook-delete-policy: before-hook-creation,hook-succeeded` so the hook surface only exists during install/upgrade. Install commands now use `helm upgrade --install --namespace cozy-system --create-namespace …` — standard pattern, matches kube-prometheus-stack / argo-cd / cert-manager. ## Verification End-to-end on a kind cluster: - Cold install: 3.3s clean, no retries needed - Upgrade (re-run): 3.0s, idempotent, namespace stays labeled - Helm uninstall + re-install: clean ## Test plan - [ ] CI on this PR passes E2E - [ ] Existing clusters: namespace already labeled correctly, hook re-asserts on next upgrade — no migration needed ```release-note fix(installer): cold installs no longer fail on the chart's Namespace resource colliding with `helm --create-namespace`; cozy-system is now bootstrapped via a pre-install hook. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added an automated pre-install/pre-upgrade step that applies system identity and pod-security labels to the system namespace during deployment, run in a hardened one-shot task before normal workloads start. * **Chores** * Separated namespace initialization from the operator install flow so the operator no longer renders or retains the system namespace directly, improving installation sequencing. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov DaniilandClaude Opus 4.7
|
25180c58f7 |
fix(virtual-machine): bump HR readiness wait from 5s to 60s
5s was too tight under runner load — both vm-disk-$name and vm-instance-$name HR waits could miss the operator's reconcile window. Aligns with the other timeout bumps in this PR. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
12ccc53d9d |
test(e2e): tighten event-driven backstops
- kafka.bats: use $name in the kafkas existence backstop, matching the variable used in the kubectl wait below it (consistency with the rest of this PR's backstops). - e2e-install-cozystack.bats: the Tenant→HelmRelease materialisation is asynchronous (cozystack-controller observes the Tenant CR and creates the HR), so guard hr/tenant-test with the same existence backstop pattern used elsewhere. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
7d292eb187 |
chore(e2e): drop seaweedfs/monitoring flux-reconcile workaround
With Strategy=RetryOnFailure + RetryInterval=30s plus the bats --set cozystackOperator.helmReleaseInterval=30s override, the existing 2m wait converges without the manual flux reconcile --force band-aid. Removing the workaround turns CI itself into the test of the fix. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
730037c85d |
fix(installer): address review feedback on cozy-system bootstrap
- Bump alpine/k8s to 1.33.4 (was 1.32.0, EOL); matches in-tree pin used by cozystack-api, seaweedfs, vm-disk, and vm-instance hooks. - Skip hostNetwork=true and not-ready/CNI tolerations on the hosted variant: managed clusters provide CNI before install, and kube-system may carry strict PSA there that would reject hostNetwork pods. - Add packages/core/installer/templates/cozy-system-namespace.yaml gated on .Values.bareNamespace (default false). The top-level Makefile's manifests target now passes --set bareNamespace=true and renders Namespace + operator into _out/assets/cozystack-operator-*.yaml so the kubectl-apply install path stays self-contained. Helm install/upgrade callers keep the default and bootstrap via --create-namespace + the pre-install labeler hook. - Add a bats assertion in hack/e2e-install-cozystack.bats checking cozy-system carries pod-security.kubernetes.io/enforce=privileged and cozystack.io/system=true after helm install. - Inline comment near hostNetwork=true documenting why dnsPolicy left at Default (cluster DNS not up at install time; ClusterFirstWithHostNet would not help). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
myasnikovdaniil
|
d9023c9b15 |
feat(cozyreport): collect Flux/cert-manager/host context + summary.txt (#2553)
## What this PR does Makes `cozyreport.tgz` actually useful when triaging an E2E failure deeper than a single pod log. Adds: - Flux controller logs (helm-controller, source-controller, notification-controller, kustomize-controller, last 2000 lines each) - Flux source resources (`HelmRepository`, `OCIRepository`, `GitRepository`, `ExternalArtifact`) - Decoded Helm storage secrets for non-Ready HRs (`base64 -d | base64 -d | gzip -d`) - Cluster events (all + warning-only filtered file) - cert-manager `Certificate` / `CertificateRequest` / `Order` / `Challenge` resources + cert-manager logs - `cozystack-operator` deployment logs (current + previous) - `Application` / `ApplicationDefinition` / `Tenant` resources - Sandbox host context per node: `df`, `free`, `ps`, `dmesg`, `talosctl logs/dmesg/kubelet/containerd` New executable `hack/cozyreport-summary.sh` writes a `summary.txt` at the archive root listing what is broken right now — the first thing to read when downloading the artifact from a CI failure. Surfaced from #2500. ### Release note ``` NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a timestamped diagnostic summary report that highlights failing resources and key cluster issues. * Expanded diagnostic collection: controller logs (current+previous), Flux source status, cert-manager certificate details, Helm release payloads, and Cozystack resource status. * Added host-level diagnostics (disk/memory/process/dmesg) and improved event/warning listings to aid troubleshooting. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
9df5950731 |
test(tests): pre-pull timing-sensitive platform images before install
Some workloads (kube-ovn raft, LINSTOR controller / piraeus-server) fail when replicas start at materially different times due to image-pull stagger across nodes. Add a DaemonSet-based pre-pull step that runs before `helm install`, ensuring every node has the images cached so replicas start within milliseconds of each other. The script accepts image refs on stdin and creates one container per image so pulls run in parallel — total time = max(any single image) rather than sum. The bats test sources image refs directly from the rendered platform charts via `yq`, walking only PodSpec-shaped objects, so version bumps stay in sync without a hardcoded list. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Assisted-By: Claude <noreply@anthropic.com> |
||
|
Myasnikov DaniilandClaude Opus 4.7
|
b40ab290eb |
fix(cozyreport): portability and Ready-filter fixes from review
- Resolve hack/cozyreport-summary.sh via SCRIPT_DIR so cozyreport.sh
can be invoked from any working directory, not only the repo root.
- Discover Talos node IPs via `kubectl get nodes` (with the existing
192.168.123.11-13 list as fallback) instead of hardcoding the E2E
sandbox topology.
- Replace broken `awk '$NF != "True"'` filter on the cozystack-apps
loop with a jsonpath check against the Ready condition; the previous
filter compared AGE ("10m" etc.) against "True" and let every
resource through, defeating the non-Ready filter.
- Switch the Flux source CRD summary loop from column parsing to
jsonpath (column positions differ between HelmRepository and
OCIRepository printer columns) and add ExternalArtifact, matching
what cozyreport.sh already collects.
- Add `kubectl get crd` guards before HR/source CRD queries so the
summary script does not abort under `set -eu` when Flux v2 is not
installed.
- Relabel "OOMKilled in last 30 min" to "Recent OOMKilled events
(last 20)" — the kubectl call only limits count, not time.
Surfaced from coderabbit and gemini review on #2553.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
Myasnikov Daniil
|
9a71dc1427 |
test(e2e): replace fixed timeouts with event-driven backstops
Sweep across `hack/*.bats` and `hack/*.sh` replacing fragile `sleep N` / `timeout N until …; do sleep …; done` patterns with `kubectl wait --for=condition=…` plus a small existence backstop: until kubectl get <resource> >/dev/null 2>&1; do sleep 2; done kubectl wait <resource> --for=condition=Ready --timeout=… Existence backstop added to 33 `kubectl wait` sites across 12 bats files (redis, kafka, openbao, harbor, mongodb, qdrant, vminstance, clickhouse, mariadb, postgres, bucket, install). Without it, `kubectl wait` against a not-yet-created resource sits silently for the full timeout; with it, the test fails fast with a clear "resource never appeared" error. OpenAPI tests' fixed `sleep 5` after starting `kubectl proxy` is now a port-readiness wait. The 3 remaining genuine sleeps (LINSTOR node membership propagation, MetalLB end-to-end HTTP path, HR-count emission heuristic) are annotated with TODO + rationale so a future reader knows they are intentional. Same green-path semantics; failures now surface in seconds instead of after a 5-minute timeout. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Assisted-By: Claude <noreply@anthropic.com> |
||
|
Myasnikov Daniil
|
4a5946a8a0 |
fix(virtual-machine): vminstance disk race + bump VM IP/ready timeouts
Three fixes for the vminstance bats, surfaced once the 3x retry was dropped from `Run E2E tests`: 1. Disk delete-recreate race. The teardown used `kubectl delete vmdisk … || true` and immediately created a new disk; sometimes the new disk reconciled before the old finalizer completed, leaving a duplicate PVC and the new VM stuck. Drop the `|| true`, block until the old disk is gone, then create. 2. VM IP wait timeout 20s -> 120s. Under nested-virt + concurrent load, a fresh VM sometimes takes 60-90s to acquire its IP via DHCP. The 20s bound was tripping deterministically. 3. VM ready timeout 20s -> 60s. Same root cause; the readiness probe needs the IP to be reachable, so the lower bound has to grow alongside the IP wait. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Assisted-By: Claude <noreply@anthropic.com> |
||
|
Myasnikov Daniil
|
aa72a232e6 |
feat(cozyreport): collect Flux/cert-manager/host context + summary.txt
`hack/cozyreport.sh` now collects materially more signal so a triage artifact can show what is broken rather than just dumping pod logs: - Flux controller logs (helm-controller, source-controller, notification-controller, kustomize-controller, last 2000 lines) - Flux source resources (HelmRepository, OCIRepository, GitRepository, ExternalArtifact) - Decoded Helm storage secrets for non-Ready HRs (`base64 -d | base64 -d | gzip -d`) - Cluster events (all + warning-only filter) - cert-manager Certificate / CertificateRequest / Order / Challenge resources + cert-manager logs - `cozystack-operator` deployment logs (current + previous) - `Application` / `ApplicationDefinition` / `Tenant` resources - Sandbox host context per node: df, free, ps, dmesg, talosctl logs/dmesg/kubelet/containerd New `hack/cozyreport-summary.sh` writes a `summary.txt` at the archive root listing what is broken right now — first thing to read when triaging an artifact. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Assisted-By: Claude <noreply@anthropic.com> |
||
|
Andrei Kvapil
|
b5a1ae57de |
feat(etcd): update operator to v0.4.3 and expose S3 backup schedules (#2428)
## What this PR does
Bumps the vendored `etcd-operator` chart from `v0.4.2` to `v0.4.3` and
wires the new backup machinery through the cozystack `Etcd` app API.
Upstream v0.4.3 ships two new CRDs — `EtcdBackup` (one-shot snapshot)
and `EtcdBackupSchedule` (periodic snapshots) — along with the
`EtcdCluster.spec.bootstrap` restore field, a new `OPERATOR_IMAGE` env
on the operator deployment, and a replacement of the deprecated
`gcr.io/kubebuilder/kube-rbac-proxy` image. The chart portion of this PR
is a clean re-vendor via `make update` — no local patches.
On top of the bump, `packages/extra/etcd` exposes a new optional
`backup.*` values block. When `backup.enabled=true`, the chart renders
an `EtcdBackupSchedule` referencing the platform `EtcdCluster/etcd` and
a `Secret/etcd-s3-creds` with the S3 credentials expected by the
backup-agent. The values field names (`destinationPath`, `endpointURL`,
`s3AccessKey`, `s3SecretKey`, `schedule`) mirror the existing `postgres`
backup pattern for consistency. The chart `fail`s at template render if
any of the four required S3 fields are empty or if `destinationPath` is
malformed (missing `s3://` prefix, `//` segments, no bucket).
Test coverage: 15 `helm-unittest` cases across
`tests/backup-secret_test.yaml` and
`tests/etcd-backup-schedule_test.yaml` (wired into `make test`, picked
up by `hack/helm-unit-tests.sh`), plus three bats cases in
`hack/e2e-apps/etcd.bats` covering the baseline `Etcd` app, the `backup:
{}` default-off path, and the full `backup.enabled=true` case that
verifies the `EtcdBackupSchedule` materializes and the operator
generates the downstream `CronJob`.
`packages/core/platform/sources/etcd-operator.yaml` now sets
`upgradeCRDs: CreateReplace` on the etcd-operator component, so Flux
applies the new CRDs on upgrade of the chart instead of silently
skipping them.
Depends on #2427.
### Scope deferred to follow-ups
- The one-shot `EtcdBackup` CRD is vendored but not exposed through this
chart — users can still hand-apply the CRD directly.
- `EtcdCluster.spec.bootstrap` (restore from an existing snapshot) is
likewise vendored but not exposed. Both gaps are called out in
`packages/extra/etcd/README.md`.
### Release note
```release-note
feat(etcd): update etcd-operator to v0.4.3 and expose S3 backup schedules on the Etcd app. Set `backup.enabled=true` along with `backup.s3AccessKey`, `backup.s3SecretKey`, `backup.destinationPath`, and `backup.endpointURL` to schedule periodic snapshots.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Scheduled S3 backups for etcd clusters (configurable cron, generated
S3 credentials) and restore/init from S3 or PVC via new backup
resources.
* **Tests**
* New end-to-end tests for etcd lifecycle and backup scenarios; added
Helm unit tests for backup templates and validations.
* **Documentation**
* Backup configuration guide and parameter reference added.
* **Chores**
* Operator chart version bump; RBAC, webhooks, and deployment wiring
updated; added chart test target.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Andrei Kvapil
|
e9351a6757 |
fix(tests): increase Bootstrap Talos cluster timeout to 60s (#2431)
## What this PR does The `Bootstrap Talos cluster` step in `hack/e2e-prepare-cluster.bats` wraps `talosctl bootstrap` in `timeout 10 sh -ec 'until talosctl bootstrap ...; do sleep 1; done'`. On a freshly booted Talos node NTP has not yet converged, so `talosctl bootstrap` returns `FailedPrecondition: time is not in sync yet` after ~5-6s. With the 10s outer timeout the embedded `until/sleep 1` loop only completes a single attempt before it is killed, and the E2E job fails during `Prepare environment`. On slower self-hosted runners NTP typically converges before the first attempt. On fast ephemeral runners (Oracle VMs) it does not, producing a consistent flake that only the outer workflow-level 3-attempt retry can paper over. Bump the outer timeout to 60s and the inter-attempt sleep to 2s. This aligns with the other bootstrap gates in the same test (etcd members check at 180s, drain RPC errors at 60s, node registration at 60s) and gives NTP plenty of time to converge. ### Release note ```release-note fix(tests): increase `Bootstrap Talos cluster` timeout in E2E from 10s to 60s to accommodate NTP convergence on fast ephemeral runners. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Enhanced cluster bootstrap process resilience by extending retry mechanisms and adjusting timing intervals to better tolerate transient failures during system initialization. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
c24a507911 |
test(gpu-operator): make variants bats test runnable under cozytest.sh
Build job in CI failed because the test relied on bats-specific machinery the in-repo cozytest.sh runner does not provide: * BATS_TEST_DIRNAME is undefined when the file is sourced as a plain shell script (cozytest.sh's mode); switch to BATS_TEST_FILENAME with a $0 fallback, matching hack/check-gpu-recording-rules.bats (REPO_ROOT pattern). * setup()/teardown() are bats lifecycle hooks; cozytest.sh runs each @test in a fresh subshell with set -u and never invokes them. With TMP only assigned in setup(), every test exited immediately on 'TMP: unbound variable'. Inline the mktemp + trap inside each @test so both runners work the same. Verified locally that bash hack/cozytest.sh hack/check-gpu-operator-variants.bats now reports all six tests OK. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
4a16c84832 |
test(gpu-operator): assert variant overlays render expected ClusterPolicy
Adds hack/check-gpu-operator-variants.bats. The test renders both
package variants with helm template and asserts the resulting
ClusterPolicy spec for the values that this PR cares about:
* ccManager.enabled=false (pinned in values.yaml; upstream v26 default
was flipped to true)
* vgpuDeviceManager.enabled=false (same — and on Ada+/Blackwell the
DaemonSet would crashloop because /sys/class/mdev_bus is empty)
* sandboxWorkloads.defaultWorkload — vm-passthrough on the default
variant, vm-vgpu on the vgpu variant; without these the upstream
'container' default makes either variant a silent no-op on
unlabelled GPU nodes
* vgpuManager.enabled=true on the vgpu variant; driver / devicePlugin
enabled=false on both variants (proprietary host driver path is
out of scope for both passthrough and SR-IOV vGPU)
Negative-tested locally: deleting any of these settings makes the
corresponding test fail, so it actually guards regressions on the
next chart bump.
Also addresses last-pass review comments:
* values-passthrough.yaml: set sandboxWorkloads.defaultWorkload=
vm-passthrough so the default variant is symmetric with the vgpu
variant — neither relies on per-node labels for the variant to
activate. Without this the upstream default 'container' would
apply on unlabelled GPU nodes and vfio-manager would never start.
* values.yaml: rewrite the vgpuDeviceManager comment to lead with
the load-bearing reason (mdev path missing on Ada+/Blackwell)
instead of the secondary 'stays at zero pods on passthrough'
rationalisation.
* docs/gpu-vgpu.md: add a 'Migrating from chart v25.x' subsection
in the Licensing section pointing at the configMapName ->
secretName rename. Old key still works upstream but emits a
deprecation warning; PR body release note already calls this out
but the docs were missing the migration breadcrumb.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
Aleksei Sviridkin
|
d15ec72327 |
fix(gpu-operator): pin upstream behavior changes; correct DLS licensing example
Addresses follow-up review of the vGPU variant + chart bump.
* values.yaml: pin ccManager.enabled=false and vgpuDeviceManager.enabled=false.
Upstream chart v26.x flipped both to true. ccManager auto-enables CC
mode on Hopper-class hardware after upgrade — a non-trivial behavior
change that should be opt-in. vgpuDeviceManager walks
/sys/class/mdev_bus/, which does not exist on Ada+ (L4/L40/L40S/B100
with vGPU 17/20). Pinning both off keeps the default footprint as it
was on v25.x; users explicitly opt in.
* hack/dcgm-default-counters.csv: refresh from dcgm-exporter tag
4.5.1-4.8.0 (matches the image bumped earlier in this PR). The
previous snapshot was pinned to 4.1.1-4.0.4. Adds DCGM_FI_DEV_FB_RESERVED
and other fields introduced upstream. Header comment updated.
hack/check-gpu-recording-rules.bats passes against the refreshed
fixture.
* docs/gpu-vgpu.md (Licensing section):
- Drop the wrong recommendation to wire the licensing Secret through
gpu-operator's driver.licensingConfig.secretName. That mounts into
the host driver pod, where SR-IOV vGPU does not consume a license.
Document guest-side delivery (cloud-init write_files for
/etc/nvidia/ClientConfigToken/ and /etc/nvidia/gridd.conf) end to
end.
- Correct FeatureType comment. Per the NVIDIA Virtual GPU Software
Licensing User Guide, FeatureType=0 is the unlicensed state, not
auto-detect; FeatureType=1 is the value that asks the driver to
auto-select the license type from the configured vGPU profile.
Switch the example to FeatureType=1 and list all four valid values
with their meaning.
- Add nvidia-smi verification snippet inside the guest.
* docs/gpu-vgpu.md (KubeVirt section): note that L40S SR-IOV VFs share
the PF device ID (10de:26b9), so a single pciVendorSelector matches
both. Recommend lspci -nn -d 10de: as a sanity check on other
generations where PF/VF IDs may differ.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
Aleksei Sviridkin
|
dff5f8d19a |
feat(kubernetes)!: replace ephemeral emptyDisk with persistent storage for worker nodes (#2454)
## What this PR does Replace the ephemeral `emptyDisk` volume on tenant Kubernetes worker VMs with persistent PVC-backed storage via KubeVirt `dataVolumeTemplates`. Previously, kubelet certificates, kubeconfig, and containerd state were lost on VM reboot, causing the node to fall out of the tenant cluster. **Changes:** - Rename `ephemeralStorage` field to `diskSize` in NodeGroup configuration - Replace `emptyDisk` volume with `dataVolumeTemplates` using CDI blank source - Add `cdi.kubevirt.io/storage.usePopulator: "false"` annotation to work around CDI volume population issues with LINSTOR CSI - Add optional per-nodeGroup `storageClass` field (defaults to cluster default) - Set fixed 2Gi ephemeral-storage resource for virt-launcher pod - Rename internal mount paths from `/ephemeral` to `/persistent` - Add helm unit tests (`cluster_test.yaml`) covering dataVolumeTemplates defaults, custom disk size, storageClass presence/absence, fixed ephemeral-storage, disk name consistency, and CDI populator annotation - Quote storage values in templates (`diskSize`, `storageClassName`) for Helm correctness - Guard bind-mount commands with `mountpoint -q` checks and create target directories to handle fresh images and cloud-init retries - Set `blockSize.matchVolume` on worker node disks for automatic block size detection, fixing DRBD 4Kn compatibility with QEMU **Known gotchas:** - CDI volume population (prime PVC rebind) causes `ClaimMisbound` errors with LINSTOR CSI driver, regardless of StorageClass (`replicated` or `local`). The `usePopulator: false` annotation switches to the legacy import path which works correctly. This is a known CDI issue documented in [kubevirt/containerized-data-importer#3146](https://github.com/kubevirt/containerized-data-importer/issues/3146). - The `replicated` StorageClass with DRBD `suspend-io` quorum policy can cause transient VM IO errors during LINSTOR satellite restarts (quorum loss). The `local` StorageClass is recommended for worker node disks. When `storageClass` is not specified in a nodeGroup, the cluster default StorageClass is used. - Existing clusters will trigger a rolling replacement of all worker nodes on upgrade due to changed KubevirtMachineTemplate hash. - DRBD volumes use 4096-byte logical sectors (`block-size 4096`). Without explicit block size configuration, QEMU defaults to 512-byte sectors, causing immediate IO errors and VM pause. The `blockSize.matchVolume` setting in the disk spec ensures KubeVirt auto-detects the correct sector size from the volume. **Tested on a live cluster:** - Verified DataVolume provisioning with `local` StorageClass - Verified nodes join the tenant cluster and reach Ready state - Verified node survives VM reboot (pod deletion) and returns to Ready with the same Node object, reusing kubelet certificates from persistent disk ### Screenshots N/A — no UI changes. ### Release note ```release-note feat(kubernetes)!: replace ephemeral emptyDisk with persistent storage for worker nodes Worker node VMs now use persistent PVC-backed disks instead of ephemeral emptyDisk volumes. Kubelet certificates, kubeconfig, and containerd state survive VM reboots, preventing nodes from falling out of the tenant cluster. BREAKING CHANGE: The `ephemeralStorage` field in nodeGroups has been renamed to `diskSize`. A new optional `storageClass` field allows per-nodeGroup storage class selection. Users must update their values overrides accordingly. Worker node disks now use `blockSize.matchVolume` to automatically detect the underlying volume's sector size, ensuring compatibility with DRBD-backed storage classes that use 4096-byte logical sectors. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Optional storageClass parameter to select storage class for worker node persistent disks (defaults to cluster default). * Makefile test target added to run Helm unit tests for the chart. * **Configuration Changes** * Worker node storage parameter renamed from ephemeralStorage to diskSize (default: 20Gi). * Kubelet/containerd state moved to persistent disk paths and node ephemeral-storage accounting fixed to 2Gi; disk provisioning aligns block sizing with backing volumes. * **Documentation** * Updated docs and chart/schema to reflect the persistent disk model and new fields. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
ohotnikov.ivan
|
f8001e23b6 |
fix(postgres): address review feedback
- Regenerate deepcopy artifacts to remove orphaned PostgreSQLParameters methods - Update e2e test to use string value for max_connections parameter - Fix template to avoid .Values mutation and use quote function for safe YAML escaping Signed-off-by: ohotnikov.ivan <ohotnikov.ivan@e-queo.net> |
||
|
Arsolitt
|
998cc77635 |
chore(kubernetes): merge main to resolve CRD schema conflict
Integrate the HAMi GPU virtualization addon from main while keeping the persistent-storage diskSize and storageClass fields introduced by this branch. Signed-off-by: Arsolitt <arsolitt@gmail.com> |
||
|
Aleksei Sviridkin
|
3977aef509 |
feat(monitoring): add GPU observability dashboards and recording rules (#2418)
## What this PR does Adds GPU observability to Cozystack: five Grafana dashboards, a set of VictoriaMetrics recording rules with a throttle-regression alert, and reference manifests for wiring the NVIDIA GPU Operator and DCGM Exporter on Talos when running GPU workloads directly in pods. ### Components 1. **Dashboards** — `dashboards/gpu/*.json`, registered in `packages/system/monitoring/dashboards-infra.list`. All follow the project convention: the Prometheus data source is selected through the `$ds_prometheus` template variable. - `gpu-performance` — per-GPU utilization (NVML, tensor pipe, graphics engine, memory copy), VRAM, power, temperature, and health (XID errors, power/thermal throttling). All metrics are shown per-GPU, filterable by the `$Hostname` selector. - `gpu-fleet` — cluster-wide inventory, capacity, utilization, and per-node power draw as percentage of TDP. - `gpu-tenants` — per-namespace allocation and 24-hour GPU-hour/kWh accounting; utilization and power are shown per-node (DCGM cannot attribute hardware metrics to namespaces). - `gpu-quotas` — allocated vs. requested GPUs, pending pods, and a per-namespace requests/limits table. All counts derive from the `namespace:gpu_count:allocated` recording rule, which excludes `Failed`/`Succeeded` pod carryover. - `gpu-efficiency` — cluster and per-GPU efficiency: tensor saturation, util-per-watt, and per-GPU power/thermal throttle fractions. 2. **Recording rules and alert** — `packages/system/monitoring-agents/alerts/gpu-recording.rules.yaml`. Four rule groups: - `gpu.recording.cluster.1m` — cluster-wide totals consumed by overview panels. - `gpu.recording.node.1m` — per-node hardware aggregates (grouped by `Hostname`) plus per-namespace allocation counts from kube resource requests; filters `cozy-*`/`kube-*`. - `gpu.recording.efficiency.1m` — per-GPU tensor saturation and util-per-watt, plus per-GPU power/thermal throttle fractions bounded via `clamp_max(..., 1)`. - `gpu.recording.throttle.validation.5m` — houses the `GPUThrottleFractionOverOne` warning alert that fires if the pre-clamp throttle fraction exceeds 1.0. This guards against DCGM counter-unit drift between GPU families; the `/1e9` divisor is verified on NVIDIA A10 with DCGM 3.x. Rules are safe on clusters without DCGM — they evaluate to empty series when no matching metrics are scraped. 3. **Reference manifests** under `packages/system/gpu-operator/examples/`. These are **not** templates; they document one working configuration: - `values-native-talos.yaml` — Cozystack Package values for the native-pod scenario. - `dcgm-custom-metrics.yaml` — ConfigMap extending the DCGM CSV with profiling, ECC, throttling, and energy counters used by the dashboards and recording rules. - `nvidia-driver-compat.yaml` — DaemonSet that stages `libnvidia-ml.so.1` and `nvidia-smi` from the Talos glibc tree into a location the GPU Operator validator inspects. Workaround for [NVIDIA/gpu-operator#1687](https://github.com/NVIDIA/gpu-operator/issues/1687). Requires `pod-security.kubernetes.io/enforce: privileged` on the target namespace when the cluster enforces baseline/restricted PodSecurityStandards. - `README.md` — explains the two deployment paths (sandbox vs. native), which DCGM metrics each dashboard depends on, and how the pieces fit together. 4. **Cross-validation test** — `hack/check-gpu-recording-rules.bats` enforces that every recording-rule reference in tracked dashboards resolves, and that every DCGM metric referenced by the rules is declared in either the upstream default CSV snapshot (`hack/dcgm-default-counters.csv`) or the custom CSV shipped under `examples/`. ### Why reference manifests, not templates The out-of-the-box `packages/system/gpu-operator/values-talos.yaml` targets **sandbox workloads** — GPUs passed through to KubeVirt VMs via VFIO, driver disabled on the host, device plugin disabled. That is the sensible default for Cozystack today. Running GPU workloads directly in pods on Talos is also possible but requires a different configuration that makes assumptions not every user shares: - the NVIDIA Talos system extension must be installed on GPU nodes, - the operator's own driver and toolkit components must be disabled because the extension provides them, - the validator's hardcoded paths need a workaround until NVIDIA/gpu-operator#1687 lands upstream. Shipping those as active templates would silently impose those assumptions. Shipping them as reference files next to the package lets operators opt in with eyes open and understand the moving parts before applying them. ### Scope note The dashboards and recording rules are useful independently of the reference manifests — any cluster that scrapes DCGM Exporter (however it is installed) will populate them. The reference manifests are self-contained and do not change any default behavior of the `gpu-operator` package. ### Screenshots <img width="2824" height="1055" alt="2026-04-27_12-36-52" src="https://github.com/user-attachments/assets/71f85de7-1c80-4817-b36c-aeb1c2bdd39a" /> <img width="2913" height="1255" alt="2026-04-27_12-36-39" src="https://github.com/user-attachments/assets/7a54daca-0727-4510-938c-080d31bdc229" /> <img width="2932" height="1251" alt="2026-04-27_12-36-30" src="https://github.com/user-attachments/assets/f6c6c215-3d05-400c-b21c-29f267c93f23" /> <img width="2864" height="1222" alt="2026-04-27_12-36-00" src="https://github.com/user-attachments/assets/e9d34bab-ade5-434b-92c3-f36712cf68d9" /> <img width="2958" height="1226" alt="image" src="https://github.com/user-attachments/assets/0047a5f8-12a5-4cda-8676-041de139cb76" /> ### Release note ```release-note feat(monitoring): add GPU observability — five Grafana dashboards (performance, fleet, tenants, quotas, efficiency), VictoriaMetrics recording rules with a throttle-regression alert, and reference manifests under packages/system/gpu-operator/examples/ for running GPU workloads natively in pods on Talos. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added five GPU dashboards for performance, efficiency, fleet overview, quotas/allocation, and tenant usage (utilization, power, throttling, allocation, billing). * Added cluster recording rules and a throttling validation alert; new Prometheus-derived metrics and per-GPU rankings/timeseries. * **Documentation** * Added GPU Operator examples and README documenting native Talos setup and required DCGM counters. * **Tests** * Added validation tests ensuring dashboard ↔ recording-rule and DCGM metric consistency. * **Chores** * Registered new GPU dashboards in infra listings. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
e197aba625 |
feat(operator): make HelmRelease Interval configurable, override to 30s in E2E
Adds --helmrelease-interval (default 5m, matching today's behaviour) to the operator and a corresponding cozystackOperator.helmReleaseInterval value on the cozy-installer chart (default empty -> no flag rendered -> operator default 5m applies). E2E install sets it to 30s. Rationale: with the 5m default, dependency- blocked HRs (waiting on cert-manager webhooks, CRDs) are requeued only every 5 min, producing an 8-10 min dead zone in the E2E install where the fast pack of HRs is Ready but the slow tail hasn't been retried yet. 30s collapses the gap. Production defaults are unchanged. The override is opt-in per install. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Arsolitt
|
633ff1a617 |
chore(kubernetes): merge main to resolve CRD schema conflict
Incorporate images.waitForKubeconfig support from main while keeping the persistent-storage diskSize and storageClass fields introduced by this branch. Signed-off-by: Arsolitt <arsolitt@gmail.com> |
||
|
Aleksei Sviridkin
|
cc7d09b458 |
fix(tests): use far-future cron in etcd backup e2e to avoid job churn
Address review feedback from coderabbitai on hack/e2e-apps/etcd.bats:89: the test verifies that the chart renders EtcdBackupSchedule and that the etcd-operator materializes a CronJob, not that backups fire. Use "0 0 1 1 *" so no Jobs run against the intentionally invalid endpoint during the ~6-minute test window. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
97eef022e5 |
fix(tests): poll for etcd HelmRelease existence before waiting on readiness
Address review feedback from coderabbitai on hack/e2e-apps/etcd.bats:39-40: replace the sleep-then-wait pattern with a wait_etcd_hr_ready helper that polls hr/etcd until the Cozystack API reconciles the Etcd CR and only then waits for condition=ready. kubectl wait exits immediately when the resource does not yet exist, which caused spurious failures when HelmRelease creation took longer than 5s. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
86a93af283 |
fix(tests): stop masking etcd e2e cleanup failures
Address review feedback from coderabbitai on hack/e2e-apps/etcd.bats:9-14: drop the '|| true' guards in setup() so real teardown failures (RBAC errors, stuck finalizers, wait timeouts) fail the test setup instead of letting the next test run against stale HelmRelease, Secret, or EtcdBackupSchedule state. --ignore-not-found still covers the empty-namespace case on a clean run. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
f8f3b29480 |
fix(tests): correct stderr redirection order in etcd diagnostics
Address review feedback from coderabbitai on hack/e2e-apps/etcd.bats:19-21: redirect stdout to fd 3 first, then dup stderr onto the new fd 1 so failures from kubectl (NotFound, RBAC, timeouts) reach the test diagnostics stream instead of being swallowed by the bats-captured default stdout. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Arsolitt
|
7b05ce0573 |
fix(kubernetes): update e2e test to use renamed diskSize field
The e2e test still used the old ephemeralStorage field name, which triggers the migration guard fail() in the Helm template. Signed-off-by: Arsolitt <arsolitt@gmail.com> |
||
|
Aleksei Sviridkin
|
4d7b4bbb33 |
fix(tests): replace bats run idiom in etcd e2e test
cozytest.sh does not implement the bats 'run <cmd>' helper and '$status' variable, so the 'Create Etcd with empty backup block (disabled by default)' test died with 'run: not found' on every attempt. Replace the bats idiom with plain shell: use '!' prefix and redirect stderr to assert the resources absent-when-backup-disabled invariant directly. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
8cae6f79d1 |
fix(tests): increase Bootstrap Talos cluster timeout to 60s
On fast ephemeral runners a freshly booted Talos node has not yet converged NTP when the previous 10s outer timeout fired, so the embedded 'until talosctl bootstrap; do sleep 1; done' retry loop only got one 5-6s attempt before the loop was killed. Every RPC returned 'FailedPrecondition: time is not in sync yet' and E2E failed in the Prepare environment step. Bump the outer timeout to 60s and the inter-attempt sleep to 2s. This aligns with the surrounding bootstrap gates (etcd members check at 180s, drain RPC errors at 60s, node registration at 60s) and gives NTP plenty of time to converge on ephemeral runners. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
ed77e9c6e3 |
fix(etcd): add helm unit tests, simplify backup guards, harden e2e
Fourth review round fixes. Simplify backup validation by dropping placeholder-regex guards and switching default values of s3AccessKey, s3SecretKey, destinationPath, and endpointURL from angle-bracketed hints to empty strings. A single 'required' check on each field is now the whole validation for presence; the regex guard was brittle (silently stopped matching if the hint ever changed) and dropping it makes the values.yaml defaults match the behavior contract the README describes. s3:// prefix and '//' rejection still live in etcd-backup-schedule.yaml as template-time fail branches. Add helm-unittest suites in packages/extra/etcd/tests/ covering both templates end-to-end: rendered output assertions for the happy path (bucket/key split, region passthrough, conditional rendering of optional fields) and failedTemplate assertions for every fail branch. A 'test' target is wired into the package Makefile so hack/helm-unit-tests.sh picks it up on 'make unit-tests'. Add type: Opaque to backup-secret.yaml for consistency with the mongodb template. E2E: wait for HelmRelease, Secret, and EtcdBackupSchedule deletion in setup() so the empty-backup case does not flake against lingering state from the previous test. Replace the fictitious minio-e2e hostname with no-such-endpoint.invalid and explicitly document that the e2e verifies schedule/CronJob materialization, not backup execution. Restore via EtcdCluster.spec.bootstrap and the one-shot EtcdBackup CRD are still not exposed through this chart; both are explicit follow-ups called out in the README. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
12fc9cd415 |
fix(etcd): address third review round
- Use name='etcd' in all three e2e cases. The chart pins the Helm release name to 'etcd' via check-release-name.yaml, so any other name causes template rendering to fail before the HelmRelease can reconcile. This is why the tenant chart hardcodes HR name 'etcd' in packages/apps/tenant/templates/etcd.yaml. - Consolidate all backup validation guards into etcd-backup-schedule.yaml so Helm's alphabetical render order no longer determines which failure a user sees first. Previously, a bad destinationPath would be masked by the earlier placeholder check in backup-secret.yaml. - Use regex match '^<.*>$' for placeholder detection instead of exact string equality. The previous check silently stopped firing if anyone updated the default placeholder string in values.yaml; the regex form catches any angle-bracketed default. - Reject destinationPath strings containing '//' after the s3:// prefix. Previously 's3://bucket//subdir' silently produced a key beginning with '/', confusing S3 clients and retention policies. - Document the backup contract in README.md: what enabling it produces, which fields are required, why placeholders are rejected, and that restore is a known follow-up gap. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
561bc4dc23 |
fix(etcd): address second review round
- Mark backup.enabled as optional in the values typedef so
generated JSON schema no longer lists it in 'required'.
Previously, 'backup: {}' (relying on the enabled:false default)
failed schema validation at CRD admission.
- Validate backup.destinationPath has s3:// prefix. Without this
check, passing a path without the scheme silently used the
first path segment as the bucket name.
- Add tail newline to values.schema.json to match repo convention
(cozyvalues-gen output lacks it — worth fixing in that project
separately).
- Add bats setup() that clears tenant-test etcd resources before
each test so a failure mid-test does not leak into subsequent
tests. Hardcoded chart-level resource names ('etcd',
'etcd-s3-creds') make two Etcd apps in the same namespace
collide; setup() enforces a single owner per test.
- Add dump_diagnostics() helper that dumps operator logs + the
relevant CRD state on wait-for-ready timeouts, so CI failures
surface actionable context instead of bare exit code 124.
- Add a third e2e case 'Create Etcd with empty backup block' that
asserts backup:{} admits and produces no backup resources — a
regression test for the required:['enabled'] bug fixed here.
Known scope: the upstream EtcdCluster.spec.bootstrap restore field
(also new in v0.4.3) is not exposed through this values.yaml yet.
Restoring a cluster from an existing snapshot is a follow-up PR.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|