These two recipes spell out their buildx flags instead of using $(BUILDX_ARGS), so they never received --builder and ran on the default docker driver. After the mode=max migration that became a hard failure -- "Cache export is not supported for the docker driver" -- since the embedded docker driver cannot export a registry cache. Add the same conditional --builder flag $(BUILDX_ARGS) carries, so WRITE_CACHE=1 builds route to the docker-container builder; an empty BUILDER still emits no flag for local builds.
Part of #2937.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
`--cache-to type=inline` only exports the final stage. 25 of 38 Dockerfiles are multistage, so the expensive `builder` stages (Go/Node compiles) were never cached even with a warm tag. Switch every image to a shared mode=max registry cache.
- New `cache-args` macro (hack/common-envs.mk) emits `--cache-from` always and `--cache-to … mode=max` only when WRITE_CACHE=1, so PR builds stay read-only and concurrent PRs never race on the cache ref (the 409 class #2711 fixed for tags). oci-mediatypes/image-manifest keep the cache manifest portable across registries.
- Cache ref is CACHE_REGISTRY/<img>:buildcache, co-located with the build registry (CACHE_REGISTRY now defaults to $(REGISTRY)).
- New build-main.yaml warms :buildcache on every push to main (WRITE_CACHE=1), serialized via concurrency.
All 31 package image recipes converted from the inline cache-from/cache-to pair to `$(call cache-args,…)`; ubuntu-container-disk keeps a per-k8s-version cache tag. Verified with `make -n image` across all packages in both PR (read-only) and main (mode=max write) contexts.
Part of #2937.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Cozystack overrides the upstream seaweedfs volumeSizeLimitMB from
1000 to 30000 (30 GB), but the e2e sandbox provisions only a 10 GiB
PVC per volume server. With maxVolumes=0 (auto), seaweedfs computes
maxVolumes as int(10240 / 30000) → 0, then rounds up to 1, so each
volume server can host exactly 1 logical volume.
With defaultReplication=001 (one replica on a different node within
the same rack) two volume servers share that single volume, giving
the cluster a hard limit of one writable collection at a time.
Since #2716 added a default bucket-cozy-backups Bucket to every
tenant-root, that bucket now grabs the only slot on install. Any
second collection — the bucket E2E (bucket-test in tenant-test),
a tenant app's own bucket, etc. — sits in 'assign volume: No
writable volumes' forever, and mc cp hangs until the test times
out. Master logs reflect this clearly:
failed to find writable volumes for collection:bucket-...
replication:001 ttl:
error: No writable volumes and no free volumes left
topo failed to pick 1 from 0 node candidates
volume_growth.go:135 create 6 volume, created 0:
Not enough data nodes found!
Lower volumeSizeLimitMB back to the upstream chart default of 1000,
which makes a 10 GiB PVC fit ~10 logical volumes per server and
unblocks all per-collection bucket workloads on small sandboxes.
Production clusters still get plenty of headroom — volume size is a
soft cap, not a preallocation.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
The seaweedfs 4.31 upstream bump (#2834) switched the s3 Service name
from "$chartname-s3" (built with the seaweedfs.name helper, evaluates
to "seaweedfs-s3") to the seaweedfs.componentName helper, which is
release-name based and now evaluates to "$release-s3" — "seaweedfs-
system-s3" for the tenant-root deployment.
The bats E2E asset hack/e2e-apps/bucket.bats hard-codes
"service/seaweedfs-s3" for the port-forward that sets up its
S3 client; backup-controller and other in-cluster consumers also
referenced the legacy name. After the bump, the legacy Service is
gone and the port-forward immediately exits, the test then times out
on the 30-second nc poll for localhost:8333.
Restore the legacy Service name. Internal references inside the chart
(s3-deployment -filer=, etc) keep using componentName for the filer
endpoint, so this change only re-exposes the public-facing name
without touching the internal topology.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
The seaweedfs 4.31 upstream bump (#2834) rewrote
packages/system/seaweedfs/charts/seaweedfs/templates/cosi/cosi-bucket-class.yaml
from scratch and silently dropped the supplementary COSI resources
that cozystack ships on top of the upstream chart:
- BucketClass {name}-lock (object lock, COMPLIANCE mode, 365 days
retention) — used by apps that store immutable backups
- BucketAccessClass {name} parameters.accessPolicy: readwrite — the
explicit grant for the default access class
- BucketAccessClass {name}-readonly (accessPolicy: readonly) — used
by every Bucket app that declares readonly users
apps/bucket still renders BucketAccess resources that reference
{name}-readonly, but the class no longer exists, so the COSI
controller never picks the BucketAccess up: no finalizer is added,
status.accessGranted stays empty, and the bucket E2E test waits 300s
and times out on the viewer access. The admin (readwrite) access
class still works because it falls back to the default BAC.
Restore the three resources so the system chart again matches the
shape that packages/extra/seaweedfs/templates/client/cosi-bucket-class.yaml
already ships for the client topology (where this never regressed).
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
The capi-controller-manager liveness probe has no initialDelaySeconds
(defaults to 0), giving the container only 30s (3×10s) to serve /healthz.
In controller-runtime, the health endpoint starts after the webhook server,
which needs its TLS cert from cert-manager. During a loaded install,
cert-manager is busy issuing many certs simultaneously and the CAPI webhook
cert can take >30s — causing a kill loop (6+ restarts observed in CI).
A startupProbe with failureThreshold=30 gives up to 300s for the initial
startup, after which the regular livenessProbe takes over for ongoing
health monitoring. This is the standard pattern for slow-starting
containers with external dependencies.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Adds `spec.timeout: 10m` to the `kubevirt-csi-node` HelmRelease template
in `packages/apps/kubernetes/templates/helmreleases/csi.yaml`, aligning
it with every other tenant addon HR in the same directory (cilium,
coredns, ingress-nginx, etc. — all set `timeout: 10m`). `csi.yaml` was
the only one missing it; commit 9f9d8f85 ("Align timeouts for
HelmReleases") missed this template.
## Symptom
E2E NFS test times out waiting for `nfs-test-pod` to reach `Succeeded`.
Cozyreport events show the tenant `csi` HR running an Install →
Uninstall → Install remediation cycle. With
`install.remediation.retries: -1` and the default 5m Helm timeout,
kstatus marks the `kubevirt-csi-node` DaemonSet `InProgress` past 5m on
a slow tenant control-plane bringup (Kamaji APIs flapping, VM image
pulls), helm-controller times out, the remediation uninstalls the
DaemonSet, then reinstalls. Any volume mount in flight at that moment is
wedged.
## Origin
Single commit lifted unchanged from #2619.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated the Container Storage Interface (CSI) Helm release settings to
include a `spec.timeout` of **10 minutes** when the etcd namespace value
is set (keeping the existing `spec.interval` at **5 minutes**).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Add `rm -rf charts/prometheus-operator-crds` before `helm pull` in the
update target so stale files from previous chart versions don't linger
across updates. This matches the convention used by cilium,
external-secrets-operator, external-dns, and cert-manager Makefiles.
Address review feedback from gemini-code-assist on
packages/system/prometheus-operator-crds/Makefile:9.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The chart previously stripped the bundle to 4 CRDs (ServiceMonitor,
PodMonitor, PrometheusRule, Probe) — the set VictoriaMetrics operator
needs for service discovery. This was correct when the chart was
vendored inside system/victoria-metrics-operator (commit ba5c785cc,
2023), but the prune was carried over when the chart was extracted
to its own package, and again when the CRDs were propagated into
tenant Kubernetes clusters (commit 1e2b66131, 2025).
In tenant clusters the partial set blocks third-party apps that ship
their own prometheus-operator (e.g. kube-prometheus-stack) — the
missing CRDs (Prometheus, Alertmanager, AlertmanagerConfig,
PrometheusAgent, ScrapeConfig, ThanosRuler) prevent the operator from
deploying its instances against the platform-managed CRD set.
Drop the rm -f line in the Makefile so the full upstream bundle ships,
and pin --version 25.0.0 to keep the operator CRD version at v0.87.0
(operator version bump is a separate concern, out of scope here).
Tenant apps can now set crds.enabled: false on their kube-prometheus-
stack release and consume the platform CRDs directly.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Aligns kubevirt-csi-node HR with every other tenant addon HR in
packages/apps/kubernetes/templates/helmreleases/ (cilium, coredns,
ingress-nginx, etc.) — all set timeout: 10m, csi.yaml was the only
exception. 9f9d8f85 ("Allign timeouts for HelmReleases") missed it.
Symptom: e2e NFS test (hack/e2e-apps/run-kubernetes.sh) times out
waiting for nfs-test-pod to reach Succeeded. Cozyreport events show
the tenant csi HR running Install -> Uninstall -> Install remediation
cycles. With install.remediation.retries: -1 and the default 5m Helm
timeout, kstatus marks the kubevirt-csi-node DaemonSet 'InProgress'
past 5m on a slow tenant control-plane bringup (Kamaji APIs flapping,
VM image pulls), helm-controller times out, the remediation uninstalls
the DaemonSet, then reinstalls. Any volume mount in flight at that
moment is wedged.
Observed on PR #2619 CI run 26142986969: csi HR remediation at
~07:07 happened ~10 min into the nfs-test-pod wait, killing the
mount and timing out the pod-Succeeded condition.
10m matches the surrounding HRs and gives the DaemonSet headroom
without changing remediation semantics on truly-broken installs.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 866588ff61)
The 4.31 chart bump (#2834) broke E2E for every PR: tenant-root
seaweedfs-system never becomes ready.
1. S3 served plaintext. Until 4.05 the -cert.file/-key.file args were
rendered whenever enableSecurity was set, so weed s3 served TLS on the
main port (8333); 4.31 nested them under 'if s3.httpsPort' and we ship
httpsPort=0, so s3 spoke HTTP while the HTTPS probes, the ingress
backend-protocol and the COSI sidecar endpoint all expected HTTPS
('server gave HTTP response to HTTPS client'). Restore the 4.05
behaviour via patch.
2. COSI provisioner ServiceAccount mismatch. cosi-service-account.yaml
and the ClusterRoleBinding name the SA
<serviceAccountName>-objectstorage-provisioner, but the Deployment
referenced componentName (<release>-objectstorage-provisioner). When
serviceAccountName != release name (tenant-root-seaweedfs vs
seaweedfs-system) the Deployment could not create pods
('serviceaccount not found'). Align the Deployment with the SA + binding.
Both fixes are vendored as patches re-applied by 'make update', and
covered by helm-unittest.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Closes#2765.
## What this PR does
Makes the passthrough and vGPU variants of `cozystack.gpu-operator`
produce a KubeVirt CR that is immediately usable, without an out-of-band
`kubectl patch`. Today the gpu-operator stack reports green,
`nvidia.com/<model>` shows up under node `.status.allocatable`, but
every `VirtualMachine` referencing the resource via
`domain.devices.hostDevices` is rejected by the admission webhook
because `HostDevices` is not in `developerConfiguration.featureGates`
and the resource is not listed under `permittedHostDevices`. That manual
reconciliation is now driven by the platform.
The change has three parts.
`packages/system/kubevirt` now parameterizes the KubeVirt CR template.
`developerConfiguration.featureGates` keeps its existing static defaults
and appends `.Values.extraFeatureGates`;
`spec.configuration.permittedHostDevices` renders verbatim from
`.Values.permittedHostDevices` when set;
`spec.configuration.mediatedDevicesConfiguration` does the same.
Defaults stay empty so the rendered CR is byte-identical to the previous
template when no overrides are supplied. The chart's `make update`
target re-applies the four sed transforms after fetching the upstream
CR, so refreshing the KubeVirt release version stays a single command.
`packages/core/platform` adds two defaults tables under `files/`.
`gpu-passthrough-defaults.yaml` covers Hopper (H100/H200), Ada Lovelace
(L4/L40/L40S), Ampere (A100 PCIe/SXM, A40, A30, A10), Turing (T4) and
Volta (V100/V100S); PCI vendor:device pairs are stable, `resourceName`
slugs follow the `<arch>_<model>_<form>_<mem>` convention
`nvidia-sandbox-device-plugin` generates as of v25.x, and
`externalResourceProvider: true` is set everywhere because those
resources are advertised by the sandbox plugin rather than KubeVirt's
in-tree device plugin. `gpu-vgpu-defaults.yaml` covers the Pascal–Ampere
mdev profile range (A100-40C/80C, A40-24Q/48Q, A30-24C, A10-24Q,
V100D-32C, T4-16Q) — the same family span the upstream
`vgpu-device-manager` walks `/sys/class/mdev_bus/` for. Ada Lovelace and
Blackwell SR-IOV vGPU is intentionally out of scope for the default
table and is expected to be advertised via the user-override hook
described below.
`templates/bundles/iaas.yaml` now selects between the two tables via a
new `bundles.iaas.gpuOperatorVariant` knob (`default` for passthrough,
`vgpu` for mediated; default is passthrough). The same variant flows
through to the gpu-operator Package CR, replacing the previous
always-default call. When the package is enabled, the platform injects
`HostDevices` into `extraFeatureGates` plus the chosen defaults into
`permittedHostDevices` for the kubevirt component's values. Operators
extend or replace via top-level platform values:
```yaml
gpu:
replaceDefaults: false # append (default) vs. wipe-and-replace
permittedHostDevices:
pciHostDevices: [] # cluster-specific PCI passthrough entries
mediatedDevices: [] # cluster-specific vGPU entries
mediatedDevicesConfiguration: {} # extra mediated-device types / per-node mappings
```
`docs/gpu-vgpu.md` is updated to drop the manual `kubectl patch` step
and describe the auto-wiring contract, the per-variant defaults, the
escape hatches, and the sandbox-device-plugin `resourceName` slug
convention so an operator whose node publishes a different name knows
where to override.
## Test coverage
`helm-unittest` covers eight branches in
`packages/core/platform/tests/bundles_gpu_kubevirt_wiring_test.yaml`:
gpu-operator disabled emits no extras; gpu-operator enabled at default
variant emits the HostDevices gate plus `pciHostDevices`; vgpu variant
switches to `mediatedDevices` plus `mediatedDevicesConfiguration`; user
overrides append by default; `replaceDefaults: true` wipes the platform
table; an unknown variant fails fast with a descriptive error.
`packages/system/kubevirt/tests/kubevirt_cr_test.yaml` pins the
empty-values render and the three populated shapes at the CR template
level. `make helm-unit-tests` passes locally on the full repo.
### Release note
```release-note
feat(platform): the KubeVirt CR is now auto-configured for GPU host-device passthrough (HostDevices feature gate plus permittedHostDevices populated from NVIDIA defaults) whenever cozystack.gpu-operator is in bundles.enabledPackages. Operators no longer need a manual kubectl patch to make VMs schedule against a GPU. The variant is selected via bundles.iaas.gpuOperatorVariant (default: passthrough; vgpu: mediated). Extend or replace the NVIDIA defaults via .gpu.permittedHostDevices and .gpu.replaceDefaults.
**Upgrade note:** the bundle now OWNS spec.configuration.permittedHostDevices on the KubeVirt CR. If you previously hand-edited it (kubectl edit kubevirt), the first reconcile after upgrade overwrites your manual entries with the rendered NVIDIA default table. Before upgrading, move any custom entries into .gpu.permittedHostDevices (set .gpu.replaceDefaults: true to keep only your list) and verify each resourceName against what your nodes advertise (kubectl describe node <node> | grep nvidia.com/) — the default table now carries the exact slug the sandbox device plugin generates, which upper-cases the card's PCI-IDs name and keeps the Tesla/GL tokens (e.g. a Tesla T4 is nvidia.com/TU104GL_TESLA_T4), but a different plugin build can publish a different string; a mismatch makes the admission webhook reject GPU VMs on their next restart or migration.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Bundle-managed GPU Operator with selectable passthrough vs vGPU
variants; platform now injects GPU settings into KubeVirt and forwards
vGPU manager overrides when enabled.
* Default platform mappings for PCI passthrough and vGPU mediated-device
resources added.
* **Documentation**
* Expanded GPU/vGPU deployment guide with bundle-managed vs manual
guidance, wiring/precedence rules, resource-name conventions, and
verification steps.
* **Tests**
* New and expanded tests for bundle→KubeVirt wiring, variant behaviors,
default/override merging, and idempotent template parameterization.
* **Chores**
* Added Helm test target and hardened KubeVirt template
update/idempotency checks.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Add a third architectural variant to `cozystack.gpu-operator` for hosts
where the NVIDIA driver and `nvidia-container-toolkit` are already
installed by the distro package manager (the typical Ubuntu/Debian GPU
node shape: apt-installed driver + apt-installed CT).
The existing variants do not produce a working setup in that shape.
`default` (passthrough) runs `vfioManager`, which unbinds the host
driver to bind `vfio-pci` — with the host driver active,
`k8s-driver-manager` early-exits and the bind never happens, so the GPU
never reaches a pod. `vgpu` requires the proprietary NVIDIA vGPU host
driver and a license server; the user-installed open or proprietary
driver does not satisfy it.
The new `container` variant maps onto upstream gpu-operator's third mode
(`sandboxWorkloads.enabled=false` + `devicePlugin.enabled=true`) with
every component that would conflict with the host install switched off:
- `driver.enabled=false` — use the host driver.
- `toolkit.enabled=false` — host already has `nvidia-container-toolkit`.
Enabling the operator's toolkit DaemonSet would overwrite the
`/etc/containerd/config.toml` the host package configured (via
`nvidia-ctk runtime configure`), breaking the host runtime wiring.
Matches NVIDIA's documented guidance and the existing
`examples/values-native-talos.yaml` reference, which pins the same flag
for the same reason.
- `devicePlugin.enabled=true` — publish `nvidia.com/gpu` to the kubelet.
- `vfioManager.enabled=false` — do not unbind the host driver. Upstream
renders this `true` unconditionally; it only stays at zero pods because
no node carries a `vm-passthrough` sandbox assignment when
`sandboxWorkloads` is off — pinned off so the rendered policy matches
intent.
- `cdi.enabled=false` — upstream defaults this `true`, but CDI spec
generation is serviced by the operator-managed toolkit DaemonSet this
variant disables. An apt host doesn't auto-generate CDI specs
(`nvidia-ctk cdi generate` is a manual step) and uses the classic
`nvidia-container-runtime` hook, so leaving CDI on would point the
device plugin at CDI injection with no specs on the host and silently
break allocation.
- `sandboxWorkloads.enabled=false` — regular pod workloads, not VMs.
With this off, upstream `defaultWorkload` is already `container`, so no
override is needed.
Host prerequisite: because the operator's toolkit component is disabled,
the node's containerd must already have the `nvidia` runtime registered
(`nvidia-ctk runtime configure --runtime=containerd`); `apt install
nvidia-container-toolkit` installs binaries only and does not wire
containerd. With `driver.enabled=false` the operator uses the
pre-installed host driver at its standard location, so a stock apt
install needs no `hostPaths.driverInstallDir` override (Talos lands the
driver under a non-standard prefix — see
`examples/values-native-talos.yaml`).
The bundle wiring (`packages/core/platform/templates/bundles/iaas.yaml`)
already enables `cozystack.gpu-operator` as an optional package — no
bundle changes required. Users opt into this variant by setting
`variant: container` in their `Package` CR.
Verified locally with `helm template` against `values.yaml` +
`values-container.yaml`: ClusterPolicy renders with
`driver.enabled=false`, `toolkit.enabled=false`,
`devicePlugin.enabled=true`, `vfioManager.enabled=false`,
`cdi.enabled=false`, `sandboxWorkloads.enabled=false`. Platform `helm
unittest` and the `gpu-operator` variant bats suite are green. Not yet
smoke-tested on a real Ubuntu GPU node — the config matches the
documented apt-host wiring (legacy `nvidia-container-runtime` hook, CDI
off), but an end-to-end CUDA-pod run on hardware is still owed.
Closes#2764
### Release note
```release-note
feat(gpu-operator): add `container` variant for hosts with a pre-installed NVIDIA driver and nvidia-container-toolkit, exposing containerized GPU workloads via the standard NVIDIA device plugin without VFIO passthrough or vGPU.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a "container" GPU operator variant to support containerized GPU
workloads on hosts with the NVIDIA driver and nvidia-container-toolkit.
* **Documentation**
* Updated variants documentation and OS support table to include the
container variant and per-host requirements (including Ubuntu and Talos
notes).
* Clarified Talos native pod example wording.
* **Tests**
* Added regression tests to verify container-variant configuration and
component enablement/disablement.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
The vm-disk **source image** dropdown in the console lists
`PersistentVolumeClaims` in `cozy-public` (the `vm-default-images-*`
golden images) to offer them when creating a VM disk. That list is
issued by the browser as the **tenant identity** (e.g.
`system:serviceaccount:tenant-root:tenant-root`), but the
`cozy:tenant:dashboard` Role in `cozy-public` only granted read on Flux
`helmrepositories`/`helmcharts`. The list returned `403 Forbidden` and
the dropdown stayed empty even when images had been enabled (e.g. via
`bundles.enabledPackages`).
This is identity-dependent, which is why it can look intermittent: a
cluster-admin login bypasses tenant RBAC and sees the images, while any
real tenant — including `tenant-root-super-admin` — hits the 403.
This PR adds `get`/`list`/`watch` on `persistentvolumeclaims` to the
Role. `cozy-public` is a shared public namespace, so read on its PVCs is
safe to expose to every tenant dashboard subject (the Role is already
bound to the tenant groups and the tenant ServiceAccount). A
helm-unittest regression pins the grant.
Verified with `kubectl auth can-i list pvc -n cozy-public`:
- `--as=system:serviceaccount:tenant-root:tenant-root` → no (before this
PR)
- `--as-group=tenant-root-super-admin` → no (before this PR)
- cluster-admin → yes (why it "worked" for admins)
No console code changed — this is an RBAC-only fix.
### Release note
```release-note
fix(dashboard): allow tenant dashboards to list VM images in cozy-public, fixing the empty source-image dropdown when creating a vm-disk
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Dashboard access permissions have been expanded to include read-level
access to persistent volume claims, enabling users to manage and select
images through the dashboard more effectively.
* **Tests**
* Added test suite to verify dashboard role permissions are correctly
configured and functional.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What
Adds a cross-tenant **Tenant Overview** dashboard to the root/infra
Grafana.
## Why
Platform admins currently have no single view of which tenants consume
what. This shows resource utilization, app counts, and health per tenant
at a glance.
## Panels
- **Fleet summary** — tenants, apps, running pods, CPU/memory/storage
totals
- **Tenant leaderboard** — per-tenant table: CPU & memory (usage vs
requests), storage, pods, PVCs, app count, not-ready apps
- **Top consumers** — Top-N bar gauges by CPU, memory, storage, app
count
- **Trends** — per-tenant CPU & memory usage over time
- **Health** — pod restart rate and not-ready HelmReleases per tenant
## Details
- Registered in `dashboards-infra.list` only, so it deploys to the
root/`cozy-monitoring` Grafana and never exposes cross-tenant data on
per-tenant instances (same pattern as `gpu/gpu-tenants`).
- Apps are counted from `gotk_resource_info` HelmReleases, excluding the
paired `*-system`/`cozy-*` charts.
- Uses only standard cadvisor / kube-state-metrics / Flux metrics.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## New Features
* Added a Tenant Overview dashboard for comprehensive tenant
infrastructure monitoring, including:
* Fleet summary with tenant counts, app deployments, and aggregate
resource usage
* Tenant performance leaderboard with CPU, memory, and storage metrics
* Top consumers visualization by resource type and app count
* Historical usage trends and pod health metrics
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
`publishing.proxyProtocol: true` is documented to enable PROXY-protocol
on the host ingress-nginx, but the three nginx config keys
(`use-proxy-protocol`, `real-ip-header`, `enable-real-ip`) never reached
the controller. They were written into the `ingress-system` component
values of the `cozystack.ingress-application` Package, but that
component has no `install:` block, so the package controller never
builds a HelmRelease for it and the values are dropped. External traffic
behind a PROXY-protocol L4 load balancer kept breaking.
This routes the switch through `_cluster.proxy-protocol` in the
`cozystack-values` Secret — the same path `expose-external-ips` already
travels to reach the host ingress release — and the `extra/ingress`
chart applies it to the external-facing ingress (the namespace named by
`expose-ingress`). The dead component-values injection is removed.
The `extra/ingress` chart also gains a first-class `proxyProtocol`
value, so PROXY-protocol can be enabled per Ingress application (a
per-app boolean knob like `cloudflareProxy`).
Two `fail` guards keep the new knobs safe: the per-app `proxyProtocol`
value is rejected on the external-facing (host) ingress — that path must
go through `publishing.proxyProtocol`, which also installs ouroboros
(hairpin handling) and arms the disable-hazard guard — and
`cloudflareProxy` + `proxyProtocol` together are refused, since
Cloudflare's HTTP proxy does not speak PROXY-protocol to origins and its
`real_ip_header` server-snippet would override `real-ip-header:
proxy_protocol`.
`use-forwarded-headers` / `compute-full-forwarded-for` are deliberately
not set: without a paired `proxy-real-ip-cidr` they would let clients
spoof `X-Forwarded-For`, and the PROXY-protocol frame is already the
source of truth for the client address.
helm-unittest coverage pins runtime delivery on the host path, the
per-app path, and that non-host tenants stay untouched; the previous
test that only checked template emission into the (non-consumed) Package
component is replaced.
Affected versions: v1.4.0–v1.4.2 and current main (introduced in #2561).
**Upgrade note (action required).** Because the flag was a silent no-op
before this fix, a v1.4.x cluster that set `publishing.proxyProtocol:
true` aspirationally — without its L4 load balancer actually injecting
PROXY frames — kept working by accident. After this fix the flag really
enables `use-proxy-protocol`, which makes ingress-nginx require a PROXY
frame on every connection; if the LB does not send one, all external
ingress breaks. Before upgrading, confirm the upstream LB injects
PROXY-protocol v1 frames, or unset `publishing.proxyProtocol`.
Closes#2796
### Release note
```release-note
fix(ingress): publishing.proxyProtocol now enables PROXY-protocol on the host ingress-nginx (the config was silently dropped before reaching the controller); the Ingress application also gains a proxyProtocol value to enable PROXY-protocol per-app. ACTION REQUIRED on upgrade: the flag was previously a no-op, so a cluster that set it without the upstream LB injecting PROXY-protocol v1 frames will lose all external ingress once it takes effect — verify the LB, or unset publishing.proxyProtocol before upgrading.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Tenant ingress support for PROXY-protocol via a new proxyProtocol
option (default: false).
* Platform-level publishing.proxyProtocol now controls external/host
ingress behavior.
* **Bug Fixes / Validation**
* Rejects enabling per-app proxyProtocol on external-facing host
ingresses.
* Rejects using proxyProtocol together with cloudflareProxy to prevent
conflicting real‑IP handling.
* **Documentation**
* Updated docs describing proxyProtocol usage, defaults, and safety
requirements.
* **Tests**
* Added tests covering proxyProtocol wiring, config emission, and
failure cases.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Drops the 3× retry loop on `Run E2E tests` and `Install Cozystack into
sandbox`. `Prepare environment` keeps its 3× retry — that step is pure
infrastructure (Talos image download, sandbox VM boot, network) where
transient runner hiccups warrant a retry.
On failure, the test step now captures `kubectl get hr -A -o wide` and
`kubectl get events -A` under a collapsible group so triage starts with
the actual broken-state snapshot.
> [!NOTE]
> An earlier revision of this PR also doubled every bats timeout. That
commit was dropped in a rebase and is intentionally **not restored**:
the timeout class that actually matters (per-app HR-Ready waits) has
since been standardized at 5m on `main` (7b9f286fa), making a blanket 2×
redundant.
**Fixes gateway.bats teardown leakage.** The nested-tenant tests deleted
tenants fire-and-forget, parent and child back-to-back. The leftover
uninstalls (each blocked on a cleanup Job, parents wedged on
still-terminating child namespaces) plus one mid-install child HR
occupied exactly 5 workers on the `--concurrent=5` tenants
helm-controller shard, starving whichever app test ran next — observed
as the harbor HR sitting unreconciled for its whole 5m HR-Ready budget
in [run
27020081550](https://github.com/cozystack/cozystack/actions/runs/27020081550),
surfaced by this PR's own retry removal + diagnostics dump. Teardown now
deletes child→parent with hard `wait hr --for=delete` between, so a
wedged tenant uninstall fails gateway.bats itself, not an innocent
neighbor.
## Why
Audit of 30 successful PR runs found that across 5 sampled failure
attempts, **25/25 retries** on `Run E2E tests` failed — the retry loop
never recovered a flake, only stretched deterministic failures and
tripled diagnostic wall-time. Same data shape on `Install Cozystack`.
Beyond wasted CI time, the retry was hiding ~10 deterministic bugs (Helm
namespace-ownership conflict, seaweedfs HR timeout, harbor BucketInfo
wiring, vminstance disk race, etc.). Each failure looked like a "flake"
because the retry sometimes coincided with whatever transient state had
cleared — the retry never fixed the bug, just delayed surfacing.
## Dependencies
The deterministic bugs the retry was masking are now fixed on `main`:
- ✅ **#2508** — installer namespace bootstrap (Helm namespace-ownership
conflict on cold install) — merged
- ✅ **#2509** — operator HelmRelease config knobs (`seaweedfs-system`
2-min wait race within Flux's 5-min reconcile windows) — merged
- ✅ **#2528** — harbor bucket-secret + BucketInfo gating (harbor
`ValuesError` on first install) — merged
- ✅ **#2529** — objectstorage-controller BucketAccess conflict retry —
merged
Companion PRs in the #2619 split (independent of this PR,
ordering-wise):
- **#2602** — Flux v2.8.0 + chart fixes
- **#2601** — seaweedfs-system split
This PR does NOT depend on #2602/#2601 — it now touches only the
workflow file and gateway.bats teardown, both on top of fresh `main`.
Surfaced from #2500.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* CI prepare-environment step now reports plain attempt counts with
clear success/failure messages.
* Install and per-app test steps no longer retry; each runs once and
fails immediately on error. Failed apps log diagnostics and job proceeds
to remaining apps while overall job fails.
* **Tests**
* End-to-end tests and install/prepare flows use longer, more tolerant
timeouts and added existence polling to reduce flakiness and improve
diagnostics.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2558?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Pin the native s3.trafficDistribution behaviour that replaced the dropped
s3-traffic-distribution.patch: PreferClose is kept on Kubernetes 1.33,
auto-converts to PreferSameZone on 1.35, and the field is omitted below the
1.31 gate. Adds the helm-unittest harness (test target + suite) the package
previously lacked.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Re-vendor the SeaweedFS Helm chart from 4.05 to 4.31. Improvements that land
with the binary: the PostgreSQL filer store moves to pgx v5 with PgBouncer
support and an ON CONFLICT upsert default, HTTPS certificates hot-reload without
a pod restart on cert-manager rotation, and more S3/volume Prometheus metrics
are exported. The S3 service now exposes a metrics port by default.
Adaptations:
- Drop s3-traffic-distribution.patch: native s3.trafficDistribution support
landed upstream in chart 4.12. The native field is gated to Kubernetes >=1.31
and auto-converts to PreferSameZone on >=1.35, which the hardcoded patch did
not do. Set s3.trafficDistribution: PreferClose in values instead.
- Migrate global.* to global.seaweedfs.* in values: chart 4.18 namespaced the
app-specific globals to avoid polluting the shared global namespace when the
chart is used as a subchart. Covers enableSecurity, serviceAccountName,
extraEnvironmentVars and monitoring.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The host PROXY-protocol switch wrote the nginx config keys into the
ingress-system component values of the cozystack.ingress-application
Package. That component has no install block, so the package controller
never built a HelmRelease for it and the values were dropped — the host
ingress-nginx never received use-proxy-protocol and external traffic
behind a PROXY-protocol L4 load balancer kept breaking.
Route the switch through _cluster.proxy-protocol in the cozystack-values
Secret instead, the same path expose-external-ips travels to reach the
host ingress release, and drop the dead component-values injection.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Expose a proxyProtocol value on the Ingress application that turns on
ingress-nginx use-proxy-protocol with real-ip-header=proxy_protocol and
enable-real-ip. The host control plane also enables it through the
_cluster.proxy-protocol switch, applied only to the external-facing
ingress (the namespace named by expose-ingress).
use-forwarded-headers / compute-full-forwarded-for are deliberately not
set: without a paired proxy-real-ip-cidr they let clients spoof
X-Forwarded-For, and the PROXY-protocol frame is already the source of
truth for the client address.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
PR #2711 switched PR builds to push only :pr-<N>-<sha> tags (PUBLISH_FLOATING=0) to stop concurrent-push 409 conflicts, but left --cache-from pointing at $(REGISTRY)/<img>:latest. PR builds set REGISTRY to the per-CI registry (iad.ocir.io) and never publish :latest there, so every PR build's registry cache lookup 404s and rebuilds cold.
Add a CACHE_REGISTRY knob (default ghcr.io/cozystack/cozystack, where release builds do publish :latest) and point every --cache-from at it. Push targets ($(REGISTRY)) and the unique per-PR tags are unchanged, so the #2711 anti-conflict behaviour is preserved while PR builds regain a warm cache from the last release.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The vgpu defaults shipped a mediatedDevicesConfiguration block whose
mediatedDeviceTypes were hardcoded nvidia-NNN ids. Those ids are not
stable identifiers — they are the per-(SKU, driver) indices under which a
host enumerates mdev types in /sys/.../mdev_supported_types/, so the same
profile gets a different number on a different card or driver branch.
Auto-injecting them made KubeVirt try to create mdev types that usually
do not exist on the operator's nodes, and duplicated work the GPU
Operator's vGPU Device Manager already does.
Delegate mdev creation to the vGPU Device Manager, which is name-based and
stable, and keep only the name-based permittedHostDevices.mediatedDevices
exposure list (mdevNameSelector) in the defaults. KubeVirt-driven creation
stays available through the .gpu.mediatedDevicesConfiguration override for
operators who supply host-verified type ids; document the lookup recipe.
Update the bundle wiring, the gpu/vgpu values comments, the vGPU guide and
the tests accordingly (the default vgpu path now emits no
mediatedDevicesConfiguration; the override path is covered explicitly).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The auto-injected permittedHostDevices only works if each resourceName
equals what nvidia-sandbox-device-plugin advertises on the node; otherwise
the admission webhook rejects the VM — the very manual step this bundle
removes. Several defaults did not match the slug the plugin
(nvcr.io/nvidia/kubevirt-gpu-device-plugin) derives from the PCI-IDs
database name, which uppercases the name and keeps the die suffix, the
Tesla brand and the form-factor/memory tokens:
10DE:1EB8 TU104GL_T4 -> TU104GL_TESLA_T4
10DE:20B7 GA100_A30 -> GA100GL_A30_PCIE
10DE:2331 GH100_H100_PCIE_80GB-> GH100_H100_PCIE
10DE:2321 GH100_H100_NVL_94GB -> GH100_H100L_94GB
10DE:1DB1/1DB4/1DF6 + new 1DB5 -> GV100GL_TESLA_V100_* family
Device 10DE:1DB1 was also mislabelled: it is the SXM2 16GB part, not the
32GB one — the 32GB SXM2 V100 is a distinct device (10DE:1DB5) that was
missing entirely. Relabel 1DB1 as 16GB and add 1DB5.
Restate the header comment as the actual generation rule and pin the
corrected slugs (T4, A30, both V100 SXM2 parts) with a regression test
that also asserts the tidy/short forms never reappear.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The update target gated the insertion of both the permittedHostDevices
and mediatedDevicesConfiguration directives behind a single
`grep permittedHostDevices` guard. A hand-merge from a newer upstream
KubeVirt CR that dropped only mediatedDevicesConfiguration left
permittedHostDevices in place, so the guard skipped the combined sed and
mediatedDevicesConfiguration was never reinserted — then the sanity-check
tail aborted the target with a non-zero exit instead of self-healing.
Split the two directives into independent guarded inserts so each
self-heals on its own. Reword the sanity-check failures to name the
directive that failed to land and the anchor its insert keys off, instead
of implying the anchor itself is missing.
Add a shell regression test that strips one directive at a time and
asserts `make update` restores it without duplicating the other; the
update target gains a CR_FILE override so the test can drive it against a
throwaway copy. Wired into the package test target and guarded to skip on
BSD sed.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The .Values.gpu accesses in the iaas bundle are fully parenthesized
((( .Values.gpu ).x).y) so the field walk stays nil-safe when an operator
nulls the whole gpu block or leaves a sub-key blank (an empty YAML key
parses to null, which Helm then drops from .Values). A bare dotted walk
would panic with "nil pointer evaluating interface {}.<field>" on the
first nil hop.
Add regression cases covering gpu: null in both passthrough and vgpu
variants plus blank sub-keys, and a comment warning against flattening
the parenthesized form. A mutation check confirms the cases fail with the
nil-pointer panic the moment the parentheses are removed.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
When cozystack.gpu-operator is in bundles.enabledPackages, the platform
now mirrors the chosen GPU variant into the KubeVirt CR — the HostDevices
feature gate is appended to developerConfiguration.featureGates, and a
starter permittedHostDevices block is filled from a per-variant defaults
table. Without this, a passthrough-variant GPU shows up under
.status.allocatable on the node but the KubeVirt admission webhook
rejects every VM that references it as a hostDevices.deviceName.
Two defaults tables ship under packages/core/platform/files/:
- gpu-passthrough-defaults.yaml covers Hopper (H100/H200), Ada
Lovelace (L4/L40/L40S), Ampere (A100 PCIe/SXM, A40, A30, A10),
Turing (T4), Volta (V100/V100S). PCI vendor:device pairs are
stable; resourceName slugs mirror nvidia-sandbox-device-plugin's
<arch>_<model>_<form>_<mem> convention. externalResourceProvider
is set on every entry because the resources are advertised by
the sandbox-device-plugin DaemonSet, not by KubeVirt itself.
- gpu-vgpu-defaults.yaml covers the Pascal-Ampere mdev profile
range the upstream vgpu-device-manager walks /sys/class/mdev_bus/
for. Ada Lovelace and Blackwell SR-IOV vGPU is out of scope for
the chart's default list (NVIDIA driver path differs) and is
expected to be advertised via the user-override hook below.
bundles.iaas.gpuOperatorVariant picks which table applies (default =
passthrough, vgpu = mediated). The same variant flows through to the
gpu-operator Package CR, replacing the previous always-default call.
Operators extend or replace the defaults via .gpu in platform values:
gpu:
replaceDefaults: false
permittedHostDevices:
pciHostDevices: [...]
mediatedDevices: [...]
mediatedDevicesConfiguration: {...}
replaceDefaults: false (the default) appends the user list to the
NVIDIA table; true drops the table entirely for clusters with only
non-NVIDIA accelerators or a strict allowlist requirement.
Companion helm-unittest suite covers the eight branches: no gpu
package emits no extras; gpu-operator at default variant emits the
HostDevices gate plus pciHostDevices; vgpu variant emits the
HostDevices gate plus mediatedDevices and mediatedDevicesConfiguration;
user overrides append by default; replaceDefaults wipes; unknown
variant fails fast.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The KubeVirt CR's developerConfiguration.featureGates list, the
permittedHostDevices block, and mediatedDevicesConfiguration are now
driven by chart values:
- extraFeatureGates is appended after the static default set so
HostDevices (or any other future gate) can be enabled without
forking the template.
- permittedHostDevices is rendered verbatim when set, allowing the
pciHostDevices (passthrough) and mediatedDevices (vGPU) shapes the
KubeVirt admission webhook requires for VMs that reference a GPU
by resourceName.
- mediatedDevicesConfiguration is rendered verbatim when set,
carrying mediatedDeviceTypes / nodeMediatedDeviceTypes for the
vGPU profile auto-creation loop.
Defaults stay empty so the rendered CR is byte-identical to the
previous template when no overrides are supplied.
The Makefile's update target re-applies the four sed transforms
in-place on templates/kubevirt-cr.yaml — it does not download anything
from upstream KubeVirt because the published CR is a near-empty stub
and would clobber the hand-curated cozystack defaults. To refresh from
a newer upstream release the operator hand-merges the new fields into
the in-tree template, then runs `make update` to re-assert the four
cozystack parameterization patches; grep sanity-checks at the end of
the target fail loudly if any anchor (developerConfiguration,
- VMExport, evictionStrategy) gets dropped during the hand-merge.
Companion helm-unittest coverage pins the default render and the
three populated shapes.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The two existing variants assume the host has neither the NVIDIA driver
nor the container toolkit: default unbinds the host driver to bind
vfio-pci for KubeVirt VMs, vgpu requires the proprietary vGPU host
driver and a license server. Neither path produces a working setup on a
node that already ships the driver and nvidia-container-toolkit via the
distro package manager — the most common shape of a Linux GPU node.
The new container variant surfaces the third architectural mode that
upstream NVIDIA gpu-operator already supports: sandboxWorkloads
disabled, devicePlugin enabled, with every component that would
conflict with the host install switched off.
- driver.enabled=false use the host driver
- toolkit.enabled=false host already has nvidia-container-toolkit;
enabling overwrites the containerd
config.toml the host configured via
nvidia-ctk runtime configure
- devicePlugin.enabled=true publish nvidia.com/gpu to the kubelet
- vfioManager.enabled=false do not unbind the host driver
- cdi.enabled=false CDI spec generation is serviced by the
disabled toolkit; an apt host uses the
legacy nvidia-container-runtime hook, so
pin off to match the host runtime wiring
- sandboxWorkloads.enabled=false regular pod workloads, not VMs
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
With `publishing.proxyProtocol: true`, the host emits a
`cozystack.ouroboros` Package using the no-override Package macro, so
the proxy inherits the wrapper chart default backend
`ingress-nginx-controller.cozy-ingress-nginx`. That layout describes a
managed Kubernetes tenant cluster, not the host: on the host,
`extra/ingress` deploys ingress-nginx into the root tenant namespace as
`root-ingress-controller` in `tenant-root`. The composed backend FQDN
never resolves, so `ouroboros-proxy` never reaches Ready.
This emits the host ouroboros Package with a `proxy.target` override
derived from `publishing.ingressName` — namespace plus
`<short>-ingress-controller` service name — mirroring the `trimPrefix
"tenant-"` transform `extra/ingress` applies to its release namespace.
The derivation reuses the existing platform invariant that
`publishing.ingressName` is the namespace where the host ingress
controller runs.
Adds bundle helm-unittest coverage for the default ingress name and a
non-default one, and corrects the misleading wrapper `values.yaml`
comment that claimed the host uses `cozy-ingress-nginx`.
Scope: this fixes the ouroboros readiness defect only. The separate
defect where the injected nginx PROXY-protocol config keys are not
consumed by the host ingress is tracked independently and not touched
here.
Closes#2797
### Release note
```release-note
fix(networking): point the host ouroboros PROXY-protocol proxy at the root-tenant ingress so it becomes Ready when publishing.proxyProtocol is enabled
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Enhanced proxy protocol configuration to automatically route through
the appropriate ingress-nginx service based on your ingress name
settings.
* **Tests**
* Added test coverage for proxy protocol bundle wiring and ingress
service routing.
* **Documentation**
* Clarified proxy target configuration behavior and fallback mechanisms.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The vm-disk source image dropdown in the console lists PersistentVolumeClaims in cozy-public (the vm-default-images-* golden images). The browser issues that list as the tenant identity (e.g. system:serviceaccount:tenant-root:tenant-root), but the cozy:tenant:dashboard Role only granted read on Flux helmrepositories and helmcharts. The list returned 403 and the dropdown stayed empty even when the images existed.
Add get/list/watch on persistentvolumeclaims to the Role. cozy-public is a shared public namespace, so read on its PVCs is safe to expose to every tenant dashboard subject. A helm-unittest regression pins the grant.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
cozystack-basics ships cluster-wide ValidatingAdmissionPolicies matching
tenants.apps.cozystack.io and the Gateway API types. KCM's VAP status
type-checker panics on a nil schema when it type-checks a policy against a
matched type that isn't resolvable yet; during a cold install this crashes
KCM, which stalls cert/token issuance, so cozystack-api never starts and the
tenants schema never resolves — a bootstrap deadlock.
Add dependsOn edges so the package installs only after cozystack-engine
(cozystack-api) and gateway-api-crds are Ready. Drift detection is off on
operator-generated HelmReleases, so a template capability gate would drop the
policy at first install and never re-add it; ordering is the reliable fix.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 03f1e7a55a)
## What this PR does
`backupstrategy-controller` has been installed by default since
494144fb9, and since f17ead3d2 its PackageSource declares a hard
`dependsOn: cozystack.velero`. But `cozystack.velero` remained an
*optional* package — so on any bundle that doesn't explicitly enable it
(including the e2e sandbox, which enables only
`cozystack.external-dns-application`), the `backupstrategy-controller`
Package stays `DependenciesNotReady` forever and the
`cozystack-platform` HelmRelease never becomes Ready:
```
Helm upgrade failed for release cozy-system/cozystack-platform: timeout waiting for:
[Package/cozystack.backupstrategy-controller status: 'InProgress']
```
Since 67b1a960d made the e2e install test gate on HelmRelease readiness,
this deterministically fails CI on every PR (e.g. runs
[27005526972](https://github.com/cozystack/cozystack/actions/runs/27005526972),
[26974531181](https://github.com/cozystack/cozystack/actions/runs/26974531181)
— the latter failed all 3 retry attempts with the same signature).
This PR promotes `cozystack.velero` from optional to default in the
system bundle, so the dependency is always satisfiable. Velero deploys
standalone (`backupStorageLocation: null`); the cozy-default BSL still
materialises only after the platform Bucket reconciles. The
`bundles.disabledPackages` opt-out keeps working (covered by a new unit
test), and the now-redundant velero entry is dropped from the
isp-full-generic `enabledPackages`.
### Release note
```release-note
fix(platform): Velero is now installed by default as a hard dependency of the backup subsystem. Existing clusters will get Velero deployed in the cozy-velero namespace on upgrade; opt out with bundles.disabledPackages if VM backups are not used.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Velero backup and restore functionality is now installed by default
with every system deployment, providing automatic backup management
capabilities out of the box.
* **Tests**
* Added test coverage for default Velero installation behavior and
package disable scenarios.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
backupstrategy-controller is installed by default since 494144fb9 and
its PackageSource declares a hard dependsOn on cozystack.velero
(f17ead3d2). With velero still optional, any bundle that does not
explicitly enable it leaves the backupstrategy-controller Package
permanently DependenciesNotReady, so the cozystack-platform HelmRelease
never becomes Ready. Since 67b1a960d made the e2e install test gate on
HelmRelease readiness, this deterministically fails CI on every PR.
Promote cozystack.velero from optional to default in the system bundle
so the dependency is always satisfiable. The bundles.disabledPackages
opt-out keeps working; drop the now-redundant velero entry from the
isp-full-generic enabledPackages list.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
The vendored COSI `objectstorage-controller` (built from upstream
`v0.2.2`) leaves dynamically-provisioned buckets without credentials.
In the dynamic-provisioning path, `provisionBucketClaimOperation` sets
`BucketClaim.status.bucketName` but hardcodes `bucketReady=false` and
never re-reads the `Bucket`. The controller does not watch `Bucket`
objects, so the readiness transition the sidecar performs after
`DriverCreateBucket` (`Bucket.status.bucketReady` `false`→`true`) is
never propagated back to the `BucketClaim`. Each periodic resync
re-enters the same branch (`Buckets().Create` returns `AlreadyExists`,
which is ignored) and re-sets `bucketReady=false`. As a result
`BucketClaim.status.bucketReady` stays `false` forever, the sidecar
refuses to grant the `BucketAccess` ("BucketAccess can't be granted to
Bucket not in Ready state"), and the credentials secret is never created
— even though the bucket exists in the backend and the `Bucket` CR
already reports `bucketReady: true`.
This adds a vendored patch that re-reads the live `Bucket` after create
and propagates its readiness onto the `BucketClaim`, so the existing
periodic resync converges the claim to ready and access can be granted.
The fix mirrors what the rewritten upstream controller already does on
`main` (`claim.Status.ReadyToUse = bucket.Status.ReadyToUse`). Upstream
`main` is a `v1alpha2` rewrite and the maintained `release-0.2` branch
still carries the bug, so this stays a vendored patch against `v0.2.2`
until the controller is bumped to a release that includes the fix.
Verified: reproduced the stuck-`false` claim with a unit test against
the upstream `v0.2.2` source and confirmed the patch resolves it,
existing controller tests pass, the patch applies cleanly to `v0.2.2`,
and the controller image builds.
### Release note
```release-note
fix(objectstorage-controller): propagate Bucket readiness to the BucketClaim so dynamically-provisioned buckets receive credentials instead of getting stuck without a granted BucketAccess
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Fixed an issue where bucket readiness status was incorrectly reported
when a bucket already existed. The status now accurately reflects the
actual state of the existing bucket instead of being unconditionally set
to not ready.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Bumps the `seaweedfs-cosi-driver` image from `v0.3.0` to `v0.3.1`.
`v0.3.1` ships the stale-socket self-heal fix. After a non-graceful exit
(SIGKILL / OOM / panic) Go does not unlink the COSI unix socket, and
because it lives on an `emptyDir` that survives container restarts, the
next start failed with `bind: address already in use` and the
`objectstorage-provisioner` wedged in `CrashLoopBackOff`. While it was
down, `BucketClaim` reconciliation stopped and buckets were never
provisioned — the COSI `Bucket` could even report `bucketReady: true`
while no bucket existed. This is the CrashLoopBackOff reported in #2429.
The driver now removes a leftover socket before binding (only when it
exists, is a socket, and has no live listener), so the provisioner
self-heals on restart.
Upstream: seaweedfs/seaweedfs-cosi-driver#8 (merged), released in
v0.3.1. Points at the upstream published multi-arch image (amd64 +
arm64) — no vendoring needed.
### Release note
```release-note
fix(seaweedfs): bump seaweedfs-cosi-driver to v0.3.1, which self-heals the object storage COSI provisioner from a CrashLoopBackOff caused by a stale unix socket
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated SeaweedFS cosi driver container image to a newer version.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The four backups.cozystack.io CRDs injected the x-cozystack-options vendor
extension into their OpenAPI validation schema. apiextensions JSONSchemaProps
is a closed struct that only preserves x-kubernetes-* vendor extensions, so
helm-controller's server-side apply rejected the unknown key ("field not
declared in schema") and the CRDs never applied — the controller crash-looped
and backups were non-functional.
Move the field-to-source mapping the dashboard needs into CRD
metadata.annotations (preserved by the apiserver), emitted via
+kubebuilder:metadata:annotations markers, and drop the post-processing awk
injector. The dashboard reattaches the extension client-side.
A helm-unittest suite pins the contract: each dropdown field carries its source
in metadata.annotations and must not carry x-cozystack-options in the schema,
guarding against re-introducing the injector.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Harbor stores registry data only in S3-compatible object storage and
has no local-filesystem backend, but the app docs never stated this.
Without SeaweedFS available in the tenant chain the registry
BucketClaim never yields its credentials secret and the HelmRelease
silently stays unreconciled — a common and confusing first-deploy
failure.
Add a Prerequisites section to the chart README (propagated to the
published app docs) explaining the requirement, how the bucket is
provisioned via COSI from SeaweedFS, how the class is inherited down
the tenant tree, and how to enable it.
Co-authored-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add a cross-tenant overview dashboard for the root/infra Grafana that
surfaces which tenants consume what resources, how many apps (Flux
HelmReleases) each runs, and per-tenant health.
Panels:
- Fleet summary stats (tenants, apps, pods, CPU/mem/storage)
- Tenant leaderboard table (CPU/mem usage vs requests, storage, pods,
PVCs, app count, not-ready apps)
- Top-N bar gauges by CPU, memory, storage and app count
- CPU/memory usage trends per tenant
- Pod restart rate and not-ready HelmReleases per tenant
Registered in dashboards-infra.list so it deploys only to the
root/cozy-monitoring Grafana, keeping cross-tenant data off per-tenant
instances. Uses standard cadvisor, kube-state-metrics and
gotk_resource_info metrics.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
v0.8.0 makes the TCP proxy log an explicit reason when its backend
readiness check fails, instead of staying silently NotReady. That
silent failure mode is exactly what makes a mispointed proxy backend
hard to diagnose, so the upgrade pays off the moment the target is
wrong. The bump also carries the v0.7.3 migration of the kubectl
sidecar image off Docker Hub to mirror.gcr.io, sidestepping anonymous
pull rate-limits on shared CI/runner IPs.
Chart manifest and controller image are re-pinned by digest in lockstep
with the version, and the image-pin test fixture is updated to match.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
With publishing.proxyProtocol enabled the host emits a cozystack.ouroboros
Package via the no-override Package macro, so the proxy inherits the wrapper
chart default backend ingress-nginx-controller.cozy-ingress-nginx. That
layout describes a managed Kubernetes tenant cluster; on the host,
extra/ingress deploys ingress-nginx into the root tenant namespace as
root-ingress-controller in tenant-root. The composed backend FQDN never
resolves, so ouroboros-proxy never reaches Ready.
Emit the host ouroboros Package with a proxy.target override derived from
publishing.ingressName (namespace plus <short>-ingress-controller service
name), mirroring the trimPrefix transform extra/ingress applies. Add bundle
helm-unittest coverage for the default and a non-default ingress name, and
correct the misleading wrapper values.yaml comment.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Adds a generic mechanism for **runtime-populated dropdowns** in
dashboard create/edit forms. Many fields are references to live cluster
resources (GPU device names, KubeVirt instancetypes/preferences, Multus
networks, VM images, bucket storage pools, storage classes, backup
classes/plans) but were free-text or static enums that drift from
reality.
- **`Option` resource** — `core.cozystack.io/v1alpha1`, namespaced,
read-only (list/get/watch). Items are computed on read by an in-process
provider registry using a privileged client, so tenants get curated
option lists without broad cluster reads. Providers: `gpu`,
`instancetype`, `instanceprofile`, `network`, `image`, `storagepool`,
`storageclass`, and backup-related `backupclass`, `plan`, `backup`,
`appkind`.
- **`x-cozystack-options` schema keyword** — app `values.yaml` declare
it via the new cozyvalues-gen `@x-<keyword>` directive (e.g. `##
@x-cozystack-options {source: instancetype}`). Schemas, Go types, and
the `-rd` ApplicationDefinitions are regenerated from source so the
extension is carried into the served OpenAPI. The dashboard renders
annotated fields as dropdowns sourced from the live cluster (UI widget
ships in a separate cozystack-ui PR).
- **RBAC** — the apiserver ServiceAccount gets the privileged reads;
tenants get read-only access to `options` in their namespace via the
aggregated tenant ClusterRoles (granted at `cozy:tenant:view:base`, so
every access level from view upward can populate dropdowns).
- Drops the redundant manual `instanceProfile` enum injection from
vm-instance's Makefile (now driven dynamically).
## Dependency on cozyvalues-gen
The `x-cozystack-options` annotations require the new vendor-extension
directive from cozystack/cozyvalues-gen#27. This PR bumps the pinned
cozyvalues-gen to **v1.6.0**. **A cozyvalues-gen v1.6.0 release must be
cut before CI can pass** — until then the pre-commit schema-generation
job cannot download the binary and will fail.
### Screenshots
N/A — no UI changes in this repository. The consuming dashboard widget
ships in a separate cozystack-ui PR.
### Release note
```release-note
feat(api): add a namespaced read-only Option resource (core.cozystack.io) and the x-cozystack-options schema keyword, enabling dashboard form fields (GPU, instancetype/profile, network, image, storage pool/class, backup class/plan) to render as dropdowns populated from live cluster resources
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Expanded dynamic dropdowns across many apps (storageClass,
storagePool, instanceType/instanceProfile, GPU, image, vmdisk, network,
backup/plan/backupclass, appkind) so chart fields can be populated from
live cluster resources.
* **Chores**
* Updated API/RBAC manifests to grant read access needed for option
providers.
* **Tests**
* Added provider and REST tests to validate option sources, scoping,
ordering, and failure handling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The vendored COSI controller (v0.2.2) sets only BucketClaim.status.bucketName
in the dynamic-provisioning path and hardcodes bucketReady=false, never
re-reading the Bucket afterwards. The controller does not watch Buckets, so the
readiness transition the sidecar performs (false->true) is never reflected on
the BucketClaim: bucketReady stays false forever and BucketAccess is never
granted, leaving provisioned buckets without credentials.
Re-read the live Bucket after create and propagate its readiness, so the
periodic resync converges the BucketClaim to ready. This matches the behavior
the rewritten upstream controller already has on its main branch.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
<img width="2770" height="1970" alt="image"
src="https://github.com/user-attachments/assets/774c7c71-0e6a-496a-9898-8ad1ed848b0a"
/>
## Summary
Grants tenant users permission to call the KubeVirt VM power-control
subresources (`subresources.kubevirt.io` →
`virtualmachines/{name}/start|stop|restart`, verb `update`) by adding
them to the `cozy:tenant:use:base` ClusterRole, which aggregates into
`tenant-use`, `tenant-admin`, and `tenant-super-admin`.
Without this, the dashboard VM power buttons return 403 even for tenant
admins: `kubectl auth can-i update virtualmachines --subresource=start`
is `no` for every tenant role today, while
`virtualmachineinstances/console|vnc|portforward` are already granted at
the same `use` level — so power control belongs there too.
## Companion PR
UI that consumes this permission (VM power controls + VNC fixes):
cozystack/cozystack-ui#27
## Test
- `helm template` renders `cozy:tenant:use:base` with the new
`virtualmachines/start|stop|restart` rule.
- After apply, `kubectl auth can-i update virtualmachines
--subresource=start|stop|restart` flips from `no` to `yes` for the
`tenant-*-use/admin/super-admin` groups.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Virtual Machine Power Control: Users with tenant-level access can now
start, stop, and restart virtual machines, providing enhanced management
capabilities for VM operations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
v0.3.1 ships the stale-socket self-heal: the driver removes a leftover
COSI unix socket before binding, so the objectstorage provisioner
recovers from CrashLoopBackOff after a non-graceful exit (SIGKILL / OOM /
panic) instead of wedging on "bind: address already in use".
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Worker-node VMs of a tenant Kubernetes cluster are created by Cluster
API / the KubeVirt provider, so their virt-launcher pods were never
stamped with the `apps.cozystack.io/application.{group,kind,name}`
lineage labels that the chart already sets on its other resources
(`ingress.yaml`, `cloud-config.yaml`). As a result the dashboard could
not attribute those pods to the owning `kubernetes` application instance
— e.g. cluster resource-consumer views showed the raw `virt-launcher-…`
pod name with no link to the cluster.
This adds the application lineage labels to the worker VM template
(`KubevirtMachineTemplate` →
`virtualMachineTemplate.spec.template.metadata.labels`), mirroring the
existing stamping in `ingress.yaml`/`cloud-config.yaml`. KubeVirt
propagates labels from this block to the VMI and the virt-launcher pods
(the same path the existing `cluster.x-k8s.io/deployment-name` label
already takes), so the dashboard now attributes worker VMs to their
Kubernetes application.
### Screenshots
N/A — no UI changes; this only affects generated pod labels.
### Release note
```release-note
fix(kubernetes): stamp tenant Kubernetes worker-node VMs with apps.cozystack.io/application.{group,kind,name} so the dashboard attributes their pods to the owning Kubernetes application
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Added application lineage labels to Kubernetes VM templates so cluster
VMs include application group, kind, and name (release-name prefix
trimmed).
* **Tests**
* Added template tests validating those labels and the name-trimming
behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Pin the apps.cozystack.io/application.{group,kind,name} labels stamped on the
worker VM template, including a case that trims the kubernetes- release prefix.
Also quote application.name so a purely-numeric cluster name renders a YAML
string rather than an integer that a label value would reject.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The tenant view aggregation grants read on the core.cozystack.io/options
virtual resource so dashboard dropdowns can be populated. Add a helm-unittest
on the cozy:tenant:view:base ClusterRole so a silent RBAC regression on the
tenant side is caught, mirroring the apiserver-side rbac test.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Myasnikov Daniil
