The flux-instance template rendered spec.distribution.artifact
unconditionally, unlike the sibling optional fields (artifactPullSecret,
imagePullSecret, variant). The umbrella default is empty to use the
operator's embedded manifests (the air-gapped default from #964), but an
empty value renders as YAML null. In the flux-operator v0.50.0 CRD the
field is optional yet typed 'string' with pattern ^oci://.*$ and is not
nullable, so the rendered null (like an empty string) fails schema
validation and the API rejects the FluxInstance.
Guest Kubernetes clusters with the fluxcd addon enabled could therefore
never install it: the sub-release looped install/uninstall, the parent
kubernetes release stayed InProgress, and the cluster never reached Ready.
Guard the field so an empty value is omitted (the operator falls back to
its embedded manifests, preserving the air-gapped default) while a
non-empty artifact still renders. The template lives in the vendored
subchart, so the change is carried as a patch applied by 'make update'.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
(cherry picked from commit da47b70e2c)
Two related issues caused the dashboard SPA to break with CORS errors
after the kc-access cookie expired:
1. KeycloakClient for dashboard had no webOrigins set, so the operator
never touched the field in Keycloak after initial creation. After
any rootUrl change the existing webOrigins value became stale and
blocked CORS from the SPA. Setting "+" tells Keycloak to derive
allowed origins from redirectUris.
2. oauth2-proxy returned 302 -> Keycloak /auth for unauthenticated
XHR/fetch on /api and /apis. The login page does not emit
Access-Control-Allow-Origin, so the SPA's fetch failed with CORS.
--api-route makes oauth2-proxy answer 401 without redirect, so the
SPA can handle the re-auth itself via top-level navigation.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
(cherry picked from commit be8f6b4739)
Refresh the vendored virt-operator manifest from v1.8.2 to v1.8.4. The
release backports validation of the VMI checksum status fields as a
uint32 range, required on Kubernetes 1.36 where strict CRD numeric
format validation otherwise leaves VMIs stuck in Scheduled. Remaining
changes are bug fixes and CVE remediations; the embedded CRDs and RBAC
are unchanged, so only the operator image tag moves.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 192246e8b6)
Reverts #3006. Dropping persistent EFI/TPM was motivated by a live-migration
concern that rested on an outdated assumption — that an RWO Filesystem
backend-storage PVC pins the VM to its node. KubeVirt has migrated
RWO-Filesystem backend storage since v1.4 (kubevirt/kubevirt#12629): on
migration it creates a fresh target state PVC and copies the small state blob,
so persistent-EFI/TPM VMs on the default replicated storage live-migrate fine
(verified: the VM reports LiveMigratable=True and the copy-on-target PVC is
created on migration). Restore upstream's persistent firmware state for the
windows.* preferences.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
(cherry picked from commit 59f035ff53)
The vendored seaweedfs chart ships connection pool settings only for the
mysql store, which cozystack does not use. The postgres2 store, which is
enabled, has no pool settings, so the Go sql pool keeps zero idle
connections and every filer metadata lookup opens a fresh PostgreSQL
connection (TCP + TLS + SCRAM, ~300ms each). With ~5 path lookups per S3
operation this adds roughly 2 seconds of latency to every S3 request,
regardless of cluster load.
Measured on a production cluster (12 nodes, 2 filers): PUT of a 4KiB
object went from 10s (p50) to 0.28s, HEAD from 2.6s to 0.16s after
setting these values; pg_stat_activity confirmed connections were
previously recycled every ~200ms and are now reused.
max_open is capped at 40 so that two filer replicas stay below the CNPG
default max_connections of 100.
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
(cherry picked from commit 49cbea8c73)
Deleting a Tenant whose etcd DataStore has 0 referencing TenantControlPlanes left the DataStore stuck Terminating (finalizer kamaji.clastix.io/TenantControlPlane), hanging the etcd helm uninstall and wedging the tenant namespace. Backports clastix/kamaji#1122 (fixes clastix/kamaji#1115) as a source patch on the pinned 26.3.6-edge build. Refs #3018.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 47638f8569)
## What this PR does
Backport of #3121 to `release-1.5`.
Makes node group `resources` optional, so a node group sized purely by
`instanceType` (with `resources` omitted) passes `values.schema.json`
validation and installs. When a group sets both `instanceType` and
explicit `resources` (both `cpu` and `memory`), the explicit resources
take precedence: the instancetype is omitted from the rendered VM and
the per-group instancetype lookup is skipped, so KubeVirt does not
reject a VM that references an instancetype while overriding
`domain.cpu`/`memory`. A partial spec (only one of `cpu`/`memory`)
alongside an `instanceType` fails fast at render time.
Cherry-picked from `a349c753`; conflicts resolved against the
`release-1.5` tree (the management-cluster Talos worker bootstrap stack
present on `main` is not on this branch, so the schema/template/test
layout differ). Generated files (`values.schema.json`, `README.md`,
cozyrds, API types) regenerated with `make generate` to match. `helm
unittest` passes (134 tests), including the four new node-group sizing
cases adapted to this branch's document layout.
### Release note
```release-note
fix(kubernetes): make node group resources optional when instanceType is set
```
PR #2279 made `replication.enabled: true` unconditional so the operator would
always create -primary/-secondary services. But the mariadb-operator validating
webhook (vmariadb-v1alpha1.kb.io) rejects replication when spec.replicas == 1,
so single-replica MariaDB could no longer be created — the HelmRelease never
became Ready.
Re-guard replication on replicas>1 and teach the service-name consumers about
the bare <release> service the operator provisions in both topologies:
- mariadb.yaml: gate `replication` on replicas>1; restore single-replica
external access via `service.type: LoadBalancer` and keep `primaryService`
LoadBalancer for replicas>1 only.
- backup-cronjob.yaml: route single-replica dumps to the bare service
(no -secondary exists without replication).
- dashboard-resourcemap.yaml: always grant RBAC on the bare service, plus
-primary/-secondary when replicas>1, matching the ApplicationDefinition.
- mariadb-rd: list the bare service in services.include so single-replica
installs are visible in the dashboard.
Adds a single-replica e2e test that reproduces the webhook rejection and
guards against regression.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
(cherry picked from commit c7e8f4d975)
A node group is sized either by instanceType or by explicit resources, but the
generated values schema required resources on every node group. A node group
with instanceType and no resources — or an empty resources dropped during
typed-CR to HelmRelease translation — failed schema validation with "missing
property 'resources'" even though sizing was fully specified.
Mark resources optional in the node group definition and regenerate the schema,
API types, README, and ApplicationDefinition. The chart template emits
domain.cpu/memory only when resources is present, so a node group sized purely
by instanceType now passes validation and installs.
When a node group sets both instanceType and explicit resources, explicit
resources take precedence: the instancetype is omitted from the rendered VM and
the per-group instancetype lookup is skipped, so KubeVirt accepts the explicit
domain.cpu/memory with no instancetype to conflict with. instanceType stays
required and defaulted (no schema change), so the console's required
instanceType dropdown keeps working — sending the default instanceType plus
explicit resources simply sizes the VM by resources. This replaces an earlier
render-time hard failure on the both-set case, which was unreachable to clear
from the dashboard (the required dropdown cannot be emptied).
Explicit resources are all-or-nothing alongside an instanceType: a partial spec
(only cpu, or only memory) on a group that carries a non-empty instanceType
would emit one domain override next to the instancetype — which KubeVirt rejects
— so it now fails at render time with a clear error. Partial resources on a
group without an instanceType (used to tune kubelet reservations) are unaffected.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit a349c753ad)
The seaweedfs COSI driver shipped at v0.1.2 hardcodes read-write S3
actions (Read, Write, List, Tagging) for every BucketAccess and never
reads the accessPolicy parameter. The chart already defines a -readonly
BucketAccessClass carrying parameters.accessPolicy: readonly, but v0.1.2
ignores it, so a credential meant to be read-only can still write
(upload and delete) objects.
Bump the driver to v0.3.1, which honours accessPolicy: readonly by
issuing Read,List-only credentials (read-write and the empty default
stay Read,Write,List,Tagging). The override lives in the package
values.yaml rather than the vendored chart so it survives make update;
the sister extra/seaweedfs chart already runs v0.3.1.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 0c6426feaa)
The s3-service-name.patch renamed the S3 Service to <name>-s3
(seaweedfs-s3), but several consumers still resolved it through
componentName -> <fullname>-s3. name and fullname only coincide when the
release name equals the chart name; the shipped release is
seaweedfs-system, so they pointed at seaweedfs-system-s3 while the real
service stayed seaweedfs-s3:
- the S3 ingress backend and the iceberg ingress backend, which left
ingress-nginx returning 503 on the external S3 endpoint.
The COSI provisioner in-cluster ENDPOINT was wrong on the same axis and
also on the port. Both the s3-gateway branch and the filer-S3 branch
rendered a portless URL that defaults to 443, while the s3 service only
listens on the S3 port (8333); the filer branch additionally targeted
the filer service, which serves no S3. Both in-cluster branches now
resolve seaweedfs-s3 on the configured S3 port.
The default path (s3.ingress.enabled=true) is repaired by the ingress
fix; the COSI in-cluster paths are latent until an operator disables the
ingress. Resolve every consumer through the same name helper the Service
uses, and pin the contract with unittests rendered under the production
release name so the mismatch cannot recur.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 8a14de9abf)
TEMPORARY revert. Restoring persistent EFI/TPM state (4cc60e3) makes
windows.11/2k22/2k25 VMs get a backend-storage PVC that is ReadWriteOnce on the
default StorageClass, which pins the VM to its node and blocks live-migration and
node drains (with evictionStrategy: LiveMigrate this can stall cluster upgrades).
See #3005.
Strip persistent EFI/TPM from the preferences again, but with `yq del` so an
emptied preferredTPM stays `{}` (the v1.8 CRD rejects null) — the bug the original
sed strip caused. Secure Boot and the vTPM stay present, just non-persistent.
Stopgap until vm-state can request an RWX backend (RWX-Block via the upstream
KubeVirt change, or an RWX-Filesystem vmStateStorageClass).
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
(cherry picked from commit 6ac2601a68)
Resolve go.mod conflict: keep main's go 1.26.4 and golang.org/x/* bumps, and keep gomodules.xyz/jsonpatch/v2 v2.4.0 as a direct dependency (imported by the flux-shard-operator admission webhook) instead of main's indirect entry. go mod tidy leaves go.sum consistent.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The shardCount description and the shard.resources value overflowed their columns, pushing the internal separators out of line. Reflow every column to its widest cell so the raw table renders cleanly.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What
Consolidated landing branch for a set of interdependent platform/CI
stabilization fixes. It began as the source PRs in the table below (in
dependency order) and has since grown with review-driven fixes and a few
production-behavior changes — the full current inventory is under
**Scope** below. Landing them as one unit lets CI run against the real
combined state instead of a fragile bottom-up merge train.
## Source PRs (dependency order)
| # | Commit | Source PR | Addresses |
|---|--------|-----------|-----------|
| 1 | fix(capi): startupProbe on capi-controller-manager | #2946 | capi
crashloop during cert provisioning |
| 2 | test(e2e): bucket.bats port-forward + S3 client reliability |
#2944 | flaky bucket test |
| 3 | fix(seaweedfs): restore -lock BucketClass, s3 svc name, drop
volumeSizeLimitMB | #2943 | missing bucket/access classes |
| 4 | test(e2e): in-cluster Cilium endpoint-leak healer (install + apps)
| #2874 | cilium "IP already in use" leak |
| 5 | refactor(build): mode=max registry cache + main-only warmer |
#2938 | build cache |
| 6 | ci(build): isolate each PR build on its own ephemeral runner VM |
#2939 | shared-buildkit contention |
| 7 | fix(e2e): LINSTOR post-install waits on a single 15m deadline |
#2928 | LINSTOR provisioning timeouts |
| 8 | test(metallb): assert digest-pinned image form, not version
literal | #2873 | brittle metallb assertion |
## Why consolidated
These form a dependency DAG (verified from CI logs): e.g. 2943 needs
2946, 2938 needs 2874, 2928 needs 2938/2939, 2873 needs 2928. Tested
bottom-up, each lower PR runs with none of the fixes above it and can't
go green alone. This branch carries all of them, so CI runs against the
real combined state.
## Verification status
CI is green on the latest head (`5653c30`): full-suite E2E passes
end-to-end and is reproducible (2 of 3 runs on this SHA green). The one
red run was a LINSTOR tie-breaker / DRBD-metadata infra flake on the
sandbox, unrelated to the diff — a different environmental subsystem
failed each run (details in the comments below). The earlier 3-hour
crust-gather snapshot hang is fixed in `5653c30`.
## Scope beyond the original 8
The branch has grown past the 8 source PRs above with review-driven
fixes and a few production-behavior changes. For reviewer transparency,
the full set:
**Review fixes (@lexfrei review):**
- **B1 (blocker)** — `8f41910` converts the SeaweedFS
`-lock`/`-readonly` BucketClasses and the s3 service-name override into
`patches/`, wired into `make update`, so a re-vendor no longer drops
them.
- **FU1** — `cc99230` grants the cilium leak-healer `delete` on pods
(both delete remedies were RBAC-forbidden).
- **FU3** — `014de3f` adds a unit test for the HelmRelease update
conflict-retry path.
- **FU4** — `6deddc6` makes the leak-healer refuse a disconnect for any
non-terminal owner phase, not just `Running`.
- **FU7** — `f3914d2` sets `persist-credentials: false` on the
build-cache checkout (SHA-pinning is handled repo-wide in #2849).
**Production-behavior changes (not in the table above):**
- VPA `updateMode: Auto → Initial` for etcd (`6e9ff90`) and monitoring
(`0997105`), plus `vmselect`/`vmstorage` `minAllowed` floors — stops
install-time eviction churn.
- cozystack-api: `retry.RetryOnConflict` on the Application→HelmRelease
update path (`728f676`).
**E2E robustness follow-ups:** per-test crust-gather snapshots
(`50e5f94`), EtcdBackupSchedule wait (`763b85e`), tenant-node
single-deadline wait (`9e43d45`), harbor BucketClaim 10m budget
(`d7f244f`), tenant API via LoadBalancer (`9beda4e`), crust-gather pin
(`95f42aa`).
**Not addressed (by decision):** FU2 — the s3 Service keeps
`seaweedfs.name`-based naming via patch (rendered output unchanged;
cozystack renders `fullname == seaweedfs`, so it matches the
`componentName`/`fullname` siblings in practice). FU6 — fork PR build
push is handled by the existing mirror-to-same-repo-branch workflow.
## Relates to
#2946#2944#2943#2874#2938#2939#2928#2873 — if this lands, those
can be closed; otherwise they remain the granular per-PR review path.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added CI build cache warming for main branch builds.
* Added a best-effort in-cluster Cilium endpoint self-heal watchdog for
e2e installs.
* Extended SeaweedFS COSI with object-lock support plus separate
readonly access.
* **Improvements**
* Reduced e2e install flakiness with a shared readiness deadline, safer
waits, and improved cleanup/diagnostics capture.
* Prevented install-time churn by switching VPA update mode to
**Initial**.
* Added container startup probes and updated SeaweedFS volume sizing.
* **Tests**
* Improved e2e robustness (etcd backup schedule waiting, digest-pinned
image checks, and corrected S3 port-forwarding).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Adds an operator-facing way to serve platform services and the root
tenant's ingress under a pre-existing wildcard TLS certificate instead
of minting per-host ACME certificates. Implements the "native
references" design decided on #2812.
A single value, `publishing.certificates.wildcardSecretName`, names a
TLS Secret the operator has already created in the publishing namespace
(`tenant-root` by default). Only the name travels through the
`cozystack-values` channel — the certificate and private key are never
broadcast, unlike the public `kube-root-ca` that already rides it.
When set:
- **ingress-nginx path** (`gateway.enabled=false`): the publishing
tenant's ingress controller serves the Secret as its
`--default-ssl-certificate`, and the platform service Ingresses drop
their cert-manager cluster-issuer annotation and `tls` block. No
per-host ACME.
- **Gateway API path** (`gateway.enabled=true`): a new `TenantGateway`
cert mode, `existingSecret`, references the Secret in the wildcard/apex
listener `certificateRefs` and mints no Issuer or Certificate. Switching
into this mode garbage-collects any ACME machinery left from
`http01`/`dns01`.
Scope is the root tenant — everything is same-namespace, so no RBAC or
NetworkPolicy changes. Extending wildcard mode to child tenants needs
the Secret replicated into each tenant namespace and is left as a
follow-up.
Closes#2812. Part of #2811.
### Screenshots
N/A — no UI changes.
### Release note
```release-note
feat(platform): operators can serve platform services and the root tenant's ingress under a pre-existing wildcard TLS certificate via `publishing.certificates.wildcardSecretName`, instead of minting per-host ACME certificates
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added an "existingSecret" TLS certificate mode and a wildcardSecretRef
option for TenantGateway and platform values to use a pre-existing
wildcard TLS Secret.
* **Platform / Templates**
* Introduced operator "wildcard secret" mode across Gateway and many
Helm charts (Ingress, apps) to skip per-host ACME provisioning and
optionally wire a default SSL cert.
* **Tests**
* Added extensive chart and controller tests covering existingSecret
behavior and cert-mode transitions.
* **Documentation**
* Updated gateway docs describing the new wildcard secret mode and
requirements.
* **Chores**
* Added Helm unit test makefile targets to several packages.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The restored COSI -lock/-readonly BucketClass/BucketAccessClass and the s3
Service name override lived only as raw edits in the vendored chart tree, so
the next `make update` (rm -rf charts && curl | tar && patch) would silently
drop them again — the same way the 4.31 bump did. Capture both as patch files
under patches/ and wire them into the Makefile update target, matching the
existing s3-tls-main-port / cosi-provisioner-sa-name customizations. No
rendered output changes; this only makes the customizations survive re-vendor.
Addresses the B1 review blocker on #2948.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
In Auto mode the VPA Updater evicts running pods to apply new recommendations. During install that eviction churn recreates VictoriaMetrics pods faster than Cilium releases their endpoint IPs, tripping the cluster-pool IPAM "IP already in use" (putEndpointIdInvalid) leak that wedges a pod and stalls the install. Initial sets requests at pod creation without ever evicting, removing the trigger (the cilium-leak healer remains as defense for non-VPA churn paths).
Also raise vmstorage/vmselect minAllowed floors to 200m/512Mi: under Initial a pod is sized once at creation and never re-sized at runtime, so the floor is the safety net against the heavy query/storage tiers being left under-provisioned.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Add `rm -rf charts/prometheus-operator-crds` before `helm pull` in the
update target so stale files from previous chart versions don't linger
across updates. This matches the convention used by cilium,
external-secrets-operator, external-dns, and cert-manager Makefiles.
Address review feedback from gemini-code-assist on
packages/system/prometheus-operator-crds/Makefile:9.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The chart previously stripped the bundle to 4 CRDs (ServiceMonitor,
PodMonitor, PrometheusRule, Probe) — the set VictoriaMetrics operator
needs for service discovery. This was correct when the chart was
vendored inside system/victoria-metrics-operator (commit ba5c785cc,
2023), but the prune was carried over when the chart was extracted
to its own package, and again when the CRDs were propagated into
tenant Kubernetes clusters (commit 1e2b66131, 2025).
In tenant clusters the partial set blocks third-party apps that ship
their own prometheus-operator (e.g. kube-prometheus-stack) — the
missing CRDs (Prometheus, Alertmanager, AlertmanagerConfig,
PrometheusAgent, ScrapeConfig, ThanosRuler) prevent the operator from
deploying its instances against the platform-managed CRD set.
Drop the rm -f line in the Makefile so the full upstream bundle ships,
and pin --version 25.0.0 to keep the operator CRD version at v0.87.0
(operator version bump is a separate concern, out of scope here).
Tenant apps can now set crds.enabled: false on their kube-prometheus-
stack release and consume the platform CRDs directly.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The 4.31 chart bump (#2834) broke E2E for every PR: tenant-root
seaweedfs-system never becomes ready.
1. S3 served plaintext. Until 4.05 the -cert.file/-key.file args were
rendered whenever enableSecurity was set, so weed s3 served TLS on the
main port (8333); 4.31 nested them under 'if s3.httpsPort' and we ship
httpsPort=0, so s3 spoke HTTP while the HTTPS probes, the ingress
backend-protocol and the COSI sidecar endpoint all expected HTTPS
('server gave HTTP response to HTTPS client'). Restore the 4.05
behaviour via patch.
2. COSI provisioner ServiceAccount mismatch. cosi-service-account.yaml
and the ClusterRoleBinding name the SA
<serviceAccountName>-objectstorage-provisioner, but the Deployment
referenced componentName (<release>-objectstorage-provisioner). When
serviceAccountName != release name (tenant-root-seaweedfs vs
seaweedfs-system) the Deployment could not create pods
('serviceaccount not found'). Align the Deployment with the SA + binding.
Both fixes are vendored as patches re-applied by 'make update', and
covered by helm-unittest.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
The README advertised cozy_flux_shard_* metrics while the endpoint was
disabled as shipped: serve them plain-HTTP on :8080 with scrape
annotations so the autosizing recommendation is actually observable.
Run the operator non-root with a read-only rootfs, dropped capabilities
and modest resources, quote the inject-ca-from annotation for YAML
tooling, and document the fresh-install bootstrap chain.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Dropping the whole tls block in wildcard mode disabled the forced
HTTP-to-HTTPS redirect: ingress-nginx only issues the 308 redirect when
TLS is enabled for the Ingress, while a host without a tls section is
still served on 443 with the default certificate but stays reachable
over plain HTTP. Keep the tls section with the host list and omit only
the per-host secretName, so ingress-nginx falls back to the operator
wildcard configured as the default SSL certificate and the redirect
stays enforced, matching the non-wildcard behaviour.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
When publishing.certificates.wildcardSecretName is set, the root
tenant's ingress-nginx controller now serves the operator-supplied
wildcard Secret as its --default-ssl-certificate, and the platform
service Ingresses stop requesting their own per-host ACME certificates.
The flag is set only on the publishing controller (Release.Namespace ==
expose-ingress) because ingress-nginx loads the Secret from its own
namespace; child-tenant controllers cannot reach a Secret in another
namespace, so they keep the existing ACME path.
Each service Ingress (dashboard, keycloak, grafana, alerta, bucket,
linstor-gui, harbor, seaweedfs S3, bootbox) drops its cert-manager
cluster-issuer annotation, the http01 challenge-class annotation, and
its tls block in wildcard mode; nginx then falls back to the default
certificate for those hosts. Other annotations and routing are
unchanged. Each file keeps its existing render guard intact, and the
annotations block collapses cleanly where the cert annotations were its
only content.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Operators who already hold a wildcard TLS certificate (purchased, or
issued by a corporate CA) had no way to make a tenant Gateway serve
under it: the only cert modes were http01 (per-listener ACME) and
dns01 (controller-minted wildcard via ACME).
Add a third mode, existingSecret, that references a pre-existing TLS
Secret in the TenantGateway's own namespace via spec.wildcardSecretRef.
In this mode the controller mints no ACME Issuer and no Certificate;
the wildcard, apex, and per-child-apex HTTPS listeners reference the
supplied Secret directly. A missing wildcardSecretRef fails fast with
Ready=False rather than producing a Gateway pointing at a nonexistent
Secret. Switching into existingSecret from http01/dns01 reclaims the
now-unused Issuer and Certificates so no ACME machinery is left behind.
The non-cert listeners (http->https redirect, TLS passthrough) are
unaffected and continue to render.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Closes#2765.
## What this PR does
Makes the passthrough and vGPU variants of `cozystack.gpu-operator`
produce a KubeVirt CR that is immediately usable, without an out-of-band
`kubectl patch`. Today the gpu-operator stack reports green,
`nvidia.com/<model>` shows up under node `.status.allocatable`, but
every `VirtualMachine` referencing the resource via
`domain.devices.hostDevices` is rejected by the admission webhook
because `HostDevices` is not in `developerConfiguration.featureGates`
and the resource is not listed under `permittedHostDevices`. That manual
reconciliation is now driven by the platform.
The change has three parts.
`packages/system/kubevirt` now parameterizes the KubeVirt CR template.
`developerConfiguration.featureGates` keeps its existing static defaults
and appends `.Values.extraFeatureGates`;
`spec.configuration.permittedHostDevices` renders verbatim from
`.Values.permittedHostDevices` when set;
`spec.configuration.mediatedDevicesConfiguration` does the same.
Defaults stay empty so the rendered CR is byte-identical to the previous
template when no overrides are supplied. The chart's `make update`
target re-applies the four sed transforms after fetching the upstream
CR, so refreshing the KubeVirt release version stays a single command.
`packages/core/platform` adds two defaults tables under `files/`.
`gpu-passthrough-defaults.yaml` covers Hopper (H100/H200), Ada Lovelace
(L4/L40/L40S), Ampere (A100 PCIe/SXM, A40, A30, A10), Turing (T4) and
Volta (V100/V100S); PCI vendor:device pairs are stable, `resourceName`
slugs follow the `<arch>_<model>_<form>_<mem>` convention
`nvidia-sandbox-device-plugin` generates as of v25.x, and
`externalResourceProvider: true` is set everywhere because those
resources are advertised by the sandbox plugin rather than KubeVirt's
in-tree device plugin. `gpu-vgpu-defaults.yaml` covers the Pascal–Ampere
mdev profile range (A100-40C/80C, A40-24Q/48Q, A30-24C, A10-24Q,
V100D-32C, T4-16Q) — the same family span the upstream
`vgpu-device-manager` walks `/sys/class/mdev_bus/` for. Ada Lovelace and
Blackwell SR-IOV vGPU is intentionally out of scope for the default
table and is expected to be advertised via the user-override hook
described below.
`templates/bundles/iaas.yaml` now selects between the two tables via a
new `bundles.iaas.gpuOperatorVariant` knob (`default` for passthrough,
`vgpu` for mediated; default is passthrough). The same variant flows
through to the gpu-operator Package CR, replacing the previous
always-default call. When the package is enabled, the platform injects
`HostDevices` into `extraFeatureGates` plus the chosen defaults into
`permittedHostDevices` for the kubevirt component's values. Operators
extend or replace via top-level platform values:
```yaml
gpu:
replaceDefaults: false # append (default) vs. wipe-and-replace
permittedHostDevices:
pciHostDevices: [] # cluster-specific PCI passthrough entries
mediatedDevices: [] # cluster-specific vGPU entries
mediatedDevicesConfiguration: {} # extra mediated-device types / per-node mappings
```
`docs/gpu-vgpu.md` is updated to drop the manual `kubectl patch` step
and describe the auto-wiring contract, the per-variant defaults, the
escape hatches, and the sandbox-device-plugin `resourceName` slug
convention so an operator whose node publishes a different name knows
where to override.
## Test coverage
`helm-unittest` covers eight branches in
`packages/core/platform/tests/bundles_gpu_kubevirt_wiring_test.yaml`:
gpu-operator disabled emits no extras; gpu-operator enabled at default
variant emits the HostDevices gate plus `pciHostDevices`; vgpu variant
switches to `mediatedDevices` plus `mediatedDevicesConfiguration`; user
overrides append by default; `replaceDefaults: true` wipes the platform
table; an unknown variant fails fast with a descriptive error.
`packages/system/kubevirt/tests/kubevirt_cr_test.yaml` pins the
empty-values render and the three populated shapes at the CR template
level. `make helm-unit-tests` passes locally on the full repo.
### Release note
```release-note
feat(platform): the KubeVirt CR is now auto-configured for GPU host-device passthrough (HostDevices feature gate plus permittedHostDevices populated from NVIDIA defaults) whenever cozystack.gpu-operator is in bundles.enabledPackages. Operators no longer need a manual kubectl patch to make VMs schedule against a GPU. The variant is selected via bundles.iaas.gpuOperatorVariant (default: passthrough; vgpu: mediated). Extend or replace the NVIDIA defaults via .gpu.permittedHostDevices and .gpu.replaceDefaults.
**Upgrade note:** the bundle now OWNS spec.configuration.permittedHostDevices on the KubeVirt CR. If you previously hand-edited it (kubectl edit kubevirt), the first reconcile after upgrade overwrites your manual entries with the rendered NVIDIA default table. Before upgrading, move any custom entries into .gpu.permittedHostDevices (set .gpu.replaceDefaults: true to keep only your list) and verify each resourceName against what your nodes advertise (kubectl describe node <node> | grep nvidia.com/) — the default table now carries the exact slug the sandbox device plugin generates, which upper-cases the card's PCI-IDs name and keeps the Tesla/GL tokens (e.g. a Tesla T4 is nvidia.com/TU104GL_TESLA_T4), but a different plugin build can publish a different string; a mismatch makes the admission webhook reject GPU VMs on their next restart or migration.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Bundle-managed GPU Operator with selectable passthrough vs vGPU
variants; platform now injects GPU settings into KubeVirt and forwards
vGPU manager overrides when enabled.
* Default platform mappings for PCI passthrough and vGPU mediated-device
resources added.
* **Documentation**
* Expanded GPU/vGPU deployment guide with bundle-managed vs manual
guidance, wiring/precedence rules, resource-name conventions, and
verification steps.
* **Tests**
* New and expanded tests for bundle→KubeVirt wiring, variant behaviors,
default/override merging, and idempotent template parameterization.
* **Chores**
* Added Helm test target and hardened KubeVirt template
update/idempotency checks.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Add a third architectural variant to `cozystack.gpu-operator` for hosts
where the NVIDIA driver and `nvidia-container-toolkit` are already
installed by the distro package manager (the typical Ubuntu/Debian GPU
node shape: apt-installed driver + apt-installed CT).
The existing variants do not produce a working setup in that shape.
`default` (passthrough) runs `vfioManager`, which unbinds the host
driver to bind `vfio-pci` — with the host driver active,
`k8s-driver-manager` early-exits and the bind never happens, so the GPU
never reaches a pod. `vgpu` requires the proprietary NVIDIA vGPU host
driver and a license server; the user-installed open or proprietary
driver does not satisfy it.
The new `container` variant maps onto upstream gpu-operator's third mode
(`sandboxWorkloads.enabled=false` + `devicePlugin.enabled=true`) with
every component that would conflict with the host install switched off:
- `driver.enabled=false` — use the host driver.
- `toolkit.enabled=false` — host already has `nvidia-container-toolkit`.
Enabling the operator's toolkit DaemonSet would overwrite the
`/etc/containerd/config.toml` the host package configured (via
`nvidia-ctk runtime configure`), breaking the host runtime wiring.
Matches NVIDIA's documented guidance and the existing
`examples/values-native-talos.yaml` reference, which pins the same flag
for the same reason.
- `devicePlugin.enabled=true` — publish `nvidia.com/gpu` to the kubelet.
- `vfioManager.enabled=false` — do not unbind the host driver. Upstream
renders this `true` unconditionally; it only stays at zero pods because
no node carries a `vm-passthrough` sandbox assignment when
`sandboxWorkloads` is off — pinned off so the rendered policy matches
intent.
- `cdi.enabled=false` — upstream defaults this `true`, but CDI spec
generation is serviced by the operator-managed toolkit DaemonSet this
variant disables. An apt host doesn't auto-generate CDI specs
(`nvidia-ctk cdi generate` is a manual step) and uses the classic
`nvidia-container-runtime` hook, so leaving CDI on would point the
device plugin at CDI injection with no specs on the host and silently
break allocation.
- `sandboxWorkloads.enabled=false` — regular pod workloads, not VMs.
With this off, upstream `defaultWorkload` is already `container`, so no
override is needed.
Host prerequisite: because the operator's toolkit component is disabled,
the node's containerd must already have the `nvidia` runtime registered
(`nvidia-ctk runtime configure --runtime=containerd`); `apt install
nvidia-container-toolkit` installs binaries only and does not wire
containerd. With `driver.enabled=false` the operator uses the
pre-installed host driver at its standard location, so a stock apt
install needs no `hostPaths.driverInstallDir` override (Talos lands the
driver under a non-standard prefix — see
`examples/values-native-talos.yaml`).
The bundle wiring (`packages/core/platform/templates/bundles/iaas.yaml`)
already enables `cozystack.gpu-operator` as an optional package — no
bundle changes required. Users opt into this variant by setting
`variant: container` in their `Package` CR.
Verified locally with `helm template` against `values.yaml` +
`values-container.yaml`: ClusterPolicy renders with
`driver.enabled=false`, `toolkit.enabled=false`,
`devicePlugin.enabled=true`, `vfioManager.enabled=false`,
`cdi.enabled=false`, `sandboxWorkloads.enabled=false`. Platform `helm
unittest` and the `gpu-operator` variant bats suite are green. Not yet
smoke-tested on a real Ubuntu GPU node — the config matches the
documented apt-host wiring (legacy `nvidia-container-runtime` hook, CDI
off), but an end-to-end CUDA-pod run on hardware is still owed.
Closes#2764
### Release note
```release-note
feat(gpu-operator): add `container` variant for hosts with a pre-installed NVIDIA driver and nvidia-container-toolkit, exposing containerized GPU workloads via the standard NVIDIA device plugin without VFIO passthrough or vGPU.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a "container" GPU operator variant to support containerized GPU
workloads on hosts with the NVIDIA driver and nvidia-container-toolkit.
* **Documentation**
* Updated variants documentation and OS support table to include the
container variant and per-host requirements (including Ubuntu and Talos
notes).
* Clarified Talos native pod example wording.
* **Tests**
* Added regression tests to verify container-variant configuration and
component enablement/disablement.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
The vm-disk **source image** dropdown in the console lists
`PersistentVolumeClaims` in `cozy-public` (the `vm-default-images-*`
golden images) to offer them when creating a VM disk. That list is
issued by the browser as the **tenant identity** (e.g.
`system:serviceaccount:tenant-root:tenant-root`), but the
`cozy:tenant:dashboard` Role in `cozy-public` only granted read on Flux
`helmrepositories`/`helmcharts`. The list returned `403 Forbidden` and
the dropdown stayed empty even when images had been enabled (e.g. via
`bundles.enabledPackages`).
This is identity-dependent, which is why it can look intermittent: a
cluster-admin login bypasses tenant RBAC and sees the images, while any
real tenant — including `tenant-root-super-admin` — hits the 403.
This PR adds `get`/`list`/`watch` on `persistentvolumeclaims` to the
Role. `cozy-public` is a shared public namespace, so read on its PVCs is
safe to expose to every tenant dashboard subject (the Role is already
bound to the tenant groups and the tenant ServiceAccount). A
helm-unittest regression pins the grant.
Verified with `kubectl auth can-i list pvc -n cozy-public`:
- `--as=system:serviceaccount:tenant-root:tenant-root` → no (before this
PR)
- `--as-group=tenant-root-super-admin` → no (before this PR)
- cluster-admin → yes (why it "worked" for admins)
No console code changed — this is an RBAC-only fix.
### Release note
```release-note
fix(dashboard): allow tenant dashboards to list VM images in cozy-public, fixing the empty source-image dropdown when creating a vm-disk
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Dashboard access permissions have been expanded to include read-level
access to persistent volume claims, enabling users to manage and select
images through the dashboard more effectively.
* **Tests**
* Added test suite to verify dashboard role permissions are correctly
configured and functional.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What
Adds a cross-tenant **Tenant Overview** dashboard to the root/infra
Grafana.
## Why
Platform admins currently have no single view of which tenants consume
what. This shows resource utilization, app counts, and health per tenant
at a glance.
## Panels
- **Fleet summary** — tenants, apps, running pods, CPU/memory/storage
totals
- **Tenant leaderboard** — per-tenant table: CPU & memory (usage vs
requests), storage, pods, PVCs, app count, not-ready apps
- **Top consumers** — Top-N bar gauges by CPU, memory, storage, app
count
- **Trends** — per-tenant CPU & memory usage over time
- **Health** — pod restart rate and not-ready HelmReleases per tenant
## Details
- Registered in `dashboards-infra.list` only, so it deploys to the
root/`cozy-monitoring` Grafana and never exposes cross-tenant data on
per-tenant instances (same pattern as `gpu/gpu-tenants`).
- Apps are counted from `gotk_resource_info` HelmReleases, excluding the
paired `*-system`/`cozy-*` charts.
- Uses only standard cadvisor / kube-state-metrics / Flux metrics.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## New Features
* Added a Tenant Overview dashboard for comprehensive tenant
infrastructure monitoring, including:
* Fleet summary with tenant counts, app deployments, and aggregate
resource usage
* Tenant performance leaderboard with CPU, memory, and storage metrics
* Top consumers visualization by resource type and app count
* Historical usage trends and pod health metrics
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Pin the native s3.trafficDistribution behaviour that replaced the dropped
s3-traffic-distribution.patch: PreferClose is kept on Kubernetes 1.33,
auto-converts to PreferSameZone on 1.35, and the field is omitted below the
1.31 gate. Adds the helm-unittest harness (test target + suite) the package
previously lacked.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Re-vendor the SeaweedFS Helm chart from 4.05 to 4.31. Improvements that land
with the binary: the PostgreSQL filer store moves to pgx v5 with PgBouncer
support and an ON CONFLICT upsert default, HTTPS certificates hot-reload without
a pod restart on cert-manager rotation, and more S3/volume Prometheus metrics
are exported. The S3 service now exposes a metrics port by default.
Adaptations:
- Drop s3-traffic-distribution.patch: native s3.trafficDistribution support
landed upstream in chart 4.12. The native field is gated to Kubernetes >=1.31
and auto-converts to PreferSameZone on >=1.35, which the hardcoded patch did
not do. Set s3.trafficDistribution: PreferClose in values instead.
- Migrate global.* to global.seaweedfs.* in values: chart 4.18 namespaced the
app-specific globals to avoid polluting the shared global namespace when the
chart is used as a subchart. Covers enableSecurity, serviceAccountName,
extraEnvironmentVars and monitoring.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Expose a proxyProtocol value on the Ingress application that turns on
ingress-nginx use-proxy-protocol with real-ip-header=proxy_protocol and
enable-real-ip. The host control plane also enables it through the
_cluster.proxy-protocol switch, applied only to the external-facing
ingress (the namespace named by expose-ingress).
use-forwarded-headers / compute-full-forwarded-for are deliberately not
set: without a paired proxy-real-ip-cidr they let clients spoof
X-Forwarded-For, and the PROXY-protocol frame is already the source of
truth for the client address.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
PR #2711 switched PR builds to push only :pr-<N>-<sha> tags (PUBLISH_FLOATING=0) to stop concurrent-push 409 conflicts, but left --cache-from pointing at $(REGISTRY)/<img>:latest. PR builds set REGISTRY to the per-CI registry (iad.ocir.io) and never publish :latest there, so every PR build's registry cache lookup 404s and rebuilds cold.
Add a CACHE_REGISTRY knob (default ghcr.io/cozystack/cozystack, where release builds do publish :latest) and point every --cache-from at it. Push targets ($(REGISTRY)) and the unique per-PR tags are unchanged, so the #2711 anti-conflict behaviour is preserved while PR builds regain a warm cache from the last release.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The update target gated the insertion of both the permittedHostDevices
and mediatedDevicesConfiguration directives behind a single
`grep permittedHostDevices` guard. A hand-merge from a newer upstream
KubeVirt CR that dropped only mediatedDevicesConfiguration left
permittedHostDevices in place, so the guard skipped the combined sed and
mediatedDevicesConfiguration was never reinserted — then the sanity-check
tail aborted the target with a non-zero exit instead of self-healing.
Split the two directives into independent guarded inserts so each
self-heals on its own. Reword the sanity-check failures to name the
directive that failed to land and the anchor its insert keys off, instead
of implying the anchor itself is missing.
Add a shell regression test that strips one directive at a time and
asserts `make update` restores it without duplicating the other; the
update target gains a CR_FILE override so the test can drive it against a
throwaway copy. Wired into the package test target and guarded to skip on
BSD sed.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The KubeVirt CR's developerConfiguration.featureGates list, the
permittedHostDevices block, and mediatedDevicesConfiguration are now
driven by chart values:
- extraFeatureGates is appended after the static default set so
HostDevices (or any other future gate) can be enabled without
forking the template.
- permittedHostDevices is rendered verbatim when set, allowing the
pciHostDevices (passthrough) and mediatedDevices (vGPU) shapes the
KubeVirt admission webhook requires for VMs that reference a GPU
by resourceName.
- mediatedDevicesConfiguration is rendered verbatim when set,
carrying mediatedDeviceTypes / nodeMediatedDeviceTypes for the
vGPU profile auto-creation loop.
Defaults stay empty so the rendered CR is byte-identical to the
previous template when no overrides are supplied.
The Makefile's update target re-applies the four sed transforms
in-place on templates/kubevirt-cr.yaml — it does not download anything
from upstream KubeVirt because the published CR is a near-empty stub
and would clobber the hand-curated cozystack defaults. To refresh from
a newer upstream release the operator hand-merges the new fields into
the in-tree template, then runs `make update` to re-assert the four
cozystack parameterization patches; grep sanity-checks at the end of
the target fail loudly if any anchor (developerConfiguration,
- VMExport, evictionStrategy) gets dropped during the hand-merge.
Companion helm-unittest coverage pins the default render and the
three populated shapes.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The two existing variants assume the host has neither the NVIDIA driver
nor the container toolkit: default unbinds the host driver to bind
vfio-pci for KubeVirt VMs, vgpu requires the proprietary vGPU host
driver and a license server. Neither path produces a working setup on a
node that already ships the driver and nvidia-container-toolkit via the
distro package manager — the most common shape of a Linux GPU node.
The new container variant surfaces the third architectural mode that
upstream NVIDIA gpu-operator already supports: sandboxWorkloads
disabled, devicePlugin enabled, with every component that would
conflict with the host install switched off.
- driver.enabled=false use the host driver
- toolkit.enabled=false host already has nvidia-container-toolkit;
enabling overwrites the containerd
config.toml the host configured via
nvidia-ctk runtime configure
- devicePlugin.enabled=true publish nvidia.com/gpu to the kubelet
- vfioManager.enabled=false do not unbind the host driver
- cdi.enabled=false CDI spec generation is serviced by the
disabled toolkit; an apt host uses the
legacy nvidia-container-runtime hook, so
pin off to match the host runtime wiring
- sandboxWorkloads.enabled=false regular pod workloads, not VMs
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
With `publishing.proxyProtocol: true`, the host emits a
`cozystack.ouroboros` Package using the no-override Package macro, so
the proxy inherits the wrapper chart default backend
`ingress-nginx-controller.cozy-ingress-nginx`. That layout describes a
managed Kubernetes tenant cluster, not the host: on the host,
`extra/ingress` deploys ingress-nginx into the root tenant namespace as
`root-ingress-controller` in `tenant-root`. The composed backend FQDN
never resolves, so `ouroboros-proxy` never reaches Ready.
This emits the host ouroboros Package with a `proxy.target` override
derived from `publishing.ingressName` — namespace plus
`<short>-ingress-controller` service name — mirroring the `trimPrefix
"tenant-"` transform `extra/ingress` applies to its release namespace.
The derivation reuses the existing platform invariant that
`publishing.ingressName` is the namespace where the host ingress
controller runs.
Adds bundle helm-unittest coverage for the default ingress name and a
non-default one, and corrects the misleading wrapper `values.yaml`
comment that claimed the host uses `cozy-ingress-nginx`.
Scope: this fixes the ouroboros readiness defect only. The separate
defect where the injected nginx PROXY-protocol config keys are not
consumed by the host ingress is tracked independently and not touched
here.
Closes#2797
### Release note
```release-note
fix(networking): point the host ouroboros PROXY-protocol proxy at the root-tenant ingress so it becomes Ready when publishing.proxyProtocol is enabled
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Enhanced proxy protocol configuration to automatically route through
the appropriate ingress-nginx service based on your ingress name
settings.
* **Tests**
* Added test coverage for proxy protocol bundle wiring and ingress
service routing.
* **Documentation**
* Clarified proxy target configuration behavior and fallback mechanisms.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The vm-disk source image dropdown in the console lists PersistentVolumeClaims in cozy-public (the vm-default-images-* golden images). The browser issues that list as the tenant identity (e.g. system:serviceaccount:tenant-root:tenant-root), but the cozy:tenant:dashboard Role only granted read on Flux helmrepositories and helmcharts. The list returned 403 and the dropdown stayed empty even when the images existed.
Add get/list/watch on persistentvolumeclaims to the Role. cozy-public is a shared public namespace, so read on its PVCs is safe to expose to every tenant dashboard subject. A helm-unittest regression pins the grant.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
myasnikovdaniil