Commit Graph
1559 Commits
Author SHA1 Message Date
Aleksei Sviridkin 2e8e6925dc chore(packages/system/nats): bump nats to 1.3.16
Re-vendors the nats Helm chart from 1.3.13 to 1.3.16 (app 2.11.8 -> 2.11.10),
the latest patch in the 1.3 line, picking up the published security fixes plus
upstream bug fixes (route-reconnect storms, stream interest loss when lowering
max-connections, JetStream meta performance). Pins the update target to
--version 1.3.16 so make update stays on the 1.3 line instead of jumping to the
2.x chart restructure. The cozystack overrides (PVC retention policy, cluster
routeURLs domain, prometheus exporter PodMonitor) stay valid.

Adds helm-unittest coverage (the package shipped none): pins the server image,
the PVC retention policy, and the PodMonitor invariant.

Adaptations:
- charts/: re-vendored via make update (helm pull nats --version 1.3.16)
- Makefile: pin update target to --version 1.3.16; add test target
- tests/nats_test.yaml: new helm-unittest coverage

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-22 20:05:54 +03:00
cozystack-ci[bot] e1aaa90ed3 Prepare release v1.5.0
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
2026-06-22 10:55:37 +00:00
Myasnikov DaniilandClaude dc9427285d Merge branch 'main' into feat/flux-shard-operator-impl
Resolve go.mod conflict: keep main's go 1.26.4 and golang.org/x/* bumps, and keep gomodules.xyz/jsonpatch/v2 v2.4.0 as a direct dependency (imported by the flux-shard-operator admission webhook) instead of main's indirect entry. go mod tidy leaves go.sum consistent.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-22 10:19:07 +05:00
Myasnikov DaniilandClaude 0421d71dd8 docs(flux-shard-operator): align parameter table columns
The shardCount description and the shard.resources value overflowed their columns, pushing the internal separators out of line. Reflow every column to its widest cell so the raw table renders cleanly.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-22 10:03:02 +05:00
myasnikovdaniil aeaedca4a4 ci: consolidated platform & e2e stabilization batch (#2948)
## What

Consolidated landing branch for a set of interdependent platform/CI
stabilization fixes. It began as the source PRs in the table below (in
dependency order) and has since grown with review-driven fixes and a few
production-behavior changes — the full current inventory is under
**Scope** below. Landing them as one unit lets CI run against the real
combined state instead of a fragile bottom-up merge train.

## Source PRs (dependency order)

| # | Commit | Source PR | Addresses |
|---|--------|-----------|-----------|
| 1 | fix(capi): startupProbe on capi-controller-manager | #2946 | capi
crashloop during cert provisioning |
| 2 | test(e2e): bucket.bats port-forward + S3 client reliability |
#2944 | flaky bucket test |
| 3 | fix(seaweedfs): restore -lock BucketClass, s3 svc name, drop
volumeSizeLimitMB | #2943 | missing bucket/access classes |
| 4 | test(e2e): in-cluster Cilium endpoint-leak healer (install + apps)
| #2874 | cilium "IP already in use" leak |
| 5 | refactor(build): mode=max registry cache + main-only warmer |
#2938 | build cache |
| 6 | ci(build): isolate each PR build on its own ephemeral runner VM |
#2939 | shared-buildkit contention |
| 7 | fix(e2e): LINSTOR post-install waits on a single 15m deadline |
#2928 | LINSTOR provisioning timeouts |
| 8 | test(metallb): assert digest-pinned image form, not version
literal | #2873 | brittle metallb assertion |

## Why consolidated

These form a dependency DAG (verified from CI logs): e.g. 2943 needs
2946, 2938 needs 2874, 2928 needs 2938/2939, 2873 needs 2928. Tested
bottom-up, each lower PR runs with none of the fixes above it and can't
go green alone. This branch carries all of them, so CI runs against the
real combined state.

## Verification status

CI is green on the latest head (`5653c30`): full-suite E2E passes
end-to-end and is reproducible (2 of 3 runs on this SHA green). The one
red run was a LINSTOR tie-breaker / DRBD-metadata infra flake on the
sandbox, unrelated to the diff — a different environmental subsystem
failed each run (details in the comments below). The earlier 3-hour
crust-gather snapshot hang is fixed in `5653c30`.

## Scope beyond the original 8

The branch has grown past the 8 source PRs above with review-driven
fixes and a few production-behavior changes. For reviewer transparency,
the full set:

**Review fixes (@lexfrei review):**
- **B1 (blocker)** — `8f41910` converts the SeaweedFS
`-lock`/`-readonly` BucketClasses and the s3 service-name override into
`patches/`, wired into `make update`, so a re-vendor no longer drops
them.
- **FU1** — `cc99230` grants the cilium leak-healer `delete` on pods
(both delete remedies were RBAC-forbidden).
- **FU3** — `014de3f` adds a unit test for the HelmRelease update
conflict-retry path.
- **FU4** — `6deddc6` makes the leak-healer refuse a disconnect for any
non-terminal owner phase, not just `Running`.
- **FU7** — `f3914d2` sets `persist-credentials: false` on the
build-cache checkout (SHA-pinning is handled repo-wide in #2849).

**Production-behavior changes (not in the table above):**
- VPA `updateMode: Auto → Initial` for etcd (`6e9ff90`) and monitoring
(`0997105`), plus `vmselect`/`vmstorage` `minAllowed` floors — stops
install-time eviction churn.
- cozystack-api: `retry.RetryOnConflict` on the Application→HelmRelease
update path (`728f676`).

**E2E robustness follow-ups:** per-test crust-gather snapshots
(`50e5f94`), EtcdBackupSchedule wait (`763b85e`), tenant-node
single-deadline wait (`9e43d45`), harbor BucketClaim 10m budget
(`d7f244f`), tenant API via LoadBalancer (`9beda4e`), crust-gather pin
(`95f42aa`).

**Not addressed (by decision):** FU2 — the s3 Service keeps
`seaweedfs.name`-based naming via patch (rendered output unchanged;
cozystack renders `fullname == seaweedfs`, so it matches the
`componentName`/`fullname` siblings in practice). FU6 — fork PR build
push is handled by the existing mirror-to-same-repo-branch workflow.

## Relates to

#2946 #2944 #2943 #2874 #2938 #2939 #2928 #2873 — if this lands, those
can be closed; otherwise they remain the granular per-PR review path.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
  * Added CI build cache warming for main branch builds.
* Added a best-effort in-cluster Cilium endpoint self-heal watchdog for
e2e installs.
* Extended SeaweedFS COSI with object-lock support plus separate
readonly access.

* **Improvements**
* Reduced e2e install flakiness with a shared readiness deadline, safer
waits, and improved cleanup/diagnostics capture.
* Prevented install-time churn by switching VPA update mode to
**Initial**.
  * Added container startup probes and updated SeaweedFS volume sizing.

* **Tests**
* Improved e2e robustness (etcd backup schedule waiting, digest-pinned
image checks, and corrected S3 port-forwarding).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-22 09:19:22 +05:00
Aleksei Sviridkin e8e750ccb0 feat(platform): operator-provided wildcard certificate for platform and tenant ingress (#2819)
## What this PR does

Adds an operator-facing way to serve platform services and the root
tenant's ingress under a pre-existing wildcard TLS certificate instead
of minting per-host ACME certificates. Implements the "native
references" design decided on #2812.

A single value, `publishing.certificates.wildcardSecretName`, names a
TLS Secret the operator has already created in the publishing namespace
(`tenant-root` by default). Only the name travels through the
`cozystack-values` channel — the certificate and private key are never
broadcast, unlike the public `kube-root-ca` that already rides it.

When set:

- **ingress-nginx path** (`gateway.enabled=false`): the publishing
tenant's ingress controller serves the Secret as its
`--default-ssl-certificate`, and the platform service Ingresses drop
their cert-manager cluster-issuer annotation and `tls` block. No
per-host ACME.
- **Gateway API path** (`gateway.enabled=true`): a new `TenantGateway`
cert mode, `existingSecret`, references the Secret in the wildcard/apex
listener `certificateRefs` and mints no Issuer or Certificate. Switching
into this mode garbage-collects any ACME machinery left from
`http01`/`dns01`.

Scope is the root tenant — everything is same-namespace, so no RBAC or
NetworkPolicy changes. Extending wildcard mode to child tenants needs
the Secret replicated into each tenant namespace and is left as a
follow-up.

Closes #2812. Part of #2811.

### Screenshots

N/A — no UI changes.

### Release note

```release-note
feat(platform): operators can serve platform services and the root tenant's ingress under a pre-existing wildcard TLS certificate via `publishing.certificates.wildcardSecretName`, instead of minting per-host ACME certificates
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added an "existingSecret" TLS certificate mode and a wildcardSecretRef
option for TenantGateway and platform values to use a pre-existing
wildcard TLS Secret.

* **Platform / Templates**
* Introduced operator "wildcard secret" mode across Gateway and many
Helm charts (Ingress, apps) to skip per-host ACME provisioning and
optionally wire a default SSL cert.

* **Tests**
* Added extensive chart and controller tests covering existingSecret
behavior and cert-mode transitions.

* **Documentation**
* Updated gateway docs describing the new wildcard secret mode and
requirements.

* **Chores**
  * Added Helm unit test makefile targets to several packages.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-21 17:37:18 +03:00
Andrei KvapilandClaude 8f41910e87 fix(seaweedfs): vendor -lock/-readonly BucketClass + s3 service name as patches
The restored COSI -lock/-readonly BucketClass/BucketAccessClass and the s3
Service name override lived only as raw edits in the vendored chart tree, so
the next `make update` (rm -rf charts && curl | tar && patch) would silently
drop them again — the same way the 4.31 bump did. Capture both as patch files
under patches/ and wire them into the Makefile update target, matching the
existing s3-tls-main-port / cosi-provisioner-sa-name customizations. No
rendered output changes; this only makes the customizations survive re-vendor.

Addresses the B1 review blocker on #2948.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-06-19 21:30:38 +02:00
Myasnikov DaniilandClaude Opus 4.8 0997105b05 fix(monitoring): set VPA updateMode to Initial to stop install-time eviction churn
In Auto mode the VPA Updater evicts running pods to apply new recommendations. During install that eviction churn recreates VictoriaMetrics pods faster than Cilium releases their endpoint IPs, tripping the cluster-pool IPAM "IP already in use" (putEndpointIdInvalid) leak that wedges a pod and stalls the install. Initial sets requests at pod creation without ever evicting, removing the trigger (the cilium-leak healer remains as defense for non-VPA churn paths).

Also raise vmstorage/vmselect minAllowed floors to 200m/512Mi: under Initial a pod is sized once at creation and never re-sized at runtime, so the floor is the safety net against the heavy query/storage tiers being left under-provisioned.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-18 12:57:23 +05:00
Myasnikov DaniilandClaude Opus 4.8 df9dc74f09 test(metallb): assert digest-pinned image form, not exact version literal (#2873)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-18 10:40:06 +05:00
Myasnikov DaniilandClaude Opus 4.8 d3958c5bcb refactor(build): mode=max registry cache with a main-only warmer (#2938)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-18 10:40:05 +05:00
Myasnikov DaniilandClaude Opus 4.8 725b19e446 fix(seaweedfs): restore -lock BucketClass, s3 service name, and drop volumeSizeLimitMB (#2943)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-18 10:40:05 +05:00
Myasnikov DaniilandClaude Opus 4.8 34c4f0a152 fix(capi): add startupProbe to capi-controller-manager to survive cert provisioning delay (#2946)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-18 10:40:05 +05:00
Myasnikov Daniil 97cf5857a3 fix(prometheus-operator-crds): clean chart dir before helm pull
Add `rm -rf charts/prometheus-operator-crds` before `helm pull` in the
update target so stale files from previous chart versions don't linger
across updates. This matches the convention used by cilium,
external-secrets-operator, external-dns, and cert-manager Makefiles.

Address review feedback from gemini-code-assist on
packages/system/prometheus-operator-crds/Makefile:9.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-16 14:19:47 +05:00
Myasnikov Daniil dbbaca16eb fix(prometheus-operator-crds): ship full upstream CRD bundle
The chart previously stripped the bundle to 4 CRDs (ServiceMonitor,
PodMonitor, PrometheusRule, Probe) — the set VictoriaMetrics operator
needs for service discovery. This was correct when the chart was
vendored inside system/victoria-metrics-operator (commit ba5c785cc,
2023), but the prune was carried over when the chart was extracted
to its own package, and again when the CRDs were propagated into
tenant Kubernetes clusters (commit 1e2b66131, 2025).

In tenant clusters the partial set blocks third-party apps that ship
their own prometheus-operator (e.g. kube-prometheus-stack) — the
missing CRDs (Prometheus, Alertmanager, AlertmanagerConfig,
PrometheusAgent, ScrapeConfig, ThanosRuler) prevent the operator from
deploying its instances against the platform-managed CRD set.

Drop the rm -f line in the Makefile so the full upstream bundle ships,
and pin --version 25.0.0 to keep the operator CRD version at v0.87.0
(operator version bump is a separate concern, out of scope here).

Tenant apps can now set crds.enabled: false on their kube-prometheus-
stack release and consume the platform CRDs directly.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-16 14:19:47 +05:00
Andrei KvapilandClaude 484d73bf78 fix(seaweedfs): restore S3 TLS and COSI provisioner SA after 4.31 bump
The 4.31 chart bump (#2834) broke E2E for every PR: tenant-root
seaweedfs-system never becomes ready.

1. S3 served plaintext. Until 4.05 the -cert.file/-key.file args were
   rendered whenever enableSecurity was set, so weed s3 served TLS on the
   main port (8333); 4.31 nested them under 'if s3.httpsPort' and we ship
   httpsPort=0, so s3 spoke HTTP while the HTTPS probes, the ingress
   backend-protocol and the COSI sidecar endpoint all expected HTTPS
   ('server gave HTTP response to HTTPS client'). Restore the 4.05
   behaviour via patch.

2. COSI provisioner ServiceAccount mismatch. cosi-service-account.yaml
   and the ClusterRoleBinding name the SA
   <serviceAccountName>-objectstorage-provisioner, but the Deployment
   referenced componentName (<release>-objectstorage-provisioner). When
   serviceAccountName != release name (tenant-root-seaweedfs vs
   seaweedfs-system) the Deployment could not create pods
   ('serviceaccount not found'). Align the Deployment with the SA + binding.

Both fixes are vendored as patches re-applied by 'make update', and
covered by helm-unittest.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-06-14 09:33:16 +03:00
Andrei KvapilandClaude c50005b7d5 feat(flux-shard-operator): expose metrics and harden the operator pod
The README advertised cozy_flux_shard_* metrics while the endpoint was
disabled as shipped: serve them plain-HTTP on :8080 with scrape
annotations so the autosizing recommendation is actually observable.
Run the operator non-root with a read-only rootfs, dropped capabilities
and modest resources, quote the inject-ca-from annotation for YAML
tooling, and document the fresh-install bootstrap chain.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-06-11 13:49:34 +03:00
Aleksei Sviridkin 13e310fe71 Merge branch 'main' into chore/seaweedfs-4.31 2026-06-10 11:50:02 +03:00
Aleksei Sviridkin 9e3ae7447b fix(ingress): keep tls section in operator wildcard mode
Dropping the whole tls block in wildcard mode disabled the forced
HTTP-to-HTTPS redirect: ingress-nginx only issues the 308 redirect when
TLS is enabled for the Ingress, while a host without a tls section is
still served on 443 with the default certificate but stays reachable
over plain HTTP. Keep the tls section with the host list and omit only
the per-host secretName, so ingress-nginx falls back to the operator
wildcard configured as the default SSL certificate and the redirect
stays enforced, matching the non-wildcard behaviour.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-10 03:18:42 +03:00
Aleksei Sviridkin b1f5d9579c feat(ingress): serve operator wildcard via default SSL certificate
When publishing.certificates.wildcardSecretName is set, the root
tenant's ingress-nginx controller now serves the operator-supplied
wildcard Secret as its --default-ssl-certificate, and the platform
service Ingresses stop requesting their own per-host ACME certificates.

The flag is set only on the publishing controller (Release.Namespace ==
expose-ingress) because ingress-nginx loads the Secret from its own
namespace; child-tenant controllers cannot reach a Secret in another
namespace, so they keep the existing ACME path.

Each service Ingress (dashboard, keycloak, grafana, alerta, bucket,
linstor-gui, harbor, seaweedfs S3, bootbox) drops its cert-manager
cluster-issuer annotation, the http01 challenge-class annotation, and
its tls block in wildcard mode; nginx then falls back to the default
certificate for those hosts. Other annotations and routing are
unchanged. Each file keeps its existing render guard intact, and the
annotations block collapses cleanly where the cert annotations were its
only content.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-10 03:08:41 +03:00
Aleksei Sviridkin 4262658f0f feat(tenantgateway): add existingSecret cert mode for operator wildcard
Operators who already hold a wildcard TLS certificate (purchased, or
issued by a corporate CA) had no way to make a tenant Gateway serve
under it: the only cert modes were http01 (per-listener ACME) and
dns01 (controller-minted wildcard via ACME).

Add a third mode, existingSecret, that references a pre-existing TLS
Secret in the TenantGateway's own namespace via spec.wildcardSecretRef.
In this mode the controller mints no ACME Issuer and no Certificate;
the wildcard, apex, and per-child-apex HTTPS listeners reference the
supplied Secret directly. A missing wildcardSecretRef fails fast with
Ready=False rather than producing a Gateway pointing at a nonexistent
Secret. Switching into existingSecret from http01/dns01 reclaims the
now-unused Issuer and Certificates so no ACME machinery is left behind.

The non-cert listeners (http->https redirect, TLS passthrough) are
unaffected and continue to render.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-10 03:08:07 +03:00
Aleksei Sviridkin 1d747b3d8d feat(platform): auto-wire KubeVirt permittedHostDevices and HostDevices feature gate when gpu-operator is enabled (#2768)
Closes #2765.

## What this PR does

Makes the passthrough and vGPU variants of `cozystack.gpu-operator`
produce a KubeVirt CR that is immediately usable, without an out-of-band
`kubectl patch`. Today the gpu-operator stack reports green,
`nvidia.com/<model>` shows up under node `.status.allocatable`, but
every `VirtualMachine` referencing the resource via
`domain.devices.hostDevices` is rejected by the admission webhook
because `HostDevices` is not in `developerConfiguration.featureGates`
and the resource is not listed under `permittedHostDevices`. That manual
reconciliation is now driven by the platform.

The change has three parts.

`packages/system/kubevirt` now parameterizes the KubeVirt CR template.
`developerConfiguration.featureGates` keeps its existing static defaults
and appends `.Values.extraFeatureGates`;
`spec.configuration.permittedHostDevices` renders verbatim from
`.Values.permittedHostDevices` when set;
`spec.configuration.mediatedDevicesConfiguration` does the same.
Defaults stay empty so the rendered CR is byte-identical to the previous
template when no overrides are supplied. The chart's `make update`
target re-applies the four sed transforms after fetching the upstream
CR, so refreshing the KubeVirt release version stays a single command.

`packages/core/platform` adds two defaults tables under `files/`.
`gpu-passthrough-defaults.yaml` covers Hopper (H100/H200), Ada Lovelace
(L4/L40/L40S), Ampere (A100 PCIe/SXM, A40, A30, A10), Turing (T4) and
Volta (V100/V100S); PCI vendor:device pairs are stable, `resourceName`
slugs follow the `<arch>_<model>_<form>_<mem>` convention
`nvidia-sandbox-device-plugin` generates as of v25.x, and
`externalResourceProvider: true` is set everywhere because those
resources are advertised by the sandbox plugin rather than KubeVirt's
in-tree device plugin. `gpu-vgpu-defaults.yaml` covers the Pascal–Ampere
mdev profile range (A100-40C/80C, A40-24Q/48Q, A30-24C, A10-24Q,
V100D-32C, T4-16Q) — the same family span the upstream
`vgpu-device-manager` walks `/sys/class/mdev_bus/` for. Ada Lovelace and
Blackwell SR-IOV vGPU is intentionally out of scope for the default
table and is expected to be advertised via the user-override hook
described below.

`templates/bundles/iaas.yaml` now selects between the two tables via a
new `bundles.iaas.gpuOperatorVariant` knob (`default` for passthrough,
`vgpu` for mediated; default is passthrough). The same variant flows
through to the gpu-operator Package CR, replacing the previous
always-default call. When the package is enabled, the platform injects
`HostDevices` into `extraFeatureGates` plus the chosen defaults into
`permittedHostDevices` for the kubevirt component's values. Operators
extend or replace via top-level platform values:

```yaml
gpu:
  replaceDefaults: false              # append (default) vs. wipe-and-replace
  permittedHostDevices:
    pciHostDevices: []                # cluster-specific PCI passthrough entries
    mediatedDevices: []               # cluster-specific vGPU entries
  mediatedDevicesConfiguration: {}    # extra mediated-device types / per-node mappings
```

`docs/gpu-vgpu.md` is updated to drop the manual `kubectl patch` step
and describe the auto-wiring contract, the per-variant defaults, the
escape hatches, and the sandbox-device-plugin `resourceName` slug
convention so an operator whose node publishes a different name knows
where to override.

## Test coverage

`helm-unittest` covers eight branches in
`packages/core/platform/tests/bundles_gpu_kubevirt_wiring_test.yaml`:
gpu-operator disabled emits no extras; gpu-operator enabled at default
variant emits the HostDevices gate plus `pciHostDevices`; vgpu variant
switches to `mediatedDevices` plus `mediatedDevicesConfiguration`; user
overrides append by default; `replaceDefaults: true` wipes the platform
table; an unknown variant fails fast with a descriptive error.
`packages/system/kubevirt/tests/kubevirt_cr_test.yaml` pins the
empty-values render and the three populated shapes at the CR template
level. `make helm-unit-tests` passes locally on the full repo.

### Release note

```release-note
feat(platform): the KubeVirt CR is now auto-configured for GPU host-device passthrough (HostDevices feature gate plus permittedHostDevices populated from NVIDIA defaults) whenever cozystack.gpu-operator is in bundles.enabledPackages. Operators no longer need a manual kubectl patch to make VMs schedule against a GPU. The variant is selected via bundles.iaas.gpuOperatorVariant (default: passthrough; vgpu: mediated). Extend or replace the NVIDIA defaults via .gpu.permittedHostDevices and .gpu.replaceDefaults.

**Upgrade note:** the bundle now OWNS spec.configuration.permittedHostDevices on the KubeVirt CR. If you previously hand-edited it (kubectl edit kubevirt), the first reconcile after upgrade overwrites your manual entries with the rendered NVIDIA default table. Before upgrading, move any custom entries into .gpu.permittedHostDevices (set .gpu.replaceDefaults: true to keep only your list) and verify each resourceName against what your nodes advertise (kubectl describe node <node> | grep nvidia.com/) — the default table now carries the exact slug the sandbox device plugin generates, which upper-cases the card's PCI-IDs name and keeps the Tesla/GL tokens (e.g. a Tesla T4 is nvidia.com/TU104GL_TESLA_T4), but a different plugin build can publish a different string; a mismatch makes the admission webhook reject GPU VMs on their next restart or migration.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Bundle-managed GPU Operator with selectable passthrough vs vGPU
variants; platform now injects GPU settings into KubeVirt and forwards
vGPU manager overrides when enabled.
* Default platform mappings for PCI passthrough and vGPU mediated-device
resources added.

* **Documentation**
* Expanded GPU/vGPU deployment guide with bundle-managed vs manual
guidance, wiring/precedence rules, resource-name conventions, and
verification steps.

* **Tests**
* New and expanded tests for bundle→KubeVirt wiring, variant behaviors,
default/override merging, and idempotent template parameterization.

* **Chores**
* Added Helm test target and hardened KubeVirt template
update/idempotency checks.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-10 02:02:01 +03:00
Aleksei Sviridkin b6108039b7 feat(gpu-operator): add container variant for preinstalled host driver (#2766)
## What this PR does

Add a third architectural variant to `cozystack.gpu-operator` for hosts
where the NVIDIA driver and `nvidia-container-toolkit` are already
installed by the distro package manager (the typical Ubuntu/Debian GPU
node shape: apt-installed driver + apt-installed CT).

The existing variants do not produce a working setup in that shape.
`default` (passthrough) runs `vfioManager`, which unbinds the host
driver to bind `vfio-pci` — with the host driver active,
`k8s-driver-manager` early-exits and the bind never happens, so the GPU
never reaches a pod. `vgpu` requires the proprietary NVIDIA vGPU host
driver and a license server; the user-installed open or proprietary
driver does not satisfy it.

The new `container` variant maps onto upstream gpu-operator's third mode
(`sandboxWorkloads.enabled=false` + `devicePlugin.enabled=true`) with
every component that would conflict with the host install switched off:

- `driver.enabled=false` — use the host driver.
- `toolkit.enabled=false` — host already has `nvidia-container-toolkit`.
Enabling the operator's toolkit DaemonSet would overwrite the
`/etc/containerd/config.toml` the host package configured (via
`nvidia-ctk runtime configure`), breaking the host runtime wiring.
Matches NVIDIA's documented guidance and the existing
`examples/values-native-talos.yaml` reference, which pins the same flag
for the same reason.
- `devicePlugin.enabled=true` — publish `nvidia.com/gpu` to the kubelet.
- `vfioManager.enabled=false` — do not unbind the host driver. Upstream
renders this `true` unconditionally; it only stays at zero pods because
no node carries a `vm-passthrough` sandbox assignment when
`sandboxWorkloads` is off — pinned off so the rendered policy matches
intent.
- `cdi.enabled=false` — upstream defaults this `true`, but CDI spec
generation is serviced by the operator-managed toolkit DaemonSet this
variant disables. An apt host doesn't auto-generate CDI specs
(`nvidia-ctk cdi generate` is a manual step) and uses the classic
`nvidia-container-runtime` hook, so leaving CDI on would point the
device plugin at CDI injection with no specs on the host and silently
break allocation.
- `sandboxWorkloads.enabled=false` — regular pod workloads, not VMs.
With this off, upstream `defaultWorkload` is already `container`, so no
override is needed.

Host prerequisite: because the operator's toolkit component is disabled,
the node's containerd must already have the `nvidia` runtime registered
(`nvidia-ctk runtime configure --runtime=containerd`); `apt install
nvidia-container-toolkit` installs binaries only and does not wire
containerd. With `driver.enabled=false` the operator uses the
pre-installed host driver at its standard location, so a stock apt
install needs no `hostPaths.driverInstallDir` override (Talos lands the
driver under a non-standard prefix — see
`examples/values-native-talos.yaml`).

The bundle wiring (`packages/core/platform/templates/bundles/iaas.yaml`)
already enables `cozystack.gpu-operator` as an optional package — no
bundle changes required. Users opt into this variant by setting
`variant: container` in their `Package` CR.

Verified locally with `helm template` against `values.yaml` +
`values-container.yaml`: ClusterPolicy renders with
`driver.enabled=false`, `toolkit.enabled=false`,
`devicePlugin.enabled=true`, `vfioManager.enabled=false`,
`cdi.enabled=false`, `sandboxWorkloads.enabled=false`. Platform `helm
unittest` and the `gpu-operator` variant bats suite are green. Not yet
smoke-tested on a real Ubuntu GPU node — the config matches the
documented apt-host wiring (legacy `nvidia-container-runtime` hook, CDI
off), but an end-to-end CUDA-pod run on hardware is still owed.

Closes #2764

### Release note

```release-note
feat(gpu-operator): add `container` variant for hosts with a pre-installed NVIDIA driver and nvidia-container-toolkit, exposing containerized GPU workloads via the standard NVIDIA device plugin without VFIO passthrough or vGPU.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added a "container" GPU operator variant to support containerized GPU
workloads on hosts with the NVIDIA driver and nvidia-container-toolkit.

* **Documentation**
* Updated variants documentation and OS support table to include the
container variant and per-host requirements (including Ubuntu and Talos
notes).
  * Clarified Talos native pod example wording.

* **Tests**
* Added regression tests to verify container-variant configuration and
component enablement/disablement.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-10 02:00:59 +03:00
Aleksei Sviridkin 28dfb40579 fix(dashboard): grant tenant dashboard read on cozy-public PVCs (#2843)
## What this PR does

The vm-disk **source image** dropdown in the console lists
`PersistentVolumeClaims` in `cozy-public` (the `vm-default-images-*`
golden images) to offer them when creating a VM disk. That list is
issued by the browser as the **tenant identity** (e.g.
`system:serviceaccount:tenant-root:tenant-root`), but the
`cozy:tenant:dashboard` Role in `cozy-public` only granted read on Flux
`helmrepositories`/`helmcharts`. The list returned `403 Forbidden` and
the dropdown stayed empty even when images had been enabled (e.g. via
`bundles.enabledPackages`).

This is identity-dependent, which is why it can look intermittent: a
cluster-admin login bypasses tenant RBAC and sees the images, while any
real tenant — including `tenant-root-super-admin` — hits the 403.

This PR adds `get`/`list`/`watch` on `persistentvolumeclaims` to the
Role. `cozy-public` is a shared public namespace, so read on its PVCs is
safe to expose to every tenant dashboard subject (the Role is already
bound to the tenant groups and the tenant ServiceAccount). A
helm-unittest regression pins the grant.

Verified with `kubectl auth can-i list pvc -n cozy-public`:
- `--as=system:serviceaccount:tenant-root:tenant-root` → no (before this
PR)
- `--as-group=tenant-root-super-admin` → no (before this PR)
- cluster-admin → yes (why it "worked" for admins)

No console code changed — this is an RBAC-only fix.

### Release note

```release-note
fix(dashboard): allow tenant dashboards to list VM images in cozy-public, fixing the empty source-image dropdown when creating a vm-disk
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Dashboard access permissions have been expanded to include read-level
access to persistent volume claims, enabling users to manage and select
images through the dashboard more effectively.

* **Tests**
* Added test suite to verify dashboard role permissions are correctly
configured and functional.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-09 23:51:46 +03:00
Aleksei Sviridkin 5982018629 feat(monitoring): add tenant overview dashboard (#2809)
## What
Adds a cross-tenant **Tenant Overview** dashboard to the root/infra
Grafana.

## Why
Platform admins currently have no single view of which tenants consume
what. This shows resource utilization, app counts, and health per tenant
at a glance.

## Panels
- **Fleet summary** — tenants, apps, running pods, CPU/memory/storage
totals
- **Tenant leaderboard** — per-tenant table: CPU & memory (usage vs
requests), storage, pods, PVCs, app count, not-ready apps
- **Top consumers** — Top-N bar gauges by CPU, memory, storage, app
count
- **Trends** — per-tenant CPU & memory usage over time
- **Health** — pod restart rate and not-ready HelmReleases per tenant

## Details
- Registered in `dashboards-infra.list` only, so it deploys to the
root/`cozy-monitoring` Grafana and never exposes cross-tenant data on
per-tenant instances (same pattern as `gpu/gpu-tenants`).
- Apps are counted from `gotk_resource_info` HelmReleases, excluding the
paired `*-system`/`cozy-*` charts.
- Uses only standard cadvisor / kube-state-metrics / Flux metrics.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

## New Features
* Added a Tenant Overview dashboard for comprehensive tenant
infrastructure monitoring, including:
* Fleet summary with tenant counts, app deployments, and aggregate
resource usage
  * Tenant performance leaderboard with CPU, memory, and storage metrics
  * Top consumers visualization by resource type and app count
  * Historical usage trends and pod health metrics

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-09 23:51:42 +03:00
Aleksei Sviridkin ed367f30dd test(packages/system/seaweedfs): cover s3.trafficDistribution version gating
Pin the native s3.trafficDistribution behaviour that replaced the dropped
s3-traffic-distribution.patch: PreferClose is kept on Kubernetes 1.33,
auto-converts to PreferSameZone on 1.35, and the field is omitted below the
1.31 gate. Adds the helm-unittest harness (test target + suite) the package
previously lacked.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-09 12:52:06 +03:00
Aleksei Sviridkin 068af203ac chore(packages/system/seaweedfs): bump seaweedfs to 4.31
Re-vendor the SeaweedFS Helm chart from 4.05 to 4.31. Improvements that land
with the binary: the PostgreSQL filer store moves to pgx v5 with PgBouncer
support and an ON CONFLICT upsert default, HTTPS certificates hot-reload without
a pod restart on cert-manager rotation, and more S3/volume Prometheus metrics
are exported. The S3 service now exposes a metrics port by default.

Adaptations:
- Drop s3-traffic-distribution.patch: native s3.trafficDistribution support
  landed upstream in chart 4.12. The native field is gated to Kubernetes >=1.31
  and auto-converts to PreferSameZone on >=1.35, which the hardcoded patch did
  not do. Set s3.trafficDistribution: PreferClose in values instead.
- Migrate global.* to global.seaweedfs.* in values: chart 4.18 namespaced the
  app-specific globals to avoid polluting the shared global namespace when the
  chart is used as a subchart. Covers enableSecurity, serviceAccountName,
  extraEnvironmentVars and monitoring.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-09 12:52:06 +03:00
Aleksei Sviridkin 8df4b0b0f1 feat(ingress): add proxyProtocol knob to ingress-nginx
Expose a proxyProtocol value on the Ingress application that turns on
ingress-nginx use-proxy-protocol with real-ip-header=proxy_protocol and
enable-real-ip. The host control plane also enables it through the
_cluster.proxy-protocol switch, applied only to the external-facing
ingress (the namespace named by expose-ingress).

use-forwarded-headers / compute-full-forwarded-for are deliberately not
set: without a paired proxy-real-ip-cidr they let clients spoof
X-Forwarded-For, and the PROXY-protocol frame is already the source of
truth for the client address.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-09 12:50:35 +03:00
Myasnikov Daniil 9868c43e53 fix(build): read --cache-from from ghcr :latest so PR builds warm-start
PR #2711 switched PR builds to push only :pr-<N>-<sha> tags (PUBLISH_FLOATING=0) to stop concurrent-push 409 conflicts, but left --cache-from pointing at $(REGISTRY)/<img>:latest. PR builds set REGISTRY to the per-CI registry (iad.ocir.io) and never publish :latest there, so every PR build's registry cache lookup 404s and rebuilds cold.

Add a CACHE_REGISTRY knob (default ghcr.io/cozystack/cozystack, where release builds do publish :latest) and point every --cache-from at it. Push targets ($(REGISTRY)) and the unique per-PR tags are unchanged, so the #2711 anti-conflict behaviour is preserved while PR builds regain a warm cache from the last release.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-09 14:24:36 +05:00
Aleksei Sviridkin 285f42aaeb fix(kubevirt): reinsert each GPU CR patch independently in make update
The update target gated the insertion of both the permittedHostDevices
and mediatedDevicesConfiguration directives behind a single
`grep permittedHostDevices` guard. A hand-merge from a newer upstream
KubeVirt CR that dropped only mediatedDevicesConfiguration left
permittedHostDevices in place, so the guard skipped the combined sed and
mediatedDevicesConfiguration was never reinserted — then the sanity-check
tail aborted the target with a non-zero exit instead of self-healing.

Split the two directives into independent guarded inserts so each
self-heals on its own. Reword the sanity-check failures to name the
directive that failed to land and the anchor its insert keys off, instead
of implying the anchor itself is missing.

Add a shell regression test that strips one directive at a time and
asserts `make update` restores it without duplicating the other; the
update target gains a CR_FILE override so the test can drive it against a
throwaway copy. Wired into the package test target and guarded to skip on
BSD sed.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-08 17:15:00 +03:00
Aleksei Sviridkin 8c507fe4c5 feat(kubevirt): parameterize CR for host-device GPU configuration
The KubeVirt CR's developerConfiguration.featureGates list, the
permittedHostDevices block, and mediatedDevicesConfiguration are now
driven by chart values:

  - extraFeatureGates is appended after the static default set so
    HostDevices (or any other future gate) can be enabled without
    forking the template.
  - permittedHostDevices is rendered verbatim when set, allowing the
    pciHostDevices (passthrough) and mediatedDevices (vGPU) shapes the
    KubeVirt admission webhook requires for VMs that reference a GPU
    by resourceName.
  - mediatedDevicesConfiguration is rendered verbatim when set,
    carrying mediatedDeviceTypes / nodeMediatedDeviceTypes for the
    vGPU profile auto-creation loop.

Defaults stay empty so the rendered CR is byte-identical to the
previous template when no overrides are supplied.

The Makefile's update target re-applies the four sed transforms
in-place on templates/kubevirt-cr.yaml — it does not download anything
from upstream KubeVirt because the published CR is a near-empty stub
and would clobber the hand-curated cozystack defaults. To refresh from
a newer upstream release the operator hand-merges the new fields into
the in-tree template, then runs `make update` to re-assert the four
cozystack parameterization patches; grep sanity-checks at the end of
the target fail loudly if any anchor (developerConfiguration,
- VMExport, evictionStrategy) gets dropped during the hand-merge.

Companion helm-unittest coverage pins the default render and the
three populated shapes.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-08 17:01:51 +03:00
Aleksei Sviridkin 9352d3ee0e feat(gpu-operator): add container variant for preinstalled host driver
The two existing variants assume the host has neither the NVIDIA driver
nor the container toolkit: default unbinds the host driver to bind
vfio-pci for KubeVirt VMs, vgpu requires the proprietary vGPU host
driver and a license server. Neither path produces a working setup on a
node that already ships the driver and nvidia-container-toolkit via the
distro package manager — the most common shape of a Linux GPU node.

The new container variant surfaces the third architectural mode that
upstream NVIDIA gpu-operator already supports: sandboxWorkloads
disabled, devicePlugin enabled, with every component that would
conflict with the host install switched off.

- driver.enabled=false        use the host driver
- toolkit.enabled=false       host already has nvidia-container-toolkit;
                              enabling overwrites the containerd
                              config.toml the host configured via
                              nvidia-ctk runtime configure
- devicePlugin.enabled=true   publish nvidia.com/gpu to the kubelet
- vfioManager.enabled=false   do not unbind the host driver
- cdi.enabled=false           CDI spec generation is serviced by the
                              disabled toolkit; an apt host uses the
                              legacy nvidia-container-runtime hook, so
                              pin off to match the host runtime wiring
- sandboxWorkloads.enabled=false   regular pod workloads, not VMs

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-08 16:58:51 +03:00
Aleksei Sviridkin 80fbfa1ed8 fix(networking): point host ouroboros proxy at the root-tenant ingress (#2800)
## What this PR does

With `publishing.proxyProtocol: true`, the host emits a
`cozystack.ouroboros` Package using the no-override Package macro, so
the proxy inherits the wrapper chart default backend
`ingress-nginx-controller.cozy-ingress-nginx`. That layout describes a
managed Kubernetes tenant cluster, not the host: on the host,
`extra/ingress` deploys ingress-nginx into the root tenant namespace as
`root-ingress-controller` in `tenant-root`. The composed backend FQDN
never resolves, so `ouroboros-proxy` never reaches Ready.

This emits the host ouroboros Package with a `proxy.target` override
derived from `publishing.ingressName` — namespace plus
`<short>-ingress-controller` service name — mirroring the `trimPrefix
"tenant-"` transform `extra/ingress` applies to its release namespace.
The derivation reuses the existing platform invariant that
`publishing.ingressName` is the namespace where the host ingress
controller runs.

Adds bundle helm-unittest coverage for the default ingress name and a
non-default one, and corrects the misleading wrapper `values.yaml`
comment that claimed the host uses `cozy-ingress-nginx`.

Scope: this fixes the ouroboros readiness defect only. The separate
defect where the injected nginx PROXY-protocol config keys are not
consumed by the host ingress is tracked independently and not touched
here.

Closes #2797

### Release note

```release-note
fix(networking): point the host ouroboros PROXY-protocol proxy at the root-tenant ingress so it becomes Ready when publishing.proxyProtocol is enabled
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Enhanced proxy protocol configuration to automatically route through
the appropriate ingress-nginx service based on your ingress name
settings.

* **Tests**
* Added test coverage for proxy protocol bundle wiring and ingress
service routing.

* **Documentation**
* Clarified proxy target configuration behavior and fallback mechanisms.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-08 16:41:15 +03:00
Myasnikov Daniil bb322138ec fix(dashboard): grant tenant dashboard read on cozy-public PVCs
The vm-disk source image dropdown in the console lists PersistentVolumeClaims in cozy-public (the vm-default-images-* golden images). The browser issues that list as the tenant identity (e.g. system:serviceaccount:tenant-root:tenant-root), but the cozy:tenant:dashboard Role only granted read on Flux helmrepositories and helmcharts. The list returned 403 and the dropdown stayed empty even when the images existed.

Add get/list/watch on persistentvolumeclaims to the Role. cozy-public is a shared public namespace, so read on its PVCs is safe to expose to every tenant dashboard subject. A helm-unittest regression pins the grant.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-08 15:31:41 +05:00
Aleksei Sviridkin bc03142c08 feat(ouroboros): bump to v0.8.0 (#2807) 2026-06-05 15:25:05 +03:00
Aleksei Sviridkin 67b406fa1a fix(objectstorage-controller): propagate Bucket readiness to BucketClaim (#2792)
## What this PR does

The vendored COSI `objectstorage-controller` (built from upstream
`v0.2.2`) leaves dynamically-provisioned buckets without credentials.

In the dynamic-provisioning path, `provisionBucketClaimOperation` sets
`BucketClaim.status.bucketName` but hardcodes `bucketReady=false` and
never re-reads the `Bucket`. The controller does not watch `Bucket`
objects, so the readiness transition the sidecar performs after
`DriverCreateBucket` (`Bucket.status.bucketReady` `false`→`true`) is
never propagated back to the `BucketClaim`. Each periodic resync
re-enters the same branch (`Buckets().Create` returns `AlreadyExists`,
which is ignored) and re-sets `bucketReady=false`. As a result
`BucketClaim.status.bucketReady` stays `false` forever, the sidecar
refuses to grant the `BucketAccess` ("BucketAccess can't be granted to
Bucket not in Ready state"), and the credentials secret is never created
— even though the bucket exists in the backend and the `Bucket` CR
already reports `bucketReady: true`.

This adds a vendored patch that re-reads the live `Bucket` after create
and propagates its readiness onto the `BucketClaim`, so the existing
periodic resync converges the claim to ready and access can be granted.

The fix mirrors what the rewritten upstream controller already does on
`main` (`claim.Status.ReadyToUse = bucket.Status.ReadyToUse`). Upstream
`main` is a `v1alpha2` rewrite and the maintained `release-0.2` branch
still carries the bug, so this stays a vendored patch against `v0.2.2`
until the controller is bumped to a release that includes the fix.

Verified: reproduced the stuck-`false` claim with a unit test against
the upstream `v0.2.2` source and confirmed the patch resolves it,
existing controller tests pass, the patch applies cleanly to `v0.2.2`,
and the controller image builds.

### Release note

```release-note
fix(objectstorage-controller): propagate Bucket readiness to the BucketClaim so dynamically-provisioned buckets receive credentials instead of getting stuck without a granted BucketAccess
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Fixed an issue where bucket readiness status was incorrectly reported
when a bucket already existed. The status now accurately reflects the
actual state of the existing bucket instead of being unconditionally set
to not ready.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-05 11:10:19 +03:00
Aleksei Sviridkin 2363f2d9be fix(backups): carry dropdown option sources in CRD annotations, not schema
The four backups.cozystack.io CRDs injected the x-cozystack-options vendor
extension into their OpenAPI validation schema. apiextensions JSONSchemaProps
is a closed struct that only preserves x-kubernetes-* vendor extensions, so
helm-controller's server-side apply rejected the unknown key ("field not
declared in schema") and the CRDs never applied — the controller crash-looped
and backups were non-functional.

Move the field-to-source mapping the dashboard needs into CRD
metadata.annotations (preserved by the apiserver), emitted via
+kubebuilder:metadata:annotations markers, and drop the post-processing awk
injector. The dashboard reattaches the extension client-side.

A helm-unittest suite pins the contract: each dropdown field carries its source
in metadata.annotations and must not carry x-cozystack-options in the schema,
guarding against re-introducing the injector.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-04 12:45:14 +03:00
Andrei KvapilandClaude 4ae34581ed feat(flux-shard-operator): automatic shard count by default
--shard-count now accepts "auto" (the new default) or an explicit
positive integer. Auto mode drives the shard count from the tenant
HelmRelease count — K = clamp(ceil(H/100), 1, min(16, T)) — applied
with hysteresis anchored on the currently provisioned shards (scale up
eagerly past 120 HR/shard, scale down lazily under 60 HR/shard) so K
does not flap around sizing boundaries.

The per-shard target is 100 HelmReleases (down from the 150 in the
design doc) to leave headroom for existing tenants growing their
HelmRelease count without an immediate reshard.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-06-04 01:32:59 +02:00
Andrei KvapilandClaude d75fac0096 feat(flux-shard-operator): add system package
New cozy-fluxcd system package (Deployment with 2 replicas, webhook
cert via cert-manager, CREATE-only MutatingWebhookConfiguration with
failurePolicy Ignore scoped to HelmReleases born on the legacy tenants
bucket), registered in the platform system bundle and the root image
build.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-06-03 19:47:19 +02:00
Myasnikov DaniilandClaude Opus 4.8 b759446fa6 feat(monitoring): add tenant overview dashboard
Add a cross-tenant overview dashboard for the root/infra Grafana that
surfaces which tenants consume what resources, how many apps (Flux
HelmReleases) each runs, and per-tenant health.

Panels:
- Fleet summary stats (tenants, apps, pods, CPU/mem/storage)
- Tenant leaderboard table (CPU/mem usage vs requests, storage, pods,
  PVCs, app count, not-ready apps)
- Top-N bar gauges by CPU, memory, storage and app count
- CPU/memory usage trends per tenant
- Pod restart rate and not-ready HelmReleases per tenant

Registered in dashboards-infra.list so it deploys only to the
root/cozy-monitoring Grafana, keeping cross-tenant data off per-tenant
instances. Uses standard cadvisor, kube-state-metrics and
gotk_resource_info metrics.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-03 17:10:54 +05:00
Aleksei Sviridkin 3a5ff3d9c8 feat(ouroboros): bump to v0.8.0
v0.8.0 makes the TCP proxy log an explicit reason when its backend
readiness check fails, instead of staying silently NotReady. That
silent failure mode is exactly what makes a mispointed proxy backend
hard to diagnose, so the upgrade pays off the moment the target is
wrong. The bump also carries the v0.7.3 migration of the kubectl
sidecar image off Docker Hub to mirror.gcr.io, sidestepping anonymous
pull rate-limits on shared CI/runner IPs.

Chart manifest and controller image are re-pinned by digest in lockstep
with the version, and the image-pin test fixture is updated to match.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-03 14:06:23 +03:00
Aleksei Sviridkin f90411da7a fix(networking): point host ouroboros proxy at the root-tenant ingress
With publishing.proxyProtocol enabled the host emits a cozystack.ouroboros
Package via the no-override Package macro, so the proxy inherits the wrapper
chart default backend ingress-nginx-controller.cozy-ingress-nginx. That
layout describes a managed Kubernetes tenant cluster; on the host,
extra/ingress deploys ingress-nginx into the root tenant namespace as
root-ingress-controller in tenant-root. The composed backend FQDN never
resolves, so ouroboros-proxy never reaches Ready.

Emit the host ouroboros Package with a proxy.target override derived from
publishing.ingressName (namespace plus <short>-ingress-controller service
name), mirroring the trimPrefix transform extra/ingress applies. Add bundle
helm-unittest coverage for the default and a non-default ingress name, and
correct the misleading wrapper values.yaml comment.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-03 11:57:01 +03:00
Aleksei Sviridkin 588c9f2d62 feat(api): add Option resource and x-cozystack-options for dynamic form dropdowns (#2778)
## What this PR does

Adds a generic mechanism for **runtime-populated dropdowns** in
dashboard create/edit forms. Many fields are references to live cluster
resources (GPU device names, KubeVirt instancetypes/preferences, Multus
networks, VM images, bucket storage pools, storage classes, backup
classes/plans) but were free-text or static enums that drift from
reality.

- **`Option` resource** — `core.cozystack.io/v1alpha1`, namespaced,
read-only (list/get/watch). Items are computed on read by an in-process
provider registry using a privileged client, so tenants get curated
option lists without broad cluster reads. Providers: `gpu`,
`instancetype`, `instanceprofile`, `network`, `image`, `storagepool`,
`storageclass`, and backup-related `backupclass`, `plan`, `backup`,
`appkind`.
- **`x-cozystack-options` schema keyword** — app `values.yaml` declare
it via the new cozyvalues-gen `@x-<keyword>` directive (e.g. `##
@x-cozystack-options {source: instancetype}`). Schemas, Go types, and
the `-rd` ApplicationDefinitions are regenerated from source so the
extension is carried into the served OpenAPI. The dashboard renders
annotated fields as dropdowns sourced from the live cluster (UI widget
ships in a separate cozystack-ui PR).
- **RBAC** — the apiserver ServiceAccount gets the privileged reads;
tenants get read-only access to `options` in their namespace via the
aggregated tenant ClusterRoles (granted at `cozy:tenant:view:base`, so
every access level from view upward can populate dropdowns).
- Drops the redundant manual `instanceProfile` enum injection from
vm-instance's Makefile (now driven dynamically).

## Dependency on cozyvalues-gen

The `x-cozystack-options` annotations require the new vendor-extension
directive from cozystack/cozyvalues-gen#27. This PR bumps the pinned
cozyvalues-gen to **v1.6.0**. **A cozyvalues-gen v1.6.0 release must be
cut before CI can pass** — until then the pre-commit schema-generation
job cannot download the binary and will fail.

### Screenshots

N/A — no UI changes in this repository. The consuming dashboard widget
ships in a separate cozystack-ui PR.

### Release note
```release-note
feat(api): add a namespaced read-only Option resource (core.cozystack.io) and the x-cozystack-options schema keyword, enabling dashboard form fields (GPU, instancetype/profile, network, image, storage pool/class, backup class/plan) to render as dropdowns populated from live cluster resources
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Expanded dynamic dropdowns across many apps (storageClass,
storagePool, instanceType/instanceProfile, GPU, image, vmdisk, network,
backup/plan/backupclass, appkind) so chart fields can be populated from
live cluster resources.

* **Chores**
* Updated API/RBAC manifests to grant read access needed for option
providers.

* **Tests**
* Added provider and REST tests to validate option sources, scoping,
ordering, and failure handling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-03 01:52:12 +03:00
Aleksei Sviridkin ed20a4322a fix(objectstorage-controller): propagate Bucket readiness to BucketClaim
The vendored COSI controller (v0.2.2) sets only BucketClaim.status.bucketName
in the dynamic-provisioning path and hardcodes bucketReady=false, never
re-reading the Bucket afterwards. The controller does not watch Buckets, so the
readiness transition the sidecar performs (false->true) is never reflected on
the BucketClaim: bucketReady stays false forever and BucketAccess is never
granted, leaving provisioned buckets without credentials.

Re-read the live Bucket after create and propagate its readiness, so the
periodic resync converges the BucketClaim to ready. This matches the behavior
the rewritten upstream controller already has on its main branch.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-03 01:48:30 +03:00
Aleksei Sviridkin 26da871c00 feat(rbac): allow tenants to start/stop/restart their VMs (#2777)
<img width="2770" height="1970" alt="image"
src="https://github.com/user-attachments/assets/774c7c71-0e6a-496a-9898-8ad1ed848b0a"
/>


## Summary

Grants tenant users permission to call the KubeVirt VM power-control
subresources (`subresources.kubevirt.io` →
`virtualmachines/{name}/start|stop|restart`, verb `update`) by adding
them to the `cozy:tenant:use:base` ClusterRole, which aggregates into
`tenant-use`, `tenant-admin`, and `tenant-super-admin`.

Without this, the dashboard VM power buttons return 403 even for tenant
admins: `kubectl auth can-i update virtualmachines --subresource=start`
is `no` for every tenant role today, while
`virtualmachineinstances/console|vnc|portforward` are already granted at
the same `use` level — so power control belongs there too.

## Companion PR

UI that consumes this permission (VM power controls + VNC fixes):
cozystack/cozystack-ui#27

## Test

- `helm template` renders `cozy:tenant:use:base` with the new
`virtualmachines/start|stop|restart` rule.
- After apply, `kubectl auth can-i update virtualmachines
--subresource=start|stop|restart` flips from `no` to `yes` for the
`tenant-*-use/admin/super-admin` groups.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Virtual Machine Power Control: Users with tenant-level access can now
start, stop, and restart virtual machines, providing enhanced management
capabilities for VM operations.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-03 00:12:53 +03:00
Aleksei Sviridkin e59f93605a test(cozystack-basics): pin tenant view grant on core.cozystack.io options
The tenant view aggregation grants read on the core.cozystack.io/options
virtual resource so dashboard dropdowns can be populated. Add a helm-unittest
on the cozy:tenant:view:base ClusterRole so a silent RBAC regression on the
tenant side is caught, mirroring the apiserver-side rbac test.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-02 18:20:51 +03:00
Aleksei Sviridkin f0f1431546 feat(kubernetes): enable GPU passthrough out-of-the-box (NvLinkDisable + gpu=on) (#2780)
## What this PR does

Makes GPU passthrough work out-of-the-box for tenant Kubernetes clusters
(the `kubernetes` app), so operators no longer have to hand-fix it. Two
defaults:

1. **Tenant gpu-operator loads the nvidia driver with
`NVreg_NvLinkDisable=1`.**
2. **Nodes in GPU node-groups are labeled `gpu=on` automatically.**

### Root cause (validated empirically on a live cluster)

cozystack passes through individual SXM GPUs (e.g. H200) into tenant
worker VMs **without** the NVSwitches. Such a GPU's driver then waits
for an NVLink fabric that can never come up — there is no NVSwitch and
fabricmanager has nothing to manage — so `nvidia-smi` reports `Fabric
State: In Progress` indefinitely. As a result every CUDA call fails with
`system not yet initialized`, `nvidia-operator-validator`
(cuda-validation) crashloops, and no GPUs are advertised.

Loading the kernel module with `NVreg_NvLinkDisable=1` flips `Fabric
State` to `N/A`, CUDA initializes, and the validators pass. This is the
correct setting for per-GPU passthrough because there is no in-VM
NVLink/NVSwitch peering — the GPU is alone in the VM, so there is
nothing to disable that the workload needs.

The second issue: HAMi's `hami-device-plugin` DaemonSet has
`nodeSelector: gpu=on` and stays at `DESIRED=0` because GPU worker nodes
lack that label, so `nvidia.com/gpu` is never advertised. Labeling the
GPU nodes `gpu=on` lets the device plugin schedule and advertise the
GPUs.

### How it works

- **`gpu=on` label:** node-groups in
`packages/apps/kubernetes/templates/cluster.yaml` that declare `gpus`
get `--node-labels=gpu=on` appended to their kubelet args (in the
`KubeadmConfigTemplate` `joinConfiguration`). Non-GPU node-groups are
untouched.
- **`NVreg_NvLinkDisable=1`:** the upstream gpu-operator chart only
references an existing ConfigMap by name
(`driver.kernelModuleConfig.name`) and never creates it. The
gpu-operator wrapper chart (`packages/system/gpu-operator`) now renders
that ConfigMap (opt-in via `kernelModuleConfig.create`) so it deploys
into the tenant `cozy-gpu-operator` namespace alongside the operator.
The tenant gpu-operator HelmRelease defaults enable it and point
`driver.kernelModuleConfig.name` at it.

Both defaults remain overridable: a user setting
`addons.gpuOperator.valuesOverride` still wins via the existing
`mergeOverwrite(defaults, overrides)` in the HelmRelease template.

### Screenshots

Not applicable — no UI changes.

### Release note

```release-note
feat(kubernetes): enable GPU passthrough out-of-the-box for tenant clusters. GPU node-groups are now labeled `gpu=on` so HAMi's device plugin schedules and advertises `nvidia.com/gpu`, and the tenant gpu-operator loads the nvidia driver with `NVreg_NvLinkDisable=1` so single-SXM-GPU passthrough (no NVSwitch in the VM) no longer hangs on `Fabric State: In Progress` / `system not yet initialized`. Both defaults are overridable via `addons.gpuOperator.valuesOverride`.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* GPU-enabled node groups now get the expected GPU label during node
registration.
* GPU operator supports optional kernel-module parameter configuration
with a default parameter set.

* **Tests**
* Added tests verifying GPU node labeling is applied only for
GPU-enabled node groups.
* Added tests validating kernel-module config creation, naming, and
content behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-02 17:53:05 +03:00
Aleksei Sviridkin b30eb4602d fix(cozystack-api): drop unused objectstorage bucketclasses RBAC grant
The Option providers RBAC block granted read on
objectstorage.k8s.io/bucketclasses, but no provider in the registry lists
BucketClasses and no schema field references a bucketclass source — the
bucket app's storage pool field uses the storagepool provider, which reads
seaweedfs HelmReleases. Drop the unused grant and pin its absence in the
chart test.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-02 17:13:18 +03:00
Aleksei Sviridkin 4cbe5b62d1 fix(cozystack-api): grant the option apiserver read on plans and backups
The plan and backup option providers list namespaced plans and backups in
the backups.cozystack.io group, but the cozystack-api ClusterRole only
granted backupclasses there. The apiserver's privileged client runs as this
ServiceAccount, so both providers got forbidden — List silently dropped the
sources and Get returned 500, leaving the plan/backup dropdowns in the
backup forms empty.

Add plans and backups to the backups.cozystack.io rule and cover the grant
with a chart test.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-02 16:42:59 +03:00
Andrei KvapilandClaude 34e370e5c5 chore(apps): regenerate schemas and ApplicationDefinitions with x-cozystack-options
Regenerate values.schema.json and embedded openAPISchema in the -rd
ApplicationDefinitions so the x-cozystack-options dynamic-source
directives declared in each app values.yaml are carried into the
generated artifacts. Brings the -rd schemas in sync with the
hand-edited values.schema.json files (CI runs make generate +
git diff --exit-code).

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-06-02 16:42:58 +03:00
Andrei KvapilandClaude 49976c02cd fix(rbac): grant tenant view role read access to options
Options were granted on cozy:tenant:use:base, leaving view-only tenant
users (cozy:tenant:view) without read access — so their create/edit
forms could not populate dynamic dropdowns. Move the grant down to
cozy:tenant:view:base, which aggregates up into use/admin/super-admin,
covering every human tenant access level. The tenant ServiceAccount role
keeps its grant via cozy:tenant:base.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-06-02 16:35:23 +03:00