mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-09 05:25:14 +09:00
6a6dcd919e075601153f3cfca6420c4dcafe0b1d
1552
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 myasnikovdaniil
|
aeaedca4a4 |
ci: consolidated platform & e2e stabilization batch (#2948)
## What Consolidated landing branch for a set of interdependent platform/CI stabilization fixes. It began as the source PRs in the table below (in dependency order) and has since grown with review-driven fixes and a few production-behavior changes — the full current inventory is under **Scope** below. Landing them as one unit lets CI run against the real combined state instead of a fragile bottom-up merge train. ## Source PRs (dependency order) | # | Commit | Source PR | Addresses | |---|--------|-----------|-----------| | 1 | fix(capi): startupProbe on capi-controller-manager | #2946 | capi crashloop during cert provisioning | | 2 | test(e2e): bucket.bats port-forward + S3 client reliability | #2944 | flaky bucket test | | 3 | fix(seaweedfs): restore -lock BucketClass, s3 svc name, drop volumeSizeLimitMB | #2943 | missing bucket/access classes | | 4 | test(e2e): in-cluster Cilium endpoint-leak healer (install + apps) | #2874 | cilium "IP already in use" leak | | 5 | refactor(build): mode=max registry cache + main-only warmer | #2938 | build cache | | 6 | ci(build): isolate each PR build on its own ephemeral runner VM | #2939 | shared-buildkit contention | | 7 | fix(e2e): LINSTOR post-install waits on a single 15m deadline | #2928 | LINSTOR provisioning timeouts | | 8 | test(metallb): assert digest-pinned image form, not version literal | #2873 | brittle metallb assertion | ## Why consolidated These form a dependency DAG (verified from CI logs): e.g. 2943 needs 2946, 2938 needs 2874, 2928 needs 2938/2939, 2873 needs 2928. Tested bottom-up, each lower PR runs with none of the fixes above it and can't go green alone. This branch carries all of them, so CI runs against the real combined state. ## Verification status CI is green on the latest head (`5653c30`): full-suite E2E passes end-to-end and is reproducible (2 of 3 runs on this SHA green). The one red run was a LINSTOR tie-breaker / DRBD-metadata infra flake on the sandbox, unrelated to the diff — a different environmental subsystem failed each run (details in the comments below). The earlier 3-hour crust-gather snapshot hang is fixed in `5653c30`. ## Scope beyond the original 8 The branch has grown past the 8 source PRs above with review-driven fixes and a few production-behavior changes. For reviewer transparency, the full set: **Review fixes (@lexfrei review):** - **B1 (blocker)** — `8f41910` converts the SeaweedFS `-lock`/`-readonly` BucketClasses and the s3 service-name override into `patches/`, wired into `make update`, so a re-vendor no longer drops them. - **FU1** — `cc99230` grants the cilium leak-healer `delete` on pods (both delete remedies were RBAC-forbidden). - **FU3** — `014de3f` adds a unit test for the HelmRelease update conflict-retry path. - **FU4** — `6deddc6` makes the leak-healer refuse a disconnect for any non-terminal owner phase, not just `Running`. - **FU7** — `f3914d2` sets `persist-credentials: false` on the build-cache checkout (SHA-pinning is handled repo-wide in #2849). **Production-behavior changes (not in the table above):** - VPA `updateMode: Auto → Initial` for etcd (`6e9ff90`) and monitoring (`0997105`), plus `vmselect`/`vmstorage` `minAllowed` floors — stops install-time eviction churn. - cozystack-api: `retry.RetryOnConflict` on the Application→HelmRelease update path (`728f676`). **E2E robustness follow-ups:** per-test crust-gather snapshots (`50e5f94`), EtcdBackupSchedule wait (`763b85e`), tenant-node single-deadline wait (`9e43d45`), harbor BucketClaim 10m budget (`d7f244f`), tenant API via LoadBalancer (`9beda4e`), crust-gather pin (`95f42aa`). **Not addressed (by decision):** FU2 — the s3 Service keeps `seaweedfs.name`-based naming via patch (rendered output unchanged; cozystack renders `fullname == seaweedfs`, so it matches the `componentName`/`fullname` siblings in practice). FU6 — fork PR build push is handled by the existing mirror-to-same-repo-branch workflow. ## Relates to #2946 #2944 #2943 #2874 #2938 #2939 #2928 #2873 — if this lands, those can be closed; otherwise they remain the granular per-PR review path. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added CI build cache warming for main branch builds. * Added a best-effort in-cluster Cilium endpoint self-heal watchdog for e2e installs. * Extended SeaweedFS COSI with object-lock support plus separate readonly access. * **Improvements** * Reduced e2e install flakiness with a shared readiness deadline, safer waits, and improved cleanup/diagnostics capture. * Prevented install-time churn by switching VPA update mode to **Initial**. * Added container startup probes and updated SeaweedFS volume sizing. * **Tests** * Improved e2e robustness (etcd backup schedule waiting, digest-pinned image checks, and corrected S3 port-forwarding). <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
e8e750ccb0 |
feat(platform): operator-provided wildcard certificate for platform and tenant ingress (#2819)
## What this PR does Adds an operator-facing way to serve platform services and the root tenant's ingress under a pre-existing wildcard TLS certificate instead of minting per-host ACME certificates. Implements the "native references" design decided on #2812. A single value, `publishing.certificates.wildcardSecretName`, names a TLS Secret the operator has already created in the publishing namespace (`tenant-root` by default). Only the name travels through the `cozystack-values` channel — the certificate and private key are never broadcast, unlike the public `kube-root-ca` that already rides it. When set: - **ingress-nginx path** (`gateway.enabled=false`): the publishing tenant's ingress controller serves the Secret as its `--default-ssl-certificate`, and the platform service Ingresses drop their cert-manager cluster-issuer annotation and `tls` block. No per-host ACME. - **Gateway API path** (`gateway.enabled=true`): a new `TenantGateway` cert mode, `existingSecret`, references the Secret in the wildcard/apex listener `certificateRefs` and mints no Issuer or Certificate. Switching into this mode garbage-collects any ACME machinery left from `http01`/`dns01`. Scope is the root tenant — everything is same-namespace, so no RBAC or NetworkPolicy changes. Extending wildcard mode to child tenants needs the Secret replicated into each tenant namespace and is left as a follow-up. Closes #2812. Part of #2811. ### Screenshots N/A — no UI changes. ### Release note ```release-note feat(platform): operators can serve platform services and the root tenant's ingress under a pre-existing wildcard TLS certificate via `publishing.certificates.wildcardSecretName`, instead of minting per-host ACME certificates ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added an "existingSecret" TLS certificate mode and a wildcardSecretRef option for TenantGateway and platform values to use a pre-existing wildcard TLS Secret. * **Platform / Templates** * Introduced operator "wildcard secret" mode across Gateway and many Helm charts (Ingress, apps) to skip per-host ACME provisioning and optionally wire a default SSL cert. * **Tests** * Added extensive chart and controller tests covering existingSecret behavior and cert-mode transitions. * **Documentation** * Updated gateway docs describing the new wildcard secret mode and requirements. * **Chores** * Added Helm unit test makefile targets to several packages. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei KvapilandClaude
|
8f41910e87 |
fix(seaweedfs): vendor -lock/-readonly BucketClass + s3 service name as patches
The restored COSI -lock/-readonly BucketClass/BucketAccessClass and the s3 Service name override lived only as raw edits in the vendored chart tree, so the next `make update` (rm -rf charts && curl | tar && patch) would silently drop them again — the same way the 4.31 bump did. Capture both as patch files under patches/ and wire them into the Makefile update target, matching the existing s3-tls-main-port / cosi-provisioner-sa-name customizations. No rendered output changes; this only makes the customizations survive re-vendor. Addresses the B1 review blocker on #2948. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
0997105b05 |
fix(monitoring): set VPA updateMode to Initial to stop install-time eviction churn
In Auto mode the VPA Updater evicts running pods to apply new recommendations. During install that eviction churn recreates VictoriaMetrics pods faster than Cilium releases their endpoint IPs, tripping the cluster-pool IPAM "IP already in use" (putEndpointIdInvalid) leak that wedges a pod and stalls the install. Initial sets requests at pod creation without ever evicting, removing the trigger (the cilium-leak healer remains as defense for non-VPA churn paths). Also raise vmstorage/vmselect minAllowed floors to 200m/512Mi: under Initial a pod is sized once at creation and never re-sized at runtime, so the floor is the safety net against the heavy query/storage tiers being left under-provisioned. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
df9dc74f09 |
test(metallb): assert digest-pinned image form, not exact version literal (#2873)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
d3958c5bcb |
refactor(build): mode=max registry cache with a main-only warmer (#2938)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
725b19e446 |
fix(seaweedfs): restore -lock BucketClass, s3 service name, and drop volumeSizeLimitMB (#2943)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
34c4f0a152 |
fix(capi): add startupProbe to capi-controller-manager to survive cert provisioning delay (#2946)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
97cf5857a3 |
fix(prometheus-operator-crds): clean chart dir before helm pull
Add `rm -rf charts/prometheus-operator-crds` before `helm pull` in the update target so stale files from previous chart versions don't linger across updates. This matches the convention used by cilium, external-secrets-operator, external-dns, and cert-manager Makefiles. Address review feedback from gemini-code-assist on packages/system/prometheus-operator-crds/Makefile:9. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
dbbaca16eb |
fix(prometheus-operator-crds): ship full upstream CRD bundle
The chart previously stripped the bundle to 4 CRDs (ServiceMonitor, PodMonitor, PrometheusRule, Probe) — the set VictoriaMetrics operator needs for service discovery. This was correct when the chart was vendored inside system/victoria-metrics-operator (commit |
||
|
Andrei KvapilandClaude
|
484d73bf78 |
fix(seaweedfs): restore S3 TLS and COSI provisioner SA after 4.31 bump
The 4.31 chart bump (#2834) broke E2E for every PR: tenant-root seaweedfs-system never becomes ready. 1. S3 served plaintext. Until 4.05 the -cert.file/-key.file args were rendered whenever enableSecurity was set, so weed s3 served TLS on the main port (8333); 4.31 nested them under 'if s3.httpsPort' and we ship httpsPort=0, so s3 spoke HTTP while the HTTPS probes, the ingress backend-protocol and the COSI sidecar endpoint all expected HTTPS ('server gave HTTP response to HTTPS client'). Restore the 4.05 behaviour via patch. 2. COSI provisioner ServiceAccount mismatch. cosi-service-account.yaml and the ClusterRoleBinding name the SA <serviceAccountName>-objectstorage-provisioner, but the Deployment referenced componentName (<release>-objectstorage-provisioner). When serviceAccountName != release name (tenant-root-seaweedfs vs seaweedfs-system) the Deployment could not create pods ('serviceaccount not found'). Align the Deployment with the SA + binding. Both fixes are vendored as patches re-applied by 'make update', and covered by helm-unittest. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Aleksei Sviridkin
|
13e310fe71 | Merge branch 'main' into chore/seaweedfs-4.31 | ||
|
Aleksei Sviridkin
|
9e3ae7447b |
fix(ingress): keep tls section in operator wildcard mode
Dropping the whole tls block in wildcard mode disabled the forced HTTP-to-HTTPS redirect: ingress-nginx only issues the 308 redirect when TLS is enabled for the Ingress, while a host without a tls section is still served on 443 with the default certificate but stays reachable over plain HTTP. Keep the tls section with the host list and omit only the per-host secretName, so ingress-nginx falls back to the operator wildcard configured as the default SSL certificate and the redirect stays enforced, matching the non-wildcard behaviour. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
b1f5d9579c |
feat(ingress): serve operator wildcard via default SSL certificate
When publishing.certificates.wildcardSecretName is set, the root tenant's ingress-nginx controller now serves the operator-supplied wildcard Secret as its --default-ssl-certificate, and the platform service Ingresses stop requesting their own per-host ACME certificates. The flag is set only on the publishing controller (Release.Namespace == expose-ingress) because ingress-nginx loads the Secret from its own namespace; child-tenant controllers cannot reach a Secret in another namespace, so they keep the existing ACME path. Each service Ingress (dashboard, keycloak, grafana, alerta, bucket, linstor-gui, harbor, seaweedfs S3, bootbox) drops its cert-manager cluster-issuer annotation, the http01 challenge-class annotation, and its tls block in wildcard mode; nginx then falls back to the default certificate for those hosts. Other annotations and routing are unchanged. Each file keeps its existing render guard intact, and the annotations block collapses cleanly where the cert annotations were its only content. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
4262658f0f |
feat(tenantgateway): add existingSecret cert mode for operator wildcard
Operators who already hold a wildcard TLS certificate (purchased, or issued by a corporate CA) had no way to make a tenant Gateway serve under it: the only cert modes were http01 (per-listener ACME) and dns01 (controller-minted wildcard via ACME). Add a third mode, existingSecret, that references a pre-existing TLS Secret in the TenantGateway's own namespace via spec.wildcardSecretRef. In this mode the controller mints no ACME Issuer and no Certificate; the wildcard, apex, and per-child-apex HTTPS listeners reference the supplied Secret directly. A missing wildcardSecretRef fails fast with Ready=False rather than producing a Gateway pointing at a nonexistent Secret. Switching into existingSecret from http01/dns01 reclaims the now-unused Issuer and Certificates so no ACME machinery is left behind. The non-cert listeners (http->https redirect, TLS passthrough) are unaffected and continue to render. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
1d747b3d8d |
feat(platform): auto-wire KubeVirt permittedHostDevices and HostDevices feature gate when gpu-operator is enabled (#2768)
Closes #2765. ## What this PR does Makes the passthrough and vGPU variants of `cozystack.gpu-operator` produce a KubeVirt CR that is immediately usable, without an out-of-band `kubectl patch`. Today the gpu-operator stack reports green, `nvidia.com/<model>` shows up under node `.status.allocatable`, but every `VirtualMachine` referencing the resource via `domain.devices.hostDevices` is rejected by the admission webhook because `HostDevices` is not in `developerConfiguration.featureGates` and the resource is not listed under `permittedHostDevices`. That manual reconciliation is now driven by the platform. The change has three parts. `packages/system/kubevirt` now parameterizes the KubeVirt CR template. `developerConfiguration.featureGates` keeps its existing static defaults and appends `.Values.extraFeatureGates`; `spec.configuration.permittedHostDevices` renders verbatim from `.Values.permittedHostDevices` when set; `spec.configuration.mediatedDevicesConfiguration` does the same. Defaults stay empty so the rendered CR is byte-identical to the previous template when no overrides are supplied. The chart's `make update` target re-applies the four sed transforms after fetching the upstream CR, so refreshing the KubeVirt release version stays a single command. `packages/core/platform` adds two defaults tables under `files/`. `gpu-passthrough-defaults.yaml` covers Hopper (H100/H200), Ada Lovelace (L4/L40/L40S), Ampere (A100 PCIe/SXM, A40, A30, A10), Turing (T4) and Volta (V100/V100S); PCI vendor:device pairs are stable, `resourceName` slugs follow the `<arch>_<model>_<form>_<mem>` convention `nvidia-sandbox-device-plugin` generates as of v25.x, and `externalResourceProvider: true` is set everywhere because those resources are advertised by the sandbox plugin rather than KubeVirt's in-tree device plugin. `gpu-vgpu-defaults.yaml` covers the Pascal–Ampere mdev profile range (A100-40C/80C, A40-24Q/48Q, A30-24C, A10-24Q, V100D-32C, T4-16Q) — the same family span the upstream `vgpu-device-manager` walks `/sys/class/mdev_bus/` for. Ada Lovelace and Blackwell SR-IOV vGPU is intentionally out of scope for the default table and is expected to be advertised via the user-override hook described below. `templates/bundles/iaas.yaml` now selects between the two tables via a new `bundles.iaas.gpuOperatorVariant` knob (`default` for passthrough, `vgpu` for mediated; default is passthrough). The same variant flows through to the gpu-operator Package CR, replacing the previous always-default call. When the package is enabled, the platform injects `HostDevices` into `extraFeatureGates` plus the chosen defaults into `permittedHostDevices` for the kubevirt component's values. Operators extend or replace via top-level platform values: ```yaml gpu: replaceDefaults: false # append (default) vs. wipe-and-replace permittedHostDevices: pciHostDevices: [] # cluster-specific PCI passthrough entries mediatedDevices: [] # cluster-specific vGPU entries mediatedDevicesConfiguration: {} # extra mediated-device types / per-node mappings ``` `docs/gpu-vgpu.md` is updated to drop the manual `kubectl patch` step and describe the auto-wiring contract, the per-variant defaults, the escape hatches, and the sandbox-device-plugin `resourceName` slug convention so an operator whose node publishes a different name knows where to override. ## Test coverage `helm-unittest` covers eight branches in `packages/core/platform/tests/bundles_gpu_kubevirt_wiring_test.yaml`: gpu-operator disabled emits no extras; gpu-operator enabled at default variant emits the HostDevices gate plus `pciHostDevices`; vgpu variant switches to `mediatedDevices` plus `mediatedDevicesConfiguration`; user overrides append by default; `replaceDefaults: true` wipes the platform table; an unknown variant fails fast with a descriptive error. `packages/system/kubevirt/tests/kubevirt_cr_test.yaml` pins the empty-values render and the three populated shapes at the CR template level. `make helm-unit-tests` passes locally on the full repo. ### Release note ```release-note feat(platform): the KubeVirt CR is now auto-configured for GPU host-device passthrough (HostDevices feature gate plus permittedHostDevices populated from NVIDIA defaults) whenever cozystack.gpu-operator is in bundles.enabledPackages. Operators no longer need a manual kubectl patch to make VMs schedule against a GPU. The variant is selected via bundles.iaas.gpuOperatorVariant (default: passthrough; vgpu: mediated). Extend or replace the NVIDIA defaults via .gpu.permittedHostDevices and .gpu.replaceDefaults. **Upgrade note:** the bundle now OWNS spec.configuration.permittedHostDevices on the KubeVirt CR. If you previously hand-edited it (kubectl edit kubevirt), the first reconcile after upgrade overwrites your manual entries with the rendered NVIDIA default table. Before upgrading, move any custom entries into .gpu.permittedHostDevices (set .gpu.replaceDefaults: true to keep only your list) and verify each resourceName against what your nodes advertise (kubectl describe node <node> | grep nvidia.com/) — the default table now carries the exact slug the sandbox device plugin generates, which upper-cases the card's PCI-IDs name and keeps the Tesla/GL tokens (e.g. a Tesla T4 is nvidia.com/TU104GL_TESLA_T4), but a different plugin build can publish a different string; a mismatch makes the admission webhook reject GPU VMs on their next restart or migration. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Bundle-managed GPU Operator with selectable passthrough vs vGPU variants; platform now injects GPU settings into KubeVirt and forwards vGPU manager overrides when enabled. * Default platform mappings for PCI passthrough and vGPU mediated-device resources added. * **Documentation** * Expanded GPU/vGPU deployment guide with bundle-managed vs manual guidance, wiring/precedence rules, resource-name conventions, and verification steps. * **Tests** * New and expanded tests for bundle→KubeVirt wiring, variant behaviors, default/override merging, and idempotent template parameterization. * **Chores** * Added Helm test target and hardened KubeVirt template update/idempotency checks. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
b6108039b7 |
feat(gpu-operator): add container variant for preinstalled host driver (#2766)
## What this PR does Add a third architectural variant to `cozystack.gpu-operator` for hosts where the NVIDIA driver and `nvidia-container-toolkit` are already installed by the distro package manager (the typical Ubuntu/Debian GPU node shape: apt-installed driver + apt-installed CT). The existing variants do not produce a working setup in that shape. `default` (passthrough) runs `vfioManager`, which unbinds the host driver to bind `vfio-pci` — with the host driver active, `k8s-driver-manager` early-exits and the bind never happens, so the GPU never reaches a pod. `vgpu` requires the proprietary NVIDIA vGPU host driver and a license server; the user-installed open or proprietary driver does not satisfy it. The new `container` variant maps onto upstream gpu-operator's third mode (`sandboxWorkloads.enabled=false` + `devicePlugin.enabled=true`) with every component that would conflict with the host install switched off: - `driver.enabled=false` — use the host driver. - `toolkit.enabled=false` — host already has `nvidia-container-toolkit`. Enabling the operator's toolkit DaemonSet would overwrite the `/etc/containerd/config.toml` the host package configured (via `nvidia-ctk runtime configure`), breaking the host runtime wiring. Matches NVIDIA's documented guidance and the existing `examples/values-native-talos.yaml` reference, which pins the same flag for the same reason. - `devicePlugin.enabled=true` — publish `nvidia.com/gpu` to the kubelet. - `vfioManager.enabled=false` — do not unbind the host driver. Upstream renders this `true` unconditionally; it only stays at zero pods because no node carries a `vm-passthrough` sandbox assignment when `sandboxWorkloads` is off — pinned off so the rendered policy matches intent. - `cdi.enabled=false` — upstream defaults this `true`, but CDI spec generation is serviced by the operator-managed toolkit DaemonSet this variant disables. An apt host doesn't auto-generate CDI specs (`nvidia-ctk cdi generate` is a manual step) and uses the classic `nvidia-container-runtime` hook, so leaving CDI on would point the device plugin at CDI injection with no specs on the host and silently break allocation. - `sandboxWorkloads.enabled=false` — regular pod workloads, not VMs. With this off, upstream `defaultWorkload` is already `container`, so no override is needed. Host prerequisite: because the operator's toolkit component is disabled, the node's containerd must already have the `nvidia` runtime registered (`nvidia-ctk runtime configure --runtime=containerd`); `apt install nvidia-container-toolkit` installs binaries only and does not wire containerd. With `driver.enabled=false` the operator uses the pre-installed host driver at its standard location, so a stock apt install needs no `hostPaths.driverInstallDir` override (Talos lands the driver under a non-standard prefix — see `examples/values-native-talos.yaml`). The bundle wiring (`packages/core/platform/templates/bundles/iaas.yaml`) already enables `cozystack.gpu-operator` as an optional package — no bundle changes required. Users opt into this variant by setting `variant: container` in their `Package` CR. Verified locally with `helm template` against `values.yaml` + `values-container.yaml`: ClusterPolicy renders with `driver.enabled=false`, `toolkit.enabled=false`, `devicePlugin.enabled=true`, `vfioManager.enabled=false`, `cdi.enabled=false`, `sandboxWorkloads.enabled=false`. Platform `helm unittest` and the `gpu-operator` variant bats suite are green. Not yet smoke-tested on a real Ubuntu GPU node — the config matches the documented apt-host wiring (legacy `nvidia-container-runtime` hook, CDI off), but an end-to-end CUDA-pod run on hardware is still owed. Closes #2764 ### Release note ```release-note feat(gpu-operator): add `container` variant for hosts with a pre-installed NVIDIA driver and nvidia-container-toolkit, exposing containerized GPU workloads via the standard NVIDIA device plugin without VFIO passthrough or vGPU. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a "container" GPU operator variant to support containerized GPU workloads on hosts with the NVIDIA driver and nvidia-container-toolkit. * **Documentation** * Updated variants documentation and OS support table to include the container variant and per-host requirements (including Ubuntu and Talos notes). * Clarified Talos native pod example wording. * **Tests** * Added regression tests to verify container-variant configuration and component enablement/disablement. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
28dfb40579 |
fix(dashboard): grant tenant dashboard read on cozy-public PVCs (#2843)
## What this PR does The vm-disk **source image** dropdown in the console lists `PersistentVolumeClaims` in `cozy-public` (the `vm-default-images-*` golden images) to offer them when creating a VM disk. That list is issued by the browser as the **tenant identity** (e.g. `system:serviceaccount:tenant-root:tenant-root`), but the `cozy:tenant:dashboard` Role in `cozy-public` only granted read on Flux `helmrepositories`/`helmcharts`. The list returned `403 Forbidden` and the dropdown stayed empty even when images had been enabled (e.g. via `bundles.enabledPackages`). This is identity-dependent, which is why it can look intermittent: a cluster-admin login bypasses tenant RBAC and sees the images, while any real tenant — including `tenant-root-super-admin` — hits the 403. This PR adds `get`/`list`/`watch` on `persistentvolumeclaims` to the Role. `cozy-public` is a shared public namespace, so read on its PVCs is safe to expose to every tenant dashboard subject (the Role is already bound to the tenant groups and the tenant ServiceAccount). A helm-unittest regression pins the grant. Verified with `kubectl auth can-i list pvc -n cozy-public`: - `--as=system:serviceaccount:tenant-root:tenant-root` → no (before this PR) - `--as-group=tenant-root-super-admin` → no (before this PR) - cluster-admin → yes (why it "worked" for admins) No console code changed — this is an RBAC-only fix. ### Release note ```release-note fix(dashboard): allow tenant dashboards to list VM images in cozy-public, fixing the empty source-image dropdown when creating a vm-disk ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Dashboard access permissions have been expanded to include read-level access to persistent volume claims, enabling users to manage and select images through the dashboard more effectively. * **Tests** * Added test suite to verify dashboard role permissions are correctly configured and functional. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
5982018629 |
feat(monitoring): add tenant overview dashboard (#2809)
## What Adds a cross-tenant **Tenant Overview** dashboard to the root/infra Grafana. ## Why Platform admins currently have no single view of which tenants consume what. This shows resource utilization, app counts, and health per tenant at a glance. ## Panels - **Fleet summary** — tenants, apps, running pods, CPU/memory/storage totals - **Tenant leaderboard** — per-tenant table: CPU & memory (usage vs requests), storage, pods, PVCs, app count, not-ready apps - **Top consumers** — Top-N bar gauges by CPU, memory, storage, app count - **Trends** — per-tenant CPU & memory usage over time - **Health** — pod restart rate and not-ready HelmReleases per tenant ## Details - Registered in `dashboards-infra.list` only, so it deploys to the root/`cozy-monitoring` Grafana and never exposes cross-tenant data on per-tenant instances (same pattern as `gpu/gpu-tenants`). - Apps are counted from `gotk_resource_info` HelmReleases, excluding the paired `*-system`/`cozy-*` charts. - Uses only standard cadvisor / kube-state-metrics / Flux metrics. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## New Features * Added a Tenant Overview dashboard for comprehensive tenant infrastructure monitoring, including: * Fleet summary with tenant counts, app deployments, and aggregate resource usage * Tenant performance leaderboard with CPU, memory, and storage metrics * Top consumers visualization by resource type and app count * Historical usage trends and pod health metrics <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
ed367f30dd |
test(packages/system/seaweedfs): cover s3.trafficDistribution version gating
Pin the native s3.trafficDistribution behaviour that replaced the dropped s3-traffic-distribution.patch: PreferClose is kept on Kubernetes 1.33, auto-converts to PreferSameZone on 1.35, and the field is omitted below the 1.31 gate. Adds the helm-unittest harness (test target + suite) the package previously lacked. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
068af203ac |
chore(packages/system/seaweedfs): bump seaweedfs to 4.31
Re-vendor the SeaweedFS Helm chart from 4.05 to 4.31. Improvements that land with the binary: the PostgreSQL filer store moves to pgx v5 with PgBouncer support and an ON CONFLICT upsert default, HTTPS certificates hot-reload without a pod restart on cert-manager rotation, and more S3/volume Prometheus metrics are exported. The S3 service now exposes a metrics port by default. Adaptations: - Drop s3-traffic-distribution.patch: native s3.trafficDistribution support landed upstream in chart 4.12. The native field is gated to Kubernetes >=1.31 and auto-converts to PreferSameZone on >=1.35, which the hardcoded patch did not do. Set s3.trafficDistribution: PreferClose in values instead. - Migrate global.* to global.seaweedfs.* in values: chart 4.18 namespaced the app-specific globals to avoid polluting the shared global namespace when the chart is used as a subchart. Covers enableSecurity, serviceAccountName, extraEnvironmentVars and monitoring. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
8df4b0b0f1 |
feat(ingress): add proxyProtocol knob to ingress-nginx
Expose a proxyProtocol value on the Ingress application that turns on ingress-nginx use-proxy-protocol with real-ip-header=proxy_protocol and enable-real-ip. The host control plane also enables it through the _cluster.proxy-protocol switch, applied only to the external-facing ingress (the namespace named by expose-ingress). use-forwarded-headers / compute-full-forwarded-for are deliberately not set: without a paired proxy-real-ip-cidr they let clients spoof X-Forwarded-For, and the PROXY-protocol frame is already the source of truth for the client address. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Myasnikov Daniil
|
9868c43e53 |
fix(build): read --cache-from from ghcr :latest so PR builds warm-start
PR #2711 switched PR builds to push only :pr-<N>-<sha> tags (PUBLISH_FLOATING=0) to stop concurrent-push 409 conflicts, but left --cache-from pointing at $(REGISTRY)/<img>:latest. PR builds set REGISTRY to the per-CI registry (iad.ocir.io) and never publish :latest there, so every PR build's registry cache lookup 404s and rebuilds cold. Add a CACHE_REGISTRY knob (default ghcr.io/cozystack/cozystack, where release builds do publish :latest) and point every --cache-from at it. Push targets ($(REGISTRY)) and the unique per-PR tags are unchanged, so the #2711 anti-conflict behaviour is preserved while PR builds regain a warm cache from the last release. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
285f42aaeb |
fix(kubevirt): reinsert each GPU CR patch independently in make update
The update target gated the insertion of both the permittedHostDevices and mediatedDevicesConfiguration directives behind a single `grep permittedHostDevices` guard. A hand-merge from a newer upstream KubeVirt CR that dropped only mediatedDevicesConfiguration left permittedHostDevices in place, so the guard skipped the combined sed and mediatedDevicesConfiguration was never reinserted — then the sanity-check tail aborted the target with a non-zero exit instead of self-healing. Split the two directives into independent guarded inserts so each self-heals on its own. Reword the sanity-check failures to name the directive that failed to land and the anchor its insert keys off, instead of implying the anchor itself is missing. Add a shell regression test that strips one directive at a time and asserts `make update` restores it without duplicating the other; the update target gains a CR_FILE override so the test can drive it against a throwaway copy. Wired into the package test target and guarded to skip on BSD sed. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
8c507fe4c5 |
feat(kubevirt): parameterize CR for host-device GPU configuration
The KubeVirt CR's developerConfiguration.featureGates list, the
permittedHostDevices block, and mediatedDevicesConfiguration are now
driven by chart values:
- extraFeatureGates is appended after the static default set so
HostDevices (or any other future gate) can be enabled without
forking the template.
- permittedHostDevices is rendered verbatim when set, allowing the
pciHostDevices (passthrough) and mediatedDevices (vGPU) shapes the
KubeVirt admission webhook requires for VMs that reference a GPU
by resourceName.
- mediatedDevicesConfiguration is rendered verbatim when set,
carrying mediatedDeviceTypes / nodeMediatedDeviceTypes for the
vGPU profile auto-creation loop.
Defaults stay empty so the rendered CR is byte-identical to the
previous template when no overrides are supplied.
The Makefile's update target re-applies the four sed transforms
in-place on templates/kubevirt-cr.yaml — it does not download anything
from upstream KubeVirt because the published CR is a near-empty stub
and would clobber the hand-curated cozystack defaults. To refresh from
a newer upstream release the operator hand-merges the new fields into
the in-tree template, then runs `make update` to re-assert the four
cozystack parameterization patches; grep sanity-checks at the end of
the target fail loudly if any anchor (developerConfiguration,
- VMExport, evictionStrategy) gets dropped during the hand-merge.
Companion helm-unittest coverage pins the default render and the
three populated shapes.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
Aleksei Sviridkin
|
9352d3ee0e |
feat(gpu-operator): add container variant for preinstalled host driver
The two existing variants assume the host has neither the NVIDIA driver
nor the container toolkit: default unbinds the host driver to bind
vfio-pci for KubeVirt VMs, vgpu requires the proprietary vGPU host
driver and a license server. Neither path produces a working setup on a
node that already ships the driver and nvidia-container-toolkit via the
distro package manager — the most common shape of a Linux GPU node.
The new container variant surfaces the third architectural mode that
upstream NVIDIA gpu-operator already supports: sandboxWorkloads
disabled, devicePlugin enabled, with every component that would
conflict with the host install switched off.
- driver.enabled=false use the host driver
- toolkit.enabled=false host already has nvidia-container-toolkit;
enabling overwrites the containerd
config.toml the host configured via
nvidia-ctk runtime configure
- devicePlugin.enabled=true publish nvidia.com/gpu to the kubelet
- vfioManager.enabled=false do not unbind the host driver
- cdi.enabled=false CDI spec generation is serviced by the
disabled toolkit; an apt host uses the
legacy nvidia-container-runtime hook, so
pin off to match the host runtime wiring
- sandboxWorkloads.enabled=false regular pod workloads, not VMs
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
Aleksei Sviridkin
|
80fbfa1ed8 |
fix(networking): point host ouroboros proxy at the root-tenant ingress (#2800)
## What this PR does With `publishing.proxyProtocol: true`, the host emits a `cozystack.ouroboros` Package using the no-override Package macro, so the proxy inherits the wrapper chart default backend `ingress-nginx-controller.cozy-ingress-nginx`. That layout describes a managed Kubernetes tenant cluster, not the host: on the host, `extra/ingress` deploys ingress-nginx into the root tenant namespace as `root-ingress-controller` in `tenant-root`. The composed backend FQDN never resolves, so `ouroboros-proxy` never reaches Ready. This emits the host ouroboros Package with a `proxy.target` override derived from `publishing.ingressName` — namespace plus `<short>-ingress-controller` service name — mirroring the `trimPrefix "tenant-"` transform `extra/ingress` applies to its release namespace. The derivation reuses the existing platform invariant that `publishing.ingressName` is the namespace where the host ingress controller runs. Adds bundle helm-unittest coverage for the default ingress name and a non-default one, and corrects the misleading wrapper `values.yaml` comment that claimed the host uses `cozy-ingress-nginx`. Scope: this fixes the ouroboros readiness defect only. The separate defect where the injected nginx PROXY-protocol config keys are not consumed by the host ingress is tracked independently and not touched here. Closes #2797 ### Release note ```release-note fix(networking): point the host ouroboros PROXY-protocol proxy at the root-tenant ingress so it becomes Ready when publishing.proxyProtocol is enabled ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Enhanced proxy protocol configuration to automatically route through the appropriate ingress-nginx service based on your ingress name settings. * **Tests** * Added test coverage for proxy protocol bundle wiring and ingress service routing. * **Documentation** * Clarified proxy target configuration behavior and fallback mechanisms. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
bb322138ec |
fix(dashboard): grant tenant dashboard read on cozy-public PVCs
The vm-disk source image dropdown in the console lists PersistentVolumeClaims in cozy-public (the vm-default-images-* golden images). The browser issues that list as the tenant identity (e.g. system:serviceaccount:tenant-root:tenant-root), but the cozy:tenant:dashboard Role only granted read on Flux helmrepositories and helmcharts. The list returned 403 and the dropdown stayed empty even when the images existed. Add get/list/watch on persistentvolumeclaims to the Role. cozy-public is a shared public namespace, so read on its PVCs is safe to expose to every tenant dashboard subject. A helm-unittest regression pins the grant. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
bc03142c08 | feat(ouroboros): bump to v0.8.0 (#2807) | ||
|
Aleksei Sviridkin
|
67b406fa1a |
fix(objectstorage-controller): propagate Bucket readiness to BucketClaim (#2792)
## What this PR does
The vendored COSI `objectstorage-controller` (built from upstream
`v0.2.2`) leaves dynamically-provisioned buckets without credentials.
In the dynamic-provisioning path, `provisionBucketClaimOperation` sets
`BucketClaim.status.bucketName` but hardcodes `bucketReady=false` and
never re-reads the `Bucket`. The controller does not watch `Bucket`
objects, so the readiness transition the sidecar performs after
`DriverCreateBucket` (`Bucket.status.bucketReady` `false`→`true`) is
never propagated back to the `BucketClaim`. Each periodic resync
re-enters the same branch (`Buckets().Create` returns `AlreadyExists`,
which is ignored) and re-sets `bucketReady=false`. As a result
`BucketClaim.status.bucketReady` stays `false` forever, the sidecar
refuses to grant the `BucketAccess` ("BucketAccess can't be granted to
Bucket not in Ready state"), and the credentials secret is never created
— even though the bucket exists in the backend and the `Bucket` CR
already reports `bucketReady: true`.
This adds a vendored patch that re-reads the live `Bucket` after create
and propagates its readiness onto the `BucketClaim`, so the existing
periodic resync converges the claim to ready and access can be granted.
The fix mirrors what the rewritten upstream controller already does on
`main` (`claim.Status.ReadyToUse = bucket.Status.ReadyToUse`). Upstream
`main` is a `v1alpha2` rewrite and the maintained `release-0.2` branch
still carries the bug, so this stays a vendored patch against `v0.2.2`
until the controller is bumped to a release that includes the fix.
Verified: reproduced the stuck-`false` claim with a unit test against
the upstream `v0.2.2` source and confirmed the patch resolves it,
existing controller tests pass, the patch applies cleanly to `v0.2.2`,
and the controller image builds.
### Release note
```release-note
fix(objectstorage-controller): propagate Bucket readiness to the BucketClaim so dynamically-provisioned buckets receive credentials instead of getting stuck without a granted BucketAccess
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Fixed an issue where bucket readiness status was incorrectly reported
when a bucket already existed. The status now accurately reflects the
actual state of the existing bucket instead of being unconditionally set
to not ready.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Aleksei Sviridkin
|
2363f2d9be |
fix(backups): carry dropdown option sources in CRD annotations, not schema
The four backups.cozystack.io CRDs injected the x-cozystack-options vendor
extension into their OpenAPI validation schema. apiextensions JSONSchemaProps
is a closed struct that only preserves x-kubernetes-* vendor extensions, so
helm-controller's server-side apply rejected the unknown key ("field not
declared in schema") and the CRDs never applied — the controller crash-looped
and backups were non-functional.
Move the field-to-source mapping the dashboard needs into CRD
metadata.annotations (preserved by the apiserver), emitted via
+kubebuilder:metadata:annotations markers, and drop the post-processing awk
injector. The dashboard reattaches the extension client-side.
A helm-unittest suite pins the contract: each dropdown field carries its source
in metadata.annotations and must not carry x-cozystack-options in the schema,
guarding against re-introducing the injector.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
Myasnikov DaniilandClaude Opus 4.8
|
b759446fa6 |
feat(monitoring): add tenant overview dashboard
Add a cross-tenant overview dashboard for the root/infra Grafana that surfaces which tenants consume what resources, how many apps (Flux HelmReleases) each runs, and per-tenant health. Panels: - Fleet summary stats (tenants, apps, pods, CPU/mem/storage) - Tenant leaderboard table (CPU/mem usage vs requests, storage, pods, PVCs, app count, not-ready apps) - Top-N bar gauges by CPU, memory, storage and app count - CPU/memory usage trends per tenant - Pod restart rate and not-ready HelmReleases per tenant Registered in dashboards-infra.list so it deploys only to the root/cozy-monitoring Grafana, keeping cross-tenant data off per-tenant instances. Uses standard cadvisor, kube-state-metrics and gotk_resource_info metrics. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
3a5ff3d9c8 |
feat(ouroboros): bump to v0.8.0
v0.8.0 makes the TCP proxy log an explicit reason when its backend readiness check fails, instead of staying silently NotReady. That silent failure mode is exactly what makes a mispointed proxy backend hard to diagnose, so the upgrade pays off the moment the target is wrong. The bump also carries the v0.7.3 migration of the kubectl sidecar image off Docker Hub to mirror.gcr.io, sidestepping anonymous pull rate-limits on shared CI/runner IPs. Chart manifest and controller image are re-pinned by digest in lockstep with the version, and the image-pin test fixture is updated to match. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
f90411da7a |
fix(networking): point host ouroboros proxy at the root-tenant ingress
With publishing.proxyProtocol enabled the host emits a cozystack.ouroboros Package via the no-override Package macro, so the proxy inherits the wrapper chart default backend ingress-nginx-controller.cozy-ingress-nginx. That layout describes a managed Kubernetes tenant cluster; on the host, extra/ingress deploys ingress-nginx into the root tenant namespace as root-ingress-controller in tenant-root. The composed backend FQDN never resolves, so ouroboros-proxy never reaches Ready. Emit the host ouroboros Package with a proxy.target override derived from publishing.ingressName (namespace plus <short>-ingress-controller service name), mirroring the trimPrefix transform extra/ingress applies. Add bundle helm-unittest coverage for the default and a non-default ingress name, and correct the misleading wrapper values.yaml comment. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
588c9f2d62 |
feat(api): add Option resource and x-cozystack-options for dynamic form dropdowns (#2778)
## What this PR does
Adds a generic mechanism for **runtime-populated dropdowns** in
dashboard create/edit forms. Many fields are references to live cluster
resources (GPU device names, KubeVirt instancetypes/preferences, Multus
networks, VM images, bucket storage pools, storage classes, backup
classes/plans) but were free-text or static enums that drift from
reality.
- **`Option` resource** — `core.cozystack.io/v1alpha1`, namespaced,
read-only (list/get/watch). Items are computed on read by an in-process
provider registry using a privileged client, so tenants get curated
option lists without broad cluster reads. Providers: `gpu`,
`instancetype`, `instanceprofile`, `network`, `image`, `storagepool`,
`storageclass`, and backup-related `backupclass`, `plan`, `backup`,
`appkind`.
- **`x-cozystack-options` schema keyword** — app `values.yaml` declare
it via the new cozyvalues-gen `@x-<keyword>` directive (e.g. `##
@x-cozystack-options {source: instancetype}`). Schemas, Go types, and
the `-rd` ApplicationDefinitions are regenerated from source so the
extension is carried into the served OpenAPI. The dashboard renders
annotated fields as dropdowns sourced from the live cluster (UI widget
ships in a separate cozystack-ui PR).
- **RBAC** — the apiserver ServiceAccount gets the privileged reads;
tenants get read-only access to `options` in their namespace via the
aggregated tenant ClusterRoles (granted at `cozy:tenant:view:base`, so
every access level from view upward can populate dropdowns).
- Drops the redundant manual `instanceProfile` enum injection from
vm-instance's Makefile (now driven dynamically).
## Dependency on cozyvalues-gen
The `x-cozystack-options` annotations require the new vendor-extension
directive from cozystack/cozyvalues-gen#27. This PR bumps the pinned
cozyvalues-gen to **v1.6.0**. **A cozyvalues-gen v1.6.0 release must be
cut before CI can pass** — until then the pre-commit schema-generation
job cannot download the binary and will fail.
### Screenshots
N/A — no UI changes in this repository. The consuming dashboard widget
ships in a separate cozystack-ui PR.
### Release note
```release-note
feat(api): add a namespaced read-only Option resource (core.cozystack.io) and the x-cozystack-options schema keyword, enabling dashboard form fields (GPU, instancetype/profile, network, image, storage pool/class, backup class/plan) to render as dropdowns populated from live cluster resources
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Expanded dynamic dropdowns across many apps (storageClass,
storagePool, instanceType/instanceProfile, GPU, image, vmdisk, network,
backup/plan/backupclass, appkind) so chart fields can be populated from
live cluster resources.
* **Chores**
* Updated API/RBAC manifests to grant read access needed for option
providers.
* **Tests**
* Added provider and REST tests to validate option sources, scoping,
ordering, and failure handling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Aleksei Sviridkin
|
ed20a4322a |
fix(objectstorage-controller): propagate Bucket readiness to BucketClaim
The vendored COSI controller (v0.2.2) sets only BucketClaim.status.bucketName in the dynamic-provisioning path and hardcodes bucketReady=false, never re-reading the Bucket afterwards. The controller does not watch Buckets, so the readiness transition the sidecar performs (false->true) is never reflected on the BucketClaim: bucketReady stays false forever and BucketAccess is never granted, leaving provisioned buckets without credentials. Re-read the live Bucket after create and propagate its readiness, so the periodic resync converges the BucketClaim to ready. This matches the behavior the rewritten upstream controller already has on its main branch. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
26da871c00 |
feat(rbac): allow tenants to start/stop/restart their VMs (#2777)
<img width="2770" height="1970" alt="image" src="https://github.com/user-attachments/assets/774c7c71-0e6a-496a-9898-8ad1ed848b0a" /> ## Summary Grants tenant users permission to call the KubeVirt VM power-control subresources (`subresources.kubevirt.io` → `virtualmachines/{name}/start|stop|restart`, verb `update`) by adding them to the `cozy:tenant:use:base` ClusterRole, which aggregates into `tenant-use`, `tenant-admin`, and `tenant-super-admin`. Without this, the dashboard VM power buttons return 403 even for tenant admins: `kubectl auth can-i update virtualmachines --subresource=start` is `no` for every tenant role today, while `virtualmachineinstances/console|vnc|portforward` are already granted at the same `use` level — so power control belongs there too. ## Companion PR UI that consumes this permission (VM power controls + VNC fixes): cozystack/cozystack-ui#27 ## Test - `helm template` renders `cozy:tenant:use:base` with the new `virtualmachines/start|stop|restart` rule. - After apply, `kubectl auth can-i update virtualmachines --subresource=start|stop|restart` flips from `no` to `yes` for the `tenant-*-use/admin/super-admin` groups. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Virtual Machine Power Control: Users with tenant-level access can now start, stop, and restart virtual machines, providing enhanced management capabilities for VM operations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
e59f93605a |
test(cozystack-basics): pin tenant view grant on core.cozystack.io options
The tenant view aggregation grants read on the core.cozystack.io/options virtual resource so dashboard dropdowns can be populated. Add a helm-unittest on the cozy:tenant:view:base ClusterRole so a silent RBAC regression on the tenant side is caught, mirroring the apiserver-side rbac test. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
f0f1431546 |
feat(kubernetes): enable GPU passthrough out-of-the-box (NvLinkDisable + gpu=on) (#2780)
## What this PR does Makes GPU passthrough work out-of-the-box for tenant Kubernetes clusters (the `kubernetes` app), so operators no longer have to hand-fix it. Two defaults: 1. **Tenant gpu-operator loads the nvidia driver with `NVreg_NvLinkDisable=1`.** 2. **Nodes in GPU node-groups are labeled `gpu=on` automatically.** ### Root cause (validated empirically on a live cluster) cozystack passes through individual SXM GPUs (e.g. H200) into tenant worker VMs **without** the NVSwitches. Such a GPU's driver then waits for an NVLink fabric that can never come up — there is no NVSwitch and fabricmanager has nothing to manage — so `nvidia-smi` reports `Fabric State: In Progress` indefinitely. As a result every CUDA call fails with `system not yet initialized`, `nvidia-operator-validator` (cuda-validation) crashloops, and no GPUs are advertised. Loading the kernel module with `NVreg_NvLinkDisable=1` flips `Fabric State` to `N/A`, CUDA initializes, and the validators pass. This is the correct setting for per-GPU passthrough because there is no in-VM NVLink/NVSwitch peering — the GPU is alone in the VM, so there is nothing to disable that the workload needs. The second issue: HAMi's `hami-device-plugin` DaemonSet has `nodeSelector: gpu=on` and stays at `DESIRED=0` because GPU worker nodes lack that label, so `nvidia.com/gpu` is never advertised. Labeling the GPU nodes `gpu=on` lets the device plugin schedule and advertise the GPUs. ### How it works - **`gpu=on` label:** node-groups in `packages/apps/kubernetes/templates/cluster.yaml` that declare `gpus` get `--node-labels=gpu=on` appended to their kubelet args (in the `KubeadmConfigTemplate` `joinConfiguration`). Non-GPU node-groups are untouched. - **`NVreg_NvLinkDisable=1`:** the upstream gpu-operator chart only references an existing ConfigMap by name (`driver.kernelModuleConfig.name`) and never creates it. The gpu-operator wrapper chart (`packages/system/gpu-operator`) now renders that ConfigMap (opt-in via `kernelModuleConfig.create`) so it deploys into the tenant `cozy-gpu-operator` namespace alongside the operator. The tenant gpu-operator HelmRelease defaults enable it and point `driver.kernelModuleConfig.name` at it. Both defaults remain overridable: a user setting `addons.gpuOperator.valuesOverride` still wins via the existing `mergeOverwrite(defaults, overrides)` in the HelmRelease template. ### Screenshots Not applicable — no UI changes. ### Release note ```release-note feat(kubernetes): enable GPU passthrough out-of-the-box for tenant clusters. GPU node-groups are now labeled `gpu=on` so HAMi's device plugin schedules and advertises `nvidia.com/gpu`, and the tenant gpu-operator loads the nvidia driver with `NVreg_NvLinkDisable=1` so single-SXM-GPU passthrough (no NVSwitch in the VM) no longer hangs on `Fabric State: In Progress` / `system not yet initialized`. Both defaults are overridable via `addons.gpuOperator.valuesOverride`. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * GPU-enabled node groups now get the expected GPU label during node registration. * GPU operator supports optional kernel-module parameter configuration with a default parameter set. * **Tests** * Added tests verifying GPU node labeling is applied only for GPU-enabled node groups. * Added tests validating kernel-module config creation, naming, and content behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
b30eb4602d |
fix(cozystack-api): drop unused objectstorage bucketclasses RBAC grant
The Option providers RBAC block granted read on objectstorage.k8s.io/bucketclasses, but no provider in the registry lists BucketClasses and no schema field references a bucketclass source — the bucket app's storage pool field uses the storagepool provider, which reads seaweedfs HelmReleases. Drop the unused grant and pin its absence in the chart test. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
4cbe5b62d1 |
fix(cozystack-api): grant the option apiserver read on plans and backups
The plan and backup option providers list namespaced plans and backups in the backups.cozystack.io group, but the cozystack-api ClusterRole only granted backupclasses there. The apiserver's privileged client runs as this ServiceAccount, so both providers got forbidden — List silently dropped the sources and Get returned 500, leaving the plan/backup dropdowns in the backup forms empty. Add plans and backups to the backups.cozystack.io rule and cover the grant with a chart test. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Andrei KvapilandClaude
|
34e370e5c5 |
chore(apps): regenerate schemas and ApplicationDefinitions with x-cozystack-options
Regenerate values.schema.json and embedded openAPISchema in the -rd ApplicationDefinitions so the x-cozystack-options dynamic-source directives declared in each app values.yaml are carried into the generated artifacts. Brings the -rd schemas in sync with the hand-edited values.schema.json files (CI runs make generate + git diff --exit-code). Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei KvapilandClaude
|
49976c02cd |
fix(rbac): grant tenant view role read access to options
Options were granted on cozy:tenant:use:base, leaving view-only tenant users (cozy:tenant:view) without read access — so their create/edit forms could not populate dynamic dropdowns. Move the grant down to cozy:tenant:view:base, which aggregates up into use/admin/super-admin, covering every human tenant access level. The tenant ServiceAccount role keeps its grant via cozy:tenant:base. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei KvapilandClaude
|
d0c74f9b89 |
feat(cozystack-api): add plan, backup and appkind Option sources
Add three dropdown sources to the Option provider registry so the dashboard's backup forms can use the unified x-cozystack-options mechanism instead of injecting client-side enum arrays: - plan / backup: namespaced, list Plans / Backups in the tenant namespace - appkind: cluster-scoped, lists application kinds from ApplicationDefinitions Annotate the backups.cozystack.io CRD schemas (Plan, BackupJob, Backup, RestoreJob) with x-cozystack-options on backupClassName (backupclass), applicationRef.kind / targetApplicationRef.kind (appkind), planRef.name (plan) and backupRef.name (backup). This also gives the previously unused backupclass source its first consumer. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei KvapilandClaude
|
ec6ec76703 |
feat(cozystack-api): add Option resource for dynamic form dropdowns
Introduce a read-only, namespaced virtual resource core.cozystack.io/v1alpha1 Option that serves named lists of dropdown options for the dashboard. Each object (one per source: gpu, instancetype, instanceprofile, network, image, storagepool, storageclass, vmdisk, backupclass) is computed on read by an in-process provider using a privileged dynamic client, so tenants need no direct access to Nodes, the KubeVirt CR, cluster instancetypes, etc. Tenants get read access via the aggregated cozy:tenant roles; the apiserver ServiceAccount gains read on the underlying sources. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Aleksei Sviridkin
|
a60c2e6e68 |
fix(gpu-operator): name kernel-module ConfigMap after the driver reference
The wrapper chart named the kernel-module ConfigMap from its own kernelModuleConfig.name key, while the NVIDIA driver mounts the ConfigMap referenced by gpu-operator.driver.kernelModuleConfig.name. The two keys defaulted to the same value but were never linked, so overriding only the driver reference left the driver pointing at a ConfigMap that is never rendered and the NVreg_NvLinkDisable=1 parameters were silently dropped. Derive the ConfigMap name from the driver's effective reference, falling back to the wrapper key when unset, so the rendered ConfigMap and the driver reference can no longer diverge. Add helm-unittest coverage for the rendered name, namespace, data key, and the create=false no-op. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
myasnikovdaniil
|
c435efa199 |
feat(flux): upgrade to v2.8.0 + chart fixes for strict SSA & kstatus (folds #2612) (#2602)
## What this PR does Upgrades Flux v2.7.3 → v2.8.0 across both the vendored tenant chart and the embedded management-cluster manifests, and folds in the chart fixes that become hard errors under the new strict helm-controller v1.5. Flux v2.8's helm-controller v1.5.0 ships: - **Server-Side Apply with `--force-conflicts`** — strict CRD schema enforcement; misplaced fields (silently dropped on v2.7) now fail the apply. - **kstatus-based health checking** by default — parent HR waits for every applied resource (including child HRs) to be Ready before flipping its own Ready, surfacing latent ordering deadlocks. - **Helm v4 embedded** — `upgrade.force: true` is deprecated and now conflicts with SSA. - **`HelmRelease.spec.healthCheckExprs`** — prerequisite for proper readiness gating (used in PR #2601 split). Folds in PR #2612 (kubevirt-instancetypes null TPM fix) since the same Flux upgrade triggers it. ## Commits **Flux upgrade itself:** - `feat(fluxcd)`: bump `flux-operator` / `flux-instance` vendored charts to v0.48.0; web UI opt-in. - `feat(flux)`: regenerate embedded management-cluster manifests via `make update` in `packages/core/flux-aio` (timoni bundle build). **Chart fixes for strict SSA — fields the chart sent that v2.7 silently dropped, v2.8 rejects:** - `fix(kubevirt-instancetypes)`: drop persistent strip that produced null `preferredTPM` (folds #2612). - `fix(foundationdb)`: relocate `faultDomain`, `imageType`, `labels`, `minimumUptimeSecondsForBounce` from inside `automationOptions` to direct children of `spec`. - `fix(kafka)`: place `enableServiceLinks` under `template.pod`, not a phantom `template.spec`. - `fix(vm-instance)`: emit `disk: {}` (not `disk:`/null) when no bus is set. - `fix(platform)`: drop deprecated `upgrade.force: true` from HelmReleases; fix `kafka` WorkloadMonitor `replicas` paths. **Ordering / deadlock fixes under v2.8 kstatus:** - `fix(vpa)`: break circular wait between parent install and nested `vpa-for-vpa` HR. - `fix(kubernetes)`: drop lookup-guarded parent-HR `dependsOn` on tenant addon child HRs (parent waits on child via kstatus, child waited on parent — deadlock). **E2E waits for v2.8 kstatus timing:** - `test(e2e)`: bump app HR-Ready waits to 5m (was 20s–100s under v2.7's faster dispatch). - `test(e2e)`: wait for parent HR Ready before downstream asserts in `run-kubernetes.sh` and `vminstance.bats`. ## Scope discipline This PR is part of the split of #2619 (the consolidated CI fixes branch) into review-friendly pieces. Companion PRs: - **PR #2601** (seaweedfs split) — folded into this PR (commits `29c6afc8`, `0e8b46d7`, `7157158c`, `dccdeb52`, `f880b324`): the seaweedfs-system → seaweedfs-db + seaweedfs-system split, its adoption migration 43 (targetVersion 44), and the configurable db resources all land here, because the strict-SSA `upgrade.force` removal and the kstatus parent-HR timeout bump only make sense together with the split. #2601 is superseded. - **PR #2558** (drop 3× retry on `Run E2E` + `Install Cozystack`) — independent, lands separately. - Several smaller standalone fixes lifted out of #2619 (startup probes, cert-manager `dependsOn`, prepull machinery, CSI HR timeout, NFS/OIDC test improvements) — opened as separate PRs. ## Verification - `helm template` renders cleanly for both `fluxcd` and `fluxcd-operator` packages with `web.enabled=false` (default) and `web.enabled=true`. - Embedded `cmd/cozystack-operator` binary contains the v1.5.0 / v1.8.0 / v2.1.0 controller image strings. - No references to the v0.39-removed `--disable-wait-interruption` flag anywhere in `packages/` or `internal/`. ### Release note ```release-note Flux upgraded to v2.8.0 (helm-controller v1.5 — Helm v4 Server-Side Apply with --force-conflicts, kstatus health checking). When upgrading existing clusters: - Kubernetes 1.33+ is now required for the platform (management) cluster, and for any tenant cluster that enables the optional (default-off) Flux addon — that addon ships the bumped Flux too. - HelmReleases no longer set `upgrade.force: true`. Helm v4 SSA resolves field-ownership conflicts automatically (`--force-conflicts`), but that is not the old client-side replace: immutable-field changes (e.g. StatefulSet volumeClaimTemplates/serviceName) no longer self-heal and require manual recreation — delete the object (e.g. `kubectl delete sts <name> --cascade=orphan`) and let Flux recreate it. - KubeVirt: persistent TPM/EFI is re-enabled for the Windows 11/2k22/2k25 preferences (KubeVirt 1.8 VMPersistentState); each affected VM provisions an extra RWO backend-storage PVC from the default StorageClass. - KubeVirt: the EOL centos.7*/centos.stream8* preferences are retained as deprecated, hidden aliases (`tags: hidden`, `instancetype.kubevirt.io/deprecated: "true"`) — existing VMInstances on these profiles keep rendering and need no action on upgrade, but the profiles are no longer offered for new VMs; repoint to centos.stream9/10 when convenient. The gn1.* GPU instancetypes are likewise retained. - FoundationDB: imageType now reaches the operator (silently dropped pre-SSA); it is pinned to `split` to match the value existing clusters effectively ran, so upgrades stay non-disruptive. Set `imageType: unified` to migrate deliberately. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Flux Status web UI: optional web server, config secret, service port, HTTPRoute/Ingress, network policy, and RBAC roles. * New SeaweedFS DB Helm chart and optional managed DB release. * **Improvements** * CRD/schema enhancements: new provider kinds, validations, variant option, and external checksum refs. * Raised Kubernetes prerequisite to 1.30+; extended e2e timeouts for reliability. * **Chores** * Bumped Flux Operator and Flux versions; documentation links updated to fluxoperator.dev. * **Bug Fixes** * Removed aggressive HelmRelease force-upgrade/install flags. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2602?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
2d58479f06 |
feat(platform): add kuberture as optional system package (#2647)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does Adds [kuberture](https://github.com/lexfrei/kuberture) as an optional system package. Kuberture is a small controller that translates the `default/kubernetes` EndpointSlice into headless Services with external-dns-style annotations, closing the gap that external-dns cannot use EndpointSlice as a source directly: ```text EndpointSlice (kubernetes) → kuberture → headless Service(s) with annotations → external-dns → DNS ``` The package is opt-in via `bundles.enabledPackages: [cozystack.kuberture]`. Operators populate `config.outputs` per HelmRelease — each output targets a specific external-dns instance by its `annotationPrefix`, so a single kuberture install can serve multiple external-dns instances (e.g. one publishing to the public zone, another to an internal corporate prefix) simultaneously. The vendored chart at `packages/system/kuberture/charts/kuberture/` is pulled from `oci://ghcr.io/lexfrei/kuberture/charts/kuberture` and pinned by both version AND OCI manifest digest in the Makefile (mirroring the ouroboros lockstep pattern). The controller image at `ghcr.io/lexfrei/kuberture` is pinned by tag+sha256 in the shim values.yaml. Both pins advance in lockstep with each upstream release; the procedure is documented inline. Install-order dependencies declared in `PackageSource`: `cozystack.networking` (CNI before pod scheduling) and `cozystack.prometheus-operator-crds` (ServiceMonitor CRD before the chart's enabled-by-default ServiceMonitor manifest). External-dns is deliberately NOT a dependency — kuberture creates annotated Services regardless of whether external-dns is installed; external-dns is a downstream consumer, not a runtime prerequisite. Coverage: - **helm-unittest** (`packages/system/kuberture/tests/kuberture_test.yaml`, 10 cases): fail-fast on empty outputs, ServiceMonitor enabled by default, source pinned to `default/kubernetes`, multi-output prefix isolation, empty-string `annotationPrefix` schema rejection, omitted-prefix fallback to controller default, ConfigMap → Deployment volume mount, controller image pinned by tag+sha256 digest, Role permits full Service lifecycle, metrics containerPort derived from `metricsBindAddress`. - **bats e2e** (`hack/e2e-apps/kuberture.bats`): applies a `Package cozystack.kuberture` with two outputs (different `annotationPrefix` values), waits for HelmRelease and controller readiness, asserts both output Services are headless and carry the right `hostname`/`target`/`ttl` annotations under their configured prefix, cross-checks per-output isolation (each Service only carries its own prefix's annotations). ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> Not applicable — no UI changes. ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note feat(platform): add kuberture as an optional system package — translates the default/kubernetes EndpointSlice into annotated headless Services so external-dns can publish the Kubernetes API endpoint to DNS. Opt-in via bundles.enabledPackages: [cozystack.kuberture]; populate config.outputs to target one or more external-dns instances by annotationPrefix. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Adds kuberture as an optional system package to publish the Kubernetes API endpoint to DNS via configurable external-dns outputs; includes Helm chart, RBAC, NetworkPolicy, PDB, ServiceMonitor, and validated chart values/schema. * **Tests** * Adds Helm unit tests and an end-to-end smoke test validating controller readiness, headless Service annotations, and external-dns routing/probe behavior. * **Documentation** * Adds a full README with usage, configuration examples, and vendoring/pinning guidance. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2647) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> --- Documentation companion: [cozystack/website#539](https://github.com/cozystack/website/pull/539). |
||
|
Andrei KvapilandClaude
|
238a3b665c |
feat(kubernetes): default NVreg_NvLinkDisable=1 for tenant gpu-operator
When a single SXM GPU (e.g. H200) is passed through into a tenant worker VM without the NVSwitches, the nvidia driver waits forever for an NVLink fabric that can never come up (Fabric State: In Progress). Every CUDA call then fails with 'system not yet initialized' and the nvidia-operator-validator crashloops. Loading the kernel module with NVreg_NvLinkDisable=1 flips Fabric State to N/A so CUDA initializes — correct for per-GPU passthrough, which has no in-VM NVLink/NVSwitch peering anyway. The upstream gpu-operator chart only references an existing ConfigMap by name via driver.kernelModuleConfig.name and never creates it. Render that ConfigMap from the gpu-operator wrapper chart (opt-in via kernelModuleConfig.create) so it lands in the tenant cozy-gpu-operator namespace alongside the operator, and wire the tenant HelmRelease defaults to enable it and point the driver at it. Remains overridable via addons.gpuOperator.valuesOverride. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei KvapilandClaude
|
8e78add8ea |
feat(rbac): allow tenants to start/stop/restart their VMs
Grant the subresources.kubevirt.io virtualmachines/start|stop|restart "update" verb in cozy:tenant:use:base so the dashboard VM power controls work for tenant users (aggregates into admin/super-admin). Without it the actions return 403 even for tenant admins. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
