## What this PR does
The vendored COSI `objectstorage-controller` (built from upstream
`v0.2.2`) leaves dynamically-provisioned buckets without credentials.
In the dynamic-provisioning path, `provisionBucketClaimOperation` sets
`BucketClaim.status.bucketName` but hardcodes `bucketReady=false` and
never re-reads the `Bucket`. The controller does not watch `Bucket`
objects, so the readiness transition the sidecar performs after
`DriverCreateBucket` (`Bucket.status.bucketReady` `false`→`true`) is
never propagated back to the `BucketClaim`. Each periodic resync
re-enters the same branch (`Buckets().Create` returns `AlreadyExists`,
which is ignored) and re-sets `bucketReady=false`. As a result
`BucketClaim.status.bucketReady` stays `false` forever, the sidecar
refuses to grant the `BucketAccess` ("BucketAccess can't be granted to
Bucket not in Ready state"), and the credentials secret is never created
— even though the bucket exists in the backend and the `Bucket` CR
already reports `bucketReady: true`.
This adds a vendored patch that re-reads the live `Bucket` after create
and propagates its readiness onto the `BucketClaim`, so the existing
periodic resync converges the claim to ready and access can be granted.
The fix mirrors what the rewritten upstream controller already does on
`main` (`claim.Status.ReadyToUse = bucket.Status.ReadyToUse`). Upstream
`main` is a `v1alpha2` rewrite and the maintained `release-0.2` branch
still carries the bug, so this stays a vendored patch against `v0.2.2`
until the controller is bumped to a release that includes the fix.
Verified: reproduced the stuck-`false` claim with a unit test against
the upstream `v0.2.2` source and confirmed the patch resolves it,
existing controller tests pass, the patch applies cleanly to `v0.2.2`,
and the controller image builds.
### Release note
```release-note
fix(objectstorage-controller): propagate Bucket readiness to the BucketClaim so dynamically-provisioned buckets receive credentials instead of getting stuck without a granted BucketAccess
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Fixed an issue where bucket readiness status was incorrectly reported
when a bucket already existed. The status now accurately reflects the
actual state of the existing bucket instead of being unconditionally set
to not ready.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Bumps the `seaweedfs-cosi-driver` image from `v0.3.0` to `v0.3.1`.
`v0.3.1` ships the stale-socket self-heal fix. After a non-graceful exit
(SIGKILL / OOM / panic) Go does not unlink the COSI unix socket, and
because it lives on an `emptyDir` that survives container restarts, the
next start failed with `bind: address already in use` and the
`objectstorage-provisioner` wedged in `CrashLoopBackOff`. While it was
down, `BucketClaim` reconciliation stopped and buckets were never
provisioned — the COSI `Bucket` could even report `bucketReady: true`
while no bucket existed. This is the CrashLoopBackOff reported in #2429.
The driver now removes a leftover socket before binding (only when it
exists, is a socket, and has no live listener), so the provisioner
self-heals on restart.
Upstream: seaweedfs/seaweedfs-cosi-driver#8 (merged), released in
v0.3.1. Points at the upstream published multi-arch image (amd64 +
arm64) — no vendoring needed.
### Release note
```release-note
fix(seaweedfs): bump seaweedfs-cosi-driver to v0.3.1, which self-heals the object storage COSI provisioner from a CrashLoopBackOff caused by a stale unix socket
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated SeaweedFS cosi driver container image to a newer version.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Two changes that together fix the v1.5.0 release blocker where the
`cozy-backup-controller` HelmRelease failed on every reconcile.
**Root cause** — the four `backups.cozystack.io` CRDs (`backupjobs`,
`backups`, `plans`, `restorejobs`) embedded the `x-cozystack-options`
vendor extension directly in their OpenAPI validation schema.
`apiextensions` `JSONSchemaProps` only preserves the documented
`x-kubernetes-*` vendor extensions — every other `x-` key is rejected by
helm-controller's server-side apply (`field not declared in schema`) and
dropped by the apiserver on decode. So the CRDs never applied and the
controller crash-looped.
This moves the field→source mapping the dashboard needs out of the
schema and into CRD `metadata.annotations` (which the apiserver
preserves), emitted declaratively via
`+kubebuilder:metadata:annotations` markers on the Go types, and removes
the post-processing awk injector (`hack/inject-cozystack-options.awk`).
A helm-unittest suite pins the contract: each dropdown field carries its
source in annotations and must not carry `x-cozystack-options` in the
schema. The dashboard reattaches the extension client-side from these
annotations.
Annotation contract: `options.cozystack.io/source.<spec-relative-path>:
<option-source>`, e.g. `options.cozystack.io/source.applicationRef.kind:
appkind`.
**Detection gap** — the install e2e let this ship green: the fanned-out
`kubectl wait` discarded child exit codes (a bare POSIX `wait` returns
0) and the readiness check printed offending HelmReleases without ever
exiting non-zero. The gate is now restored — a single `kubectl wait hr
--all -A` for trace visibility, an outcome-based re-list that also gates
late-created HelmReleases (with a short retry to absorb momentary
drift-reconcile flaps) and `exit 1`s, plus a full Ready-condition dump
per non-ready HelmRelease so the real failure reason is visible. This
commit is by @myasnikovdaniil, consolidated here from #2824 so the root
cause and the CI gap land in one PR. With the gate restored, CI on
`main` stays red until the backups fix in this PR merges — intended.
Closes#2822 · Dashboard counterpart: cozystack/cozystack-ui#40
### Release note
```release-note
fix(backups): fix backup-controller CRDs failing to apply — the dashboard dropdown hint was placed in the CRD schema where apiextensions rejects it; it now lives in CRD annotations
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a CRD option-source test suite and a package test target to
validate annotation-based option sources.
* **Bug Fixes**
* Improved end-to-end install checks with consolidated readiness waits
and richer failure diagnostics for releases.
* **Refactor**
* Migrated backup-controller CRD option metadata from legacy vendor
extensions to annotation-based keys for cleaner schema metadata.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The HR readiness gate in e2e-install-cozystack.bats was toothless in two
layers: the backgrounded kubectl-wait fan-out discarded child exit codes
(POSIX `wait` without args returns 0), and the final check echoed
"Some HelmReleases failed to reconcile" without exiting non-zero — a
regression from 1f240387f, which replaced the unavailable bats `fail`
helper with a bare echo. A permanently-failing platform HelmRelease
(e.g. the backup-controller CRD rejection, #2822) shipped through green
CI for weeks.
Replace the fan-out with a single `kubectl wait hr --all -A` for trace
visibility, gate on an outcome-based re-list (which also covers HRs
created after the snapshot, with a 120s retry to absorb momentary
drift-reconcile flaps), and dump the full Ready condition message per
non-ready HR so the real error is visible in the test output.
Refs #2822
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The four backups.cozystack.io CRDs injected the x-cozystack-options vendor
extension into their OpenAPI validation schema. apiextensions JSONSchemaProps
is a closed struct that only preserves x-kubernetes-* vendor extensions, so
helm-controller's server-side apply rejected the unknown key ("field not
declared in schema") and the CRDs never applied — the controller crash-looped
and backups were non-functional.
Move the field-to-source mapping the dashboard needs into CRD
metadata.annotations (preserved by the apiserver), emitted via
+kubebuilder:metadata:annotations markers, and drop the post-processing awk
injector. The dashboard reattaches the extension client-side.
A helm-unittest suite pins the contract: each dropdown field carries its source
in metadata.annotations and must not carry x-cozystack-options in the schema,
guarding against re-introducing the injector.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
--shard-count now accepts "auto" (the new default) or an explicit
positive integer. Auto mode drives the shard count from the tenant
HelmRelease count — K = clamp(ceil(H/100), 1, min(16, T)) — applied
with hysteresis anchored on the currently provisioned shards (scale up
eagerly past 120 HR/shard, scale down lazily under 60 HR/shard) so K
does not flap around sizing boundaries.
The per-shard target is 100 HelmReleases (down from the 150 in the
design doc) to leave headroom for existing tenants growing their
HelmRelease count without an immediate reshard.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
E2E caught the cloned shard crashlooping on Talos: the installer injects
the node-local KubePrism endpoint (KUBERNETES_SERVICE_HOST=localhost,
KUBERNETES_SERVICE_PORT=7445) into hostNetwork workloads, and the clone
carried it into a non-hostNetwork pod where localhost is dead. Sanitise
those env vars away so shards use the in-cluster defaults.
Defense in depth for the same failure mode: the placement controller now
defers reassignments whose target shard Deployment has no ready replica,
so a backfill can never hand tenants to a helm-controller that is not
actually running, and reacts to shard readiness changes directly.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Defensively removes a leftover flux-tenants Deployment only when no
HelmRelease carries sharding.fluxcd.io/key=tenants anymore (the
flux-shard-operator backfill owns the live drain), and stamps
cozystack-version 45 mirroring migration 43's labeled manifest.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
The installer no longer ships the hand-rolled tenant helm-controller
shard: flux-shard-operator provisions operator-managed shard
Deployments cloned from flux-aio at runtime instead, and deletes a
running flux-tenants Deployment only after the legacy tenants bucket
drains. flux-aio itself stays unchanged and keeps owning everything
without a shard key; the Makefile lines that synced the image/version
into the removed manifest go away with it.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
New cozy-fluxcd system package (Deployment with 2 replicas, webhook
cert via cert-manager, CREATE-only MutatingWebhookConfiguration with
failurePolicy Ignore scoped to HelmReleases born on the legacy tenants
bucket), registered in the platform system bundle and the root image
build.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Manager wiring modeled on lineage-controller-webhook: leader-elected
controllers, webhook served on all replicas, metadata-only caches for
HelmReleases (shard-key label selector) and namespaces, Deployments
cached from the flux namespace only.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Three leader-elected/served parts on metadata-only watches, so the
operator never decodes the helm-controller status-patch firehose:
- placement controller: singleton full-sync reconciler that rebuilds
the tenant index from HelmRelease labels (restart-safe), records the
assignment on the tenant namespace, self-heals HR labels and paces
reassignments so a backfill never floods a target shard;
- shard provisioner: reconciles helm-controller-shard<i> Deployments
cloned from the flux-aio helm-controller container and sanitised
(no host networking, no localhost cross-container wiring, per-shard
watch-label-selector), prunes drained extra shards and retires the
legacy flux-tenants Deployment once the tenants bucket is empty;
- CREATE-only mutating webhook: stamps the tenant's recorded shard on
newborn HelmReleases so each one is born on the correct shard; every
miss is permissive and falls back to the placement controller.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Tenant-granular placement for sharding the management-cluster tenant
helm-controller: tenant key extraction from HelmRelease metadata,
greedy least-loaded assignment weighted by HelmRelease count (with the
N-tenants-over-N-shards 1-per-shard guarantee), minimal-movement
rescale, threshold-gated rebalance with per-tenant cooldown hook,
pinned tenants, and the autosizing recommendation formula.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Harbor stores registry data only in S3-compatible object storage and
has no local-filesystem backend, but the app docs never stated this.
Without SeaweedFS available in the tenant chain the registry
BucketClaim never yields its credentials secret and the HelmRelease
silently stays unreconciled — a common and confusing first-deploy
failure.
Add a Prerequisites section to the chart README (propagated to the
published app docs) explaining the requirement, how the bucket is
provisioned via COSI from SeaweedFS, how the class is inherited down
the tenant tree, and how to enable it.
Co-authored-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Adds an `/etc/lvm/lvm.conf` overwrite to the e2e Talos machine config so
the host LVM ignores `/dev/loop` devices, alongside the existing
`drbd`/`dm`/`zd` filters. This prevents the host from scanning and
activating volume groups located inside loop-mounted disk images used
during e2e.
### Release note
```release-note
test(e2e): exclude loop devices from host LVM scanning
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated cluster preparation to adjust LVM behavior in test
environments: LVM backup/archive is disabled and device filtering was
tightened to ignore ephemeral/virtual block devices.
* This reduces interference from transient devices during end-to-end
cluster tests.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2798?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Add a cross-tenant overview dashboard for the root/infra Grafana that
surfaces which tenants consume what resources, how many apps (Flux
HelmReleases) each runs, and per-tenant health.
Panels:
- Fleet summary stats (tenants, apps, pods, CPU/mem/storage)
- Tenant leaderboard table (CPU/mem usage vs requests, storage, pods,
PVCs, app count, not-ready apps)
- Top-N bar gauges by CPU, memory, storage and app count
- CPU/memory usage trends per tenant
- Pod restart rate and not-ready HelmReleases per tenant
Registered in dashboards-infra.list so it deploys only to the
root/cozy-monitoring Grafana, keeping cross-tenant data off per-tenant
instances. Uses standard cadvisor, kube-state-metrics and
gotk_resource_info metrics.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
v0.8.0 makes the TCP proxy log an explicit reason when its backend
readiness check fails, instead of staying silently NotReady. That
silent failure mode is exactly what makes a mispointed proxy backend
hard to diagnose, so the upgrade pays off the moment the target is
wrong. The bump also carries the v0.7.3 migration of the kubectl
sidecar image off Docker Hub to mirror.gcr.io, sidestepping anonymous
pull rate-limits on shared CI/runner IPs.
Chart manifest and controller image are re-pinned by digest in lockstep
with the version, and the image-pin test fixture is updated to match.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add an /etc/lvm/lvm.conf overwrite to the e2e Talos machine config so the
host LVM ignores /dev/loop devices (alongside drbd/dm/zd). This prevents the
host from scanning and activating volume groups inside loop-mounted disk
images used during e2e.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
With publishing.proxyProtocol enabled the host emits a cozystack.ouroboros
Package via the no-override Package macro, so the proxy inherits the wrapper
chart default backend ingress-nginx-controller.cozy-ingress-nginx. That
layout describes a managed Kubernetes tenant cluster; on the host,
extra/ingress deploys ingress-nginx into the root tenant namespace as
root-ingress-controller in tenant-root. The composed backend FQDN never
resolves, so ouroboros-proxy never reaches Ready.
Emit the host ouroboros Package with a proxy.target override derived from
publishing.ingressName (namespace plus <short>-ingress-controller service
name), mirroring the trimPrefix transform extra/ingress applies. Add bundle
helm-unittest coverage for the default and a non-default ingress name, and
correct the misleading wrapper values.yaml comment.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Adds a generic mechanism for **runtime-populated dropdowns** in
dashboard create/edit forms. Many fields are references to live cluster
resources (GPU device names, KubeVirt instancetypes/preferences, Multus
networks, VM images, bucket storage pools, storage classes, backup
classes/plans) but were free-text or static enums that drift from
reality.
- **`Option` resource** — `core.cozystack.io/v1alpha1`, namespaced,
read-only (list/get/watch). Items are computed on read by an in-process
provider registry using a privileged client, so tenants get curated
option lists without broad cluster reads. Providers: `gpu`,
`instancetype`, `instanceprofile`, `network`, `image`, `storagepool`,
`storageclass`, and backup-related `backupclass`, `plan`, `backup`,
`appkind`.
- **`x-cozystack-options` schema keyword** — app `values.yaml` declare
it via the new cozyvalues-gen `@x-<keyword>` directive (e.g. `##
@x-cozystack-options {source: instancetype}`). Schemas, Go types, and
the `-rd` ApplicationDefinitions are regenerated from source so the
extension is carried into the served OpenAPI. The dashboard renders
annotated fields as dropdowns sourced from the live cluster (UI widget
ships in a separate cozystack-ui PR).
- **RBAC** — the apiserver ServiceAccount gets the privileged reads;
tenants get read-only access to `options` in their namespace via the
aggregated tenant ClusterRoles (granted at `cozy:tenant:view:base`, so
every access level from view upward can populate dropdowns).
- Drops the redundant manual `instanceProfile` enum injection from
vm-instance's Makefile (now driven dynamically).
## Dependency on cozyvalues-gen
The `x-cozystack-options` annotations require the new vendor-extension
directive from cozystack/cozyvalues-gen#27. This PR bumps the pinned
cozyvalues-gen to **v1.6.0**. **A cozyvalues-gen v1.6.0 release must be
cut before CI can pass** — until then the pre-commit schema-generation
job cannot download the binary and will fail.
### Screenshots
N/A — no UI changes in this repository. The consuming dashboard widget
ships in a separate cozystack-ui PR.
### Release note
```release-note
feat(api): add a namespaced read-only Option resource (core.cozystack.io) and the x-cozystack-options schema keyword, enabling dashboard form fields (GPU, instancetype/profile, network, image, storage pool/class, backup class/plan) to render as dropdowns populated from live cluster resources
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Expanded dynamic dropdowns across many apps (storageClass,
storagePool, instanceType/instanceProfile, GPU, image, vmdisk, network,
backup/plan/backupclass, appkind) so chart fields can be populated from
live cluster resources.
* **Chores**
* Updated API/RBAC manifests to grant read access needed for option
providers.
* **Tests**
* Added provider and REST tests to validate option sources, scoping,
ordering, and failure handling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The vendored COSI controller (v0.2.2) sets only BucketClaim.status.bucketName
in the dynamic-provisioning path and hardcodes bucketReady=false, never
re-reading the Bucket afterwards. The controller does not watch Buckets, so the
readiness transition the sidecar performs (false->true) is never reflected on
the BucketClaim: bucketReady stays false forever and BucketAccess is never
granted, leaving provisioned buckets without credentials.
Re-read the live Bucket after create and propagate its readiness, so the
periodic resync converges the BucketClaim to ready. This matches the behavior
the rewritten upstream controller already has on its main branch.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
<img width="2770" height="1970" alt="image"
src="https://github.com/user-attachments/assets/774c7c71-0e6a-496a-9898-8ad1ed848b0a"
/>
## Summary
Grants tenant users permission to call the KubeVirt VM power-control
subresources (`subresources.kubevirt.io` →
`virtualmachines/{name}/start|stop|restart`, verb `update`) by adding
them to the `cozy:tenant:use:base` ClusterRole, which aggregates into
`tenant-use`, `tenant-admin`, and `tenant-super-admin`.
Without this, the dashboard VM power buttons return 403 even for tenant
admins: `kubectl auth can-i update virtualmachines --subresource=start`
is `no` for every tenant role today, while
`virtualmachineinstances/console|vnc|portforward` are already granted at
the same `use` level — so power control belongs there too.
## Companion PR
UI that consumes this permission (VM power controls + VNC fixes):
cozystack/cozystack-ui#27
## Test
- `helm template` renders `cozy:tenant:use:base` with the new
`virtualmachines/start|stop|restart` rule.
- After apply, `kubectl auth can-i update virtualmachines
--subresource=start|stop|restart` flips from `no` to `yes` for the
`tenant-*-use/admin/super-admin` groups.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Virtual Machine Power Control: Users with tenant-level access can now
start, stop, and restart virtual machines, providing enhanced management
capabilities for VM operations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
v0.3.1 ships the stale-socket self-heal: the driver removes a leftover
COSI unix socket before binding, so the objectstorage provisioner
recovers from CrashLoopBackOff after a non-graceful exit (SIGKILL / OOM /
panic) instead of wedging on "bind: address already in use".
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Worker-node VMs of a tenant Kubernetes cluster are created by Cluster
API / the KubeVirt provider, so their virt-launcher pods were never
stamped with the `apps.cozystack.io/application.{group,kind,name}`
lineage labels that the chart already sets on its other resources
(`ingress.yaml`, `cloud-config.yaml`). As a result the dashboard could
not attribute those pods to the owning `kubernetes` application instance
— e.g. cluster resource-consumer views showed the raw `virt-launcher-…`
pod name with no link to the cluster.
This adds the application lineage labels to the worker VM template
(`KubevirtMachineTemplate` →
`virtualMachineTemplate.spec.template.metadata.labels`), mirroring the
existing stamping in `ingress.yaml`/`cloud-config.yaml`. KubeVirt
propagates labels from this block to the VMI and the virt-launcher pods
(the same path the existing `cluster.x-k8s.io/deployment-name` label
already takes), so the dashboard now attributes worker VMs to their
Kubernetes application.
### Screenshots
N/A — no UI changes; this only affects generated pod labels.
### Release note
```release-note
fix(kubernetes): stamp tenant Kubernetes worker-node VMs with apps.cozystack.io/application.{group,kind,name} so the dashboard attributes their pods to the owning Kubernetes application
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Added application lineage labels to Kubernetes VM templates so cluster
VMs include application group, kind, and name (release-name prefix
trimmed).
* **Tests**
* Added template tests validating those labels and the name-trimming
behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Two related issues caused the dashboard SPA to break with CORS errors
after the kc-access cookie expired:
1. KeycloakClient for dashboard had no webOrigins set, so the operator
never touched the field in Keycloak after initial creation. After
any rootUrl change the existing webOrigins value became stale and
blocked CORS from the SPA. Setting "+" tells Keycloak to derive
allowed origins from redirectUris.
2. oauth2-proxy returned 302 -> Keycloak /auth for unauthenticated
XHR/fetch on /api and /apis. The login page does not emit
Access-Control-Allow-Origin, so the SPA's fetch failed with CORS.
--api-route makes oauth2-proxy answer 401 without redirect, so the
SPA can handle the re-auth itself via top-level navigation.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Pin the apps.cozystack.io/application.{group,kind,name} labels stamped on the
worker VM template, including a case that trims the kubernetes- release prefix.
Also quote application.name so a purely-numeric cluster name renders a YAML
string rather than an integer that a label value would reject.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The aggregated apiserver resources in group core.cozystack.io
(TenantSecret, TenantModule, TenantNamespace) ignored the
SendInitialEvents field of ListOptions. Clients using the WatchList /
streaming-list protocol (client-go's WatchListClient feature, on by
default since v1.35) never received the required
k8s.io/initial-events-end bookmark, so their reflectors stayed stuck in
the initial-stream phase and logged "hasn't received required bookmark
event marking the end of initial events stream" every ~10s, never
reaching HasSynced.
apps.cozystack.io/Application already implemented this contract inline.
Extract that logic into a shared registry.InitialEventsBookmarker helper
(pkg/registry/utils.go) and wire it into all four translating Watch
implementations: after the initial ADDED events, the first backing
bookmark is annotated as the initial-events-end marker, with fallbacks
that emit it before the first live event and on watcher close. Application
is refactored onto the shared helper with its WorkloadMonitor
pending-event draining preserved.
Adds unit tests for the bookmarker and watch tests for each of the three
core resources asserting ADDED -> annotated Bookmark -> live event.
```release-note
[apiserver] core.cozystack.io resources (TenantSecret, TenantModule,
TenantNamespace) now emit the k8s.io/initial-events-end bookmark for
SendInitialEvents=true (WatchList) watches, so client-go informers reach
HasSynced instead of looping on the missing-bookmark warning.
```
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Simon Koudijs <simonkoudijs@gmail.com>
instancetype/instanceprofile replaced the static vm-instance enum with live
dropdowns sourced from instancetype.kubevirt.io. A wrong group/version would
make those dropdowns silently empty in every cluster (List tolerates the
error) — the failure mode the appkind GVR guard already catches. Pin both
GVRs to the served API.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The one-shot option watch emitted the current state once then blocked on
ctx.Done(). If the initial List failed (e.g. every provider forbidden by a
broken RBAC config) it blocked silently, leaving the client with an empty
stream indistinguishable from "no options". Log the error so an operator can
tell the two apart.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The tenant view aggregation grants read on the core.cozystack.io/options
virtual resource so dashboard dropdowns can be populated. Add a helm-unittest
on the cozy:tenant:view:base ClusterRole so a silent RBAC regression on the
tenant side is caught, mirroring the apiserver-side rbac test.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The gpu, image, storagepool, storageclass and vmdisk providers carry the real
parsing logic (whitelist-intersect-allocatable, prefix strip, nested pool
walk, default-class marking, size label) but had no tests — and List tolerates
a provider error by skipping it, so a logic bug ships as a silently empty or
wrong dropdown rather than a failure. Add table tests against the fake dynamic
client for each, raising option-package coverage from 39% to 77%.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The core fuzzer was migrated from gofuzz to randfill, but the apps fuzzer was
left on gofuzz. TestRoundTripTypes runs both fuzzers (apps first), so
registering a gofuzz.Continue func against the randfill-based roundtrip harness
panics ("customFuncs' second argument must be a randfill.Continue").
Migrate the apps fuzzer the same way (randfill.Continue / FillNoCustom). With
no gofuzz consumers left, go mod tidy drops the dependency. TestRoundTripTypes
now passes.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Makes GPU passthrough work out-of-the-box for tenant Kubernetes clusters
(the `kubernetes` app), so operators no longer have to hand-fix it. Two
defaults:
1. **Tenant gpu-operator loads the nvidia driver with
`NVreg_NvLinkDisable=1`.**
2. **Nodes in GPU node-groups are labeled `gpu=on` automatically.**
### Root cause (validated empirically on a live cluster)
cozystack passes through individual SXM GPUs (e.g. H200) into tenant
worker VMs **without** the NVSwitches. Such a GPU's driver then waits
for an NVLink fabric that can never come up — there is no NVSwitch and
fabricmanager has nothing to manage — so `nvidia-smi` reports `Fabric
State: In Progress` indefinitely. As a result every CUDA call fails with
`system not yet initialized`, `nvidia-operator-validator`
(cuda-validation) crashloops, and no GPUs are advertised.
Loading the kernel module with `NVreg_NvLinkDisable=1` flips `Fabric
State` to `N/A`, CUDA initializes, and the validators pass. This is the
correct setting for per-GPU passthrough because there is no in-VM
NVLink/NVSwitch peering — the GPU is alone in the VM, so there is
nothing to disable that the workload needs.
The second issue: HAMi's `hami-device-plugin` DaemonSet has
`nodeSelector: gpu=on` and stays at `DESIRED=0` because GPU worker nodes
lack that label, so `nvidia.com/gpu` is never advertised. Labeling the
GPU nodes `gpu=on` lets the device plugin schedule and advertise the
GPUs.
### How it works
- **`gpu=on` label:** node-groups in
`packages/apps/kubernetes/templates/cluster.yaml` that declare `gpus`
get `--node-labels=gpu=on` appended to their kubelet args (in the
`KubeadmConfigTemplate` `joinConfiguration`). Non-GPU node-groups are
untouched.
- **`NVreg_NvLinkDisable=1`:** the upstream gpu-operator chart only
references an existing ConfigMap by name
(`driver.kernelModuleConfig.name`) and never creates it. The
gpu-operator wrapper chart (`packages/system/gpu-operator`) now renders
that ConfigMap (opt-in via `kernelModuleConfig.create`) so it deploys
into the tenant `cozy-gpu-operator` namespace alongside the operator.
The tenant gpu-operator HelmRelease defaults enable it and point
`driver.kernelModuleConfig.name` at it.
Both defaults remain overridable: a user setting
`addons.gpuOperator.valuesOverride` still wins via the existing
`mergeOverwrite(defaults, overrides)` in the HelmRelease template.
### Screenshots
Not applicable — no UI changes.
### Release note
```release-note
feat(kubernetes): enable GPU passthrough out-of-the-box for tenant clusters. GPU node-groups are now labeled `gpu=on` so HAMi's device plugin schedules and advertises `nvidia.com/gpu`, and the tenant gpu-operator loads the nvidia driver with `NVreg_NvLinkDisable=1` so single-SXM-GPU passthrough (no NVSwitch in the VM) no longer hangs on `Fabric State: In Progress` / `system not yet initialized`. Both defaults are overridable via `addons.gpuOperator.valuesOverride`.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* GPU-enabled node groups now get the expected GPU label during node
registration.
* GPU operator supports optional kernel-module parameter configuration
with a default parameter set.
* **Tests**
* Added tests verifying GPU node labeling is applied only for
GPU-enabled node groups.
* Added tests validating kernel-module config creation, naming, and
content behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
fuzzer.go imports sigs.k8s.io/randfill directly after the gofuzz-to-randfill
migration, but go.mod still listed it under the indirect block. go mod tidy
moves it into the direct require block; commit the result so the metadata
matches the import and a contributor running go mod tidy gets no diff.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
controller-gen cannot emit the x-cozystack-options vendor extension, so the
annotations on the backup CRDs (backupjobs/backups/plans/restorejobs) were
hand-edited into generated output — the next make generate would strip them,
breaking the appkind/plan/backup form dropdowns and failing the codegen-drift
check.
Add a deterministic awk pass that re-applies the extension after
controller-gen, driven by per-CRD field targets. It inserts lines without
reserializing the YAML, so it reproduces the committed CRDs byte-for-byte and
is independent of the host yq version.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The Option REST deliberately makes List tolerate a single failing provider
(it logs and skips it, so a partial RBAC config drops one dropdown instead
of breaking the whole list) while Get surfaces the error as a 500. Neither
path was covered. Add tests that inject a failing provider and assert List
keeps the healthy sources and omits the failing one, and that Get returns an
InternalError.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The Option providers RBAC block granted read on
objectstorage.k8s.io/bucketclasses, but no provider in the registry lists
BucketClasses and no schema field references a bucketclass source — the
bucket app's storage pool field uses the storagepool provider, which reads
seaweedfs HelmReleases. Drop the unused grant and pin its absence in the
chart test.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The plan and backup option providers list namespaced plans and backups in
the backups.cozystack.io group, but the cozystack-api ClusterRole only
granted backupclasses there. The apiserver's privileged client runs as this
ServiceAccount, so both providers got forbidden — List silently dropped the
sources and Get returned 500, leaving the plan/backup dropdowns in the
backup forms empty.
Add plans and backups to the backups.cozystack.io rule and cover the grant
with a chart test.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The appkind option provider queried applicationdefinitions in the
apps.cozystack.io group, but the ApplicationDefinition CRD is served from
cozystack.io. The provider listed a resource that does not exist, so the
appkind source resolved empty on every cluster (tolerated and dropped by
List, 500 on Get), and the backup form fields that reference it never
populated.
Point the provider at cozystack.io, which the existing cozystack.io read
grant already covers. Add a guard test pinning the CRD group, since the
existing fake-client test is built from the same GVR and cannot catch a
wrong group.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Regenerate values.schema.json and embedded openAPISchema in the -rd
ApplicationDefinitions so the x-cozystack-options dynamic-source
directives declared in each app values.yaml are carried into the
generated artifacts. Brings the -rd schemas in sync with the
hand-edited values.schema.json files (CI runs make generate +
git diff --exit-code).
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Options were granted on cozy:tenant:use:base, leaving view-only tenant
users (cozy:tenant:view) without read access — so their create/edit
forms could not populate dynamic dropdowns. Move the grant down to
cozy:tenant:view:base, which aggregates up into use/admin/super-admin,
covering every human tenant access level. The tenant ServiceAccount role
keeps its grant via cozy:tenant:base.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
The instanceProfile field is now driven by an x-cozystack-options dynamic
dropdown (source: instanceprofile), so the Makefile generate step no longer
needs to post-inject a static enum into values.schema.json via yq. Remove
that injection along with the already-commented-out instanceType enum
injection so the dynamic dropdown is not fought by a stale static enum.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Add the storageclass dynamic-dropdown x-cozystack-options annotation to the
storageClass field of the data apps (openbao, clickhouse, mariadb, redis,
harbor, opensearch, rabbitmq, mongodb, postgres, nats, kafka, qdrant,
foundationdb, http-cache) at the values.yaml source, so the extension is
emitted by cozyvalues-gen instead of hand-edited into values.schema.json.
kafka carries two (Kafka and ZooKeeper structs); foundationdb is nested
under the Storage struct.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Move the dynamic-dropdown x-cozystack-options vendor extensions from the
generated values.schema.json into the values.yaml source for vm-instance,
kubernetes, vm-disk and bucket so they survive schema regeneration. The new
cozyvalues-gen @x-<keyword> directive emits these extensions from the source
annotations. Annotations on typedef @field entries (Disk, Network, GPU,
NodeGroup) are declared once and cover every place the typedef is reused.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Add three dropdown sources to the Option provider registry so the
dashboard's backup forms can use the unified x-cozystack-options
mechanism instead of injecting client-side enum arrays:
- plan / backup: namespaced, list Plans / Backups in the tenant namespace
- appkind: cluster-scoped, lists application kinds from ApplicationDefinitions
Annotate the backups.cozystack.io CRD schemas (Plan, BackupJob, Backup,
RestoreJob) with x-cozystack-options on backupClassName (backupclass),
applicationRef.kind / targetApplicationRef.kind (appkind), planRef.name
(plan) and backupRef.name (backup). This also gives the previously
unused backupclass source its first consumer.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Mark fields that reference cluster resources with the x-cozystack-options
vendor keyword so the dashboard renders them as dynamic dropdowns:
GPU, instanceType, instanceProfile, network (vm-instance), GPU/
instanceType (kubernetes nodeGroups), image (vm-disk), storagePool
(bucket), storageClass (all apps) and disks[].name (vm-instance). Drop
the stale static instanceProfile enum.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Introduce a read-only, namespaced virtual resource
core.cozystack.io/v1alpha1 Option that serves named lists of dropdown
options for the dashboard. Each object (one per source: gpu,
instancetype, instanceprofile, network, image, storagepool,
storageclass, vmdisk, backupclass) is computed on read by an in-process
provider using a privileged dynamic client, so tenants need no direct
access to Nodes, the KubeVirt CR, cluster instancetypes, etc.
Tenants get read access via the aggregated cozy:tenant roles; the
apiserver ServiceAccount gains read on the underlying sources.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Aleksei Sviridkin
