## What this PR does
Adds
[`lexfrei/ouroboros@v0.7.0`](https://github.com/lexfrei/ouroboros/releases/tag/v0.7.0)
— a Go reimplementation of
[`compumike/hairpin-proxy`](https://github.com/compumike/hairpin-proxy)
— and wires it into cozystack as a *conditional default*: a single
`publishing.proxyProtocol: true` switch turns PROXY-protocol on at the
host ingress-nginx **and** auto-deploys ouroboros to fix the resulting
hairpin-NAT problem (intra-cluster traffic to the cluster's own public
hostnames arriving at ingress-nginx without the PROXY header it now
requires). Default stays `false` — clusters that do not run
PROXY-protocol get zero new resources, zero RBAC delta, zero behaviour
change.
The chart is platform-agnostic at v0.7.0: the controller and proxy
binaries auto-detect the cluster's DNS domain from `/etc/resolv.conf`
inside the pod and compose the rewrite target FQDN and the backend host
at runtime as `<svc>.<ns>.svc.<cluster-domain>`. Cozystack tenants on
`cozy.local`, kubeadm-bootstrapped clusters on `cluster.local`,
federations on `k8s.example.com`, custom-domain RKE2/k3s — all work
without operator override. Verified end-to-end on a `cozy.local`
cluster: controller logs `cluster-domain=cozy.local` at startup, chart
emits `--proxy-service-name + --proxy-service-namespace` (no chart-baked
`--proxy-fqdn`), Corefile picks up the rewrite block between BEGIN/END
markers, in-pod `dig <ingress-host>` returns the ouroboros-proxy
ClusterIP. Operators that prefer a chart-baked FQDN can still set
`controller.clusterDomain` explicitly — the explicit-override path is
preserved for non-Service backends and for split-horizon DNS topologies.
The chart runs in `coredns` mode for the host (the host CoreDNS is
Talos-managed and does not carry an `import` directive, so the
alternative would silently no-op there) and in `coredns-import` mode for
tenant deployments (tenants run cozystack-coredns, which carries the
import directive and ships an empty `kube-system/coredns-custom`
ConfigMap; ouroboros writes plugin-only `rewrite name` snippets into the
`ouroboros.override` data key without ever touching the chart-rendered
Corefile, so it does not race the chart that re-templates
`data.Corefile` on every Flux reconcile).
### Layout
* `packages/system/ouroboros/` — vendored chart from
`oci://ghcr.io/lexfrei/charts/ouroboros@0.7.0`. Cozystack-side defaults
pinned in `values.yaml`: `controller.mode=coredns` (host scenario),
`proxy.target.namespace=cozy-ingress-nginx` (override the chart-default
`ingress-nginx` namespace; chart-default
`serviceName=ingress-nginx-controller` matches as-is), Gateway-API
watching left at upstream's default of `false`. `proxy.target.host` is
left empty so the proxy composes the backend host at runtime. Image is
digest-pinned at
`0.7.0@sha256:478665e05dd0ffc4c1f7764320ea24b52251781884ddfd668d93a03e39d9094c`
and pulled directly from `ghcr.io/lexfrei/ouroboros` — this package is
intentionally not mirrored under `ghcr.io/cozystack/*`. The chart
manifest itself is also digest-pinned via
`OUROBOROS_CHART_DIGEST=sha256:f28bfac9fd7070b1f3357983ff5aad28c16115e7de1cfc16ee568a3b4cfc9d7e`
in the package Makefile, with a Chart.yaml-version vs. Makefile-version
assert that aborts the update target on lockstep drift. Helm-unittest in
`tests/` covers controller args, RBAC narrowing per mode, Gateway-API
gating, runtime-composition flag emission in both `coredns` and
`coredns-import` modes, image pin drift.
* `packages/system/coredns/templates/coredns-custom.yaml` +
`values.yaml` — empty `coredns-custom` ConfigMap as the chart-managed
`import` prerequisite, plus the `import /etc/coredns/custom/*.override`
plugin in the `.:53` server block + `extraVolumes`/`extraVolumeMounts`
mount at `/etc/coredns/custom/`. Same package serves host and tenant
CoreDNS, so this covers both layers without duplication. Helm-unittest
pins the rendered Corefile via a snapshot so future `make update` does
not silently drop the import directive. The empty rendered ConfigMap
deliberately omits `data:` — Helm three-way merge has no template-side
keys to compare against, so apiserver-side runtime writes (the
`ouroboros.override` data key) survive every chart upgrade. The
`notExists: data` assertion in `tests/coredns_custom_test.yaml` pins the
invariant.
* `packages/core/platform/sources/ouroboros.yaml` — new
`cozystack.ouroboros` PackageSource for the host install (`dependsOn:
cozystack.networking, cozystack.ingress-application`).
* `packages/core/platform/sources/kubernetes-application.yaml` —
`kubernetes-ouroboros` component registered against `system/ouroboros`
so the tenant HelmRelease can reference it.
* `packages/apps/kubernetes/templates/helmreleases/ouroboros.yaml` —
tenant-side HelmRelease deployed via the tenant admin-kubeconfig,
mirroring the cert-manager pattern. A `cozystack.defaultOuroborosValues`
helper flips `mode` back to `coredns-import` for tenants and mirrors
per-tenant `addons.gatewayAPI.enabled` into the controller (forcing
`--gateway-api` on a tenant without Gateway-API CRDs CrashLoops the
controller on `WaitForCacheSync`). `controller.clusterDomain` is
intentionally NOT pre-set — chart 0.7.0 auto-detects per-pod from
`/etc/resolv.conf` and composes the rewrite target at runtime, so the
same wrapper renders correctly on `cluster.local`, `cozy.local`, or any
custom tenant cluster-domain. Operators on a non-default cluster-domain
can still override via
`valuesOverride.ouroboros.controller.clusterDomain`. `dependsOn` grows a
conditional `gateway-api-crds` entry on the same gate. If
`addons.ouroboros.enabled` is true while `addons.ingressNginx.enabled`
is false, the HelmRelease fails the render with a clear error rather
than silently skipping — the hairpin fix has nothing to fix against
without ingress-nginx, and a no-op addon that quietly does nothing is
harder to debug than an explicit render-time error. The valuesOverride
gateway-api guard accepts every truthy spelling Helm/YAML emit (bool
`true`, string `"true"`/`"True"`/`"TRUE"`, numeric `1`, YAML
`yes`/`Yes`/`YES`/`on`/`On`/`ON`/`y`/`Y`) via
`toString`-then-`lower`-then-string-compare; direct bool `eq` would
error on type mismatch when an operator passes the value as a string
from `--set`, and case-insensitive folding catches `--set ...=On` /
`--set ...=True` that pure-lowercase comparison would miss. Tenant
disable is single-step: flipping `addons.ouroboros.enabled: false`
deletes the rendered HelmRelease, helm-controller runs `helm uninstall`
on the tenant, and the chart's pre-delete cleanup hook
(`coredns-cleanup-hook.yaml`, vendored at 0.7.0) nulls the
`ouroboros.override` key on the tenant's `kube-system/coredns-custom`
ConfigMap automatically. No render-time guard is wired on the tenant
side — gating on a `lookup` for the leftover HelmRelease would deadlock
the routine disable (the parent render runs before helm-controller
applies the missing-child diff, so the lookup would always find the
leftover at the moment of the flag flip). If the chart's hook fails to
land (controller pod stuck CrashLooping, ConfigMap RBAC drift, Job
timeout), operators recover manually via the documented tenant cleanup
recipe.
* `packages/apps/kubernetes/values.yaml` — new `OuroborosAddon` typedef
with `enabled` + `valuesOverride`. `make generate` re-emitted README,
JSON schema, the Go struct + deepcopy, and the kubernetes-rd
ApplicationDefinition openAPISchema in lockstep.
* `packages/core/platform/values.yaml` — new `publishing.proxyProtocol`
flag with extensive comment covering the upstream-LB precondition (the
L4 LB in front of ingress-nginx must already be injecting PROXY-v1
headers before the flag flips on, otherwise external traffic breaks),
the host-vs-tenant fan-out (the flag does not cascade into tenants), the
Talos re-render flap window, and the asymmetric disable path. The host
disable path: flipping back to `false` reverts ingress-nginx
PROXY-protocol config (flip the LB off first), but the host ouroboros
itself stays installed because its Package CR carries
`helm.sh/resource-policy: keep` — an explicit `kubectl delete
package.cozystack.io cozystack.ouroboros` (or `bundles.disabledPackages`
entry) actually triggers helm uninstall, at which point the chart's
pre-delete cleanup hook fires and patches the host Corefile
automatically. The values comment ships a manual `sed`-based cleanup
recipe as a fallback for the rare case where the chart's hook fails to
land (controller stuck in CrashLoop, ConfigMap RBAC drift, hook
timeout); the `publishing.proxyProtocolAcknowledgeUnclean` gate keys off
a `lookup` against the leftover Package CR to catch operators who flip
the flag back without confirming the cleanup landed.
* `packages/core/platform/templates/bundles/system.yaml` — flips between
the default-form and explicit-form `cozystack.ingress-application`
Package emission so the host ingress-nginx gets `use-proxy-protocol` +
`real-ip-header=proxy_protocol` injected only when the flag is true
(`use-forwarded-headers` and `compute-full-forwarded-for` are
deliberately *not* on the list — they would let any upstream proxy spoof
`X-Forwarded-For`); the same conditional emits `cozystack.ouroboros`.
Host Gateway-API watching is left at the upstream chart's default of
`false` because cozystack does not install Gateway-API CRDs cluster-wide
on the host.
* `hack/e2e-apps/ouroboros.bats` — auto-discovered smoke test for the
tenant addon (HelmRelease ready, `coredns-custom` ConfigMap present in
tenant, controller pod running, and after creating a TLS Ingress the
rewrite snippet shows up in the import ConfigMap; an in-tenant `dig`
from a netshoot pod also confirms CoreDNS actually serves the rewrite —
pod-readiness alone passes vacuously since the upstream chart ships no
readiness probes).
### Why not KEP-1860 `ipMode=Proxy`
Cilium's `kubeProxyReplacement: true` (cozystack default) drops the LB
frontend entirely when `ipMode != VIP`, breaking Service ingress for
L2/BGP-announced IPs. KEP-1860's Proxy mode is reserved for clusters
fronted by an external CCM-managed proxy LB. ouroboros is the correct
fix for L2/BGP-announce topologies that cozystack ships by default.
### Coverage
Covered by helm-unittest (host chart, coredns wrapper, tenant addon —
including the `kubernetesProvider` mock that exercises the lookup-driven
`proxyProtocolAcknowledgeUnclean` gate, the lookup-returns-nil
CI/dry-run path, the gateway-api truthy-spelling acceptance across
`true`/`"true"`/`"True"`/`"TRUE"`/`1`/`yes`/`on`, the assertion that
`--proxy-service-name` + `--proxy-service-namespace` are emitted instead
of a chart-baked `--proxy-fqdn` in both `coredns` and `coredns-import`
modes, the assertion that `--target-service-name` +
`--target-service-namespace` are emitted for the proxy backend, and the
`_namespace.etcd`-empty render gate) and a tenant e2e bats test that
creates a real Ingress, asserts the rewrite snippet lands in the import
ConfigMap, and confirms in-tenant DNS serves it (with an in-pod retry
loop tolerating CoreDNS reload latency). The host-level
`publishing.proxyProtocol: true` path is not e2e-tested in this PR — it
requires a PROXY-protocol-aware upstream LB, which the cozystack e2e
harness does not stand up. That path is exercised manually and by docs
review.
End-to-end runtime auto-detect was verified manually on a `cozy.local`
cluster (dev17): chart deployed at v0.7.0, controller logged
`cluster-domain=cozy.local`, the rewrite block landed in
`kube-system/coredns` Corefile pointing at
`test-ouroboros-proxy.test-ouroboros.svc.cozy.local.`, an in-pod `dig
<ingress-host>` resolved to the ouroboros-proxy ClusterIP on the first
attempt, and the chart's pre-delete hook restored the Corefile cleanly
on `helm uninstall`.
### Supply-chain note
Image and chart are pulled from a contributor's personal namespace
(`ghcr.io/lexfrei/...`). Existing precedent
(`packages/system/etcd-operator/Makefile` pulls from
`ghcr.io/aenix-io/...`) is similar but `aenix-io` is the
cozystack-adjacent org while `lexfrei` is one human. Worth a maintainer
call.
Pairs with cozystack/website#527 (operator docs).
### Release note
```release-note
feat(networking): add `publishing.proxyProtocol: true` to enable PROXY-protocol on host ingress-nginx and auto-deploy ouroboros (a Go reimplementation of compumike/hairpin-proxy) to solve the resulting hairpin-NAT problem; tenants can opt in per-cluster via `addons.ouroboros.enabled` (requires `addons.ingressNginx.enabled`). Default behaviour unchanged.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added Ouroboros addon to address hairpin‑NAT for ingress‑nginx with
configurable enablement and tenant overrides.
* Added a packaged Ouroboros chart providing proxy, controller, CoreDNS
integration, external‑DNS and etc‑hosts modes.
* Introduced PROXY‑protocol gating with an explicit acknowledgement
flag.
* **Documentation**
* Expanded addon reference and platform docs with detailed
Ouroboros/PROXY‑protocol configuration and defaults.
* **Tests**
* Added extensive unit and e2e tests covering rendering, RBAC, uninstall
cleanup, and DNS behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Updates the NVIDIA GPU Operator from v25.3.0 to v26.3.1 and adds an
**experimental** `vgpu` variant for SR-IOV vGPU on Ada Lovelace and
newer GPUs.
Changes:
- Update vendored upstream chart to v26.3.1 (CDI enabled by default,
component version bumps).
- Pin chart version in Makefile for reproducibility.
- Add `values-vgpu.yaml` for the vGPU variant with `defaultWorkload:
vm-vgpu` so the chart activates the vGPU Manager DaemonSet without
per-node labelling.
- Add `vgpu` variant to PackageSource alongside the existing `default`
(passthrough).
- Rename `values-talos.yaml` → `values-passthrough.yaml` and pin
`defaultWorkload: vm-passthrough` for symmetry with the vgpu variant.
The overrides apply to any host OS using vfio-pci, not Talos
specifically.
- Refresh `hack/dcgm-default-counters.csv` to match the new
dcgm-exporter image tag (`4.5.1-4.8.0`) — adds `DCGM_FI_DEV_FB_RESERVED`
and other fields introduced upstream.
`hack/check-gpu-recording-rules.bats` passes against the new fixture.
- Add `hack/check-gpu-operator-variants.bats` that renders both variants
via `helm template` and asserts the pinned ClusterPolicy values, so the
next chart bump cannot silently un-pin them.
- Add internal documentation for the vGPU bring-up.
The `vgpu` variant assumes the user supplies a proprietary vGPU Manager
image via Package CR component values — the .run cannot be
redistributed.
### `vgpu` variant is experimental
NVIDIA's `vgpu-device-manager` walks `/sys/class/mdev_bus/`, which does
not exist on Ada Lovelace and Blackwell — the DaemonSet errors with "no
parent devices found" and is disabled by default. Until an SR-IOV-aware
controller is shipped, profile assignment is an out-of-band step (`echo
<id> > /sys/.../current_vgpu_type` per VF) that must be re-applied after
every node reboot. Without it, `permittedHostDevices.pciHostDevices`
reports zero allocatable resources. The variant ships so the chart bump
and the underlying skeleton (DaemonSet + image plumbing + docs) land
together; production deployments need an automated profile-assignment
mechanism in addition to this PR.
### KubeVirt requirement
KubeVirt advertises VFs via `permittedHostDevices.pciHostDevices` rather
than `mediatedDevices`. This requires kubevirt/kubevirt#16890 ("vGPU:
SRIOV support", merged 2026-04-10) which is `main`-only — released tags
`v1.6.x` / `v1.7.x` / `v1.8.x` do not include it. The patch is targeted
at the next minor KubeVirt release (v1.9.0); track the PR for the actual
release tag. Backports to released branches are not planned.
### Behavior changes inherited from upstream chart v25 → v26
The upstream chart flipped two defaults that are pinned back to `false`
here, so `default` (passthrough) deployments do not silently change
footprint on upgrade:
- `ccManager.enabled` was flipped to `true` upstream — would auto-enable
Confidential Computing on Hopper hardware (H100/H200/B100) after
upgrade.
- `vgpuDeviceManager.enabled` was flipped to `true` upstream — DaemonSet
would render even in passthrough mode.
Both stay opt-in via Package CR overrides. The new bats test guards the
pin.
Other inherited changes that are user-visible and worth pinning down
before upgrade:
- `driver.licensingConfig.configMapName` is deprecated upstream in
favour of `driver.licensingConfig.secretName`. The deprecation marker
lives in the CRD's `description` field (visible via `kubectl explain`) —
there is no helm-render-time warning, the chart template silently
prefers `secretName` and falls back to `configMapName`. Operators with
existing licensing overrides should migrate the key on the next chart
upgrade.
- `driver.version` upstream default jumped from `570.x` to `580.x`.
Invisible on the `default` variant (which sets `driver.enabled: false`),
user-visible for anyone running the upstream Ubuntu driver path.
- `vfioManager.image` switched from
`nvcr.io/nvidia/cuda:12.8.1-base-ubi9` to
`nvcr.io/nvidia/cloud-native/k8s-driver-manager:v0.10.0`. Affects every
`default` (passthrough) deployment on upgrade — different image,
different layer, different image-pull surface. Worth pre-mirroring for
air-gapped clusters.
- `dcgmExporter.image` switched to the `-distroless` flavour. Smaller
surface, fewer base-image CVEs, but no shell — operators who used to
`kubectl exec` into the exporter for ad-hoc debugging will need to
switch to `kubectl debug` or container ephemeral images.
- `CDI` (Container Device Interface) is enabled by default in v26.3.1;
this is upstream's recommended setting and is left as-is.
- `node-feature-discovery` subchart source switched from the GitHub
Pages Helm repo to `oci://registry.k8s.io/nfd/charts`. Air-gapped
installations need to mirror the new OCI URL.
### Talos
Talos is **not recommended** for the vGPU variant. NVIDIA does not
publicly distribute the vGPU guest driver — it requires NVIDIA
Enterprise Portal access — and Sidero closedsiderolabs/extensions#461
noting that they cannot support vGPU "unless NVIDIA changes their
licensing terms or provides us a way to obtain, test, and distribute the
software". Building a Talos system extension that bakes in the vGPU
driver is therefore not feasible without a private fork that violates
the EULA. Passthrough (`default` variant) on Talos is unaffected.
### Release note
```release-note
[gpu-operator] Update NVIDIA GPU Operator to v26.3.1 and add experimental `vgpu` variant for SR-IOV vGPU on Ada Lovelace / Blackwell (L4, L40, L40S, B100; Ubuntu host; requires KubeVirt v1.9.0+ for `pciHostDevices` advertising). The variant ships skeleton support — profile assignment (`current_vgpu_type` per VF) is currently out of band and must be automated by the operator before production use. On the existing `default` variant: `ccManager` and `vgpuDeviceManager` upstream defaults are pinned back to `false` to preserve prior behavior on chart upgrade; `defaultWorkload` is now pinned to `vm-passthrough` (was unset, falling back to `container`); `driver.licensingConfig.configMapName` is deprecated by upstream in favour of `secretName`; `driver.version` upstream default moves 570 → 580; `vfioManager.image` switches from CUDA UBI9 to k8s-driver-manager (relevant for air-gapped mirrors); `dcgmExporter` image switched to the `-distroless` flavour. NFD subchart source switched to `oci://registry.k8s.io/nfd/charts`.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Introduced vGPU support variant with end-to-end configuration guide,
validation procedures, and troubleshooting documentation
* **Updates**
* Upgraded GPU Operator to v26.3.1 and Node Feature Discovery to v0.18.3
* Improved component health checks and added PodDisruptionBudget
availability controls
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Adds publishing.proxyProtocol on the platform values: a single flag
that both turns PROXY-protocol on at the host ingress-nginx (via the
ingress-application Package CR's component values dict) and emits the
new cozystack.ouroboros Package CR for the host install. The two
pieces are coupled by design — turning PROXY-protocol on at
ingress-nginx without ouroboros breaks intra-cluster traffic to the
cluster's own public hostnames (kube-proxy / Cilium-KPR short-circuits
the LB IP, the connection arrives at ingress-nginx without the PROXY
header it now requires, and ingress-nginx closes the connection).
The new sources/ouroboros.yaml PackageSource declares the host chart
with dependsOn: cozystack.networking + cozystack.ingress-application
so install ordering is deterministic. The bundle template emits the
host ouroboros Package via the standard helper — the parent chart's
Gateway-API watching default of false stays in place, because cozystack
does not install Gateway-API CRDs cluster-wide on the host (system/
gateway-api-crds is referenced only from the kubernetes-application
PackageSource, tenant scope). Forcing --gateway-api on a host without
the CRDs would CrashLoop the controller on WaitForCacheSync against
missing GVRs.
The flag default is false. Clusters that do not run PROXY-protocol get
zero new resources, zero RBAC delta, zero behaviour change.
Disabling: flipping publishing.proxyProtocol back to false stops
emitting the cozystack.ouroboros Package CR, but every Package CR
carries helm.sh/resource-policy: keep — Helm leaves the existing
Package on the cluster so ouroboros stays installed and the live
Corefile rewrite block keeps pointing at the still-running
ouroboros-proxy Service. Flag-flip alone is idempotent and safe. To
actually uninstall ouroboros the operator removes the Package CR
explicitly (or adds the package name to bundles.disabledPackages) and
runs the documented Corefile cleanup recipe in the same window —
otherwise stale rewrites point at a deleted Service and in-cluster
DNS for hairpinned hostnames returns connection-refused. The values
comment ships the cleanup recipe (a JSON merge-patch on data.Corefile
that strips the BEGIN/END block while preserving labels, annotations,
and other data keys including any Talos-managed metadata) and
documents the upstream-LB precondition (the L4 LB in front of
ingress-nginx must already be injecting PROXY-v1 headers before the
flag flips on), the host-vs-tenant fan-out (the flag does not cascade
into tenant clusters; tenants opt in per-Kubernetes-CR via
addons.ouroboros.enabled), and the Talos re-render flap window
(machine-config pushes wipe the BEGIN/END markers; the controller
re-applies on next reconcile, ≤10 minutes by default).
Also drops the stale "distro-full" entry from the bundles.system.variant
options comment in core/platform/values.yaml — the variant was removed
in v1.0.1 and was never restored.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Adds the OuroborosAddon typedef, addons.ouroboros knob, and tenant
HelmRelease that deploys the cozy-ouroboros chart inside a tenant
Kubernetes cluster via the host's flux-helm-controller and the tenant
admin-kubeconfig — same shape as the cert-manager addon.
The HelmRelease overrides controller.mode back to coredns-import
because tenants run the cozystack-coredns wrapper (which carries the
`import /etc/coredns/custom/*.override` directive); the parent chart's
default of mode=coredns is the right shape only for the Talos-managed
host CoreDNS. controller.gatewayApi.enabled is mirrored from the
per-tenant addons.gatewayAPI.enabled — forcing --gateway-api on a
tenant without Gateway-API CRDs CrashLoops the controller on
WaitForCacheSync against missing GVRs. dependsOn grows a conditional
gateway-api-crds entry on the same gate.
controller.clusterDomain is intentionally NOT pre-set by the wrapper.
The chart auto-detects it inside the controller pod by parsing
/etc/resolv.conf — the search list of which is `*.<ns>.svc.<domain>`
and reflects exactly the zone the tenant CoreDNS actually serves
(governed by KamajiControlPlane.spec.networkProfile.clusterDomain
defaulting to `cluster.local`, and the cozystack-coredns wrapper
kubernetes-plugin zone, also `cluster.local`). Pre-setting from a
host-side fact like `_cluster.cluster-domain` would override the
auto-detect with a value that does not match what the tenant kube-dns
serves, silently emitting NXDOMAIN-bound rewrites. Operators that run
a tenant on a non-default cluster-domain can still override via
addons.ouroboros.valuesOverride.ouroboros.controller.clusterDomain.
If addons.ouroboros.enabled is true but ingressNginx is off, the
HelmRelease fails the render with a clear error rather than silently
skipping — the hairpin fix has nothing to fix against without
ingress-nginx, and a no-op addon that quietly does nothing is harder
to debug than an explicit render-time error.
The valuesOverride gateway-api guard accepts every truthy spelling
Helm/YAML emit (bool true, string "true"/"True"/"TRUE", numeric 1,
YAML "yes"/"Yes"/"YES"/"on"/"On"/"ON"/"y"/"Y") via
toString-then-lower-then-string-compare; direct bool eq would error
on type mismatch when an operator passes the value as a string from
--set, and case-insensitive folding catches `--set ...=On` /
`--set ...=True` that pure-lowercase comparison would miss.
The acknowledgeUncleanRemoval gate is defence-in-depth for the rare
edge case where helm uninstall fails to land cleanly (tenant
unreachable, hook timeout, manual kubectl delete bypassing helm) —
in the normal path the upstream chart's pre-delete cleanup hook
(coredns-cleanup-hook.yaml, vendored at 0.6.0) nulls the
ouroboros.override key automatically when the HelmRelease is
uninstalled. The gate only blocks render when a leftover HelmRelease
on the host kube-apiserver shows the previous install is still
sitting on the cluster.
The component is also registered in the kubernetes-application
PackageSource (as kubernetes-ouroboros at path system/ouroboros) so
the tenant HelmRelease's ExternalArtifact resolves.
make generate re-emitted README.md, values.schema.json, the Go struct
+ deepcopy, and the kubernetes-rd ApplicationDefinition openAPISchema
in lockstep.
helm-unittest covers: render gating on enabled flag, hard-fail when
ingress-nginx is off, namespaces and labels, chartRef, dependsOn
cilium/coredns/ingress-nginx + conditional gateway-api-crds, default
coredns-import mode, valuesOverride precedence, gateway-api flag
mirroring in both directions, gateway-api truthy-spelling acceptance
(including case-insensitive on/On/True/TRUE), the lookup-driven
acknowledge gate (kubernetesProvider mock asserts the failedTemplate
path when a prior HelmRelease still exists on the host), the
lookup-returns-nil CI/dry-run path (render proceeds when the cluster
mock has no objects), the _namespace.etcd-empty render gate, and
that controller.clusterDomain is NOT pre-set in the rendered
HelmRelease (so chart auto-detect runs).
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Bumps the vendored `etcd-operator` chart from `v0.4.2` to `v0.4.3` and
wires the new backup machinery through the cozystack `Etcd` app API.
Upstream v0.4.3 ships two new CRDs — `EtcdBackup` (one-shot snapshot)
and `EtcdBackupSchedule` (periodic snapshots) — along with the
`EtcdCluster.spec.bootstrap` restore field, a new `OPERATOR_IMAGE` env
on the operator deployment, and a replacement of the deprecated
`gcr.io/kubebuilder/kube-rbac-proxy` image. The chart portion of this PR
is a clean re-vendor via `make update` — no local patches.
On top of the bump, `packages/extra/etcd` exposes a new optional
`backup.*` values block. When `backup.enabled=true`, the chart renders
an `EtcdBackupSchedule` referencing the platform `EtcdCluster/etcd` and
a `Secret/etcd-s3-creds` with the S3 credentials expected by the
backup-agent. The values field names (`destinationPath`, `endpointURL`,
`s3AccessKey`, `s3SecretKey`, `schedule`) mirror the existing `postgres`
backup pattern for consistency. The chart `fail`s at template render if
any of the four required S3 fields are empty or if `destinationPath` is
malformed (missing `s3://` prefix, `//` segments, no bucket).
Test coverage: 15 `helm-unittest` cases across
`tests/backup-secret_test.yaml` and
`tests/etcd-backup-schedule_test.yaml` (wired into `make test`, picked
up by `hack/helm-unit-tests.sh`), plus three bats cases in
`hack/e2e-apps/etcd.bats` covering the baseline `Etcd` app, the `backup:
{}` default-off path, and the full `backup.enabled=true` case that
verifies the `EtcdBackupSchedule` materializes and the operator
generates the downstream `CronJob`.
`packages/core/platform/sources/etcd-operator.yaml` now sets
`upgradeCRDs: CreateReplace` on the etcd-operator component, so Flux
applies the new CRDs on upgrade of the chart instead of silently
skipping them.
Depends on #2427.
### Scope deferred to follow-ups
- The one-shot `EtcdBackup` CRD is vendored but not exposed through this
chart — users can still hand-apply the CRD directly.
- `EtcdCluster.spec.bootstrap` (restore from an existing snapshot) is
likewise vendored but not exposed. Both gaps are called out in
`packages/extra/etcd/README.md`.
### Release note
```release-note
feat(etcd): update etcd-operator to v0.4.3 and expose S3 backup schedules on the Etcd app. Set `backup.enabled=true` along with `backup.s3AccessKey`, `backup.s3SecretKey`, `backup.destinationPath`, and `backup.endpointURL` to schedule periodic snapshots.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Scheduled S3 backups for etcd clusters (configurable cron, generated
S3 credentials) and restore/init from S3 or PVC via new backup
resources.
* **Tests**
* New end-to-end tests for etcd lifecycle and backup scenarios; added
Helm unit tests for backup templates and validations.
* **Documentation**
* Backup configuration guide and parameter reference added.
* **Chores**
* Operator chart version bump; RBAC, webhooks, and deployment wiring
updated; added chart test target.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Addresses second review pass. Three blockers + one polish:
* values-vgpu.yaml: set sandboxWorkloads.defaultWorkload=vm-vgpu.
Without this the upstream default 'container' applies on any node
without the per-node nvidia.com/gpu.workload.config label, the
operator selects the container workload (driver/devicePlugin
DaemonSets), and those are disabled in this variant — so nothing
runs on the node and vgpu-manager never starts. The cluster-wide
default makes the variant functional out of the box; per-node
labels still override it for mixed clusters.
* docs/gpu-vgpu.md: imagePullSecrets is per-component on this chart
(vgpuManager.imagePullSecrets, driver.imagePullSecrets, …). The
Package CR example previously placed the field at gpu-operator: root
where it would be silently ignored at render. Move it under
vgpuManager: where the chart actually reads it.
* docs/gpu-vgpu.md, release note: mark the vgpu variant as
experimental and call out the missing profile-assignment loop
explicitly. Until an SR-IOV-aware controller writes
current_vgpu_type per VF (and re-applies after every reboot —
PCIe re-enumeration resets the value), the variant is not safe
for production. The docs now lead with that warning instead of
burying it in a caveat.
* Rename packages/system/gpu-operator/values-talos.yaml to
values-passthrough.yaml. The overrides (sandboxWorkloads.enabled,
driver and devicePlugin disabled) apply to any host OS using the
vfio-pci passthrough path — not Talos specifically. Update
PackageSource ref accordingly. Comment in the file documents the
rename so future readers understand the prior name.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Rewrites docs/gpu-vgpu.md to match the actual NVIDIA driver model on
current data-centre GPUs. Key changes:
- Drop the implicit assumption that vGPU uses mediated devices. Ada
Lovelace and newer (L4, L40, L40S, B100) on the vGPU 17/20 branch
use SR-IOV with per-VF NVIDIA sysfs (current_vgpu_type); KubeVirt
advertises them via permittedHostDevices.pciHostDevices.
- Document KubeVirt PR #16890 as a hard dependency for the SR-IOV
path. The PR is in main since 2026-04-10 and will ship in v1.9.0
(ETA July 2026); released tags v1.6.x/1.7.x/1.8.x do not include
it and backports are not planned.
- Replace the legacy NLS configuration block (ServerAddress=,
ServerPort=7070, FeatureType=1) with the current DLS flow:
ClientConfigToken file in /etc/nvidia/ClientConfigToken/, gridd.conf
with FeatureType=0 (auto-detect).
- Switch the build instructions from the archived
gitlab.com/nvidia/container-images/driver to the upstream
github.com/NVIDIA/gpu-driver-container.
- Switch sample VirtualMachine spec from gpus: (mdev resource) to
hostDevices: (PCI resource).
- Note explicitly that Talos is not recommended for the vGPU variant
— the proprietary .run cannot be redistributed in a Talos system
extension, and siderolabs/extensions#461 was closed won't-fix.
Passthrough on Talos is unaffected.
Rename packages/system/gpu-operator/values-talos-vgpu.yaml to
values-vgpu.yaml — the overrides (sandbox workloads, pod-side driver
and device plugin disabled) are not Talos-specific, they apply to
any VM-only GPU node. Update PackageSource accordingly.
Default vgpuDeviceManager.enabled: false. The DaemonSet walks
/sys/class/mdev_bus/, which does not exist on Ada+ — keeping it on
by default would crashloop modern hardware. Operators of older GPUs
(Pascal/Volta/Turing/Ampere) can flip the flag back on.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add vGPU mode as a new variant alongside the existing passthrough (default) variant.
This enables sharing a single physical GPU across multiple VMs using NVIDIA mediated devices.
Changes:
- Add values-talos-vgpu.yaml with vGPU Manager and vGPU Device Manager enabled
- Add 'vgpu' variant to PackageSource
- Add documentation for vGPU setup (driver image, NLS licensing, profiles, KubeVirt)
Users select the vgpu variant via Package CR and provide their proprietary
vGPU Manager image coordinates through component values.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The HelmRelease at packages/apps/kubernetes/templates/helmreleases/hami.yaml
references chartRef.name 'cozystack-kubernetes-application-kubevirt-kubernetes-hami',
but that component was missing from the kubernetes-application PackageSource.
The HelmRelease would sit in Stalled: ArtifactNotFound at install time.
Add kubernetes-hami next to kubernetes-gpu-operator under variant kubevirt,
mirroring the existing pattern for every other tenant-cluster addon. The
standalone cozystack.hami PackageSource is retained — same shape as
gpu-operator, which is registered both standalone (for bundles/iaas) and
as a kubernetes-application component (for tenant-cluster HelmReleases).
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Add PackageSource for HAMi with dependency on gpu-operator.
Include HAMi in the iaas bundle as an optional package, allowing
host-level deployment alongside the existing kubernetes addon path.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
v0.4.3 introduced new CRDs (EtcdBackup, EtcdBackupSchedule). Flux defaults to Skip on upgrade, so on existing clusters those CRDs never land without a manual kubectl apply.
Setting upgradeCRDs: CreateReplace on the etcd-operator component tells PackageReconciler to wire hr.Spec.Upgrade.Crds accordingly. etcd-operator's CRDs evolve additively between versions, so CreateReplace is safe here.
Depends on the PackageSource.Spec.Variants[].Components[].Install.UpgradeCRDs field — must ship with a cozystack-operator release that recognizes it. Without that, this field is ignored and the HelmRelease still defaults to Skip.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
<!-- Thank you for making a contribution! Here are some tips for you:
- Start the PR title with the [label] of Cozystack component:
- For system components: [platform], [system], [linstor], [cilium],
[kube-ovn], [dashboard], [cluster-api], etc.
- For managed apps: [apps], [tenant], [kubernetes], [postgres],
[virtual-machine] etc.
- For development and maintenance: [tests], [ci], [docs], [maintenance].
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same [label] as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
[vm-default-images] Added package that brings set of images that can be used clusterwide
[vm-disk] Updated source "image" for prettier dropdown selection
[vm-disk] Added new source for vm-disk called disk - to use as source vm-disk from same namespace.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Clone VM disks by specifying an existing vm-disk as the source.
* Global default image collection and chart to publish pre-provisioned
images.
* **UI**
* Forms provide selectable lists for default images and existing VM
disks.
* **Migration**
* Migration to rename existing image DataVolumes to the new
default-images naming and bumped migration version.
* **Documentation**
* VM disk docs and README updated to reflect image sourcing and cloning.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Adds a new Cozystack system package `linstor-gui` that ships [LINBIT's
LINSTOR web UI](https://github.com/LINBIT/linstor-gui) (GPL-3.0) so
operators can manage nodes, resources, volumes and snapshots from a
browser instead of the `linstor` CLI.
Changes:
- `packages/system/linstor-gui/` — umbrella chart with:
- `images/linstor-gui/Dockerfile` builds the image from the upstream
`pkg.linbit.com` tarball (v2.3.0) on top of
`nginxinc/nginx-unprivileged:1.29-alpine`, mirroring the build pattern
already used for `piraeus-server` and `linstor-csi`.
- `templates/configmap-nginx.yaml` — chart-supplied `nginx.conf` that
proxies `/v1` and `/metrics` to
`linstor-controller.cozy-linstor.svc:3371` over **mTLS**, using the
existing `linstor-client-tls` secret created by the `linstor` package.
- `templates/deployment.yaml` — read-only rootfs, non-root (UID 101),
`runAsNonRoot`, `RuntimeDefault` seccomp, no SA token mounted,
`reloader.stakater.com/auto` on the Deployment.
- `templates/service.yaml` — **ClusterIP only**. No Ingress is shipped
because the LINSTOR controller API is a privileged cluster-wide storage
surface and auth depends on the deployment's OIDC setup; operators wire
up ingress + auth explicitly.
- `tests/deployment_test.yaml` — `helm-unittest` covering Service shape,
TLS secret mount, securityContext, and nginx proxy+mTLS config.
- `packages/core/platform/sources/linstor-gui.yaml` — new
`PackageSource` depending on `cozystack.linstor`.
- `packages/core/platform/templates/bundles/system.yaml` — registers
`linstor-gui` as an **optional** system package (controlled via
`bundles.enabledPackages`), alongside `velero`, `telepresence`, etc.
### Why standalone (not a sidecar in the controller pod)
LINBIT's upstream Dockerfile is designed for this standalone
reverse-proxy topology, and keeping the UI as its own Deployment means
it scales, upgrades, and restarts independently of `linstor-controller`.
The price is the small amount of mTLS config in `nginx.conf`, all of
which lives in the chart's ConfigMap.
### Release note
```release-note
[linstor-gui] Add an opt-in system package that deploys LINBIT's linstor-gui web UI alongside the LINSTOR controller. Enable via `bundles.enabledPackages: [cozystack.linstor-gui]`, then `kubectl -n cozy-linstor port-forward svc/linstor-gui 3373:80` to access.
```
## Test plan
- [ ] `make unit-tests` (requires `helm-unittest` plugin) passes on the
new `tests/deployment_test.yaml`
- [ ] `make image` in `packages/system/linstor-gui/` builds and pushes
the image
- [ ] Install the package on a dev cluster with LINSTOR already
deployed; confirm the pod comes up
- [ ] `kubectl -n cozy-linstor port-forward svc/linstor-gui 3373:80` →
browser loads the UI at `localhost:3373`
- [ ] UI lists LINSTOR nodes, storage pools and resources correctly
(proves mTLS proxy to `/v1` works)
- [ ] Disable via `bundles.enabledPackages` removal; verify the
`Package` resource is cleaned up
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* LINSTOR GUI added as an optional system package deployable to the
cozy-linstor namespace.
* Web UI serves on port 3373, proxies LINSTOR controller over mTLS, and
exposes a /healthz endpoint.
* Image build and release targets added to produce multi-arch container
images.
* **Documentation**
* New README with deployment guidance, connection examples, and
configurable options (endpoint, client secret, image, replicas).
* **Tests**
* Helm chart tests validating service, deployment, config and mTLS
behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Makes `cozystack-scheduler` LINSTOR storage-aware by configuring it to
call the existing `linstor-scheduler-extender` as a scheduler extender.
This fixes#2328 (phase 1): pods with both a `SchedulingClass` and
LINSTOR PVCs now get storage-locality-aware placement instead of
bypassing LINSTOR's filter/prioritize logic.
Changes:
- **linstor-scheduler package**: expose the extender sidecar (port 8099)
via a new ClusterIP Service; patch the vendored deployment to add a
`app.kubernetes.io/component: scheduler` label for clean Service
targeting
- **cozystack-scheduler package**: bump to v0.3.0 which adds
configurable `extenders` support; set the LINSTOR extender URL in
wrapper values
- **tests**: helm-unittest for extender rendering in the ConfigMap
### Release note
```release-note
[scheduler] cozystack-scheduler now calls the LINSTOR scheduler extender for storage-aware pod placement. Pods assigned to a SchedulingClass that also use LINSTOR-backed PVCs will prefer nodes with local volume replicas.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added scheduler extender support: exposes an extender Service and adds
pod metadata to enable extender routing.
* **Chores**
* Bumped scheduler chart and image versions to 0.3.0.
* Added default extender configuration and a sample remote extender
entry.
* Updated build scripts to apply an additional patch and added a test
target.
* **Tests**
* Added Helm tests validating rendered ConfigMap with and without
extender configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Expose the existing linstor-scheduler-extender sidecar as a ClusterIP
Service and configure cozystack-scheduler to call it during the
scheduling cycle. Pods with both a SchedulingClass and LINSTOR PVCs
now get storage-locality-aware placement.
- Add extender Service (port 8099) for linstor-scheduler
- Patch vendored deployment to label pods for Service selector
- Bump cozystack-scheduler to v0.3.0 (configurable extenders)
- Add "linstor" PackageSource variant with extender values
- Default variant ships without extender for non-LINSTOR clusters
- Select linstor variant in system bundle
- Add helm-unittest tests for both packages
Ref: #2328
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Ships LINBIT's LINSTOR web UI (GPL-3.0) as an opt-in system package under
packages/system/linstor-gui so operators can manage LINSTOR nodes, resources,
and volumes from a browser instead of the linstor CLI.
The UI is built from the upstream pkg.linbit.com tarball onto an
nginx-unprivileged base image. A chart-supplied nginx.conf proxies /v1 and
/metrics to the LINSTOR controller REST API over mTLS using the existing
linstor-client-tls secret created by the linstor package.
Only a ClusterIP Service is created; no Ingress is shipped, because LINSTOR's
controller API is a privileged cluster-wide storage surface and auth depends
on the deployment's OIDC setup. Operators opt in via bundles.enabledPackages
and wire up ingress + auth themselves (port-forward works out of the box).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
E2E revealed that the kube-apiserver rejects a DaemonSet carrying a
per-container AppArmor annotation for a container that is not present
in the pod spec:
DaemonSet.apps 'cilium' is invalid:
spec.template.annotations[container.apparmor.security.beta.kubernetes.io/mount-cgroup]:
Invalid value: 'mount-cgroup': container not found
On Talos (isp-full bundle) values-talos.yaml sets
cgroup.autoMount.enabled=false, so the upstream chart does not render
the mount-cgroup init container, and our unconditional podAnnotations
entry for it becomes an invalid reference.
Move the podAnnotations block from the shared values.yaml into a new
values-apparmor.yaml and include it only in the cilium-generic and
kubeovn-cilium-generic PackageSource variants, where
cgroup.autoMount.enabled is explicitly true. Talos variants
(cilium, cilium-kilo, kubeovn-cilium) get no AppArmor annotations,
which is safe because the Talos kernel does not load the AppArmor
LSM anyway.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
<!-- Thank you for making a contribution! Here are some tips for you:
- Start the PR title with the [label] of Cozystack component:
- For system components: [platform], [system], [linstor], [cilium],
[kube-ovn], [dashboard], [cluster-api], etc.
- For managed apps: [apps], [tenant], [kubernetes], [postgres],
[virtual-machine] etc.
- For development and maintenance: [tests], [ci], [docs], [maintenance].
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same [label] as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
[backups] Added fix to roles and changed backupstrategy-controller location
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated backup controller namespace configuration.
* Enhanced backup controller permissions for leader election and event
recording capabilities.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Adds the cozystack-scheduler as an optional system package, vendored
from https://github.com/cozystack/cozystack-scheduler. The scheduler
extends the default kube-scheduler with SchedulingClass-aware affinity
plugins, allowing platform operators to define cluster-wide scheduling
constraints via a SchedulingClass CRD. Pods opt in via the
`scheduler.cozystack.io/scheduling-class` annotation.
The package includes:
- Helm chart with RBAC, ConfigMap, Deployment, and CRD
- PackageSource definition for the cozystack package system
- Optional inclusion in the platform system bundle
### Release note
```release-note
[cozystack-scheduler] Add cozystack-scheduler as an optional system
package. The custom scheduler supports SchedulingClass CRDs for
cluster-wide node affinity, pod affinity, and topology spread constraints.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added cozystack-scheduler component as an optional system package.
* Introduced SchedulingClass custom resource for advanced scheduling
configurations.
* Scheduler supports node affinity, pod affinity, pod anti-affinity, and
topology spread constraints.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Adds the cozystack-scheduler as an optional system package, vendored from
https://github.com/cozystack/cozystack-scheduler. The scheduler extends
the default kube-scheduler with SchedulingClass-aware affinity plugins,
allowing platform operators to define cluster-wide scheduling constraints
via a SchedulingClass CRD. Pods opt in via the
`scheduler.cozystack.io/scheduling-class` annotation.
The package includes:
- Helm chart with RBAC, ConfigMap, Deployment, and CRD
- PackageSource definition for the cozystack package system
- Optional inclusion in the platform system bundle
### Release note
```release-note
[cozystack-scheduler] Add cozystack-scheduler as an optional system
package. The custom scheduler supports SchedulingClass CRDs for
cluster-wide node affinity, pod affinity, and topology spread constraints.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
## What this PR does
Adds OpenBAO (open-source Vault fork) as a new managed PaaS application
in Cozystack.
**Structure follows existing app patterns (qdrant, nats):**
- System chart with vendored upstream `openbao/openbao` (chart v0.25.3,
appVersion v2.5.0)
- App chart with standalone/HA mode switching based on replicas count
- TLS via cert-manager self-signed certificates per instance
- ApplicationDefinition, PackageSource, PaaS bundle entry
- E2E test with init/unseal workflow
**Key design decisions:**
- `replicas: 1` → standalone mode with file storage; `replicas > 1` → HA
with Raft integrated storage and retry_join with TLS peer verification
- TLS enabled by default — each instance gets a self-signed Certificate
with DNS SANs covering services and pod addresses
- `disable_mlock = true` in HCL config since default security context
drops IPC_LOCK capability
- Injector and CSI provider disabled (cluster-scoped components, not
safe per-tenant)
- No auto-init/unseal — OpenBAO requires manual initialization by design
- E2E test performs full lifecycle: deploy, wait for certificate + API,
init, unseal, verify readiness, cleanup
### Release note
```release-note
[apps] Add OpenBAO as a managed secrets management service with standalone and HA Raft modes, TLS enabled by default
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **New Features**
* Added OpenBAO managed secrets management service with
high-availability and standalone deployment options
* Integrated monitoring and dashboards for operational visibility
* Enabled configurable external access and web UI
* Added automated snapshot backup capability
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Replace PVC-based registry storage with S3 via COSI BucketClaim/BucketAccess.
The system chart parses BucketInfo secret and creates a registry-s3 Secret
with REGISTRY_STORAGE_S3_* env vars that override Harbor's ConfigMap values.
- Add bucket-secret.yaml to system chart (BucketInfo parser)
- Remove storageType/size from registry config (S3 is now the only option)
- Use Harbor's existingSecret support for S3 credentials injection
- Add objectstorage-controller to PackageSource dependencies
- Update E2E test with COSI bucket provisioning waits and diagnostics
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add postgres-operator and redis-operator to PackageSource dependsOn
to ensure CRDs are available before Harbor system chart deploys.
Make persistentVolumeClaim conditional to avoid empty YAML mapping
when using S3 storage without Trivy.
Increase E2E system HelmRelease timeout from 300s to 600s to account
for CPNG + Redis + Harbor bootstrap time on QEMU.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Move Harbor from packages/extra/ to packages/apps/ as it is a
self-sufficient end-user application, not a singleton tenant module.
Update bundle entry from system to paas accordingly.
Replace registry PVC storage with S3 via COSI BucketClaim/BucketAccess,
provisioned from the namespace's SeaweedFS instance. S3 credentials are
injected into the HelmRelease via valuesFrom with targetPath.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add Harbor v2.14.2 as a tenant-level managed service with per-component
resource configuration, ingress with TLS termination, and internal
PostgreSQL/Redis.
Includes:
- extra/harbor wrapper chart with HelmRelease, WorkloadMonitors, Ingress
- system/harbor with vendored upstream chart (helm.goharbor.io v1.18.2)
- harbor-rd ApplicationDefinition for dynamic CRD registration
- PackageSource and system.yaml bundle entry
- E2E test with Secret and Service verification
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add a new networking variant that integrates Kilo with Cilium
pre-configured. Cilium is deployed with host firewall disabled and
enable-ipip-termination enabled, which are required for correct IPIP
encapsulation through Cilium's overlay.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Kilo is now integrated into the cilium-kilo networking variant instead
of being a separate package that users install manually.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Add operator and application PackageSource CRs to expose OpenSearch
in the Cozystack platform:
- opensearch-operator: operator deployment in cozy-opensearch-operator namespace
- opensearch-application: app + resource definition with cozy-lib integration
This enables OpenSearch to appear in the platform dashboard and be
deployed by tenants.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Signed-off-by: Matthieu <matthieu@hidora.com>
Add a new "cilium" variant to the kilo PackageSource that deploys kilo
with --compatibility=cilium flag. This enables Cilium-aware IPIP
encapsulation, routing outer packets through Cilium's VxLAN overlay
instead of the host network.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
The virtual-machine application is replaced by separate vm-disk and
vm-instance applications. Migration 28 handles the conversion of
existing VirtualMachine HelmReleases.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
The mysql chart actually deploys MariaDB via mariadb-operator, but was
incorrectly named "mysql". Rename all references to use the correct
"mariadb" name across the codebase.
Changes:
- Rename packages/apps/mysql -> packages/apps/mariadb
- Rename packages/system/mysql-rd -> packages/system/mariadb-rd
- Rename platform source and bundle references
- Update CRD kind from MySQL to MariaDB
- Update RBAC, e2e tests, backup controller tests
- Keep real MySQL CLI/config tool names unchanged (mysqldump, [mysqld], etc.)
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
Remove the FerretDB managed application from Cozystack. This includes
the application Helm chart, resource definition, platform source, PaaS
bundle entry, RBAC clusterrole entry, and e2e test. Historical migration
scripts are left intact for upgrade compatibility.
### Release note
```release-note
[ferretdb] Removed FerretDB managed application
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Removed FerretDB managed database service and associated Helm chart,
documentation, and test components from the platform.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Remove the FerretDB managed application, its resource definition,
platform source, RBAC entry, and e2e test. Historical migration
scripts are left intact for upgrade compatibility.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
Add a new system package `cluster-autoscaler` that supports multiple
cloud providers (Hetzner and Azure) for automatic node scaling of
Cozystack management cluster.
Key features:
- Single base package with provider-specific variants via PackageSource
- Auto-update capability from upstream Helm chart
- RBAC rules for leader election (coordination.k8s.io/leases)
- Comprehensive documentation for Hetzner setup with Talos Linux
The Hetzner documentation covers:
- Talos image creation via rescue mode snapshot
- vSwitch (private network) configuration
- Correct Talos machine config structure (nodeLabels, nodeIP,
cloud-provider)
- Testing with pod anti-affinity
- Troubleshooting common issues
- Kilo mesh networking integration
### Release note
```release-note
[system] Add cluster-autoscaler package with support for Hetzner Cloud and Azure providers
```
Cluster-autoscaler does not require privileged installation.
Remove the unnecessary privileged: true setting from both
Hetzner and Azure PackageSource definitions.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## Summary
- Create new `piraeus-operator-crds` package containing all piraeus CRDs
- Add `piraeus-operator-crds` as dependency for `piraeus-operator` in
linstor PackageSource
- Set `privileged: true` for CRDs package to ensure namespace gets
correct PodSecurity labels
- Disable `installCRDs` in piraeus-operator values
## Motivation
Helm does not reliably install all CRDs from the `templates/` directory
when the crds.yaml file is large. This causes
`linstorsatellites.piraeus.io` CRD to be missing, which breaks satellite
pod creation.
Separating CRDs into a dedicated package (similar to cert-manager-crds,
prometheus-operator-crds) ensures reliable CRD installation.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Introduced a separate component for managing Piraeus operator CRDs
with explicit dependency ordering.
* Updated operator configuration to disable built-in CRD installation
and depend on the new dedicated CRD component.
* Established installation dependency chain to ensure components
initialize in the correct sequence.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
- Remove `cozystack.cozystack-controller` from monitoring package
dependencies
- Allows monitoring to work without cozystack-engine being enabled
## Motivation
The monitoring package depends on `cozystack.cozystack-controller` which
is only installed as part of `cozystack-engine`. This prevents
monitoring from becoming ready when cozystack-engine is not enabled.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Simplified monitoring default: reduced installed sub-components and
top-level dependencies for a leaner deployment.
* Removed generation of several automated workload-monitoring manifests
to streamline resource set.
* **New Features**
* Added a curated list of monitoring dashboards for cluster, networking,
storage, databases, control-plane, ingress, observability tooling, and
key services.
* Added three ExternalName services to expose tenant monitoring
endpoints.
* **Improvements**
* Made monitoring agents namespace-aware by templating
tenant/multi-namespace references for metrics and logging endpoints.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Add PackageSource and PaaS bundle entry so the platform deploys
qdrant-rd, which provides the ApplicationDefinition that registers
the Qdrant Kind in the API server.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Replace hardcoded tenant-root with {{ .Release.Namespace }} in vmagent
and fluent-bit configs
- Add ExternalName services in cozystack-basics to redirect monitoring
traffic from cozy-monitoring to tenant-root when engine is deployed
- Add missing components to monitoring PackageSource
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Remove components that are already defined in separate PackageSources
to avoid race conditions and HelmRelease conflicts:
- vertical-pod-autoscaler
- postgres-operator
- grafana-operator
- victoria-metrics-operator
- prometheus-operator-crds
These components should be installed via their dedicated PackageSources.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Helm does not reliably install all CRDs from templates/ directory,
particularly when the crds.yaml file is large. This causes
linstorsatellites.piraeus.io CRD to be missing, breaking satellite pod
creation.
Changes:
- Create new piraeus-operator-crds package with all piraeus CRDs
- Add piraeus-operator-crds as dependency for piraeus-operator
- Set privileged: true for CRDs package to ensure namespace has correct
PodSecurity labels from the start
- Disable installCRDs in piraeus-operator since CRDs come from separate
package
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
The monitoring package depends on cozystack.cozystack-controller which
is only installed as part of cozystack-engine. This prevents monitoring
from becoming ready when cozystack-engine is not enabled.
Remove this dependency to allow monitoring to work independently.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Andrei Kvapil