Files
cozystack/packages/core/platform/values.yaml
Aleksei Sviridkin bcda5c893a feat(kubevirt)!: drop HotplugVolumes for declarative volume hotplug (#4453)
## What this PR does

This turns off the deprecated KubeVirt `HotplugVolumes` gate and moves
the platform to declarative volume hotplug. It changes how every volume
hotplug on the platform works, so the title carries `!`. Closes #4032.

This PR is stacked on #4451 and has to merge after it. #4451 adds the
kubevirt-csi flag this PR flips.

The replacement gate is already on in KubeVirt 1.9, and the old gate is
the only thing keeping it off. `DeclarativeHotplugVolumes` is Beta at
v1.9.0, and Beta gates are on by default. virt-controller skips
`handleDeclarativeVolumeHotplug` while `HotplugVolumes` is on, so
dropping the old gate is what switches the new one on.

The kubevirt-csi fallback goes off in the same change. With the old gate
off, virt-api rejects the VMI-level `removevolume` on a VM-owned VMI, so
the chart now passes `--enable-vmi-hotplug-fallback=false`. The comments
next to the gate and the flag point at each other.

This needs the rebuilt kubevirt-csi image. A driver built before #4451
does not know the flag and exits on it at startup. The image is stamped
at release preparation and ships in the same release as this change, so
only a `make apply` from main before then would crash-loop the CSI
controller.

What changes at runtime on an upgrade:

- The VM-level `addvolume` and `removevolume` calls now edit `VM.spec`
directly. virt-controller then makes the VMI hotplug volumes match it,
instead of going through `VM.status.volumeRequests`.
- Volumes hot-plugged through the VM subresource are in `VM.spec`, so
they stay attached.
- During a tenant teardown, when the VM goes before its VMI, the driver
no longer calls the VMI subresource. It waits for the VMI status to drop
the volume, and the garbage collector deletes the VMI with its VM.
Upstream waits up to two minutes. If the VMI is still there after that,
the unpublish fails and is retried.

A new platform migration keeps old VMI-only volumes attached. On its
first pass virt-controller removes every hotpluggable VMI volume that is
missing from `VM.spec`. Until July 2025 kubevirt-csi attached volumes at
the VMI level (the Role then granted
`virtualmachineinstances/addvolume`), and live migration carries such a
VMI across KubeVirt updates. Without help, a tenant node not recreated
since then would lose a mounted disk on upgrade.

Migration 59 copies each such volume and its disk from the VMI into the
VM template before anything else changes. Adding a hotplug volume to the
VM spec is a valid live update in KubeVirt, so no VM needs a restart. It
runs as the platform chart's pre-upgrade hook, before the upgrade
re-points `cozystack-packages`. The KubeVirt CR and the tenant
kubernetes charts both render from that source, so the gate and the flag
cannot change before the migration has finished. It only touches VMs
that need it, logs each one, and a second run changes nothing.

The migration also makes ephemeral hotplugs permanent. A volume a user
added at the VMI level, for example with `virtctl addvolume` without
`--persist` from an older virtctl, has the same shape as kubevirt-csi's
old attachments, because virt-api builds both the same way. There is
nothing reliable to tell them apart by, so both are copied. Leaving the
user's volume out would not keep it ephemeral: the first reconcile would
unplug it from the running VM. After the upgrade it is part of `VM.spec`
and attaches again on a restart, and `virtctl removevolume` drops it.

Ephemeral hotplug stops working on VMs. With the old gate off, virt-api
refuses VMI-level `addvolume` and `removevolume` on a VMI that a VM owns
(`vmiVolumePatch` in v1.9.0). A volume on such a VMI is added and
removed through the VM from then on. A VMI without a VM is not affected.

Only running VMIs are copied. virt-controller reconciles no other VMI,
and a stopped VM builds its next VMI from `VM.spec`, so copying from a
stopped one would bring back volumes the restart was going to drop.

Migrations run only on the isp-full, isp-full-generic and isp-hosted
variants. KubeVirt comes from the platform only on isp-full and
isp-full-generic, so those are covered. On the default variant, where
packages are installed by hand, nothing runs the migration.

This optional check lists the VMIs and volumes the migration would copy.
It uses the same rule. On the default variant, run it before upgrading
and recreate each tenant node it lists:

```bash
kubectl get vm,vmi --all-namespaces --output json | jq --raw-output '
  ([.items[] | select(.kind == "VirtualMachine")
    | {key: "\(.metadata.namespace)/\(.metadata.name)", value: [.spec.template.spec.volumes[]?.name]}]
    | from_entries) as $vm
  | .items[] | select(.kind == "VirtualMachineInstance")
  | ([.metadata.ownerReferences[]? | select(.controller == true and .kind == "VirtualMachine" and (.apiVersion | startswith("kubevirt.io/"))) | .name][0]) as $owner
  | "\(.metadata.namespace)/\($owner)" as $key
  | select(.status.phase == "Running" and $owner != null and $vm[$key] != null)
  | .spec.volumes[]? | select(.dataVolume.hotpluggable == true or .persistentVolumeClaim.hotpluggable == true)
  | .name as $n | select($vm[$key] | index($n) | not)
  | "\($key) \($n)"'
```

Empty output means nothing is affected. I checked the query against the
migration's test fixtures only. Neither the query nor the migration has
run on a live cluster.

#4287 also adds a migration 59. If it merges first, this one moves to
60: the file, the version it stamps, `migrations.targetVersion`, the
bats file (its `run_migration` argument and expected `STAMP`) and
`hack/testdata/migration-59-vmi-hotplug`.

A VM with a pending hotplug request at upgrade time may also get stuck.
virt-controller still processes `VM.status.volumeRequests` before the
declarative handler, and it applies them through the VMI subresource,
which virt-api rejects once the old gate is off. The sync then fails
before the VM spec is updated, so nothing clears the request, and every
later sync of that VM fails the same way. A retry through the VM
subresource edits the spec and clears it, which covers kubevirt-csi's
own requests. A one-shot request stays stuck until the VM restarts. I
read this from the v1.9.0 code and haven't reproduced it. Waiting until
this lists nothing before upgrading narrows the window, but a request
made during the rollout can still hit it:

```bash
kubectl get vm --all-namespaces --output json | jq --raw-output '.items[] | select((.status.volumeRequests // []) | length > 0) | .metadata.namespace + "/" + .metadata.name'
```

The two halves live in different HelmReleases, so there is a short
window during the rollout where the driver and the gate disagree. In
that window an unpublish fails and is retried until both are updated.

A tenant cluster whose `kubernetes` HelmRelease is suspended or failing
keeps its old driver for longer. Drivers from before July 2025 attach
through the VMI subresource, so on such a tenant every attach is refused
until its release reconciles. Such a driver can also attach a volume
after the migration has run and before the gate flips. The migration
doesn't see that volume, and the first declarative reconcile unplugs it
from the running VM.

The migration lists the VMs once and patches each one with a
`resourceVersion` test, so a VM that changes in between fails the hook
Job. The Job has `backoffLimit: 3`, and each retry lists again and skips
the VMs already patched. Re-reading only the conflicting VM would avoid
the retry; I left that out.

The new gate is listed in the CR even though it is on by default. With
the old gate gone it is the only hotplug path, and the chart refuses a
render that disables it. I found no other user of `HotplugVolumes` in
the platform or in the downstream repositories.

The VMI-level fallback may have been failing already. The kcsi Role has
no rule for the VMI-level `removevolume` (#4452), so that call likely
got 403 with or without the gate. I haven't checked that on a cluster.

`hack/migration-59-vmi-hotplug-volumes.bats` runs the migration against
a fake kubectl: a VM that needs the copy, VMs and VMIs that must be left
alone (a VMI that is not Running among them), a second run, a cluster
without KubeVirt and the fail-closed paths. The helm-unittests for the
KubeVirt CR pin the whole gate list, accept `HotplugVolumes` in
`disabledFeatureGates` and refuse `DeclarativeHotplugVolumes` there. The
CSI test pins the flag at false, and the `make update` idempotency test
still passes. None of that exercises hotplug itself. The real test is
e2e with volume attach and detach on a tenant cluster, including a
teardown. No CI lane upgrades from the previous release yet, so an
upgrade with a live VMI-only volume is untested; #4463 tracks that test.

### Screenshots

No UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(kubevirt)!: KubeVirt now runs with DeclarativeHotplugVolumes instead of the deprecated HotplugVolumes feature gate, and kubevirt-csi no longer uses the VMI-level hot-unplug. Volume hotplug requests are applied through VM.spec. The upgrade copies volumes that older kubevirt-csi versions attached only to the VMI into VM.spec, so they stay attached; the check in the PR description is optional. This also applies to ephemeral volumes added to running user VMs at the VMI level, such as with virtctl addvolume without --persist: they become part of VM.spec and attach again after a VM restart. virt-api now refuses VMI-level addvolume and removevolume on a VMI owned by a VM, so such volumes are added and removed through the VM. A tenant cluster whose kubernetes release is suspended or failing keeps its old kubevirt-csi, and a driver from before July 2025 cannot attach volumes there until that release reconciles, and a volume it attaches during the upgrade itself can be unplugged again. A values override that puts DeclarativeHotplugVolumes into disabledFeatureGates now fails the kubevirt render. Clusters on the default variant do not run platform migrations and should run that check before upgrading.
```





<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
  * Declarative hotplug volumes are enabled by default for KubeVirt.
* Running virtual machines keep eligible hotplugged volumes and disks
when their configuration is migrated.
* **Bug Fixes**
* The migration avoids adding duplicate volumes or disks and does not
mark the migration complete if required operations fail.
  * The Kubernetes CSI driver no longer uses the VMI hotplug fallback.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-06 17:06:03 +03:00

914 lines
50 KiB
YAML
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
sourceRef:
kind: OCIRepository
name: cozystack-platform
namespace: cozy-system
path: /
migrations:
enabled: false
# The image digest is NOT hand-edited alongside targetVersion. The release
# build rebuilds the migrations image from this same source tree (which
# carries migrations/<N> and the etcd-migrate binary) and restamps this pin
# in the same commit via `make image-migrations` (yq --inplace) — so image and
# targetVersion are only consistent at a release tag, not mid-PR. The digest
# below therefore lags targetVersion in-tree by design; run-migrations.sh
# refuses to advance past a migration file missing from the image (exit 1), so
# a stale pin fails loudly rather than silently skipping the etcd adoption.
image: ghcr.io/cozystack/cozystack/platform-migrations:v1.6.0@sha256:6777042e3e9c6bad76e7d96fb64baa115061975f39061cb3ede84e21d0e2213f
targetVersion: 60
# Adopt legacy etcd clusters onto the v1alpha2 operator WITHOUT the mandatory
# pre-adoption safety snapshot (migration 50). Leave false unless migration 50
# has actually refused to proceed and you accept the risk: adoption rewires
# ownership of LIVE etcd storage, and a botched adoption on a tenant
# control-plane etcd is UNRECOVERABLE with no snapshot.
#
# Migration 50 refuses when it cannot resolve, or cannot trust, a snapshot
# destination — backups disabled, an external S3 whose credentials were never
# staged, SeaweedFS not ready at upgrade time, or a bucket it cannot classify.
# Those are individually-supported configurations, so rather than leave such a
# cluster permanently unable to upgrade, this flag lets an operator proceed
# deliberately. Prefer fixing the condition the migration names in its log:
# the migration is idempotent, so re-running after the fix is the safe path,
# and this flag is the last resort when there is nothing to fix.
#
# Reset it to false once the upgrade is through, or the next platform upgrade
# that adds an etcd migration will skip its snapshot too.
etcdAdoptSkipBackup: false
# Bundle deployment configuration
bundles:
system:
enabled: false
variant: "isp-full" # Options: "isp-full", "isp-full-generic", "isp-hosted", "isp-slim", "isp-slim-generic", "isp-hosted-slim"
iaas:
enabled: false
# gpu-operator variant selector — applies when cozystack.gpu-operator is
# in bundles.enabledPackages. "default" runs the vfio-pci passthrough
# stack (whole GPU → one VM); "vgpu" runs the vGPU Manager stack;
# "container" serves GPUs to ordinary pods through the device plugin on
# hosts that already provide the NVIDIA driver and container toolkit.
# The platform mirrors the two VM variants into the KubeVirt CR: both
# get the HostDevices feature gate; passthrough also gets a default
# permittedHostDevices.pciHostDevices table covering common NVIDIA
# datacenter GPUs (vendor 10DE), and vgpu gets
# permittedHostDevices.mediatedDevices when
# gpu.vgpuDeviceManager.enabled is true. No mediatedDevicesConfiguration
# default ships in either variant — see the gpu block below for why.
# "container" gets none of that wiring: it keeps the host driver bound,
# so no GPU is available for VM passthrough.
# Operators with additional or non-NVIDIA cards extend the defaults via
# gpu.permittedHostDevices / gpu.mediatedDevicesConfiguration /
# gpu.replaceDefaults below — read by the two VM variants only.
gpuOperatorVariant: default
paas:
enabled: false
naas:
enabled: false
disabledPackages: []
# Packages the platform ships but does not deploy unless named here. Two of
# them belong to the Proxmox substrate, where worker VMs run on an external
# hypervisor instead of on KubeVirt in this cluster:
#
# cozystack.capi-provider-infra-proxmox
# capmox, the Cluster API infrastructure provider for Proxmox VE. Needs a
# capmox-config Secret with a Proxmox API token before it can do anything;
# enabling it pulls in the IPAM provider below automatically.
# cozystack.capi-provider-ipam-in-cluster
# Cluster API in-cluster IPAM. Required by capmox, which assigns static
# addresses and has no DHCP mode. Listed separately because it is a
# generic addon, not a Proxmox one.
#
# Both are emitted by the iaas bundle. disabledPackages still wins over this
# list, except that disabling IPAM while capmox is enabled stops the render:
# the cluster would only park capmox at Ready=False with DependenciesNotReady,
# and nothing there would name the two list entries that conflict.
enabledPackages: []
# Network configuration
networking:
clusterDomain: "cozy.local"
podCIDR: "10.244.0.0/16"
podGateway: "10.244.0.1"
serviceCIDR: "10.96.0.0/16"
joinCIDR: "100.64.0.0/16"
# KubeOVN master nodes override (optional)
# By default, KubeOVN helm chart uses `lookup` to find control-plane nodes
# by label `node-role.kubernetes.io/control-plane`. On fresh clusters or
# during initial deployment, lookup may return empty results.
# Set this to comma-separated list of master node IPs to override.
kubeovn:
MASTER_NODES: ""
# In-cluster (east-west) transport encryption. Opt-in, OFF by default.
#
# When enabled, pod-to-pod traffic that crosses nodes is encrypted at the
# network layer by the ACTIVE datapath, regardless of whether the workload
# speaks TLS — defense-in-depth for the whole mesh without per-application
# changes. This COMPLEMENTS, and does NOT replace, application-level /
# end-to-end TLS: it never reaches an external client and gives a tenant no
# verifiable endpoint certificate (see the per-app certificate series). Use
# it alongside, not instead of, TLS on exposed endpoints.
#
# The mechanism is datapath-specific (selected by bundles.system.variant):
#
# * kubeovn-cilium (isp-full) and kubeovn-cilium-generic
# (isp-full-generic): KubeOVN owns the datapath; its Geneve overlay is
# encrypted with IPsec (kube-ovn ENABLE_OVN_IPSEC). KubeOVN's own
# controller provisions the IPsec CA and per-node certificates through
# the Kubernetes CSR API — no cert-manager wiring is required. This is
# the path this toggle drives today.
# * cilium / cilium-generic (Cilium owns the datapath; isp-slim,
# isp-slim-generic): Cilium-native WireGuard/IPsec is the right mechanism
# there, but it is not wired yet, so the slim variants refuse this toggle.
# * cilium-kilo: Kilo already establishes a WireGuard mesh, but only at
# meshGranularity=location — it encrypts CROSS-LOCATION traffic and
# leaves intra-location pod-to-pod in cleartext by design. This toggle
# targets exactly that intra-location gap and composes with Kilo rather
# than conflicting with it.
#
# CAVEATS: encrypting the overlay adds per-packet ESP overhead and lowers
# the effective MTU, so east-west throughput drops and latency rises; the
# cost is highest for high-bandwidth flows. IPsec here protects the
# inter-node overlay tunnels — same-node pod-to-pod traffic never leaves the
# host and is not tunneled or encrypted. Flipping this on an existing
# cluster triggers a rollout of the CNI components. On variants without a
# cozystack-managed encryptable datapath (isp-hosted, isp-hosted-slim / noop) the toggle
# has no effect.
encryption:
enabled: false
# Install the upstream reference CNI plugins (bridge, macvlan, ipvlan, ...)
# into /opt/cni/bin on every node, from the multus image. Multus-backed
# NetworkAttachmentDefinitions delegate to these. Talos puts six binaries in
# that directory -- bridge, firewall, host-local, loopback and portmap from the
# same upstream release, plus flannel from a different repo -- and generic Linux
# gives no guarantee at all, so a bridge-type NAD fails there with
# `failed to find plugin "bridge" in path [/opt/cni/bin]`. A NAD naming one of
# the other ten needs staging on either platform.
#
# On for both variants when unset. Four of the names Talos carries are ones
# staging writes, so it rewrites those with the same upstream version (v1.9.1
# on both sides) and adds the ten it does not carry; loopback is not staged and
# flannel was never part of the reference set. Set it explicitly to decline.
#
# A spelling this chart does not recognise stops the whole platform render, not
# just multus, so a typo here halts reconciliation cluster-wide. That is
# deliberate -- reading an unreadable value as "off" would defeat the opt-out --
# but it is why the accepted spellings are worth getting right first time.
#
# When on, the plugins are written every time the Multus pod is recreated,
# replacing files of the same name. 14 of the upstream reference plugins are
# installed; loopback, dhcp, dummy and tap are left alone. See
# docs/vm-external-vlan.md.
#
# On a running cluster this is set on the `platform` component of the
# `cozystack.cozystack-platform` Package CR, the same path as backupStorage
# (see docs/operations/backup-classes.md). Editing the multus HelmRelease or
# the child Package cozystack.multus directly is reverted by the platform
# reconcile.
# stageCniPlugins: true
# Service publishing and ingress configuration
publishing:
host: "example.org"
ingressName: tenant-root
# Optional namespace of a SEPARATE Gateway / ingressClass that system
# components attach their admin routes to (currently Keycloak) — e.g. one
# bound to a private IP, unreachable from the public internet. Empty means
# admin endpoints share the same ingress as everything else (ingressName).
# The private Gateway itself is NOT created by cozystack; provision it out
# of band, then point this at its namespace to move admin traffic onto it.
ingressNameAdmin: ""
exposedServices:
- api
- dashboard
- vm-exportproxy
- cdi-uploadproxy
apiServerEndpoint: "" # example: "https://api.example.org"
externalIPs: []
# Native loadBalancerClass for the host ingress. When externalIPs is empty the
# host ingress publishes its Service as type: LoadBalancer; set this to the
# loadBalancerClass of the LoadBalancer controller that should handle it. The
# admin provisions the matching address pool / announcer out of band (a MetalLB
# IPAddressPool, a Cilium LB-IPAM pool, a cloud LB, etc.). Empty lets the
# cluster's default LoadBalancer controller claim the Service.
# spec.loadBalancerClass is immutable: on a cluster whose host ingress Service
# already exists as type: LoadBalancer, setting, changing, or clearing this
# value requires recreating that Service once (the chart fails early with the
# exact command).
loadBalancerClass: ""
# DNS suffix the dashboard appends to the name of a LoadBalancer Service on
# the application Services tab, shown under the external IP as
# `<service-name>.<serviceDomain>` (e.g. demo-external-write.svc.example.org
# for a postgres release named demo). Display-only: nothing in the platform
# creates DNS records for it. Set it only when your DNS already resolves
# exactly that name to the Service's LoadBalancer IP, e.g. external-dns with
# a matching fqdn template. A CoreDNS k8s_external zone does not fit: it
# answers <service>.<namespace>.<zone>, which this suffix cannot express.
# This is not the `<release>.<tenant host>` name the app charts put into
# their certificates, so a TLS client may see a SAN mismatch unless DNS and
# certificates are aligned by the operator. The suffix is cluster-wide, so
# two tenants with a release of the same name get the same hostname.
# Empty (the default) shows no hostname.
serviceDomain: ""
# Enable PROXY-protocol on the host ingress-nginx and auto-deploy
# ouroboros (system/ouroboros) to fix the resulting hairpin-NAT
# problem (intra-cluster traffic to the cluster's own public hostnames
# arrives at ingress-nginx without the PROXY header it now requires).
# Single cozystack-side switch: flipping this to true configures both
# pieces in one go — ingress-nginx use-proxy-protocol and the
# cozystack.ouroboros Package CR.
#
# TALOS RE-RENDER NOTE (host scope only): the host CoreDNS is a
# Talos-managed Deployment; ouroboros mutates its live Corefile in
# place between BEGIN/END markers. A Talos machine-config push or
# upgrade re-renders the Corefile and overwrites the markers — there
# is a flap window of up to one ouroboros reconcile loop (bounded
# by `controller.resync`, default 10 minutes) until the controller
# re-applies the block. The reconcile loop is ALSO event-driven on
# Ingress/Gateway watches, so on a busy cluster the window often
# closes much sooner — but on a stable production cluster with no
# incidental Ingress/Gateway churn, the upper bound is the resync
# interval. In-cluster DNS for hairpinned hostnames briefly returns
# the upstream LB IP instead of the proxy ClusterIP during that
# window, which fails the same way pre-PR behaviour fails.
# Acceptable, but worth knowing during a Talos roll.
#
# PRECONDITION: the upstream L4 load balancer / WAN edge in front of
# ingress-nginx MUST already be injecting PROXY-protocol v1 headers
# before flipping this flag. Without that, every external request
# arrives at ingress-nginx without the PROXY header, ingress-nginx
# closes the connection, and ALL ingress traffic breaks. cozystack
# cannot configure that upstream LB for you — it lives outside the
# cluster (cloud LB, F5, MetalLB+haproxy, hcloud-cloud-controller,
# etc.). PROXY-protocol frames travel from the upstream LB to
# ingress-nginx (the LB opens a fresh TCP connection to the backend
# and prepends the frame on that connection), not back to the
# client — `nc` from a laptop will never see one. Pick one of:
#
# * tcpdump --interface any 'port 80 or port 443' -A on the node
# hosting ingress-nginx and watch for a leading "PROXY TCP4 ..."
# frame on inbound TCP from the LB BEFORE flipping this flag.
# * make any external HTTP request after the LB-side change but
# before flipping this flag, and watch
# `kubectl --namespace cozy-ingress-nginx logs deploy/ingress-nginx-controller`
# for "client sent invalid proxy protocol header" — that error
# means the LB started injecting PROXY frames but ingress-nginx
# is not yet configured to accept them, which is the transition
# state this flag is meant to resolve.
#
# If neither check shows PROXY frames, fix the upstream LB before
# touching this flag. Leave it false on clusters where PROXY-protocol
# is not yet wired at the upstream LB; ouroboros wastes resources
# when the bug it fixes does not exist.
#
# FAN-OUT NOTE: this flag affects ONLY the host control plane. To
# turn the same hairpin fix on inside a tenant cluster, also set
# `addons.ouroboros.enabled: true` in the per-tenant Kubernetes CR
# (and `addons.ingressNginx.enabled: true` is a hard precondition
# there). The flag does not cascade into tenants on its own.
#
# DISABLE PATH: flipping this back from true to false has TWO
# asymmetric effects. ingress-nginx side: the switch travels through
# _cluster.proxy-protocol in the cozystack-values Secret, so the next
# reconcile re-renders the host ingress-nginx (extra/ingress) without
# the use-proxy-protocol / real-ip-header / enable-real-ip keys
# (cozystack deliberately never sets use-forwarded-headers or
# compute-full-forwarded-for — those would allow any upstream proxy
# to spoof X-Forwarded-For without a paired proxy-real-ip-cidr — see
# the rationale in extra/ingress/templates/nginx-ingress.yaml). The
# cozystack.ingress-application Package stays the bare default
# throughout; the keys never ride on it. ingress-nginx then stops
# accepting PROXY-protocol headers. If the upstream L4 LB is
# still prepending PROXY frames at that moment, every external
# request to ingress-nginx breaks until the LB is also reconfigured
# — flip the LB OFF FIRST, then flip this flag. ouroboros side:
# stops emitting the cozystack.ouroboros Package CR, but every Package
# CR carries `helm.sh/resource-policy: keep`, so Helm leaves the
# existing Package on the cluster, ouroboros stays installed, and
# the live Corefile rewrite block keeps pointing at the still-running
# ouroboros-proxy Service. Flag-flip alone leaves ouroboros idempotent.
#
# To actually uninstall ouroboros, the operator removes the Package
# CR explicitly:
#
# kubectl delete package.cozystack.io cozystack.ouroboros
#
# THAT is the path that triggers the chart's pre-delete Helm hook
# (charts/ouroboros/templates/coredns-cleanup-hook.yaml, vendored at
# 0.8.0) — the hook quiesces the controller, sed-strips the
# BEGIN/END block from the live kube-system/coredns Corefile, and
# restores the cleaned ConfigMap via kubectl replace. The hook is
# automatic; operators do NOT need to run a manual sed recipe in
# the normal disable path. Documentation of the hook's exact
# behaviour lives in the upstream chart's README.
#
# bundles.disabledPackages does NOT do this. Putting this name in
# that list only makes templates/_helpers.tpl skip emitting the
# Package document — the same suppression as leaving proxyProtocol
# false — and resource-policy: keep then stops Helm pruning a
# Package it no longer renders. The Package CR stays, so does the
# HelmRelease it owns, ouroboros keeps running, and the pre-delete
# hook never fires. No platform values setting reaches that hook:
# the operator has to delete the Package CR.
#
# Manual cleanup recipe (only if the chart's pre-delete hook fails
# to land — controller pod stuck in CrashLoop blocking the quiesce
# step, ConfigMap RBAC drift on the namespace, helm uninstall
# interrupted before the hook ran). Uses a JSON merge-patch so
# labels, annotations, and other data keys on the ConfigMap —
# including any Talos-managed metadata — are preserved:
#
# # === BEGIN cleanup-recipe ===
# existing=$(kubectl --namespace kube-system get configmap coredns \
# --output jsonpath='{.data.Corefile}')
# cleaned=$(printf '%s\n' "$existing" \
# | sed '/# === BEGIN ouroboros/,/# === END ouroboros ===/d')
# kubectl --namespace kube-system patch configmap coredns \
# --type merge \
# --patch "$(jq --null-input --arg c "$cleaned" \
# '{data: {Corefile: $c}}')"
# kubectl --namespace kube-system rollout restart deployment/coredns
# # === END cleanup-recipe ===
proxyProtocol: false
# Acknowledgement gate for the helm.sh/resource-policy: keep
# asymmetry (NOT a substitute for the chart's pre-delete cleanup
# hook — the hook handles the Corefile patch automatically when
# the chart actually uninstalls). The hazard this gate catches:
# flipping publishing.proxyProtocol from true to false stops
# emitting the cozystack.ouroboros Package CR, but resource-policy:
# keep means Helm leaves the existing Package on the cluster.
# ouroboros keeps running, the cleanup hook never fires, the
# rewrite block stays live. The platform render fails until the
# operator either restores proxyProtocol: true OR runs `kubectl
# delete package.cozystack.io cozystack.ouroboros` (which lets
# the chart actually uninstall and the hook run) and sets this
# flag to true to confirm. Once the cluster has been clean for
# one reconcile cycle, flip back to false.
#
# KNOWN HOLE: the guard keys off a `lookup` for the cozystack.ouroboros
# Package CR on the cluster. An operator who manually
# `kubectl delete package cozystack.ouroboros` before flipping the
# flag back bypasses the check entirely (the lookup returns nil, the
# render proceeds). In the chart 0.8.0 that path is mostly safe —
# `kubectl delete package` triggers helm uninstall and the chart's
# pre-delete hook patches the Corefile automatically. The hole is
# the rare case where the hook itself fails (controller pod stuck,
# ConfigMap RBAC drift, hook timeout) and the operator never
# noticed. The values comment + the acknowledge gate are
# defence-in-depth, not an airtight lock; an operator deliberately
# working around them gets to keep the broken pieces.
proxyProtocolAcknowledgeUnclean: false
certificates:
solver: http01 # "http01" or "dns01"
issuerName: letsencrypt-prod
# Shared wildcard issuance on the default ingress-nginx path
# (gateway.enabled=false). Opt-in, OFF by default. When set to true
# with solver=dns01 and no operator-provided wildcardSecretName, the
# platform issues ONE wildcard Certificate (*.<root-host> +
# <root-host>) via the DNS-01 ClusterIssuer and serves it as the
# ingress controller's default SSL certificate, so system services
# (dashboard, grafana, keycloak, harbor, …) stop minting a per-host
# ACME cert each. This avoids the Let's Encrypt per-domain rate limit
# once a deployment exceeds ~50 endpoints, matching the Gateway API
# path which issues a per-apex wildcard in dns01 mode.
#
# Ignored when solver=http01 (HTTP-01 cannot issue wildcards) and
# when gateway.enabled=true (the TenantGateway controller issues the
# wildcard there).
#
# COVERAGE CAVEAT — same contract as wildcardSecretName below: a
# single-label wildcard *.<root-host> covers <service>.<root-host>
# but NOT a custom service host outside it (e.g. a keycloak
# ingress.host / harbor host / grafana host set to auth.example.com,
# or a nested a.b.<root-host>). When this is true, services whose
# host is not covered fall back to the default SSL certificate and
# would serve the wrong certificate. Only enable it when every
# exposed system service uses a <service>.<root-host> host (the
# default), or provide a covering wildcardSecretName instead. Left
# OFF by default precisely so a dns01 cluster with a custom service
# host is never silently switched on upgrade.
#
# BLAST RADIUS — like wildcardSecretName, the chosen name rides the
# _cluster channel, which every child tenant inherits verbatim. A
# child tenant that inherits the root ingress has nested service
# hosts (<service>.<tenant>.<root-host>) that *.<root-host> does NOT
# cover; enabling this makes those services drop per-host ACME and
# serve the root default certificate too. This is not root-tenant
# scoped — only enable it when child-tenant hosts are also covered
# (or left on the Gateway path). Per-tenant wildcards are tracked
# separately.
#
# FIRST-ISSUANCE WINDOW: when this is turned on, consumers drop their
# per-host certs immediately, but the DNS-01 wildcard takes a few
# minutes to issue. Until the Secret is populated ingress-nginx serves
# its self-signed default cert (browser warning), then self-heals —
# the same delay per-host ACME has on first issue, which an
# operator-provided wildcardSecretName avoids only by pre-existing.
wildcard: false
# Operator-provided wildcard certificate. When set, platform
# services and the root tenant's ingress/Gateway serve under this
# pre-existing TLS Secret instead of minting per-host ACME certs.
# The Secret must already exist in the publishing namespace — the
# one that runs the root ingress controller / Gateway, which follows
# publishing.ingressName (tenant-root by default) — be of type
# kubernetes.io/tls, and cover the root host (typically a wildcard
# *.<root-host> + <root-host>). Only the
# NAME is distributed through the platform values channel — the
# cert/key material is never broadcast. Leave empty to keep ACME
# issuance (solver/issuerName above).
#
# Child tenants are covered automatically: the platform controller
# replicates this Secret into every tenant namespace that runs its own
# ingress controller, and into Gateway owners as long as their Gateway
# terminates TLS itself (see gateway.edgeTerminatedClasses below), so each
# per-tenant termination point serves the wildcard from its own
# namespace (no cross-namespace Secret
# read, no widened tenant RBAC). No extra input — the same name set
# here drives both the consumers and the replication. The replica
# carries the wildcard private key into each terminating tenant
# namespace (the same exposure as the per-host ACME Secret it replaces,
# now shared across tenants), and covers a child apex only if the
# certificate's SAN list does — a single *.<root-host> does not match
# *.<child-apex>, so supply SANs covering the child apexes you serve.
# On a freshly-created tenant the replica may land a moment after the
# tenant's ingress controller starts. The window is bounded: the
# platform controller creates the replica within one reconcile, and
# ingress-nginx watches its default-ssl-certificate Secret and adopts
# it when it appears (no restart needed). Until then ingress-nginx logs
# a warning and serves its built-in certificate.
wildcardSecretName: ""
# Rate-limit note: every tenant with gateway=true issues one
# Certificate for its wildcard + apex hostnames. If many tenants share
# the same apex domain with issuerName=letsencrypt-prod, the cluster
# can hit the Let's Encrypt limits (50 certs / registered domain /
# week, 5 duplicate certs / week, 300 new orders / account / 3h).
# Use letsencrypt-stage for non-production clusters or gate the
# number of certificates per tenant via tenant.spec.resourceQuotas
# with count/certificates.cert-manager.io. See
# https://letsencrypt.org/docs/rate-limits/.
#
# DNS-01 solver configuration. Read only when solver=dns01. The
# provider key selects which sub-block is consumed; others are
# ignored. Cloudflare keeps backwards-compatible defaults so the
# most common opt-in path (cloudflare token + dns01) needs only
# solver=dns01 + dns01.cloudflare.secretName.
dns01:
provider: cloudflare # cloudflare | route53 | digitalocean | rfc2136
cloudflare:
secretName: cloudflare-api-token-secret
secretKey: api-token
route53:
region: ""
accessKeyID: ""
secretName: ""
secretKey: secret-access-key
digitalocean:
secretName: digitalocean-api-token-secret
secretKey: access-token
rfc2136:
nameserver: ""
tsigKeyName: ""
tsigAlgorithm: HMACSHA256
secretName: ""
secretKey: tsig-secret-key
# Telemetry reported by platform components.
#
# Two components report: cozystack-operator (cluster facts — nodes, storage,
# packages) and cozystack-controller (application counts). This key covers the
# CONTROLLER, which the platform deploys. The operator is deployed by the
# cozy-installer chart and is switched off separately, with
# cozystackOperator.disableTelemetry=true — a full opt-out needs both, because
# the two components are installed by two different charts.
#
# What is collected, and why, is documented at
# https://cozystack.io/docs/operations/configuration/telemetry/
telemetry:
disabled: false
# Authentication configuration
authentication:
oidc:
enabled: false
insecureSkipVerify: false
keycloakExtraRedirectUri: ""
# Technical name of the Keycloak realm that owns the cozystack clients and
# the cozystack-cluster-admin roles. keycloak-configure creates the realm
# under this name, and every SSO consumer (dashboard, linstor-gui, the
# generated kubeconfig, and the derived keycloakInternalUrl below) points at
# /realms/<realmName>. The apiserver OIDC issuer is set out-of-band via talm
# oidcIssuerUrl and must carry the same realm, or the tokens minted here are
# rejected. Rename it in one place; keep it in step with talm.
#
# Set this at install time. Changing it on a running cluster does not rename
# the realm: the keycloak-operator provisions a fresh, empty realm under the
# new name and abandons the old one, so every account, session and client
# living in the previous realm is orphaned. Treat a rename on a live cluster
# as a from-scratch re-provision of identities, not an in-place migration.
#
# "master" is refused: that realm holds the Keycloak super-admin, so naming
# it here would put the platform clients, the cluster-admin roles and every
# end user inside it.
realmName: "cozy"
# In-cluster URL of the KeyCloak realm, used for backend-to-backend requests. oauth2-proxy runs
# with --skip-oidc-discovery and sends every backend call (token, jwks, userinfo, logout) to this
# URL, while browser-facing redirects keep pointing at the external host.
#
# This is the default path: the bundled KeyCloak is the only supported provider, so those calls
# have no reason to leave the cluster. Taking the external hostname instead would route them
# through public DNS and the root ingress — which does not serve yet on a fresh install, leaving
# the dashboard permanently down.
#
# The keycloak-http Service has no TLS listener, so these backend calls travel as plaintext HTTP
# between pods, as keycloak-configure's own admin calls to the same address already do.
#
# Override only for a KeyCloak that does not live at the standard in-cluster address. Setting it
# to an empty string restores full OIDC discovery against the external issuer.
#
# Left unset (null), it is derived from realmName as
# http://keycloak-http.cozy-keycloak.svc:8080/realms/<realmName>, so a realm rename reaches these
# backend calls automatically. The tri-state is: null -> derive from realmName; "" -> external
# OIDC discovery; any other value -> used verbatim. An explicit value is not rewritten, so a
# custom realm paired with an explicit URL must name the same realm.
keycloakInternalUrl: null
# Gateway API configuration
gateway:
# Enable Gateway API support across the platform. When true:
# - cert-manager ClusterIssuers use an http01.gatewayHTTPRoute solver
# that attaches to the root tenant's Gateway, so Certificate issuance
# works for tenants whose ingress is via Gateway API rather than
# ingress-nginx.
# - Exposed services whose TLS the Gateway terminates (dashboard,
# keycloak, harbor, bucket, grafana) render an HTTPRoute attached to
# the tenant's Gateway.
# - The three that present a certificate of their own (api,
# vm-exportproxy, cdi-uploadproxy) render a TLSRoute instead, pinned
# by sectionName to the tls-<name> TLS-passthrough listener the
# per-tenant gateway chart declares for it. The cozystack-api,
# kubevirt and kubevirt-cdi packages ship one each, cluster-wide,
# attached to the Gateway in the namespace publishing.ingressName
# selects.
# Per-tenant opt-in is still governed by tenant.spec.gateway=true, which
# materialises the Gateway itself. This flag only controls platform-wide
# Gateway API integration (cert-manager solver + the per-service routes).
enabled: false
# Default GatewayClass for tenant Gateways. A tenant picks another one
# with `gatewayClass` on its Tenant resource; this is the value used when
# it does not. The default is the Cilium class the platform ships. Reaches
# the per-tenant gateway chart as _cluster.gateway-class-name.
className: cilium
# GatewayClasses whose provider terminates TLS upstream of the Gateway
# (a Cloudflare Tunnel class ends TLS at the Cloudflare edge and reaches
# the Gateway over the tunnel). TLS termination is a property of the
# class, not of the tenant, so it is declared here once and every tenant
# that selects such a class gets it: the TenantGateway controller then
# issues no certificates for that Gateway, renders plain HTTP listeners
# for the apex and its wildcards, and skips the TLS-passthrough listeners
# — which leaves the tenant Kubernetes API, VM export and CDI upload
# endpoints unpublished through it, their TLSRoutes matching no listener.
#
# Listing a class here is a claim about that class and nothing validates
# it: name the bundled `cilium` and its Gateways publish every app
# hostname over plain HTTP on a public address, with no redirect and no
# certificate. Only list a class whose provider really does terminate TLS
# in front of the Gateway.
#
# Moving an existing tenant onto such a class also withdraws the operator
# wildcard from it: its namespace stops counting as a TLS termination
# point, so the replica of publishing.certificates.wildcardSecretName is
# deleted from it on the next reconcile. Ingress owners keep theirs, and a
# missing source Secret postpones the deletion rather than forcing it.
# Reaches the per-tenant gateway chart as _cluster.gateway-edge-classes.
edgeTerminatedClasses: []
# GatewayClasses a tenant may select for its own Gateway through
# `tenant.spec.gatewayClass`, on top of className above (which every
# tenant may always end up on, since it is the default). A tenant asking
# for anything outside this set fails its own gateway release with a
# message naming the class, and reaches no other tenant.
#
# This is an allowlist because Tenant is a tenant-writable object and a
# GatewayClass is cluster-scoped: without it a tenant could attach its
# Gateway to any class the cluster registers, including one an admin
# keeps for internal-only or differently-fronted traffic. Empty (the
# default) means tenants cannot choose at all and every Gateway lands on
# className. The allowed set is this list plus the CURRENT className, so
# changing className while a tenant pins the previous one by name fails
# that tenant's gateway release until it is updated too — list the old
# class here when moving the default. Reaches the per-tenant gateway
# chart as _cluster.gateway-tenant-classes.
tenantSelectableClasses: []
# Advertise HTTP/2 on every Gateway API listener served by the bundled
# Cilium dataplane. Browsers negotiate HTTP/2 exclusively through TLS ALPN
# "h2"; with this off, Envoy advertises no ALPN protocols and every client
# silently falls back to HTTP/1.1 — no error anywhere, just head-of-line
# blocking and the per-origin connection limit that HTTP/2 removes.
# ingress-nginx (which the Gateway path replaces) advertised h2 out of the
# box, so on-by-default restores the pre-migration behaviour. ALPN offers
# h2 then http/1.1, so clients without HTTP/2 keep working unchanged.
#
# Frontend hop only: gateway↔backend connections stay HTTP/1.1 unless a
# Service opts in per GEP-1911 by declaring appProtocol
# kubernetes.io/h2c on its port (that backend-protocol support is switched
# on together with ALPN; no cozystack-shipped Service declares such a
# value today, so backends see no change from this flag alone).
#
# Cluster-wide: maps to Cilium's enable-gateway-api-alpn agent config, so
# it covers the root and every tenant Gateway at once — there is no
# per-Gateway granularity in the dataplane. Only effective on bundles
# where cozystack manages Cilium (isp-full*, isp-slim*); on
# isp-hosted* the hosting-side CNI owns this setting. Flipping it re-rolls
# the cilium DaemonSet on the next platform upgrade (rollOutCiliumPods),
# the same disruption profile as any cilium config change.
http2: true
# Namespaces that are allowed to attach HTTPRoute or TLSRoute to a tenant
# Gateway. The publishing tenant namespace is always included implicitly.
# Additional entries must be the exact namespace name. The controller
# grants the access by stamping the namespace.cozystack.io/gateway label
# on each listed namespace; writing that label needs patch on Namespace,
# which no tenant role holds. The label reaches the HTTPS-terminate and
# the port-443 passthrough listeners. The port-80 listener and the
# native-port listeners from tlsPassthroughListeners select on the
# built-in kubernetes.io/metadata.name label instead: the native-port
# ones name the publishing tenant alone, so listing a namespace here
# does not let it attach to a database port, while the port-80 listener
# names the tenant plus cozy-cert-manager, where the ACME challenge
# route lands. Tenants not on this list cannot attach a route to
# somebody else's Gateway, which prevents hostname hijacking across
# tenants.
#
# The `default` namespace is included because the Kubernetes API's
# TLSRoute (shipped by the cozystack-api package) has to live next to
# the `kubernetes` Service it points at, which is always in `default`.
attachedNamespaces:
- cozy-cert-manager
- cozy-dashboard
- cozy-keycloak
- cozy-system
- cozy-harbor
- cozy-bucket
- cozy-kubevirt
- cozy-kubevirt-cdi
- cozy-monitoring
- cozy-linstor-gui
- default
# Platform monitoring configuration.
monitoring:
# Whether the root tenant (tenant-root) hosts the platform
# VictoriaMetrics/VictoriaLogs stack, i.e. Tenant/root spec.monitoring=true.
# The platform centralises metrics and logs in tenant-root, and
# cozystack-basics publishes vlinsert-generic / vminsert-* ExternalName
# Services in cozy-monitoring that redirect there. When the root tenant has
# monitoring disabled those redirects dead-end and fluent-bit silently drops
# logs (issue #3181), so the ExternalNames are only created when this is
# true. This is the single render-time source of truth for that gate:
# changing it re-renders cozystack-basics (helm-controller watches the
# cozystack-values Secret it flows into). Nothing in the charts sets
# Tenant/root spec.monitoring — it is a runtime choice on the Tenant CR
# (default false; the standard install enables it out-of-band). The default
# here is true only to preserve the historical always-create behaviour;
# operators MUST set it to match their actual Tenant/root spec.monitoring.
rootEnabled: true
# The rate, in bytes per second counted uncompressed and before the
# collector samples, at which each tenant using shared-central tracing may
# write into the traces store tenant-root hosts; each tenant's collector
# refuses data above it, with a 429 an OTLP/HTTP sender retries and a
# RESOURCE_EXHAUSTED an OTLP/gRPC sender drops, so central tenants should
# export over HTTP; a request above 3 MiB gets a 400, not retried. 0 keeps
# the default of about 97 KiB/s: over the two days VictoriaTraces always
# keeps, one tenant writes at most the disk cap of a default 10Gi store,
# counted uncompressed, and the store keeps it compressed and only the
# sampled share of it (samplingPercentage). Each collector pod starts with a
# full bucket, so a pod start lets through above the rate up to 5 MiB, or
# one second of the pod's share of the rate when that is larger. Raise it
# with the store's size, as a whole number up to 1099511627776 (1 TiB/s);
# anything else fails the render. A change upgrades every release in every
# tenant once, since it travels in the cozystack-values Secret they all
# read.
tracingCentralTenantBytesPerSecond: 0
# Backup storage overrides for the platform-managed cozy-default BackupClass.
# Everything set here is forwarded verbatim into the emitted
# cozystack.backupstrategy-controller Package CR as
# components.backupstrategy-controller.values.backupStorage, so it lands on the
# backupstrategy-controller HelmRelease and deep-merges over that chart's own
# backupStorage defaults (packages/system/backupstrategy-controller/values.yaml
# documents every knob: provisionBucket, bucketName, endpoint, region,
# forcePathStyle, systemSecretName, systemNamespaces, …). Admins set this via
# spec.components.platform.values.backupStorage on the
# cozystack.cozystack-platform Package — see docs/operations/backup-classes.md.
# Empty (the default) forwards nothing and keeps the child chart defaults.
backupStorage: {}
# backupStrategyController overrides for the platform-managed strategy CRs,
# forwarded verbatim into the same Package as
# components.backupstrategy-controller.values.backupStrategyController. The
# knob that needs this is redisBackupResources: the shipped cozy-default-redis
# Strategy CR is Helm-managed (a direct edit reverts on upgrade), so a tenant
# with a large Redis raises the restore Pod's memory / ephemeral-storage here,
# via spec.components.platform.values.backupStrategyController.redisBackupResources
# on the cozystack.cozystack-platform Package. Empty forwards nothing.
backupStrategyController: {}
# Overrides for the optional cozystack.kubernetes-worker-image package (the
# golden Talos worker OS images tenant Kubernetes node pools clone instead of
# each worker importing the raw image over HTTP). Everything set here is
# forwarded verbatim into the emitted Package CR as
# components.kubernetes-worker-image.values, so it lands on that chart's
# HelmRelease and deep-merges over its defaults
# (packages/system/kubernetes-worker-image/values.yaml documents every knob:
# imageFactoryURL, storageClass, storage, images). Admins set this via
# spec.components.platform.values.kubernetesWorkerImage on the
# cozystack.cozystack-platform Package, alongside adding
# cozystack.kubernetes-worker-image to bundles.enabledPackages -- the package is
# opt-in and this key alone does not enable it. The two knobs that decide
# whether the feature is usable at all are imageFactoryURL (an air-gapped or
# rate-limited cluster must repoint the single import) and storageClass (CDI
# cannot clone across StorageClasses, so a cluster whose worker pools are not on
# `replicated` must move the golden onto their class). Empty (the default)
# forwards nothing and keeps the child chart defaults.
kubernetesWorkerImage: {}
# Pod scheduling configuration
scheduling:
globalAppTopologySpreadConstraints: ""
# UI branding configuration. The whole map is propagated verbatim into
# _cluster.branding (the cozystack-values Secret) and consumed by the dashboard
# (config.json) and Keycloak realm provisioning.
#
# Recognized keys:
# brandName: realm displayName (the Keycloak login title reads it as
# "Sign in to <brandName>").
# brandHtmlName: realm displayNameHtml (HTML shown in the login header).
# loginTheme: Keycloak login theme name; selected as the realm login
# theme and used as the theme directory/name.
# loginThemeImage: OCI image carrying that theme under /themes/. When set
# together with loginTheme, the keycloak chart mounts it via
# an init container, so the theme ships without patching the
# StatefulSet or the keycloak package values.
#
# Example:
# branding:
# brandName: "Acme Platform"
# brandHtmlName: "<b>Acme</b> Platform"
# loginTheme: acme
# loginThemeImage: registry.example.com/acme-keycloak-theme:v1
branding: {}
# Container registry mirrors configuration
#
# Example:
# registries:
# mirrors:
# docker.io:
# endpoints:
# - http://10.0.0.1:8082
# ghcr.io:
# endpoints:
# - http://10.0.0.1:8083
# gcr.io:
# endpoints:
# - http://10.0.0.1:8084
# registry.k8s.io:
# endpoints:
# - http://10.0.0.1:8085
# quay.io:
# endpoints:
# - http://10.0.0.1:8086
# cr.fluentbit.io:
# endpoints:
# - http://10.0.0.1:8087
# docker-registry3.mariadb.com:
# endpoints:
# - http://10.0.0.1:8088
# config:
# "10.0.0.1:8082":
# tls:
# insecureSkipVerify: true
registries: {}
# Resource allocation ratios
resources:
cpuAllocationRatio: 10
memoryAllocationRatio: 1
ephemeralStorageAllocationRatio: 40
# KubeVirt CR settings that no other platform knob implies. Forwarded into the
# emitted cozystack.kubevirt Package CR as components.kubevirt.values, next to
# the entries the iaas bundle derives from resources.cpuAllocationRatio and
# from the gpu block below.
kubevirt:
# Feature gates switched off under
# spec.configuration.developerConfiguration.disabledFeatureGates, on top of
# the opt-outs the kubevirt chart pins: Template and
# ExternalNetResourceInjection always, ImageVolume below Kubernetes 1.35.
# KubeVirt 1.9 turns every Beta gate on by default, so this is how a cluster
# declines one it cannot carry — a container runtime without image volume
# support on Kubernetes 1.35 or newer is the case this list is for.
#
# kubevirt:
# disabledFeatureGates:
# - ImageVolume
#
# A gate the chart already enables cannot be switched off here. KubeVirt
# refuses a CR that names one gate in both lists, so the chart stops the
# render and names the gate rather than ship a CR the cluster would reject
# on update and misread on install.
disabledFeatureGates: []
# Passed to spec.configuration.migrations on the KubeVirt CR.
# Empty keeps KubeVirt's defaults. bandwidthPerMigration is bytes/second.
migrations: {}
# GPU host-device wiring for VirtualMachine workloads. Merged into the KubeVirt
# CR's permittedHostDevices when bundles.enabledPackages contains
# cozystack.gpu-operator and bundles.iaas.gpuOperatorVariant is one of the two
# VM variants, default or vgpu. The platform's own defaults join it there: the
# passthrough table in the default variant, the mdev table in vgpu only when
# gpu.vgpuDeviceManager.enabled is true, and neither when replaceDefaults below
# is true. So a stock vgpu cluster that sets nothing here gets the HostDevices
# gate and no device table at all. The container variant serves GPUs to pods
# and never to VMs, so nothing in this block reaches the KubeVirt CR under it.
gpu:
# When false (default), the lists below are appended to the platform's
# NVIDIA defaults. When true, the user lists fully replace the defaults
# — use this for clusters with only non-NVIDIA accelerators or where a
# strictly curated allowlist is required.
replaceDefaults: false
permittedHostDevices:
# PCI passthrough devices. Rendered when non-empty under either VM
# variant — pciHostDevices covers both the vfio-pci passthrough path
# (default variant) and the post-kubevirt#16890 SR-IOV vGPU path
# (vgpu variant on Ada Lovelace / Blackwell, where VFs are advertised
# via pciHostDevices, not via mediatedDevices). The container variant
# renders neither list. Default-table coverage:
# H100, H200, A100 PCIe/SXM, A40, A30, A10, L40, L40S, L4, T4, V100,
# V100S (PCI vendor 10DE). Item shape matches KubeVirt's pciHostDevices:
# - pciVendorSelector: "10DE:2335"
# resourceName: nvidia.com/GH100_H200_SXM_141GB
# externalResourceProvider: true
pciHostDevices: []
# Mediated (mdev) vGPU devices — pre-kubevirt#16890 path used by
# Pascal–Ampere on the legacy mdev driver branch. Default-table coverage:
# A100-40C/80C, A40-24Q/48Q, A30-24C, A10-24Q, V100D-32C, T4-16Q.
# Item shape matches KubeVirt's mediatedDevices:
# - mdevNameSelector: NVIDIA A100-40C
# resourceName: nvidia.com/A100-40C
# externalResourceProvider: true
mediatedDevices: []
# mediatedDevicesConfiguration makes KubeVirt itself create mdevs on the
# node (vgpu variant only). The platform ships NO default here: mdev
# creation is delegated to the GPU Operator's vGPU Device Manager
# (name-based), and the mediatedDeviceTypes ids below are numeric
# nvidia-NNN sysfs indices that vary per GPU SKU and driver branch — no
# value is portable, so look yours up on the host
# (/sys/bus/pci/devices/<BDF>/mdev_supported_types/*/name) before setting
# it. Supplying it opts into KubeVirt-driven creation on top of the
# Device Manager. Helm's mergeOverwrite semantics apply: a top-level key
# you supply REPLACES that key wholesale rather than appending. Shape
# matches KubeVirt's MediatedDevicesConfiguration:
# mediatedDeviceTypes:
# - nvidia-740
# nodeMediatedDeviceTypes:
# - nodeSelector:
# nvidia.com/gpu.product: NVIDIA-L40S
# mediatedDeviceTypes:
# - nvidia-740
mediatedDevicesConfiguration: {}
# NVIDIA vGPU Manager container image (vgpu variant only). The .run is
# proprietary, not redistributable, so the platform cannot ship default
# tags — the operator builds the container per
# https://github.com/NVIDIA/gpu-driver-container and supplies the
# private-registry coordinates here. The platform forwards these as
# components.gpu-operator.values.gpu-operator.vgpuManager to the emitted
# cozystack.gpu-operator Package CR when bundles.iaas.gpuOperatorVariant
# is vgpu. Forwarding triggers when ANY of repository, version, image
# (non-default), or imagePullSecrets is set — partial values still
# propagate so a private-registry-with-default-tag setup is not
# silently dropped.
vgpuManager:
repository: ""
image: vgpu-manager
version: ""
imagePullSecrets: []
# vGPU Device Manager controls whether nvidia.com's
# `vgpu-device-manager` DaemonSet is deployed (vgpu variant only). The
# DaemonSet walks /sys/class/mdev_bus/ to auto-create mediated devices
# from the mediatedDeviceTypes list. Default-off because that path
# only exists on Pascal–Ampere (V100, T4, A100, A30, …) with the mdev
# driver branch — on Ada Lovelace / Blackwell (L4, L40, L40S, B100)
# with the vGPU 17/20 driver branch the directory is missing and the
# DaemonSet crash-loops with "no parent devices found". Flip to true
# on Pascal–Ampere clusters; leave false on Ada/Blackwell and supply
# SR-IOV VFs via .gpu.permittedHostDevices.pciHostDevices instead.
# When true the platform also lets the default name-based mediatedDevices
# exposure list from files/gpu-vgpu-defaults.yaml flow into the KubeVirt
# CR; when false those defaults are suppressed (advertising mediated
# resources that nothing can ever allocate is a worse footgun than
# missing entries).
vgpuDeviceManager:
enabled: false
# VM import from external virtualization platforms (opt-in; see the iaas bundle).
vmImport:
# Container image carrying the VMware Virtual Disk Development Kit, used by
# the raw-copy transfer path. Cozystack neither ships nor mirrors it: it is
# built from a proprietary VMware SDK, so an operator who holds a licence
# builds the image, pushes it somewhere the cluster can pull from, and names
# it here. Only the reference travels to the controller, never a credential.
#
# Empty is the normal state and means the vSphere import path is unavailable
# rather than broken: a vSphere VMImportSource reports VDDKNotConfigured on
# its own status, so a tenant learns at registration time instead of watching
# a transfer die halfway. Non-VMware providers never need it.
#
# Named vmImport, not migration, to keep it clearly distinct from the
# top-level `migrations` key above, which is the platform's own schema
# migrations and has nothing to do with moving virtual machines.
vddkImage: ""