## What this PR does
Replace the ephemeral `emptyDisk` volume on tenant Kubernetes worker VMs
with persistent PVC-backed storage via KubeVirt `dataVolumeTemplates`.
Previously, kubelet certificates, kubeconfig, and containerd state were
lost on VM reboot, causing the node to fall out of the tenant cluster.
**Changes:**
- Rename `ephemeralStorage` field to `diskSize` in NodeGroup
configuration
- Replace `emptyDisk` volume with `dataVolumeTemplates` using CDI blank
source
- Add `cdi.kubevirt.io/storage.usePopulator: "false"` annotation to work
around CDI volume population issues with LINSTOR CSI
- Add optional per-nodeGroup `storageClass` field (defaults to cluster
default)
- Set fixed 2Gi ephemeral-storage resource for virt-launcher pod
- Rename internal mount paths from `/ephemeral` to `/persistent`
- Add helm unit tests (`cluster_test.yaml`) covering dataVolumeTemplates
defaults, custom disk size, storageClass presence/absence, fixed
ephemeral-storage, disk name consistency, and CDI populator annotation
- Quote storage values in templates (`diskSize`, `storageClassName`) for
Helm correctness
- Guard bind-mount commands with `mountpoint -q` checks and create
target directories to handle fresh images and cloud-init retries
- Set `blockSize.matchVolume` on worker node disks for automatic block
size detection, fixing DRBD 4Kn compatibility with QEMU
**Known gotchas:**
- CDI volume population (prime PVC rebind) causes `ClaimMisbound` errors
with LINSTOR CSI driver, regardless of StorageClass (`replicated` or
`local`). The `usePopulator: false` annotation switches to the legacy
import path which works correctly. This is a known CDI issue documented
in
[kubevirt/containerized-data-importer#3146](https://github.com/kubevirt/containerized-data-importer/issues/3146).
- The `replicated` StorageClass with DRBD `suspend-io` quorum policy can
cause transient VM IO errors during LINSTOR satellite restarts (quorum
loss). The `local` StorageClass is recommended for worker node disks.
When `storageClass` is not specified in a nodeGroup, the cluster default
StorageClass is used.
- Existing clusters will trigger a rolling replacement of all worker
nodes on upgrade due to changed KubevirtMachineTemplate hash.
- DRBD volumes use 4096-byte logical sectors (`block-size 4096`).
Without explicit block size configuration, QEMU defaults to 512-byte
sectors, causing immediate IO errors and VM pause. The
`blockSize.matchVolume` setting in the disk spec ensures KubeVirt
auto-detects the correct sector size from the volume.
**Tested on a live cluster:**
- Verified DataVolume provisioning with `local` StorageClass
- Verified nodes join the tenant cluster and reach Ready state
- Verified node survives VM reboot (pod deletion) and returns to Ready
with the same Node object, reusing kubelet certificates from persistent
disk
### Screenshots
N/A — no UI changes.
### Release note
```release-note
feat(kubernetes)!: replace ephemeral emptyDisk with persistent storage for worker nodes
Worker node VMs now use persistent PVC-backed disks instead of ephemeral emptyDisk volumes. Kubelet certificates, kubeconfig, and containerd state survive VM reboots, preventing nodes from falling out of the tenant cluster.
BREAKING CHANGE: The `ephemeralStorage` field in nodeGroups has been renamed to `diskSize`. A new optional `storageClass` field allows per-nodeGroup storage class selection. Users must update their values overrides accordingly.
Worker node disks now use `blockSize.matchVolume` to automatically detect the underlying volume's sector size, ensuring compatibility with DRBD-backed storage classes that use 4096-byte logical sectors.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Optional storageClass parameter to select storage class for worker
node persistent disks (defaults to cluster default).
* Makefile test target added to run Helm unit tests for the chart.
* **Configuration Changes**
* Worker node storage parameter renamed from ephemeralStorage to
diskSize (default: 20Gi).
* Kubelet/containerd state moved to persistent disk paths and node
ephemeral-storage accounting fixed to 2Gi; disk provisioning aligns
block sizing with backing volumes.
* **Documentation**
* Updated docs and chart/schema to reflect the persistent disk model and
new fields.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Update description in values.yaml with corrected denylist and re-run
cozyvalues-gen to propagate changes to values.schema.json, README.md,
and types.go.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
- Regenerate deepcopy artifacts to remove orphaned PostgreSQLParameters methods
- Update e2e test to use string value for max_connections parameter
- Fix template to avoid .Values mutation and use quote function for safe YAML escaping
Signed-off-by: ohotnikov.ivan <ohotnikov.ivan@e-queo.net>
Allow users to configure any PostgreSQL server parameter via the
postgresql.parameters field. All parameter values must be provided
as strings (numeric values should be quoted).
Security measures:
- Block parameters that allow shell command execution (archive_command,
restore_command, ssl_passphrase_command, log_destination, event_source)
- Validation implemented in templates to prevent security vulnerabilities
Generated files updated:
- types.go: Add Parameters field with map[string]string type
- README.md: Add parameter documentation and warnings
- values.schema.json: Update schema for new field
- CRD: Update OpenAPI schema with new field and reordered properties
CloudNativePG manages certain parameters automatically (archive_mode,
primary_conninfo, wal_level). Users should not override these as it
may break backup/replication functionality.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Integrate the HAMi GPU virtualization addon from main while
keeping the persistent-storage diskSize and storageClass fields
introduced by this branch.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
## Summary
Add `serverName` field to bootstrap configuration to support backup
recovery when the Barman server name in `backup.info` differs from the
Kubernetes cluster name.
This fixes "no target backup found" errors during CloudNativePG recovery
operations.
## Problem
CloudNativePG forms the backup path as `destinationPath + "postgres-" +
oldName`, but searches for backups using the `server_name` field from
`backup.info`. When these values don't match (e.g., cluster name is
`grafana` but `server_name` is `cloud`), recovery fails with "no target
backup found".
## Solution
- Add optional `serverName` parameter to `bootstrap` configuration
- When specified, CloudNativePG uses this value to search for backups in
S3
- Falls back to `oldName` when `serverName` is not provided (backwards
compatible)
## Changes
- Add `ServerName` field to PostgreSQL CRD type definition
- Add conditional `serverName` to Cluster `externalClusters` template
- Update `values.yaml` and README.md with `serverName` documentation
- Regenerate `values.schema.json` and `postgres.yaml` CRD
## Test plan
- [ ] Deploy postgres cluster with backups enabled
- [ ] Create backup
- [ ] Create new postgres instance with `bootstrap.enabled=true`,
`bootstrap.oldName=<original-name>`, and
`bootstrap.serverName=<value-from-backup.info>`
- [ ] Verify recovery completes successfully
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added an optional bootstrap.serverName setting to specify the Barman
server name from the original cluster’s backup.info (used when that
server name differs from the Kubernetes cluster name).
* **Documentation**
* Clarified that bootstrap.oldName must match the serverName value
recorded in backup.info; updated docs and examples to reflect the new
bootstrap.serverName option.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
The `vm-instance` chart now drives the cozy-proxy `wholeIP` and
`allowICMP` annotations explicitly so that `externalMethod: PortList`
actually filters ingress traffic to declared ports while keeping
ping/PMTU functional.
- Render `networking.cozystack.io/wholeIP: "false"` when
`externalMethod: PortList` (was always `"true"`, which silently disabled
the PortList semantics in cozy-proxy).
- Add `externalAllowICMP` value (default `true`) propagated as
`networking.cozystack.io/allowICMP` when `externalMethod: PortList`.
Without this, cozy-proxy drops ICMP in port-filter mode (ping/PMTU
broken). Operators can set `externalAllowICMP: false` to opt out.
The changelog entry is intentionally **not** part of this PR — it will
be added in a dedicated `docs: add changelog for vX.Y.Z` commit at
release time, per project convention.
## Why
`externalMethod: PortList` is documented as filtering ingress traffic to
declared ports but has been non-functional on Cozystack v1.3.0 —
verified empirically on a 3-node Talos lab. Root cause was twofold:
chart always set `wholeIP: "true"`, and cozy-proxy v0.2.0 had no
port-aware logic. The cozy-proxy side was fixed in
cozystack/cozy-proxy#11 (merged) and cozystack/cozy-proxy#12 (allowICMP
follow-up); this PR completes the user-visible fix on the chart side.
## Companion PRs
- cozystack/cozy-proxy#11 (merged) — per-service ingress port filtering
- cozystack/cozy-proxy#12 (merged) — `allowICMP` annotation for
port-filter mode
## Test plan
- [x] Built cozy-proxy with the companion fix locally, deployed on a
3-node Talos lab (Cozystack v1.3.0)
- [x] `wholeIP: "false"` Service with `spec.ports: [22]`: only port 22
reachable from outside; ports 80/443/8080/9999 filtered
- [x] WholeIP-annotated Service unchanged: all listening ports reachable
- [x] Egress IP preservation works in both modes (TCP curl + UDP DNS)
- [x] `nft list table ip cozy_proxy` confirms expected ruleset
- [x] `helm template` renders the expected annotation matrix: `PortList`
default → `wholeIP=false, allowICMP=true`; `PortList` opt-out →
`allowICMP=false`; `WholeIP` → only `wholeIP=true`
- [x] `make unit-tests` passes locally
- [ ] CI unit tests
- [ ] CI E2E
## Backport
Suggesting `backport-v1.3` once merged.
## Release note
```release-note
[vm-instance] Make `externalMethod: PortList` actually filter ingress traffic to ports listed in `externalPorts`. New `externalAllowICMP` knob (default true) propagates the cozy-proxy `allowICMP` annotation to keep ping/PMTU functional in port-filter mode. Combined with cozy-proxy v0.3.0+, only listed ports plus ICMP are reachable from the VM's LoadBalancer IP.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added `externalAllowICMP` configuration option to control ICMP traffic
acceptance for VM external access in PortList mode (enabled by default).
* **Documentation**
* Updated parameter documentation to include the new ICMP traffic
control setting.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
- Adds a pre-commit hook and a dedicated GitHub Actions workflow that
run `make generate` at the repo root to catch drift in generated API
code (CRDs, DeepCopy, clients, RBAC) before it lands on main.
- Pre-commit hook is scoped to paths that actually affect codegen
(`api/`, `pkg/apis/`, `hack/update-codegen.sh`,
`hack/boilerplate.go.txt`) so unrelated commits are not slowed down.
- CI workflow (`.github/workflows/codegen-drift.yml`) installs Go from
`go.mod`, runs `make generate`, and fails on drift with an error
pointing contributors to the local fix.
- Includes one drift fix the check surfaced: `RestoreJobSpec.Options`
(added in #2437) had no `DeepCopyInto` handling — regenerated.
### Release note
```release-note
ci(api): add pre-commit hook and GitHub Actions workflow that verify generated API code (CRDs, deepcopy, clients, RBAC) is in sync with `make generate`.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Added a CI workflow that checks generated code during PRs and fails
the build if generated artifacts diverge.
* Added a pre-commit hook that runs generation checks locally to prevent
committing outdated generated files.
* Fixed deep-copy behavior for backup restore job specs so nested
options are correctly duplicated.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
- Add `externalAllowICMP` value (default true) propagated as
`networking.cozystack.io/allowICMP` annotation on the rendered Service
when `externalMethod: PortList`. The cozy-proxy companion (released as
part of cozystack/cozy-proxy#11 + #12) drops ICMP by default in
port-filter mode, which breaks ping and PMTU discovery; defaulting the
chart to "true" preserves user expectations while still allowing
operators to opt out by setting `externalAllowICMP: false`.
- Remove the v1.3.1.md changelog entry. Project convention is to add
changelogs in a dedicated "docs: add changelog for vX.Y.Z" commit at
release time, not as part of feature/fix PRs.
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
Incorporate images.waitForKubeconfig support from main while
keeping the persistent-storage diskSize and storageClass fields
introduced by this branch.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
## What this PR does
Closes#2412. On a cold tenant-Kubernetes bootstrap, the parent
HelmRelease raced the admin-kubeconfig Secret that Kamaji provisions
asynchronously. Three CP-side Deployments (cluster-autoscaler, kccm,
kcsi-controller) mounted that Secret as a hard volume, flux
helm-controller's default wait budget was too short for Kamaji cold
start, and `install.remediation { retries: -1 }` then uninstalled the
Cluster CR and restarted the cycle forever.
Implements a defense-in-depth fix:
- `optional: true` on the admin-kubeconfig Secret volume in all three
Deployments so kubelet no longer FailedMounts while Kamaji is still
bootstrapping.
- A shared `wait-for-kubeconfig` init container (in
`templates/_helpers.tpl`) that polls for `super-admin.svc` with a 10m
deadline, strictly below the HelmRelease Install.Timeout so a broken
tenant falls into CrashLoopBackOff visibly instead of hanging forever.
- Per-Application HelmRelease Install/Upgrade timeout, driven by a new
`release.cozystack.io/helm-install-timeout` annotation on
ApplicationDefinition. Kubernetes-rd sets it to `15m`; other kinds leave
it unset and keep flux defaults, so their failed installs remediate on
the normal cadence. Parser rejects ns/us/µs (accepted by
`time.ParseDuration`, rejected by Flux's CRD pattern) at startup.
- Soft-skip when `_namespace.etcd` is empty: the CP-side Deployments,
the Cluster/KamajiControlPlane/KubevirtCluster/WorkloadMonitor CRs, and
every child HelmRelease that references admin-kubeconfig now render only
when an etcd DataStore exists for this tenant. An `awaiting-etcd`
ConfigMap is emitted as a user-visible status beacon so `helm install`
still succeeds and flux retries on its 5m interval until the Tenant
chart catches up.
- e2e remediation guard built on `.status.history[].status` (the
Snapshot shape), not on `.status.installFailures` - `ClearFailures()`
zeroes the latter on every successful reconciliation, which made the
previous guard vacuous.
Tests:
- Go unit tests for the annotation parser (accepted/rejected units) and
the HR builder (table-driven across kinds).
- helm unittest for the per-template structure (optional volume, init
container, dataStoreName, awaiting-etcd beacon).
- bats unit tests for the shell guard (every combination of
empty/zero/positive history entries, plus pinned HR v2 shape).
- Chart-wide bats invariants: every Deployment mounting admin-kubeconfig
has the guards; zero such Deployments and zero HelmReleases render when
etcd is empty.
All wired into the existing `make unit-tests` target (`go-unit-tests`
added alongside `helm-unit-tests` and `bats-unit-tests`).
Option 2 from the ticket (separate HelmRelease with `dependsOn`) was
intentionally not taken: the combination above closes the same race
without restructuring the chart's HelmRelease topology.
### Release note
```release-note
fix(kubernetes): close admin-kubeconfig race on tenant Kubernetes bootstrap. The parent HelmRelease no longer enters an uninstall/retry cycle when Kamaji control-plane cold start exceeds flux's default wait budget. A Kubernetes tenant created before the parent Tenant application has etcd enabled now renders only an awaiting-etcd beacon ConfigMap and waits quietly for the DataStore to appear, instead of producing half-installed Deployments that CrashLoopBackOff forever.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Per-application Helm install/upgrade timeout via metadata annotation.
* Init-container guards that wait for admin kubeconfig before workloads
start.
* Chart resources now render conditionally based on etcd presence.
* **Tests**
* Helm-template tests for admin-kubeconfig invariants and
remediation-cycle detection.
* New Go unit tests and CI Helm/unittest coverage plus test value files.
* **Chores**
* Added BusyBox image pin and new Makefile test targets (including Go
unit-tests).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Operators in air-gapped or rate-limited environments cannot reach
docker.io and the bundled busybox digest pin gives them no escape
hatch. Add an optional images.waitForKubeconfig chart value that, when
set, replaces the helper's image reference with any registry path
kubelet can pull. Empty value falls back to images/busybox.tag, so the
prior digest-pinned default is preserved.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add an opt-in UpgradeCRDs field to ComponentInstall that maps to
HelmRelease.Spec.Upgrade.CRDs, allowing a PackageSource component to
declare how Flux should handle CRDs from the chart's crds/ directory
on upgrade.
The helm-controller default on upgrade is Skip, which means new CRDs
added between chart versions never reach existing clusters and must be
applied manually. Setting upgradeCRDs: CreateReplace makes Flux apply
new CRDs declaratively with the chart.
Allowed values are restricted to Skip, Create, CreateReplace via a
kubebuilder enum marker. Empty / unset preserves the existing Flux
default, so all existing PackageSource resources keep working.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Specify that empty storageClass uses the management cluster default
StorageClass (annotated with is-default-class: true), not the
top-level chart storageClass value.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
The NodeGroup.StorageClass field lacked a +kubebuilder:default annotation,
causing a semantic inconsistency: the map-level default in ConfigSpec
included "storageClass": "" but CRD defaulting would not inject it at the
leaf level when a user specified a nodeGroup without storageClass.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Add storageClass: "" to nodeGroups.md0 in values.yaml so that the
code generator picks it up and places it into keysOrder for the
dashboard field ordering.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Allow each node group to specify its own storageClass for persistent
worker node disks. When empty, the cluster default StorageClass is used.
This enables using local storage for worker disks while keeping
replicated storage for tenant workloads.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Rename the ephemeralStorage field to diskSize in the NodeGroup typedef
and default values to reflect the upcoming switch from ephemeral
emptyDisk volumes to persistent dataVolumeTemplates for worker nodes.
BREAKING CHANGE: The nodeGroups.*.ephemeralStorage field has been renamed
to nodeGroups.*.diskSize. Users must update their values overrides.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
The Options field was added to RestoreJobSpec without re-running
'make generate', leaving zz_generated.deepcopy.go out of sync.
Regenerated to include the missing DeepCopyInto handling for the
runtime.RawExtension pointer.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Update oldName and serverName field descriptions based on code review
feedback to avoid confusion about their actual roles:
- oldName: Remove misleading "(matches serverName in backup.info)"
text. This field represents the Kubernetes cluster resource name,
not the Barman server name.
- serverName: Provide clearer explanation that it's the S3 path prefix
(barmanObjectStore.serverName) used by the original cluster, and should
only be set when it differs from the Kubernetes resource name.
Updated in:
- values.yaml (source of truth for field documentation)
- types.go (Go API type comments)
- values.schema.json (JSON schema for validation)
- postgres.yaml (CRD with embedded OpenAPI schema)
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Add serverName field to bootstrap configuration to explicitly specify
Barman server name from backup.info. This fixes "no target backup found"
errors when server_name in backup.info differs from Kubernetes cluster name.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Running make generate at repo root regenerates the zz_generated.deepcopy.go
files. Key fix: Source.DeepCopyInto in the vmdisk package now handles
the new Disk *SourceDisk pointer field, which was previously missing and
would cause pointer aliasing on deep-copy.
Other generated changes are incidental updates to vminstance (Networks
field added, Subnet renamed to Network) and backups CRD manifests picked
up by controller-gen from the current HEAD.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Run make generate to bring generated files up to date with current
API types. This was pre-existing staleness unrelated to any code change.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Upgrade CRDs in CozyStack dashboard controller
- Add Ingress proxy timeouts for WebSocket to work without terminations
- Add CFOMapping custom resource
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
The mysql chart actually deploys MariaDB via mariadb-operator, but was
incorrectly named "mysql". Rename all references to use the correct
"mariadb" name across the codebase.
Changes:
- Rename packages/apps/mysql -> packages/apps/mariadb
- Rename packages/system/mysql-rd -> packages/system/mariadb-rd
- Rename platform source and bundle references
- Update CRD kind from MySQL to MariaDB
- Update RBAC, e2e tests, backup controller tests
- Keep real MySQL CLI/config tool names unchanged (mysqldump, [mysqld], etc.)
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Aleksei Sviridkin