From b2c9efd31877ffdbba09fd570201af6bc3ed62ef Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Mon, 18 May 2026 20:41:01 +0500 Subject: [PATCH 1/3] docs: add changelog for v1.4.0 Signed-off-by: Myasnikov Daniil --- docs/changelogs/v1.4.0.md | 358 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 358 insertions(+) create mode 100644 docs/changelogs/v1.4.0.md diff --git a/docs/changelogs/v1.4.0.md b/docs/changelogs/v1.4.0.md new file mode 100644 index 000000000..1c59f0718 --- /dev/null +++ b/docs/changelogs/v1.4.0.md @@ -0,0 +1,358 @@ + + +# Cozystack v1.4.0 + +Cozystack v1.4.0 ships a **new schema-driven dashboard UI** (the openapi-ui + BFF stack is replaced by the rewritten cozystack-ui talking directly to the Kubernetes API), **persistent worker-node storage** for tenant Kubernetes (kubelet certs, kubeconfig, and containerd state now survive VM reboots), a unified **instance-type resource preset taxonomy** (`t1`/`c1`/`s1`/`u1`/`m1` × 8 sizes) with a migration that preserves legacy CPU/memory exactly, **declarative backup strategies** for PostgreSQL, MariaDB, ClickHouse, and FoundationDB, **HAMi-based fractional GPU sharing** as an optional system package, **ouroboros hairpin-NAT** wired into a single `publishing.proxyProtocol` switch, **per-app HelmRelease timeouts** that finally close the tenant-Kubernetes cold-bootstrap race, and **storage-class control per worker node group**. The release also rolls up every fix from v1.3.1 through v1.3.3. + +Platform components bumped in this release: **Talos v1.12.7 → v1.13.0**, **cert-manager v1.19.3 → v1.20.2**, **Cilium v1.19.1 → v1.19.3**, **NVIDIA GPU Operator v25.3.0 → v26.3.1**, **etcd-operator v0.4.2 → v0.4.3**, **KubeVirt v1.6.3 → v1.8.2**, **cozy-proxy v0.2.0 → v0.3.0**, **linstor-csi v1.10.6**, and the new **HAMi v2.8.1** and **ouroboros v0.7.2** packages. + +> **Note:** Items marked *(backported to v1.3.x)* were also shipped in v1.3.1, v1.3.2, or v1.3.3 patch releases. + +## Feature Highlights + +### New Dashboard UI: Direct Kubernetes-API Frontend + +The dashboard ships a **schema-driven rewrite** of the web console from [cozystack/cozystack-ui](https://github.com/cozystack/cozystack-ui). The old `openapi-ui` + BFF (Backend-For-Frontend) two-container stack is gone — the new React 19 + TypeScript UI talks directly to the Kubernetes API, eliminating an entire process and the proxying layer it required. + +The migration covers VNC WebSocket access for VirtualMachines (now using dynamic URLs instead of hardcoded `localhost:8001` so VNC works for any in-cluster deployment), an expanded RBAC bundle that adds read access to `ApplicationDefinition` resources for the catalog and marketplace, a **runtime branding system** that lets operators inject custom config via a ConfigMap (logos, names, brand colors) without rebuilding the image, **legacy URL redirects** so existing `/openapi-ui/*` bookmarks land at the new console, and **ConfigMap adoption + checksum-based rollouts** so dashboard upgrades cleanly inherit operator-managed branding. The dashboard is renamed end-to-end from `cozystack-dashboard` to `cozy-dashboard` for consistency with the rest of the platform ([**@IvanHunters**](https://github.com/IvanHunters) in #2507, [**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in dashboard hardening work). + +### Persistent Worker Node Storage for Tenant Kubernetes + +Tenant Kubernetes worker VMs now use **PVC-backed persistent disks** via KubeVirt `dataVolumeTemplates` instead of the previous ephemeral `emptyDisk` volume. Before this change, kubelet certificates, kubeconfig, and containerd state were wiped on every VM reboot — a node that rebooted lost its identity, fell out of the tenant cluster, and required manual rejoin. + +The change includes a **breaking rename** from `ephemeralStorage` to `diskSize` in the NodeGroup configuration, a new optional per-nodeGroup `storageClass` field (defaults to the cluster default StorageClass), and the `cdi.kubevirt.io/storage.usePopulator: "false"` annotation to work around CDI prime-PVC `ClaimMisbound` errors with LINSTOR CSI. `matchVolume` block-size detection on worker node disks fixes DRBD 4Kn compatibility with QEMU, which previously caused immediate IO errors and VM pause. Migration 39 rewrites legacy `ephemeralStorage` values to `diskSize` on upgrade; existing clusters trigger a one-time rolling replacement of all worker nodes due to the changed `KubevirtMachineTemplate` hash. The `local` StorageClass is recommended for worker node disks since DRBD `suspend-io` quorum policy can cause transient IO errors during LINSTOR satellite restarts ([**@Arsolitt**](https://github.com/Arsolitt) in #2454). + +### Instance-Type Resource Presets + +Resource presets now follow a cloud-style `.` taxonomy with **40 presets** across five CPU-to-memory series: + +| Series | Ratio | Use case | +| ------ | ----- | -------- | +| `t1` | 1:0.5 | Tiny / burstable, low memory | +| `c1` | 1:1 | Compute-balanced | +| `s1` | 1:2 | Standard — proxies, caches | +| `u1` | 1:4 | Universal — databases, messaging | +| `m1` | 1:8 | Memory — search, analytics | + +Each series ships eight sizes (`nano` → `4xlarge`). The seven legacy flat names (`nano` … `2xlarge`) are kept as deprecated aliases that map 1:1 by CPU and memory, so `helm template ... --set resourcesPreset=small` still renders `1 CPU / 512Mi` exactly as before. **Migration 39** walks every `HelmRelease.spec.values` and every app CR under `apps.cozystack.io/v1alpha1` and rewrites legacy values to their instance-type equivalents (`small → t1.small`, `medium → c1.small`, etc); the migration is idempotent and dry-runnable via `MIGRATION_DRY_RUN=1`. The `cozystack-api` emits deprecation warnings on every `Create`/`Update` against an app CR that still carries a legacy value, naming the JSON path and the 1:1 replacement. + +Operators see no behavioural change on upgrade — defaults preserve resources exactly. New deployments get the richer taxonomy and can select an explicit CPU/memory ratio per workload. See [Resource Presets](https://cozystack.io/docs/operations/resource-presets/) for the full size matrix ([**@lexfrei**](https://github.com/lexfrei) and [**@matthieu-robin**](https://github.com/matthieu-robin) in #2607). + +### Declarative Backup Strategies for Managed Apps + +The `backupstrategy-controller` is extended with four new strategy controllers — one each for **PostgreSQL** (CNPG), **MariaDB**, **ClickHouse** (Altinity-operator), and **FoundationDB**. Tenants declare a strategy plus a `BackupClass`, `Plan`, `BackupJob`, and `RestoreJob`; the controller composes the backend-specific resources (CNPG `Cluster.spec.backup`, ClickHouse `ClickHouseBackup`, MariaDB physical backup, FoundationDB) and reconciles ad-hoc snapshots, scheduled backups, in-place restores, and to-copy restores against an S3-compatible object store. + +Credentials are referenced by Kubernetes Secret name (never inline) and RBAC is tightened so the controller cannot read cluster-wide Secrets — only the ones explicitly referenced by a strategy. The new strategies join the existing per-app backup flows for VMInstance and VMDisk and complete the coverage matrix for cozystack's managed-app catalog ([**@androndo**](https://github.com/androndo) in #2552, #2598, #2605, #2606, #2645). + +### HAMi: Fractional GPU Sharing as an Optional System Package + +The new `hami` system package integrates [HAMi](https://github.com/Project-HAMi/HAMi) v2.8.1 (CNCF Sandbox) for **fractional GPU sharing in tenant clusters**. HAMi exposes per-tenant `nvidia.com/gpu`, `nvidia.com/gpumem`, and `nvidia.com/gpucores` resources so multiple pods can share a single physical GPU with explicit memory and compute slicing; the device plugin, scheduler extender, mutating webhook, and RuntimeClass are bundled. The integration auto-disables the GPU Operator's native device plugin to avoid conflict, and HAMi is exposed as an opt-in toggle (`hami.enabled`) on the tenant Kubernetes addon with a hard dependency on the GPU Operator. + +The release notes call out an important compatibility caveat: HAMi-core relies on a private glibc symbol (`_dl_sym`) removed in glibc 2.34, so compute isolation only works on container images that ship glibc < 2.34 (Ubuntu 22.04+, current PyTorch/TensorFlow official images do not). Alpine/musl is unsupported. Memory enforcement works on all images. See the package README and the [HAMi documentation](https://cozystack.io/docs/operations/kubernetes/hami/) for the supported image matrix ([**@Arsolitt**](https://github.com/Arsolitt) in #2484). + +### Ouroboros: One-Switch PROXY-Protocol + Hairpin-NAT Fix + +A new `publishing.proxyProtocol: true` switch turns on PROXY-protocol at the host ingress-nginx **and** auto-deploys [ouroboros](https://github.com/lexfrei/ouroboros) — a Go reimplementation of `compumike/hairpin-proxy` — to fix the resulting hairpin-NAT problem (intra-cluster traffic to the cluster's own public hostnames arrives at ingress-nginx without the PROXY header it now requires). + +ouroboros is exposed both as a host-level cozystack system package (`mode=coredns`) and as a per-tenant addon (`addons.ouroboros.enabled`, `mode=coredns-import`) that injects rewrite snippets into a `coredns-custom` ConfigMap without ever touching the chart-rendered Corefile. The chart auto-detects the cluster DNS domain from `/etc/resolv.conf` so it works on `cozy.local`, `cluster.local`, and any custom domain without operator override. Default behaviour is unchanged — clusters that don't enable PROXY-protocol get zero new resources ([**@lexfrei**](https://github.com/lexfrei) in #2561, #2596). + +### Per-App HelmRelease Timeouts and Operator Knobs + +The cozystack operator now exposes the HelmRelease generation knobs that have caused recurring pain across the project as both **operator flags** and **chart values**: `--helmrelease-interval` (default 5m), `--helmrelease-retry-interval` (default 30s, decoupled from Interval), `--helmrelease-install-timeout` (10m), `--helmrelease-upgrade-timeout` (10m), and `--helmrelease-max-history` (5). The retry strategy switches from `RemediateOnFailure` to `RetryOnFailure` so a slow first install no longer triggers an uninstall+reinstall loop, and Install/Upgrade `Remediation` is now left nil rather than set to `{Retries: -1}`. + +A companion change adds a **per-Application Install/Upgrade timeout** driven by a new `release.cozystack.io/helm-install-timeout` annotation on `ApplicationDefinition`. Kubernetes-rd sets it to `15m` so the parent tenant HelmRelease finally outlives Kamaji's cold-bootstrap window — the long-standing `wait hr/tenant-kubernetes timeout` failure on first install is gone. Other application kinds leave the annotation unset and keep Flux defaults ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2509, [**@lexfrei**](https://github.com/lexfrei) in #2413). + +### Kubelet Resource Reservations for Worker Nodes + +Tenant Kubernetes worker nodes now get **auto-computed kubelet reservations** so the OOM killer can't target kubelet itself under memory pressure. Memory: `systemReservedMemory` and `kubeReservedMemory` each default to 5% of effective node memory, clamped 256Mi-1Gi. CPU: `systemReservedCpu` and `kubeReservedCpu` each default to 5% of effective CPU, clamped 50m-500m. Hard eviction: configurable `evictionHardMemory` (7%) with validation that hard < soft (`evictionSoftMemory`, 10%). Disk eviction thresholds (`nodefs.available`, `imagefs.available`, `nodefs.inodesFree`) are preserved alongside memory signals. + +Cluster-autoscaler capacity annotations now report **allocatable** memory and CPU (total minus all reservations and eviction threshold) instead of raw totals, so autoscaling decisions match what the scheduler actually sees. Cgroup enforcement uses `enforce-node-allocatable=pods` — advisory reservations that the scheduler accounts for, without enforcing system-cgroup boundaries on host processes ([**@IvanHunters**](https://github.com/IvanHunters) in #2420). + +## Upgrade Notes and Required Actions + +Most operators can take v1.4.0 with no manual action — Cozystack stays on the same API surface and the in-platform migrations handle config rewrites automatically. + +* **Tenant Kubernetes worker nodes will roll once.** With #2454, `ephemeralStorage` in NodeGroup configuration is renamed to `diskSize`, and worker VMs switch from an ephemeral `emptyDisk` to a PVC-backed `dataVolumeTemplate`. **Migration 39 rewrites the field automatically** — you do not need to edit your Cluster CRs. The first reconcile after upgrade replaces all worker nodes one by one because the `KubevirtMachineTemplate` hash changes. Plan capacity for one extra worker per nodeGroup during the rollout and confirm the storage class you want (defaults to the cluster default; `local` is the safest pick for worker disks since the new persistent volumes are restart-resilient and DRBD `suspend-io` quorum policy can cause transient IO on `replicated` during LINSTOR satellite restarts). + +* **KubeVirt VMs running before the upgrade need a cold restart.** With #2502, KubeVirt jumps v1.6.3 → v1.8.2 which crosses the upstream QEMU bump (1.6.x → 1.7.x+). Live-migration of VMs that were running before the upgrade fails permanently with `qemu-kvm: error while loading state for instance ... virtio-net` / `operation failed: job 'migration in' failed: Operation not permitted` — this is upstream [kubevirt/kubevirt#16386](https://github.com/kubevirt/kubevirt/issues/16386). Switching `workloadUpdateMethods` from `[LiveMigrate, Evict]` to `[Evict]` does not help (the eviction webhook re-routes through live-migrate). **Action required:** after the platform upgrade, cold-restart pre-existing VMs (`virtctl stop` / `virtctl start`, or set `runStrategy: RerunOnFailure` and delete the VMI) so they relaunch on the new virt-launcher. New VMs created after the upgrade are unaffected. + +* **Resource presets gain new names; legacy names keep working.** With #2426 / Migration 39, presets follow a `.` taxonomy (`t1`/`c1`/`s1`/`u1`/`m1` × `nano`–`4xlarge`). The seven legacy flat names (`nano`, `micro`, `small`, `medium`, `large`, `xlarge`, `2xlarge`) **remain accepted as deprecated aliases** with the exact pre-rename CPU/memory by 1:1 mapping — no behavioural change for existing deployments. Stored values are rewritten on upgrade. You will see deprecation warnings on `Create`/`Update` against app CRs that still carry a legacy preset; migrate at your own pace before they are removed in a future release. + +* **PostgreSQL parameters are now type-checked.** With #2475, `postgresql.parameters` becomes a typed `map[string]string` with a case-insensitive denylist for parameters that would allow arbitrary code execution: `archive_command`, `restore_command`, `ssl_passphrase_command`, `dynamic_library_path`, and any `*_preload_libraries`. Common settings like `max_connections`, `shared_buffers`, `work_mem`, etc. are unaffected. If you relied on any of the denylisted parameters, the deployment will fail to render with a clear error and you must remove them. + +* **cert-manager pods now run as UID 65532.** Upstream cert-manager v1.20 changes the default container UID/GID from `1000/0` to `65532`. If you have custom PodSecurityPolicies, restrictive imagePullSecrets, or filesystem-mounted certs whose ownership is pinned to the old UID, update them. Operators who have not customised cert-manager security context need no action. + +## Platform Components + +* **Talos**: v1.12.7 → v1.13.0 (v1.12.7 was the final v1.3.x step, shipped in v1.3.2 via #2547 for CVE-2026-31431). v1.13.0 ships a Clang-built kernel with ThinLTO optimization, enables Container Device Interface by default for dynamic device-spec files under `/run/cdi`, adds a new `talosctl debug` subcommand for attaching to privileged debug containers, introduces the `EnvironmentConfig` document that replaces and deprecates `.machine.env`, adds virtiofs-based external volumes via `ExternalVolumeConfig`, machine-wide container-image signature verification via `ImageVerificationConfig`, and a new `LifecycleService` API for programmatic installs and upgrades. Component updates: Linux 6.18.24, containerd 2.2.3, etcd 3.6.9, CoreDNS 1.14.2, Kubernetes 1.36.0, CNI 1.9.1, runc 1.4.2, systemd 259.5. Built with Go 1.26.2. ([release notes](https://github.com/siderolabs/talos/releases/tag/v1.13.0)) ([**@kvaps**](https://github.com/kvaps) in #2546) + - Custom tooling that reads `EtcdConfigs`/`KubeletConfigs`/`ControllerManagerConfigs`/`SchedulerConfigs`/`APIServerConfigs` via the gRPC protobuf API: the wire format changed from `map` to `map` when consuming via `extraArgs` as a slice instead of a single string. + +* **cert-manager**: v1.19.3 → v1.20.2. The new ACME `acme.cert-manager.io/http01-ingress-ingressclassname` annotation lets operators override the `ingressClassName` field on HTTP-01 challenge solvers per-Ingress, OtherNames support is promoted to Beta, and Azure Private DNS becomes a first-class DNS-01 issuer. v1.20.2 also bumps Go to 1.26.2 to clear reported vulnerabilities and ships a moderate-severity fix for a controller panic on out-of-order DNS responses. The default container UID/GID changes to 65532 (previously 1000/0), so any custom PodSecurityPolicy or imagePullSecret bound to the old UID needs updating. ([release notes](https://github.com/cert-manager/cert-manager/releases/tag/v1.20.0)) ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2562) + +* **Cilium**: v1.19.1 → v1.19.3. Picks up upstream fixes for memory leaks during policy updates, NodePort connectivity in DSR mode when SocketLB is disabled, IPSec key-rotation races, identity-update hangs after endpoint deletion, GRPCRoute being silently excluded from Envoy config when a Gateway listener restricts `allowedRoutes.kinds`, and Hubble Relay panic on unresolvable peer addresses. Also fixes a dual-stack cluster-pool IPAM bug where an operator restart with a duplicate IPv6 PodCIDR could incorrectly free the affected node's IPv4 PodCIDR. ([release notes](https://github.com/cilium/cilium/releases/tag/v1.19.3)) ([**@lexfrei**](https://github.com/lexfrei) in #2464) + +* **NVIDIA GPU Operator**: v25.3.0 → v26.3.1. CDI is now enabled by default. The Cozystack package gains an experimental `vgpu` variant for **SR-IOV vGPU on Ada Lovelace and newer GPUs** with `defaultWorkload: vm-vgpu`, alongside the renamed `passthrough` variant (formerly `talos`) that pins `defaultWorkload: vm-passthrough`. The vGPU variant assumes the operator supplies a proprietary vGPU Manager image via Package CR component values (the `.run` cannot be redistributed). DCGM-exporter default counters refreshed to image tag `4.5.1-4.8.0` with `DCGM_FI_DEV_FB_RESERVED` and other new fields. ([release notes](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/release-notes.html#v26-3-1)) ([**@lexfrei**](https://github.com/lexfrei) in #2323) + +* **etcd-operator**: v0.4.2 → v0.4.3. Upstream ships two new CRDs — `EtcdBackup` (one-shot snapshot) and `EtcdBackupSchedule` (periodic snapshots) — plus the `EtcdCluster.spec.bootstrap` restore field. The cozystack `Etcd` app now exposes an optional `backup.*` block (`destinationPath`, `endpointURL`, `s3AccessKey`, `s3SecretKey`, `schedule`) mirroring the existing postgres backup pattern; render fails with a clear error if any required S3 field is empty or `destinationPath` is malformed. ([release notes](https://github.com/aenix-io/etcd-operator/releases/tag/v0.4.3)) ([**@lexfrei**](https://github.com/lexfrei) in #2428) + +* **KubeVirt**: v1.6.3 → v1.8.2. Two-minor jump crossing the upstream QEMU bump — VMs running before the upgrade need a cold restart (see Upgrade Notes). New CRD fields for changed block tracking (storage-agnostic incremental backups), `s390x` support added, `ppc64le` deprecated. The standalone ServiceMonitor / alerts templates are removed in favour of the Helm-rendered PrometheusRule. ([v1.7.0 notes](https://github.com/kubevirt/kubevirt/releases/tag/v1.7.0), [v1.8.0 notes](https://github.com/kubevirt/kubevirt/releases/tag/v1.8.0)) ([**@IvanHunters**](https://github.com/IvanHunters) in #2502) + - **VM Backup API + Changed Block Tracking** — new alpha `VMBackup` resource and `ChangedBlockTracking` matcher on `KubeVirt` enable storage-agnostic incremental backups; CBT also applies to hotplug volumes; libvirt checkpoints survive VM restart for resumable incremental flows. + - **Confidential VMs (experimental)**: Intel TDX support behind the `WorkloadEncryptionTDX` feature gate, AMD SEV-SNP behind `WorkloadEncruptionSEV` (alongside the existing SEV/SEV-ES paths). + - **VMPool gains a real lifecycle**: new `pool.kubevirt.io/v1beta1` API with auto-healing, Proactive/Opportunistic scale-in, statePreservation, UpdateStrategy with Selection policies, and proper finalizer cleanup. + - **Live-migration improvements**: dedicated mTLS certificate, migration allowed when host CPU model changes after a libvirt upgrade, decentralized live migration on L3 networks and across volumes with different volumeModes, live-migration for ImageVolume with modified container disk images. + - **Eviction / evacuation control**: new `/evacuate/cancel` subresource and `virtctl evacuate-cancel` command to abort an in-flight evacuation cycle; new `VirtualMachineInstanceEvictionRequested` condition for tracking. + - **Feature-gate promotions to Beta**: `ImageVolume`, `KubevirtSeccompProfile`, `IBM Secure Execution`, `VideoConfig`, `PanicDevices`, `passt` core binding with seamless migration. + - **Operator scaling**: `virt-api` replicas auto-scale by number of schedulable nodes; `virt-operator` now applies client rate limits (default 200 QPS / 400 burst), tunable via `--client-qps` / `--client-burst` or env vars. + - **virtctl breaking changes**: `--local-ssh` flag removed (native ssh/scp wrappers gone — host ssh/scp is always used); legacy dot syntax in `portforward`/`ssh`/`scp` dropped (`vm.namespace` no longer parsed); `virt-template` subresource and `virtctl virt-template` commands added. + - **API cleanup**: deprecated `instancetype.kubevirt.io/v1alpha{1,2}` API and CRDs removed; `MultiArchitecture` and `DisableMDEVConfiguration` feature gates deprecated; `foregroundDeleteVirtualMachine` replaced by the domain-qualified `kubevirt.io/foregroundDeleteVirtualMachine`. + - **Misc operator goodies**: screenshot capture without VNC, configurable `discard_granularity` for VM disks, NUMA-aware PCIe topology placement (alpha, `PCINUMAAwareTopology`), explicit feature-gate disable list via `developerConfiguration.disabledFeatureGates`. + - **Metric rename / cleanup**: `kubevirt_vm_container_free_memory_bytes_*` renamed to `kubevirt_vm_container_memory_request_margin_*`; new alerts `VirtualMachineStuckInUnhealthyState`, `VirtualMachineStuckOnNode`, plus guest vCPU queue alerts; `kubevirt_vmi_migration_disk_transfer_rate_bytes` corrected to `..._memory_transfer_rate_bytes`. + +* **HAMi**: new — v2.8.1. CNCF Sandbox project for GPU virtualization. See the Feature Highlights section above for the full description. ([**@Arsolitt**](https://github.com/Arsolitt) in #2484) + +* **ouroboros**: new — v0.7.2. Go reimplementation of `compumike/hairpin-proxy` for PROXY-protocol hairpin-NAT. v0.7.2 adds debug logging on tenants and tightens runtime cluster-domain auto-detection. See the Feature Highlights section above. ([**@lexfrei**](https://github.com/lexfrei) in #2561, #2596) + +* **cozy-proxy**: v0.2.0 → v0.3.0. Adds the `service.kubernetes.io/service-proxy-name` label-based selector (replacing label-or-annotation matching), an `allowICMP` annotation for the port-filter mode, disables probe and metrics endpoints by default, and ships per-service ingress port filtering. ([release notes](https://github.com/cozystack/cozy-proxy/releases/tag/v0.3.0)) ([**@kvaps**](https://github.com/kvaps) in #2510) + +* **linstor-csi**: bumped to v1.10.6 with the Protocol-C dual-attach fix. Live migration of KubeVirt VMs on Protocol-A/B (async) DRBD volumes no longer fails with `Protocol C required` — `linstor-csi` v1.10.6 installs a `Protocol=C` override on the resource-definition during dual-attach and reverts it on detach. `replicated-async` StorageClasses now support live migration without manual `drbdadm adjust` intervention *(backported to v1.3.1)* ([**@kvaps**](https://github.com/kvaps) in #2496). + +* **node-feature-discovery** (sub-chart of GPU Operator): v0.17.2 → v0.18.3. Adds official container images for `ppc64le` and `s390x` architectures, fixes the `test` subcommand of the kubectl-nfd plugin. (transitive via GPU Operator bump) + +* **hetzner-robotlb**: chart `appVersion` 0.0.5 → 0.0.6 (Renovate-driven refresh). ([**@sircthulhu**](https://github.com/sircthulhu) in #2465) + +## Major Features and Improvements + +* **[postgres] Support all PostgreSQL configuration parameters**: `postgresql.parameters` becomes a `map[string]string` so any PostgreSQL parameter can be set (previously only `max_connections` was applied; everything else was silently ignored despite CloudNativePG supporting them). A security denylist blocks parameters that enable arbitrary code execution as the `postgres` user: `archive_command`, `restore_command`, `ssl_passphrase_command`, `dynamic_library_path`, `local_preload_libraries`, `session_preload_libraries`, `shared_preload_libraries`. The denylist is case-insensitive. Integer values are accepted in the schema for backward compatibility. CRD, schema, README, and examples updated *(also adds the `bootstrap.serverName` parameter for backup recovery — backported to v1.3.3 via #2588)* ([**@IvanHunters**](https://github.com/IvanHunters) in #2466, #2362, backport #2588). + +* **[platform] Implement backup strategy for postgres**: Adds the CNPG-strategy controller for PostgreSQL backups with support for both in-place and to-copy restores against S3-compatible object stores. Credentials reference an existing Kubernetes Secret instead of being inlined; controller RBAC is constrained so it cannot read cluster-wide Secrets. Includes BackupClass, Plan, BackupJob, RestoreJob examples ([**@androndo**](https://github.com/androndo) in #2552). + +* **[platform] Implement backup strategy for clickhouse**: Altinity-operator-based backup strategy for ClickHouse applications, with the same controller pattern and credential model as the postgres strategy ([**@androndo**](https://github.com/androndo) in #2598). + +* **[platform] Implement backup strategy for mariadb**: Physical-backup strategy controller for the MariaDB application ([**@androndo**](https://github.com/androndo) in #2605). + +* **[platform] Add foundationdb backups**: Implementation of the backup-strategy controller for FoundationDB applications ([**@androndo**](https://github.com/androndo) in #2606). + +* **[platform] Align Altinity backup-strategy fields**: Renames and aligns Altinity backup-strategy spec fields for consistency with the other strategy controllers ([**@androndo**](https://github.com/androndo) in #2645). + +* **[apps] Instance-type resource presets with five series**: See Feature Highlights — adds 40 new presets (`t1`/`c1`/`s1`/`u1`/`m1` × 8 sizes) plus migration 39 that rewrites legacy values to the new form. Legacy aliases preserved exactly 1:1 ([**@lexfrei**](https://github.com/lexfrei) in #2607). + +* **[kubernetes] Replace ephemeral emptyDisk with persistent storage for worker nodes** (breaking): See Feature Highlights — PVC-backed dataVolumeTemplates, `ephemeralStorage` renamed to `diskSize`, per-nodeGroup `storageClass`. Triggers a rolling worker-node replacement on upgrade ([**@Arsolitt**](https://github.com/Arsolitt) in #2454). + +* **[kubernetes] Add kubelet resource reservations for worker nodes**: See Feature Highlights — auto-computed `systemReserved`/`kubeReserved` and configurable hard/soft eviction thresholds. Capacity annotations report allocatable instead of raw totals ([**@IvanHunters**](https://github.com/IvanHunters) in #2420). + +* **[networking] Add ouroboros for hairpin-NAT under PROXY-protocol**: See Feature Highlights — single `publishing.proxyProtocol: true` switch turns on PROXY-protocol at host ingress-nginx and deploys ouroboros to fix hairpin-NAT; tenants opt in via `addons.ouroboros.enabled`. Default unchanged ([**@lexfrei**](https://github.com/lexfrei) in #2561). + +* **[ouroboros] Bump to v0.7.2 with debug logging on tenants**: Refreshes the ouroboros chart to v0.7.2, adds debug logging on tenant deployments, and fixes a BATS test where `kubectl jsonpath` silently returns empty on bracket-notation paths with dotted keys ([**@lexfrei**](https://github.com/lexfrei) in #2596). + +* **[hami] Add HAMi GPU virtualization system package**: See Feature Highlights — CNCF Sandbox project for fractional GPU sharing. Optional toggle on the tenant Kubernetes addon, hard dependency on the GPU Operator ([**@Arsolitt**](https://github.com/Arsolitt) in #2484). + +* **[keycloak] Support extraEnv values and userProfile customization**: The `keycloak` chart now accepts `extraEnv` so operators can append extra `EnvVar` entries to the StatefulSet container without forking the chart (useful for custom themes that need runtime configuration or extensions that expect environment variables). The `keycloak-configure` chart exposes a top-level `userProfile` value that is mirrored verbatim into `ClusterKeycloakRealm.spec.userProfileConfig`, letting operators declare custom user-profile attributes (required fields, validators, permissions) declaratively. Both values default to empty so existing deployments are unaffected ([**@sircthulhu**](https://github.com/sircthulhu) in #2613). + +* **[etcd] Update operator to v0.4.3 and expose S3 backup schedules**: See Platform Components — `EtcdBackup` and `EtcdBackupSchedule` CRDs, `bootstrap` restore field, S3 credential block on the cozystack `Etcd` app ([**@lexfrei**](https://github.com/lexfrei) in #2428). + +* **[operator] Expose HelmRelease generation knobs**: See Feature Highlights — `--helmrelease-interval`, `--helmrelease-retry-interval`, `--helmrelease-install-timeout`, `--helmrelease-upgrade-timeout`, `--helmrelease-max-history` operator flags + chart values. Strategy switches to `RetryOnFailure` ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2509). + +* **[api] Per-Application HelmRelease Install/Upgrade Timeout**: New `release.cozystack.io/helm-install-timeout` annotation on `ApplicationDefinition` lets each application kind override the default Flux 15m wait budget. Kubernetes-rd sets it to `15m` so the parent tenant HelmRelease finally outlives Kamaji cold bootstrap; other kinds keep Flux defaults so failed installs remediate on the normal cycle ([**@lexfrei**](https://github.com/lexfrei) in #2413, refined by [**@myasnikovdaniil**](https://github.com/myasnikovdaniil) follow-ups). + +* **[operator] Add per-package upgradeCRDs policy for HelmRelease**: New per-package `upgradeCRDs` field on `PackageSource` controls whether Flux upgrades CRDs on Helm upgrade. Defaults preserve current behaviour; opt-in lets specific packages override the global policy ([**@lexfrei**](https://github.com/lexfrei) in #2427). + +* **[dashboard] Replace UI with new cozystack-ui build**: See Feature Highlights — full UI rewrite, direct Kubernetes API access (BFF removed), VNC dynamic URLs, ApplicationDefinition RBAC, runtime branding from ConfigMap, ConfigMap-adoption + checksum-based rollouts, legacy `/openapi-ui` redirect, package renamed to `cozy-dashboard` ([**@IvanHunters**](https://github.com/IvanHunters) in #2507, with dashboard hardening follow-ups from [**@myasnikovdaniil**](https://github.com/myasnikovdaniil)). + +* **[cozyreport] Collect Flux/cert-manager/host context + summary.txt**: `cozyreport.tgz` now bundles Flux controller logs (helm/source/notification/kustomize, last 2000 lines), Flux source resources, decoded Helm storage Secrets for non-Ready HRs, cluster events (all + warning-only), cert-manager `Certificate`/`CertificateRequest`/`Order`/`Challenge` and cert-manager logs, `cozystack-operator` logs (current + previous), `Application`/`ApplicationDefinition`/`Tenant` resources, and per-node host context (`df`, `free`, `ps`, `dmesg`, `talosctl logs/dmesg/kubelet/containerd`). A new `cozyreport-summary.sh` writes a `summary.txt` at the archive root listing what is broken right now — the first thing to read when downloading a CI failure artifact ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2553). + +* **[seaweedfs] Limit single PUT request size to 5GB**: Adds the `client_max_body_size` nginx annotation and `proxyBufferSize`/`proxyBuffering` knobs to the tenant SeaweedFS Ingress so single PUT requests are capped at 5GB instead of triggering arbitrary 413/504 patterns on large multipart uploads ([**@IvanHunters**](https://github.com/IvanHunters) in #2414). + +* **[monitoring] Add GPU observability dashboards and recording rules**: Five new Grafana dashboards — `gpu-performance` (per-GPU utilization, VRAM, power, temp, XID errors, throttle counters), `gpu-fleet` (cluster-wide inventory and per-node power as % of TDP), `gpu-tenants` (per-namespace allocation + 24-hour GPU-hour/kWh accounting), `gpu-quotas` (allocated vs. requested, pending pods, per-namespace requests/limits), `gpu-efficiency` (tensor saturation, util-per-watt, throttle fractions). Backed by VictoriaMetrics recording rules with a throttle-regression alert. Driven by the standard `$ds_prometheus` data-source variable and DCGM metrics ([**@Arsolitt**](https://github.com/Arsolitt) in #2418). + +* **[kilo] Add patched image build with cross mesh granularity**: New patched kilo image with a `cross` mesh granularity mode — WireGuard tunnels are created only between nodes in different locations, while intra-location traffic stays on the CNI overlay. Follows the kamaji vendor-and-patch pattern with reproducible builds via a pinned upstream commit SHA ([**@Arsolitt**](https://github.com/Arsolitt) in #2604). + +* **[vm-instance] Fix PortList not filtering ingress ports**: Adds the `externalAllowICMP` knob and fixes PortList filtering so external VM services correctly drop unlisted ingress ports under the new cozy-proxy v0.3.0 port-filter mode ([**@mattia-eleuteri**](https://github.com/mattia-eleuteri) in #2501). + +* **[cozy-proxy] Bump to v0.3.0**: See Platform Components ([**@kvaps**](https://github.com/kvaps) in #2510). + +* **[linstor] Bump linstor-csi to v1.10.6 with Protocol-C dual-attach fix**: See Platform Components ([**@kvaps**](https://github.com/kvaps) in #2496, *backported to v1.3.1*). + +## Bug Fixes + +* **[mongodb] Expose replica-set members per pod for external access**: When `external: true` was set on a non-sharded `MongoDB`, Cozystack previously rendered a single LoadBalancer Service whose selector matched all replica-set members; round-robin balancing sent writes to secondaries (`MongoServerError: not primary`) and drivers couldn't recover because the LB returned in-cluster DNS names for the primary. The hand-written `external-svc.yaml` is replaced with the Percona operator's native `replsets[].expose` feature, which creates one LoadBalancer Service per pod with `statefulset.kubernetes.io/pod-name` selectors and rewrites member `host` in `rs.conf` to external IPs. External clients can now use a standard replica-set URI. Sharded mode is unchanged (the existing single LoadBalancer fronts the stateless `mongos`). Fixes #2514 *(backported to v1.3.3 via #2590)* ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2522). + +* **[mongodb] Reclaim PVCs and operator secrets on uninstall**: Adds the Percona finalizer pair (`percona.com/delete-psmdb-pods-in-order`, `percona.com/delete-psmdb-pvc`) to the `PerconaServerMongoDB` CR so the operator performs an ordered shutdown and reclaims PVCs + operator-managed Secrets when a release is uninstalled. Previously, deleting a MongoDB application left PVCs and operator Secrets orphaned in the tenant namespace, requiring manual cleanup before the same release name could be reused. Covers replicaset and sharded modes including the config server ([**@Arsolitt**](https://github.com/Arsolitt) in #2525). + +* **[kafka] Create LoadBalancer service when external flag is enabled**: When `external: true` was set on the Kafka chart, the `external` listener was always present in the Kafka CR (rendered as `type: internal` when disabled) instead of being omitted entirely, and the dashboard role only granted access to the internal `-kafka-bootstrap` ClusterIP. The external listener is now added only when `external: true`, and dashboard RBAC includes the external bootstrap service *(backported to v1.3.3 via #2592)* ([**@IvanHunters**](https://github.com/IvanHunters) in #2578). + +* **[kafka] Bump default resourcesPreset to medium**: Changes the default `resourcesPreset` for both `kafka` and `zookeeper` from `small` (1 CPU / 512Mi) to `medium` (1 CPU / 1Gi). Strimzi sets JVM heap to 50% of container limit for Kafka and 75% for ZooKeeper, leaving only ~256Mi / 128Mi non-heap budget under `small`. That budget no longer fits the Strimzi 0.45 / Kafka 3.9 image baseline (Metaspace, code cache, direct buffers, JMX javaagent, embedded Jetty, KRaft endpoints, thread stacks), so brokers and ZooKeeper hit cgroup OOM kills (exit 137) before connecting and end up in CrashLoopBackOff. The `small` preset is intentionally kept in the enum for deployments that explicitly pin it *(backported to v1.3.3 via #2589)* ([**@IvanHunters**](https://github.com/IvanHunters) in #2537). + +* **[kubernetes] Close admin-kubeconfig race on tenant cluster bootstrap**: On a cold tenant-Kubernetes bootstrap, the parent HelmRelease raced the admin-kubeconfig Secret that Kamaji provisions asynchronously. Three CP-side Deployments (cluster-autoscaler, kccm, kcsi-controller) mounted that Secret as a hard volume; Flux helm-controller's default wait budget was too short for Kamaji cold start, and `install.remediation { retries: -1 }` then uninstalled the Cluster CR and restarted the cycle forever. The fix is defense-in-depth: `optional: true` on the admin-kubeconfig volume, a shared `wait-for-kubeconfig` init container with a 10m deadline (strictly below the HR Install.Timeout so broken tenants fall into visible CrashLoopBackOff), and the per-Application 15m timeout described above ([**@lexfrei**](https://github.com/lexfrei) in #2413). + +* **[etcd] Remove destructive post-upgrade cert-regeneration hook**: The etcd chart ran a `post-upgrade` Helm hook on every upgrade that deleted etcd TLS Secrets and then deleted etcd pods, forcing cert-manager to re-issue the entire etcd CA chain. On clusters with Kamaji-managed tenant control planes this put every tenant `kube-apiserver` into CrashLoopBackOff until each DataStore was manually re-reconciled. The hook was a one-shot `2.6.0 → 2.6.1` migration that became a permanent footgun once chart versioning moved to `0.0.0+` (always `< 2.6.1` per semver). The hook is removed entirely *(backported to v1.3.1 via #2511)* ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2462). + +* **[backups] Move velero-configmap Role to velero chart**: The `backupstrategy-controller` declared a Role/RoleBinding scoped to the `cozy-velero` namespace for managing `ResourceModifier` ConfigMaps. On bundles where Velero was not enabled, that namespace did not exist and the HelmRelease failed with `namespaces "cozy-velero" not found`, blocking installation. The Role/RoleBinding now lives in the velero chart so it is created only when velero is actually deployed *(backported to v1.3.1 via #2467)* ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2459). + +* **[kamaji] Increase memory limits and add startup probe**: The kamaji controller frequently entered CrashLoopBackOff due to OOMKills (exit 137) within ~20–25 seconds of startup, with the readiness probe failing while the controller was still finishing initialization. Memory limit raised from 500Mi to 512Mi, request from 100Mi to 256Mi, plus a 60-second startup probe (12 attempts × 5s) so the controller has room to boot *(backported to v1.3.1 via #2491)* ([**@IvanHunters**](https://github.com/IvanHunters) in #2421). + +* **[linstor] Plunger demotes idle-Primary DRBD resources**: Recovery for stuck DRBD Primaries during provisioning — `kube-linstor-plunger` now detects local Primaries with no open consumers and demotes them so CSI and provisioning can proceed *(backported to v1.3.2 via #2580)* ([**@kvaps**](https://github.com/kvaps) in #2539). + +* **[linstor] Retry drbdadm secondary after mkfs on transient EBUSY**: Adds a retry wrapper with exponential backoff for the post-mkfs demotion path to avoid initialization aborts caused by transient "device held open" errors *(backported to v1.3.2 via #2579)* ([**@IvanHunters**](https://github.com/IvanHunters) in #2538). + +* **[seaweedfs] Pass filer.replicas and s3.replicas to HelmRelease**: User-supplied replica counts on the SeaweedFS Filer and S3 components are now honored in the HelmRelease values to prevent divergence and spurious alerts *(backported to v1.3.2 via #2551)* ([**@lexfrei**](https://github.com/lexfrei) in #2533). + +* **[seaweedfs] Wait for ingress-nginx-system before creating Ingress**: Adds `dependsOn: ingress-nginx-system` to the inner `seaweedfs-system` HelmRelease so Flux blocks its install until the ingress chart is Ready. Previously, both outer HRs reconciled in parallel and SeaweedFS sometimes tried to create its tenant Ingress before the nginx ValidatingWebhookConfiguration endpoint was reachable, producing `failed calling webhook ... connect: operation not permitted` errors that needed a manual reconcile retry to clear ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2406). + +* **[harbor] Drive bucket-secret.yaml from values, gate HR on BucketInfo**: `cozy-harbor/templates/bucket-secret.yaml` no longer relies on `lookup` against the BucketAccess credentials Secret. It is now driven by `.Values.bucket.bucketInfo` (a JSON string) with `dig`-based safe accessors, so a missing/empty/partial value renders nothing instead of crashing with `index of untyped nil`. The downstream `-system` HelmRelease sources `BucketInfo` via `valuesFrom` so helm-controller refuses to compose values until the COSI BucketAccess controller has populated the Secret — both gating initial reconciliation and forcing a config-digest change (and thus a helm upgrade) when credentials arrive ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2528). + +* **[harbor] Mark credentials Secret with user-secret label**: The Harbor credentials Secret is now labeled `internal.cozystack.io/user-secret: "true"` so the dashboard correctly identifies it as a tenant-managed secret instead of treating it as platform-managed ([**@IvanHunters**](https://github.com/IvanHunters) in #2541). + +* **[objectstorage-controller] Retry on Bucket conflict during BucketAccess reconcile**: Retries on `Bucket` resource update conflicts during `BucketAccess` reconciliation so concurrent reconciles don't drop the BucketAccess into a permanent error state ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2529). + +* **[api] Apply labelSelector in TenantSecret Watch**: `Watch()` previously ignored `opts.LabelSelector` entirely; only the required `internal.cozystack.io/tenantresource=true` label was passed through. Watch consumers (e.g. dashboards using cached client subscriptions) saw every tenant Secret in the namespace regardless of the selector they requested. The user selector is now merged into the underlying watch selector, non-selectable selectors (e.g. `labels.Nothing()`) return a watcher with a closed result channel, and a defensive post-filter in the streaming goroutine ensures events never escape the requested selector ([**@IvanHunters**](https://github.com/IvanHunters) in #2542). + +* **[api] Prevent IDOR in TenantNamespace Get and Watch handlers**: Two IDOR (Insecure Direct Object Reference) vulnerabilities allowed authenticated users to read TenantNamespace metadata they had no RoleBinding for. The `Get` and `Watch` handlers now go through a new `hasAccessToNamespace()` helper that lists RoleBindings scoped only to the target namespace (orders of magnitude faster than the previous all-cluster scan), returns `NotFound` instead of leaking existence on unauthorized access, and applies the same check on the `Watch` filter path *(backported to v1.3.1 via #2524)* ([**@IvanHunters**](https://github.com/IvanHunters) in #2471). + +* **[cozystack-engine] Add managed Kubernetes without visible control-plane nodes support to lineage-controller-webhook**: The lineage-controller-webhook now handles managed-Kubernetes tenants whose control-plane nodes are not visible from the host cluster (Kamaji-style hosted control planes), so the webhook's lineage check passes for these tenants instead of rejecting all child resources ([**@dvc**](https://github.com/dvc) in #2481). + +* **[postgres-operator] Block HelmRelease Ready until the cnpg webhook actually serves**: The postgres-operator HelmRelease now blocks Ready until the CNPG webhook is actually serving (not just deployed), preventing downstream HRs from racing the webhook during cold install ([**@lexfrei**](https://github.com/lexfrei) in #2482). + +* **[kubevirt-operator] Move PrometheusRule to Helm-rendered template**: The KubeVirt operator's PrometheusRule was previously embedded in the operator manifest, which caused it to be re-applied on every reconcile and triggered spurious diffs. Moves the rule to a Helm-rendered template under the chart so it is owned and reconciled by Helm. Also replaces `max_over_time` with an instant vector + `for: 10m` to suppress spurious alerts on transient pod churn ([**@IvanHunters**](https://github.com/IvanHunters) in #2543). + +* **[cert-manager] Bump chart to v1.20.2**: See Platform Components ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2562). + +* **[virtual-machine] VMInstance disk race + bump VM IP/ready timeouts**: Fixes a race in the VMInstance disk reconciler where a freshly created disk could be marked Ready before the underlying DataVolume had finished population, and bumps the VM IP and Ready wait timeouts so slow boots are not classified as failures ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2555). + +* **[installer] Bootstrap cozy-system via --create-namespace + label hook**: Switches the platform install path to `helm install --create-namespace` plus a pre-install labeler hook that stamps the cozy-system namespace with PSA + identity labels, removing the previous chicken-and-egg where the namespace had to exist before the platform installer could be applied ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2508). + +* **[linstor] Include linstor-gui in root image build target**: The `linstor-gui` package (added in v1.3.0) was never wired into the root `Makefile`'s `build:` target, so CI never built or published the image. The missing build line is added so CI publishes the image and the per-package `Makefile` digest-pins `values.yaml` automatically *(backported to v1.3.1 via #2518)* ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2498). + +## Security + +* **[security] Bump Go dependencies to clear Dependabot CVE alerts**: Bumps root module Go deps to latest, kubeovn-webhook, kubevirt-csi-driver, token-proxy, and `api/apps/v1alpha1` deps, plus the OpenTelemetry exporter modules to align with the otel core; first-party Dockerfile builders are bumped to `golang:1.26`, and the first-party scratch images now run as non-root UID 65532. `USER 65532` is dropped from `cozystack-api` and `lineage-controller-webhook` containers in favor of the OCI-level UID so it can be overridden by PodSecurityContext ([**@lexfrei**](https://github.com/lexfrei) in #2494). + +* **[talos] Bump Talos to v1.12.7 (CVE-2026-31431)**: See Platform Components — picks up the kernel fix for the "Copy Fail" local privilege escalation in `algif_aead` *(backported to v1.3.2 via #2547)* ([**@kvaps**](https://github.com/kvaps) in #2545). + +* **[api] Prevent IDOR in TenantNamespace Get and Watch handlers**: See Bug Fixes — IDOR fix for TenantNamespace metadata access *(backported to v1.3.1 via #2524)* ([**@IvanHunters**](https://github.com/IvanHunters) in #2471). + +## System Configuration + +* **[platform] Single switch for host PROXY-protocol + ouroboros**: The new `publishing.proxyProtocol: true` value at the platform level flips ingress-nginx into PROXY-protocol mode (injecting `use-proxy-protocol` + `real-ip-header=proxy_protocol`) and emits the `cozystack.ouroboros` package. `use-forwarded-headers` and `compute-full-forwarded-for` are deliberately not on the list — they would let any upstream proxy spoof `X-Forwarded-For` ([**@lexfrei**](https://github.com/lexfrei) in #2561). + +* **[kubernetes] Expose ouroboros as a per-tenant addon**: `addons.ouroboros.enabled` on the tenant Kubernetes app turns ouroboros on per-tenant for hairpin-NAT under PROXY-protocol. Hard-fails the render if `addons.ingressNginx.enabled` is false, so the no-op case can't silently leave the addon dangling ([**@lexfrei**](https://github.com/lexfrei) in #2561). + +* **[coredns] Import /etc/coredns/custom into the Corefile**: The host and tenant CoreDNS charts now ship an empty `coredns-custom` ConfigMap with the `import /etc/coredns/custom/*.override` plugin in the `.:53` server block, so plugins like ouroboros can write rewrite snippets into the import directory without touching the chart-rendered Corefile ([**@lexfrei**](https://github.com/lexfrei) in #2561). + +* **[ouroboros] System chart**: New `packages/system/ouroboros/` package, vendored from `oci://ghcr.io/lexfrei/charts/ouroboros@0.7.0`. Cozystack-side defaults pinned in `values.yaml`; image and chart manifest are digest-pinned for reproducibility ([**@lexfrei**](https://github.com/lexfrei) in #2561). + +* **[platform] Register HAMi as optional system package**: HAMi declared as a PackageSource with a GPU Operator dependency, included in the `iaas` bundle ([**@Arsolitt**](https://github.com/Arsolitt) in #2484). + +## Dependencies & Version Updates + +See the **Platform Components** section near the top of this changelog for the full list of upstream bumps with user-facing impact summaries. + +Additional dependency-only changes: + +* **[security] Bump Go dependencies to clear Dependabot CVE alerts** — see Security ([**@lexfrei**](https://github.com/lexfrei) in #2494). +* **[hetzner-robotlb] Update robotlb chart to appVersion 0.0.6** — Renovate-driven chart refresh ([**@sircthulhu**](https://github.com/sircthulhu) in #2465). + +## Development, Testing, and CI/CD + +* **[e2e] Parallel post-install prep**: Runs the post-install preparation steps for cozystack E2E in parallel instead of sequentially, cutting roughly 5 minutes off the install path ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2557). + +* **[e2e] Replace fixed timeouts with event-driven backstops**: The E2E suite no longer waits on fixed `sleep` durations for HelmRelease readiness — `kubectl wait` and JSON-path-driven gates are used everywhere so the slowest path determines runtime, not the longest fixed timeout ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2554). + +* **[tests] Pre-pull timing-sensitive platform images before install**: Pre-pulls the LINSTOR, KubeVirt, and other timing-sensitive platform images on every E2E node before the install starts, so first-install reconciliation no longer races image pull on slow registries ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2556). + +* **[tests] Increase Bootstrap Talos cluster timeout to 60s**: Bootstrap Talos cluster wait raised from a hardcoded 30s to a more realistic 60s for slow CI hosts ([**@lexfrei**](https://github.com/lexfrei) in #2431). + +* **[ci/api] Add codegen drift check**: New CI workflow that runs `make generate && make manifests` on every PR that touches `api/` or generation tooling and fails if the working tree is dirty, catching codegen drift before merge ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2463). + +* **[ci] Replace Dosu stale and size labeling with native workflows**: Removes the Dosu integration in favor of native GitHub workflows for issue/PR staling and size labeling ([**@lexfrei**](https://github.com/lexfrei) in #2504). + +* **[ci] Adopt CNCF/k8s label conventions**: Aligns the issue/PR label palette and naming with the CNCF and Kubernetes upstream conventions ([**@lexfrei**](https://github.com/lexfrei) in #2495). + +* **[ci] Pass app token to Publish draft release step**: The Publish-draft-release step now uses the `cozystack-ci` GitHub App token instead of the workflow's default `GITHUB_TOKEN`, fixing missing-permissions errors when promoting drafts *(backported to v1.3.2 via #2572)* ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2532). + +* **[ci] Generate changelog from release tag, not from main**: Changelog generation now uses the release tag as the upper bound instead of HEAD, so a developer running the changelog tool on `main` after extra merges no longer leaks unreleased commits into the changelog *(backported to v1.3.2 via #2574)* ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2530). + +* **[ci] Stop backport workflow from cascading on its own PRs**: The backport workflow no longer triggers on PRs created by the backport workflow itself, preventing infinite cascades when a backport target overlaps an open PR ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2584). + +* **[ci] Harden changelog generation in tags.yaml (v1.3.0 regression)**: Fixes a regression in the `tags.yaml` changelog job exposed by v1.3.0 — empty changelog files are rejected before commit, the changelog-exists gate uses the tag instead of HEAD, and the Copilot step uses a read-only app token ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2460). + +## Documentation + +* **[website] Instance-type resource presets reference**: Documents the new `.` taxonomy with the full size matrix and the migration table mapping legacy names to instance-type equivalents ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#535). + +* **[website] Document publishing.proxyProtocol + ouroboros hairpin-NAT fix**: Operator-facing docs for the new platform switch and the ouroboros tenant addon ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#527). + +* **[website] vGPU setup guide for GPU sharing between VMs**: Walks through running VMs with NVIDIA vGPU on Cozystack — building the proprietary vGPU Manager container image, deploying the GPU Operator `vgpu` variant via the Package CR, assigning vGPU profiles to SR-IOV VFs through `current_vgpu_type`, configuring DLS licensing via `ClientConfigToken`, patching the KubeVirt CR with `permittedHostDevices.pciHostDevices`, and a sample `VirtualMachine` using a CDI `DataVolume` to avoid the 2.4 GiB containerDisk root overflow. Includes a vGPU profile reference table for L40S with the Q/A/B suffix taxonomy *(backported to v1.3.3)* ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#467). + +* **[website] Add GPU sharing with HAMi guide**: New documentation page covering HAMi addon enablement, profile configuration, and the glibc < 2.34 compatibility matrix ([**@Arsolitt**](https://github.com/Arsolitt) in cozystack/website#517). + +* **[website] Document Ubuntu + Secure Boot DRBD prerequisites**: Documents the `modules-load.d` step for host-loaded DRBD on Ubuntu Secure Boot systems and drops the misleading "recovery" qualifier on the manual procedure ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#529). + +* **[website] LINSTOR GUI documentation**: New `docs/storage/linstor-gui` page documenting the LINSTOR GUI addon for v1.3 and `next/` trunk ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#521). + +* **[website] vm-default-images / golden images guide update**: Refreshes the virtualization vm-image guide to reflect the v1.3 `vm-default-images` package and the new "golden images" pattern ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#520). + +* **[website] clusterDomain is pinned on tenants, drop stale 0.7.0 references**: Removes outdated references to 0.7.0 in the networking docs and clarifies that tenant cluster domains are pinned ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#534). + +* **[website] bump etcd-operator to v0.4.3**: Updates the platform-stack docs for the etcd-operator v0.4.3 bump ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#501). + +* **[website] Update managed apps reference for v1.3.0**: Automated managed-apps reference refresh for v1.3.0 ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#507). + +* **[website] Backup workloads from managed Kubernetes guide**: New guide describing how to back up tenant workloads from inside a managed Kubernetes cluster ([**@androndo**](https://github.com/androndo) in cozystack/website#528). + +* **[website] Add platform licenses reference**: Adds a per-component license overview as an oss-card grid scoped to `next/` and `v1.3` ([**@tym83**](https://github.com/tym83) in cozystack/website#530). + +* **[website] Blog: Cozystack 1.3 release announcement**: Publishes the v1.3 release-announcement blog post ([**@tym83**](https://github.com/tym83) in cozystack/website#512). + +* **[website] SEO: canonical, JSON-LD, sitemap directive, richer meta descriptions**: Adds canonical link tags, JSON-LD `SoftwareApplication` schema, `llms.txt`, sitemap directive in `robots.txt`, and noindex on legacy doc versions ([**@tym83**](https://github.com/tym83) in cozystack/website#533). + +* **[website] feat(home): add positioning taglines under hero title** ([**@tym83**](https://github.com/tym83) in cozystack/website#524). + +* **[website] Refresh OSS health snapshots monthly**: Switches the OSS-health snapshot cron from quarterly to monthly and stabilizes the telemetry snapshot ([**@tym83**](https://github.com/tym83) in cozystack/website#531). + +* **[website] Use April 2026 data as placeholder for quarter and year periods**: Telemetry snapshot fix for the OSS-health page ([**@tym83**](https://github.com/tym83) in cozystack/website#511). + +* **[website] Correct April 2026 telemetry snapshot and pause fetch cron**: Pauses the telemetry fetch cron during the April 2026 backfill window ([**@tym83**](https://github.com/tym83) in cozystack/website#508). + +* **[website] Stop autoupdate PRs from rewriting source URLs every run**: Fixes the docs autoupdate CI so it no longer churns source URLs on every run ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#516). + +* **[website] Skip prereleases when picking openapi.json release**: The openapi.json picker now skips prereleases when looking for the most recent stable cozystack release ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#515). + +* **[website] Remove obsolete update-managed-apps cron workflow** ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#519). + +* **[docs/operations] Resource presets reference**: New `docs/operations/resource-presets.md` in-repo doc documenting the new five-series taxonomy and the legacy alias mapping ([**@lexfrei**](https://github.com/lexfrei) in #2607). + +* **[talm] Release v0.25.0, v0.25.1, v0.26.0, v0.26.1, v0.27.0, v0.28.0, v0.28.1, v0.28.2** (github.com/cozystack/talm): The talm CLI gains apply-time safety gates (declared-resource existence + drift preview/verify), a wrapper UX overhaul (persistent flags, `-n` drop, crashdump, kubeconfig hint, dmesg cushion, TUI refusal), META preservation by default on `talm reset`, a pre-flight Talos version check on `apply`, multi-NIC discovery for chart defaults (filtered to IPv4 family, longest-prefix VIP-link matching), strict golangci-lint config + cross-platform CI gates, `--image` flag on `init`, cluster-name overrides via `dislogical`, and a handful of charts/engine fixes (RotateKeys atomic rotation, IPv6 bracket normalisation, `$patch:delete` no-op on absent paths). Notable PRs: ([**@lexfrei**](https://github.com/lexfrei) in cozystack/talm#127, #128, #129, #130, #133, #135, #136, #145, #146, #149, #150, #154, #158, #160, #161, #162, #163, #173, #197, #199, #200, #201, #202; [**@dislogical**](https://github.com/dislogical) in cozystack/talm#148). + +* **[ansible-cozystack] Release v1.3.0, v1.3.1, v1.3.2, v1.3.3** (github.com/cozystack/ansible-cozystack): Adds Ubuntu 26.04 support and cozy-system namespace adoption ([**@lexfrei**](https://github.com/lexfrei) in cozystack/ansible-cozystack#37), installs `drbd-dkms` from the LINBIT PPA for Secure Boot hosts ([**@lexfrei**](https://github.com/lexfrei) in cozystack/ansible-cozystack#39), bumps the k3s dependency to v1.35.4+k3s1 ([**@app/renovate**](https://github.com/apps/renovate) in cozystack/ansible-cozystack#36), and tracks cozystack v1.3.0–v1.3.3 releases ([**@app/renovate**](https://github.com/apps/renovate) in cozystack/ansible-cozystack#33, #35, #38, #40). + +* **[cozy-proxy] Release v0.3.0** (github.com/cozystack/cozy-proxy): Per-service ingress port filtering ([**@mattia-eleuteri**](https://github.com/mattia-eleuteri) in cozystack/cozy-proxy#11), service-proxy-name label selector ([**@kvaps**](https://github.com/kvaps) in cozystack/cozy-proxy#9), probe/metrics endpoints disabled by default ([**@kvaps**](https://github.com/kvaps) in cozystack/cozy-proxy#10), `allowICMP` annotation for port-filter mode ([**@kvaps**](https://github.com/kvaps) in cozystack/cozy-proxy#12), label-only selector + port-filter default ([**@kvaps**](https://github.com/kvaps) in cozystack/cozy-proxy#13). + +## Contributors + +We'd like to thank all contributors who made this release possible: + +* [**@androndo**](https://github.com/androndo) +* [**@Arsolitt**](https://github.com/Arsolitt) +* [**@dislogical**](https://github.com/dislogical) +* [**@dvc**](https://github.com/dvc) +* [**@IvanHunters**](https://github.com/IvanHunters) +* [**@kvaps**](https://github.com/kvaps) +* [**@lexfrei**](https://github.com/lexfrei) +* [**@matthieu-robin**](https://github.com/matthieu-robin) +* [**@mattia-eleuteri**](https://github.com/mattia-eleuteri) +* [**@myasnikovdaniil**](https://github.com/myasnikovdaniil) +* [**@sircthulhu**](https://github.com/sircthulhu) +* [**@tym83**](https://github.com/tym83) + +### New Contributors + +We're excited to welcome our first-time contributors: + +* [**@dvc**](https://github.com/dvc) — First contribution! +* [**@dislogical**](https://github.com/dislogical) — First contribution! + +--- + +**Full Changelog**: https://github.com/cozystack/cozystack/compare/v1.3.0...v1.4.0 From 025f4e4c01478b4fa54041a1fa1a5fdaabc9eca2 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Tue, 19 May 2026 10:13:05 +0500 Subject: [PATCH 2/3] docs(release): fix KubeVirt SEV feature gate spelling Address review feedback from gemini-code-assist on docs/changelogs/v1.4.0.md:102 and coderabbitai on :103: the AMD SEV-SNP feature gate in upstream KubeVirt source is WorkloadEncryptionSEV (see pkg/virt-config/featuregate/active.go), not WorkloadEncruptionSEV as quoted from the v1.7.0 release notes. Signed-off-by: Myasnikov Daniil --- docs/changelogs/v1.4.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/changelogs/v1.4.0.md b/docs/changelogs/v1.4.0.md index 1c59f0718..467580380 100644 --- a/docs/changelogs/v1.4.0.md +++ b/docs/changelogs/v1.4.0.md @@ -99,7 +99,7 @@ Most operators can take v1.4.0 with no manual action — Cozystack stays on the * **KubeVirt**: v1.6.3 → v1.8.2. Two-minor jump crossing the upstream QEMU bump — VMs running before the upgrade need a cold restart (see Upgrade Notes). New CRD fields for changed block tracking (storage-agnostic incremental backups), `s390x` support added, `ppc64le` deprecated. The standalone ServiceMonitor / alerts templates are removed in favour of the Helm-rendered PrometheusRule. ([v1.7.0 notes](https://github.com/kubevirt/kubevirt/releases/tag/v1.7.0), [v1.8.0 notes](https://github.com/kubevirt/kubevirt/releases/tag/v1.8.0)) ([**@IvanHunters**](https://github.com/IvanHunters) in #2502) - **VM Backup API + Changed Block Tracking** — new alpha `VMBackup` resource and `ChangedBlockTracking` matcher on `KubeVirt` enable storage-agnostic incremental backups; CBT also applies to hotplug volumes; libvirt checkpoints survive VM restart for resumable incremental flows. - - **Confidential VMs (experimental)**: Intel TDX support behind the `WorkloadEncryptionTDX` feature gate, AMD SEV-SNP behind `WorkloadEncruptionSEV` (alongside the existing SEV/SEV-ES paths). + - **Confidential VMs (experimental)**: Intel TDX support behind the `WorkloadEncryptionTDX` feature gate, AMD SEV-SNP behind `WorkloadEncryptionSEV` (alongside the existing SEV/SEV-ES paths). - **VMPool gains a real lifecycle**: new `pool.kubevirt.io/v1beta1` API with auto-healing, Proactive/Opportunistic scale-in, statePreservation, UpdateStrategy with Selection policies, and proper finalizer cleanup. - **Live-migration improvements**: dedicated mTLS certificate, migration allowed when host CPU model changes after a libvirt upgrade, decentralized live migration on L3 networks and across volumes with different volumeModes, live-migration for ImageVolume with modified container disk images. - **Eviction / evacuation control**: new `/evacuate/cancel` subresource and `virtctl evacuate-cancel` command to abort an in-flight evacuation cycle; new `VirtualMachineInstanceEvictionRequested` condition for tracking. From 0ca10142ab0322cdfbaa24031858ceac4d6d6da1 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Tue, 19 May 2026 10:13:21 +0500 Subject: [PATCH 3/3] docs(release): use linked contributor format for @dislogical Address review feedback from gemini-code-assist on docs/changelogs/v1.4.0.md:326: the in-prose mention of dislogical was rendered as inline code rather than as a contributor link. Switch to the [**@user**](https://github.com/user) format used elsewhere in the changelog. Signed-off-by: Myasnikov Daniil --- docs/changelogs/v1.4.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/changelogs/v1.4.0.md b/docs/changelogs/v1.4.0.md index 467580380..35ca0ffed 100644 --- a/docs/changelogs/v1.4.0.md +++ b/docs/changelogs/v1.4.0.md @@ -323,7 +323,7 @@ Additional dependency-only changes: * **[docs/operations] Resource presets reference**: New `docs/operations/resource-presets.md` in-repo doc documenting the new five-series taxonomy and the legacy alias mapping ([**@lexfrei**](https://github.com/lexfrei) in #2607). -* **[talm] Release v0.25.0, v0.25.1, v0.26.0, v0.26.1, v0.27.0, v0.28.0, v0.28.1, v0.28.2** (github.com/cozystack/talm): The talm CLI gains apply-time safety gates (declared-resource existence + drift preview/verify), a wrapper UX overhaul (persistent flags, `-n` drop, crashdump, kubeconfig hint, dmesg cushion, TUI refusal), META preservation by default on `talm reset`, a pre-flight Talos version check on `apply`, multi-NIC discovery for chart defaults (filtered to IPv4 family, longest-prefix VIP-link matching), strict golangci-lint config + cross-platform CI gates, `--image` flag on `init`, cluster-name overrides via `dislogical`, and a handful of charts/engine fixes (RotateKeys atomic rotation, IPv6 bracket normalisation, `$patch:delete` no-op on absent paths). Notable PRs: ([**@lexfrei**](https://github.com/lexfrei) in cozystack/talm#127, #128, #129, #130, #133, #135, #136, #145, #146, #149, #150, #154, #158, #160, #161, #162, #163, #173, #197, #199, #200, #201, #202; [**@dislogical**](https://github.com/dislogical) in cozystack/talm#148). +* **[talm] Release v0.25.0, v0.25.1, v0.26.0, v0.26.1, v0.27.0, v0.28.0, v0.28.1, v0.28.2** (github.com/cozystack/talm): The talm CLI gains apply-time safety gates (declared-resource existence + drift preview/verify), a wrapper UX overhaul (persistent flags, `-n` drop, crashdump, kubeconfig hint, dmesg cushion, TUI refusal), META preservation by default on `talm reset`, a pre-flight Talos version check on `apply`, multi-NIC discovery for chart defaults (filtered to IPv4 family, longest-prefix VIP-link matching), strict golangci-lint config + cross-platform CI gates, `--image` flag on `init`, cluster-name overrides via [**@dislogical**](https://github.com/dislogical), and a handful of charts/engine fixes (RotateKeys atomic rotation, IPv6 bracket normalisation, `$patch:delete` no-op on absent paths). Notable PRs: ([**@lexfrei**](https://github.com/lexfrei) in cozystack/talm#127, #128, #129, #130, #133, #135, #136, #145, #146, #149, #150, #154, #158, #160, #161, #162, #163, #173, #197, #199, #200, #201, #202; [**@dislogical**](https://github.com/dislogical) in cozystack/talm#148). * **[ansible-cozystack] Release v1.3.0, v1.3.1, v1.3.2, v1.3.3** (github.com/cozystack/ansible-cozystack): Adds Ubuntu 26.04 support and cozy-system namespace adoption ([**@lexfrei**](https://github.com/lexfrei) in cozystack/ansible-cozystack#37), installs `drbd-dkms` from the LINBIT PPA for Secure Boot hosts ([**@lexfrei**](https://github.com/lexfrei) in cozystack/ansible-cozystack#39), bumps the k3s dependency to v1.35.4+k3s1 ([**@app/renovate**](https://github.com/apps/renovate) in cozystack/ansible-cozystack#36), and tracks cozystack v1.3.0–v1.3.3 releases ([**@app/renovate**](https://github.com/apps/renovate) in cozystack/ansible-cozystack#33, #35, #38, #40).