From 7eed71c9064150d7c2110fc86c96ad02a7549676 Mon Sep 17 00:00:00 2001 From: "cozystack-ci[bot]" <274107086+cozystack-ci[bot]@users.noreply.github.com> Date: Mon, 22 Jun 2026 11:17:47 +0000 Subject: [PATCH 1/2] docs: add changelog for v1.5.0 Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> --- docs/changelogs/v1.5.0.md | 257 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 257 insertions(+) create mode 100644 docs/changelogs/v1.5.0.md diff --git a/docs/changelogs/v1.5.0.md b/docs/changelogs/v1.5.0.md new file mode 100644 index 000000000..0f4f4ad03 --- /dev/null +++ b/docs/changelogs/v1.5.0.md @@ -0,0 +1,257 @@ + + +# Cozystack v1.5.0 + +Cozystack v1.5.0 ships **Flux shard-operator for tenant Helm-controller sharding**, **Cilium-backed Gateway API** as a first-class networking layer, **TLS support for all major managed services** (Kafka, NATS, Qdrant, PostgreSQL), an **operator-provided wildcard certificate** for platform ingress, an **Option resource for dynamic dashboard dropdowns**, full **HelmRelease generation knobs** surfaced as operator and API flags, **tenant VM start/stop/restart RBAC**, **GPU passthrough auto-wiring**, a new **Job backup strategy**, an **etcd backup-strategy controller**, **velero installed by default**, and a **tenant overview Grafana dashboard** for platform admins. The release rolls up every fix from v1.4.1 through v1.4.4. + +Platform component bumps: **Flux v2.7.3 → v2.8.0**, **MetalLB v0.15.2 → v0.16.1**, **SeaweedFS 4.05 → 4.31**, **ouroboros v0.7.2 → v0.8.0**, **flux-operator v0.48.0**, and **seaweedfs-cosi-driver v0.3.1** *(backported to v1.4.3)*. + +> **Note:** Items marked *(backported to v1.4.x)* were also shipped in v1.4.1, v1.4.2, v1.4.3, or v1.4.4 patch releases. + +## Feature Highlights + +### Flux Shard Operator: Tenant Helm-Controller Sharding + +The new `flux-shard-operator` system package spreads tenant HelmReleases across multiple dedicated helm-controller shards, preventing a noisy tenant — such as one with a HelmRelease stuck in infinite remediation — from starving the reconciliation queue for other tenants. + +The operator has two parts: a **placement controller** that assigns each tenant to a shard using greedy least-loaded distribution weighted by HelmRelease count, and a **shard runtime** that provisions the correct number of helm-controller Deployments cloned from the existing `flux-aio` Deployment — so images and feature gates remain version-synced automatically. The unit of placement is the tenant namespace (all of its HelmReleases share one shard label), and the controller supports pinned tenants, threshold-gated rebalance, paced moves, and never moves tenants that are being deleted. Watches are metadata-only, so the placement controller does not decode the helm-controller status-patch firehose. + +`flux-aio` itself is unchanged and continues to serve all tenants without a shard key in clusters that choose not to enable the operator ([**@kvaps**](https://github.com/kvaps) in #2821). + +### Gateway API Support via Cilium + +Cozystack now ships **Cilium Gateway API** as a first-class networking layer alongside ingress-nginx. A new `TenantGateway` CRD is reconciled by `cozystack-controller`, which provisions a `Gateway`, per-tenant `Issuer`, `Certificate` (wildcard or per-listener), and HTTP→HTTPS redirect `HTTPRoute` from a single tenant-level declaration. The Helm chart does not render Gateway or Certificate resources directly, preventing the Helm-vs-controller race on `Gateway.spec.listeners` that route-driven listener additions would otherwise trigger. + +Tenants opt in by setting `spec.gateway` on their Tenant CR. A child tenant inherits the parent's Gateway by default via the `namespace.cozystack.io/gateway` label. DNS-01 SAN expansion lets child tenants register their apex domains as additional SANs on the parent wildcard certificate without a separate ACME round trip. The platform toggles ingress-nginx ↔ HTTPRoute automatically based on `_cluster.gateway-enabled`. External-DNS gains `gateway-tlsroute` support alongside `gateway-httproute`. + +All existing ingress-nginx flows are unaffected — the Gateway API layer is purely additive and disabled by default ([**@lexfrei**](https://github.com/lexfrei) in #2470). + +### TLS for Managed Services: Kafka, NATS, Qdrant, and PostgreSQL + +All four major stateful managed services now support TLS, using a consistent tri-state `tls.enabled` field: + +- **null / unset** — auto-inherits the value of `external` (TLS on when the service is published externally, off for cluster-internal deployments). This keeps the default behaviour unchanged for existing deployments. +- **true** — TLS always on. +- **false** — TLS always off. + +**Kafka** wires TLS through the Strimzi listener configuration. The operator manages the cluster PKI end-to-end and publishes `-cluster-ca-cert` and `-clients-ca-cert` for client trust. The external LoadBalancer listener on port 9094 is independently gated on `external: true`; `tls.enabled` only governs whether that listener uses TLS ([**@Arsolitt**](https://github.com/Arsolitt) in #2681). + +**NATS** and **Qdrant** use a self-contained cert-manager chain rendered in the tenant namespace: a self-signed Issuer, CA Certificate, CA Issuer, and leaf Certificate with SAN coverage for all three DNS forms per service plus the external hostname when `external: true`. Qdrant adds short-form per-replica SANs for p2p TLS hostname verification and a Reloader annotation to trigger pod restarts on cert rotation ([**@Arsolitt**](https://github.com/Arsolitt) in #2684 and #2685). + +**PostgreSQL** uses the CNPG operator's built-in cert management instead of a chart-rendered chain. The operator auto-generates a self-signed CA and signs server, client, and replication certs. When `external: true` and TLS is on, the chart sets `spec.certificates.serverAltDNSNames` so the operator appends the external hostname to the leaf cert. The trust anchor for clients is the `-ca` Opaque Secret (key `ca.crt`) ([**@Arsolitt**](https://github.com/Arsolitt) in #2686). + +### Operator-Provided Wildcard Certificate for Platform Ingress + +Operators can now serve platform services and the root tenant's ingress under a pre-existing wildcard TLS certificate by setting a single `publishing.certificates.wildcardSecretName` value that names a TLS Secret they have already provisioned. Only the Secret name travels through the `cozystack-values` channel — the certificate and private key are never broadcast to tenant namespaces. + +When a wildcard Secret is configured, `ingress-nginx` is reconfigured to use it as the default SSL certificate, eliminating the need for per-host ACME certificates on the root domain. Existing per-host ACME flows are unaffected for operators who do not set this value ([**@lexfrei**](https://github.com/lexfrei) in #2819). + +### Option Resource and Dynamic Dashboard Dropdowns + +A new `core.cozystack.io/v1alpha1 Option` resource provides **runtime-populated dropdowns** for the dashboard's create and edit forms. Fields that reference live cluster resources — GPU device names, KubeVirt instancetypes and preferences, Multus networks, VM images, bucket storage pools, storage classes, backup classes and plans — can now render as curated option lists instead of free-text inputs that drift from reality. + +Option items are computed on read by an in-process provider registry using a privileged client, so tenants see current values without needing broad cluster read access. Providers cover: `gpu`, `instancetype`, `instanceprofile`, `network`, `image`, `storagepool`, `storageclass`, `backupclass`, `plan`, `backup`, and `appkind`. + +Chart authors expose the feature via a new `@x-cozystack-options` directive in `values.yaml`; `cozyvalues-gen` propagates it through all generated artifacts (schema, Go types, and `-rd` ApplicationDefinitions) so the extension is carried into the served OpenAPI ([**@kvaps**](https://github.com/kvaps) in #2778). + +### HelmRelease Generation Knobs and Per-App Timeouts + +The cozystack API server now exposes the same HelmRelease generation flags introduced for the operator in v1.4.0: `--helmrelease-interval`, `--helmrelease-retry-interval`, `--helmrelease-install-timeout`, `--helmrelease-upgrade-timeout`, and `--helmrelease-max-history`. Previously, the `cozystack-api` path that converts Application CRs to HelmReleases hardcoded its own values (5-minute interval, `Remediation{Retries:-1}`) while the operator path used flags — creating divergent behaviour for the same application kind depending on which code path generated the HelmRelease. + +A companion change introduces a **per-Application Install/Upgrade timeout** driven by a `release.cozystack.io/helm-upgrade-timeout` annotation on ApplicationDefinition. This extends the existing `helm-install-timeout` annotation support so individual application kinds can declare asymmetric install vs. upgrade budgets — for example a managed service with a fast install but a slow rolling upgrade ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2571). + +## Major Features and Improvements + +* **feat(rbac): allow tenants to start, stop, and restart their VMs**: Grants tenant users permission to call the KubeVirt VM power-control subresources (`virtualmachines/{name}/start|stop|restart`) by adding them to the `cozy:tenant:use:base` ClusterRole. Previously the dashboard VM power buttons returned 403 even for tenant admins, while VNC and console subresources were already granted at the same level ([**@kvaps**](https://github.com/kvaps) in #2777). + +* **feat(kubernetes): enable GPU passthrough out-of-the-box**: Tenant Kubernetes clusters with GPU node-groups now have passthrough working without manual intervention. Worker nodes in GPU node-groups are automatically labeled `gpu=on`, and the tenant gpu-operator is configured to load the NVIDIA driver with `NVreg_NvLinkDisable=1` to avoid NvLink errors ([**@kvaps**](https://github.com/kvaps) in #2780). + +* **feat(platform): auto-wire KubeVirt `permittedHostDevices` and `HostDevices` feature gate when `gpu-operator` is enabled**: When the passthrough or vGPU variant of `cozystack.gpu-operator` is deployed, the platform now automatically patches the KubeVirt CR to add `HostDevices` to `developerConfiguration.featureGates` and list the detected GPU resources under `permittedHostDevices`. Previously the GPU Operator reported green and the resource appeared in node allocatable, but every VM referencing it via `domain.devices.hostDevices` was rejected by the admission webhook ([**@lexfrei**](https://github.com/lexfrei) in #2768). + +* **feat(gpu-operator): add container variant for hosts with a preinstalled driver**: A third `cozystack.gpu-operator` variant is now available for hosts where the NVIDIA driver and `nvidia-container-toolkit` are installed by the OS package manager (the typical Ubuntu/Debian GPU node shape). The existing `passthrough` variant runs `vfioManager` to bind `vfio-pci` — which conflicts with an active host driver — and `vgpu` requires a proprietary vGPU Manager; neither worked on this host shape ([**@lexfrei**](https://github.com/lexfrei) in #2766). + +* **feat(platform): add deletion-protection guardrail via `ValidatingAdmissionPolicy`**: A new VAP blocks `DELETE` on any platform object labeled `platform.cozystack.io/no-delete=true` using native in-process CEL evaluation — no webhook DaemonSet, no Service, no TLS, no additional image. The policy and its binding are themselves self-protected, and a `migration 42` backfills the label on the `cozystack-version` ConfigMap. The policy is gated on Kubernetes 1.30+ capability detection ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2650). + +* **feat(backups): implement Job backup strategy**: Adds a generic, application-agnostic `strategy.backups.cozystack.io/Job` driver that runs a user-supplied Kubernetes `Job` to create and restore backups. This complements the application-specific strategy controllers and allows operators to wire any workload into the backup framework ([**@lllamnyp**](https://github.com/lllamnyp) in #1721). + +* **feat(platform): add backup-strategy controller for etcd**: Extends the backupstrategy-controller with an etcd-specific driver, allowing operators to declare backup strategies for etcd clusters managed by etcd-operator. The controller composes `EtcdBackup` and `EtcdBackupSchedule` CRs and reconciles them against an S3-compatible object store ([**@androndo**](https://github.com/androndo) in #2641). + +* **feat(platform): add default backupclass**: The platform now ships a default `BackupClass` resource so backup strategies work out-of-the-box without operators needing to create a class manually on fresh installs ([**@androndo**](https://github.com/androndo) in #2716). + +* **feat(platform): install velero by default**: The `backupstrategy-controller` declares a hard dependency on `cozystack.velero`, but Velero remained an optional package — causing the controller's Package to stay `DependenciesNotReady` indefinitely on any bundle that did not explicitly enable it. Velero is now enabled by default in the platform bundle ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2833). + +* **feat(apps): mark `storageClass` fields as immutable on stateful applications**: The `storageClass` field on stateful app types (including managed databases and Kubernetes worker node-groups) is now marked immutable via the `x-kubernetes-immutable` schema extension, preventing silent in-place changes that would require manual PVC migration ([**@lexfrei**](https://github.com/lexfrei) in #2639). + +* **feat(platform): add kuberture as optional system package**: [kuberture](https://github.com/leaseweb/kuberture) is now available as an opt-in system package (`bundles.enabledPackages`). kuberture publishes the managed Kubernetes API endpoint as a DNS record via external-dns, enabling `kubectl` access to tenant clusters without a LoadBalancer or manual DNS entry ([**@lexfrei**](https://github.com/lexfrei) in #2647). + +* **feat(monitoring): add tenant overview dashboard**: A new **Tenant Overview** dashboard is available in the root/infra Grafana. It shows fleet-wide totals (tenants, apps, pods, CPU/memory/storage) plus a per-tenant leaderboard with resource utilization, app counts, and not-ready apps, and Top-N bar gauges by CPU, memory, storage, and app count — giving platform admins a single view of which tenants consume what ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2809). + +* **feat(dashboard): add cluster-usage RBAC for the new admin page**: Adds `get/list/watch` on `namespaces` and the cluster-usage aggregation endpoint to the `cozy:infra:admin` ClusterRole so the new admin resource-usage page in the dashboard renders without 403 errors ([**@lexfrei**](https://github.com/lexfrei) in #2743). + +* **feat(flux): upgrade to Flux v2.8.0**: Upgrades Flux from v2.7.3 to v2.8.0 across the vendored tenant chart and the embedded management-cluster manifests. Flux v2.8's helm-controller v1.5.0 introduces **Server-Side Apply with `--force-conflicts`**, enabling strict CRD schema enforcement — fields that were silently dropped on v2.7 now fail the apply. A set of chart fixes are bundled to address fields that became hard errors under the new SSA regime ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2602). + +* **refactor(seaweedfs): split `seaweedfs-system` into `seaweedfs-db` + `seaweedfs-system`**: The CNPG `Cluster/seaweedfs-db` resource is moved to its own HelmRelease (`seaweedfs-db`) that `seaweedfs-system` depends on. The new HR uses Flux v2 `HelmRelease.spec.healthCheckExprs` with a CEL expression on `Cluster.status.conditions[type=Ready]`, so `seaweedfs-system` reaches `Ready=True` only after the PostgreSQL primary is actually serving connections. This eliminates the `seaweedfs-filer` CrashLoopBackOff race during fresh tenant installs where Cilium's socket-LB returns `EPERM` on connections to ClusterIPs with no ready endpoints ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2601). + +* **feat(ouroboros): bump to v0.8.0** *(backported to v1.4.3)*: Upgrades ouroboros to v0.8.0, which logs an explicit reason when its TCP backend readiness check fails — making stuck-proxy situations immediately diagnosable. Also migrates the kubectl sidecar image from Docker Hub to `mirror.gcr.io`, avoiding anonymous pull rate-limits on shared runner IPs ([**@lexfrei**](https://github.com/lexfrei) in #2807). + +* **chore(metallb): bump MetalLB to v0.16.1**: Upgrades the vendored MetalLB chart from v0.15.2 to v0.16.1 and rebuilds the `metallb-controller` and `metallb-speaker` images. Adopts the upstream-default FRR-K8s BGP backend (FRR mode is deprecated in v0.16). The cozystack-built images move in lockstep with the chart to preserve the new probe contract ([**@lexfrei**](https://github.com/lexfrei) in #2699). + +* **chore(seaweedfs): bump SeaweedFS to 4.31**: Upgrades the vendored SeaweedFS chart and images to version 4.31 ([**@lexfrei**](https://github.com/lexfrei) in #2834). + +* **feat(hack): rewrite `check-readiness` as a Go command with test suite**: Replaces the `hack/check-readiness.sh` diagnostic with a Go command at `cmd/check-readiness`, integrated into the build the same way as `cozypkg`. The rewrite adds a test suite and makes the tool composable as a library for future automation ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2755). + +* **chore(maintainers): mark @lllamnyp as Core Maintainer**: Aleksei Sviridkin ([**@lllamnyp**](https://github.com/lllamnyp)) is promoted to Core Maintainer, reflecting sustained contributions across API design, CSI, and testing infrastructure ([**@tym83**](https://github.com/tym83) in #2719). + +## Bug Fixes + +* **fix(csi): detach orphan hot-plug volumes when VMI outlives its VM**: During node reprovisioning in tenant Kubernetes clusters, the VMI and its hot-plug pod can outlive the parent VirtualMachine. The upstream driver treated a missing VM as "detach succeeded", leaving the hot-plug pod alive and the infra storage device exclusively attached to the source host — every subsequent attach of that volume failed with `failed to set source device readwrite`. The wrapper now re-checks `VMI.status.volumeStatus` after upstream returns success and, if the hot-plug is still reported, detaches it from the VMI directly via the `removevolume` subresource ([**@kvaps**](https://github.com/kvaps) in #2866). + +* **fix(api): publish open spec as `x-kubernetes-preserve-unknown-fields`, not `additionalProperties:true`**: The aggregated `apps.cozystack.io` resources published a boolean `additionalProperties: true` for their free-form `.spec`, which caused `kube-controller-manager`'s ValidatingAdmissionPolicy type-checker to nil-dereference when it processed any VAP matching those resources. This triggered a `CrashLoopBackOff` on every control-plane node and stalled reconciliation. Switching to `x-kubernetes-preserve-unknown-fields: true` is semantically equivalent but handled safely by the type-checker ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2867). + +* **fix(platform): order `cozystack-basics` after the APIs its admission policies type-check**: `cozystack-basics` ships cluster-wide ValidatingAdmissionPolicies that match `tenants.apps.cozystack.io` and Gateway API types. On a cold install, if these policies were created before `cozystack-api` was Ready, `kube-controller-manager` would nil-dereference while type-checking the policy against the not-yet-served `tenants` schema, enter `CrashLoopBackOff`, stall cert and token issuance, and prevent `cozystack-api` from ever starting — a deadlock. `cozystack-basics` now `dependsOn` both `cozystack-api` and the Gateway API CRD installer ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2842). + +* **fix(seaweedfs): restore S3 TLS and COSI provisioner SA after SeaweedFS 4.31 upgrade**: The 4.31 chart bump nested the `-cert.file`/`-key.file` TLS args inside `{{- if .Values.s3.httpsPort }}`, but Cozystack ships `httpsPort: 0` — so no TLS args rendered, S3 spoke plaintext HTTP while all probes and clients expected HTTPS. Additionally, the COSI provisioner Deployment referenced a ServiceAccount by a different name than what the chart creates, causing pod startup failures. Both regressions are fixed via vendored patches ([**@kvaps**](https://github.com/kvaps) in #2916). + +* **fix(seaweedfs): restore `-lock` BucketClass, S3 service name, and drop `volumeSizeLimitMB`**: Three additional regressions from the SeaweedFS 4.31 bump: the `-lock` BucketClass and `-readonly` BucketAccessClass were dropped (breaking access control), the S3 service DNS name regressed to a non-prefixed form (breaking in-cluster clients), and `volumeSizeLimitMB` prevented multiple collections from sharing a volume ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2943). + +* **fix(backups): carry dropdown option sources in CRD annotations, not schema**: The four `backups.cozystack.io` CRDs embedded `x-cozystack-options` vendor extensions directly in their OpenAPI validation schema. Kubernetes only preserves `x-kubernetes-*` vendor extensions; every other `x-` key was rejected by helm-controller's SSA apply, causing the backup CRDs to never apply and the controller to crash-loop. Option sources are now stored in CRD object annotations, which are preserved unchanged ([**@lexfrei**](https://github.com/lexfrei) in #2823). + +* **fix(ingress, platform): deliver `publishing.proxyProtocol` to host ingress-nginx**: The three nginx config keys (`use-proxy-protocol`, `real-ip-header`, `enable-real-ip`) were written into a component values block that had no `install:` section, so the package controller never built a HelmRelease for it and the values were silently dropped. The switch now routes through `_cluster.proxy-protocol` in the `cozystack-values` Secret — the same path `expose-external-ips` already uses to reach the host ingress release ([**@lexfrei**](https://github.com/lexfrei) in #2799). + +* **fix(platform): add cert-manager dependency to webhook-cert consumers**: Five PackageSources (`capi-operator`, `cozystack-engine`, `ingress-nginx`, `kamaji`, `rabbitmq-operator`) now declare `dependsOn: cert-manager`. Without this, on cold installs the chart applied `cert-manager.io/v1 Certificate` CRs before the cert-manager webhook was ready, burning 2–3 minutes per affected package in retry loops ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2726). + +* **fix(info): use root-host for Keycloak OIDC issuer URL in tenant kubeconfig**: Non-root tenants had `--oidc-issuer-url` pointing at `keycloak..`, but Keycloak lives at `keycloak.`. `kubectl oidc-login` failed TLS verification against the nginx default cert, making the auto-generated kubeconfig unusable for any non-root tenant ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2704). + +* **fix(kubernetes): add `config_path` patch for containerd 2.x**: Adds support for containerd 2.x where the CRI plugin section was renamed from `io.containerd.grpc.v1.cri` to `io.containerd.cri.v1.images`. The patch regexes are also made resilient to single and double quotes around plugin names, fixing the issue on Ubuntu 24.04 where containerd 2.x uses single quotes ([**@elaugaste**](https://github.com/elaugaste) in #2723). + +* **fix: add `startupProbe` to `kubevirt-cdi-operator` and `grafana-operator`**: Both operators were missing a `startupProbe`, causing them to be killed by the `livenessProbe` during slow startup under load (slow leader election for `kubevirt-cdi-operator`, startup latency under high cluster activity for `grafana-operator`) ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2725). + +* **fix(kubernetes): add `spec.timeout` to tenant CSI HelmRelease**: The `kubevirt-csi-node` HelmRelease template was the only tenant addon HR missing `spec.timeout: 10m`, causing it to use Flux's default (5 minutes), which was insufficient on loaded clusters. All other tenant addon HelmReleases (cilium, coredns, ingress-nginx, etc.) already set 10m ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2727). + +* **fix(dashboard): unblock token-proxy startup when JWKS is briefly unreachable**: The dashboard's token-proxy process failed to start if the Keycloak JWKS endpoint was briefly unreachable during pod startup. This caused the dashboard to stay in a crash loop until Keycloak became available, rather than retrying gracefully ([**@lexfrei**](https://github.com/lexfrei) in #2745). + +* **fix(prometheus-operator-crds): ship full upstream CRD bundle**: A `rm -f` line in the Makefile stripped 6 of the 10 upstream CRDs (`Alertmanager`, `AlertmanagerConfig`, `Prometheus`, `PrometheusAgent`, `ScrapeConfig`, `ThanosRuler`). Operators depending on any of these CRDs were silently missing them from their clusters ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2660). + +* **fix(capi): add `startupProbe` to `capi-controller-manager`**: The CAPI controller manager lacked a startupProbe, causing it to be killed by the liveness probe during slow startup (e.g. when cert provisioning was delayed). This manifested as repeated restarts in clusters with high cert-manager load ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2946). + +* **fix(kafka): reclaim ZooKeeper PVCs on Kafka release deletion** *(backported to v1.4.1)*: Deleting a Kafka application freed broker PVCs but orphaned ZooKeeper PVCs, requiring manual cleanup before the same release name could be reused. `deleteClaim: true` is now set on ZooKeeper storage ([**@Arsolitt**](https://github.com/Arsolitt) in #2679). + +* **fix(postgres): accept integer values for `postgresql.parameters` in schema** *(backported to v1.4.1)*: `postgresql.parameters` declared `{type: string}` only, rejecting bare integers like `max_connections: 100`. An `intOrString` type alias is now emitted with `anyOf: [integer, string]` plus `x-kubernetes-int-or-string: true`. The same PR extends the `$dangerousParams` blocklist with `archive_cleanup_command` and `recovery_end_command` ([**@IvanHunters**](https://github.com/IvanHunters) in #2687). + +* **fix(platform): migrate `ephemeralStorage` to `diskSize` via pre-upgrade hook** *(backported to v1.4.1)*: PR #2454 renamed `nodeGroups[*].ephemeralStorage` to `nodeGroups[*].diskSize` with a hard `{{ fail }}` guard. A new platform pre-upgrade migration (migration 41) now walks all `kuberneteses.apps.cozystack.io` Application CRs and renames the field automatically before chart resources are applied ([**@IvanHunters**](https://github.com/IvanHunters) in #2688). + +* **fix(cluster-api): fix Kamaji OOM and set limits on unset providers** *(backported to v1.4.1)*: The Kamaji control-plane-provider resource override targeted a container named `manager`, but the image names it `controller` — the intended 1 CPU / 1024 Mi limits were silently dropped and Kamaji used 128 Mi, causing repeated OOMKilled restarts. The container name is corrected and modest resource requests and limits are added for the core, kubeadm-bootstrap, and kubevirt-infrastructure providers that previously ran as BestEffort ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2708). + +* **fix(opensearch-operator): replace deprecated `kube-rbac-proxy` image** *(backported to v1.4.1)*: The `gcr.io/kubebuilder/kube-rbac-proxy` image is no longer available. Replaced with `quay.io/brancz/kube-rbac-proxy` following the convention already used by etcd-operator and other components ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2689). + +* **fix(platform): add OpenSearch to PaaS bundle** *(backported to v1.4.2)*: The `opensearch-operator` and `opensearch-rd` PackageSources were missing from the PaaS bundle template, so the OpenSearch operator and its ApplicationDefinition were never deployed — tenants could not provision OpenSearch instances ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2648). + +* **fix(csi): route RWX Block volumes to upstream hotplug detach** *(backported to v1.4.2)*: `ControllerUnpublishVolume` selected the NFS cleanup branch using only `hasRWXAccessMode(pvc)`, so RWX Block PVCs (KubeVirt live-migration disks) were sent down the NFS path and left their `DataVolume` entry in `VM.spec.template.spec.volumes` permanently, blocking future attaches. The fix introduces `isNFSVolume(pvc)` requiring both `ReadWriteMany` and `VolumeMode=Filesystem` ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2658). + +* **fix(csi): verify VMI Ready after `kubevirt-csi` Publish to surface stuck-PVC failures** *(backported to v1.4.2)*: Upstream `ControllerPublishVolume` reported success as soon as the volume entry appeared in `VM.spec.template.spec.volumes`, even if QEMU never received `device_add`. The wrapper now re-reads `VMI.Status.VolumeStatus` and returns `codes.Unavailable` if the volume is not in `VolumeReady` phase, keeping `external-attacher` retrying and surfacing the underlying failure at the CSI layer ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2659). + +* **fix(dashboard): grant tenant dashboard read on `cozy-public` PVCs** *(backported to v1.4.3)*: The VM disk source-image dropdown returned 403 and stayed empty even when golden images existed in the `cozy-public` namespace. `get/list/watch` on PersistentVolumeClaims is now added to the `cozy:tenant:dashboard` Role ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2843). + +* **fix(api): emit `initial-events-end` bookmark for `core.cozystack.io` watches** *(backported to v1.4.3)*: `TenantSecret`, `TenantModule`, and `TenantNamespace` aggregated API resources never sent the `k8s.io/initial-events-end` bookmark required by the WatchList protocol. Client-go informers using `WatchListClient` never reached `HasSynced` and logged warnings every ~10 seconds ([**@sunib**](https://github.com/sunib) in #2786). + +* **fix(networking): point host ouroboros proxy at the root-tenant ingress** *(backported to v1.4.3)*: When `publishing.proxyProtocol` was enabled, the host-level ouroboros proxy inherited the wrong backend FQDN (a managed Kubernetes tenant cluster address). On the host, ingress-nginx is deployed as `root-ingress-controller` in `tenant-root`, so the proxy never became `Ready` ([**@lexfrei**](https://github.com/lexfrei) in #2800). + +* **fix(objectstorage-controller): propagate Bucket readiness to BucketClaim** *(backported to v1.4.3)*: The vendored COSI controller hardcoded `bucketReady=false` after dynamic provisioning and never re-read the `Bucket` to pick up the `true` transition, so `BucketAccess` was never granted and buckets ended up without credentials ([**@lexfrei**](https://github.com/lexfrei) in #2792). + +* **fix(kubernetes): stamp application lineage labels on worker node VMs** *(backported to v1.4.3)*: Worker-node VMs created by Cluster API were never stamped with `apps.cozystack.io/application.{group,kind,name}` lineage labels, preventing the dashboard from attributing their virt-launcher pods to the owning Kubernetes application. A companion fix quotes `application.name` so a purely-numeric cluster name renders as a YAML string ([**@kvaps**](https://github.com/kvaps) in #2779). + +## Security + +* **ci: add CodeQL workflow for Go (SAST)**: Adds a weekly CodeQL static analysis scan for the Go codebase ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2851). + +* **ci: add OpenSSF Scorecard workflow**: Enables continuous OpenSSF Scorecard scoring for the repository, covering vulnerability checks, branch protection, CI best practices, and contributor risk ([**@tym83**](https://github.com/tym83) in #2720). + +* **ci: pin Scorecard workflow actions by SHA**: Pins all third-party Actions used by the Scorecard workflow to digest-pinned SHAs to prevent supply-chain attacks via tag mutation ([**@tym83**](https://github.com/tym83) in #2721). + +* **fix(deps): bump Go to 1.26.4 and `x/net` to v0.55.0 to close OSV advisories**: Clears 12 Open Source Vulnerability advisories: 10 against the Go toolchain (fixed in 1.26.3 and 1.26.4) and 2 against `golang.org/x/net` (fixed in v0.55.0). This brings the OpenSSF Scorecard Vulnerabilities check to a passing state ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2852). + +## Dependencies + +* **chore(seaweedfs): bump `seaweedfs-cosi-driver` to v0.3.1** *(backported to v1.4.3)*: v0.3.1 ships a stale-socket self-heal: the COSI driver now removes any leftover UNIX socket before binding, so the objectstorage provisioner recovers automatically from a non-graceful exit instead of wedging on "bind: address already in use" ([**@lexfrei**](https://github.com/lexfrei) in #2791). + +## Development, Testing, and CI/CD + +* **feat(ci): test-impact analysis for E2E**: A new pre-E2E step analyses which packages changed in the PR diff and selects only the relevant bats test files. E2E runs that previously exercised the full test matrix now run only the impacted tests on normal PRs, with full-suite runs reserved for release PRs and `[full-e2e]` labelled PRs ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2559). + +* **ci(workflows): add SSH breakpoint on E2E failure for debug PRs**: E2E jobs on PRs labelled `debug` or `release` now drop into an interactive SSH session on failure, so engineers can inspect the cluster state without needing to reproduce the failure locally ([**@kvaps**](https://github.com/kvaps) in #2535). + +* **ci(release): auto-patch-release only the 2 newest release lines**: The automated patch-release cron now restricts backport CI to the two most recent `release-X.Y` branches, preventing unnecessary runs on end-of-life lines ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2856). + +* **ci(release): repair orphaned draft release on retag** *(backported to v1.4.3)*: When a git tag was deleted and re-created between draft creation and merge, GitHub orphaned the draft release by setting `tag_name` to `"untagged-"`. The finalize step now detects the orphaned form, falls back to matching by `name`, repairs `tag_name` via `updateRelease`, and then publishes ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2761). + +* **refactor(build): standardize image tagging to fix concurrent PR push conflicts**: Aligns image tagging across all build targets to prevent tag collision when multiple PRs push to the same registry prefix concurrently ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2711). + +* **refactor(build): mode=max registry cache with a main-only warmer**: Switches Docker BuildKit cache to `mode=max` and introduces a dedicated cache-warmer job that runs only on `main` pushes. PR builds warm-start from the registry cache without the warmer running on every PR ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2938). + +* **ci(build): isolate each PR build on its own ephemeral runner VM**: PR builds are now dispatched to fresh ephemeral runner VMs, preventing state bleed between concurrent builds and improving security isolation ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2939). + +* **test(api): add read-path authorization tests for `TenantNamespace` registry**: Adds direct unit coverage for the authorization-sensitive read paths (`Get`, `List`, `Watch`) of the TenantNamespace registry, including per-user filtering, privileged-group bypass, bookmark forwarding, field and label selector filtering, and goroutine lifecycle. A production gap (Watch goroutine never closing its result channel) is fixed as part of this work ([**@lexfrei**](https://github.com/lexfrei) in #2864). + +* **ci: add Go-scoped Renovate configuration**: Enables automated dependency updates for Go modules in the repository via Renovate ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2850). + +## Documentation + +* **chore(maintainers): update roster and add Emeritus section**: Updates the maintainer roster and adds an Emeritus section documenting the technical offboarding and reactivation process for inactive maintainers ([**@tym83**](https://github.com/tym83) in #2717, #2718). + +* **docs(harbor): document S3 object-storage (SeaweedFS) prerequisite**: Adds a Prerequisites section to the Harbor app README explaining that S3-compatible object storage is required and how to satisfy it (`seaweedfs: true` on the tenant), preventing a common silent first-deploy failure ([**@lexfrei**](https://github.com/lexfrei) in #2818). + +* **docs(gpu-operator): document passthrough variant host-driver prerequisite**: Adds documentation explaining that the `passthrough` variant of `cozystack.gpu-operator` requires that no host-installed NVIDIA driver is active and clarifies when the new `container` variant should be used instead ([**@lexfrei**](https://github.com/lexfrei) in #2767). + +* **chore(img): refresh Cozystack logo and add icon/stacked variants**: Updates the Cozystack brand assets — new horizontal wordmark, icon-only, and stacked variants — used across the dashboard and promotional materials ([**@kvapsova**](https://github.com/kvapsova) in #2746). + +* **[website] feat(blog): add etcd-operator v1alpha2 announcement**: Blog post announcing the donation of `etcd-operator` to the Cozystack project and the new `etcd-operator.cozystack.io/v1alpha2` API, covering the Membership-API lifecycle, key features, changes compared to `v1alpha1`, the in-place migration path, and a status comparison against the official `etcd-io/etcd-operator` roadmap ([**@kvaps**](https://github.com/kvaps) in cozystack/website#573). + +* **[website] feat(blog): add KubeVirt VM Disk and VM Instance how-to post**: A practical how-to blog post explaining Cozystack's VM Disk and VM Instance primitives and how they simplify running virtual machines on Kubernetes compared to raw KubeVirt ([**@tym83**](https://github.com/tym83) in cozystack/website#586). + +* **[website] docs(networking): Cilium Gateway API — architecture, security, and migration guide** *(backported to v1.4.2)*: Adds a comprehensive networking page covering the platform-level `gateway.enabled` toggle, per-tenant opt-in, inheritance model, HTTP-01 and DNS-01 certificate modes, per-service HTTPRoute and TLSRoute tables, a five-layer security model, and a step-by-step ingress-nginx migration path ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#509). + +* **[website] docs(talm): document DRBD sysctl tuning, keepalive toggle, and etcd quota** *(backported to v1.4.3)*: Adds documentation for talm covering DRBD sysctl performance tuning, the DRBD keepalive toggle, and the etcd quota configuration ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#567). + +* **[website] feat(blog): add Managed Kubernetes how-to post** *(backported to v1.4.3)*: Practical guide for deploying and using managed Kubernetes clusters within Cozystack ([**@tym83**](https://github.com/tym83) in cozystack/website#565). + +* **[website] feat(blog): add platform-managed backups introduction post** *(backported to v1.4.3)*: Blog post introducing Cozystack's platform-managed backup capabilities for stateful workloads ([**@tym83**](https://github.com/tym83) in cozystack/website#566). + +* **[website] feat(blog): add Cozystack vs OpenStack comparison post** *(backported to v1.4.4)*: Vendor-neutral comparison post covering seven dimensions — architecture, compute, networking, storage, managed services, operations, and multi-tenancy ([**@tym83**](https://github.com/tym83) in cozystack/website#580). + +* **[talm] feat(cozystack): add DRBD-oriented sysctl and etcd backend defaults** *(backported to v1.4.4)*: Adds production-tested TCP sysctls to the `cozystack` preset to prevent TCP port exhaustion during DRBD reconnect storms. An opt-in `tcpKeepaliveTuning` group is available for faster idle-socket failure detection. The etcd backend quota is exposed as `etcd.quotaBackendBytes` (default 8 GiB) ([**@IvanHunters**](https://github.com/IvanHunters) in cozystack/talm#131). + +* **[talm] fix(cli, engine): emit progress on stderr, enrich lookup error chain, retry transient failures** *(backported to v1.4.4)*: Fixes `talm template -f X > Y` producing YAML files with a stray progress line on stdout. Lookup failures now surface the resource kind, namespace, id, and dialed endpoints, with a six-class error taxonomy. Transient connectivity errors are retried up to three times with exponential backoff ([**@lexfrei**](https://github.com/lexfrei) in cozystack/talm#212). + +* **[talm] fix(engine): require a template before rendering the chart** *(backported to v1.4.4)*: Running `talm template` without `--file`/`--template` previously triggered live node discovery and produced a misleading `connection refused` error. The guard is now evaluated before any chart render or node dial ([**@lexfrei**](https://github.com/lexfrei) in cozystack/talm#217). + +* **[ansible-cozystack] feat(prepare): exclude loop and virtual devices from host LVM scanning** *(backported to v1.4.3)*: Sets an LVM `global_filter` in `/etc/lvm/lvm.conf` on all prepare playbooks so the host LVM does not scan or activate DRBD, device-mapper, zd-device, or loop-backed volume groups. Exposed as `cozystack_lvm_global_filter` inventory variable ([**@kvaps**](https://github.com/kvaps) in cozystack/ansible-cozystack#49). + +* **[ansible-cozystack] fix(prepare): enable `containerd device_ownership_from_security_context` for CDI block imports** *(backported to v1.4.3)*: Adds a k3s containerd drop-in config enabling `device_ownership_from_security_context` on the CRI plugin. Without this, the KubeVirt CDI importer fails with "cannot open /dev/cdi-block-volume: Permission denied" when writing VM disk images into raw block volumes ([**@lexfrei**](https://github.com/lexfrei) in cozystack/ansible-cozystack#48). + +* **[ansible-cozystack] feat(prepare): exclude loop devices from LVM `global_filter`** *(backported to v1.4.4)*: Adds `/dev/loop.*` to the LVM `global_filter` in the generated Talos machine config, preventing unexpected LVM activation on boot ([**@kvaps**](https://github.com/kvaps) in cozystack/ansible-cozystack#51). + +## Contributors + +We'd like to thank all contributors who made this release possible: + +* [**@androndo**](https://github.com/androndo) +* [**@Arsolitt**](https://github.com/Arsolitt) +* [**@elaugaste**](https://github.com/elaugaste) +* [**@IvanHunters**](https://github.com/IvanHunters) +* [**@kvaps**](https://github.com/kvaps) +* [**@kvapsova**](https://github.com/kvapsova) +* [**@lexfrei**](https://github.com/lexfrei) +* [**@lllamnyp**](https://github.com/lllamnyp) +* [**@myasnikovdaniil**](https://github.com/myasnikovdaniil) +* [**@sunib**](https://github.com/sunib) +* [**@tym83**](https://github.com/tym83) + +### New Contributors + +We're excited to welcome our first-time contributors: + +* [**@elaugaste**](https://github.com/elaugaste) - First contribution! + +**Full Changelog**: https://github.com/cozystack/cozystack/compare/v1.4.0...v1.5.0 From 62646bff1536e48e6ef6f03855441a1f97336f7b Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Mon, 22 Jun 2026 20:24:27 +0500 Subject: [PATCH 2/2] docs: expand v1.5.0 changelog with platform-component and upgrade detail Merge the AI-generated v1.5.0 changelog with a fuller pass: - Add an "Upgrade Notes and Required Actions" section (K8s 1.33+, removal of upgrade.force, GPU permittedHostDevices ownership, MetalLB FRR-K8s / HTTPS metrics, automatic TLS on externally-published services). - Expand "Platform Components" with per-component upstream highlights and CVE references (Flux v2.8 CVE-2026-23990, MetalLB v0.16 CVE-2025-22874), plus the new frr-k8s subchart and kuberture package. - Scope the storageClass-immutability entry (#2639) to UI-only enforcement, with apiserver enforcement tracked in #2657. - Broaden Development/Testing/CI coverage and add the v1.4.x website docs backports (#565, #566, #567, #580). - Correct the loop-device LVM-filter attribution to talm#215 (the prior ansible-cozystack#51 reference was a dependency bump, not this change). Signed-off-by: Myasnikov Daniil --- docs/changelogs/v1.5.0.md | 359 +++++++++++++++++++++++++------------- 1 file changed, 240 insertions(+), 119 deletions(-) diff --git a/docs/changelogs/v1.5.0.md b/docs/changelogs/v1.5.0.md index 0f4f4ad03..f2edb8d8d 100644 --- a/docs/changelogs/v1.5.0.md +++ b/docs/changelogs/v1.5.0.md @@ -4,233 +4,354 @@ https://github.com/cozystack/cozystack/releases/tag/v1.5.0 # Cozystack v1.5.0 -Cozystack v1.5.0 ships **Flux shard-operator for tenant Helm-controller sharding**, **Cilium-backed Gateway API** as a first-class networking layer, **TLS support for all major managed services** (Kafka, NATS, Qdrant, PostgreSQL), an **operator-provided wildcard certificate** for platform ingress, an **Option resource for dynamic dashboard dropdowns**, full **HelmRelease generation knobs** surfaced as operator and API flags, **tenant VM start/stop/restart RBAC**, **GPU passthrough auto-wiring**, a new **Job backup strategy**, an **etcd backup-strategy controller**, **velero installed by default**, and a **tenant overview Grafana dashboard** for platform admins. The release rolls up every fix from v1.4.1 through v1.4.4. +Cozystack v1.5.0 brings **Gateway API support via Cilium** as an opt-in ingress layer alongside ingress-nginx, **TLS for managed databases and messaging** (Kafka, NATS, Qdrant, and PostgreSQL external endpoints), **backups that work out of the box** with a platform-managed default `BackupClass`, a shared backups bucket, a new etcd backup strategy and a generic Job strategy, the **Flux v2.8 upgrade** with strict server-side apply and kstatus health checking, a new **flux-shard-operator** that spreads tenant HelmReleases across helm-controller shards so one noisy tenant can no longer stall the others, **operator-provided wildcard certificates** for platform and root-tenant ingress, **GPU passthrough that works without manual KubeVirt patching**, a **deletion-protection guardrail** for critical platform objects, and **runtime-populated dashboard dropdowns** via a new `Option` API. The release also rolls up every fix from v1.4.1 through v1.4.4. -Platform component bumps: **Flux v2.7.3 → v2.8.0**, **MetalLB v0.15.2 → v0.16.1**, **SeaweedFS 4.05 → 4.31**, **ouroboros v0.7.2 → v0.8.0**, **flux-operator v0.48.0**, and **seaweedfs-cosi-driver v0.3.1** *(backported to v1.4.3)*. +Platform components bumped in this release: **Flux v2.7.3 → v2.8.0** (flux-operator/flux-instance charts v0.33.0 → v0.50.0), **MetalLB v0.15.2 → v0.16.1** (FRR-K8s is now the default BGP backend), **SeaweedFS 4.05 → 4.31**, **etcd-operator v0.4.3 → v0.4.5**, **ouroboros v0.7.2 → v0.8.0**, **seaweedfs-cosi-driver v0.3.1**, and the new **kuberture** system package. -> **Note:** Items marked *(backported to v1.4.x)* were also shipped in v1.4.1, v1.4.2, v1.4.3, or v1.4.4 patch releases. +> **Note:** Items marked *(backported to v1.4.x)* were also shipped in the v1.4.1, v1.4.2, v1.4.3, or v1.4.4 patch releases. ## Feature Highlights -### Flux Shard Operator: Tenant Helm-Controller Sharding - -The new `flux-shard-operator` system package spreads tenant HelmReleases across multiple dedicated helm-controller shards, preventing a noisy tenant — such as one with a HelmRelease stuck in infinite remediation — from starving the reconciliation queue for other tenants. - -The operator has two parts: a **placement controller** that assigns each tenant to a shard using greedy least-loaded distribution weighted by HelmRelease count, and a **shard runtime** that provisions the correct number of helm-controller Deployments cloned from the existing `flux-aio` Deployment — so images and feature gates remain version-synced automatically. The unit of placement is the tenant namespace (all of its HelmReleases share one shard label), and the controller supports pinned tenants, threshold-gated rebalance, paced moves, and never moves tenants that are being deleted. Watches are metadata-only, so the placement controller does not decode the helm-controller status-patch firehose. - -`flux-aio` itself is unchanged and continues to serve all tenants without a shard key in clusters that choose not to enable the operator ([**@kvaps**](https://github.com/kvaps) in #2821). - ### Gateway API Support via Cilium -Cozystack now ships **Cilium Gateway API** as a first-class networking layer alongside ingress-nginx. A new `TenantGateway` CRD is reconciled by `cozystack-controller`, which provisions a `Gateway`, per-tenant `Issuer`, `Certificate` (wildcard or per-listener), and HTTP→HTTPS redirect `HTTPRoute` from a single tenant-level declaration. The Helm chart does not render Gateway or Certificate resources directly, preventing the Helm-vs-controller race on `Gateway.spec.listeners` that route-driven listener additions would otherwise trigger. +Cozystack-native services can now be exposed through the **Gateway API** backed by Cilium, as an opt-in alternative to the existing per-tenant ingress-nginx controllers. The feature is materialized per tenant through a new `gateway.cozystack.io/v1alpha1 TenantGateway` CRD reconciled by `cozystack-controller`. -Tenants opt in by setting `spec.gateway` on their Tenant CR. A child tenant inherits the parent's Gateway by default via the `namespace.cozystack.io/gateway` label. DNS-01 SAN expansion lets child tenants register their apex domains as additional SANs on the parent wildcard certificate without a separate ACME round trip. The platform toggles ingress-nginx ↔ HTTPRoute automatically based on `_cluster.gateway-enabled`. External-DNS gains `gateway-tlsroute` support alongside `gateway-httproute`. +Enable it at the platform level with `publishing.gateway.enabled=true`, then either give a tenant its own Gateway, LoadBalancer IP, and certificate with `tenant.spec.gateway=true`, or leave it unset and let the tenant inherit the nearest ancestor's Gateway through the same label-based selector model that already drives ingress inheritance. Two certificate solver modes are supported: HTTP-01 (the default — a per-app certificate with zero platform configuration for new apps) and DNS-01 (opt-in — a single wildcard certificate covering an apex, with cloudflare, route53, digitalocean, and rfc2136 providers). -All existing ingress-nginx flows are unaffected — the Gateway API layer is purely additive and disabled by default ([**@lexfrei**](https://github.com/lexfrei) in #2470). +Defaults stay on ingress-nginx, so existing clusters are unchanged. Two things to be aware of: Cilium Envoy / Gateway API is now always enabled (an extra `cilium-envoy` DaemonSet, roughly 100 MB RAM per node at idle), and `cozystack-api` now invokes admission (`createValidation` / `deleteValidation`) on Create and Delete for `apps.cozystack.io/*` — so any custom ValidatingAdmissionPolicies or webhooks on those kinds will now fire on all three verbs. See the [Gateway API guide](https://cozystack.io/docs/v1.5/operations/networking/gateway-api/) ([**@lexfrei**](https://github.com/lexfrei) in #2470). -### TLS for Managed Services: Kafka, NATS, Qdrant, and PostgreSQL +### TLS for Managed Databases and Messaging -All four major stateful managed services now support TLS, using a consistent tri-state `tls.enabled` field: +Four managed-app charts gain TLS support driven by a single `tls.enabled` value with consistent tri-state semantics: when unset it inherits `external` (TLS auto-on when the service is published externally, off when cluster-internal), and an explicit `true`/`false` always wins. In every case the trust anchor is a chart- or operator-managed self-signed CA that clients retrieve and pin — there is no publicly trusted CA. The one upgrade-time behaviour change to plan for: existing instances with `external: true` flip to TLS-on after upgrade; cluster-internal instances are unaffected. -- **null / unset** — auto-inherits the value of `external` (TLS on when the service is published externally, off for cluster-internal deployments). This keeps the default behaviour unchanged for existing deployments. -- **true** — TLS always on. -- **false** — TLS always off. +* **Kafka** serves TLS on its external LoadBalancer listener (port 9094), with certificates managed end-to-end by the Strimzi operator; clients trust via the operator-published `-cluster-ca-cert` / `-clients-ca-cert` secrets. The external listener is now gated only on `external: true`, decoupled from `tls.enabled` ([**@Arsolitt**](https://github.com/Arsolitt) in #2681). +* **NATS** and **Qdrant** gain TLS via a self-contained cert-manager chain (self-signed Issuer → CA → leaf) rendered in the tenant namespace; NATS covers both client connections and cluster routes, Qdrant covers REST and gRPC. Clients trust the `-ca` secret ([**@Arsolitt**](https://github.com/Arsolitt) in #2684, #2685). +* **PostgreSQL** — CNPG already serves TLS unconditionally, so `tls.enabled` here injects the external hostname into the operator-managed server certificate's SANs (when `external: true`), so `sslmode=verify-full` works against the external endpoint. Clients retrieve `ca.crt` from the `-credentials` secret ([**@Arsolitt**](https://github.com/Arsolitt) in #2686). -**Kafka** wires TLS through the Strimzi listener configuration. The operator manages the cluster PKI end-to-end and publishes `-cluster-ca-cert` and `-clients-ca-cert` for client trust. The external LoadBalancer listener on port 9094 is independently gated on `external: true`; `tls.enabled` only governs whether that listener uses TLS ([**@Arsolitt**](https://github.com/Arsolitt) in #2681). +### Backups That Work Out of the Box -**NATS** and **Qdrant** use a self-contained cert-manager chain rendered in the tenant namespace: a self-signed Issuer, CA Certificate, CA Issuer, and leaf Certificate with SAN coverage for all three DNS forms per service plus the external hostname when `external: true`. Qdrant adds short-form per-replica SANs for p2p TLS hostname verification and a Reloader annotation to trigger pod restarts on cert rotation ([**@Arsolitt**](https://github.com/Arsolitt) in #2684 and #2685). +This release closes the gap between "backup machinery is installed" and "backups actually work without per-app S3 configuration." -**PostgreSQL** uses the CNPG operator's built-in cert management instead of a chart-rendered chain. The operator auto-generates a self-signed CA and signs server, client, and replication certs. When `external: true` and TLS is on, the chart sets `spec.certificates.serverAltDNSNames` so the operator appends the external hostname to the leaf cert. The trust anchor for clients is the `-ca` Opaque Secret (key `ca.crt`) ([**@Arsolitt**](https://github.com/Arsolitt) in #2686). +A platform-managed default **`BackupClass` (`cozy-default`)** is now shipped, backed by a system bucket (`cozy-backups`). Apps opt in with a `useSystemBucket` flag, after which the platform projects shared backup credentials into the tenant namespace (with RBAC isolation and projection metrics) and skips per-release credential Secrets. Default strategies are provided for every backup-capable app — Velero for VMDisk/VMInstance, CNPG for PostgreSQL, MariaDB, Altinity for ClickHouse, FoundationDB, and etcd — and a Velero `BackupStorageLocation` is wired to the system bucket. The legacy per-tenant S3 fields on Postgres and ClickHouse are deprecated in favour of this default flow ([**@androndo**](https://github.com/androndo) in #2716). -### Operator-Provided Wildcard Certificate for Platform Ingress +To make that default flow reliable, **Velero is now a default system package** rather than optional. This fixes a deterministic failure where the default `backupstrategy-controller` (which hard-depends on Velero) sat in `DependenciesNotReady` and kept the platform HelmRelease from ever reaching Ready. Existing clusters get Velero in the `cozy-velero` namespace on upgrade; opt out via `bundles.disabledPackages` if you do not back up VMs ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2833). -Operators can now serve platform services and the root tenant's ingress under a pre-existing wildcard TLS certificate by setting a single `publishing.certificates.wildcardSecretName` value that names a TLS Secret they have already provisioned. Only the Secret name travels through the `cozystack-values` channel — the certificate and private key are never broadcast to tenant namespaces. +Two new backup strategies join the catalog: an **etcd** strategy (cluster-scoped `strategy.backups.cozystack.io Etcd` CRD, S3-only, with snapshot BackupJob and a destructive in-place RestoreJob), and a generic application-agnostic **Job** strategy where the operator supplies a Kubernetes Job template that Cozystack renders and runs as a one-shot backup, then re-renders with `.Mode == "restore"` for recovery — the generic counterpart of the app-specific drivers ([**@androndo**](https://github.com/androndo) in #2641, [**@lllamnyp**](https://github.com/lllamnyp) in #1721). -When a wildcard Secret is configured, `ingress-nginx` is reconfigured to use it as the default SSL certificate, eliminating the need for per-host ACME certificates on the root domain. Existing per-host ACME flows are unaffected for operators who do not set this value ([**@lexfrei**](https://github.com/lexfrei) in #2819). +### Flux v2.8 Upgrade with Strict Server-Side Apply -### Option Resource and Dynamic Dashboard Dropdowns +Flux is upgraded from v2.7.3 to v2.8.0 across both the embedded management-cluster Flux and the optional tenant Flux addon (flux-operator / flux-instance charts move v0.33.0 → v0.50.0). Flux v2.8's helm-controller v1.5 ships Server-Side Apply with `--force-conflicts` and kstatus-based health checking by default — so misplaced chart fields that v2.7 silently dropped are now hard errors (fixed here for foundationdb, kafka, kubevirt-instancetypes, vm-instance, and the platform chart), and parent HelmReleases now wait for every child resource to be Ready before reporting Ready themselves. -A new `core.cozystack.io/v1alpha1 Option` resource provides **runtime-populated dropdowns** for the dashboard's create and edit forms. Fields that reference live cluster resources — GPU device names, KubeVirt instancetypes and preferences, Multus networks, VM images, bucket storage pools, storage classes, backup classes and plans — can now render as curated option lists instead of free-text inputs that drift from reality. +**Action required on upgrade:** Kubernetes 1.33+ is now required for the management cluster (and for any tenant cluster enabling the Flux addon). The `upgrade.force: true` knob is removed, so immutable-field changes (for example StatefulSet `volumeClaimTemplates` / `serviceName`) no longer self-heal and must be recreated manually (`kubectl delete sts --cascade=orphan`). Persistent TPM/EFI is re-enabled for Windows KubeVirt preferences (each affected VM provisions one extra RWO PVC), and FoundationDB `imageType` is pinned to `split` to keep upgrades non-disruptive ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2602). -Option items are computed on read by an in-process provider registry using a privileged client, so tenants see current values without needing broad cluster read access. Providers cover: `gpu`, `instancetype`, `instanceprofile`, `network`, `image`, `storagepool`, `storageclass`, `backupclass`, `plan`, `backup`, and `appkind`. +### flux-shard-operator: Tenant helm-controller Sharding -Chart authors expose the feature via a new `@x-cozystack-options` directive in `values.yaml`; `cozyvalues-gen` propagates it through all generated artifacts (schema, Go types, and `-rd` ApplicationDefinitions) so the extension is carried into the served OpenAPI ([**@kvaps**](https://github.com/kvaps) in #2778). +A new **flux-shard-operator** spreads tenant HelmReleases across multiple helm-controller shards, so one noisy tenant — for example a HelmRelease stuck in infinite remediation — can no longer degrade reconciliation for everyone else. Placement is per-tenant (all of a tenant's HelmReleases share one shard), assigned greedily by least load, with a CREATE-time mutating webhook stamping the shard label on each HelmRelease. -### HelmRelease Generation Knobs and Per-App Timeouts +It ships with `shardCount: auto` by default, which sizes shards from the tenant HelmRelease count: small clusters stay at a single shard (today's behaviour) while large fleets shard out automatically, and an integer pins the count explicitly. The legacy hand-rolled `flux-tenants` deployment is drained and retired automatically by migration 44 ([**@kvaps**](https://github.com/kvaps) in #2821). -The cozystack API server now exposes the same HelmRelease generation flags introduced for the operator in v1.4.0: `--helmrelease-interval`, `--helmrelease-retry-interval`, `--helmrelease-install-timeout`, `--helmrelease-upgrade-timeout`, and `--helmrelease-max-history`. Previously, the `cozystack-api` path that converts Application CRs to HelmReleases hardcoded its own values (5-minute interval, `Remediation{Retries:-1}`) while the operator path used flags — creating divergent behaviour for the same application kind depending on which code path generated the HelmRelease. +### Operator-Provided Wildcard Certificates -A companion change introduces a **per-Application Install/Upgrade timeout** driven by a `release.cozystack.io/helm-upgrade-timeout` annotation on ApplicationDefinition. This extends the existing `helm-install-timeout` annotation support so individual application kinds can declare asymmetric install vs. upgrade budgets — for example a managed service with a fast install but a slow rolling upgrade ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2571). +Operators can now serve platform services and the root tenant's ingress under a pre-existing wildcard TLS certificate instead of minting per-host ACME certificates. Set `publishing.certificates.wildcardSecretName` to the name of a TLS Secret already created in the publishing namespace (`tenant-root` by default) — only the Secret name travels over the cozystack-values channel, never the key material. + +It works on both ingress paths: with ingress-nginx the controller serves it as `--default-ssl-certificate` and platform Ingresses drop their cert-manager annotations; with Gateway API a new `existingSecret` TenantGateway cert mode references the Secret directly and provisions no Issuer or Certificate. Scope is the root tenant only for now; extending wildcard mode to child tenants is a follow-up ([**@lexfrei**](https://github.com/lexfrei) in #2819). + +### GPU Passthrough Out of the Box + +GPU enablement is wired up across all three paths a GPU can reach a workload, each of which previously needed manual reconciliation. + +For **tenant Kubernetes**, node-groups declaring `gpus` automatically get the `gpu=on` kubelet label (so HAMi's device plugin schedules and advertises `nvidia.com/gpu`), and the tenant gpu-operator loads the driver with `NVreg_NvLinkDisable=1`, fixing single-SXM-GPU passthrough that previously hung at "Fabric State: In Progress" with CUDA "system not yet initialized." Both defaults are overridable via `addons.gpuOperator.valuesOverride` ([**@kvaps**](https://github.com/kvaps) in #2780). + +For **KubeVirt VMs**, enabling `cozystack.gpu-operator` now auto-populates the KubeVirt CR — injecting the `HostDevices` feature gate and filling `permittedHostDevices` (plus `mediatedDevicesConfiguration` for vGPU) from shipped NVIDIA default tables — so GPU VMs schedule without a manual `kubectl patch`. **Action required:** the bundle now owns `spec.configuration.permittedHostDevices`, so the first reconcile after upgrade overwrites any hand-edited entries; move custom device entries into `.gpu.permittedHostDevices` before upgrading and verify each `resourceName` matches what nodes advertise ([**@lexfrei**](https://github.com/lexfrei) in #2768). A third gpu-operator `container` variant is also added for hosts where the NVIDIA driver and container-toolkit are already installed by the OS, exposing GPUs to regular containerized pods via the device plugin only ([**@lexfrei**](https://github.com/lexfrei) in #2766). + +### Deletion-Protection Guardrail + +A new deletion-protection guardrail blocks `DELETE` on critical platform objects labeled `platform.cozystack.io/no-delete=true`, evaluated in-process by the kube-apiserver via a ValidatingAdmissionPolicy — no webhook, DaemonSet, TLS, or extra image. Protected objects in this release include the `cozy-system` and `tenant-root` namespaces, the `tenant-root` HelmRelease, the `cozystack-version` ConfigMap, the `cozystack-packages` OCIRepository, the cert-manager ClusterIssuers, the LinstorCluster, and the packages CRDs; a migration backfills the label onto existing resources. + +To delete a protected object, remove the label first (`kubectl label platform.cozystack.io/no-delete-`). Requires Kubernetes 1.30+ ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2650). + +### Runtime-Populated Dashboard Dropdowns + +A new generic mechanism powers runtime-populated dropdowns in dashboard create/edit forms, so fields that reference live cluster resources — GPU devices, KubeVirt instancetypes and preferences, Multus networks, VM images, storage pools, storage classes, backup classes and plans — become accurate dropdowns instead of free text or stale static enums. It introduces a namespaced, read-only `Option` resource (`core.cozystack.io/v1alpha1`) computed on read by a privileged in-process provider registry, and an `x-cozystack-options` schema keyword that app charts declare via the cozyvalues-gen `@x-cozystack-options` directive. Tenants get read-only access to options in their own namespace (from `cozy:tenant:view:base` upward), so curated lists populate without granting broad cluster reads ([**@kvaps**](https://github.com/kvaps) in #2778). + +## Upgrade Notes and Required Actions + +Most operators can take v1.5.0 with no manual action — the in-platform migrations handle config rewrites automatically — but five changes warrant attention. + +* **Kubernetes 1.33+ required for the management cluster.** With #2602, Flux jumps v2.7.3 → v2.8.0 and helm-controller v1.5 requires Kubernetes 1.33 or newer on the management cluster (and on any tenant cluster that enables the Flux addon). Upgrade Kubernetes before taking this release if you are below 1.33. + +* **`upgrade.force` is gone — immutable-field changes no longer self-heal.** Also with #2602, the helm-controller `upgrade.force: true` knob is removed. If a chart upgrade changes an immutable field (StatefulSet `volumeClaimTemplates`, `serviceName`, etc.) the apply now fails instead of silently force-replacing; recreate the object manually with `kubectl delete sts --cascade=orphan` and let Flux re-reconcile. + +* **GPU VM operators must move custom host devices before upgrading.** With #2768, when `cozystack.gpu-operator` is enabled the platform takes ownership of `KubeVirt.spec.configuration.permittedHostDevices` and overwrites it on the first reconcile. Move any hand-edited `permittedHostDevices` / `mediatedDevicesConfiguration` entries into the `.gpu.permittedHostDevices` value before upgrading, and confirm each `resourceName` matches what your nodes advertise. + +* **MetalLB switches to the FRR-K8s BGP backend; metrics are now HTTPS-only.** With #2699, MetalLB jumps v0.15.2 → v0.16.1 and adopts the upstream-default FRR-K8s backend (the classic FRR mode is deprecated). Metrics endpoints moved from plain HTTP to HTTPS only (kube-rbac-proxy replaced by native TLS + RBAC), so any scrape config pointing at the old HTTP metrics endpoints must be updated. The host-network port denylist is rotated to match the new listener set, including the dedicated `/healthz` / `/readyz` probe port 17472. + +* **Externally published databases and messaging gain TLS automatically.** With #2681 / #2684 / #2685 / #2686, instances with `external: true` flip to TLS-on after upgrade. Because the trust anchor is a self-signed CA, external clients must retrieve and pin the CA (and PostgreSQL clients using `sslmode=verify-full` benefit from the new external-hostname SAN). Cluster-internal instances are unaffected. + +## Platform Components + +* **Flux**: v2.7.3 → v2.8.0 (flux-operator / flux-instance charts v0.33.0 → v0.50.0). helm-controller v1.5 with embedded Helm v4 brings strict Server-Side Apply and kstatus health checking; see Feature Highlights for the upgrade impact. ([release notes](https://github.com/controlplaneio-fluxcd/flux-operator/releases/tag/v0.50.0)) ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2602) + - flux-operator gained an optional **Flux Status Page Web UI** with anonymous or OIDC auth and RBAC-gated actions (suspend/resume, rollout restart, run job, pod delete) — shipped opt-in in the cozystack package (v0.37.0+). + - **Security (CVE-2026-23990 / GHSA-4xh5-jcj2-ch8q, v0.40.0)**: fixed a Web UI impersonation bypass via empty OIDC claims — relevant only if the new Web UI is exposed with OIDC. + - `ResourceSetInputProvider` gained GitLab Environments, Gitea/Forgejo, AWS CodeCommit, and ExternalService provider types; `ResourceSet` gained `checksumFrom` (rollout restart on external secret/configmap change) and `includeEmptyProviders`. + - **Breaking:** the `--disable-wait-interruption` flag and `DISABLE_WAIT_INTERRUPTION` env var were removed (v0.39.0); flux-operator's CRD migration on Flux minor upgrades was fixed (v0.41.0) and a `flux-operator migrate` command added (v0.48.0). + +* **MetalLB**: v0.15.2 → v0.16.1. FRR-K8s is now the default BGP backend (Cozystack vendors the matching `frr-k8s` subchart at v0.0.25); classic FRR mode is deprecated. See Upgrade Notes for the HTTPS-only metrics change. ([release notes](https://github.com/metallb/metallb/releases/tag/v0.16.0)) ([**@lexfrei**](https://github.com/lexfrei) in #2699) + - New BGP features: per-peer `localASN` on `BGPPeer`, configurable FRR config-reload debounce, and a `ServiceSelector` on advertisements to target a subset of services. + - v0.15.3 added a `ConfigurationState` CRD to surface configuration errors, NetworkPolicy support in the chart, and fixed **CVE-2025-22874** in the images with a hardened pod security context. + +* **SeaweedFS**: 4.05 → 4.31 (chart 4.0.405 → 4.31.0). The bump clears the upstream 4.23 hazard flagged "not safe for erasure coding and multi-disk volume servers" and lands a large batch of S3-API correctness fixes (versioned-object semantics, Hadoop S3A multipart-ETag compatibility, bucket-quota read-only enforcement) plus EC bitrot detection, volume-server write-stall fixes, and new `/healthz` / `/readyz` probes. ([release notes](https://github.com/seaweedfs/seaweedfs/releases)) ([**@lexfrei**](https://github.com/lexfrei) in #2834) + - **Follow-up fixes required after the bump** (within this release): S3 TLS and the COSI provisioner ServiceAccount were restored (#2916), and the `-lock` BucketClass / s3 service name were restored with `volumeSizeLimitMB` dropped (#2943). + +* **etcd-operator**: v0.4.3 → v0.4.5. v0.4.4 added an `EtcdBackupStatus.snapshot` field describing the created backup artifact and moved `127.0.0.1` into `ipAddresses`; v0.4.5 fixed a broken restore-datadir path (restore was non-functional before this). ([release notes](https://github.com/aenix-io/etcd-operator/releases/tag/v0.4.5)) (bumped alongside the etcd backup-strategy controller by [**@androndo**](https://github.com/androndo) in #2641) + +* **ouroboros**: v0.7.2 → v0.8.0. Now logs an explicit reason when its TCP backend readiness check fails — making stuck-proxy situations immediately diagnosable instead of silently `NotReady` — and migrates the kubectl sidecar image from Docker Hub to `mirror.gcr.io` to avoid anonymous pull rate-limits *(backported to v1.4.3 via #2835)* ([**@lexfrei**](https://github.com/lexfrei) in #2807). + +* **seaweedfs-cosi-driver**: bumped to v0.3.1 with a stale-socket self-heal — the COSI driver removes any leftover UNIX socket before binding, so the objectstorage provisioner recovers automatically from `CrashLoopBackOff` after a non-graceful exit instead of wedging on "bind: address already in use" *(backported to v1.4.3 via #2827)* ([**@lexfrei**](https://github.com/lexfrei) in #2791). + +* **kuberture**: new optional system package (v0.1.1). A controller that bridges an external-dns gap (external-dns cannot read EndpointSlices): it watches the `default/kubernetes` API-server EndpointSlice and emits annotated headless Services that external-dns consumes to publish the Kubernetes API endpoint to DNS. Off by default; enable via `bundles.enabledPackages` and declare at least one `config.outputs` entry. ([sources](https://github.com/lexfrei/kuberture)) ([**@lexfrei**](https://github.com/lexfrei) in #2647) ## Major Features and Improvements -* **feat(rbac): allow tenants to start, stop, and restart their VMs**: Grants tenant users permission to call the KubeVirt VM power-control subresources (`virtualmachines/{name}/start|stop|restart`) by adding them to the `cozy:tenant:use:base` ClusterRole. Previously the dashboard VM power buttons returned 403 even for tenant admins, while VNC and console subresources were already granted at the same level ([**@kvaps**](https://github.com/kvaps) in #2777). +* **[networking] Gateway API support via Cilium**: See Feature Highlights — opt-in `TenantGateway` CRD, platform `publishing.gateway.enabled` toggle, per-tenant opt-in with HTTP-01/DNS-01 cert modes; supersedes #2213 ([**@lexfrei**](https://github.com/lexfrei) in #2470). -* **feat(kubernetes): enable GPU passthrough out-of-the-box**: Tenant Kubernetes clusters with GPU node-groups now have passthrough working without manual intervention. Worker nodes in GPU node-groups are automatically labeled `gpu=on`, and the tenant gpu-operator is configured to load the NVIDIA driver with `NVreg_NvLinkDisable=1` to avoid NvLink errors ([**@kvaps**](https://github.com/kvaps) in #2780). +* **[fluxcd] Add flux-shard-operator for tenant helm-controller sharding**: See Feature Highlights — spreads tenant HelmReleases across helm-controller shards with `shardCount: auto`; retires the hand-rolled `flux-tenants` deployment via migration 44 ([**@kvaps**](https://github.com/kvaps) in #2821). -* **feat(platform): auto-wire KubeVirt `permittedHostDevices` and `HostDevices` feature gate when `gpu-operator` is enabled**: When the passthrough or vGPU variant of `cozystack.gpu-operator` is deployed, the platform now automatically patches the KubeVirt CR to add `HostDevices` to `developerConfiguration.featureGates` and list the detected GPU resources under `permittedHostDevices`. Previously the GPU Operator reported green and the resource appeared in node allocatable, but every VM referencing it via `domain.devices.hostDevices` was rejected by the admission webhook ([**@lexfrei**](https://github.com/lexfrei) in #2768). +* **[flux] Upgrade to Flux v2.8.0**: See Feature Highlights and Platform Components — strict SSA + kstatus, chart fixes for foundationdb/kafka/kubevirt-instancetypes/vm-instance/platform, Kubernetes 1.33+ requirement, `upgrade.force` removed (folds #2612) ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2602). -* **feat(gpu-operator): add container variant for hosts with a preinstalled driver**: A third `cozystack.gpu-operator` variant is now available for hosts where the NVIDIA driver and `nvidia-container-toolkit` are installed by the OS package manager (the typical Ubuntu/Debian GPU node shape). The existing `passthrough` variant runs `vfioManager` to bind `vfio-pci` — which conflicts with an active host driver — and `vgpu` requires a proprietary vGPU Manager; neither worked on this host shape ([**@lexfrei**](https://github.com/lexfrei) in #2766). +* **[platform] Operator-provided wildcard certificate for platform and tenant ingress**: See Feature Highlights — `publishing.certificates.wildcardSecretName` serves a pre-existing wildcard cert on ingress-nginx (`--default-ssl-certificate`) and Gateway API (`existingSecret` mode); root tenant only ([**@lexfrei**](https://github.com/lexfrei) in #2819). -* **feat(platform): add deletion-protection guardrail via `ValidatingAdmissionPolicy`**: A new VAP blocks `DELETE` on any platform object labeled `platform.cozystack.io/no-delete=true` using native in-process CEL evaluation — no webhook DaemonSet, no Service, no TLS, no additional image. The policy and its binding are themselves self-protected, and a `migration 42` backfills the label on the `cozystack-version` ConfigMap. The policy is gated on Kubernetes 1.30+ capability detection ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2650). +* **[platform] Add default backupclass**: See Feature Highlights — platform-managed `cozy-default` BackupClass backed by the `cozy-backups` system bucket, `useSystemBucket` opt-in with projected shared credentials, default strategies for every backup-capable app ([**@androndo**](https://github.com/androndo) in #2716). -* **feat(backups): implement Job backup strategy**: Adds a generic, application-agnostic `strategy.backups.cozystack.io/Job` driver that runs a user-supplied Kubernetes `Job` to create and restore backups. This complements the application-specific strategy controllers and allows operators to wire any workload into the backup framework ([**@lllamnyp**](https://github.com/lllamnyp) in #1721). +* **[platform] Add backup-strategy controller for etcd**: See Feature Highlights — cluster-scoped `strategy.backups.cozystack.io Etcd` CRD with snapshot BackupJob and in-place RestoreJob; bumps etcd-operator to v0.4.5 ([**@androndo**](https://github.com/androndo) in #2641). -* **feat(platform): add backup-strategy controller for etcd**: Extends the backupstrategy-controller with an etcd-specific driver, allowing operators to declare backup strategies for etcd clusters managed by etcd-operator. The controller composes `EtcdBackup` and `EtcdBackupSchedule` CRs and reconciles them against an S3-compatible object store ([**@androndo**](https://github.com/androndo) in #2641). +* **[backups] Implement Job backup strategy**: A generic, application-agnostic `strategy.backups.cozystack.io/Job` strategy — the operator supplies a Kubernetes Job template that Cozystack renders and runs as a one-shot backup, emitting a `Backup` artifact on completion; restore re-renders the same template with `.Mode == "restore"` (cross-namespace restore is not supported) ([**@lllamnyp**](https://github.com/lllamnyp) in #1721). -* **feat(platform): add default backupclass**: The platform now ships a default `BackupClass` resource so backup strategies work out-of-the-box without operators needing to create a class manually on fresh installs ([**@androndo**](https://github.com/androndo) in #2716). +* **[platform] Install velero by default**: See Feature Highlights — Velero moves from optional to default so the default `backupstrategy-controller` no longer blocks the platform HelmRelease; opt out via `bundles.disabledPackages` ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2833). -* **feat(platform): install velero by default**: The `backupstrategy-controller` declares a hard dependency on `cozystack.velero`, but Velero remained an optional package — causing the controller's Package to stay `DependenciesNotReady` indefinitely on any bundle that did not explicitly enable it. Velero is now enabled by default in the platform bundle ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2833). +* **[api] Add Option resource and x-cozystack-options for dynamic form dropdowns**: See Feature Highlights — read-only `core.cozystack.io/v1alpha1 Option` resource computed on read, `x-cozystack-options` schema keyword, tenant-scoped read access ([**@kvaps**](https://github.com/kvaps) in #2778). -* **feat(apps): mark `storageClass` fields as immutable on stateful applications**: The `storageClass` field on stateful app types (including managed databases and Kubernetes worker node-groups) is now marked immutable via the `x-kubernetes-immutable` schema extension, preventing silent in-place changes that would require manual PVC migration ([**@lexfrei**](https://github.com/lexfrei) in #2639). +* **[api] Expose HelmRelease generation knobs as cozystack-api flags**: A follow-up to v1.4.0's #2509 that brings the same HelmRelease generation knobs to the second HelmRelease-generating path. The `cozystack-api` `convertApplicationToHelmRelease` path previously hardcoded `Interval: 5m`, `Remediation{Retries: -1}`, and no `Strategy`/`MaxHistory`; it now honours the `--helmrelease-interval`, `--helmrelease-retry-interval`, `--helmrelease-install-timeout`, `--helmrelease-upgrade-timeout`, and `--helmrelease-max-history` flags so the operator-generated and api-generated HelmReleases behave identically ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2571). -* **feat(platform): add kuberture as optional system package**: [kuberture](https://github.com/leaseweb/kuberture) is now available as an opt-in system package (`bundles.enabledPackages`). kuberture publishes the managed Kubernetes API endpoint as a DNS record via external-dns, enabling `kubectl` access to tenant clusters without a LoadBalancer or manual DNS entry ([**@lexfrei**](https://github.com/lexfrei) in #2647). +* **[platform] Add deletion-protection guardrail via ValidatingAdmissionPolicy**: See Feature Highlights — `platform.cozystack.io/no-delete=true` blocks DELETE on critical platform objects in-apiserver; remove the label to delete; requires Kubernetes 1.30+ ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2650). -* **feat(monitoring): add tenant overview dashboard**: A new **Tenant Overview** dashboard is available in the root/infra Grafana. It shows fleet-wide totals (tenants, apps, pods, CPU/memory/storage) plus a per-tenant leaderboard with resource utilization, app counts, and not-ready apps, and Top-N bar gauges by CPU, memory, storage, and app count — giving platform admins a single view of which tenants consume what ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2809). +* **[kafka] Add TLS support via Strimzi listener configuration**: See Feature Highlights — TLS on the external listener (9094), Strimzi-managed certs, external listener decoupled from TLS ([**@Arsolitt**](https://github.com/Arsolitt) in #2681). -* **feat(dashboard): add cluster-usage RBAC for the new admin page**: Adds `get/list/watch` on `namespaces` and the cluster-usage aggregation endpoint to the `cozy:infra:admin` ClusterRole so the new admin resource-usage page in the dashboard renders without 403 errors ([**@lexfrei**](https://github.com/lexfrei) in #2743). +* **[nats] Add TLS support via cert-manager**: See Feature Highlights — self-contained cert-manager chain for client connections and cluster routes ([**@Arsolitt**](https://github.com/Arsolitt) in #2684). -* **feat(flux): upgrade to Flux v2.8.0**: Upgrades Flux from v2.7.3 to v2.8.0 across the vendored tenant chart and the embedded management-cluster manifests. Flux v2.8's helm-controller v1.5.0 introduces **Server-Side Apply with `--force-conflicts`**, enabling strict CRD schema enforcement — fields that were silently dropped on v2.7 now fail the apply. A set of chart fixes are bundled to address fields that became hard errors under the new SSA regime ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2602). +* **[qdrant] Add TLS support via cert-manager**: See Feature Highlights — TLS for REST and gRPC endpoints via a single switch ([**@Arsolitt**](https://github.com/Arsolitt) in #2685). -* **refactor(seaweedfs): split `seaweedfs-system` into `seaweedfs-db` + `seaweedfs-system`**: The CNPG `Cluster/seaweedfs-db` resource is moved to its own HelmRelease (`seaweedfs-db`) that `seaweedfs-system` depends on. The new HR uses Flux v2 `HelmRelease.spec.healthCheckExprs` with a CEL expression on `Cluster.status.conditions[type=Ready]`, so `seaweedfs-system` reaches `Ready=True` only after the PostgreSQL primary is actually serving connections. This eliminates the `seaweedfs-filer` CrashLoopBackOff race during fresh tenant installs where Cilium's socket-LB returns `EPERM` on connections to ClusterIPs with no ready endpoints ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2601). +* **[postgres] Add TLS support via CNPG operator-managed certificates**: See Feature Highlights — injects the external hostname into the CNPG server certificate SANs so `sslmode=verify-full` works against the external endpoint ([**@Arsolitt**](https://github.com/Arsolitt) in #2686). -* **feat(ouroboros): bump to v0.8.0** *(backported to v1.4.3)*: Upgrades ouroboros to v0.8.0, which logs an explicit reason when its TCP backend readiness check fails — making stuck-proxy situations immediately diagnosable. Also migrates the kubectl sidecar image from Docker Hub to `mirror.gcr.io`, avoiding anonymous pull rate-limits on shared runner IPs ([**@lexfrei**](https://github.com/lexfrei) in #2807). +* **[kubernetes] Enable GPU passthrough out-of-the-box**: See Feature Highlights — `gpu=on` kubelet label on GPU node-groups and `NVreg_NvLinkDisable=1` driver flag fix single-SXM-GPU passthrough ([**@kvaps**](https://github.com/kvaps) in #2780). -* **chore(metallb): bump MetalLB to v0.16.1**: Upgrades the vendored MetalLB chart from v0.15.2 to v0.16.1 and rebuilds the `metallb-controller` and `metallb-speaker` images. Adopts the upstream-default FRR-K8s BGP backend (FRR mode is deprecated in v0.16). The cozystack-built images move in lockstep with the chart to preserve the new probe contract ([**@lexfrei**](https://github.com/lexfrei) in #2699). +* **[platform] Auto-wire KubeVirt permittedHostDevices and HostDevices feature gate**: See Feature Highlights — GPU VMs schedule without manual `kubectl patch`; the bundle now owns `permittedHostDevices` (see Upgrade Notes) ([**@lexfrei**](https://github.com/lexfrei) in #2768). -* **chore(seaweedfs): bump SeaweedFS to 4.31**: Upgrades the vendored SeaweedFS chart and images to version 4.31 ([**@lexfrei**](https://github.com/lexfrei) in #2834). +* **[gpu-operator] Add container variant for preinstalled host driver**: New `container` gpu-operator variant for hosts where the NVIDIA driver and container-toolkit are already OS-installed — exposes GPUs to containerized pods via the device plugin only ([**@lexfrei**](https://github.com/lexfrei) in #2766). -* **feat(hack): rewrite `check-readiness` as a Go command with test suite**: Replaces the `hack/check-readiness.sh` diagnostic with a Go command at `cmd/check-readiness`, integrated into the build the same way as `cozypkg`. The rewrite adds a test suite and makes the tool composable as a library for future automation ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2755). +* **[rbac] Allow tenants to start/stop/restart their VMs**: New RBAC rule grants `update` on the `virtualmachines/start`, `/stop`, and `/restart` KubeVirt subresources at the `cozy:tenant:use:base` level, so the dashboard's VM power buttons (which previously returned 403 for every tenant role) now work ([**@kvaps**](https://github.com/kvaps) in #2777). -* **chore(maintainers): mark @lllamnyp as Core Maintainer**: Aleksei Sviridkin ([**@lllamnyp**](https://github.com/lllamnyp)) is promoted to Core Maintainer, reflecting sustained contributions across API design, CSI, and testing infrastructure ([**@tym83**](https://github.com/tym83) in #2719). +* **[monitoring] Add tenant overview dashboard**: A new "Tenant Overview" Grafana dashboard for platform admins, deployed only to the root/infra Grafana in `cozy-monitoring` (never to per-tenant Grafanas, so no cross-tenant data leak). Gives a cross-tenant fleet summary, per-tenant leaderboard, top-N consumers, usage trends, and health signals ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2809). + +* **[dashboard] Add cluster-usage RBAC for the new admin page**: New `cozystack-dashboard-cluster-usage` ClusterRole (cluster-wide read on nodes/pods + `metrics.k8s.io` nodes) bound to the `cozystack-cluster-admin` group, backing the new Console → Administration → Cluster Usage page (cluster-wide and per-node utilization, including GPUs). The sidebar entry is fail-closed without the binding ([**@lexfrei**](https://github.com/lexfrei) in #2743). + +* **[apps] Mark stateful-app storageClass fields as immutable**: 16 stateful apps (clickhouse, foundationdb, harbor, http-cache, kafka, kubernetes, mariadb, mongodb, nats, openbao, opensearch, postgres, qdrant, rabbitmq, redis, vm-disk) declare `storageClass` immutable via an `x-kubernetes-validations` rule in their chart schema and the dashboard renders the field read-only on edit forms, because changing storageClass never migrates data (PVCs pin `storageClassName` at creation). **Enforcement is UI-only in this release** — the aggregated apiserver does not yet evaluate the CEL rule on `Update`, so a direct `kubectl patch` is still accepted; apiserver-level enforcement is tracked in #2657. `kubernetes.nodeGroups[].storageClass` is intentionally excluded ([**@lexfrei**](https://github.com/lexfrei) in #2639). + +* **[seaweedfs] Bump SeaweedFS to 4.31**: See Platform Components — chart 4.0.405 → 4.31.0, clears the 4.23 EC hazard ([**@lexfrei**](https://github.com/lexfrei) in #2834). + +* **[metallb] Bump to v0.16.1 and rotate host-network port denylist**: See Platform Components and Upgrade Notes — FRR-K8s default backend, HTTPS-only metrics, rebuilt controller/speaker images, rotated port denylist ([**@lexfrei**](https://github.com/lexfrei) in #2699). + +* **[ouroboros] Bump to v0.8.0**: See Platform Components ([**@lexfrei**](https://github.com/lexfrei) in #2807, *backported to v1.4.3*). + +* **[platform] Add kuberture as optional system package**: See Platform Components — publishes the Kubernetes API endpoint to DNS via external-dns ([**@lexfrei**](https://github.com/lexfrei) in #2647). ## Bug Fixes -* **fix(csi): detach orphan hot-plug volumes when VMI outlives its VM**: During node reprovisioning in tenant Kubernetes clusters, the VMI and its hot-plug pod can outlive the parent VirtualMachine. The upstream driver treated a missing VM as "detach succeeded", leaving the hot-plug pod alive and the infra storage device exclusively attached to the source host — every subsequent attach of that volume failed with `failed to set source device readwrite`. The wrapper now re-checks `VMI.status.volumeStatus` after upstream returns success and, if the hot-plug is still reported, detaches it from the VMI directly via the `removevolume` subresource ([**@kvaps**](https://github.com/kvaps) in #2866). +* **[api] Publish open spec as x-kubernetes-preserve-unknown-fields, not additionalProperties:true**: Fixes a cluster-wide kube-controller-manager CrashLoopBackOff (nil-pointer panic) that stalled reconciliation and timed out installs/upgrades. `cozystack-api` previously published the free-form `.spec` of `apps.cozystack.io` resources as `additionalProperties: true`, whose nil inner schema crashed the VAP type-checker run by KCM. The fix publishes `x-kubernetes-preserve-unknown-fields: true` instead and adds a recursive sanitizer that rewrites any boolean `additionalProperties` anywhere — including in untrusted `ApplicationDefinition.openAPISchema` input — closing the whole crash class ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2867). -* **fix(api): publish open spec as `x-kubernetes-preserve-unknown-fields`, not `additionalProperties:true`**: The aggregated `apps.cozystack.io` resources published a boolean `additionalProperties: true` for their free-form `.spec`, which caused `kube-controller-manager`'s ValidatingAdmissionPolicy type-checker to nil-dereference when it processed any VAP matching those resources. This triggered a `CrashLoopBackOff` on every control-plane node and stalled reconciliation. Switching to `x-kubernetes-preserve-unknown-fields: true` is semantically equivalent but handled safely by the type-checker ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2867). +* **[prometheus-operator-crds] Ship full upstream CRD bundle**: The package previously shipped only the four service-discovery CRDs VictoriaMetrics needs. This adds the six previously-stripped CRDs (Alertmanager, AlertmanagerConfig, Prometheus, PrometheusAgent, ScrapeConfig, ThanosRuler), so third-party apps in tenant clusters that ship their own prometheus-operator (e.g. kube-prometheus-stack) can create those CRs, or set `crds.enabled: false` and consume the platform-managed CRDs (which Flux previously kept reverting) ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2660). -* **fix(platform): order `cozystack-basics` after the APIs its admission policies type-check**: `cozystack-basics` ships cluster-wide ValidatingAdmissionPolicies that match `tenants.apps.cozystack.io` and Gateway API types. On a cold install, if these policies were created before `cozystack-api` was Ready, `kube-controller-manager` would nil-dereference while type-checking the policy against the not-yet-served `tenants` schema, enter `CrashLoopBackOff`, stall cert and token issuance, and prevent `cozystack-api` from ever starting — a deadlock. `cozystack-basics` now `dependsOn` both `cozystack-api` and the Gateway API CRD installer ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2842). +* **[info] Use root-host for Keycloak OIDC issuer URL in tenant kubeconfig**: The dashboard-issued kubeconfig for non-root tenants built `--oidc-issuer-url` from the tenant subdomain, but Keycloak's ingress and TLS cert live only at the root host, so `kubectl oidc-login` failed TLS verification and the kubeconfig was unusable. The issuer now always uses the root host (`https://keycloak./realms/cozy`), so non-root tenants can authenticate with the downloaded kubeconfig ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2704). -* **fix(seaweedfs): restore S3 TLS and COSI provisioner SA after SeaweedFS 4.31 upgrade**: The 4.31 chart bump nested the `-cert.file`/`-key.file` TLS args inside `{{- if .Values.s3.httpsPort }}`, but Cozystack ships `httpsPort: 0` — so no TLS args rendered, S3 spoke plaintext HTTP while all probes and clients expected HTTPS. Additionally, the COSI provisioner Deployment referenced a ServiceAccount by a different name than what the chart creates, causing pod startup failures. Both regressions are fixed via vendored patches ([**@kvaps**](https://github.com/kvaps) in #2916). +* **[kubernetes] Add config_path patch for containerd 2.x**: Per-registry mirror/credential config (`/etc/containerd/certs.d`) did not take effect on containerd 2.x tenant nodes because the CRI plugin config section was renamed and the old single sed was a no-op. The fix runs version-tolerant seds and loosens quote matching (also fixing Ubuntu 24.04, which emits single-quoted section headers), so `config_path` registry config works on both containerd 1.x and 2.x without manual patching ([**@elaugaste**](https://github.com/elaugaste) in #2723). -* **fix(seaweedfs): restore `-lock` BucketClass, S3 service name, and drop `volumeSizeLimitMB`**: Three additional regressions from the SeaweedFS 4.31 bump: the `-lock` BucketClass and `-readonly` BucketAccessClass were dropped (breaking access control), the S3 service DNS name regressed to a non-prefixed form (breaking in-cluster clients), and `volumeSizeLimitMB` prevented multiple collections from sharing a volume ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2943). +* **[csi] Detach orphan hot-plug volumes when VMI outlives its VM**: When a VMI outlives its owning VM, hot-plug volumes were left attached, blocking the volume from being re-attached elsewhere; the CSI wrapper now detaches the orphaned hot-plug volumes so the PVC can be reused ([**@kvaps**](https://github.com/kvaps) in #2866). -* **fix(backups): carry dropdown option sources in CRD annotations, not schema**: The four `backups.cozystack.io` CRDs embedded `x-cozystack-options` vendor extensions directly in their OpenAPI validation schema. Kubernetes only preserves `x-kubernetes-*` vendor extensions; every other `x-` key was rejected by helm-controller's SSA apply, causing the backup CRDs to never apply and the controller to crash-loop. Option sources are now stored in CRD object annotations, which are preserved unchanged ([**@lexfrei**](https://github.com/lexfrei) in #2823). +* **[csi] Route RWX Block volumes to upstream hotplug detach**: KubeVirt live-migration disks (RWX Block PVCs) were incorrectly matched by the NFS-cleanup branch in `ControllerUnpublishVolume` and never detached, so a later attach to a different VM was rejected by linstor-csi's anti-split-brain check. A new `isNFSVolume` predicate requires both `ReadWriteMany` and `VolumeMode=Filesystem`, accurately matching only NFS-backed PVCs, and the same predicate fixes in-VM disk expansion being skipped for RWX Block volumes *(backported to v1.4.2 via #2749)* ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2658). -* **fix(ingress, platform): deliver `publishing.proxyProtocol` to host ingress-nginx**: The three nginx config keys (`use-proxy-protocol`, `real-ip-header`, `enable-real-ip`) were written into a component values block that had no `install:` section, so the package controller never built a HelmRelease for it and the values were silently dropped. The switch now routes through `_cluster.proxy-protocol` in the `cozystack-values` Secret — the same path `expose-external-ips` already uses to reach the host ingress release ([**@lexfrei**](https://github.com/lexfrei) in #2799). +* **[csi] Verify VMI Ready after kubevirt-csi Publish to surface stuck-PVC failures**: Upstream `ControllerPublishVolume` has a fast path that reports success as soon as the volume entry appears in `VM.spec.template.spec.volumes`, even if the backing PVC is stuck `ClaimPending` and the hotplug never completed — surfacing later as a confusing kubelet `couldn't find device by serial id`. The wrapper now re-reads `VMI.Status.VolumeStatus` and returns `codes.Unavailable` with the upstream reason if the volume is not `VolumeReady`, keeping external-attacher retrying and surfacing the real provisioning failure at the CSI layer *(backported to v1.4.2 via #2748)* ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2659). -* **fix(platform): add cert-manager dependency to webhook-cert consumers**: Five PackageSources (`capi-operator`, `cozystack-engine`, `ingress-nginx`, `kamaji`, `rabbitmq-operator`) now declare `dependsOn: cert-manager`. Without this, on cold installs the chart applied `cert-manager.io/v1 Certificate` CRs before the cert-manager webhook was ready, burning 2–3 minutes per affected package in retry loops ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2726). +* **[capi] Add startupProbe to capi-controller-manager to survive cert provisioning delay**: The CAPI controller-manager could CrashLoop during initial certificate provisioning because the readiness probe failed while the controller was still booting; a startup probe now gives it room to come up *(backported to v1.4.x stabilization)* ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2946). -* **fix(info): use root-host for Keycloak OIDC issuer URL in tenant kubeconfig**: Non-root tenants had `--oidc-issuer-url` pointing at `keycloak..`, but Keycloak lives at `keycloak.`. `kubectl oidc-login` failed TLS verification against the nginx default cert, making the auto-generated kubeconfig unusable for any non-root tenant ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2704). +* **[cluster-api] Fix kamaji OOM and set limits on unset providers**: The Kamaji control-plane-provider resource override targeted a container named `manager`, but the upstream image names it `controller`, so the intended limits were dropped and the pod OOMKilled (exit 137) on the upstream 128 Mi default. The container name is corrected and modest requests/limits are set for the core, kubeadm-bootstrap, and kubevirt-infrastructure providers that previously ran as BestEffort *(backported to v1.4.1 via #2709)* ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2708). -* **fix(kubernetes): add `config_path` patch for containerd 2.x**: Adds support for containerd 2.x where the CRI plugin section was renamed from `io.containerd.grpc.v1.cri` to `io.containerd.cri.v1.images`. The patch regexes are also made resilient to single and double quotes around plugin names, fixing the issue on Ubuntu 24.04 where containerd 2.x uses single quotes ([**@elaugaste**](https://github.com/elaugaste) in #2723). +* **[platform] Migrate ephemeralStorage to diskSize via pre-upgrade hook**: The #2454 rename of `nodeGroups[*].ephemeralStorage` to `diskSize` added a hard `{{ fail }}` guard that blocked reconciliation of any HelmRelease still carrying the legacy field. A new platform pre-upgrade migration (migration 41) walks all `kuberneteses.apps.cozystack.io` Application CRs cluster-wide and renames the field automatically before chart resources are applied; the migration is idempotent and best-effort *(backported to v1.4.1 via #2712)* ([**@IvanHunters**](https://github.com/IvanHunters) in #2688). -* **fix: add `startupProbe` to `kubevirt-cdi-operator` and `grafana-operator`**: Both operators were missing a `startupProbe`, causing them to be killed by the `livenessProbe` during slow startup under load (slow leader election for `kubevirt-cdi-operator`, startup latency under high cluster activity for `grafana-operator`) ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2725). +* **[postgres] Accept integer values for postgresql.parameters in schema**: PostgreSQL parameters such as `max_connections` are natural integers, but the chart schema declared `{type: string}` only, so bare integers were rejected at schema-validation time. An `intOrString` alias in cozyvalues-gen now emits `anyOf: [integer, string]` plus `x-kubernetes-int-or-string: true`, making both forms valid; the `$dangerousParams` blocklist is also extended with `archive_cleanup_command` and `recovery_end_command` *(backported to v1.4.1 via #2715)* ([**@IvanHunters**](https://github.com/IvanHunters) in #2687). -* **fix(kubernetes): add `spec.timeout` to tenant CSI HelmRelease**: The `kubevirt-csi-node` HelmRelease template was the only tenant addon HR missing `spec.timeout: 10m`, causing it to use Flux's default (5 minutes), which was insufficient on loaded clusters. All other tenant addon HelmReleases (cilium, coredns, ingress-nginx, etc.) already set 10m ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2727). +* **[kafka] Remove ZooKeeper PVCs on uninstall**: The Strimzi `Kafka` CR set `deleteClaim: true` on broker JBOD volumes but left ZooKeeper persistent-claim storage at `deleteClaim: false`, orphaning the ZooKeeper PVCs on uninstall and requiring manual cleanup before the release name could be reused. `deleteClaim: true` is now set on the ZooKeeper storage as well *(backported to v1.4.1 via #2705)* ([**@Arsolitt**](https://github.com/Arsolitt) in #2679). -* **fix(dashboard): unblock token-proxy startup when JWKS is briefly unreachable**: The dashboard's token-proxy process failed to start if the Keycloak JWKS endpoint was briefly unreachable during pod startup. This caused the dashboard to stay in a crash loop until Keycloak became available, rather than retrying gracefully ([**@lexfrei**](https://github.com/lexfrei) in #2745). +* **[opensearch-operator] Replace deprecated kube-rbac-proxy image**: The `gcr.io/kubebuilder/kube-rbac-proxy` image is no longer available since the kubebuilder GCR registry was sunset. It is replaced with `quay.io/brancz/kube-rbac-proxy` (the source already used by other Cozystack components), applied via a `values.yaml` entry instead of a vendored-chart patch *(backported to v1.4.1 via #2695)* ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2689). -* **fix(prometheus-operator-crds): ship full upstream CRD bundle**: A `rm -f` line in the Makefile stripped 6 of the 10 upstream CRDs (`Alertmanager`, `AlertmanagerConfig`, `Prometheus`, `PrometheusAgent`, `ScrapeConfig`, `ThanosRuler`). Operators depending on any of these CRDs were silently missing them from their clusters ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2660). +* **[platform] Add OpenSearch to PaaS bundle**: The OpenSearch packages existed in the repo for several releases but the PaaS bundle template never referenced the two `cozystack.opensearch-*` PackageSources, so the operator was never deployed and the dashboard catalog showed no OpenSearch entry. The two missing includes are now added *(backported to v1.4.2 via #2757)* ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2648). -* **fix(capi): add `startupProbe` to `capi-controller-manager`**: The CAPI controller manager lacked a startupProbe, causing it to be killed by the liveness probe during slow startup (e.g. when cert provisioning was delayed). This manifested as repeated restarts in clusters with high cert-manager load ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2946). +* **[dashboard] Grant tenant dashboard read on cozy-public PVCs**: The VM disk source-image dropdown returned 403 and stayed empty even when golden images existed, because the `cozy:tenant:dashboard` Role only granted read on Flux HelmRepositories and HelmCharts. `get/list/watch` on PersistentVolumeClaims is added so tenant identities can list the `vm-default-images-*` PVCs *(backported to v1.4.3 via #2858)* ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2843). -* **fix(kafka): reclaim ZooKeeper PVCs on Kafka release deletion** *(backported to v1.4.1)*: Deleting a Kafka application freed broker PVCs but orphaned ZooKeeper PVCs, requiring manual cleanup before the same release name could be reused. `deleteClaim: true` is now set on ZooKeeper storage ([**@Arsolitt**](https://github.com/Arsolitt) in #2679). +* **[api] Emit initial-events-end bookmark for core.cozystack.io watches**: The `TenantSecret`, `TenantModule`, and `TenantNamespace` aggregated API resources never sent the `k8s.io/initial-events-end` bookmark required by the WatchList protocol, so client-go informers using `WatchListClient` (on by default since v1.35) never reached `HasSynced`. The bookmark is now emitted after initial `ADDED` events *(backported to v1.4.3 via #2844)* ([**@sunib**](https://github.com/sunib) in #2786). -* **fix(postgres): accept integer values for `postgresql.parameters` in schema** *(backported to v1.4.1)*: `postgresql.parameters` declared `{type: string}` only, rejecting bare integers like `max_connections: 100`. An `intOrString` type alias is now emitted with `anyOf: [integer, string]` plus `x-kubernetes-int-or-string: true`. The same PR extends the `$dangerousParams` blocklist with `archive_cleanup_command` and `recovery_end_command` ([**@IvanHunters**](https://github.com/IvanHunters) in #2687). +* **[networking] Point host ouroboros proxy at the root-tenant ingress**: When `publishing.proxyProtocol` was enabled, the host-level ouroboros proxy inherited a default backend FQDN that describes a managed tenant cluster and never resolved on the host. The host ouroboros `Package` is now emitted with a `proxy.target` override derived from `publishing.ingressName` *(backported to v1.4.3 via #2846)* ([**@lexfrei**](https://github.com/lexfrei) in #2800). -* **fix(platform): migrate `ephemeralStorage` to `diskSize` via pre-upgrade hook** *(backported to v1.4.1)*: PR #2454 renamed `nodeGroups[*].ephemeralStorage` to `nodeGroups[*].diskSize` with a hard `{{ fail }}` guard. A new platform pre-upgrade migration (migration 41) now walks all `kuberneteses.apps.cozystack.io` Application CRs and renames the field automatically before chart resources are applied ([**@IvanHunters**](https://github.com/IvanHunters) in #2688). +* **[ingress,platform] Deliver publishing.proxyProtocol to host ingress-nginx**: Threads the `publishing.proxyProtocol` setting through to the host ingress-nginx controller so PROXY-protocol mode is actually applied on the host path ([**@lexfrei**](https://github.com/lexfrei) in #2799). -* **fix(cluster-api): fix Kamaji OOM and set limits on unset providers** *(backported to v1.4.1)*: The Kamaji control-plane-provider resource override targeted a container named `manager`, but the image names it `controller` — the intended 1 CPU / 1024 Mi limits were silently dropped and Kamaji used 128 Mi, causing repeated OOMKilled restarts. The container name is corrected and modest resource requests and limits are added for the core, kubeadm-bootstrap, and kubevirt-infrastructure providers that previously ran as BestEffort ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2708). +* **[objectstorage-controller] Propagate Bucket readiness to BucketClaim**: The vendored COSI controller hardcoded `bucketReady=false` after dynamic provisioning and never re-read the `Bucket`, so `BucketAccess` was never granted and provisioned buckets ended up without credentials. The controller now re-reads the live `Bucket` and propagates its readiness *(backported to v1.4.3 via #2828)* ([**@lexfrei**](https://github.com/lexfrei) in #2792). -* **fix(opensearch-operator): replace deprecated `kube-rbac-proxy` image** *(backported to v1.4.1)*: The `gcr.io/kubebuilder/kube-rbac-proxy` image is no longer available. Replaced with `quay.io/brancz/kube-rbac-proxy` following the convention already used by etcd-operator and other components ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2689). +* **[kubernetes] Stamp application lineage labels on worker node VMs**: Worker-node VMs created by Cluster API and the KubeVirt provider were never stamped with `apps.cozystack.io/application.{group,kind,name}` lineage labels, so the dashboard could not attribute those pods to their owning Kubernetes application. The labels are now applied to the `KubevirtMachineTemplate`, and `application.name` is quoted so a purely-numeric cluster name renders as a YAML string *(backported to v1.4.3 via #2790)* ([**@kvaps**](https://github.com/kvaps) in #2779). -* **fix(platform): add OpenSearch to PaaS bundle** *(backported to v1.4.2)*: The `opensearch-operator` and `opensearch-rd` PackageSources were missing from the PaaS bundle template, so the OpenSearch operator and its ApplicationDefinition were never deployed — tenants could not provision OpenSearch instances ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2648). +* **[kubernetes] Add spec.timeout to tenant CSI HelmRelease**: Adds an explicit `spec.timeout` to the tenant CSI HelmRelease so slow first installs are not classified as failures under the new Flux kstatus waits ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2727). -* **fix(csi): route RWX Block volumes to upstream hotplug detach** *(backported to v1.4.2)*: `ControllerUnpublishVolume` selected the NFS cleanup branch using only `hasRWXAccessMode(pvc)`, so RWX Block PVCs (KubeVirt live-migration disks) were sent down the NFS path and left their `DataVolume` entry in `VM.spec.template.spec.volumes` permanently, blocking future attaches. The fix introduces `isNFSVolume(pvc)` requiring both `ReadWriteMany` and `VolumeMode=Filesystem` ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2658). +* **[platform] Add cert-manager dependency to webhook-cert consumers**: Adds an explicit `dependsOn` cert-manager to the HelmReleases that consume webhook certificates, so they no longer race cert-manager during cold install ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2726). -* **fix(csi): verify VMI Ready after `kubevirt-csi` Publish to surface stuck-PVC failures** *(backported to v1.4.2)*: Upstream `ControllerPublishVolume` reported success as soon as the volume entry appeared in `VM.spec.template.spec.volumes`, even if QEMU never received `device_add`. The wrapper now re-reads `VMI.Status.VolumeStatus` and returns `codes.Unavailable` if the volume is not in `VolumeReady` phase, keeping `external-attacher` retrying and surfacing the underlying failure at the CSI layer ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2659). +* **[platform] Order cozystack-basics after the APIs its admission policies type-check**: Reorders `cozystack-basics` so it installs after the APIs its ValidatingAdmissionPolicies reference, avoiding type-check failures when the referenced APIs are not yet registered ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2842). -* **fix(dashboard): grant tenant dashboard read on `cozy-public` PVCs** *(backported to v1.4.3)*: The VM disk source-image dropdown returned 403 and stayed empty even when golden images existed in the `cozy-public` namespace. `get/list/watch` on PersistentVolumeClaims is now added to the `cozy:tenant:dashboard` Role ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2843). +* **[dashboard] Unblock token-proxy startup when JWKS is briefly unreachable**: The dashboard token-proxy refused to start if the JWKS endpoint was momentarily unreachable; startup is now resilient to a brief JWKS outage so the dashboard comes up and recovers once JWKS is reachable ([**@lexfrei**](https://github.com/lexfrei) in #2745). -* **fix(api): emit `initial-events-end` bookmark for `core.cozystack.io` watches** *(backported to v1.4.3)*: `TenantSecret`, `TenantModule`, and `TenantNamespace` aggregated API resources never sent the `k8s.io/initial-events-end` bookmark required by the WatchList protocol. Client-go informers using `WatchListClient` never reached `HasSynced` and logged warnings every ~10 seconds ([**@sunib**](https://github.com/sunib) in #2786). +* **[kubevirt-cdi-operator,grafana-operator] Add startupProbes**: Adds startup probes to the kubevirt-cdi-operator and grafana-operator so slow boots are not killed by the readiness probe before initialization finishes ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2725). -* **fix(networking): point host ouroboros proxy at the root-tenant ingress** *(backported to v1.4.3)*: When `publishing.proxyProtocol` was enabled, the host-level ouroboros proxy inherited the wrong backend FQDN (a managed Kubernetes tenant cluster address). On the host, ingress-nginx is deployed as `root-ingress-controller` in `tenant-root`, so the proxy never became `Ready` ([**@lexfrei**](https://github.com/lexfrei) in #2800). +* **[seaweedfs] Restore S3 TLS and COSI provisioner SA after 4.31 bump**: Restores the S3 TLS configuration and the COSI provisioner ServiceAccount that regressed during the SeaweedFS 4.31 bump ([**@kvaps**](https://github.com/kvaps) in #2916). -* **fix(objectstorage-controller): propagate Bucket readiness to BucketClaim** *(backported to v1.4.3)*: The vendored COSI controller hardcoded `bucketReady=false` after dynamic provisioning and never re-read the `Bucket` to pick up the `true` transition, so `BucketAccess` was never granted and buckets ended up without credentials ([**@lexfrei**](https://github.com/lexfrei) in #2792). +* **[seaweedfs] Restore -lock BucketClass, s3 service name, and drop volumeSizeLimitMB**: Restores the `-lock` BucketClass and the s3 service name and drops the now-invalid `volumeSizeLimitMB` field after the 4.31 bump ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2943). -* **fix(kubernetes): stamp application lineage labels on worker node VMs** *(backported to v1.4.3)*: Worker-node VMs created by Cluster API were never stamped with `apps.cozystack.io/application.{group,kind,name}` lineage labels, preventing the dashboard from attributing their virt-launcher pods to the owning Kubernetes application. A companion fix quotes `application.name` so a purely-numeric cluster name renders as a YAML string ([**@kvaps**](https://github.com/kvaps) in #2779). +* **[seaweedfs] Bump seaweedfs-cosi-driver to v0.3.1**: See Platform Components — stale-socket self-heal *(backported to v1.4.3 via #2827)* ([**@lexfrei**](https://github.com/lexfrei) in #2791). + +* **[backups] Carry dropdown option sources in CRD annotations, not schema**: Moves the backup dropdown option sources from the chart schema into CRD annotations so they are consumed by the new `x-cozystack-options` mechanism without polluting the schema ([**@lexfrei**](https://github.com/lexfrei) in #2823). + +* **[backups] Fix CI broken after merge**: Repairs the backups test suite that broke after a merge conflict between concurrent backup-strategy PRs ([**@androndo**](https://github.com/androndo) in #2762). ## Security -* **ci: add CodeQL workflow for Go (SAST)**: Adds a weekly CodeQL static analysis scan for the Go codebase ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2851). +* **[deps] Bump Go to 1.26.4 and x/net to v0.55.0 to close OSV advisories**: Bumps the Go toolchain to 1.26.4 and `golang.org/x/net` to v0.55.0 to clear reported OSV/Dependabot advisories across the first-party modules ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2852). -* **ci: add OpenSSF Scorecard workflow**: Enables continuous OpenSSF Scorecard scoring for the repository, covering vulnerability checks, branch protection, CI best practices, and contributor risk ([**@tym83**](https://github.com/tym83) in #2720). +* **[ci] Add CodeQL workflow for Go (SAST)**: Adds a CodeQL static-analysis workflow for the Go codebase so security issues are flagged on PRs ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2851). -* **ci: pin Scorecard workflow actions by SHA**: Pins all third-party Actions used by the Scorecard workflow to digest-pinned SHAs to prevent supply-chain attacks via tag mutation ([**@tym83**](https://github.com/tym83) in #2721). +* **[ci] Add OpenSSF Scorecard workflow**: Adds the OpenSSF Scorecard workflow to track and publish the project's supply-chain security posture; a follow-up pins the Scorecard workflow actions by SHA ([**@tym83**](https://github.com/tym83) in #2720, #2721). -* **fix(deps): bump Go to 1.26.4 and `x/net` to v0.55.0 to close OSV advisories**: Clears 12 Open Source Vulnerability advisories: 10 against the Go toolchain (fixed in 1.26.3 and 1.26.4) and 2 against `golang.org/x/net` (fixed in v0.55.0). This brings the OpenSSF Scorecard Vulnerabilities check to a passing state ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2852). +* **[platform] Deletion-protection guardrail**: See Feature Highlights — blocks accidental DELETE of critical platform objects in-apiserver ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2650). -## Dependencies +## System Configuration -* **chore(seaweedfs): bump `seaweedfs-cosi-driver` to v0.3.1** *(backported to v1.4.3)*: v0.3.1 ships a stale-socket self-heal: the COSI driver now removes any leftover UNIX socket before binding, so the objectstorage provisioner recovers automatically from a non-graceful exit instead of wedging on "bind: address already in use" ([**@lexfrei**](https://github.com/lexfrei) in #2791). +* **[platform] Velero installed by default**: See Feature Highlights — Velero moves to a default system package; opt out via `bundles.disabledPackages` ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2833). + +* **[platform] Register kuberture as optional system package**: See Platform Components — off by default, enabled via `bundles.enabledPackages` with a required `config.outputs` entry ([**@lexfrei**](https://github.com/lexfrei) in #2647). + +* **[seaweedfs] Split seaweedfs-system into seaweedfs-db + seaweedfs-system**: Extracts the SeaweedFS CNPG postgres Cluster into its own `seaweedfs-db` HelmRelease that reports Ready only when postgres actually serves connections, fixing a fresh-install race where the filer CrashLooped against an endpoint-less ClusterIP under Cilium. An automatic, data-safe migration adopts the existing postgres cluster in place (re-annotates Helm ownership and stamps `helm.sh/resource-policy: keep`); no data is moved and no PV/PVC is re-provisioned ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2601). + +## Dependencies & Version Updates + +See the **Platform Components** section near the top of this changelog for the full list of upstream bumps with user-facing impact summaries (Flux v2.8.0, MetalLB v0.16.1, SeaweedFS 4.31, etcd-operator v0.4.5, ouroboros v0.8.0, seaweedfs-cosi-driver v0.3.1, kuberture v0.1.1). + +Additional dependency-related changes: + +* **[deps] Bump Go to 1.26.4 and x/net to v0.55.0** — see Security ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2852). +* **[ci] Bump cozyvalues-gen pin to v1.5.0, then v1.6.0** — schema-generation tooling refresh ([**@lexfrei**](https://github.com/lexfrei) in #2730, #2784). ## Development, Testing, and CI/CD -* **feat(ci): test-impact analysis for E2E**: A new pre-E2E step analyses which packages changed in the PR diff and selects only the relevant bats test files. E2E runs that previously exercised the full test matrix now run only the impacted tests on normal PRs, with full-suite runs reserved for release PRs and `[full-e2e]` labelled PRs ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2559). +* **[ci/build] Isolate each PR build on its own ephemeral runner VM**: Moves PR builds onto per-job ephemeral runner VMs to fix the cross-job buildkit single-writer-lock contention that caused intermittent build hangs on the shared runner host ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2939). -* **ci(workflows): add SSH breakpoint on E2E failure for debug PRs**: E2E jobs on PRs labelled `debug` or `release` now drop into an interactive SSH session on failure, so engineers can inspect the cluster state without needing to reproduce the failure locally ([**@kvaps**](https://github.com/kvaps) in #2535). +* **[refactor/build] mode=max registry cache with a main-only warmer**: Switches the build cache to `mode=max` with a main-branch-only cache warmer so PR builds warm-start without each PR re-pushing cache layers ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2938). -* **ci(release): auto-patch-release only the 2 newest release lines**: The automated patch-release cron now restricts backport CI to the two most recent `release-X.Y` branches, preventing unnecessary runs on end-of-life lines ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2856). +* **[build] Read --cache-from from ghcr :latest so PR builds warm-start**: Points `--cache-from` at the ghcr `:latest` tag so PR builds reuse the last published cache ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2855). -* **ci(release): repair orphaned draft release on retag** *(backported to v1.4.3)*: When a git tag was deleted and re-created between draft creation and merge, GitHub orphaned the draft release by setting `tag_name` to `"untagged-"`. The finalize step now detects the orphaned form, falls back to matching by `name`, repairs `tag_name` via `updateRelease`, and then publishes ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2761). +* **[refactor/build] Standardize image tagging to fix concurrent PR push conflicts**: Standardizes the build image-tagging scheme so concurrent PR builds no longer collide when pushing to the registry ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2711). -* **refactor(build): standardize image tagging to fix concurrent PR push conflicts**: Aligns image tagging across all build targets to prevent tag collision when multiple PRs push to the same registry prefix concurrently ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2711). +* **[ci] Create buildx builder in the build step's DOCKER_CONFIG**: Fixes buildx builder creation by scoping it to the build step's `DOCKER_CONFIG` ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2962). -* **refactor(build): mode=max registry cache with a main-only warmer**: Switches Docker BuildKit cache to `mode=max` and introduces a dedicated cache-warmer job that runs only on `main` pushes. PR builds warm-start from the registry cache without the warmer running on every PR ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2938). +* **[feat/ci] Test-impact analysis E2E (default-on) + release E2E workflow**: Adds test-impact analysis that selects which E2E suites to run based on the PR diff (default-on), plus a dedicated release E2E workflow ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2559). -* **ci(build): isolate each PR build on its own ephemeral runner VM**: PR builds are now dispatched to fresh ephemeral runner VMs, preventing state bleed between concurrent builds and improving security isolation ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2939). +* **[ci/tests] Fix OCIR registry login in Release E2E workflow**: Corrects the OCIR registry login step in the Release E2E workflow so the release E2E run authenticates correctly ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2973). -* **test(api): add read-path authorization tests for `TenantNamespace` registry**: Adds direct unit coverage for the authorization-sensitive read paths (`Get`, `List`, `Watch`) of the TenantNamespace registry, including per-user filtering, privileged-group bypass, bookmark forwarding, field and label selector filtering, and goroutine lifecycle. A production gap (Watch goroutine never closing its result channel) is fixed as part of this work ([**@lexfrei**](https://github.com/lexfrei) in #2864). +* **[ci/release] Auto-patch-release only the 2 newest release lines**: The automated patch-release cron now targets only the two newest release lines, matching the supported-version window, and documents the support window ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2856). -* **ci: add Go-scoped Renovate configuration**: Enables automated dependency updates for Go modules in the repository via Renovate ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2850). +* **[ci/release] Repair orphaned draft tag_name on retag**: When a tag was deleted and re-created between draft creation and merge, GitHub orphaned the draft release by setting `tag_name` to `untagged-`. The finalize step now detects the orphaned form, matches by `name`, repairs `tag_name`, and publishes *(backported to v1.4.3 via #2829)* ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2761). + +* **[ci] Add Go-scoped Renovate configuration**: Adds a Renovate configuration scoped to the Go modules so Go dependency updates are managed automatically ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2850). + +* **[ci/tags] Include data/versions in website docs commit**: The tags workflow now includes `data/versions` in the website docs commit so version metadata stays in sync ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2707). + +* **[ci] Consolidated platform & e2e stabilization batch**: A consolidated batch of platform and E2E stabilization changes folded into a single PR ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2948). + +* **[ci/e2e] Drop 3x retry on Run E2E + Install Cozystack, wait out gateway.bats tenant teardown**: Removes the blanket 3x retry on deterministic E2E steps and instead waits out the gateway.bats tenant teardown explicitly ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2558). + +* **[ci/workflows] Add SSH breakpoint on E2E failure for debug PRs**: Adds an opt-in SSH breakpoint on E2E failure for debug PRs so failures can be inspected live ([**@kvaps**](https://github.com/kvaps) in #2535). + +* **[test/e2e] Self-heal Cilium orphaned-endpoint leak across install and apps**: Adds an in-cluster self-heal Job that recovers from the Cilium agent's orphaned-endpoint leak (the "IP already in use" flake) during install and app churn ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2874). + +* **[test/e2e] Pin Kubernetes to v1.33.12 to avoid KCM VAP type-checker panic**: Pins the E2E Kubernetes version to v1.33.12 to avoid the kube-controller-manager VAP type-checker panic on affected Kubernetes builds (the durable apiserver-side fix ships in #2867) ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2868). + +* **[test/e2e] Drive LINSTOR post-install waits off a single 15m deadline**: Replaces per-step LINSTOR waits with a single 15-minute deadline so the slowest path determines runtime ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2928). + +* **[test/e2e] Pre-pull platform images via staged-busybox DaemonSet**: Pre-pulls timing-sensitive platform images via a staged-busybox DaemonSet before install so first-install reconciliation does not race image pulls ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2724). + +* **[test/e2e] Fix bucket.bats port-forward and S3 client reliability**: Hardens the bucket.bats test's port-forward and S3 client interaction for reliability ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2944). + +* **[test/e2e] ouroboros fold + CSI/NFS ordering + OIDC keycloakInternalUrl**: Folds the ouroboros E2E case, fixes CSI/NFS ordering, and corrects the OIDC `keycloakInternalUrl` in the suite ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2728). + +* **[test/e2e] Silence helm-template render in install-cozystack trace**: Silences the noisy helm-template render output in the install trace so failures are easier to read ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2615). + +* **[tests] Exclude loop devices from host LVM scanning**: Excludes loop devices from host LVM scanning in the test harness so loop-mounted images do not get their volume groups activated ([**@kvaps**](https://github.com/kvaps) in #2798). + +* **[e2e-sandbox] Use curl -fsSL when downloading mc**: The MinIO client was downloaded without `-L`, so a 302 redirect from `dl.min.io` wrote the HTML redirect body to `/usr/local/bin/mc` and the bucket E2E test then failed trying to execute it. Adding `-fsSL` follows the redirect and fails fast on HTTP errors *(shipped in v1.4.1)* ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2690). + +* **[test/metallb] Assert digest-pinned image form, not exact version literal**: Updates the metallb test to assert the digest-pinned image form rather than an exact version literal, so the test survives version bumps ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2873). + +* **[test/api] Add read-path authorization tests for TenantNamespace registry**: Adds read-path authorization tests for the TenantNamespace registry to lock down the IDOR fixes ([**@lexfrei**](https://github.com/lexfrei) in #2864). + +* **[feat/hack] Rewrite check-readiness as a Go command with test suite**: Rewrites the `check-readiness` helper as a Go command with a test suite, replacing the previous shell implementation ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2755). + +* **[chore/controller] Remove dead dashboard controller**: Removes the now-dead dashboard controller code left behind after the schema-driven UI rewrite ([**@kvaps**](https://github.com/kvaps) in #2694). + +* **[chore/img] Refresh Cozystack logo and add icon/stacked variants**: Refreshes the Cozystack logo and adds icon and stacked logo variants ([**@kvapsova**](https://github.com/kvapsova) in #2746). + +* **[chore/maintainers] Update roster, add Emeritus section, mark @lllamnyp as Core Maintainer**: Updates the maintainer roster, adds an Emeritus section, documents Emeritus technical offboarding and reactivation, and marks [**@lllamnyp**](https://github.com/lllamnyp) as a Core Maintainer ([**@tym83**](https://github.com/tym83) in #2717, #2718, #2719). + +* **[docs/agents] Add E2E/CI testing conventions and consolidate build/test/verify guidance**: Adds the E2E and CI testing conventions doc and consolidates the build, test, and verification guidance for agents and contributors ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2932, #2933). + +* **[docs/release] Expand release.md into a release-engineer playbook**: Expands `release.md` into a full release-engineer playbook covering the release process end to end ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2732). ## Documentation -* **chore(maintainers): update roster and add Emeritus section**: Updates the maintainer roster and adds an Emeritus section documenting the technical offboarding and reactivation process for inactive maintainers ([**@tym83**](https://github.com/tym83) in #2717, #2718). +* **[harbor] Document S3 object-storage (SeaweedFS) prerequisite**: Documents that Harbor requires an S3 object-storage backend (SeaweedFS) and how to satisfy the prerequisite ([**@lexfrei**](https://github.com/lexfrei) in #2818). -* **docs(harbor): document S3 object-storage (SeaweedFS) prerequisite**: Adds a Prerequisites section to the Harbor app README explaining that S3-compatible object storage is required and how to satisfy it (`seaweedfs: true` on the tenant), preventing a common silent first-deploy failure ([**@lexfrei**](https://github.com/lexfrei) in #2818). +* **[gpu-operator] Document passthrough variant host-driver prerequisite**: Documents the host-driver prerequisite for the gpu-operator passthrough variant ([**@lexfrei**](https://github.com/lexfrei) in #2767). -* **docs(gpu-operator): document passthrough variant host-driver prerequisite**: Adds documentation explaining that the `passthrough` variant of `cozystack.gpu-operator` requires that no host-installed NVIDIA driver is active and clarifies when the new `container` variant should be used instead ([**@lexfrei**](https://github.com/lexfrei) in #2767). +* **[website] KubeVirt VM Disk and VM Instance how-to**: New how-to explaining the VMDisk and VMInstance primitives — independent disk/compute lifecycles, golden images, cloning, fast provisioning, with Dashboard and kubectl examples and VM access methods (serial console, SSH, VNC) ([**@tym83**](https://github.com/tym83) in cozystack/website#586). -* **chore(img): refresh Cozystack logo and add icon/stacked variants**: Updates the Cozystack brand assets — new horizontal wordmark, icon-only, and stacked variants — used across the dashboard and promotional materials ([**@kvapsova**](https://github.com/kvapsova) in #2746). +* **[website] etcd-operator v1alpha2 announcement and migration guide**: Announces the donation of etcd-operator to Cozystack and the new `etcd-operator.cozystack.io/v1alpha2` API — a Membership-API lifecycle replacing the StatefulSet model — with the in-place v1alpha1 → v1alpha2 migration path ([**@kvaps**](https://github.com/kvaps) in cozystack/website#573). -* **[website] feat(blog): add etcd-operator v1alpha2 announcement**: Blog post announcing the donation of `etcd-operator` to the Cozystack project and the new `etcd-operator.cozystack.io/v1alpha2` API, covering the Membership-API lifecycle, key features, changes compared to `v1alpha1`, the in-place migration path, and a status comparison against the official `etcd-io/etcd-operator` roadmap ([**@kvaps**](https://github.com/kvaps) in cozystack/website#573). +The following documentation shipped in the v1.4.x patch releases and is included for completeness: -* **[website] feat(blog): add KubeVirt VM Disk and VM Instance how-to post**: A practical how-to blog post explaining Cozystack's VM Disk and VM Instance primitives and how they simplify running virtual machines on Kubernetes compared to raw KubeVirt ([**@tym83**](https://github.com/tym83) in cozystack/website#586). +* **[website] Add backup and recovery guides for managed applications**: Operator and tenant guides for managed-application backups covering PostgreSQL, MariaDB, ClickHouse, and FoundationDB *(shipped in v1.4.1)* ([**@androndo**](https://github.com/androndo) in cozystack/website#536). +* **[website] Cilium Gateway API — architecture, security, and migration guide**: Comprehensive `networking/gateway-api.md` page for the Cilium-backed Gateway API feature *(shipped in v1.4.2)* ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#509). +* **[website] Publish Kubernetes API endpoint via external-dns with kuberture**: Documents exposing the managed Kubernetes API endpoint through external-dns using the kuberture system package *(shipped in v1.4.3)* ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#539). +* **[ingress] Explain how ingress works in the platform**: Adds a "How ingress works" overview to the ingress package README *(shipped in v1.4.3)* ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2770). +* **[website] Managed Kubernetes how-to**: Practical guide for deploying and using managed Kubernetes clusters within Cozystack *(shipped in v1.4.3)* ([**@tym83**](https://github.com/tym83) in cozystack/website#565). +* **[website] Platform-managed backups introduction**: Introduces Cozystack's platform-managed backup capabilities for stateful workloads *(shipped in v1.4.3)* ([**@tym83**](https://github.com/tym83) in cozystack/website#566). +* **[website] talm: document DRBD sysctl tuning, keepalive toggle, and etcd quota**: Documents the talm DRBD sysctl performance tuning, the DRBD keepalive toggle, and the etcd backend quota configuration that ship in talm v0.31.0 *(shipped in v1.4.3)* ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#567). +* **[website] Cozystack vs OpenStack comparison**: Vendor-neutral comparison post covering seven dimensions — architecture, compute, networking, storage, managed services, operations, and multi-tenancy — for teams evaluating private cloud options *(shipped in v1.4.4)* ([**@tym83**](https://github.com/tym83) in cozystack/website#580). -* **[website] docs(networking): Cilium Gateway API — architecture, security, and migration guide** *(backported to v1.4.2)*: Adds a comprehensive networking page covering the platform-level `gateway.enabled` toggle, per-tenant opt-in, inheritance model, HTTP-01 and DNS-01 certificate modes, per-service HTTPRoute and TLSRoute tables, a five-layer security model, and a step-by-step ingress-nginx migration path ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#509). +## Other Repositories -* **[website] docs(talm): document DRBD sysctl tuning, keepalive toggle, and etcd quota** *(backported to v1.4.3)*: Adds documentation for talm covering DRBD sysctl performance tuning, the DRBD keepalive toggle, and the etcd quota configuration ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#567). +### talm v0.31.0 *(shipped in v1.4.4)* -* **[website] feat(blog): add Managed Kubernetes how-to post** *(backported to v1.4.3)*: Practical guide for deploying and using managed Kubernetes clusters within Cozystack ([**@tym83**](https://github.com/tym83) in cozystack/website#565). +* **[talm] Add DRBD-oriented sysctl and etcd backend defaults**: Adds production-tested TCP sysctls to the `cozystack` preset that prevent TCP port exhaustion during DRBD reconnect storms, an opt-in `tcpKeepaliveTuning` group, and a configurable `etcd.quotaBackendBytes` (default 8 GiB) ([**@IvanHunters**](https://github.com/IvanHunters) in cozystack/talm#131). +* **[talm] Emit progress on stderr, enrich lookup error chain, retry transient failures**: Fixes `talm template -f X > Y` emitting a stray progress line on stdout, enriches lookup-failure errors with a six-class hint taxonomy, and retries transient connectivity failures with exponential backoff ([**@lexfrei**](https://github.com/lexfrei) in cozystack/talm#212). +* **[talm] Require a template before rendering the chart**: Running `talm template` without `--file`/`--template` now fails fast with `templates are not set` instead of triggering live node discovery and a misleading `connection refused` ([**@lexfrei**](https://github.com/lexfrei) in cozystack/talm#217). +* **[talm] Exclude loop devices from LVM global_filter**: Adds `/dev/loop.*` to the LVM `global_filter` in the generated Talos machine config to prevent unexpected LVM activation inside loop-mounted images ([**@kvaps**](https://github.com/kvaps) in cozystack/talm#215). -* **[website] feat(blog): add platform-managed backups introduction post** *(backported to v1.4.3)*: Blog post introducing Cozystack's platform-managed backup capabilities for stateful workloads ([**@tym83**](https://github.com/tym83) in cozystack/website#566). +### ansible-cozystack *(shipped in v1.4.x)* -* **[website] feat(blog): add Cozystack vs OpenStack comparison post** *(backported to v1.4.4)*: Vendor-neutral comparison post covering seven dimensions — architecture, compute, networking, storage, managed services, operations, and multi-tenancy ([**@tym83**](https://github.com/tym83) in cozystack/website#580). - -* **[talm] feat(cozystack): add DRBD-oriented sysctl and etcd backend defaults** *(backported to v1.4.4)*: Adds production-tested TCP sysctls to the `cozystack` preset to prevent TCP port exhaustion during DRBD reconnect storms. An opt-in `tcpKeepaliveTuning` group is available for faster idle-socket failure detection. The etcd backend quota is exposed as `etcd.quotaBackendBytes` (default 8 GiB) ([**@IvanHunters**](https://github.com/IvanHunters) in cozystack/talm#131). - -* **[talm] fix(cli, engine): emit progress on stderr, enrich lookup error chain, retry transient failures** *(backported to v1.4.4)*: Fixes `talm template -f X > Y` producing YAML files with a stray progress line on stdout. Lookup failures now surface the resource kind, namespace, id, and dialed endpoints, with a six-class error taxonomy. Transient connectivity errors are retried up to three times with exponential backoff ([**@lexfrei**](https://github.com/lexfrei) in cozystack/talm#212). - -* **[talm] fix(engine): require a template before rendering the chart** *(backported to v1.4.4)*: Running `talm template` without `--file`/`--template` previously triggered live node discovery and produced a misleading `connection refused` error. The guard is now evaluated before any chart render or node dial ([**@lexfrei**](https://github.com/lexfrei) in cozystack/talm#217). - -* **[ansible-cozystack] feat(prepare): exclude loop and virtual devices from host LVM scanning** *(backported to v1.4.3)*: Sets an LVM `global_filter` in `/etc/lvm/lvm.conf` on all prepare playbooks so the host LVM does not scan or activate DRBD, device-mapper, zd-device, or loop-backed volume groups. Exposed as `cozystack_lvm_global_filter` inventory variable ([**@kvaps**](https://github.com/kvaps) in cozystack/ansible-cozystack#49). - -* **[ansible-cozystack] fix(prepare): enable `containerd device_ownership_from_security_context` for CDI block imports** *(backported to v1.4.3)*: Adds a k3s containerd drop-in config enabling `device_ownership_from_security_context` on the CRI plugin. Without this, the KubeVirt CDI importer fails with "cannot open /dev/cdi-block-volume: Permission denied" when writing VM disk images into raw block volumes ([**@lexfrei**](https://github.com/lexfrei) in cozystack/ansible-cozystack#48). - -* **[ansible-cozystack] feat(prepare): exclude loop devices from LVM `global_filter`** *(backported to v1.4.4)*: Adds `/dev/loop.*` to the LVM `global_filter` in the generated Talos machine config, preventing unexpected LVM activation on boot ([**@kvaps**](https://github.com/kvaps) in cozystack/ansible-cozystack#51). +* **[ansible-cozystack] Exclude loop and virtual devices from host LVM scanning**: Sets an LVM `global_filter` on all prepare playbooks (Ubuntu, RHEL, SUSE) so the host does not scan or activate DRBD, device-mapper, zd, or loop-backed volume groups; exposed as `cozystack_lvm_global_filter` ([**@kvaps**](https://github.com/kvaps) in cozystack/ansible-cozystack#49). +* **[ansible-cozystack] Enable containerd device_ownership_from_security_context for CDI block imports**: Adds a k3s containerd drop-in enabling `device_ownership_from_security_context` so the KubeVirt CDI importer can write VM disk images into raw block volumes instead of failing with "Permission denied" ([**@lexfrei**](https://github.com/lexfrei) in cozystack/ansible-cozystack#48). ## Contributors @@ -250,7 +371,7 @@ We'd like to thank all contributors who made this release possible: ### New Contributors -We're excited to welcome our first-time contributors: +We're excited to welcome our first-time contributor: * [**@elaugaste**](https://github.com/elaugaste) - First contribution!