213 Commits

Include renames

Author	SHA1	Message	Date
frolov.so	db97e375bd	docs(virtual-machine): correct persistent EFI migration documentation ... Assisted-by: LLM Signed-off-by: frolov.so <opsmon.sh@yandex.com>	2026-09-26 11:40:08 +03:00
Aleksei Sviridkin	2c2c0301f7	chore(deps): roll up pending Renovate updates and move to Go 1.27.1 (#4447) ... ## What this PR does This replaces the open Renovate PRs with one rollup. Go modules move to their newest releases that keep the Kubernetes libraries and controller-runtime where they are, the toolchain and builder images move to Go 1.27.1, and the first-party code gets the Go 1.27 modernizers. Tests for the code the bumps touch come first, in their own commits, and pass on both the old and the new dependencies. Before the bumps: - The e2e sandbox image builds again: dl.min.io answers 410 for mc since MinIO archived it, so mc now comes from its last GitHub release with a pinned checksum. - Renovate stops offering tags of the apimachinery fork. They mirror upstream, so #2999 is a downgrade that drops our patch. - `make unit-tests` now runs the tests of the nested Go modules (kubevirt-csi-driver, token-proxy, kubeovn-webhook, api/apps/v1alpha1). They never ran in CI before. - New tests: JWT verification in the dashboard token proxy, the kube-ovn plunger metrics, cert-manager object idempotency in the tenant gateway, values-file layering in the operator, and the kubevirt CSI driver wrapper. - One real fix. The lineage webhook decoded admitted objects with encoding/json, so an int64 above 2^53 came back rounded. jsonpatch v2.5 compares numbers exactly and would emit a replace that rewrites such a field in the pod. The webhook now decodes with apimachinery's json. Held back on purpose, with the reasons in the commit messages: - Kubernetes libraries stay on v0.35. #3001 needs the apimachinery fork rebased onto release-1.37 first. - cel-go v0.29.2 and etcd v3.6.15, the lines k8s.io/apiserver v0.35 is built and tested with. - grpc v1.83.2. v1.84.0 is affected by GO-2026-6443 and no v1.84 patch release carries the fix yet. - kubevirt.io/api and the CDI API stay on what upstream csi-driver builds against. From CDI v1.60 it no longer compiles. - api/apps/v1alpha1 keeps go 1.26.4, so consumers of the published types are not forced onto Go 1.27. - go fix runs without embedlit (Go 1.27-only syntax, breaks backports to release branches on Go 1.26) and omitzero (would make metadata, spec and status required in the published OpenAPI schemas), and one errorsastype rewrite that did not compile is left out. govulncheck reports nothing reachable in any of the five modules, standard library included. Supersedes #4311, #4310, #4309, #3452, #3993, #3927, #3902, #3805, #3451, #3415, #3414, #3413, #3302, #4050 and #3363. #3001 stays open. Not checked locally: the images that build upstream projects (kube-ovn, kamaji, cilium, linstor-csi and others) with the Go 1.27 builder. The image build jobs of this PR cover them. Closes #4487. ### Screenshots No UI changes. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note chore(deps): move the toolchain and builder images to Go 1.27.1 and raise Go dependencies to their latest compatible releases, closing the published advisories; the lineage webhook no longer rewrites int64 fields above 2^53 in admitted objects. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a restore resource policy option for Velero strategies. * Added a test target to run Go tests across tracked modules. * **Bug Fixes** * Updated the end-to-end test environment to download and verify the MinIO client from its current release location. * Added coverage for token validation, webhook patching, controller behavior, and reconciliation. * **Maintenance** * Updated the Go toolchain, dependencies, and container image versions. * Refreshed pinned workflow actions and base image digests. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-09-26 10:17:29 +04:00
Aleksei Sviridkin	a285a5ef36	feat(kubernetes): add a proxmox substrate for managed clusters and worker pools (#4088) ... ## What this PR does **Series behind #3481** — three separable pull requests: 1. #4087 — the Cluster API providers (capmox + in-cluster IPAM) 2. **#4088 — the `substrate` switch** in `kubernetes` and `kubernetes-nodes` 3. #4089 — the Proxmox CSI driver and cloud-controller-manager #4087 and #4088 share no files and can be reviewed in parallel. #4089 builds on this one. --- #4087 adds the Cluster API providers; this PR teaches the two managed-cluster charts to point at them. ### The shape A single value, `substrate: kubevirt | proxmox`, on both charts — not a parallel set of charts. The control plane is Kamaji either way; what changes is the infrastructure half: | | `kubevirt` (default) | `proxmox` | |---|---|---| | `Cluster.infrastructureRef` | `KubevirtCluster` | `ProxmoxCluster` | | worker template | `KubevirtMachineTemplate` | `ProxmoxMachineTemplate` | | apiserver Service | `ClusterIP` | `LoadBalancer` | | tenant CCM/CSI | in-cluster (`kccm`, kubevirt-csi) | rendered off; the Proxmox pair is #4089 | The KubeVirt path renders byte-for-byte what it rendered before. Checked by rendering `kubernetes-nodes` with default values at `main` and at this head and diffing the output, not only by the snapshots — the snapshots cover `nodegroup.yaml` alone, and an earlier revision of this branch changed the reconcile Job while they stayed green. The parent chart's default render differs only in the random fields of `talos-secrets`, which differ between two renders of `main` as well. `osImage` (#4171) is refused on the proxmox substrate. Only half of it could be honoured there — `factory.version` reaches the machineconfig's installer image while the worker disk keeps coming from the Proxmox template — and a field that half-works is worse than one that says no. `talos.version` and `talos.schematicID` still set the installer target. ### Three things that only show up once workers are off-cluster **The apiserver endpoint.** A KubeVirt worker runs inside this cluster and reaches the tenant apiserver by ClusterIP. A Proxmox worker does not: it lives on the hypervisor's L2 and cannot route to `10.96.0.0/16`. The machine provisions fine, `VMProvisioned=True`, `providerID` set — and then never joins, with no CSR to show for it. On the proxmox substrate the Kamaji Service is therefore a `LoadBalancer`, and the reconcile Job reads `.status.loadBalancer.ingress[0].ip` instead of `.spec.clusterIP`. **And DNS, for the same reason.** The worker machineconfig carried `nameservers: ${COREDNS_IP}` — the management cluster's CoreDNS ClusterIP, equally unreachable. The proxmox path takes its own resolvers (`proxmox.dnsServers`) and the chart refuses to render without them, rather than writing an address that cannot answer. **`externalManagedControlPlane` and the managed-by annotation.** capmox's webhook nil-derefs on a `ProxmoxCluster` whose control-plane endpoint is empty unless `spec.externalManagedControlPlane` is set, which is correct here — Kamaji owns the endpoint. Separately, the `cluster.x-k8s.io/managed-by: kamaji` annotation that the KubeVirt path carries must **not** be set on the Proxmox object: with it, CAPI never sets an ownerRef and capmox stops at `Waiting for Cluster Controller to set OwnerRef`, leaving the address pool uncreated and every worker waiting on infrastructure. ### Linked clones by default `proxmox.full: false` — worker disks are linked clones of the template rather than full copies. Measured on a ZFS-backed store: **8 KiB per worker instead of 10.2 GiB**. Operators who want independent disks set `full: true`; the value is plumbed straight through to `ProxmoxMachineTemplate`. `proxmox.storage` is a full-clone parameter and is refused with a linked clone: a linked clone always lives on the template's storage, and both Proxmox (`parameter 'storage' not allowed for linked clones`) and capmox's own CRD (`Must set full=true when specifying storage`) reject the pair. The chart says so at render time, naming both values, instead of failing at apply time with an error about a `ProxmoxMachineTemplate` nobody wrote. ### Sizing `ProxmoxMachineTemplate` takes `numCores` and `memoryMiB`, not Kubernetes quantities, so the pool's `resources.cpu`/`memory` are converted (millicores → cores, bytes → MiB, both rounded up). The rounding is pinned in `tests/substrate_branches_test.yaml` with `1500m` and `8G`, where rounding up and down disagree; `tests/substrate_proxmox_sizing_test.yaml` covers the refusal to size a pool that gives neither. ### Known upstream behaviour this surfaces `TalosConfigTemplate.spec` is immutable by schema, so the reconcile Job cannot rewrite an endpoint it has already published — `kubectl apply` fails with `TalosConfigTemplate.Spec is immutable`, and the template has to be deleted for the Job to recreate it. A fresh install never meets it; it appears when the apiserver address changes on an existing cluster. Handling it changes the KubeVirt path too, so it is not in this PR: it is a separate change on `proxmox/tct-replace`, with its own upgrade note. ### Evidence ``` $ KUBECONFIG=<tenant> kubectl get nodes -o wide kubernetes-px-md0-jmzds-ndnl6 Ready <none> 75s v1.35.6 10.0.0.160 Talos (v1.13.6) $ KUBECONFIG=<tenant> kubectl get pods -A cozy-cilium cilium-… Running kube-system konnectivity-agent Running kube-system coredns-… Running ``` No management-cluster address remains in the rendered `TalosConfigTemplate`. ### Testing `make unit-tests` — exit 0. `apps/kubernetes` 26 suites / 314 tests; `apps/kubernetes-nodes` 29 suites / 160 tests / 12 snapshots, all unchanged on the default path. Each of the ten commits passes both suites on its own. Every substrate branch is asserted on both sides, and every assert goes red on the mutation it exists for: inverting the `SVC_IP` condition, inverting the `nameservers` condition, `ceil`→`floor` in the sizing, removing the proxmox gate from each of the kccm and csi templates (the two RBAC bindings included), pinning the retention lookup back to `KubevirtMachineTemplate`, widening the IPv6 class of the `dnsServers` validation, dropping the parent chart's `dnsServers` validation, the storage-with-linked-clone refusal, and the `osImage` refusal. Negated `containsDocument` asserts carry `any: true`. `hack/talos-reconcile-heredoc_test.bats` additionally runs the rendered heredoc through a real shell: a `dnsServers` entry with a substitution is refused and named, an IPv6-shaped one with a substitution is refused too, and a real address survives the shell unchanged. ### Screenshots Not applicable — no UI change. ### Downstream repositories I walked the trigger map in `docs/agents/contributing.md` against this diff, file by file. What it matches is below. **No box is ticked**, because every follow-up it points at depends on a decision that is not mine to make: #3481 asks whether Proxmox becomes a supported platform at all, and writing its documentation or its Terraform schema before that is answered would be work built on an assumption. I have not filed speculative PRs or issues in those repositories for the same reason. If the direction is accepted, I open them; a human should decide which. **cozystack/terraform-provider-cozystack — matches, no follow-up yet.** - `packages/apps/kubernetes/values.schema.json` and `packages/apps/kubernetes-nodes/values.schema.json` both gain fields: `substrate`, and a `proxmox` object (`allowedNodes`, `dnsServers`, `ipv4Config`, and on the pool side `templateTags`, `network`, `storage`, `full`). The provider hand-maintains a schema, a model and an expand/flatten pair per Kind, with no codegen from these files. - `values.yaml` gains defaults for those fields (`substrate: kubevirt`, `proxmox.full: false`, `prefix: 24`). The provider restates defaults it models and sends its own, so a stale one wins silently rather than showing as a diff. **Everything else: no match.** No app added, renamed or removed (website app lists); no version enum, `kind`, `plural`, `release.prefix` or output Secret/Service name changed; `packages/core/platform/values.yaml` untouched; the `*-rd` files are regenerated, not renamed (ccp); no `hack/` change; no node prerequisites (talm, ansible); no telemetry metric; no commit-convention change (community). - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(kubernetes): add a `substrate` value to the `kubernetes` and `kubernetes-nodes` applications, selecting whether worker VMs run on KubeVirt in this cluster (`kubevirt`, the default and unchanged) or on an external Proxmox VE cluster through capmox (`proxmox`). On the Proxmox substrate the tenant apiserver is published through a LoadBalancer and worker disks default to linked clones. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added support for choosing KubeVirt or Proxmox for Kubernetes clusters and node pools. * Added Proxmox configuration for nodes, networking, storage, DNS, templates, cloning, and IPv4 allocation. * Proxmox deployments now use the appropriate cluster, machine, networking, and control-plane resources. * **Validation** * Added checks for supported substrates, required Proxmox settings, sizing, and incompatible options. * **Documentation & Tests** * Documented the new settings and expanded coverage for rendering, validation, cloning, sizing, and template replacement. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-09-25 22:09:23 +04:00
Aleksei Sviridkin	b16edec3f3	fix(tcp-balancer): rename backends and pin HAProxy so the chart starts on 3.3+ (#4366) ... ## What this PR does `haproxy:latest` now resolves to 3.3 and newer. Since 3.3 a backend named like a frontend is a fatal config error, so every tcp-balancer pod fails at config parse (#1966). We hit it on Cozystack 1.6.1 with `haproxy:3.4.4` behind the tag. - backends get a `_backend` suffix and every `default_backend` follows; the rename is the one from #2321 by @officialasishkumar - the image is `haproxy:{{ .Chart.AppVersion }}` with `appVersion: 3.4.4`, the current 3.4 LTS tag - the whitelist `acl`, `accept` and `reject` lines of each frontend render inside one guard - `tests/` holds helm-unittest suites, and the package `Makefile` gets the `test` target the sweep looks for - the `whitelistHTTP` description says it needs a non-empty `whitelist` Two inputs change behaviour on upgrade. On `main` both fail at config parse with `no such ACL : 'whitelist'` on every HAProxy version, because the `acl` line sits inside the `with` and `accept if whitelist` does not: - `whitelistHTTP: true` with an empty `whitelist` now fails the render with `whitelistHTTP is enabled but whitelist is empty`. Its HelmRelease upgrade fails and the release stays on its old revision, which is already crash-looping. - `kubernetesAndTalos` with an empty `whitelist` now starts, and ports 6443 and 50000 accept any source. With `external: true` the Service is a LoadBalancer and carries both ports. `whitelist` is the only switch those frontends have, so an empty list means no filtering. Every other input renders the same config apart from the backend names. The image and the `checksum/config` annotation change, so the Deployment rolls, and for releases already crash-looping on `latest` the rollout is the fix. Validation: renders of `main` and of this branch for the default, whitelist and empty-whitelist inputs were checked with `haproxy -c` on 2.9.7 and 3.4.4. `helm unittest .` passes 9/9, and removing the `reject` or `accept` line from any single frontend turns a test red. Fixes #1966 ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map against the diff: chart templates, `Chart.yaml`, the package `Makefile`, `tests/`, and a description-only change in `values.yaml` with its generated carriers. No field is added, removed or renamed and no default changes, so nothing in the map fires. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(tcp-balancer): rename HAProxy backends and pin the image to the chart appVersion so the balancer starts on HAProxy 3.3+. whitelistHTTP with an empty whitelist now fails the render; kubernetesAndTalos with an empty whitelist leaves ports 6443 and 50000 unfiltered. ```	2026-09-25 19:29:39 +04:00
Marian Koreniuk	8722462066	fix(kubernetes-nodes): refuse osImage on proxmox instead of honouring half of it ... #4171 gave a worker pool an osImage, choosing where its system disk comes from: a CDI clone of a shared golden image, or an HTTP import from an Image Factory. A Proxmox worker's disk comes from neither. capmox clones a VM template that already carries Talos, and nothing in this chart can change what that template holds. Half of the field would still appear to work, which is why it is refused rather than ignored. osImage.factory.version reaches the machineconfig's installer image and changes the reconcile Job, so a pool could set it, watch the Job roll, and end up running one Talos while booting another: v1.12.6 moves the installer to .../installer/<schematic>:v1.12.6 and leaves the ProxmoxMachineTemplate untouched. osImage.builtin has no meaning here at all. Nothing is lost by refusing it: the installer target on this substrate is settable through talos.version and talos.schematicID, which do reach it. The branches suite gains the refusal case. The substrate parameter's description said changing the value "rolls the pool". The chart refuses the change on a live pool, and the description says so; it reaches the README, the values schema and the CRD. Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM	2026-09-25 16:46:15 +02:00
Marian Koreniuk	517baf945d	fix(kubernetes,kubernetes-nodes): validate dnsServers, refuse a substrate switch, drop the Pod threshold on proxmox ... Three faults in the proxmox substrate, none of which the render refused. 1. proxmox.dnsServers reached the reconcile Job's shell unescaped. The Job applies the worker TalosConfigTemplate through an unquoted heredoc, so every byte of its body is subject to parameter expansion and command substitution at Job runtime. The INVARIANT comment above that block says each tenant-controlled value must be shell-escaped or pattern-validated; the new nameserver list was neither. `quote` there is YAML quoting. Rendered with `$(id -u)` and a backtick pair in dnsServers, extracted the way hack/talos-reconcile-heredoc_test.bats does and run under sh, the shell executed both and put their output in the worker's resolver list. The Job's pod holds the parent cluster's Talos token, cluster id, cluster secret and bootstrap token in its environment, and its Role can read the Talos CA and patch the parent KamajiControlPlane. Each entry is now validated as an IPv4 or IPv6 address at render time, in both charts. Validation rather than escaping because Talos takes addresses here: an address cannot carry a metacharacter, so the property holds by construction, and the ordinary mistake of writing a hostname is caught in the same move. IPv6 is matched on its character set rather than its full grammar — what matters is which bytes may reach the shell, and the apiserver rejects a malformed address anyway. 2. Nothing stopped a live cluster or pool from being switched between substrates. `substrate` had no immutability of any kind, and the `## @immutable` marker this adds is dashboard-enforced only, so both charts now read the live object. Switching a cluster removes the KubevirtCluster, the cloud-controller-manager, the kubevirt-csi controller and the in-tenant -csi HelmRelease with all their RBAC from the render. Flux then uninstalls the CSI driver behind every PVC the tenant has. CAPI does not prevent it — its Cluster webhook forbids removing spec.infrastructureRef but admits a change of kind. Switching a pool rolls every worker to the other hypervisor and drops the machine template the draining MachineSet still references, because the retention loop looks up only the kind the new substrate uses. A pool whose substrate differs from its parent's was not rejected either, though the README says the two must match. Both guards are inert offline, like assertNoForeignPool and assertParentVersion beside them: with no apiserver the lookup is empty and a first install renders. 3. The worker WorkloadMonitor reported a proxmox pool as broken. The controller lists Pods matching the monitor's selector and sets Operational=false while fewer than minReplicas are ready. On kubevirt those Pods are the virt-launcher Pods. A proxmox worker is a VM on another machine with no Pod here, so the count is always zero and any threshold pinned the pool — and through the lineage labels its whole application — to WorkloadsReady=False while every node was healthy. The monitor still renders for the dashboard and the lineage; what it no longer carries on proxmox is a threshold counted in Pods. Two smaller things. The wait-loop log line is written per substrate, so a default kubevirt render of kubernetes-nodes is byte-identical to main and no existing pool gets a new reconcile Job on upgrade. And the comment claiming a rename of $kubevirtmachinetemplate would roll every live worker VM is replaced: the hash is taken over the rendered template, not the variable name, so renaming it leaves the render byte-identical. Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM	2026-09-25 16:46:15 +02:00
Marian Koreniuk	ceb7119a1b	fix(api): regenerate deepcopy for the proxmox value structs ... The proxmox blocks added to the Kubernetes and KubernetesNodes values carry string slices, and the generated deepcopy never learned about them: a copy shared allowedNodes, dnsServers and templateTags with its original instead of owning them. Cause was procedural rather than conceptual — the per-package `make generate` under packages/apps regenerates the chart schema and the Go types, but the deepcopy functions come from the repository-root target. Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM	2026-09-25 16:46:15 +02:00
Marian Koreniuk	c5996c1d0f	fix(kubernetes,kubernetes-nodes): give Proxmox workers an endpoint they can reach ... A worker on the proxmox substrate is a VM on the hypervisor network, not a Pod in this cluster. It was handed two addresses only an in-cluster consumer can route to: the apiserver as a ClusterIP, and the management cluster's CoreDNS ClusterIP as its resolver. The VM booted, the Machine reported Ready with a providerID, the disk was where it should be — and no CSR ever arrived. Nothing in that picture points at an address. The control-plane chart now publishes the tenant apiserver as a LoadBalancer on the proxmox path, putting the endpoint on the network the workers share, and the pool's reconcile Job reads the Service's LoadBalancer address instead of its ClusterIP for the host entry the workers dial. The nameservers follow the same reasoning: proxmox pools carry their own resolvers rather than pointing at a CoreDNS they cannot reach, and the chart refuses to render a pool that has not named any. KubeVirt keeps the ClusterIP on both counts. Its workers are in this cluster, where a ClusterIP is the right answer and a LoadBalancer address would be an unnecessary one. Verified end to end on a real tenant cluster: the node joins, its CSR is approved, and cilium, konnectivity and coredns come up in the tenant. One operational note this surfaced, not caused by it: TalosConfigTemplate.spec is immutable, so the reconcile Job cannot rewrite an endpoint that has already been published — the template has to be deleted for the Job to recreate it. That matters only when an existing cluster's apiserver address changes. Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM	2026-09-25 16:46:15 +02:00
Marian Koreniuk	d78af928fc	feat(kubernetes-nodes): default Proxmox worker disks to linked clones ... A full clone on a ZFS pool configured with `sparse 0` reserves the whole volsize the moment the VM is created: 10.2G for a worker whose Talos image occupies 337M. A linked clone writes only its own increment — measured at 8K for the same worker on the same host — because it shares the template's base snapshot until it diverges. The cost is that the clone holds that snapshot, so the template cannot be rotated while any linked clone still references it. That is bounded rather than permanent: consolidation detaches each worker 24 hours after creation, so template rotation is blocked for at most a day. Consolidation is not a chart concern, but two of its constraints are worth recording where the flag is set, because both were discovered by testing on a live host rather than from the docs. A linked clone can only be detached by moving its disks to a *different* storage — Proxmox refuses a same-storage move outright. And both disks must move: leaving efidisk0 behind keeps the template pinned just as effectively as the system disk would. Set full: true for a pool that must be independent of the template from the first second. Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM	2026-09-25 16:46:14 +02:00
Marian Koreniuk	db839d1198	feat(kubernetes): add a proxmox substrate for managed clusters ... Companion to the same switch in kubernetes-nodes. The control plane is Kamaji on either substrate, so this changes exactly one object — the one Cluster.infrastructureRef points at — plus the two tenant addons that exist only because the workers are KubeVirt VMs living in this cluster. A KubevirtCluster needs no spec: its workers are in the same cluster and reach the apiserver over the pod network. A ProxmoxCluster addresses an external hypervisor, so it carries what capmox cannot infer — which Proxmox nodes may host VMs, which addresses they may take, and who resolves names for them. controlPlaneEndpoint stays absent on both: the cluster.x-k8s.io/managed-by annotation hands that to Kamaji. The address pool is required rather than defaulted. capmox has no DHCP mode; it allocates from an InClusterIPPool built out of ProxmoxCluster.spec.ipv4Config, so an unset pool is not a smaller configuration but a ProxmoxCluster that never reaches Ready and worker Machines that wait on it forever. Refusing at render time turns that into a message instead of a silence. The KubeVirt cloud-controller-manager, its cloud-config and the kubevirt-csi driver are gated off on the proxmox path. They reach the workers' VirtualMachines and DataVolumes in this cluster; on Proxmox those objects do not exist, and leaving them rendered would put controllers in the tenant reconciling nothing. Their Proxmox equivalents are separate packages and land separately. The default path is unchanged: rendering it before and after this commit differs only in the per-render random bootstrap tokens, and all 19 existing suites pass untouched. Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM	2026-09-25 16:29:09 +02:00
Marian Koreniuk	85f576756b	feat(kubernetes-nodes): add a proxmox substrate for worker pools ... Worker pools are already Talos machines created by Cluster API; only the substrate underneath them is fixed to KubeVirt. This adds `substrate: kubevirt|proxmox` and swaps exactly two things: the machine template object and every reference to it. Autoscaler capacity annotations, kubelet reservations, MachineDeployment, MachineHealthCheck, WorkloadMonitor and the talos-reconcile Job are substrate-agnostic and rendered once, which is why this is a branch in the existing chart rather than a second copy of it. The default path is unchanged down to the bytes: the render snapshot and all 15 existing suites pass untouched, and the hash-named template keeps its variable name so no live worker VM rolls on upgrade. Five things the Proxmox path must do differently, each learned from a failure that pointed somewhere else: - Skip capmox's guest-agent and cloud-init-status probes. Talos ships neither, and left on they leave the VM booted and the Machine stuck on "error waiting for agent: the operation has timed out". This is the same class of problem `virtualMachineBootstrapCheck.checkStrategy: none` solves for CAPK's SSH probe; capmox's CRD names TalOS in the skipCloudInitStatus description. - Clone a template carrying the nocloud image, not openstack. capmox injects bootstrap data as a nocloud ISO, and an openstack image ignores it exactly as silently as a nocloud image ignores the configdrive the KubeVirt path depends on. - Pass templateTags through verbatim. capmox compares the template's tag set with slices.Equal after sorting, so a selector listing a subset matches nothing and reports "found 0 VM templates with tags ..." while the tags are plainly present. - Require resources.cpu and resources.memory. Proxmox has no VirtualMachineClusterInstancetype, so the lookup that normally supplies effective CPU and memory has nothing to read — and those two numbers feed the kubelet reservations and the autoscaler capacity annotations. A pool that cannot state them would be scheduled against invented capacity, so it is refused at render time. KubeVirt-only knobs (gpus, logSerialConsole, podCpuLimit/Request) are refused too rather than accepted and ignored. - Refuse proxmox.storage unless full is true. A target storage is a full-clone parameter: a linked clone always lives on the template's storage, and both Proxmox ("parameter 'storage' not allowed for linked clones") and capmox's own CRD ("Must set full=true when specifying storage") reject the pair. Left to them, a pool with storage set and a linked clone fails at apply time with an error about a ProxmoxMachineTemplate the operator never wrote; failing the render names the two values instead. The Talos image is not a chart input on this path: capmox clones a template that already exists on the hypervisor, so building and rotating that template is an operator responsibility with no equivalent on the KubeVirt side. Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM	2026-09-25 16:29:08 +02:00
scooby87	03e2da5805	feat(postgres,mariadb)!: drop plaintext user passwords (#4078) ... ## What this PR does Removes the plaintext `users[].password` field from the `apps/postgres` and `apps/mariadb` charts (and the mariadb `root` account), so managed-database passwords can no longer be pinned in tenant values or committed to git. Passwords are always chart-generated into the `<release>-credentials` Secret and preserved across reconciles via `lookup`, matching the model `redis`/`valkey` already use. This closes the "User passwords" item of the portal upstream epic (aenix-org/cozyportal#1060). The generated password never lands in the init-script Secret or the SQL it runs: the postgres init-job reads each user's password from the mounted `<release>-credentials` Secret at run time and applies it through a psql variable (`ALTER ROLE … WITH PASSWORD :'pw'`), so it does not linger in the init-script manifest a tenant can read. It does live in cleartext `stringData` in the `<release>-credentials` Secret itself, which is a chart manifest and therefore kept in Helm release history for `MaxHistory` revisions like any chart-generated secret — that residue is exactly why rotation cannot be built on the chart (see below). A `password` or a `passwordRotation` key left over in a tenant's values is accepted-but-inert and draws an admission warning, so an operator editing it is told it has no effect rather than getting a silent 200. Backup/restore into a copy keeps the target's chart-managed credentials working: the CNPG driver clears `bootstrap.enabled` once recovery is healthy so the init-job re-applies the generated passwords onto the recovered roles, and the MariaDB driver excludes `mysql.global_priv` from the dump (`ignoreGlobalPriv`) for backups captured from this release onward, so restoring one of those into a copy does not clobber the target's chart-managed users or `root`. Excluding the grant table has a flip side, and it applies to every MariaDB restore, not only a to-copy one: only accounts declared through `User` CRs (the chart's `users` map) are captured, so an account created out of band directly in MySQL is in no backup and a restore brings back its data and `mysql.db` grants but not the account itself. This is also a backup-side fix only: a restore from a backup taken **before** this change still replays the source's `mysql.global_priv` and overwrites the target's users and `root`, and `root` then needs an out-of-band reset (the operator applies `rootPasswordSecretKeyRef` only at datadir bootstrap, so there is no in-band path). See `docs/operations/backup-classes.md` for the convergence window, the out-of-band-account limitation, and this old-backup caveat. Password **rotation** is deliberately **not** in this PR. A chart-rendered credentials Secret keeps its cleartext in Helm release history for `MaxHistory` revisions, so a rotation built on it cannot revoke a leaked password; rotation is being reworked as a controller that owns the Secret — design proposal in cozystack/community#72. Migration is seamless: existing releases keep their current passwords on upgrade (the value already lives in `<release>-credentials`, which `lookup` re-reads and preserves); a `password` still present in a tenant's values is ignored, so no existing HelmRelease fails validation. Verification: `helm unittest` passes for both charts (postgres 87, mariadb 126, including render-time username guards that reject any name k8s would later reject as an invalid Secret key or User resource name); `go build ./...`, `go vet`, the backup-controller tests and the apps-apiserver tests are green; `make generate` was run with the CI-pinned `cozyvalues-gen` v1.6.0 so `values.schema.json`, the READMEs, the API types, and the `cozyrds` ApplicationDefinitions are all in sync (the branch was rebased onto `main`, which reconciled cleanly with the read-replica autoscaling feature that landed there). ### Screenshots N/A — no UI change. ### Downstream repositories - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: affected — `content/en/docs/next/cozystack-api/go-types.md` is hand-written and still shows `Password:` in the Go examples, and the managed-app reference pages regenerate from these chart READMEs, so both document `users[name].password` until refreshed (not yet filed). - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: affected — the provider hand-maintains the postgres/mariadb schemas and per-field defaults; this PR removes `users[].password`. A follow-up issue/PR is needed there (not yet filed). `terraform-provider-cozystack` and `website` are reached by this diff; both boxes are left unticked because the follow-ups are not yet filed. Same-repo future work this change suggests: the controller-owned password rotation (design in cozystack/community#72), and extending the "passwords only in the Secret" model to the other engines that still accept plaintext (`clickhouse`, `mongodb`, `nats`, `opensearch`, `rabbitmq`, `vpn`). ### Release note ```release-note feat(postgres,mariadb)!: passwords for managed PostgreSQL and MariaDB users can no longer be set via `users[].password` — they are always auto-generated and stored only in the `<release>-credentials` Secret, never rendered into the init-script or the SQL it runs. Existing releases keep their current passwords on upgrade; a `password` (or a `passwordRotation`) left in values is ignored with an admission warning. Password rotation is deferred to a follow-up controller design (cozystack/community#72). MariaDB backups now exclude the `mysql.global_priv` grant table (`ignoreGlobalPriv`), so a restore recreates only accounts declared through the chart's `users` map — an account created out of band directly in MySQL is not captured and must be recreated after a restore. ```	2026-09-25 15:49:53 +03:00
Alexey Artamonov	9f857f64e1	feat(mariadb)!: generate every user password, root included ... users[].password is gone from the chart, its schema and the API type. Every password, root included, is generated into <release>-credentials and kept across reconciles by lookup, and a password left in values is ignored. An upgrade that finds no credentials Secret fails instead of minting new passwords, because mariadb-operator applies root only at datadir bootstrap and would advertise a root it never set. The Secret carries k8s.mariadb.com/watch so the operator applies a changed password without waiting for its requeue interval. A username or database name becomes a Secret key and a resource name, so render-time guards reject a name outside the DNS-1123 rule or past its limit: 80 for a username, as the User CRD allows, and 64 for a database, as the server allows. Names that differ only in "_" and "-" collide on one resource; that is rejected on install only, since an existing release already applied the overwrite. A user in both admin and readonly of a database keeps a single ALL grant under the unchanged Grant name, because renaming a Grant makes the operator revoke the live privilege. A role member past 128 characters, the server's account-name limit, and a database name of 65-80 characters, which the Database CRD accepts but the server does not, are rejected on install and on upgrade only when their Grant or Database does not exist yet, so a release that already applied one keeps upgrading. User CR names and keys are quoted so a username that reads as a YAML scalar stays a string. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>	2026-09-24 18:10:57 +03:00
Alexey Artamonov	2b5877eff9	feat(postgres)!: generate user passwords and read them at runtime ... users[].password is gone from the chart, its schema and the API type. Every application-user password is generated into <release>-credentials and kept across reconciles by lookup, so an upgraded release keeps the password it already has and a password left in values is ignored. The init-job mounts that Secret and applies each password with ALTER ROLE and a psql variable, so no password is rendered into the init-script Secret that Helm keeps in release history. While bootstrap is enabled the Secret is marked credentials-pending, and NOTES say the generated passwords apply once the init-job runs. Names reach superuser SQL and the init-script shell, so render-time guards reject a username, database or extension name outside [-._a-zA-Z0-9] or past the length PostgreSQL silently truncates: 63 bytes for a role or extension, 54 for a database, whose <db>_readonly role would otherwise truncate and fail the next run. A username over 63 bytes that the credentials Secret already carries still upgrades. users.postgres, the CNPG superuser, is rejected in every bootstrap state. Changing a database's key in values drops the database, so the reject message and the README give the rename that keeps the data: a platform administrator renames it and its roles as the CNPG superuser before the key changes. The init-job cannot hang its Helm hook: the readiness wait ends on elapsed time below activeDeadlineSeconds and the Job never retries, so a purged restore target fails with a diagnostic. The delete-users pass reads managed users from pg_shdescription instead of parsing the version-dependent \du+ layout, which on PostgreSQL 16 and later matched nothing and left a removed user able to log in; it compares names as CREATE ROLE stores them and reassigns a removed user's objects in every database before dropping it. A database without roles renders, and extension names are quoted for hyphenated names and folded to lowercase so they resolve as they did unquoted. A value that carries its own surrounding quotes, the only way a hyphenated extension rendered before, has that pair removed and keeps its case, so such a release still upgrades. The README reads a password with jq, since jsonpath splits a dotted key. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>	2026-09-24 18:10:57 +03:00
Aleksei Sviridkin	db6c20f841	chore(deps): raise Go modules to their latest compatible releases ... Several pinned modules carry published advisories: golang.org/x/crypto, net, text and mod, grpc, the opentelemetry SDK and exporters, velero, the etcd client and cel-go. Rather than take only the minimum fixed versions, every module outside the Kubernetes lockstep set moves to its newest release whose own requirements keep k8s.io/* and controller-runtime where they are and fit Go 1.27.1. Where the newest release asks for more, the module stops at the last one that does not: cert-manager v1.19.6, source-controller/api v1.8.5, source-watcher/api v2.1.1, fluxcd meta v1.26.0; helm-controller/api, fluxcd kustomize, gateway-api and external-snapshotter stay put. A few modules stop below their newest release on purpose. cel-go and the etcd modules stay on the lines k8s.io/apiserver v0.35 was built and tested with (cel-go v0.29.2, the version Kubernetes itself ships, and etcd v3.6.15), since the apiserver is their only consumer here. grpc stays on v1.83.2: v1.84.0 predates the fix for GO-2026-6443, a server panic the CSI driver's gRPC server can reach, and no v1.84 patch release carries it yet. kubevirt.io/api and the CDI API stay on the versions upstream csi-driver builds against: from CDI v1.60 StorageSpec.Resources changes type and upstream no longer compiles. velero v1.18 adds RestoreSpec.ResourcePolicy, which the strategy CRD embeds, so the CRD is regenerated. The backup controller only consumes the velero v1 types; the new Queued and ReadyToStart backup phases fall through to its requeue path like any other in-progress phase. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-09-24 17:23:18 +03:00
Aleksei Sviridkin	a92e89e969	docs(kubernetes): describe the OIDC UID header from v1.33 ... The tenant OIDC guide, the version field description and the e2e helper comment still described a --feature-gates term rendered on v1.32. Left as they were, they would tell an operator that the chart switches that alpha gate on, which it no longer does. The guide now says what a v1.32 tenant has to write itself to get the UID, and which --emulated-version values fail the render, so an operator who meets that refusal finds it explained. The breaking-changes note in the README said the chart refuses the mixed gate spelling while OIDC is on; with that refusal gone it would promise a render error that no longer comes. The version description returns to its earlier wording in values.yaml and in the update-versions.sh template, because that template rewrites the line on every version bump and would otherwise bring the old text back. make generate carries it into the schema, README, API types and ApplicationDefinition. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-09-24 15:10:53 +03:00
Aleksei Sviridkin	4e9f83ac0d	fix(kubernetes)!: move or refuse extraArgs flags the control plane owns ... From 26.9.4-edge the tenant control plane sets a list of kube-apiserver flags itself and drops any extraArgs entry naming one before the apiserver sees the list; 26.3.6-edge let such an entry win. An existing tenant that set, for example, its own admission plugins through extraArgs would lose them on the first reconcile after the upgrade with nothing to say so, and a flag written with its value in a separate entry leaves that value behind as a positional argument the apiserver refuses. Two of those flags have a KamajiControlPlane field the control plane renders them from. An --enable-admission-plugins=value or --kubelet-preferred-address-types=value entry is moved into that field while the field is empty, and left out of the rendered extraArgs, so the tenant keeps its setting across the upgrade without an edit. The move is computed from the values on every render; the last entry wins, as it did while the control plane kept one entry per flag name, and a repeated address type is dropped because that field is a set. The post-upgrade Job that patches extraArgs onto the KamajiControlPlane reads the same list. Every other entry naming an owned flag is refused at render time in every OIDC mode, since the chart cannot stop the drop and the refusal is where the operator learns of it. So are the two moved flags beside a non-empty field, with a value that names nothing or is quoted, and in the two-entry spelling. The list mirrors the Kamaji builder for the shape of control plane this chart renders, and the extraArgs description names the same flags. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-09-24 14:59:12 +03:00
Aleksei Sviridkin	0f9b8dc16c	feat(kubernetes): expose admission controllers and kubelet address types ... The KamajiControlPlane carries two lists the tenant control plane renders into kube-apiserver flags: spec.admissionControllers into --enable-admission-plugins and spec.kubelet.preferredAddressTypes into --kubelet-preferred-address-types. The chart rendered neither from the tenant values; the address types were a fixed list. Pass both through. Admission controllers are rendered only when the list is non-empty, so a tenant that sets none keeps the control plane's own default set, and a list replaces that default rather than adding to it. An empty or absent list of address types renders the list the chart rendered before, so a tenant that sets nothing renders exactly as it did. The names are declared as enums copied from the KamajiControlPlane CRD at the provider pin, so the application API refuses a misspelled name where the tenant edits it instead of at apply time. Both fields are optional. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-09-24 14:59:11 +03:00
Aleksei Sviridkin	88264216b7	docs(kubernetes): restate the OIDC UID flags for the merged reading ... The migration note told operators to move the extraArgs entry count in the same edit so the running control plane would rewrite its apiserver arguments; the control plane compares a hash of the list now, so any change already forces the rebuild and no second edit is owed in either direction. The paragraphs describing a value that survives only as the last entry of its flag, and an empty value reaching the apiserver as a bare flag, describe a build that is no longer shipped. What replaces them is the accumulation each flag actually performs: a uid-headers list the apiserver validates as the union of every entry, feature gates settled across entries with the last term winning and one spelling of the kube component throughout, and an emulated version that two entries cannot both name. One contract the earlier text sold is narrowed to what the chart keeps. The render checks run only where the chart renders something, so the three readings it refuses to guess at switch them off as well, and a shape that would otherwise be refused passes through to a control plane that does not start. A flag and value written as two entries is a narrower case, and the page says which checks hold next to one. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-09-24 14:59:01 +03:00
myasnikovdaniil	9bec21461d	feat(kubernetes): boot tenant worker disks from a shared golden Talos image (CDI clone) (#4171) ... ## What this PR does Tenant worker boots from talos OS disk image that CDI streams over HTTP from image factory, and it does that once per worker. Same bytes N times, and N chances for a fetch to fail. In closed or rate-limited environment that is wrong shape. This PR adds opt-in catalog and per-pool selector that reads from it. `packages/system/kubernetes-worker-image` imports one golden DataVolume into `cozy-public` per `(schematicID, version)`, named `talos-worker-<schematicID>-<version>`. Pool opts in with `osImage.builtin` on its own `KubernetesNodes`, and its `disk-system` DataVolume then carries `source.pvc` pointing at that golden, so CDI does storage layer CSI clone instead of HTTP import. One import per flavor, shared by every worker of every tenant. Catalog is disabled by default, enable by adding `cozystack.kubernetes-worker-image` to `bundles.enabledPackages`. Its knobs (`imageFactoryURL`, `storageClass`, `storage`, `images`) are set through `spec.components.platform.values.kubernetesWorkerImage` on cozystack-platform Package. That needed optional package helper to learn components argument, which `cozystack.platform.package.optional.default` did not have, so every optional package registered through that wrapper could name variant and nothing else. Thirteen other packages get same path for free. `osImage` has two mutually exclusive forms: - `osImage.builtin` boots pool workers by CDI-cloning golden from catalog. Clone is storage layer CSI copy, so OS image never crosses pod network and factory answers once for whole fleet instead of once per worker. - `osImage.factory` keeps HTTP import and lets pool pin own `imageFactoryURL`, `schematicID` and `version`. This is bring-your-own path for operator running own factory or caching mirror, and it is what pool gets when `osImage` is omitted. In-guest talos installer written by reconcile job follows same resolution, so pool that pins release boots it and upgrades into it instead of being flipped back to cluster default halfway. This is pool half of what #3950 asks for. That issue wants platform level defaults for two channels, worker OS image and in-guest registry mirrors. Here image channel gets per-pool source and shared cache, platform default plumbing is left to the issue. ## Existing workers do not roll With `osImage` omitted rendered `KubevirtMachineTemplate` is byte identical to what chart produced before. KMT is named by hash of its own content, so any drift would re-image every worker VM in fleet on next upgrade. Stored render snapshot is untouched by this PR, that is the evidence and not the claim. Editing `osImage` on existing pool re-images that pool, like changing any other image reference. ## What fails at render Clone that would silently degrade into something slower or wrong is refused instead of rendered. Golden absent, or its import in terminal `Failed` phase. Pool `storageClass` different from golden one, because CDI cant CSI-clone across storage classes and falls back to host assisted copy over pod network, which is exactly the transfer `osImage.builtin` removes. Empty class on either side resolves against cluster default class before comparing, so pool at `storageClass: ""` is checked too. `diskSize` below golden storage, because clone target cannot be smaller than source. Talos release the matrix lists as unsupported, whether pool reached it through `talos.version` or through `osImage.*` override. Matrix passes talos minor it does not list, so what it promises is that known-bad pairing is refused, not that every pairing is known. `schematicID` that is not plain lowercase alphanumeric, or version that is not `vMAJOR.MINOR.PATCH`, both are interpolated into DataVolume name and image URL while `values.schema.json` types them as bare strings. Golden that is only mid-import is not an error. CDI holds worker disk until source populates, and failing there would abort whole pool HelmRelease for length of a download. Same reasoning covers pool that already cloned: absent or `Failed` golden fails render only for pool still waiting for its first disk, otherwise reconcile of healthy pool would abort over source it no longer reads. Golden that declares no StorageClass is read from its PVC, not from whatever default is in force today, because those differ once cluster default moves. Cluster with two default classes resolves same way Kubernetes does, newest wins. Pool that already cloned is identified by annotation naming exactly that pool, not by disk name prefix, because pool `md0-gpu` produces disks carrying `md0` prefix, and it reads its own class from its own bound PVC, so admin moving `storageclass.kubernetes.io/is-default-class` does not fail release of pool whose disks are all on right class. That resolved class is then written onto worker disk. Reading one class and binding another is how pool at empty `storageClass` passed the comparison and still took host assisted copy on its next worker, so guard and disk now come from one value. HTTP path leaves empty class to cluster as before, which is what keeps its render byte identical. Catalog side refuses duplicate `(schematicID, version)` entries, entries whose schematic or version cannot produce name pool can reconstruct, and edit to `storage`, `storageClass` or source URL of entry whose golden already exists. Last one is not patchable in place, so without guard it lands as rejected patch on next upgrade. Goldens carry `helm.sh/resource-policy: keep`, so dropping entry stops managing golden instead of deleting it. Missing golden aborts whole Kubernetes HelmRelease of every tenant pinned to it, not just the disk, and each cloned worker disk annotates golden it came from, so "does anything still clone this" is answerable before manual delete. Three drift guards live in `hack/check-worker-image-catalog-defaults.bats`, because values that have to agree sit in files nobody edits together. One-sided talos bump trips all three. ## Upgrade Three pool values that never produced working image rendered before and hard-fail now, so pool carrying any of them flips its HelmRelease to `InstallFailed` on upgrade and not on next edit: `talos.version` in `vMAJOR.MINOR` form, `schematicID` that is not plain lowercase alphanumeric, and empty `talos.imageFactoryURL`, which used to render schemeless `/image/...` source URL. None of the three could boot worker, so this is failing earlier and not a regression, but it surfaces on pool nobody touched and fix in each case is to correct the value. CR carrying such value is admitted by API with value intact, and later unrelated edit reaches `.spec.values` of HelmRelease while it stays `InstallFailed`, so edit is taken everywhere and applied nowhere. ## Why e2e does not exercise the clone path Clone is storage layer copy, so it needs StorageClass whose volumes are reachable from any node. Merge-gating e2e has none: those lanes run on talos containers, which share runner kernel and cannot load DRBD, so node-local `local` is only class there. Nothing would arrange the co-location such a class requires either, golden in `cozy-public` has no consumer and binds where CDI importer landed, while worker disk binds where its VM is scheduled. So the suites keep importing over HTTP with `osImage` omitted, which is also the render the no-roll guarantee rests on. This PR touches no e2e harness file: `hack/` differs from main only by the two guards this feature owns, one tying catalog default golden to pool chart talos and StorageClass defaults, one on in-guest installer heredoc. Switching suites over does not work on that substrate. Golden asks for `replicated`, lane does not have it, CDI resolves no StorageProfile, and catalog HelmRelease times out on DataVolume stuck in `ImportInProgress`. Passing lane own class would fix the class and leave the co-location. Nightly and tag build run same suites on QEMU with DRBD and `replicated`, which is the substrate clone path was verified on. That is where e2e coverage belongs, as its own change with that lane storage headroom measured rather than assumed. Worth being straight about what this leaves, because it bears on whether the feature earns its place. The e2e argument was never the strong one: golden fetches from image factory exactly once, and #3513 established that node-join stall tracks host `kvm_amd` `vgif` flag rather than registry egress. What clone genuinely removes is the fanout, N workers streaming over pod network becomes N storage layer copies. That is property of storage layer, so it is worth having where DRBD is and worth nothing where it is not. This ships as production capability with the constraint stated on the field, not as CI improvement. ## Testing On this tree: - full `make unit-tests` plus `make test-controllers`: 95 charts, 2520 helm-unittest cases, 1710 bats cases, `go test ./pkg/...` and `./internal/...`, exit 0 - `pre-commit run --all-files`: exit 0, root `make generate` and per-app generate both clean, so no generated-file drift - render snapshot unchanged, which is what pins byte identical render for pool that does not adopt `osImage` - mutation pass over render guards, each term deleted or inverted in turn and suite required to notice it by name. Restoring presence gate on catalog StorageClass comparison reddens exactly one case, inverting its `ne` to `eq` reddens five. Reverting worker disk to omit its resolved class reddens three cases that pin where builtin disk binds, and forcing that class non-empty on every path reddens HTTP-path case and all three render snapshots, with snapshot diff naming the changed KMT hash `E2E Tests` passed on run 35831830568. That run was on the previous head, 4fd8e7ba, and this head differs from it only by one test comment and the commit message. ## Verified on a live cluster Throwaway three-node talos stand (OCI, talos v1.13.6, LINSTOR/ZFS, `replicated` StorageClass) installed from this branch `packages/` tree. This covers what neither `helm template` nor unit suites can answer. It predates the change that writes resolved StorageClass onto worker disk, and that stand ran with explicit `replicated` on both sides, which is the value that change now renders. - The catalog wires up from `bundles.enabledPackages` alone: `PackageSource cozystack.kubernetes-worker-image` reconciled, its HelmRelease installed, and the golden `talos-worker-<schematic>-v1.13.6` imported to `Succeeded` in 108s (6Gi, RWX, `replicated`). - A pool on `osImage.builtin` clones it on the storage layer. Both worker DataVolumes went `CSICloneInProgress` then `Succeeded` in 43s with `Running=False reason=Completed message="Clone Complete"`, carrying `source.pvc` and no `source.http`, and no importer or upload Pod appeared in the tenant namespace. - The clone yields a bootable talos root disk. Both worker VMs reached `Running`/`Ready`, joined the tenant cluster and went `Ready` about 66s after creation, and the tenant cilium, coredns, csi, ingress-nginx and metrics-server addons all installed on them. - Access modes match across the hop: golden `ReadWriteMany` 6Gi to worker `ReadWriteMany` 20Gi on the same `replicated` class, grown by CDI. - The written annotations are correct in practice: each worker disk carries `kubernetes-worker-image.cozystack.io/golden` naming the golden and `.../pool` naming its pool. - helm-controller converges on a builtin pool: forcing a reconcile left the content-hashed KMT name unchanged and the VMI and DataVolume UIDs identical, so a reconcile does not roll the pool. - Cross-namespace clone RBAC is in place as shipped: `create datavolumes --subresource=source -n cozy-public` resolves to `yes` for a tenant ServiceAccount through the `cdi-clone-dv` RoleBinding, and CDI runs with `cloneStrategyOverride: csi-clone`. This PR adds no RBAC of its own. ### Screenshots Not applicable, no UI change. ### Downstream repositories Trigger map points at two repositories. Provider models every pool field by hand, so `osImage` needs schema, model and expand/flatten pair there. Already tracked as cozystack/terraform-provider-cozystack#39, filed when the field was still called `image`. `content/en/docs/next/operations/configuration/platform-package.md` is a hand-written table of `spec.components.platform.values.*` keys and nothing generates it, so `kubernetesWorkerImage` needs a row there. Filed as cozystack/website#707. - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/issues/707 - [x] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: https://github.com/cozystack/terraform-provider-cozystack/issues/39 - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(kubernetes): tenant worker pools can boot their system disk from a shared golden talos image instead of importing it from the Image Factory once per worker. Enable the opt-in `cozystack.kubernetes-worker-image` package via `bundles.enabledPackages` to publish one golden DataVolume per `(schematicID, version)` in `cozy-public`, then set `osImage.builtin` on a `KubernetesNodes` pool to have CDI clone it on the storage layer. A pool that omits `osImage` keeps importing over HTTP and renders exactly as before. The clone path needs the pool and the golden on the same StorageClass, and one whose volumes are reachable from any node, in practice a replicated/DRBD class. ```	2026-09-24 12:10:07 +05:00
Myasnikov Daniil	bc8c54d6e6	feat(kubernetes): boot tenant worker disks from a shared golden Talos image ... Every tenant worker VM imports the same Talos raw image from the Image Factory over HTTP, once per worker: a 20-worker cluster fetches the identical ~4.15 GiB artifact 20 times, and each fetch is a live dependency on the Factory at exactly the moment a node joins. packages/system/kubernetes-worker-image is a new opt-in catalog that holds one golden DataVolume per (schematicID, version) in cozy-public, named talos-worker-<schematicID>-<version>. A worker pool opts in with osImage.builtin on its own KubernetesNodes resource, and its disk-system DataVolume then carries source.pvc pointing at that golden, so CDI performs a storage-layer CSI clone instead of an HTTP import -- one import per flavor, shared by every worker of every tenant. Confirmed on a LINSTOR-backed cluster: the worker disk is a CSI clone plus a resize, with no host-assisted upload pod. Both halves are off unless asked for. The catalog is registered through the platform's optional-package helper and disabled by default, so a cluster that does not enable it creates no golden and keeps importing over HTTP. The disk source is derived entirely from osImage, which is absent by default, so a pool that does not set it renders byte-identical to the pre-feature chart and adopting the feature never rolls existing workers. Editing osImage on an existing pool re-images that pool, as changing any other image reference does. What the render refuses, rather than degrading into a silent fallback: - A pool whose golden has no catalog entry, or whose golden terminally failed to import. A golden that is merely mid-import is not an error; CDI holds the worker disk until the source populates. - A pool whose StorageClass differs from its golden's. CDI cannot CSI-clone across classes and falls back to a host-assisted copy over the pod network, which is the transfer this feature exists to remove. An empty class on either side means the cluster default and is resolved before the comparison, and a pool that has already cloned reads its class from its own bound PVC rather than from today's default -- the golden's side is frozen the same way, and an admin moving storageclass.kubernetes.io/is-default-class must not fail the release of a pool whose disks are all on the right class. That resolved class is then written onto the worker disk, so the disk binds where the guard checked instead of on whatever the default has become by then -- reading one class and binding another is how a pool at an empty storageClass passed the comparison and still took the host-assisted copy. The HTTP path leaves an empty class to the cluster as before, which is what keeps its render, and the KubevirtMachineTemplate hash named after it, byte-identical. - A diskSize below the golden's storage size, which a clone target cannot be. - A schematicID that is not a plain lowercase alphanumeric identifier, or a version that is not a vMAJOR.MINOR.PATCH release. Both are interpolated into a DataVolume name and an image URL, and values.schema.json types them as bare strings, so a malformed one otherwise reaches the render. The catalog applies the same patterns, since both sides build the same name from the same fields. - An images[] entry edited after its golden exists. A DataVolume spec cannot be patched in place, so the message names the entry and the supported move -- a new (schematicID, version) -- instead of leaving the API server to reject the patch on the next upgrade. Goldens carry helm.sh/resource-policy: keep, so dropping an entry stops managing a golden rather than deleting it: a missing golden aborts the whole Kubernetes HelmRelease of every tenant pinned to it, not just the disk. Each cloned worker disk annotates the golden it came from and the pool that owns it, so "does anything still clone this?" is answerable before a manual delete instead of after. A builtin pool needs a StorageClass whose volumes are reachable from any node, in practice a replicated/DRBD one. The golden in cozy-public has no consumer and binds as soon as it imports, wherever CDI's importer landed, while a worker disk binds where its VM is scheduled, and nothing here arranges for those to be the same node. The platform ships cloneStrategyOverride: csi-clone, which rules out the host-assisted strategy that would cross nodes. What a node-pinned class does in that situation is not measured, so values.yaml states the requirement and not a symptom. ComputePlane pools forward only the per-pool shape fields, and osImage is excluded there by name: the worker OS image, its Talos installer and its Image Factory are operator decisions about the platform's own images. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-09-23 15:35:44 +03:00
Aleksei Sviridkin	c290dee945	fix(vm-instance)!: refuse resources that cannot size the VM ... resources.cpu and resources.sockets are typed as quantities, but they land in domain.cpu.cores and domain.cpu.sockets, which KubeVirt declares as uint32. The chart cast them with int64, which turns 500m, 1.5 or 1e1 into 0 without an error, and KubeVirt boots a zero core count as one core. The user asked for half a core and got a one-core VM with nothing reporting it. The raw spelling is now decoded as YAML, the way an integer field reads it, and the render stops unless the result is a positive whole number that fits in uint32. A leading zero is refused, because YAML reads 010 as octal 8. The check keys on presence rather than truthiness, so an unquoted 0 is refused instead of being read as unset. Memory gets the same presence check. A quoted "0" was truthy and rendered requests.memory: "0", while an unquoted 0 was falsy and dropped, so one quantity took two paths. A zero or negative amount now stops the render in both spellings. BREAKING CHANGE: values that used to render now either fail or render differently. A cpu or sockets that does not decode to a plain whole number within uint32 (a millicore, suffixed such as 1k, fractional, zero, negative or leading-zero value), and a zero or negative memory, fail the next upgrade of the release until the value is fixed. This includes cpu or sockets set on its own beside an instanceType, which used to be ignored whatever it held. A spelling that the int64 cast truncated to 0 but that YAML decodes to a whole number, such as the exponent 1e1 or the trailing dot 2., used to boot one core and now sizes the VM with the count it spells, so such a VM changes size on upgrade. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-09-23 11:31:07 +03:00
Yan Bondarenko	eee0596661	docs(tcp-balancer): say that whitelistHTTP needs a non-empty whitelist ... The render now fails when whitelistHTTP is set with an empty whitelist, so the values description tells users that before they hit the error. Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>	2026-09-23 11:18:19 +04:00
Aleksei Sviridkin	4bf95815a8	fix(kubernetes): render the aggregation-layer UID header for tenant OIDC (#3539) ... ## What this PR does Tenant clusters with `spec.oidc.mode: System` or `CustomConfig` authenticate users fine, but a UID does not survive the hop to an aggregated API server. The tenant kube-apiserver publishes `requestheader-uid-headers` into the `extension-apiserver-authentication` ConfigMap only when the flag is set, and the chart never set it. The aggregation layer still sends `X-Remote-Uid`, but the extension server trusts only the header names published in that ConfigMap, so it ignores the one it gets and the call arrives without a UID. The field documentation even told operators to add the flag by hand via `controlPlane.apiServer.extraArgs`. It forwards a UID rather than creating one. ServiceAccount tokens carry one, and for those the aggregation hop starts working. Users authenticated by `spec.oidc.mode: System` still have none, because the generated AuthenticationConfiguration maps `username` and `groups` only; giving them one means mapping a claim, which changes the identity of users who have none today, and is tracked separately. The chart now renders `--requestheader-uid-headers=X-Remote-Uid` on v1.32 and newer, and adds `--feature-gates=RemoteRequestHeaderUID=true` on v1.32, where the gate is still alpha and the apiserver rejects the flag without it. From v1.33 the gate is beta and on by default, so only the header flag is rendered. v1.31 gets nothing: the flag does not exist there, and writing it by hand stops the apiserver from starting. `--emulated-version` in the same passthrough moves that decision, because the gate resolves through the versioned spec at the emulated version. Rendering the gate on every version from v1.32 rather than on v1.32 alone was the alternative, and it would remove the migration step entirely, because an opt-out would then drop two entries instead of one and move the entry count by itself. It loses on what it leaves behind: a redundant gate term on every cluster, which stops the apiserver from starting on the day the gate is retired from the known list. The design also prefers rendering over failing for the same reason the fail branches are narrow, since a cluster stranded on a HelmRelease that no longer converges is a worse outcome than one that never gets the UID fix. The chart reads an operator entry the way the apiserver reads it: header list elements matched whole and untrimmed and folded as `EqualFold` folds them, gate names compared exactly, gate values resolved as `strconv.ParseBool` resolves them, `AllAlpha` and `AllBeta` blankets honoured, an empty value treated as no entry. On v1.32 an entry that says nothing about `RemoteRequestHeaderUID` gets a second one carrying the operator's own terms plus `RemoteRequestHeaderUID=true`. Carrying those terms into the merged copy is what makes it land correctly whether the tenant control plane collapses duplicate flags into the last one or hands both to the apiserver, which merges them in the order it reads them. Tests cover the render matrix over mode, version and operator input: every fail path with its exact message, the ownership and merge paths, duplicate entries in both orders, blanket gates, empty values, and case variants on the header name and on the gate value. The latest-version tenant lane creates its tenant with `spec.oidc.mode: System` and asserts the flag on the rendered KamajiControlPlane, so the tenant apiserver starts with the flag in CI. Nothing asserts the UID arriving at an aggregated apiserver; that needs an extension server inside the tenant and is not in this change. The layer where the count-keyed reset actually bites is the tenant Deployment's arguments, and asserting there needs a live tenant control plane, so it is not in this change either. The rule this follows is that a configuration which renders and boots today keeps rendering. Both flags are new, so an `extraArgs` shape reaching this code was rendering before it existed, and stranding a running tenant on a HelmRelease that no longer converges is a worse outcome than one that never gets the fix. A few shapes do stop rendering, and reaching any of them takes writing `--requestheader-uid-headers` into the passthrough by hand, attempting exactly what the chart now does and getting it wrong; in each the kube-apiserver fails option validation and never starts, so nothing that was working is lost. `docs/oidc-tenant.md` covers these shapes. One known limitation is in there and worth naming here: on v1.33 and newer, adding the documented opt-out is count-neutral in `extraArgs`, and so is removing it again, and on v1.33 and v1.34 adding or removing `--emulated-version=1.31`. The pinned Kamaji build rewrites a running apiserver's arguments only when that count changes and otherwise keeps every flag that has left the list, so either the header flag stays beside a now-disabled gate or the disabled gate stays beside the header flag, and the apiserver refuses to start. Changing the entry count in the same edit is the mitigation, and the fix belongs to the pin rather than to this chart, tracked in #3541. The paths that need no operator edit are clear: turning the feature on for a default tenant moves the count, and a tenant already carrying the opt-out renders byte-identically before and after. Out of scope: `mode: None` still publishes no UID header even though ServiceAccount tokens carry UIDs. That is pre-existing, and widening the condition changes the render for every default tenant. Tracked separately, along with the claim mapping above. Closes #3377 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change Checked the trigger map against the diff: no package added or removed, no schema field, enum or default changed (description text only), no ApplicationDefinition surface touched, no hack/ layout moved. The managed-apps reference page picks up the new field description through release-time README regeneration. ### Release note ```release-note fix(kubernetes): tenant clusters with `spec.oidc.mode` set to `System` or `CustomConfig` now render `--requestheader-uid-headers=X-Remote-Uid` on the tenant kube-apiserver, plus the `RemoteRequestHeaderUID` feature gate on v1.32, so that an aggregated API server can read the caller's UID out of the request headers. Without that flag the tenant apiserver never publishes `requestheader-uid-headers`, and an aggregated API server ignores the UID header even for callers that have one, ServiceAccount tokens among them. It forwards a UID rather than creating one: users authenticated by `spec.oidc.mode: System` still have none, because the generated AuthenticationConfiguration maps `username` and `groups` only, and that is tracked separately. Adding the flag by hand through `controlPlane.apiServer.extraArgs` is no longer necessary, and an entry of your own is respected: the chart steps aside when yours already carries `X-Remote-Uid` or already enables the gate, merges `RemoteRequestHeaderUID=true` into a copy of a v1.32 `--feature-gates` entry that says nothing about the gate, and renders neither flag when you switch the gate off. The render fails where a `--requestheader-uid-headers` list of your own would stop the kube-apiserver from starting anyway: one that omits `X-Remote-Uid`, one with an empty element, and a list of your own while the gate is off. On v1.33 and newer there is a migration step, and it applies in both directions: the pinned Kamaji build rewrites a running apiserver's arguments only when the number of `extraArgs` entries changes, and otherwise keeps every flag that has left the list. Adding the documented opt-out keeps that number the same, so the header flag already on the Deployment stays beside the now-disabled gate. Removing the opt-out later keeps it the same too, so the disabled gate stays on the Deployment beside the header flag the chart renders again, although your spec no longer shows it. Adding or removing `--emulated-version=1.31` has the same shape both ways on v1.33 and v1.34. The apiserver refuses to start in each case, so change the entry count in the same edit. ```	2026-09-23 00:03:20 +04:00
Aleksei Sviridkin	12e64a99cd	fix(rabbitmq): right-size the default resources preset to s1.nano (#3936) ... ## What this PR does Raises the RabbitMQ default resource preset from `t1.nano` (128Mi memory) to `s1.nano` (250m CPU / 512Mi memory). A default-sized RabbitMQ 4.2 broker on Cozystack 1.6.1 was OOMKilled with exit 137, reached 10 restarts, and never became Ready. A canary on a larger preset reached 3/3 Ready with zero restarts; its working set peaked at 125.8Mi. Against the 128Mi `t1.nano` ceiling that is 98% of the limit in steady state, and cgroup accounting (working set plus page cache) crosses it under load. The canary and its storage were removed after validation. ### Sizing `s1.nano` keeps the CPU of `t1.nano` (250m) and quadruples memory to 512Mi — a 4x margin over the observed 125.8Mi peak. An earlier revision of this PR proposed `u1.nano` (1Gi). Review correctly objected that 1Gi reserves 8x the observed working set with no measured peak to justify it, and asked to either report that peak or right-size the default. The peak is reported above, and the default is right-sized to the smallest preset that clears the ceiling with a real margin rather than to the first one that is comfortably large. ### Upgrade impact cozystack-api applies OpenAPI schema defaults on the read path and stores what the client sent: `Get` runs `applySpecDefaults` on its way out (`pkg/registry/apps/application/rest.go:1232`) and the write puts `app.Spec` into `HelmRelease.spec.values` (`rest.go:1652`). That splits existing brokers in two. **Brokers with no stored `resourcesPreset`** pick up the new chart default on the next Helm reconcile. For them this upgrade is: - a rolling restart of every replica, and - a rise in guaranteed memory reservation from 128Mi to 512Mi per replica, because the sanitize helper sets memory request equal to limit — roughly 1.1Gi extra reserved per 3-replica instance. **Brokers whose values went through a read-modify-write keep `t1.nano` and are not resized.** `Update` opens with `r.Get` (`rest.go:489`), so a GET-then-PUT — `kubectl edit` is the unambiguous case, and any client that reads before writing behaves the same — freezes the schema default of the moment into the stored values, and a stored value beats the chart default. Those brokers are not restarted, not resized, and keep hitting the ceiling this PR raises. Nothing downstream can separate a deliberate `t1.nano` from one a round trip baked in, so this is documented rather than migrated. After the upgrade the affected brokers are the ones that still report the old preset: ``` kubectl get rabbitmq -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PRESET:.spec.resourcesPreset ``` The same command before the upgrade tells you nothing: the cozyrd default is still `t1.nano` then, so the read path fills it in for both groups. Afterwards, anything reading `t1.nano` carries a stored value; setting `resourcesPreset: s1.nano` explicitly on the ones that should move is the fix. PVCs are untouched, so there is no data loss, but node headroom should be checked before upgrading. Also worth knowing when reading the parameter table: the preset does not only apply to a fully omitted `resources`. `cozy-lib.resources.defaultingSanitize` merges it key by key, so a broker pinned to `resources: {cpu: 700m}` keeps its CPU and moves from 128Mi to 512Mi with everyone else. The field descriptions now say so, and a test pins it. The values, generated schema, ApplicationDefinition, typed API, and README all carry the same default. Tests cover the new default, an explicit legacy override, explicit resources, and a partially set `resources` map inheriting the preset for what it leaves out. Validation: - `helm template` with the chart's `cozy-lib` dependency renders `limits 250m / 512Mi` and `requests 0.025 / 536870912` for the default; an explicit `t1.nano` renders `128Mi / 134217728`; explicit `700m / 2Gi` renders `0.07 / 2147483648`. The unit test asserts exactly these values. - Provider counterpart: cozystack/terraform-provider-cozystack#34. ### Screenshots Not a UI change. ### Downstream repositories The default is restated by the Terraform provider, so the linked provider PR updates it and documents the state-plan implication. No other trigger-map entry matches this diff. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [x] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: https://github.com/cozystack/terraform-provider-cozystack/pull/34 - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(rabbitmq): raise the default resource preset from t1.nano to s1.nano so default-sized brokers are no longer OOMKilled. Brokers with no stored `resourcesPreset` are restarted on upgrade and their guaranteed memory reservation grows from 128Mi to 512Mi per replica; check node headroom before upgrading. Brokers whose values already pin `resourcesPreset`, including ones where a read-modify-write through the API froze the old default into them, keep their old size: list them after the upgrade with `kubectl get rabbitmq -A -o custom-columns=NAME:.metadata.name,PRESET:.spec.resourcesPreset` and set `resourcesPreset: s1.nano` explicitly on the ones that should move. The preset also fills any resource a partially set `resources` leaves out, so a broker pinned to cpu alone moves too. A tenant that opted into `resourceQuotas` and sits near its memory ceiling has the extra reservation rejected by quota admission, which shows up as a StatefulSet that never scales rather than as pods left Pending. ```	2026-09-22 21:53:02 +04:00
scooby87	7d07bbe3ce	[kafka] Migrate from ZooKeeper to KRaft mode (#2352) ... ## Summary - Replace ZooKeeper-based Kafka deployments with KRaft (Kafka Raft) metadata mode - Existing ZooKeeper instances migrate automatically and in place via a Helm `pre-install,pre-upgrade` hook Job that drives Strimzi's ZK-to-KRaft migration state machine - New instances deploy directly in KRaft mode with broker and controller `KafkaNodePool`s - The node pools are owned by the migration hook, not by regular chart manifests (see "Why the pools are hook-managed") - Kafka version is user-configurable (v3.9 default, v3.8 available) - Strimzi operator is unchanged in this PR (stays at 0.45.1-rc1); 0.45.x is the last line that supports ZooKeeper→KRaft migration - ZooKeeper removed from the dashboard UI (ApplicationDefinition) ## Migration strategy Both fresh installs and ZooKeeper→KRaft migrations are driven by a single Helm hook Job (`templates/migration-hook.yaml`, `pre-install,pre-upgrade`) that owns the `KafkaNodePool`s. On every install and upgrade the Job: 1. Ensures both node pools exist (idempotent): a broker pool named exactly `kafka` for a migrating cluster — so Strimzi adopts the existing `<cluster>-kafka-N` brokers and their PVCs in place — or `b-<hash>` for a fresh cluster, plus a controller pool `c-<hash>`. The controller pool pins `strimzi.io/next-node-ids` above the broker range so the broker pool keeps node IDs `[0..N-1]` and Strimzi cannot hand the existing broker IDs to the fresh controllers. The shared `kafka` name is claimed atomically (via `kubectl create`) so only one migration per namespace can win it. 2. Inspects the Kafka CR: no CR → fresh install (pools ensured; the main apply then creates the KRaft Kafka CR, nothing to migrate); already `strimzi.io/kraft=enabled` or `status.kafkaMetadataState: KRaft` → done; any other read error → fail closed. Otherwise (a ZooKeeper cluster) it migrates. 3. Annotates `strimzi.io/node-pools=enabled` then `strimzi.io/kraft=migration`, polls `status.kafkaMetadataState` through `ZooKeeper → KRaftMigration → KRaftDualWriting → KRaftPostMigration → PreKRaft → KRaft`, and only once a safe state is reached flips `strimzi.io/kraft=enabled`. If it times out in an intermediate state it aborts without finalising, so Helm fails the upgrade and the next reconcile retries from where it left off. Data is migrated in place: brokers keep their node IDs and reuse their existing PVCs, nothing is copied. Migration is one-way — Strimzi has no supported KRaft→ZooKeeper rollback once complete, so lowering the `version` field or rolling the app back to a pre-KRaft chart is unsupported (take a backup first). ### Why the pools are hook-managed The hook must create the pools and walk the migration state machine *before* the main release apply. A resource a hook creates cannot also be a regular chart manifest across the ZK→KRaft boundary: Helm's main apply finds the pool live in the cluster but absent from the previous (ZooKeeper) release revision and aborts with `original object KafkaNodePool ... not found`, leaving the release `failed` even though the cluster migrated fine (reproduced under server-side apply, the helm-controller default). Managing the pools only from the hook avoids that, so `templates/kafkanodepools.yaml` and the deployed-version gate ConfigMap (`templates/version.yaml`) are removed and the pools are applied idempotently by the Job on every install and upgrade. ## Bug fixes included - ZooKeeper `metricsConfig` referenced wrong ConfigMap key (`kafka-metrics-config.yml` instead of `zookeeper-metrics-config.yml`) - ZooKeeper `storageClass` used `kafka.storageClass` instead of `zookeeper.storageClass` - WorkloadMonitor used `.Values.replicas` instead of `.Values.kafka.replicas` - `validateNameLength` now caps a Kafka app name at the KRaft controller pod-hostname budget (`<release>-c-<hash>-<id>` must fit 63 chars), instead of admitting names that later fail the chart render forever ## What's next A follow-up PR will upgrade Strimzi to 0.51.0 and Kafka to 4.2 once all instances are confirmed migrated to KRaft. That PR is already prepared on branch `KRaft-upgrade-strimzi-051`. ## Test plan - [x] `helm template` renders correctly for new KRaft instances (no ZooKeeper refs) - [x] `helm template` with custom values (1 replica, external, topics, v3.8) - [x] Version validation rejects unsupported versions via JSON schema - [x] helm-unittest: 86/86 across 9 suites - [x] `hack/kafka-migration-hook.bats`: 12/12 — the real hook script executed against a stub kubectl (resolution branches, fail-closed reads, atomic collision guard, finalize guard) - [x] Operator template renders with the unchanged Strimzi 0.45.1-rc1 - [x] Live migration on a disposable cluster under server-side apply: fresh install → healthy KRaft, release `deployed`; in-place migration of a 20000-message ZooKeeper cluster → release `deployed`, broker PVCs adopted in place (same UIDs), and the KRaft data byte-for-byte identical to the pre-migration ZooKeeper data (matching sha256 over every `partition:offset:key:value`) ### Release note ```release-note [kafka] Enable KRaft metadata mode with automatic, in-place ZooKeeper-to-KRaft migration. Kafka version is now configurable (v3.9, v3.8). The Strimzi operator is unchanged (0.45.1-rc1). ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added chart-level Kafka version selector (default v3.9) and KRaft-aware node pool rendering with controller storage sizing. * **Updates** * Default Kafka appVersion updated to 3.9.0; operator and image tag defaults aligned to 0.45.0. * Templates now omit ZooKeeper resources and metrics when KRaft is in use. * **Tests** * New Helm unit tests for Kafka CR, node pools, metrics, topics, dashboard mappings, and workload monitors. * **Documentation** * Values, schema, and README updated to document new version and controller storage options. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-09-22 19:19:21 +03:00
Aleksei Sviridkin	f4e923aa82	docs(kubernetes): restate the apiServer extraArgs contract for OIDC ... The extraArgs field documentation told operators to add --requestheader-uid-headers themselves, which the chart now renders whenever OIDC is on. Describe what it renders per version, when it defers to an operator entry, when it merges into one, and that it still refuses to render shapes the apiserver would not start on. The version field says it too: choosing v1.32 is what makes the chart render the feature gate next to extraArgs, and a reader of that field alone would otherwise not learn that the two interact. The package's hack/update-versions.sh writes that description itself on every make update, so it carries the same text, or the next version bump would drop the sentence again. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-09-22 18:23:48 +03:00
Yan Bondarenko	6d5666f7e6	fix(rabbitmq): describe the preset as filling what resources leaves unset ... cozy-lib merges the preset under resources key by key, so a partially set resources map still takes the preset for everything it omits: with resources.cpu=700m alone the broker renders 700m of CPU against the preset's 512Mi of memory, and this upgrade moves that broker from 128Mi to 512Mi as well. The description said the preset applies when resources is omitted, and cozyvalues-gen copies it verbatim into values.schema.json, the README parameter table and the cozyrd openAPISchema the dashboard renders, so one sentence made the claim on three surfaces. The suite covered a fully omitted resources map and one that sets both cpu and memory, which is the one shape that does not exercise the merge. The new case pins it: reverting defaultingSanitize to replace the preset instead of merging over it turns that case red and leaves the other three green. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>	2026-09-22 17:47:55 +04:00
Yan Bondarenko	cf749988a4	fix(rabbitmq): right-size the default resources preset to s1.nano ... Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> Assisted-by: LLM	2026-09-22 17:47:55 +04:00
Alexey Artamonov	ff637ed9a2	fix(platform): validate the realm-name charset and regenerate kubernetes artifacts ... The realm-name guard checked only the type and echoed the parsed value back, so its hint told an operator who wrote `on` to set realmName: "true" — the token YAML had already discarded. Drop the echo and add a slug-charset check: the name is spliced verbatim into URL path segments and a shell token, so a "/" or a space must be rejected at the source rather than split a URL or the hook script downstream. The description edit in the previous commit was hand-applied to the generated kubernetes README and never reached the Go types, leaving pre-commit red; regenerate both with cozyvalues-gen. Finish the oidc-tenant System-mode prose that still read cozy as a fixed name. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>	2026-09-21 12:15:10 +03:00
Alexey Artamonov	8ef230f0bf	Merge remote-tracking branch 'origin/main' into kafka-multi-instance ... Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> # Conflicts: # docs/storage-immutability.md # packages/apps/kafka/tests/topics_test.yaml # packages/system/kafka-rd/cozyrds/kafka.yaml	2026-09-21 12:05:33 +03:00
Yan Bondarenko	8ec76a3614	fix(kafka): make topics[].config optional ... Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>	2026-09-18 17:30:06 +04:00
Alexey Artamonov	710cdd18bb	fix(kafka): address KRaft-migration review — broker pool adoption, hook safety, tests ... Addresses the branch review on the ZK->KRaft migration path. CRITICAL — broker node pool adoption. Strimzi derives broker/PVC names as <cluster>-<pool>-<id>; a pre-node-pool ZooKeeper cluster has brokers <cluster>-kafka-N, so the adopting broker pool must be named exactly "kafka" or the operator provisions fresh empty brokers and orphans the tenant's data. A new kafka.brokerPoolName helper resolves the broker pool to "kafka" for an existing (migrating) cluster and keeps the release-scoped "<release>-broker" for fresh installs, so multiple Kafka instances still coexist in one namespace. The migration hook creates the pool as "kafka" and fails closed if a "kafka" pool already belongs to another cluster (node pool names are namespace-unique, so in-place migration of co-namespaced ZK clusters is unsupported). Wired through kafkanodepools.yaml, workloadmonitor.yaml and the hook. MAJOR — migration robustness. - Raise the HelmRelease timeout: the migration Job can run tens of minutes but the release timeout defaults to 10m, so helm-controller aborted and thrashed the pre-upgrade hook. Add release.cozystack.io/helm-install-timeout=45m on the kafka ApplicationDefinition, above the Job's activeDeadlineSeconds (2400s). - Idempotent migration: do not re-issue kraft=migration once the cluster is past the ZooKeeper phase (an invalid backward transition Strimzi rejects on retry). - Fail closed on the decision gates: distinguish a genuine NotFound (skip) from any other kubectl error (abort), and never treat an empty kafkaMetadataState on an existing CR as "migrated". MINOR — version downgrade documented as unsupported; integer (not lexical) compare in the version gate; deleteClaim/PVC-retention and the one-per-namespace migration limitation documented in the README; docs/storage-immutability.md updated for KRaft. Tests — helm-unittest coverage for the broker pool name, the hook guards and the idempotent/fail-closed logic; a bats check for the helm-install-timeout budget ordering; and a Chainsaw ZK->KRaft migration suite that seeds a ZooKeeper cluster, drives the migration, and asserts the original broker PVC survives (not recreated) and the cluster reaches KRaft. The migration sequence and the Chainsaw assertions were validated against a live cluster. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>	2026-09-17 11:09:10 +03:00
Matthieu	42db2ca141	feat(kafka): migrate from ZooKeeper to KRaft mode with automatic migration ... Replace ZooKeeper-based Kafka with KRaft (Kafka Raft) metadata mode. Strimzi 0.45 is the last line that still supports ZooKeeper, so this prepares the transition before ZooKeeper support is removed upstream. Chart changes: - The Kafka CR always renders strimzi.io/kraft=enabled with node pools and no spec.zookeeper block. Storage, replicas and resources move to separate broker + controller KafkaNodePools (a homogeneous separate-pool layout works for both fresh installs and migrations; Strimzi 0.45 does not support migration into a combined pool). - Kafka version is now user-configurable via the standard files/versions.yaml pattern (v3.9 default, v3.8 available); appVersion bumped to 3.9.0. - WorkloadMonitors track the broker and controller pools (no zookeeper monitor); the dashboard RBAC grants the controller monitor and drops the zookeeper one. Migration of existing ZooKeeper clusters is handled by a Helm pre-upgrade Job (templates/migration-hook.yaml) gated by a <release>-kafka-deployed-version ConfigMap, following the established seaweedfs/etcd hook pattern. The Job creates the broker + controller pools with Helm ownership labels, drives the Strimzi ZK->KRaft state machine (node-pools=enabled, kraft=migration -> kraft=enabled once KRaftPostMigration is reached) and is skipped once the ConfigMap records version >= "1". Testing: - Helm unit tests cover the Kafka CR, node pools, migration hook, metrics, topics, version map, dashboard RBAC and workload monitors. - The e2e suite is ported to the current Chainsaw framework (hack/e2e-chainsaw/kafka): it asserts the KRaft topology (both node pools, kafkaMetadataState=KRaft), the absence of ZooKeeper, and bootstrap endpoints. Rebased onto current main: TLS listener support, entity-operator resource guards, x-cozystack-options schema annotations and the WorkloadMonitor replica-path fix are all preserved and reconciled with KRaft. Signed-off-by: Matthieu <matthieu@hidora.com> Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>	2026-09-17 11:09:10 +03:00
Andrei Kvapil	d7c0332d5e	chore(dashboard,api): drop the HTTPCache catalog entry and values type ... The dashboard no longer registers a brand icon for the kind, and the ServicesTab fixture moves to TCPBalancer, which the catalog still ships. An instance that survives the removal still renders, on the generic sidebar icon. api/apps/v1alpha1/httpcache goes too. types.go is generated by values-gen out of packages/apps/http-cache, which is gone, so the file could never be regenerated, and no Go file in this repository imports it. Surviving instances do not depend on it: the aggregated API serves the schema recorded in the in-cluster ApplicationDefinition. The terraform provider's cozystack_httpcache resource is hand-written and does not import it either; only its per-kind schema-guard test does, and that goes away with the resource. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com>	2026-09-14 13:11:19 +02:00
Aleksei Sviridkin	cecb4b11d3	feat(opensearch)!: issue HTTP and Dashboards certificates from cert-manager (#2682) ... ## What this PR does The OpenSearch HTTP API (9200) and Dashboards (5601) already served TLS, from the operator's own private CA and with a SAN built from `cluster.local` while the credentials Secret advertised the release on the platform domain, so nothing a client was handed could verify. Both certificates now come from a per-release cert-manager CA in the tenant namespace, and the CA reaches tenants as a key-free trust anchor. - `tls.issuer` selects who issues the HTTP server certificate, `cert-manager` or `operator`, and is named for the issuer because TLS is served under both. Unset resolves from `external` (cert-manager for externally published services, the operator for cluster-internal); an explicit value always wins. - Renders a self-contained cert-manager chain in the tenant namespace: a self-signed Issuer, a CA Certificate, a CA Issuer, and two leaf Certificates, one for the HTTP API and one for Dashboards, each with the SANs of its own Service and the external hostname when `external: true`. Leaf keys are written as PKCS8, since the JVM loader rejects the PKCS1/SEC1 encoding cert-manager emits by default, and the CA subject is pinned so the operator's signature check accepts the chain. - The CA Secret is `<release>.http-ca`. The dot separates the chart-chosen suffix from release names, so no other release can produce that name; the two leaf Secrets stay hyphenated because none of the chart's suffixes is a tail of another. The operator's own transport CA remains at `<release>-ca` and is not touched. - Transport mTLS (9300) stays operator-managed (`transport.generate: true`, `perNode: true`). The security plugin requires it and the chart cannot turn it off. - `hack/e2e-chainsaw/opensearch/` runs the move on a cluster: it installs the app on the operator's HTTP listener, removes `tls.issuer` so it resolves from `external`, and asserts the cert-manager chain, the CR dropping `http.generate`, the admin certificate reissued under the chart CA, the StatefulSet rolling onto the three subPath mounts, the tenant anchor, and the certificate the endpoint actually presents to a client holding that anchor. ### Sysctl carrier and single-node rolls The E2E suite for this PR was red because the chart set `spec.general.setVMMaxMapCount: true`, which makes the operator prepend a privileged `init-sysctl` container to every OpenSearch pod. Tenant namespaces run at the cluster's default PodSecurity level, baseline on Talos, so admission refused the container and the StatefulSet never produced a pod (#4131). The carrier for the sysctl is now on main: #4152 added a DaemonSet to the `opensearch-operator` package that raises `vm.max_map_count` to a floor of 262144 on every node, from a namespace the platform labels privileged. This branch is rebased onto that main and sets the flag to an explicit `false`, with a template comment saying why and a unit test pinning it, so a release no longer asks the operator for a container admission refuses. The value stays with the DaemonSet until the node contract carries it (#4144); the DaemonSet's own template comment, which said the chart still asks, now says it does not. Three more things came out of the same pass. The chart asked the operator to drain data nodes before every rolling restart, and with a single data node the drain never finishes: at v2.8.0 `PreparePodForDelete` excludes the node from allocation and then waits for it to hold no shards, which the only data node can never satisfy, so a one-replica release could not roll for a version bump or for the certificate move this PR relies on. The chart now drains only when a second data node exists, pinned both ways in the unit tests. The chart maps an empty `storageClass` to `replicated` at one replica, and the PR lane runs LINSTOR without DRBD and creates only `local`, so the E2E fixture takes its class from the `storageClass` value the harness passes with `--set-string` from `COZY_E2E_STORAGE_CLASS`, the way the vminstance fixtures do; on the nightly QEMU lane that still resolves to `replicated`. And the external Dashboards Service selected labels no pod carries. `pkg/builders/dashboards.go` at v2.8.0 stamps `opensearch.cluster.dashboards: <cluster name>` on the Deployment, its selector and the pod template, and nothing else; the chart asked for `opster.io/opensearch-cluster` together with `app.kubernetes.io/component: dashboards`, so the LoadBalancer came up with no endpoints and no error to show for it. It selects the label the operator sets now. ### securityadmin cross-trust, still blocked on the DNS base Chart-managed HTTP TLS previously broke the operator's securityadmin reconciliation: the admin certificate was signed by the transport CA while the HTTP listener trusted the cert-manager CA, so the two never verified against each other and users and roles silently never applied. This chart puts one CA under both sides, handing the HTTP CA to the operator through `spec.security.tls.http.caSecret` so that it signs the admin certificate with the CA the listener trusts. That closes the CA mismatch. The other half, the job resolving a name the operator built from `cluster.local` on a cluster that does not use it, was closed separately and is on main, so the security configuration applies once this lands. No operator bump is required; the API exists in the pinned version. That path is version-gated upstream at OpenSearch 2.0.0: below it the operator falls back to the transport CA and the mismatch returns with no error anywhere. The chart therefore refuses chart-managed HTTP TLS on `version: v1` with an explicit render-time failure that names the mechanism and offers the alternatives, rather than letting it fail silently at runtime. ### Trust anchor for tenants The HTTP server's private key is not exposed to tenants. The chart renders a `TenantProjection` sentinel that names `<release>.http-ca` as its source; the CA-extraction controller publishes a `ca.crt`-only copy as `<release>.tenant-ca`, and the ResourceDefinition selects it with the engine-agnostic `internal.cozystack.io/tenant-ca` label. The sentinel is gated on the same condition as the chain, so a plaintext release renders no sentinel instead of one stuck at `SourceNotFound`. Tenants get what they need to verify the server and nothing more. ## Known limitations These are stated rather than left to be discovered: 1. **securityadmin needed both trust and reachability, and both are closed now.** This branch put one CA under the admin certificate and the HTTP listener. The other half was the operator resolving `<svc>.<ns>.svc.<DNS_BASE>` with `DNS_BASE` left at the vendored `cluster.local`, so the job waited on a name that never resolved whichever CA had signed; the platform now renders that value from `networking.clusterDomain` and it is on main, which is what unblocks this branch. 2. **The e2e suite reaches the move, not the chart bump that triggers it.** The suite above arrives at the chart-managed chain by removing `tls.issuer` from a running release, which leaves exactly the values an upgraded release arrives with. The one step it cannot run is the one where the previous chart version wrote the CR, because there is no second chart version to install from in this tree. Everything downstream of that write is exercised on a cluster. 3. **A reissue does not reach a running pod.** Neither service reloads its certificate, and cert-manager reissues on any change to `external`, the tenant host or the cluster domain, so an ordinary edit leaves the pods serving material that no longer matches their own names with no error, event or condition to show for it. Completing such a change is manual. So is removing the CA private key cert-manager leaves behind when TLS is turned off again, since the platform runs with `enableCertificateOwnerRef: false` and deleting a `Certificate` does not delete its `Secret`. `packages/apps/opensearch/README.md` carries both, with the commands. ### Downstream repositories `packages/apps/opensearch/values.schema.json` gains the `tls` object, so `cozystack/terraform-provider-cozystack` needs the field modelled for `cozystack_opensearch`; tracked in cozystack/terraform-provider-cozystack#41. ### Upgrade impact An existing `external: true` release with `tls.issuer` unset serves operator-issued HTTP TLS today and moves to chart-managed TLS on its first reconcile after this lands: the cluster rolls once and is re-anchored on a new per-release CA, so any client pinned to the operator's CA has to be repointed at the key-free `<release>.tenant-ca`. Setting `tls.issuer: operator` before the upgrade keeps that half as it is. Releases that are not external are untouched. Every existing release rolls once regardless of `tls.issuer` and `external`, because the pod template loses the operator's `init-sysctl` container; on a single-replica release that roll is the first one the operator can complete, since draining the only data node used to block every rolling restart. The published external-dns name changes on the same upgrade whatever `tls.issuer` says, because the Services are gated on `external` alone: they move from `<release>.<namespace>.<cluster-domain>`, an in-cluster name that never resolved outside the cluster, to `<release>.<tenant-host>` and `<release>-dashboards.<tenant-host>`. external-dns is configured upsert-only here, so the old record is not withdrawn and keeps pointing at the LoadBalancer until it is removed from the zone by hand. That cleanup is per zone rather than per release and nothing enumerates the records, so the README shows how to derive the list from the cluster. ### Release note ```release-note feat(opensearch)!: the OpenSearch HTTP API and Dashboards already served TLS, from the operator's own private CA and with a SAN built from cluster.local while the credentials Secret advertised the release on the platform domain; both certificates now come from a per-release cert-manager CA whose anchor the platform publishes into the tenant as <release>.tenant-ca. Any client that worked against the old endpoint was skipping verification, because no hostname it was handed could have matched that SAN. Existing external releases move from the operator's CA to the chart's on their first reconcile, and their published external-dns name moves from the in-cluster domain to <release>.<tenant-host> with the old record left in place; set tls.issuer: operator beforehand to opt out of the CA move. With dashboards.enabled the application name is limited to 41 characters, and to 32 when external is on as well; past either bound the chart fails the render and withholds every object in the release. A release already above a bound is not broken by the guard, it is told why it has not been upgrading: the Dashboards Service the operator builds from that name exceeds the 63-character limit for a DNS label, the operator returns that failure on every pass, and the reconcilers queued behind Dashboards, which are upgrade and restart and snapshot repository, have not run since the release was created. Set dashboards.enabled to false to drop the bound. The external Dashboards Service also selected labels the operator does not put on Dashboards pods, so it stood up with no endpoints; it selects opensearch.cluster.dashboards now. The drain of data nodes is now requested only with the data role on and more than two replicas, since at two the operator gives up on a full drain itself. The chart stops asking the operator for its privileged init-sysctl container, since the opensearch-operator package now sets vm.max_map_count from a DaemonSet, so every existing release rolls once on upgrade; single-replica releases no longer ask the operator to drain their only data node, which used to block every rolling restart ```	2026-09-11 13:02:47 +04:00
Arsolitt	85fa60a33f	feat(opensearch)!: issue the HTTP API certificate from cert-manager ... The HTTP listener and Dashboards ran on certificates the operator minted itself, so a client outside the cluster had no anchor it could obtain through the platform and no way to verify the endpoint it was handed. Issue the HTTP chain from cert-manager instead, and give the operator the same CA through spec.security.tls.http.caSecret so it signs the securityadmin admin certificate with it. Without that the admin certificate would keep the operator's own CA while the listener trusted cert-manager's, and securityadmin could no longer authenticate to apply users, roles and audit policy. tls.issuer names the authority rather than a switch, because the listener serves TLS under either value and only the signer changes. It is tri-state: an explicit value is obeyed, an absent one follows external, because a release nobody publishes has no third party to prove anything to. The cert-manager issuer needs OpenSearch 2.0.0 or newer, since the operator honours the HTTP CA only from that version and silently signs with its transport CA below it. An explicit choice under that floor fails the render and one resolved from external falls back to the operator, so asking for a verifiable certificate never quietly yields less than was asked, and turning on external access never breaks an upgrade. BREAKING CHANGE: an externally published release on OpenSearch 2.0.0 or newer serves the HTTP API from a cert-manager CA rather than the operator's own. A client pinned to the operator CA must move to the trust anchor the platform projects into the tenant. BREAKING CHANGE: the external-dns name of an externally published release moves from <release>.<namespace>.<cluster-domain> to <release>.<tenant-host>, and the Dashboards name with it. This half is gated on external alone and does not depend on the issuer. external-dns runs upsert-only here, so the old record stays in the zone until it is removed by hand. BREAKING CHANGE: with dashboards enabled the application name is limited to 41 characters, and to 32 when external access is also on. A release above either limit now fails to render rather than converging. Such a release was already stalled: the Dashboards Service built from its name exceeds the 63-character DNS label limit, and the operator returns that failure ahead of its upgrade, restart and snapshot-repository reconcilers, so it has not upgraded or rolled since it was created. Set dashboards.enabled to false to converge again, or recreate the application under a shorter name. Assisted-by: LLM Signed-off-by: Arsolitt <arsolitt@gmail.com> [lexfrei: reworded subject and body for the issuer field rename] Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-09-11 01:23:13 +03:00
Aleksei Sviridkin	4b6b1dc2da	fix(vm-instance): let complete resources replace the instance type (#4109) ... ## What this PR does Closes #2909. The matcher keyed off `instanceType`, the domain sizing keyed off `resources`, and neither side looked at the other, so the chart could render both. KubeVirt refuses that, and the two halves of its rule are different shapes: `validateCPU` conflicts on `domain.cpu.cores` and `domain.cpu.sockets` only when they are non-zero, while `validateMemory` conflicts on `domain.resources.requests.memory` whenever the key is present. `values.yaml` ships `instanceType: "u1.medium"`, so clearing the field in the dashboard just falls back to that default. Fill in cpu, sockets and memory after clearing it and you get a VM the webhook rejects, naming fields the form gives you no way to clear. Resources that size the VM completely now suppress the matcher and size it directly. `packages/apps/kubernetes-nodes` already does this for its worker VMs. Beside an instance type, sizing that is only partial stops the render, naming what sized the VM and what did not; with no instance type in play there is nothing to conflict with, so that half renders as it does today. Letting resources win unconditionally would silently drop the instance type for a VM that sets memory alone, which is a different bug in the same place. Completeness is decided from what `domainResources` emitted, never from which keys the user set, and all three fields go through that one reading. The two readings come apart in more than one place. `resources.cpu` on its own emits nothing, so it conflicts with nothing and goes on rendering a plain instancetype-sized VM. cpu and sockets pass through `int64`, which yields 0 for a quantity the schema accepts, so `cpu: 500m` beside a whole sockets and a memory would read as complete from the values, drop the matcher, and hand KubeVirt a `cores: 0` the user never asked for. And a `resources` block nulled outright leaves `.Values.resources` nil rather than the chart's `{}`, so a raw field read is a render-time nil pointer where the emitted object is merely empty. The existence check and the pre-upgrade update hook read the same effective value. Keying the `lookup` on the raw one would fail a resource-sized render over an instance type it never references, which is the same confusing error this fixes. Reading `.Values.instanceType` in the hook would see the leftover value, skip the removal patch, and leave `spec.instancetype.revisionName` on a VM Helm had just given explicit domain resources. ### What changes in the rendered output | `instanceType` | `resources` | Before | After | | --- | --- | --- | --- | | set | unset | matcher only | unchanged | | unset | cpu, sockets and memory | domain cpu and resources | unchanged | | set | cpu, sockets and memory | matcher plus domain sizing, refused by KubeVirt | domain sizing, no matcher | | set | any subset with memory, or cpu with sockets | matcher plus a conflicting domain field, refused by KubeVirt | render stops, message names both sides | | set | cpu alone, or sockets alone | matcher only | unchanged | | set | cpu and sockets that both truncate to zero | matcher plus `cores: 0` | unchanged | | unset | anything short of all three | already refused by the existing check | unchanged | The rows that stay unchanged are deliberate. `virtual-machine.domainResources` writes `domain.cpu` only when cpu and sockets arrive together, so cpu alone emits no override at all. A `cores: 0` beside a matcher is not a conflict either, because `validateCPU` guards on non-zero, so KubeVirt admits it and the instance type sizes the VM. Both shapes render today and go on rendering. `cpuModel` is left alone for a related reason: it writes `domain.cpu.model`, which KubeVirt flags only when the instance type declares a model of its own, and no instance type shipped in `packages/system/kubevirt-instancetypes` declares one. ### Why no working release breaks Every shape that now fails at render was already being rejected by the KubeVirt validating webhook at admission, so it could not have produced a running VM in the first place. The failure moves from an opaque admission error to a render error that names the fields. Nothing that worked stops working. The unchanged rows above are measured. Each non-failing shape in the table, plus firmware on and off, an explicitly nulled `resources` block and the three update-hook branches, was rendered against a mocked cluster on both sides and compared document by document. Across 14 value shapes and 78 rendered documents, 77 come out byte-identical. The one that differs is the row the table marks as changed, which is also what the harness was checked against first, so it is known to be able to fail. ### Live-VM impact This changes a rendered VM spec, so an upgrade can resize or restart a running VM. Adding a complete `resources` block to a VM currently sized by an instance type drops its matcher, and the pre-upgrade hook patches `spec.instancetype` off the live object and writes the domain resources in its place. Cozystack runs KubeVirt with `vmRolloutStrategy: LiveUpdate` (`packages/system/kubevirt/templates/kubevirt-cr.yaml:125`), so KubeVirt applies what it can to the running VMI and marks the rest `RestartRequired`; I have not measured which half a given resize lands in, so plan the upgrade as if it restarts. Dropping the matcher drops the whole instance type, sizing and everything else it carries. Of the 69 shipped cluster instancetypes, 36 declare `dedicatedCPUPlacement`, 36 declare `hugepages` and 26 declare both. 20 declare NUMA passthrough, 8 declare `realtime`. A VM moved from `rt1.medium` to explicit resources loses those characteristics along with the type. Before this change that combination was rejected outright, so the VM never existed; now it is admitted, and the field help says so. ### Tests `tests/instancetype_resources_test.yaml` covers matcher only, resources only, both complete, each failing partial shape, the shapes that emit nothing, both cpu and sockets truncating to zero, `cpuModel` beside a matcher and beside complete resources, a stale instance type with and without complete resources, an explicit `instanceType: null`, an explicitly nulled `resources` block, and every branch of the update hook's instancetype selector, in both directions plus the two no-op cases. 24 cases in this package, 2127 across `make helm-unit-tests`, all green. Every case pinning new behaviour was checked against a reverted template to confirm it goes red; the rest pin behaviour this change deliberately leaves alone. A ten-mutation ladder over the helper, `vm.yaml` and the update hook leaves no survivor. The memory branch took the most work. Swapping its emitted read for a raw `.Values.resources.memory` looks like an equivalent mutant, since `domainResources` emits `resources.requests.memory` under the same truthiness condition. It is not: a `resources` block nulled outright leaves `.Values.resources` nil, and the raw read dereferences it. The case pinning that shape kills the mutant. The update-hook cases assert the `kubectl patch` command the Job will run, which is as far as a render-level test reaches. They do not show that KubeVirt accepts the resulting object, that helm-controller converges on a live VM whose `spec.instancetype` also carries a `revisionName` written by virt-controller, or which half of a resize lands as a live update rather than `RestartRequired`. That needs a cluster. The `dig` calls survive the shapes that reach them. `values.schema.json` rejects a non-object `resources` before render; Helm coalescing turns a nulled one into a nil `.Values.resources`, which digs to an empty object; and for any other non-map intermediate, current Helm turns the underlying sprig type assertion into a render error rather than a panic. ### Interaction with the firmware block `firmware` (UEFI/BIOS selection, Secure Boot, persistent EFI NVRAM) landed in this chart in the meantime, and it does not touch the sizing question. `domainResources` does not read it, so it takes no part in the completeness decision. KubeVirt's instancetype spec carries neither firmware nor `domain.features`, so nothing an instance type declares can conflict with what the firmware block renders. The boot mode a VM inherits when the block is empty comes from the preference, and the preference is not what gets dropped when complete resources replace the matcher. Both firmware states are covered on each side by the byte-identity comparison above. `README.md`, `values.schema.json`, `api/apps/v1alpha1/vminstance/types.go` and the ApplicationDefinition all come out of `make generate` in the package, not out of a hand merge. Re-running it on the result produces no further diff, and the root `make generate` reports no drift on top of it. ### On the ticket's second proposal #2909 also asks whether the non-empty `u1.medium` default belongs in the chart at all. That requirement is met, and not by removing the default. The default was a problem because a schema-generated form gave no way to say "no instance type": clearing the field falls back to `u1.medium`. Filling cpu, sockets and memory now says it directly, so the obstacle the ticket names is gone while the default stays. Dropping the default on top of that would change the rendered spec of every release that never set the field, with nothing left to gain. ### Named improvements, not done here The failure message names the fields that did not size the VM, computed from what was emitted, so for `cpu: 4` with no `sockets` it names `resources.cpu` even though the user set it. The message states the pair rule that explains this, but naming the actually-missing field first would read better. The two tests that pin `domain.cpu.cores: 0` document the pre-existing `int64` truncation in prose rather than through a `FIXME` reference, so a later reader has to read the comment to tell a contract from a bug we chose not to fix. Both are wording; neither changes behaviour. `resources.cpu`, `sockets` and `memory` are checked for shape and not for value, so `memory: "0"` and `-1Gi` pass the schema and fail further down. Both forms fail before and after this change and only the place of the failure moves, so that belongs with [#4104](https://github.com/cozystack/cozystack/issues/4104) rather than here. The new template comments run to about 46 lines against about 28 lines of logic. All of it is why rather than what and all of it is accurate, but the helper's third paragraph restates the commit body, and two sentences would carry that weight. The seven-line block in `vm.yaml` is the one worth keeping: swapping that gate to the effective value survives the entire suite, so what the block records is an equivalence no test can pin, and trimming it would delete the only record of it. The `instanceType` description runs to about 370 characters and reaches the dashboard as tooltip help, through `values.schema.json` and the ApplicationDefinition. That is long for a tooltip. It is accurate as it stands, and rewriting accurate text for length is a fresh unmeasured claim in the surface with the most readers, so it is not done here: the place for the detail about what a dropped instance type takes with it is the chart README, leaving the field help carrying the precedence rule alone. Choosing a `cx1.*` type and then giving it complete resources drops what the type guaranteed along with the type, and nothing says so at runtime. That is a consequence of the precedence rule rather than a defect in it, and the field help states it. A runtime signal would need the chart to compare what the type declares against what `resources` sets, which is its own task. Defaulting `sockets` to 1 would let a user size a VM from cpu and memory alone and would remove the commonest way to land in the partial case. It is more than the ticket asked for and it changes the rendered spec of stored releases that set cpu and memory without sockets, so it needs its own call on compatibility. ### Not changed Two pre-existing quirks stay as they are, because both reach past the conflict this fixes. `resources` fields are read for truthiness, so an explicit `0` reads as unset, in `domainResources` and in the existing "Either instanceType or resources" check alike. And with no instance type in play, a fractional `resources.cpu` still truncates to `cores: 0` and renders a VM sized by nothing, with no error. The new guard covers only the half where a matcher is present. The tests pin both as they stand, which is what keeps working releases rendering. ### Screenshots No dashboard code is touched. The only user-visible surface is two field descriptions the dashboard renders as help text, regenerated into `values.schema.json` and the ApplicationDefinition. `instanceType` now reads "Ignored when `resources` sizes the VM, which needs whole-number cpu and sockets plus memory; the whole type is dropped then, so any dedicated CPU placement, hugepages, NUMA or realtime settings it carries go with it. While this field is set, a `resources` block that sizes only part of the VM is rejected outright instead of ignored, so complete it or clear it." `resources` states the same rule from its own side. ### Downstream repositories - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: I walked the trigger map against the diff and no listed trigger fires: no package added, renamed or removed; no `values.schema.json` field added, renamed or removed; no default changed; nothing under `hack/` moved; `templates/service.yaml` and its `wholeIP` and `allowICMP` annotations untouched. Two repositories do carry prose this makes incomplete. I left every box empty rather than push follow-ups nobody agreed to, so a maintainer decides. `cozystack/website`: `content/en/docs/next/virtualization/resources.md` opens with "Each virtual machine has these two configuration settings" and lists only `instanceType` and `instanceProfile`. The page is hand-written, so the release-time bot that regenerates app reference pages from `README.md` will not reach it. `cozystack/terraform-provider-cozystack`: `internal/provider/vminstance_schema.go` describes `instance_type` as "Virtual machine instance type." Its `kubernetes_schema.go` and `kubernetes_nodes_schema.go` already spell out the equivalent precedence rule for their own charts. No schema field or default changed here, so its new-field test guard has nothing to catch. ### Release note ```release-note fix(vm-instance): a complete `resources` block (whole-number cpu and sockets, plus memory) now sizes the VM directly and suppresses the `instanceType` matcher, instead of rendering both and having KubeVirt refuse the VM. Beside an `instanceType`, a `resources` block that sizes the VM only in part now stops the render with a message naming both sides, where it previously produced a VM the admission webhook rejected. Note that suppressing the matcher drops the whole instance type, including any dedicated CPU placement, hugepages, NUMA or realtime settings it carries. A release whose values already carry both converts on the next platform upgrade rather than at a moment the operator picks: the pre-upgrade hook strips `spec.instancetype` off the live VM and writes the domain sizing in its place, so plan that upgrade as if the VM restarts. ```	2026-09-10 19:21:59 +04:00
scooby87	afc003c464	feat(postgres): database read-replica autoscaling via KEDA (#3954) ... ## What this PR does Adds automatic horizontal autoscaling of a PostgreSQL database's **read replicas** in response to load, implementing the merged design proposal [Database Horizontal Autoscaler](https://github.com/cozystack/community/tree/main/design-proposals/database-horizontal-autoscaling) (community#44). The mechanism is entirely stock — no bespoke operator, no new CRD: - **KEDA as an optional platform component** (`packages/system/keda`, wired into `core/platform` as a source + system-bundle entry). It claims the `external.metrics.k8s.io` APIService that nothing served before. - **`cozy-lib.keda.scaledObject` helper** (`packages/library/cozy-lib/templates/_keda.tpl`) renders a KEDA `ScaledObject`, fail-closed on missing/invalid arguments. - **`packages/apps/postgres` autoscaling** — an `autoscaling` values block (`enabled`, `minReplicas`, `maxReplicas`, `metric`, `target`, `maxReplicationLagSeconds`, `dryRun`, `transition`) validated by `values.schema.json`; under active autoscaling the chart renders a `ScaledObject` and seeds `spec.instances` to a constant floor (`max(replicas, effectiveMin)`), while the postgres release is forced to client-side (two-way) Helm apply so the seed is a merge no-op on upgrade and the KEDA-managed HPA — the runtime writer via the CNPG `Cluster` scale subresource — is not reverted. - **Single-value metric (§1)** — one PromQL query returns `Σ(read load over standbys) + target`, so a stock HPA computes `desired = ceil((Σ+target)/target) = 1 + ceil(Σ/target) = primary + desiredRead`, with no per-pod emission. - **The four static brakes (§5)** — quorum floor `max(minReplicas, maxSyncReplicas+1, 2)` (quorum wins over `maxReplicas`), scale-down pacing (`Pods/1/600s`), a replication-lag clamp folded into the query (freeze both directions while lag exceeds the threshold and the primary is writing, with `max_over_time` hysteresis), and dry-run (render paused). - **Two-phase enable/disable migration** via a `transition` sub-flag: phase 1 stages `spec.instances` to the live count and brings KEDA up with its HPA paused in both directions; flipping to phase 2 hands the runtime count to KEDA, and because the seed re-renders the same count the flip is a merge no-op with no dip. - **Observability** — a database-autoscaling Grafana dashboard and PrometheusRule alerts. **Live validation (dev cluster, CNPG 1.30.0).** The full path works end-to-end: the CNPG `Cluster` scale subresource serves `status.selector`, KEDA renders the HPA, the metric reaches it (`ScalingActive=ValidMetricFound`, `TARGETS 151/150`, `desiredReplicas=2 = 1 primary + 1 replica` — the §1 arithmetic), and the lag-clamp query returns the base value on live VictoriaMetrics. Two packaging fixes were needed and are included: KEDA must run on the cozystack `cozy.local` cluster domain (the upstream `cluster.local` default makes the metrics gRPC target NXDOMAIN and every HPA hangs on `FailedGetExternalMetric`), and the operator's client-go throttle is raised so a single-replica operator does not lose its leader lease under scale-subresource polling. Implements #3953. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff. Two repositories are affected; both are follow-ups to open once this lands (and after its dependency #3951 merges and the values surface is final), left unticked here for a maintainer to confirm rather than claimed with a speculative link: - **cozystack/terraform-provider-cozystack** — this adds the nested `autoscaling` block to `packages/apps/postgres/values.schema.json`. The provider hand-models each app's fields (schema + model + expand/flatten), so it needs the new block to expose it in Terraform. - **cozystack/website** — KEDA is a new platform component (`guides/platform-stack`), and the postgres reference page gains the `autoscaling` params. The managed-app reference is regenerated from `README.md` by the release bot; the platform-stack component list is hand-maintained. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: to open (new platform component + postgres autoscaling params) - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: to open (new `autoscaling` block in postgres values.schema.json) ### Depends on - #3951 — bump CloudNativePG to 1.30.0. The HPA can only drive the CNPG `Cluster` scale subresource on CNPG ≥ 1.28.4, where it serves `status.selector`; without it the HPA fails with `InvalidSelector`. This PR should not merge before #3951. ### Release note ```release-note feat(postgres): add optional horizontal autoscaling of read replicas via KEDA, configured through the new `autoscaling` values block; KEDA ships as a new optional platform component. Requires CloudNativePG >= 1.28.4 for the Cluster scale subresource selector. Note: every Postgres release now applies client-side (`release.cozystack.io/helm-server-side-apply: "false"`) so the autoscaler's live `/scale` count is not reverted by Helm; a kind-wide side effect is that chart-rendered fields on Postgres releases are no longer re-asserted by server-side-apply drift correction (driftDetection is unset), a documented trade-off with a per-instance apply strategy as follow-up. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added configurable PostgreSQL read-replica autoscaling with KEDA. * Added CPU and read-connection metrics, replica bounds, replication-lag safeguards, dry-run mode, and transition handling. * Added optional KEDA platform support, Grafana autoscaling dashboard, and Prometheus alerts. * **Documentation** * Clarified integer thresholds, prerequisites, configuration, and transition guidance. * **Bug Fixes** * Improved validation and actionable errors for unavailable KEDA or invalid settings. * **Tests** * Expanded coverage for autoscaling, observability, package selection, and validation. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-09-09 21:28:34 +03:00
Aleksei Sviridkin	59731429a8	fix(vm-instance): let complete resources replace the instance type ... KubeVirt refuses a VirtualMachine that carries an instancetype matcher and explicit domain sizing at once. validateCPU in pkg/instancetype/apply conflicts on domain.cpu.cores and domain.cpu.sockets when they are non-zero, and validateMemory conflicts on domain.resources.requests.memory whenever the key is present. The chart rendered both, because the matcher keyed off instanceType while the domain sizing keyed off resources, and neither side looked at the other. values.yaml ships instanceType "u1.medium", so clearing that field in the dashboard only falls back to the default. A user who then fills in cpu, sockets and memory gets a VM the webhook rejects, with an error naming fields the form gives them no way to clear. Resources that size the VM completely now replace the matcher, the same way the kubernetes-nodes chart sizes its worker VMs. Beside an instance type, sizing that is only partial stops the render, because there is then no safe reading: dropping the matcher would size the VM from fields the user never finished, and keeping it either loses to the webhook or silently discards what was typed. With no instance type in play there is nothing to conflict with, so that half is left as it renders today. Completeness is decided from what domainResources emitted, not from which keys the user set, and the two readings come apart. resources.cpu on its own emits nothing, so it conflicts with nothing and goes on rendering a plain instancetype-sized VM. cpu and sockets pass through int64, which yields 0 for a quantity the schema accepts, so a cpu of 500m beside a whole sockets and a memory would read as complete from the values, drop the matcher, and hand KubeVirt a cores of 0 the user never asked for. And a resources block nulled outright leaves .Values.resources nil, where a raw field read is a render-time nil pointer and the emitted object is merely empty. The existence check and the update hook read the same effective value. Keying the lookup on the raw one would fail a resource-sized render over an instance type it never references, which is the same confusing error this fixes. Reading .Values in the hook would see the leftover instanceType, skip the removal patch, and leave spec.instancetype.revisionName on a VM Helm had just given explicit domain resources. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-09-08 00:26:21 +03:00
Alexey Artamonov	678d22f209	docs(postgres): move the serverAddress namespace-pin note out of a template comment ... Closes IvanHunters' MINOR on scaledobject.yaml:97. The API-owner-gate marker was a Go template comment, stripped at render, so the ownership question and the pin's residual surface were recorded where nobody reads them again. State the constraint on the serverAddress field doc (propagated into values.schema.json, README and the cozyrds openAPISchema) including the accepted residual: tenant-root is always in the allow list because the derived default points there, so a tenant can still aim the operator at a root Service as a blind in-cluster probe (no response body reaches the tenant), which the egress NetworkPolicy does not narrow. The API-owner sign-off is tracked on the PR, not in a stripped comment. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>	2026-09-07 21:03:15 +03:00
Alexey Artamonov	07f1daa22d	docs(postgres): correct the replicas "restored on disable" claim ... Closes IvanHunters' MINOR on values.yaml:58. The replicas description (propagated into values.schema.json, README and the cozyrds openAPISchema) said the count is "restored on disable". Under client-side apply it is not: the two-way merge writes instances only when the rendered value changes, so a cluster KEDA grew past replicas stays there after autoscaling.enabled: false with no autoscaler left to reduce it — the sibling dryRun doc already states the true rule, so the two contradicted each other. Reword both the @param and the @typedef to say the live count is not auto-restored and that the operator lowers replicas to reclaim the grown capacity; regenerated the derived artifacts. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>	2026-09-07 21:01:30 +03:00
Andrei Kvapil	54621bc046	feat(migration): tenant self-service VM import from VMware vSphere ... Hidora's customers are leaving VMware now, and a tenant has to be able to import their own machines without a platform administrator. The chart approach could not express that: a tenant can neither create the Secret holding vCenter credentials nor name the VDDK image, so both had to come from somewhere they cannot reach. Two CRDs in a new `forklift.cozystack.io` group, split by lifecycle. A VMImportSource is a long-lived connection -- type, endpoint, credentials, readiness -- and a VMImportTask is the one-shot operation that names machines and produces VMDisks and VMInstances. The outputs carry no owner reference back to the task, so deleting a finished import leaves the machines alone. Konveyor Forklift does the transfer underneath; the controller renders its objects, mirrors its verdicts, and hands each finished volume into a VMDisk without copying it. What a live vCenter migration forced out, and no amount of source reading would have: - A guest copied verbatim keeps the drivers it had under VMware and no virtio, so on a virtio disk it does not fail the import -- it succeeds and then does not boot. Imported disks go on SATA. - VDDK connects straight to the ESXi host at the address vCenter advertises, which is routinely unreachable, and on the first cluster collided with the Service CIDR. `spec.hosts` redirects per host. - Firmware must be carried across or a UEFI guest imports as BIOS and never boots (stacked on #3002). - Deleting a task mid-transfer orphaned its DataVolume; a finalizer now removes the engine's volumes and leaves the outputs. The console gains a Migration section: connections are editable, and the VM field is a picker built from a machine list the controller publishes, since the aggregated API cannot reach the inventory and should not hold a source's credentials to try. Two general mechanisms fall out of that -- an option source may take an argument, and one named in a CRD annotation may reference a sibling field. Design: cozystack/community#62. Docs: cozystack/website#673. Signed-off-by: Andrei Kvapil <kvapss@gmail.com>	2026-09-07 17:03:44 +02:00
Aleksei Sviridkin	d000d02b99	feat(rabbitmq): add TLS support for AMQPS and management HTTPS (#2683) ... ## What this PR does Adds TLS for AMQP and the management API, and hands the tenant a key-free trust anchor instead of a private key. - A chart-authored cert-manager chain issues the broker certificate. The chart names its CA Secret in a `TenantProjection` sentinel; the CA-extraction controller reads `ca.crt` from that Secret and writes a key-free projection, stamping `internal.cozystack.io/tenant-ca` on the projection it owns. That label is what the ResourceDefinition selects. Nothing labels the CA Secret itself, which holds the CA private key and is excluded by name. - The leaf `<release>-tls` carries the broker's private key and reaches no tenant. Neither the dashboard Role nor the ResourceDefinition grants it, and the ResourceDefinition excludes it by name as well, so a label mis-stamped onto it later still does not get through. - SAN coverage now lists every resolvable form explicitly, including the bare `*.<release>-nodes` wildcard that the operator uses for `RABBITMQ_NODENAME`. ## TLS is opt-in here, unlike the sibling charts `tls.enabled` defaults to `false` and is not inferred from `external`. That diverges from nats, qdrant and kafka, which infer it, and the divergence is deliberate for a reason specific to this operator. Setting `spec.tls.secretName` makes the cluster operator rewrite the `port` key in the `<release>-default-user` Secret from 5672 to 5671, on the mere fact of TLS and regardless of whether the plaintext listener is still open. That Secret is the Service Binding the chart hands the tenant. A client with a hardcoded `:5672` keeps working. A client that reads the endpoint from the binding, which is the documented way, gets moved to the TLS port and must then speak TLS and be given a CA to verify against, neither of which it was configured for. Inferring TLS from `external` would fire all of that on an ordinary upgrade, with no configuration change, against exactly the clients that followed the documentation. Whether the other engines should follow is an epic-level question rather than this PR's to answer; for nats it is raised in #3348. ## The plaintext ports are still published, and this PR says so There is no operator-supported way to publish the TLS ports alone. `spec.override.service.spec.ports` is a strategic merge keyed on `port` against defaults the operator rebuilds every reconcile, so it adds and mutates entries but never subtracts one, and a TLS-only port list there would read as a guarantee while changing nothing. The chart renders no service override at all and says why at the callsite. `tls.disableNonTLSListeners` is offered as an explicit opt-in, and it refuses to render rather than failing silently in three situations. - Together with `users` or `vhosts`. The flag forces messaging-topology-operator onto TLS regardless of its own configuration, after which it verifies the broker against the system certificate pool, where a per-release CA cannot be. Every topology object bound to the release would stop reconciling, silently: already-created users keep working while password rotation and new users simply never apply. There is no working route left for it, so the chart refuses instead. Managing those objects as hand-written CRs is not a way out, because the operator takes TLS from the `RabbitmqCluster` and not from whoever created the CR, so manual objects break identically and without the render-time refusal. `rabbitmqctl` or a definitions import are. - While topology objects are still live in the namespace. See the next section. - With TLS off, where no listener of either kind would remain. Erlang distribution (25672) and epmd (4369) stay plaintext under every setting; the flag does not touch them, so `disableNonTLSListeners: true` is not "TLS everywhere". A third route, closing AMQP at the broker through `spec.rabbitmq.additionalConfig` while leaving the operator on plaintext management, has been suggested and is plausible, but the operator's sources are not vendored here and it was not verified. It is recorded as an open question rather than promised. ## Closing the listeners takes two upgrades, and the chart enforces it Clearing `users` and `vhosts` and setting `tls.disableNonTLSListeners: true` in one edit is the natural thing to try, and it wedges the objects being removed. The values-level refusal cannot catch it, because by then both maps are empty. Helm applies every object in the target release before pruning the ones the release dropped, so the broker closes plaintext and loses the `rabbitmq.com/operator-connection-uri` annotation while the topology objects still hold `deletion.finalizers.<plural>.rabbitmq.com`. Their operator then has to reach a management API it can only get to over a certificate it verifies against the system trust pool. The delete never completes, and the upgrade reports success over objects left in `Terminating`, with the users still live in the broker and `kubectl delete namespace` on that tenant hanging. The chart now reads the namespace rather than the values. When the flag is being introduced and any topology object is still there, the render refuses and names them, gives the two-upgrade order and the `kubectl get` that confirms nothing bound to this release is left, and says how to unwedge a release that is already stuck. The check keys on the live `RabbitmqCluster` as well, so it applies only to the transition: once the listeners are closed, an object a tenant creates afterwards is theirs to clean up rather than something that stops the release rendering. Eleven unit cases cover the firing and the silent paths. The set is wider than the three kinds this chart creates. All twelve kinds `messaging-topology-operator` drives through its shared `TopologyReconciler` carry that finalizer and share one deletion path, so a `Queue` or a `Shovel` written by hand against the release wedges exactly the same way, and a runbook step that checks only users, vhosts and permissions says the namespace is clear when it is not. The list is derived in one place and read from both the lookup and the `kubectl get` the refusal prints, and a unit test holds it against the operator's own CRDs so a bump cannot introduce a kind the guard has never heard of. `SuperStream` is the one CRD left out: it runs on its own reconciler with no finalizer, and the objects it owns are already on the list. An object that binds through `connectionSecret` instead of naming the cluster stays outside the guard, which the comment says. ## Breaking for metrics scrapers Enabling TLS moves the metrics endpoint from 15692 to 15691. A ServiceMonitor selecting the port by the name `prometheus` silently stops matching and the instance's metrics disappear with no error anywhere. Nothing in this repository scrapes RabbitMQ, so only user-defined monitoring is affected, silently, which is why it is stated here. ## Known limitations 1. Turning TLS back off leaves the raw CA material behind. The anchor itself is collected: the projection carries an owner reference to the `TenantProjection` sentinel, and Helm deletes the sentinel along with the rest of the chain. What survives is `<release>-ca` and `<release>-tls`, because cert-manager runs with `enableCertificateOwnerRef: false` and does not delete the Secrets it issued, so the README says to remove them by hand. #3349 asks the same question of the label-driven leg of the contract, which this chart does not use. 2. No e2e coverage for the TLS path. The one-upgrade wedge above is the case that most deserves it: install with TLS and users, upgrade to no users plus the flag, and assert the `User` is actually gone rather than merely marked for deletion. A template test cannot reach that. On certificate renewal, the leaf carries `duration: 8760h` and `renewBefore: 720h`, so cert-manager rewrites the Secret around day 335. The cluster operator mounts it as a projected volume with no `subPath`, so the kubelet refreshes the files in place, and the operator does not watch a user-supplied TLS Secret, so nothing rolls the pods. The last hop, the broker picking the new file up, rests on Erlang's `ssl_pem_cache` re-reading by mtime and has only been read from sources rather than exercised against a live broker. The merged qdrant chart ships the same duration and `rotationPolicy`, so this is a platform-wide property rather than something this PR introduces. ### Downstream repositories `packages/apps/rabbitmq/values.schema.json` gains the `tls` object, so `cozystack/terraform-provider-cozystack` needs the field modelled for `cozystack_rabbitmq`; tracked in cozystack/terraform-provider-cozystack#42. ### Release note ```release-note feat(rabbitmq): opt-in TLS for AMQP and the management API, with the CA published to tenants as a key-free trust anchor. Enabling TLS moves the metrics port from 15692 to 15691 (ServiceMonitors selecting the port named `prometheus` must be updated) and moves the `port` value in the `<release>-default-user` binding Secret from 5672 to 5671. Plaintext AMQP and management ports remain published; `tls.disableNonTLSListeners` closes them but also drops in-cluster plaintext clients, and must be enabled in a separate upgrade after the topology objects are gone. ```	2026-09-07 18:51:34 +04:00
myasnikovdaniil	94a202bc5a	test(e2e): run the merge-gating lanes on Talos containers (#4020) ... Replaces QEMU substrate with Talos containers on both lanes that gate a merge. srv1-srv3 run as containers, so tenant worker sits at L2 instead of L3. Measured in CI on commits where both substrates ran the same tree: | | qemu | container | |---|---|---| | `kubernetes-latest` / `-previous` | 1143-2040s, red on every vgif-absent runner | 466-809s, green in both strata | | whole job | | 24-49 min faster | Almost all of it is in Chainsaw. Install takes about half an hour either way. Node-join soft-red gate goes with it. It was a prosthetic for nested virt: a worker that registers in minutes elsewhere could miss any deadline this test can afford, and #3513 established the discriminator is host `kvm_amd` `vgif`, not anything about the product. Remove the nesting level and there is nothing left to tolerate, so `hack/e2e-node-join-soft-red.sh`, the marker and the `soft_red` outputs are gone from all four workflows. ### What moves off the per-PR path DRBD, and with it `replicated` StorageClass, its `Immediate` binding mode, tenant StorageClass propagation (applies only to remotely-accessible classes), and cozystack talos node image with its extensions. None of it is lost - `nightly.yaml` and `e2e-tag.yaml` still run the suite on QEMU, so all four keep nightly and release-candidate coverage. Live migration is not on that list because nothing tests it today anyway. One more belongs on the list and was missing from it: the nocloud disk build. `build-talos` no longer runs `make -C packages/core/talos talos-nocloud`, so `images/talos/profiles/nocloud.yaml` is compiled by `nightly.yaml` and by the tag build's `make assets` rather than by anything a PR runs, and `promote-rc.yaml` still hard-fails a promotion whose rc release has no `nocloud-amd64.raw.xz`. Nothing in that chain breaks, because the tag build supplies the asset as before. What a PR no longer catches is narrow: it still compiles `installer.yaml` through the same imager, and the two profiles are generated by the same script and differ only in `platform`, `kind`, `imageOptions` and `outFormat`. ### Product fixes Four, all found by measurement while building the lane, each on its own commit: The linstor `drbd.enabled` gate. Without it drbd-logger sidecar exits, satellite is never Ready, piraeus registers zero nodes and every PVC hangs Pending behind a cluster that looks healthy. linstor `--strict-topology`. Without it external-provisioner passes every topology segment as `requisite`, so a node-pinned volume can be provisioned away from its pod and the pod is then unschedulable forever with nothing erroring anywhere. kubevirt-cdi importer resources are configurable now. Default restates CDI's own values exactly so production behaviour does not move, and the lane raises the ceiling through its own Package. vm-disk binds standalone disks immediately. A VMDisk on `local` silently never populated, while the same disk on `replicated` did. ### Where the run stands The lane has gone green twice, most recently on `cc70cef7b` where `E2E (in-tree)` took 2h3m against its 215m cap with 45 Chainsaw tests passing. The measurements quoted elsewhere in this description come from the first of those runs, on a commit that has since been rebased away: the two tenant-Kubernetes suites came in at 569.72s (`kubernetes-latest`) and 470.19s (`kubernetes-previous`) against a 67m operation. Both runs answer the four earlier reds, all of which were the lua-protobuf segfault from #4012 once `vminstance`/`vmdisk` were fixed on the branch. ghcr.io mirror removal is carried here too and is the same change as #4007, whichever lands first makes the other a no-op. ### What review changed `drbd.enabled=false` now refuses to render together with `talos.enabled=false`. It removed the logger sidecar only, while the two DRBD initContainers piraeus contributes unconditionally are deleted by the Talos configuration alone, so that combination produced a satellite that can never reach Ready. `isp-full-generic` sets `talos.enabled=false`, so it was reachable and not theoretical. vm-disk keeps unconditional immediate binding, a standalone disk has no consumer to wait for. What it costs on a node-pinned class is in the chart README now, and the annotation is pinned per source type and on the existing-DataVolume path where `helm upgrade` retrofits it. The rest is the harness holding its own promises: HelmRelease gate reads are bounded and its minimum-count half is pinned by a test that fails without it, two Chainsaw ops sit above the deadlines they contain, the fallback-StorageClass render restores `replicated` when it fails, container lane compose project and pool files are per-checkout, and three comments that said the opposite of the code now say what is true (QEMU is still wired into nightly and e2e-tag, the lane replaces two Packages and not one, the argument-count assertion cannot see upstream drift). ### Release note ```release-note fix(linstor): CSI provisioning is constrained to the node the scheduler chose (`--strict-topology`), and the DRBD sidecar can be switched off with `drbd.enabled` on substrates whose kernel cannot run DRBD. On an existing cluster the new default re-templates `spec.csiController.podTemplate`, so piraeus rolls `linstor-csi-controller` once during the upgrade. `drbd.enabled=false` requires `talos.enabled=true` and the chart now refuses the other combination at render time, because only the Talos satellite configuration removes the DRBD initContainers. fix(kubevirt-cdi): CDI worker pod resources are configurable through `importerResources`, which reaches the importer, the uploader and the host-assisted cloner. The default restates CDI's own built-in values exactly, so an install that sets nothing behaves as before. fix(vm-disk): every disk requests immediate binding, not only `upload` sources, so a standalone disk on a WaitForFirstConsumer StorageClass is populated instead of waiting for a consumer that may never arrive. On a node-pinned class such as `local` the volume then binds where CDI's worker pod was scheduled rather than where the VM will run; see the chart README before choosing such a class for VM disks. The default `replicated` is unaffected. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added container-based end-to-end testing with local storage support. * Added optional backup-access preflight checks for database backup examples. * Added configurable CDI worker resources and LINSTOR storage settings. * Added configurable immediate storage binding for VM disks. * Added stronger readiness checks, diagnostics, and parallel test execution. * **Bug Fixes** * Improved cleanup failure reporting, capacity validation, and missing-image detection. * Improved FoundationDB health verification and storage placement behavior. * **Removed** * Removed GHCR mirror support and soft-red node-join tolerance. * Removed standalone OIDC render test suites. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-09-07 19:33:20 +05:00
Myasnikov Daniil	a115745c69	feat(vm-disk): let the operator defer binding to the consumer ... Immediate binding for every source fixed a real bug: a standalone disk on a WaitForFirstConsumer class has no consumer to wait for, so it never populated. It also took away the path that is correct for the other case. For a disk a VMInstance consumes, KubeVirt renders a temporary launcher pod carrying the VM's own affinity and nodeSelector, so the volume binds where the VM can actually run; with the annotation always present the CDI worker becomes the first consumer instead and the volume lands where that pod happened to be scheduled. On a node-pinned class the VM is then unschedulable against a node-affinity conflict, and storageClass is immutable, so the only exit is deleting the disk. The two cases want opposite defaults and only the operator knows which one a given disk is. bindImmediately defaults to true, so nothing moves for existing disks or for anyone on `replicated`, which binds Immediate either way. Both branches are pinned in tests, including the existing-DataVolume path, which renders its own annotation block. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Assisted-By: LLM	2026-09-07 16:27:24 +05:00
Alexey Artamonov	c693ed36fb	fix(postgres): force client-side apply so the autoscaler seed survives ... Under active read-replica autoscaling the chart seeds spec.instances at a constant max(replicas, effectiveMin) rather than omitting it (an omitted field defaults to 1, which the CNPG quorum webhook rejects, and — under server-side apply — the phase-2 omit prunes the field whenever KEDA has not yet written /scale, collapsing an idle-at-floor cluster to 1 and shedding its replica PVCs). Keeping the constant only works if Flux does not force-own and revert it. The postgres ApplicationDefinition therefore carries release.cozystack.io/helm-server-side-apply: "false", so the tenant HelmRelease applies client-side: helm-controller patches the CNPG Cluster from the previous-vs-new rendered manifest without consulting live state, the unchanged constant yields no patch, and KEDA's higher live /scale count is preserved. The constant also equals the disabled/dryRun/transition render (instances: replicas) whenever replicas sits at/above the floor, so those flips are patch-free too. Correct the merge-semantics comments across db.yaml, _autoscaling.tpl and values.yaml to describe the two-way previous-vs-new patch precisely (not a "three-way no-op"), and document the residual floor-change hazard: editing replicas / minReplicas / quorum.maxSyncReplicas moves the seed, so that one reconcile does patch spec.instances to the new seed. Add autoscaling.serverAddress so a monitoring:true tenant can point the ScaledObject at a vmselect that carries both the CNPG metrics and kube_pod_labels. Regenerated README/schema/types/cozyrds. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>	2026-09-04 23:19:25 +03:00
Alexey Artamonov	826f08972f	fix(apps/postgres): make autoscaler fail-safe, dip-free, and observable ... Address review findings on the KEDA read-replica autoscaler where the implementation diverged from the merged design proposal. Fail-safe read load. Gate the `or vector(0)` floor on the existence of ANY CNPG instance pod (`count(kube_pod_labels{...instance_role=~".+"}) > 0`) — primary or replica — so the query is empty ONLY when no CNPG pod labels exist at all (a broken kube-state-metrics pipeline), not during initial provisioning when the sole replica-side pod is CNPG's join Job (jobRole, not instanceRole). Also pin `ignoreNullValues: "false"` on the trigger: KEDA's prometheus scaler defaults it true, turning an EMPTY result into metric 0 (not an error) — the HPA would then desire 0 and clamp to the floor, i.e. silently scale a broken-pipeline cluster DOWN. With it false an empty result is a scaler error, so the HPA HOLDS and DatabaseAutoscalerScalerErrors fires; gating on any instance role keeps a healthy cluster (including a long basebackup) from tripping that alert. Constant seed = max(replicas, effectiveMin), so enable/dryRun/transition flips are a merge no-op with no dip and never undercut the quorum floor. Fail closed when the quorum floor exceeds maxReplicas (was: effectiveMax silently raised both bounds, unbounded for a large maxSyncReplicas). Dry-run / transition keep the HPA (unidirectional pause pair, not the HPA-deleting full pause) but both SelectPolicy=Disabled, so desired normalizes to current — the recommendation is read from the read-load metric, not a diverging desired count. Observability. DatabaseAutoscalerAtMax keys on desired >= maxReplicas, NOT the ScalingLimited condition (which is also true at the floor, so it fired on every idle cluster with wrong "raise maxReplicas" advice). Dashboard read-load panels are per-cluster (label_cnpg_io_cluster) with a CPU panel for ReadCPUUtilization and the exact clamp in descriptions; ScaleStuck's Pending intersection is scoped to CNPG pods; alerts.yaml narrative comments moved to Go-template comments so they do not ship into the PrometheusRule. Guard the seeding invariant: TestBuildHelmReleaseSpecNoForceOrDriftDetection. Regenerated README/schema/types. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>	2026-09-04 23:19:25 +03:00
Alexey Artamonov	6e3da63586	fix(postgres): guard the disable-wedge, role-scope the freeze count, and harden edges ... Address the remaining IvanHunters review points on #3954: - [MAJOR] Disabling autoscaling wedged the HelmRelease under synchronous quorum. An autoscaled + maxSyncReplicas>=1 cluster runs with instances seeded from effectiveMin while replicas stays at its default, so flipping autoscaling off rendered instances: <replicas> next to the quorum and CNPG's webhook rejected it ("maxSyncReplicas must be lower than the number of instances"). Add a symmetric render-time guard on the static-count path (covers off/dryRun/ transition): fail with an actionable message when replicas <= maxSyncReplicas. - [MINOR] Role-scope the freeze brake's currentInstances count (label_cnpg_io_instance_role=~".+") so lingering CNPG join/bootstrap Job pods (cnpg.io/cluster but jobRole, not instanceRole) don't inflate it and nudge desired up while the brake is meant to hold it. - [MINOR] Default maxReplicationLagSeconds to 0 (brake opt-in): only the non-braking path is validated live, so ship the validated default and let the freeze be enabled explicitly until its branch is exercised under load. - [MINOR] Document the floor-change and non-quorum-disable dips (same three-way- merge hazard) and the pin-replicas-first mitigation. - [NIT] Coerce the cozy-lib helper's `paused` (eq "true") so a caller passing the string "false" does not pause the ScaledObject. - [NIT] Trim the keda.clusterDomain injection comment so it no longer ships ~8 verbatim lines into the cozystack-values Secret on every cluster. Regenerate types/schema/README/postgres-rd. New tests cover the disable-wedge guard, the role-scoped freeze count, and the string-"false" paused coercion. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>	2026-09-04 23:19:25 +03:00
Alexey Artamonov	842a59a957	fix(postgres): correct the migration-dip docs and pin the KEDA metricType ... Two review findings: - The enablement note claimed the two-phase `transition` flow avoids the count dip when enabling autoscaling on an existing cluster. It does not: helm-controller upgrades with Helm's three-way merge (not SSA), which keys off the old-vs-new manifest diff, not KEDA's live /scale value. At phase 2 the chart stops rendering the staged static count, so the field is deleted (no quorum -> CNPG defaults to 1, a brief collapse) or changed (quorum -> rebased to effectiveMin), then the warm KEDA restores it. Rewrite the docs to state the flow reduces (KEDA is pre-warmed, no cold-start) but does not eliminate the dip, and spell out the suspend + scale-subresource pin that actually makes the handoff dip-free. This is the SSA-vs-three-way-merge handoff the design proposal left for the PoC to settle. - Pin metricType: AverageValue on the Prometheus trigger in cozy-lib.keda.scaledObject. The desired = ceil(query / threshold) identity holds only for an AverageValue external metric; it was relying on KEDA's default. Make it explicit and assert it so a re-vendor or default change cannot silently break the arithmetic. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>	2026-09-04 23:18:47 +03:00