717 Commits

Include renames

Author	SHA1	Message	Date
Myasnikov Daniil	47e24f01d1	fix(platform): hand seaweedfs-db over for every instance name, not just the default ... Migration 43 performs the v1.5.0 db-split hand-over: it re-owns Cluster/seaweedfs-db from <name>-system to the new <name>-db release and stamps helm.sh/resource-policy: keep, so the <name>-system upgrade — whose post-split chart no longer renders the Cluster — does not prune it as a removed resource. It compared the owning release name against the literal "seaweedfs-system". SeaweedFS is a user-creatable kind, so an instance named `foo` is owned by `foo-system` and was silently skipped: no re-own, no keep. The <name>-system upgrade then deleted the Cluster, CNPG took its PVC with it, and the filer metadata — the index for every object in that tenant's S3 — was destroyed. Data loss, not an outage: the volume PVCs still hold the bytes, nothing can find them. Verified live on the upgrade stand, which is a natural control group. All five tenants sit in the SAME failed-upgrade loop (<name>-system last-deployed is rev 1, a pre-split revision whose manifest still contains the Cluster, so every retry recomputes the same deletion): tenant-root/dsplit/l instance `seaweedfs` owner=seaweedfs-db keep Cluster alive tenant-named instance `foo` no Cluster at all Cluster GONE The only difference is that migration 43 fired for the default-named instances and skipped `foo`. That also answers whether `keep` is sufficient: three tenants survive the identical prune loop with it. Correction to the reported mechanism: this is the 1.4->1.5 db split (PR #2601, v1.5.0), not a 1.5->1.6 change — v1.5.3 and main are identical on every db-split file. And the split does protect the object; it just only protected the instances someone happened to name `seaweedfs`. - lib/seaweedfs-db-adopt.sh: shared hand-over matching the `-system` SUFFIX, so `foo-system` -> `foo-db` exactly as `seaweedfs-system` -> `seaweedfs-db`. Idempotent, and refuses a release named literally `-system` rather than annotating an owner of `-db` that no release would claim. - Migration 43 sources it: covers clusters upgrading from before 43 (1.4.x). - Migration 53 (new) re-runs it: migrations never re-run, so any cluster already at >= 44 ran the hardcoded version and is still exposed. As a pre-upgrade hook it lands before <name>-system re-renders, closing the window on THIS upgrade. This is the one that matters for 1.5.x -> 1.6. - targetVersion 53 -> 54. Tests fail on unfixed code: the non-default-name case produces no ANNOTATE at all against the hardcoded 43, while the default-name case passes — which is precisely the bug's shape. They drive the real migration scripts against a fake kubectl, following hack/migration-50-etcd-adopt.bats. LIMIT, stated plainly: a Cluster that is already deleted cannot be recovered by either migration — there is nothing left to re-annotate and the PVC went with it. Such a tenant needs seaweedfs-db restored from a backup. Runbook Step 0 adds the ownership audit that distinguishes at-risk from already-lost, and records that <name>-db can report Ready while its Cluster is gone (it rendered fine; a later <name>-system prune removed it and Flux has not re-checked) — so the audit trusts `kubectl get cluster`, not the HelmRelease status. Review round 2 — two ways this still lost the database: - OWNERSHIP IS NOT SAFETY. The helper skipped any Cluster already owned by <name>-db, treating ownership as proof the hand-over was done. It is not: where the hand-over was skipped, <name>-system prunes the Cluster and <name>-db RECREATES it under its own ownership with NO keep, while <name>-system's prune baseline still lists it — so the next reconcile deletes it again. That is the observed delete/recreate loop, and the skip walked straight past it. Verified on the stand: tenant-l and tenant-root are seaweedfs-db-owned AND their <name>-system deployed revision (rev 1) still contains the Cluster, so only keep saves them; tenant-fresh is seaweedfs-db-owned without keep but was installed after the split, so its baseline never had one. Telling those apart needs the release's deployed manifest, which this script cannot read cheaply or reliably. The costs are asymmetric — a needless keep leaves an orphan on delete (now reclaimed by the cleanup hook), a missing one loses the database — so keep is stamped on every Cluster owned by either side of the split. - FAIL OPEN, THEN STAMP. `set -euo pipefail` does not abort on a failing command substitution in a `for` word-list: the loop ran zero times, the script continued, and stamp_cozystack_version ran anyway. Migrations never re-run, so one throttle, RBAC hiccup, or not-yet-established CNPG apiservice behind the pre-upgrade hook permanently left every at-risk tenant exposed with no later migration to catch them. Reproduced: against the old helper a failing fleet scan still logged STAMP 54. Now every kubectl failure is fatal EXCEPT the two that genuinely mean "nothing to do" — the CNPG resource type not being served (the fail-open that IS load-bearing: a cluster without CNPG must still upgrade) and a Cluster vanishing between scan and read. An unreadable annotation is no longer indistinguishable from an absent one, and the unowned case now warns instead of passing silently. - ORPHAN ON DELETE. `keep` is permanent (removal deferred to the 1.7 batch migrations) and also survives the <name>-db release's own uninstall, so `kubectl delete seaweedfs <name>` left the CNPG Cluster, its Postgres pods and its PVCs behind — the cleanup hook selects app.kubernetes.io/instance=<name>-system, which the db PVCs (cnpg.io/cluster=seaweedfs-db) do not carry. Migration 53 widened that from pre-1.5 default-named tenants to every instance name, so the hook now deletes the Cluster explicitly; its PVCs carry ownerReferences to it and follow. - Tests assert the stamped VERSION (44 / 54), not a bare "STAMP": a wrong number would loop run-migrations.sh forever and the old assertion could not tell. Review round 3 — the reclaim introduced a data-loss path of its own: - DELETING ONE APP DELETED ANOTHER APP'S DATABASE. The db chart hardcodes the Cluster name `seaweedfs-db` for every instance, and the reclaim was scoped by that name in the namespace rather than by what the release owns. templates/seaweedfs-db.yaml gates the <name>-db HelmRelease on `topology != "Client"`; the cleanup hook had no such gate. Verified by render: `helm template s3-remote . --set topology=Client` emits ZERO s3-remote-db HelmReleases yet still emitted the Role granting delete on clusters/seaweedfs-db AND `kubectl delete cluster.postgresql.cnpg.io -n tenant-root seaweedfs-db`. So in a namespace running a server instance plus a Client, deleting the CLIENT destroyed the SERVER's filer metadata — CNPG takes the PVC via ownerReferences — for an app the operator never touched. Reachable the same way via a second Simple instance, whose <name>-db loses Helm's ownership check but which still installs and still carries the hook. main's hook was instance-scoped; this was new, and mine. Now gated exactly as seaweedfs-db.yaml is, and the Job re-checks meta.helm.sh/release-name == <release>-db at run time before deleting. The gate stops the permission being granted where it can never be legitimate; the runtime check stops the delete where it can. - `|| echo` turned any API/authz/admission failure into exit zero, so backoffLimit never retried and hook-delete-policy removed the only cleanup Job while the keep-protected Cluster stayed orphaned. A failed reclaim now fails the Job. - TEST SHELL. The migrations run under /bin/sh = busybox ash (the image is FROM alpine, run-migrations.sh is #!/bin/sh), but the tests drove them with bash — so the fail-closed guarantees were asserted in a shell that never runs them, which matters more than usual given the bug they cover was a fail-open. The scripts are now invoked via `sh` and the fake kubectl is POSIX sh. Verified directly against the production base image (busybox 1.37.0 / alpine:3.24): `set -o pipefail` is supported, every script passes `ash -n`, and the failing-fleet-scan path exits non-zero without stamping under real ash. The recipe is in the bats header. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-07-20 15:40:48 +05:00
Myasnikov Daniil	efa85a4f61	feat(platform): make the etcd-adoption skip-backup hatch reachable ... Migration 50 has honoured ETCD_ADOPT_SKIP_BACKUP since it was written, and its failure paths tell operators to "re-run with ETCD_ADOPT_SKIP_BACKUP=1". But migration-hook.yaml passes only NAMESPACE/CURRENT_VERSION/ TARGET_VERSION, and the Job is a Helm hook the chart re-renders, so there has never been a way to set it. The script advertised an escape the platform made impossible. The endpoint-classification work landing alongside this makes that worse: it adds refuse-rather-than-guess paths, so more clusters can now legitimately stop here. Plumb it through migrations.etcdAdoptSkipBackup, default false. Skipping the pre-adoption snapshot rewires ownership of LIVE etcd storage with no way back, so it stays opt-in and deliberate — the flag is the last resort for a cluster that has nothing to fix, not the first response to a refusal. Render "1"/"0" rather than the bare bool. The script matches the literal string "1", so a bool would render "true", sail through the Job spec looking correct, and be ignored — a safety valve that is set but silently disregarded is worse than one that was never offered, because the operator believes they opted in. The value is normalised through a string, so a bool, a quoted "true" out of the Package CR's Values JSON, or a numeric 1 all land on the same answer, while nil or garbage resolves to "0": taking this hatch must require an affirmative spelling, never a typo. Resolved with `dig` so a Platform Package predating the key renders "0" instead of failing the render of the entire platform chart and bricking every upgrade. Widen the script's own check to accept 1/true/yes as well. The template guarantees the GitOps path, but an operator running this image by hand will reasonably type "true", and that is exactly the moment they are already stuck. The two layers cover independent failure modes. Emit the var unconditionally, including the "0" default, so the rendered Job states the cluster's safety posture outright rather than leaving an operator to diff values to find out whether a snapshot was skipped. Point the failure messages at the values path that actually exists, and describe it honestly: editing the Platform Package and letting Flux re-render is a real operation, not a one-liner, and the flag has to be reverted afterwards or the next etcd migration skips its snapshot too. Fixing the condition the migration names remains the recommended path — the migration is idempotent, so re-running after a fix is safe. templates/migration-hook.yaml had no coverage at all, because the Job only renders when a lookup of the cozystack-version ConfigMap reports a version below targetVersion, and lookup is nil under helm template. The new suite mocks that ConfigMap and asserts the RENDERED VALUE — enabled, disabled, key absent, stringified true, garbage, and the pre-existing three env vars — rather than merely that the variable exists, which is what let the trap above go unnoticed in the first place. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-07-20 11:29:09 +05:00
Myasnikov Daniil	0ba5337526	fix(migrations): key the etcd-adoption endpoint scheme off the claimed bucket ... Deriving the safety-snapshot target from the projected cozy-backups-creds Secret is right, but detecting the admin-managed external-S3 case (provisionBucket: false) by "the projected endpoint is absent" is not: that state does not exist. The projector substitutes BACKUP_STORAGE_ENDPOINT and fails ReasonSourceMalformed when both sources are empty, so a projected Secret always carries a non-empty, scheme-stripped endpoint. So for provisionBucket: false against a plaintext external S3 the projected endpoint is a bare host, the script saw it non-empty, forced https:// and never consulted the Strategy CR — a regression against main, which read the CR verbatim and passed http:// through. Being a pre-upgrade hook with no ETCD_ADOPT_SKIP_BACKUP plumbed through migration-hook.yaml, that is a total upgrade block with no operator escape. Classify on data instead: match the projected bucketName against the .status.bucketName of any BucketClaim. The COSI driver assigns that name and the projector republishes exactly it (packages/system/bucket user-credentials.yaml reads BucketInfo spec.bucketName), so a match means the creds describe a COSI bucket => the host is the always-TLS S3 ingress => force https, as the 1.6 chart helper does. No match means external S3. Keying on data rather than on the claim's name and namespace matters twice. backupStorage.namespace/.bucketName are supported Package-CR overrides that this hook cannot see, so a name lookup would miss a renamed bucket and fall back to the v1.5.x plaintext CR — the original P0. And a BucketClaim wedged Terminating (its cosi bucketclaim-protection finalizer outlives an uninstalled COSI controller) keeps answering a name lookup while its status names some other bucket. Split the two paths by which source is authoritative. On COSI the projected coordinates win: the bucket is the COSI-assigned name the CR can only reproduce through a live lookup that may never have run. On external S3 the Strategy CR wins WHOLESALE, not just for the endpoint — the CR renders backupStorage.* and points at this same cozy-backups-creds, so CR coordinates plus these credentials is exactly what the cluster's own BackupJobs already use, and the bucket an operator will look in. Keeping the Secret's bucket while taking the CR's endpoint assembled a pairing nothing else produces: the projector copies bucketName straight from the admin-managed source Secret and never from backupStorage.bucketName, so the two can disagree outright and the snapshot could land in a bucket the platform never writes. Relying on the CR is safe on this path alone, because bucketName short-circuits to values before the BucketClaim lookup that races on the COSI path, so the CR is guaranteed present. Match with a single jq `any`, never `jq -r | grep -q`: grep exits on the first match and SIGPIPEs jq, and pipefail reports jq's 141 rather than grep's 0, so a MATCH returns as a failure and the cluster is misclassified external. It is a race on the list outgrowing one stdio write (~8KB) with only a last-line match safe, so it strikes exactly the big multi-tenant clusters that can least afford it, and backoffLimit 3 can decide differently on each retry. Measured with the match first, 20 runs each: 300 claims went 0/20 correct (rc=141 every time) and 2000 likewise, while `any` is 20/20 at 3, 300 and 2000. Refuse rather than guess when the answer is not knowable. Three states qualify: the BucketClaim API is unreadable; the list is not interpretable (no producer emits that, but without a shape gate it reads as "no claim" = external = the plaintext P0, silently); or the bucket is claimed ONLY by a Terminating claim. That last one is ambiguous — an admin who moved COSI -> external S3 reusing the bucket name, versus a still-COSI cluster whose claim is mid-recreate — and both readings fail as the same confusing handshake error when wrong, so a guess buys no reliability while reading it as "external" specifically resurrects the plaintext-:8333 P0. A live claim still wins over an unrelated Terminating one, so healthy clusters are unaffected. Each refusal names the condition and the operator action; the migration is idempotent, so re-running after resolving it is the remedy. Behaviour change worth noting: a cluster on external S3 that still has COSI installed now refuses on a sustained BucketClaim-API failure where main would have proceeded off the Strategy CR. That is the intended trade — main proceeded only because it never had to tell the two apart — but it does mean an apiserver outage can now block this hook where it previously did not. ETCD_ADOPT_SKIP_BACKUP is still not plumbed through migration-hook.yaml, so the messages deliberately point at resolving the condition rather than at an escape hatch the operator cannot reach. Force the scheme only onto a non-empty host: an absent endpoint must degrade to the CR as main did, not to a literal "https://" that would pass the final non-empty guard and reach etcd-migrate as a destination. The old test modelled external S3 by dropping the projected endpoint entirely — a state the projector cannot produce — so it green-lit the regression. It now models what the projector really writes, and the suite pins the stale-claim, overridden-coordinates, empty-endpoint, 300-claim-multi-tenant, Terminating-match, hybrid-coordinate and unreadable/uninterpretable API cases. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-07-20 11:29:09 +05:00
Myasnikov Daniil	9ceccd2b07	fix(migrations): derive the etcd-adoption snapshot target from the projected bucket creds ... Migration 50 is a pre-upgrade hook, so every chart-rendered resource it reads still belongs to the version being upgraded FROM. It read the snapshot target from the live cozy-default-etcd Strategy CR, which on a v1.5.x cluster is broken in one of two ways — and the platform upgrade is gated behind this snapshot, so neither can self-heal first: 1. Absent. strategy-etcd-default.yaml is guarded by `if $bucketName`, whose helper looks up the BucketClaim status. That BucketClaim is created by the same chart, so on first render the lookup is empty and the Strategy is silently skipped. Helm lookup only runs at install/upgrade time, so a cluster that reached 1.5.x in one hop never grows the CR on reconcile. 2. Present, carrying the static v1.5.x default http://seaweedfs-s3.tenant-root.svc.cozy.local:8333. Cozystack ships SeaweedFS with enableSecurity=true, so :8333 is a TLS listener and plaintext against it fails the handshake. Fixing the scheme alone is not enough: the Etcd Strategy's S3 schema has no caCert/insecureSkipVerify, so the self-signed in-cluster endpoint is unusable at any scheme and a trusted-cert endpoint is required. Either way migration 50 hard-fails and blocks the upgrade, and the documented ETCD_ADOPT_SKIP_BACKUP escape hatch cannot be reached — migration-hook.yaml passes only NAMESPACE/CURRENT_VERSION/TARGET_VERSION. Take the coordinates from cozy-backups-creds instead. The projector writes it from the COSI-provisioned bucket's system credentials — bucket provisioning, not chart values — so it describes the live bucket regardless of which version rendered the charts, and it carries endpoint, bucketName, region and forcePathStyle alongside the AWS keys. resolve_platform_backup_args already read this Secret to validate those keys, so preferring it adds no new dependency. The endpoint is a bare host fronted by the always-TLS S3 ingress, so force https://, exactly as the 1.6 chart helper backupstrategy-controller.endpoint does. The Strategy CR remains the fallback for the admin-managed external-S3 case (provisionBucket: false), where .Values.backupStorage.endpoint is authoritative and may legitimately be plaintext against a private store; that path is detected by the absence of a projected endpoint and its scheme is preserved verbatim. The bats fixture modelled cozy-backups-creds as carrying only the AWS keys, and test 1 asserted the broken plaintext endpoint as expected output — the test was written against the implementation rather than the requirement, which is why the defect shipped green. The fixture now carries the coordinates the real projected Secret has, test 1 asserts the derived https endpoint, and two cases are added: an absent Strategy CR still resolving, and external S3 keeping the CR endpoint verbatim. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-07-20 11:29:09 +05:00
Aleksei Sviridkin	0238231303	fix(linstor-scheduler): re-vendor to upstream 0.3.1 with native admission webhook (#3201) ... ## What this PR does Re-vendors the `linstor-scheduler` chart from upstream **0.3.1** (appVersion **v0.3.6**). Upstream now ships the admission webhook natively (contributed via [piraeusdatastore/helm-charts#94](https://github.com/piraeusdatastore/helm-charts/pull/94)), so the loose downstream admission templates — which round-tripped every Pod through an old `k8s.io/api` and silently stripped fields they did not know about, the `image` volume source and the native-sidecar `restartPolicy` among them, breaking KubeVirt containerDisk VMs and native sidecars cluster-wide — are dropped in favour of the upstream copy. - The admission webhook comes from the upstream chart now; the previous downstream copy and its patch are removed. `make update` pulls 0.3.1 and applies only `cozystack-scheduler-deployment.patch` (the scheduler pod component label and the distro-suffix-stripped kube-scheduler image tag, neither of which is upstream yet), so vendoring stays deterministic and reproducible. - The upstream admission Deployment keeps the same name but changes its immutable pod selector, so an in-place upgrade of an existing install would fail with `field is immutable`. Migration 52 deletes the old admission Deployment before the upgrade so the new one is recreated cleanly (`targetVersion` 52 → 53). - appVersion v0.3.6 also carries the webhook's TLS certificate hot-reload fix ([piraeusdatastore/linstor-scheduler-extender#22](https://github.com/piraeusdatastore/linstor-scheduler-extender/pull/22)): the webhook used to load its certificate once at startup, so after the mounted Secret was rotated it kept serving a stale certificate and — with `failurePolicy: Ignore` — silently stopped mutating until it was restarted. It now reloads on rotation, so no external cert-reload workaround is needed here. - Wrapper values keep `replicaCount: 2`, now that the admission Pods have their own PodDisruptionBudget and no longer share the scheduler's. - Adds a helm-unittest suite covering the admission resources, PodDisruptionBudget selector isolation, least-privilege RBAC, the TLS provisioning modes and their rejection paths, the scheduler image-tag strip, and backward compatibility with the webhook disabled. Closes #3198. ### Screenshots N/A — no UI changes. ### Downstream repositories I walked the trigger map against the diff, file by file. It touches `packages/system/linstor-scheduler/` and, under `packages/core/platform/`, only migration 52 and the `migrations.targetVersion` bump — an internal, release-managed counter, not a user-facing `spec.components.platform.values.*` key documented on the website. No trigger matches. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(linstor-scheduler): the LINSTOR scheduler admission webhook no longer strips unknown Pod fields (the `image` volume source and native-sidecar `restartPolicy`), which had broken KubeVirt containerDisk VMs and native sidecars, and it now reloads its TLS certificate after rotation instead of silently failing. The chart is re-vendored from upstream 0.3.1; an automatic upgrade migration replaces the admission Deployment whose immutable selector changed. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Expanded the optional Linstor Scheduler admission webhook with richer configuration, including automated `schedulerName` mutation for LINSTOR CSI-backed PVC Pods. * **Improvements** * Updated the Linstor Scheduler Helm chart to **0.3.1** (scheduler app **v0.3.6**), including updated admission webhook behavior, safer defaults, and **2 replicas**. * Platform migrations now advance to **version 53**, with upgrade-safe cleanup of the existing admission deployment. * **Documentation** * Updated Helm README and post-install notes with detailed guidance for admission webhook and TLS modes. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-07-16 19:08:54 +03:00
Andrei Kvapil	fbbe4af7df	refactor(network): drop ExposureClass/ServiceExposure for native loadBalancerClass (#3218) ... ## What Removes the `network.cozystack.io` API group — `ExposureClass` + `ServiceExposure` (#3081) — and its reconciler, in favor of native `Service` `type: LoadBalancer` + `loadBalancerClass`. ## Why Full rationale and discussion in #3164. For **managed applications** the chart already owns the Service, so: - `type: LoadBalancer` + `loadBalancerClass` + an admin-provisioned pool covers vendor-neutral external exposure without a dedicated API group and controller; - the `externalIPs` (node-IP) default never went through `ServiceExposure` (its backend is a no-op) and is left unchanged; - the only current consumer, the host ingress, keeps rendering `type: LoadBalancer` and just drops the `ServiceExposure`; the admin provisions the address pool / announcer out of band, as the install docs already assume. `ServiceExposure`'s one unique feature — dynamic per-namespace scoping of a shared VIP pool — is a convenience the managed-app path does not need, and it does not justify a second tenant-networking API group next to `sdn.cozystack.io`. ## Changes - Delete `api/network/v1alpha1` and `internal/controller/serviceexposure` (+ CRD definitions, e2e bats, controller registration, RBAC). - Host ingress: drop the `ServiceExposure` template; add an optional `publishing.loadBalancerClass` that sets `spec.loadBalancerClass` on the host-ingress LoadBalancer Service. - The `externalIPs` path and default behavior are unchanged. ## Verification - `go build ./...` — green. - `helm unittest` — `core/platform` 72/72, `extra/ingress` 25/25 (incl. host-vs-tenant `loadBalancerClass` scoping and the immutable-field migration guard). ## Migration None required. `network.cozystack.io` (#3081) has never shipped in a release — it is on `main` only (not backported to `release-1.5`; absent from v1.4.x / v1.5.x), so no upgrading cluster has these CRDs and this removes it before it is ever released. Clusters that track `main` and explicitly set `publishing.exposureClass` (opt-in, only since 2026-06-30) should provision their MetalLB / Cilium address pool themselves and set `publishing.loadBalancerClass`; any orphaned `cozystack-<class>` pool can be deleted by hand. Default clusters (externalIPs path) are unaffected. ## Status **Draft / RFC for the #3164 discussion** — this removes an API group that landed only days ago, so nothing here should merge before the design call. cc @lexfrei (author of #3081) and @lllamnyp (#3164). Deferred: when Kubernetes actually removes `Service.spec.externalIPs`, the node-IP default migrates via a MetalLB `IPAddressPool` with no `L2Advertisement` (allocate-but-don't-announce) — a separate change. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added support for configuring the ingress host Service with a native `loadBalancerClass` value. * Expanded chart tests to verify class wiring and immutability behavior across install and upgrade scenarios. * **Bug Fixes** * Removed the legacy exposure-class/service-exposure flow and related controller behavior. * Updated templates and CRD packaging so the platform no longer renders or depends on those deprecated resources. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-07-16 17:44:28 +02:00
Aleksei Sviridkin	f4aef73053	fix(linstor-scheduler): re-vendor to upstream 0.3.1 with native admission webhook ... Re-vendor the linstor-scheduler chart from upstream 0.3.1 (appVersion v0.3.6), which carries the admission webhook binary that stopped stripping unknown Pod fields — the image VolumeSource and native-sidecar restartPolicy, which broke KubeVirt containerDisk VMs and native sidecars cluster-wide — and now reloads its TLS certificate on rotation instead of serving a stale one until restarted. The admission webhook ships in the upstream chart, so the previous downstream copy is dropped: `make update` pulls it directly and applies only cozystack-scheduler-deployment.patch (pod component label and a distro-suffix-stripped image tag), keeping the vendoring deterministic. Because the webhook now reloads its own certificate, the external cert-reload workaround is dropped. The upstream admission Deployment keeps the same name but changes its immutable pod selector, so an in-place upgrade of an existing install would fail. A migration deletes the old admission Deployment before the upgrade so the new one is recreated cleanly, and bumps the migration target version. Add a helm-unittest suite covering the admission resources, PDB selector isolation, least-privilege RBAC, the scheduler image tag, and backward compatibility. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-07-16 16:57:41 +03:00
myasnikovdaniil	f6a03e93f8	fix(etcd): complete v1alpha2 transition for in-cluster 1.5→1.6 upgrades (#3270) ... Consolidates the etcd `v1alpha2` transition fix for **in-cluster 1.5 → 1.6 upgrades** into a single PR, rebased on current `main`. Supersedes #3265 and #3261 — their commits are carried here (authorship preserved), so those PRs can be closed once this lands. Fresh installs use the new shapes directly and are unaffected, so fresh-install CI does not catch these; the 1.5 → 1.6 e2e upgrade path is the authoritative regression guard. ## What this PR does ### 1. Keep the legacy `etcd-headless` Service alive during adoption — `packages/extra/etcd` `etcd-migrate` adopts legacy clusters **in place**: the Pods keep their original `spec.subdomain: etcd-headless` and are dialed at `etcd-<i>.etcd-headless.<ns>.svc` until they roll onto the operator's native `<member>.etcd.<ns>.svc` domain. The v1alpha2 operator only creates the native `etcd` Service and the legacy `etcd-headless` Service is pruned, so those per-pod names stop resolving (`no such host`), `MemberList` fails, and `status.readyMembers` never populates — the `EtcdCluster` never goes `Ready` even though etcd is healthy and in quorum. We ship a chart-managed transitional headless `etcd-headless` Service (selector mirrors the operator's native `etcd` Service via `etcd-operator.cozystack.io/cluster`, `publishNotReadyAddresses: true`) — the DNS counterpart of the legacy `*.etcd-headless.<ns>.svc` SAN already kept for this window. Removable together with that SAN once members roll onto the native subdomain. ### 2. Survive the immutable controller-Deployment selector on upgrade — `packages/system/etcd-operator` (#3242) #2859 replaced the upstream etcd-operator chart with the cozystack-authored one, changing `Deployment.spec.selector.matchLabels`. `spec.selector` is immutable, so `helm upgrade` cannot patch the existing Deployment and the whole HelmRelease upgrade fails (`field is immutable`). A **pre-upgrade hook** (`templates/pre-upgrade-selector-fix.yaml`: ServiceAccount + Role + RoleBinding + Job) deletes the Deployment **only** when its live selector is the pre-1.6 one, so Helm recreates it cleanly. No-op when the selector already matches, never runs on fresh install. ### 3. Raise the operator's memory cold-start floor — `packages/system/etcd-operator` Steady-state working set is ~250Mi; the static `limits.memory: 128Mi` OOMKills a Pod that starts before the VPA admission webhook rewrites it. Raise the floor to `256Mi` (and VPA `minAllowed` to match) as defense in depth so the operator never depends on VPA timing to avoid crashing. ### 4. Make migration 50 (etcd adoption) robust in-cluster — `packages/core/platform/images/migrations/migrations/50` (was #3261) Exact server/peer cert-SAN match, Secret-gated wait on the adoption Secret, and in-cluster (IPv6-safe) kubeconfig handling, so the migration script drives the adoption reliably from inside the cluster. Covered by `hack/migration-50-etcd-adopt.bats`. ### 5. Hardening / review fixes (this PR's original scope) - **Hook `runAsUser: 65532`.** `pre-upgrade-selector-fix.yaml` set `runAsNonRoot: true` but no numeric `runAsUser`; `clastix/kubectl`'s image user is the non-numeric name `nonroot`, which the kubelet cannot verify against `runAsNonRoot` — so the hook Pod fails admission and silently blocks the very upgrade it exists to unblock. Adds `runAsUser: 65532`, matching the postgres-operator webhook-ready hook that runs the same image. - **Digest-pin the kubectl image.** The values comment claimed digest-pinning but shipped a floating `v1.32` tag. Reuses the digest postgres-operator vendors, adds the `renovate` annotation, and templates `repo:tag@digest`. - **Tests.** `etcd-operator/tests/selector-fix-hook_test.yaml` (hook wiring, weight ordering, namespaced least-privilege RBAC, numeric-non-root security context, digest-pinned image), `etcd-operator/tests/deployment_test.yaml` (256Mi cold-start floor), `extra/etcd/tests/etcd-cluster_test.yaml` (transitional `etcd-headless` Service). Each assertion was mutation-tested. ### 6. Derive the default S3 endpoint from the provisioned bucket — `packages/system/backupstrategy-controller` Derive the default S3 endpoint (and per-driver scheme / TLS / `secure_connection`) from the provisioned bucket Secret instead of requiring it to be hand-set, so etcd (and other) backup strategies get a working endpoint by default. Docs in `docs/operations/backup-classes.md`; covered by `tests/endpoint_form_test.yaml`. ### Verification - `helm unittest` green on current `main`: etcd-operator **18/18**, extra/etcd **18/18**, backupstrategy-controller **11/11**. - Behaviours 1 & 2 were reproduced and confirmed live on a 1.5.2 → 1.6.0-rc.1 adoption (3-node cluster) — recreating the `etcd-headless` Service took the adopted cluster to `readyMembers=3 / Available=True`. ```release-note fix(etcd): complete the v1alpha2 transition on in-cluster 1.5→1.6 upgrades — keep the legacy etcd-headless Service alive so adopted members stay resolvable, delete the pre-1.6 operator Deployment via a pre-upgrade hook to get past the immutable selector, raise the operator's memory floor so it does not OOM before the VPA scales it, and make the etcd adoption migration robust in-cluster. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Improved backup storage endpoint handling across supported backup drivers, including provisioned and external S3 storage. * Added compatibility support for legacy etcd pod discovery during migration. * Added an automated upgrade safeguard for etcd operator deployments. * **Bug Fixes** * Improved certificate SAN detection and etcd migration authentication. * Increased the etcd operator’s minimum startup memory to prevent early restarts. * **Documentation** * Clarified backup endpoint, TLS, and driver-specific behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-07-16 12:37:00 +05:00
Andrey Kolkov	47d239c519	fix(platform): address review on migration 50 (exact SAN match, Secret-gated wait, IPv6 kubeconfig) ... Review feedback on #3261 (gemini-code-assist, coderabbitai, myasnikovdaniil): - Exact SAN match. _san_present read the Certificate SANs as {.spec.dnsNames} (a bracketed JSON blob that `tr ','` cannot split) and matched with a substring `grep -qF`, so a wildcard that is a substring of a longer SAN (e.g. *.etcd.<ns>.svc inside *.etcd.<ns>.svc.cluster.local) could false-positive and skip the re-issue. Read {.spec.dnsNames[*]} (space separated) and match exactly with `grep -qxF`, matching the Secret-annotation branch. - Wait on the re-issued Secret, not the patched spec. The post-patch wait loop called _san_present, which returns as soon as the Certificate spec.dnsNames contains the wildcard -- but the patch just added it there, so the loop broke on the first iteration before cert-manager re-issued the Secret, and the "re-issued" log printed unconfirmed. Add _secret_has_san (Secret alt-names annotation only, same empty-read retry + exact match) and gate the wait on it. - IPv6-safe kubeconfig server. The synthesized kubeconfig used https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT, which is an invalid URL on an IPv6-only cluster (a bare IPv6 host must be bracketed). Use https://kubernetes.default.svc -- IP-family-agnostic, validated by the mounted SA CA, and the same server URL kubectl synthesizes in-cluster. Tests (the review blocker): extend the bats suite (and its fake kubectl) with two cases pinning these contracts -- a superstring dnsName must NOT skip the re-issue patch, and etcd-migrate must be invoked with a synthesized in-cluster kubeconfig whose server is kubernetes.default.svc (not the bare IPv6 host), authenticating via the SA token file + CA. Both the kubeconfig path and the SA directory are now env-overridable (ETCD_MIGRATE_KUBECONFIG, ETCD_ADOPT_SA_DIR) so the synthesis is exercisable off-cluster; production defaults are unchanged. Signed-off-by: Andrey Kolkov <androndo@gmail.com>	2026-07-15 19:11:35 +05:00
Andrey Kolkov	c2539ca935	fix(platform): make etcd v1alpha2 adoption (migration 50) robust in-cluster ... Migration 50 (etcd.aenix.io -> etcd-operator.cozystack.io/v1alpha2 adoption) had two defects that blocked every in-cluster 1.5 -> 1.6 upgrade on a cluster with an existing etcd: 1. Cert-SAN wait treated transient kubectl failures as "SAN absent". ensure_wildcard_sans checked/awaited the wildcard SAN with `kubectl get ... 2>/dev/null | grep`, so any transient GET failure (API discovery refresh, apiserver blip, throttling) produced an empty string indistinguishable from a genuine absence -> false miss, and the 120s wait never recovered. Replace the two ad-hoc checks with a _san_present helper that retries on an empty read (a real Certificate/Secret never has empty dnsNames/alt-names) and accepts the native wildcard from EITHER the issued Secret's cert-manager.io/alt-names annotation OR the Certificate spec.dnsNames (the source of truth for what cert-manager will issue). 2. etcd-migrate had no kubeconfig in-cluster. etcd-migrate only reads a kubeconfig file (-k/--kubeconfig, default /root/.kube/config) and, unlike kubectl, does not fall back to the mounted in-cluster ServiceAccount. The hook Job set no KUBECONFIG and passed no --kubeconfig, so both the dry-run and --apply aborted with "error building kubeconfig: stat /root/.kube/config: no such file". Synthesize an in-cluster kubeconfig from the mounted ServiceAccount and pass --kubeconfig to both etcd-migrate invocations. Verified end-to-end on a 1.5.2 -> 1.6.0-rc.1 upgrade: the adoption now completes in-place (pods never restarted, data intact) and the cluster reaches readyMembers=3 / Available=True. Refs: #3243, #3255 Signed-off-by: Andrey Kolkov <androndo@gmail.com>	2026-07-15 19:11:35 +05:00
IvanHunters	d4f59af624	chore(monitoring): sync PackageSource comment with removed Keycloak groups ... Update the oidc-variant comment on `packages/core/platform/sources/monitoring-application.yaml` so it no longer claims `spec.oidc.mode: System` renders three KeycloakRealmGroups. The dependsOn edge and the fail-close guard in `templates/grafana/oidc-keycloak.yaml` stay unchanged: the KeycloakClient + KeycloakClientScope path still needs the `v1.edp.epam.com` CRDs from `cozystack.keycloak-operator` to be registered before the monitoring chart reconciles. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>	2026-07-15 15:31:45 +03:00
Ivan Okhotnikov	b46543e5f2	fix(monitoring): defer platform GrafanaAdmin lever to a future phase ... The MAJOR finding from the cozy-review agent: the platform-side `oidc.grafanaAdmin: true` documented in the PR body and in docs/oidc-grafana.md never actually reached the platform Grafana's HelmRelease. `Package.spec.components["monitoring-system"].values.oidc.grafanaAdmin` was set in packages/core/platform/templates/bundles/system.yaml, but `monitoring-system` (and `monitoring`) have no `install:` block in packages/core/platform/sources/monitoring-application.yaml — only `monitoring-rd` does. PackageReconciler skips components without an `install:` block before ever reaching the assignment that copies `pkg.Spec.Components[].Values` onto `hr.Spec.Values` (internal/operator/package_reconciler.go around lines 197-201 and 292-293). The values-override dictionary was silently dropped, no HR carried `oidc.grafanaAdmin: true`, and the platform Grafana rendered with the chart default (`false`). The `bundles_monitoring_grafana_admin_test.yaml` unit test only asserted against the emitted Package CR spec — exactly the layer where the value still existed but was about to be discarded — so the broken propagation slipped past. Trim the broken promise from Phase 1. The chart-side helper (`monitoring.oidc.allowAssignGrafanaAdmin`) stays as-is so a future phase that routes the override through the tenant-root chart's monitoring HR values (or adds an `install:` block on the `monitoring-system` component) lands on a green chart-side contract. The helm-unittest that verifies chart-side emission stays too, but with a docstring calling out that end-to-end wiring is deferred. Everything else that promised the lever works — bundle line, docs, platform bundle test — is either removed or rewritten to match reality. Also picks up two MINOR findings from the same review: * Drop the misleading `secret-hash` annotation on the OIDC client-secret Secret. The comment claimed it "rotates the KeycloakClient CR and therefore the Grafana Deployment env when the client-secret changes", but nothing downstream watches an annotation on the Secret itself — no rotation is triggered. The `lookup`-based fallback already keeps the value stable across reconciles, so no rotation is even needed. Removing the dead annotation avoids planting a broken assumption for future readers. * Add `release.cozystack.io/helm-install-timeout: "15m"` on the monitoring cozyrds ApplicationDefinition. Under `mode: System` the chart now emits, on top of the existing Grafana + Alerta + VMCluster + CNPG stack, four additional Keycloak CRs whose reconciliation is asynchronous against an operator in `cozy-keycloak` (not the tenant namespace). Flux's default 5m install window was already tight for the pre-OIDC stack; 15m matches the precedent set by tenant-rd/kubernetes-rd cozyrds. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>	2026-07-15 15:31:45 +03:00
Ivan Okhotnikov	be52e80289	fix(monitoring): split monitoring-application PackageSource into default/oidc variants ... Follow-up to the previous self-review-fix commit: the unconditional `- cozystack.keycloak-operator` in dependsOn was itself a CRITICAL cold-install regression on any cluster where `authentication.oidc.enabled: false` (the default). The keycloak-operator Package is only emitted by `packages/core/platform/templates/bundles/system.yaml:112` when that flag is on — a hard dep against it from the default variant would leave `cozystack.monitoring-application` stuck on DependenciesNotReady forever, cascading a `cozystack-platform` HR timeout that hits every OIDC-off customer. Fix by mirroring the cozystack-engine variant split: - `default` variant keeps the baseline reconciliation edges (no keycloak-operator dep). - `oidc` variant additionally waits for `cozystack.keycloak-operator` so the v1.edp.epam.com CRDs are registered before the monitoring chart reconciles and the `.Capabilities.APIVersions.Has` guard in templates/grafana/oidc-keycloak.yaml can safely emit KeycloakClient/Scope/Groups. The bundle template swings between variants on `.Values.authentication.oidc.enabled`, mirroring the cozystack-engine invocation two lines above. The `monitoring-system` component override (oidc.grafanaAdmin: true) is preserved across both branches — the values field is a no-op when mode: None (no auth.generic_oauth block is emitted at all), so keeping it identical avoids two divergent code paths. Also widened the platform helm-unittest coverage to lock: - default variant does NOT depend on keycloak-operator (permanent DependenciesNotReady guard). - oidc variant DOES. - Bundle emits the right variant based on authentication.oidc.enabled AND keeps the GrafanaAdmin uplift on the monitoring-system component in either branch. 83 platform tests + 23 monitoring tests still green. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>	2026-07-15 15:31:44 +03:00
Ivan Okhotnikov	ffeb8e5e4f	fix(monitoring): close self-review blockers for the OIDC PR ... Address the two blocking-class findings from the cozy-review pass on this PR: (1) Missing PackageSource dependency on cozystack.keycloak-operator (MAJOR cold-install race). Without this edge, the monitoring chart can reconcile before the v1.edp.epam.com CRDs are registered on a fresh cluster — the `.Capabilities.APIVersions.Has` guard in templates/grafana/oidc-keycloak.yaml then silently drops all Keycloak-side objects for mode: System, and Grafana's auth.generic_oauth block points at a client Keycloak never provisions. Add the ordering edge, plus a sources_monitoring_application_dependson_test.yaml regression guard so a future edit cannot drop it silently. (2) allow_assign_grafana_admin driven by `.Release.Name == monitoring-system` (MINOR fragile coupling). Replace the release-name sniff with an explicit `oidc.grafanaAdmin: false` values field, and flip it on for the platform Grafana release via Package.spec.components["monitoring-system"].values.oidc.grafanaAdmin in packages/core/platform/templates/bundles/system.yaml — mirroring the pattern monitoring-agents already uses for per-component overrides. Tenant Grafana releases inherit the chart default false and stay at org-level Admin. Two new helm-unittest cases in packages/system/monitoring/tests/ lock the values-driven behaviour on both sides (grafanaAdmin=true → Admin uplift, default → tenant stays at org Admin), plus bundles_monitoring_grafana_admin_test.yaml in packages/core/platform/tests/ regression-guards the Package's component override so the platform release keeps its uplift on future edits. Also document the mode-toggle group-membership loss in docs/oidc-grafana.md — mode: System → CustomConfig/None deletes the three chart-owned KeycloakRealmGroups; membership is not restored on toggle-back. Not a bug, but the operator UX cost of the toggle deserves a bullet in Failure modes. Regenerated artifacts (cozyvalues-gen + hack/update-crd.sh): packages/extra/monitoring/values.schema.json packages/extra/monitoring/README.md packages/system/monitoring-rd/cozyrds/monitoring.yaml Test suites still green: packages/system/monitoring 23/23, packages/core/platform 81/81. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>	2026-07-15 15:31:44 +03:00
Aleksei Sviridkin	249f7b5a5e	feat(platform, ingress): propagate operator wildcard certificate to per-tenant termination points (#2990) ... ## What this PR does The operator-provided wildcard certificate (`publishing.certificates.wildcardSecretName`) currently serves the root tenant only: everything is same-namespace, so the root ingress controller and Gateway read the operator's TLS Secret directly. Child tenants run their own ingress controller / Gateway in their own namespace under a default-deny policy and cannot read that Secret across namespaces, so they fall back to per-host ACME. This propagates the operator wildcard to per-tenant termination points by replicating the Secret into each tenant namespace that owns one, keeping the root-MVP invariant intact: the wildcard is deliberately distributed to tenant namespaces, but no tenant RBAC is widened — every consumer reads only its own-namespace copy. A new controller in `cozystack-controller` does the replication with no extra operator input. It reads the same platform values channel the consumers read (`cozy-system/cozystack-values`): the wildcard Secret name from `_cluster.wildcard-secret-name` and the publishing namespace from `_cluster.expose-ingress`. It mirrors that Secret into every tenant namespace whose `namespace.cozystack.io/ingress` or `namespace.cozystack.io/gateway` label equals its own name — i.e. the namespace runs its own controller / Gateway. Because the source is derived from the same value that makes the consumers reference it, the replica is created whenever the consumers expect it — no manual labelling and no upgrade hazard where a child controller references a Secret that nothing creates. Replicas are refreshed on rotation and pruned when a namespace stops terminating TLS. Teardown of every replica happens on exactly one trigger: clearing `wildcardSecretName` (explicit disable). A source that is merely absent or mistyped, or a transiently-missing values channel, leaves existing replicas in place, so a brief gap, a delete+recreate rotation, or a misconfigured publishing namespace never drops tenant TLS. The platform controller's own ServiceAccount gains Secret writes; tenant RBAC is untouched. The controller does not cache every Secret in the cluster: the manager's Secret informer is scoped to managed replicas and the values channel only, and the operator source / foreign collisions are read through the uncached APIReader. The source is not watched (its name is dynamic), so an in-place rotation is picked up by a bounded periodic resync while the publishing tenant serves the source with no lag. On the consumer side, each tenant ingress controller now sets `--default-ssl-certificate` to the wildcard Secret in its own namespace (`.Release.Namespace/<name>`), so a child tenant serves the replica instead of minting per-host ACME. The Gateway path already renders `certMode: existingSecret` with a same-namespace `wildcardSecretRef` for every tenant, so it consumes the replica with no chart change. ### Why not clustersecret-operator The issue proposed the shipped `clustersecret-operator`. It materializes inline template data into selected namespaces; it cannot mirror an existing source Secret, and it is not installed in any default bundle. Driving it would mean embedding the certificate and private key as literal data in a cluster-scoped object rendered from a Helm `lookup` — that broadcasts key material the design keeps off any shared channel, and a `lookup` is invisible to the Flux helm-controller digest so it would not re-render on rotation. Its namespace selector also cannot express "the namespace whose owner label equals its own name", so it would over-replicate the key into every tenant namespace. A small dedicated reconciler targets exactly the termination namespaces and keeps the key off the values channel. ### Coverage caveat Replication delivers the certificate bytes, not SAN coverage: a single `*.<root-host>` does not match `*.<child-apex>`. Operators serving child-tenant subdomains supply a certificate whose SANs cover those apexes; the docs spell this out. The replica carries the wildcard private key into each terminating tenant namespace — the same exposure as the per-host ACME Secret it replaces, now shared across tenants. ### Tests - Go fake-client tests for the reconciler: termination-only replication (ingress-owning and gateway-owning namespaces only, never inheriting ones, never over the source), copy markers, rotation, prune on namespace demotion, the non-destructive source-deletion contract, replica self-heal after out-of-band deletion, the publishing-namespace self-replication guard, disable-via-config teardown, source rename, foreign-Secret collision safety (no overwrite, no requeue, does not block siblings), transient-error requeue, non-TLS source ignored, absent-channel keeps replicas, publishing-namespace mismatch keeps and requeues, the cache-scope pin, the enqueue-wiring map, an `ownsTerminationPoint` table, and the active-only requeue contract. - helm-unittest for the ingress `default-ssl-certificate` wiring (including a child-tenant controller pointing at its own-namespace replica) and for the cozystack-controller ClusterRole's new Secret-write rule. ### Screenshots N/A — no UI changes. ### Release note ```release-note feat(platform): the operator-provided wildcard TLS certificate (`publishing.certificates.wildcardSecretName`) is now served by per-tenant ingress controllers and Gateways automatically — the platform controller replicates it into each tenant namespace that terminates TLS, with no extra operator input and no cross-namespace Secret read ``` Closes #2820 Part of #2811 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes * **New Features** * Wildcard TLS Secret now replicates from the platform to tenant namespaces that terminate TLS via their own ingress controller or Gateway, including child-tenant scenarios. * **Documentation** * Clarified configuration, timing, and SAN matching requirements for wildcard replication. * Updated gateway and ingress-nginx default certificate documentation for replica mirroring behavior. * **Bug Fixes** * Improved handling for missing/invalid source secrets (no unnecessary pruning) and safe collision behavior. * **Tests** * Added comprehensive reconciliation/replication test coverage. * Added Helm unittest target for the controller chart. * **Chores** * Updated controller RBAC to support wildcard TLS Secret replication. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-07-15 11:32:08 +03:00
Aleksei Sviridkin	f24b56327b	fix(platform): route OIDC backend calls through the in-cluster Keycloak by default ... authentication.oidc.keycloakInternalUrl defaulted to an empty string, which left oauth2-proxy performing OIDC discovery against the external Keycloak hostname. That request resolves through public DNS and lands on the root ingress controller, which does not serve yet on a fresh install: the root tenant's IngressClass only exists once tenant-root is ready, and tenant-root is still reconciling while the dashboard starts. Discovery has nothing to talk to, the gatekeeper fails, and the dashboard never comes up. The bundled Keycloak is the only supported provider — the external hostname is templated as keycloak.<host> and the realm is fixed — so backend-to-backend calls have no reason to leave the cluster. Default the value to http://keycloak-http.cozy-keycloak.svc:8080/realms/cozy, the same Service address keycloak-configure already uses. The dashboard and linstor-gui gatekeepers then run with --skip-oidc-discovery and send token, jwks, userinfo and logout requests to the Service, while browser redirects stay on the external host. Those calls travel as plaintext HTTP between pods, as keycloak-configure's admin calls to the same address already do. The value remains overridable for a Keycloak that does not live at the standard in-cluster address, and setting it back to an empty string restores external discovery. Existing OIDC clusters that never set it get a gatekeeper rollout onto the internal path on upgrade. Enabling OIDC swaps the dashboard's token-proxy container for oauth2-proxy, so the E2E install now drives that swap to completion: it waits for the auth-proxy container to appear, then gates on the gatekeeper rollout. Readiness alone proves nothing there — the HelmRelease is still Ready from the token-proxy install, and the Deployment keeps the old ReplicaSet Available while a new pod crashloops, so only the rollout catches a regression back to external discovery. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-07-14 14:22:46 +03:00
myasnikovdaniil	99a28d82c2	test(e2e): migrate app suite from BATS to Chainsaw and wire into CI (#2826) ... ## What this PR does Migrates the E2E **app** test suite from BATS to [Kyverno Chainsaw](https://github.com/kyverno/chainsaw) — a declarative, Kubernetes-native E2E framework (CNCF, part of the Kyverno project; successor to KUTTL) — and wires it into CI as the replacement for the per-app BATS loop. This grows the original pilot (postgres + bucket) into the full suite. All 22 app suites are ported under `hack/e2e-chainsaw/` and the `hack/e2e-apps/*.bats` files are removed: `postgres`, `bucket`, `mariadb`, `mongodb`, `redis`, `qdrant`, `clickhouse`, `kafka`, `etcd`, `openbao`, `harbor`, `foundationdb`, `external-dns`, `kuberture`, `vminstance`, `gateway`, `kubernetes-latest`, `kubernetes-previous`, `kubernetes-oidc-system`, `kubernetes-oidc-customconfig`, `securitygroup`, `serviceexposure`. **Approach** - **Declarative** suites assert on `status.conditions` and concrete fields. The `timeout N sh -ec "until kubectl get ..."` + `kubectl wait` pair that appeared ~100 times collapses into a single `assert` that polls existence and state together, with a structured diff on failure and automatic `events`/`describe`/`podLogs` capture via `catch` (previously only `harbor.bats` did this, by hand). - **Imperative** suites keep their logic in `script` steps: `openbao` init/unseal, `kuberture` external-dns split-horizon probes, `vminstance`, the `gateway` admission/impersonation cases, and `kubernetes-latest`/`previous`, which wrap the relocated `hack/e2e-chainsaw/_lib/run-kubernetes.sh` verbatim (Kamaji bring-up, LB/NFS/ouroboros checks). - `gateway` tests derive the tenant apex from the namespace `namespace.cozystack.io/host` label at runtime, so they are host-independent. **CI** - The `e2e` job now runs `chainsaw test hack/e2e-chainsaw/` via a new `test-chainsaw` target and uploads the JUnit `chainsaw-report.xml`. - The `chainsaw` binary is added to the e2e-sandbox image. - `install-cozystack` and `test-openapi` stay BATS (cluster bootstrap + OpenAPI checks). - The per-app 3-retry loop is dropped — assertion polling replaces the fixed-timeout flakiness it papered over. **Validation** Ran against a development cluster: the DB/app, storage, and VM suites pass; the three suites that depend on platform features not present on that cluster (`gateway`, `kuberture`, `external-dns`) and the two heavyweight `kubernetes-*` suites are exercised by this PR's CI run on a freshly installed platform. Note for reviewers: service-port asserts use the `(ports[*].port)` projection form rather than a number-literal filter (`` ports[?port == `N`] ``), because Chainsaw v0.2.15 mis-evaluates JMESPath number-literal comparisons. **2026-07-09: reconciled with main (~520 commits of drift)** - etcd suite re-ported to the v1alpha2 operator contracts (#2859): `readyMembers=3` gate, pod-label/`/scale`/WorkloadMonitor/metrics/defrag contracts, plus the `ETCD_E2E_S3_ROUNDTRIP`-gated backup round-trip driving `examples/backups/etcd` - 12 new `ingress-hostname-policy` gateway cases ported (apex derived at runtime from the tenant-root namespace label) - post-branch bats drift folded in: `mariadb-single` webhook guard, kafka `c1.small` presets, bucket/harbor 2m BucketClaim fail-fast, bucket readonly-denial promoted to a hard fail (cosi-driver v0.3.1), qdrant PVC-reclaim guard (#3059), kuberture fail-loud negations, SC-fallback-default test (#2872 B1) - 4 suites that only exist on main since the branch was cut are ported: `kubernetes-oidc-system`, `kubernetes-oidc-customconfig`, `securitygroup`, `serviceexposure` - CI seams: `nightly.yaml` converted to the chainsaw invocation, `run-kubernetes.sh` reconciled (Talos/CABPT waits, tenant drain, LINSTOR pool wait, talos-image-cache under `_lib/`), `e2e-capture-dataplane.sh` replicated in the Chainsaw global catch ### Release note ```release-note fix(kubernetes): tenant Kubernetes teardown no longer hangs when the cluster has no working nodes — the pre-delete hook now bounds its wait for the in-tenant HelmReleases and force-clears their Flux finalizers on timeout (#3271) ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * End-to-end testing is now driven by a unified Chainsaw suite (platform apps, Kubernetes, networking, security groups, service exposure). * Test Impact Analysis selects affected suites, with an option to run the full suite. * CI produces a JUnit report artifact and collects suite-scoped diagnostics on failure. * **Bug Fixes** * CI E2E now fails fast and emits clearer Kubernetes/HelmRelease and sandbox bucket state diagnostics. * Improved teardown reliability with bounded wait + finalizer handling to avoid hangs. * **Documentation** * Updated E2E testing docs and local guidance to reflect the new Chainsaw-based workflow and suite paths. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-07-12 11:58:08 +05:00
Myasnikov DaniilandClaude Opus 4.8	302a72f14a	Merge origin/main into feat/e2e-chainsaw-example ... Reconciles the talos-image-cache conflicts created when #3254 merged to main: keep the Chainsaw _lib/ port (hack/e2e-chainsaw/_lib/talos-image-cache.sh, a superset of #3254's logic + the node-join diagnose helper), drop the BATS-era hack/e2e-apps/talos-image-cache.sh, and keep the _lib/-path versions of the mirror manifest, install bats, and unit test. Also picks up main's zizmor action-pinning (#3223, unblocks pre-commit/zizmor) and Talos v1.13.6 (#3240). Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-07-10 12:00:01 +05:00
renovate[bot]	5b9b420ffe	chore(deps): update container image & action digests	2026-07-10 05:29:23 +00:00
Aleksei Sviridkin	3e9b5600e3	fix(talos): bump Talos to v1.13.6 (CVE-2026-53359) ... Talos v1.13.5 ships Linux 6.18.36, which carries the fix for CVE-2026-46113 but not the one for CVE-2026-53359. The two are related KVM x86 shadow-paging bugs allowing a guest to escape to the host, and both patches are needed for the mitigation to hold. Talos v1.13.6 ships Linux 6.18.38, the first release on the 6.18 stable line that has both. Profiles regenerated with hack/gen-profiles.sh v1.13.6. Alongside the Talos version and the installer reference, this refreshes the pinned system-extension digests: the firmware extensions move from the 20260519 to the 20260622 build, and intel-ucode stays on its 20260512 tag but resolves to a new digest because upstream re-pushed that tag. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-07-09 20:29:40 +03:00
Myasnikov DaniilandClaude Fable 5	1f84d2d49e	Merge origin/main into feat/e2e-chainsaw-example (post-#3244) ... Fold the in-sandbox Talos image cache work into the Chainsaw layout: the new shared helper relocates to hack/e2e-chainsaw/_lib/ talos-image-cache.sh and run-kubernetes.sh sources it from there (the ouroboros DNS-check no-retry rework merges clean). The OIDC bats are resolved as deleted again; their Chainsaw fixtures deliberately skip the talos.imageFactoryURL injection — the helper only emits it when the cache is up, and the zero-worker render-side tests instantiate no DataVolume imports for it to redirect. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-07-09 10:38:35 +05:00
scooby87	f91841fce6	fix(monitoring): surface missing VictoriaLogs and gate dead-end vlinsert ExternalName (#3215) ... ## What this PR does Addresses #3181, where platform log ingestion could fail **silently**: fluent-bit shipped to `vlinsert-generic` while the monitoring `HelmRelease` reported `Ready`, and a `vlinsert-generic` `ExternalName` in `cozy-monitoring` dead-ended at a `tenant-root` VictoriaLogs that did not exist when the root tenant had monitoring disabled. Logs were dropped with no failing signal. Three complementary layers make the failure loud instead of silent: 1. **Readiness gate** — `packages/extra/monitoring/templates/helmrelease.yaml` now sets a `poller` `waitStrategy` and a `healthCheckExprs` entry on the `VLCluster` `status.updateStatus`. A missing/unhealthy log store makes the `monitoring-system` release **not-ready** instead of flipping `Ready` the moment the CR is applied. Mirrors the existing seaweedfs CNPG pattern. 2. **No empty-log-store hole** — `packages/system/monitoring/templates/vlogs/vlogs.yaml` fails the render on an empty `logsStorages`. With no entry no `VLCluster` exists, the readiness gate has nothing to match, and fluent-bit would silently drop logs — so this is now a loud misconfiguration. 3. **No silent fallback** — the `cozy-monitoring → tenant-root` `vlinsert`/`vminsert` `ExternalName` redirects in `cozystack-basics` are created only when the root tenant hosts monitoring. The template reads the live `Tenant/root` `spec.monitoring` via `lookup` (source of truth, so it cannot drift from the tenant's real state) and falls back to a new platform value `monitoring.rootEnabled` (via `_cluster.monitoring-enabled`) when `lookup` is empty (dry-run / first render). The default preserves the historical always-create behaviour. Covered by helm-unittest across all four affected charts (141 tests): readiness-gate fields, empty-vs-populated `logsStorages` contract, `ExternalName` present/absent fallback contract, and the platform wiring into the `cozystack-values` Secret. Also adds the missing `test:` target to `packages/extra/monitoring/Makefile` so its suite runs in CI. **Known limitation / follow-up:** the legacy platform monitoring release installed directly into `cozy-monitoring` via the `cozystack.monitoring` PackageSource cannot carry the readiness gate, because the engine's `ComponentInstall`/HelmRelease builder and the vendored `helm-controller` API v1.4.3 lack `waitStrategy`/`healthCheckExprs`. That source is not referenced by any active bundle; full coverage there needs a helm-controller bump plus engine support. ### Release note ```release-note fix(monitoring): surface missing platform VictoriaLogs as not-ready and stop the dangling vlinsert-generic ExternalName from silently dropping logs (#3181) ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a platform setting (`monitoring.rootEnabled`) to control whether root-level monitoring is enabled. * Monitoring redirect ExternalName services are now rendered only when root monitoring is enabled (defaulting to enabled). * **Bug Fixes** * Ensures an explicitly set `monitoring.rootEnabled=false` is preserved when generating platform configuration. * Prevents incomplete monitoring deployments by failing rendering when `logsStorages` is empty. * Improves VictoriaMetrics readiness by delaying “ready” until the operator reports an operational status. * **Tests** * Added/expanded Helm-unittest suites covering monitoring wiring, redirects, readiness gating, and `logsStorages` requirements. * Added a `make test` target to run chart unittests. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-07-09 01:55:01 +03:00
Алексей Артамонов	f700c9043c	fix(monitoring): backfill release label onto pre-existing VM/VL storage PVCs ... Re-review on #3094 (IvanHunters, MAJOR): the volumeClaimTemplate label lands on PVCs only at creation — the StatefulSet controller never re-labels existing PVCs and vm-operator's PVC reconcile is resize-only. So on an already-deployed cluster (exactly the #3091 population) the pre-upgrade VM/VL storage PVCs stay unlabeled and the cleanup hook's selector misses them. Add migration 49 (mirrors the ClickHouse-keeper backfill in migration 48 for the sibling #3072). It relabels the pre-existing vmstorage/vmselect/vlstorage PVCs of every cozystack-managed VMCluster/VLCluster with apps.cozystack.io/application.name read from the CR's own spec.managedMetadata, so only monitoring's clusters are touched (a user's own VMCluster carries no such label) and the value always matches what the cleanup hook selects. Best-effort like #48: a skipped relabel is no worse than the pre-existing leak, so transient apiserver errors never abort the platform upgrade. Bump platform targetVersion 49 -> 50. Also correct the misleading comments (IvanHunters MINOR): spec.managedMetadata is never applied to PVCs by vm-operator at any current version, so the claim-template label is the only mechanism — do not imply a v0.71.0 crossover. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> Assisted-By: Claude <noreply@anthropic.com>	2026-07-08 22:27:42 +03:00
Alexey Artamonov	f9afc73096	fix(monitoring): surface missing VictoriaLogs and gate dead-end vlinsert ExternalName (#3181) ... Platform log ingestion could fail silently: fluent-bit shipped to vlinsert-generic while the monitoring HelmRelease reported Ready, and a vlinsert-generic ExternalName in cozy-monitoring dead-ended at a tenant-root VictoriaLogs that did not exist when the root tenant had monitoring disabled. Logs were dropped with no failing signal. Three complementary changes: - Readiness (do not stay silently Ready): the monitoring-system HelmRelease now uses a `poller` waitStrategy and a healthCheckExpr on its VLCluster (status.updateStatus), so a missing/unhealthy log store makes the release not-ready instead of flipping Ready as soon as the CR is applied. Mirrors the seaweedfs CNPG pattern. - No empty-log-store hole: system/monitoring rejects an empty logsStorages at render time (helm fail). With no entry no VLCluster is created, the readiness gate has nothing to match, and fluent-bit would silently drop logs — so this is made a loud misconfiguration rather than a documented footgun. - No silent fallback: the cozy-monitoring -> tenant-root vlinsert/vminsert ExternalName redirects in cozystack-basics are now created only when the root tenant hosts monitoring, gated on _cluster.monitoring-enabled — fed from a new platform value monitoring.rootEnabled (default true, preserving the historical always-create behaviour). monitoring.rootEnabled is the single render-time source of truth: changing it re-renders cozystack-basics (helm-controller watches the cozystack-values Secret it flows into). A live `lookup` of Tenant/root spec.monitoring is deliberately NOT used: cozystack-basics is an operator-generated HelmRelease with drift detection off, so it re-renders only on install/upgrade/values-change — never on a reconcile tick or when a lookup result changes — and a lookup would go stale in both directions. Operators must keep monitoring.rootEnabled in sync with Tenant/root spec.monitoring. Covered by helm-unittest across all four charts: readiness gate fields, empty-vs-populated logsStorages contract, ExternalName present/absent and null-guard contract, and the platform wiring into the cozystack-values Secret. The CEL health predicates are exercised live, not in helm-unittest (which cannot evaluate CEL). Adds the missing `test:` target to packages/extra/monitoring/Makefile so its suite runs in CI. Known limitation (follow-up): the legacy cozystack.monitoring PackageSource that installs system/monitoring directly into cozy-monitoring cannot carry the readiness gate, because the engine's ComponentInstall/HelmRelease builder and the vendored helm-controller API v1.4.3 lack waitStrategy/healthCheckExprs. That source is not referenced by any active bundle; full coverage needs a helm-controller bump plus engine support. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>	2026-07-08 22:27:22 +03:00
Myasnikov DaniilandClaude Fable 5	9297e51170	Merge origin/main into feat/e2e-chainsaw-example ... Reconcile the Chainsaw e2e migration with ~500 commits of main drift: - keep main's run-kubernetes.sh helpers (kubectl_wait_retry, tenant drain, LINSTOR pool-free wait, SC-fallback probe) under the branch's _lib/ rename with the Chainsaw finally/snapshot deltas re-applied - resolve modify/delete on 10 drifted hack/e2e-apps/*.bats as deleted; their post-branch assertion drift is re-ported into the Chainsaw suites in follow-up commits - drop the 4 new main-side bats (kubernetes-oidc-*, securitygroup, serviceexposure) from _lib/ relocation; ported as Chainsaw suites in follow-up commits - docs/agents/e2e-testing.md: merge main's plan-job TIA wording with the Chainsaw suite phrasing Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-07-08 18:04:08 +05:00
myasnikovdaniil	cece5c23b5	feat(release): immutable tags & rc→stable promotion (#2677) (#3017) ... ## What this PR does Implements the immutable-tag + rc-promotion flow from #2677. A **stable release becomes a renamed release-candidate**: the bytes shipped as `vX.Y.Z` are bit-for-bit the bytes built and e2e-tested as `vX.Y.Z-rc.N`. No tag is ever force-moved, and stable is never rebuilt — it is *promoted* by retagging the rc's existing images. Layered onto the build matrix from #2937/#2983 (this PR is **stacked on `refactor/build-matrix-2937`** and must merge after it). ## The five force-retag sites, removed | Site | Before | After | |---|---|---| | `tags.yaml` api/apps/v1alpha1 tag | `git tag -f` / `push -f` | write-once (create-if-absent, fail if it would move) | | `tags.yaml` release-X.Y.Z branch | `git branch -f` / `push -f` | compare-before-force (no-op if unchanged; staging branch only) | | `pull-requests-release.yaml` stable tag | `git tag -f` / `push -f` | write-once at the PR merge commit (force impossible by construction) | | `pull-requests-release.yaml` maintenance branch | `updateRef force:true` | fast-forward-only | | `auto-release.yaml` patch tags | delete-recreate (cron) | **workflow deleted** — stable only via explicit promote | ## Version decoupling (the enabler) The operator baked its version into the image at build time, so an rc image self-reported the rc string — blocking retag-promotion. It now reads `COZYSTACK_VERSION` from the environment (threaded via `cozystackOperator.platformVersion` → Deployment env, stamped by `make manifests`), falling back to the build-time value when unset. The same image bits can report any release name. The only runtime reader is the telemetry metric `cozy_cluster_info{cozystack_version=...}`. ## Promotion flow `promote-rc.yaml` (`workflow_dispatch`, `rc_tag=vX.Y.Z-rc.N`): 1. Validate the rc release exists and the stable tag does not. 2. `hack/promote-retag.sh` reads the rc's digest-pinned image refs from `packages/*/*/values.yaml` and `skopeo copy`s each — **by digest** — to `:vX.Y.Z` and `:latest`, verifying with `skopeo inspect`. 3. Rewrite the cosmetic `-rc.N` substring in vendored tags to the stable version (the `@sha256` wins regardless), restamp the version-stamped assets (operator manifests, cozypkg, openapi; the heavy Talos assets are copied verbatim from the rc draft), open a `release-X.Y.Z` PR. 4. Merging the PR reuses the existing release-PR e2e and `pull-requests-release.yaml` finalize to cut the **write-once** stable tag at the merge commit and publish the release. Squash is disallowed (the tag needs a real merge commit). `nightly.yaml` cuts write-once `*-nightly.<date>` tags (gated by `NIGHTLY_ENABLED`); `retention.yaml` keeps the newest 14 per line (dry-run by default). ## Validation **Local end-to-end — the core claim is proven, not asserted.** The whole design rests on "copy-by-digest to a new tag preserves the digest, so stable == the e2e-tested rc, bit-for-bit." This was exercised against a real registry (`ttl.sh`) with real `skopeo`: - Pushed a multi-arch image under a `:v1.4.0-rc.2` tag, built a temp tree exercising all three `values.yaml` digest shapes (single `image:` string, split `repository`/`tag`/`digest` map, and the `platformSourceRef` OCI artifact), then ran the actual `hack/promote-retag.sh v1.4.0`. - **Result: `:v1.4.0-rc.2`, `:v1.4.0`, and `:latest` all resolve to the identical digest** (`sha256:fd8d9aa6…`), across all 17 platform manifests. The script's own `skopeo inspect` post-check passed, and an independent re-inspection confirmed it. Local validation caught (and this PR fixes) two bugs that only surface against a real registry — never in static lint: 1. `skopeo copy --multi-arch all --all` — mutually exclusive flags, fatal error; every retag would have failed on CI. Now `--multi-arch all`. 2. The same digest was retagged twice when an image appeared in two value shapes; the ref set is now deduped on the canonical `repo@digest`. **Other local checks:** `go test ./pkg/version` + `go build`/`vet`; `helm template` renders `COZYSTACK_VERSION` in all 3 operator variants (omitted when unset); `make manifests` stamps the version into the install assets; `shellcheck` clean on `promote-retag.sh`; `actionlint` clean and `act -l` resolves the job graph on all workflows; the rc-tag parser and nightly version-math unit-tested (accept/reject + minor-vs-patch bumps). **Still CI-only (cannot be exercised offline):** the full workflow runtime — github-script API calls, OCIR auth, the rc-release / staging-branch preconditions, the nightly `base_ref` push mechanics, and self-hosted runners. The *logic* inside the steps is unit-tested; the *orchestration* is not. Treat it as unproven until the workflows run. ## Out of scope / required follow-up (needs repo admin — not doable from a PR) - Tag-protection rules: `v*` and `api/apps/v1alpha1/*` create-only by the CI app, no delete/update; "limit branches/tags updated in a single push" = 1. - Enable `NIGHTLY_ENABLED` (and `RETENTION_APPLY` when ready) repo variables. - Registry-side pruning of `*-nightly.*` image tags (no OCIR delete parity yet — tracked TODO in `retention.yaml`). ## Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added nightly publishing (mirror-by-digest, disk build, e2e validation) and nightly retention pruning. * Added RC-to-stable promotion via digest retagging (no rebuild). * Operator/installer and console now expose version/platform metadata, including runtime override via `COZYSTACK_VERSION`. * **Bug Fixes** * Enforced write-once tag behavior and fast-forward-only maintenance updates across release workflows. * Made nightly mirroring/selection and retention pruning more selective and safer. * **Tests** * Added Bats coverage for nightly mirroring and RC retagging, plus new Helm/unit checks. * **Documentation** * Updated the release model to center RC promotion and tag immutability. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-07-07 17:46:06 +05:00
Aleksei Sviridkin	43104c9075	fix(ingress): guard the immutable loadBalancerClass migration on existing Services ... spec.loadBalancerClass is immutable once a Service is type LoadBalancer, so setting, changing, or clearing publishing.loadBalancerClass on a cluster whose host ingress Service already exists would make Flux wedge on an API-rejected patch instead of migrating. Fail the render early with the exact recreate command whenever an existing type: LoadBalancer Service's class differs from the requested value; a fresh install (lookup nil) or a ClusterIP Service transitioning to LoadBalancer is left untouched. Note the caveat in the values comment and cover the guard with helm-unittest. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-07-07 12:39:30 +03:00
Aleksei Sviridkin	a1748009ac	fix(ingress): scope host loadBalancerClass to the external-facing ingress ... publishing.loadBalancerClass is a host-publishing setting, but its _cluster.load-balancer-class block was emitted on any ingress taking the LoadBalancer branch — including tenant ingresses — unlike expose-external-ips and proxy-protocol, which gate on the host namespace. A tenant ingress would then be claimed by an LB controller the tenant never selected. Gate it the same way, and add helm-unittest coverage for the host/tenant split and the platform to _cluster wiring. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-07-07 12:39:30 +03:00
Andrei Kvapil	3dbc7d3662	refactor(network): drop ExposureClass/ServiceExposure for native loadBalancerClass ... Remove the network.cozystack.io API group (ExposureClass + ServiceExposure) and its reconciler. Managed-application charts own their Service, so native type: LoadBalancer + loadBalancerClass + an admin-provisioned pool covers external exposure without a dedicated API group and controller. The host ingress -- the only consumer -- keeps rendering type: LoadBalancer and gains an optional publishing.loadBalancerClass to pick the LB controller; the externalIPs (node-IP) default path is unchanged. Rationale in cozystack#3164. Assisted-By: Claude Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>	2026-07-07 12:39:30 +03:00
Myasnikov DaniilandClaude Opus 4.8	6d3d4908de	test(installer): cover the generic variant in the version test ... The COZYSTACK_VERSION env test pinned talos and hosted but not generic, leaving the generic branch of the endpoint-env interleave (KUBERNETES_ SERVICE_HOST/PORT from cozystack.apiServerHost/Port) unguarded. Add a generic + platformVersion case so a refactor can't silently drop COZYSTACK_VERSION or mis-render the generic endpoint block. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-07-06 16:49:42 +05:00
Andrey Kolkov	d0740ed8c7	test(etcd): gate S3 backup round-trip behind ETCD_E2E_S3_ROUNDTRIP ... Signed-off-by: Andrey Kolkov <androndo@gmail.com>	2026-07-06 14:51:48 +04:00
Andrey Kolkov	011d6d1639	fix FUs ... Signed-off-by: Andrey Kolkov <androndo@gmail.com>	2026-07-06 14:41:33 +04:00
Andrey Kolkov	f2866edcf0	review fixes ... Signed-off-by: Andrey Kolkov <androndo@gmail.com>	2026-07-06 14:41:33 +04:00
Andrey Kolkov	087a8c4be9	chore(etcd-operator): bump to v0.5.2 ... Signed-off-by: Andrey Kolkov <androndo@gmail.com>	2026-07-06 14:41:33 +04:00
Andrey Kolkov	cc5d688e15	feat(platform): migrate to etcd-operator v1alpha2 ... Signed-off-by: Andrey Kolkov <androndo@gmail.com>	2026-07-06 14:41:33 +04:00
Myasnikov Daniil	22a9437aa6	test(installer): cover COZYSTACK_VERSION env rendering ... The COZYSTACK_VERSION env is what lets a release-candidate image be promoted to stable by retagging alone, and it is interleaved with the per-variant endpoint env under one conditional. Add a helm-unittest (and the package test target so `make unit-tests` runs it) pinning both axes: talos/hosted x platformVersion set/unset, asserting COZYSTACK_VERSION presence and the variant endpoint block. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-07-06 10:03:48 +05:00
Myasnikov DaniilandClaude Opus 4.8	fd7fcbda4e	refactor(operator): read platform version from env (decouple from build) ... The operator's platform version was baked into the image at build time (`-ldflags -X pkg/version.Version` from `git describe`), so a `vX.Y.Z-rc.N` image literally carried the rc string. That blocks promoting a release-candidate to stable by retag: the bits would still self-report the rc version. Resolve the version from the COZYSTACK_VERSION env var at runtime, falling back to the build-time value when unset (local builds still work). The installer chart threads `cozystackOperator.platformVersion` into the operator Deployment as that env var, and `make manifests` stamps it into the kubectl-apply release artifacts. The same image bits can now report any release name, so promotion needs no rebuild. The only behavioral reader is the telemetry metric `cozy_cluster_info{cozystack_version=...}`; cozypkg/check-readiness keep their own build-time Version and are unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-07-06 10:03:46 +05:00
IvanHunters	a1c779854b	fix(vertical-pod-autoscaler): migrate admission-controller TLS to cert-manager to stop reconcile-driven ReplicaSet churn (#3178) ... ## Summary The VPA admission-controller webhook TLS Secret is currently rendered by an in-chart `genCA` + `genSignedCert` template. Those functions are not idempotent across `helm template` invocations — every Flux HelmRelease reconcile produces a fresh CA + cert. A `checksum/tls-secret` podAnnotation on the admission-controller Deployment then flips its sha, materialising a new ReplicaSet on every reconcile. Under load — the E2E install phase where cert-manager / CDI / LINSTOR cascade races repeatedly refire the VPA HelmRelease — this produces 10+ admission-controller ReplicaSets in minutes and blocks the whole cozystack-platform HR: the admission-controller pod livelocks in startup (waits on its own election lease which its previous ReplicaSet is still holding, hits livenessProbe timeout, CrashLoopBackOffs), VPA never becomes Ready, PackageSources stay InProgress, and cozystack-platform aborts at its 15-minute install timeout. Fix: replace the source of the TLS Secret with a namespace-scoped cert-manager `Issuer` + `Certificate`. cert-manager only rewrites the Secret when the certificate is close to expiry (`renewBefore`), so reconcile-driven cert churn stops. Pod rotation is delegated to a runtime-lookup-based `checksum/tls-cert` podAnnotation that only changes when the Secret's `tls.crt` content actually changes — not on every render, not on every reconcile. ## What - `charts/vertical-pod-autoscaler/templates/admission-controller/certificate.yaml` (new): namespace-scoped selfSigned Issuer + Certificate. The Certificate mirrors the legacy `genSignedCert` shape (single DNS SAN `vpa-webhook.<ns>.svc`, 365d validity, matching Secret name), so cert-manager on the first upgrade adopts the same Secret name the legacy path wrote — no rename cascade in the Deployment volume mount. Rendered only when `admissionController.tls.certManagerEnabled=true` AND no `existingSecret` is supplied. - `charts/vertical-pod-autoscaler/templates/admission-controller/tls-secret.yaml` (modified): wrapped in an extra `certManagerEnabled=false` guard so the legacy genCA path remains available as an explicit opt-out. - `charts/vertical-pod-autoscaler/templates/admission-controller/deployment.yaml` (modified): the legacy `checksum/tls-secret` annotation is gated behind `certManagerEnabled=false`. Under the cert-manager path the Deployment carries a `checksum/tls-cert` annotation computed from `lookup`-ing the runtime Secret's `tls.crt` — see the template comment for the full timing analysis and the day-350 renewal trade-off it documents. - `charts/vertical-pod-autoscaler/values.yaml` (modified): new `admissionController.tls.certManagerEnabled` value defaulting to `true`. Values-block header documents the switch as effectively one-way (cert-manager writes `type: kubernetes.io/tls`; `Secret.type` is immutable, so falling back requires `kubectl delete` first). - `patches/cert-manager-admission-tls.diff` (new) + `Makefile update` extended: the four subchart edits above live as a single reviewable diff that Makefile re-applies over the freshly vendored chart on every `make update`, matching the fluxcd-operator package convention. Prevents a bump from silently reverting the fix. - `packages/core/platform/sources/vertical-pod-autoscaler.yaml` (modified): VPA PackageSource now depends on `cozystack.cert-manager`. Non-circular: cert-manager depends on `cozystack.networking` + `cozystack.gateway-api-crds`, neither reaches VPA. - `tests/vpa_test.yaml` (modified): +11 helm-unittest cases (15 total, 4 pre-existing) covering `certManagerEnabled={true,false}`, `existingSecret` bypass, disabled admission-controller, and the offline-render behaviour of the `checksum/tls-cert` lookup. ## Why (evidence) Snapshot from CI run 28587656430 (this branch, pre-workaround-review): ``` admission-controller ReplicaSets after first-install churn window: vertical-pod-autoscaler-admission-controller-55d888d6d4 vertical-pod-autoscaler-admission-controller-56f8d4c747 vertical-pod-autoscaler-admission-controller-5775dbc9b7 vertical-pod-autoscaler-admission-controller-585b7c56c9 vertical-pod-autoscaler-admission-controller-6fc44944fb vertical-pod-autoscaler-admission-controller-74bccf4849 vertical-pod-autoscaler-admission-controller-84df98f98b vertical-pod-autoscaler-admission-controller-8666dffb85 vertical-pod-autoscaler-admission-controller-86b469457c vertical-pod-autoscaler-admission-controller-9b979f9d6 → 10 ReplicaSets, one per Flux reconcile, each with a different genCA-rooted TLS Secret sha in checksum/tls-secret. ``` ## Migration for existing clusters On upgrade the legacy Opaque Secret disappears from the release manifest, Helm garbage-collects it, and cert-manager immediately writes a fresh `kubernetes.io/tls` Secret of the same name via the new Certificate. `Secret.type` is immutable, so in-place adoption is impossible — the delete/create pass is the correct path, and the seconds-long window it produces is safe: the `vpa-webhook-config` MutatingWebhookConfiguration ships with `failurePolicy: Ignore` (verified on cozy-dev3), so Pod admissions in that window skip VPA mutation but never get rejected — no cluster-wide disruption. The Deployment rolls in the same upgrade because the new `checksum/tls-cert` annotation differs from anything the pod had before, so the admission-controller re-registers `vpa-webhook-config` with the fresh caBundle on the same pass. ## Test plan - [x] `helm unittest packages/system/vertical-pod-autoscaler` — 15/15 pass. - [x] `helm template` diff between main and this branch is scoped exactly to the cert-manager path (new Certificate + Issuer, new `checksum/tls-cert` annotation, guarded legacy Secret, new values key, patch file + Makefile step, PackageSource dependsOn). No unrelated lines change. - [x] `make update`-style simulation (rm -rf charts && helm pull && patch) produces a byte-identical subchart to what git tracks — the patch stays clean across future upstream bumps until it genuinely conflicts. - [ ] Live migration on cozy-dev3: `helm upgrade` VPA release from pre-fix version → observe cert-manager write the new tls Secret → verify `checksum/tls-cert` on the new pod's podTemplate → verify MutatingWebhookConfiguration `vpa-webhook-config` caBundle updated in place → confirm no ReplicaSet churn on subsequent Flux reconciles. Blocking the draft's removal. - [ ] Full E2E on this branch — verify only 1 admission-controller ReplicaSet after install, VPA HR reaches Ready inside `cozystack-platform`'s 15m install timeout, and `kubernetes-latest.bats` / `kubernetes-previous.bats` no longer time out on VPA-attributable resource churn. ## Review pass — kvaps' CHANGES_REQUESTED addressed - ✅ Vendored chart mods now carried as `patches/*.diff` re-applied by `Makefile update` (matches fluxcd-operator convention). - ✅ Retired the tls-migrator hook — kvaps correctly showed it was a no-op: Helm's delete decision is driven by the stored release manifest not by live labels (`ResourcePolicyAnno` is the only opt- out), and `Secret.type` is immutable so cert-manager could not have adopted the legacy Opaque Secret in place anyway. ~200 lines removed. - ✅ PR description rewritten to match the actual diff (no more reference to the removed `tls-rollout.yaml` post-upgrade hook, real test count, dev3 verification honestly listed as pending). - ✅ `checksum/tls-cert` timing comment corrected: `lookup` runs on Helm actions only, day-350 renewal trade-off spelled out. - ✅ `certManagerEnabled=false` documented as effectively one-way (Secret.type immutable). - ✅ Dropped the `secretTemplate` labels — `app.kubernetes.io/managed- by: Helm` on a cert-manager-owned Secret was misleading. - ✅ Banner comment indentation restored in `vpa_test.yaml`. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * VPA admission-controller webhook TLS now uses cert-manager by default, with options to disable cert-manager or supply an existing TLS secret. * Webhook pod restarts are now tied to cert-manager Secret contents via a content-based checksum. * **Bug Fixes** * Improved webhook TLS provisioning reliability when TLS readiness is delayed. * Corrected conditional rendering/rollout behavior to avoid mismatched TLS ownership. * **Tests** * Added rendering-invariant tests covering cert-manager, legacy TLS, existing-secret bypass, and admission-controller disabled behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-07-04 22:48:00 +03:00
IvanHunters	59964043e3	fix(platform): migration 49 backfills tenant ancestor labels ... The tenant chart fix makes new and re-reconciled tenant namespaces self-compute their full tenant.cozystack.io/<ancestor> chain, but existing namespaces whose HelmRelease does not promptly re-reconcile (suspended or failed releases) keep the incomplete labels — and with them the broken <ancestor>-egress reachability that surfaces as upstream timeouts to nested tenants. Backfill the labels directly so clusters heal on upgrade regardless of per-tenant HelmRelease state. Bump targetVersion to 50 so run-migrations.sh iterates the new script (seq CURRENT..49); the missing-file guard otherwise never fires. Best-effort and idempotent: kubectl label --overwrite adds only the listed labels, and per-namespace failures are tolerated so a transient apiserver error cannot abort the platform upgrade. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>	2026-07-03 09:54:12 +03:00
Ivan Okhotnikov	9dc64b9278	feat(platform): VPA depends on cert-manager after admission-controller TLS migration ... The VPA admission-controller webhook TLS is now provisioned via a cert-manager Certificate + Issuer instead of the in-chart genCA Secret. On a fresh install the VPA package must not reconcile until cert-manager is Ready; otherwise the Certificate resource sits Pending forever, the admission-controller Secret is never written, and the pod CrashLoops on missing TLS mount — cascading up through the cozystack-platform PackageSource in exactly the way the old genCA path did, only for a different reason. Add cozystack.cert-manager to the variant's dependsOn list. Verified non-circular against packages/core/platform/sources/cert-manager.yaml (cert-manager depends on cozystack.networking + cozystack.gateway-api-crds, neither of which reaches VPA). Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>	2026-07-02 18:57:01 +03:00
Алексей Артамонов	a2ae1621af	fix(clickhouse): surface cleanup hook failures and backfill legacy keeper PVC labels ... Address review on #3072: - Cleanup hook no longer masks kubectl delete failures: the command now fails (exit 1) when PVC deletion fails instead of swallowing the error with || echo. The hook-delete-policy drops hook-succeeded (keeps only before-hook-creation) so a failed cleanup Job survives for inspection. - Add migration 45 (45 -> 46) that backfills app.kubernetes.io/instance onto keeper PVCs created before this change; the altinity keeper operator never re-stamps volumeClaimTemplate labels onto existing PVCs, so without this the cleanup selector misses pre-upgrade keeper PVCs and #3057 persists for existing releases. Bump migrations.targetVersion to 46. - Unify helm.sh/hook-weight key quoting across the four hook manifests. - Extend the unittest to assert the new hook-delete-policy and exit-on-failure. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>	2026-07-01 18:31:05 +03:00
Aleksei Sviridkin	c841c70866	feat(operator): replicate operator wildcard certificate to tenant namespaces ... The operator-provided wildcard MVP serves the root tenant only: the root ingress controller and Gateway read a TLS Secret from their own namespace. Child tenants run their own ingress controller / Gateway in a separate, default-deny namespace and cannot read that Secret cross-namespace. Add a controller that mirrors the operator wildcard Secret into every tenant namespace that owns a TLS termination point (its namespace.cozystack.io/ingress or .../gateway label equals its own name). The source is identified by name from the platform values channel (_cluster.wildcard-secret-name) with no extra operator input; replicas carry a managed-by label and refresh on rotation. An individual replica is removed when its namespace stops terminating TLS; every replica is torn down only on an explicit disable (clearing wildcard-secret-name). A missing or mistyped source, or a transiently-absent values channel, keeps existing replicas, so tenant TLS never drops on a transient gap. The controller's Secret cache is scoped to managed replicas and the values channel, not every cluster Secret. Only the platform controller's ServiceAccount gains Secret writes — no tenant RBAC is widened, and every consumer reads only its own-namespace copy. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-07-01 10:47:14 +03:00
Aleksei Sviridkin	41ff94eca9	fix(flux): tune source-watcher concurrency and CPU under load ... source-watcher reconciles one ArtifactGenerator per PackageSource and builds the resulting ExternalArtifacts. On the management cluster that is ~100 artifact builds during install, and the platform HelmRelease blocks until all of them are Ready. It was the only Flux controller left untuned. In the bootstrap all-in-one pod it ran without an explicit --concurrent (binary default 10) and with the shared 100m CPU request; in the production FluxInstance it was missing from the --concurrent / CPU patch that source-, kustomize- and helm-controller already receive. Under load its per-artifact builds finish at staggered times, leaving a random tail of not-yet-Ready artifacts that times out the platform install. Raise source-watcher to --concurrent=20 in both deployments. In the bootstrap all-in-one pod also give it a 500m CPU request; the pod has no CPU limits, so the request is what drives its CPU share under contention. In the production FluxInstance add it to the shared reconciler patch, so it now receives the same --requeue-dependency=5s and 2000m/2048Mi limits the other tuned controllers already have. The flux-aio module exposes a resources override for the watcher (set via CUE) but no args override, so the bootstrap --concurrent bump is a post-render patch in the Makefile. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-07-01 07:42:33 +03:00
Aleksei Sviridkin	cbc314a8c5	fix(kubevirt-cdi): order behind cozystack-basics so cdi-clone-dv RoleBinding finds cozy-public ... The kubevirt-cdi default variant only depended on cozystack.networking, which reconciles long before cozystack-basics provisions the cozy-public namespace. kubevirt-cdi ships the cdi-clone-dv RoleBinding into cozy-public, so on a fresh install the RoleBinding can race namespace creation and fail with `namespaces "cozy-public" not found`. Flux retries normally mask it, but under slow reconcile it surfaces as a hard UpgradeFailed. Add a cozystack.cozystack-basics edge so the namespace exists first, mirroring the backupstrategy-controller ordering. No cycle: cozystack-basics' own dependencies (tenant-application, cozystack-engine, gateway-api-crds) do not reach kubevirt-cdi. Pin the edge with a helm-unittest regression guard. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-06-30 19:51:46 +03:00
Aleksei Sviridkin	bf14a73b39	feat(api): add SecurityGroup network policy resource (sdn.cozystack.io) (#2922) ... ## What this PR does Introduces a tenant-facing `SecurityGroup` resource in a new `sdn.cozystack.io/v1alpha1` API group, served by the Cozystack aggregated API server as a 1:1 projection of a `CiliumNetworkPolicy`. A SecurityGroup attaches to one of the tenant's managed applications by reference (`spec.targetRef: {apiGroup, kind, name}`) and declares allow-list ingress/egress for that application. Tenants manage network policy for their own apps entirely through `securitygroups.sdn.cozystack.io`, without being granted any access to the `cilium.io` API group. The translation is synchronous and stateless — the API server converts each SecurityGroup into a CiliumNetworkPolicy in the same namespace on write and back on read; there is no controller. The backing policy's `endpointSelector` is **derived** from `targetRef` (matched on the application's lineage labels `apps.cozystack.io/application.{group,kind,name}`), never copied from tenant input, so the selector is machine-generated and can only ever match the referenced application's own pods in the tenant's namespace. Policies owned by this API carry an `sdn.cozystack.io/securitygroup` marker label that the storage always re-asserts and hides from the view, so it only ever surfaces or mutates its own policies and never touches platform-managed network policies. The value is self-service network policy on a Cozystack-native, RBAC-scoped surface, without exposing Cilium's version-coupled CRDs to tenants. Because the selector is derived from an authorized application reference rather than free-form tenant input, the RBAC boundary is structural, not selector-validated: a tenant cannot express network policy against arbitrary or platform-owned pods. The full design — selector derivation from lineage labels, the structural boundary, the additive-policy reality (a SecurityGroup widens a managed application's allowed traffic but cannot sever its platform-managed management plane, and equally cannot tighten below the platform baseline), the in-tree CiliumNetworkPolicy mirror that avoids a Cilium Go-module dependency, RBAC, and deferred extensions (raw-pod targeting, cluster scope, reusable groups) — is in `pkg/apis/sdn/DESIGN.md`. Implements the attachment model from #1614. ### Release note ```release-note feat(api): add the SecurityGroup resource (sdn.cozystack.io/v1alpha1) — a tenant-facing, namespace-scoped firewall that attaches to a managed application by reference and projects 1:1 onto a CiliumNetworkPolicy with a machine-derived endpoint selector, letting tenants manage their application's network policy without cilium.io access. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added the `SecurityGroup` API resource (`sdn.cozystack.io` v1alpha1) with a namespaced projection backed by Cilium network policies. * Enabled the SDN `APIService` and wired SDN routing, scheme, and REST support. * Expanded RBAC to include SecurityGroup access and required Cilium policy permissions. * **Documentation** * Added a design document describing projection behavior and rule semantics. * **Bug Fixes** * Improved API rule validation coverage for missing list-type fields. * **Tests** * Added SDN round-trip, fuzzing, RBAC, and comprehensive REST create/get/list/update/delete/watch/resource-version tests. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-06-30 14:30:23 +03:00
Aleksei Sviridkin	f9d1138c6e	feat(keycloak): configurable login theme via platform branding (#3124) ... ## What this PR does Makes a realm's Keycloak login theme configurable through platform `branding` values, closing a gap: `branding` could set the realm display name but not its login theme, and the keycloak chart's `themes:` list was not reachable from platform values. Two new keys under `branding` (propagated via `_cluster.branding`, the same path that already feeds `displayName`): - `branding.loginTheme` — keycloak-configure renders it as `ClusterKeycloakRealm.spec.themes.loginTheme`, so the realm selects that login theme. - `branding.loginThemeImage` — when set together with `loginTheme`, the keycloak chart appends `{name: loginTheme, image: loginThemeImage}` to its effective theme list, so the existing init-container mounts the theme image into `/opt/keycloak/themes/`. No StatefulSet patching or per-package values plumbing. `loginTheme` alone (no image) still selects the theme — for themes already baked into the Keycloak image. Explicit chart `themes:` entries keep working; the branding-derived entry is appended after them. Flow: platform `branding` → `_cluster.branding` (cozystack-values Secret) → keycloak-configure (realm loginTheme) + keycloak (theme init container). Covered by helm-unittest in both charts. ### Screenshots No bundled UI assets — this PR adds the mechanism only; the login theme is operator-supplied via `loginThemeImage`, so the visual result depends on that image. ### Release note ```release-note feat(keycloak): select the realm login theme and ship its theme image through platform `branding` values (`branding.loginTheme`, `branding.loginThemeImage`) ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added support for configuring a custom login theme from platform branding, including automatic selection in the realm setup. * Theme entries can now be combined with existing theme settings and rendered in the expected order. * **Documentation** * Expanded configuration guidance for branding and theme-related settings with clearer examples. * **Tests** * Added coverage for login theme rendering and fallback behavior when theme settings are missing. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-06-30 12:29:52 +03:00
Aleksei Sviridkin	31c15b282f	feat(network): mechanism-agnostic LoadBalancer exposure (ExposureClass + ServiceExposure) (#3081) ... ## What this PR does Lays the foundation for migrating tenant LoadBalancer exposure off the deprecated `Service.spec.externalIPs` (KEP-5707) and, eventually, off MetalLB onto Cilium-native announcement — the mechanism-agnostic shape prescribed when the earlier chart-side approach (#2468) was reverted in #2603. It adds a new `network.cozystack.io/v1alpha1` API group and a controller that translates intent into whichever LoadBalancer backend the cluster admin configured: - `ExposureClass` (cluster-scoped, admin): binds a logical class name to a backend (`externalIPs` | `metallb` | `cilium` | `robotlb`) and its VIP address scope — the StorageClass/IngressClass pattern. - `ServiceExposure` (namespaced, tenant intent): "expose this Service via class X"; status reports the resolved backend and assigned IPs. - A `serviceexposure` controller renders the backend's pool + announcer once per class (MetalLB `IPAddressPool` + `L2Advertisement`; Cilium `CiliumLoadBalancerIPPool` + `CiliumL2AnnouncementPolicy`), scoped to the namespaces that opted in. Backends are pluggable behind a small Go interface, so adding BGP or another cloud LB is one implementation, not an API change. The controller never mutates the target Service — the chart still owns its shape. Platform wiring: `publishing.exposureClass.*` renders the admin `ExposureClass` and the host ingress publishes through it. The default (no class set, `externalIPs`) leaves every existing cluster's behaviour byte-for-byte unchanged; the MetalLB→Cilium announcer migration becomes a single `ExposureClass.spec.backend` change. This is the foundation only — Part of #2586. Converting per-app exposure, flipping Cilium `l2announcements` on, changing the default, and removing MetalLB are deferred to later phases. ### Release note ```release-note feat(network): add ExposureClass and ServiceExposure for mechanism-agnostic LoadBalancer exposure. Opt-in foundation; the default externalIPs publishing path is unchanged. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added new `ExposureClass` and `ServiceExposure` custom resources with support for multiple backends (externalIPs, MetalLB, Cilium, RobotLB), including optional L2 and address configuration. * Added Helm wiring and templates to enable service/ingress publishing via `ServiceExposure`, selecting a configured (or default) `ExposureClass`. * **Bug Fixes** * Improved reconciliation, finalizer-based cleanup, and status updates for readiness and assigned IP reporting, including safer handling when backends or managed resources change. * **Tests** * Added end-to-end and unit test coverage for backend rendering, readiness, and cleanup flows. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-06-30 12:29:48 +03:00
Myasnikov Daniil	b9749cec80	fix(platform): md0 migration -- fail closed and preserve the no-delete label ... Migration 47 pins the implicit default md0 node group before the kubernetes chart upgrade so the live md0 MachineDeployment is not pruned. Two correctness fixes: - Stamp the cozystack-version ConfigMap via the shared stamp_cozystack_version helper instead of `kubectl create configmap | kubectl apply`. The label-less apply by the same field manager stripped the platform.cozystack.io/no-delete label migration 42 added, dropping cozystack-version out of the cozystack-no-delete-guardrail policy; the helper re-emits the label and is enforced for every migration >= 42 by hack/cozystack-version-stamp.bats. - Fail closed. A non-NotFound read error is now counted as a failure rather than silently skipped, and any failure exits 1 so the pre-upgrade hook aborts before Helm rolls out the new chart against clusters whose md0 is not yet pinned. The version stamp stays at 47 and the migration retries next upgrade. Also narrow the merge patch to only md0 so a concurrent edit to a sibling node group between the read and the patch cannot be reverted. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-06-30 12:08:17 +05:00
mattia-eleuteri	602b8d06fd	fix(platform): add migration 47 to pin implicit md0 node group ... After the kubernetes chart stops force-merging a default md0 node group, clusters that relied on the implicit md0 (defined other nodeGroups without md0) would have their live md0 MachineDeployment pruned by Helm on the next reconcile. Migration 47 makes md0 explicit on those clusters to preserve the current node topology; it is idempotent, treats a missing CRD as a no-op, and skips clusters that already define md0. Signed-off-by: mattia-eleuteri <mattia@hidora.io>	2026-06-30 12:08:17 +05:00
Aleksei Sviridkin	60c9b39578	docs(platform): document branding keys including login theme ... The branding map shipped as an undocumented empty map. Describe the keys it actually supports (brandName, brandHtmlName) and the new login-theme keys (loginTheme, loginThemeImage) consumed by the Keycloak charts, with an example. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-06-28 05:08:47 +03:00