mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-09 14:02:56 +09:00
8f4c19daaeb00a2aeccf761cd96879867a4d86ce
5128
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Andrey KolkovandClaude Fable 5
|
1ea7ef54c0 |
docs(backups): correct the CRD-upgrade and barman-binaries claims
Review follow-ups: the vendored cloudnative-pg chart ships the CNPG CRDs as ordinary release manifests (crds.create), so helm upgrade updates them with the operator — the out-of-band kubectl apply claim was wrong (the historical skew came from the image.tag pin, which this PR removes). And the missing barman-cloud-* binaries only affect the standard image variant keycloak-db pins; apps/postgres pins bare system-flavor tags that still ship them — its breakage was the operator/CRD skew. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey KolkovandClaude Fable 5
|
f2f39d8f8c |
test(postgres): materialise restore-target backup Secrets in the example harness
The CNPG strategy template renders <app>-cnpg-backup-creds / -ca against whichever application it drives, so a to-copy restore looks the pair up under the TARGET's name. The harness only created the source's pair; on a cluster with stale target Secrets from an earlier run the restore failed TLS verification against the wrong CA. Materialise the pair for the target too (kubectl apply, so stale content is corrected) and delete both pairs in cleanup. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey KolkovandClaude Fable 5
|
67db8f7da7 |
fix(backups): make the barman-cloud serverName stable across every flow
Two defects of the same class — the WAL-archive S3 prefix (serverName) changing or defaulting differently between components: Recovery default regression (review blocker): native CNPG defaulted an absent externalClusters serverName to the entry name (= bootstrap.oldName), but the barman-cloud plugin defaults it to the NEW cluster's name, so a legacy-flow restore with bootstrap.serverName unset would read the wrong S3 prefix (hard failure, or silently restoring another cluster's data). The chart now always emits serverName, defaulting to bootstrap.oldName. Live-cluster serverName flip (found by the gated postgres backup e2e on a real cluster): the chart renders spec.plugins without serverName (effective prefix = cluster name) while a strategy template may name a different one; the driver's ForceOwnership patch then flipped the prefix mid-stream, splitting the WAL archive across two prefixes — the WAL holding the backup's start was lost between them and the restore failed with "WAL not found". The driver now preserves the live Cluster's effective serverName (explicit parameter, or the cluster name when the plugin is attached without one) and only applies the strategy's value to a Cluster with no plugin attached; both charts pin serverName explicitly, and the example strategy no longer sets one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey KolkovandClaude Fable 5
|
8967205353 |
docs(backups): document the mixed-version BackupJob window caught by the e2e
Running the gated postgres backup e2e against a cluster whose apps still
render the pre-upgrade chart surfaced the CNPG webhook rejection
("Cannot enable a WAL archiver plugin when barmanObjectStore is
configured"): the driver cannot attach spec.plugins while the native
config is still present. Document the window and its resolution in the
upgrade notes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
|
||
|
Andrey KolkovandClaude Opus 4.8
|
bdc0b564e2 |
test(postgres): add gated barman-cloud backup/restore e2e round-trip
Port examples/backups/postgres into an automated e2e, following the etcd backup precedent: the numbered example manifests become a runnable harness (00-helpers.sh, run-all.sh, cleanup.sh) that a human reads and the test drives, so the documented flow and the test can never drift. run-all.sh provisions a Bucket, deploys the source Postgres (writing a sentinel row), derives the two Secrets the barman-cloud ObjectStore references from the Bucket's BucketInfo and seaweedfs' CA, submits an ad-hoc BackupJob and waits for Succeeded, then restores to a copy and asserts the sentinel round-tripped through S3 — the in-cluster witness that a plugin backup is both taken and restorable end to end. A second Test document (postgres-2-backup-roundtrip) in the existing hack/e2e-chainsaw/postgres suite drives run-all.sh, gated out of CI behind POSTGRES_E2E_S3_ROUNDTRIP=1 (kind has no reachable, CA-trusted S3 endpoint, and a tenant Pod cannot cross the Cilium egress policy to tenant-root's seaweedfs). Placing it in the postgres suite (rather than a new dir) keeps it triggered by packages/apps/postgres changes under Test Impact Analysis. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey KolkovandClaude Opus 4.8
|
a1c135c2a1 |
fix(backups): omit empty retentionPolicy from the ObjectStore
Review follow-up (cozystack#3300, non-blocking): the barman-cloud ObjectStore CRD
validates spec.retentionPolicy against ^[1-9][0-9]*[dwm]$, so an operator who
explicitly sets backup.retentionPolicy="" would render retentionPolicy: "" and
the API server would reject the ObjectStore, breaking backup provisioning. Guard
the field in the apps/postgres and keycloak chart ObjectStores with
{{- if .Values.backup.retentionPolicy }} (the Go driver path is already safe via
omitempty). Add a helm-unittest asserting an empty retentionPolicy is omitted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
|
||
|
Andrey KolkovandClaude Opus 4.8
|
604e79437b |
docs(backups): disclose CNPG-bump + LINSTOR scheduler change; fix stale e2e fixture
Review follow-up (cozystack#3300): - docs/operations/backup-classes.md: add an operator-facing upgrade note for the two platform-wide changes this migration carries — the CloudNativePG operator bump to 1.28.1 (with the CRD-skew fix and the Helm-CRD-upgrade caveat) and the LINSTOR scheduler admission-webhook removal, which makes data-locality scheduling advisory cluster-wide (spec.schedulerName no longer auto-assigned; opt in via explicit schedulerName). These affect more than Postgres backups and were previously only noted in a chart values.yaml comment. - hack/e2e-chainsaw/postgres/postgres.yaml: drop stale restic-era backup fields (s3Region, s3Bucket, cleanupStrategy, resticPassword) that no longer exist in the Postgres backup schema; use current fields (destinationPath, endpointURL, retentionPolicy, 6-field schedule). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey KolkovandClaude Opus 4.8
|
5fbc90047d |
docs(apps/postgres): modernize backup annotations/README for the plugin flow
Review follow-up (cozystack#3300): the apps/postgres user-facing docs still described the removed native spec.backup.barmanObjectStore / archive_command model. Update the values.yaml @field/@typedef/@param annotations (useSystemBucket, schedule, S3CredentialsSecret, EndpointCA, endpointCA, the @param backup note, bootstrap.serverName) and the hand-written README Operations prose to the barman-cloud plugin + ObjectStore model, then regenerate values.schema.json, the README parameters table, the api types, and the postgres-rd ApplicationDefinition schema (which feed the dashboard form help + cozyrd schema). Field names, defaults, and the storageClass immutability marker are unchanged; only backup/ bootstrap description text changed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey KolkovandClaude Opus 4.8
|
268ec29856 |
test(backups): assert driver-initiated Backup uses method=plugin
Review follow-up (cozystack#3300): ensureCNPGBackup stamps spec.method=plugin + spec.pluginConfiguration on the postgresql.cnpg.io/Backup it creates — the field that routes a driver-initiated run through the barman-cloud plugin. If it regressed to the default (barmanObjectStore) method, CNPG would run the legacy method against a Cluster that no longer has one and every platform backup would fail. Add TestEnsureCNPGBackup_UsesPluginMethod asserting method, plugin name, and cluster ref. (The chart ScheduledBackup path was already covered; the driver path was not.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey KolkovandClaude Opus 4.8
|
2212367842 |
fix(backups): own driver-created ObjectStore + correct stale barman references
Review follow-ups (cozystack#3300): - Lifecycle/leak: applyClusterPluginBackup now sets an ownerReference from the driver-created barmancloud.cnpg.io/ObjectStore to its cnpg.io/Cluster, so Kubernetes garbage-collects it when the Cluster is deleted (tenant app teardown or the RestoreJob purge). In the platform useSystemBucket flow the chart does not render this ObjectStore, so previously it was orphaned in the tenant namespace. Drop the now-accurate-only 'delete' verb on objectstores from the driver RBAC (GC handles removal; the driver never deletes it) and correct the RBAC comment that falsely claimed the purge deletes it. - Diagnostics/comments: the RestoreJob WAL-archive-deadline failure message and several doc comments still told operators to check spec.backup.barmanObjectStore / externalClusters[].barmanObjectStore, fields no longer emitted after the migration. Point them at the barman-cloud plugin (spec.plugins + ObjectStore, externalClusters[].plugin) instead so they don't mislead during an incident. - Test: TestApplyClusterPluginBackup_PatchesExistingCluster now asserts the ObjectStore carries an ownerReference to the Cluster (kind/name/uid). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey KolkovandClaude Opus 4.8
|
302a0193f1 |
docs+test(backups): update upgrade notes for the plugin flow + cover recovery rendering
Review follow-ups for the CNPG barman-cloud plugin migration (cozystack#3300): - docs/operations/backup-classes.md: the 'Upgrade notes from chart-managed backups' section still described the removed native spec.backup.barmanObjectStore behavior. Rewrite it for the plugin flow (chart renders spec.plugins + a barmancloud.cnpg.io/ObjectStore; the driver SSA-applies an ObjectStore and patches spec.plugins), and document the real upgrade hazard: the plugin-barman-cloud operator + cert-manager must be present in cozy-postgres-operator or the Cluster references an unregistered plugin and backups/WAL archiving stall. - apps/postgres backup_storage_test.yaml: add a positive bootstrap-recovery rendering test. Asserts externalClusters[].plugin references barman-cloud.cloudnative-pg.io with barmanObjectName=<release>-recovery and serverName on the plugin parameter, and that the -recovery ObjectStore carries spec.configuration but NOT serverName (the plugin forbids it there). Previously the recovery path had only fail-fast guard tests, no positive coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey KolkovandClaude Opus 4.8
|
39bd8623c7 |
fix(keycloak): back up keycloak-db via the barman-cloud plugin
The keycloak-db backup added in #3174 uses the in-tree native spec.backup.barmanObjectStore, whose barman-cloud-* binaries ship only in the CNPG system image variant; keycloak-db is pinned to the standard variant, so base backups fail with 'barman-cloud-backup: executable file not found'. The proposed workaround (#3306) switches keycloak-db to the deprecated system image variant when backups are enabled. Migrate keycloak-db to the barman-cloud plugin instead, matching the rest of this PR: render spec.plugins referencing a barmancloud.cnpg.io ObjectStore (the S3/barman config moves there) and a method=plugin ScheduledBackup, keeping the Cluster on the standard image (the plugin runs the barman tooling in a sidecar). This removes the need for the system-variant workaround entirely, so #3306 can be closed. cozystack#3300. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey Kolkov
|
13ff340277 |
fix(backups): migrate CNPG backups to the barman-cloud plugin
Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Myasnikov Daniil
|
90254efa6d |
fix(ci): resolve nightly packages digest without image-specific skopeo ops
The Nightly workflow has never completed a run. Every scheduled run since
NIGHTLY_ENABLED was set has failed in `mirror` at "Resolve source commit
and version", ~80s in:
level=fatal msg="unsupported image-specific operation on artifact with
type \"application/vnd.cncf.flux.config.v1+json\""
cozystack-packages is a Flux OCI artifact, not a container image: its
config mediaType is application/vnd.cncf.flux.config.v1+json. Plain
`skopeo inspect` parses the config blob as an image config and fatals on
any non-image artifact, so resolving the digest with
`inspect --format '{{.Digest}}'` could never work.
Fetch the :main manifest once with `inspect --raw` and derive both the
digest (`skopeo manifest-digest`, which just hashes the manifest bytes —
no image-specific parsing) and the revision annotation from those exact
bytes. This also tightens the race the original comment describes: the
digest and the annotation now come from a single fetch, so there is no
window at all between pinning the tag and reading its revision, and one
registry round-trip is saved.
Because the workflow never ran past this step, the rest was unproven.
Also fix both `flux pull artifact` calls, which passed an --output
directory that does not exist and would have failed with "invalid output
path": create it first, matching the established idiom in
pull-requests.yaml ("Pull current-main packages tree").
Verified against the live OCIR artifact: the fixed step resolves
packages_ref to the same digest the registry reports and yields main's
current HEAD sha; the fixed pull extracts the tree. skopeo copy on Flux
artifacts, the flux push digest awk parse, the cozy-installer chart name
and nightly-mirror.sh's per-image digest verification were each checked
and are correct as-is.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
Aleksei Sviridkin
|
a5cabca73e |
fix(ouroboros): render controller rollout strategy in every mode (#3323)
## What this PR does Bumps the ouroboros chart and controller image to 0.8.1, which fixes a latent upgrade break in the controller Deployment's rollout strategy, and pins the fixed contract from the cozystack side. Up to chart 0.8.0 the controller Deployment rendered a rollout strategy **only** in `external-dns` mode, as `type: Recreate`, and rendered no strategy block at all in the other modes. The apiserver defaults a `rollingUpdate` block onto any Deployment created without one, so switching a live release into `external-dns` laid `Recreate` on top of that defaulted block, and the apiserver rejected the object: ``` spec.strategy.rollingUpdate: Forbidden: may not be specified when strategy `type` is 'Recreate' ``` 0.8.1 renders the same `RollingUpdate` block, with no surge, in every mode. The strategy no longer changes when `controller.mode` does, so the rejected combination cannot arise on any transition. Rendering `Recreate` unconditionally would not have worked: existing `coredns`-mode releases already carry the defaulted `rollingUpdate` block and would hit the identical rejection on their first upgrade. Always-`RollingUpdate` is upgrade-safe from both existing states, and rollback works for the same reason. **Reachability, stated plainly:** this is a latent break, not an active one. The host pins `coredns` and tenants pin `coredns-import`, neither of which rendered a strategy in 0.8.0, and no shipped configuration selects `external-dns`. But `controller.mode` is a documented, user-settable value that the tenant wrapper exposes through `valuesOverride`, so the transition is reachable by configuration rather than only in theory. **Why it went unnoticed:** the break only surfaces under server-side apply, which helm-controller uses by default. `DeploymentSpec.Strategy` carries `patchStrategy:"retainKeys"`, so a client-side strategic merge clears `rollingUpdate` on the way through and a plain `helm upgrade` never reproduces it. Only the upgrade breaks, never the install, so nothing catches it at render time either. **Why no surge, in every mode:** the controller has no leader-election in any mode and no serving path — no ports, no probes — so surging a second pod buys no availability anywhere. In `external-dns` mode it also removes a real hazard: that reconciler `Update`s DNSEndpoints with no conflict retry, so two overlapping reconcilers race on Get→Update. The `coredns` paths wrap Get→Update in `RetryOnConflict` and converge instead, which is why only `external-dns` rejects `replicaCount > 1`; that asymmetry is deliberate and is now called out in the template so it does not get tidied away later. The upstream comment is explicit that `maxSurge: 0` is *not* Recreate-equivalent exclusivity — the surge budget counts desired replicas, not live pods — and rests the argument on what is actually guaranteed: the old pod's SIGTERM precedes the new pod's creation, and SIGTERM cancels the reconciler context. The proxy Deployment is untouched and keeps its default rolling surge, as the data-path component. The fix was made upstream in `lexfrei/ouroboros` and released as v0.8.1 rather than patched into the vendored tree here, since `make update` regenerates `charts/` wholesale. The vendored chart in this PR is a verbatim `make update` pull of the published 0.8.1 artifact, byte-identical to the registry. Chart manifest digest and controller image digest are bumped in lockstep with the version, as the package Makefile requires. 0.8.1 also carries an upstream Renovate bump of the cleanup-hook kubectl image to 1.36.1 (verified to resolve, multi-arch). The `update` target additionally drops the empty digest-named directory that `helm pull` leaves beside the untarred chart when pulling by digest. Git ignores empty directories, which is exactly why it lingered unnoticed. ### Testing `make -C packages/system/ouroboros test` → 17/17 pass. The three new cases pin the rendered strategy on `coredns`, `coredns-import` and `external-dns`. `external-dns` is included deliberately: it is the only mode that ever rendered `Recreate`, so a future bump that re-gated the block would slip past a coredns-only suite. Verified by re-gating the bug surgically against the new suite — it fails on exactly that case, and passes without it. Upstream additionally tests this contract across every mode in its own suite. ### Screenshots Not applicable — no UI change. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff file by file. The diff touches only `packages/system/ouroboros/`: the package Makefile's version and digest pins, the vendored chart (regenerated), the image pin in `values.yaml`, and the package's own tests. It adds, renames and removes no package under `packages/apps/` or `packages/extra/`; touches no `packages/core/platform/values.yaml` key, no platform variant, bundle or component; changes no `ApplicationDefinition`, no `values.schema.json`, no version enum, no `release.prefix`; touches nothing under `hack/`, no namespace, and no node prerequisite. No trigger matches. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(ouroboros): render the controller rollout strategy in every controller.mode. Up to chart 0.8.0 the strategy was rendered only in external-dns mode, as type: Recreate, so switching a live release into that mode applied Recreate on top of the rollingUpdate block the apiserver defaults onto a Deployment created without one, and the upgrade was rejected with "spec.strategy.rollingUpdate: Forbidden: may not be specified when strategy `type` is 'Recreate'". The rejection only occurs under server-side apply, which helm-controller uses by default. The strategy is now identical across all modes, so a mode switch changes no strategy field. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Updated Ouroboros to version 0.8.1 with refreshed, pinned images. * Improved controller rollouts using a consistent rolling-update strategy across supported modes. * Updated cleanup hook tooling to version 1.36.1. * **Bug Fixes** * Prevented leftover chart directories from accumulating during chart updates. * Improved deployment behavior by limiting rollout disruption while maintaining controller availability. * **Tests** * Added coverage validating rollout behavior across all supported controller modes. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
90342063c0 |
chore(github): add area/keycloak, area/tenant, area/kamaji labels
Three scopes recur across PR history with no fitting area/* label (keycloak, tenant, kamaji). Add the labels and map their scopes in the auto-labeler, and fill two gaps in existing mappings (mongodb -> area/database, csi -> area/storage) so these stop landing in area/uncategorized. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
4cbfb31348 |
chore(github): auto-label new issues triage/needs-triage
Every issue created through a template now gets triage/needs-triage so the untriaged backlog is visible at a glance and the stale bot's 60-day clock has a clear signal for maintainers to act on. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
myasnikovdaniil
|
4a00b502d7 |
fix(seaweedfs): adopt legacy workloads in place across the 4.31 rename (#3282)
## What this PR does Fixes a v1.5.0 upgrade regression where a tenant ends up with **two** SeaweedFS workload sets. **Root cause.** v1.5.0 bumped the vendored SeaweedFS chart `4.0.405 → 4.31.0`. Upstream changed resource naming: before 4.31 the chart named workloads after the **chart** (`seaweedfs-*`, ignoring the release name), and 4.31 names them after the **release**. The data-plane release is `<name>-system`, so every StatefulSet wanted to become `seaweedfs-system-*`. StatefulSet names are immutable, so Helm could not rename in place — it stood up a second, empty set beside the running one while the data stayed on the original `data1-seaweedfs-volume-*` PVCs. Where the node count let both sets run, they shared one `seaweedfs-s3` Service (identical pod labels) and one `seaweedfs-db` filer metadata database, so the empty set answered reads and the two master sets mixed volume IDs from independent sequences into one metadata table. **Fix.** Pin `fullnameOverride` in `system/seaweedfs` values so the rendered names stop following the release name, which is the convention already used by `cozy-proxy`, `flux-operator`, `grafana-operator`, `linstor-scheduler` and `victoria-metrics-operator`. The names go back to what every pre-1.5 cluster already runs, so the upgrade adopts the running workloads **and their volumes** in place. No data migration, no platform migration. **The one case that cannot be adopted.** A tenant installed *fresh* on 1.5.x wrote its data under the renamed workloads, so its volumes are not the chart-named ones. Pinning the chart name there would rename the workloads *away* from that data and bring up an empty cluster beside it. Helm cannot move data between PVCs, so `extra/seaweedfs` refuses to render for such a tenant and points at the recovery runbook, which re-binds the existing PVs onto the adopted PVC names (no copying — the claim is renamed, the volume is not touched). The guard is fail-closed: it fires only when renamed volumes are present and no adopted ones are. Two naming subtleties the guard has to respect, both verified by rendering the shipped 1.5 chart: - Before 4.31 the chart ignored the release name, so an instance running under a non-default name (`SeaweedFS` is a user-creatable kind) *also* stored its data on `data1-seaweedfs-volume-*`. Such a tenant is adopted, not blocked. - On 4.31 the fullname helper appends the chart name when the release name does not contain it, and the component helper truncates to 56 characters. A fresh 1.5.x instance named `archive` therefore wrote to `data1-archive-system-seaweedfs-volume-*`, and a long name loses the chart name from its tail entirely. The guard matches volumes by shape and by the chart labels on the StatefulSet instead of reconstructing those names, so both are caught. **On the duplicate set.** Once the fix reconciles, the `seaweedfs-system-*` StatefulSets and Deployments are no longer part of the release and Helm removes them on its own. What it cannot remove — PVCs it did not template, and cert-manager Secrets that carry no owner reference — is inert, and the runbook clears it explicitly. A pre-upgrade migration was considered and dropped: it would duplicate what Helm already does, while its only unique action (deleting PVCs) cannot distinguish "the duplicate never served" from "it served and is currently down", and any transient delete failure under `set -e` would fail the platform's pre-upgrade hook and block delivery of this very fix. ## Testing `helm unittest` — `system/seaweedfs` 21 tests, `extra/seaweedfs` 17 tests. The guard's `lookup` branches are covered by mocking the Kubernetes provider (the pattern already used in `core/platform` and `system/cozystack-api`): adopted-only, renamed-only, both sets, zoned volumes, a non-default instance name, a truncated long name, and an unrelated `data1-*-volume-*` PVC that must not trip the guard. The s3-consumer suite now re-creates the name/fullname divergence explicitly, so it keeps guarding that regression regardless of the shipped default. The remaining regression test for this class of bug is an upgrade E2E (seed a pre-4.31 named set, upgrade, assert a single set with data intact) — no fresh-install E2E or `helm template` can reproduce an upgrade over pre-existing state. That belongs in the chainsaw upgrade lane and is a follow-up. ### Screenshots N/A — no UI changes. ### Release note ```release-note fix(seaweedfs): adopt existing SeaweedFS workloads and volumes in place when upgrading past the 4.31 chart rename, instead of standing up a second, empty cluster beside them ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added an audit & recovery runbook for SeaweedFS 4.31 rename impacts. * **Bug Fixes** * Stabilized SeaweedFS workload/PVC naming across upgrades, including non-default instance names. * Added preflight safeguards to prevent upgrade scenarios that could split or wedge existing data. * Updated post-delete cleanup to remove secrets from both legacy and current naming schemes. * **Tests** * Expanded Helm unit coverage for naming guards, cleanup behavior, and S3 service name consumers. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
0238231303 |
fix(linstor-scheduler): re-vendor to upstream 0.3.1 with native admission webhook (#3201)
## What this PR does Re-vendors the `linstor-scheduler` chart from upstream **0.3.1** (appVersion **v0.3.6**). Upstream now ships the admission webhook natively (contributed via [piraeusdatastore/helm-charts#94](https://github.com/piraeusdatastore/helm-charts/pull/94)), so the loose downstream admission templates — which round-tripped every Pod through an old `k8s.io/api` and silently stripped fields they did not know about, the `image` volume source and the native-sidecar `restartPolicy` among them, breaking KubeVirt containerDisk VMs and native sidecars cluster-wide — are dropped in favour of the upstream copy. - The admission webhook comes from the upstream chart now; the previous downstream copy and its patch are removed. `make update` pulls 0.3.1 and applies only `cozystack-scheduler-deployment.patch` (the scheduler pod component label and the distro-suffix-stripped kube-scheduler image tag, neither of which is upstream yet), so vendoring stays deterministic and reproducible. - The upstream admission Deployment keeps the same name but changes its immutable pod selector, so an in-place upgrade of an existing install would fail with `field is immutable`. Migration 52 deletes the old admission Deployment before the upgrade so the new one is recreated cleanly (`targetVersion` 52 → 53). - appVersion v0.3.6 also carries the webhook's TLS certificate hot-reload fix ([piraeusdatastore/linstor-scheduler-extender#22](https://github.com/piraeusdatastore/linstor-scheduler-extender/pull/22)): the webhook used to load its certificate once at startup, so after the mounted Secret was rotated it kept serving a stale certificate and — with `failurePolicy: Ignore` — silently stopped mutating until it was restarted. It now reloads on rotation, so no external cert-reload workaround is needed here. - Wrapper values keep `replicaCount: 2`, now that the admission Pods have their own PodDisruptionBudget and no longer share the scheduler's. - Adds a helm-unittest suite covering the admission resources, PodDisruptionBudget selector isolation, least-privilege RBAC, the TLS provisioning modes and their rejection paths, the scheduler image-tag strip, and backward compatibility with the webhook disabled. Closes #3198. ### Screenshots N/A — no UI changes. ### Downstream repositories I walked the trigger map against the diff, file by file. It touches `packages/system/linstor-scheduler/` and, under `packages/core/platform/`, only migration 52 and the `migrations.targetVersion` bump — an internal, release-managed counter, not a user-facing `spec.components.platform.values.*` key documented on the website. No trigger matches. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(linstor-scheduler): the LINSTOR scheduler admission webhook no longer strips unknown Pod fields (the `image` volume source and native-sidecar `restartPolicy`), which had broken KubeVirt containerDisk VMs and native sidecars, and it now reloads its TLS certificate after rotation instead of silently failing. The chart is re-vendored from upstream 0.3.1; an automatic upgrade migration replaces the admission Deployment whose immutable selector changed. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Expanded the optional Linstor Scheduler admission webhook with richer configuration, including automated `schedulerName` mutation for LINSTOR CSI-backed PVC Pods. * **Improvements** * Updated the Linstor Scheduler Helm chart to **0.3.1** (scheduler app **v0.3.6**), including updated admission webhook behavior, safer defaults, and **2 replicas**. * Platform migrations now advance to **version 53**, with upgrade-safe cleanup of the existing admission deployment. * **Documentation** * Updated Helm README and post-install notes with detailed guidance for admission webhook and TLS modes. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
fbbe4af7df |
refactor(network): drop ExposureClass/ServiceExposure for native loadBalancerClass (#3218)
## What Removes the `network.cozystack.io` API group — `ExposureClass` + `ServiceExposure` (#3081) — and its reconciler, in favor of native `Service` `type: LoadBalancer` + `loadBalancerClass`. ## Why Full rationale and discussion in #3164. For **managed applications** the chart already owns the Service, so: - `type: LoadBalancer` + `loadBalancerClass` + an admin-provisioned pool covers vendor-neutral external exposure without a dedicated API group and controller; - the `externalIPs` (node-IP) default never went through `ServiceExposure` (its backend is a no-op) and is left unchanged; - the only current consumer, the host ingress, keeps rendering `type: LoadBalancer` and just drops the `ServiceExposure`; the admin provisions the address pool / announcer out of band, as the install docs already assume. `ServiceExposure`'s one unique feature — dynamic per-namespace scoping of a shared VIP pool — is a convenience the managed-app path does not need, and it does not justify a second tenant-networking API group next to `sdn.cozystack.io`. ## Changes - Delete `api/network/v1alpha1` and `internal/controller/serviceexposure` (+ CRD definitions, e2e bats, controller registration, RBAC). - Host ingress: drop the `ServiceExposure` template; add an optional `publishing.loadBalancerClass` that sets `spec.loadBalancerClass` on the host-ingress LoadBalancer Service. - The `externalIPs` path and default behavior are unchanged. ## Verification - `go build ./...` — green. - `helm unittest` — `core/platform` 72/72, `extra/ingress` 25/25 (incl. host-vs-tenant `loadBalancerClass` scoping and the immutable-field migration guard). ## Migration None required. `network.cozystack.io` (#3081) has never shipped in a release — it is on `main` only (not backported to `release-1.5`; absent from v1.4.x / v1.5.x), so no upgrading cluster has these CRDs and this removes it before it is ever released. Clusters that track `main` and explicitly set `publishing.exposureClass` (opt-in, only since 2026-06-30) should provision their MetalLB / Cilium address pool themselves and set `publishing.loadBalancerClass`; any orphaned `cozystack-<class>` pool can be deleted by hand. Default clusters (externalIPs path) are unaffected. ## Status **Draft / RFC for the #3164 discussion** — this removes an API group that landed only days ago, so nothing here should merge before the design call. cc @lexfrei (author of #3081) and @lllamnyp (#3164). Deferred: when Kubernetes actually removes `Service.spec.externalIPs`, the node-IP default migrates via a MetalLB `IPAddressPool` with no `L2Advertisement` (allocate-but-don't-announce) — a separate change. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added support for configuring the ingress host Service with a native `loadBalancerClass` value. * Expanded chart tests to verify class wiring and immutability behavior across install and upgrade scenarios. * **Bug Fixes** * Removed the legacy exposure-class/service-exposure flow and related controller behavior. * Updated templates and CRD packaging so the platform no longer renders or depends on those deprecated resources. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
f4aef73053 |
fix(linstor-scheduler): re-vendor to upstream 0.3.1 with native admission webhook
Re-vendor the linstor-scheduler chart from upstream 0.3.1 (appVersion v0.3.6), which carries the admission webhook binary that stopped stripping unknown Pod fields — the image VolumeSource and native-sidecar restartPolicy, which broke KubeVirt containerDisk VMs and native sidecars cluster-wide — and now reloads its TLS certificate on rotation instead of serving a stale one until restarted. The admission webhook ships in the upstream chart, so the previous downstream copy is dropped: `make update` pulls it directly and applies only cozystack-scheduler-deployment.patch (pod component label and a distro-suffix-stripped image tag), keeping the vendoring deterministic. Because the webhook now reloads its own certificate, the external cert-reload workaround is dropped. The upstream admission Deployment keeps the same name but changes its immutable pod selector, so an in-place upgrade of an existing install would fail. A migration deletes the old admission Deployment before the upgrade so the new one is recreated cleanly, and bumps the migration target version. Add a helm-unittest suite covering the admission resources, PDB selector isolation, least-privilege RBAC, the scheduler image tag, and backward compatibility. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
d542f66368 |
fix(ouroboros): render controller rollout strategy in every mode
Bump the chart and controller image to 0.8.1, which renders the controller Deployment's rollout strategy in every controller.mode instead of only in external-dns mode. Up to 0.8.0 the chart rendered a strategy only in external-dns mode, as type: Recreate, and nothing in the other modes. The apiserver defaults a rollingUpdate block onto a Deployment created without one, so switching a live release into external-dns laid Recreate on top of that defaulted block and the apiserver rejected the object: spec.strategy.rollingUpdate: Forbidden: may not be specified when strategy `type` is 'Recreate' The strategy is now identical across all modes, so no mode switch changes a strategy field at all. This is a latent break rather than an active one: the host pins coredns and tenants pin coredns-import, neither of which rendered a strategy in 0.8.0, and no shipped configuration selects external-dns. But controller.mode is user-settable and the chart supports the mode, so the transition is reachable by configuration. It surfaces only under server-side apply, which helm-controller uses by default, so no local helm upgrade reproduces it: a client-side strategic merge carries retainKeys on strategy and clears rollingUpdate on the way through. The rendered strategy is pinned on all three modes, external-dns included — it is the only mode that ever rendered Recreate, so a bump that re-gated the block would otherwise slip past a coredns-only suite. 0.8.1 also carries an upstream bump of the cleanup-hook kubectl image to 1.36.1. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Myasnikov Daniil
|
f1619215ff |
docs(api): note disableWait silently defeats healthCheckExprs
The field docs already say healthCheckExprs are only evaluated under the poller wait strategy, but that is only half the precedence rule. Upstream evaluates them only when the Helm action itself has wait enabled (helm-controller/api@v1.5.1 v2/helmrelease_types.go:193-199), so the release.cozystack.io/helm-install-disable-wait annotation makes them a silent no-op whatever waitStrategy says — no error, no warning, the release just reports Ready as before. This is reachable today: packages/system/kubernetes-rd already sets that annotation and is a plausible candidate for CEL expressions in phase 3. Document it on the ApplicationDefinition field (so it reaches the CRD description) and next to HelmInstallDisableWaitAnnotation. ComponentInstall has no disableWait, so its field doc is unchanged. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
b916d7475b |
fix(api): declare OpenAPIModelName for apps types so SSA resolves catalog kinds
After the k8s 0.35 bump, server-side apply of any apps.cozystack.io kind (e.g. Bucket) fails at field management with "no corresponding type for apps.cozystack.io/v1alpha1, Kind=Bucket", blocking backupstrategy-controller and the whole install. The apps API multiplexes every catalog kind (Bucket, Postgres, …) onto the single Application Go type via Scheme.AddKnownTypeWithName. The SSA field manager attaches a kind's GroupVersionKind to the Application OpenAPI model only when the model's name matches the name the apiserver's DefinitionNamer computed for it. Since Kubernetes 0.35 the DefinitionNamer keys that lookup by Scheme.ToOpenAPIDefinitionName (the "friendly" reversed-path form, com.github.cozystack…Application) while the model builder still uses the Go import-path form. With no OpenAPIModelName the two disagree, the GVK extension is never attached, and the type converter ends up with no apps kinds at all. Declare OpenAPIModelName on Application/ApplicationList/ApplicationStatus so GetCanonicalTypeName, ToOpenAPIDefinitionName and the generated openapi map key all agree on the friendly form. This is done via the +k8s:openapi-model-package package tag plus hand-written methods rather than code-generator's --output-model-name-file flag, because that flag also rewrites zz_generated.model_name.go inside the read-only apimachinery / apiextensions module-cache packages the shared gen_openapi helper always passes as inputs, which fails on any consumer (including CI) that vendors deps from the cache. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
1ea4eb452c |
fix(api): regenerate OpenAPI with code-generator v0.35.0
This PR bumps the runtime k8s libraries to 0.35 (k8s.io/apiserver,
kube-openapi 20250910) but hack/update-codegen.sh still pinned the
openapi code-generator at v0.34.1.
The 0.34.1 openapi-gen keys the version.Info definition by its Go
import path ("k8s.io/apimachinery/pkg/version.Info"), whereas
kube-openapi 0.35 now resolves that model via the new OpenAPIModelNamer
interface — version.Info{}.OpenAPIModelName() ->
"io.k8s.apimachinery.pkg.version.Info". The key mismatch makes the
aggregated apiserver fatal while building its root OpenAPI spec
("cannot find model definition for io.k8s.apimachinery.pkg.version.Info"
-> klog.Fatalf, exit 255), crashlooping cozystack-api. That takes down
the apps.cozystack.io aggregated API, so the Bucket kind can no longer
be resolved and the backupstrategy-controller HelmRelease install fails,
reddening E2E.
Bump the code-generator pin to v0.35.0 (matching the runtime) and commit
the regenerated artifacts. No hand edits to generated files.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
Myasnikov Daniil
|
96a9652fb1 |
feat(api): typed waitStrategy/healthCheckExprs for kstatus readiness
Add first-class fields so a package/app can declare the Flux helm-controller v1.5.0 kstatus health gate for the custom resource it wraps, so HelmRelease.Ready means "the CR is actually healthy" instead of "helm applied the CR" (#2642). API-owner approved in #3263. - api: WaitStrategy (scalar, maps to HelmReleaseSpec.WaitStrategy.Name — a deliberate simplification) + HealthCheckExprs ([]kustomize.CustomHealthCheck, the upstream type verbatim) on both ApplicationDefinition.spec.release (the apiserver path) and ComponentInstall (the operator path). - config.ResolveWaitStrategy: shared by both HelmRelease builders. Couples the default — healthCheckExprs imply the poller strategy when none is set, since exprs are only evaluated under poller. This removes the "must also set waitStrategy: poller" footgun and does not depend on the controller's global default. - thread through buildHelmReleaseSpec (operator) and convertApplicationToHelmRelease (apiserver); the apiserver loads the typed spec.release fields in start.go. - regenerate deepcopy, the ApplicationDefinition and PackageSource CRDs, and openapi. The backupstrategy CRDs also regenerate — benign k8s-0.35 PodSpec description drift, not a functional change. disableWait is intentionally not added (YAGNI): the only known need (the kubernetes tenant-addon deadlock) was resolved child-side, and waitStrategy: legacy is not equivalent (legacy still waits Helm-v3-style; disableWait skips waiting entirely). Add it later when a concrete leaf needs a true no-gate escape hatch. Refs #2642 Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
83f0d3bc92 |
build(deps): bump helm-controller/api to v1.5.1 for kstatus types
The kstatus readiness work (#2642) sets WaitStrategy / kustomize.CustomHealthCheck / HelmReleaseSpec.HealthCheckExprs on generated HelmReleases. Those Go types do not exist at helm-controller/api v1.4.3 — the runtime CRD is already v1.5.0 (shipped by #2602), so this closes a Go-types-vs-runtime skew rather than getting ahead of the runtime. The bump is not isolated: helm-controller/api v1.5.x requires k8s 0.35 + controller-runtime 0.23, cascading the k8s stack 0.34 -> 0.35. cozystack replaces k8s.io/apimachinery with a fork carrying the (still-open) upstream ConvertToVersion patch k/k#135537; that single commit is rebased cleanly onto apimachinery v0.35.0 at cozystack/apimachinery@release-1.35-pr135537. The repo builds with no source changes under the new stack. Refs #2642 Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
IvanHunters
|
7212e9b8c9 |
fix(backups/postgres): purge stale recovery cluster on repeat in-place restore (#3318)
## What this PR does Fixes a silent data-integrity failure in the CNPG backup driver: a repeat in-place PostgreSQL `RestoreJob` into a target that was already restored once reported `Succeeded` but did nothing. The PVC was untouched (same uid), no `*-full-recovery` pods appeared, and the database kept its pre-restore contents. Root cause: the restore patch sets `spec.bootstrap.enabled=true` on the Postgres app and never resets it, so the chart keeps rendering `spec.bootstrap.recovery` on the live cnpg.io Cluster. On a second `RestoreJob` the re-purge guard (`cnpgPurgeNeeded`) saw that leftover `bootstrap.recovery` and treated it as "an earlier reconcile of this restore already purged", skipping the destructive Cluster + PVC purge. That guard was meant only to avoid re-purging a freshly-recovered Cluster when the post-purge status write races, but it could not tell that case apart from a Cluster left over by a previous completed restore. The fix distinguishes the two by the live Cluster's `creationTimestamp` relative to the `RestoreJob`'s `StartedAt`: - created after `StartedAt`: freshly recovered by this restore's own purge, so it is skipped (unchanged behaviour for the status-write-race path); - created at or before `StartedAt`: leftover from an earlier completed restore, so it is purged and the restore re-bootstraps from the backup. Design note: an identity-based alternative (record the purged Cluster's UID on the `TargetPurged` condition and compare UIDs on later reconciles) was considered and rejected. It is immune to clock skew but leans on the same status write that the status-write-race path assumes can fail, so it cannot cover the exact case the skip exists for. The timestamp comparison needs no extra persisted state, and the fresh-vs-stale gap (a full purge plus chart re-render cycle) always dwarfs any plausible control-plane clock skew. The comparison is strict, so an exact-second tie resolves to purge (the conservative direction). ### Tests - New `cnpgClusterFreshlyRecovered` unit test: fresh / stale / boundary / missing-timestamp cases. - `TestCNPGPurgeNeeded` extended with the repeat-restore (stale recovery cluster) case that reproduced the bug. - `recoveryBootstrapClusterState` test updated to assert the returned `creationTimestamp`. - New reconcile-level regression test driving `reconcileCNPGRestore` end to end: a stale leftover recovery Cluster is purged, a freshly-recovered Cluster is not re-purged. Bug report: #3311 ### Screenshots Not applicable (no UI changes). ### Downstream repositories The diff touches only `internal/backupcontroller` (operator reconcile logic); no API types, CRDs, chart values, or docs change, so no downstream repository is affected. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(postgres): a repeat in-place restore no longer silently succeeds without restoring; a recovery cluster left over from an earlier completed restore is now purged so the restore re-bootstraps from the backup ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Bug Fixes** - Improved CNPG restore handling for repeat in-place restores. - Stale recovery clusters are now correctly purged instead of being mistaken for newly recovered clusters. - Newly recovered clusters are protected from unnecessary destructive cleanup. - Restore behavior is more reliable when recovery clusters are left over from earlier attempts. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
f6a03e93f8 |
fix(etcd): complete v1alpha2 transition for in-cluster 1.5→1.6 upgrades (#3270)
Consolidates the etcd `v1alpha2` transition fix for **in-cluster 1.5 → 1.6 upgrades** into a single PR, rebased on current `main`. Supersedes #3265 and #3261 — their commits are carried here (authorship preserved), so those PRs can be closed once this lands. Fresh installs use the new shapes directly and are unaffected, so fresh-install CI does not catch these; the 1.5 → 1.6 e2e upgrade path is the authoritative regression guard. ## What this PR does ### 1. Keep the legacy `etcd-headless` Service alive during adoption — `packages/extra/etcd` `etcd-migrate` adopts legacy clusters **in place**: the Pods keep their original `spec.subdomain: etcd-headless` and are dialed at `etcd-<i>.etcd-headless.<ns>.svc` until they roll onto the operator's native `<member>.etcd.<ns>.svc` domain. The v1alpha2 operator only creates the native `etcd` Service and the legacy `etcd-headless` Service is pruned, so those per-pod names stop resolving (`no such host`), `MemberList` fails, and `status.readyMembers` never populates — the `EtcdCluster` never goes `Ready` even though etcd is healthy and in quorum. We ship a chart-managed transitional headless `etcd-headless` Service (selector mirrors the operator's native `etcd` Service via `etcd-operator.cozystack.io/cluster`, `publishNotReadyAddresses: true`) — the DNS counterpart of the legacy `*.etcd-headless.<ns>.svc` SAN already kept for this window. Removable together with that SAN once members roll onto the native subdomain. ### 2. Survive the immutable controller-Deployment selector on upgrade — `packages/system/etcd-operator` (#3242) #2859 replaced the upstream etcd-operator chart with the cozystack-authored one, changing `Deployment.spec.selector.matchLabels`. `spec.selector` is immutable, so `helm upgrade` cannot patch the existing Deployment and the whole HelmRelease upgrade fails (`field is immutable`). A **pre-upgrade hook** (`templates/pre-upgrade-selector-fix.yaml`: ServiceAccount + Role + RoleBinding + Job) deletes the Deployment **only** when its live selector is the pre-1.6 one, so Helm recreates it cleanly. No-op when the selector already matches, never runs on fresh install. ### 3. Raise the operator's memory cold-start floor — `packages/system/etcd-operator` Steady-state working set is ~250Mi; the static `limits.memory: 128Mi` OOMKills a Pod that starts before the VPA admission webhook rewrites it. Raise the floor to `256Mi` (and VPA `minAllowed` to match) as defense in depth so the operator never depends on VPA timing to avoid crashing. ### 4. Make migration 50 (etcd adoption) robust in-cluster — `packages/core/platform/images/migrations/migrations/50` (was #3261) Exact server/peer cert-SAN match, Secret-gated wait on the adoption Secret, and in-cluster (IPv6-safe) kubeconfig handling, so the migration script drives the adoption reliably from inside the cluster. Covered by `hack/migration-50-etcd-adopt.bats`. ### 5. Hardening / review fixes (this PR's original scope) - **Hook `runAsUser: 65532`.** `pre-upgrade-selector-fix.yaml` set `runAsNonRoot: true` but no numeric `runAsUser`; `clastix/kubectl`'s image user is the non-numeric name `nonroot`, which the kubelet cannot verify against `runAsNonRoot` — so the hook Pod fails admission and silently blocks the very upgrade it exists to unblock. Adds `runAsUser: 65532`, matching the postgres-operator webhook-ready hook that runs the same image. - **Digest-pin the kubectl image.** The values comment claimed digest-pinning but shipped a floating `v1.32` tag. Reuses the digest postgres-operator vendors, adds the `renovate` annotation, and templates `repo:tag@digest`. - **Tests.** `etcd-operator/tests/selector-fix-hook_test.yaml` (hook wiring, weight ordering, namespaced least-privilege RBAC, numeric-non-root security context, digest-pinned image), `etcd-operator/tests/deployment_test.yaml` (256Mi cold-start floor), `extra/etcd/tests/etcd-cluster_test.yaml` (transitional `etcd-headless` Service). Each assertion was mutation-tested. ### 6. Derive the default S3 endpoint from the provisioned bucket — `packages/system/backupstrategy-controller` Derive the default S3 endpoint (and per-driver scheme / TLS / `secure_connection`) from the provisioned bucket Secret instead of requiring it to be hand-set, so etcd (and other) backup strategies get a working endpoint by default. Docs in `docs/operations/backup-classes.md`; covered by `tests/endpoint_form_test.yaml`. ### Verification - `helm unittest` green on current `main`: etcd-operator **18/18**, extra/etcd **18/18**, backupstrategy-controller **11/11**. - Behaviours 1 & 2 were reproduced and confirmed live on a 1.5.2 → 1.6.0-rc.1 adoption (3-node cluster) — recreating the `etcd-headless` Service took the adopted cluster to `readyMembers=3 / Available=True`. ```release-note fix(etcd): complete the v1alpha2 transition on in-cluster 1.5→1.6 upgrades — keep the legacy etcd-headless Service alive so adopted members stay resolvable, delete the pre-1.6 operator Deployment via a pre-upgrade hook to get past the immutable selector, raise the operator's memory floor so it does not OOM before the VPA scales it, and make the etcd adoption migration robust in-cluster. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Improved backup storage endpoint handling across supported backup drivers, including provisioned and external S3 storage. * Added compatibility support for legacy etcd pod discovery during migration. * Added an automated upgrade safeguard for etcd operator deployments. * **Bug Fixes** * Improved certificate SAN detection and etcd migration authentication. * Increased the etcd operator’s minimum startup memory to prevent early restarts. * **Documentation** * Clarified backup endpoint, TLS, and driver-specific behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
IvanHunters
|
3fb8ec5b0e |
feat(monitoring): per-instance OIDC selector for the Grafana instance (Phase 1) (#3176)
<!-- Follow-up to cozystack/cozystack#3044 — same design family, different subject. --> ## What this PR does Adds a Phase-1 OIDC selector to the Grafana instance shipped by `packages/system/monitoring/`, mirroring the shape of the tenant kube-apiserver PR ([cozystack/cozystack#3044](https://github.com/cozystack/cozystack/pull/3044)) so that operators can use one mental model for both. Rationale in [cozystack/community#24](https://github.com/cozystack/community/pull/24) — per-cluster (here: per-Monitoring-release) audience isolation on the flat `cozy` realm, no per-tenant realm. Authorization is **app-side** via a chart-owned `users:` map reconciled by a post-install/post-upgrade Job into Grafana org membership. The chart does NOT own Keycloak directory objects (no `KeycloakRealmGroup`s) and does NOT rely on Grafana's `role_attribute_path` — same design shape as #3044's ClusterRoleBinding reconcile, adapted for Grafana orgs. See `docs/oidc-grafana.md` for the full rationale. The `Monitoring` CR grows a new field: ```yaml spec: oidc: mode: None | System | CustomConfig # default None customConfig: # only for CustomConfig config: {} # inline grafana.ini keys, or... secretRef: name: "" # ...an operator-managed Secret users: # optional; ignored in mode=None - email: alice@example.com role: Admin | Editor | Viewer ``` ### Modes - **`None`** (default) — no OIDC; `admin_user` / `admin_password` Secret is the only auth path. Existing releases render byte-identical. - **`System`** — chart provisions, in the release namespace of the Monitoring CR: - `KeycloakClient` (`public: false`, confidential, `secret` sourced from a chart-owned Secret via the EDP `$<secret>:<key>` reference; `redirectUris` locked to `https://grafana.<host>/login/generic_oauth`). - `KeycloakClientScope` with an `oidc-audience-mapper` pinning `id_token.aud` to the per-release clientId — the isolation primitive. - Persistent Kubernetes Secret carrying the confidential `client-secret` (random on first install, preserved via `lookup`+fallback, same pattern as `packages/system/dashboard`). - Grafana CR's `spec.config.auth.generic_oauth` section wired to the `cozy` realm issuer + per-release audience scope, with two chart-forced settings the users-Job depends on: `skip_org_role_sync: "true"` (login never overwrites the Job's org-role assignments) and `oauth_allow_insecure_email_lookup: "true"` (OIDC identity binds to the pre-provisioned local account by email). - `GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET` env on the Grafana Deployment sourced from the Secret. - Post-install / post-upgrade **users-Job** reconciling `spec.oidc.users` into Grafana's Main Org.: pre-provisions a local account per listed email, `POST /api/orgs/1/users` + `PATCH /api/orgs/1/users/{userId}` to converge role changes, prunes every non-admin Main-Org member whose email is not in the list. - Server-level `GrafanaAdmin` promotion (`allow_assign_grafana_admin`) is out of scope for Phase 1. Every Grafana instance (platform and tenant) caps at org-level `Admin`. - **`CustomConfig`** — tenant supplies the whole `[auth.generic_oauth]` payload; `cozy` is not in the path. Two mutually exclusive paths: - `customConfig.config: {...}` — inline map of `grafana.ini` keys. The chart **merges** the two chart-forced settings (`skip_org_role_sync`, `oauth_allow_insecure_email_lookup`) on top so the users-Job contract holds. The operator can NOT override those two (merge wins). - `customConfig.secretRef.name: ...` — operator-owned Secret with an `auth.ini` fragment; the chart mounts it under `/etc/grafana/oidc` and wires `GF_PATHS_CUSTOM_INI`, and does NOT emit `auth.generic_oauth` under `spec.config` (the ini fragment is authoritative). Because the chart cannot inject the two settings into a mounted ini, `spec.oidc.users` is NOT supported in this branch — the chart hard-fails render on that combination. ### Fail-close guards - `mode: System` without `_cluster.oidc-enabled == "true"` (the platform-level `authentication.oidc.enabled` flag) → hard-fail with an actionable message pointing at the platform values. - `mode: System` on a cluster where the EDP Keycloak operator CRDs (`v1.edp.epam.com/v1`) are not registered → hard-fail (symmetric on `oidc-keycloak.yaml` and `grafana.yaml`). - `mode: CustomConfig` with both or neither of `config` / `secretRef.name` → hard-fail. - `mode: CustomConfig` with `secretRef.name` AND non-empty `users` → hard-fail (chart cannot inject the two forced settings into an operator-mounted ini; forcing an explicit choice avoids silent contract loss). - `spec.oidc.users[]` entry with missing / unknown `role` or malformed `email` → hard-fail at render time so the users-Job never fires with a body that would 400 on Grafana's admin API. ### Break-glass posture `admin_user` / `admin_password` and `disable_login_form: false` are unchanged in every mode. Locking the form off under `mode: System` is a documented follow-up hardening (see `docs/oidc-grafana.md`). ### What's NOT in this PR (documented as out of scope) - Per-tenant Keycloak realms (Phase 2 territory, tracked in community#24). - Full-logout through Keycloak's end-session endpoint (`backend-logout-url` and matching Keycloak client attribute). - CEL `claimValidationRules` gate on `email_verified` — layered guarantees are described in `docs/oidc-grafana.md`. - Multi-issuer composition on one Monitoring release (mutually exclusive modes). - Server-level `GrafanaAdmin` promotion. ### Docs - New `docs/oidc-grafana.md` — full operator guide (parallel to `docs/oidc-tenant.md`). - Website page follows in a paired PR against cozystack/website (`docs/monitoring-oidc-authentication` branch, under `content/en/docs/next/operations/services/monitoring/oidc-authentication.md`). ### Tests - 33 `helm-unittest` cases in `tests/oidc_test.yaml` + `tests/oidc_crd_missing_test.yaml` cover: all three modes; defensive nil paths (`oidc: ~`); System-mode Keycloak client + audience scope shape; CustomConfig chart-forced merge (positive assertion) AND `mode=CustomConfig inline — operator cannot override chart-forced OIDC settings` (regression guard); CustomConfig secretRef + `users` incompat fail-fast; `spec.oidc.users[].role` missing/unknown fail-fast; users-Job shape (hook annotations, RBAC-free, activeDeadlineSeconds, `ttlSecondsAfterFinished: 3600`, admin credential wiring); no chart-owned `KeycloakRealmGroup`s in any mode; no `role_attribute_path` in Grafana config. - Render-side bats in `hack/e2e-apps/monitoring-oidc-{system,customconfig}.bats` — mirror the tenant kube-apiserver's e2e shape (no live browser flow — deferred to a follow-up integration suite, same as #3044). Asserts on live cluster: users-Job appears with `DESIRED_USERS_JSON` populated, Grafana config carries `skip_org_role_sync=true` and no `role_attribute_path`, no chart-owned `KeycloakRealmGroup`s land in `cozy`. ### Screenshots N/A — no UI change on default values; the visual delta is Grafana's own "Sign in with Keycloak" button appearing under the login form when a release opts in. ### Release note ```release-note feat(monitoring): Each `Monitoring` CR can now opt its Grafana instance into OIDC via a flat selector — `spec.oidc.mode: System | CustomConfig | None` (default `None`). `System` trusts the platform `cozy` realm via a per-instance confidential Keycloak client + audience binding. Authorization is app-side: `spec.oidc.users: [{email, role: Admin|Editor|Viewer}]` is reconciled into Grafana's Main Org. by a chart-owned post-install/post-upgrade Job (pre-provision → org add → PATCH role → prune stale). `CustomConfig` accepts a tenant-supplied `[auth.generic_oauth]` payload — inline as a map (chart merges the two contract-critical settings `skip_org_role_sync` + `oauth_allow_insecure_email_lookup` on top) or as a Secret with an `auth.ini` key (in which case `spec.oidc.users` is not supported and the chart fails render on that combination). Server-level `GrafanaAdmin` promotion is out of scope for Phase 1. The `admin_user`/`admin_password` Secret stays a documented break-glass path in every mode. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added OIDC authentication for Monitoring Grafana with `None`, platform-managed `System`, and tenant-controlled `CustomConfig` modes. * System mode provisions platform-backed OIDC for each instance (including persistent client credentials) and can optionally promote matching users to server-level `GrafanaAdmin`. * CustomConfig supports inline OAuth settings or Secret-backed `auth.ini` (with mutual exclusivity validation). |
||
|
Ivan Okhotnikov
|
a8494d874f |
test(backups/postgres): reconcile-level repeat-restore regression + conservative freshness tie
Address review feedback on the repeat in-place restore fix: - Make the freshness discriminator strict (creationTimestamp > StartedAt) so an exact tie resolves to "not fresh" and the caller purges, matching the function's stated conservative default; a fresh Cluster is always created well after StartedAt so this never misclassifies the status-write-race path. - Document why a wall-clock timestamp comparison was chosen over recording the purged Cluster's UID on the TargetPurged condition (the UID approach leans on the same status write the race path assumes can fail). - Add a reconcile-level regression test that drives reconcileCNPGRestore end to end with a fake client: a stale leftover recovery Cluster from a prior restore is purged, and a Cluster this restore just re-created is not re-purged. This covers the actual call site where #3311 lived, not just the pure helpers. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
4af5a82796 |
fix(etcd-operator,backups): address review nits — trim endpoint + hook resources
- backupstrategy-controller: trim() the decoded S3 endpoint before scheme stripping so a trailing newline in the Secret can't yield a malformed URL - etcd-operator: give the pre-upgrade selector-fix hook container resource requests/limits (matches postgres-operator webhook-ready), +unittest Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
7e34e87cd9 |
docs(backups): document derive-from-Secret S3 endpoint behavior
The `backupstrategy-controller.endpoint` helper changed how the S3 endpoint is resolved (for a provisioned bucket it derives from the COSI system Secret and forces https://, with `backupStorage.endpoint` demoted to a fallback), but docs/operations/backup-classes.md still described the old single-verbatim-value behavior and directly contradicted the code: - the "Endpoint format per driver" intro said the templates adapt "the single backupStorage.endpoint value"; - it claimed drivers "pull from backupStorage.endpoint in chart values, not from the Secret" — now precisely backwards for provisioned buckets; - the values-table `endpoint` row omitted the fallback/derive semantics. Rewrite those to match the new helper + the values.yaml comment. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
e160a0dc87 |
fix(backups): derive default S3 endpoint from the provisioned bucket
The default backupStorage.endpoint is http://seaweedfs-s3...svc:8333, but Cozystack ships SeaweedFS with global.seaweedfs.enableSecurity=true, so its in-cluster S3 serves TLS on :8333 behind the self-signed "SeaweedFS CA". Every cozy-default backup (etcd/mariadb/velero/fdb/cnpg) therefore hits a TLS listener over plaintext and fails the handshake. This is fatal for the etcd v1alpha2 adoption migration: its mandatory pre-upgrade safety snapshot writes to this endpoint, so migration 50 hard-fails and blocks the 1.5->1.6 upgrade on any cluster with a legacy etcd. The Etcd Strategy S3 schema has no caCert/insecureSkipVerify field, so it cannot target the self-signed in-cluster endpoint at all. Add a "backupstrategy-controller.endpoint" helper that, for a provisioned bucket, resolves the endpoint from the COSI bucket's system credentials Secret (backupStorage.systemSecretName) — the external S3 ingress with an ACME cert, the same trusted endpoint COSI advertises and every backup operator can verify — and forces the https:// scheme. All Strategy CRs, the Velero BackupStorageLocation and the controller Deployment env are routed through it. It falls back to .Values.backupStorage.endpoint for external S3 (provisionBucket=false) and for offline `helm template`/unit renders and the pre-reconcile first install, where the Secret lookup returns nothing (Flux re-renders on spec.interval once the Secret exists). helm-unittest extended to assert every consumer picks up the resolved endpoint (CNPG/Etcd/Velero keep the full URL, MariaDB/FDB strip the scheme and derive their secure flag) plus the external-S3 verbatim fallback. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Andrey Kolkov
|
47d239c519 |
fix(platform): address review on migration 50 (exact SAN match, Secret-gated wait, IPv6 kubeconfig)
Review feedback on #3261 (gemini-code-assist, coderabbitai, myasnikovdaniil): - Exact SAN match. _san_present read the Certificate SANs as {.spec.dnsNames} (a bracketed JSON blob that `tr ','` cannot split) and matched with a substring `grep -qF`, so a wildcard that is a substring of a longer SAN (e.g. *.etcd.<ns>.svc inside *.etcd.<ns>.svc.cluster.local) could false-positive and skip the re-issue. Read {.spec.dnsNames[*]} (space separated) and match exactly with `grep -qxF`, matching the Secret-annotation branch. - Wait on the re-issued Secret, not the patched spec. The post-patch wait loop called _san_present, which returns as soon as the Certificate spec.dnsNames contains the wildcard -- but the patch just added it there, so the loop broke on the first iteration before cert-manager re-issued the Secret, and the "re-issued" log printed unconfirmed. Add _secret_has_san (Secret alt-names annotation only, same empty-read retry + exact match) and gate the wait on it. - IPv6-safe kubeconfig server. The synthesized kubeconfig used https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT, which is an invalid URL on an IPv6-only cluster (a bare IPv6 host must be bracketed). Use https://kubernetes.default.svc -- IP-family-agnostic, validated by the mounted SA CA, and the same server URL kubectl synthesizes in-cluster. Tests (the review blocker): extend the bats suite (and its fake kubectl) with two cases pinning these contracts -- a superstring dnsName must NOT skip the re-issue patch, and etcd-migrate must be invoked with a synthesized in-cluster kubeconfig whose server is kubernetes.default.svc (not the bare IPv6 host), authenticating via the SA token file + CA. Both the kubeconfig path and the SA directory are now env-overridable (ETCD_MIGRATE_KUBECONFIG, ETCD_ADOPT_SA_DIR) so the synthesis is exercisable off-cluster; production defaults are unchanged. Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey Kolkov
|
c2539ca935 |
fix(platform): make etcd v1alpha2 adoption (migration 50) robust in-cluster
Migration 50 (etcd.aenix.io -> etcd-operator.cozystack.io/v1alpha2 adoption) had two defects that blocked every in-cluster 1.5 -> 1.6 upgrade on a cluster with an existing etcd: 1. Cert-SAN wait treated transient kubectl failures as "SAN absent". ensure_wildcard_sans checked/awaited the wildcard SAN with `kubectl get ... 2>/dev/null | grep`, so any transient GET failure (API discovery refresh, apiserver blip, throttling) produced an empty string indistinguishable from a genuine absence -> false miss, and the 120s wait never recovered. Replace the two ad-hoc checks with a _san_present helper that retries on an empty read (a real Certificate/Secret never has empty dnsNames/alt-names) and accepts the native wildcard from EITHER the issued Secret's cert-manager.io/alt-names annotation OR the Certificate spec.dnsNames (the source of truth for what cert-manager will issue). 2. etcd-migrate had no kubeconfig in-cluster. etcd-migrate only reads a kubeconfig file (-k/--kubeconfig, default /root/.kube/config) and, unlike kubectl, does not fall back to the mounted in-cluster ServiceAccount. The hook Job set no KUBECONFIG and passed no --kubeconfig, so both the dry-run and --apply aborted with "error building kubeconfig: stat /root/.kube/config: no such file". Synthesize an in-cluster kubeconfig from the mounted ServiceAccount and pass --kubeconfig to both etcd-migrate invocations. Verified end-to-end on a 1.5.2 -> 1.6.0-rc.1 upgrade: the adoption now completes in-place (pods never restarted, data intact) and the cluster reaches readyMembers=3 / Available=True. Refs: #3243, #3255 Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
ca4f99411f |
fix(etcd): harden v1alpha2 transition — hook runAsUser, digest-pin kubectl, tests
Addresses review on #3265: - pre-upgrade-selector-fix hook: add runAsUser: 65532. clastix/kubectl's image user is the non-numeric name `nonroot`, which the kubelet cannot verify against runAsNonRoot: true — so the Pod fails admission and the hook never runs, silently blocking the very 1.5->1.6 upgrade it exists to unblock. Matches postgres-operator's webhook-ready hook (same image). - Digest-pin the kubectl image (the comment already claimed digest-pinning but shipped a floating v1.32 tag). Reuse the digest postgres-operator already vendors, add the renovate annotation, and template repo:tag@digest. - Tests (both charts have CI helm-unittest suites, none previously covered these): - etcd-operator/tests/selector-fix-hook_test.yaml: hook wiring, namespaced least-privilege RBAC, numeric-non-root securityContext (guards the runAsUser regression above), digest-pinned image. - etcd-operator/tests/deployment_test.yaml: assert the 256Mi cold-start memory floor so it can't silently drop back to an OOMKilling value. - extra/etcd/tests/etcd-cluster_test.yaml: assert the transitional etcd-headless Service (headless, publishNotReadyAddresses, member selector, client/peer ports). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Andrey Kolkov
|
9daaa9ff5a |
fix(etcd): fix etcd-operator 1.5->1.6 upgrade (immutable selector + OOM floor)
Two defects that break the etcd-operator on an in-cluster 1.5 -> 1.6 upgrade (#2859 swapped the upstream chart for the cozystack-authored one): - #3242: the controller Deployment's spec.selector.matchLabels changed ({instance,name} -> {name,control-plane}). spec.selector is immutable, so `helm upgrade` cannot patch the existing Deployment and the whole HelmRelease upgrade fails ("field is immutable") -- the operator and the v1alpha2 CRDs it serves never come up, which blocks the etcd v1alpha2 adoption. Fresh installs use the new selector directly, so fresh-install CI does not catch it. Add a pre-upgrade hook that deletes the Deployment ONLY when its live selector is the pre-1.6 one (lacks control-plane=controller-manager), so Helm recreates it cleanly. No-op when the selector already matches (rc.1 -> later) and never runs on a fresh install (pre-upgrade only). Keeping the selector stable in the chart is not an option: rc.1 already shipped the new selector, so aligning it back would merely move the immutable break to rc.1 -> next. - Raise the manager's cold-start memory limit floor 128Mi -> 256Mi (and the VPA minAllowed to match). The steady-state working set is ~250Mi (the VPA's own recommendation); at 128Mi a Pod that starts before the VPA admission webhook rewrites it (e.g. a Deployment recreated out of band, or the webhook briefly unavailable during upgrade) OOMKills into a crash loop. Defense in depth so the operator never depends on VPA timing merely to avoid crashing; the VPA still scales it further under load up to maxAllowed. Verified live on a 1.5 -> 1.6 adoption: the delete-Deployment step plus the VPA re-applying 256Mi+ let the operator come up and the adoption complete. Refs: #3242, #3243 Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey Kolkov
|
d51d6914d7 |
fix(etcd): keep legacy etcd-headless Service alive during v1alpha2 adoption
etcd-migrate adopts legacy clusters IN PLACE, so the Pods keep their original spec.subdomain: etcd-headless and are dialed at etcd-<i>.etcd-headless.<ns>.svc until they eventually roll onto the operator's native <member>.etcd.<ns>.svc domain. The v1alpha2 operator only creates the native `etcd` Service, and the legacy `etcd-headless` Service is pruned during the transition, so those names stop resolving (no such host), the operator's MemberList fails, and status.readyMembers never populates -- the EtcdCluster never goes Ready even though the etcd processes are healthy and in quorum. Ship a chart-managed transitional headless `etcd-headless` Service (selector mirrors the operator's native `etcd` Service via etcd-operator.cozystack.io/cluster, publishNotReadyAddresses: true). This is the DNS counterpart of the legacy *.etcd-headless.<ns>.svc wildcard already kept in the server/peer cert SANs for the same transition window -- the TLS half of the compat was done, the DNS half was missing. Safe to remove, together with that SAN, once the members have rolled onto the native `etcd` subdomain. Verified live: recreating this Service on an adopted 3-node cluster restored per-pod DNS and the cluster went readyMembers=3 / Available=True. Refs: #3243 Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Ivan Okhotnikov
|
cf1b6ba28e |
fix(backups/postgres): purge stale recovery cluster on repeat in-place restore
A repeat in-place PostgreSQL RestoreJob into an already-restored target silently no-opped: the destructive purge was skipped because the live Cluster still carried spec.bootstrap.recovery left over from the previous restore, and the re-purge guard mistook that for a freshly-recovered Cluster produced by this restore's own purge. The job then reported Succeeded while the PVC, disk, and data were never touched. Tell the two apart by the Cluster's creationTimestamp relative to the RestoreJob's StartedAt: only a recovery Cluster created at/after StartedAt is treated as this restore's own and skipped; one that predates StartedAt is leftover from an earlier completed restore and is purged so the restore re-bootstraps from the backup. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |
||
|
myasnikovdaniil
|
215bd430a9 |
test(e2e): diff-driven install selector select-install.sh (#3279)
## What this PR does Adds `hack/select-install.sh` — the install-side companion to `hack/select-e2e.sh`. Where `select-e2e.sh` picks *which* Chainsaw suites run for a diff (a reverse dependency walk), `select-install.sh` picks *what must be installed* for those suites: the forward `dependsOn` closure over the `PackageSource` graph. Given the suite names `select-e2e.sh` emits, it maps each back to its `*-application` `PackageSource` and walks `spec.variants[].dependsOn` forward to the transitive closure — the minimal set of packages to enable instead of provisioning the full `isp-full` platform. A `postgres` chart change, for example, resolves to 9 of 97 sources. Design notes: - Reuses the real `PackageSource` graph in `packages/core/platform/sources/` — no parallel dependency-declaration files to keep in sync. - Unlike `select-e2e.sh`'s reverse walk, the forward walk **keeps** the `cozystack.cozystack-engine` edge: for test *selection* it is noise (it would fan every app out to every other), but for *install* it is a genuine prerequisite (the app's `*-rd` HelmRelease needs the engine to register the ApplicationDefinition CRD). Because that edge lives only on `*-application` sources, a suite backed by a **system** package (e.g. `securitygroup`, whose `SecurityGroup` in `sdn.cozystack.io` is served by the aggregated `cozystack-api` — a `cozystack-engine` component) can't reach the engine via the walk, so the engine is **seeded explicitly for any non-empty closure** (and a graph missing it is a hard error). - `--validate` mode checks the whole graph — every `dependsOn` target resolves to a real `PackageSource` (reachability) and there are no dependency cycles — **and** the suite mapping: every `hack/e2e-chainsaw/*/` suite resolves to a real `PackageSource` or to `-` (a missing suites dir is itself a failure). The graph lives next to the packages and can't drift from the suite dirs; the hand-maintained mapping is the only part that can, so this is the cheap CI guard. - Suite→source mapping follows the `<suite>-application` convention with a bare-`<suite>` fallback, plus explicit cases for the irregular names (`kubernetes`/`vminstance`/`securitygroup`) and for core-platform features that have no separately-enabled package (`serviceexposure`, whose `ExposureClass`/`ServiceExposure` CRDs live in `packages/core/platform`). A suite may reference other app kinds by selector without standing them up (e.g. `securitygroup`'s `SecurityGroup` names `Postgres`/`Kubernetes` as selectors only) — only the primary package is mapped here; the forward closure and the seeded engine cover its prerequisites, and full multi-package modelling is left to the `test-minimal` follow-up. A suite with no known mapping is a **hard error** (fail closed), not a silent skip. Includes a 17-case unit suite (`hack/select-install_test.bats`) covering single-app closure, transitivity, multi-suite union, stdin input, the kubernetes/irregular/core-platform suite mappings, the `securitygroup` engine-seed regression, empty input, fail-closed unmapped suites (all reported, no partial output), the missing-engine hard error, and the `--validate` failure modes (dangling dep, cycle, unmapped suite dir, missing suites dir). It is auto-discovered by `make unit-tests` (`BATS_UNIT_FILES` globs `hack/*.bats`). Verified end-to-end against real git diffs across six path types (app chart, shared operator, docs-only, per-suite Chainsaw edit, library, shared e2e helper): a `postgres` chart change scopes the install to 9 of 97 sources; the full-suite fallback to 39 of 97. This is Track A of #2847 (diff-driven & scenario-based E2E); tracked in #3278. Consuming the output — a `test-minimal` platform variant/bundle plus wiring into the `pull-requests.yaml` install step — is a deliberate follow-up, since it depends on a platform-variant design decision. ### Screenshots Not applicable — CI/tooling-only change. ### Release note ```release-note test(e2e): add hack/select-install.sh to compute the minimal package install set for a diff-scoped E2E run (forward PackageSource dependency closure), with a --validate graph-consistency mode ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a tool to select the minimal package set required for one or more end-to-end test suites. * Supports combining multiple suites, resolving transitive dependencies, and handling unmapped or empty suite selections. * Added validation for missing dependencies and circular dependency definitions. * **Tests** * Added coverage for package selection, dependency resolution, suite mappings, empty selections, and validation failures. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Ivan Okhotnikov
|
5d13c15d65 |
fix(apps/kubernetes): compile OIDC groups claim rule via dyn() so the tenant apiserver boots
The System-mode AuthenticationConfiguration rendered a claimValidationRule ranging over claims.groups, which is statically typed `any` in the apiserver CEL environment. The .exists() comprehension rejects a range of type any, so the tenant kube-apiserver failed config compilation at startup and CrashLooped — the control plane never booted and cluster teardown then hung the e2e cleanup budget. Wrap the range in dyn() so CEL treats it as dynamic; the has() guard still short-circuits when groups is absent and the gate semantics are unchanged. Strengthen the oidc unit test to pin dyn() (mutation-verified). Verified on a live tenant apiserver (v1.32). Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |
||
|
Ivan Okhotnikov
|
0164a6afa4 |
fix(monitoring): set groups_attribute_path=groups so allowed_groups actually works on Grafana v11.x
Verified end-to-end on dev3 with Grafana v11.6.15: OAuth flow reaches Grafana, id_token and userinfo response both carry the correct `groups` claim (`[cozystack-cluster-admin, tenant-root-super-admin]` for kvaps), `allowed_groups` in grafana.ini lists `tenant-root-super-admin`, and login still fails with `user not a member of one of the required groups`. Grafana debug log shows `Attributes: map[]` — the extracted user's Groups slice is empty. On v11.x+ Grafana does not auto-map the `groups` claim onto user.Groups unless `groups_attribute_path` is set explicitly (the field is documented as optional but the gate is a silent no-op without it — a claim-name-string is not enough). Fix: hard-wire `groups_attribute_path=groups` in the chart-forced System-mode block alongside `allowed_groups`. "groups" points at the top-level `groups` array from userinfo, matching the shape Keycloak's oidc-group-membership-mapper emits (verified on the same dev3 run). Two helm-unittest cases updated to assert `groups_attribute_path` lives next to `allowed_groups` in both the users-map-active and empty-users branches (rejecting the silent no-op regression). Docs also updated to name the requirement. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
874ca9427f |
fix(apps/kubernetes): close cross-tenant kube-apiserver OIDC hole via CEL claim gate
Symmetric with the Grafana-side allowed_groups gate — round-3 review from lllamnyp on PR#3176 covered both surfaces, and #3044's original System-mode wiring shipped without this guard. The rationale is identical: audience binding stops REPLAY of a minted token across tenant kube-apiservers, but nothing stopped Keycloak from ISSUING a usable token to a caller outside the tenant. Absent a claim-side gate, any authenticated cozy-realm user could `kubectl oidc-login` against another tenant's cluster and land at `system:authenticated`. RBAC default-denies named-resource access, but `system:authenticated` still leaks discovery (kubectl api-resources, OpenAPI schemata) — tenant-alice could enumerate tenant-bob's cluster's CRD shape and built-in resource surface. Fix: add a CEL `claimValidationRules` entry to the rendered AuthenticationConfiguration (System mode only) that requires at least one of the release's four tenant-scoped Keycloak groups (`<namespace>-{view,use,admin,super-admin}`) to be present in the token's `groups` claim. The `has(claims.groups) &&` guard is required — CEL evaluation on a missing claim surfaces as HTTP 500 from the authenticator instead of a clean 401. CustomConfig mode is not touched — the tenant's own AuthenticationConfiguration is authoritative and the tenant is responsible for claim-side guards on their own issuer. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
b002c967ac |
fix(monitoring): close cross-tenant Grafana OIDC hole via tenant-group gate
Round-3 review from lllamnyp: any user in the shared `cozy` Keycloak
realm could sign into any tenant's Grafana. The audience mapper stops
replay of a minted token across releases, but does not gate who may
mint one — tenant-alice could complete the OAuth flow against
tenant-bob's Grafana. With `spec.oidc.users` non-empty the earlier
`allow_sign_up=false` fix mostly closed the door (though tenant-alice
still hit `error 401` after landing in Grafana, revealing the release
exists); with empty users under mode: System — a config the schema
permits and the docs describe as hands-off — the login succeeded and
minted a Viewer account for every cozy-realm identity.
The fix uses the KeycloakRealmGroups the tenant chart already
provisions (`<namespace>-{view,use,admin,super-admin}` in
packages/apps/tenant/templates/keycloakgroups.yaml) — no new
directory objects owned by the Monitoring release. Two chart edits:
1. oidc-keycloak.yaml: add the platform-managed realm-default `groups`
scope to `defaultClientScopes` explicitly. EDP's KeycloakClient
does NOT merge realm-level defaults into `defaultClientScopes`, so
the scope must be listed here or the `groups` claim would be
missing from the token.
2. grafana.yaml: in System mode, unconditionally emit
`allowed_groups: <ns>-view <ns>-use <ns>-admin <ns>-super-admin`
in `auth.generic_oauth` and add `groups` to `scopes`. Grafana
rejects any login whose token does not carry at least one of the
groups.
The gate is chart-owned and does not depend on the users-map — with
`users: []` tenant members still get a self-service Viewer through
`auto_assign_org_role`, cross-tenant users get 401 in Keycloak's OAuth
callback. Docs updated to reflect the two-layered contract
(tenant-membership gate + optional users-map).
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
|
||
|
IvanHunters
|
f64ba91872 |
fix(monitoring): do not render users-Job when spec.oidc.users is empty
Address PR#3176 review round-2 gap-2: the users-Job's prune pass deletes every non-admin Main-Org member whose email is not in the desired list. When the desired list is empty every helm-upgrade wiped the whole org. Three empty-users states used to fall into this trap: * customConfig.secretRef: users[] is forbidden by assertCustomSecretRefUsersEmpty, so the desired list was structurally always empty and the chart wiped an org whose auth.ini fragment the operator advertised as authoritative. * System with users[] unset: the operator opted into OIDC but not into the chart-managed users-map, yet the Job still pruned every Main-Org member on their behalf. * CustomConfig-inline with users[] unset: same shape as System. The mode=None guard (from an earlier round) already had the same shape; extend the render gate to cover the other empty-list doors — now the Job renders only when `mode != None AND users non-empty`. The trade-off: taking `users:` back to `[]` no longer prunes the last entry — matching the already-documented `System → None` posture — called out explicitly in the docs. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
ff4ef2c189 |
fix(monitoring): force allow_sign_up=false and gate isolation dict on users-map
Address PR#3176 review round-2 gap-1: the docs promise "users not in spec.oidc.users get nothing", but Grafana's default allow_sign_up=true combined with the chart-forced skip_org_role_sync=true was minting a Viewer account for every cozy-realm identity that completed the OAuth flow, then reaping it on the next helm-upgrade prune pass. The chart-forced isolation dict (now three keys: skip_org_role_sync, oauth_allow_insecure_email_lookup, allow_sign_up=false) is applied only when spec.oidc.users is non-empty. A BYO-IdP operator who does not use the users-map gets none of the three forced on top of their inline config, matching the review's minor: forcing insecure_email_lookup against an issuer that may not verify emails is a needless account- linking surface, and allow_sign_up=false with no pre-provisioned accounts would lock everyone out. Tests split the two branches: users-active asserts all three keys are present and the operator cannot override them via CustomConfig-inline; users-empty asserts the auth.generic_oauth block is emitted without any of the isolation keys. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
3776c447b9 |
test(e2e): poll users-Job DESIRED_USERS_JSON for target email, not just Job existence
CI E2E showed the users-Job assertion returning `DESIRED_USERS_JSON:
[]` — but the CR body had `users: [{email: e2e-admin@example.com,
role: Admin}]`. Root cause: `hook-delete-policy: before-hook-creation`
keeps the previous hook Job around across helm upgrades, so `until
kubectl get job` matches a stale Job from the previous customconfig-
mode test where users may have been unset (the secretRef swap CR had
no `users:` field, so the schema default `[]` propagated to the
prior Job's DESIRED_USERS_JSON). The System test then grabbed that
stale Job before the helm upgrade for the new CR completed and swapped
in the fresh Job.
Fix: poll on the target env value itself — the new Job with the
correct users list is the first Job whose DESIRED_USERS_JSON string
contains `e2e-admin@example.com`. 600s window matches the inner HR's
15m timeout with room for a slow reconcile.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
|
||
|
IvanHunters
|
b2fd32385e |
fix(monitoring): rewrite users-Job shell in POSIX (alpine BusyBox compat)
CI E2E caught the Job container crash-looping within 20s of the first Start event. Root cause: the `alpine/k8s:1.33.4` image is Alpine-based (BusyBox coreutils + BusyBox grep), and the Job shell used four GNU-specific constructs that BusyBox does not implement: - `head --bytes=N` (BusyBox `head` only accepts `-c N`). - `tr --delete` (BusyBox `tr` only accepts `-d`). - `jq --raw-output0` (NUL-terminated output — jq 1.7+ feature; also paired with `read -r -d ''` which is a bash-ism, not POSIX `sh` and not honoured by BusyBox `ash`). - `grep --null-data --fixed-strings --line-regexp` (BusyBox `grep` supports `-F` and `-x` short forms but NOT `-z`/`--null-data`). Rewrite: - `head -c 24 /dev/urandom | base64 | tr -d '=+/' | head -c 32` — same random-32-char password, portable to BusyBox. - Emit desired emails and org membership one record per line (jq `--raw-output`) instead of NUL-terminated. Safe because the template-level `monitoring.oidc.assertUsersEmailShape` guard already rejects any email containing whitespace / control characters, so line-oriented iteration cannot get confused. - Read TSV membership records with `IFS="$(printf '\t')"` (POSIX- portable TAB) + plain `read -r uid ulogin uemail` — no need for `-d ''` once records are newline-terminated. - Match desired list with `grep -Fxq -e "$uemail" -e "$ulogin"` — short-form flags supported by both GNU and BusyBox `grep`. Syntax validated with both `sh -n` and `bash -n` on the extracted inline script. helm-unittest 34/34 still green (render shape is unchanged, only the shell literal is different). Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
1422495b88 |
fix(monitoring): do not render users-Job when mode=None (upgrade safety)
Caught by dev3 end-to-end test on the PR working tree
(cozystack-pr-test skill): the users-Job was rendered in every mode
including `None` on the rationale that a `mode: System → None`
switch should garbage-collect OIDC-provisioned local accounts. That
rationale ignored the upgrade path from the pre-PR chart: an
existing tenant that never had `spec.oidc` on their Monitoring CR
would receive `mode: None` on first render under the PR, the Job
would fire on every Flux reconcile of the outer HR, and its prune
pass would DELETE every non-admin Main-Org member — including users
the tenant added by hand through the Grafana UI on the pre-PR chart.
Silent breaking change with data loss.
Wrap the whole Job resource in `{{- if ne $mode "None" }} … {{- end }}`
so `mode: None` = "chart owns nothing in Grafana orgs, we don't
touch what's there". Operators who need to prune OIDC-provisioned
accounts after switching System | CustomConfig → None do it
themselves through the Grafana UI or admin API — same posture as
"we don't own the Keycloak directory objects" from the earlier
architectural change.
The `assertUsersPrologue` include stays outside the `if` so an
operator listing users in `mode: None` still gets the schema and
email-shape validation at render time — the Job body just doesn't
emit.
Test coverage:
- Flip the previous "mode=None still renders the users-Job so a
mode switch can prune stale members" case to
`mode=None does NOT render the users-Job (upgrade safety guard)`
as a regression lock.
- Add `mode=None with non-empty users still fails render` covering
the prologue-only-fires-on-schema-issue path.
Docs: rewrite the "Users and RBAC" section step 3 to name the
upgrade-safety motivation and the manual-cleanup handoff.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
|