mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-09 14:02:56 +09:00
ba67fea7df3d757885b654e2ab9f386cb8d2e126
5130
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Myasnikov Daniil
|
ba67fea7df |
ci(release): drop paths-ignore from the release finalize trigger
The promote PR now carries docs/changelogs/vX.Y.Z.md, and Prepare stable branch explicitly tolerates producing no tag-string changes when the digests are already stable. Together those can yield a docs-only promote PR, which `paths-ignore: docs/**/*` would filter out — finalize never runs, so the stable tag is never cut and the release is never published, with no error raised anywhere. The `if:` gate (merged + `release` label + release-bot author) is what actually decides whether this workflow does anything, so let it do the filtering. The cost is a skipped job on unrelated closed PRs. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
75b5192f1b |
ci(release): produce changelog and docs before the release goes public
Changelog generation and the website-docs PR both lived in tags.yaml, gated on the stable tag push. Under the promote flow that tag is created by pull-requests-release.yaml finalize, which also publishes the release — so both jobs could only ever run once the release was already public. A reader hitting the release in that window saw an empty body, and the site lagged until someone merged the docs PR. Move both to promote time: - promote-rc.yaml gains a `changelog` job that runs in parallel with `promote` and hands the file to a new `open-pr` join job, which commits it onto the release-X.Y.Z branch so it rides the promote PR. Generation is AI-driven and slow while `promote` spends its wall clock on the Talos assets; neither needs the other, so both start at once. - The website-docs job moves to promote-rc.yaml too, depending only on `promote`. Different repo, so it cannot ride the PR — it is opened early and finalize merges it after the release publishes. - finalize reads docs/changelogs/vX.Y.Z.md from the merge commit and uses it as the published release body. update-releasenotes.yaml is no longer on the critical path: it triggers on the same push and would race this job, and if it won it would find no published release, skip, and never fire again. It also only watches main, so a patch release promoted onto release-X.Y was never synced at all. - tags.yaml keeps generate-changelog as a backstop only. It self-skips when the changelog is already on main, and covers the two cases where promotion does not put it there: generation failed (non-blocking), or the promote PR targeted a maintenance line. Generation stays non-blocking, so a flaky AI step cannot wedge a promotion. When it fails the promote PR body says so explicitly, and the escape hatch is to commit docs/changelogs/vX.Y.Z.md to the branch — which also makes a hand-written changelog first-class: if the file is already in the rc tree, generation is skipped entirely. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Timofei Larkin
|
2823b76802 |
feat(gateway): advertise HTTP/2 via ALPN on Gateway API listeners (#3398)
## What this PR does Browsers negotiate HTTP/2 exclusively through TLS ALPN: if the terminating listener does not advertise `h2`, every client silently falls back to HTTP/1.1, with no error surfaced anywhere. Cozystack's Gateway API path is served by Cilium's embedded Envoy, and Cilium ships with `enable-gateway-api-alpn=false` — so every Gateway listener advertised no ALPN protocols and all browser traffic through a Gateway ran over HTTP/1.1 (observable in browser dev tools as `http/1.1` on every request). The ingress-nginx path this replaces advertised `h2` out of the box, so the migration to Gateway API silently downgraded clients: head-of-line blocking and the ~6-connections-per-origin limit are back for anything served through a Gateway. This PR enables ALPN (`h2`, then `http/1.1`) on Gateway API listeners, default-on, with an opt-out at each management layer: - `packages/system/cilium/values.yaml` now defaults `gatewayAPI.enableAlpn: true`, which renders `enable-gateway-api-alpn: "true"` into the `cilium-config` ConfigMap. - The platform chart gains `gateway.http2` (default `true`) next to the existing `gateway.enabled`/`gateway.attachedNamespaces`. The `isp-full` and `isp-full-generic` bundles thread it as inline component values into the `cozystack.networking` Package CR (`components.cilium.values.cilium.gatewayAPI.enableAlpn`); inline HelmRelease values override the package `valuesFiles`, so an explicit `gateway.http2: false` actually reaches the dataplane rather than being shadowed by the new chart default. On `isp-hosted` the hosting-side CNI owns this setting and the flag is a no-op. - Tenant Kubernetes clusters (`packages/apps/kubernetes`) get the same fix: when the `gatewayAPI` addon is enabled, the tenant cilium HelmRelease now sets `enableAlpn: true`; individual clusters opt out via `addons.cilium.valuesOverride.cilium.gatewayAPI.enableAlpn=false`. ### Why default-on rather than opt-in HTTP/2 for browsers was the platform's behavior before the Gateway API migration, so keeping ALPN off would preserve a regression, not a status quo. Defaulting on is safe because ALPN is a negotiation: clients that cannot speak HTTP/2 simply keep negotiating `http/1.1`, exactly as before. Operators who need the old wire behavior can set `gateway.http2: false` platform-wide. ### Frontend vs backend hop This change affects the client↔gateway hop only. Gateway↔backend connections stay HTTP/1.1 unless a Service explicitly opts in per GEP-1911 by declaring `appProtocol: kubernetes.io/h2c` on its port. Cilium couples that backend-protocol support to the ALPN flag (`enable-gateway-api-app-protocol` flips to true together with it), but it only acts on Services that declare a recognized `appProtocol` value — and no cozystack-shipped Service does today (only plain `http`/`https` values exist in-tree, which Cilium ignores for protocol selection). So upstreams see no change from this PR alone; h2c/gRPC backends become possible to configure, not configured. The setting is cluster-wide Cilium agent config, so it covers the root Gateway and all tenant Gateways at once — Cilium offers no per-Gateway granularity here. Flipping it re-rolls the cilium DaemonSet on upgrade (`rollOutCiliumPods: true`), the same disruption profile as any other cilium config change. ### How it was verified - New helm-unittest suites: `packages/core/platform/tests/bundles_gateway_http2_wiring_test.yaml` asserts the Package CR carries `enableAlpn` true by default and false with `gateway.http2: false` for both cilium bundles; `packages/apps/kubernetes/tests/cilium_gateway_alpn_test.yaml` asserts the tenant HelmRelease default, the addon-off case, and the `valuesOverride` opt-out. Full suites for both charts pass (93 and 184 tests). - `helm template` of `packages/system/cilium` confirms the value is consumed, not just rendered: default produces `enable-gateway-api-alpn: "true"` (and `enable-gateway-api-app-protocol: "true"`) in `cilium-config`; the threaded `false` produces `"false"` for both. ### Screenshots Not a UI change. ### Downstream repositories The website's hand-written platform values table (`content/en/docs/next/operations/configuration/platform-package.md`) needs a row for `gateway.http2` — filed as cozystack/website#625. No other downstream repository restates anything in this diff (no app `values.schema.json`, ansible-consumed platform key, or hack/ tooling changed). - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/pull/625 - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note feat(gateway): Gateway API listeners now advertise HTTP/2 via TLS ALPN (h2, then http/1.1) by default, restoring the browser HTTP/2 support lost in the migration from ingress-nginx. Opt out platform-wide with gateway.http2=false, or per tenant Kubernetes cluster via addons.cilium.valuesOverride.cilium.gatewayAPI.enableAlpn=false. Backend (gateway-to-upstream) connections are unchanged and stay HTTP/1.1 unless a Service declares appProtocol kubernetes.io/h2c. ``` |
||
|
Aleksei Sviridkin
|
0ba79c0168 |
fix(cozystack-basics): gate the hostname VAP policies on the VAP API (#3409)
## What this PR does Wraps the route, gateway, and ingress hostname `ValidatingAdmissionPolicy` templates in a `.Capabilities.APIVersions.Has` gate so they render only where the VAP API is available. Without it, an operator-generated HelmRelease with drift detection off renders the policies out at first install on a cluster missing the API and never adds them back. Each template gains a sibling test asserting it renders zero documents when the API is absent. Install-robustness fix, split out from #3299. ```release-note fix(cozystack-basics): hostname ValidatingAdmissionPolicies render only where the VAP API is available, so a first install on a cluster without it no longer drops them permanently. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Hostname validation policies now render only when the Kubernetes admission policy API is available, preventing unsupported policy resources on older clusters. * **Tests** * Added/updated unit tests to explicitly simulate both missing and present admission policy API capabilities, ensuring templates render (or not) as expected. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
af397bb62c |
chore: ignore root-level Go binaries built from cmd/ (#3412)
## What this PR does Adds the root-level binary names that `go build ./cmd/<name>` (without `-o`) drops in the repository root to `.gitignore`, so `git add .` cannot accidentally commit an ~80 MB executable. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated repository settings to exclude generated Go executables from version control. * Added explanatory comments for the ignored build artifacts. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
5a7e4c5391 |
feat(api)!: require the application release prefix to be dot-free (#3411)
## What this PR does Constrains `ApplicationDefinition.spec.application.release.prefix` to `^[a-z0-9-]*$`, which excludes the dot. Release names are `<prefix><app>`, and the tenant CA trust anchor is projected to `<release>.tenant-ca`; the dot is the separator that keeps that name unreachable by any engine, so a prefix carrying a dot would break the guarantee. Every in-tree prefix already conforms. Split out from #3299. ```release-note feat(api)!: `ApplicationDefinition.spec.application.release.prefix` must match `^[a-z0-9-]*$` (dot-free). A definition with a prefix containing a dot or any other character is rejected at admission; correct it before applying this CRD. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Added validation for release prefixes to allow only lowercase letters, numbers, and hyphens. * Updated guidance explaining how prefixes affect generated release names and tenant CA trust anchors. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
6ecb36ffc7 |
feat(cozystack-controller): extract application CA into a key-free tenant Secret (#3407)
## What this PR does Adds an engine-agnostic controller that publishes a key-free `<release>.tenant-ca` Secret (only `ca.crt`) for a managed application, so a tenant can verify the application's TLS without ever seeing a private key. The source is declared, not guessed: a chart renders a namespaced `TenantProjection` sentinel (group `internal.cozystack.io`) naming the CA Secret to lift; the controller watches the sentinel, extracts `ca.crt`, and writes the projection owner-referenced to the sentinel, so garbage collection is native. The projection carries the `internal.cozystack.io/tenant-ca` label the lineage webhook turns into tenant visibility. Postgres is the first consumer. The trust boundary is RBAC: no tenant role grants any verb on `internal.cozystack.io`, so a tenant cannot forge a sentinel. A companion `ValidatingAdmissionPolicy` that pins the writer to helm-controller ships separately as defence in depth. This is the controller-and-CRD slice, split out from #3299 for focused review. It is the only piece required for the feature to work; the render-time helper, the writer policy, the capability-gate, and the release-prefix validation land as their own PRs. ### Downstream repositories Walked the trigger map against the diff. This adds an internal CRD and controller and a sentinel to the Postgres chart; no app is added or renamed and no `values.schema.json` changes, so the typed provider and Ansible are unaffected. The one to confirm is the Postgres README edit — if the website mirrors it, it needs a refresh. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: ### Release note ```release-note feat(cozystack-controller): publish a key-free per-application CA trust anchor (`<release>.tenant-ca`) that tenants read to verify managed-application TLS, declared through a new `TenantProjection` CRD. Postgres is the first consumer. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added tenant-facing CA certificate projections for PostgreSQL deployments. * Tenant secrets now provide only the CA certificate, excluding private keys. * Added support for configuring and monitoring tenant CA projections. * **Documentation** * Updated PostgreSQL TLS instructions to retrieve certificates through tenant secrets. * **Bug Fixes** * Improved protection against invalid certificates, secret collisions, and accidental exposure of private key material. * Added validation to ensure projected certificates remain synchronized with their sources. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrey Kolkov
|
8aeedde088 |
feat(backups): support PostgreSQL point-in-time recovery (PITR) (#3383)
## What this PR does Makes PostgreSQL point-in-time recovery (PITR) a genuinely supported, tested capability for the new `backups.cozystack.io` API. Fixes #2774. Was **stacked on #3313** (`fix/3300-cnpg-barman-plugin`) for the `recoveryTime` → CNPG `bootstrap.recovery.recoveryTarget.targetTime` plumbing; #3313 has since merged and this branch is rebased on `main`. This PR adds the reliability + verification layer #2774 asked for. ### Changes - **Precise failure reason for an unreachable target.** The restore deadline (`spec.options.restoreTimeoutSeconds`, default 30m) stays the sole authority for "this restore is stuck" — the client waits the full window either way. What changes is the *diagnosis*: once the deadline has elapsed, if `recoveryTime` was set and the recovery Cluster never went healthy, the CNPG driver reads the `full-recovery` pod logs and — when several distinct attempts exited with PostgreSQL's `FATAL: recovery ended before configured recovery target was reached` — fails the RestoreJob with `RecoveryConverged=False` / `Ready=False` (reason `RecoveryTargetUnreachable`) and an actionable message, instead of a bare generic timeout. Timing is unchanged; only the reason/message differ. (For a faster rejection, set a shorter `restoreTimeoutSeconds`.) Design points: - **We deliberately do NOT trip early off the FATAL.** A valid near-now target emits that same FATAL transiently — repeatedly, for many attempts, on a slow archiver — before it converges, so any earlier trip would wrongly reject a recoverable restore (observed twice in CI). The FATAL is used only to *explain* a failure the deadline has already declared. - **Health wins unconditionally** — the deadline/classification check precedes the healthy check, but a recovery that converges within the window reaches success and is never classified as failed (a live dev7 restore did exactly this). - **Counting distinct FATAL-ing attempts** (not pod failures) tolerates a one-off archive-lag miss on a near-now target and ignores transient recovery-pod crashes (a node blip / brief API-server unreachability never emits that FATAL). - Adds the `pods/log` RBAC grant the log read needs (distinct subresource from `pods`); a `Forbidden` is logged loudly and surfaced in the failure message rather than silently disabling the classification. - **e2e** (`examples/backups/postgres/run-all.sh`, wired into the Chainsaw `postgres-2-backup-roundtrip` suite): - Step 45 — write `before`, capture server timestamp (µs precision), write `after`, restore to that timestamp, assert `before` present and `after` absent. - Step 46 — a `recoveryTime` an hour ahead must fail with reason `RecoveryTargetUnreachable` once the deadline elapses (asserts the *reason*, so it proves the classification fired, not just that a timeout occurred). - Chainsaw budget raised 25m → 40m to cover three sequential restores + the post-deadline classification. - **Docs + example**: uncommented `45-restorejob-pitr.yaml`; a `docs/operations/backup-classes.md` PITR section (recoverable window, discovering earliest/latest restorable time from the backup catalog since `firstRecoverabilityPoint` is unreliable under the plugin, archive-lag tolerance, idempotency under GitOps). ### Verification - `go build` / `go vet` clean; unit tests pass (`logIndicatesRecoveryTargetUnreachable`, `recoveryUnreachableFromLogs`, `recoveryPodsToInspect`, nil-Clientset guard, a reconcile-level success-path test asserting `Succeeded` + `RecoveryConverged=True`, and — added per review — `TestReconcileCNPGRestore_DeadlineClassifiesRecoveryTargetUnreachable`, a reconcile-level test driving the deadline branch through all three sub-paths: FATAL found → reason `RecoveryTargetUnreachable`, FATAL absent → generic timeout, `Forbidden` log read → generic timeout + missing-`pods/log`-RBAC note. It fails if the call-site reason is reverted to the generic one). - Reviewed via adversarial branch-review; fixes include a critical missing-RBAC dead-on-arrival, a classification false-positive on transient recovery-pod crashes, and a sub-second e2e flake. - **dev7**: backup + WAL archiving + recovery convergence verified live on the #3313 build. The load-bearing assumption of the classification — CNPG **accumulates** failed `full-recovery` pods rather than reusing one — was confirmed (observed 7 simultaneous failed recovery pods). A fully-green live step-46 run was blocked by intermittent COSI bucket-provisioning flakiness on dev7; the negative path is covered by the CI Chainsaw e2e. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff. The only user-facing surface is the new `docs/operations/backup-classes.md` PITR section; the website keeps a hand-mirrored copy per docs version. - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: PITR docs mirror warranted (not yet opened; same pending mirror as #3313's backup-classes changes) - [x] No other downstream repository is affected by this change ```release-note feat(backups): PostgreSQL RestoreJob now supports point-in-time recovery (PITR). Set spec.options.recoveryTime (RFC3339) to restore to an exact instant. A recoveryTime past the latest archived WAL still runs to the restore deadline (spec.options.restoreTimeoutSeconds, default 30m), but the failure is now reported with the precise reason RecoveryTargetUnreachable and an actionable message instead of a generic timeout; set a shorter restoreTimeoutSeconds for a faster rejection. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added PostgreSQL point-in-time recovery (PITR) support via RFC3339 `recoveryTime`. * Introduced a PITR RestoreJob example, plus helper variables and an unreachable-target scenario. * **Bug Fixes** * Improved restore deadline handling with deterministic, log-based fail-fast classification for unreachable recovery targets, including clearer conditions. * **Documentation** * Added detailed PITR operational documentation (recovery windows, failure behavior, and GitOps/idempotent behavior). * Expanded README and example comments for PITR verification and failure expectations. * **Tests** * Extended unit tests and end-to-end chainsaw workflow to cover PITR success and unreachable failures. * **Chores** * Updated RBAC to allow pod log reads; refined cleanup and run scripts for additional RestoreJob artifacts. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
267ea513ad |
fix(cozystack-basics): gate the hostname VAP policies on the VAP API
Render the route, gateway and ingress hostname ValidatingAdmissionPolicies only where the admissionregistration.k8s.io/v1/ValidatingAdmissionPolicy API is served, matching the deletion-protection guard, so a cluster without that API does not receive an unrenderable resource. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
myasnikovdaniil
|
18e459a1cb |
fix(e2e): improve failure diagnostics (#3421)
## What this PR does Improves E2E failure diagnostics tracked in #3368: - preserves scheduled, non-terminal NotReady pods before a pod IP is assigned - always captures load-balancer state while gating IP-dependent diagnostics - collects Talos dmesg, kubelet, and containerd logs per host node using explicit endpoint and node arguments - renders COSI Bucket catch output as YAML - makes pre-commit generation locking work from linked worktrees - adds regression coverage for dataplane and Talos collection Tenant-worker Talos logs remain outside this PR because the runner does not currently have a tenant talosconfig. Part of #3368. ## Testing - `hack/cozytest.sh hack/capture-dataplane.bats` - `hack/cozytest.sh hack/cozyreport-talos.bats` - `sh -n hack/e2e-capture-dataplane.sh` - `sh -n hack/cozyreport.sh` - `chainsaw test --no-cluster --config hack/e2e-chainsaw/.chainsaw.yaml --include-test-regex '^$' hack/e2e-chainsaw/bucket hack/e2e-chainsaw/harbor` - `pre-commit run --all-files` ### Screenshots Not applicable. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(e2e): improve diagnostic capture for dataplane, Talos, and COSI failures ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Bug Fixes** - Improved dataplane diagnostics for scheduled pods that are NotReady, including cases without an assigned IP. - LoadBalancer checks now proceed even when no affected pods are found. - Standardized Talos node diagnostics collection for consistent kernel, kubelet, and containerd logs. - Improved reliability of generation hooks by using a git-common lockfile path for concurrent runs. - **Tests** - Added/expanded Bats coverage for dataplane pod filtering and Talos diagnostics argument correctness. - Improved end-to-end bucket test retrieval/formatting for clearer failure output. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
IvanHunters
|
f077420fbe |
fix(http-cache,vpa): fix haproxy 3.3 backend name conflict and vpa-for-vpa OOM (#2654)
## What this PR does Fixes two independent issues found during cluster operations: **http-cache (HAProxy 3.3 breaking change)** HAProxy 3.3 no longer allows a frontend and backend to share the same name. The existing config had both `frontend http` and `backend http`, which causes haproxy to refuse to start on upgrade. Rename `backend http` → `backend cache` and update the `default_backend` directive accordingly. **vpa-for-vpa (OOMKilled recommender)** The VPA recommender deployed by `vpa-for-vpa` was hitting its 512Mi memory limit and being OOMKilled. Increase the limit to 1Gi to keep the recommender running stably. ### Release note ```release-note fix(http-cache): rename haproxy backend to avoid HAProxy 3.3 name conflict between frontend and backend fix(vpa): increase vpa-for-vpa recommender memory limit from 512Mi to 1Gi to prevent OOMKilled restarts ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Corrected HAProxy routing so HTTP traffic is directed to the cache backend. * Increased the VPA recommender’s memory limit to improve reliability under higher workloads. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrey Kolkov
|
c503ae29d9 |
fix(backups): request S3 checksum only when required for barman-cloud (non-AWS S3 / Ceph RGW) (#3417)
## What this PR does Fixes CNPG **barman-cloud plugin** backups to non-AWS, S3-compatible object stores (Ceph RADOS Gateway, and some MinIO / Cloudflare R2 builds). The plugin's sidecar uploads via **boto3**. Since botocore ~1.36 (early 2025) the default `RequestChecksumCalculation` is `when_supported`, so a flexible checksum (CRC32) plus the corresponding `x-amz-content-sha256` handling is attached to every `PutObject`. AWS S3 accepts it, but several S3-compatible backends reject the request: ``` InvalidArgument: x-amz-content-sha256 must be UNSIGNED-PAYLOAD, STREAMING-AWS4-HMAC-SHA256-PAYLOAD or a valid sha256 value. ``` so every backup / WAL-archive upload fails against them and the `ScheduledBackup` never stores anything. This sets `spec.instanceSidecarConfiguration.env` on the rendered `ObjectStore`s to carry `AWS_REQUEST_CHECKSUM_CALCULATION=when_required` (botocore reads this env var from the sidecar's environment). `when_required` computes a checksum only for operations that mandate one; a plain `PutObject` then carries none, which both AWS S3 and the affected backends accept — a safe default everywhere. No API/CRD change, no new value, no new dependency. Applied to every barman-cloud `ObjectStore` Cozystack creates: - **chart-rendered** — `packages/system/keycloak/templates/db.yaml` (Keycloak system DB) and `packages/apps/postgres/templates/db.yaml` (postgres app — backup **and** bootstrap-recovery `ObjectStore`s), via a shared `cozy-lib` helper (`cozy-lib.barman.checksumSidecarConfiguration`); - **Go-constructed** — the platform-managed `useSystemBucket=true` `ObjectStore` SSA-applied by `internal/backupcontroller/cnpgstrategy_controller.go` (`barmanSidecarConfiguration`), whose default system bucket is SeaweedFS. Regression coverage: a Go unit test on the SSA-applied `ObjectStore`, plus helm-unittest assertions on `spec.instanceSidecarConfiguration.env` for all three rendered `ObjectStore`s (mutation-checked — reverting the change fails them). Mirrors the same fix already applied to the etcd path in `cozystack/etcd-operator#342` (released in `v0.5.3`) for the same Ceph RGW backend. **Verification:** reproduced end-to-end against a real Ceph RGW endpoint. With default botocore, `PutObject` fails with the `InvalidArgument` above; with `AWS_REQUEST_CHECKSUM_CALCULATION=when_required` in the environment it succeeds. Deployed the plugin + a throwaway CNPG cluster whose `ObjectStore` carried this env: continuous WAL archiving reported `ContinuousArchiving=True` and an on-demand `Backup` completed, with the base backup and WAL segments present in the bucket. `helm template` renders the env on all three `ObjectStore`s; the `packages/apps/postgres` unit-test suite still passes. ### Screenshots N/A — no UI changes. ### Downstream repositories <!-- Walked the trigger map in docs/agents/contributing.md against the diff, file by file. The diff touches only two Helm templates (ObjectStore renders); it adds no package, no values.yaml / values.schema.json field, no API/CRD change, no variant/component/asset change, no ApplicationDefinition change, and no developer-tooling change. None of the trigger-map entries match. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(backups): CNPG barman-cloud backups now work against non-AWS S3-compatible object stores (Ceph RADOS Gateway, some MinIO/Cloudflare R2). The backup sidecar requests an S3 request checksum only when required, avoiding the "x-amz-content-sha256 must be UNSIGNED-PAYLOAD, ..." error that previously broke every backup and WAL-archive upload to those backends. No change for AWS S3. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved compatibility with S3-compatible, non-AWS storage backends for PostgreSQL and Keycloak backups and recovery. * Pin the barman-cloud sidecar to compute AWS request checksums only when required (`AWS_REQUEST_CHECKSUM_CALCULATION=when_required`) to prevent failures with strict S3 gateways. * **Tests** * Updated/added assertions in controller and Helm/YAML suites to verify the checksum pin is applied to generated and recovery ObjectStores (Postgres and Keycloak). <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
a2acfcde1a |
docs(release): add changelog for v1.6.0 (#3429)
## What this PR does Adds `docs/changelogs/v1.6.0.md` (667 lines), generated locally with the process in `docs/agents/changelog.md`. Normally `tags.yaml`'s `generate-changelog` job produces this on the stable tag push, but that job never ran for `v1.6.0` — see the note below. **Coverage boundary.** The changelog runs through #3386, i.e. the release period ending at the `v1.6.0-rc.4` cut. Roughly eight PRs merged to `main` after that cut are deliberately absent, because `v1.6.0` is a promotion rather than a rebuild: every shipped artifact is pinned to an rc.4 digest (`cozystack-packages@sha256:bf682087…`, `cozystack-operator@sha256:93b305f7…`), so that later code is in the tagged tree but not in anything users install. The changelog describes what shipped. **Why this is a manual PR.** `generate-changelog` and `update-website-docs` are jobs in `tags.yaml`, triggered by `push: tags`. The `v1.6.0` tag was pushed by `pull-requests-release.yaml`'s finalize job using the `cozystack-ci` app token, and no workflow run was created for that ref — every rc tag, pushed by `cut-prerelease.yaml` with the same app, did trigger it. `v1.6.0` is the first stable tag created through the promote flow, so this path had never run before. The cause is not yet established and needs a repro rather than a guess; the website docs versioning is still outstanding for the same reason. ### Screenshots N/A — documentation only. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff, which is a single new file under `docs/changelogs/`. No trigger matches: the map covers package layout, release assets, CRDs, workflow behaviour and naming contracts, none of which a changelog file touches. The separate repository list in `docs/agents/changelog.md` was already consumed while generating this file — it feeds the notes, it is not a code coupling. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Added comprehensive release notes for version 1.6.0. * Documented breaking changes, upgrade prerequisites, manual upgrade steps, and post-upgrade expectations. * Highlighted major features, deprecations, security updates, bug fixes, operational notes, and component version updates. * Included configuration, CI, development, and related-project changelog updates. * Added contributor acknowledgements and a link to the full changelog. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
764e30418d |
chore(release): promote v1.6.0-rc.4 -> v1.6.0 (#3425)
Promotes `v1.6.0-rc.4` to stable `v1.6.0` — no rebuild. The tree pins the rc's images by digest. On merge, pull-requests-release.yaml creates the write-once `v1.6.0` tag at the merge commit, retags those digests to `v1.6.0` (+`:latest` when this is the newest stable), publishes the stable cozy-installer chart, and publishes the release — so the stable artifacts are bit-for-bit the rc that passed e2e. The `full-e2e` label forces the full suite on this PR. Do NOT squash-merge (decision B): the stable tag must attach to a real merge commit.v1.6.0 api/apps/v1alpha1/v1.6.0 |
||
|
myasnikovdaniil
|
d3815a4ce6 |
Merge branch 'main' into release-1.6.0
Signed-off-by: myasnikovdaniil <60174387+myasnikovdaniil@users.noreply.github.com> |
||
|
cozystack-ci[bot]
|
235d38dae1 |
Prepare release v1.6.0 (promoted from v1.6.0-rc.4)
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> |
||
|
myasnikovdaniil
|
706c3f06fc |
fix(release): reuse only an OPEN promote PR, not one in any state (#3424)
## What this PR does The promote step guarded PR creation with `gh pr view <branch>`, which resolves a pull request by head branch **in any state**. A closed one satisfied the guard, so creation was skipped while the step still exited 0 — a promotion that reports success and leaves nothing to merge. Same silent-skip shape as the enumeration bug fixed in #3404: the run is green, the artifact is missing, and nothing says so. `STABLE_BRANCH` is derived from the stable version (`release-${version}`), not from the rc number, so every rc promoted to that version shares it. Once any promote PR for a version has been closed, both re-dispatch paths this workflow documents as supported are wedged: re-dispatching the same rc, and promoting a newer rc to the same version. That second one is named in the workflow's own header as the reason a leftover draft is tolerated — the tolerance was implemented for the release draft and missed for the PR. Neither path recovers by reopening the old PR, because the step's own `git checkout -B` plus force-push makes that PR's head unreachable, and GitHub then refuses: ``` 422 Validation Failed state cannot be changed. The release-1.6.0 branch was force-pushed or recreated. ``` The fix asks the question the guard meant to ask — is there an **open** PR for this head — and creates one otherwise. **Hit live.** Promoting `v1.6.0-rc.4` after #3397 had been closed: [run 29917852639](https://github.com/cozystack/cozystack/actions/runs/29917852639) went green, logged `PR already open for release-1.6.0`, and opened no PR. This currently blocks the v1.6.0 release. **Verified in both directions** against live data, so the change is not vacuous: | head branch | PR state | old guard | new guard | | --- | --- | --- | --- | | `release-1.6.0` | #3397 closed | `TRUE` → skip (bug) | `FALSE` → create ✅ | | `chore/gitignore` | #3412 open | `TRUE` → skip | `TRUE` → skip ✅ | `actionlint` exits 0 and `zizmor` reports no findings. The guard reads `${STABLE_BRANCH}` as a shell variable from the step's `env:` block rather than as a `${{ }}` expansion inside `run:`, so no expression-injection surface is added. ### Screenshots N/A — no UI change. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff, which is one file: `.github/workflows/promote-rc.yaml`. The only workflow-related trigger in the map is cozystack/ccp on "change release-prep behaviour in `.github/workflows/tags.yaml`", which this does not touch. This change also restores the documented behaviour of `promote-rc.yaml` rather than altering its contract, so nothing downstream sees a result different from what the docs already promise. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(release): promoting a release candidate now opens the promotion pull request even when an earlier promotion attempt for the same version was abandoned. Previously the workflow mistook a closed pull request for an open one, skipped creating a new one, and reported success with nothing left to merge. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved release PR creation checks so closed or previously merged pull requests no longer prevent new release PRs from being opened. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
6a41d65f16 |
test(e2e): avoid ripgrep dependency in diagnostics tests
Assisted-By: GPT-5 <noreply@openai.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
ef2fa628f5 |
fix(ci): make pre-commit locking work in worktrees
Assisted-By: GPT-5 <noreply@openai.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
4e0cfd994d |
test(e2e): render COSI catch output as YAML
Assisted-By: GPT-5 <noreply@openai.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
ad1adbf785 |
test(e2e): fix diagnostic collector blind spots
Assisted-By: GPT-5 <noreply@openai.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
myasnikovdaniil
|
55195ded0c |
test(e2e): capture previous container logs and stop killing the snapshot (#3346)
## What this PR does Root-causing a recent E2E failure was nearly blocked because the diagnostics captured on failure had collected almost nothing. Two independent mechanisms were responsible, and both had been silently in place for a long time. **The failing container's previous logs were never captured.** The pod had crash-looped, so the decisive evidence lived in the first container instance, reachable only through `kubectl logs --previous`. Chainsaw's `podLogs` action has no `previous` field, so this could not be expressed per-suite; the conclusion had to be reconstructed from indirect evidence instead. A new capture runs from the global catch and from the BATS failure hook, selecting containers at runtime by `restartCount > 0` across both init and regular containers, so a healthy run stays quiet and future suites are covered without per-suite wiring. **The catch was structurally guaranteed to be killed.** A Chainsaw `script` op is bounded by `timeouts.exec`, which is 5m, while the catch's own inner budgets summed to roughly 960s. It reported `SCRIPT | ERROR | signal: killed` and produced nothing — and its second leg, the data-plane capture, had therefore never run at all. The kill signature was itself misleading: `timeout` exits 124 and cannot produce that message, which comes from Go's `exec.Cmd` killing the process group. Each catch op now carries an explicit timeout above the budgets it contains, so the inner one fires and leaves a partial, usable artifact instead of losing everything. **The snapshot was truncated at 60 seconds.** `crust-gather --duration` defaults to 60s and was never passed, so every snapshot ever taken was cut short, with nothing in the artifact saying so. The API discovery it performs first is not covered by that budget at all, which is where the outer wall-clock was going on a cluster with a degraded aggregated APIService — precisely the state a failed run is in. Both budgets are now set explicitly, and the collector's output goes to a log inside the snapshot with a completeness verdict, so "is this snapshot complete?" is answerable from the artifact. Failure paths are reported rather than swallowed: an unreachable API says so instead of implying nothing had restarted, a truncated capture is kept and marked, and every dropped item is logged, since silent truncation reads as full coverage. ### What is not verified here The mechanism claims are read from the pinned versions of Chainsaw and crust-gather. Everything cluster-facing is not verified by this PR: the go-template's output shape against a live API server, `--previous` returning the first instance for a real crash-looper, and whether the new collection budgets suffice on a degraded cluster. The bats suite covers the selection helpers only, and those cases are mutation-tested. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Diagnostics** * Improved end-to-end failure evidence by capturing “previous instance” logs for crash-looping pods (including restarted init containers) and prioritizing relevant namespaces. * Added time-bounded diagnostics collection with explicit outer timeouts; collector output is saved to per-snapshot log files. * Classifies collector outcomes as complete, timeout-truncated, or failed, while preserving the original test failure signal. * **Documentation** * Expanded E2E failure-diagnostics guidance with new timeout and evidence-capture conventions. * **Tests** * Added unit-style tests for previous-log selection, prioritization, row limiting, and deterministic log artifact naming. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
baf0158d81 |
fix(release): reuse only an OPEN promote PR, not one in any state
The promote step guarded PR creation with `gh pr view <branch>`, which resolves a pull request by head branch in any state. A closed one satisfied the guard, so creation was skipped while the step still exited 0 — a promotion that reports success and leaves nothing to merge, the same silent-skip shape as the enumeration bug fixed in #3404. STABLE_BRANCH is derived from the stable version, so every rc promoted to that version shares it. Both re-dispatch paths this workflow documents as supported are therefore wedged once any promote PR for that version has been closed: re-dispatching the same rc, and promoting a newer rc to the same version. Neither recovers by reopening the old PR, because the step's own force-push makes its head unreachable and GitHub refuses with "state cannot be changed. The <branch> branch was force-pushed or recreated." Hit live while promoting v1.6.0-rc.4 after #3397 had been closed: the run went green and opened no PR. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Timofei Larkin
|
d5806eb1a0 |
feat(kubernetes-nodes): split worker node pool into its own app (Phase 2, additive) (#3314)
## What this PR does Phase 2 of the Kubernetes-app split ([cozystack/community#8](https://github.com/cozystack/community/pull/8)) — the **additive** half. Introduces `kubernetes-nodes`: a per-pool application that renders the worker node-pool objects (`KubevirtMachineTemplate`, `MachineDeployment`, `MachineHealthCheck`, worker `WorkloadMonitor`, and the `talos-reconcile` Job + RBAC) extracted from the monolithic `kubernetes` chart. A `KubernetesNodes` CR named `<cluster>-<pool>` attaches to a parent `Kubernetes` cluster in the same namespace by name convention (`kubernetes-<cluster>`), mirroring the `vm-instance`/`vm-disk` precedent. This PR is additive: the parent `kubernetes` chart gets exactly one scoping fix (its old-KMT preservation block is limited to KMTs owned by its own Helm release) so that attaching a `kubernetes-nodes` pool to a live cluster cannot break the parent's next upgrade. Existing clusters keep provisioning workers via `nodeGroups`. Removing `nodeGroups` from the parent and adopting existing pools into `kubernetes-nodes` releases is a follow-up **breaking** PR (Phase 2b). Key properties: - Pool objects render **byte-identical** to the parent for the same group (golden-parity check in `tests/render-parity.sh`), so the follow-up adoption never churns live worker VMs — the content-hashed `KubevirtMachineTemplate` name is preserved. - cluster-autoscaler (kept in the parent chart) discovers the pool's `MachineDeployment`s cross-release via the unchanged `cluster.x-k8s.io/cluster-name` label, namespace, and min/max annotations. - Registered as a dedicated `cozystack.kubernetes-nodes-application` PackageSource + `kubernetes-nodes-rd` ApplicationDefinition, wired into the iaas bundle (mirrors `vm-instance`/`vm-disk`). ### Downstream repositories Left unticked deliberately — this is a draft. A new user-facing `KubernetesNodes` API is added, so cozystack/website (docs) and cozystack/terraform-provider-cozystack likely need follow-ups; the trigger-map walk will be completed before this leaves draft. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: ```release-note Add the `kubernetes-nodes` application: manage the worker node pools of a managed Kubernetes cluster as independent resources, one HelmRelease per pool. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a new Kubernetes worker node pool offering with configurable sizing (CPU/memory/instance, disk), roles, storage class, GPUs, and Talos worker image settings. * Introduced a dedicated Helm package, dashboard/resource-map access, and Flux application wiring for installing/managing pools. * **Bug Fixes** * Improved safeguards against pool naming/ownership conflicts and tightened upgrade behavior for reusing previously created templates. * **Documentation** * Added full node pool configuration docs and operational notes (including supported Kubernetes versions and reconciliation expectations). * **Tests** * Added node pool rendering/validation tests plus golden parity checks between chart layouts. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrey KolkovandClaude Opus 4.8
|
80a03b1e09 |
test(backups): assert the barman-cloud sidecar checksum env on rendered ObjectStores
helm-unittest coverage for the S3 request-checksum fix (was mutation-negative before): assert spec.instanceSidecarConfiguration.env pins AWS_REQUEST_CHECKSUM_CALCULATION=when_required on all rendered barman-cloud ObjectStores — postgres backup + bootstrap-recovery (packages/apps/postgres/tests/backup_storage_test.yaml) and the Keycloak DB (new packages/system/keycloak/tests/db_backup_test.yaml). Verified by mutation: reverting the template change now fails these assertions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io> |
||
|
Myasnikov Daniil
|
284a17ba1f |
fix(release): unblock the v1.6.0 promote on the rc.4 staging tree
Promoting this branch fails in two independent ways, both landing after finalize has already created the write-once stable tag, published the GitHub release and moved :latest. chBackupClientImage had no producer and froze at v1.4.0-rc.2 while .migrations.image advanced to v1.6.0-rc.4, leaving platform-migrations pinned at two digests. promote-retag.sh copies every collected <repo>@<digest> to <repo>:<stable>, so both compete for one destination tag: the lower-sorting v1.4.0-rc.2 digest wins it and the second copy trips the write-once guard, aborting the retag with five repositories still untagged and the installer chart unpublished. Align it to the digest this branch already carries for the migrations image. Both come from the same tree, so the runner keeps the curl, jq and ca-certificates the Altinity strategy Pod needs, and that Pod overrides the ENTRYPOINT with its own command. promote-rc.yaml on main now calls hack/promote-rewrite-tags.sh, which does not exist here, so the dispatch would fail outright. Vendor it and hack/lib/image-refs.sh verbatim from main (identical blobs). That also fixes the rewrite's coverage: the old inline glob enumerated 155 files and missed 10 rc-bearing ones, which is why the earlier promote left 11 references reading v1.6.0-rc.4. The shared enumeration covers all 168 and fails the promotion if any rc reference survives. Both fixes are on main via #3405 and #3404; this carries them onto the rc.4 staging tree so v1.6.0 ships the bytes rc.4 tested. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
myasnikovdaniil
|
d0c0f399f8 |
fix(ci): cover every image-ref storage shape in promote, retag and mirror (#3404)
## The bug Promotion rewrites the rc version substring to the stable version across the vendored image references. That rewrite lived inline in `promote-rc.yaml` and globbed the depth-2 package `values.yaml` plus `packages/apps/kubernetes/images/*.tag` **only**, on the premise that the kubernetes app was the only one whose `.tag` files carry the cozystack version. Nine other `.tag` files do, and `system/multus` seds its reference straight into a vendored upstream daemonset manifest. #3397 (promote `v1.6.0-rc.4` → `v1.6.0`) is therefore staged with **33 of 55 first-party references still reading `v1.6.0-rc.4`**. `hack/promote-retag.sh` and `hack/nightly-mirror.sh` had the same blind spot from the other direction — both scanned the depth-2 `values.yaml` alone, so those images were never retagged to the stable version (**30 references selected before, 42 after**) and never mirrored to the public registry for a nightly. A retag miss leaves an image short of a tag, since the digest still resolves inside one registry. A mirror miss is worse: the host rewrite walks the same file list, so a reference that is neither mirrored nor rewritten leaves the published tree pointing at the private build registry. `v1.6.0` is the first release to go through promotion — every earlier release was a full rebuild from the tag, which stamped every file uniformly. The glob has always been wrong; the old path just made it unobservable. ## Why it could not be caught before The rewrite was workflow-inline, so there was nothing to unit test. The only way to observe the miss was to cut a release. It is a script now — `hack/promote-rewrite-tags.sh` — and `hack/promote-rewrite-tags_test.bats` round-trips it against the real tree under `make unit-tests`, with no cluster, registry or release involved. ## The fix The three call sites each carried their own idea of where a reference can live, which is how they drifted apart. That knowledge moves into `hack/lib/image-refs.sh` as the single enumeration — `image_ref_files()` for the files, `collect_image_refs()` for the references inside them — and all three source it, so a new storage location is declared once and reaches every consumer. The rewrite also asserts its own postcondition: it scans **wider** than it rewrites and fails the promotion when a reference survives in a location the enumeration does not know about, rather than shipping it quietly. The wide scan is shape-filtered to image-reference position, because `X.Y.Z-rc.N` is an ordinary version string that other things legitimately contain — `kubevirt-csi-driver/go.sum` pins `github.com/golang/protobuf v1.4.0-rc.2`, and `v1.4.0-rc.2` is a cozystack rc that was actually cut. Narrowing it to `${RC}@sha256:` instead would have been wrong: `system/cilium/values.yaml` keeps `tag:` and `digest:` under separate keys. `docs/agents/image-refs.md` records the contract — three tag classes, three storage shapes, the invariants, and which consumer does what. ## Not a bug: kamaji `cluster-api-control-plane-provider-kamaji` stays at `v0.19.0-cozystack.0` across the promotion. It is a first-party rebuild of an upstream component, versioned by that component rather than by the cozystack version line, so no rewrite is owed. A test pins this so a future fix here cannot over-reach. ## Documented known gaps, deliberately not fixed here Both are pre-existing and in the **host** dimension; the version dimension this PR owns is complete, because the version always lives in a `tag` key regardless of where the host sits. - **The nightly host rewrite cannot reach a split host.** It is a literal `<src-registry>/` substring replace, so `keycloak-operator` (host in a sibling `registry:` key) and `kubeovn` (`global.registry.address`, no trailing slash) are mirrored but keep pointing at the build registry in the published tree. Fixing it needs structure-aware rewriting. - **A reference inside a gzip is invisible.** `capi-providers-cpprovider` ships `files/components.gz`, and the promote postcondition cannot catch this class at all since `grep -rIl` skips binaries. Nothing is owed today — kamaji is component-versioned and the image is still mirrored and retagged via its `.tag` — so the residual is that a nightly's kamaji ConfigMap keeps the build-registry host. ## Testing - 17 tests in the new suite; `make bats-unit-tests` green (312 tests, exit 0); `shellcheck -x` clean; `zizmor` clean on the workflow. - Verified against four real rc versions (`1.4.0-rc.2`, `1.4.0-rc.4`, `1.5.0-rc.2`, `1.6.0-rc.4`) in both directions — no false positives, no false negatives. - Every test is mutation-proven. Reverting the enumeration to the old glob, deleting any of the five YAML shapes, corrupting emitted digests, reverting extras to parse-only, dropping the unreadable-file guard, broadening shape 4 beyond `global.images`, or planting an unenumerated first-party reference in any storage shape each fail a named test. ## Reviewer note, out of scope `platform-migrations` is pinned at two different digests — `core/platform/values.yaml` at `v1.5.0` and `backupstrategy-controller/values.yaml` at `v1.4.0-rc.2` (an unowned pin; no Makefile writes that key, tracked in #3143). A `promote-retag.sh` dry-run emits **two copies to the same destination tag**, so the write-once guard will fail the promotion. This is independent of this PR and needs a decision about the backup-client image version, so it is left alone here — but it blocks the 1.6.0 promote and wants fixing before the re-cut. ## Release note ```release-note Promotion, retagging and nightly mirroring now cover every image reference the build stamps, including those stored in `images/*.tag` files and in vendored manifests. Previously a promoted release could ship images still tagged with the release-candidate version, and those images received no stable tag in the registry. ``` Fixes the promote half of #3143. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Image promotion and nightly mirroring now detect digest-pinned image references across values files, standalone tag files, and supported templates. - Stable release preparation supports RC→stable tag rewriting via a dedicated promotion script, with broader verification to ensure no RC references remain. - **Bug Fixes** - Improved completeness of ref discovery and scanning, including previously missed template/tag locations. - **Documentation** - Added and linked guidance documenting image-reference invariants and how tooling enumerates supported storage shapes. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
bbfe9f9a9d |
fix(backupstrategy-controller): give chBackupClientImage a producer (#3405)
## This is a workaround `backupstrategy-controller` should not be running the platform **migrations** image. This PR does not fix that — it makes the existing reuse *safe* so the v1.6.0 promote can proceed, and documents in the tree exactly what is wrong and what should replace it. Treat it as a stopgap with a TODO, not as an endorsement of the design. ## What is actually wrong The Altinity strategy Pod drives clickhouse-backup's HTTP API. Its script uses exactly four binaries — `curl`, `jq`, `sleep`, `date`. For that it currently pulls `platform-migrations`: | | | | --- | --- | | Size | **349 MB** (a dedicated `alpine + curl + jq + ca-certificates` measures **23 MB**) | | Also contains | `kubectl`, `helm`, `git`, `cozyhr`, the `etcd-migrate` binary, every migration script, vendored etcd CRDs | | `ENTRYPOINT` | `run-migrations.sh` | Two consequences worth naming. A tenant-adjacent backup Pod carries cluster tooling it has no use for. And because the image's own entrypoint is the migration runner, the `command:` override in the strategy template is load-bearing — drop that one line and the Pod runs platform migrations with `kubectl` and `helm` already on the PATH. It also creates the coupling that caused the bug this PR fixes: bumping `ETCD_OPERATOR_VERSION` or editing any migration script re-digests the image, forcing the backup client's pin to move for reasons that have nothing to do with backups. ## The release blocker `platform-migrations` was pinned at two different digests in the committed tree: | Location | Pin | Producer | | --- | --- | --- | | `packages/core/platform/values.yaml` `.migrations.image` | `v1.5.0@sha256:8bf61f17…` | `packages/core/platform/Makefile` | | `backupstrategy-controller/values.yaml` `.chBackupClientImage` | `v1.4.0-rc.2@sha256:17390197…` | **none** | Promotion retags **by digest**: `hack/promote-retag.sh` copies every collected `<repo>@<digest>` to `<repo>:<stable-version>`. Two digests under one repository produce two copies competing for the same destination tag — the first wins, the second hits the write-once guard, and the promotion fails. A dry run against `main` emits exactly that pair: ``` ▸ ghcr.io/cozystack/cozystack/platform-migrations sha256:17390197… → docker://ghcr.io/cozystack/cozystack/platform-migrations:v9.9.9 ▸ ghcr.io/cozystack/cozystack/platform-migrations sha256:8bf61f17… → docker://ghcr.io/cozystack/cozystack/platform-migrations:v9.9.9 ``` `v1.6.0` is the first release cut through promotion rather than a full rebuild. Every earlier release ran `make build`, which restamped both copies as a side effect and hid the missing producer. ## What this PR does `packages/core/platform/Makefile` now stamps both keys with the same ref, so they move in lockstep by construction rather than by instruction, and the committed value is aligned to the digest `core/platform` already carried. Moving the runner forward two minors is safe: it is used only as a curl + jq container, and the image built at `v1.5.0` carries them (`git show v1.5.0:…/migrations/Dockerfile` → `apk add … jq ca-certificates bash curl`). ## What should replace it A dedicated `ch-backup-client` image built by **this package's own Makefile** — measured at 23 MB, 15× smaller, with none of the cluster tooling and none of the entrypoint hazard. That also deletes the cross-package stamp this PR adds, because the pin would be produced by the package that owns it. The objection that has kept this reuse in place — "a second tag release CI would have to re-digest each cut" — does not survive contact with the actual defect. Every other first-party image already pays that cost, automatically. What broke here was not an extra tag; it was **a pin with no producer**, which a package-owned image cannot reproduce. I am deliberately not doing that here: it means a new image, a new build target and a new digest introduced during a release cut. It belongs in a follow-up. ## The guard `hack/image-pin-consistency.bats` asserts the general invariant — no repository pinned at more than one digest — driven through the real promotion selector so it tracks whatever set promotion actually acts on, rather than a reimplementation that could drift from it. Plus a specific lockstep assertion so a recurrence names its cause. This check could not have been added before now: it fails on the very duplicate this PR removes. ## Testing - `hack/image-pin-consistency.bats` — both tests pass; verified they fail when the drifted pin is restored. - `make bats-unit-tests` green (294 tests, exit 0). - `make -C packages/system/backupstrategy-controller test` — 11 tests, 3 suites, green. - `helm template … --set backupStorage.bucketNameOverride=test-bucket` renders the strategy Pod with the new ref. - The Makefile's `yq` step was simulated directly to confirm the relative path and key path resolve and that comments survive the in-place edit. - Merged locally with #3404 onto `main` and re-run: the expanded ref collector in that PR surfaces no additional duplicate pins (314 tests, exit 0). ## Relationship to other work Relates to #3143 — this is one instance of its unowned-pin class, the one its discussion flagged as having no producer. The four never-built packages there remain untouched. Independent of #3404 (promote/retag/mirror storage-shape coverage); the two touch no common files and can merge in either order. Both are needed before the `v1.6.0` promote is re-cut. ## Release note ```release-note Fixed a release-blocking duplicate pin: the ClickHouse backup client image is now stamped from the same source as the platform migrations image it reuses, so both carry one digest. Previously they could drift, which made promotion attempt to retag two digests to the same stable tag and fail. Reusing the migrations image here remains a documented workaround pending a dedicated backup-client image. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Ensured the platform migrations image digest is pinned consistently between the platform and backup strategy components. * Updated the backup strategy component to reference the new pinned `platform-migrations` digest. * **Tests** * Added promotion/retag consistency checks to fail on conflicting source digests for the same stable tag. * Added drift detection to ensure `platform-migrations` image references match across consumer configuration files. * **Documentation** * Expanded in-chart comments and values documentation clarifying the coupling and workaround behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrey KolkovandClaude Opus 4.8
|
b574e2176f |
fix(backups): cover Go-driven system-bucket ObjectStore + share the checksum helper
Address review on the S3 request-checksum fix: - Extend the fix to the platform-managed useSystemBucket=true flow, whose ObjectStore is built in Go by applyClusterPluginBackup (SSA-applied), not chart-rendered. Add InstanceSidecarConfiguration to cnpgtypes.ObjectStoreSpec and set AWS_REQUEST_CHECKSUM_CALCULATION=when_required on it via barmanSidecarConfiguration(). The platform's own default system bucket is SeaweedFS, a non-AWS S3 gateway of the same class as Ceph RGW. Unit test in cnpgstrategy_controller_test.go asserts the applied ObjectStore carries it. - Factor the duplicated instanceSidecarConfiguration block (3x across two charts) into a shared cozy-lib helper (cozy-lib.barman.checksumSidecarConfiguration). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io> |
||
|
Andrey Kolkov
|
ad3f8b2487 |
fix(backups): check cluster health before the restore deadline (PITR)
Addresses review CHANGES_REQUESTED on #3383. CRITICAL: reconcileCNPGRestore checked the restore deadline BEFORE the cluster-health check, so a reconcile that first observes "deadline elapsed" AFTER the recovery Cluster already reached a healthy state (a controller restart or stalled workqueue spanning restoreTimeoutSeconds) would mark an already-converged restore Failed / RecoveryTargetUnreachable. A false Failed can then escalate to data loss: a resubmitted RestoreJob's purge guard treats the recovered Cluster as a stale leftover and deletes it + its PVCs. Fix: check hasRecovery + cnpgClusterHealthy first and return Succeeded unconditionally when healthy, regardless of how late the reconcile runs; only a not-yet-healthy restore reaches the deadline/classification branch. Adds TestReconcileCNPGRestore_HealthyPastDeadlineSucceeds (StartedAt 72h past the deadline + healthy Cluster + recoveryTime set; the injected readPodLog fails the test if the classification path is entered, proving health short-circuits). MINOR: handle the deferred Close error on the pod-log reader (errcheck). Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey KolkovandClaude Opus 4.8
|
0c586b9d72 |
fix(backups): request S3 checksum only when required for barman-cloud
The barman-cloud plugin's sidecar uploads via boto3. Since botocore ~1.36 (early 2025) the default RequestChecksumCalculation is when_supported, which attaches a flexible checksum to every PutObject. Non-AWS S3-compatible backends (Ceph RADOS Gateway, some MinIO / Cloudflare R2 builds) reject the accompanying header with "InvalidArgument: x-amz-content-sha256 must be UNSIGNED-PAYLOAD, STREAMING-AWS4-HMAC-SHA256-PAYLOAD or a valid sha256 value", so every backup/WAL-archive upload fails against them. Pin the sidecar to when_required via the ObjectStore's spec.instanceSidecarConfiguration.env (botocore honors the AWS_REQUEST_CHECKSUM_CALCULATION env var) on both the keycloak system DB and the postgres app (backup + recovery ObjectStores). when_required is also accepted by AWS S3 on a plain PutObject, so it is a safe default everywhere. This mirrors the etcd-operator fix (cozystack/etcd-operator#342, released in v0.5.3), which applied the same WhenRequired policy to its aws-sdk-go-v2 S3 client for the same Ceph RGW backend. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io> |
||
|
Andrey KolkovandClaude Opus 4.8
|
b5d03f889b |
test(backups): reconcile-level test for the PITR deadline classification
Review of #3383 found the RecoveryTargetUnreachable classification branch in reconcileCNPGRestore was only covered through the pure helpers and recoveryTargetUnreachable directly, never through reconcileCNPGRestore itself — a mutation swapping the call-site reason back to the generic RestoreFailed kept the suite green. Add TestReconcileCNPGRestore_DeadlineClassifiesRecoveryTargetUnreachable: it backdates StartedAt past the restore deadline with a recoveryTime set and drives the real reconcile through the readPodLog seam, asserting all three deadline sub-paths — FATAL present (reason RecoveryTargetUnreachable on both RecoveryConverged and Ready), FATAL absent (generic timeout, no RecoveryTargetUnreachable), and a Forbidden log read (generic timeout + the missing-pods/log-RBAC note). Verified it fails when the call-site reason is reverted to the generic RestoreFailed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io> |
||
|
Myasnikov Daniil
|
676c8b888a |
docs: add changelog for v1.6.0
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Andrey Kolkov
|
8ea589230f |
test(mariadb): add MariaDb backup/restore e2e test (#3384)
## What Reworks the never-merged [`backup-mariadb-e2e`](https://github.com/cozystack/cozystack/tree/backup-mariadb-e2e) branch onto the modern Chainsaw e2e approach ([`docs/agents/e2e-testing.md`](https://github.com/cozystack/cozystack/blob/main/docs/agents/e2e-testing.md)), rebuilt on fresh `main`. That branch bundled the whole MariaDB backup-strategy controller **plus** a `hack/e2e-apps/backup-mariadb.bats` test. The controller and the `examples/backups/mariadb/` manifests have since landed on `main` independently (`feat(platform): implement backup strategy for mariadb`), and the `hack/e2e-apps/*.bats` app suite has been replaced by `hack/e2e-chainsaw/`. So the only piece still missing was the e2e coverage — this PR delivers it the modern way. ## Changes - **`examples/backups/mariadb/` → executable harness.** Converts the demo from static numbered YAML into the same script harness the other backup demos use (`00-helpers.sh`, `run-all.sh`, `cleanup.sh`), keeping the numbered manifests as the human-readable flow that `run-all.sh` drives — so the documented flow and the automated test can never drift. `run-all.sh` provisions a `Bucket`, derives the two per-app Secrets the strategy references from `BucketInfo` + the seaweedfs CA, deploys a source MariaDB with a sentinel row, fires an ad-hoc `BackupJob`, restores to a copy with a to-copy `RestoreJob`, and asserts the sentinel round-tripped through S3 into the copy while the source is left untouched. - **`hack/e2e-chainsaw/mariadb/` → second Chainsaw Test** (`mariadb-2-backup-roundtrip`) that drives `run-all.sh` + `cleanup.sh`, following the merged `etcd-2-backup-roundtrip` harness-driving shape: runs in `tenant-root` (the isolated e2e tenant cannot reach `tenant-root`'s seaweedfs across the Cilium egress policy), `skipDelete` since the harness owns the resources, an idempotent pre-clean step so a run killed before `finally` cannot leave a stale `Succeeded` BackupJob that false-passes, fail-fast on terminal `BackupJob`/`RestoreJob` phases, and scoped `catch` diagnostics. Cleanup also prunes the operator-side `k8s.mariadb.com` `Backup`/`Restore` CRs the Cozystack jobs own only by label. - **`hack/select-e2e.sh` → TIA rule for `examples/backups/<app>/`.** The harness lives outside the suite dir, so without this a harness-only diff produced an empty selection, which `pull-requests.yaml` turns into "skip the whole E2E stage" — a future bug-fix to `run-all.sh` would merge with its own round-trip test never run. The new rule maps `examples/backups/<app>/` to the `<app>` suite that drives it (dir name == suite name for every backup demo except `vmi` → `vminstance`), with three cases added to `hack/select-e2e_test.bats`. ## Conventions followed - No retries on deterministic steps; the single `script` step carries a justified 30m budget (2 MariaDB instances × 310s first-boot startup probe + dump/restore) and fails fast rather than polling out. - Self-contained trap-based temp-file cleanup only inside a single step; no test-level EXIT trap. - Controller-created artifacts Chainsaw cannot reclaim are pruned explicitly by `cleanup.sh`. - TIA verified: `examples/backups/mariadb/run-all.sh` now selects the `mariadb` suite (`hack/select-e2e.sh`), and editing `select-e2e.sh` itself escalates this PR to the full suite. ## Downstream Repositories Walked the trigger map in `docs/agents/contributing.md` against the diff. The change touches only e2e test material, backup examples, and CI suite-selection — no package added/renamed/removed, no `values.schema.json` / `values.yaml` / platform-values / variant / `ApplicationDefinition` / namespace / release-asset changes, and no `hack/` file moved/renamed. No downstream repository is reached. - [x] `cozystack/website` — not affected (no package or platform/docs-source change) - [x] `cozystack/terraform-provider-cozystack` — not affected (no app/schema/enum/default/prefix change) - [x] `cozystack/ansible-cozystack` — not affected (no installer values / namespace / variant / node-prereq change) - [x] `cozystack/ccp` — not affected (no `hack/` move-or-rename, no make-target behavior change) ## Notes No `make generate` needed — the change touches only `examples/`, `hack/`, and a Chainsaw suite (the edited README is under `examples/`, not a package). ## Release note ```release-note Add an end-to-end Chainsaw test for the MariaDB backup/restore round-trip, and convert the `examples/backups/mariadb/` demo into an executable `run-all.sh`/`cleanup.sh` harness. ``` 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added an end-to-end MariaDB backup and restore demonstration workflow. * Added automated backup round-trip validation, including data integrity checks. * Added reusable scripts for running the demo and cleaning up created resources. * Added automatic handling of backup credentials, endpoints, and trusted certificates. * **Documentation** * Expanded setup, configuration, execution, cleanup, and troubleshooting guidance. * Documented same-namespace restore support and cross-tenant limitations. * **Tests** * Added automated Chainsaw coverage for the MariaDB backup and restore workflow. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrey Kolkov
|
efa4e41e9c |
feat(etcd-operator): bump to v0.5.3 (S3 checksum fix for non-AWS backends) (#3403)
## What this PR does Bumps the etcd-operator to **v0.5.3**, which carries the S3 request-checksum fix ([cozystack/etcd-operator#342](https://github.com/cozystack/etcd-operator/pull/342)). `git compare v0.5.2...v0.5.3` is exactly #340, #341, #342. **Why:** scheduled etcd snapshots to non-AWS S3-compatible backends (**Ceph RGW** confirmed) fail at upload. Since early 2025 `aws-sdk-go-v2` defaults `RequestChecksumCalculation` to `WhenSupported`, which stamps a CRC32 on every upload; over HTTPS a multipart part rides it as `x-amz-content-sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER`, and RGW rejects it with `400 InvalidArgument`. v0.5.3 sets `WhenRequired` on both the S3 client and the transfer manager, so multipart uploads (>5 MiB — every real etcd snapshot) carry no checksum trailer. **Changes:** - `system/etcd-operator` — `appVersion` v0.5.2 → v0.5.3 (image tag follows AppVersion; `values.yaml` keeps `tag: ""`), bump `ETCD_OPERATOR_REF`, update deployment unittest expectations. - `system/etcd-operator-crds` — re-vendor CRDs from v0.5.3. `etcdmembers` gains the additive #340 `status.version` field + **Running** printer column and picks up the `peerAutoTLS` spec field the vendored copy was already missing (all additive, backward-compatible; `etcdclusters`/`etcdsnapshots` unchanged). Fixes the stale `make update` (upstream has no `config/crd` kustomization — CRDs live in `charts/etcd-operator/crd-bases`) and makes it fail loudly on a fetch error (temp file under `set -e`, not a pipe). **Verification:** `helm unittest` 18/18 and `helm template` green on both packages; regenerated CRDs are byte-identical to upstream `crd-bases` at v0.5.3. End-to-end on a live Ceph RGW backend (freedom-portal-stage): the v0.5.2 (client-only) agent fails a multipart snapshot upload with the 400; v0.5.3 uploads an 11 MiB multipart snapshot successfully, and the operator-driven CronJob→EtcdSnapshot path reaches `Complete`. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` file-by-file against the diff (`packages/system/etcd-operator{,-crds}/**` only — a component version bump + additive CRD re-vendor). Nothing matches: not an `apps/`/`extra/` package add/rename/remove, no `core/platform` values, no variant/bundle, no platform component add/remove, no `hack/` layout or shared-tooling change, and the provider does not type the `etcd-operator.cozystack.io` CRDs (and the change is additive/opaque regardless). - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(etcd-operator): bump to v0.5.3 — etcd snapshot uploads to non-AWS S3-compatible backends (Ceph RGW, some MinIO/R2) no longer fail with `400 InvalidArgument: x-amz-content-sha256 ...`; the snapshot agent now requests a checksum only when required, on both the S3 client and the multipart transfer manager. Also brings observed EtcdMember versions (#340) and `--watch-namespace` (#341). ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added runtime etcd version visibility to `EtcdMember` (including a “Running” status column). * Added `peerAutoTLS` support to `EtcdMember` for operator-managed peer TLS behavior. * **Updates** * Updated the etcd-operator to version 0.5.3. * Refreshed bundled CRDs and ensured CRDs are preserved during Helm lifecycle operations. * **Tests** * Updated deployment test expectations to use the 0.5.3 operator image. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
a302bd0ddc |
docs(backupstrategy-controller): reframe the migrations-image reuse as a workaround
Running the platform migrations image as the ClickHouse backup client is not a design decision worth defending, and the in-tree comments read as though it were. Reframe them. The strategy script uses exactly curl, jq, sleep and date. For that it pulls a 349 MB image carrying kubectl, helm, git, cozyhr, the etcd-migrate binary and every migration script into a tenant-adjacent Pod, and that image's ENTRYPOINT is run-migrations.sh -- so the `command:` override in the strategy template is load-bearing rather than incidental. A dedicated alpine + curl + jq + ca-certificates image measures 23 MB. The standing objection to a dedicated image -- "a second tag release CI would have to re-digest each cut" -- does not survive contact with the actual defect. Every other first-party image already pays that cost automatically. What broke was a pin with NO producer, which an image owned by its own package cannot reproduce; that fix would also delete the cross-package stamp introduced here. Comment-only. No rendered output changes. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
1f302d0d78 |
docs(agents): correct issue ref, mirror claim and example in image-refs
The build-list violations bullet pointed at #3143, which tracks the adjacent gap (packages in the list going stale between releases) and never covers packages absent from the list; they are now tracked in the newly filed #3416. The nightly split-host paragraph claimed both affected images are mirrored while the published tree points at the build registry — today neither ref carries the source-registry host (keycloak-operator is never rebuilt by CI, kubeovn is built externally by cozystack/kubeovn-chart), so the ownership filter skips them and the gap is latent, not live; reworded to say exactly that. The opening example paired a v1.6.0 tag with the real v1.5.0 grafana digest — a ref that will never exist — and grafana is slated to stop being built; replaced with the literal committed cozystack-api ref. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Andrey KolkovandClaude Opus 4.8
|
4605bbf3db |
test(mariadb): pre-clean the backup roundtrip against stale tenant-root state
Address review feedback on #3384. The TIA rule for
examples/backups/<app>/ that an earlier revision of this commit added
has since landed on main independently (
|
||
|
Andrey KolkovandClaude Opus 4.8
|
2829f7a300 |
docs(mariadb): address review notes on backup example
- 00-bucket.yaml: correct the Secret-name comment to include the "bucket-" release prefix (bucket-<bucket>-<user>), matching the README and 00-helpers.sh. - README: note that S3_CA_NAMESPACE defaults to tenant-root independently of NAMESPACE, so a non-root tenant run must override it (and S3_CA_SECRET) too. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey KolkovandClaude Opus 4.8
|
d1f2822018 |
test(mariadb): add backup/restore round-trip e2e (chainsaw) via example harness
Reworks the never-merged backup-mariadb-e2e branch onto the modern Chainsaw approach (docs/agents/e2e-testing.md). The bats port that branch carried (hack/e2e-apps/backup-mariadb.bats) is dropped — the whole hack/e2e-apps/*.bats app suite has been replaced by hack/e2e-chainsaw/, and the mariadb strategy controller + example manifests it depended on have since landed on main independently. Only the e2e coverage was still missing. Convert examples/backups/mariadb/ from static numbered YAML into the same executable harness the other backup demos use (00-helpers.sh, run-all.sh, cleanup.sh), keeping the numbered manifests as the human-readable flow that run-all.sh drives — so the documented flow and the automated test can never drift. run-all.sh provisions a Bucket, derives the two per-app Secrets the strategy references from BucketInfo + the seaweedfs CA, deploys a source MariaDB with a sentinel row, fires an ad-hoc BackupJob, restores to a copy with a to-copy RestoreJob, and asserts the sentinel round-tripped through S3 into the copy while the source is left untouched. Add a second Chainsaw Test (mariadb-2-backup-roundtrip) to hack/e2e-chainsaw/mariadb/ that drives run-all.sh + cleanup.sh, mirroring postgres-2-backup-roundtrip: runs in tenant-root (the isolated e2e tenant cannot reach tenant-root's seaweedfs across the Cilium egress policy), skipDelete since the harness (not Chainsaw) owns the resources, fail-fast on terminal BackupJob/RestoreJob phases, and scoped catch diagnostics. Cleanup also prunes the operator-side k8s.mariadb.com Backup/Restore CRs the Cozystack jobs own only by label. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey KolkovandClaude Opus 4.8
|
6a1e2d0d06 |
feat(backups): support PostgreSQL point-in-time recovery (PITR)
Makes RestoreJob.spec.options.recoveryTime a genuinely supported, tested capability for the new backups.cozystack.io API (issue #2774). The recoveryTime -> CNPG bootstrap.recovery.recoveryTarget.targetTime plumbing already existed on the barman-cloud-plugin branch; this adds the reliability and verification layer the issue asked for: - Fail-fast for an unreachable target: when a RestoreJob has recoveryTime set and the recovery Cluster is not healthy, the driver reads the CNPG full-recovery pod logs and, once several distinct attempts have exited with PostgreSQL's "recovery ended before configured recovery target was reached" FATAL, fails the RestoreJob with conditions RecoveryConverged=False / Ready=False (reason RecoveryTargetUnreachable) instead of hanging to the restore deadline. Health is checked first, so a converged recovery always wins; counting distinct FATAL-ing attempts (not pod failures) tolerates a transient archive-lag miss on a near-now target and ignores transient recovery-pod crashes. Adds the pods/log RBAC grant the log read requires. - e2e (examples/backups/postgres/run-all.sh, wired into the Chainsaw postgres-2-backup-roundtrip suite): step 45 writes a before marker, captures the server timestamp, writes an after marker, restores to that timestamp and asserts the before row survived and the after row did not; step 46 submits a recoveryTime an hour ahead and asserts it fails fast with reason RecoveryTargetUnreachable. - Working, uncommented example (45-restorejob-pitr.yaml) and a docs section (docs/operations/backup-classes.md) covering the recoverable window, how to discover the earliest/latest restorable time from the backup catalog, the archive-lag tolerance, and idempotency under GitOps. Refs #2774 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey KolkovandClaude Opus 4.8
|
ae946c7143 |
fix(etcd-operator): make CRD vendoring fail loudly on fetch error
Review of #3403 noted the `curl … | awk > file` recipe swallowed curl failures: the pipe's exit status is awk's, and neither the Makefile nor hack/package.mk sets pipefail, so a 404 / network error / upstream path rename on a future bump would write an empty/truncated CRD template and still exit 0. Fetch to a temp file under `set -e` (curl -f already exits non-zero on HTTP error), then awk from it — shell-agnostic (no pipefail, which dash lacks). Verified: the regenerated templates are byte-identical to the committed ones, and a bad ref now exits non-zero leaving no template files instead of silently writing empties. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io> |
||
|
Aleksei Sviridkin
|
480c6d8a01 |
chore: ignore root-level Go binaries built from cmd/
`go build ./cmd/<name>` without -o drops an ~80 MB executable in the repo root under the package name. Ignore each cmd/ binary so a stray `git add` cannot commit one. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
d443a092ad |
feat(api)!: require the application release prefix to be dot-free
Constrain ApplicationDefinition spec.application.release.prefix to lowercase DNS-1123 characters (^[a-z0-9-]*$), which excludes the dot. Release names are "<prefix><app name>" and the tenant CA trust anchor is projected to "<release>.tenant-ca"; the dot is the separator that makes that name unreachable by any engine, so a prefix carrying a dot would break the guarantee. BREAKING CHANGE: an ApplicationDefinition whose release prefix contains a dot, or any character outside [a-z0-9-], is now rejected at admission. Existing definitions with such a prefix must be corrected before this CRD is applied. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
99f04d99bd |
test(postgres): verify the tenant CA projection end to end
Assert the live controller publishes <release>.tenant-ca carrying only ca.crt, that its certificate matches CloudNativePG's CA while the private key never crosses over, and that the anchor is reachable through the tenantsecrets API while the key-bearing source Secret is not. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
858c1c04fe |
feat(postgres): declare the tenant CA trust anchor
CloudNativePG stores its self-signed CA in <release>-ca alongside the private key and offers no way to label it, so a tenant cannot be handed the trust anchor without also being handed the key. Render a TenantProjection that lifts ca.crt alone into the key-free <release>.tenant-ca Secret, and select that label from the postgres tenant-resource definition so the projection is served through the tenantsecrets API. Document the new retrieval path in the chart README. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
37a91f98b8 |
feat(cozystack-controller): extract application CA into a key-free tenant Secret
Every catalog engine issues a per-release CA, but the object holding ca.crt usually holds a private key beside it (cert-manager's tls.key, CloudNativePG's ca.key), so granting a tenant read on it hands over key material too. Add a controller that publishes the trust anchor as one canonical, key-free Secret per release, "<release>.tenant-ca", holding only ca.crt. A chart declares the anchor by rendering a namespaced TenantProjection sentinel (internal.cozystack.io/v1alpha1) that names the source Secret and key. The controller rebuilds ca.crt from the parsed certificate DER rather than copying the source, refuses any value carrying private key material, and owns the projection by owner reference so a foreign Secret at the same name is never overwritten. No tenant RBAC grants any verb on internal.cozystack.io; a ValidatingAdmissionPolicy pins writes to helm-controller as defence in depth. The controller does not grant tenant read access — the lineage webhook does, from the ApplicationDefinition's spec.secrets. This side stamps the engine-agnostic internal.cozystack.io/tenant-ca label and forces a re-admission when those selectors change. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Ivan Okhotnikov
|
b2e1707632 |
fix(kubernetes-nodes): drop dead ephemeralStorage guard, clarify parity scope
The ephemeralStorage guard could never fire (the pool group is built from a fixed set of chart values that never includes ephemeralStorage), so it read as protection that is not there — remove it. Note in the parity script header that only the four pool objects are compared, not the talos-reconcile Job. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |
||
|
Ivan Okhotnikov
|
326dbe0a5b |
test(kubernetes-nodes): cover GPU and kubelet pool shapes in render parity
Extend the golden-parity matrix beyond the resources-sized pool to also diff a GPU pool and a kubelet-override pool against the parent render, so future drift in those branches (which would roll the content-hash-named KubevirtMachineTemplate) is caught in CI. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |
||
|
Ivan Okhotnikov
|
f21365c4f0 |
fix(kubernetes-nodes): address review nits in guard comment and NOTES
Reference the adoption migration by role (Phase 2b) instead of a number that does not exist on this branch, and correct the NOTES operational-note count. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |