admin/cozystack
1
0 Fork 0
mirror of https://github.com/cozystack/cozystack.git synced 2026-11-08 21:14:56 +09:00
Code Issues Packages Projects Releases Wiki Activity
5,116 Commits 569 Branches 294 Tags
8aeedde088e69411dce3aec6ba9a1f07b8151d5b
Commit Graph
5116 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Andrey Kolkov 8aeedde088 feat(backups): support PostgreSQL point-in-time recovery (PITR) (#3383) 
## What this PR does

Makes PostgreSQL point-in-time recovery (PITR) a genuinely supported,
tested capability for the new `backups.cozystack.io` API. Fixes #2774.

Was **stacked on #3313** (`fix/3300-cnpg-barman-plugin`) for the
`recoveryTime` → CNPG `bootstrap.recovery.recoveryTarget.targetTime`
plumbing; #3313 has since merged and this branch is rebased on `main`.
This PR adds the reliability + verification layer #2774 asked for.

### Changes

- **Precise failure reason for an unreachable target.** The restore
deadline (`spec.options.restoreTimeoutSeconds`, default 30m) stays the
sole authority for "this restore is stuck" — the client waits the full
window either way. What changes is the *diagnosis*: once the deadline
has elapsed, if `recoveryTime` was set and the recovery Cluster never
went healthy, the CNPG driver reads the `full-recovery` pod logs and —
when several distinct attempts exited with PostgreSQL's `FATAL: recovery
ended before configured recovery target was reached` — fails the
RestoreJob with `RecoveryConverged=False` / `Ready=False` (reason
`RecoveryTargetUnreachable`) and an actionable message, instead of a
bare generic timeout. Timing is unchanged; only the reason/message
differ. (For a faster rejection, set a shorter `restoreTimeoutSeconds`.)
Design points:
- **We deliberately do NOT trip early off the FATAL.** A valid near-now
target emits that same FATAL transiently — repeatedly, for many
attempts, on a slow archiver — before it converges, so any earlier trip
would wrongly reject a recoverable restore (observed twice in CI). The
FATAL is used only to *explain* a failure the deadline has already
declared.
- **Health wins unconditionally** — the deadline/classification check
precedes the healthy check, but a recovery that converges within the
window reaches success and is never classified as failed (a live dev7
restore did exactly this).
- **Counting distinct FATAL-ing attempts** (not pod failures) tolerates
a one-off archive-lag miss on a near-now target and ignores transient
recovery-pod crashes (a node blip / brief API-server unreachability
never emits that FATAL).
- Adds the `pods/log` RBAC grant the log read needs (distinct
subresource from `pods`); a `Forbidden` is logged loudly and surfaced in
the failure message rather than silently disabling the classification.
- **e2e** (`examples/backups/postgres/run-all.sh`, wired into the
Chainsaw `postgres-2-backup-roundtrip` suite):
- Step 45 — write `before`, capture server timestamp (µs precision),
write `after`, restore to that timestamp, assert `before` present and
`after` absent.
- Step 46 — a `recoveryTime` an hour ahead must fail with reason
`RecoveryTargetUnreachable` once the deadline elapses (asserts the
*reason*, so it proves the classification fired, not just that a timeout
occurred).
- Chainsaw budget raised 25m → 40m to cover three sequential restores +
the post-deadline classification.
- **Docs + example**: uncommented `45-restorejob-pitr.yaml`; a
`docs/operations/backup-classes.md` PITR section (recoverable window,
discovering earliest/latest restorable time from the backup catalog
since `firstRecoverabilityPoint` is unreliable under the plugin,
archive-lag tolerance, idempotency under GitOps).

### Verification

- `go build` / `go vet` clean; unit tests pass
(`logIndicatesRecoveryTargetUnreachable`, `recoveryUnreachableFromLogs`,
`recoveryPodsToInspect`, nil-Clientset guard, a reconcile-level
success-path test asserting `Succeeded` + `RecoveryConverged=True`, and
— added per review —
`TestReconcileCNPGRestore_DeadlineClassifiesRecoveryTargetUnreachable`,
a reconcile-level test driving the deadline branch through all three
sub-paths: FATAL found → reason `RecoveryTargetUnreachable`, FATAL
absent → generic timeout, `Forbidden` log read → generic timeout +
missing-`pods/log`-RBAC note. It fails if the call-site reason is
reverted to the generic one).
- Reviewed via adversarial branch-review; fixes include a critical
missing-RBAC dead-on-arrival, a classification false-positive on
transient recovery-pod crashes, and a sub-second e2e flake.
- **dev7**: backup + WAL archiving + recovery convergence verified live
on the #3313 build. The load-bearing assumption of the classification —
CNPG **accumulates** failed `full-recovery` pods rather than reusing one
— was confirmed (observed 7 simultaneous failed recovery pods). A
fully-green live step-46 run was blocked by intermittent COSI
bucket-provisioning flakiness on dev7; the negative path is covered by
the CI Chainsaw e2e.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff. The only user-facing surface is the new
`docs/operations/backup-classes.md` PITR section; the website keeps a
hand-mirrored copy per docs version.

- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: PITR docs mirror warranted (not yet opened; same pending
mirror as #3313's backup-classes changes)
- [x] No other downstream repository is affected by this change

```release-note
feat(backups): PostgreSQL RestoreJob now supports point-in-time recovery (PITR). Set spec.options.recoveryTime (RFC3339) to restore to an exact instant. A recoveryTime past the latest archived WAL still runs to the restore deadline (spec.options.restoreTimeoutSeconds, default 30m), but the failure is now reported with the precise reason RecoveryTargetUnreachable and an actionable message instead of a generic timeout; set a shorter restoreTimeoutSeconds for a faster rejection.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added PostgreSQL point-in-time recovery (PITR) support via RFC3339
`recoveryTime`.
* Introduced a PITR RestoreJob example, plus helper variables and an
unreachable-target scenario.
* **Bug Fixes**
* Improved restore deadline handling with deterministic, log-based
fail-fast classification for unreachable recovery targets, including
clearer conditions.
* **Documentation**
* Added detailed PITR operational documentation (recovery windows,
failure behavior, and GitOps/idempotent behavior).
* Expanded README and example comments for PITR verification and failure
expectations.
* **Tests**
* Extended unit tests and end-to-end chainsaw workflow to cover PITR
success and unreachable failures.
* **Chores**
* Updated RBAC to allow pod log reads; refined cleanup and run scripts
for additional RestoreJob artifacts.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-23 14:14:58 +04:00
myasnikovdaniil 18e459a1cb fix(e2e): improve failure diagnostics (#3421) 
## What this PR does

Improves E2E failure diagnostics tracked in #3368:

- preserves scheduled, non-terminal NotReady pods before a pod IP is
assigned
- always captures load-balancer state while gating IP-dependent
diagnostics
- collects Talos dmesg, kubelet, and containerd logs per host node using
explicit endpoint and node arguments
- renders COSI Bucket catch output as YAML
- makes pre-commit generation locking work from linked worktrees
- adds regression coverage for dataplane and Talos collection

Tenant-worker Talos logs remain outside this PR because the runner does
not currently have a tenant talosconfig.

Part of #3368.

## Testing

- `hack/cozytest.sh hack/capture-dataplane.bats`
- `hack/cozytest.sh hack/cozyreport-talos.bats`
- `sh -n hack/e2e-capture-dataplane.sh`
- `sh -n hack/cozyreport.sh`
- `chainsaw test --no-cluster --config hack/e2e-chainsaw/.chainsaw.yaml
--include-test-regex '^$' hack/e2e-chainsaw/bucket
hack/e2e-chainsaw/harbor`
- `pre-commit run --all-files`

### Screenshots

Not applicable.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(e2e): improve diagnostic capture for dataplane, Talos, and COSI failures
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Bug Fixes**
- Improved dataplane diagnostics for scheduled pods that are NotReady,
including cases without an assigned IP.
- LoadBalancer checks now proceed even when no affected pods are found.
- Standardized Talos node diagnostics collection for consistent kernel,
kubelet, and containerd logs.
- Improved reliability of generation hooks by using a git-common
lockfile path for concurrent runs.

- **Tests**
- Added/expanded Bats coverage for dataplane pod filtering and Talos
diagnostics argument correctness.
- Improved end-to-end bucket test retrieval/formatting for clearer
failure output.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-23 12:07:25 +05:00
IvanHunters f077420fbe fix(http-cache,vpa): fix haproxy 3.3 backend name conflict and vpa-for-vpa OOM (#2654) 
## What this PR does

Fixes two independent issues found during cluster operations:

**http-cache (HAProxy 3.3 breaking change)**

HAProxy 3.3 no longer allows a frontend and backend to share the same
name. The existing config had both `frontend http` and `backend http`,
which causes haproxy to refuse to start on upgrade. Rename `backend
http` → `backend cache` and update the `default_backend` directive
accordingly.

**vpa-for-vpa (OOMKilled recommender)**

The VPA recommender deployed by `vpa-for-vpa` was hitting its 512Mi
memory limit and being OOMKilled. Increase the limit to 1Gi to keep the
recommender running stably.

### Release note

```release-note
fix(http-cache): rename haproxy backend to avoid HAProxy 3.3 name conflict between frontend and backend
fix(vpa): increase vpa-for-vpa recommender memory limit from 512Mi to 1Gi to prevent OOMKilled restarts
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Corrected HAProxy routing so HTTP traffic is directed to the cache
backend.
* Increased the VPA recommender’s memory limit to improve reliability
under higher workloads.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-22 20:57:45 +03:00
Andrey Kolkov c503ae29d9 fix(backups): request S3 checksum only when required for barman-cloud (non-AWS S3 / Ceph RGW) (#3417) 
## What this PR does

Fixes CNPG **barman-cloud plugin** backups to non-AWS, S3-compatible
object
stores (Ceph RADOS Gateway, and some MinIO / Cloudflare R2 builds).

The plugin's sidecar uploads via **boto3**. Since botocore ~1.36 (early
2025)
the default `RequestChecksumCalculation` is `when_supported`, so a
flexible
checksum (CRC32) plus the corresponding `x-amz-content-sha256` handling
is
attached to every `PutObject`. AWS S3 accepts it, but several
S3-compatible
backends reject the request:

```
InvalidArgument: x-amz-content-sha256 must be UNSIGNED-PAYLOAD,
STREAMING-AWS4-HMAC-SHA256-PAYLOAD or a valid sha256 value.
```

so every backup / WAL-archive upload fails against them and the
`ScheduledBackup` never stores anything.

This sets `spec.instanceSidecarConfiguration.env` on the rendered
`ObjectStore`s to carry `AWS_REQUEST_CHECKSUM_CALCULATION=when_required`
(botocore reads this env var from the sidecar's environment).
`when_required`
computes a checksum only for operations that mandate one; a plain
`PutObject`
then carries none, which both AWS S3 and the affected backends accept —
a safe
default everywhere. No API/CRD change, no new value, no new dependency.

Applied to every barman-cloud `ObjectStore` Cozystack creates:
- **chart-rendered** — `packages/system/keycloak/templates/db.yaml`
(Keycloak
system DB) and `packages/apps/postgres/templates/db.yaml` (postgres app
—
backup **and** bootstrap-recovery `ObjectStore`s), via a shared
`cozy-lib`
  helper (`cozy-lib.barman.checksumSidecarConfiguration`);
- **Go-constructed** — the platform-managed `useSystemBucket=true`
`ObjectStore`
  SSA-applied by `internal/backupcontroller/cnpgstrategy_controller.go`
(`barmanSidecarConfiguration`), whose default system bucket is
SeaweedFS.

Regression coverage: a Go unit test on the SSA-applied `ObjectStore`,
plus
helm-unittest assertions on `spec.instanceSidecarConfiguration.env` for
all
three rendered `ObjectStore`s (mutation-checked — reverting the change
fails
them).

Mirrors the same fix already applied to the etcd path in
`cozystack/etcd-operator#342` (released in `v0.5.3`) for the same Ceph
RGW
backend.

**Verification:** reproduced end-to-end against a real Ceph RGW
endpoint. With
default botocore, `PutObject` fails with the `InvalidArgument` above;
with
`AWS_REQUEST_CHECKSUM_CALCULATION=when_required` in the environment it
succeeds.
Deployed the plugin + a throwaway CNPG cluster whose `ObjectStore`
carried this
env: continuous WAL archiving reported `ContinuousArchiving=True` and an
on-demand `Backup` completed, with the base backup and WAL segments
present in
the bucket. `helm template` renders the env on all three `ObjectStore`s;
the
`packages/apps/postgres` unit-test suite still passes.

### Screenshots

N/A — no UI changes.

### Downstream repositories

<!-- Walked the trigger map in docs/agents/contributing.md against the
diff,
file by file. The diff touches only two Helm templates (ObjectStore
renders);
it adds no package, no values.yaml / values.schema.json field, no
API/CRD
change, no variant/component/asset change, no ApplicationDefinition
change, and
no developer-tooling change. None of the trigger-map entries match. -->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(backups): CNPG barman-cloud backups now work against non-AWS S3-compatible object stores (Ceph RADOS Gateway, some MinIO/Cloudflare R2). The backup sidecar requests an S3 request checksum only when required, avoiding the "x-amz-content-sha256 must be UNSIGNED-PAYLOAD, ..." error that previously broke every backup and WAL-archive upload to those backends. No change for AWS S3.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved compatibility with S3-compatible, non-AWS storage backends
for PostgreSQL and Keycloak backups and recovery.
* Pin the barman-cloud sidecar to compute AWS request checksums only
when required (`AWS_REQUEST_CHECKSUM_CALCULATION=when_required`) to
prevent failures with strict S3 gateways.
* **Tests**
* Updated/added assertions in controller and Helm/YAML suites to verify
the checksum pin is applied to generated and recovery ObjectStores
(Postgres and Keycloak).

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-22 21:02:05 +04:00
myasnikovdaniil a2acfcde1a docs(release): add changelog for v1.6.0 (#3429) 
## What this PR does

Adds `docs/changelogs/v1.6.0.md` (667 lines), generated locally with the
process in `docs/agents/changelog.md`. Normally `tags.yaml`'s
`generate-changelog` job produces this on the stable tag push, but that
job never ran for `v1.6.0` — see the note below.

**Coverage boundary.** The changelog runs through #3386, i.e. the
release period ending at the `v1.6.0-rc.4` cut. Roughly eight PRs merged
to `main` after that cut are deliberately absent, because `v1.6.0` is a
promotion rather than a rebuild: every shipped artifact is pinned to an
rc.4 digest (`cozystack-packages@sha256:bf682087…`,
`cozystack-operator@sha256:93b305f7…`), so that later code is in the
tagged tree but not in anything users install. The changelog describes
what shipped.

**Why this is a manual PR.** `generate-changelog` and
`update-website-docs` are jobs in `tags.yaml`, triggered by `push:
tags`. The `v1.6.0` tag was pushed by `pull-requests-release.yaml`'s
finalize job using the `cozystack-ci` app token, and no workflow run was
created for that ref — every rc tag, pushed by `cut-prerelease.yaml`
with the same app, did trigger it. `v1.6.0` is the first stable tag
created through the promote flow, so this path had never run before. The
cause is not yet established and needs a repro rather than a guess; the
website docs versioning is still outstanding for the same reason.

### Screenshots

N/A — documentation only.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff, which is a single new file under `docs/changelogs/`. No trigger
matches: the map covers package layout, release assets, CRDs, workflow
behaviour and naming contracts, none of which a changelog file touches.
The separate repository list in `docs/agents/changelog.md` was already
consumed while generating this file — it feeds the notes, it is not a
code coupling.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
  * Added comprehensive release notes for version 1.6.0.
* Documented breaking changes, upgrade prerequisites, manual upgrade
steps, and post-upgrade expectations.
* Highlighted major features, deprecations, security updates, bug fixes,
operational notes, and component version updates.
* Included configuration, CI, development, and related-project changelog
updates.
  * Added contributor acknowledgements and a link to the full changelog.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-22 21:12:34 +05:00
myasnikovdaniil 764e30418d chore(release): promote v1.6.0-rc.4 -> v1.6.0 (#3425) 
Promotes `v1.6.0-rc.4` to stable `v1.6.0` — no rebuild. The tree pins
the rc's images by digest. On merge, pull-requests-release.yaml creates
the write-once `v1.6.0` tag at the merge commit, retags those digests to
`v1.6.0` (+`:latest` when this is the newest stable), publishes the
stable cozy-installer chart, and publishes the release — so the stable
artifacts are bit-for-bit the rc that passed e2e. The `full-e2e` label
forces the full suite on this PR. Do NOT squash-merge (decision B): the
stable tag must attach to a real merge commit.
v1.6.0 api/apps/v1alpha1/v1.6.0 		2026-07-22 20:20:09 +05:00
myasnikovdaniil d3815a4ce6 Merge branch 'main' into release-1.6.0 
Signed-off-by: myasnikovdaniil <60174387+myasnikovdaniil@users.noreply.github.com>
 2026-07-22 18:12:04 +05:00
cozystack-ci[bot] 235d38dae1 Prepare release v1.6.0 (promoted from v1.6.0-rc.4) 
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
 2026-07-22 13:00:10 +00:00
myasnikovdaniil 706c3f06fc fix(release): reuse only an OPEN promote PR, not one in any state (#3424) 
## What this PR does

The promote step guarded PR creation with `gh pr view <branch>`, which
resolves a pull request by head branch **in any state**. A closed one
satisfied the guard, so creation was skipped while the step still exited
0 — a promotion that reports success and leaves nothing to merge. Same
silent-skip shape as the enumeration bug fixed in #3404: the run is
green, the artifact is missing, and nothing says so.

`STABLE_BRANCH` is derived from the stable version
(`release-${version}`), not from the rc number, so every rc promoted to
that version shares it. Once any promote PR for a version has been
closed, both re-dispatch paths this workflow documents as supported are
wedged: re-dispatching the same rc, and promoting a newer rc to the same
version. That second one is named in the workflow's own header as the
reason a leftover draft is tolerated — the tolerance was implemented for
the release draft and missed for the PR.

Neither path recovers by reopening the old PR, because the step's own
`git checkout -B` plus force-push makes that PR's head unreachable, and
GitHub then refuses:

```
422 Validation Failed
  state cannot be changed. The release-1.6.0 branch was force-pushed or recreated.
```

The fix asks the question the guard meant to ask — is there an **open**
PR for this head — and creates one otherwise.

**Hit live.** Promoting `v1.6.0-rc.4` after #3397 had been closed: [run
29917852639](https://github.com/cozystack/cozystack/actions/runs/29917852639)
went green, logged `PR already open for release-1.6.0`, and opened no
PR. This currently blocks the v1.6.0 release.

**Verified in both directions** against live data, so the change is not
vacuous:

| head branch | PR state | old guard | new guard |
| --- | --- | --- | --- |
| `release-1.6.0` | #3397 closed | `TRUE` → skip (bug) | `FALSE` →
create ✅ |
| `chore/gitignore` | #3412 open | `TRUE` → skip | `TRUE` → skip ✅ |

`actionlint` exits 0 and `zizmor` reports no findings. The guard reads
`${STABLE_BRANCH}` as a shell variable from the step's `env:` block
rather than as a `${{ }}` expansion inside `run:`, so no
expression-injection surface is added.

### Screenshots

N/A — no UI change.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff, which is one file: `.github/workflows/promote-rc.yaml`. The only
workflow-related trigger in the map is cozystack/ccp on "change
release-prep behaviour in `.github/workflows/tags.yaml`", which this
does not touch. This change also restores the documented behaviour of
`promote-rc.yaml` rather than altering its contract, so nothing
downstream sees a result different from what the docs already promise.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(release): promoting a release candidate now opens the promotion pull request even when an earlier promotion attempt for the same version was abandoned. Previously the workflow mistook a closed pull request for an open one, skipped creating a new one, and reported success with nothing left to merge.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved release PR creation checks so closed or previously merged
pull requests no longer prevent new release PRs from being opened.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-22 17:36:30 +05:00
Myasnikov Daniil 6a41d65f16 test(e2e): avoid ripgrep dependency in diagnostics tests 
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-22 17:27:02 +05:00
Myasnikov Daniil ef2fa628f5 fix(ci): make pre-commit locking work in worktrees 
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-22 17:27:02 +05:00
Myasnikov Daniil 4e0cfd994d test(e2e): render COSI catch output as YAML 
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-22 17:27:02 +05:00
Myasnikov Daniil ad1adbf785 test(e2e): fix diagnostic collector blind spots 
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-22 17:27:02 +05:00
myasnikovdaniil 55195ded0c test(e2e): capture previous container logs and stop killing the snapshot (#3346) 
## What this PR does

Root-causing a recent E2E failure was nearly blocked because the
diagnostics captured on failure had collected almost nothing. Two
independent mechanisms were responsible, and both had been silently in
place for a long time.

**The failing container's previous logs were never captured.** The pod
had crash-looped, so the decisive evidence lived in the first container
instance, reachable only through `kubectl logs --previous`. Chainsaw's
`podLogs` action has no `previous` field, so this could not be expressed
per-suite; the conclusion had to be reconstructed from indirect evidence
instead. A new capture runs from the global catch and from the BATS
failure hook, selecting containers at runtime by `restartCount > 0`
across both init and regular containers, so a healthy run stays quiet
and future suites are covered without per-suite wiring.

**The catch was structurally guaranteed to be killed.** A Chainsaw
`script` op is bounded by `timeouts.exec`, which is 5m, while the
catch's own inner budgets summed to roughly 960s. It reported `SCRIPT |
ERROR | signal: killed` and produced nothing — and its second leg, the
data-plane capture, had therefore never run at all. The kill signature
was itself misleading: `timeout` exits 124 and cannot produce that
message, which comes from Go's `exec.Cmd` killing the process group.
Each catch op now carries an explicit timeout above the budgets it
contains, so the inner one fires and leaves a partial, usable artifact
instead of losing everything.

**The snapshot was truncated at 60 seconds.** `crust-gather --duration`
defaults to 60s and was never passed, so every snapshot ever taken was
cut short, with nothing in the artifact saying so. The API discovery it
performs first is not covered by that budget at all, which is where the
outer wall-clock was going on a cluster with a degraded aggregated
APIService — precisely the state a failed run is in. Both budgets are
now set explicitly, and the collector's output goes to a log inside the
snapshot with a completeness verdict, so "is this snapshot complete?" is
answerable from the artifact.

Failure paths are reported rather than swallowed: an unreachable API
says so instead of implying nothing had restarted, a truncated capture
is kept and marked, and every dropped item is logged, since silent
truncation reads as full coverage.

### What is not verified here

The mechanism claims are read from the pinned versions of Chainsaw and
crust-gather. Everything cluster-facing is not verified by this PR: the
go-template's output shape against a live API server, `--previous`
returning the first instance for a real crash-looper, and whether the
new collection budgets suffice on a degraded cluster. The bats suite
covers the selection helpers only, and those cases are mutation-tested.

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Diagnostics**
* Improved end-to-end failure evidence by capturing “previous instance”
logs for crash-looping pods (including restarted init containers) and
prioritizing relevant namespaces.
* Added time-bounded diagnostics collection with explicit outer
timeouts; collector output is saved to per-snapshot log files.
* Classifies collector outcomes as complete, timeout-truncated, or
failed, while preserving the original test failure signal.
* **Documentation**
* Expanded E2E failure-diagnostics guidance with new timeout and
evidence-capture conventions.
* **Tests**
* Added unit-style tests for previous-log selection, prioritization, row
limiting, and deterministic log artifact naming.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-22 17:22:11 +05:00
Myasnikov Daniil baf0158d81 fix(release): reuse only an OPEN promote PR, not one in any state 
The promote step guarded PR creation with `gh pr view <branch>`, which
resolves a pull request by head branch in any state. A closed one
satisfied the guard, so creation was skipped while the step still exited
0 — a promotion that reports success and leaves nothing to merge, the
same silent-skip shape as the enumeration bug fixed in #3404.

STABLE_BRANCH is derived from the stable version, so every rc promoted to
that version shares it. Both re-dispatch paths this workflow documents as
supported are therefore wedged once any promote PR for that version has
been closed: re-dispatching the same rc, and promoting a newer rc to the
same version. Neither recovers by reopening the old PR, because the
step's own force-push makes its head unreachable and GitHub refuses with
"state cannot be changed. The <branch> branch was force-pushed or
recreated."

Hit live while promoting v1.6.0-rc.4 after #3397 had been closed: the run
went green and opened no PR.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-22 17:18:00 +05:00
Timofei Larkin d5806eb1a0 feat(kubernetes-nodes): split worker node pool into its own app (Phase 2, additive) (#3314) 
## What this PR does

Phase 2 of the Kubernetes-app split
([cozystack/community#8](https://github.com/cozystack/community/pull/8))
— the **additive** half.

Introduces `kubernetes-nodes`: a per-pool application that renders the
worker node-pool objects (`KubevirtMachineTemplate`,
`MachineDeployment`, `MachineHealthCheck`, worker `WorkloadMonitor`, and
the `talos-reconcile` Job + RBAC) extracted from the monolithic
`kubernetes` chart. A `KubernetesNodes` CR named `<cluster>-<pool>`
attaches to a parent `Kubernetes` cluster in the same namespace by name
convention (`kubernetes-<cluster>`), mirroring the
`vm-instance`/`vm-disk` precedent.

This PR is additive: the parent `kubernetes` chart gets exactly one
scoping fix (its old-KMT preservation block is limited to KMTs owned by
its own Helm release) so that attaching a `kubernetes-nodes` pool to a
live cluster cannot break the parent's next upgrade. Existing clusters
keep provisioning workers via `nodeGroups`. Removing `nodeGroups` from
the parent and adopting existing pools into `kubernetes-nodes` releases
is a follow-up **breaking** PR (Phase 2b).

Key properties:
- Pool objects render **byte-identical** to the parent for the same
group (golden-parity check in `tests/render-parity.sh`), so the
follow-up adoption never churns live worker VMs — the content-hashed
`KubevirtMachineTemplate` name is preserved.
- cluster-autoscaler (kept in the parent chart) discovers the pool's
`MachineDeployment`s cross-release via the unchanged
`cluster.x-k8s.io/cluster-name` label, namespace, and min/max
annotations.
- Registered as a dedicated `cozystack.kubernetes-nodes-application`
PackageSource + `kubernetes-nodes-rd` ApplicationDefinition, wired into
the iaas bundle (mirrors `vm-instance`/`vm-disk`).

### Downstream repositories

Left unticked deliberately — this is a draft. A new user-facing
`KubernetesNodes` API is added, so cozystack/website (docs) and
cozystack/terraform-provider-cozystack likely need follow-ups; the
trigger-map walk will be completed before this leaves draft.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:

```release-note
Add the `kubernetes-nodes` application: manage the worker node pools of a managed Kubernetes cluster as independent resources, one HelmRelease per pool.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added a new Kubernetes worker node pool offering with configurable
sizing (CPU/memory/instance, disk), roles, storage class, GPUs, and
Talos worker image settings.
* Introduced a dedicated Helm package, dashboard/resource-map access,
and Flux application wiring for installing/managing pools.
* **Bug Fixes**
* Improved safeguards against pool naming/ownership conflicts and
tightened upgrade behavior for reusing previously created templates.
* **Documentation**
* Added full node pool configuration docs and operational notes
(including supported Kubernetes versions and reconciliation
expectations).
* **Tests**
* Added node pool rendering/validation tests plus golden parity checks
between chart layouts.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-22 15:57:43 +04:00
Andrey KolkovandClaude Opus 4.8 80a03b1e09 test(backups): assert the barman-cloud sidecar checksum env on rendered ObjectStores 
helm-unittest coverage for the S3 request-checksum fix (was mutation-negative
before): assert spec.instanceSidecarConfiguration.env pins
AWS_REQUEST_CHECKSUM_CALCULATION=when_required on all rendered barman-cloud
ObjectStores — postgres backup + bootstrap-recovery
(packages/apps/postgres/tests/backup_storage_test.yaml) and the Keycloak DB
(new packages/system/keycloak/tests/db_backup_test.yaml). Verified by mutation:
reverting the template change now fails these assertions.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
 2026-07-22 15:50:25 +04:00
Myasnikov Daniil 284a17ba1f fix(release): unblock the v1.6.0 promote on the rc.4 staging tree 
Promoting this branch fails in two independent ways, both landing after
finalize has already created the write-once stable tag, published the
GitHub release and moved :latest.

chBackupClientImage had no producer and froze at v1.4.0-rc.2 while
.migrations.image advanced to v1.6.0-rc.4, leaving platform-migrations
pinned at two digests. promote-retag.sh copies every collected
<repo>@<digest> to <repo>:<stable>, so both compete for one destination
tag: the lower-sorting v1.4.0-rc.2 digest wins it and the second copy
trips the write-once guard, aborting the retag with five repositories
still untagged and the installer chart unpublished. Align it to the
digest this branch already carries for the migrations image. Both come
from the same tree, so the runner keeps the curl, jq and ca-certificates
the Altinity strategy Pod needs, and that Pod overrides the ENTRYPOINT
with its own command.

promote-rc.yaml on main now calls hack/promote-rewrite-tags.sh, which
does not exist here, so the dispatch would fail outright. Vendor it and
hack/lib/image-refs.sh verbatim from main (identical blobs). That also
fixes the rewrite's coverage: the old inline glob enumerated 155 files
and missed 10 rc-bearing ones, which is why the earlier promote left 11
references reading v1.6.0-rc.4. The shared enumeration covers all 168
and fails the promotion if any rc reference survives.

Both fixes are on main via #3405 and #3404; this carries them onto the
rc.4 staging tree so v1.6.0 ships the bytes rc.4 tested.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-22 16:48:32 +05:00
myasnikovdaniil d0c0f399f8 fix(ci): cover every image-ref storage shape in promote, retag and mirror (#3404) 
## The bug

Promotion rewrites the rc version substring to the stable version across
the vendored image references. That rewrite lived inline in
`promote-rc.yaml` and globbed the depth-2 package `values.yaml` plus
`packages/apps/kubernetes/images/*.tag` **only**, on the premise that
the kubernetes app was the only one whose `.tag` files carry the
cozystack version. Nine other `.tag` files do, and `system/multus` seds
its reference straight into a vendored upstream daemonset manifest.

#3397 (promote `v1.6.0-rc.4` → `v1.6.0`) is therefore staged with **33
of 55 first-party references still reading `v1.6.0-rc.4`**.

`hack/promote-retag.sh` and `hack/nightly-mirror.sh` had the same blind
spot from the other direction — both scanned the depth-2 `values.yaml`
alone, so those images were never retagged to the stable version (**30
references selected before, 42 after**) and never mirrored to the public
registry for a nightly. A retag miss leaves an image short of a tag,
since the digest still resolves inside one registry. A mirror miss is
worse: the host rewrite walks the same file list, so a reference that is
neither mirrored nor rewritten leaves the published tree pointing at the
private build registry.

`v1.6.0` is the first release to go through promotion — every earlier
release was a full rebuild from the tag, which stamped every file
uniformly. The glob has always been wrong; the old path just made it
unobservable.

## Why it could not be caught before

The rewrite was workflow-inline, so there was nothing to unit test. The
only way to observe the miss was to cut a release.

It is a script now — `hack/promote-rewrite-tags.sh` — and
`hack/promote-rewrite-tags_test.bats` round-trips it against the real
tree under `make unit-tests`, with no cluster, registry or release
involved.

## The fix

The three call sites each carried their own idea of where a reference
can live, which is how they drifted apart. That knowledge moves into
`hack/lib/image-refs.sh` as the single enumeration — `image_ref_files()`
for the files, `collect_image_refs()` for the references inside them —
and all three source it, so a new storage location is declared once and
reaches every consumer.

The rewrite also asserts its own postcondition: it scans **wider** than
it rewrites and fails the promotion when a reference survives in a
location the enumeration does not know about, rather than shipping it
quietly. The wide scan is shape-filtered to image-reference position,
because `X.Y.Z-rc.N` is an ordinary version string that other things
legitimately contain — `kubevirt-csi-driver/go.sum` pins
`github.com/golang/protobuf v1.4.0-rc.2`, and `v1.4.0-rc.2` is a
cozystack rc that was actually cut. Narrowing it to `${RC}@sha256:`
instead would have been wrong: `system/cilium/values.yaml` keeps `tag:`
and `digest:` under separate keys.

`docs/agents/image-refs.md` records the contract — three tag classes,
three storage shapes, the invariants, and which consumer does what.

## Not a bug: kamaji

`cluster-api-control-plane-provider-kamaji` stays at
`v0.19.0-cozystack.0` across the promotion. It is a first-party rebuild
of an upstream component, versioned by that component rather than by the
cozystack version line, so no rewrite is owed. A test pins this so a
future fix here cannot over-reach.

## Documented known gaps, deliberately not fixed here

Both are pre-existing and in the **host** dimension; the version
dimension this PR owns is complete, because the version always lives in
a `tag` key regardless of where the host sits.

- **The nightly host rewrite cannot reach a split host.** It is a
literal `<src-registry>/` substring replace, so `keycloak-operator`
(host in a sibling `registry:` key) and `kubeovn`
(`global.registry.address`, no trailing slash) are mirrored but keep
pointing at the build registry in the published tree. Fixing it needs
structure-aware rewriting.
- **A reference inside a gzip is invisible.**
`capi-providers-cpprovider` ships `files/components.gz`, and the promote
postcondition cannot catch this class at all since `grep -rIl` skips
binaries. Nothing is owed today — kamaji is component-versioned and the
image is still mirrored and retagged via its `.tag` — so the residual is
that a nightly's kamaji ConfigMap keeps the build-registry host.

## Testing

- 17 tests in the new suite; `make bats-unit-tests` green (312 tests,
exit 0); `shellcheck -x` clean; `zizmor` clean on the workflow.
- Verified against four real rc versions (`1.4.0-rc.2`, `1.4.0-rc.4`,
`1.5.0-rc.2`, `1.6.0-rc.4`) in both directions — no false positives, no
false negatives.
- Every test is mutation-proven. Reverting the enumeration to the old
glob, deleting any of the five YAML shapes, corrupting emitted digests,
reverting extras to parse-only, dropping the unreadable-file guard,
broadening shape 4 beyond `global.images`, or planting an unenumerated
first-party reference in any storage shape each fail a named test.

## Reviewer note, out of scope

`platform-migrations` is pinned at two different digests —
`core/platform/values.yaml` at `v1.5.0` and
`backupstrategy-controller/values.yaml` at `v1.4.0-rc.2` (an unowned
pin; no Makefile writes that key, tracked in #3143). A
`promote-retag.sh` dry-run emits **two copies to the same destination
tag**, so the write-once guard will fail the promotion. This is
independent of this PR and needs a decision about the backup-client
image version, so it is left alone here — but it blocks the 1.6.0
promote and wants fixing before the re-cut.

## Release note

```release-note
Promotion, retagging and nightly mirroring now cover every image reference the build stamps, including those stored in `images/*.tag` files and in vendored manifests. Previously a promoted release could ship images still tagged with the release-candidate version, and those images received no stable tag in the registry.
```

Fixes the promote half of #3143.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Image promotion and nightly mirroring now detect digest-pinned image
references across values files, standalone tag files, and supported
templates.
- Stable release preparation supports RC→stable tag rewriting via a
dedicated promotion script, with broader verification to ensure no RC
references remain.

- **Bug Fixes**
- Improved completeness of ref discovery and scanning, including
previously missed template/tag locations.

- **Documentation**
- Added and linked guidance documenting image-reference invariants and
how tooling enumerates supported storage shapes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-22 16:41:17 +05:00
myasnikovdaniil bbfe9f9a9d fix(backupstrategy-controller): give chBackupClientImage a producer (#3405) 
## This is a workaround

`backupstrategy-controller` should not be running the platform
**migrations** image. This PR does not fix that — it makes the existing
reuse *safe* so the v1.6.0 promote can proceed, and documents in the
tree exactly what is wrong and what should replace it. Treat it as a
stopgap with a TODO, not as an endorsement of the design.

## What is actually wrong

The Altinity strategy Pod drives clickhouse-backup's HTTP API. Its
script uses exactly four binaries — `curl`, `jq`, `sleep`, `date`. For
that it currently pulls `platform-migrations`:

| | |
| --- | --- |
| Size | **349 MB** (a dedicated `alpine + curl + jq + ca-certificates`
measures **23 MB**) |
| Also contains | `kubectl`, `helm`, `git`, `cozyhr`, the `etcd-migrate`
binary, every migration script, vendored etcd CRDs |
| `ENTRYPOINT` | `run-migrations.sh` |

Two consequences worth naming. A tenant-adjacent backup Pod carries
cluster tooling it has no use for. And because the image's own
entrypoint is the migration runner, the `command:` override in the
strategy template is load-bearing — drop that one line and the Pod runs
platform migrations with `kubectl` and `helm` already on the PATH.

It also creates the coupling that caused the bug this PR fixes: bumping
`ETCD_OPERATOR_VERSION` or editing any migration script re-digests the
image, forcing the backup client's pin to move for reasons that have
nothing to do with backups.

## The release blocker

`platform-migrations` was pinned at two different digests in the
committed tree:

| Location | Pin | Producer |
| --- | --- | --- |
| `packages/core/platform/values.yaml` `.migrations.image` |
`v1.5.0@sha256:8bf61f17…` | `packages/core/platform/Makefile` |
| `backupstrategy-controller/values.yaml` `.chBackupClientImage` |
`v1.4.0-rc.2@sha256:17390197…` | **none** |

Promotion retags **by digest**: `hack/promote-retag.sh` copies every
collected `<repo>@<digest>` to `<repo>:<stable-version>`. Two digests
under one repository produce two copies competing for the same
destination tag — the first wins, the second hits the write-once guard,
and the promotion fails. A dry run against `main` emits exactly that
pair:

```
▸ ghcr.io/cozystack/cozystack/platform-migrations  sha256:17390197…
  → docker://ghcr.io/cozystack/cozystack/platform-migrations:v9.9.9
▸ ghcr.io/cozystack/cozystack/platform-migrations  sha256:8bf61f17…
  → docker://ghcr.io/cozystack/cozystack/platform-migrations:v9.9.9
```

`v1.6.0` is the first release cut through promotion rather than a full
rebuild. Every earlier release ran `make build`, which restamped both
copies as a side effect and hid the missing producer.

## What this PR does

`packages/core/platform/Makefile` now stamps both keys with the same
ref, so they move in lockstep by construction rather than by
instruction, and the committed value is aligned to the digest
`core/platform` already carried. Moving the runner forward two minors is
safe: it is used only as a curl + jq container, and the image built at
`v1.5.0` carries them (`git show v1.5.0:…/migrations/Dockerfile` → `apk
add … jq ca-certificates bash curl`).

## What should replace it

A dedicated `ch-backup-client` image built by **this package's own
Makefile** — measured at 23 MB, 15× smaller, with none of the cluster
tooling and none of the entrypoint hazard. That also deletes the
cross-package stamp this PR adds, because the pin would be produced by
the package that owns it.

The objection that has kept this reuse in place — "a second tag release
CI would have to re-digest each cut" — does not survive contact with the
actual defect. Every other first-party image already pays that cost,
automatically. What broke here was not an extra tag; it was **a pin with
no producer**, which a package-owned image cannot reproduce.

I am deliberately not doing that here: it means a new image, a new build
target and a new digest introduced during a release cut. It belongs in a
follow-up.

## The guard

`hack/image-pin-consistency.bats` asserts the general invariant — no
repository pinned at more than one digest — driven through the real
promotion selector so it tracks whatever set promotion actually acts on,
rather than a reimplementation that could drift from it. Plus a specific
lockstep assertion so a recurrence names its cause. This check could not
have been added before now: it fails on the very duplicate this PR
removes.

## Testing

- `hack/image-pin-consistency.bats` — both tests pass; verified they
fail when the drifted pin is restored.
- `make bats-unit-tests` green (294 tests, exit 0).
- `make -C packages/system/backupstrategy-controller test` — 11 tests, 3
suites, green.
- `helm template … --set backupStorage.bucketNameOverride=test-bucket`
renders the strategy Pod with the new ref.
- The Makefile's `yq` step was simulated directly to confirm the
relative path and key path resolve and that comments survive the
in-place edit.
- Merged locally with #3404 onto `main` and re-run: the expanded ref
collector in that PR surfaces no additional duplicate pins (314 tests,
exit 0).

## Relationship to other work

Relates to #3143 — this is one instance of its unowned-pin class, the
one its discussion flagged as having no producer. The four never-built
packages there remain untouched.

Independent of #3404 (promote/retag/mirror storage-shape coverage); the
two touch no common files and can merge in either order. Both are needed
before the `v1.6.0` promote is re-cut.

## Release note

```release-note
Fixed a release-blocking duplicate pin: the ClickHouse backup client image is now stamped from the same source as the platform migrations image it reuses, so both carry one digest. Previously they could drift, which made promotion attempt to retag two digests to the same stable tag and fail. Reusing the migrations image here remains a documented workaround pending a dedicated backup-client image.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Ensured the platform migrations image digest is pinned consistently
between the platform and backup strategy components.
* Updated the backup strategy component to reference the new pinned
`platform-migrations` digest.

* **Tests**
* Added promotion/retag consistency checks to fail on conflicting source
digests for the same stable tag.
* Added drift detection to ensure `platform-migrations` image references
match across consumer configuration files.

* **Documentation**
* Expanded in-chart comments and values documentation clarifying the
coupling and workaround behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-22 16:41:13 +05:00
Andrey KolkovandClaude Opus 4.8 b574e2176f fix(backups): cover Go-driven system-bucket ObjectStore + share the checksum helper 
Address review on the S3 request-checksum fix:

- Extend the fix to the platform-managed useSystemBucket=true flow, whose
  ObjectStore is built in Go by applyClusterPluginBackup (SSA-applied), not
  chart-rendered. Add InstanceSidecarConfiguration to cnpgtypes.ObjectStoreSpec
  and set AWS_REQUEST_CHECKSUM_CALCULATION=when_required on it via
  barmanSidecarConfiguration(). The platform's own default system bucket is
  SeaweedFS, a non-AWS S3 gateway of the same class as Ceph RGW. Unit test in
  cnpgstrategy_controller_test.go asserts the applied ObjectStore carries it.

- Factor the duplicated instanceSidecarConfiguration block (3x across two
  charts) into a shared cozy-lib helper (cozy-lib.barman.checksumSidecarConfiguration).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
 2026-07-22 15:35:32 +04:00
Andrey Kolkov ad3f8b2487 fix(backups): check cluster health before the restore deadline (PITR) 
Addresses review CHANGES_REQUESTED on #3383.

CRITICAL: reconcileCNPGRestore checked the restore deadline BEFORE the
cluster-health check, so a reconcile that first observes "deadline elapsed"
AFTER the recovery Cluster already reached a healthy state (a controller
restart or stalled workqueue spanning restoreTimeoutSeconds) would mark an
already-converged restore Failed / RecoveryTargetUnreachable. A false Failed
can then escalate to data loss: a resubmitted RestoreJob's purge guard treats
the recovered Cluster as a stale leftover and deletes it + its PVCs.

Fix: check hasRecovery + cnpgClusterHealthy first and return Succeeded
unconditionally when healthy, regardless of how late the reconcile runs; only
a not-yet-healthy restore reaches the deadline/classification branch. Adds
TestReconcileCNPGRestore_HealthyPastDeadlineSucceeds (StartedAt 72h past the
deadline + healthy Cluster + recoveryTime set; the injected readPodLog fails
the test if the classification path is entered, proving health short-circuits).

MINOR: handle the deferred Close error on the pod-log reader (errcheck).
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-07-22 15:31:31 +04:00
Andrey KolkovandClaude Opus 4.8 0c586b9d72 fix(backups): request S3 checksum only when required for barman-cloud 
The barman-cloud plugin's sidecar uploads via boto3. Since botocore ~1.36
(early 2025) the default RequestChecksumCalculation is when_supported, which
attaches a flexible checksum to every PutObject. Non-AWS S3-compatible
backends (Ceph RADOS Gateway, some MinIO / Cloudflare R2 builds) reject the
accompanying header with "InvalidArgument: x-amz-content-sha256 must be
UNSIGNED-PAYLOAD, STREAMING-AWS4-HMAC-SHA256-PAYLOAD or a valid sha256
value", so every backup/WAL-archive upload fails against them.

Pin the sidecar to when_required via the ObjectStore's
spec.instanceSidecarConfiguration.env (botocore honors the
AWS_REQUEST_CHECKSUM_CALCULATION env var) on both the keycloak system DB and
the postgres app (backup + recovery ObjectStores). when_required is also
accepted by AWS S3 on a plain PutObject, so it is a safe default everywhere.

This mirrors the etcd-operator fix (cozystack/etcd-operator#342, released in
v0.5.3), which applied the same WhenRequired policy to its aws-sdk-go-v2 S3
client for the same Ceph RGW backend.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
 2026-07-22 15:29:20 +04:00
Andrey KolkovandClaude Opus 4.8 b5d03f889b test(backups): reconcile-level test for the PITR deadline classification 
Review of #3383 found the RecoveryTargetUnreachable classification branch in
reconcileCNPGRestore was only covered through the pure helpers and
recoveryTargetUnreachable directly, never through reconcileCNPGRestore itself —
a mutation swapping the call-site reason back to the generic RestoreFailed kept
the suite green.

Add TestReconcileCNPGRestore_DeadlineClassifiesRecoveryTargetUnreachable: it
backdates StartedAt past the restore deadline with a recoveryTime set and drives
the real reconcile through the readPodLog seam, asserting all three deadline
sub-paths — FATAL present (reason RecoveryTargetUnreachable on both
RecoveryConverged and Ready), FATAL absent (generic timeout, no
RecoveryTargetUnreachable), and a Forbidden log read (generic timeout + the
missing-pods/log-RBAC note). Verified it fails when the call-site reason is
reverted to the generic RestoreFailed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
 2026-07-22 13:06:00 +04:00
Myasnikov Daniil 676c8b888a docs: add changelog for v1.6.0 
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-22 14:01:14 +05:00
Andrey Kolkov 8ea589230f test(mariadb): add MariaDb backup/restore e2e test (#3384) 
## What

Reworks the never-merged
[`backup-mariadb-e2e`](https://github.com/cozystack/cozystack/tree/backup-mariadb-e2e)
branch onto the modern Chainsaw e2e approach
([`docs/agents/e2e-testing.md`](https://github.com/cozystack/cozystack/blob/main/docs/agents/e2e-testing.md)),
rebuilt on fresh `main`.

That branch bundled the whole MariaDB backup-strategy controller
**plus** a `hack/e2e-apps/backup-mariadb.bats` test. The controller and
the `examples/backups/mariadb/` manifests have since landed on `main`
independently (`feat(platform): implement backup strategy for mariadb`),
and the `hack/e2e-apps/*.bats` app suite has been replaced by
`hack/e2e-chainsaw/`. So the only piece still missing was the e2e
coverage — this PR delivers it the modern way.

## Changes

- **`examples/backups/mariadb/` → executable harness.** Converts the
demo from static numbered YAML into the same script harness the other
backup demos use (`00-helpers.sh`, `run-all.sh`, `cleanup.sh`), keeping
the numbered manifests as the human-readable flow that `run-all.sh`
drives — so the documented flow and the automated test can never drift.
`run-all.sh` provisions a `Bucket`, derives the two per-app Secrets the
strategy references from `BucketInfo` + the seaweedfs CA, deploys a
source MariaDB with a sentinel row, fires an ad-hoc `BackupJob`,
restores to a copy with a to-copy `RestoreJob`, and asserts the sentinel
round-tripped through S3 into the copy while the source is left
untouched.
- **`hack/e2e-chainsaw/mariadb/` → second Chainsaw Test**
(`mariadb-2-backup-roundtrip`) that drives `run-all.sh` + `cleanup.sh`,
following the merged `etcd-2-backup-roundtrip` harness-driving shape:
runs in `tenant-root` (the isolated e2e tenant cannot reach
`tenant-root`'s seaweedfs across the Cilium egress policy), `skipDelete`
since the harness owns the resources, an idempotent pre-clean step so a
run killed before `finally` cannot leave a stale `Succeeded` BackupJob
that false-passes, fail-fast on terminal `BackupJob`/`RestoreJob`
phases, and scoped `catch` diagnostics. Cleanup also prunes the
operator-side `k8s.mariadb.com` `Backup`/`Restore` CRs the Cozystack
jobs own only by label.
- **`hack/select-e2e.sh` → TIA rule for `examples/backups/<app>/`.** The
harness lives outside the suite dir, so without this a harness-only diff
produced an empty selection, which `pull-requests.yaml` turns into "skip
the whole E2E stage" — a future bug-fix to `run-all.sh` would merge with
its own round-trip test never run. The new rule maps
`examples/backups/<app>/` to the `<app>` suite that drives it (dir name
== suite name for every backup demo except `vmi` → `vminstance`), with
three cases added to `hack/select-e2e_test.bats`.

## Conventions followed

- No retries on deterministic steps; the single `script` step carries a
justified 30m budget (2 MariaDB instances × 310s first-boot startup
probe + dump/restore) and fails fast rather than polling out.
- Self-contained trap-based temp-file cleanup only inside a single step;
no test-level EXIT trap.
- Controller-created artifacts Chainsaw cannot reclaim are pruned
explicitly by `cleanup.sh`.
- TIA verified: `examples/backups/mariadb/run-all.sh` now selects the
`mariadb` suite (`hack/select-e2e.sh`), and editing `select-e2e.sh`
itself escalates this PR to the full suite.

## Downstream Repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff. The change touches only e2e test material, backup examples, and CI
suite-selection — no package added/renamed/removed, no
`values.schema.json` / `values.yaml` / platform-values / variant /
`ApplicationDefinition` / namespace / release-asset changes, and no
`hack/` file moved/renamed. No downstream repository is reached.

- [x] `cozystack/website` — not affected (no package or
platform/docs-source change)
- [x] `cozystack/terraform-provider-cozystack` — not affected (no
app/schema/enum/default/prefix change)
- [x] `cozystack/ansible-cozystack` — not affected (no installer values
/ namespace / variant / node-prereq change)
- [x] `cozystack/ccp` — not affected (no `hack/` move-or-rename, no
make-target behavior change)

## Notes

No `make generate` needed — the change touches only `examples/`,
`hack/`, and a Chainsaw suite (the edited README is under `examples/`,
not a package).

## Release note

```release-note
Add an end-to-end Chainsaw test for the MariaDB backup/restore round-trip, and convert the `examples/backups/mariadb/` demo into an executable `run-all.sh`/`cleanup.sh` harness.
```

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added an end-to-end MariaDB backup and restore demonstration workflow.
* Added automated backup round-trip validation, including data integrity
checks.
* Added reusable scripts for running the demo and cleaning up created
resources.
* Added automatic handling of backup credentials, endpoints, and trusted
certificates.

* **Documentation**
* Expanded setup, configuration, execution, cleanup, and troubleshooting
guidance.
* Documented same-namespace restore support and cross-tenant
limitations.

* **Tests**
* Added automated Chainsaw coverage for the MariaDB backup and restore
workflow.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-22 12:54:27 +04:00
Andrey Kolkov efa4e41e9c feat(etcd-operator): bump to v0.5.3 (S3 checksum fix for non-AWS backends) (#3403) 
## What this PR does

Bumps the etcd-operator to **v0.5.3**, which carries the S3
request-checksum fix
([cozystack/etcd-operator#342](https://github.com/cozystack/etcd-operator/pull/342)).
`git compare v0.5.2...v0.5.3` is exactly #340, #341, #342.

**Why:** scheduled etcd snapshots to non-AWS S3-compatible backends
(**Ceph RGW** confirmed) fail at upload. Since early 2025
`aws-sdk-go-v2` defaults `RequestChecksumCalculation` to
`WhenSupported`, which stamps a CRC32 on every upload; over HTTPS a
multipart part rides it as `x-amz-content-sha256:
STREAMING-UNSIGNED-PAYLOAD-TRAILER`, and RGW rejects it with `400
InvalidArgument`. v0.5.3 sets `WhenRequired` on both the S3 client and
the transfer manager, so multipart uploads (>5 MiB — every real etcd
snapshot) carry no checksum trailer.

**Changes:**
- `system/etcd-operator` — `appVersion` v0.5.2 → v0.5.3 (image tag
follows AppVersion; `values.yaml` keeps `tag: ""`), bump
`ETCD_OPERATOR_REF`, update deployment unittest expectations.
- `system/etcd-operator-crds` — re-vendor CRDs from v0.5.3.
`etcdmembers` gains the additive #340 `status.version` field +
**Running** printer column and picks up the `peerAutoTLS` spec field the
vendored copy was already missing (all additive, backward-compatible;
`etcdclusters`/`etcdsnapshots` unchanged). Fixes the stale `make update`
(upstream has no `config/crd` kustomization — CRDs live in
`charts/etcd-operator/crd-bases`) and makes it fail loudly on a fetch
error (temp file under `set -e`, not a pipe).

**Verification:** `helm unittest` 18/18 and `helm template` green on
both packages; regenerated CRDs are byte-identical to upstream
`crd-bases` at v0.5.3. End-to-end on a live Ceph RGW backend
(freedom-portal-stage): the v0.5.2 (client-only) agent fails a multipart
snapshot upload with the 400; v0.5.3 uploads an 11 MiB multipart
snapshot successfully, and the operator-driven CronJob→EtcdSnapshot path
reaches `Complete`.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` file-by-file
against the diff (`packages/system/etcd-operator{,-crds}/**` only — a
component version bump + additive CRD re-vendor). Nothing matches: not
an `apps/`/`extra/` package add/rename/remove, no `core/platform`
values, no variant/bundle, no platform component add/remove, no `hack/`
layout or shared-tooling change, and the provider does not type the
`etcd-operator.cozystack.io` CRDs (and the change is additive/opaque
regardless).

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(etcd-operator): bump to v0.5.3 — etcd snapshot uploads to non-AWS S3-compatible backends (Ceph RGW, some MinIO/R2) no longer fail with `400 InvalidArgument: x-amz-content-sha256 ...`; the snapshot agent now requests a checksum only when required, on both the S3 client and the multipart transfer manager. Also brings observed EtcdMember versions (#340) and `--watch-namespace` (#341).
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added runtime etcd version visibility to `EtcdMember` (including a
“Running” status column).
* Added `peerAutoTLS` support to `EtcdMember` for operator-managed peer
TLS behavior.

* **Updates**
  * Updated the etcd-operator to version 0.5.3.
* Refreshed bundled CRDs and ensured CRDs are preserved during Helm
lifecycle operations.

* **Tests**
* Updated deployment test expectations to use the 0.5.3 operator image.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-22 12:53:59 +04:00
Myasnikov Daniil a302bd0ddc docs(backupstrategy-controller): reframe the migrations-image reuse as a workaround 
Running the platform migrations image as the ClickHouse backup client is
not a design decision worth defending, and the in-tree comments read as
though it were. Reframe them.

The strategy script uses exactly curl, jq, sleep and date. For that it
pulls a 349 MB image carrying kubectl, helm, git, cozyhr, the
etcd-migrate binary and every migration script into a tenant-adjacent
Pod, and that image's ENTRYPOINT is run-migrations.sh -- so the
`command:` override in the strategy template is load-bearing rather than
incidental. A dedicated alpine + curl + jq + ca-certificates image
measures 23 MB.

The standing objection to a dedicated image -- "a second tag release CI
would have to re-digest each cut" -- does not survive contact with the
actual defect. Every other first-party image already pays that cost
automatically. What broke was a pin with NO producer, which an image
owned by its own package cannot reproduce; that fix would also delete
the cross-package stamp introduced here.

Comment-only. No rendered output changes.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-22 12:29:15 +05:00
Myasnikov Daniil 1f302d0d78 docs(agents): correct issue ref, mirror claim and example in image-refs 
The build-list violations bullet pointed at #3143, which tracks the
adjacent gap (packages in the list going stale between releases) and
never covers packages absent from the list; they are now tracked in
the newly filed #3416. The nightly split-host paragraph claimed both
affected images are mirrored while the published tree points at the
build registry — today neither ref carries the source-registry host
(keycloak-operator is never rebuilt by CI, kubeovn is built externally
by cozystack/kubeovn-chart), so the ownership filter skips them and
the gap is latent, not live; reworded to say exactly that. The opening
example paired a v1.6.0 tag with the real v1.5.0 grafana digest — a
ref that will never exist — and grafana is slated to stop being built;
replaced with the literal committed cozystack-api ref.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-22 12:23:29 +05:00
Andrey KolkovandClaude Opus 4.8 4605bbf3db test(mariadb): pre-clean the backup roundtrip against stale tenant-root state 
Address review feedback on #3384. The TIA rule for
examples/backups/<app>/ that an earlier revision of this commit added
has since landed on main independently (50fb3c87c, "run the postgres
backup round-trip e2e in CI under TIA"), so only the suite change
remains here after the rebase.

mariadb-2-backup-roundtrip runs in the shared, persistent tenant-root
namespace with fixed resource names, so a run killed before the finally
leaves stale state behind; a leftover BackupJob still carrying
status.phase=Succeeded would satisfy the Succeeded wait without a real
backup ever running. Add an idempotent pre-clean step (run cleanup.sh
before run-all.sh), mirroring the etcd suite, and reword the suite
comment to reference the merged etcd-2-backup-roundtrip pattern instead
of postgres-2-backup-roundtrip.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-07-22 10:45:32 +04:00
Andrey KolkovandClaude Opus 4.8 2829f7a300 docs(mariadb): address review notes on backup example 
- 00-bucket.yaml: correct the Secret-name comment to include the
  "bucket-" release prefix (bucket-<bucket>-<user>), matching the
  README and 00-helpers.sh.
- README: note that S3_CA_NAMESPACE defaults to tenant-root
  independently of NAMESPACE, so a non-root tenant run must override
  it (and S3_CA_SECRET) too.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-07-22 10:44:42 +04:00
Andrey KolkovandClaude Opus 4.8 d1f2822018 test(mariadb): add backup/restore round-trip e2e (chainsaw) via example harness 
Reworks the never-merged backup-mariadb-e2e branch onto the modern
Chainsaw approach (docs/agents/e2e-testing.md). The bats port that
branch carried (hack/e2e-apps/backup-mariadb.bats) is dropped — the
whole hack/e2e-apps/*.bats app suite has been replaced by
hack/e2e-chainsaw/, and the mariadb strategy controller + example
manifests it depended on have since landed on main independently. Only
the e2e coverage was still missing.

Convert examples/backups/mariadb/ from static numbered YAML into the
same executable harness the other backup demos use (00-helpers.sh,
run-all.sh, cleanup.sh), keeping the numbered manifests as the
human-readable flow that run-all.sh drives — so the documented flow and
the automated test can never drift. run-all.sh provisions a Bucket,
derives the two per-app Secrets the strategy references from BucketInfo
+ the seaweedfs CA, deploys a source MariaDB with a sentinel row, fires
an ad-hoc BackupJob, restores to a copy with a to-copy RestoreJob, and
asserts the sentinel round-tripped through S3 into the copy while the
source is left untouched.

Add a second Chainsaw Test (mariadb-2-backup-roundtrip) to
hack/e2e-chainsaw/mariadb/ that drives run-all.sh + cleanup.sh, mirroring
postgres-2-backup-roundtrip: runs in tenant-root (the isolated e2e tenant
cannot reach tenant-root's seaweedfs across the Cilium egress policy),
skipDelete since the harness (not Chainsaw) owns the resources, fail-fast
on terminal BackupJob/RestoreJob phases, and scoped catch diagnostics.
Cleanup also prunes the operator-side k8s.mariadb.com Backup/Restore CRs
the Cozystack jobs own only by label.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-07-22 10:44:42 +04:00
Andrey KolkovandClaude Opus 4.8 6a1e2d0d06 feat(backups): support PostgreSQL point-in-time recovery (PITR) 
Makes RestoreJob.spec.options.recoveryTime a genuinely supported, tested
capability for the new backups.cozystack.io API (issue #2774). The
recoveryTime -> CNPG bootstrap.recovery.recoveryTarget.targetTime plumbing
already existed on the barman-cloud-plugin branch; this adds the reliability
and verification layer the issue asked for:

- Fail-fast for an unreachable target: when a RestoreJob has recoveryTime set
  and the recovery Cluster is not healthy, the driver reads the CNPG
  full-recovery pod logs and, once several distinct attempts have exited with
  PostgreSQL's "recovery ended before configured recovery target was reached"
  FATAL, fails the RestoreJob with conditions RecoveryConverged=False /
  Ready=False (reason RecoveryTargetUnreachable) instead of hanging to the
  restore deadline. Health is checked first, so a converged recovery always
  wins; counting distinct FATAL-ing attempts (not pod failures) tolerates a
  transient archive-lag miss on a near-now target and ignores transient
  recovery-pod crashes. Adds the pods/log RBAC grant the log read requires.

- e2e (examples/backups/postgres/run-all.sh, wired into the Chainsaw
  postgres-2-backup-roundtrip suite): step 45 writes a before marker, captures
  the server timestamp, writes an after marker, restores to that timestamp and
  asserts the before row survived and the after row did not; step 46 submits a
  recoveryTime an hour ahead and asserts it fails fast with reason
  RecoveryTargetUnreachable.

- Working, uncommented example (45-restorejob-pitr.yaml) and a docs section
  (docs/operations/backup-classes.md) covering the recoverable window, how to
  discover the earliest/latest restorable time from the backup catalog, the
  archive-lag tolerance, and idempotency under GitOps.

Refs #2774

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-07-22 10:40:18 +04:00
Andrey KolkovandClaude Opus 4.8 ae946c7143 fix(etcd-operator): make CRD vendoring fail loudly on fetch error 
Review of #3403 noted the `curl … | awk > file` recipe swallowed curl failures:
the pipe's exit status is awk's, and neither the Makefile nor hack/package.mk sets
pipefail, so a 404 / network error / upstream path rename on a future bump would
write an empty/truncated CRD template and still exit 0.

Fetch to a temp file under `set -e` (curl -f already exits non-zero on HTTP error),
then awk from it — shell-agnostic (no pipefail, which dash lacks). Verified: the
regenerated templates are byte-identical to the committed ones, and a bad ref now
exits non-zero leaving no template files instead of silently writing empties.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
 2026-07-22 10:37:20 +04:00
Ivan Okhotnikov b2e1707632 fix(kubernetes-nodes): drop dead ephemeralStorage guard, clarify parity scope 
The ephemeralStorage guard could never fire (the pool group is built from a
fixed set of chart values that never includes ephemeralStorage), so it read
as protection that is not there — remove it. Note in the parity script header
that only the four pool objects are compared, not the talos-reconcile Job.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:35:11 +03:00
Ivan Okhotnikov 326dbe0a5b test(kubernetes-nodes): cover GPU and kubelet pool shapes in render parity 
Extend the golden-parity matrix beyond the resources-sized pool to also
diff a GPU pool and a kubelet-override pool against the parent render, so
future drift in those branches (which would roll the content-hash-named
KubevirtMachineTemplate) is caught in CI.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:35:11 +03:00
Ivan Okhotnikov f21365c4f0 fix(kubernetes-nodes): address review nits in guard comment and NOTES 
Reference the adoption migration by role (Phase 2b) instead of a number
that does not exist on this branch, and correct the NOTES operational-note
count.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:35:11 +03:00
Ivan Okhotnikov 9e2f7cc953 test(kubernetes-nodes): run render parity in make test 
Wire tests/render-parity.sh into the chart's test target so CI verifies
the pool objects stay byte-identical to the parent render.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:35:10 +03:00
Ivan Okhotnikov 3d10a1c431 fix(kubernetes-nodes): keep applied TalosConfigTemplate byte-identical to the parent 
Restore the inline machineconfig comments (nameservers, GPU node labels)
inside the reconcile Job's data block so the applied TalosConfigTemplate
matches the parent's, avoiding drift on adoption.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:35:10 +03:00
Ivan Okhotnikov 73be7032a3 fix(kubernetes-nodes): drop unbacked cluster option source 
The cluster field referenced x-cozystack-options source 'kubernetes',
which has no provider (cozystack-api returns NotFound), so the dashboard
dropdown 404s. Drop the annotation so the field is a plain text input;
a real provider listing Kubernetes CRs is a follow-up.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:35:10 +03:00
Ivan Okhotnikov fe32fd8e58 fix(kubernetes-nodes): scope KMT preservation by release and guard foreign pool names 
Replace the collision-prone name-prefix match (pool 'gpu' matched
sibling 'gpu-large-<hash>') with a release-name annotation scope, and
add a render-time guard that fails cleanly when a pool's MachineDeployment
already exists under a different release (e.g. naming a pool after a
parent nodeGroup such as the default md0). Document the naming caveat in
NOTES.txt.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:35:10 +03:00
Ivan Okhotnikov 1e8d44995c fix(kubernetes): scope worker KMT preservation to the release 
The preserve-old-KubevirtMachineTemplate block re-emitted any KMT owned
by the CAPI Cluster and referenced by a MachineSet. A kubernetes-nodes
pool's KMT lives in the same namespace under the same Cluster, so the
parent would re-emit it with a foreign meta.helm.sh/release-name and its
next upgrade would fail Helm ownership validation. Restrict the block to
KMTs whose release-name annotation matches the parent release.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:35:10 +03:00
Ivan Okhotnikov e7947dd0e7 feat(platform): register kubernetes-nodes application 
Add the kubernetes-nodes-rd package (ApplicationDefinition), the
cozystack.kubernetes-nodes-application PackageSource (kubevirt variant,
kubernetes-nodes + kubernetes-nodes-rd components), and wire it into the
iaas bundle. Mirrors the vm-instance/vm-disk registration precedent.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:35:10 +03:00
Ivan Okhotnikov 277b32306c test(kubernetes-nodes): add unit tests and golden-parity check 
Unit tests cover naming/selectors, clusterName wiring, autoscaler
annotations, omitted replicas, bootstrap TCT reference, KMT hash name,
worker WorkloadMonitor, and the empty-cluster / mis-prefixed-release
render guards. render-parity.sh diffs the pool objects against the
parent kubernetes chart and asserts byte-identity.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:35:10 +03:00
Ivan Okhotnikov f842c0a17e feat(kubernetes-nodes): generate API types for KubernetesNodes 
Generated by cozyvalues-gen + controller-gen from values.yaml.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:35:10 +03:00
Ivan Okhotnikov fd14126bea feat(kubernetes-nodes): add worker node pool chart 
Introduce the kubernetes-nodes application chart: one HelmRelease per
worker node pool, attached to a parent Kubernetes cluster by name
convention (kubernetes-<cluster>). Renders the KubevirtMachineTemplate,
MachineDeployment, MachineHealthCheck, worker WorkloadMonitor and the
talos-reconcile Job with its RBAC, extracted from the monolithic
kubernetes chart. Pool objects are byte-identical to the parent's render
for the same group, so a later adoption never churns live worker VMs.

This is the additive half of the Phase 2 split (community#8); the parent
kubernetes chart is left unchanged.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:35:10 +03:00
Myasnikov Daniil 427ed68761 fix(backupstrategy-controller): give chBackupClientImage a producer 
platform-migrations was pinned at two different digests: v1.5.0 in
packages/core/platform/values.yaml, stamped by its Makefile, and
v1.4.0-rc.2 as .backupStrategyController.chBackupClientImage, stamped by
nothing at all. The second is the Altinity strategy Pod's curl+jq runner,
deliberately reusing the migrations image rather than shipping a second
one-binary tag that release CI would have to re-digest every cut. That
reuse only holds while the two move together, and the values.yaml comment
said so -- but a comment was the only thing enforcing it, so the copy with
no producer froze two minors back.

This is a release blocker, not just staleness. Promotion retags by digest,
copying every collected <repo>@<digest> to <repo>:<stable>. Two digests
under one repository produce two copies competing for one destination tag:
the first wins and the second hits the write-once guard in
hack/promote-retag.sh, which fails the promotion. A dry run against the
tree emitted exactly that pair. v1.6.0 is the first release cut through
promotion rather than a full rebuild, so it would have been the first to
hit it -- every earlier release ran `make build`, which restamped both
copies as a side effect and hid the missing producer.

packages/core/platform/Makefile now stamps both keys with the same ref, so
they are in lockstep by construction rather than by instruction, and the
committed value is aligned to the digest core/platform already carried.
The image is used only as a curl + jq + ca-certificates runner, so moving
it forward two minors changes no behaviour the strategy depends on; it
renders and the chart's helm-unittest suite passes.

hack/image-pin-consistency.bats adds the general guard -- no repository
pinned at more than one digest, driven through the real promotion selector
so it tracks whatever set promotion acts on -- plus a specific lockstep
assertion so a recurrence names its cause. The general check could not be
added before now: it fails on the very duplicate this commit removes.

Relates to #3143 (the unowned-pin class) and unblocks the v1.6.0 promote.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-21 23:15:46 +05:00
Myasnikov Daniil 91c076137f test(ci): pin shape 3's per-map digest binding 
The shape-3 fixture held a single repository/digest pair, so a rule that
binds one digest per file and reuses it for every repository satisfied
it -- and passed all three suites. That is not a theoretical mutation:
against the real linstor values it hands piraeus-server's digest to
linstor-csi, which retags and mirrors the wrong image under the right
name. Silently shipping wrong bytes is worse than the missing-tag bug
this branch started from.

The fixture now carries two shape-3 maps with distinct digests in one
file and asserts both exact canonical refs, so the digest must be bound
per map rather than per file.

Assisted-By: Claude <noreply@anthropic.com>
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-21 22:54:19 +05:00
Myasnikov Daniil a16f218875 test(ci): correct a stale comment and isolate the fail-open fixtures 
Two review nits, both comment- or fixture-level.

The completeness oracle's comment still said the per-shape test "matches
on the repository substring, so a shape that loses its registry rejoin
still satisfies it". That stopped being true in the same commit that
introduced it -- the substring match was replaced with an exact
canonical-ref assertion, which kills both the shape-2 and shape-3 rejoin
mutations. The comment understated coverage.

The fail-open test copied the tree once and looped three rc versions over
it, so the second and third iterations ran against a tree the first
rewrite had already mutated. Each now gets a fresh copy, so each tests
the input it names.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-21 22:51:00 +05:00
Myasnikov Daniil 539a7b8e0b test(ci): catch shape-4 stapling inside global, and spaced marker keys 
Round-5 review. The shape-4 negative fixture placed its decoy outside
`global` entirely, so it only caught a document-wide broadening. Mutating
the rule to `.global | ..` instead passed the per-shape test and all three
suites while still stapling the document-level registry onto unrelated
maps inside `global` -- manufacturing first-party refs that promotion
would then retag and mirror to destinations nothing ever built. The
fixture now carries two decoys at different depths, one inside `global`
but outside `global.images` and one outside `global`, and the mutation is
caught.

Markers also required the key to be immediately followed by its colon, so
`registry : ghcr.io/...` was missed. No producer writes that spelling and
it is valid YAML rather than anything in the tree, so this is theoretical
-- but tolerating whitespace costs one character class and keeps the
comment's exhaustiveness claim true lexically as well as structurally.
Marker count is unchanged at 43 files with the same two allowlist entries.

Assisted-By: Claude <noreply@anthropic.com>
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-21 22:26:38 +05:00