Commit Graph
92 Commits
Author SHA1 Message Date
Andrey Kolkov 01967b56d3 docs(backups): correct the Kafka .Mode doc and the embedded-script templating note
The KafkaSpec.Template doc listed .Mode as "backup" or "restore", but the
driver also renders it as "cleanup" when reaping a removed Backup's object;
the comment is copied verbatim into the CRD, so kubectl explain and the
dashboard showed two values while a custom strategy's restore branch would
run on the cleanup pass. List all three, matching the Redis sibling, and
regenerate the CRD.

The strategy template also claimed the embedded script "reads only env
vars, never Helm/controller values, so no tpl is applied". The controller
renders every string leaf of the PodTemplateSpec through internal/template,
args[0] included, so the script is templated on every run. State that
instead, without literal template braces that Helm would itself expand.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-14 17:55:48 +04:00
Andrey Kolkov 02bf3e9970 fix(backups): probe live Kafka topics without a pipe and isolate the scratch file
The restore existence check piped the live topic list into `grep -qxF`.
grep exits at the first match, so once the matched topic is far enough from
the end of a list larger than the 64 KiB pipe buffer, the producing printf
takes SIGPIPE, pipefail turns the pipeline into a failure, the `if` reads
false, and a topic that exists is sent to --create, which fails with
TopicExistsException. Read the list from a here-string instead: no pipe, no
SIGPIPE, decided by content rather than list size.

Also make the local metadata scratch file overridable (METADATA_FILE),
like BIN already is, so the bats suite can point each test at its own path
instead of sharing /tmp/kafka-metadata.txt across runs. The Job never sets
it and keeps the default.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-14 17:55:48 +04:00
Andrey Kolkov 17825d4a32 fix(backups): declare the shell dialect in the Kafka backup script
The script opens with `set -euo pipefail` and no shebang, so shellcheck
reports SC2148 and cannot tell which dialect to lint against. Declare it
with a `# shellcheck shell=bash` directive as the first line: shellcheck
and pre-commit now lint it as bash, and the line is an inert comment once
the script is embedded into the Job's pod argument.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:23:53 +04:00
Andrey Kolkov 2cb2b459fa fix(backups): reject a non-numeric recorded partition count on Kafka restore
The restore path validated only the shape parsed from the live cluster,
not the partition count and replication factor read from the stored
object. A non-numeric recorded partition count made `[ "${lp}" -lt
"${parts}" ]` and its `elif` exit 2; an `if` condition is exempt from
errexit, so both read as false, the topic was skipped, and the loop still
printed its per-topic success line - so the RestoreJob reached Succeeded
without ever touching the topic. A well-formed but wrong-content object,
which curl -f cannot catch, must fail loudly like the unrecognised-record
guard.

Validate the recorded parts and rf are numeric before the comparisons and
--create, with a bats case that reddens when the guard is dropped.

Also correct a stale to-copy example comment: the strategy reads the exact
object the source backup recorded in status.artifact.uri (injected as
ARTIFACT_URI), not a key derived from .Backup.ApplicationRef.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:23:53 +04:00
Andrey Kolkov 4925a84c26 fix(backups): pin the restore-side guards and bound a running Kafka Job
- delete_topic waited with `grep -qx` (BRE), so deleting `audit.events`
  matched the live `audit-events` and spun to its timeout — the e2e
  failed at step 25 on every run that reached it. Use `grep -qxF`.
- The two -k/--connect-to guards were written as `! grep`, which cannot
  fail under hack/cozytest.sh (it wraps each @test in `set -e` + a
  trailing `return 0`, and bash exempts a `!`-negated command from
  errexit). Rewrite them as positive `if grep ...; then return 1; fi`;
  verified the -k reinstatement now reddens the suite under cozytest.
- Cover the restore-side literal-topic guard that was unpinned at every
  level: a bats case runs the existing-topic branch with a colliding
  sibling and asserts the `--alter` argv carries `\Q…\E`, and run-all.sh
  now leaves `audit.events` live so the e2e exercises that branch instead
  of only `--create`. Add bats cases for the partition-shrink refusal, the
  `-f` upload guard, and `--exclude-internal`.
- Bound a running Job: reconcileKafka / reconcileKafkaRestore now fail the
  BackupJob/RestoreJob once kafkaDefaultBackupDeadline elapses in the
  Job-running branch (buildJobStrategyBatchJob sets no
  activeDeadlineSeconds), so an unschedulable pod no longer pins it in
  Running forever. Pinned by TestReconcileKafka_FailsWhenJobNeverCompletes.
- Fail closed on an unknown record kind and an unknown MODE, and validate
  the live partition/RF parsed on restore (an empty value silently skipped
  reconciliation). Correct the stale "image pinned in the strategy" comment
  in the example helpers and the chainsaw test (the controller now resolves
  the target broker's image as .ClientImage).

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:23:52 +04:00
Andrey Kolkov 0d918e1f91 fix(backups): resolve Kafka backup client image from the target broker
The Kafka metadata strategy pinned kafkaBackupClientImage to an upstream
Strimzi tag with no producer: nothing stamped it and Renovate does not
manage helm-values, so it drifted from the broker version the operator
provisions. A dedicated small client image (the sibling curl-only drivers'
follow-up) is not viable here because the driver runs the Kafka Admin-API
CLI, which only the broker image ships.

Resolve the client image from a live broker pod of the target Strimzi Kafka
at reconcile time instead, so the CLI always matches the broker (any version
the operator picks, not only the chart default), and record it on the Backup
so cleanup can reuse it once the source cluster is gone. The chart no longer
pins the image; the strategy template renders it as the .ClientImage
placeholder the controller fills.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:23:52 +04:00
Andrey Kolkov ebb3118694 fix(backups): key Kafka --connect-to off the request host in both S3 modes
The port workaround built --connect-to's HOST1 from ${host}, but the
virtual-hosted branch (forcePathStyle:false) sends to ${bucket}.${host}.
curl matches HOST1 against the request host, so on a virtual-hosted AND
non-default-port endpoint the redirect never fired and the request hit the
wrong port. Derive HOST1 from the host the URL actually uses; the real
connection target stays ${host}:${port}. Path-style is unaffected. Add a
bats case for the virtual-hosted ported mode, which the suite did not cover.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:22:32 +04:00
Andrey Kolkov 14c6af122e fix(backups): restore Kafka S3 port workaround and hold the script with a test
- Regression fix: an earlier change dropped the curl --connect-to port workaround along
  with -k, but curl 7.76.1 (the image's) 403s on ANY ported endpoint,
  which is the shape backupStorage.endpoint ships (:8333) and what
  provisionBucket:false and s3.ingress.enabled:false produce. Restore
  --connect-to (sign a portless URL, redirect to the real port), keeping
  -k off so TLS is still verified. IPv6-safe host/port split, path prefix
  preserved. The port-less ingress passed; the ported case is the one that regressed.
- Correct the S3 envelope claim in values.yaml and backup-classes.md: the
  403 is decided by the port, not a body-hash the backend recomputes, and
  --connect-to handles it on SeaweedFS and MinIO alike.
- Hold the script fixes at the layer the contract lives at. Externalise
  the driver to files/kafka-backup.sh (embedded verbatim via .Files.Get,
  render byte-identical) and add hack/kafka-backup-script.bats, which
  EXECUTES it against stubbed kafka CLIs + curl and asserts behaviour: \Q
  literal match on the colliding pair, --connect-to with no -k on a ported
  endpoint, fail-closed --list, comma-value bracket grouping, RF-divergence
  abort, and the cleanup delete's HTTP handling. Not a template grep.
- Pin the Backup-delete dispatch and the readiness deadline in Go:
  TestCleanupOnDelete_Kafka_RoutesToKafkaCleanup and
  TestReconcileKafka_FailsAfterReadinessDeadline.
- Docs: config restore is additive (no --delete-config); export time
  scales with topic count and the Job has no wall clock (as its siblings).

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:22:32 +04:00
Andrey Kolkov ed3cb34cbf fix(backups): adopt-guard Kafka Backup artifact and correct URI-template doc
- createKafkaBackupArtifact now rejects adopting a same-named Backup that
  describes a different application, so a retained Backup reused for a
  different app of the same name is not reported as this run's success.
  Mirrors the RabbitMQ driver's guard; pinned by a Go test.
- ArtifactURITemplate godoc said empty leaves status.artifact unset, but
  the driver hard-fails an empty template. Reword to state it is required,
  and regenerate the CRD description to match.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:22:07 +04:00
Andrey Kolkov 0270650237 fix(backups): harden Kafka metadata S3 handling and own the artifact
- Drop `-k`: mirror the RabbitMQ S3 layer (ARTIFACT_URI as the single
  source of truth, forcePathStyle honoured, whole base URL kept) so
  curl --aws-sigv4 verifies TLS against the ACME endpoint the helper
  returns. The derived endpoint is default-port, so the curl 7.76.1
  --connect-to port workaround is gone too; this also fixes the IPv6
  and path-prefix mangling in the old host-splitting.
- Own the artifact: a cleanup mode in the script plus cleanupKafkaBackup
  wired into BackupReconciler.cleanupOnDelete delete a deleted Backup's
  object (best-effort, waited on) instead of orphaning it, matching the
  RabbitMQ precedent. Kafka has the identical own-your-object property.
- Restore probes existence with --list (fail closed), so an unreachable
  broker aborts instead of being misread as "topic absent" and created.
- Hold the fixes with tests at the right level: a colliding topic pair
  (audit.events / audit-events, distinct partition counts) in the e2e so
  a revert of the \Q...\E literal-match breaks the round-trip, and Go
  tests pinning the scope stamp and the cleanup Job.
- Correct the docs/comments: endpoint-shape row, the "cached on nodes"
  claim, and the narrower curl 7.76.1 S3 envelope (empty-body payload
  hash) that makes a small modern-curl client image the right follow-up.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:22:06 +04:00
Andrey Kolkov 537aadf389 fix(backups): fix Kafka backup S3 signing and literal topic names
Reproduced end-to-end against a real Kafka 3.9.1 broker:

- [CRITICAL] The Strimzi image ships curl 7.76.1, whose --aws-sigv4
  omits a non-default port from the SigV4 canonical Host while sending it
  in the Host header, so every S3 PUT/GET 403s and the driver never
  uploads or downloads. Sign/send the bare host and redirect the
  connection to the real port with --connect-to (the same fix the
  topic-data example already carries), and accept the in-cluster
  self-signed cert with -k.
- [CRITICAL] kafka-topics --topic is a Java regex, not a literal: a
  backup entry for "audit.events" also matched a live "audit-events", so
  restore repartitioned an unrelated topic irreversibly and backup
  recorded the wrong partition/replication values. Wrap the name in
  \Q...\E on every filter call (--describe on both paths, --alter on
  restore) so it matches exactly one topic; --create and
  kafka-configs --entity-name are already literal. Applied in the example
  helpers too.
- [MAJOR] cozy-default's Kafka route looked like a data backup at
  runtime. Stamp scope=topic-metadata on the produced Backup's
  DriverMetadata and say "topic definitions and configs only; no message
  data" in the Ready condition, so kubectl describe backup is honest.
- [MAJOR] Add reconcile tests for the two Go behaviours a mutation could
  silently drop: the Strimzi-Ready precondition on backup (BackupJob
  stays out of Succeeded and creates no Job) and the applicationRef Kind
  gate on the restore target.
- [MINOR] Fail closed on the config-capture grep (exit != 1 aborts) and
  clarify the image pin has no automated producer.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:21:25 +04:00
Andrey Kolkov 405eb25095 fix(backups): harden Kafka metadata backup
- Fail closed on topic listing: set -o pipefail, capture kafka-topics
  --list --exclude-internal exit separately, and abort on failure so an
  unreachable broker can no longer produce an empty backup reported as
  Succeeded. An empty list after a successful call stays legitimate.
- Check each kafka-topics --describe exit and validate the parsed
  partition count / replication factor before writing a record, so a
  timed-out describe cannot write a corrupt orders\t\t\t line.
- Store topic configs one per line with key and value in separate
  tab-delimited fields, sourced from kafka-configs --describe (one config
  per line), so a value containing a comma (cleanup.policy=compact,delete)
  round-trips; on restore each config is applied with the value bracketed
  only when it contains a comma.
- Detect topic divergence on restore instead of a silent --if-not-exists
  no-op: grow partitions with --alter when the backup asks for more, and
  fail loudly (naming the topic and both values) when it asks for fewer
  partitions or a different replication factor. A restore that cannot
  reach the recorded state no longer reports Succeeded.
- Add KafkaStrategyKind to the RestoreJob cleanup-dispatch no-op group so
  deleting a Kafka RestoreJob does not run a pointless Velero cleanup; add
  a regression test alongside the siblings, plus a reconcile Kind-gate
  test and a rendering test that actually renders the strategy template.
- Wire the kafka-metadata Chainsaw suite into both selectors
  (select-install suite_to_source, select-e2e src_to_suites) so editing
  the kafka app or the suite selects it and the bats unit tests pass.
- Pin the client image to the operator's default broker version
  (kafka 3.9.1) with a digest, and align the example default; fix the
  values comment. Point the RabbitMQ comment references (which shipped in
  the CRD description) at the Job strategy actually mirrored, and trim the
  in-pod comments that rendered into the manifest.
- Document the Strimzi Topic Operator caveat: for a topic owned by a
  KafkaTopic CR the operator reconciles its config from the CR, so an
  in-place config restore is durable only for out-of-band topics.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:21:24 +04:00
Andrey Kolkov a410a3f48b feat(backups): Kafka topic-metadata backup via a dedicated strategy driver
Add a strategy.backups.cozystack.io/Kafka driver that backs up the topic
metadata of a Cozystack-managed Kafka application — partition counts,
replication factors and non-default topic configs — through the Kafka
Admin API, and restores it with kafka-topics --create + kafka-configs
--alter. Modelled on the Job/Rabbitmq strategies: the driver renders a
one-shot batch Job, gates on the Strimzi Kafka cluster reporting Ready,
and keys each backup by .BackupName so runs do not overwrite each other.

Scope is deliberately narrow: message payloads, consumer-group offsets,
ACLs, quotas and KafkaUsers are out of scope, and there is no PITR.
Restored topics are created directly (unmanaged on the target) — the
accepted tradeoff of an Admin-API-only driver. Both restore flows are
implemented: in-place recreates dropped topics into the source, to-copy
applies the source's topics onto a freshly-bootstrapped empty cluster.

Wire it into cozy-default (strategy-kafka-default.yaml gated on the
bucket name, a BackupClass binding, read RBAC on kafka.strimzi.io/kafkas
for the readiness gate, and the Strimzi Kafka client image), document it
in backup-classes.md, and add an honest round-trip example plus Chainsaw
e2e (examples/backups/kafka-metadata, kafka-3-metadata-roundtrip) that
proves a per-run retention.ms sentinel survives both restores.

Unit tests cover the driver's pure helpers (application-ref validation,
readiness verdict, parameter round-trip, per-run artifact-URI keying).

Not yet verified on a live cluster.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:20:27 +04:00
Aleksei Sviridkin 1f04e31ed3 feat(backups): add Redis backup/restore strategy driver (#3992)
## What this PR does

Adds backup/restore support for the `Redis` app (spotahome
`RedisFailover`) via a dedicated `strategy.backups.cozystack.io/Redis`
strategy controller. The spotahome operator ships no backup CRD, so the
driver follows the generic-script-Job pattern (a specialisation of the
Job/Altinity drivers): it runs a one-shot Job that dumps an RDB snapshot
from the current master to the system bucket and replays it on restore.

- **Backup**: discover the current master via the operator's sentinel
Service (`rfs-<name>`, master `mymaster`) → `redis-cli --rdb` → `curl
--aws-sigv4` PUT to `s3://<bucket>/<namespace>/<app>/dump.rdb`.
- **Restore**: GET the object → convert RDB → RESP (`hdt3213/rdb`) →
`FLUSHALL` + `redis-cli --pipe` into the target's master. Both
**in-place** and **to-copy** are implemented; a to-copy restore reads
the source's prefix via `.Backup.ApplicationRef.Name` while writing into
a differently-named target.
- **PITR**: not supported — RDB is a discrete snapshot with no
continuous log to replay to an arbitrary time (the etcd-like case).

The driver is a specialisation of the Job/Altinity script-Job flow with
a `Redis` applicationRef gate and an app-readiness precondition
(surfaces a precise `Ready=False` instead of a silent hang; blocks only
on an explicit `Ready=False` so a missing condition never wedges
backups). Strategy parameters round-trip through the Backup's
`DriverMetadata`; there is no `underlyingResources` snapshot (the Job
pattern re-renders on restore).

### System bucket

Coordinates and credentials reach the Job as env `secretKeyRef` from the
projected `cozy-backups-creds` (the ClickHouse/Altinity delivery model),
so **the app chart is untouched** — no `backup.useSystemBucket`, no new
RBAC (`batch/jobs` and `apps.cozystack.io/*` are already granted). The
bare-host `endpoint` gets `https://` prepended when unscheme'd; against
a self-signed endpoint the strategy verifies with the CA from
`backupStorage.endpointCASecretName` when set, otherwise against the
image's system CA store; skip-verify is a separate explicit
`backupStorage.insecureSkipTLSVerify` opt-in, never a fallback. The
`cozy-default-redis` strategy is gated on `$bucketName` (CRD-ordering)
and bound in the `cozy-default` BackupClass.

### Changes

- `api/backups/strategy/v1alpha1/redis_types.go` + generated CRD and
deepcopy.
- `internal/backupcontroller/redisstrategy_controller.go` (+ unit tests)
wired into both dispatch switches, `supportedBackupStrategyKinds`, and
the no-op restore-cleanup case.
- `cozy-default-redis` strategy template + `cozy-default` BackupClass
route + helm rendering-gate coverage.
- `images/redis-backup` (redis-cli + curl + RDB→RESP converter) +
Makefile target + `backupStrategyController.redisBackupClientImage`.
- `examples/backups/redis/` — an honest marker round-trip for both
restore flows.

### Validation

`go build`/`vet`, `go test ./internal/backupcontroller/...
./api/backups/...`, `make generate` (clean), and `helm unittest` (10/10)
all pass. Not yet cluster-verified.

## Downstream

None.

```release-note
Add backup and restore support for the Redis application (spotahome RedisFailover) via a new Redis backup strategy, wired into the cozy-default BackupClass. Supports in-place and to-copy restore; PITR is not supported.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added Redis backup and restore support using RDB snapshots in
S3-compatible storage.
  * Supports restoring data in place or to a separate Redis instance.
* Added configuration for Redis backup images, custom HTTPS certificate
authorities, and TLS verification.
* Added a complete Redis backup and restore demonstration with automated
setup, verification, and cleanup.

* **Documentation**
* Documented Redis storage, credentials, endpoints, TLS, and restore
behavior.
  * Added administrator and tenant scenario guides.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-10 19:17:26 +04:00
Andrey Kolkov 92edfb810b fix(backups): run the strategy bats under bash; move template essays to {{/* */}}
The behavioural bats test ran the rendered `set -euo pipefail` script via
`sh`; the CI unit-tests runner's /bin/sh is dash, which rejects `-o pipefail`
at line 1 and reddened the whole suite (production is busybox ash, which
supports it, so the script itself is fine). Run it via `bash`, matching ash
semantics and the hack/migration-50-etcd-adopt.bats precedent.

Convert the multi-paragraph `#` rationale blocks in the shipped strategy
template to `{{/* */}}` so Helm strips them at render instead of copying the
prose verbatim into every rendered Strategy CR, where it drifts as the code
changes. Short one- to three-line guard comments stay as `#`.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-09 19:23:37 +04:00
Andrey Kolkov 1289f522fa fix(backups): unbreak install, kill destructive cleanup, reach the resource knob
Store the Redis strategy Template schemaless, as the Rabbitmq strategy
already does: without the markers controller-gen emitted the full
PodTemplateSpec schema and the CRD reached ~602 KB, pushing the chart's
Helm-release Secret past the 1 MiB cap so backupstrategy-controller failed
to install at all.

Give the example strategy a cleanup branch, and both scripts an explicit
`elif restore` / `else exit 1`, so a deleted Backup rendered with
Mode=cleanup deletes its object instead of falling into the restore path
and FLUSHALLing the live master.

Forward a backupStrategyController block through the platform bundle
alongside backupStorage, so redisBackupResources — the restore-Pod sizing
knob the loader-not-ready message points at — is reachable via
spec.components.platform.values.backupStrategyController, not merely
settable on a Helm-managed Strategy CR that reverts on the next upgrade.

Correct the endpointCASecretName doc row to match the now-optional mount,
and `int` the activeDeadlineSeconds interpolation so a large values-file
override renders as an integer, not exponent form.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-09 16:53:50 +04:00
Andrey Kolkov 651a6e3023 fix(backups): harden the redis restore AOF wait and image build
Wait for the throwaway loader's AOF rewrite to finish - no rewrite in
flight, last one ok, base AOF on disk - instead of a bare
aof_rewrite_in_progress:0 check, which right after `appendonly yes` can
fall through before the scheduled rewrite starts and replay an absent or
half-written base AOF. Same fix in the example strategy.

Declare the package image targets PHONY and serialize image-redis-backup
after image-backupstrategy-controller so their `yq -i values.yaml` digest
stamps cannot race under `make -j`. Run the demo cleanup under `set -e`
(every delete is already --ignore-not-found) so a real API failure aborts
instead of printing "Cleanup complete" over a half-torn-down demo.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-09 16:23:12 +04:00
Andrey Kolkov b4d621f4f7 fix(backups): delete Redis backup objects on cleanup and size the restore Pod
Delete the backup object when its Backup is removed. The Redis driver
owns its S3 object (one per BackupJob, no engine retention prunes it), so
it now spawns a cleanup-mode Job that deletes the object and holds the
Backup's finalizer until it completes, mirroring the Rabbitmq driver.
Without this a retention-pruned Plan grew the bucket unboundedly and the
key was lost with the CR.

Size the strategy Pod from backupStrategyController.redisBackupResources
so an admin can raise the restore ceiling (the throwaway loader holds the
whole dataset in memory) via chart values, which a direct edit of the
cluster-scoped, Helm-managed Strategy CR could not survive. The
loader-not-ready diagnostic now names the memory limit, the likeliest
cause, instead of only the image version.

Make the endpoint CA volume optional: nothing projects that Secret into
the app namespace, so a required mount wedged every backup Pod on
FailedMount; a missing ca.crt now falls through to the system CA store
and fails the handshake fast, which is not the same as skipping verify.

Promote the Pod-level activeDeadlineSeconds to the Job so a wedged run is
bounded once, not (backoffLimit+1) times. Use find, not ls, for the base
AOF so its no-match diagnostic is not dead code under pipefail.

Pin the object key reaching the rendered Job: both fixtures now render
--key={{ .ObjectKey }} and assert it, so zeroing the key (which left the
package green) is caught. Document .ObjectKey and the cleanup mode in the
CRD render-context, list cozy-default-redis among the Helm-managed CRs,
and mirror the CA/TLS and find fixes into the example strategy.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-09 12:19:02 +04:00
Andrey Kolkov 56e10b1580 fix(backupstrategy-controller): gate the default MongoDB Strategy on its CRD (#3980)
## What breaks today

Upgrading any existing cluster to a build that carries the new MongoDB
backup driver wedges the `backupstrategy-controller` HelmRelease
permanently, with `resource mapping not found for name:
"cozy-default-mongodb" ... no matches for kind "MongoDB" in version
"strategy.backups.cozystack.io/v1alpha1"; ensure CRDs are installed
first`.

This chart ships its `strategy.backups.cozystack.io` CRDs as ordinary
templates, and Helm resolves every document in a release manifest
through the cluster's RESTMapper in `Build()` **before** it applies any
of them. A Strategy CR whose CRD is not served yet therefore fails
`Build()`, so the release applies **zero** objects — including the CRD
that would have made the mapping resolve — and every retry renders the
same manifest. Nothing recovers on its own.

## Why the gate that exists does not cover it

The chart already knows this trap: it is why every default Strategy
gates on the bucket name. That gate covers the **first install** of a
fresh cluster, where nothing is resolvable yet and no Strategy renders
at all. It cannot cover shipping a **new kind** to a chart that is
already installed — there the bucket name resolved releases ago, so the
first upgraded revision renders the new CRD and the new Strategy CR
together and wedges.

## The fix

Gate the CR on its CRD actually being served, through a `crdEstablished`
helper. The helper asks about the **CRD object** rather than listing the
custom kind, because `lookup` on a kind the apiserver does not serve is
a render *error*, not an empty result, which would trade this wedge for
a different permanent failure; `apiextensions.k8s.io/v1` is always
served, so that lookup can only answer yes or no. It requires
`Established=True` rather than mere existence, because that is when the
kind enters the RESTMapper.

Convergence afterwards is the mechanism the bucket gate already relies
on, and is **not** the HelmRelease interval: once the CRD is served, the
controller's `DefaultObjectsGate` sees the kind mapped and the CR absent
and forces a real Helm upgrade whose render includes it. Kinds that do
not map at all are skipped there (`meta.IsNoMatchError`), which is what
stops the gate forcing upgrades against a render that cannot yet produce
the object.

`bucketNameOverride` short-circuits the CRD lookup for the same reason
it short-circuits the bucket lookup: it marks an offline render — `helm
template`, a CI diff, a unit test — where there is no apiserver to ask.
Live deploys go through Flux and never set it.

## How it was found

The release upgrade E2E lane in #3276, on the `v1.6.2 → main` path.
`mongodbs.strategy.backups.cozystack.io` is the only definition added to
this chart since the tag, so this is reachable only by upgrading an
existing install — no install-only suite can see it.

## Verification

Reproduced and fixed against a live cluster that happens to be in
exactly the pre-upgrade state (the seven older strategy CRDs present,
`mongodbs` absent). `helm template --dry-run=server` of main's chart
fails with the error above and produces zero documents; the same render
with this change succeeds with 21 documents and the Strategy correctly
withheld. The helper's two branches were probed the same way against
that cluster: a served CRD answers `true`, an absent one answers empty.

The new unit case pins the gate offline through the external-S3 path,
where the bucket name resolves without the offline override so that only
the CRD gate can hold the CR back. Removing the gate turns exactly that
case red; the suite is 19/19 green with it.

```release-note
Fixed a permanent failure of the backupstrategy-controller release when upgrading a cluster to a version that adds a new backup-strategy kind: the default MongoDB Strategy is now withheld until its CRD is served, instead of failing the whole release with "no matches for kind".
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved backup strategy rendering during upgrades by waiting until
the required MongoDB resource definition is available.
* Preserved offline rendering when an explicit bucket name is provided.
* Prevented invalid default strategies from being generated when the
required resource definition is unavailable.

* **Tests**
* Added coverage for offline rendering and unavailable
resource-definition scenarios.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-09 11:17:28 +04:00
Andrey Kolkov 45a2f54c62 feat(backups): back up TLS-enabled Redis apps via the default strategy
The shipped cozy-default-redis strategy connected to Sentinel and the
master in plaintext, so a BackupJob against a Redis app with tls.enabled
(the operator moves both to a TLS-only listener) failed at master
resolution. Mount the app CA the operator publishes at
redis-<app>.ca-cert as an optional Secret, and dial redis-cli with
--tls --cacert exactly when its ca.crt is present, so a non-TLS app
keeps the plaintext path and the throwaway restore loader stays
local-only. Covers the default authClients: no; mutual TLS is out of
scope, since the platform issues no client certificate.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-08 18:47:13 +04:00
Andrey Kolkov b0eb104276 feat(backups): add Redis backup/restore strategy driver
The spotahome RedisFailover operator ships no backup CRD, so add a
dedicated strategy.backups.cozystack.io/Redis driver that runs a one-shot
Job: it dumps an RDB snapshot from the current master (discovered via the
operator's sentinel Service) to the system bucket over S3, and on restore
replays it (RDB -> RESP via redis-cli --pipe) into the target's master.

Structurally a specialisation of the Job/Altinity script-Job pattern with
a Redis applicationRef gate and an app-readiness precondition. Backup
parameters round-trip through the Backup's DriverMetadata (no
underlyingResources snapshot); restore drives from .Backup.ApplicationRef
so a to-copy restore reads the source's object-storage prefix while
writing into a differently-named target. Both in-place and to-copy are
supported; PITR is not (RDB is a discrete snapshot).

- api: Redis/RedisList strategy type + generated CRD and deepcopy
- driver: reconcileRedis / reconcileRedisRestore + pure helpers, wired
  into both dispatch switches, supportedBackupStrategyKinds and the no-op
  restore cleanup case
- default: gated cozy-default-redis strategy (system-bucket coords via
  cozy-backups-creds env secretKeyRef, https:// scheme + optional CA) and
  the Redis route in the cozy-default BackupClass
- image: images/redis-backup bundling redis-cli + curl + an RDB->RESP
  converter, wired via backupStrategyController.redisBackupClientImage
- tests: driver unit tests (gate, precondition, render, param round-trip,
  both reconcile paths) and the helm rendering-gate coverage
- example: examples/backups/redis honest marker round-trip for both
  restore flows

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-By: LLM
2026-09-08 17:56:20 +04:00
Andrey KolkovandClaude Opus 4.8 ba32df69c0 feat(backups): add RabbitMQ definitions backup via a dedicated strategy driver
RabbitMQ's cluster operator ships no backup CRD and RabbitMQ has no
consistent online export of message payloads, so back up broker
DEFINITIONS (vhosts, users, permissions, queues, exchanges, bindings,
policies, parameters) over the management HTTP API and store them in the
system bucket. Message payloads and PITR are out of scope - a full-data
backup is a Velero volume snapshot, a separate strategy.

Add a first-class strategy.backups.cozystack.io/Rabbitmq driver, following
the Altinity pattern (a dedicated driver that renders a one-shot Job for an
HTTP-API engine):

  - reconcileRabbitmq / reconcileRabbitmqRestore render the strategy's
    PodTemplateSpec (curl GET/POST /api/definitions, push/pull to S3 with
    --aws-sigv4) as a batch/v1.Job, for both in-place and to-copy restore.
  - A Go-read precondition gates each run on the RabbitmqCluster reporting
    AllReplicasReady (via the rabbitmqtypes partial type), surfacing a
    precise Ready=False wait instead of a Job that starts and fails.
  - The produced Backup records the object location on status.artifact.uri
    (rendered from spec.artifactURITemplate; the Backup CRD has no status
    subresource, so it persists on Create, as in the Etcd driver).

Wire dispatch in both controllers + supportedBackupStrategyKinds, register
rabbitmqtypes in the scheme, and grant read on rabbitmqclusters. Ship
cozy-default-rabbitmq (a $bucketName-gated Rabbitmq strategy) and route
apps.cozystack.io/RabbitMQ to it in the cozy-default BackupClass. The
application chart is untouched.

Add unit tests for the driver's pure helpers and the precondition gate, a
runnable example, and an honest-roundtrip Chainsaw e2e that seeds a
sentinel vhost, backs it up, drops it, and asserts it round-trips through
object storage on both restore flows.

The Job pod reuses the platform-migrations image (curl + ca-certificates)
under a new rabbitmqBackupClientImage key, stamped in lockstep with
chBackupClientImage so the repository stays pinned at one digest; a
dedicated slim image is a follow-up.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-01 18:21:10 +04:00
Myasnikov Daniil 50ed307fef fix(backupstrategy-controller): gate the default MongoDB Strategy on its CRD
Upgrading a cluster to a build that carries the new MongoDB backup driver
wedges the backupstrategy-controller release permanently:

  Helm upgrade failed: resource mapping not found for name:
  "cozy-default-mongodb" ... no matches for kind "MongoDB" in version
  "strategy.backups.cozystack.io/v1alpha1"; ensure CRDs are installed first

This chart ships its strategy.backups.cozystack.io CRDs as ordinary
templates, and Helm resolves every document in a release manifest through
the cluster's RESTMapper in Build() BEFORE applying any of them. A
Strategy CR whose CRD is not served yet therefore fails Build(), so the
release applies ZERO objects — including the CRD that would have made the
mapping resolve — and every retry renders the same manifest. Nothing
recovers on its own.

The chart already knows about this trap: it is why every default Strategy
gates on the bucket name. That gate covers the first install of a fresh
cluster, where nothing is resolvable and no Strategy renders at all. It
cannot cover shipping a NEW kind to a chart that is already installed,
which is what happened here: the bucket name resolved releases ago, so
the first upgraded revision renders the new CRD and the new Strategy CR
together and wedges.

Gate the CR on its CRD being served, via a crdEstablished helper that
asks about the CRD OBJECT rather than listing the custom kind — `lookup`
on a kind the apiserver does not serve is a render error, not an empty
result, which would trade this wedge for a different permanent failure.
apiextensions.k8s.io/v1 is always served, so that lookup can only answer
yes or no. Established=True rather than mere existence, because that is
when the kind enters the RESTMapper.

Convergence afterwards is the mechanism the bucket gate already relies on
and is NOT the HelmRelease interval: once the CRD is served, the
controller's DefaultObjectsGate sees the kind mapped and the CR absent
and forces a real Helm upgrade whose render includes it. Kinds that do
not map at all are skipped there (meta.IsNoMatchError), which is what
stops the gate forcing upgrades against a render that cannot yet produce
the object.

Found by the release upgrade E2E lane on the v1.6.2 → main path, where
`mongodbs.strategy.backups.cozystack.io` is the only definition added
since the tag.

Verified against a live cluster that is in exactly the pre-upgrade state
(seven older strategy CRDs, no mongodbs): `helm template --dry-run=server`
of main's chart fails with the error above and produces zero documents,
while the same render with this change succeeds with 21 documents and the
Strategy correctly withheld. The helper's two branches were probed the
same way — a served CRD answers true, an absent one answers empty.

The new unit case pins the gate offline through the external-S3 path,
where the bucket name resolves without the offline override so only the
CRD gate can hold the CR back; removing the gate turns exactly that case
red.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-28 11:29:59 +05:00
Myasnikov Daniil 4f9c20cac0 fix(backupstrategy-controller): gate the default MongoDB strategy on bootstrap
A fresh install of packages/system/backupstrategy-controller fails
permanently and applies nothing:

  Helm install failed for release cozy-backup-controller/backupstrategy-controller
  resource mapping not found for name: "cozy-default-mongodb"
  no matches for kind "MongoDB" in version "strategy.backups.cozystack.io/v1alpha1"

The chart ships its own strategy.backups.cozystack.io CRDs as ordinary
templates (templates/crds.yaml globs definitions/*.yaml), and Helm
resolves every document in a release manifest through the cluster's
RESTMapper in Build() before it applies any of them. An ungated Strategy
CR therefore has no mapping on the first install, Build() fails, and the
release applies ZERO objects, including the CRDs that would have made
the mapping resolve. Nothing recovers: every retry renders the same
manifest, so this is permanent rather than a race, and Helm's kind
sorter cannot help because it orders the apply, not the Build.

The seven sibling strategies never hit this because each gates on a
resolved $bucketName, which is empty until the COSI BucketClaim status
is reconciled. That gate is documented as "the CR needs the bucket
name", and it silently doubles as the CRD-ordering guard. MongoDB
genuinely does not need the bucket name, so the gate was left off and
the accidental protection went with it.

Gate it too, for the ordering reason rather than the value, and record
that reason in the template so the next bucket-independent strategy
keeps the gate. Convergence is unchanged: DefaultObjectsGate already
routes apps.cozystack.io/MongoDB to cozy-default-mongodb and forces a
second revision once any routed object is missing, which is how the
seven siblings materialise today.

Verified by rendering both windows. Before: the bootstrap render emits
one Strategy, MongoDB, alongside the eight not-yet-applied CRDs. After:
it emits none, and with backupStorage.bucketNameOverride set it emits
all eight.

tests/rendering_test.yaml asserted count: 1 for MongoDB under a case
named "bootstrap window (no resolvable bucket name)", so the defect was
codified green. helm-unittest has no RESTMapper and cannot see it.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 14:53:13 +05:00
Andrey KolkovandClaude Opus 4.8 dff56aa998 feat(backups): add a native MongoDB (psmdb) backup-strategy driver
Adds a MongoDB (Percona Server for MongoDB / psmdb) backup-strategy driver so
`apps.cozystack.io/MongoDB` reaches feature parity with the CNPG, MariaDB and
ClickHouse native drivers — logical backups, restore-to-differently-named
instances, and point-in-time recovery through the unified
BackupJob/RestoreJob/BackupClass/Plan interface (closes #3545).

The driver orchestrates the operator's PerconaServerMongoDBBackup /
PerconaServerMongoDBRestore CRDs:

- BackupJob: creates a one-shot PerconaServerMongoDBBackup, polls
  status.state, and persists the S3 destination + storage descriptor
  (by-reference, no credentials) onto the Cozystack Backup artifact.
- RestoreJob: drives PerconaServerMongoDBRestore via backupSource (read back
  from the source backup status, snapshot fallback) so the same artifact
  restores in-place or into a differently-named instance; PITR via
  spec.options.pitr (date/latest).

Because psmdb resolves the backup storage by name against the cluster's
spec.backup.storages, the driver names the chart's storage (s3-storage) via the
cozy-default-mongodb Strategy rather than injecting it, and surfaces a clear
Ready=False precondition when a target has backups disabled.

Wiring:
- api/backups/strategy/v1alpha1: MongoDB strategy type + generated CRD/deepcopy.
- internal/backupcontroller: psmdbstrategy_controller + psmdbtypes/mongodbapp
  partial-schema mirrors; dispatch, supported-kinds and scheme registration.
- backupstrategy-controller: cozy-default MongoDB route, cozy-default-mongodb
  Strategy CR, and psmdb.percona.com RBAC.
- apps/mongodb: backup.insecureSkipTLSVerify knob (psmdb s3 storage has no
  CA-bundle field) so backups can target Cozystack's self-signed seaweedfs.

Tests + e2e:
- unit tests for the driver (validation, ensure-idempotency, precondition,
  backupSource resolution, snapshot round-trip, PITR option parsing).
- examples/backups/mongodb + a mongodb-2-backup-roundtrip Chainsaw suite that
  drives run-all.sh end-to-end (sentinel document round-trips through S3 into a
  restored copy while the source stays untouched), per docs/agents/e2e-testing.md.
- docs/operations/backup-classes.md updated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-08-12 12:00:33 +04:00
IvanHunters cfaf2e086c fix(backupstrategy-controller): repair lookup-gated backup objects (#3524)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->

## What this PR does

Fixes #3518.

Objects in `packages/system/backupstrategy-controller` and
`packages/system/bucket` are gated on a Helm `lookup` of an object that
is still being created. When the lookup is empty the object is skipped —
and since helm-controller does not re-render a release whose chart and
values did not change (drift detection is off on operator-generated
HelmReleases), **the skip is permanent**, not a bootstrap window. On the
reporter's cluster the install-timing window was 4 seconds and the
objects were still missing months later:

- the 7 default backup `Strategy` CRs and the Velero
`BackupStorageLocation`, gated on `include
"backupstrategy-controller.bucketName"` → a `lookup` of `BucketClaim`;
- the `<bucket>-<user>-credentials` Secret
(`packages/system/bucket/templates/user-credentials.yaml`), gated on a
`lookup` of the COSI Secret.

Only `BackupClass cozy-default` was present, because it renders
unconditionally. Its own comment predicted this case while assuming
"Flux re-renders the chart", which does not happen — the same trap
`packages/core/platform/sources/cozystack-basics.yaml` already
documents: *"Drift detection is off on operator-generated HelmReleases,
so a capability gate in the templates would render the policy out at
first install and never add it back."*

Consequence: migration 50 (etcd adoption, v1.6.0) is fail-closed on the
snapshot target. It resolves the target from the projected credentials
Secret, which has no source without `<bucket>-<user>-credentials`, so
its Job reboots forever and the cluster cannot be upgraded at all.

### 1. `backupstrategy-controller` reconciles the objects into existence

New leader-elected `DefaultObjectsGate` runnable
(`internal/backupcontroller/default_objects_gate.go`), modelled on the
existing `SystemCredentialsProjector`. Every minute:

- resolves the bucket name from the **projector's source Secret**
(`parseSourceSecret`) — no new dependency, no `lookup`, and it covers
the external-S3 path too. While it is unresolvable the gate does
nothing: forcing then would only re-run the same empty lookup;
- reads `BackupClass cozy-default` as the manifest of what must exist
(its `strategyRef`s), plus the `cozy-default` BSL when
`velero.bslEnabled`;
- when any is missing, stamps `reconcile.fluxcd.io/forceAt` **and**
`requestedAt` on its own HelmRelease. Both are required: `requestedAt`
alone only requests a reconcile, which is a no-op for an unchanged
release; `forceAt` is what triggers a real Helm upgrade, and it is only
honoured together with `requestedAt`;
- throttled to one forced upgrade per 5 minutes, and a no-op in the
steady state. A `strategyRef` whose CRD is not installed is not counted
as missing, so a missing CRD cannot drive an endless force loop.

Two new metrics: `cozystack_backup_default_objects_missing{backupclass}`
— the alert that would have caught this, since it is non-zero regardless
of the HelmRelease's `Ready` condition — and
`cozystack_backup_default_objects_force_reconciles_total`.

RBAC delta is two lines: `patch` on `helmreleases` (`get` was already
there) and `get` on `backupstoragelocations`. Both accesses are point
Gets through the dynamic client, so no cluster-wide informer and no
`list`/`watch`.

Off switch: `backupStorage.reconcileDefaultObjects: false`.

**Why not a render-time `fail` in this chart:** it would deadlock.
`templates/bucket.yaml` in this same chart is the producer of the
`Bucket` whose `BucketClaim` the lookup reads, so a failed render never
applies it and the condition can never resolve.

**Why not template the objects in Go:** their bodies are values-driven
(endpoint, region, `forcePathStyle`, the Altinity strategy's whole
PodTemplateSpec). Helm stays their single author; the gate only makes
sure the render that produces them actually happens.

**Why not `dependsOn`:** an edge guarantees the `Bucket` CR is
*applied*, not that its claim is *reconciled*, so it does not close the
race.

### 2. The gate also repairs the credentials Secret, one release earlier

The `<bucket>-<user>-credentials` Secret is the same trap as the
Strategy CRs, one release up the chain:
`packages/system/bucket/templates/user-credentials.yaml` renders it
behind a `lookup` of the COSI Secret, and skips it just as permanently.
It is also the *first* domino — the projector reads it, so without it
there is no `cozy-backups-creds`, no strategy, no Velero, and no
migration 50.

An earlier revision of this PR made that render `fail` so the release
would retry and converge. [@IvanHunters pointed
out](https://github.com/cozystack/cozystack/pull/3524#issuecomment-5214269078)
that Helm cannot render a partial set, so the `fail` aborts the
**whole** `<bucket>-system` release — the other users' Secrets and the
bucket UI Deployment/Service/Ingress/HTTPRoute with them. One declared
user whose `BucketAccess` never provisions would take down every other
user of that bucket, and on an installed release park it in `Failed`,
blocking every later upgrade. That was a real blast-radius regression on
a universal path, and it is gone.

The repair now lives in the gate, where it is per-object:

- while the source Secret is absent, or carries no bucket name, the gate
stamps `forceAt` + `requestedAt` on the bucket's `bucket-<name>-system`
HelmRelease instead of doing nothing;
- **no self-deadlock:** the `BucketClaim` and the `BucketAccess` whose
COSI Secret that lookup reads are rendered unconditionally by the
**parent** release (`packages/apps/bucket/templates/bucketclaim.yaml`),
not by the one being forced;
- **no RBAC delta:** `patch` on `helmreleases` is already
cluster-scoped;
- the two releases are throttled **independently**. They are forced in
sequence on a bootstrap (credentials, then the objects the resolved
bucket unblocks), so one shared timestamp would delay the second by a
full `MinForceInterval` for no reason;
- coordinates come from the chart under `provisionBucket`, and are
omitted on external S3, where the Secret is admin-managed and no release
renders it.

The bucket chart keeps its partial render. Its comment now explains why
the skip is not self-healing and why failing is the wrong lever, and a
test renders the UI with *every* user unresolvable so a reintroduced
`fail` fails the suite.

**Scope this narrows, deliberately:** tenant buckets keep the
pre-existing silent skip. The gate owns only the platform bucket, which
is the cluster-breaking path. Generalising the repair to every bucket
needs a controller that owns tenant buckets and is a separate change.

Two more corrections from the same review:

- `cozystack_backup_default_objects_missing` was only written on the
happy path, so the state this PR exists to catch reported `0` and the
documented alert never fired. The credentials Secret now counts in
`missing`, so an absent or rotated-away Secret moves the gauge. On an
API error the gauge is deliberately left alone rather than flapping, and
a new `cozystack_backup_default_objects_check_errors_total` marks it
stale; the doc pairs the two instead of overstating the gauge.
`force_reconciles_total` gained `namespace`/`name`, since two releases
can now be forced.
- `forceHelmRelease` does a point `Get` and skips a release with
`spec.suspend: true`. helm-controller ignores both annotations while
suspended (`cozyhr suspend` sets exactly that), so the gate was
re-stamping every `MinForceInterval` for the whole suspension and
climbing the force counter — which the runbook attributes to a render
that is not producing the objects. The skip is logged and not counted.

### 3. Corrected comments and docs

`_helpers.tpl`, `backupclass-default.yaml`, `values.yaml` and
`docs/operations/backup-classes.md` all claimed the skip self-heals on
the HelmRelease interval, and the doc suggested a `flux reconcile` that
does not re-render. They now say it does not, and why. The two-release
manual recovery for already-affected clusters is documented, including
the easy-to-miss part: the credentials Secret is rendered by the
`bucket-<name>-system` release, not by `bucket-<name>`.

### Manual recovery for already-affected clusters

Only needed on a version without the gate. A plain `flux reconcile
helmrelease` does **nothing** — it does not re-render.

```bash
# 0. Confirm the bucket name is resolvable first — forcing before that
#    just re-runs the same empty lookup.
kubectl -n tenant-root get bucketclaim bucket-cozy-backups -o jsonpath='{.status.bucketName}'

# 1. The credentials Secret. Rendered by the -system release, NOT by
#    bucket-cozy-backups — easy to miss, and the projector (hence every
#    strategy, Velero, and migration 50) has no source without it.
ts=$(date -u +%Y-%m-%dT%H:%M:%SZ)
kubectl -n tenant-root annotate helmrelease bucket-cozy-backups-system \
  reconcile.fluxcd.io/forceAt="$ts" reconcile.fluxcd.io/requestedAt="$ts" --overwrite

# 2. The Strategy CRs and the Velero BSL.
ts=$(date -u +%Y-%m-%dT%H:%M:%SZ)
kubectl -n cozy-backup-controller annotate helmrelease backupstrategy-controller \
  reconcile.fluxcd.io/forceAt="$ts" reconcile.fluxcd.io/requestedAt="$ts" --overwrite
```

### Validation

- `go test ./internal/... -count=1` — pass, including 15 cases for the
gate: force on missing, no-op in the steady state, throttling, unmapped
kinds ignored, BSL skipped when disabled, patch failure surfaced without
advancing the throttle, absent HelmRelease tolerated, plus
force-on-absent-Secret, force-on-empty-bucket-name, external-S3
reports-but-does-not-force, independent throttles, suspend skip on each
of the two releases, gauge written on the unresolved path, and errors
counter on a failed check.
- `make helm-unit-tests` — pass, with 2 new suites: 3 cases for the
bucket chart (partial render preserved, and the UI still renders with
every user unresolvable — the guard against reintroducing the `fail`), 7
for the gate's env + RBAC wiring. Those include that `.Release.Name` and
`.Release.Namespace` match the HelmRelease the operator generates
(`releaseName` becomes the HR's `metadata.name`, and no
`spec.releaseName`/`targetNamespace` is set, so the Helm release name
equals the HR name), that the bucket release coordinates track a renamed
bucket together with `systemSecretName`, and that they are unwired on
`provisionBucket: false`.
- `make generate` — no diff.
- `make go-unit-tests rd-presets-check test-check-readiness
migrations-target-check` — pass.
- `make bats-unit-tests` — every suite passes except
`hack/migration-seaweedfs-db-adopt.bats`, which needs a Docker daemon
not available in this environment ("docker is required: these tests run
the migrations inside alpine"). Unrelated to this diff.

**Not verified — please read this before approving.** There was still no
live-cluster run. What that gap covers is smaller than it was: dropping
the chart `fail` removed the universal behaviour change on every
bucket's install path, so there is no longer a render-blind failure mode
whose convergence has to be observed. What remains is one mechanism —
`forceAt` + `requestedAt` on an unchanged release — applied to two
releases instead of one, argued from read code rather than an observed
install: `Strategy.Name = RetryOnFailure` on operator-generated
HelmReleases (`internal/operator/package_reconciler.go`), and
`bucketName = "bucket-" + claim.UID` in
`packages/system/objectstorage-controller/images/objectstorage/patches/92-bucketclaim-propagate-ready.diff`.
It is the same procedure as the manual recovery below, which has been
run by hand on the reporter's cluster. Someone with a bootstrap cluster
should still confirm that the gate fires within a minute or two of the
bucket becoming ready, and that forcing `bucket-<name>-system`
materialises the credentials Secret.

### Screenshots

Not applicable: no UI change. This PR touches a controller, two system
charts, and docs.

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

Walked file by file against the diff of this PR.

**`website` — affected, follow-up issue opened.**
`content/en/docs/next/operations/services/backup-classes.md` mirrors
this repo's `docs/operations/backup-classes.md` and carries the exact
stale claim this PR corrects, plus a `flux reconcile helmrelease`
recovery suggestion that has no effect.
`operations/configuration/platform-package.md` enumerates the forwarded
`backupStorage` keys, and `reconcileDefaultObjects` is admin-settable
through `spec.components.platform.values.backupStorage` (the whole block
is deep-merged), so that list becomes incomplete. Filed as an issue
rather than a PR because the wording depends on the shape this PR lands
in, and whether to touch the frozen `v1.5`/`v1.6` copies is a maintainer
call. No existing website issue or PR covered it.

**`terraform-provider-cozystack` — not affected.** Checked, not assumed:
`packages/apps/**` is untouched by this diff (`git show --stat HEAD --
packages/apps/` is empty), so no app `values.schema.json`, no
`values.yaml` default, no version enum, and no `ApplicationDefinition`
`kind`/`plural`/`release.prefix` changed. `reconcileDefaultObjects` is a
key on a **system** chart, which has no `values.schema.json` and is not
modelled by the provider. (An earlier revision added
`requireUserCredentials` to the `bucket` system chart; that is gone, and
no key was added to `apps/bucket`, so the app API is untouched.) The
`<bucket>-<user>-credentials` Secret the provider looks up by literal
name is **not renamed** — this PR only changes whether it gets created,
which if anything removes a case where the provider silently read null
from a Secret that never existed.

Everything else: no `hack/` layout or make-target change (`ccp`,
`external-apps-example`, `cozyhr`), no `ApplicationDefinition` CRD
change, no namespace rename, no node prerequisites in
`hack/e2e-prepare-cluster.bats` (`talm`, `ansible-cozystack`), no
`cozyhr.cozystack.io/values-files` annotation or chart-source-kind
change, no `service-proxy-name` / `wholeIP` / `allowICMP` rename
(`cozy-proxy`), no telemetry metric or label rename
(`cozystack-telemetry-server` — the two metrics here are new, not
renamed), no node or network requirement change (`examples`).

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/issues/639
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(backupstrategy-controller): the default backup Strategy CRs, the Velero BackupStorageLocation and the bucket user-credentials Secret were skipped permanently when their install-time Helm `lookup` came back empty, leaving clusters with a BackupClass but no strategies and blocking the v1.6.0 etcd migration. The controller now detects the missing objects and forces the Helm upgrade that creates them, including on the bucket release that renders the credentials Secret.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added automatic detection and recovery for missing default backup
resources.
- Missing resources trigger a throttled Helm upgrade once bucket
configuration is available.
- Added configuration to enable or disable default-object
reconciliation.
- Added validation requiring user credential Secrets during bucket
installation by default, with an offline rendering option.

- **Bug Fixes**
- Improved recovery from incomplete backup and Velero storage
configuration.

- **Documentation**
- Expanded operational guidance, monitoring metrics, alerts, and manual
recovery steps.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-10 13:10:08 +03:00
Aleksei Sviridkin 8aadbd5731 test(backupstrategy-controller): anchor the metrics-absence guard to its container
The two notContains assertions in the metrics.enabled=false case address
the controller by index. notContains passes whenever its path resolves
to something that lacks the forbidden content, and a different container
at that index qualifies, so a sidecar inserted ahead of the controller
leaves both assertions green while the flag they forbid comes back on
the real container.

Assert the container's name at the same index first. That fails when the
index stops pointing at the controller, which is the case the negatives
structurally cannot see.

Verified by rendering a sidecar ahead of the controller: before this
change the case stayed green and only unrelated tests in the same file
went red, by accident rather than by design; after it, the anchor fails.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-07 15:08:56 +03:00
Mattia Eleuteri f859b710a4 fix(backupstrategy-controller): repair the bucket credentials Secret from the gate, not a chart fail
Addresses the four findings on #3524.

**The `fail` in the bucket chart (MAJOR).** The <bucket>-<user>-credentials
Secret was made to `fail` the render while its COSI lookup was empty, so the
release would retry and converge instead of skipping the Secret permanently.
Helm cannot render a partial set, so that fail aborts the WHOLE
<bucket>-system release: the other users' Secrets, and the bucket UI
Deployment/Service/Ingress/HTTPRoute with them. One declared user whose
BucketAccess never provisions takes down every other user of that bucket,
and on an already-installed release parks it in Failed, blocking every later
upgrade.

**No per-bucket escape (MAJOR).** Rather than add one, remove the need for
it. The gate already forces the release that renders the lookup-gated
Strategy CRs; the credentials Secret is the same trap one release earlier,
and the gate reads that Secret to resolve the bucket name in the first
place. So while it is absent, or carries no bucket name, force the bucket's
<bucket>-system release instead of doing nothing. That cannot deadlock its
own precondition: the BucketClaim and the BucketAccess whose COSI Secret the
lookup reads are rendered unconditionally by the parent release, not by the
one being forced.

The two releases are throttled independently — they are forced in sequence
on a bootstrap, so a shared timestamp would delay the second by a full
MinForceInterval for no reason. Coordinates come from the chart under
provisionBucket, and are omitted on external S3 where the Secret is
admin-managed and no release renders it. No RBAC delta: patch on
helmreleases is already cluster-scoped.

The chart goes back to a partial render, with the comment now explaining why
the skip is not self-healing and why failing is the wrong lever, and a test
that renders the UI with every user unresolvable so a reintroduced `fail`
fails the suite. Tenant buckets keep their pre-existing behaviour;
generalising the repair to every bucket needs a controller that owns them
and is tracked separately.

**Stale gauge (MINOR).** cozystack_backup_default_objects_missing was only
written on the happy path, so the state the gate exists to catch reported 0
and the documented alert never fired. It is now written on the unresolved
path too — the credentials Secret counts as one of the objects the default
backups depend on, which is what it always was. On an API error the gauge is
deliberately left alone rather than flapping, so a new
cozystack_backup_default_objects_check_errors_total marks it stale, and the
doc pairs the two instead of overstating the gauge.

**Suspended releases (MINOR).** forceHelmRelease now does a point Get and
skips a release with spec.suspend: true. helm-controller ignores forceAt and
requestedAt while suspended (`cozyhr suspend` sets exactly that), so the gate
was re-stamping every MinForceInterval for the whole suspension and climbing
the force counter — which the runbook attributes to a render that is not
producing the objects, the wrong diagnosis. The skip is logged and not
counted, so a climbing counter keeps its documented meaning.

Reported-by: IvanHunters <xorokhotnikov@gmail.com>
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Assisted-By: Claude <noreply@anthropic.com>
2026-08-07 11:10:41 +02:00
IvanHunters aa3accd128 fix(backupstrategy-controller): stop the default-objects gate looping when velero.bslEnabled=false
The cozy-default BackupClass routes VMInstance/VMDisk to the Velero
Strategy CRs unconditionally, but the chart gates those CRs on
velero.bslEnabled. With the BSL disabled they never render, so the gate
counted them as permanently missing and forced a Helm upgrade every
MinForceInterval forever, climbing force_reconciles_total and pinning the
missing gauge. Skip Velero-kind strategyRefs when the BSL is disabled
(empty VeleroNamespace), the same flag that gates their render. The
regression test now omits the manually-created Velero objects so it fails
without the skip. Correct the values.yaml comment that wrongly claimed the
Strategy CRs still ship in this mode.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-05 09:52:18 +03:00
Mattia Eleuteri 51ad226706 fix(backupstrategy-controller): repair lookup-gated backup objects
The default Strategy CRs and the Velero BackupStorageLocation are
Helm-templated behind a `lookup` of the BucketClaim the same chart
creates, and the <bucket>-<user>-credentials Secret behind a `lookup` of
the COSI Secret. When those lookups are empty at install time the objects
were silently skipped, and helm-controller does not re-render a release
whose chart and values did not change (drift detection is off on
operator-generated HelmReleases), so the skip was permanent: clusters ran
for months with only BackupClass cozy-default and no Strategy CRs at all.
That also fail-closes the pre-adoption snapshot in the v1.6.0 etcd
migration, which reads the projected credentials Secret.

Add a DefaultObjectsGate runnable to backupstrategy-controller. Once the
bucket name is resolvable from the projector's source Secret, it checks
that every object cozy-default routes to exists and forces one real Helm
upgrade (reconcile.fluxcd.io/forceAt + requestedAt) when any is missing.
Helm remains the objects' only author; the gate only makes sure the render
that produces them actually happens. It cannot be a render-time `fail` in
this chart: the chart is the producer of the Bucket its own lookup reads,
so a failed render would deadlock the condition.

The bucket chart's user-credentials template CAN fail safely — the
BucketAccess that produces the COSI Secret belongs to the parent release
and that release retries forever — so make it fail loudly instead of
skipping, with requireUserCredentials=false as the offline-render escape.

Also correct the chart and doc comments that claimed Flux re-renders on
its interval, and document the two-release manual recovery for clusters
already affected.

Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-08-03 17:16:55 +02:00
Andrey Kolkov 8aeedde088 feat(backups): support PostgreSQL point-in-time recovery (PITR) (#3383)
## What this PR does

Makes PostgreSQL point-in-time recovery (PITR) a genuinely supported,
tested capability for the new `backups.cozystack.io` API. Fixes #2774.

Was **stacked on #3313** (`fix/3300-cnpg-barman-plugin`) for the
`recoveryTime` → CNPG `bootstrap.recovery.recoveryTarget.targetTime`
plumbing; #3313 has since merged and this branch is rebased on `main`.
This PR adds the reliability + verification layer #2774 asked for.

### Changes

- **Precise failure reason for an unreachable target.** The restore
deadline (`spec.options.restoreTimeoutSeconds`, default 30m) stays the
sole authority for "this restore is stuck" — the client waits the full
window either way. What changes is the *diagnosis*: once the deadline
has elapsed, if `recoveryTime` was set and the recovery Cluster never
went healthy, the CNPG driver reads the `full-recovery` pod logs and —
when several distinct attempts exited with PostgreSQL's `FATAL: recovery
ended before configured recovery target was reached` — fails the
RestoreJob with `RecoveryConverged=False` / `Ready=False` (reason
`RecoveryTargetUnreachable`) and an actionable message, instead of a
bare generic timeout. Timing is unchanged; only the reason/message
differ. (For a faster rejection, set a shorter `restoreTimeoutSeconds`.)
Design points:
- **We deliberately do NOT trip early off the FATAL.** A valid near-now
target emits that same FATAL transiently — repeatedly, for many
attempts, on a slow archiver — before it converges, so any earlier trip
would wrongly reject a recoverable restore (observed twice in CI). The
FATAL is used only to *explain* a failure the deadline has already
declared.
- **Health wins unconditionally** — the deadline/classification check
precedes the healthy check, but a recovery that converges within the
window reaches success and is never classified as failed (a live dev7
restore did exactly this).
- **Counting distinct FATAL-ing attempts** (not pod failures) tolerates
a one-off archive-lag miss on a near-now target and ignores transient
recovery-pod crashes (a node blip / brief API-server unreachability
never emits that FATAL).
- Adds the `pods/log` RBAC grant the log read needs (distinct
subresource from `pods`); a `Forbidden` is logged loudly and surfaced in
the failure message rather than silently disabling the classification.
- **e2e** (`examples/backups/postgres/run-all.sh`, wired into the
Chainsaw `postgres-2-backup-roundtrip` suite):
- Step 45 — write `before`, capture server timestamp (µs precision),
write `after`, restore to that timestamp, assert `before` present and
`after` absent.
- Step 46 — a `recoveryTime` an hour ahead must fail with reason
`RecoveryTargetUnreachable` once the deadline elapses (asserts the
*reason*, so it proves the classification fired, not just that a timeout
occurred).
- Chainsaw budget raised 25m → 40m to cover three sequential restores +
the post-deadline classification.
- **Docs + example**: uncommented `45-restorejob-pitr.yaml`; a
`docs/operations/backup-classes.md` PITR section (recoverable window,
discovering earliest/latest restorable time from the backup catalog
since `firstRecoverabilityPoint` is unreliable under the plugin,
archive-lag tolerance, idempotency under GitOps).

### Verification

- `go build` / `go vet` clean; unit tests pass
(`logIndicatesRecoveryTargetUnreachable`, `recoveryUnreachableFromLogs`,
`recoveryPodsToInspect`, nil-Clientset guard, a reconcile-level
success-path test asserting `Succeeded` + `RecoveryConverged=True`, and
— added per review —
`TestReconcileCNPGRestore_DeadlineClassifiesRecoveryTargetUnreachable`,
a reconcile-level test driving the deadline branch through all three
sub-paths: FATAL found → reason `RecoveryTargetUnreachable`, FATAL
absent → generic timeout, `Forbidden` log read → generic timeout +
missing-`pods/log`-RBAC note. It fails if the call-site reason is
reverted to the generic one).
- Reviewed via adversarial branch-review; fixes include a critical
missing-RBAC dead-on-arrival, a classification false-positive on
transient recovery-pod crashes, and a sub-second e2e flake.
- **dev7**: backup + WAL archiving + recovery convergence verified live
on the #3313 build. The load-bearing assumption of the classification —
CNPG **accumulates** failed `full-recovery` pods rather than reusing one
— was confirmed (observed 7 simultaneous failed recovery pods). A
fully-green live step-46 run was blocked by intermittent COSI
bucket-provisioning flakiness on dev7; the negative path is covered by
the CI Chainsaw e2e.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff. The only user-facing surface is the new
`docs/operations/backup-classes.md` PITR section; the website keeps a
hand-mirrored copy per docs version.

- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: PITR docs mirror warranted (not yet opened; same pending
mirror as #3313's backup-classes changes)
- [x] No other downstream repository is affected by this change

```release-note
feat(backups): PostgreSQL RestoreJob now supports point-in-time recovery (PITR). Set spec.options.recoveryTime (RFC3339) to restore to an exact instant. A recoveryTime past the latest archived WAL still runs to the restore deadline (spec.options.restoreTimeoutSeconds, default 30m), but the failure is now reported with the precise reason RecoveryTargetUnreachable and an actionable message instead of a generic timeout; set a shorter restoreTimeoutSeconds for a faster rejection.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added PostgreSQL point-in-time recovery (PITR) support via RFC3339
`recoveryTime`.
* Introduced a PITR RestoreJob example, plus helper variables and an
unreachable-target scenario.
* **Bug Fixes**
* Improved restore deadline handling with deterministic, log-based
fail-fast classification for unreachable recovery targets, including
clearer conditions.
* **Documentation**
* Added detailed PITR operational documentation (recovery windows,
failure behavior, and GitOps/idempotent behavior).
* Expanded README and example comments for PITR verification and failure
expectations.
* **Tests**
* Extended unit tests and end-to-end chainsaw workflow to cover PITR
success and unreachable failures.
* **Chores**
* Updated RBAC to allow pod log reads; refined cleanup and run scripts
for additional RestoreJob artifacts.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-23 14:14:58 +04:00
myasnikovdaniil d3815a4ce6 Merge branch 'main' into release-1.6.0
Signed-off-by: myasnikovdaniil <60174387+myasnikovdaniil@users.noreply.github.com>
2026-07-22 18:12:04 +05:00
cozystack-ci[bot] 235d38dae1 Prepare release v1.6.0 (promoted from v1.6.0-rc.4)
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
2026-07-22 13:00:10 +00:00
Myasnikov Daniil 284a17ba1f fix(release): unblock the v1.6.0 promote on the rc.4 staging tree
Promoting this branch fails in two independent ways, both landing after
finalize has already created the write-once stable tag, published the
GitHub release and moved :latest.

chBackupClientImage had no producer and froze at v1.4.0-rc.2 while
.migrations.image advanced to v1.6.0-rc.4, leaving platform-migrations
pinned at two digests. promote-retag.sh copies every collected
<repo>@<digest> to <repo>:<stable>, so both compete for one destination
tag: the lower-sorting v1.4.0-rc.2 digest wins it and the second copy
trips the write-once guard, aborting the retag with five repositories
still untagged and the installer chart unpublished. Align it to the
digest this branch already carries for the migrations image. Both come
from the same tree, so the runner keeps the curl, jq and ca-certificates
the Altinity strategy Pod needs, and that Pod overrides the ENTRYPOINT
with its own command.

promote-rc.yaml on main now calls hack/promote-rewrite-tags.sh, which
does not exist here, so the dispatch would fail outright. Vendor it and
hack/lib/image-refs.sh verbatim from main (identical blobs). That also
fixes the rewrite's coverage: the old inline glob enumerated 155 files
and missed 10 rc-bearing ones, which is why the earlier promote left 11
references reading v1.6.0-rc.4. The shared enumeration covers all 168
and fails the promotion if any rc reference survives.

Both fixes are on main via #3405 and #3404; this carries them onto the
rc.4 staging tree so v1.6.0 ships the bytes rc.4 tested.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-22 16:48:32 +05:00
Myasnikov Daniil a302bd0ddc docs(backupstrategy-controller): reframe the migrations-image reuse as a workaround
Running the platform migrations image as the ClickHouse backup client is
not a design decision worth defending, and the in-tree comments read as
though it were. Reframe them.

The strategy script uses exactly curl, jq, sleep and date. For that it
pulls a 349 MB image carrying kubectl, helm, git, cozyhr, the
etcd-migrate binary and every migration script into a tenant-adjacent
Pod, and that image's ENTRYPOINT is run-migrations.sh -- so the
`command:` override in the strategy template is load-bearing rather than
incidental. A dedicated alpine + curl + jq + ca-certificates image
measures 23 MB.

The standing objection to a dedicated image -- "a second tag release CI
would have to re-digest each cut" -- does not survive contact with the
actual defect. Every other first-party image already pays that cost
automatically. What broke was a pin with NO producer, which an image
owned by its own package cannot reproduce; that fix would also delete
the cross-package stamp introduced here.

Comment-only. No rendered output changes.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-22 12:29:15 +05:00
Andrey KolkovandClaude Opus 4.8 6a1e2d0d06 feat(backups): support PostgreSQL point-in-time recovery (PITR)
Makes RestoreJob.spec.options.recoveryTime a genuinely supported, tested
capability for the new backups.cozystack.io API (issue #2774). The
recoveryTime -> CNPG bootstrap.recovery.recoveryTarget.targetTime plumbing
already existed on the barman-cloud-plugin branch; this adds the reliability
and verification layer the issue asked for:

- Fail-fast for an unreachable target: when a RestoreJob has recoveryTime set
  and the recovery Cluster is not healthy, the driver reads the CNPG
  full-recovery pod logs and, once several distinct attempts have exited with
  PostgreSQL's "recovery ended before configured recovery target was reached"
  FATAL, fails the RestoreJob with conditions RecoveryConverged=False /
  Ready=False (reason RecoveryTargetUnreachable) instead of hanging to the
  restore deadline. Health is checked first, so a converged recovery always
  wins; counting distinct FATAL-ing attempts (not pod failures) tolerates a
  transient archive-lag miss on a near-now target and ignores transient
  recovery-pod crashes. Adds the pods/log RBAC grant the log read requires.

- e2e (examples/backups/postgres/run-all.sh, wired into the Chainsaw
  postgres-2-backup-roundtrip suite): step 45 writes a before marker, captures
  the server timestamp, writes an after marker, restores to that timestamp and
  asserts the before row survived and the after row did not; step 46 submits a
  recoveryTime an hour ahead and asserts it fails fast with reason
  RecoveryTargetUnreachable.

- Working, uncommented example (45-restorejob-pitr.yaml) and a docs section
  (docs/operations/backup-classes.md) covering the recoverable window, how to
  discover the earliest/latest restorable time from the backup catalog, the
  archive-lag tolerance, and idempotency under GitOps.

Refs #2774

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-22 10:40:18 +04:00
Myasnikov Daniil 427ed68761 fix(backupstrategy-controller): give chBackupClientImage a producer
platform-migrations was pinned at two different digests: v1.5.0 in
packages/core/platform/values.yaml, stamped by its Makefile, and
v1.4.0-rc.2 as .backupStrategyController.chBackupClientImage, stamped by
nothing at all. The second is the Altinity strategy Pod's curl+jq runner,
deliberately reusing the migrations image rather than shipping a second
one-binary tag that release CI would have to re-digest every cut. That
reuse only holds while the two move together, and the values.yaml comment
said so -- but a comment was the only thing enforcing it, so the copy with
no producer froze two minors back.

This is a release blocker, not just staleness. Promotion retags by digest,
copying every collected <repo>@<digest> to <repo>:<stable>. Two digests
under one repository produce two copies competing for one destination tag:
the first wins and the second hits the write-once guard in
hack/promote-retag.sh, which fails the promotion. A dry run against the
tree emitted exactly that pair. v1.6.0 is the first release cut through
promotion rather than a full rebuild, so it would have been the first to
hit it -- every earlier release ran `make build`, which restamped both
copies as a side effect and hid the missing producer.

packages/core/platform/Makefile now stamps both keys with the same ref, so
they are in lockstep by construction rather than by instruction, and the
committed value is aligned to the digest core/platform already carried.
The image is used only as a curl + jq + ca-certificates runner, so moving
it forward two minors changes no behaviour the strategy depends on; it
renders and the chart's helm-unittest suite passes.

hack/image-pin-consistency.bats adds the general guard -- no repository
pinned at more than one digest, driven through the real promotion selector
so it tracks whatever set promotion acts on -- plus a specific lockstep
assertion so a recurrence names its cause. The general check could not be
added before now: it fails on the very duplicate this commit removes.

Relates to #3143 (the unowned-pin class) and unblocks the v1.6.0 promote.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-21 23:15:46 +05:00
myasnikovdaniil 960d853706 feat(api): typed waitStrategy/healthCheckExprs for kstatus readiness (#2642) (#3273)
## What

Adds typed `waitStrategy` + `healthCheckExprs` fields so a package/app
can declare the Flux helm-controller v1.5.0 kstatus health gate for the
custom resource it wraps — making `HelmRelease.Ready` mean "the CR is
actually healthy" instead of "helm applied the CR". This is the
operator/apiserver foundation of the kstatus epic (#2642); the API
surface was reviewed and approved by the API owner in #3263.

Commits (per the review's "isolated step-0" request):
1. `build(deps)` — the dependency bump (see below).
2. `feat(api)` — the fields, plumbing, tests, and regenerated code.
3. `fix(api)` — match the openapi code-generator pin to the 0.35 runtime
(fixes an aggregated-apiserver crash the bump surfaced).
4. `fix(api)` — declare `OpenAPIModelName` on the apps types so SSA
resolves catalog kinds.
5. `docs(api)` — note that `disableWait` silently defeats
`healthCheckExprs`.

## API surface

Typed fields on **both** HelmRelease-building paths:
- `ApplicationDefinition.spec.release` (apiserver / catalog apps →
`convertApplicationToHelmRelease`)
- `ComponentInstall` (operator / platform packages →
`buildHelmReleaseSpec`)

```go
// WaitStrategy maps to HelmReleaseSpec.WaitStrategy.Name — a deliberate scalar
// simplification of the upstream {name} object. One of poller|legacy.
WaitStrategy string
// HealthCheckExprs maps to HelmReleaseSpec.HealthCheckExprs (the upstream
// kustomize.CustomHealthCheck type verbatim).
HealthCheckExprs []kustomize.CustomHealthCheck
```

`config.ResolveWaitStrategy` is shared by both builders and **couples
the default**: `healthCheckExprs` are only evaluated under the `poller`
strategy, so when expressions are set and no strategy is given, the
generated HelmRelease defaults to `poller`. A package that sets only
`healthCheckExprs` is therefore self-contained, independent of the
controller's global default.

## The dependency bump (commit 1)

The `WaitStrategy` / `kustomize.CustomHealthCheck` /
`HelmReleaseSpec.HealthCheckExprs` types do not exist below
`helm-controller/api` v1.5.0 — the runtime is already v1.5.0 (shipped by
#2602), so this closes a Go-types-vs-runtime skew. It is **not** an
isolated bump: v1.5.x requires **k8s 0.35 + controller-runtime 0.23**,
cascading the k8s stack 0.34 → 0.35 (and the go directive to 1.26).

cozystack replaces `k8s.io/apimachinery` with a fork carrying the
still-open upstream `ConvertToVersion` patch (k/k#135537). That single
commit rebases cleanly onto apimachinery v0.35.0; published at
`cozystack/apimachinery@release-1.35-pr135537`, and `go.mod` pins its
pseudo-version. **The repo builds with zero source changes under the new
stack.**

> Maintainer note: please confirm the `release-1.35-pr135537` fork
branch naming, or fold it into the `cozystack` branch as you prefer —
the pin can be updated to match.

## Not included (conscious YAGNI)

`disableWait` is intentionally omitted. The only known need (the
kubernetes tenant-addon deadlock) was resolved child-side, and
`waitStrategy: legacy` is not equivalent (legacy still waits
Helm-v3-style; `disableWait` skips waiting entirely). Add it later when
a concrete leaf needs a true no-gate escape hatch.

## Tests

- `config.TestResolveWaitStrategy` — the couple-the-default logic
(unset+exprs→poller; explicit legacy honored; unset+no-exprs→nil).
- Operator + apiserver cases mirror each other
(`TestBuildHelmReleaseSpecHealthCheckExprs` /
`TestConvertApplicationToHelmRelease_HealthCheckExprs`).
- `go build ./...` clean, `go vet` clean, `gofmt` clean, codegen
regenerated and committed.

The 4 backupstrategy CRDs regenerate with benign k8s-0.35 PodSpec
description drift (DynamicResourceAllocation alpha→stable, toleration
Lt/Gt, resizePolicy note) — no functional change.

## Follow-up

Phase 3 (per-leaf adoption — postgres/kafka/mongodb/etc. declaring their
CEL via these fields) builds on this. The full CRD→CEL catalog is
maintained in the epic's readiness audit.

Refs #2642

```release-note
feat(api): add `waitStrategy` and `healthCheckExprs` to ApplicationDefinition and ComponentInstall, so a HelmRelease reports Ready only once the custom resource it wraps is actually healthy
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added configurable CEL health-check expressions for application
releases and package components.
* Added `waitStrategy` support (`poller` and `legacy`) with automatic
`poller` selection when health checks are configured and no strategy is
set.
* Propagated these settings into generated Helm releases, affecting
readiness/wait behavior.

* **Documentation**
* Updated Application and PackageSource schemas to include
`healthCheckExprs` and `waitStrategy`.
* Extended related Kubernetes CRD documentation, including projected pod
certificate annotations and workload reference fields.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-21 17:02:37 +05:00
cozystack-ci[bot] d95685c427 Prepare release v1.6.0-rc.4
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
2026-07-21 10:31:23 +00:00
Andrey KolkovandClaude Opus 4.8 2212367842 fix(backups): own driver-created ObjectStore + correct stale barman references
Review follow-ups (cozystack#3300):

- Lifecycle/leak: applyClusterPluginBackup now sets an ownerReference from the
  driver-created barmancloud.cnpg.io/ObjectStore to its cnpg.io/Cluster, so
  Kubernetes garbage-collects it when the Cluster is deleted (tenant app
  teardown or the RestoreJob purge). In the platform useSystemBucket flow the
  chart does not render this ObjectStore, so previously it was orphaned in the
  tenant namespace. Drop the now-accurate-only 'delete' verb on objectstores
  from the driver RBAC (GC handles removal; the driver never deletes it) and
  correct the RBAC comment that falsely claimed the purge deletes it.
- Diagnostics/comments: the RestoreJob WAL-archive-deadline failure message and
  several doc comments still told operators to check spec.backup.barmanObjectStore
  / externalClusters[].barmanObjectStore, fields no longer emitted after the
  migration. Point them at the barman-cloud plugin (spec.plugins + ObjectStore,
  externalClusters[].plugin) instead so they don't mislead during an incident.
- Test: TestApplyClusterPluginBackup_PatchesExistingCluster now asserts the
  ObjectStore carries an ownerReference to the Cluster (kind/name/uid).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-16 21:44:41 +04:00
Andrey Kolkov 13ff340277 fix(backups): migrate CNPG backups to the barman-cloud plugin
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-16 21:44:41 +04:00
Myasnikov Daniil 96a9652fb1 feat(api): typed waitStrategy/healthCheckExprs for kstatus readiness
Add first-class fields so a package/app can declare the Flux helm-controller
v1.5.0 kstatus health gate for the custom resource it wraps, so
HelmRelease.Ready means "the CR is actually healthy" instead of "helm applied
the CR" (#2642). API-owner approved in #3263.

- api: WaitStrategy (scalar, maps to HelmReleaseSpec.WaitStrategy.Name — a
  deliberate simplification) + HealthCheckExprs ([]kustomize.CustomHealthCheck,
  the upstream type verbatim) on both ApplicationDefinition.spec.release (the
  apiserver path) and ComponentInstall (the operator path).
- config.ResolveWaitStrategy: shared by both HelmRelease builders. Couples the
  default — healthCheckExprs imply the poller strategy when none is set, since
  exprs are only evaluated under poller. This removes the "must also set
  waitStrategy: poller" footgun and does not depend on the controller's global
  default.
- thread through buildHelmReleaseSpec (operator) and
  convertApplicationToHelmRelease (apiserver); the apiserver loads the typed
  spec.release fields in start.go.
- regenerate deepcopy, the ApplicationDefinition and PackageSource CRDs, and
  openapi. The backupstrategy CRDs also regenerate — benign k8s-0.35 PodSpec
  description drift, not a functional change.

disableWait is intentionally not added (YAGNI): the only known need (the
kubernetes tenant-addon deadlock) was resolved child-side, and waitStrategy:
legacy is not equivalent (legacy still waits Helm-v3-style; disableWait skips
waiting entirely). Add it later when a concrete leaf needs a true no-gate
escape hatch.

Refs #2642

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-16 16:45:16 +05:00
Myasnikov DaniilandClaude Opus 4.8 4af5a82796 fix(etcd-operator,backups): address review nits — trim endpoint + hook resources
- backupstrategy-controller: trim() the decoded S3 endpoint before scheme
  stripping so a trailing newline in the Secret can't yield a malformed URL
- etcd-operator: give the pre-upgrade selector-fix hook container resource
  requests/limits (matches postgres-operator webhook-ready), +unittest

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-15 19:11:36 +05:00
Myasnikov DaniilandClaude Opus 4.8 e160a0dc87 fix(backups): derive default S3 endpoint from the provisioned bucket
The default backupStorage.endpoint is http://seaweedfs-s3...svc:8333, but
Cozystack ships SeaweedFS with global.seaweedfs.enableSecurity=true, so its
in-cluster S3 serves TLS on :8333 behind the self-signed "SeaweedFS CA".
Every cozy-default backup (etcd/mariadb/velero/fdb/cnpg) therefore hits a TLS
listener over plaintext and fails the handshake. This is fatal for the etcd
v1alpha2 adoption migration: its mandatory pre-upgrade safety snapshot writes
to this endpoint, so migration 50 hard-fails and blocks the 1.5->1.6 upgrade
on any cluster with a legacy etcd. The Etcd Strategy S3 schema has no
caCert/insecureSkipVerify field, so it cannot target the self-signed
in-cluster endpoint at all.

Add a "backupstrategy-controller.endpoint" helper that, for a provisioned
bucket, resolves the endpoint from the COSI bucket's system credentials
Secret (backupStorage.systemSecretName) — the external S3 ingress with an
ACME cert, the same trusted endpoint COSI advertises and every backup
operator can verify — and forces the https:// scheme. All Strategy CRs, the
Velero BackupStorageLocation and the controller Deployment env are routed
through it. It falls back to .Values.backupStorage.endpoint for external S3
(provisionBucket=false) and for offline `helm template`/unit renders and the
pre-reconcile first install, where the Secret lookup returns nothing (Flux
re-renders on spec.interval once the Secret exists).

helm-unittest extended to assert every consumer picks up the resolved
endpoint (CNPG/Etcd/Velero keep the full URL, MariaDB/FDB strip the scheme
and derive their secure flag) plus the external-S3 verbatim fallback.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-15 19:11:36 +05:00
renovate[bot] 5b9b420ffe chore(deps): update container image & action digests 2026-07-10 05:29:23 +00:00
Andrey Kolkov cc5d688e15 feat(platform): migrate to etcd-operator v1alpha2
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-06 14:41:33 +04:00
cozystack-ci[bot] e1aaa90ed3 Prepare release v1.5.0
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
2026-06-22 10:55:37 +00:00
Myasnikov DaniilandClaude Opus 4.8 d3958c5bcb refactor(build): mode=max registry cache with a main-only warmer (#2938)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-18 10:40:05 +05:00
Myasnikov Daniil 9868c43e53 fix(build): read --cache-from from ghcr :latest so PR builds warm-start
PR #2711 switched PR builds to push only :pr-<N>-<sha> tags (PUBLISH_FLOATING=0) to stop concurrent-push 409 conflicts, but left --cache-from pointing at $(REGISTRY)/<img>:latest. PR builds set REGISTRY to the per-CI registry (iad.ocir.io) and never publish :latest there, so every PR build's registry cache lookup 404s and rebuilds cold.

Add a CACHE_REGISTRY knob (default ghcr.io/cozystack/cozystack, where release builds do publish :latest) and point every --cache-from at it. Push targets ($(REGISTRY)) and the unique per-PR tags are unchanged, so the #2711 anti-conflict behaviour is preserved while PR builds regain a warm cache from the last release.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-09 14:24:36 +05:00
Andrey Kolkov f17ead3d26 feat(platform): add default backupclass
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-05-28 13:49:59 +04:00