Commit Graph
53 Commits
Author SHA1 Message Date
Andrey Kolkov 8aeedde088 feat(backups): support PostgreSQL point-in-time recovery (PITR) (#3383)
## What this PR does

Makes PostgreSQL point-in-time recovery (PITR) a genuinely supported,
tested capability for the new `backups.cozystack.io` API. Fixes #2774.

Was **stacked on #3313** (`fix/3300-cnpg-barman-plugin`) for the
`recoveryTime` → CNPG `bootstrap.recovery.recoveryTarget.targetTime`
plumbing; #3313 has since merged and this branch is rebased on `main`.
This PR adds the reliability + verification layer #2774 asked for.

### Changes

- **Precise failure reason for an unreachable target.** The restore
deadline (`spec.options.restoreTimeoutSeconds`, default 30m) stays the
sole authority for "this restore is stuck" — the client waits the full
window either way. What changes is the *diagnosis*: once the deadline
has elapsed, if `recoveryTime` was set and the recovery Cluster never
went healthy, the CNPG driver reads the `full-recovery` pod logs and —
when several distinct attempts exited with PostgreSQL's `FATAL: recovery
ended before configured recovery target was reached` — fails the
RestoreJob with `RecoveryConverged=False` / `Ready=False` (reason
`RecoveryTargetUnreachable`) and an actionable message, instead of a
bare generic timeout. Timing is unchanged; only the reason/message
differ. (For a faster rejection, set a shorter `restoreTimeoutSeconds`.)
Design points:
- **We deliberately do NOT trip early off the FATAL.** A valid near-now
target emits that same FATAL transiently — repeatedly, for many
attempts, on a slow archiver — before it converges, so any earlier trip
would wrongly reject a recoverable restore (observed twice in CI). The
FATAL is used only to *explain* a failure the deadline has already
declared.
- **Health wins unconditionally** — the deadline/classification check
precedes the healthy check, but a recovery that converges within the
window reaches success and is never classified as failed (a live dev7
restore did exactly this).
- **Counting distinct FATAL-ing attempts** (not pod failures) tolerates
a one-off archive-lag miss on a near-now target and ignores transient
recovery-pod crashes (a node blip / brief API-server unreachability
never emits that FATAL).
- Adds the `pods/log` RBAC grant the log read needs (distinct
subresource from `pods`); a `Forbidden` is logged loudly and surfaced in
the failure message rather than silently disabling the classification.
- **e2e** (`examples/backups/postgres/run-all.sh`, wired into the
Chainsaw `postgres-2-backup-roundtrip` suite):
- Step 45 — write `before`, capture server timestamp (µs precision),
write `after`, restore to that timestamp, assert `before` present and
`after` absent.
- Step 46 — a `recoveryTime` an hour ahead must fail with reason
`RecoveryTargetUnreachable` once the deadline elapses (asserts the
*reason*, so it proves the classification fired, not just that a timeout
occurred).
- Chainsaw budget raised 25m → 40m to cover three sequential restores +
the post-deadline classification.
- **Docs + example**: uncommented `45-restorejob-pitr.yaml`; a
`docs/operations/backup-classes.md` PITR section (recoverable window,
discovering earliest/latest restorable time from the backup catalog
since `firstRecoverabilityPoint` is unreliable under the plugin,
archive-lag tolerance, idempotency under GitOps).

### Verification

- `go build` / `go vet` clean; unit tests pass
(`logIndicatesRecoveryTargetUnreachable`, `recoveryUnreachableFromLogs`,
`recoveryPodsToInspect`, nil-Clientset guard, a reconcile-level
success-path test asserting `Succeeded` + `RecoveryConverged=True`, and
— added per review —
`TestReconcileCNPGRestore_DeadlineClassifiesRecoveryTargetUnreachable`,
a reconcile-level test driving the deadline branch through all three
sub-paths: FATAL found → reason `RecoveryTargetUnreachable`, FATAL
absent → generic timeout, `Forbidden` log read → generic timeout +
missing-`pods/log`-RBAC note. It fails if the call-site reason is
reverted to the generic one).
- Reviewed via adversarial branch-review; fixes include a critical
missing-RBAC dead-on-arrival, a classification false-positive on
transient recovery-pod crashes, and a sub-second e2e flake.
- **dev7**: backup + WAL archiving + recovery convergence verified live
on the #3313 build. The load-bearing assumption of the classification —
CNPG **accumulates** failed `full-recovery` pods rather than reusing one
— was confirmed (observed 7 simultaneous failed recovery pods). A
fully-green live step-46 run was blocked by intermittent COSI
bucket-provisioning flakiness on dev7; the negative path is covered by
the CI Chainsaw e2e.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff. The only user-facing surface is the new
`docs/operations/backup-classes.md` PITR section; the website keeps a
hand-mirrored copy per docs version.

- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: PITR docs mirror warranted (not yet opened; same pending
mirror as #3313's backup-classes changes)
- [x] No other downstream repository is affected by this change

```release-note
feat(backups): PostgreSQL RestoreJob now supports point-in-time recovery (PITR). Set spec.options.recoveryTime (RFC3339) to restore to an exact instant. A recoveryTime past the latest archived WAL still runs to the restore deadline (spec.options.restoreTimeoutSeconds, default 30m), but the failure is now reported with the precise reason RecoveryTargetUnreachable and an actionable message instead of a generic timeout; set a shorter restoreTimeoutSeconds for a faster rejection.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added PostgreSQL point-in-time recovery (PITR) support via RFC3339
`recoveryTime`.
* Introduced a PITR RestoreJob example, plus helper variables and an
unreachable-target scenario.
* **Bug Fixes**
* Improved restore deadline handling with deterministic, log-based
fail-fast classification for unreachable recovery targets, including
clearer conditions.
* **Documentation**
* Added detailed PITR operational documentation (recovery windows,
failure behavior, and GitOps/idempotent behavior).
* Expanded README and example comments for PITR verification and failure
expectations.
* **Tests**
* Extended unit tests and end-to-end chainsaw workflow to cover PITR
success and unreachable failures.
* **Chores**
* Updated RBAC to allow pod log reads; refined cleanup and run scripts
for additional RestoreJob artifacts.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-23 14:14:58 +04:00
Andrey KolkovandClaude Opus 4.8 b574e2176f fix(backups): cover Go-driven system-bucket ObjectStore + share the checksum helper
Address review on the S3 request-checksum fix:

- Extend the fix to the platform-managed useSystemBucket=true flow, whose
  ObjectStore is built in Go by applyClusterPluginBackup (SSA-applied), not
  chart-rendered. Add InstanceSidecarConfiguration to cnpgtypes.ObjectStoreSpec
  and set AWS_REQUEST_CHECKSUM_CALCULATION=when_required on it via
  barmanSidecarConfiguration(). The platform's own default system bucket is
  SeaweedFS, a non-AWS S3 gateway of the same class as Ceph RGW. Unit test in
  cnpgstrategy_controller_test.go asserts the applied ObjectStore carries it.

- Factor the duplicated instanceSidecarConfiguration block (3x across two
  charts) into a shared cozy-lib helper (cozy-lib.barman.checksumSidecarConfiguration).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
2026-07-22 15:35:32 +04:00
Andrey Kolkov ad3f8b2487 fix(backups): check cluster health before the restore deadline (PITR)
Addresses review CHANGES_REQUESTED on #3383.

CRITICAL: reconcileCNPGRestore checked the restore deadline BEFORE the
cluster-health check, so a reconcile that first observes "deadline elapsed"
AFTER the recovery Cluster already reached a healthy state (a controller
restart or stalled workqueue spanning restoreTimeoutSeconds) would mark an
already-converged restore Failed / RecoveryTargetUnreachable. A false Failed
can then escalate to data loss: a resubmitted RestoreJob's purge guard treats
the recovered Cluster as a stale leftover and deletes it + its PVCs.

Fix: check hasRecovery + cnpgClusterHealthy first and return Succeeded
unconditionally when healthy, regardless of how late the reconcile runs; only
a not-yet-healthy restore reaches the deadline/classification branch. Adds
TestReconcileCNPGRestore_HealthyPastDeadlineSucceeds (StartedAt 72h past the
deadline + healthy Cluster + recoveryTime set; the injected readPodLog fails
the test if the classification path is entered, proving health short-circuits).

MINOR: handle the deferred Close error on the pod-log reader (errcheck).
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-22 15:31:31 +04:00
Andrey KolkovandClaude Opus 4.8 b5d03f889b test(backups): reconcile-level test for the PITR deadline classification
Review of #3383 found the RecoveryTargetUnreachable classification branch in
reconcileCNPGRestore was only covered through the pure helpers and
recoveryTargetUnreachable directly, never through reconcileCNPGRestore itself —
a mutation swapping the call-site reason back to the generic RestoreFailed kept
the suite green.

Add TestReconcileCNPGRestore_DeadlineClassifiesRecoveryTargetUnreachable: it
backdates StartedAt past the restore deadline with a recoveryTime set and drives
the real reconcile through the readPodLog seam, asserting all three deadline
sub-paths — FATAL present (reason RecoveryTargetUnreachable on both
RecoveryConverged and Ready), FATAL absent (generic timeout, no
RecoveryTargetUnreachable), and a Forbidden log read (generic timeout + the
missing-pods/log-RBAC note). Verified it fails when the call-site reason is
reverted to the generic RestoreFailed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
2026-07-22 13:06:00 +04:00
Andrey KolkovandClaude Opus 4.8 6a1e2d0d06 feat(backups): support PostgreSQL point-in-time recovery (PITR)
Makes RestoreJob.spec.options.recoveryTime a genuinely supported, tested
capability for the new backups.cozystack.io API (issue #2774). The
recoveryTime -> CNPG bootstrap.recovery.recoveryTarget.targetTime plumbing
already existed on the barman-cloud-plugin branch; this adds the reliability
and verification layer the issue asked for:

- Fail-fast for an unreachable target: when a RestoreJob has recoveryTime set
  and the recovery Cluster is not healthy, the driver reads the CNPG
  full-recovery pod logs and, once several distinct attempts have exited with
  PostgreSQL's "recovery ended before configured recovery target was reached"
  FATAL, fails the RestoreJob with conditions RecoveryConverged=False /
  Ready=False (reason RecoveryTargetUnreachable) instead of hanging to the
  restore deadline. Health is checked first, so a converged recovery always
  wins; counting distinct FATAL-ing attempts (not pod failures) tolerates a
  transient archive-lag miss on a near-now target and ignores transient
  recovery-pod crashes. Adds the pods/log RBAC grant the log read requires.

- e2e (examples/backups/postgres/run-all.sh, wired into the Chainsaw
  postgres-2-backup-roundtrip suite): step 45 writes a before marker, captures
  the server timestamp, writes an after marker, restores to that timestamp and
  asserts the before row survived and the after row did not; step 46 submits a
  recoveryTime an hour ahead and asserts it fails fast with reason
  RecoveryTargetUnreachable.

- Working, uncommented example (45-restorejob-pitr.yaml) and a docs section
  (docs/operations/backup-classes.md) covering the recoverable window, how to
  discover the earliest/latest restorable time from the backup catalog, the
  archive-lag tolerance, and idempotency under GitOps.

Refs #2774

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-22 10:40:18 +04:00
Andrey KolkovandClaude Fable 5 67db8f7da7 fix(backups): make the barman-cloud serverName stable across every flow
Two defects of the same class — the WAL-archive S3 prefix (serverName)
changing or defaulting differently between components:

Recovery default regression (review blocker): native CNPG defaulted an
absent externalClusters serverName to the entry name (= bootstrap.oldName),
but the barman-cloud plugin defaults it to the NEW cluster's name, so a
legacy-flow restore with bootstrap.serverName unset would read the wrong
S3 prefix (hard failure, or silently restoring another cluster's data).
The chart now always emits serverName, defaulting to bootstrap.oldName.

Live-cluster serverName flip (found by the gated postgres backup e2e on a
real cluster): the chart renders spec.plugins without serverName
(effective prefix = cluster name) while a strategy template may name a
different one; the driver's ForceOwnership patch then flipped the prefix
mid-stream, splitting the WAL archive across two prefixes — the WAL
holding the backup's start was lost between them and the restore failed
with "WAL not found". The driver now preserves the live Cluster's
effective serverName (explicit parameter, or the cluster name when the
plugin is attached without one) and only applies the strategy's value to
a Cluster with no plugin attached; both charts pin serverName explicitly,
and the example strategy no longer sets one.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-16 21:44:41 +04:00
Andrey KolkovandClaude Opus 4.8 268ec29856 test(backups): assert driver-initiated Backup uses method=plugin
Review follow-up (cozystack#3300): ensureCNPGBackup stamps spec.method=plugin +
spec.pluginConfiguration on the postgresql.cnpg.io/Backup it creates — the field
that routes a driver-initiated run through the barman-cloud plugin. If it
regressed to the default (barmanObjectStore) method, CNPG would run the legacy
method against a Cluster that no longer has one and every platform backup would
fail. Add TestEnsureCNPGBackup_UsesPluginMethod asserting method, plugin name,
and cluster ref. (The chart ScheduledBackup path was already covered; the driver
path was not.)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-16 21:44:41 +04:00
Andrey KolkovandClaude Opus 4.8 2212367842 fix(backups): own driver-created ObjectStore + correct stale barman references
Review follow-ups (cozystack#3300):

- Lifecycle/leak: applyClusterPluginBackup now sets an ownerReference from the
  driver-created barmancloud.cnpg.io/ObjectStore to its cnpg.io/Cluster, so
  Kubernetes garbage-collects it when the Cluster is deleted (tenant app
  teardown or the RestoreJob purge). In the platform useSystemBucket flow the
  chart does not render this ObjectStore, so previously it was orphaned in the
  tenant namespace. Drop the now-accurate-only 'delete' verb on objectstores
  from the driver RBAC (GC handles removal; the driver never deletes it) and
  correct the RBAC comment that falsely claimed the purge deletes it.
- Diagnostics/comments: the RestoreJob WAL-archive-deadline failure message and
  several doc comments still told operators to check spec.backup.barmanObjectStore
  / externalClusters[].barmanObjectStore, fields no longer emitted after the
  migration. Point them at the barman-cloud plugin (spec.plugins + ObjectStore,
  externalClusters[].plugin) instead so they don't mislead during an incident.
- Test: TestApplyClusterPluginBackup_PatchesExistingCluster now asserts the
  ObjectStore carries an ownerReference to the Cluster (kind/name/uid).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-16 21:44:41 +04:00
Andrey Kolkov 13ff340277 fix(backups): migrate CNPG backups to the barman-cloud plugin
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-16 21:44:41 +04:00
Ivan Okhotnikov a8494d874f test(backups/postgres): reconcile-level repeat-restore regression + conservative freshness tie
Address review feedback on the repeat in-place restore fix:

- Make the freshness discriminator strict (creationTimestamp > StartedAt) so
  an exact tie resolves to "not fresh" and the caller purges, matching the
  function's stated conservative default; a fresh Cluster is always created
  well after StartedAt so this never misclassifies the status-write-race path.
- Document why a wall-clock timestamp comparison was chosen over recording the
  purged Cluster's UID on the TargetPurged condition (the UID approach leans on
  the same status write the race path assumes can fail).
- Add a reconcile-level regression test that drives reconcileCNPGRestore end to
  end with a fake client: a stale leftover recovery Cluster from a prior restore
  is purged, and a Cluster this restore just re-created is not re-purged. This
  covers the actual call site where #3311 lived, not just the pure helpers.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
2026-07-15 17:38:42 +03:00
Ivan Okhotnikov cf1b6ba28e fix(backups/postgres): purge stale recovery cluster on repeat in-place restore
A repeat in-place PostgreSQL RestoreJob into an already-restored target
silently no-opped: the destructive purge was skipped because the live
Cluster still carried spec.bootstrap.recovery left over from the previous
restore, and the re-purge guard mistook that for a freshly-recovered
Cluster produced by this restore's own purge. The job then reported
Succeeded while the PVC, disk, and data were never touched.

Tell the two apart by the Cluster's creationTimestamp relative to the
RestoreJob's StartedAt: only a recovery Cluster created at/after StartedAt
is treated as this restore's own and skipped; one that predates StartedAt
is leftover from an earlier completed restore and is purged so the restore
re-bootstraps from the backup.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
2026-07-15 16:59:01 +03:00
Myasnikov DaniilandClaude Fable 5 9297e51170 Merge origin/main into feat/e2e-chainsaw-example
Reconcile the Chainsaw e2e migration with ~500 commits of main drift:
- keep main's run-kubernetes.sh helpers (kubectl_wait_retry, tenant
  drain, LINSTOR pool-free wait, SC-fallback probe) under the branch's
  _lib/ rename with the Chainsaw finally/snapshot deltas re-applied
- resolve modify/delete on 10 drifted hack/e2e-apps/*.bats as deleted;
  their post-branch assertion drift is re-ported into the Chainsaw
  suites in follow-up commits
- drop the 4 new main-side bats (kubernetes-oidc-*, securitygroup,
  serviceexposure) from _lib/ relocation; ported as Chainsaw suites in
  follow-up commits
- docs/agents/e2e-testing.md: merge main's plan-job TIA wording with
  the Chainsaw suite phrasing

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-08 18:04:08 +05:00
Andrey Kolkov cc5d688e15 feat(platform): migrate to etcd-operator v1alpha2
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-06 14:41:33 +04:00
Myasnikov DaniilandClaude Opus 4.8 c82b55c621 docs(agents): align E2E guide and references with the Chainsaw migration
Rewrite docs/agents/e2e-testing.md for the Chainsaw suite: declarative
asserts replace the until-get/wait backstops, Chainsaw cleanup replaces
test-level EXIT traps, catch blocks replace the `|| { ...; false; }` idiom,
and the TIA/diagnostics sections describe suites + `make test-chainsaw
CHAINSAW_SUITES`. Move the BATS->Chainsaw bullet out of 'in-flight' since it
is now the standard.

Fix references the migration left stale: hack/e2e-apps/ -> hack/e2e-chainsaw/
in the gemini styleguide and the nats/etcd backup READMEs. The
backup-postgres.bats / backup-foundationdb.bats files cited by the postgres
examples, the postgres README, and the foundationdb controller comment never
existed -- drop the false 'CI e2e drives this' claims and describe those
backup flows as the manual reference flows they are.

Assisted-By: Claude <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-25 13:43:50 +05:00
Andrey Kolkov 1319fb78bf fix(backups): CI was broken on unittests
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-05-28 14:46:22 +04:00
Andrey Kolkov f17ead3d26 feat(platform): add default backupclass
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-05-28 13:49:59 +04:00
Timofei Larkin 72c4eb9c08 feat(backups): implement Job backup strategy (#1721)
## What

Implements the `strategy.backups.cozystack.io/Job` backup strategy — a
generic, application-agnostic driver that runs a user-supplied
Kubernetes Job to create (and restore) a backup.

## Background

This PR began as a draft written against an earlier shape of the backups
API. The API has since moved to a `BackupClass`-based dispatch model,
which broke the original draft on three counts:

- `BackupJob.Spec.StrategyRef` and `BackupJob.Spec.StorageRef` were
removed; the strategy and its parameters are now resolved from a
`BackupClass` into a `ResolvedBackupConfig`.
- Strategy handling moved from standalone per-strategy reconcilers to a
dispatch model: `BackupJobReconciler.Reconcile` /
`RestoreJobReconciler.Reconcile` resolve the strategy and switch into
per-strategy methods (`reconcileVelero`, `reconcileCNPG`,
`reconcileAltinity`, …).
- The draft's template context (`.Values`, `.Storage`) diverged from the
convention the other strategies adopted, and its `.Storage` half
targeted a storage API that does not exist.

## What changed

- **`reconcileJob` / `reconcileJobRestore`**
(`internal/backupcontroller/jobstrategy_controller.go`): replaces the
standalone `BackupJobStrategyReconciler` with the two dispatch methods
the `BackupJob` / `RestoreJob` reconcilers call. The Job strategy is the
generic counterpart of the Altinity driver — same lifecycle (render
`PodTemplateSpec` → run a one-shot `batch/v1.Job` → poll its terminal
condition → emit a `Backup`), minus the ClickHouse-specific
`applicationRef` gate, so it runs for any application. Shared package
helpers (`renderJobTemplate`, `jobConditionState`,
`getApplicationUnstructured`, status helpers, deterministic Job naming,
controller refs) are reused.
- **Template context** follows the in-tree convention used by the other
drivers: `.Application` (live application object), `.Release.Name` /
`.Release.Namespace`, `.Mode` (`backup` / `restore`), `.Parameters`
(resolved `BackupClass` parameters), and `.Backup` (restore runs only).
Strategy parameters round-trip through the `Backup`'s `driverMetadata`
so a later restore re-renders against the same values.
- **Restore** re-renders the same template with `.Mode == "restore"`.
Cross-namespace restore is unsupported (the RestoreJob target reference
is namespace-local), matching the Altinity driver.
- **`JobSpec.Template` doc + regenerated CRD**: the field doc previously
advertised `.Values`; it now documents the real context, and
`strategy.backups.cozystack.io_jobs.yaml` is regenerated to match (the
change is limited to that field's description).
- **Tests** (`jobstrategy_controller_test.go`): backup-Job creation /
args / labels / status, completion → `Backup` artifact with parameter
round-trip, Job-failure surfacing, missing-strategy terminal failure,
restore into the target namespace, restore completion, and the
parameter-prefix unit. A deliberately generic application kind pins that
the driver never gates on `applicationRef.kind`.

No RBAC changes are required: the controller already holds
`strategy.backups.cozystack.io/*`, `batch/jobs`, `apps.cozystack.io/*`,
and `backups` permissions.

## Notes for reviewers

- Backup and restore share one template, so a strategy author's
`PodTemplateSpec` must branch on `.Mode` to do anything
restore-specific. This matches the existing convention; worth confirming
it's the intended contract for the Job strategy's docs/examples.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Implemented full backup/restore job strategy with deterministic Job
creation, idempotent execution, and status/condition updates; added demo
example scripts and reusable shell helpers for a NATS JetStream
backup/restore workflow.

* **Documentation**
* Clarified Job template rendering context and listed supported template
variables; added README and step-by-step scenario docs for the NATS
example.

* **Tests**
* Added comprehensive unit tests covering job creation,
completion/failure handling, parameter extraction, and unmappable-target
failure cases.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/1721?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-05-27 14:45:19 +04:00
Timofei Larkin 5029af6d83 test(backups): harden Job strategy tests
Address review feedback on the Job backup strategy:

- Assert .Application is templated: the rendered backup Job now reads
  .Application.spec.replicas, the load-bearing context key that lets a
  generic backup tool read live app spec (previously only .Release,
  .Mode, .Parameters and .Backup had coverage).
- Pin failure-path messages so a reordered error path can't pass on the
  wrong branch: the missing-strategy test asserts the strategy name, and
  the unmappable-kind tests assert "kind not registered".
- Document why the full Status().Update in the success branches is safe
  (the object is freshly Get() each pass; the StartedAt write requeues).
- Note the fixed batch/v1.Job backoffLimit=2 and link the follow-up to
  make it configurable (#2742).

Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2026-05-26 17:25:42 +03:00
Aleksei Sviridkin d2e851f63b feat: Gateway API support via Cilium
Replace ingress-nginx as the default ingress layer for cozystack-native
services with Cilium Gateway API, materialised per-tenant via a new
TenantGateway CRD reconciled by cozystack-controller.

The extra/gateway chart renders one gateway.cozystack.io/v1alpha1
TenantGateway per tenant; the controller owns the resulting Gateway,
per-tenant Issuer, Certificate(s), and the http->https redirect
HTTPRoute. This avoids the helm-vs-controller race on
Gateway.spec.listeners when route-driven listeners are added.

publishing.certificates.solver selects HTTP-01 (default; per-app HTTPS
listener + per-app Certificate) or DNS-01 (opt-in; single wildcard
Certificate per apex; providers: cloudflare, route53, digitalocean,
rfc2136). HTTP-01 keeps zero-config per-app onboarding; DNS-01 saves
Let's Encrypt budget on apex-shared clusters.

Security model -- seven independent layers, all admission-time gates
fail-closed:

  1. Listener allowedRoutes namespace whitelist via the
     kube-apiserver-written kubernetes.io/metadata.name label;
     plain-HTTP listener is strictly narrower than HTTPS, and HTTPS /
     passthrough listeners restrict allowedRoutes.kinds to
     HTTPRoute / TLSRoute respectively.
  2. cozystack-gateway-hostname-policy VAP on Gateway.
  3. cozystack-gateway-attached-namespaces-policy VAP on Package --
     blocks tenant-* in gateway.attachedNamespaces.
  4. cozystack-tenant-host-policy VAP on Tenant -- blocks spec.host
     writes from non-trusted callers.
  5. cozystack-namespace-host-label-policy VAP on Namespace -- blocks
     writes to namespace.cozystack.io/host from non-trusted callers.
  6. Render-time fail in cozystack-basics for tenant-* in
     gateway.attachedNamespaces.
  7. cozystack-route-hostname-policy VAP on HTTPRoute / TLSRoute,
     scoped to tenant-* namespaces.

Within-apex cross-namespace hostname conflict is resolved at reconcile
time by a HostnameConflict condition (cozy-* namespace wins).

Five reconcile paths refuse to rewrite pre-existing objects with the
controller-derived name but no OwnerReference back to the
TenantGateway: Gateway, redirect HTTPRoute, per-tenant Issuer, wildcard
Certificate, per-listener Certificate.

Derived-apex tenants (tenants whose host is a subdomain of a parent
tenant's) auto-enable Gateway when the platform-level gateway.enabled
is on.

Bonus fix: pkg/registry/apps/application/rest.go was ignoring the
createValidation and deleteValidation callbacks that genericapiserver
hands to every storage method, silently bypassing admission for Create
and Delete on every apps.cozystack.io resource (Tenant, MariaDB,
Postgres, Kubernetes, ...). Aligned with the existing updateValidation
pattern.

Always-on changes: cilium.envoy.enabled + cilium.gatewayAPI.enabled
flip to true (extra cilium-envoy DaemonSet, ~100 MB RAM/node idle),
and the cozystack-api admission-chain fix. Everything else is gated
behind gateway.enabled=false and tenant.spec.gateway=false; existing
clusters see no behavioural change until an operator opts in.

Tests: 60+ Go cases in internal/controller/tenantgateway/, admission-
chain regression tests in pkg/registry/apps/application/, helm-unittest
across 7 charts (extra/gateway, core/platform, apps/tenant,
system/cozystack-basics, system/cert-manager-issuers,
system/cozystack-api, system/dashboard), and 14 e2e bats scenarios in
hack/e2e-apps/gateway.bats. New make test-controllers target wired
into pull-requests.yaml.

Deps: sigs.k8s.io/gateway-api v1.4.1, cert-manager v1.17.4.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-05-25 17:07:02 +03:00
Timofei Larkin 02911493f4 feat(backups): implement Job backup strategy
Complete the strategy.backups.cozystack.io/Job driver as the generic,
app-agnostic counterpart to the Altinity driver: render the strategy
PodTemplateSpec, run it as a one-shot batch/v1.Job, poll its terminal
condition, and emit a Backup. Wire it into the existing BackupClass
dispatch via reconcileJob/reconcileJobRestore rather than a standalone
reconciler, matching the post-BackupClass backups API (the old draft
read BackupJob.Spec.StrategyRef/StorageRef, which no longer exist).

Template context follows the in-tree convention (.Application, .Release,
.Mode, .Parameters, .Backup); the JobSpec.Template doc and regenerated
CRD are updated to match. Restore re-renders the same template with
.Mode=restore; cross-namespace restore is unsupported, mirroring
Altinity.

Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2026-05-22 16:46:42 +03:00
Andrey Kolkov d80f67e799 feat(platform): add implementation of backup-strategy controller for etcd
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-05-22 14:35:08 +04:00
Andrey Kolkov 84874353ce fix(platform): align field of Altinity backup-strategy for similarity with others
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-05-13 17:15:51 +04:00
Andrey Kolkov 92930ad1aa feat(platform): add implementation of backup-strategy controller for foundationdb
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-05-13 09:49:19 +04:00
Andrey Kolkov 62605a198b feat(platform): implement backup strategy for mariadb
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-05-12 13:56:29 +04:00
Andrey Kolkov 250e1b3376 feat(platform): implement backup strategy for clickhouse application
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-05-11 14:46:48 +04:00
Andrey Kolkov ed5ae16e36 feat(platform): implement backup strategy for postgres application
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-05-07 13:12:54 +04:00
Andrey Kolkov 2234824feb feat(backups): restore vmi to copy in another namespace
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-04-09 14:06:33 +04:00
Andrey Kolkov 16223dcffa fix(backup): abstract underlying resources by applicationRef.kind
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-04-01 21:39:47 +04:00
Andrey Kolkov 15d319bfa1 feat(backup): no manual actions required to perform restorejob for
VMInstance

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-03-31 12:01:04 +04:00
Andrei KvapilandClaude bce5300116 refactor: rename mysql application to mariadb
The mysql chart actually deploys MariaDB via mariadb-operator, but was
incorrectly named "mysql". Rename all references to use the correct
"mariadb" name across the codebase.

Changes:
- Rename packages/apps/mysql -> packages/apps/mariadb
- Rename packages/system/mysql-rd -> packages/system/mariadb-rd
- Rename platform source and bundle references
- Update CRD kind from MySQL to MariaDB
- Update RBAC, e2e tests, backup controller tests
- Keep real MySQL CLI/config tool names unchanged (mysqldump, [mysqld], etc.)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-02-12 15:15:20 +01:00
Timofei Larkin 000b5ff76c [backups]
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2026-02-04 15:50:23 +03:00
Timofei Larkin 75a4b8ecbd Merge branch 'feat/backup-work-consolidation' into feat/backup-recovery
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2026-02-03 21:20:59 +04:00
Andrey Kolkov d1d0627f0e Revert "fix about wrong strategy ref type"
This reverts commit 73d6e3013e.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-01-23 00:33:37 +04:00
Andrey Kolkov 73d6e3013e fix
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-01-22 14:00:17 +04:00
Andrey Kolkov 8755497869 fixes
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-01-22 13:55:00 +04:00
Andrey Kolkov ffde02c992 fix setup cache for backupClasses
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-01-22 12:59:42 +04:00
Andrey Kolkov f6641c1547 fixes
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-01-22 10:39:33 +04:00
Andrey Kolkov f254c5f03e fix test
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-01-22 10:39:33 +04:00
Andrey Kolkov e3aab24810 fixes
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-01-22 10:39:33 +04:00
Andrey Kolkov 02ace2e482 Simplify checks in tests
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-01-22 10:39:33 +04:00
Andrey Kolkov 79bd3ad0d5 fix templating
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-01-22 10:39:33 +04:00
Andrey Kolkov ba04063662 buildfix: restore against cyclic dependency
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-01-22 10:39:33 +04:00
Andrey Kolkov 5924c484c9 clean-up comments
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-01-22 10:39:33 +04:00
Andrey Kolkov d7ae3213ff removed unused
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-01-22 10:39:33 +04:00
Andrey Kolkov 41646b253e use everywhere NormalizeApplicationRef
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-01-22 10:39:33 +04:00
Andrey Kolkov 9e94a699a0 feat(backups): implement interface changes related backups class introduction
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-01-22 10:39:33 +04:00
Andrey Kolkov 5e15a75d89 [backups] add restore jobs controller
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-01-08 18:57:42 +03:00
Timofei Larkin f4228ffc20 [backups] Add templating of velero backups
## What this PR does

This patch narrows the scope of the Velero backup strategy controller to
simply template Velero Backups according to the application being backed
up and the template in the strategy. Creating storage locations is now
out of scope.

### Release note

```release-note
[backups] Implement templating for Velero backups and remove creation of
backup storage locations and volume snapshot locations.
```

Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2026-01-04 13:31:15 +03:00
Andrey Kolkov bfafcaa3ab [backups] Implement Velero strategy controller
## What this PR does

This patch implements the Reconcile function for BackupJobs with a
Velero strategy ref.

### Release note

```release-note
[backups] Implement the Velero backup strategy controller.
```

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-01-04 12:55:24 +03:00
Timofei Larkin 8cf68149cb [backups] Scaffold a Velero strategy
## What this PR does

This patch adds boilerplate code for a Velero strategy for the backups
API. Implementation can be added into the Reconcile function in a
subsequent PR.

###

```release-note
[backups] Add the boilerplate for a new Velero strategy for backups.
```

Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2025-12-26 10:14:29 +03:00