Commit Graph
20 Commits
Author SHA1 Message Date
Andrey Kolkov a410a3f48b feat(backups): Kafka topic-metadata backup via a dedicated strategy driver
Add a strategy.backups.cozystack.io/Kafka driver that backs up the topic
metadata of a Cozystack-managed Kafka application — partition counts,
replication factors and non-default topic configs — through the Kafka
Admin API, and restores it with kafka-topics --create + kafka-configs
--alter. Modelled on the Job/Rabbitmq strategies: the driver renders a
one-shot batch Job, gates on the Strimzi Kafka cluster reporting Ready,
and keys each backup by .BackupName so runs do not overwrite each other.

Scope is deliberately narrow: message payloads, consumer-group offsets,
ACLs, quotas and KafkaUsers are out of scope, and there is no PITR.
Restored topics are created directly (unmanaged on the target) — the
accepted tradeoff of an Admin-API-only driver. Both restore flows are
implemented: in-place recreates dropped topics into the source, to-copy
applies the source's topics onto a freshly-bootstrapped empty cluster.

Wire it into cozy-default (strategy-kafka-default.yaml gated on the
bucket name, a BackupClass binding, read RBAC on kafka.strimzi.io/kafkas
for the readiness gate, and the Strimzi Kafka client image), document it
in backup-classes.md, and add an honest round-trip example plus Chainsaw
e2e (examples/backups/kafka-metadata, kafka-3-metadata-roundtrip) that
proves a per-run retention.ms sentinel survives both restores.

Unit tests cover the driver's pure helpers (application-ref validation,
readiness verdict, parameter round-trip, per-run artifact-URI keying).

Not yet verified on a live cluster.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:20:27 +04:00
Andrey Kolkov b0eb104276 feat(backups): add Redis backup/restore strategy driver
The spotahome RedisFailover operator ships no backup CRD, so add a
dedicated strategy.backups.cozystack.io/Redis driver that runs a one-shot
Job: it dumps an RDB snapshot from the current master (discovered via the
operator's sentinel Service) to the system bucket over S3, and on restore
replays it (RDB -> RESP via redis-cli --pipe) into the target's master.

Structurally a specialisation of the Job/Altinity script-Job pattern with
a Redis applicationRef gate and an app-readiness precondition. Backup
parameters round-trip through the Backup's DriverMetadata (no
underlyingResources snapshot); restore drives from .Backup.ApplicationRef
so a to-copy restore reads the source's object-storage prefix while
writing into a differently-named target. Both in-place and to-copy are
supported; PITR is not (RDB is a discrete snapshot).

- api: Redis/RedisList strategy type + generated CRD and deepcopy
- driver: reconcileRedis / reconcileRedisRestore + pure helpers, wired
  into both dispatch switches, supportedBackupStrategyKinds and the no-op
  restore cleanup case
- default: gated cozy-default-redis strategy (system-bucket coords via
  cozy-backups-creds env secretKeyRef, https:// scheme + optional CA) and
  the Redis route in the cozy-default BackupClass
- image: images/redis-backup bundling redis-cli + curl + an RDB->RESP
  converter, wired via backupStrategyController.redisBackupClientImage
- tests: driver unit tests (gate, precondition, render, param round-trip,
  both reconcile paths) and the helm rendering-gate coverage
- example: examples/backups/redis honest marker round-trip for both
  restore flows

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-By: LLM
2026-09-08 17:56:20 +04:00
Andrey KolkovandClaude Opus 4.8 ba32df69c0 feat(backups): add RabbitMQ definitions backup via a dedicated strategy driver
RabbitMQ's cluster operator ships no backup CRD and RabbitMQ has no
consistent online export of message payloads, so back up broker
DEFINITIONS (vhosts, users, permissions, queues, exchanges, bindings,
policies, parameters) over the management HTTP API and store them in the
system bucket. Message payloads and PITR are out of scope - a full-data
backup is a Velero volume snapshot, a separate strategy.

Add a first-class strategy.backups.cozystack.io/Rabbitmq driver, following
the Altinity pattern (a dedicated driver that renders a one-shot Job for an
HTTP-API engine):

  - reconcileRabbitmq / reconcileRabbitmqRestore render the strategy's
    PodTemplateSpec (curl GET/POST /api/definitions, push/pull to S3 with
    --aws-sigv4) as a batch/v1.Job, for both in-place and to-copy restore.
  - A Go-read precondition gates each run on the RabbitmqCluster reporting
    AllReplicasReady (via the rabbitmqtypes partial type), surfacing a
    precise Ready=False wait instead of a Job that starts and fails.
  - The produced Backup records the object location on status.artifact.uri
    (rendered from spec.artifactURITemplate; the Backup CRD has no status
    subresource, so it persists on Create, as in the Etcd driver).

Wire dispatch in both controllers + supportedBackupStrategyKinds, register
rabbitmqtypes in the scheme, and grant read on rabbitmqclusters. Ship
cozy-default-rabbitmq (a $bucketName-gated Rabbitmq strategy) and route
apps.cozystack.io/RabbitMQ to it in the cozy-default BackupClass. The
application chart is untouched.

Add unit tests for the driver's pure helpers and the precondition gate, a
runnable example, and an honest-roundtrip Chainsaw e2e that seeds a
sentinel vhost, backs it up, drops it, and asserts it round-trips through
object storage on both restore flows.

The Job pod reuses the platform-migrations image (curl + ca-certificates)
under a new rabbitmqBackupClientImage key, stamped in lockstep with
chBackupClientImage so the repository stays pinned at one digest; a
dedicated slim image is a follow-up.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-01 18:21:10 +04:00
Andrey KolkovandClaude Opus 4.8 dff56aa998 feat(backups): add a native MongoDB (psmdb) backup-strategy driver
Adds a MongoDB (Percona Server for MongoDB / psmdb) backup-strategy driver so
`apps.cozystack.io/MongoDB` reaches feature parity with the CNPG, MariaDB and
ClickHouse native drivers — logical backups, restore-to-differently-named
instances, and point-in-time recovery through the unified
BackupJob/RestoreJob/BackupClass/Plan interface (closes #3545).

The driver orchestrates the operator's PerconaServerMongoDBBackup /
PerconaServerMongoDBRestore CRDs:

- BackupJob: creates a one-shot PerconaServerMongoDBBackup, polls
  status.state, and persists the S3 destination + storage descriptor
  (by-reference, no credentials) onto the Cozystack Backup artifact.
- RestoreJob: drives PerconaServerMongoDBRestore via backupSource (read back
  from the source backup status, snapshot fallback) so the same artifact
  restores in-place or into a differently-named instance; PITR via
  spec.options.pitr (date/latest).

Because psmdb resolves the backup storage by name against the cluster's
spec.backup.storages, the driver names the chart's storage (s3-storage) via the
cozy-default-mongodb Strategy rather than injecting it, and surfaces a clear
Ready=False precondition when a target has backups disabled.

Wiring:
- api/backups/strategy/v1alpha1: MongoDB strategy type + generated CRD/deepcopy.
- internal/backupcontroller: psmdbstrategy_controller + psmdbtypes/mongodbapp
  partial-schema mirrors; dispatch, supported-kinds and scheme registration.
- backupstrategy-controller: cozy-default MongoDB route, cozy-default-mongodb
  Strategy CR, and psmdb.percona.com RBAC.
- apps/mongodb: backup.insecureSkipTLSVerify knob (psmdb s3 storage has no
  CA-bundle field) so backups can target Cozystack's self-signed seaweedfs.

Tests + e2e:
- unit tests for the driver (validation, ensure-idempotency, precondition,
  backupSource resolution, snapshot round-trip, PITR option parsing).
- examples/backups/mongodb + a mongodb-2-backup-roundtrip Chainsaw suite that
  drives run-all.sh end-to-end (sentinel document round-trips through S3 into a
  restored copy while the source stays untouched), per docs/agents/e2e-testing.md.
- docs/operations/backup-classes.md updated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-08-12 12:00:33 +04:00
Andrey Kolkov f17ead3d26 feat(platform): add default backupclass
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-05-28 13:49:59 +04:00
Andrey Kolkov d80f67e799 feat(platform): add implementation of backup-strategy controller for etcd
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-05-22 14:35:08 +04:00
Andrey Kolkov 92930ad1aa feat(platform): add implementation of backup-strategy controller for foundationdb
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-05-13 09:49:19 +04:00
Andrey Kolkov 62605a198b feat(platform): implement backup strategy for mariadb
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-05-12 13:56:29 +04:00
Andrey Kolkov 250e1b3376 feat(platform): implement backup strategy for clickhouse application
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-05-11 14:46:48 +04:00
Andrey Kolkov ed5ae16e36 feat(platform): implement backup strategy for postgres application
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-05-07 13:12:54 +04:00
Timofei Larkin 000b5ff76c [backups]
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2026-02-04 15:50:23 +03:00
Timofei Larkin 75a4b8ecbd Merge branch 'feat/backup-work-consolidation' into feat/backup-recovery
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2026-02-03 21:20:59 +04:00
Andrey Kolkov ffde02c992 fix setup cache for backupClasses
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-01-22 12:59:42 +04:00
Andrey Kolkov f6641c1547 fixes
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-01-22 10:39:33 +04:00
Andrey Kolkov 41646b253e use everywhere NormalizeApplicationRef
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-01-22 10:39:33 +04:00
Andrey Kolkov 9e94a699a0 feat(backups): implement interface changes related backups class introduction
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-01-22 10:39:33 +04:00
Andrey Kolkov 5e15a75d89 [backups] add restore jobs controller
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-01-08 18:57:42 +03:00
Timofei Larkin f4228ffc20 [backups] Add templating of velero backups
## What this PR does

This patch narrows the scope of the Velero backup strategy controller to
simply template Velero Backups according to the application being backed
up and the template in the strategy. Creating storage locations is now
out of scope.

### Release note

```release-note
[backups] Implement templating for Velero backups and remove creation of
backup storage locations and volume snapshot locations.
```

Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2026-01-04 13:31:15 +03:00
Andrey Kolkov bfafcaa3ab [backups] Implement Velero strategy controller
## What this PR does

This patch implements the Reconcile function for BackupJobs with a
Velero strategy ref.

### Release note

```release-note
[backups] Implement the Velero backup strategy controller.
```

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-01-04 12:55:24 +03:00
Timofei Larkin 8cf68149cb [backups] Scaffold a Velero strategy
## What this PR does

This patch adds boilerplate code for a Velero strategy for the backups
API. Implementation can be added into the Reconcile function in a
subsequent PR.

###

```release-note
[backups] Add the boilerplate for a new Velero strategy for backups.
```

Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2025-12-26 10:14:29 +03:00