The KafkaSpec.Template doc listed .Mode as "backup" or "restore", but the
driver also renders it as "cleanup" when reaping a removed Backup's object;
the comment is copied verbatim into the CRD, so kubectl explain and the
dashboard showed two values while a custom strategy's restore branch would
run on the cleanup pass. List all three, matching the Redis sibling, and
regenerate the CRD.
The strategy template also claimed the embedded script "reads only env
vars, never Helm/controller values, so no tpl is applied". The controller
renders every string leaf of the PodTemplateSpec through internal/template,
args[0] included, so the script is templated on every run. State that
instead, without literal template braces that Helm would itself expand.
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
- createKafkaBackupArtifact now rejects adopting a same-named Backup that
describes a different application, so a retained Backup reused for a
different app of the same name is not reported as this run's success.
Mirrors the RabbitMQ driver's guard; pinned by a Go test.
- ArtifactURITemplate godoc said empty leaves status.artifact unset, but
the driver hard-fails an empty template. Reword to state it is required,
and regenerate the CRD description to match.
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
- Fail closed on topic listing: set -o pipefail, capture kafka-topics
--list --exclude-internal exit separately, and abort on failure so an
unreachable broker can no longer produce an empty backup reported as
Succeeded. An empty list after a successful call stays legitimate.
- Check each kafka-topics --describe exit and validate the parsed
partition count / replication factor before writing a record, so a
timed-out describe cannot write a corrupt orders\t\t\t line.
- Store topic configs one per line with key and value in separate
tab-delimited fields, sourced from kafka-configs --describe (one config
per line), so a value containing a comma (cleanup.policy=compact,delete)
round-trips; on restore each config is applied with the value bracketed
only when it contains a comma.
- Detect topic divergence on restore instead of a silent --if-not-exists
no-op: grow partitions with --alter when the backup asks for more, and
fail loudly (naming the topic and both values) when it asks for fewer
partitions or a different replication factor. A restore that cannot
reach the recorded state no longer reports Succeeded.
- Add KafkaStrategyKind to the RestoreJob cleanup-dispatch no-op group so
deleting a Kafka RestoreJob does not run a pointless Velero cleanup; add
a regression test alongside the siblings, plus a reconcile Kind-gate
test and a rendering test that actually renders the strategy template.
- Wire the kafka-metadata Chainsaw suite into both selectors
(select-install suite_to_source, select-e2e src_to_suites) so editing
the kafka app or the suite selects it and the bats unit tests pass.
- Pin the client image to the operator's default broker version
(kafka 3.9.1) with a digest, and align the example default; fix the
values comment. Point the RabbitMQ comment references (which shipped in
the CRD description) at the Job strategy actually mirrored, and trim the
in-pod comments that rendered into the manifest.
- Document the Strimzi Topic Operator caveat: for a topic owned by a
KafkaTopic CR the operator reconciles its config from the CR, so an
in-place config restore is durable only for out-of-band topics.
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
Add a strategy.backups.cozystack.io/Kafka driver that backs up the topic
metadata of a Cozystack-managed Kafka application — partition counts,
replication factors and non-default topic configs — through the Kafka
Admin API, and restores it with kafka-topics --create + kafka-configs
--alter. Modelled on the Job/Rabbitmq strategies: the driver renders a
one-shot batch Job, gates on the Strimzi Kafka cluster reporting Ready,
and keys each backup by .BackupName so runs do not overwrite each other.
Scope is deliberately narrow: message payloads, consumer-group offsets,
ACLs, quotas and KafkaUsers are out of scope, and there is no PITR.
Restored topics are created directly (unmanaged on the target) — the
accepted tradeoff of an Admin-API-only driver. Both restore flows are
implemented: in-place recreates dropped topics into the source, to-copy
applies the source's topics onto a freshly-bootstrapped empty cluster.
Wire it into cozy-default (strategy-kafka-default.yaml gated on the
bucket name, a BackupClass binding, read RBAC on kafka.strimzi.io/kafkas
for the readiness gate, and the Strimzi Kafka client image), document it
in backup-classes.md, and add an honest round-trip example plus Chainsaw
e2e (examples/backups/kafka-metadata, kafka-3-metadata-roundtrip) that
proves a per-run retention.ms sentinel survives both restores.
Unit tests cover the driver's pure helpers (application-ref validation,
readiness verdict, parameter round-trip, per-run artifact-URI keying).
Not yet verified on a live cluster.
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
Store the Redis strategy Template schemaless, as the Rabbitmq strategy
already does: without the markers controller-gen emitted the full
PodTemplateSpec schema and the CRD reached ~602 KB, pushing the chart's
Helm-release Secret past the 1 MiB cap so backupstrategy-controller failed
to install at all.
Give the example strategy a cleanup branch, and both scripts an explicit
`elif restore` / `else exit 1`, so a deleted Backup rendered with
Mode=cleanup deletes its object instead of falling into the restore path
and FLUSHALLing the live master.
Forward a backupStrategyController block through the platform bundle
alongside backupStorage, so redisBackupResources — the restore-Pod sizing
knob the loader-not-ready message points at — is reachable via
spec.components.platform.values.backupStrategyController, not merely
settable on a Helm-managed Strategy CR that reverts on the next upgrade.
Correct the endpointCASecretName doc row to match the now-optional mount,
and `int` the activeDeadlineSeconds interpolation so a large values-file
override renders as an integer, not exponent form.
Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Delete the backup object when its Backup is removed. The Redis driver
owns its S3 object (one per BackupJob, no engine retention prunes it), so
it now spawns a cleanup-mode Job that deletes the object and holds the
Backup's finalizer until it completes, mirroring the Rabbitmq driver.
Without this a retention-pruned Plan grew the bucket unboundedly and the
key was lost with the CR.
Size the strategy Pod from backupStrategyController.redisBackupResources
so an admin can raise the restore ceiling (the throwaway loader holds the
whole dataset in memory) via chart values, which a direct edit of the
cluster-scoped, Helm-managed Strategy CR could not survive. The
loader-not-ready diagnostic now names the memory limit, the likeliest
cause, instead of only the image version.
Make the endpoint CA volume optional: nothing projects that Secret into
the app namespace, so a required mount wedged every backup Pod on
FailedMount; a missing ca.crt now falls through to the system CA store
and fails the handshake fast, which is not the same as skipping verify.
Promote the Pod-level activeDeadlineSeconds to the Job so a wedged run is
bounded once, not (backoffLimit+1) times. Use find, not ls, for the base
AOF so its no-match diagnostic is not dead code under pipefail.
Pin the object key reaching the rendered Job: both fixtures now render
--key={{ .ObjectKey }} and assert it, so zeroing the key (which left the
package green) is caught. Document .ObjectKey and the cleanup mode in the
CRD render-context, list cozy-default-redis among the Helm-managed CRs,
and mirror the CA/TLS and find fixes into the example strategy.
Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
The spotahome RedisFailover operator ships no backup CRD, so add a
dedicated strategy.backups.cozystack.io/Redis driver that runs a one-shot
Job: it dumps an RDB snapshot from the current master (discovered via the
operator's sentinel Service) to the system bucket over S3, and on restore
replays it (RDB -> RESP via redis-cli --pipe) into the target's master.
Structurally a specialisation of the Job/Altinity script-Job pattern with
a Redis applicationRef gate and an app-readiness precondition. Backup
parameters round-trip through the Backup's DriverMetadata (no
underlyingResources snapshot); restore drives from .Backup.ApplicationRef
so a to-copy restore reads the source's object-storage prefix while
writing into a differently-named target. Both in-place and to-copy are
supported; PITR is not (RDB is a discrete snapshot).
- api: Redis/RedisList strategy type + generated CRD and deepcopy
- driver: reconcileRedis / reconcileRedisRestore + pure helpers, wired
into both dispatch switches, supportedBackupStrategyKinds and the no-op
restore cleanup case
- default: gated cozy-default-redis strategy (system-bucket coords via
cozy-backups-creds env secretKeyRef, https:// scheme + optional CA) and
the Redis route in the cozy-default BackupClass
- image: images/redis-backup bundling redis-cli + curl + an RDB->RESP
converter, wired via backupStrategyController.redisBackupClientImage
- tests: driver unit tests (gate, precondition, render, param round-trip,
both reconcile paths) and the helm rendering-gate coverage
- example: examples/backups/redis honest marker round-trip for both
restore flows
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-By: LLM
RabbitMQ's cluster operator ships no backup CRD and RabbitMQ has no
consistent online export of message payloads, so back up broker
DEFINITIONS (vhosts, users, permissions, queues, exchanges, bindings,
policies, parameters) over the management HTTP API and store them in the
system bucket. Message payloads and PITR are out of scope - a full-data
backup is a Velero volume snapshot, a separate strategy.
Add a first-class strategy.backups.cozystack.io/Rabbitmq driver, following
the Altinity pattern (a dedicated driver that renders a one-shot Job for an
HTTP-API engine):
- reconcileRabbitmq / reconcileRabbitmqRestore render the strategy's
PodTemplateSpec (curl GET/POST /api/definitions, push/pull to S3 with
--aws-sigv4) as a batch/v1.Job, for both in-place and to-copy restore.
- A Go-read precondition gates each run on the RabbitmqCluster reporting
AllReplicasReady (via the rabbitmqtypes partial type), surfacing a
precise Ready=False wait instead of a Job that starts and fails.
- The produced Backup records the object location on status.artifact.uri
(rendered from spec.artifactURITemplate; the Backup CRD has no status
subresource, so it persists on Create, as in the Etcd driver).
Wire dispatch in both controllers + supportedBackupStrategyKinds, register
rabbitmqtypes in the scheme, and grant read on rabbitmqclusters. Ship
cozy-default-rabbitmq (a $bucketName-gated Rabbitmq strategy) and route
apps.cozystack.io/RabbitMQ to it in the cozy-default BackupClass. The
application chart is untouched.
Add unit tests for the driver's pure helpers and the precondition gate, a
runnable example, and an honest-roundtrip Chainsaw e2e that seeds a
sentinel vhost, backs it up, drops it, and asserts it round-trips through
object storage on both restore flows.
The Job pod reuses the platform-migrations image (curl + ca-certificates)
under a new rabbitmqBackupClientImage key, stamped in lockstep with
chBackupClientImage so the repository stays pinned at one digest; a
dedicated slim image is a follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Adds a MongoDB (Percona Server for MongoDB / psmdb) backup-strategy driver so
`apps.cozystack.io/MongoDB` reaches feature parity with the CNPG, MariaDB and
ClickHouse native drivers — logical backups, restore-to-differently-named
instances, and point-in-time recovery through the unified
BackupJob/RestoreJob/BackupClass/Plan interface (closes#3545).
The driver orchestrates the operator's PerconaServerMongoDBBackup /
PerconaServerMongoDBRestore CRDs:
- BackupJob: creates a one-shot PerconaServerMongoDBBackup, polls
status.state, and persists the S3 destination + storage descriptor
(by-reference, no credentials) onto the Cozystack Backup artifact.
- RestoreJob: drives PerconaServerMongoDBRestore via backupSource (read back
from the source backup status, snapshot fallback) so the same artifact
restores in-place or into a differently-named instance; PITR via
spec.options.pitr (date/latest).
Because psmdb resolves the backup storage by name against the cluster's
spec.backup.storages, the driver names the chart's storage (s3-storage) via the
cozy-default-mongodb Strategy rather than injecting it, and surfaces a clear
Ready=False precondition when a target has backups disabled.
Wiring:
- api/backups/strategy/v1alpha1: MongoDB strategy type + generated CRD/deepcopy.
- internal/backupcontroller: psmdbstrategy_controller + psmdbtypes/mongodbapp
partial-schema mirrors; dispatch, supported-kinds and scheme registration.
- backupstrategy-controller: cozy-default MongoDB route, cozy-default-mongodb
Strategy CR, and psmdb.percona.com RBAC.
- apps/mongodb: backup.insecureSkipTLSVerify knob (psmdb s3 storage has no
CA-bundle field) so backups can target Cozystack's self-signed seaweedfs.
Tests + e2e:
- unit tests for the driver (validation, ensure-idempotency, precondition,
backupSource resolution, snapshot round-trip, PITR option parsing).
- examples/backups/mongodb + a mongodb-2-backup-roundtrip Chainsaw suite that
drives run-all.sh end-to-end (sentinel document round-trips through S3 into a
restored copy while the source stays untouched), per docs/agents/e2e-testing.md.
- docs/operations/backup-classes.md updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
The four backups.cozystack.io CRDs injected the x-cozystack-options vendor
extension into their OpenAPI validation schema. apiextensions JSONSchemaProps
is a closed struct that only preserves x-kubernetes-* vendor extensions, so
helm-controller's server-side apply rejected the unknown key ("field not
declared in schema") and the CRDs never applied — the controller crash-looped
and backups were non-functional.
Move the field-to-source mapping the dashboard needs into CRD
metadata.annotations (preserved by the apiserver), emitted via
+kubebuilder:metadata:annotations markers, and drop the post-processing awk
injector. The dashboard reattaches the extension client-side.
A helm-unittest suite pins the contract: each dropdown field carries its source
in metadata.annotations and must not carry x-cozystack-options in the schema,
guarding against re-introducing the injector.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What
Implements the `strategy.backups.cozystack.io/Job` backup strategy — a
generic, application-agnostic driver that runs a user-supplied
Kubernetes Job to create (and restore) a backup.
## Background
This PR began as a draft written against an earlier shape of the backups
API. The API has since moved to a `BackupClass`-based dispatch model,
which broke the original draft on three counts:
- `BackupJob.Spec.StrategyRef` and `BackupJob.Spec.StorageRef` were
removed; the strategy and its parameters are now resolved from a
`BackupClass` into a `ResolvedBackupConfig`.
- Strategy handling moved from standalone per-strategy reconcilers to a
dispatch model: `BackupJobReconciler.Reconcile` /
`RestoreJobReconciler.Reconcile` resolve the strategy and switch into
per-strategy methods (`reconcileVelero`, `reconcileCNPG`,
`reconcileAltinity`, …).
- The draft's template context (`.Values`, `.Storage`) diverged from the
convention the other strategies adopted, and its `.Storage` half
targeted a storage API that does not exist.
## What changed
- **`reconcileJob` / `reconcileJobRestore`**
(`internal/backupcontroller/jobstrategy_controller.go`): replaces the
standalone `BackupJobStrategyReconciler` with the two dispatch methods
the `BackupJob` / `RestoreJob` reconcilers call. The Job strategy is the
generic counterpart of the Altinity driver — same lifecycle (render
`PodTemplateSpec` → run a one-shot `batch/v1.Job` → poll its terminal
condition → emit a `Backup`), minus the ClickHouse-specific
`applicationRef` gate, so it runs for any application. Shared package
helpers (`renderJobTemplate`, `jobConditionState`,
`getApplicationUnstructured`, status helpers, deterministic Job naming,
controller refs) are reused.
- **Template context** follows the in-tree convention used by the other
drivers: `.Application` (live application object), `.Release.Name` /
`.Release.Namespace`, `.Mode` (`backup` / `restore`), `.Parameters`
(resolved `BackupClass` parameters), and `.Backup` (restore runs only).
Strategy parameters round-trip through the `Backup`'s `driverMetadata`
so a later restore re-renders against the same values.
- **Restore** re-renders the same template with `.Mode == "restore"`.
Cross-namespace restore is unsupported (the RestoreJob target reference
is namespace-local), matching the Altinity driver.
- **`JobSpec.Template` doc + regenerated CRD**: the field doc previously
advertised `.Values`; it now documents the real context, and
`strategy.backups.cozystack.io_jobs.yaml` is regenerated to match (the
change is limited to that field's description).
- **Tests** (`jobstrategy_controller_test.go`): backup-Job creation /
args / labels / status, completion → `Backup` artifact with parameter
round-trip, Job-failure surfacing, missing-strategy terminal failure,
restore into the target namespace, restore completion, and the
parameter-prefix unit. A deliberately generic application kind pins that
the driver never gates on `applicationRef.kind`.
No RBAC changes are required: the controller already holds
`strategy.backups.cozystack.io/*`, `batch/jobs`, `apps.cozystack.io/*`,
and `backups` permissions.
## Notes for reviewers
- Backup and restore share one template, so a strategy author's
`PodTemplateSpec` must branch on `.Mode` to do anything
restore-specific. This matches the existing convention; worth confirming
it's the intended contract for the Job strategy's docs/examples.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Implemented full backup/restore job strategy with deterministic Job
creation, idempotent execution, and status/condition updates; added demo
example scripts and reusable shell helpers for a NATS JetStream
backup/restore workflow.
* **Documentation**
* Clarified Job template rendering context and listed supported template
variables; added README and step-by-step scenario docs for the NATS
example.
* **Tests**
* Added comprehensive unit tests covering job creation,
completion/failure handling, parameter extraction, and unmappable-target
failure cases.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/1721?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Complete the strategy.backups.cozystack.io/Job driver as the generic,
app-agnostic counterpart to the Altinity driver: render the strategy
PodTemplateSpec, run it as a one-shot batch/v1.Job, poll its terminal
condition, and emit a Backup. Wire it into the existing BackupClass
dispatch via reconcileJob/reconcileJobRestore rather than a standalone
reconciler, matching the post-BackupClass backups API (the old draft
read BackupJob.Spec.StrategyRef/StorageRef, which no longer exist).
Template context follows the in-tree convention (.Application, .Release,
.Mode, .Parameters, .Backup); the JobSpec.Template doc and regenerated
CRD are updated to match. Restore re-renders the same template with
.Mode=restore; cross-namespace restore is unsupported, mirroring
Altinity.
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
The Options field was added to RestoreJobSpec without re-running
'make generate', leaving zz_generated.deepcopy.go out of sync.
Regenerated to include the missing DeepCopyInto handling for the
runtime.RawExtension pointer.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Running make generate at repo root regenerates the zz_generated.deepcopy.go
files. Key fix: Source.DeepCopyInto in the vmdisk package now handles
the new Disk *SourceDisk pointer field, which was previously missing and
would cause pointer aliasing on deep-copy.
Other generated changes are incidental updates to vminstance (Networks
field added, Subnet renamed to Network) and backups CRD manifests picked
up by controller-gen from the current HEAD.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The mysql chart actually deploys MariaDB via mariadb-operator, but was
incorrectly named "mysql". Rename all references to use the correct
"mariadb" name across the codebase.
Changes:
- Rename packages/apps/mysql -> packages/apps/mariadb
- Rename packages/system/mysql-rd -> packages/system/mariadb-rd
- Rename platform source and bundle references
- Update CRD kind from MySQL to MariaDB
- Update RBAC, e2e tests, backup controller tests
- Keep real MySQL CLI/config tool names unchanged (mysqldump, [mysqld], etc.)
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
This patch narrows the scope of the Velero backup strategy controller to
simply template Velero Backups according to the application being backed
up and the template in the strategy. Creating storage locations is now
out of scope.
### Release note
```release-note
[backups] Implement templating for Velero backups and remove creation of
backup storage locations and volume snapshot locations.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
## What this PR does
This patch implements the Reconcile function for BackupJobs with a
Velero strategy ref.
### Release note
```release-note
[backups] Implement the Velero backup strategy controller.
```
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
## What this PR does
This patch adds boilerplate code for a Velero strategy for the backups
API. Implementation can be added into the Reconcile function in a
subsequent PR.
###
```release-note
[backups] Add the boilerplate for a new Velero strategy for backups.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
## What this PR does
This PR introduces a bare-bones Job backup strategy API type and stubs
out all the boilerplate for a new controller that will handle this
strategy, as well as any others in the `strategy.backups.cozystack.io`
API group.
### Release note
```release-note
[backups] Create stubs and minimal implmentations for controllers for
the strategy.backups.cozystack.io API group.
```
## What this PR does
This adds custom indexable fields on core backup resources to enable
filtering by backed-up application. This will later be useful for
displaying backup resources in the dashboard: it will be possible to
filter them with a field selector to display only those backup resources
that are relevant to a given application.
### Release note
```release-note
[backups] Enable filtering backup resources by backed up application for
per-app views of backups.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **New Features**
* Introduced new Backup, BackupJob, Plan, and RestoreJob API resources
for enhanced backup management.
* Added selectable fields to backup resources, enabling efficient
filtering and querying by application reference.
* Added status subresource with conditions to Plan resources for
improved status tracking and observability.
* **Chores**
* Updated repository metadata configuration.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
This PR introduces a bare-bones Job backup strategy API type and stubs
out all the boilerplate for a new controller that will handle this
strategy, as well as any others in the `strategy.backups.cozystack.io`
API group.
### Release note
```release-note
[backups] Create stubs and minimal implmentations for controllers for
the strategy.backups.cozystack.io API group.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
## What this PR does
This adds custom indexable fields on core backup resources to enable
filtering by backed-up application. This will later be useful for
displaying backup resources in the dashboard: it will be possible to
filter them with a field selector to display only those backup resources
that are relevant to a given application.
### Release note
```release-note
[backups] Enable filtering backup resources by backed up application for
per-app views of backups.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
## What this PR does
This patch adds the boilerplate for the `strategy.backups.cozystack.io`
API group that will contain reference backup/restore strategy
implementations that will live in Cozystack core.
### Release note
```release-note
[backups] Scaffold the `strategy.backups.cozystack.io` API group to
provide a well-defined point for adding reference strategy
implementations.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
## What this PR does
This patch creates a new backups.cozystack.io API group which includes
backup-related resources owned exclusively by Cozystack core. A
cronjob-like controller is implemented to create backup jobs that will
be handled by appropriate third-party or external controllers.
### Release note
```release-note
[backups] Implement the core backup API and an accompanying controller.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Andrey Kolkov
