Nothing stopped two RestoreJobs from rewriting one application at once.
Each restores the target from its own Backup, so the application ends up
on whichever finishes last, and the CNPG driver's purge decision cannot
tell which of the two produced the live Cluster: the second job could
skip its purge and report Succeeded on the first job's data.
A RestoreJob that has not started now fails with reason
ConcurrentRestore when another unfinished RestoreJob in the namespace
writes into the same application and goes first. A started restore goes
before one that has not; between two that have not, the older one goes
first, by creation time and then name, so a running restore is never
failed by a newer one. The refused job fails instead of waiting, so the
application is not restored a second time later, after the first
restore, by a job its author may no longer expect to run.
The gate lists RestoreJobs through the manager's API reader rather than
the informer cache. Creation times have one-second precision, so two
jobs created in the same second are ordered by name, and a job reconciled
before the cache held its peer's StartedAt would see that peer as not
started and proceed alongside it. The controller runs one worker, so a
peer's StartedAt written by an earlier reconcile is on the API server by
the time the next job reads it; SetupWithManager now states that.
The target is resolved as the BackupJob hold resolves it, so a Velero
restore into another namespace does not collide with one in place.
Refs #3322
Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
The BackupJob and RestoreJob reconcilers decide a job is still active
from the informer cache, and the cache can trail their own writes. A
driver's first pass stamps StartedAt, which requeues the job, and may
then fail the job in the same pass on a transient error, such as a 503
from the aggregated apps.cozystack.io API while it reads the
application. The requeued pass can start before the informer has seen
Failed: it retries the step that failed and, once the error has
cleared, creates the driver object anyway. Its own status write then
conflicts, the pass after it sees Failed and skips the job, and a
Velero Backup is left running that no Backup artifact will reference
and nothing tracks.
Against a real apiserver and the manager's cache, with one 503 on the
first application read of each job, 5 to 10 BackupJobs out of 40 ended
Failed with a Velero Backup.
A restore pass that starts on such a stale copy does worse, since the
step it retries can be destructive: suspending the VM's HelmRelease and
halting the VM before creating a Velero Restore, or patching the
Postgres application and purging its cluster.
Both reconcilers now re-read the job through the manager's uncached
reader when the cached copy says it is still active, and skip it when
the apiserver holds it as finished. The check sits ahead of the
dispatch, so it covers every driver, and a job the cache already
reports as finished costs no extra request.
Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
renamePVC created the -orig PVC, then read it back through the cached
client to learn its UID. The informer had often not seen the PVC yet, so
the read failed with NotFound. On a two-disk VM restore one disk was
kept and the other was not; with that error turned into an event, its
DataVolume was deleted, the original PVC stayed in place and Velero
skipped it, and the restore reported success without restoring the
disk. The -orig PVC was left Pending on the same PV, by then Retain.
Create already returns the UID, so the read-back goes. The other reads
the rename decides on, the -orig and original PVCs and the PV, go
through the manager's uncached reader, and the PV rebind retries on
conflict, since the PV controller writes the PV once the original PVC
is deleted.
A rename that fails for any reason other than an -orig name taken by
another PVC now requeues the restore instead of failing it. The
DataVolume is still there, and the retry finds its own -orig PVC by the
RestoreJob UID and resumes where it stopped.
Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Run go fix over every first-party module: interface{} becomes any,
hand-written loops become slices, maps, min/max and range-over-int,
errors.As becomes errors.AsType, pointer helpers become new(expr) and
are inlined at their call sites, and so on. Every rewrite keeps
behaviour.
Three rewrites stay out. embedlit flattens embedded-field literals
such as ObjectMeta, syntax only Go 1.27 accepts, which would break
backports to release branches still on Go 1.26. omitzero drops
omitempty from struct fields, and openapi-gen reads omitempty as
optional, so metadata, spec and status would turn required in the
published schemas. And errorsastype rewrote an errors.As whose
target, apierrors.APIStatus, does not implement error, which
errors.AsType rejects at compile time.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Fail closed on topic listing: set -o pipefail, capture kafka-topics
--list --exclude-internal exit separately, and abort on failure so an
unreachable broker can no longer produce an empty backup reported as
Succeeded. An empty list after a successful call stays legitimate.
- Check each kafka-topics --describe exit and validate the parsed
partition count / replication factor before writing a record, so a
timed-out describe cannot write a corrupt orders\t\t\t line.
- Store topic configs one per line with key and value in separate
tab-delimited fields, sourced from kafka-configs --describe (one config
per line), so a value containing a comma (cleanup.policy=compact,delete)
round-trips; on restore each config is applied with the value bracketed
only when it contains a comma.
- Detect topic divergence on restore instead of a silent --if-not-exists
no-op: grow partitions with --alter when the backup asks for more, and
fail loudly (naming the topic and both values) when it asks for fewer
partitions or a different replication factor. A restore that cannot
reach the recorded state no longer reports Succeeded.
- Add KafkaStrategyKind to the RestoreJob cleanup-dispatch no-op group so
deleting a Kafka RestoreJob does not run a pointless Velero cleanup; add
a regression test alongside the siblings, plus a reconcile Kind-gate
test and a rendering test that actually renders the strategy template.
- Wire the kafka-metadata Chainsaw suite into both selectors
(select-install suite_to_source, select-e2e src_to_suites) so editing
the kafka app or the suite selects it and the bats unit tests pass.
- Pin the client image to the operator's default broker version
(kafka 3.9.1) with a digest, and align the example default; fix the
values comment. Point the RabbitMQ comment references (which shipped in
the CRD description) at the Job strategy actually mirrored, and trim the
in-pod comments that rendered into the manifest.
- Document the Strimzi Topic Operator caveat: for a topic owned by a
KafkaTopic CR the operator reconciles its config from the CR, so an
in-place config restore is durable only for out-of-band topics.
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
Add a strategy.backups.cozystack.io/Kafka driver that backs up the topic
metadata of a Cozystack-managed Kafka application — partition counts,
replication factors and non-default topic configs — through the Kafka
Admin API, and restores it with kafka-topics --create + kafka-configs
--alter. Modelled on the Job/Rabbitmq strategies: the driver renders a
one-shot batch Job, gates on the Strimzi Kafka cluster reporting Ready,
and keys each backup by .BackupName so runs do not overwrite each other.
Scope is deliberately narrow: message payloads, consumer-group offsets,
ACLs, quotas and KafkaUsers are out of scope, and there is no PITR.
Restored topics are created directly (unmanaged on the target) — the
accepted tradeoff of an Admin-API-only driver. Both restore flows are
implemented: in-place recreates dropped topics into the source, to-copy
applies the source's topics onto a freshly-bootstrapped empty cluster.
Wire it into cozy-default (strategy-kafka-default.yaml gated on the
bucket name, a BackupClass binding, read RBAC on kafka.strimzi.io/kafkas
for the readiness gate, and the Strimzi Kafka client image), document it
in backup-classes.md, and add an honest round-trip example plus Chainsaw
e2e (examples/backups/kafka-metadata, kafka-3-metadata-roundtrip) that
proves a per-run retention.ms sentinel survives both restores.
Unit tests cover the driver's pure helpers (application-ref validation,
readiness verdict, parameter round-trip, per-run artifact-URI keying).
Not yet verified on a live cluster.
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
The spotahome RedisFailover operator ships no backup CRD, so add a
dedicated strategy.backups.cozystack.io/Redis driver that runs a one-shot
Job: it dumps an RDB snapshot from the current master (discovered via the
operator's sentinel Service) to the system bucket over S3, and on restore
replays it (RDB -> RESP via redis-cli --pipe) into the target's master.
Structurally a specialisation of the Job/Altinity script-Job pattern with
a Redis applicationRef gate and an app-readiness precondition. Backup
parameters round-trip through the Backup's DriverMetadata (no
underlyingResources snapshot); restore drives from .Backup.ApplicationRef
so a to-copy restore reads the source's object-storage prefix while
writing into a differently-named target. Both in-place and to-copy are
supported; PITR is not (RDB is a discrete snapshot).
- api: Redis/RedisList strategy type + generated CRD and deepcopy
- driver: reconcileRedis / reconcileRedisRestore + pure helpers, wired
into both dispatch switches, supportedBackupStrategyKinds and the no-op
restore cleanup case
- default: gated cozy-default-redis strategy (system-bucket coords via
cozy-backups-creds env secretKeyRef, https:// scheme + optional CA) and
the Redis route in the cozy-default BackupClass
- image: images/redis-backup bundling redis-cli + curl + an RDB->RESP
converter, wired via backupStrategyController.redisBackupClientImage
- tests: driver unit tests (gate, precondition, render, param round-trip,
both reconcile paths) and the helm rendering-gate coverage
- example: examples/backups/redis honest marker round-trip for both
restore flows
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-By: LLM
RabbitMQ's cluster operator ships no backup CRD and RabbitMQ has no
consistent online export of message payloads, so back up broker
DEFINITIONS (vhosts, users, permissions, queues, exchanges, bindings,
policies, parameters) over the management HTTP API and store them in the
system bucket. Message payloads and PITR are out of scope - a full-data
backup is a Velero volume snapshot, a separate strategy.
Add a first-class strategy.backups.cozystack.io/Rabbitmq driver, following
the Altinity pattern (a dedicated driver that renders a one-shot Job for an
HTTP-API engine):
- reconcileRabbitmq / reconcileRabbitmqRestore render the strategy's
PodTemplateSpec (curl GET/POST /api/definitions, push/pull to S3 with
--aws-sigv4) as a batch/v1.Job, for both in-place and to-copy restore.
- A Go-read precondition gates each run on the RabbitmqCluster reporting
AllReplicasReady (via the rabbitmqtypes partial type), surfacing a
precise Ready=False wait instead of a Job that starts and fails.
- The produced Backup records the object location on status.artifact.uri
(rendered from spec.artifactURITemplate; the Backup CRD has no status
subresource, so it persists on Create, as in the Etcd driver).
Wire dispatch in both controllers + supportedBackupStrategyKinds, register
rabbitmqtypes in the scheme, and grant read on rabbitmqclusters. Ship
cozy-default-rabbitmq (a $bucketName-gated Rabbitmq strategy) and route
apps.cozystack.io/RabbitMQ to it in the cozy-default BackupClass. The
application chart is untouched.
Add unit tests for the driver's pure helpers and the precondition gate, a
runnable example, and an honest-roundtrip Chainsaw e2e that seeds a
sentinel vhost, backs it up, drops it, and asserts it round-trips through
object storage on both restore flows.
The Job pod reuses the platform-migrations image (curl + ca-certificates)
under a new rabbitmqBackupClientImage key, stamped in lockstep with
chBackupClientImage so the repository stays pinned at one digest; a
dedicated slim image is a follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Adds a MongoDB (Percona Server for MongoDB / psmdb) backup-strategy driver so
`apps.cozystack.io/MongoDB` reaches feature parity with the CNPG, MariaDB and
ClickHouse native drivers — logical backups, restore-to-differently-named
instances, and point-in-time recovery through the unified
BackupJob/RestoreJob/BackupClass/Plan interface (closes#3545).
The driver orchestrates the operator's PerconaServerMongoDBBackup /
PerconaServerMongoDBRestore CRDs:
- BackupJob: creates a one-shot PerconaServerMongoDBBackup, polls
status.state, and persists the S3 destination + storage descriptor
(by-reference, no credentials) onto the Cozystack Backup artifact.
- RestoreJob: drives PerconaServerMongoDBRestore via backupSource (read back
from the source backup status, snapshot fallback) so the same artifact
restores in-place or into a differently-named instance; PITR via
spec.options.pitr (date/latest).
Because psmdb resolves the backup storage by name against the cluster's
spec.backup.storages, the driver names the chart's storage (s3-storage) via the
cozy-default-mongodb Strategy rather than injecting it, and surfaces a clear
Ready=False precondition when a target has backups disabled.
Wiring:
- api/backups/strategy/v1alpha1: MongoDB strategy type + generated CRD/deepcopy.
- internal/backupcontroller: psmdbstrategy_controller + psmdbtypes/mongodbapp
partial-schema mirrors; dispatch, supported-kinds and scheme registration.
- backupstrategy-controller: cozy-default MongoDB route, cozy-default-mongodb
Strategy CR, and psmdb.percona.com RBAC.
- apps/mongodb: backup.insecureSkipTLSVerify knob (psmdb s3 storage has no
CA-bundle field) so backups can target Cozystack's self-signed seaweedfs.
Tests + e2e:
- unit tests for the driver (validation, ensure-idempotency, precondition,
backupSource resolution, snapshot round-trip, PITR option parsing).
- examples/backups/mongodb + a mongodb-2-backup-roundtrip Chainsaw suite that
drives run-all.sh end-to-end (sentinel document round-trips through S3 into a
restored copy while the source stays untouched), per docs/agents/e2e-testing.md.
- docs/operations/backup-classes.md updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Makes RestoreJob.spec.options.recoveryTime a genuinely supported, tested
capability for the new backups.cozystack.io API (issue #2774). The
recoveryTime -> CNPG bootstrap.recovery.recoveryTarget.targetTime plumbing
already existed on the barman-cloud-plugin branch; this adds the reliability
and verification layer the issue asked for:
- Fail-fast for an unreachable target: when a RestoreJob has recoveryTime set
and the recovery Cluster is not healthy, the driver reads the CNPG
full-recovery pod logs and, once several distinct attempts have exited with
PostgreSQL's "recovery ended before configured recovery target was reached"
FATAL, fails the RestoreJob with conditions RecoveryConverged=False /
Ready=False (reason RecoveryTargetUnreachable) instead of hanging to the
restore deadline. Health is checked first, so a converged recovery always
wins; counting distinct FATAL-ing attempts (not pod failures) tolerates a
transient archive-lag miss on a near-now target and ignores transient
recovery-pod crashes. Adds the pods/log RBAC grant the log read requires.
- e2e (examples/backups/postgres/run-all.sh, wired into the Chainsaw
postgres-2-backup-roundtrip suite): step 45 writes a before marker, captures
the server timestamp, writes an after marker, restores to that timestamp and
asserts the before row survived and the after row did not; step 46 submits a
recoveryTime an hour ahead and asserts it fails fast with reason
RecoveryTargetUnreachable.
- Working, uncommented example (45-restorejob-pitr.yaml) and a docs section
(docs/operations/backup-classes.md) covering the recoverable window, how to
discover the earliest/latest restorable time from the backup catalog, the
archive-lag tolerance, and idempotency under GitOps.
Refs #2774
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Andrey Kolkov