Files
cozystack/packages/system
Alexey Artamonov 993b79bb37 fix(backups): take over a running ClickHouse restore on Job retry
restore_remote drops every table it restores before recreating it, so
running it twice discards whatever was written after the first run
finished. The strategy Job retries a failed Pod up to twice, and with
the start gate waiting for an idle sidecar instead of hitting 423, a
retry whose predecessor died after its POST waited for that restore to
finish and then started a second one. A retry now takes over a
restore_remote of the same archive that is still running.

A finished row is not taken over: the sidecar never trims its action
log, so a success row of the archive may belong to an earlier
RestoreJob, and treating it as this one would report a restore that
never ran. A retry that arrives after the restore finished therefore
still runs it again.

Once the operation is accepted, one failed poll no longer ends the
attempt under errexit; six in a row do. An operation that ends
cancelled now fails the Pod instead of being polled forever.

A retried backup still starts a new create_remote, because its archive
name changes on every attempt; taking it over needs a name fixed for
the whole BackupJob.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-06 11:23:28 +03:00
..
…
…
…