The artifact took its destination from the record written when the
cnpg.io/Backup was created, but its serverName, the archive prefix
under that destination, from the live Cluster when the backup
completed. A restore that re-renders the Cluster in between gives it
another serverName, and the artifact then pairs the original
destination with a prefix the backup was never written to, so a later
restore searches the wrong archive. The record now carries the
serverName too; a record written before it did still falls back to the
live Cluster.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
A backup job that is about to write the WAL archive destination first
checks for a restore of its application, because a write landing during
the restore puts the restored Cluster back on the source's WAL prefix
and the restore fails with an expected-empty-archive error after it has
purged the target. That check read RestoreJobs from the informer cache,
which can still miss one created a moment ago.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The rollout check takes the later of the latch and the Cluster's attach
record, and the record is never rewritten while the plugin stays
attached. A record that parses but lies in the future made every
instance look older than the plugin and counted the rollout deadline
from that time, so the job waited past the deadline by as far as the
record lay ahead, without end for one far enough out. Such a record
says no more about when the plugin was attached than one that does not
parse, and fails the job the same way.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
A status write that fails with the apiserver unavailable takes the same
path as one lost to a conflict, but the attach it follows can itself be
the retry of an apply that failed first, which leaves its own condition
on the job. The test now runs that sequence as well, so the rollout check
is held to it whichever way the record was lost.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The rollout check takes the later of the latch and the Cluster's attach
record. A record left by an earlier attach of a plugin since removed is
older than the latch when someone who keeps no record attaches the
plugin again, and taking it would accept an instance recreated before
the latch as one that rolled with the plugin.
The record is absent not only when no backup job attached the plugin
but also when the chart did, or a controller that kept no record, so
the comment on the field says that.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The rollout check took the attach time from the PluginRolledOut
condition on the BackupJob, which the pass that attaches the plugin
writes only after the Cluster apply went through. When that status
write is lost to a conflict or a controller restart, the next pass
finds the plugin attached and falls back to the latch written before
the attach. An instance recreated between the latch and the attach then
counts as one that rolled with the plugin, and the cnpg.io/Backup is
created against an instance that has no plugin sidecar; a latch older
than the deadline also fails a rollout that has only just begun.
The apply that attaches the plugin now records the time on the Cluster
itself, so the record cannot be lost apart from the attach. The rollout
check and its deadline use the later of the two times. A record that
does not parse fails the job rather than guess which instances rolled.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The queue that keeps two BackupJobs of one application from writing the
WAL archive destination at once listed cnpg.io/Backups from the
informer cache. The controller runs one reconcile at a time under
leader election, so a Backup created by one job's pass already exists
when the next pass starts, but the cache can still miss it. The next
job then took the queue as empty, rewrote the destination under the
running backup and left that backup's recorded destination naming a
place its WAL no longer went. A job could likewise miss its own fresh
Backup, write the destination again and create a second Backup.
The queue now lists Backups from the apiserver, and a job that finds
no Backup of its own in the cache checks the apiserver before it
writes or creates one. Only a leader handover, when two processes
reconcile for a moment, can still let two jobs through.
The pass that creates the Backup no longer looks it up a second time
right before the create. Nothing can create it in between, and once
the lookup can reach the apiserver, a failed second lookup would come
after the destination write and fail the job instead of retrying.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The WAL archive destination is written by the pass that creates the
cnpg.io/Backup and not again while it runs, yet the artifact took the
destination, the endpoint and the credentials Secret from the strategy
rendered when the Backup completed. A strategy or BackupClass edited
while the backup ran gave an artifact that pointed somewhere nothing
had been written, and a restore from it found no data.
The pass that writes the destination now records it on the
cnpg.io/Backup it creates, and the artifact is built from that record.
The record lives on the Backup rather than on the BackupJob because a
tenant can edit its BackupJobs but has no access to postgresql.cnpg.io
objects: a record a tenant could rewrite would point the artifact, and
a restore from it, which takes its S3 keys from the strategy, at an
endpoint of the tenant's choosing.
The artifact size probe measures the same place: the recorded
location and credentials Secret, with the Secret keys and the endpoint
CA, which the record does not carry, from the strategy. Should those no
longer fit after an edit, the probe reports no size instead of
measuring somewhere else.
A record that cannot be read, or that names no destination path,
creates no artifact and fails the job, since the strategy may no longer
match the write. A Backup without a record was created by a controller
that kept none, and its artifact is built from the strategy as before.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Only the first attach after the latch restarted the rollout. In the
platform flow the chart renders no spec.plugins, so an in-place restore
recreates the Cluster without the plugin, and a backup job held through
that restore attaches it again once released. The rollout condition
then still carried its earlier start, the restored instances were newer
than it and ready, and the Backup was created in the same pass, before
CloudNativePG had rolled anything; such a Backup fails with "requested
plugin is not available" or "instance manager was restarted".
Every write of the job that finds the plugin missing now starts the
rollout over. A plugin attached by another writer is still timed from
the latch.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
A job that latched the barman-cloud plugin rollout and then waited for
another backup of the application, or for a restore of it, had that
wait charged to the rollout deadline. Released after more than the
deadline, it failed on the first instance that was not ready yet with
"rollout did not finish", although no rollout of its own had been
running while it waited.
The hold is now recorded in a DestinationWriteHeld condition, and its
release time restarts the rollout deadline when it is later than the
rollout's start. The instance pods are still measured against the
rollout's start: one that rolled while the job was held carries the
plugin and counts.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The deadline no longer applies to a failed cnpg.io/Backup, which now
fails its BackupJob at once; it only bounds the wait for the
barman-cloud plugin rollout, and the comment still claimed the former.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The phase test read the BackupJob it passed to the reconciler, which
holds the phase in memory whether or not the status write happened. A
completed Backup could then leave the stored BackupJob Running, so the
Plan would never see the run finish, and no test in the package noticed.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
When two jobs of the application hold the queue, the job waiting
behind them names one of them in its Ready message, and the sort is
what keeps that name stable however the cache orders their Backups.
Without it the message can change from one poll to the next, and each
change writes the BackupJob status again; nothing failed when the sort
was removed.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The rollout deadline starts at the latch when another writer attached
the plugin and at the write otherwise, and a wait for a restore resets
the destination retry budget just as a wait for another backup does.
The comments on the deadline and on the two conditions still described
only the attach and the queue.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
After a successful destination write each path of the CNPG driver sets
or removes Ready later in the same pass, and writes the status after
it, so clearing a stale WaitingForOtherBackup or
ClusterPluginApplyRetrying reason there could never outlive the pass,
and no test could pin it.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
Three guards of the CNPG driver could be reverted with the package
tests still green. The controller watches only BackupJobs, so a job
waiting in the queue or retrying the destination write is woken by
nothing but its own requeue, and the tests dropped the result. A job
whose Backup runs must not consult the queue, or two jobs that passed
it together wait on each other forever, and must patch neither the
ObjectStore nor the Cluster plugin, yet only the ObjectStore content
was checked.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The PluginRolledOut latch is recorded before the Cluster write that
attaches the barman-cloud plugin, and its time was used both as the
start of the rollout deadline and as the age every instance pod must
exceed to count as rolled. Now that a failing write is retried for up
to the backup deadline, the attach can come long after the latch: the
deadline was then spent before the rollout began, and a pod recreated
between the latch and the attach, without the plugin sidecar, passed
as rolled, so a Backup was created that CloudNativePG fails for good.
The latch now carries its own reason, and the write that finds the
plugin missing and attaches it re-stamps the condition, once, with the
time it was attempted. A plugin someone else attached after the latch,
another backup job or a restore, keeps the latch time: the rollout it
started can be over by the time this job writes, and its instances
would otherwise never count as rolled.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
The restore gate in Reconcile looks only at a BackupJob that has not
started, and the CNPG driver stamps StartedAt on its first pass. A job
that then waits behind another backup of the application, for as long
as that backup runs, writes the WAL archive destination and the
Cluster plugin once released without looking for a restore that began
in the meantime. A write landing while the restore re-renders the
Cluster can put the restored Cluster back on the source's WAL prefix,
where recovery fails with "Expected empty archive".
Every pass that is about to write the destination now looks for a
restore of the application and holds the job the way the gate does.
A job whose Backup already runs writes nothing and is not held. The
hold resets the retry budget of the destination write, as the wait
for another backup does, so time spent behind a restore does not fail
the job on the first error after it.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
The ObjectStore a Postgres Cluster archives its WAL to is shared by
every backup of the app, yet each running BackupJob server-side applied
its own strategy's destination onto it every five seconds with forced
ownership. Two BackupJobs of one app with different BackupClasses
flipped the destination back and forth until the healthy backup failed
too, and a webhook or apiserver blip during that apply failed the
BackupJob at once.
A BackupJob now writes the destination only before its cnpg.io/Backup
exists, and waits, with a WaitingForOtherBackup condition, while
another running BackupJob of the app has a Backup in flight. The check
reads the informer cache, so jobs released together, or one created
while another's Backup is not yet cached, can still both proceed; an
atomic per-application claim is a separate change.
Write errors that cannot succeed on retry (invalid, bad request,
forbidden) still fail the job at once. Any other is retried for the
backup deadline counted from its first failure since the job last left
the queue, recorded in a DestinationApplied condition. Once the Backup
exists the Cluster is only read, and a failed read is returned for a
controller retry instead of failing the running backup. The plugin
rollout deadline counts from the attach rather than from the job's
start, so queueing before the attach no longer shortens it.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->
## What this PR does
Part of #5201. The remaining part, a bound on a Backup that stays
`pending`, is in #5209.
A Postgres BackupJob whose `cnpg.io/Backup` had failed stayed `Running`
for 30 minutes with reason `CNPGBackupTransientFailure`, on the premise
that CloudNativePG may move a Backup out of the failed phase. It does
not: in CloudNativePG 1.30.0 the backup controller returns early on any
Backup already `failed` or `completed`, so the wait only delayed the
verdict. The BackupJob now fails as soon as the Backup is `failed`, with
the error CloudNativePG recorded. When that error is only the plugin's
exit status, the message points to the `plugin-barman-cloud` sidecar
log, where the storage error (`AccessDenied`, a connect timeout) is.
A Backup in any other phase keeps the BackupJob waiting as before, but
the BackupJob now carries a `Ready=False` / `WaitingForCNPGBackup`
condition naming the Backup and its phase instead of no condition at
all. These phases stay uncapped on purpose: CloudNativePG keeps a
healthy Backup `pending` while another backup of the same Cluster runs
and while it restarts one on another instance, so a fixed deadline here
would fail healthy backups, in particular scheduled ones that queue
behind a long previous run. #5209 tracks bounding a Backup that is stuck
rather than queued, together with the Plan concurrency that makes
queueing common.
A table-driven test drives the reconciler through each Backup phase and
asserts the job phase, the message, the persisted condition, that an
unchanged phase does not rewrite the BackupJob status, and that only a
completed Backup produces a Cozystack `Backup`.
### Screenshots
<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->
### Downstream repositories
<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.
Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.
IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:
1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
fix(postgres): a Postgres BackupJob now fails as soon as its CloudNativePG Backup fails, with the recorded error, instead of after 30 minutes, and a BackupJob waiting on a CloudNativePG Backup reports a WaitingForCNPGBackup condition with the Backup's phase.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **Bug Fixes**
* CNPG backups that enter a failed state now fail their BackupJob
immediately, with the recorded error shown when available.
* BackupJobs waiting for a CNPG backup to finish remain in a waiting
state, even after the backup deadline; the deadline applies while
waiting for the backup to be created.
* Waiting status now includes a clear reason, and failure messages
provide a sidecar-log hint when relevant.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Refs #5205.
The S3 endpoint the platform backups use is copied twice, and neither
copy is refreshed after `publishing.host` changes. The COSI sidecar
writes `BucketInfo` once per `BucketAccess`, so the credentials Secret
`bucket-cozy-backups-system-credentials` rendered from it keeps the old
endpoint. The Velero BSL and the default Strategy CRs keep whatever
endpoint the last render of `backupstrategy-controller` saw. The default
objects gate only acted on missing objects, so backups kept going to the
old host and nothing reported it.
The gate now also forces a render when a copy is stale:
- the bucket's `<bucket>-system` release, when the credentials Secret's
endpoint differs from the COSI Secret it is rendered from;
- `backupstrategy-controller`, when the endpoint an object was rendered
with differs from the credentials Secret's. The chart stamps the
rendered endpoint on the BSL and on every default Strategy as
`backups.cozystack.io/endpoint`, because the endpoint itself sits at a
different path in every kind. Objects without the annotation count as
stale until a render stamps it; `MinForceInterval` bounds the forces.
Stale credentials are forced before the BackupClass and the routed
objects are read, so an absent BackupClass or an unreadable Strategy
still fails the check but does not keep the old endpoint in the
credentials Secret. A `BucketInfo` without an endpoint is not compared,
since rendering from it would write an empty endpoint or fail the bucket
release. Repairing missing objects keeps working while the credentials
comparison is stale or fails, and a `systemSecretName` that is not
derived from a COSI Secret (no `-credentials` suffix, or the raw COSI
Secret itself) is not compared at all. RBAC is unchanged: the controller
already holds cluster-wide `get` on Secrets. No new metric: nothing
would consume it.
What this does not do: the COSI Secret still holds the endpoint the
`BucketAccess` was granted with, so after a host change the endpoint
follows only once an operator deletes the `BucketAccess` and forces
`bucket-cozy-backups` to recreate it (this issues new S3 keys). From
there the gate replaces the remaining manual steps of the recovery
described in the issue. Automatic re-grant is left for a separate issue,
and the operator runbook goes to the website, hence `Refs` rather than
`Fixes`.
Known limitation: the credentials comparison runs only when
`backupStorage.systemSecretName` is
`bucket-<bucketName>-system-credentials`, the Secret the forced
`bucket-<bucketName>-system` release renders. If an operator sets
`bucketName` and `systemSecretName` inconsistently, the comparison is
switched off rather than forcing a release that never rewrites the
compared Secret, so a stale endpoint in that Secret is not repaired.
Repairing a missing credentials Secret still forces the release derived
from `bucketName`, as before this PR.
Tested with unit tests for the gate (including the comparison
normalisation shared with the chart helpers), helm-unittest for the
annotation and the wiring, and a bats guard that every rendered Strategy
CR and the BSL carry the annotation. On a live cluster, a server-side
render of the chart produced exactly the annotation the gate derives
from the real credentials Secret, so the steady state does not force
anything.
### Screenshots
Not a UI change.
### Downstream repositories
- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: an operator runbook for changing `publishing.host` with
platform backups enabled (re-grant the `BucketAccess`, then let the gate
converge; restart ClickHouse pods that read the credentials from
environment variables). Not filed yet, pending approval of the text.
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(backups): the backup controller now re-renders the Velero BSL, the default backup strategies and the platform bucket's credentials Secret when their S3 endpoint is stale, so after the platform bucket's BucketAccess is re-granted for a new publishing.host, backups move to the new endpoint without forcing releases by hand.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **New Features**
* Backup strategies and storage locations now record their storage
endpoint, enabling detection and repair of outdated endpoint settings
after bucket access changes.
* Bucket credentials and backup objects are refreshed independently when
their endpoints are missing or out of date.
* **Documentation**
* Clarified when endpoint changes trigger updates and how suspended
releases are handled.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
An object without backups.cozystack.io/endpoint stopped counting as
stale as soon as one forced render had been requested, whether or not
that render succeeded. A failed upgrade left the objects unannotated
and the gate silent until the controller restarted, which also hid the
climbing force counter meant to report exactly that failure. Such an
object now stays stale until a render stamps it; MinForceInterval
already bounds the cost.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
A BucketInfo without an endpoint differs from every credentials Secret
that has one, so the gate forced the bucket release on every throttle
window. That render cannot converge: an empty endpoint would be written
into the Secret every driver reads, and with the key absent the
template's trimPrefix fails and parks the release in Failed.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The credentials endpoint comparison needs only the credentials Secret and
the COSI Secret it is rendered from, yet its forced render was issued only
after the BackupClass and every routed object had been read. A missing
BackupClass or one unreadable Strategy therefore kept the old endpoint in
the credentials Secret, which the projector hands to every driver, not
just the one whose object failed.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The credentials comparison returned early when the credentials Secret
was stale or the COSI Secret could not be read, so a missing default
Strategy or BSL was no longer created on that tick, nor on any later
one while the bucket release stayed suspended, kept failing, or the
COSI Secret stayed unreadable. Repairing missing objects is what the
gate did before the comparison existed, and it does not depend on the
answer.
Run the object check on every tick and force each release for what it
renders, reporting a comparison error alongside instead of in place of
the repair.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
backupStorage.systemSecretName may name a Secret the bucket chart does
not render from a COSI Secret: the raw COSI Secret itself, which the
projector accepts, or any Secret without the -credentials suffix. The
COSI Secret name was then the source name unchanged, so the gate
compared a Secret with itself and either reported it stale on every
tick or failed the check, and stopped repairing missing default objects
as well. Skip the comparison when there is no distinct COSI Secret or
no endpoint key to compare.
The gate compares the annotation with what the chart's endpoint
helper renders, and the helper renders a present but blank endpoint as
a bare https:// instead of falling back, so the gate has to compute
the same value or its expectation and the chart disagree for that
Secret. The gate's docs, metric help and log lines named only missing
objects, so they did not cover a stale endpoint they now report.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
## What this PR does
A ClickHouse `RestoreJob` restored the newest archive under the app
prefix, whichever `Backup` it named. Strategy pod generated the archive
name with `date -u` and nothing was recorded on the `Backup`, so restore
could only list the remote archives and take the last one. Restoring an
older `Backup` brought back the latest data.
Now the controller names the archive before it creates the backup Job
(`clickhouse-<app>-<startedAt>` from persisted `status.startedAt`, so
retried pods write the same one), passes it to the pod as `.ArchiveName`
and records it in the `Backup` `driverMetadata`. Restore takes exactly
that archive and fails with its name when storage has no such archive.
The name is recorded only when the Job's pod got it, so a Job left from
the previous controller during an upgrade, or a custom strategy with its
own names, leaves no name instead of a wrong one.
A retried pod reuses the name, so it waits for an attempt the sidecar
still runs (no 423) and asks for `resume`.
A `Backup` outlives its `BackupJob` and a Plan names `BackupJob`s after
the schedule slot, so a `BackupJob` created again under a used name met
the earlier run's `Backup` when its Job completed, took it as its result
and left the archive it just wrote recorded nowhere. Now it fails if
that `Backup` records another archive, and the message names both. A
retry of the same run still gets its own `Backup` back.
Old `Backup`s have no recorded name and stay restorable: the pod logs a
`WARNING` and takes the newest archive stamped no later than `takenAt`,
and fails when nothing qualifies. While the chart rolls out, previous
controller pod can still render the new template and pass no `takenAt`,
so a restore of a `Backup` without recorded name fails before it touches
any table, and works when created again after the rollout.
Fixes https://github.com/cozystack/cozystack/issues/5199
Tests are go unit tests for naming, recording, a reused `BackupJob` name
and restore render, and `hack/altinity-backup-strategy.bats`, which runs
the shipped script against a stubbed clickhouse-backup API (the restore
tests go red on the old template). Checked on a live ClickHouse with two
backups of 1500 and 2000 rows: restore of the older one by recorded name
ends with 1500 rows, fallback for an unnamed older backup picks the same
archive with the warning, a missing archive fails without touching the
table, a retry over an existing local backup completes under the same
name. The controller binary itself was not run on the cluster, only its
rendered output in the pod.
### Downstream repositories
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(clickhouse): a RestoreJob now restores the archive its Backup was taken as, not the newest one. New Backups record the archive name, Backups taken before this release are restored best effort from the newest archive stamped no later than their takenAt, with a warning in the restore pod log. A ClickHouse BackupJob created again under a name whose Backup still exists now fails instead of reporting that Backup as its result.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **New Features**
* Backups now use a consistent archive name tied to the backup job, and
completed backups retain that name for future restores.
* Restores can select the recorded archive and backup timestamp. If no
archive was recorded, the system selects the newest eligible archive for
the application.
* **Bug Fixes**
* A restore now fails when its recorded archive is missing instead of
silently selecting a different one.
* Backup requests avoid starting a duplicate action when the matching
action is already in progress.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
A Backup outlives the BackupJob that made it, so a BackupJob created
again under a name already used meets the earlier run's Backup when its
own Job completes. That happens by hand, and from a Plan reconciled
again inside its starting deadline, since a Plan names each BackupJob
after its schedule slot. The Altinity driver took the earlier Backup as
its result: the BackupJob turned Succeeded with a BackupRef recording
the earlier run's archive, the archive this run wrote was recorded on
nothing, and restoring what the BackupJob reported brought back the
older data.
An existing Backup is now taken as this run's only when it records this
run's archive, which is what an earlier reconcile of the same run
leaves when its status write is lost. Any other fails the BackupJob,
naming both archives, and leaves the earlier Backup untouched.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The S3 endpoint the platform backups use is copied twice and neither
copy is refreshed. The COSI sidecar writes BucketInfo once per
BucketAccess, so the credentials Secret rendered from it keeps its
endpoint, and the Velero BSL and the default Strategy CRs keep whatever
endpoint the last render of backupstrategy-controller saw. The default
objects gate only acted on objects that were missing, so after a
publishing.host change every backup kept going to the old host with
nothing reporting it.
The gate now also forces a render when a copy is stale: the bucket's
<bucket>-system release when the credentials Secret's endpoint differs
from the COSI Secret it is rendered from, and backupstrategy-controller
when the endpoint an object was rendered with differs from the
credentials Secret's. That endpoint sits at a different path in every
kind, in some only as a bare host, so the chart stamps the rendered
value on each object as backups.cozystack.io/endpoint and the gate
compares that. Objects rendered before the annotation existed count as
stale until one forced render has gone through, which stamps it.
The COSI Secret itself still holds the endpoint the BucketAccess was
granted with; it follows a host change only once the BucketAccess is
re-granted.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Every backup of a MariaDB application writes its dump under one shared
prefix, so the probe took the dump whose file-name timestamp (the
backup pod's clock) and LastModified (the object store's clock) both
fall in the operator Backup's lifetime, with a minute of slack for
skew. That keeps a neighbour's dump out only while this run's own dump
is in the window. With the pod clock more than the slack behind, the
own dump is named before the window opens, and a previous run that
finished within the minute before this one was created becomes the
single match: a 123-byte dump was reported as 666 bytes. A pod clock
ahead or an object store clock behind does the same with the next run,
and an own dump that is gone by the time it is looked up, as it can be
for a restore long after the backup, leaves a neighbour as the single
match with no skew at all.
A dump is now attributed only when ownership is proven:
- it is the one dump that either clock places in the window. A dump
that one clock places there and the other does not may be this run's
own with a skewed clock, so it leaves any other match unproven, the
way a second dump in the window already did, and is not taken itself;
- both of its clocks place it in the window;
- neither clock places it in the window of another operator Backup
that writes under the same bucket and prefix and has completed. This
rejects a neighbour's dump when this run's own is missing, which the
clocks alone cannot tell.
Operator Backups still running are not counted as neighbours: their
window has no end, and one stuck short of Complete (a Job whose pod
never starts stays JobRunning for good) would otherwise keep every
later run of the application from being sized. A dump such a run
writes inside this run's window still competes with this run's own.
Narrower slack does not prove ownership: a neighbour that finished
within the narrower slack still matches, and a smaller skew pushes the
own dump out. The Job's start and completion times are stamped by the
control plane, like the operator Backup's own, so they narrow the
window by seconds without removing the dependence on the two clocks a
dump carries. What is left: a neighbour's dump is taken only when this
run's own is missing from the listing or misplaced on both clocks and
the neighbour's operator Backup no longer exists or has not completed.
Every other unproven case leaves the size unset, as two runs that
overlap or follow each other within the slack already did.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The size probe took the bucket, endpoint, prefix, credentials and TLS
settings from the strategy as it renders on the reconcile that creates
the Backup, which is not necessarily what it rendered when the operator
Backup was created: an existing operator Backup is reused untouched,
and a backup can run for up to the driver's 30-minute deadline. An
admin who edits the strategy in between sends the probe to the new
location, where it measures the wrong prefix or nothing at all.
The operator Backup's own spec.storage.s3 is where the dump went, so
the probe reads that. It is also the only source a restore can use,
since it targets that operator Backup long after the strategy may have
changed. An operator Backup that does not store to S3 is refused before
anything is listed.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
A MariaDB restore has to name the dump it restores from: without a
target time the operator restores the newest dump under the prefix,
which is not necessarily the one of the Backup being restored. The size
probe already finds that dump, but it returns only its size, folds a
missing, an ambiguous and an unreadable listing into the same 0, and
lives on BackupJobReconciler, which the restore reconciler is not.
Split the probe in two: findMariaDBDump reads the bucket and returns the
matched object (key, file name, the timestamp in its name, size and
LastModified) or an error that tells "not found" from "ambiguous"; the
size path keeps its timeout and reads only the size. The Secret lookup
and the S3 client move onto an objectStore value that any reconciler
can build from its own client, and BackupJobReconciler keeps one-line
wrappers so the CNPG probe is unchanged. sizeBytes is reported exactly
as before.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
mariadb-operator marks a Backup Complete=True for a failed Job as well
as a successful one; only the reason tells them apart (JobComplete with
message Success, or JobFailed with message Failed). The fixtures that
stand for a finished backup set Complete=True and nothing else, so they
describe a state the operator never produces and would read as a
failure to a driver that checks the reason. Give them the success shape
the operator writes.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
With an endpoint CA the probe trusted the system roots plus that CA.
barman-cloud gets endpointCA as AWS_CA_BUNDLE, which replaces the
default trust store, so a CNPG probe could accept an endpoint the backup
itself would have refused. Let the connection say whether the CA adds to
the system roots or replaces them, and replace them for CNPG.
mariadb-operator appends its CA to the system pool, so MariaDB keeps
that.
Pin in reconcile tests that a Create the apiserver rejects still fails
the BackupJob, so only the read-back of an existing Backup is retried.
Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
A Backup that exists but cannot be read back is retried through the
controller's backoff, which left the BackupJob Running with nothing on
it saying why. Set Ready=False with reason BackupNotReadBack before
handing the error back.
When the check for an existing Backup itself fails, the Backup may well
exist, so a size probe would only be discarded, and a read that keeps
failing reran it, up to the probe timeout on the single BackupJob
worker, on every retry. Skip the probe in that case and create the
Backup without a size.
Drive both behaviours through the MariaDB and CNPG reconciles in tests.
Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
When Create collides with a Backup an earlier reconcile already made and
the read that follows fails, the drivers failed the BackupJob, although
the backup and its Backup both exist. Return a distinct error instead,
which the drivers hand back to the controller so the reconcile is
retried with backoff and then finds the Backup.
minio.New already refuses an endpoint with a path prefix and accepts a
trailing slash, so drop the duplicate check in the shared client; the
CNPG probe keeps its own check because it parses the URL itself and
would otherwise drop the path before minio sees it.
Say in the sizeBytes API doc what the value covers, such as the base
backup without the shared WAL archive for CNPG, and that strategies
which cannot attribute stored bytes to one backup leave it unset.
Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
The check for a Backup an earlier reconcile already created returned
its read error, and the drivers turn any error from that path into a
terminally Failed BackupJob. An apiserver timeout right after the
operator backup completed therefore lost a backup whose data was safely
in S3. The check only saves a probe, and the Create that follows already
handles an existing Backup, so log the error and go on to Create.
A trailing slash on the S3 endpoint is not a path prefix; the operators
accept it, so do not refuse it.
Pin the uncached AlreadyExists refetch, the upper slack of the MariaDB
window and the probe timeout with tests.
Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
The MariaDB probe identified this run's dump by the timestamp in its
file name, which comes from the backup pod's clock. With that clock
skewed past the slack, this run's dump falls outside the window while a
neighbouring run's dump can be the single match left, and its size is
reported as this backup's. A dump now has to fall inside the window by
both its name and its LastModified, which the object store stamps, so a
single skewed clock leaves the size unset instead.
The name parser also accepts the layout later mariadb-operator releases
write for compressed dumps, backup.<ts>.sql.<ext>, and checks the
compression token in either layout, so an operator upgrade does not
silently stop the measurement.
The retry check and the AlreadyExists refetch read the Backup through
the cache. A retry usually follows a Create made moments earlier; a
cache that does not show it yet turned the refetch into NotFound and
failed a backup that had succeeded. Read it from the API server instead.
Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
An endpoint such as https://s3.example.org/prefix cannot be addressed
by minio-go, which has no path-prefix option, and the prefix cannot be
added below the client either, because the SigV4 signature covers the
request path. The CNPG probe was worse than failing: it kept only the
host of endpointURL and listed the bucket at a location barman never
writes to.
Reject such endpoints with an explicit error, in the CNPG probe and in
the shared client, so the size is left unset instead of measured
somewhere else.
Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
minio-go ends a listing whose context is cancelled between pages by
closing the channel without an error item. When the probe timeout
expired between pages of a large CNPG base backup, the pages already
read were summed and reported as the measured size. Check the context
once the listing channel closes and leave the size unset instead.
The probe ran before the Backup was created, so a reconcile that retried
after the Backup already existed probed the bucket again, held the
single BackupJob worker for up to the timeout, and then discarded the
result in favour of the existing object. Return an existing Backup
before probing.
Log the ambiguous MariaDB case (several dumps in the window) at info like
the other reasons the size stays unset, and cover the session-token
credential path with a test.
Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
The probe runs inside the reconcile that creates the Backup, and the
BackupJob controller has a single worker, so its timeout delays every
other BackupJob while an endpoint hangs. Lower it from 30s to 10s.
List the MariaDB prefix non-recursively, as mariadb-operator does. The
default MariaDB prefix and the default CNPG destinationPath are the same
<namespace>/<name>/ path, so a recursive listing of a MariaDB backed up
next to a same-named Postgres walked the whole WAL archive and ran out of
time. The CNPG base-backup prefix is still listed recursively.
Build the custom-CA transport from minio.DefaultTransport so it keeps the
proxy-from-environment and dial/TLS timeout settings a bare
http.Transport drops, and cover it with a test against a TLS endpoint
with a self-signed certificate.
Log the reasons the size could not be measured (credentials, client,
listing, empty CNPG prefix) at info. The chart runs the controller at
info, which hid them before. The expected ambiguity of several MariaDB
dumps in the window stays at debug.
Skip an endpointCA without a Secret name, as buildBarmanObjectStore does.
Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
CNPG backups arrived without Backup.status.artifact.sizeBytes:
cnpg.io/Backup status carries no size, and the data lives in barman's
object-store layout rather than in a single object.
barman-cloud uploads every base backup under
<destinationPath>/<serverName>/base/<backupID>/ (backup.info plus the
data and tablespace tarballs, possibly split into parts). On completion
the driver now lists that prefix with the credentials and endpoint CA of
the rendered barmanObjectStore and records the total. The WAL archive is
not counted: it is one stream shared by every base backup of the server.
The field stays unset when the backup ID is unknown, the bucket cannot be
read, or nothing is stored under the prefix.
The S3 client and Secret lookup are now shared with the MariaDB driver.
Refs #4316
Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Backup.status.artifact.sizeBytes was only ever set by the etcd strategy,
so MariaDB backups always arrived without a size.
mariadb-operator writes each logical dump as
<prefix>/backup.<RFC3339>[.<compression>].sql under a prefix shared by
every backup of the application, and the URI the driver records is
synthetic, so there is no single object to HEAD. On completion the driver
now lists the prefix with the tenant credentials the operator Backup was
rendered with and takes the size of the one dump whose timestamp falls
inside the operator Backup's lifetime. When the bucket cannot be read or
the window does not match exactly one dump, the field stays unset.
The API doc now states that an absent sizeBytes means "unknown", and the
Velero strategy documents why it leaves the field unset: kopia stores
data content-addressed across backups, so there is no per-backup stored
size.
Refs #4316
Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
A failed cnpg.io/Backup kept its BackupJob Running for 30 minutes as a
"transient" failure, on the premise that CloudNativePG may move a
Backup out of the failed phase. It does not: its backup controller
returns early on any Backup already failed or completed, so the wait
only delayed the verdict and hid it behind a misleading reason.
A failed Backup now fails the BackupJob at once with the error CNPG
recorded, pointing to the plugin-barman-cloud sidecar log when that
error is only the plugin's exit status. A Backup in any other phase
keeps the BackupJob waiting as before, now with a WaitingForCNPGBackup
condition naming the Backup and its phase instead of no condition at
all. It is still not capped: CloudNativePG keeps a healthy Backup
pending while another backup of the same Cluster runs or while it
restarts one, and a fixed deadline would fail those.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The Altinity strategy named each archive inside its Pod with date -u
and recorded nothing on the Backup, so a restore could only guess: it
listed the remote archives under the application's prefix and took the
newest. A RestoreJob of an older Backup therefore restored whichever
backup had run last.
The controller now names the archive before it creates the backup Job,
from the BackupJob's persisted StartedAt, so every reconcile and every
retried Pod of the Job writes the same one. The name is recorded in the
Backup's driverMetadata next to the strategy parameters, and a restore
renders it into the strategy Pod, which restores exactly that archive
and fails, naming it, when remote storage no longer holds it. The name
is recorded only when the Job's Pod was given it, so a Job created by
an earlier controller, or a strategy that names its own archives,
leaves the Backup without one instead of pointing it at an archive
that does not exist.
Because a retried Pod reuses the name, it waits on an attempt the
sidecar is still running instead of being refused with 423, and asks
for resume, so a local or partial remote backup an earlier attempt left
is completed rather than refused as existing.
Backups taken before names were recorded stay restorable: the Pod picks
the newest archive whose name is stamped no later than the Backup's
takenAt, which is set only after the backup Job finished, and logs a
warning that the choice is a best effort.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Seven guides under `docs/` were operator facing and written alongside
the code, and left in the source tree where no user looks. They are on
the website now, so this deletes them here and repoints every reference.
```
docs/operations/backup-classes.md -> operations/services/backup-classes
docs/operations/resource-presets.md -> guides/resource-management
docs/storage-immutability.md -> operations/storage-class-immutability
docs/operations/system-memory-limits.md -> operations/configuration/system-memory-limits
docs/gateway-classes.md -> networking/gateway-classes
docs/observability/distributed-tracing.md -> operations/services/monitoring/distributed-tracing
docs/operations/seaweedfs-capacity-for-backups.md -> operations/services/object-storage/capacity-for-backups
```
`docs/release.md`, `docs/deprecating-applications.md` and the two
SeaweedFS runbooks in `docs/operations/` stay. A workflow, `AGENTS.md`,
a migration and a bats suite read them by path, which is the test from
#3777.
References point at `https://cozystack.io/docs/v1/...`. The site
rewrites that short prefix to the latest version in `layouts/404.html`,
so the links follow releases and pin none. Backup Classes, resource
management and `storageClass` immutability exist in the latest version
already. The other four pages are only in `next/` until 1.7.0 is
promoted, and their links start to resolve then. The generated mongodb
files come from `values.yaml` through `make generate` with
cozyvalues-gen v1.7.0, the version pinned in
`.github/workflows/pre-commit.yml`.
Tests: helm unit tests for mongodb (151), postgres (209), vm-disk (114)
and kubernetes-nodes (223), go tests for backupcontroller, presets and
the operator, rd-presets-check, the migration-50 bats suite.
### Downstream repositories
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#763
Walked the rest of the trigger map against the diff. No provider
follow-up: the mongodb `values.schema.json` diff is one `description`
string, with no field added, removed or renamed, no enum, no default,
and no `kind`, `plural` or `release.prefix` change.
`packages/core/platform/values.yaml` changed in comments only and no
key, so the hand written platform-package table on the site is
unaffected.
Please merge after cozystack/website#763, the links in the code only
resolve once those pages deploy.
### Release note
```release-note
docs: seven guides move from the docs/ directory of this repository to cozystack.io (Backup Classes, resource presets, storageClass immutability, system component memory limits, GatewayClass selection, distributed tracing, SeaweedFS capacity for backups), and in-repo references point there.
```
Seven guides in docs/ were written next to the code and left in the
source tree, where no operator looks. The website carries them now, so
a second copy here only lets the two drift again: the Backup Classes
page had already fallen behind its original.
docs/release.md, docs/deprecating-applications.md and the two SeaweedFS
runbooks in docs/operations stay. A migration, a bats suite, a workflow
or AGENTS.md reads each of them by path, which is the test for keeping
a document in this repository.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The restore controller and the user-restore example described the
allowClaimAdoption annotation its resource modifiers put on restored
PVCs as what lets CDI adopt them. CDI v1.66 reads that annotation only
on the DataVolume; on a PVC it reads nothing but createdForDataVolume.
Adoption rests on the vm-disk chart annotating the DataVolume Flux
recreates, and a reader trusting the PVC rule could rely on it for a
restore without such a DataVolume, or drop the DataVolume annotation as
redundant.
Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
collectUnderlyingResources returned nothing for a non-VMInstance kind, so
prepareForRestore did nothing for a standalone VMDisk: no HelmRelease suspend and
no DataVolume delete. An in-place VMDisk restore then raced Flux/CDI — the
restored HelmRelease reinstalled and CDI recreated the DataVolume over the same
PVC name the data mover was repopulating, wedging the RestoreJob.
Report a standalone VMDisk's own disk as its underlying resource, so the existing
(VMInstance-proven) restore preparation suspends the vm-disk-<name> HelmRelease,
deletes its DataVolume and — with keepOriginalPVC — renames the live PVC before
the data mover recreates and repopulates it. No VM, so no IP/MAC is collected.
A VMInstance is halted before its disks are touched; a VMDisk target is not,
and every VMInstance disk is a VMDisk. Renaming the PVC of a disk a running VM
mounts leaves the original Terminating behind pvc-protection, Velero skips a
PVC that still exists, and the RestoreJob would report Succeeded having moved
nothing while the VM loses its disk on the next restart. An in-place VMDisk
restore now fails before changing anything when a Pod that has not terminated
mounts the disk's PVC, naming the Pod and telling the user to stop the VM or
restore the VMInstance instead.
A VMDisk Backup taken before the disk was recorded as its underlying
resource has none, so its disk is now derived from the application name
and such a restore is prepared and checked for a mounting Pod as well.
The per-disk orLabelSelectors stay a VMInstance concern. A VMDisk
strategy already selects its disk, and Velero rejects a Backup or
Restore that carries both labelSelector and orLabelSelectors, so
appending them for a VMDisk would break a strategy written with
labelSelector.
When an in-place restore fails after preparation, nothing is rolled
back: resuming the disk's HelmRelease while its PVC is renamed and its
DataVolume deleted would have CDI import a fresh disk under the original
name, which looks healthy and hides that the data sits in the -orig-
PVC. The failure message and a Warning event now name the suspended
HelmReleases and the -orig- PVCs so the user can recover them, read back
from the cluster, so a failure before a step ran does not name what it
would have left.
Preparation takes the target apart before Velero validates the Restore,
so it is refused up front when the Velero Backup is gone or not
Completed. That check now runs on every pass behind the same failure
path as the rest of preparation, and a FailedValidation Restore fails
the RestoreJob like a Failed one instead of leaving it Running. A
HelmRelease suspend or a VM halt that did not happen now stops
preparation instead of renaming and deleting under a running release or
VM, and a VMInstance's disks are renamed only once its launcher Pod has
let go of them. A VMDisk held only by a Pod already being deleted is
waited for rather than refused, for
up to 30 minutes after the RestoreJob started.
Between renaming a kept disk's PVC and deleting its DataVolume, CDI can
recreate an empty PVC under the original name. Velero then skips the
PVC as already present, the RestoreJob reports Succeeded and the VM
boots a blank disk. Once the DataVolume is gone, preparation now
deletes a PVC found under the original name, read uncached, before the
Velero Restore is created.
Preparation spans several reconciles, so the RestoreJob records a
PreparationStarted condition before it first changes the target, and
retries without changing anything when recording it fails. A
refusal on a later pass (the Velero Backup expired, a VM started on the
disk) then names what preparation left behind instead of claiming that
nothing changed, and a preparation still waiting 30 minutes after it
started fails the same way instead of polling forever.
Now that a VMDisk target is prepared, keepOriginalPVC=false reaches it
too: its DataVolume is deleted with orphan propagation and the bound PVC
is deleted before the Restore, as for a VMInstance disk. Left in place,
Velero would skip it and CDI would refuse the DataVolume the resumed
HelmRelease creates.
Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Valkey runs on the same spotahome RedisFailover operator as Redis and
speaks the same replication protocol and RDB format, so it needs no
strategy kind of its own. The only difference the backup Job sees is the
chart's resource-name prefix: valkey-<app> instead of redis-<app> for the
sentinel Service and the auth Secret.
The Redis driver now accepts applicationRef kind Valkey as well as Redis
and hands the template .Release.Prefix, derived from the app's kind, which
cozy-default-redis uses for every name it builds. A kind with no known
prefix fails the render instead of producing an empty one. cozy-default
routes apps.cozystack.io/Valkey to cozy-default-redis. A restore must
target an app of the same kind as the backup's source: replaying a Redis
dump into Valkey, or the reverse, is not something the strategy can
promise across engine releases.
The shipped template reads the prefix as `.Release.Prefix | default
"redis-"`: during an upgrade the previous controller, which has no Prefix
in its render context and serves only Redis, renders the new CR, and a
bare reference would print "<no value>" into the names.
The redis:8 image already loads RDB v11, which both shipped Valkey
versions (7.2, 8.1) write.
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-By: LLM