Commit Graph
141 Commits
Author SHA1 Message Date
Andrey Kolkov 906eeb2372 feat(backups): add Kubernetes backup strategy for tenant clusters (#4931)
## What this PR does

Adds a `Kubernetes` backup strategy so a tenant Kubernetes application
can be backed up and restored through the backups framework, and routes
`apps.cozystack.io/Kubernetes` to it in `cozy-default`.

None of the existing strategies fits a tenant cluster:

- Its API state is a key prefix (`/<namespace>_<release>/`, the
TenantControlPlane's datastore schema) in the etcd `DataStore` shared by
every cluster of the tenant. The `Etcd` strategy snapshots the whole
DataStore, so restoring it would roll every tenant cluster back at once.
- Its control-plane CA, front-proxy CA, service-account key and Talos
secrets cannot be regenerated: new ones break every node, kubeconfig and
token issued under the old ones.
- Its PersistentVolumes are DataVolumes in the tenant namespace
(`pvc-<tenant pvc uid>`, labelled `cluster.x-k8s.io/cluster-name`), and
deleting the application deletes them.

**Backup** (`strategy.backups.cozystack.io/Kubernetes`, driver in
`internal/backupcontroller/kubernetesstrategy_*.go`):

1. Checks the control-plane and Talos certificates and sets a
`CertificatesExpiring` condition (plus a Warning event) when a CA
expires before the backup's TTL or a leaf within
`certificateWarningThreshold` (720h). This never fails the run.
2. A Job in the application namespace dumps the cluster's prefix with
`etcdctl` to the platform bucket. It authenticates with the cluster's
own `kubernetes-<name>-datastore-certificate`, the one its
kube-apiserver uses. It runs on a new `kubernetes-backup` image (alpine
+ curl + jq + etcdctl matching the platform's etcd).
3. A Velero backup takes the PKI Secrets (selected by
`kamaji.clastix.io/name` / `app.kubernetes.io/instance`; the driver adds
the instance label to the cert-manager-issued Talos CA Secrets, which
carry neither) and the data DataVolumes/PVCs. Worker system disks are
not backed up.
4. The Backup records the prefix, the endpoints, the data DataVolumes
and the HelmReleases of the cluster and its `KubernetesNodes` pools. The
HelmReleases are not in Velero because the lineage webhook labels worker
disks after the same applications, so a label selector for the releases
would also pick the disks.

**Restore** goes into the application the backup came from (rename /
cross-namespace is rejected). `RestoreJob.spec.options` takes `etcd` and
`volumes` (both default `true`) to restore either half alone:

- **Application gone**:
  1. Restore the PKI Secrets and the volumes.
2. Recreate the cluster's HelmRelease. The Kamaji release this repo
ships gives a new TenantControlPlane a UID-keyed datastore schema, so
the restore waits for that schema.
3. Stop the new control plane, write the dump under the new prefix (keys
are moved from the old prefix), and start it again.
4. Recreate the worker pools last, so workers join the restored cluster
with the restored CA and Talos secrets rather than an empty one.
- **Application live**: suspend the cluster's HelmReleases, then stop
the MachineDeployments and the cluster-autoscaler (for `volumes`,
deleting the backed-up DataVolumes/PVCs once the Machines are gone), and
then the KamajiControlPlane (for `etcd`). Workers go first because CAPI
removes a worker's Node through the tenant API. Then write the selected
halves back and restore every replica count. Each original count is
recorded on the object it belongs to, so the cluster is put back on
success, on failure, after a 30-minute stop deadline, and when the
RestoreJob is deleted midway. A retry keeps the counts from before the
first attempt, and a HelmRelease suspended before the restore stays
suspended.

Before changing anything, a restore checks that the Velero backup is
still there and Completed, and that the dump object exists (a
verify-mode run of the Job). The restore mode is recorded in the
RestoreJob's status, so a RestoreJob cannot claim that a live cluster is
gone.

Before it runs `del --prefix`, the restore Job downloads and validates
the dump and saves the current prefix to `<dump>.pre-restore`. That copy
is never overwritten, so a retry cannot replace it with a partial
prefix, and it is dropped after a successful restore. The Job writes a
marker to its termination message, and the outcome depends on it:

- A failure before `del` resumes the cluster.
- A failure after `del` leaves the control plane stopped
(`KubernetesEtcdWriteFailed`) rather than started on a partial prefix.
- Deleting the RestoreJob while its restore Job runs also leaves it
stopped.

Events and master leases are skipped only as top-level resources. Velero
cannot overwrite an existing claim, so a live-cluster volume restore
deletes the current claims first; this is documented.

A backup whose Velero half fails deletes the etcd dump it already
uploaded, and a transient error creating the Backup is retried rather
than failing the run. Deleting a Backup removes the Velero backup
(DeleteBackupRequest) and the dump object plus its pre-restore copy (a
cleanup-mode Job, given up after five failed attempts with a Warning,
plus the same skip-artifact-cleanup escape hatch as Redis/Rabbitmq).
With `velero.bslEnabled=false` the strategy is not rendered, and the
DefaultObjectsGate does not count it as missing, the same as the Velero
strategies. Like every default strategy, it is also gated on its CRD
being established, so the upgrade that first ships the CRD to an
installed chart cannot wedge the release.

Also included: RBAC for
TenantControlPlanes/DataStores/KamajiControlPlanes/MachineDeployments/Machines/Deployments,
chart values (`kubernetesBackupClientImage`,
`kubernetesBackupActiveDeadlineSeconds`, `kubernetesBackupResources`), a
`make image-kubernetes-backup` target, docs in
`docs/operations/backup-classes.md`, and the
`examples/backups/kubernetes/` walkthrough.

### Verification

- `go test ./internal/backupcontroller/`: the new tests cover:
- the certificate windows, pool HelmRelease discovery, option parsing
and the restore-phase selectors;
  - the full backup path;
  - in-place restores with default options and etcd only;
  - a pre-suspended HelmRelease;
- a failed etcd write leaving the control plane stopped, including on
RestoreJob deletion;
  - RestoreJob deletion resuming a stopped cluster;
  - the stop deadline;
  - the preflight: missing or incomplete Velero backup, missing dump;
  - a rebuild, and an unreadable HelmRelease record;
- rename rejection, a non-etcd datastore, and the cleanup attempt limit.

Each safeguard in the Go driver was checked by mutating it out and
confirming that a test fails. The rendered Job script has no committed
test: a unit test would have to extract the script from the rendered
chart, and its behaviour belongs in the chainsaw e2e follow-up instead.
It was run against stub `etcdctl`/`curl` in every mode and branch:
  - a foreign key or an empty value stops it before any etcd call;
- an S3 answer other than 200/404 on the pre-restore copy stops it
before any etcd call;
- an existing copy is kept, and a missing one is uploaded before `del`;
  - keys are moved from the old prefix to the new one;
  - an object in a namespace named `events` is kept.
- `helm unittest` (29/29), `hack/backupstrategy-crd-gate.bats` and `helm
lint` on `backupstrategy-controller`.
- On a live cluster (Velero 1.18.1), with a tenant cluster holding a
ConfigMap marker and a file on a PVC:
- The same flow done by hand before the driver existed: back up, delete
the application, rebuild. The CA, front-proxy CA, SA key and Talos
secrets were byte-identical afterwards, the pre-backup admin kubeconfig
still worked, namespace and ConfigMap UIDs were unchanged, and the PVC
data came back.
- Through this driver: the BackupJob succeeded (432 keys dumped, Velero
backup Completed, Backup metadata as described above). The tenant
namespace holding the markers was then deleted, which also deleted its
PVC and the backing DataVolume, and a RestoreJob with default options
rolled the live cluster back. It stopped the workers and then the
control plane, wrote the prefix back, restored the volume, and restored
the control plane to 2 replicas, the pool and the autoscaler to 1, and
unsuspended the HelmRelease. A new worker joined. The namespace came
back with its original UID, the ConfigMap and the PVC (bound to the same
volume) with their data. That run caught an ordering bug, now fixed and
covered by a test: scaling the control plane down first left the old
Machine stuck in `Deleting`, because CAPI removes the Node through the
tenant API.
- On the current head, through the driver, against the same tenant
cluster:
- **Backup.** It succeeded. The volume list it recorded was taken before
Velero started.
- **`{"volumes":false}` restore after a ConfigMap was deleted and
another created.** The deleted ConfigMap came back and the newer one was
gone. The worker was left running and the volume data was untouched. The
restore waited for terminating control-plane Pods before writing. The
pre-restore copy was saved before `del` and dropped afterwards.
- **Rebuild after the application was deleted, with default options.**
It ran the full sequence: secrets, volumes, cluster HelmRelease,
datastore schema, stop, etcd write, resume. It succeeded in about 3
minutes. The CA and Talos CA were byte-identical, a new worker joined,
and the namespace and ConfigMap UIDs and the PVC data were back. The
tenant cluster runs on a Kamaji that still names schemas `<ns>_<name>`,
so this run did not exercise a prefix change; that path is covered by
the unit tests and the stub runs.

### Not in this PR

- A chainsaw e2e. The tenant Kubernetes suites are bounded by tight op
budgets that tests pin, so the round trip needs its own suite or a
budget change; it will be a follow-up.
- `rotationPolicy` on the Talos CA Certificate in the kubernetes chart
(cert-manager's default `Always` would rotate the CA key at renewal).
- Restore into another application name or namespace.
- The tenant's shared etcd DataStore runs with auth disabled, so any
cluster's datastore certificate can read its neighbours' prefixes. This
predates this change; the dump Job uses the same certificate the
cluster's kube-apiserver already holds.

### Screenshots

N/A — no UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Walked against the diff: no package under `packages/apps|extra` is added
or renamed, `packages/core/platform` and installer values are untouched,
the new CRD is in `strategy.backups.cozystack.io` (not typed by
terraform-provider), and `RestoreJob.spec.options` stays opaque JSON.

### Release note

```release-note
feat(backups): tenant Kubernetes applications can now be backed up and restored through the cozy-default BackupClass. A backup captures the cluster's etcd state and its control-plane PKI and PersistentVolume data, and warns when a control-plane CA would expire before the backup does. A restore rebuilds a deleted cluster with its original CA or rolls a live one back, and can restore the etcd state and the volumes separately via RestoreJob options `etcd` and `volumes`.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

- **New Features**
- Added Kubernetes application backup and restore support, including
etcd data and persistent volumes.
- The default backup class now routes Kubernetes applications to the new
strategy.
- Added an end-to-end walkthrough for creating a cluster, backing it up,
restoring it in place or after deletion, and cleaning up.
- Backup status can report expiring certificates, and restore validation
helps protect cluster identity and state.
- **Documentation**
- Added guidance on backup contents, restore behavior, and walkthrough
configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 15:40:01 +07:00
Alexey Artamonov dae76d186c Merge branch 'main' into fix/clickhouse-idle-before-backup
main pins the archive name a backup Pod keeps across a wait for
late bucket credentials, against the controller's archive helper.
That helper now names a run's archive with part of the BackupJob's
UID, so the merged test has to give its BackupJob one.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-12 17:20:19 +03:00
Aleksei Sviridkin 3badbdef74 fix(clickhouse): let a running backup sidecar pick up system-bucket credentials (#5232)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

A ClickHouse release with `backup.useSystemBucket: true` can start
before the platform projects `cozy-backups-creds` into its namespace.
Its `clickhouse-backup` sidecar took the S3 settings from environment
variables, which are fixed when the container starts, so the first
BackupJob or RestoreJob of such a release failed with `no EC2 IMDS role
found` until the Pod was restarted, and a rotated key stayed stale the
same way.

The projector now also writes a `clickhouse-backup.yml` key into
`cozy-backups-creds` (bucket, `https://` endpoint, region when set,
keys, `force_path_style` from `backupStorage.forcePathStyle`). On the
system-bucket flow the chart mounts that key as an optional Secret
volume and points `CLICKHOUSE_BACKUP_CONFIG` at it, instead of setting
`S3_*` variables. clickhouse-backup reloads its config file in each API
handler that reaches S3 (among them the `create_remote`,
`restore_remote` and `list` calls the strategy makes; the `status` and
`actions` reads it polls with do not) and kubelet rewrites a mounted
Secret in place, so credentials projected after the Pod started, or
rotated since, reach the running sidecar. `S3_PATH` stays in the
environment because the prefix belongs to the release.

Kubelet takes about a minute to refresh the volume, so the projector
stamps `backups.cozystack.io/credentials-updated-at` when the projected
content changes, and the Altinity driver holds a new backup or restore
Job for up to two minutes after that stamp, with `Ready=False` and
reason `WaitingForCredentials`; the condition is dropped once the Job
runs. A stamp that cannot be read, or one further ahead than clock skew
explains, is logged and ignored rather than holding the namespace's jobs
for good. The hold applies to every Altinity job in a namespace whose
projected Secret just changed, including releases on their own Secret,
where it is only a delay.

The new e2e Test `clickhouse-3-system-bucket-roundtrip` starts a
ClickHouse on the system bucket with no projected Secret, then runs a
backup and an in-place restore through `cozy-default` and checks that
the sidecar that started without credentials served both jobs. In CI the
system bucket's advertised host `s3.example.org` neither resolves nor
gets an ACME certificate, so for the duration of its own step the Test
serves a SeaweedFS-CA certificate under the S3 Ingress TLS Secret name
and adds a marked `rewrite name exact` line for that host to the host
CoreDNS Corefile, removing both on exit; it proves the served
certificate before any job starts.

The bug was reproduced on a test cluster running the code before this
change: a ClickHouse started before the projection failed its first
BackupJob with `no EC2 IMDS role found`. The new e2e Test has not run
green yet outside CI, because it edits the host CoreDNS, which is shared
on that cluster.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#799 (draft until this PR merges):
`operations/services/backup-classes.md` for the mounted config file, the
credentials wait and the upgrade window, without the Pod restart this
change removes
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(clickhouse): a ClickHouse release on the platform system bucket (`backup.useSystemBucket: true`) now backs up and restores without a Pod restart when its namespace had no projected backup credentials at start, and picks up rotated credentials while running. The sidecar reads its S3 settings from a `clickhouse-backup.yml` file in `cozy-backups-creds` instead of environment variables, so the upgrade rolls every such ClickHouse Pod once. The first Altinity backup or restore in a namespace after its projected credentials change waits up to two minutes (`WaitingForCredentials`) for them to reach the sidecar. Upgrade the platform as a whole: a system-bucket BackupJob started while the new chart runs against the previous controller fails, because that controller does not write the new key.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* ClickHouse system-bucket backups now use a projected configuration
file for S3 settings, supporting HTTPS endpoints and optional region and
path-style settings.
* Backup and restore jobs wait briefly for recently updated projected
credentials before starting.
* Path-style settings accept common true and false values and are
normalized consistently across backup storage integrations.
* **Tests**
* Added end-to-end coverage for system-bucket backup and restore,
verifying restored data without restarting the ClickHouse Pod or backup
sidecar.
* Added checks for TLS certificate validation and safe restoration of
shared network and ingress settings.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-12 16:15:26 +03:00
Andrey Kolkov 0f44cf9e25 feat(backups): add Kubernetes backup strategy for tenant clusters
A tenant Kubernetes application had no backup route in cozy-default, and
none of the existing strategies fits it. Its API state is a key prefix in
the etcd DataStore the whole tenant shares, so the Etcd strategy's snapshot
would roll every tenant cluster back at once; its control-plane CA, service
account key and Talos secrets cannot be regenerated without breaking every
node and credential issued under them; and its PersistentVolumes are
DataVolumes in the tenant namespace that the application's own deletion
removes.

The new Kubernetes strategy backs up the cluster's etcd prefix with a Job
that authenticates with the cluster's own datastore certificate, and the PKI
Secrets and data DataVolumes with Velero. The HelmReleases of the cluster
and its node pools are recorded on the Backup rather than in Velero,
because a label selector for them would also pick up the worker system
disks. A backup warns when a CA would expire before the backup does.

A restore rebuilds a deleted cluster under its old CA, or rolls a live one
back after stopping its control plane and workers, and lets the etcd
prefix and the volumes be restored separately.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-11-10 15:28:34 +07:00
Alexey Artamonov d530ca8cf5 fix(backups): keep a ClickHouse archive an earlier attempt finished
A backup Pod that loses sight of its upload, by dying or by losing six
polls in a row, is retried while the sidecar finishes that upload. The
retry found it in the action log as a success, took the sidecar for
idle and asked for create_remote with resume again. clickhouse-backup
then rebuilds the local backup and uploads it once more over the
archive already in the bucket, which is left broken if that second
upload dies halfway.

The archive name belongs to one BackupJob, so a success row of exactly
this upload can only be an earlier attempt's, and the retry now takes
it as its result. An error or cancel row still leads to a resumed
upload, and a restore still runs again, since a restore is not named
per RestoreJob.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-09 16:39:46 +03:00
Alexey Artamonov 89f8cc02ee Merge branch 'main' into fix/clickhouse-idle-before-backup
main now names the ClickHouse archive in the controller and restores
the archive a Backup recorded, in the same strategy script this branch
gates on an idle sidecar. The merged script keeps both: the archive
name and its restore selection from main, the idle gate, retried polls
and lost-operation check from this branch.

A retried backup is taken over through the same idle gate as a restore,
matched on the whole command line the sidecar logs. With resume
requested that line reads "create_remote --resume <name>", so the gate
matches that form; matching on the name alone would also take another
operation's row ending in the same name for this one.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-09 16:39:43 +03:00
Aleksei Sviridkin 9e8e10c307 fix(backups): re-render the default backup objects when the S3 endpoint changes (#5230)
## What this PR does

Refs #5205.

The S3 endpoint the platform backups use is copied twice, and neither
copy is refreshed after `publishing.host` changes. The COSI sidecar
writes `BucketInfo` once per `BucketAccess`, so the credentials Secret
`bucket-cozy-backups-system-credentials` rendered from it keeps the old
endpoint. The Velero BSL and the default Strategy CRs keep whatever
endpoint the last render of `backupstrategy-controller` saw. The default
objects gate only acted on missing objects, so backups kept going to the
old host and nothing reported it.

The gate now also forces a render when a copy is stale:

- the bucket's `<bucket>-system` release, when the credentials Secret's
endpoint differs from the COSI Secret it is rendered from;
- `backupstrategy-controller`, when the endpoint an object was rendered
with differs from the credentials Secret's. The chart stamps the
rendered endpoint on the BSL and on every default Strategy as
`backups.cozystack.io/endpoint`, because the endpoint itself sits at a
different path in every kind. Objects without the annotation count as
stale until a render stamps it; `MinForceInterval` bounds the forces.

Stale credentials are forced before the BackupClass and the routed
objects are read, so an absent BackupClass or an unreadable Strategy
still fails the check but does not keep the old endpoint in the
credentials Secret. A `BucketInfo` without an endpoint is not compared,
since rendering from it would write an empty endpoint or fail the bucket
release. Repairing missing objects keeps working while the credentials
comparison is stale or fails, and a `systemSecretName` that is not
derived from a COSI Secret (no `-credentials` suffix, or the raw COSI
Secret itself) is not compared at all. RBAC is unchanged: the controller
already holds cluster-wide `get` on Secrets. No new metric: nothing
would consume it.

What this does not do: the COSI Secret still holds the endpoint the
`BucketAccess` was granted with, so after a host change the endpoint
follows only once an operator deletes the `BucketAccess` and forces
`bucket-cozy-backups` to recreate it (this issues new S3 keys). From
there the gate replaces the remaining manual steps of the recovery
described in the issue. Automatic re-grant is left for a separate issue,
and the operator runbook goes to the website, hence `Refs` rather than
`Fixes`.

Known limitation: the credentials comparison runs only when
`backupStorage.systemSecretName` is
`bucket-<bucketName>-system-credentials`, the Secret the forced
`bucket-<bucketName>-system` release renders. If an operator sets
`bucketName` and `systemSecretName` inconsistently, the comparison is
switched off rather than forcing a release that never rewrites the
compared Secret, so a stale endpoint in that Secret is not repaired.
Repairing a missing credentials Secret still forces the release derived
from `bucketName`, as before this PR.

Tested with unit tests for the gate (including the comparison
normalisation shared with the chart helpers), helm-unittest for the
annotation and the wiring, and a bats guard that every rendered Strategy
CR and the BSL carry the annotation. On a live cluster, a server-side
render of the chart produced exactly the annotation the gate derives
from the real credentials Secret, so the steady state does not force
anything.

### Screenshots

Not a UI change.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: an operator runbook for changing `publishing.host` with
platform backups enabled (re-grant the `BucketAccess`, then let the gate
converge; restart ClickHouse pods that read the credentials from
environment variables). Not filed yet, pending approval of the text.
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(backups): the backup controller now re-renders the Velero BSL, the default backup strategies and the platform bucket's credentials Secret when their S3 endpoint is stale, so after the platform bucket's BucketAccess is re-granted for a new publishing.host, backups move to the new endpoint without forcing releases by hand.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Backup strategies and storage locations now record their storage
endpoint, enabling detection and repair of outdated endpoint settings
after bucket access changes.
* Bucket credentials and backup objects are refreshed independently when
their endpoints are missing or out of date.
* **Documentation**
* Clarified when endpoint changes trigger updates and how suspended
releases are handled.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-09 12:02:42 +03:00
Alexey Artamonov c6c107fd08 docs(backups): state when the gate compares endpoints in the values reference
The values reference read as if the credentials endpoint were always
compared and an object without the endpoint annotation never forced a
render. Operators sizing the force counter or choosing a custom
systemSecretName need both conditions spelled out.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-08 20:18:29 +03:00
Alexey Artamonov dc909c778c fix(backups): compare the credentials endpoint only for the forced bucket's Secret
The gate forces bucket-<bucketName>-system, which renders the
credentials Secret of that bucket's system user and nothing else. The
COSI Secret name was derived from systemSecretName instead, so a value
naming another bucket's Secret made a lagging endpoint there force the
platform bucket release every throttle window without ever refreshing
the Secret that was compared.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-08 20:18:29 +03:00
Alexey Artamonov d0f47f7dd5 fix(backups): compare the credentials endpoint only for a Secret derived from COSI
backupStorage.systemSecretName may name a Secret the bucket chart does
not render from a COSI Secret: the raw COSI Secret itself, which the
projector accepts, or any Secret without the -credentials suffix. The
COSI Secret name was then the source name unchanged, so the gate
compared a Secret with itself and either reported it stale on every
tick or failed the check, and stopped repairing missing default objects
as well. Skip the comparison when there is no distinct COSI Secret or
no endpoint key to compare.

The gate compares the annotation with what the chart's endpoint
helper renders, and the helper renders a present but blank endpoint as
a bare https:// instead of falling back, so the gate has to compute
the same value or its expectation and the chart disagree for that
Secret. The gate's docs, metric help and log lines named only missing
objects, so they did not cover a stale endpoint they now report.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-08 20:18:29 +03:00
Alexey Artamonov 86c541368d Merge branch 'main' into fix/clickhouse-backup-late-credentials
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-06 17:56:59 +03:00
Alexey Artamonov 9376ba0078 fix(backups): hand every driver the same forcePathStyle
backupStorage.forcePathStyle reaches drivers that read it differently.
The controller, Velero and the ClickHouse sidecar parse it with
strconv.ParseBool, the etcd and MongoDB strategies write it into a
boolean field that Helm decodes as YAML 1.1, and the Kafka and
RabbitMQ scripts take anything but the string "false" as path style.
So "False" meant path style to Kafka and RabbitMQ only; "yes" meant
path style to etcd, MongoDB, Kafka and RabbitMQ and was refused by
Velero and the ClickHouse sidecar; "T" and "1" could not be applied to
the boolean fields.

The chart now accepts the spellings of strconv.ParseBool and of YAML
1.1 booleans, a superset of the boolean spellings the etcd and MongoDB
strategies accepted, and passes every consumer "true" or "false", so
they agree. Unset stays unset, so each driver keeps its own default.
Any other value fails the render with a message naming it.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-06 17:56:59 +03:00
Alexey Artamonov ae1e1830ce test(backups): cover a ClickHouse restore against an unreachable sidecar
The restore path resolves its archive from list/remote before the start
gate runs, so it carries its own wait for a sidecar that is not up, and
only the backup path had a test for a sidecar that never answers. The
retry comment also miscounted: the limit is attempts, not retries.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-06 11:31:41 +03:00
Alexey Artamonov 993b79bb37 fix(backups): take over a running ClickHouse restore on Job retry
restore_remote drops every table it restores before recreating it, so
running it twice discards whatever was written after the first run
finished. The strategy Job retries a failed Pod up to twice, and with
the start gate waiting for an idle sidecar instead of hitting 423, a
retry whose predecessor died after its POST waited for that restore to
finish and then started a second one. A retry now takes over a
restore_remote of the same archive that is still running.

A finished row is not taken over: the sidecar never trims its action
log, so a success row of the archive may belong to an earlier
RestoreJob, and treating it as this one would report a restore that
never ran. A retry that arrives after the restore finished therefore
still runs it again.

Once the operation is accepted, one failed poll no longer ends the
attempt under errexit; six in a row do. An operation that ends
cancelled now fails the Pod instead of being polled forever.

A retried backup still starts a new create_remote, because its archive
name changes on every attempt; taking it over needs a name fixed for
the whole BackupJob.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-06 11:23:28 +03:00
Alexey Artamonov d0a73302d9 fix(backups): fail a ClickHouse backup whose operation the sidecar lost
The sidecar keeps its action log only in memory, so a sidecar restart
after create_remote was accepted drops the operation's row, and the
strategy kept polling for it forever under a Job with no deadline.
Until now the resumed upload re-added a row ending in the archive name
and ended the loop, but resume is now off so a failed attempt never
reaches the bucket, which leaves nothing to end it. The row is logged
before the POST is acknowledged, so its absence means the operation is
gone and the attempt has to fail.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-05 21:11:59 +03:00
Alexey Artamonov 2e37aaec5f docs(backups): drop the claim that a region change needs a ClickHouse restart
The ClickHouse sidecar on the system bucket no longer reads S3 settings
from environment set at start, so a re-projected region reaches it
without a Pod rollout, and the contracts fixture no longer depends on
optional Secret keys to start its Pod.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-04 19:31:59 +03:00
Alexey Artamonov a25db5aeb6 fix(backups): wait for an idle ClickHouse backup sidecar before starting
The sidecar refuses a new operation with 423 while another one is in
progress, but the strategy only waited for the API to answer, and went
on to start its operation even when it never did. An operation already
running in the sidecar therefore failed the attempt within seconds.

Poll the action log until nothing is in progress, and fail the Pod
instead of proceeding when the sidecar stays busy or unreachable, so
the failure names the cause rather than surfacing as a 423 or a
connection error from the operation itself.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-04 19:26:23 +03:00
Alexey Artamonov cd91138f33 fix(backups): re-render the default backup objects when the S3 endpoint changes
The S3 endpoint the platform backups use is copied twice and neither
copy is refreshed. The COSI sidecar writes BucketInfo once per
BucketAccess, so the credentials Secret rendered from it keeps its
endpoint, and the Velero BSL and the default Strategy CRs keep whatever
endpoint the last render of backupstrategy-controller saw. The default
objects gate only acted on objects that were missing, so after a
publishing.host change every backup kept going to the old host with
nothing reporting it.

The gate now also forces a render when a copy is stale: the bucket's
<bucket>-system release when the credentials Secret's endpoint differs
from the COSI Secret it is rendered from, and backupstrategy-controller
when the endpoint an object was rendered with differs from the
credentials Secret's. That endpoint sits at a different path in every
kind, in some only as a bare host, so the chart stamps the rendered
value on each object as backups.cozystack.io/endpoint and the gate
compares that. Objects rendered before the annotation existed count as
stale until one forced render has gone through, which stamps it.

The COSI Secret itself still holds the endpoint the BucketAccess was
granted with; it follows a host change only once the BucketAccess is
re-granted.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-04 19:13:55 +03:00
Myasnikov Daniil 47b497194e fix(backups): restore the ClickHouse archive a Backup was taken as
The Altinity strategy named each archive inside its Pod with date -u
and recorded nothing on the Backup, so a restore could only guess: it
listed the remote archives under the application's prefix and took the
newest. A RestoreJob of an older Backup therefore restored whichever
backup had run last.

The controller now names the archive before it creates the backup Job,
from the BackupJob's persisted StartedAt, so every reconcile and every
retried Pod of the Job writes the same one. The name is recorded in the
Backup's driverMetadata next to the strategy parameters, and a restore
renders it into the strategy Pod, which restores exactly that archive
and fails, naming it, when remote storage no longer holds it. The name
is recorded only when the Job's Pod was given it, so a Job created by
an earlier controller, or a strategy that names its own archives,
leaves the Backup without one instead of pointing it at an archive
that does not exist.

Because a retried Pod reuses the name, it waits on an attempt the
sidecar is still running instead of being refused with 423, and asks
for resume, so a local or partial remote backup an earlier attempt left
is completed rather than refused as existing.

Backups taken before names were recorded stay restorable: the Pod picks
the newest archive whose name is stamped no later than the Backup's
takenAt, which is set only after the backup Job finished, and logs a
warning that the choice is a best effort.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-04 11:10:42 +00:00
Myasnikov Daniil b83cc087cc docs: move the repo-only guides onto the website
Seven guides in docs/ were written next to the code and left in the
source tree, where no operator looks. The website carries them now, so
a second copy here only lets the two drift again: the Backup Classes
page had already fallen behind its original.

docs/release.md, docs/deprecating-applications.md and the two SeaweedFS
runbooks in docs/operations stay. A migration, a bats suite, a workflow
or AGENTS.md reads each of them by path, which is the test for keeping
a document in this repository.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-21 17:50:46 +05:00
Myasnikov Daniil 2f6c6a27b8 fix(backups): store the Altinity, Job and FoundationDB pod templates schemaless
backupstrategy-controller renders its strategy CRDs from templates, so
Helm stores each of them twice in every release revision, and a
revision is one Secret capped at 1 MiB. The Altinity, Job and
FoundationDB CRDs each inline the full upstream PodTemplateSpec schema,
about 600 KB apiece; once the default Strategy CRs render as well, the
stored release no longer fits and the forced upgrade that creates them
fails with "Secret ... is invalid: data: Too long".

The Kafka, RabbitMQ and Redis strategies already carry their pod
template schemaless for this reason. Give the three older fields the
same markers. The Go types stay typed, the drivers decode the template
as before, and the apiserver still validates the pod when the driver
creates the workload; every description is kept, and the largest
revision drops to about a quarter of the limit. The rationale moves
to the package doc, so the six field docs only tell the reader what
schemaless means for them.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-19 07:55:31 +00:00
Myasnikov Daniil f47bab7280 test(backups): fail CI on any default Strategy CR rendered without its CRD gate
The CRD gate has now been missed twice: MongoDB shipped without it, and
Kafka, RabbitMQ and Redis followed after it existed. helm-unittest only
checks the templates a case lists, so a new strategy-*-default.yaml
that forgets the gate stays green until an upgrade wedges.

Render the chart with the bucket gate open and no CRD served, and fail
on any strategy.backups.cozystack.io object from any template. Also
check that every rendered Strategy CR gates on the CRD this chart ships
for its kind, because a gate naming the wrong CRD passes the first
check and keeps the CR out of every live render.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-14 13:31:48 +00:00
Myasnikov Daniil 7204992452 fix(backups): gate every default Strategy CR on its CRD
The chart ships its strategy CRDs as ordinary templates, and Helm maps
every document of a release through the RESTMapper before it applies
any. A default Strategy CR gated only on the bucket name therefore
fails Build() with "no matches for kind" whenever the bucket name
resolves before the CRD is served, the release applies nothing, the
CRDs are never created, and every retry renders the same manifest.

That happens on the upgrade that first ships a kind: the Kafka,
Rabbitmq and Redis CRs leave a v1.6 cluster's HelmRelease stuck in
UpgradeFailed on the way to v1.7. It also happens on the first render
of an external-S3 install (provisionBucket=false), where the bucket
name comes from values and every older default CR renders next to its
absent CRD.

MongoDB already gates on crdEstablished for this reason. Give every
default Strategy the same gate. On an installed cluster the CRDs are
established and the render is unchanged; otherwise the first revision
creates the CRDs and skips the CRs, and DefaultObjectsGate, which reads
the cozy-default BackupClass routes, forces the follow-up upgrade that
renders them once the kinds are served.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-14 13:31:45 +00:00
Andrey Kolkov 0c2f131192 feat(backups): back up Valkey with the Redis strategy
Valkey runs on the same spotahome RedisFailover operator as Redis and
speaks the same replication protocol and RDB format, so it needs no
strategy kind of its own. The only difference the backup Job sees is the
chart's resource-name prefix: valkey-<app> instead of redis-<app> for the
sentinel Service and the auth Secret.

The Redis driver now accepts applicationRef kind Valkey as well as Redis
and hands the template .Release.Prefix, derived from the app's kind, which
cozy-default-redis uses for every name it builds. A kind with no known
prefix fails the render instead of producing an empty one. cozy-default
routes apps.cozystack.io/Valkey to cozy-default-redis. A restore must
target an app of the same kind as the backup's source: replaying a Redis
dump into Valkey, or the reverse, is not something the strategy can
promise across engine releases.

The shipped template reads the prefix as `.Release.Prefix | default
"redis-"`: during an upgrade the previous controller, which has no Prefix
in its render context and serves only Redis, renders the new CR, and a
bare reference would print "<no value>" into the names.

The redis:8 image already loads RDB v11, which both shipped Valkey
versions (7.2, 8.1) write.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-By: LLM
2026-10-13 12:34:24 +04:00
Andrey Kolkov 647407545d fix(backupstrategy-controller): run backup clients on a package-owned image
The Altinity and RabbitMQ strategy Pods ran platform-migrations, a 349 MB
image carrying kubectl, helm and etcd-migrate whose ENTRYPOINT is the
migration runner, and whose pins were stamped from packages/core/platform
so every migration change moved the backup clients too.

Build images/backup-http-client (alpine + curl + jq + ca-certificates,
non-root) from this package's Makefile and stamp both client keys from
it. The cross-package stamp in packages/core/platform/Makefile and the
platform-migrations-specific check in hack/image-pin-consistency.bats go
away; the general one-digest-per-repository check still covers the two
keys sharing one image.

Fixes #3420

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-10-12 09:54:52 +04:00
Aleksei Sviridkin f4bde9e50d fix(backups): delete the original PVC of an in-place VM restore without keepOriginalPVC (#4672)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

An in-place VMInstance restore with `keepOriginalPVC: false` reports
`Succeeded` without restoring the disks, and leaves every disk it
touched unable to come back. `prepareForRestore` deletes each DataVolume
with Orphan propagation and leaves the PVC in place, on the assumption,
stated in the `deleteDataVolume` comment, that Velero overwrites it.
Velero does not overwrite a PVC. Its CSI restore action returns early
for a claim that already exists, so no DataDownload is ever created, and
with `existingResourcePolicy: update` the merge patch it attempts
instead is rejected because a bound claim's spec is immutable. All that
reaches the claim is the labels-only fallback patch, which also drops
the `cdi.kubevirt.io/allowClaimAdoption` annotation the resource
modifier added.

The claim left behind then blocks the disk for good. Orphan propagation
has removed its owner reference to the DataVolume, and it lacks the
`cdi.kubevirt.io/storage.populatedFor` annotation that
kubevirt-velero-plugin writes on the backed-up copy, so when the
HelmRelease resumes, the CDI webhook refuses the DataVolume it creates
with `Destination PVC <ns>/<name> already exists`, on every reconcile.
Deleting the disk through the API leaves the PVC behind, and a new disk
of the same name fails the same way.

Observed on a v1.6.3 cluster, for a VMInstance restore with
`keepOriginalPVC: false` whose disk was attached to the VM. The Velero
log for the restore, trimmed:

```text
level=info msg="Retrieved per-restore resource modifiers from configmap cozy-velero/restore-modifiers-..."
level=warning msg="PVC already exists and ExistingVolumeDataPolicy is not in-place restore. Skip restore this PVC." Action=PVCCSIRestoreItemAction PVC=tenant-datalab/vm-disk-e2edisk
level=info msg="Applying resource modifier patch on tenant-datalab/vm-disk-e2edisk"
level=warning msg="patch attempt failed for /v1, Kind=PersistentVolumeClaim tenant-datalab/vm-disk-e2edisk: PersistentVolumeClaim \"vm-disk-e2edisk\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound claims"
level=info msg="backup/restore labels successfully updated for PersistentVolumeClaim tenant-datalab/vm-disk-e2edisk"
```

`velero restore describe --details` lists the PVC as `(failed)` and the
Restore is `Completed` with two warnings. Afterwards the claim had its
original creation timestamp and UID, no owner reference, no
`allowClaimAdoption`, no `populatedFor`, and the disk's HelmRelease was
`Ready=False` with the webhook error above. That cluster runs a Velero
build from `main`, whose wording is quoted above; the released v1.17.1
and v1.18.3 return early at the same point with `PVC already exists.
Skip restore this PVC.` (`pkg/restore/actions/csi/pvc_action.go`).

The fix deletes the PVC when `keepOriginalPVC` is false, only once its
DataVolume is confirmed gone so that CDI cannot recreate it, and holds
the Velero Restore until the claim has actually disappeared. A claim
still mounted stays `Terminating` behind `kubernetes.io/pvc-protection`,
and creating the Restore then would hit the same existing-claim path.
Velero then creates the PVC from the backup, the data mover fills it,
and the recreated DataVolume adopts it through `populatedFor`. A failed
deletion now fails the RestoreJob instead of letting it report
`Succeeded` over the old data. Before 2234824fe the DataVolume was
deleted with background propagation, so garbage collection removed the
claim, but nothing waited for it.

Deleting the claim is only safe when Velero can put the disk back, so
before any preparation the reconciler now gets the Velero Backup and
fails the RestoreJob unless it exists and is `Completed`. The Backup
artifact outlives the Velero Backup, which Velero garbage-collects at
its TTL, and without the check a restore of an expired backup would halt
the VM and delete its disks before failing. The check runs for every
Velero restore, whatever `keepOriginalPVC` says. A DataVolume that
cannot be deleted now fails the RestoreJob too, and the preparation
waits for every DataVolume to be gone before it goes further: a
surviving DataVolume makes CDI recreate the claim, which Velero then
skips. Reading the DataVolume back needs `get` on `datavolumes`, which
the backupstrategy-controller ClusterRole now grants next to `delete`.

`TestPrepareForRestore_KeepOriginalPVCFalse_SkipsRename` asserted the
old behaviour, that the original PVC survives, and is replaced by
`TestPrepareForRestore_KeepOriginalPVCFalse_DeletesPVC`.
`TestPrepareForRestore_KeepOriginalPVCFalse_WaitsForTerminatingPVC`
covers the wait. Both fail on `upstream/main` without the controller
change. `TestReconcileVeleroRestore_UnusableVeleroBackupTouchesNothing`
covers an expired Velero Backup and one being deleted,
`TestPrepareForRestore_KeepOriginalPVCFalse_KeepsPVCWhileDataVolumeRemains`
a failed and a pending DataVolume deletion, and
`TestPrepareForRestore_KeepOriginalPVCTrue_FailedRenameKeepsPVC` a
failed rename under `keepOriginalPVC: true`, which must leave the claim
in place. A helm-unittest suite, `tests/restore_rbac_test.yaml`, checks
the new verb. Each test fails with its fix reverted, with one exception:
`TestPrepareForRestore_KeepOriginalPVCTrue_FailedRenameKeepsPVC` still
passes with the Step 5 condition cut to `vmRes != nil`, because a failed
rename holds the restore back before Step 5 is reached. That guard is
pinned by `TestPrepareForRestore_KeepOriginalPVC_CacheLagsBehindWrites`,
which fails with the same cut. `helm unittest
packages/system/backupstrategy-controller` passes. `go test
./internal/backupcontroller/...` passes.

Related: #4601 reports the same Velero mechanism for a standalone
VMDisk, where `prepareForRestore` does nothing at all because
`collectUnderlyingResources` records no disk. #3427 makes a VMDisk
report its own disk; with `keepOriginalPVC: false` that path then runs
into this defect too, and this change covers it once both are in. This
PR does not touch the VMDisk path itself.

### Screenshots

N/A — no UI changes.

### Downstream repositories

Walked the trigger map against the diff, which is the restore
preparation in `internal/backupcontroller/velerostrategy_controller.go`
and its unit tests, plus one verb in the backupstrategy-controller
ClusterRole and its helm-unittest: no package added or renamed, no
`values.yaml` or `values.schema.json` change, no CRD,
ApplicationDefinition, label, annotation, namespace or metric change.
`keepOriginalPVC` lives in the opaque `RestoreJob.spec.options`, and a
search of the organisation finds it only in this repository, so neither
the website nor the Terraform provider documents or models it.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(backups): an in-place VMInstance RestoreJob with keepOriginalPVC=false now deletes each original disk PVC, and waits for it to be gone, before Velero restores it. Previously the PVC was left in place, Velero skipped restoring its data, the RestoreJob still reported Succeeded, and the disk's HelmRelease could never recreate its DataVolume ("Destination PVC already exists"). A Velero RestoreJob now also fails before touching the application when its Velero Backup is gone or not Completed, for example after the Backup expired at its TTL.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* When a restore is configured not to keep the original disk claim, the
claim is deleted before Velero restores it, allowing Velero to recreate
it.
* Restores wait while the original disk claim or its DataVolume is being
deleted. Deletion or status-check errors stop restore preparation.
* Restores configured to keep the original disk claim continue to
preserve it.
* Restores stop before changing resources if the referenced Velero
backup is missing or has not completed.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-10 22:26:10 +03:00
Mattia Eleuteri 6d064a8602 fix(backups): delete a restored disk's PVC only once the restore can run
Deleting the original PVC of a keepOriginalPVC=false restore is only
safe when Velero can put the disk back. The Backup artifact outlives
the Velero Backup it points to, which Velero garbage-collects at TTL,
and nothing checked it before prepareForRestore halted the VM and
deleted the disks: a restore of an expired backup lost the volume where
it used to fail with the disk intact. The Velero Backup must now exist
and be Completed before any preparation starts.

A DataVolume that survives its deletion makes CDI recreate the claim
under the original name, which Velero then skips as an existing PVC,
so the restore would end Succeeded on a freshly imported disk. A failed
DataVolume deletion now fails the restore, and the PVCs are deleted
only once every DataVolume is gone. Reading the DataVolume back needs
get on datavolumes, which the ClusterRole did not grant.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-10-07 11:23:37 +02:00
Mattia Eleuteri 5956b459fb fix(backups): give each Postgres instance its own default WAL prefix
The default CNPG strategy archived a Postgres app under
<namespace>-<name>, inside a destinationPath that is also keyed by
namespace and name, and the driver fell back to the Cluster name when a
strategy set no serverName. Deleting the app does not empty the bucket,
so an app recreated under the same name archived its WALs and base
backups into its predecessor's stream: both start on timeline 1, and a
later restore of the old archive could pick the new cluster's base
backup or WAL segments.

barman-cloud refuses a non-empty archive only while
.check-empty-wal-archive exists in PGDATA. CNPG removes that file as
soon as ContinuousArchiving turns True, and with no plugin attached
archive_command is a successful no-op, so the file is gone minutes
after initdb, before the first BackupJob attaches the plugin.

Suffix the default serverName with the application UID, in the
strategy template and in the driver's fallback. The template guards
the UID with `with`, so an object without one renders the previous name
rather than "<no value>". A Cluster that already has the plugin keeps
its serverName, so only clusters never backed up take the new prefix,
and restores keep reading the source prefix from driverMetadata.

Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Assisted-by: LLM
2026-10-06 16:35:42 +03:00
Aleksei Sviridkin 2af8b0c229 ci(build): fall back to direct, retry and cache Go module downloads (#4660)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

One transient error from proxy.golang.org is enough to fail an image
build or the `Unit & controller tests` job today. The error is an HTTP/2
stream reset (`stream error ... INTERNAL_ERROR`), and nothing retries or
falls back after it. This PR adds three things: a direct fallback, a
bounded retry and a module cache.

Image builds now fall back to fetching modules from their origin when
the proxy fails. They get `GOPROXY=https://proxy.golang.org|direct` as a
build arg from `hack/common-envs.mk`, and every Go builder stage
declares `ARG GOPROXY`. With a pipe Go goes direct on any proxy error.
With Go's default comma it does that only on a 404 or 410. The value
comes from `IMAGE_GOPROXY` and not from the host's `GOPROXY`, because
buildkit can't reach a mirror on localhost or a `file://` proxy. Every
workflow job that runs Go on the runner sets the same value as env.

Explicit module downloads in Dockerfiles are retried three times with a
growing pause. The loop is inline, in one form that the contract test
pins. I tried a shared script from a named build context first, but
buildx refuses any `--build-context` for the four Dockerfiles still
pinned to the `dockerfile:1.2` frontend. The images also build from
different contexts, so no single file can be copied in. Raising those
pins changes how those images build and needs a real build to check, so
I left it for a follow-up. Two alpine stages, token-proxy and
shadowbox's `ss-server`, left the download to `go build`. The alpine
golang images have no git, so the direct fallback can't fetch for them,
and they now get the same download step. The runner jobs use
`hack/go-mod-download.sh` with the same bound.

The unit test job now starts from a module cache instead of downloading
everything. It restores `GOMODCACHE` with `actions/cache`, keyed on
every `go.sum` in the tree, and then prefetches all tracked modules
through the retry, so a cache miss still works. A cache saved by a PR
run is visible only to that PR, but one saved on the default branch can
be restored by any PR
([docs](https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching#restrictions-for-accessing-a-cache)).
That is why the build cache warmer on main writes the same key before
its image build. This way the first run of a PR, which is the one that
flakes, starts warm. The warmer restores only an exact key, so old
module versions don't pile up, and its Go steps can't fail the image
build. Both runner labels are Linux, so the key matches. Both also check
out under `/home/ubuntu/_work`, so `go env GOMODCACHE` should give the
same cache path, but I have not seen it on the warmer's runner yet. The
release jobs in `tags.yaml` and `promote-rc.yaml` get the GOPROXY env
only, no cache.

`hack/go-module-fetch-contract.bats` covers all three. Each test is red
on main, and I checked each guard with a mutation.

Closes #4591

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
ci(build): Go module downloads in image builds and CI jobs now fall back to direct fetching and retry on transient proxy.golang.org errors. The unit test job reuses a module cache written from main.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Reliability**
- Go module downloads now fall back to direct access when the public
proxy encounters errors.
- Builds and automated checks retry failed module downloads, helping
them recover from temporary network issues.
- **Performance**
- CI caches Go modules to reduce repeated downloads and speed up builds
and tests.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:14:47 +03:00
Aleksei Sviridkin bf03c84711 chore(deps): move to the Kubernetes 1.37 libraries without the apimachinery fork (#4473)
## What this PR does

This moves the root module's `k8s.io/*` from v0.35 to v0.37.1 and
controller-runtime from v0.23 to v0.25.1, and drops the
`k8s.io/apimachinery` fork on the way. Supersedes #3001, #4467 and
#4472; those were a stack of the same commits.

The four commits are meant to be read one by one, and each builds and
passes the tests on its own:

1. **Plain watches stop getting the initial-events-end bookmark.** The
apiserver defaults `sendInitialEvents=true` onto any watch with an empty
or `0` resourceVersion and leaves bookmarks off, and our aggregated
storages took that flag alone as a WatchList request. From
`k8s.io/apiserver` v0.36 the watch handler compiled into cozystack-api
panics on that bookmark, so `kubectl get -w` on any cozystack-api
resource would die after the bump. The marker now needs both flags, the
same condition the watch cache uses. There are tests per storage and one
through the real handler chain.
2. **The apimachinery fork goes.** Every apps kind shares the
`Application` Go type, and stock `Scheme.ConvertToVersion` picks the
first kind registered for it. The fork patched that; the apps group now
encodes through a serializer that keeps the kind the object carries. A
handler-chain test covers create, get, list, update, watch, apply and
table for two kinds and fails without it.
3. **MutatingAdmissionPolicy stays out of admission.** From v0.36 the
plugin is on by default and refuses every write until it has synced
`mutatingadmissionpolicies`, which cozystack-api has no RBAC to list and
which clusters before 1.36 do not serve. Without this, every create,
update and delete would fail after the bump. It is also re-applied in
`Complete`, since `--disable-admission-plugins` replaces the list.
4. **The bump itself**, with the regenerated clients, openapi and
strategy CRDs.

Commits 2 and 4 are where the review effort goes; 1 and 3 are small.

Known changes in behaviour:

- Without the fork, the row objects of a Table requested with
`includeObject=Object` carry the first registered app kind. kubectl asks
for that only with `--sort-by`, and it sorts on fields, not on the kind.
Nothing at this layer can fix it, because the handler converts through
`APIGroupInfo.Scheme`, a concrete `*runtime.Scheme`. Admission webhooks
and ValidatingAdmissionPolicy still see the right kind.
- The alpha PodSpec field `workloadRef` became `schedulingGroup`
upstream. Once the new strategy CRDs are applied, the apiserver prunes
`workloadRef` from stored strategies it serves, and their next write
erases it. It only ever worked behind an alpha feature gate.

The branch is rebased onto the Renovate rollup (#4447), with `go mod
tidy` and codegen rerun on top of it.

### Screenshots

No UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
chore(deps): cozystack-api and the controllers build against the Kubernetes 1.37 libraries without the apimachinery fork. Plain watches no longer receive the WatchList initial-events-end bookmark, and objects in Table rows requested with includeObject=Object may carry another app's kind.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Application API responses now preserve the requested resource kind
across operations, including lists and watches.
* Backup strategy resources support additional Kubernetes pod settings,
including probe protocols, volume ownership options, and scheduling
groups.
* Backup strategy schemas include updated Kubernetes pod fields and
descriptions.
* **Bug Fixes**
* Initial-event watch bookmarks are sent only when requested; plain
watches continue without an ending bookmark.
* Mutating Admission Policy is disabled by default, including when
admission plugin options are customized.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-02 21:45:51 +03:00
Aleksei Sviridkin 548d33f01f build(images): fall back to direct and retry Go module downloads
A transient proxy.golang.org HTTP/2 stream reset fails an image build
outright: Go's default GOPROXY falls back to direct only on a 404 or
410. Image builds now get GOPROXY=https://proxy.golang.org|direct as a
build arg from hack/common-envs.mk, declared by every Go builder stage,
so any proxy error falls back to fetching from the origin. The value
comes from IMAGE_GOPROXY rather than the host's GOPROXY, which may
name a mirror that buildkit cannot reach. The alpine golang images
carry no git, so in their stages a module hosted on GitHub cannot be
fetched directly and the retry is the cover.

Explicit `go mod download` steps retry three times with a growing
pause. The loop is inline, in one form the contract test pins: the
images build from different contexts, so no shared script can be
copied in, and a named build context in the shared buildx arguments
makes buildx refuse every Dockerfile pinned to a frontend older than
dockerfile:1.4.

The two alpine stages that left the download to `go build` get the
same download step, because without git neither the fallback nor a
retry covered them.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 19:37:00 +03:00
Aleksei Sviridkin 9b56e70ca6 feat(mongodb): support the system-bucket backup flow (useSystemBucket) (#3967)
## What this PR does

MongoDB was the only `cozy-default` app with **no `useSystemBucket`
flow**: its backup storage was always static on the
`PerconaServerMongoDB` CR (bucket/endpoint/credentials from `backup.*`
values plus a chart-emitted `<release>-s3-creds` Secret), so a tenant
could not back up to the platform system bucket out of the box the way
Postgres/ClickHouse/MariaDB/etcd do. This closes that gap so the default
MongoDB backup-restore works OOTB.

PBM takes `bucket`/`endpointUrl`/`region`/`prefix` as **static CR
fields** (not secret refs), and the platform bucket name/endpoint are
only known at BackupJob time — so, unlike ClickHouse's runtime
`secretKeyRef` env, the chart cannot render them. The implementation
mirrors the CNPG driver's pattern:

- **Chart** (`packages/apps/mongodb`): a new `backup.useSystemBucket`
flag keeps `spec.backup.enabled` (so the pbm agents run) but omits the
static `storages`/`tasks`/`pitr` and the `<release>-s3-creds` Secret.
Helm never owns `spec.backup.storages`, so a Flux re-render can't fight
the driver. A `bootstrap.enabled + useSystemBucket` render guard mirrors
Postgres, and `useSystemBucket: true` with `backup.enabled: false` is
refused both at render time and by an `x-kubernetes-validations` rule on
`backup` in the schema (the same mechanism as the `storageClass`
immutability rule). Note on the schema rule: the aggregated apiserver
does not evaluate CEL on application-backed resources yet (#2657), so
today it is a UI/schema-level marker and the render-time refusal is what
stops a Flux release; it also trips the API review gate (“validation
rule added”), so this PR needs an API-owner approval on top of the code
review.
- **Strategy** (`cozy-default-mongodb`): now carries the S3 coordinates
(`endpointUrl` already scheme'd by the
`backupstrategy-controller.endpoint` helper, `credentialsSecret:
cozy-backups-creds`, `prefix: <namespace>/<application>`).
- **Driver** (`psmdbstrategy_controller.go`): SSA-injects
`spec.backup.storages[s3-storage]` onto the live `PerconaServerMongoDB`
at BackupJob time under a dedicated field manager
(`cozystack-psmdb-backup-driver`) with `ForceOwnership`. Injection is
**gated on the app's `backup.useSystemBucket` flag** (a legacy app
leaves it false and is never touched; a useSystemBucket app is
re-injected once per mint (once per BackupJob, plus once per
storage-race re-mint), before its operator CR is created, so a later
coordinate change catches up on the next job while a streaming dump is
never patched under). The backup CR is minted only once the operator
reports the storage pushed into pbm (`status.observedGeneration` caught
up and `PBMReady` True): pbm writes to what its own config says, not to
the spec, so a CR minted before that push would land at the previous
coordinates while its Backup recorded the new ones; the job waits at
`PerconaServerMongoDBPBMConfigPending` meanwhile, bounded by the backup
deadline. Once a job has its CR, none of the mint-time reads or holds
can fail it: the strategy and app reads the artifact still needs are
bounded by the 24h running ceiling instead, and the cluster is not read
at all **and on the release having rendered that flag**: the app CR is
projected from the HelmRelease's desired values, so the flag reads true
before helm-controller renders the revision, while the tenant's own
storage, scheduled tasks and PITR stream from the previous render still
sit on the cluster. As long as the cluster carries `spec.backup.tasks`
or `spec.backup.pitr`, the driver does not inject and holds the
BackupJob at `Ready=False PerconaServerMongoDBLegacyRender` (failing it
at the deadline), so a wedged legacy revision never has its nightly dump
and oplog redirected into the shared bucket. The backup snapshot falls
back to the strategy coordinates when the operator status echo is empty;
that fallback is gated on the CR's own `percona.com/delete-backup`
finalizer (the flow the CR was minted on), not on the app flag re-read
at completion, so a legacy backup never records the platform bucket and
a flag flipped mid-dump does not change the artifact. The restore path
rebuilds `backupSource` from it (the `cozy-backups-creds` reference is
projected into the restore namespace before dispatch).
- **Retention/cleanup**: on the `useSystemBucket` flow the driver owns
the shared-bucket archive (the chart's psmdb `tasks[].keep` retention is
gated out, and an on-demand backup is not covered by task retention). It
stamps the operator's `percona.com/delete-backup` finalizer on each
backup it mints **against a storage that carries the platform
coordinates** — the strategy's bucket and `credentialsSecret`, read back
from the live cluster after the injection — never on the flag alone: a
strategy without an `s3` block, or a storage that points anywhere else,
holds the job (`MongoDBStrategyHasNoS3` /
`PerconaServerMongoDBStorageForeign`) instead of minting a finalized
dump against a bucket the platform does not own. On Cozystack `Backup`
deletion (by hand today; by platform retention once it lands, since a
`Plan` only creates BackupJobs and never deletes a `Backup`) the driver
deletes the operator `PerconaServerMongoDBBackup` CR so that finalizer
frees the object — the finalizer's presence is the ownership marker
cleanup reads back. That reading is unambiguous because cleanup resolves
the CR by the name recorded in `driverMetadata` (the one the driver
minted), not by scanning: the psmdb operator does stamp
`percona.com/delete-backup` on its own `cron-*` task-minted backups, but
those are never in scope. A legacy backup keeps the pre-existing
hands-off contract, decided from the snapshot before any read of the
operator CR. A terminating namespace or the
`backups.cozystack.io/skip-artifact-cleanup: "true"` annotation strips
the driver's finalizer and releases; a `Get` of the operator CR that
fails with anything other than NotFound/NoMatchError still requeues, and
the operator's own `release-lock` finalizer can hold the CR while the
operator is down, so those two are what can keep a namespace in
`Terminating`. `schedule` and `retentionPolicy` are inert on this flow
(there is no engine-side retention and a `Plan` has no retention field
yet); platform-level retention is tracked in
[#4192](https://github.com/cozystack/cozystack/pull/4192) and teardown
GC of orphaned backups in
[#3966](https://github.com/cozystack/cozystack/issues/3966), so until
those land bound `cozy-backups` with an S3 lifecycle policy.
- **Bounds on the waits**: the injection race with the operator's
cluster cache (an `unable to get storage` error on a freshly injected
storage) is retried by deleting the errored CR and minting a fresh one,
with the retry count kept on the BackupJob
(`backups.cozystack.io/mongodb-storage-race-retries`) because the
deleted CR is reaped before the next poll; after three retries the
operator's error is terminal. A system-bucket dump that has started
streaming is not cancelled on the 30-minute deadline, but fails after 24
hours in `running` (`BackupAbandoned` warning naming the operator CR,
which is left in place as the handle to whatever pbm still writes). The
deadline cancel of a not-yet-started CR pins its delete to the
resourceVersion the live read answered for.
- **RBAC**: `patch` on `perconaservermongodbs` for the injection
(server-side apply), and `create`/`delete`/`patch` on
`perconaservermongodbbackups` — `delete` to prune the shared-bucket
archive on `Backup` deletion, `patch` to strip the driver's finalizer
when releasing a wedged cleanup.

Coverage: Go unit tests over `internal/backupcontroller` — the injection
gate keyed off the app flag (legacy cluster never patched) and off the
applied render (a cluster still carrying tasks or pitr is held,
untouched), the SSA-injection precondition with the whole entry
(per-application prefix included) asserted on the cluster, strategy
templating per application, the finalizer refused against a storage
without platform coordinates (strategy without `s3` over a legacy entry,
foreign coordinates on the merged entry), the retry budget driven with
the operator simulated reaping the deleted CR between polls, the restore
snapshot fallback and credential swap across all four legacy/system flow
combinations, the finalizer-owned cleanup with its escape hatches (skip
annotation before any read, `NoMatchError` treated as gone,
terminating-namespace release using the pre-read CR, a failed strip
surfaced as an Event, a legacy CR never deleted, a legacy snapshot
released without a read, a system-bucket snapshot reaching the prune),
the deadline cancelling a not-yet-running operator backup only on a
positive live read (including no live client at all, and a CR that moved
since the read), and the running ceiling. Plus helm-unittest on both
charts (chart skips the creds Secret and omits
`spec.backup.storages`/`tasks`/`pitr` on `useSystemBucket` and keeps
them on the legacy flow; the strategy renders the S3 coordinates and the
`insecureSkipTLSVerify` knob). A live `useSystemBucket` backup+restore
e2e against the platform bucket is a follow-up (needs the system bucket
wired in CI, same status as the Postgres/ClickHouse useSystemBucket
e2e). One item to confirm in e2e: the psmdb operator accepting
`spec.backup.enabled=true` with an empty `storages` between install and
the first BackupJob.

### Screenshots

Not applicable (no UI changes).

### Downstream repositories

The tenant-facing schema surface is one new field,
`spec.backup.useSystemBucket`, on the MongoDB `values.schema.json` / CRD
(regenerated), plus a validation rule on `spec.backup`.
`terraform-provider-cozystack` hand-models MongoDB and may want to
expose it, but it is a new optional boolean that defaults false (no
drift for existing users), and the chart `README.md` regenerates on
release. Leaving every box unticked per the trigger-map guidance so a
maintainer decides on the provider follow-up.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
feat(mongodb): MongoDB backups now support the platform system bucket via backup.useSystemBucket, so the default cozy-default BackupClass works without per-app S3 credentials — the backup driver injects the system-bucket storage onto the cluster at backup time, once the release has rendered the flag. On this flow nothing prunes the shared bucket automatically yet (a Plan carries no retention field; tracked in #4192 and #3966), so an S3 lifecycle policy on cozy-backups is a prerequisite for enabling it, not a suggestion. useSystemBucket requires backup.enabled=true; the chart refuses the combination with backups disabled at render time, and the schema carries the same rule as a marker until the apiserver enforces CEL (#2657). A system-bucket dump still running after 24 hours fails its BackupJob with the operator backup left in place. A MongoDB restore now adopts the target cluster's S3 credentials only for a storage on the same bucket as the archive (it used to adopt them unconditionally); a legacy restore into a target on a different bucket that relied on one key pair covering both must keep or re-create the source release's -s3-creds Secret by name, and is held at Ready=False RestoreCredentialsMissing until it exists.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
  * Added opt-in system bucket mode for MongoDB backups.
* Automatically injects platform bucket storage without
application-managed S3 credentials.
* Supports endpoint, region, path prefix, addressing, and TLS settings.
* Manages shared-bucket backup artifacts when system bucket backups are
deleted.

* **Bug Fixes**
* Improved restore credential selection to avoid cross-bucket
mismatches.
  * Preserved backup metadata when storage details are unavailable.

* **Documentation**
* Documented configuration, scheduling, retention, PITR limitations, and
restore guidance.

* **Validation**
* Prevented incompatible bootstrap and system bucket restore
configurations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 18:10:56 +03:00
Andrey Kolkov 6555ae16b8 fix(mongodb): start a backup only once the operator has pushed the storage into pbm
The wait after a storage injection watched the cluster's spec, which is
not what pbm writes to. At the pinned operator version the backup
reconciler hands pbm only a storage name and records status.s3 from the
spec entry it read, while the storage pbm uses is whatever the cluster
reconciler last pushed into pbm's own config; that push flips PBMReady
False and back to True and the reconcile stamps observedGeneration when
it ends. A backup started between the apply and that push dumped to the
previous coordinates while its Backup recorded the new ones, so both its
restore and its prune pointed at the wrong bucket. The wait was also no
wait at all: the status write that named it re-queued the job at once,
and the live read always showed the apply. The mint now waits for
observedGeneration to reach the cluster's generation and for PBMReady
to be True, and fails at the backup deadline if the operator never gets
there.

The strategy, app and cluster reads still ran ahead of the operator CR
lookup and were bounded by the backup deadline, so a strategy absent for
one reconcile 40 minutes into a streaming dump failed the job and left
the finalized CR unreachable. The CR is resolved first now; with one in
hand the cluster is not read at all, and the reads the artifact needs
are bounded by the running ceiling, past which the job fails naming the
CR left in place.

The settle check treated a field the entry lacked as agreement, so a
strategy that newly set forcePathStyle minted in the same pass as the
apply that changed addressing. A missing field now counts as its zero
value, which keeps an entry the operator rewrote without an empty region
settled while catching a value the apply adds; the storage type is
compared outright.

The runbook still said the finalizer keeps a Plan from accumulating
objects, the RBAC comment still said the timeout cancel runs on either
flow, and four wait messages read a duration as a delay. All corrected;
the runbook also describes the pbm wait.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-29 11:02:57 +04:00
Aleksei Sviridkin e62f510faf chore(deps): move the Kubernetes libraries to v0.37.1
The root module's k8s.io/* move from v0.35 to v0.37.1 and
controller-runtime from v0.23 to v0.25.1, its release for Kubernetes
1.37. Two things held them back: the apimachinery fork, which had to be
rebased onto every minor, and the watch handler that from
k8s.io/apiserver v0.36 panicked on the initial-events-end bookmark the
aggregated storages sent to plain watches. Both are gone.

The rest of the upgrades are what these releases require: cel-go
v0.29.2, the etcd client v3.7.0, kube-openapi, structured-merge-diff and
the x/ modules. code-generator moves with the libraries.

Regenerating the strategy CRDs that embed core types adds the
Kubernetes 1.37 fields and follows one removal: the alpha PodSpec field
workloadRef became schedulingGroup upstream. Once the new CRD is
applied, the apiserver prunes workloadRef from every stored strategy it
serves, and the object's next write erases it from etcd. It only ever
took effect behind an alpha feature gate.

The gateway README and the tenantgateway CRD tests cite the
apiextensions-apiserver list-type ratchet at the pinned version; it is
byte-identical in v0.37.1, so only the version they name changes.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-26 16:08:25 +03:00
Aleksei Sviridkin d9cbbec5ff fix(build): cross-compile image builders on the build platform
The remaining Go builder stages, plus the kube-ovn, console and
piraeus-server builders, ran on the target platform. A multi-platform
build therefore ran the whole compiler under QEMU: slow, and for Go
unreliable, as the migration-controller Dockerfile already records a
"found bad pointer in Go heap" crash from exactly that. The builders
now run on $BUILDPLATFORM and produce the target architecture from
TARGETARCH.

Two builders only produced the right architecture because they were
emulated. token-proxy ran `go build` with no GOOS/GOARCH, and kube-ovn
calls upstream `make build-go`, which hardcodes GOARCH=amd64 even on an
arm64 build. On a native builder both would have shipped host binaries
inside the other architecture's image, and nothing fails until a node
of that architecture starts the container. token-proxy now passes the
target explicitly; kube-ovn picks build-go or its arm64 twin
build-go-arm from TARGETARCH.

The console builder emits static JS and linstor-server's debian/control
declares every package Architecture: all, so both run natively without
a target.

A bats test requires every stage that compiles to sit on a BUILDPLATFORM
FROM and to read TARGETARCH or TARGETPLATFORM on a RUN or ENV line. A
bare ARG declaration does not count, which is what let token-proxy
through.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-26 15:05:06 +03:00
Aleksei Sviridkin db6c20f841 chore(deps): raise Go modules to their latest compatible releases
Several pinned modules carry published advisories: golang.org/x/crypto,
net, text and mod, grpc, the opentelemetry SDK and exporters, velero,
the etcd client and cel-go. Rather than take only the minimum fixed
versions, every module outside the Kubernetes lockstep set moves to
its newest release whose own requirements keep k8s.io/* and
controller-runtime where they are and fit Go 1.27.1. Where the newest
release asks for more, the module stops at the last one that does not:
cert-manager v1.19.6, source-controller/api v1.8.5, source-watcher/api
v2.1.1, fluxcd meta v1.26.0; helm-controller/api, fluxcd kustomize,
gateway-api and external-snapshotter stay put.

A few modules stop below their newest release on purpose. cel-go and
the etcd modules stay on the lines k8s.io/apiserver v0.35 was built
and tested with (cel-go v0.29.2, the version Kubernetes itself ships,
and etcd v3.6.15), since the apiserver is their only consumer here.
grpc stays on v1.83.2: v1.84.0 predates the fix for GO-2026-6443, a
server panic the CSI driver's gRPC server can reach, and no v1.84
patch release carries it yet. kubevirt.io/api and the CDI API stay on
the versions upstream csi-driver builds against: from CDI v1.60
StorageSpec.Resources changes type and upstream no longer compiles.

velero v1.18 adds RestoreSpec.ResourcePolicy, which the strategy CRD
embeds, so the CRD is regenerated. The backup controller only consumes
the velero v1 types; the new Queued and ReadyToStart backup phases
fall through to its requeue path like any other in-progress phase.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 17:23:18 +03:00
Aleksei Sviridkin ec7387109e build(go): move the toolchain and builder images to Go 1.27.1
Go 1.26.4 carries standard-library advisories in net/http, net/url,
crypto/tls, encoding/asn1 and html/template that govulncheck reports
as reachable from the first-party binaries. The shipped stdlib comes
from the builder image, so the builder tags move together with the
go directive; the digests are the multi-arch indexes that golang:1.27,
1.27-alpine and 1.27-bookworm resolve to at 1.27.1.

api/apps/v1alpha1 keeps go 1.26.4. It is a published types-only
module, no binary here links its stdlib, and raising its directive
would force every consumer onto Go 1.27.

linstor-csi v1.11.2 pins grpc v1.81.0, which no longer builds on
Go 1.27: golang.org/x/net drops http2.TrailerPrefix there. Its image
now raises grpc to v1.83.2 before building, as the kamaji image does
for its advisories; the upstream unit tests pass on it.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 17:23:18 +03:00
renovate[bot] a7dfbe6f8c chore(deps): update container image & action digests
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 16:55:04 +03:00
Andrey Kolkov 0aaf5e1e05 docs(mongodb): condense the psmdb-backups RBAC comment so it does not ship into the ClusterRole
The delete/patch grant justification had grown to an 11-line `#` block, which
(unlike a `{{/* */}}` comment) renders verbatim into every cluster's ClusterRole
and drifts out of date as the code moves. Condense it to a short pointer at
reconcileMongoDB / cleanupMongoDBBackup and the docs, where the detail is
versioned; keep only enough to explain the non-obvious grant.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-22 12:03:58 +04:00
Andrey Kolkov d091c9951c fix(mongodb): requeue after a fresh storage injection so the operator's cache can observe it
The first BackupJob of a useSystemBucket app SSA-injected the storage and minted
the operator PerconaServerMongoDBBackup in one reconcile. The psmdb operator
resolves spec.backup.storages from a cached cluster read when it services that
CR, and a cache that has not yet observed the apply takes the not-found branch
and latches the CR at State=error — which the operator never re-drives (it
returns for a terminal-state CR and watches only the Backup CR and Pods) and the
driver reads as a terminal failure. Only the first job per app is exposed; later
ones re-apply an already-declared storage.

When the pre-apply cluster did not already declare the storage, requeue after
the apply instead of minting in the same pass. The next reconcile finds the
storage in this driver's own cache (a good proxy that the operator's has caught
up too) and only then mints the CR — one 5s poll per app, well inside the
deadline. A fake client has no informer lag, so this is pinned by asserting the
first pass requeues and mints nothing rather than by reproducing the timing.

Also correct the RBAC delete-grant comment: the 30m timeout branch cancels a
not-yet-started operator CR on either flow, not just the useSystemBucket
cleanup, so a Failed BackupJob never leaves a CR the operator would run later.
And pin the wantBucket=="" restore-credential short-circuit with a no-bucket
source case.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-22 12:03:58 +04:00
Andrey Kolkov e653a4fa5d fix(mongodb): correct the finalizer-prune claims, pin the pre-read release CR, and fix the retention field text
The delete-backup finalizer prunes storage only for a `ready` backup at
v1.22.0; a timed-out backup never reaches `ready`, so the deadline-cancel
comment no longer claims the finalizer "takes anything partially written" —
nothing is written until the backup runs, which the cancel prevents. The
psmdbBackupTimedOut godoc no longer says the operator "has not started" the
backup, which is wrong for `requested` (the operator sets it as it dispatches);
it now describes the not-yet-running states it actually gates.

The pre-read CR handed to releaseMongoDBCleanup on the terminating path had no
test that failed when the argument was dropped, because the existing cases
either call the function directly or let the second Get succeed. Added a
cleanupMongoDBBackup case whose operator-CR Get fails after the first: the
release must strip from the passed object, not re-read, or the teardown wedges.

The useSystemBucket field said "retention move[s] to the Plan", but a Plan has
no retention field; corrected to "scheduling moves to a Plan; this flow has no
retention yet" and regenerated the schema, README and mongodb-rd. Softened the
"never pins a namespace in Terminating" wording in the cleanup godoc and RBAC
comment: the driver only bounds its own finalizer, the operator's release-lock
finalizer can still hold the CR while the operator is down.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-22 12:03:58 +04:00
Andrey Kolkov f8554d289d fix(mongodb): stop the system-bucket flow stranding archives, and reject an inert retentionPolicy
Two ways the shared cozy-backups bucket could accumulate objects nothing prunes.

A dump that outran the 30-minute deadline was declared failed while pbm kept
writing. The BackupJob's terminal-phase guard then never runs the branch that
mints the Cozystack Backup, so the archive lands with no Backup object to
represent it and no retention can reach it (the operator CR carries no ownerRef
either). Leave a running dump to finish — a real dataset routinely outruns 30m,
and a broken one still terminates through the operator's error/rejected state.
The deadline keeps bounding the case it was written for: the operator never
starts the backup (state stays "", requested or waiting), where nothing has
been written and failing strands nothing.

retentionPolicy is inert on this flow (the chart's tasks[].keep is gated out and
no Plan-level retention exists yet), so a value a tenant set expecting it to work
would grow the bucket silently. Reject a retentionPolicy changed from the chart
default when useSystemBucket is on, rather than ignore it; the default still
renders so an untouched install is unaffected.

Also on the cleanup path: hand the terminating-namespace release the operator CR
the caller already read and confirmed owned, so the finalizer strip is not gated
on a second Get whose transient failure would silently drop it and leave the CR
finalized in a namespace being torn down. Correct the ownership comments — the
delete-backup finalizer is not one the operator never sets (it finalizes
task-minted backups on the legacy flow); what keeps it unambiguous is that
cleanup only ever resolves the one CR named in driverMetadata. Add the missing
isolation test that a legacy app's own s3-storage is never overwritten by the
injection call site, and move the strategy's storage comment into a Go-template
comment so it no longer ships into every rendered Strategy CR.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-22 12:03:58 +04:00
Andrey Kolkov cc51e040ae fix(mongodb): keep the cleanup escape hatch off legacy operator CRs, and surface a stuck finalizer
The skip-artifact-cleanup annotation reaches releaseMongoDBCleanup before the
ownership check, and the operator-CR delete sat outside the finalizer gate, so
a legacy backup (no delete-backup finalizer) lost its PerconaServerMongoDBBackup
record when a tenant set the annotation to release a stuck Backup or applied it
across a namespace during teardown. That record is what a psmdb restore-by-name
resolves against, and both the RBAC comment and the docs promise it is never
touched. Gate every write in the release on the finalizer being present: an
unowned (legacy) CR or one that cannot be read at all is left intact, and the
Cozystack Backup still releases.

Surface a failed finalizer strip. The release must proceed even against an
unreachable CR, but the swallowed patch error left the operator CR holding
percona.com/delete-backup — pinning its namespace in Terminating, the outcome
the branch exists to prevent — while the Event reported the success case. Emit
a FinalizerNotStripped Warning naming the object so an operator can find it.

Mark the sibling backup fields (schedule, retentionPolicy, destinationPath,
endpointURL, insecureSkipTLSVerify, s3AccessKey, s3SecretKey) as ignored under
useSystemBucket, so a tenant editing one instead of the flag itself is not
misled by a description that describes only the legacy flow; regenerate the
schema, README and mongodb-rd. Pair the insecureSkipTLSVerify rendering
assertion with a case that flips the platform knob on, so the two together
discriminate the rendered value from a hardcoded default.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-22 12:03:58 +04:00
Andrey Kolkov 542f42ef67 fix(mongodb): make the system-bucket cleanup teardown-safe and single-source its ownership
Four issues in the system-bucket cleanup path.

Teardown wedge. The cleanup requeued forever waiting for the operator to
prune, and the escape-hatch annotation released the Cozystack Backup without
touching the delete-backup finalizer the driver stamps on the operator CR.
So a namespace teardown, where the operator's delete-backup step cannot run
(it execs against a pbm agent that is going away), left the operator CR — and
the namespace — stuck in Terminating with no way out but hand-patching
finalizers. Cleanup now detects a terminating namespace (as Redis/Rabbitmq
do) and, on that or the annotation, strips its own finalizer and releases,
leaving the object for a bucket lifecycle policy. Needs patch on
perconaservermongodbbackups for the finalizer strip.

Ownership was read from two places written at different times: the finalizer
(stamped when the operator CR is created) and a driverMetadata flag (written
when the artifact is created, a later reconcile). If the app's useSystemBucket
flag flipped between them they disagreed. Read ownership from the finalizer on
the one CR named in driverMetadata (the one the driver minted): resolving it by
name, never by scan, keeps the operator's own cron-* task-minted backups (which
do carry percona.com/delete-backup at v1.22.0) out of scope, so the finalizer on
that named CR is an unambiguous single marker. Drops the driverMetadata flag.

Injection error was terminal. A transient server-side apply — an apiserver
hiccup, or a conflict while the operator writes the same cluster — failed the
BackupJob outright. Requeue with backoff instead, bounded by the existing
deadline, matching the cluster/app read errors alongside it.

Injection ran before the enabled check, mutating a cluster that can never
back up. Gate it on backup.enabled so a disabled cluster is left untouched.

Also: reframe the restore S3 backfill as the defensive guard it is (the
pinned operator always echoes status.s3, so the live path normally carries
coordinates), and correct the psmdbtypes doc that still claimed the driver
never patches the operator CRs.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-22 12:03:58 +04:00
Andrey Kolkov f27a02f1b6 docs(mongodb): correct the inlined-storage story and surface the TLS knob
The strategy godoc still said the storage config is NOT inlined and the
strategy only names the storage — which controller-gen copies verbatim into
the CRD, so kubectl explain served the new s3 field and its own denial from
one object. Rewrite it (and the StorageName field doc) to describe both
flows, and regenerate the CRD.

Render insecureSkipTLSVerify on the default MongoDB strategy: the field is
plumbed end to end into the driver-injected storage entry and backed by a
platform knob, but the template set only forcePathStyle, pinning it to
false so a self-signed external S3 that Redis backups honour would fail
MongoDB x509 verification. Pin it in the rendering test alongside the other
s3 coordinates.

Fix the useSystemBucket field description: angle brackets rendered as empty
HTML tags in the README table and the dashboard schema, so wrap the path in
backticks (as the postgres field already does) and note that schedule,
retentionPolicy and PITR are ignored on this flow. Regenerate the chart
schema, README and mongodb-rd.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-22 12:03:58 +04:00
Andrey Kolkov 1d05dafb2e fix(mongodb): own and prune the system-bucket archive, and heal the restore read path
Two correctness gaps on the useSystemBucket flow.

Retention. Gating spec.backup.tasks out of the chart also dropped the keep
that psmdb task retention used, and an on-demand PerconaServerMongoDBBackup
is not covered by task retention, so nothing pruned the shared cozy-backups
bucket — a retention-swept Plan grew it unboundedly for every tenant at
once. Own the archive the way the Redis/Rabbitmq drivers own theirs: stamp
the operator's percona.com/delete-backup finalizer onto the backups minted
on this flow, and on Cozystack Backup deletion delete that operator CR and
wait for the finalizer to free the object before releasing. A legacy backup
(tenant's own bucket) keeps the pre-existing hands-off no-op. Adds the
delete verb on perconaservermongodbbackups and a skip-artifact-cleanup
escape hatch for a wedged operator, matching the sibling drivers. Also drops
the unused update verb from the psmdb cluster RBAC (server-side apply is a
patch, not an update).

Restore. resolveMongoDBBackupSource returned early off the live operator
Backup CR whenever it reported a destination, copying status.s3 as-is. On
this flow the operator can echo a destination with status.s3 empty (the
storage was injected, not declared), and the target has no declared storage
to resolve from either, so the restore stranded with no coordinates.
Backfill a missing live S3 block from the snapshot the BackupJob recorded.

Also correct the precondition godoc: the pbm agents start on
spec.backup.enabled alone; the declared storage is required by the backup
CR's resolution, not by the agents.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-22 12:03:58 +04:00
Andrey Kolkov 9545bee233 fix(mongodb): correct system-bucket prose, pin strategy rendering, make restore credential swap flow-aware
Several comments and the useSystemBucket field description had drifted from
what the code does.

The strategy header still said the $bucketName gate was "not because it
needs the value" and that the sibling strategies needed the bucket name
while this one only asked for ordering, but the strategy now renders
bucket: {{ $bucketName }} for the injected coordinates, so it needs the
value; the precondition comment claimed the s3-storage entry exists on
backup.enabled=true, which is only the legacy flow. The rendering-test
comment carried the same "needs no bucket coordinates" claim. The
useSystemBucket field description said injection happens "at first BackupJob
time" (it now runs on every BackupJob) and that backups scope to
<namespace>/<release> (the prefix templates the application name, not the
release), and it ran ~700 chars of implementation detail into the CRD that
backup-classes.md already carries; trimmed and corrected, then regenerated
schema/README/types/cozyrds. Same <release> slip fixed in the driver
comment and the docs.

Nothing asserted the new strategy rendering. Added the MongoDB row to the
endpoint-form suite (s3.endpointUrl, scheme preserved both ways) and field
assertions to the rendering suite — bucket, endpointUrl, region,
credentialsSecret, forcePathStyle, and prefix, which must reach the CR as a
literal go-template string so the driver can template it per-application;
a stray brace would empty it with every doc-count test still green.

The restore credential swap broke across flows. It replaced the source
credentialsSecret with whatever the target declared, comparing no buckets,
so a system-bucket backup restored into a legacy target kept the platform
bucket against the tenant's own -s3-creds, and a legacy backup restored
into a system-bucket target got cozy-backups-creds against the tenant
bucket - both AccessDenied. Skip the swap when the source already names the
platform-projected cozy-backups-creds (it outlives the source app, nothing
to repair), and only adopt a target credential whose storage is on the
same bucket as the archive. psmdbStorageS3 now reads .s3.bucket alongside
.s3.credentialsSecret; TestPsmdbTargetCredentialsSecret covers the match,
the cross-bucket refusal, and the empty-bucket no-op.

Also drop the post-apply re-Get: the typed SSA Patch decodes the server's
merged object back into the passed struct, so the precondition reads the
injected storage from it instead of a follow-up Get that hit the informer
cache and briefly reported Ready=False on the happy path.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-By: LLM
2026-09-22 12:03:58 +04:00
Andrey Kolkov 4ea4888685 fix(mongodb): key system-bucket injection off the app flag, not live storage
The MongoDB backup driver decided what to inject and what to record from
the live cluster's storage set rather than from the app's opt-in flag. That
had three consequences.

The snapshot fallback fired for legacy apps too. Its gate was
`snap.S3 == nil && rendered.S3 != nil`, but the coordinates moved onto the
shared cozy-default strategy, so rendered.S3 is now non-nil for every
MongoDB app. A legacy backup written to the tenant's own bucket got its
snapshot rewritten to the platform bucket, sending restore to the wrong
place. Gate the fallback on the app's backup.useSystemBucket flag instead,
threaded through createMongoDBBackupArtifact, and pin the legacy case.

The injected storage never caught up with the strategy. Injection was
gated on the named storage being absent, so it ran once per cluster and
froze the coordinates at the first backup; a later endpoint/region change
or bucket re-provision was a silent no-op, and Helm could not repair it
because the useSystemBucket chart renders no storages. Key injection off
the app flag and re-apply on every BackupJob (the prefix is deterministic,
so this never splits the archive the way CNPG's serverName would). Add
UseSystemBucket to mongodbapp.MongoDBSpec so a legacy app is protected by
the flag, not by whatever happens to be on the cluster.

The docs said the opposite of what the driver does. backup-classes.md
claimed "the MongoDB driver cannot inject its S3 target per-backup";
correct it and add an opt-in-to-the-system-bucket section that names which
values stop being read and that scheduled backups + PITR must move to a
Plan. Fix the endpoint table row, the tuning reference, the RBAC comment
that said the driver never patches, the strategy header/inline comments,
and the example comment. Also assert notExists on spec.backup.pitr in the
helm test, and lift the duplicated cozy-backups-creds default into a const.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-By: LLM
2026-09-22 12:03:58 +04:00
Andrey Kolkov 730592f323 feat(mongodb): support the system-bucket backup flow (useSystemBucket)
MongoDB was the only cozy-default app with no useSystemBucket flow: its
backup storage was always static on the app CR (bucket/endpoint/creds
from backup.* values + a chart-emitted <release>-s3-creds Secret), so a
tenant could not back up to the platform system bucket out of the box the
way postgres/clickhouse/etc. do.

PBM takes bucket/endpointUrl/region/prefix as static CR fields (not
secret refs) and the platform bucket name/endpoint are only known at
BackupJob time, so - unlike ClickHouse's runtime env secretKeyRef - the
chart cannot render them. The fix mirrors the CNPG driver: the chart adds
a useSystemBucket flag that keeps spec.backup.enabled (so the pbm agents
run) but omits the static storages/tasks/pitr and the creds Secret; the
cozy-default MongoDB strategy carries the coordinates (endpointUrl already
scheme'd by the endpoint helper, credentialsSecret cozy-backups-creds);
and the psmdb driver SSA-injects spec.backup.storages[s3-storage] onto the
live PerconaServerMongoDB at BackupJob time under its own field manager
with ForceOwnership, so a Flux re-render never reverts it. Injection is
gated on the storage being absent, so a legacy app that ships its own
static storage is left untouched. The backup snapshot falls back to the
strategy coordinates when the operator status echo is empty, so the
restore path rebuilds backupSource (credentialsSecret cozy-backups-creds
is projected into the restore namespace before dispatch).

RBAC grows patch/update on perconaservermongodbs for the injection. A
bootstrap.enabled + useSystemBucket render guard mirrors postgres.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-By: LLM
2026-09-22 12:03:58 +04:00
Andrey Kolkov 01967b56d3 docs(backups): correct the Kafka .Mode doc and the embedded-script templating note
The KafkaSpec.Template doc listed .Mode as "backup" or "restore", but the
driver also renders it as "cleanup" when reaping a removed Backup's object;
the comment is copied verbatim into the CRD, so kubectl explain and the
dashboard showed two values while a custom strategy's restore branch would
run on the cleanup pass. List all three, matching the Redis sibling, and
regenerate the CRD.

The strategy template also claimed the embedded script "reads only env
vars, never Helm/controller values, so no tpl is applied". The controller
renders every string leaf of the PodTemplateSpec through internal/template,
args[0] included, so the script is templated on every run. State that
instead, without literal template braces that Helm would itself expand.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-14 17:55:48 +04:00