Commit Graph
788 Commits
Author SHA1 Message Date
Maxim Belyy a4ea942a2c Merge branch 'main' into feat/observability-proxy-app-scope
Assisted-by: LLM
Signed-off-by: Maxim Belyy <maximbel2003@gmail.com>
2026-11-13 16:07:33 +07:00
Andrey Kolkov 906eeb2372 feat(backups): add Kubernetes backup strategy for tenant clusters (#4931)
## What this PR does

Adds a `Kubernetes` backup strategy so a tenant Kubernetes application
can be backed up and restored through the backups framework, and routes
`apps.cozystack.io/Kubernetes` to it in `cozy-default`.

None of the existing strategies fits a tenant cluster:

- Its API state is a key prefix (`/<namespace>_<release>/`, the
TenantControlPlane's datastore schema) in the etcd `DataStore` shared by
every cluster of the tenant. The `Etcd` strategy snapshots the whole
DataStore, so restoring it would roll every tenant cluster back at once.
- Its control-plane CA, front-proxy CA, service-account key and Talos
secrets cannot be regenerated: new ones break every node, kubeconfig and
token issued under the old ones.
- Its PersistentVolumes are DataVolumes in the tenant namespace
(`pvc-<tenant pvc uid>`, labelled `cluster.x-k8s.io/cluster-name`), and
deleting the application deletes them.

**Backup** (`strategy.backups.cozystack.io/Kubernetes`, driver in
`internal/backupcontroller/kubernetesstrategy_*.go`):

1. Checks the control-plane and Talos certificates and sets a
`CertificatesExpiring` condition (plus a Warning event) when a CA
expires before the backup's TTL or a leaf within
`certificateWarningThreshold` (720h). This never fails the run.
2. A Job in the application namespace dumps the cluster's prefix with
`etcdctl` to the platform bucket. It authenticates with the cluster's
own `kubernetes-<name>-datastore-certificate`, the one its
kube-apiserver uses. It runs on a new `kubernetes-backup` image (alpine
+ curl + jq + etcdctl matching the platform's etcd).
3. A Velero backup takes the PKI Secrets (selected by
`kamaji.clastix.io/name` / `app.kubernetes.io/instance`; the driver adds
the instance label to the cert-manager-issued Talos CA Secrets, which
carry neither) and the data DataVolumes/PVCs. Worker system disks are
not backed up.
4. The Backup records the prefix, the endpoints, the data DataVolumes
and the HelmReleases of the cluster and its `KubernetesNodes` pools. The
HelmReleases are not in Velero because the lineage webhook labels worker
disks after the same applications, so a label selector for the releases
would also pick the disks.

**Restore** goes into the application the backup came from (rename /
cross-namespace is rejected). `RestoreJob.spec.options` takes `etcd` and
`volumes` (both default `true`) to restore either half alone:

- **Application gone**:
  1. Restore the PKI Secrets and the volumes.
2. Recreate the cluster's HelmRelease. The Kamaji release this repo
ships gives a new TenantControlPlane a UID-keyed datastore schema, so
the restore waits for that schema.
3. Stop the new control plane, write the dump under the new prefix (keys
are moved from the old prefix), and start it again.
4. Recreate the worker pools last, so workers join the restored cluster
with the restored CA and Talos secrets rather than an empty one.
- **Application live**: suspend the cluster's HelmReleases, then stop
the MachineDeployments and the cluster-autoscaler (for `volumes`,
deleting the backed-up DataVolumes/PVCs once the Machines are gone), and
then the KamajiControlPlane (for `etcd`). Workers go first because CAPI
removes a worker's Node through the tenant API. Then write the selected
halves back and restore every replica count. Each original count is
recorded on the object it belongs to, so the cluster is put back on
success, on failure, after a 30-minute stop deadline, and when the
RestoreJob is deleted midway. A retry keeps the counts from before the
first attempt, and a HelmRelease suspended before the restore stays
suspended.

Before changing anything, a restore checks that the Velero backup is
still there and Completed, and that the dump object exists (a
verify-mode run of the Job). The restore mode is recorded in the
RestoreJob's status, so a RestoreJob cannot claim that a live cluster is
gone.

Before it runs `del --prefix`, the restore Job downloads and validates
the dump and saves the current prefix to `<dump>.pre-restore`. That copy
is never overwritten, so a retry cannot replace it with a partial
prefix, and it is dropped after a successful restore. The Job writes a
marker to its termination message, and the outcome depends on it:

- A failure before `del` resumes the cluster.
- A failure after `del` leaves the control plane stopped
(`KubernetesEtcdWriteFailed`) rather than started on a partial prefix.
- Deleting the RestoreJob while its restore Job runs also leaves it
stopped.

Events and master leases are skipped only as top-level resources. Velero
cannot overwrite an existing claim, so a live-cluster volume restore
deletes the current claims first; this is documented.

A backup whose Velero half fails deletes the etcd dump it already
uploaded, and a transient error creating the Backup is retried rather
than failing the run. Deleting a Backup removes the Velero backup
(DeleteBackupRequest) and the dump object plus its pre-restore copy (a
cleanup-mode Job, given up after five failed attempts with a Warning,
plus the same skip-artifact-cleanup escape hatch as Redis/Rabbitmq).
With `velero.bslEnabled=false` the strategy is not rendered, and the
DefaultObjectsGate does not count it as missing, the same as the Velero
strategies. Like every default strategy, it is also gated on its CRD
being established, so the upgrade that first ships the CRD to an
installed chart cannot wedge the release.

Also included: RBAC for
TenantControlPlanes/DataStores/KamajiControlPlanes/MachineDeployments/Machines/Deployments,
chart values (`kubernetesBackupClientImage`,
`kubernetesBackupActiveDeadlineSeconds`, `kubernetesBackupResources`), a
`make image-kubernetes-backup` target, docs in
`docs/operations/backup-classes.md`, and the
`examples/backups/kubernetes/` walkthrough.

### Verification

- `go test ./internal/backupcontroller/`: the new tests cover:
- the certificate windows, pool HelmRelease discovery, option parsing
and the restore-phase selectors;
  - the full backup path;
  - in-place restores with default options and etcd only;
  - a pre-suspended HelmRelease;
- a failed etcd write leaving the control plane stopped, including on
RestoreJob deletion;
  - RestoreJob deletion resuming a stopped cluster;
  - the stop deadline;
  - the preflight: missing or incomplete Velero backup, missing dump;
  - a rebuild, and an unreadable HelmRelease record;
- rename rejection, a non-etcd datastore, and the cleanup attempt limit.

Each safeguard in the Go driver was checked by mutating it out and
confirming that a test fails. The rendered Job script has no committed
test: a unit test would have to extract the script from the rendered
chart, and its behaviour belongs in the chainsaw e2e follow-up instead.
It was run against stub `etcdctl`/`curl` in every mode and branch:
  - a foreign key or an empty value stops it before any etcd call;
- an S3 answer other than 200/404 on the pre-restore copy stops it
before any etcd call;
- an existing copy is kept, and a missing one is uploaded before `del`;
  - keys are moved from the old prefix to the new one;
  - an object in a namespace named `events` is kept.
- `helm unittest` (29/29), `hack/backupstrategy-crd-gate.bats` and `helm
lint` on `backupstrategy-controller`.
- On a live cluster (Velero 1.18.1), with a tenant cluster holding a
ConfigMap marker and a file on a PVC:
- The same flow done by hand before the driver existed: back up, delete
the application, rebuild. The CA, front-proxy CA, SA key and Talos
secrets were byte-identical afterwards, the pre-backup admin kubeconfig
still worked, namespace and ConfigMap UIDs were unchanged, and the PVC
data came back.
- Through this driver: the BackupJob succeeded (432 keys dumped, Velero
backup Completed, Backup metadata as described above). The tenant
namespace holding the markers was then deleted, which also deleted its
PVC and the backing DataVolume, and a RestoreJob with default options
rolled the live cluster back. It stopped the workers and then the
control plane, wrote the prefix back, restored the volume, and restored
the control plane to 2 replicas, the pool and the autoscaler to 1, and
unsuspended the HelmRelease. A new worker joined. The namespace came
back with its original UID, the ConfigMap and the PVC (bound to the same
volume) with their data. That run caught an ordering bug, now fixed and
covered by a test: scaling the control plane down first left the old
Machine stuck in `Deleting`, because CAPI removes the Node through the
tenant API.
- On the current head, through the driver, against the same tenant
cluster:
- **Backup.** It succeeded. The volume list it recorded was taken before
Velero started.
- **`{"volumes":false}` restore after a ConfigMap was deleted and
another created.** The deleted ConfigMap came back and the newer one was
gone. The worker was left running and the volume data was untouched. The
restore waited for terminating control-plane Pods before writing. The
pre-restore copy was saved before `del` and dropped afterwards.
- **Rebuild after the application was deleted, with default options.**
It ran the full sequence: secrets, volumes, cluster HelmRelease,
datastore schema, stop, etcd write, resume. It succeeded in about 3
minutes. The CA and Talos CA were byte-identical, a new worker joined,
and the namespace and ConfigMap UIDs and the PVC data were back. The
tenant cluster runs on a Kamaji that still names schemas `<ns>_<name>`,
so this run did not exercise a prefix change; that path is covered by
the unit tests and the stub runs.

### Not in this PR

- A chainsaw e2e. The tenant Kubernetes suites are bounded by tight op
budgets that tests pin, so the round trip needs its own suite or a
budget change; it will be a follow-up.
- `rotationPolicy` on the Talos CA Certificate in the kubernetes chart
(cert-manager's default `Always` would rotate the CA key at renewal).
- Restore into another application name or namespace.
- The tenant's shared etcd DataStore runs with auth disabled, so any
cluster's datastore certificate can read its neighbours' prefixes. This
predates this change; the dump Job uses the same certificate the
cluster's kube-apiserver already holds.

### Screenshots

N/A — no UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Walked against the diff: no package under `packages/apps|extra` is added
or renamed, `packages/core/platform` and installer values are untouched,
the new CRD is in `strategy.backups.cozystack.io` (not typed by
terraform-provider), and `RestoreJob.spec.options` stays opaque JSON.

### Release note

```release-note
feat(backups): tenant Kubernetes applications can now be backed up and restored through the cozy-default BackupClass. A backup captures the cluster's etcd state and its control-plane PKI and PersistentVolume data, and warns when a control-plane CA would expire before the backup does. A restore rebuilds a deleted cluster with its original CA or rolls a live one back, and can restore the etcd state and the volumes separately via RestoreJob options `etcd` and `volumes`.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

- **New Features**
- Added Kubernetes application backup and restore support, including
etcd data and persistent volumes.
- The default backup class now routes Kubernetes applications to the new
strategy.
- Added an end-to-end walkthrough for creating a cluster, backing it up,
restoring it in place or after deletion, and cleaning up.
- Backup status can report expiring certificates, and restore validation
helps protect cluster identity and state.
- **Documentation**
- Added guidance on backup contents, restore behavior, and walkthrough
configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 15:40:01 +07:00
scooby87 e978ed7f2f fix(postgres): stop backup jobs from fighting over the WAL archive destination (#5221)
## What this PR does

Part of #5202. A second PR will restore the last good destination when a
job that changed it fails.

The ObjectStore a Postgres Cluster archives its WAL to is shared by
every backup of the application, yet every running BackupJob server-side
applied its own strategy's destination onto it every five seconds with
forced ownership. Two BackupJobs of one application with different
BackupClasses flipped the destination back and forth until the healthy
backup failed too, and a webhook or apiserver blip during that apply
failed the BackupJob at once.

- A BackupJob writes the destination and the Cluster plugin only before
its `cnpg.io/Backup` exists. Afterwards it only reads the live Cluster,
and a failed read is returned for a controller retry instead of failing
the running backup.
- Before writing, a BackupJob waits, with a `Ready=False` /
`WaitingForOtherBackup` condition naming the other job, while another
running BackupJob of the application has a `cnpg.io/Backup` in flight.
The wait is not capped, for the same reason as in #5211. The queue reads
the Cluster's own Backups from the apiserver; with leader election one
reconcile runs at a time, so only a leader handover can still let two
jobs through. An atomic per-application claim is #4559.
- Before writing, a BackupJob also checks the apiserver for a restore of
its application and holds while one runs, including a job released from
the queue: a write landing during the restore would put the restored
Cluster back on the source's WAL prefix.
- Write errors that cannot succeed on retry (invalid, bad request,
forbidden) still fail the job at once. Any other is retried with a
`ClusterPluginApplyRetrying` condition for the backup deadline, counted
from the first failure since the job last waited, and recorded in a new
`DestinationApplied` condition.
- The pass that creates the `cnpg.io/Backup` records on it the
destination, endpoint, credentials Secret and serverName it wrote
(`backups.cozystack.io/cnpg-written-destination`), and the artifact and
its size probe are built from that record, so a strategy edited, or a
Cluster re-rendered, while the backup runs does not redirect the
artifact. A Backup without the record falls back to the strategy as
before.
- The plugin rollout deadline and the instance check count from the
write that attached the plugin, not from the job's start. The apply that
attaches it also stamps the time on the Cluster
(`backups.cozystack.io/barman-plugin-attached-at`), so a lost BackupJob
status write after the attach does not fall back to an older time, and a
job that starts while another job's recent attach is still rolling waits
for that rollout too. A stamp a few minutes ahead of the controller's
clock is waited out; one further ahead is treated as corrupt.

Tests drive the reconciler through a running backup that must not
rewrite the destination, the queue and its release, a restore starting
while a job waits, each class of write error, a recovered write, a
failed Cluster read while the Backup runs, lost status writes around the
attach, another job's rollout, clock drift on the attach stamp, the
recorded destination and serverName, and the rollout deadline. Each
guard was checked by mutation.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(postgres): Postgres BackupJobs of the same application no longer overwrite each other's WAL archive destination: a job writes it only before its backup starts and waits while another backup or a restore of the application is in flight. A transient error while writing it is retried instead of failing the job at once, and the backup artifact names the destination and serverName the backup was actually written to.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Reliability**
* CNPG backups wait while another backup or restore is using the shared
destination, and retry temporary destination-write failures while
observing the backup deadline.
* Backup artifacts and size checks use the destination recorded when the
backup was created, even if destination settings later change.
* Backup processing handles plugin rollout timing and destination-hold
release more consistently.
* **Restore**
* Restore operations account for shared WAL archive destination
conflicts and guard against incompatible backup selections.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 08:58:55 +03:00
Alexey Artamonov 583f18437d test(clickhouse): give the credentials-wait BackupJob a UID
The test pins that a backup held for late bucket credentials keeps
its archive name, and a run's archive name now carries part of the
BackupJob's UID, so the fixture needs one for the name it expects to
be the name the controller mints.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-12 17:21:51 +03:00
Alexey Artamonov dae76d186c Merge branch 'main' into fix/clickhouse-idle-before-backup
main pins the archive name a backup Pod keeps across a wait for
late bucket credentials, against the controller's archive helper.
That helper now names a run's archive with part of the BackupJob's
UID, so the merged test has to give its BackupJob one.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-12 17:20:19 +03:00
Alexey Artamonov 5c6fc10a23 test(postgres): pin three guards no test caught when changed
A job whose own Backup exists skips the wait for another job's plugin
rollout, and the size probe keeps the strategy's Secret key names while
it takes the Secret from the record, adding a credentials block only
when the strategy has none. Each of these could be changed with the
suite still green: dropping the own-Backup check, or forcing the
missing-credentials branch either way. Cases for a job with a running
Backup next to a recent foreign attach, a strategy without credentials
and a strategy with custom key names now fail on each change.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-12 17:02:40 +03:00
Aleksei Sviridkin 238b3ee491 fix(seaweedfs): enforce S3 bucket lifecycle rules (#5228)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

SeaweedFS accepts and stores an S3 bucket lifecycle configuration, but
only the `s3_lifecycle` job of its plugin worker applies it, and that
job is scheduled by the admin server. Cozystack deployed neither, so a
lifecycle rule returned 200 and no object ever expired, on the platform
backup bucket and on tenant buckets alike. That also made the documented
way to bound backup storage a no-op.

This PR adds the admin server and a worker to Simple and MultiZone
SeaweedFS instances, behind a new `lifecycle.enabled` value that is
**off by default**. A Client instance is unchanged, because its rules
are enforced where the instance it connects to runs. The worker
registers for `s3_lifecycle` only, so vacuum, balancing and erasure
coding are not started as a side effect. The admin keeps its job state
on a 1Gi volume on `lifecycle.storageClass`, or on the cluster default
when that is empty; `volume.storageClass` is not reused, because in
MultiZone it can name a class local to one zone or node. The admin UI is
protected by a generated password, because a tenant namespace admits
traffic from the whole cluster. The dashboard Role lists the two new
WorkloadMonitors.

Enabling the worker alone was not enough on SeaweedFS 4.45. Its chart
renders no `[grpc.s3]` section, so the s3 gateway served its internal
gRPC in plaintext while the worker dials it with mTLS, and every
lifecycle delete failed the TLS handshake. The section is backported
from upstream (seaweedfs/seaweedfs#11157, released in 4.46) as a chart
patch, applied with `--forward` so the next `make update` stops at the
point where the patch must be dropped instead of silently reversing it.
The s3 pods are restarted once so they load the section. This also
repairs IAM cache propagation between s3 replicas, which failed in the
same way, and it applies whether or not lifecycle is enabled.

The data-plane release now retries a failed install or upgrade in place
(`RetryOnFailure`) instead of rolling it back. With the previous default
remediation and unlimited retries, an upgrade that waited out its
timeout, for example on a tenant quota with no room for the admin
volume, rolled back and retried forever, restarting the s3 pods twice
per cycle. The worker's hostname anti-affinity is relaxed to a
preference so a rollout finishes on a single-node cluster.

#### Why lifecycle is off by default

The daily walk that SeaweedFS uses to expire objects sends its deletes
without an object identity, and the s3 server skips its compare-and-swap
when no identity is given; it does not re-check the object's age either.
A key overwritten between the walk's listing and the delete loses its
new content. This path is unchanged from 4.45 through 4.52 and on
upstream master (`weed/s3api/s3lifecycle/dailyrun/walker_dispatcher.go`,
`identityMatches` in `weed/s3api/s3api_internal_lifecycle.go`). Backup
buckets write unique keys and are barely exposed; a tenant bucket that
rewrites a fixed key is. On top of that, turning enforcement on would
start applying, on upgrade, every rule that was stored and ignored until
now. Both are retention changes an operator should choose, so nothing
changes for existing instances until `lifecycle.enabled` is set. Making
it the default again is left to a separate PR that carries object
identity on the walk path; upstream report:
https://github.com/seaweedfs/seaweedfs/issues/12451.

#### Turning lifecycle on and off

When `lifecycle.enabled` is set, rules that have been stored but ignored
start applying within minutes, and objects they have already expired are
deleted on that first pass, so review existing bucket lifecycle rules
first. Set `lifecycle.storageClass` before enabling, and prefer a
replicated class: the class is part of the admin StatefulSet's volume
claim template, which cannot change, so changing it while lifecycle is
on makes every upgrade of the instance fail until it is reverted. Moving
the volume means disabling lifecycle, deleting the
`admin-data-seaweedfs-admin-0` PVC, changing the class and enabling
lifecycle again. Each instance gains two small pods and a 1Gi PVC; when
the tenant quota has no room for them, the data-plane release keeps
retrying in place until it has.

When `lifecycle.enabled` is turned off again, the parent release removes
the admin credentials Secret while the data-plane release removes the
admin StatefulSet only on its own reconcile, shortly after. An admin pod
that restarts in that window fails to start on the missing Secret until
the StatefulSet is gone; nothing is lost, and the order is left as is.
The admin PVC stays until the instance is deleted, and a later re-enable
starts the admin with a new password.

Verified on a test cluster with the 4.45 chart: without the backport,
the worker found the expired objects and every delete failed with `tls:
first record does not look like a TLS handshake`; with this branch and
lifecycle enabled, a rule with a past expiration date removed the
matching objects and left the others, both on a manual run and on the
first scheduled pass.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up needed, not opened yet:
`operations/services/backup-classes.md` (v1.7 and next) advises an S3
lifecycle policy, which the bundled SeaweedFS applies only once
`lifecycle.enabled` is set; the SeaweedFS reference page gains
`lifecycle.enabled` and `lifecycle.storageClass` from the regenerated
README on the next release
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The terraform provider does not model the SeaweedFS application, so the
new values do not reach it.

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(seaweedfs): S3 bucket lifecycle rules can now be enforced. Simple and MultiZone SeaweedFS instances run the SeaweedFS admin server and a lifecycle worker when the new `lifecycle.enabled` value is set; it is off by default, so existing instances keep their current behaviour. It stays off by default because the upstream daily expiry walk deletes without checking that the object it listed is still the one stored, so an object overwritten in that window can be deleted although it is new. When enabled, rules that were stored but ignored start applying within minutes and objects they have already expired are deleted on that first pass; review existing rules first. Each enabled instance gains two small pods and a 1Gi volume, placed on `lifecycle.storageClass` or the cluster default. Independently of the switch, the s3 gateway now serves its internal gRPC over mTLS, which repairs IAM cache propagation between s3 replicas and restarts the s3 pods once on upgrade.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Added optional, daily S3 bucket lifecycle-rule processing for
supported SeaweedFS configurations. It is disabled by default; Client
instances rely on lifecycle rules configured on the connected SeaweedFS.
* Added configurable storage for lifecycle job state and dashboard
visibility for lifecycle components.
* Enabled the S3 gRPC server to use its internal TLS identity when
security is configured.

* **Bug Fixes**
* Improved storage-class handling so numeric-looking names remain
strings and unset classes are omitted where appropriate.
* Clarified backup cleanup guidance, including when lifecycle rules
apply and how to reclaim storage.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-12 16:21:52 +03:00
Aleksei Sviridkin 3badbdef74 fix(clickhouse): let a running backup sidecar pick up system-bucket credentials (#5232)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

A ClickHouse release with `backup.useSystemBucket: true` can start
before the platform projects `cozy-backups-creds` into its namespace.
Its `clickhouse-backup` sidecar took the S3 settings from environment
variables, which are fixed when the container starts, so the first
BackupJob or RestoreJob of such a release failed with `no EC2 IMDS role
found` until the Pod was restarted, and a rotated key stayed stale the
same way.

The projector now also writes a `clickhouse-backup.yml` key into
`cozy-backups-creds` (bucket, `https://` endpoint, region when set,
keys, `force_path_style` from `backupStorage.forcePathStyle`). On the
system-bucket flow the chart mounts that key as an optional Secret
volume and points `CLICKHOUSE_BACKUP_CONFIG` at it, instead of setting
`S3_*` variables. clickhouse-backup reloads its config file in each API
handler that reaches S3 (among them the `create_remote`,
`restore_remote` and `list` calls the strategy makes; the `status` and
`actions` reads it polls with do not) and kubelet rewrites a mounted
Secret in place, so credentials projected after the Pod started, or
rotated since, reach the running sidecar. `S3_PATH` stays in the
environment because the prefix belongs to the release.

Kubelet takes about a minute to refresh the volume, so the projector
stamps `backups.cozystack.io/credentials-updated-at` when the projected
content changes, and the Altinity driver holds a new backup or restore
Job for up to two minutes after that stamp, with `Ready=False` and
reason `WaitingForCredentials`; the condition is dropped once the Job
runs. A stamp that cannot be read, or one further ahead than clock skew
explains, is logged and ignored rather than holding the namespace's jobs
for good. The hold applies to every Altinity job in a namespace whose
projected Secret just changed, including releases on their own Secret,
where it is only a delay.

The new e2e Test `clickhouse-3-system-bucket-roundtrip` starts a
ClickHouse on the system bucket with no projected Secret, then runs a
backup and an in-place restore through `cozy-default` and checks that
the sidecar that started without credentials served both jobs. In CI the
system bucket's advertised host `s3.example.org` neither resolves nor
gets an ACME certificate, so for the duration of its own step the Test
serves a SeaweedFS-CA certificate under the S3 Ingress TLS Secret name
and adds a marked `rewrite name exact` line for that host to the host
CoreDNS Corefile, removing both on exit; it proves the served
certificate before any job starts.

The bug was reproduced on a test cluster running the code before this
change: a ClickHouse started before the projection failed its first
BackupJob with `no EC2 IMDS role found`. The new e2e Test has not run
green yet outside CI, because it edits the host CoreDNS, which is shared
on that cluster.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#799 (draft until this PR merges):
`operations/services/backup-classes.md` for the mounted config file, the
credentials wait and the upgrade window, without the Pod restart this
change removes
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(clickhouse): a ClickHouse release on the platform system bucket (`backup.useSystemBucket: true`) now backs up and restores without a Pod restart when its namespace had no projected backup credentials at start, and picks up rotated credentials while running. The sidecar reads its S3 settings from a `clickhouse-backup.yml` file in `cozy-backups-creds` instead of environment variables, so the upgrade rolls every such ClickHouse Pod once. The first Altinity backup or restore in a namespace after its projected credentials change waits up to two minutes (`WaitingForCredentials`) for them to reach the sidecar. Upgrade the platform as a whole: a system-bucket BackupJob started while the new chart runs against the previous controller fails, because that controller does not write the new key.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* ClickHouse system-bucket backups now use a projected configuration
file for S3 settings, supporting HTTPS endpoints and optional region and
path-style settings.
* Backup and restore jobs wait briefly for recently updated projected
credentials before starting.
* Path-style settings accept common true and false values and are
normalized consistently across backup storage integrations.
* **Tests**
* Added end-to-end coverage for system-bucket backup and restore,
verifying restored data without restarting the ClickHouse Pod or backup
sidecar.
* Added checks for TLS certificate validation and safe restoration of
shared network and ingress settings.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-12 16:15:26 +03:00
Alexey Artamonov 1d22013646 fix(postgres): hold a backup until the plugin sees its new destination
The plugin-barman-cloud sidecar reads the ObjectStore through its own
cache, kept for ten seconds (DefaultTTLSeconds in plugin-barman-cloud
v0.13.0) and invalidated only by the sidecar's own writes, and WAL
archiving keeps it warm. The pass that changed the destination created
the cnpg.io/Backup a moment later, so the base backup could be taken
against the previous destination while the artifact names the new one,
and a restore from it finds nothing. On a live cluster with the shipped
plugin, a Backup created two to four seconds after switching the
destination landed in the old bucket in two of three tries; with a
fifteen second wait it landed in the new one every time.

The apply that changes the ObjectStore configuration now stamps the
time on the ObjectStore itself, so the record cannot be lost apart from
the change, and the job creates its Backup only once that change is
older than the cache can be. A change is judged on the configuration a
server-side dry run of the same apply returns, so values the apiserver
defaults or another field manager owns do not read as a change on every
pass.

That wait leaves a job with the destination written and no Backup yet,
which the queue, looking only for Backups in flight, did not see: a
second job with another destination wrote its own, the first took that
as a change and stamped again, and the two restarted each other's wait
forever. Every apply of the ObjectStore therefore also names the job
that made it, which only a job without a Backup of its own and let
through by the queue does, and the queue holds every other job while
that one runs and has no Backup yet. Once it has one, that Backup holds
the destination for as long as CloudNativePG runs it, as before, so a
job stuck after its Backup finished does not hold every later backup of
the application. This also covers a job waiting for the
rollout of the plugin it attached, during which a second job used to
switch the destination as well, including when its own write left the
configuration as a restore had re-rendered it.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-12 15:13:58 +03:00
Alexey Artamonov 5a95bcdd64 fix(clickhouse): keep a run's archive name within the old name's length
The archive name is part of every object key, a key is capped at 1024
bytes, and the legacy s3PathOverride prefix has no length limit. The
full UID made the name 37 characters longer than before, so a prefix
that fit the old name pushed the manifest key past the cap and failed
the backup.

The run's name now drops "clickhouse-" and the "T" and "Z" of the
timestamp and spends the room on the first 12 hex digits of the UID,
so it is exactly as long as the old name for every application.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-12 11:10:43 +03:00
Aleksei Sviridkin 9978771858 feat(observability-proxy): serve tenant namespace logs through an authorizing proxy (#5214)
## What this PR does

Adds `observability-proxy`, a new system component that lets a tenant
user read the logs of their own namespace from VictoriaLogs with the
same Kubernetes credentials they already use for the console, and with
Kubernetes RBAC deciding what they can read. This is the first part of
the design proposals [Tenant application
logs](https://github.com/cozystack/community/pull/95) and [Tenant
resource-consumption
metrics](https://github.com/cozystack/community/pull/94); this PR
implements the namespace logs root only. Metrics and the per-application
roots follow in separate PRs.

Why a proxy: today the log stores are shared per root tenant and have no
notion of a caller, so tenants cannot be given direct read access
without seeing other tenants' logs. The proxy authenticates the caller,
authorizes the request from its path, and rebuilds the upstream query so
that its scope cannot be widened by the client.

Backward compatibility: this adds a new package and a new HelmRelease;
nothing existing is renamed, removed or changed, so no migration is
needed.

What is included:

- `cmd/observability-proxy` and `internal/observabilityproxy`: the
proxy.
- Serves
`/apis/cozystack.io/v1alpha1/namespaces/<ns>/logs/select/logsql/<endpoint>`
for the VictoriaLogs read endpoints the console needs (`query`, `hits`,
`stats_query_range`, `tail`, `field_values`, `stream_field_values`,
`facets`). Anything else under the root answers `404`.
- Authentication by `TokenReview`, authorization by one
`SubjectAccessReview` built from the path (`list` on `cozystack.io/logs`
in the namespace), both cached briefly.
- The upstream request is rebuilt from an allowlist of parameters; the
namespace scope is always forced as `extra_stream_filters`.
Client-supplied `extra_filters`, `extra_stream_filters`, any LogsQL
`options(...)` block, and tenant headers such as `AccountID`/`ProjectID`
are not forwarded or are refused.
- Non-canonical paths (`..`, `//`, encoded separators) are refused with
`400` before any authorization is built, because the console nginx
matches on the normalized path but forwards the raw one.
- Time window, step, bucket count and result limits are clamped;
per-user rate, concurrency and open-tail limits answer `429`. Invalid
time, step and limit parameters answer `400`.
- The store is resolved from the namespace's
`namespace.cozystack.io/monitoring` label.
- Live `tail` streams are passed through unbuffered, and the upstream
tail request disables compression so headers reach the client before the
first line.
- `packages/system/observability-proxy`: chart and image build, wired
into the root `make build`.
- Platform wiring: a new `cozystack.observability-proxy` PackageSource,
enabled by default on the full variants and opt-in on the slim ones, the
same way as `monitoring-agents`.
- RBAC: an aggregated `cozy:observability:view` ClusterRole grants
`list` on `cozystack.io/logs` to tenant `view` and above. No API server
serves this resource; the rule exists only for the proxy's
`SubjectAccessReview`. The proxy's own ClusterRole is limited to the
calls it makes (`tokenreviews`, `subjectaccessreviews`, `list`/`watch`
on namespaces).
- Console nginx: when the proxy is not installed or cannot be reached,
the observability roots answer `404` with a Kubernetes `Status` body,
which the console shows as not available on this cluster; the proxy's
own `502`/`504` for an unreachable store pass through unchanged.
- Console: one regex `location` in the console nginx routes the four
observability roots to the proxy ahead of `/apis`. The upstream is
resolved at request time, so the console starts even when the proxy is
not installed; those paths then answer `404` with a Kubernetes `Status`
body. The console Deployment now receives the cluster domain for that
resolution.
- E2E: a new Chainsaw suite `hack/e2e-chainsaw/observability-proxy/`
with two tenants, covering `401`/`403`/`200`, scope-widening attempts,
path normalization and a long-lived `tail`.

#### How it was tested

- `go test -race` and `go vet` for the proxy packages.
- helm-unittest for `observability-proxy`, `dashboard`, `tenant` and
`core/platform`; bats tests for the console nginx routing.
- The Chainsaw suite was not run as-is: it deploys its own VictoriaLogs
as the root store and refuses to run when a real one exists. Its cases
were replayed by a script against the real store on a test cluster.
- Verified on a test cluster:
- `401` without a token and with an invalid token, `403` for another
tenant's namespace, `200` for the caller's own.
- Client `extra_stream_filters`/`extra_filters` (GET and POST), an
`AccountID` header and another tenant's stream named inside the query do
not widen the scope; `options(...)` answers `400`; a path with `..`
pointing at another tenant answers `400`; an endpoint outside the
allowlist answers `404`.
- Through the full browser chain (OIDC login, gatekeeper, gateway,
console nginx), a single `tail` stayed open for more than two minutes
without reconnecting, and requests from the same session to other
tenants' roots answered `403`.
- Not verified: the `429` limits under real load. `golangci-lint` has
not been run locally.

#### Known limitations

- The proxy is reachable only through the console route; tenant
workloads have no egress rule to it. Every request takes its concurrency
or tail slot before the body is read; the token and access reviews and
the body read run under `--request-timeout`, so a stalled kube-apiserver
answers `503` rather than holding the request; and writes to the client
carry a deadline (renewed per line for tails). `--request-timeout=0`
disables those bounds and should not be used in production.
- The first lines a newly started pod writes can be missing from an
already open `tail`, although they are in the store. The likely cause is
that Fluent Bit picks up a new log file later than the VictoriaLogs
live-tail window; later lines of the same pod arrive. A history query
shows them.
- The per-application roots (`.../<plural>/<name>/log`, `.../metrics`)
and the namespace metrics root answer `404` in this PR.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/issues/787 (docs land
once the implementation PRs merge)
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Downstream: `cozystack/website` is reached. A new platform component
(`packages/core/platform/sources/observability-proxy.yaml`) triggers
`content/en/docs/next/guides/platform-stack/_index.md` and
`operations/configuration/licenses.md`; adding it to the system bundle
(`packages/core/platform/templates/bundles/system.yaml`) may also need a
line in `operations/configuration/variants.md`. A user-facing page for
the logs API and the new tenant `view` permission is not required by the
trigger map but would be useful. Tracked in cozystack/website#787. Other
repositories: no trigger matched. `packages/core/platform/values.yaml`
is not changed (only a test fixture), no package under `packages/apps/`
gains a values field, and no CRD changes. The root `Makefile` `build`
target builds one more image; this does not touch `hack/` or the make
targets `ccp` relies on, so `ccp` is not considered affected, but a
reviewer may want to confirm.

### Release note

```release-note
feat(observability-proxy): add an authorizing proxy that lets tenant users read their namespace logs from VictoriaLogs with their Kubernetes credentials, under `/apis/cozystack.io/v1alpha1/namespaces/<ns>/logs/`. Access requires the tenant `view` role or higher. The component is enabled by default on full variants and opt-in on slim variants.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Added an observability proxy for tenant-scoped log access through the
console, including live log streaming. Requests are authenticated,
access-controlled, and subject to configurable rate and concurrency
limits.
* Added platform packaging and deployment support, with console routing
for observability requests.
* **Tests**
* Added automated coverage for access control, tenant scoping, query
handling, streaming, proxy behavior, and deployment configuration.
* **Documentation**
* Updated end-to-end test guidance with host requirements for running
the observability proxy suite.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-11 16:24:25 +03:00
Alexey Artamonov 8ba0380043 docs(clickhouse): name the clickhouse-backup handlers that reload the config
clickhouse-backup reloads its config file in each handler that reaches
S3, among them the create_remote, restore_remote and list calls the
strategy makes. The status and action-log endpoints the strategy polls
read in-memory state and do not reload it, so the mounted file reaches
the sidecar on the calls that need credentials rather than on every
request.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 14:56:11 +03:00
Andrey Kolkov 3b4db69f8e feat(backups): record the RDB snapshot location and size for Redis backups (#5046)
## What this PR does

Builds on the shared S3 probe from #5223 (MariaDB and CNPG), which is on
`main`. Refs #4316.

Backups made by the Redis strategy carry no `Backup.status.artifact` at
all: no URI and no size. This covers both the Redis and Valkey apps,
which share the driver. The snapshot is a single object per backup,
`<namespace>/<app>/<backup>.rdb` (`redisObjectKey`). The driver already
records that key in `driverMetadata`. The default strategy uploads the
object using the S3 coordinates from the backup credentials Secret,
which the controller projects into the application namespace
(`endpoint`, `bucketName`, `region` and the access keys).

When a backup completes, the driver now:
1. reads that projected Secret (`CredentialsConfig.TargetSecretName`) in
the application namespace. The projector stores its `endpoint` without a
scheme, and the strategy script reaches it over https; an endpoint with
a path prefix is refused by minio-go, which leaves the artifact unset;
2. reads the TLS trust from the pod spec of the completed backup Job, in
the order the script applies it: `ca.crt`, then `tls.crt`, from the
Secret mounted as the `s3-ca` volume
(`backupStorage.endpointCASecretName`). Only when there is no `s3-ca`
volume does it honour `S3_INSECURE_SKIP_VERIFY=true`, and only when
every container of the pod sets it, since the probe cannot tell which
container uploaded. The Secret is read at probe time; the default
strategy mounts `s3-ca` only when an endpoint CA is configured, so when
the volume yields no CA now (the Secret is gone or lacks both keys, or
the volume is not a Secret), the probe leaves the artifact unset rather
than trust the system roots or skip verification where the Job may have
verified. The CA is trusted on its own, as `curl --cacert` does, not on
top of the system roots;
3. runs `StatObject` on the recorded key; a response without
`Content-Length`, which minio-go reports as size -1, counts as
unmeasured;
4. records `status.artifact.uri = s3://<bucket>/<key>` and
`status.artifact.sizeBytes` = the object's size. An empty object is
recorded too, but `sizeBytes` is `omitempty`, so on the Backup it reads
like an unmeasured size; since a valid RDB snapshot always carries a
header, a Warning `EmptySnapshot` Event on the BackupJob flags it as a
broken backup.

The artifact is set **only when the object is found**. A custom strategy
that writes elsewhere would otherwise get a URI pointing at an object
that does not exist. On any failure (Secret missing or incomplete,
endpoint invalid, CA unreadable, object not found, TLS error) the
artifact stays absent and the backup still succeeds. The full error is
logged at info, and once the Backup is created a Normal
`ArtifactNotMeasured` Event on the BackupJob gives a fixed reason, also
when a failed existence check skipped the probe; the Event lives in the
application namespace, so it never carries the raw error, which may name
the platform's endpoint and bucket. The probe shares the 10s timeout of
the MariaDB and CNPG probes. The retry check is shared with those
drivers too: a `Backup` that already exists is returned without probing,
a failed existence check creates the Backup without probing, a `Create`
collision is resolved by reading past the cache, and an existing Backup
that cannot be read back is retried with `Ready=False` instead of
failing the BackupJob.

The size is useful because an RDB snapshot is the compressed dataset. It
shows growth, and a snapshot that is suddenly tiny is a broken backup.

Kafka is deliberately left out. Its backup is a topic-metadata export of
a few kilobytes, so the size would say nothing about the data and might
suggest the data itself was backed up.

Tests:
- the stat uses the projected credentials;
- every path that leaves the artifact unset: object missing, Secret
missing, bucket, endpoint, access key or secret key missing, no target
Secret configured, a size the endpoint did not report;
- trust resolution: `ca.crt` wins over `tls.crt` and skip-verify;
`items` remapping of the volume, and a key the `items` do not project is
never trusted; the Redis CA volume mounted next to `s3-ca` is not taken
as the S3 CA; an `s3-ca` volume that yields no CA (Secret gone, Secret
without the keys, a ConfigMap source) is refused with and without
skip-verify; skip-verify applies only on an exact `true` of that
variable in every container, the last entry of a repeated variable
winning;
- the real minio client against a TLS stub: its certificate is trusted
only through the `s3-ca` volume or skip-verify, a CA that did not sign
it is refused even with skip-verify set on the Job, and the shared
client enforces a configured CA with and without the system roots; the
client sends a signed `HEAD` and parses `Content-Length`;
- the Events: an unmeasured artifact or a failed existence check raises
`ArtifactNotMeasured` without the raw error; an empty snapshot is
recorded and raises a Warning `EmptySnapshot`; a measured artifact, a
rejected `Create` and a `Create` collision raise none;
- a reconcile test pinning that the completed Job's pod spec reaches the
probe; a Backup that already exists is returned without a probe or a
`Create`; the `Create` collision is read back from the API server past
the cache.

The suite fails if any guard of the trust lookup (volume name, Secret
source, variable name, every-container rule, refusal of an unreadable
CA), the CA enforcement in the transport, the negative-size check, the
Events, the reason-only message, their place after `Create`, the
empty-snapshot warning, the existing-Backup check, the wiring, any term
of the Secret check, or the API-server read-back is removed.

### Screenshots

N/A — no UI changes.

### Downstream repositories

I walked the trigger map against the diff. The change is Go-only in
`internal/backupcontroller`, with no API, CRD, chart, value or schema
change. Existing `Backup` fields are now filled for one more driver.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(backups): Redis and Valkey backups now record `status.artifact` with the `s3://` location and the size of the RDB snapshot. If the snapshot cannot be found with the platform backup credentials, the artifact stays unset and a Normal `ArtifactNotMeasured` Event on the BackupJob says why. An empty snapshot raises a Warning `EmptySnapshot` Event.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Redis backups now include artifact location and size when they can be
measured.
* Backups are still created when artifact measurement fails, with status
reporting that the artifact was not measured.
* **Bug Fixes**
* Improved handling of existing backups and temporary read-back failures
to support reliable reconciliation.
* Improved compatibility with S3 endpoints using custom certificate
authorities or configured TLS verification settings.
  * Zero-size snapshots are now reported with a warning.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-11 15:52:54 +04:00
Alexey Artamonov 59a11c1788 refactor(postgres): judge an attach record's clock drift in one place
The job that attached the plugin and a job waiting on another job's
attach each parsed the Cluster's attach record and applied the
clock-drift margin on their own, so the two could drift apart on the
next change. Both now take the record from one parser that also
refuses one too far ahead, and differ only in what they do with a
record they cannot use. The margin is now pinned for the waiting job
from both sides: a record a minute ahead still holds it for the
rollout.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-10 16:00:09 +03:00
Andrey Kolkov 0f44cf9e25 feat(backups): add Kubernetes backup strategy for tenant clusters
A tenant Kubernetes application had no backup route in cozy-default, and
none of the existing strategies fits it. Its API state is a key prefix in
the etcd DataStore the whole tenant shares, so the Etcd strategy's snapshot
would roll every tenant cluster back at once; its control-plane CA, service
account key and Talos secrets cannot be regenerated without breaking every
node and credential issued under them; and its PersistentVolumes are
DataVolumes in the tenant namespace that the application's own deletion
removes.

The new Kubernetes strategy backs up the cluster's etcd prefix with a Job
that authenticates with the cluster's own datastore certificate, and the PKI
Secrets and data DataVolumes with Velero. The HelmReleases of the cluster
and its node pools are recorded on the Backup rather than in Velero,
because a label selector for them would also pick up the worker system
disks. A backup warns when a CA would expire before the backup does.

A restore rebuilds a deleted cluster under its old CA, or rolls a live one
back after stopping its control plane and workers, and lets the etcd
prefix and the volumes be restored separately.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-11-10 15:28:34 +07:00
Aleksei Sviridkin 456473dec1 fix(mariadb): fail backup and restore jobs whose operator Job failed (#5207)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

When the Job behind a mariadb-operator `Backup` or `Restore` fails, the
operator sets `Complete=True` with reason `JobFailed`
(`SetCompleteWithJob` in mariadb-operator 25.10.2). The MariaDB driver
read `Complete=True` alone as success, so a backup whose dump or upload
failed ended `Succeeded` and produced a `Ready` Cozystack `Backup`
pointing at an archive that does not exist, and a failed restore ended
`Succeeded` too. A scheduled `Plan` kept producing such backups.

The driver now treats `Complete=True` as success only with reason
`JobComplete`, and fails the BackupJob or RestoreJob on `JobFailed`
whatever the status, without creating a `Backup`. The failure message
names the operator object and carries the Job's own `Failed` reason and
message, since the operator's condition message is only `Failed` and the
Job's pod, with its logs, is already deleted when the Job fails. A Job
that is missing or cannot be read is named in the message instead. Any
other `Complete` state keeps waiting under the existing deadline, as
before; that includes `Complete=False` with reason `Failed`, which the
operator sets with its own cause (`Error creating Job`, `Backup not
complete`) for a failure outside the Job and then retries, and the
deadline failure now carries that message.

Regression tests drive the backup and restore reconcilers through every
`Complete` status/reason combination, an absent `Complete` condition,
and expired deadlines for an absent condition, an unknown reason and
reason `Failed`. They assert the job phase and message as persisted in
the API rather than on the in-memory object, a requeue while the run is
in progress, and, for backups, that no `Backup` is created on failure.
Separate tests cover the failure message for a Job that cannot be read
and for a Job without a `Failed` condition.

Backups that already exist are not touched. A `Ready` MariaDB `Backup`
created by a failed run before this fix may have no archive of its own.
Restoring from it can fail or select another dump from the shared
prefix.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(mariadb): a MariaDB BackupJob or RestoreJob whose operator Job failed now ends Failed instead of Succeeded, and a failed backup no longer creates a Ready Backup without an archive. Backups created by failed runs before this fix are not removed; restoring from one can fail or select another dump from the shared prefix.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Backup and restore operations now report failed jobs as failures,
including details from the underlying job when available.
* Operations are marked successful only when the job completes
successfully. Other outcomes continue to be monitored until the existing
deadline, after which failures include available condition details.
  * Failed backups no longer create backup artifacts.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-10 11:21:19 +03:00
Andrey Kolkov 1093f390ae feat(backups): warn on the BackupJob when the Redis snapshot is empty
A valid RDB snapshot always carries a header, so a 0-byte object means
the backup is broken. sizeBytes is omitempty, though, so such a Backup
read exactly like one whose size was never measured. The controller
still records the artifact, and now raises a Warning EmptySnapshot
Event on the BackupJob once the Backup is created.

A test now also pins that a retry over an existing Backup returns it
without calling Create.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-10 12:18:07 +07:00
Andrey Kolkov 2013e624ca fix(backups): raise ArtifactNotMeasured only for a Backup this pass created
The Event was recorded before the Backup was created, so a Create that
then failed left the BackupJob Failed next to an Event saying the
backup succeeded, and a Create that collided with a Backup another pass
had made reported on that pass's Backup. The Event is now recorded once
Create succeeds. It also covers the one unmeasured path that had none:
a failed existence check, which skips the probe.

A test now also pins that a CA Secret key the s3-ca volume's items do
not project is never trusted.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-10 11:44:33 +07:00
Andrey Kolkov 6f3d26c491 feat(backups): tell on the BackupJob why a Redis artifact was not recorded
A Backup without status.artifact looked the same whether a custom
strategy wrote elsewhere, the credentials Secret was unusable or the
endpoint CA was gone; only the controller log told them apart. The
probe now returns why it gave up, and the controller records it as a
Normal ArtifactNotMeasured Event on the BackupJob. The Event lives in
the application namespace, so it carries a fixed reason only; the raw
error, which may name the platform's endpoint and bucket, stays in the
log.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-10 11:21:46 +07:00
Andrey Kolkov d61a5054a1 fix(backups): refuse to probe a Redis snapshot behind a CA that is gone
The default strategy mounts the s3-ca volume only when the admin
configured an endpoint CA, and the script then trusts that CA alone. When
the volume yields no CA by the time the probe runs, because the Secret
was deleted or emptied after the Job or the volume has another source,
the probe used to fall back to the system roots, trusting more than the
Job may have. It now leaves the artifact unset instead, with or without
skip-verify.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-10 11:21:46 +07:00
Andrey Kolkov 43bed8f6d9 fix(backups): keep the Redis probe verifying TLS unless every container skips
The probe read S3_INSECURE_SKIP_VERIFY from any container of the backup
Job, so a custom strategy whose sidecar sets it while the uploading
container verifies made the probe skip verification the upload did
not. The probe cannot tell which container uploaded, so it now skips
only when every container sets the variable to true, the last entry
winning as in the kubelet.

An s3-ca volume of a source other than a Secret yields no CA the probe
can read; it is now treated like a CA Secret that is gone, so it never
turns into skip-verify.

Tests pin the guards that decide the probe's trust: the Redis CA volume
mounted next to the S3 one, a decoy variable set to true, a CA that did
not sign the endpoint, and the AlreadyExists read-back past the cache.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-10 11:01:18 +07:00
Andrey Kolkov e03986e8d0 fix(backups): treat an S3 object without a reported size as unmeasured
minio-go returns size -1 when a HEAD answer carries no Content-Length,
which would land in Backup.status.artifact.sizeBytes as -1. The probe
now reports such an object as unmeasured, so the field stays unset.

The shared client's trust settings also gain a test against an endpoint
whose certificate the configured CA did not sign, with the CA alone and
on top of the system roots: nothing pinned that a configured CA is
actually enforced.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-10 11:01:18 +07:00
Andrey Kolkov 631f6cfc0d fix(backups): mirror the Redis script's CA trust exactly
The script passes the endpoint CA as curl --cacert, which replaces the
default trust store, while the probe trusted the system roots on top of
it. Trust only the CA, as the script does.

A mounted CA Secret that is still there but no longer carries ca.crt or
tls.crt let the probe skip TLS verification when the strategy opts into
it, although the Job may have verified against a key since removed.
Treat a volume that yields no CA now like a vanished Secret and refuse
to skip verification in its place.

Pin the probe timeout and the reconcile's retry of an unreadable Backup
for Redis with tests.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-10 10:52:45 +07:00
Andrey Kolkov 50b1a2e5ee test(backups): pin the Redis retry check
Cover the two retry-check paths the Redis driver shares with MariaDB and
CNPG: a failed existence check creates the Backup without probing the
bucket, and an existing Backup that cannot be read back is returned as
errBackupNotReadBack so the reconcile is retried.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-10 10:52:45 +07:00
Andrey Kolkov bfea468b6c fix(backups): never skip TLS verification in place of a vanished CA
The Redis probe reads the CA Secret when it runs, not when the backup
Job did. If the pod mounted a CA Secret that has since been deleted and
the strategy also opts into S3_INSECURE_SKIP_VERIFY, the probe skipped
verification against an endpoint the Job may have verified. Refuse to
probe in that case. Without skip-verify a missing CA Secret still falls
back to the system roots, as the script does.

The endpoint comment claimed a bare host; it can carry a path, which the
shared client now refuses.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-10 10:52:45 +07:00
Andrey Kolkov 2ff547348d fix(backups): probe the Redis snapshot with the trust its Job used
The default Redis strategy script verifies the S3 endpoint against
ca.crt, then tls.crt, from the Secret mounted as the s3-ca volume
(backupStorage.endpointCASecretName), and only without either honours
S3_INSECURE_SKIP_VERIFY=true. The probe always verified against the
system roots, so on a cluster with a private CA, such as a self-signed
SeaweedFS, every Redis and Valkey backup succeeded with no artifact.

Read the same settings from the pod spec of the completed backup Job, in
the order the script applies them, and pass them to the S3 connection.
The shared transport gains an InsecureSkipVerify switch that is set only
from that explicit opt-in.

The projector always stores the endpoint as a bare host, so the branch
that parsed a scheme out of it was unreachable; drop it and its test.
Cover the endpoint and access-key terms of the Secret completeness check.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-10 10:52:45 +07:00
Andrey Kolkov c8cdefc665 feat(backups): record the RDB snapshot location and size for Redis backups
Backups made by the Redis strategy, which also serves Valkey, carried no
Backup.status.artifact at all: neither a URI nor a size. The snapshot is
one object per backup, <namespace>/<app>/<backup>.rdb, whose key the
driver already records, and the default strategy uploads it with the S3
coordinates of the backup credentials Secret the controller projects
into the application namespace.

On completion the driver now stats that object with the same coordinates
and records its s3:// URI and size. The artifact is set only when the
object is found, so a custom strategy that writes elsewhere gets no URI
pointing at a missing object. A Backup that already exists is returned
without probing, as for the MariaDB and CNPG drivers.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-10 10:52:45 +07:00
Alexey Artamonov 5635d9d815 test(postgres): pin that a rollout dated by the attach record runs out
The deadline tests dated every rollout by the job's own condition, so a
change that let a rollout dated by the Cluster's attach record wait
forever would have passed them.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-10 01:27:21 +03:00
Alexey Artamonov a27de054ca fix(postgres): wait for the rollout another backup job's attach started
Only the job that attaches the barman-cloud plugin waited for the
instance rollout it causes. A second job of the same application that
started meanwhile found the plugin already there, had no rollout of its
own to wait for and created its cnpg.io/Backup at once, against
instances CNPG was still recreating, and such a Backup fails for good.
The queue does not hold it either, since the attaching job has no
Backup yet.

The attach record on the Cluster says when the plugin was attached, so
a job without a rollout of its own now waits for instances newer than
it while the attach is recent enough for that rollout to be running.
An older attach has had its rollout judged by the job that made it.
The record belongs to that job, so one this job cannot read, or one
further ahead than controller clocks drift, is skipped rather than
failing it; the latter would otherwise hold every later job of the
application for as long as it lies ahead.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-10 01:27:21 +03:00
Alexey Artamonov d37701a327 fix(postgres): tolerate an attach record slightly ahead of the clock
An attach record in the future failed the backup job at once, because
it would hold every instance as older than the plugin. A record a few
minutes ahead is also what a leadership change between controllers
whose clocks differ leaves behind after an attach that worked, and
failing then throws away a backup that would have run. Within that
margin the job now keeps waiting: the instances predate the record, so
the rollout check holds the backup, and the deadline counted from the
record is not spent by the wait. Only a record further ahead is taken
as corrupt and fails the job.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-10 00:08:41 +03:00
Alexey Artamonov c69787ff49 perf(postgres): list only the Cluster's own backups for the queue
Every pass of a backup job waiting for another one reads the
cnpg.io/Backups from the apiserver, and the read covered every Backup in
the namespace, so a namespace with a long backup history paid for it on
each poll of each waiting job. The queue only looks at Backups this
controller created for the Cluster, which have carried the cluster and
owning-job labels since the driver was introduced, so it now asks the
apiserver for those alone.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-10 00:08:41 +03:00
Alexey Artamonov 9a8025ca14 fix(postgres): record the serverName a backup was archived under
The artifact took its destination from the record written when the
cnpg.io/Backup was created, but its serverName, the archive prefix
under that destination, from the live Cluster when the backup
completed. A restore that re-renders the Cluster in between gives it
another serverName, and the artifact then pairs the original
destination with a prefix the backup was never written to, so a later
restore searches the wrong archive. The record now carries the
serverName too; a record written before it did still falls back to the
live Cluster.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-10 00:08:41 +03:00
Alexey Artamonov b4551eb379 fix(postgres): read restores from the apiserver before writing the destination
A backup job that is about to write the WAL archive destination first
checks for a restore of its application, because a write landing during
the restore puts the restored Cluster back on the source's WAL prefix
and the restore fails with an expected-empty-archive error after it has
purged the target. That check read RestoreJobs from the informer cache,
which can still miss one created a moment ago.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-10 00:08:41 +03:00
Alexey Artamonov 8a631311a7 fix(postgres): fail a backup job whose plugin attach record is in the future
The rollout check takes the later of the latch and the Cluster's attach
record, and the record is never rewritten while the plugin stays
attached. A record that parses but lies in the future made every
instance look older than the plugin and counted the rollout deadline
from that time, so the job waited past the deadline by as far as the
record lay ahead, without end for one far enough out. Such a record
says no more about when the plugin was attached than one that does not
parse, and fails the job the same way.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-10 00:08:41 +03:00
Alexey Artamonov fe03589d91 test(postgres): cover a lost attach record on the retry of a failed attach
A status write that fails with the apiserver unavailable takes the same
path as one lost to a conflict, but the attach it follows can itself be
the retry of an apply that failed first, which leaves its own condition
on the job. The test now runs that sequence as well, so the rollout check
is held to it whichever way the record was lost.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-10 00:08:41 +03:00
Alexey Artamonov 8d92aee796 test(postgres): pin that an older attach record keeps the latch
The rollout check takes the later of the latch and the Cluster's attach
record. A record left by an earlier attach of a plugin since removed is
older than the latch when someone who keeps no record attaches the
plugin again, and taking it would accept an instance recreated before
the latch as one that rolled with the plugin.

The record is absent not only when no backup job attached the plugin
but also when the chart did, or a controller that kept no record, so
the comment on the field says that.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-10 00:08:41 +03:00
Alexey Artamonov 325d2b815c fix(postgres): keep the plugin attach time when its status write is lost
The rollout check took the attach time from the PluginRolledOut
condition on the BackupJob, which the pass that attaches the plugin
writes only after the Cluster apply went through. When that status
write is lost to a conflict or a controller restart, the next pass
finds the plugin attached and falls back to the latch written before
the attach. An instance recreated between the latch and the attach then
counts as one that rolled with the plugin, and the cnpg.io/Backup is
created against an instance that has no plugin sidecar; a latch older
than the deadline also fails a rollout that has only just begun.

The apply that attaches the plugin now records the time on the Cluster
itself, so the record cannot be lost apart from the attach. The rollout
check and its deadline use the later of the two times. A record that
does not parse fails the job rather than guess which instances rolled.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-10 00:08:41 +03:00
Alexey Artamonov e5ccde82e2 fix(postgres): read cnpg Backups from the apiserver before writing the destination
The queue that keeps two BackupJobs of one application from writing the
WAL archive destination at once listed cnpg.io/Backups from the
informer cache. The controller runs one reconcile at a time under
leader election, so a Backup created by one job's pass already exists
when the next pass starts, but the cache can still miss it. The next
job then took the queue as empty, rewrote the destination under the
running backup and left that backup's recorded destination naming a
place its WAL no longer went. A job could likewise miss its own fresh
Backup, write the destination again and create a second Backup.

The queue now lists Backups from the apiserver, and a job that finds
no Backup of its own in the cache checks the apiserver before it
writes or creates one. Only a leader handover, when two processes
reconcile for a moment, can still let two jobs through.

The pass that creates the Backup no longer looks it up a second time
right before the create. Nothing can create it in between, and once
the lookup can reach the apiserver, a failed second lookup would come
after the destination write and fail the job instead of retrying.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-10 00:08:41 +03:00
Alexey Artamonov a6af14ce74 fix(postgres): build a backup's artifact from the destination it was written to
The WAL archive destination is written by the pass that creates the
cnpg.io/Backup and not again while it runs, yet the artifact took the
destination, the endpoint and the credentials Secret from the strategy
rendered when the Backup completed. A strategy or BackupClass edited
while the backup ran gave an artifact that pointed somewhere nothing
had been written, and a restore from it found no data.

The pass that writes the destination now records it on the
cnpg.io/Backup it creates, and the artifact is built from that record.
The record lives on the Backup rather than on the BackupJob because a
tenant can edit its BackupJobs but has no access to postgresql.cnpg.io
objects: a record a tenant could rewrite would point the artifact, and
a restore from it, which takes its S3 keys from the strategy, at an
endpoint of the tenant's choosing.

The artifact size probe measures the same place: the recorded
location and credentials Secret, with the Secret keys and the endpoint
CA, which the record does not carry, from the strategy. Should those no
longer fit after an edit, the probe reports no size instead of
measuring somewhere else.

A record that cannot be read, or that names no destination path,
creates no artifact and fails the job, since the strategy may no longer
match the write. A Backup without a record was created by a controller
that kept none, and its artifact is built from the strategy as before.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-10 00:08:41 +03:00
ZverGuy 6fe3e5f6ff fix(observability-proxy): refuse a partial application lookup
A cluster vmselect answers label values with isPartial when a storage
node does not respond, and the list it returns misses that node's
names. Scoping the query to it would drop those pods' series without a
word, which the lookup exists to prevent, so a partial list now fails
the request as any other failed lookup does.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy 3a5eb4fbf2 test(observability-proxy): cover stalled reviews on the application roots
Application roots take two access reviews before any body is read; a
POST to either must answer 503 within the request timeout when
kube-apiserver stalls on them.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy 19dbee15fc docs(observability-proxy): explain why both application roots need the namespace root
The reason given for the second access review held only for the metrics
root. The log root needs it too, because an ownerless pod keeps the
lineage labels its creator set and can claim any application. The
tenant ClusterRole comment also named only the namespace roots, though
the application roots now depend on the same rule.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy e8e0cede61 fix(observability-proxy): forward the default instant query time
An instant query without time had its application lookup sized at the
proxy's clock, but the time was not forwarded, so vmselect evaluated at
its own later clock. With lookups taking most of the request deadline,
the evaluation lands well past the end of the lookup and leaves out pods
first sampled in between. Forward the instant the lookup was sized for.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy 79ea8561de fix(observability-proxy): end a backquoted string only at an unescaped quote
The string tracking treated a backslash inside backquotes as literal,
but metricsql's scanString lets a backslash escape the closing quote
in all three quote kinds, and a quoted label name keeps that escape
({`\``=""} parses). The proxy ended such a string early, read the rest
of the query as string content, and accepted an escaped
rollup_candlestick after it with the lookup ending too soon.

Inside any quote a backslash now skips the next character, as
scanString does, and the comment no longer claims metricsql refuses
an escaped backquote.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy ac5a3c88c4 fix(observability-proxy): put the latency window in the instant start
The second latency offset was added to the application lookup only,
so the other readers of the instant query's earliest read still
assumed time minus 30s: the store choice could send a query to the
short-term store up to 30 seconds too late near its retention edge,
the lookback cap compared against a start the lookup no longer used,
and the field no longer meant the earliest instant read.

The earliest read of an instant query now starts from time minus two
latency offsets, the earliest instant vmselect can evaluate it at, and
the lookup uses it as is.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy cf45a23bf3 fix(observability-proxy): refuse comments outside strings on the app root
The escape check tracked string literals but not comments: in
metricsql a "#" outside a string comments out the rest of the line, so
a quote inside a comment made the check treat the next line's code as
string content, and rollup\_candlestick there was accepted again. The
bracket and offset scans do not understand comments either.

A "#" outside a string now makes the lookahead unknown as well, which
answers 422 on the application root, rather than teaching every scan
about comments; API clients do not send them.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy a54760fd9c fix(observability-proxy): start the instant app lookup a latency early
vmselect evaluates an instant query whose time is within 30s of its
own clock at that clock minus 30s (queryOffset in prometheus.go), not
at time minus 30s. For a time slightly in the future the evaluation
starts earlier than the proxy assumed: time=now+29s with a 1s step
read from six minutes before, while the lookup started at five
minutes 32 seconds, and pods present only in that gap were missing
from a 200.

The evaluation instant is always within (time-60s, time], so the
lookup of an instant query now starts one more latency offset early.
That needs no assumption about the proxy's clock matching vmselect's.
The store choice on the namespace root is left as it is.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy 3198de893a fix(observability-proxy): refuse escapes outside strings on the app root
metricsql unescapes identifiers (unescapeIdent in parser.go), so
rollup\_candlestick(x + y) is rollup_candlestick to vmselect while the
lexical scan saw another name: the lookup ended ten minutes past the
evaluation time and vmselect read nearly thirteen. Any escaped keyword
the scan looks for, offset included, had the same hole. A backslash
outside a string literal now makes the scan unknown, which answers 422
on the application root; escapes inside label values are unaffected.

With rollup_candlestick refused, two steps per "(" stay enough: a
subquery evaluates at most one step past its end plus the alignment of
that end to the step (evalRollupFuncWithSubquery), and the one step a
candlestick over a non-selector would add on top cannot occur.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy 8e491d15dd fix(observability-proxy): accept only a token end after an offset
metricsql also reads a trailing $__interval as part of an offset's
duration: offset -1h$__interval is -(1h + step). The scan stopped at
the "$" and read one hour, so the lookup missed pods created in the
last step and the query answered 200 without them. Refusing a minus
alone left every other continuation open.

Only what certainly ends the duration token is now accepted after it:
whitespace, a closing bracket or separator, or a binary operator other
than minus; anything else answers 422 on the application root. The
grammar comment no longer claims upper-case offset units are refused,
as offsets are scanned lower-cased like metricsql reads them.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00
ZverGuy 054fac6ab0 docs(observability-proxy): name subquery steps in the app lookup texts
Since the lookup reaches subquery steps past the evaluation time, the
applicationScope comment still described only negative offsets, and a
query refused for a subquery step the scan cannot read got a 422 that
listed every other cause but that one.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:24 +07:00