Commit Graph
7820 Commits
Author SHA1 Message Date
Alexey Artamonov 8ba0380043 docs(clickhouse): name the clickhouse-backup handlers that reload the config
clickhouse-backup reloads its config file in each handler that reaches
S3, among them the create_remote, restore_remote and list calls the
strategy makes. The status and action-log endpoints the strategy polls
read in-memory state and do not reload it, so the mounted file reaches
the sidecar on the calls that need credentials rather than on every
request.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 14:56:11 +03:00
Alexey Artamonov c26d2773a5 test(clickhouse): name what really leaves the round-trip's Secret behind
The projected Secret the preconditions delete is left by whichever
BackupJob ran in tenant-root before this Test, from this suite or
another: Chainsaw does not fix the order of the Tests in a suite, so no
particular one can be named. The restore budget is also not spent on a
credentials hold: by then the stamp is older than the backup's own
two-minute wait.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 14:42:11 +03:00
Alexey Artamonov 72a94ee426 test(clickhouse): give the round-trip's preconditions room for their deletes
The step's ceiling SIGKILLs the shell and reports only "signal: killed",
while a kubectl --timeout that fires first names the object that would
not go. The deletes it runs now wait up to 270s in all, past the 3m
ceiling, so a stuck delete late in the list would be reported by the
ceiling instead of by kubectl.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-09 14:26:02 +03:00
Alexey Artamonov 0dc5a54731 test(clickhouse): remove the round-trip's TLS objects a killed run left
Chainsaw deletes an applied object only when its own apply created
it; one left behind by a run killed past its cleanup is patched and
kept. The Certificate would then outlive every later run and keep
re-issuing its Secret after each step cleanup, and a probe Pod whose
sleep had ended never turns Ready, failing the next run's assert.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-09 13:49:42 +03:00
Alexey Artamonov 4345a346cc test(clickhouse): check why the round-trip restore failed, not only that it did
The refused-delete and failed-lookup cases asserted a non-zero status
alone, which any earlier failure in the restore also produces, so a stub
drifting from the script would have kept them green for the wrong
reason. They now also require the delete to have been attempted and the
lookup failure to be named.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-08 20:37:38 +03:00
Alexey Artamonov d20b26d2ed test(clickhouse): pin that a refused Secret delete fails the round-trip restore
The owned Ingress Secret's delete only became reachable once its lookup
stopped asking for a name and a selector together, and nothing failed
the stub's delete, so dropping the delete's status from the restore's
result passed every test. The CoreDNS half of the restore already has
this guarantee pinned.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-08 20:34:28 +03:00
Alexey Artamonov 28f3613450 test(clickhouse): delete the round-trip's TLS Secret in its own step's cleanup
Chainsaw applied the Certificate, so the scripted delete of it in the
run step's finally was hand-rolled teardown of a resource Chainsaw
already reclaims. Chainsaw deletes a step's applied resources, waiting
for them to go, before it runs that step's cleanup operations, which
puts the Secret's delete after the Certificate's without any ordering
written into a script.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-08 20:28:53 +03:00
Alexey Artamonov 6e23bd5629 test(clickhouse): delete the round-trip's Certificate before its Secret
cert-manager issues the Secret of a Certificate that still exists again
as soon as it is deleted, and the Certificate is only removed by
Chainsaw's own cleanup after the step's finally has run. Deleting the
Secret first therefore left a freshly issued copy behind in tenant-root,
carrying no owner reference to take it down with the Certificate.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-08 20:22:43 +03:00
Alexey Artamonov 1003cdb1b6 test(clickhouse): find the round-trip's Ingress Secret by its owner label alone
kubectl refuses a resource name together with a label selector, and the
restore asked for both with its stderr discarded, so the lookup always
came back empty and the Secret the Test serves under the platform's S3
Ingress name was never deleted. The next run then stops on its own
precondition that the Secret must not exist. A lookup that does fail
now fails the restore, since an empty answer from a failed read is what
kept this hidden.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-08 20:22:35 +03:00
Alexey Artamonov 6280d06d01 test(clickhouse): describe the round-trip's guards by the archive-recording restore
Restores now take the archive the Backup recorded rather than the
newest one, and a leftover Backup of the BackupJob's name that records
another archive fails the new BackupJob instead of being adopted. The
comments justifying the sentinel row and the Backup cleanup, in the
round-trip and in its unit test, still gave the earlier reasons, which
would send the next reader after behaviour the driver no longer has.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-06 17:56:59 +03:00
Alexey Artamonov c76080cc07 test(clickhouse): pin the archive name across a credentials wait
A backup's archive is named from the BackupJob's persisted StartedAt,
and the restore of a Backup that records no archive picks one stamped
no later than its takenAt. The credentials wait holds the Job after
StartedAt is set and writes the job's status while it holds. Should a
held reconcile restamp StartedAt, the Pod would write an archive the
earlier reconciles never named; nothing caught that.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-06 17:56:59 +03:00
Alexey Artamonov 86c541368d Merge branch 'main' into fix/clickhouse-backup-late-credentials
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-06 17:56:59 +03:00
Alexey Artamonov 477540b8f1 fix(backups): keep projecting credentials when forcePathStyle cannot be parsed
The credentials projection carries every backup driver's keys, but
only the ClickHouse sidecar's file needs forcePathStyle parsed as a
boolean. A value that failed the parse failed the whole projection as
a malformed source Secret, a terminal reason pointing at the wrong
object, so CNPG, MariaDB, etcd and the other drivers lost their
backups over a parse that serves one file. The value is now left out
of that file and the reason is logged at error level with the
namespace; the rest of the projection proceeds. The chart refuses such
a value at render time, so this covers an environment set around it.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-06 17:56:59 +03:00
Alexey Artamonov 9376ba0078 fix(backups): hand every driver the same forcePathStyle
backupStorage.forcePathStyle reaches drivers that read it differently.
The controller, Velero and the ClickHouse sidecar parse it with
strconv.ParseBool, the etcd and MongoDB strategies write it into a
boolean field that Helm decodes as YAML 1.1, and the Kafka and
RabbitMQ scripts take anything but the string "false" as path style.
So "False" meant path style to Kafka and RabbitMQ only; "yes" meant
path style to etcd, MongoDB, Kafka and RabbitMQ and was refused by
Velero and the ClickHouse sidecar; "T" and "1" could not be applied to
the boolean fields.

The chart now accepts the spellings of strconv.ParseBool and of YAML
1.1 booleans, a superset of the boolean spellings the etcd and MongoDB
strategies accepted, and passes every consumer "true" or "false", so
they agree. Unset stays unset, so each driver keeps its own default.
Any other value fails the render with a message naming it.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-06 17:56:59 +03:00
myasnikovdaniil 970191dc8a fix(clickhouse): restore the archive a Backup was taken as (#5208)
## What this PR does

A ClickHouse `RestoreJob` restored the newest archive under the app
prefix, whichever `Backup` it named. Strategy pod generated the archive
name with `date -u` and nothing was recorded on the `Backup`, so restore
could only list the remote archives and take the last one. Restoring an
older `Backup` brought back the latest data.

Now the controller names the archive before it creates the backup Job
(`clickhouse-<app>-<startedAt>` from persisted `status.startedAt`, so
retried pods write the same one), passes it to the pod as `.ArchiveName`
and records it in the `Backup` `driverMetadata`. Restore takes exactly
that archive and fails with its name when storage has no such archive.
The name is recorded only when the Job's pod got it, so a Job left from
the previous controller during an upgrade, or a custom strategy with its
own names, leaves no name instead of a wrong one.

A retried pod reuses the name, so it waits for an attempt the sidecar
still runs (no 423) and asks for `resume`.

A `Backup` outlives its `BackupJob` and a Plan names `BackupJob`s after
the schedule slot, so a `BackupJob` created again under a used name met
the earlier run's `Backup` when its Job completed, took it as its result
and left the archive it just wrote recorded nowhere. Now it fails if
that `Backup` records another archive, and the message names both. A
retry of the same run still gets its own `Backup` back.

Old `Backup`s have no recorded name and stay restorable: the pod logs a
`WARNING` and takes the newest archive stamped no later than `takenAt`,
and fails when nothing qualifies. While the chart rolls out, previous
controller pod can still render the new template and pass no `takenAt`,
so a restore of a `Backup` without recorded name fails before it touches
any table, and works when created again after the rollout.

Fixes https://github.com/cozystack/cozystack/issues/5199

Tests are go unit tests for naming, recording, a reused `BackupJob` name
and restore render, and `hack/altinity-backup-strategy.bats`, which runs
the shipped script against a stubbed clickhouse-backup API (the restore
tests go red on the old template). Checked on a live ClickHouse with two
backups of 1500 and 2000 rows: restore of the older one by recorded name
ends with 1500 rows, fallback for an unnamed older backup picks the same
archive with the warning, a missing archive fails without touching the
table, a retry over an existing local backup completes under the same
name. The controller binary itself was not run on the cluster, only its
rendered output in the pod.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(clickhouse): a RestoreJob now restores the archive its Backup was taken as, not the newest one. New Backups record the archive name, Backups taken before this release are restored best effort from the newest archive stamped no later than their takenAt, with a warning in the restore pod log. A ClickHouse BackupJob created again under a name whose Backup still exists now fails instead of reporting that Backup as its result.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Backups now use a consistent archive name tied to the backup job, and
completed backups retain that name for future restores.
* Restores can select the recorded archive and backup timestamp. If no
archive was recorded, the system selects the newest eligible archive for
the application.
* **Bug Fixes**
* A restore now fails when its recorded archive is missing instead of
silently selecting a different one.
* Backup requests avoid starting a duplicate action when the matching
action is already in progress.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-06 11:54:38 +05:00
myasnikovdaniil 18779dddfc fix(backups): report sizeBytes for MariaDB and CNPG backup artifacts (#5223)
## What this PR does

Fixes https://github.com/cozystack/cozystack/issues/4316.

Continues https://github.com/cozystack/cozystack/pull/5015 by androndo
on a separate branch. All ten of their commits are kept as they were
(authorship, messages and order, only rebased on current `main`).
MariaDB and CNPG backups stored in S3 report
`Backup.status.artifact.sizeBytes`, the size of the dump
mariadb-operator uploaded or the total of the barman base backup, and
the field stays unset when the size cannot be attributed to that one
backup.

Four commits are added on top:

- `test(backups)`: operator Backup fixtures carry the condition
mariadb-operator writes on success, `Complete=True` with reason
`JobComplete` and message `Success`. Operator sets `Complete=True` for a
failed Job too (reason `JobFailed`), so a fixture without reason
describes a state that never happens.
https://github.com/cozystack/cozystack/pull/5207 makes the driver check
that reason, the tests pass with and without it.
- `refactor(backups)`: the MariaDB matcher returns the matched dump
(key, file name, timestamp from the name, size, LastModified) and tells
"not found" from "ambiguous", and the Secret lookup and S3 client live
on a small value any reconciler can build from its own client. A MariaDB
restore needs this to restore the dump of the Backup it was given and
not the newest one under the prefix. `sizeBytes` does not change here.
- `fix(backups)`: the probe reads bucket, prefix, endpoint, credentials
and TLS from the operator Backup's own `spec.storage.s3`, not from the
strategy as it renders now, so editing the strategy during a backup does
not send the probe somewhere else.
- `fix(backups)`: a dump is attributed only when its ownership is
proven. Before, a backup pod with a clock more than a minute behind,
right after a previous run, got that run's dump size (666 bytes for a
123-byte dump in a test). Now the dump has to be the only one that
either clock places in the operator Backup's lifetime, both clocks have
to agree, and neither may place it in the lifetime of another finished
operator Backup with the same bucket and prefix. Otherwise the size
stays unset. Two cases are still not covered. First, this run's own dump
is missing from the listing while the operator Backup of a neighbour was
deleted or has not finished, then the neighbour's dump can be taken.
Second, backup pod clock and object store clock are both off the same
way by more than the minute of slack, which moves a finished neighbour's
dump into this run's window, and nothing the clocks record tells them
apart. A run whose Job failed has no dump of its own and reaches the
probe only because the driver takes `Complete=True` for success,
https://github.com/cozystack/cozystack/pull/5207 fixes that.

Of the open threads on the original PR, the cache reads, the dump file
name layout and the default AWS endpoint test were already handled by
its later commits, the clock skew one is the last commit here and the
failed Job case is https://github.com/cozystack/cozystack/pull/5207.

`minio-go` is new to this tree, this PR adds it as a direct dependency
and `internal/backupcontroller/artifact_size.go` is its only importer.
The open Bucket strategy PR
https://github.com/cozystack/cozystack/pull/4235 picks the same client
at the same version.

Each clock skew case, and a run whose own dump is gone while the
neighbour's operator Backup still exists and has finished, have a unit
test that fails on the old matcher, and every new guard has a mutant
that turns a test red. I also ran the matcher against a real bucket
after four MariaDB backups in a row. Runs that finished more than a
minute apart mapped to the dump their operator Job logged, the two back
to back came back unset instead of taking each other's dump, and with a
run's own dump hidden from the listing the matcher refused the
neighbour's dump that the old one returned.

### Screenshots

N/A, no UI changes.

### Downstream repositories

The only API change is the description of the existing
`Backup.status.artifact.sizeBytes` field, no new field, value, default
or enum, no package, values or tooling change, and neither
cozystack/website nor cozystack/terraform-provider-cozystack references
`sizeBytes`.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(backups): MariaDB and CNPG (Postgres) backups stored in S3 now report `status.artifact.sizeBytes`: the size of the dump mariadb-operator uploaded, or the total size of the barman base backup (WAL archive excluded). A MariaDB backup takes the size of the one dump that falls inside its run on both the backup pod's and the object store's clock and inside no other finished run, and the size is left unset rather than set to 0 whenever it cannot be measured or attributed. Velero-based backups intentionally keep it unset.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Backup artifacts now report the stored size attributable to an
individual MariaDB or CNPG backup when it can be determined. Shared or
ambiguous storage may leave the size unset.
* **Bug Fixes**
* Improved handling of backup artifact creation and lookup errors,
including retrying when an existing backup cannot be read and failing
jobs when artifact creation is rejected.
* **Documentation**
* Clarified that an unset size means unknown; zero does not indicate an
unknown size.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-06 11:54:28 +05:00
Alexey Artamonov 33608326f1 test(clickhouse): pin that an unreadable hold state keeps the job held
The credentials hold returns the error when it cannot read the strategy
Job or the projected Secret, and both the backup and the restore path
pass it on, so the reconcile retries instead of starting the Job.
Nothing tested that: swallowing either error, in the hold or at either
call site, left the suite green while letting the Job start before the
credentials reached the sidecar, which is the failure the hold exists
to prevent.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-05 21:22:48 +03:00
Alexey Artamonov 603114cd7c test(clickhouse): check the round-trip restored this run's sentinel row
Every run inserted the same sentinel row and accepted any row after the
restore, while the Altinity restore takes the newest remote archive of
the release. A restore of an older run's archive therefore passed. The
row now carries a per-run value and exactly that row has to come back.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-05 21:09:15 +03:00
Alexey Artamonov dee5e6d8bf test(clickhouse): mark the round-trip's reachability poll as a fixed wait
The probe loop sleeps between attempts because no Kubernetes condition
reports that the host CoreDNS serves the rewrite and the Ingress serves
the Test certificate. The e2e conventions allow such a sleep only under
the marker that records it is waiting for something to model it.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-05 21:08:59 +03:00
Alexey Artamonov cabe9a07e2 test(clickhouse): reclaim the round-trip Backup by its name
The Altinity driver names the Backup after the BackupJob and adopts one
that already exists under that name. The round-trip's cleanup deleted
the Backup only through the BackupJob's status.backupRef, so a Backup
whose BackupJob never recorded it survived, and the next run's
BackupJob would have adopted it in place of a fresh backup. Both the
cleanup and the preconditions now delete it by name; the preconditions
ceiling grows to cover the extra wait.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-05 21:08:50 +03:00
Alexey Artamonov a662198991 test(clickhouse): restore the round-trip's shared objects after a step ceiling
The round-trip changes the host CoreDNS Corefile and the S3 Ingress TLS
Secret and puts both back from its EXIT trap. When the run step's
ceiling fires, Chainsaw kills the shell and the trap never runs, so the
rewrite stays in a cluster-wide object and the next run refuses to
start. The step's finally runs regardless, so it repeats the restore;
after a run whose trap already restored them it finds nothing to undo.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-05 21:07:51 +03:00
Alexey Artamonov 003bc3ed0b test(clickhouse): fail the round-trip when CoreDNS cannot be restored
The round-trip puts the host CoreDNS Corefile back from its EXIT trap,
which runs with errexit off. A refused kubectl replace, for instance a
resourceVersion conflict with another writer, fell through to the
success message and returned 0, so the run stayed green while the
rewrite remained in a cluster-wide object and the next run refused to
start. The restore now returns the failure.

The restore moves into a file of its own so a unit test can run it
against a stubbed kubectl and so other callers can share it.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-05 21:06:53 +03:00
Alexey Artamonov 0ad642e68d test(clickhouse): size the system-bucket round-trip server for SQL
The round-trip ClickHouse ran on the nano preset, whose 128Mi limit
the clickhouse-1 contract suite never exceeds because it sends no
query. This suite writes a sentinel row before the backup, and the
server was OOM-killed on that first statement (lastState OOMKilled,
exit 137), so the Test failed before it reached the behaviour under
test. clickhouse-2 runs the same statements on the small preset.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-05 21:05:31 +03:00
myasnikovdaniil e5bb3cda07 fix(kubernetes): move the OIDC users out of the bootstrap Job's script (#5213)
## What this PR does

Found while checking for other places with the failure from
https://github.com/cozystack/cozystack/issues/4968: data that grows with
configuration is passed to a process as one argument, and Linux allows
128 KiB for that. This one is in the `kubernetes` app and there is no
issue about it yet.

The OIDC bootstrap Job wrote a ClusterRoleBinding for every `oidc.users`
entry into its `sh -c` script, 526 bytes per user. I rendered current
main with `helm template` and users like `user123@example.org`: 238
users give a 130,889 bytes script that starts, 239 give 131,415 bytes
and `exec` fails with E2BIG. Then the container can't start, the
post-install and post-upgrade hook fails after its retries and the
HelmRelease of the tenant cluster stays unready. Schema has no limit for
`oidc.users` and each person needs own entry, so a cluster shared by 240
people gets there.

Now the bindings and the list of their names are rendered by the new
`oidc-rbac-bindings.yaml` into a ConfigMap `<release>-oidc-bindings`
that the Job mounts, so the script is 6,380 bytes for any number of
users. The ConfigMap has its own 1 MiB limit, about 2,500 users at 417
bytes each. Names, labels and roles of the bindings stay the same. I
compared the render of old and new chart for eight emails (ones YAML
reads as a boolean or null, a number, a quote, an apostrophe, unicode
and two plain) and the bindings are identical, so existing ones are
updated in place and nothing is orphaned.

The prune step reads the same ConfigMap. If the list is not mounted the
Job stops before it deletes anything, because `grep` on a missing file
exits 2, `! grep` is true and every binding of the release would be
pruned. A release with no users applies nothing (`kubectl apply` fails
on an empty file) and prunes all stale bindings, same as before.

`hack/kubernetes-oidc-bindings.bats` renders 300 users, checks the
script is under the limit without any email in it and that every user
has one binding with the old name and role. Three more tests run the
rendered script against a fake `kubectl` and check what it applies and
deletes. The prune condition forced to true or to false passes all
helm-unittest cases, so this part is tested in bats. The bug is in
release-1.6 and release-1.7 as well.

### Downstream repositories

I walked the trigger map against the diff (one chart template, its tests
and a bats file): no package added or removed, no values, schema or
default changed, no output Secret or Service renamed.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(kubernetes): the OIDC bootstrap Job no longer fails to start for a tenant cluster with about 240 or more oidc.users, the bindings now come from a ConfigMap the Job mounts.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Kubernetes OIDC configuration now supports per-user role bindings,
with administrator and viewer access assigned according to each user’s
role.
* OIDC role bindings are kept in sync with the configured user list,
including removal of obsolete bindings.
* Deployments with no OIDC users or with OIDC mode disabled continue to
avoid creating user bindings.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-05 19:37:16 +05:00
myasnikovdaniil ab13f9f72c fix(cilium): pass the CiliumNode list to jq on stdin in the IPAM hook (#5212)
## What this PR does

Fixes https://github.com/cozystack/cozystack/issues/4968. The cilium
IPAM migration hook passed the whole `kubectl get ciliumnodes -o json`
to `jq` as one argument, and Linux allows 128 KiB for a single argument.
Cilium copies Node labels onto every CiliumNode (22 to 30 KiB per node
with Talos, NFD or GPU labels), so the list is over the limit within six
nodes. `jq` can't start (`Argument list too long`), the hook Job fails
before it moves any node, `cilium` HelmRelease loops on `UpgradeFailed`
and everything that depends on networking stays in `DependencyNotReady`.

The hook is rendered on every upgrade of cilium in `cluster-pool` mode,
so a cluster that already finished the migration hits the same on its
next values change. The failing step only logs the pods that still hold
an old Ingress IP, but it runs before the node loop, so on a cluster
that is not migrated yet it blocks the migration itself.

Now the list goes to `jq` on stdin together with the IP objects, same as
the other `jq` calls in the script get their input.

The new test in `hack/cilium-ipam-migration.bats` makes six CiliumNodes
over the limit (one 24 KiB annotation each instead of labels, and it
checks the list is over 128 KiB even printed compact). It fails on the
old script with status 126 and passes now. It also checks that each pod
is named with the node whose Ingress IP it holds and that a pod on an
unrelated address is not named, because a join that always names the
first node, or reports every IP object, passed the old tests.

I did not take two suggestions from the issue. Skipping the hook when
every CiliumNode is in the pool changes what one existing test pins (the
hook adds the pool to `excludeIps` in that case) and doesn't help
clusters that are not migrated yet. A value to disable the hook leaves
the default broken. `cozystack-operator` rewriting `disableHooks` and
helm-controller rolling the DaemonSet on every retry are other
components. The bug is in release-1.6 and release-1.7 as well.

### Downstream repositories

I walked the trigger map against the diff (one hook script and its bats
test): no package, values, schema, default or output name changed.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(cilium): the IPAM migration hook no longer fails with "Argument list too long" on clusters with more than a few nodes, which blocked the cilium upgrade and every package that depends on networking.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved IP address migration handling for environments with large
node lists, avoiding failures caused by operating system argument-size
limits.
* Migration continues to identify affected pods by node and distinguish
unrelated pods.
* **Tests**
* Added coverage for migrations involving large node lists, multiple IP
objects, and unrelated pods.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-05 19:37:05 +05:00
Myasnikov Daniil d7ae34dfc3 fix(backups): fail a ClickHouse BackupJob whose Backup name is taken
A Backup outlives the BackupJob that made it, so a BackupJob created
again under a name already used meets the earlier run's Backup when its
own Job completes. That happens by hand, and from a Plan reconciled
again inside its starting deadline, since a Plan names each BackupJob
after its schedule slot. The Altinity driver took the earlier Backup as
its result: the BackupJob turned Succeeded with a BackupRef recording
the earlier run's archive, the archive this run wrote was recorded on
nothing, and restoring what the BackupJob reported brought back the
older data.

An existing Backup is now taken as this run's only when it records this
run's archive, which is what an earlier reconcile of the same run
leaves when its status write is lost. Any other fails the BackupJob,
naming both archives, and leaves the earlier Backup untouched.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-05 07:56:44 +00:00
Myasnikov Daniil d5559bb934 test(backups): remove the Altinity strategy tests' scratch directories
Every test in hack/altinity-backup-strategy.bats made a directory with
mktemp -d and never removed it, so each passing run of the file left
nine directories in TMPDIR, and they pile up on any machine that keeps
it between runs.

Each test now removes its directory as its last statement. A failed
assertion aborts the body before that line, so a failing test keeps
its stub log and rendered script for inspection.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-05 07:56:39 +00:00
Alexey Artamonov 31b905a1ce docs(clickhouse): note that the backup path stays in the sidecar environment
The chart comment said every S3 setting of the system-bucket flow comes
from the mounted config file, but S3_PATH is still set in the
environment: the platform's file carries no path, and the prefix belongs
to the release rather than to the platform.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-04 19:51:38 +03:00
Aleksei Sviridkin 3d317c5f67 feat(api): expose HelmRelease failure state on Application status (#5171)
## What this PR does

Tenants usually read Applications but not the HelmReleases behind them,
and the Application status only carried the `Ready` and `Released`
conditions. A client could see that a release was failing, but not how
many times it had failed or whether helm-controller had stopped retrying
it.

This PR copies the `Stalled` condition onto the Application and adds
three optional status fields taken from the HelmRelease:
`installFailures`, `upgradeFailures` and `lastAttemptedReleaseAction`.
All of them are additive and `omitempty`, so existing clients are
unaffected.

What the new fields do and do not tell a client:

- With the `RetryOnFailure` policy the platform generates today, a
failed release is retried rather than stalled, so the counters are the
signal. That does not hold for every state: a release missing from Helm
storage while an install failure count is retained still stalls (the
case in #3554), a release on a remediation policy stalls once its
retries run out, and terminal errors such as an invalid CEL expression
or denied access stall too.
- `Stalled` is copied only when it was observed for the HelmRelease's
current generation. The Application carries no generation, so after a
spec update a client would otherwise read the previous spec's failure as
terminal for the new values; until helm-controller reconciles the new
spec, the Application shows no `Stalled`.
- The counters only count attempts that recorded a Helm release. A
failure before Helm records one, such as a chart that fails to render
(for example a `fail` guard rejecting the values), shows only as
`Ready=False` and leaves them at zero; a failure to apply the rendered
resources is counted.
- helm-controller resets the counters when the desired state changes
(generation, chart version, values), and on success only under
`RetryOnFailure`, so a `Ready` release can still carry non-zero
counters. The field descriptions say so.

`TenantModule` is left unchanged on purpose: it still copies only
`Ready` and `Released`.

Related: #5109, #4155

Tests cover the copied condition and counters, a `Stalled` left over
from an older generation after a spec update, the counters kept on a
`Ready` release, and that `Reconciling` is still not copied. The OpenAPI
schema and apply configurations are regenerated, and every per-kind
status schema picks up the new fields.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The Terraform provider reads only the `Ready` condition by type, so the
extra condition and fields do not change its behaviour. The website only
shows an example Application output, not a description of the status
contract.

### Release note

```release-note
feat(api): Applications now expose the HelmRelease `Stalled` condition and the `installFailures`, `upgradeFailures` and `lastAttemptedReleaseAction` status fields, so tenants without access to HelmReleases can see how a release is failing. Tools that compute health with kstatus, such as Flux Kustomizations waiting on Applications, now report an Application whose release has stalled as failed.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Application status now displays install and upgrade failure counts,
along with the most recently attempted release action. Failure counts
reflect attempts recorded as releases and reset when the desired state
changes.
* Current stalled release conditions are now reflected in application
status, helping clarify when a release is blocked.
* Failure details remain available when a release is ready, while
reconciling conditions are not shown.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-04 19:48:48 +03:00
Alexey Artamonov dbcc2af679 test(clickhouse): require the system-bucket round-trip to run on the sidecar that started without credentials
A Pod restart during the round-trip would hand the sidecar fresh
credentials at start, which is how the old environment-based sidecar
got them, and the Test would pass without the running sidecar ever
picking them up. The Pod uid and the sidecar's restart count must stay
the same across both jobs.

The probe deadline is now measured on the clock: counting only its
pauses let a hanging exec stretch it past the step budget, so the step
timeout would fire before the probe could say what stalled.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-04 19:43:55 +03:00
Alexey Artamonov 6345b939d2 fix(clickhouse): stop reporting a credentials wait once the strategy job runs
The hold sets Ready=False/WaitingForCredentials with a countdown, and
nothing removed it once the Job was created, so a backup or restore
that ran for minutes kept saying it was still waiting. The condition is
dropped when the Job runs; a Ready condition set for another reason is
left alone.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-04 19:43:15 +03:00
Alexey Artamonov 5f636ce624 fix(clickhouse): do not hold strategy jobs on a credentials stamp that cannot expire
The projector rewrites the credentials-updated-at stamp only when the
projected content changes, so a stamp the hold cannot use stays put.
An unreadable one failed every reconcile with no condition on the job,
and one far in the future held each job again on every reconcile; in
both cases every ClickHouse backup and restore in the namespace stopped
for good. Such a stamp is now logged and ignored. A stamp only slightly
ahead, as a skewed clock writes it, still waits, and that wait ends.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-04 19:43:14 +03:00
Alexey Artamonov 38b00730ff fix(clickhouse): keep the clickhouse-backup default region when none is configured
The environment the sidecar used to read had no S3_REGION when the
platform carried no region, which left clickhouse-backup on its own
us-east-1. Writing an empty region into the config file replaces that
default with an empty one, and requests are then signed for no region.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-04 19:40:37 +03:00
Alexey Artamonov 2e37aaec5f docs(backups): drop the claim that a region change needs a ClickHouse restart
The ClickHouse sidecar on the system bucket no longer reads S3 settings
from environment set at start, so a re-projected region reaches it
without a Pod rollout, and the contracts fixture no longer depends on
optional Secret keys to start its Pod.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-04 19:31:59 +03:00
Alexey Artamonov e752952047 test(clickhouse): back up and restore a running release through the system bucket
A ClickHouse release on the system bucket starts before the platform
projects cozy-backups-creds into its namespace, and no suite exercised
that order: the round-trips bring their own Bucket and endpoint, so the
sidecar never had to pick up credentials that arrived after it started.

The new Test starts the release without the projected Secret and runs a
backup and an in-place restore through cozy-default against the bucket
as advertised. In CI that host neither resolves nor has a certificate,
so the step makes it reachable from tenant-root for its own duration
and proves the served certificate before any job starts, which keeps a
DNS or TLS problem from reading as a backup failure.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-04 19:31:29 +03:00
Aleksei Sviridkin c1ba05574f feat(monitoring-agents): keep application identity in container log streams (#5217)
## What this PR does

Makes Fluent Bit keep the pod lineage labels
`apps.cozystack.io/application.{group,kind,name}` on container log
lines, as the stream fields
`apps_cozystack_io_application_{group,kind,name}`, for both the default
container-log output and the per-tenant routed output.

Why: the per-application logs scope in the design proposal [Tenant
application logs](https://github.com/cozystack/community/pull/95)
(section 6) filters by these stream fields. Today the kubernetes filter
fetches pod labels, but they are dropped together with the rest of the
`kubernetes_*` keys before output.

What changes in the pipeline:

- A second `nest` lift of the pod labels, then a rename of the three
lineage keys to their final names before the existing `Remove_wildcard
kubernetes_` runs. All other labels are still removed. Every lifted key
is prefixed, so no other label can collide into the identity names.
- The three keys are added to `_stream_fields` of both container-log
outputs. Events and audit pipelines are unchanged.
- A parsed JSON log line is kept under `log_processed` (already on
`main`), so a line's content cannot create the identity keys.
- Guest Kubernetes clusters are not affected: their monitoring-agents
release replaces the filters and outputs wholesale. A regression test in
`packages/apps/kubernetes` pins that guest streams carry no identity
fields.

Effect on existing streams: VictoriaLogs treats a missing stream field
like an empty one, and the rename writes nothing when the label is
absent. Pods without lineage labels keep exactly the same stream. Pods
with lineage labels get a new stream once after the rollout; their
earlier lines stay reachable by namespace.

#### How it was tested

- helm-unittest for `monitoring-agents` (lift order, rename before
`Remove_wildcard`, stream fields on both outputs, events and audit
untouched) and for `kubernetes` (guest streams unchanged). The
events/audit guard was mutation-checked.
- Verified on a test cluster: real pod log lines arrive, Redis pods'
lines carry the three `apps_cozystack_io_application_*` stream fields,
and Fluent Bit logs show no errors.

#### Known limitation

Guest clusters write to the parent log store without authentication, so
a guest administrator can forge these stream fields as well as the
namespace field, within the store their tenant shares. This is the
write-path trust risk already documented in the proposal.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Downstream: no trigger matched. Candidate: none affected, because the
diff changes `packages/system/monitoring-agents/values.yaml` (a system
package with no schema) and adds tests; the file under
`packages/apps/kubernetes` is a test only, so the app's `values.yaml`
and `values.schema.json` are unchanged.

### Release note

```release-note
feat(monitoring-agents): container log lines of pods that belong to a Cozystack application now carry `apps_cozystack_io_application_{group,kind,name}` stream fields in VictoriaLogs. Streams of pods without application labels are unchanged; streams of application pods start a new stream once after the upgrade.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Container-log streams now include application group, kind, and name
metadata for root and tenant stores.
* Guest monitoring streams include pod name, container name, and
namespace fields.
* **Bug Fixes**
* Application metadata is added to container-log streams without
changing the existing stream fields for event and audit outputs.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-04 18:27:20 +03:00
Aleksei Sviridkin e2283b3758 feat(monitoring-agents): export PVC labels from kube-state-metrics (#5215)
## What this PR does

Adds `persistentvolumeclaims=[*]` to the kube-state-metrics
`--metric-labels-allowlist`, next to the existing `pods=[*]` and
`deployments=[*]`. kube-state-metrics then publishes
`kube_persistentvolumeclaim_labels`, one series per PVC, with the PVC's
labels as `label_*`.

Why: the per-application metrics scope in the design proposal [Tenant
resource-consumption
metrics](https://github.com/cozystack/community/pull/94) (Phase 0) finds
an application's volumes through the `apps.cozystack.io/application.*`
lineage labels on its PVCs. Without this series, volume usage
(`kubelet_volume_stats_*`, which carries `persistentvolumeclaim` but not
`pod`) cannot be attributed to an application.

`[*]` rather than an explicit key list: kube-state-metrics has no prefix
matching, so the three lineage keys would have to be enumerated, and the
series count is the same either way (one per PVC). `[*]` stays
consistent with the existing pods and deployments entries and does not
silently miss a lineage key added later. PVCs are far fewer than pods,
which already export all labels.

#### How it was tested

- helm-unittest for `monitoring-agents`: a new suite asserts the exact
flag in the rendered kube-state-metrics Deployment.
- Verified on a test cluster: `kube_persistentvolumeclaim_labels` is
present and carries `label_apps_cozystack_io_application_name`.
- Not verified: the memory impact on kube-state-metrics.

#### Known limitation

PVCs created from a StatefulSet's `volumeClaimTemplates` usually have no
owner reference and no Flux label, so the lineage webhook gives them no
lineage labels and this series cannot attribute them to an application.
Fixing that belongs to the lineage webhook.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Downstream: no trigger matched. Candidate: none affected, because the
diff touches only `packages/system/monitoring-agents/values.yaml` and a
test in that system package; it adds a series but renames no telemetry
metric, platform value, app field or CRD.

### Release note

```release-note
feat(monitoring-agents): kube-state-metrics now exports PVC labels as `kube_persistentvolumeclaim_labels`, one series per PVC, so volume usage can be attributed to the application that owns it.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Kubernetes monitoring metrics now include labels from pods,
deployments, and persistent volume claims. This expands the resource
metadata available alongside metrics, including lineage labels used by
the tenant metrics API. Metrics for these resource labels are emitted
only for resources covered by the configured label allowlist.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-04 18:22:09 +03:00
Alexey Artamonov f5f50d2a77 fix(clickhouse): hold a strategy job until projected credentials reach the sidecar
The clickhouse-backup sidecar now reads the projected credentials from a
mounted Secret, which kubelet rewrites only after its sync period,
about 75 seconds after the change in practice. The first BackupJob
projects the Secret and started the strategy Job a second later, so its
attempts raced the volume update and the backup could still fail on a
sidecar without credentials.

The projector records when the projected content last changed, and the
ClickHouse driver holds back creating a backup or restore Job until two
minutes have passed since, with a WaitingForCredentials condition. An
unchanged re-projection keeps the time, so only the first projection
and a rotation cause a wait, and a Job already running is never held.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-04 17:15:10 +03:00
Alexey Artamonov c9ce4c480f fix(clickhouse): let the backup sidecar pick up credentials projected after it started
With backup.useSystemBucket the clickhouse-backup sidecar took its S3
settings from optional secretKeyRefs on cozy-backups-creds. The platform
projects that Secret into the namespace at the first BackupJob, usually
after the ClickHouse pod started, and a container's environment is fixed
at start, so the sidecar ran without credentials and every backup and
restore failed until the pod was restarted by hand. Rotated keys stayed
stale the same way.

The projector now also writes the s3 section of a clickhouse-backup
config into the Secret, and the chart mounts that key, optionally, as
the sidecar's config file instead of setting S3 environment variables.
clickhouse-backup reloads its config file in each API handler that
reaches S3 and kubelet rewrites a mounted Secret in place, so projected
and rotated credentials reach the running sidecar. The flat keys stay
for the Redis strategy and for ClickHouse releases still rendered by
the previous chart.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-04 17:12:31 +03:00
Myasnikov Daniil 59f0013b1f fix(backups): attribute a MariaDB dump only when no other run can own it
Every backup of a MariaDB application writes its dump under one shared
prefix, so the probe took the dump whose file-name timestamp (the
backup pod's clock) and LastModified (the object store's clock) both
fall in the operator Backup's lifetime, with a minute of slack for
skew. That keeps a neighbour's dump out only while this run's own dump
is in the window. With the pod clock more than the slack behind, the
own dump is named before the window opens, and a previous run that
finished within the minute before this one was created becomes the
single match: a 123-byte dump was reported as 666 bytes. A pod clock
ahead or an object store clock behind does the same with the next run,
and an own dump that is gone by the time it is looked up, as it can be
for a restore long after the backup, leaves a neighbour as the single
match with no skew at all.

A dump is now attributed only when ownership is proven:

- it is the one dump that either clock places in the window. A dump
  that one clock places there and the other does not may be this run's
  own with a skewed clock, so it leaves any other match unproven, the
  way a second dump in the window already did, and is not taken itself;
- both of its clocks place it in the window;
- neither clock places it in the window of another operator Backup
  that writes under the same bucket and prefix and has completed. This
  rejects a neighbour's dump when this run's own is missing, which the
  clocks alone cannot tell.

Operator Backups still running are not counted as neighbours: their
window has no end, and one stuck short of Complete (a Job whose pod
never starts stays JobRunning for good) would otherwise keep every
later run of the application from being sized. A dump such a run
writes inside this run's window still competes with this run's own.

Narrower slack does not prove ownership: a neighbour that finished
within the narrower slack still matches, and a smaller skew pushes the
own dump out. The Job's start and completion times are stamped by the
control plane, like the operator Backup's own, so they narrow the
window by seconds without removing the dependence on the two clocks a
dump carries. What is left: a neighbour's dump is taken only when this
run's own is missing from the listing or misplaced on both clocks and
the neighbour's operator Backup no longer exists or has not completed.
Every other unproven case leaves the size unset, as two runs that
overlap or follow each other within the slack already did.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-04 13:24:09 +00:00
Myasnikov Daniil 560c44f71e fix(backups): read a MariaDB dump from where its operator Backup wrote it
The size probe took the bucket, endpoint, prefix, credentials and TLS
settings from the strategy as it renders on the reconcile that creates
the Backup, which is not necessarily what it rendered when the operator
Backup was created: an existing operator Backup is reused untouched,
and a backup can run for up to the driver's 30-minute deadline. An
admin who edits the strategy in between sends the probe to the new
location, where it measures the wrong prefix or nothing at all.

The operator Backup's own spec.storage.s3 is where the dump went, so
the probe reads that. It is also the only source a restore can use,
since it targets that operator Backup long after the strategy may have
changed. An operator Backup that does not store to S3 is refused before
anything is listed.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-04 13:18:39 +00:00
Myasnikov Daniil 21f51a2e74 refactor(backups): return the matched MariaDB dump, not only its size
A MariaDB restore has to name the dump it restores from: without a
target time the operator restores the newest dump under the prefix,
which is not necessarily the one of the Backup being restored. The size
probe already finds that dump, but it returns only its size, folds a
missing, an ambiguous and an unreadable listing into the same 0, and
lives on BackupJobReconciler, which the restore reconciler is not.

Split the probe in two: findMariaDBDump reads the bucket and returns the
matched object (key, file name, the timestamp in its name, size and
LastModified) or an error that tells "not found" from "ambiguous"; the
size path keeps its timeout and reads only the size. The Secret lookup
and the S3 client move onto an objectStore value that any reconciler
can build from its own client, and BackupJobReconciler keeps one-line
wrappers so the CNPG probe is unchanged. sizeBytes is reported exactly
as before.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-04 13:16:15 +00:00
Myasnikov Daniil c140354977 test(backups): build operator Backups with the operator's success shape
mariadb-operator marks a Backup Complete=True for a failed Job as well
as a successful one; only the reason tells them apart (JobComplete with
message Success, or JobFailed with message Failed). The fixtures that
stand for a finished backup set Complete=True and nothing else, so they
describe a state the operator never produces and would read as a
failure to a driver that checks the reason. Give them the success shape
the operator writes.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-04 13:13:54 +00:00
Andrey Kolkov 92fa979690 fix(backups): trust only the endpoint CA where the backup writer does
With an endpoint CA the probe trusted the system roots plus that CA.
barman-cloud gets endpointCA as AWS_CA_BUNDLE, which replaces the
default trust store, so a CNPG probe could accept an endpoint the backup
itself would have refused. Let the connection say whether the CA adds to
the system roots or replaces them, and replace them for CNPG.
mariadb-operator appends its CA to the system pool, so MariaDB keeps
that.

Pin in reconcile tests that a Create the apiserver rejects still fails
the BackupJob, so only the read-back of an existing Backup is retried.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:23 +00:00
Andrey Kolkov 080df1b2f2 fix(backups): surface a Backup that cannot be read back and skip its probe
A Backup that exists but cannot be read back is retried through the
controller's backoff, which left the BackupJob Running with nothing on
it saying why. Set Ready=False with reason BackupNotReadBack before
handing the error back.

When the check for an existing Backup itself fails, the Backup may well
exist, so a size probe would only be discarded, and a read that keeps
failing reran it, up to the probe timeout on the single BackupJob
worker, on every retry. Skip the probe in that case and create the
Backup without a size.

Drive both behaviours through the MariaDB and CNPG reconciles in tests.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov 523d74a503 fix(backups): retry instead of failing when an existing Backup cannot be read
When Create collides with a Backup an earlier reconcile already made and
the read that follows fails, the drivers failed the BackupJob, although
the backup and its Backup both exist. Return a distinct error instead,
which the drivers hand back to the controller so the reconcile is
retried with backoff and then finds the Backup.

minio.New already refuses an endpoint with a path prefix and accepts a
trailing slash, so drop the duplicate check in the shared client; the
CNPG probe keeps its own check because it parses the URL itself and
would otherwise drop the path before minio sees it.

Say in the sizeBytes API doc what the value covers, such as the base
backup without the shared WAL archive for CNPG, and that strategies
which cannot attribute stored bytes to one backup leave it unset.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov 2b21763577 fix(backups): do not fail a backup on a transient retry-check read
The check for a Backup an earlier reconcile already created returned
its read error, and the drivers turn any error from that path into a
terminally Failed BackupJob. An apiserver timeout right after the
operator backup completed therefore lost a backup whose data was safely
in S3. The check only saves a probe, and the Create that follows already
handles an existing Backup, so log the error and go on to Create.

A trailing slash on the S3 endpoint is not a path prefix; the operators
accept it, so do not refuse it.

Pin the uncached AlreadyExists refetch, the upper slack of the MariaDB
window and the probe timeout with tests.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov a3d854739d fix(backups): attribute MariaDB dumps on two clocks and read retries uncached
The MariaDB probe identified this run's dump by the timestamp in its
file name, which comes from the backup pod's clock. With that clock
skewed past the slack, this run's dump falls outside the window while a
neighbouring run's dump can be the single match left, and its size is
reported as this backup's. A dump now has to fall inside the window by
both its name and its LastModified, which the object store stamps, so a
single skewed clock leaves the size unset instead.

The name parser also accepts the layout later mariadb-operator releases
write for compressed dumps, backup.<ts>.sql.<ext>, and checks the
compression token in either layout, so an operator upgrade does not
silently stop the measurement.

The retry check and the AlreadyExists refetch read the Backup through
the cache. A retry usually follows a Create made moments earlier; a
cache that does not show it yet turned the refetch into NotFound and
failed a backup that had succeeded. Read it from the API server instead.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov 3b86f57803 fix(backups): refuse to probe an S3 endpoint behind a path prefix
An endpoint such as https://s3.example.org/prefix cannot be addressed
by minio-go, which has no path-prefix option, and the prefix cannot be
added below the client either, because the SigV4 signature covers the
request path. The CNPG probe was worse than failing: it kept only the
host of endpointURL and listed the bucket at a location barman never
writes to.

Reject such endpoints with an explicit error, in the CNPG probe and in
the shared client, so the size is left unset instead of measured
somewhere else.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov 038f35376a fix(backups): do not report a partial listing or re-probe on retry
minio-go ends a listing whose context is cancelled between pages by
closing the channel without an error item. When the probe timeout
expired between pages of a large CNPG base backup, the pages already
read were summed and reported as the measured size. Check the context
once the listing channel closes and leave the size unset instead.

The probe ran before the Backup was created, so a reconcile that retried
after the Backup already existed probed the bucket again, held the
single BackupJob worker for up to the timeout, and then discarded the
result in favour of the existing object. Return an existing Backup
before probing.

Log the ambiguous MariaDB case (several dumps in the window) at info like
the other reasons the size stays unset, and cover the session-token
credential path with a test.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00