Commit Graph
7796 Commits
Author SHA1 Message Date
scooby87 a7240ba0ee fix(backups): delete a MariaDB dump from S3 when its Backup is deleted (#5233)
## What this PR does

Deleting a MariaDB `Backup` removed only the Cozystack object: the dump
stayed in the bucket and the `k8s.mariadb.com/Backup` stayed in the
namespace. Nothing else prunes such a dump on the platform bucket.
mariadb-operator's `maxRetention` runs only inside the next backup run
of the same prefix, and a `Plan` never deletes a `Backup`.

Deleting a MariaDB `Backup` now deletes its dump, then the operator
`Backup` that wrote it, and only then lets the `Backup` go. Every run of
an application writes under the same prefix, so the only key ever
deleted is one proven to belong to this `Backup`:

- a `Backup` taken with #5227 records its dump in `driverMetadata`
(`k8s.mariadb.com/dump-file`), and that key is deleted unless another
run of the same prefix, finished or not, could have written it. The size
probe that records the name lets only runs finished by then contest it,
so a run that uploaded its dump and pruned this one through
`maxRetention` before completing can have its dump recorded here; the
delete contests the recorded dump again against every run as it stands,
keeps it when any could own it, and releases the `Backup` with a
`Warning`;
- a `Backup` taken before that is matched with the same ownership-proven
attribution #5223 introduced for the size probe and #5227 uses for
legacy restores: one dump in the operator `Backup`'s lifetime on both
clocks (the timestamp in the name and `LastModified`), and in the
lifetime of no other run of the same prefix, finished or not: a run that
has not completed yet may already have uploaded its dump and pruned this
one through `maxRetention`, so its window is treated as still open. The
match is recorded on the `Backup` before the delete, so a retry deletes
the same key by name;
- when no dump is proven, the `Backup` is released with a `Warning`
event and nothing is deleted, the operator `Backup` included, since it
is what any later attribution starts from.

The delete runs inside the controller through the minio client #5223
added, the same library mariadb-operator writes with, so TLS is verified
the same way. Bucket, prefix and dump name come from the operator
`Backup`. When the operator `Backup` writes to the platform bucket,
which it names by the bucket and by the projected `cozy-backups-creds`
Secret, the endpoint and keys are the platform's current ones (read from
the Secret `cozy-backups-creds` is projected from, TLS from the scheme
of the platform endpoint, as the default strategy decides it, still
trusting a CA the strategy pinned), so a `Backup` taken before the
endpoint moved is still deleted. Any other bucket is reached as the
operator `Backup` recorded it.

Other cases:

- a restore from the `Backup` holds the deletion while it runs: a
`RestoreJob` of the `Backup` that has not finished, or an operator
`Restore` of its operator `Backup` that mariadb-operator has not marked
`Complete` (it does so once the Restore's Job has completed or failed).
The second check matters because a `RestoreJob` can fail on its deadline
while its operator `Restore` keeps reading the dump. Both are listed
from the API server rather than the informer cache, so a restore that
started just before the delete is not missed;
- a restore does not start from a `Backup` that is being deleted or is
gone: the MariaDB restore reads the `Backup` from the API server before
creating the operator `Restore` and fails the `RestoreJob` otherwise,
since mariadb-operator restores the nearest other dump once the
requested one is deleted. A failed lookup of an existing operator
`Restore` is retried rather than failing the `RestoreJob`;
- a failed dump delete keeps the `Backup` in `Terminating` and retries
every 30 seconds with a `Warning` naming the
`backups.cozystack.io/skip-artifact-cleanup` annotation, which releases
the `Backup` and deletes nothing; a failed delete of the operator
`Backup` is retried with the controller's backoff;
- a missing or malformed platform credentials Secret falls back to the
credentials the operator `Backup` names, so it never releases a `Backup`
on its own: a delete that then fails holds it as above. A failed read of
that Secret (an API error) is retried instead, because the recorded
endpoint may be the one the platform moved off;
- a recorded dump that is not in the listing is not deleted again: the
operator `Backup` is deleted and a `Normal` `ArtifactAlreadyAbsent`
event names the key and the endpoint that was listed, so a dump left on
storage the platform moved off can still be found;
- in a terminating namespace a failed delete releases the `Backup` with
a `Warning`, so it does not hold the namespace;
- an operator `Backup` that is gone, that stores on a PVC or volume, or
whose Job failed (`Complete=True` with reason `JobFailed`, which wrote
no dump) releases the `Backup` with a `Warning` and nothing is deleted,
as does a cluster without the mariadb-operator CRDs.

The controller gains the `delete` verb on `k8s.mariadb.com/backups`. No
Job, image or flag is added.

Known limitation: when a namespace is deleted with its `Backup`s, the
namespace controller may delete the operator `Backup` first, and the
`Backup` is then released with a `Warning` and its dump left behind,
because the bucket and prefix are read from the operator `Backup`.

Known limitation: an operator `Backup` that never completes, for example
one whose Job pod never starts, contests every later dump its lifetime
covers on that prefix, so those `Backup`s are released with a `Warning`
and their dumps left behind. That leaks storage rather than deleting a
dump another `Backup` may restore from. The contest reads the runs from
the operator `Backup`s that still exist, so a run whose operator
`Backup` was deleted by hand no longer contests a dump. Likewise, an
operator `Restore` that never completes holds its `Backup`'s deletion
until the skip annotation releases it. These and the namespace-teardown
limitation above are tracked in #5291.

Stacked on #5227 and targets its branch until that merges; only the
commits after #5227's head belong here.

Tests: controller tests on the fake client cover each path above,
including the window bound (`takenAt`, not the operator's `Complete`
transition) of the legacy match, ordering (the operator `Backup`
survives a failed dump delete, and a legacy `Backup` retried after a
failed operator `Backup` delete finishes by its recorded name) and the
finalizer at `Reconcile` level. The minio adapter is tested against an
HTTP server for the escaped key and an error response. A helm-unittest
pins the RBAC verbs. An unfinished neighbour run is covered both ways:
one that started inside the `Backup`'s lifetime keeps its candidate
dump, one that started after it does not contest an earlier dump. The
coordination with restores is covered from both sides: a `RestoreJob` or
operator `Restore` that has not finished, including one only the API
server has seen yet, holds the deletion, and a restore from a `Backup`
being deleted or gone is refused without creating an operator `Restore`.
A recorded dump is contested the same way, down to the full sequence of
the size probe recording a dump an unfinished run uploaded and the
`Backup` then being deleted, and a recorded dump already gone is not
contested. Every condition was mutated, each mutation turns a test red,
including the timeout that bounds an unreachable endpoint. Checked on a
test cluster against SeaweedFS with the controller function and its real
minio client: over the in-cluster `seaweedfs-s3.<ns>:8333` endpoint with
its wildcard certificate a recorded dump and a legacy dump were deleted
with their operator `Backup`s, a neighbouring dump was kept, and a
recorded dump already gone completed; with a dead recorded endpoint the
dump on the platform bucket was deleted through the platform's current
endpoint; on a foreign bucket behind a dead endpoint the `Backup` was
held with everything kept. The contest of a recorded dump and the
coordination with restores were added after that run and are covered by
the controller tests only.

### Screenshots

Not a UI change.

### Downstream repositories

Walked against the diff. It touches no CRD or API type, no app
`values.schema.json`, no `packages/core/platform/values.yaml`, no
`hack/` tooling the trigger map names, and no ApplicationDefinition.
Only `cozystack/website` is affected:
`content/en/docs/next/operations/services/backup-classes.md` describes
deletion of the artifact on `Backup` removal for Rabbitmq, Redis, Kafka
and the MongoDB system bucket but not for MariaDB.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: needed, not opened yet: document MariaDB dump deletion on
`Backup` removal and the skip annotation
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(backups): deleting a MariaDB Backup on S3 storage now deletes its dump and its k8s.mariadb.com/Backup before the Backup goes. Only a dump proven to be the Backup's own is deleted: the one it records, unless another run could have written it, or for older Backups the one that can be attributed to its run. Otherwise the Backup is released with a Warning event and nothing is deleted. A failed delete keeps the Backup Terminating and retries; the backups.cozystack.io/skip-artifact-cleanup annotation releases it and keeps the dump. Deletion waits while a RestoreJob or an operator Restore reads the Backup's dump, and a restore from a Backup being deleted is refused.
```
2026-11-13 15:07:50 +03:00
Alexey Artamonov b3ca798bc4 fix(backups): read the runs that may own a MariaDB dump uncached
Before deleting a dump the cleanup checks every other operator Backup
on the same prefix that could have written it, and it read them from
the informer cache. A run whose operator Backup had not reached the
cache yet but had already uploaded its dump was missed, and its dump
deleted as this Backup's. Both the recorded and the legacy paths now
list the runs from the API server, as the restore checks already do.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-12 17:01:29 +03:00
Alexey Artamonov a6a704768c test(backups): pin a failed RestoreJob listing in the MariaDB cleanup
A RestoreJob listing that fails says nothing about whether a restore
reads the dump. Swallowing that error left the whole suite green, while
it would let the cleanup delete the dump under a running restore.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 15:41:03 +03:00
Alexey Artamonov f2507b4b84 docs(backups): name what the MariaDB cleanup reads uncached
The reader's comment named only RestoreJobs, and the restore path said
the cleanup holds a Backup only for RestoreJobs it can see, while the
cleanup now lists both RestoreJobs and operator Restores straight from
the API server.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 15:35:33 +03:00
Alexey Artamonov 0195a66fae fix(backups): release a MariaDB Backup when the operator CRDs are gone
Listing the operator Restores of a Backup before reading its operator
Backup made the cleanup fail on a cluster without the mariadb-operator
CRDs, retried for good, so the Backup and its namespace never went
away. The operator Backup read that follows already releases the
Backup in that case; with no Restore kind no restore can run, so the
listing now lets the cleanup reach it.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 15:35:33 +03:00
Alexey Artamonov 8246808289 fix(backups): hold a MariaDB dump while an operator Restore reads it
The cleanup waited for a restore by the phase of its RestoreJob, but the
phase is not the restore: a RestoreJob fails on its deadline while the
operator Restore it created keeps running, and that Restore still reads
the dump. Deleting the dump then made mariadb-operator restore the
nearest other one.

The cleanup now also holds the Backup while an operator Restore of its
operator Backup has not been marked Complete, which mariadb-operator
does once the Restore's Job has completed or failed, so what keeps the
dump is the restore that reads it rather than the status of the
request.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 15:30:41 +03:00
Alexey Artamonov ee6404a0c8 fix(backups): retry a failed lookup of a MariaDB operator Restore
The lookup now goes to the API server, which can time out or throttle,
and a failed lookup failed the RestoreJob. It says nothing about
whether the operator Restore exists, so the RestoreJob could be failed
while its Restore was running, and the Backup's cleanup would then stop
waiting for it and delete the dump the Restore reads.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 15:28:45 +03:00
Alexey Artamonov 14a842ad16 fix(backups): find a MariaDB operator Restore the cache has not seen
The MariaDB restore creates its operator Restore once and then follows
it, finding it by label in the informer cache. Nothing watches operator
Restores, so a reconcile that ran before the new Restore reached the
cache saw none: it created a second one, or, now that a restore refuses
a Backup being deleted, failed the RestoreJob while its Restore was
still running, after which the cleanup deleted the dump that Restore
was reading.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 15:15:18 +03:00
Alexey Artamonov d57ebbe808 docs(backups): state what the MariaDB dump delete does not cover
The comment on who deletes a MariaDB Backup listed namespace teardown
without saying that teardown can remove the operator Backup first, in
which case the dump is left behind with a Warning, so it promised a
delete the code does not make. The remover contract claimed a missing
key is never an error, which holds only for a backend that answers the
delete with 204, the one status minio-go accepts.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 15:09:15 +03:00
Alexey Artamonov 3334152da7 fix(backups): report a recorded MariaDB dump that was not found
A recorded dump missing from the listing was logged as deleted, with no
event. That is right when an earlier attempt or maxRetention removed it,
but the listing goes through the platform's current endpoint, and after
an endpoint move the dump can still sit on the old storage: the cleanup
then deleted the operator Backup that pointed at it and said it had
deleted the dump.

The cleanup now records a Normal ArtifactAlreadyAbsent event naming the
key and the endpoint it listed, so whoever later looks for that dump
can tell where it was not found.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 15:08:43 +03:00
Alexey Artamonov bbeca34535 fix(backups): retry a failed read of the platform backup credentials
The MariaDB cleanup reaches the platform bucket through the platform's
current endpoint so that a Backup taken before the endpoint moved is
still deleted. A failed read of the platform credentials made it fall
back to the endpoint recorded on the operator Backup, which may be the
old address: there the dump is not found, a missing dump counts as
already deleted, and the operator Backup that pointed at the real dump
was deleted with a success log.

A read that fails with an API error now holds the Backup and retries.
A missing or malformed platform Secret still falls back, since it is a
standing state that a retry would not change.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 15:07:03 +03:00
Alexey Artamonov 37a74e5939 test(backups): reach the S3 calls the MariaDB cleanup timeout bounds
Contesting a recorded dump lists the bucket before the delete, and the
timeout test supplied no lister, so a real client failed on DNS before
the blocking delete was ever called and the suite stayed green with the
timeout removed. The test now lists the dump through the fake bucket,
blocks in the listing and in the delete in turn, and requires that the
blocking call was entered.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 15:05:51 +03:00
Alexey Artamonov 654c543264 fix(backups): see a MariaDB restore the cache has not caught up with
A MariaDB restore now checks on the API server that its Backup is not
being deleted before it starts, so a restore that started first exists
on the API server before the delete. The cleanup read RestoreJobs from
the informer cache, which can lag behind, and could delete the dump
under a restore it had not seen yet. Reading them uncached closes that
window from both sides.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 15:04:53 +03:00
Alexey Artamonov 320d0299ad fix(backups): refuse a MariaDB restore from a Backup being deleted
Deleting a MariaDB Backup now deletes its dump, and mariadb-operator
restores the dump nearest the requested time when the exact one is
gone. A restore that starts while the Backup's cleanup runs can lose
its dump between the check and the operator's read, and then completes
on another run's data without any error.

The restore now reads the Backup from the API server before creating
the operator Restore and fails when it is being deleted or is gone. The
copy the reconcile was handed comes from the cache and can predate the
delete; a transient read error keeps the RestoreJob waiting like any
other failed dump check.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 15:03:59 +03:00
Alexey Artamonov 74fbb34aca docs(backups): say who deletes a MariaDB Backup
The comment justified deleting a dump in a tenant's own bucket by the
tenant asking for it, but tenant roles grant no delete on Backups: only
an administrator or the namespace teardown deletes one, and the reader
of this branch should know whose request the delete answers.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-09 18:47:08 +03:00
Alexey Artamonov 0972690f0d fix(backups): contest a recorded MariaDB dump before deleting it
The size probe records the dump it attributes to a Backup, and it lets
only runs that had already completed contest the attribution. A run can
upload its dump and prune an older run's through maxRetention before its
own operator Backup completes, so the probe of the pruned run can record
the newer run's dump, and deleting that Backup then deleted a dump that
another Backup still restores from.

Deletion now contests the recorded dump against every run on the same
bucket and prefix as they stand at deletion time, the unfinished ones
included, and leaves it behind with a Warning when any could have
written it, as the unrecorded path already does. A run that never
completes therefore keeps every later dump its lifetime covers, which
leaks storage rather than losing a backup. The probe is unchanged here.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-09 14:03:21 +03:00
Alexey Artamonov e66b764734 fix(backups): let an unfinished MariaDB run block a legacy dump delete
A Backup taken before dumps were recorded finds its dump by window, and
only finished neighbour runs were allowed to contest a candidate. A run
uploads its dump and prunes older ones through maxRetention before its
operator Backup turns Complete, so in that gap the pruned run's cleanup
saw the newer run's dump as the only candidate and deleted it, leaving
that Backup Ready with nothing to restore.

The size probe and the restore keep ignoring unfinished runs, where a
wrong guess costs a size or a refused restore. Deletion now treats a run
that has not completed as one whose window is still open, so the dump is
left behind with a Warning, as for any other unproven attribution.
Holding the Backup would not help: once that run completes its window
still covers the candidate, and the attribution stays ambiguous.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-08 20:21:22 +03:00
Alexey Artamonov f147b22d86 fix(backups): delete a MariaDB dump when its Backup is deleted
Deleting a MariaDB Backup left its dump in the bucket for good:
mariadb-operator's maxRetention prunes only from inside the next run of
the same prefix, and a Plan never deletes a Backup, so on the platform
bucket nothing else ever removes it.

Every run of an application writes under one prefix, so the key deleted
has to be proven to be this Backup's. It is the dump the Backup records,
or, for a Backup taken before dumps were recorded, the one the existing
ownership match attributes to its operator Backup. That match is
recorded on the Backup before the delete, so a retry deletes the same
key by name instead of matching again against a bucket that no longer
holds it. Without proof the Backup is released with a Warning and
nothing is deleted, the operator Backup included, since any later
attribution starts from it. The operator Backup is deleted only after
its dump, and the delete waits while a RestoreJob still restores from
the Backup.

The delete runs in the controller through the same minio client the
size probe uses, so it verifies TLS the way mariadb-operator did when
it wrote the dump. A dump on the platform bucket, which the operator
Backup names by its bucket and by the projected credentials Secret, is
reached through the platform's current endpoint and keys, still
trusting a CA the strategy pinned, so a Backup taken before the
endpoint moved is still deleted.

Platform credentials that cannot be read fall back to the ones the
operator Backup names. A failed delete holds the Backup and retries
with a Warning naming the skip-artifact-cleanup annotation, except in
a terminating namespace, where the Backup is released with a Warning
so it does not hold the namespace.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-06 17:24:29 +03:00
Alexey Artamonov 55f1fd33a8 fix(backups): let the controller delete operator MariaDB Backups
Deleting a MariaDB Backup now deletes the k8s.mariadb.com/Backup that
wrote its dump, once the dump itself is gone, so the controller needs
the delete verb on that resource. Restores stay create-only.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-06 16:04:04 +03:00
Myasnikov Daniil 195de033e8 fix(mariadb): pin legacy Backups to a proven dump, refuse failed runs
A Backup taken before Backups recorded their dump gives a restore
nothing to pin. Refusing such Backups would leave every existing one
unrestorable, so a restore of one now makes the same ownership-proven
match the backup side makes, against the operator Backup it names, and
restores that dump by its timestamp. When no dump can be proven to be
the run's own, or the storage cannot be read, mariadb-operator restores
the newest dump under the prefix as it always did, and a BestEffortDump
Warning on the RestoreJob says so and why.

An operator Backup whose Job failed is Complete=True with reason
JobFailed, and a BackupJob that took Complete=True for success made a
Backup of it. That run left no dump of its own, and restoring it would
quietly bring back another run's dump, so the RestoreJob now fails
instead. The condition is read by its reason, because its status does
not tell a failure from a success.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-04 15:57:20 +00:00
Myasnikov Daniil 21d1e9f02b fix(mariadb): restore the dump a Backup records
A RestoreJob created a mariadb-operator Restore that named only the
operator Backup. The operator then lists the prefix that Backup wrote
to and restores the dump nearest to the moment the restore started:
the application's newest dump, not the one the Backup was taken as.

When the Backup records its dump, the Restore now carries the
timestamp in that dump's name as targetRecoveryTime. mariadb-operator
returns the dump whose timestamp equals the target before it looks for
the nearest one, and the nearest one is exactly what it would restore
if the recorded dump were gone. So the controller first checks, with
its own bounded call, that the dump is still in the operator Backup's
storage. A dump that is gone fails the RestoreJob, naming the dump,
instead of restoring a neighbour in its place. A check that cannot be
made keeps the RestoreJob waiting with Ready=False until the restore
deadline, then fails it with the cause: the Backup could record its
dump only because the controller reached that storage when it was
taken, so an unreachable storage is not a reason to guess.

The dump is chosen once, as the operator Restore is created; a Restore
that already exists is followed as before.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-04 15:53:38 +00:00
Myasnikov Daniil a042f33590 fix(mariadb): record the dump a Backup was taken as
Every run of a MariaDB application writes its dump under the same
storage prefix, and the Backup did not say which of those dumps is its
own. A restore had nothing to give mariadb-operator but the operator
Backup, which resolves to the newest dump under the prefix.

The Backup now records the dump's file name and the timestamp in that
name in driverMetadata, taken from the match the size probe already
makes. Nothing is recorded when that match cannot prove the dump
belongs to the operator Backup, so a Backup never names another run's
dump.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-04 15:49:35 +00:00
Myasnikov Daniil 59f0013b1f fix(backups): attribute a MariaDB dump only when no other run can own it
Every backup of a MariaDB application writes its dump under one shared
prefix, so the probe took the dump whose file-name timestamp (the
backup pod's clock) and LastModified (the object store's clock) both
fall in the operator Backup's lifetime, with a minute of slack for
skew. That keeps a neighbour's dump out only while this run's own dump
is in the window. With the pod clock more than the slack behind, the
own dump is named before the window opens, and a previous run that
finished within the minute before this one was created becomes the
single match: a 123-byte dump was reported as 666 bytes. A pod clock
ahead or an object store clock behind does the same with the next run,
and an own dump that is gone by the time it is looked up, as it can be
for a restore long after the backup, leaves a neighbour as the single
match with no skew at all.

A dump is now attributed only when ownership is proven:

- it is the one dump that either clock places in the window. A dump
  that one clock places there and the other does not may be this run's
  own with a skewed clock, so it leaves any other match unproven, the
  way a second dump in the window already did, and is not taken itself;
- both of its clocks place it in the window;
- neither clock places it in the window of another operator Backup
  that writes under the same bucket and prefix and has completed. This
  rejects a neighbour's dump when this run's own is missing, which the
  clocks alone cannot tell.

Operator Backups still running are not counted as neighbours: their
window has no end, and one stuck short of Complete (a Job whose pod
never starts stays JobRunning for good) would otherwise keep every
later run of the application from being sized. A dump such a run
writes inside this run's window still competes with this run's own.

Narrower slack does not prove ownership: a neighbour that finished
within the narrower slack still matches, and a smaller skew pushes the
own dump out. The Job's start and completion times are stamped by the
control plane, like the operator Backup's own, so they narrow the
window by seconds without removing the dependence on the two clocks a
dump carries. What is left: a neighbour's dump is taken only when this
run's own is missing from the listing or misplaced on both clocks and
the neighbour's operator Backup no longer exists or has not completed.
Every other unproven case leaves the size unset, as two runs that
overlap or follow each other within the slack already did.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-04 13:24:09 +00:00
Myasnikov Daniil 560c44f71e fix(backups): read a MariaDB dump from where its operator Backup wrote it
The size probe took the bucket, endpoint, prefix, credentials and TLS
settings from the strategy as it renders on the reconcile that creates
the Backup, which is not necessarily what it rendered when the operator
Backup was created: an existing operator Backup is reused untouched,
and a backup can run for up to the driver's 30-minute deadline. An
admin who edits the strategy in between sends the probe to the new
location, where it measures the wrong prefix or nothing at all.

The operator Backup's own spec.storage.s3 is where the dump went, so
the probe reads that. It is also the only source a restore can use,
since it targets that operator Backup long after the strategy may have
changed. An operator Backup that does not store to S3 is refused before
anything is listed.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-04 13:18:39 +00:00
Myasnikov Daniil 21f51a2e74 refactor(backups): return the matched MariaDB dump, not only its size
A MariaDB restore has to name the dump it restores from: without a
target time the operator restores the newest dump under the prefix,
which is not necessarily the one of the Backup being restored. The size
probe already finds that dump, but it returns only its size, folds a
missing, an ambiguous and an unreadable listing into the same 0, and
lives on BackupJobReconciler, which the restore reconciler is not.

Split the probe in two: findMariaDBDump reads the bucket and returns the
matched object (key, file name, the timestamp in its name, size and
LastModified) or an error that tells "not found" from "ambiguous"; the
size path keeps its timeout and reads only the size. The Secret lookup
and the S3 client move onto an objectStore value that any reconciler
can build from its own client, and BackupJobReconciler keeps one-line
wrappers so the CNPG probe is unchanged. sizeBytes is reported exactly
as before.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-04 13:16:15 +00:00
Myasnikov Daniil c140354977 test(backups): build operator Backups with the operator's success shape
mariadb-operator marks a Backup Complete=True for a failed Job as well
as a successful one; only the reason tells them apart (JobComplete with
message Success, or JobFailed with message Failed). The fixtures that
stand for a finished backup set Complete=True and nothing else, so they
describe a state the operator never produces and would read as a
failure to a driver that checks the reason. Give them the success shape
the operator writes.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-04 13:13:54 +00:00
Andrey Kolkov 92fa979690 fix(backups): trust only the endpoint CA where the backup writer does
With an endpoint CA the probe trusted the system roots plus that CA.
barman-cloud gets endpointCA as AWS_CA_BUNDLE, which replaces the
default trust store, so a CNPG probe could accept an endpoint the backup
itself would have refused. Let the connection say whether the CA adds to
the system roots or replaces them, and replace them for CNPG.
mariadb-operator appends its CA to the system pool, so MariaDB keeps
that.

Pin in reconcile tests that a Create the apiserver rejects still fails
the BackupJob, so only the read-back of an existing Backup is retried.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:23 +00:00
Andrey Kolkov 080df1b2f2 fix(backups): surface a Backup that cannot be read back and skip its probe
A Backup that exists but cannot be read back is retried through the
controller's backoff, which left the BackupJob Running with nothing on
it saying why. Set Ready=False with reason BackupNotReadBack before
handing the error back.

When the check for an existing Backup itself fails, the Backup may well
exist, so a size probe would only be discarded, and a read that keeps
failing reran it, up to the probe timeout on the single BackupJob
worker, on every retry. Skip the probe in that case and create the
Backup without a size.

Drive both behaviours through the MariaDB and CNPG reconciles in tests.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov 523d74a503 fix(backups): retry instead of failing when an existing Backup cannot be read
When Create collides with a Backup an earlier reconcile already made and
the read that follows fails, the drivers failed the BackupJob, although
the backup and its Backup both exist. Return a distinct error instead,
which the drivers hand back to the controller so the reconcile is
retried with backoff and then finds the Backup.

minio.New already refuses an endpoint with a path prefix and accepts a
trailing slash, so drop the duplicate check in the shared client; the
CNPG probe keeps its own check because it parses the URL itself and
would otherwise drop the path before minio sees it.

Say in the sizeBytes API doc what the value covers, such as the base
backup without the shared WAL archive for CNPG, and that strategies
which cannot attribute stored bytes to one backup leave it unset.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov 2b21763577 fix(backups): do not fail a backup on a transient retry-check read
The check for a Backup an earlier reconcile already created returned
its read error, and the drivers turn any error from that path into a
terminally Failed BackupJob. An apiserver timeout right after the
operator backup completed therefore lost a backup whose data was safely
in S3. The check only saves a probe, and the Create that follows already
handles an existing Backup, so log the error and go on to Create.

A trailing slash on the S3 endpoint is not a path prefix; the operators
accept it, so do not refuse it.

Pin the uncached AlreadyExists refetch, the upper slack of the MariaDB
window and the probe timeout with tests.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov a3d854739d fix(backups): attribute MariaDB dumps on two clocks and read retries uncached
The MariaDB probe identified this run's dump by the timestamp in its
file name, which comes from the backup pod's clock. With that clock
skewed past the slack, this run's dump falls outside the window while a
neighbouring run's dump can be the single match left, and its size is
reported as this backup's. A dump now has to fall inside the window by
both its name and its LastModified, which the object store stamps, so a
single skewed clock leaves the size unset instead.

The name parser also accepts the layout later mariadb-operator releases
write for compressed dumps, backup.<ts>.sql.<ext>, and checks the
compression token in either layout, so an operator upgrade does not
silently stop the measurement.

The retry check and the AlreadyExists refetch read the Backup through
the cache. A retry usually follows a Create made moments earlier; a
cache that does not show it yet turned the refetch into NotFound and
failed a backup that had succeeded. Read it from the API server instead.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov 3b86f57803 fix(backups): refuse to probe an S3 endpoint behind a path prefix
An endpoint such as https://s3.example.org/prefix cannot be addressed
by minio-go, which has no path-prefix option, and the prefix cannot be
added below the client either, because the SigV4 signature covers the
request path. The CNPG probe was worse than failing: it kept only the
host of endpointURL and listed the bucket at a location barman never
writes to.

Reject such endpoints with an explicit error, in the CNPG probe and in
the shared client, so the size is left unset instead of measured
somewhere else.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov 038f35376a fix(backups): do not report a partial listing or re-probe on retry
minio-go ends a listing whose context is cancelled between pages by
closing the channel without an error item. When the probe timeout
expired between pages of a large CNPG base backup, the pages already
read were summed and reported as the measured size. Check the context
once the listing channel closes and leave the size unset instead.

The probe ran before the Backup was created, so a reconcile that retried
after the Backup already existed probed the bucket again, held the
single BackupJob worker for up to the timeout, and then discarded the
result in favour of the existing object. Return an existing Backup
before probing.

Log the ambiguous MariaDB case (several dumps in the window) at info like
the other reasons the size stays unset, and cover the session-token
credential path with a test.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov f222e52863 fix(backups): harden the artifact size probe
The probe runs inside the reconcile that creates the Backup, and the
BackupJob controller has a single worker, so its timeout delays every
other BackupJob while an endpoint hangs. Lower it from 30s to 10s.

List the MariaDB prefix non-recursively, as mariadb-operator does. The
default MariaDB prefix and the default CNPG destinationPath are the same
<namespace>/<name>/ path, so a recursive listing of a MariaDB backed up
next to a same-named Postgres walked the whole WAL archive and ran out of
time. The CNPG base-backup prefix is still listed recursively.

Build the custom-CA transport from minio.DefaultTransport so it keeps the
proxy-from-environment and dial/TLS timeout settings a bare
http.Transport drops, and cover it with a test against a TLS endpoint
with a self-signed certificate.

Log the reasons the size could not be measured (credentials, client,
listing, empty CNPG prefix) at info. The chart runs the controller at
info, which hid them before. The expected ambiguity of several MariaDB
dumps in the window stays at debug.

Skip an endpointCA without a Secret name, as buildBarmanObjectStore does.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov de60488a2c fix(backups): report sizeBytes for CNPG backup artifacts
CNPG backups arrived without Backup.status.artifact.sizeBytes:
cnpg.io/Backup status carries no size, and the data lives in barman's
object-store layout rather than in a single object.

barman-cloud uploads every base backup under
<destinationPath>/<serverName>/base/<backupID>/ (backup.info plus the
data and tablespace tarballs, possibly split into parts). On completion
the driver now lists that prefix with the credentials and endpoint CA of
the rendered barmanObjectStore and records the total. The WAL archive is
not counted: it is one stream shared by every base backup of the server.
The field stays unset when the backup ID is unknown, the bucket cannot be
read, or nothing is stored under the prefix.

The S3 client and Secret lookup are now shared with the MariaDB driver.

Refs #4316

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov d4ad5612d3 fix(backups): report sizeBytes for MariaDB backup artifacts
Backup.status.artifact.sizeBytes was only ever set by the etcd strategy,
so MariaDB backups always arrived without a size.

mariadb-operator writes each logical dump as
<prefix>/backup.<RFC3339>[.<compression>].sql under a prefix shared by
every backup of the application, and the URI the driver records is
synthetic, so there is no single object to HEAD. On completion the driver
now lists the prefix with the tenant credentials the operator Backup was
rendered with and takes the size of the one dump whose timestamp falls
inside the operator Backup's lifetime. When the bucket cannot be read or
the window does not match exactly one dump, the field stays unset.

The API doc now states that an absent sizeBytes means "unknown", and the
Velero strategy documents why it leaves the field unset: kopia stores
data content-addressed across backups, so there is no per-backup stored
size.

Refs #4316

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
myasnikovdaniil 4087218a02 docs: move the repo-only guides onto the website (#5034)
## What this PR does

Seven guides under `docs/` were operator facing and written alongside
the code, and left in the source tree where no user looks. They are on
the website now, so this deletes them here and repoints every reference.

```
docs/operations/backup-classes.md                  -> operations/services/backup-classes
docs/operations/resource-presets.md                -> guides/resource-management
docs/storage-immutability.md                       -> operations/storage-class-immutability
docs/operations/system-memory-limits.md            -> operations/configuration/system-memory-limits
docs/gateway-classes.md                            -> networking/gateway-classes
docs/observability/distributed-tracing.md          -> operations/services/monitoring/distributed-tracing
docs/operations/seaweedfs-capacity-for-backups.md  -> operations/services/object-storage/capacity-for-backups
```

`docs/release.md`, `docs/deprecating-applications.md` and the two
SeaweedFS runbooks in `docs/operations/` stay. A workflow, `AGENTS.md`,
a migration and a bats suite read them by path, which is the test from
#3777.

References point at `https://cozystack.io/docs/v1/...`. The site
rewrites that short prefix to the latest version in `layouts/404.html`,
so the links follow releases and pin none. Backup Classes, resource
management and `storageClass` immutability exist in the latest version
already. The other four pages are only in `next/` until 1.7.0 is
promoted, and their links start to resolve then. The generated mongodb
files come from `values.yaml` through `make generate` with
cozyvalues-gen v1.7.0, the version pinned in
`.github/workflows/pre-commit.yml`.

Tests: helm unit tests for mongodb (151), postgres (209), vm-disk (114)
and kubernetes-nodes (223), go tests for backupcontroller, presets and
the operator, rd-presets-check, the migration-50 bats suite.

### Downstream repositories

- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#763

Walked the rest of the trigger map against the diff. No provider
follow-up: the mongodb `values.schema.json` diff is one `description`
string, with no field added, removed or renamed, no enum, no default,
and no `kind`, `plural` or `release.prefix` change.
`packages/core/platform/values.yaml` changed in comments only and no
key, so the hand written platform-package table on the site is
unaffected.

Please merge after cozystack/website#763, the links in the code only
resolve once those pages deploy.

### Release note

```release-note
docs: seven guides move from the docs/ directory of this repository to cozystack.io (Backup Classes, resource presets, storageClass immutability, system component memory limits, GatewayClass selection, distributed tracing, SeaweedFS capacity for backups), and in-repo references point there.
```
2026-11-03 23:09:46 +05:00
Timofei Larkin e31f518df7 ci(api-review-gate): publish the gate as a commit status (#3440)
## What this PR does

Publishes the `API Review Gate` merge context as a commit status, so a
skipped or cancelled run cannot pass it or poison it, and gives fork PRs
the same gate through a privileged `workflow_run` workflow.

### Why

The merge context was a job name. A required job that skips counts as
passed, and check runs are not deduplicated per name, so one cancelled
run on a head keeps the context failed until the next push. Measured on
this repo: `41d1153c` has four check runs with the gate's job name (one
`cancelled`, three `success`). Commit statuses are deduplicated per
context and the newest wins, so a run with no verdict posts nothing and
the previous status stays.

### What changes

Same-repo PRs get status from `api-gate-report`. A `pending` opener
marks the head before the evaluation on approve, dismiss and
request-changes reviews, so an old `success` is not mergeable for the
minute the evaluation takes. `COMMENTED` reviews no longer count as a
reviewer's latest review, so an owner who approved and then commented
keeps the approval. Concurrency is split, so bot comment run cannot
cancel live evaluation.

Fork PRs: `api-review-gate-fork.yaml` runs from the default branch and
reaches the verdict itself, the fork's run is only a trigger. `resolve`
finds PR by head SHA in open PR list (review runs on a fork report base
repo as head repository and have no pull requests). `detect` fetches the
head as data and runs the detector in a job with `contents: read` only.
`conclude` applies the approval rule and posts. A sweep every 15 minutes
moves the status of fork PRs whose runs wait for a maintainer in
`action_required`, those never fire `workflow_run`. This is lllamnyp's
patch, applied whole, with these changes on top:

- two open fork PRs on one head fail closed in `resolve` and in the
sweep, a head carried by a same-repo PR stays with `api-gate-report`
- detector is built from default branch only, merge base is picked by PR
author and an old one would mean an old detector
- the fork `detect` step runs under `bash -e` and detector exit 2 ended
it before the code was read, so it has `set +e` like the same-repo job
- head is checked out with links as links, like the same-repo gate does,
because flattening them hid a CRD behind a file link from the detector
while Helm still ships it

### Limits

- A fork PR whose push run waits for a maintainer's approval has no
status until that run executes, same as `E2E Tests`.
- A fork PR on a head that a same-repo PR also carries shows the verdict
standing there, one commit status cannot hold a verdict per base.
- The sweep can publish a stale `success` if it reads an approval,
stalls, and the owner withdraws it meanwhile. The next sweep flips it
back, so the window is up to 15 minutes.
- Two copies of one first-party CRD and CRDs behind directory links are
gaps of the detector on main, same for same-repo PRs, they are in
https://github.com/cozystack/cozystack/issues/5170.

### Testing

`actionlint` is clean on both workflows. `zizmor` is clean too, but
`dangerous-triggers` and `template-injection` are disabled repo-wide in
`.github/zizmor.yml`, so that says little about a privileged workflow. I
ran the `github-script` bodies under node with a mocked API (every
conclusion, approval and shared head case) and the run steps under `bash
-e` with a stub detector exiting 0, 2 and 1. `make unit-tests` passes
with `LC_ALL=C` (a kafka bats test depends on the locale, nothing to do
with this PR).

Not exercised live: the fork path and the sweep run only from the
default branch, so their first run is after merge.

### Repo admin

Branch protection requires the old job name `Require API owner review
for sizeable API changes`. This PR renames that job, so after merge the
required context has to be changed to the `API Review Gate` status.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```
2026-11-02 19:15:18 +04:00
Aleksei Sviridkin a2cc9a3016 fix(dashboard): link a tenant row to its detail page (#4004)
## What this PR does

Fixes #3822

Tenant list showed name as plain text and the only link in a row was
Edit button, so tenant detail page with its tabs and Delete was
reachable only by opening edit form and cancelling out of it. Name cell
is a link to that page now, it appears exactly where Edit appears and
points to the same CR. Plain click switches selected tenant to the row's
parent first, same as Edit does, because detail page takes namespace
from selected tenant. Ctrl, cmd and shift click don't run that switch,
so the tab you open a tenant from stays where it was.

First commit fixes what the link depends on. Console recovers a tenant's
parent from namespace name (`tenant-a-b` sits under `tenant-a`), but
root is the one level that naming skips: tenant `acme` ordered in
`tenant-root` owns `tenant-acme`, not `tenant-root-acme`. So
`realParentNamespace` returned undefined for every top level tenant,
which is most tenants on a normal cluster, and Edit button has been
missing there for as long as it exists. Ancestor labels carry the parent
that the name does not. Root level tenant named `root-x` resolves to CR
`root-x` now, not `x`.

### Notes

I first tried to make links carry their tenant in the url (`?tenant=`),
so a detail page opened in new tab resolves on its own. That turned into
questions about tenant context itself: what to do with a tenant in a
link the user can't see, what localStorage may hold, and TenantNamespace
LIST ignoring `labelSelector`. Detail route addresses a Tenant CR by
relative name and takes namespace from selected tenant, and making that
robust is a refactor of tenant context and routing, which should start
with study and planning, not with patches on patches. So this PR only
closes the UX gap (no way from tenant list into a tenant's page), going
deeper is out of scope.

Known limit that comes with it: link has no tenant in it, so opening it
in a new tab (ctrl/cmd/middle click, open in new tab, copied url)
resolves against the tenant last selected in any tab, because selection
lives in shared localStorage. For a row directly under that tenant it
opens the right page. For a row deeper in the tree the page says
`Application <name> not found.` and there is no Delete. If that tenant
also has a direct child tenant with the same relative name, new tab
opens that other tenant's page with working Edit and Delete, and delete
confirm says only `Delete Tenant "<name>"?`. Plain click in the same tab
is not affected. Detail page's own tabs and Edit link already behave
this way, it's how detail route resolves its namespace today and fixing
it properly is the refactor above.

A row links, and offers Edit, exactly when namespace holding its Tenant
CR is one the apiserver returns to this user. It returns a tenant
namespace only if the user is subject of a RoleBinding in it, so
sub-tenant admin that cant act in `tenant-root` sees no link there.

No row level Delete button. Issue allows either that or a route to
detail page, I took the route so confirm and mutation stay in one place.

Covered by tests, every source change here fails a test when reverted:

- top level tenant, nested one, root level tenant named `root-x`, forest
root without ancestor labels
- bridged row whose real parent is invisible gets neither link nor Edit
- plain click switches to the row's parent and opens detail page
- ctrl click leaves selection and location alone

### Screenshots

Tenant list under root on a dev cluster. Every row name is a link to
tenant page now (`pr4078` is hovered), top level tenants had neither
link nor Edit before. Host column redacted.

<!-- SCREENSHOT: drag in 01-tenant-list-links.png here -->
<img width="1440" height="900" alt="01-tenant-list-links"
src="https://github.com/user-attachments/assets/39cd723a-1ac5-4d3b-8c36-b70ddc626961"
/>


### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

All changed files are under `packages/system/dashboard/images/console`,
console has no downstream copy (standalone UI repo is archived).

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(dashboard): tenant name in the tenant list links to its detail page now (where Delete is), and top level tenants get their Edit button back.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Root-level tenant names are now displayed without incorrectly removing
prefixes, and tenants ordered directly under the root are correctly
recognized in the hierarchy.
* Tenant rows link to their detail pages when the parent tenant is
accessible. Same-tab navigation updates the active tenant, while opening
a link in another tab leaves the current selection unchanged.
  * Edit links and navigation now use consistent tenant routes.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-02 16:33:51 +03:00
Myasnikov Daniil b5e3b37a9a ci(api-review-gate): build the fork detector from the default branch
The merge base is the commit the fork's author branched from, so a
branch cut from an old commit selected an old detector, and a fix to
the detector never reached the PRs an attacker controls. The workflow
itself comes from the default branch, so the detector now does too.

The head is also checked out with its links intact. Flattening them
hid a CRD behind a file link from the detector while Helm still
shipped it, and made the verdict differ from the one the same-repo
gate reaches.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-02 14:47:51 +05:00
Aleksei Sviridkin d32092ac9a feat(release): attach an SBOM to every release image (#4997)
## What this PR does

This PR attaches an SPDX SBOM to every first-party image of a release.
BuildKit adds it at build time as an attestation, so it sits in the
image's index and moves with the digest the release pins, through
promotion. `docker buildx imagetools inspect <image>@<digest> --format
'{{ json .SBOM }}'` prints it, one document per platform. Releases cut
from this change on have it. The Talos installer image (built by the
imager, pushed by skopeo) and the third-party images don't.

It is more than `SBOM: '1'` in `tags.yaml`, because three things broke
on the way:

- An image built with an attestation is an index of the image and an
attestation manifest. `image_arch` in `hack/stitch-multiarch.sh` read
that as not one amd64 and one arm64 image, so the stitch skipped every
such image and still exited 0, and `hack/verify-multiarch.sh` then
failed the release with "index without linux/arm64". `image_arch` and
the rerun check look through the index now, and the stitched index keeps
both attestations.
- The release `make build` ran on the default docker driver, which
cannot attach attestations. It runs on the docker-container builder with
the docker.io mirror now, like the arm64 leg. `use: false` stays, so a
plain `docker build` keeps the default driver.
- `--sbom=true` pulls `docker/buildkit-syft-scanner:stable-1`, a
floating tag. `SBOM=1` passes `--attest type=sbom,generator=` with the
scanner pinned by digest (1.12.0). The comment in `hack/common-envs.mk`
called the output CycloneDX, BuildKit writes SPDX, and the same word in
`docs/security/self-assessment.md` is fixed too.

Only the release lane changes. PR, main and nightly builds push to OCIR
and are not what users install, so the OCIR attestation probe the
tracker (A2 in #3278) listed is not on this path, release images go to
GHCR.

### How it was checked

- Bats 1.14.0: four new tests in `hack/stitch-multiarch_test.bats` (the
arm64 leg's workflow shape, a stitch of SBOM images, a rerun after one,
an index of two linux images that is skipped), two in
`hack/common-envs_test.bats` (the flags, and the Renovate manager held
to the `SBOM_SCANNER` line), and the matchbox workflow test rewritten
for the new builder. The stitch tests fail on `main`'s
`stitch-multiarch.sh`, and each of these mutations fails one: dropping
the `length == 1` check, counting the attestation as a platform, looking
for the amd64 index itself in the rerun, taking `SBOM`, `BUILDER` or the
mirror config out of `tags.yaml`, going back to `--sbom=true`, dropping
the scanner digest, turning `SBOM` on by default, reformatting the
`SBOM_SCANNER` line out of the Renovate manager's regexp.
- A rehearsal on a localhost registry (docker 29.8, buildx 0.37.1) with
the real scripts. Images built through the `BUILDX_ARGS` of
`common-envs.mk` for amd64 and arm64, one without SBOM beside them, then
`stitch-multiarch.sh` (2 stitched, 0 skipped), `verify-multiarch.sh` (0
failed), `imagetools inspect` reading SPDX 2.3 with 15 packages per
platform from the stitched index, and `skopeo copy --multi-arch all`
keeping the index digest. `containerimage.digest` in `--metadata-file`
equals the pushed index digest, so the digest the package Makefiles
stamp is the index. The same run with `main`'s stitch skips the SBOM
image, exits 0 and fails the gate.
- `make bats-unit-tests`: 2444 of 2445. The one failure, "all-topics
backup skips double-underscore internals and keeps single-underscore
user topics" in `hack/kafka-topic-backup-strategy.bats`, also fails on
unmodified `main` on the same machine. `make bats-posix-compat-tests`
passes, and the stitch test file passes under dash through
`cozytest.sh`. `hack/check-commit-trailers.sh` passes. zizmor and
actionlint aren't installed here, so the workflow change (two env keys
and one renamed builder step) is covered only by the tests above.
- Not checked: that GHCR takes the attestations, what retention does to
the child manifests, the fork lane's OCI export (it doesn't set `SBOM`),
and a real release build. The first alpha cut after the merge tests the
first and the last. Renovate itself did not run on the new manager: the
config passes the Renovate JSON schema, and the regexp takes the tag and
the digest off the live line, which a test holds.

### Decisions for the release owners

- Scope is `tags.yaml` only. SBOMs on main and nightly images would need
OCIR to take attestations (not probed) and would change what the nightly
mirror copies.
- The scanner is 1.12.0 with syft 1.51.0 inside, which is in the range
of GHSA-mw2c-m758-9v5q (memory exhaustion parsing binary inputs, fixed
in syft 1.52.0). It reads images our own builds produce, so I didn't
hold the change for it. A Renovate manager on the `SBOM_SCANNER` line
moves the version and the digest together when the scanner ships a newer
syft.
- The scanner adds a step per image and I didn't time it on the release
runners.
- A stitch rerun reads the amd64 index the tree pins, which stays in the
registry untagged after a stitch. If the registry drops untagged
manifests before the rerun, it fails at that read rather than repinning.
- After the first alpha with attestations I'll move A2b in #3278 and the
SBOM item in #2937 to done. The SBOM row in
`docs/security/self-assessment.md` still says SBOMs aren't published,
and stays until that alpha shows they are.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

I walked the trigger map against the diff. It touches `tags.yaml`,
`hack/common-envs.mk`, the stitch and verify scripts and three docs. No
release asset is renamed, `hack/package.mk` and the commit convention
are untouched, and the flow a maintainer follows to cut and promote a
release is the same.

### Release note

```release-note
feat(release): every first-party container image of a release carries an SPDX SBOM as a BuildKit attestation in its index, readable with `docker buildx imagetools inspect <image>@<digest> --format '{{ json .SBOM }}'`. Releases cut from this change on have it, the Talos installer image and third-party images don't.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

- **New Features**
- Supported release image builds now include SPDX-format software bills
of materials (SBOMs) as attestations.
- Multi-architecture image stitching, retagging, and mirroring preserve
SBOM attestations.
- **Documentation**
- Updated image-reference guidance to explain how to handle attestation
entries in multi-architecture indexes.
- Clarified which images include SBOMs, how to inspect them, and that
SBOM generation remains disabled by default outside configured release
builds.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-01 22:35:38 +03:00
IvanHunters 94ada11eef fix(marketplace): pin a tapped repository to a digest instead of a mutable tag (#5082)
## What this PR does

A tapped (community-connected) repository followed a mutable OCI tag:
the Flux `OCIRepository` carried only `spec.ref.tag` with a 5-minute
interval, so whoever controlled the tag controlled what ran in the
management cluster within that interval, down to the HelmReleases of
already-running tenant application instances. Fixes #4979.

A tap is now pinned to a digest. `cozypkg tap` and the connect API
accept an explicit `oci://host/org/repo@sha256:...` reference; a tag
reference is resolved and pinned to its digest by the tap materializer
once the artifact for the current spec generation is ready, from the
digest Flux reports in `status.artifact.revision`, with the tag kept
informational. A tap connected before this change is pinned to its
current digest on the first reconcile after upgrade, and a re-connect to
a new tag waits for source-controller to re-resolve before pinning, so a
stale revision is never pinned. Moving the tag upstream no longer
changes anything in the cluster; an explicit re-connect moves the pin.

Connect also accepts an optional keyless cosign verification policy
(`--certificate-identity` / `--certificate-oidc-issuer` on the CLI,
`spec.verify` on the API), anchored to an exact identity so a
superstring cannot match, and preserves it, and the pull secret, across
repeat connects rather than silently dropping them. With those flags the
CLI verifies the signature at tap time and pins the verified digest
before pulling, so the content it applies is what a trusted identity
signed; it fails closed if the verified digest cannot be determined.

### Screenshots

Not applicable: no UI changes.

### Downstream repositories

I walked the trigger map against the diff. The change is confined to
`cozypkg`, the `Tap` aggregated API (a virtual, cluster-scoped resource
with no CRD), the tap materializer, and the community-index docs. It
does not touch `packages/apps`, `packages/core/platform/values.yaml`,
installer values, platform variants/components, release assets,
`ApplicationDefinition` semantics, the `hack/` make targets, or the
commit convention, so the provider, ansible, ccp and external-apps paths
are not reached.

One path is arguable and needs a human decision, so I have left every
box unticked rather than tick or dismiss it:

- `cozystack/website` — `cozypkg tap` gains flags and digest-pinning
behaviour. The marketplace tap flow is new and I am not sure it has a
reference page on the site yet, so whether a docs follow-up is warranted
(and where) is a maintainer call.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(marketplace): a tapped repository is pinned to a digest instead of following a mutable tag, so moving the tag upstream no longer changes what runs in the cluster; connect accepts oci://...@sha256:... and an optional keyless cosign verify policy (anchored to an exact identity and preserved across re-connects), and with verify flags the CLI verifies the signature and pins the verified digest at tap time.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

- **New Features**
- OCI repositories can be connected using a SHA-256 digest, with or
without a tag. Tag-based sources are pinned to the resolved artifact
digest.
- Optional Cosign keyless verification is available during connection
using certificate identity and OIDC issuer details.
- Reconnecting preserves existing verification and pull-credential
settings when replacement values aren’t provided.
- **Bug Fixes**
- Reconnecting with a new tag clears previous operator-managed pins so
the requested tag can resolve to its current artifact.
- **Documentation**
- Clarified tag pinning, signature verification, and where tapped app
charts run.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-30 18:09:34 +03:00
IvanHunters ccc216652e fix(monitoring-agents)!: isolate merged log payload under log_processed (#5167)
## What this PR does

Fluent Bit's `kubernetes` filter runs with `Merge_Log On` but without
`Merge_Log_Key`, so a parsed JSON log line is merged into the record
root.
An application that logs JSON can thereby set top-level fields that
later
pipeline stages trust for routing and tenant isolation:
`kubernetes_namespace_name`
(rewrite_tag routing and the `_stream_fields` stream identity),
`kubernetes.labels`, and the `tenant` / `cluster` / `log_source`
markers.
A tenant workload can forge another tenant's values on its own log
lines.

Setting `Merge_Log_Key log_processed` nests the parsed payload under a
dedicated key, so application log content can no longer override the
top-level service fields.

The change is applied to both Fluent Bit configurations that carry this
filter: the platform agent
(`packages/system/monitoring-agents/values.yaml`)
and the guest-cluster agent (`packages/apps/kubernetes`).

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff,
file by file. The change touches only the Fluent Bit filter config
inside
`monitoring-agents`; it does not alter package layout, any
`values.schema.json`,
platform values keys, `ApplicationDefinition`, any CRD, `hack/`,
namespaces,
telemetry metric names, or the commit convention. None of the listed
couplings match.

- [x] No downstream repository is affected by this change

### Release note

```release-note
fix(monitoring-agents)!: Fluent Bit now nests parsed JSON log content under a `log_processed` key instead of merging it into the record root. This prevents an application log line from forging the service fields used for log routing and tenant isolation. BREAKING: top-level fields previously extracted from JSON logs (for example `level`, `msg`, and any application-specific keys) now live under `log_processed.*`; external dashboards and LogsQL queries that read them from the record root must be updated.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Improvements**
* Merged Kubernetes log content is now grouped under `log_processed` in
Fluent Bit output.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-30 17:14:45 +03:00
Myasnikov Daniil 7f55ff5d85 fix(dashboard): link a tenant row to its detail page
The tenant list rendered the name as plain text and put the row's only
link on an Edit button, so nothing in the list reached the detail page
that every other kind gets, tabs and a Delete action included. The only
way there was to open the edit form and cancel out of it.

The name cell becomes a link wherever Edit is offered, to the same CR.
The detail page reads the namespace from the tenant context, so a plain
click aligns the context to the row's parent first, the way Edit
already does. React runs that handler for a ctrl-, cmd- or shift-click
too, which the browser turns into a new tab or window, so it runs only
for the click that navigates its own tab; otherwise opening a tenant
beside the current one would move the tab left behind as well.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-30 16:38:53 +05:00
Myasnikov Daniil e44e41d307 fix(dashboard): resolve the Tenant CR of a tenant ordered in the root
A tenant's CR lives in its parent's namespace, and the console recovered
the parent from the namespace name: `tenant-a-b` sits under `tenant-a`.
The root is the one level that naming skips. A tenant `acme` ordered in
`tenant-root` owns `tenant-acme`, not `tenant-root-acme`, so the prefix
test never matched a top-level tenant and `realParentNamespace` returned
undefined for it. That is most tenants on a normal cluster, and the Edit
button, offered only where the CR's namespace resolves, was missing for
all of them.

The ancestor labels carry the parent the name does not, so the lookup
falls back to `tenant-root` when the chain names it. Deriving the CR name
keys on the parent being the root rather than on the child's name
happening to start with it, which keeps a root-level tenant called
`root-x` from resolving to the CR `x`, and the row label agrees with it
rather than showing `x`.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-30 16:23:10 +05:00
Myasnikov Daniil dfa2283a13 ci(api-review-gate): close a standing success on a shared fork head
A commit status belongs to the commit. When two open fork PRs carry the
same head, a success already standing on it may have been earned by one
of them on another base, or may rest on an approval that was withdrawn
on a PR whose run waits for a maintainer and so never reaches resolve.
The sweep skipped such a head and left that success in place, which let
a duplicate fork PR keep a withdrawn approval from taking effect.

The sweep now writes the same failure resolve writes, once per head,
when a success set by a workflow stands on a head that only fork PRs
carry. It still never creates a status, and a head that an open
same-repo PR carries stays with api-gate-report.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-30 16:09:00 +05:00
Myasnikov Daniil 0538e70591 ci(api-review-gate): let a sizeable fork change reach the approval check
api-gate exits 2 to report a sizeable change, and the runner runs a step
without an explicit shell as bash -e. The fork detect step switched
errexit off only after reading the exit code, so the first failing
command ended the step with 2 before sizeable=true was recorded. Every
fork PR with a sizeable API change then got "gate evaluation did not
complete", which the sweep deliberately ignores, so an owner's approval
could not turn it green by either path. The same-repo step already
switches errexit off before the call.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-30 16:09:00 +05:00
Myasnikov Daniil 878f569e61 ci(api-review-gate): leave a shared head to its same-repo publisher
A commit status belongs to the commit, so two open PRs that carry the
same head share one "API Review Gate" status. The fork workflow
attributed that status to a fork PR by head SHA alone. A fork PR opened
at a commit of this repository could therefore overwrite the verdict of
the same-repo PR that owns it: resolve wrote a failure after
api-gate-report had posted success, and the sweep applied the fork PR's
reviews to a status the other PR had earned. A fork can reference any
commit of the repository's network, so anyone able to open a PR could
hold another PR's gate red.

The fork workflow now leaves a head that an open same-repo PR carries to
api-gate-report, and the sweep skips any head carried by more than one
open PR. Two fork PRs at one head still fail closed in resolve, since
their bases, and so their verdicts, can differ.

The header of the same-repo workflow said skipped and cancelled runs
post nothing. That stopped being true once a review posts pending ahead
of its evaluation, so it now says they post no verdict.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-30 16:09:00 +05:00
Myasnikov Daniil 4c37ef3a61 ci(api-review-gate): let owner approval reach fork PRs
On a fork PR an API owner's approval could never turn the status green.
For a pull_request_review run on a fork PR, workflow_run.head_repository
names the base repository and pull_requests is empty, so the fork
publisher took every review-triggered run for a same-repo one and
skipped it, while the same-repo reporter skipped it for being a fork. A
sizeable fork PR stayed on the failure its push run had posted.

Fixing the guard alone would leave most forks stuck. The repository
makes every run from an external contributor wait for a maintainer's
approval, and a run in action_required never fires workflow_run, so
their review-triggered runs never reach the publisher. And a fork's run
executes the fork's own copy of the gate workflow, so the conclusion the
publisher relayed was whatever the fork made it.

The privileged workflow therefore reaches the verdict itself, from
inputs a fork cannot forge, and a scheduled sweep covers the runs that
never execute. The sweep only moves a status the full evaluation already
set after finding a sizeable change, so it cannot create a pass.

The same-repo gate now posts pending as soon as an actionable review
arrives: a dismissal can turn a green head red, and the old success
would stay mergeable for the minute the evaluation takes. Both paths
also ignore COMMENTED reviews when taking each reviewer's latest, since
an owner who approved and then commented had withdrawn their own
approval as far as the gate was concerned, which GitHub does not do.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-30 16:09:00 +05:00
Myasnikov Daniil 939496ad78 docs(e2e): record the gate-status pattern's second consumer
The "E2E Tests" commit-status pattern in §10 now has a second consumer
in api-review-gate.yaml, and the two GitHub behaviours that make it
load-bearing were not written down anywhere.

Check runs are not deduplicated per name across workflow runs — each run
gets its own check suite, so filter=latest returns the same set as
filter=all and the aggregate is worst-of, meaning a cancelled run
poisons a context until the next push — while commit statuses are
deduplicated per context with the newest winning. And the fork
read-only token restriction covers pull_request_review as well as
pull_request, so an approval on a fork PR cannot publish a status
either, which is why both gates need a privileged workflow_run half
rather than only handling the push path.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-30 16:09:00 +05:00