Commit Graph
7800 Commits
Author SHA1 Message Date
Alexey Artamonov 0f336e99c5 test(backups): drop the comment restating what accepted() returns
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-09 16:39:47 +03:00
Alexey Artamonov 1291e616c7 fix(clickhouse): give every backup run an archive of its own
An archive was named from the application and the BackupJob's start
time, which has one-second resolution. Two runs of one application
started in the same second, as two Plans on the same schedule are,
wrote the same archive. The strategy takes a finished upload of its
archive as an earlier attempt of its own run, so the later run could
succeed on the other run's upload and record an archive missing any
table created since.

The BackupJob's UID now ends the name. It goes after the timestamp so
names keep sorting chronologically, and the restore of a Backup that
records no archive, which accepts only a bare timestamp after the
release prefix, never picks one of these.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-09 16:39:47 +03:00
Alexey Artamonov d530ca8cf5 fix(backups): keep a ClickHouse archive an earlier attempt finished
A backup Pod that loses sight of its upload, by dying or by losing six
polls in a row, is retried while the sidecar finishes that upload. The
retry found it in the action log as a success, took the sidecar for
idle and asked for create_remote with resume again. clickhouse-backup
then rebuilds the local backup and uploads it once more over the
archive already in the bucket, which is left broken if that second
upload dies halfway.

The archive name belongs to one BackupJob, so a success row of exactly
this upload can only be an earlier attempt's, and the retry now takes
it as its result. An error or cancel row still leads to a resumed
upload, and a restore still runs again, since a restore is not named
per RestoreJob.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-09 16:39:46 +03:00
Alexey Artamonov 89f8cc02ee Merge branch 'main' into fix/clickhouse-idle-before-backup
main now names the ClickHouse archive in the controller and restores
the archive a Backup recorded, in the same strategy script this branch
gates on an idle sidecar. The merged script keeps both: the archive
name and its restore selection from main, the idle gate, retried polls
and lost-operation check from this branch.

A retried backup is taken over through the same idle gate as a restore,
matched on the whole command line the sidecar logs. With resume
requested that line reads "create_remote --resume <name>", so the gate
matches that form; matching on the name alone would also take another
operation's row ending in the same name for this one.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-09 16:39:43 +03:00
Alexey Artamonov ae1e1830ce test(backups): cover a ClickHouse restore against an unreachable sidecar
The restore path resolves its archive from list/remote before the start
gate runs, so it carries its own wait for a sidecar that is not up, and
only the backup path had a test for a sidecar that never answers. The
retry comment also miscounted: the limit is attempts, not retries.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-06 11:31:41 +03:00
Alexey Artamonov 993b79bb37 fix(backups): take over a running ClickHouse restore on Job retry
restore_remote drops every table it restores before recreating it, so
running it twice discards whatever was written after the first run
finished. The strategy Job retries a failed Pod up to twice, and with
the start gate waiting for an idle sidecar instead of hitting 423, a
retry whose predecessor died after its POST waited for that restore to
finish and then started a second one. A retry now takes over a
restore_remote of the same archive that is still running.

A finished row is not taken over: the sidecar never trims its action
log, so a success row of the archive may belong to an earlier
RestoreJob, and treating it as this one would report a restore that
never ran. A retry that arrives after the restore finished therefore
still runs it again.

Once the operation is accepted, one failed poll no longer ends the
attempt under errexit; six in a row do. An operation that ends
cancelled now fails the Pod instead of being polled forever.

A retried backup still starts a new create_remote, because its archive
name changes on every attempt; taking it over needs a name fixed for
the whole BackupJob.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-06 11:23:28 +03:00
myasnikovdaniil 970191dc8a fix(clickhouse): restore the archive a Backup was taken as (#5208)
## What this PR does

A ClickHouse `RestoreJob` restored the newest archive under the app
prefix, whichever `Backup` it named. Strategy pod generated the archive
name with `date -u` and nothing was recorded on the `Backup`, so restore
could only list the remote archives and take the last one. Restoring an
older `Backup` brought back the latest data.

Now the controller names the archive before it creates the backup Job
(`clickhouse-<app>-<startedAt>` from persisted `status.startedAt`, so
retried pods write the same one), passes it to the pod as `.ArchiveName`
and records it in the `Backup` `driverMetadata`. Restore takes exactly
that archive and fails with its name when storage has no such archive.
The name is recorded only when the Job's pod got it, so a Job left from
the previous controller during an upgrade, or a custom strategy with its
own names, leaves no name instead of a wrong one.

A retried pod reuses the name, so it waits for an attempt the sidecar
still runs (no 423) and asks for `resume`.

A `Backup` outlives its `BackupJob` and a Plan names `BackupJob`s after
the schedule slot, so a `BackupJob` created again under a used name met
the earlier run's `Backup` when its Job completed, took it as its result
and left the archive it just wrote recorded nowhere. Now it fails if
that `Backup` records another archive, and the message names both. A
retry of the same run still gets its own `Backup` back.

Old `Backup`s have no recorded name and stay restorable: the pod logs a
`WARNING` and takes the newest archive stamped no later than `takenAt`,
and fails when nothing qualifies. While the chart rolls out, previous
controller pod can still render the new template and pass no `takenAt`,
so a restore of a `Backup` without recorded name fails before it touches
any table, and works when created again after the rollout.

Fixes https://github.com/cozystack/cozystack/issues/5199

Tests are go unit tests for naming, recording, a reused `BackupJob` name
and restore render, and `hack/altinity-backup-strategy.bats`, which runs
the shipped script against a stubbed clickhouse-backup API (the restore
tests go red on the old template). Checked on a live ClickHouse with two
backups of 1500 and 2000 rows: restore of the older one by recorded name
ends with 1500 rows, fallback for an unnamed older backup picks the same
archive with the warning, a missing archive fails without touching the
table, a retry over an existing local backup completes under the same
name. The controller binary itself was not run on the cluster, only its
rendered output in the pod.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(clickhouse): a RestoreJob now restores the archive its Backup was taken as, not the newest one. New Backups record the archive name, Backups taken before this release are restored best effort from the newest archive stamped no later than their takenAt, with a warning in the restore pod log. A ClickHouse BackupJob created again under a name whose Backup still exists now fails instead of reporting that Backup as its result.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Backups now use a consistent archive name tied to the backup job, and
completed backups retain that name for future restores.
* Restores can select the recorded archive and backup timestamp. If no
archive was recorded, the system selects the newest eligible archive for
the application.
* **Bug Fixes**
* A restore now fails when its recorded archive is missing instead of
silently selecting a different one.
* Backup requests avoid starting a duplicate action when the matching
action is already in progress.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-06 11:54:38 +05:00
myasnikovdaniil 18779dddfc fix(backups): report sizeBytes for MariaDB and CNPG backup artifacts (#5223)
## What this PR does

Fixes https://github.com/cozystack/cozystack/issues/4316.

Continues https://github.com/cozystack/cozystack/pull/5015 by androndo
on a separate branch. All ten of their commits are kept as they were
(authorship, messages and order, only rebased on current `main`).
MariaDB and CNPG backups stored in S3 report
`Backup.status.artifact.sizeBytes`, the size of the dump
mariadb-operator uploaded or the total of the barman base backup, and
the field stays unset when the size cannot be attributed to that one
backup.

Four commits are added on top:

- `test(backups)`: operator Backup fixtures carry the condition
mariadb-operator writes on success, `Complete=True` with reason
`JobComplete` and message `Success`. Operator sets `Complete=True` for a
failed Job too (reason `JobFailed`), so a fixture without reason
describes a state that never happens.
https://github.com/cozystack/cozystack/pull/5207 makes the driver check
that reason, the tests pass with and without it.
- `refactor(backups)`: the MariaDB matcher returns the matched dump
(key, file name, timestamp from the name, size, LastModified) and tells
"not found" from "ambiguous", and the Secret lookup and S3 client live
on a small value any reconciler can build from its own client. A MariaDB
restore needs this to restore the dump of the Backup it was given and
not the newest one under the prefix. `sizeBytes` does not change here.
- `fix(backups)`: the probe reads bucket, prefix, endpoint, credentials
and TLS from the operator Backup's own `spec.storage.s3`, not from the
strategy as it renders now, so editing the strategy during a backup does
not send the probe somewhere else.
- `fix(backups)`: a dump is attributed only when its ownership is
proven. Before, a backup pod with a clock more than a minute behind,
right after a previous run, got that run's dump size (666 bytes for a
123-byte dump in a test). Now the dump has to be the only one that
either clock places in the operator Backup's lifetime, both clocks have
to agree, and neither may place it in the lifetime of another finished
operator Backup with the same bucket and prefix. Otherwise the size
stays unset. Two cases are still not covered. First, this run's own dump
is missing from the listing while the operator Backup of a neighbour was
deleted or has not finished, then the neighbour's dump can be taken.
Second, backup pod clock and object store clock are both off the same
way by more than the minute of slack, which moves a finished neighbour's
dump into this run's window, and nothing the clocks record tells them
apart. A run whose Job failed has no dump of its own and reaches the
probe only because the driver takes `Complete=True` for success,
https://github.com/cozystack/cozystack/pull/5207 fixes that.

Of the open threads on the original PR, the cache reads, the dump file
name layout and the default AWS endpoint test were already handled by
its later commits, the clock skew one is the last commit here and the
failed Job case is https://github.com/cozystack/cozystack/pull/5207.

`minio-go` is new to this tree, this PR adds it as a direct dependency
and `internal/backupcontroller/artifact_size.go` is its only importer.
The open Bucket strategy PR
https://github.com/cozystack/cozystack/pull/4235 picks the same client
at the same version.

Each clock skew case, and a run whose own dump is gone while the
neighbour's operator Backup still exists and has finished, have a unit
test that fails on the old matcher, and every new guard has a mutant
that turns a test red. I also ran the matcher against a real bucket
after four MariaDB backups in a row. Runs that finished more than a
minute apart mapped to the dump their operator Job logged, the two back
to back came back unset instead of taking each other's dump, and with a
run's own dump hidden from the listing the matcher refused the
neighbour's dump that the old one returned.

### Screenshots

N/A, no UI changes.

### Downstream repositories

The only API change is the description of the existing
`Backup.status.artifact.sizeBytes` field, no new field, value, default
or enum, no package, values or tooling change, and neither
cozystack/website nor cozystack/terraform-provider-cozystack references
`sizeBytes`.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(backups): MariaDB and CNPG (Postgres) backups stored in S3 now report `status.artifact.sizeBytes`: the size of the dump mariadb-operator uploaded, or the total size of the barman base backup (WAL archive excluded). A MariaDB backup takes the size of the one dump that falls inside its run on both the backup pod's and the object store's clock and inside no other finished run, and the size is left unset rather than set to 0 whenever it cannot be measured or attributed. Velero-based backups intentionally keep it unset.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Backup artifacts now report the stored size attributable to an
individual MariaDB or CNPG backup when it can be determined. Shared or
ambiguous storage may leave the size unset.
* **Bug Fixes**
* Improved handling of backup artifact creation and lookup errors,
including retrying when an existing backup cannot be read and failing
jobs when artifact creation is rejected.
* **Documentation**
* Clarified that an unset size means unknown; zero does not indicate an
unknown size.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-06 11:54:28 +05:00
Alexey Artamonov d0a73302d9 fix(backups): fail a ClickHouse backup whose operation the sidecar lost
The sidecar keeps its action log only in memory, so a sidecar restart
after create_remote was accepted drops the operation's row, and the
strategy kept polling for it forever under a Job with no deadline.
Until now the resumed upload re-added a row ending in the archive name
and ended the loop, but resume is now off so a failed attempt never
reaches the bucket, which leaves nothing to end it. The row is logged
before the POST is acknowledged, so its absence means the operation is
gone and the attempt has to fail.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-05 21:11:59 +03:00
myasnikovdaniil e5bb3cda07 fix(kubernetes): move the OIDC users out of the bootstrap Job's script (#5213)
## What this PR does

Found while checking for other places with the failure from
https://github.com/cozystack/cozystack/issues/4968: data that grows with
configuration is passed to a process as one argument, and Linux allows
128 KiB for that. This one is in the `kubernetes` app and there is no
issue about it yet.

The OIDC bootstrap Job wrote a ClusterRoleBinding for every `oidc.users`
entry into its `sh -c` script, 526 bytes per user. I rendered current
main with `helm template` and users like `user123@example.org`: 238
users give a 130,889 bytes script that starts, 239 give 131,415 bytes
and `exec` fails with E2BIG. Then the container can't start, the
post-install and post-upgrade hook fails after its retries and the
HelmRelease of the tenant cluster stays unready. Schema has no limit for
`oidc.users` and each person needs own entry, so a cluster shared by 240
people gets there.

Now the bindings and the list of their names are rendered by the new
`oidc-rbac-bindings.yaml` into a ConfigMap `<release>-oidc-bindings`
that the Job mounts, so the script is 6,380 bytes for any number of
users. The ConfigMap has its own 1 MiB limit, about 2,500 users at 417
bytes each. Names, labels and roles of the bindings stay the same. I
compared the render of old and new chart for eight emails (ones YAML
reads as a boolean or null, a number, a quote, an apostrophe, unicode
and two plain) and the bindings are identical, so existing ones are
updated in place and nothing is orphaned.

The prune step reads the same ConfigMap. If the list is not mounted the
Job stops before it deletes anything, because `grep` on a missing file
exits 2, `! grep` is true and every binding of the release would be
pruned. A release with no users applies nothing (`kubectl apply` fails
on an empty file) and prunes all stale bindings, same as before.

`hack/kubernetes-oidc-bindings.bats` renders 300 users, checks the
script is under the limit without any email in it and that every user
has one binding with the old name and role. Three more tests run the
rendered script against a fake `kubectl` and check what it applies and
deletes. The prune condition forced to true or to false passes all
helm-unittest cases, so this part is tested in bats. The bug is in
release-1.6 and release-1.7 as well.

### Downstream repositories

I walked the trigger map against the diff (one chart template, its tests
and a bats file): no package added or removed, no values, schema or
default changed, no output Secret or Service renamed.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(kubernetes): the OIDC bootstrap Job no longer fails to start for a tenant cluster with about 240 or more oidc.users, the bindings now come from a ConfigMap the Job mounts.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Kubernetes OIDC configuration now supports per-user role bindings,
with administrator and viewer access assigned according to each user’s
role.
* OIDC role bindings are kept in sync with the configured user list,
including removal of obsolete bindings.
* Deployments with no OIDC users or with OIDC mode disabled continue to
avoid creating user bindings.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-05 19:37:16 +05:00
myasnikovdaniil ab13f9f72c fix(cilium): pass the CiliumNode list to jq on stdin in the IPAM hook (#5212)
## What this PR does

Fixes https://github.com/cozystack/cozystack/issues/4968. The cilium
IPAM migration hook passed the whole `kubectl get ciliumnodes -o json`
to `jq` as one argument, and Linux allows 128 KiB for a single argument.
Cilium copies Node labels onto every CiliumNode (22 to 30 KiB per node
with Talos, NFD or GPU labels), so the list is over the limit within six
nodes. `jq` can't start (`Argument list too long`), the hook Job fails
before it moves any node, `cilium` HelmRelease loops on `UpgradeFailed`
and everything that depends on networking stays in `DependencyNotReady`.

The hook is rendered on every upgrade of cilium in `cluster-pool` mode,
so a cluster that already finished the migration hits the same on its
next values change. The failing step only logs the pods that still hold
an old Ingress IP, but it runs before the node loop, so on a cluster
that is not migrated yet it blocks the migration itself.

Now the list goes to `jq` on stdin together with the IP objects, same as
the other `jq` calls in the script get their input.

The new test in `hack/cilium-ipam-migration.bats` makes six CiliumNodes
over the limit (one 24 KiB annotation each instead of labels, and it
checks the list is over 128 KiB even printed compact). It fails on the
old script with status 126 and passes now. It also checks that each pod
is named with the node whose Ingress IP it holds and that a pod on an
unrelated address is not named, because a join that always names the
first node, or reports every IP object, passed the old tests.

I did not take two suggestions from the issue. Skipping the hook when
every CiliumNode is in the pool changes what one existing test pins (the
hook adds the pool to `excludeIps` in that case) and doesn't help
clusters that are not migrated yet. A value to disable the hook leaves
the default broken. `cozystack-operator` rewriting `disableHooks` and
helm-controller rolling the DaemonSet on every retry are other
components. The bug is in release-1.6 and release-1.7 as well.

### Downstream repositories

I walked the trigger map against the diff (one hook script and its bats
test): no package, values, schema, default or output name changed.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(cilium): the IPAM migration hook no longer fails with "Argument list too long" on clusters with more than a few nodes, which blocked the cilium upgrade and every package that depends on networking.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved IP address migration handling for environments with large
node lists, avoiding failures caused by operating system argument-size
limits.
* Migration continues to identify affected pods by node and distinguish
unrelated pods.
* **Tests**
* Added coverage for migrations involving large node lists, multiple IP
objects, and unrelated pods.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-05 19:37:05 +05:00
Myasnikov Daniil d7ae34dfc3 fix(backups): fail a ClickHouse BackupJob whose Backup name is taken
A Backup outlives the BackupJob that made it, so a BackupJob created
again under a name already used meets the earlier run's Backup when its
own Job completes. That happens by hand, and from a Plan reconciled
again inside its starting deadline, since a Plan names each BackupJob
after its schedule slot. The Altinity driver took the earlier Backup as
its result: the BackupJob turned Succeeded with a BackupRef recording
the earlier run's archive, the archive this run wrote was recorded on
nothing, and restoring what the BackupJob reported brought back the
older data.

An existing Backup is now taken as this run's only when it records this
run's archive, which is what an earlier reconcile of the same run
leaves when its status write is lost. Any other fails the BackupJob,
naming both archives, and leaves the earlier Backup untouched.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-05 07:56:44 +00:00
Myasnikov Daniil d5559bb934 test(backups): remove the Altinity strategy tests' scratch directories
Every test in hack/altinity-backup-strategy.bats made a directory with
mktemp -d and never removed it, so each passing run of the file left
nine directories in TMPDIR, and they pile up on any machine that keeps
it between runs.

Each test now removes its directory as its last statement. A failed
assertion aborts the body before that line, so a failing test keeps
its stub log and rendered script for inspection.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-05 07:56:39 +00:00
Aleksei Sviridkin 3d317c5f67 feat(api): expose HelmRelease failure state on Application status (#5171)
## What this PR does

Tenants usually read Applications but not the HelmReleases behind them,
and the Application status only carried the `Ready` and `Released`
conditions. A client could see that a release was failing, but not how
many times it had failed or whether helm-controller had stopped retrying
it.

This PR copies the `Stalled` condition onto the Application and adds
three optional status fields taken from the HelmRelease:
`installFailures`, `upgradeFailures` and `lastAttemptedReleaseAction`.
All of them are additive and `omitempty`, so existing clients are
unaffected.

What the new fields do and do not tell a client:

- With the `RetryOnFailure` policy the platform generates today, a
failed release is retried rather than stalled, so the counters are the
signal. That does not hold for every state: a release missing from Helm
storage while an install failure count is retained still stalls (the
case in #3554), a release on a remediation policy stalls once its
retries run out, and terminal errors such as an invalid CEL expression
or denied access stall too.
- `Stalled` is copied only when it was observed for the HelmRelease's
current generation. The Application carries no generation, so after a
spec update a client would otherwise read the previous spec's failure as
terminal for the new values; until helm-controller reconciles the new
spec, the Application shows no `Stalled`.
- The counters only count attempts that recorded a Helm release. A
failure before Helm records one, such as a chart that fails to render
(for example a `fail` guard rejecting the values), shows only as
`Ready=False` and leaves them at zero; a failure to apply the rendered
resources is counted.
- helm-controller resets the counters when the desired state changes
(generation, chart version, values), and on success only under
`RetryOnFailure`, so a `Ready` release can still carry non-zero
counters. The field descriptions say so.

`TenantModule` is left unchanged on purpose: it still copies only
`Ready` and `Released`.

Related: #5109, #4155

Tests cover the copied condition and counters, a `Stalled` left over
from an older generation after a spec update, the counters kept on a
`Ready` release, and that `Reconciling` is still not copied. The OpenAPI
schema and apply configurations are regenerated, and every per-kind
status schema picks up the new fields.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The Terraform provider reads only the `Ready` condition by type, so the
extra condition and fields do not change its behaviour. The website only
shows an example Application output, not a description of the status
contract.

### Release note

```release-note
feat(api): Applications now expose the HelmRelease `Stalled` condition and the `installFailures`, `upgradeFailures` and `lastAttemptedReleaseAction` status fields, so tenants without access to HelmReleases can see how a release is failing. Tools that compute health with kstatus, such as Flux Kustomizations waiting on Applications, now report an Application whose release has stalled as failed.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Application status now displays install and upgrade failure counts,
along with the most recently attempted release action. Failure counts
reflect attempts recorded as releases and reset when the desired state
changes.
* Current stalled release conditions are now reflected in application
status, helping clarify when a release is blocked.
* Failure details remain available when a release is ready, while
reconciling conditions are not shown.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-04 19:48:48 +03:00
Alexey Artamonov 3a10f956cc test(backups): cover a restore that starts on an idle ClickHouse sidecar
The strategy suite only exercised restore against a busy sidecar, so a
gate that wrongly blocked restore on an idle one would have stayed
green.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-04 19:30:54 +03:00
Alexey Artamonov a25db5aeb6 fix(backups): wait for an idle ClickHouse backup sidecar before starting
The sidecar refuses a new operation with 423 while another one is in
progress, but the strategy only waited for the API to answer, and went
on to start its operation even when it never did. An operation already
running in the sidecar therefore failed the attempt within seconds.

Poll the action log until nothing is in progress, and fail the Pod
instead of proceeding when the sidecar stays busy or unreachable, so
the failure names the cause rather than surfacing as a 423 or a
connection error from the operation itself.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-04 19:26:23 +03:00
Alexey Artamonov 4f3f1a5cad fix(clickhouse): stop the backup sidecar from publishing failed attempts
create_remote keeps its local backup when the upload fails, and with
the upstream defaults the sidecar never prunes local backups and
resumes every unfinished upload when it starts. A strategy attempt that
fails mid-upload therefore lands in the bucket on the next sidecar
restart as an archive no Backup refers to, and restore can pick it as
the newest one. The resumed upload also holds the API lock, so the
strategy Pods racing it get 423 and burn the Job's retries.

Turning the resume off keeps a failed attempt out of the bucket, and a
negative local retention leaves at most the newest local backup and
drops it once its upload succeeds, so leftovers no longer accumulate on
the data volume.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-04 19:26:23 +03:00
Aleksei Sviridkin c1ba05574f feat(monitoring-agents): keep application identity in container log streams (#5217)
## What this PR does

Makes Fluent Bit keep the pod lineage labels
`apps.cozystack.io/application.{group,kind,name}` on container log
lines, as the stream fields
`apps_cozystack_io_application_{group,kind,name}`, for both the default
container-log output and the per-tenant routed output.

Why: the per-application logs scope in the design proposal [Tenant
application logs](https://github.com/cozystack/community/pull/95)
(section 6) filters by these stream fields. Today the kubernetes filter
fetches pod labels, but they are dropped together with the rest of the
`kubernetes_*` keys before output.

What changes in the pipeline:

- A second `nest` lift of the pod labels, then a rename of the three
lineage keys to their final names before the existing `Remove_wildcard
kubernetes_` runs. All other labels are still removed. Every lifted key
is prefixed, so no other label can collide into the identity names.
- The three keys are added to `_stream_fields` of both container-log
outputs. Events and audit pipelines are unchanged.
- A parsed JSON log line is kept under `log_processed` (already on
`main`), so a line's content cannot create the identity keys.
- Guest Kubernetes clusters are not affected: their monitoring-agents
release replaces the filters and outputs wholesale. A regression test in
`packages/apps/kubernetes` pins that guest streams carry no identity
fields.

Effect on existing streams: VictoriaLogs treats a missing stream field
like an empty one, and the rename writes nothing when the label is
absent. Pods without lineage labels keep exactly the same stream. Pods
with lineage labels get a new stream once after the rollout; their
earlier lines stay reachable by namespace.

#### How it was tested

- helm-unittest for `monitoring-agents` (lift order, rename before
`Remove_wildcard`, stream fields on both outputs, events and audit
untouched) and for `kubernetes` (guest streams unchanged). The
events/audit guard was mutation-checked.
- Verified on a test cluster: real pod log lines arrive, Redis pods'
lines carry the three `apps_cozystack_io_application_*` stream fields,
and Fluent Bit logs show no errors.

#### Known limitation

Guest clusters write to the parent log store without authentication, so
a guest administrator can forge these stream fields as well as the
namespace field, within the store their tenant shares. This is the
write-path trust risk already documented in the proposal.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Downstream: no trigger matched. Candidate: none affected, because the
diff changes `packages/system/monitoring-agents/values.yaml` (a system
package with no schema) and adds tests; the file under
`packages/apps/kubernetes` is a test only, so the app's `values.yaml`
and `values.schema.json` are unchanged.

### Release note

```release-note
feat(monitoring-agents): container log lines of pods that belong to a Cozystack application now carry `apps_cozystack_io_application_{group,kind,name}` stream fields in VictoriaLogs. Streams of pods without application labels are unchanged; streams of application pods start a new stream once after the upgrade.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Container-log streams now include application group, kind, and name
metadata for root and tenant stores.
* Guest monitoring streams include pod name, container name, and
namespace fields.
* **Bug Fixes**
* Application metadata is added to container-log streams without
changing the existing stream fields for event and audit outputs.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-04 18:27:20 +03:00
Aleksei Sviridkin e2283b3758 feat(monitoring-agents): export PVC labels from kube-state-metrics (#5215)
## What this PR does

Adds `persistentvolumeclaims=[*]` to the kube-state-metrics
`--metric-labels-allowlist`, next to the existing `pods=[*]` and
`deployments=[*]`. kube-state-metrics then publishes
`kube_persistentvolumeclaim_labels`, one series per PVC, with the PVC's
labels as `label_*`.

Why: the per-application metrics scope in the design proposal [Tenant
resource-consumption
metrics](https://github.com/cozystack/community/pull/94) (Phase 0) finds
an application's volumes through the `apps.cozystack.io/application.*`
lineage labels on its PVCs. Without this series, volume usage
(`kubelet_volume_stats_*`, which carries `persistentvolumeclaim` but not
`pod`) cannot be attributed to an application.

`[*]` rather than an explicit key list: kube-state-metrics has no prefix
matching, so the three lineage keys would have to be enumerated, and the
series count is the same either way (one per PVC). `[*]` stays
consistent with the existing pods and deployments entries and does not
silently miss a lineage key added later. PVCs are far fewer than pods,
which already export all labels.

#### How it was tested

- helm-unittest for `monitoring-agents`: a new suite asserts the exact
flag in the rendered kube-state-metrics Deployment.
- Verified on a test cluster: `kube_persistentvolumeclaim_labels` is
present and carries `label_apps_cozystack_io_application_name`.
- Not verified: the memory impact on kube-state-metrics.

#### Known limitation

PVCs created from a StatefulSet's `volumeClaimTemplates` usually have no
owner reference and no Flux label, so the lineage webhook gives them no
lineage labels and this series cannot attribute them to an application.
Fixing that belongs to the lineage webhook.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Downstream: no trigger matched. Candidate: none affected, because the
diff touches only `packages/system/monitoring-agents/values.yaml` and a
test in that system package; it adds a series but renames no telemetry
metric, platform value, app field or CRD.

### Release note

```release-note
feat(monitoring-agents): kube-state-metrics now exports PVC labels as `kube_persistentvolumeclaim_labels`, one series per PVC, so volume usage can be attributed to the application that owns it.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Kubernetes monitoring metrics now include labels from pods,
deployments, and persistent volume claims. This expands the resource
metadata available alongside metrics, including lineage labels used by
the tenant metrics API. Metrics for these resource labels are emitted
only for resources covered by the configured label allowlist.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-04 18:22:09 +03:00
Myasnikov Daniil 59f0013b1f fix(backups): attribute a MariaDB dump only when no other run can own it
Every backup of a MariaDB application writes its dump under one shared
prefix, so the probe took the dump whose file-name timestamp (the
backup pod's clock) and LastModified (the object store's clock) both
fall in the operator Backup's lifetime, with a minute of slack for
skew. That keeps a neighbour's dump out only while this run's own dump
is in the window. With the pod clock more than the slack behind, the
own dump is named before the window opens, and a previous run that
finished within the minute before this one was created becomes the
single match: a 123-byte dump was reported as 666 bytes. A pod clock
ahead or an object store clock behind does the same with the next run,
and an own dump that is gone by the time it is looked up, as it can be
for a restore long after the backup, leaves a neighbour as the single
match with no skew at all.

A dump is now attributed only when ownership is proven:

- it is the one dump that either clock places in the window. A dump
  that one clock places there and the other does not may be this run's
  own with a skewed clock, so it leaves any other match unproven, the
  way a second dump in the window already did, and is not taken itself;
- both of its clocks place it in the window;
- neither clock places it in the window of another operator Backup
  that writes under the same bucket and prefix and has completed. This
  rejects a neighbour's dump when this run's own is missing, which the
  clocks alone cannot tell.

Operator Backups still running are not counted as neighbours: their
window has no end, and one stuck short of Complete (a Job whose pod
never starts stays JobRunning for good) would otherwise keep every
later run of the application from being sized. A dump such a run
writes inside this run's window still competes with this run's own.

Narrower slack does not prove ownership: a neighbour that finished
within the narrower slack still matches, and a smaller skew pushes the
own dump out. The Job's start and completion times are stamped by the
control plane, like the operator Backup's own, so they narrow the
window by seconds without removing the dependence on the two clocks a
dump carries. What is left: a neighbour's dump is taken only when this
run's own is missing from the listing or misplaced on both clocks and
the neighbour's operator Backup no longer exists or has not completed.
Every other unproven case leaves the size unset, as two runs that
overlap or follow each other within the slack already did.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-04 13:24:09 +00:00
Myasnikov Daniil 560c44f71e fix(backups): read a MariaDB dump from where its operator Backup wrote it
The size probe took the bucket, endpoint, prefix, credentials and TLS
settings from the strategy as it renders on the reconcile that creates
the Backup, which is not necessarily what it rendered when the operator
Backup was created: an existing operator Backup is reused untouched,
and a backup can run for up to the driver's 30-minute deadline. An
admin who edits the strategy in between sends the probe to the new
location, where it measures the wrong prefix or nothing at all.

The operator Backup's own spec.storage.s3 is where the dump went, so
the probe reads that. It is also the only source a restore can use,
since it targets that operator Backup long after the strategy may have
changed. An operator Backup that does not store to S3 is refused before
anything is listed.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-04 13:18:39 +00:00
Myasnikov Daniil 21f51a2e74 refactor(backups): return the matched MariaDB dump, not only its size
A MariaDB restore has to name the dump it restores from: without a
target time the operator restores the newest dump under the prefix,
which is not necessarily the one of the Backup being restored. The size
probe already finds that dump, but it returns only its size, folds a
missing, an ambiguous and an unreadable listing into the same 0, and
lives on BackupJobReconciler, which the restore reconciler is not.

Split the probe in two: findMariaDBDump reads the bucket and returns the
matched object (key, file name, the timestamp in its name, size and
LastModified) or an error that tells "not found" from "ambiguous"; the
size path keeps its timeout and reads only the size. The Secret lookup
and the S3 client move onto an objectStore value that any reconciler
can build from its own client, and BackupJobReconciler keeps one-line
wrappers so the CNPG probe is unchanged. sizeBytes is reported exactly
as before.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-04 13:16:15 +00:00
Myasnikov Daniil c140354977 test(backups): build operator Backups with the operator's success shape
mariadb-operator marks a Backup Complete=True for a failed Job as well
as a successful one; only the reason tells them apart (JobComplete with
message Success, or JobFailed with message Failed). The fixtures that
stand for a finished backup set Complete=True and nothing else, so they
describe a state the operator never produces and would read as a
failure to a driver that checks the reason. Give them the success shape
the operator writes.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-04 13:13:54 +00:00
Andrey Kolkov 92fa979690 fix(backups): trust only the endpoint CA where the backup writer does
With an endpoint CA the probe trusted the system roots plus that CA.
barman-cloud gets endpointCA as AWS_CA_BUNDLE, which replaces the
default trust store, so a CNPG probe could accept an endpoint the backup
itself would have refused. Let the connection say whether the CA adds to
the system roots or replaces them, and replace them for CNPG.
mariadb-operator appends its CA to the system pool, so MariaDB keeps
that.

Pin in reconcile tests that a Create the apiserver rejects still fails
the BackupJob, so only the read-back of an existing Backup is retried.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:23 +00:00
Andrey Kolkov 080df1b2f2 fix(backups): surface a Backup that cannot be read back and skip its probe
A Backup that exists but cannot be read back is retried through the
controller's backoff, which left the BackupJob Running with nothing on
it saying why. Set Ready=False with reason BackupNotReadBack before
handing the error back.

When the check for an existing Backup itself fails, the Backup may well
exist, so a size probe would only be discarded, and a read that keeps
failing reran it, up to the probe timeout on the single BackupJob
worker, on every retry. Skip the probe in that case and create the
Backup without a size.

Drive both behaviours through the MariaDB and CNPG reconciles in tests.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov 523d74a503 fix(backups): retry instead of failing when an existing Backup cannot be read
When Create collides with a Backup an earlier reconcile already made and
the read that follows fails, the drivers failed the BackupJob, although
the backup and its Backup both exist. Return a distinct error instead,
which the drivers hand back to the controller so the reconcile is
retried with backoff and then finds the Backup.

minio.New already refuses an endpoint with a path prefix and accepts a
trailing slash, so drop the duplicate check in the shared client; the
CNPG probe keeps its own check because it parses the URL itself and
would otherwise drop the path before minio sees it.

Say in the sizeBytes API doc what the value covers, such as the base
backup without the shared WAL archive for CNPG, and that strategies
which cannot attribute stored bytes to one backup leave it unset.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov 2b21763577 fix(backups): do not fail a backup on a transient retry-check read
The check for a Backup an earlier reconcile already created returned
its read error, and the drivers turn any error from that path into a
terminally Failed BackupJob. An apiserver timeout right after the
operator backup completed therefore lost a backup whose data was safely
in S3. The check only saves a probe, and the Create that follows already
handles an existing Backup, so log the error and go on to Create.

A trailing slash on the S3 endpoint is not a path prefix; the operators
accept it, so do not refuse it.

Pin the uncached AlreadyExists refetch, the upper slack of the MariaDB
window and the probe timeout with tests.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov a3d854739d fix(backups): attribute MariaDB dumps on two clocks and read retries uncached
The MariaDB probe identified this run's dump by the timestamp in its
file name, which comes from the backup pod's clock. With that clock
skewed past the slack, this run's dump falls outside the window while a
neighbouring run's dump can be the single match left, and its size is
reported as this backup's. A dump now has to fall inside the window by
both its name and its LastModified, which the object store stamps, so a
single skewed clock leaves the size unset instead.

The name parser also accepts the layout later mariadb-operator releases
write for compressed dumps, backup.<ts>.sql.<ext>, and checks the
compression token in either layout, so an operator upgrade does not
silently stop the measurement.

The retry check and the AlreadyExists refetch read the Backup through
the cache. A retry usually follows a Create made moments earlier; a
cache that does not show it yet turned the refetch into NotFound and
failed a backup that had succeeded. Read it from the API server instead.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov 3b86f57803 fix(backups): refuse to probe an S3 endpoint behind a path prefix
An endpoint such as https://s3.example.org/prefix cannot be addressed
by minio-go, which has no path-prefix option, and the prefix cannot be
added below the client either, because the SigV4 signature covers the
request path. The CNPG probe was worse than failing: it kept only the
host of endpointURL and listed the bucket at a location barman never
writes to.

Reject such endpoints with an explicit error, in the CNPG probe and in
the shared client, so the size is left unset instead of measured
somewhere else.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov 038f35376a fix(backups): do not report a partial listing or re-probe on retry
minio-go ends a listing whose context is cancelled between pages by
closing the channel without an error item. When the probe timeout
expired between pages of a large CNPG base backup, the pages already
read were summed and reported as the measured size. Check the context
once the listing channel closes and leave the size unset instead.

The probe ran before the Backup was created, so a reconcile that retried
after the Backup already existed probed the bucket again, held the
single BackupJob worker for up to the timeout, and then discarded the
result in favour of the existing object. Return an existing Backup
before probing.

Log the ambiguous MariaDB case (several dumps in the window) at info like
the other reasons the size stays unset, and cover the session-token
credential path with a test.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov f222e52863 fix(backups): harden the artifact size probe
The probe runs inside the reconcile that creates the Backup, and the
BackupJob controller has a single worker, so its timeout delays every
other BackupJob while an endpoint hangs. Lower it from 30s to 10s.

List the MariaDB prefix non-recursively, as mariadb-operator does. The
default MariaDB prefix and the default CNPG destinationPath are the same
<namespace>/<name>/ path, so a recursive listing of a MariaDB backed up
next to a same-named Postgres walked the whole WAL archive and ran out of
time. The CNPG base-backup prefix is still listed recursively.

Build the custom-CA transport from minio.DefaultTransport so it keeps the
proxy-from-environment and dial/TLS timeout settings a bare
http.Transport drops, and cover it with a test against a TLS endpoint
with a self-signed certificate.

Log the reasons the size could not be measured (credentials, client,
listing, empty CNPG prefix) at info. The chart runs the controller at
info, which hid them before. The expected ambiguity of several MariaDB
dumps in the window stays at debug.

Skip an endpointCA without a Secret name, as buildBarmanObjectStore does.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov de60488a2c fix(backups): report sizeBytes for CNPG backup artifacts
CNPG backups arrived without Backup.status.artifact.sizeBytes:
cnpg.io/Backup status carries no size, and the data lives in barman's
object-store layout rather than in a single object.

barman-cloud uploads every base backup under
<destinationPath>/<serverName>/base/<backupID>/ (backup.info plus the
data and tablespace tarballs, possibly split into parts). On completion
the driver now lists that prefix with the credentials and endpoint CA of
the rendered barmanObjectStore and records the total. The WAL archive is
not counted: it is one stream shared by every base backup of the server.
The field stays unset when the backup ID is unknown, the bucket cannot be
read, or nothing is stored under the prefix.

The S3 client and Secret lookup are now shared with the MariaDB driver.

Refs #4316

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov d4ad5612d3 fix(backups): report sizeBytes for MariaDB backup artifacts
Backup.status.artifact.sizeBytes was only ever set by the etcd strategy,
so MariaDB backups always arrived without a size.

mariadb-operator writes each logical dump as
<prefix>/backup.<RFC3339>[.<compression>].sql under a prefix shared by
every backup of the application, and the URI the driver records is
synthetic, so there is no single object to HEAD. On completion the driver
now lists the prefix with the tenant credentials the operator Backup was
rendered with and takes the size of the one dump whose timestamp falls
inside the operator Backup's lifetime. When the bucket cannot be read or
the window does not match exactly one dump, the field stays unset.

The API doc now states that an absent sizeBytes means "unknown", and the
Velero strategy documents why it leaves the field unset: kopia stores
data content-addressed across backups, so there is no per-backup stored
size.

Refs #4316

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Alexey Artamonov 6e36385e00 feat(api): expose HelmRelease failure state on Application status
Tenants usually read Applications but not the HelmReleases behind them,
and only Ready and Released were copied over. A client could therefore
see that a release was failing but not how often it had failed, nor
whether helm-controller had stopped retrying it.

Copy the Stalled condition, which helm-controller sets once it stops
reconciling a release (remediation retries exhausted, rollback target
missing, invalid CEL, access denied), and surface the install and
upgrade failure counters with the last attempted release action.

Stalled is dropped when it was observed for an older generation than
the HelmRelease's current one: the Application carries no generation,
so after a spec update a client would otherwise read the previous
spec's failure as terminal for the new values.

With the RetryOnFailure policy the platform generates today, a failed
release is retried rather than stalled, so the counters are the signal.
That does not hold for every state: a release missing from storage
while an install failure count is retained still stalls, and a release
on a remediation policy stalls once its retries run out.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-04 15:50:24 +03:00
Myasnikov Daniil 47b497194e fix(backups): restore the ClickHouse archive a Backup was taken as
The Altinity strategy named each archive inside its Pod with date -u
and recorded nothing on the Backup, so a restore could only guess: it
listed the remote archives under the application's prefix and took the
newest. A RestoreJob of an older Backup therefore restored whichever
backup had run last.

The controller now names the archive before it creates the backup Job,
from the BackupJob's persisted StartedAt, so every reconcile and every
retried Pod of the Job writes the same one. The name is recorded in the
Backup's driverMetadata next to the strategy parameters, and a restore
renders it into the strategy Pod, which restores exactly that archive
and fails, naming it, when remote storage no longer holds it. The name
is recorded only when the Job's Pod was given it, so a Job created by
an earlier controller, or a strategy that names its own archives,
leaves the Backup without one instead of pointing it at an archive
that does not exist.

Because a retried Pod reuses the name, it waits on an attempt the
sidecar is still running instead of being refused with 423, and asks
for resume, so a local or partial remote backup an earlier attempt left
is completed rather than refused as existing.

Backups taken before names were recorded stay restorable: the Pod picks
the newest archive whose name is stamped no later than the Backup's
takenAt, which is set only after the backup Job finished, and logs a
warning that the choice is a best effort.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-04 11:10:42 +00:00
Myasnikov Daniil 7b7444feaf fix(kubernetes): move the OIDC users out of the bootstrap Job's script
The bootstrap Job wrote a ClusterRoleBinding per oidc.users entry into
its script, and the script reaches the container as one `sh -c`
argument. Linux caps a single argument at 128 KiB, which about 240
users cross at roughly 530 bytes each. The container then cannot start,
the post-install and post-upgrade hook fails, and the tenant cluster's
HelmRelease stays unready. The values schema puts no bound on
oidc.users, so a shared cluster reaches that size with ordinary
per-person entries.

The bindings and the list of their names now reach the Job through a
ConfigMap it mounts, so the script is the same size for any list. Names,
labels and roles of the bindings are unchanged, so the bindings already
in a tenant cluster are updated in place and none is orphaned.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-04 07:06:31 +00:00
Myasnikov Daniil a2d0e4cb50 fix(cilium): pass the CiliumNode list to jq on stdin in the IPAM hook
The hook fed the whole `kubectl get ciliumnodes -o json` to jq as one
argument. Cilium copies every Node label onto its CiliumNode, 22 to 30
KiB per node on Talos or with NFD and GPU labels, so the list passes the
128 KiB Linux allows a single argument within six nodes. jq then fails
to start with "Argument list too long", the hook Job fails before it
moves any node, and the cilium HelmRelease loops on UpgradeFailed with
the HelmReleases behind it stuck in DependencyNotReady.

The hook renders on every upgrade of cilium in cluster-pool mode, so a
cluster that already finished the migration hits the same failure on its
next values change.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-04 06:57:03 +00:00
ZverGuy 95dc591198 test(kubernetes): keep application identity out of guest log streams
Guest pods are tenant-controlled, so their lineage labels are not a
trustworthy application identity, and the platform must not promote
them to the stream fields the parent scopes application logs by. The
guest release replaces the monitoring-agents filters and outputs
wholesale, which is what keeps the parent's label lift out of guests;
pin that so a later change to the guest overrides cannot bring it in.

This only covers what the platform ships. A guest admin can still edit
the guest's fluent-bit or write to the parent's vlinsert directly; that
needs a write-path control, which the logs proposal leaves open.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-04 04:18:09 +07:00
ZverGuy e5f1714836 feat(monitoring): keep application identity on container log lines
A reader that scopes logs to one application needs the owning
application on every line, and the lineage webhook already puts it on
the pod as apps.cozystack.io/application.{group,kind,name}. Fluent Bit
fetched those labels and then dropped them with the rest of the
kubernetes map, so the store could only be scoped by namespace.

Keep the three labels as apps_cozystack_io_application_{group,kind,name},
the names the metrics side uses, and make them stream fields on every
container-log output so the scope is an indexed stream filter. A pod
without the labels produces no such fields, and VictoriaLogs builds the
stream from the fields present, so existing streams keep their
identity; a labelled pod starts a new stream once.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-04 04:18:09 +07:00
ZverGuy 7de70ac2fc feat(monitoring-agents): export PVC labels from kube-state-metrics
The tenant metrics API scopes a query to one application by looking up
its pods and PVCs through their apps.cozystack.io/application.* lineage
labels. kube-state-metrics emits kube_persistentvolumeclaim_labels only
for resources on --metric-labels-allowlist, so without PVCs there the
PVC half of that lookup always comes back empty and volume series of
an application fall out of its scope.

All labels are allowed, as for pods and deployments: kube-state-metrics
emits one series per PVC whatever keys are listed, and PVCs are far
fewer than pods, which already export every label.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-04 04:12:15 +07:00
myasnikovdaniil 4087218a02 docs: move the repo-only guides onto the website (#5034)
## What this PR does

Seven guides under `docs/` were operator facing and written alongside
the code, and left in the source tree where no user looks. They are on
the website now, so this deletes them here and repoints every reference.

```
docs/operations/backup-classes.md                  -> operations/services/backup-classes
docs/operations/resource-presets.md                -> guides/resource-management
docs/storage-immutability.md                       -> operations/storage-class-immutability
docs/operations/system-memory-limits.md            -> operations/configuration/system-memory-limits
docs/gateway-classes.md                            -> networking/gateway-classes
docs/observability/distributed-tracing.md          -> operations/services/monitoring/distributed-tracing
docs/operations/seaweedfs-capacity-for-backups.md  -> operations/services/object-storage/capacity-for-backups
```

`docs/release.md`, `docs/deprecating-applications.md` and the two
SeaweedFS runbooks in `docs/operations/` stay. A workflow, `AGENTS.md`,
a migration and a bats suite read them by path, which is the test from
#3777.

References point at `https://cozystack.io/docs/v1/...`. The site
rewrites that short prefix to the latest version in `layouts/404.html`,
so the links follow releases and pin none. Backup Classes, resource
management and `storageClass` immutability exist in the latest version
already. The other four pages are only in `next/` until 1.7.0 is
promoted, and their links start to resolve then. The generated mongodb
files come from `values.yaml` through `make generate` with
cozyvalues-gen v1.7.0, the version pinned in
`.github/workflows/pre-commit.yml`.

Tests: helm unit tests for mongodb (151), postgres (209), vm-disk (114)
and kubernetes-nodes (223), go tests for backupcontroller, presets and
the operator, rd-presets-check, the migration-50 bats suite.

### Downstream repositories

- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#763

Walked the rest of the trigger map against the diff. No provider
follow-up: the mongodb `values.schema.json` diff is one `description`
string, with no field added, removed or renamed, no enum, no default,
and no `kind`, `plural` or `release.prefix` change.
`packages/core/platform/values.yaml` changed in comments only and no
key, so the hand written platform-package table on the site is
unaffected.

Please merge after cozystack/website#763, the links in the code only
resolve once those pages deploy.

### Release note

```release-note
docs: seven guides move from the docs/ directory of this repository to cozystack.io (Backup Classes, resource presets, storageClass immutability, system component memory limits, GatewayClass selection, distributed tracing, SeaweedFS capacity for backups), and in-repo references point there.
```
2026-11-03 23:09:46 +05:00
Timofei Larkin e31f518df7 ci(api-review-gate): publish the gate as a commit status (#3440)
## What this PR does

Publishes the `API Review Gate` merge context as a commit status, so a
skipped or cancelled run cannot pass it or poison it, and gives fork PRs
the same gate through a privileged `workflow_run` workflow.

### Why

The merge context was a job name. A required job that skips counts as
passed, and check runs are not deduplicated per name, so one cancelled
run on a head keeps the context failed until the next push. Measured on
this repo: `41d1153c` has four check runs with the gate's job name (one
`cancelled`, three `success`). Commit statuses are deduplicated per
context and the newest wins, so a run with no verdict posts nothing and
the previous status stays.

### What changes

Same-repo PRs get status from `api-gate-report`. A `pending` opener
marks the head before the evaluation on approve, dismiss and
request-changes reviews, so an old `success` is not mergeable for the
minute the evaluation takes. `COMMENTED` reviews no longer count as a
reviewer's latest review, so an owner who approved and then commented
keeps the approval. Concurrency is split, so bot comment run cannot
cancel live evaluation.

Fork PRs: `api-review-gate-fork.yaml` runs from the default branch and
reaches the verdict itself, the fork's run is only a trigger. `resolve`
finds PR by head SHA in open PR list (review runs on a fork report base
repo as head repository and have no pull requests). `detect` fetches the
head as data and runs the detector in a job with `contents: read` only.
`conclude` applies the approval rule and posts. A sweep every 15 minutes
moves the status of fork PRs whose runs wait for a maintainer in
`action_required`, those never fire `workflow_run`. This is lllamnyp's
patch, applied whole, with these changes on top:

- two open fork PRs on one head fail closed in `resolve` and in the
sweep, a head carried by a same-repo PR stays with `api-gate-report`
- detector is built from default branch only, merge base is picked by PR
author and an old one would mean an old detector
- the fork `detect` step runs under `bash -e` and detector exit 2 ended
it before the code was read, so it has `set +e` like the same-repo job
- head is checked out with links as links, like the same-repo gate does,
because flattening them hid a CRD behind a file link from the detector
while Helm still ships it

### Limits

- A fork PR whose push run waits for a maintainer's approval has no
status until that run executes, same as `E2E Tests`.
- A fork PR on a head that a same-repo PR also carries shows the verdict
standing there, one commit status cannot hold a verdict per base.
- The sweep can publish a stale `success` if it reads an approval,
stalls, and the owner withdraws it meanwhile. The next sweep flips it
back, so the window is up to 15 minutes.
- Two copies of one first-party CRD and CRDs behind directory links are
gaps of the detector on main, same for same-repo PRs, they are in
https://github.com/cozystack/cozystack/issues/5170.

### Testing

`actionlint` is clean on both workflows. `zizmor` is clean too, but
`dangerous-triggers` and `template-injection` are disabled repo-wide in
`.github/zizmor.yml`, so that says little about a privileged workflow. I
ran the `github-script` bodies under node with a mocked API (every
conclusion, approval and shared head case) and the run steps under `bash
-e` with a stub detector exiting 0, 2 and 1. `make unit-tests` passes
with `LC_ALL=C` (a kafka bats test depends on the locale, nothing to do
with this PR).

Not exercised live: the fork path and the sweep run only from the
default branch, so their first run is after merge.

### Repo admin

Branch protection requires the old job name `Require API owner review
for sizeable API changes`. This PR renames that job, so after merge the
required context has to be changed to the `API Review Gate` status.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```
2026-11-02 19:15:18 +04:00
Aleksei Sviridkin a2cc9a3016 fix(dashboard): link a tenant row to its detail page (#4004)
## What this PR does

Fixes #3822

Tenant list showed name as plain text and the only link in a row was
Edit button, so tenant detail page with its tabs and Delete was
reachable only by opening edit form and cancelling out of it. Name cell
is a link to that page now, it appears exactly where Edit appears and
points to the same CR. Plain click switches selected tenant to the row's
parent first, same as Edit does, because detail page takes namespace
from selected tenant. Ctrl, cmd and shift click don't run that switch,
so the tab you open a tenant from stays where it was.

First commit fixes what the link depends on. Console recovers a tenant's
parent from namespace name (`tenant-a-b` sits under `tenant-a`), but
root is the one level that naming skips: tenant `acme` ordered in
`tenant-root` owns `tenant-acme`, not `tenant-root-acme`. So
`realParentNamespace` returned undefined for every top level tenant,
which is most tenants on a normal cluster, and Edit button has been
missing there for as long as it exists. Ancestor labels carry the parent
that the name does not. Root level tenant named `root-x` resolves to CR
`root-x` now, not `x`.

### Notes

I first tried to make links carry their tenant in the url (`?tenant=`),
so a detail page opened in new tab resolves on its own. That turned into
questions about tenant context itself: what to do with a tenant in a
link the user can't see, what localStorage may hold, and TenantNamespace
LIST ignoring `labelSelector`. Detail route addresses a Tenant CR by
relative name and takes namespace from selected tenant, and making that
robust is a refactor of tenant context and routing, which should start
with study and planning, not with patches on patches. So this PR only
closes the UX gap (no way from tenant list into a tenant's page), going
deeper is out of scope.

Known limit that comes with it: link has no tenant in it, so opening it
in a new tab (ctrl/cmd/middle click, open in new tab, copied url)
resolves against the tenant last selected in any tab, because selection
lives in shared localStorage. For a row directly under that tenant it
opens the right page. For a row deeper in the tree the page says
`Application <name> not found.` and there is no Delete. If that tenant
also has a direct child tenant with the same relative name, new tab
opens that other tenant's page with working Edit and Delete, and delete
confirm says only `Delete Tenant "<name>"?`. Plain click in the same tab
is not affected. Detail page's own tabs and Edit link already behave
this way, it's how detail route resolves its namespace today and fixing
it properly is the refactor above.

A row links, and offers Edit, exactly when namespace holding its Tenant
CR is one the apiserver returns to this user. It returns a tenant
namespace only if the user is subject of a RoleBinding in it, so
sub-tenant admin that cant act in `tenant-root` sees no link there.

No row level Delete button. Issue allows either that or a route to
detail page, I took the route so confirm and mutation stay in one place.

Covered by tests, every source change here fails a test when reverted:

- top level tenant, nested one, root level tenant named `root-x`, forest
root without ancestor labels
- bridged row whose real parent is invisible gets neither link nor Edit
- plain click switches to the row's parent and opens detail page
- ctrl click leaves selection and location alone

### Screenshots

Tenant list under root on a dev cluster. Every row name is a link to
tenant page now (`pr4078` is hovered), top level tenants had neither
link nor Edit before. Host column redacted.

<!-- SCREENSHOT: drag in 01-tenant-list-links.png here -->
<img width="1440" height="900" alt="01-tenant-list-links"
src="https://github.com/user-attachments/assets/39cd723a-1ac5-4d3b-8c36-b70ddc626961"
/>


### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

All changed files are under `packages/system/dashboard/images/console`,
console has no downstream copy (standalone UI repo is archived).

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(dashboard): tenant name in the tenant list links to its detail page now (where Delete is), and top level tenants get their Edit button back.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Root-level tenant names are now displayed without incorrectly removing
prefixes, and tenants ordered directly under the root are correctly
recognized in the hierarchy.
* Tenant rows link to their detail pages when the parent tenant is
accessible. Same-tab navigation updates the active tenant, while opening
a link in another tab leaves the current selection unchanged.
  * Edit links and navigation now use consistent tenant routes.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-02 16:33:51 +03:00
Myasnikov Daniil b5e3b37a9a ci(api-review-gate): build the fork detector from the default branch
The merge base is the commit the fork's author branched from, so a
branch cut from an old commit selected an old detector, and a fix to
the detector never reached the PRs an attacker controls. The workflow
itself comes from the default branch, so the detector now does too.

The head is also checked out with its links intact. Flattening them
hid a CRD behind a file link from the detector while Helm still
shipped it, and made the verdict differ from the one the same-repo
gate reaches.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-02 14:47:51 +05:00
Aleksei Sviridkin d32092ac9a feat(release): attach an SBOM to every release image (#4997)
## What this PR does

This PR attaches an SPDX SBOM to every first-party image of a release.
BuildKit adds it at build time as an attestation, so it sits in the
image's index and moves with the digest the release pins, through
promotion. `docker buildx imagetools inspect <image>@<digest> --format
'{{ json .SBOM }}'` prints it, one document per platform. Releases cut
from this change on have it. The Talos installer image (built by the
imager, pushed by skopeo) and the third-party images don't.

It is more than `SBOM: '1'` in `tags.yaml`, because three things broke
on the way:

- An image built with an attestation is an index of the image and an
attestation manifest. `image_arch` in `hack/stitch-multiarch.sh` read
that as not one amd64 and one arm64 image, so the stitch skipped every
such image and still exited 0, and `hack/verify-multiarch.sh` then
failed the release with "index without linux/arm64". `image_arch` and
the rerun check look through the index now, and the stitched index keeps
both attestations.
- The release `make build` ran on the default docker driver, which
cannot attach attestations. It runs on the docker-container builder with
the docker.io mirror now, like the arm64 leg. `use: false` stays, so a
plain `docker build` keeps the default driver.
- `--sbom=true` pulls `docker/buildkit-syft-scanner:stable-1`, a
floating tag. `SBOM=1` passes `--attest type=sbom,generator=` with the
scanner pinned by digest (1.12.0). The comment in `hack/common-envs.mk`
called the output CycloneDX, BuildKit writes SPDX, and the same word in
`docs/security/self-assessment.md` is fixed too.

Only the release lane changes. PR, main and nightly builds push to OCIR
and are not what users install, so the OCIR attestation probe the
tracker (A2 in #3278) listed is not on this path, release images go to
GHCR.

### How it was checked

- Bats 1.14.0: four new tests in `hack/stitch-multiarch_test.bats` (the
arm64 leg's workflow shape, a stitch of SBOM images, a rerun after one,
an index of two linux images that is skipped), two in
`hack/common-envs_test.bats` (the flags, and the Renovate manager held
to the `SBOM_SCANNER` line), and the matchbox workflow test rewritten
for the new builder. The stitch tests fail on `main`'s
`stitch-multiarch.sh`, and each of these mutations fails one: dropping
the `length == 1` check, counting the attestation as a platform, looking
for the amd64 index itself in the rerun, taking `SBOM`, `BUILDER` or the
mirror config out of `tags.yaml`, going back to `--sbom=true`, dropping
the scanner digest, turning `SBOM` on by default, reformatting the
`SBOM_SCANNER` line out of the Renovate manager's regexp.
- A rehearsal on a localhost registry (docker 29.8, buildx 0.37.1) with
the real scripts. Images built through the `BUILDX_ARGS` of
`common-envs.mk` for amd64 and arm64, one without SBOM beside them, then
`stitch-multiarch.sh` (2 stitched, 0 skipped), `verify-multiarch.sh` (0
failed), `imagetools inspect` reading SPDX 2.3 with 15 packages per
platform from the stitched index, and `skopeo copy --multi-arch all`
keeping the index digest. `containerimage.digest` in `--metadata-file`
equals the pushed index digest, so the digest the package Makefiles
stamp is the index. The same run with `main`'s stitch skips the SBOM
image, exits 0 and fails the gate.
- `make bats-unit-tests`: 2444 of 2445. The one failure, "all-topics
backup skips double-underscore internals and keeps single-underscore
user topics" in `hack/kafka-topic-backup-strategy.bats`, also fails on
unmodified `main` on the same machine. `make bats-posix-compat-tests`
passes, and the stitch test file passes under dash through
`cozytest.sh`. `hack/check-commit-trailers.sh` passes. zizmor and
actionlint aren't installed here, so the workflow change (two env keys
and one renamed builder step) is covered only by the tests above.
- Not checked: that GHCR takes the attestations, what retention does to
the child manifests, the fork lane's OCI export (it doesn't set `SBOM`),
and a real release build. The first alpha cut after the merge tests the
first and the last. Renovate itself did not run on the new manager: the
config passes the Renovate JSON schema, and the regexp takes the tag and
the digest off the live line, which a test holds.

### Decisions for the release owners

- Scope is `tags.yaml` only. SBOMs on main and nightly images would need
OCIR to take attestations (not probed) and would change what the nightly
mirror copies.
- The scanner is 1.12.0 with syft 1.51.0 inside, which is in the range
of GHSA-mw2c-m758-9v5q (memory exhaustion parsing binary inputs, fixed
in syft 1.52.0). It reads images our own builds produce, so I didn't
hold the change for it. A Renovate manager on the `SBOM_SCANNER` line
moves the version and the digest together when the scanner ships a newer
syft.
- The scanner adds a step per image and I didn't time it on the release
runners.
- A stitch rerun reads the amd64 index the tree pins, which stays in the
registry untagged after a stitch. If the registry drops untagged
manifests before the rerun, it fails at that read rather than repinning.
- After the first alpha with attestations I'll move A2b in #3278 and the
SBOM item in #2937 to done. The SBOM row in
`docs/security/self-assessment.md` still says SBOMs aren't published,
and stays until that alpha shows they are.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

I walked the trigger map against the diff. It touches `tags.yaml`,
`hack/common-envs.mk`, the stitch and verify scripts and three docs. No
release asset is renamed, `hack/package.mk` and the commit convention
are untouched, and the flow a maintainer follows to cut and promote a
release is the same.

### Release note

```release-note
feat(release): every first-party container image of a release carries an SPDX SBOM as a BuildKit attestation in its index, readable with `docker buildx imagetools inspect <image>@<digest> --format '{{ json .SBOM }}'`. Releases cut from this change on have it, the Talos installer image and third-party images don't.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

- **New Features**
- Supported release image builds now include SPDX-format software bills
of materials (SBOMs) as attestations.
- Multi-architecture image stitching, retagging, and mirroring preserve
SBOM attestations.
- **Documentation**
- Updated image-reference guidance to explain how to handle attestation
entries in multi-architecture indexes.
- Clarified which images include SBOMs, how to inspect them, and that
SBOM generation remains disabled by default outside configured release
builds.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-01 22:35:38 +03:00
IvanHunters 94ada11eef fix(marketplace): pin a tapped repository to a digest instead of a mutable tag (#5082)
## What this PR does

A tapped (community-connected) repository followed a mutable OCI tag:
the Flux `OCIRepository` carried only `spec.ref.tag` with a 5-minute
interval, so whoever controlled the tag controlled what ran in the
management cluster within that interval, down to the HelmReleases of
already-running tenant application instances. Fixes #4979.

A tap is now pinned to a digest. `cozypkg tap` and the connect API
accept an explicit `oci://host/org/repo@sha256:...` reference; a tag
reference is resolved and pinned to its digest by the tap materializer
once the artifact for the current spec generation is ready, from the
digest Flux reports in `status.artifact.revision`, with the tag kept
informational. A tap connected before this change is pinned to its
current digest on the first reconcile after upgrade, and a re-connect to
a new tag waits for source-controller to re-resolve before pinning, so a
stale revision is never pinned. Moving the tag upstream no longer
changes anything in the cluster; an explicit re-connect moves the pin.

Connect also accepts an optional keyless cosign verification policy
(`--certificate-identity` / `--certificate-oidc-issuer` on the CLI,
`spec.verify` on the API), anchored to an exact identity so a
superstring cannot match, and preserves it, and the pull secret, across
repeat connects rather than silently dropping them. With those flags the
CLI verifies the signature at tap time and pins the verified digest
before pulling, so the content it applies is what a trusted identity
signed; it fails closed if the verified digest cannot be determined.

### Screenshots

Not applicable: no UI changes.

### Downstream repositories

I walked the trigger map against the diff. The change is confined to
`cozypkg`, the `Tap` aggregated API (a virtual, cluster-scoped resource
with no CRD), the tap materializer, and the community-index docs. It
does not touch `packages/apps`, `packages/core/platform/values.yaml`,
installer values, platform variants/components, release assets,
`ApplicationDefinition` semantics, the `hack/` make targets, or the
commit convention, so the provider, ansible, ccp and external-apps paths
are not reached.

One path is arguable and needs a human decision, so I have left every
box unticked rather than tick or dismiss it:

- `cozystack/website` — `cozypkg tap` gains flags and digest-pinning
behaviour. The marketplace tap flow is new and I am not sure it has a
reference page on the site yet, so whether a docs follow-up is warranted
(and where) is a maintainer call.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(marketplace): a tapped repository is pinned to a digest instead of following a mutable tag, so moving the tag upstream no longer changes what runs in the cluster; connect accepts oci://...@sha256:... and an optional keyless cosign verify policy (anchored to an exact identity and preserved across re-connects), and with verify flags the CLI verifies the signature and pins the verified digest at tap time.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

- **New Features**
- OCI repositories can be connected using a SHA-256 digest, with or
without a tag. Tag-based sources are pinned to the resolved artifact
digest.
- Optional Cosign keyless verification is available during connection
using certificate identity and OIDC issuer details.
- Reconnecting preserves existing verification and pull-credential
settings when replacement values aren’t provided.
- **Bug Fixes**
- Reconnecting with a new tag clears previous operator-managed pins so
the requested tag can resolve to its current artifact.
- **Documentation**
- Clarified tag pinning, signature verification, and where tapped app
charts run.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-30 18:09:34 +03:00
IvanHunters ccc216652e fix(monitoring-agents)!: isolate merged log payload under log_processed (#5167)
## What this PR does

Fluent Bit's `kubernetes` filter runs with `Merge_Log On` but without
`Merge_Log_Key`, so a parsed JSON log line is merged into the record
root.
An application that logs JSON can thereby set top-level fields that
later
pipeline stages trust for routing and tenant isolation:
`kubernetes_namespace_name`
(rewrite_tag routing and the `_stream_fields` stream identity),
`kubernetes.labels`, and the `tenant` / `cluster` / `log_source`
markers.
A tenant workload can forge another tenant's values on its own log
lines.

Setting `Merge_Log_Key log_processed` nests the parsed payload under a
dedicated key, so application log content can no longer override the
top-level service fields.

The change is applied to both Fluent Bit configurations that carry this
filter: the platform agent
(`packages/system/monitoring-agents/values.yaml`)
and the guest-cluster agent (`packages/apps/kubernetes`).

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff,
file by file. The change touches only the Fluent Bit filter config
inside
`monitoring-agents`; it does not alter package layout, any
`values.schema.json`,
platform values keys, `ApplicationDefinition`, any CRD, `hack/`,
namespaces,
telemetry metric names, or the commit convention. None of the listed
couplings match.

- [x] No downstream repository is affected by this change

### Release note

```release-note
fix(monitoring-agents)!: Fluent Bit now nests parsed JSON log content under a `log_processed` key instead of merging it into the record root. This prevents an application log line from forging the service fields used for log routing and tenant isolation. BREAKING: top-level fields previously extracted from JSON logs (for example `level`, `msg`, and any application-specific keys) now live under `log_processed.*`; external dashboards and LogsQL queries that read them from the record root must be updated.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Improvements**
* Merged Kubernetes log content is now grouped under `log_processed` in
Fluent Bit output.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-30 17:14:45 +03:00
Myasnikov Daniil 7f55ff5d85 fix(dashboard): link a tenant row to its detail page
The tenant list rendered the name as plain text and put the row's only
link on an Edit button, so nothing in the list reached the detail page
that every other kind gets, tabs and a Delete action included. The only
way there was to open the edit form and cancel out of it.

The name cell becomes a link wherever Edit is offered, to the same CR.
The detail page reads the namespace from the tenant context, so a plain
click aligns the context to the row's parent first, the way Edit
already does. React runs that handler for a ctrl-, cmd- or shift-click
too, which the browser turns into a new tab or window, so it runs only
for the click that navigates its own tab; otherwise opening a tenant
beside the current one would move the tab left behind as well.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-30 16:38:53 +05:00
Myasnikov Daniil e44e41d307 fix(dashboard): resolve the Tenant CR of a tenant ordered in the root
A tenant's CR lives in its parent's namespace, and the console recovered
the parent from the namespace name: `tenant-a-b` sits under `tenant-a`.
The root is the one level that naming skips. A tenant `acme` ordered in
`tenant-root` owns `tenant-acme`, not `tenant-root-acme`, so the prefix
test never matched a top-level tenant and `realParentNamespace` returned
undefined for it. That is most tenants on a normal cluster, and the Edit
button, offered only where the CR's namespace resolves, was missing for
all of them.

The ancestor labels carry the parent the name does not, so the lookup
falls back to `tenant-root` when the chain names it. Deriving the CR name
keys on the parent being the root rather than on the child's name
happening to start with it, which keeps a root-level tenant called
`root-x` from resolving to the CR `x`, and the row label agrees with it
rather than showing `x`.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-30 16:23:10 +05:00
Myasnikov Daniil dfa2283a13 ci(api-review-gate): close a standing success on a shared fork head
A commit status belongs to the commit. When two open fork PRs carry the
same head, a success already standing on it may have been earned by one
of them on another base, or may rest on an approval that was withdrawn
on a PR whose run waits for a maintainer and so never reaches resolve.
The sweep skipped such a head and left that success in place, which let
a duplicate fork PR keep a withdrawn approval from taking effect.

The sweep now writes the same failure resolve writes, once per head,
when a success set by a workflow stands on a head that only fork PRs
carry. It still never creates a status, and a head that an open
same-repo PR carries stays with api-gate-report.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-30 16:09:00 +05:00