mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 12:13:55 +09:00
d530ca8cf534eb4387934eccae6a94402ddfe01b
7798
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
d530ca8cf5 |
fix(backups): keep a ClickHouse archive an earlier attempt finished
A backup Pod that loses sight of its upload, by dying or by losing six polls in a row, is retried while the sidecar finishes that upload. The retry found it in the action log as a success, took the sidecar for idle and asked for create_remote with resume again. clickhouse-backup then rebuilds the local backup and uploads it once more over the archive already in the bucket, which is left broken if that second upload dies halfway. The archive name belongs to one BackupJob, so a success row of exactly this upload can only be an earlier attempt's, and the retry now takes it as its result. An error or cancel row still leads to a resumed upload, and a restore still runs again, since a restore is not named per RestoreJob. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
89f8cc02ee |
Merge branch 'main' into fix/clickhouse-idle-before-backup
main now names the ClickHouse archive in the controller and restores the archive a Backup recorded, in the same strategy script this branch gates on an idle sidecar. The merged script keeps both: the archive name and its restore selection from main, the idle gate, retried polls and lost-operation check from this branch. A retried backup is taken over through the same idle gate as a restore, matched on the whole command line the sidecar logs. With resume requested that line reads "create_remote --resume <name>", so the gate matches that form; matching on the name alone would also take another operation's row ending in the same name for this one. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
ae1e1830ce |
test(backups): cover a ClickHouse restore against an unreachable sidecar
The restore path resolves its archive from list/remote before the start gate runs, so it carries its own wait for a sidecar that is not up, and only the backup path had a test for a sidecar that never answers. The retry comment also miscounted: the limit is attempts, not retries. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
993b79bb37 |
fix(backups): take over a running ClickHouse restore on Job retry
restore_remote drops every table it restores before recreating it, so running it twice discards whatever was written after the first run finished. The strategy Job retries a failed Pod up to twice, and with the start gate waiting for an idle sidecar instead of hitting 423, a retry whose predecessor died after its POST waited for that restore to finish and then started a second one. A retry now takes over a restore_remote of the same archive that is still running. A finished row is not taken over: the sidecar never trims its action log, so a success row of the archive may belong to an earlier RestoreJob, and treating it as this one would report a restore that never ran. A retry that arrives after the restore finished therefore still runs it again. Once the operation is accepted, one failed poll no longer ends the attempt under errexit; six in a row do. An operation that ends cancelled now fails the Pod instead of being polled forever. A retried backup still starts a new create_remote, because its archive name changes on every attempt; taking it over needs a name fixed for the whole BackupJob. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
970191dc8a |
fix(clickhouse): restore the archive a Backup was taken as (#5208)
## What this PR does A ClickHouse `RestoreJob` restored the newest archive under the app prefix, whichever `Backup` it named. Strategy pod generated the archive name with `date -u` and nothing was recorded on the `Backup`, so restore could only list the remote archives and take the last one. Restoring an older `Backup` brought back the latest data. Now the controller names the archive before it creates the backup Job (`clickhouse-<app>-<startedAt>` from persisted `status.startedAt`, so retried pods write the same one), passes it to the pod as `.ArchiveName` and records it in the `Backup` `driverMetadata`. Restore takes exactly that archive and fails with its name when storage has no such archive. The name is recorded only when the Job's pod got it, so a Job left from the previous controller during an upgrade, or a custom strategy with its own names, leaves no name instead of a wrong one. A retried pod reuses the name, so it waits for an attempt the sidecar still runs (no 423) and asks for `resume`. A `Backup` outlives its `BackupJob` and a Plan names `BackupJob`s after the schedule slot, so a `BackupJob` created again under a used name met the earlier run's `Backup` when its Job completed, took it as its result and left the archive it just wrote recorded nowhere. Now it fails if that `Backup` records another archive, and the message names both. A retry of the same run still gets its own `Backup` back. Old `Backup`s have no recorded name and stay restorable: the pod logs a `WARNING` and takes the newest archive stamped no later than `takenAt`, and fails when nothing qualifies. While the chart rolls out, previous controller pod can still render the new template and pass no `takenAt`, so a restore of a `Backup` without recorded name fails before it touches any table, and works when created again after the rollout. Fixes https://github.com/cozystack/cozystack/issues/5199 Tests are go unit tests for naming, recording, a reused `BackupJob` name and restore render, and `hack/altinity-backup-strategy.bats`, which runs the shipped script against a stubbed clickhouse-backup API (the restore tests go red on the old template). Checked on a live ClickHouse with two backups of 1500 and 2000 rows: restore of the older one by recorded name ends with 1500 rows, fallback for an unnamed older backup picks the same archive with the warning, a missing archive fails without touching the table, a retry over an existing local backup completes under the same name. The controller binary itself was not run on the cluster, only its rendered output in the pod. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(clickhouse): a RestoreJob now restores the archive its Backup was taken as, not the newest one. New Backups record the archive name, Backups taken before this release are restored best effort from the newest archive stamped no later than their takenAt, with a warning in the restore pod log. A ClickHouse BackupJob created again under a name whose Backup still exists now fails instead of reporting that Backup as its result. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Backups now use a consistent archive name tied to the backup job, and completed backups retain that name for future restores. * Restores can select the recorded archive and backup timestamp. If no archive was recorded, the system selects the newest eligible archive for the application. * **Bug Fixes** * A restore now fails when its recorded archive is missing instead of silently selecting a different one. * Backup requests avoid starting a duplicate action when the matching action is already in progress. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
18779dddfc |
fix(backups): report sizeBytes for MariaDB and CNPG backup artifacts (#5223)
## What this PR does Fixes https://github.com/cozystack/cozystack/issues/4316. Continues https://github.com/cozystack/cozystack/pull/5015 by androndo on a separate branch. All ten of their commits are kept as they were (authorship, messages and order, only rebased on current `main`). MariaDB and CNPG backups stored in S3 report `Backup.status.artifact.sizeBytes`, the size of the dump mariadb-operator uploaded or the total of the barman base backup, and the field stays unset when the size cannot be attributed to that one backup. Four commits are added on top: - `test(backups)`: operator Backup fixtures carry the condition mariadb-operator writes on success, `Complete=True` with reason `JobComplete` and message `Success`. Operator sets `Complete=True` for a failed Job too (reason `JobFailed`), so a fixture without reason describes a state that never happens. https://github.com/cozystack/cozystack/pull/5207 makes the driver check that reason, the tests pass with and without it. - `refactor(backups)`: the MariaDB matcher returns the matched dump (key, file name, timestamp from the name, size, LastModified) and tells "not found" from "ambiguous", and the Secret lookup and S3 client live on a small value any reconciler can build from its own client. A MariaDB restore needs this to restore the dump of the Backup it was given and not the newest one under the prefix. `sizeBytes` does not change here. - `fix(backups)`: the probe reads bucket, prefix, endpoint, credentials and TLS from the operator Backup's own `spec.storage.s3`, not from the strategy as it renders now, so editing the strategy during a backup does not send the probe somewhere else. - `fix(backups)`: a dump is attributed only when its ownership is proven. Before, a backup pod with a clock more than a minute behind, right after a previous run, got that run's dump size (666 bytes for a 123-byte dump in a test). Now the dump has to be the only one that either clock places in the operator Backup's lifetime, both clocks have to agree, and neither may place it in the lifetime of another finished operator Backup with the same bucket and prefix. Otherwise the size stays unset. Two cases are still not covered. First, this run's own dump is missing from the listing while the operator Backup of a neighbour was deleted or has not finished, then the neighbour's dump can be taken. Second, backup pod clock and object store clock are both off the same way by more than the minute of slack, which moves a finished neighbour's dump into this run's window, and nothing the clocks record tells them apart. A run whose Job failed has no dump of its own and reaches the probe only because the driver takes `Complete=True` for success, https://github.com/cozystack/cozystack/pull/5207 fixes that. Of the open threads on the original PR, the cache reads, the dump file name layout and the default AWS endpoint test were already handled by its later commits, the clock skew one is the last commit here and the failed Job case is https://github.com/cozystack/cozystack/pull/5207. `minio-go` is new to this tree, this PR adds it as a direct dependency and `internal/backupcontroller/artifact_size.go` is its only importer. The open Bucket strategy PR https://github.com/cozystack/cozystack/pull/4235 picks the same client at the same version. Each clock skew case, and a run whose own dump is gone while the neighbour's operator Backup still exists and has finished, have a unit test that fails on the old matcher, and every new guard has a mutant that turns a test red. I also ran the matcher against a real bucket after four MariaDB backups in a row. Runs that finished more than a minute apart mapped to the dump their operator Job logged, the two back to back came back unset instead of taking each other's dump, and with a run's own dump hidden from the listing the matcher refused the neighbour's dump that the old one returned. ### Screenshots N/A, no UI changes. ### Downstream repositories The only API change is the description of the existing `Backup.status.artifact.sizeBytes` field, no new field, value, default or enum, no package, values or tooling change, and neither cozystack/website nor cozystack/terraform-provider-cozystack references `sizeBytes`. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(backups): MariaDB and CNPG (Postgres) backups stored in S3 now report `status.artifact.sizeBytes`: the size of the dump mariadb-operator uploaded, or the total size of the barman base backup (WAL archive excluded). A MariaDB backup takes the size of the one dump that falls inside its run on both the backup pod's and the object store's clock and inside no other finished run, and the size is left unset rather than set to 0 whenever it cannot be measured or attributed. Velero-based backups intentionally keep it unset. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Backup artifacts now report the stored size attributable to an individual MariaDB or CNPG backup when it can be determined. Shared or ambiguous storage may leave the size unset. * **Bug Fixes** * Improved handling of backup artifact creation and lookup errors, including retrying when an existing backup cannot be read and failing jobs when artifact creation is rejected. * **Documentation** * Clarified that an unset size means unknown; zero does not indicate an unknown size. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
d0a73302d9 |
fix(backups): fail a ClickHouse backup whose operation the sidecar lost
The sidecar keeps its action log only in memory, so a sidecar restart after create_remote was accepted drops the operation's row, and the strategy kept polling for it forever under a Job with no deadline. Until now the resumed upload re-added a row ending in the archive name and ended the loop, but resume is now off so a failed attempt never reaches the bucket, which leaves nothing to end it. The row is logged before the POST is acknowledged, so its absence means the operation is gone and the attempt has to fail. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
e5bb3cda07 |
fix(kubernetes): move the OIDC users out of the bootstrap Job's script (#5213)
## What this PR does Found while checking for other places with the failure from https://github.com/cozystack/cozystack/issues/4968: data that grows with configuration is passed to a process as one argument, and Linux allows 128 KiB for that. This one is in the `kubernetes` app and there is no issue about it yet. The OIDC bootstrap Job wrote a ClusterRoleBinding for every `oidc.users` entry into its `sh -c` script, 526 bytes per user. I rendered current main with `helm template` and users like `user123@example.org`: 238 users give a 130,889 bytes script that starts, 239 give 131,415 bytes and `exec` fails with E2BIG. Then the container can't start, the post-install and post-upgrade hook fails after its retries and the HelmRelease of the tenant cluster stays unready. Schema has no limit for `oidc.users` and each person needs own entry, so a cluster shared by 240 people gets there. Now the bindings and the list of their names are rendered by the new `oidc-rbac-bindings.yaml` into a ConfigMap `<release>-oidc-bindings` that the Job mounts, so the script is 6,380 bytes for any number of users. The ConfigMap has its own 1 MiB limit, about 2,500 users at 417 bytes each. Names, labels and roles of the bindings stay the same. I compared the render of old and new chart for eight emails (ones YAML reads as a boolean or null, a number, a quote, an apostrophe, unicode and two plain) and the bindings are identical, so existing ones are updated in place and nothing is orphaned. The prune step reads the same ConfigMap. If the list is not mounted the Job stops before it deletes anything, because `grep` on a missing file exits 2, `! grep` is true and every binding of the release would be pruned. A release with no users applies nothing (`kubectl apply` fails on an empty file) and prunes all stale bindings, same as before. `hack/kubernetes-oidc-bindings.bats` renders 300 users, checks the script is under the limit without any email in it and that every user has one binding with the old name and role. Three more tests run the rendered script against a fake `kubectl` and check what it applies and deletes. The prune condition forced to true or to false passes all helm-unittest cases, so this part is tested in bats. The bug is in release-1.6 and release-1.7 as well. ### Downstream repositories I walked the trigger map against the diff (one chart template, its tests and a bats file): no package added or removed, no values, schema or default changed, no output Secret or Service renamed. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(kubernetes): the OIDC bootstrap Job no longer fails to start for a tenant cluster with about 240 or more oidc.users, the bindings now come from a ConfigMap the Job mounts. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Kubernetes OIDC configuration now supports per-user role bindings, with administrator and viewer access assigned according to each user’s role. * OIDC role bindings are kept in sync with the configured user list, including removal of obsolete bindings. * Deployments with no OIDC users or with OIDC mode disabled continue to avoid creating user bindings. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
ab13f9f72c |
fix(cilium): pass the CiliumNode list to jq on stdin in the IPAM hook (#5212)
## What this PR does Fixes https://github.com/cozystack/cozystack/issues/4968. The cilium IPAM migration hook passed the whole `kubectl get ciliumnodes -o json` to `jq` as one argument, and Linux allows 128 KiB for a single argument. Cilium copies Node labels onto every CiliumNode (22 to 30 KiB per node with Talos, NFD or GPU labels), so the list is over the limit within six nodes. `jq` can't start (`Argument list too long`), the hook Job fails before it moves any node, `cilium` HelmRelease loops on `UpgradeFailed` and everything that depends on networking stays in `DependencyNotReady`. The hook is rendered on every upgrade of cilium in `cluster-pool` mode, so a cluster that already finished the migration hits the same on its next values change. The failing step only logs the pods that still hold an old Ingress IP, but it runs before the node loop, so on a cluster that is not migrated yet it blocks the migration itself. Now the list goes to `jq` on stdin together with the IP objects, same as the other `jq` calls in the script get their input. The new test in `hack/cilium-ipam-migration.bats` makes six CiliumNodes over the limit (one 24 KiB annotation each instead of labels, and it checks the list is over 128 KiB even printed compact). It fails on the old script with status 126 and passes now. It also checks that each pod is named with the node whose Ingress IP it holds and that a pod on an unrelated address is not named, because a join that always names the first node, or reports every IP object, passed the old tests. I did not take two suggestions from the issue. Skipping the hook when every CiliumNode is in the pool changes what one existing test pins (the hook adds the pool to `excludeIps` in that case) and doesn't help clusters that are not migrated yet. A value to disable the hook leaves the default broken. `cozystack-operator` rewriting `disableHooks` and helm-controller rolling the DaemonSet on every retry are other components. The bug is in release-1.6 and release-1.7 as well. ### Downstream repositories I walked the trigger map against the diff (one hook script and its bats test): no package, values, schema, default or output name changed. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(cilium): the IPAM migration hook no longer fails with "Argument list too long" on clusters with more than a few nodes, which blocked the cilium upgrade and every package that depends on networking. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * Improved IP address migration handling for environments with large node lists, avoiding failures caused by operating system argument-size limits. * Migration continues to identify affected pods by node and distinguish unrelated pods. * **Tests** * Added coverage for migrations involving large node lists, multiple IP objects, and unrelated pods. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
d7ae34dfc3 |
fix(backups): fail a ClickHouse BackupJob whose Backup name is taken
A Backup outlives the BackupJob that made it, so a BackupJob created again under a name already used meets the earlier run's Backup when its own Job completes. That happens by hand, and from a Plan reconciled again inside its starting deadline, since a Plan names each BackupJob after its schedule slot. The Altinity driver took the earlier Backup as its result: the BackupJob turned Succeeded with a BackupRef recording the earlier run's archive, the archive this run wrote was recorded on nothing, and restoring what the BackupJob reported brought back the older data. An existing Backup is now taken as this run's only when it records this run's archive, which is what an earlier reconcile of the same run leaves when its status write is lost. Any other fails the BackupJob, naming both archives, and leaves the earlier Backup untouched. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
d5559bb934 |
test(backups): remove the Altinity strategy tests' scratch directories
Every test in hack/altinity-backup-strategy.bats made a directory with mktemp -d and never removed it, so each passing run of the file left nine directories in TMPDIR, and they pile up on any machine that keeps it between runs. Each test now removes its directory as its last statement. A failed assertion aborts the body before that line, so a failing test keeps its stub log and rendered script for inspection. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
3d317c5f67 |
feat(api): expose HelmRelease failure state on Application status (#5171)
## What this PR does Tenants usually read Applications but not the HelmReleases behind them, and the Application status only carried the `Ready` and `Released` conditions. A client could see that a release was failing, but not how many times it had failed or whether helm-controller had stopped retrying it. This PR copies the `Stalled` condition onto the Application and adds three optional status fields taken from the HelmRelease: `installFailures`, `upgradeFailures` and `lastAttemptedReleaseAction`. All of them are additive and `omitempty`, so existing clients are unaffected. What the new fields do and do not tell a client: - With the `RetryOnFailure` policy the platform generates today, a failed release is retried rather than stalled, so the counters are the signal. That does not hold for every state: a release missing from Helm storage while an install failure count is retained still stalls (the case in #3554), a release on a remediation policy stalls once its retries run out, and terminal errors such as an invalid CEL expression or denied access stall too. - `Stalled` is copied only when it was observed for the HelmRelease's current generation. The Application carries no generation, so after a spec update a client would otherwise read the previous spec's failure as terminal for the new values; until helm-controller reconciles the new spec, the Application shows no `Stalled`. - The counters only count attempts that recorded a Helm release. A failure before Helm records one, such as a chart that fails to render (for example a `fail` guard rejecting the values), shows only as `Ready=False` and leaves them at zero; a failure to apply the rendered resources is counted. - helm-controller resets the counters when the desired state changes (generation, chart version, values), and on success only under `RetryOnFailure`, so a `Ready` release can still carry non-zero counters. The field descriptions say so. `TenantModule` is left unchanged on purpose: it still copies only `Ready` and `Released`. Related: #5109, #4155 Tests cover the copied condition and counters, a `Stalled` left over from an older generation after a spec update, the counters kept on a `Ready` release, and that `Reconciling` is still not copied. The OpenAPI schema and apply configurations are regenerated, and every per-kind status schema picks up the new fields. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The Terraform provider reads only the `Ready` condition by type, so the extra condition and fields do not change its behaviour. The website only shows an example Application output, not a description of the status contract. ### Release note ```release-note feat(api): Applications now expose the HelmRelease `Stalled` condition and the `installFailures`, `upgradeFailures` and `lastAttemptedReleaseAction` status fields, so tenants without access to HelmReleases can see how a release is failing. Tools that compute health with kstatus, such as Flux Kustomizations waiting on Applications, now report an Application whose release has stalled as failed. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Application status now displays install and upgrade failure counts, along with the most recently attempted release action. Failure counts reflect attempts recorded as releases and reset when the desired state changes. * Current stalled release conditions are now reflected in application status, helping clarify when a release is blocked. * Failure details remain available when a release is ready, while reconciling conditions are not shown. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
3a10f956cc |
test(backups): cover a restore that starts on an idle ClickHouse sidecar
The strategy suite only exercised restore against a busy sidecar, so a gate that wrongly blocked restore on an idle one would have stayed green. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
a25db5aeb6 |
fix(backups): wait for an idle ClickHouse backup sidecar before starting
The sidecar refuses a new operation with 423 while another one is in progress, but the strategy only waited for the API to answer, and went on to start its operation even when it never did. An operation already running in the sidecar therefore failed the attempt within seconds. Poll the action log until nothing is in progress, and fail the Pod instead of proceeding when the sidecar stays busy or unreachable, so the failure names the cause rather than surfacing as a 423 or a connection error from the operation itself. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
4f3f1a5cad |
fix(clickhouse): stop the backup sidecar from publishing failed attempts
create_remote keeps its local backup when the upload fails, and with the upstream defaults the sidecar never prunes local backups and resumes every unfinished upload when it starts. A strategy attempt that fails mid-upload therefore lands in the bucket on the next sidecar restart as an archive no Backup refers to, and restore can pick it as the newest one. The resumed upload also holds the API lock, so the strategy Pods racing it get 423 and burn the Job's retries. Turning the resume off keeps a failed attempt out of the bucket, and a negative local retention leaves at most the newest local backup and drops it once its upload succeeds, so leftovers no longer accumulate on the data volume. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
c1ba05574f |
feat(monitoring-agents): keep application identity in container log streams (#5217)
## What this PR does
Makes Fluent Bit keep the pod lineage labels
`apps.cozystack.io/application.{group,kind,name}` on container log
lines, as the stream fields
`apps_cozystack_io_application_{group,kind,name}`, for both the default
container-log output and the per-tenant routed output.
Why: the per-application logs scope in the design proposal [Tenant
application logs](https://github.com/cozystack/community/pull/95)
(section 6) filters by these stream fields. Today the kubernetes filter
fetches pod labels, but they are dropped together with the rest of the
`kubernetes_*` keys before output.
What changes in the pipeline:
- A second `nest` lift of the pod labels, then a rename of the three
lineage keys to their final names before the existing `Remove_wildcard
kubernetes_` runs. All other labels are still removed. Every lifted key
is prefixed, so no other label can collide into the identity names.
- The three keys are added to `_stream_fields` of both container-log
outputs. Events and audit pipelines are unchanged.
- A parsed JSON log line is kept under `log_processed` (already on
`main`), so a line's content cannot create the identity keys.
- Guest Kubernetes clusters are not affected: their monitoring-agents
release replaces the filters and outputs wholesale. A regression test in
`packages/apps/kubernetes` pins that guest streams carry no identity
fields.
Effect on existing streams: VictoriaLogs treats a missing stream field
like an empty one, and the rename writes nothing when the label is
absent. Pods without lineage labels keep exactly the same stream. Pods
with lineage labels get a new stream once after the rollout; their
earlier lines stay reachable by namespace.
#### How it was tested
- helm-unittest for `monitoring-agents` (lift order, rename before
`Remove_wildcard`, stream fields on both outputs, events and audit
untouched) and for `kubernetes` (guest streams unchanged). The
events/audit guard was mutation-checked.
- Verified on a test cluster: real pod log lines arrive, Redis pods'
lines carry the three `apps_cozystack_io_application_*` stream fields,
and Fluent Bit logs show no errors.
#### Known limitation
Guest clusters write to the parent log store without authentication, so
a guest administrator can forge these stream fields as well as the
namespace field, within the store their tenant shares. This is the
write-path trust risk already documented in the proposal.
### Screenshots
No UI changes.
### Downstream repositories
- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
Downstream: no trigger matched. Candidate: none affected, because the
diff changes `packages/system/monitoring-agents/values.yaml` (a system
package with no schema) and adds tests; the file under
`packages/apps/kubernetes` is a test only, so the app's `values.yaml`
and `values.schema.json` are unchanged.
### Release note
```release-note
feat(monitoring-agents): container log lines of pods that belong to a Cozystack application now carry `apps_cozystack_io_application_{group,kind,name}` stream fields in VictoriaLogs. Streams of pods without application labels are unchanged; streams of application pods start a new stream once after the upgrade.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **New Features**
* Container-log streams now include application group, kind, and name
metadata for root and tenant stores.
* Guest monitoring streams include pod name, container name, and
namespace fields.
* **Bug Fixes**
* Application metadata is added to container-log streams without
changing the existing stream fields for event and audit outputs.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
e2283b3758 |
feat(monitoring-agents): export PVC labels from kube-state-metrics (#5215)
## What this PR does Adds `persistentvolumeclaims=[*]` to the kube-state-metrics `--metric-labels-allowlist`, next to the existing `pods=[*]` and `deployments=[*]`. kube-state-metrics then publishes `kube_persistentvolumeclaim_labels`, one series per PVC, with the PVC's labels as `label_*`. Why: the per-application metrics scope in the design proposal [Tenant resource-consumption metrics](https://github.com/cozystack/community/pull/94) (Phase 0) finds an application's volumes through the `apps.cozystack.io/application.*` lineage labels on its PVCs. Without this series, volume usage (`kubelet_volume_stats_*`, which carries `persistentvolumeclaim` but not `pod`) cannot be attributed to an application. `[*]` rather than an explicit key list: kube-state-metrics has no prefix matching, so the three lineage keys would have to be enumerated, and the series count is the same either way (one per PVC). `[*]` stays consistent with the existing pods and deployments entries and does not silently miss a lineage key added later. PVCs are far fewer than pods, which already export all labels. #### How it was tested - helm-unittest for `monitoring-agents`: a new suite asserts the exact flag in the rendered kube-state-metrics Deployment. - Verified on a test cluster: `kube_persistentvolumeclaim_labels` is present and carries `label_apps_cozystack_io_application_name`. - Not verified: the memory impact on kube-state-metrics. #### Known limitation PVCs created from a StatefulSet's `volumeClaimTemplates` usually have no owner reference and no Flux label, so the lineage webhook gives them no lineage labels and this series cannot attribute them to an application. Fixing that belongs to the lineage webhook. ### Screenshots No UI changes. ### Downstream repositories - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: Downstream: no trigger matched. Candidate: none affected, because the diff touches only `packages/system/monitoring-agents/values.yaml` and a test in that system package; it adds a series but renames no telemetry metric, platform value, app field or CRD. ### Release note ```release-note feat(monitoring-agents): kube-state-metrics now exports PVC labels as `kube_persistentvolumeclaim_labels`, one series per PVC, so volume usage can be attributed to the application that owns it. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Kubernetes monitoring metrics now include labels from pods, deployments, and persistent volume claims. This expands the resource metadata available alongside metrics, including lineage labels used by the tenant metrics API. Metrics for these resource labels are emitted only for resources covered by the configured label allowlist. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
59f0013b1f |
fix(backups): attribute a MariaDB dump only when no other run can own it
Every backup of a MariaDB application writes its dump under one shared prefix, so the probe took the dump whose file-name timestamp (the backup pod's clock) and LastModified (the object store's clock) both fall in the operator Backup's lifetime, with a minute of slack for skew. That keeps a neighbour's dump out only while this run's own dump is in the window. With the pod clock more than the slack behind, the own dump is named before the window opens, and a previous run that finished within the minute before this one was created becomes the single match: a 123-byte dump was reported as 666 bytes. A pod clock ahead or an object store clock behind does the same with the next run, and an own dump that is gone by the time it is looked up, as it can be for a restore long after the backup, leaves a neighbour as the single match with no skew at all. A dump is now attributed only when ownership is proven: - it is the one dump that either clock places in the window. A dump that one clock places there and the other does not may be this run's own with a skewed clock, so it leaves any other match unproven, the way a second dump in the window already did, and is not taken itself; - both of its clocks place it in the window; - neither clock places it in the window of another operator Backup that writes under the same bucket and prefix and has completed. This rejects a neighbour's dump when this run's own is missing, which the clocks alone cannot tell. Operator Backups still running are not counted as neighbours: their window has no end, and one stuck short of Complete (a Job whose pod never starts stays JobRunning for good) would otherwise keep every later run of the application from being sized. A dump such a run writes inside this run's window still competes with this run's own. Narrower slack does not prove ownership: a neighbour that finished within the narrower slack still matches, and a smaller skew pushes the own dump out. The Job's start and completion times are stamped by the control plane, like the operator Backup's own, so they narrow the window by seconds without removing the dependence on the two clocks a dump carries. What is left: a neighbour's dump is taken only when this run's own is missing from the listing or misplaced on both clocks and the neighbour's operator Backup no longer exists or has not completed. Every other unproven case leaves the size unset, as two runs that overlap or follow each other within the slack already did. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
560c44f71e |
fix(backups): read a MariaDB dump from where its operator Backup wrote it
The size probe took the bucket, endpoint, prefix, credentials and TLS settings from the strategy as it renders on the reconcile that creates the Backup, which is not necessarily what it rendered when the operator Backup was created: an existing operator Backup is reused untouched, and a backup can run for up to the driver's 30-minute deadline. An admin who edits the strategy in between sends the probe to the new location, where it measures the wrong prefix or nothing at all. The operator Backup's own spec.storage.s3 is where the dump went, so the probe reads that. It is also the only source a restore can use, since it targets that operator Backup long after the strategy may have changed. An operator Backup that does not store to S3 is refused before anything is listed. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
21f51a2e74 |
refactor(backups): return the matched MariaDB dump, not only its size
A MariaDB restore has to name the dump it restores from: without a target time the operator restores the newest dump under the prefix, which is not necessarily the one of the Backup being restored. The size probe already finds that dump, but it returns only its size, folds a missing, an ambiguous and an unreadable listing into the same 0, and lives on BackupJobReconciler, which the restore reconciler is not. Split the probe in two: findMariaDBDump reads the bucket and returns the matched object (key, file name, the timestamp in its name, size and LastModified) or an error that tells "not found" from "ambiguous"; the size path keeps its timeout and reads only the size. The Secret lookup and the S3 client move onto an objectStore value that any reconciler can build from its own client, and BackupJobReconciler keeps one-line wrappers so the CNPG probe is unchanged. sizeBytes is reported exactly as before. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
c140354977 |
test(backups): build operator Backups with the operator's success shape
mariadb-operator marks a Backup Complete=True for a failed Job as well as a successful one; only the reason tells them apart (JobComplete with message Success, or JobFailed with message Failed). The fixtures that stand for a finished backup set Complete=True and nothing else, so they describe a state the operator never produces and would read as a failure to a driver that checks the reason. Give them the success shape the operator writes. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
92fa979690 |
fix(backups): trust only the endpoint CA where the backup writer does
With an endpoint CA the probe trusted the system roots plus that CA. barman-cloud gets endpointCA as AWS_CA_BUNDLE, which replaces the default trust store, so a CNPG probe could accept an endpoint the backup itself would have refused. Let the connection say whether the CA adds to the system roots or replaces them, and replace them for CNPG. mariadb-operator appends its CA to the system pool, so MariaDB keeps that. Pin in reconcile tests that a Create the apiserver rejects still fails the BackupJob, so only the read-back of an existing Backup is retried. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
080df1b2f2 |
fix(backups): surface a Backup that cannot be read back and skip its probe
A Backup that exists but cannot be read back is retried through the controller's backoff, which left the BackupJob Running with nothing on it saying why. Set Ready=False with reason BackupNotReadBack before handing the error back. When the check for an existing Backup itself fails, the Backup may well exist, so a size probe would only be discarded, and a read that keeps failing reran it, up to the probe timeout on the single BackupJob worker, on every retry. Skip the probe in that case and create the Backup without a size. Drive both behaviours through the MariaDB and CNPG reconciles in tests. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
523d74a503 |
fix(backups): retry instead of failing when an existing Backup cannot be read
When Create collides with a Backup an earlier reconcile already made and the read that follows fails, the drivers failed the BackupJob, although the backup and its Backup both exist. Return a distinct error instead, which the drivers hand back to the controller so the reconcile is retried with backoff and then finds the Backup. minio.New already refuses an endpoint with a path prefix and accepts a trailing slash, so drop the duplicate check in the shared client; the CNPG probe keeps its own check because it parses the URL itself and would otherwise drop the path before minio sees it. Say in the sizeBytes API doc what the value covers, such as the base backup without the shared WAL archive for CNPG, and that strategies which cannot attribute stored bytes to one backup leave it unset. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
2b21763577 |
fix(backups): do not fail a backup on a transient retry-check read
The check for a Backup an earlier reconcile already created returned its read error, and the drivers turn any error from that path into a terminally Failed BackupJob. An apiserver timeout right after the operator backup completed therefore lost a backup whose data was safely in S3. The check only saves a probe, and the Create that follows already handles an existing Backup, so log the error and go on to Create. A trailing slash on the S3 endpoint is not a path prefix; the operators accept it, so do not refuse it. Pin the uncached AlreadyExists refetch, the upper slack of the MariaDB window and the probe timeout with tests. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
a3d854739d |
fix(backups): attribute MariaDB dumps on two clocks and read retries uncached
The MariaDB probe identified this run's dump by the timestamp in its file name, which comes from the backup pod's clock. With that clock skewed past the slack, this run's dump falls outside the window while a neighbouring run's dump can be the single match left, and its size is reported as this backup's. A dump now has to fall inside the window by both its name and its LastModified, which the object store stamps, so a single skewed clock leaves the size unset instead. The name parser also accepts the layout later mariadb-operator releases write for compressed dumps, backup.<ts>.sql.<ext>, and checks the compression token in either layout, so an operator upgrade does not silently stop the measurement. The retry check and the AlreadyExists refetch read the Backup through the cache. A retry usually follows a Create made moments earlier; a cache that does not show it yet turned the refetch into NotFound and failed a backup that had succeeded. Read it from the API server instead. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
3b86f57803 |
fix(backups): refuse to probe an S3 endpoint behind a path prefix
An endpoint such as https://s3.example.org/prefix cannot be addressed by minio-go, which has no path-prefix option, and the prefix cannot be added below the client either, because the SigV4 signature covers the request path. The CNPG probe was worse than failing: it kept only the host of endpointURL and listed the bucket at a location barman never writes to. Reject such endpoints with an explicit error, in the CNPG probe and in the shared client, so the size is left unset instead of measured somewhere else. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
038f35376a |
fix(backups): do not report a partial listing or re-probe on retry
minio-go ends a listing whose context is cancelled between pages by closing the channel without an error item. When the probe timeout expired between pages of a large CNPG base backup, the pages already read were summed and reported as the measured size. Check the context once the listing channel closes and leave the size unset instead. The probe ran before the Backup was created, so a reconcile that retried after the Backup already existed probed the bucket again, held the single BackupJob worker for up to the timeout, and then discarded the result in favour of the existing object. Return an existing Backup before probing. Log the ambiguous MariaDB case (several dumps in the window) at info like the other reasons the size stays unset, and cover the session-token credential path with a test. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
f222e52863 |
fix(backups): harden the artifact size probe
The probe runs inside the reconcile that creates the Backup, and the BackupJob controller has a single worker, so its timeout delays every other BackupJob while an endpoint hangs. Lower it from 30s to 10s. List the MariaDB prefix non-recursively, as mariadb-operator does. The default MariaDB prefix and the default CNPG destinationPath are the same <namespace>/<name>/ path, so a recursive listing of a MariaDB backed up next to a same-named Postgres walked the whole WAL archive and ran out of time. The CNPG base-backup prefix is still listed recursively. Build the custom-CA transport from minio.DefaultTransport so it keeps the proxy-from-environment and dial/TLS timeout settings a bare http.Transport drops, and cover it with a test against a TLS endpoint with a self-signed certificate. Log the reasons the size could not be measured (credentials, client, listing, empty CNPG prefix) at info. The chart runs the controller at info, which hid them before. The expected ambiguity of several MariaDB dumps in the window stays at debug. Skip an endpointCA without a Secret name, as buildBarmanObjectStore does. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
de60488a2c |
fix(backups): report sizeBytes for CNPG backup artifacts
CNPG backups arrived without Backup.status.artifact.sizeBytes: cnpg.io/Backup status carries no size, and the data lives in barman's object-store layout rather than in a single object. barman-cloud uploads every base backup under <destinationPath>/<serverName>/base/<backupID>/ (backup.info plus the data and tablespace tarballs, possibly split into parts). On completion the driver now lists that prefix with the credentials and endpoint CA of the rendered barmanObjectStore and records the total. The WAL archive is not counted: it is one stream shared by every base backup of the server. The field stays unset when the backup ID is unknown, the bucket cannot be read, or nothing is stored under the prefix. The S3 client and Secret lookup are now shared with the MariaDB driver. Refs #4316 Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
d4ad5612d3 |
fix(backups): report sizeBytes for MariaDB backup artifacts
Backup.status.artifact.sizeBytes was only ever set by the etcd strategy, so MariaDB backups always arrived without a size. mariadb-operator writes each logical dump as <prefix>/backup.<RFC3339>[.<compression>].sql under a prefix shared by every backup of the application, and the URI the driver records is synthetic, so there is no single object to HEAD. On completion the driver now lists the prefix with the tenant credentials the operator Backup was rendered with and takes the size of the one dump whose timestamp falls inside the operator Backup's lifetime. When the bucket cannot be read or the window does not match exactly one dump, the field stays unset. The API doc now states that an absent sizeBytes means "unknown", and the Velero strategy documents why it leaves the field unset: kopia stores data content-addressed across backups, so there is no per-backup stored size. Refs #4316 Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
6e36385e00 |
feat(api): expose HelmRelease failure state on Application status
Tenants usually read Applications but not the HelmReleases behind them, and only Ready and Released were copied over. A client could therefore see that a release was failing but not how often it had failed, nor whether helm-controller had stopped retrying it. Copy the Stalled condition, which helm-controller sets once it stops reconciling a release (remediation retries exhausted, rollback target missing, invalid CEL, access denied), and surface the install and upgrade failure counters with the last attempted release action. Stalled is dropped when it was observed for an older generation than the HelmRelease's current one: the Application carries no generation, so after a spec update a client would otherwise read the previous spec's failure as terminal for the new values. With the RetryOnFailure policy the platform generates today, a failed release is retried rather than stalled, so the counters are the signal. That does not hold for every state: a release missing from storage while an install failure count is retained still stalls, and a release on a remediation policy stalls once its retries run out. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
47b497194e |
fix(backups): restore the ClickHouse archive a Backup was taken as
The Altinity strategy named each archive inside its Pod with date -u and recorded nothing on the Backup, so a restore could only guess: it listed the remote archives under the application's prefix and took the newest. A RestoreJob of an older Backup therefore restored whichever backup had run last. The controller now names the archive before it creates the backup Job, from the BackupJob's persisted StartedAt, so every reconcile and every retried Pod of the Job writes the same one. The name is recorded in the Backup's driverMetadata next to the strategy parameters, and a restore renders it into the strategy Pod, which restores exactly that archive and fails, naming it, when remote storage no longer holds it. The name is recorded only when the Job's Pod was given it, so a Job created by an earlier controller, or a strategy that names its own archives, leaves the Backup without one instead of pointing it at an archive that does not exist. Because a retried Pod reuses the name, it waits on an attempt the sidecar is still running instead of being refused with 423, and asks for resume, so a local or partial remote backup an earlier attempt left is completed rather than refused as existing. Backups taken before names were recorded stay restorable: the Pod picks the newest archive whose name is stamped no later than the Backup's takenAt, which is set only after the backup Job finished, and logs a warning that the choice is a best effort. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
7b7444feaf |
fix(kubernetes): move the OIDC users out of the bootstrap Job's script
The bootstrap Job wrote a ClusterRoleBinding per oidc.users entry into its script, and the script reaches the container as one `sh -c` argument. Linux caps a single argument at 128 KiB, which about 240 users cross at roughly 530 bytes each. The container then cannot start, the post-install and post-upgrade hook fails, and the tenant cluster's HelmRelease stays unready. The values schema puts no bound on oidc.users, so a shared cluster reaches that size with ordinary per-person entries. The bindings and the list of their names now reach the Job through a ConfigMap it mounts, so the script is the same size for any list. Names, labels and roles of the bindings are unchanged, so the bindings already in a tenant cluster are updated in place and none is orphaned. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
a2d0e4cb50 |
fix(cilium): pass the CiliumNode list to jq on stdin in the IPAM hook
The hook fed the whole `kubectl get ciliumnodes -o json` to jq as one argument. Cilium copies every Node label onto its CiliumNode, 22 to 30 KiB per node on Talos or with NFD and GPU labels, so the list passes the 128 KiB Linux allows a single argument within six nodes. jq then fails to start with "Argument list too long", the hook Job fails before it moves any node, and the cilium HelmRelease loops on UpgradeFailed with the HelmReleases behind it stuck in DependencyNotReady. The hook renders on every upgrade of cilium in cluster-pool mode, so a cluster that already finished the migration hits the same failure on its next values change. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
95dc591198 |
test(kubernetes): keep application identity out of guest log streams
Guest pods are tenant-controlled, so their lineage labels are not a trustworthy application identity, and the platform must not promote them to the stream fields the parent scopes application logs by. The guest release replaces the monitoring-agents filters and outputs wholesale, which is what keeps the parent's label lift out of guests; pin that so a later change to the guest overrides cannot bring it in. This only covers what the platform ships. A guest admin can still edit the guest's fluent-bit or write to the parent's vlinsert directly; that needs a write-path control, which the logs proposal leaves open. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
e5f1714836 |
feat(monitoring): keep application identity on container log lines
A reader that scopes logs to one application needs the owning
application on every line, and the lineage webhook already puts it on
the pod as apps.cozystack.io/application.{group,kind,name}. Fluent Bit
fetched those labels and then dropped them with the rest of the
kubernetes map, so the store could only be scoped by namespace.
Keep the three labels as apps_cozystack_io_application_{group,kind,name},
the names the metrics side uses, and make them stream fields on every
container-log output so the scope is an indexed stream filter. A pod
without the labels produces no such fields, and VictoriaLogs builds the
stream from the fields present, so existing streams keep their
identity; a labelled pod starts a new stream once.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
|
||
|
|
7de70ac2fc |
feat(monitoring-agents): export PVC labels from kube-state-metrics
The tenant metrics API scopes a query to one application by looking up its pods and PVCs through their apps.cozystack.io/application.* lineage labels. kube-state-metrics emits kube_persistentvolumeclaim_labels only for resources on --metric-labels-allowlist, so without PVCs there the PVC half of that lookup always comes back empty and volume series of an application fall out of its scope. All labels are allowed, as for pods and deployments: kube-state-metrics emits one series per PVC whatever keys are listed, and PVCs are far fewer than pods, which already export every label. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
4087218a02 |
docs: move the repo-only guides onto the website (#5034)
## What this PR does Seven guides under `docs/` were operator facing and written alongside the code, and left in the source tree where no user looks. They are on the website now, so this deletes them here and repoints every reference. ``` docs/operations/backup-classes.md -> operations/services/backup-classes docs/operations/resource-presets.md -> guides/resource-management docs/storage-immutability.md -> operations/storage-class-immutability docs/operations/system-memory-limits.md -> operations/configuration/system-memory-limits docs/gateway-classes.md -> networking/gateway-classes docs/observability/distributed-tracing.md -> operations/services/monitoring/distributed-tracing docs/operations/seaweedfs-capacity-for-backups.md -> operations/services/object-storage/capacity-for-backups ``` `docs/release.md`, `docs/deprecating-applications.md` and the two SeaweedFS runbooks in `docs/operations/` stay. A workflow, `AGENTS.md`, a migration and a bats suite read them by path, which is the test from #3777. References point at `https://cozystack.io/docs/v1/...`. The site rewrites that short prefix to the latest version in `layouts/404.html`, so the links follow releases and pin none. Backup Classes, resource management and `storageClass` immutability exist in the latest version already. The other four pages are only in `next/` until 1.7.0 is promoted, and their links start to resolve then. The generated mongodb files come from `values.yaml` through `make generate` with cozyvalues-gen v1.7.0, the version pinned in `.github/workflows/pre-commit.yml`. Tests: helm unit tests for mongodb (151), postgres (209), vm-disk (114) and kubernetes-nodes (223), go tests for backupcontroller, presets and the operator, rd-presets-check, the migration-50 bats suite. ### Downstream repositories - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: cozystack/website#763 Walked the rest of the trigger map against the diff. No provider follow-up: the mongodb `values.schema.json` diff is one `description` string, with no field added, removed or renamed, no enum, no default, and no `kind`, `plural` or `release.prefix` change. `packages/core/platform/values.yaml` changed in comments only and no key, so the hand written platform-package table on the site is unaffected. Please merge after cozystack/website#763, the links in the code only resolve once those pages deploy. ### Release note ```release-note docs: seven guides move from the docs/ directory of this repository to cozystack.io (Backup Classes, resource presets, storageClass immutability, system component memory limits, GatewayClass selection, distributed tracing, SeaweedFS capacity for backups), and in-repo references point there. ``` |
||
|
|
e31f518df7 |
ci(api-review-gate): publish the gate as a commit status (#3440)
## What this PR does Publishes the `API Review Gate` merge context as a commit status, so a skipped or cancelled run cannot pass it or poison it, and gives fork PRs the same gate through a privileged `workflow_run` workflow. ### Why The merge context was a job name. A required job that skips counts as passed, and check runs are not deduplicated per name, so one cancelled run on a head keeps the context failed until the next push. Measured on this repo: `41d1153c` has four check runs with the gate's job name (one `cancelled`, three `success`). Commit statuses are deduplicated per context and the newest wins, so a run with no verdict posts nothing and the previous status stays. ### What changes Same-repo PRs get status from `api-gate-report`. A `pending` opener marks the head before the evaluation on approve, dismiss and request-changes reviews, so an old `success` is not mergeable for the minute the evaluation takes. `COMMENTED` reviews no longer count as a reviewer's latest review, so an owner who approved and then commented keeps the approval. Concurrency is split, so bot comment run cannot cancel live evaluation. Fork PRs: `api-review-gate-fork.yaml` runs from the default branch and reaches the verdict itself, the fork's run is only a trigger. `resolve` finds PR by head SHA in open PR list (review runs on a fork report base repo as head repository and have no pull requests). `detect` fetches the head as data and runs the detector in a job with `contents: read` only. `conclude` applies the approval rule and posts. A sweep every 15 minutes moves the status of fork PRs whose runs wait for a maintainer in `action_required`, those never fire `workflow_run`. This is lllamnyp's patch, applied whole, with these changes on top: - two open fork PRs on one head fail closed in `resolve` and in the sweep, a head carried by a same-repo PR stays with `api-gate-report` - detector is built from default branch only, merge base is picked by PR author and an old one would mean an old detector - the fork `detect` step runs under `bash -e` and detector exit 2 ended it before the code was read, so it has `set +e` like the same-repo job - head is checked out with links as links, like the same-repo gate does, because flattening them hid a CRD behind a file link from the detector while Helm still ships it ### Limits - A fork PR whose push run waits for a maintainer's approval has no status until that run executes, same as `E2E Tests`. - A fork PR on a head that a same-repo PR also carries shows the verdict standing there, one commit status cannot hold a verdict per base. - The sweep can publish a stale `success` if it reads an approval, stalls, and the owner withdraws it meanwhile. The next sweep flips it back, so the window is up to 15 minutes. - Two copies of one first-party CRD and CRDs behind directory links are gaps of the detector on main, same for same-repo PRs, they are in https://github.com/cozystack/cozystack/issues/5170. ### Testing `actionlint` is clean on both workflows. `zizmor` is clean too, but `dangerous-triggers` and `template-injection` are disabled repo-wide in `.github/zizmor.yml`, so that says little about a privileged workflow. I ran the `github-script` bodies under node with a mocked API (every conclusion, approval and shared head case) and the run steps under `bash -e` with a stub detector exiting 0, 2 and 1. `make unit-tests` passes with `LC_ALL=C` (a kafka bats test depends on the locale, nothing to do with this PR). Not exercised live: the fork path and the sweep run only from the default branch, so their first run is after merge. ### Repo admin Branch protection requires the old job name `Require API owner review for sizeable API changes`. This PR renames that job, so after merge the required context has to be changed to the `API Review Gate` status. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note NONE ``` |
||
|
|
a2cc9a3016 |
fix(dashboard): link a tenant row to its detail page (#4004)
## What this PR does Fixes #3822 Tenant list showed name as plain text and the only link in a row was Edit button, so tenant detail page with its tabs and Delete was reachable only by opening edit form and cancelling out of it. Name cell is a link to that page now, it appears exactly where Edit appears and points to the same CR. Plain click switches selected tenant to the row's parent first, same as Edit does, because detail page takes namespace from selected tenant. Ctrl, cmd and shift click don't run that switch, so the tab you open a tenant from stays where it was. First commit fixes what the link depends on. Console recovers a tenant's parent from namespace name (`tenant-a-b` sits under `tenant-a`), but root is the one level that naming skips: tenant `acme` ordered in `tenant-root` owns `tenant-acme`, not `tenant-root-acme`. So `realParentNamespace` returned undefined for every top level tenant, which is most tenants on a normal cluster, and Edit button has been missing there for as long as it exists. Ancestor labels carry the parent that the name does not. Root level tenant named `root-x` resolves to CR `root-x` now, not `x`. ### Notes I first tried to make links carry their tenant in the url (`?tenant=`), so a detail page opened in new tab resolves on its own. That turned into questions about tenant context itself: what to do with a tenant in a link the user can't see, what localStorage may hold, and TenantNamespace LIST ignoring `labelSelector`. Detail route addresses a Tenant CR by relative name and takes namespace from selected tenant, and making that robust is a refactor of tenant context and routing, which should start with study and planning, not with patches on patches. So this PR only closes the UX gap (no way from tenant list into a tenant's page), going deeper is out of scope. Known limit that comes with it: link has no tenant in it, so opening it in a new tab (ctrl/cmd/middle click, open in new tab, copied url) resolves against the tenant last selected in any tab, because selection lives in shared localStorage. For a row directly under that tenant it opens the right page. For a row deeper in the tree the page says `Application <name> not found.` and there is no Delete. If that tenant also has a direct child tenant with the same relative name, new tab opens that other tenant's page with working Edit and Delete, and delete confirm says only `Delete Tenant "<name>"?`. Plain click in the same tab is not affected. Detail page's own tabs and Edit link already behave this way, it's how detail route resolves its namespace today and fixing it properly is the refactor above. A row links, and offers Edit, exactly when namespace holding its Tenant CR is one the apiserver returns to this user. It returns a tenant namespace only if the user is subject of a RoleBinding in it, so sub-tenant admin that cant act in `tenant-root` sees no link there. No row level Delete button. Issue allows either that or a route to detail page, I took the route so confirm and mutation stay in one place. Covered by tests, every source change here fails a test when reverted: - top level tenant, nested one, root level tenant named `root-x`, forest root without ancestor labels - bridged row whose real parent is invisible gets neither link nor Edit - plain click switches to the row's parent and opens detail page - ctrl click leaves selection and location alone ### Screenshots Tenant list under root on a dev cluster. Every row name is a link to tenant page now (`pr4078` is hovered), top level tenants had neither link nor Edit before. Host column redacted. <!-- SCREENSHOT: drag in 01-tenant-list-links.png here --> <img width="1440" height="900" alt="01-tenant-list-links" src="https://github.com/user-attachments/assets/39cd723a-1ac5-4d3b-8c36-b70ddc626961" /> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: All changed files are under `packages/system/dashboard/images/console`, console has no downstream copy (standalone UI repo is archived). ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(dashboard): tenant name in the tenant list links to its detail page now (where Delete is), and top level tenants get their Edit button back. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * Root-level tenant names are now displayed without incorrectly removing prefixes, and tenants ordered directly under the root are correctly recognized in the hierarchy. * Tenant rows link to their detail pages when the parent tenant is accessible. Same-tab navigation updates the active tenant, while opening a link in another tab leaves the current selection unchanged. * Edit links and navigation now use consistent tenant routes. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
b5e3b37a9a |
ci(api-review-gate): build the fork detector from the default branch
The merge base is the commit the fork's author branched from, so a branch cut from an old commit selected an old detector, and a fix to the detector never reached the PRs an attacker controls. The workflow itself comes from the default branch, so the detector now does too. The head is also checked out with its links intact. Flattening them hid a CRD behind a file link from the detector while Helm still shipped it, and made the verdict differ from the one the same-repo gate reaches. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
d32092ac9a |
feat(release): attach an SBOM to every release image (#4997)
## What this PR does
This PR attaches an SPDX SBOM to every first-party image of a release.
BuildKit adds it at build time as an attestation, so it sits in the
image's index and moves with the digest the release pins, through
promotion. `docker buildx imagetools inspect <image>@<digest> --format
'{{ json .SBOM }}'` prints it, one document per platform. Releases cut
from this change on have it. The Talos installer image (built by the
imager, pushed by skopeo) and the third-party images don't.
It is more than `SBOM: '1'` in `tags.yaml`, because three things broke
on the way:
- An image built with an attestation is an index of the image and an
attestation manifest. `image_arch` in `hack/stitch-multiarch.sh` read
that as not one amd64 and one arm64 image, so the stitch skipped every
such image and still exited 0, and `hack/verify-multiarch.sh` then
failed the release with "index without linux/arm64". `image_arch` and
the rerun check look through the index now, and the stitched index keeps
both attestations.
- The release `make build` ran on the default docker driver, which
cannot attach attestations. It runs on the docker-container builder with
the docker.io mirror now, like the arm64 leg. `use: false` stays, so a
plain `docker build` keeps the default driver.
- `--sbom=true` pulls `docker/buildkit-syft-scanner:stable-1`, a
floating tag. `SBOM=1` passes `--attest type=sbom,generator=` with the
scanner pinned by digest (1.12.0). The comment in `hack/common-envs.mk`
called the output CycloneDX, BuildKit writes SPDX, and the same word in
`docs/security/self-assessment.md` is fixed too.
Only the release lane changes. PR, main and nightly builds push to OCIR
and are not what users install, so the OCIR attestation probe the
tracker (A2 in #3278) listed is not on this path, release images go to
GHCR.
### How it was checked
- Bats 1.14.0: four new tests in `hack/stitch-multiarch_test.bats` (the
arm64 leg's workflow shape, a stitch of SBOM images, a rerun after one,
an index of two linux images that is skipped), two in
`hack/common-envs_test.bats` (the flags, and the Renovate manager held
to the `SBOM_SCANNER` line), and the matchbox workflow test rewritten
for the new builder. The stitch tests fail on `main`'s
`stitch-multiarch.sh`, and each of these mutations fails one: dropping
the `length == 1` check, counting the attestation as a platform, looking
for the amd64 index itself in the rerun, taking `SBOM`, `BUILDER` or the
mirror config out of `tags.yaml`, going back to `--sbom=true`, dropping
the scanner digest, turning `SBOM` on by default, reformatting the
`SBOM_SCANNER` line out of the Renovate manager's regexp.
- A rehearsal on a localhost registry (docker 29.8, buildx 0.37.1) with
the real scripts. Images built through the `BUILDX_ARGS` of
`common-envs.mk` for amd64 and arm64, one without SBOM beside them, then
`stitch-multiarch.sh` (2 stitched, 0 skipped), `verify-multiarch.sh` (0
failed), `imagetools inspect` reading SPDX 2.3 with 15 packages per
platform from the stitched index, and `skopeo copy --multi-arch all`
keeping the index digest. `containerimage.digest` in `--metadata-file`
equals the pushed index digest, so the digest the package Makefiles
stamp is the index. The same run with `main`'s stitch skips the SBOM
image, exits 0 and fails the gate.
- `make bats-unit-tests`: 2444 of 2445. The one failure, "all-topics
backup skips double-underscore internals and keeps single-underscore
user topics" in `hack/kafka-topic-backup-strategy.bats`, also fails on
unmodified `main` on the same machine. `make bats-posix-compat-tests`
passes, and the stitch test file passes under dash through
`cozytest.sh`. `hack/check-commit-trailers.sh` passes. zizmor and
actionlint aren't installed here, so the workflow change (two env keys
and one renamed builder step) is covered only by the tests above.
- Not checked: that GHCR takes the attestations, what retention does to
the child manifests, the fork lane's OCI export (it doesn't set `SBOM`),
and a real release build. The first alpha cut after the merge tests the
first and the last. Renovate itself did not run on the new manager: the
config passes the Renovate JSON schema, and the regexp takes the tag and
the digest off the live line, which a test holds.
### Decisions for the release owners
- Scope is `tags.yaml` only. SBOMs on main and nightly images would need
OCIR to take attestations (not probed) and would change what the nightly
mirror copies.
- The scanner is 1.12.0 with syft 1.51.0 inside, which is in the range
of GHSA-mw2c-m758-9v5q (memory exhaustion parsing binary inputs, fixed
in syft 1.52.0). It reads images our own builds produce, so I didn't
hold the change for it. A Renovate manager on the `SBOM_SCANNER` line
moves the version and the digest together when the scanner ships a newer
syft.
- The scanner adds a step per image and I didn't time it on the release
runners.
- A stitch rerun reads the amd64 index the tree pins, which stays in the
registry untagged after a stitch. If the registry drops untagged
manifests before the rerun, it fails at that read rather than repinning.
- After the first alpha with attestations I'll move A2b in #3278 and the
SBOM item in #2937 to done. The SBOM row in
`docs/security/self-assessment.md` still says SBOMs aren't published,
and stays until that alpha shows they are.
### Screenshots
<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->
### Downstream repositories
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
I walked the trigger map against the diff. It touches `tags.yaml`,
`hack/common-envs.mk`, the stitch and verify scripts and three docs. No
release asset is renamed, `hack/package.mk` and the commit convention
are untouched, and the flow a maintainer follows to cut and promote a
release is the same.
### Release note
```release-note
feat(release): every first-party container image of a release carries an SPDX SBOM as a BuildKit attestation in its index, readable with `docker buildx imagetools inspect <image>@<digest> --format '{{ json .SBOM }}'`. Releases cut from this change on have it, the Talos installer image and third-party images don't.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
- **New Features**
- Supported release image builds now include SPDX-format software bills
of materials (SBOMs) as attestations.
- Multi-architecture image stitching, retagging, and mirroring preserve
SBOM attestations.
- **Documentation**
- Updated image-reference guidance to explain how to handle attestation
entries in multi-architecture indexes.
- Clarified which images include SBOMs, how to inspect them, and that
SBOM generation remains disabled by default outside configured release
builds.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
94ada11eef |
fix(marketplace): pin a tapped repository to a digest instead of a mutable tag (#5082)
## What this PR does A tapped (community-connected) repository followed a mutable OCI tag: the Flux `OCIRepository` carried only `spec.ref.tag` with a 5-minute interval, so whoever controlled the tag controlled what ran in the management cluster within that interval, down to the HelmReleases of already-running tenant application instances. Fixes #4979. A tap is now pinned to a digest. `cozypkg tap` and the connect API accept an explicit `oci://host/org/repo@sha256:...` reference; a tag reference is resolved and pinned to its digest by the tap materializer once the artifact for the current spec generation is ready, from the digest Flux reports in `status.artifact.revision`, with the tag kept informational. A tap connected before this change is pinned to its current digest on the first reconcile after upgrade, and a re-connect to a new tag waits for source-controller to re-resolve before pinning, so a stale revision is never pinned. Moving the tag upstream no longer changes anything in the cluster; an explicit re-connect moves the pin. Connect also accepts an optional keyless cosign verification policy (`--certificate-identity` / `--certificate-oidc-issuer` on the CLI, `spec.verify` on the API), anchored to an exact identity so a superstring cannot match, and preserves it, and the pull secret, across repeat connects rather than silently dropping them. With those flags the CLI verifies the signature at tap time and pins the verified digest before pulling, so the content it applies is what a trusted identity signed; it fails closed if the verified digest cannot be determined. ### Screenshots Not applicable: no UI changes. ### Downstream repositories I walked the trigger map against the diff. The change is confined to `cozypkg`, the `Tap` aggregated API (a virtual, cluster-scoped resource with no CRD), the tap materializer, and the community-index docs. It does not touch `packages/apps`, `packages/core/platform/values.yaml`, installer values, platform variants/components, release assets, `ApplicationDefinition` semantics, the `hack/` make targets, or the commit convention, so the provider, ansible, ccp and external-apps paths are not reached. One path is arguable and needs a human decision, so I have left every box unticked rather than tick or dismiss it: - `cozystack/website` — `cozypkg tap` gains flags and digest-pinning behaviour. The marketplace tap flow is new and I am not sure it has a reference page on the site yet, so whether a docs follow-up is warranted (and where) is a maintainer call. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(marketplace): a tapped repository is pinned to a digest instead of following a mutable tag, so moving the tag upstream no longer changes what runs in the cluster; connect accepts oci://...@sha256:... and an optional keyless cosign verify policy (anchored to an exact identity and preserved across re-connects), and with verify flags the CLI verifies the signature and pins the verified digest at tap time. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit - **New Features** - OCI repositories can be connected using a SHA-256 digest, with or without a tag. Tag-based sources are pinned to the resolved artifact digest. - Optional Cosign keyless verification is available during connection using certificate identity and OIDC issuer details. - Reconnecting preserves existing verification and pull-credential settings when replacement values aren’t provided. - **Bug Fixes** - Reconnecting with a new tag clears previous operator-managed pins so the requested tag can resolve to its current artifact. - **Documentation** - Clarified tag pinning, signature verification, and where tapped app charts run. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
ccc216652e |
fix(monitoring-agents)!: isolate merged log payload under log_processed (#5167)
## What this PR does Fluent Bit's `kubernetes` filter runs with `Merge_Log On` but without `Merge_Log_Key`, so a parsed JSON log line is merged into the record root. An application that logs JSON can thereby set top-level fields that later pipeline stages trust for routing and tenant isolation: `kubernetes_namespace_name` (rewrite_tag routing and the `_stream_fields` stream identity), `kubernetes.labels`, and the `tenant` / `cluster` / `log_source` markers. A tenant workload can forge another tenant's values on its own log lines. Setting `Merge_Log_Key log_processed` nests the parsed payload under a dedicated key, so application log content can no longer override the top-level service fields. The change is applied to both Fluent Bit configurations that carry this filter: the platform agent (`packages/system/monitoring-agents/values.yaml`) and the guest-cluster agent (`packages/apps/kubernetes`). ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff, file by file. The change touches only the Fluent Bit filter config inside `monitoring-agents`; it does not alter package layout, any `values.schema.json`, platform values keys, `ApplicationDefinition`, any CRD, `hack/`, namespaces, telemetry metric names, or the commit convention. None of the listed couplings match. - [x] No downstream repository is affected by this change ### Release note ```release-note fix(monitoring-agents)!: Fluent Bit now nests parsed JSON log content under a `log_processed` key instead of merging it into the record root. This prevents an application log line from forging the service fields used for log routing and tenant isolation. BREAKING: top-level fields previously extracted from JSON logs (for example `level`, `msg`, and any application-specific keys) now live under `log_processed.*`; external dashboards and LogsQL queries that read them from the record root must be updated. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Improvements** * Merged Kubernetes log content is now grouped under `log_processed` in Fluent Bit output. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
7f55ff5d85 |
fix(dashboard): link a tenant row to its detail page
The tenant list rendered the name as plain text and put the row's only link on an Edit button, so nothing in the list reached the detail page that every other kind gets, tabs and a Delete action included. The only way there was to open the edit form and cancel out of it. The name cell becomes a link wherever Edit is offered, to the same CR. The detail page reads the namespace from the tenant context, so a plain click aligns the context to the row's parent first, the way Edit already does. React runs that handler for a ctrl-, cmd- or shift-click too, which the browser turns into a new tab or window, so it runs only for the click that navigates its own tab; otherwise opening a tenant beside the current one would move the tab left behind as well. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
e44e41d307 |
fix(dashboard): resolve the Tenant CR of a tenant ordered in the root
A tenant's CR lives in its parent's namespace, and the console recovered the parent from the namespace name: `tenant-a-b` sits under `tenant-a`. The root is the one level that naming skips. A tenant `acme` ordered in `tenant-root` owns `tenant-acme`, not `tenant-root-acme`, so the prefix test never matched a top-level tenant and `realParentNamespace` returned undefined for it. That is most tenants on a normal cluster, and the Edit button, offered only where the CR's namespace resolves, was missing for all of them. The ancestor labels carry the parent the name does not, so the lookup falls back to `tenant-root` when the chain names it. Deriving the CR name keys on the parent being the root rather than on the child's name happening to start with it, which keeps a root-level tenant called `root-x` from resolving to the CR `x`, and the row label agrees with it rather than showing `x`. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
dfa2283a13 |
ci(api-review-gate): close a standing success on a shared fork head
A commit status belongs to the commit. When two open fork PRs carry the same head, a success already standing on it may have been earned by one of them on another base, or may rest on an approval that was withdrawn on a PR whose run waits for a maintainer and so never reaches resolve. The sweep skipped such a head and left that success in place, which let a duplicate fork PR keep a withdrawn approval from taking effect. The sweep now writes the same failure resolve writes, once per head, when a success set by a workflow stands on a head that only fork PRs carry. It still never creates a status, and a head that an open same-repo PR carries stays with api-gate-report. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
0538e70591 |
ci(api-review-gate): let a sizeable fork change reach the approval check
api-gate exits 2 to report a sizeable change, and the runner runs a step without an explicit shell as bash -e. The fork detect step switched errexit off only after reading the exit code, so the first failing command ended the step with 2 before sizeable=true was recorded. Every fork PR with a sizeable API change then got "gate evaluation did not complete", which the sweep deliberately ignores, so an owner's approval could not turn it green by either path. The same-repo step already switches errexit off before the call. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
878f569e61 |
ci(api-review-gate): leave a shared head to its same-repo publisher
A commit status belongs to the commit, so two open PRs that carry the same head share one "API Review Gate" status. The fork workflow attributed that status to a fork PR by head SHA alone. A fork PR opened at a commit of this repository could therefore overwrite the verdict of the same-repo PR that owns it: resolve wrote a failure after api-gate-report had posted success, and the sweep applied the fork PR's reviews to a status the other PR had earned. A fork can reference any commit of the repository's network, so anyone able to open a PR could hold another PR's gate red. The fork workflow now leaves a head that an open same-repo PR carries to api-gate-report, and the sweep skips any head carried by more than one open PR. Two fork PRs at one head still fail closed in resolve, since their bases, and so their verdicts, can differ. The header of the same-repo workflow said skipped and cancelled runs post nothing. That stopped being true once a review posts pending ahead of its evaluation, so it now says they post no verdict. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |