Commit Graph
8019 Commits
Author SHA1 Message Date
Aleksei Sviridkin 2cf80c02a4 fix(clickhouse): keep failed backup attempts out of the bucket (#5229)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

A ClickHouse BackupJob attempt that fails during upload leaves its local
backup on the data volume, and with the upstream defaults the
clickhouse-backup sidecar never prunes local backups and resumes every
unfinished upload when it starts. On the next sidecar restart the failed
attempt is uploaded to the bucket as an archive no Backup refers to, and
while that resumed upload runs the sidecar answers new operations with
423, which burns the strategy Job's retries in seconds. The strategy
itself only waited for the sidecar API to respond and then started its
operation regardless.

The sidecar now runs with `API_COMPLETE_RESUMABLE_AFTER_RESTART=false`,
so a failed attempt is never published, and `BACKUPS_TO_KEEP_LOCAL=-1`,
so at most the newest local backup is kept and it is removed once its
upload succeeds. Backups already left on existing volumes are cleaned up
by the first backup after the upgrade. The Altinity strategy now waits
until the sidecar has no operation in progress before it starts a backup
or a restore, and fails the attempt with a clear message instead of
proceeding when the sidecar stays busy or unreachable.

Waiting for an idle sidecar changes what a retried strategy Pod does.
The Job retries a failed Pod up to twice, and a Pod can die after its
operation was accepted (a dropped poll under errexit, an eviction, a
node drain) while the sidecar keeps running it. Before, the retry hit
423 and failed; with the idle gate it would wait for the running
operation to finish and start it again. For a restore that is
destructive, because `restore_remote` drops every table it restores
before recreating it, so the second run discards whatever was written
after the first one finished. A retried restore therefore takes over a
`restore_remote` of the same archive that is still running and waits for
its result instead of starting another. Once an operation is accepted,
polls tolerate up to six failures in a row before the attempt fails, and
the sidecar losing the operation from its in-memory action log, or
ending it cancelled, fails the Pod instead of leaving it polling.

A retried backup works from the archive name the controller records for
its BackupJob. That name is now `<application>-<YYYYMMDDHHMMSS>-<first
12 hex digits of the BackupJob UID>`, so two runs of one application
started in the same second, as two Plans on one schedule are, never
share an archive. It is exactly as long as the previous
`clickhouse-<application>-<YYYYMMDDTHHMMSSZ>`, so an `s3PathOverride`
prefix that fit before still keeps every object key within the 1024-byte
limit. A retry takes over a `create_remote --resume` of its archive that
is still running, counts an upload of its archive whose latest run ended
in `success` as its own result instead of uploading the archive again,
and otherwise resumes the upload. Restoring a Backup that records no
archive still picks an archive by timestamp, and that path accepts only
`clickhouse-<application>-<timestamp>` names, so it never picks a
per-run archive.

One limit remains. A finished `restore_remote` row is not taken over:
the sidecar never trims its action log, so a `success` row of the same
archive may be left by an earlier RestoreJob, and counting it as this
one would report a restore that never ran. A retry that arrives after
the restore it replaces had already finished, but before that Pod saw
the result, therefore still runs the restore a second time.

Upgrading changes the ClickHouseInstallation pod template, so ClickHouse
pods with `backup.enabled` are rolled once. A backup Job started before
the upgrade by a controller that named archives the previous way is not
re-rendered and keeps writing that name, so the controller records the
previous name when the new one is not on the Job. The local copy of the
latest failed attempt stays on the volume until the next backup. Objects
partially written to the bucket by an upload that fails midway are not
removed by clickhouse-backup and are left for a follow-up.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(clickhouse): failed ClickHouse backup attempts are no longer uploaded to the bucket when the backup sidecar restarts, local backups no longer accumulate on the data volume, the backup strategy waits for the sidecar to be idle instead of failing with HTTP 423, a retried ClickHouse restore takes over the restore that is still running instead of starting it a second time, and a retried ClickHouse backup takes over or reuses its own upload instead of starting a new one. Every ClickHouse backup run now writes an archive of its own, even when two runs start in the same second, under a name no longer than before. ClickHouse pods with backups enabled are restarted once on upgrade.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Backup and restore operations wait for the backup service to become
idle and join a matching operation already in progress rather than
starting a duplicate.
* Restore operations use the archive the Backup records and fail when it
is not in remote storage; only a Backup that records no archive falls
back to the newest eligible archive by timestamp.
* New backups use distinct archive names, including for runs started in
the same second.
* Local backup copies are discarded after upload, and completed uploads
are not resumed after a service restart.
* **Bug Fixes**
* Operations now fail when the service cannot be queried, an operation
disappears from its activity log, or it ends with an error or
cancellation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 08:58:33 +03:00
Aleksei Sviridkin bc268ce759 fix(monitoring-agents): let the release uninstall when the VictoriaMetrics operator is gone first (#5152)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Disabling the Monitoring Agents addon of a tenant Kubernetes cluster
leaves `<cluster>-monitoring-agents` (release `cozy-monitoring-agents`,
installed into the tenant cluster) in `UninstallFailed` for good. Seen
in production on 1.6.5 and still reproducible from `main`.

`monitoring-agents`, `victoria-metrics-operator` and
`vertical-pod-autoscaler-crds` in
`packages/apps/kubernetes/templates/helmreleases/` are all gated on
`addons.monitoringAgents.enabled`, so they disappear together and Flux
uninstalls them in parallel: `dependsOn` is not honoured on deletion.
Two things then break the uninstall of `monitoring-agents`:

1. The VictoriaMetrics operator is often gone first. The operator's
`apps.victoriametrics.com/finalizer` stays on the `VMAgent`, nobody
removes it, and the deletion hangs. The manual workaround was to strip
the finalizer, after which GC removes the VMAgent's Deployment, Service,
Secrets and ServiceAccount through their owner references.
2. The VPA CRD was deleted with its own release while the stored
manifest still contains `VerticalPodAutoscaler vpa-vmagent`, so Helm
cannot build the manifest to uninstall. That root cause is fixed by
#3586 (`helm.sh/resource-policy: keep` on the tenant addon CRDs), which
this PR does not redo.

This PR:

- adds a `pre-delete` hook (`templates/vm-cleanup-hook.yaml`) that
deletes every VictoriaMetrics object the release owns, waits up to 60s
for a live operator to finalize them properly, and clears their
finalizers by hand when it has not. Objects are selected through Helm's
`meta.helm.sh/release-name` / `release-namespace` annotations rather
than by name, so the 34 `VMRule`s rendered from `alerts/` and the
`VMServiceScrape` in `cozy-victoria-metrics-operator` are covered along
with the VMAgent. The kinds handled are every VictoriaMetrics kind the
chart renders: `VMAgent`, `VMRule`, `VMServiceScrape`, `VMPodScrape`,
`VMNodeScrape`. Operator v0.68.4 only finalizes the VMAgent; the other
kinds are covered in case another operator version finalizes them, and
for them the patch is a no-op. A kind whose CRD is already gone is
skipped, and the script always exits 0, since a failing pre-delete hook
would itself block the uninstall. The Job is bounded by
`activeDeadlineSeconds: 180` and `backoffLimit: 0`; its kubectl calls
carry no `--request-timeout`, which would make kubectl skip the
in-cluster credentials (`hack/check-hook-request-timeout.bats`) and
reuses the digest-pinned `docker.io/clastix/kubectl:v1.32` image the
etcd-operator hook uses, via a new `kubectlImage` value. The shape
follows
`packages/system/forklift/templates/forklift-cr-cleanup-hook.yaml`.
- guards `templates/vpa.yaml` on `.Capabilities.APIVersions.Has
"autoscaling.k8s.io/v1/VerticalPodAutoscaler"`, so a cluster that does
not serve the CRD never records a VPA in the release manifest. Cheap
defence in depth next to #3586.

Points for review:

- The hook's RBAC is a ClusterRole/ClusterRoleBinding, limited to `get,
list, delete, patch` on the five `operator.victoriametrics.com`
resources, rather than Roles. The objects span two namespaces, and a
Role in `cozy-victoria-metrics-operator` could not be created if that
namespace were already gone or terminating, which would fail the hook
and the uninstall with it. All hook objects carry
`hook-succeeded,hook-failed,before-hook-creation` except the Job, which
keeps a failed pod for its logs (`ttlSecondsAfterFinished: 600`), as in
the forklift hook.
- When the operator is already gone, clearing the finalizer skips
whatever the operator does on finalize, notably the cluster-scoped RBAC
it creates for the VMAgent, which has no owner reference. That residue
is the same the manual workaround left, and it is no worse than today's
stuck release; removing it would mean hard-coding the operator's naming
scheme for those objects, which I preferred not to do.
- When the operator is alive, the hook only brings forward deletions
Helm was about to make, so the operator's normal finalization still
runs.

Tests: `tests/uninstall_test.yaml` asserts that the hook is a
`pre-delete` Job in the release namespace with a bounded runtime, that
its RBAC is hook-scoped and grants exactly the verbs needed on every
VictoriaMetrics kind the chart renders, that the binding targets the
hook's ServiceAccount, that the script selects by release ownership,
deletes, clears finalizers and exits 0, and that the VPA renders only
when the capability is present (using `capabilities.apiVersions`). With
the hook removed and `vpa.yaml` reverted, 6 of the 7 tests fail; with
the fix, `helm unittest packages/system/monitoring-agents` passes 26/26.

`hack/monitoring-agents-vm-cleanup-hook.bats` runs the rendered script
against a fake `kubectl` that keeps the objects in a state file, with
fake `date` and `sleep`. It checks that only objects annotated for this
release are deleted and patched, that with the operator gone the hook
polls through the grace period and then clears the finalizers, that a
live operator's objects are never patched, that a missing CRD still
exits 0 and cleans the other kinds, and that a release with nothing left
exits 0 without deleting anything. Removing the ownership condition,
making the patch a no-op, or inverting the polling condition each turns
this file red. The hook has not yet been run on a live tenant cluster.

Backport: labelled `kind/backport` (release-1.7). The release-1.6
backport is done by hand in #5158. The commit cherry-picks onto
`release-1.6` with a single trivial conflict in `values.yaml` (context
only: 1.6 lacks the `prometheus-node-exporter` resources block the new
`kubectlImage` block sits after on `main`). Resolved locally, the
chart's tests pass on 1.6 (9/9); 1.6's chart renders the same
VictoriaMetrics kinds. Note that `release-1.6`'s monitoring-agents
`Makefile` has no `test:` target, so CI does not run this chart's unit
tests on that branch.

### Screenshots

Not a UI change.

### Downstream repositories

I walked the trigger map against the diff (a system chart's templates,
`values.yaml` and tests): no package added or renamed, no platform
values, no app schema, no CRD, no tooling or convention change. I also
searched cozystack/website for a documented VMAgent finalizer workaround
that this would make obsolete and found none.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(monitoring-agents): disabling the Monitoring Agents addon of a tenant Kubernetes cluster no longer leaves the monitoring-agents release stuck in UninstallFailed when the VictoriaMetrics operator is removed first; a pre-delete hook now releases the VictoriaMetrics objects' finalizers, and the VPA is only rendered when its CRD is served.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* VictoriaMetrics resources belonging to a release are cleaned up during
uninstall, including resources that remain because of finalizers.
Cleanup is limited to resources owned by that release.
* Vertical Pod Autoscaler resources are rendered only when the cluster
supports the required API.
* **Configuration**
* Added an image setting for the uninstall cleanup job, including a
pinned image version and pull policy.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 08:58:10 +03:00
Alexey Artamonov 583f18437d test(clickhouse): give the credentials-wait BackupJob a UID
The test pins that a backup held for late bucket credentials keeps
its archive name, and a run's archive name now carries part of the
BackupJob's UID, so the fixture needs one for the name it expects to
be the name the controller mints.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-12 17:21:51 +03:00
Alexey Artamonov dae76d186c Merge branch 'main' into fix/clickhouse-idle-before-backup
main pins the archive name a backup Pod keeps across a wait for
late bucket credentials, against the controller's archive helper.
That helper now names a run's archive with part of the BackupJob's
UID, so the merged test has to give its BackupJob one.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-12 17:20:19 +03:00
Aleksei Sviridkin 238b3ee491 fix(seaweedfs): enforce S3 bucket lifecycle rules (#5228)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

SeaweedFS accepts and stores an S3 bucket lifecycle configuration, but
only the `s3_lifecycle` job of its plugin worker applies it, and that
job is scheduled by the admin server. Cozystack deployed neither, so a
lifecycle rule returned 200 and no object ever expired, on the platform
backup bucket and on tenant buckets alike. That also made the documented
way to bound backup storage a no-op.

This PR adds the admin server and a worker to Simple and MultiZone
SeaweedFS instances, behind a new `lifecycle.enabled` value that is
**off by default**. A Client instance is unchanged, because its rules
are enforced where the instance it connects to runs. The worker
registers for `s3_lifecycle` only, so vacuum, balancing and erasure
coding are not started as a side effect. The admin keeps its job state
on a 1Gi volume on `lifecycle.storageClass`, or on the cluster default
when that is empty; `volume.storageClass` is not reused, because in
MultiZone it can name a class local to one zone or node. The admin UI is
protected by a generated password, because a tenant namespace admits
traffic from the whole cluster. The dashboard Role lists the two new
WorkloadMonitors.

Enabling the worker alone was not enough on SeaweedFS 4.45. Its chart
renders no `[grpc.s3]` section, so the s3 gateway served its internal
gRPC in plaintext while the worker dials it with mTLS, and every
lifecycle delete failed the TLS handshake. The section is backported
from upstream (seaweedfs/seaweedfs#11157, released in 4.46) as a chart
patch, applied with `--forward` so the next `make update` stops at the
point where the patch must be dropped instead of silently reversing it.
The s3 pods are restarted once so they load the section. This also
repairs IAM cache propagation between s3 replicas, which failed in the
same way, and it applies whether or not lifecycle is enabled.

The data-plane release now retries a failed install or upgrade in place
(`RetryOnFailure`) instead of rolling it back. With the previous default
remediation and unlimited retries, an upgrade that waited out its
timeout, for example on a tenant quota with no room for the admin
volume, rolled back and retried forever, restarting the s3 pods twice
per cycle. The worker's hostname anti-affinity is relaxed to a
preference so a rollout finishes on a single-node cluster.

#### Why lifecycle is off by default

The daily walk that SeaweedFS uses to expire objects sends its deletes
without an object identity, and the s3 server skips its compare-and-swap
when no identity is given; it does not re-check the object's age either.
A key overwritten between the walk's listing and the delete loses its
new content. This path is unchanged from 4.45 through 4.52 and on
upstream master (`weed/s3api/s3lifecycle/dailyrun/walker_dispatcher.go`,
`identityMatches` in `weed/s3api/s3api_internal_lifecycle.go`). Backup
buckets write unique keys and are barely exposed; a tenant bucket that
rewrites a fixed key is. On top of that, turning enforcement on would
start applying, on upgrade, every rule that was stored and ignored until
now. Both are retention changes an operator should choose, so nothing
changes for existing instances until `lifecycle.enabled` is set. Making
it the default again is left to a separate PR that carries object
identity on the walk path; upstream report:
https://github.com/seaweedfs/seaweedfs/issues/12451.

#### Turning lifecycle on and off

When `lifecycle.enabled` is set, rules that have been stored but ignored
start applying within minutes, and objects they have already expired are
deleted on that first pass, so review existing bucket lifecycle rules
first. Set `lifecycle.storageClass` before enabling, and prefer a
replicated class: the class is part of the admin StatefulSet's volume
claim template, which cannot change, so changing it while lifecycle is
on makes every upgrade of the instance fail until it is reverted. Moving
the volume means disabling lifecycle, deleting the
`admin-data-seaweedfs-admin-0` PVC, changing the class and enabling
lifecycle again. Each instance gains two small pods and a 1Gi PVC; when
the tenant quota has no room for them, the data-plane release keeps
retrying in place until it has.

When `lifecycle.enabled` is turned off again, the parent release removes
the admin credentials Secret while the data-plane release removes the
admin StatefulSet only on its own reconcile, shortly after. An admin pod
that restarts in that window fails to start on the missing Secret until
the StatefulSet is gone; nothing is lost, and the order is left as is.
The admin PVC stays until the instance is deleted, and a later re-enable
starts the admin with a new password.

Verified on a test cluster with the 4.45 chart: without the backport,
the worker found the expired objects and every delete failed with `tls:
first record does not look like a TLS handshake`; with this branch and
lifecycle enabled, a rule with a past expiration date removed the
matching objects and left the others, both on a manual run and on the
first scheduled pass.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up needed, not opened yet:
`operations/services/backup-classes.md` (v1.7 and next) advises an S3
lifecycle policy, which the bundled SeaweedFS applies only once
`lifecycle.enabled` is set; the SeaweedFS reference page gains
`lifecycle.enabled` and `lifecycle.storageClass` from the regenerated
README on the next release
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The terraform provider does not model the SeaweedFS application, so the
new values do not reach it.

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(seaweedfs): S3 bucket lifecycle rules can now be enforced. Simple and MultiZone SeaweedFS instances run the SeaweedFS admin server and a lifecycle worker when the new `lifecycle.enabled` value is set; it is off by default, so existing instances keep their current behaviour. It stays off by default because the upstream daily expiry walk deletes without checking that the object it listed is still the one stored, so an object overwritten in that window can be deleted although it is new. When enabled, rules that were stored but ignored start applying within minutes and objects they have already expired are deleted on that first pass; review existing rules first. Each enabled instance gains two small pods and a 1Gi volume, placed on `lifecycle.storageClass` or the cluster default. Independently of the switch, the s3 gateway now serves its internal gRPC over mTLS, which repairs IAM cache propagation between s3 replicas and restarts the s3 pods once on upgrade.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Added optional, daily S3 bucket lifecycle-rule processing for
supported SeaweedFS configurations. It is disabled by default; Client
instances rely on lifecycle rules configured on the connected SeaweedFS.
* Added configurable storage for lifecycle job state and dashboard
visibility for lifecycle components.
* Enabled the S3 gRPC server to use its internal TLS identity when
security is configured.

* **Bug Fixes**
* Improved storage-class handling so numeric-looking names remain
strings and unset classes are omitted where appropriate.
* Clarified backup cleanup guidance, including when lifecycle rules
apply and how to reclaim storage.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-12 16:21:52 +03:00
Aleksei Sviridkin 3badbdef74 fix(clickhouse): let a running backup sidecar pick up system-bucket credentials (#5232)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

A ClickHouse release with `backup.useSystemBucket: true` can start
before the platform projects `cozy-backups-creds` into its namespace.
Its `clickhouse-backup` sidecar took the S3 settings from environment
variables, which are fixed when the container starts, so the first
BackupJob or RestoreJob of such a release failed with `no EC2 IMDS role
found` until the Pod was restarted, and a rotated key stayed stale the
same way.

The projector now also writes a `clickhouse-backup.yml` key into
`cozy-backups-creds` (bucket, `https://` endpoint, region when set,
keys, `force_path_style` from `backupStorage.forcePathStyle`). On the
system-bucket flow the chart mounts that key as an optional Secret
volume and points `CLICKHOUSE_BACKUP_CONFIG` at it, instead of setting
`S3_*` variables. clickhouse-backup reloads its config file in each API
handler that reaches S3 (among them the `create_remote`,
`restore_remote` and `list` calls the strategy makes; the `status` and
`actions` reads it polls with do not) and kubelet rewrites a mounted
Secret in place, so credentials projected after the Pod started, or
rotated since, reach the running sidecar. `S3_PATH` stays in the
environment because the prefix belongs to the release.

Kubelet takes about a minute to refresh the volume, so the projector
stamps `backups.cozystack.io/credentials-updated-at` when the projected
content changes, and the Altinity driver holds a new backup or restore
Job for up to two minutes after that stamp, with `Ready=False` and
reason `WaitingForCredentials`; the condition is dropped once the Job
runs. A stamp that cannot be read, or one further ahead than clock skew
explains, is logged and ignored rather than holding the namespace's jobs
for good. The hold applies to every Altinity job in a namespace whose
projected Secret just changed, including releases on their own Secret,
where it is only a delay.

The new e2e Test `clickhouse-3-system-bucket-roundtrip` starts a
ClickHouse on the system bucket with no projected Secret, then runs a
backup and an in-place restore through `cozy-default` and checks that
the sidecar that started without credentials served both jobs. In CI the
system bucket's advertised host `s3.example.org` neither resolves nor
gets an ACME certificate, so for the duration of its own step the Test
serves a SeaweedFS-CA certificate under the S3 Ingress TLS Secret name
and adds a marked `rewrite name exact` line for that host to the host
CoreDNS Corefile, removing both on exit; it proves the served
certificate before any job starts.

The bug was reproduced on a test cluster running the code before this
change: a ClickHouse started before the projection failed its first
BackupJob with `no EC2 IMDS role found`. The new e2e Test has not run
green yet outside CI, because it edits the host CoreDNS, which is shared
on that cluster.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#799 (draft until this PR merges):
`operations/services/backup-classes.md` for the mounted config file, the
credentials wait and the upgrade window, without the Pod restart this
change removes
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(clickhouse): a ClickHouse release on the platform system bucket (`backup.useSystemBucket: true`) now backs up and restores without a Pod restart when its namespace had no projected backup credentials at start, and picks up rotated credentials while running. The sidecar reads its S3 settings from a `clickhouse-backup.yml` file in `cozy-backups-creds` instead of environment variables, so the upgrade rolls every such ClickHouse Pod once. The first Altinity backup or restore in a namespace after its projected credentials change waits up to two minutes (`WaitingForCredentials`) for them to reach the sidecar. Upgrade the platform as a whole: a system-bucket BackupJob started while the new chart runs against the previous controller fails, because that controller does not write the new key.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* ClickHouse system-bucket backups now use a projected configuration
file for S3 settings, supporting HTTPS endpoints and optional region and
path-style settings.
* Backup and restore jobs wait briefly for recently updated projected
credentials before starting.
* Path-style settings accept common true and false values and are
normalized consistently across backup storage integrations.
* **Tests**
* Added end-to-end coverage for system-bucket backup and restore,
verifying restored data without restarting the ClickHouse Pod or backup
sidecar.
* Added checks for TLS certificate validation and safe restoration of
shared network and ingress settings.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-12 16:15:26 +03:00
Aleksei Sviridkin aa97451bc2 fix(platform): leave tenant-root's ingress empty when the root tenant has none (#5226)
## What this PR does

Enabling SeaweedFS on the root tenant without ingress no longer hangs.
`cozystack-basics` wrote `_namespace.ingress: tenant-root` for the root
tenant whatever its `spec.ingress` said, so `seaweedfs-system` depended
on an `ingress-nginx-system` release that nothing creates, and the
platform backup bucket waited with it. Every other tenant already
derives the key in `packages/apps/tenant/templates/namespace.yaml` and
leaves it empty without ingress.

`cozystack-basics` cannot read the root Tenant, so the platform now
carries its ingress state the same way it carries monitoring: the new
platform value `ingress.rootEnabled` reaches the `cozystack-values`
Secret as `_cluster.ingress-enabled`, and `_namespace.ingress` is
written only when it is true. The default is `true`, which keeps the
value every existing cluster renders today, and a Secret written before
the key existed keeps it as well. An operator running the root tenant
without ingress sets `ingress.rootEnabled: false`, kept in sync with
`Tenant/root spec.ingress` by hand, as with `monitoring.rootEnabled`.

With the key empty, `seaweedfs-system` drops the dependency through the
check it already had (`eq $ingress .Release.Namespace`), and nothing
reads live cluster state for it.

An empty `_namespace.ingress` now reaches tenant-root and every tenant
inheriting from it, so the consumers that would break on it are adjusted
in the same PR. The API rejects `ingressClassName: ""` (checked with a
server-side dry run), and two consumers quote the value:

- `kubernetes`: the proxied ingress-nginx addon Ingress is not rendered
without an ingress; its Service still is.
- `monitoring`: Grafana's `spec.ingress` is left out, since
grafana-operator copies it into the Ingress verbatim.
- `seaweedfs`: the S3 Ingress is disabled instead of rendering with no
controller to serve it.

The `KamajiControlPlane` `network.ingress` block is unchanged on
purpose: its `hostname` is what the Kamaji provider uses as the control
plane endpoint, and Kamaji itself skips an empty class name. Harbor,
bucket, bootbox and alerta write the class unquoted, so an empty value
renders as an unset class, which the API accepts.

Tests:
`packages/core/platform/tests/apps_ingress_enabled_wiring_test.yaml`
pins the flag reaching `_cluster` both ways and with `ingress` set to
null;
`packages/system/cozystack-basics/tests/cozystack-values-secret_test.yaml`
pins `_namespace.ingress` for a missing, string, bool and `"false"`
flag, and that the other keys stay when ingress is off;
`packages/extra/seaweedfs/tests/ingress_dependency_test.yaml` pins the
dependency and the S3 Ingress for an own, empty and inherited ingress;
the kubernetes and monitoring suites pin the gated Ingress. Reverting
any of the gates, or flipping the missing-key default, fails the suites.

Not covered here:

- `_namespace.etcd` has the same hardcoded shape (#4783) and is left for
its own change.
- Toggling `Tenant/root spec.ingress` without changing
`ingress.rootEnabled` leaves the value stale, as
`monitoring.rootEnabled` does today.

### Screenshots

Not a UI change.

### Downstream repositories

The diff adds a key to `packages/core/platform/values.yaml`, which the
website documents by hand.

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#800 documents `ingress.rootEnabled` in the
platform Package reference
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(platform): a root tenant without ingress no longer leaves seaweedfs-system waiting for an ingress-nginx-system release that is never created. Set the new platform value ingress.rootEnabled to false to match Tenant/root spec.ingress: false.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Added a root-tenant ingress setting, enabled by default, that can be
turned off.
* **Bug Fixes**
* Kubernetes and Grafana ingress resources are now rendered only when
ingress is configured.
  * SeaweedFS S3 ingress is enabled only when ingress is configured.
* Ingress-dependent components now respect the root-tenant setting,
while other platform services remain available when root ingress is off.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-12 15:06:12 +03:00
myasnikovdaniil 76a34d3e53 fix(seaweedfs): bump the COSI driver to v0.4.0 for the lost-grant fix (#5284)
## What this PR does

This PR bumps seaweedfs COSI driver to v0.4.0, because v0.3.1 can lose a
grant for good: BucketAccess reports `accessGranted` and its Secret is
written, but S3 gateways keep rejecting the key with
`InvalidAccessKeyId`. Fixes #4912.

Driver v0.3.1 grants access by rewriting legacy
`/etc/iam/identity.json`, and every S3 gateway migrates this file to
per-identity files and renames it away on each change, so grant written
during migration is dropped. v0.4.0 also writes
`/etc/iam/identities/<user>.json` on grant
(seaweedfs/seaweedfs-cosi-driver#15). First version of this PR built the
driver from that upstream commit, now upstream released v0.4.0 so it's
just a tag bump.

Rest of v0.4.0 doesn't reach the platform:

- `readwrite-governance-bypass` access policy
(seaweedfs/seaweedfs-cosi-driver#16) is opt-in, none of our
BucketAccessClasses use it.
- Bucket deletion is refused while objects hold an active lock
(seaweedfs/seaweedfs-cosi-driver#18), only for object-lock buckets. All
`-lock` BucketClasses are `Retain`, and sidecar calls DriverDeleteBucket
only for `Delete`.
- Filer errors (seaweedfs/seaweedfs-cosi-driver#20) are now matched by
exact not-found message, I checked it against seaweedfs 4.45 filer and
formats match. One behavior change: bucket deletion doesn't ignore
errors of the recursive delete anymore, so sidecar retries instead of
reporting success.

How I tested (seaweedfs 4.45, two S3 gateways): Bucket with one user,
then three more users added in one change, after a minute ListObjects
with each user key, rejected key retried once more after 30s. Five runs
for each driver, the fixed one is a build of `5efd120`, the commit
v0.4.0 takes the grant fix from:

| driver | keys rejected for good |
| --- | --- |
| v0.3.1 | 5 of 20 (4 runs out of 5 lost at least one) |
| `5efd120` | 0 of 20 |

Same failure broke v1.7.1-rc.1 e2e, RabbitMQ backup round-trip got `403
InvalidAccessKeyId` for freshly granted key:
https://github.com/cozystack/cozystack/actions/runs/42980741995

Keys already lost on running clusters stay broken after upgrade.
Workaround from #4912 still works: remove user from Bucket, wait until
its BucketAccess is gone, add it back.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(seaweedfs): bucket users granted together could get S3 keys that never work (403 InvalidAccessKeyId). COSI driver is bumped to v0.4.0, which writes each grant to per-identity IAM file.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Updates**
  * Updated the SeaweedFS COSI driver to version `v0.4.0`.
* The updated driver writes IAM files per identity, helping preserve
grants when identity files are updated.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-12 16:31:44 +05:00
Aleksei Sviridkin 0b284a65d5 test(release): isolate report deduplication from workflow attempts (#5282)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, security (a fix that comes through a security
advisory), docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Fixes merged to `main` reach the release lines the backport batch runs
for automatically, so do not open backport PRs by hand:
https://github.com/cozystack/cozystack/blob/main/docs/release/backports.md
-->

## What this PR does

Pins the report deduplication test to the first workflow attempt. In a
CI rerun, the inherited attempt otherwise adds `/attempts/2` to the
synthetic run URL and fails the existing exact-URL assertion. Production
run URLs and the deduplication assertions stay unchanged.

Validation: a source-derived check of the test setup and original URL
assertion fails with an inherited second attempt and passes with the
proposed first-attempt setup. Controls with the pin after preview or set
to 2 still fail. The full Git-backed test was not run locally; CI must
verify it.

### Screenshots

Not applicable; no UI changes.

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
test(release): make the report deduplication test independent of the surrounding workflow attempt.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Tests**
* Updated automated test configuration to account for GitHub Actions run
attempts. No user-facing behavior changes.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-12 13:01:17 +03:00
Alexey Artamonov 5a95bcdd64 fix(clickhouse): keep a run's archive name within the old name's length
The archive name is part of every object key, a key is capped at 1024
bytes, and the legacy s3PathOverride prefix has no length limit. The
full UID made the name 37 characters longer than before, so a prefix
that fit the old name pushed the manifest key past the cap and failed
the backup.

The run's name now drops "clickhouse-" and the "T" and "Z" of the
timestamp and spends the room on the first 12 hex digits of the UID,
so it is exactly as long as the old name for every application.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-12 11:10:43 +03:00
Myasnikov Daniil 6a93765351 fix(seaweedfs): bump the COSI driver to v0.4.0 for the lost-grant fix
seaweedfs-cosi-driver v0.3.1 grants access by rewriting the legacy
/etc/iam/identity.json. Each S3 gateway migrates that file to
per-identity files and renames it away whenever it changes, so a grant
written while a gateway is migrating is lost: the BucketAccess reports
accessGranted and its Secret is written, but every gateway rejects the
key with InvalidAccessKeyId and never recovers. Users added to a Bucket
in one change lose a key regularly. v0.4.0 also writes the per-identity
file on grant.

The rest of v0.4.0 does not reach the platform. The
readwrite-governance-bypass access policy is opt-in and no
BucketAccessClass uses it. The refusal to delete a bucket that still
holds locked objects applies only to object-lock buckets, and their
classes are Retain, so the sidecar never calls DriverDeleteBucket for
them.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-12 07:53:34 +00:00
Alexey Artamonov 67fcadf689 fix(seaweedfs): disable the S3 Ingress in a tenant without ingress
The S3 Ingress kept rendering with no ingress-nginx to serve it. Now
that seaweedfs-system no longer waits on ingress-nginx-system there, the
release completes and the unserved Ingress would stay behind.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 18:20:38 +03:00
Alexey Artamonov 275c840a8c fix(monitoring): leave Grafana's ingress out in a tenant without ingress
grafana-operator creates the Ingress from spec.ingress verbatim, so an
empty _namespace.ingress reached the API as ingressClassName "" and was
rejected. With no ingress-nginx in the tenant there is nothing for that
Ingress to attach to.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 18:20:38 +03:00
Alexey Artamonov c61e6a842e fix(kubernetes): skip the proxied addon Ingress in a tenant without ingress
An empty _namespace.ingress means no ingress-nginx serves the tenant,
which tenant-root now reports when the root Tenant has ingress off. The
Ingress would carry ingressClassName "", which the API rejects, and the
release would fail to install where it used to leave an unserved
Ingress behind.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 18:20:38 +03:00
Alexey Artamonov b26001b6f9 test(seaweedfs): pin the ingress-nginx dependency to the namespace's ingress
seaweedfs-system waits on ingress-nginx-system only when _namespace.ingress
names its own namespace. With tenant-root's key now empty when the root
Tenant runs no ingress, an empty value has to drop the dependency, and a
value inherited from a parent must not add it.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 18:04:36 +03:00
Alexey Artamonov 5b6708c7f9 fix(platform): leave tenant-root's ingress empty when the root has none
cozystack-basics wrote _namespace.ingress: tenant-root whatever the root
Tenant's spec.ingress said, so applications in tenant-root were told an
ingress-nginx runs there when nothing deploys it. SeaweedFS then waited
forever on ingress-nginx-system, and the platform backup bucket with it.
Every other tenant already derives the key and leaves it empty without
ingress.

cozystack-basics cannot read the root Tenant, so the platform carries
its ingress state the way it carries monitoring: ingress.rootEnabled
reaches the Secret as _cluster.ingress-enabled and gates the key. The
default keeps the value existing clusters render, and a Secret written
before the key existed keeps it too.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 18:04:35 +03:00
Aleksei Sviridkin 9978771858 feat(observability-proxy): serve tenant namespace logs through an authorizing proxy (#5214)
## What this PR does

Adds `observability-proxy`, a new system component that lets a tenant
user read the logs of their own namespace from VictoriaLogs with the
same Kubernetes credentials they already use for the console, and with
Kubernetes RBAC deciding what they can read. This is the first part of
the design proposals [Tenant application
logs](https://github.com/cozystack/community/pull/95) and [Tenant
resource-consumption
metrics](https://github.com/cozystack/community/pull/94); this PR
implements the namespace logs root only. Metrics and the per-application
roots follow in separate PRs.

Why a proxy: today the log stores are shared per root tenant and have no
notion of a caller, so tenants cannot be given direct read access
without seeing other tenants' logs. The proxy authenticates the caller,
authorizes the request from its path, and rebuilds the upstream query so
that its scope cannot be widened by the client.

Backward compatibility: this adds a new package and a new HelmRelease;
nothing existing is renamed, removed or changed, so no migration is
needed.

What is included:

- `cmd/observability-proxy` and `internal/observabilityproxy`: the
proxy.
- Serves
`/apis/cozystack.io/v1alpha1/namespaces/<ns>/logs/select/logsql/<endpoint>`
for the VictoriaLogs read endpoints the console needs (`query`, `hits`,
`stats_query_range`, `tail`, `field_values`, `stream_field_values`,
`facets`). Anything else under the root answers `404`.
- Authentication by `TokenReview`, authorization by one
`SubjectAccessReview` built from the path (`list` on `cozystack.io/logs`
in the namespace), both cached briefly.
- The upstream request is rebuilt from an allowlist of parameters; the
namespace scope is always forced as `extra_stream_filters`.
Client-supplied `extra_filters`, `extra_stream_filters`, any LogsQL
`options(...)` block, and tenant headers such as `AccountID`/`ProjectID`
are not forwarded or are refused.
- Non-canonical paths (`..`, `//`, encoded separators) are refused with
`400` before any authorization is built, because the console nginx
matches on the normalized path but forwards the raw one.
- Time window, step, bucket count and result limits are clamped;
per-user rate, concurrency and open-tail limits answer `429`. Invalid
time, step and limit parameters answer `400`.
- The store is resolved from the namespace's
`namespace.cozystack.io/monitoring` label.
- Live `tail` streams are passed through unbuffered, and the upstream
tail request disables compression so headers reach the client before the
first line.
- `packages/system/observability-proxy`: chart and image build, wired
into the root `make build`.
- Platform wiring: a new `cozystack.observability-proxy` PackageSource,
enabled by default on the full variants and opt-in on the slim ones, the
same way as `monitoring-agents`.
- RBAC: an aggregated `cozy:observability:view` ClusterRole grants
`list` on `cozystack.io/logs` to tenant `view` and above. No API server
serves this resource; the rule exists only for the proxy's
`SubjectAccessReview`. The proxy's own ClusterRole is limited to the
calls it makes (`tokenreviews`, `subjectaccessreviews`, `list`/`watch`
on namespaces).
- Console nginx: when the proxy is not installed or cannot be reached,
the observability roots answer `404` with a Kubernetes `Status` body,
which the console shows as not available on this cluster; the proxy's
own `502`/`504` for an unreachable store pass through unchanged.
- Console: one regex `location` in the console nginx routes the four
observability roots to the proxy ahead of `/apis`. The upstream is
resolved at request time, so the console starts even when the proxy is
not installed; those paths then answer `404` with a Kubernetes `Status`
body. The console Deployment now receives the cluster domain for that
resolution.
- E2E: a new Chainsaw suite `hack/e2e-chainsaw/observability-proxy/`
with two tenants, covering `401`/`403`/`200`, scope-widening attempts,
path normalization and a long-lived `tail`.

#### How it was tested

- `go test -race` and `go vet` for the proxy packages.
- helm-unittest for `observability-proxy`, `dashboard`, `tenant` and
`core/platform`; bats tests for the console nginx routing.
- The Chainsaw suite was not run as-is: it deploys its own VictoriaLogs
as the root store and refuses to run when a real one exists. Its cases
were replayed by a script against the real store on a test cluster.
- Verified on a test cluster:
- `401` without a token and with an invalid token, `403` for another
tenant's namespace, `200` for the caller's own.
- Client `extra_stream_filters`/`extra_filters` (GET and POST), an
`AccountID` header and another tenant's stream named inside the query do
not widen the scope; `options(...)` answers `400`; a path with `..`
pointing at another tenant answers `400`; an endpoint outside the
allowlist answers `404`.
- Through the full browser chain (OIDC login, gatekeeper, gateway,
console nginx), a single `tail` stayed open for more than two minutes
without reconnecting, and requests from the same session to other
tenants' roots answered `403`.
- Not verified: the `429` limits under real load. `golangci-lint` has
not been run locally.

#### Known limitations

- The proxy is reachable only through the console route; tenant
workloads have no egress rule to it. Every request takes its concurrency
or tail slot before the body is read; the token and access reviews and
the body read run under `--request-timeout`, so a stalled kube-apiserver
answers `503` rather than holding the request; and writes to the client
carry a deadline (renewed per line for tails). `--request-timeout=0`
disables those bounds and should not be used in production.
- The first lines a newly started pod writes can be missing from an
already open `tail`, although they are in the store. The likely cause is
that Fluent Bit picks up a new log file later than the VictoriaLogs
live-tail window; later lines of the same pod arrive. A history query
shows them.
- The per-application roots (`.../<plural>/<name>/log`, `.../metrics`)
and the namespace metrics root answer `404` in this PR.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/issues/787 (docs land
once the implementation PRs merge)
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Downstream: `cozystack/website` is reached. A new platform component
(`packages/core/platform/sources/observability-proxy.yaml`) triggers
`content/en/docs/next/guides/platform-stack/_index.md` and
`operations/configuration/licenses.md`; adding it to the system bundle
(`packages/core/platform/templates/bundles/system.yaml`) may also need a
line in `operations/configuration/variants.md`. A user-facing page for
the logs API and the new tenant `view` permission is not required by the
trigger map but would be useful. Tracked in cozystack/website#787. Other
repositories: no trigger matched. `packages/core/platform/values.yaml`
is not changed (only a test fixture), no package under `packages/apps/`
gains a values field, and no CRD changes. The root `Makefile` `build`
target builds one more image; this does not touch `hack/` or the make
targets `ccp` relies on, so `ccp` is not considered affected, but a
reviewer may want to confirm.

### Release note

```release-note
feat(observability-proxy): add an authorizing proxy that lets tenant users read their namespace logs from VictoriaLogs with their Kubernetes credentials, under `/apis/cozystack.io/v1alpha1/namespaces/<ns>/logs/`. Access requires the tenant `view` role or higher. The component is enabled by default on full variants and opt-in on slim variants.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Added an observability proxy for tenant-scoped log access through the
console, including live log streaming. Requests are authenticated,
access-controlled, and subject to configurable rate and concurrency
limits.
* Added platform packaging and deployment support, with console routing
for observability requests.
* **Tests**
* Added automated coverage for access control, tenant scoping, query
handling, streaming, proxy behavior, and deployment configuration.
* **Documentation**
* Updated end-to-end test guidance with host requirements for running
the observability proxy suite.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-11 16:24:25 +03:00
Aleksei Sviridkin 9fe06a874b test(release): isolate report deduplication from workflow attempts
CI reruns expose GITHUB_RUN_ATTEMPT to a test that expects a URL for
the first attempt. Keep its synthetic run identity independent of the
surrounding workflow.

Fixes: 5a997fb7d4 ("feat(release): post an immutable report of every classification and write record version 2")
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-11 15:35:18 +03:00
Alexey Artamonov 35823c6c9c test(clickhouse): list the system-bucket round-trip in the suite header
The header names the backup Tests of the suite and why they are split;
it still described two of them after the third was added, so a reader
starting from it would miss the only Test that covers the system-bucket
path.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 14:56:26 +03:00
Alexey Artamonov 8ba0380043 docs(clickhouse): name the clickhouse-backup handlers that reload the config
clickhouse-backup reloads its config file in each handler that reaches
S3, among them the create_remote, restore_remote and list calls the
strategy makes. The status and action-log endpoints the strategy polls
read in-memory state and do not reload it, so the mounted file reaches
the sidecar on the calls that need credentials rather than on every
request.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 14:56:11 +03:00
Andrey Kolkov 3b4db69f8e feat(backups): record the RDB snapshot location and size for Redis backups (#5046)
## What this PR does

Builds on the shared S3 probe from #5223 (MariaDB and CNPG), which is on
`main`. Refs #4316.

Backups made by the Redis strategy carry no `Backup.status.artifact` at
all: no URI and no size. This covers both the Redis and Valkey apps,
which share the driver. The snapshot is a single object per backup,
`<namespace>/<app>/<backup>.rdb` (`redisObjectKey`). The driver already
records that key in `driverMetadata`. The default strategy uploads the
object using the S3 coordinates from the backup credentials Secret,
which the controller projects into the application namespace
(`endpoint`, `bucketName`, `region` and the access keys).

When a backup completes, the driver now:
1. reads that projected Secret (`CredentialsConfig.TargetSecretName`) in
the application namespace. The projector stores its `endpoint` without a
scheme, and the strategy script reaches it over https; an endpoint with
a path prefix is refused by minio-go, which leaves the artifact unset;
2. reads the TLS trust from the pod spec of the completed backup Job, in
the order the script applies it: `ca.crt`, then `tls.crt`, from the
Secret mounted as the `s3-ca` volume
(`backupStorage.endpointCASecretName`). Only when there is no `s3-ca`
volume does it honour `S3_INSECURE_SKIP_VERIFY=true`, and only when
every container of the pod sets it, since the probe cannot tell which
container uploaded. The Secret is read at probe time; the default
strategy mounts `s3-ca` only when an endpoint CA is configured, so when
the volume yields no CA now (the Secret is gone or lacks both keys, or
the volume is not a Secret), the probe leaves the artifact unset rather
than trust the system roots or skip verification where the Job may have
verified. The CA is trusted on its own, as `curl --cacert` does, not on
top of the system roots;
3. runs `StatObject` on the recorded key; a response without
`Content-Length`, which minio-go reports as size -1, counts as
unmeasured;
4. records `status.artifact.uri = s3://<bucket>/<key>` and
`status.artifact.sizeBytes` = the object's size. An empty object is
recorded too, but `sizeBytes` is `omitempty`, so on the Backup it reads
like an unmeasured size; since a valid RDB snapshot always carries a
header, a Warning `EmptySnapshot` Event on the BackupJob flags it as a
broken backup.

The artifact is set **only when the object is found**. A custom strategy
that writes elsewhere would otherwise get a URI pointing at an object
that does not exist. On any failure (Secret missing or incomplete,
endpoint invalid, CA unreadable, object not found, TLS error) the
artifact stays absent and the backup still succeeds. The full error is
logged at info, and once the Backup is created a Normal
`ArtifactNotMeasured` Event on the BackupJob gives a fixed reason, also
when a failed existence check skipped the probe; the Event lives in the
application namespace, so it never carries the raw error, which may name
the platform's endpoint and bucket. The probe shares the 10s timeout of
the MariaDB and CNPG probes. The retry check is shared with those
drivers too: a `Backup` that already exists is returned without probing,
a failed existence check creates the Backup without probing, a `Create`
collision is resolved by reading past the cache, and an existing Backup
that cannot be read back is retried with `Ready=False` instead of
failing the BackupJob.

The size is useful because an RDB snapshot is the compressed dataset. It
shows growth, and a snapshot that is suddenly tiny is a broken backup.

Kafka is deliberately left out. Its backup is a topic-metadata export of
a few kilobytes, so the size would say nothing about the data and might
suggest the data itself was backed up.

Tests:
- the stat uses the projected credentials;
- every path that leaves the artifact unset: object missing, Secret
missing, bucket, endpoint, access key or secret key missing, no target
Secret configured, a size the endpoint did not report;
- trust resolution: `ca.crt` wins over `tls.crt` and skip-verify;
`items` remapping of the volume, and a key the `items` do not project is
never trusted; the Redis CA volume mounted next to `s3-ca` is not taken
as the S3 CA; an `s3-ca` volume that yields no CA (Secret gone, Secret
without the keys, a ConfigMap source) is refused with and without
skip-verify; skip-verify applies only on an exact `true` of that
variable in every container, the last entry of a repeated variable
winning;
- the real minio client against a TLS stub: its certificate is trusted
only through the `s3-ca` volume or skip-verify, a CA that did not sign
it is refused even with skip-verify set on the Job, and the shared
client enforces a configured CA with and without the system roots; the
client sends a signed `HEAD` and parses `Content-Length`;
- the Events: an unmeasured artifact or a failed existence check raises
`ArtifactNotMeasured` without the raw error; an empty snapshot is
recorded and raises a Warning `EmptySnapshot`; a measured artifact, a
rejected `Create` and a `Create` collision raise none;
- a reconcile test pinning that the completed Job's pod spec reaches the
probe; a Backup that already exists is returned without a probe or a
`Create`; the `Create` collision is read back from the API server past
the cache.

The suite fails if any guard of the trust lookup (volume name, Secret
source, variable name, every-container rule, refusal of an unreadable
CA), the CA enforcement in the transport, the negative-size check, the
Events, the reason-only message, their place after `Create`, the
empty-snapshot warning, the existing-Backup check, the wiring, any term
of the Secret check, or the API-server read-back is removed.

### Screenshots

N/A — no UI changes.

### Downstream repositories

I walked the trigger map against the diff. The change is Go-only in
`internal/backupcontroller`, with no API, CRD, chart, value or schema
change. Existing `Backup` fields are now filled for one more driver.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(backups): Redis and Valkey backups now record `status.artifact` with the `s3://` location and the size of the RDB snapshot. If the snapshot cannot be found with the platform backup credentials, the artifact stays unset and a Normal `ArtifactNotMeasured` Event on the BackupJob says why. An empty snapshot raises a Warning `EmptySnapshot` Event.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Redis backups now include artifact location and size when they can be
measured.
* Backups are still created when artifact measurement fails, with status
reporting that the artifact was not measured.
* **Bug Fixes**
* Improved handling of existing backups and temporary read-back failures
to support reliable reconciliation.
* Improved compatibility with S3 endpoints using custom certificate
authorities or configured TLS verification settings.
  * Zero-size snapshots are now reported with a warning.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-11 15:52:54 +04:00
Alexey Artamonov c26d2773a5 test(clickhouse): name what really leaves the round-trip's Secret behind
The projected Secret the preconditions delete is left by whichever
BackupJob ran in tenant-root before this Test, from this suite or
another: Chainsaw does not fix the order of the Tests in a suite, so no
particular one can be named. The restore budget is also not spent on a
credentials hold: by then the stamp is older than the backup's own
two-minute wait.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 14:42:11 +03:00
Aleksei Sviridkin fb5ac1826a test(observability-proxy): prepare seed stubs in both unit runners
The POSIX runner does not invoke Bats setup hooks. Initialize the
capture stub from the test body so the compatibility lane stays offline.

Fixes: 5a2582e21c ("test(observability-proxy): preserve JSON seed request bodies")
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-11 12:12:11 +03:00
Aleksei Sviridkin 66acdb80e1 ci(e2e): skip final cleanup on disposable PR runners (#5260)
## What this PR does

Remove the final sandbox teardown from the two PR E2E workflows. The
runners are documented as disposable VMs, so destroying their local
containers and zpools after testing delays the result without protecting
a later run. Cleanup between preparation attempts and the Makefile
targets for reused hosts stay unchanged.

The job limit stays at 215 minutes. Suite-length budgeting, including
nightly and tag runs, remains separate in #5261. This change relies on
the existing disposable-runner contract; it does not change runner
provisioning.

### Screenshots

Not applicable.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **CI Workflows**
* Removed automatic sandbox and cluster teardown after end-to-end report
uploads.
* Added checks that cluster cleanup runs before environment preparation
is retried.

These changes affect automated test workflows and do not alter
user-facing product features.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-11 12:10:26 +03:00
Alexey Artamonov f5114ae04c fix(seaweedfs): keep the metadata database storage class a string
db.storageClass went unquoted into the values of the database release
and again into its CNPG Cluster. A class name made only of digits,
which Kubernetes allows, became a number that the Cluster rejects, and
a line break in the value could add keys to the database release
values. It is quoted at both steps.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 10:30:11 +03:00
Alexey Artamonov e6eb3345c6 fix(seaweedfs): escape pool and zone storage classes in the release values
Pool and zone storage classes were wrapped in quotes by hand, which
does not escape a quote or a backslash in the value: a quote closes the
string early and the data-plane release values stop parsing, with a
YAML error that does not name the field. They are now quoted like the
volume and admin classes.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 10:29:48 +03:00
Alexey Artamonov f83db514b0 fix(seaweedfs): quote every storage class the upstream claims carry
The master, filer, volume index, log and all-in-one claims of the
upstream chart write storageClassName unquoted, like the data claims
did, so a class name made only of digits renders as a number and the
claim is rejected. None of them is set by Cozystack today, but they
are values of the same chart that a later change can start handing on,
and the patch already exists for the data claims. They follow the same
rule: quoted, and left out when no class is named so that the cluster
default applies. The deprecated filer.enablePVC claim is left alone:
it renders only with filer.data unset, and the template fails earlier on
an unset filer.data, so it cannot be reached.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 10:29:28 +03:00
myasnikovdaniil c6cba19946 feat(release): keep a line's backport record and reports across batch pull requests (#5264)
## What this PR does

This PR keeps backport batch record of a line across batch pull
requests. Before it lived only in body of open batch PR, so once that PR
merged or closed it was gone, and a conflict deferred in merged batch
showed up as missing in `check` and on cut. Three `check --cut` results
also did not match the docs, all three are fixed here.

- `check --cut` fails when workflow original is the only thing left
waiting. Before it passed as soon as pick next to it got `backport/skip`
or landed by hand.
- `run`, `check`, `fold` and `notify` read line record from newest
trusted batch PR in any state (opened by bot or by someone with write or
admin access). If newest record is in doubt (author no longer trusted,
bot batch without block, unreadable block, closed record newer than open
batch) command stops and doesn't fall back to older one.
- Record is written only while its PR is open. Merged and closed batch
PRs are never edited or commented, so run that picks nothing with no
open batch writes nothing and says so.
- Every run makes report: each original merged to `main` since line
branched in exactly one category, what the run read (heads, policy, tool
commit, Actions run, flags) and each drop with its reason and merge
commit. Latest report sits in body next to record and each one also goes
to immutable comment. Report ID is hash of classification without PR
number, so identical runs post one comment. `backport-batch report --pr
N [--id ID]` reads it back, merged batches too, it needs `main` in local
clone to check drops.
- `run --defer --push` of last pick used to refuse with "close batch
pull request instead". Now it writes report and record with reason while
PR is open, then closes PR and deletes branch, both only while head is
still the same. If close stops halfway next `run --push` finishes it,
and it never deletes branch that moved or that new batch uses.
- `run --drop N --reason R` keeps original off one line, `--undrop`
withdraws it. Drop is stored in line record bound to merge commit and
carried to next batches. Original the line already carries can't be
dropped, and `backport/include` doesn't lift a drop.
- Output names policy rules instead of internal numbers (no more "rule
101"), `--push` prints PR URL.

Reasons and file names from record are accepted only in the clean
one-line form a run writes them, and everything printed is escaped.

Record version 2 reads unversioned blocks that open batches carry today.
I ran new tool next to current one without `--push` on today's history
for release-1.7 and release-1.6, every original got same category and
`check` output is the same.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(release): backport batch keeps line record and report of every run across batch pull requests, check --cut fails on workflow change left waiting alone, deferred conflict stays deferred after its batch merges, --defer of last pick closes the batch, and new run --drop keeps a fix off one line.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Maintainers can permanently exclude a change from a release line with
`--drop`, or undo that decision with `--undrop`.
* Added a `report` command to view validated batch reports in JSON or a
readable summary.
* Deferring every remaining change can close an emptied batch. A later
`run --push` can complete an interrupted close.
* Batch reports now show change classifications, including deferred and
dropped items.

* **Bug Fixes**
* Checks more clearly distinguish missing, deferred, skipped, and
carried changes, with cut checks enforcing outstanding workflow
requirements.
* Batch operations refuse edited or untrusted records and avoid writing
to closed pull requests.

* **Documentation**
* Updated command and release-process guidance for drops, reports, cut
checks, and batch closure.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-11 12:12:39 +05:00
Alexey Artamonov e7b5a5a1a9 fix(seaweedfs): keep storage class names strings in the volume claims
The upstream admin and volume StatefulSets write storageClassName
unquoted, so a class name made only of digits, which Kubernetes allows
and the parent chart now hands on as a string, turned back into a
number there and the claim was rejected. The field is quoted, and left
out when no class is named: quoting an empty value would set an empty
storageClassName, which disables dynamic provisioning instead of using
the cluster default. Upstream 4.52 and master still write both fields
unquoted, so this is a chart patch, applied with --forward so make
update stops once upstream carries an equivalent change.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 02:18:50 +03:00
Alexey Artamonov 7c9aa2fe10 fix(seaweedfs): hand storage class names to the data plane as strings
The volume and admin storage classes were written unquoted into the
values of the data-plane release. A class name made only of digits,
which Kubernetes allows, arrived there as a number, and a value with a
line break could add keys next to the field, past the schema that is
the tenant's input gate. Both are now quoted. lifecycle.storageClass,
new in this release, also carries the StorageClass name rules in its
schema, so a value that cannot name a class is refused when the
application is written instead of leaving the admin claim pending.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-10 23:43:06 +03:00
Aleksei Sviridkin 5a2582e21c test(observability-proxy): preserve JSON seed request bodies
Send JSON seeds with their content type so VictoriaLogs does not consume
log records while parsing form parameters before ingestion.

Fixes: ebb5a9cec4 ("test(observability-proxy): cover the namespace logs scope end to end")
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-10 23:21:56 +03:00
Aleksei Sviridkin 9cd51cac2e ci(e2e): skip final cleanup on disposable PR runners
Destroying runner-local state after a completed suite can push the job
past its deadline even though the runner is discarded. Keep cleanup
between preparation attempts and for reused hosts instead.

Fixes: 186547d835 ("test(e2e): run the merge-gating lanes on Talos containers")
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-10 17:06:15 +03:00
Aleksei Sviridkin c117420f5e fix(kubernetes, monitoring): send tenant cluster metrics to the tenant's vminsert (#5024)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The vmagent that `packages/apps/kubernetes` deploys into a tenant
Kubernetes cluster has not been shipping metrics anywhere, because the
app and the `monitoring-agents` chart disagree on two values keys, and
Helm drops a key nobody reads without a word.

- **Remote write URL.** #697 changed the chart to read
`vmagent.remoteWrite.urls` as a list, but the app still passed a single
`vmagent.remoteWrite.url` (the cluster domain was later added to that
ignored key in #2086). The chart then applied its own defaults,
`vminsert-{shortterm,longterm}.cozy-monitoring.svc`, which do not exist
inside a tenant cluster. vmagent fails with `no such host` and queues
the failed writes on the node disk, up to
`-remoteWrite.maxDiskUsagePerURL` (1 GiB) per URL, which is enough to
push a small node into DiskPressure.
- **Tenant label.** The chart writes `tenant: {{ .Values.global.target
}}`, which only the root bundle sets, so every tenant cluster series was
labelled `tenant="cozy-monitoring"` and the
`vmagent.externalLabels.tenant` the app passes was ignored.

The fix:

- the app passes `remoteWrite.urls` with the tenant's shortterm
vminsert;
- the chart uses `vmagent.externalLabels.tenant` when set, and falls
back to `global.target`. The label is deliberately not fixed by passing
`global.target` from the app, because the tenant label must name the
tenant that owns the cluster (`.Release.Namespace`), while the
monitoring target can be a parent tenant.

The root install sets neither key: `helm template` of
`templates/vmagent.yaml` with `global.target=tenant-root` renders the
same URLs and label values before and after; the only textual change is
that the `tenant` label is now quoted, so a tenant name YAML would read
as a boolean or number still renders as a string.

Tests: `packages/apps/kubernetes/tests/monitoring_agents_test.yaml` pins
the keys the app sends, and
`packages/system/monitoring-agents/tests/vmagent_remote_write_test.yaml`
pins how the chart reads them, for both the root install and a tenant
cluster. Reverting either template change fails its suite. `helm
unittest` passes on both packages (358 and 23 tests).

Upgrade note for operators: on tenant clusters the `tenant` label of
vmagent series moves from `cozy-monitoring` to the owning tenant's
namespace, so dashboards or alerts that filtered on
`tenant="cozy-monitoring"` for those series need updating. Before this
change those series never reached the tenant's Monitoring in the first
place.

Left out on purpose: the shortterm store stays hardcoded, as it was in
the app before; deriving it from the target Monitoring's
`metricsStorages`, and bounding vmagent's on-disk buffer, are separate
changes.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(kubernetes, monitoring): the vmagent of tenant Kubernetes clusters now writes to the tenant's vminsert and labels series with the owning tenant instead of `cozy-monitoring`. Previously no tenant cluster metric reached the tenant's Monitoring.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

- **Bug Fixes**
- Monitoring agents now configure remote-write endpoints using a URL
list.
- Tenant labels use explicitly configured values when available, with a
fallback to the global target.
- Tenant and cluster labels, remote-write URLs, and external labels are
applied consistently across root and tenant-cluster deployments.

- **Tests**
- Added coverage for root and tenant-cluster monitoring configurations,
including remote-write settings and metric labels.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-10 13:36:27 +03:00
Aleksei Sviridkin ba0312b35f chore(kubernetes): align monitoring fix with current platform base
The previous base differs from the published package configuration,
so the image overlay retains release controllers instead of current
builds.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-10 11:45:36 +03:00
Aleksei Sviridkin 456473dec1 fix(mariadb): fail backup and restore jobs whose operator Job failed (#5207)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

When the Job behind a mariadb-operator `Backup` or `Restore` fails, the
operator sets `Complete=True` with reason `JobFailed`
(`SetCompleteWithJob` in mariadb-operator 25.10.2). The MariaDB driver
read `Complete=True` alone as success, so a backup whose dump or upload
failed ended `Succeeded` and produced a `Ready` Cozystack `Backup`
pointing at an archive that does not exist, and a failed restore ended
`Succeeded` too. A scheduled `Plan` kept producing such backups.

The driver now treats `Complete=True` as success only with reason
`JobComplete`, and fails the BackupJob or RestoreJob on `JobFailed`
whatever the status, without creating a `Backup`. The failure message
names the operator object and carries the Job's own `Failed` reason and
message, since the operator's condition message is only `Failed` and the
Job's pod, with its logs, is already deleted when the Job fails. A Job
that is missing or cannot be read is named in the message instead. Any
other `Complete` state keeps waiting under the existing deadline, as
before; that includes `Complete=False` with reason `Failed`, which the
operator sets with its own cause (`Error creating Job`, `Backup not
complete`) for a failure outside the Job and then retries, and the
deadline failure now carries that message.

Regression tests drive the backup and restore reconcilers through every
`Complete` status/reason combination, an absent `Complete` condition,
and expired deadlines for an absent condition, an unknown reason and
reason `Failed`. They assert the job phase and message as persisted in
the API rather than on the in-memory object, a requeue while the run is
in progress, and, for backups, that no `Backup` is created on failure.
Separate tests cover the failure message for a Job that cannot be read
and for a Job without a `Failed` condition.

Backups that already exist are not touched. A `Ready` MariaDB `Backup`
created by a failed run before this fix may have no archive of its own.
Restoring from it can fail or select another dump from the shared
prefix.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(mariadb): a MariaDB BackupJob or RestoreJob whose operator Job failed now ends Failed instead of Succeeded, and a failed backup no longer creates a Ready Backup without an archive. Backups created by failed runs before this fix are not removed; restoring from one can fail or select another dump from the shared prefix.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Backup and restore operations now report failed jobs as failures,
including details from the underlying job when available.
* Operations are marked successful only when the job completes
successfully. Other outcomes continue to be monitored until the existing
deadline, after which failures include available condition details.
  * Failed backups no longer create backup artifacts.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-10 11:21:19 +03:00
Myasnikov Daniil 2c1cb2f41c feat(release): drop an original from one line with run --drop
A fix the rules take for every line can still be wrong for one of
them, typically because it builds on code that line does not have.
The only ways to keep it out were backport/skip, which keeps it off
every line, and --defer, which holds it out of one batch: the next
batch tried it again, and a conflict that would never resolve came
back every run.

run --drop N --reason R records a drop in the line's record, bound to
the original's merge commit on main, and every later record carries
it, a new batch's included. What the open batch carries of a dropped
original comes off it the way --defer takes one off; any other drop
changes only the record, so a batch on a line that moved since stays
where it is. --undrop withdraws one. Both need an open batch, as every
write to the record does.

A drop comes after the rules and the carries: backport/skip, rules 2
to 4 or the line itself decide first and the drop stays on record,
and backport/include does not lift it. check notes a dropped original,
the body lists it, fold refuses it, and a record whose drop names
another original's merge stops every command. Each report keeps the
record's drops with the merge commit each is bound to and its reason,
held on read to the same rules, so an earlier report still says why
after an undrop.

A reason now travels from a body to terminals, bodies and comments, so
a reason or a file name in a record is believed only in the form a run
writes it, one clean line, and so is a report's diagnostic. Output
that repeats a reason, file names, a report ID or a state GitHub
reports escapes it.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-10 07:20:32 +00:00
Myasnikov Daniil 79b335e461 fix(release): close the batch --defer leaves empty, recording why while it is open
docs/release/backport-tasks/failed-ci.md says to defer the culprit with
run --defer --push. When the culprit was all the batch carried, the
push stopped with "close batch pull request #N instead", and closing
it by hand left nothing that said why: once closed, the deferred
original was simply missing.

The run now closes the batch itself, and does every write while the
pull request is open: the report, then the body, whose record keeps
the deferrals and says the batch is closing, at the head the rebuild
started from. The close and the deletion of the branch go only while
GitHub and the remote still show that head, and the deletion is
leased on it.

A close that stops part way is finished by the next run --push, before
anything else, and by no other command; a head the close did not
freeze is refused, never adopted. The run that finishes it stops
there, and fails when it was asked for --defer, since no open batch
is left to defer on. Recovery never writes the closed
pull request. A same-day batch that reuses the branch name and fails
to open its pull request leaves a branch the next run names for a
person to delete.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-10 07:20:00 +00:00
Myasnikov Daniil 5a997fb7d4 feat(release): post an immutable report of every classification and write record version 2
Each run rewrote the batch body whole, so the only account of what a
run found was gone at the next run, and a merged batch kept only its
last one. Nothing said what a classification was made of either: not
the heads it read, the policy, the tool or the flags.

A run now makes a report: every original merged into main since the
line branched, in exactly one category, with what the run read. Its
ID hashes the classification and names no pull request, so a new
batch's report is whole before GitHub numbers the pull request, and a
run that finds the same as the last one makes the same report, which
is posted once and keeps its first bytes. The body carries the latest
report under the record; a comment the tool never edits keeps each
one, so report reads any of them back, from merged batches too.

The comment goes up before the push and the body edit on an open
batch, and after the pull request a new batch opens with the report in
its body; a later run posts a report the body kept but no comment did,
byte for byte, before its own. Every write is preceded by a read of
the pull request, and nothing is written once it is merged or closed.
Both the comment and the body are rendered and measured before the
first write, so an oversized one stops the run with nothing written.

The record becomes version 2: what it was built from, with the run's
--no-workflow-files, which --defer keeps rather than resets, and the
report it goes with.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-10 07:20:00 +00:00
Andrey Kolkov 1093f390ae feat(backups): warn on the BackupJob when the Redis snapshot is empty
A valid RDB snapshot always carries a header, so a 0-byte object means
the backup is broken. sizeBytes is omitempty, though, so such a Backup
read exactly like one whose size was never measured. The controller
still records the artifact, and now raises a Warning EmptySnapshot
Event on the BackupJob once the Backup is created.

A test now also pins that a retry over an existing Backup returns it
without calling Create.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-10 12:18:07 +07:00
Andrey Kolkov 2013e624ca fix(backups): raise ArtifactNotMeasured only for a Backup this pass created
The Event was recorded before the Backup was created, so a Create that
then failed left the BackupJob Failed next to an Event saying the
backup succeeded, and a Create that collided with a Backup another pass
had made reported on that pass's Backup. The Event is now recorded once
Create succeeds. It also covers the one unmeasured path that had none:
a failed existence check, which skips the probe.

A test now also pins that a CA Secret key the s3-ca volume's items do
not project is never trusted.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-10 11:44:33 +07:00
Andrey Kolkov 6f3d26c491 feat(backups): tell on the BackupJob why a Redis artifact was not recorded
A Backup without status.artifact looked the same whether a custom
strategy wrote elsewhere, the credentials Secret was unusable or the
endpoint CA was gone; only the controller log told them apart. The
probe now returns why it gave up, and the controller records it as a
Normal ArtifactNotMeasured Event on the BackupJob. The Event lives in
the application namespace, so it carries a fixed reason only; the raw
error, which may name the platform's endpoint and bucket, stays in the
log.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-10 11:21:46 +07:00
Andrey Kolkov d61a5054a1 fix(backups): refuse to probe a Redis snapshot behind a CA that is gone
The default strategy mounts the s3-ca volume only when the admin
configured an endpoint CA, and the script then trusts that CA alone. When
the volume yields no CA by the time the probe runs, because the Secret
was deleted or emptied after the Job or the volume has another source,
the probe used to fall back to the system roots, trusting more than the
Job may have. It now leaves the artifact unset instead, with or without
skip-verify.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-10 11:21:46 +07:00
Andrey Kolkov 43bed8f6d9 fix(backups): keep the Redis probe verifying TLS unless every container skips
The probe read S3_INSECURE_SKIP_VERIFY from any container of the backup
Job, so a custom strategy whose sidecar sets it while the uploading
container verifies made the probe skip verification the upload did
not. The probe cannot tell which container uploaded, so it now skips
only when every container sets the variable to true, the last entry
winning as in the kubelet.

An s3-ca volume of a source other than a Secret yields no CA the probe
can read; it is now treated like a CA Secret that is gone, so it never
turns into skip-verify.

Tests pin the guards that decide the probe's trust: the Redis CA volume
mounted next to the S3 one, a decoy variable set to true, a CA that did
not sign the endpoint, and the AlreadyExists read-back past the cache.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-10 11:01:18 +07:00
Andrey Kolkov e03986e8d0 fix(backups): treat an S3 object without a reported size as unmeasured
minio-go returns size -1 when a HEAD answer carries no Content-Length,
which would land in Backup.status.artifact.sizeBytes as -1. The probe
now reports such an object as unmeasured, so the field stays unset.

The shared client's trust settings also gain a test against an endpoint
whose certificate the configured CA did not sign, with the CA alone and
on top of the system roots: nothing pinned that a configured CA is
actually enforced.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-10 11:01:18 +07:00
Andrey Kolkov 631f6cfc0d fix(backups): mirror the Redis script's CA trust exactly
The script passes the endpoint CA as curl --cacert, which replaces the
default trust store, while the probe trusted the system roots on top of
it. Trust only the CA, as the script does.

A mounted CA Secret that is still there but no longer carries ca.crt or
tls.crt let the probe skip TLS verification when the strategy opts into
it, although the Job may have verified against a key since removed.
Treat a volume that yields no CA now like a vanished Secret and refuse
to skip verification in its place.

Pin the probe timeout and the reconcile's retry of an unreadable Backup
for Redis with tests.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-10 10:52:45 +07:00
Andrey Kolkov 50b1a2e5ee test(backups): pin the Redis retry check
Cover the two retry-check paths the Redis driver shares with MariaDB and
CNPG: a failed existence check creates the Backup without probing the
bucket, and an existing Backup that cannot be read back is returned as
errBackupNotReadBack so the reconcile is retried.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-10 10:52:45 +07:00
Andrey Kolkov bfea468b6c fix(backups): never skip TLS verification in place of a vanished CA
The Redis probe reads the CA Secret when it runs, not when the backup
Job did. If the pod mounted a CA Secret that has since been deleted and
the strategy also opts into S3_INSECURE_SKIP_VERIFY, the probe skipped
verification against an endpoint the Job may have verified. Refuse to
probe in that case. Without skip-verify a missing CA Secret still falls
back to the system roots, as the script does.

The endpoint comment claimed a bare host; it can carry a path, which the
shared client now refuses.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-10 10:52:45 +07:00
Andrey Kolkov 2ff547348d fix(backups): probe the Redis snapshot with the trust its Job used
The default Redis strategy script verifies the S3 endpoint against
ca.crt, then tls.crt, from the Secret mounted as the s3-ca volume
(backupStorage.endpointCASecretName), and only without either honours
S3_INSECURE_SKIP_VERIFY=true. The probe always verified against the
system roots, so on a cluster with a private CA, such as a self-signed
SeaweedFS, every Redis and Valkey backup succeeded with no artifact.

Read the same settings from the pod spec of the completed backup Job, in
the order the script applies them, and pass them to the S3 connection.
The shared transport gains an InsecureSkipVerify switch that is set only
from that explicit opt-in.

The projector always stores the endpoint as a bare host, so the branch
that parsed a scheme out of it was unreachable; drop it and its test.
Cover the endpoint and access-key terms of the Secret completeness check.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-10 10:52:45 +07:00
Andrey Kolkov c8cdefc665 feat(backups): record the RDB snapshot location and size for Redis backups
Backups made by the Redis strategy, which also serves Valkey, carried no
Backup.status.artifact at all: neither a URI nor a size. The snapshot is
one object per backup, <namespace>/<app>/<backup>.rdb, whose key the
driver already records, and the default strategy uploads it with the S3
coordinates of the backup credentials Secret the controller projects
into the application namespace.

On completion the driver now stats that object with the same coordinates
and records its s3:// URI and size. The artifact is set only when the
object is found, so a custom strategy that writes elsewhere gets no URI
pointing at a missing object. A Backup that already exists is returned
without probing, as for the MariaDB and CNPG drivers.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-10 10:52:45 +07:00
Aleksei Sviridkin 5e9c392352 feat(api): add the reservation evaluator for tenant quotas (#4983)
## What this PR does

First phase of the accepted [tenant quotas as reservation
limits](https://github.com/cozystack/community/pull/48) design: a
standalone, importable `pkg/reservation` package that computes what an
application consumes from a tenant's budget from its declared values.
Nothing calls it yet; the usage oracle, the generalized admission gate
and the per-kind declarations come in the following phases.

Today a tenant quota is declared in instance-type units but enforced
against `ResourceQuota.status.used`, which counts virt-launcher pods
with their additive virtualization overhead and drifts without
recomputation. The design makes the declared reservation the accounting
authority instead, and this package is the evaluator both the apiserver
and the controller will read.

- **Declarations are data, not Go.** A kind describes its consumption as
CEL over its values (`exempt`, ordered `let` bindings, one expression
per quota key), so the evaluator holds no per-kind knowledge and a kind
from an out-of-tree catalog participates without a change here. The
`Declaration` type is ready to be embedded in
`ApplicationDefinitionSpec` in phase 2. Environment functions are
`instanceType(name)`, `preset(name)` and `quantity(x)`, on top of the
Kubernetes CEL quantity library; scaling is `quantity.mul(n)`, because
the standard `_*_` cannot take a quantity overload.
- **Instance types resolve to scalars keyed by class**, read from the
type's spec and never from its name: `cpu`, `dedicated-cpu` (guest cores
plus the isolated emulator thread, the extra parity core when the type
itself carries the `EmulatorThreadCompleteToEvenParity` annotation, and
a supplemental I/O pool), `memory`, `hugepages-<size>`,
`overcommitted-memory`. The test pins the classification against every
`VirtualMachineClusterInstancetype` shipped in
`packages/system/kubevirt-instancetypes`.
- **Fail closed.** An unknown instance type or preset, a value of the
wrong type, an unguarded absent key, an unparseable quantity or a
negative figure is an error, and values are checked against the kind's
schema before defaulting, which would otherwise replace a mistyped value
with its default, never a smaller reservation. A definition that
declares neither its consumption nor an exemption yields
`ErrNoDeclaration`.
- **Stored values are defaulted first.** The aggregated apiserver
persists a tenant's spec without its schema defaults, so an evaluator
reading them raw would count zero for `kubernetes-nodes.maxReplicas` on
every pool created with defaults. The structural-schema defaulting moves
from `pkg/registry/apps/application` into `pkg/apis/apps/defaulting` so
the registry and the evaluator apply the same defaults; the registry now
only delegates to it, with no behaviour change.
- **Aggregation** sums reservations per namespace and returns the
applications that could not be evaluated separately, so a caller never
mistakes an incomplete sum for a complete one.

The evaluator tests run against the shipped OpenAPI schemas of
`postgres`, `vm-instance` and `kubernetes-nodes`: the cozy-lib per-key
preset fill, a `{cpu: 4, sockets: 2}` block reserving 8 vCPUs, the
autoscaling ceiling, `external: true` reserving a load balancer, and a
ComputePlane-shaped `KubernetesNodes` release carrying only `{roles,
minReplicas}` reserving exactly what the same pool with its defaults
written out reserves.

Left for the next phases on purpose: where presets are resolved from
(the design's direction is platform-shipped environment objects; here
they come through the `Environment` interface), the
`ApplicationDefinition` field itself, and any consumer.

### Screenshots

Not a UI change.

### Downstream repositories

The diff adds two Go packages and moves the defaulting helpers without
changing behaviour. It touches no package under `packages/`, no
`values.yaml`, no CRD and no `ApplicationDefinition` field, so none of
the triggers in the map is reached.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(api): add the reservation evaluator that computes an application's tenant-quota consumption from its declared values. Internal only, with no consumer yet and no change in behaviour.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

- **New Features**
- Added reservation evaluation for application resources, including CPU,
memory, hugepages, and overcommitted memory.
- Added support for instance types, resource presets, schema-based
defaults, and CEL expressions.
- Added namespace-level reservation aggregation with clear reporting for
applications that cannot be evaluated.
- Added validation for invalid, negative, unsupported, or excessive
resource values.
- Added classification of KubeVirt instance types, including dedicated
CPUs, emulator threads, supplemental I/O threads, and hugepages.

- **Bug Fixes**
- Schema defaults are applied consistently to nested application values.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-10 03:26:01 +03:00