Commit Graph
844 Commits
Author SHA1 Message Date
scooby87 4f813fb6b2 fix(postgres): hold the WAL archive destination for backups no job tracks (#5294)
## What this PR does

Stacked on #5221; review that one first. Once it merges this PR
retargets to `main`.

After #5221 a Postgres BackupJob writes the shared WAL archive
destination only while no other running BackupJob of the application has
a `cnpg.io/Backup` in flight. A `cnpg.io/Backup` that no running
BackupJob tracks was not counted: one created by hand or by a
`ScheduledBackup`, or one whose BackupJob was deleted or had already
finished. A BackupJob using another BackupClass could then rewrite the
destination while CloudNativePG was running such a Backup, leaving its
base backup in one place and the WAL it needs in another, so the backup
could not be restored.

Such a Backup now holds the destination while CloudNativePG executes it,
that is in the `started` and `running` phases
(`BackupStatus.IsExecuting` in CloudNativePG 1.30). While a Backup of a
Cluster is in one of those phases, CloudNativePG's own
`BackupList.CanExecuteBackup` starts no other Backup of that Cluster, so
the hold never lasts longer than the queue the operator already imposes
and needs no separate deadline. A `pending` Backup does not hold,
including one that ran before and was sent back to `pending`:
CloudNativePG starts its base backup over against the destination
current at that time.

These Backups are found by `spec.cluster.name` in the controller's
cache, because a Backup created by hand need carry no label. The waiting
BackupJob reports `Ready=False` with reason `WaitingForUntrackedBackup`
and names the Backup.

### Screenshots

Not applicable: no UI change.

### Downstream repositories

The diff touches only `internal/backupcontroller`: no CRD, values, chart
or naming change, so nothing in the trigger map applies.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(postgres): a Postgres BackupJob no longer rewrites the WAL archive destination while CloudNativePG is running a backup of the same cluster that no BackupJob tracks, such as one created by hand or by a ScheduledBackup; it waits with reason WaitingForUntrackedBackup.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Prevented shared WAL archive destinations from being written while an
untracked backup is actively running.
* Pending and finalizing backups no longer hold the destination, and the
destination becomes available again after the active backup completes.
* Improved handling of backup status and retry conditions when multiple
backups are in progress.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-16 15:05:08 +03:00
Aleksei Sviridkin 46cb99df5b fix(controller, platform): give unbounded pool members the root's LimitRange (#5265)
## What this PR does

Since tenant quotas became hierarchical (#3086), an unbounded sub-tenant
(no `resourceQuotas` of its own) shares its parent's pool and receives a
controller-owned `tenant-quota-allocated` ResourceQuota. The tenant
chart, however, renders the `tenant-range-limits` LimitRange only next
to the `tenant-quota` of a tenant that declares `resourceQuotas`, so
these namespaces end up with a quota on `limits.cpu`/`limits.memory` and
no container defaults. Admission then rejects every container that omits
one of those limits.

Pods that predate the quota keep running and fail as soon as they are
recreated. In production this took down CNPG postgres instances with
backups enabled: the `plugin-barman-cloud` sidecar injected by the
barman-cloud plugin sets CPU and memory requests and a memory limit but
no CPU limit, so the instance pod is refused on its next restart. The
same happened to the tenant `kubernetes` app: the `wait-for-kubeconfig`
init container of `cluster-autoscaler`, `kcsi-controller` and the
`bootstrap-token` jobs has no resources, so their ReplicaSets and Jobs
were refused with `failed quota: tenant-quota-allocated: must specify
limits.cpu` in every unbounded sub-tenant hosting a tenant cluster. On a
production cluster, 131 tenant pods currently get their CPU or memory
from the tenant LimitRange (VictoriaMetrics
`config-init`/`config-reloader`, Harbor exporter, MariaDB agent, Strimzi
entity operator, among others), so the defect is not specific to one
chart: any of them breaks once it lands in such a namespace. Workloads
that set every limit explicitly are unaffected.

The controller already decides which namespaces it clamps, so it now
copies the pool root's chart LimitRange into each unbounded member whose
allocated quota bounds `limits.cpu` or `limits.memory` as
`tenant-range-limits-allocated`, keeps it in step with the root, and
deletes it once the namespace leaves the pool (for instance when the
sub-tenant declares its own quota and the chart renders its own
LimitRange). The chart renders its LimitRange under any non-empty
`resourceQuotas`, including one that bounds no compute (#4930), so the
copy is limited to members whose quota bounds a compute limit: elsewhere
it would impose a 128Mi default memory limit the quota never asked for
and refuse pods, virt-launcher among them, that are admitted today. This
keeps the change independent of #4954. The copy has its own name so that
Helm can still create the chart-owned object in that case without an
ownership conflict. The root is the single source of truth for the
defaults, so the values stay defined in
`packages/apps/tenant/templates/quota.yaml` only. The controller gains
`limitranges` RBAC and watches the two LimitRange names so a change
propagates without waiting for the 60 s resync.

Tests: five reconciler cases cover a member under a non-compute quota
(no copy, and a stale one removed), the copy into an unbounded member
(and not into the root), propagation of a change on the root, garbage
collection once the member is bounded (the chart-owned object is left
alone), and a root without a LimitRange. Disabling the copy makes the
first two fail.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff. The code touches only
`internal/controller/tenantquota/` and the `cozystack-controller` RBAC
template, but the website's `system-memory-limits.md` states that tenant
namespaces get defaults only from the chart's `tenant-range-limits`,
which this change makes untrue; the follow-up is an issue, since the
page should change once this is merged.

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#806
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(controller, platform): sub-tenants without their own resourceQuotas now receive their parent pool's container defaults (LimitRange) alongside the allocated quota, when that quota bounds a CPU or memory limit, so pods that omit such a limit, such as the postgres barman-cloud backup sidecar, are no longer rejected at admission.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Unbounded pool members now receive their pool’s CPU and memory limit
settings when their allocated quota sets those limits. Pool setting
updates are reflected in members during reconciliation.
* When a member has its own quota, inherited limit settings are removed
while settings configured directly for that member remain unchanged.
* No inherited settings are created when the pool has no limit settings
or the allocated quota does not set CPU or memory limits.
* Existing inherited settings are retained if an update temporarily
fails.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-15 02:33:34 +03:00
Maxim Belyy b24a9b4cb9 Merge branch 'main' into feat/observability-proxy-exported-namespace
Resolve the overlap with the per-application roots: the namespace root
keeps its platform exporter branch, the application root still replaces
the whole scope with its own selectors, and the label APIs mark the
repeated scope so that replacement reaches match[] as well.

Assisted-by: LLM
Signed-off-by: Maxim Belyy <maximbel2003@gmail.com>
2026-11-14 00:49:51 +07:00
Aleksei Sviridkin 4f7eb63447 feat(observability-proxy): serve per-application logs and metrics (#5220)
## What this PR does

Depends on #5219. This PR is stacked on that branch; the description
covers only what it adds on top.

Adds the per-application roots to the observability proxy, as described
in the design proposals [Tenant application
logs](https://github.com/cozystack/community/pull/95) and [Tenant
resource-consumption
metrics](https://github.com/cozystack/community/pull/94):

```text
/apis/apps.cozystack.io/v1alpha1/namespaces/<ns>/<plural>/<name>/log/select/logsql/<endpoint>
/apis/apps.cozystack.io/v1alpha1/namespaces/<ns>/<plural>/<name>/metrics/api/v1/<endpoint>
```

This lets the console show the logs and resource consumption of one
application rather than of the whole namespace.

What is added:

- Plural-to-Kind mapping from an informer over `ApplicationDefinitions`.
The Kind is looked up only after the `SubjectAccessReview` (`get
<plural>/log` or `get <plural>/metrics`), so an unauthenticated caller
cannot probe which plurals exist. An unknown plural answers `404`; an
unsynced cache or a plural declared by two kinds answers `503`.
- Logs: a second `extra_stream_filters` value with the application Kind
and name, on top of the unchanged namespace filter. VictoriaLogs ANDs
the two, on every endpoint including `tail`.
- Metrics: the forced `extra_filters[]` becomes an OR of three branches,
each still bound to the namespace:
  1. series carrying the application identity labels;
  2. series of the application's pods, found through `kube_pod_labels`;
3. series of the application's PVCs, found through
`kube_persistentvolumeclaim_labels`.
- The pod and PVC lookups cap the number of names
(`--metrics-app-max-pods`, `--metrics-app-max-pvcs`); over the cap
answers `422`. A failed pod or PVC lookup fails the request (`502`, or
`504` on the deadline) rather than answer without the application's
series, and so does a lookup that vmselect marks `isPartial`, since a
storage node that did not answer leaves names out (a partial list
already over the cap still answers `422`); vmselect's series-limit
refusal of a lookup is passed through as `422`. A query that reads
further back than `--metrics-app-lookback` (default 720h, chart value
`observabilityProxy.metrics.appLookback`), or by an unknown amount, and
a query whose samples lie at a time the lookup cannot cover answer `422`
on the application root; this includes every query with an `@` modifier,
`WITH` with a negative offset, and offsets that are not plain durations.
The namespace root still serves them. `rollup_candlestick`, a backslash
or a `#` comment outside a string, an offset directly followed by a
minus (compound durations such as `-1h-30m` or `1h-2h`), and a subquery
step that is not a plain duration are refused as well. The lookup
reaches past the evaluation time by any negative offset and by two steps
per subquery level.
- Application routes require read access to the namespace root as well
as the application subresource; a grant narrowed with `resourceNames`
alone answers `403`.
- The proxy's ClusterRole gains `list`/`watch` on
`cozystack.io/applicationdefinitions`. No tenant RBAC change is needed:
the `view` role already allows `get` on every `apps.cozystack.io`
resource and subresource.
- `/readyz` now also waits for the `ApplicationDefinitions` cache, so a
missing CRD or RBAC rule keeps the whole proxy unready, namespace roots
included.

#### Data comes from the identity labels

The application roots find data only through identity labels added by
three PRs that are now in main:

- #5215 `feat(monitoring-agents)`: export PVC labels from
kube-state-metrics (PVC branch).
- #5216 `feat(monitoring)`: stamp application identity on scraped series
(identity branch).
- #5217 `feat(monitoring-agents)`: keep application identity in
container log streams (logs root).

Merge after #5219.

The pod branch does not depend on them, because `kube_pod_labels`
already exports all pod labels. Logs and series written before those
changes rolled out carry no identity and are reachable only through the
namespace root.

#### Security condition: a convenience filter, not an isolation boundary

The application roots narrow the namespace roots; they do not isolate
applications from each other. Each application route therefore requires
two grants: the application subresource (`get <plural>/log` or `get
<plural>/metrics`) and read access to the namespace root (`list
cozystack.io/logs` or `cozystack.io/metrics` in the namespace). A grant
narrowed with `resourceNames` alone is refused. Every tenant role that
can read applications already has both. The application scope cannot
isolate yet because:

1. The lineage webhook keeps `apps.cozystack.io/application.*` labels
that a user sets on a pod or PVC that has no application owner, so such
an object can claim another application's identity.
2. The webhook's `objectSelector` skips any object created with
`internal.cozystack.io/managed-by-cozystack`, pods included, so forged
labels on such an object are never examined.
3. The pod and PVC branches match by name over the lookup window, so an
object that reuses the name of another application's recently deleted
pod or PVC pulls that object's older series into scope.

The proposed fix (a separate change to the lineage webhook) is to strip
the identity labels when no owner resolves, and to stop skipping CREATE
requests for objects carrying the managed-by label.

#### Other known gaps

- PVCs created from a StatefulSet's `volumeClaimTemplates` usually get
no lineage labels, so their volume series are not in the application
scope.
- ingress-nginx series intentionally carry no application identity, so
they are reachable only through the namespace root.
- A request for an application that does not exist answers `200` with an
empty result rather than `404`: the proxy checks authorization, not
existence. Nothing leaks; whether a `404` is wanted is open for review.

#### How it was tested

- `go test -race` and `go vet` for the proxy packages, including a
narrowed-grant test (access to application `a` only): every application
path for `b` answers `403` and no request reaches the store. Name
escaping in the pod and PVC branches is covered for regex metacharacters
and quotes.
- helm-unittest for `observability-proxy` (new RBAC rule).
- Verified on a test cluster, with the three identity PRs above applied:
the application logs root of a Redis instance returned only that
instance's lines and none from a neighbouring pod in the same namespace;
its application metrics root returned only the Redis pods' series.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Downstream: no trigger matched. Candidate: none affected, because the
diff on top of #5219 is confined to the proxy's Go code and its own
chart RBAC; it reads `ApplicationDefinitions` but does not change their
semantics or the CRD. The security condition above may deserve a note in
the design proposals in `cozystack/community`, which is not a
trigger-map item.

### Release note

```release-note
feat(observability-proxy): add per-application logs and metrics roots under `/apis/apps.cozystack.io/v1alpha1/namespaces/<ns>/<plural>/<name>/{log,metrics}/`, scoped by application identity labels. Data appears for logs and series collected after the application identity changes to Fluent Bit, VMAgent and kube-state-metrics are rolled out.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Added a tenant-aware observability proxy for logs and metrics, with
namespace and application scoping, access checks, and per-user request
limits.
* Enabled the proxy in the system package bundle and routed console log
and metrics requests through it.
* Added live log streaming and health, readiness, and metrics endpoints.
* **Bug Fixes**
* Invalid or excessive query parameters are rejected; requests with
uncertain application-metric scope are refused rather than returning
incomplete results.
* **Tests**
* Added end-to-end and integration coverage for tenant isolation,
console routing, access controls, and log streaming.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 20:47:10 +03:00
Aleksei Sviridkin 526f08e17d fix(mariadb): restore the dump a Backup was taken as (#5227)
## What this PR does

A RestoreJob that names a MariaDB `Backup` restored the newest dump of
the application, not the dump of that `Backup`. The driver created the
operator `Restore` with only `backupRef`, and mariadb-operator then
restores the dump nearest to the moment the restore starts, from the
prefix every run of the app shares. Restoring an older `Backup` brought
back the latest data, and when its own dump was already gone it silently
restored another one.

The `Backup` now records the dump its operator `Backup` wrote, the file
name in `k8s.mariadb.com/dump-file` and the timestamp from that name in
`k8s.mariadb.com/dump-time`. They come from the ownership-proven match
the size probe already makes, and nothing is recorded when ownership is
not proven. A restore first checks with a bounded S3 listing of the
operator `Backup`'s own storage that the dump still exists, then sets
`targetRecoveryTime` on the operator `Restore` to the dump's own
timestamp, which mariadb-operator matches exactly. If the dump is gone
the RestoreJob fails and names it. If the check cannot be made it waits
with `Ready=False/DumpCheckPending` until the restore deadline and then
fails with the cause.

`Backup`s taken before this stay restorable. The restore makes the same
ownership-proven match against their operator `Backup` and pins the dump
when it is provable, otherwise it restores the newest dump as before and
a `BestEffortDump` warning event on the RestoreJob says so. One case is
not covered: the `Backup`'s own dump is gone, a neighbour's dump from
within the minute of slack is still there and its operator `Backup` was
deleted, so that dump is taken as the match. Nothing in the platform
deletes operator `Backup`s, it needs someone to delete one by hand. A
`Backup` made of an operator `Backup` whose Job failed (`Complete` with
reason `JobFailed`) has no dump, so its RestoreJob now fails instead of
restoring another run's dump.

Stacked on https://github.com/cozystack/cozystack/pull/5223 and targets
its branch until that merges, only the last three commits belong here.

Checked on a test cluster with mariadb-operator 25.10.2. An operator
`Restore` of an older `Backup` with `targetRecoveryTime` set to its dump
timestamp restored exactly that dump (`obtained target backup` names it
and a row inserted after it is absent from the target), the same
`Restore` without the target restored the newest one. Against the real
bucket and operator `Backup`s the recording probe attributed to each
`Backup` the dump its backup pod logged, the existence check passed for
those dumps and refused a pruned one, legacy `Backup`s with a provable
dump got pinned while back-to-back runs fell back with the warning, and
the two failed runs on that cluster were refused. The controller itself
was not run on the cluster, only its code driven by a test against the
real bucket.

Fixes https://github.com/cozystack/cozystack/issues/5196

### Screenshots

Not a UI change.

### Downstream repositories

The diff is confined to the backup controller's Go code. It changes no
CRD or API type (`driverMetadata` is an opaque map), no chart, values,
package layout or `hack/` script, so no entry of the trigger map
applies.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(mariadb): a RestoreJob now restores the dump of the Backup it names instead of the newest dump of the application. It fails with a clear message when that dump no longer exists in storage, or when the Backup was made of a failed backup run. Backups taken before this release are restored by the one dump that can be matched to them, or else, as before, by the newest dump, with a BestEffortDump warning event on the RestoreJob.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* MariaDB backups now record the selected dump’s filename and timestamp,
allowing restores to target the corresponding recovery time.
* For older backups without dump metadata, restores use a uniquely
matched dump when possible or fall back to the latest dump with a
warning.
* Restore processing verifies recorded dumps, retries temporary storage
checks, and reports failures for invalid or missing recorded dumps.
* **Bug Fixes**
* Deleting a MariaDB backup now removes its associated dump and operator
backup when ownership is verified and no restore is active. Cleanup
avoids deleting artifacts when it cannot safely identify them.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 20:46:45 +03:00
Aleksei Sviridkin 6f8171db89 fix(workloadmonitor): report a bucket's workloads not ready until its claim is provisioned (#5096)
## What this PR does

A Bucket app reports WorkloadsReady=True while its BucketClaim is never
provisioned. On a v1.6.5 cluster a tenant got a BucketClaim for bucket
class `tenant-root`, the root tenant there runs no SeaweedFS, and for
three days the claim had no status while the objectstorage controller
kept emitting `FailedCreateBucket` with
`bucketclasses.objectstorage.k8s.io "tenant-root" not found`. The Bucket
app showed Ready and WorkloadsReady the whole time.

The WorkloadMonitor controller already writes a Workload per BucketClaim
with `Operational` set from `status.bucketReady`, but the monitor itself
counts only pods. The bucket chart renders its monitor with `replicas:
0` and `minReplicas: 0`, so the monitor stays operational whatever
happens to the claim.

The monitor now names every selected BucketClaim that is not provisioned
in `status.message`, sets `status.reason` to the new
`BucketClaimNotReady` and turns `Operational` false, the same way it
already reports a DataVolume that is not ready. A DataVolume verdict
keeps the fields when both apply. BucketClaims are listed on every pass,
so a stored BucketClaim verdict is recomputed rather than kept like an
unread DataVolume verdict, and it clears once the claim is provisioned.
Any monitor whose selector picks a BucketClaim follows its readiness,
which is what the Workload objects already say. The Application then
shows WorkloadsReady=False with that message and reason. Its Ready
condition keeps following the HelmRelease, as
`pkg/registry/apps/application/rest.go` intends.

Three new tests cover a provisioned claim, a claim not provisioned with
and without a class, a claim outside the selector, the verdict clearing
after provisioning with and without a DataVolume reader, and the
DataVolume verdict winning over the BucketClaim one. Against the
controller on main the four not-provisioned and clearing cases fail and
the rest pass. `go test ./internal/controller/...` and `go vet` pass.
`make generate` does not run on my machine, gengo fails to parse the
go1.27.1 standard library, so I ran the controller-gen `object` and
`crd` steps of `hack/update-codegen.sh` for `./api/v1alpha1/...`
directly. The deepcopy file is unchanged and the generated
WorkloadMonitor CRD matches
`packages/system/cozystack-controller/definitions/cozystack.io_workloadmonitors.yaml`.
I have not run the change on a live cluster.

#5083 changes the Workload names of the same BucketClaims and does not
touch the monitor status, so the two do not overlap.

### Screenshots

Not a UI change.

### Downstream repositories

The controller, one exported constant and its tests. No CRD schema,
values key, default, telemetry metric or package change, so no trigger
map entry applies.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(workloadmonitor): a Bucket app reports WorkloadsReady=False with reason BucketClaimNotReady while its BucketClaim is not provisioned, instead of WorkloadsReady=True.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Workload monitors now report when a selected bucket claim is not
provisioned and mark the monitor as non-operational. Status messages
include the bucket class when available and list multiple unprovisioned
claims in sorted order.
* **Bug Fixes**
* The bucket-claim status clears when the claim is provisioned or no
longer selected. When DataVolume information is available, its status
takes precedence over the bucket-claim status.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 18:11:43 +03:00
Aleksei Sviridkin f12300943d chore: clear golangci-lint findings and handle the errors they flagged (#5306)
## What this PR does

This clears the golangci-lint findings on main and fixes the errors
several of them pointed at. With the default linter set (errcheck,
govet, ineffassign, staticcheck, unused), main had 197 findings when
this branch started. Rebased on main at 0f169540, the branch leaves 38,
all deprecations that need a design decision rather than a lint fix;
they are listed below.

Most commits are mechanical. They remove unused helpers and constants,
reword error strings, take the staticcheck simplifications, discard
best-effort cleanup errors explicitly, and move to the FieldsV1 and ptr
accessors. A loop in pkg/lineage that always ended in its first pass
becomes a helper, with a regression test for each fallback. These change
behaviour:

- Lost output is now a failure for the commands this PR touches: cozypkg
list, the validation report of validate, push and tap, the push status
line, the tap summary, and check-readiness. The command exits non-zero
and says whether the action it reports on already happened. A closed
pipe ends the process with SIGPIPE, as for other Unix tools.
- backport-batch writes its output before it pushes or publishes. Once a
write to stdout or stderr fails, the run changes nothing more: no push,
no GitHub write, no local branch move, and no new pick. A pick already
in flight still finishes its rollback. The run exits 2, and it catches
SIGPIPE, so a closed pipe is handled the same way. A confirmation
printed after an action is lost only when stderr itself fails, so a lost
confirmation can only be reported by the exit code.
- The SSE server drops a client as soon as a write to it fails. A stream
still open at shutdown no longer panics on a double close, the handler
waits for its pump before returning, and a stream that opens during
shutdown gets 503.
- The kube-ovn plunger now fails its manager when the status server
cannot bind :18080, as a metrics server bind failure does. Before, it
ran without the endpoint and logged nothing.
- A timed-out check-readiness --wait still ends with its verdict, and
the help lists the exit statuses.

The 37 findings left are SA1019 deprecations, kept out of this PR on
purpose:

- GetEventRecorderFor (12, in cmd/*/main.go). The events.k8s.io recorder
has a different interface and needs RBAC for that API group. It changes
what events look like, so it needs its own design.
- client.Apply (12, one more site arrived from main). Moving to
Client.Apply with apply configurations changes how every server-side
apply call is built and how the fake client in tests handles it. It
overlaps the dependency rollup in #4869 and should come after it.
- Result.Requeue (10). Requeue retries with rate-limited backoff,
RequeueAfter with a fixed delay. The interval has to come from whoever
owns each controller.
- scheme.Builder (3, in api/*). Replacing it changes the type of the
exported SchemeBuilder that other repositories import.
- cel.Functions (1, in pkg/reservation). It stays for its late binding,
as the comment next to it explains.

Each behaviour change has tests that failed before the fix. I ran go
test on every changed package (for cmd/backport-batch, the tests around
output and publishing, since the full package takes about ten minutes),
go build ./..., go vet on the changed packages, and a final
golangci-lint run with uncapped output on the rebased tree:
`golangci-lint run --verbose --max-same-issues=0
--max-issues-per-linter=0`, which reports the 38 findings above and
nothing else.

The tests in pkg/lineage, pkg/ovnstatus, pkg/reservation, cmd/cozypkg
and cmd/check-readiness now run in CI: a separate commit adds those
packages to go-unit-tests. Some packages with tests are still in no CI
target: cmd/cozystack-controller, cmd/cozystack-operator,
cmd/observability-proxy, pkg/apiserver, pkg/apis/apps/defaulting,
pkg/apis/apps/install, pkg/apis/apps/presets, pkg/apis/apps/validation,
pkg/apis/core/install, pkg/apis/sdn/install and pkg/version. That
predates this PR.

One known nit: when backport-batch refuses an action after its output
was lost, stderr can show two "output was lost" lines.

### Screenshots

No UI change.

### Downstream repositories

I walked the trigger map against the diff. cozypkg counts as developer
tooling, but only its error paths changed, not a flag, its usage or its
output format, so the development page needs no update. Telemetry metric
names and labels are unchanged.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
chore: cozypkg, check-readiness and backport-batch now fail when their output cannot be written, and backport-batch pushes and publishes nothing once that happens. The kube-ovn plunger fails when its status server cannot start, drops status clients whose writes fail, and no longer panics when it shuts down with a client connected.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Commands now report output failures instead of silently succeeding.
Backport operations stop before further external changes when required
output cannot be written; readiness checks return a failure status when
output writing fails.
* Package listings, validation reports, push status messages, and tap
summaries now report output errors. Tap commands indicate when changes
were applied before a summary could be written.
* Live event streams handle failed writes and shutdown more reliably,
and status-server startup errors are reported.

* **Documentation**
* Readiness-check help now explains exit statuses and closed-pipe
behavior.

* **Improvements**
* Error messages for several backup and inventory operations now provide
clearer context.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 16:44:32 +03:00
Aleksei Sviridkin 5d453c4d23 test(backups): give the fake S3 bucket a StatObject
main added StatObject to the object lister interface, so the fake
bucket the MariaDB cleanup tests use has to implement it to compile.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 16:33:57 +03:00
Aleksei Sviridkin 8a37a70681 Merge branch 'main' into fix/mariadb-restore-dump
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 16:33:50 +03:00
Aleksei Sviridkin 03a2edf3db Merge branch 'main' into feat/observability-proxy-app-scope
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 16:31:05 +03:00
Aleksei Sviridkin 17bbca2088 feat(observability-proxy): serve tenant namespace metrics (#5219)
## What this PR does

Depends on #5214. This PR is stacked on that branch; the description
covers only what it adds on top.

Backward compatibility: this adds a new package and a new HelmRelease;
nothing existing is renamed, removed or changed, so no migration is
needed.

Adds the namespace metrics root to the observability proxy, as described
in the design proposal [Tenant resource-consumption
metrics](https://github.com/cozystack/community/pull/94). A tenant user
can query the metrics of their own namespace from VictoriaMetrics with
their Kubernetes credentials, without seeing other tenants' series.

What is added:

- Serves
`/apis/cozystack.io/v1alpha1/namespaces/<ns>/metrics/api/v1/{query,query_range,series,labels,label/<name>/values}`,
GET and POST. Anything else under the root answers `404`.
- Authorization by one `SubjectAccessReview` (`list` on
`cozystack.io/metrics` in the namespace); the aggregated tenant `view`
role gains that rule.
- The upstream request is rebuilt from an allowlist (`query`, `start`,
`end`, `step`, `time`, `match[]`, `timeout`). The scope is a single
forced `extra_filters[]={namespace="<ns>"}`; client-supplied
`extra_filters`, `extra_filters[]` and `extra_label` are dropped,
because vmselect would OR them with the forced filter. The query itself
is not rewritten.
- Label endpoints always carry a `match[]` (the namespace scope when the
client sent none), so they stay scoped even if vmselect's
`-search.ignoreExtraFiltersAtLabelsAPI` is ever enabled.
- Cost limits: maximum range (30 days), maximum points (11000), and a
clamped `timeout` (30 seconds); query-shape limits answer `422`, the
concurrency limit answers `429`. Times are accepted as unix seconds or
RFC3339; durations are rounded up to whole milliseconds before limits
are checked.
- Store selection between the short-term and long-term VictoriaMetrics
stores, configured by new chart values (`observabilityProxy.metrics`).

#### Store selection: a known limitation and an open question

The proxy has to send each query to the short-term store (3 days of
retention, finer deduplication) or the long-term store. The choice
depends on how far back the query reads, which is not only the request
window but also the windows, subquery steps and offsets inside the
expression.

The current implementation estimates that lookback with a conservative
scan of the expression text, not with a parser. It sums every `[...]`
window and subquery step, every positive `offset`, and allowances for
implicit subqueries and vmselect's default step and latency offset.
Anything it cannot read with certainty (`@`, `WITH` templates, step
multiples such as `5i`, template variables, unusual duration syntax)
goes to the long-term store. The query is forwarded unchanged, so the
estimate affects only the store choice, never the scope.

This is a heuristic. It is designed to err only towards the long-term
store, which holds the same recent data at coarser deduplication, so a
wrong guess returns coarser data rather than missing data in most cases.
A very short `rate` window sent to the long-term store can still come
back empty. Two review rounds each found new expressions the scan
underestimated, which were then fixed; there may be more.

Open question for reviewers: should the scan be replaced by parsing the
expression with `github.com/VictoriaMetrics/metricsql` (the library
vmselect itself uses), only for the store choice? That gives an exact
lookback at the cost of a new Go dependency. The namespace scope would
still be enforced by vmselect through the single `extra_filters[]`, so
the isolation design does not change either way.

A user-visible consequence of counting the lookback: a range whose start
sits right at the short-term retention edge (for example "last 3 days")
now goes to the long-term store.

#### How it was tested

- `go test -race` and `go vet` for the proxy packages; table tests for
the lookback estimate, including mutation checks of the `@` and
unparseable-window cases.
- helm-unittest for `observability-proxy` (new args and the tenant role
rule).
- Verified on a test cluster: `401` and `403` where expected; results
contain only the caller's namespace; a query naming another tenant's
namespace and a client-supplied `extra_filters[]` return nothing extra;
`query_range`, `label/<name>/values` and `series` are scoped; an admin
endpoint answers `404`.
- Not verified: the limits under load, and the store choice on windows
longer than 3 days (the test cluster had less than an hour of data).

#### Known limitations

- Relative times (`start=-1h`, `end=now`) are refused with `400`, while
the logs root accepts durations. The console sends unix seconds, so it
is not affected.
- The proxy bounds the evaluation window, the step, the number of points
of `query_range` and the timeout, not how far an expression reads:
windows and offsets inside the expression (`up[10y]`, `x[30d:1ms]`,
`offset 20d`) are bounded only by the store's retention and vmselect's
own limits.
- Times and durations are sent in a form vmselect parses exactly
(RFC3339 with milliseconds for times, durations that truncate to the
checked milliseconds); times outside 1970..2262 are refused with `400`.
A test repeats vmselect's parsing of the exact strings the proxy sends.
- `--metrics-max-range` defaults to 30 days while the longterm store
keeps 14, so the older part of a long range comes back empty rather than
refused. The two should be aligned or documented.
- The store estimate errs towards longterm: deeply nested expressions
with a large step can go to longterm although shortterm holds the data,
and longterm deduplicates at 5 minutes, so short windows such as
`rate(x[1m])` can come back sparse there.
- The `namespace` label is only as trustworthy as the writer. Any pod in
a child tenant may write to the vminsert of every ancestor tenant
(`packages/apps/tenant/templates/networkpolicy.yaml`), and vminsert does
not authenticate writers, so a child tenant can write series labelled
with another tenant's namespace. Through this proxy the owner of that
namespace would see those series as its own, and a flood of them could
push its queries over the series limit. Reading another tenant's data is
not possible this way. The write path predates this PR and already
affects the root tenant's dashboards and alerts; closing it needs the
write side to mark untrusted series, for example a vmauth in front of
vminsert that forces an `extra_label` the proxy then excludes.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Downstream: no trigger matched. Candidate: none affected, because the
diff on top of #5214 is confined to the proxy's Go code and its own
chart under `packages/system/observability-proxy` (no platform values,
no package under `packages/apps/`, no CRD, no `hack/` change). A
user-facing page for the metrics API on the website would be useful but
is not required by the trigger map.

### Release note

```release-note
feat(observability-proxy): let tenant users query their namespace metrics from VictoriaMetrics with their Kubernetes credentials, under `/apis/cozystack.io/v1alpha1/namespaces/<ns>/metrics/api/v1/`. Access requires the tenant `view` role or higher. Ranges are limited to 30 days and 11000 points.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* The observability proxy now supports tenant-scoped metrics queries
alongside logs.
* Metrics requests are routed to short- or long-term stores based on
their time range, with configurable retention and query limits.
* Metrics access is subject to tenant authorization, request validation,
and concurrency limits.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 16:26:22 +03:00
Mattia Eleuteri cc412d44f2 test(controller): exercise the missing-root-LimitRange path under a compute-limit quota
TestReconcile_PoolRootWithoutLimitRange declared its root quota on cpu,
which the compute-limit gate does not count, so the sweep never looked
for the root's LimitRange and the test passed whatever that path did.
Declaring limits.cpu makes it reach the lookup, and it now fails if a
missing root LimitRange produces an empty copy.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-11-13 13:07:08 +01:00
Aleksei Sviridkin 7c46e1e384 fix(backups): give the barman-cloud sidecar an explicit CPU limit (#5280)
## What this PR does

The barman-cloud sidecar of Postgres backups is deliberately left
without a CPU limit, so that a throttled sidecar slows WAL archiving
down instead of failing it. In a tenant with `resourceQuotas` set that
intent does not hold: `packages/apps/tenant/templates/quota.yaml`
creates the `tenant-range-limits` LimitRange, whose default CPU limit is
`250m`, and the sidecar receives it. Full backups and WAL segments are
then gzip-compressed on a quarter of a core. On a production tenant the
first full backup of a 1.6 GB database took 30 minutes, uploading at
about 0.5 MB/s with the sidecar pinned at its 250m limit, while a small
database on the same destination finished in 35 seconds.

This sets an explicit `limits.cpu: "1"` on the sidecar, in the two
places that build its configuration: `barmanSidecarConfiguration` in the
backup driver (platform-managed ObjectStores) and
`cozy-lib.barman.sidecarConfiguration` (ObjectStores rendered by the
postgres and keycloak charts). This mirrors how the memory request and
limit there already override the LimitRange's 128Mi default. One core
removes the 250m ceiling without letting the sidecar starve the
instance. Inside a tenant it counts against the `limits.cpu` quota, one
core per instance pod; the comments now say so.

The sidecar is injected when the instance pod is created, so existing
instances pick the new limit up at their next restart.

Tests: the Go test that asserted the absence of a CPU limit now asserts
one core, and so does
`packages/apps/postgres/tests/barman_resources_test.yaml`. Both fail
with the implementation reverted and pass with it. `go test
./internal/backupcontroller/`, `helm unittest packages/apps/postgres`
(220) and `helm unittest packages/system/keycloak` (121) pass.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff (the backup driver, `cozy-lib`'s
barman template, two tests): no app, value, schema, default, CRD,
annotation or tooling change, so none of the listed repositories
restates anything touched here.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(backups): the barman-cloud sidecar of Postgres backups now has an explicit one-core CPU limit instead of inheriting the tenant LimitRange's 250m default, which made full backups and WAL archiving several times slower. Each instance pod of a tenant with a CPU quota now counts 750m more against `limits.cpu`; check the headroom before instance pods are recreated. Existing ObjectStores pick the limit up at their next backup (platform-managed) or HelmRelease upgrade (chart-rendered), and running pods at their next restart.
```
2026-11-13 14:54:51 +03:00
Aleksei Sviridkin e8c8fa3ec7 Merge branch 'main' into fix/workloadmonitor-bucketclaim-ready
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 14:07:23 +03:00
Mattia Eleuteri e6127f7e42 fix(controller, platform): copy the LimitRange only under a compute-limit quota
The tenant chart renders tenant-range-limits under any non-empty
resourceQuotas, including one that bounds no compute, such as
services.loadbalancers alone. Copying it into an unbounded member whose
allocated quota bounds neither limits.cpu nor limits.memory imposes a
128Mi default memory limit the quota never asked for: a container that
requests more and sets no limit, virt-launcher among them, is refused
where it was admitted before. Only a quota on limits.cpu or
limits.memory makes admission reject a container that omits a limit, so
the copy is now limited to that case, and removed otherwise.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-11-13 10:42:20 +01:00
Aleksei Sviridkin 6cab97f010 style: use tagged switches and drop unused test assignments
Use tagged switches where staticcheck suggests them and drop two test
assignments whose values are never read. Behaviour is unchanged.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 12:35:10 +03:00
Aleksei Sviridkin 6f2d6ed17b fix(sse): refuse streams that open while the server shuts down
Shutdown stopped the clients registered at that moment and then shut
the HTTP server down. A stream accepted in between registered after the
stop loop, so nothing ended it, and http.Server.Shutdown waited for its
handler. The kube-ovn plunger calls Shutdown without a deadline, so
such a stream delayed stopping until the manager's graceful-shutdown
timeout and then failed it. Shutdown now marks the server as closing
under the same lock, and a stream that opens after that is answered 503
without being registered.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 12:35:06 +03:00
Aleksei Sviridkin 40d340a3fc fix(sse): wait for the pump before returning
When a client disconnected, the handler stopped its pump and returned at
once, while the pump could still be writing an event or a heartbeat to
the response. net/http does not allow a write to a response after
ServeHTTP has returned. The handler now waits for the pump to exit; a
pump blocked in a write returns once the connection is gone.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 12:35:06 +03:00
Aleksei Sviridkin 86da6c7a5b style(lineage-webhook): prefix sentinel error names with Err
Go names sentinel error variables Err...; the package is internal and
the webhook is their only user.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 12:35:05 +03:00
Aleksei Sviridkin 648928f272 refactor(workloadmonitor): use k8s.io/utils/ptr instead of pointer
The pointer package is deprecated; its Bool and BoolPtr are aliases of
ptr.To, so the values written are unchanged.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 12:35:04 +03:00
Aleksei Sviridkin 939a1ec44b refactor: read managed fields through FieldsV1 accessors
Direct access to FieldsV1.Raw is deprecated; GetRawBytes returns the
same bytes and is nil-safe, and NewFieldsV1 builds the test fixtures.

The write-options test still covers a raw-only orphanDependents delete
option, now decoded from its wire form, and asserts that the converted
options keep it.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 12:35:03 +03:00
Aleksei Sviridkin 7a481c4765 chore(workloadmonitor): discard best-effort close errors explicitly
The Prometheus response body is closed after it has been read, and the
test stubs' writes fail on the client side of the test, so neither
error changes a result.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 12:35:01 +03:00
Aleksei Sviridkin 87b794a992 chore: handle errors dropped in cleanup paths and tests
The drain tests now assert that RunThenDrain returns the run's nil
error, the idle-tail test fails on any error other than the stream
ending or being cut, and a test fixture file fails the test when it
cannot be closed.

Removing a temporary directory and closing a response body that has
been read are best-effort: a failure leaves nothing for the caller to
act on and does not change any result, so those errors are discarded
explicitly. The same holds for the stub servers' response writes, whose
failure surfaces on the client side of the test.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 12:35:01 +03:00
Aleksei Sviridkin bd998d9653 fix(kube-ovn): fail the plunger when its status server cannot start
The status stream was started in a bare goroutine whose error was
dropped, and the runnable's registration error was discarded too, so a
port that could not be bound left the plunger running with no status
endpoint and nothing in its logs. The server now runs inside its
runnable: a listen error fails the manager, as a metrics server bind
failure does, and stopping the manager is not reported as an error.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 12:35:00 +03:00
Aleksei Sviridkin 41ad82842a fix(sse): drop a client as soon as a write to it fails
Write errors on the event stream were ignored, so a client whose
connection broke kept its handler, pump and registration until the
request context ended, and every event was written into the dead
connection meanwhile. A failed write now ends the stream and
unregisters the client, and the handler also returns once its pump
ends, so Shutdown no longer leaves streams open.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 12:34:59 +03:00
Aleksei Sviridkin 20ce6c691e fix(sse): close a client's channel once when the server shuts down
Shutdown closed every open client's channel, and the client's handler
closed the same channel again when its request ended, so any stream
still open at shutdown panicked with a close of a closed channel.
Stopping a client is now idempotent.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 12:34:58 +03:00
Maxim Belyy 473cfdf478 Merge branch 'main' into feat/observability-proxy-exported-namespace
Assisted-by: LLM
Signed-off-by: Maxim Belyy <maximbel2003@gmail.com>
2026-11-13 16:07:33 +07:00
Maxim Belyy a4ea942a2c Merge branch 'main' into feat/observability-proxy-app-scope
Assisted-by: LLM
Signed-off-by: Maxim Belyy <maximbel2003@gmail.com>
2026-11-13 16:07:33 +07:00
Maxim Belyy 34843c1a91 Merge branch 'main' into feat/observability-proxy-metrics
Assisted-by: LLM
Signed-off-by: Maxim Belyy <maximbel2003@gmail.com>
2026-11-13 16:07:33 +07:00
Andrey Kolkov 906eeb2372 feat(backups): add Kubernetes backup strategy for tenant clusters (#4931)
## What this PR does

Adds a `Kubernetes` backup strategy so a tenant Kubernetes application
can be backed up and restored through the backups framework, and routes
`apps.cozystack.io/Kubernetes` to it in `cozy-default`.

None of the existing strategies fits a tenant cluster:

- Its API state is a key prefix (`/<namespace>_<release>/`, the
TenantControlPlane's datastore schema) in the etcd `DataStore` shared by
every cluster of the tenant. The `Etcd` strategy snapshots the whole
DataStore, so restoring it would roll every tenant cluster back at once.
- Its control-plane CA, front-proxy CA, service-account key and Talos
secrets cannot be regenerated: new ones break every node, kubeconfig and
token issued under the old ones.
- Its PersistentVolumes are DataVolumes in the tenant namespace
(`pvc-<tenant pvc uid>`, labelled `cluster.x-k8s.io/cluster-name`), and
deleting the application deletes them.

**Backup** (`strategy.backups.cozystack.io/Kubernetes`, driver in
`internal/backupcontroller/kubernetesstrategy_*.go`):

1. Checks the control-plane and Talos certificates and sets a
`CertificatesExpiring` condition (plus a Warning event) when a CA
expires before the backup's TTL or a leaf within
`certificateWarningThreshold` (720h). This never fails the run.
2. A Job in the application namespace dumps the cluster's prefix with
`etcdctl` to the platform bucket. It authenticates with the cluster's
own `kubernetes-<name>-datastore-certificate`, the one its
kube-apiserver uses. It runs on a new `kubernetes-backup` image (alpine
+ curl + jq + etcdctl matching the platform's etcd).
3. A Velero backup takes the PKI Secrets (selected by
`kamaji.clastix.io/name` / `app.kubernetes.io/instance`; the driver adds
the instance label to the cert-manager-issued Talos CA Secrets, which
carry neither) and the data DataVolumes/PVCs. Worker system disks are
not backed up.
4. The Backup records the prefix, the endpoints, the data DataVolumes
and the HelmReleases of the cluster and its `KubernetesNodes` pools. The
HelmReleases are not in Velero because the lineage webhook labels worker
disks after the same applications, so a label selector for the releases
would also pick the disks.

**Restore** goes into the application the backup came from (rename /
cross-namespace is rejected). `RestoreJob.spec.options` takes `etcd` and
`volumes` (both default `true`) to restore either half alone:

- **Application gone**:
  1. Restore the PKI Secrets and the volumes.
2. Recreate the cluster's HelmRelease. The Kamaji release this repo
ships gives a new TenantControlPlane a UID-keyed datastore schema, so
the restore waits for that schema.
3. Stop the new control plane, write the dump under the new prefix (keys
are moved from the old prefix), and start it again.
4. Recreate the worker pools last, so workers join the restored cluster
with the restored CA and Talos secrets rather than an empty one.
- **Application live**: suspend the cluster's HelmReleases, then stop
the MachineDeployments and the cluster-autoscaler (for `volumes`,
deleting the backed-up DataVolumes/PVCs once the Machines are gone), and
then the KamajiControlPlane (for `etcd`). Workers go first because CAPI
removes a worker's Node through the tenant API. Then write the selected
halves back and restore every replica count. Each original count is
recorded on the object it belongs to, so the cluster is put back on
success, on failure, after a 30-minute stop deadline, and when the
RestoreJob is deleted midway. A retry keeps the counts from before the
first attempt, and a HelmRelease suspended before the restore stays
suspended.

Before changing anything, a restore checks that the Velero backup is
still there and Completed, and that the dump object exists (a
verify-mode run of the Job). The restore mode is recorded in the
RestoreJob's status, so a RestoreJob cannot claim that a live cluster is
gone.

Before it runs `del --prefix`, the restore Job downloads and validates
the dump and saves the current prefix to `<dump>.pre-restore`. That copy
is never overwritten, so a retry cannot replace it with a partial
prefix, and it is dropped after a successful restore. The Job writes a
marker to its termination message, and the outcome depends on it:

- A failure before `del` resumes the cluster.
- A failure after `del` leaves the control plane stopped
(`KubernetesEtcdWriteFailed`) rather than started on a partial prefix.
- Deleting the RestoreJob while its restore Job runs also leaves it
stopped.

Events and master leases are skipped only as top-level resources. Velero
cannot overwrite an existing claim, so a live-cluster volume restore
deletes the current claims first; this is documented.

A backup whose Velero half fails deletes the etcd dump it already
uploaded, and a transient error creating the Backup is retried rather
than failing the run. Deleting a Backup removes the Velero backup
(DeleteBackupRequest) and the dump object plus its pre-restore copy (a
cleanup-mode Job, given up after five failed attempts with a Warning,
plus the same skip-artifact-cleanup escape hatch as Redis/Rabbitmq).
With `velero.bslEnabled=false` the strategy is not rendered, and the
DefaultObjectsGate does not count it as missing, the same as the Velero
strategies. Like every default strategy, it is also gated on its CRD
being established, so the upgrade that first ships the CRD to an
installed chart cannot wedge the release.

Also included: RBAC for
TenantControlPlanes/DataStores/KamajiControlPlanes/MachineDeployments/Machines/Deployments,
chart values (`kubernetesBackupClientImage`,
`kubernetesBackupActiveDeadlineSeconds`, `kubernetesBackupResources`), a
`make image-kubernetes-backup` target, docs in
`docs/operations/backup-classes.md`, and the
`examples/backups/kubernetes/` walkthrough.

### Verification

- `go test ./internal/backupcontroller/`: the new tests cover:
- the certificate windows, pool HelmRelease discovery, option parsing
and the restore-phase selectors;
  - the full backup path;
  - in-place restores with default options and etcd only;
  - a pre-suspended HelmRelease;
- a failed etcd write leaving the control plane stopped, including on
RestoreJob deletion;
  - RestoreJob deletion resuming a stopped cluster;
  - the stop deadline;
  - the preflight: missing or incomplete Velero backup, missing dump;
  - a rebuild, and an unreadable HelmRelease record;
- rename rejection, a non-etcd datastore, and the cleanup attempt limit.

Each safeguard in the Go driver was checked by mutating it out and
confirming that a test fails. The rendered Job script has no committed
test: a unit test would have to extract the script from the rendered
chart, and its behaviour belongs in the chainsaw e2e follow-up instead.
It was run against stub `etcdctl`/`curl` in every mode and branch:
  - a foreign key or an empty value stops it before any etcd call;
- an S3 answer other than 200/404 on the pre-restore copy stops it
before any etcd call;
- an existing copy is kept, and a missing one is uploaded before `del`;
  - keys are moved from the old prefix to the new one;
  - an object in a namespace named `events` is kept.
- `helm unittest` (29/29), `hack/backupstrategy-crd-gate.bats` and `helm
lint` on `backupstrategy-controller`.
- On a live cluster (Velero 1.18.1), with a tenant cluster holding a
ConfigMap marker and a file on a PVC:
- The same flow done by hand before the driver existed: back up, delete
the application, rebuild. The CA, front-proxy CA, SA key and Talos
secrets were byte-identical afterwards, the pre-backup admin kubeconfig
still worked, namespace and ConfigMap UIDs were unchanged, and the PVC
data came back.
- Through this driver: the BackupJob succeeded (432 keys dumped, Velero
backup Completed, Backup metadata as described above). The tenant
namespace holding the markers was then deleted, which also deleted its
PVC and the backing DataVolume, and a RestoreJob with default options
rolled the live cluster back. It stopped the workers and then the
control plane, wrote the prefix back, restored the volume, and restored
the control plane to 2 replicas, the pool and the autoscaler to 1, and
unsuspended the HelmRelease. A new worker joined. The namespace came
back with its original UID, the ConfigMap and the PVC (bound to the same
volume) with their data. That run caught an ordering bug, now fixed and
covered by a test: scaling the control plane down first left the old
Machine stuck in `Deleting`, because CAPI removes the Node through the
tenant API.
- On the current head, through the driver, against the same tenant
cluster:
- **Backup.** It succeeded. The volume list it recorded was taken before
Velero started.
- **`{"volumes":false}` restore after a ConfigMap was deleted and
another created.** The deleted ConfigMap came back and the newer one was
gone. The worker was left running and the volume data was untouched. The
restore waited for terminating control-plane Pods before writing. The
pre-restore copy was saved before `del` and dropped afterwards.
- **Rebuild after the application was deleted, with default options.**
It ran the full sequence: secrets, volumes, cluster HelmRelease,
datastore schema, stop, etcd write, resume. It succeeded in about 3
minutes. The CA and Talos CA were byte-identical, a new worker joined,
and the namespace and ConfigMap UIDs and the PVC data were back. The
tenant cluster runs on a Kamaji that still names schemas `<ns>_<name>`,
so this run did not exercise a prefix change; that path is covered by
the unit tests and the stub runs.

### Not in this PR

- A chainsaw e2e. The tenant Kubernetes suites are bounded by tight op
budgets that tests pin, so the round trip needs its own suite or a
budget change; it will be a follow-up.
- `rotationPolicy` on the Talos CA Certificate in the kubernetes chart
(cert-manager's default `Always` would rotate the CA key at renewal).
- Restore into another application name or namespace.
- The tenant's shared etcd DataStore runs with auth disabled, so any
cluster's datastore certificate can read its neighbours' prefixes. This
predates this change; the dump Job uses the same certificate the
cluster's kube-apiserver already holds.

### Screenshots

N/A — no UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Walked against the diff: no package under `packages/apps|extra` is added
or renamed, `packages/core/platform` and installer values are untouched,
the new CRD is in `strategy.backups.cozystack.io` (not typed by
terraform-provider), and `RestoreJob.spec.options` stays opaque JSON.

### Release note

```release-note
feat(backups): tenant Kubernetes applications can now be backed up and restored through the cozy-default BackupClass. A backup captures the cluster's etcd state and its control-plane PKI and PersistentVolume data, and warns when a control-plane CA would expire before the backup does. A restore rebuilds a deleted cluster with its original CA or rolls a live one back, and can restore the etcd state and the volumes separately via RestoreJob options `etcd` and `volumes`.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

- **New Features**
- Added Kubernetes application backup and restore support, including
etcd data and persistent volumes.
- The default backup class now routes Kubernetes applications to the new
strategy.
- Added an end-to-end walkthrough for creating a cluster, backing it up,
restoring it in place or after deletion, and cleaning up.
- Backup status can report expiring certificates, and restore validation
helps protect cluster identity and state.
- **Documentation**
- Added guidance on backup contents, restore behavior, and walkthrough
configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 15:40:01 +07:00
Aleksei Sviridkin 04fa87ccd1 style(backups): clear staticcheck findings in overlapping controllers
Drop a field-manager constant nothing uses in the MariaDB driver, switch
on the MongoDB operator's state as a tagged switch, and reword the error
strings of the MongoDB driver and the workload monitor so they do not
start with a capital. The errors keep their meaning and nothing matches
their text.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 11:37:42 +03:00
Aleksei Sviridkin 4c3a4b42cc style: clear unused code and style findings reported by staticcheck
Remove helpers and constants nothing references, escape format
characters in test literals, and take the simplifications staticcheck
suggests where they read the same. Error strings no longer start with a
capital: they keep their meaning, with the kind or component name moved
out of first position. Callers match errors by type, not text; the one
test that matches text, for Kafka, keeps the phrase it looks for.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 11:37:41 +03:00
scooby87 e978ed7f2f fix(postgres): stop backup jobs from fighting over the WAL archive destination (#5221)
## What this PR does

Part of #5202. A second PR will restore the last good destination when a
job that changed it fails.

The ObjectStore a Postgres Cluster archives its WAL to is shared by
every backup of the application, yet every running BackupJob server-side
applied its own strategy's destination onto it every five seconds with
forced ownership. Two BackupJobs of one application with different
BackupClasses flipped the destination back and forth until the healthy
backup failed too, and a webhook or apiserver blip during that apply
failed the BackupJob at once.

- A BackupJob writes the destination and the Cluster plugin only before
its `cnpg.io/Backup` exists. Afterwards it only reads the live Cluster,
and a failed read is returned for a controller retry instead of failing
the running backup.
- Before writing, a BackupJob waits, with a `Ready=False` /
`WaitingForOtherBackup` condition naming the other job, while another
running BackupJob of the application has a `cnpg.io/Backup` in flight.
The wait is not capped, for the same reason as in #5211. The queue reads
the Cluster's own Backups from the apiserver; with leader election one
reconcile runs at a time, so only a leader handover can still let two
jobs through. An atomic per-application claim is #4559.
- Before writing, a BackupJob also checks the apiserver for a restore of
its application and holds while one runs, including a job released from
the queue: a write landing during the restore would put the restored
Cluster back on the source's WAL prefix.
- Write errors that cannot succeed on retry (invalid, bad request,
forbidden) still fail the job at once. Any other is retried with a
`ClusterPluginApplyRetrying` condition for the backup deadline, counted
from the first failure since the job last waited, and recorded in a new
`DestinationApplied` condition.
- The pass that creates the `cnpg.io/Backup` records on it the
destination, endpoint, credentials Secret and serverName it wrote
(`backups.cozystack.io/cnpg-written-destination`), and the artifact and
its size probe are built from that record, so a strategy edited, or a
Cluster re-rendered, while the backup runs does not redirect the
artifact. A Backup without the record falls back to the strategy as
before.
- The plugin rollout deadline and the instance check count from the
write that attached the plugin, not from the job's start. The apply that
attaches it also stamps the time on the Cluster
(`backups.cozystack.io/barman-plugin-attached-at`), so a lost BackupJob
status write after the attach does not fall back to an older time, and a
job that starts while another job's recent attach is still rolling waits
for that rollout too. A stamp a few minutes ahead of the controller's
clock is waited out; one further ahead is treated as corrupt.

Tests drive the reconciler through a running backup that must not
rewrite the destination, the queue and its release, a restore starting
while a job waits, each class of write error, a recovered write, a
failed Cluster read while the Backup runs, lost status writes around the
attach, another job's rollout, clock drift on the attach stamp, the
recorded destination and serverName, and the rollout deadline. Each
guard was checked by mutation.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(postgres): Postgres BackupJobs of the same application no longer overwrite each other's WAL archive destination: a job writes it only before its backup starts and waits while another backup or a restore of the application is in flight. A transient error while writing it is retried instead of failing the job at once, and the backup artifact names the destination and serverName the backup was actually written to.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Reliability**
* CNPG backups wait while another backup or restore is using the shared
destination, and retry temporary destination-write failures while
observing the backup deadline.
* Backup artifacts and size checks use the destination recorded when the
backup was created, even if destination settings later change.
* Backup processing handles plugin rollout timing and destination-hold
release more consistently.
* **Restore**
* Restore operations account for shared WAL archive destination
conflicts and guard against incompatible backup selections.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 08:58:55 +03:00
Alexey Artamonov 9bd8d12cb8 fix(postgres): hold the WAL archive destination for backups no job tracks
A BackupJob rewrote the shared WAL archive destination while a
cnpg.io/Backup of the Cluster was running if nothing tracked that
Backup through a running BackupJob: one created by hand or by a
ScheduledBackup, or one whose BackupJob was deleted or had already
finished. The base backup then stayed where it started while the WAL
it needs went to the new location, and the backup could not be
restored.

Such a Backup now holds the destination while CloudNativePG executes
it. Only the phases in which CloudNativePG itself starts no other
Backup of the Cluster count, so the hold never outlasts the queue the
operator already imposes: a Backup that waits in pending, or was sent
back to it, starts its base backup over against the destination
current then.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-12 18:44:17 +03:00
Alexey Artamonov 583f18437d test(clickhouse): give the credentials-wait BackupJob a UID
The test pins that a backup held for late bucket credentials keeps
its archive name, and a run's archive name now carries part of the
BackupJob's UID, so the fixture needs one for the name it expects to
be the name the controller mints.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-12 17:21:51 +03:00
Alexey Artamonov dae76d186c Merge branch 'main' into fix/clickhouse-idle-before-backup
main pins the archive name a backup Pod keeps across a wait for
late bucket credentials, against the controller's archive helper.
That helper now names a run's archive with part of the BackupJob's
UID, so the merged test has to give its BackupJob one.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-12 17:20:19 +03:00
Alexey Artamonov 5c6fc10a23 test(postgres): pin three guards no test caught when changed
A job whose own Backup exists skips the wait for another job's plugin
rollout, and the size probe keeps the strategy's Secret key names while
it takes the Secret from the record, adding a credentials block only
when the strategy has none. Each of these could be changed with the
suite still green: dropping the own-Backup check, or forcing the
missing-credentials branch either way. Cases for a job with a running
Backup next to a recent foreign attach, a strategy without credentials
and a strategy with custom key names now fail on each change.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-12 17:02:40 +03:00
Alexey Artamonov b3ca798bc4 fix(backups): read the runs that may own a MariaDB dump uncached
Before deleting a dump the cleanup checks every other operator Backup
on the same prefix that could have written it, and it read them from
the informer cache. A run whose operator Backup had not reached the
cache yet but had already uploaded its dump was missed, and its dump
deleted as this Backup's. Both the recorded and the legacy paths now
list the runs from the API server, as the restore checks already do.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-12 17:01:29 +03:00
Aleksei Sviridkin 238b3ee491 fix(seaweedfs): enforce S3 bucket lifecycle rules (#5228)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

SeaweedFS accepts and stores an S3 bucket lifecycle configuration, but
only the `s3_lifecycle` job of its plugin worker applies it, and that
job is scheduled by the admin server. Cozystack deployed neither, so a
lifecycle rule returned 200 and no object ever expired, on the platform
backup bucket and on tenant buckets alike. That also made the documented
way to bound backup storage a no-op.

This PR adds the admin server and a worker to Simple and MultiZone
SeaweedFS instances, behind a new `lifecycle.enabled` value that is
**off by default**. A Client instance is unchanged, because its rules
are enforced where the instance it connects to runs. The worker
registers for `s3_lifecycle` only, so vacuum, balancing and erasure
coding are not started as a side effect. The admin keeps its job state
on a 1Gi volume on `lifecycle.storageClass`, or on the cluster default
when that is empty; `volume.storageClass` is not reused, because in
MultiZone it can name a class local to one zone or node. The admin UI is
protected by a generated password, because a tenant namespace admits
traffic from the whole cluster. The dashboard Role lists the two new
WorkloadMonitors.

Enabling the worker alone was not enough on SeaweedFS 4.45. Its chart
renders no `[grpc.s3]` section, so the s3 gateway served its internal
gRPC in plaintext while the worker dials it with mTLS, and every
lifecycle delete failed the TLS handshake. The section is backported
from upstream (seaweedfs/seaweedfs#11157, released in 4.46) as a chart
patch, applied with `--forward` so the next `make update` stops at the
point where the patch must be dropped instead of silently reversing it.
The s3 pods are restarted once so they load the section. This also
repairs IAM cache propagation between s3 replicas, which failed in the
same way, and it applies whether or not lifecycle is enabled.

The data-plane release now retries a failed install or upgrade in place
(`RetryOnFailure`) instead of rolling it back. With the previous default
remediation and unlimited retries, an upgrade that waited out its
timeout, for example on a tenant quota with no room for the admin
volume, rolled back and retried forever, restarting the s3 pods twice
per cycle. The worker's hostname anti-affinity is relaxed to a
preference so a rollout finishes on a single-node cluster.

#### Why lifecycle is off by default

The daily walk that SeaweedFS uses to expire objects sends its deletes
without an object identity, and the s3 server skips its compare-and-swap
when no identity is given; it does not re-check the object's age either.
A key overwritten between the walk's listing and the delete loses its
new content. This path is unchanged from 4.45 through 4.52 and on
upstream master (`weed/s3api/s3lifecycle/dailyrun/walker_dispatcher.go`,
`identityMatches` in `weed/s3api/s3api_internal_lifecycle.go`). Backup
buckets write unique keys and are barely exposed; a tenant bucket that
rewrites a fixed key is. On top of that, turning enforcement on would
start applying, on upgrade, every rule that was stored and ignored until
now. Both are retention changes an operator should choose, so nothing
changes for existing instances until `lifecycle.enabled` is set. Making
it the default again is left to a separate PR that carries object
identity on the walk path; upstream report:
https://github.com/seaweedfs/seaweedfs/issues/12451.

#### Turning lifecycle on and off

When `lifecycle.enabled` is set, rules that have been stored but ignored
start applying within minutes, and objects they have already expired are
deleted on that first pass, so review existing bucket lifecycle rules
first. Set `lifecycle.storageClass` before enabling, and prefer a
replicated class: the class is part of the admin StatefulSet's volume
claim template, which cannot change, so changing it while lifecycle is
on makes every upgrade of the instance fail until it is reverted. Moving
the volume means disabling lifecycle, deleting the
`admin-data-seaweedfs-admin-0` PVC, changing the class and enabling
lifecycle again. Each instance gains two small pods and a 1Gi PVC; when
the tenant quota has no room for them, the data-plane release keeps
retrying in place until it has.

When `lifecycle.enabled` is turned off again, the parent release removes
the admin credentials Secret while the data-plane release removes the
admin StatefulSet only on its own reconcile, shortly after. An admin pod
that restarts in that window fails to start on the missing Secret until
the StatefulSet is gone; nothing is lost, and the order is left as is.
The admin PVC stays until the instance is deleted, and a later re-enable
starts the admin with a new password.

Verified on a test cluster with the 4.45 chart: without the backport,
the worker found the expired objects and every delete failed with `tls:
first record does not look like a TLS handshake`; with this branch and
lifecycle enabled, a rule with a past expiration date removed the
matching objects and left the others, both on a manual run and on the
first scheduled pass.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up needed, not opened yet:
`operations/services/backup-classes.md` (v1.7 and next) advises an S3
lifecycle policy, which the bundled SeaweedFS applies only once
`lifecycle.enabled` is set; the SeaweedFS reference page gains
`lifecycle.enabled` and `lifecycle.storageClass` from the regenerated
README on the next release
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The terraform provider does not model the SeaweedFS application, so the
new values do not reach it.

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(seaweedfs): S3 bucket lifecycle rules can now be enforced. Simple and MultiZone SeaweedFS instances run the SeaweedFS admin server and a lifecycle worker when the new `lifecycle.enabled` value is set; it is off by default, so existing instances keep their current behaviour. It stays off by default because the upstream daily expiry walk deletes without checking that the object it listed is still the one stored, so an object overwritten in that window can be deleted although it is new. When enabled, rules that were stored but ignored start applying within minutes and objects they have already expired are deleted on that first pass; review existing rules first. Each enabled instance gains two small pods and a 1Gi volume, placed on `lifecycle.storageClass` or the cluster default. Independently of the switch, the s3 gateway now serves its internal gRPC over mTLS, which repairs IAM cache propagation between s3 replicas and restarts the s3 pods once on upgrade.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Added optional, daily S3 bucket lifecycle-rule processing for
supported SeaweedFS configurations. It is disabled by default; Client
instances rely on lifecycle rules configured on the connected SeaweedFS.
* Added configurable storage for lifecycle job state and dashboard
visibility for lifecycle components.
* Enabled the S3 gRPC server to use its internal TLS identity when
security is configured.

* **Bug Fixes**
* Improved storage-class handling so numeric-looking names remain
strings and unset classes are omitted where appropriate.
* Clarified backup cleanup guidance, including when lifecycle rules
apply and how to reclaim storage.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-12 16:21:52 +03:00
Aleksei Sviridkin 3badbdef74 fix(clickhouse): let a running backup sidecar pick up system-bucket credentials (#5232)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

A ClickHouse release with `backup.useSystemBucket: true` can start
before the platform projects `cozy-backups-creds` into its namespace.
Its `clickhouse-backup` sidecar took the S3 settings from environment
variables, which are fixed when the container starts, so the first
BackupJob or RestoreJob of such a release failed with `no EC2 IMDS role
found` until the Pod was restarted, and a rotated key stayed stale the
same way.

The projector now also writes a `clickhouse-backup.yml` key into
`cozy-backups-creds` (bucket, `https://` endpoint, region when set,
keys, `force_path_style` from `backupStorage.forcePathStyle`). On the
system-bucket flow the chart mounts that key as an optional Secret
volume and points `CLICKHOUSE_BACKUP_CONFIG` at it, instead of setting
`S3_*` variables. clickhouse-backup reloads its config file in each API
handler that reaches S3 (among them the `create_remote`,
`restore_remote` and `list` calls the strategy makes; the `status` and
`actions` reads it polls with do not) and kubelet rewrites a mounted
Secret in place, so credentials projected after the Pod started, or
rotated since, reach the running sidecar. `S3_PATH` stays in the
environment because the prefix belongs to the release.

Kubelet takes about a minute to refresh the volume, so the projector
stamps `backups.cozystack.io/credentials-updated-at` when the projected
content changes, and the Altinity driver holds a new backup or restore
Job for up to two minutes after that stamp, with `Ready=False` and
reason `WaitingForCredentials`; the condition is dropped once the Job
runs. A stamp that cannot be read, or one further ahead than clock skew
explains, is logged and ignored rather than holding the namespace's jobs
for good. The hold applies to every Altinity job in a namespace whose
projected Secret just changed, including releases on their own Secret,
where it is only a delay.

The new e2e Test `clickhouse-3-system-bucket-roundtrip` starts a
ClickHouse on the system bucket with no projected Secret, then runs a
backup and an in-place restore through `cozy-default` and checks that
the sidecar that started without credentials served both jobs. In CI the
system bucket's advertised host `s3.example.org` neither resolves nor
gets an ACME certificate, so for the duration of its own step the Test
serves a SeaweedFS-CA certificate under the S3 Ingress TLS Secret name
and adds a marked `rewrite name exact` line for that host to the host
CoreDNS Corefile, removing both on exit; it proves the served
certificate before any job starts.

The bug was reproduced on a test cluster running the code before this
change: a ClickHouse started before the projection failed its first
BackupJob with `no EC2 IMDS role found`. The new e2e Test has not run
green yet outside CI, because it edits the host CoreDNS, which is shared
on that cluster.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#799 (draft until this PR merges):
`operations/services/backup-classes.md` for the mounted config file, the
credentials wait and the upgrade window, without the Pod restart this
change removes
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(clickhouse): a ClickHouse release on the platform system bucket (`backup.useSystemBucket: true`) now backs up and restores without a Pod restart when its namespace had no projected backup credentials at start, and picks up rotated credentials while running. The sidecar reads its S3 settings from a `clickhouse-backup.yml` file in `cozy-backups-creds` instead of environment variables, so the upgrade rolls every such ClickHouse Pod once. The first Altinity backup or restore in a namespace after its projected credentials change waits up to two minutes (`WaitingForCredentials`) for them to reach the sidecar. Upgrade the platform as a whole: a system-bucket BackupJob started while the new chart runs against the previous controller fails, because that controller does not write the new key.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* ClickHouse system-bucket backups now use a projected configuration
file for S3 settings, supporting HTTPS endpoints and optional region and
path-style settings.
* Backup and restore jobs wait briefly for recently updated projected
credentials before starting.
* Path-style settings accept common true and false values and are
normalized consistently across backup storage integrations.
* **Tests**
* Added end-to-end coverage for system-bucket backup and restore,
verifying restored data without restarting the ClickHouse Pod or backup
sidecar.
* Added checks for TLS certificate validation and safe restoration of
shared network and ingress settings.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-12 16:15:26 +03:00
Alexey Artamonov 1d22013646 fix(postgres): hold a backup until the plugin sees its new destination
The plugin-barman-cloud sidecar reads the ObjectStore through its own
cache, kept for ten seconds (DefaultTTLSeconds in plugin-barman-cloud
v0.13.0) and invalidated only by the sidecar's own writes, and WAL
archiving keeps it warm. The pass that changed the destination created
the cnpg.io/Backup a moment later, so the base backup could be taken
against the previous destination while the artifact names the new one,
and a restore from it finds nothing. On a live cluster with the shipped
plugin, a Backup created two to four seconds after switching the
destination landed in the old bucket in two of three tries; with a
fifteen second wait it landed in the new one every time.

The apply that changes the ObjectStore configuration now stamps the
time on the ObjectStore itself, so the record cannot be lost apart from
the change, and the job creates its Backup only once that change is
older than the cache can be. A change is judged on the configuration a
server-side dry run of the same apply returns, so values the apiserver
defaults or another field manager owns do not read as a change on every
pass.

That wait leaves a job with the destination written and no Backup yet,
which the queue, looking only for Backups in flight, did not see: a
second job with another destination wrote its own, the first took that
as a change and stamped again, and the two restarted each other's wait
forever. Every apply of the ObjectStore therefore also names the job
that made it, which only a job without a Backup of its own and let
through by the queue does, and the queue holds every other job while
that one runs and has no Backup yet. Once it has one, that Backup holds
the destination for as long as CloudNativePG runs it, as before, so a
job stuck after its Backup finished does not hold every later backup of
the application. This also covers a job waiting for the
rollout of the plugin it attached, during which a second job used to
switch the destination as well, including when its own write left the
configuration as a restore had re-rendered it.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-12 15:13:58 +03:00
Alexey Artamonov 5a95bcdd64 fix(clickhouse): keep a run's archive name within the old name's length
The archive name is part of every object key, a key is capped at 1024
bytes, and the legacy s3PathOverride prefix has no length limit. The
full UID made the name 37 characters longer than before, so a prefix
that fit the old name pushed the manifest key past the cap and failed
the backup.

The run's name now drops "clickhouse-" and the "T" and "Z" of the
timestamp and spends the room on the first 12 hex digits of the UID,
so it is exactly as long as the old name for every application.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-12 11:10:43 +03:00
Aleksei Sviridkin 9978771858 feat(observability-proxy): serve tenant namespace logs through an authorizing proxy (#5214)
## What this PR does

Adds `observability-proxy`, a new system component that lets a tenant
user read the logs of their own namespace from VictoriaLogs with the
same Kubernetes credentials they already use for the console, and with
Kubernetes RBAC deciding what they can read. This is the first part of
the design proposals [Tenant application
logs](https://github.com/cozystack/community/pull/95) and [Tenant
resource-consumption
metrics](https://github.com/cozystack/community/pull/94); this PR
implements the namespace logs root only. Metrics and the per-application
roots follow in separate PRs.

Why a proxy: today the log stores are shared per root tenant and have no
notion of a caller, so tenants cannot be given direct read access
without seeing other tenants' logs. The proxy authenticates the caller,
authorizes the request from its path, and rebuilds the upstream query so
that its scope cannot be widened by the client.

Backward compatibility: this adds a new package and a new HelmRelease;
nothing existing is renamed, removed or changed, so no migration is
needed.

What is included:

- `cmd/observability-proxy` and `internal/observabilityproxy`: the
proxy.
- Serves
`/apis/cozystack.io/v1alpha1/namespaces/<ns>/logs/select/logsql/<endpoint>`
for the VictoriaLogs read endpoints the console needs (`query`, `hits`,
`stats_query_range`, `tail`, `field_values`, `stream_field_values`,
`facets`). Anything else under the root answers `404`.
- Authentication by `TokenReview`, authorization by one
`SubjectAccessReview` built from the path (`list` on `cozystack.io/logs`
in the namespace), both cached briefly.
- The upstream request is rebuilt from an allowlist of parameters; the
namespace scope is always forced as `extra_stream_filters`.
Client-supplied `extra_filters`, `extra_stream_filters`, any LogsQL
`options(...)` block, and tenant headers such as `AccountID`/`ProjectID`
are not forwarded or are refused.
- Non-canonical paths (`..`, `//`, encoded separators) are refused with
`400` before any authorization is built, because the console nginx
matches on the normalized path but forwards the raw one.
- Time window, step, bucket count and result limits are clamped;
per-user rate, concurrency and open-tail limits answer `429`. Invalid
time, step and limit parameters answer `400`.
- The store is resolved from the namespace's
`namespace.cozystack.io/monitoring` label.
- Live `tail` streams are passed through unbuffered, and the upstream
tail request disables compression so headers reach the client before the
first line.
- `packages/system/observability-proxy`: chart and image build, wired
into the root `make build`.
- Platform wiring: a new `cozystack.observability-proxy` PackageSource,
enabled by default on the full variants and opt-in on the slim ones, the
same way as `monitoring-agents`.
- RBAC: an aggregated `cozy:observability:view` ClusterRole grants
`list` on `cozystack.io/logs` to tenant `view` and above. No API server
serves this resource; the rule exists only for the proxy's
`SubjectAccessReview`. The proxy's own ClusterRole is limited to the
calls it makes (`tokenreviews`, `subjectaccessreviews`, `list`/`watch`
on namespaces).
- Console nginx: when the proxy is not installed or cannot be reached,
the observability roots answer `404` with a Kubernetes `Status` body,
which the console shows as not available on this cluster; the proxy's
own `502`/`504` for an unreachable store pass through unchanged.
- Console: one regex `location` in the console nginx routes the four
observability roots to the proxy ahead of `/apis`. The upstream is
resolved at request time, so the console starts even when the proxy is
not installed; those paths then answer `404` with a Kubernetes `Status`
body. The console Deployment now receives the cluster domain for that
resolution.
- E2E: a new Chainsaw suite `hack/e2e-chainsaw/observability-proxy/`
with two tenants, covering `401`/`403`/`200`, scope-widening attempts,
path normalization and a long-lived `tail`.

#### How it was tested

- `go test -race` and `go vet` for the proxy packages.
- helm-unittest for `observability-proxy`, `dashboard`, `tenant` and
`core/platform`; bats tests for the console nginx routing.
- The Chainsaw suite was not run as-is: it deploys its own VictoriaLogs
as the root store and refuses to run when a real one exists. Its cases
were replayed by a script against the real store on a test cluster.
- Verified on a test cluster:
- `401` without a token and with an invalid token, `403` for another
tenant's namespace, `200` for the caller's own.
- Client `extra_stream_filters`/`extra_filters` (GET and POST), an
`AccountID` header and another tenant's stream named inside the query do
not widen the scope; `options(...)` answers `400`; a path with `..`
pointing at another tenant answers `400`; an endpoint outside the
allowlist answers `404`.
- Through the full browser chain (OIDC login, gatekeeper, gateway,
console nginx), a single `tail` stayed open for more than two minutes
without reconnecting, and requests from the same session to other
tenants' roots answered `403`.
- Not verified: the `429` limits under real load. `golangci-lint` has
not been run locally.

#### Known limitations

- The proxy is reachable only through the console route; tenant
workloads have no egress rule to it. Every request takes its concurrency
or tail slot before the body is read; the token and access reviews and
the body read run under `--request-timeout`, so a stalled kube-apiserver
answers `503` rather than holding the request; and writes to the client
carry a deadline (renewed per line for tails). `--request-timeout=0`
disables those bounds and should not be used in production.
- The first lines a newly started pod writes can be missing from an
already open `tail`, although they are in the store. The likely cause is
that Fluent Bit picks up a new log file later than the VictoriaLogs
live-tail window; later lines of the same pod arrive. A history query
shows them.
- The per-application roots (`.../<plural>/<name>/log`, `.../metrics`)
and the namespace metrics root answer `404` in this PR.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/issues/787 (docs land
once the implementation PRs merge)
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Downstream: `cozystack/website` is reached. A new platform component
(`packages/core/platform/sources/observability-proxy.yaml`) triggers
`content/en/docs/next/guides/platform-stack/_index.md` and
`operations/configuration/licenses.md`; adding it to the system bundle
(`packages/core/platform/templates/bundles/system.yaml`) may also need a
line in `operations/configuration/variants.md`. A user-facing page for
the logs API and the new tenant `view` permission is not required by the
trigger map but would be useful. Tracked in cozystack/website#787. Other
repositories: no trigger matched. `packages/core/platform/values.yaml`
is not changed (only a test fixture), no package under `packages/apps/`
gains a values field, and no CRD changes. The root `Makefile` `build`
target builds one more image; this does not touch `hack/` or the make
targets `ccp` relies on, so `ccp` is not considered affected, but a
reviewer may want to confirm.

### Release note

```release-note
feat(observability-proxy): add an authorizing proxy that lets tenant users read their namespace logs from VictoriaLogs with their Kubernetes credentials, under `/apis/cozystack.io/v1alpha1/namespaces/<ns>/logs/`. Access requires the tenant `view` role or higher. The component is enabled by default on full variants and opt-in on slim variants.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Added an observability proxy for tenant-scoped log access through the
console, including live log streaming. Requests are authenticated,
access-controlled, and subject to configurable rate and concurrency
limits.
* Added platform packaging and deployment support, with console routing
for observability requests.
* **Tests**
* Added automated coverage for access control, tenant scoping, query
handling, streaming, proxy behavior, and deployment configuration.
* **Documentation**
* Updated end-to-end test guidance with host requirements for running
the observability proxy suite.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-11 16:24:25 +03:00
Alexey Artamonov a6a704768c test(backups): pin a failed RestoreJob listing in the MariaDB cleanup
A RestoreJob listing that fails says nothing about whether a restore
reads the dump. Swallowing that error left the whole suite green, while
it would let the cleanup delete the dump under a running restore.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 15:41:03 +03:00
Alexey Artamonov f2507b4b84 docs(backups): name what the MariaDB cleanup reads uncached
The reader's comment named only RestoreJobs, and the restore path said
the cleanup holds a Backup only for RestoreJobs it can see, while the
cleanup now lists both RestoreJobs and operator Restores straight from
the API server.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 15:35:33 +03:00
Alexey Artamonov 0195a66fae fix(backups): release a MariaDB Backup when the operator CRDs are gone
Listing the operator Restores of a Backup before reading its operator
Backup made the cleanup fail on a cluster without the mariadb-operator
CRDs, retried for good, so the Backup and its namespace never went
away. The operator Backup read that follows already releases the
Backup in that case; with no Restore kind no restore can run, so the
listing now lets the cleanup reach it.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 15:35:33 +03:00
Alexey Artamonov 8246808289 fix(backups): hold a MariaDB dump while an operator Restore reads it
The cleanup waited for a restore by the phase of its RestoreJob, but the
phase is not the restore: a RestoreJob fails on its deadline while the
operator Restore it created keeps running, and that Restore still reads
the dump. Deleting the dump then made mariadb-operator restore the
nearest other one.

The cleanup now also holds the Backup while an operator Restore of its
operator Backup has not been marked Complete, which mariadb-operator
does once the Restore's Job has completed or failed, so what keeps the
dump is the restore that reads it rather than the status of the
request.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 15:30:41 +03:00
Alexey Artamonov ee6404a0c8 fix(backups): retry a failed lookup of a MariaDB operator Restore
The lookup now goes to the API server, which can time out or throttle,
and a failed lookup failed the RestoreJob. It says nothing about
whether the operator Restore exists, so the RestoreJob could be failed
while its Restore was running, and the Backup's cleanup would then stop
waiting for it and delete the dump the Restore reads.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 15:28:45 +03:00
Alexey Artamonov 14a842ad16 fix(backups): find a MariaDB operator Restore the cache has not seen
The MariaDB restore creates its operator Restore once and then follows
it, finding it by label in the informer cache. Nothing watches operator
Restores, so a reconcile that ran before the new Restore reached the
cache saw none: it created a second one, or, now that a restore refuses
a Backup being deleted, failed the RestoreJob while its Restore was
still running, after which the cleanup deleted the dump that Restore
was reading.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 15:15:18 +03:00
Alexey Artamonov d57ebbe808 docs(backups): state what the MariaDB dump delete does not cover
The comment on who deletes a MariaDB Backup listed namespace teardown
without saying that teardown can remove the operator Backup first, in
which case the dump is left behind with a Warning, so it promised a
delete the code does not make. The remover contract claimed a missing
key is never an error, which holds only for a backend that answers the
delete with 204, the one status minio-go accepts.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 15:09:15 +03:00