Commit Graph
1640 Commits
Author SHA1 Message Date
Maxim Belyy 2468b309ee fix(ci): count a failed bare-ref rewrite and word the drift log by result
A rewrite that cannot replace the file now counts as failed and leaves
no temporary file behind, instead of reporting lines as replaced. A
drifted file whose bare refs took main's images no longer logs that it
kept its committed ref.

Assisted-by: LLM
Signed-off-by: Maxim Belyy <maximbel2003@gmail.com>
2026-11-14 19:58:33 +07:00
Maxim Belyy 854a3f40e4 fix(ci): limit the drifted-file bare-ref overlay to first-party images
Main can bump a third-party image together with its config, so a PR
behind it must keep the committed pair rather than take main's image
alone. Only ghcr.io/cozystack/cozystack refs, matched against an
artifact line for the same first-party image, are replaced now.

State the trade-off for first-party refs that do exist, and name the
packages that took main's images in their own annotation.

Assisted-by: LLM
Signed-off-by: Maxim Belyy <maximbel2003@gmail.com>
2026-11-14 19:54:31 +07:00
Maxim Belyy fd15ad282e fix(ci): take the main ref for a bare image ref in a drifted file
A file whose configuration drifted from the cozystack-packages:main
artifact keeps its committed refs. For a digest-pinned ref that runs an
older image; for the bare <repo>:<tag> a new image is first committed
with, it names a tag that was never pushed, so the component cannot
pull. The merge commit CI tests is fixed when the run starts, so a PR
whose merge commit predates a config change to observability-proxy's
values.yaml installed observability-proxy:v0.0.0 and failed E2E.

In a drifted values file, a bare ref under an image key now takes the
artifact line with the same key, the same image name and a digest, when
exactly one exists. The rest of the file stays as the PR has it.

Assisted-by: LLM
Signed-off-by: Maxim Belyy <maximbel2003@gmail.com>
2026-11-14 19:51:20 +07:00
myasnikovdaniil a427ceb582 ci: run only the backport-batch tests for a backport-batch change (#5287)
## What this PR does

Stacked on #5288, merge that one first.

PR touching only `cmd/backport-batch` was treated as code change, so it
rebuilt every image, installed platform in e2e sandbox and ran full
chainsaw suite plus whole unit lane. #5283 (one-line test fix) spent 3.5
hours in that e2e and hit job timeout. And backport-batch go tests were
running on every code PR, whatever it touched.

- plan job `code` output is now false when every changed path is under
`docs/`, `cmd/backport-batch/` or `cmd/api-gate/` (CI-only tools, no
image builds them). Builds, finalize and e2e are skipped and `E2E Tests`
posted green, same as for docs-only PRs. `e2e-fork.yaml` skips on same
prefixes, `hack/gate-concurrency-contract.bats` fails if two lists
diverge.
- `hack/build-matrix.sh` and `hack/select-e2e.sh` don't escalate to full
rebuild or full suite because of these two dirs, so mixed PR is scoped
by its other paths. Other `cmd/` dirs still escalate.
- `test-backport-batch` removed from `make unit-tests`. Checks job runs
it only when diff touches path from `cmd/backport-batch/test-inputs.txt`
(tool itself, in-repo packages it imports, docs pages its tests read,
`go.mod`, `go.sum`). `TestInputsListed` compares this list with `go list
-deps -test` and fails when import is missing there.
- new `docs` output keeps bats lane for docs changes and drops it for
tool-only ones.
- includes test fix from #5283 (pin run attempt in
`TestReportDedupAndTrust`), this PR replaces it.

### Screenshots

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
ci: changes only in docs/, cmd/backport-batch/ or cmd/api-gate/ don't build images and don't run e2e anymore, backport-batch tests run only when their inputs change.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* CI now skips builds, E2E tests, and full-pipeline checks for changes
limited to documentation and designated CI-only tools.
* Backport-batch tests run when relevant files change, without being
included in the general unit-test target.
* Changes to other command-line tools continue to trigger the
appropriate build and E2E checks.

* **Documentation**
* Clarified how documentation, dashboard, and backport-batch changes
affect test selection.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 23:40:56 +03:00
myasnikovdaniil aa36d3fad6 fix(kubernetes)!: remove the talos block and other leftovers of the worker pool split (#5191)
## What this PR does

Cleanup after the Kubernetes app was split into `Kubernetes` (control
plane and addons) and `KubernetesNodes` (one release per worker pool):
what the split left behind in values, docs, templates and tests.

### Kubernetes

The `Kubernetes` application has no `talos` field any more. Worker pools
are `KubernetesNodes` resources and take their Talos release, image,
installer and registry mirrors from their own `talos` and `osImage`, so
the cluster's copies did nothing. The last one, `talos.version`, fed a
duplicate of the Talos to Kubernetes support-matrix check. The pool
chart already runs that check against the Talos release each pool
actually boots, including one picked through an `osImage` override; the
cluster's copy compared the Kubernetes version against a release no node
runs. A stored spec that still carries `talos.*` keeps rendering,
because the values schema does not close the object and the API does not
validate `spec` against it, and a unit test pins that.

The pool's check now has render coverage. Its own suite pins the
pairings it must accept, and a bats test renders the refusals through
`talos.version`, `osImage.factory.version` and `osImage.builtin.version`
with schema validation skipped, since the schema offers no Kubernetes
version the matrix refuses.

I fixed the README and field descriptions that described behavior the
chart no longer has, or never had: a dedicated etcd cluster, a
`blockSize.matchVolume` setting that does not exist, a default `md0`
pool, the host default, the substrate mismatch behavior, and rollout
phase names. The README's Breaking Changes section becomes a pointer to
the per-release upgrade notes on cozystack.io. Unused helpers and stale
comments are removed; the chart renders the same bytes.

### KubernetesNodes

Five field descriptions and the `talos` block told users to keep them in
sync with parent fields that do not exist. NOTES warned about colliding
with a parent-managed `md0` pool, although the parent renders no pools
and `md0` is the normal name of a pool. The installer image was credited
to an upgrade Job that never runs one. The values descriptions, README
intro and NOTES now say what the chart renders today, including which
fields have no effect on `substrate: proxmox`: `storageClass`,
`diskSize`, `instanceType`, `talos.imageFactoryURL` (beyond its format
check) and `talos.imageFactoryCA`.

The render error for a pool whose MachineDeployment belongs to another
release told users to remove the pool from the parent's `nodeGroups`,
which cannot clear the conflict. It now names a remedy that works.
Split-era history is gone from Helm and test comments, along with two
`storageClass` fallbacks that can only return the value they fall back
from, a constant variable, and two `.helmignore` rules that match
nothing.

The tenant pod and service CIDRs are literals in both charts, and
nothing checked that they agree. A new bats test renders both charts and
fails when they differ.

No rendered manifest of the pool chart changes. I rendered it for the
snapshot values, GPU, kubelet-override, factory/CA, storageClass
variants and three Proxmox variants on main and on every commit, and the
output is byte-identical. The snapshot and TalosConfigTemplate hash
tests pass unchanged, so no worker rolls. YAML comments inside the
hashed machine config and machine template are left as they are for that
reason.

### The rule

`docs/agents/overview.md` gains a rule so this does not come back with
the next change: app READMEs and field descriptions describe current
behavior, a change users must act on goes to the website's upgrade page,
and a description does not ask the reader to keep it in sync with
another chart.

```release-note
[kubernetes] The Kubernetes application no longer has a `talos` field. Talos settings for workers, and the Talos to Kubernetes compatibility check, live on each KubernetesNodes pool. Existing clusters that still store `talos` values keep working; the values are ignored. KubernetesNodes field descriptions, README and install notes now describe the worker pool as it works today, including which fields have no effect on Proxmox pools.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Configuration**
* Worker OS image, installer, and registry-mirror settings are
configured per worker pool; these settings are no longer available at
the cluster level.
* Kubernetes and Talos version compatibility is checked for each worker
pool. A worker Kubernetes version cannot be ahead of its parent cluster.
* Worker pools support KubeVirt and Proxmox, with pool-specific sizing
and disk settings.
* Rendering is refused when a pool uses a different substrate from its
cluster or conflicts with an existing worker pool owned by another
release.
  * The default cluster hostname now includes the `kubernetes-` prefix.
* Existing cluster configurations that still include Talos settings
receive a warning; those settings are no longer used.

* **Documentation**
* Updated guidance for cluster and worker-pool settings, storage,
upgrades, and version compatibility.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-14 00:34:44 +05:00
Aleksei Sviridkin 8d29097e34 Merge branch 'main' into ci/skip-tooling-paths
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 19:16:34 +03:00
Aleksei Sviridkin 29e8522f4b fix(cozystack-basics): refuse a direct delete of a tenant namespace (#5011)
## What this PR does

Fixes #5002. A DELETE of a tenant namespace is refused unless it comes
from the identities that remove it on the normal path, and the denial
says to delete the Tenant instead.

The new ValidatingAdmissionPolicy in cozystack-basics matches DELETE of
core namespaces, and its binding selects namespaces labelled
`tenant.cozystack.io/tenant-root`. The tenant chart puts that label on
every tenant namespace below tenant-root (`tenant.ancestorTenantLabels`,
and migration 49 backfilled older ones). tenant-root itself has no such
label and stays with `cozystack-no-delete-guardrail`. The `tenant-*`
probe namespaces of the e2e suites carry only
`namespace.cozystack.io/host` and are not selected either.

Two groups pass. `system:serviceaccounts:cozy-fluxcd`, because
helm-controller and its shards run as `flux` there, and a Tenant
uninstall is helm-controller deleting the namespace after the pre-delete
hook. `system:serviceaccounts:kube-system`, for the garbage collector
following the ownerReference of a child namespace once its parent is
gone. The namespace controller never sends a DELETE for a namespace, it
updates `finalize`, so it is not affected
(`pkg/controller/namespace/deletion` and
`pkg/registry/core/namespace/storage` at v1.33.12 and v1.34.1).
`system:masters` is not exempt, the same as with the no-delete
guardrail.

This is the denial kube-apiserver 1.35 returns.

```
namespaces "tenant-foo" is forbidden: ValidatingAdmissionPolicy 'cozystack-tenant-namespace-delete-policy' with binding 'cozystack-tenant-namespace-delete-policy' denied request: Namespace tenant-foo belongs to the Tenant foo in tenant-root. Delete the Tenant instead with kubectl delete tenants.apps.cozystack.io foo --namespace tenant-root. Deleting the namespace directly can remove the tenant etcd before the Kubernetes clusters that use it and leave the namespace stuck in Terminating. If the Tenant is already gone, remove the label first with kubectl label namespace tenant-foo tenant.cozystack.io/tenant-root-
```

There are a few limits.

- The bypass is removing the label, like with the guardrail. That also
takes the namespace out of the tenant-root egress policy and the
dashboard tree, which only matters for a namespace that is about to go.
- Migration 49 labelled every `tenant-*` namespace that existed when it
ran, so a non-tenant namespace with such a name is refused too and needs
the same label removal.
- Because the garbage collector is exempt, a parent namespace whose
label was removed and that is deleted with foreground propagation still
cascades into its children directly.
- A DELETE of a namespace that is already Terminating is answered before
admission, so the policy does not see it. It changes nothing either.
- A cluster that made helm-controller impersonate another account
(`--default-service-account`, or `spec.serviceAccountName` on Tenant
HelmReleases) would need that account allowed. Cozystack sets neither.
- It does not change how Kamaji cleans up once the order is broken,
which is the second idea in #5002.

The helper rename in `external_tls_policy_test.go` (`loadPolicy(t,
path)` and `compilePolicy`) is byte for byte the one in #2622, so the
two merge in either order.

Checked locally.

- `go test ./pkg/registry/apps/application/ -run
'TestTenantNamespaceDeletePolicy|TestDatabaseExternalTLSPolicy' -count=1
-v` passes 4 tests, the new `TestTenantNamespaceDeletePolicy_Decisions`
with 11 cases. `go test ./pkg/registry/... ./pkg/config/...
./pkg/cmd/server/...` passes in all 11 packages that have tests.
- `helm unittest` in `packages/system/cozystack-basics` with
helm-unittest 1.1.2 in docker passes 12 suites and 70 tests, against 11
and 66 on main. The new tests on main without the template fail, both Go
tests and 4 helm tests in 2 suites.
- 12 mutations of the template each fail a test. They exempt
`system:masters`, drop either exempt group, select on the no-delete
label, add UPDATE, always name tenant-root as the parent, drop the
capability gate, label tenant-root, switch to Warn, switch failurePolicy
to Ignore, drop the bypass from the message and drop the objectSelector.
- The rendered policy, the guardrail and namespaces rendered from the
tenant chart were applied to kube-apiserver v1.33.0, v1.35.0 and v1.36.2
from setup-envtest, 16 checks on each. Real and dry-run deletes by a
cluster admin, by an OIDC user bound to cluster-admin and by a
cozy-system and a tenant ServiceAccount are refused with 403. The flux
and garbage collector identities get their delete accepted. tenant-root
gets the guardrail's denial, and with the guardrail binding removed a
dry-run delete of it goes through, so this policy does not select it. A
`tenant-*` namespace without the label and a tenant namespace after its
label is removed are accepted for deletion. With this binding removed, a
dry-run delete by the cluster admin goes through. That harness is not
part of the PR.
- `hack/bats-no-negated-assert.bats`, `hack/bats-no-exit-trap.bats`,
`hack/bats-runner-coverage.bats`, `hack/helmrelease-readiness.bats` and
`hack/route-hostname-policy-version-coverage.bats` pass under
`hack/cozytest.sh`.

The new e2e case in `hack/e2e-install-cozystack.bats` creates an empty
Tenant, checks that a server dry-run delete of its namespace is refused
with the message, deletes the Tenant and waits for the namespace to go.
It was not run here, there is no Cozystack cluster at hand. Its kubectl
steps were run against the envtest apiserver 1.35, the selector and
field selector read, the refused dry-run with the exact strings it greps
for, and `kubectl wait --for=delete` on a namespace that is gone. The
Chainsaw suites that delete Tenants, the gateway inheritance ones among
them, go through the uninstall path as well.

The release-1.6 backport is #5012.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The trigger map was walked against the diff. It adds a policy and tests
to cozystack-basics and an e2e case, and no entry covers that. The
website docs do not tell users to delete a tenant namespace directly,
and terraform-provider-cozystack deletes Tenants through the apps API.

### Release note

```release-note
fix(cozystack-basics): refuse a direct DELETE of a tenant namespace unless it comes from helm-controller or the garbage collector, and point to deleting the Tenant instead, so a tenant namespace deleted by hand no longer gets stuck Terminating on Kamaji finalizers
```
2026-11-13 18:25:04 +03:00
Aleksei Sviridkin 51282a1243 fix(piraeus-operator): keep the LINSTOR error-report alerts quiet when the controller pod moves (#4963)
## What this PR does

`linstorSatelliteErrorRate` fires for every node that has any old
LINSTOR error report whenever the LINSTOR controller pod moves to
another node, and it keeps firing for 15 minutes.
`linstor_error_reports_count` is exported by the controller, so its
series carry the controller pod's `instance` and `controller_node`, and
a moved pod starts new series that hold the existing counts.
VictoriaMetrics'
[`increase()`](https://docs.victoriametrics.com/victoriametrics/metricsql/#increase)
starts from the last sample before the window, and when a series has
none,
[`rollupDelta`](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/v1.136.0/app/vmselect/promql/rollup.go#L1859-L1901)
counts it from zero whenever its first value is below ten times the next
change plus one. That is the case the docs describe as `increase`
ignoring "the first value in a series if it is too big" under
[`increase_pure`](https://docs.victoriametrics.com/victoriametrics/metricsql/#increase_pure),
and report counts are never that big, so every new series counts all of
its reports as new. The `up` guard from #2265 matches `on(instance,
job)`, and the new controller is up, so it lets them through.

Seen on a v1.6.5 cluster with VictoriaMetrics v1.136.0. During the
upgrade the controller pod moved to another node, and the alert fired
for all three nodes with values 1, 1 and 2 for 15 minutes, while
`linstor error-reports list --since 1h` was empty and the four reports
that exist were weeks old. The day before, it fired the same way for
about 15 minutes when only the `cluster` external label of vmagent
changed.

A satellite coming back to the metric does the same. In LINSTOR v1.33.3
the controller asks every satellite for its reports on each scrape and
leaves out any satellite that is not connected
([`prepareErrRequestApi`](https://github.com/LINBIT/linstor-server/blob/v1.33.3/controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlErrorListApiCallHandler.java#L240-L259)),
so after a node reboot or any other disconnect longer than five minutes
the satellite's old count is counted again.

Both rules now take the count per `hostname`, hold the last value for an
hour with `last_over_time`, and run `increase()` over that as a
one-minute
[subquery](https://docs.victoriametrics.com/victoriametrics/metricsql/#subqueries),
whose inner points VictoriaMetrics aligns to that step. Keyed on
`hostname`, a moved controller or a changed label continues the same
series, and the hold bridges a disconnect or a slow controller start of
up to an hour. The `up` guard matches `on()`, since the aggregated
result has no `instance` any more. A new report still fires with the
same value and stays for 15 minutes, at most a minute later than before
because of the one-minute step. The alerts now carry `hostname` and
`severity`, and no longer the controller's `instance`, `job` and
`controller_node`, the `module`, or the external labels of the scrape.

`linstorControllerErrorRate` gets the same form. Its `hostname` is the
controller pod name and the piraeus controller keeps its reports in an
[`emptyDir`](https://github.com/piraeusdatastore/piraeus-operator/blob/v2.10.2/pkg/resources/cluster/controller/controller-deployment.yaml#L111-L112),
so a new pod starts with no reports and a move alone does not trip it,
but a label change does whenever the controller has reports.

A gap longer than the hour is not covered. After it the series starts
over, and as before the `up` guard holds the alert back only if the
controller target recorded failed scrapes during the gap.

I checked it against the stored metrics of the cluster above with
read-only queries to vmselect. Over the upgrade the current expression
returns the three nodes from 15:45:30 to 16:00:00 UTC at a 30-second
step, the same interval as vmalert's `ALERTS` series, and over the label
change from 20:27:30 to 20:42:00. The new expression returns nothing
over both windows, and without `> 0` it returns each of the three nodes
with 0 at every step, 88 and 121 points. The same storage has six gaps
of 8 to 113 minutes over one night with no failed scrape recorded in
them. Replaying vmalert's instant query of the `increase()` part every
30 seconds in the 154 evaluations after those gaps, the current
expression returns at least one node in 99, the `hostname` key without
the hold in 84, and the new expression in 30, all of them after the
113-minute gap. Retention there does not reach back to a real new
report, so the firing side is covered by the tests.

`hack/linstor-error-report-alerts.bats` replays both rules through
`vmalert-tool` v1.136.0, pinned by digest, which is the VictoriaMetrics
release victoria-metrics-operator v0.68.4 [deploys by
default](https://github.com/VictoriaMetrics/operator/blob/v0.68.4/internal/config/config.go#L38),
with 30-second samples and 15-second evaluations like the platform's
vmagent and vmalert. The suite needs docker, like the migration suites,
and only the image pull touches the network.
`hack/testdata/linstor-error-report-alerts/unittest.yaml` has 20 cases
in five groups, which cover a controller move followed by a new report,
a changed external label for both rules, one satellite away for 40
minutes and one that comes back with a new report, a new report and a
first report on a node for both rules, and a controller that cannot be
scraped for 70 minutes. Against the rule on main 13 cases fail, 9 of
them because old reports fire again after the move, the label change or
the disconnect, and the other 4 on the label set alone, since the
current alerts carry the controller's labels. Each of seven mutations of
the new expressions turns at least one case red, namely no hold, a
15-minute hold, the guard on `instance`, no guard, the key on `hostname`
and `instance`, the key on `hostname` and `cluster`, and the controller
rule as on main. With Bats 1.14.0 at the commit CI pins, the new suite
with `alert-severity-contract`, `bats-strict-setup`,
`bats-no-negated-assert`, `bats-no-exit-trap`, `bats-runner-coverage`
and `image-pin-consistency` passes 74 of 74, and `helm template` renders
the same 15 rules as main. I have not run the new rules in vmalert on a
live cluster.

The commit cherry-picks onto release-1.7 as is, and the suite passes
there. On release-1.6 it applies without conflicts, but that branch runs
`hack/*.bats` through `hack/cozytest.sh` and has no
`hack/test_helper.bash`, so the suite there has to drop its `load
test_helper` line.

### Screenshots

Not a UI change.

### Downstream repositories

A rule file under `packages/system/piraeus-operator/alerts/`, a new Bats
suite under `hack/` and its test data. No values, schema, README, image,
make target or `ApplicationDefinition` change, and the website does not
mention these rules or the metric, so no trigger map entry applies.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(piraeus-operator): the LINSTOR error-report alerts no longer fire on old reports when the LINSTOR controller pod moves to another node, when an external label of the scrape changes, or when a satellite reconnects within an hour. They now count reports per node, and their alerts carry the node's hostname instead of the controller pod's instance, job and controller_node labels.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved LINSTOR controller and satellite error alerts to account for
recent error reports over a 15-minute window, even when metrics are
intermittent or scrape labels change.
* Updated alert matching to better handle controller moves between nodes
and satellites that disappear from metrics while disconnected.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 18:18:47 +03:00
Aleksei Sviridkin 4fa6bfe30b feat(monitoring): alert when pods of one workload share a node (#4993)
## What this PR does

On three 3-node clusters running v1.6.4 and v1.6.5 we found a tenant
etcd with two of its three members on one node, a tenant Redis with two
of its three Sentinels on the node that also ran the master, and several
VictoriaMetrics and VictoriaLogs pairs (vmstorage, vmselect,
vmalertmanager, vlstorage) sharing a node. Nothing alerted. Losing one
of those nodes would have cost the etcd or Sentinel quorum, or both
copies of a storage tier. Spread constraints keep new pods apart once
they are in place (#4986 for etcd, #4989 for monitoring, #4990 for
Redis), but a placement that already exists, or that comes from a soft
rule or a drain, stays as it is, because the scheduler does not move
running pods.

This adds `PodReplicasColocated`, a warning in a new
`workload-placement` group under
`packages/system/monitoring-agents/alerts/`. It fires when two or more
running pods of one StatefulSet, ReplicaSet, CNPG `Cluster` or
`StrimziPodSet` have been on one node for an hour while the cluster has
at least as many schedulable nodes as the workload has running pods, so
five replicas on three nodes stay quiet. The alert carries `cluster`,
`namespace`, `owner_kind`, `owner_name` and `node`, plus the `severity`,
`exported_instance` and `service` labels the other platform alerts set,
and its value is the number of pods on that node. DaemonSets, Jobs and
every other owner kind are not in the list, so a DaemonSet surge pod
next to its predecessor does not count.

etcd-operator gives every member pod its own `EtcdMember` as controller
owner
([etcdmember_controller.go#L546](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdmember_controller.go#L546)),
so grouping by owner would see one pod per group. Members get their
names from `GenerateName: <cluster>-`
([etcdcluster_controller.go#L434](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdcluster_controller.go#L434),
[#L839](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdcluster_controller.go#L839)),
and members adopted from the old StatefulSet keep names like `etcd-0`
([adopt.go#L103](https://github.com/cozystack/etcd-operator/blob/v0.5.6/internal/migrate/adopt.go#L103)).
The rule drops the last dash-separated part of the member name and
reports the group as `EtcdCluster <name>`. With the etcd package that is
one group per tenant namespace, and two etcd clusters in one namespace
would still be told apart. CNPG instance pods carry a controller
reference to their `Cluster`
([cluster_create.go#L1580](https://github.com/cloudnative-pg/cloudnative-pg/blob/v1.30.0/internal/controller/cluster_create.go#L1580)),
and Strimzi pods one to their `StrimziPodSet`
([StrimziPodSetController.java#L369](https://github.com/strimzi/strimzi-kafka-operator/blob/0.45.1-rc1/cluster-operator/src/main/java/io/strimzi/operator/cluster/operator/assembly/StrimziPodSetController.java#L369)).

Owner and node both come from `kube_pod_info`. kube-state-metrics fills
`created_by_kind` and `created_by_name` from the controller owner
reference and `node` from `spec.nodeName`
([pod.go#L638-L653](https://github.com/kubernetes/kube-state-metrics/blob/v2.15.0/internal/store/pod.go#L638-L653)).
`kube_pod_owner` and `kube_pod_status_phase` have no node label of their
own, and the kube-state-metrics scrape adds `node` from
`__meta_kubernetes_pod_node_name` with `honorLabels: true`
(`templates/kube-state-metrics-scrape.yaml`), so on those series `node`
is the node of the kube-state-metrics pod. The test data keeps that
label on the phase series. A pod counts while
`kube_pod_status_phase{phase="Running"}` is 1, so a Failed or Pending
pod on the same node does not. `kube_pod_labels` does carry every pod
label here (`--metric-labels-allowlist=pods=[*]`), but the rule does not
depend on it.

A node counts as schedulable unless it has a `NoSchedule` or `NoExecute`
taint. A cordon shows up as the
`node.kubernetes.io/unschedulable:NoSchedule` taint
([node_lifecycle_controller.go#L549-L555](https://github.com/kubernetes/kubernetes/blob/v1.33.12/pkg/controller/nodelifecycle/node_lifecycle_controller.go#L549-L555)),
and not ready and unreachable nodes get theirs from their conditions
([#L87-L99](https://github.com/kubernetes/kubernetes/blob/v1.33.12/pkg/controller/nodelifecycle/node_lifecycle_controller.go#L87-L99)),
so one clause covers all three, and a tainted control plane does not
count either. Pods that a drain had to put together on the remaining
nodes therefore stay quiet while the node is out, and the hour starts
once it is back.

The alert waits an hour. A replacement pod runs next to its terminating
predecessor only for the predecessor's grace period, and node
maintenance takes the drained node out of the count, while a placement
that has held for an hour does not fix itself. The alert is about a risk
rather than an outage, so the wait costs nothing. The VMAlert sets both
`remoteWrite` and `remoteRead`, so the pending state survives a vmalert
restart ([vmalert
docs](https://docs.victoriametrics.com/victoriametrics/vmalert/#alerts-state-on-restarts)).

Not covered are workloads that run one pod per owner, such as ClickHouse
with one StatefulSet per replica
([patterns.go#L45-L53](https://github.com/Altinity/clickhouse-operator/blob/release-0.25.2/pkg/model/chi/namer/patterns.go#L45-L53))
and the KubeVirt VMs behind a tenant Kubernetes node group. They would
need grouping by pod labels, which this rule avoids. The node count is
cluster-wide, so a workload that a node selector or affinity restricts
to fewer nodes than it has pods fires too.

`hack/workload-placement-alerts.bats` takes `.spec` of the VMRule and
replays it through `vmalert-tool` v1.136.0 pinned by digest, the release
vm-operator v0.68.4 deploys by default
([config.go#L38](https://github.com/VictoriaMetrics/operator/blob/v0.68.4/internal/config/config.go#L38))
and the same image #4963 and #4985 use.
`hack/testdata/workload-placement-alerts/unittest.yaml` has 13 cases
with 24 checks. Two of three StatefulSet pods on one node fire at 61
minutes and not at 59, with labels and annotations pinned. Spread pods,
five ReplicaSet pods on three nodes, a DaemonSet surge pod, two pods of
one Job, a Failed pod next to a running one, and one pod each in two
clusters that share workload and node names stay quiet. etcd members
fire as one `EtcdCluster` per namespace, for generated and adopted
member names. A replacement pod next to its terminating predecessor for
twenty minutes stays quiet, and the same overlap held past the hour
fires and resolves when it ends. A cordoned node keeps the alert quiet
until it is uncordoned, and a node with unreachable taints keeps it
quiet throughout. ReplicaSet, CNPG `Cluster` and `StrimziPodSet` pods
fire with their owner kind.

```console
$ bats hack/workload-placement-alerts.bats
1..1
ok 1 workload placement alerts pass their vmalert-tool cases
```

With the expression replaced by one that never fires, the suite prints
`not ok 1` and vmalert-tool reports 5 of the 24 checks, which are all
the checks that expect an alert. On main, where the rule file does not
exist, the suite fails as well. Each of these changes to the rule fails
at least one check, with the number of failing checks in brackets. `for:
15m` (5), no phase filter (1), no node count (6), no `cluster` label
(6), DaemonSet and Job in the kind list (3), no taint clause (4), `<`
instead of `<=` (5), etcd members grouped per member (1).

With Bats 1.14.0 at the commit CI pins, the suite passes together with
`alert-severity-contract`, `bats-strict-setup`,
`bats-no-negated-assert`, `bats-no-exit-trap`, `bats-runner-coverage`,
`bats-runner-contract`, `image-pin-consistency` and
`check-commit-trailers`, 109 of 109, and the severity contract lists the
new alert as `warning`. helm-unittest 1.1.2, the version CI pins, passes
the monitoring-agents chart with 19 of 19 tests in 6 suites, and `helm
template` with Helm 3.17.3 renders the VMRule the same as the file. Like
the migration suites the new suite needs docker, and only the image pull
touches the network. I have not run the rule in vmalert on a live
cluster.

On release-1.6 kube-state-metrics 2.15.0, its scrape config, the alerts
template, the VMAlert and vm-operator v0.68.4 are the same, and
etcd-operator v0.5.5 and CNPG 1.28.2 set the same owners and member
names. The new files apply there without changes. That backport is a
separate PR with one more commit that drops the `load test_helper` line,
because release-1.6 runs `hack/*.bats` through `hack/cozytest.sh`.

### Screenshots

Not a UI change.

### Downstream repositories

A new alert file under `packages/system/monitoring-agents/alerts/`, a
Bats suite under `hack/` and its test data. No values, schema, README,
image or `ApplicationDefinition` change, and the website does not list
platform alerts by name, so no trigger map entry applies.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(monitoring): new warning alert PodReplicasColocated fires when two or more running pods of one StatefulSet, ReplicaSet, CNPG Cluster, StrimziPodSet or etcd cluster have shared a node for an hour while the cluster has a schedulable node for each of them.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added monitoring for workloads whose running pod replicas are
colocated on the same node.
* Alerts include workload, node, cluster, and severity details, along
with quorum-loss guidance.
* Coverage includes StatefulSets, ReplicaSets, Clusters, StrimziPodSets,
and EtcdMembers.

* **Tests**
* Added validation for workload-placement alerts across scheduling,
availability, cluster-isolation, and pod-replacement scenarios.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 17:38:58 +03:00
Aleksei Sviridkin d089decab9 Merge branch 'main' into fix/tenant-namespace-direct-delete-guard
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 14:20:44 +03:00
Aleksei Sviridkin 054852cfce Merge branch 'main' into feat/pod-replicas-colocated-alert
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 14:20:35 +03:00
Aleksei Sviridkin f1067fdd64 Merge branch 'main' into fix/linstor-error-report-alerts-controller-move
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 14:20:23 +03:00
Aleksei Sviridkin e8475963ee Merge branch 'main' into fix/monitoring-infoinhibitor
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 14:07:04 +03:00
Myasnikov Daniil 48eae9ad36 ci: run only the backport-batch tests for a backport-batch change
A pull request that changed only cmd/backport-batch built every image,
installed the platform for e2e and ran the whole unit lane, so a
one-line test fix waited hours on suites that cannot exercise it. The
backport-batch tests themselves ran on every code pull request,
although only a change to the tool, to what it imports or to the pages
its tests read can break them.

Their inputs are listed beside the tool and checked against its import
graph, because a hand-kept list that fell behind would switch the
tests off without anyone noticing.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-13 13:11:46 +05:00
Myasnikov Daniil 1619cb4eb9 ci: keep the CI-only Go tools from escalating builds and e2e
cmd/backport-batch and cmd/api-gate are built into no image and run
nowhere in the e2e sandbox, yet a change to either rebuilt every image
and ran the full Chainsaw suite, and widened a package change made in
the same pull request to everything. Every other directory under cmd/
is a shipped binary and keeps escalating.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-13 13:11:46 +05:00
myasnikovdaniil f061739ce2 ci(e2e): raise the E2E job limit from 215 to 265 minutes (#5288)
## What this PR does

Full chainsaw run doesn't fit in 215 minutes anymore. Last four nightly
runs were cancelled at the limit, and PRs that select full suite end
same way while every suite that got to run passed (#5283 for example).

This raises limit for all four jobs that run the suite (pull-requests,
e2e-fork, e2e-tag, nightly) from 215 to 265, so they stay on one figure.
Comments that state the cap follow it. Making e2e itself shorter is
separate work.

Part of #5261, `Collect report` ceiling in fork workflow from there is
not covered here.

### Screenshots

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
ci(e2e): E2E job limit raised from 215 to 265 minutes.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Chores**
* Increased the time limit for end-to-end test runs from 215 to 265
minutes across applicable workflows. This allows longer test runs to
complete before the overall job is stopped.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 09:16:13 +03:00
Aleksei Sviridkin 2cf80c02a4 fix(clickhouse): keep failed backup attempts out of the bucket (#5229)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

A ClickHouse BackupJob attempt that fails during upload leaves its local
backup on the data volume, and with the upstream defaults the
clickhouse-backup sidecar never prunes local backups and resumes every
unfinished upload when it starts. On the next sidecar restart the failed
attempt is uploaded to the bucket as an archive no Backup refers to, and
while that resumed upload runs the sidecar answers new operations with
423, which burns the strategy Job's retries in seconds. The strategy
itself only waited for the sidecar API to respond and then started its
operation regardless.

The sidecar now runs with `API_COMPLETE_RESUMABLE_AFTER_RESTART=false`,
so a failed attempt is never published, and `BACKUPS_TO_KEEP_LOCAL=-1`,
so at most the newest local backup is kept and it is removed once its
upload succeeds. Backups already left on existing volumes are cleaned up
by the first backup after the upgrade. The Altinity strategy now waits
until the sidecar has no operation in progress before it starts a backup
or a restore, and fails the attempt with a clear message instead of
proceeding when the sidecar stays busy or unreachable.

Waiting for an idle sidecar changes what a retried strategy Pod does.
The Job retries a failed Pod up to twice, and a Pod can die after its
operation was accepted (a dropped poll under errexit, an eviction, a
node drain) while the sidecar keeps running it. Before, the retry hit
423 and failed; with the idle gate it would wait for the running
operation to finish and start it again. For a restore that is
destructive, because `restore_remote` drops every table it restores
before recreating it, so the second run discards whatever was written
after the first one finished. A retried restore therefore takes over a
`restore_remote` of the same archive that is still running and waits for
its result instead of starting another. Once an operation is accepted,
polls tolerate up to six failures in a row before the attempt fails, and
the sidecar losing the operation from its in-memory action log, or
ending it cancelled, fails the Pod instead of leaving it polling.

A retried backup works from the archive name the controller records for
its BackupJob. That name is now `<application>-<YYYYMMDDHHMMSS>-<first
12 hex digits of the BackupJob UID>`, so two runs of one application
started in the same second, as two Plans on one schedule are, never
share an archive. It is exactly as long as the previous
`clickhouse-<application>-<YYYYMMDDTHHMMSSZ>`, so an `s3PathOverride`
prefix that fit before still keeps every object key within the 1024-byte
limit. A retry takes over a `create_remote --resume` of its archive that
is still running, counts an upload of its archive whose latest run ended
in `success` as its own result instead of uploading the archive again,
and otherwise resumes the upload. Restoring a Backup that records no
archive still picks an archive by timestamp, and that path accepts only
`clickhouse-<application>-<timestamp>` names, so it never picks a
per-run archive.

One limit remains. A finished `restore_remote` row is not taken over:
the sidecar never trims its action log, so a `success` row of the same
archive may be left by an earlier RestoreJob, and counting it as this
one would report a restore that never ran. A retry that arrives after
the restore it replaces had already finished, but before that Pod saw
the result, therefore still runs the restore a second time.

Upgrading changes the ClickHouseInstallation pod template, so ClickHouse
pods with `backup.enabled` are rolled once. A backup Job started before
the upgrade by a controller that named archives the previous way is not
re-rendered and keeps writing that name, so the controller records the
previous name when the new one is not on the Job. The local copy of the
latest failed attempt stays on the volume until the next backup. Objects
partially written to the bucket by an upload that fails midway are not
removed by clickhouse-backup and are left for a follow-up.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(clickhouse): failed ClickHouse backup attempts are no longer uploaded to the bucket when the backup sidecar restarts, local backups no longer accumulate on the data volume, the backup strategy waits for the sidecar to be idle instead of failing with HTTP 423, a retried ClickHouse restore takes over the restore that is still running instead of starting it a second time, and a retried ClickHouse backup takes over or reuses its own upload instead of starting a new one. Every ClickHouse backup run now writes an archive of its own, even when two runs start in the same second, under a name no longer than before. ClickHouse pods with backups enabled are restarted once on upgrade.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Backup and restore operations wait for the backup service to become
idle and join a matching operation already in progress rather than
starting a duplicate.
* Restore operations use the archive the Backup records and fail when it
is not in remote storage; only a Backup that records no archive falls
back to the newest eligible archive by timestamp.
* New backups use distinct archive names, including for runs started in
the same second.
* Local backup copies are discarded after upload, and completed uploads
are not resumed after a service restart.
* **Bug Fixes**
* Operations now fail when the service cannot be queried, an operation
disappears from its activity log, or it ends with an error or
cancellation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 08:58:33 +03:00
Aleksei Sviridkin bc268ce759 fix(monitoring-agents): let the release uninstall when the VictoriaMetrics operator is gone first (#5152)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Disabling the Monitoring Agents addon of a tenant Kubernetes cluster
leaves `<cluster>-monitoring-agents` (release `cozy-monitoring-agents`,
installed into the tenant cluster) in `UninstallFailed` for good. Seen
in production on 1.6.5 and still reproducible from `main`.

`monitoring-agents`, `victoria-metrics-operator` and
`vertical-pod-autoscaler-crds` in
`packages/apps/kubernetes/templates/helmreleases/` are all gated on
`addons.monitoringAgents.enabled`, so they disappear together and Flux
uninstalls them in parallel: `dependsOn` is not honoured on deletion.
Two things then break the uninstall of `monitoring-agents`:

1. The VictoriaMetrics operator is often gone first. The operator's
`apps.victoriametrics.com/finalizer` stays on the `VMAgent`, nobody
removes it, and the deletion hangs. The manual workaround was to strip
the finalizer, after which GC removes the VMAgent's Deployment, Service,
Secrets and ServiceAccount through their owner references.
2. The VPA CRD was deleted with its own release while the stored
manifest still contains `VerticalPodAutoscaler vpa-vmagent`, so Helm
cannot build the manifest to uninstall. That root cause is fixed by
#3586 (`helm.sh/resource-policy: keep` on the tenant addon CRDs), which
this PR does not redo.

This PR:

- adds a `pre-delete` hook (`templates/vm-cleanup-hook.yaml`) that
deletes every VictoriaMetrics object the release owns, waits up to 60s
for a live operator to finalize them properly, and clears their
finalizers by hand when it has not. Objects are selected through Helm's
`meta.helm.sh/release-name` / `release-namespace` annotations rather
than by name, so the 34 `VMRule`s rendered from `alerts/` and the
`VMServiceScrape` in `cozy-victoria-metrics-operator` are covered along
with the VMAgent. The kinds handled are every VictoriaMetrics kind the
chart renders: `VMAgent`, `VMRule`, `VMServiceScrape`, `VMPodScrape`,
`VMNodeScrape`. Operator v0.68.4 only finalizes the VMAgent; the other
kinds are covered in case another operator version finalizes them, and
for them the patch is a no-op. A kind whose CRD is already gone is
skipped, and the script always exits 0, since a failing pre-delete hook
would itself block the uninstall. The Job is bounded by
`activeDeadlineSeconds: 180` and `backoffLimit: 0`; its kubectl calls
carry no `--request-timeout`, which would make kubectl skip the
in-cluster credentials (`hack/check-hook-request-timeout.bats`) and
reuses the digest-pinned `docker.io/clastix/kubectl:v1.32` image the
etcd-operator hook uses, via a new `kubectlImage` value. The shape
follows
`packages/system/forklift/templates/forklift-cr-cleanup-hook.yaml`.
- guards `templates/vpa.yaml` on `.Capabilities.APIVersions.Has
"autoscaling.k8s.io/v1/VerticalPodAutoscaler"`, so a cluster that does
not serve the CRD never records a VPA in the release manifest. Cheap
defence in depth next to #3586.

Points for review:

- The hook's RBAC is a ClusterRole/ClusterRoleBinding, limited to `get,
list, delete, patch` on the five `operator.victoriametrics.com`
resources, rather than Roles. The objects span two namespaces, and a
Role in `cozy-victoria-metrics-operator` could not be created if that
namespace were already gone or terminating, which would fail the hook
and the uninstall with it. All hook objects carry
`hook-succeeded,hook-failed,before-hook-creation` except the Job, which
keeps a failed pod for its logs (`ttlSecondsAfterFinished: 600`), as in
the forklift hook.
- When the operator is already gone, clearing the finalizer skips
whatever the operator does on finalize, notably the cluster-scoped RBAC
it creates for the VMAgent, which has no owner reference. That residue
is the same the manual workaround left, and it is no worse than today's
stuck release; removing it would mean hard-coding the operator's naming
scheme for those objects, which I preferred not to do.
- When the operator is alive, the hook only brings forward deletions
Helm was about to make, so the operator's normal finalization still
runs.

Tests: `tests/uninstall_test.yaml` asserts that the hook is a
`pre-delete` Job in the release namespace with a bounded runtime, that
its RBAC is hook-scoped and grants exactly the verbs needed on every
VictoriaMetrics kind the chart renders, that the binding targets the
hook's ServiceAccount, that the script selects by release ownership,
deletes, clears finalizers and exits 0, and that the VPA renders only
when the capability is present (using `capabilities.apiVersions`). With
the hook removed and `vpa.yaml` reverted, 6 of the 7 tests fail; with
the fix, `helm unittest packages/system/monitoring-agents` passes 26/26.

`hack/monitoring-agents-vm-cleanup-hook.bats` runs the rendered script
against a fake `kubectl` that keeps the objects in a state file, with
fake `date` and `sleep`. It checks that only objects annotated for this
release are deleted and patched, that with the operator gone the hook
polls through the grace period and then clears the finalizers, that a
live operator's objects are never patched, that a missing CRD still
exits 0 and cleans the other kinds, and that a release with nothing left
exits 0 without deleting anything. Removing the ownership condition,
making the patch a no-op, or inverting the polling condition each turns
this file red. The hook has not yet been run on a live tenant cluster.

Backport: labelled `kind/backport` (release-1.7). The release-1.6
backport is done by hand in #5158. The commit cherry-picks onto
`release-1.6` with a single trivial conflict in `values.yaml` (context
only: 1.6 lacks the `prometheus-node-exporter` resources block the new
`kubectlImage` block sits after on `main`). Resolved locally, the
chart's tests pass on 1.6 (9/9); 1.6's chart renders the same
VictoriaMetrics kinds. Note that `release-1.6`'s monitoring-agents
`Makefile` has no `test:` target, so CI does not run this chart's unit
tests on that branch.

### Screenshots

Not a UI change.

### Downstream repositories

I walked the trigger map against the diff (a system chart's templates,
`values.yaml` and tests): no package added or renamed, no platform
values, no app schema, no CRD, no tooling or convention change. I also
searched cozystack/website for a documented VMAgent finalizer workaround
that this would make obsolete and found none.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(monitoring-agents): disabling the Monitoring Agents addon of a tenant Kubernetes cluster no longer leaves the monitoring-agents release stuck in UninstallFailed when the VictoriaMetrics operator is removed first; a pre-delete hook now releases the VictoriaMetrics objects' finalizers, and the VPA is only rendered when its CRD is served.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* VictoriaMetrics resources belonging to a release are cleaned up during
uninstall, including resources that remain because of finalizers.
Cleanup is limited to resources owned by that release.
* Vertical Pod Autoscaler resources are rendered only when the cluster
supports the required API.
* **Configuration**
* Added an image setting for the uninstall cleanup job, including a
pinned image version and pull policy.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 08:58:10 +03:00
Alexey Artamonov dae76d186c Merge branch 'main' into fix/clickhouse-idle-before-backup
main pins the archive name a backup Pod keeps across a wait for
late bucket credentials, against the controller's archive helper.
That helper now names a run's archive with part of the BackupJob's
UID, so the merged test has to give its BackupJob one.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-12 17:20:19 +03:00
Aleksei Sviridkin 3badbdef74 fix(clickhouse): let a running backup sidecar pick up system-bucket credentials (#5232)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

A ClickHouse release with `backup.useSystemBucket: true` can start
before the platform projects `cozy-backups-creds` into its namespace.
Its `clickhouse-backup` sidecar took the S3 settings from environment
variables, which are fixed when the container starts, so the first
BackupJob or RestoreJob of such a release failed with `no EC2 IMDS role
found` until the Pod was restarted, and a rotated key stayed stale the
same way.

The projector now also writes a `clickhouse-backup.yml` key into
`cozy-backups-creds` (bucket, `https://` endpoint, region when set,
keys, `force_path_style` from `backupStorage.forcePathStyle`). On the
system-bucket flow the chart mounts that key as an optional Secret
volume and points `CLICKHOUSE_BACKUP_CONFIG` at it, instead of setting
`S3_*` variables. clickhouse-backup reloads its config file in each API
handler that reaches S3 (among them the `create_remote`,
`restore_remote` and `list` calls the strategy makes; the `status` and
`actions` reads it polls with do not) and kubelet rewrites a mounted
Secret in place, so credentials projected after the Pod started, or
rotated since, reach the running sidecar. `S3_PATH` stays in the
environment because the prefix belongs to the release.

Kubelet takes about a minute to refresh the volume, so the projector
stamps `backups.cozystack.io/credentials-updated-at` when the projected
content changes, and the Altinity driver holds a new backup or restore
Job for up to two minutes after that stamp, with `Ready=False` and
reason `WaitingForCredentials`; the condition is dropped once the Job
runs. A stamp that cannot be read, or one further ahead than clock skew
explains, is logged and ignored rather than holding the namespace's jobs
for good. The hold applies to every Altinity job in a namespace whose
projected Secret just changed, including releases on their own Secret,
where it is only a delay.

The new e2e Test `clickhouse-3-system-bucket-roundtrip` starts a
ClickHouse on the system bucket with no projected Secret, then runs a
backup and an in-place restore through `cozy-default` and checks that
the sidecar that started without credentials served both jobs. In CI the
system bucket's advertised host `s3.example.org` neither resolves nor
gets an ACME certificate, so for the duration of its own step the Test
serves a SeaweedFS-CA certificate under the S3 Ingress TLS Secret name
and adds a marked `rewrite name exact` line for that host to the host
CoreDNS Corefile, removing both on exit; it proves the served
certificate before any job starts.

The bug was reproduced on a test cluster running the code before this
change: a ClickHouse started before the projection failed its first
BackupJob with `no EC2 IMDS role found`. The new e2e Test has not run
green yet outside CI, because it edits the host CoreDNS, which is shared
on that cluster.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#799 (draft until this PR merges):
`operations/services/backup-classes.md` for the mounted config file, the
credentials wait and the upgrade window, without the Pod restart this
change removes
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(clickhouse): a ClickHouse release on the platform system bucket (`backup.useSystemBucket: true`) now backs up and restores without a Pod restart when its namespace had no projected backup credentials at start, and picks up rotated credentials while running. The sidecar reads its S3 settings from a `clickhouse-backup.yml` file in `cozy-backups-creds` instead of environment variables, so the upgrade rolls every such ClickHouse Pod once. The first Altinity backup or restore in a namespace after its projected credentials change waits up to two minutes (`WaitingForCredentials`) for them to reach the sidecar. Upgrade the platform as a whole: a system-bucket BackupJob started while the new chart runs against the previous controller fails, because that controller does not write the new key.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* ClickHouse system-bucket backups now use a projected configuration
file for S3 settings, supporting HTTPS endpoints and optional region and
path-style settings.
* Backup and restore jobs wait briefly for recently updated projected
credentials before starting.
* Path-style settings accept common true and false values and are
normalized consistently across backup storage integrations.
* **Tests**
* Added end-to-end coverage for system-bucket backup and restore,
verifying restored data without restarting the ClickHouse Pod or backup
sidecar.
* Added checks for TLS certificate validation and safe restoration of
shared network and ingress settings.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-12 16:15:26 +03:00
Myasnikov Daniil e56e66ae81 test(kubernetes-nodes): fail when the pool and cluster CIDRs disagree
The tenant pod and service CIDRs are literals in two charts: the
kubernetes chart writes them into the CAPI Cluster, the pool chart
into every worker's Talos machine config. The parent's comment says
they must match and keeps them "in one place", but since the pools
moved out the second copy has had nothing tying it to the first, and
a mismatch renders cleanly on both sides while workers route over a
range the control plane does not use.

The test renders both charts and compares the lists, reading the
worker side from the TalosConfigTemplate the reconcile Job's heredoc
actually emits rather than grepping the Job text.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-12 18:03:44 +05:00
Myasnikov Daniil 6b0a564bc4 test(kubernetes-nodes): cover the Talos support-matrix guard by render
The pool chart checks every worker's Talos release against the
Kubernetes version it joins, once for talos.version and once for the
release osImage resolves to, and that is now the only such check. Its
coverage was two grep tests that the call sites exist; nothing rendered
a refusal, and the pairings it has to let through were pinned only in
the parent chart, against a copy of the table.

The pool's own suite pins the in-window pairings, through talos.version
at both ends of the schema's version range and through
osImage.factory.version. The refusals need a version the schema does
not offer and the versions map does not hold, so they render a copy of
the chart with --skip-schema-validation and one extra versions.yaml row,
the way the postgres preload allowlist test reaches its backstop. They
fail if the second call is fed talos.version instead of the resolved
release, if the table is keyed on the full release instead of the
minor, or if either call is dropped.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-12 16:57:46 +05:00
Myasnikov Daniil 90f0b043a6 fix(kubernetes)!: remove the talos block the chart no longer uses
The kubernetes chart declared a talos block (version, schematicID,
imageFactoryURL, installerRepository, registryMirrors). Workers are
KubernetesNodes pools and take every one of these from their own talos
values, so an operator who set them on the cluster got no error and no
effect. The only reader left was a copy of the Talos to Kubernetes
support-matrix check, and the pool chart already runs that check
itself, against the Talos release each pool actually boots, including
one an osImage override selects. The cluster's copy compared the
Kubernetes version against a Talos release no node runs.

The block leaves the API schema, hence the "!". Stored specs that still
carry it keep rendering: the values schema does not close the object
and the API does not validate spec against it. A helm-unittest case
pins that.

Readers outside the chart moved with it. The e2e install read
talos.schematicID from this chart's values.yaml in a best-effort block
that only warns, so removing the field would have stopped the image
cache being seeded without failing anything; it now reads the pool
chart, as the proxmox template script already does. The e2e no longer
writes a talos block into the Kubernetes CR, the spec test checks the
pool chart for the key it emits, the release checklist diffs the pool
chart's pins as well, and the catalog guard drops the two tests that
compared or read the parent's block. The null-normalization Go tests
used talos keys of the real Kubernetes schema as their fixtures; they
use oidc.mode, the same required-with-default shape under an optional
root section, and addons.cilium.valuesOverride for the free-form case.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-12 16:51:38 +05:00
Myasnikov Daniil c63e83466a ci(e2e): raise the E2E job limit from 215 to 265 minutes
A green run of the full Chainsaw suite no longer fits in 215 minutes:
the last four nightly runs were cancelled at the limit, and pull
requests that select the full suite end the same way while every
suite that got to run passed. All lanes that run the suite keep one
figure, so a change that passes on a pull request is not cut off on
nightly or on a tag.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-12 15:42:25 +05:00
Alexey Artamonov 5a95bcdd64 fix(clickhouse): keep a run's archive name within the old name's length
The archive name is part of every object key, a key is capped at 1024
bytes, and the legacy s3PathOverride prefix has no length limit. The
full UID made the name 37 characters longer than before, so a prefix
that fit the old name pushed the manifest key past the cap and failed
the backup.

The run's name now drops "clickhouse-" and the "T" and "Z" of the
timestamp and spends the room on the first 12 hex digits of the UID,
so it is exactly as long as the old name for every application.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-12 11:10:43 +03:00
Aleksei Sviridkin 9978771858 feat(observability-proxy): serve tenant namespace logs through an authorizing proxy (#5214)
## What this PR does

Adds `observability-proxy`, a new system component that lets a tenant
user read the logs of their own namespace from VictoriaLogs with the
same Kubernetes credentials they already use for the console, and with
Kubernetes RBAC deciding what they can read. This is the first part of
the design proposals [Tenant application
logs](https://github.com/cozystack/community/pull/95) and [Tenant
resource-consumption
metrics](https://github.com/cozystack/community/pull/94); this PR
implements the namespace logs root only. Metrics and the per-application
roots follow in separate PRs.

Why a proxy: today the log stores are shared per root tenant and have no
notion of a caller, so tenants cannot be given direct read access
without seeing other tenants' logs. The proxy authenticates the caller,
authorizes the request from its path, and rebuilds the upstream query so
that its scope cannot be widened by the client.

Backward compatibility: this adds a new package and a new HelmRelease;
nothing existing is renamed, removed or changed, so no migration is
needed.

What is included:

- `cmd/observability-proxy` and `internal/observabilityproxy`: the
proxy.
- Serves
`/apis/cozystack.io/v1alpha1/namespaces/<ns>/logs/select/logsql/<endpoint>`
for the VictoriaLogs read endpoints the console needs (`query`, `hits`,
`stats_query_range`, `tail`, `field_values`, `stream_field_values`,
`facets`). Anything else under the root answers `404`.
- Authentication by `TokenReview`, authorization by one
`SubjectAccessReview` built from the path (`list` on `cozystack.io/logs`
in the namespace), both cached briefly.
- The upstream request is rebuilt from an allowlist of parameters; the
namespace scope is always forced as `extra_stream_filters`.
Client-supplied `extra_filters`, `extra_stream_filters`, any LogsQL
`options(...)` block, and tenant headers such as `AccountID`/`ProjectID`
are not forwarded or are refused.
- Non-canonical paths (`..`, `//`, encoded separators) are refused with
`400` before any authorization is built, because the console nginx
matches on the normalized path but forwards the raw one.
- Time window, step, bucket count and result limits are clamped;
per-user rate, concurrency and open-tail limits answer `429`. Invalid
time, step and limit parameters answer `400`.
- The store is resolved from the namespace's
`namespace.cozystack.io/monitoring` label.
- Live `tail` streams are passed through unbuffered, and the upstream
tail request disables compression so headers reach the client before the
first line.
- `packages/system/observability-proxy`: chart and image build, wired
into the root `make build`.
- Platform wiring: a new `cozystack.observability-proxy` PackageSource,
enabled by default on the full variants and opt-in on the slim ones, the
same way as `monitoring-agents`.
- RBAC: an aggregated `cozy:observability:view` ClusterRole grants
`list` on `cozystack.io/logs` to tenant `view` and above. No API server
serves this resource; the rule exists only for the proxy's
`SubjectAccessReview`. The proxy's own ClusterRole is limited to the
calls it makes (`tokenreviews`, `subjectaccessreviews`, `list`/`watch`
on namespaces).
- Console nginx: when the proxy is not installed or cannot be reached,
the observability roots answer `404` with a Kubernetes `Status` body,
which the console shows as not available on this cluster; the proxy's
own `502`/`504` for an unreachable store pass through unchanged.
- Console: one regex `location` in the console nginx routes the four
observability roots to the proxy ahead of `/apis`. The upstream is
resolved at request time, so the console starts even when the proxy is
not installed; those paths then answer `404` with a Kubernetes `Status`
body. The console Deployment now receives the cluster domain for that
resolution.
- E2E: a new Chainsaw suite `hack/e2e-chainsaw/observability-proxy/`
with two tenants, covering `401`/`403`/`200`, scope-widening attempts,
path normalization and a long-lived `tail`.

#### How it was tested

- `go test -race` and `go vet` for the proxy packages.
- helm-unittest for `observability-proxy`, `dashboard`, `tenant` and
`core/platform`; bats tests for the console nginx routing.
- The Chainsaw suite was not run as-is: it deploys its own VictoriaLogs
as the root store and refuses to run when a real one exists. Its cases
were replayed by a script against the real store on a test cluster.
- Verified on a test cluster:
- `401` without a token and with an invalid token, `403` for another
tenant's namespace, `200` for the caller's own.
- Client `extra_stream_filters`/`extra_filters` (GET and POST), an
`AccountID` header and another tenant's stream named inside the query do
not widen the scope; `options(...)` answers `400`; a path with `..`
pointing at another tenant answers `400`; an endpoint outside the
allowlist answers `404`.
- Through the full browser chain (OIDC login, gatekeeper, gateway,
console nginx), a single `tail` stayed open for more than two minutes
without reconnecting, and requests from the same session to other
tenants' roots answered `403`.
- Not verified: the `429` limits under real load. `golangci-lint` has
not been run locally.

#### Known limitations

- The proxy is reachable only through the console route; tenant
workloads have no egress rule to it. Every request takes its concurrency
or tail slot before the body is read; the token and access reviews and
the body read run under `--request-timeout`, so a stalled kube-apiserver
answers `503` rather than holding the request; and writes to the client
carry a deadline (renewed per line for tails). `--request-timeout=0`
disables those bounds and should not be used in production.
- The first lines a newly started pod writes can be missing from an
already open `tail`, although they are in the store. The likely cause is
that Fluent Bit picks up a new log file later than the VictoriaLogs
live-tail window; later lines of the same pod arrive. A history query
shows them.
- The per-application roots (`.../<plural>/<name>/log`, `.../metrics`)
and the namespace metrics root answer `404` in this PR.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/issues/787 (docs land
once the implementation PRs merge)
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Downstream: `cozystack/website` is reached. A new platform component
(`packages/core/platform/sources/observability-proxy.yaml`) triggers
`content/en/docs/next/guides/platform-stack/_index.md` and
`operations/configuration/licenses.md`; adding it to the system bundle
(`packages/core/platform/templates/bundles/system.yaml`) may also need a
line in `operations/configuration/variants.md`. A user-facing page for
the logs API and the new tenant `view` permission is not required by the
trigger map but would be useful. Tracked in cozystack/website#787. Other
repositories: no trigger matched. `packages/core/platform/values.yaml`
is not changed (only a test fixture), no package under `packages/apps/`
gains a values field, and no CRD changes. The root `Makefile` `build`
target builds one more image; this does not touch `hack/` or the make
targets `ccp` relies on, so `ccp` is not considered affected, but a
reviewer may want to confirm.

### Release note

```release-note
feat(observability-proxy): add an authorizing proxy that lets tenant users read their namespace logs from VictoriaLogs with their Kubernetes credentials, under `/apis/cozystack.io/v1alpha1/namespaces/<ns>/logs/`. Access requires the tenant `view` role or higher. The component is enabled by default on full variants and opt-in on slim variants.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Added an observability proxy for tenant-scoped log access through the
console, including live log streaming. Requests are authenticated,
access-controlled, and subject to configurable rate and concurrency
limits.
* Added platform packaging and deployment support, with console routing
for observability requests.
* **Tests**
* Added automated coverage for access control, tenant scoping, query
handling, streaming, proxy behavior, and deployment configuration.
* **Documentation**
* Updated end-to-end test guidance with host requirements for running
the observability proxy suite.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-11 16:24:25 +03:00
Alexey Artamonov 35823c6c9c test(clickhouse): list the system-bucket round-trip in the suite header
The header names the backup Tests of the suite and why they are split;
it still described two of them after the third was added, so a reader
starting from it would miss the only Test that covers the system-bucket
path.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 14:56:26 +03:00
Alexey Artamonov c26d2773a5 test(clickhouse): name what really leaves the round-trip's Secret behind
The projected Secret the preconditions delete is left by whichever
BackupJob ran in tenant-root before this Test, from this suite or
another: Chainsaw does not fix the order of the Tests in a suite, so no
particular one can be named. The restore budget is also not spent on a
credentials hold: by then the stamp is older than the backup's own
two-minute wait.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 14:42:11 +03:00
Aleksei Sviridkin fb5ac1826a test(observability-proxy): prepare seed stubs in both unit runners
The POSIX runner does not invoke Bats setup hooks. Initialize the
capture stub from the test body so the compatibility lane stays offline.

Fixes: 5a2582e21c ("test(observability-proxy): preserve JSON seed request bodies")
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-11 12:12:11 +03:00
Aleksei Sviridkin 66acdb80e1 ci(e2e): skip final cleanup on disposable PR runners (#5260)
## What this PR does

Remove the final sandbox teardown from the two PR E2E workflows. The
runners are documented as disposable VMs, so destroying their local
containers and zpools after testing delays the result without protecting
a later run. Cleanup between preparation attempts and the Makefile
targets for reused hosts stay unchanged.

The job limit stays at 215 minutes. Suite-length budgeting, including
nightly and tag runs, remains separate in #5261. This change relies on
the existing disposable-runner contract; it does not change runner
provisioning.

### Screenshots

Not applicable.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **CI Workflows**
* Removed automatic sandbox and cluster teardown after end-to-end report
uploads.
* Added checks that cluster cleanup runs before environment preparation
is retried.

These changes affect automated test workflows and do not alter
user-facing product features.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-11 12:10:26 +03:00
Aleksei Sviridkin 5a2582e21c test(observability-proxy): preserve JSON seed request bodies
Send JSON seeds with their content type so VictoriaLogs does not consume
log records while parsing form parameters before ingestion.

Fixes: ebb5a9cec4 ("test(observability-proxy): cover the namespace logs scope end to end")
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-10 23:21:56 +03:00
Aleksei Sviridkin 9cd51cac2e ci(e2e): skip final cleanup on disposable PR runners
Destroying runner-local state after a completed suite can push the job
past its deadline even though the runner is discarded. Keep cleanup
between preparation attempts and for reused hosts instead.

Fixes: 186547d835 ("test(e2e): run the merge-gating lanes on Talos containers")
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-10 17:06:15 +03:00
Aleksei Sviridkin b53ca1fede fix(postgres): refuse system databases and never drop them (#5155)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

`databases` accepts `postgres`, `template0` and `template1`, and the
init job treats them like application databases: it comments them
`database managed by helm`, creates `<db>_admin` and `<db>_readonly`,
makes `<db>_admin` the owner of the database and its schemas, and
installs the `trigger_auto_grant` event trigger. When the entry later
leaves values, the delete pass runs `DROP DATABASE` on it, which can
only fail:

- `template1`, `template0`: `cannot drop a template database`;
- `postgres`: `cannot drop the currently open database`, since the job
is connected to it.

The init job then fails on every upgrade, so the HelmRelease sits in
`UpgradeFailed` with `post-upgrade hooks failed`, and putting the entry
back does not help. `template1` also hands the takeover (owner of
`public`, default privileges, trigger) to every database created
afterwards. Once its roles are dropped by hand, the trigger left behind
names a role that no longer exists, so `CREATE SCHEMA` fails in every
new database, even as superuser.

This PR:

- **Refuses** the three names at render with a message that says what to
do. `cozyvalues-gen` has no way to constrain the keys of a
`map[string]Database`, so the refusal sits next to the existing
database-name guard, like the `users.postgres` refusal. The field
description says it too.
- **Never selects them for `DROP DATABASE`.** The delete pass excludes
them whatever values say.
- **Hands a taken-over system database back.** In a new `release system
databases` pass before the delete pass, for each of the three still
commented `database managed by helm`: drop the trigger and
`auto_grant_schema_privileges()`, `REASSIGN OWNED` / `DROP OWNED` for
its two roles in that database, give `public` back to
`pg_database_owner` (PostgreSQL 15+), and clear the comment. The
existing orphaned-roles pass then drops the roles; it never visits
`template1`, hence the per-database step. `template0` accepts no
connection and the job that commented it never got further, so only its
comment is cleared.

**Upgrade note.** A release that declares one of the three names fails
to render until the entry is removed. Removing it is now safe: the next
run releases the database instead of dropping it.

**Testing**

- `tests/system_databases_test.yaml`: 5 of its 6 cases fail on `main`
without the change. `helm unittest packages/apps/postgres`: 215 passed.
- I replayed the scenario against a `postgres:17` container, running the
rendered `init.sh` with `PGUSER=postgres`:
1. The `main` script with `databases: {appdb, template1, postgres}`
reproduces the takeover.
2. The `main` script with only `appdb` then fails with `cannot drop the
currently open database`.
3. This branch's script on that same state exits 0. It releases both
databases (owner `postgres`, no comment, no event trigger, `public`
owned by `pg_database_owner`) and drops `postgres_admin`,
`postgres_readonly`, `template1_admin` and `template1_readonly`. A
database created afterwards gets no trigger, and `CREATE SCHEMA`
succeeds in it.
  4. A second run is a no-op.
5. A `template0` commented and given roles by hand is released the same
way.

Open PR #5091 touches the same script (stable database order); the hunks
do not overlap.

Backport: the automatic cherry-pick onto `release-1.6` conflicts in
`init-script.yaml` and the generated cozyrd, so the backport to
`release-1.6` is done by hand in #5157 (same SQL, replayed against that
branch's script). `release-1.7` keeps `kind/backport`.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change (walked the
map: a template guard, the init script, a test suite and one field
description regenerated into README, schema, Go types and cozyrd; no
field, default, enum or name change)
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(postgres): databases now refuses the PostgreSQL system databases postgres, template0 and template1, and the init job never runs DROP DATABASE on them. A system database a previous version took over is handed back to the superuser on the next upgrade (event trigger removed, ownership and public schema restored, its <db>_admin and <db>_readonly roles dropped). A release that declares one of them must remove the entry before upgrading.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Behavior Changes**
* PostgreSQL system databases (`postgres`, `template0`, and `template1`)
are refused as managed application databases.
* Previously managed system databases are released during cleanup and
excluded from database deletion. Cleanup restores eligible ownership and
security settings and clears system database comments.
* **Documentation**
* Updated PostgreSQL configuration guidance to clarify that these system
database names are refused. Names that merely contain these strings
remain accepted.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-09 21:09:23 +03:00
ZverGuy a2eb175139 test(dashboard): start the pass-through nginx on fresh ports
The second nginx of the not-installed test reused the stand-in port
of the first one right after killing it, behind a fixed sleep, so a
slow exit could leave the port taken. It now waits for the first nginx
to exit and takes a new port. The file header also names the fourth
property it now checks.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:07 +07:00
ZverGuy f2944c572d fix(dashboard): answer not-installed when the observability proxy is absent
The proxy can be left out of an installation, and then the console
nginx answered the observability roots with its own 502. The console
read that as an unreachable store: three errors, a Retry button and a
live tail reconnecting forever, instead of a notice that the feature
is not available on this cluster.

nginx now turns the 502 and 504 it generates itself, when it cannot
resolve or reach the proxy, into a 404 Kubernetes Status, which the
console already treats as not installed. Upstream errors are not
intercepted, so the proxy's own 502 and 504 for a store still reach
the console unchanged. The bats test runs the extracted locations in a
real nginx when one is installed.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:06 +07:00
ZverGuy 3e907e6f9a test(observability-proxy): declare the shell of the e2e helper library
The library is sourced by sh scripts and has no shebang, so shellcheck
could not tell which shell to check it as and refused the file.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:06 +07:00
ZverGuy f775948d2c test(observability-proxy): replay the early line into the e2e tail
The background tail goes through the console route and the proxy's
reviews before it subscribes, so the early ingest can land first, and
a tail without start_offset returns only lines written after it
starts. The tail now replays the last minute, which still leaves the
hold and the late-line check meaningful.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:06 +07:00
Alexey Artamonov 0f336e99c5 test(backups): drop the comment restating what accepted() returns
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-09 16:39:47 +03:00
Alexey Artamonov 1291e616c7 fix(clickhouse): give every backup run an archive of its own
An archive was named from the application and the BackupJob's start
time, which has one-second resolution. Two runs of one application
started in the same second, as two Plans on the same schedule are,
wrote the same archive. The strategy takes a finished upload of its
archive as an earlier attempt of its own run, so the later run could
succeed on the other run's upload and record an archive missing any
table created since.

The BackupJob's UID now ends the name. It goes after the timestamp so
names keep sorting chronologically, and the restore of a Backup that
records no archive, which accepts only a bare timestamp after the
release prefix, never picks one of these.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-09 16:39:47 +03:00
Alexey Artamonov d530ca8cf5 fix(backups): keep a ClickHouse archive an earlier attempt finished
A backup Pod that loses sight of its upload, by dying or by losing six
polls in a row, is retried while the sidecar finishes that upload. The
retry found it in the action log as a success, took the sidecar for
idle and asked for create_remote with resume again. clickhouse-backup
then rebuilds the local backup and uploads it once more over the
archive already in the bucket, which is left broken if that second
upload dies halfway.

The archive name belongs to one BackupJob, so a success row of exactly
this upload can only be an earlier attempt's, and the retry now takes
it as its result. An error or cancel row still leads to a resumed
upload, and a restore still runs again, since a restore is not named
per RestoreJob.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-09 16:39:46 +03:00
Alexey Artamonov 89f8cc02ee Merge branch 'main' into fix/clickhouse-idle-before-backup
main now names the ClickHouse archive in the controller and restores
the archive a Backup recorded, in the same strategy script this branch
gates on an idle sidecar. The merged script keeps both: the archive
name and its restore selection from main, the idle gate, retried polls
and lost-operation check from this branch.

A retried backup is taken over through the same idle gate as a restore,
matched on the whole command line the sidecar logs. With resume
requested that line reads "create_remote --resume <name>", so the gate
matches that form; matching on the name alone would also take another
operation's row ending in the same name for this one.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-09 16:39:43 +03:00
Alexey Artamonov 72a94ee426 test(clickhouse): give the round-trip's preconditions room for their deletes
The step's ceiling SIGKILLs the shell and reports only "signal: killed",
while a kubectl --timeout that fires first names the object that would
not go. The deletes it runs now wait up to 270s in all, past the 3m
ceiling, so a stuck delete late in the list would be reported by the
ceiling instead of by kubectl.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-09 14:26:02 +03:00
Alexey Artamonov 0dc5a54731 test(clickhouse): remove the round-trip's TLS objects a killed run left
Chainsaw deletes an applied object only when its own apply created
it; one left behind by a run killed past its cleanup is patched and
kept. The Certificate would then outlive every later run and keep
re-issuing its Secret after each step cleanup, and a probe Pod whose
sleep had ended never turns Ready, failing the next run's assert.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-09 13:49:42 +03:00
Mattia Eleuteri 4f35af6694 fix(monitoring-agents): let the release uninstall when the VictoriaMetrics operator is gone first
Disabling the Monitoring Agents addon of a tenant cluster removes this
release, the VictoriaMetrics operator and the VPA CRDs together, and
Flux deletes them in parallel without honouring dependsOn. When the
operator goes first, nothing clears apps.victoriametrics.com/finalizer
on the VMAgent and the other VictoriaMetrics objects this chart created,
so they stay Terminating and the release sits in UninstallFailed for
good.

Clearing those finalizers has to happen inside the uninstall and must
never be what blocks it, and a live operator still has to get the first
chance to finalize its objects. Only objects of this release may be
touched, and release names are unique only within a namespace, so the
release name alone does not identify them. kubectl loads in-cluster
credentials only while the client config is otherwise default, which a
--request-timeout breaks. A VPA recorded in the release manifest while
its CRD is not served cannot be mapped at uninstall either.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-11-09 11:18:49 +01:00
Mattia Eleuteri f003851338 fix(postgres): refuse system databases and never drop them
postgres, template0 and template1 were accepted in databases and handled
like an application database. The init job made <db>_admin their owner,
installed the auto-grant event trigger in them and, once the entry left
values, ran DROP DATABASE on them. That DROP can only fail (template
databases cannot be dropped, and the job is connected to postgres), so
every later upgrade failed in the init job and the release stayed in
UpgradeFailed even after the entry was put back. template1 also passed
the takeover on to every database created from it afterwards, where the
trigger, naming a role about to be dropped, makes CREATE SCHEMA fail.

A release a previous version let take one of them over has to be
repaired without dropping it, and without touching what this chart did
not create: a role or owner that merely carries an expected name may
belong to a tenant or an administrator, and a role comment can be set
by any CREATEROLE holder, so only the cozystack:managed mark is trusted.
What the marked roles own goes to cozystack:orphaned rather than to the
superuser, since a tenant's SECURITY DEFINER function would otherwise
run with superuser rights. A failed lookup of the databases cloned from
template1 has to stop the run before template1 stops being selected, or
those clones would never be repaired.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-11-09 11:18:27 +01:00
Aleksei Sviridkin da7f2c2f50 fix(tests): keep two unit tests green after the release image restamp (#5180)
## What this PR does

Same change as #5179, on `main`. Both tests pass here today, because
`main` still carries the pre-release image pins; they fail on every
release line as soon as the first rc is cut, since the `Prepare release`
commit rewrites `packages/*/values.yaml` and two unit tests read exactly
the fields it rewrites. On `release-1.7` this has kept `Unit &
controller tests` red on every PR since v1.7.0-rc.3
([job](https://github.com/cozystack/cozystack/actions/runs/41162009892/job/124421449910)
on #5172), and `release-1.8` would repeat it.

| test | reads | before the release build | after (release-1.7 today) |
|---|---|---|---|
| `hack/cni-plugins-staging-contract.bats` "the kube-ovn release the CNI
plugin pin was checked against is the one vendored" |
`.global.images.kubeovn.tag` in `packages/system/kubeovn/values.yaml` |
`v1.15.26@sha256:…` | `v1.7.0@sha256:…` |
| `packages/system/monitoring/tests/vtraces_collector_test.yaml` "pins
the collector image through the cozy-lib.image helper" |
`tracingCollector.image` |
`docker.io/otel/opentelemetry-collector-contrib:0.136.0@sha256:…` |
`ghcr.io/cozystack/cozystack/otelcol-tracing:v0.136.0-cozystack.0@sha256:…`
|

Changes:

- The bats test takes the vendored kube-ovn release from `appVersion` in
`packages/system/kubeovn/charts/kube-ovn/Chart.yaml`. That is the value
`packages/system/kubeovn/Makefile` builds the image from, and a release
build leaves it alone. The hint printed on a mismatch names the new
file.
- The helm unittest case no longer names the image at all. It sets
`_cluster.images-registry` and asserts that the registry lands in front
of the rendered image and that the image stays digest-pinned (the
`grafana_upstream_image_plugins_test.yaml` pattern). Removing the
`cozy-lib.image` call from the template makes it fail; the old regex did
not.

Same single commit as #5179; the only difference is the unrelated `DOC=`
line next to the edit, kept as `main` has it.

Verified locally on this branch: `bats
hack/cni-plugins-staging-contract.bats` 10/10, `helm unittest
packages/system/monitoring -f tests/vtraces_collector_test.yaml` 28/28.

Refs #5178 (the release-1.7 report), #5179, #5172.

### Downstream repositories

- [x] No downstream repository is affected by this change

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Tests**
* Version validation now compares the checked version with the chart’s
application version and reports missing or mismatched versions with the
chart path.
* Monitoring image checks now verify use of the configured mirror
registry and require a 64-character lowercase SHA-256 digest, without
requiring a specific image name.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-09 12:19:21 +03:00
Aleksei Sviridkin 9e8e10c307 fix(backups): re-render the default backup objects when the S3 endpoint changes (#5230)
## What this PR does

Refs #5205.

The S3 endpoint the platform backups use is copied twice, and neither
copy is refreshed after `publishing.host` changes. The COSI sidecar
writes `BucketInfo` once per `BucketAccess`, so the credentials Secret
`bucket-cozy-backups-system-credentials` rendered from it keeps the old
endpoint. The Velero BSL and the default Strategy CRs keep whatever
endpoint the last render of `backupstrategy-controller` saw. The default
objects gate only acted on missing objects, so backups kept going to the
old host and nothing reported it.

The gate now also forces a render when a copy is stale:

- the bucket's `<bucket>-system` release, when the credentials Secret's
endpoint differs from the COSI Secret it is rendered from;
- `backupstrategy-controller`, when the endpoint an object was rendered
with differs from the credentials Secret's. The chart stamps the
rendered endpoint on the BSL and on every default Strategy as
`backups.cozystack.io/endpoint`, because the endpoint itself sits at a
different path in every kind. Objects without the annotation count as
stale until a render stamps it; `MinForceInterval` bounds the forces.

Stale credentials are forced before the BackupClass and the routed
objects are read, so an absent BackupClass or an unreadable Strategy
still fails the check but does not keep the old endpoint in the
credentials Secret. A `BucketInfo` without an endpoint is not compared,
since rendering from it would write an empty endpoint or fail the bucket
release. Repairing missing objects keeps working while the credentials
comparison is stale or fails, and a `systemSecretName` that is not
derived from a COSI Secret (no `-credentials` suffix, or the raw COSI
Secret itself) is not compared at all. RBAC is unchanged: the controller
already holds cluster-wide `get` on Secrets. No new metric: nothing
would consume it.

What this does not do: the COSI Secret still holds the endpoint the
`BucketAccess` was granted with, so after a host change the endpoint
follows only once an operator deletes the `BucketAccess` and forces
`bucket-cozy-backups` to recreate it (this issues new S3 keys). From
there the gate replaces the remaining manual steps of the recovery
described in the issue. Automatic re-grant is left for a separate issue,
and the operator runbook goes to the website, hence `Refs` rather than
`Fixes`.

Known limitation: the credentials comparison runs only when
`backupStorage.systemSecretName` is
`bucket-<bucketName>-system-credentials`, the Secret the forced
`bucket-<bucketName>-system` release renders. If an operator sets
`bucketName` and `systemSecretName` inconsistently, the comparison is
switched off rather than forcing a release that never rewrites the
compared Secret, so a stale endpoint in that Secret is not repaired.
Repairing a missing credentials Secret still forces the release derived
from `bucketName`, as before this PR.

Tested with unit tests for the gate (including the comparison
normalisation shared with the chart helpers), helm-unittest for the
annotation and the wiring, and a bats guard that every rendered Strategy
CR and the BSL carry the annotation. On a live cluster, a server-side
render of the chart produced exactly the annotation the gate derives
from the real credentials Secret, so the steady state does not force
anything.

### Screenshots

Not a UI change.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: an operator runbook for changing `publishing.host` with
platform backups enabled (re-grant the `BucketAccess`, then let the gate
converge; restart ClickHouse pods that read the credentials from
environment variables). Not filed yet, pending approval of the text.
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(backups): the backup controller now re-renders the Velero BSL, the default backup strategies and the platform bucket's credentials Secret when their S3 endpoint is stale, so after the platform bucket's BucketAccess is re-granted for a new publishing.host, backups move to the new endpoint without forcing releases by hand.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Backup strategies and storage locations now record their storage
endpoint, enabling detection and repair of outdated endpoint settings
after bucket access changes.
* Bucket credentials and backup objects are refreshed independently when
their endpoints are missing or out of date.
* **Documentation**
* Clarified when endpoint changes trigger updates and how suspended
releases are handled.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-09 12:02:42 +03:00
ZverGuy 8784863632 test(observability-proxy): wait for every seeded line before checking scope
The suite checked that B's line and the account-7 line are absent from
A's answers but only waited for A's line to become searchable, so an
absence could come from a line the store had not indexed yet. The step
budgets in the comments now count the calls the steps make.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-09 15:48:57 +07:00
ZverGuy ebb5a9cec4 test(observability-proxy): cover the namespace logs scope end to end
Unit tests mock the API server and the store, so nothing shows that the
console's nginx, the proxy's TokenReview and SubjectAccessReview and a
real VictoriaLogs agree. The suite drives a viewer of one tenant through
the console against a seeded store: it reads its own namespace, is
refused a sibling's, cannot widen the scope with client filters, an
AccountID header or dot segments, and keeps a tail open past a minute.

The root store is off in e2e, so the suite runs its own VictoriaLogs
under the root store's Service name rather than waiting on Fluent Bit.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-09 15:48:57 +07:00
ZverGuy d452b30e42 test(dashboard): name the templated nginx config in the buffer check
The console nginx config now ships as an entrypoint template, and a
failure message naming the old conf.d path sends the reader to a file
the Containerfile no longer writes.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-09 15:48:57 +07:00