Commit Graph
2955 Commits
Author SHA1 Message Date
Mattia Eleuteri f7ee616f6c fix(linstor): rebase the snapshot restore patch on linstor-csi v1.13.2
The import block of pkg/client/linstor.go changed in v1.13.2, so the
patch no longer applied and the linstor-csi image failed to build.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-11-13 17:27:48 +01:00
Aleksei Sviridkin ba5948ed29 Merge branch 'main' into fix/linstor-csi-snapshot-restore-node
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 18:54:57 +03:00
Aleksei Sviridkin f61cfdf0b8 feat(bucket)!: add publicRead to serve a bucket without credentials (#5252)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, security (a fix that comes through a security
advisory), docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Fixes merged to `main` reach the release lines the backport batch runs
for automatically, so do not open backport PRs by hand:
https://github.com/cozystack/cozystack/blob/main/docs/release/backports.md
-->

## What this PR does

Adds `publicRead` to the Bucket application. A bucket cannot serve its
objects without a signature today: every BucketAccess issues a key, and
those keys grant neither `WriteAcp` nor `PutBucketPolicy`, so `aws s3api
put-bucket-acl --acl public-read` is refused and a tenant has no way to
serve static assets (images, downloads) straight from a bucket. The only
workaround is an operator running `weed shell` `s3.anonymous.set` by
hand, which nothing reconciles and which outlives the bucket.

With `publicRead: true` the chart renders one more BucketAccess,
`<bucket>-public-read`, on a new `-public` BucketAccessClass whose
`accessPolicy` is `public-read`. The class is added in the three places
that define access classes: the Client topology, the per-storage-pool
classes, and the vendored chart (through its existing
`cosi-bucket-class-lock-readonly.patch`). The COSI driver handles the
policy by adding `Read:<bucket>` to the gateway's `anonymous` identity
and, on revoke, removing only that action, so turning the flag off or
deleting the bucket closes it again without touching other public
buckets. Anonymous callers can `GET`/`HEAD` objects at
`<endpoint>/<bucket>/<key>`; listing and writes stay refused.

A user named `public-read` would own the same BucketAccess name, so that
combination fails to render with an explicit message rather than having
two owners fight over one object.

The seaweedfs pool name check now also reserves `public` and the
`-public` suffix, next to the existing `-lock` and `-readonly`: a pool
`<x>-public` would render a readwrite BucketAccessClass with the same
name as pool `<x>`'s public-read class, and a pool `public` would take
the instance's own `-public` class. This is a breaking change for an
instance that already defines a pool with such a name: its seaweedfs
release stops rendering with the pool name error. Pool keys are part of
the volume server and PVC names, so renaming such a pool moves its data
to new volumes; that migration is the operator's to plan before
upgrading.

**Known limitation until the next driver release.** On v0.5.0, revoking
public read (turning `publicRead` off or deleting the bucket) can leave
the bucket readable anonymously if an S3 gateway migrates
`identity.json` while the revoke is in flight: the migrated
`identities/anonymous.json` keeps `Read:<bucket>`, and SeaweedFS loads
it over `identity.json` without ever repairing it. The fix is
seaweedfs/seaweedfs-cosi-driver#22 (the revoke always writes
`anonymous.json`, after the legacy file). The driver release carrying it
will be pinned here if it lands before this PR merges, otherwise in a
follow-up. Until then, an operator can check `s3.anonymous.list` in
`weed shell` after a revoke and clear a leftover with `s3.anonymous.set
-bucket <bucket> -access none`.

**Driver: pins the upstream seaweedfs-cosi-driver v0.5.0.** The
`public-read` policy merged upstream in
seaweedfs/seaweedfs-cosi-driver#21 and v0.5.0 is tagged on that merge,
so this PR moves the image pin from v0.4.0 (#5284) to v0.5.0 in
`packages/system/seaweedfs/values.yaml`, the client-mode `.tag` file and
`cosi_image_test.yaml`. An earlier revision of this branch built the
driver in-tree while no release carried the policy; that build is gone,
and the intermediate commits that added it are undone by the merge of
`main`, so squashing leaves only the feature and the pin.

v0.5.0 differs from v0.4.0 only by the `public-read` policy
(seaweedfs/seaweedfs-cosi-driver#21) and its README note. Upstream
documents one `public-read` BucketAccess per bucket as the supported
shape, which is what the chart renders.

Tested: new helm-unittest suites for the bucket app (default off, class
name with and without storage pool, locked bucket, coexistence with
users, the name collision), the seaweedfs extra chart (Client and
storage-pool classes) and the vendored chart; they fail when the class
suffix or the policy is removed. `helm unittest` passes on
`packages/apps/bucket`, `packages/extra/seaweedfs`,
`packages/system/seaweedfs` and `packages/system/bucket`, as do
`test-guard-fail-closed` and `test-image-version`;
`cosi_image_test.yaml` pins
`ghcr.io/seaweedfs/seaweedfs-cosi-driver:v0.5.0`, which is published for
amd64 and arm64. The anonymous-read mechanism itself was checked by hand
on SeaweedFS 4.45 (`weed shell` `s3.anonymous.set -access Read`):
anonymous `GET` returns 200 on both gateways,
`ListObjects`/`ListObjectsV2` and `PUT` return 403, and the identity
survives grants and revokes of regular BucketAccesses. The end-to-end
path through the new driver has not run on a cluster yet.

### Screenshots

No UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: cozystack/terraform-provider-cozystack#80
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Walked the trigger map against the diff: the only hit is the new field
in `packages/apps/bucket/values.schema.json` (and
`api/apps/v1alpha1/bucket/types.go`), which the Terraform provider
models by hand. Follow-up filed as
cozystack/terraform-provider-cozystack#80. The website's bucket page is
hand-written and could mention the flag, but no listed trigger covers an
added field, so it is left unticked.

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
feat(bucket): add `publicRead` to serve a bucket's objects without credentials at `<endpoint>/<bucket>/<key>`. Anonymous callers can read objects but cannot list the bucket or write to it. The COSI driver is bumped to seaweedfs-cosi-driver v0.5.0, which adds the `public-read` access policy. The seaweedfs pool names `public` and `*-public` are now reserved: an instance that already defines such a pool fails to render until the pool is renamed, which moves its data to new volumes.
```





<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Bucket configurations can enable anonymous reads of objects through
the bucket endpoint. Anonymous access does not allow listing buckets or
writing objects.
* SeaweedFS provides public-read access classes across supported bucket
configurations, alongside read-only and read-write access.
* Added a retained bucket class with object locking and a 365-day
compliance retention period.
* **Documentation**
* Added guidance for the `publicRead` setting, which is disabled by
default.
* **Updates**
* Updated the SeaweedFS COSI driver to v0.5.0. Pool names `public` and
names ending in `-public` are reserved.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 18:47:57 +03:00
Aleksei Sviridkin 29e8522f4b fix(cozystack-basics): refuse a direct delete of a tenant namespace (#5011)
## What this PR does

Fixes #5002. A DELETE of a tenant namespace is refused unless it comes
from the identities that remove it on the normal path, and the denial
says to delete the Tenant instead.

The new ValidatingAdmissionPolicy in cozystack-basics matches DELETE of
core namespaces, and its binding selects namespaces labelled
`tenant.cozystack.io/tenant-root`. The tenant chart puts that label on
every tenant namespace below tenant-root (`tenant.ancestorTenantLabels`,
and migration 49 backfilled older ones). tenant-root itself has no such
label and stays with `cozystack-no-delete-guardrail`. The `tenant-*`
probe namespaces of the e2e suites carry only
`namespace.cozystack.io/host` and are not selected either.

Two groups pass. `system:serviceaccounts:cozy-fluxcd`, because
helm-controller and its shards run as `flux` there, and a Tenant
uninstall is helm-controller deleting the namespace after the pre-delete
hook. `system:serviceaccounts:kube-system`, for the garbage collector
following the ownerReference of a child namespace once its parent is
gone. The namespace controller never sends a DELETE for a namespace, it
updates `finalize`, so it is not affected
(`pkg/controller/namespace/deletion` and
`pkg/registry/core/namespace/storage` at v1.33.12 and v1.34.1).
`system:masters` is not exempt, the same as with the no-delete
guardrail.

This is the denial kube-apiserver 1.35 returns.

```
namespaces "tenant-foo" is forbidden: ValidatingAdmissionPolicy 'cozystack-tenant-namespace-delete-policy' with binding 'cozystack-tenant-namespace-delete-policy' denied request: Namespace tenant-foo belongs to the Tenant foo in tenant-root. Delete the Tenant instead with kubectl delete tenants.apps.cozystack.io foo --namespace tenant-root. Deleting the namespace directly can remove the tenant etcd before the Kubernetes clusters that use it and leave the namespace stuck in Terminating. If the Tenant is already gone, remove the label first with kubectl label namespace tenant-foo tenant.cozystack.io/tenant-root-
```

There are a few limits.

- The bypass is removing the label, like with the guardrail. That also
takes the namespace out of the tenant-root egress policy and the
dashboard tree, which only matters for a namespace that is about to go.
- Migration 49 labelled every `tenant-*` namespace that existed when it
ran, so a non-tenant namespace with such a name is refused too and needs
the same label removal.
- Because the garbage collector is exempt, a parent namespace whose
label was removed and that is deleted with foreground propagation still
cascades into its children directly.
- A DELETE of a namespace that is already Terminating is answered before
admission, so the policy does not see it. It changes nothing either.
- A cluster that made helm-controller impersonate another account
(`--default-service-account`, or `spec.serviceAccountName` on Tenant
HelmReleases) would need that account allowed. Cozystack sets neither.
- It does not change how Kamaji cleans up once the order is broken,
which is the second idea in #5002.

The helper rename in `external_tls_policy_test.go` (`loadPolicy(t,
path)` and `compilePolicy`) is byte for byte the one in #2622, so the
two merge in either order.

Checked locally.

- `go test ./pkg/registry/apps/application/ -run
'TestTenantNamespaceDeletePolicy|TestDatabaseExternalTLSPolicy' -count=1
-v` passes 4 tests, the new `TestTenantNamespaceDeletePolicy_Decisions`
with 11 cases. `go test ./pkg/registry/... ./pkg/config/...
./pkg/cmd/server/...` passes in all 11 packages that have tests.
- `helm unittest` in `packages/system/cozystack-basics` with
helm-unittest 1.1.2 in docker passes 12 suites and 70 tests, against 11
and 66 on main. The new tests on main without the template fail, both Go
tests and 4 helm tests in 2 suites.
- 12 mutations of the template each fail a test. They exempt
`system:masters`, drop either exempt group, select on the no-delete
label, add UPDATE, always name tenant-root as the parent, drop the
capability gate, label tenant-root, switch to Warn, switch failurePolicy
to Ignore, drop the bypass from the message and drop the objectSelector.
- The rendered policy, the guardrail and namespaces rendered from the
tenant chart were applied to kube-apiserver v1.33.0, v1.35.0 and v1.36.2
from setup-envtest, 16 checks on each. Real and dry-run deletes by a
cluster admin, by an OIDC user bound to cluster-admin and by a
cozy-system and a tenant ServiceAccount are refused with 403. The flux
and garbage collector identities get their delete accepted. tenant-root
gets the guardrail's denial, and with the guardrail binding removed a
dry-run delete of it goes through, so this policy does not select it. A
`tenant-*` namespace without the label and a tenant namespace after its
label is removed are accepted for deletion. With this binding removed, a
dry-run delete by the cluster admin goes through. That harness is not
part of the PR.
- `hack/bats-no-negated-assert.bats`, `hack/bats-no-exit-trap.bats`,
`hack/bats-runner-coverage.bats`, `hack/helmrelease-readiness.bats` and
`hack/route-hostname-policy-version-coverage.bats` pass under
`hack/cozytest.sh`.

The new e2e case in `hack/e2e-install-cozystack.bats` creates an empty
Tenant, checks that a server dry-run delete of its namespace is refused
with the message, deletes the Tenant and waits for the namespace to go.
It was not run here, there is no Cozystack cluster at hand. Its kubectl
steps were run against the envtest apiserver 1.35, the selector and
field selector read, the refused dry-run with the exact strings it greps
for, and `kubectl wait --for=delete` on a namespace that is gone. The
Chainsaw suites that delete Tenants, the gateway inheritance ones among
them, go through the uninstall path as well.

The release-1.6 backport is #5012.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The trigger map was walked against the diff. It adds a policy and tests
to cozystack-basics and an e2e case, and no entry covers that. The
website docs do not tell users to delete a tenant namespace directly,
and terraform-provider-cozystack deletes Tenants through the apps API.

### Release note

```release-note
fix(cozystack-basics): refuse a direct DELETE of a tenant namespace unless it comes from helm-controller or the garbage collector, and point to deleting the Tenant instead, so a tenant namespace deleted by hand no longer gets stuck Terminating on Kamaji finalizers
```
2026-11-13 18:25:04 +03:00
Aleksei Sviridkin fa09cdc184 fix(keycloak, foundationdb): apply operator CRD updates on upgrade (#4946)
## What this PR does

The generated HelmReleases leave `spec.upgrade.crds` unset unless the
PackageSource sets `upgradeCRDs`, so helm-controller never replaces a
CRD that a chart ships in `crds/`. keycloak-operator and
foundationdb-operator ship their CRDs that way and set no policy. On a
v1.6.4 cluster that came through earlier releases, the CRDs of both
operators were older than the ones in the v1.6.4 charts. The v1.25.0
ClusterKeycloakRealm schema has no `spec.sessions`, `spec.login` or
`spec.organizationsEnabled`, so the apiserver dropped the session
settings keycloak-configure sets and the operator never applied them to
the realm. Keycloak also rejected the dashboard KeycloakClient with HTTP
400 on that cluster, and replacing the 17 CRDs from the charts and
restarting keycloak-operator cleared it.

This opts both PackageSources into `upgradeCRDs: CreateReplace`, the
policy mongodb-operator, redis-operator and velero already use, and pins
it with a platform test for each. The keycloak-operator change is not
purely additive. v1.32.0 drops the `realm`, `targetRealm` and
`keycloakOwner` fields that v1.25.0 marked deprecated and requires
`realmRef` and `keycloakRef` instead. Every Keycloak object the platform
renders already uses the refs, and each of the 14 CRDs keeps its single
version, so the replace loses nothing the platform relies on and drops
no stored version. A hand-made object that still uses a deprecated field
would lose it on its next write, and that write needs the ref. The
foundationdb-operator CRDs only add fields from v2.13.0 to v2.30.0, plus
a pattern on `perpetual_storage_wiggle_locality`, which the platform
does not set.

helm-controller applies the policy only when it upgrades a release, and
it upgrades only when the release artifact or its values change. A
PackageSource edit changes neither. The PR therefore also gives the
keycloak-operator package a helm-unittest suite, which pins the realm
fields keycloak-configure sets against the ClusterKeycloakRealm CRD the
chart ships and changes the keycloak-operator artifact, so the CRDs are
replaced on the upgrade that brings this change even where nothing else
in the package changes, as on a backport to release-1.6. The platform
release writes the new PackageSource in the same apply that moves
`cozystack-packages` to the new revision, so the policy should reach the
HelmRelease before the new keycloak-operator artifact does. The
foundationdb-operator package does not change here, so its CRDs are
replaced at the next change to that package, and main already carries
one since v1.6.5.

Objects written while the CRDs were stale get the pruned fields back
only when their chart applies them again. For the cozy realm that is the
next keycloak-configure upgrade. On main it comes with the same platform
upgrade, since keycloak-configure changed after v1.6.5, and its
HelmRelease depends on the keycloak-operator one through the oidc
variant. On a branch where keycloak-configure does not change, the realm
keeps its current session limits until it does or until that release is
forced to upgrade.

#4901 adds the same `upgradeCRDs: CreateReplace` to the
foundationdb-operator PackageSource as part of the FoundationDB backup
work, and it changes the foundationdb-operator package as well. The
foundationdb commit here is separate from the keycloak one and does not
depend on it, so it can be dropped if #4901 lands first.

The other packages that ship CRDs only in `crds/` and set no policy are
bootbox, clickhouse-operator, clustersecret-operator,
cozystack-scheduler, external-dns, grafana-operator, kafka-operator and
snapshot-controller. None of their CRDs changed since v1.0.0, or since
the package was added, apart from description text in external-dns, so
no cluster is behind on them today. kafka-operator gets CreateReplace
with the Strimzi move in #4900. gpu-operator replaces its CRDs with its
own pre-upgrade Job, and the remaining packages render their CRDs from
templates or leave them to an operator. Each of the eight will need the
same review and opt-in the next time its CRDs change.

`make -C packages/core/platform test` passes with 58 suites and 287
tests, and `make -C packages/system/keycloak-operator test` passes. With
either PackageSource reverted to main, its suite fails with `unknown
path spec.variants[0].components[0].install.upgradeCRDs`. With the
v1.25.0 ClusterKeycloakRealm CRD in the chart, the keycloak-operator
suite fails all four assertions. The full `hack/helm-unit-tests.sh`
sweep passes.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Two PackageSources under `packages/core/platform/sources/` gain an
install option, three test suites are new, and
`packages/system/keycloak-operator/Makefile` gains the same `test`
target other packages have. Platform `values.yaml`, variants, bundles,
CRDs, `hack/` and the installer are untouched. The website's
`development.md` does not describe per-package test targets, so the
Makefile change needs nothing there, and no website page documents a
manual CRD replace for these operators.

### Release note

```release-note
fix(keycloak, foundationdb): keycloak-operator and foundationdb-operator now replace their CRDs on upgrade. Clusters upgraded from before v1.2.0 kept the v1.25.0 keycloak-operator CRDs, under which the apiserver dropped the Keycloak realm session settings and Keycloak could reject the dashboard client, and clusters upgraded from before v1.6.0 kept the v2.13.0 foundationdb-operator CRDs.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Existing FoundationDB and Keycloak installations can now upgrade their
custom resource definitions with replacement behavior, including updates
to schemas and removal of deprecated fields.
* **Tests**
* Added checks for the operator upgrade settings and Keycloak CRD schema
fields.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 18:24:45 +03:00
Aleksei Sviridkin 51282a1243 fix(piraeus-operator): keep the LINSTOR error-report alerts quiet when the controller pod moves (#4963)
## What this PR does

`linstorSatelliteErrorRate` fires for every node that has any old
LINSTOR error report whenever the LINSTOR controller pod moves to
another node, and it keeps firing for 15 minutes.
`linstor_error_reports_count` is exported by the controller, so its
series carry the controller pod's `instance` and `controller_node`, and
a moved pod starts new series that hold the existing counts.
VictoriaMetrics'
[`increase()`](https://docs.victoriametrics.com/victoriametrics/metricsql/#increase)
starts from the last sample before the window, and when a series has
none,
[`rollupDelta`](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/v1.136.0/app/vmselect/promql/rollup.go#L1859-L1901)
counts it from zero whenever its first value is below ten times the next
change plus one. That is the case the docs describe as `increase`
ignoring "the first value in a series if it is too big" under
[`increase_pure`](https://docs.victoriametrics.com/victoriametrics/metricsql/#increase_pure),
and report counts are never that big, so every new series counts all of
its reports as new. The `up` guard from #2265 matches `on(instance,
job)`, and the new controller is up, so it lets them through.

Seen on a v1.6.5 cluster with VictoriaMetrics v1.136.0. During the
upgrade the controller pod moved to another node, and the alert fired
for all three nodes with values 1, 1 and 2 for 15 minutes, while
`linstor error-reports list --since 1h` was empty and the four reports
that exist were weeks old. The day before, it fired the same way for
about 15 minutes when only the `cluster` external label of vmagent
changed.

A satellite coming back to the metric does the same. In LINSTOR v1.33.3
the controller asks every satellite for its reports on each scrape and
leaves out any satellite that is not connected
([`prepareErrRequestApi`](https://github.com/LINBIT/linstor-server/blob/v1.33.3/controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlErrorListApiCallHandler.java#L240-L259)),
so after a node reboot or any other disconnect longer than five minutes
the satellite's old count is counted again.

Both rules now take the count per `hostname`, hold the last value for an
hour with `last_over_time`, and run `increase()` over that as a
one-minute
[subquery](https://docs.victoriametrics.com/victoriametrics/metricsql/#subqueries),
whose inner points VictoriaMetrics aligns to that step. Keyed on
`hostname`, a moved controller or a changed label continues the same
series, and the hold bridges a disconnect or a slow controller start of
up to an hour. The `up` guard matches `on()`, since the aggregated
result has no `instance` any more. A new report still fires with the
same value and stays for 15 minutes, at most a minute later than before
because of the one-minute step. The alerts now carry `hostname` and
`severity`, and no longer the controller's `instance`, `job` and
`controller_node`, the `module`, or the external labels of the scrape.

`linstorControllerErrorRate` gets the same form. Its `hostname` is the
controller pod name and the piraeus controller keeps its reports in an
[`emptyDir`](https://github.com/piraeusdatastore/piraeus-operator/blob/v2.10.2/pkg/resources/cluster/controller/controller-deployment.yaml#L111-L112),
so a new pod starts with no reports and a move alone does not trip it,
but a label change does whenever the controller has reports.

A gap longer than the hour is not covered. After it the series starts
over, and as before the `up` guard holds the alert back only if the
controller target recorded failed scrapes during the gap.

I checked it against the stored metrics of the cluster above with
read-only queries to vmselect. Over the upgrade the current expression
returns the three nodes from 15:45:30 to 16:00:00 UTC at a 30-second
step, the same interval as vmalert's `ALERTS` series, and over the label
change from 20:27:30 to 20:42:00. The new expression returns nothing
over both windows, and without `> 0` it returns each of the three nodes
with 0 at every step, 88 and 121 points. The same storage has six gaps
of 8 to 113 minutes over one night with no failed scrape recorded in
them. Replaying vmalert's instant query of the `increase()` part every
30 seconds in the 154 evaluations after those gaps, the current
expression returns at least one node in 99, the `hostname` key without
the hold in 84, and the new expression in 30, all of them after the
113-minute gap. Retention there does not reach back to a real new
report, so the firing side is covered by the tests.

`hack/linstor-error-report-alerts.bats` replays both rules through
`vmalert-tool` v1.136.0, pinned by digest, which is the VictoriaMetrics
release victoria-metrics-operator v0.68.4 [deploys by
default](https://github.com/VictoriaMetrics/operator/blob/v0.68.4/internal/config/config.go#L38),
with 30-second samples and 15-second evaluations like the platform's
vmagent and vmalert. The suite needs docker, like the migration suites,
and only the image pull touches the network.
`hack/testdata/linstor-error-report-alerts/unittest.yaml` has 20 cases
in five groups, which cover a controller move followed by a new report,
a changed external label for both rules, one satellite away for 40
minutes and one that comes back with a new report, a new report and a
first report on a node for both rules, and a controller that cannot be
scraped for 70 minutes. Against the rule on main 13 cases fail, 9 of
them because old reports fire again after the move, the label change or
the disconnect, and the other 4 on the label set alone, since the
current alerts carry the controller's labels. Each of seven mutations of
the new expressions turns at least one case red, namely no hold, a
15-minute hold, the guard on `instance`, no guard, the key on `hostname`
and `instance`, the key on `hostname` and `cluster`, and the controller
rule as on main. With Bats 1.14.0 at the commit CI pins, the new suite
with `alert-severity-contract`, `bats-strict-setup`,
`bats-no-negated-assert`, `bats-no-exit-trap`, `bats-runner-coverage`
and `image-pin-consistency` passes 74 of 74, and `helm template` renders
the same 15 rules as main. I have not run the new rules in vmalert on a
live cluster.

The commit cherry-picks onto release-1.7 as is, and the suite passes
there. On release-1.6 it applies without conflicts, but that branch runs
`hack/*.bats` through `hack/cozytest.sh` and has no
`hack/test_helper.bash`, so the suite there has to drop its `load
test_helper` line.

### Screenshots

Not a UI change.

### Downstream repositories

A rule file under `packages/system/piraeus-operator/alerts/`, a new Bats
suite under `hack/` and its test data. No values, schema, README, image,
make target or `ApplicationDefinition` change, and the website does not
mention these rules or the metric, so no trigger map entry applies.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(piraeus-operator): the LINSTOR error-report alerts no longer fire on old reports when the LINSTOR controller pod moves to another node, when an external label of the scrape changes, or when a satellite reconnects within an hour. They now count reports per node, and their alerts carry the node's hostname instead of the controller pod's instance, job and controller_node labels.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved LINSTOR controller and satellite error alerts to account for
recent error reports over a 15-minute window, even when metrics are
intermittent or scrape labels change.
* Updated alert matching to better handle controller moves between nodes
and satellites that disappear from metrics while disconnected.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 18:18:47 +03:00
Aleksei Sviridkin 868b97dbe6 fix(kubevirt): let tenant admins upload a VMDisk image with virtctl (#5114)
## What this PR does

A VMDisk created with `source: upload` waits in `UploadReady` for an
image, but no tenant role can fill it. `virtctl image-upload` reads the
disk PVC and CDI's PVC Prime and requests an upload token, and the
tenant roles grant neither `get` on persistentvolumeclaims nor `create`
on `upload.cdi.kubevirt.io/uploadtokenrequests`, so the command stops on
a forbidden PVC read.

`cozy:tenant:admin:base` now grants both. Tenant admins and super-admins
get them in their own namespaces through the existing RoleBindings, the
use and view tiers do not. Reading the DataVolume is already granted for
the disk's own name by the vm-disk chart's dashboard role, so no
DataVolume rule is added. The vm-disk notes print the command with
`--no-create`, because the chart has already created the DataVolume, and
the security self-assessment and threat model list the new grants.

This is the RBAC part of #3847, split out so tenants can upload while
the console upload panel and the upload proxy certificate stay in review
there. The rules, their tests, the notes change and the docs lines are
from @myasnikovdaniil, who is co-author on the commit.

Checked live on a cluster with KubeVirt v1.8.4 and CDI v1.66.1, with
virtctl v1.8.4 impersonating a user in the `<tenant>-admin` group.

- With the stock roles `virtctl image-upload dv vm-disk-<name>
--no-create` fails with `persistentvolumeclaims "vm-disk-<name>" is
forbidden`.
- With exactly these two rules added as a temporary Role, the same
command reads `prime-<uid>`, uploads a 16 MiB image and the DataVolume
reaches `Succeeded`. `get` on the DataVolume came from the vm-disk
dashboard role.
- The upload went through a port-forward to `cdi-uploadproxy` with
`--insecure`. A trusted certificate on the published proxy is the other
half of #3847 and is not changed here.

Unit tests with `helm unittest` 1.1.2, the version CI pins.
`packages/system/cozystack-basics` passes 72 of 72, against 66 on main,
and `packages/apps/vm-disk` 117 of 117, against 116. Removing the upload
token rule fails one case, and dropping `--no-create` from the notes
fails one case.

Fixes #3759

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff. Two tenant RBAC rules, their tests, the vm-disk notes and two docs
files. No values key, schema, default, README or package change, so no
trigger map entry applies.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(kubevirt): tenant admins can upload an image into a VMDisk created with source upload using virtctl image-upload. The tenant admin and super-admin roles get get on persistentvolumeclaims and create on upload.cdi.kubevirt.io/uploadtokenrequests, and the vm-disk notes print the command with --no-create.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
  * Tenant administrators can now upload images into existing VM disks.
* **Improvements**
* The displayed upload command now explicitly targets the release
namespace and avoids creating a new disk.
* **Documentation**
* Security guidance now clarifies the access needed for uploads and
notes that persistent-volume permissions remain limited by namespace
scope.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 18:12:46 +03:00
Aleksei Sviridkin df42104b13 fix(seaweedfs, cozy-lib): cap the Go heap so components survive their memory limit (#4234)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Every SeaweedFS component is a Go binary running under a cgroup memory
limit, and none of them set `GOMEMLIMIT`. The collector then works from
`GOGC` alone, which is a target relative to the live heap with no
absolute ceiling, so a burst of allocation grows the heap past the limit
and the kernel kills the container.

This is not a leak. On the cluster where it showed up, the S3 gateway
held 6.4 GiB of `heap_sys` against an 8 GiB limit while `heap_inuse` was
389 MiB: the container was killed over memory the runtime was holding
and not using. Ten kills in seven days, each one an interruption of the
object store every tenant writes its backups to. Raising the limit does
not help, because nothing bounds the growth; it only moves the ceiling
the heap climbs to.

This PR adds a `cozy-lib.resources.goMemLimit` helper next to the
existing `javaHeap` one and wires it into the master, volume, filer and
s3 components and into every volume pool and zone, each sized from the
memory limit already computed for it. Pools need their own value because
the subchart merges `volume` into every entry of `volumes`: without one,
a `t1.nano` pool would inherit the value sized for the default volume's
512Mi limit, above its own 128Mi. The value is 75% of the limit, the
same fraction `javaHeap` uses for the JVM; the remaining quarter covers
what `GOMEMLIMIT` does not account for, goroutine stacks and mmap'd
regions among them.

The lifecycle admin and worker run the same binary under limits the
parent chart does not set (256Mi and 512Mi in
`packages/system/seaweedfs/values.yaml`), so their value is pinned next
to those limits at the same 75%: 192MiB and 384MiB.

A limit under 4/3 Mi (1398102 bytes) floors to `0MiB`, which Go accepts
and then collects almost continuously, so the helper fails the render
instead.

The upstream chart already supports `extraEnvironmentVars` on every one
of these components, so nothing had to change there. No values key is
added, so this needs no configuration and changes no public schema.

Tested with `helm unittest`: a suite on the seaweedfs package asserts
the value tracks the memory limit rather than a fixed string, for the
default volume, Simple pools, MultiZone zones and zone pools; the system
seaweedfs suite asserts the admin and worker env; and a helper suite in
`cozy-lib-tests` pins the unit grammar and the rejection of a zero
value. Reverting each fix makes its suite fail.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(seaweedfs): set GOMEMLIMIT on master, volume (and each pool and zone), filer, s3 and the lifecycle admin and worker so the Go runtime collects before the container hits its memory limit instead of being OOM-killed
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* SeaweedFS master, volume, pool, zonal volume, filer, and S3 workloads
now receive a Go memory limit set to 75% of their configured memory
limit.
* Pool workloads use their own resource settings when provided, falling
back to the default volume preset when needed.
* SeaweedFS admin and worker components now have Go memory limits
aligned with their configured memory limits.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 17:53:25 +03:00
Aleksei Sviridkin 4fa6bfe30b feat(monitoring): alert when pods of one workload share a node (#4993)
## What this PR does

On three 3-node clusters running v1.6.4 and v1.6.5 we found a tenant
etcd with two of its three members on one node, a tenant Redis with two
of its three Sentinels on the node that also ran the master, and several
VictoriaMetrics and VictoriaLogs pairs (vmstorage, vmselect,
vmalertmanager, vlstorage) sharing a node. Nothing alerted. Losing one
of those nodes would have cost the etcd or Sentinel quorum, or both
copies of a storage tier. Spread constraints keep new pods apart once
they are in place (#4986 for etcd, #4989 for monitoring, #4990 for
Redis), but a placement that already exists, or that comes from a soft
rule or a drain, stays as it is, because the scheduler does not move
running pods.

This adds `PodReplicasColocated`, a warning in a new
`workload-placement` group under
`packages/system/monitoring-agents/alerts/`. It fires when two or more
running pods of one StatefulSet, ReplicaSet, CNPG `Cluster` or
`StrimziPodSet` have been on one node for an hour while the cluster has
at least as many schedulable nodes as the workload has running pods, so
five replicas on three nodes stay quiet. The alert carries `cluster`,
`namespace`, `owner_kind`, `owner_name` and `node`, plus the `severity`,
`exported_instance` and `service` labels the other platform alerts set,
and its value is the number of pods on that node. DaemonSets, Jobs and
every other owner kind are not in the list, so a DaemonSet surge pod
next to its predecessor does not count.

etcd-operator gives every member pod its own `EtcdMember` as controller
owner
([etcdmember_controller.go#L546](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdmember_controller.go#L546)),
so grouping by owner would see one pod per group. Members get their
names from `GenerateName: <cluster>-`
([etcdcluster_controller.go#L434](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdcluster_controller.go#L434),
[#L839](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdcluster_controller.go#L839)),
and members adopted from the old StatefulSet keep names like `etcd-0`
([adopt.go#L103](https://github.com/cozystack/etcd-operator/blob/v0.5.6/internal/migrate/adopt.go#L103)).
The rule drops the last dash-separated part of the member name and
reports the group as `EtcdCluster <name>`. With the etcd package that is
one group per tenant namespace, and two etcd clusters in one namespace
would still be told apart. CNPG instance pods carry a controller
reference to their `Cluster`
([cluster_create.go#L1580](https://github.com/cloudnative-pg/cloudnative-pg/blob/v1.30.0/internal/controller/cluster_create.go#L1580)),
and Strimzi pods one to their `StrimziPodSet`
([StrimziPodSetController.java#L369](https://github.com/strimzi/strimzi-kafka-operator/blob/0.45.1-rc1/cluster-operator/src/main/java/io/strimzi/operator/cluster/operator/assembly/StrimziPodSetController.java#L369)).

Owner and node both come from `kube_pod_info`. kube-state-metrics fills
`created_by_kind` and `created_by_name` from the controller owner
reference and `node` from `spec.nodeName`
([pod.go#L638-L653](https://github.com/kubernetes/kube-state-metrics/blob/v2.15.0/internal/store/pod.go#L638-L653)).
`kube_pod_owner` and `kube_pod_status_phase` have no node label of their
own, and the kube-state-metrics scrape adds `node` from
`__meta_kubernetes_pod_node_name` with `honorLabels: true`
(`templates/kube-state-metrics-scrape.yaml`), so on those series `node`
is the node of the kube-state-metrics pod. The test data keeps that
label on the phase series. A pod counts while
`kube_pod_status_phase{phase="Running"}` is 1, so a Failed or Pending
pod on the same node does not. `kube_pod_labels` does carry every pod
label here (`--metric-labels-allowlist=pods=[*]`), but the rule does not
depend on it.

A node counts as schedulable unless it has a `NoSchedule` or `NoExecute`
taint. A cordon shows up as the
`node.kubernetes.io/unschedulable:NoSchedule` taint
([node_lifecycle_controller.go#L549-L555](https://github.com/kubernetes/kubernetes/blob/v1.33.12/pkg/controller/nodelifecycle/node_lifecycle_controller.go#L549-L555)),
and not ready and unreachable nodes get theirs from their conditions
([#L87-L99](https://github.com/kubernetes/kubernetes/blob/v1.33.12/pkg/controller/nodelifecycle/node_lifecycle_controller.go#L87-L99)),
so one clause covers all three, and a tainted control plane does not
count either. Pods that a drain had to put together on the remaining
nodes therefore stay quiet while the node is out, and the hour starts
once it is back.

The alert waits an hour. A replacement pod runs next to its terminating
predecessor only for the predecessor's grace period, and node
maintenance takes the drained node out of the count, while a placement
that has held for an hour does not fix itself. The alert is about a risk
rather than an outage, so the wait costs nothing. The VMAlert sets both
`remoteWrite` and `remoteRead`, so the pending state survives a vmalert
restart ([vmalert
docs](https://docs.victoriametrics.com/victoriametrics/vmalert/#alerts-state-on-restarts)).

Not covered are workloads that run one pod per owner, such as ClickHouse
with one StatefulSet per replica
([patterns.go#L45-L53](https://github.com/Altinity/clickhouse-operator/blob/release-0.25.2/pkg/model/chi/namer/patterns.go#L45-L53))
and the KubeVirt VMs behind a tenant Kubernetes node group. They would
need grouping by pod labels, which this rule avoids. The node count is
cluster-wide, so a workload that a node selector or affinity restricts
to fewer nodes than it has pods fires too.

`hack/workload-placement-alerts.bats` takes `.spec` of the VMRule and
replays it through `vmalert-tool` v1.136.0 pinned by digest, the release
vm-operator v0.68.4 deploys by default
([config.go#L38](https://github.com/VictoriaMetrics/operator/blob/v0.68.4/internal/config/config.go#L38))
and the same image #4963 and #4985 use.
`hack/testdata/workload-placement-alerts/unittest.yaml` has 13 cases
with 24 checks. Two of three StatefulSet pods on one node fire at 61
minutes and not at 59, with labels and annotations pinned. Spread pods,
five ReplicaSet pods on three nodes, a DaemonSet surge pod, two pods of
one Job, a Failed pod next to a running one, and one pod each in two
clusters that share workload and node names stay quiet. etcd members
fire as one `EtcdCluster` per namespace, for generated and adopted
member names. A replacement pod next to its terminating predecessor for
twenty minutes stays quiet, and the same overlap held past the hour
fires and resolves when it ends. A cordoned node keeps the alert quiet
until it is uncordoned, and a node with unreachable taints keeps it
quiet throughout. ReplicaSet, CNPG `Cluster` and `StrimziPodSet` pods
fire with their owner kind.

```console
$ bats hack/workload-placement-alerts.bats
1..1
ok 1 workload placement alerts pass their vmalert-tool cases
```

With the expression replaced by one that never fires, the suite prints
`not ok 1` and vmalert-tool reports 5 of the 24 checks, which are all
the checks that expect an alert. On main, where the rule file does not
exist, the suite fails as well. Each of these changes to the rule fails
at least one check, with the number of failing checks in brackets. `for:
15m` (5), no phase filter (1), no node count (6), no `cluster` label
(6), DaemonSet and Job in the kind list (3), no taint clause (4), `<`
instead of `<=` (5), etcd members grouped per member (1).

With Bats 1.14.0 at the commit CI pins, the suite passes together with
`alert-severity-contract`, `bats-strict-setup`,
`bats-no-negated-assert`, `bats-no-exit-trap`, `bats-runner-coverage`,
`bats-runner-contract`, `image-pin-consistency` and
`check-commit-trailers`, 109 of 109, and the severity contract lists the
new alert as `warning`. helm-unittest 1.1.2, the version CI pins, passes
the monitoring-agents chart with 19 of 19 tests in 6 suites, and `helm
template` with Helm 3.17.3 renders the VMRule the same as the file. Like
the migration suites the new suite needs docker, and only the image pull
touches the network. I have not run the rule in vmalert on a live
cluster.

On release-1.6 kube-state-metrics 2.15.0, its scrape config, the alerts
template, the VMAlert and vm-operator v0.68.4 are the same, and
etcd-operator v0.5.5 and CNPG 1.28.2 set the same owners and member
names. The new files apply there without changes. That backport is a
separate PR with one more commit that drops the `load test_helper` line,
because release-1.6 runs `hack/*.bats` through `hack/cozytest.sh`.

### Screenshots

Not a UI change.

### Downstream repositories

A new alert file under `packages/system/monitoring-agents/alerts/`, a
Bats suite under `hack/` and its test data. No values, schema, README,
image or `ApplicationDefinition` change, and the website does not list
platform alerts by name, so no trigger map entry applies.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(monitoring): new warning alert PodReplicasColocated fires when two or more running pods of one StatefulSet, ReplicaSet, CNPG Cluster, StrimziPodSet or etcd cluster have shared a node for an hour while the cluster has a schedulable node for each of them.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added monitoring for workloads whose running pod replicas are
colocated on the same node.
* Alerts include workload, node, cluster, and severity details, along
with quorum-loss guidance.
* Coverage includes StatefulSets, ReplicaSets, Clusters, StrimziPodSets,
and EtcdMembers.

* **Tests**
* Added validation for workload-placement alerts across scheduling,
availability, cluster-isolation, and pod-replacement scenarios.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 17:38:58 +03:00
Aleksei Sviridkin 5fe356f57e fix(postgres): accept a standard five-field backup schedule (#5119)
## What this PR does

The postgres app hands `backup.schedule` to a CloudNativePG
`ScheduledBackup`, and the field asks for the six-field CloudNativePG
cron with leading seconds. The mongodb app takes a standard five-field
cron in the same field, and so do backup Plans and the Plan form in the
console, so tenants copy a value from one app into the other and support
has to explain the difference.

CloudNativePG does not reject a five-field value. CloudNativePG 1.30.0,
which main pins, parses the schedule with `cron.Parse` from robfig/cron
v1.2.0 in both the ScheduledBackup webhook and the controller. That
parser takes five or six fields, reads the first one as seconds and
makes day-of-week optional, and the webhook only returns a warning when
the count is not six. CloudNativePG's own webhook test "warn the user if
the schedule has a wrong number of arguments" covers this with `1 2 3 4
5`. So `0 2 * * *`, meant as every day at 02:00, fires every hour at
minute 2, and `5 * * * *` fires every minute.

The chart now counts the fields. A five-field value gets a leading `0`
seconds field, so `0 2 * * *` renders `0 0 2 * * *`. A six-field value
and a descriptor such as `@daily` or `@every 6h` render exactly as
before. Any other field count fails the render with a message that shows
both forms, where before it reached the webhook and was rejected there.
An empty schedule still renders no ScheduledBackup. The field
description that the dashboard form shows and the README now name both
forms.

One thing changes on upgrade. A release that already sets a five-field
value gets the standard meaning of it, so `0 2 * * *` moves from hourly
to daily. Before this change such a value was outside the documented
format and drew the webhook warning.

`packages/system/keycloak` renders its own CloudNativePG ScheduledBackup
from its `backup.schedule`, with a six-field default and a values
comment that says so. It is a platform component with its own template,
so it is not changed here. The other apps with a backup schedule
(mongodb, mariadb, clickhouse) and backup Plans already take five
fields.

- Ran robfig/cron v1.2.0 `Parse` locally from 10:00:00. `5 * * * *`
fires at 10:00:05, 10:01:05 and 10:02:05, and `0 2 * * *` at 10:02:00
and 11:02:00. With the leading `0` they fire at 10:05:00 and 11:05:00,
and daily at 02:00.
- Rendered the whole chart with helm 3.17.3 and the `configured` render
fixture. The output matches main for the defaults, for `0 2 * * * *` and
for `@daily`, and for `0 2 * * *` only the `schedule` line differs.

Unit tests with `helm unittest` 1.1.2, the version CI pins.
`packages/apps/postgres` passes 216 of 216 in 27 suites, against 209 in
26 on main. The new suite has seven cases. With the old template five of
them fail. The six-field and descriptor cases pass on both, and they
fail when the template prepends to six fields too or drops the
descriptor check.

### Screenshots

Not a UI change. The dashboard form picks up the new field description
from the schema.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff. A chart template, a new test suite, one field description and the
README, schema, Go types and cozyrd regenerated from it. No field is
added, removed or renamed and no default or enum changes. The terraform
provider does not model `backup.schedule`. The website reference page
for postgres is generated from this README on a stable tag, and the Go
types example there uses a six-field schedule, which keeps working.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(postgres): backup.schedule accepts a standard five-field cron such as 0 2 * * *, and the chart prepends a 0 seconds field before handing it to CloudNativePG, which reads the first field as seconds. Six-field values and descriptors render as before, and any other field count fails the render. A release that already sets five fields gets the standard meaning on upgrade, so 0 2 * * * moves from hourly to daily.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* PostgreSQL backup schedules support standard five-field cron
expressions as well as six-field expressions with a leading seconds
field. Five-field schedules are converted automatically; six-field
schedules and cron descriptors are used as provided.
* **Schedule Changes**
* After upgrading, `0 2 * * *` runs daily at 02:00 instead of hourly at
minute 2. To keep the previous hourly schedule, use `0 2 * * * *`.
* Invalid schedule formats produce a rendering error. Empty schedules
continue to disable chart-managed backups.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 17:38:05 +03:00
Aleksei Sviridkin a4537127e7 fix(vertical-pod-autoscaler): run two admission controllers spread over nodes (#5098)
## What this PR does

The VPA admission controller runs as one replica, and its webhook is
registered with `failurePolicy: Ignore`. While that one pod is being
rescheduled, every pod created in the cluster is admitted without its
VPA recommendation, and with `updateMode: Initial` it keeps those
resources until it is recreated. On a v1.6.5 cluster
`vmstorage-shortterm-0` was created during a node drain while the
admission controller moved to another node. It has no `vpaUpdates`
annotation and runs with the vm-operator defaults, requests 250m/500Mi
and limits 1/1500Mi, while its VPA recommends 920m and 3.42Gi and the
other replica got 5.13Gi through VPA. It was OOMKilled 18 days later.

This PR sets `replicaCount: 2`, enables the chart's PodDisruptionBudget
with `minAvailable: 1` and spreads the two replicas over nodes with a
`ScheduleAnyway` topology spread, so single-node clusters still schedule
both. A drain then leaves one controller serving. The controller still
deletes and recreates its MutatingWebhookConfiguration on start
(`selfRegistration` in `pkg/admission-controller/config.go`, VPA 1.5.1),
so a restart leaves only that short gap instead of the whole reschedule.
`failurePolicy` stays `Ignore`, a VPA outage should not block pod
creation.

Two new cases in `tests/vpa_test.yaml` check the replica count, the
spread selector against the pod labels and the PDB. `helm unittest .` in
`packages/system/vertical-pod-autoscaler` passes 18 of 18 with
helm-unittest 1.1.2, and against the values on main the two new cases
fail and the other 16 pass. `helm template` renders the Deployment with
2 replicas and the PDB with the admission controller selector. I have
not run the change on a live cluster.

#4540 is the same trap from the other side, a component that keeps a bad
size because Initial never revisits it.

### Screenshots

Not a UI change.

### Downstream repositories

Values of `packages/system/vertical-pod-autoscaler` and its tests only.
No platform values, app schema, default exposed to tenants or package
change, so no trigger map entry applies.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(vertical-pod-autoscaler): the admission controller runs two replicas spread over nodes with a PodDisruptionBudget, so pods created during a drain still get their VPA recommendation.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Improvements**
* The VPA admission controller now runs with two replicas, increasing
availability.
* Its pods are spread across hostnames when scheduling permits, helping
reduce the impact of node-level disruptions.
* A disruption budget requires at least one admission-controller pod to
remain available during voluntary disruptions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 16:32:43 +03:00
Aleksei Sviridkin 24f2b8009f feat(redis): add maxmemory, eviction policy and snapshot toggle (#5310)
## What this PR does

Lets a tenant run Redis as a pure cache without giving up the persistent
volume. Until now the chart only turned snapshots off together with the
volume (when `size` is empty), and offered no way to bound memory: a
`CONFIG SET maxmemory` is lost when the pod restarts. On a cache-only
instance, RDB snapshots and replica full syncs then filled the volume,
and each SIGKILL during a save left an orphaned `temp-*.rdb` behind, so
the instance crashlooped.

New values:

- `maxmemory` (quantity, unset by default): rendered in bytes. Unset
renders `maxmemory 0`, the Redis default (no limit).
- `maxmemoryPolicy` (enum, default `noeviction`): rendered as
`maxmemory-policy`.
- `snapshots` (bool, default `true`): `false` renders `save ""` while
keeping the volume. Snapshots stay off when `size` is empty, as before.

Every directive is rendered with an explicit value, defaults included.
The operator applies `customConfig` with `CONFIG SET` on every reconcile
and never resets a directive whose line disappears
(`service/redis/client.go` in freshworks-oss/redis-operator v3.3.5), so
leaving the defaults implicit would turn reverting a setting into a
no-op until the next pod restart. As a consequence, existing instances
gain `save 3600 1 300 100 60 10000`, `maxmemory 0` and `maxmemory-policy
noeviction` on upgrade. These equal the Redis 7 and 8 defaults, but they
overwrite any value an operator set by hand with `CONFIG SET`.

`maxmemory` is refused at render time above 90% of the Redis memory
request: above that threshold (`reservedPodMemoryPercent`, default 10)
the operator logs an error and skips the line, which would leave Redis
unbounded with nothing visible to the tenant.

Not changed here: a replica still writes a temporary RDB to disk during
a full sync (`repl-diskless-load` defaults to `disabled`), even with
`snapshots: false`. `stop-writes-on-bgsave-error` keeps its default,
since no background save runs once snapshots are off.

Tests: `tests/cache_config_test.yaml` covers the rendered defaults, the
snapshot toggle with and without a volume, byte conversion from presets
and explicit resources, and both refusals. `helm unittest
packages/apps/redis` passes (93 tests).

### Screenshots

Not a UI change.

### Downstream repositories

The new fields in `values.schema.json` reach
`cozystack/terraform-provider-cozystack`, whose redis schema and
expand/flatten pair are hand-written; no follow-up has been opened there
yet, so the box is left unticked. The website reference page is
regenerated from `README.md` by the release bot.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(redis): add `maxmemory`, `maxmemoryPolicy` and `snapshots` values so Redis can run as a pure cache (bounded memory, eviction, no RDB snapshots) while keeping its persistent volume. Existing instances now render the Redis defaults for these directives explicitly, which resets any value set by hand with `CONFIG SET`.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Added configurable Redis memory limits, with a default of unlimited,
and eviction policies, defaulting to `noeviction`. Configured limits
must be whole-byte quantities and cannot exceed 90% of the pod’s memory
request.
* Added controls for scheduled Redis snapshots. Snapshots are enabled by
default when storage is configured; disabling them does not remove
existing snapshot files, and snapshots are off when storage size is
empty.
* **Documentation**
* Documented the new Redis memory and snapshot settings, including
defaults, constraints, and effects.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 16:32:31 +03:00
Aleksei Sviridkin 4e6ce4655a feat(dashboard): add logs and metrics views to the console (#5218)
## What this PR does

Adds logs and resource-consumption views to the console, backed by the
observability proxy from the design proposals [Tenant application
logs](https://github.com/cozystack/community/pull/95) and [Tenant
resource-consumption
metrics](https://github.com/cozystack/community/pull/94).

This PR is useful only together with #5214, which adds the console nginx
route to the proxy. The application tabs need #5219 and the
per-application roots from its follow-up PR. Until those are installed,
each view shows a quiet "not available on this cluster" state instead of
an error, so the order of merging does not break the console.

What is added:

- A "Logs" page at the namespace level, under a new "Observability"
sidebar group, reading the namespace logs root. It lives at
`/console/-/observability/logs`: every one- or two-segment console path
is an application list or detail page, and `-` cannot start an
application plural or name, so the page never shadows an application's
own pages.
- "Logs" and "Metrics" tabs on application pages (every application kind
except VMDisk, which runs no containers), reading the per-application
roots.
- Logs view:
- time range, a LogsQL filter, and pod and container pickers that turn
into a stream filter;
- "Follow" opens a live tail. The status shows "Live" as soon as the
stream is open, even before the first line; on a dropped stream or a
load refusal (`429`) it reconnects from the newest line seen, with
backoff; it stops on `400`/`401`/`403`/`404`;
- the reader can scroll up while lines stream; the newest 1000 lines are
kept;
- while a new range or filter loads, the previous lines stay on screen
with a note saying so, or the view says it is loading; it never reports
"no lines" for a query it has not read yet.
- Metrics view: per-pod CPU and memory charts and application-wide
network charts, refreshed every minute, with ranges from 1 hour to 7
days. The `rate` window is at least 5 minutes and four steps, so queries
routed to the long-term store (coarser deduplication) still return data.
No query names a namespace or kind; the proxy adds the scope.
- Error handling by proxy status: scope-wide answers (`401`/`403`/`404`)
replace the whole view with one notice; limit answers keep the proxy's
own advice. A Kubernetes `Status` body on a proxy path means the request
reached kube-apiserver instead of the proxy and is shown as "not
available".
- Shared UI pieces: an accessible `Select` and a text `Input` matching
the console's controls, and an `EmptyState`.
- `K8sClient.raw()` for untyped requests over the same credential path
as `/apis`; the client now keeps caller headers given in any form.
- Charts use uPlot, loaded in a separate lazy chunk, so the main bundle
does not grow.

#### How it was tested

- `pnpm typecheck`, `pnpm test`, `pnpm lint` and `pnpm build` pass.
- Screenshots of the views were taken against a mock API and on a test
cluster.
- Verified on a test cluster through the full browser chain (OIDC login,
gatekeeper, gateway, console nginx, proxy), as a user with the `view`
role in one tenant: the Logs page works; Follow keeps a single tail open
for more than two minutes without reconnecting; Follow on a quiet filter
shows "Live"; the Metrics tab of a Redis instance draws its charts;
requests from the same session to another tenant's roots answer `403`.
- Not verified: a screen reader (roles and names are asserted in tests
only) and viewports narrower than 1440px.

#### Known limitation

The first lines a newly started pod writes can be missing from an
already open tail, although a history query shows them. This comes from
the log store's live-tail window relative to log collection delay, not
from the console.

### Screenshots

<img width="1440" height="900" alt="cluster-follow-live"
src="https://github.com/user-attachments/assets/6efd9bfb-f05b-4498-bd20-0af4e5cda01f"
/>
<img width="1440" height="900" alt="cluster-logs-empty"
src="https://github.com/user-attachments/assets/673d8295-cd3d-409d-b871-290705d0a6a2"
/>
<img width="1440" height="900" alt="cluster-logs"
src="https://github.com/user-attachments/assets/621c2312-8427-4353-b199-44bdd12b78f7"
/>
<img width="1440" height="900" alt="cluster-pod-select"
src="https://github.com/user-attachments/assets/92ea9c72-4a3c-4562-9c3a-2170730ab085"
/>
<img width="1400" height="1000" alt="cluster-redis-metrics"
src="https://github.com/user-attachments/assets/3533d069-01b9-41de-bba9-70b940d05c6e"
/>
<img width="1440" height="900" alt="follow-waiting"
src="https://github.com/user-attachments/assets/eb318d13-5188-4632-842a-43c0bbbcff5a"
/>
<img width="1440" height="900" alt="logs-empty"
src="https://github.com/user-attachments/assets/c3d89e81-7249-444d-a3fa-9ab44c1ca86e"
/>
<img width="1440" height="900" alt="logs"
src="https://github.com/user-attachments/assets/ec6caa9c-5eda-4361-99ea-359e5d002f04"
/>
<img width="1440" height="900" alt="metrics-empty"
src="https://github.com/user-attachments/assets/db16da15-a618-4dbe-a531-593bc9b916dc"
/>


### Downstream repositories

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Downstream: no trigger matched. Candidate: none affected, because the
whole diff is inside the vendored console
(`packages/system/dashboard/images/console`), which per the trigger map
belongs to this repository and has no downstream copy. A user guide page
for the new console views on the website would be useful but is not
required by the trigger map.

### Release note

```release-note
feat(dashboard): the console gains a namespace Logs page and Logs and Metrics tabs on application pages, with live log following, per-pod CPU and memory charts, and application-wide network charts. The views need the observability proxy; without it they show "not available on this cluster".
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Added Metrics and Logs tabs to supported application details, plus a
tenant-wide Logs page in the sidebar.
* View CPU, memory, and network charts over selectable time ranges, with
optional sample tables and automatic refresh.
* Search logs using pod and container filters, choose a history range,
and follow live updates with connection status.
* Added distinct loading, empty, and error messages for observability
views, including guidance for unavailable services and query limits.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 16:26:55 +03:00
Aleksei Sviridkin 17bbca2088 feat(observability-proxy): serve tenant namespace metrics (#5219)
## What this PR does

Depends on #5214. This PR is stacked on that branch; the description
covers only what it adds on top.

Backward compatibility: this adds a new package and a new HelmRelease;
nothing existing is renamed, removed or changed, so no migration is
needed.

Adds the namespace metrics root to the observability proxy, as described
in the design proposal [Tenant resource-consumption
metrics](https://github.com/cozystack/community/pull/94). A tenant user
can query the metrics of their own namespace from VictoriaMetrics with
their Kubernetes credentials, without seeing other tenants' series.

What is added:

- Serves
`/apis/cozystack.io/v1alpha1/namespaces/<ns>/metrics/api/v1/{query,query_range,series,labels,label/<name>/values}`,
GET and POST. Anything else under the root answers `404`.
- Authorization by one `SubjectAccessReview` (`list` on
`cozystack.io/metrics` in the namespace); the aggregated tenant `view`
role gains that rule.
- The upstream request is rebuilt from an allowlist (`query`, `start`,
`end`, `step`, `time`, `match[]`, `timeout`). The scope is a single
forced `extra_filters[]={namespace="<ns>"}`; client-supplied
`extra_filters`, `extra_filters[]` and `extra_label` are dropped,
because vmselect would OR them with the forced filter. The query itself
is not rewritten.
- Label endpoints always carry a `match[]` (the namespace scope when the
client sent none), so they stay scoped even if vmselect's
`-search.ignoreExtraFiltersAtLabelsAPI` is ever enabled.
- Cost limits: maximum range (30 days), maximum points (11000), and a
clamped `timeout` (30 seconds); query-shape limits answer `422`, the
concurrency limit answers `429`. Times are accepted as unix seconds or
RFC3339; durations are rounded up to whole milliseconds before limits
are checked.
- Store selection between the short-term and long-term VictoriaMetrics
stores, configured by new chart values (`observabilityProxy.metrics`).

#### Store selection: a known limitation and an open question

The proxy has to send each query to the short-term store (3 days of
retention, finer deduplication) or the long-term store. The choice
depends on how far back the query reads, which is not only the request
window but also the windows, subquery steps and offsets inside the
expression.

The current implementation estimates that lookback with a conservative
scan of the expression text, not with a parser. It sums every `[...]`
window and subquery step, every positive `offset`, and allowances for
implicit subqueries and vmselect's default step and latency offset.
Anything it cannot read with certainty (`@`, `WITH` templates, step
multiples such as `5i`, template variables, unusual duration syntax)
goes to the long-term store. The query is forwarded unchanged, so the
estimate affects only the store choice, never the scope.

This is a heuristic. It is designed to err only towards the long-term
store, which holds the same recent data at coarser deduplication, so a
wrong guess returns coarser data rather than missing data in most cases.
A very short `rate` window sent to the long-term store can still come
back empty. Two review rounds each found new expressions the scan
underestimated, which were then fixed; there may be more.

Open question for reviewers: should the scan be replaced by parsing the
expression with `github.com/VictoriaMetrics/metricsql` (the library
vmselect itself uses), only for the store choice? That gives an exact
lookback at the cost of a new Go dependency. The namespace scope would
still be enforced by vmselect through the single `extra_filters[]`, so
the isolation design does not change either way.

A user-visible consequence of counting the lookback: a range whose start
sits right at the short-term retention edge (for example "last 3 days")
now goes to the long-term store.

#### How it was tested

- `go test -race` and `go vet` for the proxy packages; table tests for
the lookback estimate, including mutation checks of the `@` and
unparseable-window cases.
- helm-unittest for `observability-proxy` (new args and the tenant role
rule).
- Verified on a test cluster: `401` and `403` where expected; results
contain only the caller's namespace; a query naming another tenant's
namespace and a client-supplied `extra_filters[]` return nothing extra;
`query_range`, `label/<name>/values` and `series` are scoped; an admin
endpoint answers `404`.
- Not verified: the limits under load, and the store choice on windows
longer than 3 days (the test cluster had less than an hour of data).

#### Known limitations

- Relative times (`start=-1h`, `end=now`) are refused with `400`, while
the logs root accepts durations. The console sends unix seconds, so it
is not affected.
- The proxy bounds the evaluation window, the step, the number of points
of `query_range` and the timeout, not how far an expression reads:
windows and offsets inside the expression (`up[10y]`, `x[30d:1ms]`,
`offset 20d`) are bounded only by the store's retention and vmselect's
own limits.
- Times and durations are sent in a form vmselect parses exactly
(RFC3339 with milliseconds for times, durations that truncate to the
checked milliseconds); times outside 1970..2262 are refused with `400`.
A test repeats vmselect's parsing of the exact strings the proxy sends.
- `--metrics-max-range` defaults to 30 days while the longterm store
keeps 14, so the older part of a long range comes back empty rather than
refused. The two should be aligned or documented.
- The store estimate errs towards longterm: deeply nested expressions
with a large step can go to longterm although shortterm holds the data,
and longterm deduplicates at 5 minutes, so short windows such as
`rate(x[1m])` can come back sparse there.
- The `namespace` label is only as trustworthy as the writer. Any pod in
a child tenant may write to the vminsert of every ancestor tenant
(`packages/apps/tenant/templates/networkpolicy.yaml`), and vminsert does
not authenticate writers, so a child tenant can write series labelled
with another tenant's namespace. Through this proxy the owner of that
namespace would see those series as its own, and a flood of them could
push its queries over the series limit. Reading another tenant's data is
not possible this way. The write path predates this PR and already
affects the root tenant's dashboards and alerts; closing it needs the
write side to mark untrusted series, for example a vmauth in front of
vminsert that forces an `extra_label` the proxy then excludes.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Downstream: no trigger matched. Candidate: none affected, because the
diff on top of #5214 is confined to the proxy's Go code and its own
chart under `packages/system/observability-proxy` (no platform values,
no package under `packages/apps/`, no CRD, no `hack/` change). A
user-facing page for the metrics API on the website would be useful but
is not required by the trigger map.

### Release note

```release-note
feat(observability-proxy): let tenant users query their namespace metrics from VictoriaMetrics with their Kubernetes credentials, under `/apis/cozystack.io/v1alpha1/namespaces/<ns>/metrics/api/v1/`. Access requires the tenant `view` role or higher. Ranges are limited to 30 days and 11000 points.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* The observability proxy now supports tenant-scoped metrics queries
alongside logs.
* Metrics requests are routed to short- or long-term stores based on
their time range, with configurable retention and query limits.
* Metrics access is subject to tenant authorization, request validation,
and concurrency limits.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 16:26:22 +03:00
Aleksei Sviridkin 185c5eca72 fix(monitoring): route InfoInhibitor to the blackhole receiver (#4948)
## What this PR does

`InfoInhibitor` in
`packages/system/monitoring-agents/alerts/general.rules.yaml` is the
kube-prometheus rule that exists only to feed an Alertmanager inhibit
rule, and its own description says it should go to a null receiver. The
`update` target that first imported the rules rewrote its `severity:
none` to `major`, and the platform Alertmanager config has no inhibit
rule, so the alert goes to Alerta one level below critical. Alerta's
Telegram and Slack plugins forward it with the default
`disabledSeverity: []`, and once email delivery is configured, a
`notify_email="true"` label inherited from the info alert gets it
emailed as well.

It fires whenever an alert with `severity="info"` is active and no
warning or critical alert fires in the same namespace and cluster. #3799
moved the last first-party rule off `info`, but rules the platform does
not write still use it. CDI v1.66.1 generates
`CDIStorageProfilesIncomplete`, KubeVirt v1.9.0 generates
`KubeVirtVMGuestMemoryAvailableLow` and
`KubeVirtDeprecatedAPIRequested`, all at `info`, and operators add their
own. We hit it on a v1.6.4 cluster with one custom `severity: info` rule
that is always firing, and Alerta showed a permanent MAJOR
`InfoInhibitor`. The info alert itself cannot show up there, because
Alerta's default alarm model has no `info`.

This puts the label back to `none`, the value kube-prometheus ships. The
platform Alertmanager config already routes `severity="none"` to the
`blackhole` receiver ahead of every route that delivers, and the
Monitoring parameters already document that such alerts never reach
email. `hack/alert-severity-contract.bats` refuses `none` on purpose,
because an alert carrying it is never delivered, so InfoInhibitor is
listed as the one alert for which that is the point. The exemption
covers `none` and that name only.

The contract now also checks that InfoInhibitor carries `none`, that
`none` still fails on any other alert, and that the exemption does not
let InfoInhibitor through with a severity Alerta refuses. A new
helm-unittest suite in `packages/system/monitoring` pins the route
order, so that only the heartbeat route can come ahead of the blackhole,
with and without the email routes.

There is no inhibit rule here, and the expression still matches `info`
only. Wiring the inhibition up would hide every informational alert from
Alerta unless a warning fires in the same namespace, which changes
behaviour on every cluster and is a separate decision. Dropping
InfoInhibitor altogether would be equivalent for delivery, and I can
rework the PR that way if you prefer it.

v1.6.5 ships the same rule and the same routes, so a backport to
`release-1.6` would be appreciated. The commit cherry-picks onto it
without conflicts.

### Validation

- `bats hack/alert-severity-contract.bats` passes 12 of 12, and so does
the same file under `hack/cozytest.sh`. Without the label change the new
InfoInhibitor test fails on `[ "$delivered" -eq 0 ]`. With the exemption
removed the tree-wide contract fails on
`packages/system/monitoring-agents/alerts/general.rules.yaml:
InfoInhibitor -> 'none'`. With the exemption widened to any alert, or to
any severity, the matching fixture test fails.
- `helm unittest` in `packages/system/monitoring` passes 25 suites and
306 tests. Deleting the `severity="none"` route fails both new cases,
and an email route placed ahead of it fails the email case.
- `helm unittest` in `packages/system/monitoring-agents` passes 6 suites
and 19 tests, and the rendered `alerts-general.rules` VMRule carries
`severity: none` for InfoInhibitor.
- Rendered the Alertmanager config and ran `amtool config routes test`
from Alertmanager v0.31.0, the default of VM operator v0.68.4, with
InfoInhibitor's labels. `severity=major` routes to `alerta`, and to
`alerta,email` once email is configured and `notify_email=true` is
inherited. `severity=none` routes to `blackhole` in every case. `amtool
check-config` reports 0 inhibit rules.
- On `release-1.6` with this commit cherry-picked, the contract and both
helm-unittest suites pass.
- Not exercised on a live cluster.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Walked the trigger map against the diff. It changes one alert label
under `packages/system/monitoring-agents/alerts/`, extends the test in
`hack/alert-severity-contract.bats` and adds a test suite under
`packages/system/monitoring/tests/`. No package is added or renamed, and
no values, schema, CRD, platform values, variant, make target, tool the
satellites call, node prerequisite or metric changes. The website
already says that `severity="none"` alerts are blackholed and never
reach email, and no page mentions InfoInhibitor, so nothing there goes
stale.

### Release note

```release-note
fix(monitoring): InfoInhibitor no longer reaches Alerta as a major alert. It carries severity none again, which the platform Alertmanager routes to the blackhole, as upstream kube-prometheus intends. Before, any active alert with severity info, from a custom rule or from one CDI or KubeVirt generates, raised a major InfoInhibitor in Alerta and in its Telegram and Slack notifications.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* `InfoInhibitor` alerts are now treated as non-notifying, preventing
them from generating alert notifications. Other alerts retain their
existing severity handling, and heartbeat alerts continue to follow
their configured route.
* The non-notifying route takes precedence over email delivery rules
that include `InfoInhibitor`.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 15:56:58 +03:00
Aleksei Sviridkin dd6ff661bd fix(monitoring-agents): make a silent Watchdog an alarm in Alerta (#5102)
## What this PR does

The Watchdog is meant to prove the alerting pipeline is alive, and
Alerta turns it into a heartbeat because it carries `event: Heartbeat`.
When the heartbeat goes stale, nothing alarms.

- The Alerta 9.1.0 heartbeat plugin (`alerta/plugins/heartbeat.py`)
stores the alert severity in the heartbeat attributes, and the Watchdog
has `severity: ok`.
- The `alerta/alerta-web:9.1.0` image sets `HEARTBEAT_SEVERITY=major`,
and supervisord runs `alerta heartbeats --alert --severity major` about
every minute. The client it ships, 8.5.3, takes the severity of a stale
heartbeat from the heartbeat attributes ahead of that flag
(`want_severity = b.attributes.pop('severity', severity)` in
`alertaclient/commands/cmd_heartbeats.py`). The HeartbeatFail it raises
for the Watchdog is therefore severity `ok`, a normal alert.
- The Watchdog has no `timeout` label, so the Alerta webhook leaves the
timeout unset and the heartbeat only goes stale after a day (timeout
86400 on three v1.6.5 clusters), while the `event="Heartbeat"` route
resends it every minute. `/api/management/healthcheck` compares against
four timeouts and never reports it.

The Watchdog now has `severity: major`, matching the image's own
`HEARTBEAT_SEVERITY`, and `timeout: "300"`, five missed resends. The
`event="Heartbeat"` route still sends it only to `alerta_no_resolve` and
stops, so the severity reaches no email or other receiver, and the
template has no inhibit rules it could trigger. In the Alertmanager and
vmalert lists the Watchdog now shows as major instead of ok.

A new suite `tests/watchdog_heartbeat_test.yaml` checks the severity,
the timeout and the event label. `helm unittest .` in
`packages/system/monitoring-agents` passes 21 of 21 with helm-unittest
1.1.2, and against the rules on main the two new cases fail and the
other 19 pass. `hack/alert-severity-contract.bats` passes 9 of 9. I have
not run the change on a live cluster, and I have not read a heartbeat
record through the Alerta API, the chain above is from the sources of
the versions the chart pins.

The rule file is the same on release-1.6, and that backport is a
separate PR.

### Screenshots

Not a UI change.

### Downstream repositories

One rule in
`packages/system/monitoring-agents/alerts/general.rules.yaml` and a
test. No values key, schema, default or package change, so no trigger
map entry applies.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(monitoring-agents): a Watchdog that stops reaching Alerta now raises a major HeartbeatFail about six minutes later. It used to raise an ok alert after a day.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Monitoring**
* The Watchdog alert now reports a major severity and includes a
300-second timeout.
  * The alert identifies heartbeat events.
* **Tests**
* Added checks to verify the Watchdog alert’s severity, timeout, and
event.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 15:56:16 +03:00
Aleksei Sviridkin e4d059cedd fix(etcd): require an even spread of members over nodes by default (#4986)
## What this PR does

The tenant etcd chart spreads members with `maxSkew: 1` on
`kubernetes.io/hostname` and `whenUnsatisfiable: ScheduleAnyway`, the
default since #119. `ScheduleAnyway` only adds to the score of a node
and other scores can outweigh it, so two members of one etcd can still
start on the same node, where a node-local volume then keeps them. This
PR makes the default `DoNotSchedule` with `nodeTaintsPolicy: Ignore`.
The branch for `_cluster.scheduling.globalAppTopologySpreadConstraints`
is untouched, so a platform that sets its own policy renders exactly as
before, and the soft pod anti-affinity from #3962 stays.

Seen on a three-node cluster running v1.6.4, whose `etcd-cluster.yaml`
and etcd-operator v0.5.5 are the same as in v1.6.5. A new tenant etcd
got the Pods of its first two members on one node, created 26 seconds
apart, and the third on another node 18 seconds later. The Pods carry
the `ScheduleAnyway` constraint and were scheduled by
`linstor-scheduler`, whose profile adds the LINSTOR extender's
prioritize score to the same sum with weight 5. The data volumes are on
a node-local StorageClass with `WaitForFirstConsumer` and
`allowRemoteVolumeAccess: "false"`, so the two members cannot leave that
node, and losing it takes the quorum. A second tenant etcd created there
the next day came up the same way. On a test cluster four tenant etcds
created within one week each had two members on one node, and we moved
one member of each by hand.

`DoNotSchedule` does not stop an etcd with more members than nodes from
starting when every node can run its Pods, unlike a required
anti-affinity, because the skew is counted between the nodes that exist.
kube-scheduler rejects a node only when the matching Pods on it plus the
new one exceed the node with the fewest by more than `maxSkew`
([filtering.go](https://github.com/kubernetes/kubernetes/blob/v1.34.1/pkg/scheduler/framework/plugins/podtopologyspread/filtering.go#L334-L352)).
One node takes all three members, two nodes take two and one, and three
or more nodes take one each. A member whose only allowed node cannot
take it, for example because the storage pool there is full, now waits
Pending instead of joining another member.

`nodeTaintsPolicy: Ignore` is the Kubernetes default and is written out
on purpose. A cordoned node carries `node.kubernetes.io/unschedulable`,
and `Honor` would take it out of the count
([common.go](https://github.com/kubernetes/kubernetes/blob/v1.34.1/pkg/scheduler/framework/plugins/podtopologyspread/common.go#L50-L54)),
so a member created during a drain would start next to another member
and stay there on a node-local volume. With `Ignore` it waits Pending
until the node is back, while the other two keep quorum. The cost is
that a node no member can run on, such as a tainted control plane,
counts as an empty node too, so on a cluster with such a node each
schedulable node takes at most one member and the members beyond that do
not start. #4947 makes the same choice for the SeaweedFS masters and
checked both policies in kwok. A platform that wants the old behaviour
can set `scheduling.globalAppTopologySpreadConstraints` to
`ScheduleAnyway`, which the other charts that read it pick up as well,
and a new test case pins that the chart renders such a policy as
written.

On upgrade nothing is recreated. The etcd-operator adopts a changed
`spec.topologySpreadConstraints` into `status.observed` once the cluster
has reached its current target, or its progress deadline has passed
([etcdcluster_controller.go#L203-L214](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdcluster_controller.go#L203-L214)).
It copies the value from there into the spec of an `EtcdMember` only
when it creates one, at bootstrap
([#L446](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdcluster_controller.go#L446))
and on scale-up
([#L851](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdcluster_controller.go#L851)),
and nothing in the operator writes it into an existing `EtcdMember`. The
member controller creates a Pod only when it is missing and builds it
from that member's own copy
([etcdmember_controller.go#L499-L512](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdmember_controller.go#L499-L512),
[#L545-L549](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdmember_controller.go#L545-L549),
[#L860](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdmember_controller.go#L860)).
So running members keep `ScheduleAnyway`, and so does a member whose Pod
is evicted, deleted or replaced after a terminal phase
([#L164-L186](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdmember_controller.go#L164-L186)),
since the new Pod is built from the same copy. Only members created
after the upgrade get `DoNotSchedule`, which means a scale-up, a
crash-loop replacement or a member whose `EtcdMember` was deleted. No
member is rolled, neither one at a time nor all at once. This matches
the field docs
([etcdcluster_types.go#L542-L549](https://github.com/cozystack/etcd-operator/blob/v0.5.6/api/v1alpha2/etcdcluster_types.go#L542-L549),
[etcdmember_types.go#L112-L116](https://github.com/cozystack/etcd-operator/blob/v0.5.6/api/v1alpha2/etcdmember_types.go#L112-L116)).
The code paths are the same in v0.5.8, and in v0.5.5, which v1.6.4 and
v1.6.5 ship, except that v0.5.5 does not yet replace a Pod left in a
terminal phase.

Members that already share a node therefore stay there. One of them
moves only when a new `EtcdMember` takes its place, and the constraint
then sends the replacement to the node without a member. That is how we
moved the members on the test cluster, with `DoNotSchedule` set on the
`EtcdCluster` by hand. The operator's docs say a scheduling change can
be applied by deleting one Pod at a time
([concepts.md#L272](https://github.com/cozystack/etcd-operator/blob/v0.5.6/docs/concepts.md#L272)),
but the recreated Pod gets the member's old copy, so that does not move
it.

The release note now declares two cases from review. In the first, an
etcd created while the cluster had fewer nodes than members has two or
three members on one node, and once a node is added a recreated Pod of
one of them stays Pending, since `maxSkew` rejects its own node and its
node-local volume rules out the others. Deleting that member's
`EtcdMember` while the other members keep quorum replaces it on a node
without a member. In v0.5.5 the member finalizer removes the member from
etcd through the remaining members before the object goes
([`handleDeletion`](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdmember_controller.go#L195-L231),
[`removeMemberFromEtcd`](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdmember_controller.go#L243-L310)),
its Pod and its `data-<member>` PVC are owned by it and deleted with it
([#L403](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdmember_controller.go#L403),
[#L490](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdmember_controller.go#L490)),
and the cluster controller waits for the deletion, then adds a member
under a new name with the constraints from `status.observed`
([etcdcluster_controller.go#L254-L312](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdcluster_controller.go#L254-L312),
[`scaleUp`](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdcluster_controller.go#L820-L851)).
Its new PVC binds where its Pod lands, and under `DoNotSchedule` only a
node without a member passes. The operator documents the same steps for
a broken member, along with the limit that `MemberRemove` cannot run
without quorum
([operations.md#L208-L231](https://github.com/cozystack/etcd-operator/blob/v0.5.5/docs/operations.md#L208-L231)),
and v0.5.6, which main and `release-1.6` ship now, has the same path.
The upstream chart of the operator installs a policy that denies
`EtcdMember` deletion without the
`etcd-operator.cozystack.io/allow-deletion=true` annotation
([member-deletion-policy.yaml](https://github.com/cozystack/etcd-operator/blob/v0.5.5/charts/etcd-operator/templates/member-deletion-policy.yaml)),
and the `etcd-operator` package here does not install it.

On a three-node cluster running v0.5.5 a tenant etcd had two members on
one node with `DoNotSchedule` in effect. We deleted the `EtcdMember` of
one of the two while the other members kept quorum. The replacement's
Pod and its new node-local PVC were scheduled on the node without a
member, the member joined, and the `EtcdCluster` stayed `Available` with
quorum the whole time, polled every 5 seconds.

In the second case `nodeTaintsPolicy: Ignore` counts a node the Pods
cannot tolerate as one with zero members, so on a cluster with such a
node and fewer schedulable nodes than members the members beyond the
schedulable nodes do not start. The way out is the soft global policy in
the platform `Package`, set before the upgrade.

```yaml
apiVersion: cozystack.io/v1alpha1
kind: Package
metadata:
  name: cozystack.cozystack-platform
spec:
  components:
    platform:
      values:
        scheduling:
          globalAppTopologySpreadConstraints: |
            topologySpreadConstraints:
            - maxSkew: 1
              topologyKey: kubernetes.io/hostname
              whenUnsatisfiable: ScheduleAnyway
```

The platform chart writes `scheduling` into `_cluster` of the
`cozystack-values` Secret
([apps.yaml#L292-L295](https://github.com/cozystack/cozystack/blob/55aad6f91456cb545601a83aaecf39118e943b04/packages/core/platform/templates/apps.yaml#L292-L295)),
the tenant chart copies `_cluster` into the Secret of each tenant
namespace
([namespace.yaml#L109-L120](https://github.com/cozystack/cozystack/blob/55aad6f91456cb545601a83aaecf39118e943b04/packages/apps/tenant/templates/namespace.yaml#L109-L120)),
and this chart renders the value as written, which the new test case
pins. Rendering the platform and tenant charts with the value above, on
main and on `release-1.6`, gives the same string in both Secrets. The
operator adopts a changed spec into `status.observed` only once all
members are Ready or the progress deadline has passed
([etcdcluster_controller.go#L196-L213](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdcluster_controller.go#L196-L213)),
and it clears the deadline when the cluster converges
([#L1448-L1456](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdcluster_controller.go#L1448-L1456)).
A new etcd stuck short of its members adopts the policy once its first
deadline passes, and deleting the `EtcdMember` of the Pending member
then brings a replacement that starts. An etcd that had converged before
one of its replacements went Pending adopts the policy only after the
deadline is forced as the operator docs describe
([operations.md#L197-L206](https://github.com/cozystack/etcd-operator/blob/v0.5.5/docs/operations.md#L197-L206)),
which is why the release note says to set it before the upgrade.

#4817 moves this block into `_etcd.tpl` in cozy-lib without changing it,
so whichever of the two lands second has to carry the two lines over.

`tests/topology-spread_test.yaml` now pins the whole default constraint,
and a new case pins that a `ScheduleAnyway` policy from
`globalAppTopologySpreadConstraints` renders as written, without
`nodeTaintsPolicy`. The existing zone override case is unchanged.
Against the template on main the suite fails only the default case, 1
failed and 25 passed. Switching the default to `Honor`, to
`ScheduleAnyway` or to `maxSkew: 2` fails the default case, and adding
`nodeTaintsPolicy` to the override branch fails the new case.

```console
$ docker run --rm -v "$PWD":/apps -w /apps/packages/extra/etcd helmunittest/helm-unittest:4.2.3-1.1.2 .
Charts:      1 passed, 1 total
Test Suites: 9 passed, 9 total
Tests:       26 passed, 26 total
```

On main the same command gives 25 tests in 9 suites, with helm-unittest
1.1.2 as CI pins it. With default values `helm template` differs from
main in the two lines and in the comment above the affinity block, and
with a global policy set only in that comment. The docs commit changes
the `affinity` description, from which `make generate` rewrites the
README, `values.schema.json` and the `ApplicationDefinition` with
nothing but that text, and the suite stays at 26 tests. I have not
rolled this chart change out on a cluster.

The spread block and the test file are the same on release-1.6 and
release-1.7. On release-1.6 the template has no affinity block yet, so
the fix commit cherry-picks without conflicts and the docs commit has
nothing to change there. That backport is a separate PR.

### Screenshots

Not a UI change.

### Downstream repositories

One changed and one added line in
`packages/extra/etcd/templates/etcd-cluster.yaml` and its test suite,
and new text for the `affinity` description and the comment above the
affinity block. The description reaches `values.schema.json`, the README
and the `ApplicationDefinition` as text only, with no key, type or
default change, and the website rebuilds the etcd reference page from
the README on release. The website mentions
`globalAppTopologySpreadConstraints` only as a platform setting, which
stays as it is, so no trigger map entry applies.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(etcd): tenant etcd members now spread over nodes with whenUnsatisfiable DoNotSchedule by default, so two members of one etcd no longer start on the same node while another node has none. Members that exist are not moved, only members created after the upgrade get the new constraint.

An etcd created while the cluster had fewer nodes than members puts two or three members on one node. Once a node is added, such a member whose Pod is recreated (drain, eviction, graceful node shutdown) stays Pending, because maxSkew rejects its own node and its node-local volume rules out the others. With two nodes grown to three the etcd runs two of three members and its PodDisruptionBudget blocks draining the other nodes. With one node grown to three a reboot of that node brings back one member of three and quorum is lost. A stranded member is replaced by deleting its EtcdMember while the other members keep quorum. Its Pod and PVC go with it, and the replacement gets a new Pod and PVC on the node without a member. Right after adding nodes, doing this for one co-located member at a time while all members are Ready avoids the case.

With nodeTaintsPolicy Ignore a node the etcd Pods cannot tolerate, such as a tainted control plane, counts as a node with zero members, so no node can take a second member. On a cluster with such a node and fewer schedulable nodes than members, a new etcd starts only as many members as there are schedulable nodes, one of three on a single schedulable node. An existing etcd there keeps its members, but a member it replaces stays Pending the same way.

Either case is avoided by setting scheduling.globalAppTopologySpreadConstraints in spec.components.platform.values of the cozystack.cozystack-platform Package to a topologySpreadConstraints entry with maxSkew 1, topologyKey kubernetes.io/hostname and whenUnsatisfiable ScheduleAnyway before the upgrade. Every app chart that reads this value applies it, two members of a new etcd can share a node again, and the operator gives it only to members created after it adopts the changed EtcdCluster spec into status.observed. Once status.observed.topologySpreadConstraints of an EtcdCluster shows ScheduleAnyway, a member of it that stays Pending is replaced by deleting its EtcdMember.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

- **Bug Fixes**
- Default etcd placement now requires even spreading across hosts and
ignores node taints when evaluating placement.
- Configurable cluster-wide spreading continues to use a flexible
scheduling policy when selected.
- **Documentation**
- Clarified how affinity and topology spreading work together, when
required anti-affinity or node-local storage can leave members pending,
and that affinity changes take effect as members are replaced.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 15:39:53 +03:00
Aleksei Sviridkin 76204b10aa fix(ingress-nginx): raise client keepalive_time above the 1h default (#5250)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, security (a fix that comes through a security
advisory), docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Fixes merged to `main` reach the release lines the backport batch runs
for automatically, so do not open backport PRs by hand:
https://github.com/cozystack/cozystack/blob/main/docs/release/backports.md
-->

## What this PR does

ingress-nginx exposes no ConfigMap key for the client-side
`keepalive_time`, so nginx keeps its built-in 1h default: every client
connection older than an hour is closed, and on HTTP/2 that is a
`GOAWAY` (`ErrCode=NO_ERROR`). A client that multiplexes a long
operation over one connection loses the request in flight. We hit it
with a kopia repository maintenance run by Velero against a SeaweedFS S3
endpoint published through this controller: the run opened its
connection at 18:57:05, finished the heavy part, and failed at 19:57:38
while listing blobs with `http2: server sent GOAWAY and closed the
connection`, because kopia does not retry that listing. A shorter
maintenance on the previous day succeeded on the same repository.

This sets `keepalive_time 12h` through `http-snippet` in the controller
ConfigMap. Idle connections are still closed after `keep-alive` (75s),
so only connections that are actually in use live longer. No cozystack
template sets `http-snippet` today; the tenant ingress HelmRelease only
sets other `controller.config` keys, which Helm merges with this
default, and a test covers that merge.

Applied by hand on a production cluster before this PR: the controllers
reloaded cleanly and S3 traffic was unaffected.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(ingress-nginx): raise the client keepalive_time from the nginx 1h default to 12h, so long-lived HTTP/2 connections (for example kopia maintenance against S3) are no longer cut by a GOAWAY after one hour
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* The ingress controller now configures NGINX to allow HTTP connections
to remain open for up to 12 hours, supporting long-running operations
such as S3 transfers.
* **Tests**
* Added checks confirming the setting remains in place when PROXY
protocol and whitelist configuration are enabled.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 15:32:52 +03:00
Aleksei Sviridkin 7ec746a360 fix(monitoring): select the vmalert Service the operator creates (#5094)
## What this PR does

`templates/vm/vmalert-scrape.yaml` selects `app.kubernetes.io/instance:
vmalert`. The chart names the VMAlert `vmalert-<first metricsStorages
name>` (`templates/vm/vmalert.yaml` renders only the first storage), and
the operator labels its Service with that name, so with the default
storages the Service `vmalert-vmalert-shortterm` carries
`app.kubernetes.io/instance=vmalert-shortterm`. The VMServiceScrape
matches no Service and the metrics of vmalert itself never reach the
storage.

Seen on three v1.6.5 clusters. The `vmalert` VMServiceScrape in
`tenant-root` selects no Service, `count({__name__=~"vmalert_.*"})` in
vmselect-shortterm returns nothing while
`count(up{job="vmalertmanager-alertmanager"} == 1)` returns 3, and the
`/metrics` endpoint of vmalert does serve those series. No other scrape
object covers it, and no rule under `packages/` uses `vmalert_*`
metrics, so rule evaluation and notifier errors of vmalert are not
visible anywhere in the storage.

The scrape dates from v0.0.1, when the VMAlert was called `vmalert`.
4471b4ba (#289) renamed it to `vmalert-<storage>` in August 2024 and the
selector stayed, so the metrics of vmalert have not been collected
since.

The selector now takes the instance from the first `metricsStorages`
entry, the one `vmalert.yaml` renders, and the scrape is not rendered
when there are no storages, like the VMAlert itself. Keeping `instance`
instead of dropping it keeps the selector pinned to the one VMAlert this
release owns.

A new suite `tests/vmalert_scrape_test.yaml` covers the selector with
two storages and the empty case. `helm unittest .` in
`packages/system/monitoring` passes 306 of 306 with helm-unittest 1.1.2
as CI pins it, and against the template on main both new cases fail. A
full `helm template` of the chart with the default storages renders the
scrape with `app.kubernetes.io/instance: vmalert-shortterm`. I have not
run the change on a live cluster.

The template is the same on release-1.6, and that backport is a separate
PR.

### Screenshots

Not a UI change.

### Downstream repositories

One chart template and a new unittest suite in
`packages/system/monitoring`. No values key, schema, default, README or
package change, so no trigger map entry applies.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(monitoring): vmalert metrics are collected again. The vmalert VMServiceScrape selected app.kubernetes.io/instance=vmalert while the Service is labelled vmalert-<first metrics storage>, so it matched nothing.
```
2026-11-13 15:16:23 +03:00
Aleksei Sviridkin 900baf2215 fix(linstor): update piraeus-operator to v2.12.1 (#5166)
## What this PR does

Updates the LINSTOR stack to the current stable piraeus-operator
release: operator v2.12.1, LINSTOR server 1.35.3 and LINSTOR CSI
v1.13.2, the last two built here with our patches. This started as a fix
for the NFS server image hanging up the host console on bare-metal
storage nodes (Ubuntu 24.04, Cozystack 1.6.5 and 1.7.0). That needed
nfs-server v1.10.6 or later. Operators also asked for per-volume IO
limits, which need linstor-csi v1.11.3 or later and piraeus-operator
v2.10.8 or later, so the update goes to the current release rather than
the minimum.

The first commit fixes a problem the update exposed. The operator runs
the LINSTOR database migration in a `run-migration` init container, and
this package pinned only the main controller container to our build.
With the operator update, that init container ran the operator's default
LINSTOR 1.35.3 and migrated the database before the controller image
changed. The 1.33.3 controller then crash-looped until the linstor
release caught up. Both init containers that run the LINSTOR server
image (`run-migration` and the satellite's `setup-lvm-configuration`)
now use the platform image, with a test pinning it. This protects later
upgrades where the operator default is newer than our build. It does not
help this upgrade: the pin ships in the linstor release, and Flux
reconciles piraeus-operator first.

Patches:

- linstor-csi: the Protocol C override for dual attach is dropped. Since
v1.12 the driver asks LINSTOR to manage dual-primary, and LINSTOR 1.35
sets protocol C on the connections that need it. Upstream closed the
same change as superseded (piraeusdatastore/linstor-csi#435). The
relocate-after-clone patch is carried forward.
- LINSTOR server: six of the nine patches are in upstream by 1.35.3 and
are removed. `fix-k8s-create-already-exists`,
`retry-adjust-after-stale-bitmap` and `native-protoc` are carried
forward. LINSTOR 1.34 requires Java 21, so the image moves from bookworm
to trixie.

Behaviour that changes for users:

- No downgrade. LINSTOR 1.35 migrates the database and drops the `SEC_*`
tables, so a cluster that has run this version cannot go back to 1.33.
- The LINSTOR controller is down for a short time during this upgrade.
piraeus-operator is reconciled before the linstor release, so the
database is migrated to 1.35 while the old controller still runs. That
controller crash-loops until the linstor release rolls out, and if the
rollout fails there is no way back.
- A volume on a Protocol A or B resource group that was attached to two
nodes while the CSI driver was updated (a live migration in progress)
keeps the Protocol C override the old driver set on its resource
definition. The new driver does not remove it. Data is not affected, the
resource just stays on protocol C. Such resource definitions have the
`Aux/csi-protocol-override` property. To remove the override, run
`linstor resource-definition set-property <rd> DrbdOptions/Net/protocol`
and `linstor resource-definition set-property --aux <rd>
csi-protocol-override`.
- The CSIDriver moves to `fsGroupPolicy: File`, so `fsGroup` is also
applied to ReadWriteMany filesystem volumes.
- The operator manages Secrets only in its own namespace, and the CSI
controller loses cluster-wide read access to Secrets. Nothing in this
repository references snapshotter Secrets.
- The csi-health-monitor sidecar is removed.
- LINSTOR's default evict timeout goes from 1h to 12h; this package does
not set it.
- Volume group snapshots turn on only when
`groupsnapshot.storage.k8s.io/v1` is served. The snapshot controller
here still serves v1alpha1, so they stay off; #5239 changes that.
- IO limits can be set through `qos.linbit.com/{rbps,wbps,riops,wiops}`
StorageClass parameters. Enforcing them needs NRI enabled on the nodes
and the nri-volume-qos plugin, neither of which this PR adds.

Tested on a three-node Talos cluster, upgrading from the current main
packages to this branch:

- LINSTOR database migrated to 1.35.3, no faulty resources, no error
reports.
- Data written before the upgrade is intact, checked by checksum.
- A snapshot taken before the upgrade restores, and all three replicas
come up `UpToDate`; cloning and new snapshots work.
- KubeVirt live migration works for a ReplicatedBlock volume, including
one on a Protocol A resource group: LINSTOR switches the connection to
protocol C for the migration and restores A afterwards.
- A StorageClass with QoS parameters provisions, and the limits reach
the resource definition and the VolumeAttachment metadata.
- ZFS userspace in the satellite is now 2.4.4, matching the kernel
module.

Not tested: an arm64 build.

### Screenshots

No UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(linstor): update piraeus-operator to v2.12.1, LINSTOR to 1.35.3 and LINSTOR CSI to v1.13.2. Fixes the NFS server breaking console login on storage nodes and adds IO QoS StorageClass parameters. LINSTOR 1.35 migrates its database, and a downgrade to an earlier version is not possible after the upgrade.
```
2026-11-13 15:15:43 +03:00
Aleksei Sviridkin 30c4232be1 fix(seaweedfs): stop the master crashing on the crowded volume map (#5308)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Fixes merged to `main` reach the release lines the backport batch runs
for automatically, so do not open backport PRs by hand:
https://github.com/cozystack/cozystack/blob/main/docs/release/backports.md
-->

## What this PR does

Backports seaweedfs/seaweedfs#11216 ("topology: fix fatal concurrent map
read/write on VolumeLayout.crowded", released in 4.46) as a patch on the
in-tree 4.45 image, next to `reader-cache-prefetch-window.diff`.

In 4.45, `(*VolumeLayout).SetVolumeCrowded` mutates the `crowded` map
under `accessLock.RLock()`, while
`(*VolumeLayout).GetWritableVolumeCount` reads the same map under
`RLock()` on the `Assign` path. Two read-lock holders, one writing and
one reading the map, make the Go runtime abort the master with `fatal
error: concurrent map read and map write`. This cannot be recovered, so
the process exits and S3 requests that need an assign fail until a
master is back. The fix takes the write lock in `SetVolumeCrowded`;
every other mutation of `crowded` already holds it.

`patches/volume-layout-crowded-lock.diff` is the upstream squash commit
5d5ea18287cfbf9e274f4b78be40a0ae220cb55d (files under `weed/topology/`),
regenerated against the 4.45 source so it applies without offset. Its
content is unchanged, including the regression test
`TestSetVolumeCrowdedNoRaceWithGetWritableVolumeCount`, which the
Dockerfile's existing `go test` step runs on every build.

#### Seen in production

On a production platform running Cozystack v1.7.0 (in-tree SeaweedFS
4.45), with a dedicated S3 cluster used as the Velero backup target:

- Since the upgrade to 4.45 on 2026-11-05, all 3 masters crash every
night with exit code 2 under the backup load, between 01:00 and 03:00
UTC: 2 crashes on 2026-11-06, about 23 on 2026-11-13, 33 to 43 restarts
per master over 8 days.
- The stack is always the same:
  ```
  fatal error: concurrent map read and map write
  ...

github.com/seaweedfs/seaweedfs/weed/topology.(*VolumeLayout).GetWritableVolumeCount(...)
      weed/topology/volume_layout.go:834
  github.com/seaweedfs/seaweedfs/weed/server.(*MasterServer).Assign(...)
      weed/server/master_grpc_server_assign.go:97
  ```
- Clients see HTTP 500 `InternalError` from the S3 gateway, and the
nightly backups end `Failed` or `PartiallyFailed` (4 `Failed` and 3
`PartiallyFailed` on the night of 2026-11-12 to 13).

Upstream issue: seaweedfs/seaweedfs#11211.

#### Testing

- `git apply --check` of both patches against the 4.45 source archive
(`SOURCE_SHA256` in the Dockerfile matches).
- Regression test on 4.45 with only the test applied, `CGO_ENABLED=0` as
in the image build: `fatal error: concurrent map writes`, FAIL. With the
fix: `go test ./weed/topology/` passes, and the new test passes under
`-race` too.
- `docker buildx build images/seaweedfs --platform linux/amd64` builds,
running the patched packages' tests.
- `make test` in `packages/system/seaweedfs`: 107 helm unit tests, the
naming-guard check and the image/chart version check pass.

#### Lifecycle

The patch is dropped once the vendored chart reaches 4.46 or later, for
example with #4879 (4.48 bump). From that release on it no longer
applies, and `make update-image` fails until it is removed, as the
Dockerfile header says. This PR is kept small so that it does not wait
for #4879.

#### Backport

This is a `fix` with no CRD, migration or API change, and `release-1.7`
carries the same `images/seaweedfs/` as `main`, so the backport batch
should take it onto `release-1.7` with a clean cherry-pick for the next
v1.7.x. Clusters on v1.7 are the ones that crash today.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff touches only `packages/system/seaweedfs/images/seaweedfs/` (a
Go patch and a Dockerfile comment): no package added or renamed, no
values, schema, CRD, `hack/` or installer change, so no entry of the
trigger map applies.

### Release note

```release-note
fix(seaweedfs): stop SeaweedFS masters crashing with "concurrent map read and map write" under write load, which made S3 answer 500 (backport of seaweedfs/seaweedfs#11216 onto 4.45).
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved reliability when updating crowded-volume state concurrently
with writable-volume queries.
* **Documentation**
* Added notes about the upstream fix and when the local patch can be
removed.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 14:52:45 +03:00
Aleksei Sviridkin 3fce040268 Merge branch 'main' into fix/etcd-strict-member-spread
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 14:20:49 +03:00
Aleksei Sviridkin d089decab9 Merge branch 'main' into fix/tenant-namespace-direct-delete-guard
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 14:20:44 +03:00
Aleksei Sviridkin 054852cfce Merge branch 'main' into feat/pod-replicas-colocated-alert
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 14:20:35 +03:00
Aleksei Sviridkin f1067fdd64 Merge branch 'main' into fix/linstor-error-report-alerts-controller-move
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 14:20:23 +03:00
Aleksei Sviridkin d2c5bb05c8 Merge branch 'main' into fix/operator-crds-upgrade
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 14:20:08 +03:00
Aleksei Sviridkin 95364b8ba0 Merge branch 'main' into fix/tenant-cdi-upload-rbac
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 14:07:40 +03:00
Aleksei Sviridkin 5543cfc37c Merge branch 'main' into fix/watchdog-heartbeat-alarm
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 14:07:36 +03:00
Aleksei Sviridkin 78d79b22f6 Merge branch 'main' into fix/vpa-admission-two-replicas
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 14:07:27 +03:00
Aleksei Sviridkin 4f19e038c9 Merge branch 'main' into fix/monitoring-vmalert-scrape
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 14:07:18 +03:00
Aleksei Sviridkin e8475963ee Merge branch 'main' into fix/monitoring-infoinhibitor
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 14:07:04 +03:00
Aleksei Sviridkin 97e0618258 Merge branch 'main' into fix/postgres-backup-schedule-five-fields
Regenerate the postgres ApplicationDefinition schema on top of main.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 13:39:51 +03:00
Mattia Eleuteri e3e2018075 fix(seaweedfs, cozy-lib): cap the Go heap so components survive their memory limit
Every SeaweedFS component is a Go binary running under a cgroup memory
limit, and none of them set GOMEMLIMIT. The collector then works from
GOGC alone, a target relative to the live heap with no absolute ceiling,
so a burst of allocation grows the heap past the limit and the kernel
kills the container.

Seen on an S3 gateway holding 6.4 GiB of heap_sys against an 8 GiB limit
while only 389 MiB was actually in use: ten kills in seven days, each one
an interruption of the object store every tenant writes its backups to.
Raising the limit does not help, since nothing bounds the growth; it only
moves the ceiling the heap climbs to.

Derive the value from the limit already computed for each component, at
the same 75% the javaHeap helper uses for the JVM. The remaining quarter
covers what GOMEMLIMIT does not account for, goroutine stacks and mmap'd
regions among them.

Volume pools and zones carry their own value: the subchart merges volume
into every entry of volumes, so a pool would otherwise inherit the value
sized for the default volume's limit, above its own. The lifecycle admin
and worker run the same binary under limits the parent chart does not
set, so their value is pinned next to those limits.

A limit under 2Mi floors to 0MiB, which Go accepts and then collects
almost continuously; the render fails instead.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-11-13 11:25:08 +01:00
Aleksei Sviridkin c2d370e95c fix(seaweedfs): pin the COSI driver release with public-read access
The public BucketAccessClass relies on the public-read accessPolicy,
which the driver supports from v0.5.0; v0.4.0 would hand out keys
instead of anonymous read access.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 13:02:13 +03:00
Mattia Eleuteri 21fd2b7b69 feat(bucket): add publicRead to serve a bucket without credentials
A bucket cannot serve its objects without a signature: every
BucketAccess issues a key, and the keys grant neither WriteAcp nor
PutBucketPolicy, so a tenant cannot open the bucket themselves with
put-bucket-acl. Static assets then need a proxy, presigned URLs or
an operator running weed shell by hand, which no reconcile tracks
and which outlives the bucket.

publicRead renders one more BucketAccess on a new -public access
class whose accessPolicy is public-read. The COSI driver handles that
policy by adding Read:<bucket> to the gateway's anonymous identity
and removing only that action on revoke, so deleting the bucket or
turning the flag off closes it again without touching other public
buckets. Anonymous callers can read objects but cannot list the
bucket or write to it.

The access is named <bucket>-public-read, the name a user called
public-read would take, so that combination fails to render instead
of fighting over one BucketAccess.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-11-13 13:02:02 +03:00
Aleksei Sviridkin 5e07258acb Merge branch 'main' into feat/ingress-nginx-keepalive-time
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 12:44:57 +03:00
Aleksei Sviridkin 56224a156c feat(virtual-machine): let each disk set its own boot order (#5266)
## What this PR does

A `vm-instance` boots from its disks in the order they appear in
`disks`: the template hardcodes `bootOrder: {{ add $i 1 }}`. Booting
from installer or rescue media, such as an ISO attached as a CD-ROM next
to the system disk, therefore means moving that media to the front of
the list, and anything that treats `disks[0]` as the system disk then
finds the ISO there instead.

This adds an optional `bootOrder` to each entry of `disks`:

- When set, it is the disk's boot priority (1 is tried first),
independent of its position in the list.
- When unset, the disk keeps its positional order (`index + 1`), so
existing VMs render exactly as before.
- The values schema requires `bootOrder >= 1`.
- Duplicate effective boot orders are refused at render time with the
names of both disks, rather than surfacing as a KubeVirt admission error
on the VirtualMachine. This includes an explicit value colliding with
another disk's positional one, so booting from a CD-ROM in second
position means setting both disks:

```yaml
disks:
  - name: system
    bootOrder: 2
  - name: install-iso
    bus: sata
    bootOrder: 1
```

`make generate` was run in the package, so the README,
`values.schema.json`, the Go type in `api/apps/v1alpha1/vminstance`
(with its regenerated deepcopy, since the field is a pointer) and the
`vm-instance-rd` OpenAPI schema are all updated.

A new suite, `tests/disk_boot_order_test.yaml`, mocks the DataVolumes
the template looks up and covers: positional order when no disk sets
`bootOrder`; a CD-ROM in second position booting first while the list
order and volumes stay unchanged; an explicit value colliding with a
positional one; two equal explicit values; and a value below 1 rejected
by the schema. Run against the previous template, the CD-ROM and
collision cases fail. `helm unittest packages/apps/vm-instance`: 6
suites, 77 tests passing.

### Screenshots

No UI change.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

This adds a field to `vm-instance`'s `values.schema.json`, which the
trigger map routes to `cozystack/terraform-provider-cozystack`: its
`vminstance_schema.go` and `vminstance_model.go` model `disks` by hand,
so `bootOrder` needs adding there. The follow-up has not been opened yet
and the box is left blank until it is linked. The website reference page
is regenerated from the README by the release bot; nothing else in the
map is touched.

### Release note

```release-note
feat(virtual-machine): disks in a vm-instance accept an optional `bootOrder`, so a VM can boot from an attached ISO or other rescue media without reordering its disks. Unset keeps the previous positional boot order.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Configure a boot priority for each VM disk. Lower values are tried
first; disks without an explicit priority use their position in the disk
list.
* Disk priorities must be unique, and values below 1 are rejected.
Priority changes do not reorder disks or their associated volumes.
* **Documentation**
  * Added guidance and an example for configuring disk boot priorities.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 12:40:31 +03:00
Maxim Belyy 34843c1a91 Merge branch 'main' into feat/observability-proxy-metrics
Assisted-by: LLM
Signed-off-by: Maxim Belyy <maximbel2003@gmail.com>
2026-11-13 16:07:33 +07:00
Maxim Belyy e123cef283 Merge branch 'main' into feat/console-observability
Assisted-by: LLM
Signed-off-by: Maxim Belyy <maximbel2003@gmail.com>
2026-11-13 16:07:32 +07:00
Aleksei Sviridkin b6f5a4fa35 feat(monitoring): stamp application identity on scraped series (#5216)
## What this PR does

Adds a default scrape class `application-identity` to both VMAgents: the
cluster-wide one in `monitoring-agents` and the per-tenant one in
`monitoring`. Every series scraped from a pod that carries the lineage
labels `apps.cozystack.io/application.{group,kind,name}` gets them as
`apps_cozystack_io_application_{group,kind,name}`.

Why: the per-application metrics scope in the design proposal [Tenant
resource-consumption
metrics](https://github.com/cozystack/community/pull/94) (Design 4a)
needs to tell which application an exporter series belongs to. Today
that is only possible indirectly, through pod names.

How the labels are protected against forgery by an exporter:

- At the target stage, the class always sets all three labels: to the
pod's real lineage value, or to a placeholder `-` when the pod has none
(`-` is not a valid Kubernetes label value). With `honorLabels: false`,
any copy the exporter publishes therefore collides with the target label
and is renamed to `exported_*`.
- At the metric stage, the class drops every `exported_*` copy.
- The placeholder is removed by the VMAgent's global remote-write
relabeling (`inlineRelabelConfig`), not at the metric stage: vmagent
adds `up` and `scrape_*` with the target labels after metric relabeling,
so only the remote-write stage sees those series. Targets without
lineage therefore keep exactly the label set they have today on every
series, including `up` and `scrape_*`.
- With `honorLabels: true` the exporter's value wins and no relabel rule
can tell it apart. ingress-nginx scrapes with `honorLabels: true`, and
its series describe Ingress objects in any namespace, so the
controller's identity would misattribute traffic; its scrape now drops
the identity labels explicitly, from the target (so `up` and `scrape_*`
stay unchanged) and from the scraped samples (so an exporter's copy
cannot survive). Switching it to `honorLabels: false` was not done,
because that renames `namespace` and breaks the existing ingress
dashboards.

#### How it was tested

- helm-unittest for `monitoring-agents`, `monitoring` and `ingress`.
These tests check the rendered config only; the relabeling behaviour was
verified offline by running vmagent v1.136.0, built from source, against
the rendered rules: `up` and `scrape_*` carry no identity for targets
without lineage, and the placeholder appears on no series.
- The rendered `scrapeClasses` validate against the vendored VMAgent CRD
schema.
- The rendered rules were exercised on a locally built vmagent v1.136.0
(the version the bundled operator deploys) writing to a local
VictoriaMetrics: with and without lineage, with and without a forging
exporter, `honorLabels` true and false, and the ingress-nginx endpoint
rules. Checked on exporter samples and on `up`, `scrape_samples_scraped`
and `scrape_series_added`; the previous rules reproduce the `-`
placeholder on the generated series of targets without lineage, the new
ones leave no placeholder on any series.
- Verified on a test cluster with the earlier revision that removed the
placeholder at the metric stage (the move to remote-write relabeling is
not yet verified on a cluster): series carry the application labels;
metrics read for a Redis instance through the per-application proxy root
contained only that instance's pods; vmagent logs show no errors.

#### Known limitations

- Scrapes with `honorLabels: true` other than ingress-nginx are not
protected against an exporter that publishes the identity labels itself.
Today these are the vendored qdrant and harbor ServiceMonitors; their
exporters are platform images with fixed label names, and their series
describe the application's own pods.
- A tenant's raw `vmagent.inlineScrapeConfig` bypasses scrape classes
entirely, and so do `additionalScrapeConfigs`. VMProbe takes only auth
settings from a scrape class (operator v0.68.4); no VMProbe exists in
the platform today.
- A scrape object's own relabel rules run after the class and can
overwrite the labels; tenants cannot create VictoriaMetrics operator
objects, so this is platform-controlled.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Downstream: no trigger matched. Candidate: none affected, because the
diff changes templates and tests in `packages/system/monitoring-agents`,
`packages/system/monitoring` and `packages/extra/ingress` without
adding, renaming or removing a package and without touching any
`values.yaml` or schema. The telemetry metrics the telemetry server
reads (`cozy_cluster_info`, `cozy_application_count`) are produced by
the operator, not by these scrapes. The proposal text in
`cozystack/community` (#94, Design 4a) should be corrected to name
qdrant and harbor as further `honorLabels: true` scrapes; that is not a
trigger-map item.

### Release note

```release-note
feat(monitoring): VMAgent now adds `apps_cozystack_io_application_{group,kind,name}` labels to every series scraped from a pod that carries application lineage labels. Because the label set of those series changes, each of them becomes a new series after the upgrade: graphs and dashboards show a break at the rollout, queries over a range that spans it see two series per pod, and alert `for:` timers on those series restart. Series of targets without lineage labels, including node-level series (cAdvisor, kubelet) and kube-state-metrics, and all ingress-nginx series keep their current label set; this includes their `up` and `scrape_*` series.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Improvements**
* Monitoring metrics now use application identity labels derived from
Kubernetes pod metadata for covered scrape targets. Exporter-supplied
duplicates and placeholder values are removed, including from `up` and
`scrape_*` metrics.
* Ingress metrics no longer retain application identity labels at either
target or metric relabeling stages.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 12:03:57 +03:00
Mattia Eleuteri c6711d6fe9 fix(linstor): prefer a schedulable snapshot node over topology hints
The preferred nodes come from the requisite topology, which may leave
out a snapshot node. Filtering on them first kept a cordoned node when
the only schedulable snapshot node was outside that list, so the
restored volume could still be unusable. Schedulability now decides
before the topology hints, which only break ties.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-11-13 09:43:20 +01:00
Mattia Eleuteri d308e9e5e5 fix(redis): refuse sub-byte maxmemory and apply the policy first
A quantity such as 512m is 0.512 bytes; the integer cast truncated it,
or -512m, to 0, which Redis reads as no limit, so the bound a tenant
set could vanish without an error. The operator also sends the
directives in list order, and lowering maxmemory below the dataset
evicts at once, so the policy has to be set before the bound.

Turning snapshots off does not remove an RDB already on the volume,
which Redis still loads on restart; the parameter no longer says the
data is lost.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-11-13 09:43:05 +01:00
Andrey Kolkov 906eeb2372 feat(backups): add Kubernetes backup strategy for tenant clusters (#4931)
## What this PR does

Adds a `Kubernetes` backup strategy so a tenant Kubernetes application
can be backed up and restored through the backups framework, and routes
`apps.cozystack.io/Kubernetes` to it in `cozy-default`.

None of the existing strategies fits a tenant cluster:

- Its API state is a key prefix (`/<namespace>_<release>/`, the
TenantControlPlane's datastore schema) in the etcd `DataStore` shared by
every cluster of the tenant. The `Etcd` strategy snapshots the whole
DataStore, so restoring it would roll every tenant cluster back at once.
- Its control-plane CA, front-proxy CA, service-account key and Talos
secrets cannot be regenerated: new ones break every node, kubeconfig and
token issued under the old ones.
- Its PersistentVolumes are DataVolumes in the tenant namespace
(`pvc-<tenant pvc uid>`, labelled `cluster.x-k8s.io/cluster-name`), and
deleting the application deletes them.

**Backup** (`strategy.backups.cozystack.io/Kubernetes`, driver in
`internal/backupcontroller/kubernetesstrategy_*.go`):

1. Checks the control-plane and Talos certificates and sets a
`CertificatesExpiring` condition (plus a Warning event) when a CA
expires before the backup's TTL or a leaf within
`certificateWarningThreshold` (720h). This never fails the run.
2. A Job in the application namespace dumps the cluster's prefix with
`etcdctl` to the platform bucket. It authenticates with the cluster's
own `kubernetes-<name>-datastore-certificate`, the one its
kube-apiserver uses. It runs on a new `kubernetes-backup` image (alpine
+ curl + jq + etcdctl matching the platform's etcd).
3. A Velero backup takes the PKI Secrets (selected by
`kamaji.clastix.io/name` / `app.kubernetes.io/instance`; the driver adds
the instance label to the cert-manager-issued Talos CA Secrets, which
carry neither) and the data DataVolumes/PVCs. Worker system disks are
not backed up.
4. The Backup records the prefix, the endpoints, the data DataVolumes
and the HelmReleases of the cluster and its `KubernetesNodes` pools. The
HelmReleases are not in Velero because the lineage webhook labels worker
disks after the same applications, so a label selector for the releases
would also pick the disks.

**Restore** goes into the application the backup came from (rename /
cross-namespace is rejected). `RestoreJob.spec.options` takes `etcd` and
`volumes` (both default `true`) to restore either half alone:

- **Application gone**:
  1. Restore the PKI Secrets and the volumes.
2. Recreate the cluster's HelmRelease. The Kamaji release this repo
ships gives a new TenantControlPlane a UID-keyed datastore schema, so
the restore waits for that schema.
3. Stop the new control plane, write the dump under the new prefix (keys
are moved from the old prefix), and start it again.
4. Recreate the worker pools last, so workers join the restored cluster
with the restored CA and Talos secrets rather than an empty one.
- **Application live**: suspend the cluster's HelmReleases, then stop
the MachineDeployments and the cluster-autoscaler (for `volumes`,
deleting the backed-up DataVolumes/PVCs once the Machines are gone), and
then the KamajiControlPlane (for `etcd`). Workers go first because CAPI
removes a worker's Node through the tenant API. Then write the selected
halves back and restore every replica count. Each original count is
recorded on the object it belongs to, so the cluster is put back on
success, on failure, after a 30-minute stop deadline, and when the
RestoreJob is deleted midway. A retry keeps the counts from before the
first attempt, and a HelmRelease suspended before the restore stays
suspended.

Before changing anything, a restore checks that the Velero backup is
still there and Completed, and that the dump object exists (a
verify-mode run of the Job). The restore mode is recorded in the
RestoreJob's status, so a RestoreJob cannot claim that a live cluster is
gone.

Before it runs `del --prefix`, the restore Job downloads and validates
the dump and saves the current prefix to `<dump>.pre-restore`. That copy
is never overwritten, so a retry cannot replace it with a partial
prefix, and it is dropped after a successful restore. The Job writes a
marker to its termination message, and the outcome depends on it:

- A failure before `del` resumes the cluster.
- A failure after `del` leaves the control plane stopped
(`KubernetesEtcdWriteFailed`) rather than started on a partial prefix.
- Deleting the RestoreJob while its restore Job runs also leaves it
stopped.

Events and master leases are skipped only as top-level resources. Velero
cannot overwrite an existing claim, so a live-cluster volume restore
deletes the current claims first; this is documented.

A backup whose Velero half fails deletes the etcd dump it already
uploaded, and a transient error creating the Backup is retried rather
than failing the run. Deleting a Backup removes the Velero backup
(DeleteBackupRequest) and the dump object plus its pre-restore copy (a
cleanup-mode Job, given up after five failed attempts with a Warning,
plus the same skip-artifact-cleanup escape hatch as Redis/Rabbitmq).
With `velero.bslEnabled=false` the strategy is not rendered, and the
DefaultObjectsGate does not count it as missing, the same as the Velero
strategies. Like every default strategy, it is also gated on its CRD
being established, so the upgrade that first ships the CRD to an
installed chart cannot wedge the release.

Also included: RBAC for
TenantControlPlanes/DataStores/KamajiControlPlanes/MachineDeployments/Machines/Deployments,
chart values (`kubernetesBackupClientImage`,
`kubernetesBackupActiveDeadlineSeconds`, `kubernetesBackupResources`), a
`make image-kubernetes-backup` target, docs in
`docs/operations/backup-classes.md`, and the
`examples/backups/kubernetes/` walkthrough.

### Verification

- `go test ./internal/backupcontroller/`: the new tests cover:
- the certificate windows, pool HelmRelease discovery, option parsing
and the restore-phase selectors;
  - the full backup path;
  - in-place restores with default options and etcd only;
  - a pre-suspended HelmRelease;
- a failed etcd write leaving the control plane stopped, including on
RestoreJob deletion;
  - RestoreJob deletion resuming a stopped cluster;
  - the stop deadline;
  - the preflight: missing or incomplete Velero backup, missing dump;
  - a rebuild, and an unreadable HelmRelease record;
- rename rejection, a non-etcd datastore, and the cleanup attempt limit.

Each safeguard in the Go driver was checked by mutating it out and
confirming that a test fails. The rendered Job script has no committed
test: a unit test would have to extract the script from the rendered
chart, and its behaviour belongs in the chainsaw e2e follow-up instead.
It was run against stub `etcdctl`/`curl` in every mode and branch:
  - a foreign key or an empty value stops it before any etcd call;
- an S3 answer other than 200/404 on the pre-restore copy stops it
before any etcd call;
- an existing copy is kept, and a missing one is uploaded before `del`;
  - keys are moved from the old prefix to the new one;
  - an object in a namespace named `events` is kept.
- `helm unittest` (29/29), `hack/backupstrategy-crd-gate.bats` and `helm
lint` on `backupstrategy-controller`.
- On a live cluster (Velero 1.18.1), with a tenant cluster holding a
ConfigMap marker and a file on a PVC:
- The same flow done by hand before the driver existed: back up, delete
the application, rebuild. The CA, front-proxy CA, SA key and Talos
secrets were byte-identical afterwards, the pre-backup admin kubeconfig
still worked, namespace and ConfigMap UIDs were unchanged, and the PVC
data came back.
- Through this driver: the BackupJob succeeded (432 keys dumped, Velero
backup Completed, Backup metadata as described above). The tenant
namespace holding the markers was then deleted, which also deleted its
PVC and the backing DataVolume, and a RestoreJob with default options
rolled the live cluster back. It stopped the workers and then the
control plane, wrote the prefix back, restored the volume, and restored
the control plane to 2 replicas, the pool and the autoscaler to 1, and
unsuspended the HelmRelease. A new worker joined. The namespace came
back with its original UID, the ConfigMap and the PVC (bound to the same
volume) with their data. That run caught an ordering bug, now fixed and
covered by a test: scaling the control plane down first left the old
Machine stuck in `Deleting`, because CAPI removes the Node through the
tenant API.
- On the current head, through the driver, against the same tenant
cluster:
- **Backup.** It succeeded. The volume list it recorded was taken before
Velero started.
- **`{"volumes":false}` restore after a ConfigMap was deleted and
another created.** The deleted ConfigMap came back and the newer one was
gone. The worker was left running and the volume data was untouched. The
restore waited for terminating control-plane Pods before writing. The
pre-restore copy was saved before `del` and dropped afterwards.
- **Rebuild after the application was deleted, with default options.**
It ran the full sequence: secrets, volumes, cluster HelmRelease,
datastore schema, stop, etcd write, resume. It succeeded in about 3
minutes. The CA and Talos CA were byte-identical, a new worker joined,
and the namespace and ConfigMap UIDs and the PVC data were back. The
tenant cluster runs on a Kamaji that still names schemas `<ns>_<name>`,
so this run did not exercise a prefix change; that path is covered by
the unit tests and the stub runs.

### Not in this PR

- A chainsaw e2e. The tenant Kubernetes suites are bounded by tight op
budgets that tests pin, so the round trip needs its own suite or a
budget change; it will be a follow-up.
- `rotationPolicy` on the Talos CA Certificate in the kubernetes chart
(cert-manager's default `Always` would rotate the CA key at renewal).
- Restore into another application name or namespace.
- The tenant's shared etcd DataStore runs with auth disabled, so any
cluster's datastore certificate can read its neighbours' prefixes. This
predates this change; the dump Job uses the same certificate the
cluster's kube-apiserver already holds.

### Screenshots

N/A — no UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Walked against the diff: no package under `packages/apps|extra` is added
or renamed, `packages/core/platform` and installer values are untouched,
the new CRD is in `strategy.backups.cozystack.io` (not typed by
terraform-provider), and `RestoreJob.spec.options` stays opaque JSON.

### Release note

```release-note
feat(backups): tenant Kubernetes applications can now be backed up and restored through the cozy-default BackupClass. A backup captures the cluster's etcd state and its control-plane PKI and PersistentVolume data, and warns when a control-plane CA would expire before the backup does. A restore rebuilds a deleted cluster with its original CA or rolls a live one back, and can restore the etcd state and the volumes separately via RestoreJob options `etcd` and `volumes`.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

- **New Features**
- Added Kubernetes application backup and restore support, including
etcd data and persistent volumes.
- The default backup class now routes Kubernetes applications to the new
strategy.
- Added an end-to-end walkthrough for creating a cluster, backing it up,
restoring it in place or after deletion, and cleaning up.
- Backup status can report expiring certificates, and restore validation
helps protect cluster identity and state.
- **Documentation**
- Added guidance on backup contents, restore behavior, and walkthrough
configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 15:40:01 +07:00
Mattia Eleuteri 0f16bf9c73 feat(redis): add maxmemory, eviction policy and snapshot toggle
A Redis used purely as a cache had no durable way to bound its memory
or stop writing RDB snapshots: the chart only disabled snapshots
together with the persistent volume, and a CONFIG SET is lost when the
pod restarts. Snapshots and replica full syncs then filled the volume
and left the instance crashlooping.

maxmemory, maxmemory-policy and save are always rendered with an
explicit value, because the operator applies customConfig with CONFIG
SET on every reconcile and never resets a directive whose line
disappears; leaving the defaults implicit would make reverting a
setting a no-op until the next pod restart. maxmemory is refused above
90% of the memory request, the threshold above which the operator
silently skips the line and leaves Redis unbounded.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-11-13 09:30:50 +01:00
Mattia Eleuteri 1725226d23 fix(linstor): restore snapshots on a schedulable node
A volume restored from a snapshot is pinned to the one node it is
restored on. linstor-csi picked that node among every ONLINE snapshot
node: its preferred-node filter matched all nodes, so the node the
scheduler selected got no priority, and a cordoned node was as good as
any other. A pod whose volume landed on a cordoned node stayed Pending
until the node was uncordoned, which made Velero data mover backups
time out on every storage node cordoned ahead of maintenance, even when
schedulable replicas held the same snapshot.

Prefer the selected node when it holds the snapshot, and skip cordoned
nodes while another node holds it.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-11-13 09:30:36 +01:00
Mattia Eleuteri 9e861c1178 fix(seaweedfs): stop the master crashing on the crowded volume map
SeaweedFS 4.45 updates the crowded-volume map of a volume layout under
the read lock, while Assign reads that map under the same read lock to
count writable volumes. Under a steady write load the Go runtime aborts
the master with "concurrent map read and map write", a fatal error that
recover cannot catch, and the S3 gateway answers 500 until a master is
elected again. Taking the write lock for the update, as every other
mutation of the map already does, removes the race.

Upstream fixed it in seaweedfs/seaweedfs#11216, released in 4.46. The
patch carries that change and its regression test unchanged, so the
build runs the test and the patch stops applying, and is dropped, once
the vendored release carries the fix.

Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Assisted-by: LLM
2026-11-13 09:19:25 +01:00
Aleksei Sviridkin c1652a730d fix(linstor): update LINSTOR server to v1.35.3
piraeus-operator v2.12.1 ships LINSTOR 1.35.3 as its controller and
satellite version, and the in-repo piraeus-server build has to match it.

LINSTOR 1.34 raised the minimum Java to 21, so the build and runtime
stages move from bookworm to trixie, and the system Gradle used by
debian/rules follows the upstream wrapper to 8.14.2.

Six carried patches are part of upstream by v1.35.3 and are dropped;
the stale-bitmap adjust retry is rebased onto the new DrbdAdm.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 11:05:37 +03:00
Aleksei Sviridkin e1ed8c7719 fix(linstor): update linstor-csi to v1.13.2
piraeus-operator v2.12.1 is released against LINSTOR CSI v1.13.2, and
the release line carries IO QoS limits through the qos.linbit.com/*
StorageClass parameters.

The operator gives the CSI pods their client certificate as mounted
files through LS_USER_CERTIFICATE_FILE and LS_USER_KEY_FILE, instead
of inline values in LS_USER_CERTIFICATE and LS_USER_KEY. The
linstor-wait-until v0.3.1 in this image is built on a golinstor
release that reads only the inline variables, so its init containers
present no client certificate, the LINSTOR API closes the TLS
connection, and they retry on EOF forever. Move to
linstor-wait-until v1.0.1, the version upstream v1.13.2 pins.

The Protocol C override for dual attach is dropped. Since v1.12 the
driver no longer sets allow-two-primaries itself: it asks LINSTOR to
manage dual-primary through make-available, and LINSTOR 1.35 sets
protocol C on the connections that need it. Upstream closed the
equivalent change as superseded for that reason
(piraeusdatastore/linstor-csi#435).

The relocate-after-clone patch is carried forward. Volume IDs became
a struct for consistency groups; Clone and VolFromSnap handle
standalone resources, so relocation keys on the resource name. The
enumer output is regenerated rather than hand-patched.

Upstream now pins grpc v1.84.0, which builds on Go 1.27 and includes
the fix for GO-2026-6443, so the grpc override is no longer needed.

The golden-image clone guard in vm-disk was re-checked: parameter
parsing for allowRemoteVolumeAccess and layerList, and the default
derived from the layer list, are unchanged in this range.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 11:05:36 +03:00
Aleksei Sviridkin acfae30136 fix(linstor): update piraeus-operator to v2.12.1
The nfs-server DaemonSet runs piraeus-csi-nfs-server v1.10.3, which
boots systemd as PID 1 in a privileged container with the host /dev
mounted. Its getty units hang up the host's getty@tty1 about a second
after the pod starts; systemd restarts the host getty until it hits
the start limit and gives up, so login on the local or out-of-band
console stops working on nodes that run the NFS server. Seen on
bare-metal Ubuntu 24.04 nodes with Cozystack 1.6.5 and 1.7.0.

Upstream fixed the image by disabling the services the NFS server does
not need (piraeusdatastore/linstor-csi#417, reported as
piraeusdatastore/piraeus-operator#961); v1.10.6 includes the fix.

v2.12.1 is the current stable release. Its image set is LINSTOR
1.35.3 and LINSTOR CSI v1.13.2, which this repository builds itself;
both builds follow in the next commits. The CSI release line is what
carries IO QoS limits (qos.linbit.com/* StorageClass parameters,
added in v1.11.3), which operators have asked for.

Behaviour that changes with this range:

- The CSIDriver moves to fsGroupPolicy: File, so fsGroup is applied
  to ReadWriteMany filesystem volumes too.
- The operator manages Secrets only in its own namespace, and the CSI
  controller loses its cluster-wide read on Secrets; nothing in this
  repository references snapshotter Secrets.
- The csi-health-monitor sidecar is removed.
- Volume group snapshots are enabled only when the
  groupsnapshot.storage.k8s.io/v1 API is served; the snapshot
  controller here still ships v1alpha1, so they stay off.

The csi-provisioner arguments are unchanged in this range, so the
restated strictTopology list stays complete.

The chart is vendored the way `make update` does it, which also drops
the stale CRD copy from the vendored chart; piraeus-operator-crds
installs the CRDs. Declare the local chart dependency so Helm can
validate the umbrella chart without changing its rendered resources.

This commit is not deployable on its own. The operator now hands the
CSI pods their client certificate through LS_USER_CERTIFICATE_FILE and
LS_USER_KEY_FILE, which the linstor-wait-until in the current
linstor-csi image does not read, so the CSI init containers retry
forever until the linstor-csi update in the next commit.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 11:05:36 +03:00
Aleksei Sviridkin 6297ebd708 fix(linstor): run LINSTOR init containers on the platform server image
piraeus-operator fills in its own default LINSTOR server image for every
container this package leaves unset, and two of those are init
containers: run-migration in the controller pod and
setup-lvm-configuration in the satellite pods. Only the main containers
were pinned to the platform build.

run-migration upgrades the LINSTOR database in place. When the operator
default is newer than the platform build, it migrates the schema ahead
of the controller, the controller then fails to read the migrated
database and crash-loops, and the migration cannot be rolled back. This
stayed hidden while the operator default was older than the version
built here; an operator update that moves the default past it exposes
it during the window between the operator and linstor releases
reconciling.

Pin both init containers to the same image as the main containers, so
the database moves only together with the controller.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 11:05:35 +03:00