Commit Graph
4316 Commits
Author SHA1 Message Date
Aleksei Sviridkin 889bb79236 feat(harbor): restrict the portal and API to listed source ranges (#4300)
## What this PR does

Adds `managementWhitelist` to the Harbor app, a list of IP addresses or
CIDR ranges allowed to reach the web portal and the Harbor API, and I
made it a typed list on purpose. The obvious alternative, a free-form
`ingress.annotations` map, gives tenants a way into the shared ingress
controller. The platform ingress-nginx runs with
`allowSnippetAnnotations: true` (the kubernetes app needs a
`configuration-snippet`), and v1.11.5 defaults `annotations-risk-level`
to `Critical`, so it accepts any annotation, `server-snippet` included.
Tenants can't create Ingress objects themselves, but they can write any
field of an app spec. An annotation map on the Harbor spec would let any
tenant put arbitrary nginx config into a controller that serves every
tenant. A typed list is also what the repo already does for this:
`filer.whitelist` in `extra/seaweedfs`, `whitelist` in `extra/ingress`
and `tcp-balancer`.

The use case is a registry that serves customers: pulls and pushes come
from anywhere, the admin UI and API should answer only to an office or
VPN range. Right now the portal, `/api/v2.0` and `/v2/` sit on one
Ingress and one public host, and a tenant can't restrict any of them.

With a non-empty list the existing Ingress gets
`nginx.ingress.kubernetes.io/whitelist-source-range`, and `/v2` and
`/service/token` move to a second Ingress, `<release>-registry-ingress`,
on the same host. ingress-nginx applies that annotation only to the
locations of the Ingress that carries it, so those two stay open. They
are what a registry client needs: Harbor answers an anonymous `/v2/`
request with a Bearer challenge whose realm is `/service/token` on the
same host. Everything else under `/service/` is
`/service/notifications/*`, internal job callbacks that the bundled
nginx already answers with 404, so it stays on the restricted side. The
second Ingress uses the same TLS secret and has no cert-manager
annotations, otherwise ingress-shim gets two owners for one Certificate.
The HTTP-01 solver is its own Ingress and is not restricted. With the
default empty list the chart renders exactly what it renders today.

An HTTPRoute can't express a source restriction, so in Gateway API mode
a non-empty list fails the render. Dropping it silently would leave the
portal public while the spec says it is restricted.

Each entry has to be an IP address or CIDR range, and a bad one fails
the render. ingress-nginx reads an empty `whitelist-source-range` as no
restriction and turns an unparseable one into a 503 for the whole
location, and the values schema generator can't validate list items.

Three things users should know. Harbor-to-Harbor replication that pulls
from this registry goes through its API, so the other registry's address
has to be on the list. The check uses the client address that
ingress-nginx sees: with `externalTrafficPolicy: Local` or PROXY
protocol that is the real client, behind another proxy it is the proxy.
And for the portal and API the list replaces the tenant ingress
`whitelist` rather than narrowing it.

Disabling the portal is not here. The vendored Harbor chart (1.19.1)
always renders the portal Deployment and its internal nginx always
proxies `/` to it, so there is no small switch to wire.

The new Ingress is added to the dashboard Role and to the ingress
selector in the ApplicationDefinition.

### Screenshots

No UI code change. The field reaches the dashboard form through the
regenerated schema and `keysOrder`.

### Downstream repositories

The terraform provider models Harbor by hand, so the new field needs a
follow-up there: cozystack/terraform-provider-cozystack#46. The website
reference page is generated from `README.md` on release. Nothing else in
the trigger map is touched: no package added or renamed, no platform,
installer or `hack/` change.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
https://github.com/cozystack/terraform-provider-cozystack/issues/46
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(harbor): add `managementWhitelist`, a list of IP addresses or CIDR ranges allowed to reach the Harbor web portal and API. Only `/v2/` and `/service/token` stay reachable from any address. The default is an empty list, which keeps the current behaviour. Not supported with Gateway API publishing: a non-empty list fails the render there. Each entry must be an IP address or CIDR range; for the portal and API the list replaces the tenant ingress `whitelist`.
```
2026-10-01 17:39:56 +03:00
Aleksei Sviridkin 54159736bc feat(talos)!: bump Talos to v1.14.2 (#4598)
## What this PR does

Bumps the Talos image Cozystack builds from v1.13.6 to v1.14.2 to get a
newer DRBD. v1.13.6 ships DRBD 9.3.2, v1.13.7 and later and
v1.14.0/v1.14.1 ship 9.3.3, and v1.14.2 is the first release with 9.3.4.
ZFS moves from 2.4.3 to 2.4.4.

On 9.3.2 I've seen a peer reboot leave the DRBD sender thread spinning
on a failed send (EPIPE). What followed was RCU stalls, processes stuck
in D-state, a hanging `drbdsetup disconnect` and two-phase-commit
timeouts. The [9.3.4
ChangeLog](https://github.com/LINBIT/drbd/blob/drbd-9.3.4/ChangeLog) has
fixes for a sender thread pinning a CPU while its connection is down,
for two-phase-commit issues and a hanging `drbdadm disconnect`, and for
two nodes ending UpToDate with different data after a reconnect. None of
them are in 9.3.3. I matched the trace to those entries by their
descriptions only, not against the commits.

The base installer image had to change. Talos v1.14 no longer publishes
`ghcr.io/siderolabs/installer`, so `gen-profiles.sh` now uses
`ghcr.io/siderolabs/installer-base`, same as upstream's own v1.14 imager
profiles. I built the amd64 installer from the new profile with
`imager:v1.14.2` locally and it builds without errors.

PR CI will not boot the new image. It builds the installer and matchbox
images, but the e2e runs on the upstream `talos:v1.13.5` container from
`hack/e2e-compose.yaml`. The first run that boots the new kernel, DRBD
and ZFS is the nightly after merge, which builds the nocloud disk from
these profiles and runs it in QEMU.

On Talos v1.14.2 the `drbd_transport_tcp` module is no longer loaded on
demand. After upgrading a node from v1.13.6, `drbd` was loaded and the
transport module was not, `drbdsetup new-peer` failed with "Failed to
create transport (drbd_transport_xxx module missing?)", and every DRBD
resource on the node stayed in Connecting. Listing the module in
`machine.kernel.modules` fixed it without a reboot. The cause is
[siderolabs/pkgs#1565](https://github.com/siderolabs/pkgs/pull/1565), in
Talos since v1.14.0: the kernel is built with an empty modprobe path, so
it no longer loads modules on `request_module()`, which is how DRBD asks
for its transport. Any module that used to load that way now has to be
listed explicitly (reported upstream as siderolabs/talos#14501). Outside
DRBD this includes `dm-thin-pool` and `dm-multipath`, which the v1.14.2
kernel builds as modules, so LVM-thin or multipath users on Talos need
them listed too. The e2e node config now loads `drbd_transport_tcp`
explicitly, and the talm preset (cozystack/talm#252) and the install
docs (cozystack/website#720) get the same line. The module ships in the
drbd extension on v1.13 too, so the extra line is safe before the
upgrade.

The order matters for host nodes. Either update the cozystack preset in
the talm project to a version with cozystack/talm#252 and re-render and
apply the node config, or add `drbd_transport_tcp` to
`machine.kernel.modules` by hand. Only then upgrade the node to v1.14.2.
A node upgraded first loses DRBD replication until the module is added.

The host network policy now also denies port 2383 to `world`. Talos
v1.14 moved etcd's `/metrics`, `/health` and gRPC-gateway JSON API from
2379 to a separate listener on 2383, and the policy only blocked 2379
and 2380. Without this, the etcd JSON API on v1.14 control-plane nodes
would be reachable from outside, protected only by client mTLS.

Two smaller changes come with the bump. The multus
`talos-cni-plugins-checked-against` marker moves to v1.14.2, because the
pkgs release behind Talos v1.14.2 still pins CNI plugins v1.9.1, same as
multus. The system memory limits doc now describes the v1.14 OOM
trigger, since v1.14.0 dropped its global memory PSI clause
([siderolabs/talos#13895](https://github.com/siderolabs/talos/pull/13895)).

Before upgrading hosts, operators should know a few things from the
[v1.14.0 release
notes](https://github.com/siderolabs/talos/releases/tag/v1.14.0) and the
Talos compatibility code:

- Talos v1.14 accepts a host upgrade from v1.12.0 or later. It supports
Kubernetes 1.32 through 1.37, while v1.13 still accepted 1.31, so a
cluster on Kubernetes 1.31 has to upgrade Kubernetes first.
- etcd's HTTP endpoints (`/metrics`, `/health`) moved from port 2379 to
2383. The Cozystack etcd scrape proxy reads a separate metrics listener
on `127.0.0.1:2381`, which this change does not move. Anything else that
scraped or health-checked etcd on 2379 has to switch to 2383, and a
firewall outside the cluster that blocked 2379 should block 2383 too.
- etcd and kube-apiserver now require TLS 1.3, so a client that can only
speak TLS 1.2 stops connecting.
- Workload isolation (`SecurityProfileConfig`) stays off on upgraded
clusters, but `talosctl gen config` turns it on for new ones. I haven't
tested Cozystack with it enabled.
- `talosctl apply-config --mode=reboot` is gone.

### Screenshots

Not a UI change.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#720
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [x] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
cozystack/talm#252
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The website PR refreshes the `next` version pins for v1.14.2 and adds
the module to the Talos install pages. The talm PR adds it to the
cozystack preset.

### Release note

```release-note
feat(talos): Bump Talos to v1.14.2, which ships DRBD 9.3.4 and ZFS 2.4.4. DRBD 9.3.4 fixes a sender thread spinning on a dead connection, two-phase-commit hangs and a data divergence after a reconnect. Before upgrading a node, add `drbd_transport_tcp` to its Talos `machine.kernel.modules` next to `drbd`: Talos v1.14.2 no longer loads it on demand, and without it DRBD cannot connect to any peer. We recommend skipping the v1.13 line and upgrading nodes straight to v1.14.2. Talos accepts a direct upgrade from v1.12.0 or later. Talos v1.14 moves etcd's HTTP metrics and health endpoints from port 2379 to 2383, and the Cozystack host network policy now blocks 2383 from outside the cluster.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Updates**
* Upgraded host node support to Talos v1.14.2 across machine images,
with refreshed firmware and storage extensions.
* Updated memory-pressure guidance for Talos v1.14, including
version-specific behavior and configuration recommendations.
  * Added protection for Talos v1.14’s etcd HTTP endpoint on port 2383.
* Updated cluster preparation to load the TCP transport module required
for DRBD.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-01 14:03:30 +03:00
Aleksei Sviridkin de570a3806 feat(platform): add slim installation variants (#4595)
## What this PR does

This adds three minimal platform variants for small installs, such as
arm64 labs, where isp-full is too heavy. They are `isp-slim` for Talos,
`isp-slim-generic` for k3s, kubeadm or RKE2, and `isp-hosted-slim` for
clusters where the host provides CNI and storage.

A slim variant installs only the base platform: the engine, API and
dashboard, tenants, ingress and Gateway API. The two non-hosted variants
also get LINSTOR. objectstorage-controller stays on too, because
cozystack-controller watches BucketClaim unconditionally and crash-loops
without that CRD. Everything else is opt-in through
`bundles.enabledPackages`, including every paas and naas application and
its operator, monitoring, backups, etcd, SeaweedFS, metrics-server, VPA,
Multus and MetalLB. The iaas bundle is refused, so there are no VMs and
no managed Kubernetes. With the stock preset a slim variant emits 20
Packages (15 for hosted), where isp-full emits 77.

On the two non-hosted slim variants, networking is Cilium alone. Without
VMs nothing needs Kube-OVN, and Cilium L2 announcements take the place
of MetalLB. The admin creates a `CiliumLoadBalancerIPPool` and a
`CiliumL2AnnouncementPolicy`, or uses `publishing.externalIPs`. Pod
ranges come from `node.spec.podCIDR`, which Talos and k3s allocate by
default. `networking.encryption` only drives Kube-OVN IPsec, so the slim
variants refuse it instead of silently ignoring it.

The tenant application needs its own slim variant. The default one waits
on the monitoring, etcd and SeaweedFS applications, so on slim
cozystack-basics and tenant-root would never become ready. tenant-rd and
cozystack-basics still reference the default variant's artifacts by
name. Those artifacts exist anyway, because the PackageSource builds
them for every variant.

The full variants don't change. I rendered isp-full, isp-full-generic
and isp-hosted against main and got byte-identical Packages. The only
new output there is the slim variant in the tenant-application
PackageSource.

Things to know before using it:

- An opt-in package does not pull in its dependencies. The presets and
the docs list the chains. A package enabled without its chain stays
`DependenciesNotReady`.
- Slim is for new installs. Switching a live isp-full cluster to
isp-slim moves the networking Package from `kubeovn-cilium` to `cilium`,
the operator removes the Kube-OVN release, and running pods lose
networking. The other Packages are kept by `helm.sh/resource-policy:
keep`, so the switch doesn't make the cluster smaller either.
- No e2e suite runs a slim variant yet. Helm unit tests cover the
presets, opt-in, OIDC, the networking values and both refusals, and a
dependency check over every emitted Package passes for all three
presets.

I installed isp-slim from a build of this branch on a fresh three-node
Talos 1.13 cluster (amd64). All Packages and HelmReleases went Ready,
with no Kube-OVN, MetalLB, monitoring, backup or KubeVirt pods. From
outside, the dashboard and the API answered by name with valid
certificates. A LoadBalancer Service got an address from a Cilium pool
and answered through L2 announcements. Pod-to-pod traffic across nodes,
cluster DNS and a replicated LINSTOR volume worked. Opting in
postgres-operator and postgres-application gave a Ready Postgres. That
run found the BucketClaim crash-loop above, which is fixed here. The
base platform used about 215m CPU and 3.2 GiB of memory, not counting
the Kubernetes control plane.

### Screenshots

Not a UI change.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#718
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [x]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up: cozystack/ansible-cozystack#79
- [x] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
cozystack/ccp#23
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(platform): add the isp-slim, isp-slim-generic and isp-hosted-slim variants. They install only the base platform (engine, dashboard, tenants, ingress, gateway, and LINSTOR outside hosted) with Cilium-only networking and no MetalLB; applications, operators, monitoring, backups, etcd and SeaweedFS are opt-in through bundles.enabledPackages, and the iaas bundle is not available.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added the `isp-slim`, `isp-slim-generic`, and `isp-hosted-slim`
platform variants, with support for PaaS and NaaS bundles.
* Slim variants use Cilium networking and include a reduced set of
packages by default. Additional packages and their dependencies can be
enabled as needed.
* **Limitations**
* IaaS and the platform encryption toggle are not supported on slim
variants. Cilium-native encryption is not configured.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-01 14:03:22 +03:00
Aleksei Sviridkin 312409ecd2 test(e2e): fail the suite whose pods outlive its cleanup (#4492)
## What this PR does

A Chainsaw suite could pass while the workloads it created kept running
after its own cleanup, and nothing noticed until a later suite failed.
On one E2E run the brokers of four Kafka releases from suites that had
passed and deleted their applications (#4488), plus a TenantControlPlane
sitting on its finalizer with two apiserver pods (#4489), pushed pod
memory requests to the node limit. Harbor then failed on `Insufficient
memory` two suites later, under the wrong name.

This makes the suite that leaked fail, and name what it left behind.

- Chainsaw v0.2.15 has no hook that runs after a test's cleanup: its
`Configuration` carries `error.catch`, which runs only on failure. So
`make test-chainsaw` now runs `hack/e2e-chainsaw-run.sh`, which invokes
`chainsaw test` once per suite directory and checks the cluster between
invocations with `hack/e2e-chainsaw/_lib/leak-guard.sh`.
- Attribution: a pod is charged to a suite when all three hold. It is
not terminal. Its UID was not in the pod list taken before the suite.
The root of its ownerReferences chain was created at or after the suite
started. A platform pod replaced mid-suite by a rollout, an eviction or
a crash traces back to an object the install created, so it is not
charged. A pre-existing pod adopted by a controller recreated mid-suite
is excluded by the UID baseline. A leak is charged once, to its own
suite.
- The suite start is read off an apiserver, from the `creationTimestamp`
of a server-side dry-run create, rather than off the sandbox. With
several control-plane nodes the reading and a root's timestamp can come
from different apiservers, so the rule assumes their clocks agree more
closely than the gap between the reading and the suite's first object;
no skew margin is added. Source for that behaviour: the generic registry
`Store.create` stamps it through `rest.FillObjectMetaSystemFields`, then
`DryRunnableStorage.Create` copies it back without persisting
(`staging/src/k8s.io/apiserver/pkg/registry/generic/registry/store.go`
and `dryrun.go` at Kubernetes v1.33.0).
- Budget: charged pods get 300s to disappear, the default
`timeouts.cleanup` in `.chainsaw.yaml`; a unit test keeps the two equal.
A suite that overrides its own cleanup timeout (kafka sets 8m) is
covered the same way, because the check runs only after `chainsaw test`
has returned, which is after every cleanup in the suite has finished
under its own timeout; the 300s start from there. The leaks this targets
sat about 110 minutes (Kafka brokers) and 30+ minutes (the
TenantControlPlane) past their suites. Every green suite logs
`leak-guard: suite=<dir> clear after Ns`, so the number can be checked
against later runs.
- Past the budget the run prints `LEAKED-PODS suite=<dir> budget=300s
count=N`, then one line per pod with its phase, deletion state, root and
`app.kubernetes.io/instance`. It raises a `leaked-pods` error annotation
and adds a failed `leak-guard` test case to `chainsaw-report.xml`. The
run continues through the remaining suites and exits non-zero at the
end, as the single invocation did on a red test.
- A suite that already failed on its own is checked once without the
wait. Its leftovers are reported as a `::warning` with `LEAKED-PODS
suite=<dir> checked-once=chainsaw-failed`, since some may still be
tearing down.
- Each pod gets one `kubectl wait` inside the remaining budget, and
nothing is retried. A timeout or a spent budget gives the verdict from
the last good pod list, also when the re-list after it fails, so a leak
fails the suite. A wait that fails only after running for its whole
`--timeout` counts as a timeout whatever it printed: kubectl maps an
expired context to "timed out waiting for the condition" only inside the
watch, so a deadline that passes during its initial lookup surfaces as
"context deadline exceeded" (`wait.go` and `delete.go` in kubectl
v1.33.0). A spent budget is caught before calling kubectl and never
passed as `--timeout=0`, which means "check once".
- Any other error on a wait or a re-list, before the budget is spent and
before any timeout, stops the waiting and is reported as `LEAKED-PODS
suite=<dir> count=N inconclusive="<error>"`. That is a warning, not a
leak failure, so an apiserver blip cannot become a leak red.
- A failed first reading after a suite, including a pod list that cannot
be parsed, leaves the suite unchecked. It fails with a `leak-guard`
annotation and a report case saying the cluster could not be read, not
that it leaked. Errors that are not leaks carry the annotation title
`leak-guard`, not `leaked-pods`.
- Known leaks are listed in `hack/e2e-chainsaw/_lib/leak-allowlist.txt`
as `<suite-dir> <issue-ref>`: `kubernetes-latest` and
`kubernetes-previous` against #4489 (both suites create a
TenantControlPlane through `_lib/run-kubernetes.sh`). `kafka` and
`kafka-metadata` were listed against #4488 until #4490 fixed it; those
entries are gone. An allowlisted suite prints its leak as a `::warning`
and does not fail. An entry that names a directory that is not a suite
stops the run before the first suite, so a rename cannot keep an
exemption alive. The PR fixing each issue deletes its lines.

Suite selection is unchanged. The runner runs the named directories when
there are any, and otherwise every directory directly under
`hack/e2e-chainsaw/`, each handed to Chainsaw's own discovery; it
refuses a tree with a test at its top level, where that split would stop
being exact. I verified this with Chainsaw v0.2.15's own
`discovery.DiscoverTests`, built from source at that tag. For the full
tree, for `kafka redis harbor` and for `kubernetes-latest`, the union of
the per-suite discoveries equals what a single `chainsaw test [suites]`
discovers: 57, 7 and 2 tests, identical lists. 57 matches the test count
in an earlier full run's report. A unit test pins that the runner passes
chainsaw the suites and flags it was given, in order. The order changes
too: suites used to run in Chainsaw's own order and now run one
directory at a time in alphabetical order, which moves the kubernetes
suites after the kafka ones. Before #4490 fixed #4488 this starved them
(see the E2E criterion below). With the fix merged the order has no
effect, and a future leak that lands before a memory-heavy suite is
named at the suite that leaked. One behaviour does change: a selected
name that is not a directory used to fail the single invocation before
any test ran; now only that suite fails and the others run.

`chainsaw-report.xml` keeps its path and shape: one `<testsuites>` root
with the summed counts, one `<testsuite>` per suite directory. Its
consumers are the `Collect chainsaw report` / `Upload chainsaw report`
steps in `pull-requests.yaml`, `e2e-fork.yaml`, `nightly.yaml` and
`e2e-tag.yaml`, which `docker cp` the file and upload it as an artifact.
Nothing in the repository parses it. The merge is unit-tested on two
per-suite reports in Chainsaw's real shape, one of them failed, and the
result is checked to parse as XML.

Tests: `hack/chainsaw-leak-guard_test.bats` (31 tests, in `make
unit-tests`; named without the `e2e-` prefix, which the unit lane
excludes) covers the attribution rule, the budget, the allowlist, the
merge and the runner, all against stub `kubectl` and `chainsaw`. Every
rule was mutation-checked: removing the guard call, having the Makefile
bypass the runner, inverting the allowlist match, flipping or tightening
the root-timestamp comparison, removing the UID baseline, removing the
terminal-phase filter, skipping the budget wait, changing the budget,
lowering or raising the depth cap, accepting a stale allowlist entry,
reading an unreadable pod list as empty, dropping the leak case from the
report, skipping directories in the selection, treating any failed owner
read as a missing owner, dropping the separate report for a suite that
already failed, dropping the not-run reason from the report, reading any
wait error as a timeout, removing the inconclusive branch, letting a
failed re-list after a timeout turn a leak inconclusive, ignoring a
failed re-list, reporting an unreadable cluster as a leak, reading an
unparsable pod list as clean, reporting a kubectl warning instead of the
error, passing the report name without its extension, wording a failed
re-list as a failed wait, dropping an allowlist's last line when it has
no newline, dropping the check that `--set-string` carries a value, and
reading a wait that ran its whole timeout but printed a context error as
inconclusive. Each one turns its named test red, and restoring turns it
green.

E2E criterion, registered before the run: in the `Run E2E tests` log of
this PR, every suite that ran prints `leak-guard: suite=<dir> clear
after`, a `LEAKED-PODS suite=<dir>` line carrying `allowlisted=`, or one
carrying `checked-once=` (a suite that failed on its own) or
`inconclusive=` (waiting stopped on a failed read), and no line matches
`LEAKED-PODS suite=.* budget=`. With #4490 merged, `kafka` and
`kafka-metadata` are no longer allowlisted and must print `leak-guard:
suite=kafka clear after` and `leak-guard: suite=kafka-metadata clear
after`, which is the live check that #4490 fixed #4488. The first run of
this PR (before #4490, run 36223896793) is the positive control: it
printed `LEAKED-PODS suite=kafka count=15` and `suite=kafka-metadata
count=8`, allowlisted, and every other suite cleared in 0-37s. That run
also went red on both kubernetes suites, because those 23 leaked brokers
left no memory for the second tenant worker VM (`0/3 nodes are
available: 3 Insufficient memory`), which the order change described
above made possible. The guard has not yet run against a real cluster,
so the first run of this PR should carry the `full-e2e` label; its
`clear after` lines are the first measured teardown times, and any
unexpected `LEAKED-PODS` line is either a real leak or a false positive
to understand before merge.

Follow-ups, known and out of scope here:

- The allowlist exempts a whole suite, so a new leak in an allowlisted
suite stays hidden until the issue's fix removes the line.
- Attribution is per suite directory, not per Chainsaw `Test`; the leak
lines carry the release instance, which usually names the test.
- Only pods are checked. A leaked PVC, LoadBalancer IP or bucket holds
no node memory and is not seen.
- The 300s budget rests on Chainsaw's cleanup timeout, not on a measured
pod-gone latency; the `clear after` lines are where that measurement
will come from.
- A platform object recreated mid-suite (for example a HelmRelease
uninstall/reinstall remediation, or a force-replace on an immutable
field) has a root newer than the suite start, and its pods would be
charged to whichever suite was running as a leak; the first CI runs are
where to look for this.
- The per-pod wait recomputes the remaining budget before each pod, and
no unit test pins that.
- `kubernetes-previous` is allowlisted for #4489 although the stuck
finalizer was seen only in `kubernetes-latest`; both suites create a
TenantControlPlane the same way, and the exemption hides any new leak
there until #4489 is fixed.
- Each clean suite now waits for its own pods to finish terminating
before the next starts, where the tail used to overlap the next suite.
The first full run's `clear after` lines say what that costs against the
E2E job's 215-minute cap.
- After a timeout followed by a failed re-list, the leak lines come from
the list taken before the wait, so a pod whose own wait had already
succeeded is still named and the count can be too high; the verdict is
unaffected.
- Likewise, when every wait succeeded and the re-list then fails, the
inconclusive report still lists the pods whose deletion was just
confirmed. In the boundary case where the budget runs out during the
last successful wait and the re-list then fails, the result is
inconclusive rather than a verdict from the last good list, because the
guard only treats the budget as spent when it finds it exhausted at a
check.
- A mistyped `COZY_LEAK_ALLOWLIST` path passes allowlist validation; it
fails closed, since no suite is then exempt.
- An allowlisted suite is not waited on, so `kubernetes-latest` and
`kubernetes-previous` will print `LEAKED-PODS ... allowlisted=` on most
runs even for pods still in normal teardown; waiting out the budget
first and applying the allowlist only to what remains would stop that
noise.
- A slow teardown in each of several suites can add up to 300s apiece,
against the E2E job's 215-minute cap. On the first full run of this PR
(run 36223896793) the `clear after` values of the suites that were
waited on summed to 210s; the last full run before this change took 158
of the 215 minutes.
- A pod whose owner's resource type is no longer served gets "the server
doesn't have a resource type" rather than NotFound, so its suite is
reported unchecked instead of leaked; it still fails.
- `kubectl wait --for=delete` waits for deletion, while the guard skips
pods in `Succeeded` or `Failed`; a pod that finishes but is never
deleted (a Job under a leaked root) spends the rest of the budget before
the suite shows `clear`. The verdict is right; only job time is lost.
- The unit tests do not pin the one-line empty report Chainsaw writes
for a directory with no tests (`_lib`), nor the branch that catches a
spent budget before calling `kubectl wait`; both are verified against
the Chainsaw and kubectl sources rather than by a test.
- The runner's per-suite report directory from `mktemp -d` is not
removed; the sandbox is discarded after the job.
- Each non-suite directory under `hack/e2e-chainsaw/` (today `_lib`)
costs one empty Chainsaw invocation, because the runner leaves the
decision of what is a test to Chainsaw.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff adds files under `hack/`, changes the `test-chainsaw` target,
and rewords one comment and the failure line in the four E2E workflows.
The trigger map's `hack/` entry is for cozystack/ccp, whose skills
anchor on `hack/package.mk`, `hack/common-envs.mk` and `make generate`,
none of which change here. A GitHub code search for `chainsaw` across
the cozystack organisation finds hits only in this repository.

### Release note

```release-note
test(e2e): the E2E app suite now fails the suite whose pods outlive its own cleanup by more than 300s, naming each leftover pod and its owner; suites with a filed leak are listed in hack/e2e-chainsaw/_lib/leak-allowlist.txt and only warn.
```
2026-10-01 14:03:13 +03:00
scooby87 031ded71ee feat(seaweedfs): alert before an instance stops taking writes (#4573)
## What this PR does

A SeaweedFS instance that runs out of room stops taking writes without
anything paging. A volume server below its `minFreeSpacePercent` reserve
turns every volume it holds read-only; before that, it can run out of
slots, and a bucket whose volumes are all full then cannot get a new
one, even with half the disk free. Backups archiving into the instance
fail in turn, and a Postgres cluster doing continuous WAL archiving
keeps the segments on its own volume until that fills too.

The generic `KubePersistentVolumeFillingUp` alert does not catch this:
it pages below 3% free, while a volume server stalls at its 5% reserve
or, when slots run out, with the disk still half empty; and its early
warning needs the disk to keep filling, which stops once writes fail.

This PR adds per-instance rules to the SeaweedFS system chart, on
signals the servers already publish and the platform already scrapes,
rendered under the same `monitoring.enabled` gate as the chart's
ServiceMonitors:

- `SeaweedFSVolumeServerDiskLow` (warning): a volume server has less
than 10% of its disk left above the reserve.
- `SeaweedFSVolumeServerFull` (warning): a volume server cannot host
another volume, because it is at the reserve or has no slot left.
- `SeaweedFSWritesFailing` (critical): the master has failed to place a
write in every 5-minute window for 15 minutes. Writes may still land in
between; the usual cause is a bucket with no writable volume left that
the master cannot grow. A new bucket's first writes fail a few picks
while its volumes grow, which the 15-minute hold does not reach unless
new buckets keep arriving.

Every expression is pinned to the release namespace, because the root
vmalert evaluates each instance's rules against every namespace's
metrics.

It also adds `docs/operations/seaweedfs-capacity-for-backups.md`: how to
size an instance for continuous WAL archiving, what the rules mean, and
how to grow an instance.

The expressions were validated live rather than by unit tests, since the
repository has no rule-evaluation tooling: on a test cluster, a
standalone SeaweedFS 4.45 on a 1 GiB volume was scraped by the
platform's vmagent and the rules rendered from this chart were evaluated
by the root vmalert. All three loaded healthy and each reached `firing`:
`SeaweedFSVolumeServerDiskLow` with 5.7% of the disk left above the
reserve, `SeaweedFSVolumeServerFull` both at the reserve and on slot
exhaustion, which stopped one bucket at 50% disk use, and
`SeaweedFSWritesFailing` under continuous failing writes, resolving once
they stopped. A new bucket's first successful writes on a free disk did
fail a few picks, which is why that rule holds for 15 minutes. On the
healthy platform instance all three stay quiet. helm-unittest pins what
a render can break: the namespace selector of each rule, the shared
label set of the two arms of the full-server rule, the gate, the
critical rule's hold, and that alert templating reaches vmalert
unrendered; each assertion was checked to fail when its line is mutated.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff touches a system chart template, its test and a document under
`docs/operations/`; no app values, platform values,
ApplicationDefinition, tooling or `hack/` path from the trigger map.

### Release note

```release-note
feat(seaweedfs): alert when a SeaweedFS volume server nears its free-space reserve, cannot host another volume, or keeps failing to place writes, and document how to size an instance for continuous WAL archiving.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added namespace-scoped SeaweedFS monitoring alerts for low or
exhausted disk capacity, unavailable volume slots, and write-placement
failures. Capacity alerts use a 10% disk threshold, and alerts trigger
after 15 minutes; write failures are marked critical.
- **Documentation**
- Added guidance on estimating SeaweedFS storage needs for Postgres
backups, interpreting capacity alerts, and expanding storage when
supported by the StorageClass.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-01 10:29:35 +03:00
Aleksei Sviridkin c983b5c404 fix(linstor): use severity warning instead of warn in piraeus-datastore alerts (#3799)
## What this PR does

Seven of the piraeus-datastore alert rules label themselves `severity:
warn`. Alerta, which the monitoring package ships as the alert sink,
validates that value and `warn` is not on its list. Every webhook
delivery for these alerts dies with a 500.

```
Severity (warn) is not one of security, critical, major, minor, warning,
indeterminate, informational, normal, ok, cleared, debug, trace, unknown
```

Alertmanager retries seven times and drops the notification, and the
alert never shows up anywhere. The affected rules are the LINSTOR/DRBD
ones, such as storage pool at capacity, lost quorum, unintentional
diskless and stuck resync. These are exactly the alerts you least want
silently dropped.

Found this on a v1.4.6 cluster where `linstorStoragePoolAtCapacity` was
genuinely firing. The cluster had 153 dropped notifications in 40
minutes and nothing in Alerta. Renaming the label to `warning`, which
the other three rules in the same file already use, stopped the failures
immediately and the alerts came through.

The same labels sit on `main` and in `v1.6.1`, so a backport to
`release-1.6` would be appreciated.

The test commit adds a bats guard so the next one gets caught in CI. It
walks every `PrometheusRule` and `VMRule` under `packages/` outside
vendored `charts/`, helm templates included, and fails on a severity
outside Alerta's default alarm model, which is exactly where `warn`,
`info`, `error` and `fatal` fall. Helm actions are stripped before the
file is read, so a rule behind a `{{- if }}` or an unquoted `{{
.Release.Namespace }}` is checked rather than skipped, and a file that
still does not parse fails the contract instead of thinning it out. On
the current tree that is 194 alerting rules in 41 scanned files.

The guard found one more on `main`. `DatabaseAutoscalerAtMax` in
`packages/system/keda/templates/alerts.yaml` carries `severity: info`,
which Alerta refuses the same way. It is renamed to `informational`, the
value the model has for that level and the one seven other rules in the
tree already use.

Moving that alert off `info` also stops `InfoInhibitor` in
`packages/system/monitoring-agents/alerts/general.rules.yaml` from
firing. That rule matches any `severity="info"` alert and carries
`severity: major`, so on `main` it usually reached Alerta whenever
`DatabaseAutoscalerAtMax` fired, even though the keda alert itself was
refused. After this change no rule carries `info`, so operators no
longer see that `InfoInhibitor` notification.

Fixes #2411.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff. It touches severity labels
inside `packages/system/piraeus-operator/alerts/` and
`packages/system/keda/templates/`, plus a bats file under `hack/`. No
package is added or renamed, and there is no values schema, CRD, tooling
the satellites gate on or node contract change. Nothing downstream
restates these labels.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(linstor): LINSTOR/DRBD alerts carried severity `warn`, which Alerta rejects, so they were never delivered. They now use the standard `warning` severity, and the keda at-max autoscaler alert moves from `info` to `informational` for the same reason.
```
2026-09-30 21:44:57 +03:00
Aleksei Sviridkin 2c2f2c194e fix(dashboard, foundationdb): show the connection ConfigMap to tenants (#4612)
## What this PR does

Fixes #4611.

Since #4148 a tenant may read the `<release>-config` ConfigMap that
holds the FoundationDB cluster file, but the console has no view for
ConfigMaps, so the connection string was still reachable only with
kubectl.

Chart:

- `<release>-resourcemap` is rendered regardless of
`monitoring.enabled`, because the console now reads it;
- the tenant Role grants `get`, `list` and `watch` on
`<release>-resourcemap` next to `<release>-config`, still by
`resourceNames`, with no wider ConfigMap access.

Console:

- the application page gets a ConfigMaps tab when the application's
resource map lists at least one ConfigMap in the same namespace;
- each listed ConfigMap is read and watched with a `metadata.name` field
selector, which RBAC checks against `resourceNames`, so a tenant never
needs `list` on all ConfigMaps;
- values are shown as plain text with a copy button, the cluster file
needs no decoding;
- a resource map that is missing or forbidden hides the tab, a listed
ConfigMap that cannot be read shows the API error instead of looking
empty, and a malformed map or an invalid name is reported.

The tab is generic, but today only FoundationDB ships a resource map, so
it is the only application that gets it.

#3490 keeps `monitoring.enabled` as the switch for the resource map and
asks whether the map should render unconditionally. This PR answers that
for FoundationDB by rendering it always, so the wording in #3490 would
need to follow if both land.

Tests: `helm unittest` for the chart (6 tests), `tsc --noEmit`, `vitest
run` (438 tests in 60 files) and `eslint` on the changed files pass. The
new tests cover reading only the listed names in the active namespace,
the watch, copying, a clipboard failure, a read denial, and malformed
maps. The chart renders with `monitoring.enabled` true and false.

### Screenshots

The ConfigMaps tab of a FoundationDB application, rendered by the
console from this branch against a local stub API with synthetic data:

![ConfigMaps
tab](https://raw.githubusercontent.com/yankawai/cozystack/29ae3c94e539edc05b29700788a9b875b5a13d54/foundationdb-configmaps-tab.png)

### Downstream repositories

Walked the trigger map against the diff: it changes a FoundationDB chart
template and its test, and the console. No `values.yaml`,
`values.schema.json`, README, CRD or `hack/` file is touched. The
website FoundationDB page is generated from the chart README and says
nothing about connecting, and the Terraform provider does not read these
ConfigMaps. The console is vendored in this repository.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(dashboard, foundationdb): the application page shows the FoundationDB connection ConfigMap with the cluster file to tenants who may read it
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Application details now include a ConfigMaps tab when configuration
maps are available. View their values and copy them to the clipboard.
  * The tab displays loading, empty, unavailable, and error states.
* **Bug Fixes**
* The resource map remains available when monitoring is disabled, and
its access permissions are included.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-30 21:23:05 +03:00
scooby87 51155f1003 feat(postgres-operator): alert when the WAL archive backlog does not drain (#4572)
## What this PR does

`LastFailedArchiveTime` compares a cluster's last failed WAL archival
with its last successful one, so it resolves as soon as any segment gets
through. Archiving that fails intermittently, or runs slower than the
cluster writes WAL, keeps clearing that alert while the queue of
segments waiting on the Postgres volume grows, until the volume is full
and the database stops. The alert is also only a warning.

This PR adds `CNPGWALArchiveBacklog`, a critical alert that fires when a
primary's `archive_status` queue has not been empty once in 30 minutes.
A healthy archiver empties the queue within seconds of each WAL switch,
so a queue that never drains means point-in-time recovery is falling
behind, whatever the archiver reports. Idle clusters and replicas have
an empty queue and stay quiet.

Validated live, since the repository has no rule-evaluation tooling: on
a test cluster, two CNPG clusters archived into an object store that
refused connections and into one that accepted them and never answered,
while writing WAL continuously. The rule, evaluated by the root vmalert,
fired for both. Over the previous seven days of the same cluster's
metrics, none of its healthy clusters would have matched; those clusters
carry light load, so a cluster whose archiver runs near its limit all
the time has not been observed and would page, which is the intended
reading of a queue that never drains. The unit test pins only that the
rule ships in the rendered PrometheusRule as critical, since the rules
reach the cluster through a file glob that drops a moved file silently.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff touches an operator chart's alert rules and its test; no path
from the trigger map.

### Release note

```release-note
feat(postgres-operator): page when a Postgres cluster's WAL archive queue has not drained for 30 minutes, which the existing archiving alert misses while some archives still succeed.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added a critical alert for sustained PostgreSQL WAL archive backlogs.
It triggers when the minimum observed ready archive status remains above
zero over a 30-minute window, then stays pending for 10 minutes before
firing. The alert describes potential impacts on point-in-time recovery
and storage usage.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-30 21:21:22 +03:00
Aleksei Sviridkin 97fa2a619f feat(dashboard): show a connection hostname for LoadBalancer services
Users who provision a service through the dashboard had no hostname to
connect to: the Services tab showed only the raw LoadBalancer IP, while
the pre-1.4.0 UI composed <service>.<root host> for every Service. Render
<service>.<suffix> under the External IP of every LoadBalancer Service
(also while the IP is still Pending), and nothing for ClusterIP Services.

The suffix is config.json serviceDomain, set by the platform value
publishing.serviceDomain. There is deliberately no fallback: the app
charts fix their external name as <release>.<tenant host> while the
LoadBalancer Services are named <release>-external-write and similar,
so a name derived from the tenant apex would match neither the DNS nor
the certificate. Without a configured suffix nothing is rendered.

Runtime config previously reached only the Shell as a prop; it is now
published through AppConfigContext and read from there by the Shell and
the Services tab alike, so a single test on the rendered version pins
the wiring.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-30 18:17:38 +03:00
Aleksei Sviridkin 67adfbce3c feat(dashboard): surface service-domain in the console config.json
The console loads config.json once at startup and treats a missing
serviceDomain key as "show no hostname", so the chart copies
_cluster.service-domain into config.json only when it is non-empty rather
than shipping an empty string the SPA would have to special-case. The
helm-unittest suite pins the unset, empty and set cases.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-30 18:17:37 +03:00
Aleksei Sviridkin d463e00da1 feat(platform): add publishing.serviceDomain for the dashboard Services tab
The dashboard has no way to show a connection hostname for a LoadBalancer
Service: before 1.4.0 the old UI composed one from the Service name and
the cluster's root host, and operators whose DNS publishes LoadBalancer
IPs under a dedicated zone still need that. Add the suffix as a platform
value and propagate it into the cozystack-values map as service-domain,
next to cluster-domain and branding, so the dashboard chart can read it.

The value is display-only and opt-in: empty (the default) means the
console shows no hostname, because nothing in the platform publishes DNS
for a guessed name. Leading and trailing dots are stripped here, the one
place every consumer reads from, so an operator's ".svc.example.org."
cannot render as "demo..svc.example.org."; the value is coerced to a
string first because an unquoted numeric scalar would otherwise abort
the whole platform render. The helm-unittest suite pins the empty
default, a configured value landing in the Secret, the dot trimming and
the numeric case.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-30 18:17:37 +03:00
Aleksei Sviridkin 0bc3f9f25c fix(kubernetes): name TalosConfigTemplate by content hash (#3523)
## What this PR does

Fixes #3515.

`TalosConfigTemplate.spec` is immutable — the
`vtalosconfigtemplate.cluster.x-k8s.io` webhook denies any mutation with
`TalosConfigTemplate.Spec is immutable` — but the `talos-reconcile` Job
applied it under a fixed `<release>-<group>` name on every reconcile.
Once the template existed, **no change to the rendered worker machine
config could ever reach the cluster**: the apply was rejected
server-side while the HelmRelease stayed `Ready`. Reported with
`machine.network.searchDomains` after a `clusterDomain` correction; the
same held for `certSANs`, `machine.registries` and nameservers.

- Name the `TalosConfigTemplate` `<release>-<group>-<6-char hash of its
spec>`, mirroring the `KubevirtMachineTemplate` content-hash idiom the
chart already uses, and point
`MachineDeployment.spec.template.spec.bootstrap.configRef` at that name.
A content change now creates a **new** template instead of attempting a
rejected mutation, and CAPI rolls the workers onto it.
- Extract the machine config into a single named template
(`_talosconfigtemplate.tpl`) shared by the two consumers that have to
agree on the name byte-for-byte: the Job, which embeds it in the heredoc
it `kubectl apply`s, and the `MachineDeployment`, which hashes it into
`configRef`.
- Assemble the node group once, in `kubernetes-nodes.group`, and read it
back at both call sites. Sharing the rendering was not enough: each file
also built its own `$group` dict, so the two names agreed only by
coincidence of the keys the spec helper happens to read.
- The runtime values the Job substitutes (`${TALOS_CA_B64}`,
`${SVC_IP}`, `${COREDNS_IP}`, tokens) stay shell placeholders and are
deliberately **not** part of the hash: they are facts of an
already-provisioned cluster, not chart inputs, so a certificate rotation
or a Service re-address does not roll workers.

### What the shared spec carries

Worker node pools live in the `kubernetes-nodes` chart, which owns
`_talosconfigtemplate.tpl` outright: `make update` copies nothing from
the parent chart.

The helper renders everything `main` puts into the worker machine
config, so the hashed spec is the one the cluster gets:

- The heredoc escaping from #3513: backslash, dollar and backtick
escaping on `talosVersion`, `installerRepository` and `schematicID`, the
escaped `machine.registries.mirrors` block, and the `INVARIANT` comment
above `data:`, which belongs in that file since it is where the next
field will be added. The spec is still emitted from an unquoted heredoc
in a host-cluster Pod, so an unescaped copy would be command injection.
- The per-pool install image. `_talosconfigtemplate.tpl` calls
`kubernetes-nodes.resolveOsImage` itself instead of reading
`talos.schematicID` and `talos.version`, so a pool on `osImage.builtin`
or `osImage.factory` reinstalls the schematic and version it actually
booted. The install image is part of the hashed spec, so the hash has to
be taken over the resolved image, or an `osImage` change would render a
different config under an unchanged name.
- The Proxmox branches: `nameservers` from `proxmox.dnsServers` where a
KubeVirt pool points at the cluster's CoreDNS, and the kubelet's
`cloud-provider: external`, without which the Proxmox
cloud-controller-manager never initialises the Node.
- `machine.kernel.modules`, emitted only when the pool resolves to a
non-empty list.

`kubernetes-nodes.group` carries every key those read, `podCpuLimit`,
`podCpuRequest`, `osImage`, `proxmox` and `kernelModules` included. A
key missing there is the failure this PR exists to prevent: with
`kernelModules` read at one call site only, an explicit module list
hashes to two different names and the `MachineDeployment` points at a
template nothing creates.

The `kubectl apply` stays `--server-side --force-conflicts` (#4163), so
the Job is not OOMKilled on the merge path; with a content-hash name the
apply is normally a create or a no-op, and server-side apply keeps the
no-op cheap too.

### The replace-on-immutable path and the Role

The runtime values stay outside the hash, so one failure is left that
the hash cannot absorb: a new apiserver address renders a different spec
under an unchanged name, and the webhook rejects it. The replace path
from #4089 handles exactly that case and is kept, with its `kubectl
delete` pointed at `${TCT_NAME}`.

The talos-reconcile Role pins its TalosConfigTemplate `get`, `patch`,
`update` and `delete` rights to the pool's own template, so it names the
hashed template too. A server-side apply of a template that does not
exist yet is still a PATCH on its name: a rule left on
`<release>-<group>` would leave the Job Forbidden and the pool with no
template to scale onto, with nothing turning red. The name-agreement
guard below therefore covers the Role as well.

## Upgrade impact — please read

**Existing clusters roll their Talos workers once.** The `configRef`
moves from `<release>-<group>` to the hashed name; CAPI creates a new
`MachineSet` for it and rolls machines onto it under the existing
`RollingUpdate` strategy (`maxSurge: maxReplicas`, `maxUnavailable: 1`)
plus the group's `MachineHealthCheck`. The rendered machine config
itself is byte-identical to `main` for the same values, so this
particular roll delivers no config change — it is the one-time cost of
adopting the content-hash name, the same cost `KubevirtMachineTemplate`
already paid. Proxmox pools roll the same way: the
`ProxmoxMachineTemplate` name does not move, but the `configRef` does.

**Until the new template exists, the pool cannot scale.** The Helm
upgrade points the `MachineDeployment`'s `configRef` at the hashed name
before the `talos-reconcile` Job has created that template, and there is
no render-time ordering fix, since the template needs values Helm does
not have. Existing workers are untouched in that window, but the pool is
frozen: in CAPI v1.10.1, which this repository ships,
`getTemplatesAndSetOwner` records the missing bootstrap template, and
`rolloutRolling` then fails in `getAllMachineSetsAndSyncRevision` with
`cannot create a new MachineSet when templates do not exist` and returns
before it scales either the new or the old `MachineSet`. A replica
change from the cluster-autoscaler therefore reaches no `MachineSet`:
scale-up and scale-down both wait. On an upgraded cluster the Job's
inputs already exist, so the window is normally a few seconds.

**A failed `talos-reconcile` Job now blocks autoscaling, and nothing
turns red.** After `backoffLimit: 30` the Job is not retried until the
next Helm upgrade of the pool release, and the HelmRelease stays `Ready`
meanwhile, as it did in #3515. Before this PR the same failure left the
pool on its old template and still scaling; with it, the pool cannot
scale at all until the template exists. Where to look:

- the `MachineDeployment` `kubernetes-<cluster>-<group>`: its
`ScalingUp` condition under `status.v1beta2.conditions` reads `Scaling
up from N to M replicas is blocked because TalosConfigTemplate does not
exist` (or `Scaling up would be blocked because …` while no scale-up is
pending), and the capi-controller-manager logs carry `cannot create a
new MachineSet when templates do not exist` for it;
- whether a `TalosConfigTemplate` named after its
`spec.template.spec.bootstrap.configRef.name` exists in the tenant
namespace;
- the `talos-reconcile` Job's status and Pod logs — but the Job has
`ttlSecondsAfterFinished: 600`, so a failed Job and its logs are deleted
ten minutes after it gives up. Past that, the `MachineDeployment`
condition and the missing template are the only evidence left.

The pre-existing fixed-name template is **not** deleted — the outgoing
`MachineSet` still references it — and is garbage-collected with the
`KamajiControlPlane` when the cluster is deleted. Hashed templates
therefore accumulate, one per distinct machine config. Pruning them
safely is filed as #4355 rather than widened into this PR.

**The one roll is not the only one.** Hashing the whole rendered spec
makes anything inside it a roll trigger, including two things that are
not edits to this cluster's resource. For a pool sized by
`instanceType`, the kubelet reservations come from a live `lookup` of
the `VirtualMachineClusterInstancetype`, so a later platform-side edit
to that instancetype rotates the name and rolls the pool again — and the
`KubevirtMachineTemplate` hash this mirrors does not have that property,
since it embeds the instancetype by name and never its resolved size.
And `talos.registryMirrors` is inside the hashed spec, so setting or
changing a mirror now rolls the pool rather than applying only to nodes
created later; the field's documentation said nothing about this and now
says it. The same holds on Proxmox for `proxmox.dnsServers`, and on
every substrate for `kernelModules`: both are inside the hashed spec, so
changing either rolls the pool, and the `kernelModules` documentation
now says so instead of describing the fixed name as the current state.

**The roll is not free on a node group holding a scarce resource.** From
exercising this same rollout path on a production fleet of 13 tenant
clusters, by rotating the `KubevirtMachineTemplate` hash on a GPU node
group. `maxSurge: maxReplicas` is dangerous when the resource is
constrained: with 6 GPUs and 1 free, CAPI created all 5 new machines at
once and 3 sat `Pending` on `Insufficient nvidia.com/...`, each having
already imported a 200 GiB disk. It converges, but it wastes a lot of
import. The natural reflex, `maxSurge: 0` with `maxUnavailable: 1`,
deadlocks instead: CAPI computes `maxScaledDown = available - (desired -
maxUnavailable)`, which was `4 - 4 = 0`, so it removed no old machine
while the new ones waited for exactly those machines to release their
GPUs; nothing moved until `maxUnavailable` was loosened to 2. And the
`MachineHealthCheck` amplifies both — at `maxUnhealthy: 100%` and
`nodeStartupTimeout: 10m`, machines queuing for a GPU were declared
unhealthy and remediated in a loop, three deleted and recreated while
perfectly healthy, each iteration re-importing the disk. The
per-node-group `nodeStartupTimeout` override fixes that, but the default
is shorter than a plausible queue wait. Anyone upgrading a fleet with a
GPU pool hits all three, and this PR is what starts the roll.

## Validation

- `helm unittest packages/apps/kubernetes-nodes` passes, snapshots
included. Pointing `configRef` back at the fixed name fails
`nodegroup_test.yaml`, the snapshot and the name-agreement bats;
dropping `osImage` from `kubernetes-nodes.group` fails the
`worker_image_*` suites; removing the Proxmox nameserver branch fails
`substrate_branches_test.yaml` and the Proxmox heredoc case.
- `hack/check-talosconfigtemplate-name-agreement.bats` is new. The
unittest suites pin each name against its own literal in its own file,
so a divergence shows up as one literal having moved, which reads as a
stale fixture and invites re-pasting the value. The guard renders the
chart once and compares the Job's `TCT_NAME`, the Role's
TalosConfigTemplate `resourceNames` and the `MachineDeployment`'s
`configRef.name` against each other, so the failure states the invariant
rather than a number. Asserted on a GPU-shaped pool as well as the
default one, because the default renders `gpus []`, `kubelet {}` and
`instanceType ""` as falsy and a consumer reading a subset of the group
is indistinguishable there from one reading all of it; on an explicit
`kernelModules` list, which neither of those shapes sets; and a last
case pins that the shapes hash differently, so the others cannot hold
vacuously. Verified by injecting each divergence: dropping `gpus` at one
call site, dropping `kernelModules` at one call site, and pointing the
Role back at `<release>-<group>` each fail it, the last one also failing
the Role cases in `talos_tct_immutable_test.yaml`, as does pointing the
`delete` back at the fixed name.
- `hack/cozytest.sh hack/talos-reconcile-heredoc_test.bats` passes, the
Proxmox cases included. This is the execution-level evidence that the
escaping survived the move into `_talosconfigtemplate.tpl`: it renders
with a hostile `registryMirrors` endpoint and a hostile
`installerRepository`, runs the extracted heredoc through a real shell,
and asserts the values come out literal rather than command-substituted;
hostile `schematicID` and `version` values, and `proxmox.dnsServers`
entries that are not addresses, are asserted refused at render; and the
Proxmox heredoc is asserted to emit the nameservers it was given. A
rendered-string regex cannot catch a heredoc the shell refuses to emit;
this does.
- `make generate` in `packages/apps/kubernetes-nodes` — no diff.
- The "byte-identical to `main`" claim was measured against `main` on
six pool shapes: default, GPU, an explicit `kernelModules` list,
`kernelModules: []` on a GPU pool, `osImage.factory` and Proxmox. For
each, the `TalosConfigTemplate` manifest the Job applies differs only in
its name (`${RELEASE}-${GROUP_NAME}` → `${TCT_NAME}`); the spec is
unchanged, the Job's `TCT_NAME` equals the `MachineDeployment`'s
`configRef.name`, and the `KubevirtMachineTemplate` and
`ProxmoxMachineTemplate` names do not move.
- **Not verified:** no live-cluster or e2e run (needs bare-metal nodes).
The upgrade sequence above, the scaling freeze included, comes from
reading the CAPI v1.10.1 `MachineDeployment` controller source and the
chart's existing `KubevirtMachineTemplate` precedent, not from watching
it end to end.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

Every box is left empty on purpose, including the first, because I am
not confident about one line of the map and would rather a maintainer
decided than have me tick it. The change adds no field and changes no
default: the only schema movement is the description text of
`talos.registryMirrors` and `kernelModules`, regenerated from their
`values.yaml` doc comments. Nothing else downstream restates this
chart's templates, the hash naming is internal to the chart, and no
image reference, CI behaviour or CRD shape moves. The open question is
whether `terraform-provider-cozystack` restates field descriptions as
well as field shapes; if it does, those description changes reach it and
want a follow-up.

## Release note

```release-note
fix(kubernetes): name the worker TalosConfigTemplate after a hash of its content so machine-config changes (searchDomains, certSANs, registries, nameservers) actually propagate instead of being silently rejected by the immutability webhook. Upgrading rolls existing Talos workers once, on KubeVirt and Proxmox pools alike. Until the talos-reconcile Job has created the new template, the pool cannot scale, so a failed talos-reconcile Job now blocks cluster-autoscaler scale-up and scale-down while the HelmRelease stays Ready; check the MachineDeployment's ScalingUp condition ("blocked because TalosConfigTemplate does not exist") and the Job status, which is deleted ten minutes after it fails. On a node group holding a scarce resource such as GPUs, review the MachineDeployment surge settings and the MachineHealthCheck nodeStartupTimeout first: `maxSurge: 0` can deadlock the roll and the default startup timeout can trigger spurious remediation while machines queue for the resource.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Worker Talos configuration templates now use content-hashed names for
consistent reconciliation and bootstrap references.
* Changes to worker machine configuration, including GPU and registry
mirror settings, automatically roll existing workers.

* **Documentation**
* Updated registry mirror documentation to explain that configuration
changes roll all workers in the pool.

* **Tests**
* Added coverage for hashed template names, configuration-triggered
worker rotations, and matching references across worker resources.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-30 17:43:09 +03:00
Yan Bondarenko 2fa7fea29b fix(dashboard): show ConfigMaps listed in the application resource map
FoundationDB keeps its connection string in a ConfigMap that its operator
writes, and tenants may read it, but the console had no view for
ConfigMaps. Tenant roles grant ConfigMaps by name only, so the tab takes
the names from the application's resource map and reads each one with a
metadata.name field selector, which RBAC checks against resourceNames.
An application without a readable resource map shows no tab, while a
listed ConfigMap that cannot be read is reported instead of looking
empty.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-09-30 17:29:49 +03:00
Yan Bondarenko e3ae8033e6 fix(foundationdb): let tenants read the dashboard resource map
The resource map is the only object that names the connection ConfigMap
for a client such as the dashboard. It was rendered only with monitoring
enabled, and the tenant role could not read it, so a tenant session had
no way to find the ConfigMap it is allowed to read.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-09-30 17:29:49 +03:00
Yan Bondarenko 17aaf3e038 feat(monitoring): add optional email alert delivery
Platform alerts need an email delivery path alongside Alerta without
exposing arbitrary namespace Secrets through tenant-controlled values.
SMTP credentials therefore use a fixed Secret name.

Alertmanager parses the addresses and connects to the SMTP host only
when it sends a notification, so a malformed value there drops alerts
instead of failing the release. The chart validates these fields when
it renders. Addresses must be literal dot-atom mailboxes, and a domain
literal must hold a dotted-quad IPv4 address, because IPv6 literal
syntax differs across the supported Go mail parsers. A bracketed
smarthost must be a valid IPv4 or IPv6 address, and its port must be
between 1 and 65535 without leading zeros. The repeat interval follows
the Alertmanager duration grammar and range, so a value that renders is
one Alertmanager loads.

Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
Assisted-by: LLM
2026-09-30 13:55:12 +03:00
Yan Bondarenko 0cf6e8c98b fix(keda): use severity informational instead of info in the at-max alert
Alerta validates the severity label against its alarm model, which has
no info in it, so the DatabaseAutoscalerAtMax notification is refused
the same way the piraeus warn alerts were. informational is the value
the model carries for this level, and the one the other rules in the
tree already use.

Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
Assisted-by: LLM
2026-09-30 13:53:06 +03:00
Yan Bondarenko 3ade16a380 fix(linstor): use severity warning instead of warn in piraeus-datastore alerts
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-09-30 13:53:06 +03:00
Alexey Artamonov 2c5c230435 fix(postgres-operator): count every scraped empty queue in the WAL backlog rule
The per-pod subquery stepped once a minute and took the latest sample
at each step, so with a 30-second scrape it looked at half the samples.
A queue that emptied only in a skipped sample did not count, and the
rule could page on a cluster that was keeping up. While a recreated pod
overlapped its old series, taking the maximum also hid the new
series' zero. The minimum of each raw series, then the minimum per pod
name, sees every sample and still keeps the window across a new
instance label.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-30 11:11:03 +03:00
Mattia Eleuteri 9afe7f1293 fix(kubernetes): assemble the node group once in kubernetes-nodes
The TalosConfigTemplate name is a hash of the machine-config spec, and two
templates have to agree on it: talos-reconcile-job.yaml creates the object,
nodegroup.yaml points MachineDeployment.bootstrap.configRef at it. Sharing the
rendering was not enough — each file also assembled its own $group dict, one
carrying every pool value and one only the few the spec reads, so the two
names agreed only by coincidence of the keys the spec helper happens to read.
Dropping "gpus" from the Job's dict diverged them on a GPU pool with the whole
suite still green, which is the "templates do not exist" deadlock this fix
exists to make impossible.

Move the assembly into a single kubernetes-nodes.group helper, read back with
fromYaml at both call sites, so there is one input as well as one rendering.

Cover it too, since the seam can be reintroduced. The chart's helm-unittest
suites pin each name against its own literal in its own file, which catches a
divergence but reports it as one literal having moved — indistinguishable from
a stale fixture, so re-pasting the new value is the obvious and wrong repair.
hack/check-talosconfigtemplate-name-agreement.bats renders the chart once and
compares the two values against each other, so the failure states the invariant
instead of a number. It asserts on a GPU-shaped pool as well as the default
one: the default renders gpus [], kubelet {} and instanceType "" as falsy, so a
consumer reading a subset of the group is indistinguishable there from one
reading all of it. kernelModules gets a case of its own, because it has no
default and stays unset on both of those shapes. A last case pins that the
shapes hash differently, so the others cannot hold vacuously.

The same guard compares the talos-reconcile Role's resourceNames on
TalosConfigTemplates against that name. The Job applies server-side, and
that is a PATCH on the name even when it creates the object, so a Role
naming anything else leaves the Job Forbidden and the pool without a
template, with the same silent outcome.

Document that talos.registryMirrors and kernelModules sit inside the hashed
spec, so changing either rolls the pool's existing workers instead of only
affecting nodes created later. The kernelModules text described the
fixed-name template as the present state and the content-hash name as a
future one.

No rendered output changes: the golden render snapshots are unchanged.

Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Assisted-by: LLM
2026-09-30 08:59:03 +02:00
Mattia Eleuteri 200226686e fix(kubernetes): name TalosConfigTemplate by content hash
TalosConfigTemplate.spec is immutable — vtalosconfigtemplate.cluster.x-k8s.io
denies any mutation with "TalosConfigTemplate.Spec is immutable" — but the
talos-reconcile Job applied it under a fixed <release>-<group> name on every
reconcile. Once the template existed, no change to the rendered worker machine
config could reach the cluster: the apply was rejected server-side while the
HelmRelease stayed Ready. Observed with machine.network.searchDomains after a
clusterDomain correction; the same held for certSANs, machine.registries and
nameservers.

Name the template after a 6-char hash of its rendered spec, the idiom
KubevirtMachineTemplate already uses in the same chart, and point
MachineDeployment.spec.template.spec.bootstrap.configRef at that name. A content
change now creates a new template instead of attempting a rejected mutation, and
CAPI rolls the workers onto it.

The spec is rendered from one named template shared by the Job (which embeds it
in its heredoc) and the MachineDeployment (which hashes it into configRef), so
the two cannot compute different names. The runtime values the Job substitutes
(CA certificates, Service ClusterIP, tokens) stay shell placeholders and are not
hashed, so a certificate rotation does not roll workers.

The talos-reconcile Role pins its TalosConfigTemplate get, patch, update
and delete rights to the pool's own template, so it names the hashed
template too. A server-side apply of a template that does not exist yet
is still a PATCH on that name: a rule left on <release>-<group> would
leave the Job Forbidden and the pool with no template to scale onto. The
replace-on-immutable path is kept and pointed at the same name, because
the runtime values sit outside the hash: a new apiserver address still
renders a different spec under an unchanged name.

Upgrade note: existing clusters roll their Talos workers once, because the
configRef moves from <release>-<group> to the hashed name and CAPI creates a new
MachineSet for it. The rendered machine config itself is unchanged by this
commit. The pre-existing fixed-name template is left in place — the outgoing
MachineSet still references it — and is garbage-collected with the
KamajiControlPlane when the cluster is deleted.

A pool sized by instanceType rather than by explicit resources has a
second roll trigger. Its kubelet reservations are resolved through a live
lookup of the VirtualMachineClusterInstancetype, so that object's size is
part of the hashed spec even though the chart does not own it: a
platform-side change to the instancetype rotates the name and rolls the
pool again. That is the mechanism working as intended - the reservations
really did change - but it is a roll the operator did not initiate, and
the KubevirtMachineTemplate hash this mirrors does not have the property,
since it embeds the instancetype by name and never its resolved size.

Fixes #3515

Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Assisted-by: LLM
2026-09-30 08:52:36 +02:00
Aleksei Sviridkin 4570672186 fix(cilium): deny world ingress to etcd's HTTP port on Talos v1.14
Talos v1.14 serves etcd's /metrics, /health and gRPC-gateway JSON API
on a dedicated listener on 2383 instead of the client port, and its
release notes ask for 2383 to be blocked wherever 2379 was. The host
policy denied 2379 and 2380 to world, so on v1.14 nodes the JSON KV
API would be reachable from outside behind client mTLS alone.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-30 04:23:38 +03:00
Aleksei Sviridkin e002f8d5e5 feat(talos)!: bump Talos to v1.14.2
Talos v1.14.2 ships DRBD 9.3.4 and ZFS 2.4.4. The v1.13 line tops
out at DRBD 9.3.3, which lacks the 9.3.4 fixes for a sender thread
pinning a CPU while its connection is down, for two-phase-commit
retries and hangs that leave drbdadm disconnect stuck, and for two
nodes ending UpToDate with different data after a reconnect.

Talos v1.14 no longer publishes ghcr.io/siderolabs/installer, so the
imager profiles take installer-base as the base installer, which is
also what upstream's own v1.14 imager profiles use.

BREAKING CHANGE: Talos v1.14 no longer loads kernel modules on demand,
so every node must list drbd_transport_tcp in machine.kernel.modules
before it is upgraded, or DRBD cannot connect to its peers.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-30 04:23:38 +03:00
Aleksei Sviridkin 506dc7a4a7 feat(platform): add isp-slim, isp-slim-generic and isp-hosted-slim variants
Small installs such as arm64 labs had to take the whole isp-full set:
KubeVirt, backups, monitoring, etcd, SeaweedFS and every paas/naas
operator. The slim variants keep only the engine, tenants, ingress,
gateway, LINSTOR storage and networking, and turn everything else into
opt-in packages listed in bundles.enabledPackages. Full variants render
the same Packages as before.

Networking on isp-slim and isp-slim-generic is Cilium alone: without
VMs nothing needs KubeOVN, and Cilium L2 announcements take the place of
MetalLB, which stays available as an opt-in package. KubeOVN IPsec is
what networking.encryption drives, so the slim variants refuse it
rather than ignore it. isp-hosted-slim keeps the noop datapath.

tenant-application gains a slim variant whose dependsOn drops the
monitoring, etcd and seaweedfs applications; the default variant waits
on them, which would leave cozystack-basics and tenant-root never Ready.
The iaas bundle stays limited to the isp-full variants.

objectstorage-controller stays on everywhere: cozystack-controller
watches BucketClaim unconditionally, so without that CRD its manager
never syncs its caches and crash-loops.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-30 03:12:00 +03:00
Myasnikov Daniil b8bc0dfd5f fix(platform): make mongodb-application depend on its operator
The mongodb chart renders kind: PerconaServerMongoDB from
psmdb.percona.com/v1, but cozystack.mongodb-application depended only on
networking and cozystack-engine. So mongodb-rd registers the
ApplicationDefinition as soon as the engine is up, and a tenant can
create a MongoDB before the operator exists; its HelmRelease then fails
with no matches for kind "PerconaServerMongoDB".

This is the same defect #3817 fixed for etcd.
Every other operator-backed app here already carries the edge.

It also closes a test-selection gap: without it cozystack.mongodb-operator
reaches no runnable suite, so every change to the operator escalates to
a run of all suites instead of selecting mongodb.

select-install.sh --validate reports the graph still has no cycle.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-30 02:18:28 +03:00
Aleksei Sviridkin d6b38f0a79 fix(kubernetes): fail the OIDC bootstrap Job when the tenant CA is missing (#4232)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

In `spec.oidc.mode: System`, the post-install/post-upgrade bootstrap Job
patches the chart-owned `<release>-oidc-kubeconfig` Secret with a real
kubeconfig built from the tenant admin kubeconfig's CA data. When that
CA data was empty, the script logged a WARNING and continued, then
exited 0. The hook was satisfied, the HelmRelease reported `Ready`, and
the Secret was left holding the install-time placeholder kubeconfig,
which cannot authenticate. This PR makes the Job fail (`exit 1`)
instead, so the failure surfaces as a red install and can be retried
(the Job already has `backoffLimit: 6` and `restartPolicy: OnFailure`)
rather than reported as a falsely healthy release. This is the smallest
of the three directions raised on the linked issue, and matches how the
neighbouring failure mode in the same script already behaves (a failed
`kubectl apply` of the ClusterRoleBindings already fails the Job under
`set -euo pipefail`). Behaviour for `mode: None` and `mode:
CustomConfig` is unchanged, and the common case — CA data present — runs
the identical patch commands as before.

A regression test was added to
`packages/apps/kubernetes/tests/oidc_test.yaml` asserting the rendered
Job script contains the `exit 1` on empty `CA_DATA` and no longer
contains the old warn-and-continue text. Confirmed it fails against the
pre-fix template and passes against the fix.

Validation:
- `helm unittest .` in `packages/apps/kubernetes` — 150/150 tests pass
(up from 149 before the new test).
- `helm template ... --show-only templates/oidc-rbac-job.yaml` with
representative `System`-mode values renders valid YAML; the extracted
bootstrap script passes `bash -n` and `shellcheck -s sh` introduces no
new warnings versus the pre-change script.
- `make generate` was not run/needed — this change touches neither
`values.yaml`, `values.schema.json`, `Chart.yaml`, nor `README.md`.
- Not run: a live cluster reproduction (Kamaji-issued admin kubeconfig
actually missing CA data at Job run time). The defect is provable from
the script logic itself and the added unit test demonstrates the
fail-then-pass behaviour; there is no live cluster available in this
environment.

### Screenshots

Not applicable — no UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

This change is confined to the shell logic inside
`packages/apps/kubernetes/templates/oidc-rbac-job.yaml`'s post-install
hook Job (a `Values.oidc.mode: System`-only failure path) and its
accompanying helm-unittest test. It does not add, rename or remove a
package, a `values.yaml` key, a CRD, a namespace, a release-owned object
name, or any other surface the trigger map lists, so no downstream
repository is affected.

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(kubernetes): a System-mode OIDC install no longer reports Ready while the tenant OIDC kubeconfig Secret still holds the install-time placeholder — the bootstrap Job now fails (and retries) instead of only logging a warning when the tenant cluster's CA data isn't available yet.
```

Fixes #3765


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* OIDC setup now fails with an error if the tenant admin kubeconfig is
missing certificate authority data, rather than leaving the OIDC
kubeconfig Secret at its install-time placeholder.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-30 01:50:22 +03:00
Pujitha Paladugu 4fe45eba32 fix(kubernetes): fail the OIDC bootstrap Job when the tenant CA is missing
In mode=System the post-install hook Job patches the chart-owned
<release>-oidc-kubeconfig Secret with a real kubeconfig built from the
tenant admin kubeconfig's CA data. When that CA data was empty the
script only warned and exited 0, so the hook was satisfied, the
HelmRelease reported Ready, and the Secret kept the install-time
placeholder kubeconfig, which cannot authenticate. The failure only
showed up when a user tried to use the downloaded kubeconfig.

Exit 1 instead, so the install fails visibly and is retried. The Job
already has backoffLimit: 6 and restartPolicy: OnFailure, and a failed
ClusterRoleBinding apply in the same script already fails the Job the
same way.

Assisted-by: LLM
Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
2026-09-30 00:36:44 +03:00
Marian Koreniuk e5dc086338 fix(kubernetes-nodes): replace the TalosConfigTemplate CABPT's webhook will not let us patch
CABPT's validating webhook rejects any change to TalosConfigTemplate.spec
(TalosConfigTemplate.Spec is immutable), so the worker reconcile Job's
`kubectl apply` fails the moment any input the render depends on genuinely
changes. The apiserver address is the one that moves in practice: the Job
writes it into certSANs and into the worker's endpoint, and a cluster that
gets a new address leaves the Job dead with

  TalosConfigTemplate.bootstrap.cluster.x-k8s.io "…" is invalid:
  spec: Invalid value: …: Spec is immutable

from then on, until somebody deletes the object by hand. Nothing in the
cluster says that is what has to happen.

The Job now recognises that one failure and replaces the object: delete, then
apply the same manifest. Every other apply error still stops the Job, because a
delete followed by a create that fails for the same reason — an apply rejected
for its content, say — would leave the pool with no template, unable to scale
until the render is fixed. The replace itself does not roll the pool: Machines
reference the template by name, and the name does not change. The match is on
the message for that reason, and the Role's new `delete` is bound to this pool's
own template by name, so the Job cannot reach a sibling pool's.

While the Role is being touched anyway, its other TalosConfigTemplate rights are
pinned to the same name. `get`, `patch` and `update` were unnamed, and several
clusters can share a tenant namespace: a Job that can patch a sibling pool's
template can rewrite the machineconfig that pool's next workers boot. `create`
stays unnamed because RBAC matches resourceNames only on verbs that name an
object.

The manifest is captured into a variable rather than written to a file: the
container runs with readOnlyRootFilesystem, and one variable also means the
replace applies exactly what the first attempt did.

Both applies stay server-side, as the Job's single apply already is: a
client-side merge pulls the full OpenAPI schema and runs the Job out of its
256Mi. The webhook rejects a spec change the same way under server-side apply,
so the `is immutable` match still fires.

UPGRADE NOTE. This changes the KubeVirt path as well, and the render with it:
the talos-reconcile Role gains the `delete` rule, its TalosConfigTemplate rights
are narrowed to this pool's own object, and the Job's script changes,
which moves the Job's content-hash name. Every existing worker pool therefore
gets a new reconcile Job on upgrade. On a pool whose live TalosConfigTemplate
no longer matches the current render, that Job will now delete and recreate it
instead of failing. Machines already running are unaffected — their bootstrap
data was rendered when they joined — but Machines created afterwards boot the
new machineconfig. On a pool where template and render agree, nothing happens.

Both new asserts were checked against the mutation they exist for: deleting the
`exit 1` from the `*)` branch and pointing the delete at `${RELEASE}` each turn
the matching test red. A bare `exit 1` pattern does not — the input-timeout path
near the top of the script already has one.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-29 21:09:09 +02:00
Marian Koreniuk 5be222ad75 docs(kubernetes): say the token's user needs the pool privileges too
A privilege-separated Proxmox token never exceeds its user's permissions.
Granting the README's roles to the token alone left the CSI and CCM tokens
with an empty effective set on the pool and the storage; granting the same
roles to the user made the token's own ACL the effective one. Seen on
Proxmox VE 9.2.20 while running the storage pair against a pool-scoped
user, and worth one sentence next to the table so nobody repeats it.

Assisted-by: LLM
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
2026-09-29 21:04:06 +02:00
Marian Koreniuk 36b04218d7 test(kubernetes): assert where the hypervisor token is and where it is not
Every assert here was written against the mutation it exists for, and each was
checked to turn red on it and green without: putting the controller back in the
tenant, giving the tenant a non-empty config.clusters, pointing a sidecar at the
wrong kubeconfig, restoring the blanket toleration, hardcoding the driver's
default owner id, dropping capmox mode, and letting the controller ignore
proxmox.csi.enabled.

The negations carry `any: true`. Without it a negated containsDocument passes
whether or not the document is in the render.

Two of the cases are about objects rather than values: the tenant release must
carry no valuesFrom at all, and two clusters must derive different volume owner
ids. Both are properties a reader can check against the rendered output, which
is why they are asserted that way rather than by reading the templates.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-29 21:04:05 +02:00
Marian Koreniuk 5ab36023e0 fix(kubernetes): release Proxmox volumes before the driver that frees them goes
Deleting a tenant cluster left its PVC disks on the hypervisor. The pre-delete
hook deletes every HelmRelease labelled with the cluster in Step 2, the CSI
driver among them, and nothing before that deleted the tenant's PVCs — a volume
is freed only by the driver answering DeleteVolume. capmox then destroys the
worker VMs, and Proxmox frees only what the destroyed VMID owns, which these
volumes are not. They stayed on storage with the tenant's data in them, and the
PV objects that could have named their owner went with the control plane.

Step 1b deletes the claims while the driver is still up, and it deletes the
claims before the Pods that hold them: a claim being deleted is one the
scheduler places no new Pod on, so a Pod that a StatefulSet or Deployment brings
back cannot mount it again and hold it to the backstop. The Pods go second: a
claim in use carries the kubernetes.io/pvc-protection finalizer, so it clears
only once nothing mounts it, and the Pods are going with the cluster in any
case. Then it waits for the PVs rather than the PVCs, because the claim
disappears as soon as its finalizer clears while the disk is freed only when the
PV is gone.

The Pods holding a claim are found by listing every Pod with the claims it
mounts and matching in the shell. A JSONPath filter on the claim name cannot do
it: with a Pod that mounts two claims its left side yields two values, client-go
refuses the whole expression ("can only compare one element at a time"), and no
Pod in that namespace would be found for any claim.

A kubeconfig that exists is not an apiserver that answers, and listing claims
against one that does not comes back empty — the clean case. Step 1b asks
/readyz first and sends an apiserver that stays silent through the backstop, so
the summary carries it. The probe is bounded by `timeout` rather than by
--request-timeout: a non-zero --request-timeout on an in-cluster kubectl call
diverts it to localhost:8080, which is why hack/check-hook-request-timeout.bats
refuses the flag in hook templates, and the check cannot tell this call, which
carries its own kubeconfig, from one that does not.

Best effort, and loud about it: a tenant that never came up has no apiserver to
ask, and a teardown must not wedge on that, but a timeout is reported through
the same backstop the other steps use.

The hook's Role gains one Secret by name — the tenant's admin kubeconfig, which
this step needs. Without it the step takes its no-kubeconfig branch and reports
that there is nothing to release while the volumes stay behind.

Step 5 no longer fails the whole hook where CDI is not installed. It deletes
DataVolumes, which are a KubeVirt-path component, and kubectl exits non-zero on
an unknown resource type; under set -e that failed the teardown after every
earlier step had already run, leaving the cluster half removed and the hook
reporting failure.

Verified on a throwaway tenant: a PVC bound and mounted by a running Pod, then
`helm uninstall`, and the volume freed by the driver rather than by hand.

delete_hook_test.yaml pins Step 1b and the Role's admin-kubeconfig entry on
proxmox and their absence on kubevirt, the /readyz probe, and the order of the
claim and Pod deletes.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-29 21:04:05 +02:00
Marian Koreniuk d902421f47 feat(kubernetes): deploy the Proxmox cloud-controller-manager with the cluster
On this substrate the worker kubelet runs with --cloud-provider=external, which
taints every node node.cloudprovider.kubernetes.io/uninitialized until a cloud
provider clears it. Without a cloud-controller-manager the taint is permanent:
nothing but tolerating DaemonSets schedules, and the CSI node plugin — which
reads topology.kubernetes.io/region and zone off its own Node and exits fatally
without them — never starts.

It runs in the management cluster for the same reason as the CSI controller
beside it: it authenticates to the hypervisor, and the tenant is cluster-admin
of its own cluster. Its whole job is to write on the tenant's Nodes, and it
does that over the tenant's admin kubeconfig from out of cluster, which also
means there is no tenant-side RBAC left to install.

capmox mode, not the default one. The default resolves a node to its VM by name
and writes providerID as proxmox://<node>/<vmid>, while capmox reads the CAPI
Machine and writes proxmox://<SMBIOS UUID> — the form capmox itself puts on the
Machine. With the default the Node and its Machine carry two different
identifiers for the same VM.

The tolerations are spelled out rather than a blanket `operator: Exists`. That
one also matches node.kubernetes.io/unreachable:NoExecute with no
tolerationSeconds, which stops admission adding the bounded 300s one and lets
the Pod outlive the Node it runs on.

`proxmox.insecure` defaults to false: the controllers verify the hypervisor's
certificate unless an operator turns that off, and both run here, so the value
describes what the management cluster trusts, not the tenant. The README gains
a section on what the two tokens can reach — the controller acts on whatever
VMID and volume handle the tenant's objects name — and on the per-tenant
Proxmox scope that contains it: a storage of the tenant's own and one ACL on
the resource pool proxmox.pool puts its workers in, which follows every VM
capmox creates — VM.Audit there for the CCM too, which deletes a NotReady Node
whose VM it cannot see.

ccm, csi and insecure are optional in the schema. Each has a default, and what
the substrate needs is enforced at render time — a proxmox cluster without
ccm.credentialsSecretName does not render — so a Kubernetes object written
before these fields existed stays valid, and adding them is not a breaking
change to the API.

The kubelet flag is asserted on both substrates: present on proxmox, absent on
kubevirt, where nothing would ever clear the taint it causes.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-29 21:04:05 +02:00
Marian Koreniuk 93c89c8bf6 feat(kubernetes): add the Proxmox CSI driver, with its controller in this cluster
Tenant clusters on the proxmox substrate get persistent volumes backed by the
hypervisor's storage. The driver is sergelogvinov/proxmox-csi-plugin, vendored
as a chart that reproduces byte-for-byte from `helm pull`.

It is deployed in halves, and which half goes where is the whole design. The
controller authenticates to Proxmox, so it runs HERE, in the tenant's namespace
of the management cluster, and drives the tenant over its admin kubeconfig —
the same shape templates/csi/deploy.yaml already uses for kubevirt-csi. The
tenant gets the node plugin, the CSIDriver object and the storage classes; its
DaemonSet mounts no credentials.

Putting the controller in the tenant would hand the token to the tenant. The
resource definition lists the admin kubeconfig in secrets.include, so the
tenant is cluster-admin of its own cluster and can read any Secret there, and
the chart writes the driver's config.yaml — token included — wherever it is
installed. The privileges the driver documents are granted on `/`, so that
token reaches every VM and every volume on the Proxmox cluster, not just this
tenant's.

Three consequences of the split are handled here rather than discovered later:

  - The vendored chart has no switch for its controller, so the tenant release
    sets replicaCount: 0. The Deployment object still renders; no Pod does.
  - Its Secret template is gated on a non-empty config.clusters, so an empty
    list is what keeps the credential out.
  - CSIStorageCapacity is published by the provisioner sidecar, which now runs
    here and owns those objects through a Pod that does not exist in the
    tenant. Left on, CSIDriver.storageCapacity=true and the scheduler refuses
    every Pod with "node(s) did not have enough free storage". The chart
    hardcodes that field, so a kustomize postRenderer patches it rather than
    forking the chart.

The controller follows proxmox.csi.enabled, like the tenant release: with the
driver switched off there is nothing for it to serve and no token to hold.

Volumes are owned per tenant. The driver names them vm-<controllerVmID>-pvc-…
and defaults the id to 9999; on shared storage that made every tenant's volumes
indistinguishable to an operator and to a VM-level ACL. Each cluster derives its
own from its release name.

What this does not do is make one tenant unable to reach another's volumes: a
tenant can still create a static PV naming a foreign volumeHandle. That needs a
per-tenant Proxmox scope — a dedicated storage and a VM ACL per tenant — which
is a deployment requirement rather than a chart change.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-29 21:04:05 +02:00
Marian Koreniuk fc729299f4 feat(kubernetes-nodes): put Proxmox workers into a resource pool
The privileges a Proxmox token needs — VM.Audit and VM.Config.Disk for the CSI
controller, VM.Audit and VM.GuestAgent.Audit for the cloud-controller-manager —
are granted on an ACL path, and the only path that follows every VM capmox
creates, replacements included, is a resource pool. Without one the choices are
`/`, which reaches every VM on the hypervisor, or each VMID, which has to be
granted again after every scale-up and every Machine replacement, and until it
is the new worker gets no disks and no providerID.

proxmox.pool names the pool; capmox adds each worker to it as it clones the VM
(ProxmoxMachineTemplate.spec.template.spec.pool). The pool has to exist on the
hypervisor — capmox does not create it — and every node pool of a tenant cluster
sets the same one, or a worker outside it is a worker the tenant's tokens cannot
reach. Empty adds the workers to no pool, which is what the chart did before.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-29 21:04:05 +02:00
Aleksei Sviridkin 08c344ab5f feat(vpc): expose allowSubnets per subnet (#4478)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Adds an optional `allowSubnets` list to each entry of `subnets` in the
VirtualPrivateCloud chart, rendered into `spec.allowSubnets` of the
kube-ovn `Subnet`.

Every VPC subnet is rendered `private: true`, so kube-ovn drops any
packet whose source is outside the subnet's own CIDR. A tenant who runs
a VPN or router VM on a subnet already gets the forward path through
`routes` (a static route with the VM as `nextHopIP`), but traffic
arriving from the remote network is dropped by the private ACL. Today
the only fix is an operator patching the cluster-scoped `Subnet` by
hand: the tenant cannot do it, no application value records it, and it
is lost whenever the subnet is recreated.

```yaml
spec:
  subnets:
    - name: gateway
      cidr: 172.16.1.0/24
      allowSubnets:
        - 192.168.1.0/24
  routes:
    - cidr: 192.168.1.0/24
      nextHopIP: 172.16.1.10
```

The subnet stays private; the list only widens the ingress ACL of the
subnet that declares it, inside the tenant's own VPC. An absent or empty
list renders nothing, so existing releases produce the same manifests.

The chart had no helm-unittest suite, so this adds a `test` target and
`tests/subnet_test.yaml`, covering a subnet without the field, one with
it, the field staying scoped to its own subnet, and an empty list. Two
of the four cases fail against the template on `main`.

`allowSubnets` is a kube-ovn field. #4034 teaches this chart to render
Cozyplane objects when kube-ovn is absent, so whichever of the two lands
second has to decide how the other backend treats the list.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: cozystack/terraform-provider-cozystack#61 (the provider
models `subnets` by hand in `vpc_schema.go` and `vpc_model.go`, so the
new nested field needs a schema and expand/flatten change there; an
issue rather than a PR, since it should follow this one merging)
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
feat(vpc): subnets accept an optional `allowSubnets` list of CIDRs admitted into the private subnet, so a VM on the subnet can act as a VPN or router gateway without a manual patch of the kube-ovn Subnet.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Private subnets can now allow additional source CIDR ranges, such as
remote networks routed through a gateway. This setting is optional and
can be configured for individual subnets.
* **Tests**
* Added checks for rendering subnet CIDRs, including cases where the
setting is omitted or applied to only one subnet.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 21:50:01 +03:00
Aleksei Sviridkin 3e4030d4d9 feat(kubernetes): per-pool kernel modules for Talos worker nodes (#3571)
## What this PR does

Implements gap 1 of #3563: a pool-scoped `kernelModules` field on
`KubernetesNodes`, emitted as Talos `machine.kernel.modules`.

A Talos system extension installs a kernel module but does not load it —
loading is `machine.kernel.modules`' job. The worker machine config had
no such block and no values key could add one, so a GPU pool on Talos
workers was unusable as shipped: the extension supplied the NVIDIA
module, nothing loaded it, and the failure was silent end to end. The VM
held the PCI device, the node advertised no GPU, and no component logged
an error. Found in a production 1.6.0 environment, where the worker
rollover took the 4 GPUs of a tenant cluster out of service with no
error anywhere.

### Design: three states, and no default

The shape was arbitrated in #3563 (option 3 of the three proposed
there).

- **absent** — the chart chooses. A pool holding at least one
`nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`),
`nvidia_uvm`, `nvidia_drm`, `nvidia_modeset`; any other pool gets no
`kernel` block at all.
- **non-empty list** — taken verbatim, replacing the chart's choice
outright.
- **`[]`** — explicit opt-out: no modules even on a GPU pool.

The order is not cosmetic: Talos loads the list in sequence, `nvidia`
has to come first because the other three depend on it, and `nvidia_uvm`
is what CUDA unified memory needs. This is the order validated against a
production GB202 passthrough node. `NVreg_NvLinkDisable=1` is what the
gpu-operator driver container applies on non-Talos workers; on Talos
that container has to be off, so nothing else would carry it, and
without it the driver waits forever for an NVLink fabric that the
passed-through GPU does not have.

The three states are distinguishable only because the field carries no
default — no entry in `values.yaml`, no `default` in
`values.schema.json`. A default of `[]` would collapse "absent" into
"opted out" for every pool and make the automatic set unreachable. The
generated parameter table prints `[]` in the Default column for any
array with no default; the field description says so explicitly. For the
same reason `keysOrder` does not list the field, so the dashboard
renders it at the bottom of the form.

Automatic emission is keyed on the `nvidia.com/` resource prefix, not on
the mere presence of `gpus`, so an AMD pool is not given NVIDIA modules
on a guess.

The field loads modules; it does not put them in the image. Which
extensions the image carries is the pool's schematic, so a GPU pool
points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`)
at a schematic carrying the NVIDIA extensions.

Module names and parameters reach the reconcile Job's unquoted heredoc,
so both are validated at render, and the emitted list is rebuilt from
those two fields alone: the schema does not prune unknown keys, and any
other key passed through `toYaml` would reach the shell unchecked. A
parameter must be printable ASCII with no whitespace, and must not
contain `$`, a backtick, a backslash or quotes. The ASCII half is there
because RE2's `\s` and `[:cntrl:]` are ASCII-only: without it U+2028 and
U+2029 pass, `toYaml` writes them raw, and Helm's YAML parser reads them
as line breaks that end the Job's `command` block scalar early. Inside
ASCII the guard denies what the shell expands rather than allowing a
fixed set, because NVIDIA's documented multi-value parameter is
semicolon-separated.

### Rollout, and #3523

A pool that resolves to no modules renders the machine config it
rendered before, byte for byte: `tests/render_snapshot_test.yaml` and
its committed golden snapshot pass unmodified, including the
`talos-reconcile` Job's content-hash name.

For a pool that does get modules, what happens depends on how its
`TalosConfigTemplate` is named, and the field description states both
cases:

- **On main today** the template has a fixed name and an immutable spec,
so the reconcile Job's apply of a changed machine config is refused
(#3515). Running workers keep what they booted with, and new workers
boot from the old template too. This covers the upgrade itself: every
existing pool holding an `nvidia.com/*` GPU with the field unset gets
the automatic set, and its apply is refused the same way. Only pools
created afterwards pick the modules up.
- **Once #3523 lands** the template is named by its content, so the same
change renames it and CAPI rolls the pool. On the upgrade that brings
both, every existing pool holding an `nvidia.com/*` GPU with the field
unset rolls onto the automatic set. That roll is the intended outcome —
those pools have no working driver without it — but it is a roll, and
the release note says so.

### Tests

- `packages/apps/kubernetes-nodes/tests/kernel_modules_test.yaml`, 12
cases: the NVIDIA set and its order for an `nvidia.com/*` pool, no
`kernel` block for a pool with neither GPUs nor modules, an explicit
list taken verbatim including `parameters`, an explicit list replacing
the NVIDIA default, `[]` opting out while leaving the `gpu=on` label
intact, no NVIDIA assumption for another vendor, a semicolon-separated
parameter accepted, a hostile module name and a hostile parameter each
rejected, a parameter carrying U+2028 and one carrying U+2029 each
rejected with the kernelModules message rather than a YAML parse error,
and an unvalidated extra key on an item dropped. The patterns pin
absolute indentation, because `modules:` one level out is still valid
YAML and silently loads nothing.
- `hack/talos-reconcile-heredoc_test.bats` lists `kernelModules` in its
validated arm and gains two execution-level cases: a hostile name and a
hostile parameter each refused at render, and a semicolon-separated
parameter surviving the real shell literally. Loosening the guards to
`^.+$` makes the refusal case fail.
- `helm unittest packages/apps/kubernetes-nodes`: 31 suites, 174 tests,
12 snapshots, all passing. `hack/cozytest.sh
hack/talos-reconcile-heredoc_test.bats`: 8 passed.
- `make generate` in the package, and `controller-gen object` for the
`api/apps/v1alpha1` submodule.

### Not in scope

The rest of #3563: the `nvidia-operator-validator` → device-plugin chain
that cannot validate an OS-provided driver, CDI spec generation and the
`nvidia` containerd runtime default, and the `maxSurge` /
`nodeStartupTimeout` defaults.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/issues/643
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
https://github.com/cozystack/terraform-provider-cozystack/issues/20
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

Both follow-ups are issues rather than PRs.
`terraform-provider-cozystack` is hand-written, so the field needs a
schema entry, a model entry and an expand/flatten pair on
`KubernetesNodes`; the undefaulted three-state behaviour is exactly
where that provider's habit of sending its own defaults would break
users silently, since materialising `kernel_modules = []` would disable
the automatic NVIDIA set for every GPU pool managed through Terraform.
`website` needs the three-state behaviour written down, the GPU pool
example pointed at `osImage.factory.schematicID`, and the Blackwell
constraint documented: on GB202 the schematic must carry
`siderolabs/nvidia-open-gpu-kernel-modules-production`, because with the
proprietary extension the module loads, `/dev/nvidia0` appears, and
`nvidia-smi -L` then reports `No devices found`.

The `hack/talos-reconcile-heredoc_test.bats` change adds test cases and
does not change node prerequisites, so `ansible-cozystack` and `talm`
are not reached. No other repository in the map is touched: no package,
platform component, bundle, variant or release asset is added, and
`ApplicationDefinition` semantics are unchanged.

### Release note

```release-note
feat(kubernetes): Worker node pools (`KubernetesNodes`) accept a `kernelModules` field, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a kernel module without loading it, so a GPU pool previously came up with no working driver and no way to fix it through values. Left unset, a pool holding at least one `nvidia.com/*` GPU now loads `nvidia` (with `NVreg_NvLinkDisable=1`), `nvidia_uvm`, `nvidia_drm` and `nvidia_modeset` automatically; an explicit list replaces that set, and an explicit `[]` opts out. The modules still have to be in the image: point the GPU pool's `osImage.factory.schematicID` at a schematic carrying the NVIDIA extensions, and on Blackwell (GB202) use the open kernel modules extension, not the proprietary one. Pools without GPUs and without the field render exactly what they rendered before. NOTE: while the worker `TalosConfigTemplate` keeps a fixed name (#3515), a changed module list does not reach a pool that already exists — including the automatic set on existing GPU pools — and only pools created afterwards pick it up. Once that template is named by its content (#3523), the same change rolls the pool, so existing GPU pools that leave the field unset are rolled onto the automatic set.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Worker node pools can now configure Talos kernel modules, including
optional module parameters.
* When no list is specified, NVIDIA GPU pools automatically receive
NVIDIA modules; other pools receive none. A custom list replaces this
selection, while an empty list disables modules.
* Invalid module names or parameters are rejected during configuration
rendering.
* **Documentation**
* Added guidance on module selection, parameters, image-extension
requirements, and pool rollout behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 21:43:43 +03:00
Aleksei Sviridkin 877d2ff43c fix(linstor): build piraeus-server's protobuf code on an arm64 builder (#4576)
## What this PR does

The arm64 half of a release fails to build piraeus-server.
linstor-server's build always downloads the `linux-x86_64` protoc, and
the builder stage runs on the build platform. Since the release builds
arm64 on a native arm64 runner, that protoc cannot execute there, and
`./gradlew generateJava` fails with `Syntax error: ")" unexpected`. The
first multi-arch pre-release run, v1.7.0-alpha.2, stopped on it.

A new patch to linstor-server picks the protoc build for the host's
architecture. An amd64 builder still gets the same x86_64 protoc as
before. The generated code is Java, so one native builder serves both
target platforms.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Builds on ARM-based hosts now select a compatible Protocol Buffers
compiler download, avoiding failures caused by using an x86_64-only
build. Builds on other supported hosts continue to use the x86_64
compiler. This improves compatibility across build architectures without
changing the resulting application behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 21:21:54 +03:00
Aleksei Sviridkin e56478cf2f feat(keycloak): expose organizationsEnabled realm toggle (#3031)
## What this PR does

- Adds an `organizationsEnabled` value to the `cozy-keycloak-configure`
chart (default `false`, no behaviour change).
- Maps it to `ClusterKeycloakRealm.spec.organizationsEnabled` so the
Keycloak
[Organizations](https://www.keycloak.org/docs/latest/server_admin/#_managing_organizations)
feature can be turned on declaratively for the platform realm.

### Why

The chart hardcoded the realm without exposing this field. Enabling
Organizations through the Keycloak admin console is not durable: the EDP
keycloak-operator reconciles `ClusterKeycloakRealm` by `PUT`ting the
chart-rendered spec back to the realm, so the console change is reverted
on the next reconcile (operator restart, periodic resync, or any CR
change). Verified on Keycloak 26.5.2.

### Release note

```release-note
feat(keycloak): add `organizationsEnabled` value to the keycloak-configure chart to enable the Keycloak Organizations feature on the platform realm (default: false)
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added a configuration option to enable or disable organizations in the
platform-managed Keycloak realm.
* Organizations remain disabled by default. Set the option to `true` to
enable them.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 21:19:47 +03:00
Aleksei Sviridkin 018b49929f feat(monitoring): auto-route tenant logs using namespace labels (#2195)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

A tenant that runs its own monitoring stack gets a VictoriaLogs
instance, but the node fluent-bit in `cozy-monitoring` sends every
container log to `global.target` (`tenant-root`), so that instance stays
empty. This PR routes each namespace's container logs to the
VictoriaLogs named by its `namespace.cozystack.io/monitoring` label, the
label the tenant chart already sets and the per-tenant vmagent already
selects on.

**Chart (`packages/system/monitoring-agents`).**
`templates/_tenant-log-routes.tpl` looks up the namespaces and keeps
those whose label is set and names a tenant other than `global.target`.
For those, fluent-bit gets one `rewrite_tag` filter with a `Rule` per
target (a single emitter however many tenants are routed), one `http`
output per target to `vlinsert-generic.<target>.svc:9481`, and a
per-target `modify Set tenant <target>`, because the re-tagged record
re-enters the filter chain where the `Match *` `Add tenant` cannot
overwrite the key. Unlabelled namespaces, empty labels and labels equal
to `global.target` stay on the default outputs.

**Controller (`internal/controller/tenantlogrouting`).** `lookup`
results are not part of the digest helm-controller compares, so the
chart alone would not pick up a tenant that turns monitoring on, or drop
one that is removed, until an unrelated upgrade. The new reconciler in
cozystack-controller digests the monitoring label across all namespaces
and, when the digest moves, sets `reconcile.fluxcd.io/requestedAt` and
`reconcile.fluxcd.io/forceAt` on `cozy-monitoring/monitoring-agents`,
which forces an upgrade and re-runs the lookup. It digests every
labelled namespace rather than repeating the chart's selection, so it
cannot drift from the chart; the cost is an occasional upgrade that
renders the same config. The digest it acted on is kept in its own
annotation, so a manual `flux reconcile` does not make it force another
upgrade. The existing ClusterRole already grants the verbs it needs.

**Behaviour change for the platform operator.** Routed records are
re-tagged with `keep false`: once a tenant runs its own monitoring, its
namespaces' container logs go to that tenant's VictoriaLogs and no
longer to the root one. Events and audit logs are not routed and stay on
the root store.

**Why not a per-tenant collector.** An earlier comment on this PR
proposed a fluent-bit per tenant, following the per-tenant vmagent and
the OTLP collector from #3763. That collector receives OTLP over the
network; a log collector has to tail `/var/log` through `hostPath`,
which the PodSecurity level of tenant namespaces rejects, and running it
in a privileged system namespace instead would mean one DaemonSet per
tenant tailing every node's logs. The routing therefore stays in the one
node DaemonSet, with the reconciler closing the reconcile gap.

Tests: `tests/fluent-bit_tenant_log_routing_test.yaml` mocks the
namespaces with `kubernetesProvider` and pins which get a rule and an
output, which are skipped (unlabelled, empty label, label equal to
`global.target`), and that a routed record carries the target tenant.
Three of its cases fail against the previous revision of this branch.
`tests/fluent-bit_tenant_log_routing_offline_test.yaml` pins that an
offline render routes nothing.
`internal/controller/tenantlogrouting/reconciler_test.go` covers first
sight, no-op while labels are unchanged (including after a manual
reconcile), a tenant turning monitoring on, a routed namespace being
deleted, and a missing release.

Refs #2194

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

No entry of the trigger map matches the diff: it touches
`packages/system/monitoring-agents` (values, a template helper, tests),
a new reconciler in `internal/controller`, its registration in
`cmd/cozystack-controller`, and a comment in the cozystack-controller
ClusterRole. The one open question is `cozystack/website`:
`content/en/docs/next/operations/services/monitoring/logs.md` describes
log collection in general terms and does not say where a tenant's
container logs land, so the behaviour change above could deserve a
sentence there. I have left every box empty for a maintainer to decide.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
feat(monitoring): container logs from a tenant that runs its own monitoring stack now go to that tenant's VictoriaLogs instead of the root one. The node fluent-bit routes them by the namespace.cozystack.io/monitoring label, and cozystack-controller re-renders the node agents when that label changes on any namespace.
```
2026-09-29 21:18:52 +03:00
Aleksei Sviridkin 811f030f15 fix(cozy-lib): round the sanitized CPU request up to a whole milli-CPU (#3522)
## What this PR does

Fixes #2917.

`cozy-lib.resources.sanitize` divides the CPU limit by
`cpuAllocationRatio` and rendered the quotient as a bare float. When the
ratio does not divide the value evenly the result is finer than a
milli-CPU, and the VPA admission webhook rejects the object outright:

```
admission webhook "vpa.k8s.io" denied the request:
minAllowed: CPU [62500u] must be a whole number of milli CPUs
```

`packages/extra/etcd/templates/vpa.yaml` routes its VPA
`minAllowed`/`maxAllowed` through the helper, so with
`cpuAllocationRatio: 4` its floor renders as `250m / 4 = 62.5m` and the
`etcd` HelmRelease loops `UpgradeFailed` → `RollbackFailed` forever: the
rollback target renders the same invalid value. Since v1.6.0
HelmReleases wait on each other, so that one stuck app also blocks
`tenant-root` → `cozystack-basics` → `kubevirt-cdi-operator`, and
without CDI the Talos worker rollover cannot import the worker OS image.
Only the divisors of 250 worked as ratios, which is an artefact of one
chart's hardcoded `250m` floor rather than a real constraint on the
operator.

- `packages/library/cozy-lib/templates/_resources.tpl`: round the
sanitized CPU request **up** to the next whole milli-CPU. Up, never
down, so the request stays at or above the intended fraction of the
limit. And only when rounding is actually needed: an exactly divisible
value keeps its current float rendering, so nothing changes on the
default ratio.
- The whole-milli test is exact, with no tolerance around it. sprig's
`divf`/`mulf` run on `shopspring/decimal` rather than float64, so `mulf
(divf 0.25 10.0) 1000.0` renders exactly `25` in Helm and there is no
binary-float noise on this codepath to smooth over. A tolerance would in
fact reintroduce the bug: a value that is whole-milli only to within
`5e-7` would read as exact and render verbatim, so `cpu: 100.0000004m`
at ratio 1 would render `0.1000000004`, rejected for the same reason
`62.5m` is. There is a test for that case.
- `packages/tests/cozy-lib-tests/`: new `resources` template plus a
`resources_test.yaml` suite covering the indivisible ratio, the round-up
direction, the two unchanged-rendering cases, a
whole-milli-only-within-a-tolerance value, sub-milli input at an
ordinary and at an extreme ratio, a genuinely zero request, and that
memory is untouched.
- `packages/extra/etcd/tests/vpa_test.yaml`: regression guard asserting
the chart's rendered VPA `minAllowed.cpu` is a whole milli-CPU (`63m`)
with `cpu-allocation-ratio: 4`, and that the default ratio still renders
`0.025` / `0.5`.

### Why in `sanitize` and not in the etcd chart

The issue also suggests taking `minAllowed`/`maxAllowed` out of
`sanitize` in the etcd chart, on the grounds that VPA policy bounds are
not workload requests. I looked at that first and it is the wrong fix
here:

- `minAllowed`/`maxAllowed` bound the **request** the VPA writes, and in
Cozystack every request is `limit / cpuAllocationRatio` —
`packages/extra/etcd/templates/etcd-cluster.yaml` sanitizes
`.Values.resources` the same way. On the default ratio the etcd request
is `1000m / 10 = 100m`, bracketed by `minAllowed 25m` and `maxAllowed
500m`. Dropping `sanitize` would make the floor a literal `250m`, i.e.
**10x** the intended request floor, and let the ceiling reach `5000m`.
It would also change rendered output on the default ratio, for every
existing cluster.
- It fixes one chart. The same helper feeds every app's requests, and at
`cpuAllocationRatio: 4` seven other charts (`http-cache`, `mariadb`,
`nats`, `rabbitmq`, `redis`, `tcp-balancer`, `valkey` — all from the
`250m` nano presets) also render `cpu: "0.0625"`. Those go into pod
requests rather than a VPA object so they are not rejected, but they are
still not quantities we should be emitting.
- No other chart routes `minAllowed` through `sanitize`, so there is no
consistency argument either way: the other VPA templates
(`etcd-operator`, `monitoring`, `monitoring-agents`, `vpa-for-vpa`) use
hand-picked literals in system charts where no allocation ratio applies.

### How it was validated

- `hack/helm-unit-tests.sh`, full run: 81 packages green, before and
after. The only difference between the two runs is the two new suites
appearing as `PASS`; no existing suite changed status.
- Both new suites were confirmed to **fail** against `main`'s helper (5
tests in `resources_test.yaml`, 1 in the etcd `vpa_test.yaml`), so they
guard the regression rather than merely describing the new behaviour.
- Breadth of the bug: over the 905000 combinations of a whole-milli
limit `1m`..`1000m` against an integer ratio `1`..`905`, `main` renders
a quantity finer than 1m for **898026** of them (99.2%) and this branch
for **none**.
- Rendered-output check on the default ratio: `helm template` over all
164 chart directories under `packages/{apps,extra,system,core}`, before
and after, byte-diffed. 129 render standalone; all 129 are identical
apart from 8 whose diff is only their per-render random secrets and
self-signed certs (no `cpu` line differs in any of them).
- Same sweep with `--set _cluster.cpu-allocation-ratio=4`: `main` emits
11 sub-milli `cpu: "0.0625"` requests across 7 charts, this branch emits
**0**.
- The quoted webhook error was checked against the VPA we actually ship
(1.5.1): `pkg/admission-controller/resource/vpa/handler.go` wraps the
quantity error as lowercase `minAllowed:`/`maxAllowed:`, and `62.5m`
canonicalises to `62500u`.
- Not verified: nothing from this branch was applied to a live cluster,
so the end of the chain (the webhook accepting the object, the etcd
HelmRelease converging, CDI unblocking) is inferred from the rendered
manifests and the webhook's own error message, not observed.

### Screenshots

Not applicable, no UI change.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff. The diff is one Helm library template plus test files: no package
added, renamed or removed; no `values.yaml`, `values.schema.json`,
version enum or default changed; no `ApplicationDefinition`, CRD,
namespace, `hack/` file, package layout or release asset touched. The
terraform provider restates app defaults, but no default moves here —
the rendered value only changes for a non-default `cpuAllocationRatio`,
which the provider does not model.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(cozy-lib): round the sanitized CPU request up to a whole milli-CPU. A `cpuAllocationRatio` that does not divide a chart's CPU value evenly (for example `4` against the `250m` VPA floor of the etcd chart) produced a quantity finer than 1m, which the VPA admission webhook rejects, leaving the etcd HelmRelease looping UpgradeFailed/RollbackFailed and — since v1.6.0, where HelmReleases wait on each other — blocking tenant-root, cozystack-basics and CDI behind it. Any allocation ratio now renders a valid quantity; rendering is unchanged whenever the ratio divides evenly, including on the default ratio of 10. Where a ratio does not divide a configured CPU value evenly, the rendered request changes once on upgrade: for example `cpu: 125m` at the default ratio goes from `0.0125` to `13m`, and so does every nano preset at a ratio like 4. The API server already rounds pod requests up the same way, but the string stored in operator-managed resources (CNPG clusters, KubeVirt VMs and so on) changes, and some operators may roll their pods for it.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Bug Fixes**
- CPU resource requests are now sanitized to avoid floating-point
precision issues.
- Fractional milli-CPU requests are rounded up to the nearest whole
milli-CPU, with positive sub-milli requests clamped to `1m`.
  - Exact and zero CPU values remain unchanged.
- CPU limits and memory requests continue to be preserved without
modification.

- **Documentation**
  - Updated resource guidance to describe CPU request rounding behavior.

- **Tests**
- Added coverage for allocation ratios, default bounds, rounding,
precision, and resource preservation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 21:17:44 +03:00
Aleksei Sviridkin 6ab3e0be78 feat(postgres): allow preloading an allowlist of shipped modules (#3506)
## What this PR does

- Adds `postgresql.sharedPreloadLibraries`, a closed enum of four module
names that already ship in the CloudNativePG PostgreSQL image, rendered
onto the CNPG `Cluster` field of the same meaning.
- Leaves the `shared_preload_libraries` entry in the
`postgresql.parameters` blocklist exactly as it is.
- Adds `tests/shared_preload_libraries_test.yaml`: absent by default,
single and multiple libraries, no interference with `parameters`, schema
rejection of a non-allowlisted name and of an absolute path, and the
parameter blocklist still firing when the allowlist is in use.
- Adds `hack/check-postgres-preload-allowlist.bats`, which renders the
chart with `--skip-schema-validation` to exercise the template-side
allowlist that helm-unittest never reaches, and checks that it matches
the schema enum.

`shared_preload_libraries` is blocked in `postgresql.parameters`, and
rightly so: the parameter is free text and accepts a path, so it can
point at a writable location such as the PGDATA volume. The block does
not make every preloadable module unreachable, though. CloudNativePG
carries a managed extensions list (`ManagedExtensions` in
`pkg/postgres/configuration.go`, checked on v1.30.0, the operator
version in this repo): as soon as a parameter under the
`pg_stat_statements.`, `auto_explain.` or `pgaudit.` prefix is set, the
operator adds that library to `shared_preload_libraries` itself, and for
`pg_stat_statements` and `pgaudit` it also runs `CREATE EXTENSION`. The
chart already passes such keys through, so
`postgresql.parameters."pg_stat_statements.track": all` works on main
today.

What this PR adds is narrower than a missing capability. `pg_prewarm`
has no route at all, and its autoprewarm worker only runs when the
module is preloaded. For the other three, the field is an explicit
switch that does not require setting one of their tuning parameters,
even to its default value, just to get the library loaded. The two
routes combine rather than override: the operator merges this list with
the libraries it preloads on its own, so removing a module from
`sharedPreloadLibraries` does not unload it while a parameter under its
prefix is still set. The field description says so.

The distinction this PR draws is between a library *name* and a library
*path*. Nothing needs to be injected to preload these four:
`pg_stat_statements.so`, `auto_explain.so`, `pgaudit.so` and
`pg_prewarm.so` are already in `$libdir` in the image, and that
directory is on the read-only root filesystem, so it cannot be written
to. `dynamic_library_path` stays blocked, so `$libdir` remains the only
search path. A bare name from a fixed enum therefore has no way to reach
code that is not already in the image, which is the property the blanket
block was protecting.

The enum in `values.schema.json` is the primary guard and rejects a bad
value before rendering starts. The template carries the same list as a
backstop for renders that bypass schema validation, using the existing
fail-fast `SECURITY:` idiom from the parameter blocklist a few lines
below it. helm-unittest validates the schema on every render, so it
never reaches that backstop; the bats suite renders with
`--skip-schema-validation` instead. I checked that it fails when the
template condition is replaced with `false`, and when the template list
gains a name the enum does not have.

I verified the mechanism end to end on a live CloudNativePG 1.27.3
cluster rather than assuming it: with
`spec.postgresql.shared_preload_libraries` set, the operator writes
`shared_preload_libraries = 'pg_stat_statements'` into
`postgresql.conf`, `SHOW shared_preload_libraries` reports it, and
`SELECT ... FROM pg_stat_statements` returns rows. Adding a second
module to the list took effect too, and the CNPG instance manager
restarts PostgreSQL in place for it, with no pod restart. That restart
covers every instance, the primary included, so writes pause briefly.
The CRD documents the field as additive to the operator's own defaults,
so this does not clobber anything CNPG needs to preload itself.

Scope note: the four names are the modules in the image that are useful
to a tenant and safe to preload. Anything requiring a background worker
or a shipped-but-privileged module was deliberately left out, and the
enum is the single place to extend if that changes.

### Screenshots

No UI change.

### Downstream repositories

`packages/apps/postgres/values.schema.json` gains a field, which per the
trigger map reaches `cozystack/terraform-provider-cozystack`: its
postgres resource carries a hand-maintained schema and its own
expand/flatten pair, with no codegen from `values.schema.json`, so
`sharedPreloadLibraries` will be invisible to Terraform users until it
is added there by hand. The enum matters twice over there, because the
provider restates enums as `stringvalidator.OneOf` lists. I searched
that repository and found no open PR or issue covering it (the only open
PR is a dependabot Go module bump). I have not filed the follow-up, so I
am leaving the box unticked rather than claiming work that does not
exist. The same gap applies to `users.<name>.createRole` in #3505; one
follow-up there can cover both fields.

I walked the rest of the map against the diff and found nothing else: no
package is added, renamed or removed, no existing version enum changes,
no default changes, `release.prefix` and the output Secret and Service
names are untouched, and no CRD outside the regenerated
`packages/system/postgres-rd/cozyrds/postgres.yaml` is affected. The
`README.md` parameter table is regenerated here and postgres is already
in the website generator's app list, so the website path stays
automated.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: not filed yet, see the note above
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
feat(postgres): add `postgresql.sharedPreloadLibraries`, a closed allowlist of modules shipped in the PostgreSQL image (`pg_stat_statements`, `auto_explain`, `pgaudit`, `pg_prewarm`) that are preloaded at server start. It is the first way to preload `pg_prewarm`, and lets the other three be preloaded without setting one of their parameters, which CloudNativePG already treats as a request to preload them. The `shared_preload_libraries` parameter stays blocked: only bare names from the allowlist are accepted, never a path. Changing the list restarts PostgreSQL on every instance, the primary included.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added optional PostgreSQL shared preload library configuration for
`pg_stat_statements`, `auto_explain`, `pgaudit`, and `pg_prewarm`.
* Preserves the configured library order. CloudNativePG may
automatically preload the first three libraries when their related
parameters are set; `pg_prewarm` must be enabled through this list.
* Changing the list restarts every PostgreSQL instance, including the
primary.
* **Validation**
* Rejects unsupported modules, absolute paths, and attempts to set
preload libraries through free-form parameters.
* **Documentation**
* Added configuration guidance covering supported libraries and restart
behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 21:12:04 +03:00
Aleksei Sviridkin 53829ea109 fix(kubernetes): reconcile KCCM EndpointSlices when tenant node VMIs change (#3500)
## What this PR does

Adds `patches/vmi-informer.diff` to the `kubevirt-cloud-provider` image
of the `kubernetes` package. The file is named after what it does: no
upstream pull request carries this change yet
(kubevirt/cloud-provider-kubevirt#382 covered the same desync and was
closed as rotten), so a number would point at an unrelated upstream PR
and invite someone bumping the pin to drop it.

The `kubevirteps` controller derives the infra EndpointSlices of a
LoadBalancer Service from the VMIs backing the tenant nodes
(`getDesiredEndpoints`, and the all-VMIs fallback added by `379.diff`).
But it registers informers only on infra Services, tenant EndpointSlices
and infra EndpointSlices, and both factories are built with a resync
period of `0`. VMI addresses are resolved with a one-off `Get` during a
reconcile.

Consequently a VMI appearing, disappearing or changing its address
triggers no reconciliation, and no periodic relist ever catches up. The
EndpointSlices keep pointing at VMIs that no longer exist and the
LoadBalancer blackholes the share of the inbound connections hashed onto
those dead addresses — sporadic inbound timeouts, with nothing wrong on
the tenant side.

Services whose tenant EndpointSlice churns along with its pods recover
on their own, because that churn produces an event. Cilium Gateway API
Services do not: their tenant slice is a static sentinel
(`192.192.192.192`, no `NodeName`), so nothing ever forces a recompute.
Seen in production on a cluster whose GPU MachineDeployment is rolled by
the cluster-autoscaler: 5 of the 9 endpoints of each of the 3 gateways
pointed at destroyed VMs (13 failures out of 20 requests). Worth noting
for anyone debugging this: the KCCM looks dead (no log for hours) while
its lease is still being renewed every second. The process is healthy,
it is simply never woken up.

The patch:

- adds an informer on the infra `VirtualMachineInstances` that requeues
the LoadBalancer Services of this cluster (those labelled
`cluster.x-k8s.io/cluster-name=<clusterName>`, the ones `reconcile` does
not skip) on add/delete, and on updates filtered by a fingerprint of
what actually shapes an endpoint: the VMI's cluster label,
`Status.NodeName`, `Status.Phase`, and the address of the `default`
interface;
- gives both informer factories a 5 minute resync, as a safety net if an
event is ever missed;
- scopes the all-VMIs fallback to the VMIs labelled
`cluster.x-k8s.io/cluster-name=<clusterName>`. An infra namespace can
hold several tenant clusters as well as standalone `VirtualMachines`,
and their VMIs do not serve this Service's NodePort, so selecting them
adds endpoints that blackhole traffic. The fallback widens to every VMI
only when none of them carries the label, so an environment that does
not label them does not lose all of its endpoints; once any VMI is
labelled, an empty selection means this cluster has no node right now
and is not widened to another cluster's VMIs.

No RBAC change: the KCCM role already grants `watch` on
`virtualmachineinstances`.

On `main`, the Gateway scenario reaches the all-VMIs fallback without
any other KCCM patch: Cilium 1.19.5 (shipped since v1.6.0) sets
`kubernetes.io/service-name` on the tenant slice of a `cilium-gateway-*`
Service (cilium/cilium#46242), so the existing lookup finds the sentinel
slice and falls back on its missing `NodeName`. This is why #3374, which
added that lookup to the KCCM, was closed. On v1.5.x and earlier, which
ship Cilium 1.19.3, a Gateway Service finds no tenant slice at all, so
it has no endpoints to go stale in the first place.

## Testing

- `341 + 379 + vmi-informer` applies onto the pinned `a0acf33`; the
Dockerfile steps (`go get`, `go mod tidy`, `go mod vendor`,
`-mod=vendor` build of the manager) pass, `go vet` and `gofmt` are
clean.
- The patch adds 6 specs, and the suite runs 21 of 21 green: fallback
scoped to the cluster, fallback preserved when no VMI is labelled, no
widening when only another cluster's VMIs are present, unlabelled VMIs
ignored once some are labelled, VMI add/delete requeues the Service of
this cluster and not the one of another cluster, and VMI updates driven
through the fake dynamic client: an annotation-only update does not
requeue, while a change of phase, node, address, or cluster label does.
- Mutation-tested, each one turns the suite red: always requeuing on
update; dropping the phase, the node, the address or the cluster label
from the fingerprint; requeuing every Service of the namespace; dropping
the delete requeue.
- Under `-race`, the suite reports an intermittent data race at
`kubevirteps_controller_test.go:820` (a `PrependReactor` racing an
informer list). It reproduces identically on `341 + 379` without this
patch, and the new specs pass on every `-race` run.
- Validated on a live cluster with an image built from an earlier
revision of this patch (same informer and fallback, before the
cluster-scoped requeue and the label in the fingerprint): endpoint added
in under 2s when a VMI appears, dead address removed in 33s when it is
deleted. Before the patch the dead address stayed forever.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff is one patch file of the KCCM image: it touches no values,
schema, CRD, `hack/` tooling, namespace, telemetry metric or node
prerequisite, so the trigger map comes out empty.

### Release note

```release-note
fix(kubernetes): the KCCM now reconciles the LoadBalancer EndpointSlices when the VMIs backing the tenant nodes change, instead of keeping stale addresses that blackhole part of the inbound traffic
```
2026-09-29 19:32:42 +03:00
Aleksei Sviridkin 924d28cf3d feat(kubernetes): default gpu-operator to an OS-provided driver on Talos (#3585)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->

## What this PR does

Closes the gpu-operator half of #3563 — gap 3 (the operator cannot
validate a driver it did not install) and gap 4 (nothing routes CUDA
libraries to a tenant pod).

Tenant clusters have run Talos workers since 1.6.0, where the NVIDIA
driver comes from a system extension baked into the worker image rather
than from the operator's driver container. The addon default still
described the Ubuntu shape, so a GPU node group came up with the
operator fighting the OS for the driver and no path at all for the CUDA
libraries.

Three changes, each closing one silent failure:

| Change | Without it |
|---|---|
| `driver.enabled: false`, `toolkit.enabled: false` | the driver
container loads its own module and clashes with the extension's; the
toolkit container rewrites `/etc/containerd/config.toml`, which Talos
owns and which already registers the `nvidia` runtime with `enable_cdi =
true` and `cdi_spec_dirs = ['/run/cdi']` |
| `validator.driver.env`: `DRIVER_ROOT=/`, `NVIDIA_DRIVER_ROOT=/`,
`DISABLE_DEV_CHAR_SYMLINK_CREATION=true` | the driver validator looks
for a driver-container install, finds none, never writes its sentinel,
and the device plugin stays blocked behind `toolkit-validation` — the
loop on `Attempting to validate a driver container installation` (**gap
3**) |
| `devicePlugin.env`: `DEVICE_LIST_STRATEGY=cdi-annotations`,
`NVIDIA_CTK_PATH=/usr/local/bin/nvidia-ctk`, plus `DEVICE_ID_STRATEGY`,
`NVIDIA_DRIVER_ROOT`, `CDI_ANNOTATION_PREFIX` | tenant pods carry no
`runtimeClassName` and did not need one on Ubuntu, where the toolkit
made `nvidia` the default containerd runtime. Talos does not, so CDI is
the only route: the pod gets the device and no CUDA libraries (**gap
4**) |

Every value is the configuration validated on a production Talos GPU
node group, minus the parts that only mattered while the driver
container was in play. No DaemonSet, no new image, no new package — the
fix is values.

### The removed default

The previous default fed `NVreg_NvLinkDisable=1` to the driver container
through the `nvidia-kernel-module-params` ConfigMap. With that container
off the ConfigMap is mounted by nothing, so this removes the default. On
Talos the parameter belongs to the worker machine config, which the
`kernelModules` field of the worker pool writes: that field is top-level
in the KubernetesNodes app (`packages/apps/kubernetes-nodes`), added by
#3571, and its automatic set for a pool holding an `nvidia.com/*` GPU
already loads `nvidia` with `NVreg_NvLinkDisable=1`. The wrapper chart
keeps the ConfigMap feature for an operator who re-enables the driver
container through `valuesOverride`. A test pins the absence so the dead
default cannot come back and offer a second, inert source for the same
setting.

### Relationship to #3571

Independent files, no conflict, but only useful together: this makes
`driver.enabled: false` correct, and #3571 is what actually loads the
modules the extension ships. Merged alone, this changes a GPU node group
from "operator fights the OS" to "operator stays out of the way and the
modules still are not loaded" — better, but not yet working. #3571 alone
leaves the operator fighting for the driver. Either order is safe; both
are needed for a GPU node group to work.

### What is verified, and what is not

Verified here: the rendered HelmRelease values, that `valuesOverride`
still wins over each new key, and that the removed default is gone. The
individual values are the ones running in production today, applied by
hand through `addons.gpuOperator.valuesOverride`.

Not verified here: this exact default exercised end-to-end as the
chart's own output. That needs a Talos GPU node group whose modules
load, which needs #3571 released. I would rather say so than imply a
green e2e that does not exist — happy to hold this as draft until I have
run a single-node L40S cluster against the chart default, and I will
report the result on this PR either way.

### Not covered

HAMi. When `addons.hami.enabled` the operator's device plugin stays off
and HAMi serves the GPUs; HAMi's own plugin needs equivalent CDI wiring
on Talos, which this default does not attempt. Flagging rather than
guessing — the production cluster this was derived from runs HAMi
disabled, so I have no validated configuration for that path. Worth its
own issue if a maintainer confirms the shape.

Also not in scope, and unchanged from the merge base:
`talos.installerRepository` and `talos.version` reach the worker machine
config unvalidated (being tracked separately by @lexfrei), and the
`nvidia-operator-validator` chain for the *management* cluster's own
passthrough/vgpu variants, which never load the host driver at all.

### Screenshots

No UI changes.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/issues/643
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
feat(kubernetes): The GPU Operator addon of tenant Kubernetes clusters now defaults to a driver provided by the operating system, which is the shape of the Talos workers these clusters have run since 1.6.0. `driver` and `toolkit` are disabled (the driver comes from a Talos system extension, and containerd configuration belongs to the machine config), the driver validator is pointed at the host root so it stops looping on "Attempting to validate a driver container installation", and the device plugin is switched to CDI (`DEVICE_LIST_STRATEGY=cdi-annotations`) so tenant pods receive the CUDA libraries — previously they received the GPU device and nothing else, since Talos does not make `nvidia` the default containerd runtime. The previous default, which fed `NVreg_NvLinkDisable=1` to the operator's driver container through a ConfigMap, is removed because that container is disabled; on Talos the parameter is carried by the worker machine config, through the `kernelModules` field of the worker pool in the KubernetesNodes app (`packages/apps/kubernetes-nodes`), which loads `nvidia` with `NVreg_NvLinkDisable=1` by default on a pool holding an `nvidia.com/*` GPU. Operators who overrode `addons.gpuOperator.valuesOverride` to work around this by hand can drop those overrides. NOTE: an override that sets `devicePlugin.env` replaces the list rather than merging it, so restate the full list when overriding it or CDI stops working. HAMi is not covered: when `addons.hami.enabled` the operator's device plugin stays off and HAMi's own plugin needs equivalent CDI wiring, which this change does not attempt.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Configuration**
* GPU workloads on Talos workers now use host-provided NVIDIA drivers
and tooling by default; the GPU Operator’s driver and toolkit are
disabled.
* Device discovery uses CDI annotations and UUID-based device IDs. When
HAMi is enabled, the Operator’s device plugin is disabled.
  * Host-root validation no longer creates `/dev/char` symlinks.
  * Driver kernel-module configuration is no longer created by default.

* **Documentation**
* Clarified when kernel-module configuration applies and what settings
are needed to re-enable the driver.
  * Noted that `valuesOverride` replaces lists wholesale.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 19:28:17 +03:00
Aleksei Sviridkin d466fd96b3 fix(rabbitmq): bound the topology operator Secret cache (#4130)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Fixes #4128.


`messaging-topology-operator` 1.14.2 caches **every Secret in the
cluster** — its `ClusterRole` grants `secrets: list, watch` cluster-wide
with no namespace or label restriction — and the vendored manifest
limits it to `500Mi`. On a cluster that has grown enough Secrets it
OOMKills and never recovers, because the initial `LIST` alone no longer
fits. While it is down, no topology CR reconciles for any tenant.

Measured on the affected cluster: 8 211 Secrets across 291 namespaces,
293 MB of Secret payload (87 % of it Helm release history), against 21
topology CRs and 1 `RabbitmqCluster`. Working set sat flat at 331–335 Mi
for 30+ hours, crossed 500Mi, then crashlooped 154 times. The operator
was caching 8 211 Secrets in order to read 2.

Upstream fixed exactly this, and it is not a bigger limit — their
default is still `512Mi`:

- `329aa9e4e` (2026-06-30, released in v1.20.0) gates the Secret
informer on `rabbitmq.com/topology-operator: "true"`, to bound "memory
usage on large clusters".
- `0850c3ae8` (2026-07-22, released in v1.20.1) bypasses the cache for
`Service` reads.

v1.20.2 is the current patch release carrying both; v1.19.x and earlier
carry neither. This PR refreshes the vendored manifest to v1.20.2
through the package's existing `make update` pipeline (fetch, drop the
`Namespace` object, rewrite `rabbitmq-system` →
`cozy-rabbitmq-operator`).

**The bump alone would break every install**, which is why the app chart
moves with it. The app chart's credentials Secret is referenced by
`User.spec.importCredentialsSecret`, and from 1.20 the admission webhook
rejects a user-provided Secret without the label:

```
secret %q must have label rabbitmq.com/topology-operator=true to be used by the Topology Operator
```

Verified against the 1.20.2 source, two things deliberately do **not**
change:

- The `RabbitmqCluster` `default-user` Secret needs no label —
`rabbitmqclient/cluster_reference.go` reads it through `apiReader.Get`,
uncached.
- The `Status.Binding` contract is unchanged since 1.14.2, so the
vendored `cluster-operator` 2.9.0 stays compatible and is left alone to
keep this PR focused.

The v1.20.2 manifest is self-contained: it ships its own `Issuer` and
both `Certificate` objects (webhook + metrics), and adds probes, a
`securityContext` that satisfies `restricted` PodSecurity, a defaulting
`MutatingWebhookConfiguration` and secure metrics. Same 13 CRDs, none
added or removed.

### Tests

- `packages/apps/rabbitmq/tests/user_credentials_test.yaml` renders two
users and asserts that each `User` imports its own
`<release>-<user>-credentials` Secret and that every credentials Secret
carries `rabbitmq.com/topology-operator: "true"`. Deleting the label
line from the template fails it.
- The rabbitmq e2e suite now creates its cluster with one user and
asserts `User/rabbitmq-test-app` reaches `Ready`, so the webhook and the
User reconcile run against the real 1.20.2 operator. Its `catch` gains a
describe of that `User` and the topology operator's logs.

### Upgrade path

Existing installs carry unlabeled credentials Secrets until each
tenant's `rabbitmq-*` HelmRelease re-renders. No migration is added: in
1.20.2 the User controller watches Secrets and maps an import Secret
back to its Users through the `.spec.importCredentialsSecret.name` field
index (`secretToUserRequests`), so relabelling the Secret requeues the
User immediately rather than on the next interval, and nothing is
removed from the broker in between. The import path no longer writes
`<user>-user-credentials`, and nothing in this repo reads the ones
1.14.2 left behind.

### Verification

- `make generate` in both touched packages — no diff beyond the
intentional changes
- `helm unittest packages/apps/rabbitmq` — 103/103, and the new suite
fails with the label removed

### Screenshots

No UI changes.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

Walked the trigger map against the diff, file by file.
`packages/apps/rabbitmq/templates/rabbitmq.yaml` adds one label to a
rendered Secret: no package added or renamed, no `values.schema.json`
field, no version enum, no `values.yaml` default, no `release.prefix`,
and the Secret's *name* is unchanged, so the provider's literal-name
lookups are unaffected.
`packages/system/rabbitmq-operator/templates/messaging-topology-operator.yml`
is a vendored upstream manifest for an already-present component: no
package layout change, no `cozyrds` rename, no namespace rename, no
installer or platform values key. `packages/apps/rabbitmq/tests/` and
`hack/e2e-chainsaw/rabbitmq/` are test-only and appear nowhere in the
map.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(rabbitmq): update messaging-topology-operator to 1.20.2, which bounds its Secret cache to Secrets labeled `rabbitmq.com/topology-operator=true` instead of caching every Secret in the cluster. Before this, the operator's memory grew with the cluster's total Secret count and OOMKilled against its 500Mi limit on large clusters, stopping reconciliation of all Queue, Exchange, Binding, User, Permission, Vhost and Policy resources for every tenant. The managed RabbitMQ chart now labels the credentials Secret it creates, so existing users need no action. If you manage topology resources by hand and point `connectionSecret`, `importCredentialsSecret`, `uriSecret` or `upstreamSecret` at your own Secret, add the label `rabbitmq.com/topology-operator: "true"` to it or the admission webhook will reject the resource.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added configuration options for resource cleanup, queue types,
connection and channel limits, shovel queue arguments, and federation
settings.
* Added metrics support, health checks, leader-election configuration,
and queue webhook capabilities.
* Improved compatibility of generated credential Secrets with the
RabbitMQ Topology Operator.
* **Bug Fixes**
* Improved validation and documentation for topology resources,
including requiring an exchange for topic permissions.
  * Updated operator components and Kubernetes resource configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 19:25:30 +03:00
Mattia Eleuteri 3c657a4cc1 feat(vpc): expose allowSubnets per subnet
Every VPC subnet is rendered private, so kube-ovn drops any packet
whose source lies outside the subnet's own CIDR. A tenant that runs a
VPN or router VM on a subnet and points a static route at it gets the
forward path from the existing routes value, but traffic arriving
from the remote network is then dropped by the private ACL.

Subnet is cluster-scoped, so the only workaround so far has been an
operator patching spec.allowSubnets by hand: something the tenant cannot
do, that no application value records, and that is lost whenever the
subnet is recreated. Declaring it per subnet keeps the widening in the
tenant's own VPC and scoped to the one subnet that needs it.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-09-29 18:47:18 +03:00
Mattia Eleuteri 91c87b84cd fix(kubernetes): refuse non-ASCII kernelModules parameters
RE2's \s and [:cntrl:] only cover ASCII, so the parameter deny class
passed U+2028 and U+2029. toYaml writes either one raw, and Helm's YAML
parser reads it as a line break that ends the reconcile Job's command
block scalar early. The render then failed on an unparseable manifest
rather than on the guard, which is the last check before the unquoted
heredoc.

Pinning parameters to printable ASCII closes every non-ASCII line break
at once, and the deny class keeps refusing what the heredoc expands.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-09-29 17:35:49 +02:00
Aleksei Sviridkin 6c7d22edab fix(linstor): build piraeus-server's protobuf code on an arm64 builder
linstor-server's build always downloads the linux-x86_64 protoc, and
the builder stage runs on the build platform. The release now builds
the arm64 images on a native arm64 runner, where that protoc cannot
execute and generateJava fails. A patch picks the protoc build for the
host's architecture.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-29 18:21:34 +03:00
Mattia Eleuteri 1f9b1b85db fix(kubernetes): reconcile KCCM EndpointSlices when tenant node VMIs change
The kubevirteps controller derives the infra EndpointSlices of a
LoadBalancer Service from the VMIs backing the tenant nodes, but it
watches no VMI and its informers had no resync period. A VMI appearing,
disappearing or changing its address therefore triggered no
reconciliation, and no periodic relist ever caught up: the EndpointSlices
kept pointing at VMIs that no longer existed and the LoadBalancer
blackholed the share of the inbound connections hashed onto those dead
addresses.

Services backed by a tenant EndpointSlice that churns with its pods
recovered on their own. Cilium Gateway API Services did not: their
tenant slice is a static sentinel, so nothing ever forced a recompute.
Observed in production on a cluster whose GPU MachineDeployment is
rolled by the cluster-autoscaler: 5 of the 9 endpoints of every gateway
pointed at destroyed VMs.

Add an informer on the infra VirtualMachineInstances that requeues the
LoadBalancer Services of this cluster, ignoring the updates that cannot
change an endpoint, and give the informers a 5 minute resync as a
safety net. Services of other tenant clusters sharing the namespace are
not requeued, since reconcile would skip them.

Also scope the all-VMIs fallback to the VMIs labelled with the name of
the cluster being reconciled: an infra namespace can hold several
tenant clusters and standalone VirtualMachines, whose VMIs do not serve
the Service NodePort. The fallback widens to every VMI only when none of
them carries the label, so that environments that do not label them keep
their endpoints. Once any VMI is labelled, an empty selection means this
cluster has no node right now, and widening it would publish the
addresses of another cluster.

The patch is named after what it does rather than after an upstream
pull request, because none carries this change yet.

Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Assisted-by: LLM
2026-09-29 17:21:19 +02:00
Alexey Artamonov 35e3593c85 fix(seaweedfs): describe the writes-failing alert by what it measures
The rule fires when the master fails a volume pick in every 5-minute
window for 15 minutes. That proves recurring refusals, not that no
upload gets through: writes into other buckets can succeed in between,
and a steady stream of new buckets, each failing a few picks while its
first volumes grow, produces the same pattern. The master publishes no
count of successful picks to tell the two apart, so the rule stays and
its description and the operations page now say what it has seen.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-29 18:17:01 +03:00
Alexey Artamonov e23c6af59b fix(postgres-operator): keep the WAL backlog window across a recreated pod
The scraped series carries the pod IP in its instance label, so a
recreated primary starts a new series, and min_over_time over it saw
only the samples since the restart. The rule could then page ten
minutes after the pod came back with a queue that was still draining,
not after the 30 minutes its description promises. Grouping by pod name
first keeps one series per pod across the restart, and the description
now says what happens to a pod name that is newer than the window.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-29 18:16:23 +03:00
Aleksei Sviridkin 9b56e70ca6 feat(mongodb): support the system-bucket backup flow (useSystemBucket) (#3967)
## What this PR does

MongoDB was the only `cozy-default` app with **no `useSystemBucket`
flow**: its backup storage was always static on the
`PerconaServerMongoDB` CR (bucket/endpoint/credentials from `backup.*`
values plus a chart-emitted `<release>-s3-creds` Secret), so a tenant
could not back up to the platform system bucket out of the box the way
Postgres/ClickHouse/MariaDB/etcd do. This closes that gap so the default
MongoDB backup-restore works OOTB.

PBM takes `bucket`/`endpointUrl`/`region`/`prefix` as **static CR
fields** (not secret refs), and the platform bucket name/endpoint are
only known at BackupJob time — so, unlike ClickHouse's runtime
`secretKeyRef` env, the chart cannot render them. The implementation
mirrors the CNPG driver's pattern:

- **Chart** (`packages/apps/mongodb`): a new `backup.useSystemBucket`
flag keeps `spec.backup.enabled` (so the pbm agents run) but omits the
static `storages`/`tasks`/`pitr` and the `<release>-s3-creds` Secret.
Helm never owns `spec.backup.storages`, so a Flux re-render can't fight
the driver. A `bootstrap.enabled + useSystemBucket` render guard mirrors
Postgres, and `useSystemBucket: true` with `backup.enabled: false` is
refused both at render time and by an `x-kubernetes-validations` rule on
`backup` in the schema (the same mechanism as the `storageClass`
immutability rule). Note on the schema rule: the aggregated apiserver
does not evaluate CEL on application-backed resources yet (#2657), so
today it is a UI/schema-level marker and the render-time refusal is what
stops a Flux release; it also trips the API review gate (“validation
rule added”), so this PR needs an API-owner approval on top of the code
review.
- **Strategy** (`cozy-default-mongodb`): now carries the S3 coordinates
(`endpointUrl` already scheme'd by the
`backupstrategy-controller.endpoint` helper, `credentialsSecret:
cozy-backups-creds`, `prefix: <namespace>/<application>`).
- **Driver** (`psmdbstrategy_controller.go`): SSA-injects
`spec.backup.storages[s3-storage]` onto the live `PerconaServerMongoDB`
at BackupJob time under a dedicated field manager
(`cozystack-psmdb-backup-driver`) with `ForceOwnership`. Injection is
**gated on the app's `backup.useSystemBucket` flag** (a legacy app
leaves it false and is never touched; a useSystemBucket app is
re-injected once per mint (once per BackupJob, plus once per
storage-race re-mint), before its operator CR is created, so a later
coordinate change catches up on the next job while a streaming dump is
never patched under). The backup CR is minted only once the operator
reports the storage pushed into pbm (`status.observedGeneration` caught
up and `PBMReady` True): pbm writes to what its own config says, not to
the spec, so a CR minted before that push would land at the previous
coordinates while its Backup recorded the new ones; the job waits at
`PerconaServerMongoDBPBMConfigPending` meanwhile, bounded by the backup
deadline. Once a job has its CR, none of the mint-time reads or holds
can fail it: the strategy and app reads the artifact still needs are
bounded by the 24h running ceiling instead, and the cluster is not read
at all **and on the release having rendered that flag**: the app CR is
projected from the HelmRelease's desired values, so the flag reads true
before helm-controller renders the revision, while the tenant's own
storage, scheduled tasks and PITR stream from the previous render still
sit on the cluster. As long as the cluster carries `spec.backup.tasks`
or `spec.backup.pitr`, the driver does not inject and holds the
BackupJob at `Ready=False PerconaServerMongoDBLegacyRender` (failing it
at the deadline), so a wedged legacy revision never has its nightly dump
and oplog redirected into the shared bucket. The backup snapshot falls
back to the strategy coordinates when the operator status echo is empty;
that fallback is gated on the CR's own `percona.com/delete-backup`
finalizer (the flow the CR was minted on), not on the app flag re-read
at completion, so a legacy backup never records the platform bucket and
a flag flipped mid-dump does not change the artifact. The restore path
rebuilds `backupSource` from it (the `cozy-backups-creds` reference is
projected into the restore namespace before dispatch).
- **Retention/cleanup**: on the `useSystemBucket` flow the driver owns
the shared-bucket archive (the chart's psmdb `tasks[].keep` retention is
gated out, and an on-demand backup is not covered by task retention). It
stamps the operator's `percona.com/delete-backup` finalizer on each
backup it mints **against a storage that carries the platform
coordinates** — the strategy's bucket and `credentialsSecret`, read back
from the live cluster after the injection — never on the flag alone: a
strategy without an `s3` block, or a storage that points anywhere else,
holds the job (`MongoDBStrategyHasNoS3` /
`PerconaServerMongoDBStorageForeign`) instead of minting a finalized
dump against a bucket the platform does not own. On Cozystack `Backup`
deletion (by hand today; by platform retention once it lands, since a
`Plan` only creates BackupJobs and never deletes a `Backup`) the driver
deletes the operator `PerconaServerMongoDBBackup` CR so that finalizer
frees the object — the finalizer's presence is the ownership marker
cleanup reads back. That reading is unambiguous because cleanup resolves
the CR by the name recorded in `driverMetadata` (the one the driver
minted), not by scanning: the psmdb operator does stamp
`percona.com/delete-backup` on its own `cron-*` task-minted backups, but
those are never in scope. A legacy backup keeps the pre-existing
hands-off contract, decided from the snapshot before any read of the
operator CR. A terminating namespace or the
`backups.cozystack.io/skip-artifact-cleanup: "true"` annotation strips
the driver's finalizer and releases; a `Get` of the operator CR that
fails with anything other than NotFound/NoMatchError still requeues, and
the operator's own `release-lock` finalizer can hold the CR while the
operator is down, so those two are what can keep a namespace in
`Terminating`. `schedule` and `retentionPolicy` are inert on this flow
(there is no engine-side retention and a `Plan` has no retention field
yet); platform-level retention is tracked in
[#4192](https://github.com/cozystack/cozystack/pull/4192) and teardown
GC of orphaned backups in
[#3966](https://github.com/cozystack/cozystack/issues/3966), so until
those land bound `cozy-backups` with an S3 lifecycle policy.
- **Bounds on the waits**: the injection race with the operator's
cluster cache (an `unable to get storage` error on a freshly injected
storage) is retried by deleting the errored CR and minting a fresh one,
with the retry count kept on the BackupJob
(`backups.cozystack.io/mongodb-storage-race-retries`) because the
deleted CR is reaped before the next poll; after three retries the
operator's error is terminal. A system-bucket dump that has started
streaming is not cancelled on the 30-minute deadline, but fails after 24
hours in `running` (`BackupAbandoned` warning naming the operator CR,
which is left in place as the handle to whatever pbm still writes). The
deadline cancel of a not-yet-started CR pins its delete to the
resourceVersion the live read answered for.
- **RBAC**: `patch` on `perconaservermongodbs` for the injection
(server-side apply), and `create`/`delete`/`patch` on
`perconaservermongodbbackups` — `delete` to prune the shared-bucket
archive on `Backup` deletion, `patch` to strip the driver's finalizer
when releasing a wedged cleanup.

Coverage: Go unit tests over `internal/backupcontroller` — the injection
gate keyed off the app flag (legacy cluster never patched) and off the
applied render (a cluster still carrying tasks or pitr is held,
untouched), the SSA-injection precondition with the whole entry
(per-application prefix included) asserted on the cluster, strategy
templating per application, the finalizer refused against a storage
without platform coordinates (strategy without `s3` over a legacy entry,
foreign coordinates on the merged entry), the retry budget driven with
the operator simulated reaping the deleted CR between polls, the restore
snapshot fallback and credential swap across all four legacy/system flow
combinations, the finalizer-owned cleanup with its escape hatches (skip
annotation before any read, `NoMatchError` treated as gone,
terminating-namespace release using the pre-read CR, a failed strip
surfaced as an Event, a legacy CR never deleted, a legacy snapshot
released without a read, a system-bucket snapshot reaching the prune),
the deadline cancelling a not-yet-running operator backup only on a
positive live read (including no live client at all, and a CR that moved
since the read), and the running ceiling. Plus helm-unittest on both
charts (chart skips the creds Secret and omits
`spec.backup.storages`/`tasks`/`pitr` on `useSystemBucket` and keeps
them on the legacy flow; the strategy renders the S3 coordinates and the
`insecureSkipTLSVerify` knob). A live `useSystemBucket` backup+restore
e2e against the platform bucket is a follow-up (needs the system bucket
wired in CI, same status as the Postgres/ClickHouse useSystemBucket
e2e). One item to confirm in e2e: the psmdb operator accepting
`spec.backup.enabled=true` with an empty `storages` between install and
the first BackupJob.

### Screenshots

Not applicable (no UI changes).

### Downstream repositories

The tenant-facing schema surface is one new field,
`spec.backup.useSystemBucket`, on the MongoDB `values.schema.json` / CRD
(regenerated), plus a validation rule on `spec.backup`.
`terraform-provider-cozystack` hand-models MongoDB and may want to
expose it, but it is a new optional boolean that defaults false (no
drift for existing users), and the chart `README.md` regenerates on
release. Leaving every box unticked per the trigger-map guidance so a
maintainer decides on the provider follow-up.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
feat(mongodb): MongoDB backups now support the platform system bucket via backup.useSystemBucket, so the default cozy-default BackupClass works without per-app S3 credentials — the backup driver injects the system-bucket storage onto the cluster at backup time, once the release has rendered the flag. On this flow nothing prunes the shared bucket automatically yet (a Plan carries no retention field; tracked in #4192 and #3966), so an S3 lifecycle policy on cozy-backups is a prerequisite for enabling it, not a suggestion. useSystemBucket requires backup.enabled=true; the chart refuses the combination with backups disabled at render time, and the schema carries the same rule as a marker until the apiserver enforces CEL (#2657). A system-bucket dump still running after 24 hours fails its BackupJob with the operator backup left in place. A MongoDB restore now adopts the target cluster's S3 credentials only for a storage on the same bucket as the archive (it used to adopt them unconditionally); a legacy restore into a target on a different bucket that relied on one key pair covering both must keep or re-create the source release's -s3-creds Secret by name, and is held at Ready=False RestoreCredentialsMissing until it exists.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
  * Added opt-in system bucket mode for MongoDB backups.
* Automatically injects platform bucket storage without
application-managed S3 credentials.
* Supports endpoint, region, path prefix, addressing, and TLS settings.
* Manages shared-bucket backup artifacts when system bucket backups are
deleted.

* **Bug Fixes**
* Improved restore credential selection to avoid cross-bucket
mismatches.
  * Preserved backup metadata when storage details are unavailable.

* **Documentation**
* Documented configuration, scheduling, retention, PITR limitations, and
restore guidance.

* **Validation**
* Prevented incompatible bootstrap and system bucket restore
configurations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 18:10:56 +03:00