Commit Graph
6956 Commits
Author SHA1 Message Date
Aleksei Sviridkin 08c344ab5f feat(vpc): expose allowSubnets per subnet (#4478)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Adds an optional `allowSubnets` list to each entry of `subnets` in the
VirtualPrivateCloud chart, rendered into `spec.allowSubnets` of the
kube-ovn `Subnet`.

Every VPC subnet is rendered `private: true`, so kube-ovn drops any
packet whose source is outside the subnet's own CIDR. A tenant who runs
a VPN or router VM on a subnet already gets the forward path through
`routes` (a static route with the VM as `nextHopIP`), but traffic
arriving from the remote network is dropped by the private ACL. Today
the only fix is an operator patching the cluster-scoped `Subnet` by
hand: the tenant cannot do it, no application value records it, and it
is lost whenever the subnet is recreated.

```yaml
spec:
  subnets:
    - name: gateway
      cidr: 172.16.1.0/24
      allowSubnets:
        - 192.168.1.0/24
  routes:
    - cidr: 192.168.1.0/24
      nextHopIP: 172.16.1.10
```

The subnet stays private; the list only widens the ingress ACL of the
subnet that declares it, inside the tenant's own VPC. An absent or empty
list renders nothing, so existing releases produce the same manifests.

The chart had no helm-unittest suite, so this adds a `test` target and
`tests/subnet_test.yaml`, covering a subnet without the field, one with
it, the field staying scoped to its own subnet, and an empty list. Two
of the four cases fail against the template on `main`.

`allowSubnets` is a kube-ovn field. #4034 teaches this chart to render
Cozyplane objects when kube-ovn is absent, so whichever of the two lands
second has to decide how the other backend treats the list.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: cozystack/terraform-provider-cozystack#61 (the provider
models `subnets` by hand in `vpc_schema.go` and `vpc_model.go`, so the
new nested field needs a schema and expand/flatten change there; an
issue rather than a PR, since it should follow this one merging)
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
feat(vpc): subnets accept an optional `allowSubnets` list of CIDRs admitted into the private subnet, so a VM on the subnet can act as a VPN or router gateway without a manual patch of the kube-ovn Subnet.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Private subnets can now allow additional source CIDR ranges, such as
remote networks routed through a gateway. This setting is optional and
can be configured for individual subnets.
* **Tests**
* Added checks for rendering subnet CIDRs, including cases where the
setting is omitted or applied to only one subnet.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 21:50:01 +03:00
Aleksei Sviridkin 3e4030d4d9 feat(kubernetes): per-pool kernel modules for Talos worker nodes (#3571)
## What this PR does

Implements gap 1 of #3563: a pool-scoped `kernelModules` field on
`KubernetesNodes`, emitted as Talos `machine.kernel.modules`.

A Talos system extension installs a kernel module but does not load it —
loading is `machine.kernel.modules`' job. The worker machine config had
no such block and no values key could add one, so a GPU pool on Talos
workers was unusable as shipped: the extension supplied the NVIDIA
module, nothing loaded it, and the failure was silent end to end. The VM
held the PCI device, the node advertised no GPU, and no component logged
an error. Found in a production 1.6.0 environment, where the worker
rollover took the 4 GPUs of a tenant cluster out of service with no
error anywhere.

### Design: three states, and no default

The shape was arbitrated in #3563 (option 3 of the three proposed
there).

- **absent** — the chart chooses. A pool holding at least one
`nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`),
`nvidia_uvm`, `nvidia_drm`, `nvidia_modeset`; any other pool gets no
`kernel` block at all.
- **non-empty list** — taken verbatim, replacing the chart's choice
outright.
- **`[]`** — explicit opt-out: no modules even on a GPU pool.

The order is not cosmetic: Talos loads the list in sequence, `nvidia`
has to come first because the other three depend on it, and `nvidia_uvm`
is what CUDA unified memory needs. This is the order validated against a
production GB202 passthrough node. `NVreg_NvLinkDisable=1` is what the
gpu-operator driver container applies on non-Talos workers; on Talos
that container has to be off, so nothing else would carry it, and
without it the driver waits forever for an NVLink fabric that the
passed-through GPU does not have.

The three states are distinguishable only because the field carries no
default — no entry in `values.yaml`, no `default` in
`values.schema.json`. A default of `[]` would collapse "absent" into
"opted out" for every pool and make the automatic set unreachable. The
generated parameter table prints `[]` in the Default column for any
array with no default; the field description says so explicitly. For the
same reason `keysOrder` does not list the field, so the dashboard
renders it at the bottom of the form.

Automatic emission is keyed on the `nvidia.com/` resource prefix, not on
the mere presence of `gpus`, so an AMD pool is not given NVIDIA modules
on a guess.

The field loads modules; it does not put them in the image. Which
extensions the image carries is the pool's schematic, so a GPU pool
points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`)
at a schematic carrying the NVIDIA extensions.

Module names and parameters reach the reconcile Job's unquoted heredoc,
so both are validated at render, and the emitted list is rebuilt from
those two fields alone: the schema does not prune unknown keys, and any
other key passed through `toYaml` would reach the shell unchecked. A
parameter must be printable ASCII with no whitespace, and must not
contain `$`, a backtick, a backslash or quotes. The ASCII half is there
because RE2's `\s` and `[:cntrl:]` are ASCII-only: without it U+2028 and
U+2029 pass, `toYaml` writes them raw, and Helm's YAML parser reads them
as line breaks that end the Job's `command` block scalar early. Inside
ASCII the guard denies what the shell expands rather than allowing a
fixed set, because NVIDIA's documented multi-value parameter is
semicolon-separated.

### Rollout, and #3523

A pool that resolves to no modules renders the machine config it
rendered before, byte for byte: `tests/render_snapshot_test.yaml` and
its committed golden snapshot pass unmodified, including the
`talos-reconcile` Job's content-hash name.

For a pool that does get modules, what happens depends on how its
`TalosConfigTemplate` is named, and the field description states both
cases:

- **On main today** the template has a fixed name and an immutable spec,
so the reconcile Job's apply of a changed machine config is refused
(#3515). Running workers keep what they booted with, and new workers
boot from the old template too. This covers the upgrade itself: every
existing pool holding an `nvidia.com/*` GPU with the field unset gets
the automatic set, and its apply is refused the same way. Only pools
created afterwards pick the modules up.
- **Once #3523 lands** the template is named by its content, so the same
change renames it and CAPI rolls the pool. On the upgrade that brings
both, every existing pool holding an `nvidia.com/*` GPU with the field
unset rolls onto the automatic set. That roll is the intended outcome —
those pools have no working driver without it — but it is a roll, and
the release note says so.

### Tests

- `packages/apps/kubernetes-nodes/tests/kernel_modules_test.yaml`, 12
cases: the NVIDIA set and its order for an `nvidia.com/*` pool, no
`kernel` block for a pool with neither GPUs nor modules, an explicit
list taken verbatim including `parameters`, an explicit list replacing
the NVIDIA default, `[]` opting out while leaving the `gpu=on` label
intact, no NVIDIA assumption for another vendor, a semicolon-separated
parameter accepted, a hostile module name and a hostile parameter each
rejected, a parameter carrying U+2028 and one carrying U+2029 each
rejected with the kernelModules message rather than a YAML parse error,
and an unvalidated extra key on an item dropped. The patterns pin
absolute indentation, because `modules:` one level out is still valid
YAML and silently loads nothing.
- `hack/talos-reconcile-heredoc_test.bats` lists `kernelModules` in its
validated arm and gains two execution-level cases: a hostile name and a
hostile parameter each refused at render, and a semicolon-separated
parameter surviving the real shell literally. Loosening the guards to
`^.+$` makes the refusal case fail.
- `helm unittest packages/apps/kubernetes-nodes`: 31 suites, 174 tests,
12 snapshots, all passing. `hack/cozytest.sh
hack/talos-reconcile-heredoc_test.bats`: 8 passed.
- `make generate` in the package, and `controller-gen object` for the
`api/apps/v1alpha1` submodule.

### Not in scope

The rest of #3563: the `nvidia-operator-validator` → device-plugin chain
that cannot validate an OS-provided driver, CDI spec generation and the
`nvidia` containerd runtime default, and the `maxSurge` /
`nodeStartupTimeout` defaults.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/issues/643
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
https://github.com/cozystack/terraform-provider-cozystack/issues/20
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

Both follow-ups are issues rather than PRs.
`terraform-provider-cozystack` is hand-written, so the field needs a
schema entry, a model entry and an expand/flatten pair on
`KubernetesNodes`; the undefaulted three-state behaviour is exactly
where that provider's habit of sending its own defaults would break
users silently, since materialising `kernel_modules = []` would disable
the automatic NVIDIA set for every GPU pool managed through Terraform.
`website` needs the three-state behaviour written down, the GPU pool
example pointed at `osImage.factory.schematicID`, and the Blackwell
constraint documented: on GB202 the schematic must carry
`siderolabs/nvidia-open-gpu-kernel-modules-production`, because with the
proprietary extension the module loads, `/dev/nvidia0` appears, and
`nvidia-smi -L` then reports `No devices found`.

The `hack/talos-reconcile-heredoc_test.bats` change adds test cases and
does not change node prerequisites, so `ansible-cozystack` and `talm`
are not reached. No other repository in the map is touched: no package,
platform component, bundle, variant or release asset is added, and
`ApplicationDefinition` semantics are unchanged.

### Release note

```release-note
feat(kubernetes): Worker node pools (`KubernetesNodes`) accept a `kernelModules` field, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a kernel module without loading it, so a GPU pool previously came up with no working driver and no way to fix it through values. Left unset, a pool holding at least one `nvidia.com/*` GPU now loads `nvidia` (with `NVreg_NvLinkDisable=1`), `nvidia_uvm`, `nvidia_drm` and `nvidia_modeset` automatically; an explicit list replaces that set, and an explicit `[]` opts out. The modules still have to be in the image: point the GPU pool's `osImage.factory.schematicID` at a schematic carrying the NVIDIA extensions, and on Blackwell (GB202) use the open kernel modules extension, not the proprietary one. Pools without GPUs and without the field render exactly what they rendered before. NOTE: while the worker `TalosConfigTemplate` keeps a fixed name (#3515), a changed module list does not reach a pool that already exists — including the automatic set on existing GPU pools — and only pools created afterwards pick it up. Once that template is named by its content (#3523), the same change rolls the pool, so existing GPU pools that leave the field unset are rolled onto the automatic set.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Worker node pools can now configure Talos kernel modules, including
optional module parameters.
* When no list is specified, NVIDIA GPU pools automatically receive
NVIDIA modules; other pools receive none. A custom list replaces this
selection, while an empty list disables modules.
* Invalid module names or parameters are rejected during configuration
rendering.
* **Documentation**
* Added guidance on module selection, parameters, image-extension
requirements, and pool rollout behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 21:43:43 +03:00
Aleksei Sviridkin 8ed34166cf perf(lineage): cache owner GETs and short-circuit re-admission (#2667)
## What this PR does

The lineage mutating webhook walks the owner chain (Pod, StatefulSet,
HelmRelease, application CR) on every admission, with one uncached
dynamic GET per level, and sibling Pods of the same parent repeat the
same lookups. On a multi-tenant cluster this put `lineage.cozystack.io`
at 300-500ms p99 in
`apiserver_admission_webhook_admission_duration_seconds`, the largest
share of admission latency.

Owners are now cached in-process for 60 seconds, and only successful
lookups are stored. A failed lookup (RESTMapping, NotFound, timeout)
leaves the walk without an owner, so the object is admitted with no
lineage labels and stays unlabeled until its next UPDATE runs the walk
again. Secrets, Services and PVCs can go a long time between updates, so
a cached failure would leave every object admitted within the TTL
unlabeled for that long. The aggregated cozystack-api makes such
failures routine, since discovery of `apps.cozystack.io` fails while it
rolls. The one staleness left is the scheduling class, which is read
from the application's spec: a change to it reaches newly admitted Pods
up to 60 seconds later.

The cache is not swept below 4096 entries, so up to that many owner
objects stay in memory for the life of the process. Above that, each
insert drops the expired entries among a random sample of 64, which
keeps the write lock short.

Non-Pod objects that already carry every lineage label,
`internal.cozystack.io/tenantresource` included, skip the walk, because
the objectSelector still matches an UPDATE whose old object lacks the
label. Pods go through regardless, as each Pod admission also sets the
scheduling class.

### Screenshots

Not a UI change.

### Downstream repositories

The diff is Go under `pkg/lineage` and
`internal/lineagecontrollerwebhook` only: no label, annotation, CRD,
values key, `hack/` file or telemetry metric is added or renamed, so no
entry of the trigger map applies.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
perf(lineage): the lineage mutating webhook caches owner lookups for 60 seconds and skips the owner walk for non-Pod objects that already carry lineage labels, cutting admission latency on clusters with many tenants.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Lineage and owner lookups now use a time-limited in-memory cache,
reducing repeated API requests during admission processing.
Scheduling-class updates may take up to 60 seconds to appear on new
Pods.
* Non-Pod objects with complete lineage labels are allowed without
additional owner lookup or mutation.
* **Tests**
* Added coverage for cache hits, misses, expiration, and retrying
lookups after errors.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 21:34:19 +03:00
Aleksei Sviridkin 877d2ff43c fix(linstor): build piraeus-server's protobuf code on an arm64 builder (#4576)
## What this PR does

The arm64 half of a release fails to build piraeus-server.
linstor-server's build always downloads the `linux-x86_64` protoc, and
the builder stage runs on the build platform. Since the release builds
arm64 on a native arm64 runner, that protoc cannot execute there, and
`./gradlew generateJava` fails with `Syntax error: ")" unexpected`. The
first multi-arch pre-release run, v1.7.0-alpha.2, stopped on it.

A new patch to linstor-server picks the protoc build for the host's
architecture. An amd64 builder still gets the same x86_64 protoc as
before. The generated code is Java, so one native builder serves both
target platforms.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Builds on ARM-based hosts now select a compatible Protocol Buffers
compiler download, avoiding failures caused by using an x86_64-only
build. Builds on other supported hosts continue to use the x86_64
compiler. This improves compatibility across build architectures without
changing the resulting application behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
v1.7.0-alpha.3
2026-09-29 21:21:54 +03:00
Aleksei Sviridkin e56478cf2f feat(keycloak): expose organizationsEnabled realm toggle (#3031)
## What this PR does

- Adds an `organizationsEnabled` value to the `cozy-keycloak-configure`
chart (default `false`, no behaviour change).
- Maps it to `ClusterKeycloakRealm.spec.organizationsEnabled` so the
Keycloak
[Organizations](https://www.keycloak.org/docs/latest/server_admin/#_managing_organizations)
feature can be turned on declaratively for the platform realm.

### Why

The chart hardcoded the realm without exposing this field. Enabling
Organizations through the Keycloak admin console is not durable: the EDP
keycloak-operator reconciles `ClusterKeycloakRealm` by `PUT`ting the
chart-rendered spec back to the realm, so the console change is reverted
on the next reconcile (operator restart, periodic resync, or any CR
change). Verified on Keycloak 26.5.2.

### Release note

```release-note
feat(keycloak): add `organizationsEnabled` value to the keycloak-configure chart to enable the Keycloak Organizations feature on the platform realm (default: false)
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added a configuration option to enable or disable organizations in the
platform-managed Keycloak realm.
* Organizations remain disabled by default. Set the option to `true` to
enable them.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 21:19:47 +03:00
Aleksei Sviridkin 018b49929f feat(monitoring): auto-route tenant logs using namespace labels (#2195)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

A tenant that runs its own monitoring stack gets a VictoriaLogs
instance, but the node fluent-bit in `cozy-monitoring` sends every
container log to `global.target` (`tenant-root`), so that instance stays
empty. This PR routes each namespace's container logs to the
VictoriaLogs named by its `namespace.cozystack.io/monitoring` label, the
label the tenant chart already sets and the per-tenant vmagent already
selects on.

**Chart (`packages/system/monitoring-agents`).**
`templates/_tenant-log-routes.tpl` looks up the namespaces and keeps
those whose label is set and names a tenant other than `global.target`.
For those, fluent-bit gets one `rewrite_tag` filter with a `Rule` per
target (a single emitter however many tenants are routed), one `http`
output per target to `vlinsert-generic.<target>.svc:9481`, and a
per-target `modify Set tenant <target>`, because the re-tagged record
re-enters the filter chain where the `Match *` `Add tenant` cannot
overwrite the key. Unlabelled namespaces, empty labels and labels equal
to `global.target` stay on the default outputs.

**Controller (`internal/controller/tenantlogrouting`).** `lookup`
results are not part of the digest helm-controller compares, so the
chart alone would not pick up a tenant that turns monitoring on, or drop
one that is removed, until an unrelated upgrade. The new reconciler in
cozystack-controller digests the monitoring label across all namespaces
and, when the digest moves, sets `reconcile.fluxcd.io/requestedAt` and
`reconcile.fluxcd.io/forceAt` on `cozy-monitoring/monitoring-agents`,
which forces an upgrade and re-runs the lookup. It digests every
labelled namespace rather than repeating the chart's selection, so it
cannot drift from the chart; the cost is an occasional upgrade that
renders the same config. The digest it acted on is kept in its own
annotation, so a manual `flux reconcile` does not make it force another
upgrade. The existing ClusterRole already grants the verbs it needs.

**Behaviour change for the platform operator.** Routed records are
re-tagged with `keep false`: once a tenant runs its own monitoring, its
namespaces' container logs go to that tenant's VictoriaLogs and no
longer to the root one. Events and audit logs are not routed and stay on
the root store.

**Why not a per-tenant collector.** An earlier comment on this PR
proposed a fluent-bit per tenant, following the per-tenant vmagent and
the OTLP collector from #3763. That collector receives OTLP over the
network; a log collector has to tail `/var/log` through `hostPath`,
which the PodSecurity level of tenant namespaces rejects, and running it
in a privileged system namespace instead would mean one DaemonSet per
tenant tailing every node's logs. The routing therefore stays in the one
node DaemonSet, with the reconciler closing the reconcile gap.

Tests: `tests/fluent-bit_tenant_log_routing_test.yaml` mocks the
namespaces with `kubernetesProvider` and pins which get a rule and an
output, which are skipped (unlabelled, empty label, label equal to
`global.target`), and that a routed record carries the target tenant.
Three of its cases fail against the previous revision of this branch.
`tests/fluent-bit_tenant_log_routing_offline_test.yaml` pins that an
offline render routes nothing.
`internal/controller/tenantlogrouting/reconciler_test.go` covers first
sight, no-op while labels are unchanged (including after a manual
reconcile), a tenant turning monitoring on, a routed namespace being
deleted, and a missing release.

Refs #2194

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

No entry of the trigger map matches the diff: it touches
`packages/system/monitoring-agents` (values, a template helper, tests),
a new reconciler in `internal/controller`, its registration in
`cmd/cozystack-controller`, and a comment in the cozystack-controller
ClusterRole. The one open question is `cozystack/website`:
`content/en/docs/next/operations/services/monitoring/logs.md` describes
log collection in general terms and does not say where a tenant's
container logs land, so the behaviour change above could deserve a
sentence there. I have left every box empty for a maintainer to decide.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
feat(monitoring): container logs from a tenant that runs its own monitoring stack now go to that tenant's VictoriaLogs instead of the root one. The node fluent-bit routes them by the namespace.cozystack.io/monitoring label, and cozystack-controller re-renders the node agents when that label changes on any namespace.
```
2026-09-29 21:18:52 +03:00
Aleksei Sviridkin 811f030f15 fix(cozy-lib): round the sanitized CPU request up to a whole milli-CPU (#3522)
## What this PR does

Fixes #2917.

`cozy-lib.resources.sanitize` divides the CPU limit by
`cpuAllocationRatio` and rendered the quotient as a bare float. When the
ratio does not divide the value evenly the result is finer than a
milli-CPU, and the VPA admission webhook rejects the object outright:

```
admission webhook "vpa.k8s.io" denied the request:
minAllowed: CPU [62500u] must be a whole number of milli CPUs
```

`packages/extra/etcd/templates/vpa.yaml` routes its VPA
`minAllowed`/`maxAllowed` through the helper, so with
`cpuAllocationRatio: 4` its floor renders as `250m / 4 = 62.5m` and the
`etcd` HelmRelease loops `UpgradeFailed` → `RollbackFailed` forever: the
rollback target renders the same invalid value. Since v1.6.0
HelmReleases wait on each other, so that one stuck app also blocks
`tenant-root` → `cozystack-basics` → `kubevirt-cdi-operator`, and
without CDI the Talos worker rollover cannot import the worker OS image.
Only the divisors of 250 worked as ratios, which is an artefact of one
chart's hardcoded `250m` floor rather than a real constraint on the
operator.

- `packages/library/cozy-lib/templates/_resources.tpl`: round the
sanitized CPU request **up** to the next whole milli-CPU. Up, never
down, so the request stays at or above the intended fraction of the
limit. And only when rounding is actually needed: an exactly divisible
value keeps its current float rendering, so nothing changes on the
default ratio.
- The whole-milli test is exact, with no tolerance around it. sprig's
`divf`/`mulf` run on `shopspring/decimal` rather than float64, so `mulf
(divf 0.25 10.0) 1000.0` renders exactly `25` in Helm and there is no
binary-float noise on this codepath to smooth over. A tolerance would in
fact reintroduce the bug: a value that is whole-milli only to within
`5e-7` would read as exact and render verbatim, so `cpu: 100.0000004m`
at ratio 1 would render `0.1000000004`, rejected for the same reason
`62.5m` is. There is a test for that case.
- `packages/tests/cozy-lib-tests/`: new `resources` template plus a
`resources_test.yaml` suite covering the indivisible ratio, the round-up
direction, the two unchanged-rendering cases, a
whole-milli-only-within-a-tolerance value, sub-milli input at an
ordinary and at an extreme ratio, a genuinely zero request, and that
memory is untouched.
- `packages/extra/etcd/tests/vpa_test.yaml`: regression guard asserting
the chart's rendered VPA `minAllowed.cpu` is a whole milli-CPU (`63m`)
with `cpu-allocation-ratio: 4`, and that the default ratio still renders
`0.025` / `0.5`.

### Why in `sanitize` and not in the etcd chart

The issue also suggests taking `minAllowed`/`maxAllowed` out of
`sanitize` in the etcd chart, on the grounds that VPA policy bounds are
not workload requests. I looked at that first and it is the wrong fix
here:

- `minAllowed`/`maxAllowed` bound the **request** the VPA writes, and in
Cozystack every request is `limit / cpuAllocationRatio` —
`packages/extra/etcd/templates/etcd-cluster.yaml` sanitizes
`.Values.resources` the same way. On the default ratio the etcd request
is `1000m / 10 = 100m`, bracketed by `minAllowed 25m` and `maxAllowed
500m`. Dropping `sanitize` would make the floor a literal `250m`, i.e.
**10x** the intended request floor, and let the ceiling reach `5000m`.
It would also change rendered output on the default ratio, for every
existing cluster.
- It fixes one chart. The same helper feeds every app's requests, and at
`cpuAllocationRatio: 4` seven other charts (`http-cache`, `mariadb`,
`nats`, `rabbitmq`, `redis`, `tcp-balancer`, `valkey` — all from the
`250m` nano presets) also render `cpu: "0.0625"`. Those go into pod
requests rather than a VPA object so they are not rejected, but they are
still not quantities we should be emitting.
- No other chart routes `minAllowed` through `sanitize`, so there is no
consistency argument either way: the other VPA templates
(`etcd-operator`, `monitoring`, `monitoring-agents`, `vpa-for-vpa`) use
hand-picked literals in system charts where no allocation ratio applies.

### How it was validated

- `hack/helm-unit-tests.sh`, full run: 81 packages green, before and
after. The only difference between the two runs is the two new suites
appearing as `PASS`; no existing suite changed status.
- Both new suites were confirmed to **fail** against `main`'s helper (5
tests in `resources_test.yaml`, 1 in the etcd `vpa_test.yaml`), so they
guard the regression rather than merely describing the new behaviour.
- Breadth of the bug: over the 905000 combinations of a whole-milli
limit `1m`..`1000m` against an integer ratio `1`..`905`, `main` renders
a quantity finer than 1m for **898026** of them (99.2%) and this branch
for **none**.
- Rendered-output check on the default ratio: `helm template` over all
164 chart directories under `packages/{apps,extra,system,core}`, before
and after, byte-diffed. 129 render standalone; all 129 are identical
apart from 8 whose diff is only their per-render random secrets and
self-signed certs (no `cpu` line differs in any of them).
- Same sweep with `--set _cluster.cpu-allocation-ratio=4`: `main` emits
11 sub-milli `cpu: "0.0625"` requests across 7 charts, this branch emits
**0**.
- The quoted webhook error was checked against the VPA we actually ship
(1.5.1): `pkg/admission-controller/resource/vpa/handler.go` wraps the
quantity error as lowercase `minAllowed:`/`maxAllowed:`, and `62.5m`
canonicalises to `62500u`.
- Not verified: nothing from this branch was applied to a live cluster,
so the end of the chain (the webhook accepting the object, the etcd
HelmRelease converging, CDI unblocking) is inferred from the rendered
manifests and the webhook's own error message, not observed.

### Screenshots

Not applicable, no UI change.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff. The diff is one Helm library template plus test files: no package
added, renamed or removed; no `values.yaml`, `values.schema.json`,
version enum or default changed; no `ApplicationDefinition`, CRD,
namespace, `hack/` file, package layout or release asset touched. The
terraform provider restates app defaults, but no default moves here —
the rendered value only changes for a non-default `cpuAllocationRatio`,
which the provider does not model.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(cozy-lib): round the sanitized CPU request up to a whole milli-CPU. A `cpuAllocationRatio` that does not divide a chart's CPU value evenly (for example `4` against the `250m` VPA floor of the etcd chart) produced a quantity finer than 1m, which the VPA admission webhook rejects, leaving the etcd HelmRelease looping UpgradeFailed/RollbackFailed and — since v1.6.0, where HelmReleases wait on each other — blocking tenant-root, cozystack-basics and CDI behind it. Any allocation ratio now renders a valid quantity; rendering is unchanged whenever the ratio divides evenly, including on the default ratio of 10. Where a ratio does not divide a configured CPU value evenly, the rendered request changes once on upgrade: for example `cpu: 125m` at the default ratio goes from `0.0125` to `13m`, and so does every nano preset at a ratio like 4. The API server already rounds pod requests up the same way, but the string stored in operator-managed resources (CNPG clusters, KubeVirt VMs and so on) changes, and some operators may roll their pods for it.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Bug Fixes**
- CPU resource requests are now sanitized to avoid floating-point
precision issues.
- Fractional milli-CPU requests are rounded up to the nearest whole
milli-CPU, with positive sub-milli requests clamped to `1m`.
  - Exact and zero CPU values remain unchanged.
- CPU limits and memory requests continue to be preserved without
modification.

- **Documentation**
  - Updated resource guidance to describe CPU request rounding behavior.

- **Tests**
- Added coverage for allocation ratios, default bounds, rounding,
precision, and resource preservation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 21:17:44 +03:00
Aleksei Sviridkin 6ab3e0be78 feat(postgres): allow preloading an allowlist of shipped modules (#3506)
## What this PR does

- Adds `postgresql.sharedPreloadLibraries`, a closed enum of four module
names that already ship in the CloudNativePG PostgreSQL image, rendered
onto the CNPG `Cluster` field of the same meaning.
- Leaves the `shared_preload_libraries` entry in the
`postgresql.parameters` blocklist exactly as it is.
- Adds `tests/shared_preload_libraries_test.yaml`: absent by default,
single and multiple libraries, no interference with `parameters`, schema
rejection of a non-allowlisted name and of an absolute path, and the
parameter blocklist still firing when the allowlist is in use.
- Adds `hack/check-postgres-preload-allowlist.bats`, which renders the
chart with `--skip-schema-validation` to exercise the template-side
allowlist that helm-unittest never reaches, and checks that it matches
the schema enum.

`shared_preload_libraries` is blocked in `postgresql.parameters`, and
rightly so: the parameter is free text and accepts a path, so it can
point at a writable location such as the PGDATA volume. The block does
not make every preloadable module unreachable, though. CloudNativePG
carries a managed extensions list (`ManagedExtensions` in
`pkg/postgres/configuration.go`, checked on v1.30.0, the operator
version in this repo): as soon as a parameter under the
`pg_stat_statements.`, `auto_explain.` or `pgaudit.` prefix is set, the
operator adds that library to `shared_preload_libraries` itself, and for
`pg_stat_statements` and `pgaudit` it also runs `CREATE EXTENSION`. The
chart already passes such keys through, so
`postgresql.parameters."pg_stat_statements.track": all` works on main
today.

What this PR adds is narrower than a missing capability. `pg_prewarm`
has no route at all, and its autoprewarm worker only runs when the
module is preloaded. For the other three, the field is an explicit
switch that does not require setting one of their tuning parameters,
even to its default value, just to get the library loaded. The two
routes combine rather than override: the operator merges this list with
the libraries it preloads on its own, so removing a module from
`sharedPreloadLibraries` does not unload it while a parameter under its
prefix is still set. The field description says so.

The distinction this PR draws is between a library *name* and a library
*path*. Nothing needs to be injected to preload these four:
`pg_stat_statements.so`, `auto_explain.so`, `pgaudit.so` and
`pg_prewarm.so` are already in `$libdir` in the image, and that
directory is on the read-only root filesystem, so it cannot be written
to. `dynamic_library_path` stays blocked, so `$libdir` remains the only
search path. A bare name from a fixed enum therefore has no way to reach
code that is not already in the image, which is the property the blanket
block was protecting.

The enum in `values.schema.json` is the primary guard and rejects a bad
value before rendering starts. The template carries the same list as a
backstop for renders that bypass schema validation, using the existing
fail-fast `SECURITY:` idiom from the parameter blocklist a few lines
below it. helm-unittest validates the schema on every render, so it
never reaches that backstop; the bats suite renders with
`--skip-schema-validation` instead. I checked that it fails when the
template condition is replaced with `false`, and when the template list
gains a name the enum does not have.

I verified the mechanism end to end on a live CloudNativePG 1.27.3
cluster rather than assuming it: with
`spec.postgresql.shared_preload_libraries` set, the operator writes
`shared_preload_libraries = 'pg_stat_statements'` into
`postgresql.conf`, `SHOW shared_preload_libraries` reports it, and
`SELECT ... FROM pg_stat_statements` returns rows. Adding a second
module to the list took effect too, and the CNPG instance manager
restarts PostgreSQL in place for it, with no pod restart. That restart
covers every instance, the primary included, so writes pause briefly.
The CRD documents the field as additive to the operator's own defaults,
so this does not clobber anything CNPG needs to preload itself.

Scope note: the four names are the modules in the image that are useful
to a tenant and safe to preload. Anything requiring a background worker
or a shipped-but-privileged module was deliberately left out, and the
enum is the single place to extend if that changes.

### Screenshots

No UI change.

### Downstream repositories

`packages/apps/postgres/values.schema.json` gains a field, which per the
trigger map reaches `cozystack/terraform-provider-cozystack`: its
postgres resource carries a hand-maintained schema and its own
expand/flatten pair, with no codegen from `values.schema.json`, so
`sharedPreloadLibraries` will be invisible to Terraform users until it
is added there by hand. The enum matters twice over there, because the
provider restates enums as `stringvalidator.OneOf` lists. I searched
that repository and found no open PR or issue covering it (the only open
PR is a dependabot Go module bump). I have not filed the follow-up, so I
am leaving the box unticked rather than claiming work that does not
exist. The same gap applies to `users.<name>.createRole` in #3505; one
follow-up there can cover both fields.

I walked the rest of the map against the diff and found nothing else: no
package is added, renamed or removed, no existing version enum changes,
no default changes, `release.prefix` and the output Secret and Service
names are untouched, and no CRD outside the regenerated
`packages/system/postgres-rd/cozyrds/postgres.yaml` is affected. The
`README.md` parameter table is regenerated here and postgres is already
in the website generator's app list, so the website path stays
automated.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: not filed yet, see the note above
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
feat(postgres): add `postgresql.sharedPreloadLibraries`, a closed allowlist of modules shipped in the PostgreSQL image (`pg_stat_statements`, `auto_explain`, `pgaudit`, `pg_prewarm`) that are preloaded at server start. It is the first way to preload `pg_prewarm`, and lets the other three be preloaded without setting one of their parameters, which CloudNativePG already treats as a request to preload them. The `shared_preload_libraries` parameter stays blocked: only bare names from the allowlist are accepted, never a path. Changing the list restarts PostgreSQL on every instance, the primary included.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added optional PostgreSQL shared preload library configuration for
`pg_stat_statements`, `auto_explain`, `pgaudit`, and `pg_prewarm`.
* Preserves the configured library order. CloudNativePG may
automatically preload the first three libraries when their related
parameters are set; `pg_prewarm` must be enabled through this list.
* Changing the list restarts every PostgreSQL instance, including the
primary.
* **Validation**
* Rejects unsupported modules, absolute paths, and attempts to set
preload libraries through free-form parameters.
* **Documentation**
* Added configuration guidance covering supported libraries and restart
behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 21:12:04 +03:00
Aleksei Sviridkin e126158404 fix(ci): keep the controller Go tests running when a unit test fails first (#4233)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The `checks` job in `.github/workflows/pull-requests.yaml` runs `make
unit-tests test-controllers -j4 --output-sync=target` as a single step.
`unit-tests` and `test-controllers` are independent goals on that
command line, and without `-k` GNU Make stops handing out new work the
instant any earlier-processed goal's prerequisite fails — a goal that
had not yet been scheduled never starts. So whenever a BATS file inside
`unit-tests` fails before `test-controllers` gets a job slot, the entire
Go controller suite (`go test ./internal/...`) silently never runs, and
the PR page shows only the unit-test failure with no signal at all about
the controllers. This adds `-k` so already-schedulable goals keep
running after an earlier one fails; the job still exits non-zero and
still fails whenever anything failed, `-k` only stops make from
abandoning independent work.

This addresses the second half of
https://github.com/cozystack/cozystack/issues/3776. The first half of
that report — whether the job's comment overclaims that it gates the
merge, since `Unit & controller tests` is not currently in `main`'s
required status checks — needs a maintainer's call between two mutually
exclusive fixes (weaken the comment, or add the job to branch
protection), and is deliberately left untouched here; the issue itself
frames the two as independent and separately useful.

Note: `hack/unit-test-parallelism.bats`, which pins the literal
invocation string in a test, was updated to match.

### Screenshots

N/A — CI workflow and test-script change only, no UI.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

I walked the trigger map: this PR only touches CI scheduling flags
inside this repo's own `.github/workflows/pull-requests.yaml` and a
`hack/*.bats` meta-test that asserts against that same file. Nothing in
the trigger map names either file, and the change has no effect on any
package, API, chart, or contract another repository consumes.

### Validation

Reproduced the exact scheduling gap in an isolated Makefile mirroring
this repo's real dependency shape (a `unit-tests`-like goal depending on
several independent prerequisites, one of which fails immediately, plus
a sibling goal with no dependency on it), under the same GNU Make 4.4.1
`-j4 --output-sync=target` invocation used here. Without `-k` the
sibling goal never ran, across 5 repeated trials; with `-k` added it ran
to completion every time across 3 repeated trials, and the overall exit
code stayed non-zero either way. `bats hack/unit-test-parallelism.bats`
passes locally against the real files. I also ran `make unit-tests
test-controllers -j4 -k --output-sync=target` against this repo
directly: local disk space (a limit of my sandbox, unrelated to this
change) made `go test ./internal/...` fail to compile with "no space
left on device", so I could not confirm a clean end-to-end pass of
`test-controllers` itself in this environment — but the run still
exercised the fix: after that build failure, unrelated `hack/*.bats`
prerequisites kept being scheduled and ran to completion instead of the
whole invocation aborting, which is exactly the behavior `-k` adds over
the prior command line. I could not run this repo's actual CI runners to
confirm disk is not similarly constrained there; nothing in the workflow
or runner spec suggests it would be.

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(ci): the `Unit & controller tests` PR check no longer silently skips the Go controller test suite when a Helm-unittest or BATS unit test fails first; both suites now always run and report.
```

---
Part of #3776

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Tests**
* Updated automated test execution so unit and controller test suites
continue running when one test target fails, while the job still reports
failures.
  * Updated workflow validation to reflect the revised test command.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 20:00:54 +03:00
Aleksei Sviridkin 53829ea109 fix(kubernetes): reconcile KCCM EndpointSlices when tenant node VMIs change (#3500)
## What this PR does

Adds `patches/vmi-informer.diff` to the `kubevirt-cloud-provider` image
of the `kubernetes` package. The file is named after what it does: no
upstream pull request carries this change yet
(kubevirt/cloud-provider-kubevirt#382 covered the same desync and was
closed as rotten), so a number would point at an unrelated upstream PR
and invite someone bumping the pin to drop it.

The `kubevirteps` controller derives the infra EndpointSlices of a
LoadBalancer Service from the VMIs backing the tenant nodes
(`getDesiredEndpoints`, and the all-VMIs fallback added by `379.diff`).
But it registers informers only on infra Services, tenant EndpointSlices
and infra EndpointSlices, and both factories are built with a resync
period of `0`. VMI addresses are resolved with a one-off `Get` during a
reconcile.

Consequently a VMI appearing, disappearing or changing its address
triggers no reconciliation, and no periodic relist ever catches up. The
EndpointSlices keep pointing at VMIs that no longer exist and the
LoadBalancer blackholes the share of the inbound connections hashed onto
those dead addresses — sporadic inbound timeouts, with nothing wrong on
the tenant side.

Services whose tenant EndpointSlice churns along with its pods recover
on their own, because that churn produces an event. Cilium Gateway API
Services do not: their tenant slice is a static sentinel
(`192.192.192.192`, no `NodeName`), so nothing ever forces a recompute.
Seen in production on a cluster whose GPU MachineDeployment is rolled by
the cluster-autoscaler: 5 of the 9 endpoints of each of the 3 gateways
pointed at destroyed VMs (13 failures out of 20 requests). Worth noting
for anyone debugging this: the KCCM looks dead (no log for hours) while
its lease is still being renewed every second. The process is healthy,
it is simply never woken up.

The patch:

- adds an informer on the infra `VirtualMachineInstances` that requeues
the LoadBalancer Services of this cluster (those labelled
`cluster.x-k8s.io/cluster-name=<clusterName>`, the ones `reconcile` does
not skip) on add/delete, and on updates filtered by a fingerprint of
what actually shapes an endpoint: the VMI's cluster label,
`Status.NodeName`, `Status.Phase`, and the address of the `default`
interface;
- gives both informer factories a 5 minute resync, as a safety net if an
event is ever missed;
- scopes the all-VMIs fallback to the VMIs labelled
`cluster.x-k8s.io/cluster-name=<clusterName>`. An infra namespace can
hold several tenant clusters as well as standalone `VirtualMachines`,
and their VMIs do not serve this Service's NodePort, so selecting them
adds endpoints that blackhole traffic. The fallback widens to every VMI
only when none of them carries the label, so an environment that does
not label them does not lose all of its endpoints; once any VMI is
labelled, an empty selection means this cluster has no node right now
and is not widened to another cluster's VMIs.

No RBAC change: the KCCM role already grants `watch` on
`virtualmachineinstances`.

On `main`, the Gateway scenario reaches the all-VMIs fallback without
any other KCCM patch: Cilium 1.19.5 (shipped since v1.6.0) sets
`kubernetes.io/service-name` on the tenant slice of a `cilium-gateway-*`
Service (cilium/cilium#46242), so the existing lookup finds the sentinel
slice and falls back on its missing `NodeName`. This is why #3374, which
added that lookup to the KCCM, was closed. On v1.5.x and earlier, which
ship Cilium 1.19.3, a Gateway Service finds no tenant slice at all, so
it has no endpoints to go stale in the first place.

## Testing

- `341 + 379 + vmi-informer` applies onto the pinned `a0acf33`; the
Dockerfile steps (`go get`, `go mod tidy`, `go mod vendor`,
`-mod=vendor` build of the manager) pass, `go vet` and `gofmt` are
clean.
- The patch adds 6 specs, and the suite runs 21 of 21 green: fallback
scoped to the cluster, fallback preserved when no VMI is labelled, no
widening when only another cluster's VMIs are present, unlabelled VMIs
ignored once some are labelled, VMI add/delete requeues the Service of
this cluster and not the one of another cluster, and VMI updates driven
through the fake dynamic client: an annotation-only update does not
requeue, while a change of phase, node, address, or cluster label does.
- Mutation-tested, each one turns the suite red: always requeuing on
update; dropping the phase, the node, the address or the cluster label
from the fingerprint; requeuing every Service of the namespace; dropping
the delete requeue.
- Under `-race`, the suite reports an intermittent data race at
`kubevirteps_controller_test.go:820` (a `PrependReactor` racing an
informer list). It reproduces identically on `341 + 379` without this
patch, and the new specs pass on every `-race` run.
- Validated on a live cluster with an image built from an earlier
revision of this patch (same informer and fallback, before the
cluster-scoped requeue and the label in the fingerprint): endpoint added
in under 2s when a VMI appears, dead address removed in 33s when it is
deleted. Before the patch the dead address stayed forever.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff is one patch file of the KCCM image: it touches no values,
schema, CRD, `hack/` tooling, namespace, telemetry metric or node
prerequisite, so the trigger map comes out empty.

### Release note

```release-note
fix(kubernetes): the KCCM now reconciles the LoadBalancer EndpointSlices when the VMIs backing the tenant nodes change, instead of keeping stale addresses that blackhole part of the inbound traffic
```
2026-09-29 19:32:42 +03:00
Aleksei Sviridkin 924d28cf3d feat(kubernetes): default gpu-operator to an OS-provided driver on Talos (#3585)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->

## What this PR does

Closes the gpu-operator half of #3563 — gap 3 (the operator cannot
validate a driver it did not install) and gap 4 (nothing routes CUDA
libraries to a tenant pod).

Tenant clusters have run Talos workers since 1.6.0, where the NVIDIA
driver comes from a system extension baked into the worker image rather
than from the operator's driver container. The addon default still
described the Ubuntu shape, so a GPU node group came up with the
operator fighting the OS for the driver and no path at all for the CUDA
libraries.

Three changes, each closing one silent failure:

| Change | Without it |
|---|---|
| `driver.enabled: false`, `toolkit.enabled: false` | the driver
container loads its own module and clashes with the extension's; the
toolkit container rewrites `/etc/containerd/config.toml`, which Talos
owns and which already registers the `nvidia` runtime with `enable_cdi =
true` and `cdi_spec_dirs = ['/run/cdi']` |
| `validator.driver.env`: `DRIVER_ROOT=/`, `NVIDIA_DRIVER_ROOT=/`,
`DISABLE_DEV_CHAR_SYMLINK_CREATION=true` | the driver validator looks
for a driver-container install, finds none, never writes its sentinel,
and the device plugin stays blocked behind `toolkit-validation` — the
loop on `Attempting to validate a driver container installation` (**gap
3**) |
| `devicePlugin.env`: `DEVICE_LIST_STRATEGY=cdi-annotations`,
`NVIDIA_CTK_PATH=/usr/local/bin/nvidia-ctk`, plus `DEVICE_ID_STRATEGY`,
`NVIDIA_DRIVER_ROOT`, `CDI_ANNOTATION_PREFIX` | tenant pods carry no
`runtimeClassName` and did not need one on Ubuntu, where the toolkit
made `nvidia` the default containerd runtime. Talos does not, so CDI is
the only route: the pod gets the device and no CUDA libraries (**gap
4**) |

Every value is the configuration validated on a production Talos GPU
node group, minus the parts that only mattered while the driver
container was in play. No DaemonSet, no new image, no new package — the
fix is values.

### The removed default

The previous default fed `NVreg_NvLinkDisable=1` to the driver container
through the `nvidia-kernel-module-params` ConfigMap. With that container
off the ConfigMap is mounted by nothing, so this removes the default. On
Talos the parameter belongs to the worker machine config, which the
`kernelModules` field of the worker pool writes: that field is top-level
in the KubernetesNodes app (`packages/apps/kubernetes-nodes`), added by
#3571, and its automatic set for a pool holding an `nvidia.com/*` GPU
already loads `nvidia` with `NVreg_NvLinkDisable=1`. The wrapper chart
keeps the ConfigMap feature for an operator who re-enables the driver
container through `valuesOverride`. A test pins the absence so the dead
default cannot come back and offer a second, inert source for the same
setting.

### Relationship to #3571

Independent files, no conflict, but only useful together: this makes
`driver.enabled: false` correct, and #3571 is what actually loads the
modules the extension ships. Merged alone, this changes a GPU node group
from "operator fights the OS" to "operator stays out of the way and the
modules still are not loaded" — better, but not yet working. #3571 alone
leaves the operator fighting for the driver. Either order is safe; both
are needed for a GPU node group to work.

### What is verified, and what is not

Verified here: the rendered HelmRelease values, that `valuesOverride`
still wins over each new key, and that the removed default is gone. The
individual values are the ones running in production today, applied by
hand through `addons.gpuOperator.valuesOverride`.

Not verified here: this exact default exercised end-to-end as the
chart's own output. That needs a Talos GPU node group whose modules
load, which needs #3571 released. I would rather say so than imply a
green e2e that does not exist — happy to hold this as draft until I have
run a single-node L40S cluster against the chart default, and I will
report the result on this PR either way.

### Not covered

HAMi. When `addons.hami.enabled` the operator's device plugin stays off
and HAMi serves the GPUs; HAMi's own plugin needs equivalent CDI wiring
on Talos, which this default does not attempt. Flagging rather than
guessing — the production cluster this was derived from runs HAMi
disabled, so I have no validated configuration for that path. Worth its
own issue if a maintainer confirms the shape.

Also not in scope, and unchanged from the merge base:
`talos.installerRepository` and `talos.version` reach the worker machine
config unvalidated (being tracked separately by @lexfrei), and the
`nvidia-operator-validator` chain for the *management* cluster's own
passthrough/vgpu variants, which never load the host driver at all.

### Screenshots

No UI changes.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/issues/643
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
feat(kubernetes): The GPU Operator addon of tenant Kubernetes clusters now defaults to a driver provided by the operating system, which is the shape of the Talos workers these clusters have run since 1.6.0. `driver` and `toolkit` are disabled (the driver comes from a Talos system extension, and containerd configuration belongs to the machine config), the driver validator is pointed at the host root so it stops looping on "Attempting to validate a driver container installation", and the device plugin is switched to CDI (`DEVICE_LIST_STRATEGY=cdi-annotations`) so tenant pods receive the CUDA libraries — previously they received the GPU device and nothing else, since Talos does not make `nvidia` the default containerd runtime. The previous default, which fed `NVreg_NvLinkDisable=1` to the operator's driver container through a ConfigMap, is removed because that container is disabled; on Talos the parameter is carried by the worker machine config, through the `kernelModules` field of the worker pool in the KubernetesNodes app (`packages/apps/kubernetes-nodes`), which loads `nvidia` with `NVreg_NvLinkDisable=1` by default on a pool holding an `nvidia.com/*` GPU. Operators who overrode `addons.gpuOperator.valuesOverride` to work around this by hand can drop those overrides. NOTE: an override that sets `devicePlugin.env` replaces the list rather than merging it, so restate the full list when overriding it or CDI stops working. HAMi is not covered: when `addons.hami.enabled` the operator's device plugin stays off and HAMi's own plugin needs equivalent CDI wiring, which this change does not attempt.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Configuration**
* GPU workloads on Talos workers now use host-provided NVIDIA drivers
and tooling by default; the GPU Operator’s driver and toolkit are
disabled.
* Device discovery uses CDI annotations and UUID-based device IDs. When
HAMi is enabled, the Operator’s device plugin is disabled.
  * Host-root validation no longer creates `/dev/char` symlinks.
  * Driver kernel-module configuration is no longer created by default.

* **Documentation**
* Clarified when kernel-module configuration applies and what settings
are needed to re-enable the driver.
  * Noted that `valuesOverride` replaces lists wholesale.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 19:28:17 +03:00
Aleksei Sviridkin d466fd96b3 fix(rabbitmq): bound the topology operator Secret cache (#4130)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Fixes #4128.


`messaging-topology-operator` 1.14.2 caches **every Secret in the
cluster** — its `ClusterRole` grants `secrets: list, watch` cluster-wide
with no namespace or label restriction — and the vendored manifest
limits it to `500Mi`. On a cluster that has grown enough Secrets it
OOMKills and never recovers, because the initial `LIST` alone no longer
fits. While it is down, no topology CR reconciles for any tenant.

Measured on the affected cluster: 8 211 Secrets across 291 namespaces,
293 MB of Secret payload (87 % of it Helm release history), against 21
topology CRs and 1 `RabbitmqCluster`. Working set sat flat at 331–335 Mi
for 30+ hours, crossed 500Mi, then crashlooped 154 times. The operator
was caching 8 211 Secrets in order to read 2.

Upstream fixed exactly this, and it is not a bigger limit — their
default is still `512Mi`:

- `329aa9e4e` (2026-06-30, released in v1.20.0) gates the Secret
informer on `rabbitmq.com/topology-operator: "true"`, to bound "memory
usage on large clusters".
- `0850c3ae8` (2026-07-22, released in v1.20.1) bypasses the cache for
`Service` reads.

v1.20.2 is the current patch release carrying both; v1.19.x and earlier
carry neither. This PR refreshes the vendored manifest to v1.20.2
through the package's existing `make update` pipeline (fetch, drop the
`Namespace` object, rewrite `rabbitmq-system` →
`cozy-rabbitmq-operator`).

**The bump alone would break every install**, which is why the app chart
moves with it. The app chart's credentials Secret is referenced by
`User.spec.importCredentialsSecret`, and from 1.20 the admission webhook
rejects a user-provided Secret without the label:

```
secret %q must have label rabbitmq.com/topology-operator=true to be used by the Topology Operator
```

Verified against the 1.20.2 source, two things deliberately do **not**
change:

- The `RabbitmqCluster` `default-user` Secret needs no label —
`rabbitmqclient/cluster_reference.go` reads it through `apiReader.Get`,
uncached.
- The `Status.Binding` contract is unchanged since 1.14.2, so the
vendored `cluster-operator` 2.9.0 stays compatible and is left alone to
keep this PR focused.

The v1.20.2 manifest is self-contained: it ships its own `Issuer` and
both `Certificate` objects (webhook + metrics), and adds probes, a
`securityContext` that satisfies `restricted` PodSecurity, a defaulting
`MutatingWebhookConfiguration` and secure metrics. Same 13 CRDs, none
added or removed.

### Tests

- `packages/apps/rabbitmq/tests/user_credentials_test.yaml` renders two
users and asserts that each `User` imports its own
`<release>-<user>-credentials` Secret and that every credentials Secret
carries `rabbitmq.com/topology-operator: "true"`. Deleting the label
line from the template fails it.
- The rabbitmq e2e suite now creates its cluster with one user and
asserts `User/rabbitmq-test-app` reaches `Ready`, so the webhook and the
User reconcile run against the real 1.20.2 operator. Its `catch` gains a
describe of that `User` and the topology operator's logs.

### Upgrade path

Existing installs carry unlabeled credentials Secrets until each
tenant's `rabbitmq-*` HelmRelease re-renders. No migration is added: in
1.20.2 the User controller watches Secrets and maps an import Secret
back to its Users through the `.spec.importCredentialsSecret.name` field
index (`secretToUserRequests`), so relabelling the Secret requeues the
User immediately rather than on the next interval, and nothing is
removed from the broker in between. The import path no longer writes
`<user>-user-credentials`, and nothing in this repo reads the ones
1.14.2 left behind.

### Verification

- `make generate` in both touched packages — no diff beyond the
intentional changes
- `helm unittest packages/apps/rabbitmq` — 103/103, and the new suite
fails with the label removed

### Screenshots

No UI changes.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

Walked the trigger map against the diff, file by file.
`packages/apps/rabbitmq/templates/rabbitmq.yaml` adds one label to a
rendered Secret: no package added or renamed, no `values.schema.json`
field, no version enum, no `values.yaml` default, no `release.prefix`,
and the Secret's *name* is unchanged, so the provider's literal-name
lookups are unaffected.
`packages/system/rabbitmq-operator/templates/messaging-topology-operator.yml`
is a vendored upstream manifest for an already-present component: no
package layout change, no `cozyrds` rename, no namespace rename, no
installer or platform values key. `packages/apps/rabbitmq/tests/` and
`hack/e2e-chainsaw/rabbitmq/` are test-only and appear nowhere in the
map.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(rabbitmq): update messaging-topology-operator to 1.20.2, which bounds its Secret cache to Secrets labeled `rabbitmq.com/topology-operator=true` instead of caching every Secret in the cluster. Before this, the operator's memory grew with the cluster's total Secret count and OOMKilled against its 500Mi limit on large clusters, stopping reconciliation of all Queue, Exchange, Binding, User, Permission, Vhost and Policy resources for every tenant. The managed RabbitMQ chart now labels the credentials Secret it creates, so existing users need no action. If you manage topology resources by hand and point `connectionSecret`, `importCredentialsSecret`, `uriSecret` or `upstreamSecret` at your own Secret, add the label `rabbitmq.com/topology-operator: "true"` to it or the admission webhook will reject the resource.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added configuration options for resource cleanup, queue types,
connection and channel limits, shovel queue arguments, and federation
settings.
* Added metrics support, health checks, leader-election configuration,
and queue webhook capabilities.
* Improved compatibility of generated credential Secrets with the
RabbitMQ Topology Operator.
* **Bug Fixes**
* Improved validation and documentation for topology resources,
including requiring an exchange for topic permissions.
  * Updated operator components and Kubernetes resource configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 19:25:30 +03:00
Mattia Eleuteri 3c657a4cc1 feat(vpc): expose allowSubnets per subnet
Every VPC subnet is rendered private, so kube-ovn drops any packet
whose source lies outside the subnet's own CIDR. A tenant that runs a
VPN or router VM on a subnet and points a static route at it gets the
forward path from the existing routes value, but traffic arriving
from the remote network is then dropped by the private ACL.

Subnet is cluster-scoped, so the only workaround so far has been an
operator patching spec.allowSubnets by hand: something the tenant cannot
do, that no application value records, and that is lost whenever the
subnet is recreated. Declaring it per subnet keeps the widening in the
tenant's own VPC and scoped to the one subnet that needs it.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-09-29 18:47:18 +03:00
Mattia Eleuteri 91c87b84cd fix(kubernetes): refuse non-ASCII kernelModules parameters
RE2's \s and [:cntrl:] only cover ASCII, so the parameter deny class
passed U+2028 and U+2029. toYaml writes either one raw, and Helm's YAML
parser reads it as a line break that ends the reconcile Job's command
block scalar early. The render then failed on an unparseable manifest
rather than on the guard, which is the last check before the unquoted
heredoc.

Pinning parameters to printable ASCII closes every non-ASCII line break
at once, and the deny class keeps refusing what the heredoc expands.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-09-29 17:35:49 +02:00
Mattia Eleuteri ca24d8f5f6 perf(lineage): cache owner GETs and short-circuit re-admission
Every admission walks the owner chain (Pod, StatefulSet, HelmRelease,
application CR) with one uncached GET per level, and sibling Pods
repeat the same lookups. On a cluster with many tenants that put the
lineage webhook at 300-500ms p99, the largest share of apiserver
admission latency.

Owners are now reused for 60 seconds, successful lookups only. A failed
lookup, whether RESTMapping, NotFound or a timeout, leaves the walk
without an owner, so the object is admitted with no lineage labels and
stays unlabeled until its next UPDATE runs the walk again. Secrets,
Services and PVCs can go a long time between updates, so a cached
failure would leave every object admitted within the TTL unlabeled for
that long, and the aggregated cozystack-api makes such failures
routine: discovery of apps.cozystack.io fails while it rolls.

Non-Pod objects that already carry every lineage label, tenantresource
included, skip the walk, since the objectSelector still matches an
UPDATE whose old object lacks the label. Pods go through it regardless,
as each Pod admission also sets the scheduling class.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-09-29 17:32:13 +02:00
Aleksei Sviridkin d6e5c1bf2b fix(api): preserve foreign HelmRelease metadata on Application update (#2914)
## What this PR does

The aggregated `apps.cozystack.io` API rebuilds the HelmRelease from the
Application on `Update` and issues a full-replace PUT. This strips
metadata owned by other controllers:

- `metadata.finalizers` — including helm-controller's
`finalizers.fluxcd.io`
- `metadata.ownerReferences`
- labels and annotations outside the `apps.cozystack.io-` prefix (e.g.
`reconcile.fluxcd.io/requestedAt`)

Losing the Flux finalizer is the harmful case: helm-controller re-adds
it on the next reconcile, but a delete landing in that window removes
the HelmRelease **without running `helm uninstall`**, orphaning every
resource of the release (VMs, databases, PVCs...). A label patch
immediately followed by a delete — a common client pattern — hits that
window almost every time. We traced ~106 orphaned releases across 39
tenants on a production cluster back to this.

Reproduction:
1. `kubectl patch <any apps.cozystack.io resource> --type merge -p
'{"metadata":{"labels":{"foo":"bar"}}}'` → the backing HelmRelease loses
`finalizers.fluxcd.io`
2. `kubectl delete` the resource right after → HelmRelease is removed
instantly, Helm uninstall never runs, child resources stay alive

Changes:

- `Update` now fetches the current HelmRelease and carries over
`finalizers`, `ownerReferences`, and non-prefixed labels/annotations
before the PUT
- Keys under the `apps.cozystack.io-` prefix remain owned by the API and
are recomputed from the Application, so label/annotation deletions still
propagate
- Regression test `TestUpdate_PreservesForeignHelmReleaseMetadata` pins
the carry-over contract (fails on the previous behavior with exactly the
stripped fields)

## Release note

```release-note
fix(api): updating an Application no longer strips finalizers, ownerReferences, and foreign labels/annotations from the backing HelmRelease, which could cause deletion to skip helm uninstall and orphan the release's resources
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Application updates now preserve metadata managed by external
controllers, including finalizers, owner references, labels,
annotations, and shard information, when synchronizing related releases.
* Conflict retries now retain the latest release metadata, preventing
changes made between attempts from being lost.
* Creating an application without labels no longer alters shared release
configuration labels.

* **Tests**
* Added coverage for metadata preservation during updates and conflict
retries, and for label handling during application creation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 18:30:27 +03:00
Aleksei Sviridkin 6c7d22edab fix(linstor): build piraeus-server's protobuf code on an arm64 builder
linstor-server's build always downloads the linux-x86_64 protoc, and
the builder stage runs on the build platform. The release now builds
the arm64 images on a native arm64 runner, where that protoc cannot
execute and generateJava fails. A patch picks the protoc build for the
host's architecture.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-29 18:21:34 +03:00
Mattia Eleuteri 1f9b1b85db fix(kubernetes): reconcile KCCM EndpointSlices when tenant node VMIs change
The kubevirteps controller derives the infra EndpointSlices of a
LoadBalancer Service from the VMIs backing the tenant nodes, but it
watches no VMI and its informers had no resync period. A VMI appearing,
disappearing or changing its address therefore triggered no
reconciliation, and no periodic relist ever caught up: the EndpointSlices
kept pointing at VMIs that no longer existed and the LoadBalancer
blackholed the share of the inbound connections hashed onto those dead
addresses.

Services backed by a tenant EndpointSlice that churns with its pods
recovered on their own. Cilium Gateway API Services did not: their
tenant slice is a static sentinel, so nothing ever forced a recompute.
Observed in production on a cluster whose GPU MachineDeployment is
rolled by the cluster-autoscaler: 5 of the 9 endpoints of every gateway
pointed at destroyed VMs.

Add an informer on the infra VirtualMachineInstances that requeues the
LoadBalancer Services of this cluster, ignoring the updates that cannot
change an endpoint, and give the informers a 5 minute resync as a
safety net. Services of other tenant clusters sharing the namespace are
not requeued, since reconcile would skip them.

Also scope the all-VMIs fallback to the VMIs labelled with the name of
the cluster being reconciled: an infra namespace can hold several
tenant clusters and standalone VirtualMachines, whose VMIs do not serve
the Service NodePort. The fallback widens to every VMI only when none of
them carries the label, so that environments that do not label them keep
their endpoints. Once any VMI is labelled, an empty selection means this
cluster has no node right now, and widening it would publish the
addresses of another cluster.

The patch is named after what it does rather than after an upstream
pull request, because none carries this change yet.

Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Assisted-by: LLM
2026-09-29 17:21:19 +02:00
Aleksei Sviridkin 9b56e70ca6 feat(mongodb): support the system-bucket backup flow (useSystemBucket) (#3967)
## What this PR does

MongoDB was the only `cozy-default` app with **no `useSystemBucket`
flow**: its backup storage was always static on the
`PerconaServerMongoDB` CR (bucket/endpoint/credentials from `backup.*`
values plus a chart-emitted `<release>-s3-creds` Secret), so a tenant
could not back up to the platform system bucket out of the box the way
Postgres/ClickHouse/MariaDB/etcd do. This closes that gap so the default
MongoDB backup-restore works OOTB.

PBM takes `bucket`/`endpointUrl`/`region`/`prefix` as **static CR
fields** (not secret refs), and the platform bucket name/endpoint are
only known at BackupJob time — so, unlike ClickHouse's runtime
`secretKeyRef` env, the chart cannot render them. The implementation
mirrors the CNPG driver's pattern:

- **Chart** (`packages/apps/mongodb`): a new `backup.useSystemBucket`
flag keeps `spec.backup.enabled` (so the pbm agents run) but omits the
static `storages`/`tasks`/`pitr` and the `<release>-s3-creds` Secret.
Helm never owns `spec.backup.storages`, so a Flux re-render can't fight
the driver. A `bootstrap.enabled + useSystemBucket` render guard mirrors
Postgres, and `useSystemBucket: true` with `backup.enabled: false` is
refused both at render time and by an `x-kubernetes-validations` rule on
`backup` in the schema (the same mechanism as the `storageClass`
immutability rule). Note on the schema rule: the aggregated apiserver
does not evaluate CEL on application-backed resources yet (#2657), so
today it is a UI/schema-level marker and the render-time refusal is what
stops a Flux release; it also trips the API review gate (“validation
rule added”), so this PR needs an API-owner approval on top of the code
review.
- **Strategy** (`cozy-default-mongodb`): now carries the S3 coordinates
(`endpointUrl` already scheme'd by the
`backupstrategy-controller.endpoint` helper, `credentialsSecret:
cozy-backups-creds`, `prefix: <namespace>/<application>`).
- **Driver** (`psmdbstrategy_controller.go`): SSA-injects
`spec.backup.storages[s3-storage]` onto the live `PerconaServerMongoDB`
at BackupJob time under a dedicated field manager
(`cozystack-psmdb-backup-driver`) with `ForceOwnership`. Injection is
**gated on the app's `backup.useSystemBucket` flag** (a legacy app
leaves it false and is never touched; a useSystemBucket app is
re-injected once per mint (once per BackupJob, plus once per
storage-race re-mint), before its operator CR is created, so a later
coordinate change catches up on the next job while a streaming dump is
never patched under). The backup CR is minted only once the operator
reports the storage pushed into pbm (`status.observedGeneration` caught
up and `PBMReady` True): pbm writes to what its own config says, not to
the spec, so a CR minted before that push would land at the previous
coordinates while its Backup recorded the new ones; the job waits at
`PerconaServerMongoDBPBMConfigPending` meanwhile, bounded by the backup
deadline. Once a job has its CR, none of the mint-time reads or holds
can fail it: the strategy and app reads the artifact still needs are
bounded by the 24h running ceiling instead, and the cluster is not read
at all **and on the release having rendered that flag**: the app CR is
projected from the HelmRelease's desired values, so the flag reads true
before helm-controller renders the revision, while the tenant's own
storage, scheduled tasks and PITR stream from the previous render still
sit on the cluster. As long as the cluster carries `spec.backup.tasks`
or `spec.backup.pitr`, the driver does not inject and holds the
BackupJob at `Ready=False PerconaServerMongoDBLegacyRender` (failing it
at the deadline), so a wedged legacy revision never has its nightly dump
and oplog redirected into the shared bucket. The backup snapshot falls
back to the strategy coordinates when the operator status echo is empty;
that fallback is gated on the CR's own `percona.com/delete-backup`
finalizer (the flow the CR was minted on), not on the app flag re-read
at completion, so a legacy backup never records the platform bucket and
a flag flipped mid-dump does not change the artifact. The restore path
rebuilds `backupSource` from it (the `cozy-backups-creds` reference is
projected into the restore namespace before dispatch).
- **Retention/cleanup**: on the `useSystemBucket` flow the driver owns
the shared-bucket archive (the chart's psmdb `tasks[].keep` retention is
gated out, and an on-demand backup is not covered by task retention). It
stamps the operator's `percona.com/delete-backup` finalizer on each
backup it mints **against a storage that carries the platform
coordinates** — the strategy's bucket and `credentialsSecret`, read back
from the live cluster after the injection — never on the flag alone: a
strategy without an `s3` block, or a storage that points anywhere else,
holds the job (`MongoDBStrategyHasNoS3` /
`PerconaServerMongoDBStorageForeign`) instead of minting a finalized
dump against a bucket the platform does not own. On Cozystack `Backup`
deletion (by hand today; by platform retention once it lands, since a
`Plan` only creates BackupJobs and never deletes a `Backup`) the driver
deletes the operator `PerconaServerMongoDBBackup` CR so that finalizer
frees the object — the finalizer's presence is the ownership marker
cleanup reads back. That reading is unambiguous because cleanup resolves
the CR by the name recorded in `driverMetadata` (the one the driver
minted), not by scanning: the psmdb operator does stamp
`percona.com/delete-backup` on its own `cron-*` task-minted backups, but
those are never in scope. A legacy backup keeps the pre-existing
hands-off contract, decided from the snapshot before any read of the
operator CR. A terminating namespace or the
`backups.cozystack.io/skip-artifact-cleanup: "true"` annotation strips
the driver's finalizer and releases; a `Get` of the operator CR that
fails with anything other than NotFound/NoMatchError still requeues, and
the operator's own `release-lock` finalizer can hold the CR while the
operator is down, so those two are what can keep a namespace in
`Terminating`. `schedule` and `retentionPolicy` are inert on this flow
(there is no engine-side retention and a `Plan` has no retention field
yet); platform-level retention is tracked in
[#4192](https://github.com/cozystack/cozystack/pull/4192) and teardown
GC of orphaned backups in
[#3966](https://github.com/cozystack/cozystack/issues/3966), so until
those land bound `cozy-backups` with an S3 lifecycle policy.
- **Bounds on the waits**: the injection race with the operator's
cluster cache (an `unable to get storage` error on a freshly injected
storage) is retried by deleting the errored CR and minting a fresh one,
with the retry count kept on the BackupJob
(`backups.cozystack.io/mongodb-storage-race-retries`) because the
deleted CR is reaped before the next poll; after three retries the
operator's error is terminal. A system-bucket dump that has started
streaming is not cancelled on the 30-minute deadline, but fails after 24
hours in `running` (`BackupAbandoned` warning naming the operator CR,
which is left in place as the handle to whatever pbm still writes). The
deadline cancel of a not-yet-started CR pins its delete to the
resourceVersion the live read answered for.
- **RBAC**: `patch` on `perconaservermongodbs` for the injection
(server-side apply), and `create`/`delete`/`patch` on
`perconaservermongodbbackups` — `delete` to prune the shared-bucket
archive on `Backup` deletion, `patch` to strip the driver's finalizer
when releasing a wedged cleanup.

Coverage: Go unit tests over `internal/backupcontroller` — the injection
gate keyed off the app flag (legacy cluster never patched) and off the
applied render (a cluster still carrying tasks or pitr is held,
untouched), the SSA-injection precondition with the whole entry
(per-application prefix included) asserted on the cluster, strategy
templating per application, the finalizer refused against a storage
without platform coordinates (strategy without `s3` over a legacy entry,
foreign coordinates on the merged entry), the retry budget driven with
the operator simulated reaping the deleted CR between polls, the restore
snapshot fallback and credential swap across all four legacy/system flow
combinations, the finalizer-owned cleanup with its escape hatches (skip
annotation before any read, `NoMatchError` treated as gone,
terminating-namespace release using the pre-read CR, a failed strip
surfaced as an Event, a legacy CR never deleted, a legacy snapshot
released without a read, a system-bucket snapshot reaching the prune),
the deadline cancelling a not-yet-running operator backup only on a
positive live read (including no live client at all, and a CR that moved
since the read), and the running ceiling. Plus helm-unittest on both
charts (chart skips the creds Secret and omits
`spec.backup.storages`/`tasks`/`pitr` on `useSystemBucket` and keeps
them on the legacy flow; the strategy renders the S3 coordinates and the
`insecureSkipTLSVerify` knob). A live `useSystemBucket` backup+restore
e2e against the platform bucket is a follow-up (needs the system bucket
wired in CI, same status as the Postgres/ClickHouse useSystemBucket
e2e). One item to confirm in e2e: the psmdb operator accepting
`spec.backup.enabled=true` with an empty `storages` between install and
the first BackupJob.

### Screenshots

Not applicable (no UI changes).

### Downstream repositories

The tenant-facing schema surface is one new field,
`spec.backup.useSystemBucket`, on the MongoDB `values.schema.json` / CRD
(regenerated), plus a validation rule on `spec.backup`.
`terraform-provider-cozystack` hand-models MongoDB and may want to
expose it, but it is a new optional boolean that defaults false (no
drift for existing users), and the chart `README.md` regenerates on
release. Leaving every box unticked per the trigger-map guidance so a
maintainer decides on the provider follow-up.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
feat(mongodb): MongoDB backups now support the platform system bucket via backup.useSystemBucket, so the default cozy-default BackupClass works without per-app S3 credentials — the backup driver injects the system-bucket storage onto the cluster at backup time, once the release has rendered the flag. On this flow nothing prunes the shared bucket automatically yet (a Plan carries no retention field; tracked in #4192 and #3966), so an S3 lifecycle policy on cozy-backups is a prerequisite for enabling it, not a suggestion. useSystemBucket requires backup.enabled=true; the chart refuses the combination with backups disabled at render time, and the schema carries the same rule as a marker until the apiserver enforces CEL (#2657). A system-bucket dump still running after 24 hours fails its BackupJob with the operator backup left in place. A MongoDB restore now adopts the target cluster's S3 credentials only for a storage on the same bucket as the archive (it used to adopt them unconditionally); a legacy restore into a target on a different bucket that relied on one key pair covering both must keep or re-create the source release's -s3-creds Secret by name, and is held at Ready=False RestoreCredentialsMissing until it exists.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
  * Added opt-in system bucket mode for MongoDB backups.
* Automatically injects platform bucket storage without
application-managed S3 credentials.
* Supports endpoint, region, path prefix, addressing, and TLS settings.
* Manages shared-bucket backup artifacts when system bucket backups are
deleted.

* **Bug Fixes**
* Improved restore credential selection to avoid cross-bucket
mismatches.
  * Preserved backup metadata when storage details are unavailable.

* **Documentation**
* Documented configuration, scheduling, retention, PITR limitations, and
restore guidance.

* **Validation**
* Prevented incompatible bootstrap and system bucket restore
configurations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 18:10:56 +03:00
Mattia Eleuteri e510d49b3d feat(monitoring): re-render node agents when a monitoring label changes
The node fluent-bit builds its tenant routes from a lookup of namespace
labels, and helm-controller upgrades a release only when its chart or
values digest changes. A tenant turning monitoring on therefore got no
route, and a removed one kept its route, until an unrelated change
upgraded monitoring-agents.

The reconciler digests the label across all namespaces and requests a
forced upgrade of the release when the digest moves. It digests every
labelled namespace rather than repeating the chart's selection, so it
cannot drift from the chart; the cost is an occasional upgrade that
renders the same config.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-09-29 17:10:55 +02:00
Mattia Eleuteri cdff5ed380 feat(monitoring): route tenant logs to the tenant's own VictoriaLogs
A tenant that runs its own monitoring stack gets a VictoriaLogs instance,
but the node fluent-bit sends every container log to global.target, so
that instance stays empty. The tenant chart already names the stack a
namespace reports to in namespace.cozystack.io/monitoring, the label the
per-tenant vmagent selects on, so the log path keys off the same label.

The routed record leaves the default outputs (keep false): a tenant with
its own monitoring holds its logs, the root store no longer does. The
re-tagged record re-enters the filter chain, where the Match * "Add
tenant" cannot overwrite the key, so a per-target Set labels it with the
tenant it is sent to. One rewrite_tag carries every rule, keeping a
single emitter however many tenants are routed.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-09-29 17:10:54 +02:00
Mattia Eleuteri 874590e200 test(keycloak): pin organizationsEnabled rendering in configure-kk
Cover both paths of the new value: the chart default renders
spec.organizationsEnabled: false (no behaviour change), and setting
organizationsEnabled: true renders through to the ClusterKeycloakRealm.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-09-29 17:10:10 +02:00
mattia-eleuteri adb2918daf feat(keycloak): expose organizationsEnabled realm toggle
The keycloak-configure chart hardcoded the realm with the Organizations
feature off, with no value to enable it. Operators who flip
organizationsEnabled in the Keycloak admin console see it reverted on the
next ClusterKeycloakRealm reconcile, since the EDP operator pushes the
chart-rendered spec back to the realm.

Expose organizationsEnabled in values.yaml (default false, no behaviour
change) and map it to ClusterKeycloakRealm.spec.organizationsEnabled so
the feature can be enabled declaratively and survive reconciles.

Assisted-by: LLM
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
2026-09-29 17:10:09 +02:00
Mattia Eleuteri f7c04b7a39 feat(kubernetes): expose kernel modules for Talos worker pools
A Talos system extension installs a kernel module but does not load it;
loading is machine.kernel.modules' job. The worker machine config had no
such block and no values key could add one, so a GPU pool on Talos
workers was unusable as shipped: the extension supplied the NVIDIA
module, nothing loaded it, and the failure was silent end to end. The VM
held the PCI device, the node advertised no GPU, and no component logged
an error.

The field is three-state, and the states are distinguishable because it
carries no default: absent lets the chart choose (a pool holding at least
one nvidia.com/* GPU gets nvidia, nvidia_uvm, nvidia_drm, nvidia_modeset,
in that order, since Talos loads the list in sequence and the last three
depend on the first; any other pool gets nothing), a non-empty list
replaces that choice outright, and an explicit [] opts out even on a GPU
pool. A default of [] would collapse absent into opted-out for every pool
and make the automatic set unreachable.

The automatic nvidia entry carries NVreg_NvLinkDisable=1. Individual
GPUs are passed into worker VMs without their NVSwitches, and on
non-Talos workers the gpu-operator driver container applies that
parameter; on Talos that container has to be off, so nothing else would.

Names and parameters reach the reconcile Job's unquoted heredoc, so they
are validated at render and the emitted list is rebuilt from those two
fields alone: the schema does not prune unknown keys, and any other key
passed through toYaml would reach the shell unchecked.

A pool that resolves to no modules renders the machine config it
rendered before, byte for byte, so its content-hash Job name does not
rotate.

Refs: #3563
Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-09-29 17:09:42 +02:00
Mattia Eleuteri 669fb2416a feat(kubernetes): default gpu-operator to an OS-provided driver on Talos
Closes the gpu-operator half of #3563. Tenant clusters have run Talos
workers since 1.6.0, where the NVIDIA driver comes from a system extension
baked into the worker image rather than from the operator's driver
container. The addon default still described the Ubuntu shape, so a GPU
node group came up with the operator fighting the OS for the driver and no
path at all for CUDA libraries to reach a tenant pod.

Three things change, each fixing one silent failure:

driver.enabled and toolkit.enabled go false. The driver container would
load its own module and clash with the extension's; the toolkit container
would rewrite /etc/containerd/config.toml, which Talos owns and which
already registers the nvidia runtime with enable_cdi = true and
cdi_spec_dirs = ['/run/cdi'].

validator.driver.env points DRIVER_ROOT and NVIDIA_DRIVER_ROOT at / and
disables /dev/char symlink creation. The driver validator otherwise looks
for a driver container install, finds none, never writes its sentinel, and
leaves the device plugin blocked behind toolkit-validation — the loop on
"Attempting to validate a driver container installation".

devicePlugin.env selects cdi-annotations and points NVIDIA_CTK_PATH at the
host's nvidia-ctk. Tenant pods carry no runtimeClassName and did not need
one on Ubuntu, where the toolkit made nvidia the default containerd
runtime; Talos does not, so CDI is the only route for the CUDA libraries.
DEVICE_ID_STRATEGY, NVIDIA_DRIVER_ROOT and CDI_ANNOTATION_PREFIX are set
and tested alongside it: CDI device names key on the UUID strategy, the
plugin resolves the driver from NVIDIA_DRIVER_ROOT, and the annotation
prefix has to be the one containerd reads, so a regression in any of them
breaks device resolution on its own.

The previous default fed NVreg_NvLinkDisable=1 to the driver container
through the nvidia-kernel-module-params ConfigMap. With that container off
the ConfigMap is mounted by nothing, so the default is removed. On Talos a
module parameter belongs to the worker machine config, which the
kernelModules field of the KubernetesNodes app
(packages/apps/kubernetes-nodes, #3571) writes. The wrapper chart keeps
the ConfigMap for an operator who re-enables the driver container.

Every value here is the configuration validated on a production Talos GPU
node group, minus the parts that only mattered while the driver container
was in play.

Not covered: HAMi. When addons.hami.enabled the operator's device plugin
stays off and HAMi serves the GPUs, and HAMi's own plugin needs equivalent
CDI wiring on Talos that this default does not attempt.

Refs: #3563

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-09-29 17:09:13 +02:00
Mattia Eleuteri bb3ddee233 fix(rabbitmq): bound the topology operator Secret cache
messaging-topology-operator 1.14.2 caches every Secret in the cluster: its
ClusterRole grants secrets list/watch cluster-wide with no namespace or label
restriction, so the controller-runtime cache holds the whole set. Against the
vendored 500Mi limit this OOMKills on any cluster that has grown enough
Secrets, and it never recovers, because the initial LIST alone no longer fits.
While it is down no topology CR reconciles, for any tenant.

Observed on a cluster with 8211 Secrets across 291 namespaces (293 MB, 87% of
it Helm release history): a flat 333Mi working set for days, then the ceiling,
then a permanent crashloop. The operator was caching 8211 Secrets to read 2.

Upstream fixed this by gating the Secret informer on a label, released in
v1.20.0, and by bypassing the cache for Service reads, released in v1.20.1.
Refresh the vendored manifest to v1.20.2, the current patch release carrying
both, through the package's existing make update pipeline.

The bump alone would break every install: the app chart's credentials Secret is
referenced by User.spec.importCredentialsSecret, and from 1.20 the admission
webhook rejects a user-provided Secret that does not carry
rabbitmq.com/topology-operator=true. Label it in the same change, pin the
label in the chart's unit tests, and give the e2e RabbitMQ a user so the
webhook and the User reconcile are exercised against the real operator.

The RabbitmqCluster default-user Secret needs no label, it is read uncached via
apiReader, and the Status.Binding contract is unchanged since 1.14.2, so the
vendored cluster-operator 2.9.0 stays compatible and does not move here.

Refs #4128

Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Assisted-by: LLM
2026-09-29 17:08:17 +02:00
Mattia Eleuteri 7c9ab0e28e fix(cozy-lib): round the sanitized CPU request up to a whole milli-CPU
cozy-lib.resources.sanitize divides the CPU limit by cpuAllocationRatio
and rendered the quotient as a bare float. When the ratio does not
divide the value evenly the result is finer than a milli-CPU, and the
VPA admission webhook rejects it outright:

  admission webhook "vpa.k8s.io" denied the request:
  minAllowed: CPU [62500u] must be a whole number of milli CPUs

packages/extra/etcd routes its VPA minAllowed/maxAllowed through the
helper, so 250m / 4 = 62.5m left the etcd HelmRelease looping
UpgradeFailed -> RollbackFailed forever: the rollback target renders the
same invalid value. Since v1.6.0 HelmReleases wait on each other, so
that one stuck app also blocks tenant-root, cozystack-basics and
kubevirt-cdi-operator, and without CDI the Talos worker rollover cannot
import the worker OS image. Only the divisors of 250 worked as ratios,
which is an artefact of one chart's hardcoded floor rather than a real
constraint on the operator.

The request is rounded up, never down, so it stays at or above the
intended fraction of the limit. It is rounded only when it has to be:
if the milli-CPU value equals its own ceiling, the float keeps its
current rendering, so nothing changes on the default ratio of 10 or on
any ratio that divides evenly; otherwise it renders as the ceiling in
m. The comparison is exact, with no tolerance. sprig's divf and mulf run
on shopspring/decimal rather than float64, so mulf (divf 0.25 10.0)
1000.0 renders exactly 25 and there is no binary-float noise on this
codepath to absorb. A tolerance would reintroduce the bug: a value
whole-milli only to within it, such as 100.0000004m at ratio 1, would
read as exact and render "0.1000000004", rejected for the same reason
62.5m is. The same predicate covers sub-milli input, since ceil carries
any positive request to at least 1m, and a zero request stays "0".

Fixes #2917

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-09-29 17:04:00 +02:00
Mattia Eleuteri 25cdd51881 feat(postgres): allow preloading an allowlist of shipped modules
shared_preload_libraries is blocked in postgresql.parameters, and
rightly so: the parameter is free text and accepts a path, so it can
point at a writable location such as the PGDATA volume. What the block
leaves is an indirect route. CloudNativePG preloads pg_stat_statements,
auto_explain and pgaudit by itself as soon as a parameter under their
prefix is set, such as pg_stat_statements.track, so enabling one of
them today means setting one of its tuning parameters, even to its
default value. pg_prewarm has no route at all, and its autoprewarm
worker only runs when the module is preloaded.

postgresql.sharedPreloadLibraries is the explicit switch. It is a
closed enum of four module names that already ship in the CloudNativePG
PostgreSQL image, rendered onto the CNPG Cluster field of the same
meaning, which the operator merges with the libraries it preloads on
its own. A bare name can only resolve inside $libdir, which sits on the
read-only image filesystem, and dynamic_library_path stays blocked, so
no path can be introduced. The blocklist on postgresql.parameters is
left untouched.

The template repeats the enum as a backstop for renders that skip
schema validation. helm-unittest always validates the schema and never
reaches it, so a bats suite renders with --skip-schema-validation to
keep the backstop, and its agreement with the enum, under test.

Verified on a live CloudNativePG 1.27.3 cluster: with the field set,
the operator writes shared_preload_libraries into postgresql.conf,
SHOW reports the modules, and pg_stat_statements returns rows.
Changing the list restarts PostgreSQL on every instance, the primary
included, through the instance manager and without restarting the pod.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-09-29 17:02:44 +02:00
Aleksei Sviridkin 0a2054e228 fix(kubernetes): apply worker TalosConfigTemplate server-side (#4163)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The `talos-reconcile` Job runs under a 256Mi limit and applies its
`TalosConfigTemplate` with a client-side apply. Building the merge patch
means pulling the cluster's full OpenAPI schema down, which on a
management cluster carrying a few hundred CRDs costs around 290Mi of
RSS, so the kernel OOMKills kubectl before the Job finishes.

Only the merge path pays that cost, which is what makes this hide so
well. The first run creates the object and stays cheap, so a fresh
install goes green; every reconcile afterwards is an update, so the Job
CrashLoopBackOffs on every chart upgrade until it burns its
`backoffLimit: 30`. The KamajiControlPlane patch lands before the apply
dies, so the cluster keeps working and the failure reads as cosmetic
noise.

Measured against a 438-CRD management cluster with the Job's own kubectl
(`alpine/k8s:1.33.4`), cold discovery cache:

| command | peak RSS |
| --- | --- |
| `get -o jsonpath` | 52Mi |
| `apply`, object absent (create) | 52Mi |
| `apply`, object present (merge) | **292Mi** — OOMKilled at 256Mi |
| `apply --server-side` | 56Mi |

`--validate=false` changes nothing, so the cost is the merge strategy
lookup rather than validation. Server-side apply leaves the merge to the
apiserver and the client never needs the schema. `--force-conflicts`
keeps the Job authoritative over an object an earlier client-side apply
created.

Raising the memory limit instead would work today and rot tomorrow: the
cost tracks the cluster's CRD count, so the ceiling would need lifting
again as the API surface grows.

Includes a regression test asserting the rendered Job applies
server-side and carries no client-side fallback. It fails on the pre-fix
template.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(kubernetes): apply the worker TalosConfigTemplate server-side so the talos-reconcile Job is no longer OOMKilled on clusters with a large CRD count. The Job previously failed on every reconcile after the first install, which meant on every chart upgrade.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved Kubernetes reconciliation reliability by using server-side
updates for Talos configuration templates.
* Prevented reconciliation jobs from exceeding their memory limit during
chart upgrades.
* Ensured the reconciliation job remains authoritative when updating
previously created resources.

* **Tests**
* Added coverage verifying server-side application behavior and conflict
handling.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 17:37:49 +03:00
Aleksei Sviridkin 8b861668b8 ci(release): build pre-release images for amd64 and arm64 and gate releases on it (#4505)
## What this PR does

Pre-release builds now publish every image as an amd64 and arm64 index,
and a release fails if anything it ships is single-arch. Main, release
lines and pull requests keep building amd64 only. It builds on #4498,
which made every image buildable for both architectures. Part of #1961.

**This PR is on hold until everything under "Merge after" has landed.**
Merged earlier, the new gate fails the next rc on the images those PRs
fix.

### How a pre-release is built

A pre-release tag gets a second job, for arm64, next to the existing
release job. That covers `-rc.N`, `-beta.N` and `-alpha.N` tags. It runs
on the CNCF arm64 pool (`oracle-vm-24cpu-96gb-arm64`), builds the same
images with `PLATFORM=linux/arm64`, and pushes them as `<tag>-arm64`.
`prepare-release` waits for it, and if the arm64 job fails the rc fails
with it. There is no amd64-only fallback. Stable tags do not run it:
promotion copies the rc digests with `skopeo copy --multi-arch all`, so
a stable release inherits the indexes.

After the amd64 build, the stitch joins each amd64 image with its arm64
twin into one index. It lives in `hack/stitch-multiarch.sh`. Every tag
of the amd64 image moves to the index, the component-versioned ones
included. The script then rewrites the digests in the tree and
republishes the packages artifact and the installer chart pinned on
them. It reads refs through `hack/lib/image-refs.sh`, and it fails if an
old digest survives anywhere outside `charts/`.

### The gate

A new check fails the rc when a pinned digest in the tree is not an
index with both linux/amd64 and linux/arm64. It lives in
`hack/verify-multiarch.sh`. It checks first-party and third-party refs
alike. Images that are amd64 by nature go into
`hack/multiarch-allowlist`, one repository per line, each with a
mandatory reason. Today that is only the e2e sandbox. An entry that
matches nothing is reported as stale.

The static gate only sees digest-pinned refs in the tree. A tag-only
ref, an image from a vendored chart default, or one an operator starts
at runtime is invisible to it. So the rc e2e also audits every image its
nodes pulled, with the same check and the same allowlist. Neither covers
a package the e2e never installs.

### Nightly

A new nightly workflow builds main for arm64. It keeps the arm64 build
cache warm, since the rc job only reads it. It also prints the
single-arch refs that would block the next rc. Nothing it builds is
published.

### Smaller changes

- `CACHE_TAG` in `hack/common-envs.mk` moves the default cache ref, so
the arm64 build keeps a cache of its own.
- `MATRIX_ARCH=arm64` makes `hack/build-matrix.sh` leave out the
amd64-only e2e sandbox.
- The kamaji provider image is pushed under the build's `IMAGE_TAG` like
every other image. Its Makefile set `IMAGE_TAG` itself, so concurrent
builds overwrote each other's component tag. Closes #4503.
- `hack/nightly-mirror.sh` verifies a copy against the raw manifest
digest. Plain `skopeo inspect` resolves an index to one platform's
child.
- matchbox is rebuilt for both platforms in the amd64 release job. The
arm64 leg skips the talos package, so there is no arm64 half to stitch,
and its Dockerfile only copies files, so no QEMU is needed. Releases
then network-boot arm64 machines too. Closes #4524.

The stitch also rewrites the kamaji ref inside `files/components.gz`,
recompressed with `gzip -n` so the bytes are reproducible, so the kamaji
control-plane provider ships multi-arch too. keda and kuberture now name
the repository next to their pinned digest, so the gate can resolve
them; their rendered manifests do not change.

### Merge after

- #4552 builds the Talos installer, matchbox, Harbor and the Velero
KubeVirt plugin for arm64, and adds flux-plunger, keycloak-operator,
kilo and migration-controller to the root `build:` list. It replaces
#4507, #4515, #4525 and #4528.
- #4549 moves ingress-nginx, cozy-proxy, cozystack-scheduler and
keycloak-kms-proxy to their multi-arch releases, which are already
published.

The rest of the stack is merged: #4485, #4498, #4512, #4517, #4521 and
#4522 here, and the multi-arch PRs in
ingress-nginx-with-protobuf-exporter, cozy-proxy, cozystack-scheduler
and keycloak-kms-proxy.

### Verification

The bats suites pass: stitch, verify-multiarch, build-matrix,
common-envs, nightly-mirror and the release contracts. The first three
also pass under `hack/cozytest.sh` with dash. Each new test was red
before its implementation, and the verify-multiarch and stitch tests
each have a mutation that turns them red. I ran `verify-multiarch
--report` against main. It lists the first-party refs the stitch will
fix, plus the third-party and special cases above. The workflows have
not run yet. The arm64 job's tools and duration, the stitch against real
registries, and the audit are checked for the first time on the nightly
and on the next rc.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Nothing under `hack/` is moved or renamed, and no make target changes
its default behaviour: `CACHE_TAG` and `MATRIX_ARCH` are opt-in. The
satellite repositories listed under "Merge after" are prerequisites, not
follow-ups this change forces on them.

### Release note

```release-note
ci(release): pre-release builds publish every image as an amd64 and arm64 multi-arch index, and a release fails if any image it ships or pulls in e2e lacks either architecture. Stable releases inherit the indexes through promotion.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Release candidates now build amd64 and arm64 images in parallel and
combine eligible images into verified multi-architecture indexes before
publication.
* A nightly arm64 image build is available for testing and supplies a
build cache for release-candidate builds.
* **Bug Fixes**
* End-to-end checks audit whether pulled images support both
architectures. Audit failures block prerelease checks, while stable
releases continue with a warning.
* End-to-end test artifacts now include collected image references and
multi-architecture audit results.
* **Documentation**
* Updated release and image guidance covers multi-architecture builds,
verification checks, and troubleshooting, including arm64 build and
stitching failures.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
v1.7.0-alpha.2
2026-09-29 15:35:13 +03:00
Mattia Eleuteri 31948c8838 fix(api): redo the HelmRelease metadata carry-over on a conflict retry
A 409 on the HelmRelease PUT means the live object changed after the
read the carry-over was built from, and the writer behind it is normally
helm-controller - the controller whose finalizer the carry-over exists
to protect, and which re-adds it on the reconcile following a
metadata-only update. Retrying with the metadata read before the
conflict wrote that stale copy back and dropped the finalizer the
reconcile had just restored, reopening the orphaning window the
carry-over closes.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-09-29 15:10:03 +03:00
mattia-eleuteri 6d541c63eb fix(api): make mergeMaps always allocate, never return an input map
mergeMaps returned one of its inputs whenever the other was nil. Create
and Update seed the HelmRelease labels with
mergeMaps(r.releaseConfig.Labels, ...) and then write the application
metadata labels into the result in place, so for an Application with no
labels the writes landed on the long-lived config map shared by every
request for the kind: requests cross-contaminate (application.name
varies per request) and concurrent map writes crash the apiserver.

Signed-off-by: mattia-eleuteri <mattia@hidora.io>
2026-09-29 15:09:46 +03:00
mattia-eleuteri a18cbc581c fix(api): preserve foreign HelmRelease metadata on Application update
Update rebuilds the HelmRelease from the Application and issues a
full-replace PUT, which stripped metadata owned by other controllers:
finalizers, ownerReferences, and any labels or annotations outside the
apps.cozystack.io- prefix. Losing helm-controller's
finalizers.fluxcd.io finalizer is the harmful case: a delete landing
before helm-controller re-adds it removes the HelmRelease without
running helm uninstall, orphaning every resource of the release (VMs,
databases, PVCs). A label patch immediately followed by a delete - a
common client pattern - hits that window almost every time.

Fetch the current HelmRelease before the PUT and carry over its
finalizers, ownerReferences, and non-prefixed labels and annotations.
Keys under the apps.cozystack.io- prefix remain owned by the API and
are recomputed from the Application, so label/annotation deletions
still propagate.

Signed-off-by: mattia-eleuteri <mattia@hidora.io>
2026-09-29 15:09:46 +03:00
Aleksei Sviridkin d9ed566326 fix(migration): restore the class reclaim policy on imported volumes (#4480)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The VM import handoff sets the transferred `PersistentVolume` to
`Retain` so the data survives the swap from the Forklift-populated claim
to the VMDisk's claim, and then never gives the policy back. Every
imported disk therefore ends up with `persistentVolumeReclaimPolicy:
Retain`, while its StorageClass says `Delete` and every VMDisk created
directly carries `Delete`. When the tenant later deletes an imported
VMDisk through the normal path, its DataVolume and claim go away, but
the volume stays `Released` with its backing storage: an orphaned,
billed volume that no tenant can see or remove.

#### How it was measured

On a lab cluster with a LINSTOR-backed `replicated` class
(`reclaimPolicy: Delete`): a 16 GiB VM imported from vSphere produced a
VMDisk whose PV was `Retain`, while every non-imported VMDisk's PV in
the cluster was `Delete`. Deleting the VMInstance and then the VMDisk
removed the DataVolume and the claim and left the PV `Released`, still
`Retain`, with its LINSTOR resource in place. Patching that PV back to
`Delete` made the CSI provisioner delete both the PV and the LINSTOR
resource, which confirms the reclaim policy is the only thing keeping
it.

#### The fix

`Retain` is now held only for the window it protects. A new step,
`settleReclaimPolicy`, restores the policy the volume's StorageClass
declares (`Delete` when the class names none or no longer exists) once
the handoff is provably over, and until then reports "not yet" and
touches nothing:

- the replacement claim is `Bound` and the PV is `Bound` with a
`claimRef` naming that claim by UID, as the PV controller reports it
rather than as the controller wrote it, so no release is in flight;
- the transferred claim is gone;
- the claim's one and only owner is the VMDisk's DataVolume, as a
controller reference whose UID matches the live DataVolume (CDI's
adoption), so the only deletion that can reach the volume is the
VMDisk's own;
- the VMDisk exists and carries this task's output markers.

`fulfill` runs this step after adopting every disk and before creating
the VMInstance and deleting the scaffolding VM, so a VM is reported
`Succeeded` only once its volumes carry their class's policy, and an
import that fails after this point still leaves disks that clean up like
any other. The step is a compare-and-set, so repeated passes and
restarts are no-ops once settled. It also runs on the path that recovers
outputs whose status was never written, for tasks caught mid-handoff by
an upgrade. The comment that called permanent `Retain` "part of the
contract" and the matching paragraph in
`packages/system/migration-controller/README.md` are updated; the README
also says how to fix volumes left on `Retain` by imports completed
before this change, since those tasks are terminal and are not
reconciled again.

In practice a VM now stays in `Creating` for one more requeue (about 20
s) after the swap, while the PV controller rebinds and CDI adopts the
claim.

#### Tests

- `TestSettleRestoresTheClassReclaimPolicy`: a finished handoff gets the
class's policy back: `Delete`, `Retain` when the class asks for it, and
`Delete` when the class names none or is gone.
- `TestSettleIsIdempotent`: a second pass over a settled volume reports
settled and does not rewrite the PV.
- `TestSettleKeepsRetainWhileTheHandoffIsOpen`: thirteen cases, each
breaking one condition (claim not created by this import, claim not
bound, PV still `Released`, `claimRef` still on the old claim,
`claimRef` on the new claim with a stale UID, transferred claim still
present, claim not yet adopted by CDI, an extra owner, a DataVolume
owner that is not the controller, an owner reference to a previous
DataVolume, DataVolume missing, VMDisk missing, VMDisk not created by
this import). Every one keeps `Retain`.
- `TestFulfillRestoresTheReclaimPolicyBeforeFinishing`: behavioural
regression over `fulfill`. The first pass swaps the volume, keeps
`Retain`, creates no VMInstance and keeps the scaffolding VM. After
simulating the PV controller and CDI, the second pass restores `Delete`,
creates the VMInstance and removes the scaffolding VM. With the new step
turned into a no-op, this test fails.
- `TestFulfillSettlesReclaimedOutputs`: the recovery path for outputs
whose status was never written. With the handoff still open the volume
keeps `Retain`; once it is finished the policy goes back to `Delete`.
- The existing `adoptVolume` tests still assert `Retain` during the
swap.

`go test ./internal/migrationcontroller/...` and `make test-controllers`
pass, `go vet` is clean, and `make generate` leaves the tree unchanged.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff touches only `internal/migrationcontroller/`,
`packages/system/migration-controller/README.md` and a comment in
`packages/system/migration-controller/templates/rbac.yaml`, which match
no entry of the trigger map; the website's VMware migration page does
not describe the reclaim policy, so it needs no change.

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(migration): disks imported from VMware no longer keep a Retain reclaim policy after the import. The volume gets its StorageClass policy back once it is bound to its VMDisk, so deleting an imported VMDisk deletes its volume instead of leaving an orphaned Released PersistentVolume. Volumes imported before this fix keep Retain and have to be patched back by hand.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Imported volumes are protected during claim transfers, then returned
to their StorageClass reclaim policy after the replacement claim and
volume are safely adopted. If the class or policy is unavailable, the
policy defaults to `Delete`.
* VM imports wait for volume adoption and reclaim-policy settlement
before completing, helping prevent premature volume deletion and
incomplete imports. Previously unrecorded handoff outputs are also
settled before being returned.
* **Documentation**
* Clarified reclaim-policy behavior during imports and how to restore
cleanup behavior for volumes from older imports.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 14:41:04 +03:00
Mattia Eleuteri 1728b49e0c fix(kubernetes): apply worker TalosConfigTemplate server-side
The talos-reconcile Job runs under a 256Mi limit and applies its
TalosConfigTemplate client-side. Building the merge patch means pulling
the cluster's full OpenAPI schema down, which on a management cluster
carrying a few hundred CRDs costs around 290Mi of RSS, so the kernel
OOMKills kubectl before the Job can finish.

Only the merge path pays that cost. The first run creates the object and
stays cheap, so a fresh install succeeds and the Job then
CrashLoopBackOffs on every reconcile afterwards, which means on every
chart upgrade. Measured against a 438-CRD cluster with the Job's own
kubectl (alpine/k8s:1.33.4), cold discovery cache:

  get -o jsonpath                    52Mi
  apply, object absent (create)      52Mi
  apply, object present (merge)     292Mi   <- OOMKilled at 256Mi
  apply --server-side                56Mi

--validate=false changes nothing, so the cost is the merge strategy
lookup rather than validation.

Server-side apply leaves the merge to the apiserver and the client never
needs the schema. --force-conflicts keeps the Job authoritative over an
object an earlier client-side apply created.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-09-29 14:38:56 +03:00
Aleksei Sviridkin 717d091970 fix(backups, postgres): start a Postgres restore from the backup it names (#4443)
## What this PR does

A Postgres `RestoreJob` now starts from the base backup its `Backup`
refers to, and the PITR documentation describes the window the recovery
actually has.

**Starting from the named backup.** Without `recoveryTarget.backupID`
the barman-cloud plugin picks the base backup itself
(`internal/cnpgi/restore/restore.go` in plugin-barman-cloud, then
`FindBackupInfo` in barman-cloud `pkg/catalog`): the newest backup of
the catalog when no `recoveryTime` is set, otherwise the newest one
whose end is not after `recoveryTime`, on any timeline since no
`targetTLI` is passed. A `RestoreJob` naming an older `Backup` therefore
started from another one, and on an archive that holds sibling timelines
— left by in-place restores that archived back under the source
`serverName`, before restored clusters got a fresh `newServerName` — it
could start from another branch and bring back that branch's data.
`FindBackupInfo` gives `backupID` precedence over time-based selection,
and CNPG's webhook does not count it among the mutually exclusive
targets, so it combines with `targetTime`.

- on restore the CNPG driver reads barman's `backupId` and the backup's
`stoppedAt` from the `Backup`'s `cnpg.io/Backup` and sets the new chart
value `bootstrap.backupID`, rendered as
`bootstrap.recovery.recoveryTarget.backupID`;
- it does not pin for a `recoveryTime` earlier than one second past
`stoppedAt`: the named backup cannot serve an instant before it is
consistent, and `stoppedAt` is truncated to the second while the catalog
compares barman's end time to the microsecond. Those requests keep the
plugin's time-based choice, as today;
- it does not pin either when the `cnpg.io/Backup` is gone.
plugin-barman-cloud's retention deletes it together with the base backup
it describes (`deleteBackupsNotInCatalog`), and the driver deletes one
only with its artifact, so a missing one means the catalog has lost that
base backup: pinning it would purge the target and then fail the
recovery. The plugin's choice recovers to the same end of the archive,
and the `RestoreJob` gets a `BaseBackupGone` warning event. A base
backup deleted from the object store by other means is still pinned for
as long as its `cnpg.io/Backup` stays, and that restore fails after the
purge; plugin-barman-cloud deletes such a `cnpg.io/Backup` on its next
maintenance pass (`deleteBackupsNotInCatalog`, every
`retentionPolicyIntervalSeconds`, retention policy or not), but only
from the primary of the cluster that took it, under the same UID.
Telling it apart earlier would need reading the barman catalog, which
the driver does not do; the documentation says so, and that deleting the
`cnpg.io/Backup` by hand is the way out;
- a re-restore into the same target clears a `backupID` left by a
previous one.

**Documentation** (`docs/operations/backup-classes.md`, PostgreSQL
PITR):

- the upper bound of the window is the last transaction (commit or
abort) in the archived WAL, not the newest segment: recovery ends just
before the first transaction after the target, and a segment switch or a
base backup is not one. On a database at rest every instant after its
last write fails with `recovery ended before configured recovery target
was reached`, including "at/before the most recent completed backup",
which the page recommended as the safe choice; the state as of the last
write is obtained by omitting `recoveryTime`;
- aiming at a backup's `stoppedAt` misses that backup (the plugin keeps
only a backup whose end is not after the target), falling back to an
older one or to `no target backup found`; the page now says to aim one
second past it;
- when a base backup removed from the bucket outside retention is still
pinned, and how to unpin it;
- how to read the upper bound from the archive, what pinning the base
backup changes, and a section on archives with several timelines:
PostgreSQL follows `recovery_target_timeline: latest` from the backup's
timeline through the consecutively numbered history files it can find,
so a missing `.history` confines the replay to the backup's own timeline
and the bound is the last transaction on that timeline, while a targeted
timeline that does not descend from the backup stops recovery with
`requested timeline N is not a child of this server's history`.

**Tests.** `TestCNPGRestoreBackupID` (the pinning rule),
`TestReconcileCNPGRestore_PinsTheRequestedBackup` (drives a restore
through the purge and reads back the patched `Postgres` app, including a
`cnpg.io/Backup` removed by retention, with and without `recoveryTime`),
`TestBuildPostgresAppRestorePatch_ReplacesBackupID`, a deep-copy test
for the new `stoppedAt` field, and three `helm unittest` cases for
`recoveryTarget`. With the rule disabled, 4 of the rule cases and 3 of
the reconcile cases fail; with the template change removed, 2 chart
cases fail; the retention case fails against the first version of this
PR, which pinned an id recorded on the artifact; the deep-copy test
fails once `StoppedAt` is dropped from the generated `DeepCopyInto`. `go
test ./internal/backupcontroller/...` and `helm unittest
packages/apps/postgres` (139/139, after rebasing on main) pass. Not run:
the e2e suite (`hack/e2e-chainsaw`), and a restore against a live
cluster with this driver.

### Screenshots

No UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/issues/716.
`content/en/docs/next/operations/services/backup-classes.md` (and
`v1.6/`) carries a copy of the corrected "The recoverable window" and
"Discovering the earliest / latest restorable time" sections; the window
and `stoppedAt` corrections apply to released versions too, the
`backupID` part only once this is released.
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: none expected. The diff adds `bootstrap.backupID` to the
postgres `values.schema.json`, but the provider does not manage the
`bootstrap` block (`internal/provider/postgresql_schema.go`: "bootstrap
blocks use server defaults"), and the `RestoreJob` CRD is unchanged.
Left unticked for a maintainer to confirm.
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(backups): a Postgres RestoreJob now starts from the base backup its Backup refers to (recoveryTarget.backupID) instead of whichever backup the barman-cloud plugin picks, which on an archive with several timelines could restore another branch's data; a recoveryTime before that backup's end keeps the previous time-based choice. The PITR documentation now bounds the recoverable window by the last archived transaction rather than the newest WAL segment.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Restores can target a specific base backup and optionally replay to a
requested recovery time. If no backup is specified, the newest eligible
backup is selected automatically.
* **Bug Fixes**
* Restore handling warns when the referenced base backup is unavailable
and allows the plugin to select a backup from the catalog.
* **Documentation**
* Clarified recovery-window boundaries, timestamp precision, idle
databases, timeline branches, and behavior when a requested target
cannot be reached or predates the earliest base backup.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 14:26:49 +03:00
Aleksei Sviridkin ff0752d8e9 fix(kubernetes): probe Keycloak CRDs by output in oidc-bootstrap cleanup (#3520)
## What this PR does

Fixes #3516.

- `packages/apps/kubernetes/templates/oidc-rbac-job.yaml`: the
`mode=None` cleanup path of the `oidc-bootstrap` Job guarded its
Keycloak deletes with `if kubectl api-resources
--api-group=v1.edp.epam.com >/dev/null 2>&1`. That command **exits 0 for
an absent API group** (it simply prints an empty list), so the guard
always fell through and the delete failed with `error: the server
doesn't have a resource type "keycloakclient"` — `--ignore-not-found`
covers a missing *object*, not an unknown *type*. Since the Job is a
`post-install,post-upgrade` Helm hook with `backoffLimit: 6`, this
blocked the Helm upgrade of every `kubernetes-<cluster>` release on any
cluster without the EDP Keycloak operator CRDs, i.e. at the chart
default `oidc.mode: None`.
The probe now captures the discovery output once and gates each delete
on its own resource type being present, matching the idiom already used
in `packages/core/platform/images/migrations/migrations/{34,38,39}` and
`packages/system/ouroboros/.../cleanup-hook.yaml`. `api-resources` is
kept (rather than `get crd`) on purpose: it hits discovery, which
`system:discovery` grants to every authenticated principal, so the
namespaced Job Role needs no cluster-scoped CRD read.
- `packages/apps/kubernetes/tests/oidc_test.yaml`: new unit test
asserting the rendered script probes by output and that the exit-code
form does not come back. All four of its assertions fail against the
pre-fix template.

`cmd/check-readiness` and the platform migrations were checked too: they
already test the output, not the exit code.

## Validation

- `helm unittest packages/apps/kubernetes` on the branch rebased onto
current `main`: 317/317 pass. Putting `main`'s template back fails only
the new case, all 4 assertions, so it is a real regression guard.
- `make generate` in `packages/apps/kubernetes`: no diff.
- Extracted the rendered `mode=None` script and ran `sh -n` /
`shellcheck -s sh` — clean (only pre-existing `SC3040` for `pipefail`
and one pre-existing `SC2086`).
- Executed the extracted script against a stub `kubectl` reproducing the
two real behaviors (`api-resources` exits 0 with empty output for an
absent group; `delete <unknown type>` exits 1 despite
`--ignore-not-found`): the pre-fix script exits 1 with `the server
doesn't have a resource type "keycloakclient"`, the fixed script exits 0
with the CRDs absent and still issues both deletes with the CRDs
present.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

The diff touches no `values.schema.json` and no behaviour the docs
describe: the template change restores the documented `oidc.mode: None`
path rather than altering it.

## Release note

```release-note
fix(kubernetes): the oidc-bootstrap hook no longer fails on clusters without the Keycloak operator CRDs, which blocked Helm upgrades of kubernetes-* releases at oidc.mode=None
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* OIDC cleanup now skips deleting Keycloak resources when their resource
types are unavailable, preventing cleanup failures.
* **Tests**
* Added coverage for resource discovery and conditional cleanup
behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 14:26:00 +03:00
Aleksei Sviridkin abc07a3927 fix(backups): hold a BackupJob while its application is being restored (#4544)
## What this PR does

Nothing on the backup path looks at restores. A Plan creates its
BackupJobs on schedule (`plan_controller.go`), and `BackupJobReconciler`
dispatches them to their driver whatever else is happening to the
application. A backup taken during a restore captures a half-restored
state at best, and for CNPG it does worse.

We hit it on a v1.6.3 cluster. helm-controller applied the restored
Cluster with `bootstrap.newServerName`, and four seconds later a
BackupJob of the same application re-applied the barman plugin with the
source's `serverName`. The restored cluster archived onto the source's
WAL prefix, and its recovery job died with `WAL archive check failed for
server <cluster>: Expected empty archive`. Correcting `serverName` by
hand let the same restore finish, which is what identified the writer.
`currentBarmanServerName` preserves the live value on most paths, but
not in the window where the restored Cluster is being re-rendered. An
on-demand run can be refused by whoever creates it, but a scheduled one
cannot, and there is no `suspend` on a Plan. With an hourly plan and a
CNPG restore allowed thirty minutes, the overlap happens about half the
time.

A BackupJob that has not started yet (`status.startedAt` unset) now
waits while a non-terminal RestoreJob in its namespace writes into the
same application. The target is resolved as the drivers resolve it
(`resolveCNPGRestoreTarget` and the Job-based paths): the referenced
Backup's `applicationRef`, with the target's `kind` and `name` when a
`targetApplicationRef` is set, and the target's API group only when it
names one, so a target without one stays in the Backup's group. A Velero
restore always stays in the Backup's group, since the Velero driver
drops the target's API group and restores the Backup's resources. A
RestoreJob whose Backup is gone is skipped, since the restore controller
fails it on its next pass. `options.targetNamespace` is honoured only
when the Backup was taken by the Velero strategy, the one driver that
reads it: the CNPG and Job-based drivers ignore the option and restore
in place, so their restores hold whatever it says. The held job stays
`Pending` with a `Ready=False` condition, reason `RestoreInProgress`,
naming the restore, and is requeued every 30s. It is not failed: a
scheduled run should still happen once the restore ends. A failed write
of that status is returned, so the manager retries it, and once the
restore ends the condition is removed before the job starts, since it
would otherwise read "waiting" until the driver next writes `Ready`,
which for a healthy run is when the backup completes.

The gate sits after BackupClass resolution and strategy validation, so
third-party strategies are unaffected, and before credentials
projection, so a held job projects nothing.

What happens to a delayed scheduled run: the Plan controller only
decides when to *create* a BackupJob (a slot within `startingDeadline`,
300s, of now). Once created, the job is not subject to that deadline, so
a held run keeps its `<plan>-<unix-minutes>` name and runs once the
restore ends, later than its slot. If the restore outlasts the next
slot, that slot's job is created and held too, and both run after the
restore. Nothing is skipped, and nothing is deduplicated.

Not covered, deliberately:
- A backup already running when a restore starts is not interrupted. Its
driver owns state the gate would strand. The reverse guard, a RestoreJob
waiting for a running BackupJob, would belong in the restore path,
together with a per-application claim that closes the race between the
two checks, and is tracked in #4559.
- A Velero RestoreJob in another namespace that targets this one through
`options.targetNamespace` does not hold this namespace's backups. The
lookup is namespaced.
- The Velero restore path has no deadline, so a stuck VM restore holds
that application's backups until the restore is ended or deleted. That
matches the restore already blocking the application.

Tests: `TestActiveRestoreTargeting` covers the resolution matrix (in
place, to-copy into and out of the application, terminal phases, another
kind or another API group under the same name, a to-copy target
inheriting the Backup's API group, a Velero target naming another API
group, a Velero restore sent to another namespace or naming its own, a
Postgres restore carrying `targetNamespace`, a missing Backup with or
without a target). `TestReconcile_RetriesAHoldItCouldNotRecord` and
`TestReleaseRestoreHold` cover the status writes, and the held-job test
reconciles again after the restore ends to check the condition is gone.
`TestReconcile_HoldsABackupJobWhileItsApplicationIsRestored` is the
behavioural regression: it fails with the gate removed (the job reaches
projection and dispatch). `TestReconcile_DoesNotHoldARunningBackupJob`
checks that a started job is not held.

Backport: a candidate for `release-1.6`, where the reproduction was
seen. `backupjob_controller.go` differs from `main` only in the dispatch
switch and the Go 1.27 modernizers, so the cherry-pick should be clean
or close to it.

### Screenshots

N/A, no UI change.

### Downstream repositories

The diff touches only `internal/backupcontroller/` Go code. It changes
no CRD, API type, chart, values file, hack script or naming, so the
trigger map comes out empty.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(backups): a BackupJob that has not started yet, scheduled or on demand, now waits while a RestoreJob is restoring the same application, instead of running against it. It stays Pending with a Ready=False RestoreInProgress condition and runs once the restore ends. This stops a CNPG backup that has not started yet from re-pointing an in-place restored cluster at the source's WAL archive; a backup already running when a restore starts is not held.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Unstarted backup jobs now wait when an active restore targets the same
application. They remain pending and are retried after a short delay,
then can proceed once the restore finishes.
* Backup jobs that have already started continue without being held by
an active restore. Completed or failed restores do not hold pending
backup jobs.
* Restore matching accounts for the specified target and, for Velero
restores, the target namespace. A backup job’s restore hold is cleared
when no matching active restore remains.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 14:24:41 +03:00
Aleksei Sviridkin afe23a2c3f ci(release): pin flux where the release pushes the packages artifact
prepare-release now runs flux push artifact to publish the rc's packages
artifact, and it installed whatever flux was latest, while build-amd64
and the candidate checks pin one version. It installs that version now.
The arm64 leg and the nightly never run flux, so they no longer install
it. The rc-e2e comment gave a stable tag as the reason for its
!cancelled(), but the job only runs for an rc; it now says it follows
the rule every job below build-arm64 has.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-29 12:36:56 +03:00
Aleksei Sviridkin 3f6de5eac0 ci(release): keep every installer tarball out of the release assets
The release job now builds the Talos installer for both arches, so
_out/assets holds an arm64 tarball next to the amd64 one. Both are the
image the build pushed, not something upload-assets publishes, and the
artifact only excluded the amd64 one.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-29 12:36:56 +03:00
Aleksei Sviridkin 51a2d859ac docs(release): describe what the release job builds and what blocks an rc
The amd64 release job builds the Talos installer and matchbox for both
arches, so the pages that said CI never builds both in one job, and
that build-amd64 builds amd64 only, were wrong. Every rc stitch also
skips matchbox and the e2e sandbox, and the nightly report lists
matchbox and the refs an in-tree rebuild stamps over; the page said
every such skip and line fails the next rc. Name the ones that do not.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-29 12:36:56 +03:00
Aleksei Sviridkin 71144d4fd6 ci(release): pin the matchbox rebuild in the release job
tags.yaml only runs on tags, so no pull request exercises it. The
matchbox rebuild is the only thing that makes its ref an index, and
losing the step, moving it past the tree capture or letting it switch
the default builder would surface as a failed gate on the next rc. The
comments that called the talos images amd64 only now say they are
built for both arches in the amd64 job.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-29 11:52:08 +03:00
Kirill Ilin b314514950 chore(hetzner-robotlb): update robotlb to 0.1.0 (#4557)
## What this PR does

Updates the bundled robotlb to 0.1.0. The upstream project moved from
the `intreecom` GitHub organization to `treetscom`, so the chart and the
image are now pulled from the new location.

Upgrade note from upstream: balancer targets now follow the Service's
`externalTrafficPolicy`. A Service that leaves it unset gets the
Kubernetes default `Cluster`, and its balancer targets every eligible
node instead of only the nodes running its pods. Set
`externalTrafficPolicy: Local` to keep the previous targeting.

### Screenshots

Not applicable, no UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#715
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The website still links to the old `Intreecom/robotlb` repository in the
Hetzner install guide, the platform-stack guide and the licenses page.
GitHub redirects those links, so nothing is broken;
cozystack/website#715 updates them.

### Release note

```release-note
chore(hetzner-robotlb): update robotlb to 0.1.0 and pull it from the new treetscom organization. Load balancer targets now follow the Service's externalTrafficPolicy; set it to Local to keep targeting only the nodes that run the Service's pods.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Updates**
* Updated the RobotLB installation to use the `treetscom` chart and
container image sources.
* Updated the installation notes to link to the RobotLB project’s
current documentation location.
  * Updated the RobotLB chart’s application version to 0.1.0.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 13:51:32 +05:00
Aleksei Sviridkin 6ef23da6e7 ci(release): publish matchbox as an amd64 and arm64 index
matchbox already carries the kernel and initramfs of both arches, but
the release built it for amd64 only: the arm64 leg skips the talos
package, so there is no arm64 half to stitch, and the gate let it
through on an allowlist entry. Its Dockerfile only copies files, so the
amd64 job now rebuilds it for both platforms on a docker-container
builder and the stamped ref pins the index. The allowlist entry goes.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-29 11:29:04 +03:00
Aleksei Sviridkin c740031453 Merge remote-tracking branch 'origin/main' into ci/build-arm64-leg
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-29 11:20:27 +03:00
Mattia Eleuteri 1cefadcef6 docs(backups): say when a base backup gone from the bucket is still pinned
A restore leaves the base backup to the plugin only once the
cnpg.io/Backup is gone. A base backup removed from the bucket by a
lifecycle rule or by hand keeps its cnpg.io/Backup until the plugin's
next maintenance pass on the source cluster, and indefinitely once
that cluster no longer runs under the same UID. A restore in that
window fails after the purge and on every retry, with nothing on the
RestoreJob to unpin it; deleting the cnpg.io/Backup is the way out and
does not touch the object store.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-09-29 09:39:49 +02:00
Mattia Eleuteri 28e42fbe4d fix(backups): match a Velero restore in its Backup's API group
The Velero driver resolves a restore's target from the Backup's
applicationRef, overriding only the kind and name, and restores the
Backup's resources. A targetApplicationRef.apiGroup never reaches it.
The restore gate honoured that group all the same, so a Velero
RestoreJob naming another group let a BackupJob of the application
actually being restored start against it, and held one of an
application the restore never touches.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-09-29 09:38:36 +02:00
Mattia Eleuteri 64e837727c docs(backups): say what recovery does when the latest timeline is a sibling
recovery_target_timeline latest targets the highest consecutively
numbered history file, without checking that it descends from the base
backup. On an archive holding sibling branches that can be another
branch, and recovery then stops with "requested timeline N is not a
child of this server's history" rather than following the backup's
own line of descent. The page also names the second case where a
restore does not pin its base backup: a cnpg.io/Backup removed by
retention.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-09-29 09:38:12 +02:00
Mattia Eleuteri 6999609585 docs(backups): bound Postgres PITR by the last archived transaction
The recoverable window was described as ending at the newest archived
WAL segment, and the safe fallback as "at/before the most recent
completed backup". A time-targeted recovery ends just before the first
transaction after the target, and a segment switch or a backup is not
one, so on a database at rest every instant after its last write
fails, including the end of a later backup. Aiming at a backup's
stoppedAt also misses it: stoppedAt is truncated to the second and the
plugin compares barman's end time to the microsecond.

Describe the bound the recovery actually has, how to read it from the
archive, the one-second margin on stoppedAt, what pinning the base
backup changes, and why archives with several timelines need it.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-09-29 09:38:12 +02:00
Mattia Eleuteri 497dd6fc4f fix(backups): start a Postgres restore from the backup it names
Without recoveryTarget.backupID the barman-cloud plugin chooses the
base backup itself: the newest one in the catalog when no recoveryTime
is set, and the newest one ending by recoveryTime otherwise, on any
timeline since no targetTLI is passed. A RestoreJob naming an older
Backup therefore started from another one, and on an archive holding
sibling timelines - left by in-place restores that archived back under
the source serverName - from another branch, bringing back that
branch's data.

The restore reads barman's backupId and the backup's stoppedAt from
the live cnpg.io/Backup behind the Backup, and passes the id through
the chart as bootstrap.recovery.recoveryTarget.backupID. The plugin
gives backupID precedence over time-based selection, and CNPG does not
count it among the mutually exclusive targets, so it combines with
targetTime.

Two cases keep the plugin's choice. A recoveryTime earlier than one
second past stoppedAt: the named backup cannot serve an instant before
it is consistent, and stoppedAt is truncated to the second while
barman compares its end time to the microsecond. And a cnpg.io/Backup
that is gone: plugin-barman-cloud's retention deletes it together with
the base backup it describes, and the driver deletes one only with its
artifact, so pinning its id would purge the target and then fail the
recovery on every retry. That restore records a BaseBackupGone warning
event instead.

A re-restore into the same target clears a backupID left by a previous
one.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-09-29 09:38:11 +02:00