admin/cozystack
1
0 Fork 0
mirror of https://github.com/cozystack/cozystack.git synced 2026-11-08 21:14:56 +09:00
Code Issues Packages Projects Releases Wiki Activity
7,071 Commits 569 Branches 294 Tags
e4e249bbf66624c0332748c1673fdbbafd15f47d
Commit Graph
501 Commits
Author SHA1 Message Date
Aleksei Sviridkin c4e6d169a0 fix(e2e): give the cozytest caller a dataplane budget that fits its backstop 
The collector's default budget is sized under the 300s Chainsaw
backstop. The cozytest.sh caller allows 600s, and on the default it
would give up more than half of that window, so it passes a budget
matching its own backstop. The budget test reads each caller's
override and fails when a budget leaves more of its backstop unused
than the backstop's kill grace.

The Chainsaw call-site comment still described the fan-out caps as the
bound, and the e2e testing guide described an EndpointSlice exception
to the notes contract that the code does not have; both now describe
what the collector does.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-10-02 03:29:13 +03:00
Aleksei Sviridkin 94b328c6eb test(e2e): hold every raised backup wait to a start margin 
The wait-budget guard accepted a wait one second over the install
timeout, although the wait's clock starts before the install's by the
Application reconcile, the chart fetch and the helm-controller queue. A
wait lowered from 660 to 601 passed every test and brought back the bare
assert timeout the budgets exist to avoid. A wait now has to clear its
floor by a named 60s margin. Nothing in the tree bounds those three
delays, so the value is policy: the headroom every current wait already
carries. The sizing guide in docs/agents/e2e-testing.md asked only for
"strictly above", which the guard now rejects; it points at the margin
instead of restating a number.

The mongodb, rabbitmq and clickhouse walkthrough waits were raised for
the same reason as mariadb's, and their Chainsaw ops grown with them,
but nothing held either. Each suite is now a row in one table and gets
its own floor and op tests, so a regression reds only its own suite.
Floors come from the api-server install-timeout default and, for
mariadb, the chart's startup probe; each suite's RD is checked for an
override instead of being assumed not to carry one. Each op keeps the
remainder it had before the raise over every HelmRelease wait it
reaches, the bucket one included, so raising a wait and its op together
keeps the remainder instead of widening the slack. The mongodb target
behind SKIP_RESTORE=1 is not reached and is left out until the op stops
passing it.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-10-01 17:40:20 +03:00
Aleksei Sviridkin 54159736bc feat(talos)!: bump Talos to v1.14.2 (#4598) 
## What this PR does

Bumps the Talos image Cozystack builds from v1.13.6 to v1.14.2 to get a
newer DRBD. v1.13.6 ships DRBD 9.3.2, v1.13.7 and later and
v1.14.0/v1.14.1 ship 9.3.3, and v1.14.2 is the first release with 9.3.4.
ZFS moves from 2.4.3 to 2.4.4.

On 9.3.2 I've seen a peer reboot leave the DRBD sender thread spinning
on a failed send (EPIPE). What followed was RCU stalls, processes stuck
in D-state, a hanging `drbdsetup disconnect` and two-phase-commit
timeouts. The [9.3.4
ChangeLog](https://github.com/LINBIT/drbd/blob/drbd-9.3.4/ChangeLog) has
fixes for a sender thread pinning a CPU while its connection is down,
for two-phase-commit issues and a hanging `drbdadm disconnect`, and for
two nodes ending UpToDate with different data after a reconnect. None of
them are in 9.3.3. I matched the trace to those entries by their
descriptions only, not against the commits.

The base installer image had to change. Talos v1.14 no longer publishes
`ghcr.io/siderolabs/installer`, so `gen-profiles.sh` now uses
`ghcr.io/siderolabs/installer-base`, same as upstream's own v1.14 imager
profiles. I built the amd64 installer from the new profile with
`imager:v1.14.2` locally and it builds without errors.

PR CI will not boot the new image. It builds the installer and matchbox
images, but the e2e runs on the upstream `talos:v1.13.5` container from
`hack/e2e-compose.yaml`. The first run that boots the new kernel, DRBD
and ZFS is the nightly after merge, which builds the nocloud disk from
these profiles and runs it in QEMU.

On Talos v1.14.2 the `drbd_transport_tcp` module is no longer loaded on
demand. After upgrading a node from v1.13.6, `drbd` was loaded and the
transport module was not, `drbdsetup new-peer` failed with "Failed to
create transport (drbd_transport_xxx module missing?)", and every DRBD
resource on the node stayed in Connecting. Listing the module in
`machine.kernel.modules` fixed it without a reboot. The cause is
[siderolabs/pkgs#1565](https://github.com/siderolabs/pkgs/pull/1565), in
Talos since v1.14.0: the kernel is built with an empty modprobe path, so
it no longer loads modules on `request_module()`, which is how DRBD asks
for its transport. Any module that used to load that way now has to be
listed explicitly (reported upstream as siderolabs/talos#14501). Outside
DRBD this includes `dm-thin-pool` and `dm-multipath`, which the v1.14.2
kernel builds as modules, so LVM-thin or multipath users on Talos need
them listed too. The e2e node config now loads `drbd_transport_tcp`
explicitly, and the talm preset (cozystack/talm#252) and the install
docs (cozystack/website#720) get the same line. The module ships in the
drbd extension on v1.13 too, so the extra line is safe before the
upgrade.

The order matters for host nodes. Either update the cozystack preset in
the talm project to a version with cozystack/talm#252 and re-render and
apply the node config, or add `drbd_transport_tcp` to
`machine.kernel.modules` by hand. Only then upgrade the node to v1.14.2.
A node upgraded first loses DRBD replication until the module is added.

The host network policy now also denies port 2383 to `world`. Talos
v1.14 moved etcd's `/metrics`, `/health` and gRPC-gateway JSON API from
2379 to a separate listener on 2383, and the policy only blocked 2379
and 2380. Without this, the etcd JSON API on v1.14 control-plane nodes
would be reachable from outside, protected only by client mTLS.

Two smaller changes come with the bump. The multus
`talos-cni-plugins-checked-against` marker moves to v1.14.2, because the
pkgs release behind Talos v1.14.2 still pins CNI plugins v1.9.1, same as
multus. The system memory limits doc now describes the v1.14 OOM
trigger, since v1.14.0 dropped its global memory PSI clause
([siderolabs/talos#13895](https://github.com/siderolabs/talos/pull/13895)).

Before upgrading hosts, operators should know a few things from the
[v1.14.0 release
notes](https://github.com/siderolabs/talos/releases/tag/v1.14.0) and the
Talos compatibility code:

- Talos v1.14 accepts a host upgrade from v1.12.0 or later. It supports
Kubernetes 1.32 through 1.37, while v1.13 still accepted 1.31, so a
cluster on Kubernetes 1.31 has to upgrade Kubernetes first.
- etcd's HTTP endpoints (`/metrics`, `/health`) moved from port 2379 to
2383. The Cozystack etcd scrape proxy reads a separate metrics listener
on `127.0.0.1:2381`, which this change does not move. Anything else that
scraped or health-checked etcd on 2379 has to switch to 2383, and a
firewall outside the cluster that blocked 2379 should block 2383 too.
- etcd and kube-apiserver now require TLS 1.3, so a client that can only
speak TLS 1.2 stops connecting.
- Workload isolation (`SecurityProfileConfig`) stays off on upgraded
clusters, but `talosctl gen config` turns it on for new ones. I haven't
tested Cozystack with it enabled.
- `talosctl apply-config --mode=reboot` is gone.

### Screenshots

Not a UI change.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#720
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [x] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
cozystack/talm#252
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The website PR refreshes the `next` version pins for v1.14.2 and adds
the module to the Talos install pages. The talm PR adds it to the
cozystack preset.

### Release note

```release-note
feat(talos): Bump Talos to v1.14.2, which ships DRBD 9.3.4 and ZFS 2.4.4. DRBD 9.3.4 fixes a sender thread spinning on a dead connection, two-phase-commit hangs and a data divergence after a reconnect. Before upgrading a node, add `drbd_transport_tcp` to its Talos `machine.kernel.modules` next to `drbd`: Talos v1.14.2 no longer loads it on demand, and without it DRBD cannot connect to any peer. We recommend skipping the v1.13 line and upgrading nodes straight to v1.14.2. Talos accepts a direct upgrade from v1.12.0 or later. Talos v1.14 moves etcd's HTTP metrics and health endpoints from port 2379 to 2383, and the Cozystack host network policy now blocks 2383 from outside the cluster.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Updates**
* Upgraded host node support to Talos v1.14.2 across machine images,
with refreshed firmware and storage extensions.
* Updated memory-pressure guidance for Talos v1.14, including
version-specific behavior and configuration recommendations.
  * Added protection for Talos v1.14’s etcd HTTP endpoint on port 2383.
* Updated cluster preparation to load the TCP transport module required
for DRBD.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-10-01 14:03:30 +03:00
Aleksei Sviridkin 312409ecd2 test(e2e): fail the suite whose pods outlive its cleanup (#4492) 
## What this PR does

A Chainsaw suite could pass while the workloads it created kept running
after its own cleanup, and nothing noticed until a later suite failed.
On one E2E run the brokers of four Kafka releases from suites that had
passed and deleted their applications (#4488), plus a TenantControlPlane
sitting on its finalizer with two apiserver pods (#4489), pushed pod
memory requests to the node limit. Harbor then failed on `Insufficient
memory` two suites later, under the wrong name.

This makes the suite that leaked fail, and name what it left behind.

- Chainsaw v0.2.15 has no hook that runs after a test's cleanup: its
`Configuration` carries `error.catch`, which runs only on failure. So
`make test-chainsaw` now runs `hack/e2e-chainsaw-run.sh`, which invokes
`chainsaw test` once per suite directory and checks the cluster between
invocations with `hack/e2e-chainsaw/_lib/leak-guard.sh`.
- Attribution: a pod is charged to a suite when all three hold. It is
not terminal. Its UID was not in the pod list taken before the suite.
The root of its ownerReferences chain was created at or after the suite
started. A platform pod replaced mid-suite by a rollout, an eviction or
a crash traces back to an object the install created, so it is not
charged. A pre-existing pod adopted by a controller recreated mid-suite
is excluded by the UID baseline. A leak is charged once, to its own
suite.
- The suite start is read off an apiserver, from the `creationTimestamp`
of a server-side dry-run create, rather than off the sandbox. With
several control-plane nodes the reading and a root's timestamp can come
from different apiservers, so the rule assumes their clocks agree more
closely than the gap between the reading and the suite's first object;
no skew margin is added. Source for that behaviour: the generic registry
`Store.create` stamps it through `rest.FillObjectMetaSystemFields`, then
`DryRunnableStorage.Create` copies it back without persisting
(`staging/src/k8s.io/apiserver/pkg/registry/generic/registry/store.go`
and `dryrun.go` at Kubernetes v1.33.0).
- Budget: charged pods get 300s to disappear, the default
`timeouts.cleanup` in `.chainsaw.yaml`; a unit test keeps the two equal.
A suite that overrides its own cleanup timeout (kafka sets 8m) is
covered the same way, because the check runs only after `chainsaw test`
has returned, which is after every cleanup in the suite has finished
under its own timeout; the 300s start from there. The leaks this targets
sat about 110 minutes (Kafka brokers) and 30+ minutes (the
TenantControlPlane) past their suites. Every green suite logs
`leak-guard: suite=<dir> clear after Ns`, so the number can be checked
against later runs.
- Past the budget the run prints `LEAKED-PODS suite=<dir> budget=300s
count=N`, then one line per pod with its phase, deletion state, root and
`app.kubernetes.io/instance`. It raises a `leaked-pods` error annotation
and adds a failed `leak-guard` test case to `chainsaw-report.xml`. The
run continues through the remaining suites and exits non-zero at the
end, as the single invocation did on a red test.
- A suite that already failed on its own is checked once without the
wait. Its leftovers are reported as a `::warning` with `LEAKED-PODS
suite=<dir> checked-once=chainsaw-failed`, since some may still be
tearing down.
- Each pod gets one `kubectl wait` inside the remaining budget, and
nothing is retried. A timeout or a spent budget gives the verdict from
the last good pod list, also when the re-list after it fails, so a leak
fails the suite. A wait that fails only after running for its whole
`--timeout` counts as a timeout whatever it printed: kubectl maps an
expired context to "timed out waiting for the condition" only inside the
watch, so a deadline that passes during its initial lookup surfaces as
"context deadline exceeded" (`wait.go` and `delete.go` in kubectl
v1.33.0). A spent budget is caught before calling kubectl and never
passed as `--timeout=0`, which means "check once".
- Any other error on a wait or a re-list, before the budget is spent and
before any timeout, stops the waiting and is reported as `LEAKED-PODS
suite=<dir> count=N inconclusive="<error>"`. That is a warning, not a
leak failure, so an apiserver blip cannot become a leak red.
- A failed first reading after a suite, including a pod list that cannot
be parsed, leaves the suite unchecked. It fails with a `leak-guard`
annotation and a report case saying the cluster could not be read, not
that it leaked. Errors that are not leaks carry the annotation title
`leak-guard`, not `leaked-pods`.
- Known leaks are listed in `hack/e2e-chainsaw/_lib/leak-allowlist.txt`
as `<suite-dir> <issue-ref>`: `kubernetes-latest` and
`kubernetes-previous` against #4489 (both suites create a
TenantControlPlane through `_lib/run-kubernetes.sh`). `kafka` and
`kafka-metadata` were listed against #4488 until #4490 fixed it; those
entries are gone. An allowlisted suite prints its leak as a `::warning`
and does not fail. An entry that names a directory that is not a suite
stops the run before the first suite, so a rename cannot keep an
exemption alive. The PR fixing each issue deletes its lines.

Suite selection is unchanged. The runner runs the named directories when
there are any, and otherwise every directory directly under
`hack/e2e-chainsaw/`, each handed to Chainsaw's own discovery; it
refuses a tree with a test at its top level, where that split would stop
being exact. I verified this with Chainsaw v0.2.15's own
`discovery.DiscoverTests`, built from source at that tag. For the full
tree, for `kafka redis harbor` and for `kubernetes-latest`, the union of
the per-suite discoveries equals what a single `chainsaw test [suites]`
discovers: 57, 7 and 2 tests, identical lists. 57 matches the test count
in an earlier full run's report. A unit test pins that the runner passes
chainsaw the suites and flags it was given, in order. The order changes
too: suites used to run in Chainsaw's own order and now run one
directory at a time in alphabetical order, which moves the kubernetes
suites after the kafka ones. Before #4490 fixed #4488 this starved them
(see the E2E criterion below). With the fix merged the order has no
effect, and a future leak that lands before a memory-heavy suite is
named at the suite that leaked. One behaviour does change: a selected
name that is not a directory used to fail the single invocation before
any test ran; now only that suite fails and the others run.

`chainsaw-report.xml` keeps its path and shape: one `<testsuites>` root
with the summed counts, one `<testsuite>` per suite directory. Its
consumers are the `Collect chainsaw report` / `Upload chainsaw report`
steps in `pull-requests.yaml`, `e2e-fork.yaml`, `nightly.yaml` and
`e2e-tag.yaml`, which `docker cp` the file and upload it as an artifact.
Nothing in the repository parses it. The merge is unit-tested on two
per-suite reports in Chainsaw's real shape, one of them failed, and the
result is checked to parse as XML.

Tests: `hack/chainsaw-leak-guard_test.bats` (31 tests, in `make
unit-tests`; named without the `e2e-` prefix, which the unit lane
excludes) covers the attribution rule, the budget, the allowlist, the
merge and the runner, all against stub `kubectl` and `chainsaw`. Every
rule was mutation-checked: removing the guard call, having the Makefile
bypass the runner, inverting the allowlist match, flipping or tightening
the root-timestamp comparison, removing the UID baseline, removing the
terminal-phase filter, skipping the budget wait, changing the budget,
lowering or raising the depth cap, accepting a stale allowlist entry,
reading an unreadable pod list as empty, dropping the leak case from the
report, skipping directories in the selection, treating any failed owner
read as a missing owner, dropping the separate report for a suite that
already failed, dropping the not-run reason from the report, reading any
wait error as a timeout, removing the inconclusive branch, letting a
failed re-list after a timeout turn a leak inconclusive, ignoring a
failed re-list, reporting an unreadable cluster as a leak, reading an
unparsable pod list as clean, reporting a kubectl warning instead of the
error, passing the report name without its extension, wording a failed
re-list as a failed wait, dropping an allowlist's last line when it has
no newline, dropping the check that `--set-string` carries a value, and
reading a wait that ran its whole timeout but printed a context error as
inconclusive. Each one turns its named test red, and restoring turns it
green.

E2E criterion, registered before the run: in the `Run E2E tests` log of
this PR, every suite that ran prints `leak-guard: suite=<dir> clear
after`, a `LEAKED-PODS suite=<dir>` line carrying `allowlisted=`, or one
carrying `checked-once=` (a suite that failed on its own) or
`inconclusive=` (waiting stopped on a failed read), and no line matches
`LEAKED-PODS suite=.* budget=`. With #4490 merged, `kafka` and
`kafka-metadata` are no longer allowlisted and must print `leak-guard:
suite=kafka clear after` and `leak-guard: suite=kafka-metadata clear
after`, which is the live check that #4490 fixed #4488. The first run of
this PR (before #4490, run 36223896793) is the positive control: it
printed `LEAKED-PODS suite=kafka count=15` and `suite=kafka-metadata
count=8`, allowlisted, and every other suite cleared in 0-37s. That run
also went red on both kubernetes suites, because those 23 leaked brokers
left no memory for the second tenant worker VM (`0/3 nodes are
available: 3 Insufficient memory`), which the order change described
above made possible. The guard has not yet run against a real cluster,
so the first run of this PR should carry the `full-e2e` label; its
`clear after` lines are the first measured teardown times, and any
unexpected `LEAKED-PODS` line is either a real leak or a false positive
to understand before merge.

Follow-ups, known and out of scope here:

- The allowlist exempts a whole suite, so a new leak in an allowlisted
suite stays hidden until the issue's fix removes the line.
- Attribution is per suite directory, not per Chainsaw `Test`; the leak
lines carry the release instance, which usually names the test.
- Only pods are checked. A leaked PVC, LoadBalancer IP or bucket holds
no node memory and is not seen.
- The 300s budget rests on Chainsaw's cleanup timeout, not on a measured
pod-gone latency; the `clear after` lines are where that measurement
will come from.
- A platform object recreated mid-suite (for example a HelmRelease
uninstall/reinstall remediation, or a force-replace on an immutable
field) has a root newer than the suite start, and its pods would be
charged to whichever suite was running as a leak; the first CI runs are
where to look for this.
- The per-pod wait recomputes the remaining budget before each pod, and
no unit test pins that.
- `kubernetes-previous` is allowlisted for #4489 although the stuck
finalizer was seen only in `kubernetes-latest`; both suites create a
TenantControlPlane the same way, and the exemption hides any new leak
there until #4489 is fixed.
- Each clean suite now waits for its own pods to finish terminating
before the next starts, where the tail used to overlap the next suite.
The first full run's `clear after` lines say what that costs against the
E2E job's 215-minute cap.
- After a timeout followed by a failed re-list, the leak lines come from
the list taken before the wait, so a pod whose own wait had already
succeeded is still named and the count can be too high; the verdict is
unaffected.
- Likewise, when every wait succeeded and the re-list then fails, the
inconclusive report still lists the pods whose deletion was just
confirmed. In the boundary case where the budget runs out during the
last successful wait and the re-list then fails, the result is
inconclusive rather than a verdict from the last good list, because the
guard only treats the budget as spent when it finds it exhausted at a
check.
- A mistyped `COZY_LEAK_ALLOWLIST` path passes allowlist validation; it
fails closed, since no suite is then exempt.
- An allowlisted suite is not waited on, so `kubernetes-latest` and
`kubernetes-previous` will print `LEAKED-PODS ... allowlisted=` on most
runs even for pods still in normal teardown; waiting out the budget
first and applying the allowlist only to what remains would stop that
noise.
- A slow teardown in each of several suites can add up to 300s apiece,
against the E2E job's 215-minute cap. On the first full run of this PR
(run 36223896793) the `clear after` values of the suites that were
waited on summed to 210s; the last full run before this change took 158
of the 215 minutes.
- A pod whose owner's resource type is no longer served gets "the server
doesn't have a resource type" rather than NotFound, so its suite is
reported unchecked instead of leaked; it still fails.
- `kubectl wait --for=delete` waits for deletion, while the guard skips
pods in `Succeeded` or `Failed`; a pod that finishes but is never
deleted (a Job under a leaked root) spends the rest of the budget before
the suite shows `clear`. The verdict is right; only job time is lost.
- The unit tests do not pin the one-line empty report Chainsaw writes
for a directory with no tests (`_lib`), nor the branch that catches a
spent budget before calling `kubectl wait`; both are verified against
the Chainsaw and kubectl sources rather than by a test.
- The runner's per-suite report directory from `mktemp -d` is not
removed; the sandbox is discarded after the job.
- Each non-suite directory under `hack/e2e-chainsaw/` (today `_lib`)
costs one empty Chainsaw invocation, because the runner leaves the
decision of what is a test to Chainsaw.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff adds files under `hack/`, changes the `test-chainsaw` target,
and rewords one comment and the failure line in the four E2E workflows.
The trigger map's `hack/` entry is for cozystack/ccp, whose skills
anchor on `hack/package.mk`, `hack/common-envs.mk` and `make generate`,
none of which change here. A GitHub code search for `chainsaw` across
the cozystack organisation finds hits only in this repository.

### Release note

```release-note
test(e2e): the E2E app suite now fails the suite whose pods outlive its own cleanup by more than 300s, naming each leftover pod and its owner; suites with a filed leak are listed in hack/e2e-chainsaw/_lib/leak-allowlist.txt and only warn.
```
 2026-10-01 14:03:13 +03:00
Aleksei Sviridkin cdc4b88a90 fix(ci): close e2e suite selection gaps in select-e2e and select-install (#4634) 
## What this PR does

The e2e test selector picked the wrong Chainsaw suites for some diffs.
In one case it lost an escalation, in another it ran the full suite
where a subset was enough, and in two more it missed a suite or ran
nothing. The fixes are in `hack/select-e2e.sh`, with the `.yml` part
also in `hack/select-install.sh`.

- A changed file in a directory under `hack/e2e-chainsaw/` that holds no
suite escalated only when it was the whole diff. Next to any path that
selected a suite, the escalation was dropped with nothing on stderr. The
directory name is now checked against the suite list right where the
path is classified, like the `hack/e2e-apps/` rule already did. The
empty-selection backstop can't be reached anymore. I kept it as a guard,
and the doc says its line has no test.
- A seaweedfs change ran the full suite, because
`cozystack.seaweedfs-application` mapped to a `seaweedfs` suite that
does not exist. Now it selects the suites that use tenant-root's object
storage: bucket, the backup round-trips that write through the
in-cluster `seaweedfs-s3` endpoint, and harbor, which keeps its registry
in a BucketClaim served by the seaweedfs COSI driver. The issue says
this mapping would break the round-trip test. It doesn't, since that
test only walks suite to source to suite, and `vm-disk-application`
already maps to `vminstance` the same way. The test takes the expected
set from the suites' own yaml.
- A suite written as `chainsaw-test.yml` was invisible to both
selectors. Chainsaw v0.2.15 loads that name when there is no
`chainsaw-test.yaml` (`pkg/discovery/load.go`), so such a suite runs but
never gets selected. Both scripts list it now. `select-install.sh
--validate` also passed when `find` failed, and now it fails.
- An edit under `examples/backups/_lib/` was read as an app named `_lib`
and selected nothing. It escalates now, same as
`hack/e2e-chainsaw/_lib/`.
- The selector comments quoted bats file counts and escalation-branch
counts. All of them were stale, so I removed them. The doc half of #4120
was already fixed on main.

Each fix has a test that fails on main. I also mutated each fix: its own
test goes red and no other test does.

Three related gaps outside these lines are tracked in #4630: the
`packages/core/testing/Makefile` comment that names only
`chainsaw-test.yaml`, suites built from numbered step files, and a file
placed directly in `examples/backups/`.

Two things worth knowing before merge:

- Any file except the `.disabled` one in a parked suite's directory,
like `hack/e2e-chainsaw/kubernetes-proxmox/tenant.yaml`, now always runs
the full suite. That is the safe direction. If it gets expensive, those
directories could be made inert.
- The `examples/backups/_lib/` rule and its doc line point at a
directory another PR adds. If this PR merges first, the rule just has
nothing to match yet.

Closes #3816, closes #3710, closes #4283, closes #4120, closes #4205,
closes #4626.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(ci): e2e test selection no longer drops an escalation for a non-suite path under hack/e2e-chainsaw/, selects the suites that use seaweedfs instead of the full run, sees suites written as chainsaw-test.yml, and escalates edits to examples/backups/_lib/.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* End-to-end test selection now recognizes suites configured with either
supported YAML filename and reports discovery failures instead of
silently proceeding.
* Changes to shared backup walkthrough helpers and unrecognized
test-suite directories now trigger broader test coverage.
* Storage-related changes in SeaweedFS now select all relevant test
suites.

* **Documentation**
* Updated testing guidance to describe suite discovery, file-change
matching, and when the full test suite is selected.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-10-01 13:16:55 +03:00
scooby87 031ded71ee feat(seaweedfs): alert before an instance stops taking writes (#4573) 
## What this PR does

A SeaweedFS instance that runs out of room stops taking writes without
anything paging. A volume server below its `minFreeSpacePercent` reserve
turns every volume it holds read-only; before that, it can run out of
slots, and a bucket whose volumes are all full then cannot get a new
one, even with half the disk free. Backups archiving into the instance
fail in turn, and a Postgres cluster doing continuous WAL archiving
keeps the segments on its own volume until that fills too.

The generic `KubePersistentVolumeFillingUp` alert does not catch this:
it pages below 3% free, while a volume server stalls at its 5% reserve
or, when slots run out, with the disk still half empty; and its early
warning needs the disk to keep filling, which stops once writes fail.

This PR adds per-instance rules to the SeaweedFS system chart, on
signals the servers already publish and the platform already scrapes,
rendered under the same `monitoring.enabled` gate as the chart's
ServiceMonitors:

- `SeaweedFSVolumeServerDiskLow` (warning): a volume server has less
than 10% of its disk left above the reserve.
- `SeaweedFSVolumeServerFull` (warning): a volume server cannot host
another volume, because it is at the reserve or has no slot left.
- `SeaweedFSWritesFailing` (critical): the master has failed to place a
write in every 5-minute window for 15 minutes. Writes may still land in
between; the usual cause is a bucket with no writable volume left that
the master cannot grow. A new bucket's first writes fail a few picks
while its volumes grow, which the 15-minute hold does not reach unless
new buckets keep arriving.

Every expression is pinned to the release namespace, because the root
vmalert evaluates each instance's rules against every namespace's
metrics.

It also adds `docs/operations/seaweedfs-capacity-for-backups.md`: how to
size an instance for continuous WAL archiving, what the rules mean, and
how to grow an instance.

The expressions were validated live rather than by unit tests, since the
repository has no rule-evaluation tooling: on a test cluster, a
standalone SeaweedFS 4.45 on a 1 GiB volume was scraped by the
platform's vmagent and the rules rendered from this chart were evaluated
by the root vmalert. All three loaded healthy and each reached `firing`:
`SeaweedFSVolumeServerDiskLow` with 5.7% of the disk left above the
reserve, `SeaweedFSVolumeServerFull` both at the reserve and on slot
exhaustion, which stopped one bucket at 50% disk use, and
`SeaweedFSWritesFailing` under continuous failing writes, resolving once
they stopped. A new bucket's first successful writes on a free disk did
fail a few picks, which is why that rule holds for 15 minutes. On the
healthy platform instance all three stay quiet. helm-unittest pins what
a render can break: the namespace selector of each rule, the shared
label set of the two arms of the full-server rule, the gate, the
critical rule's hold, and that alert templating reaches vmalert
unrendered; each assertion was checked to fail when its line is mutated.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff touches a system chart template, its test and a document under
`docs/operations/`; no app values, platform values,
ApplicationDefinition, tooling or `hack/` path from the trigger map.

### Release note

```release-note
feat(seaweedfs): alert when a SeaweedFS volume server nears its free-space reserve, cannot host another volume, or keeps failing to place writes, and document how to size an instance for continuous WAL archiving.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added namespace-scoped SeaweedFS monitoring alerts for low or
exhausted disk capacity, unavailable volume slots, and write-placement
failures. Capacity alerts use a 10% disk threshold, and alerts trigger
after 15 minutes; write failures are marked critical.
- **Documentation**
- Added guidance on estimating SeaweedFS storage needs for Postgres
backups, interpreting capacity alerts, and expanding storage when
supported by the StorageClass.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-10-01 10:29:35 +03:00
Aleksei Sviridkin 08011cef09 fix(ci): escalate an edit to the shared backup walkthrough helpers 
examples/backups/<app>/ selects that app's suite, and a directory whose
name is no suite selects nothing. A helpers directory shared by every
walkthrough, examples/backups/_lib/, falls into the second case: an
edit to it alone produces an empty selection, which both lanes read as
"skip Chainsaw", so a change reaching every backup round-trip would run
none.

It now escalates to the full suite like hack/e2e-chainsaw/_lib/. The
set of suites that run a walkthrough is spread across the suites' own
scripts, and a derived subset that misses one would fail open.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-10-01 01:06:25 +03:00
Aleksei Sviridkin a6ec30d56f fix(ci): discover Chainsaw suites written as chainsaw-test.yml 
Chainsaw loads chainsaw-test.yml when a directory has no
chainsaw-test.yaml, but select-e2e and select-install enumerated the
.yaml name only. A suite spelled that way ran whenever the whole
directory was handed to chainsaw, yet no escalation listed it and no
edit to it could select it, so it silently dropped out of every
narrowed run.

select-install also sent find's errors to /dev/null inside a pipeline
that reported sort's status, so a failed listing validated no suite at
all and passed. It is now captured and fails validation.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-10-01 01:06:24 +03:00
Aleksei Sviridkin e070bc7d1d fix(ci): escalate a per-suite Chainsaw path that names no suite 
A path under hack/e2e-chainsaw/<dir>/ put <dir> into the selection
without checking that it is a suite, leaving the final intersection to
drop it. Alone the path escalated through the empty-selection backstop,
but beside any path that contributed a suite the escalation vanished
and the run narrowed to that suite with nothing on stderr. Shared
material next to _lib/ or a suite nested deeper than discovery looks
then went untested in exactly the pull requests that also touch an app.

The directory name is now membership-tested where the path is
classified, as the hack/e2e-apps/ arm already does, so the decision is
a property of the path and names it.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-10-01 01:06:13 +03:00
Aleksei Sviridkin 9ac67d76a2 test(tests): preserve Bats failure reports during fixture cleanup (#3848) 
## What this PR does

Test-level `EXIT` cleanup overrides the Bats TAP handler, so a failed
assertion can disappear from test results. This moves cleanup in eight
unit files after their assertions. Failed tests keep scratch files for
inspection.

Prerequisite for [the Bats runner
change](https://github.com/cozystack/cozystack/pull/3849). Existing
subshell traps in the OpenAPI and CAPK tests remain supported.

Closes #3623

### Validation

`make unit-tests test-controllers -j4 --output-sync=target` passed. The
converted files passed under Bats 1.14.0 and `dash`. A deliberately
failed assertion produces `not ok` after this change, and the trap audit
rejects an added test-level handler.

### Downstream repositories

Test cleanup and its documentation only. No downstream consumer changes.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```
 2026-10-01 00:45:09 +03:00
Andrey Kolkov f6a615f693 fix(backups): scrub legacy plaintext passwords from CNPG Backup snapshots (#4558) 
## What this PR does

A CNPG `Backup` taken while the Postgres app still accepted
`users[].password` copied that password into
`status.underlyingResources`, where anyone with `get backups` in the
namespace can read it. #4078 stopped new snapshots from carrying the
field but left every existing `Backup` as it was (#4179).

This adds `LegacyPasswordScrubber`, a leader-elected runnable in the
backup strategy controller that, on start, lists `Backup` objects and
removes `users.*.password` from every snapshot whose `kind`/`apiVersion`
is the CNPG snapshot's.

- **Why on every leader start and not a platform migration.** The
migration hook runs pre-upgrade, while the old controller is still
leader, so a `Backup` the old replica writes during the rollout would
keep its password and the migration never re-runs. With `--leader-elect`
on, the old replicas stop writing before the new leader's scrub runs,
and a `Backup` brought back from an object-level export is cleaned on
the next start too. On a clean cluster the whole pass is one List from
the manager cache, which `BackupReconciler` already populates.
- **Write shape.** The patch is a merge patch computed from the diff, so
the request body only nulls the `password` keys and cannot overwrite
status another writer changed since the List. The snapshot is handled as
generic JSON (numbers kept as `json.Number`), so fields the current
snapshot type does not know survive.
- **Failure handling.** A `Backup` that cannot be decoded or patched is
logged and skipped, and a failed List is logged too; `Start` always
returns nil, because a returned error would stop the manager and take
every backup down over one object.
- **Restore compatibility.** Restore already ignored the key; a test
decodes a scrubbed snapshot through `unmarshalCNPGBackupSnapshot` to pin
that.
- RBAC: `list` and `patch` on `backups.cozystack.io/backups` are already
granted to the controller, so the chart is unchanged.
- `docs/operations/backup-classes.md` no longer tells operators to
delete pre-upgrade `Backup` objects, and says what the scrub does not
reach (audit logs, etcd backups, external exports of the objects, Helm
history of the init-script Secret), so a password once set through
`users[].password` is still treated as exposed until rotated.

The runnable carries a `TODO(#4179)` to remove it once upgrading from a
release that accepted `users[].password` is no longer supported.

Verification: `go test ./internal/backupcontroller/...` and `go build
./cmd/backupstrategy-controller/` are green. The new tests cover
scrubbing with everything else kept (including an unknown field with a
large integer), leaving non-CNPG, other-apiVersion, clean and
snapshot-less Backups unpatched, and continuing past a Backup that fails
to decode or patch. Disabling the delete, or returning on the first
patch error, makes the tests fail. Not exercised on a live cluster yet.

Fixes #4179

### Screenshots

N/A — no UI changes.

### Downstream repositories

Walked the trigger map against the diff: it touches only
`internal/backupcontroller`, `cmd/backupstrategy-controller` and the
in-repo `docs/operations/backup-classes.md`, with no package, values,
schema, CRD, variant or asset change. The website carries no "delete
pre-upgrade Backups" workaround this would make obsolete; its stale
`users[name].password` rows are the #4078 follow-up already noted there.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(backups): the backup strategy controller now removes plaintext `users[].password` values left in `status.underlyingResources` of CNPG `Backup` objects taken before managed Postgres stopped accepting user passwords. It runs on every controller start and needs no action. It does not reach copies outside the cluster (audit logs, etcd backups, exported `Backup` objects), so a password once set through `users[].password` should still be treated as exposed until it is rotated.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Security**
* At controller startup, plaintext user passwords are removed from
matching CNPG backup snapshots. Other snapshot data is preserved.
* This cleanup does not affect external copies or remove older
init-script manifests from Helm history. Treat passwords previously set
through `users[].password` as exposed until rotated.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-09-30 11:58:06 +04:00
Aleksei Sviridkin 4d3c8dd7bc docs(talos): describe the Talos v1.14 OOM trigger 
Talos v1.14.0 dropped the global memory PSI clause from the default
OOM trigger, so the page no longer matches the version Cozystack
ships. The OOMConfig example now serves host nodes still on v1.13.x,
which can shed the clause without upgrading.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-30 04:23:38 +03:00
Marian Koreniuk 7500a9fbae test(kubernetes): add the Proxmox e2e suite, parked disabled 
Upstream CI has no hypervisor, so this suite cannot run there; it ships
`.disabled` and is run by a lane that owns a Proxmox stand.

Named for what the chart actually creates. The resource definition prefixes the
release, so a Kubernetes CR called k8s-e2e produces
ProxmoxCluster/kubernetes-k8s-e2e in the tenant's own namespace, and the admin
kubeconfig Secret is kubernetes-k8s-e2e-admin-kubeconfig.

Stand-specific values are variables rather than hardcoded. Every one of them is
a bare ${COZY_PVE_*} that run-proxmox.sh expands with envsubst, supplying the
defaults itself — GNU envsubst leaves a ${VAR:-default} in the text exactly as
it is, set or not, and that reaches the apiserver as a string where the schema
wants an integer — and refusing a fixture with a variable still in it. The
StorageClass and the PVC probe read the same COZY_PROXMOX_STORAGE, so the volume
lands where the probe looks. The runner reads the tenant kubeconfig's .conf key,
the address a runner outside the management cluster can reach.
The stand's resource pool comes in through COZY_PVE_POOL, empty by default.
hack/proxmox-e2e-fixture_test.bats pins the shape of the fixture and the render.

The PVC probe reads the storage again after the claim is deleted: a deleted
claim whose disk is still there is the leak the pre-delete hook's Step 1b
exists to prevent. keep_tenant is honoured by the suite's own teardown, not
only by the workflow's. The workflow carries no fallback for the stand's
variables: an unset one fails the job before a key is sent anywhere. The
schedule runs only where vars.PROXMOX_NIGHTLY is true: the job targets a
self-hosted runner with the `proxmox` label, and a scheduled run on a repository
without one would sit in the queue until GitHub cancels it.

The suite also asserts the half of the deployment that lives in the management
cluster — the CSI controller and the cloud-controller-manager — because that is
the property the pair is designed around and a tenant-only assertion would pass
with the token in the wrong place.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
 2026-09-29 21:04:06 +02:00
Alexey Artamonov 35e3593c85 fix(seaweedfs): describe the writes-failing alert by what it measures 
The rule fires when the master fails a volume pick in every 5-minute
window for 15 minutes. That proves recurring refusals, not that no
upload gets through: writes into other buckets can succeed in between,
and a steady stream of new buckets, each failing a few picks while its
first volumes grow, produces the same pattern. The master publishes no
count of successful picks to tell the two apart, so the rule stays and
its description and the operations page now say what it has seen.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
 2026-09-29 18:17:01 +03:00
Aleksei Sviridkin 9b56e70ca6 feat(mongodb): support the system-bucket backup flow (useSystemBucket) (#3967) 
## What this PR does

MongoDB was the only `cozy-default` app with **no `useSystemBucket`
flow**: its backup storage was always static on the
`PerconaServerMongoDB` CR (bucket/endpoint/credentials from `backup.*`
values plus a chart-emitted `<release>-s3-creds` Secret), so a tenant
could not back up to the platform system bucket out of the box the way
Postgres/ClickHouse/MariaDB/etcd do. This closes that gap so the default
MongoDB backup-restore works OOTB.

PBM takes `bucket`/`endpointUrl`/`region`/`prefix` as **static CR
fields** (not secret refs), and the platform bucket name/endpoint are
only known at BackupJob time — so, unlike ClickHouse's runtime
`secretKeyRef` env, the chart cannot render them. The implementation
mirrors the CNPG driver's pattern:

- **Chart** (`packages/apps/mongodb`): a new `backup.useSystemBucket`
flag keeps `spec.backup.enabled` (so the pbm agents run) but omits the
static `storages`/`tasks`/`pitr` and the `<release>-s3-creds` Secret.
Helm never owns `spec.backup.storages`, so a Flux re-render can't fight
the driver. A `bootstrap.enabled + useSystemBucket` render guard mirrors
Postgres, and `useSystemBucket: true` with `backup.enabled: false` is
refused both at render time and by an `x-kubernetes-validations` rule on
`backup` in the schema (the same mechanism as the `storageClass`
immutability rule). Note on the schema rule: the aggregated apiserver
does not evaluate CEL on application-backed resources yet (#2657), so
today it is a UI/schema-level marker and the render-time refusal is what
stops a Flux release; it also trips the API review gate (“validation
rule added”), so this PR needs an API-owner approval on top of the code
review.
- **Strategy** (`cozy-default-mongodb`): now carries the S3 coordinates
(`endpointUrl` already scheme'd by the
`backupstrategy-controller.endpoint` helper, `credentialsSecret:
cozy-backups-creds`, `prefix: <namespace>/<application>`).
- **Driver** (`psmdbstrategy_controller.go`): SSA-injects
`spec.backup.storages[s3-storage]` onto the live `PerconaServerMongoDB`
at BackupJob time under a dedicated field manager
(`cozystack-psmdb-backup-driver`) with `ForceOwnership`. Injection is
**gated on the app's `backup.useSystemBucket` flag** (a legacy app
leaves it false and is never touched; a useSystemBucket app is
re-injected once per mint (once per BackupJob, plus once per
storage-race re-mint), before its operator CR is created, so a later
coordinate change catches up on the next job while a streaming dump is
never patched under). The backup CR is minted only once the operator
reports the storage pushed into pbm (`status.observedGeneration` caught
up and `PBMReady` True): pbm writes to what its own config says, not to
the spec, so a CR minted before that push would land at the previous
coordinates while its Backup recorded the new ones; the job waits at
`PerconaServerMongoDBPBMConfigPending` meanwhile, bounded by the backup
deadline. Once a job has its CR, none of the mint-time reads or holds
can fail it: the strategy and app reads the artifact still needs are
bounded by the 24h running ceiling instead, and the cluster is not read
at all **and on the release having rendered that flag**: the app CR is
projected from the HelmRelease's desired values, so the flag reads true
before helm-controller renders the revision, while the tenant's own
storage, scheduled tasks and PITR stream from the previous render still
sit on the cluster. As long as the cluster carries `spec.backup.tasks`
or `spec.backup.pitr`, the driver does not inject and holds the
BackupJob at `Ready=False PerconaServerMongoDBLegacyRender` (failing it
at the deadline), so a wedged legacy revision never has its nightly dump
and oplog redirected into the shared bucket. The backup snapshot falls
back to the strategy coordinates when the operator status echo is empty;
that fallback is gated on the CR's own `percona.com/delete-backup`
finalizer (the flow the CR was minted on), not on the app flag re-read
at completion, so a legacy backup never records the platform bucket and
a flag flipped mid-dump does not change the artifact. The restore path
rebuilds `backupSource` from it (the `cozy-backups-creds` reference is
projected into the restore namespace before dispatch).
- **Retention/cleanup**: on the `useSystemBucket` flow the driver owns
the shared-bucket archive (the chart's psmdb `tasks[].keep` retention is
gated out, and an on-demand backup is not covered by task retention). It
stamps the operator's `percona.com/delete-backup` finalizer on each
backup it mints **against a storage that carries the platform
coordinates** — the strategy's bucket and `credentialsSecret`, read back
from the live cluster after the injection — never on the flag alone: a
strategy without an `s3` block, or a storage that points anywhere else,
holds the job (`MongoDBStrategyHasNoS3` /
`PerconaServerMongoDBStorageForeign`) instead of minting a finalized
dump against a bucket the platform does not own. On Cozystack `Backup`
deletion (by hand today; by platform retention once it lands, since a
`Plan` only creates BackupJobs and never deletes a `Backup`) the driver
deletes the operator `PerconaServerMongoDBBackup` CR so that finalizer
frees the object — the finalizer's presence is the ownership marker
cleanup reads back. That reading is unambiguous because cleanup resolves
the CR by the name recorded in `driverMetadata` (the one the driver
minted), not by scanning: the psmdb operator does stamp
`percona.com/delete-backup` on its own `cron-*` task-minted backups, but
those are never in scope. A legacy backup keeps the pre-existing
hands-off contract, decided from the snapshot before any read of the
operator CR. A terminating namespace or the
`backups.cozystack.io/skip-artifact-cleanup: "true"` annotation strips
the driver's finalizer and releases; a `Get` of the operator CR that
fails with anything other than NotFound/NoMatchError still requeues, and
the operator's own `release-lock` finalizer can hold the CR while the
operator is down, so those two are what can keep a namespace in
`Terminating`. `schedule` and `retentionPolicy` are inert on this flow
(there is no engine-side retention and a `Plan` has no retention field
yet); platform-level retention is tracked in
[#4192](https://github.com/cozystack/cozystack/pull/4192) and teardown
GC of orphaned backups in
[#3966](https://github.com/cozystack/cozystack/issues/3966), so until
those land bound `cozy-backups` with an S3 lifecycle policy.
- **Bounds on the waits**: the injection race with the operator's
cluster cache (an `unable to get storage` error on a freshly injected
storage) is retried by deleting the errored CR and minting a fresh one,
with the retry count kept on the BackupJob
(`backups.cozystack.io/mongodb-storage-race-retries`) because the
deleted CR is reaped before the next poll; after three retries the
operator's error is terminal. A system-bucket dump that has started
streaming is not cancelled on the 30-minute deadline, but fails after 24
hours in `running` (`BackupAbandoned` warning naming the operator CR,
which is left in place as the handle to whatever pbm still writes). The
deadline cancel of a not-yet-started CR pins its delete to the
resourceVersion the live read answered for.
- **RBAC**: `patch` on `perconaservermongodbs` for the injection
(server-side apply), and `create`/`delete`/`patch` on
`perconaservermongodbbackups` — `delete` to prune the shared-bucket
archive on `Backup` deletion, `patch` to strip the driver's finalizer
when releasing a wedged cleanup.

Coverage: Go unit tests over `internal/backupcontroller` — the injection
gate keyed off the app flag (legacy cluster never patched) and off the
applied render (a cluster still carrying tasks or pitr is held,
untouched), the SSA-injection precondition with the whole entry
(per-application prefix included) asserted on the cluster, strategy
templating per application, the finalizer refused against a storage
without platform coordinates (strategy without `s3` over a legacy entry,
foreign coordinates on the merged entry), the retry budget driven with
the operator simulated reaping the deleted CR between polls, the restore
snapshot fallback and credential swap across all four legacy/system flow
combinations, the finalizer-owned cleanup with its escape hatches (skip
annotation before any read, `NoMatchError` treated as gone,
terminating-namespace release using the pre-read CR, a failed strip
surfaced as an Event, a legacy CR never deleted, a legacy snapshot
released without a read, a system-bucket snapshot reaching the prune),
the deadline cancelling a not-yet-running operator backup only on a
positive live read (including no live client at all, and a CR that moved
since the read), and the running ceiling. Plus helm-unittest on both
charts (chart skips the creds Secret and omits
`spec.backup.storages`/`tasks`/`pitr` on `useSystemBucket` and keeps
them on the legacy flow; the strategy renders the S3 coordinates and the
`insecureSkipTLSVerify` knob). A live `useSystemBucket` backup+restore
e2e against the platform bucket is a follow-up (needs the system bucket
wired in CI, same status as the Postgres/ClickHouse useSystemBucket
e2e). One item to confirm in e2e: the psmdb operator accepting
`spec.backup.enabled=true` with an empty `storages` between install and
the first BackupJob.

### Screenshots

Not applicable (no UI changes).

### Downstream repositories

The tenant-facing schema surface is one new field,
`spec.backup.useSystemBucket`, on the MongoDB `values.schema.json` / CRD
(regenerated), plus a validation rule on `spec.backup`.
`terraform-provider-cozystack` hand-models MongoDB and may want to
expose it, but it is a new optional boolean that defaults false (no
drift for existing users), and the chart `README.md` regenerates on
release. Leaving every box unticked per the trigger-map guidance so a
maintainer decides on the provider follow-up.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
feat(mongodb): MongoDB backups now support the platform system bucket via backup.useSystemBucket, so the default cozy-default BackupClass works without per-app S3 credentials — the backup driver injects the system-bucket storage onto the cluster at backup time, once the release has rendered the flag. On this flow nothing prunes the shared bucket automatically yet (a Plan carries no retention field; tracked in #4192 and #3966), so an S3 lifecycle policy on cozy-backups is a prerequisite for enabling it, not a suggestion. useSystemBucket requires backup.enabled=true; the chart refuses the combination with backups disabled at render time, and the schema carries the same rule as a marker until the apiserver enforces CEL (#2657). A system-bucket dump still running after 24 hours fails its BackupJob with the operator backup left in place. A MongoDB restore now adopts the target cluster's S3 credentials only for a storage on the same bucket as the archive (it used to adopt them unconditionally); a legacy restore into a target on a different bucket that relied on one key pair covering both must keep or re-create the source release's -s3-creds Secret by name, and is held at Ready=False RestoreCredentialsMissing until it exists.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
  * Added opt-in system bucket mode for MongoDB backups.
* Automatically injects platform bucket storage without
application-managed S3 credentials.
* Supports endpoint, region, path prefix, addressing, and TLS settings.
* Manages shared-bucket backup artifacts when system bucket backups are
deleted.

* **Bug Fixes**
* Improved restore credential selection to avoid cross-bucket
mismatches.
  * Preserved backup metadata when storage details are unavailable.

* **Documentation**
* Documented configuration, scheduling, retention, PITR limitations, and
restore guidance.

* **Validation**
* Prevented incompatible bootstrap and system bucket restore
configurations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-09-29 18:10:56 +03:00
Alexey Artamonov 1ead6ed90d docs(seaweedfs): size an instance for continuous WAL archiving 
The default instance holds a little under 10 GiB, while a Postgres
cluster archiving WAL into it keeps every segment for the whole
retention window, 30 days by default. Nothing in the tree says how to
work out what a bucket needs or what happens when it does not fit, so
the first sign is usually a database that has stopped.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
 2026-09-29 15:53:44 +03:00
Aleksei Sviridkin 8b861668b8 ci(release): build pre-release images for amd64 and arm64 and gate releases on it (#4505) 
## What this PR does

Pre-release builds now publish every image as an amd64 and arm64 index,
and a release fails if anything it ships is single-arch. Main, release
lines and pull requests keep building amd64 only. It builds on #4498,
which made every image buildable for both architectures. Part of #1961.

**This PR is on hold until everything under "Merge after" has landed.**
Merged earlier, the new gate fails the next rc on the images those PRs
fix.

### How a pre-release is built

A pre-release tag gets a second job, for arm64, next to the existing
release job. That covers `-rc.N`, `-beta.N` and `-alpha.N` tags. It runs
on the CNCF arm64 pool (`oracle-vm-24cpu-96gb-arm64`), builds the same
images with `PLATFORM=linux/arm64`, and pushes them as `<tag>-arm64`.
`prepare-release` waits for it, and if the arm64 job fails the rc fails
with it. There is no amd64-only fallback. Stable tags do not run it:
promotion copies the rc digests with `skopeo copy --multi-arch all`, so
a stable release inherits the indexes.

After the amd64 build, the stitch joins each amd64 image with its arm64
twin into one index. It lives in `hack/stitch-multiarch.sh`. Every tag
of the amd64 image moves to the index, the component-versioned ones
included. The script then rewrites the digests in the tree and
republishes the packages artifact and the installer chart pinned on
them. It reads refs through `hack/lib/image-refs.sh`, and it fails if an
old digest survives anywhere outside `charts/`.

### The gate

A new check fails the rc when a pinned digest in the tree is not an
index with both linux/amd64 and linux/arm64. It lives in
`hack/verify-multiarch.sh`. It checks first-party and third-party refs
alike. Images that are amd64 by nature go into
`hack/multiarch-allowlist`, one repository per line, each with a
mandatory reason. Today that is only the e2e sandbox. An entry that
matches nothing is reported as stale.

The static gate only sees digest-pinned refs in the tree. A tag-only
ref, an image from a vendored chart default, or one an operator starts
at runtime is invisible to it. So the rc e2e also audits every image its
nodes pulled, with the same check and the same allowlist. Neither covers
a package the e2e never installs.

### Nightly

A new nightly workflow builds main for arm64. It keeps the arm64 build
cache warm, since the rc job only reads it. It also prints the
single-arch refs that would block the next rc. Nothing it builds is
published.

### Smaller changes

- `CACHE_TAG` in `hack/common-envs.mk` moves the default cache ref, so
the arm64 build keeps a cache of its own.
- `MATRIX_ARCH=arm64` makes `hack/build-matrix.sh` leave out the
amd64-only e2e sandbox.
- The kamaji provider image is pushed under the build's `IMAGE_TAG` like
every other image. Its Makefile set `IMAGE_TAG` itself, so concurrent
builds overwrote each other's component tag. Closes #4503.
- `hack/nightly-mirror.sh` verifies a copy against the raw manifest
digest. Plain `skopeo inspect` resolves an index to one platform's
child.
- matchbox is rebuilt for both platforms in the amd64 release job. The
arm64 leg skips the talos package, so there is no arm64 half to stitch,
and its Dockerfile only copies files, so no QEMU is needed. Releases
then network-boot arm64 machines too. Closes #4524.

The stitch also rewrites the kamaji ref inside `files/components.gz`,
recompressed with `gzip -n` so the bytes are reproducible, so the kamaji
control-plane provider ships multi-arch too. keda and kuberture now name
the repository next to their pinned digest, so the gate can resolve
them; their rendered manifests do not change.

### Merge after

- #4552 builds the Talos installer, matchbox, Harbor and the Velero
KubeVirt plugin for arm64, and adds flux-plunger, keycloak-operator,
kilo and migration-controller to the root `build:` list. It replaces
#4507, #4515, #4525 and #4528.
- #4549 moves ingress-nginx, cozy-proxy, cozystack-scheduler and
keycloak-kms-proxy to their multi-arch releases, which are already
published.

The rest of the stack is merged: #4485, #4498, #4512, #4517, #4521 and
#4522 here, and the multi-arch PRs in
ingress-nginx-with-protobuf-exporter, cozy-proxy, cozystack-scheduler
and keycloak-kms-proxy.

### Verification

The bats suites pass: stitch, verify-multiarch, build-matrix,
common-envs, nightly-mirror and the release contracts. The first three
also pass under `hack/cozytest.sh` with dash. Each new test was red
before its implementation, and the verify-multiarch and stitch tests
each have a mutation that turns them red. I ran `verify-multiarch
--report` against main. It lists the first-party refs the stitch will
fix, plus the third-party and special cases above. The workflows have
not run yet. The arm64 job's tools and duration, the stitch against real
registries, and the audit are checked for the first time on the nightly
and on the next rc.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Nothing under `hack/` is moved or renamed, and no make target changes
its default behaviour: `CACHE_TAG` and `MATRIX_ARCH` are opt-in. The
satellite repositories listed under "Merge after" are prerequisites, not
follow-ups this change forces on them.

### Release note

```release-note
ci(release): pre-release builds publish every image as an amd64 and arm64 multi-arch index, and a release fails if any image it ships or pulls in e2e lacks either architecture. Stable releases inherit the indexes through promotion.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Release candidates now build amd64 and arm64 images in parallel and
combine eligible images into verified multi-architecture indexes before
publication.
* A nightly arm64 image build is available for testing and supplies a
build cache for release-candidate builds.
* **Bug Fixes**
* End-to-end checks audit whether pulled images support both
architectures. Audit failures block prerelease checks, while stable
releases continue with a warning.
* End-to-end test artifacts now include collected image references and
multi-architecture audit results.
* **Documentation**
* Updated release and image guidance covers multi-architecture builds,
verification checks, and troubleshooting, including arm64 build and
stitching failures.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-09-29 15:35:13 +03:00
Aleksei Sviridkin 51a2d859ac docs(release): describe what the release job builds and what blocks an rc 
The amd64 release job builds the Talos installer and matchbox for both
arches, so the pages that said CI never builds both in one job, and
that build-amd64 builds amd64 only, were wrong. Every rc stitch also
skips matchbox and the e2e sandbox, and the nightly report lists
matchbox and the refs an in-tree rebuild stamps over; the page said
every such skip and line fails the next rc. Name the ones that do not.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-29 12:36:56 +03:00
Aleksei Sviridkin 71144d4fd6 ci(release): pin the matchbox rebuild in the release job 
tags.yaml only runs on tags, so no pull request exercises it. The
matchbox rebuild is the only thing that makes its ref an index, and
losing the step, moving it past the tree capture or letting it switch
the default builder would surface as a failed gate on the next rc. The
comments that called the talos images amd64 only now say they are
built for both arches in the amd64 job.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-29 11:52:08 +03:00
Aleksei Sviridkin c740031453 Merge remote-tracking branch 'origin/main' into ci/build-arm64-leg 
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-29 11:20:27 +03:00
Mattia Eleuteri 1cefadcef6 docs(backups): say when a base backup gone from the bucket is still pinned 
A restore leaves the base backup to the plugin only once the
cnpg.io/Backup is gone. A base backup removed from the bucket by a
lifecycle rule or by hand keeps its cnpg.io/Backup until the plugin's
next maintenance pass on the source cluster, and indefinitely once
that cluster no longer runs under the same UID. A restore in that
window fails after the purge and on every retry, with nothing on the
RestoreJob to unpin it; deleting the cnpg.io/Backup is the way out and
does not touch the object store.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
 2026-09-29 09:39:49 +02:00
Mattia Eleuteri 64e837727c docs(backups): say what recovery does when the latest timeline is a sibling 
recovery_target_timeline latest targets the highest consecutively
numbered history file, without checking that it descends from the base
backup. On an archive holding sibling branches that can be another
branch, and recovery then stops with "requested timeline N is not a
child of this server's history" rather than following the backup's
own line of descent. The page also names the second case where a
restore does not pin its base backup: a cnpg.io/Backup removed by
retention.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
 2026-09-29 09:38:12 +02:00
Mattia Eleuteri 6999609585 docs(backups): bound Postgres PITR by the last archived transaction 
The recoverable window was described as ending at the newest archived
WAL segment, and the safe fallback as "at/before the most recent
completed backup". A time-targeted recovery ends just before the first
transaction after the target, and a segment switch or a backup is not
one, so on a database at rest every instant after its last write
fails, including the end of a later backup. Aiming at a backup's
stoppedAt also misses it: stoppedAt is truncated to the second and the
plugin compares barman's end time to the microsecond.

Describe the bound the recovery actually has, how to read it from the
archive, the one-second margin on stoppedAt, what pinning the base
backup changes, and why archives with several timelines need it.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
 2026-09-29 09:38:12 +02:00
Myasnikov Daniil 10727c0391 test(tests): preserve Bats failure reports during fixture cleanup 
Test-level EXIT handlers replace the handler Bats uses for TAP output,
so a failed assertion can disappear from the reported test results.
Cleanup must preserve those failures before the unit runner changes.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-09-29 12:38:02 +05:00
Andrey Kolkov 19764c4bba fix(backups): scrub legacy plaintext passwords from CNPG Backup snapshots 
A CNPG Backup taken while the Postgres app still accepted users[].password
copied that password into status.underlyingResources, where anyone with
`get backups` in the namespace can read it. Dropping the field stopped new
snapshots from carrying it but left every existing Backup exposed.

The backup strategy controller now removes users.*.password from CNPG
snapshots when its leader starts. It runs on every start rather than once,
so a Backup written by a pre-upgrade replica during the rollout, or brought
back from an object-level export, is still cleaned; on a clean cluster it
is a single cached List. The patch is a diff-computed merge patch that
only nulls the password keys, and a Backup that cannot be decoded or
patched is logged and skipped instead of stopping the manager.

Fixes #4179

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-09-29 11:21:08 +04:00
Andrey Kolkov 6555ae16b8 fix(mongodb): start a backup only once the operator has pushed the storage into pbm 
The wait after a storage injection watched the cluster's spec, which is
not what pbm writes to. At the pinned operator version the backup
reconciler hands pbm only a storage name and records status.s3 from the
spec entry it read, while the storage pbm uses is whatever the cluster
reconciler last pushed into pbm's own config; that push flips PBMReady
False and back to True and the reconcile stamps observedGeneration when
it ends. A backup started between the apply and that push dumped to the
previous coordinates while its Backup recorded the new ones, so both its
restore and its prune pointed at the wrong bucket. The wait was also no
wait at all: the status write that named it re-queued the job at once,
and the live read always showed the apply. The mint now waits for
observedGeneration to reach the cluster's generation and for PBMReady
to be True, and fails at the backup deadline if the operator never gets
there.

The strategy, app and cluster reads still ran ahead of the operator CR
lookup and were bounded by the backup deadline, so a strategy absent for
one reconcile 40 minutes into a streaming dump failed the job and left
the finalized CR unreachable. The CR is resolved first now; with one in
hand the cluster is not read at all, and the reads the artifact needs
are bounded by the running ceiling, past which the job fails naming the
CR left in place.

The settle check treated a field the entry lacked as agreement, so a
strategy that newly set forcePathStyle minted in the same pass as the
apply that changed addressing. A missing field now counts as its zero
value, which keeps an entry the operator rewrote without an empty region
settled while catching a value the apply adds; the storage type is
compared outright.

The runbook still said the finalizer keeps a Plan from accumulating
objects, the RBAC comment still said the timeout cancel runs on either
flow, and four wait messages read a duration as a delay. All corrected;
the runbook also describes the pbm wait.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-09-29 11:02:57 +04:00
Andrey Kolkov 841933dcec docs(mongodb): stop saying a Plan prunes the shared bucket 
A Plan creates BackupJobs and never deletes a Backup, and PlanSpec has
no retention field, so nothing on the platform bounds a MongoDB archive
on the shared bucket today except an S3 lifecycle policy or a manual
Backup delete. The runbook's tuning section said a Plan does, two
hundred lines after the same page said a Plan carries no retention, and
three driver comments named "Plan retention" as the path that reclaims
the bucket. An operator reading the first sentence would skip the
lifecycle policy the release note calls a prerequisite. All of them now
say what deletes a Backup today, and that the finalizer frees the object
whenever one is deleted, whatever does the deleting.

The settle check after an apply compared only the fields the desired
entry carries, so an entry written with forcePathStyle while the strategy
still set it counted as settled once the strategy dropped the field, and
the apply that removed it minted in the same pass. The check now treats
a field the apply is about to remove as a change too.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-09-29 10:14:44 +04:00
Andrey Kolkov 707e38d17d fix(mongodb): keep every mint-time hold off a job that already has its CR 
The injection and the ownership switch were gated on the job having no
operator CR yet; the precondition between them was not, and neither was
the pair of holds ahead of them. Each of those fails the job at the
30-minute deadline, so a tenant disabling backups, or the platform
dropping the strategy's s3 block, while a system-bucket dump streamed
failed the BackupJob at 30 minutes with the CR left finalized and no
Backup object to reach it, bypassing the 24-hour ceiling that exists so
a shared-bucket archive is never stranded. Everything up to the mint now
runs only while there is no CR; a job with one goes straight to the
state switch.

The settle wait after the apply was keyed on the storage key being
present, so a change to the strategy coordinates rewrote an entry the
operator had cached and minted in the same pass, the case the wait
exists for. It is now keyed on the entry's content: only an apply that
changes what the entry says costs a poll.

The mint-time reads (the render's tasks and pitr, the storage entry)
came from the manager's cache, which can lag a re-render back to the
legacy flow by one reconcile and show the flag true with the tasks not
yet visible. With a dynamic client they now come from the apiserver,
the way the deadline cancel already reads the operator CR.

Tests age the existing-CR case past the deadline and vary the three
holds, pin the coordinate-change wait, the live read and the term that
keeps the storage-race retry from swallowing an unrelated operator
error. The runbook's tuning section no longer summarises the defaults
as 30-day retention: only the CNPG and Velero strategies carry one.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-09-29 10:14:44 +04:00
Andrey Kolkov 46a8bd7363 fix(mongodb): apply the storage once per job, and pin the terms that decide ownership 
The system-bucket storage was applied on every reconcile of a job, not
once per job: with the operator CR already minted nothing returned before
the apply, so a streaming dump saw an SSA patch every poll for up to a
day, and the only thing keeping that patch off a cluster that had
re-acquired the chart's tasks and pitr since the mint was a predicate
term no test held. The storage a CR names is fixed for its dump, so the
apply now runs only before the CR exists; the legacy-render hold covers
that window, and a test pins that an existing CR is never injected over.

The other predicate that decides whether a Backup owns an archive
compared the live entry's bucket and credentialsSecret against the
strategy, and only the bucket half had a fixture. The credential is what
tells a tenant's entry from the platform's when a bucket name collides,
so a case varying only that term joins the foreign-bucket one. The
cleanup shortcut's snapshot-kind term and the 24h running ceiling are
pinned the same way, the latter as an absolute boundary pair.

The generated field description said the schema rejects useSystemBucket
without backup.enabled. The aggregated apiserver does not evaluate CEL on
application schemas yet, so the render-time refusal is the one that
holds; the description and the runbook now say so. The runbook's restore
paragraph still described the unconditional credential swap this branch
narrowed, exempted MariaDB from a retention warning its shipped strategy
does not earn, and pointed at a Plan retention field that does not
exist; the example README and a dispatcher test's comment still carried
the pre-branch claim that the driver never injects and never deletes.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-09-29 10:14:44 +04:00
Andrey Kolkov a6bb0556fc fix(mongodb): own archives by where the storage points, and bound the open loops 
The prune finalizer on a driver-minted PerconaServerMongoDBBackup means
the platform owns the archive: cleanup deletes the operator CR on that
marker and the operator removes the object from storage. It was stamped
from the app's useSystemBucket flag, which says nothing about where the
storage the CR names actually points. With a strategy carrying no s3
block nothing was injected, the presence-only precondition passed on a
legacy tenant-bucket entry, and the CR went out finalized against the
tenant's own bucket; deleting the Cozystack Backup then deleted the
tenant's archive. The finalizer now follows the storage's coordinates as
read back from the live cluster after the injection, and a strategy
without s3 coordinates, or a storage pointing anywhere else, holds the
job instead of minting.

The injection keyed on the same desired value. The app CR is projected
from the HelmRelease's spec.values, so the flag reads true before
helm-controller renders the revision, while the tenant's own storage,
scheduled tasks and PITR stream stay on the cluster. Injecting over that
entry redirected the tenant's nightly dump and oplog into the shared
bucket under platform credentials. A cluster still carrying tasks or
pitr is now left alone and the job held until the release renders the
flag, and the schema carries a rule marking useSystemBucket without
backup.enabled invalid, the pair the chart already refuses at render.

The storage-race retry counted surviving operator CRs against its
budget, but an errored CR is reaped the moment it is deleted (the
operator drops delete-backup without pruning on a non-ready CR), so the
count never left one and the job minted a CR per poll for the whole
deadline. The count now lives on the BackupJob and survives the reap.

A system-bucket dump in running had no bound at all: a wedged pbm agent
left the job Running forever and a nightly Plan stacked another beside
it. It now fails after 24 hours with the operator CR left in place as
the handle to whatever pbm still writes. The deadline cancel pins its
delete to the resourceVersion the live read answered for, and the
artifact-retry failure names the window that was actually in force.

Tests pin the guards that were exercised only in the harmless direction:
a system-bucket snapshot reaching the prune, the resolved prefix reaching
the cluster, a missing live client deferring the cancel, and the
finalizer refused against foreign coordinates.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-09-29 10:14:35 +04:00
Aleksei Sviridkin cb035f8eac docs(agents): stop calling the image-ref known gaps unfixed 
Two of the gaps listed there are handled in the tree now, and each entry
already says whether it is. The heading sentence still claimed all of
them were open.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-28 22:17:07 +03:00
Aleksei Sviridkin 79854b16fb Merge remote-tracking branch 'origin/fix/build-list-orphan-images' into feat/arm64-image-builds 
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-28 19:49:54 +03:00
Aleksei Sviridkin 2f08832392 Merge remote-tracking branch 'origin/feat/talos-arm64' into feat/arm64-image-builds 
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-28 19:49:46 +03:00
Aleksei Sviridkin 6d759cc73e feat(build): build package images for amd64 and arm64 (#4498) 
## What this PR does

Package images can now be built for amd64 and arm64 in one `make image`,
and a published build is multi-arch unless `PLATFORM` says otherwise.
This is the build half of arm64 support from #1961 and #903. CI still
publishes amd64 only until it gets a native arm64 leg, which comes in a
separate PR.

Builder stages now run on the build platform and compile for the target
one, so Go never runs under emulation. Under emulation it is slow and it
crashes: the migration-controller Dockerfile already records a "bad
pointer in Go heap" from exactly that. On top of this, `PLATFORM`
defaults to `linux/amd64,linux/arm64`. A build on an arm64 workstation
can't publish an arm64-only digest any more. That happened once already,
the cilium pin was arm64-only until 3f36a1b45 rebuilt it by hand.

Two images were only right because they were emulated. On a native
builder they would ship the wrong architecture. token-proxy ran `go
build` with no GOARCH, and kube-ovn calls upstream `make build-go`,
which hardcodes `GOARCH=amd64`. Both are fixed now. The new bats test
wants every compiling stage on `$BUILDPLATFORM` and actually using
`TARGETARCH`. A bare `ARG TARGETARCH` does not count, this is how
token-proxy got through.

Some things stay amd64. talos and testing pin it with `override`,
because matchbox serves amd64 Talos assets and the sandbox runs
`qemu-system-x86_64`. `LOAD=1` builds for the host only, since the
classic docker image store can't load an index. Every CI build job pins
`PLATFORM: linux/amd64`, so CI output does not change.

This changes local builds. A `make image` on the default docker driver
with the classic image store now fails with "Multi-platform build is not
supported for the docker driver", where before it built for the host.
Use a `docker-container` builder (`BUILDER=<name>`), or pass
`PLATFORM=linux/amd64` (or `LOAD=1`) to build one architecture.

I checked it on an arm64 host with a docker-container builder.
cozystack-api, dashboard, kubeovn, platform-migrations, cilium and
capi-providers-cpprovider build and push as amd64+arm64 indexes.
Binaries pulled from each platform are x86-64 and aarch64:
cozystack-api, token-proxy, the kamaji provider manager, cilium-agent
and all four kube-ovn binaries. piraeus-server builds for both
platforms, but the push to my test registry failed on registry 500s. The
capi-providers-cpprovider inline cache works with a multi-platform push.
Not covered is running the arm64 images on an arm64 cluster, CI has
none.

This is stacked on #4485 and targets its branch, so the diff here is
only the two commits on top.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#711
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(build): package images build for linux/amd64 and linux/arm64 by default, with Go builders cross-compiling on the build platform; a local `make image` now needs a docker-container builder, or PLATFORM=linux/amd64 / LOAD=1 for a single architecture. CI still publishes amd64.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Build Improvements**
* Container image builds now target both amd64 and arm64 by default when
building without loading images locally.
* CI and release builds are pinned to amd64; Talos and testing images
remain amd64-only.
* Cross-platform builds now compile on the builder’s native platform
while targeting the requested architecture.
* **Documentation**
* Clarified platform defaults and architecture requirements for local
and CI builds.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-09-28 19:43:47 +03:00
cozystack-ci[bot] 59faddc98a docs: add changelog for v1.6.4 
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
 2026-09-28 07:39:44 +00:00
Aleksei Sviridkin 5924b3fd34 docs: describe how a pre-release is built and gated for arm64 
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-28 01:32:37 +03:00
Aleksei Sviridkin 144d080342 feat(build): gate a release on every pinned image being multi-arch 
A release pins one ref per image for clusters of either architecture.
A ref that resolves to a single-platform manifest, or to an index
missing amd64 or arm64, installs on one and fails to pull on the
other, and nothing before an arm64 install would notice.

hack/verify-multiarch.sh resolves every ref hack/lib/image-refs.sh
enumerates, third-party ones included since an arm64 node pulls them
too, and fails naming each ref that is not an index carrying both
linux/amd64 and linux/arm64. A ref that cannot be resolved fails as
well, rather than passing unchecked. A rate limit or a 5xx is the one
answer retried, a bounded number of times, and a ref still refused
fails as rate-limited rather than as single-arch. --report prints the
same list without failing, for a tree that is not being released.

Images that cannot be built for both architectures go in
hack/multiarch-allowlist by repository, each with a reason, so a bump
needs no edit; they are still listed, and an entry that matches no
single-arch ref is named as stale.

--refs-file checks a list of refs instead of a tree, a tag-only ref
resolved by its tag, with the same allowlist and output. It is for
images the tree does not pin by digest, such as the ones an e2e run's
nodes actually pulled.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-28 01:32:36 +03:00
Aleksei Sviridkin 7753394bfe build: rebuild every package that has an image target 
flux-plunger, keycloak-operator, kilo and migration-controller have
image targets but were missing from the root build list, which the
build matrix, the main build and release preparation all iterate. No
CI path rebuilt them, so their pins stayed on images pushed by hand
while Dockerfile, patch and base-image changes never shipped, and the
multi-arch pre-release build would find single-arch pins it cannot
replace. migration-controller shipped a bare :latest with no digest.

Add the four packages, and a test that derives the set from the
package Makefiles so a new image target cannot be left out again.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-28 00:57:24 +03:00
Aleksei Sviridkin f02963a2c7 fix(keycloak-operator): keep the image host inside repository 
The image target stamped only `tag`, leaving the committed
`registry: ghcr.io` in place, so a build against the CI registry would
render the public host with a digest that exists only in the build
registry. Stamping the build host into `registry` instead would split
the source registry across two keys, where the nightly mirror's
substring host rewrite cannot reach it and the published tree would
keep pointing at the private registry.

Set `registry: ""` and keep the whole host in `repository`, stamped
by the image target. The upstream chart joins the two keys, so the
rendered image string is unchanged, and the ref now has the same
shape as every other stamped reference the mirror and promotion
already handle.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-28 00:57:24 +03:00
Aleksei Sviridkin 769d3b3897 fix(build): push the kamaji provider image under the build's IMAGE_TAG 
The capi-providers-cpprovider Makefile assigned IMAGE_TAG itself, which
beats the value CI puts in the environment. Every PR, main and release
line build therefore pushed the same component-version tag and
overwrote each other's, and its second tag was built from an undefined
TAG variable and ended in a bare "-".

The image now goes through image-tags like every other package: the
build-unique IMAGE_TAG always, the component version only under
PUBLISH_VERSIONED=1. The stamped refs keep the component version as
their tag, so the promote, retag and mirror tooling see the same refs
as before.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-27 07:57:31 +03:00
Aleksei Sviridkin 9a7094427e feat(build): stitch amd64 and arm64 builds into multi-arch indexes 
The arm64 images are built on native arm64 runners, apart from the
amd64 build, so each build pushes a single-platform image. The stitch
joins every first-party image with its -arm64 twin into one index,
moves every tag this build pushed for it to that index, and repins
the tree on the index digest, so one ref serves both architectures.

Most refs in the tree name only the build's IMAGE_TAG, while under
PUBLISH_VERSIONED=1 the build also pushes the component version, so
the refs alone cannot say which tags to move. The image-tags macro
therefore appends every tag it pushes to PUSHED_TAGS_LOG when that is
set, and the stitch moves exactly those, plus IMAGE_TAG and the tag a
ref carries. It reads no other tag: another build's tag that sits on
the same amd64 bytes is not this build's to move, and scanning every
tag of every repository would cost thousands of registry reads per
release.

It reads refs through hack/lib/image-refs.sh like every other
consumer, rewrites only the files that enumeration names, and then
searches the whole tree outside charts/, gzip payloads included, for
any old digest it re-pointed. A ref whose digest is also pinned where
the rewrite cannot reach is skipped instead of half rewritten.

The kamaji control-plane provider's chart ships its ref only inside
files/components.gz, built from a plain copy with gzip -n, which no
text search sees. The stitch rewrites both copies, recompressing with
gzip -n so the result is what the package's image target would have
written, rather than leaving that image amd64 only.

An image with no arm64 twin (talos, the e2e sandbox), or a pair that
is not one amd64 and one arm64 image, is skipped and named. A
registry read answered with a rate limit or a 5xx is retried a
bounded number of times; any other registry error fails the run at
once rather than reading as a missing arm64 image.

On a rerun after a stitch that failed partway, the build tag can
already sit on an index of the pinned image and an arm64 image, not
necessarily this run's if the arm64 leg rebuilt it. The stitch repins
on that index instead of skipping the image, and copies it to any of
the build's other tags still left on the amd64 image. A tag moved to
anything else fails the run, since the tree then no longer matches
the registry.

MATRIX_ARCH=arm64 makes hack/build-matrix.sh leave out the e2e
sandbox, which pins linux/amd64: an arm64 leg building it would push
a second amd64 image under its -arm64 tag.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-27 07:57:30 +03:00
Aleksei Sviridkin 6907c22302 docs(agents): describe the talos package as multi-arch 
The build notes said the talos package pins linux/amd64, which no
longer holds and would steer an agent into re-adding the pin.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-26 17:44:19 +03:00
Aleksei Sviridkin b6a30bd504 docs(e2e): describe the per-suite runner and the pod-leak guard 
The agent guide, the suite README and a workflow comment said the E2E
step is a single chainsaw invocation; it is now one per suite, and a
suite whose pods outlive its cleanup fails. Say how a pod is charged
to a suite, what the budget is, and how the allowlist is kept honest.
The workflows' failure line now points at LEAKED-PODS as well, since a
run that failed only on a leak has no assertion diff to point at.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-26 15:19:49 +03:00
Aleksei Sviridkin 084ac47237 feat(build): build published images for amd64 and arm64 by default 
PLATFORM defaulted to empty, so buildx built for the host alone. On an
arm64 workstation `make image` published an arm64-only digest, which
amd64 nodes then fail to pull; the cilium pin shipped exactly that
until it was rebuilt by hand. A build now targets linux/amd64 and
linux/arm64 unless PLATFORM is set, and the digest a package stamps is
the index digest.

LOAD=1 keeps building for the host only. The classic docker image store
cannot load a multi-platform index, and loading is the documented local
workflow.

This changes local pushes: a `make image` on the default docker driver
without the containerd image store now fails with "Multi-platform build
is not supported for the docker driver", where it used to build for
the host. Such a build needs a docker-container builder (BUILDER), or
LOAD=1 or an explicit PLATFORM for a single architecture.

talos and testing pin linux/amd64 with `override`, placed before the
include because common-envs.mk expands BUILDX_ARGS immediately. The
Talos assets are amd64: matchbox bakes the kernel and initramfs into
its image, and the e2e sandbox boots the disk under qemu-system-x86_64.
An arm64 variant of either would still serve or boot amd64 files.

CI pins linux/amd64 on every image build. Nothing there sets up QEMU,
fork builds export a single-platform archive, and tags.yaml builds on
the default docker driver, so the default would break or emulate those
jobs. Native arm64 builds belong in a leg of their own.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-26 15:14:12 +03:00
Aleksei Sviridkin 368d00b5d4 test(e2e): remove unused monitoring suites and audit BATS runners (#4019) 
## What this PR does

I removed the two monitoring OIDC BATS suites that had no runner after
the Chainsaw migration, and brought the runner audit and wrapper OIDC
test from [#3778](https://github.com/cozystack/cozystack/pull/3778).

Also added checks for audience mapper and `OIDC_MODE=System`. The named
`KeycloakRealmGroup` test documents coverage the document-count and kind
assertions already provided.

Runner audit reports unused BATS suites, missing files in runner paths,
and unresolved Make variables (including `$(E2E_SUITE)`).

Live Monitoring API admission, Grafana operator reconciliation and mode
transitions are not tested here.

### Validation

- `make -j4 unit-tests SHELL=/usr/bin/dash` passed.
- Monitoring Helm tests: 53 system-chart tests and 10 wrapper-chart
tests passed.
- Runner audit: 12 cases passed under dash and Bats 1.14.0. The full
shell syntax and BATS sweep passed under dash too.
- Three chart mutations and seven runner mutations failed as expected.

### Downstream repositories

- [x] No downstream repository is affected by this change

### Release note

```release-note
NONE
```
 2026-09-25 16:55:05 +04:00
scooby87 03e2da5805 feat(postgres,mariadb)!: drop plaintext user passwords (#4078) 
## What this PR does

Removes the plaintext `users[].password` field from the `apps/postgres`
and `apps/mariadb` charts (and the mariadb `root` account), so
managed-database passwords can no longer be pinned in tenant values or
committed to git. Passwords are always chart-generated into the
`<release>-credentials` Secret and preserved across reconciles via
`lookup`, matching the model `redis`/`valkey` already use. This closes
the "User passwords" item of the portal upstream epic
(aenix-org/cozyportal#1060).

The generated password never lands in the init-script Secret or the SQL
it runs: the postgres init-job reads each user's password from the
mounted `<release>-credentials` Secret at run time and applies it
through a psql variable (`ALTER ROLE … WITH PASSWORD :'pw'`), so it does
not linger in the init-script manifest a tenant can read. It does live
in cleartext `stringData` in the `<release>-credentials` Secret itself,
which is a chart manifest and therefore kept in Helm release history for
`MaxHistory` revisions like any chart-generated secret — that residue is
exactly why rotation cannot be built on the chart (see below). A
`password` or a `passwordRotation` key left over in a tenant's values is
accepted-but-inert and draws an admission warning, so an operator
editing it is told it has no effect rather than getting a silent 200.

Backup/restore into a copy keeps the target's chart-managed credentials
working: the CNPG driver clears `bootstrap.enabled` once recovery is
healthy so the init-job re-applies the generated passwords onto the
recovered roles, and the MariaDB driver excludes `mysql.global_priv`
from the dump (`ignoreGlobalPriv`) for backups captured from this
release onward, so restoring one of those into a copy does not clobber
the target's chart-managed users or `root`. Excluding the grant table
has a flip side, and it applies to every MariaDB restore, not only a
to-copy one: only accounts declared through `User` CRs (the chart's
`users` map) are captured, so an account created out of band directly in
MySQL is in no backup and a restore brings back its data and `mysql.db`
grants but not the account itself. This is also a backup-side fix only:
a restore from a backup taken **before** this change still replays the
source's `mysql.global_priv` and overwrites the target's users and
`root`, and `root` then needs an out-of-band reset (the operator applies
`rootPasswordSecretKeyRef` only at datadir bootstrap, so there is no
in-band path). See `docs/operations/backup-classes.md` for the
convergence window, the out-of-band-account limitation, and this
old-backup caveat.

Password **rotation** is deliberately **not** in this PR. A
chart-rendered credentials Secret keeps its cleartext in Helm release
history for `MaxHistory` revisions, so a rotation built on it cannot
revoke a leaked password; rotation is being reworked as a controller
that owns the Secret — design proposal in cozystack/community#72.

Migration is seamless: existing releases keep their current passwords on
upgrade (the value already lives in `<release>-credentials`, which
`lookup` re-reads and preserves); a `password` still present in a
tenant's values is ignored, so no existing HelmRelease fails validation.

Verification: `helm unittest` passes for both charts (postgres 87,
mariadb 126, including render-time username guards that reject any name
k8s would later reject as an invalid Secret key or User resource name);
`go build ./...`, `go vet`, the backup-controller tests and the
apps-apiserver tests are green; `make generate` was run with the
CI-pinned `cozyvalues-gen` v1.6.0 so `values.schema.json`, the READMEs,
the API types, and the `cozyrds` ApplicationDefinitions are all in sync
(the branch was rebased onto `main`, which reconciled cleanly with the
read-replica autoscaling feature that landed there).

### Screenshots

N/A — no UI change.

### Downstream repositories

- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: affected — `content/en/docs/next/cozystack-api/go-types.md`
is hand-written and still shows `Password:` in the Go examples, and the
managed-app reference pages regenerate from these chart READMEs, so both
document `users[name].password` until refreshed (not yet filed).
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: affected — the provider hand-maintains the postgres/mariadb
schemas and per-field defaults; this PR removes `users[].password`. A
follow-up issue/PR is needed there (not yet filed).

`terraform-provider-cozystack` and `website` are reached by this diff;
both boxes are left unticked because the follow-ups are not yet filed.
Same-repo future work this change suggests: the controller-owned
password rotation (design in cozystack/community#72), and extending the
"passwords only in the Secret" model to the other engines that still
accept plaintext (`clickhouse`, `mongodb`, `nats`, `opensearch`,
`rabbitmq`, `vpn`).

### Release note

```release-note
feat(postgres,mariadb)!: passwords for managed PostgreSQL and MariaDB users can no longer be set via `users[].password` — they are always auto-generated and stored only in the `<release>-credentials` Secret, never rendered into the init-script or the SQL it runs. Existing releases keep their current passwords on upgrade; a `password` (or a `passwordRotation`) left in values is ignored with an admission warning. Password rotation is deferred to a follow-up controller design (cozystack/community#72). MariaDB backups now exclude the `mysql.global_priv` grant table (`ignoreGlobalPriv`), so a restore recreates only accounts declared through the chart's `users` map — an account created out of band directly in MySQL is not captured and must be recreated after a restore.
```
 2026-09-25 15:49:53 +03:00
myasnikovdaniil 0cb37dba9e docs(release): add changelog for v1.6.3 (#4077) 
This PR adds the changelog for release `v1.6.3`.

✅ Changelog has been automatically generated in
`docs/changelogs/v1.6.3.md`.
 2026-09-25 15:41:55 +05:00
Aleksei Sviridkin 564f540f46 fix(kubernetes): render the OIDC UID header only from v1.33 (#4407) 
## What this PR does

Stacked on #4398; review the diff against `fix/kamaji-pin-hash-reset`.

The tenant Kubernetes chart now renders
`--requestheader-uid-headers=X-Remote-Uid` only from an effective
Kubernetes version of v1.33, where `RemoteRequestHeaderUID` is beta and
on by default. It no longer renders a `--feature-gates` term of its own
on any version. On v1.32 it used to add a term enabling that gate, which
is alpha there, whenever it could read the operator's entries and they
left the gate unset; no release tag contains that path, so dropping it
now changes nothing a user has run.

The chart still reads `--emulated-version`, because emulation moves the
effective version: `--emulated-version=1.32` on a v1.35 tenant puts the
gate back to alpha-and-off, and the header flag is not rendered there.
It still reads the operator's `--feature-gates` entries for an opt-out
(`RemoteRequestHeaderUID=false`, or `AllBeta=false` with no explicit
`=true`), so the header flag never lands next to a disabled gate.
`AllAlpha` is no longer read, since it says nothing about a beta gate.

An unquoted `--emulated-version=value` entry whose kube version has a
non-zero patch component now fails the render and names the entry. The
rule mirrors upstream rather than the regex in #4396: `toVersionMap`
parses each value with `version.Parse` and refuses only `ver.Patch() !=
0` (`staging/src/k8s.io/component-base/compatibility/registry.go` lines
320-326 at v1.35.0, the same code in
`staging/src/k8s.io/apiserver/pkg/util/version/registry.go` at v1.31.0).
`version.Parse` takes `^\s*v?([0-9]+(?:\.[0-9]+)*)(.*)*$`
(`versionMatchRE` in `k8s.io/apimachinery/pkg/util/version/version.go`,
identical at v0.31.1 and v0.35.0). So `1.32`, `v1.32`, `1.32-rc`,
`1.32.0` and `1.32.0-rc` all start an apiserver emulating 1.32, and
`1.32.4` and `1.32.4-rc` do not. A `^[0-9]+\.[0-9]+$` check would have
refused the first four. The chart extracts the numeric run with the same
pattern, reads the minor from it, and fails only on a non-zero third
component. A quoted value is still left unread.

Two deliberate narrowings:

- The chart no longer refuses a mix of bare and `kube:`-prefixed
`--feature-gates` terms. Once the chart writes no gate term, the
spelling of the operator's terms no longer concerns anything it renders.
The apiserver still refuses the mix
(`component-base/compatibility/registry.go:405` at v1.35.0), and the
chart now passes it through like the other fatal gate values it already
leaves to the apiserver: unknown names, unparseable values, quoted
terms.
- On v1.32 the chart now neither renders nor judges the UID flags, as it
already did on v1.31. An operator's own `--requestheader-uid-headers`
entry without the gate reaches an apiserver that refuses it, which is
how every released chart version behaved. A cluster built from `main`
that already runs a v1.32 tenant with OIDC on loses the chart's
`--requestheader-uid-headers` and gate term on the next render; no
release carried them. That holds only while no release is cut from
`main` before this merges; a release that carries the v1.32 gate term
first would turn this into a breaking change for v1.32 tenants, whose
migration is to write both flags into `extraArgs` themselves.

An element such as `kube=1.32=x` is now split at its first `=` only, the
way `toVersionMap` splits it with `SplitN(compVer, "=", 2)`. The chart
used to skip that element, which the apiserver reads as 1.32, and so
rendered the header flag against an emulated 1.32. That gap predates
this branch.

Rebased onto #4398 at `fb2e57c3b`, which reads a two-element
`--feature-gates` and `--emulated-version` separately. That behaviour
carries over unchanged under the v1.33 floor: behind a hidden
`--emulated-version` only an explicit `RemoteRequestHeaderUID=false` is
refused beside a header entry, while a blanket `AllBeta=false` waits for
a reading, because a lower emulated version can make the gate alpha
again; nothing is refused as an opt-out behind a hidden
`--feature-gates` value; the empty-element refusal applies behind both;
and behind either one `X-Remote-Uid` is added only beside a uid-headers
list of the operator's. The README breaking-changes note from #4398 now
says the chart refuses neither the mixed gate spelling nor two kube
emulated versions. In `docs/oidc-tenant.md` the paragraphs that list
refusals now name the managed-flag move and refusal from #4398 and no
longer carry a count, and the two-element paragraph now says what the
chart renders behind a hidden value.

Tests: `helm unittest` goes from 308 on #4398 to 298. 26 cases are
deleted. 25 of them pinned the v1.32 gate term, `AllAlpha` handling or
the mixed-spelling refusal. One emulated a version below 1.32, which the
below-floor emulation case now covers. v1.32 cases that tested parsing
rather than the gate are retargeted to v1.34. Sixteen cases are added:
the patch-level refusal, the version parsing below the floor and above
it (on v1.35 emulating `1.33-rc`, `v1.33`, `1.33.0` and `kube=1.33=x`,
where only a correct minor renders the header), the first-`=` split, a
pin that a mixed gate spelling now passes through, emulation landing
exactly on 1.33, and a v1.32 opt-out beside a header entry passing
through untouched.

Coverage notes, not applied: no case pins that the patch-level refusal
stays off with `spec.oidc.mode: None`, where the block that holds it
does not run. The case "splits a component element at its first equals
sign only" pins that the kube element of `kube=1.32=x` is read. It does
not pin the split itself, because the version pattern ignores what
follows the numbers either way.

The `extraArgs` field description still calls the field the escape hatch
for apiserver flags unrelated to OIDC. It is unchanged here, and it
stays accurate for the v1.32 advice in the guide, because the UID flags
belong to the aggregation layer rather than to OIDC.

Optional, not applied: on v1.32 the chart could refuse an operator's
`--requestheader-uid-headers` entry that comes without the gate, since
the gate is off by default there and the apiserver will not start. The
same skip also drops two refusals the branch history had at 1.32, a
header entry beside `RemoteRequestHeaderUID=false` and a header list
ending in a comma; both now reach an apiserver that refuses them. It
stays out so that v1.32 matches v1.31 and the released charts, where the
chart judges none of these flags.

Not applied: for a value with a zero-prefixed major such as `01.32.4`,
the patch refusal suggests `01.32`. The apiserver refuses both values
over the leading zero, so the configuration fails either way and only
the suggested fix is off.

Why the patch refusal stays while the mixed-spelling refusal goes: the
patch refusal is about `--emulated-version`, which the chart reads
itself to decide what to render. The mixed spelling concerns
`--feature-gates`, which the chart no longer writes to.

Not split out: the `--emulated-version` parsing and the patch-level
refusal share the fix commit with the floor move, because both change
how the same value is read. The change also removes a comment from
`hack/e2e-chainsaw/_lib/run-kubernetes.sh` that became false, so
select-e2e runs the full suite for this PR.

Default path: renders of the parent and this branch through
`tests/values/common.yaml` and `tests/values/oidc-system.yaml` at v1.31
to v1.35 are byte-identical, once the randomly generated Kamaji token
fields are masked, except for OIDC System at v1.32, which loses exactly
the two UID lines.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The only field that changes is the description of `version`, which
returns to its earlier wording. The website's kubernetes reference page
is regenerated from the README on the next tag, and the Terraform
provider keeps its own copy of that description, which already reads
"Kubernetes major.minor version to deploy.".

Closes #4373

Closes #4396

### Release note

```release-note
fix(kubernetes): the tenant chart renders the aggregation-layer UID header flag with OIDC on only from an effective Kubernetes v1.33, and never renders a --feature-gates term of its own. v1.32 tenants render no UID flags, which matches every released version, so nothing regresses. This supersedes the v1.32 clause of the #3539 release note. An unquoted --emulated-version entry with a patch-level kube version such as 1.32.4, which the kube-apiserver refuses, now fails the render and names the entry.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* OIDC-enabled clusters on Kubernetes 1.33 and later now receive the UID
request-header setting without an added feature-gate flag.
* Chart rendering now checks Kubernetes and Talos version compatibility
and prevents rendering when worker deployments do not meet ownership or
retention requirements.
* **Bug Fixes**
* Emulated Kubernetes versions with non-zero patch components now fail
with a clear error; supported version formats are recognized more
consistently.
* **Documentation**
  * Updated version and OIDC guidance to reflect current chart behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-09-25 00:01:56 +04:00
Aleksei Sviridkin 339cffe70f feat(kamaji)!: update to 26.9.4-edge and read every apiServer extraArgs entry (#4398) 
## What this PR does

BREAKING CHANGE: a Kubernetes application whose
`controlPlane.apiServer.extraArgs` names a flag the tenant control plane
now owns fails to render until that entry is removed, except that an
`--enable-admission-plugins=` entry is moved into the new
`controlPlane.apiServer.admissionControllers` and a
`--kubelet-preferred-address-types=` entry into the new
`controlPlane.kubelet.preferredAddressTypes`.

Closes #3541.

Updates the Kamaji build this repo ships and rewrites the part of the
tenant Kubernetes chart whose correctness rested on the old one. The new
pin is `26.9.4-edge`, which is what `make update` picks.

Two upstream changes reach us. The tenant control plane no longer folds
the kube-apiserver argument list into a map keyed by flag name, so
entries supplied through the control plane spec arrive at the apiserver
verbatim, duplicates included. And the reset that rewrites those
arguments from scratch keys on a hash of the list rather than on its
length, so swapping one flag for another no longer strands the old one
on the running Deployment.

The OIDC block in the tenant chart read only the last entry of each
flag. That was right while earlier entries were dropped before the
apiserver saw them, and it is wrong now. An operator whose `extraArgs`
carried `--feature-gates=RemoteRequestHeaderUID=false` followed by an
unrelated `--feature-gates` entry got
`--requestheader-uid-headers=X-Remote-Uid` rendered next to a gate the
apiserver has switched off, and the control plane would not start. A
blanket `AllBeta=false` has the same shape, and so does an
`--emulated-version` entry naming kube followed by one naming another
component.

The chart now reads every entry with the accumulation each flag
performs. Occurrences of `--requestheader-uid-headers` append into one
list, so where a list of the operator's own omits `X-Remote-Uid` the
chart adds its own element beside it instead of refusing to render: the
union is what the apiserver validates, and their header keeps working.
Feature-gate terms collect per component and are applied in one pass, so
the chart carries only its own term, placed last where it wins. Naming
the kube component both bare and `kube:`-prefixed now fails the render,
because the apiserver refuses that configuration and nothing the chart
renders repairs it. `--emulated-version` accumulates too, and two
entries naming kube stop the apiserver outright, so the chart renders
nothing rather than pick a version that will not exist.

The same upstream change narrows what `extraArgs` can do, and the chart
now refuses the part that stopped working. Kamaji 26.9.4-edge sets a
list of kube-apiserver flags itself and drops any `extraArgs` entry that
names one before the apiserver sees the list, where `26.3.6-edge` merged
`extraArgs` last and let the entry win. A tenant with
`--enable-admission-plugins=NodeRestriction,AlwaysPullImages` loses both
plugins on the first reconcile after the upgrade, with nothing to say
so. The drop happens on the running control plane whatever the chart
renders, so the chart cannot prevent it; what it can do is refuse to
render such an entry, in every `oidc.mode`, and name it. The list
mirrors `buildKubeAPIServerCommand` at the pinned tag for the control
plane this chart renders (an etcd DataStore, konnectivity on, no
`dataStoreOverrides`), and the `controlPlane.apiServer.extraArgs`
description in the chart README carries it. A flag the control plane
only defaults, such as `--authorization-mode` or
`--requestheader-group-headers`, is still yours to set.
`--enable-admission-plugins` gets a replacement: the new
`controlPlane.apiServer.admissionControllers` passes a list through to
the KamajiControlPlane `spec.admissionControllers`, which the control
plane renders into that flag, and the application API checks each name
against the enum the KamajiControlPlane CRD declares. The list is
rendered only when non-empty, so a tenant that sets nothing renders
exactly as before, and a non-empty list replaces the control plane's
default set, as the `extraArgs` entry did under the previous build.
`--kubelet-preferred-address-types` gets the same treatment: the list
the chart used to hardcode on the KamajiControlPlane is now
`controlPlane.kubelet.preferredAddressTypes`, which renders that list
when left empty. Neither entry is refused where it can be moved: an
`--enable-admission-plugins=` or `--kubelet-preferred-address-types=`
entry is moved into its field while the field is empty, on every render,
and left out of the rendered `extraArgs`, so a tenant carrying one keeps
its setting across the Kamaji update with no edit and no ordering to get
right. The entry is refused beside a non-empty field, when it names
nothing, and in the two-entry spelling.

A separate defect in the same package is fixed here. The
TenantControlPlane schema reaches existing clusters through the
`kamaji-crds` subchart's templated CRDs, which every release upgrade
applies; `crds/` holds the copies a fresh install lays down first.
Directory-based renders lost that schema entirely, `helm template`,
`lint` and `package` alike, because the parent `.helmignore` matched the
subchart's `hack/` directory at any depth and that is where the
templates keep the schema they read with `.Files.Get`. Helm honours
`.helmignore` for a directory and skips it for an archive, so the
cluster never saw the difference and no local check could catch a schema
defect. The patterns are anchored to the chart root and a unit test
asserts the rendered CRDs carry their versions.

The bump is not cleanly reversible, and that is the one thing to know
before rolling it out. The args annotation holds a checksum once the new
build has written it, and the old build reads that annotation with
`strconv.Atoi`. The parse fails and `resetKubeAPIServerFlags` returns
early, ahead of both the args reset and the annotation write that would
put a number back. So after a rollback to `26.3.6-edge` the count-keyed
reset is disabled on every TenantControlPlane: a later removal of an
`extraArgs` entry no longer clears the flag from the running Deployment,
and it stays that way until someone writes a number into
`kube-apiserver.kamaji.clastix.io/args` by hand.

Findings from a live upgrade on a dev cluster, none of them caused by
the diff. The DNS-service-IP check moved out of a CEL rule on the CRD
into the admission webhook (`internal/webhook/handlers/tcp_dns.go`), so
it now depends on that webhook being reachable where the apiserver used
to enforce it by itself. The manager's readiness probe targets port 8081
while the webhook serves on 9443, so a Ready pod does not prove 9443
accepts connections; with `failurePolicy: Fail` on the
TenantControlPlane webhook, that window rejects writes rather than
letting them through unvalidated. The readiness probe this build adds to
kube-scheduler and kube-controller-manager survives a rollback, since
the old build neither sets nor clears those two; that one is benign.

Seven limits worth naming rather than leaving to be found. The
mixed-spelling refusal quotes the entry carrying each spelling, so one
entry holding both, `--feature-gates=Foo=true,kube:Bar=true`, is quoted
twice in its own message; the refusal is right and the message reads
oddly. The unit test that pins the rendered CRD schemas asserts on a
count, so a CRD missing from the render altogether fails on an empty
comparison rather than naming the CRD that went missing. There is one
window during the upgrade itself: the chart used to fold an operator's
own `--feature-gates` terms into the entry it rendered, because the old
build would otherwise drop them, and it now renders only its own term.
Nothing orders a tenant release after the Kamaji operator has rolled, so
a `v1.32` tenant that sets gate terms of its own with OIDC on can lose
them until the new operator reconciles it, which it then does on its
own. The same window takes a `--requestheader-uid-headers` list of the
operator's own that lacks `X-Remote-Uid`: the chart now renders its own
entry beside it instead of refusing, and the old operator keeps one
value per flag name, the chart's, so the operator's header is dropped
until the new operator reconciles. Which entry an error message quotes
is pinned only where one entry is eligible. Three variables choose
between equally eligible entries, the gate dictionary taking the last
setting for a name and the uid-headers ones the first, and no test
distinguishes first from last when two entries qualify, so a future edit
could make a message name the wrong entry of the two without a test
noticing. What the chart renders and what it refuses is pinned
exhaustively; it is the provenance in three message strings that is not.
The block is deliberately inconsistent about two shapes that stop the
apiserver equally hard: two `--emulated-version` entries naming kube
make it withhold silently, while a kube-prefixed and a bare gate
spelling in the same configuration make it refuse the render. The
refusal is right where the apiserver rejects the configuration whatever
the chart does; the withholding is right where the chart cannot read a
version it must render against. They are two rules, not one, and a third
that covers both is fair to ask for. Moving the refusal out from under
those conditions so it fires everywhere is the obvious third rule and it
is not free: the apiserver refuses the mix on `v1.31` too, but from
`k8s.io/apiserver/pkg/util/version` rather than from the component-base
`componentGlobalsRegistry.Set()` the message names, so hoisting it means
rewriting that sentence as well. Reading a flag and its value written as
two entries would be better than withholding for it, and is a change of
behaviour rather than a correction, so it is not here. And two sentences
about `--emulated-version` in `docs/oidc-tenant.md` sit thirty lines
apart and read as a retraction in sequence, though each is true at its
own layer: `toVersionMap` accepts an element naming a foreign component,
and the registry rejects it later.

What this leaves open, by name. The guard for a valueless flag at the
end of `extraArgs` covers `--requestheader-uid-headers`,
`--feature-gates` and `--emulated-version` only; any other flag that
takes a value swallows the appended `--authentication-config` the same
way, which is an input mistake the chart does not look for. The
managed-flag refusal matches names as written, as Kamaji does: an
underscore spelling such as `--secure_port` passes both, and the
apiserver normalises it to `--secure-port`, where the operator's value
wins as the later one, the same as before this change. Rendering
`--authentication-config` ahead of the operator's entries would close
that class for OIDC, since only the UID flags would then follow them; it
changes the rendered order, and so rolls every OIDC tenant that has
`extraArgs` of its own, and belongs in a separate change after the OIDC
floor move. One pre-existing comment needs checking: the note above
`extraArgs` in `templates/cluster.yaml` says that omitting the field
would leave the live array untouched under JSON merge patch, which reads
against Helm's `createPatch`, where a key the previous manifest carried
and the new one does not is patched to null; what that comment was
guarding against is still to be verified. The KamajiControlPlane CRD
accepts only the plugin names it lists, so a plugin outside that list
that the apiserver leaves off by default can no longer be enabled; that
limit is Kamaji's, since it owns the flag and restricts the field. A
name moved out of an `extraArgs` entry is not checked against that list
by the chart, so one outside it fails when helm-controller applies the
KamajiControlPlane, with an error that names it, rather than at render;
the chart keeps no copy of the list to check it with. A repeated address
type in a moved entry is dropped, keeping the first place it appears,
because the field is a set. With `oidc.mode: None` the chart does not
read `extraArgs` for the OIDC flags at all, and two shapes regress with
the bump there: a kube feature-gate term written both bare and
`kube:`-prefixed across entries, and two `--emulated-version` entries
naming kube. The old build kept one entry per flag, the new one passes
both, and the apiserver refuses both at startup. Next to a flag and
value written as two entries the chart keeps the refusals that hold
whatever the hidden value is, but it does not read positions: a hidden
`--feature-gates` term keeps the opt-out refusal off even when it sits
before the explicit opt-out and so cannot undo it.

What the live run could not exercise: the konnectivity datapath, because
the stand had no tenant workers, and `--requestheader-uid-headers`,
which comes from the tenant Kubernetes chart and no image swap reaches.
Both are covered by render and unit tests only.

One shape the old collapse was quietly repairing, with OIDC on, needs a
guard now that it is gone. A bare `--requestheader-uid-headers` in
`extraArgs` used to be overwritten by the entry of the same name the
chart renders, because the collapse kept one value per flag name; the
same held for a bare `--feature-gates` on `v1.32`, where the chart
renders a gate entry too. A bare `--emulated-version`, or a bare
`--feature-gates` on any other version, had no entry of the chart's to
be overwritten by and was emitted bare in sorted position, where it took
the next flag as its value, so it never worked. Without it those entries
reach the apiserver as written, and the apiserver gives a valueless long
flag the next argument whatever that argument looks like. What follows
the operator's last entry is the `--authentication-config` the chart
appends, so the flag swallows it and the AuthenticationConfiguration is
never applied. On `--requestheader-uid-headers` the path is accepted as
a header name and the control plane comes up with OIDC silently off,
failing every login for that tenant with nothing in the cluster to say
why; the other two reject it and the apiserver exits. The chart refuses
to render that shape and says which entry to fix.

Two things in the image build fall out of the bump. The datastore
deletion fix carried here as a patch landed upstream verbatim, so its
patch no longer applies, and one failing patch fails the whole apply of
the directory. The Go module overlay is obsolete for the opposite
reason: this tag already carries `golang.org/x/net`,
`golang.org/x/crypto` and `go.opentelemetry.io/otel/sdk` above the
versions the overlay forced, so keeping it would downgrade all three.

What I checked: `helm unittest` passes at 307 cases for the package,
with the OIDC suite carrying every shape above, and the image builder
stage builds end to end at the new tag with the one remaining patch.
Kamaji's own support for Kubernetes 1.36 and 1.37 does not move the
tenant version enum here; that is a separate update of the `kubernetes`
package. No image reference in `packages/system/kamaji/values.yaml` is
touched, since CI stamps that digest.

### Screenshots

No UI changes.

### Downstream repositories

I walked the trigger map file by file against the diff. The chart
templates, the vendored Kamaji chart, the image build and the docs reach
nothing downstream. The one generated file that reaches downstream is
`packages/apps/kubernetes/values.schema.json`, which gains
`controlPlane.apiServer.admissionControllers` and
`controlPlane.kubelet.preferredAddressTypes` and a changed `extraArgs`
description; both new fields are optional with defaults, so a stored
Kubernetes object needs no change, no field is removed or renamed and no
version enum moves. The new fields are the one trigger the map fires on:
the Terraform provider keeps a hand-written schema.
`packages/apps/kubernetes/README.md` feeds the website's managed-apps
reference pages, which a bot regenerates on a stable tag.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: model `controlPlane.apiServer.admissionControllers` and
`controlPlane.kubelet.preferredAddressTypes` on the kubernetes resource
(schema, model, expand/flatten).
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(kamaji): update Kamaji to 26.9.4-edge. Tenant kube-apiserver arguments are now derived from a hash of the control plane's extraArgs instead of from its length, so on the first reconcile after the upgrade every tenant control plane rewrites those arguments from scratch and rolls once. The rebuilt list also orders them differently, Kamaji's own flags sorted first and the operator's kept verbatim after them, so the roll happens whether or not the tenant sets any extraArgs. A tenant with OIDC enabled takes a second roll when the chart's own flags land. Rolling the Kamaji image back to an earlier build does not restore the old behaviour: the older code reads that annotation as a number, gives up when it finds a checksum, and from then on never clears a flag you remove from extraArgs, on any TenantControlPlane, until the annotation is set back to a number by hand. The tenant Kubernetes chart reads every extraArgs entry the way the apiserver accumulates them: a --requestheader-uid-headers list that omits X-Remote-Uid now gets the chart's own element beside it instead of failing the render, a feature gate set in an earlier entry is honoured, and naming the kube feature-gates component both bare and kube:-prefixed fails the render, because the apiserver refuses that configuration outright. Writing --feature-gates or --emulated-version as a flag and a value in two separate entries now keeps the chart from rendering UID flags on its own, because an entry the chart cannot read may hide an opt-out; a --requestheader-uid-headers list of your own still gets X-Remote-Uid beside it, and write --flag=value to get the rest back. Action required for tenants that pass their own apiserver flags: Kamaji now sets a list of flags itself and drops a controlPlane.apiServer.extraArgs entry naming one, where the previous build let that entry win, and the Kubernetes application now refuses to render such an entry, except for the two it moves into a field. A tenant carrying one of the others stops reconciling after the upgrade, and no later change to that tenant applies, until the entry is removed; the entry no longer has an effect either way. The list is in the description of controlPlane.apiServer.extraArgs. An --enable-admission-plugins=value entry is moved into the new controlPlane.apiServer.admissionControllers, a list that replaces the control plane's default set as the entry did, and a --kubelet-preferred-address-types=value entry into the new controlPlane.kubelet.preferredAddressTypes, while that field is empty; nothing has to be edited for them. Before upgrading, list the affected entries with: kubectl get kuberneteses.apps.cozystack.io --all-namespaces --output json | jq --raw-output '["--advertise-address","--client-ca-file","--enable-admission-plugins","--service-cluster-ip-range","--kubelet-client-certificate","--kubelet-client-key","--kubelet-preferred-address-types","--proxy-client-cert-file","--proxy-client-key-file","--requestheader-allowed-names","--requestheader-client-ca-file","--secure-port","--service-account-key-file","--service-account-signing-key-file","--tls-cert-file","--tls-private-key-file","--etcd-servers","--etcd-cafile","--etcd-certfile","--etcd-keyfile","--etcd-prefix","--etcd-compaction-interval","--egress-selector-config-file"] as $owned | .items[] | .metadata as $m | (.spec.controlPlane.apiServer.extraArgs // [])[] | select((split("=")[0]) as $f | $owned | index($f)) | "\($m.namespace)/\($m.name): \(.)"' Entries it lists for --enable-admission-plugins or --kubelet-preferred-address-types need a change only when written as two entries, when they name nothing, or next to a non-empty field of their own.
```






<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Configure tenant control-plane admission controllers and kubelet
preferred address types.
- Choose a cert-manager issuer for webhook certificates; self-signed
certificates remain the default.
- **Bug Fixes**
- Improved OIDC argument handling for repeated, split, quoted, and
conflicting settings, with clearer render-time errors.
- DataStores are recognized as unused when no tenant control planes
reference them.
- **Validation**
- DataStore endpoints are checked for length, count, and IPv6 address
formatting.
- Chart-managed tenant control-plane arguments are rejected when
supplied through `extraArgs`.
- **Documentation**
  - Updated Kubernetes OIDC and certificate issuer guidance.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-09-24 22:26:41 +04:00
Alexey Artamonov ce74fdeea9 docs(backups): describe restored credentials and pre-upgrade residue 
Say that a restored copy converges its application passwords shortly
after recovery, that MariaDB backups leave out the grant table, and that
dropping the password field does not rewrite copies made before the
upgrade: old Backup snapshots and Helm revisions still hold the password
that stays live. The credentials projector comment counts only the
engines that still take a password from values.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
 2026-09-24 18:11:07 +03:00