Commit Graph
7497 Commits
Author SHA1 Message Date
Andrey Kolkov e4e991e837 fix(backups): skip the CNPG restore purge only for this restore's Cluster
The in-place CNPG restore skips deleting the live Cluster when it
carries bootstrap.recovery and was created after the RestoreJob started,
taking that as "this restore's own purge and re-render produced it". The
timestamp does not say which restore rendered the Cluster. A Cluster
deleted while the restore waits for the WAL archive is re-rendered by
the still-active HelmRelease from the previous restore's values and is
created after StartedAt too; the restore then skipped its purge and
reported Succeeded on the previous restore's data.

The restore patch already sets bootstrap.newServerName to
restoredServerName(cluster, RestoreJob UID). The chart now annotates the
Cluster with it as postgres.cozystack.io/restored-server-name, whether
or not it renders the barman plugin that also carries the value, and the
purge is skipped only when the annotation names this RestoreJob. The
mark lives on the rendered Cluster rather than in RestoreJob status, so
it still covers the failed status write the skip exists for. A Cluster
rendered by a chart that predates the annotation keeps the creation-time
check.

Fixes #3322

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-10-10 23:49:08 +03:00
Andrey Kolkov 3daf942dce fix(backups): refuse a restore into an application another restore holds
Nothing stopped two RestoreJobs from rewriting one application at once.
Each restores the target from its own Backup, so the application ends up
on whichever finishes last, and the CNPG driver's purge decision cannot
tell which of the two produced the live Cluster: the second job could
skip its purge and report Succeeded on the first job's data.

A RestoreJob that has not started now fails with reason
ConcurrentRestore when another unfinished RestoreJob in the namespace
writes into the same application and goes first. A started restore goes
before one that has not; between two that have not, the older one goes
first, by creation time and then name, so a running restore is never
failed by a newer one. The refused job fails instead of waiting, so the
application is not restored a second time later, after the first
restore, by a job its author may no longer expect to run.

The gate lists RestoreJobs through the manager's API reader rather than
the informer cache. Creation times have one-second precision, so two
jobs created in the same second are ordered by name, and a job reconciled
before the cache held its peer's StartedAt would see that peer as not
started and proceed alongside it. The controller runs one worker, so a
peer's StartedAt written by an earlier reconcile is on the API server by
the time the next job reads it; SetupWithManager now states that.

The target is resolved as the BackupJob hold resolves it, so a Velero
restore into another namespace does not collide with one in place.

Refs #3322

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-10-10 23:49:08 +03:00
Aleksei Sviridkin 22e0fdbc63 test(tests): stop unit tests from reaching a cluster (#4804)
## What this PR does

One unit test was reading from whatever cluster the caller's kubeconfig
selected. The knob test in `hack/run-kubernetes-node-join_test.bats`
runs `cozy_report_node_join_failure` in a fresh `bash -c` and stubs only
`kubectl`, as a shell function. `cozy_diag_read` runs every read through
`timeout`, and `timeout` execs the real `kubectl` from PATH, so the stub
never ran. With a recording `kubectl` first on PATH, one run of that
file logged 170 calls such as `kubectl -n tenant-test describe
virtualmachineinstances.kubevirt.io`. All of them were reads, but a unit
suite should not touch a cluster at all.

The PR has three parts:

1. Both `bash -c` blocks in that file now stub `timeout`, as the
post-source test already did. They also put recording `kubectl` and
`talosctl` executables first on the subshell's PATH and fail if either
one is executed. The declined-path block made no calls in my runs,
because a zero budget declines every read. It gets the same guard
because it has the same shape.
2. The unit and controller targets now run with no cluster to reach. The
root `Makefile` sets `KUBECONFIG=/dev/null` and an empty
`KUBERNETES_SERVICE_HOST` for `unit-tests`, each of its prerequisites,
and `test-controllers`, so the CI command and each target run on its own
are covered. The second variable is needed because client-go falls back
to in-cluster credentials when the kubeconfig is empty and that variable
plus a service-account token are present. The new
`hack/unit-lane-no-cluster.bats` runs every prerequisite of `unit-tests`
as its own goal with a sentinel kubeconfig and service host in the
environment, and fails if any target receives them.
3. One unit file was missing the strict-mode helper.
`hack/bats-strict-setup.bats` requires a `load test_helper` line in
every unit file, and `hack/fork-new-images.bats` is the only one without
it, so the bats lane is red on main and would stay red here. #4805
carries the same one-line fix on its own; if it lands first, this commit
drops out on rebase.

I ran the whole lane (`make unit-tests test-controllers`) with recording
fakes for 20 cluster clients first on PATH. It made no `kubectl` call.
`helm` was called 444 times, all `template` or `unittest`, none with
`--validate`. `flux` was called once, for `push artifact --help`.
Deleting either export line from the Makefile, or removing one target
from the list, turns the new test red. Removing the `timeout` stubs
turns the knob test red, and the failure lists the reads that escaped.

Closes #4793

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
test(tests): unit and controller test targets run with no kubeconfig and no in-cluster service host, so a unit test can no longer reach a cluster.
```
2026-10-10 23:47:52 +03:00
Aleksei Sviridkin bcde187858 fix(dashboard): small Console fixes from the #3828 triage (#3837)
## What this PR does

Six small Console fixes from the #3828 triage. Each one is small and
most of the code sits in `apps/console/src`, so they go together in one
PR.

`Fixes #3105`: sizes between 1Ki and 1Mi printed as a raw byte count.
`humanizeBytes` had branches for Ti, Gi and Mi and then fell through to
bytes. It now has a Ki branch with the same `toFixed(0)` as the Mi
branch. The `1023B` case still holds, and the test covers 1Ki and 512Ki.

`Fixes #3106`: the tenant picker showed on admin pages where the tenant
changes nothing. It is now hidden on the cluster-scoped admin pages
only: Capacity, Backup classes, External IPs and Modules. The rest of
`/admin` mounts the same tenant-scoped pages as `/console`, and those
take their namespace from the tenant context, so the picker stays there.
`/admin/tenants` keeps it too, because its rows come from the
picker-filtered tenant list.

`Fixes #3107`: both capacity drill-downs showed one generic error, so a
permission failure looked like a broken page. Both now use the `error
instanceof K8sApiError && error.status === 403` check that
`ClusterStorageSection` already uses, with a test for the forbidden case
and one for a generic error on each page.

`Fixes #3102`: an immutable leaf was not restored when the YAML edit
dropped its parent object. `overlayPath` stopped as soon as the
submitted body had nothing at a path segment. It now creates `{}` for
the missing parent, but only when the target there is undefined or null
and the original really has the leaf to restore. So a scalar or an
explicit null the user put there survives. A wildcard with more path
under it does not rebuild a deleted parent, since that would put back
entries the user removed. The test uses foundationdb's
`storage.storageClass`, which is a shipped path of this shape.

`Fixes #3135`: a blocked submit did not scroll to the field that blocked
it. Plain `focusOnFirstError` crashes here, because RJSF's built-in
handler reads `form.elements` and this form is `tagName="div"`. So the
form passes a small handler that finds the field by its generated id and
scrolls it into view. The ids come from one shared definition, and the
widgets that had none now carry it: the `DynamicOptionsWidget` select,
which every `x-cozystack-options` field uses, and the `SourceField`
controls. The nested form that `AdditionalPropertiesField` mounts per
map entry continues the outer id namespace instead of restarting at
`root`, so fields like `nodeGroups.*.instanceType` resolve. A map key
with a dot in it, such as a mariadb user `john.doe`, resolves by trying
both readings of the dotted error path.

`Fixes #3108`: `cilium`, `coredns` and `verticalPodAutoscaler` have only
`valuesOverride` and no `enabled`, so they rendered as plain groups next
to the toggleable addons and looked like a switch that failed to appear.
Setting `<addon>.enabled: true` in the YAML editor is accepted, because
the schema sets no `additionalProperties`, and the API stores the field
and echoes it back. Nothing reads it. The three addons are not the same
case. `cilium` and `coredns` are always installed.
`verticalPodAutoscaler` has no switch of its own and is installed and
removed together with `addons.monitoringAgents.enabled`. The three are
now marked in `values.yaml` with a new `x-cozystack-no-enable-switch`
schema keyword, and the form renders them with a note that there is no
enable switch and an `enabled` field in YAML does nothing. The per-addon
reason goes into the schema description, which the form renders and
which regenerates into the README, the Go types and the
ApplicationDefinition.

The generated files are regenerated with cozyvalues-gen v1.7.0, the
version CI pins, and the pre-commit check passes. The console typecheck
and tests pass in CI.

### Screenshots

##### Ki formatting
<img width="1182" height="184" alt="image"
src="https://github.com/user-attachments/assets/62662265-4f77-455d-b90e-85e417684bb9"
/>

##### Admin without tenant picker
<img width="1440" height="245" alt="image"
src="https://github.com/user-attachments/assets/99aff4f3-fda9-4c4c-8cf8-0c1eb01fb973"
/>

##### Capacity 403 error
<img width="1232" height="270" alt="image"
src="https://github.com/user-attachments/assets/f4e4b328-ee5e-4516-b78f-9b5084b6bbc8"
/>
<img width="1232" height="944" alt="image"
src="https://github.com/user-attachments/assets/142e438e-f538-436e-aae8-dfb79ec747e8"
/>

##### Validation focus
<img width="1166" height="135" alt="image"
src="https://github.com/user-attachments/assets/89abb73b-a624-406f-9736-3bf38bb8a923"
/>

##### Addon lifecycle
<img width="1166" height="591" alt="image"
src="https://github.com/user-attachments/assets/14a95891-84d1-4073-a009-bc0eab94f114"
/>

### Downstream repositories

I walked the trigger map in `docs/agents/contributing.md` against the
diff, file by file. This PR changes field descriptions, adds the
`x-cozystack-no-enable-switch` schema keyword on three addon objects,
and changes the console. No field is added, removed or renamed, and no
default, version enum, `kind`, `plural`, `release.prefix`, output Secret
or Service changes, so none of the terraform provider triggers fire. No
package is added or removed, and nothing changes in
`packages/core/platform/values.yaml`, the variants, `hack/` or the node
prerequisites. cozyvalues-gen itself is unchanged, since the keyword
goes through its generic `@x-*` passthrough. The kubernetes reference
page on the website regenerates from `README.md` on a stable tag, which
is automated rather than a follow-up.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(dashboard): Ki-range sizes now render as Ki instead of raw bytes, the tenant picker is hidden on cluster-scoped admin pages, capacity drill-downs distinguish a permission error from a broken page, an immutable field whose parent object is absent is now applied, a blocked submit scrolls to the field that blocked it including fields inside maps and the source and dropdown widgets, and the cluster addons with no enable switch now say so instead of showing a toggle that never appears
```
2026-10-10 23:47:46 +03:00
Aleksei Sviridkin 1b0a5a2829 fix(velero): bump velero-plugin-for-aws to v1.14.4 so backups work on R2 (#4834)
## What this PR does

Bumps the `velero-plugin-for-aws` init container in
`packages/system/velero/values.yaml` from `v1.14.1` to `v1.14.4`, and
updates the pin in `tests/velero_test.yaml`.

`v1.14.1` sets `x-amz-tagging` on every `PutObject`, even when the
BackupStorageLocation configures no tags, in which case the header is
empty. Cloudflare R2 rejects that with `501 NotImplemented: Header
'x-amz-tagging' with value '' not implemented`. Velero then cannot
upload backup metadata, so every Velero backup to an R2-backed location
fails. That includes Cozystack's own `cozy-default`
BackupStorageLocation when the platform backup bucket is on R2.

Upstream fixed this in v1.14.3: velero-plugin-for-aws#309 (a backport of
#299, "Only set PutObject Tagging when tags are configured"). `v1.14.4`
is the current patch release of the 1.14 line. Its only change on top of
v1.14.3 is CVE dependency bumps (#336). v1.14.3 itself also carries #310
(SSE-C presigned URLs), #313 (no amz-checksum encapsulation when
`checksumAlgorithm=""`) and CVE/Go bumps. The plugin's compatibility
table lists `v1.14.x` for Velero `v1.18.x`, which is what the chart
ships (`velero:v1.18.1`), so this stays inside the supported pairing.

Testing done:

- `make test` in `packages/system/velero` (helm-unittest v1.1.2): 7
passed, with the existing init-container pin assertion moved to
`v1.14.4`.
- `helm template` renders `velero/velero-plugin-for-aws:v1.14.4` as
`initContainers[0]` of the `velero` Deployment.
- On a Cozystack v1.6.4 staging cluster with the backup bucket on
Cloudflare R2: before the bump, Velero backups failed on the empty
`x-amz-tagging` header. After pinning the same plugin image through a
Package values override
(`docker.io/velero/velero-plugin-for-aws:v1.14.4@sha256:c85a7fa803569e3128c33e4f5766f84ab156c8d171c9f1dd2715581d45ce98fa`),
all of our volume backup and restore drills pass. That covers CSI
snapshots with the Kopia data mover to R2, and incremental backups
confirmed.

Backwards compatibility: same plugin minor line and the same Velero
version. The init container image changes, so the `velero` Deployment
rolls once on upgrade. Users who never set `tagging` on a location now
get no tagging header at all; users who do set it get the same header as
before.

The reference stays in the existing shape (tag only, no digest), like
the current `v1.14.1` line. If you'd rather pin third-party init
containers by digest, the v1.14.4 multi-arch index digest is
`sha256:c85a7fa803569e3128c33e4f5766f84ab156c8d171c9f1dd2715581d45ce98fa`.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Trigger map walked against the diff: a third-party image tag in a system
package values file and its test; no app schema, platform values, names
or tooling.

Fixes #4852

### Release note

```release-note
fix(velero): velero-plugin-for-aws is bumped from v1.14.1 to v1.14.4, which no longer sends an empty x-amz-tagging header, so Velero backups work against S3-compatible stores that reject it, such as Cloudflare R2.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Updates**
  * Updated the AWS backup plugin used by Velero to version 1.14.4.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-10 23:37:43 +03:00
Aleksei Sviridkin cd1345e661 fix(redis): re-send the Sentinel auth-pass when the operator resets a Sentinel (#4833)
Fixes #4851

## What this PR does

Adds a downstream patch to the redis-operator image
(`packages/system/redis-operator/images/redis-operator/patches/unauthenticated-sentinel.diff`)
so that a Sentinel which lost its `auth-pass` is repaired by the
operator instead of being reset forever.

### The failure

After a node power cycle on a Cozystack v1.6.4 cluster (3 nodes, Talos,
HA Redis with `authEnabled: true`), the Sentinel pod on that node came
back monitoring the correct master address, but with no `auth-pass`. The
master answered it with `NOAUTH`, it reported the master as
`s_down,disconnected`, and `SENTINEL CKQUORUM mymaster` on it returned
`NOQUORUM 1 usable Sentinels`. The operator logged `mismatch number of
sentinels in memory. resetting` on every resync and sent `SENTINEL RESET
*`. That does not touch `auth-pass`, so the Sentinel never recovered on
its own: the instance keeps running with two working Sentinels out of
three until someone deletes that pod. One more failure then leaves
Sentinel without a majority.

The mechanism, which I reproduced locally with Redis 8.6.2 (script and
output in #4851):

1. The pod restarts in place (same pod, same IP), so the
`sentinel-config-copy` init container writes the operator's initial
`sentinel.conf` again: `sentinel monitor mymaster 127.0.0.1 6379 2` and
no `auth-pass`.
2. The other Sentinels still know that address and keep sending their
hello messages to it directly. A failover has raised the config epoch,
so the restarted Sentinel adopts the current master address from a peer
(`+config-update-from sentinel ... +switch-master mymaster 127.0.0.1 ...
<current master>`) before the operator looks at it.
3. `CheckAndHeal` then finds the right address in `CheckSentinelMonitor`
and skips `NewSentinelMonitor`, the only call that sends `SENTINEL SET
<master> auth-pass`.
4. `INFO sentinel` reports `status=sdown`, so
`CheckSentinelNumberInMemory` fails and `RestoreSentinel` sends
`SENTINEL RESET *`. The reset keeps the empty `auth-pass`, and the cycle
repeats.

The same code is on `main`: freshworks-oss v3.3.5 sets `auth-pass` only
in `MonitorRedisWithPort`, and none of the three patches Cozystack
carries touch this path.

### The change

`RestoreSentinel` sends `SENTINEL SET <master> auth-pass <password>`
right after `SENTINEL RESET *` when the RedisFailover has auth enabled.
It uses the same password `NewSentinelMonitor` uses
(`k8s.GetRedisPassword`), the same TLS config, and the same master name
(`rf.MasterName()`, so `disableMyMaster` is respected).

- It only runs on the unhealthy path. `RestoreSentinel` is called only
when a Sentinel's view is already wrong: its sentinel or replica count
does not match, or its status is not `ok`. A healthy Sentinel is never
touched.
- It does not disturb a working Sentinel. `SENTINEL SET auth-pass` drops
the Sentinel's links to the monitored instances, but the `RESET` just
before it has already done that.
- Without auth, `GetRedisPassword` returns an empty string and nothing
is sent, so the behaviour is exactly as before.
- Bootstrap mode goes through the same function and gets the same
password `NewSentinelMonitorWithPort` already sends there.

Patch contents: a `SetSentinelAuthPass` method on
`service/redis.Client`, its mockery-style mock, the two-step
`RestoreSentinel`, and `TestRestoreSentinel` (5 cases: auth off, auth
on, reset error, unreadable Secret, `SET` error).

The file name sorts after `tls.diff` on purpose: the patch is written on
top of it (same functions, new `tlsConfig` parameters), and the
Dockerfile applies `patches/*.diff` in glob order. If you prefer
numbered patch names, I can rename it.

Like `tls.diff`, this belongs upstream too: the same fix is open as
freshworks-oss/redis-operator#90. This carried patch can be dropped once
a release containing it is vendored here.

### Why not a quorum-checking liveness or readiness probe

The RedisFailover CRD does allow `spec.sentinel.customLivenessProbe` /
`customReadinessProbe`, so a `SENTINEL CKQUORUM` probe could be set from
the chart. I looked at that first and decided against it:

- A liveness restart does not repair this. It restarts the container,
not the pod. `sentinel.conf` lives on an `emptyDir` that survives a
container restart, and only the init container rewrites it, on pod
start. The restarted Sentinel reads the same file without `auth-pass`
and fails again, so the probe would restart it in a loop.
- `CKQUORUM` does not reliably detect it. In the local reproduction the
broken Sentinel still answered `OK 3 usable Sentinels`, because its
peers reach it directly. On the cluster it answered `NOQUORUM`. What
does show the problem every time is `status=sdown` in `INFO sentinel`,
which the operator already reads.
- It hurts in real outages. With two of three nodes down, a quorum-based
liveness probe would restart the last Sentinel, and a quorum-based
readiness probe would empty the `rfs-` Service, at the moment clients
most need it.

The operator already detects this state and already runs a repair for
it. The repair just did not include the password.

### Testing done

- Applied the four patches to a fresh v3.3.5 tarball in Dockerfile order
(`git apply patches/*.diff`). Then ran the Dockerfile's test step: `go
test .` and `go vet -tags integration .` in every directory the patches
touch, all 11 pass. Also ran `GOOS=linux GOARCH=amd64 go build
./cmd/redisoperator`. I could not run `make image` because Docker was
not available on my machine.
- `TestRestoreSentinel` fails on 3 of its 5 cases without the `heal.go`
change and passes with it. The existing `operator/redisfailover/...` and
`service/redis` tests pass unchanged.
- End-to-end against real Sentinels (Redis 8.6.2, local): master +
replica with `requirepass`, three Sentinels configured the way the
operator configures them, a `SENTINEL FAILOVER`, then one Sentinel
restarted on the same port with the operator's initial config. Result:
that Sentinel switched to the new master from a peer's hello and sat in
`status=sdown`, `flags s_down,master,disconnected`. `SENTINEL RESET *`
alone left it in `sdown`. The patched `service/redis` client
(`ResetSentinel` then `SetSentinelAuthPass`) brought it to
`status=ok,...,slaves=1,sentinels=3`.
- The mock was written in the format mockery v2.45.0 generates; v2.45.0
itself does not build with Go 1.26+ (`golang.org/x/tools` v0.17.0), so I
could not regenerate it mechanically.

Backwards compatibility: no API or chart change. The operator image has
to be rebuilt to pick up the patch, like the existing patches (the image
digest in `packages/system/redis-operator/values.yaml` is stamped by the
build). Instances without auth are unaffected.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Trigger map walked against the diff: one new file under
`packages/system/redis-operator/images/redis-operator/patches/`; no
values, schema, CRD, name or tooling change.

### Release note

```release-note
fix(redis): the redis-operator now re-sends the Redis password to a Sentinel it resets, so a Sentinel that restarted in place (for example after a node reboot) and lost its auth-pass recovers on the next check instead of staying down until its pod is deleted.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Sentinel recovery now restores the Redis authentication password after
a successful reset when a password is configured.
* Recovery stops if the reset fails and reports errors encountered while
retrieving or applying the password.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-10 23:36:12 +03:00
Aleksei Sviridkin f4bde9e50d fix(backups): delete the original PVC of an in-place VM restore without keepOriginalPVC (#4672)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

An in-place VMInstance restore with `keepOriginalPVC: false` reports
`Succeeded` without restoring the disks, and leaves every disk it
touched unable to come back. `prepareForRestore` deletes each DataVolume
with Orphan propagation and leaves the PVC in place, on the assumption,
stated in the `deleteDataVolume` comment, that Velero overwrites it.
Velero does not overwrite a PVC. Its CSI restore action returns early
for a claim that already exists, so no DataDownload is ever created, and
with `existingResourcePolicy: update` the merge patch it attempts
instead is rejected because a bound claim's spec is immutable. All that
reaches the claim is the labels-only fallback patch, which also drops
the `cdi.kubevirt.io/allowClaimAdoption` annotation the resource
modifier added.

The claim left behind then blocks the disk for good. Orphan propagation
has removed its owner reference to the DataVolume, and it lacks the
`cdi.kubevirt.io/storage.populatedFor` annotation that
kubevirt-velero-plugin writes on the backed-up copy, so when the
HelmRelease resumes, the CDI webhook refuses the DataVolume it creates
with `Destination PVC <ns>/<name> already exists`, on every reconcile.
Deleting the disk through the API leaves the PVC behind, and a new disk
of the same name fails the same way.

Observed on a v1.6.3 cluster, for a VMInstance restore with
`keepOriginalPVC: false` whose disk was attached to the VM. The Velero
log for the restore, trimmed:

```text
level=info msg="Retrieved per-restore resource modifiers from configmap cozy-velero/restore-modifiers-..."
level=warning msg="PVC already exists and ExistingVolumeDataPolicy is not in-place restore. Skip restore this PVC." Action=PVCCSIRestoreItemAction PVC=tenant-datalab/vm-disk-e2edisk
level=info msg="Applying resource modifier patch on tenant-datalab/vm-disk-e2edisk"
level=warning msg="patch attempt failed for /v1, Kind=PersistentVolumeClaim tenant-datalab/vm-disk-e2edisk: PersistentVolumeClaim \"vm-disk-e2edisk\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound claims"
level=info msg="backup/restore labels successfully updated for PersistentVolumeClaim tenant-datalab/vm-disk-e2edisk"
```

`velero restore describe --details` lists the PVC as `(failed)` and the
Restore is `Completed` with two warnings. Afterwards the claim had its
original creation timestamp and UID, no owner reference, no
`allowClaimAdoption`, no `populatedFor`, and the disk's HelmRelease was
`Ready=False` with the webhook error above. That cluster runs a Velero
build from `main`, whose wording is quoted above; the released v1.17.1
and v1.18.3 return early at the same point with `PVC already exists.
Skip restore this PVC.` (`pkg/restore/actions/csi/pvc_action.go`).

The fix deletes the PVC when `keepOriginalPVC` is false, only once its
DataVolume is confirmed gone so that CDI cannot recreate it, and holds
the Velero Restore until the claim has actually disappeared. A claim
still mounted stays `Terminating` behind `kubernetes.io/pvc-protection`,
and creating the Restore then would hit the same existing-claim path.
Velero then creates the PVC from the backup, the data mover fills it,
and the recreated DataVolume adopts it through `populatedFor`. A failed
deletion now fails the RestoreJob instead of letting it report
`Succeeded` over the old data. Before 2234824fe the DataVolume was
deleted with background propagation, so garbage collection removed the
claim, but nothing waited for it.

Deleting the claim is only safe when Velero can put the disk back, so
before any preparation the reconciler now gets the Velero Backup and
fails the RestoreJob unless it exists and is `Completed`. The Backup
artifact outlives the Velero Backup, which Velero garbage-collects at
its TTL, and without the check a restore of an expired backup would halt
the VM and delete its disks before failing. The check runs for every
Velero restore, whatever `keepOriginalPVC` says. A DataVolume that
cannot be deleted now fails the RestoreJob too, and the preparation
waits for every DataVolume to be gone before it goes further: a
surviving DataVolume makes CDI recreate the claim, which Velero then
skips. Reading the DataVolume back needs `get` on `datavolumes`, which
the backupstrategy-controller ClusterRole now grants next to `delete`.

`TestPrepareForRestore_KeepOriginalPVCFalse_SkipsRename` asserted the
old behaviour, that the original PVC survives, and is replaced by
`TestPrepareForRestore_KeepOriginalPVCFalse_DeletesPVC`.
`TestPrepareForRestore_KeepOriginalPVCFalse_WaitsForTerminatingPVC`
covers the wait. Both fail on `upstream/main` without the controller
change. `TestReconcileVeleroRestore_UnusableVeleroBackupTouchesNothing`
covers an expired Velero Backup and one being deleted,
`TestPrepareForRestore_KeepOriginalPVCFalse_KeepsPVCWhileDataVolumeRemains`
a failed and a pending DataVolume deletion, and
`TestPrepareForRestore_KeepOriginalPVCTrue_FailedRenameKeepsPVC` a
failed rename under `keepOriginalPVC: true`, which must leave the claim
in place. A helm-unittest suite, `tests/restore_rbac_test.yaml`, checks
the new verb. Each test fails with its fix reverted, with one exception:
`TestPrepareForRestore_KeepOriginalPVCTrue_FailedRenameKeepsPVC` still
passes with the Step 5 condition cut to `vmRes != nil`, because a failed
rename holds the restore back before Step 5 is reached. That guard is
pinned by `TestPrepareForRestore_KeepOriginalPVC_CacheLagsBehindWrites`,
which fails with the same cut. `helm unittest
packages/system/backupstrategy-controller` passes. `go test
./internal/backupcontroller/...` passes.

Related: #4601 reports the same Velero mechanism for a standalone
VMDisk, where `prepareForRestore` does nothing at all because
`collectUnderlyingResources` records no disk. #3427 makes a VMDisk
report its own disk; with `keepOriginalPVC: false` that path then runs
into this defect too, and this change covers it once both are in. This
PR does not touch the VMDisk path itself.

### Screenshots

N/A — no UI changes.

### Downstream repositories

Walked the trigger map against the diff, which is the restore
preparation in `internal/backupcontroller/velerostrategy_controller.go`
and its unit tests, plus one verb in the backupstrategy-controller
ClusterRole and its helm-unittest: no package added or renamed, no
`values.yaml` or `values.schema.json` change, no CRD,
ApplicationDefinition, label, annotation, namespace or metric change.
`keepOriginalPVC` lives in the opaque `RestoreJob.spec.options`, and a
search of the organisation finds it only in this repository, so neither
the website nor the Terraform provider documents or models it.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(backups): an in-place VMInstance RestoreJob with keepOriginalPVC=false now deletes each original disk PVC, and waits for it to be gone, before Velero restores it. Previously the PVC was left in place, Velero skipped restoring its data, the RestoreJob still reported Succeeded, and the disk's HelmRelease could never recreate its DataVolume ("Destination PVC already exists"). A Velero RestoreJob now also fails before touching the application when its Velero Backup is gone or not Completed, for example after the Backup expired at its TTL.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* When a restore is configured not to keep the original disk claim, the
claim is deleted before Velero restores it, allowing Velero to recreate
it.
* Restores wait while the original disk claim or its DataVolume is being
deleted. Deletion or status-check errors stop restore preparation.
* Restores configured to keep the original disk claim continue to
preserve it.
* Restores stop before changing resources if the referenced Velero
backup is missing or has not completed.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-10 22:26:10 +03:00
Aleksei Sviridkin c290b75923 docs: link the documentation of the current release (#4761)
## What this PR does

Points the documentation links in the README and GOVERNANCE.md at the
current release.

The unversioned `/docs/...` paths on cozystack.io are Hugo aliases owned
by the v0 docs. Each one is a meta-refresh to the v0 page, so these
links sent readers to the v0 documentation, a major version behind the
current release:

| Link | Went to | Now |
|---|---|---|
| README `/docs/guides/use-cases/public-cloud/` |
`/docs/v0/guides/use-cases/public-cloud/` |
`/docs/v1.6/guides/use-cases/public-cloud/` |
| README `/docs/guides/use-cases/private-cloud/` |
`/docs/v0/guides/use-cases/private-cloud/` |
`/docs/v1.6/guides/use-cases/private-cloud/` |
| README `/docs/guides/use-cases/kubernetes-distribution/` |
`/docs/v0/guides/use-cases/kubernetes-distribution/` |
`/docs/v1.6/guides/use-cases/kubernetes-distribution/` |
| README `/docs/getting-started/` | `/docs/v0/getting-started/` |
`/docs/v1.6/getting-started/` |
| README `/docs/operations/troubleshooting/` |
`/docs/v0/operations/troubleshooting/` |
`/docs/v1.6/operations/troubleshooting/` |
| GOVERNANCE.md `/docs/operations/configuration/telemetry/` |
`/docs/v0/operations/configuration/telemetry/` |
`/docs/v1.6/operations/configuration/telemetry/` |

In the v1 docs the kubernetes-distribution path holds Build Your Own
Platform (BYOP), installing only the packages you need onto an existing
cluster that already has networking and storage, rather than the v0
"Kubernetes distribution for bare metal" page. The README entry is
rewritten to match: "Using Cozystack to build your own platform", with
"You can install only the Cozystack components you need on an existing
Kubernetes cluster".

The full version path follows the other versioned docs links in the
repository, which all name `/docs/v1.6/`. The short `/docs/v1/...` path
is not an option: only the site's 404 page redirects it, in JavaScript.
All six new URLs return 200 without a redirect, and the same pages exist
under the website's `content/en/docs/next/`, so the next release keeps
them.

The same alias problem in the website's own `llms.txt` is fixed in
cozystack/website#737.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
docs: link the v1.6 documentation from the README and GOVERNANCE.md instead of the v0 pages the unversioned cozystack.io paths redirect to, and describe the kubernetes-distribution use case as Build Your Own Platform, which is what that page documents in v1.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Documentation**
* Updated README use-case and documentation links to point to the
Cozystack v1.6 documentation, including the Telemetry link in the
governance guide.
* Renamed the Kubernetes-distribution use case to “Build your own
platform” and clarified that selected Cozystack components can be
installed on an existing Kubernetes cluster.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-10 22:21:09 +03:00
Aleksei Sviridkin b1192bfcbc fix(backups): create no driver object for a job that already finished (#4673)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The BackupJob and RestoreJob reconcilers decide a job is still active
from the informer cache, and the cache can trail their own writes. A
driver's first pass stamps `StartedAt`, which requeues the job, and may
then fail the job in the same pass on a transient error, for instance a
503 from the aggregated `apps.cozystack.io` API while
`createVeleroBackup` reads the application. The requeued pass can start
before the informer has seen `Failed`: it retries the step that failed
and, once the error has cleared, creates the driver object anyway. Its
own status write then conflicts on the stale resourceVersion, the next
pass sees `Failed` and skips the job, and a Velero Backup is left
running that no `Backup` artifact will reference and nothing tracks.

Reproduced against a real apiserver (envtest 1.34) with the real manager
cache and the reconciler wired through `SetupWithManager`, nothing else
writing the BackupJobs. The only fault injected is one 503 on the first
GET of each application through `apps.cozystack.io`. Out of 40
BackupJobs, every one ends `Failed` with `failed to create Velero
Backup: the server is currently unable to handle the request`, and on
`main` (6addef4d3) 10, 8 and 5 of them still got a Velero Backup over
three runs. With this change: 0, 0 and 0. The harness is in the
collapsed block below; it needs `KUBEBUILDER_ASSETS` and `REPO_ROOT`,
and is not part of the PR.

A restore pass that starts on such a stale copy does worse, since the
step it retries can be destructive: suspending the VM's HelmRelease and
halting the VM before creating a Velero Restore, or patching the
Postgres application and purging its cluster. The reproduction above
covers the backup side only; on the restore side the unit test below
shows what a pass on a stale copy does.

Both reconcilers now re-read the job through the manager's uncached
reader when the cached copy says it is still active, and skip it when
the apiserver holds it as finished. The check sits ahead of the
dispatch, so it covers all eleven drivers at once. The cost is one GET
per pass of an active job; a job the cache already reports as finished
costs nothing extra.

`TestReconcile_DoesNotRetryAJobItAlreadyFailed` and
`TestReconcileRestore_DoesNotRetryAJobItAlreadyFailed` run the full
`Reconcile` on a cache holding the job active while the apiserver holds
it `Failed`. Each first checks that the same fixture does create the
Velero Backup, or suspend the HelmRelease and create the Velero Restore,
when the apiserver agrees with the cache. With the re-read reverted to
the cached client, both fail: a Velero Backup is created and the phase
is written back to `Running`, the HelmRelease is suspended and a Velero
Restore is created. `go test ./internal/backupcontroller/...` passes.

<details>
<summary>envtest reproduction harness</summary>

```go
//go:build repro

// Reproduction, not for commit: does the real BackupJob controller, on a real
// apiserver and the manager's real cache, create a Velero Backup for a
// BackupJob it has itself marked Failed?
//
// The only fault injected is one 503 on the first GET of each application
// through apps.cozystack.io, the aggregated cozystack-api, standing for a
// blip of that API. Nothing else writes the BackupJob.
//
// Run: KUBEBUILDER_ASSETS=<envtest bin dir> go test -tags repro -run TestReproFinishedJobRace -v ./internal/backupcontroller/
package backupcontroller

import (
	"context"
	"fmt"
	"io"
	"net/http"
	"os"
	"strings"
	"sync"
	"testing"
	"time"

	corev1 "k8s.io/api/core/v1"
	apiextensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
	"k8s.io/apimachinery/pkg/runtime"
	"k8s.io/apimachinery/pkg/runtime/schema"
	"k8s.io/client-go/kubernetes/scheme"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/envtest"
	"sigs.k8s.io/controller-runtime/pkg/log/zap"
	metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server"

	strategyv1alpha1 "github.com/cozystack/cozystack/api/backups/strategy/v1alpha1"
	backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1"
	velerov1 "github.com/vmware-tanzu/velero/pkg/apis/velero/v1"
)

type blipOnce struct {
	next http.RoundTripper
	mu   sync.Mutex
	seen map[string]bool
}

func (b *blipOnce) RoundTrip(req *http.Request) (*http.Response, error) {
	p := req.URL.Path
	if req.Method == http.MethodGet && strings.Contains(p, "/apis/apps.cozystack.io/v1alpha1/namespaces/") {
		b.mu.Lock()
		first := !b.seen[p]
		b.seen[p] = true
		b.mu.Unlock()
		if first {
			body := `{"kind":"Status","apiVersion":"v1","status":"Failure","message":"the server is currently unable to handle the request","reason":"ServiceUnavailable","code":503}`
			return &http.Response{
				StatusCode: http.StatusServiceUnavailable,
				Header:     http.Header{"Content-Type": []string{"application/json"}},
				Body:       io.NopCloser(strings.NewReader(body)),
				Request:    req,
			}, nil
		}
	}
	return b.next.RoundTrip(req)
}

func TestReproFinishedJobRace(t *testing.T) {
	ctrl.SetLogger(zap.New(zap.WriteTo(io.Discard)))
	root := os.Getenv("REPO_ROOT")
	dummyCRD := &apiextensionsv1.CustomResourceDefinition{
		ObjectMeta: metav1.ObjectMeta{Name: "dummies.apps.cozystack.io"},
		Spec: apiextensionsv1.CustomResourceDefinitionSpec{
			Group: "apps.cozystack.io",
			Names: apiextensionsv1.CustomResourceDefinitionNames{Kind: "Dummy", ListKind: "DummyList", Plural: "dummies", Singular: "dummy"},
			Scope: apiextensionsv1.NamespaceScoped,
			Versions: []apiextensionsv1.CustomResourceDefinitionVersion{{
				Name: "v1alpha1", Served: true, Storage: true,
				Schema: &apiextensionsv1.CustomResourceValidation{OpenAPIV3Schema: &apiextensionsv1.JSONSchemaProps{
					Type: "object", XPreserveUnknownFields: new(true),
				}},
			}},
		},
	}
	env := &envtest.Environment{
		CRDInstallOptions: envtest.CRDInstallOptions{
			Paths: []string{
				root + "/packages/system/backup-controller/definitions",
				root + "/packages/system/backupstrategy-controller/definitions",
				root + "/packages/system/velero/charts/velero/crds/backups.yaml",
			},
			CRDs:               []*apiextensionsv1.CustomResourceDefinition{dummyCRD},
			ErrorIfPathMissing: true,
		},
	}
	cfg, err := env.Start()
	if err != nil {
		t.Fatalf("envtest: %v", err)
	}
	defer env.Stop()

	s := runtime.NewScheme()
	_ = scheme.AddToScheme(s)
	_ = backupsv1alpha1.AddToScheme(s)
	_ = strategyv1alpha1.AddToScheme(s)
	_ = velerov1.AddToScheme(s)

	direct, err := client.New(cfg, client.Options{Scheme: s})
	if err != nil {
		t.Fatal(err)
	}
	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	mgrCfg := *cfg
	mgrCfg.WrapTransport = func(rt http.RoundTripper) http.RoundTripper {
		return &blipOnce{next: rt, seen: map[string]bool{}}
	}
	mgr, err := ctrl.NewManager(&mgrCfg, ctrl.Options{Scheme: s, Metrics: metricsserver.Options{BindAddress: "0"}})
	if err != nil {
		t.Fatal(err)
	}
	r := &BackupJobReconciler{Client: mgr.GetClient(), Scheme: s, Recorder: mgr.GetEventRecorderFor("repro")}
	if err := r.SetupWithManager(mgr); err != nil {
		t.Fatal(err)
	}
	go func() { _ = mgr.Start(ctx) }()

	must := func(o client.Object) {
		if err := direct.Create(ctx, o); err != nil {
			t.Fatalf("create %T %s: %v", o, o.GetName(), err)
		}
	}
	must(&corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: "tenant"}})
	must(&corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: veleroNamespace}})
	must(&strategyv1alpha1.Velero{ObjectMeta: metav1.ObjectMeta{Name: "velero"}})
	strategyGroup := strategyv1alpha1.GroupVersion.Group
	appGroup := "apps.cozystack.io"
	must(&backupsv1alpha1.BackupClass{
		ObjectMeta: metav1.ObjectMeta{Name: "bc"},
		Spec: backupsv1alpha1.BackupClassSpec{Strategies: []backupsv1alpha1.BackupClassStrategy{{
			StrategyRef: corev1.TypedLocalObjectReference{APIGroup: &strategyGroup, Kind: strategyv1alpha1.VeleroStrategyKind, Name: "velero"},
			Application: backupsv1alpha1.ApplicationSelector{APIGroup: &appGroup, Kind: "Dummy"},
		}}},
	})
	time.Sleep(2 * time.Second) // let the cache list the fixtures

	n := 40
	if v := os.Getenv("REPRO_N"); v != "" {
		fmt.Sscan(v, &n)
	}
	for i := range n {
		app := &unstructured.Unstructured{}
		app.SetGroupVersionKind(schema.GroupVersionKind{Group: appGroup, Version: "v1alpha1", Kind: "Dummy"})
		app.SetNamespace("tenant")
		app.SetName(fmt.Sprintf("app-%d", i))
		must(app)
		must(&backupsv1alpha1.BackupJob{
			ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: fmt.Sprintf("bj-%d", i)},
			Spec: backupsv1alpha1.BackupJobSpec{
				BackupClassName: "bc",
				ApplicationRef:  corev1.TypedLocalObjectReference{APIGroup: &appGroup, Kind: "Dummy", Name: app.GetName()},
			},
		})
		time.Sleep(200 * time.Millisecond)
	}
	time.Sleep(5 * time.Second)

	phases := map[backupsv1alpha1.BackupJobPhase]int{}
	orphans := 0
	for i := range n {
		j := &backupsv1alpha1.BackupJob{}
		if err := direct.Get(ctx, client.ObjectKey{Namespace: "tenant", Name: fmt.Sprintf("bj-%d", i)}, j); err != nil {
			t.Fatal(err)
		}
		phases[j.Status.Phase]++
		vbs := &velerov1.BackupList{}
		if err := direct.List(ctx, vbs, client.InNamespace(veleroNamespace), client.MatchingLabels{
			backupsv1alpha1.OwningJobNamespaceLabel: "tenant", backupsv1alpha1.OwningJobNameLabel: j.Name,
		}); err != nil {
			t.Fatal(err)
		}
		if j.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed && len(vbs.Items) > 0 {
			orphans++
			if orphans <= 3 {
				t.Logf("%s: phase=%s message=%q, yet Velero Backup %s/%s exists (created %s, job completedAt %s)",
					j.Name, j.Status.Phase, j.Status.Message, veleroNamespace, vbs.Items[0].Name,
					vbs.Items[0].CreationTimestamp.Format(time.RFC3339Nano), j.Status.CompletedAt.Format(time.RFC3339Nano))
			}
		}
	}
	t.Logf("jobs=%d phases=%v failed-with-velero-backup=%d", n, phases, orphans)
}
```

</details>
### Screenshots

N/A — no UI changes.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

Walked the trigger map against the diff, which is the two job
reconcilers in `internal/backupcontroller/` and a test file: no package,
`values.yaml`, `values.schema.json`, CRD, ApplicationDefinition, label,
annotation, namespace or `hack/` change.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(backups): a BackupJob or RestoreJob that the controller has just marked Failed no longer gets a driver object created for it by the next reconcile, which could start on a cache that had not yet seen the Failed status and retry the step that failed — leaving, for instance, an untracked Velero Backup behind a Failed BackupJob, or a restore that halts the VM after its RestoreJob failed.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Backup and restore jobs that have already succeeded or failed are no
longer retried based on outdated status, preventing duplicate operations
and unnecessary changes to related resources.
* Reconciliation stops when a job has been removed and reports errors if
its current status cannot be checked.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-10 22:15:54 +03:00
Aleksei Sviridkin a2e10312de fix(platform): stop treating a failed ConfigMap read as a fresh cluster (#4792)
## What this PR does

A single failed API read at the start of the platform migration Job
could skip every pending migration and still end green.
`run-migrations.sh` checked whether the `cozystack-version` ConfigMap
exists with a silenced `kubectl get` and looked only at the exit code. A
timeout, a dropped connection or an apiserver restart during the upgrade
all counted as "no ConfigMap", so the script stamped the target version
and exited 0. The hook Job is rendered only when that ConfigMap already
exists, so in practice an error was the only way into this branch.

Now the script creates the ConfigMap only when it is really missing. The
check uses `--ignore-not-found --output name`: kubectl drops NotFound
and nothing else, so empty output means a genuine NotFound. Any other
error keeps a non-zero exit, `set -e` aborts the script, and the Job
retries the pod. The rest of the script already works this way: the read
of the stamped version aborts on error, and the stamp runs under
`pipefail`.

The tests cover both branches. A new case in `hack/run-migrations.bats`
makes the existence check fail with a connection error and expects a
non-zero exit, no migration run and the stamp left where it was. It
fails against the old check. The fake kubectl returns empty output for a
missing ConfigMap only when `--ignore-not-found` is passed, as real
kubectl does, so dropping the flag breaks the existing "no ConfigMap"
case.

No migration is added, so `targetVersion` stays. The script ships in the
migrations image, and the release build restamps that image's digest as
usual.

Several older migrations have the same problem in a smaller form. They
check for an object with `kubectl get ... >/dev/null 2>&1` and skip
their own work when the check fails, for example migrations 31, 32, 36,
48, 50 and 51. A transient error there skips one migration's work, not
all of them. Editing migrations that already shipped has its own risk,
so I'd track that separately.

Closes #4579

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(platform): a transient API error while the platform migration Job checks for the cozystack-version ConfigMap now fails the pod so the Job retries, instead of stamping the target version and silently skipping every pending migration.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Migration checks now distinguish a missing version record from an
error while checking for it. When the check fails, the pod stops instead
of treating the record as missing and proceeding with migrations.
* Added coverage for TLS-handshake timeouts during the check, confirming
that migrations do not run and the stamped version remains unchanged.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-10 20:37:53 +03:00
Marian Koreniuk c61fcf0ee4 docs(governance): link the Telemetry page of the current release
The unversioned /docs/operations/configuration/telemetry/ path is an alias owned by the v0 docs, so GOVERNANCE.md sent readers to the v0 Telemetry page. It now names the v1.6 page, like the Licenses link a few lines below.

Assisted-by: LLM
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
2026-10-10 19:08:16 +02:00
Marian Koreniuk b39f623338 docs(readme): describe the use case the kubernetes-distribution page documents now
In the v1 docs the kubernetes-distribution use-case path holds Build Your Own Platform: installing only the packages you need onto an existing cluster that already has networking and storage. The README entry still described the v0 page, Cozystack as a Kubernetes distribution for bare metal, so its label and description no longer matched the page it links to. Both now describe BYOP.

Assisted-by: LLM
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
2026-10-10 19:08:16 +02:00
Marian Koreniuk 59af194b73 docs(readme): link the documentation of the current release
The unversioned /docs/... paths on cozystack.io are aliases owned by the v0 docs, so the use-case, getting-started and troubleshooting links sent readers to the v0 documentation, a major version behind the current release. They now name the v1.6 pages, in line with the other versioned links in the repository.

Assisted-by: LLM
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
2026-10-10 19:07:50 +02:00
Aleksei Sviridkin be2387323b feat(gateway)!: reserve a passthrough listener's name by the spec alone (#4638)
## What this PR does

This keeps the TLS-passthrough services on port 443 of the tenant
Gateway and makes the passthrough rules hold on Cilium 1.20.2. It is
stacked on #3343 and has to land with it (stack #4717).

On 1.20, upstream Cilium marks an HTTPS-terminate listener and a
TLS-passthrough listener on one port `Conflicted` when their hostnames
overlap, and drops both from Envoy. That took `*.<apex>` and the `api`,
`vm-exportproxy` and `cdi-uploadproxy` passthrough listeners down
together on every tenant Gateway in `dns01` and `existingSecret` mode
(#3912). #3343 now ships a patched operator that conflicts the pair only
when the hostnames are equal, so the layout stays as it is on main: one
Gateway, one address, everything on 443. What remains is the
equal-hostname case and the cleanup that 1.20 allows.

Under `http01`, a name a passthrough listener answers is now reserved by
the spec alone. Before, the controller dropped the terminate listener
for such a name only once a TLSRoute was attached to the passthrough
listener and its backend Service resolved, which copied how Cilium
builds filter chains. Without a working TLSRoute, an HTTPRoute on
`api.<apex>` kept its terminate listener next to `tls-api`, and on 1.20
that equal-hostname pair takes both listeners down. Now such an
HTTPRoute gets no listener and no certificate, TLSRoute or not. If all
its hostnames are reserved it reads `Accepted=False` with
`NoMatchingListenerHostname`. If it also claims a free hostname it stays
`Accepted` and is served there. The backend lookup, its Service and
ReferenceGrant watches and their RBAC rules are gone.

Other changes:

- `tlsPassthroughListeners` is accepted under `dns01` and
`existingSecret`. The old refusal relied on 1.19.5 serving all ports
from one Envoy listener; 1.20 splits ports whose SNI overlaps.
- A TenantGateway reads `Ready=False` with reason
`GatewayClassUnsupported` when its GatewayClass lists features in
`status.supportedFeatures` but not `TLSRoute`, while it renders
passthrough listeners. A class that lists nothing is not judged. Cilium
1.20.2 lists `TLSRoute`.
- Comments, the README and one error message no longer describe Cilium
1.19.5 as the shipped release.
- The gateway chart writes `tlsPassthroughServices` on the publishing
tenant only, behind the same `gateway.enabled` switch and
`publishing.exposedServices` filter as the platform TLSRoutes. Before,
every tenant Gateway got the three passthrough listeners with nothing
routed to them, and the reservation above would have taken those names
from a child tenant's own HTTPRoutes.

The uniform `allowedRoutes.kinds` on port 443 from #3087 stays. The
Cilium bug behind it is fixed since 1.19.6, but several tests pin it, so
removing it is a separate cleanup.

### Upgrade notes

Only the publishing tenant keeps the `api`, `vm-exportproxy` and
`cdi-uploadproxy` passthrough listeners. That is `tenant-root` unless
`publishing.ingressName` says otherwise, and only for names in
`publishing.exposedServices` while `gateway.enabled` is on. Other
tenants lose these listeners on upgrade. Nothing shipped was attached to
them, so a child tenant's own HTTPRoute on `api.<child-apex>` keeps its
listener and certificate.

Two narrow cases do change. On the publishing tenant under `http01`, an
HTTPRoute on an exposed passthrough name loses its listener and
certificate when the platform TLSRoute for that name forwards nowhere
(the owning package is not installed, or its backends do not resolve). A
TLSRoute attached by hand to a child tenant's `tls-*` listener loses its
parent. To keep that listener, merge-patch `tlsPassthroughServices` into
the tenant's `TenantGateway` after the upgrade. From then on the chart
does not render the key there, and later upgrades leave the patch alone.
`dns01` and `existingSecret` do not change: the wildcard listener still
covers such a name, and the passthrough listener answers it by its exact
name.

Tested locally: Go tests for the controller, `go vet`, helm-unittest for
`extra/gateway` and `cozystack-controller`, `make generate` and `make
manifests` with no diff. The gateway e2e scenario checks that a child
tenant renders no passthrough listener. It then declares `tls-api` and a
passthrough listener on a native port on that tenant, checks that the
tenant stays Ready with no route behind either, and checks the
reservation against `tls-api`. Unit tests catch a regression in either
change. In e2e, dropping the reservation turns the gateway suite red,
and dropping the chart scoping does not, because CI never sets
`gateway.enabled` (#4846). The controller was not run on a live cluster;
the operator patch it depends on was, see #3343.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/issues/725
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(gateway)!: the tenant Gateway renders the api, vm-exportproxy and cdi-uploadproxy passthrough listeners only on the publishing tenant, for the names publishing.exposedServices lists while gateway.enabled is on. Under certMode http01 a name a TLS-passthrough listener answers no longer gets an HTTPS listener on 443, with or without a TLSRoute behind it. On the publishing tenant, an HTTPRoute on such a name whose TLSRoute forwards nowhere loses its listener and certificate. A TLSRoute attached by hand to a tls-* listener of another tenant loses its parent. An HTTPRoute with some reserved and some free hostnames used to be rejected whole; it is now Accepted and its free hostnames get a listener and a certificate. tlsPassthroughListeners is accepted under dns01 and existingSecret. A TenantGateway whose GatewayClass does not list TLSRoute in its supported features reads Ready=False while it renders passthrough listeners.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* TLS passthrough listeners are supported with HTTP-01, DNS-01, and
existing-secret certificate modes. Edge-terminated Gateways remain
unsupported.
* Declaring a passthrough hostname reserves it from HTTPS termination
and certificate creation, even when no route is attached.
* Passthrough listeners are rendered only for exposed services on the
enabled publishing Gateway.
* **Bug Fixes**
* Gateways that declare passthrough listeners but explicitly lack
TLSRoute support are now reported as not ready.
* Routes targeting a hostname served by a passthrough listener on a
different Gateway are correctly withdrawn.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-10 16:59:17 +03:00
Aleksei Sviridkin d9689bd6e7 fix(gateway): stop describing Cilium 1.19.5 as the shipped release
With Cilium at 1.20.2 the tenant gateway comments, README and one
status message still explained behaviour as that of "the pin" at
1.19.5. Several of those claims no longer hold: a Gateway with two
routed passthrough ports now gets an Envoy listener per port, Cilium
judges a route's kinds against the listener the route names, and the
shipped operator accepts an HTTPS and a passthrough listener on one
port when the passthrough hostname is strictly narrower.

Behaviour that is unchanged is re-anchored on the 1.20.2 source,
behaviour that is gone is stated as history, and the refusal of one
hostname on two passthrough ports no longer blames a Cilium limitation
the shipped release does not have.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-10 06:32:42 +03:00
Aleksei Sviridkin ad3f15d4f3 feat(gateway)!: reserve a passthrough listener's name by the spec alone
The TenantGateway controller withheld the HTTPS-terminate listener for a
name a passthrough listener answers only once a TLSRoute was attached
there and resolved a backend, mirroring how Cilium builds its filter
chains. That kept a served endpoint up on Cilium 1.19.5, but it tied
every route's status to one implementation's translator and needed
Service and ReferenceGrant watches to land the deferred decision.

The shipped operator accepts a wildcard terminate listener beside a
narrower passthrough listener on 443 and keeps conflicting every other
overlap. A terminate listener for the passthrough name itself is not
narrower, so on 443 the rule now protects the dataplane outright; on a
native port Gateway API would serve the pair, and the rule holds as a
policy that keeps a platform name with its passthrough listener. Reading
it off the spec makes the outcome the same on every implementation.

Under http01 an HTTPRoute claiming such a name gets no listener and no
certificate whether or not a TLSRoute is attached. A route whose every
hostname is reserved reads Accepted=False with
NoMatchingListenerHostname; one that also claims a free hostname stays
Accepted and is served there, as Gateway API accepts a route whose
hostnames intersect at least one listener. dns01 and existingSecret are
unchanged: their wildcard listener still covers such a name, and the
passthrough listener answers it by its exact server name.

The chart writes the passthrough services on the publishing tenant
alone, so the names reserved by default are the ones the platform
publishes there through its own TLSRoutes. The shape an upgrade
changes is an HTTPRoute on such a name whose platform TLSRoute forwards
nowhere, because it never attached (the package owning the service is
not installed) or because none of its backends resolve: that route kept
a terminate listener and now gets none.

The backend resolution, its watches and the RBAC rules that served them
go away.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-10 06:32:42 +03:00
Aleksei Sviridkin 6144805bea fix(mariadb): mount the chart's my.cnf under a .cnf name (#4843)
## What this PR does

The MariaDB chart's my.cnf has not been read by the server since the
mariadb-operator bump to v0.28.1. Up to v0.0.27 the operator mounted the
referenced ConfigMap key as `/etc/mysql/conf.d/my.cnf` whatever the key
was called. Since v0.0.28 the file takes the key's own name, so the
chart's key `config` became `/etc/mysql/conf.d/config`. The server's
`!includedir` reads only `*.cnf` files, so it skipped it, and every
instance has been running on stock MariaDB defaults. On the default
`t1.nano` preset that means a 128M buffer pool inside a 128Mi memory
limit.

This renames the key to `my.cnf` in the ConfigMap and in the MariaDB CR.
The file content is unchanged. I started all four shipped versions
(10.6.24, 10.11.15, 11.4.9, 11.8.5) with the rendered file under a 128m
memory limit, and each one came up with the chart's values. The
MySQL-only options in the file log a warning and are otherwise ignored.
They and the deprecated `innodb_flush_method` are tracked in #4842.

Tests: a helm-unittest suite pins the key on both sides, and the mariadb
chainsaw suite gets a step that compares `@@innodb_buffer_pool_size` on
the running server with the value in the rendered my.cnf.

This changes behaviour on every existing instance, see the release note.
Each instance rolls its pods once because the projected volume in the
pod template changes: replicas first, then the primary with a
switchover. After that the server runs with every setting in the chart's
file. Instances created before the operator bump (May 2024) ran with
these settings before. Newer ones never did.

Closes #4836

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(mariadb): the chart's my.cnf is applied again. Since mariadb-operator v0.28.1 it was mounted under a name the server ignores, so MariaDB instances ran on stock defaults. Every existing instance rolls its pods once on upgrade (replicas first, then the primary with a switchover) and then runs with the chart's settings. That changes behaviour: buffer pool, redo log, connection limits and caches are sized from the instance's resources; sql_mode becomes NO_ENGINE_SUBSTITUTION, so strict mode is off and invalid values are truncated with a warning instead of rejected; innodb_flush_log_at_trx_commit=2 can lose up to about a second of committed transactions if the node crashes; innodb_fast_shutdown=0 makes every pod shutdown slower.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* MariaDB configuration is now stored and read using the `my.cnf` key. A
verification check confirms the running server applies the configured
buffer-pool size.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-10 04:38:28 +03:00
Aleksei Sviridkin 4346a30f4b ci(e2e): run nightly and release-tag e2e on the container lane (#4828)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Nightly and the release-tag e2e (`e2e-tag.yaml`, the rc gate) now run on
the Talos-container substrate the PR lanes moved to in #4020. This is
the interim step from #4642, not the whole issue, so it does not close
it.

On QEMU the nightly full suite has been cancelled at its 215-minute
timeout on every night since 2026-10-01 except 2026-10-07. From
2026-10-02 on, kubernetes-latest and kubernetes-previous failed their
tenant node-join deadline almost every night, each failure took about 35
minutes, and the suites after them alphabetically never ran. In the 18
PR runs since 2026-10-03 that I checked and that ran them, they passed
every time, at 7 to 10 minutes each. A nightly dispatched from this
branch (run 37934155231) passed all 62 tests in 193 minutes. The
kubernetes suites took 9 to 11 minutes there.

Changes:

- `nightly.yaml` and `e2e-tag.yaml` get the same host module step,
`prepare-env-container`, endpoint and storage-class env and two-part
teardown as `pull-requests.yaml`. Nightly's e2e no longer waits for the
nocloud disk, which it does not boot. The disk is still built and
published.
- `e2e-tag.yaml` still requires the release to carry its disk but no
longer downloads it. A dispatch against a tag cut before the container
lane (v1.6.3 and older, v1.7.0-alpha.1) fails early and says to dispatch
with `--ref <tag>`. When `tags.yaml` calls it, it runs the copy from the
tag's own commit, so the guard does not fire there.
- The comment above `timeout-minutes: 215` in all four e2e workflows
claimed the budget covers two failing tenant Kubernetes suites. It does
not, and the comment now says so. The value is unchanged.
- `platform-packages_test.bats` now checks the container-lane env on all
four workflows, and `collect-images_test.bats` covers the new guard.

What loses coverage: DRBD and the `replicated` StorageClass (Immediate
binding, tenant StorageClass propagation, RWX), the Cozystack Talos node
image with its extensions, and the stock 600M CDI importer limit. None
of them can run on containers, so no CI lane tests them until #4642 gets
a lane with a kernel per node. The docs and comments that pointed at
nightly and rc as that coverage now say this.

Release lines keep their own `e2e-tag.yaml`, so rc tags on release-1.6
stay on QEMU until this is backported. That is why `kind/backport` is on
this PR. Leftover QEMU-only text and diagnostics are collected in #4825.

A dispatch of `e2e-tag.yaml` from this branch against `v1.7.0-alpha.3`
is running as run 37958852914. I'll post the result here.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
ci(e2e): nightly and release-tag e2e run on Talos containers instead of QEMU guests. DRBD, the replicated StorageClass and the Cozystack Talos node image are no longer covered by any e2e lane.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **CI and Testing**
* Release-tag and nightly end-to-end tests now run against
container-based environments rather than booting disk images.
* Release-tag checks still verify that the published release includes
its disk image.
* CI no longer covers QEMU-based DRBD scenarios or certain storage and
Talos extension behaviors; these coverage gaps are documented.
* **Documentation**
* Updated E2E and release guidance to reflect the container-based test
coverage and its limitations.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-10 03:56:39 +03:00
Aleksei Sviridkin 8ab6d4948f test(mariadb): assert the server runs with the chart's my.cnf
The chart's unit tests can pin the ConfigMap key and the CR reference,
but not what mariadb-operator does with them or which files the server
then reads from /etc/mysql/conf.d. Compare innodb_buffer_pool_size as
the live server reports it against the value in the rendered my.cnf,
which differs from the server default on this fixture's preset.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-10 02:47:37 +03:00
Aleksei Sviridkin d76bd1f263 fix(mariadb): mount the chart's my.cnf under a .cnf name
mariadb-operator up to v0.0.27 mounted the referenced ConfigMap key as
/etc/mysql/conf.d/my.cnf whatever the key was called. Since v0.0.28 the
file takes the key's own name, so the chart's key "config" became
/etc/mysql/conf.d/config, and the server's !includedir reads only *.cnf
files from that directory. Every setting in the chart's my.cnf has been
ignored since that operator bump: instances run on stock defaults,
including a 128M InnoDB buffer pool inside the 128Mi default limit.

Renaming the key restores the file as intended. Existing instances roll
their pods once, since the projected volume in the pod template changes,
and come back with the chart's settings in effect, including sql-mode
and innodb_flush_log_at_trx_commit, not only the resource sizing.

Fixes: a9c42c8ef0 ("Update mariadb-operator v0.28.1 (#124)")
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-10 02:35:48 +03:00
Yasar Rahman 83aaa65ed1 fix(velero): bump velero-plugin-for-aws to v1.14.4 so backups work on R2
velero-plugin-for-aws v1.14.1 sets x-amz-tagging on every PutObject,
empty when the BackupStorageLocation configures no tags. Cloudflare R2
answers that with 501 NotImplemented ("Header 'x-amz-tagging' with value
'' not implemented"), so Velero cannot write backup metadata and every
backup to an R2 location fails, the cozy-default location included when
the platform bucket is on R2.

The plugin sends the header only when tags are configured since v1.14.3
(velero-plugin-for-aws#309, a backport of #299). v1.14.4 is the current
patch release of the 1.14 line, which upstream lists as the one
compatible with Velero 1.18, and adds only CVE dependency bumps on top
of v1.14.3.

Signed-off-by: Yasar Rahman <yasar.rehman@gmail.com>
Assisted-by: LLM
2026-10-10 01:18:13 +05:00
Yasar Rahman 8bfd4e5f5e fix(redis): re-send the Sentinel auth-pass when the operator resets a Sentinel
A Sentinel that restarts on the same pod IP, as after a node reboot,
comes back with the operator's initial sentinel.conf, which monitors a
127.0.0.1 placeholder and carries no auth-pass. Its peers still know
that address and keep publishing hello messages to it, and once a
failover has raised the config epoch the restarted Sentinel adopts the
current master address from them before the operator looks at it. The
operator then finds the right master address and skips
NewSentinelMonitor, the only call that sends SENTINEL SET auth-pass. The
master answers that Sentinel with NOAUTH, it reports the master as
s_down, and the operator's resulting SENTINEL RESET on every check keeps
the empty auth-pass, so the instance runs with one Sentinel fewer until
someone deletes the pod.

RestoreSentinel now sends the RedisFailover's password with SENTINEL SET
auth-pass after the reset. It runs only when the Sentinel already looks
unhealthy, and the reset has just dropped its links, so a Sentinel that
had the password loses nothing. Without auth there is nothing to send.

The patch file sorts after tls.diff because it is written on top of it;
the Dockerfile applies patches/*.diff in glob order.

Signed-off-by: Yasar Rahman <yasar.rehman@gmail.com>
Assisted-by: LLM
2026-10-10 01:18:13 +05:00
Aleksei Sviridkin f61e13fdef fix(cilium): size the agent memory limit for its BPF maps (#4803)
## What this PR does

This raises the cilium-agent memory limit from 4Gi to 24Gi. The 4Gi
limit from #3495 has no room for the agent's BPF maps. On nodes with
roughly a terabyte of RAM or more, the agent is OOM-killed on every
start after a reboot, and the node loses pod networking.

The maps grow with node RAM. Cilium sizes its CT, NAT, neighbor and
socket rev-NAT maps as a share of total RAM
(`bpf-map-dynamic-size-ratio`, 0.0025 by default), and caps each map at
16Mi entries. These are LRU hash maps, so the kernel preallocates them
in full. Since Linux 5.11 that memory is charged to the memcg of the
process that creates the maps, which is the agent container.

This is also why the upgrade itself works. A map reused from bpffs stays
charged to the cgroup that created it. The failure shows up on the first
reboot after the upgrade, when the maps are recreated inside the limited
cgroup.

Policy maps don't belong on that list. The issue and the upstream
chart's configmap comment both say the ratio sizes them, but at this
version they are LPM tries sized by `bpf.policyMapMax`, and they
allocate per entry.

One DaemonSet serves every node, so the limit can't follow node RAM. The
map sizes are capped, though, so a fixed limit can cover any node. With
every map at 16Mi entries for both IPv4 and IPv6, the maps take just
under 20Gi of kernel memory. Add the 4Gi the agent already had and you
get 24Gi. The request stays at 256Mi.

The limit doesn't reserve memory, and the pod cgroup still gets
`memory.max`, which is what #3495 was after. What it costs is the leak
backstop. The agent is now stopped at 24Gi minus its own map memory, so
on most nodes a leak in its heap is no longer caught before the node
feels it. If someone prefers a tighter backstop over IPv6 headroom, they
can lower `cilium.resources.limits.memory` in the networking component
values. About 13Gi still covers the IPv4-only map ceiling.

I didn't pin the map sizes (`bpf.ctTcpMax`, `bpf.natMax` and so on). No
single value works for every node. A value small enough for a 16Gi node
cuts the conntrack capacity of a 1.5TiB node many times over. A pinned
size also differs from what almost every node above ~14Gi of RAM runs
today, so the agent would recreate those maps on upgrade and drop their
conntrack and NAT state. A lower ratio has the same problem and doesn't
lower the ceiling. The new test keeps the ratio at its default and the
sizes unpinned, so changing either has to be deliberate.

The figures are computed, not measured. I read the sizing code at cilium
1.19.5 (`pkg/option/config.go`), and 1.20.2, which #3343 moves to, has
the same code. Memory comes from that formula plus the kernel hash map
layout: a 48-byte element header, the key and value rounded up to 8
bytes, and a 16-byte bucket per power of two of `max_entries`. By that
math the IPv4 maps alone pass 4Gi at roughly a terabyte of RAM. Bucket
rounding moves the exact point, so I wouldn't read more precision into
it. For the 1.5TiB node in the report, the issue's 3.9GiB is the agent's
own budget (0.25% of RAM), and the kernel's actual charge comes out
higher.

Follow-ups, not in this PR:

- Operators who want smaller maps on large nodes can size them per node
with a `CiliumNodeConfig` selected by node label. This needs a
documentation note.
- The reporter's `bpftool` measurement of the real map memory is still
pending. That is the measured check for the numbers above.

The cilium helm-unittest suite passes on this branch, and also with this
change applied over the 1.20.2 chart from #3343. Each new assertion
fails when its value changes: the limit, the CT, NAT and neighbor sizes,
the socket rev-NAT size set through `extraConfig`, and the ratio.

Closes #4800

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff changes only `packages/system/cilium/values.yaml` and its test.
None of the trigger-map entries apply: no app, schema, platform value,
variant, release asset or tooling changes, and no website page documents
the agent limit.

### Release note

```release-note
fix(cilium): raise the cilium-agent memory limit from 4Gi to 24Gi so it covers the agent's BPF maps, which cilium sizes from node RAM. On nodes with around a terabyte of RAM or more, the 4Gi limit got the agent OOM-killed on every start after a reboot, leaving the node without pod networking.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
  * Increased the Cilium agent memory limit to 24 GiB.
* Updated validation for BPF map sizing: checks the dynamic size ratio
and confirms that specific map-size settings are absent.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-09 20:50:44 +03:00
Aleksei Sviridkin 52f09d1aed fix(gateway)!: render passthrough services only where platform routes attach
The chart wrote the tlsPassthroughServices default onto every non-edge
tenant's TenantGateway, so every tenant Gateway carried tls-api,
tls-vm-exportproxy and tls-cdi-uploadproxy. The TLSRoutes that serve
those names are platform routes: the packages owning the services render
one per name in publishing.exposedServices while gateway.enabled is on,
parented to the publishing tenant's Gateway alone. Everywhere else no
shipped route attached to the listeners, and a passthrough listener
holds its name whether or not anything is routed through it.

The chart now writes the field only on the publishing tenant, with the
same gateway.enabled switch and the same exposedServices filter those
routes use, so a listener exists exactly where a platform route can
attach. A TLSRoute attached by hand to one of those listeners on another
tenant loses its parent.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-09 18:46:47 +03:00
Aleksei Sviridkin bdca165f4c feat(gateway): report a GatewayClass that does not serve TLSRoute
A GatewayClass lists the features it serves in status.supportedFeatures,
and TLS passthrough is TLSRoute. On a class that leaves it out the
passthrough listeners are accepted and serve nothing, so the
TenantGateway now reads Ready=False/GatewayClassUnsupported naming the
class instead of Ready. A class that lists no features at all is not
judged: implementations predating the field leave it empty.

The class writes that list after it is accepted and may change it when
its controller is upgraded, so a GatewayClass event requeues the
TenantGateways on it.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-09 17:53:25 +03:00
Aleksei Sviridkin d624049a3a feat(gateway): allow tlsPassthroughListeners under dns01 and existingSecret
The refusal existed because Cilium 1.19.5 served every Gateway port
from one Envoy listener matched by SNI alone, so the "*.<apex>"
terminate listener of the wildcard modes and a passthrough listener
under the apex could reach each other's backend. The shipped Cilium
1.20.2 gives each port its own Envoy listener when SNI overlaps across
ports, and Gateway API treats listeners on different ports as
compatible, so the pair is served as declared. edge still refuses the
field: it renders no TLS listener for an entry to bind to.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-09 17:53:25 +03:00
Aleksei Sviridkin e1319b5a6c docs(cilium): say the SNAT backport is dropped by hand
The patch header cited a commit of cilium/cilium#49232, an open PR
whose branch is force-pushed, so the commit is reachable from no ref.
Cite the PR alone.

It also read as if the build would flag the patch once upstream
carries the fix. It does not: a fix that lands in another place leaves
this patch applying cleanly as a second early return, so the drop is a
manual step.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-09 17:53:16 +03:00
Aleksei Sviridkin e816522366 chore(renovate): leave the cilium-operator image pins to make update
The operator rebuild compiles one Cilium release with the Go toolchain
upstream's builder image pins at that tag, and make update re-pins the
builder line from there, along with the operator-generic base, its
digest and the source archive checksum. A Renovate bump of that line, by
tag or by digest, would build the release with a toolchain upstream
never shipped it with, and the next make update would revert it without
notice.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-09 17:53:15 +03:00
Aleksei Sviridkin 1bfa64f83a fix(cilium): keep per-type xDS while the 1.20 upgrade rolls
The 1.20 chart renders envoy-xds-mode "ads" and an adsConfig bootstrap
for cilium-envoy unless xdsMode or upgradeCompatibility says otherwise.
A 1.19 agent never registers the aggregated discovery service, and a
1.20 agent in ADS mode registers nothing else. The agent and envoy
DaemonSets roll independently, so on every node where their versions
differ envoy gets no configuration until both pods are replaced, and
Gateway traffic stops on all nodes at once during the upgrade.

Pin xdsMode to split, the per-type xDS both versions serve and the
agent's own default when the key is absent. Moving to ADS is a separate
step once every node runs 1.20.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-09 17:53:14 +03:00
Aleksei Sviridkin 5b92ca3fc8 fix(cilium): serve HTTPS and passthrough on one port from one listener
When a Gateway needs per-port Envoy listeners, the operator emits one
listener per HTTPS port and, with two or more passthrough ports, one per
passthrough port, all named listener-<port>. A port carrying both gets
two listeners with one name, and the agent skips the whole
CiliumEnvoyConfig, so every listener of the Gateway stops serving while
its status stays green. With one passthrough port the port is instead
mapped to two listeners in the CEC services. A tenant Gateway with
HTTPS and passthrough on 443 plus a passthrough listener on a native
port reaches the first case.

The patch puts the passthrough filter chains of such a port on its
HTTPS listener and leaves the port out of the passthrough listeners and
service mappings.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-09 17:53:14 +03:00
Aleksei Sviridkin cebf9f1e1e feat(cilium): accept wildcard HTTPS beside specific TLS passthrough
Cilium 1.19.6 and later mark an HTTPS-terminate listener and a TLS
passthrough listener on the same port Conflicted whenever their
hostnames intersect, and 1.20 drops both from Envoy. That rejects
*.<apex> HTTPS next to api.<apex> passthrough on 443, which a tenant
Gateway with one address needs. Envoy selects the most specific
server_names match before the transport protocol, so the pair maps every
SNI to exactly one filter chain.

The operator is rebuilt from the upstream tag with a patch that accepts
the pair when the passthrough hostname is strictly narrower than the
HTTPS one, and keeps every other overlap conflicted. The asymmetry is
needed because passthrough chains take their server_names from the
attached TLSRoutes: under a passthrough wildcard a route may claim the
HTTPS listener's exact name, and two chains with one match make Envoy
refuse the whole listener.

The binary replaces the one in the upstream operator-generic image, and
the builder runs the patched packages' tests, so a tag bump that breaks
the patch fails the build. make update re-pins both upstream images by
the digests the vendored chart carries, the builder by upstream's own Go
toolchain, and the source archive by its checksum. The image ref sits
under operator.image.override: committed as the upstream operator the
rebuild starts from, so a tree rendered without a build still pulls an
image that exists, and stamped by make image.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-09 17:53:14 +03:00
Aleksei Sviridkin c1fe1e9cb2 ci(e2e): say what the 215-minute e2e budget actually covers
The comment on every lane claimed the budget fits two tenant
Kubernetes suites failing at their node-join deadline. A green full
suite on the container lane already spends most of it, and a suite that
runs into the deadline costs about 25 minutes more than one that
passes, so the claim does not hold. State the real bound so a cancelled
run is read as a red suite, not as a budget to raise.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-09 16:57:44 +03:00
Aleksei Sviridkin da28ea7ee8 ci(e2e): run nightly and release-tag e2e on the container lane
Nightly and the rc gate still booted the management nodes as QEMU
guests, the substrate the merge-gating lanes left behind. On it the
tenant Kubernetes suites have missed their node-join deadline on most
nights since 2026-10-02, each failure spends the whole deadline, and
the job reaches its timeout before the alphabetically later suites
run. Over the same days those suites passed on every PR run that
selected them, where the nodes are Talos containers and a tenant
worker sits one virtualization level shallower.

Both lanes now use the PR lanes' container substrate. Nightly's e2e no
longer waits for the nocloud disk, which it does not boot; the disk is
still built and published. A manual e2e-tag dispatch against a tag
that predates the container lane fails early and names the ref to
dispatch from instead.

DRBD, the replicated StorageClass and the Cozystack Talos node image
lose their last e2e coverage with this. The docs and comments that
pointed at nightly and the rc gate as that coverage now say so.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-09 16:57:43 +03:00
Aleksei Sviridkin aa804aaa83 fix(dashboard): allow editing free-form objects in application forms (#4682)
## What this PR does

Free-form object fields, such as NATS `config.merge` and
`config.resolver`, get a JSON editor. The form keeps nested objects,
arrays and primitive values. Input that is not a JSON object stays
visible, and schema validation blocks the create or update request until
it is fixed. A JSON object that fails the schema marks the field invalid
and points it at the error list. Typed maps keep their key and value
editor, and read-only and immutable fields stay protected.

Fixes #4676.

### Validation

- All 463 console tests in 63 files pass, and so does the TypeScript
check.
- Tests cover typing into the editor, nested NATS objects next to the
typed users editor, input that is not a JSON object, schema errors on a
valid object, existing values, immutable fields and free-form objects
inside arrays. On the order page they cover a create after a form and
YAML round trip, saving an edited object, and blocking input that is not
JSON in create and edit mode.
- The two schema-error tests fail on the component without the change.

### Screenshots

![Nested
JSON](https://github.com/user-attachments/assets/04c95513-4e7d-465a-9c7d-e4384e13a47c)

![Invalid
JSON](https://github.com/user-attachments/assets/8cbb0e88-965a-4765-b0f9-9c0ab8c7f065)

### Downstream repositories

This change preserves the public resource schema and client contract.
The file-level trigger map was reviewed; no companion repository update
is required.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(dashboard): allow editing free-form objects in application forms
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added a JSON editor for configuration fields that accept objects with
arbitrary properties. Edit nested values directly, with changes retained
when switching between form and YAML views.
* **Bug Fixes**
* Invalid JSON is flagged and prevents create or update submissions
until corrected. The editor supports empty objects, read-only fields,
and objects nested in arrays, and makes validation errors accessible.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-09 02:21:30 +03:00
Aleksei Sviridkin 6f49e5f427 feat(openbao): optional static-key auto-unseal and apiserver egress label (#4766)
## What this PR does

This continues #4168 by @txmazing, which has been idle since 2026-09-10.
The first commit is the work from #4168 squashed into one, with
@txmazing as author. The other two commits constrain the new values and
complete the documentation.

Five changes to the managed OpenBAO app (`packages/apps/openbao`).

1. **Optional static-key auto-unseal** (#2787, #2483). `seal.type` is
`shamir`, the unchanged default, or `static`. With `static` the chart
mounts the Secret named in `seal.secretName` through the upstream
chart's `server.extraVolumes` and adds OpenBAO's `seal "static"` stanza
to the standalone and the Raft config, so the pods unseal themselves
after a restart. `seal.previousSecretName` and `seal.previousKeyId`
carry the n-1 key during a rotation. The chart never generates the key.
A cluster administrator creates the Secret in the tenant namespace,
because the `cozy:tenant:*` roles grant nothing on Secrets. A way for
tenants to supply secrets by reference is proposed separately in
cozystack/community#82.
2. **No silent seal change.** The template reads the seal in use from
the `<release>-config` ConfigMap and refuses a different `seal.type`
unless `seal.allowMigration: true` acknowledges a migration. Going from
`static` back to `shamir` is refused, because the chart cannot render
the disabled stanza that direction needs.
3. **Apiserver egress label** (#2793). The server pods carry
`policy.cozystack.io/allow-to-apiserver: "true"`, so
`service_registration "kubernetes"` reaches the API server and HA pods
no longer hang at start.
4. **Constrained values.** The upstream chart passes the server config
through `tpl`, and its start script replaces upper case placeholders
such as `HOSTNAME` in it. A key identifier with template delimiters
would run as a template inside helm-controller, which runs as
cluster-admin, so `seal.keyId` and `seal.previousKeyId` take lowercase
alphanumerics, hyphens, dots and underscores. The Secret names become
pod volume names `userconfig-<name>`, so they take DNS labels of at most
52 characters. A previous Secret or key identifier equal to the current
one fails at render. The apps API only defaults against the schema, and
Helm enforces these patterns at install and upgrade.
5. **README.** The trade-off of `static`, the key Secret, the rotation
with the pod replacement that the `OnDelete` strategy requires, the
migration from Shamir step by step after the OpenBAO guide, and the
one-time pod replacement after an upgrade so that existing pods get the
label.

### Validation

- `helm unittest .` in `packages/apps/openbao` passes 28 of 28 with
helm-unittest 1.1.2, the version CI pins, and with 1.2.1.
- Dropping the key identifier patterns, moving the Secret name limit to
51 or 53, or removing either equality guard turns the suite red each
time.
- Rendering this chart and then the vendored `openbao` chart with a
template in `seal.keyId` now stops at the schema. Without the pattern
the same two-stage render executed it, `lookup` included.
- With default values, standalone and with `replicas: 3`, `helm
template` renders the same as `main` apart from the six-line
`extraLabels` block.
- `make generate` in the package with cozyvalues-gen v1.7.0 and at the
repository root with Go 1.27.1 leaves the tree unchanged. `make
unit-tests` passes apart from seven BATS tests that fail the same way on
a clean `main` under macOS, most of them because macOS ships no
`timeout`. In `make test-controllers` only `internal/backupcontroller`
fails, because its tests do not build on current `main` either, which
#4768 fixes. `hack/check-commit-trailers.sh` passes and `cmd/api-gate`
reports no sizeable API change.
- Nothing here was run on a cluster. #4168 reports a Cozystack v1.6.3
cluster where a `replicas: 1` instance with `seal.type: static` came
back `Sealed false` after `kubectl delete pod`.

TLS between pods and an init or bootstrap job are not covered. Both are
on the #2787 list and independent of this change.

### Screenshots

No dashboard code change. The application form gains the `seal` group
from the regenerated `openAPISchema` and `keysOrder`.

### Downstream repositories

Walked the trigger map against the diff. The new `seal` field in
`packages/apps/openbao/values.schema.json` reaches the hand-written
`cozystack_openbao` resource of the Terraform provider. That is tracked
in cozystack/terraform-provider-cozystack#44, which also needs
`allowMigration` and the value constraints. The website reference page
is generated from this `README.md` for the apps listed in its
`Makefile`, and `openbao` is in `APPS`, so the page follows the next
stable tag without a follow-up. The website pages that mention the
apiserver label describe it in general terms and carry no OpenBAO
workaround. There is no `hack/` change, no platform values change, no
variant and no change to ApplicationDefinition semantics.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
https://github.com/cozystack/terraform-provider-cozystack/issues/44
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(openbao): the OpenBAO app can auto-unseal with OpenBAO's static-key seal (`seal.type: static`) from a Secret that a cluster administrator creates in the tenant namespace, with n-1 rotation through `seal.previousSecretName` and `seal.previousKeyId`. Shamir stays the default, and changing the seal of an existing instance needs `seal.allowMigration`. Server pods now carry `policy.cozystack.io/allow-to-apiserver: "true"`, so HA instances no longer hang at start behind the apiserver egress policy.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* OpenBAO supports static sealing with keys stored in an existing
Secret; Shamir sealing remains the default.
* Configure current and previous keys to support key rotation in
standalone and high-availability deployments. Keep the previous key
until stored keys have been re-wrapped.
* Seal-type changes are validated. Shamir-to-static migration requires
explicit acknowledgement; static-to-Shamir migration is not supported.
* Static-seal settings validate required key details and Secret names.
Server pods receive a label needed by the configured API-server egress
policy.
* **Documentation**
* Added guidance for configuring static sealing, managing Secrets,
rotating keys, and migrating seal types.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-09 02:17:21 +03:00
Aleksei Sviridkin 8c8f2522ee fix(cilium): keep the host mark on host traffic to a local pod
Since v1.20.0 the bpf_host SNAT conflict check fires for any packet
carrying the host mark on every device bpf_host runs on, and the SNAT
path replaces that mark with SNAT_DONE. With kube-ovn chaining a host
connection to a pod on the same node leaves through ovn0, so the pod's
to-container program sees SNAT_DONE and classifies the source as world
instead of host. The implicit localhost allow then stops matching:
kubelet probes fail under any default-deny ingress policy that does not
admit world, and a world deny drops the API server calling a webhook on
its own node.

Carry a verbatim backport of cilium/cilium#49232, which skips the
conflict check when the destination is a local endpoint: such traffic
never leaves the node and cannot collide with pod egress SNAT. Keeping
the hunks identical to upstream lets the patch be dropped as soon as a
shipped cilium release contains the fix. Without chaining, host traffic
to a local pod does not pass to-netdev and nothing changes.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-09 02:04:40 +03:00
Aleksei Sviridkin b64e12b0cf docs(cilium): re-anchor comments the v1.20.2 bump made stale
The admission-webhook deny policies in ingress-nginx and kubeovn-webhook
cite Cilium source and the vendored values by line number. The bump
moved endpointLockdownOnMapOverflow and the policy-map ordering block in
pkg/endpoint/bpf.go, so the cited lines pointed elsewhere. Name the key
and the function instead, which survive the next bump, and move the
getEndpointSelector citation to v1.20.2, where the cited lines are
unchanged.

The kubeProxyReplacement pin explained the service translation in terms
of v1.19.5. With Gateway API on, v1.20.2 keeps socket LB in the host
namespace, so pod-namespace connections are translated per packet in
bpf_lxc before egress policy runs, which is what the pin relies on. The
same holds for the e2e image mirror, whose egress-policy comment in
hack/e2e-talos-image-cache.yaml described socket LB.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-09 02:04:40 +03:00
Aleksei Sviridkin e60461dc56 docs(cilium): stop tying the endpoint-leak race note to one release
The note said the restore-window race had no released fix as of
v1.19.5, which reads as current on every later pin whether or not
anything changed upstream.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-09 02:04:39 +03:00
Aleksei Sviridkin 0f31e7a148 fix(cilium): bump vendored chart to v1.20.2
v1.19.5 rejects platform HTTPRoutes on a shared tenant Gateway with
NotAllowedByListeners (cilium#42159). The 1.19 line that fixes it also
marks a same-port HTTPS-terminate and TLS-passthrough pair
ProtocolConflict while still programming both, which leaves the
TenantGateway Ready=False on a layout that serves. 1.20 drops such a
pair from Envoy outright, so the listener layout is fixed against 1.20
rather than against a 1.19 status quirk, and the bump goes straight
there.

The update target now follows the 1.20 line. v1.20 moved the ARP case of
do_netdev under ENABLE_IPV4, which broke the anchor of the IPv6
passthrough patch; the patch's ETH_P_IPV6 case now sits right before
`default:`, outside the ENABLE_IPV4 block. The other two cases apply
unchanged.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-09 02:04:39 +03:00
Aleksei Sviridkin 3dc0d046b3 fix(postgres): cap the default shared_buffers at a quarter of the memory limit (#4724)
## What this PR does

The chart renders no `shared_buffers`, and CloudNativePG does not set
one either (its default parameters in `pkg/postgres/configuration.go` at
v1.30.0, the operator version this repo ships, have no such key).
PostgreSQL therefore starts with its built-in 128MB whatever the
container gets. A full buffer pool is the whole memory limit on
`t1.nano` and half of it on `t1.micro`, which is the chart's default
preset. The pool is shared memory that the kernel cannot reclaim without
swap. A small or idle database never fills it, but a dump, index scans
over a large table or a replica catching up after a failover can, and
then the instance is OOMKilled.

Observed on a cluster running Cozystack 1.4.3, whose chart renders the
same parameters, with a three-instance `t1.nano` database. A
`pg_dumpall` run through `kubectl exec` inside the primary's container
got that container OOMKilled. The dump client ran in the same container
and shared its limit, so that first kill is weak evidence on its own.
After the failover the old primaries could not rejoin. Their `pg_rewind`
against the new 128Mi primary kept failing with `could not fetch file
list: SSL SYSCALL error: EOF detected` until that primary's memory limit
was raised to 512Mi, and then both rewinds completed.

Below a 512Mi memory limit the chart defaults `shared_buffers` to a
quarter of the limit with a 16MB floor, capped at the rendered memory
request rounded down to whole MiB. CloudNativePG rejects a Cluster when
`shared_buffers` exceeds that request, which can happen with memory
overcommit. That is 32MB on `t1.nano` and 64MB on `t1.micro`. From 512Mi
up a quarter is at least 128MB, so nothing is rendered and those
clusters keep the PostgreSQL default exactly as before. An explicit
`shared_buffers` in `postgresql.parameters` still wins at any size. The
limit is read through `cozy-lib.resources.defaultingSanitize`, the
helper that fills `spec.resources`, which is also how
`packages/apps/mariadb/templates/_tuning.tpl` sizes the InnoDB buffer
pool. Presets and partial `resources` overrides therefore resolve
exactly as they do for the container.

A quarter is the starting point the PostgreSQL documentation gives for a
dedicated server with 1GB or more, and it advises a smaller share below
1GB, so this is a ceiling for small instances rather than tuning. Larger
presets are left alone on purpose. Deriving a bigger pool for them is a
tuning change that would restart every PostgreSQL instance on upgrade,
and it does not belong in a fix.

### Upgrade impact

`shared_buffers` needs a restart. On upgrade every cluster whose memory
limit is below 512Mi and that does not set `shared_buffers` gets the new
value, and that includes every cluster created with the default preset.
The chart leaves `primaryUpdateStrategy` and `primaryUpdateMethod` at
their CloudNativePG defaults, `unsupervised` and `restart`, so the
operator restarts the replicas first and then the primary in place, and
writes pause for the length of one PostgreSQL restart. A single-instance
cluster is unavailable for that time. Clusters from 512Mi up render the
same manifest as before and are not touched.

### Validation

- `helm unittest` passes all 166 tests in 21 suites. The new suite
covers the default preset, `t1.nano`, a `resources` override that sets
only cpu, a 500Mi limit just under the threshold, the 16MB floor,
`t1.small` and `t1.large` rendering no key, an explicit value on a small
and on a large preset, and the other default parameters. Seven
additional cases cover memory overcommit, rounding down a fractional MiB
request, a request below the 16MB floor, explicit values and the
unchanged 512Mi threshold. Four cases fail against the previous PR
template.
- Breaking the threshold comparison, the 16MB floor or the divisor, or
dropping the new block, turns at least one case of the suite red each
time.
- `helm template` of the whole chart renders `t1.small` exactly as the
base does, and the default preset differs only by `shared_buffers:
"64MB"`.
- `make generate` with cozyvalues-gen v1.7.0, the version CI pins,
leaves `README.md`, `values.schema.json`, the Go types and the
ApplicationDefinition unchanged, because `values.yaml` only gains a
comment.
- The new default was not exercised on a live cluster.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff. It touches one chart template, a comment in `values.yaml` and a
new test file. No package is added, renamed or removed, no field in
`values.schema.json` is added, removed or renamed, and no default in
`values.yaml` changes, so the Terraform provider has nothing to restate
and the generated README is identical. Neither the website nor the
provider mentions `shared_buffers`. No entry matches.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(postgres): default shared_buffers to a quarter of the memory limit with a 16MB floor, capped at the memory request, when the limit is below 512Mi and shared_buffers is not set. PostgreSQL's built-in 128MB was the whole memory limit on t1.nano and half of it on t1.micro, the default preset, so a dump, index scans over a large table or a replica resync could fill the buffer pool and get the instance OOMKilled. Affected clusters restart their instances once on upgrade, the primary last, with a short pause in writes. Clusters from 512Mi up and clusters that set shared_buffers explicitly are unchanged.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* PostgreSQL now automatically adjusts its default memory allocation for
deployments with memory limits below 512 MiB, while retaining
PostgreSQL’s default for larger limits. Explicitly configured values
continue to take precedence.
* **Documentation**
* Clarified how the automatic memory allocation works, including its
minimum and memory-request cap.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-09 00:10:40 +03:00
Aleksei Sviridkin e7bd3f4c58 fix(mongodb): set the s3 bucket and prefix on the bootstrap restore (#4779)
## What this PR does

helm fails on every install or upgrade of a mongodb release with
`bootstrap.enabled: true`. the chart renders a
`PerconaServerMongoDBRestore` without `spec.backupSource.s3.bucket`, the
operator crd marks that field required, and the api server rejects the
object.

```text
PerconaServerMongoDBRestore.psmdb.percona.com "<release>-restore" is invalid: [spec.backupSource.s3.bucket: Required value, <nil>: Invalid value: "null": some validation rules were not checked because the object was invalid]
```

the bucket is not the only gap. for a `backupSource` restore the
operator builds the pbm storage from `backupSource.s3` alone. the only
location data it takes from `backupSource.destination` is the backup
name, its last path segment. that is `getBackup` and `getStorage` in
`pkg/controller/perconaservermongodbrestore` at v1.22.0, the operator
version this repo ships. so a restore with a bucket and no prefix would
pass validation and then look for the backup at the bucket root.

the fix adds two helpers in `templates/_backup.tpl` that take the bucket
and the prefix from `backup.destinationPath`, with the same expressions
the backup storage already used. `mongodb.yaml` and `restore.yaml` both
call them, so the restore reads from where the backups are written. both
places quote the values now. a numeric bucket or prefix like
`s3://2024/10` used to render as ints that the crd rejects and now stays
a string. the backup storage still leaves out an empty prefix, so for
every other `destinationPath` running releases keep the same object. the
restore always sets the prefix, an empty one as `""`. a
`destinationPath` that names no bucket, like `s3://`, now fails at
render time instead of reaching the operator with an empty bucket.

the bug goes back to when the chart was added. both crd versions the
repo has shipped, 1.21.1 and 1.22.0, require the bucket, so no chart
version could ever create this restore. release-1.6 has the same
template and gets a backport.

tested with helm-unittest 1.1.2, the version ci pins.

- `helm unittest` passes 151 of 151 tests in 11 suites. before this
change it was 143 in 10.
- six new cases in `tests/restore_test.yaml` cover the bucket and prefix
with and without a trailing slash, a bucket root, a numeric bucket and
prefix, the render-time failure, and the check staying off when
bootstrap is disabled.
- the new `tests/restore-storage_test.yaml` renders the cluster and the
restore together and checks that bucket, prefix, endpoint, credentials
secret, `insecureSkipTLSVerify` and `forcePathStyle` match on both
sides, and that a numeric bucket and prefix stay strings on both.
- against the old templates 7 of the 8 new cases fail. the eighth only
pins that the new check stays inside the bootstrap block, so it passes
either way.
- `helm template` of the `PerconaServerMongoDB` for 8 `destinationPath`
forms. 5 render the same storage as before. for the two bucket root
forms the only difference is that a `prefix: null` the api server prunes
anyway is no longer rendered. `s3://2024/10` goes from ints the crd
rejects to strings.
- the rendered restore was checked offline against the crd in
`packages/system/mongodb-operator` with the apiextensions validation
code the api server runs. before the fix all 8 forms fail with
`spec.backupSource.s3.bucket: Required value`. after it 7 pass and
`s3://` stops at render time.
- `make generate` with cozyvalues-gen v1.7.0, the version ci pins,
leaves every generated file unchanged.
- no restore was run against a live cluster.

on upgrade, releases with `bootstrap.enabled: false` render the same
manifests as before. a release that still has `bootstrap.enabled: true`
creates its restore on the first reconcile after the upgrade, and the
operator starts restoring into the cluster as it is at that moment. the
release note says so.

three related things are out of scope here. the restore runs in the
pbm-agent sidecar, which the operator only adds when
`spec.backup.enabled` is true, so a bootstrap with `backup.enabled:
false` still cannot complete. the credential mismatch in #3832 is not
addressed, a bootstrap restore goes through the same pbm logical restore
that issue describes. and a restored release backs up to the same
`backup.destinationPath` it restored from, since the chart has one field
for both, so with backups on it writes into the prefix of the source.

### Screenshots

not a ui change.

### Downstream repositories

walked the trigger map in `docs/agents/contributing.md` against the
diff. it touches two templates, one new helper file and two test files
in `packages/apps/mongodb`. no package is added or renamed,
`values.yaml`, `values.schema.json` and `README.md` are unchanged, and
nothing under `hack/` moves. the website page for mongodb is generated
from the readme and has no bootstrap walkthrough to update. no entry
matches.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(mongodb): a mongodb app created with bootstrap.enabled now gets a valid PerconaServerMongoDBRestore. the restore carries the s3 bucket and prefix of backup.destinationPath, the same location the backup storage writes to, so the api server accepts it and the operator looks for the backup in the right place. before this the api server rejected the restore because the bucket was missing, and the release failed.

on upgrade, an app that still has bootstrap.enabled set to true, for example after an earlier restore attempt that failed on the missing bucket, creates its restore on the first reconcile, and the operator restores the backup into the cluster as it is at that moment. set bootstrap.enabled to false before the upgrade on every mongodb app that should not be restored.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* MongoDB bootstrap restores now use the bucket and prefix from the
configured S3 backup destination, keeping restore storage settings
aligned with backup settings.
* **Bug Fixes**
* Bootstrap now reports an error when the S3 destination does not
specify a bucket.
* Numeric-looking bucket and prefix values are preserved as strings in
backup and restore settings.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-08 23:54:44 +03:00
Yan Bondarenko 4b606036bb fix(mongodb): set the s3 bucket and prefix on the bootstrap restore
With bootstrap.enabled the chart rendered a PerconaServerMongoDBRestore
without backupSource.s3.bucket. The operator CRD requires that field, so
the API server rejected the restore and the HelmRelease failed. The prefix
was missing too, and it matters just as much. For a backupSource restore
the operator builds the PBM storage from backupSource.s3 alone and takes
only the backup name from backupSource.destination, so a restore without
the prefix would look for the backup at the bucket root.

The restore now takes the bucket and prefix from the same helpers that
fill the backup storage, so it reads from where the backups are written.
Both places quote the two values, so a numeric bucket or prefix stays a
string instead of making the resource invalid. The backup storage still
leaves out an empty prefix, so running releases keep the same object. A
destinationPath that names no bucket fails at render time instead of
reaching the operator with an empty bucket.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-10-08 22:39:14 +03:00
Yan Bondarenko 3ea7a95108 docs(openbao): document the static seal trade-off and the Shamir migration
With the static seal the key sits in a Secret in the same cluster as the
data it unwraps, so the barrier is only as strong as access to Secrets
in that namespace, to etcd and to backups. The README now says so before
the reader picks a seal, and it says what the key identifier is for,
since OpenBAO picks the key for stored data by it.

Tenant roles grant nothing on Secrets and no pods/log, so a cluster
administrator creates the key Secret and reads the log line that
confirms a re-wrap.

The StatefulSet uses the OnDelete strategy, so neither a seal migration
nor the apiserver egress label reaches a running pod until it is
replaced. The README describes the migration from Shamir step by step,
following the OpenBAO seal migration guide, and says that pods created
before the label have to be replaced once.

The migration starts with a backup. The standalone file backend is not
transactional, so its volume is copied with the server stopped or
snapshotted atomically. The upgrade that sets allowMigration is the
point of no return for the chart, and the README says how a cluster
administrator backs out of it before any pod is replaced.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-10-08 22:39:10 +03:00
Yan Bondarenko bfbb5ac75f fix(openbao): constrain the seal values that reach the server config
The upstream chart passes the server config through tpl, and its start
script replaces upper case placeholders such as HOSTNAME anywhere in the
copied file. A key identifier with template delimiters would run as a
template inside helm-controller, which runs as cluster-admin, and an
upper case one could come out different on every pod. Both key
identifiers now take lowercase alphanumerics, hyphens, dots and
underscores.

The Secret names become pod volume names of the form userconfig-<name>,
which Kubernetes validates as DNS labels. Both name fields now take a
DNS label of at most 52 characters, and the volume entries are quoted.
The apps API only defaults against the generated schema, so Helm
enforces these patterns when it installs or upgrades the release.

A previous Secret equal to the current one mounts the same volume twice,
which the API server rejects for the StatefulSet, and OpenBAO refuses to
start when equal key identifiers name different key material. Both
cases now fail at render. The deepcopy functions for the Seal type are
regenerated as well.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-10-08 22:39:10 +03:00
txmazing 0c52dedbe2 feat(openbao): optional static-key auto-unseal and apiserver egress label
The OpenBAO app has no auto-unseal (#2787). Every restart of a server,
after a node drain, an image bump or an OOM kill, leaves it sealed until
an operator enters the Shamir key shares into each replica. In HA mode
the pods also hang at start (#2793). Cozystack's allow-to-apiserver
policy only admits egress from pods labelled
policy.cozystack.io/allow-to-apiserver, the app never set that label,
and service_registration "kubernetes" cannot reach the API server
without it.

A new seal value selects shamir, which stays the default, or OpenBAO's
seal "static". With static the chart renders the static seal stanza into
the standalone and the Raft config and mounts the key Secret named in
seal.secretName through the upstream chart's server.extraVolumes. The
chart never generates, stores or rotates the key, because a missed
lookup would replace it and leave the data unreadable.
seal.previousSecretName and seal.previousKeyId carry the n-1 key that
the static seal accepts during a rotation. The server pods always carry
the apiserver egress label.

Changing the seal of an initialized instance is a seal migration that
OpenBAO performs under bao operator unseal -migrate. The chart reads the
seal in use from the config ConfigMap the previous release rendered and
refuses a different seal.type unless seal.allowMigration acknowledges
the migration. Moving away from static needs the old stanza kept with
disabled = "true", which the chart does not render, so that direction is
refused regardless. The README documents the key Secret, the rotation
with the pod replacement that the OnDelete update strategy requires, and
the log line that confirms the re-wrap.

Signed-off-by: txmazing <txmazing@gmail.com>
Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-10-08 22:39:09 +03:00
Yan Bondarenko 9c0085c358 fix(postgres): cap shared_buffers at the memory request
CloudNativePG rejects clusters whose shared_buffers exceeds the memory
request. Memory overcommit can put that request below both a quarter of
the limit and the 16MB floor.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-10-08 22:39:05 +03:00
Yan Bondarenko 6dbd76ae50 fix(postgres): cap the default shared_buffers at a quarter of the memory limit
The chart renders no shared_buffers, so PostgreSQL starts with its 128MB
default whatever the container gets. A full pool is the whole memory
limit on t1.nano and half of it on t1.micro, the default preset. The
pool is shared memory that the kernel cannot reclaim without swap, so
when a dump, index scans over a large table or a replica catching up
after a failover fill it, the instance is OOMKilled.

Instances below a 512Mi memory limit now default to a quarter of the
limit, at least 16MB. From 512Mi up a quarter is at least 128MB, so
those instances render exactly as before. shared_buffers needs a
restart, so on upgrade CloudNativePG restarts every instance below
512Mi that does not set it explicitly.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-10-08 22:39:05 +03:00
Yan Bondarenko 96b35893a5 fix(dashboard): mark a free-form field invalid when the schema rejects it
A JSON object that parses but fails the schema, for example on
minProperties, left the textarea with aria-invalid false, and nothing
linked it to the error list RJSF renders under the field, so assistive
technology announced the field as valid. The field now reports the
schema errors RJSF passes in and points aria-describedby at their list,
unless hideError keeps that list from rendering. The red help text
stays tied to input that is not a JSON object.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-10-08 22:39:01 +03:00
Yan Bondarenko d639113caf fix(dashboard): allow editing free-form objects in application forms
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
Assisted-by: LLM
2026-10-08 22:39:01 +03:00
Aleksei Sviridkin 87a64071a7 fix(cilium)!: take cilium's own addresses from a block kube-ovn does not hand out (#4778)
## What this PR does

fixes #3740.

on the kube-ovn variants cilium allocates no pod ips, but the agent
still takes its router ip (`cilium_host`, `CiliumInternalIP`) and the
gateway api ingress ip from its own allocation cidr. with `ipam.mode:
kubernetes` that's `Node.spec.podCIDR`, a /24 out of the same /16
kube-ovn hands pods from, so sooner or later kube-ovn gives a pod an
address cilium already holds. we hit it on three clusters, details in
https://github.com/cozystack/cozystack/issues/3740#issuecomment-6022394243.

**what changes**

- the kube-ovn variants run cilium with `ipam.mode: cluster-pool`, a /29
per node out of the last /21 of `networking.podCIDR` (`10.244.248.0/21`
for the default /16). the platform derives the block from
`networking.podCIDR` and passes the same range to kube-ovn as
`--default-exclude-ips`, so a fresh install never hands it to pods.
- the block sits inside the pod cidr on purpose. envoy reaches gateway
api backends from the node's ingress ip, and kube-ovn masquerades host
originated traffic only when the source is in `ovn40subnets`. with the
`100.65.0.0/16` pool from the issue a nodeport through the gateway
answered 200 only on the backend's node and 503 `upstream connect error
... connection timeout` on the other two. with the block inside the pod
cidr it's 200 from all three. worth a look on the ci side too. the ci
sandbox sets talos `podSubnets` to `100.65.0.0/16`, so cilium's ingress
ips there already come from outside `ovn40subnets`, and the gateway e2e
checks Accepted, Programmed and the lb service but sends no request
through a listener, so it wouldn't notice. with this pr the sandbox
takes its block from the pod cidr like everyone else.
- the end of the /16 is what kube-ovn reaches last. it hands out the
lowest free address and reuses released ones only when the free range
runs dry (`pkg/ipam/subnet.go` and `pkg/ipam/ip_range_list.go` at
v1.15.26). a /21 cut into /29s covers 256 nodes.
- the slim variants and guest clusters stay on `ipam.mode: kubernetes`.
cilium is the only cni there and `Node.spec.podCIDR` is the pod range.
- `networking.podCIDR` now needs /20 or wider on the kube-ovn variants,
otherwise the render fails and says why. an install that rendered before
can stop rendering at upgrade, hence the `!`. a /21 cut into /29s also
caps these variants at 256 nodes, a fresh install included.

**existing clusters**

a post-upgrade hook in the cilium chart moves them
(`templates/ipam-migration.yaml`, `files/ipam-migration.sh`). before it
touches anything it checks that the block has a /29 for every CiliumNode
and that it lies inside kube-ovn's default subnet, and stops with both
cidrs printed otherwise. a node cleared past the pool's capacity would
lose pod networking on its next agent restart, and a block outside the
subnet puts the agents outside `ovn40subnets` again. it has to be
post-upgrade. platform migrations run as the platform's pre-upgrade
hook, while every agent still runs `ipam.mode: kubernetes` and writes
`Node.spec.podCIDR` back into a cleared CiliumNode
(`pkg/nodediscovery/nodediscovery.go` at v1.19.5), so the move can only
happen after the cilium release switched the mode. the job runs on the
host network against the agent's own api endpoint (kubeprism on talos),
because it restarts the agents.

1. adds the block and every node's current ingress ip to `excludeIps` of
the default subnet, in one json patch with a `test` op, and reads the
subnet again on conflict, since kube-ovn rewrites that list into its own
order right after a patch. the old ingress ip goes in because cilium
1.19.5 keeps a former ingress ip in the agent's ipcache as
`reserved:ingress` after the move. we saw 2 to 3 such entries per node
hours later. old router ips leave the ipcache and go back to kube-ovn.
2. stops right there if an address in the block is still allocated,
names it, and moves no agent. it first checks the subnet status it
patched against, then waits until `status.v4availableIPrange` stops
offering the block, which is kube-ovn having applied the exclusion, and
then reads the `ips.kubeovn.io` objects of the default subnet. the
status alone can't answer it, kube-ovn leaves excluded addresses out of
`v4usingIPrange` (`GetSubnetIPRangeString` in `pkg/ipam/ipam.go` at
v1.15.26), so a pod still in the block would be invisible on a retry
where the exclusion is already in place. recreate those pods and the
next upgrade retry goes through.
3. one node at a time, clears the inherited /24 from
`spec.ipam.podCIDRs` (again with a `test` op, so a concurrent writer
fails the patch instead of losing an update), waits for the operator to
allocate a /29 from the block and restarts that node's agent. a node
that isn't ready is skipped and its agent picks up the block when it
starts. nodes already moved are left alone, so a retry after a timeout
just continues. the hook runs inside the cilium upgrade, so where
restarting every agent takes longer than the upgrade timeout, 10m by
default, the release first reports a failed upgrade and the move
finishes on the retry.

running pods keep their addresses. every agent restarts twice on
upgrade, once when the config change rolls the daemonset and once from
the hook, and while it restarts host to pod traffic on that node blips
for a few seconds because the router ip moves. a pod that sits on a
router ip today is reachable again after the move, the node no longer
holds that address. a pod on a node's current ingress ip keeps it and
that node keeps seeing it as `reserved:ingress`, so recreate it. the
hook logs those pods by name.

### Validation

- `helm unittest` passes cilium 23/23, platform 246/246 and kube-ovn
22/22. the new suites pin the rendered `cilium-config` keys per variant,
the hook shape, env and rbac, and the platform wiring for the talos and
generic branches, a custom pod cidr, host bits in the network address,
the refusals for a too small or non ipv4 cidr, and the slim variant.
- on a trial merge with main at cbc9d0c4e, where #3849 runs
`hack/*.bats` under real bats 1.14 with `set -u`, the 27 bats tests pass
after the second commit loads `test_helper`, the strict setup audit is
clean, and helm unittest passes cilium 23/23, platform 263/263 and
kube-ovn 22/22.
- `hack/cilium-ipam-migration.bats` passes 27/27 and runs the real
script against a fake kubectl that models the operator, the agent
daemonset and kube-ovn publishing the subnet status. thirteen mutations
turn it red each (exclusion after the move, no ingress exclusion, no
retry, no status check, an ingress ip inside the block listed on its
own, no `test` op, no wait for kube-ovn, no ip object check, no subnet
filter on the ip objects, no capacity check, a block outside the subnet
let through, overlap turned into containment, no log of ingress ip
holders). the containment one is pinned by a usage range that crosses
the start of the block, `10.244.247.250-10.244.248.3`.
- live on a 3 node talos test cluster with cozystack v1.6.4, cilium
1.19.5 and kube-ovn v1.15.26
- moved from `kubernetes` to the block, exclusion first and then the
node moves this script does. 3 nodes in about 40 seconds, every agent
took its router and ingress ip from its /29, no pod changed its address.
- gateway api through a nodeport 9/9 answered 200 and through the
loadbalancer ip 6/6, against 3/9 with `100.65.0.0/16`.
- the final hook job on the moved cluster is a no-op in 2 seconds,
including the ip object read with the job's own rbac. the same ip object
pipeline over that cluster's real objects finds 77 in a populated /24
and none in the block.
- the exact `test` plus `replace` patch on `ovn-default` goes through,
kube-ovn re-sorts the list right after, and the same patch against the
old list is rejected. that's the conflict path the retry covers.
- shellcheck is clean and `hack/check-commit-trailers.sh` passes.
- not run here. the upgrade move from an older release in ci, e2e only
covers the fresh install.

### Screenshots

not a ui change.

### Downstream repositories

walked the trigger map against the diff.
`packages/core/platform/values.yaml` only gains a comment, but
`networking.podCIDR` gets a new constraint on the kube-ovn variants,
which the hand written key table on the website should say. the ansible
role sets `networking.podCIDR` to `10.42.0.0/16`, which yields
`10.42.248.0/21` and needs nothing there. the talm preset keeps
`podSubnets: 10.244.0.0/16`, which no longer matters on these variants
because cilium stops reading `Node.spec.podCIDR` there. the terraform
provider doesn't model platform values.

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/pull/739
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(cilium)!: on the kube-ovn variants cilium now takes its router and gateway api ingress ips from the last /21 of networking.podCIDR, which kube-ovn no longer hands out, so pods stop landing on addresses cilium holds. existing clusters move on upgrade through a post-upgrade hook, and each cilium agent restarts twice, once for the config change and once in the hook. networking.podCIDR needs /20 or wider on these variants, and a /21 of /29s caps them at 256 nodes.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Kube-OVN-enabled variants now reserve the final `/21` of the pod CIDR
for Cilium router and Gateway API Ingress addresses, excluding it from
Kube-OVN pod allocation. Pod CIDRs must be IPv4 with a prefix of `/20`
or shorter.
* Added a post-upgrade migration that moves existing Cilium IP
allocations into the cluster pool and updates affected node networking.
* **Documentation**
* Documented the pod CIDR requirements and node capacity limit for
Kube-OVN variants.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-08 21:41:23 +03:00