The Job in templates/talos/talos-reconcile-job.yaml is the only
producer of the TalosConfigTemplate that each worker
MachineDeployment names in spec.template.spec.bootstrap.configRef;
cluster.yaml renders the reference but deliberately not the object.
Its loop read .Values.nodeGroups directly, while the
MachineDeployment loop reads the effective set through the
kubernetes.nodeGroups helper.
For a cluster that supplies no nodeGroups the two disagree: the
helper's else-branch emits the built-in md0 group, so the md0
MachineDeployment renders, but the raw map is empty and no Job
renders at all. Every Machine the cluster-autoscaler adds to md0
then blocks indefinitely on a TalosConfigTemplate that nothing will
ever create, and the KamajiControlPlane spec.network.certSANs patch
the same Job performs is skipped too.
Range over the helper so the Job set tracks the MachineDeployment
set exactly. No change for a cluster that declares its own groups,
and md0 stays removable: the helper's if-branch keeps a
user-supplied map authoritative.
The gap survived because every helm-unittest fixture and e2e suite
declares md0 explicitly. tests/nodegroups_default_test.yaml does
render the empty-nodeGroups case but lists only
templates/cluster.yaml, so it never looked at the Job. The new
suite pins both halves of the helper contract and fails without
this fix.
Fixes#3504
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 4e9c5ce8ca)
The ephemeralStorage guard could never fire (the pool group is built from a
fixed set of chart values that never includes ephemeralStorage), so it read
as protection that is not there — remove it. Note in the parity script header
that only the four pool objects are compared, not the talos-reconcile Job.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
Extend the golden-parity matrix beyond the resources-sized pool to also
diff a GPU pool and a kubelet-override pool against the parent render, so
future drift in those branches (which would roll the content-hash-named
KubevirtMachineTemplate) is caught in CI.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
Reference the adoption migration by role (Phase 2b) instead of a number
that does not exist on this branch, and correct the NOTES operational-note
count.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
Wire tests/render-parity.sh into the chart's test target so CI verifies
the pool objects stay byte-identical to the parent render.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
Restore the inline machineconfig comments (nameservers, GPU node labels)
inside the reconcile Job's data block so the applied TalosConfigTemplate
matches the parent's, avoiding drift on adoption.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
The cluster field referenced x-cozystack-options source 'kubernetes',
which has no provider (cozystack-api returns NotFound), so the dashboard
dropdown 404s. Drop the annotation so the field is a plain text input;
a real provider listing Kubernetes CRs is a follow-up.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
Replace the collision-prone name-prefix match (pool 'gpu' matched
sibling 'gpu-large-<hash>') with a release-name annotation scope, and
add a render-time guard that fails cleanly when a pool's MachineDeployment
already exists under a different release (e.g. naming a pool after a
parent nodeGroup such as the default md0). Document the naming caveat in
NOTES.txt.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
The preserve-old-KubevirtMachineTemplate block re-emitted any KMT owned
by the CAPI Cluster and referenced by a MachineSet. A kubernetes-nodes
pool's KMT lives in the same namespace under the same Cluster, so the
parent would re-emit it with a foreign meta.helm.sh/release-name and its
next upgrade would fail Helm ownership validation. Restrict the block to
KMTs whose release-name annotation matches the parent release.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
Unit tests cover naming/selectors, clusterName wiring, autoscaler
annotations, omitted replicas, bootstrap TCT reference, KMT hash name,
worker WorkloadMonitor, and the empty-cluster / mis-prefixed-release
render guards. render-parity.sh diffs the pool objects against the
parent kubernetes chart and asserts byte-identity.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
Introduce the kubernetes-nodes application chart: one HelmRelease per
worker node pool, attached to a parent Kubernetes cluster by name
convention (kubernetes-<cluster>). Renders the KubevirtMachineTemplate,
MachineDeployment, MachineHealthCheck, worker WorkloadMonitor and the
talos-reconcile Job with its RBAC, extracted from the monolithic
kubernetes chart. Pool objects are byte-identical to the parent's render
for the same group, so a later adoption never churns live worker VMs.
This is the additive half of the Phase 2 split (community#8); the parent
kubernetes chart is left unchanged.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
quay.io serves only currently-tagged manifests: when a rolling tag moves,
the superseded digest immediately returns MANIFEST_UNKNOWN. The pinned
`centos:stream9` digest has died twice in six days (#3303, #3386), each
time breaking `make build` on main and on every PR that triggers a full
rebuild.
Repinning is not a fix -- the newly pinned digest is always the live tag,
so it is next in line to be retired. Move the runtime stage to a registry
that provably retains superseded manifests instead, which makes the digest
pin durable rather than decorative. Pinning policy is unchanged.
Verified equivalent: after `dnf install -y e2fsprogs xfsprogs nfs-utils`,
both bases provide mkfs.ext4, mkfs.xfs, blkid, mount, umount, fsck.ext4,
fsck.xfs and mount.nfs at identical paths. e2fsprogs and xfsprogs are
identical NVRs. The driver is built CGO_ENABLED=0, so the base libc is not
linked. The pinned digest is the OCI index covering amd64, arm64, ppc64le
and s390x -- the same architecture set as the previous base.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
Fixes#3300. Migrates CNPG Postgres backups off the deprecated native
`spec.backup.barmanObjectStore` (removed in CNPG 1.29; the
`barman-cloud-*` binaries it needs are absent from the `standard` image
variant keycloak-db pins, and the bare system-flavor tags apps/postgres
pins are deprecated upstream) onto the [CloudNativePG Barman Cloud
plugin](https://github.com/cloudnative-pg/plugin-barman-cloud), across
every Postgres backup surface: `apps/postgres`, `system/keycloak`, and
the `cozy-default-cnpg` BackupClass strategy driver. The chart renders
`spec.plugins` + a `barmancloud.cnpg.io/ObjectStore` CR; the
backup-controller driver SSA-applies an `ObjectStore` and patches
`spec.plugins`; `Backup`/`ScheduledBackup` use `method: plugin`. The
`postgres-operator` chart vendors and co-locates the
`plugin-barman-cloud` operator (with its cert-manager-issued mTLS) so
CNPG can discover it.
One platform-level change is required for the plugin to run and ships
with this PR (see the upgrade notes in
`docs/operations/backup-classes.md` and the release note below): the
CloudNativePG operator is bumped to 1.28.1 (also fixing an operator/CRD
version skew that broke all backups). The other prerequisite — the
LINSTOR scheduler admission webhook stripping
`initContainers[].restartPolicy` cluster-wide, which broke the plugin's
restartable sidecar — is already fixed on `main` (f4aef7305 re-vendors
`linstor-scheduler` to upstream chart 0.3.1 / extender v0.3.6, which
keeps the webhook and stops stripping unknown pod fields), so after the
rebase this PR no longer touches `linstor-scheduler`; just do not hold
that package back on an older version when rolling out the plugin.
Supersedes #3306 (the keycloak `system`-image workaround): keycloak-db
now backs up via the plugin on the `standard` image, so that stopgap is
no longer needed.
### Screenshots
<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->
### Downstream repositories
<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.
Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.
IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:
1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->
- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: docs/operations/backup-classes.md changed; a website mirror
follow-up is warranted (not yet opened).
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
fix(backups): migrate CNPG Postgres backups to the CloudNativePG Barman Cloud plugin
Postgres backups (apps/postgres, system/keycloak, and the cozy-default-cnpg BackupClass strategy) now use the CloudNativePG Barman Cloud plugin (spec.plugins + a barmancloud.cnpg.io/ObjectStore, method: plugin) instead of the deprecated native spec.backup.barmanObjectStore (removed in CNPG 1.29; the barman-cloud-* binaries it needs are absent from the standard image variant keycloak-db pins, and the bare system-flavor tags apps/postgres pins are deprecated upstream). The barman-cloud plugin operator is installed alongside the CNPG operator.
One platform-level change rides along and requires operator attention on upgrade:
- The CloudNativePG operator is bumped to 1.28.1 (fleet-wide minor upgrade); this also fixes an operator-newer-than-its-CRDs skew that made the operator report "instance manager was restarted during backup" and fail every backup. The vendored chart ships the CNPG CRDs as ordinary release manifests (crds.create), so helm upgrade updates them with the operator; the skew came from the image.tag pin outrunning the chart's CRDs, which this PR removes.
- The plugin's restartable sidecar requires the linstor-scheduler admission webhook that no longer strips initContainers[].restartPolicy (upstream chart 0.3.1 / extender v0.3.6, shipped separately on main); do not hold the linstor-scheduler package back on an older version when rolling out the plugin.
```
Close#3300, #3246
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Switched PostgreSQL CNPG backups/restores to the plugin-based Barman
Cloud workflow, using dedicated ObjectStore resources for backup and
recovery.
* Added the Barman Cloud CNPG-I plugin chart (deployment, TLS, RBAC,
leader election).
* **Bug Fixes**
* Improved backup/restore attachment messaging when the CNPG Cluster
isn’t ready yet.
* Preserves server-name behavior to avoid WAL-archive prefix issues;
clarified endpoint CA handling.
* **Changes**
* Helm integration no longer uses the legacy
`spec.backup.barmanObjectStore` path; updated gating, credential
projection, and ObjectStore-related RBAC/cleanup.
* Removed the linstor-scheduler admission webhook; updated
CloudNativePG/operator versions and CRDs.
* **Tests**
* Updated controller and Helm-unittest checks for the plugin flow;
added/extended end-to-end backup/restore coverage.
* **Documentation**
* Expanded upgrade/migration and backup documentation; updated backup
example materials and comments.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
quay.io/centos/centos:stream9 is a rolling tag; upstream deletes the pinned
digest whenever it publishes a new stream9 build. The digest re-pinned in
ce4b263ca (sha256:a93e3316...) has now itself been deleted, so every
kubevirt-csi-driver image build fails with:
quay.io/centos/centos:stream9@sha256:a93e3316...: not found
failed to resolve source metadata ...
make: *** [image-kubevirt-csi-driver] Error 1
breaking the "Build packages/apps/kubernetes" job on main and every PR.
Re-pin to the current stream9 manifest-list digest
(sha256:3714c89f1903dac5a5e1eb6c02d84189ff6d962a0c18df01feba0a5406856926),
resolved via `docker buildx imagetools inspect quay.io/centos/centos:stream9`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
## What this PR does
The chart shipped a my.cnf with sizing hardcoded for a large dedicated
host — `innodb_buffer_pool_size=60G`, `innodb_log_file_size=4096M`,
`key_buffer_size=1024M`, `max_connections=4096` — while the default
preset `t1.nano` gives the pod 250m CPU and 128Mi of memory, and the
default volume is 10Gi. Nothing connected the two, so the defaults asked
a 128Mi container for a 60G buffer pool and made every first boot
preallocate a redo log worth 40% of the volume.
That is not only a slow start. On the current default, MariaDB 10.6 is
OOM-killed before it finishes checking its configuration, and on 11.8
the server refuses to start at higher limits with `Cannot map
innodb_buffer_pool_size_max`. It survives in practice only because a
large node's overcommit heuristics allow the mapping and a small
database never touches most of it.
Sizing now derives from the instance's own limits through the same
helper that fills `resources` on the CR, so both `resourcesPreset` and
an explicit `resources` block work. The buffer pool takes half the
memory limit, rounded down to a 128 MiB multiple because InnoDB on the
10.6 line allocates in chunks of that size and rounds requests up — a
naive value would hand the pod a pool larger than its own limit. Half
rather than the conventional 70-80% because that advice assumes a
dedicated host where the remainder is page cache and overshoot means
swapping; under a cgroup limit the same budget also has to cover
per-connection buffers, temporary tables, the key cache and mysqld's own
heap, and overshoot means the kernel kills the process. The redo log
follows the smaller of the pool and 5% of the volume, and the remaining
allocations scale from memory or CPU with floors at the server's own
defaults.
`innodb_buffer_pool_instances` and `innodb_thread_concurrency` are
removed rather than scaled: every shipped server version logs that they
were removed and now do nothing, so this only drops a startup warning.
Settings that describe durability or disk behaviour rather than instance
size — `innodb_flush_log_at_trx_commit`, `innodb_flush_method`,
`innodb_io_capacity`, `innodb_fast_shutdown` — are deliberately left
alone, since the chart cannot derive them from a pod's resources.
### Upgrade safety
An existing instance has a 4096M redo log and will get a smaller one.
Shrinking it was verified on every shipped server version — 10.6.24,
10.11.15, 11.4.9 and 11.8.5 — after both a clean shutdown and a SIGKILL
with writes still unflushed. In all cases the server replayed crash
recovery first and resized the log afterwards, with data intact by
checksum and `CHECK TABLE ... EXTENDED`.
The ConfigMap carries no operator watch label, so a running healthy
instance keeps its current sizing until its next restart; an instance
already crash-looping on the old values picks the new ones up when it
restarts.
### Release note
```release-note
fix(mariadb): size my.cnf from the instance's own CPU, memory and volume instead of hardcoded values that assumed a large dedicated host. Instances on small presets previously advertised `max_connections=4096`, which they could not serve; the limit now scales with memory and starts at the server default of 151.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- MariaDB server settings now scale automatically based on CPU, memory,
storage, and configured resource limits.
- Improved buffer pool, redo log, connection, cache, temporary table,
and thread sizing across instance presets.
- Large instances can use buffer pools beyond the previous fixed limit.
- **Bug Fixes**
- Improved sizing for small-volume deployments.
- Removed obsolete MariaDB configuration options for improved version
compatibility.
- Explicit resource settings now take precedence over preset values.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Three package Makefiles — `apps/mariadb`, `apps/kafka` and
`extra/monitoring` — declared an identical `test` target twice, so every
invocation printed:
```text
Makefile:19: warning: overriding commands for target `test'
Makefile:7: warning: ignoring old commands for target `test'
```
Make names the lines in that warning itself: it was already running the
second recipe and discarding the first, so the earlier declaration was
dead in each case and removing it changes nothing but the noise. All six
recipes were `helm unittest .`.
Which copy is dead is not the same across the three, so this is not a
mechanical edit. In `monitoring` the `.PHONY: test` line sits above the
*dead* recipe rather than the live one; since `.PHONY` applies to the
target name wherever it is declared, the declaration stays and only the
recipe beneath it goes.
Default goals are unaffected: `hack/package.mk` sets
`.DEFAULT_GOAL=help` and is included ahead of any target in these files,
so `test` was never selected by first-target-wins. Confirmed with `make
-p --dry-run` before and after, and each package's tests still run — 11,
35 and 9 respectively, with no warnings left.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Streamlined build workflows for MariaDB, Kafka, and Monitoring
applications.
* Removed redundant test target definitions while preserving the
existing Helm unit test capability.
* Other generation, update, and image-building workflows remain
unchanged.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Three package Makefiles declared an identical `test` target twice, so
every invocation printed "overriding commands for target 'test'" and
"ignoring old commands". Make names the lines in that warning: it was
already running the second recipe and discarding the first, so the first
declaration was dead in each case and removing it changes nothing but
the noise.
Which copy is dead is not the same in all three, so the edit is not
mechanical. In monitoring the `.PHONY: test` line sits above the dead
recipe rather than the live one; `.PHONY` applies to the target name
wherever it is declared, so the declaration stays and only the recipe
under it goes.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The package Makefile declared an identical `test` target twice, so
every invocation printed "overriding commands for target 'test'" and
"ignoring old commands". Make already ran the second recipe and
discarded the first, so removing the first is behaviour-preserving.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The shipped my.cnf hardcoded the tuning of a large bare-metal host: a 60G
buffer pool, a 4096M redo log, a 1024M MyISAM key cache, 4096 connections
and 24 pool threads. The chart defaults to t1.nano, which is 250m of CPU
and 128Mi of memory with a 10Gi volume, so by default a 128Mi container
was asked to map 60G of buffer pool and preallocate a redo log worth 40%
of its data volume.
That is not merely wasteful. With the Linux default heuristic overcommit
the 60G mapping is refused outright on a node without the memory to back
it, and mariadbd exits with "Cannot map innodb_buffer_pool_size_max"; where
the mapping is permitted, the eagerly allocated key cache alone exceeds a
small instance's limit and the pod is OOM killed during startup. A nano
instance only survives on a node large enough for the mapping to be granted
and with a working set small enough never to touch it.
Every value that is a memory allocation or a thread count now derives from
the memory and CPU limits resolved by the same cozy-lib helper that fills
the MariaDB CR's resource stanza, so an explicit resources block and a
preset both flow through. Settings that encode durability or storage
behaviour rather than size -- the flush method, flush_log_at_trx_commit,
io_capacity, fast_shutdown -- are left alone, because the chart cannot
infer them from the pod.
The buffer pool takes half of the memory limit. The usual advice to give
InnoDB 70-80% of RAM assumes a dedicated host where the rest is page cache
and overshooting merely swaps; under a cgroup limit the same budget must
also cover per-connection buffers, temp tables, the key cache and mysqld's
own heap, and overshooting is a kill rather than a slowdown. Sizes above
128 MiB are rounded down to a whole number of 128 MiB chunks. On the 10.6
line InnoDB allocates the pool in 128 MiB chunks and rounds a request up,
which would hand the pod more pool than its limit allows; the newer lines
either autosize the chunk or ignore it outright, so there the rounding only
costs a little pool on a limit that is at least 256 MiB and not a multiple
of it -- the rounding applies to half the limit, and below one whole chunk
it does not fire at all -- and buys one effective size across every version
the chart ships.
Most caps are the previously shipped value, so an instance large enough to
reach one is sized as it is today, and most floors are the server's own
default, so a small instance is not configured more restrictively than an
untuned MariaDB. Three clamps deliberately break that symmetry. The buffer
pool has no cap: pinning it to the old 60G would keep the largest instances
sized for a host they do not run on, so an instance with 128Gi or more of
memory is the one case where a value goes up rather than down, to half of a
limit it already has. The MyISAM key cache goes the other way, capped at
128M against the 1024M shipped and floored at 8M against a 128M default,
because it is allocated eagerly at startup and MariaDB's own system tables
are Aria rather than MyISAM, so on a small instance it is close to dead
weight charged against the limit. The IO threads are the third, floored at
2 against a default of 4, because they should follow the CPU limit rather
than the host. The one user-visible narrowing is
max_connections on small instances, which falls from 4096 to the stock
default of 151 -- a limit those instances could not have served anyway, and
one that now rises automatically as the instance is sized up.
Changing the redo log size is safe on every version this chart ships.
InnoDB replays the existing log during crash recovery and only then rewrites
it at the new size. This was verified on all four shipped versions --
10.6.24, 10.11.15, 11.4.9 and 11.8.5 -- after both a clean shutdown and a
SIGKILL mid-write, with the data intact afterwards. The size is also
bounded by a twentieth of the data volume, so first boot no longer
preallocates a large part of it -- below roughly 2Gi of volume the 96M
floor wins instead and the server's own default is what lands.
The ConfigMap is not labelled k8s.mariadb.com/watch, so the operator does
not roll an instance when it changes. That is left as it was. An instance
crashlooping on the old config picks the new one up as soon as the kubelet
restarts it, which is the case this fixes, while a healthy instance keeps
its current sizing until it next restarts for some other reason. Rolling
every tenant database on a chart upgrade would be the larger risk.
innodb_buffer_pool_instances and innodb_thread_concurrency are dropped
rather than scaled. Both were removed from the server in 10.6, below the
oldest version this chart ships, and each start logged "was removed. It
does nothing now and exists only for compatibility with old my.cnf files".
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The operator builds the MariaDB startup probe without setting
failureThreshold, so it falls back to the Kubernetes default of 3. The
kubelet only starts probing once initialDelaySeconds has elapsed and
kills the container on the Nth consecutive failure, so with the
operator's initialDelaySeconds of 20 and periodSeconds of 10 the budget
is 20 + (3 - 1) * 10 = 40 seconds -- exactly what the liveness probe
allows, which leaves the startup probe deferring nothing.
Bootstrapping a fresh datadir does not reliably fit in 40 seconds under
a constrained CPU limit or slow replicated storage. Overrunning the
budget is not a slow start that recovers on the next attempt: the
kubelet kills the container after mariadb-install-db has created the
system tables but before the entrypoint's temporary server applies the
root grant. The restarted entrypoint keys DATABASE_ALREADY_EXISTS off
that directory and skips initialization for good, so the server comes
up with the install-time root account -- local only and passwordless --
rather than the credentials the operator holds. Every probe from then
on fails with "Access denied for user 'root'", so the pod crash-loops
permanently with no path back and the install times out waiting for a
MariaDB that can never become ready.
Raise failureThreshold to 30, giving first boot 310 seconds. Only the
threshold is set, so the operator keeps its own probe handler -- the
agent HTTP endpoint when replication is enabled, the exec check
otherwise -- and the liveness and readiness probes keep their
defaults, so a server that hangs after a successful start is still
caught as quickly as before.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The db.yaml plugins-block comment still carried the over-generalized
claim that the pinned postgresql image variants no longer ship the
barman-cloud-* binaries; mirror the scoped wording from
docs/operations/backup-classes.md (only the standard variant keycloak-db
pins lacks them; this chart's bare system-flavor tags still ship them
but the flavor is deprecated upstream).
After the rebase onto main this PR no longer touches linstor-scheduler:
main re-vendored it to upstream chart 0.3.1 / extender v0.3.6, which
keeps the admission webhook and stops stripping unknown pod fields
(including initContainers[].restartPolicy the barman sidecar needs). The
upgrade note claiming the webhook is removed and locality scheduling
becomes advisory is rewritten as a hold-back warning.
Also: afterwards -> afterward (American English, review nit).
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
make -C packages/apps/postgres generate re-pads the Bootstrap parameters
table for the new bootstrap.serverName row; the committed README still
carried the old padding and kept the pre-commit generate check red.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Two defects of the same class — the WAL-archive S3 prefix (serverName)
changing or defaulting differently between components:
Recovery default regression (review blocker): native CNPG defaulted an
absent externalClusters serverName to the entry name (= bootstrap.oldName),
but the barman-cloud plugin defaults it to the NEW cluster's name, so a
legacy-flow restore with bootstrap.serverName unset would read the wrong
S3 prefix (hard failure, or silently restoring another cluster's data).
The chart now always emits serverName, defaulting to bootstrap.oldName.
Live-cluster serverName flip (found by the gated postgres backup e2e on a
real cluster): the chart renders spec.plugins without serverName
(effective prefix = cluster name) while a strategy template may name a
different one; the driver's ForceOwnership patch then flipped the prefix
mid-stream, splitting the WAL archive across two prefixes — the WAL
holding the backup's start was lost between them and the restore failed
with "WAL not found". The driver now preserves the live Cluster's
effective serverName (explicit parameter, or the cluster name when the
plugin is attached without one) and only applies the strategy's value to
a Cluster with no plugin attached; both charts pin serverName explicitly,
and the example strategy no longer sets one.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Review follow-up (cozystack#3300, non-blocking): the barman-cloud ObjectStore CRD
validates spec.retentionPolicy against ^[1-9][0-9]*[dwm]$, so an operator who
explicitly sets backup.retentionPolicy="" would render retentionPolicy: "" and
the API server would reject the ObjectStore, breaking backup provisioning. Guard
the field in the apps/postgres and keycloak chart ObjectStores with
{{- if .Values.backup.retentionPolicy }} (the Go driver path is already safe via
omitempty). Add a helm-unittest asserting an empty retentionPolicy is omitted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Review follow-up (cozystack#3300): the apps/postgres user-facing docs still
described the removed native spec.backup.barmanObjectStore / archive_command
model. Update the values.yaml @field/@typedef/@param annotations (useSystemBucket,
schedule, S3CredentialsSecret, EndpointCA, endpointCA, the @param backup note,
bootstrap.serverName) and the hand-written README Operations prose to the
barman-cloud plugin + ObjectStore model, then regenerate values.schema.json, the
README parameters table, the api types, and the postgres-rd ApplicationDefinition
schema (which feed the dashboard form help + cozyrd schema). Field names,
defaults, and the storageClass immutability marker are unchanged; only backup/
bootstrap description text changed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Review follow-ups for the CNPG barman-cloud plugin migration (cozystack#3300):
- docs/operations/backup-classes.md: the 'Upgrade notes from chart-managed
backups' section still described the removed native spec.backup.barmanObjectStore
behavior. Rewrite it for the plugin flow (chart renders spec.plugins + a
barmancloud.cnpg.io/ObjectStore; the driver SSA-applies an ObjectStore and
patches spec.plugins), and document the real upgrade hazard: the
plugin-barman-cloud operator + cert-manager must be present in
cozy-postgres-operator or the Cluster references an unregistered plugin and
backups/WAL archiving stall.
- apps/postgres backup_storage_test.yaml: add a positive bootstrap-recovery
rendering test. Asserts externalClusters[].plugin references
barman-cloud.cloudnative-pg.io with barmanObjectName=<release>-recovery and
serverName on the plugin parameter, and that the -recovery ObjectStore carries
spec.configuration but NOT serverName (the plugin forbids it there). Previously
the recovery path had only fail-fast guard tests, no positive coverage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
The System-mode AuthenticationConfiguration rendered a claimValidationRule
ranging over claims.groups, which is statically typed `any` in the apiserver
CEL environment. The .exists() comprehension rejects a range of type any, so
the tenant kube-apiserver failed config compilation at startup and CrashLooped
— the control plane never booted and cluster teardown then hung the e2e
cleanup budget. Wrap the range in dyn() so CEL treats it as dynamic; the has()
guard still short-circuits when groups is absent and the gate semantics are
unchanged. Strengthen the oidc unit test to pin dyn() (mutation-verified).
Verified on a live tenant apiserver (v1.32).
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
Symmetric with the Grafana-side allowed_groups gate — round-3 review
from lllamnyp on PR#3176 covered both surfaces, and #3044's original
System-mode wiring shipped without this guard. The rationale is
identical: audience binding stops REPLAY of a minted token across
tenant kube-apiservers, but nothing stopped Keycloak from ISSUING a
usable token to a caller outside the tenant.
Absent a claim-side gate, any authenticated cozy-realm user could
`kubectl oidc-login` against another tenant's cluster and land at
`system:authenticated`. RBAC default-denies named-resource access, but
`system:authenticated` still leaks discovery (kubectl api-resources,
OpenAPI schemata) — tenant-alice could enumerate tenant-bob's cluster's
CRD shape and built-in resource surface.
Fix: add a CEL `claimValidationRules` entry to the rendered
AuthenticationConfiguration (System mode only) that requires at least
one of the release's four tenant-scoped Keycloak groups
(`<namespace>-{view,use,admin,super-admin}`) to be present in the
token's `groups` claim. The `has(claims.groups) &&` guard is required
— CEL evaluation on a missing claim surfaces as HTTP 500 from the
authenticator instead of a clean 401.
CustomConfig mode is not touched — the tenant's own
AuthenticationConfiguration is authoritative and the tenant is
responsible for claim-side guards on their own issuer.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
quay.io/centos/centos:stream9 is a rolling tag; upstream deleted the pinned
digest (sha256:0c9db821...) when it published a new stream9 build, so every
build of image-kubevirt-csi-driver fails with "not found" — on open PRs and
on main. Re-pin to the current live stream9 digest (sha256:a93e3316...).
This is an emergency CI unblock, not a durable fix: stream9 is garbage-collected,
so upstream will delete this digest too on its next rebuild and the same
"not found" breakage recurs until the pin is bumped again — the same operation
Renovate's weekly, non-automerged digest PR performs. A durable fix (mirror the
base image into cozystack's own registry, or move to an immutable-digest base
such as UBI9) is out of scope for this hotfix and left as follow-up.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
This PR contains the following updates:
| Package | Type | Update | Change |
|---|---|---|---|
|
[github.com/kubevirt/csi-driver](https://redirect.github.com/kubevirt/csi-driver)
| replace | digest | `51c6bca` → `27b52aa` |
---
### Configuration
📅 **Schedule**: (UTC)
- Branch creation
- At any time (no schedule defined)
- Automerge
- At any time (no schedule defined)
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box
---
This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/cozystack/cozystack).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4yNDIuMiIsInVwZGF0ZWRJblZlciI6IjQzLjI1OS4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJhdXRvbWF0ZWQiLCJkby1ub3QtbWVyZ2UvaG9sZCJdfQ==-->
This PR contains the following updates:
| Package | Type | Update | Change |
|---|---|---|---|
|
[altinity/clickhouse-backup](https://redirect.github.com/Altinity/clickhouse-backup)
| final | patch | `2.7.2` → `2.7.4` |
---
### Release Notes
<details>
<summary>Altinity/clickhouse-backup
(altinity/clickhouse-backup)</summary>
###
[`v2.7.4`](https://redirect.github.com/Altinity/clickhouse-backup/blob/HEAD/ChangeLog.md#v274)
[Compare
Source](https://redirect.github.com/Altinity/clickhouse-backup/compare/v2.7.3...v2.7.4)
NEW FEATURES
- add `clickhouse.parts_columns_max_bytes_before_external_group_by` (env
`CLICKHOUSE_PARTS_COLUMNS_MAX_BYTES_BEFORE_EXTERNAL_GROUP_BY`, default
`100000000`) and `clickhouse.parts_columns_max_memory_usage` (env
`CLICKHOUSE_PARTS_COLUMNS_MAX_MEMORY_USAGE`, default `200000000`) — make
the memory caps injected into the `system.parts_columns` check query
configurable instead of hardcoded, `0` means don't inject the
corresponding setting (pre-2.7.2 behavior), fix
[#​1420](https://redirect.github.com/Altinity/clickhouse-backup/issues/1420)
BUG FIXES
- fix `--restore-database-mapping`/`--restore-table-mapping` losing
unrelated comma-separated `--tables` pattern items, or applying the
mapping to the wrong item when the same source database/table name was
repeated across items — each comma-separated pattern item is now matched
and substituted independently, fix
[#​1421](https://redirect.github.com/Altinity/clickhouse-backup/issues/1421)
###
[`v2.7.3`](https://redirect.github.com/Altinity/clickhouse-backup/blob/HEAD/ChangeLog.md#v273)
[Compare
Source](https://redirect.github.com/Altinity/clickhouse-backup/compare/v2.7.2...v2.7.3)
NEW FEATURES
- add `general.max_broken_part_ratio` (env `MAX_BROKEN_PART_RATIO`,
default `0`) — when >0, `create`/`create_remote` skips broken data parts
instead of aborting the whole backup as long as the broken/total part
ratio stays at or below the configured fraction; skipped parts are
logged as warnings and the backup is marked successful but partial; `0`
preserves the legacy any-broken-part-fails behavior, fix
[#​1418](https://redirect.github.com/Altinity/clickhouse-backup/issues/1418)
- add `clickhouse.rebind_replica_path_if_exists` (env
`CLICKHOUSE_REBIND_REPLICA_PATH_IF_EXISTS`, CLI
`--rebind-replica-path-if-exists`, also accepted by the REST API `POST
/backup/restore` endpoint) — opt-in rebind of the replicated table ZK
path to `default_replica_path` during `restore` when the resolved path
already has leftover children but no local table uses it (async-stale
state after DROP); MUST stay `false` during a concurrent multi-replica
restore to avoid split-brain, fix
[#​849](https://redirect.github.com/Altinity/clickhouse-backup/issues/849)
- add `gcs.disable_http2` (env `GCS_DISABLE_HTTP2`) — force HTTP/1.1 for
the GCS client to avoid single-TCP-connection HTTP/2 multiplexing
becoming the bottleneck for high-throughput parallel downloads, fix
[#​1437](https://redirect.github.com/Altinity/clickhouse-backup/pull/1437)
IMPROVEMENTS
- switch the S3 client retry mode from `aws.RetryModeStandard` to
`aws.RetryModeAdaptive`, so upload/download retries back off with a
client-side rate limiter under throttling (`SlowDown`/503) instead of a
fixed token bucket
- support tables with `ENGINE=Alias` during `restore` — strip the
explicit column list (Alias inherits columns from the target table) and
add `allow_experimental_alias_table_engine=1` for ClickHouse 25.11+, fix
[#​1426](https://redirect.github.com/Altinity/clickhouse-backup/issues/1426)
- skip tables from databases with engine `DataLakeCatalog` (read-only
external catalogs) during backup, in addition to
`MySQL`/`PostgreSQL`/`MaterializedPostgreSQL`, fix
[#​1417](https://redirect.github.com/Altinity/clickhouse-backup/issues/1417)
- speed up `create` on servers with many tables: batch the in-progress
`system.mutations` lookup instead of issuing one query per table (O(N^2)
-> O(N)), fix
[#​1431](https://redirect.github.com/Altinity/clickhouse-backup/pull/1431)
- speed up `restore`/`download` table list resolution: eliminate the
O(N^2) linear duplicate scan in
`getTableListByPatternLocal`/`getTableListByPatternRemote` via a shared
O(1) dedup keyed on `metadata.TableTitle`, fix
[#​1430](https://redirect.github.com/Altinity/clickhouse-backup/pull/1430)
- extend TestFlows FIPS 140-3 test coverage: dedicated test steps,
stronger `--fips-info`/version-output and TLS-handshake assertions,
updated requirements documentation, fix
[#​1443](https://redirect.github.com/Altinity/clickhouse-backup/pull/1443)
BUG FIXES
- restore the default `general.compression_use_multi_thread` to `true`:
[#​1378](https://redirect.github.com/Altinity/clickhouse-backup/issues/1378)
defaulted it to `false`, which silently dropped gzip/zstd compression
from multi-threaded (pre-1378 gzip always used `pgzip`) to
single-threaded and caused \~30% slower upload throughput for backups
dominated by one large table (where `upload_concurrency` provides no
per-stream parallelism); the option is now silently ignored instead of
failing config validation for formats other than gzip/zstd
- fix v2.7.1 regression where `restore --schema` on a second replica
rebound the ZK replica path even while the first replica was active and
healthy — a foreign/renamed local table occupying the resolved ZK path
is still detected automatically via `system.replicas` and rebound to
`default_replica_path`, but a non-empty ZK path without a local table
now defaults to the HA-safe join and requires the new opt-in
`rebind_replica_path_if_exists`, fix
[#​1428](https://redirect.github.com/Altinity/clickhouse-backup/issues/1428)
- fix inconsistent `--rbac` behavior: `restore --rbac` no longer returns
an error when the backup contains no RBAC `access` directory, matching
the `download --rbac` behavior, fix
[#​1432](https://redirect.github.com/Altinity/clickhouse-backup/issues/1432)
- inject `ON CLUSTER` for `LIVE VIEW` and `WINDOW VIEW` restored with
explicit column lists when `restore_schema_on_cluster` is set —
`enrichQueryWithOnCluster` missed these statements and the views were
created only on the local replica, fix
[#​1433](https://redirect.github.com/Altinity/clickhouse-backup/pull/1433)
- mask sensitive values (passwords, storage credentials, encryption
keys) when logging `--env` config overrides, and in the API server
action logs and `/backup/status` output, fix
[#​1441](https://redirect.github.com/Altinity/clickhouse-backup/pull/1441)
</details>
---
### Configuration
📅 **Schedule**: (UTC)
- Branch creation
- At any time (no schedule defined)
- Automerge
- At any time (no schedule defined)
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box
---
This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/cozystack/cozystack).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4yNDIuMiIsInVwZGF0ZWRJblZlciI6IjQzLjI0Mi4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJhdXRvbWF0ZWQiLCJkby1ub3QtbWVyZ2UvaG9sZCJdfQ==-->
The prerequisites section described the resolved SeaweedFS class as being
surfaced on the tenant namespace as the `namespace.cozystack.io/seaweedfs`
annotation. The tenant chart renders that key under `metadata.labels`,
alongside the sibling `/etcd`, `/ingress`, `/gateway`, `/monitoring` and
`/host` keys; no annotation of that name exists.
The wrong noun sends an operator to the wrong field. When a Harbor release
stays unreconciled waiting on `BucketInfo`, the documented check is to
confirm the tenant chain resolved a SeaweedFS class. Looking for it in
`.metadata.annotations` returns nothing even on a correctly configured
tenant, which invites the wrong conclusion that the tenant chain is broken
and sends the investigation away from the actual cause.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A leftover debug field "foo" in the <release>-urls Secret template
violates the core/v1 Secret schema. It went unnoticed under client-side
apply but breaks Helm upgrades under server-side apply with
".foo: field not declared in schema", leaving the release in
UpgradeFailed state. Drop the debug block.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
Add optional nodeGroups[name].maxUnhealthy and nodeGroups[name].nodeStartupTimeout
fields that override the cluster-wide nodeHealthCheck defaults per worker node
group. When unset, a group inherits nodeHealthCheck.{maxUnhealthy,nodeStartupTimeout},
so existing values are unaffected. The MachineHealthCheck render resolves the
per-group value first, reuses the existing int-or-string validation, and names
the offending group in the fail() message when an override is invalid.
Delivers the per-group granularity of the standalone maxUnhealthy proposal on
top of the cluster-wide nodeHealthCheck from the Talos worker bootstrap PR.
Co-authored-by: mattia-eleuteri <mattia@hidora.io>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Migrates the E2E **app** test suite from BATS to [Kyverno
Chainsaw](https://github.com/kyverno/chainsaw) — a declarative,
Kubernetes-native E2E framework (CNCF, part of the Kyverno project;
successor to KUTTL) — and wires it into CI as the replacement for the
per-app BATS loop. This grows the original pilot (postgres + bucket)
into the full suite.
All 22 app suites are ported under `hack/e2e-chainsaw/` and the
`hack/e2e-apps/*.bats` files are removed:
`postgres`, `bucket`, `mariadb`, `mongodb`, `redis`, `qdrant`,
`clickhouse`, `kafka`, `etcd`, `openbao`, `harbor`, `foundationdb`,
`external-dns`, `kuberture`, `vminstance`, `gateway`,
`kubernetes-latest`, `kubernetes-previous`, `kubernetes-oidc-system`,
`kubernetes-oidc-customconfig`, `securitygroup`, `serviceexposure`.
**Approach**
- **Declarative** suites assert on `status.conditions` and concrete
fields. The `timeout N sh -ec "until kubectl get ..."` + `kubectl wait`
pair that appeared ~100 times collapses into a single `assert` that
polls existence and state together, with a structured diff on failure
and automatic `events`/`describe`/`podLogs` capture via `catch`
(previously only `harbor.bats` did this, by hand).
- **Imperative** suites keep their logic in `script` steps: `openbao`
init/unseal, `kuberture` external-dns split-horizon probes,
`vminstance`, the `gateway` admission/impersonation cases, and
`kubernetes-latest`/`previous`, which wrap the relocated
`hack/e2e-chainsaw/_lib/run-kubernetes.sh` verbatim (Kamaji bring-up,
LB/NFS/ouroboros checks).
- `gateway` tests derive the tenant apex from the namespace
`namespace.cozystack.io/host` label at runtime, so they are
host-independent.
**CI**
- The `e2e` job now runs `chainsaw test hack/e2e-chainsaw/` via a new
`test-chainsaw` target and uploads the JUnit `chainsaw-report.xml`.
- The `chainsaw` binary is added to the e2e-sandbox image.
- `install-cozystack` and `test-openapi` stay BATS (cluster bootstrap +
OpenAPI checks).
- The per-app 3-retry loop is dropped — assertion polling replaces the
fixed-timeout flakiness it papered over.
**Validation**
Ran against a development cluster: the DB/app, storage, and VM suites
pass; the three suites that depend on platform features not present on
that cluster (`gateway`, `kuberture`, `external-dns`) and the two
heavyweight `kubernetes-*` suites are exercised by this PR's CI run on a
freshly installed platform.
Note for reviewers: service-port asserts use the `(ports[*].port)`
projection form rather than a number-literal filter (`` ports[?port ==
`N`] ``), because Chainsaw v0.2.15 mis-evaluates JMESPath number-literal
comparisons.
**2026-07-09: reconciled with main (~520 commits of drift)**
- etcd suite re-ported to the v1alpha2 operator contracts (#2859):
`readyMembers=3` gate, pod-label/`/scale`/WorkloadMonitor/metrics/defrag
contracts, plus the `ETCD_E2E_S3_ROUNDTRIP`-gated backup round-trip
driving `examples/backups/etcd`
- 12 new `ingress-hostname-policy` gateway cases ported (apex derived at
runtime from the tenant-root namespace label)
- post-branch bats drift folded in: `mariadb-single` webhook guard,
kafka `c1.small` presets, bucket/harbor 2m BucketClaim fail-fast, bucket
readonly-denial promoted to a hard fail (cosi-driver v0.3.1), qdrant
PVC-reclaim guard (#3059), kuberture fail-loud negations,
SC-fallback-default test (#2872 B1)
- 4 suites that only exist on main since the branch was cut are ported:
`kubernetes-oidc-system`, `kubernetes-oidc-customconfig`,
`securitygroup`, `serviceexposure`
- CI seams: `nightly.yaml` converted to the chainsaw invocation,
`run-kubernetes.sh` reconciled (Talos/CABPT waits, tenant drain, LINSTOR
pool wait, talos-image-cache under `_lib/`), `e2e-capture-dataplane.sh`
replicated in the Chainsaw global catch
### Release note
```release-note
fix(kubernetes): tenant Kubernetes teardown no longer hangs when the cluster has no working nodes — the pre-delete hook now bounds its wait for the in-tenant HelmReleases and force-clears their Flux finalizers on timeout (#3271)
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* End-to-end testing is now driven by a unified Chainsaw suite (platform
apps, Kubernetes, networking, security groups, service exposure).
* Test Impact Analysis selects affected suites, with an option to run
the full suite.
* CI produces a JUnit report artifact and collects suite-scoped
diagnostics on failure.
* **Bug Fixes**
* CI E2E now fails fast and emits clearer Kubernetes/HelmRelease and
sandbox bucket state diagnostics.
* Improved teardown reliability with bounded wait + finalizer handling
to avoid hangs.
* **Documentation**
* Updated E2E testing docs and local guidance to reflect the new
Chainsaw-based workflow and suite paths.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Tenant worker VMs were pinned to Talos v1.13.0, which ships Linux
6.18.24. That kernel predates the fixes for two related KVM x86
shadow-paging use-after-free bugs in `arch/x86/kvm/mmu/mmu.c`:
CVE-2026-46113, fixed upstream in
`0cb2af2ea66ad8ff195c156ea690f11216285bdf` and released in 6.18.30, and
CVE-2026-53359, fixed in `81ccda30b4e83d8f5cc4fd50503c44e3a33abfeb` and
released in 6.18.38. A worker's own kernel runs KVM whenever a tenant
enables nested virtualization inside the VM, and either bug lets a
nested guest escape into the worker.
This bumps the worker pin to Talos v1.13.6, which ships Linux 6.18.38.
Stable kernel branches are cumulative, so v1.13.6 is the first v1.13
patch release carrying both fixes. The pinned v1.13.0 ships 6.18.24 and
carries neither; v1.13.5 ships 6.18.36 and carries only CVE-2026-46113.
Nested virtualization stays available to tenants; the kernel bump is the
fix, not a capability removal.
The image-factory schematic is unchanged and resolves for the new
version: both the `openstack-amd64.raw.xz` artifact and the
`installer/<schematicID>:v1.13.6` manifest are served by the public
factory. Talos v1.13 supports Kubernetes 1.31 through 1.36, so the
chart's Talos-to-Kubernetes support window is unaffected and no tenant
`spec.version` needs to move.
Applying this rolls each worker pool once. The new pin changes the
rendered worker spec, which rotates the content hash in the
`KubevirtMachineTemplate` and `talos-reconcile` Job names, and CAPI
replaces the worker VMs to deliver the new kernel. The pinned hashes in
the chart tests are recomputed to match.
A second commit fixes the support-matrix lookup itself. The matrix dict
was keyed on the full patch version, so every Talos patch bump dropped
out of the dict and silently disabled the render-time guard — exactly
the mismatch the guard exists to catch. The lookup now derives the minor
from `talos.version`. The supported-Kubernetes window is unchanged;
unknown minors are still skipped, leaving compatibility to whoever
overrode the pin.
### Screenshots
Not applicable — no UI changes.
### Release note
```release-note
fix(kubernetes): bump tenant worker Talos to v1.13.6, picking up the Linux 6.18.38 kernel fixes for CVE-2026-46113 and CVE-2026-53359 (KVM shadow-paging use-after-free, nested-guest escape into the worker VM). Nested virtualization remains available. Applying this upgrade rolls each tenant worker pool once, as the worker template is renamed to carry the new image.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Updated the default Talos worker release to v1.13.6 for Kubernetes
cluster provisioning.
- Improved compatibility checks to apply consistently across Talos patch
releases within the same minor version.
- **Bug Fixes**
- Corrected generated resource naming expectations when storage settings
or GPU configuration change.
- **Tests**
- Added coverage confirming Kubernetes clusters render successfully with
Talos v1.13.6.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The pre-delete hook (delete.yaml) deleted the tenant's addon HelmReleases with
an unbounded `kubectl delete … --wait=true`. When the tenant has no working
nodes (LINSTOR/ZFS exhaustion, worker VMs that never joined, or a nodeGroup at
minReplicas: 0), those uninstalls cannot converge — in-tenant cilium stays
InstallFailed because cilium-operator has nothing to schedule on — so the hook
hung indefinitely, the parent Helm uninstall failed on the pre-cleanup Job, and
the Kubernetes CR was stuck deleting.
Bound the wait with --timeout=120s; on expiry, force-clear the Flux finalizer
from any child HelmRelease still present so it can be garbage-collected — safe
because the whole tenant (control plane torn down in the steps below) is being
destroyed, so there is nothing left to orphan. A healthy tenant's children
uninstall well within the timeout and never reach the force path.
Also unblocks the render-side kubernetes-oidc-{system,customconfig} Chainsaw
e2e suites, which deterministically hit this teardown hang under the Chainsaw
cleanup budget.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Tenant worker VMs were pinned to Talos v1.13.0, which ships Linux 6.18.24.
That kernel predates both fixes for a pair of related KVM x86 shadow-paging
use-after-free bugs: CVE-2026-46113, fixed in 6.18.30, and CVE-2026-53359,
fixed in 6.18.38. A worker's own kernel runs KVM whenever a tenant enables
nested virtualization inside the VM, and either bug lets a nested guest
escape into the worker.
Talos v1.13.6 ships Linux 6.18.38. Stable kernel branches are cumulative, so
it is the first v1.13 patch release carrying both fixes: the pinned v1.13.0
ships 6.18.24 and carries neither, while v1.13.5 ships 6.18.36 and carries
only CVE-2026-46113. Nested virtualization stays available to tenants — the
kernel bump is the fix.
The new pin changes the rendered worker spec, so the content hash in the
KubevirtMachineTemplate and talos-reconcile Job names rotates and each
worker pool rolls once on upgrade. Pinned hashes in the chart tests are
recomputed accordingly.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The Talos<->Kubernetes support window is declared per Talos minor release,
and patch releases never move it. Keying the chart's matrix dict on the full
patch version meant every Talos patch bump fell out of the dict and silently
disabled the render-time guard — precisely the mismatch the guard exists to
catch.
Derive the minor from talos.version before the lookup, so the guard keeps
firing across patch releases of a known minor. Unknown minors are still
skipped, leaving compatibility to the operator who overrode the pin.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Fold the in-sandbox Talos image cache work into the Chainsaw layout:
the new shared helper relocates to hack/e2e-chainsaw/_lib/
talos-image-cache.sh and run-kubernetes.sh sources it from there (the
ouroboros DNS-check no-retry rework merges clean). The OIDC bats are
resolved as deleted again; their Chainsaw fixtures deliberately skip
the talos.imageFactoryURL injection — the helper only emits it when
the cache is up, and the zero-worker render-side tests instantiate no
DataVolume imports for it to redirect.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Match the full rendered worker-disk URL shape (version + artifact + end anchor) like the sibling override assertion, so the trailing-slash normalization case cannot pass on a partial match.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
cozystack-ci[bot]