fix(kubernetes)!: remove the talos block and other leftovers of the worker pool split (#5191)

## What this PR does

Cleanup after the Kubernetes app was split into `Kubernetes` (control
plane and addons) and `KubernetesNodes` (one release per worker pool):
what the split left behind in values, docs, templates and tests.

### Kubernetes

The `Kubernetes` application has no `talos` field any more. Worker pools
are `KubernetesNodes` resources and take their Talos release, image,
installer and registry mirrors from their own `talos` and `osImage`, so
the cluster's copies did nothing. The last one, `talos.version`, fed a
duplicate of the Talos to Kubernetes support-matrix check. The pool
chart already runs that check against the Talos release each pool
actually boots, including one picked through an `osImage` override; the
cluster's copy compared the Kubernetes version against a release no node
runs. A stored spec that still carries `talos.*` keeps rendering,
because the values schema does not close the object and the API does not
validate `spec` against it, and a unit test pins that.

The pool's check now has render coverage. Its own suite pins the
pairings it must accept, and a bats test renders the refusals through
`talos.version`, `osImage.factory.version` and `osImage.builtin.version`
with schema validation skipped, since the schema offers no Kubernetes
version the matrix refuses.

I fixed the README and field descriptions that described behavior the
chart no longer has, or never had: a dedicated etcd cluster, a
`blockSize.matchVolume` setting that does not exist, a default `md0`
pool, the host default, the substrate mismatch behavior, and rollout
phase names. The README's Breaking Changes section becomes a pointer to
the per-release upgrade notes on cozystack.io. Unused helpers and stale
comments are removed; the chart renders the same bytes.

### KubernetesNodes

Five field descriptions and the `talos` block told users to keep them in
sync with parent fields that do not exist. NOTES warned about colliding
with a parent-managed `md0` pool, although the parent renders no pools
and `md0` is the normal name of a pool. The installer image was credited
to an upgrade Job that never runs one. The values descriptions, README
intro and NOTES now say what the chart renders today, including which
fields have no effect on `substrate: proxmox`: `storageClass`,
`diskSize`, `instanceType`, `talos.imageFactoryURL` (beyond its format
check) and `talos.imageFactoryCA`.

The render error for a pool whose MachineDeployment belongs to another
release told users to remove the pool from the parent's `nodeGroups`,
which cannot clear the conflict. It now names a remedy that works.
Split-era history is gone from Helm and test comments, along with two
`storageClass` fallbacks that can only return the value they fall back
from, a constant variable, and two `.helmignore` rules that match
nothing.

The tenant pod and service CIDRs are literals in both charts, and
nothing checked that they agree. A new bats test renders both charts and
fails when they differ.

No rendered manifest of the pool chart changes. I rendered it for the
snapshot values, GPU, kubelet-override, factory/CA, storageClass
variants and three Proxmox variants on main and on every commit, and the
output is byte-identical. The snapshot and TalosConfigTemplate hash
tests pass unchanged, so no worker rolls. YAML comments inside the
hashed machine config and machine template are left as they are for that
reason.

### The rule

`docs/agents/overview.md` gains a rule so this does not come back with
the next change: app READMEs and field descriptions describe current
behavior, a change users must act on goes to the website's upgrade page,
and a description does not ask the reader to keep it in sync with
another chart.

```release-note
[kubernetes] The Kubernetes application no longer has a `talos` field. Talos settings for workers, and the Talos to Kubernetes compatibility check, live on each KubernetesNodes pool. Existing clusters that still store `talos` values keep working; the values are ignored. KubernetesNodes field descriptions, README and install notes now describe the worker pool as it works today, including which fields have no effect on Proxmox pools.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Configuration**
* Worker OS image, installer, and registry-mirror settings are
configured per worker pool; these settings are no longer available at
the cluster level.
* Kubernetes and Talos version compatibility is checked for each worker
pool. A worker Kubernetes version cannot be ahead of its parent cluster.
* Worker pools support KubeVirt and Proxmox, with pool-specific sizing
and disk settings.
* Rendering is refused when a pool uses a different substrate from its
cluster or conflicts with an existing worker pool owned by another
release.
  * The default cluster hostname now includes the `kubernetes-` prefix.
* Existing cluster configurations that still include Talos settings
receive a warning; those settings are no longer used.

* **Documentation**
* Updated guidance for cluster and worker-pool settings, storage,
upgrades, and version compatibility.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
myasnikovdaniil authored and GitHub committed 2026-11-14 00:34:44 +05:00
commit aa36d3fad6
53 files changed
+686 -713

No files matched your search

@@ -0,0 +1,104 @@
#!/usr/bin/env bats
# -----------------------------------------------------------------------------
# The tenant cluster's pod and service CIDRs are written by two charts:
#
# * packages/apps/kubernetes renders them into the CAPI Cluster's
# spec.clusterNetwork, which the control plane is built from;
# * packages/apps/kubernetes-nodes renders them into every worker's Talos
# machine config (cluster.network.podSubnets / serviceSubnets), inside the
# TalosConfigTemplate the talos-reconcile Job applies.
#
# Each chart carries its own literal, and neither render fails when the two
# disagree: the workers boot, join, and route pod and Service traffic over
# ranges the control plane does not use. This guard renders both charts and
# compares the lists, so a range moved on one side only fails here.
#
# The worker side is read the way the Job produces it: the heredoc is cut out
# of the Job's command, run through sh, and the TalosConfigTemplate it prints is
# parsed, then the machine config inside it. A text grep of the Job would also
# match a comment.
#
# Requires: helm, yq (mikefarah v4+). Runs under both `bats` and the in-repo
# cozytest.sh runner, so TMP is provisioned inline per test and cleaned without
# an EXIT trap.
# -----------------------------------------------------------------------------
load test_helper
REPO_ROOT="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")/.." && pwd)"
# $1: output directory. Writes pods and services, one CIDR per line.
cluster_cidrs() {
helm template kubernetes-myk8s "$REPO_ROOT/packages/apps/kubernetes" \
--namespace tenant-test \
-f "$REPO_ROOT/packages/apps/kubernetes/tests/values-ci.yaml" \
> "$1/cluster.yaml" 2> "$1/cluster.err" \
|| { cat "$1/cluster.err" >&2; return 1; }
yq 'select(.kind == "Cluster" and .apiVersion == "cluster.x-k8s.io/v1beta1") | .spec.clusterNetwork.pods.cidrBlocks[]' \
"$1/cluster.yaml" > "$1/cluster.pods"
yq 'select(.kind == "Cluster" and .apiVersion == "cluster.x-k8s.io/v1beta1") | .spec.clusterNetwork.services.cidrBlocks[]' \
"$1/cluster.yaml" > "$1/cluster.services"
}
# $1: output directory. Writes pool.pods and pool.services, one CIDR per line.
pool_cidrs() {
helm template kubernetes-nodes-myk8s-md0 "$REPO_ROOT/packages/apps/kubernetes-nodes" \
--namespace tenant-test \
--set cluster=myk8s \
--set _cluster.cluster-domain=cozy.local \
--set instanceType="" \
--set resources.cpu=2 --set resources.memory=4Gi \
--show-only templates/talos-reconcile-job.yaml \
> "$1/pool.yaml" 2> "$1/pool.err" \
|| { cat "$1/pool.err" >&2; return 1; }
yq 'select(.kind == "Job") | .spec.template.spec.containers[0].command[2]' "$1/pool.yaml" > "$1/cmd.sh"
awk '
/(^|="?\$\()cat <<EOF/ { print "cat <<EOF"; inblock=1; next }
inblock && /^EOF$/ { print "EOF"; inblock=0; next }
inblock { print }
' "$1/cmd.sh" > "$1/heredoc.sh"
sh "$1/heredoc.sh" > "$1/tct.yaml" 2> "$1/heredoc.err" \
|| { cat "$1/heredoc.err" >&2; return 1; }
yq 'select(.kind == "TalosConfigTemplate") | .spec.template.spec.data' "$1/tct.yaml" > "$1/machineconfig.yaml"
yq '.cluster.network.podSubnets[]' "$1/machineconfig.yaml" > "$1/pool.pods"
yq '.cluster.network.serviceSubnets[]' "$1/machineconfig.yaml" > "$1/pool.services"
}
# $1: directory, $2: pods or services, $3: what to call it in a message.
assert_agree() {
cluster=$(cat "$1/cluster.$2")
pool=$(cat "$1/pool.$2")
# An empty read means a field moved or a render came out short; comparing
# "" with "" would pass on nothing.
if [ -z "$cluster" ] || [ -z "$pool" ]; then
echo "FAIL: could not read the $3 CIDRs from both renders."
echo " kubernetes Cluster.spec.clusterNetwork: '${cluster:-<missing>}'"
echo " kubernetes-nodes machine config cluster.network: '${pool:-<missing>}'"
return 1
fi
if [ "$cluster" != "$pool" ]; then
echo "FAIL: the tenant $3 CIDRs differ between the two charts."
echo " packages/apps/kubernetes (CAPI Cluster): $cluster"
echo " packages/apps/kubernetes-nodes (worker machine config): $pool"
echo "Workers would route $3 traffic over a range the control plane does not"
echo "use. Change both charts together. Moving the worker side also renames"
echo "the content-hashed TalosConfigTemplate and rolls every worker."
return 1
fi
}
@test "kubernetes and kubernetes-nodes render the same pod CIDRs" {
TMP=$(mktemp -d)
cluster_cidrs "$TMP"
pool_cidrs "$TMP"
assert_agree "$TMP" pods pod
rm -rf "$TMP"
}
@test "kubernetes and kubernetes-nodes render the same service CIDRs" {
TMP=$(mktemp -d)
cluster_cidrs "$TMP"
pool_cidrs "$TMP"
assert_agree "$TMP" services service
rm -rf "$TMP"
}
+9 -39
View File
@@ -78,19 +78,16 @@ CATALOG_VALUES="$REPO_ROOT/packages/system/kubernetes-worker-image/values.yaml"
# -----------------------------------------------------------------------------
# Second family of default-drift guards on the same chart: the Talos <-> Kubernetes
# support matrix. The render guard keyed only on `talos.version`, so a pool could
# reach a different Talos release through `osImage.builtin.version` /
# `osImage.factory.version` and boot a kubelet outside its support window with no
# render failure. The table now lives once in templates/_helpers.tpl and both
# paths consult it.
# support matrix. A pool reaches a Talos release through `talos.version` or through
# `osImage.builtin.version` / `osImage.factory.version`, and the table in
# templates/_helpers.tpl has to be consulted for whichever one the pool boots.
#
# The symmetry is structural, not observable from a render: the matrix ships one
# row (`v1.13`) whose Kubernetes list is a superset of `values.schema.json`'s
# `version` enum, so no schema-valid pairing can make either call site fail today.
# A helm-unittest case would have to put an out-of-enum `version` in, and
# helm-unittest validates the schema, so it cannot. These two guards cover what a
# render cannot: that both call sites exist, and that the table still means
# something for the defaults the chart ships.
# The matrix ships one row (`v1.13`) whose Kubernetes list is a superset of
# `values.schema.json`'s `version` enum, so no schema-valid pairing makes either
# call site fail today. hack/kubernetes-nodes-talos-matrix.bats renders the
# refusals with the schema skipped; these two guards cover what it does not: that
# both call sites stay wired, and that the table still means something for the
# defaults the chart ships.
NODES_TEMPLATE="$REPO_ROOT/packages/apps/kubernetes-nodes/templates/nodegroup.yaml"
NODES_SCHEMA="$REPO_ROOT/packages/apps/kubernetes-nodes/values.schema.json"
@@ -148,30 +145,3 @@ NODES_SCHEMA="$REPO_ROOT/packages/apps/kubernetes-nodes/values.schema.json"
esac
done
}
# -----------------------------------------------------------------------------
# Third guard, and the premise the first one rests on. The catalog is keyed off
# the kubernetes-nodes chart's `talos` defaults, but the parent kubernetes chart
# carries its own copy of the same block since the Phase 2 split, and its own
# values.yaml says to keep the two in sync. Nothing enforced that: the pool
# chart's `make update` copies files/versions.yaml and images/kubectl.tag from
# the parent and not the talos block, so a one-sided bump leaves the parent's
# support-matrix guard reasoning about one Talos release while every worker boots
# another — and the catalog check above would go on passing, because it only ever
# reads one of the two files.
PARENT_VALUES="$REPO_ROOT/packages/apps/kubernetes/values.yaml"
@test "the parent kubernetes chart and kubernetes-nodes agree on their Talos defaults" {
for key in version schematicID imageFactoryURL installerRepository registryMirrors; do
parent=$(yq ".talos.${key}" "$PARENT_VALUES")
pool=$(yq ".talos.${key}" "$APP_VALUES")
if [ "$parent" != "$pool" ]; then
echo "talos.${key} differs between the two charts:" >&2
echo " packages/apps/kubernetes/values.yaml: $parent" >&2
echo " packages/apps/kubernetes-nodes/values.yaml: $pool" >&2
echo "both are declared 'keep in sync' and a worker pool follows the second while the parent's support-matrix guard reads the first" >&2
return 1
fi
done
}
+4 -4
View File
@@ -18,7 +18,7 @@ cozy_e2e_storage_class() {
esac
}
# talos_spec_block: emit the tenant Kubernetes CR `spec.talos` block with the Talos
# talos_spec_block: emit the tenant KubernetesNodes CR `spec.talos` block with the Talos
# OS image cache (imageFactoryURL) when the in-sandbox cache is up. Prints nothing
# when it falls back to the public default, so the chart default applies.
# Indented for insertion directly under `spec:` in the heredoc below.
@@ -5702,8 +5702,9 @@ YAML
# Point worker DataVolume imports at the in-sandbox Talos OS image cache when it
# is up, falling back to the public factory otherwise. Worker image pulls use
# the chart's public-registry default. Emitted right under spec: as
# `talos: { imageFactoryURL: ... }`, or nothing when the default applies.
# the chart's public-registry default. Emitted right under spec: of the
# KubernetesNodes pool below as `talos: { imageFactoryURL: ... }`, or nothing
# when the default applies. The Kubernetes CR takes no worker image field.
local talos_block
talos_block=$(talos_spec_block)
@@ -5714,7 +5715,6 @@ metadata:
name: "${test_name}"
namespace: tenant-test
spec:
${talos_block}
${oidc_block}
addons:
certManager:
+3 -3
View File
@@ -49,10 +49,10 @@
# Best-effort: never fail the suite on the band-aid. Remove once tenant workers
# no longer bulk-pull the OS image from the public internet in CI.
local sid ver
sid=$(yq '.talos.schematicID' packages/apps/kubernetes/values.yaml 2>/dev/null)
ver=$(yq '.talos.version' packages/apps/kubernetes/values.yaml 2>/dev/null)
sid=$(yq '.talos.schematicID' packages/apps/kubernetes-nodes/values.yaml 2>/dev/null)
ver=$(yq '.talos.version' packages/apps/kubernetes-nodes/values.yaml 2>/dev/null)
if [ -z "$sid" ] || [ "$sid" = "null" ] || [ -z "$ver" ] || [ "$ver" = "null" ]; then
echo "WARNING: could not read talos.schematicID/version from values.yaml — skipping mirror (fallback to public factory)"
echo "WARNING: could not read talos.schematicID/version from packages/apps/kubernetes-nodes/values.yaml — skipping mirror (fallback to public factory)"
return 0
fi
# The CiliumClusterwideNetworkPolicy in the manifest is intentionally NOT applied
+2 -2
View File
@@ -14,7 +14,7 @@
# This Deployment fetches the image ONCE at e2e-install time with an aggressive
# curl retry loop (decoupling the one flaky external fetch from the many
# per-worker imports), then serves it locally over plain HTTP with range support.
# Tenant `Kubernetes` CRs point `spec.talos.imageFactoryURL` at this Service
# Tenant `KubernetesNodes` CRs point `spec.talos.imageFactoryURL` at this Service
# (see hack/e2e-chainsaw/_lib/run-kubernetes.sh) so every worker imports from the
# warm local copy instead of the public internet. Tenant namespaces run under a
# default-deny Cilium egress installed by the tenant chart, whose allow rules cover
@@ -28,7 +28,7 @@
# mirror can only help, never make CI worse.
#
# The __SCHEMATIC_ID__ / __TALOS_VERSION__ placeholders are substituted at apply
# time by hack/e2e-install-cozystack.bats from packages/apps/kubernetes/values.yaml.
# time by hack/e2e-install-cozystack.bats from packages/apps/kubernetes-nodes/values.yaml.
#
# CI band-aid, not a product artifact. The durable fix is the overridable
# `talos.imageFactoryURL` chart value this manifest exercises; a self-hosted
+81
View File
@@ -0,0 +1,81 @@
#!/usr/bin/env bats
# Unit test: the refusals of the Talos to Kubernetes support-matrix guard in
# packages/apps/kubernetes-nodes (kubernetes-nodes.assertTalosSupportsKubernetes,
# called from templates/nodegroup.yaml for talos.version and again for the
# release osImage resolves to).
#
# On the shipped chart a refusal is unreachable: the values schema offers only
# Kubernetes versions the v1.13 row lists, and files/versions.yaml, which another
# template fails on first, holds the same set. The guard is there for the day a
# Talos row stops listing a version the chart still offers. So this renders a
# copy of the chart with --skip-schema-validation and one extra versions.yaml row
# for a version the matrix does not list. The pairings the guard lets through
# are in the chart's own tests/talos_k8s_support_matrix_test.yaml.
load test_helper
REPO_ROOT="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")/.." && pwd)"
CHART="$REPO_ROOT/packages/apps/kubernetes-nodes"
# render_pool <args...> renders the pool chart copy with the given --set flags and
# prints helm's output, errors included. The copy lives in the test's own tmpdir.
render_pool() {
copy="$BATS_TEST_TMPDIR/chart"
if [ ! -d "$copy" ]; then
cp -RL "$CHART" "$copy"
printf '"v1.30": "v1.30.14"\n' >> "$copy/files/versions.yaml"
fi
helm template kubernetes-nodes-myk8s-md0 "$copy" --namespace tenant-test \
--skip-schema-validation \
--set cluster=myk8s \
--set _cluster.cluster-domain=cozy.local \
--set minReplicas=1 --set maxReplicas=3 \
--set instanceType= --set resources.cpu=2 --set resources.memory=4Gi \
--set diskSize=20Gi --set storageClass=replicated \
"$@" 2>&1
}
@test "an in-window pairing renders on the same copy, so the refusals below are the guard's" {
out=$(render_pool --set version=v1.31) || { echo "$out" >&2; return 1; }
printf '%s\n' "$out" | grep -q '^kind: KubevirtMachineTemplate$'
}
@test "a Kubernetes version outside the row of talos.version is refused" {
if out=$(render_pool --set version=v1.30); then
echo "render succeeded with Kubernetes v1.30 on the default Talos" >&2
return 1
fi
printf '%s\n' "$out" | grep -qF "Kubernetes v1.30 is not supported by Talos v1.13.6. Supported versions: [v1.31 v1.32 v1.33 v1.34 v1.35 v1.36]. Bump 'version' or set talos.version explicitly." \
|| { echo "$out" >&2; return 1; }
}
@test "an older patch release of a listed Talos minor is checked against the same row" {
if out=$(render_pool --set version=v1.30 --set talos.version=v1.13.0); then
echo "render succeeded with Kubernetes v1.30 on Talos v1.13.0: the guard is not keyed on the minor" >&2
return 1
fi
printf '%s\n' "$out" | grep -qF 'Kubernetes v1.30 is not supported by Talos v1.13.0.' \
|| { echo "$out" >&2; return 1; }
}
# talos.version names a minor the matrix does not list, so the first call lets the
# pool through and only the call fed the osImage release can refuse it.
@test "a Kubernetes version outside the row of osImage.factory.version is refused" {
if out=$(render_pool --set version=v1.30 --set talos.version=v1.12.0 --set osImage.factory.version=v1.13.0); then
echo "render succeeded with Kubernetes v1.30 on Talos v1.13.0 reached through osImage.factory" >&2
return 1
fi
printf '%s\n' "$out" | grep -qF 'Kubernetes v1.30 is not supported by Talos v1.13.0.' \
&& printf '%s\n' "$out" | grep -qF 'nodeGroup md0 reaches this Talos release through its osImage override' \
|| { echo "$out" >&2; return 1; }
}
@test "a Kubernetes version outside the row of osImage.builtin.version is refused" {
if out=$(render_pool --set version=v1.30 --set talos.version=v1.12.0 --set osImage.builtin.version=v1.13.0); then
echo "render succeeded with Kubernetes v1.30 on Talos v1.13.0 reached through osImage.builtin" >&2
return 1
fi
printf '%s\n' "$out" | grep -qF 'Kubernetes v1.30 is not supported by Talos v1.13.0.' \
&& printf '%s\n' "$out" | grep -qF 'nodeGroup md0 reaches this Talos release through its osImage override' \
|| { echo "$out" >&2; return 1; }
}
+4 -4
View File
@@ -2,7 +2,7 @@
# -----------------------------------------------------------------------------
# Unit tests for talos_spec_block in hack/e2e-chainsaw/_lib/run-kubernetes.sh
#
# It is the composition point for the tenant Kubernetes CR's `spec.talos`: the
# It is the composition point for the tenant KubernetesNodes CR's `spec.talos`: the
# Talos OS image cache contributes imageFactoryURL when available. The result is
# spliced into an unquoted heredoc directly under `spec:`, so an indentation slip
# does not fail this function -- it produces a CR the API rejects, and every
@@ -32,7 +32,7 @@ load test_helper
# Splice a talos_spec_block result under `spec:` of a minimal CR and print the
# document, so yq can be asked what the tenant apiserver would see.
spec_doc() {
printf 'apiVersion: apps.cozystack.io/v1alpha1\nkind: Kubernetes\nmetadata:\n name: t\nspec:\n%s\n host: ""\n' "$1"
printf 'apiVersion: apps.cozystack.io/v1alpha1\nkind: KubernetesNodes\nmetadata:\n name: t\nspec:\n%s\n cluster: t\n' "$1"
}
@test "no Talos image cache renders nothing, so the chart default applies" {
@@ -59,6 +59,6 @@ spec_doc() {
# talos_spec_block writes imageFactoryURL by hand. If the chart ever renames
# it, the CR would be rejected only at e2e time, and this file is the cheapest
# place to notice.
yq -e '.talos | has("imageFactoryURL")' packages/apps/kubernetes/values.yaml >/dev/null \
|| { echo "packages/apps/kubernetes/values.yaml has no talos.imageFactoryURL; talos_spec_block would emit a key the chart rejects" >&2; exit 1; }
yq -e '.talos | has("imageFactoryURL")' packages/apps/kubernetes-nodes/values.yaml >/dev/null \
|| { echo "packages/apps/kubernetes-nodes/values.yaml has no talos.imageFactoryURL; talos_spec_block would emit a key the pool chart ignores" >&2; exit 1; }
}