Commit Graph
77 Commits
Author SHA1 Message Date
myasnikovdaniil 4087218a02 docs: move the repo-only guides onto the website (#5034)
## What this PR does

Seven guides under `docs/` were operator facing and written alongside
the code, and left in the source tree where no user looks. They are on
the website now, so this deletes them here and repoints every reference.

```
docs/operations/backup-classes.md                  -> operations/services/backup-classes
docs/operations/resource-presets.md                -> guides/resource-management
docs/storage-immutability.md                       -> operations/storage-class-immutability
docs/operations/system-memory-limits.md            -> operations/configuration/system-memory-limits
docs/gateway-classes.md                            -> networking/gateway-classes
docs/observability/distributed-tracing.md          -> operations/services/monitoring/distributed-tracing
docs/operations/seaweedfs-capacity-for-backups.md  -> operations/services/object-storage/capacity-for-backups
```

`docs/release.md`, `docs/deprecating-applications.md` and the two
SeaweedFS runbooks in `docs/operations/` stay. A workflow, `AGENTS.md`,
a migration and a bats suite read them by path, which is the test from
#3777.

References point at `https://cozystack.io/docs/v1/...`. The site
rewrites that short prefix to the latest version in `layouts/404.html`,
so the links follow releases and pin none. Backup Classes, resource
management and `storageClass` immutability exist in the latest version
already. The other four pages are only in `next/` until 1.7.0 is
promoted, and their links start to resolve then. The generated mongodb
files come from `values.yaml` through `make generate` with
cozyvalues-gen v1.7.0, the version pinned in
`.github/workflows/pre-commit.yml`.

Tests: helm unit tests for mongodb (151), postgres (209), vm-disk (114)
and kubernetes-nodes (223), go tests for backupcontroller, presets and
the operator, rd-presets-check, the migration-50 bats suite.

### Downstream repositories

- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#763

Walked the rest of the trigger map against the diff. No provider
follow-up: the mongodb `values.schema.json` diff is one `description`
string, with no field added, removed or renamed, no enum, no default,
and no `kind`, `plural` or `release.prefix` change.
`packages/core/platform/values.yaml` changed in comments only and no
key, so the hand written platform-package table on the site is
unaffected.

Please merge after cozystack/website#763, the links in the code only
resolve once those pages deploy.

### Release note

```release-note
docs: seven guides move from the docs/ directory of this repository to cozystack.io (Backup Classes, resource presets, storageClass immutability, system component memory limits, GatewayClass selection, distributed tracing, SeaweedFS capacity for backups), and in-repo references point there.
```
2026-11-03 23:09:46 +05:00
Aleksei Sviridkin 3abdff5900 fix(kubernetes-nodes): refuse a partial resources spec on every pool (#4973)
The `resources.cpu`/`resources.memory` pair check in `nodegroup.yaml`
ran only when an `instanceType` was set, while the field description
(values.yaml, README, schema, the KubernetesNodes OpenAPI schema)
promised a render-time error for a partial spec on any pool. Without an
instancetype a cpu-only pool renders a VM with no memory, which KubeVirt
refuses when the Machine is created, and a memory-only pool silently
sizes the guest at KubeVirt's default of one vCPU.

- The check now runs on every kubevirt pool. The proxmox branch already
required both, so this is the kubevirt half of one rule, and the field
descriptions are true as written (no doc change). On a pool with no
`instanceType` the message ends in "or neither and set instanceType", so
it does not send the user straight into the next refusal.
- A kubevirt pool with neither an `instanceType` nor both resources
fails at render with a message naming both, the way `vm-instance` does.
No such pool could have passed admission (no memory), so this refusal
breaks nothing that runs today. This is the decision the issue asked
for; if the gate should stay narrower, this commit can be dropped on its
own.
- **Upgrade impact.** One shape that renders on main stops rendering:
`instanceType: ""` with only `resources.memory`. On main it gets
`domain.memory.guest` and no `domain.cpu`, so KubeVirt runs the worker
with one vCPU; with this PR it fails with the partial-spec message, and
the pool's HelmRelease sits in a failed upgrade until the values are
edited. To keep the current size such a pool needs `resources.cpu: "1"`.
Adding that line renames the machine template (for the test values,
`kubernetes-myk8s-md0-28d812` becomes `kubernetes-myk8s-md0-1ce103`, the
same name on main and on this branch), so the pool's workers are
replaced once. cpu-only with `instanceType: ""` is not in this class: it
renders a VM with no memory, which KubeVirt refuses, so it never ran.
- `cores:` printed an unquoted (float64) core count of one million or
more in exponent form (`1e+06`). It is printed as an integer now, with
the same bytes for every smaller whole number.

Tests: `tests/resources_partial_test.yaml` covers both halves with and
without an instancetype, the no-sizing case, both fields set, and the
float64 formatting. The existing unquoted-zero case in
`resources_cpu_test.yaml` now reads the field as unset on an
instancetype-sized pool (with a `kubernetesProvider` stub for the
lookup), and zero beside memory is pinned as a partial spec.

Byte stability, checked by snapshotting the `KubevirtMachineTemplate` on
main for nine spellings of `resources.cpu` and replaying the snapshots
on this branch: `"2"`, `2`, `2.0`, `"2.0"`, `"+2"`, `"1e1"`, `"010"`,
`999999` render identical bytes and template names. Only unquoted
`4294967295` changes, from `cores: 4.294967295e+09` to `cores:
4294967295`.

Verified with `helm unittest .` in `packages/apps/kubernetes-nodes` (38
suites, 233 tests, 12 snapshots), and on a trial merge with current main
(39 suites, 237 tests).

Closes #4546. The leading-zero inconsistency from the issue's comment is
addressed separately, together with #4554.

### Release note

```release-note
fix(kubernetes-nodes): a partial `resources` spec (only `cpu` or only `memory`) is now refused at render time on every kubevirt worker pool, not only on a pool with an `instanceType`, and a pool with neither an `instanceType` nor both resources is refused as well. Action required: a pool with `instanceType: ""` and only `resources.memory` rendered before (KubeVirt ran it with one vCPU) and now fails to upgrade; add `resources.cpu: "1"` to keep its current size. Adding that line renames the pool's machine template, so its workers are replaced once. Whole-number core counts of one million or more are now printed as integers instead of in exponent form.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* KubeVirt worker pools now require either an instance type or both CPU
and memory values. Configurations with only one resource or no sizing
information are rejected.
* Whole-number CPU values render without exponent notation, including
values written with a decimal fraction.
* An unquoted zero CPU value is treated as unset when an instance type
is provided; without an instance type, pairing zero CPU with memory
fails validation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-29 11:22:48 +03:00
Marian Koreniuk f7913329b2 fix(kubernetes-nodes): plan autoscaler capacity on the cores the VM actually gets
On the kubevirt substrate resources.cpu is written into domain.cpu.cores
as spelled and decoded as YAML, so "010" gives the VM eight cores, while
the cluster-autoscaler capacity annotation parsed the same string as
decimal ten. The autoscaler then planned 9000m on a node that offers
7000m.

Keep the YAML-decoded value the range check already computes and start
the capacity annotation and the reserved-CPU check from it on kubevirt.
The kubelet reservations stay as they are: the worker machineconfig
sizes them from the decimal string, and the TalosConfigTemplate name is
a hash of that config, so changing them would roll every leading-zero
pool once. The annotation therefore subtracts the reservations the
machineconfig really carries (2 x 500m for "010") from the cores the VM
really has, which is the node's real allocatable. Neither template name
changes for any spelling, so no pool rolls. Proxmox parses the quantity
as decimal on both sides and is unchanged. The field description says
the value is decoded as YAML and that the reservations still read it as
a decimal.

This closes #4554 without the migration and the render-time refusal it
proposes: a refusal breaks stored pools on their next upgrade, a schema
pattern breaks updates of existing CRs, and a platform-wide migration of
every KubernetesNodes and nodeGroups entry is a lot of blast radius for a
spelling nobody has reported in a live cluster.

Closes #4554

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-10-28 21:04:27 +01:00
Marian Koreniuk 0e77e2b10f fix(kubernetes-nodes): name instanceType in the partial-spec refusal
On a pool with no instanceType, "set both resources.cpu and
resources.memory, or neither" sent the user straight into the next
refusal, since neither sizing source is refused as well. The message now
ends in "or neither and set instanceType" on such a pool; a pool sized by
an instancetype keeps the old wording.

The tests anchor both spellings, so the suffix cannot leak onto, or drop
off, either kind of pool.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-10-28 17:06:39 +01:00
Andrei Kvapil 1636349a63 fix(api): point the removed image overrides at node registry mirrors
The warning for a removed image override told an air-gapped install to
use the platform-wide registry, but nothing in the platform writes
_cluster.images-registry, so there is nothing to set. What serves these
images offline is what serves every other platform image: the registry
mirrors configured on the nodes, as the air-gapped install guide sets
up. The tests that hold the cozy-lib.image routing now say what they
check rather than promising a platform knob.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-10-23 19:14:16 +05:00
Andrei Kvapil dfff5c2dac test(kubernetes): hold the routing of the busybox and unpin images
The busybox helper shared by four workloads and the pool's pre-delete
Job had no test holding their route through cozy-lib.image, so
reverting either line left both suites green.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-10-23 19:14:16 +05:00
Andrei Kvapil d99a1b9f2f fix(kubernetes): route the helper images through cozy-lib.image
These images now go through cozy-lib.image like the harbor, kafka and
mariadb hook images, so they follow _cluster.images-registry wherever
that is set rather than any per-application value. An air-gapped
install serves them the way it serves every other platform image,
through the registry mirrors configured on its nodes.

The talos-csr-signer sidecar and the OIDC Jobs had no test holding
their image, which is where an override could come back unnoticed.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-10-23 19:14:15 +05:00
Andrei Kvapil 941b50c675 fix(kubernetes-nodes)!: drop the kubectl image override
images.kubectl picked the image of the talos-reconcile and pre-delete
unpin Jobs. Both run in the management cluster under service accounts
that read the pool's Talos and cluster CA secrets and patch its CAPI
objects, so a tenant-chosen image ran with access the apps API never
grants. The image now always comes from images/kubectl.tag.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-10-23 19:14:14 +05:00
Myasnikov Daniil b83cc087cc docs: move the repo-only guides onto the website
Seven guides in docs/ were written next to the code and left in the
source tree, where no operator looks. The website carries them now, so
a second copy here only lets the two drift again: the Backup Classes
page had already fallen behind its original.

docs/release.md, docs/deprecating-applications.md and the two SeaweedFS
runbooks in docs/operations stay. A migration, a bats suite, a workflow
or AGENTS.md reads each of them by path, which is the test for keeping
a document in this repository.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-21 17:50:46 +05:00
Marian Koreniuk 1211095693 fix(kubernetes-nodes): refuse a partial resources spec on every pool
The resources.cpu/resources.memory pair check ran only when an
instanceType was set, while the field description promised a render-time
error for a partial spec on any pool. Without an instancetype a cpu-only
pool renders a VM with no memory, which KubeVirt refuses when the Machine
is created, and a memory-only pool silently sizes the guest at one vCPU.

Run the check on every kubevirt pool; the proxmox branch already requires
both. A kubevirt pool with neither an instanceType nor both resources now
fails at render with a message naming both, the way vm-instance does; no
such pool ever passed admission, so nothing that runs is affected.

A float64 core count of one million or more printed in exponent form,
which the CRD's integer field cannot take. Print it as an integer; every
smaller whole number prints the same bytes, so no template name moves.

Closes #4546

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-10-19 23:29:09 +02:00
Marian Koreniuk bbd39c94a4 feat(kubernetes-nodes): build the Proxmox worker template from the pool's Talos values
A proxmox pool clones its workers from a VM template the chart never sees,
and nothing said which Talos image that template has to carry. The one an
operator reaches for, the platform release's nocloud-amd64.raw.xz, is the
bare-metal node image: its zfs extension's service waits for a /dev/zfs no
VM has, the boot sequence never completes, and Talos reboots every worker
each time the boot timeout runs out, about every 70 minutes on v1.13. It
also follows the host nodes' Talos version, not the pool's talos.version
that the installer in the worker machineconfig names.

The chart's default schematic has a trap of its own: its qemu-guest-agent
service waits the same way for the agent's virtio port, which Proxmox adds
only to a VM with the agent enabled.

hack/proxmox-talos-template.sh builds the template from the chart's
talos.schematicID and talos.version in the nocloud flavour with the agent
enabled, refuses a tag set another template already carries, because
capmox then finds two and clones neither, and prints the templateTags
that select it.

Assisted-by: LLM
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
2026-10-11 01:23:10 +02:00
Fedor Batonogov 68cd1028bd feat(kubernetes-nodes): let CDI trust a self-hosted Talos image factory
talos.imageFactoryURL is documented for self-hosted factories and mirrors
in air-gapped, rate-limited or flaky-egress environments. Such an endpoint
is normally served with a certificate from an internal CA, and CDI — which
trusts only its system store — refuses the import with "x509: certificate
signed by unknown authority". The DataVolume never completes, no worker
joins, and the release sits in install until it times out.

DataVolume.spec.source.http.certConfigMap exists for exactly this, but the
chart rendered source.http.url alone and no value reached the field.

Add an optional talos.imageFactoryCA taking the CA in PEM form. When set,
the chart renders it into a ConfigMap in the release namespace — CDI
requires certConfigMap to sit alongside the DataVolume — and references it
from the worker DataVolume on the HTTP import path; a pool cloning a
golden image via osImage.builtin builds no source.http and renders
neither. When empty, nothing is rendered and the output is byte-for-byte
what it was.

CDI appends the bundle to the system pool rather than replacing it, so a
release pulling from a publicly-trusted endpoint keeps verifying either
way.

The value is emitted verbatim into a ConfigMap readable by anything that
can read the namespace, so the render fails closed on private key material
and on anything that is not complete PEM certificate blocks. Those guards
live in a new validate-only cozy-lib.tls.validateCACert helper that takes
the caller's field name for its messages, and cozy-lib.tls.caCertSecret is
refactored onto it, so the check exists in exactly one place.

Assisted-by: LLM
Signed-off-by: Fedor Batonogov <fedor@batonogov.ru>
2026-10-10 22:36:49 +03:00
IvanHunters ec4d26df54 fix(kubernetes): accept a golden worker image that exists only as a PVC (#4578)
## What this PR does

A worker pool on `osImage.builtin` refused a golden image that exists in
`cozy-public` as a PVC without its DataVolume, and the whole
KubernetesNodes release failed to render. CDI clones the PVC, not the
DataVolume, so that golden is a valid source, and vm-disk already
accepts it. This can happen once CDI garbage-collects DataVolumes
(`dataVolumeTTLSeconds`) or when someone restores a golden PVC on its
own.

Now the render fails only when neither the DataVolume nor the PVC
exists. The StorageClass and size checks already fall back to the PVC,
so a PVC-only golden is still checked against the pool, and the new test
suite pins that too.

Closes #4565

### Screenshots

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(kubernetes): a worker pool on osImage.builtin now accepts a golden image that exists in cozy-public only as a PVC, matching vm-disk; it is refused only when neither the DataVolume nor the PVC exists.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Built-in golden images can now be used when available as either a
DataVolume or a PVC.
* **Bug Fixes**
* Missing golden images now produce an error that identifies the missing
image and suggests ways to resolve the issue. The error is suppressed
when the pool already has a successful clone.
* PVC-based golden images are checked against the pool’s storage class
and minimum disk size requirements.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:45:50 +03:00
Aleksei Sviridkin 4583ee595f fix(kubernetes-nodes): refuse a resources.cpu that cannot be a core count
On the kubevirt substrate a worker pool's resources.cpu, although
schema-typed as a Kubernetes quantity, is written verbatim into
domain.cpu.cores, which KubeVirt declares as uint32 and the
KubevirtMachineTemplate CRD as an integer. A value such as 500m or
"1.5" therefore rendered a template the API server refuses, and
because the same raw string was truthy it also counted as explicit
sizing and dropped the pool's instancetype matcher first. A whole
count past the uint32 ceiling rendered too, although KubeVirt cannot
hold it.

The render now decodes the value the way the emitted cores field is
decoded and stops unless that is a whole number between 1 and the
uint32 maximum, before the value reaches the VM template or the
precedence decision. Only inputs that could not produce a working
pool are refused. Every value that rendered one still renders the
same bytes, so no pool rolls: a bare numeric zero stays falsy and
means unset, and a leading-zero spelling is still emitted as written.

The proxmox substrate is left out of the check, because it converts
the quantity to millicores and rounds up to whole cores, so 500m is a
valid size there. The field description now says what the value
means on each substrate.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 22:10:40 +03:00
Mattia Eleuteri 9afe7f1293 fix(kubernetes): assemble the node group once in kubernetes-nodes
The TalosConfigTemplate name is a hash of the machine-config spec, and two
templates have to agree on it: talos-reconcile-job.yaml creates the object,
nodegroup.yaml points MachineDeployment.bootstrap.configRef at it. Sharing the
rendering was not enough — each file also assembled its own $group dict, one
carrying every pool value and one only the few the spec reads, so the two
names agreed only by coincidence of the keys the spec helper happens to read.
Dropping "gpus" from the Job's dict diverged them on a GPU pool with the whole
suite still green, which is the "templates do not exist" deadlock this fix
exists to make impossible.

Move the assembly into a single kubernetes-nodes.group helper, read back with
fromYaml at both call sites, so there is one input as well as one rendering.

Cover it too, since the seam can be reintroduced. The chart's helm-unittest
suites pin each name against its own literal in its own file, which catches a
divergence but reports it as one literal having moved — indistinguishable from
a stale fixture, so re-pasting the new value is the obvious and wrong repair.
hack/check-talosconfigtemplate-name-agreement.bats renders the chart once and
compares the two values against each other, so the failure states the invariant
instead of a number. It asserts on a GPU-shaped pool as well as the default
one: the default renders gpus [], kubelet {} and instanceType "" as falsy, so a
consumer reading a subset of the group is indistinguishable there from one
reading all of it. kernelModules gets a case of its own, because it has no
default and stays unset on both of those shapes. A last case pins that the
shapes hash differently, so the others cannot hold vacuously.

The same guard compares the talos-reconcile Role's resourceNames on
TalosConfigTemplates against that name. The Job applies server-side, and
that is a PATCH on the name even when it creates the object, so a Role
naming anything else leaves the Job Forbidden and the pool without a
template, with the same silent outcome.

Document that talos.registryMirrors and kernelModules sit inside the hashed
spec, so changing either rolls the pool's existing workers instead of only
affecting nodes created later. The kernelModules text described the
fixed-name template as the present state and the content-hash name as a
future one.

No rendered output changes: the golden render snapshots are unchanged.

Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Assisted-by: LLM
2026-09-30 08:59:03 +02:00
Mattia Eleuteri 200226686e fix(kubernetes): name TalosConfigTemplate by content hash
TalosConfigTemplate.spec is immutable — vtalosconfigtemplate.cluster.x-k8s.io
denies any mutation with "TalosConfigTemplate.Spec is immutable" — but the
talos-reconcile Job applied it under a fixed <release>-<group> name on every
reconcile. Once the template existed, no change to the rendered worker machine
config could reach the cluster: the apply was rejected server-side while the
HelmRelease stayed Ready. Observed with machine.network.searchDomains after a
clusterDomain correction; the same held for certSANs, machine.registries and
nameservers.

Name the template after a 6-char hash of its rendered spec, the idiom
KubevirtMachineTemplate already uses in the same chart, and point
MachineDeployment.spec.template.spec.bootstrap.configRef at that name. A content
change now creates a new template instead of attempting a rejected mutation, and
CAPI rolls the workers onto it.

The spec is rendered from one named template shared by the Job (which embeds it
in its heredoc) and the MachineDeployment (which hashes it into configRef), so
the two cannot compute different names. The runtime values the Job substitutes
(CA certificates, Service ClusterIP, tokens) stay shell placeholders and are not
hashed, so a certificate rotation does not roll workers.

The talos-reconcile Role pins its TalosConfigTemplate get, patch, update
and delete rights to the pool's own template, so it names the hashed
template too. A server-side apply of a template that does not exist yet
is still a PATCH on that name: a rule left on <release>-<group> would
leave the Job Forbidden and the pool with no template to scale onto. The
replace-on-immutable path is kept and pointed at the same name, because
the runtime values sit outside the hash: a new apiserver address still
renders a different spec under an unchanged name.

Upgrade note: existing clusters roll their Talos workers once, because the
configRef moves from <release>-<group> to the hashed name and CAPI creates a new
MachineSet for it. The rendered machine config itself is unchanged by this
commit. The pre-existing fixed-name template is left in place — the outgoing
MachineSet still references it — and is garbage-collected with the
KamajiControlPlane when the cluster is deleted.

A pool sized by instanceType rather than by explicit resources has a
second roll trigger. Its kubelet reservations are resolved through a live
lookup of the VirtualMachineClusterInstancetype, so that object's size is
part of the hashed spec even though the chart does not own it: a
platform-side change to the instancetype rotates the name and rolls the
pool again. That is the mechanism working as intended - the reservations
really did change - but it is a roll the operator did not initiate, and
the KubevirtMachineTemplate hash this mirrors does not have the property,
since it embeds the instancetype by name and never its resolved size.

Fixes #3515

Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Assisted-by: LLM
2026-09-30 08:52:36 +02:00
Marian Koreniuk e5dc086338 fix(kubernetes-nodes): replace the TalosConfigTemplate CABPT's webhook will not let us patch
CABPT's validating webhook rejects any change to TalosConfigTemplate.spec
(TalosConfigTemplate.Spec is immutable), so the worker reconcile Job's
`kubectl apply` fails the moment any input the render depends on genuinely
changes. The apiserver address is the one that moves in practice: the Job
writes it into certSANs and into the worker's endpoint, and a cluster that
gets a new address leaves the Job dead with

  TalosConfigTemplate.bootstrap.cluster.x-k8s.io "…" is invalid:
  spec: Invalid value: …: Spec is immutable

from then on, until somebody deletes the object by hand. Nothing in the
cluster says that is what has to happen.

The Job now recognises that one failure and replaces the object: delete, then
apply the same manifest. Every other apply error still stops the Job, because a
delete followed by a create that fails for the same reason — an apply rejected
for its content, say — would leave the pool with no template, unable to scale
until the render is fixed. The replace itself does not roll the pool: Machines
reference the template by name, and the name does not change. The match is on
the message for that reason, and the Role's new `delete` is bound to this pool's
own template by name, so the Job cannot reach a sibling pool's.

While the Role is being touched anyway, its other TalosConfigTemplate rights are
pinned to the same name. `get`, `patch` and `update` were unnamed, and several
clusters can share a tenant namespace: a Job that can patch a sibling pool's
template can rewrite the machineconfig that pool's next workers boot. `create`
stays unnamed because RBAC matches resourceNames only on verbs that name an
object.

The manifest is captured into a variable rather than written to a file: the
container runs with readOnlyRootFilesystem, and one variable also means the
replace applies exactly what the first attempt did.

Both applies stay server-side, as the Job's single apply already is: a
client-side merge pulls the full OpenAPI schema and runs the Job out of its
256Mi. The webhook rejects a spec change the same way under server-side apply,
so the `is immutable` match still fires.

UPGRADE NOTE. This changes the KubeVirt path as well, and the render with it:
the talos-reconcile Role gains the `delete` rule, its TalosConfigTemplate rights
are narrowed to this pool's own object, and the Job's script changes,
which moves the Job's content-hash name. Every existing worker pool therefore
gets a new reconcile Job on upgrade. On a pool whose live TalosConfigTemplate
no longer matches the current render, that Job will now delete and recreate it
instead of failing. Machines already running are unaffected — their bootstrap
data was rendered when they joined — but Machines created afterwards boot the
new machineconfig. On a pool where template and render agree, nothing happens.

Both new asserts were checked against the mutation they exist for: deleting the
`exit 1` from the `*)` branch and pointing the delete at `${RELEASE}` each turn
the matching test red. A bare `exit 1` pattern does not — the input-timeout path
near the top of the script already has one.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-29 21:09:09 +02:00
Marian Koreniuk d902421f47 feat(kubernetes): deploy the Proxmox cloud-controller-manager with the cluster
On this substrate the worker kubelet runs with --cloud-provider=external, which
taints every node node.cloudprovider.kubernetes.io/uninitialized until a cloud
provider clears it. Without a cloud-controller-manager the taint is permanent:
nothing but tolerating DaemonSets schedules, and the CSI node plugin — which
reads topology.kubernetes.io/region and zone off its own Node and exits fatally
without them — never starts.

It runs in the management cluster for the same reason as the CSI controller
beside it: it authenticates to the hypervisor, and the tenant is cluster-admin
of its own cluster. Its whole job is to write on the tenant's Nodes, and it
does that over the tenant's admin kubeconfig from out of cluster, which also
means there is no tenant-side RBAC left to install.

capmox mode, not the default one. The default resolves a node to its VM by name
and writes providerID as proxmox://<node>/<vmid>, while capmox reads the CAPI
Machine and writes proxmox://<SMBIOS UUID> — the form capmox itself puts on the
Machine. With the default the Node and its Machine carry two different
identifiers for the same VM.

The tolerations are spelled out rather than a blanket `operator: Exists`. That
one also matches node.kubernetes.io/unreachable:NoExecute with no
tolerationSeconds, which stops admission adding the bounded 300s one and lets
the Pod outlive the Node it runs on.

`proxmox.insecure` defaults to false: the controllers verify the hypervisor's
certificate unless an operator turns that off, and both run here, so the value
describes what the management cluster trusts, not the tenant. The README gains
a section on what the two tokens can reach — the controller acts on whatever
VMID and volume handle the tenant's objects name — and on the per-tenant
Proxmox scope that contains it: a storage of the tenant's own and one ACL on
the resource pool proxmox.pool puts its workers in, which follows every VM
capmox creates — VM.Audit there for the CCM too, which deletes a NotReady Node
whose VM it cannot see.

ccm, csi and insecure are optional in the schema. Each has a default, and what
the substrate needs is enforced at render time — a proxmox cluster without
ccm.credentialsSecretName does not render — so a Kubernetes object written
before these fields existed stays valid, and adding them is not a breaking
change to the API.

The kubelet flag is asserted on both substrates: present on proxmox, absent on
kubevirt, where nothing would ever clear the taint it causes.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-29 21:04:05 +02:00
Marian Koreniuk fc729299f4 feat(kubernetes-nodes): put Proxmox workers into a resource pool
The privileges a Proxmox token needs — VM.Audit and VM.Config.Disk for the CSI
controller, VM.Audit and VM.GuestAgent.Audit for the cloud-controller-manager —
are granted on an ACL path, and the only path that follows every VM capmox
creates, replacements included, is a resource pool. Without one the choices are
`/`, which reaches every VM on the hypervisor, or each VMID, which has to be
granted again after every scale-up and every Machine replacement, and until it
is the new worker gets no disks and no providerID.

proxmox.pool names the pool; capmox adds each worker to it as it clones the VM
(ProxmoxMachineTemplate.spec.template.spec.pool). The pool has to exist on the
hypervisor — capmox does not create it — and every node pool of a tenant cluster
sets the same one, or a worker outside it is a worker the tenant's tokens cannot
reach. Empty adds the workers to no pool, which is what the chart did before.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-29 21:04:05 +02:00
Aleksei Sviridkin 52aaffb5bd fix(kubernetes): accept a golden worker image that exists only as a PVC
CDI clones the golden's PVC, not its DataVolume, so a golden whose
DataVolume was garbage-collected (dataVolumeTTLSeconds) or whose PVC was
restored on its own is still a usable clone source. vm-disk already
accepts one; a worker pool refused it and failed the whole release.

The absence guard now fires only when neither object exists. With no
DataVolume the StorageClass and size guards already fall back to the
golden's PVC, so a PVC-only golden is still checked against the pool.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-29 18:50:16 +03:00
Mattia Eleuteri 91c87b84cd fix(kubernetes): refuse non-ASCII kernelModules parameters
RE2's \s and [:cntrl:] only cover ASCII, so the parameter deny class
passed U+2028 and U+2029. toYaml writes either one raw, and Helm's YAML
parser reads it as a line break that ends the reconcile Job's command
block scalar early. The render then failed on an unparseable manifest
rather than on the guard, which is the last check before the unquoted
heredoc.

Pinning parameters to printable ASCII closes every non-ASCII line break
at once, and the deny class keeps refusing what the heredoc expands.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-09-29 17:35:49 +02:00
Mattia Eleuteri f7c04b7a39 feat(kubernetes): expose kernel modules for Talos worker pools
A Talos system extension installs a kernel module but does not load it;
loading is machine.kernel.modules' job. The worker machine config had no
such block and no values key could add one, so a GPU pool on Talos
workers was unusable as shipped: the extension supplied the NVIDIA
module, nothing loaded it, and the failure was silent end to end. The VM
held the PCI device, the node advertised no GPU, and no component logged
an error.

The field is three-state, and the states are distinguishable because it
carries no default: absent lets the chart choose (a pool holding at least
one nvidia.com/* GPU gets nvidia, nvidia_uvm, nvidia_drm, nvidia_modeset,
in that order, since Talos loads the list in sequence and the last three
depend on the first; any other pool gets nothing), a non-empty list
replaces that choice outright, and an explicit [] opts out even on a GPU
pool. A default of [] would collapse absent into opted-out for every pool
and make the automatic set unreachable.

The automatic nvidia entry carries NVreg_NvLinkDisable=1. Individual
GPUs are passed into worker VMs without their NVSwitches, and on
non-Talos workers the gpu-operator driver container applies that
parameter; on Talos that container has to be off, so nothing else would.

Names and parameters reach the reconcile Job's unquoted heredoc, so they
are validated at render and the emitted list is rebuilt from those two
fields alone: the schema does not prune unknown keys, and any other key
passed through toYaml would reach the shell unchecked.

A pool that resolves to no modules renders the machine config it
rendered before, byte for byte, so its content-hash Job name does not
rotate.

Refs: #3563
Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-09-29 17:09:42 +02:00
Mattia Eleuteri 1728b49e0c fix(kubernetes): apply worker TalosConfigTemplate server-side
The talos-reconcile Job runs under a 256Mi limit and applies its
TalosConfigTemplate client-side. Building the merge patch means pulling
the cluster's full OpenAPI schema down, which on a management cluster
carrying a few hundred CRDs costs around 290Mi of RSS, so the kernel
OOMKills kubectl before the Job can finish.

Only the merge path pays that cost. The first run creates the object and
stays cheap, so a fresh install succeeds and the Job then
CrashLoopBackOffs on every reconcile afterwards, which means on every
chart upgrade. Measured against a 438-CRD cluster with the Job's own
kubectl (alpine/k8s:1.33.4), cold discovery cache:

  get -o jsonpath                    52Mi
  apply, object absent (create)      52Mi
  apply, object present (merge)     292Mi   <- OOMKilled at 256Mi
  apply --server-side                56Mi

--validate=false changes nothing, so the cost is the merge strategy
lookup rather than validation.

Server-side apply leaves the merge to the apiserver and the client never
needs the schema. --force-conflicts keeps the Job authoritative over an
object an earlier client-side apply created.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-09-29 14:38:56 +03:00
Marian Koreniuk 8722462066 fix(kubernetes-nodes): refuse osImage on proxmox instead of honouring half of it
#4171 gave a worker pool an osImage, choosing where its system disk comes from:
a CDI clone of a shared golden image, or an HTTP import from an Image Factory.
A Proxmox worker's disk comes from neither. capmox clones a VM template that
already carries Talos, and nothing in this chart can change what that template
holds.

Half of the field would still appear to work, which is why it is refused rather
than ignored. osImage.factory.version reaches the machineconfig's installer
image and changes the reconcile Job, so a pool could set it, watch the Job
roll, and end up running one Talos while booting another: v1.12.6 moves the
installer to .../installer/<schematic>:v1.12.6 and leaves the
ProxmoxMachineTemplate untouched. osImage.builtin has no meaning here at all.

Nothing is lost by refusing it: the installer target on this substrate is
settable through talos.version and talos.schematicID, which do reach it.

The branches suite gains the refusal case. The substrate parameter's description said changing the value "rolls the
pool". The chart refuses the change on a live pool, and the description says
so; it reaches the README, the values schema and the CRD.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:46:15 +02:00
Marian Koreniuk 4bc6621a5c test(kubernetes,kubernetes-nodes): assert both sides of every substrate branch
Every `if substrate` in both charts is asserted on both sides, so an inverted
condition cannot pass:

- the reconcile Job reads SVC_IP from the LoadBalancer status on proxmox and
  from the ClusterIP on kubevirt;
- the machineconfig carries the pool's nameservers on proxmox and none on
  kubevirt;
- the proxmox gate on each kubevirt-only template in the parent chart, the two
  RBAC bindings included, so a proxmox cluster renders no binding to a Role,
  ClusterRole or ServiceAccount it does not render;
- the sizing conversion at 1500m and 8G, where ceil and floor disagree
  (2 cores and 7630 MiB against 1 and 7629);
- the old-template retention loop keeps a ProxmoxMachineTemplate a MachineSet
  still references;
- the proxmox worker monitor carries no Pod threshold while the kubevirt one
  does;
- a live cluster and a live pool refuse a substrate switch while a first install
  still renders;
- both charts refuse a dnsServers entry that is not an address, the parent
  chart included, where the value reaches ProxmoxCluster.spec.dnsServers.

The negations carry `any: true`: without it a negated containsDocument passes
whether or not the document is in the render.

hack/talos-reconcile-heredoc_test.bats gains the dnsServers cases: the render
refuses a command substitution and names the entry; an entry with a colon in it
is refused too, which pins the IPv6 branch of the validation and not only the
IPv4 one; and a real address survives the Job's shell unchanged. The heredoc is
extracted by shape — `cat <<EOF` at the start of a line or right after `=$(` /
`="$(` — so the same case keeps working whichever way the Job writes it.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:46:15 +02:00
Marian Koreniuk 517baf945d fix(kubernetes,kubernetes-nodes): validate dnsServers, refuse a substrate switch, drop the Pod threshold on proxmox
Three faults in the proxmox substrate, none of which the render refused.

1. proxmox.dnsServers reached the reconcile Job's shell unescaped.

The Job applies the worker TalosConfigTemplate through an unquoted heredoc, so
every byte of its body is subject to parameter expansion and command
substitution at Job runtime. The INVARIANT comment above that block says each
tenant-controlled value must be shell-escaped or pattern-validated; the new
nameserver list was neither. `quote` there is YAML quoting.

Rendered with `$(id -u)` and a backtick pair in dnsServers, extracted the way
hack/talos-reconcile-heredoc_test.bats does and run under sh, the shell executed
both and put their output in the worker's resolver list. The Job's pod holds the
parent cluster's Talos token, cluster id, cluster secret and bootstrap token in
its environment, and its Role can read the Talos CA and patch the parent
KamajiControlPlane.

Each entry is now validated as an IPv4 or IPv6 address at render time, in both
charts. Validation rather than escaping because Talos takes addresses here: an
address cannot carry a metacharacter, so the property holds by construction, and
the ordinary mistake of writing a hostname is caught in the same move. IPv6 is
matched on its character set rather than its full grammar — what matters is which
bytes may reach the shell, and the apiserver rejects a malformed address anyway.

2. Nothing stopped a live cluster or pool from being switched between substrates.

`substrate` had no immutability of any kind, and the `## @immutable` marker this
adds is dashboard-enforced only, so both charts now read the live object.

Switching a cluster removes the KubevirtCluster, the cloud-controller-manager,
the kubevirt-csi controller and the in-tenant -csi HelmRelease with all their
RBAC from the render. Flux then uninstalls the CSI driver behind every PVC the
tenant has. CAPI does not prevent it — its Cluster webhook forbids removing
spec.infrastructureRef but admits a change of kind.

Switching a pool rolls every worker to the other hypervisor and drops the
machine template the draining MachineSet still references, because the retention
loop looks up only the kind the new substrate uses. A pool whose substrate
differs from its parent's was not rejected either, though the README says the
two must match.

Both guards are inert offline, like assertNoForeignPool and assertParentVersion
beside them: with no apiserver the lookup is empty and a first install renders.

3. The worker WorkloadMonitor reported a proxmox pool as broken.

The controller lists Pods matching the monitor's selector and sets
Operational=false while fewer than minReplicas are ready. On kubevirt those Pods
are the virt-launcher Pods. A proxmox worker is a VM on another machine with no
Pod here, so the count is always zero and any threshold pinned the pool — and
through the lineage labels its whole application — to WorkloadsReady=False while
every node was healthy. The monitor still renders for the dashboard and the
lineage; what it no longer carries on proxmox is a threshold counted in Pods.

Two smaller things. The wait-loop log line is written per substrate, so a
default kubevirt render of kubernetes-nodes is byte-identical to main and no
existing pool gets a new reconcile Job on upgrade. And the comment claiming a
rename of $kubevirtmachinetemplate would roll every live worker VM is replaced:
the hash is taken over the rendered template, not the variable name, so renaming
it leaves the render byte-identical.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:46:15 +02:00
Marian Koreniuk 33e1de52d2 test(kubernetes-nodes): pin the linked-clone default and three silent guards
Nothing failed when `full` was wrong: the pool came up, the workers joined, and
each one quietly cost the whole template disk instead of the increment — 10.2
GiB against 8 KiB on a ZFS store. A default whose only symptom is disk usage is
exactly the kind that drifts, so both it and its override are now asserted.

The bridge and dnsServers guards were two of the guards on this path with no
coverage. Both fire on absence, so the suite omits each and the tests supply the
other one back; declaring either at suite level would make it impossible to take
away, which is why the sizing guard already lives alone.

The storage guard is the third: `storage` with a linked clone is refused and the
message names both values, while `storage` with `full: true` reaches the
ProxmoxMachineTemplate.

Each assert goes red on the edit it exists for: flipping the default reds the
clone suite, deleting a guard reds the guard suite.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:46:15 +02:00
Marian Koreniuk c5996c1d0f fix(kubernetes,kubernetes-nodes): give Proxmox workers an endpoint they can reach
A worker on the proxmox substrate is a VM on the hypervisor network, not a Pod
in this cluster. It was handed two addresses only an in-cluster consumer can
route to: the apiserver as a ClusterIP, and the management cluster's CoreDNS
ClusterIP as its resolver. The VM booted, the Machine reported Ready with a
providerID, the disk was where it should be — and no CSR ever arrived. Nothing
in that picture points at an address.

The control-plane chart now publishes the tenant apiserver as a LoadBalancer on
the proxmox path, putting the endpoint on the network the workers share, and
the pool's reconcile Job reads the Service's LoadBalancer address instead of its
ClusterIP for the host entry the workers dial. The nameservers follow the same
reasoning: proxmox pools carry their own resolvers rather than pointing at a
CoreDNS they cannot reach, and the chart refuses to render a pool that has not
named any.

KubeVirt keeps the ClusterIP on both counts. Its workers are in this cluster,
where a ClusterIP is the right answer and a LoadBalancer address would be an
unnecessary one.

Verified end to end on a real tenant cluster: the node joins, its CSR is
approved, and cilium, konnectivity and coredns come up in the tenant.

One operational note this surfaced, not caused by it: TalosConfigTemplate.spec
is immutable, so the reconcile Job cannot rewrite an endpoint that has already
been published — the template has to be deleted for the Job to recreate it.
That matters only when an existing cluster's apiserver address changes.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:46:15 +02:00
Marian Koreniuk d78af928fc feat(kubernetes-nodes): default Proxmox worker disks to linked clones
A full clone on a ZFS pool configured with `sparse 0` reserves the whole
volsize the moment the VM is created: 10.2G for a worker whose Talos image
occupies 337M. A linked clone writes only its own increment — measured at 8K
for the same worker on the same host — because it shares the template's base
snapshot until it diverges.

The cost is that the clone holds that snapshot, so the template cannot be
rotated while any linked clone still references it. That is bounded rather
than permanent: consolidation detaches each worker 24 hours after creation, so
template rotation is blocked for at most a day.

Consolidation is not a chart concern, but two of its constraints are worth
recording where the flag is set, because both were discovered by testing on a
live host rather than from the docs. A linked clone can only be detached by
moving its disks to a *different* storage — Proxmox refuses a same-storage
move outright. And both disks must move: leaving efidisk0 behind keeps the
template pinned just as effectively as the system disk would.

Set full: true for a pool that must be independent of the template from the
first second.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:46:14 +02:00
Marian Koreniuk 85f576756b feat(kubernetes-nodes): add a proxmox substrate for worker pools
Worker pools are already Talos machines created by Cluster API; only the
substrate underneath them is fixed to KubeVirt. This adds `substrate:
kubevirt|proxmox` and swaps exactly two things: the machine template object
and every reference to it. Autoscaler capacity annotations, kubelet
reservations, MachineDeployment, MachineHealthCheck, WorkloadMonitor and the
talos-reconcile Job are substrate-agnostic and rendered once, which is why
this is a branch in the existing chart rather than a second copy of it.

The default path is unchanged down to the bytes: the render snapshot and all
15 existing suites pass untouched, and the hash-named template keeps its
variable name so no live worker VM rolls on upgrade.

Five things the Proxmox path must do differently, each learned from a
failure that pointed somewhere else:

- Skip capmox's guest-agent and cloud-init-status probes. Talos ships
  neither, and left on they leave the VM booted and the Machine stuck on
  "error waiting for agent: the operation has timed out". This is the same
  class of problem `virtualMachineBootstrapCheck.checkStrategy: none` solves
  for CAPK's SSH probe; capmox's CRD names TalOS in the skipCloudInitStatus
  description.

- Clone a template carrying the nocloud image, not openstack. capmox injects
  bootstrap data as a nocloud ISO, and an openstack image ignores it exactly
  as silently as a nocloud image ignores the configdrive the KubeVirt path
  depends on.

- Pass templateTags through verbatim. capmox compares the template's tag set
  with slices.Equal after sorting, so a selector listing a subset matches
  nothing and reports "found 0 VM templates with tags ..." while the tags are
  plainly present.

- Require resources.cpu and resources.memory. Proxmox has no
  VirtualMachineClusterInstancetype, so the lookup that normally supplies
  effective CPU and memory has nothing to read — and those two numbers feed
  the kubelet reservations and the autoscaler capacity annotations. A pool
  that cannot state them would be scheduled against invented capacity, so it
  is refused at render time. KubeVirt-only knobs (gpus, logSerialConsole,
  podCpuLimit/Request) are refused too rather than accepted and ignored.

- Refuse proxmox.storage unless full is true. A target storage is a full-clone
  parameter: a linked clone always lives on the template's storage, and both
  Proxmox ("parameter 'storage' not allowed for linked clones") and capmox's
  own CRD ("Must set full=true when specifying storage") reject the pair.
  Left to them, a pool with storage set and a linked clone fails at apply time
  with an error about a ProxmoxMachineTemplate the operator never wrote;
  failing the render names the two values instead.

The Talos image is not a chart input on this path: capmox clones a template
that already exists on the hypervisor, so building and rotating that template
is an operator responsibility with no equivalent on the KubeVirt side.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:29:08 +02:00
Myasnikov Daniil bc8c54d6e6 feat(kubernetes): boot tenant worker disks from a shared golden Talos image
Every tenant worker VM imports the same Talos raw image from the Image Factory
over HTTP, once per worker: a 20-worker cluster fetches the identical ~4.15 GiB
artifact 20 times, and each fetch is a live dependency on the Factory at exactly
the moment a node joins.

packages/system/kubernetes-worker-image is a new opt-in catalog that holds one
golden DataVolume per (schematicID, version) in cozy-public, named
talos-worker-<schematicID>-<version>. A worker pool opts in with osImage.builtin
on its own KubernetesNodes resource, and its disk-system DataVolume then carries
source.pvc pointing at that golden, so CDI performs a storage-layer CSI clone
instead of an HTTP import -- one import per flavor, shared by every worker of
every tenant. Confirmed on a LINSTOR-backed cluster: the worker disk is a CSI
clone plus a resize, with no host-assisted upload pod.

Both halves are off unless asked for. The catalog is registered through the
platform's optional-package helper and disabled by default, so a cluster that
does not enable it creates no golden and keeps importing over HTTP. The disk
source is derived entirely from osImage, which is absent by default, so a pool
that does not set it renders byte-identical to the pre-feature chart and
adopting the feature never rolls existing workers. Editing osImage on an
existing pool re-images that pool, as changing any other image reference does.

What the render refuses, rather than degrading into a silent fallback:

- A pool whose golden has no catalog entry, or whose golden terminally failed to
  import. A golden that is merely mid-import is not an error; CDI holds the
  worker disk until the source populates.
- A pool whose StorageClass differs from its golden's. CDI cannot CSI-clone
  across classes and falls back to a host-assisted copy over the pod network,
  which is the transfer this feature exists to remove. An empty class on either
  side means the cluster default and is resolved before the comparison, and a
  pool that has already cloned reads its class from its own bound PVC rather
  than from today's default -- the golden's side is frozen the same way, and an
  admin moving storageclass.kubernetes.io/is-default-class must not fail the
  release of a pool whose disks are all on the right class. That resolved class
  is then written onto the worker disk, so the disk binds where the guard
  checked instead of on whatever the default has become by then -- reading one
  class and binding another is how a pool at an empty storageClass passed the
  comparison and still took the host-assisted copy. The HTTP path leaves an
  empty class to the cluster as before, which is what keeps its render, and the
  KubevirtMachineTemplate hash named after it, byte-identical.
- A diskSize below the golden's storage size, which a clone target cannot be.
- A schematicID that is not a plain lowercase alphanumeric identifier, or a
  version that is not a vMAJOR.MINOR.PATCH release. Both are interpolated into
  a DataVolume name and an image URL, and values.schema.json types them as bare
  strings, so a malformed one otherwise reaches the render. The catalog applies
  the same patterns, since both sides build the same name from the same fields.
- An images[] entry edited after its golden exists. A DataVolume spec cannot be
  patched in place, so the message names the entry and the supported move -- a
  new (schematicID, version) -- instead of leaving the API server to reject the
  patch on the next upgrade.

Goldens carry helm.sh/resource-policy: keep, so dropping an entry stops
managing a golden rather than deleting it: a missing golden aborts the whole
Kubernetes HelmRelease of every tenant pinned to it, not just the disk. Each
cloned worker disk annotates the golden it came from and the pool that owns it,
so "does anything still clone this?" is answerable before a manual delete
instead of after.

A builtin pool needs a StorageClass whose volumes are reachable from any node,
in practice a replicated/DRBD one. The golden in cozy-public has no consumer and
binds as soon as it imports, wherever CDI's importer landed, while a worker disk
binds where its VM is scheduled, and nothing here arranges for those to be the
same node. The platform ships cloneStrategyOverride: csi-clone, which rules out
the host-assisted strategy that would cross nodes. What a node-pinned class does
in that situation is not measured, so values.yaml states the requirement and not
a symptom.

ComputePlane pools forward only the per-pool shape fields, and osImage is
excluded there by name: the worker OS image, its Talos installer and its Image
Factory are operator decisions about the platform's own images.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-23 15:35:44 +03:00
IvanHunters d93e136e23 fix(computeplane): provision worker pools via kubernetes-nodes after Phase 2 split
Phase 2 moved worker nodeGroups out of apps/kubernetes into apps/kubernetes-nodes,
but the ComputePlane module still passed nodeGroups to the wrapped kubernetes
chart, which now ignores it: a fresh ComputePlane cluster came up with zero
workers (and the pinned ingress-nginx addon stuck Pending), while existing ones
were unreachable by migration 56 (it only adopts kubernetes-* releases, not
computeplane-cluster).

The module now emits one kubernetes-nodes child HelmRelease per nodeGroup,
shaped like a natively created pool, and materialises the pre-split default md0
(roles: [ingress-nginx]) when nodeGroups is empty. The pool chart's clusterName
gains an internal clusterReleaseName override so pools attach to the module's
fixed computeplane-cluster release, which is off the aggregated API's
kubernetes-<cluster> convention; the default keeps every API-created pool
byte-identical.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-24 11:19:54 +03:00
IvanHunters 69da032103 docs(kubernetes): point references at migration 56 after the renumber
The merge renumbered the worker-pool adoption migration from 54 to 56, but
six references still named 54 (README, pre-delete-unpin hook, cluster.yaml,
worker_adoption_guard test), which now points at main's redis-operator
adoption migration. Repoint them at 56.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-21 00:35:05 +03:00
IvanHunters cd72d81bb4 Merge branch 'main' into feat/kubernetes-nodes-phase2b
Reconcile the phase-2b split with main's configurable worker-health
feature (MHC condition timeouts, maxNodeProvisionTime, podCpuLimit/
podCpuRequest), which landed on both charts after this branch forked:

- kubernetes (parent): pool rendering stays removed; main's parent
  worker-pool additions do not apply post-split.
- kubernetes-nodes (pool): absorb the feature; keep the directional
  parent-version skew guard and the immutable cluster/storageClass rules.
  KubevirtMachineTemplate content hashes are unchanged (render snapshot).
- migration 54: carry podCpuLimit/podCpuRequest (KMT-hash inputs) and the
  cluster-wide readyUnknownTimeout/readyFalseTimeout/maxNodeProvisionTime
  into adopted pools, so adoption neither rolls nor resets a tuned pool.
  Covered by a new adopt-values bats case.
- drop the parent chart's obsolete MHC-timeout suites; the
  kubernetes-nodes twins cover the feature.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-19 13:14:11 +03:00
Aleksei Sviridkin 3710541278 fix(kubernetes): validate MachineHealthCheck condition timeouts at render time
The CAPI webhook validates nodeStartupTimeout with a 30s floor but does
not validate unhealthyConditions timeouts at all, so a zero or negative
readyUnknownTimeout or readyFalseTimeout applied and remediated a
Machine the moment its Ready condition flipped. Route both fields
through the same render-time duration guard as maxNodeProvisionTime,
extracted into a kubernetes.positiveDuration helper in each chart.

The guard now also accepts the canonical form a duration serializes
back to ("20m0s", "1h0m0s"): metav1.Duration prints that shape, so a
value copied from kubectl output previously failed the render. The
first segment must still be positive and later segments may be zero,
which keeps every accepted string a positive duration.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-08-18 22:12:54 +03:00
Aleksei Sviridkin ddb15ebc6a fix(kubernetes): widen the worker node startup budget to 20m
nodeStartupTimeout is the window a Machine has to produce a Node at
all, measured from the later of the Machine's creation and its
InfrastructureReady transition. What has to fit inside it is the guest
boot and the kubelet registering with the tenant apiserver, and on a
host under load neither is quick. At 10m a worker that was still
booting was deleted before it ever joined, and its replacement started
the same race from the beginning.

Also record what a percentage maxUnhealthy does at small group sizes,
because the interaction is not what the field name suggests: the value
is scaled against the machines the check covers and rounded down, and
remediation proceeds only while the unhealthy count is at or below the
result. At the default "50%" a group of one never remediates, and a
group of two remediates one unhealthy machine but not two.

The golden-parity gate pinned the child chart's MachineHealthCheck
values to the parent's defaults as inputs, which made a default moved
on both sides fail there for the wrong reason. Drop the pins so the
comparison is between the two charts' defaults, which is the contract
it exists to hold, and pin the defaults themselves in each chart's own
unit tests instead.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-08-18 20:09:26 +03:00
Aleksei Sviridkin 33701800d6 fix(kubernetes): make the worker MachineHealthCheck timeouts configurable
The two unhealthyConditions timeouts were fixed in the template at 30s
for `Ready=Unknown` and 300s for `Ready=False`, and both are too tight
for a worker joining on a loaded host.

`Ready=Unknown` is what the node lifecycle controller writes when a
kubelet stops reporting, on a grace period of its own, so 30s on top
of that deleted the VM of a guest that had merely stopped being
scheduled for a stretch, before it had a chance to answer again. And
`Ready=False` is the condition a registered Node holds until its CNI
is up, so 300s was the entire budget a first boot got for the CNI
rollout to reach it -- a worker deleted mid-rollout takes the rollout
with it and its replacement starts over.

Move both to values, defaulting to 5m and 10m. The remediation of a
worker that is genuinely dead is correspondingly later, which costs
less than it appears to: Pods leave an unreachable Node on the node
lifecycle controller's taint-based eviction and the toleration that
carries, not on either of these timers.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-08-18 20:09:13 +03:00
Aleksei Sviridkin 83f0eef645 fix(kubernetes): give the autoscaler the real worker join time
The cluster-autoscaler treats a machine as long-unregistered once its
own max-node-provision-time elapses, and a worker that has not
registered by then stops counting towards its node group's upcoming
capacity. On a host slow enough that the Talos image import, the guest
boot and the CNI rollout do not fit inside that window, the timer
fires on machines that were still joining: the group reads as short of
capacity, so the autoscaler asks for replacements, and the extra
provisioning load makes the join it was waiting for slower still.

Render the per-node-group override the autoscaler reads off the
MachineDeployment, sourced from a new maxNodeProvisionTime value that
defaults to 30m. The duration is checked at render time because the
autoscaler discards one it cannot parse and falls back to its own
default without failing, and because the values schema types the field
as a string and so accepts every malformed duration.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-08-18 19:41:00 +03:00
IvanHunters 5297df5cf3 Merge remote-tracking branch 'origin/main' into pr-3315-lgtm
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>

# Conflicts:
#	api/apps/v1alpha1/kubernetes/types.go
#	api/apps/v1alpha1/kubernetes/zz_generated.deepcopy.go
#	hack/e2e-chainsaw/_lib/run-kubernetes.sh
#	packages/apps/kubernetes-nodes/tests/render-parity.sh
#	packages/apps/kubernetes/README.md
#	packages/apps/kubernetes/templates/cluster.yaml
#	packages/apps/kubernetes/values.schema.json
#	packages/apps/kubernetes/values.yaml
#	packages/system/kubernetes-nodes-rd/cozyrds/kubernetes-nodes.yaml
#	packages/system/kubernetes-rd/cozyrds/kubernetes.yaml
2026-08-17 11:25:29 +03:00
Aleksei Sviridkin 3cb131785c fix(kubernetes): settle what a pod CPU quantity of zero or less means
Both podCpuLimit and podCpuRequest were rendered under `with`, which
skips a zero as readily as an absent field, and their guards were keyed
the same way. The generated schema behind them admits a leading minus
and a zero as readily as a sane value, so rather than patch whichever
edge turned up, the whole value domain is settled in one place.

Presence, not truth, is the predicate everywhere either field is read.
A negative value is refused: it never reaches a running Pod. A zero
request on its own is allowed and reserves nothing. A zero request
beside a non-zero limit is refused: KubeVirt's defaulter tests the
request with IsZero, which an explicit zero satisfies exactly as an
absent key does, so it overwrites the zero with the limit and the
worker asks the scheduler for its whole ceiling. That is the outcome
podCpuRequest exists to prevent, reached silently and reported nowhere,
which is why it is refused rather than written down.

A zero limit removes the ceiling rather than being the tightest one --
the kubelet turns a zero CPU limit into no CFS quota -- but it reaches
a Pod only paired with a zero request. KubeVirt keeps deriving a
request from the vCPU count when the VM declares none, the defaulter
that would have copied the limit across does not fire because the limit
is itself zero, and API server validation refuses a request above a
limit that exists. Zero is a limit that exists. So a lone zero limit is
refused too.

Both fields document the same rules and each class has a test on both
charts. A group that sets neither field renders the bytes it rendered
before, so the content-hash name of the worker VM template does not
move.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-08-16 22:53:53 +03:00
Aleksei Sviridkin a6770d0f1b fix(cozy-lib): convert micro and nano quantities instead of zero
cozy-lib.resources.toFloat knew every suffix of the Kubernetes quantity
grammar except `u` and `n`. A suffix it does not recognise is not an
error: the unit letter stays on the string, `float64` cannot parse the
result and yields zero silently, and the caller divides or compares
that zero as if it were the value.

The node group CPU request guard is one such caller. A request of 100u
above a limit of 50u compared zero against zero and rendered, so the
pair reached API server validation on the worker's Pod instead of
failing where the values were written.

Pinned by a suite over the whole suffix grammar, one value per form,
because the failure has no error to notice it by.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-08-16 22:20:17 +03:00
Aleksei Sviridkin 1b8340298e feat(kubernetes): let a node group set the CPU request of its worker VMs
Adds nodeGroups.<name>.podCpuRequest, rendering into
domain.resources.requests.cpu on the worker VM, and the same field on
the kubernetes-nodes chart, which is the other half of a byte-identity
contract rather than a separate feature surface.

podCpuLimit on its own cannot raise the ceiling of a worker without
raising what that worker asks the scheduler for, because KubeVirt
copies a declared CPU limit into the request when the VM declares none.
The request is the half that has to hold still, so the pair can now be
moved apart.

Both fields conflict with instancetype sizing for the same reason and
in the same upstream check, so the request gets the guard the limit
already had. A request above its own limit fails the render too, rather
than reaching API server validation on a Pod two objects away from the
CR that named the values; the quantities are compared as numbers, so
500m under a ceiling of 1 is accepted where a lexical guard would
refuse it.

A node group that sets neither field renders the bytes it rendered
before, which the content-hash name of the KubevirtMachineTemplate
depends on, and the render-parity gate gains a case that compares the
pair across the two charts.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-08-16 21:21:10 +03:00
Aleksei Sviridkin 669c3e60ab docs(kubernetes): correct what podCpuLimit does to the CPU request
The field was documented as moving the ceiling alone and leaving the CPU
request at the value KubeVirt derives from the guest vCPU count and the
cluster CPU allocation ratio. KubeVirt does the opposite: its VMI
defaulter copies a declared CPU limit into the CPU request whenever the
VM declares no request of its own, so a node group that sets only the
limit asks the scheduler for its whole ceiling, and a ceiling the nodes
cannot spare leaves the workers Pending on Insufficient cpu.

The same paragraph promised a floor that does not exist either. A limit
below the derived request was said to leave the container with a request
above its limit; the copy makes the two equal instead, whatever the
limit is.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-08-16 21:15:33 +03:00
Aleksei Sviridkin 78b9bc3e6e feat(kubernetes-nodes): mirror the worker pod CPU limit into the pool chart
The split-out pool chart renders the same worker VM as the monolithic
one, and tests/render-parity.sh exists to prove it renders it
byte-identically, because the KubevirtMachineTemplate is content-hash
named and any divergence rolls every live worker VM. A field that only
one of the two charts knows breaks that quietly rather than loudly: a
pool adopted into a kubernetes-nodes release would render without the
CPU limit it was given, the hash would change, and the ceiling would
fall back to the vCPU count with nothing reporting it.

So the field, its render-time guard against combining it with an
instancetype, and its tests arrive here in the shape the parent chart
has them, and the parity gate gains a case that compares the two
renders with the limit set.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-08-16 17:06:17 +03:00
IvanHunters 5999807aa6 fix(kubernetes-nodes): guard against silent clusterName drift on pool adoption
The release name kubernetes-nodes-<cluster>-<pool> does not encode where the
cluster segment ends, so several (.Values.cluster, pool) splits reconstruct the
same object name kubernetes-<cluster>-<pool>. An operator who edits spec.cluster
to another such value (immutability is enforced only by the dashboard, not the
aggregated apiserver, per docs/storage-immutability.md) prunes no worker VM but
silently flips spec.clusterName and the pool WorkloadMonitor selector; CAPI
rejects the immutable MachineDeployment.spec.clusterName Update loudly while the
WorkloadMonitor drift stays silent.

Extend assertNoForeignPool to refuse the render when the live MachineDeployment
of the reconstructed name carries a spec.clusterName that disagrees with the
clusterName this pool renders, naming the value to restore. Inert offline.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-13 18:39:38 +03:00
IvanHunters b4a2f6f791 Merge remote-tracking branch 'origin/main' into pr-3315-lgtm
# Conflicts:
#	api/apps/v1alpha1/kubernetes/types.go
#	api/apps/v1alpha1/kubernetes/zz_generated.deepcopy.go
#	hack/e2e-chainsaw/_lib/run-kubernetes.sh
#	packages/apps/kubernetes-nodes/tests/render-parity.sh
#	packages/apps/kubernetes/README.md
#	packages/apps/kubernetes/templates/cluster.yaml
#	packages/apps/kubernetes/templates/talos/talos-reconcile-job.yaml
#	packages/apps/kubernetes/tests/talos_templates_test.yaml
#	packages/apps/kubernetes/values.schema.json
#	packages/apps/kubernetes/values.yaml
#	packages/system/kubernetes-nodes-rd/cozyrds/kubernetes-nodes.yaml
#	packages/system/kubernetes-rd/cozyrds/kubernetes.yaml
2026-08-10 22:11:21 +03:00
Aleksei Sviridkin 3ae1e10c92 docs(kubernetes): correct the registryMirrors field and README claims
State in the README that in-guest registry pulls are routable again via
talos.registryMirrors, which knob covers which artifact (the mirror
routes registry pulls; the worker OS disk image arrives as a raw HTTP
artifact under talos.imageFactoryURL and no registry mirror touches
it), and the limits: skipFallback is needed for air-gap,
machine.registries.config is not passed through, and existing nodes
keep their boot-time config until replaced. Trim the field doc comment.
The regenerated schema, README tables and cozyrds openAPISchema follow
from those sources.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-09 16:17:45 +03:00
Aleksei Sviridkin 3a1292d682 fix(kubernetes): escape tenant values in the worker reconcile heredoc
The worker TalosConfigTemplate is applied from an unquoted
`cat <<EOF | kubectl apply` heredoc, so every free-form value
interpolated into it is subject to shell parameter expansion and
command substitution at Job runtime. Escape backslash, dollar and
backtick on talosVersion, installerRepository, schematicID and the
registryMirrors block so each renders as a literal, and record the
invariant above the data block in both charts: a new free-form field
here must be escaped or render-time pattern-validated, or it reopens
command injection into a host-cluster Pod. Escaping covers every byte
by construction; the chain is a no-op for ordinary values, so the
default render and its content-hash Job name are unchanged.

Tests render both charts and run the heredoc through a real shell with
hostile values, which a rendered-string regex cannot do, and the
default-omits-mirrors assertion selects the Job by kind rather than by
document index so it cannot pass against the wrong document.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-09 16:17:26 +03:00
IvanHunters 58910c0edc feat(kubernetes): route tenant worker image pulls through a registry mirror
Add a talos.registryMirrors passthrough to the kubernetes and kubernetes-nodes
charts that renders into the worker TalosConfigTemplate machine.registries.mirrors,
so an operator can point a registry such as ghcr.io at an in-cluster pull-through
mirror and a worker boot no longer depends on live public egress. Default {}
keeps the rendered worker reconcile Job byte-identical to before, content-hash
Job name included. Free-form mirror values are shell-escaped for the unquoted
apply heredoc, covered by an execution-level bats test.

Also add the e2e ghcr-mirror (registry:2 pull-through plus a Cilium egress
policy) and wire it into the kubernetes chainsaw suites, so CI exercises the
mirror path and takes the flaky public ghcr.io kubelet-image pull off live
egress.

Refs #3513 (addresses the egress variant of the tenant-worker node-join
failures; the dominant variant is CI-runner CPU starvation, tracked in the
issue).

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-09 14:10:16 +03:00
Aleksei Sviridkin 50c5e2c14c test(e2e): capture the tenant worker guest serial console
The tenant Kubernetes suites fail the node-Ready budget with zero
Nodes registered, and the time is lost inside the guest between the
worker VMI reaching Running and Talos requesting its apid
certificate. Nothing collected today covers that window: the existing
in-guest capture drives talosctl, so it needs the very apid the
worker never reached, and on the management side there is neither a
Node nor a certificate request to read.

Turn on the guest console log for the suite's node group and collect
each virt-launcher's guest-console-log container from the management
cluster. Reads start at the beginning of the stream rather than
tailing it, so a guest that repaints its console cannot push the boot
output out of the window; every read is bounded, and the walk stops
at a cap that records both counts, so the collector cannot become the
unbounded term in a failure path that already runs several others
under one deadline.

A read that returns no console is not filed as a quiet guest. Three
outcomes get three labels, because only one of them says anything
about the guest: the read failed and returned nothing, the read
succeeded and returned nothing, or it broke off part way. Each read's
stderr is kept beside the capture rather than inside it, since
kubectl's own "container is not valid for pod" would otherwise fill
the file and a console that never started would read as one that had
spoken. That case is the headline one here, so the Pod state and
events that explain it are collected alongside, once for the whole
selector.

Enabling the console overrides a cluster-wide setting the platform
holds for an open upstream defect, so the run is also an experiment,
and its own failure mode is the one that hides best: the container
wedges virt-launcher in Init, no worker boots, and the suite reports
"fewer than 2 tenant nodes Ready" -- the same sentence the failure
under study produces. Read as the known signature, the experiment's
answer would be filed as the flake it was meant to explain. So the
failure path names that case first, before any other diagnostic, and
only when it is true.

The passing path checks separately that the container attached at
all, and reports the ratio it saw rather than a bare success, since
the check is deliberately satisfied by any one Pod. It keeps three
outcomes apart too: a read that did not answer, a selector that
matched no Pod, and Pods that matched and carry nothing. It reads the
Pod name alongside the container names so the last two cannot
collapse, and both container lists so a future move of the container
cannot fail a run over a diagnostic. It runs last, after everything
the suite exists to prove, because a debugging aid must not preempt
the assertions that cover the cluster.

The capture runs before the talosctl one because it is the only of
the two that can describe a worker which never reached apid, the
shape this failure keeps taking. kubectl reports a warning and an
empty result alike, on stderr with a zero status, so the exit status
decides which artifact its output lands in.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-08 05:48:38 +03:00