Files
cozystack/packages
Aleksei Sviridkin 0a2054e228 fix(kubernetes): apply worker TalosConfigTemplate server-side (#4163)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The `talos-reconcile` Job runs under a 256Mi limit and applies its
`TalosConfigTemplate` with a client-side apply. Building the merge patch
means pulling the cluster's full OpenAPI schema down, which on a
management cluster carrying a few hundred CRDs costs around 290Mi of
RSS, so the kernel OOMKills kubectl before the Job finishes.

Only the merge path pays that cost, which is what makes this hide so
well. The first run creates the object and stays cheap, so a fresh
install goes green; every reconcile afterwards is an update, so the Job
CrashLoopBackOffs on every chart upgrade until it burns its
`backoffLimit: 30`. The KamajiControlPlane patch lands before the apply
dies, so the cluster keeps working and the failure reads as cosmetic
noise.

Measured against a 438-CRD management cluster with the Job's own kubectl
(`alpine/k8s:1.33.4`), cold discovery cache:

| command | peak RSS |
| --- | --- |
| `get -o jsonpath` | 52Mi |
| `apply`, object absent (create) | 52Mi |
| `apply`, object present (merge) | **292Mi** — OOMKilled at 256Mi |
| `apply --server-side` | 56Mi |

`--validate=false` changes nothing, so the cost is the merge strategy
lookup rather than validation. Server-side apply leaves the merge to the
apiserver and the client never needs the schema. `--force-conflicts`
keeps the Job authoritative over an object an earlier client-side apply
created.

Raising the memory limit instead would work today and rot tomorrow: the
cost tracks the cluster's CRD count, so the ceiling would need lifting
again as the API surface grows.

Includes a regression test asserting the rendered Job applies
server-side and carries no client-side fallback. It fails on the pre-fix
template.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(kubernetes): apply the worker TalosConfigTemplate server-side so the talos-reconcile Job is no longer OOMKilled on clusters with a large CRD count. The Job previously failed on every reconcile after the first install, which meant on every chart upgrade.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved Kubernetes reconciliation reliability by using server-side
updates for Talos configuration templates.
* Prevented reconciliation jobs from exceeding their memory limit during
chart upgrades.
* Ensured the reconciliation job remains authoritative when updating
previously created resources.

* **Tests**
* Added coverage verifying server-side application behavior and conflict
handling.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 17:37:49 +03:00
..