Commit Graph
197 Commits
Author SHA1 Message Date
Myasnikov Daniil b83cc087cc docs: move the repo-only guides onto the website
Seven guides in docs/ were written next to the code and left in the
source tree, where no operator looks. The website carries them now, so
a second copy here only lets the two drift again: the Backup Classes
page had already fallen behind its original.

docs/release.md, docs/deprecating-applications.md and the two SeaweedFS
runbooks in docs/operations stay. A migration, a bats suite, a workflow
or AGENTS.md reads each of them by path, which is the test for keeping
a document in this repository.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-21 17:50:46 +05:00
Myasnikov Daniil f3486f4e3a docs: move the operator guides out of docs/ to where readers find them
Five guides in docs/ were written next to the feature code that needed
them and left in the source tree, where no operator looks. They are on
the website, so a second copy here only drifts: edits kept landing in
docs/ after the website copy was made.

docs/vm-external-vlan.md is split rather than moved. Its section on
staging the reference CNI plugins is a contract that
hack/cni-plugins-staging-contract.bats asserts against the multus
Dockerfile, the daemonset manifest and the platform values, so that half
is package documentation and goes to packages/system/multus/README.md,
which the suite now reads. The operator half goes to the website.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-14 16:24:16 +05:00
Aleksei Sviridkin 1f74ad4ee7 fix(multus): drop the CPU limit and make resources configurable (#4830)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The multus daemon was too small to recover a node after a CNI outage,
and its limits could not be raised from values. The 100m CPU limit and
the 1Gi memory limit of `kube-multus` were hardcoded in the patched
upstream template.

On a node with about 600 dead sandboxes and 220 pods waiting, multus was
throttled in every CFS period. Its calls took longer than kubelet's
two-minute CRI deadline, so every call was retried and the queue only
grew. With the CPU limit lifted it OOM-looped at 1Gi. With 2 CPU and 4Gi
it cleared the backlog in about three minutes.

Now the resources come from a new `resources` value. The default keeps
the same requests (100m CPU, 100Mi memory), has no CPU limit, and raises
the memory limit to 4Gi. The memory limit stays on purpose. Without it
the pod cgroup has no `memory.max`, and multus becomes a candidate for
the Talos userspace OOM handler again.

The chart now refuses to render when an override removes the memory
limit or the memory request. Without a request Kubernetes uses the limit
as the request, so a node with less allocatable memory than the limit
would be left without multus.

The change is in `patches/customize-deployment.patch`, so `make update`
keeps it. I applied the patch to upstream v4.3.0 and got the committed
template, except the image references. The platform reverts a direct
edit of the child `cozystack.multus` Package, so a running cluster sets
the value through the new `networking.multus.resources` key of the
platform Package. The key is passed on only when set, and a value that
is not a map stops the render. The system-defaults LimitRange in the
namespace sets memory only, so it does not bring a CPU limit back.

Closes #4829

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

Not a UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/pull/747
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff adds a platform value, so the hand-written key table on the
website needs it. The follow-up PR there is a draft until this one
merges.

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(multus): the kube-multus container no longer has a CPU limit, and its memory limit is raised from 1Gi to 4Gi, so a node with hundreds of pods can recover after a CNI outage. Its resources can be overridden through the new `networking.multus.resources` platform value. An override that removes the memory limit or the memory request now fails the render.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added optional CNI plugin staging during Multus startup, configurable
through chart values. Staging skips when the plugin directory is
unavailable and reports individual copy failures while continuing.
* Made Multus resource settings configurable, with defaults of 100m CPU
and 100Mi memory requests and a 4Gi memory limit. Rendering requires
memory requests and limits.
* Updated the Multus deployment to use the `cozy-multus` namespace and
DaemonSet name, and Cilium as the master CNI configuration.
* **Bug Fixes**
  * Added a 128Mi memory limit to the CNI installation step.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-12 17:24:46 +03:00
Aleksei Sviridkin 938abdc4d2 fix(platform)!: refuse a networking variant change on a live cluster (#4790)
## What this PR does

Changing the platform variant on a running cluster can delete its CNI,
and nothing stopped that before. A new platform variant (`spec.variant`
of the `cozystack.cozystack-platform` Package) can give the
`cozystack.networking` Package a different variant, and the operator
then deletes every HelmRelease of that Package the new variant does not
install. Moving from `isp-full` to `isp-hosted` deletes Cilium and
Kube-OVN, so every pod loses networking. Moving from `isp-full` to
`isp-slim` deletes Kube-OVN.

With this PR the platform render looks up the live
`cozystack.networking` Package. If its variant differs from the one
about to be rendered, the render fails, and the error names the value
that lets it through: `networking.acknowledgeVariantChange: "<new
variant>"`. I made the acknowledgement a variant name instead of a
boolean like `publishing.proxyProtocolAcknowledgeUnclean`. A leftover
`true` would let every later switch through unchecked, while a variant
name covers only the switch it names.

The guard does nothing when there is no live Package to compare against:
`helm template`, a client-side dry-run, or a first install. It is also
skipped when `cozystack.networking` is listed in
`bundles.disabledPackages`, because the platform does not render the
Package then. It lives in the shared Package helper and not in
`bundles/system.yaml`, so it covers both places that emit the networking
Package, including the `noop` call in the `isp-hosted` branch. The other
Packages whose variant follows the bundle, `cozystack-scheduler` and
`tenant-application`, keep the same release names in every variant, so
changing their variant deletes nothing and they need no guard.

The new helm-unittest cases cover the refused switches, an unchanged
variant, a matching and a mismatching acknowledgement, a bare boolean,
`disabledPackages`, an offline render, and a live cluster with no
networking Package yet.

A cluster where someone set the Helm-owned networking Package to
`cilium-kilo` by hand now fails its platform upgrade, and the error
points it at `bundles.disabledPackages`. Acknowledging does not help
there, because the render writes the bundle's variant over
`cilium-kilo`. helm-controller v1.5.0 installs a release with
server-side apply by default, which takes over every rendered field. On
a release it installed, the upgrade replaces the variant and the
operator deletes Kilo. A release first installed with Helm 3 keeps the
apply method recorded in its history, so there the outcome depends on
that history.

Known gap, left for a follow-up: the guard fires on any variant change,
including Talos to generic (`kubeovn-cilium` to
`kubeovn-cilium-generic`). Those pairs share release names, so nothing
is deleted and only values change. Refusing is safe, and the error does
not claim a deletion for such a switch.

Closes #4596

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

This PR adds a key to `packages/core/platform/values.yaml`, and the
website keeps a hand-written table of those keys in
`operations/configuration/platform-package.md`.

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/issues/746
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(platform): the platform render now refuses to change the variant of a live cozystack.networking Package, because the operator would then delete the CNI HelmReleases the new variant does not install (isp-full to isp-hosted removes Cilium and Kube-OVN). To go ahead with such a change, set networking.acknowledgeVariantChange to the new networking variant named in the error. A cluster whose cozystack.networking variant was set by hand (for example cilium-kilo) needs cozystack.networking in bundles.disabledPackages before the upgrade, otherwise the platform render refuses.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Changing the networking variant on a running cluster now requires
explicit confirmation naming the requested variant. A change without a
matching confirmation is blocked.
* The confirmation applies only to the specified switch and must be
updated for a later variant change. If the running package has no
recorded variant, it is treated as using the default variant.
* New installations without an existing networking package are
unaffected, and networking can still be disabled through its package
setting.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-12 17:24:09 +03:00
Aleksei Sviridkin ddd00f8639 feat(platform): pass multus resources from networking.multus.resources
The multus chart takes its resources from values, but the platform
renders the cozystack.multus Package with stageCniPlugins alone, and an
edit to that child Package is reverted by the platform reconcile. A
running cluster therefore had no way to raise the multus limits.

networking.multus.resources is passed to the Package only when set, so
an unset key leaves the chart default in charge. A value that is not a
map stops the render here; the multus chart keeps refusing a render
without a memory limit or request.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-10 20:59:26 +03:00
Aleksei Sviridkin 1245b32a32 fix(platform)!: refuse a networking variant change on a live cluster
Changing the variant of the cozystack.cozystack-platform Package on a
running cluster changes bundles.system.variant and can re-render the
cozystack.networking Package with another variant. The operator then
deletes every HelmRelease of that Package the new variant does not
install, so isp-full -> isp-hosted removes Cilium and Kube-OVN and the
cluster loses its CNI; isp-full -> isp-slim removes Kube-OVN.

The platform render now looks up the live Package and fails when its
variant differs from the one about to be rendered, unless
networking.acknowledgeVariantChange names the new variant. The
acknowledgement is a variant name rather than a boolean so that a value
left behind after one switch does not cover a later one. The check is
inert where lookup returns nothing (helm template, client-side dry-run,
first install) and skipped when cozystack.networking is in
bundles.disabledPackages, since the platform then does not render it.

A live variant set by hand, such as cilium-kilo, is pointed at
bundles.disabledPackages rather than at the acknowledgement: the
acknowledged render would write the bundle's variant over it.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-10 20:57:17 +03:00
Aleksei Sviridkin 87a64071a7 fix(cilium)!: take cilium's own addresses from a block kube-ovn does not hand out (#4778)
## What this PR does

fixes #3740.

on the kube-ovn variants cilium allocates no pod ips, but the agent
still takes its router ip (`cilium_host`, `CiliumInternalIP`) and the
gateway api ingress ip from its own allocation cidr. with `ipam.mode:
kubernetes` that's `Node.spec.podCIDR`, a /24 out of the same /16
kube-ovn hands pods from, so sooner or later kube-ovn gives a pod an
address cilium already holds. we hit it on three clusters, details in
https://github.com/cozystack/cozystack/issues/3740#issuecomment-6022394243.

**what changes**

- the kube-ovn variants run cilium with `ipam.mode: cluster-pool`, a /29
per node out of the last /21 of `networking.podCIDR` (`10.244.248.0/21`
for the default /16). the platform derives the block from
`networking.podCIDR` and passes the same range to kube-ovn as
`--default-exclude-ips`, so a fresh install never hands it to pods.
- the block sits inside the pod cidr on purpose. envoy reaches gateway
api backends from the node's ingress ip, and kube-ovn masquerades host
originated traffic only when the source is in `ovn40subnets`. with the
`100.65.0.0/16` pool from the issue a nodeport through the gateway
answered 200 only on the backend's node and 503 `upstream connect error
... connection timeout` on the other two. with the block inside the pod
cidr it's 200 from all three. worth a look on the ci side too. the ci
sandbox sets talos `podSubnets` to `100.65.0.0/16`, so cilium's ingress
ips there already come from outside `ovn40subnets`, and the gateway e2e
checks Accepted, Programmed and the lb service but sends no request
through a listener, so it wouldn't notice. with this pr the sandbox
takes its block from the pod cidr like everyone else.
- the end of the /16 is what kube-ovn reaches last. it hands out the
lowest free address and reuses released ones only when the free range
runs dry (`pkg/ipam/subnet.go` and `pkg/ipam/ip_range_list.go` at
v1.15.26). a /21 cut into /29s covers 256 nodes.
- the slim variants and guest clusters stay on `ipam.mode: kubernetes`.
cilium is the only cni there and `Node.spec.podCIDR` is the pod range.
- `networking.podCIDR` now needs /20 or wider on the kube-ovn variants,
otherwise the render fails and says why. an install that rendered before
can stop rendering at upgrade, hence the `!`. a /21 cut into /29s also
caps these variants at 256 nodes, a fresh install included.

**existing clusters**

a post-upgrade hook in the cilium chart moves them
(`templates/ipam-migration.yaml`, `files/ipam-migration.sh`). before it
touches anything it checks that the block has a /29 for every CiliumNode
and that it lies inside kube-ovn's default subnet, and stops with both
cidrs printed otherwise. a node cleared past the pool's capacity would
lose pod networking on its next agent restart, and a block outside the
subnet puts the agents outside `ovn40subnets` again. it has to be
post-upgrade. platform migrations run as the platform's pre-upgrade
hook, while every agent still runs `ipam.mode: kubernetes` and writes
`Node.spec.podCIDR` back into a cleared CiliumNode
(`pkg/nodediscovery/nodediscovery.go` at v1.19.5), so the move can only
happen after the cilium release switched the mode. the job runs on the
host network against the agent's own api endpoint (kubeprism on talos),
because it restarts the agents.

1. adds the block and every node's current ingress ip to `excludeIps` of
the default subnet, in one json patch with a `test` op, and reads the
subnet again on conflict, since kube-ovn rewrites that list into its own
order right after a patch. the old ingress ip goes in because cilium
1.19.5 keeps a former ingress ip in the agent's ipcache as
`reserved:ingress` after the move. we saw 2 to 3 such entries per node
hours later. old router ips leave the ipcache and go back to kube-ovn.
2. stops right there if an address in the block is still allocated,
names it, and moves no agent. it first checks the subnet status it
patched against, then waits until `status.v4availableIPrange` stops
offering the block, which is kube-ovn having applied the exclusion, and
then reads the `ips.kubeovn.io` objects of the default subnet. the
status alone can't answer it, kube-ovn leaves excluded addresses out of
`v4usingIPrange` (`GetSubnetIPRangeString` in `pkg/ipam/ipam.go` at
v1.15.26), so a pod still in the block would be invisible on a retry
where the exclusion is already in place. recreate those pods and the
next upgrade retry goes through.
3. one node at a time, clears the inherited /24 from
`spec.ipam.podCIDRs` (again with a `test` op, so a concurrent writer
fails the patch instead of losing an update), waits for the operator to
allocate a /29 from the block and restarts that node's agent. a node
that isn't ready is skipped and its agent picks up the block when it
starts. nodes already moved are left alone, so a retry after a timeout
just continues. the hook runs inside the cilium upgrade, so where
restarting every agent takes longer than the upgrade timeout, 10m by
default, the release first reports a failed upgrade and the move
finishes on the retry.

running pods keep their addresses. every agent restarts twice on
upgrade, once when the config change rolls the daemonset and once from
the hook, and while it restarts host to pod traffic on that node blips
for a few seconds because the router ip moves. a pod that sits on a
router ip today is reachable again after the move, the node no longer
holds that address. a pod on a node's current ingress ip keeps it and
that node keeps seeing it as `reserved:ingress`, so recreate it. the
hook logs those pods by name.

### Validation

- `helm unittest` passes cilium 23/23, platform 246/246 and kube-ovn
22/22. the new suites pin the rendered `cilium-config` keys per variant,
the hook shape, env and rbac, and the platform wiring for the talos and
generic branches, a custom pod cidr, host bits in the network address,
the refusals for a too small or non ipv4 cidr, and the slim variant.
- on a trial merge with main at cbc9d0c4e, where #3849 runs
`hack/*.bats` under real bats 1.14 with `set -u`, the 27 bats tests pass
after the second commit loads `test_helper`, the strict setup audit is
clean, and helm unittest passes cilium 23/23, platform 263/263 and
kube-ovn 22/22.
- `hack/cilium-ipam-migration.bats` passes 27/27 and runs the real
script against a fake kubectl that models the operator, the agent
daemonset and kube-ovn publishing the subnet status. thirteen mutations
turn it red each (exclusion after the move, no ingress exclusion, no
retry, no status check, an ingress ip inside the block listed on its
own, no `test` op, no wait for kube-ovn, no ip object check, no subnet
filter on the ip objects, no capacity check, a block outside the subnet
let through, overlap turned into containment, no log of ingress ip
holders). the containment one is pinned by a usage range that crosses
the start of the block, `10.244.247.250-10.244.248.3`.
- live on a 3 node talos test cluster with cozystack v1.6.4, cilium
1.19.5 and kube-ovn v1.15.26
- moved from `kubernetes` to the block, exclusion first and then the
node moves this script does. 3 nodes in about 40 seconds, every agent
took its router and ingress ip from its /29, no pod changed its address.
- gateway api through a nodeport 9/9 answered 200 and through the
loadbalancer ip 6/6, against 3/9 with `100.65.0.0/16`.
- the final hook job on the moved cluster is a no-op in 2 seconds,
including the ip object read with the job's own rbac. the same ip object
pipeline over that cluster's real objects finds 77 in a populated /24
and none in the block.
- the exact `test` plus `replace` patch on `ovn-default` goes through,
kube-ovn re-sorts the list right after, and the same patch against the
old list is rejected. that's the conflict path the retry covers.
- shellcheck is clean and `hack/check-commit-trailers.sh` passes.
- not run here. the upgrade move from an older release in ci, e2e only
covers the fresh install.

### Screenshots

not a ui change.

### Downstream repositories

walked the trigger map against the diff.
`packages/core/platform/values.yaml` only gains a comment, but
`networking.podCIDR` gets a new constraint on the kube-ovn variants,
which the hand written key table on the website should say. the ansible
role sets `networking.podCIDR` to `10.42.0.0/16`, which yields
`10.42.248.0/21` and needs nothing there. the talm preset keeps
`podSubnets: 10.244.0.0/16`, which no longer matters on these variants
because cilium stops reading `Node.spec.podCIDR` there. the terraform
provider doesn't model platform values.

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/pull/739
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(cilium)!: on the kube-ovn variants cilium now takes its router and gateway api ingress ips from the last /21 of networking.podCIDR, which kube-ovn no longer hands out, so pods stop landing on addresses cilium holds. existing clusters move on upgrade through a post-upgrade hook, and each cilium agent restarts twice, once for the config change and once in the hook. networking.podCIDR needs /20 or wider on these variants, and a /21 of /29s caps them at 256 nodes.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Kube-OVN-enabled variants now reserve the final `/21` of the pod CIDR
for Cilium router and Gateway API Ingress addresses, excluding it from
Kube-OVN pod allocation. Pod CIDRs must be IPv4 with a prefix of `/20`
or shorter.
* Added a post-upgrade migration that moves existing Cilium IP
allocations into the cluster pool and updates affected node networking.
* **Documentation**
* Documented the pod CIDR requirements and node capacity limit for
Kube-OVN variants.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-08 21:41:23 +03:00
Aleksei Sviridkin 0a2bb5ef98 fix(cert-manager): enable the Gateway API solver in the host cert-manager (#4039)
## What this PR does

With the platform Gateway enabled, the ClusterIssuers in
`cert-manager-issuers` and the per-tenant Issuers rendered for
`TenantGateway` carry an `http01.gatewayHTTPRoute` solver, but the
controller refuses it: the vendored chart never turns Gateway API
support on, so every HTTP-01 challenge behind the Gateway fails with
`gateway api is not enabled` and no certificate is issued.

This passes a `ControllerConfiguration` with `enableGatewayAPI: true` to
the host cert-manager through the component values of its platform
Package. The host ships the Gateway API CRDs on every variant and orders
cert-manager after them, so the flag is safe there.

It is deliberately not a default of the `system/cert-manager` chart. The
same chart is installed into tenant Kubernetes clusters, where the CRDs
are an opt-in addon, and cert-manager v1.20 exits at startup when
Gateway API support is on and the CRDs are missing. The tenant
HelmRelease in `packages/apps/kubernetes` already passes the same
`ControllerConfiguration` when its `gatewayAPI` addon is enabled, so
both clusters now opt in from the layer that knows whether the CRDs are
there, and the chart itself stays neutral.

The new tests pin both halves: the host Package carries the flag and the
preconditions that make it safe, and the chart wires `config` through to
the controller while shipping none by default.

Closes #3725

### Screenshots

Not applicable, no UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(cert-manager): enable the Gateway API solver in the host cert-manager, so ACME HTTP-01 challenges succeed when the platform Gateway is enabled
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Supported system bundles can now use cert-manager’s Gateway API
HTTPRoute solver to issue certificates.
- Hosted deployments include the required Gateway API custom resource
definitions, installed before cert-manager.
- Gateway API solver support remains opt-in in environments that do not
provide the required definitions, avoiding unintended changes to
certificate management there.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-07 17:47:41 +03:00
Aleksei Sviridkin 76b9e55102 fix(cert-manager): enable the Gateway API solver in the controller
With the platform Gateway enabled, the ClusterIssuers in
cert-manager-issuers and the per-tenant Issuers rendered for
TenantGateway carry an http01.gatewayHTTPRoute solver, but the
controller only honours that solver when Gateway API support is
switched on, and the vendored chart leaves it off. Every HTTP-01
challenge behind the Gateway then fails with "gateway api is not
enabled" and no ACME certificate is issued.

Pass a ControllerConfiguration with enableGatewayAPI to the host
cert-manager through the component values of its platform Package.
The host always ships the Gateway API CRDs and orders cert-manager
after them, so the flag is safe on every variant.

It is not a default of the system/cert-manager chart: the same chart
is installed into tenant clusters, where the CRDs are an opt-in addon
and a controller started with Gateway API support and no CRDs exits
at startup. The tenant side keeps opting in on its own.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
Assisted-by: LLM
2026-10-07 14:14:35 +03:00
Myasnikov Daniil d61fed46cc feat(local-storage): add opt-in OpenEBS LocalPV hostpath package
LINSTOR needs the DRBD and ZFS kernel modules, so a cluster without
them has no dynamic PV provisioning. OpenEBS LocalPV hostpath needs no
kernel modules and also serves deliberately node-local workloads.

The package is opt-in so it cannot change an existing cluster. Its
class is named cozystack-local-hostpath so it cannot collide with an
existing class name, and it is not the default so PVCs that omit
storageClassName keep their current binding. The vendored RBAC is
patched down to what volume reconciliation calls; helper Pods, Leases
and analytics state stay in the install namespace.

Images stay on docker.io, pinned by index digest.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 11:30:45 +03:00
Yan Bondarenko dc362fa591 fix(cilium)!: take cilium's own addresses from a block kube-ovn does not hand out
On the kube-ovn variants Cilium allocates no pod addresses, but the agent
still takes its router IP and the Gateway API Ingress IP from its
allocation CIDR. In ipam.mode kubernetes that is Node.spec.podCIDR, which
the documented installs slice from the same range kube-ovn allocates pods
from, so kube-ovn sooner or later hands a pod an address Cilium already
holds.

Cilium now runs ipam.mode cluster-pool on the kube-ovn variants, with a
/29 per node from the last /21 of networking.podCIDR, and kube-ovn
excludes that /21 through --default-exclude-ips. The block stays inside
the pod CIDR on purpose. Envoy reaches Gateway API backends from the
node's Ingress IP, and kube-ovn masquerades host-originated traffic only
when the source lies in one of its subnets, so a block outside them such
as 100.65.0.0/16 breaks Gateway API traffic that crosses nodes.

A post-upgrade hook moves existing clusters. It first checks that the
block has a /29 for every node and lies inside kube-ovn's default
subnet, and stops without changing anything otherwise. It adds the block
and every node's current Ingress IP to excludeIps of that subnet,
because the agent keeps a former Ingress IP in its ipcache as
reserved:ingress. It then waits until kube-ovn stops offering the block
and stops if an address in it is still allocated. kube-ovn leaves
excluded addresses out of the subnet status, so that check reads its IP
objects. Then, one node at a time, it clears the inherited /24 from the
CiliumNode, waits for the operator to allocate from the block and
restarts the agent.

networking.podCIDR now needs a prefix of /20 or shorter on the kube-ovn
variants, and the render fails otherwise. A /21 cut into /29s also caps
these variants at 256 nodes.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-10-06 22:11:02 +03:00
Alexey Artamonov 3497855e15 fix(platform): read the tracing rate only as plain decimal digits
Sprig's int64 reads a string with a leading zero as octal and turns a
parse error into 0, so "0100" would reach the collectors as 64, and
"08" or digits beyond int64 would leave the default in place without
a word. A float beyond int64 converts differently on each CPU
architecture, and the collector adds its replica count to the rate
before splitting it.

A string rate must now be plain decimal digits, and any rate above
1 TiB/s fails the render; the collector also falls back to the default
on digits it cannot hold rather than start with a rate of zero, which
the processor refuses.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-06 09:39:04 +03:00
Alexey Artamonov a0aea139fc fix(platform): refuse a malformed shared-central tracing rate
A rate written as a quantity or a fraction ("1Mi", "100KiB", 1.5) was
read as zero, so the platform wrote no key and every collector kept
the default without a word. The platform render now fails on anything
but a whole number of bytes per second. Helm reads a number from
values as a float64, which prints a million as 1e+06, so a whole
number is checked as a number and written as int64 digits.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-06 09:39:04 +03:00
Alexey Artamonov f8deb8e171 fix(platform): write the tracing rate only when it is positive
apps.yaml wrote the rate for any non-zero value, so a negative one
reached every tenant's cozystack-values Secret and upgraded every
release in every tenant, while the collectors ignored it and kept the
default.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-06 09:39:04 +03:00
Alexey Artamonov 177b207a27 feat(monitoring): hold each shared-central tenant to a write rate
The design has the per-tenant collector rate-limit its tenant, so that
a noisy or hostile tenant cannot exhaust a shared backend. The disk cap
alone does not do that: VictoriaTraces always keeps the last two days,
so a tenant writing faster than the cap allows over two days fills the
disk for every tenant on the store.

The platform sets the rate with
monitoring.tracingCentralTenantBytesPerSecond, and it reaches every
tenant in _cluster through the cozystack-values Secret. A change there is a change in the values each
monitoring HelmRelease reads, so Flux upgrades them all; a rate read
with lookup at render time would never reach a tenant whose release is
already in sync. The platform writes the key only when it is set, since
every release in every tenant reads that Secret. Without it the
collector holds the tenant to the rate of a default 10Gi store.

Each central tenant's collector splits the rate across its replicas
and refuses data above it before the batch processor, so the sender
sees the refusal. An OTLP/HTTP sender gets a 429 and retries; the
limiter answers gRPC with RESOURCE_EXHAUSTED and no RetryInfo, which
the OTLP specification makes final, so central tenants should export
over HTTP.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-06 09:39:04 +03:00
Aleksei Sviridkin de570a3806 feat(platform): add slim installation variants (#4595)
## What this PR does

This adds three minimal platform variants for small installs, such as
arm64 labs, where isp-full is too heavy. They are `isp-slim` for Talos,
`isp-slim-generic` for k3s, kubeadm or RKE2, and `isp-hosted-slim` for
clusters where the host provides CNI and storage.

A slim variant installs only the base platform: the engine, API and
dashboard, tenants, ingress and Gateway API. The two non-hosted variants
also get LINSTOR. objectstorage-controller stays on too, because
cozystack-controller watches BucketClaim unconditionally and crash-loops
without that CRD. Everything else is opt-in through
`bundles.enabledPackages`, including every paas and naas application and
its operator, monitoring, backups, etcd, SeaweedFS, metrics-server, VPA,
Multus and MetalLB. The iaas bundle is refused, so there are no VMs and
no managed Kubernetes. With the stock preset a slim variant emits 20
Packages (15 for hosted), where isp-full emits 77.

On the two non-hosted slim variants, networking is Cilium alone. Without
VMs nothing needs Kube-OVN, and Cilium L2 announcements take the place
of MetalLB. The admin creates a `CiliumLoadBalancerIPPool` and a
`CiliumL2AnnouncementPolicy`, or uses `publishing.externalIPs`. Pod
ranges come from `node.spec.podCIDR`, which Talos and k3s allocate by
default. `networking.encryption` only drives Kube-OVN IPsec, so the slim
variants refuse it instead of silently ignoring it.

The tenant application needs its own slim variant. The default one waits
on the monitoring, etcd and SeaweedFS applications, so on slim
cozystack-basics and tenant-root would never become ready. tenant-rd and
cozystack-basics still reference the default variant's artifacts by
name. Those artifacts exist anyway, because the PackageSource builds
them for every variant.

The full variants don't change. I rendered isp-full, isp-full-generic
and isp-hosted against main and got byte-identical Packages. The only
new output there is the slim variant in the tenant-application
PackageSource.

Things to know before using it:

- An opt-in package does not pull in its dependencies. The presets and
the docs list the chains. A package enabled without its chain stays
`DependenciesNotReady`.
- Slim is for new installs. Switching a live isp-full cluster to
isp-slim moves the networking Package from `kubeovn-cilium` to `cilium`,
the operator removes the Kube-OVN release, and running pods lose
networking. The other Packages are kept by `helm.sh/resource-policy:
keep`, so the switch doesn't make the cluster smaller either.
- No e2e suite runs a slim variant yet. Helm unit tests cover the
presets, opt-in, OIDC, the networking values and both refusals, and a
dependency check over every emitted Package passes for all three
presets.

I installed isp-slim from a build of this branch on a fresh three-node
Talos 1.13 cluster (amd64). All Packages and HelmReleases went Ready,
with no Kube-OVN, MetalLB, monitoring, backup or KubeVirt pods. From
outside, the dashboard and the API answered by name with valid
certificates. A LoadBalancer Service got an address from a Cilium pool
and answered through L2 announcements. Pod-to-pod traffic across nodes,
cluster DNS and a replicated LINSTOR volume worked. Opting in
postgres-operator and postgres-application gave a Ready Postgres. That
run found the BucketClaim crash-loop above, which is fixed here. The
base platform used about 215m CPU and 3.2 GiB of memory, not counting
the Kubernetes control plane.

### Screenshots

Not a UI change.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#718
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [x]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up: cozystack/ansible-cozystack#79
- [x] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
cozystack/ccp#23
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(platform): add the isp-slim, isp-slim-generic and isp-hosted-slim variants. They install only the base platform (engine, dashboard, tenants, ingress, gateway, and LINSTOR outside hosted) with Cilium-only networking and no MetalLB; applications, operators, monitoring, backups, etcd and SeaweedFS are opt-in through bundles.enabledPackages, and the iaas bundle is not available.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added the `isp-slim`, `isp-slim-generic`, and `isp-hosted-slim`
platform variants, with support for PaaS and NaaS bundles.
* Slim variants use Cilium networking and include a reduced set of
packages by default. Additional packages and their dependencies can be
enabled as needed.
* **Limitations**
* IaaS and the platform encryption toggle are not supported on slim
variants. Cilium-native encryption is not configured.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-01 14:03:22 +03:00
Aleksei Sviridkin d463e00da1 feat(platform): add publishing.serviceDomain for the dashboard Services tab
The dashboard has no way to show a connection hostname for a LoadBalancer
Service: before 1.4.0 the old UI composed one from the Service name and
the cluster's root host, and operators whose DNS publishes LoadBalancer
IPs under a dedicated zone still need that. Add the suffix as a platform
value and propagate it into the cozystack-values map as service-domain,
next to cluster-domain and branding, so the dashboard chart can read it.

The value is display-only and opt-in: empty (the default) means the
console shows no hostname, because nothing in the platform publishes DNS
for a guessed name. Leading and trailing dots are stripped here, the one
place every consumer reads from, so an operator's ".svc.example.org."
cannot render as "demo..svc.example.org."; the value is coerced to a
string first because an unquoted numeric scalar would otherwise abort
the whole platform render. The helm-unittest suite pins the empty
default, a configured value landing in the Secret, the dot trimming and
the numeric case.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-30 18:17:37 +03:00
Aleksei Sviridkin 506dc7a4a7 feat(platform): add isp-slim, isp-slim-generic and isp-hosted-slim variants
Small installs such as arm64 labs had to take the whole isp-full set:
KubeVirt, backups, monitoring, etcd, SeaweedFS and every paas/naas
operator. The slim variants keep only the engine, tenants, ingress,
gateway, LINSTOR storage and networking, and turn everything else into
opt-in packages listed in bundles.enabledPackages. Full variants render
the same Packages as before.

Networking on isp-slim and isp-slim-generic is Cilium alone: without
VMs nothing needs KubeOVN, and Cilium L2 announcements take the place of
MetalLB, which stays available as an opt-in package. KubeOVN IPsec is
what networking.encryption drives, so the slim variants refuse it
rather than ignore it. isp-hosted-slim keeps the noop datapath.

tenant-application gains a slim variant whose dependsOn drops the
monitoring, etcd and seaweedfs applications; the default variant waits
on them, which would leave cozystack-basics and tenant-root never Ready.
The iaas bundle stays limited to the isp-full variants.

objectstorage-controller stays on everywhere: cozystack-controller
watches BucketClaim unconditionally, so without that CRD its manager
never syncs its caches and crash-loops.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-30 03:12:00 +03:00
Marian Koreniuk 18b83a72b3 feat(platform): make the Proxmox substrate installable from a bundle
Until now the two Cluster API packages behind the Proxmox substrate existed but
no bundle emitted them, so a Cozystack install shipped neither and the substrate
was reachable only by applying Package CRs by hand.

They join the iaas bundle as opt-in packages, where the KubeVirt infrastructure
provider is unconditional. The asymmetry is the point: capmox drives a
hypervisor outside this cluster and needs a Secret holding a Proxmox API token,
which capi-operator reads before it installs anything — with no Secret the
provider is never deployed at all. Enabling it by default would ask every
Cozystack install that has no Proxmox to supply hypervisor credentials.

Enabling capmox brings the in-cluster IPAM provider with it — capmox allocates
worker addresses from an InClusterIPPool and has no DHCP mode, so on its own it
never gets a ProxmoxCluster to Ready. IPAM stays independently enableable,
since assigning addresses is a generic Cluster API concern rather than a Proxmox
one.

Naming capmox in enabledPackages while disabling IPAM stops the render. The
cluster would catch that combination on its own: the capmox PackageSource
depends on the IPAM Package, so with that Package absent capmox goes
Ready=False with DependenciesNotReady and no HelmRelease is ever created. What
it would not do is say why. Failing the render names the two conflicting list
entries at the moment someone writes them, instead of leaving a condition to
decode later.

The negated containsDocument asserts in the wiring suite carry `any: true`.
Without it a negated containsDocument passes whether the document is in the
render or not, so the two properties the suite exists for — off by default,
and disabledPackages winning — could not fail.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:25:00 +02:00
myasnikovdaniil 9bec21461d feat(kubernetes): boot tenant worker disks from a shared golden Talos image (CDI clone) (#4171)
## What this PR does

Tenant worker boots from talos OS disk image that CDI streams over HTTP
from image factory, and it does that once per worker. Same bytes N
times, and N chances for a fetch to fail. In closed or rate-limited
environment that is wrong shape.

This PR adds opt-in catalog and per-pool selector that reads from it.

`packages/system/kubernetes-worker-image` imports one golden DataVolume
into `cozy-public` per `(schematicID, version)`, named
`talos-worker-<schematicID>-<version>`. Pool opts in with
`osImage.builtin` on its own `KubernetesNodes`, and its `disk-system`
DataVolume then carries `source.pvc` pointing at that golden, so CDI
does storage layer CSI clone instead of HTTP import. One import per
flavor, shared by every worker of every tenant.

Catalog is disabled by default, enable by adding
`cozystack.kubernetes-worker-image` to `bundles.enabledPackages`. Its
knobs (`imageFactoryURL`, `storageClass`, `storage`, `images`) are set
through `spec.components.platform.values.kubernetesWorkerImage` on
cozystack-platform Package. That needed optional package helper to learn
components argument, which `cozystack.platform.package.optional.default`
did not have, so every optional package registered through that wrapper
could name variant and nothing else. Thirteen other packages get same
path for free.

`osImage` has two mutually exclusive forms:

- `osImage.builtin` boots pool workers by CDI-cloning golden from
catalog. Clone is storage layer CSI copy, so OS image never crosses pod
network and factory answers once for whole fleet instead of once per
worker.
- `osImage.factory` keeps HTTP import and lets pool pin own
`imageFactoryURL`, `schematicID` and `version`. This is bring-your-own
path for operator running own factory or caching mirror, and it is what
pool gets when `osImage` is omitted.

In-guest talos installer written by reconcile job follows same
resolution, so pool that pins release boots it and upgrades into it
instead of being flipped back to cluster default halfway.

This is pool half of what #3950 asks for. That issue wants platform
level defaults for two channels, worker OS image and in-guest registry
mirrors. Here image channel gets per-pool source and shared cache,
platform default plumbing is left to the issue.

## Existing workers do not roll

With `osImage` omitted rendered `KubevirtMachineTemplate` is byte
identical to what chart produced before. KMT is named by hash of its own
content, so any drift would re-image every worker VM in fleet on next
upgrade. Stored render snapshot is untouched by this PR, that is the
evidence and not the claim. Editing `osImage` on existing pool re-images
that pool, like changing any other image reference.

## What fails at render

Clone that would silently degrade into something slower or wrong is
refused instead of rendered.

Golden absent, or its import in terminal `Failed` phase. Pool
`storageClass` different from golden one, because CDI cant CSI-clone
across storage classes and falls back to host assisted copy over pod
network, which is exactly the transfer `osImage.builtin` removes. Empty
class on either side resolves against cluster default class before
comparing, so pool at `storageClass: ""` is checked too. `diskSize`
below golden storage, because clone target cannot be smaller than
source. Talos release the matrix lists as unsupported, whether pool
reached it through `talos.version` or through `osImage.*` override.
Matrix passes talos minor it does not list, so what it promises is that
known-bad pairing is refused, not that every pairing is known.
`schematicID` that is not plain lowercase alphanumeric, or version that
is not `vMAJOR.MINOR.PATCH`, both are interpolated into DataVolume name
and image URL while `values.schema.json` types them as bare strings.

Golden that is only mid-import is not an error. CDI holds worker disk
until source populates, and failing there would abort whole pool
HelmRelease for length of a download. Same reasoning covers pool that
already cloned: absent or `Failed` golden fails render only for pool
still waiting for its first disk, otherwise reconcile of healthy pool
would abort over source it no longer reads.

Golden that declares no StorageClass is read from its PVC, not from
whatever default is in force today, because those differ once cluster
default moves. Cluster with two default classes resolves same way
Kubernetes does, newest wins. Pool that already cloned is identified by
annotation naming exactly that pool, not by disk name prefix, because
pool `md0-gpu` produces disks carrying `md0` prefix, and it reads its
own class from its own bound PVC, so admin moving
`storageclass.kubernetes.io/is-default-class` does not fail release of
pool whose disks are all on right class.

That resolved class is then written onto worker disk. Reading one class
and binding another is how pool at empty `storageClass` passed the
comparison and still took host assisted copy on its next worker, so
guard and disk now come from one value. HTTP path leaves empty class to
cluster as before, which is what keeps its render byte identical.

Catalog side refuses duplicate `(schematicID, version)` entries, entries
whose schematic or version cannot produce name pool can reconstruct, and
edit to `storage`, `storageClass` or source URL of entry whose golden
already exists. Last one is not patchable in place, so without guard it
lands as rejected patch on next upgrade.

Goldens carry `helm.sh/resource-policy: keep`, so dropping entry stops
managing golden instead of deleting it. Missing golden aborts whole
Kubernetes HelmRelease of every tenant pinned to it, not just the disk,
and each cloned worker disk annotates golden it came from, so "does
anything still clone this" is answerable before manual delete.

Three drift guards live in
`hack/check-worker-image-catalog-defaults.bats`, because values that
have to agree sit in files nobody edits together. One-sided talos bump
trips all three.

## Upgrade

Three pool values that never produced working image rendered before and
hard-fail now, so pool carrying any of them flips its HelmRelease to
`InstallFailed` on upgrade and not on next edit: `talos.version` in
`vMAJOR.MINOR` form, `schematicID` that is not plain lowercase
alphanumeric, and empty `talos.imageFactoryURL`, which used to render
schemeless `/image/...` source URL. None of the three could boot worker,
so this is failing earlier and not a regression, but it surfaces on pool
nobody touched and fix in each case is to correct the value. CR carrying
such value is admitted by API with value intact, and later unrelated
edit reaches `.spec.values` of HelmRelease while it stays
`InstallFailed`, so edit is taken everywhere and applied nowhere.

## Why e2e does not exercise the clone path

Clone is storage layer copy, so it needs StorageClass whose volumes are
reachable from any node. Merge-gating e2e has none: those lanes run on
talos containers, which share runner kernel and cannot load DRBD, so
node-local `local` is only class there. Nothing would arrange the
co-location such a class requires either, golden in `cozy-public` has no
consumer and binds where CDI importer landed, while worker disk binds
where its VM is scheduled.

So the suites keep importing over HTTP with `osImage` omitted, which is
also the render the no-roll guarantee rests on. This PR touches no e2e
harness file: `hack/` differs from main only by the two guards this
feature owns, one tying catalog default golden to pool chart talos and
StorageClass defaults, one on in-guest installer heredoc.

Switching suites over does not work on that substrate. Golden asks for
`replicated`, lane does not have it, CDI resolves no StorageProfile, and
catalog HelmRelease times out on DataVolume stuck in `ImportInProgress`.
Passing lane own class would fix the class and leave the co-location.

Nightly and tag build run same suites on QEMU with DRBD and
`replicated`, which is the substrate clone path was verified on. That is
where e2e coverage belongs, as its own change with that lane storage
headroom measured rather than assumed.

Worth being straight about what this leaves, because it bears on whether
the feature earns its place. The e2e argument was never the strong one:
golden fetches from image factory exactly once, and #3513 established
that node-join stall tracks host `kvm_amd` `vgif` flag rather than
registry egress. What clone genuinely removes is the fanout, N workers
streaming over pod network becomes N storage layer copies. That is
property of storage layer, so it is worth having where DRBD is and worth
nothing where it is not. This ships as production capability with the
constraint stated on the field, not as CI improvement.

## Testing

On this tree:

- full `make unit-tests` plus `make test-controllers`: 95 charts, 2520
helm-unittest cases, 1710 bats cases, `go test ./pkg/...` and
`./internal/...`, exit 0
- `pre-commit run --all-files`: exit 0, root `make generate` and per-app
generate both clean, so no generated-file drift
- render snapshot unchanged, which is what pins byte identical render
for pool that does not adopt `osImage`
- mutation pass over render guards, each term deleted or inverted in
turn and suite required to notice it by name. Restoring presence gate on
catalog StorageClass comparison reddens exactly one case, inverting its
`ne` to `eq` reddens five. Reverting worker disk to omit its resolved
class reddens three cases that pin where builtin disk binds, and forcing
that class non-empty on every path reddens HTTP-path case and all three
render snapshots, with snapshot diff naming the changed KMT hash

`E2E Tests` passed on run 35831830568. That run was on the previous
head, 4fd8e7ba, and this head differs from it only by one test comment
and the commit message.

## Verified on a live cluster

Throwaway three-node talos stand (OCI, talos v1.13.6, LINSTOR/ZFS,
`replicated` StorageClass) installed from this branch `packages/` tree.
This covers what neither `helm template` nor unit suites can answer. It
predates the change that writes resolved StorageClass onto worker disk,
and that stand ran with explicit `replicated` on both sides, which is
the value that change now renders.

- The catalog wires up from `bundles.enabledPackages` alone:
`PackageSource cozystack.kubernetes-worker-image` reconciled, its
HelmRelease installed, and the golden `talos-worker-<schematic>-v1.13.6`
imported to `Succeeded` in 108s (6Gi, RWX, `replicated`).
- A pool on `osImage.builtin` clones it on the storage layer. Both
worker DataVolumes went `CSICloneInProgress` then `Succeeded` in 43s
with `Running=False reason=Completed message="Clone Complete"`, carrying
`source.pvc` and no `source.http`, and no importer or upload Pod
appeared in the tenant namespace.
- The clone yields a bootable talos root disk. Both worker VMs reached
`Running`/`Ready`, joined the tenant cluster and went `Ready` about 66s
after creation, and the tenant cilium, coredns, csi, ingress-nginx and
metrics-server addons all installed on them.
- Access modes match across the hop: golden `ReadWriteMany` 6Gi to
worker `ReadWriteMany` 20Gi on the same `replicated` class, grown by
CDI.
- The written annotations are correct in practice: each worker disk
carries `kubernetes-worker-image.cozystack.io/golden` naming the golden
and `.../pool` naming its pool.
- helm-controller converges on a builtin pool: forcing a reconcile left
the content-hashed KMT name unchanged and the VMI and DataVolume UIDs
identical, so a reconcile does not roll the pool.
- Cross-namespace clone RBAC is in place as shipped: `create datavolumes
--subresource=source -n cozy-public` resolves to `yes` for a tenant
ServiceAccount through the `cdi-clone-dv` RoleBinding, and CDI runs with
`cloneStrategyOverride: csi-clone`. This PR adds no RBAC of its own.

### Screenshots

Not applicable, no UI change.

### Downstream repositories

Trigger map points at two repositories.

Provider models every pool field by hand, so `osImage` needs schema,
model and expand/flatten pair there. Already tracked as
cozystack/terraform-provider-cozystack#39, filed when the field was
still called `image`.

`content/en/docs/next/operations/configuration/platform-package.md` is a
hand-written table of `spec.components.platform.values.*` keys and
nothing generates it, so `kubernetesWorkerImage` needs a row there.
Filed as cozystack/website#707.

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/issues/707
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
https://github.com/cozystack/terraform-provider-cozystack/issues/39
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(kubernetes): tenant worker pools can boot their system disk from a shared golden talos image instead of importing it from the Image Factory once per worker. Enable the opt-in `cozystack.kubernetes-worker-image` package via `bundles.enabledPackages` to publish one golden DataVolume per `(schematicID, version)` in `cozy-public`, then set `osImage.builtin` on a `KubernetesNodes` pool to have CDI clone it on the storage layer. A pool that omits `osImage` keeps importing over HTTP and renders exactly as before. The clone path needs the pool and the golden on the same StorageClass, and one whose volumes are reachable from any node, in practice a replicated/DRBD class.
```
2026-09-24 12:10:07 +05:00
Myasnikov Daniil bc8c54d6e6 feat(kubernetes): boot tenant worker disks from a shared golden Talos image
Every tenant worker VM imports the same Talos raw image from the Image Factory
over HTTP, once per worker: a 20-worker cluster fetches the identical ~4.15 GiB
artifact 20 times, and each fetch is a live dependency on the Factory at exactly
the moment a node joins.

packages/system/kubernetes-worker-image is a new opt-in catalog that holds one
golden DataVolume per (schematicID, version) in cozy-public, named
talos-worker-<schematicID>-<version>. A worker pool opts in with osImage.builtin
on its own KubernetesNodes resource, and its disk-system DataVolume then carries
source.pvc pointing at that golden, so CDI performs a storage-layer CSI clone
instead of an HTTP import -- one import per flavor, shared by every worker of
every tenant. Confirmed on a LINSTOR-backed cluster: the worker disk is a CSI
clone plus a resize, with no host-assisted upload pod.

Both halves are off unless asked for. The catalog is registered through the
platform's optional-package helper and disabled by default, so a cluster that
does not enable it creates no golden and keeps importing over HTTP. The disk
source is derived entirely from osImage, which is absent by default, so a pool
that does not set it renders byte-identical to the pre-feature chart and
adopting the feature never rolls existing workers. Editing osImage on an
existing pool re-images that pool, as changing any other image reference does.

What the render refuses, rather than degrading into a silent fallback:

- A pool whose golden has no catalog entry, or whose golden terminally failed to
  import. A golden that is merely mid-import is not an error; CDI holds the
  worker disk until the source populates.
- A pool whose StorageClass differs from its golden's. CDI cannot CSI-clone
  across classes and falls back to a host-assisted copy over the pod network,
  which is the transfer this feature exists to remove. An empty class on either
  side means the cluster default and is resolved before the comparison, and a
  pool that has already cloned reads its class from its own bound PVC rather
  than from today's default -- the golden's side is frozen the same way, and an
  admin moving storageclass.kubernetes.io/is-default-class must not fail the
  release of a pool whose disks are all on the right class. That resolved class
  is then written onto the worker disk, so the disk binds where the guard
  checked instead of on whatever the default has become by then -- reading one
  class and binding another is how a pool at an empty storageClass passed the
  comparison and still took the host-assisted copy. The HTTP path leaves an
  empty class to the cluster as before, which is what keeps its render, and the
  KubevirtMachineTemplate hash named after it, byte-identical.
- A diskSize below the golden's storage size, which a clone target cannot be.
- A schematicID that is not a plain lowercase alphanumeric identifier, or a
  version that is not a vMAJOR.MINOR.PATCH release. Both are interpolated into
  a DataVolume name and an image URL, and values.schema.json types them as bare
  strings, so a malformed one otherwise reaches the render. The catalog applies
  the same patterns, since both sides build the same name from the same fields.
- An images[] entry edited after its golden exists. A DataVolume spec cannot be
  patched in place, so the message names the entry and the supported move -- a
  new (schematicID, version) -- instead of leaving the API server to reject the
  patch on the next upgrade.

Goldens carry helm.sh/resource-policy: keep, so dropping an entry stops
managing a golden rather than deleting it: a missing golden aborts the whole
Kubernetes HelmRelease of every tenant pinned to it, not just the disk. Each
cloned worker disk annotates the golden it came from and the pool that owns it,
so "does anything still clone this?" is answerable before a manual delete
instead of after.

A builtin pool needs a StorageClass whose volumes are reachable from any node,
in practice a replicated/DRBD one. The golden in cozy-public has no consumer and
binds as soon as it imports, wherever CDI's importer landed, while a worker disk
binds where its VM is scheduled, and nothing here arranges for those to be the
same node. The platform ships cloneStrategyOverride: csi-clone, which rules out
the host-assisted strategy that would cross nodes. What a node-pinned class does
in that situation is not measured, so values.yaml states the requirement and not
a symptom.

ComputePlane pools forward only the per-pool shape fields, and osImage is
excluded there by name: the worker OS image, its Talos installer and its Image
Factory are operator decisions about the platform's own images.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-23 15:35:44 +03:00
Aleksei Sviridkin 08bbed3fb5 feat(talos-log-collector): add talos-log-collector package (#3260)
## What this PR does

Adds `talos-log-collector`, an optional system package: a node-local
Vector
DaemonSet that receives Talos machine and kernel logs and forwards them
to the
tenant VictoriaLogs (vlinsert), alongside the container/audit/event logs
already collected by monitoring-agents.

Talos keeps system and kernel (kmsg) logs inside the OS and does not
write them
to a host file the Fluent Bit `tail` input could read, so these logs are
currently never collected. Vector receives them over the node loopback
(Talos pushes to 127.0.0.1:5170). Talos exposes the two streams through
independent config paths, and both must point at the loopback socket:
`machine.logging.destinations` forwards the JSON-lines service logs, and
a
`KmsgLogConfig` document forwards the kernel (kmsg) stream (including
DRBD
kmsg). The node-side setup for both is in the README.

Design notes:
- The pod stays on the pod network so cluster DNS resolves vlinsert with
no
  extra config. Talos pushes to 127.0.0.1:5170; a `hostPort` bound to
`hostIP: 127.0.0.1` publishes that loopback socket into the pod via
portmap,
  keeping the receiver off the host network (no node-wide pod exposure).
- The receiver binds inside the pod netns, so a NetworkPolicy is
load-bearing:
a CiliumNetworkPolicy restricts ingress to the host entity when Cilium
is
present, otherwise a default-deny-ingress `networking.k8s.io/v1`
fallback
  keeps the socket protected on non-Cilium variants.
- Runs non-root, read-only root filesystem, all capabilities dropped.
- Optional package (opt-in via `bundles.enabledPackages`): inert until
each
Talos node is configured to push logs. Node-side setup is in the README.

Verified end-to-end on a Talos v1.12 dev cluster: service and kernel
logs
(including DRBD kmsg) reach VictoriaLogs; pods run non-root with 0
restarts.

### Release note

```release-note
feat(talos-log-collector): add talos-log-collector package to collect Talos machine and kernel logs into VictoriaLogs
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added an optional node-local Talos log collector (via system bundle
packages) that forwards Talos service and kernel logs over loopback to
the platform log ingestion endpoint.
* Introduced the collector Helm chart (Vector-based) with configurable
log level, loopback TCP receive port, digest-pinned image, default
resources, and routing target.
* Added conditional Cilium network policy support with a safe
deny-by-default fallback.
* **Security**
* Hardened the collector to run non-root with reduced privileges,
read-only filesystem, and dropped Linux capabilities (and disabled
service account token automount).
* **Documentation**
* Documented configuration requirements (including kernel log
enablement), operational behavior, and teardown steps.
* **Build & Release**
  * Extended the build to include building the collector’s image.
* **Tests**
* Added Helm rendering/contract tests covering daemonset, configmap,
service account, and network policy.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-22 23:49:33 +04:00
Kirill Ilin c96fe3d3ef fix(platform): refuse master as the OIDC realm name
The realm named here is where keycloak-configure creates every platform
client and the cluster-admin role, and where dashboard users sign in. Naming
master puts all of that inside the realm that holds the Keycloak super-admin,
and publishes its login page wherever the platform realm is published — and
the guard belongs where the value is set, since the Keycloak chart only reads
it once a separate admin hostname is configured.

Assisted-By: LLM
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
2026-09-22 10:35:31 +05:00
Alexey Artamonov 142a14cb69 fix(platform): reject an empty realm name instead of coercing it to cozy
realmName resolved through `| default "cozy"`, which fires on both null
and "". Unset (null) legitimately means the default, but an explicit ""
is a meaningless value and coercing it hid the mistake. Split the two:
null falls back to cozy, an empty string reaches the charset guard and is
rejected — the same null-vs-"" distinction keycloakInternalUrl already
draws a few lines below.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-21 12:15:10 +03:00
Alexey Artamonov ff637ed9a2 fix(platform): validate the realm-name charset and regenerate kubernetes artifacts
The realm-name guard checked only the type and echoed the parsed value
back, so its hint told an operator who wrote `on` to set realmName:
"true" — the token YAML had already discarded. Drop the echo and add a
slug-charset check: the name is spliced verbatim into URL path segments
and a shell token, so a "/" or a space must be rejected at the source
rather than split a URL or the hook script downstream.

The description edit in the previous commit was hand-applied to the
generated kubernetes README and never reached the Go types, leaving
pre-commit red; regenerate both with cozyvalues-gen. Finish the
oidc-tenant System-mode prose that still read cozy as a fixed name.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-21 12:15:10 +03:00
Alexey Artamonov 949096694b fix(platform): reject non-string realm names and shell-quote the hook realm
A realmName that YAML resolves to an int or a bool (a bare 123, or a
reserved word like on/off) was coerced with toString, silently renaming
the realm to something the operator never typed (on -> "true"). Fail the
render with a quoting hint at the source instead of surfacing it later as
a broken issuer URL.

The update-email hook Job interpolated the realm into its shell script
unquoted, so a name carrying a space or a shell metacharacter could break
or inject into the script; single-quote it. The Grafana and managed-cluster
issuer helpers build the URL with printf %s straight off the channel value,
so guard them with toString for the same non-string reason.

Name the configured realm in the tenant kube-apiserver oidc.mode
description and the Grafana identity-mode comment, which still read cozy as
if it were fixed.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-21 12:15:10 +03:00
Alexey Artamonov 80d32b3ab7 fix(keycloak): complete realm-name coverage and guard non-string names
The enable-update-email hook Job in keycloak-configure still pinned
REALM=cozy, so on a custom realmName it waited on a realm the chart never
provisions and the cold install timed out — the one seventh consumer left
behind, and the only value nobody could set without breaking the install.

A bare-integer or YAML-reserved realm name reached the printf %s that
builds the derived keycloakInternalUrl as a Go int, rendering a
"%!s(...)" format error into the URL; coerce the name to a string once
at the source.

Document that renaming the realm on a running cluster provisions a fresh
empty realm and orphans the accounts in the old one, and note in the OIDC
tenant and grafana guides that the realm name is now the configurable
default. info/Makefile carried NAME=etcd, so cozyhr drove the wrong
release from that package; correct it while touching the file.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-21 12:15:10 +03:00
Alexey Artamonov 44a9a8e735 feat(platform): make the Keycloak realm name configurable
The realm name "cozy" was hardcoded in keycloak-configure (which owns
the realm, its clients and the cozystack-cluster-admin roles), in the
dashboard and linstor-gui gatekeepers, in the info kubeconfig, and in the
Grafana and managed-Kubernetes apiserver OIDC issuers. Operators could
not rename the realm without patching vendored templates, which
`make update` reverts. The apiserver OIDC issuer is already configurable
via talm oidcIssuerUrl, so pointing it at a custom realm left every SSO
component on "cozy" and broke login with an issuer mismatch.

authentication.oidc.realmName (default "cozy") now rides the _cluster
channel so every consumer resolves the same realm, and the default
keycloakInternalUrl is derived from it so the realm is named in one place.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-21 12:15:10 +03:00
Timofei Larkin c398e0e116 Merge branch 'main' into chore/drop-http-cache
main took migration 57 and moved migrations.targetVersion to 58 while this
branch was open, so that line is the one conflict: 57 at the merge base, 58
on main, 59 here. Resolved as 59, which is what this branch's migration 58
requires, since run-migrations.sh loops up to targetVersion - 1.

Merged rather than rebased because the branch carries commits that are not
mine to rewrite.

Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2026-09-18 13:20:03 +03:00
Yan Bondarenko 5ed076cff8 feat(kubevirt): expose migration configuration through platform values
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-09-18 13:24:36 +04:00
Andrei Kvapil 06bd17621f chore(platform): stop shipping the http-cache package
Drops the cozystack.http-cache-application PackageSource and its entry
in the naas bundle.

No migration accompanies this. Platform Package objects carry
helm.sh/resource-policy: keep, so an existing cluster keeps its
http-cache Package and its http-cache-rd HelmRelease, and with them the
ApplicationDefinition that makes the kind resolvable. Instances that are
already deployed therefore keep running and stay addressable; the
platform simply stops delivering and updating the application. Removing
those objects would be the more invasive choice, not the safer one,
because there is no successor in the catalog to migrate an instance onto.

The httpcaches grant in cozy:tenant:admin:base stays for now. It is the
only rule in the tenant admin chain that can delete an instance, and no
tenant role carries any verb on helm.toolkit.fluxcd.io, so without it
the owner of a surviving HTTPCache could not remove it at all.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-09-14 13:11:00 +02:00
Aleksei Sviridkin 1f04e31ed3 feat(backups): add Redis backup/restore strategy driver (#3992)
## What this PR does

Adds backup/restore support for the `Redis` app (spotahome
`RedisFailover`) via a dedicated `strategy.backups.cozystack.io/Redis`
strategy controller. The spotahome operator ships no backup CRD, so the
driver follows the generic-script-Job pattern (a specialisation of the
Job/Altinity drivers): it runs a one-shot Job that dumps an RDB snapshot
from the current master to the system bucket and replays it on restore.

- **Backup**: discover the current master via the operator's sentinel
Service (`rfs-<name>`, master `mymaster`) → `redis-cli --rdb` → `curl
--aws-sigv4` PUT to `s3://<bucket>/<namespace>/<app>/dump.rdb`.
- **Restore**: GET the object → convert RDB → RESP (`hdt3213/rdb`) →
`FLUSHALL` + `redis-cli --pipe` into the target's master. Both
**in-place** and **to-copy** are implemented; a to-copy restore reads
the source's prefix via `.Backup.ApplicationRef.Name` while writing into
a differently-named target.
- **PITR**: not supported — RDB is a discrete snapshot with no
continuous log to replay to an arbitrary time (the etcd-like case).

The driver is a specialisation of the Job/Altinity script-Job flow with
a `Redis` applicationRef gate and an app-readiness precondition
(surfaces a precise `Ready=False` instead of a silent hang; blocks only
on an explicit `Ready=False` so a missing condition never wedges
backups). Strategy parameters round-trip through the Backup's
`DriverMetadata`; there is no `underlyingResources` snapshot (the Job
pattern re-renders on restore).

### System bucket

Coordinates and credentials reach the Job as env `secretKeyRef` from the
projected `cozy-backups-creds` (the ClickHouse/Altinity delivery model),
so **the app chart is untouched** — no `backup.useSystemBucket`, no new
RBAC (`batch/jobs` and `apps.cozystack.io/*` are already granted). The
bare-host `endpoint` gets `https://` prepended when unscheme'd; against
a self-signed endpoint the strategy verifies with the CA from
`backupStorage.endpointCASecretName` when set, otherwise against the
image's system CA store; skip-verify is a separate explicit
`backupStorage.insecureSkipTLSVerify` opt-in, never a fallback. The
`cozy-default-redis` strategy is gated on `$bucketName` (CRD-ordering)
and bound in the `cozy-default` BackupClass.

### Changes

- `api/backups/strategy/v1alpha1/redis_types.go` + generated CRD and
deepcopy.
- `internal/backupcontroller/redisstrategy_controller.go` (+ unit tests)
wired into both dispatch switches, `supportedBackupStrategyKinds`, and
the no-op restore-cleanup case.
- `cozy-default-redis` strategy template + `cozy-default` BackupClass
route + helm rendering-gate coverage.
- `images/redis-backup` (redis-cli + curl + RDB→RESP converter) +
Makefile target + `backupStrategyController.redisBackupClientImage`.
- `examples/backups/redis/` — an honest marker round-trip for both
restore flows.

### Validation

`go build`/`vet`, `go test ./internal/backupcontroller/...
./api/backups/...`, `make generate` (clean), and `helm unittest` (10/10)
all pass. Not yet cluster-verified.

## Downstream

None.

```release-note
Add backup and restore support for the Redis application (spotahome RedisFailover) via a new Redis backup strategy, wired into the cozy-default BackupClass. Supports in-place and to-copy restore; PITR is not supported.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added Redis backup and restore support using RDB snapshots in
S3-compatible storage.
  * Supports restoring data in place or to a separate Redis instance.
* Added configuration for Redis backup images, custom HTTPS certificate
authorities, and TLS verification.
* Added a complete Redis backup and restore demonstration with automated
setup, verification, and cleanup.

* **Documentation**
* Documented Redis storage, credentials, endpoints, TLS, and restore
behavior.
  * Added administrator and tenant scenario guides.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-10 19:17:26 +04:00
Aleksei Sviridkin 857ad4a128 fix(opensearch-operator): resolve in-cluster names on the platform domain
The operator reads DNS_BASE and builds every in-cluster name it needs
from it: the securityconfig job target, its own OpenSearch API client
URL, the Dashboards host, and the SANs it writes into node
certificates. The vendored chart defaults that to cluster.local, so on
a cluster whose domain is cozy.local the securityconfig job never
resolves its target, the .opendistro_security index is never created,
and the OpenSearch nodes never go Ready.

The domain the platform already holds could not reach the operator
before: _cluster stops at each package's own templates and never
reaches a vendored subchart, and the wrapper's values.yaml is not
templated. Carry it in the shared values Secret under the subchart key
instead, the same way the KEDA cluster domain travels, so the vendored
chart picks it up with no patch and a package rendered without the
Secret keeps the upstream cluster.local fallback.

A global key would have worked for the operator but reaches every
release on the cluster, and the vendored smee chart in bootbox cannot
render once global is non-empty. Scoping the value to the subchart
that consumes it avoids putting every other package on that path.

The diagnosis and the first working fix came from fuad00 in an earlier
pull request.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-09 23:56:12 +03:00
scooby87 afc003c464 feat(postgres): database read-replica autoscaling via KEDA (#3954)
## What this PR does

Adds automatic horizontal autoscaling of a PostgreSQL database's **read
replicas** in response to load, implementing the merged design proposal
[Database Horizontal
Autoscaler](https://github.com/cozystack/community/tree/main/design-proposals/database-horizontal-autoscaling)
(community#44). The mechanism is entirely stock — no bespoke operator,
no new CRD:

- **KEDA as an optional platform component** (`packages/system/keda`,
wired into `core/platform` as a source + system-bundle entry). It claims
the `external.metrics.k8s.io` APIService that nothing served before.
- **`cozy-lib.keda.scaledObject` helper**
(`packages/library/cozy-lib/templates/_keda.tpl`) renders a KEDA
`ScaledObject`, fail-closed on missing/invalid arguments.
- **`packages/apps/postgres` autoscaling** — an `autoscaling` values
block (`enabled`, `minReplicas`, `maxReplicas`, `metric`, `target`,
`maxReplicationLagSeconds`, `dryRun`, `transition`) validated by
`values.schema.json`; under active autoscaling the chart renders a
`ScaledObject` and seeds `spec.instances` to a constant floor
(`max(replicas, effectiveMin)`), while the postgres release is forced to
client-side (two-way) Helm apply so the seed is a merge no-op on upgrade
and the KEDA-managed HPA — the runtime writer via the CNPG `Cluster`
scale subresource — is not reverted.
- **Single-value metric (§1)** — one PromQL query returns `Σ(read load
over standbys) + target`, so a stock HPA computes `desired =
ceil((Σ+target)/target) = 1 + ceil(Σ/target) = primary + desiredRead`,
with no per-pod emission.
- **The four static brakes (§5)** — quorum floor `max(minReplicas,
maxSyncReplicas+1, 2)` (quorum wins over `maxReplicas`), scale-down
pacing (`Pods/1/600s`), a replication-lag clamp folded into the query
(freeze both directions while lag exceeds the threshold and the primary
is writing, with `max_over_time` hysteresis), and dry-run (render
paused).
- **Two-phase enable/disable migration** via a `transition` sub-flag:
phase 1 stages `spec.instances` to the live count and brings KEDA up
with its HPA paused in both directions; flipping to phase 2 hands the
runtime count to KEDA, and because the seed re-renders the same count
the flip is a merge no-op with no dip.
- **Observability** — a database-autoscaling Grafana dashboard and
PrometheusRule alerts.

**Live validation (dev cluster, CNPG 1.30.0).** The full path works
end-to-end: the CNPG `Cluster` scale subresource serves
`status.selector`, KEDA renders the HPA, the metric reaches it
(`ScalingActive=ValidMetricFound`, `TARGETS 151/150`, `desiredReplicas=2
= 1 primary + 1 replica` — the §1 arithmetic), and the lag-clamp query
returns the base value on live VictoriaMetrics. Two packaging fixes were
needed and are included: KEDA must run on the cozystack `cozy.local`
cluster domain (the upstream `cluster.local` default makes the metrics
gRPC target NXDOMAIN and every HPA hangs on `FailedGetExternalMetric`),
and the operator's client-go throttle is raised so a single-replica
operator does not lose its leader lease under scale-subresource polling.

Implements #3953.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff. Two repositories are affected; both are follow-ups to open once
this lands (and after its dependency #3951 merges and the values surface
is final), left unticked here for a maintainer to confirm rather than
claimed with a speculative link:

- **cozystack/terraform-provider-cozystack** — this adds the nested
`autoscaling` block to `packages/apps/postgres/values.schema.json`. The
provider hand-models each app's fields (schema + model +
expand/flatten), so it needs the new block to expose it in Terraform.
- **cozystack/website** — KEDA is a new platform component
(`guides/platform-stack`), and the postgres reference page gains the
`autoscaling` params. The managed-app reference is regenerated from
`README.md` by the release bot; the platform-stack component list is
hand-maintained.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: to open (new platform component + postgres autoscaling
params)
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: to open (new `autoscaling` block in postgres
values.schema.json)

### Depends on

- #3951 — bump CloudNativePG to 1.30.0. The HPA can only drive the CNPG
`Cluster` scale subresource on CNPG ≥ 1.28.4, where it serves
`status.selector`; without it the HPA fails with `InvalidSelector`. This
PR should not merge before #3951.

### Release note

```release-note
feat(postgres): add optional horizontal autoscaling of read replicas via KEDA, configured through the new `autoscaling` values block; KEDA ships as a new optional platform component. Requires CloudNativePG >= 1.28.4 for the Cluster scale subresource selector. Note: every Postgres release now applies client-side (`release.cozystack.io/helm-server-side-apply: "false"`) so the autoscaler's live `/scale` count is not reverted by Helm; a kind-wide side effect is that chart-rendered fields on Postgres releases are no longer re-asserted by server-side-apply drift correction (driftDetection is unset), a documented trade-off with a per-instance apply strategy as follow-up.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
  * Added configurable PostgreSQL read-replica autoscaling with KEDA.
* Added CPU and read-connection metrics, replica bounds, replication-lag
safeguards, dry-run mode, and transition handling.
* Added optional KEDA platform support, Grafana autoscaling dashboard,
and Prometheus alerts.

* **Documentation**
* Clarified integer thresholds, prerequisites, configuration, and
transition guidance.

* **Bug Fixes**
* Improved validation and actionable errors for unavailable KEDA or
invalid settings.

* **Tests**
* Expanded coverage for autoscaling, observability, package selection,
and validation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-09 21:28:34 +03:00
Andrey Kolkov 1289f522fa fix(backups): unbreak install, kill destructive cleanup, reach the resource knob
Store the Redis strategy Template schemaless, as the Rabbitmq strategy
already does: without the markers controller-gen emitted the full
PodTemplateSpec schema and the CRD reached ~602 KB, pushing the chart's
Helm-release Secret past the 1 MiB cap so backupstrategy-controller failed
to install at all.

Give the example strategy a cleanup branch, and both scripts an explicit
`elif restore` / `else exit 1`, so a deleted Backup rendered with
Mode=cleanup deletes its object instead of falling into the restore path
and FLUSHALLing the live master.

Forward a backupStrategyController block through the platform bundle
alongside backupStorage, so redisBackupResources — the restore-Pod sizing
knob the loader-not-ready message points at — is reachable via
spec.components.platform.values.backupStrategyController, not merely
settable on a Helm-managed Strategy CR that reverts on the next upgrade.

Correct the endpointCASecretName doc row to match the now-optional mount,
and `int` the activeDeadlineSeconds interpolation so a large values-file
override renders as an integer, not exponent form.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-09 16:53:50 +04:00
Aleksei Sviridkin a6caad7c57 fix(platform): correct what disabledPackages does to ouroboros (#4098)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

`bundles.disabledPackages` does not uninstall anything. Two comments in
the platform chart said it does, offered next to `kubectl delete
package.cozystack.io cozystack.ouroboros` as if the two were
interchangeable.

A name in that list only stops `_helpers.tpl` emitting the Package
document, which is the same suppression you already get from leaving
`publishing.proxyProtocol` false. Every emitted Package carries
`helm.sh/resource-policy: keep`, and that annotation is there precisely
so disabling a package stops deleting it, so Helm leaves the Package CR
alone. The CR stays, the HelmRelease it owns stays, ouroboros keeps
running, and the pre-delete hook never fires because nothing uninstalls
the release. Deleting the Package CR is what gets there: GC takes the
HelmRelease, Flux uninstalls, the hook runs on the way out.

There was a second cost. The guard in the system bundle looks up the
live Package CR and never reads `disabledPackages`, so an operator who
followed the old advice hit the guard anyway with no idea why. New test
case pins that.

Comments only in the templates, so rendered output is unchanged. Chart
suite green at 156 tests.

### Screenshots

Not a UI change.

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

Walked the trigger map against the diff. One repo is reached, and I left
its box empty rather than tick it without a link.

`cozystack/website` has the same wrong instruction verbatim in three
files: `content/en/docs/next/networking/hairpin-proxy-protocol.md:122`,
and the identical line under `v1.4/` and `v1.5/`. It tells the reader
that removing the Package CR "(or add it to `bundles.disabledPackages`)"
triggers helm uninstall and fires the pre-delete hook. It also
contradicts `operations/configuration/components.md:72` on the same
site, which says `disabledPackages` will not remove components that are
already installed. The site fix is cozystack/website#683. It turned out
to be four files rather than three: `origin/main` there also carries a
`v1.6` tree, and all four copies were byte-identical.

Nothing else is reached: no package added or renamed, no
`values.schema.json`, no version enum, no default, no key the Ansible
role sets, nothing under `hack/`.

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/pull/683
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(platform): the platform chart no longer says `bundles.disabledPackages` uninstalls a package. A name in that list only stops the Package document being rendered. `helm.sh/resource-policy: keep` leaves the existing Package CR on the cluster and the component keeps running, so deleting the Package CR is what uninstalls it.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Documentation**
- Clarified proxy protocol safety guidance, including the effects of
disabling the Ouroboros package and the limitations of using
`bundles.disabledPackages`.
- Documented that disabling package rendering does not remove an
existing live package or stop its associated resources.

- **Tests**
- Added regression coverage confirming that disabling the package does
not bypass the proxy protocol safety guard when the live package remains
active.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-08 02:52:00 +04:00
Aleksei Sviridkin 675163399d fix(platform): accept the container gpu-operator variant in iaas (#4101)
## What this PR does

`bundles.iaas.gpuOperatorVariant: container` failed the whole platform
render. The `cozystack.gpu-operator` PackageSource ships that variant,
`values-container.yaml` configures it deliberately, and
`hack/check-gpu-operator-variants.bats` covers it, but the `iaas` bundle
guard only accepted `default` and `vgpu`.

The exclusion made sense when it was written and stopped making sense a
few minutes later. The variant and the guard came from parallel
branches, neither an ancestor of the other, variant merged first. Before
the guard the bundle hardcoded `default`, so `container` was selected by
hand-writing a Package CR, and leaving it out of a bundle-side list cost
nothing. The guard made the bundle the variant owner and closed that
route without noticing it was one. Its comment calls itself typo
protection and names `passthrough` as the example typo.

Allowing the value is not the whole fix. The same file puts
`extraFeatureGates: [HostDevices]` on the KubeVirt CR whenever
gpu-operator is enabled, and builds a `permittedHostDevices` table under
it. Both are VM-facing. The container variant keeps the host driver
bound and pins `vfioManager` off, so no GPU reaches a VM: the gate would
advertise something the node cannot serve, and the table would name
resources no sandbox device plugin registers there. Both now render for
`default` and `vgpu` only, and `container` gets the gpu-operator Package
with the kubevirt component untouched.

The helm-unittest case that pinned the rejection is gone. Two replace
it: the Package renders at the container variant with no component
overrides, and the KubeVirt wiring stays absent even with
`gpu.permittedHostDevices` and `gpu.mediatedDevicesConfiguration` set.
Comments and docs that called the mirroring unconditional on variant, or
the container variant unmerged, are fixed in the same pass.

No issue for this one, so the history is in the body.

### Screenshots

No UI change.

### Downstream repositories

Walked the trigger map against the diff. The only file it touches that
appears there is `packages/core/platform/values.yaml`, and the website
hit is not the hand-written key table (that page has no GPU section): it
is the documented workaround this change makes obsolete.
`content/en/docs/next/operations/gpu-container-workloads.md` currently
tells readers the bundle rejects `container` and routes them to a
hand-crafted Package CR, and `virtualization/gpu.md` / `vgpu.md`
describe the KubeVirt mirroring as unconditional on variant.

Filed as an issue rather than a PR because the docs must not land ahead
of this change, and the `v1.6` copies describe released behaviour and
stay correct as they are.

Nothing else matches: no package added, renamed or removed, no
`values.schema.json` or version enum touched, no `ApplicationDefinition`
semantics, no installer values key, no `hack/` layout change, no
platform variant or bundle recomposed. `gpuOperatorVariant` appears in
none of `ccp`, `ansible-cozystack` or `terraform-provider-cozystack`.

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/issues/685
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(platform): accept `container` as a value for `bundles.iaas.gpuOperatorVariant`, so the gpu-operator variant the platform already ships can be selected from platform values instead of a hand-crafted Package CR. The KubeVirt `HostDevices` feature gate and `permittedHostDevices` table are now emitted for the `default` and `vgpu` variants only; `container` serves GPUs to pods and gets neither.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added the `container` GPU operator variant for workloads that do not
require VM passthrough.
* Container workloads no longer receive KubeVirt host-device or
mediated-device GPU wiring.
* Clarified GPU configuration behavior for passthrough, vGPU, SR-IOV,
and container workloads.

* **Documentation**
* Updated passthrough guidance, configuration references, examples, and
KubeVirt wiring instructions for all supported variants.
* Added details for manually configured package resources and
platform-specific container setup.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-08 02:50:33 +04:00
Andrei Kvapil 54621bc046 feat(migration): tenant self-service VM import from VMware vSphere
Hidora's customers are leaving VMware now, and a tenant has to be able to
import their own machines without a platform administrator. The chart
approach could not express that: a tenant can neither create the Secret
holding vCenter credentials nor name the VDDK image, so both had to come
from somewhere they cannot reach.

Two CRDs in a new `forklift.cozystack.io` group, split by lifecycle. A
VMImportSource is a long-lived connection -- type, endpoint, credentials,
readiness -- and a VMImportTask is the one-shot operation that names
machines and produces VMDisks and VMInstances. The outputs carry no owner
reference back to the task, so deleting a finished import leaves the
machines alone. Konveyor Forklift does the transfer underneath; the
controller renders its objects, mirrors its verdicts, and hands each
finished volume into a VMDisk without copying it.

What a live vCenter migration forced out, and no amount of source reading
would have:

- A guest copied verbatim keeps the drivers it had under VMware and no
  virtio, so on a virtio disk it does not fail the import -- it succeeds
  and then does not boot. Imported disks go on SATA.
- VDDK connects straight to the ESXi host at the address vCenter
  advertises, which is routinely unreachable, and on the first cluster
  collided with the Service CIDR. `spec.hosts` redirects per host.
- Firmware must be carried across or a UEFI guest imports as BIOS and
  never boots (stacked on #3002).
- Deleting a task mid-transfer orphaned its DataVolume; a finalizer now
  removes the engine's volumes and leaves the outputs.

The console gains a Migration section: connections are editable, and the
VM field is a picker built from a machine list the controller publishes,
since the aggregated API cannot reach the inventory and should not hold a
source's credentials to try. Two general mechanisms fall out of that --
an option source may take an argument, and one named in a CRD annotation
may reference a sibling field.

Design: cozystack/community#62. Docs: cozystack/website#673.

Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-09-07 17:03:44 +02:00
Aleksei Sviridkin 6e8566a94c fix(platform): accept the container gpu-operator variant in iaas
The cozystack.gpu-operator PackageSource ships three variants: default,
vgpu and container. The iaas bundle's variant guard accepted only the
first two, so selecting container failed the render on a value the
package ships, values-container.yaml configures deliberately, and the
variant bats suite already covers.

The exclusion was coherent when it was written and stopped being so
immediately. The container variant and the guard landed within minutes
of each other from parallel branches, neither an ancestor of the other,
variant first. Until the guard, the bundle hardcoded the default variant
and container was selected by hand-crafting a Package CR, so leaving it
out of a bundle-side list cost nothing. The guard then made the bundle
the variant owner and closed that route without noticing it was one; its
comment describes itself as typo protection and names passthrough as the
example typo.

The KubeVirt side is not a bare allowlist edit. The container variant
serves GPUs to pods through the device plugin and runs no VM-facing
GPU stack: sandboxWorkloads and vfioManager are both off and the host
driver stays bound. Both KubeVirt-side pieces the bundle wires are
VM-facing — the HostDevices feature gate advertises a capability such
a node cannot serve, and permittedHostDevices lists resources no
sandbox device plugin registers there. Both are now emitted only for
default and vgpu, and container yields the gpu-operator Package with
the kubevirt component untouched.

Comments and docs that described the KubeVirt mirroring as
unconditional on variant, or the container variant as unmerged and
unavailable, are corrected to match.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-07 04:58:02 +03:00
Aleksei Sviridkin cd6a9b3a8f fix(platform): correct what disabledPackages does to ouroboros
Both the publishing.proxyProtocol comment in values.yaml and the
disable-hazard guard comment in the system bundle told operators that
adding a package name to bundles.disabledPackages triggers the vendored
ouroboros chart's pre-delete Corefile cleanup hook, alongside an
explicit `kubectl delete package.cozystack.io cozystack.ouroboros`.

It does not. A name in disabledPackages only makes the rendering
helper in _helpers.tpl skip emitting the Package document, which is
the same suppression that leaving proxyProtocol false already
performs. Every emitted Package carries helm.sh/resource-policy: keep,
so Helm does not prune the Package it has stopped rendering — that
annotation was added precisely to stop disabling a package from
deleting it. The Package CR survives, the HelmRelease it owns via
ownerReference survives with it, ouroboros keeps running and keeps
re-applying its rewrite block to kube-system/coredns, and the
pre-delete hook never runs because nothing uninstalls the release.

No platform values setting reaches that hook; the operator has to
delete the Package CR. Anyone who followed the old advice was left
with ouroboros still installed and, on the proxyProtocol false
transition, still facing the render guard with no idea why.

Pins the behaviour with a case asserting the guard still fails while
cozystack.ouroboros sits in disabledPackages and the Package CR is
live.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-07 01:15:06 +03:00
Aleksei Sviridkin 6935df8c7f chore(kubevirt): update KubeVirt to v1.9.0 (#4015)
## What this PR does

KubeVirt operator manifest goes from v1.8.4 to v1.9.0. The 1.9 release
turns every Beta feature gate on by default and adds
`disabledFeatureGates` to the CR for opting out, so the CR side needed a
few decisions.

`Template` is disabled. With it on, virt-operator deploys virt-template:
two Deployments, two APIServices, a webhook and a
ValidatingAdmissionPolicy that nothing here uses. One more aggregated
API server is one more way to break discovery. This one is not a knob.
`- Template` is written into the CR unconditionally and there is no
value that takes it back out, because nothing on the platform uses
VirtualMachineTemplate; if something starts to, the static entry becomes
a value then. Asking for it through `extraFeatureGates` is refused at
render time with an error that says so, rather than producing a CR
KubeVirt would reject.

`ImageVolume` is the host-dependent one: off below Kubernetes 1.35, left
on from 1.35, with a platform value for runtimes that cannot serve image
volumes even there (#3190 has the k3s case). Nothing the platform ships
uses containerDisk. The value is `kubevirt.disabledFeatureGates` on the
platform values, which the iaas bundle forwards into the emitted
`cozystack.kubevirt` Package as
`components.kubevirt.values.disabledFeatureGates`:

Closes #3190.

```yaml
kubevirt:
  disabledFeatureGates:
  - ImageVolume
```

The chart refuses to render a CR that names one gate in both lists.
KubeVirt's `kubevirt-update-validator.kubevirt.io` webhook rejects that
combination and runs with `failurePolicy: Fail`, so on a cluster that
already has the CR the write is refused and the `kubevirt` HelmRelease
stops converging with everything else the CR carries. The create
admitter runs no validation at all (its own comment says the update
validator is where that happens, and it only gets installed after the CR
exists), and `IsEnabled` in virt-config checks `FeatureGates` before
`DisabledFeatureGates`, so on a fresh install the same CR is admitted
and the gate silently stays on. Render-time is the only place left where
an operator gets told.

A name repeated inside one list is not an error: upstream compares the
two lists against each other, and both fields are
`x-kubernetes-list-type: atomic` in the CRD, so that CR is accepted and
the guard stays quiet. The gates the chart pins off can arrive a second
time from a supported values file, since the chart writes them itself
and an operator listing the gates their cluster declines has no reason
to know that, so the range skips them and the CR states each opt-out
once. The guard walks its own copy of the seven gates the chart enables.
That copy exists because the rendered list has to stay a literal list
for `make update` to find its `- VMExport` anchor, and a gate quietly
dropped from the copy would keep rendering under `featureGates` while
leaving the guard. The suite carries one case per gate, so removing any
single name turns exactly one of the seven red.

`HotplugVolumes` stays in the list even though 1.9 deprecates it. A
deprecated gate listed in the CR is still enabled, and while it is
enabled virt-controller skips the declarative hotplug path. Dropping it
would switch on declarative hotplug, but then virt-api rejects the
VMI-level `removevolume` subresource on VM-owned VMIs, and kubevirt-csi
uses that as a fallback during teardown. The CSI driver has to move off
that subresource first. Upstream names no removal release for it: the
`Deprecated` state's own doc comment says "discontinued in the following
release", but `MultiArchitecture` and `DisableMDEVConfiguration` have
been deprecated since 1.8.0 and are still registered at 1.9.0, so that
is a direction rather than a date.

What makes that opt-out list matter is a semantics change: on 1.8
`isFeatureGateEnabled` ended in `return false`, so a Beta gate nobody
listed was off; on 1.9 `IsEnabled` ends in `return fg.State == Beta`.
Sixteen gates are Beta at v1.9.0 and none of them is in this chart's
`featureGates`, so all sixteen arrive on with the bump. Two are switched
back off here, one is switched off below Kubernetes 1.35, and I went
through the rest:

- `Template`: off. Reason above.
- `ExternalNetResourceInjection`: off. With it on, virt-controller stops
calling `multus.NetworkToResource`, so a NAD's
`k8s.v1.cni.cncf.io/resourceName` no longer becomes a resource request
on the launcher pod. That is correct for clusters running a
network-resources-injector; this platform ships none, so a VM on an
SR-IOV NAD would schedule without its device.
- `ImageVolume`: off below Kubernetes 1.35, left alone from 1.35 on.
KubeVirt renders containerDisk images as `VolumeSource.Image` volumes
when the gate is on, and it does not check the host: `dropImageVolumes`
in the apiserver strips every image volume and the mounts referencing it
while Kubernetes' own `ImageVolume` gate is off, which is the default
until 1.35 (`kube_features.go` has it `Default: false` at 1.31 and
1.33). The pod is admitted without the disk and virt-launcher fails
looking for it. The management cluster is supported from 1.33, so on
1.33 and 1.34 the chart keeps the KubeVirt gate off and the
containerDisk sidecar path in use; from 1.35 the gate stands and
`extraFeatureGates` can name it. The apiserver version is a proxy for a
runtime capability, not the capability itself, so a cluster at 1.35 or
newer whose container runtime cannot serve image volumes (the k3s case
in #3190) switches the gate off through `kubevirt.disabledFeatureGates`;
that is what the value is for.
- `Snapshot`: on. `VirtualMachineSnapshot` needed the gate listed on
1.8; snapshot and export were exercised on the test cluster.
- `DeclarativeHotplugVolumes`: on and inert.
`handleDeclarativeVolumeHotplug` returns early while `HotplugVolumes` is
enabled, which this chart still does.
- `GPUsWithDRA`, `HostDevicesWithDRA`: on; each only lets a VMI request
a DRA-provisioned device, and nothing here creates one.
- `WorkloadEncryptionSEV`: on; needs SEV hardware and a VM that asks for
it.
- `PasstBinding`: on; needs a VM using the passt binding.
- `RebootPolicy`: on; adds a field on the VMI domain spec, unset means
the old behaviour.
- `VmiMemoryOverheadReport`: on; fills `status.memory.memoryOverhead`,
reporting only.
- `OptOutRoleAggregation`: on; it enables the `RoleAggregationStrategy`
field, and aggregation stops only for a CR that sets it to `Manual`.
This one does not.
- `KubevirtSeccompProfile`: on; installs the profile, and a workload
uses it only where `seccompConfiguration` says so.
- `NodeRestriction`: on; virt-handler node authorization, needs the
Kubernetes `ServiceAccountTokenPodNodeInfo` gate, GA since 1.32.
- `LibvirtHooksServerAndClient`: on; pre-migration hooks on the target
launcher, and no hooks ship here.
- `PodSecondaryInterfaceNamingUpgrade`: flips on; it adds
`--upgrade-ordinal-ifaces` to virt-launcher and a pre-migration hook
that renames the ordinal secondary interfaces of VMs still carrying
them. The test cluster's migrations exercised that path.

Five of the seven gates the chart enables (`ExpandDisks`,
`LiveMigration`, `AutoResourceLimitsGate`, `GPU`, `VMExport`) reached GA
by 1.9 and are inert: `IsEnabled` returns true for a GA gate whether or
not the CR names it. I left the list as it is; dropping the five is a
follow-up, because it changes what the CR says without changing what the
cluster does and it belongs in its own commit.

QEMU is 10.1.0 on both sides of this jump; the el9 build revision moves
from -10 to -20 and libvirt goes 11.9 to 11.10. A moving build revision
is the thing to check rather than a major:
[kubevirt/kubevirt#16386](https://github.com/kubevirt/kubevirt/issues/16386),
where every existing VM lost live migration, was 1.6.2 to 1.7.0 on
qemu-kvm 9.1.0-19 to 9.1.0-20, one build apart within the same QEMU
version. So I tested it: VMs running on the 1.8.4 launcher live-migrated
onto the 1.9.0 one on a three-node cluster.

`make update` in `kubevirt-operator` used to `rm -rf templates` before
the fetch and took the local PrometheusRule template with it. Now it
only overwrites the file it downloads.

`make update` in `kubevirt` now greps for every line of the new block
before it passes: the range directive, the `disabledFeatureGates:` key,
the three static entries, and the guard header that decides which static
entries render. Every insert in that target is guarded by one directive
it also inserts, so a hand-merge that keeps the directive and drops a
sibling line leaves the guard satisfied and the insert skipped. That
shape covers the directive and nothing else. What makes it worth a grep
per line here is the static entries: they are the only lines those
inserts add that are CR content in their own right rather than a wrapper
around a value, so they render on every install and their absence cannot
surface as a value that stopped taking effect. The CR just goes back to
deploying virt-template and to letting KubeVirt skip NAD resource
injection. The target cannot rebuild a partial block, so it names the
missing line and stops. The payload lines of the other four blocks are
exposed the same way; I checked, and both dropping `- {{ . }}` from the
`extraFeatureGates` range and dropping the `permittedHostDevices:` key
from its `with` render green and lose the operator's values. That is a
wider change than this PR and I left it out.

Two sharp edges I know about and left. `kubevirt.disabledFeatureGates:
[HostDevices]` on a cluster that also has `cozystack.gpu-operator` in
`enabledPackages` makes the bundle forward the same gate into both
lists, and the platform renders fine while the child kubevirt
HelmRelease then stops on the guard: fail-closed, with the gate named,
but diagnosed one level below where it was configured. And the
version-conditional opt-out is evaluated at render time, so a cluster
upgraded from 1.34 to 1.35 keeps `ImageVolume` disabled until something
re-renders the chart; the direction is the safe one, since the gate
stays off and containerDisk keeps the path it had on 1.8.4.

Two things I decided against doing here. The guard's copy of the gate
list is pinned against a name being dropped, not against a name being
added to the rendered list only; the copy exists because `make update`
needs the literal `- VMExport` anchor, so closing the other direction
means changing the self-heal contract. And
`hack/run-kubernetes-block-io_test.bats` still names a
`virt-launcher:v1.8.4` image in a fixture: inert, the test matches on
the container name, and the file is outside this change.

Tested on a three-node Talos v1.13.6 / Kubernetes v1.36.2 cluster with
Flux suspended. The upgrade took about ten minutes. virt-operator had
already deployed virt-template before the CR change landed and removed
it afterwards, CRDs included. Two VMs started on the 1.8.4 launcher, one
DataVolume-backed and one containerDisk-backed, live-migrated to the
1.9.0 launcher. VMDisk plus VMInstance through the apps API boots.
`addvolume` on the VM subresource reaches Ready, `removevolume` on the
VMI subresource is accepted. VirtualMachineSnapshot and
VirtualMachineExport from it reach Ready. Restore from snapshot and PVC
clone failed on that cluster inside the LINSTOR API layer, unrelated to
this change. CDI v1.66.0 from #3920 on top of it: HTTP imports to Block
and Filesystem, blank population and hotplug all work. `helm unittest`
is 26/26 for the kubevirt chart and 142/142 for the platform chart, and
the update idempotency script passes its eleven cases.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff. No app added or renamed, no
variant, release asset, `ApplicationDefinition` or tooling contract
touched, and none of the `packages/core/platform/values.yaml` keys the
ansible role sets are in the diff.

One repository is reached. `packages/core/platform/values.yaml` gains a
top-level key, `kubevirt.disabledFeatureGates`, defaulting to the empty
list, and `operations/configuration/platform-package.md` on the website
is a hand-written table of `spec.components.platform.values.*` that
nothing generates. The row is in cozystack/website#678, which lands
after this PR merges.

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/pull/678
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
chore(kubevirt): update KubeVirt to v1.9.0. Beta feature gates are on by default upstream now. The platform keeps `Template` off, which would deploy virt-template, and `ExternalNetResourceInjection`, which would hand `resourceName` injection for NetworkAttachmentDefinitions to an injector the platform does not run. `ImageVolume` is disabled on management clusters below Kubernetes 1.35: containerDisk-backed VMs are rendered with native image volumes when it is on, and an apiserver whose own ImageVolume gate is off, the default before 1.35, drops those volumes silently, leaving the VM without its disk. On 1.35 and newer the gate stands, and that is the first release where containerDisk-backed VMs are rendered with image volumes instead of sidecars; the bundled Talos ships Kubernetes 1.36, so this is the default there. A `kubevirt.disabledFeatureGates` list in the platform values opts out of any other gate a cluster cannot carry.
```
2026-09-05 00:30:05 +04:00
Alexey Artamonov 6e3da63586 fix(postgres): guard the disable-wedge, role-scope the freeze count, and harden edges
Address the remaining IvanHunters review points on #3954:

- [MAJOR] Disabling autoscaling wedged the HelmRelease under synchronous quorum.
  An autoscaled + maxSyncReplicas>=1 cluster runs with instances seeded from
  effectiveMin while replicas stays at its default, so flipping autoscaling off
  rendered instances: <replicas> next to the quorum and CNPG's webhook rejected
  it ("maxSyncReplicas must be lower than the number of instances"). Add a
  symmetric render-time guard on the static-count path (covers off/dryRun/
  transition): fail with an actionable message when replicas <= maxSyncReplicas.

- [MINOR] Role-scope the freeze brake's currentInstances count
  (label_cnpg_io_instance_role=~".+") so lingering CNPG join/bootstrap Job pods
  (cnpg.io/cluster but jobRole, not instanceRole) don't inflate it and nudge
  desired up while the brake is meant to hold it.

- [MINOR] Default maxReplicationLagSeconds to 0 (brake opt-in): only the
  non-braking path is validated live, so ship the validated default and let the
  freeze be enabled explicitly until its branch is exercised under load.

- [MINOR] Document the floor-change and non-quorum-disable dips (same three-way-
  merge hazard) and the pin-replicas-first mitigation.

- [NIT] Coerce the cozy-lib helper's `paused` (eq "true") so a caller passing the
  string "false" does not pause the ScaledObject.

- [NIT] Trim the keda.clusterDomain injection comment so it no longer ships ~8
  verbatim lines into the cozystack-values Secret on every cluster.

Regenerate types/schema/README/postgres-rd. New tests cover the disable-wedge
guard, the role-scoped freeze count, and the string-"false" paused coercion.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-04 23:19:25 +03:00
Alexey Artamonov 42a533fc50 fix(platform): inject networking.clusterDomain into the KEDA package
KEDA's clusterDomain is a vendored-subchart value baked into the metrics-apiserver
gRPC target (keda-operator.<ns>.svc.<clusterDomain>:9666) and the operator's
serving-cert SANs at render time. A chart template cannot derive a subchart value
from _cluster, so the keda package hardcoded cozy.local — which silently breaks
(NXDOMAIN, every DB HPA FailedGetExternalMetric) on any cluster whose domain is
not the default. Inject the authoritative networking.clusterDomain into the shared
cozystack-values Secret under the keda subchart key so the value tracks the
platform config; the keda package keeps cozy.local as a standalone fallback. The
key is ignored by packages without a keda subchart (no app/extra schema sets root
additionalProperties:false). Add platform unittests asserting the Secret carries
keda.clusterDomain and that it tracks a non-default networking.clusterDomain.

Addresses the hardcoded-clusterDomain portability gap found in branch review.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-04 23:18:47 +03:00
Alexey Artamonov 7f8f2c1d86 feat(keda): add KEDA as an optional platform component
Vendor the upstream KEDA chart (kedacore/keda 2.20.2) as a new optional
system package and wire it into the platform, so database read-replica
autoscaling (which renders KEDA ScaledObjects) has a backend.

- packages/system/keda: umbrella cozy-keda + vendored chart (Makefile
  pins KEDA_CHART_VERSION with a lockstep assert), shim values enabling
  the operator / metrics-server / webhook ServiceMonitors under the
  platform Prometheus.
- core/platform: PackageSource cozystack.keda (namespace cozy-keda,
  dependsOn networking + prometheus-operator-crds; cert-manager is not
  needed, KEDA self-generates its webhook TLS) plus an optional bundle
  entry, opt-in via bundles.enabledPackages. It is not a dependsOn of any
  default package, so the platform HelmRelease never blocks on it.
- KEDA claims the cluster-wide external.metrics.k8s.io APIService
  (distinct from metrics-server's metrics.k8s.io — no conflict).

Covered by helm unittest: platform bundle enable/disable/opt-out, the
PackageSource dependsOn, and the vendored chart's external-metrics
APIService + shim-enabled ServiceMonitor.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-04 23:18:47 +03:00
Aleksei Sviridkin 8adf3a0863 feat(platform): forward kubevirt.disabledFeatureGates to the kubevirt CR
The kubevirt chart reads disabledFeatureGates from its own values, and
the iaas bundle is the only writer of the cozystack.kubevirt Package, so
nothing could set it: a Package patched by hand is reverted the next
time the platform re-renders it.

Platform values gain a documented kubevirt.disabledFeatureGates list,
forwarded into the emitted Package as
components.kubevirt.values.disabledFeatureGates alongside the entries
derived from resources.cpuAllocationRatio and the gpu block. The
parenthesized value walk is nil-safe, matching the gpu block, so a
cluster that nulls out the whole kubevirt key still renders.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-04 15:29:09 +03:00
Aleksei Sviridkin 5976ddf02f fix(gateway): normalise the GatewayClass name and refuse a blank one
The class lists arrive over the values channel comma-joined, so every
reader trims each entry as it splits them. The scalar those entries are
compared against was not trimmed anywhere. A class name carrying a
stray space therefore failed `has` against its own padded self, and the
two readers of that scalar disagreed about the same class: the gateway
chart put the tenant on an edge-terminated class, while the
cluster-wide ACME issuers still pinned their solver to a listener an
edge Gateway does not render. That stops every HTTP-01 certificate in
the cluster from issuing, with nothing in any status naming the cause.

Normalise once at the single writer so both readers see a clean value,
and keep a defensive trim at each reader. The tenant's own choice does
not travel that channel — it comes from tenant.spec.gatewayClass — so
it is trimmed where the gateway chart reads it.

Unset and whitespace-only are different events and no longer share an
outcome. Unset means the choice was not made, and falling back to the
bundled class stays its documented behaviour. Whitespace-only means a
choice was made and is garbage, in the field that decides where TLS
terminates: resolving it quietly would take a tenant the operator meant
to put behind an edge terminator and publish it directly, with a real
certificate on a real address. Both the platform default and the
tenant's own choice now refuse it by name.

The kind check beside that refusal is part of the same fix, not a tidy
up. Through Helm's values path an unquoted numeric scalar arrives as a
float64, and the whitespace check compares it against a string, so
`className: 123` dies at that comparison — one line before the trim —
with an error naming neither the key nor the file. Refusing whitespace
without checking the kind first would therefore have made the numeric
case worse than it was. `kindIs "string"` is the right test because it
never has to name the type that actually arrived.

The order of the guard's steps is part of the guard. The value is read
before anything normalises it, so a default that fires on any falsy
value cannot launder 0 or false into the empty string and resolve them
to the bundled class ahead of the kind check. Unset is the one case
that still falls back, and each of those four inputs is pinned, so a
normalisation drifting back ahead of a check fails rather than quietly
covering only the loudest bad input.

Also correct a cleanup log line: the wildcard Certificate is deleted
for every mode that is not DNS-01, so naming HTTP-01 in that message is
wrong for existingSecret and for edge alike. And state the listener
name invariant plainly: the cap is 253, not the 63 of a DNS label, and
the longest name the builders produce is 84.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-09-04 14:33:35 +03:00
Aleksei Sviridkin 487cafac3e feat(platform): let a tenant pick its Gateway class from an allowlist
Tenant Gateways were pinned to the bundled Cilium class. Give the
platform a default in gateway.className, the set a tenant may choose
from in gateway.tenantSelectableClasses, and the set whose provider
terminates TLS upstream of the Gateway in
gateway.edgeTerminatedClasses. A tenant picks its own with
tenant.spec.gatewayClass, read only when it owns a Gateway; an
inheriting tenant runs on its ancestor's.

The selectable set is an allowlist rather than free choice because
Tenant is a tenant-writable object while a GatewayClass is
cluster-scoped: without it a tenant could attach its Gateway to a class
an admin keeps for internal-only traffic. Naming anything outside the
set fails that tenant's own gateway release and reaches no other
tenant.

Where TLS ends is a property of the class, so the gateway chart derives
certMode from whichever class the tenant landed on instead of taking it
as a separate input. The two therefore cannot be configured out of
step.

The publishing tenant cannot cross that line by itself. The
cluster-wide ACME issuers point their HTTP-01 solver at its Gateway and
are rendered from the platform default alone, so a mismatch would pin
the solver to a listener that Gateway does not have. Those issuers now
drop the sectionName pin when the default class is edge-terminated,
letting the challenge attach by hostname instead.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-09-04 14:26:54 +03:00
Andrei Kvapil 4e6313b608 feat(valkey): register Valkey application in the platform
Wire the managed Valkey app into the platform: add the valkey-rd package
carrying the Valkey ApplicationDefinition, the valkey-application PackageSource
(reusing the existing redis-operator), enable it in the paas bundle, and grant
tenant roles access to the valkeys resource.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-09-01 11:43:10 +04:00
Andrei Kvapil c3cead8f64 fix(telemetry): make the documented opt-out cover both collectors
Telemetry is reported by two components, cozystack-operator (cluster facts)
and cozystack-controller (application counts), but only the operator had a
reachable switch. The documented opt-out sets
cozystackOperator.disableTelemetry on the cozy-installer chart, which cannot
reach a package the platform deploys, so cozystack-controller kept reporting
after an operator had followed the procedure — silently, with nothing to
indicate the opt-out was partial.

The platform now exposes telemetry.disabled and threads it into the
cozystack-controller component of the cozystack-engine Package, the same way
the bundle already threads cilium, multus and linstor values. The installer
key is also declared in values.yaml: it was only ever referenced from the
template, so the documented --set target did not appear in the chart's own
values.

Both keys stay false by default; this changes no behaviour for anyone who has
not opted out.

Assisted-By: Claude
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-08-12 19:48:37 +02:00
Aleksei Sviridkin 7739d1e294 feat(multus)!: stage reference CNI plugins from the multus image
Multus-backed NetworkAttachmentDefinitions delegate to the upstream
containernetworking reference plugins (bridge, macvlan, ipvlan, ...),
and nothing guarantees a node has them in the directory Multus
searches.  On k3s they live under /var/lib/rancher/k3s/data/cni while
Multus and Cilium install into /opt/cni/bin, so a bridge-type NAD fails
there with `failed to find plugin "bridge" in path [/opt/cni/bin]`.

Bundle 14 of the release's plugins into the multus image and add an
install-cni-plugins init container that installs them into the host
/opt/cni/bin, mirroring the existing install-multus-binary container.
loopback, dhcp, tap and dummy are left out; the Dockerfile records why.

Gate it on a new chart value, stageCniPlugins, resolved per bundle and
on for both.  Talos bundles only bridge, firewall, flannel, host-local,
loopback and portmap, so the other ten are missing there exactly as on
generic Linux; the four that overlap are rewritten with the same
upstream version, v1.9.1 on both sides, rather than an older one.  A
marker in the image Dockerfile records the Talos release that pin was
compared against, and a test holds it to the release the platform
ships.  multus therefore moves out of system.common-packages into the
two variant branches, the way linstor already does for talos.enabled.
networking.stageCniPlugins overrides the variant default in either
direction.

Where staging is on the install is unconditional: a plugin is replaced
on every pod recreation, not only when the destination is missing.  A
destination that is already a directory is reported and left alone,
since replacing it is not something this can do safely.  Turning the
value off afterwards does not undo it; nothing here removes or
restores a plugin.

The container is not privileged and takes no mount propagation.  It
runs in the spc_t SELinux domain, without which an enforcing node
refuses the write.  Each plugin is installed under a temporary name and
swapped in with a rename: /opt/cni/bin is shared with cilium and
kube-ovn.  The plugin tarball is checksummed before extraction.

A plugin that cannot be installed is reported on stderr and to
/dev/termination-log, and the container still exits 0: Multus stays the
node's primary CNI and nothing removes its config, so an init container
that never completes would leave the shim with no daemon behind it.

The manifest change lives in customize-deployment.patch so it survives
make update.

BREAKING CHANGE: enabling stageCniPlugins replaces reference CNI
plugins in /opt/cni/bin on existing nodes, Talos included -- four of the
names it writes are ones the Talos node image also carries.  Opt out
before upgrading: declining afterwards does not restore what was
replaced.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-05 14:19:47 +03:00