Commit Graph
12 Commits
Author SHA1 Message Date
Aleksei Sviridkin be2387323b feat(gateway)!: reserve a passthrough listener's name by the spec alone (#4638)
## What this PR does

This keeps the TLS-passthrough services on port 443 of the tenant
Gateway and makes the passthrough rules hold on Cilium 1.20.2. It is
stacked on #3343 and has to land with it (stack #4717).

On 1.20, upstream Cilium marks an HTTPS-terminate listener and a
TLS-passthrough listener on one port `Conflicted` when their hostnames
overlap, and drops both from Envoy. That took `*.<apex>` and the `api`,
`vm-exportproxy` and `cdi-uploadproxy` passthrough listeners down
together on every tenant Gateway in `dns01` and `existingSecret` mode
(#3912). #3343 now ships a patched operator that conflicts the pair only
when the hostnames are equal, so the layout stays as it is on main: one
Gateway, one address, everything on 443. What remains is the
equal-hostname case and the cleanup that 1.20 allows.

Under `http01`, a name a passthrough listener answers is now reserved by
the spec alone. Before, the controller dropped the terminate listener
for such a name only once a TLSRoute was attached to the passthrough
listener and its backend Service resolved, which copied how Cilium
builds filter chains. Without a working TLSRoute, an HTTPRoute on
`api.<apex>` kept its terminate listener next to `tls-api`, and on 1.20
that equal-hostname pair takes both listeners down. Now such an
HTTPRoute gets no listener and no certificate, TLSRoute or not. If all
its hostnames are reserved it reads `Accepted=False` with
`NoMatchingListenerHostname`. If it also claims a free hostname it stays
`Accepted` and is served there. The backend lookup, its Service and
ReferenceGrant watches and their RBAC rules are gone.

Other changes:

- `tlsPassthroughListeners` is accepted under `dns01` and
`existingSecret`. The old refusal relied on 1.19.5 serving all ports
from one Envoy listener; 1.20 splits ports whose SNI overlaps.
- A TenantGateway reads `Ready=False` with reason
`GatewayClassUnsupported` when its GatewayClass lists features in
`status.supportedFeatures` but not `TLSRoute`, while it renders
passthrough listeners. A class that lists nothing is not judged. Cilium
1.20.2 lists `TLSRoute`.
- Comments, the README and one error message no longer describe Cilium
1.19.5 as the shipped release.
- The gateway chart writes `tlsPassthroughServices` on the publishing
tenant only, behind the same `gateway.enabled` switch and
`publishing.exposedServices` filter as the platform TLSRoutes. Before,
every tenant Gateway got the three passthrough listeners with nothing
routed to them, and the reservation above would have taken those names
from a child tenant's own HTTPRoutes.

The uniform `allowedRoutes.kinds` on port 443 from #3087 stays. The
Cilium bug behind it is fixed since 1.19.6, but several tests pin it, so
removing it is a separate cleanup.

### Upgrade notes

Only the publishing tenant keeps the `api`, `vm-exportproxy` and
`cdi-uploadproxy` passthrough listeners. That is `tenant-root` unless
`publishing.ingressName` says otherwise, and only for names in
`publishing.exposedServices` while `gateway.enabled` is on. Other
tenants lose these listeners on upgrade. Nothing shipped was attached to
them, so a child tenant's own HTTPRoute on `api.<child-apex>` keeps its
listener and certificate.

Two narrow cases do change. On the publishing tenant under `http01`, an
HTTPRoute on an exposed passthrough name loses its listener and
certificate when the platform TLSRoute for that name forwards nowhere
(the owning package is not installed, or its backends do not resolve). A
TLSRoute attached by hand to a child tenant's `tls-*` listener loses its
parent. To keep that listener, merge-patch `tlsPassthroughServices` into
the tenant's `TenantGateway` after the upgrade. From then on the chart
does not render the key there, and later upgrades leave the patch alone.
`dns01` and `existingSecret` do not change: the wildcard listener still
covers such a name, and the passthrough listener answers it by its exact
name.

Tested locally: Go tests for the controller, `go vet`, helm-unittest for
`extra/gateway` and `cozystack-controller`, `make generate` and `make
manifests` with no diff. The gateway e2e scenario checks that a child
tenant renders no passthrough listener. It then declares `tls-api` and a
passthrough listener on a native port on that tenant, checks that the
tenant stays Ready with no route behind either, and checks the
reservation against `tls-api`. Unit tests catch a regression in either
change. In e2e, dropping the reservation turns the gateway suite red,
and dropping the chart scoping does not, because CI never sets
`gateway.enabled` (#4846). The controller was not run on a live cluster;
the operator patch it depends on was, see #3343.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/issues/725
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(gateway)!: the tenant Gateway renders the api, vm-exportproxy and cdi-uploadproxy passthrough listeners only on the publishing tenant, for the names publishing.exposedServices lists while gateway.enabled is on. Under certMode http01 a name a TLS-passthrough listener answers no longer gets an HTTPS listener on 443, with or without a TLSRoute behind it. On the publishing tenant, an HTTPRoute on such a name whose TLSRoute forwards nowhere loses its listener and certificate. A TLSRoute attached by hand to a tls-* listener of another tenant loses its parent. An HTTPRoute with some reserved and some free hostnames used to be rejected whole; it is now Accepted and its free hostnames get a listener and a certificate. tlsPassthroughListeners is accepted under dns01 and existingSecret. A TenantGateway whose GatewayClass does not list TLSRoute in its supported features reads Ready=False while it renders passthrough listeners.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* TLS passthrough listeners are supported with HTTP-01, DNS-01, and
existing-secret certificate modes. Edge-terminated Gateways remain
unsupported.
* Declaring a passthrough hostname reserves it from HTTPS termination
and certificate creation, even when no route is attached.
* Passthrough listeners are rendered only for exposed services on the
enabled publishing Gateway.
* **Bug Fixes**
* Gateways that declare passthrough listeners but explicitly lack
TLSRoute support are now reported as not ready.
* Routes targeting a hostname served by a passthrough listener on a
different Gateway are correctly withdrawn.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-10 16:59:17 +03:00
Aleksei Sviridkin f61e13fdef fix(cilium): size the agent memory limit for its BPF maps (#4803)
## What this PR does

This raises the cilium-agent memory limit from 4Gi to 24Gi. The 4Gi
limit from #3495 has no room for the agent's BPF maps. On nodes with
roughly a terabyte of RAM or more, the agent is OOM-killed on every
start after a reboot, and the node loses pod networking.

The maps grow with node RAM. Cilium sizes its CT, NAT, neighbor and
socket rev-NAT maps as a share of total RAM
(`bpf-map-dynamic-size-ratio`, 0.0025 by default), and caps each map at
16Mi entries. These are LRU hash maps, so the kernel preallocates them
in full. Since Linux 5.11 that memory is charged to the memcg of the
process that creates the maps, which is the agent container.

This is also why the upgrade itself works. A map reused from bpffs stays
charged to the cgroup that created it. The failure shows up on the first
reboot after the upgrade, when the maps are recreated inside the limited
cgroup.

Policy maps don't belong on that list. The issue and the upstream
chart's configmap comment both say the ratio sizes them, but at this
version they are LPM tries sized by `bpf.policyMapMax`, and they
allocate per entry.

One DaemonSet serves every node, so the limit can't follow node RAM. The
map sizes are capped, though, so a fixed limit can cover any node. With
every map at 16Mi entries for both IPv4 and IPv6, the maps take just
under 20Gi of kernel memory. Add the 4Gi the agent already had and you
get 24Gi. The request stays at 256Mi.

The limit doesn't reserve memory, and the pod cgroup still gets
`memory.max`, which is what #3495 was after. What it costs is the leak
backstop. The agent is now stopped at 24Gi minus its own map memory, so
on most nodes a leak in its heap is no longer caught before the node
feels it. If someone prefers a tighter backstop over IPv6 headroom, they
can lower `cilium.resources.limits.memory` in the networking component
values. About 13Gi still covers the IPv4-only map ceiling.

I didn't pin the map sizes (`bpf.ctTcpMax`, `bpf.natMax` and so on). No
single value works for every node. A value small enough for a 16Gi node
cuts the conntrack capacity of a 1.5TiB node many times over. A pinned
size also differs from what almost every node above ~14Gi of RAM runs
today, so the agent would recreate those maps on upgrade and drop their
conntrack and NAT state. A lower ratio has the same problem and doesn't
lower the ceiling. The new test keeps the ratio at its default and the
sizes unpinned, so changing either has to be deliberate.

The figures are computed, not measured. I read the sizing code at cilium
1.19.5 (`pkg/option/config.go`), and 1.20.2, which #3343 moves to, has
the same code. Memory comes from that formula plus the kernel hash map
layout: a 48-byte element header, the key and value rounded up to 8
bytes, and a 16-byte bucket per power of two of `max_entries`. By that
math the IPv4 maps alone pass 4Gi at roughly a terabyte of RAM. Bucket
rounding moves the exact point, so I wouldn't read more precision into
it. For the 1.5TiB node in the report, the issue's 3.9GiB is the agent's
own budget (0.25% of RAM), and the kernel's actual charge comes out
higher.

Follow-ups, not in this PR:

- Operators who want smaller maps on large nodes can size them per node
with a `CiliumNodeConfig` selected by node label. This needs a
documentation note.
- The reporter's `bpftool` measurement of the real map memory is still
pending. That is the measured check for the numbers above.

The cilium helm-unittest suite passes on this branch, and also with this
change applied over the 1.20.2 chart from #3343. Each new assertion
fails when its value changes: the limit, the CT, NAT and neighbor sizes,
the socket rev-NAT size set through `extraConfig`, and the ratio.

Closes #4800

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff changes only `packages/system/cilium/values.yaml` and its test.
None of the trigger-map entries apply: no app, schema, platform value,
variant, release asset or tooling changes, and no website page documents
the agent limit.

### Release note

```release-note
fix(cilium): raise the cilium-agent memory limit from 4Gi to 24Gi so it covers the agent's BPF maps, which cilium sizes from node RAM. On nodes with around a terabyte of RAM or more, the 4Gi limit got the agent OOM-killed on every start after a reboot, leaving the node without pod networking.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
  * Increased the Cilium agent memory limit to 24 GiB.
* Updated validation for BPF map sizing: checks the dynamic size ratio
and confirms that specific map-size settings are absent.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-09 20:50:44 +03:00
Aleksei Sviridkin 1bfa64f83a fix(cilium): keep per-type xDS while the 1.20 upgrade rolls
The 1.20 chart renders envoy-xds-mode "ads" and an adsConfig bootstrap
for cilium-envoy unless xdsMode or upgradeCompatibility says otherwise.
A 1.19 agent never registers the aggregated discovery service, and a
1.20 agent in ADS mode registers nothing else. The agent and envoy
DaemonSets roll independently, so on every node where their versions
differ envoy gets no configuration until both pods are replaced, and
Gateway traffic stops on all nodes at once during the upgrade.

Pin xdsMode to split, the per-type xDS both versions serve and the
agent's own default when the key is absent. Moving to ADS is a separate
step once every node runs 1.20.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-09 17:53:14 +03:00
Aleksei Sviridkin b64e12b0cf docs(cilium): re-anchor comments the v1.20.2 bump made stale
The admission-webhook deny policies in ingress-nginx and kubeovn-webhook
cite Cilium source and the vendored values by line number. The bump
moved endpointLockdownOnMapOverflow and the policy-map ordering block in
pkg/endpoint/bpf.go, so the cited lines pointed elsewhere. Name the key
and the function instead, which survive the next bump, and move the
getEndpointSelector citation to v1.20.2, where the cited lines are
unchanged.

The kubeProxyReplacement pin explained the service translation in terms
of v1.19.5. With Gateway API on, v1.20.2 keeps socket LB in the host
namespace, so pod-namespace connections are translated per packet in
bpf_lxc before egress policy runs, which is what the pin relies on. The
same holds for the e2e image mirror, whose egress-policy comment in
hack/e2e-talos-image-cache.yaml described socket LB.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-09 02:04:40 +03:00
Aleksei Sviridkin 87a64071a7 fix(cilium)!: take cilium's own addresses from a block kube-ovn does not hand out (#4778)
## What this PR does

fixes #3740.

on the kube-ovn variants cilium allocates no pod ips, but the agent
still takes its router ip (`cilium_host`, `CiliumInternalIP`) and the
gateway api ingress ip from its own allocation cidr. with `ipam.mode:
kubernetes` that's `Node.spec.podCIDR`, a /24 out of the same /16
kube-ovn hands pods from, so sooner or later kube-ovn gives a pod an
address cilium already holds. we hit it on three clusters, details in
https://github.com/cozystack/cozystack/issues/3740#issuecomment-6022394243.

**what changes**

- the kube-ovn variants run cilium with `ipam.mode: cluster-pool`, a /29
per node out of the last /21 of `networking.podCIDR` (`10.244.248.0/21`
for the default /16). the platform derives the block from
`networking.podCIDR` and passes the same range to kube-ovn as
`--default-exclude-ips`, so a fresh install never hands it to pods.
- the block sits inside the pod cidr on purpose. envoy reaches gateway
api backends from the node's ingress ip, and kube-ovn masquerades host
originated traffic only when the source is in `ovn40subnets`. with the
`100.65.0.0/16` pool from the issue a nodeport through the gateway
answered 200 only on the backend's node and 503 `upstream connect error
... connection timeout` on the other two. with the block inside the pod
cidr it's 200 from all three. worth a look on the ci side too. the ci
sandbox sets talos `podSubnets` to `100.65.0.0/16`, so cilium's ingress
ips there already come from outside `ovn40subnets`, and the gateway e2e
checks Accepted, Programmed and the lb service but sends no request
through a listener, so it wouldn't notice. with this pr the sandbox
takes its block from the pod cidr like everyone else.
- the end of the /16 is what kube-ovn reaches last. it hands out the
lowest free address and reuses released ones only when the free range
runs dry (`pkg/ipam/subnet.go` and `pkg/ipam/ip_range_list.go` at
v1.15.26). a /21 cut into /29s covers 256 nodes.
- the slim variants and guest clusters stay on `ipam.mode: kubernetes`.
cilium is the only cni there and `Node.spec.podCIDR` is the pod range.
- `networking.podCIDR` now needs /20 or wider on the kube-ovn variants,
otherwise the render fails and says why. an install that rendered before
can stop rendering at upgrade, hence the `!`. a /21 cut into /29s also
caps these variants at 256 nodes, a fresh install included.

**existing clusters**

a post-upgrade hook in the cilium chart moves them
(`templates/ipam-migration.yaml`, `files/ipam-migration.sh`). before it
touches anything it checks that the block has a /29 for every CiliumNode
and that it lies inside kube-ovn's default subnet, and stops with both
cidrs printed otherwise. a node cleared past the pool's capacity would
lose pod networking on its next agent restart, and a block outside the
subnet puts the agents outside `ovn40subnets` again. it has to be
post-upgrade. platform migrations run as the platform's pre-upgrade
hook, while every agent still runs `ipam.mode: kubernetes` and writes
`Node.spec.podCIDR` back into a cleared CiliumNode
(`pkg/nodediscovery/nodediscovery.go` at v1.19.5), so the move can only
happen after the cilium release switched the mode. the job runs on the
host network against the agent's own api endpoint (kubeprism on talos),
because it restarts the agents.

1. adds the block and every node's current ingress ip to `excludeIps` of
the default subnet, in one json patch with a `test` op, and reads the
subnet again on conflict, since kube-ovn rewrites that list into its own
order right after a patch. the old ingress ip goes in because cilium
1.19.5 keeps a former ingress ip in the agent's ipcache as
`reserved:ingress` after the move. we saw 2 to 3 such entries per node
hours later. old router ips leave the ipcache and go back to kube-ovn.
2. stops right there if an address in the block is still allocated,
names it, and moves no agent. it first checks the subnet status it
patched against, then waits until `status.v4availableIPrange` stops
offering the block, which is kube-ovn having applied the exclusion, and
then reads the `ips.kubeovn.io` objects of the default subnet. the
status alone can't answer it, kube-ovn leaves excluded addresses out of
`v4usingIPrange` (`GetSubnetIPRangeString` in `pkg/ipam/ipam.go` at
v1.15.26), so a pod still in the block would be invisible on a retry
where the exclusion is already in place. recreate those pods and the
next upgrade retry goes through.
3. one node at a time, clears the inherited /24 from
`spec.ipam.podCIDRs` (again with a `test` op, so a concurrent writer
fails the patch instead of losing an update), waits for the operator to
allocate a /29 from the block and restarts that node's agent. a node
that isn't ready is skipped and its agent picks up the block when it
starts. nodes already moved are left alone, so a retry after a timeout
just continues. the hook runs inside the cilium upgrade, so where
restarting every agent takes longer than the upgrade timeout, 10m by
default, the release first reports a failed upgrade and the move
finishes on the retry.

running pods keep their addresses. every agent restarts twice on
upgrade, once when the config change rolls the daemonset and once from
the hook, and while it restarts host to pod traffic on that node blips
for a few seconds because the router ip moves. a pod that sits on a
router ip today is reachable again after the move, the node no longer
holds that address. a pod on a node's current ingress ip keeps it and
that node keeps seeing it as `reserved:ingress`, so recreate it. the
hook logs those pods by name.

### Validation

- `helm unittest` passes cilium 23/23, platform 246/246 and kube-ovn
22/22. the new suites pin the rendered `cilium-config` keys per variant,
the hook shape, env and rbac, and the platform wiring for the talos and
generic branches, a custom pod cidr, host bits in the network address,
the refusals for a too small or non ipv4 cidr, and the slim variant.
- on a trial merge with main at cbc9d0c4e, where #3849 runs
`hack/*.bats` under real bats 1.14 with `set -u`, the 27 bats tests pass
after the second commit loads `test_helper`, the strict setup audit is
clean, and helm unittest passes cilium 23/23, platform 263/263 and
kube-ovn 22/22.
- `hack/cilium-ipam-migration.bats` passes 27/27 and runs the real
script against a fake kubectl that models the operator, the agent
daemonset and kube-ovn publishing the subnet status. thirteen mutations
turn it red each (exclusion after the move, no ingress exclusion, no
retry, no status check, an ingress ip inside the block listed on its
own, no `test` op, no wait for kube-ovn, no ip object check, no subnet
filter on the ip objects, no capacity check, a block outside the subnet
let through, overlap turned into containment, no log of ingress ip
holders). the containment one is pinned by a usage range that crosses
the start of the block, `10.244.247.250-10.244.248.3`.
- live on a 3 node talos test cluster with cozystack v1.6.4, cilium
1.19.5 and kube-ovn v1.15.26
- moved from `kubernetes` to the block, exclusion first and then the
node moves this script does. 3 nodes in about 40 seconds, every agent
took its router and ingress ip from its /29, no pod changed its address.
- gateway api through a nodeport 9/9 answered 200 and through the
loadbalancer ip 6/6, against 3/9 with `100.65.0.0/16`.
- the final hook job on the moved cluster is a no-op in 2 seconds,
including the ip object read with the job's own rbac. the same ip object
pipeline over that cluster's real objects finds 77 in a populated /24
and none in the block.
- the exact `test` plus `replace` patch on `ovn-default` goes through,
kube-ovn re-sorts the list right after, and the same patch against the
old list is rejected. that's the conflict path the retry covers.
- shellcheck is clean and `hack/check-commit-trailers.sh` passes.
- not run here. the upgrade move from an older release in ci, e2e only
covers the fresh install.

### Screenshots

not a ui change.

### Downstream repositories

walked the trigger map against the diff.
`packages/core/platform/values.yaml` only gains a comment, but
`networking.podCIDR` gets a new constraint on the kube-ovn variants,
which the hand written key table on the website should say. the ansible
role sets `networking.podCIDR` to `10.42.0.0/16`, which yields
`10.42.248.0/21` and needs nothing there. the talm preset keeps
`podSubnets: 10.244.0.0/16`, which no longer matters on these variants
because cilium stops reading `Node.spec.podCIDR` there. the terraform
provider doesn't model platform values.

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/pull/739
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(cilium)!: on the kube-ovn variants cilium now takes its router and gateway api ingress ips from the last /21 of networking.podCIDR, which kube-ovn no longer hands out, so pods stop landing on addresses cilium holds. existing clusters move on upgrade through a post-upgrade hook, and each cilium agent restarts twice, once for the config change and once in the hook. networking.podCIDR needs /20 or wider on these variants, and a /21 of /29s caps them at 256 nodes.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Kube-OVN-enabled variants now reserve the final `/21` of the pod CIDR
for Cilium router and Gateway API Ingress addresses, excluding it from
Kube-OVN pod allocation. Pod CIDRs must be IPv4 with a prefix of `/20`
or shorter.
* Added a post-upgrade migration that moves existing Cilium IP
allocations into the cluster pool and updates affected node networking.
* **Documentation**
* Documented the pod CIDR requirements and node capacity limit for
Kube-OVN variants.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-08 21:41:23 +03:00
Aleksei Sviridkin 9be997b6e2 fix(cilium): size the agent memory limit for its BPF maps
cilium sizes its CT, NAT, neighbor and socket rev-NAT maps as a share
of node RAM. They are preallocated LRU maps charged to the agent's
memcg, so on nodes with around a terabyte of RAM or more the maps alone
exceed the 4Gi limit and the agent is OOM-killed on every start after
a reboot, leaving the node without pod networking.

Raise the limit to the map ceiling (every map at LimitTableMax, both
address families) plus the agent's own 4Gi budget. A per-node limit
cannot be rendered into one DaemonSet, and pinning the map sizes would
cut connection capacity on large nodes and reset conntrack and NAT
state on upgrade, so the map sizes stay dynamic and a test keeps them
unpinned.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-07 19:10:40 +03:00
Myasnikov Daniil c9558f5658 fix(networking): correct webhook policy assumptions
Read the ingress admission port from the effective subchart values and fail if the port is absent. Describe kube-proxy replacement as a required service-routing invariant while retaining target-side enforcement.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:00:03 +03:00
Yan Bondarenko dc362fa591 fix(cilium)!: take cilium's own addresses from a block kube-ovn does not hand out
On the kube-ovn variants Cilium allocates no pod addresses, but the agent
still takes its router IP and the Gateway API Ingress IP from its
allocation CIDR. In ipam.mode kubernetes that is Node.spec.podCIDR, which
the documented installs slice from the same range kube-ovn allocates pods
from, so kube-ovn sooner or later hands a pod an address Cilium already
holds.

Cilium now runs ipam.mode cluster-pool on the kube-ovn variants, with a
/29 per node from the last /21 of networking.podCIDR, and kube-ovn
excludes that /21 through --default-exclude-ips. The block stays inside
the pod CIDR on purpose. Envoy reaches Gateway API backends from the
node's Ingress IP, and kube-ovn masquerades host-originated traffic only
when the source lies in one of its subnets, so a block outside them such
as 100.65.0.0/16 breaks Gateway API traffic that crosses nodes.

A post-upgrade hook moves existing clusters. It first checks that the
block has a /29 for every node and lies inside kube-ovn's default
subnet, and stops without changing anything otherwise. It adds the block
and every node's current Ingress IP to excludeIps of that subnet,
because the agent keeps a former Ingress IP in its ipcache as
reserved:ingress. It then waits until kube-ovn stops offering the block
and stops if an address in it is still allocated. kube-ovn leaves
excluded addresses out of the subnet status, so that check reads its IP
objects. Then, one node at a time, it clears the inherited /24 from the
CiliumNode, waits for the operator to allocate from the block and
restarts the agent.

networking.podCIDR now needs a prefix of /20 or shorter on the kube-ovn
variants, and the render fails otherwise. A /21 cut into /29s also caps
these variants at 256 nodes.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-10-06 22:11:02 +03:00
Myasnikov Daniil 686a81a491 docs(networking): stop claiming the churn coupling is removed
Restoring the target-side pod deny put a cluster-spanning peer selector back on
the webhook endpoint, which is the exact thing the source-side rules were added
to take off it. The comments still said the coupling was "structural and
removed", and the release note still said deleting a namespace no longer
rewrites that endpoint's policy map. Both were true of the previous revision
and are not true of this one.

What ships now is two denies that fail in different directions: the source-side
one keeps enforcing while the webhook endpoint's policy is being recomputed or
has overflowed its map, the target-side one keeps enforcing where Cilium is not
the service load balancer, and neither subsumes the other. So the coupling is
not eliminated, it is made non-load-bearing -- whichever end is degraded the
other still denies, and the map writes are writes rather than an outage.
Eliminating it outright would mean dropping the target-side rule and accepting
a silent fail-open, which is the worse trade.

The chainsaw case's header claimed the deletion "briefly made fail-closed
admission webhooks unreachable". That is the unconfirmed hypothesis, and the
errno in the report contradicts it: EHOSTUNREACH cannot come from a policy
denial, which drops after connect() has returned. It now says what the case
actually asserts -- both webhooks still admit across a namespace deletion --
rather than claiming to reproduce a cause.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-09-02 20:52:06 +05:00
Myasnikov Daniil 3daa83f4b5 feat(cilium): surface a kube-proxy-replacement override instead of nothing
The pin added earlier guards a cozystack developer changing the default. It
cannot observe an operator override in the field, and neither can the e2e,
which only ever runs the default configuration -- so the path Ivan's review
actually names, addons.cilium.valuesOverride on a tenant cluster and the
cozystack.networking Package's cilium component on the host, produced no signal
at all. Both webhook deny policies name their webhook's backend port only,
which covers a pod dialling the ClusterIP solely because Cilium performs the
service DNAT ahead of its own egress hook, so an override degrades that half
silently.

A status-beacon ConfigMap rather than a fail(), following the -awaiting-etcd
beacon in packages/apps/kubernetes/templates/cluster.yaml. Whether Cilium
without kube-proxy replacement is a supported cozystack configuration is a
platform decision, and refusing to render the CNI over a weakened
defence-in-depth control is not a call these charts should make. The host
beacon fires on a bool false, on the string "false", and on the umbrella key
being deleted, because the subchart's own "false" default coalesces into the
parent. The tenant beacon fires on an explicit non-true override only, since
absence means the chart default applies, and it renders on the management
cluster next to the HelmRelease carrying the override -- which is why the
mismatch is detectable there at all rather than inside the guest cluster.

Also withdraws the argument the suite header used to carry, that a target-side
ingressDeny could not be re-added as a fallback. It was wrong, and the next
commit adds that rule. The header now records what the two halves each buy.

Makefile gets .PHONY: test for the reason this branch already argued for
kubeovn-webhook, which applies with more force here now that the suite is
load-bearing for two other packages: hack/package.mk's .PHONY list deliberately
carries only the targets that file defines, so a path named `test` would
otherwise make the target up to date and skip the recipe. Proven both ways.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-09-02 20:44:53 +05:00
Myasnikov Daniil 6a289cf6d0 test(cilium): pin the service-LB assumption both webhook denies rest on
Both admission-webhook deny policies name only their webhook's backend port.
That covers a pod dialling the ClusterIP because Cilium translates a service to
its backends before it enforces egress policy -- in v1.19.5, bpf/bpf_lxc.c runs
__per_packet_lb_svc_xlate_4 and lb4_local() and only then tail-calls into
handle_ipv4_from_lxc where policy_can_egress4 evaluates the rule.

That ordering exists only while Cilium performs the DNAT, and the only reason
it does is cilium's kubeProxyReplacement: true, set against an upstream chart
default of "false". Set it false and the DNAT moves downstream of Cilium's
egress hook: the packet arrives with dst = ClusterIP, matches no endpoint
identity, and with default-deny off it is allowed. Both webhooks regain a pod
-reachable ClusterIP path while the backend deny keeps working, so nothing
fails and nothing is logged. Same shape as the port disagreement
namespace-annotation-webhook.port already exists to prevent, and it had no pin.

The suite asserts the rendered cilium-config key rather than .Values, so an
upstream bump that renames it or stops honouring the override is caught too,
and it spells out each of the five variant valuesFiles combinations from
sources/networking.yaml so a future variant file that overrides the value reds
here. Proven both ways: flipping values.yaml reds all six cases, and adding the
override to values-kilo.yaml alone reds exactly the cilium-kilo case.

The suite also records why a target-side ingressDeny is not available as a
belt-and-braces fallback. A target-side rule needs a peer selector covering
pods, and the peer selector is what carries namespace churn -- the failure
being removed here. pkg/policy/selectorcache.go UpdateIdentities notifies a
selector only when the churned identities were in its own cached selections,
and pkg/policy/l4.go IdentitySelectionUpdated then writes the policy map of
every endpoint the rule's endpointSelector selects, which for a target-side
rule is the fail-closed webhook pods. The one churn-free peer is the wildcard,
which Cilium collapses to a single ANY-identity entry and skips identity
updates for -- but ANY covers reserved:host, remote-node and kube-apiserver,
so it would deny the API server. There is no third shape.

Refs #3983.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-09-02 19:17:24 +05:00
Myasnikov Daniil c9ba7a3cca fix(platform): give node DaemonSets memory requests and limits
The Talos userspace OOM handler (v1.12+) ranks cgroups with

  memory_max.hasValue() ? 0.0 :
    {Besteffort: 1.0, Burstable: 0.5, ...}[class] * memory_current

and discards every cgroup scoring zero, so only a memory limit takes a
pod out of the victim set. Selection is decoupled from the trigger: a
tenant workload thrashing against its own limit drives node memory PSI,
and a limit-free system DaemonSet is killed for it. metallb-speaker and
linstor-satellite were the observed victims.

Set requests and limits on the node DaemonSets that had none. Requests
come from 14d peak working set measured on a 3-node dev cluster; limits
are deliberately loose, sized as a runaway backstop rather than a fitted
ceiling, because several of these scale with cluster size that a dev
cluster understates and because undersizing converts a rare
pressure-driven kill into a deterministic one. A limit costs nothing
until it is hit -- only the request reserves.

memory.max lands on the POD cgroup only when EVERY container in the pod
carries a limit, so sidecars and init containers are covered too; one
limit-free container returns the whole pod to the victim set.

Two DaemonSets are not chart-rendered and needed their own seam:
linstor-satellite via the LinstorSatelliteConfiguration podTemplate, and
virt-handler via KubeVirt customizeComponents.patches, since virt-operator
owns that DaemonSet. Note linstor-controller stays deliberately limit-free
-- it is a JVM that must not be OOM-killed mid-operation, and cluster_test
pins that.

Two things the multus manifest needed beyond the values change.
customize-deployment.patch is the source for that template, so the
limits are carried there rather than only in the generated file, which
`make update` would discard. And the reason kube-multus had no memory
limit was that this patch DELETED upstream's 50Mi rather than raising
it -- 50Mi being well under the 230Mi measured peak -- which is what
removed memory.max from the pod cgroup in the first place.

Not covered, because upstream hardcodes the containers with no resources
knob: the four frr-k8s cp-* init containers, the kube-ovn hostpath-init
and install-cni init containers, and cozy-proxy, whose vendored chart
exposes no resources value at all. hami and kilo are left alone as
optional components with no usage measurements behind them. All of these
still get a limit from the default LimitRange the operator maintains in
every system namespace.

Every limit is pinned by a helm-unittest assertion, and every assertion
was mutation-tested: changing the rendered value turns its suite red.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-13 16:47:52 +05:00