mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 12:13:55 +09:00
docs/roadmap-dco
126
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
f61e13fdef |
fix(cilium): size the agent memory limit for its BPF maps (#4803)
## What this PR does This raises the cilium-agent memory limit from 4Gi to 24Gi. The 4Gi limit from #3495 has no room for the agent's BPF maps. On nodes with roughly a terabyte of RAM or more, the agent is OOM-killed on every start after a reboot, and the node loses pod networking. The maps grow with node RAM. Cilium sizes its CT, NAT, neighbor and socket rev-NAT maps as a share of total RAM (`bpf-map-dynamic-size-ratio`, 0.0025 by default), and caps each map at 16Mi entries. These are LRU hash maps, so the kernel preallocates them in full. Since Linux 5.11 that memory is charged to the memcg of the process that creates the maps, which is the agent container. This is also why the upgrade itself works. A map reused from bpffs stays charged to the cgroup that created it. The failure shows up on the first reboot after the upgrade, when the maps are recreated inside the limited cgroup. Policy maps don't belong on that list. The issue and the upstream chart's configmap comment both say the ratio sizes them, but at this version they are LPM tries sized by `bpf.policyMapMax`, and they allocate per entry. One DaemonSet serves every node, so the limit can't follow node RAM. The map sizes are capped, though, so a fixed limit can cover any node. With every map at 16Mi entries for both IPv4 and IPv6, the maps take just under 20Gi of kernel memory. Add the 4Gi the agent already had and you get 24Gi. The request stays at 256Mi. The limit doesn't reserve memory, and the pod cgroup still gets `memory.max`, which is what #3495 was after. What it costs is the leak backstop. The agent is now stopped at 24Gi minus its own map memory, so on most nodes a leak in its heap is no longer caught before the node feels it. If someone prefers a tighter backstop over IPv6 headroom, they can lower `cilium.resources.limits.memory` in the networking component values. About 13Gi still covers the IPv4-only map ceiling. I didn't pin the map sizes (`bpf.ctTcpMax`, `bpf.natMax` and so on). No single value works for every node. A value small enough for a 16Gi node cuts the conntrack capacity of a 1.5TiB node many times over. A pinned size also differs from what almost every node above ~14Gi of RAM runs today, so the agent would recreate those maps on upgrade and drop their conntrack and NAT state. A lower ratio has the same problem and doesn't lower the ceiling. The new test keeps the ratio at its default and the sizes unpinned, so changing either has to be deliberate. The figures are computed, not measured. I read the sizing code at cilium 1.19.5 (`pkg/option/config.go`), and 1.20.2, which #3343 moves to, has the same code. Memory comes from that formula plus the kernel hash map layout: a 48-byte element header, the key and value rounded up to 8 bytes, and a 16-byte bucket per power of two of `max_entries`. By that math the IPv4 maps alone pass 4Gi at roughly a terabyte of RAM. Bucket rounding moves the exact point, so I wouldn't read more precision into it. For the 1.5TiB node in the report, the issue's 3.9GiB is the agent's own budget (0.25% of RAM), and the kernel's actual charge comes out higher. Follow-ups, not in this PR: - Operators who want smaller maps on large nodes can size them per node with a `CiliumNodeConfig` selected by node label. This needs a documentation note. - The reporter's `bpftool` measurement of the real map memory is still pending. That is the measured check for the numbers above. The cilium helm-unittest suite passes on this branch, and also with this change applied over the 1.20.2 chart from #3343. Each new assertion fails when its value changes: the limit, the CT, NAT and neighbor sizes, the socket rev-NAT size set through `extraConfig`, and the ratio. Closes #4800 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The diff changes only `packages/system/cilium/values.yaml` and its test. None of the trigger-map entries apply: no app, schema, platform value, variant, release asset or tooling changes, and no website page documents the agent limit. ### Release note ```release-note fix(cilium): raise the cilium-agent memory limit from 4Gi to 24Gi so it covers the agent's BPF maps, which cilium sizes from node RAM. On nodes with around a terabyte of RAM or more, the 4Gi limit got the agent OOM-killed on every start after a reboot, leaving the node without pod networking. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Increased the Cilium agent memory limit to 24 GiB. * Updated validation for BPF map sizing: checks the dynamic size ratio and confirms that specific map-size settings are absent. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
87a64071a7 |
fix(cilium)!: take cilium's own addresses from a block kube-ovn does not hand out (#4778)
## What this PR does
fixes #3740.
on the kube-ovn variants cilium allocates no pod ips, but the agent
still takes its router ip (`cilium_host`, `CiliumInternalIP`) and the
gateway api ingress ip from its own allocation cidr. with `ipam.mode:
kubernetes` that's `Node.spec.podCIDR`, a /24 out of the same /16
kube-ovn hands pods from, so sooner or later kube-ovn gives a pod an
address cilium already holds. we hit it on three clusters, details in
https://github.com/cozystack/cozystack/issues/3740#issuecomment-6022394243.
**what changes**
- the kube-ovn variants run cilium with `ipam.mode: cluster-pool`, a /29
per node out of the last /21 of `networking.podCIDR` (`10.244.248.0/21`
for the default /16). the platform derives the block from
`networking.podCIDR` and passes the same range to kube-ovn as
`--default-exclude-ips`, so a fresh install never hands it to pods.
- the block sits inside the pod cidr on purpose. envoy reaches gateway
api backends from the node's ingress ip, and kube-ovn masquerades host
originated traffic only when the source is in `ovn40subnets`. with the
`100.65.0.0/16` pool from the issue a nodeport through the gateway
answered 200 only on the backend's node and 503 `upstream connect error
... connection timeout` on the other two. with the block inside the pod
cidr it's 200 from all three. worth a look on the ci side too. the ci
sandbox sets talos `podSubnets` to `100.65.0.0/16`, so cilium's ingress
ips there already come from outside `ovn40subnets`, and the gateway e2e
checks Accepted, Programmed and the lb service but sends no request
through a listener, so it wouldn't notice. with this pr the sandbox
takes its block from the pod cidr like everyone else.
- the end of the /16 is what kube-ovn reaches last. it hands out the
lowest free address and reuses released ones only when the free range
runs dry (`pkg/ipam/subnet.go` and `pkg/ipam/ip_range_list.go` at
v1.15.26). a /21 cut into /29s covers 256 nodes.
- the slim variants and guest clusters stay on `ipam.mode: kubernetes`.
cilium is the only cni there and `Node.spec.podCIDR` is the pod range.
- `networking.podCIDR` now needs /20 or wider on the kube-ovn variants,
otherwise the render fails and says why. an install that rendered before
can stop rendering at upgrade, hence the `!`. a /21 cut into /29s also
caps these variants at 256 nodes, a fresh install included.
**existing clusters**
a post-upgrade hook in the cilium chart moves them
(`templates/ipam-migration.yaml`, `files/ipam-migration.sh`). before it
touches anything it checks that the block has a /29 for every CiliumNode
and that it lies inside kube-ovn's default subnet, and stops with both
cidrs printed otherwise. a node cleared past the pool's capacity would
lose pod networking on its next agent restart, and a block outside the
subnet puts the agents outside `ovn40subnets` again. it has to be
post-upgrade. platform migrations run as the platform's pre-upgrade
hook, while every agent still runs `ipam.mode: kubernetes` and writes
`Node.spec.podCIDR` back into a cleared CiliumNode
(`pkg/nodediscovery/nodediscovery.go` at v1.19.5), so the move can only
happen after the cilium release switched the mode. the job runs on the
host network against the agent's own api endpoint (kubeprism on talos),
because it restarts the agents.
1. adds the block and every node's current ingress ip to `excludeIps` of
the default subnet, in one json patch with a `test` op, and reads the
subnet again on conflict, since kube-ovn rewrites that list into its own
order right after a patch. the old ingress ip goes in because cilium
1.19.5 keeps a former ingress ip in the agent's ipcache as
`reserved:ingress` after the move. we saw 2 to 3 such entries per node
hours later. old router ips leave the ipcache and go back to kube-ovn.
2. stops right there if an address in the block is still allocated,
names it, and moves no agent. it first checks the subnet status it
patched against, then waits until `status.v4availableIPrange` stops
offering the block, which is kube-ovn having applied the exclusion, and
then reads the `ips.kubeovn.io` objects of the default subnet. the
status alone can't answer it, kube-ovn leaves excluded addresses out of
`v4usingIPrange` (`GetSubnetIPRangeString` in `pkg/ipam/ipam.go` at
v1.15.26), so a pod still in the block would be invisible on a retry
where the exclusion is already in place. recreate those pods and the
next upgrade retry goes through.
3. one node at a time, clears the inherited /24 from
`spec.ipam.podCIDRs` (again with a `test` op, so a concurrent writer
fails the patch instead of losing an update), waits for the operator to
allocate a /29 from the block and restarts that node's agent. a node
that isn't ready is skipped and its agent picks up the block when it
starts. nodes already moved are left alone, so a retry after a timeout
just continues. the hook runs inside the cilium upgrade, so where
restarting every agent takes longer than the upgrade timeout, 10m by
default, the release first reports a failed upgrade and the move
finishes on the retry.
running pods keep their addresses. every agent restarts twice on
upgrade, once when the config change rolls the daemonset and once from
the hook, and while it restarts host to pod traffic on that node blips
for a few seconds because the router ip moves. a pod that sits on a
router ip today is reachable again after the move, the node no longer
holds that address. a pod on a node's current ingress ip keeps it and
that node keeps seeing it as `reserved:ingress`, so recreate it. the
hook logs those pods by name.
### Validation
- `helm unittest` passes cilium 23/23, platform 246/246 and kube-ovn
22/22. the new suites pin the rendered `cilium-config` keys per variant,
the hook shape, env and rbac, and the platform wiring for the talos and
generic branches, a custom pod cidr, host bits in the network address,
the refusals for a too small or non ipv4 cidr, and the slim variant.
- on a trial merge with main at
|
||
|
|
9be997b6e2 |
fix(cilium): size the agent memory limit for its BPF maps
cilium sizes its CT, NAT, neighbor and socket rev-NAT maps as a share of node RAM. They are preallocated LRU maps charged to the agent's memcg, so on nodes with around a terabyte of RAM or more the maps alone exceed the 4Gi limit and the agent is OOM-killed on every start after a reboot, leaving the node without pod networking. Raise the limit to the map ceiling (every map at LimitTableMax, both address families) plus the agent's own 4Gi budget. A per-node limit cannot be rendered into one DaemonSet, and pinning the map sizes would cut connection capacity on large nodes and reset conntrack and NAT state on upgrade, so the map sizes stay dynamic and a test keeps them unpinned. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
c9558f5658 |
fix(networking): correct webhook policy assumptions
Read the ingress admission port from the effective subchart values and fail if the port is absent. Describe kube-proxy replacement as a required service-routing invariant while retaining target-side enforcement. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
dc362fa591 |
fix(cilium)!: take cilium's own addresses from a block kube-ovn does not hand out
On the kube-ovn variants Cilium allocates no pod addresses, but the agent still takes its router IP and the Gateway API Ingress IP from its allocation CIDR. In ipam.mode kubernetes that is Node.spec.podCIDR, which the documented installs slice from the same range kube-ovn allocates pods from, so kube-ovn sooner or later hands a pod an address Cilium already holds. Cilium now runs ipam.mode cluster-pool on the kube-ovn variants, with a /29 per node from the last /21 of networking.podCIDR, and kube-ovn excludes that /21 through --default-exclude-ips. The block stays inside the pod CIDR on purpose. Envoy reaches Gateway API backends from the node's Ingress IP, and kube-ovn masquerades host-originated traffic only when the source lies in one of its subnets, so a block outside them such as 100.65.0.0/16 breaks Gateway API traffic that crosses nodes. A post-upgrade hook moves existing clusters. It first checks that the block has a /29 for every node and lies inside kube-ovn's default subnet, and stops without changing anything otherwise. It adds the block and every node's current Ingress IP to excludeIps of that subnet, because the agent keeps a former Ingress IP in its ipcache as reserved:ingress. It then waits until kube-ovn stops offering the block and stops if an address in it is still allocated. kube-ovn leaves excluded addresses out of the subnet status, so that check reads its IP objects. Then, one node at a time, it clears the inherited /24 from the CiliumNode, waits for the operator to allocate from the block and restarts the agent. networking.podCIDR now needs a prefix of /20 or shorter on the kube-ovn variants, and the render fails otherwise. A /21 cut into /29s also caps these variants at 256 nodes. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> |
||
|
|
a7dfbe6f8c |
chore(deps): update container image & action digests
Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
1b2beac3f8 |
[cilium] Set resources requests/limits on cilium-operator (#2618)
## Summary Companion to #2617 (cilium-agent). The upstream cilium chart ships with `operator.resources: {}`, which triggers a `no CPU limit` conformance warning on the `cilium-operator` Deployment. Override `packages/system/cilium/values.yaml` to set the sizing recommended by the upstream `values.yaml` comment: - `limits` : `cpu: 1000m`, `memory: 1Gi` - `requests` : `cpu: 100m`, `memory: 128Mi` ## Test plan - [ ] `helm template . --namespace kube-system` from `packages/system/cilium/` renders a `resources` block on the `cilium-operator` container of the Deployment (verified locally) - [ ] Re-run the conformance/lint tool — the `cilium-operator has no CPU limit` warning should be gone - [ ] CI passes ### Release note \`\`\`release-note Set CPU/memory requests and limits on the cilium-operator Deployment to clear the "no CPU limit" conformance warning. Values follow the upstream chart's recommended sizing (limits 1000m/1Gi, requests 100m/128Mi). \`\`\` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated Cilium system component resource configuration with CPU and memory requests and limits. [](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2618) <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
686a81a491 |
docs(networking): stop claiming the churn coupling is removed
Restoring the target-side pod deny put a cluster-spanning peer selector back on the webhook endpoint, which is the exact thing the source-side rules were added to take off it. The comments still said the coupling was "structural and removed", and the release note still said deleting a namespace no longer rewrites that endpoint's policy map. Both were true of the previous revision and are not true of this one. What ships now is two denies that fail in different directions: the source-side one keeps enforcing while the webhook endpoint's policy is being recomputed or has overflowed its map, the target-side one keeps enforcing where Cilium is not the service load balancer, and neither subsumes the other. So the coupling is not eliminated, it is made non-load-bearing -- whichever end is degraded the other still denies, and the map writes are writes rather than an outage. Eliminating it outright would mean dropping the target-side rule and accepting a silent fail-open, which is the worse trade. The chainsaw case's header claimed the deletion "briefly made fail-closed admission webhooks unreachable". That is the unconfirmed hypothesis, and the errno in the report contradicts it: EHOSTUNREACH cannot come from a policy denial, which drops after connect() has returned. It now says what the case actually asserts -- both webhooks still admit across a namespace deletion -- rather than claiming to reproduce a cause. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
3daa83f4b5 |
feat(cilium): surface a kube-proxy-replacement override instead of nothing
The pin added earlier guards a cozystack developer changing the default. It cannot observe an operator override in the field, and neither can the e2e, which only ever runs the default configuration -- so the path Ivan's review actually names, addons.cilium.valuesOverride on a tenant cluster and the cozystack.networking Package's cilium component on the host, produced no signal at all. Both webhook deny policies name their webhook's backend port only, which covers a pod dialling the ClusterIP solely because Cilium performs the service DNAT ahead of its own egress hook, so an override degrades that half silently. A status-beacon ConfigMap rather than a fail(), following the -awaiting-etcd beacon in packages/apps/kubernetes/templates/cluster.yaml. Whether Cilium without kube-proxy replacement is a supported cozystack configuration is a platform decision, and refusing to render the CNI over a weakened defence-in-depth control is not a call these charts should make. The host beacon fires on a bool false, on the string "false", and on the umbrella key being deleted, because the subchart's own "false" default coalesces into the parent. The tenant beacon fires on an explicit non-true override only, since absence means the chart default applies, and it renders on the management cluster next to the HelmRelease carrying the override -- which is why the mismatch is detectable there at all rather than inside the guest cluster. Also withdraws the argument the suite header used to carry, that a target-side ingressDeny could not be re-added as a fallback. It was wrong, and the next commit adds that rule. The header now records what the two halves each buy. Makefile gets .PHONY: test for the reason this branch already argued for kubeovn-webhook, which applies with more force here now that the suite is load-bearing for two other packages: hack/package.mk's .PHONY list deliberately carries only the targets that file defines, so a path named `test` would otherwise make the target up to date and skip the recipe. Proven both ways. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
6a289cf6d0 |
test(cilium): pin the service-LB assumption both webhook denies rest on
Both admission-webhook deny policies name only their webhook's backend port. That covers a pod dialling the ClusterIP because Cilium translates a service to its backends before it enforces egress policy -- in v1.19.5, bpf/bpf_lxc.c runs __per_packet_lb_svc_xlate_4 and lb4_local() and only then tail-calls into handle_ipv4_from_lxc where policy_can_egress4 evaluates the rule. That ordering exists only while Cilium performs the DNAT, and the only reason it does is cilium's kubeProxyReplacement: true, set against an upstream chart default of "false". Set it false and the DNAT moves downstream of Cilium's egress hook: the packet arrives with dst = ClusterIP, matches no endpoint identity, and with default-deny off it is allowed. Both webhooks regain a pod -reachable ClusterIP path while the backend deny keeps working, so nothing fails and nothing is logged. Same shape as the port disagreement namespace-annotation-webhook.port already exists to prevent, and it had no pin. The suite asserts the rendered cilium-config key rather than .Values, so an upstream bump that renames it or stops honouring the override is caught too, and it spells out each of the five variant valuesFiles combinations from sources/networking.yaml so a future variant file that overrides the value reds here. Proven both ways: flipping values.yaml reds all six cases, and adding the override to values-kilo.yaml alone reds exactly the cilium-kilo case. The suite also records why a target-side ingressDeny is not available as a belt-and-braces fallback. A target-side rule needs a peer selector covering pods, and the peer selector is what carries namespace churn -- the failure being removed here. pkg/policy/selectorcache.go UpdateIdentities notifies a selector only when the churned identities were in its own cached selections, and pkg/policy/l4.go IdentitySelectionUpdated then writes the policy map of every endpoint the rule's endpointSelector selects, which for a target-side rule is the fail-closed webhook pods. The one churn-free peer is the wildcard, which Cilium collapses to a single ANY-identity entry and skips identity updates for -- but ANY covers reserved:host, remote-node and kube-apiserver, so it would deny the API server. There is no third shape. Refs #3983. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
c9ba7a3cca |
fix(platform): give node DaemonSets memory requests and limits
The Talos userspace OOM handler (v1.12+) ranks cgroups with
memory_max.hasValue() ? 0.0 :
{Besteffort: 1.0, Burstable: 0.5, ...}[class] * memory_current
and discards every cgroup scoring zero, so only a memory limit takes a
pod out of the victim set. Selection is decoupled from the trigger: a
tenant workload thrashing against its own limit drives node memory PSI,
and a limit-free system DaemonSet is killed for it. metallb-speaker and
linstor-satellite were the observed victims.
Set requests and limits on the node DaemonSets that had none. Requests
come from 14d peak working set measured on a 3-node dev cluster; limits
are deliberately loose, sized as a runaway backstop rather than a fitted
ceiling, because several of these scale with cluster size that a dev
cluster understates and because undersizing converts a rare
pressure-driven kill into a deterministic one. A limit costs nothing
until it is hit -- only the request reserves.
memory.max lands on the POD cgroup only when EVERY container in the pod
carries a limit, so sidecars and init containers are covered too; one
limit-free container returns the whole pod to the victim set.
Two DaemonSets are not chart-rendered and needed their own seam:
linstor-satellite via the LinstorSatelliteConfiguration podTemplate, and
virt-handler via KubeVirt customizeComponents.patches, since virt-operator
owns that DaemonSet. Note linstor-controller stays deliberately limit-free
-- it is a JVM that must not be OOM-killed mid-operation, and cluster_test
pins that.
Two things the multus manifest needed beyond the values change.
customize-deployment.patch is the source for that template, so the
limits are carried there rather than only in the generated file, which
`make update` would discard. And the reason kube-multus had no memory
limit was that this patch DELETED upstream's 50Mi rather than raising
it -- 50Mi being well under the 230Mi measured peak -- which is what
removed memory.max from the pod cgroup in the first place.
Not covered, because upstream hardcodes the containers with no resources
knob: the four frr-k8s cp-* init containers, the kube-ovn hostpath-init
and install-cni init containers, and cozy-proxy, whose vendored chart
exposes no resources value at all. hami and kilo are left alone as
optional components with no usage measurements behind them. All of these
still get a limit from the default LimitRange the operator maintains in
every system namespace.
Every limit is pinned by a helm-unittest assertion, and every assertion
was mutation-tested: changing the rendered value turns its suite red.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
|
2823b76802 |
feat(gateway): advertise HTTP/2 via ALPN on Gateway API listeners (#3398)
## What this PR does Browsers negotiate HTTP/2 exclusively through TLS ALPN: if the terminating listener does not advertise `h2`, every client silently falls back to HTTP/1.1, with no error surfaced anywhere. Cozystack's Gateway API path is served by Cilium's embedded Envoy, and Cilium ships with `enable-gateway-api-alpn=false` — so every Gateway listener advertised no ALPN protocols and all browser traffic through a Gateway ran over HTTP/1.1 (observable in browser dev tools as `http/1.1` on every request). The ingress-nginx path this replaces advertised `h2` out of the box, so the migration to Gateway API silently downgraded clients: head-of-line blocking and the ~6-connections-per-origin limit are back for anything served through a Gateway. This PR enables ALPN (`h2`, then `http/1.1`) on Gateway API listeners, default-on, with an opt-out at each management layer: - `packages/system/cilium/values.yaml` now defaults `gatewayAPI.enableAlpn: true`, which renders `enable-gateway-api-alpn: "true"` into the `cilium-config` ConfigMap. - The platform chart gains `gateway.http2` (default `true`) next to the existing `gateway.enabled`/`gateway.attachedNamespaces`. The `isp-full` and `isp-full-generic` bundles thread it as inline component values into the `cozystack.networking` Package CR (`components.cilium.values.cilium.gatewayAPI.enableAlpn`); inline HelmRelease values override the package `valuesFiles`, so an explicit `gateway.http2: false` actually reaches the dataplane rather than being shadowed by the new chart default. On `isp-hosted` the hosting-side CNI owns this setting and the flag is a no-op. - Tenant Kubernetes clusters (`packages/apps/kubernetes`) get the same fix: when the `gatewayAPI` addon is enabled, the tenant cilium HelmRelease now sets `enableAlpn: true`; individual clusters opt out via `addons.cilium.valuesOverride.cilium.gatewayAPI.enableAlpn=false`. ### Why default-on rather than opt-in HTTP/2 for browsers was the platform's behavior before the Gateway API migration, so keeping ALPN off would preserve a regression, not a status quo. Defaulting on is safe because ALPN is a negotiation: clients that cannot speak HTTP/2 simply keep negotiating `http/1.1`, exactly as before. Operators who need the old wire behavior can set `gateway.http2: false` platform-wide. ### Frontend vs backend hop This change affects the client↔gateway hop only. Gateway↔backend connections stay HTTP/1.1 unless a Service explicitly opts in per GEP-1911 by declaring `appProtocol: kubernetes.io/h2c` on its port. Cilium couples that backend-protocol support to the ALPN flag (`enable-gateway-api-app-protocol` flips to true together with it), but it only acts on Services that declare a recognized `appProtocol` value — and no cozystack-shipped Service does today (only plain `http`/`https` values exist in-tree, which Cilium ignores for protocol selection). So upstreams see no change from this PR alone; h2c/gRPC backends become possible to configure, not configured. The setting is cluster-wide Cilium agent config, so it covers the root Gateway and all tenant Gateways at once — Cilium offers no per-Gateway granularity here. Flipping it re-rolls the cilium DaemonSet on upgrade (`rollOutCiliumPods: true`), the same disruption profile as any other cilium config change. ### How it was verified - New helm-unittest suites: `packages/core/platform/tests/bundles_gateway_http2_wiring_test.yaml` asserts the Package CR carries `enableAlpn` true by default and false with `gateway.http2: false` for both cilium bundles; `packages/apps/kubernetes/tests/cilium_gateway_alpn_test.yaml` asserts the tenant HelmRelease default, the addon-off case, and the `valuesOverride` opt-out. Full suites for both charts pass (93 and 184 tests). - `helm template` of `packages/system/cilium` confirms the value is consumed, not just rendered: default produces `enable-gateway-api-alpn: "true"` (and `enable-gateway-api-app-protocol: "true"`) in `cilium-config`; the threaded `false` produces `"false"` for both. ### Screenshots Not a UI change. ### Downstream repositories The website's hand-written platform values table (`content/en/docs/next/operations/configuration/platform-package.md`) needs a row for `gateway.http2` — filed as cozystack/website#625. No other downstream repository restates anything in this diff (no app `values.schema.json`, ansible-consumed platform key, or hack/ tooling changed). - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/pull/625 - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note feat(gateway): Gateway API listeners now advertise HTTP/2 via TLS ALPN (h2, then http/1.1) by default, restoring the browser HTTP/2 support lost in the migration from ingress-nginx. Opt out platform-wide with gateway.http2=false, or per tenant Kubernetes cluster via addons.cilium.valuesOverride.cilium.gatewayAPI.enableAlpn=false. Backend (gateway-to-upstream) connections are unchanged and stay HTTP/1.1 unless a Service declares appProtocol kubernetes.io/h2c. ``` |
||
|
|
235d38dae1 |
Prepare release v1.6.0 (promoted from v1.6.0-rc.4)
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> |
||
|
|
ebe111c439 |
feat(gateway): advertise HTTP/2 via ALPN on Gateway API listeners
Browsers negotiate HTTP/2 exclusively through TLS ALPN "h2". Cilium's Gateway API implementation ships with enable-gateway-api-alpn=false, so every Gateway listener advertised no ALPN protocols and every browser silently fell back to HTTP/1.1 — a regression against the ingress-nginx path, which advertised h2 out of the box. Enable ALPN (h2, then http/1.1) by default on the client-facing hop: - packages/system/cilium: default gatewayAPI.enableAlpn=true. - platform chart: new gateway.http2 value (default true), threaded as inline component values into the cozystack.networking Package for the isp-full and isp-full-generic bundles, so an explicit false overrides the system/cilium valuesFiles default either way. - apps/kubernetes: tenant-cluster cilium gets enableAlpn=true whenever the gatewayAPI addon is enabled; opt out per cluster via addons.cilium.valuesOverride. The backend hop is unchanged: gateway-to-upstream stays HTTP/1.1 unless a Service opts in per GEP-1911 via appProtocol kubernetes.io/h2c (that support is switched on together with ALPN; no cozystack-shipped Service declares such a value today). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Timofei Larkin <lllamnyp@gmail.com> |
||
|
|
d95685c427 |
Prepare release v1.6.0-rc.4
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> |
||
|
|
5b9b420ffe | chore(deps): update container image & action digests | ||
|
|
6df2b3a797 |
chore(packages/system/cilium): bump cilium to 1.19.5
Re-vendors the cilium chart from 1.19.3 to 1.19.5 (latest 1.19 patch) and points the agent image build at v1.19.5. Picks up the published security fixes plus upstream bug fixes (CiliumEgressGatewayPolicy NamespaceSelector corruption, BGP PeerConfig status cleanup, reused-endpoint BPF state race). The make-update vendoring patches re-apply cleanly: the SYS_MODULE/iptables capability lines and the k8s<1.30 AppArmor annotation block are stripped (the guard confirms the AppArmor block is gone, and the rendered agent DaemonSet carries the annotation exactly once via cozystack podAnnotations). Upstream dropped the dev-only values.yaml.tmpl / README.md.gotmpl from the 1.19.5 release tarball. The defunct l2podAnnouncements.interface removal does not affect cozystack, which uses interfacePattern. The BPF host patch (pass-ipv6-to-stack-when-ipv6-datapath-disabled.diff) is refreshed against v1.19.5 bpf_host.c: identical logic, only context lines re-synced. The deployed agent image pin in values.yaml is intentionally left on the current release-line digest; the multi-arch cilium agent is rebuilt and re-pinned during release preparation. Adaptations: - charts/: re-vendored (helm pull cilium 1.19.5 + capability/AppArmor patches) - images/cilium/Dockerfile: ARG VERSION v1.19.3 -> v1.19.5 - images/cilium/patches/: BPF host patch context refreshed for v1.19.5 Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
b1662f7cf6 |
fix(cilium): pass IPv6 to kernel stack when IPv6 datapath is disabled (#2871)
## What this PR does Cozystack enables Cilium's host firewall by default while the Cilium IPv6 datapath stays disabled (`ipv6.enabled=false`). In upstream Cilium this combination drops all node IPv6 on managed devices as "Unsupported L3 protocol" (`DROP_UNKNOWN_L3` in `bpf_host.c`) before any policy evaluation, in both directions. This breaks ICMPv6 Neighbor Discovery — and with it all node-level IPv6, e.g. BGP unnumbered over link-local addresses on L3 fabrics. No `CiliumClusterwideNetworkPolicy` can allow this traffic, because the drop happens before policy enforcement (upstream report: cilium/cilium#33155, closed stale). The cozystack cilium image now carries a minimal BPF patch that passes IPv6 to the kernel stack instead, mirroring the behavior with the host firewall disabled: - `packages/system/cilium/images/cilium/patches/` — three `ETH_P_IPV6` passthrough cases under `#ifndef ENABLE_IPV6` in `bpf_host.c`. The fourth `DROP_UNKNOWN_L3` site (`from_host_to_lxc`) is intentionally not covered: bpf_lxc drops IPv6 without the v6 datapath regardless, so a passthrough there restores nothing. - The multi-stage Dockerfile applies the patch with `patch --fuzz=0` plus a marker-count check, so upstream drift fails the image build loudly. - `make update` dry-runs the patch against the freshly vendored upstream tag, so a cilium version bump surfaces drift immediately. - `values.yaml` documents the consequence: Cilium host policies apply to IPv4 only; node IPv6 is not filtered by Cilium. The same fix was submitted upstream: cilium/cilium#46473. The local patch is carried until it lands in a release Cozystack ships. Verification: the image builds and contains the patched source; a corrupted patch context fails the build with a clear error; the upstream variant of this change passes new BPF unit tests that fail without the fix with exactly the "Unsupported L3 protocol" drop from the field report; existing host-firewall and IPv6 BPF tests still compile. PR e2e exercises runtime BPF compilation of the patched source, since the agent compiles `bpf_host.c` at startup. Docs: cozystack/website#574 Closes #2806 ### Screenshots Not a UI change. ### Release note ```release-note fix(cilium): node IPv6 (ICMPv6 Neighbor Discovery, BGP unnumbered over link-local) now works with the host firewall enabled; Cilium host policies apply to IPv4 only, node IPv6 is not filtered by Cilium ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Pass IPv6 traffic to the kernel stack when the Cilium IPv6 datapath is disabled, preventing it from being treated as an unknown L3 protocol. * **Documentation** * Added clarification on host firewall behavior: with hostFirewall enabled and IPv6 datapath disabled, policies apply to IPv4 only. * **Chores** * Improved the Cilium image build/update flow by validating the IPv6 patch against the upstream Cilium version to ensure compatibility. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
ec0b266f29 |
docs(cilium): document IPv6 passthrough behavior with hostFirewall
State next to the hostFirewall default why the cozystack cilium image deviates from upstream: with the IPv6 datapath disabled, upstream drops all node IPv6 pre-policy, while the patched image hands IPv6 to the kernel stack. Cilium host policies therefore apply to IPv4 only. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
fbfabbfdf2 |
chore(cilium): verify BPF patch applies during make update
The image build already fails on patch drift, but only when the image is built. Check the patch against the freshly vendored upstream tag inside make update as well, so the author of a cilium version bump sees the drift immediately instead of at the next image build. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
5c0d461aaa |
fix(cilium): pass IPv6 to kernel stack when IPv6 datapath is disabled
Cozystack ships Cilium with the IPv6 datapath disabled
(ipv6.enabled=false) and the host firewall enabled. In this combination
upstream bpf_host.c drops all IPv6 on managed devices as unknown L3
("Unsupported L3 protocol") before any policy evaluation, in both
directions. This kills ICMPv6 Neighbor Discovery and BGP unnumbered over
link-local addresses on IPv6/L3 fabrics, and no
CiliumClusterwideNetworkPolicy can allow the traffic back because the
drop happens before policy enforcement.
Carry a BPF patch in the cozystack cilium image that adds an ETH_P_IPV6
passthrough case to the host-firewall protocol switches, mirroring the
behaviour without the host firewall: IPv6 is handed to the kernel stack
instead of dropped. IPv4 host policies remain fully enforced; node IPv6
is not filtered by Cilium.
The patch is applied with patch --fuzz=0 in a multi-stage build, so any
upstream drift in the patched context fails the image build loudly.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
|
e1aaa90ed3 |
Prepare release v1.5.0
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> |
||
|
|
d3958c5bcb |
refactor(build): mode=max registry cache with a main-only warmer (#2938)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
9868c43e53 |
fix(build): read --cache-from from ghcr :latest so PR builds warm-start
PR #2711 switched PR builds to push only :pr-<N>-<sha> tags (PUBLISH_FLOATING=0) to stop concurrent-push 409 conflicts, but left --cache-from pointing at $(REGISTRY)/<img>:latest. PR builds set REGISTRY to the per-CI registry (iad.ocir.io) and never publish :latest there, so every PR build's registry cache lookup 404s and rebuilds cold. Add a CACHE_REGISTRY knob (default ghcr.io/cozystack/cozystack, where release builds do publish :latest) and point every --cache-from at it. Push targets ($(REGISTRY)) and the unique per-PR tags are unchanged, so the #2711 anti-conflict behaviour is preserved while PR builds regain a warm cache from the last release. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
3756f5c459 |
feat(networking): Gateway API support via Cilium (supersedes #2213) (#2470)
## What this PR does
Replaces ingress-nginx as the default ingress layer for cozystack-native
services with **Cilium Gateway API**, materialised per-tenant via a new
`TenantGateway` CRD reconciled by `cozystack-controller`. Supersedes
#2213 / #2208. Stacked on top of #2468.
## Architecture
The chart renders one `gateway.cozystack.io/v1alpha1 TenantGateway` CR
per tenant that owns its own Gateway. The cozystack-controller
reconciles the actual `Gateway`, per-tenant `Issuer`, `Certificate`
(wildcard or per-listener), and HTTP→HTTPS redirect `HTTPRoute` from
there. Helm does not render `Gateway` or `Certificate` directly — that
prevents the Helm-vs-controller race on `Gateway.spec.listeners` that
route-driven listener additions would otherwise cause.
```
┌── TenantGateway CR (rendered by extra/gateway chart for owning tenants)
│
└──▶ cozystack-controller reconciles:
├─ Gateway (with dynamic listeners, label-based allowedRoutes)
├─ Issuer (per-tenant ACME account)
├─ Certificate(s) — wildcard + per-child SANs in DNS-01, per-listener in HTTP-01
├─ HTTPRoute (controller-owned http→https redirect)
├─ Namespace label patching (cozy-* system namespaces from spec.AttachedNamespaces)
└─ Watches HTTPRoute / TLSRoute attachments to materialise listeners
```
## Inheritance: tenants attach by default, opt into their own Gateway
A tenant gets a dedicated Gateway (own LB Service, own LB IP, own
Certificate) only when it explicitly asks via
`tenant.spec.gateway=true`. Every other tenant in the tree publishes
through the Gateway of the nearest ancestor that owns one — same shape
as the existing `_namespace.ingress` inheritance.
The `apps/tenant` chart writes a `namespace.cozystack.io/gateway` label
on each tenant namespace, carrying either the tenant's own name (when
owning a Gateway) or the inherited ancestor name (when inheriting). The
Gateway's listener `allowedRoutes.namespaces.selector` is keyed on that
label, so the same selector admits routes from every namespace pointing
at the owner — descendants and cozy-* system namespaces alike.
cozystack-controller patches the same label onto every namespace in
`tgw.Spec.AttachedNamespaces` and garbage-collects labels it wrote when
an entry is removed from the attach list (annotation-gated GC so labels
written by the apps/tenant chart are never stripped).
In DNS-01 mode, the controller extends the parent's wildcard Certificate
with `<child-apex>` + `*.<child-apex>` SANs per inheriting child, and
adds a `*.<child-apex>` HTTPS listener per child apex referencing the
same cert. Without this expansion the parent's single-level wildcard
cannot match a child route's hostname (`harbor.alice.example.org` is two
labels past the parent's `*.example.org`).
Opt-out into own Gateway is the right choice when: a tenant needs its
own LB IP (DNS pinned, firewall rule), the apex is not derived from the
parent (custom `host` value the ancestor's cert can't cover), or the
tenant wants its own ACME account / cert authority. Otherwise leave
`gateway` unset and inherit.
## End-to-end flow
1. Platform: `gateway.enabled=true` toggles cert-manager solver,
switches per-system-app templates from Ingress to HTTPRoute / TLSRoute.
2. Per tenant: explicit `tenant.spec.gateway=true` renders a
`TenantGateway` CR and triggers the controller to materialise the owner
Gateway. Tenants with the field unset inherit through
`_namespace.gateway` propagation in the tenant chart.
3. Adding a published app under any tenant (owner or inheriting) is just
deploying its HTTPRoute. The controller picks up the hostname, adds a
per-listener Certificate (HTTP-01) or extends the wildcard cert's SANs
(DNS-01), and updates the owner Gateway's listeners.
Defaults stay legacy: `gateway.enabled=false`,
`tenant.spec.gateway=false`. Existing clusters upgrade unchanged.
## External IP allocation — mechanism-agnostic
The per-tenant Gateway's auto-created `LoadBalancer` Service draws its
IP from whatever LB allocator the cluster admin has configured at the
platform layer — same shape as ingress-nginx today. The tenant API stays
mechanism-agnostic — no `gatewayIP` field, no allocator-specific
manifest in the tenant chart. Cozystack itself ships MetalLB installed
but does not render any `IPAddressPool` / `L2Advertisement` /
`BGPAdvertisement` from this chart; admins set up the allocator that
suits their environment (MetalLB pool with L2 / BGP, Cilium LB-IPAM with
announcer, robotlb against a cloud provider, or
`Service.spec.externalIPs` pinning).
If a tenant needs a specific address (DNS already pinned, firewall rule,
etc.), the operator pre-allocates it on the admin side: either
pre-create the Service with `loadBalancerIP` set, or hand the tenant a
reference to a named admin-managed pool. Per-Service IP uniqueness is
the allocator's responsibility — same as for any other LoadBalancer
Service.
## Cert mode: HTTP-01 (default) vs DNS-01 (opt-in)
`publishing.certificates.solver` selects how the controller sources TLS
certs.
**HTTP-01 (default)** — per-app HTTPS listener + per-app Certificate.
New apps require zero platform-side config: deploy the HTTPRoute,
controller does the rest. Inheriting children's hostnames pick up
per-listener certs the same way the owner's own apps do.
**DNS-01 (opt-in)** — single wildcard Certificate (`<apex>` +
`*.<apex>`) covering every published app under the apex, plus per-child
SANs (`<child-apex>` + `*.<child-apex>`) for every inheriting tenant.
Pick this for clusters where many apps share the apex and Let's Encrypt
rate limits matter. The DNS provider account must be able to write TXT
records under every apex level the parent serves — for deeply-nested
inherited children that requires zone delegation or a wide-scope
provider credential.
| `publishing.certificates.dns01.provider` | Required
`publishing.certificates.dns01.<provider>` keys |
| --- | --- |
| `cloudflare` (default) | `cloudflare.secretName`,
`cloudflare.secretKey` |
| `route53` | `route53.region`, `route53.secretName` (and `accessKeyID`
if not on IRSA) |
| `digitalocean` | `digitalocean.secretName` |
| `rfc2136` | `rfc2136.nameserver`, `rfc2136.tsigKeyName`,
`rfc2136.secretName` |
The platform chart wires every provider's keys into `_cluster.dns01-*`;
the per-tenant gateway chart and the cluster-wide `cluster-issuers.yaml`
both read them.
## Strengthens admission coverage on apps.cozystack.io/*
The `cozystack-tenant-host-policy` probe in `gateway.bats` initially
looked like a flake — the apply that was supposed to be VAP-rejected
just succeeded. Investigation traced it to the custom REST handler at
`pkg/registry/apps/application/rest.go`. genericapiserver hands every
storage method a callback into the validating admission chain:
`createValidation` for Create, `updateValidation` for Update,
`deleteValidation` for Delete. `Update` was correctly invoking its
callback since
|
||
|
|
d2e851f63b |
feat: Gateway API support via Cilium
Replace ingress-nginx as the default ingress layer for cozystack-native
services with Cilium Gateway API, materialised per-tenant via a new
TenantGateway CRD reconciled by cozystack-controller.
The extra/gateway chart renders one gateway.cozystack.io/v1alpha1
TenantGateway per tenant; the controller owns the resulting Gateway,
per-tenant Issuer, Certificate(s), and the http->https redirect
HTTPRoute. This avoids the helm-vs-controller race on
Gateway.spec.listeners when route-driven listeners are added.
publishing.certificates.solver selects HTTP-01 (default; per-app HTTPS
listener + per-app Certificate) or DNS-01 (opt-in; single wildcard
Certificate per apex; providers: cloudflare, route53, digitalocean,
rfc2136). HTTP-01 keeps zero-config per-app onboarding; DNS-01 saves
Let's Encrypt budget on apex-shared clusters.
Security model -- seven independent layers, all admission-time gates
fail-closed:
1. Listener allowedRoutes namespace whitelist via the
kube-apiserver-written kubernetes.io/metadata.name label;
plain-HTTP listener is strictly narrower than HTTPS, and HTTPS /
passthrough listeners restrict allowedRoutes.kinds to
HTTPRoute / TLSRoute respectively.
2. cozystack-gateway-hostname-policy VAP on Gateway.
3. cozystack-gateway-attached-namespaces-policy VAP on Package --
blocks tenant-* in gateway.attachedNamespaces.
4. cozystack-tenant-host-policy VAP on Tenant -- blocks spec.host
writes from non-trusted callers.
5. cozystack-namespace-host-label-policy VAP on Namespace -- blocks
writes to namespace.cozystack.io/host from non-trusted callers.
6. Render-time fail in cozystack-basics for tenant-* in
gateway.attachedNamespaces.
7. cozystack-route-hostname-policy VAP on HTTPRoute / TLSRoute,
scoped to tenant-* namespaces.
Within-apex cross-namespace hostname conflict is resolved at reconcile
time by a HostnameConflict condition (cozy-* namespace wins).
Five reconcile paths refuse to rewrite pre-existing objects with the
controller-derived name but no OwnerReference back to the
TenantGateway: Gateway, redirect HTTPRoute, per-tenant Issuer, wildcard
Certificate, per-listener Certificate.
Derived-apex tenants (tenants whose host is a subdomain of a parent
tenant's) auto-enable Gateway when the platform-level gateway.enabled
is on.
Bonus fix: pkg/registry/apps/application/rest.go was ignoring the
createValidation and deleteValidation callbacks that genericapiserver
hands to every storage method, silently bypassing admission for Create
and Delete on every apps.cozystack.io resource (Tenant, MariaDB,
Postgres, Kubernetes, ...). Aligned with the existing updateValidation
pattern.
Always-on changes: cilium.envoy.enabled + cilium.gatewayAPI.enabled
flip to true (extra cilium-envoy DaemonSet, ~100 MB RAM/node idle),
and the cozystack-api admission-chain fix. Everything else is gated
behind gateway.enabled=false and tenant.spec.gateway=false; existing
clusters see no behavioural change until an operator opts in.
Tests: 60+ Go cases in internal/controller/tenantgateway/, admission-
chain regression tests in pkg/registry/apps/application/, helm-unittest
across 7 charts (extra/gateway, core/platform, apps/tenant,
system/cozystack-basics, system/cert-manager-issuers,
system/cozystack-api, system/dashboard), and 14 e2e bats scenarios in
hack/e2e-apps/gateway.bats. New make test-controllers target wired
into pull-requests.yaml.
Deps: sigs.k8s.io/gateway-api v1.4.1, cert-manager v1.17.4.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
|
ed0297eb87 |
refactor(build): standardize image tagging to fix concurrent PR push conflicts
Every package Makefile tagged images via `settag` which, on a non-release commit, collapsed to `:latest`. Concurrent PR builds therefore raced on every image's `:latest` manifest in the OCIR registry. Two outliers (metallb, redis-operator) used hardcoded version tags and collided across PRs even when `settag` would not have. Replace `settag` and the `TAG=latest` fallback with an `image-tags` macro driven by three env vars: - IMAGE_TAG — build-unique tag, always pushed - PUBLISH_VERSIONED — also push :<component-version> - PUBLISH_FLOATING — also push :latest PR CI sets IMAGE_TAG=pr-<N>-<sha> with both PUBLISH_* off, so every push is unique across runs. Release CI sets IMAGE_TAG=<git-tag> with both PUBLISH_* on, preserving the previous release output and adding a uniform :<git-tag> tag across every image. Local builds default to IMAGE_TAG=dev with PUSH=0 already off. Image consumers reference digests pinned into values.yaml by each Makefile, so the tag is decorative for cluster-side pulls; the build-unique tag is sufficient for E2E. Drops the redundant `:<component-version>-<git-tag>` form (cilium, linstor, matchbox, ubuntu-container-disk, kubevirt-*, cluster-autoscaler) — nothing in the repo or downstream values.yaml consumed it. The same information is preserved via the new uniform `:<git-tag>` plus the existing `:<component-version>` on release. ubuntu-container-disk builds a separate image per Kubernetes version in a loop. Switched its primary tag to `:<k8s-ver>-<IMAGE_TAG>` so loop iterations within a single build do not push to the same tag. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
252c18f3e8 |
Prepare release v1.4.0-rc.1
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> |
||
|
|
a4d1423435 |
[cilium] Set resources requests/limits on cilium-operator
The cilium chart ships with operator.resources: {}, which triggers a
'no CPU limit' conformance warning on the cilium-operator Deployment.
Apply the sizing recommended by the upstream values.yaml comment
(limits 1000m/1Gi, requests 100m/128Mi). Companion to the cilium-agent
fix in a previous PR.
Signed-off-by: Matthieu <matthieu@hidora.com>
|
||
|
|
3f36a1b45b |
fix(cilium): rebuild image multi-arch and pin tag to 1.19.3
The previous image digest in values.yaml pointed at a single-arch linux/arm64 manifest because 'make image' was run from an arm64 host with the default buildx platform. Cozystack targets amd64 (Talos build output, E2E runners, most real-world clusters) and also arm64 for hybrid fleets, so Helm install would fail on amd64 nodes with 'no matching manifest for linux/amd64 in the manifest list entries' whenever somebody installed directly from this commit between merge and the next release-tag CI rebuild. Fix: rebuilt the image locally with PLATFORM='linux/amd64,linux/arm64' make image from a buildx docker-container driver, pushed the multi-arch manifest, and refreshed values.yaml with: - digest of the new multi-arch manifest list (verified via 'docker manifest inspect': amd64 sha256:e1977323..., arm64 sha256:8f5ab529...). - tag bumped from 'latest' (emitted by the common-envs.mk settag macro on a non-tagged checkout) to '1.19.3', matching the established convention in every other packages/system/*/values.yaml so reviewers and incident response have a human-readable version anchor independent of digest chasing. The Makefile is left untouched so the CI builder (which only uses the default docker driver) keeps building single-arch for whatever architecture it runs on; multi-arch is a responsibility of the release-tag pipeline or an explicit local rebuild. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
a78505e932 |
chore(cilium): refresh image digest for v1.19.3
Built ghcr.io/cozystack/cozystack/cilium from the refreshed upstream v1.19.3 base image and updated values.yaml with the new digest. Previously values.yaml still pointed at the v1.19.1 cozystack rebuild by digest while Chart.yaml and the Dockerfile were on v1.19.3 — with chart default useDigest=true that would have silently pulled v1.19.1 until the next release-tag rebuild. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
0e4b66a70f |
chore(cilium): bump to v1.19.3
Vendored chart refreshed via make update in packages/system/cilium. Motivation: v1.19.2 fixes a cert-manager HTTP-01 bug on hostnames with both HTTP and HTTPS listeners (cilium#44492, backport PR #44517). This is a prerequisite for upcoming Gateway API work. v1.19.3 is the latest stable release in the v1.19.x line (15 Apr 2026). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
abd6667eb3 |
[cilium] Move AppArmor podAnnotations to a non-Talos-only values file
E2E revealed that the kube-apiserver rejects a DaemonSet carrying a
per-container AppArmor annotation for a container that is not present
in the pod spec:
DaemonSet.apps 'cilium' is invalid:
spec.template.annotations[container.apparmor.security.beta.kubernetes.io/mount-cgroup]:
Invalid value: 'mount-cgroup': container not found
On Talos (isp-full bundle) values-talos.yaml sets
cgroup.autoMount.enabled=false, so the upstream chart does not render
the mount-cgroup init container, and our unconditional podAnnotations
entry for it becomes an invalid reference.
Move the podAnnotations block from the shared values.yaml into a new
values-apparmor.yaml and include it only in the cilium-generic and
kubeovn-cilium-generic PackageSource variants, where
cgroup.autoMount.enabled is explicitly true. Talos variants
(cilium, cilium-kilo, kubeovn-cilium) get no AppArmor annotations,
which is safe because the Talos kernel does not load the AppArmor
LSM anyway.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
|
fc9ef55c42 |
[cilium] Harden vendoring guard and clarify Talos annotation comment
- Broaden the grep guard in the Makefile update: target to match any container.apparmor.security.beta.kubernetes.io key, not only the quoted cilium-agent line. The previous pattern would silently pass if upstream switched to an unquoted annotation value, masking a failed perl patch; grepping for the prefix catches any residual hardcoded AppArmor annotation regardless of format. - Rewrite the values.yaml comment to distinguish the two Talos-safe reasons: mount-cgroup is simply not rendered when cgroup.autoMount.enabled is false (Talos default), while the other annotations are harmless because kubelet ignores AppArmor metadata on nodes without the LSM loaded. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
0b0d3de99d |
[cilium] Clarify nsenter/AppArmor comment wording
Replace the misleading 'nsenter --ptrace' shorthand with an accurate description: nsenter joins the host cgroup/mount namespace, and the cri-containerd.apparmor.d profile reports the denial as a blocked 'ptrace' operation (operation name, not an nsenter CLI flag). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
39cd2658b5 |
[cilium] Add fail-fast guard for the vendored daemonset patch
The perl multi-line delete in the update: target silently becomes a no-op if the upstream template is ever reformatted. Verify the patch applied by grepping for one of the stripped annotation keys after the perl runs; fail the update: target loudly if it still exists, so a future upstream reformat is caught instead of silently reintroducing the duplicate-key rendering on k8s < 1.30. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
5a50df800d |
[cilium] Strip the entire k8s<1.30 AppArmor block from vendored daemonset
The previous sed patch removed only the four annotation lines and
left an empty nested '{{- if ... }}{{- end }}{{- end }}{{- end }}'
skeleton behind. That skeleton was dead code, but it also meant that
the next make update would re-fetch the upstream chart with the
annotations back inside the same block, and the sed would strip them
again — leaving the skeleton behind forever.
Replace the per-line sed patch with a single perl -i -0pe multi-line
delete that removes the entire <1.30 AppArmor block in one step, and
apply the same patch to the currently vendored copy for consistency.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
|
182fc0b52d |
[cilium] Strip hardcoded AppArmor annotations from vendored daemonset
The upstream chart emits per-container AppArmor unconfined annotations inside a k8s<1.30 branch of cilium-agent/daemonset.yaml. Cozystack now owns these annotations via cilium.podAnnotations in values.yaml so they also take effect on k8s>=1.30 (where upstream drops them in favour of the containerd-broken podSecurityContext.appArmorProfile). Having both sources emit the same keys produced a duplicate mapping on k8s<1.30. Add a new SED_INPLACE invocation to the package Makefile that removes the four hardcoded annotation lines from the vendored template, and apply the patch to the currently vendored copy so make update will reproduce the same state on future refreshes. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
703dbca734 |
[cilium] Document duplicate-key rendering on unsupported k8s < 1.30
On Kubernetes versions below 1.30 the upstream cilium chart still emits the deprecated per-container AppArmor annotations from its own semverCompare '<1.30.0' branch. Our podAnnotations override then emits the same keys again, yielding a YAML mapping with duplicate entries (identical values, last-wins by kubernetes API server JSON semantics). Cozystack's supported Kubernetes matrix starts at 1.30 (packages/apps/kubernetes/files/versions.yaml), so this rendering quirk never affects any shipped version. Document the behaviour in a comment so future readers are not surprised. No functional change. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
2531ab661a |
[cilium] Drop mount-bpf-fs from AppArmor unconfined annotations
mount-bpf-fs runs as privileged in the vendored chart, and the Linux kernel does not apply AppArmor profiles to privileged containers. The per-container unconfined annotation for this container is a no-op and only widens the pod's admission-policy surface for no reason. Keep the four meaningful annotations (cilium-agent, clean-cilium-state, mount-cgroup, apply-sysctl-overwrites) which target unprivileged agent and init containers actually affected by the cri-containerd.apparmor.d profile. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
6c0ae2570e |
[cilium] Opt-out of cri-containerd.apparmor.d for nsenter init containers
Cilium 1.19 init containers mount-cgroup, apply-sysctl-overwrites, mount-bpf-fs and clean-cilium-state call nsenter --ptrace, which is denied by the cri-containerd.apparmor.d profile on Ubuntu 22.04+ and any other distro loading this AppArmor profile. As a result cilium agent pods land in Init:CrashLoopBackOff and downstream HelmReleases cascade into "dependency not ready". The modern k8s >= 1.30 pod-level appArmorProfile: Unconfined field is accepted into the DaemonSet spec but not honoured by containerd CRI at runtime (kubernetes/kubernetes#125069), so the deprecated per-container AppArmor annotations remain the only reliable opt-out. Add the annotations via podAnnotations in the wrapper values.yaml. The change is harmless on systems without AppArmor (Talos, RHEL/Rocky/AlmaLinux with SELinux) because kubelet silently ignores unknown AppArmor annotations when the LSM is not loaded. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
748f814523 |
chore: update cilium to 1.19.1
Signed-off-by: Artem Bortnikov <brongineer747@gmail.com> |
||
|
|
b3b7307105 |
fix(kilo): use official kilo image and clean up cilium-kilo config
Switch kilo image to official ghcr.io/cozystack/cozystack/kilo:v0.8.2, remove unnecessary enable-ipip-termination from cilium-kilo values, and update platform source digest. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
|
536766cffc |
feat(platform): add cilium-kilo networking variant
Add a new networking variant that integrates Kilo with Cilium pre-configured. Cilium is deployed with host firewall disabled and enable-ipip-termination enabled, which are required for correct IPIP encapsulation through Cilium's overlay. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
|
f20c7c4890 |
[cilium] change cilium-operator replicas to 1 (#1784)
<!-- Thank you for making a contribution! Here are some tips for you: - Start the PR title with the [label] of Cozystack component: - For system components: [platform], [system], [linstor], [cilium], [kube-ovn], [dashboard], [cluster-api], etc. - For managed apps: [apps], [tenant], [kubernetes], [postgres], [virtual-machine] etc. - For development and maintenance: [tests], [ci], [docs], [maintenance]. - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same [label] as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note [cilium] change cilium-operator replicas to 1 ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated operator configuration settings. Added support for configuring the number of operator replicas, enabling adjustment of deployment specifications based on infrastructure requirements and resource availability. <sub>✏️ Tip: You can customize this high-level summary in your review settings.</sub> <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
4ffe453351 |
Prepare release v1.0.0-alpha.2
Signed-off-by: cozystack-bot <217169706+cozystack-bot@users.noreply.github.com> |
||
|
|
faa419f535 |
Added a customizable kilo system chart
Co-authored-by: Andrei Kvapil <kvapss@gmail.com> Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
e469021e87 |
[kilo] Introduce kilo
Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
|
464b6b3bb6 |
[cilium] Update cilium to v1.18.6
Signed-off-by: Kirill Ilin <stitch14@yandex.ru> |
||
|
|
09290b5586 |
Prepare release v1.0.0-alpha.1
Signed-off-by: cozystack-bot <217169706+cozystack-bot@users.noreply.github.com> |