mirror of
https://github.com/cozystack/cozystack.git
synced 2026-10-23 03:27:50 +09:00
Aleksei Sviridkin
4e53f5e448
The cilium leak healer escalates an unhealed leaked-IP reuse to a Tier-2 cilium-agent restart. For one variant that restart is both futile and harmful: when the disputed IP is the node's per-node Cilium ingress IP (CiliumNode.spec.ingress.ipv4), held by the reserved:ingress endpoint (allocated from the node pod CIDR whenever Gateway API / Envoy is on), restarting the agent cannot clear it -- the agent re-derives the same ingress IP from the CiliumNode spec on every boot -- and can re-trigger the very race that caused the wedge: the restart rebuilds the host-scope (kubernetes) IPAM bitmap empty, then re-reserves the ingress IP via a path decoupled from pod-endpoint restore, so a CNI ADD in that window can be handed the ingress IP again. This is an upstream cilium bug in host-scope IPAM with Gateway API; no released cilium fixes it as of v1.19.5. Detect that variant by an exact match of the wedged IP against the node's CiliumNode.spec.ingress.ipv4 and skip the Tier-2 restart -- surface the case and reschedule the wedged pod best-effort instead. The match is gated so an empty ingress IP (CiliumNode absent or RBAC denied) never matches, so a false positive can never suppress a restart that would clear a real in-memory leak. Add a CiliumNode read to the healer RBAC and a unit test pinning the exact-match-and-non-empty contract. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
112 lines
4.1 KiB
YAML
112 lines
4.1 KiB
YAML
# Interim in-cluster self-heal for the Cilium endpoint leak (cilium/cilium#38313
|
|
# class). Applied during e2e install (hack/e2e-install-cozystack.bats) and left
|
|
# running for the whole cluster lifetime, so it covers install AND every app
|
|
# test. The heal logic lives in hack/e2e-cilium-endpoint-leak-healer.sh and is
|
|
# shipped to the pod via a ConfigMap built from that file at apply time (single
|
|
# source of truth). REMOVE this file, that script, and the bats wiring once a
|
|
# fixed Cilium ships. CI band-aid, not a product fix.
|
|
apiVersion: v1
|
|
kind: ServiceAccount
|
|
metadata:
|
|
name: cilium-leak-healer
|
|
namespace: kube-system
|
|
labels:
|
|
app.kubernetes.io/name: cilium-leak-healer
|
|
---
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
kind: ClusterRole
|
|
metadata:
|
|
name: cilium-leak-healer
|
|
labels:
|
|
app.kubernetes.io/name: cilium-leak-healer
|
|
rules:
|
|
# find FailedCreatePodSandBox "IP already in use" rejections, cluster-wide
|
|
- apiGroups: [""]
|
|
resources: ["events"]
|
|
verbs: ["get", "list", "watch"]
|
|
# inspect the wedged pod and the endpoint's claimed owner; delete the wedged
|
|
# pod to reschedule it onto a free IP (Tier 1, when the disputed IP is held by
|
|
# a reserved/infra endpoint or a disconnect otherwise fails); and delete the
|
|
# node's cilium-agent pod in cozy-cilium to restart it (Tier 2 escalation,
|
|
# which clears a stale in-memory endpoint leak the DaemonSet then rebuilds
|
|
# without). cluster-wide `delete` already covers cozy-cilium, so no extra verb.
|
|
- apiGroups: [""]
|
|
resources: ["pods"]
|
|
verbs: ["get", "list", "delete"]
|
|
# run `cilium-dbg endpoint ...` inside the owning node's cilium-agent
|
|
- apiGroups: [""]
|
|
resources: ["pods/exec"]
|
|
verbs: ["create"]
|
|
# read CiliumNode.spec.ingress.ipv4 to recognise the per-node reserved:ingress
|
|
# IP. When the disputed IP is exactly that, the Tier-2 agent restart is futile
|
|
# (the agent re-derives the same ingress IP on boot) and harmful (it can
|
|
# re-trigger the host-scope IPAM restore-window race), so the healer skips it.
|
|
- apiGroups: ["cilium.io"]
|
|
resources: ["ciliumnodes"]
|
|
verbs: ["get", "list"]
|
|
---
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
kind: ClusterRoleBinding
|
|
metadata:
|
|
name: cilium-leak-healer
|
|
labels:
|
|
app.kubernetes.io/name: cilium-leak-healer
|
|
roleRef:
|
|
apiGroup: rbac.authorization.k8s.io
|
|
kind: ClusterRole
|
|
name: cilium-leak-healer
|
|
subjects:
|
|
- kind: ServiceAccount
|
|
name: cilium-leak-healer
|
|
namespace: kube-system
|
|
---
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
name: cilium-leak-healer
|
|
namespace: kube-system
|
|
labels:
|
|
app.kubernetes.io/name: cilium-leak-healer
|
|
spec:
|
|
# The pod runs an endless heal loop and is torn down with the ephemeral e2e
|
|
# cluster, so it never completes on its own. backoffLimit only matters if the
|
|
# watchdog itself ever crashes — keep restarting it.
|
|
backoffLimit: 100
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app.kubernetes.io/name: cilium-leak-healer
|
|
spec:
|
|
serviceAccountName: cilium-leak-healer
|
|
restartPolicy: OnFailure
|
|
# Immune to the very leak it heals: host networking needs no Cilium
|
|
# endpoint, so this pod can never be wedged by the leak and can reach the
|
|
# API server before pod networking is fully up. Broad tolerations let it
|
|
# schedule before nodes are Ready and stay put through later taints.
|
|
hostNetwork: true
|
|
dnsPolicy: ClusterFirstWithHostNet
|
|
tolerations:
|
|
- operator: Exists
|
|
containers:
|
|
- name: healer
|
|
# Reuse the in-tree, digest-pinned kubectl+jq image already shipped by
|
|
# the installer (packages/core/installer/templates/cozy-system-labels.yaml).
|
|
image: docker.io/alpine/k8s:1.36.2@sha256:44ef4942e171939b9c665a4a84beb80e2dcdb9a24330d4651cfdfd2e9deecc47
|
|
command: ["/bin/sh", "/opt/healer/heal.sh"]
|
|
env:
|
|
- name: CILIUM_NS
|
|
value: cozy-cilium
|
|
volumeMounts:
|
|
- name: script
|
|
mountPath: /opt/healer
|
|
resources:
|
|
requests:
|
|
cpu: 10m
|
|
memory: 32Mi
|
|
limits:
|
|
memory: 128Mi
|
|
volumes:
|
|
- name: script
|
|
configMap:
|
|
name: cilium-leak-healer
|