mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-09 14:02:56 +09:00
IvanHunters
42eafd4381
## What Add a bounded escalation tier to the E2E cilium endpoint-leak healer: when a pod stays wedged on `FailedCreatePodSandBox` "IP ipv4:<X> is already in use" for the same IP across several heal cycles, restart the cilium-agent on that node. Keep the existing per-endpoint remedies as the first tier. ## Why The leak (cilium/cilium#38313 class) sometimes lives purely in the cilium-agent's in-memory endpoint state. The existing remedies — `cilium-dbg endpoint disconnect` and delete-rescheduling the wedged pod — cannot reach it: IPAM keeps re-handing the SAME released IP, so the recreated pod re-wedges on it, and on some variants the stale entry is not even in the queryable endpoint registry, so disconnect is a no-op. Both are inescapable loops that fail the install. The only thing that clears an in-memory leak is restarting the agent, which drops the node's endpoint registry and rebuilds it from CRDs/IPAM without the stale entry. ## How - Track recurrence per `(node, ip)`. After `ESCALATE_AFTER` (default 3) heal cycles of a futile first-tier remedy on the same IP, restart that node's cilium-agent pod. - Bounded by `MAX_ESCALATIONS_PER_NODE` (default 2); after the cap, log loudly and surface the failure instead of restarting again. State is per-node and resets after each restart so one restart covers every leaked IP on the node. - A genuine duplicate-IP (endpoint backs a live owner) is refused before counting, so an agent restart can never mask a real product bug. - RBAC unchanged: the existing cluster-wide `pods` `get/list/delete` already covers deleting a cilium-agent pod in `cozy-cilium`. E2E-test-only resilience; no product code change. Remove alongside the rest of the healer once a fixed Cilium ships.