mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-04 11:07:03 +09:00
Myasnikov Daniil
660b79401f
A churn-heavy platform install intermittently wedges a pod in
ContainerCreating because the cilium-agent rejects its sandbox with
'[PUT /endpoint/{id}][400] putEndpointIdInvalid "IP ipv4:<X> is already
in use"', repeated until the agent restarts or the install times out.
The disputed IP has no live owner: IPAM and the ipcache released it, but
the agent's in-memory endpointManager still holds a stale Endpoint from a
previously-deleted pod whose CNI DEL was skipped or raced (so
removeReferencesLocked never ran). Neither the operator's CiliumEndpoint
GC nor the agent's veth-based endpoint GC can reap it. This surfaced as
~6 of the post-#2558 e2e failures, each on a different victim pod
(velero node-agent, fluent-bit, fdb-operator, cdi-operator, capi
bootstrap), failing PRs unrelated to the change under test. Tracked
upstream at cilium/cilium#38313 (closed stale); no released or
pre-release Cilium fixes it and no flag disables it.
hack/e2e-cilium-endpoint-leak-healer.sh runs as a file-scoped background
watchdog (setup_file/teardown_file). It detects a wedged pod, confirms
the IP is a true orphan -- an endpoint in the owning node's registry
holds it, that endpoint backs no live Running pod, and a different pod is
wedged requesting the same IP -- then evicts only that endpoint via
'cilium-dbg endpoint disconnect ipv4:<X>' (DELETE /endpoint/{id} ->
unexpose -> removeReferencesLocked). Blast radius is one dead endpoint,
not an agent restart. It is read-only until an orphan is positively
confirmed and refuses (loudly) to touch an endpoint backing a live pod,
so a genuine duplicate-IP bug stays visible rather than masked.
This is an interim CI mitigation, not a product fix; remove the script
and the bats hooks once a fixed Cilium ships.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>