mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-09 14:02:56 +09:00
Aleksei Sviridkin
2abc7c2ab0
token-proxy panicked at startup whenever the very first JWKS fetch against kubernetes.default.svc failed, which is routine on a fresh install: CoreDNS, the CNI, or the API-server endpoint may not be ready in the seconds between the pod scheduling and the cache initialization running. The pod then entered CrashLoopBackOff with exit code 2 and no logs, the dashboard HelmRelease never installed, and every release that depended on it stalled behind it. Two paths into init() touched the network synchronously: jwk.Cache.Register with the default WaitReady(true), and an explicit follow-up Refresh. WaitReady has no built-in timeout, so a permanently-unreachable JWKS would turn the panic into a different failure mode (an unbounded startup wait) rather than fixing it. The explicit Refresh on top of Register was redundant. Drop the redundant Refresh and pass WithWaitReady(false) on Register so the cache setup returns immediately. The background worker keeps retrying: a pure network failure (DNS, dial, TLS) skips Sync's SetNext and falls back to httprc's 1-second tick, while an HTTP error response (apiserver 5xx, RBAC 403) honors MinInterval — lower it from 5 minutes to 5 seconds so a single bad response can no longer hold the pod NotReady for several minutes. Move the cache construction out of init() into a testable function; init() now only registers flags. Background fetch failures are wired into the container log via httprc.WithErrorSink so a persistent JWKS misconfig produces a recognisable stdout signature instead of being invisible behind the readiness probe. To prevent the new failure mode (silently broken sign-in when JWKS is permanently misconfigured) from regressing observability, expose /healthz/live and /healthz/ready: liveness is process-up; readiness calls jwkCache.Lookup so a permanent misconfig surfaces as 0/1 Ready and keeps the pod out of service endpoints. A startupProbe pointed at /healthz/ready with a generous failure budget (~5 minutes) serves as a backstop: a genuinely stuck pod eventually restarts so the operator sees RestartCount/CrashLoopBackOff instead of a silently NotReady pod that nothing escalates. The oauth2-proxy branch keeps its own /ping probes unchanged. The sign-in handler distinguishes errJWKSNotReady from a real invalid token: during the cold-start window users see "Service is starting, please retry in a moment" instead of the misleading "Invalid token". Tests cover: WaitReady(false) contract (Register returns in <1s against unreachable JWKS), background-worker recovery from transient HTTP 5xx without an explicit Refresh (the production cold-start path), lazy population, catastrophic-misconfig surface (missing/malformed CA still returns hard error), errJWKSNotReady disambiguation, SA token injection, and /healthz/ready 503-then-200 transition. The package Makefile gains a test-go target wired into the existing 'test' target so hack/helm-unit-tests.sh picks up the new Go tests automatically. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>