token-proxy panicked at startup whenever the very first JWKS fetch
against kubernetes.default.svc failed, which is routine on a fresh
install: CoreDNS, the CNI, or the API-server endpoint may not be ready
in the seconds between the pod scheduling and the cache initialization
running. The pod then entered CrashLoopBackOff with exit code 2 and no
logs, the dashboard HelmRelease never installed, and every release
that depended on it stalled behind it.
Two paths into init() touched the network synchronously: jwk.Cache.Register
with the default WaitReady(true), and an explicit follow-up Refresh.
WaitReady has no built-in timeout, so a permanently-unreachable JWKS
would turn the panic into a different failure mode (an unbounded
startup wait) rather than fixing it. The explicit Refresh on top of
Register was redundant.
Drop the redundant Refresh and pass WithWaitReady(false) on Register
so the cache setup returns immediately. The background worker keeps
retrying: a pure network failure (DNS, dial, TLS) skips Sync's SetNext
and falls back to httprc's 1-second tick, while an HTTP error response
(apiserver 5xx, RBAC 403) honors MinInterval — lower it from 5 minutes
to 5 seconds so a single bad response can no longer hold the pod
NotReady for several minutes. Move the cache construction out of
init() into a testable function; init() now only registers flags.
Background fetch failures are wired into the container log via
httprc.WithErrorSink so a persistent JWKS misconfig produces a
recognisable stdout signature instead of being invisible behind the
readiness probe.
To prevent the new failure mode (silently broken sign-in when JWKS is
permanently misconfigured) from regressing observability, expose
/healthz/live and /healthz/ready: liveness is process-up; readiness
calls jwkCache.Lookup so a permanent misconfig surfaces as 0/1 Ready
and keeps the pod out of service endpoints. A startupProbe pointed at
/healthz/ready with a generous failure budget (~5 minutes) serves as
a backstop: a genuinely stuck pod eventually restarts so the operator
sees RestartCount/CrashLoopBackOff instead of a silently NotReady pod
that nothing escalates. The oauth2-proxy branch keeps its own /ping
probes unchanged.
The sign-in handler distinguishes errJWKSNotReady from a real invalid
token: during the cold-start window users see "Service is starting,
please retry in a moment" instead of the misleading "Invalid token".
Tests cover: WaitReady(false) contract (Register returns in <1s
against unreachable JWKS), background-worker recovery from transient
HTTP 5xx without an explicit Refresh (the production cold-start path),
lazy population, catastrophic-misconfig surface (missing/malformed CA
still returns hard error), errJWKSNotReady disambiguation, SA token
injection, and /healthz/ready 503-then-200 transition. The package
Makefile gains a test-go target wired into the existing 'test' target
so hack/helm-unit-tests.sh picks up the new Go tests automatically.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Aleksei Sviridkin
2abc7c2ab0