mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-09 05:25:14 +09:00
Aleksei Sviridkin
7f67d02fb7
Reloader binds its :9090 listener as the last statement of startReloader, so /live and /metrics answer connection refused until everything before it has finished. The blocking part of that prologue is the package-level IsOpenshift = isOpenshift() in pkg/kube, which issues a GET /apis/project.openshift.io against the API server on a context carrying no deadline, before main() is even entered. That latency belongs to the API server, and the container waits it out at a 100m CPU limit. The chart renders no startupProbe, so the only grace period is the liveness budget: initialDelaySeconds 10 + periodSeconds 10 * failureThreshold 5, about 60 seconds. A first start that needs longer is killed mid-prologue, and the restart does not resume it, it runs it again from the beginning, so the loop sustains itself. The blast radius reaches past reloader: the LINSTOR stack lists cozystack.reloader in dependsOn, which fans out to every component of that stack, so a reloader stuck in this loop blocks the whole install. Render a values-driven startupProbe on /live so liveness only begins evaluating once the listener is up. Budget = periodSeconds 5 times failureThreshold 45, so about 225 seconds. timeoutSeconds matches the 5s the chart already gives liveness rather than the kubelet default of 1s, which a throttled process can exceed while holding the socket open. Both liveness knobs the chart already exposes could buy the same grace with no patch to carry. Widening failureThreshold buys it permanently, loosening hang detection for the life of the container. initialDelaySeconds is start-up-only and would genuinely do, but it is a fixed blind window on every start, whereas a startup probe returns liveness to its tight budget the moment the listener binds. The chart is vendored, so the template change is carried as patches/deployment-startup-probe.diff and reapplied by `make update`, which now drives the shared %-update rule with an exact chart version pin instead of pulling whatever is latest. The patch is applied with --forward because patch(1) defaults diverge by implementation over a tree that already carries the change: GNU 2.7.6 refuses and exits 1, Apple 2.0-12u11 takes its "Assume -R? [y]" default and strips the hunk back out with exit 0, and GNU under --batch does the same. --forward refuses in all three. A helm unittest pins the probe, its budget, the no-op path the template promises for an unset value, and the liveness numbers the budget is sized against, so the documented arithmetic cannot drift away from the chart without a red test. The same target asserts that the vendored Chart.yaml still records the pinned version, so raising the pin without re-running `make update` fails instead of leaving the tree and the pin quietly disagreeing. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
63 lines
3.1 KiB
YAML
63 lines
3.1 KiB
YAML
reloader:
|
|
reloader:
|
|
reloadStrategy: annotations
|
|
deployment:
|
|
resources:
|
|
limits:
|
|
cpu: 100m
|
|
memory: 512Mi
|
|
requests:
|
|
cpu: 10m
|
|
memory: 128Mi
|
|
|
|
# -- Startup probe for the `reloader` container. This is the canonical
|
|
# explanation for the cozystack `patches/deployment-startup-probe.diff`
|
|
# template patch, which is what makes the key render at all; the template
|
|
# itself only points back here.
|
|
#
|
|
# Reloader binds its :9090 listener as the last statement of
|
|
# `startReloader`, so /live and /metrics answer connection refused until
|
|
# everything before it has finished. The blocking part of that prologue is
|
|
# not the informer construction, which is lazy: it is the package-level
|
|
# `IsOpenshift = isOpenshift()` in pkg/kube, which issues a
|
|
# GET /apis/project.openshift.io against the API server, with a
|
|
# context.TODO() that carries no deadline of its own, before main() is
|
|
# even entered. Its latency is the API server's, and the container runs at
|
|
# a 100m CPU limit while it waits.
|
|
#
|
|
# Without a startup probe the only grace period is the liveness budget:
|
|
# initialDelaySeconds 10 + periodSeconds 10 * failureThreshold 5, about
|
|
# 60s. A first start that needs longer is killed mid-prologue, and the
|
|
# restart does not resume it, it runs it again from the beginning, so the
|
|
# loop sustains itself.
|
|
#
|
|
# Budget here = periodSeconds * failureThreshold = 5 * 45, about 225s.
|
|
# Several times the window that was not enough, still well inside the
|
|
# install timeout that applies (not Flux's own default: cozystack-operator
|
|
# stamps every generated HelmRelease with --helmrelease-install-timeout,
|
|
# 10m by default), and short enough that a container which is genuinely
|
|
# dead surfaces within a few minutes rather than hiding behind the probe.
|
|
#
|
|
# Why a startup probe rather than the liveness knobs the chart already
|
|
# exposes, neither of which would need a patch:
|
|
# - livenessProbe.failureThreshold buys the grace permanently, loosening
|
|
# steady-state hang detection for the life of the container.
|
|
# - livenessProbe.initialDelaySeconds is start-up-only and would genuinely
|
|
# do, but it is a fixed blind window on every start; a startup probe
|
|
# returns liveness to its tight budget the moment the listener binds.
|
|
# Adaptive against fixed is the trade. It is narrower than it sounds while
|
|
# reloader.enableHA is unset, since /live is then a constant 200 and
|
|
# liveness can only catch the listener dying outright.
|
|
#
|
|
# timeoutSeconds matches the 5s the chart gives liveness and readiness,
|
|
# rather than the kubelet default of 1s, because at a 100m CPU limit the
|
|
# listener can be up while a throttled process still needs more than a
|
|
# second to answer.
|
|
startupProbe:
|
|
httpGet:
|
|
path: /live
|
|
port: http
|
|
periodSeconds: 5
|
|
failureThreshold: 45
|
|
timeoutSeconds: 5
|