mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 20:23:39 +09:00
Reloader binds its :9090 listener as the last statement of startReloader, so /live and /metrics answer connection refused until everything before it has finished. The blocking part of that prologue is the package-level IsOpenshift = isOpenshift() in pkg/kube, which issues a GET /apis/project.openshift.io against the API server on a context carrying no deadline, before main() is even entered. That latency belongs to the API server, and the container waits it out at a 100m CPU limit. The chart renders no startupProbe, so the only grace period is the liveness budget: initialDelaySeconds 10 + periodSeconds 10 * failureThreshold 5, about 60 seconds. A first start that needs longer is killed mid-prologue, and the restart does not resume it, it runs it again from the beginning, so the loop sustains itself. The blast radius reaches past reloader: the LINSTOR stack lists cozystack.reloader in dependsOn, which fans out to every component of that stack, so a reloader stuck in this loop blocks the whole install. Render a values-driven startupProbe on /live so liveness only begins evaluating once the listener is up. Budget = periodSeconds 5 times failureThreshold 45, so about 225 seconds. timeoutSeconds matches the 5s the chart already gives liveness rather than the kubelet default of 1s, which a throttled process can exceed while holding the socket open. Both liveness knobs the chart already exposes could buy the same grace with no patch to carry. Widening failureThreshold buys it permanently, loosening hang detection for the life of the container. initialDelaySeconds is start-up-only and would genuinely do, but it is a fixed blind window on every start, whereas a startup probe returns liveness to its tight budget the moment the listener binds. The chart is vendored, so the template change is carried as patches/deployment-startup-probe.diff and reapplied by `make update`, which now drives the shared %-update rule with an exact chart version pin instead of pulling whatever is latest. The patch is applied with --forward because patch(1) defaults diverge by implementation over a tree that already carries the change: GNU 2.7.6 refuses and exits 1, Apple 2.0-12u11 takes its "Assume -R? [y]" default and strips the hunk back out with exit 0, and GNU under --batch does the same. --forward refuses in all three. A helm unittest pins the probe, its budget, the no-op path the template promises for an unset value, and the liveness numbers the budget is sized against, so the documented arithmetic cannot drift away from the chart without a red test. The same target asserts that the vendored Chart.yaml still records the pinned version, so raising the pin without re-running `make update` fails instead of leaving the tree and the pin quietly disagreeing. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
37 lines
1.9 KiB
Diff
37 lines
1.9 KiB
Diff
# The upstream chart renders liveness and readiness from values but no
|
|
# startupProbe at all, so a slow first start has no grace period of its own and
|
|
# is killed by liveness inside ~60s. Reloader binds its :9090 listener as the
|
|
# last statement of startReloader, after a package-level
|
|
# `IsOpenshift = isOpenshift()` in pkg/kube issues a GET
|
|
# /apis/project.openshift.io on a context with no deadline, before main() is
|
|
# entered, so the probes are refused for as long as that takes. The budget and
|
|
# the reasoning behind the numbers live in the package values.yaml.
|
|
#
|
|
# Chart source: https://github.com/stakater/stakater-charts (reloader)
|
|
# This belongs upstream: the chart already renders the other two probes from
|
|
# values, so a startupProbe block is the same shape of change. Not reported
|
|
# upstream yet. Still absent as of chart 2.2.14 / app v1.4.19, where
|
|
# isOpenshift() is also still eager, so the patch is needed on current upstream
|
|
# and not only on the pinned 2.0.0. Drop this hunk once upstream renders one.
|
|
diff --git a/charts/reloader/templates/deployment.yaml b/charts/reloader/templates/deployment.yaml
|
|
--- a/charts/reloader/templates/deployment.yaml
|
|
+++ b/charts/reloader/templates/deployment.yaml
|
|
@@ -161,6 +161,17 @@
|
|
ports:
|
|
- name: http
|
|
containerPort: 9090
|
|
+ {{- /*
|
|
+ cozystack (patches/deployment-startup-probe.diff): the upstream chart
|
|
+ renders no startupProbe, which leaves a slow first start no grace period
|
|
+ of its own. Why that matters and how the budget was chosen is documented
|
|
+ next to the knob, in the package values.yaml. Rendered with `with`, so an
|
|
+ unset value emits nothing and the template stays upstream-compatible.
|
|
+ */}}
|
|
+ {{- with .Values.reloader.deployment.startupProbe }}
|
|
+ startupProbe:
|
|
+ {{- toYaml . | nindent 10 }}
|
|
+ {{- end }}
|
|
livenessProbe:
|
|
httpGet:
|
|
path: /live
|