fix(seaweedfs): raise master probe timeouts above the cluster-status backoff

The master's /cluster/status:9333 handler calls topo.Leader(), which spins
an exponential backoff with MaxElapsedTime=20s before returning 200 whenever
the node has no leader in its view — e.g. a master that was briefly the initial
raft leader during bootstrap and then was demoted. The chart's master liveness
and readiness probes both httpGet /cluster/status with timeoutSeconds: 10, which
is shorter than that worst-case backoff. Such a master is then liveness-killed
roughly every 137s into a permanent CrashLoop, the master StatefulSet never
reaches full readiness, and the helm --wait install times out.

Raise both master probe timeouts to 30s so each probe outlasts the 20s backoff
(the volume component already uses 30s). The override sets only timeoutSeconds
and relies on Helm's deep-merge to keep the rest of the subchart probe defaults.
Both probes must be raised: leaving readiness at 10s would keep the master from
becoming Ready and still stall the StatefulSet.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
This commit is contained in:
Aleksei Sviridkin committed 2026-06-25 13:39:34 +03:00
1 parent fef9e76505
commit bfc1fbd555
2 files changed
+77

No files matched your search

@@ -0,0 +1,64 @@
suite: seaweedfs master probe timeouts
# Pins the master probe timeout overrides that keep master-0 alive during raft
# bootstrap. The master's /cluster/status handler calls topo.Leader(), which
# backs off up to ~20s (MaxElapsedTime=20s) while the node has no leader in
# view — e.g. a master that was briefly the initial raft leader then was demoted.
# The subchart default 10s probe timeout is shorter than that worst case, so such
# a master is liveness-killed into a permanent CrashLoop and the master
# StatefulSet never reaches full readiness. Both probes must allow more than the
# 20s backoff. The overlay overrides only timeoutSeconds and relies on Helm's
# deep-merge to keep every other subchart probe field, so this also guards that
# the merge does not drop the surrounding probe configuration.
templates:
- charts/seaweedfs/templates/master/master-statefulset.yaml
release:
name: seaweedfs
namespace: cozy-seaweedfs
tests:
- it: raises both master probe timeouts above the cluster-status backoff
asserts:
- equal:
path: spec.template.spec.containers[0].livenessProbe.timeoutSeconds
value: 30
- equal:
path: spec.template.spec.containers[0].readinessProbe.timeoutSeconds
value: 30
- it: preserves the other master probe fields through the deep-merge
asserts:
# liveness: only timeoutSeconds is overridden, the rest survive
- equal:
path: spec.template.spec.containers[0].livenessProbe.httpGet.path
value: /cluster/status
- equal:
path: spec.template.spec.containers[0].livenessProbe.failureThreshold
value: 4
- equal:
path: spec.template.spec.containers[0].livenessProbe.successThreshold
value: 1
- equal:
path: spec.template.spec.containers[0].livenessProbe.initialDelaySeconds
value: 20
- equal:
path: spec.template.spec.containers[0].livenessProbe.periodSeconds
value: 30
# readiness: only timeoutSeconds is overridden, the rest survive
- equal:
path: spec.template.spec.containers[0].readinessProbe.httpGet.path
value: /cluster/status
- equal:
path: spec.template.spec.containers[0].readinessProbe.failureThreshold
value: 100
- equal:
path: spec.template.spec.containers[0].readinessProbe.successThreshold
value: 2
- equal:
path: spec.template.spec.containers[0].readinessProbe.initialDelaySeconds
value: 10
- equal:
path: spec.template.spec.containers[0].readinessProbe.periodSeconds
value: 45
+13
View File
@@ -32,6 +32,19 @@ seaweedfs:
type: "emptyDir"
logs:
type: ""
# The master's /cluster/status handler calls topo.Leader(), which backs off
# up to ~20s (MaxElapsedTime=20s) while the node has no leader in view — e.g.
# a master that was briefly the initial raft leader then was demoted during
# bootstrap. The default 10s probe timeout is shorter than that worst case, so
# such a master is liveness-killed every ~137s into a permanent CrashLoop and
# the master StatefulSet never reaches full readiness, timing out the install.
# Both probes must allow more than the 20s backoff (the volume component in
# this chart already uses 30s). Override only timeoutSeconds; Helm deep-merges
# these onto the subchart defaults, keeping enabled/path/initialDelay/etc.
livenessProbe:
timeoutSeconds: 30
readinessProbe:
timeoutSeconds: 30
volume:
replicas: 2
resizeHook: