mirror of
https://github.com/cozystack/cozystack.git
synced 2026-10-30 16:47:20 +09:00
Aleksei Sviridkin
3a82ed43bc
The metrics contract runs a one-shot curl Pod and reads a `code=<status>` marker back through `kubectl run --attach`. Attach connects only after the Pod reaches Running, so a container that exits before the connection is up writes its stdout into the void: the step then sees only kubectl's own "If you don't see a command prompt, try pressing enter." and reports the metrics endpoint as broken. curl against a Pod IP finishes in milliseconds, so losing that race is routine, and `--rm` compounded it by deleting the Pod the instant it exited, taking the log that still held the real output. The result was a false negative on a healthy cluster, non-deterministic in the way that reads as a flake because whether attach wins depends on Pod startup timing rather than on anything under test. Drop `--rm`, keep the Pod until its result is in hand, and re-read the log when the attach stream carried no marker. The recovery cannot mask a real failure: the marker encodes the outcome, so a recovered `code=503` stays a failure, and a stream that already carried a marker is never second-guessed. The pre-run delete stays blocking so the re-read cannot find a previous incarnation's marker, and the post-run delete is guarded so a transient API error cannot abort the step with the result in hand but unreported. Pin `--pod-running-timeout` at 90s rather than leaning on kubectl's 1m default. curlimages/curl is not otherwise used on these nodes, so the probe pays a cold pull, and a pull that outruns the budget produces the same markerless output this change exists to explain. The resolver is split into a sourced helper so the recovery is unit-tested without a cluster; the kubectl orchestration around it stays covered by the e2e run. The image-cache probe carries the same recovery inline for the same race. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
66 lines
3.5 KiB
Bash
66 lines
3.5 KiB
Bash
# shellcheck shell=bash
|
|
# Sourced by the chainsaw etcd Tests after `cd` to the repo root. Provides:
|
|
# etcd_probe_resolve — recover a throwaway probe Pod's result marker when
|
|
# `kubectl run --attach` lost the container's stream
|
|
# Sourcing has no side effects: it defines a function and nothing else. The
|
|
# function itself assigns two `_probe_`-prefixed variables without `local`,
|
|
# which is not POSIX and so unavailable to the `sh` Chainsaw runs these steps
|
|
# under; the prefix is what keeps them from colliding with a caller's names.
|
|
#
|
|
# Why this exists. The metrics contract in
|
|
# hack/e2e-chainsaw/etcd/chainsaw-test.yaml proves the member Pods serve
|
|
# Prometheus metrics over plaintext http by running a one-shot curl Pod and
|
|
# reading a `code=<status>` marker off its output. `kubectl run --attach`
|
|
# establishes the attach connection only after the Pod reaches Running, and a
|
|
# container that exits before that connection is up writes its stdout into the
|
|
# void — the caller gets kubectl's own chatter ("If you don't see a command
|
|
# prompt, try pressing enter.") and no marker. curl against a Pod IP completes
|
|
# in milliseconds, so losing the race is routine, not exotic.
|
|
#
|
|
# The failure that follows is a false negative: the step reports the metrics
|
|
# endpoint as broken on a cluster that was serving 200 throughout. It is also
|
|
# non-deterministic in the way that reads as a flake — whether attach wins
|
|
# depends on Pod startup timing (CNI interface setup and image pull have each
|
|
# been observed adding seconds), so the same suite passes on the next run
|
|
# against identical code.
|
|
#
|
|
# The recovery is that the container's stdout is not actually gone: kubelet
|
|
# wrote it to the Pod's log. So a lost stream is re-read rather than believed.
|
|
# hack/e2e-chainsaw/_lib/talos-image-cache.sh solves the same race inline for
|
|
# the image-cache probe (root cause from the same class of bug); that inline
|
|
# copy is what identified this call site as still carrying the race. This is
|
|
# the extracted, unit-tested form — see hack/etcd-probe_test.bats. It is not
|
|
# yet shared: talos-image-cache.sh still carries its own copy, and folding it
|
|
# onto this helper would need a name that is not etcd-specific.
|
|
#
|
|
# Three conditions make the re-read safe rather than a way to mask real
|
|
# failures. The marker encodes the outcome, so a `code=503` recovered from the
|
|
# log stays a failure. An attach stream that already carried a marker is never
|
|
# second-guessed. And the caller must delete any previous Pod of the same name
|
|
# *and wait for it to be gone* before running the probe, then not pass `--rm`:
|
|
# without the blocking pre-delete the re-read could find a previous
|
|
# incarnation's marker, and `--rm` would destroy the log this recovery reads.
|
|
#
|
|
# Usage:
|
|
# out=$(etcd_probe_resolve "$attach_output" kubectl -n "$ns" logs "$pod")
|
|
#
|
|
# Returns the attach output when it carries a marker; otherwise the log re-read
|
|
# when that carries one; otherwise the attach output unchanged, so whatever
|
|
# kubectl said about the failure survives into the caller's error message. A
|
|
# failing re-read (evicted or already-collected Pod) is tolerated rather than
|
|
# propagated, so it cannot kill a `set -e` caller before it reports the result.
|
|
etcd_probe_resolve() {
|
|
_probe_attach_out="$1"
|
|
shift
|
|
if printf '%s' "$_probe_attach_out" | grep -q 'code='; then
|
|
printf '%s' "$_probe_attach_out"
|
|
return 0
|
|
fi
|
|
_probe_log_out="$("$@" 2>/dev/null || true)"
|
|
if printf '%s' "$_probe_log_out" | grep -q 'code='; then
|
|
printf '%s' "$_probe_log_out"
|
|
return 0
|
|
fi
|
|
printf '%s' "$_probe_attach_out"
|
|
}
|