Files
cozystack/hack/etcd-probe_test.bats
T
Aleksei Sviridkin 3a82ed43bc test(etcd): stop losing the metrics probe's output to the attach race
The metrics contract runs a one-shot curl Pod and reads a `code=<status>`
marker back through `kubectl run --attach`. Attach connects only after the
Pod reaches Running, so a container that exits before the connection is up
writes its stdout into the void: the step then sees only kubectl's own
"If you don't see a command prompt, try pressing enter." and reports the
metrics endpoint as broken. curl against a Pod IP finishes in milliseconds,
so losing that race is routine, and `--rm` compounded it by deleting the Pod
the instant it exited, taking the log that still held the real output.

The result was a false negative on a healthy cluster, non-deterministic in
the way that reads as a flake because whether attach wins depends on Pod
startup timing rather than on anything under test.

Drop `--rm`, keep the Pod until its result is in hand, and re-read the log
when the attach stream carried no marker. The recovery cannot mask a real
failure: the marker encodes the outcome, so a recovered `code=503` stays a
failure, and a stream that already carried a marker is never second-guessed.
The pre-run delete stays blocking so the re-read cannot find a previous
incarnation's marker, and the post-run delete is guarded so a transient API
error cannot abort the step with the result in hand but unreported.

Pin `--pod-running-timeout` at 90s rather than leaning on kubectl's 1m
default. curlimages/curl is not otherwise used on these nodes, so the probe
pays a cold pull, and a pull that outruns the budget produces the same
markerless output this change exists to explain.

The resolver is split into a sourced helper so the recovery is unit-tested
without a cluster; the kubectl orchestration around it stays covered by the
e2e run. The image-cache probe carries the same recovery inline for the same
race.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-07-19 20:16:01 +03:00

93 lines
4.5 KiB
Bash

#!/usr/bin/env bats
# -----------------------------------------------------------------------------
# Unit tests for the probe-output resolver in
# hack/e2e-chainsaw/_lib/etcd-probe.sh.
#
# The etcd suite proves the member Pods serve Prometheus metrics over plaintext
# http by running a throwaway curl Pod and reading back a `code=<status>` marker.
# `kubectl run --attach` is how that marker comes back, and it loses the stream
# of a container that exits before the attach connection is established -- curl
# against a Pod IP finishes in milliseconds, so the loss is routine rather than
# exotic. When it happens the
# caller sees only kubectl's own chatter ("If you don't see a command prompt,
# try pressing enter."), reads no marker, and reports the metrics endpoint as
# broken -- a false negative on a healthy cluster.
#
# etcd_probe_resolve is the recovery: the container's stdout is still on disk in
# the Pod's log, so a lost attach stream is re-read rather than believed. These
# tests pin that recovery, the pass-through when attach did carry the marker,
# and the honest surfacing of a genuine non-200 (which must NOT be masked by the
# fallback -- a real failure has to stay a failure).
#
# The resolver is separated from the kubectl orchestration precisely so it can
# be tested without a cluster; the `kubectl run`/`kubectl logs` calls around it
# need a live cluster and are covered by the e2e run, matching how the other
# hack/ helpers are tested.
#
# cozytest.sh's awk parser recognizes only @test blocks and a bare `}` on its
# own line; there is no bats `run` or `$status`, and setup()/teardown() are not
# honored. Each test runs under `set -eu -x`; assertions are direct shell tests
# that exit non-zero on failure. Titles are delimited by ASCII double quotes and
# only [A-Za-z0-9] survives into the generated function name, so keep them
# distinctive in their alphanumeric run.
#
# Run with: hack/cozytest.sh hack/etcd-probe_test.bats
# -----------------------------------------------------------------------------
HACK_DIR="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")" && pwd)"
# shellcheck source=/dev/null
. "$HACK_DIR/e2e-chainsaw/_lib/etcd-probe.sh"
@test "resolver returns the attach output unchanged when it carries the marker" {
out="$(etcd_probe_resolve 'code=200' printf 'code=500')"
# The fallback must not run at all when attach already delivered a marker:
# re-reading the log would be a second source of truth for the same probe.
[ "$out" = "code=200" ]
}
@test "resolver rereads the pod log when the attach stream carried no marker" {
lost="If you don't see a command prompt, try pressing enter."
out="$(etcd_probe_resolve "$lost" printf 'code=200')"
# This is the regression under test: before the fallback existed the lost
# stream was reported verbatim as the probe result and failed the suite on a
# cluster whose metrics endpoint was serving 200 the whole time.
[ "$out" = "code=200" ]
}
@test "resolver rereads the pod log when the attach stream was entirely empty" {
out="$(etcd_probe_resolve '' printf 'code=200')"
[ "$out" = "code=200" ]
}
@test "resolver surfaces a genuine non200 from the attach stream" {
# A real failure must survive the resolver untouched. If this ever returns
# the fallback's value instead, the fallback has become a way to launder a
# broken metrics endpoint into a pass.
out="$(etcd_probe_resolve 'code=503' printf 'code=200')"
[ "$out" = "code=503" ]
}
@test "resolver surfaces a genuine non200 recovered from the pod log" {
lost="If you don't see a command prompt, try pressing enter."
out="$(etcd_probe_resolve "$lost" printf 'code=503')"
[ "$out" = "code=503" ]
}
@test "resolver keeps the lost attach output when the log reread also yields nothing" {
# Both sources empty means the probe genuinely produced no marker. Returning
# the attach text keeps whatever kubectl said about why in the failure
# message, instead of reporting an empty string the reader cannot act on.
lost="Error from server: pods etcd-metrics-probe not found"
out="$(etcd_probe_resolve "$lost" printf '')"
[ "$out" = "$lost" ]
}
@test "resolver tolerates a failing log reread without aborting the caller" {
# `kubectl logs` against an evicted or already-collected Pod exits non-zero.
# Under the suite's `set -e` that must not kill the step before it can
# report the probe result.
lost="If you don't see a command prompt, try pressing enter."
out="$(etcd_probe_resolve "$lost" false)"
[ "$out" = "$lost" ]
}