mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-09 22:13:18 +09:00
Aleksei Sviridkin
e80b8e370f
The failure snapshot selected pods by their STATUS column, so a pod whose container runs while its readiness probe never passes -- STATUS Running, READY 0/1 -- was dropped from the report entirely. That is a common way for a suite to time out: the replica never joins, its owning HelmRelease sits in InProgress, and the artifact carries no trace of the pod the run died on. Select on the READY column instead, which also covers the partially ready multi-container case, and keep every phase that was collected before. On one degraded cluster that is four of the seven broken pods the old filter reported nothing about. The VMI and VM sections had the same blind spot and now share one selector that reads READY as the last field, because a Pending object prints fewer columns than a Running one and the two kinds differ only in how many columns precede READY. READY counts ready containers rather than the pod's Ready condition, so a pod held back by an unsatisfied readiness gate is out of scope; a test pins that rather than leaving it to be found in an empty directory. Logs are requested for every container of a selected pod rather than skipped for Pending ones, and the per-pod collection moved into a helper so it can be driven against a stub kubectl. kubectl's refusal is the finding, and an empty result is ambiguous on its own: a read against an unscheduled pod exits 0 having printed nothing, which is what a container that started and stayed quiet also leaves. Every site that classifies a failed read now orders its branches the same way, naming the timeout first where both a cutoff and an uncapturable stderr apply. The CRD gate in the summary had no such branch at all: it returned in silence, which is byte-identical to the CRD being absent, so every section behind that gate vanished from a report whose only real problem was that the machine writing it had nowhere to put a temp file. Collection used to stop silently. A walk that hit its cap said so in the job log and nowhere in the artifact. Now the pod walk leaves COLLECTION-TRUNCATED.txt when it stops at COZY_REPORT_PODS_MAX or COZY_REPORT_PODS_BUDGET, CONTAINER-LOGS-TRUNCATED.txt on a pod cut off part way through, COLLECTION-FAILED.txt when the pod list never came back rather than the empty tree a healthy cluster produces, and COLLECTION-UNBOUNDED.txt when timeout is missing locally, which is a fact about the machine writing the report and not about the cluster. Tenant namespaces are collected first, so a cap that only bites on a broadly degraded cluster is not spent on platform pods while the pod the suite died on falls off the end. Container logs are read with --tail=2000 where they were previously read whole, and every bounded file states the bound it was written under, because --tail drops the oldest lines and nothing else in the output would say so. COZY_REPORT_POD_TAIL=-1 restores the complete log. The bounded whole-object read no longer merges stderr into pod.yaml, vm.yaml and vmi.yaml, which are read by yq rather than by eye: a warning on an otherwise complete read left an unparseable file. stderr goes to a scratch file and is copied in only when kubectl returned no object, and the truncation markers those files carry are written as YAML comments, since a bare marker line after a complete mapping is a second top-level node. Six latent faults surfaced while making these reads report. The kamaji module was gated on the LINSTOR controller, so a cluster without LINSTOR got no kamaji evidence at all, which is what a tenant-kubernetes bootstrap timeout needs most. The Flux sources section nested one jsonpath filter inside another, which kubectl rejects outright, so it printed nothing on every report ever taken and read as "all sources Ready". The HelmRelease listing took a single field as STATUS where STATUS is the Ready condition's whole message, so every failed release rendered as one word. The services section filtered for a pending LoadBalancer address on the CLUSTER-IP column, which never carries that value. The cozystack module read its image from a Deployment the installer stopped shipping when it moved to the operator, putting kubectl's NotFound in the first file of the report. And the cozystack apps section gated all three of its kinds on a CRD in an aggregated API group, so it never ran once; two of those names were wrong, and both are collected again. The tests avoid EXIT-trap cleanup, and e2e-testing.md says why: a bats test that installs one and then fails prints no TAP line at all, so output grepped for `not ok` reads as green, while hack/cozytest.sh reports the same failure correctly. A guard enforces that over a named list of files rather than a glob with exceptions, because a list of exceptions is the same claim written so that forgetting to shorten it makes it false. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
64 lines
2.8 KiB
Bash
64 lines
2.8 KiB
Bash
#!/usr/bin/env bats
|
|
# Regression coverage for host-node Talos diagnostics in hack/cozyreport.sh.
|
|
# The full report needs a cluster; this test sources only the focused helper and
|
|
# replaces talosctl with a shell function that records the exact argument vector.
|
|
|
|
HACK_DIR="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")" && pwd)"
|
|
SCRIPT="$HACK_DIR/cozyreport.sh"
|
|
|
|
COZYREPORT_LIB=1
|
|
# shellcheck source=/dev/null
|
|
. "$SCRIPT"
|
|
|
|
# cozytest.sh's parser ends an @test block at the first bare `}`, so keep the
|
|
# talosctl mock at top level rather than nesting a function inside the test.
|
|
talosctl() {
|
|
printf '%s\n' "$*" >> "$calls"
|
|
[ -z "${TALOSCTL_FAIL:-}" ] && return 0
|
|
echo 'error: rpc error: code = Unavailable desc = connection refused' >&2
|
|
return 1
|
|
}
|
|
|
|
@test "Talos host diagnostics pass an endpoint and use valid dmesg flags" {
|
|
# Cleanup at the end of the body, never a `trap ... EXIT`. An EXIT trap inside
|
|
# an @test replaces the one the bats binary installs for its own bookkeeping, and
|
|
# a test that then fails prints no TAP line at all -- it does not appear as
|
|
# `not ok`, it disappears, so a reader grepping the output sees a green suite.
|
|
# See docs/agents/e2e-testing.md. Both runners set `set -e`, so on a failure the
|
|
# cleanup line is never reached and the directory survives for inspection, which
|
|
# is what you want from a test that just failed.
|
|
tmp=$(mktemp -d)
|
|
calls="$tmp/talosctl.calls"
|
|
|
|
cozyreport_collect_talos_node /workspace/talosconfig 192.0.2.11 "$tmp"
|
|
|
|
[ "$(sed -n '1p' "$calls")" = "--talosconfig /workspace/talosconfig -e 192.0.2.11 -n 192.0.2.11 dmesg" ]
|
|
[ "$(sed -n '2p' "$calls")" = "--talosconfig /workspace/talosconfig -e 192.0.2.11 -n 192.0.2.11 logs kubelet --tail=500" ]
|
|
[ "$(sed -n '3p' "$calls")" = "--talosconfig /workspace/talosconfig -e 192.0.2.11 -n 192.0.2.11 logs containerd --tail=500" ]
|
|
case "$(sed -n '1p' "$calls")" in *--tail*)
|
|
echo "dmesg must not receive the boolean --tail flag" >&2
|
|
false
|
|
esac
|
|
|
|
rm -rf "$tmp"
|
|
}
|
|
|
|
@test "a Talos node that refuses the connection leaves the reason in its files" {
|
|
tmp=$(mktemp -d)
|
|
calls="$tmp/talosctl.calls"
|
|
|
|
TALOSCTL_FAIL=1 cozyreport_collect_talos_node /workspace/talosconfig 192.0.2.11 "$tmp"
|
|
|
|
# The reads are best-effort and must not abort the report, but "did not abort"
|
|
# and "produced nothing" are different outcomes and only one of them is about
|
|
# the node. Each read merges stderr into its own file, so a node that refused
|
|
# the connection says so where a reader looking for its dmesg will find it,
|
|
# rather than leaving three empty files that read like a quiet node.
|
|
for f in dmesg.txt kubelet.log containerd.log; do
|
|
t="$tmp/talos-192.0.2.11-$f"
|
|
[ -s "$t" ] || { echo "FAIL: $t is empty on a refused connection"; false; }
|
|
grep -q 'connection refused' "$t" || { echo "FAIL: $t does not carry the reason"; cat "$t"; false; }
|
|
done
|
|
rm -rf "$tmp"
|
|
}
|