docs(e2e): make the op-budget comment name what it guarantees

The 50m ceiling on the kubernetes-* script ops is justified by a comment
enumerating what the failure path can spend, and the enumeration was
both incomplete and reassuring in the wrong direction. It listed some
collectors, said nothing about the node-join diagnostics or the
image-cache diagnosis, carried a figure for the guest Talos capture that
was half its real cost, and closed by asking the reader to keep the op
above the sum, which no ceiling can do because the sum runs past what is
left of the op on its own.

It now names the phase budget as the thing that keeps the snapshot
reachable and lists the collectors this ceiling covers, in the order the
phase spends it. It states no sums and no per-collector figures. Why the
budget is that number is derived once, beside the budget, and pointed at
from here: a figure repeated across three files is a figure that drifts,
and the drift is what the reader trusts.

It also says what the ceiling does not promise. The budget bounds when a
collector may start rather than when the phase ends; several collectors
on this path carry no wall-clock bound at all; and the bringup's own
waits already exceed this op on ceilings before a single diagnostic
runs. So no worst case is claimed, because none can be stated while that
holds.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
This commit is contained in:
Aleksei Sviridkin committed 2026-08-09 02:51:06 +03:00
1 parent 0d7beaf630
commit 9278702018
2 files changed
+50 -24

No files matched your search

@@ -19,18 +19,31 @@ spec:
try:
- script:
# ~25m Kamaji bringup + worker join + LB/NFS/ouroboros assertions.
# The failure path can spend ~5m10s in bounded guest Talos capture,
# 35s per virt-launcher Pod reading the guest serial console, one 35s
# list read, one 35s read of the wedge check that runs before any
# other diagnostic, and when any console read comes back silent two
# more 35s reads for the Pod state that says whether the container
# started (30s each plus the 5s kill grace, so ~3m30s at the pool's
# minReplicas of 2 once a silent read pulls in the Pod-state pair,
# and ~5m50s at worst -- that walk is capped at 6
# Pods and the Pod-state reads are taken once, so neither follows
# maxReplicas), and another 6m30s in its in-process tenant snapshot.
# Keep the outer op above those inner budgets with useful
# scheduling/API-call slack.
# The on-failure diagnostics phase carries its own 8m wall-clock budget
# (COZY_DIAG_PHASE_BUDGET in _lib/run-kubernetes.sh): once it is spent, the
# collectors that have not started are declined out loud rather than
# attempted, so the in-process tenant crust-gather snapshot, the one the
# failing exit triggers and the largest artifact this path produces, still
# has its time. Why that budget and not another is derived beside it, from
# the op ceiling here minus the bringup and the snapshot; it is not repeated
# here, because a number kept in two places is a number that drifts.
#
# What this ceiling covers, in the order the phase spends it, cheapest first
# so that a budget running out drops the collectors that cost minutes rather
# than the reads that discriminate between failure modes:
# node-join diagnostics reads
# serial-console family (~5m50s at worst, ~3m30s at the pool's minimum)
# guest Talos capture
# talos-image-cache diagnosis
# and then, after the phase rather than inside it, the in-process tenant
# crust-gather snapshot.
#
# Those do not fit what is left of this op once the bringup ahead of them and
# the snapshot behind them are counted, which is why the phase is bounded as a
# phase rather than this ceiling raised. None of it is a worst case and none is
# claimed: several of those collectors carry no wall-clock bound at all, and
# the bringup's own waits already exceed this op on ceilings before a single
# diagnostic runs. Those residuals are tracked in cozystack/cozystack#3666.
timeout: 50m
# Co-locate the tenant crust-gather snapshot (captured by
# run-kubernetes.sh on failure) under the same snapshots/<test>/ dir as
@@ -13,18 +13,31 @@ spec:
- name: tenant-kubernetes-previous-version
try:
- script:
# The failure path can spend ~5m10s in bounded guest Talos capture,
# 35s per virt-launcher Pod reading the guest serial console, one 35s
# list read, one 35s read of the wedge check that runs before any
# other diagnostic, and when any console read comes back silent two
# more 35s reads for the Pod state that says whether the container
# started (30s each plus the 5s kill grace, so ~3m30s at the pool's
# minReplicas of 2 once a silent read pulls in the Pod-state pair,
# and ~5m50s at worst -- that walk is capped at 6
# Pods and the Pod-state reads are taken once, so neither follows
# maxReplicas), and another 6m30s in its in-process tenant snapshot.
# Keep the outer op above those inner budgets with useful
# scheduling/API-call slack.
# The on-failure diagnostics phase carries its own 8m wall-clock budget
# (COZY_DIAG_PHASE_BUDGET in _lib/run-kubernetes.sh): once it is spent, the
# collectors that have not started are declined out loud rather than
# attempted, so the in-process tenant crust-gather snapshot, the one the
# failing exit triggers and the largest artifact this path produces, still
# has its time. Why that budget and not another is derived beside it, from
# the op ceiling here minus the bringup and the snapshot; it is not repeated
# here, because a number kept in two places is a number that drifts.
#
# What this ceiling covers, in the order the phase spends it, cheapest first
# so that a budget running out drops the collectors that cost minutes rather
# than the reads that discriminate between failure modes:
# node-join diagnostics reads
# serial-console family (~5m50s at worst, ~3m30s at the pool's minimum)
# guest Talos capture
# talos-image-cache diagnosis
# and then, after the phase rather than inside it, the in-process tenant
# crust-gather snapshot.
#
# Those do not fit what is left of this op once the bringup ahead of them and
# the snapshot behind them are counted, which is why the phase is bounded as a
# phase rather than this ceiling raised. None of it is a worst case and none is
# claimed: several of those collectors carry no wall-clock bound at all, and
# the bringup's own waits already exceed this op on ceilings before a single
# diagnostic runs. Those residuals are tracked in cozystack/cozystack#3666.
timeout: 50m
# Co-locate the tenant crust-gather snapshot (captured by
# run-kubernetes.sh on failure) under the same snapshots/<test>/ dir as