mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-09 14:02:56 +09:00
Aleksei Sviridkin
61edf33ef4
A tenant worker that misses the node-Ready deadline while its kubelet image pull crawls leaves two mechanisms behind one symptom, and the artifact could not tell them apart. Either the link is slow, or the pull keeps restarting and discarding what it fetched; from inside the guest both are a progress line that barely moves. Per-worker network counters, read from the kubelet's cAdvisor endpoint the CPU throttling capture already uses, count every byte that crossed into the Pod including the ones a restart threw away, so the gap between that and the progress the guest reported separates the two. Which row carries it is not obvious and getting it wrong inverts the reading, so the collector states it. cAdvisor publishes one row per interface in the Pod netns, and a bridge-bound worker has four: the one still named eth0 is a dummy holding the Pod IP, carries no traffic, and reports zero -- the row a reader reaches for first, where zero reads as a worker that received nothing. The renamed NIC's receive is the number that answers; the bridge counts the same bytes again and the tap is named from the host's end. No interface predicate is applied, so all four reach the artifact and can be told apart. Two further properties of the family fail by capturing nothing rather than loudly: the dropped-packet series are spelled packets_dropped, and every row carries an empty container label because cAdvisor publishes them on the Pod sandbox, so a filter borrowed from the CPU counters beside it matches nothing at all. Both are stated where the filter is. The fixture's rows are real kubelet output relabelled; its interface names are read from the KubeVirt version this repo ships rather than sampled, since no virt-launcher Pod was available. The guest side gains the service state machine and the link table. Talos creates a service's log buffer when its runner opens the log writer, as it starts the process, so a kubelet still waiting on its volumes, on time sync, or on the container runtime reporting healthy never reached its runner, has no log at all, and the capture recorded an rpc error naming a log that was never registered: true about the log, silent about the service, while the state and the awaited condition were the finding. The link table is read as yaml because MTU lives in the spec rather than among the printed columns, and it is the value that decides whether a path-MTU reading of a stalled transfer survives. Both carry a tighter bound than the two reads beside them, being single tabular RPCs rather than streams, and the walk they sit in is the term the phase budget sizes itself against. A worker whose reads all failed now says so once for the worker. Each log already carried its own error and exit code, which is accurate per file and adds up to a directory indistinguishable from a worker that had little to report. The phase budget drops from 8m to 7m to pay for the two added guest reads, because the inequality keeping the tenant snapshot reachable had ten seconds of slack. A shorter budget declines more, and what it declines first is what is gated last: the mirror capture, whose subject is partly recoverable from its own request counts and the host snapshot, and the image-cache re-probe, which has no substitute and was already the first to go. The second cost lands on the guest captures, which are gated after the new counters and reach their gate with both the shortened budget and a listing plus a read per node already spent -- up to 100s at the read bound and the three-node cap, a ceiling those numbers impose rather than a duration measured on a live cluster. Sharing that read would mean the two captures becoming one walk, which is left out for scope rather than merit: the phase gate is a deadline rather than a per-collector allowance, so one walk admitted at the first of the two gates would cost less and collect more. The cost is stated rather than netted away, and the source says which way the argument runs. Both captures run one shared body, which also settles what the two copies had already broken once: the audit that catches an unbounded read pins each collector by something only it produces, and two collectors issuing byte-identical reads left neither with anything of its own, so stubbing one out of the audit went unnoticed. It is pinned by the report directory each writes. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>