mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 20:23:39 +09:00
03e2da580549fbefcecb677caaf1f4dc36e5b838
15
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
18be4ee272 |
test(e2e): capture host-netns address attribution and a healthy-pod reference
The collector characterises the host->local-pod route for an affected pod through ovn0 alone, so when a pod IP turns out to be a local address of the host netns instead, nothing records which interface actually holds it. The host netns capture now also takes `ip -o addr show` across every interface and `ip route show table local`, so the owner can be named on the next occurrence rather than only ruled out on ovn0. The collector also inspects NotReady pods only, so a run has no same-window baseline for what a normal route and conntrack state look like on the same node. The per-pod capture becomes a reusable function and runs a second time against one Ready, Running, non-hostNetwork pod on that node, into the same file, so the comparison is in front of whoever opens it. hostNetwork is excluded because such a pod's podIP IS the node's own address and would show the same local-route fingerprint as the fault for an entirely ordinary reason. That baseline depends on the node and nothing else -- the candidates are the rows of the one snapshot the affected set was computed from, and a pod reporting Ready there is by definition not one this run is investigating -- so it is captured once per node and folded into each affected pod's file from a cache. Several affected pods on one node is what a wedged host->local-pod datapath looks like rather than an edge case, and capturing the same healthy pod once per affected pod would spend the caller's envelope re-asking a question already answered -- the cost the node-global dedup and the lookup memo beside it exist to avoid. This adds no read of its own. The cluster-wide pod list gains one column instead of being asked a second time, and that single read's status is what lets the artifact separate a node with no eligible pod from a list that never answered: one line for both would make an unchecked baseline read as an established absence. The two host-netns additions are best-effort execs, like the captures they sit beside. The new row filter carries the field count its jsonpath emits, so a list cut mid-record cannot hand over a fragment as the pod every reading of the affected one is compared against. The unit suite pins these on what they are rather than on what they leave behind. Both host-netns captures are asserted through the kubectl argv the collector issues, container included, because the section headers are separate echoes that outlive the exec being removed. The baseline is pinned from both sides: several affected pods on one node share a single capture, and an affected pod is served the healthy pod from its own node rather than from whichever node the walk reached first. The pod list's column order is asserted against the positions its readers index, because every runtime test answers that list from a stub written by hand and none of them can observe the jsonpath at all. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
9b11db1b03 |
fix(e2e): report an LB no probe reached as unprobed, not reachable
Five inputs walked into the LB probe gate's skip branch having run zero probes, and all five came out as "probe: LB reachable or not probeable ... -- reachable, skipped": no node to probe from, a Service with no port, a probe node the lookup says runs no kube-ovn-cni pod, a host netns with no nc/curl/wget, and an exec that could not run (no timeout on PATH, exec denied, the pod gone mid-probe). None of them observed the address. An e2e image without a probe client would stamp every LoadBalancer in the cluster reachable and skip the whole heavy capture, which reads as a healthy datapath. A sixth way in, the cni-server lookup not answering at all, already reported unknown and still does. Zero probes now routes to that same explicit unknown outcome, and each way of getting there carries a reason the run can actually support. The two structural inputs are named where they are detected. The rest are told apart by the shape of the outcome set, which is the only place the difference survives: the probe helper emits an explicit unknown token only when the lookup itself did not answer, so a set that is present and wholly unknown names the lookup, while an empty set names the three ways a probe dies after the lookup answered. Those three stay together -- the exec's status is swallowed by design, since this collector never fails a job -- but none of them is reported as a lookup that went unanswered, because it did not. The reachable line now says which evidence produced it. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
ed1c9e774b |
fix(e2e): say when an LB backend is unread rather than absent
The two EndpointSlice reads in the LB section discarded stderr and were interpreted by emptiness alone, so a refused or cut-off read wrote the same `<none>` into the artifact that a Service with no endpoints does. That is the assertion without the refutation: a reader holding the uploaded report, and not the run, has no way to tell which happened, and the section is at its weakest exactly when the apiserver is struggling. Both reads now carry their status beside the value, in the vocabulary the per-node captures already use. Reporting the status arms a landmine the silence used to hide: the target port was read with a jsonpath indexing `items[0]`, and client-go's evalArray has no allowMissingKeys escape, so an empty list is a hard error and kubectl exits 1. A Service with no endpointslices is an ordinary answer here and would have started reporting as a failed read, so the read now asks with `items[*]` -- empty output, exit 0 -- and takes the first value, the way the pod lookups already do. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
33f672f3fc |
fix(e2e): skip a tcpdump leg whose node was never identified
pod_on_node turns its node argument into `--field-selector spec.nodeName=<node>`, and an empty node asks for pods whose nodeName is empty -- the unscheduled ones. So an unidentified node did not yield "no pod found": it yielded the first pending kube-ovn-cni pod in the namespace, and the tcpdump then ran against a pod that is not on that node and may not be running at all. Both halves of the cross-node capture had this, and each gap opens in the very state its own leg exists to characterise. The announcer is unknown exactly when MetalLB is misbehaving; a Service with no ready endpoint is the ordinary shape of an LB outage, and the endpoint side carried two such lookups, the cni-server pod and the backend's OVS interface. In either state the artifact came out holding a tcpdump from an unrelated pending pod under an ANNOUNCER or ENDPOINT label, or a confusing exec failure -- both pointing away from the actual problem. Each leg is now guarded the way the static capture above it already was, and records in the artifact that it was skipped rather than leaving a silent gap. The guards stay at the call sites rather than moving into pod_on_node: that helper serves five callers, and an early return there would have to invent a status for a lookup nobody made. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
0cc49ad34b |
fix(e2e): drop truncated records in the data-plane capture filters
The record filters selected rows by field value and never checked the field count, so a list cut mid-record parsed as a valid record: an IP cut mid-octet still looks like an IP, and a cut inside the nodeName column still looks like a scheduled pod. The capture then probed an address the cluster never had and opened a per-node file for a node that does not exist -- confidently labelled data about objects that are artifacts of the cut. The kubectl jsonpath behind each filter emits a fixed field count per record, so a shorter row is a fragment by construction: seven for the Service list, six for the pod list, five for the endpoint list. Dropping it costs the reader nothing -- the read's own note already says the list did not finish. The endpoint filter is the one that matters most, because it stops at the first row it accepts: a fragment at its head is not one bad row among many, it is the backend, and its half-parsed node is the node every endpoint-side capture then aims at. Where the class lives in this file, so the map does not have to be rebuilt: four sites parse records, and the two loops walking pods and LoadBalancers are not among them -- they consume what these filters already returned, so a cut is caught upstream of them. Three of the four are the filters guarded here. The fourth, the MetalLB speaker loop, is at the limit of what a field count can do: its records carry two fields, a cut in the first leaves no separator and its existing emptiness guard already rejects that, and a cut in the second delivers every separator, which nothing in-band tells apart from a genuinely short node name. That limit is not confined to the speaker loop, and the comment on the first filter now says so rather than implying a guard that holds everywhere: a cut inside any record's LAST field passes a field count. It lands on a display-only column for Services, and on a decision column for the other two, where a truncated value matches neither state being excluded and the row is kept -- over-capturing rather than dropping evidence. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
f266705784 |
docs(e2e): name both callers of the data-plane capture in its header
The header credited cozytest.sh alone, while the Chainsaw global catch invokes the same script under a different wall-clock backstop -- 300s against cozytest.sh's 600s. A reader sizing this collector's budget against the caller the header names can still overrun the one it omits, and the arithmetic further down the file already assumes both exist. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
f92306681d |
fix(e2e): an unanswered read is not the same as an absent thing
Five places turned an empty or missing result into a fact about the cluster: no affected pods, no ovn-central, no LoadBalancer service, no cilium-agent or ovs on a node, and a LoadBalancer recorded as reachable. Each is also what the script produces when the read behind it never answered, and with a note now saying the read failed, lines sat next to each other and disagreed. Each claim has a counterpart chosen by the status of the read that produced it, and each says what is left unknown rather than what the read did: no scheduled NotReady pods / whether any pod is affected no ovn-central pod in <ns> / whether <ns> runs ovn-central no Service type=LoadBalancer / whether any LoadBalancer needs capturing (no cilium-agent pod found) / could not determine whether one runs there LB reachable, skipped / whether the LB is reachable is unknown The last of those is the subtlest, because nothing about it looks like a claim of absence. host_http_probe resolves a cni-server before probing, and a lookup that never answered produced no probe outcome at all, which the decision helper read as "nothing failed" and the artifact stamped as reachable -- a verdict about an address the script never touched. The probe now emits an explicit unknown, and a set of outcomes that is entirely unknown stays unknown rather than collapsing into either answer. Only a wholly unknown set: one unrun probe beside real failures leaves the failures standing and still captures, because they are the evidence the capture exists to characterise. Wording that describes the read instead would be wrong in a case these bounds create. A list can fail after emitting rows, since a bound firing mid-stream leaves output on stdout and 124 in $?, so an empty result with a non-zero status can mean the read said nothing or that it said only uninteresting things. Before the bounds a read either answered or hung forever, and a partial answer was not a state this script could reach. For the same reason a failure note is unconditional on the status while the consequence -- skipped, unresolved -- is stated only by the branch that takes it, which is why the note inside pod_on_node names none at all: five call sites share it and their consequences differ. Two of these five are in the artifact rather than the log, and they are there because of the bounds. Unbounded, the per-node lookup blocked until the caller's backstop killed the script and no capture file was written; bounded, the script survives and writes the file, so a read that never answered would have been recorded as a pod that is not there or an address that answered. pod_on_node returns its status with the name and the memo stores it, since a hit that dropped it would launder a cutoff into a confident absence. The status travels packed with the value through stdout because every caller reads the helper through a command substitution, where a variable dies with the subshell -- the same property that makes the memo itself a file. Only answers and cutoffs are memoised: an instant failure costs nothing to ask again, and caching it would make one transient permanent for the whole run. Every note also ships beside the capture it explains, not only in the job log, because the reader who has the uploaded report and not the run is exactly the reader who cannot otherwise tell a capture that found nothing from one that never ran. Both sibling collectors hold that contract and this script's notes are modelled on them. The two phrasing helpers move above the sourcing guard so the unit suite can assert their branches directly. The stderr sink stays below it, because anything created above is created again by every test that sources this file and by every host that exits early for want of kubectl. What is deliberately left is the same distinction for the two EndpointSlice reads, which still discard stderr and report neither outcome. Each branch is pinned by a stub that answers one read and fails another, each partial-answer case by a stub that emits a row and then fails, the artifact cases in the artifact rather than the log, and the memo's status round-trip by consuming one lookup from both a miss and a hit. Every one of them carries a positive anchor as well, so a stub that stops short of the branch fails rather than passing quietly. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
b921952fb0 |
perf(e2e): ask each per-node pod lookup once
The same (namespace, label, node) is asked up to three times per affected pod across the sections of this script. That was free while the first call hung forever and the script died inside it; with a per-read bound each repeat costs its own timeout, and against a wedged apiserver the repetition becomes the largest consumer of the caller's envelope -- enough that a 300s budget buys two pods where it could buy five. Answers are memoised, empty ones included: "there is no cilium-agent on this node" is exactly the lookup worth not repeating. The memo is a file rather than a variable, which is not a style choice. The walk over affected pods runs on the right-hand side of a pipeline, so it executes in a subshell and a variable memo is discarded at the end of it, leaving every lookup to be paid for again in the sections that follow. With the variable form the suite still recorded each of the three lookups twice. That is also why this is not the shape of the node memo above it, which is a space-delimited string matched with a case glob and is correct for a dedup living entirely inside the walk. Keys are compared as whole tab-separated fields, so the '=' inside a label selector is ordinary payload; what the scheme rests on is that no namespace, label selector or node name contains a tab, which Kubernetes names and selectors cannot. An empty pod list is also separated from one that never answered, so the branch below it stops reporting a failed read as a cluster with nothing wrong. Pinned by a stub that records every per-node lookup it is given: the test fails if any is issued twice. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
7d1137a889 |
fix(e2e): bound the dataplane capture's reads and name why one failed
Six `kubectl get` calls ran with no bound of their own, while all 27
`kubectl exec` calls in the same script went through one. Against an
apiserver that hangs rather than refuses, the first of them -- the
cluster-wide pod list -- held the script until the caller's backstop
killed it, so the capture wrote nothing at all. From the Chainsaw catch
that is worse than empty: the run is cut off by the outer `timeout -k`
and the script says nothing about having been cut short, so a truncated
capture reads exactly like a complete one that found no affected pods.
Measured against a kubectl stub that never returns: before, the script
was still blocked on its first call when a 60s backstop killed it;
after, it reaches its own end and says which read was cut off and what
that leaves missing. Roughly 4s at a 1s bound, 40s at the 20s default.
A per-read bound buys forward progress, not a smaller total. The sum of
the bounds here is far past any caller's envelope and deliberately so,
because the MAX_PODS and MAX_LBS caps bound work while the outer
`timeout -k` bounds wall clock. What changes is that no single hung read
can consume that envelope before anything is written.
The reporting around them is built to survive being read by someone who
was not there. Only 124 and 137 may name a timeout, since those are the
only statuses a bound produces and every other one is kubectl answering
-- refused, denied, a kind the cluster does not serve -- so the reads
keep their stderr instead of discarding it and quote it back. 137 is
described as what it is, a SIGKILL that cannot be told apart from the
bound's own kill grace. Notes print through `printf` rather than `echo`,
because under /bin/sh echo expands backslash escapes and kubectl's
stderr now flows through the logger: a message holding a literal \n
would otherwise split the note and forge a second "[capture-dataplane]"
line that reads as this script's own verdict, which needs no malice
given the jsonpath in these reads contains {"\n"} and kubectl quotes the
expression back when it fails to parse it.
The per-node lookups ask with `{.items[*].metadata.name}` rather than
`{.items[0]...}`. client-go's evalArray has no allowMissingKeys escape
the way evalField does, so indexing into an empty list is a hard error
and kubectl exits 1 -- which meant nothing while the status was
discarded, and would report "there is no cilium-agent on this node", the
ordinary answer, as a read that failed.
The tests drive the script as a subprocess against a stub kubectl that
hangs, fails instantly, or returns a forged line, and read what it
wrote. The stub that proves forward progress answers the pod list and
hangs afterwards, so the walk over affected pods is actually entered.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
|
ad1adbf785 |
test(e2e): fix diagnostic collector blind spots
Assisted-By: GPT-5 <noreply@openai.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
07c67a8913 |
test(e2e): capture previous container logs and stop killing the snapshot
Root-causing a crash-loop failure was blocked twice over by the diagnostics collected on failure. `kubectl logs` shows only the current container instance, so for a crash-looping pod the suites' `podLogs` collector captured the Nth restart -- a fresh replay of the same startup path -- while the evidence that explained the failure sat in the first, interrupted instance. Chainsaw's podLogs collector has no `previous` option, so add e2e-capture-previous-logs.sh and drive it from the global catch and from the BATS failure hook. It discovers containers with restartCount > 0 at runtime (init containers included, where an interrupted bootstrap lives), orders the app suites' namespace first, and dumps each to the job log and the snapshot tree. Containers that never restarted are skipped, so a healthy run emits one line rather than a page of "previous terminated container not found". Every way that capture can come up short is named rather than inferred, because kubectl exits 1 for a garbage-collected container, an RBAC denial and an unreachable node alike: the container cap, a pod list that never returned (which is not the same as nothing having restarted), a read that timed out versus one kubectl refused and the reason it gave, and a previous instance that produced no output. A partial capture is kept and marked truncated rather than deleted -- it is the only copy of a log already gone from the cluster. The crust-gather snapshot was killed by the Chainsaw op containing it. A script op is bounded by timeouts.exec (5m), and the catch's own wall-clock wrappers summed to 960s, so the script was SIGKILLed partway through -- taking out the process group mid-write and skipping the data-plane capture entirely. Give each catch op an explicit timeout above the budgets it contains, so an inner timeout fires instead and leaves a partial, still-servable snapshot behind. The data-plane leg drops from 600s to 300s at that call site so both legs fit one envelope; the BATS hook keeps the wider budget, having no containing op. Set crust-gather --duration explicitly as well. It is the collection phase's own budget, it defaults to 60s, and on elapse it abandons the collection and skips its finish step, leaving a truncated tree with no indication anything is missing. The outer wall-clock has to exceed it because the API discovery that runs first is not covered by that budget at all -- with an unhealthy aggregated APIService, the state a failed run is usually in, discovery alone can consume the whole timeout before a single object is written. Send each collector's output to a log beside its snapshot instead of /dev/null, with a complete/truncated/failed verdict, so that question is answerable from the artifact. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
3ece61ba04 |
test(e2e): make the LoadBalancer announcer detection failover-robust
The dataplane diagnostic's announcer-node detection reported the wrong
node in two failover situations and could match the wrong IP:
- A node that announced an IP and then withdrew it (with no
re-announce) was still reported as the announcer, because a withdraw
did not retract a prior announce.
- Speaker logs are concatenated per-pod, not globally time-sorted, so
a stale former owner's announce could outrank the true current
owner purely by its position in the input.
- The IP was matched as a substring, so a query for 192.0.2.5 also
matched a 192.0.2.50 line.
Reimplement the detection to report a node only when that node's LAST
own announce/withdraw event for the exact IP is an announce, which is
robust to both concat order and L2 failover, and anchor the IP match to
a whole IP-literal token instead of a substring.
Also fix the heavy-capture budget: the MAX_LBS cap counted every
enumerated LB, so a broken LB enumerated after a cap's worth of
reachable LBs was never captured. Count only LBs actually captured, via
a small pure budget helper, so reachable/skipped LBs do not consume the
budget. Document that this cap bounds work, not wall-clock -- at two or
more unreachable LBs the real wall-clock bound is the outer backstop at
the call site.
Add bats coverage pinning the fixed announcer behavior (withdraw,
concat-order, exact-IP) and the captured-budget accounting.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
|
631cdaa069 |
test(e2e): capture the LoadBalancer datapath when an LB IP is unreachable
The on-failure dataplane diagnostic only captured host->pod CNI state for NotReady pods. A recurrent flake presents differently: a tenant Service type=LoadBalancer IP goes unreachable while its backend pod/VMI stays Ready and Running, so there is no NotReady pod and the hook captured nothing -- leaving the host->cross-node->bridged-worker-VMI->nodePort path uncharacterised, which is exactly where root-causing has stalled. Add a gated LoadBalancer-datapath section. It enumerates LoadBalancer services with an ingress IP, derives the EndpointSlice backend, the nodePort/externalTrafficPolicy, and the announcer node from the hostNetwork MetalLB speaker logs, then probes the LB IP from a host netns. Only an LB whose every probe fails gets the heavy capture, and that probe doubles as the traffic for the tcpdumps. For a failing LB it captures, on both the announcer and endpoint nodes, the host cilium LB map, kernel conntrack/neigh, br-int flows, ovn-controller timing, and a live tcpdump on the geneve tunnel and the backend tap -- localising the failing hop to host-cilium programming vs kube-ovn delivery. In-guest VMI capture is documented as a deferred stretch that needs tenant-cluster access. The existing NotReady-pod capture is unchanged. The new section is best-effort and time-boxed like the rest of the script, and a unit test pins the pure enumeration, announcer-node, and capture-or-skip logic against mock input. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
594c887916 |
test(e2e): capture OVN flow-programming datums in dataplane diagnostic
Extend the host->pod data-plane capture with the three datums that make an OVN incremental-processing lag dispositive on the next recurrence, all diagnostic-only and gated on the existing on-failure path: - ovn-controller.log from the ovs-ovn pod (grep for physical_flow_output, if_status_mgr, took <N>ms, recompute, long poll interval; plus a bounded tail) -- shows whether ovn-installed was set before the LSP local-delivery flow was programmed. - per-interface ovn-installed flag + ovn-installed-ts on the pod's OVS interface, looked up by iface-id <pod>.<ns>. - ovs-ovn cgroup cpu.stat (nr_throttled / throttled_usec, cgroup v2 with v1 fallback) -- shows whether ovn-controller was CPU-starved. Node-global captures land in node-<node>.txt, the per-interface flag in the per-pod file. Every command stays timeboxed and || true, no behavior or exit-code change. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
079e4079d3 |
test(e2e): capture host->pod data-plane on failure
The E2E install intermittently fails on a CNI host->local-pod data-plane transient: kubelet on a node reaches a local pod's readiness/startup probe with "connection refused" for several minutes while overlay pod->pod traffic works, then it self-heals. This is rooted in the cilium+kube-ovn chaining config (forced enable-host-legacy-routing, CNI InstallEndpointRoute:false), which delegates host->local-pod routing to kube-ovn/ovn0. The crust-gather failure snapshot records Kubernetes object state but not the node's L3 forwarding state, so the mechanism cannot be root-caused after the fact. On test failure, for each NotReady pod that is scheduled, has a podIP, and is not terminal (Succeeded/Failed excluded), collect the host->pod data-plane state from the pod's node into the crust-gather snapshot dir (so it ships in the uploaded report): - host netns via the kube-ovn cni-server (hostNetwork): ip route get <podIP>, ip neigh, ip rule, ip addr show ovn0, and the KERNEL conntrack entries for the podIP (conntrack -L, falling back to /proc/net/nf_conntrack). Under enable-host-legacy-routing the host->local-pod path traverses the kernel netfilter stack, not cilium BPF, so this is the authoritative conntrack table and route/rule state for the transient; - cilium-agent on the node: endpoint list, bpf CT entries for the podIP, a bounded monitor --type drop, and hubble dropped verdicts when present (complementary: confirms the path is not handled in BPF); - OVS/OVN on the node: ovs-ofctl dump-flows br-int, the OVN Port_Binding / Logical_Switch_Port for the pod, and a bounded ovn-sbctl lflow-list. Diagnostic-only: it runs after the existing crust-gather collect and only on an already-failed test, never mutates the cluster, and cannot change the pass/fail outcome. Every live capture is time-boxed, every command is || true, the whole run has a wall-clock backstop, node-global captures are deduped per node, and the pod count is capped. It no-ops cleanly when no pods are affected or when a tool/pod is absent. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |