admin/cozystack
1
0 Fork 0
mirror of https://github.com/cozystack/cozystack.git synced 2026-11-08 21:14:56 +09:00
Code Issues Packages Projects Releases Wiki Activity

test(e2e): capture host-netns address attribution and a healthy-pod reference for the dataplane collector (#3617)

Browse Source 
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->

## What this PR does

Extends `hack/e2e-capture-dataplane.sh`, the diagnostic collector that
runs after an e2e test failure, with the instrumentation issue #3615
asks for to close its address-attribution gap.

Issue #3615 found that a NotReady pod's IP can turn out to be a local
address of the node's host netns instead of reachable through the ovn0
route, which makes kubelet's probes connect to the node's own loopback
and fail forever. The collector already captured `ip addr show ovn0` for
the affected pod's node, but nothing that could name which interface
actually holds a stolen address, since it only ever looked at ovn0.

This adds `ip -o addr show` (every interface) and `ip route show table
local` to the same host netns capture, so the interface actually holding
a local address can be identified on the next occurrence.

The collector also only ever inspected NotReady pods, so a run had no
same-window baseline for what a normal route and conntrack state look
like on the same node. This extracts the existing per-pod capture into a
reusable function and runs it a second time against one Ready, Running
pod on the same node as an in-run healthy-pod control, picked while
explicitly excluding hostNetwork pods (their podIP is the node's own
address, so one would show the exact same local-route fingerprint as the
bug for an unrelated reason). When no eligible Ready pod exists on the
node, that absence is recorded in the artifact instead of left silent.

This is diagnostic tooling only: it changes what gets captured after a
failure, not any test's pass/fail outcome.

### Known limitations

The baseline runs at a reduced scope: the host route and the kernel
conntrack for the healthy pod's address, and nothing else. What it drops
-- cilium CT, the OVN port binding, the OVS `ovn-installed` flag, and
the pod's own Ready conditions and events -- describes how a pod was
programmed, and a pod picked for being Ready and Running is programmed
correctly by definition. That scope is what holds the pod path's worst
case at 579s against the 600s the `cozytest.sh` trap allows; the
remaining 21 seconds are what this path leaves for the LoadBalancer
section that runs after it, not headroom the run has. The other caller's
envelope is 300s, and the worst case does not fit it -- nor did it at
the merge base, where the same walk bounds at 519s. That overrun is
tracked in #3671, and the term-by-term arithmetic is in the `MAX_PODS`
note in the script.

The 579s above assumes the lookup memo exists. When `mktemp` fails the
memo is empty, every per-node pod lookup runs live again, and the same
single-pod case bounds at 667s -- past the 600s envelope as well as the
300s one. At the merge base that memo-less case bounds at 585s, inside
600s, so this change is what moves it out. The degradation is visible
rather than silent: the caller's backstop truncates the capture and the
run reports `data-plane capture INCOMPLETE (exit 124)`.

A baseline can be present and still carry nothing. Every capture at that
scope goes through the node's `kube-ovn-cni` pod, so when that lookup
does not resolve, the reference section is its banner and the scope note
with nothing beneath them -- and, unlike `node-<node>.txt` next to it,
no line saying why. Two different standards of honesty inside one
artifact. The affected-pod path has the same shape today, so this
extends an existing gap rather than opening one, and it is left alone
here.

The per-node baseline is cached under `TMPDIR` for the run. A SIGKILL --
which is what a caller's wall-clock backstop delivers -- skips the
cleanup at the end of the script and leaves that directory behind. The
lookup memo this script already used has the same exposure.

Nodes are visited in the order their affected pods appear in the
cluster-wide list, which the API chooses. Once `MAX_PODS` cuts the walk,
which pods and therefore which nodes got captured is a function of that
ordering and not of anything a reader would pick.


### Screenshots

Not applicable, no UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
test(e2e): capture host netns address attribution and an in-run healthy-pod reference for the dataplane failure collector
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
  * Added diagnostics for node addresses and local routing information.
* Added healthy, same-node reference captures for dataplane
troubleshooting, reused across affected pods.
* Expanded pod diagnostics to include status, events, connection
tracking, routes, network bindings, and interface state.

* **Tests**
* Added coverage for pod selection, retries, caching, exclusions,
incomplete records, unavailable responses, and missing baselines.
  * Recorded diagnostic failure reasons in artifacts.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
Aleksei Sviridkin authored and GitHub committed 2026-08-09 15:39:36 +03:00
2 parents dda339f351 18be4ee272
commit 02a3fda471
2 files changed
+936 -126
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

hack/capture-dataplane.bats
+555 -19
View File
@@ -15,6 +15,8 @@
# succeeded, unknown when none ran or none could.
# - pod_filter_affected -- retain scheduled NotReady pods even without IPs,
# and only from whole rows.
# - pod_first_ready -- pick the healthy-pod baseline for one node out
# of the cluster-wide snapshot, whole rows only.
#
# The tests come in two shapes, and neither needs a cluster.
#
@@ -57,25 +59,150 @@ E2E_CAPTURE_DATAPLANE_LIB=1
@test "pod_filter_affected keeps a scheduled NotReady pod without a podIP" {
rows="$(printf '%s\n' \
'tenant|no-ip||node-a|False|Pending' \
'tenant|with-ip|192.0.2.80|node-b|False|Running')"
'tenant|no-ip||node-a|False|Pending||eol' \
'tenant|with-ip|192.0.2.80|node-b|False|Running||eol')"
out="$(printf '%s\n' "$rows" | pod_filter_affected)"
[ "$(printf '%s\n' "$out" | awk 'END { print NR }')" -eq 2 ]
printf '%s\n' "$out" | awk '$0 == "tenant|no-ip||node-a|False|Pending" { found = 1 } END { exit !found }'
printf '%s\n' "$out" | awk '$0 == "tenant|no-ip||node-a|False|Pending||eol" { found = 1 } END { exit !found }'
}
@test "pod_filter_affected drops Ready unscheduled and terminal pods" {
rows="$(printf '%s\n' \
'tenant|ready|192.0.2.81|node-a|True|Running' \
'tenant|unscheduled|||False|Pending' \
'tenant|succeeded|192.0.2.82|node-b|False|Succeeded' \
'tenant|failed|192.0.2.83|node-b|False|Failed')"
'tenant|ready|192.0.2.81|node-a|True|Running||eol' \
'tenant|unscheduled|||False|Pending||eol' \
'tenant|succeeded|192.0.2.82|node-b|False|Succeeded||eol' \
'tenant|failed|192.0.2.83|node-b|False|Failed||eol')"
[ -z "$(printf '%s\n' "$rows" | pod_filter_affected)" ]
}
@test "pod_first_ready picks the first Ready Running pod on the named node" {
# Rows are the cluster-wide snapshot, so the filter scopes to a node itself:
# ns|name|podIP|nodeName|ready|phase|hostNetwork.
rows="$(printf '%s\n' \
'tenant|not-ready|192.0.2.90|node-a|False|Running||eol' \
'tenant|pending|192.0.2.91|node-a|True|Pending||eol' \
'tenant|other-node|192.0.2.94|node-b|True|Running||eol' \
'tenant|healthy-1|192.0.2.92|node-a|True|Running||eol' \
'tenant|healthy-2|192.0.2.93|node-a|True|Running||eol')"
out="$(printf '%s\n' "$rows" | pod_first_ready node-a)"
[ "$out" = "tenant|healthy-1|192.0.2.92" ]
}
@test "pod_first_ready ignores an eligible pod on a different node" {
# The scoping is the whole reason this reads the cluster-wide list safely: a
# baseline from another machine would compare the wedged pod's route against
# a route that never shared a kernel with it.
rows="$(printf '%s\n' 'tenant|healthy|192.0.2.95|node-b|True|Running||eol')"
[ -z "$(printf '%s\n' "$rows" | pod_first_ready node-a)" ]
}
@test "pod_first_ready emits nothing when no pod on the node is Ready and Running" {
rows="$(printf '%s\n' \
'tenant|not-ready|192.0.2.94|node-a|False|Running||eol' \
'tenant|pending|192.0.2.95|node-a|True|Pending||eol')"
[ -z "$(printf '%s\n' "$rows" | pod_first_ready node-a)" ]
}
@test "pod_first_ready emits nothing on empty input" {
[ -z "$(printf '' | pod_first_ready node-a)" ]
}
@test "pod_first_ready rejects a hostNetwork row cut at the separator" {
# The half a value check cannot reach on its own. A stream cut RIGHT AFTER the
# sixth separator leaves the hostNetwork column empty, and empty is
# byte-identical to the omitempty-false the filter must accept -- so `$7 ==
# ""` alone seats a hostNetwork pod whose `true` was cut away entirely. The
# trailing constant is what separates them, and the two halves of that guard
# catch different cuts, so both are exercised here:
# row 1 loses its tail before the last separator -> NF is 7, not 8
# row 2 keeps the count but the sentinel is cut -> NF is 8, $8 is not eol
# Neither may be picked; the whole row behind them must be.
rows="$(printf '%s\n' \
'cozy-cilium|cilium-abcde|192.0.2.10|node-a|True|Running|' \
'cozy-cilium|cilium-fghij|192.0.2.13|node-a|True|Running||eo' \
'tenant|workload|192.0.2.12|node-a|True|Running||eol')"
out="$(printf '%s\n' "$rows" | pod_first_ready node-a)"
[ "$out" = "tenant|workload|192.0.2.12" ]
}
@test "pod_first_ready rejects a hostNetwork row cut inside its last value" {
# The one filter here where a truncated last value must NOT be kept.
# `tru` carries all eight fields and passes every other gate, so a deny-list
# (`$7 != "true"`) accepts it -- and because this filter stops at the row it
# takes, the baseline then IS a hostNetwork pod, whose podIP is the node's own
# address. That is the fingerprint the exclusion exists to keep out of the
# comparison, so the control would be the thing it excludes. The allow-list
# form rejects any value that is not an exact known state.
rows="$(printf '%s\n' \
'cozy-cilium|cilium-abcde|192.0.2.10|node-a|True|Running|tru|eol' \
'tenant|workload|192.0.2.11|node-a|True|Running||eol')"
out="$(printf '%s\n' "$rows" | pod_first_ready node-a)"
[ "$out" = "tenant|workload|192.0.2.11" ]
}
@test "pod_first_ready skips a Ready pod that has no podIP yet" {
# The filter stops at the row it takes, so an addressless pod does not just
# produce a thin baseline -- it consumes the only pick and the pod behind it
# is never considered.
rows="$(printf '%s\n' \
'tenant|no-ip-yet||node-a|True|Running||eol' \
'tenant|healthy|192.0.2.99|node-a|True|Running||eol')"
out="$(printf '%s\n' "$rows" | pod_first_ready node-a)"
[ "$out" = "tenant|healthy|192.0.2.99" ]
}
@test "pod_first_ready drops a pod row cut off mid-record" {
# The cut has to land where the fragment still satisfies every value test, or
# the row is rejected for an unrelated reason and the count is never
# consulted: here the stream ends right after `Running`, one separator short
# of the hostNetwork column.
rows="$(printf '%s\n' \
'tenant|cut-off|192.0.2.9|node-a|True|Running' \
'tenant|healthy|192.0.2.98|node-a|True|Running||eol')"
out="$(printf '%s\n' "$rows" | pod_first_ready node-a)"
[ "$out" = "tenant|healthy|192.0.2.98" ]
}
@test "pod_first_ready skips a hostNetwork pod even when it is Ready Running and first" {
# cilium-agent / kube-ovn-cni / ovs-ovn run hostNetwork on every node, and a
# hostNetwork pod's podIP IS the node's own address -- picking one as the
# baseline would resolve to a local route for an entirely ordinary reason,
# which is the same fingerprint the address-attribution capture exists to
# recognise, so the baseline would defeat itself.
rows="$(printf '%s\n' \
'cozy-cilium|cilium-agent-abcde|192.0.2.10|node-a|True|Running|true|eol' \
'tenant|workload|192.0.2.96|node-a|True|Running||eol')"
out="$(printf '%s\n' "$rows" | pod_first_ready node-a)"
[ "$out" = "tenant|workload|192.0.2.96" ]
}
@test "pod_first_ready treats a blank hostNetwork column as NOT hostNetwork" {
# The API omits spec.hostNetwork (omitempty) when it is false/unset, so
# kubectl's jsonpath emits an empty string, not the literal "false".
rows="$(printf '%s\n' 'tenant|workload|192.0.2.97|node-a|True|Running||eol')"
out="$(printf '%s\n' "$rows" | pod_first_ready node-a)"
[ "$out" = "tenant|workload|192.0.2.97" ]
}
@test "runtime checks LoadBalancers when there are no affected pods" {
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
@@ -138,11 +265,13 @@ EOF
}
@test "pod_filter_affected drops a pod row cut off mid-record" {
# The same cut over the six-field pod jsonpath. The fragment ends inside the
# The same cut over the eight-field pod jsonpath. The fragment ends inside the
# nodeName column, so the half-parsed `nod` reads as a scheduled pod on a node
# of that name, and the capture opens a node-nod.txt for it.
# of that name, and the capture opens a node-nod.txt for it. Note the cut has
# to land in a column this filter reads: hostNetwork is last now, and a cut
# inside THAT one leaves every column this filter decides on intact.
rows="$(printf '%s\n' \
'tenant|wedged|192.0.2.80|node-b|False|Running' \
'tenant|wedged|192.0.2.80|node-b|False|Running||eol' \
'tenant|cut||nod')"
out="$(printf '%s\n' "$rows" | pod_filter_affected)"
@@ -381,7 +510,7 @@ dp_hanging_kubectl_dir() {
#!/bin/sh
for a in "$@"; do
case $a in
pods) echo 'tenant-test|wedged|10.0.0.1|node-a|False|Running'; exit 0 ;;
pods) echo 'tenant-test|wedged|10.0.0.1|node-a|False|Running||eol'; exit 0 ;;
esac
done
sleep 300
@@ -466,7 +595,7 @@ STUB
#!/bin/sh
for a in "$@"; do
case $a in
pods) echo 'tenant-test|wedged|10.0.0.1|node-a|False|Running'; exit 0 ;;
pods) echo 'tenant-test|wedged|10.0.0.1|node-a|False|Running||eol'; exit 0 ;;
esac
done
# The behaviour under test: client-go's evalArray has no allowMissingKeys
@@ -508,7 +637,7 @@ STUB
#!/bin/sh
for a in "$@"; do
case $a in
pods) echo 'tenant-test|wedged|10.0.0.1|node-a|False|Running'; exit 0 ;;
pods) echo 'tenant-test|wedged|10.0.0.1|node-a|False|Running||eol'; exit 0 ;;
esac
done
case "$*" in
@@ -556,7 +685,7 @@ STUB
#!/bin/sh
for a in "$@"; do
case $a in
pods) echo 'tenant-test|wedged|10.0.0.1|node-a|False|Running'; exit 0 ;;
pods) echo 'tenant-test|wedged|10.0.0.1|node-a|False|Running||eol'; exit 0 ;;
esac
done
case "$*" in
@@ -683,7 +812,7 @@ STUB
#!/bin/sh
for a in "$@"; do
case $a in
pods) echo 'tenant-test|wedged|10.0.0.1|node-a|False|Running'; exit 0 ;;
pods) echo 'tenant-test|wedged|10.0.0.1|node-a|False|Running||eol'; exit 0 ;;
esac
done
case "$*" in
@@ -717,7 +846,7 @@ STUB
#!/bin/sh
for a in "$@"; do
case $a in
pods) echo 'tenant-test|healthy|10.0.0.9|node-a|True|Running'; exit 1 ;;
pods) echo 'tenant-test|healthy|10.0.0.9|node-a|True|Running||eol'; exit 1 ;;
esac
done
exit 0
@@ -744,7 +873,7 @@ STUB
#!/bin/sh
for a in "$@"; do
case $a in
pods) echo 'tenant-test|wedged|10.0.0.1|node-a|False|Running'; exit 0 ;;
pods) echo 'tenant-test|wedged|10.0.0.1|node-a|False|Running||eol'; exit 0 ;;
esac
done
case "$*" in
@@ -765,6 +894,413 @@ STUB
rm -rf "$d"
}
@test "the host netns capture names every address, not only ovn0" {
# The reason this commit exists: when a podIP turns out to be a local address
# of the host netns, `ip addr show ovn0` cannot say which interface holds it.
# Nothing else in this suite reaches these two captures -- deleting both left
# every other test green -- so without this the collector could quietly lose
# the evidence the change was written to collect.
#
# Asserted on the kubectl argv, and the section headers are the secondary
# check rather than the primary one. A header is an `echo` the script emits
# before the exec, so dropping the exec and keeping the label leaves an empty
# section behind and every header grep green -- either capture alone, or both
# together, and the whole suite stays green. The argv also pins WHERE the
# capture runs -- `-c cni-server`, the container that shares the host netns --
# which is half the property and one a header cannot carry at all.
d=$(mktemp -d)
mkdir -p "$d/bin"
cat >"$d/bin/kubectl" <<'STUB'
#!/bin/sh
case "$*" in
*'ip -o addr show'*|*'ip route show table local'*) echo "$*" >>"$STUB_CALLS" ;;
esac
for a in "$@"; do
case $a in
pods) echo 'tenant-test|wedged|10.0.0.1|node-a|False|Running||eol'; exit 0 ;;
esac
done
case "$*" in
*'app=kube-ovn-cni'*) echo 'cni-abc'; exit 0 ;;
esac
exit 0
STUB
chmod +x "$d/bin/kubectl"
STUB_CALLS="$d/calls" PATH="$d/bin:$PATH" timeout 120 "$SCRIPT" "$d/out" >"$d/log" 2>&1 || true
[ -f "$d/out/node-node-a.txt" ]
# Positive control: the host-netns section ran, so a missing address capture
# below is that capture's absence and not the whole block being skipped.
grep -q 'host netns: ip rule' "$d/out/node-node-a.txt"
# The commands the collector actually issued into the host netns ...
grep -q -- '-c cni-server -- ip -o addr show' "$d/calls"
grep -q -- '-c cni-server -- ip route show table local' "$d/calls"
# ... and the labels a reader of the artifact finds them under.
grep -q 'ip -o addr show, all interfaces' "$d/out/node-node-a.txt"
grep -q 'ip route show table local' "$d/out/node-node-a.txt"
rm -rf "$d"
}
@test "a healthy pod on the node is captured beside the affected one" {
# The baseline is the point of the reference leg, and it only pays off in the
# SAME file as the affected pod: a reader comparing a suspicious route against
# a normal one should not have to go and find the normal one.
d=$(mktemp -d)
mkdir -p "$d/bin"
cat >"$d/bin/kubectl" <<'STUB'
#!/bin/sh
for a in "$@"; do
case $a in
pods)
echo 'tenant-test|wedged|10.0.0.1|node-a|False|Running||eol'
echo 'cozy-cilium|cilium-xyz|10.0.0.5|node-a|True|Running|true|eol'
echo 'tenant|healthy|10.0.0.9|node-a|True|Running||eol'
exit 0 ;;
esac
done
case "$*" in
# A cni-server on the node, so the IP-specific legs actually run. Without it
# they are skipped and the baseline would be a header with no comparison in
# it -- which is the whole thing this leg is for.
*'app=kube-ovn-cni'*) echo 'cni-abc'; exit 0 ;;
esac
exit 0
STUB
chmod +x "$d/bin/kubectl"
PATH="$d/bin:$PATH" timeout 90 "$SCRIPT" "$d/out" >"$d/log" 2>&1 || true
[ -f "$d/out/pod-tenant-test-wedged.txt" ]
grep -q 'POD tenant-test/wedged' "$d/out/pod-tenant-test-wedged.txt"
grep -q 'REFERENCE (Ready)' "$d/out/pod-tenant-test-wedged.txt"
grep -q 'POD tenant/healthy' "$d/out/pod-tenant-test-wedged.txt"
# And the baseline carries the evidence it exists to provide: the route and
# conntrack for the HEALTHY pod's address, not just a section header. A
# baseline without these compares nothing.
grep -q 'ip route get 10.0.0.9' "$d/out/pod-tenant-test-wedged.txt"
grep -q 'kernel conntrack for 10.0.0.9' "$d/out/pod-tenant-test-wedged.txt"
# The hostNetwork pod sorts first in the snapshot and must not have been
# taken: its podIP is the node's own address, the fingerprint the baseline
# exists to rule out.
if grep -q 'POD cozy-cilium/cilium-xyz' "$d/out/pod-tenant-test-wedged.txt"; then
echo "a hostNetwork pod was taken as the baseline:"
cat "$d/out/pod-tenant-test-wedged.txt"
exit 1
fi
rm -rf "$d"
}
@test "the baseline is captured at route-only scope, not the full one" {
# The baseline exists to show a normal route and conntrack beside a suspicious
# one. Cilium CT, the OVN port binding and the OVS ovn-installed flag describe
# how a pod was PROGRAMMED, and a pod chosen for being healthy is correctly
# programmed by definition -- they compare nothing and cost 105s of exec bound
# on the leg most likely to be cut. Dropping them is what keeps one affected
# pod on one node inside the 600s backstop, so the scope is load-bearing and
# not a preference.
d=$(mktemp -d)
mkdir -p "$d/bin"
cat >"$d/bin/kubectl" <<'STUB'
#!/bin/sh
for a in "$@"; do
case $a in
pods)
echo 'tenant-test|wedged|10.0.0.1|node-a|False|Running||eol'
echo 'tenant|healthy|10.0.0.9|node-a|True|Running||eol'
exit 0 ;;
esac
done
case "$*" in
# Everything the full scope needs is present, so its absence from the
# baseline is a decision and not a missing dependency.
*'k8s-app=cilium'*) echo 'cilium-xyz'; exit 0 ;;
*'app=ovs'*) echo 'ovs-xyz'; exit 0 ;;
*'app=kube-ovn-cni'*) echo 'cni-abc'; exit 0 ;;
*'app=ovn-central'*) echo 'ovn-central-0'; exit 0 ;;
esac
exit 0
STUB
chmod +x "$d/bin/kubectl"
PATH="$d/bin:$PATH" timeout 120 "$SCRIPT" "$d/out" >"$d/log" 2>&1 || true
f="$d/out/pod-tenant-test-wedged.txt"
[ -f "$f" ]
# Positive control: the affected pod gets the full scope, so the blocks exist
# and their absence below is scope and not a stub that answered nothing.
awk '/REFERENCE \(Ready\)/ { exit } { print }' "$f" | grep -q 'cilium-dbg bpf ct list global'
awk '/REFERENCE \(Ready\)/ { exit } { print }' "$f" | grep -q 'OVN Port_Binding'
# The baseline keeps what it is read for ...
awk '/REFERENCE \(Ready\)/,0' "$f" | grep -q 'ip route get 10.0.0.9'
awk '/REFERENCE \(Ready\)/,0' "$f" | grep -q 'kernel conntrack for 10.0.0.9'
# ... says in the artifact that the rest was a decision ...
awk '/REFERENCE \(Ready\)/,0' "$f" | grep -q 'scope: route and conntrack only'
awk '/REFERENCE \(Ready\)/,0' "$f" | grep -q 'absence below is a decision, not a lookup'
# ... and drops what compares nothing. Matched against SECTION HEADERS only:
# the marker line above names the very blocks it skips, so a bare substring
# search finds the explanation and calls it the block.
for section in 'cilium-dbg bpf ct list global' 'OVN Port_Binding' 'ovn-installed' 'Ready conditions'; do
if awk '/REFERENCE \(Ready\)/,0' "$f" | grep '^=== ' | grep -q "$section"; then
echo "the baseline was captured at full scope: section '$section' present after REFERENCE"
awk '/REFERENCE \(Ready\)/,0' "$f" | grep '^=== '
exit 1
fi
done
rm -rf "$d"
}
@test "the baseline column is read from the cluster-wide snapshot" {
# The seventh column is what lets the baseline be chosen without a second
# read. Drop it from the jsonpath and every row is a six-field fragment that
# the NF guards reject -- no affected pods, no baseline, nothing captured. The
# stub answers only a request that asks for hostNetwork, so a jsonpath that
# stopped asking produces an empty list and this test goes red.
d=$(mktemp -d)
mkdir -p "$d/bin"
cat >"$d/bin/kubectl" <<'STUB'
#!/bin/sh
case "$*" in
*'spec.hostNetwork'*)
echo 'tenant-test|wedged|10.0.0.1|node-a|False|Running||eol'
echo 'tenant|healthy|10.0.0.9|node-a|True|Running||eol'
exit 0 ;;
esac
for a in "$@"; do
case $a in
pods) exit 0 ;;
esac
done
exit 0
STUB
chmod +x "$d/bin/kubectl"
PATH="$d/bin:$PATH" timeout 90 "$SCRIPT" "$d/out" >"$d/log" 2>&1 || true
[ -f "$d/out/pod-tenant-test-wedged.txt" ]
grep -q 'POD tenant/healthy' "$d/out/pod-tenant-test-wedged.txt"
rm -rf "$d"
}
@test "two affected pods on one node share a single baseline capture" {
# The baseline depends on the node alone, and several affected pods on one
# node is what a wedged datapath looks like rather than an edge case. What
# must not repeat is the CAPTURE -- a second full per-pod capture of the same
# healthy pod, against a backstop the pod leg already strains. Counted by the
# baseline pod's own conditions read, which happens once per capture.
d=$(mktemp -d)
mkdir -p "$d/bin"
cat >"$d/bin/kubectl" <<'STUB'
#!/bin/sh
case "$*" in
# The baseline runs at route-only scope, so its conditions read is gone --
# count the route exec, which is the read that scope does make.
*'ip route get 10.0.0.9'*) echo ref-capture >>"$STUB_CALLS" ;;
esac
for a in "$@"; do
case $a in
pods)
echo 'tenant-test|wedged-a|10.0.0.1|node-a|False|Running||eol'
echo 'tenant-test|wedged-b|10.0.0.2|node-a|False|Running||eol'
echo 'tenant|healthy|10.0.0.9|node-a|True|Running||eol'
exit 0 ;;
esac
done
case "$*" in
*'app=kube-ovn-cni'*) echo 'cni-abc'; exit 0 ;;
esac
exit 0
STUB
chmod +x "$d/bin/kubectl"
STUB_CALLS="$d/calls" PATH="$d/bin:$PATH" timeout 120 "$SCRIPT" "$d/out" >"$d/log" 2>&1 || true
# Both pods carry the baseline: the saving must not cost one of them its copy.
grep -q 'POD tenant/healthy' "$d/out/pod-tenant-test-wedged-a.txt"
grep -q 'POD tenant/healthy' "$d/out/pod-tenant-test-wedged-b.txt"
n=$(grep -c ref-capture "$d/calls")
if [ "$n" -ne 1 ]; then
echo "the baseline was captured $n times for one node instead of once"
exit 1
fi
rm -rf "$d"
}
@test "the pod list jsonpath emits its columns in the order its readers index" {
# Every runtime test below answers the pod list from a stub that writes rows
# by hand, so none of them can observe the jsonpath's column ORDER: swap two
# fields in it and the suite stays green while the filters go on reading
# position 6 as phase and 7 as hostNetwork. Measured -- swapping
# `.status.phase` with `.spec.hostNetwork` passed 72 of 72.
#
# Against a cluster that swap makes pod_filter_affected compare a boolean
# against Succeeded/Failed and pod_first_ready compare a phase against
# "false", so the collector selects the wrong pods and says nothing. The
# positions are a contract between one jsonpath and three readers -- both awk
# filters and the `read` in the capture loop -- and no stub can hold it.
#
# Asserted as ORDER rather than as an exact string, so reformatting the
# jsonpath is free and reordering it is not.
#
# Each field carries its closing brace, and that is load-bearing rather than
# tidy: `.metadata.name` is a prefix of `.metadata.namespace`, so searching
# for it bare finds the namespace field and reports the two columns at one
# offset. The brace is what makes each search hit its own column.
line=$(grep -F '{range .items[*]}' "$SCRIPT" | grep -F 'spec.hostNetwork')
[ -n "$line" ]
prev=0
for f in '.metadata.namespace}' '.metadata.name}' '.status.podIP}' '.spec.nodeName}' \
'type=="Ready")].status}' '.status.phase}' '.spec.hostNetwork}' '|eol"}'; do
cur=$(printf '%s\n' "$line" | awk -v s="$f" '{ print index($0, s) }')
# index() returns 0 for absent, so this one comparison catches a field that
# moved and a field that vanished.
if [ "$cur" -le "$prev" ]; then
echo "the pod list jsonpath no longer emits its columns in the order its readers assume"
echo "field '$f' is at offset $cur, expected after $prev"
echo "line: $line"
exit 1
fi
prev=$cur
done
}
@test "an affected pod gets the baseline from its own node" {
# The complement of the test above, and the half a shared cache can break
# without breaking that one: the cache is keyed by node, and a key that
# stopped telling nodes apart would serve every node the first node's
# answer. The test above counts captures and a single shared answer is
# exactly one capture, so it would stay green.
#
# What the reader would then get is a route and a conntrack table read on
# one kernel, filed under a heading naming a different node, with nothing
# in the artifact saying so -- a capture reporting what it never observed,
# which is the failure this collector exists to rule out. So the assertion
# is on WHICH healthy pod each file carries, per file, not on how many
# captures ran.
d=$(mktemp -d)
mkdir -p "$d/bin"
cat >"$d/bin/kubectl" <<'STUB'
#!/bin/sh
for a in "$@"; do
case $a in
pods)
echo 'tenant-test|wedged-a|10.0.0.1|node-a|False|Running||eol'
echo 'tenant-test|wedged-b|10.0.0.2|node-b|False|Running||eol'
echo 'tenant|healthy-a|10.0.0.9|node-a|True|Running||eol'
echo 'tenant|healthy-b|10.0.0.8|node-b|True|Running||eol'
exit 0 ;;
esac
done
case "$*" in
*'app=kube-ovn-cni'*) echo 'cni-abc'; exit 0 ;;
esac
exit 0
STUB
chmod +x "$d/bin/kubectl"
PATH="$d/bin:$PATH" timeout 120 "$SCRIPT" "$d/out" >"$d/log" 2>&1 || true
fa="$d/out/pod-tenant-test-wedged-a.txt"
fb="$d/out/pod-tenant-test-wedged-b.txt"
[ -f "$fa" ]
[ -f "$fb" ]
# Each file names its own node's healthy pod ...
grep -q 'POD tenant/healthy-a' "$fa"
grep -q 'POD tenant/healthy-b' "$fb"
# ... and carries that pod's address in the evidence, not just in a heading.
grep -q 'ip route get 10.0.0.9' "$fa"
grep -q 'ip route get 10.0.0.8' "$fb"
# ... and neither carries the other node's, which is what a node-agnostic
# cache key produces: both files served whichever node was reached first.
if grep -q 'POD tenant/healthy-b' "$fa" || grep -q 'POD tenant/healthy-a' "$fb"; then
echo "a baseline crossed nodes:"
grep -H 'POD tenant/healthy' "$fa" "$fb"
exit 1
fi
rm -rf "$d"
}
@test "a partially answered pod list is not called unanswered by the baseline" {
# The absence branch fires on "read failed AND no eligible pod", which is what
# a partial answer produces: rows arrive, none of them can serve -- the
# affected pod is not Ready, the hostNetwork pod is excluded -- and the list
# then dies mid-stream. Saying it never answered would assert what the run did
# not observe. It did not FINISH, which is true either way.
#
# With one snapshot this is the only way to reach that branch at all: a list
# that answered nothing leaves no affected pods, so no pod file is written.
d=$(mktemp -d)
mkdir -p "$d/bin"
cat >"$d/bin/kubectl" <<'STUB'
#!/bin/sh
for a in "$@"; do
case $a in
pods)
echo 'tenant-test|wedged|10.0.0.1|node-a|False|Running||eol'
echo 'cozy-cilium|cilium-xyz|10.0.0.5|node-a|True|Running|true|eol'
exit 1 ;;
esac
done
exit 0
STUB
chmod +x "$d/bin/kubectl"
PATH="$d/bin:$PATH" timeout 90 "$SCRIPT" "$d/out" >"$d/log" 2>&1 || true
[ -f "$d/out/pod-tenant-test-wedged.txt" ]
grep -q 'REFERENCE (Ready) pod on node=node-a: unknown' "$d/out/pod-tenant-test-wedged.txt"
grep -q 'the pod list did not finish' "$d/out/pod-tenant-test-wedged.txt"
rm -rf "$d"
}
@test "a node whose snapshot holds no eligible pod says none found" {
# The other side of the same branch, and the reason it needs two wordings: a
# complete list with nothing eligible on the node is an answer, not a gap.
d=$(mktemp -d)
mkdir -p "$d/bin"
cat >"$d/bin/kubectl" <<'STUB'
#!/bin/sh
for a in "$@"; do
case $a in
pods)
echo 'tenant-test|wedged|10.0.0.1|node-a|False|Running||eol'
echo 'cozy-cilium|cilium-xyz|10.0.0.5|node-a|True|Running|true|eol'
exit 0 ;;
esac
done
exit 0
STUB
chmod +x "$d/bin/kubectl"
PATH="$d/bin:$PATH" timeout 90 "$SCRIPT" "$d/out" >"$d/log" 2>&1 || true
grep -q 'none found -- no baseline available' "$d/out/pod-tenant-test-wedged.txt"
if grep -q 'did not finish' "$d/out/pod-tenant-test-wedged.txt"; then
echo "a complete list was reported as unfinished:"
cat "$d/out/pod-tenant-test-wedged.txt"
exit 1
fi
rm -rf "$d"
}
@test "an affected pod is never offered as a healthy baseline" {
# Now true by construction rather than by exclusion: candidates come from the
# same rows the affected set came from, and one row cannot be both Ready and
# not-Ready. The test stays because the property is what matters, not the
# mechanism -- a future change that reads the node separately would reopen it.
d=$(mktemp -d)
mkdir -p "$d/bin"
cat >"$d/bin/kubectl" <<'STUB'
#!/bin/sh
for a in "$@"; do
case $a in
pods)
echo 'tenant-test|wedged-a|10.0.0.1|node-a|False|Running||eol'
echo 'tenant-test|wedged-b|10.0.0.2|node-a|False|Running||eol'
exit 0 ;;
esac
done
exit 0
STUB
chmod +x "$d/bin/kubectl"
PATH="$d/bin:$PATH" timeout 120 "$SCRIPT" "$d/out" >"$d/log" 2>&1 || true
[ -f "$d/out/pod-tenant-test-wedged-a.txt" ]
[ -f "$d/out/pod-tenant-test-wedged-b.txt" ]
for f in "$d"/out/pod-tenant-test-wedged-a.txt "$d"/out/pod-tenant-test-wedged-b.txt; do
if grep 'REFERENCE' "$f" | grep -q 'tenant-test/wedged'; then
echo "a pod under investigation was used as the healthy baseline in $(basename "$f"):"
grep '^# POD' "$f"
exit 1
fi
done
grep -q 'none found -- no baseline available' "$d/out/pod-tenant-test-wedged-b.txt"
rm -rf "$d"
}
@test "a memo hit reports the same answer as the miss that filled it" {
# The status has to survive the memo. Every caller reads pod_on_node through a
# command substitution, so a status kept in a variable dies with that subshell
@@ -776,7 +1312,7 @@ STUB
#!/bin/sh
for a in "$@"; do
case $a in
pods) echo 'tenant-test|wedged|10.0.0.1|node-a|False|Running'; exit 0 ;;
pods) echo 'tenant-test|wedged|10.0.0.1|node-a|False|Running||eol'; exit 0 ;;
svc) echo 'tenant|web|LoadBalancer|192.0.2.10|80|30080|Cluster'; exit 0 ;;
endpointslices) echo '10.0.0.1|node-a|tenant-test|wedged|true'; exit 0 ;;
esac
@@ -826,7 +1362,7 @@ STUB
#!/bin/sh
for a in "$@"; do
case $a in
pods) echo 'tenant-test|wedged|10.0.0.1|node-a|False|Running'; exit 0 ;;
pods) echo 'tenant-test|wedged|10.0.0.1|node-a|False|Running||eol'; exit 0 ;;
svc) echo 'tenant|web|LoadBalancer|192.0.2.10|80|30080|Cluster'; exit 0 ;;
endpointslices) echo '10.0.0.1|node-a|tenant-test|wedged|true'; exit 0 ;;
esac
hack/e2e-capture-dataplane.sh
+381 -107
View File
@@ -9,7 +9,8 @@
# snapshot dir so the output lands in the uploaded cozyreport artifact, and both
# wrap this script in a wall-clock backstop -- but not the same one (600s and
# 300s respectively), so a change sized against one caller can still overrun the
# other. The MAX_LBS note below carries that arithmetic.
# other. The MAX_LBS and MAX_PODS notes below carry that arithmetic, one per
# fan-out cap.
#
# Why this exists: a recurrent install failure is a CNI host->local-pod
# data-plane transient -- kubelet on a node reaches a *local* pod's
@@ -39,6 +40,16 @@
# kernel-path misforward / missing route / wrong rule / unresolved neighbor
# -- the actual mechanism behind host->local-pod "connection refused" on the
# legacy-routing path (the crux of the transient).
# - Address attribution for a podIP that turns out to be a LOCAL address of
# the host netns instead of reachable via ovn0 (a route to nowhere-in-
# particular is not enough to name the owner): the same host netns above
# also gets `ip -o addr show` (every interface) and `ip route show table
# local`, and one Ready, Running, non-hostNetwork pod on the same node that
# has an address gets the route and conntrack half of the per-pod capture
# below as an in-run healthy-pod control, since this collector otherwise
# inspects NotReady pods only and had no same-window baseline. hostNetwork
# is excluded because its podIP IS the node's own address, so it would show
# the same local-route fingerprint as the fault for an unrelated reason.
# - OVS/OVN on the node: `ovs-ofctl dump-flows br-int`, and the OVN
# Port_Binding / Logical_Switch_Port for the pod (LSP name = <pod>.<ns> in
# kube-ovn) plus a bounded `ovn-sbctl lflow-list`.
@@ -94,9 +105,9 @@
# In-guest capture INSIDE the worker VMI (tenant `cilium-dbg bpf lb list` /
# `tcpdump eth0`) is the one hop this cannot reach from the host; it is left
# as a documented stretch at the call site (it needs a tenant
# kubeconfig/virtctl that neither caller of this diagnostic has). The host-side
# ANNOUNCER/ENDPOINT split is the deliverable and already localises the
# failing hop to host-cilium vs kube-ovn delivery.
# kubeconfig/virtctl that neither caller of this diagnostic has). The
# host-side ANNOUNCER/ENDPOINT split is the deliverable and already
# localises the failing hop to host-cilium vs kube-ovn delivery.
#
# Robustness contract (matches docs/agents/e2e-testing.md): pure diagnostics,
# no retries, no behavior change, no traps. Every live capture is time-boxed,
@@ -134,13 +145,24 @@ set -u
# Service. Dropping it loses nothing a reader needs -- the read's own note
# already says the list did not finish.
#
# What a field count cannot catch: a cut inside the LAST field still delivers
# every separator, so `...|30080|Clus` counts seven and passes, and nothing
# in-band separates a truncated final value from a short one. Here the last
# column is display-only. In the two filters below it is a decision column
# (`phase`, `ready`), where a truncated value matches neither state being
# excluded and the row is kept -- over-capturing rather than dropping evidence,
# and the read's own cut-off note is what tells the reader the list ended early.
# What a field count cannot catch on its own: a cut inside the LAST field
# still delivers every separator, so `...|30080|Clus` counts seven and passes,
# and nothing in-band separates a truncated final value from a short one. Here
# the last column is display-only, so it costs nothing.
# `lb_first_ready_endpoint` decides on its own last column (`ready`) as a
# deny-list against the single value `false`, so a truncated value matches
# nothing it excludes and the row is kept -- over-capturing rather than
# dropping evidence, the safe direction for a diagnostic.
#
# The two pod filters do not rely on that reasoning at all. Their jsonpath
# ends in a constant `eol` column, and both require it present and intact, so
# a cut anywhere in a row either drops the field count or corrupts the
# sentinel. That matters for `pod_first_ready` specifically: it decides on
# `hostNetwork`, and a cut landing exactly on the preceding separator leaves
# that column EMPTY, which is byte-identical to the omitempty-false the filter
# must accept. No test of the value itself can tell those apart -- only a
# marker after it can. Either way the read's own cut-off note is what tells
# the reader the list ended early.
lb_filter_services() {
awk -F'|' 'NF == 7 && $3 == "LoadBalancer" && $4 != "" { print }'
}
@@ -247,19 +269,69 @@ lb_budget_ok() {
}
# pod_filter_affected: stdin =
# `namespace|name|podIP|nodeName|ready|phase` rows. Keep scheduled, non-terminal
# pods whose Ready condition is not True. podIP is deliberately NOT a gate: the
# cilium endpointManager leak this collector diagnoses can strand a pod before
# an IP is assigned, and the node-global Cilium/OVN state plus pod events remain
# useful in that state. IP-specific commands are gated later at their call sites.
# `namespace|name|podIP|nodeName|ready|phase|hostNetwork` rows. Keep scheduled,
# non-terminal pods whose Ready condition is not True. podIP is deliberately
# NOT a gate: the cilium endpointManager leak this collector diagnoses can
# strand a pod before an IP is assigned, and the node-global Cilium/OVN state
# plus pod events remain useful in that state. IP-specific commands are gated
# later at their call sites.
#
# NF == 6 for the reason lb_filter_services carries NF == 7: a list cut
# mid-record leaves a fragment whose remaining columns still read as values, and
# a cut landing inside the nodeName column produces a half-parsed node this
# capture would then open a node-<name>.txt for. Six fields per record is what
# the jsonpath emits.
# NF == 8 for the reason lb_filter_services carries its own count, though the
# counts differ because the jsonpaths do. A list cut mid-record leaves a
# fragment whose remaining columns still read as values, and a cut landing
# inside the nodeName column produces a half-parsed node this capture would then
# open a node-<name>.txt for. Eight fields per record is what the jsonpath
# emits: six this filter reads, then hostNetwork for the baseline filter below,
# then a constant `eol` whose only job is to be the thing a truncation destroys.
# Both are checked here even though neither is read, because that is what makes
# a cut anywhere in the row detectable.
pod_filter_affected() {
awk -F'|' 'NF == 6 && $4 != "" && $5 != "True" && $6 != "Succeeded" && $6 != "Failed"'
awk -F'|' 'NF == 8 && $8 == "eol" && $4 != "" && $5 != "True" && $6 != "Succeeded" && $6 != "Failed"'
}
# pod_first_ready <node>: stdin = the cluster-wide
# `namespace|name|podIP|nodeName|ready|phase|hostNetwork` rows, unfiltered.
# Emits `namespace|name|podIP` for the first pod ON <node> that is Ready=True,
# Running, addressed and NOT hostNetwork -- the healthy-pod baseline for that
# node -- then stops, or nothing when the node has no eligible pod.
#
# It scopes itself rather than being handed a per-node list, and that is the
# design rather than a convenience. The rows are the same snapshot the affected
# set was computed from, so an affected pod cannot be selected here:
# pod_filter_affected keeps rows whose $5 is not True, this one requires $5 to
# be True, and a row cannot be both. Reading the node separately would need an
# exclusion list instead, because a pod that recovered between the two reads
# reports Ready in the later one and could be offered as the baseline for its
# own failure.
#
# hostNetwork is excluded because such a pod's podIP IS the node's own address,
# and cilium-agent / kube-ovn-cni / ovs-ovn run hostNetwork on every node. A
# baseline picked from those would resolve `ip route get <ip>` to `local <ip>
# dev lo table local` for an entirely ordinary reason -- the exact fingerprint
# the address-attribution capture above exists to recognise -- so the
# comparison would defeat itself. A blank hostNetwork column counts as NOT
# hostNetwork, since the API omits the field when it is false.
#
# The test is written as an allow-list -- empty or the literal "false" -- rather
# than as `!= "true"`, which rejects a value cut mid-token: `tru` carries every
# field and passes every other gate, and because this filter stops at the row it
# takes, a deny-list would seat a hostNetwork pod as the baseline. That alone
# is not enough, though: a cut landing on the separator BEFORE hostNetwork
# leaves the column empty and indistinguishable from a genuine false. That case
# is caught by the constant `eol` column instead -- the row loses the sentinel
# with its tail. The two guards cover different cuts and neither replaces the
# other.
#
# An addressless pod is skipped for the same `exit`-at-first-match reason: a
# Ready pod that has no podIP yet would become the baseline and then carry none
# of the route or conntrack evidence the comparison exists for.
#
# NF == 8 and an intact `eol` for the reason the filters above carry their own
# counts: a list cut mid-record leaves a fragment whose remaining columns still
# read as values, and a fragment here would name a pod that does not exist as
# the baseline every reading of the affected pod gets compared against.
pod_first_ready() {
awk -F'|' -v node="$1" 'NF == 8 && $8 == "eol" && $4 == node && $3 != "" && $5 == "True" && $6 == "Running" && ($7 == "" || $7 == "false") { print $1 "|" $2 "|" $3; exit }'
}
# How a cutoff should be described, mirroring prevlog_cutoff_desc in the sibling
@@ -315,8 +387,52 @@ CILIUM_NS="${COZY_CILIUM_NS:-cozy-cilium}"
KUBEOVN_NS="${COZY_KUBEOVN_NS:-cozy-kubeovn}"
# Cap how many pods we inspect so a fully-wedged cluster cannot explode the
# runtime; the per-command timeouts and the call-site wall-clock wrapper are the
# other two bounds. Node-global captures are deduped per node, so the effective
# work is closer to (#affected-nodes) than (#affected-pods).
# other two bounds. Node-global captures are deduped per node, and so is the
# healthy-pod baseline, so the total is (#affected-pods) per-pod captures plus
# (#affected-nodes) baselines and node-global captures -- not two per pod.
#
# The header asks for this arithmetic against the tighter caller rather than
# only the roomier one, so here it is. Every term counts the OUTER bound of
# each call and adds the bounded lookups that call makes; a nested inner
# timeout is not a second wait. The terms are the whole path before
# capture_lb_datapath, not only the per-pod ones: the preamble is 82s of the
# total, and a sum that omits it reports a margin roomy enough for another
# collector where the real one has no room at all.
# - preamble, once per run: 82s -- the cluster pod list at 28s+2s, the
# ovn-central lookup at 20s+2s, and the 30s ovn-sbctl lflow-list;
# - a node capture bounds at 288s: 222s of execs (the `cilium-dbg monitor`
# leg counts its outer 12s, not the inner 8s it wraps) plus three 22s
# pod_on_node lookups, memo misses the first time a node is seen;
# - an affected pod's capture bounds at 179s: 135s of execs plus two 20s
# reads with their 2s kill graces;
# - the baseline adds 30s per affected node: it runs at route-only scope, so
# the route and the conntrack and nothing else, and it issues no read of
# its own.
#
# One affected pod on one node therefore bounds at 579s, inside the 600s the
# cozytest.sh trap allows and still past the 300s the Chainsaw catch shares
# with its snapshot leg. That margin is 21 seconds, so the next collector
# added to this path does not fit and the number has to be recomputed rather
# than assumed. Read the 21s as what this path leaves the rest of the run, not
# as headroom the run has: capture_lb_datapath executes unconditionally after
# it and inside the same backstop, and the MAX_LBS note above says that section
# bounds work rather than wall clock, with each heavy capture costing tens of
# seconds.
#
# The 579s holds only WHILE the lookup memo exists. When mktemp has failed and
# _POD_MEMO is empty every pod_on_node is live again, each capture pays for
# its own, and the same case bounds at 667s -- outside both backstops. The
# guarantee is conditional and this is the condition.
#
# Route-only scope is what buys it. At full scope the baseline costs 179s
# instead of 30s, the single-pod case bounds at 728s, and it would lose a
# backstop it clears at the merge base (519s there). The three blocks that
# scope drops describe how a pod was programmed, and a pod chosen for being
# healthy is programmed correctly by definition -- they compare nothing.
#
# What the numbers do say is that this cap counts pods while the ceiling is
# measured in seconds, and past one affected pod the two stop meeting: raising
# MAX_PODS buys nothing the envelope can pay for.
MAX_PODS="${COZY_DATAPLANE_MAX_PODS:-12}"
# LoadBalancer-datapath section tunables (see the header block and the
@@ -411,9 +527,18 @@ log() {
# Failed pods (completed Jobs, hook pods) are Ready!=True too but are not either
# symptom, so they are excluded to keep the MAX_PODS budget on actual wedges.
# jsonpath keeps this dependency-free (no jq / go-template reassignment).
#
# hostNetwork rides along on this one read even though only the healthy-pod
# baseline consults it, and only for nodes that turn out to have an affected
# pod. Asking per node instead would be a second read of the same objects, and
# the two answers could then disagree: a pod that recovered in between reports
# Ready in the later one and could be offered as the baseline for its own
# failure. One snapshot makes that impossible rather than guarded -- the
# affected set and the baseline candidates are the same rows -- and the extra
# column costs one boolean per pod on a list this script already takes.
# shellcheck disable=SC2086 # empty DP_LIST_BOUND must vanish, not become ""
_pods_raw=$($DP_LIST_BOUND kubectl get pods -A \
-o jsonpath='{range .items[*]}{.metadata.namespace}{"|"}{.metadata.name}{"|"}{.status.podIP}{"|"}{.spec.nodeName}{"|"}{.status.conditions[?(@.type=="Ready")].status}{"|"}{.status.phase}{"\n"}{end}' \
-o jsonpath='{range .items[*]}{.metadata.namespace}{"|"}{.metadata.name}{"|"}{.status.podIP}{"|"}{.spec.nodeName}{"|"}{.status.conditions[?(@.type=="Ready")].status}{"|"}{.status.phase}{"|"}{.spec.hostNetwork}{"|eol"}{"\n"}{end}' \
2>"${DP_ERR:-/dev/null}")
_pods_rc=$?
if [ "$_pods_rc" -ne 0 ]; then
@@ -497,6 +622,11 @@ mark_node() { _SEEN_NODES="$_SEEN_NODES$1 "; }
# be paid for again in the sections that follow. Empty when mktemp fails, which
# only costs the memo.
_POD_MEMO=$(mktemp "${TMPDIR:-/tmp}/dataplane-podmemo.XXXXXX" 2>/dev/null) || _POD_MEMO=""
# Cache for the per-node healthy-pod baseline, one file per node, for the same
# reason and with the same subshell constraint as the memo above. Empty when
# mktemp fails, which costs the saving and not the evidence: the baseline is
# then captured per pod, the way it was before the cache existed.
_REF_CACHE=$(mktemp -d "${TMPDIR:-/tmp}/dataplane-refcache.XXXXXX" 2>/dev/null) || _REF_CACHE=""
# The stored row carries the read's status beside its value. A hit that dropped
# it would hand a later caller an empty answer with no way to tell "there is no
# such pod" from "the read never said", which is the distinction the callers
@@ -613,6 +743,23 @@ capture_node() {
echo "=== host netns: ip addr show ovn0 ==="
timeout 15 kubectl exec -n "$KUBEOVN_NS" "$cni" -c cni-server -- \
ip addr show ovn0 2>&1 || true
echo
echo "=== host netns: ip -o addr show, all interfaces (address attribution) ==="
# ovn0 alone cannot name the owner when a podIP turns out to be a LOCAL
# address instead: the interface actually holding it might be anything on
# the host (see the header block). This is the same info scoped to every
# interface, not just ovn0.
timeout 15 kubectl exec -n "$KUBEOVN_NS" "$cni" -c cni-server -- \
ip -o addr show 2>&1 || true
echo
echo "=== host netns: ip route show table local (address attribution) ==="
# A podIP resolving to `local <ip> dev lo table local` here is the
# fingerprint from the header block: it wins over the ovn0 route in
# `main` because table local is consulted first (see `ip rule` above).
timeout 15 kubectl exec -n "$KUBEOVN_NS" "$cni" -c cni-server -- \
ip route show table local 2>&1 || true
else
echo
if [ "$_cni_ok" -eq 0 ]; then
@@ -624,6 +771,210 @@ capture_node() {
} >> "$nf" 2>&1 || true
}
# capture_pod_dataplane <ns> <pod> <podip> <node> <label> [scope] -- the
# pod-specific captures for ONE pod on <node>.
#
# scope is `full` (default) or `route-only`. Full takes everything: Ready
# conditions and events, cilium CT, the host route, kernel conntrack, the OVN
# port binding and the OVS ovn-installed flag. route-only takes the route and
# the conntrack and nothing else, and the healthy-pod baseline asks for it.
# Three of the four blocks it drops -- cilium CT, the OVN port binding, the OVS
# ovn-installed flag -- describe how a pod was PROGRAMMED, and a pod chosen for
# being healthy is correctly programmed by definition; they compare nothing and
# cost 105s of the 135s of exec bounds, on the leg most likely to be cut. The
# fourth is the Ready-conditions and events pair, which a pod selected for
# reporting Ready=True and Running has already answered, and dropping it is what
# leaves the baseline with no read of its own -- the 30s the MAX_PODS note above
# spends on it, against 179s at full scope. What a baseline is read for is the
# route and the conntrack beside the wedged pod's own. <label> is
# stamped in the section header and says which role the pod plays: the affected
# pod under investigation, or the Ready pod captured beside it as a baseline.
# Writes to stdout; the caller redirects into the pod's output file. $central
# (the ovn-central pod, or empty) is read from the caller's scope rather than
# threaded as a parameter, matching how the rest of this script shares
# node-global lookups.
capture_pod_dataplane() {
_cpd_ns=$1; _cpd_pod=$2; _cpd_ip=$3; _cpd_node=$4; _cpd_label=$5; _cpd_scope=${6:-full}
# Banner first, then the lookups: pod_on_node writes its failure note to
# stderr, and this whole function is redirected into the pod's file, so a
# lookup resolved above the banner puts that note on line 1 and pushes the
# header down. The note still reaches capture-notes.txt either way.
echo "################################################################"
echo "# POD $_cpd_ns/$_cpd_pod podIP=${_cpd_ip:-<none>} node=$_cpd_node ($_cpd_label)"
echo "# node-global captures are in node-$_cpd_node.txt"
echo "################################################################"
# Only the cni-server is needed at route-only scope; resolving the other two
# there would cost a live lookup each whenever the memo is unavailable, for
# blocks that scope does not take.
_cpd_cni=$(pod_on_node "$KUBEOVN_NS" app=kube-ovn-cni "$_cpd_node")
_cpd_agent=""
_cpd_ovs=""
if [ "$_cpd_scope" = full ]; then
_cpd_agent=$(pod_on_node "$CILIUM_NS" k8s-app=cilium "$_cpd_node")
_cpd_ovs=$(pod_on_node "$KUBEOVN_NS" app=ovs "$_cpd_node")
else
echo
echo "=== scope: route and conntrack only ==="
echo "This is a reference pod, so the cilium CT, OVN Port_Binding and OVS"
echo "ovn-installed blocks are deliberately NOT taken: they describe how a"
echo "pod was programmed, and this one was chosen for being healthy. Its"
echo "Ready conditions and events go for the same reason: reporting Ready"
echo "and Running is what selected it."
echo "Their absence below is a decision, not a lookup that came back empty."
fi
if [ "$_cpd_scope" = full ]; then
echo
echo "=== pod Ready conditions + recent probe events ==="
# shellcheck disable=SC2086 # empty DP_BOUND must vanish, not become ""
$DP_BOUND kubectl get pod -n "$_cpd_ns" "$_cpd_pod" \
-o jsonpath='{range .status.conditions[*]}{.type}={.status} reason={.reason}: {.message}{"\n"}{end}' 2>&1 \
|| echo "(reading this pod's Ready conditions $(dp_read_outcome "$?" "$DP_READ_TIMEOUT" "$DP_BOUND"))"
# shellcheck disable=SC2086 # empty DP_BOUND must vanish, not become ""
$DP_BOUND kubectl get events -n "$_cpd_ns" --field-selector "involvedObject.name=$_cpd_pod" \
-o jsonpath='{range .items[*]}{.lastTimestamp}{" "}{.reason}{": "}{.message}{"\n"}{end}' 2>&1 \
|| echo "(reading this pod's events $(dp_read_outcome "$?" "$DP_READ_TIMEOUT" "$DP_BOUND"))"
fi
if [ -z "$_cpd_ip" ]; then
echo
echo "=== pod has no podIP; IP-specific route/conntrack capture skipped ==="
echo "Node-global Cilium/OVN state and pod allocation events remain captured."
fi
if [ "$_cpd_scope" = full ] && [ -n "$_cpd_ip" ] && [ -n "$_cpd_agent" ]; then
echo
echo "=== cilium-dbg bpf ct list global | grep $_cpd_ip (node=$_cpd_node agent=$_cpd_agent) ==="
timeout 25 kubectl exec -n "$CILIUM_NS" "$_cpd_agent" -c cilium-agent -- \
sh -c "cilium-dbg bpf ct list global 2>/dev/null | grep -F '$_cpd_ip' || echo 'no CT entries for $_cpd_ip'" 2>&1 || true
fi
if [ -n "$_cpd_ip" ] && [ -n "$_cpd_cni" ]; then
echo
echo "=== host netns: ip route get $_cpd_ip (via kube-ovn cni-server) ==="
timeout 15 kubectl exec -n "$KUBEOVN_NS" "$_cpd_cni" -c cni-server -- \
ip route get "$_cpd_ip" 2>&1 || true
echo
echo "=== host netns: kernel conntrack for $_cpd_ip ==="
# Under enable-host-legacy-routing the host->local-pod path is in the
# kernel netfilter conntrack table, NOT cilium BPF -- this is the
# authoritative table for the transient. Prefer the conntrack CLI; fall
# back to /proc/net/nf_conntrack when it is absent from the image.
timeout 15 kubectl exec -n "$KUBEOVN_NS" "$_cpd_cni" -c cni-server -- \
sh -c "if command -v conntrack >/dev/null 2>&1; then conntrack -L 2>/dev/null | grep -F '$_cpd_ip' || echo 'no conntrack entries for $_cpd_ip'; else grep -F '$_cpd_ip' /proc/net/nf_conntrack 2>/dev/null || echo 'no conntrack CLI; no /proc/net/nf_conntrack match for $_cpd_ip'; fi" 2>&1 || true
fi
if [ "$_cpd_scope" = full ] && [ -n "$central" ]; then
echo
echo "=== OVN Port_Binding / Logical_Switch_Port for $_cpd_pod.$_cpd_ns ==="
# kube-ovn names the OVN logical port <pod>.<namespace>.
timeout 20 kubectl exec -n "$KUBEOVN_NS" "$central" -c ovn-central -- \
ovn-sbctl --no-leader-only find port_binding "logical_port=$_cpd_pod.$_cpd_ns" 2>&1 || true
timeout 20 kubectl exec -n "$KUBEOVN_NS" "$central" -c ovn-central -- \
ovn-nbctl --no-leader-only find logical_switch_port "name=$_cpd_pod.$_cpd_ns" 2>&1 || true
fi
if [ "$_cpd_scope" = full ] && [ -n "$_cpd_ovs" ]; then
echo
echo "=== OVS interface ovn-installed flag + ts for iface-id=$_cpd_pod.$_cpd_ns (ovs pod $_cpd_ovs) ==="
# kube-ovn stamps external_ids:ovn-installed (+ -ts) on the pod's OVS
# interface once ovn-controller reports the port programmed -- this is
# the barrier the CNI waits on. The OVS interface name is opaque, so
# look it up by iface-id (<pod>.<ns>), then read the flag + timestamp.
# An ovn-installed-ts set early (before physical_flow_output installed
# the local-delivery flow, see node-$node.txt) is the I-P-lag signature.
_cpd_ovsif=$(timeout 20 kubectl exec -n "$KUBEOVN_NS" "$_cpd_ovs" -c openvswitch -- \
ovs-vsctl --no-heading --columns=name find interface "external_ids:iface-id=$_cpd_pod.$_cpd_ns" 2>/dev/null \
| head -n 1 | tr -d '" ')
if [ -n "$_cpd_ovsif" ]; then
echo "ovs interface = $_cpd_ovsif"
timeout 20 kubectl exec -n "$KUBEOVN_NS" "$_cpd_ovs" -c openvswitch -- \
ovs-vsctl get interface "$_cpd_ovsif" external_ids:ovn-installed external_ids:ovn-installed-ts 2>&1 || true
else
echo "no OVS interface with external_ids:iface-id=$_cpd_pod.$_cpd_ns"
fi
fi
}
# capture_pod_reference <node> <outfile> -- the healthy-pod baseline for a
# NODE, appended to <outfile>. This collector otherwise inspects NotReady pods
# only, so a run has nothing to compare a suspicious route or conntrack state
# against from the same node in the same window. Picks one Ready, Running,
# non-hostNetwork pod on <node> that this run is not already investigating, and
# runs the route-only half of the per-pod capture against it -- the route and
# the conntrack, which is what a baseline is read for.
#
# The baseline belongs to the node, not to any one affected pod, and nothing
# about it depends on which pod asked: the candidates are the rows of the one
# snapshot, and pod_first_ready keeps only the Ready=True ones, which every
# affected pod fails by definition. That is what keeps the answer a function of
# the node alone, and that is the precondition for the cache below. Had
# eligibility been decided by excluding the asking pod instead, one node would
# yield a different baseline for each caller, and a shared cache would then hand
# one pod's answer to another -- including to the pod the answer names. Hence
# the signature: a node, and a file to append to.
#
# The absence is recorded rather than left silent, and so is the difference
# between the two ways of having none: a node with no eligible pod is not a
# read that never answered, and the artifact says which one happened.
capture_pod_reference() {
_cpr_node=$1; _cpr_of=$2
# Captured once per NODE, then folded into every affected pod's file from a
# cache. The pod is a function of the node and nothing else, so several
# affected pods on one node -- the ordinary shape of a wedged host->local-pod
# datapath rather than an edge -- would otherwise each pay for a second full
# per-pod capture of the same healthy pod. The cache is about that capture,
# not about any read: the candidates come from the cluster-wide pod list this
# script has already taken.
_cpr_cache="${_REF_CACHE:+$_REF_CACHE/$_cpr_node}"
if [ -n "$_cpr_cache" ] && [ -f "$_cpr_cache" ]; then
cat "$_cpr_cache" >> "$_cpr_of" 2>/dev/null || true
return 0
fi
# With no cache directory the capture still happens, once per pod as before.
_cpr_sink="${_cpr_cache:-$_cpr_of}"
# Picked out of the SAME snapshot the affected set came from, which is what
# makes an affected pod ineligible by construction rather than by exclusion:
# pod_filter_affected keeps rows whose Ready is not True, pod_first_ready
# requires Ready True, and one row cannot satisfy both. A second read would
# reopen that -- a pod recovering between the two reports Ready in the later
# one, and could be offered as the baseline for its own failure.
_cpr_ref=$(printf '%s\n' "$_pods_raw" | pod_first_ready "$_cpr_node")
if [ -z "$_cpr_ref" ]; then
{
echo
echo "################################################################"
if [ "$_pods_rc" -eq 0 ]; then
echo "# REFERENCE (Ready) pod on node=$_cpr_node: none found -- no baseline available"
else
echo "# REFERENCE (Ready) pod on node=$_cpr_node: unknown -- the pod list did not finish, so whether a baseline exists was never established"
fi
echo "################################################################"
} >> "$_cpr_sink" 2>&1 || true
if [ -n "$_cpr_cache" ]; then
cat "$_cpr_cache" >> "$_cpr_of" 2>/dev/null || true
fi
return 0
fi
_cpr_refns=$(printf '%s' "$_cpr_ref" | cut -d'|' -f1)
_cpr_refpod=$(printf '%s' "$_cpr_ref" | cut -d'|' -f2)
_cpr_refip=$(printf '%s' "$_cpr_ref" | cut -d'|' -f3)
{
echo
capture_pod_dataplane "$_cpr_refns" "$_cpr_refpod" "$_cpr_refip" "$_cpr_node" "REFERENCE (Ready) on node=$_cpr_node" route-only
} >> "$_cpr_sink" 2>&1 || true
if [ -n "$_cpr_cache" ]; then
cat "$_cpr_cache" >> "$_cpr_of" 2>/dev/null || true
fi
}
if [ -z "$affected" ] && [ "$_pods_rc" -ne 0 ]; then
# Says what is unknown rather than what the read did. The list can answer in
# part and name only healthy pods, which lands here too, so a line asserting
@@ -671,7 +1022,7 @@ else
i=0
printf '%s\n' "$affected" | {
while IFS='|' read -r ns pod podip node _ready _phase; do
while IFS='|' read -r ns pod podip node _ready _phase _hostnet _eol; do
[ -n "$ns" ] && [ -n "$pod" ] && [ -n "$node" ] || continue
i=$((i + 1))
if [ "$i" -gt "$MAX_PODS" ]; then
@@ -683,88 +1034,10 @@ else
# ip rule, ovn0 addr) -- captured once per node.
capture_node "$node"
# Pod-specific state.
# Pod-specific state, plus a healthy pod on the same node as a baseline.
pf="$OUT/pod-$ns-$pod.txt"
agent=$(pod_on_node "$CILIUM_NS" k8s-app=cilium "$node")
ovs=$(pod_on_node "$KUBEOVN_NS" app=ovs "$node")
{
echo "################################################################"
echo "# POD $ns/$pod podIP=${podip:-<none>} node=$node (Ready=$_ready)"
echo "# node-global captures are in node-$node.txt"
echo "################################################################"
echo
echo "=== pod Ready conditions + recent probe events ==="
# shellcheck disable=SC2086 # empty DP_BOUND must vanish, not become ""
$DP_BOUND kubectl get pod -n "$ns" "$pod" \
-o jsonpath='{range .status.conditions[*]}{.type}={.status} reason={.reason}: {.message}{"\n"}{end}' 2>&1 \
|| echo "(reading this pod's Ready conditions $(dp_read_outcome "$?" "$DP_READ_TIMEOUT" "$DP_BOUND"))"
# shellcheck disable=SC2086 # empty DP_BOUND must vanish, not become ""
$DP_BOUND kubectl get events -n "$ns" --field-selector "involvedObject.name=$pod" \
-o jsonpath='{range .items[*]}{.lastTimestamp}{" "}{.reason}{": "}{.message}{"\n"}{end}' 2>&1 \
|| echo "(reading this pod's events $(dp_read_outcome "$?" "$DP_READ_TIMEOUT" "$DP_BOUND"))"
if [ -z "$podip" ]; then
echo
echo "=== pod has no podIP; IP-specific route/conntrack capture skipped ==="
echo "Node-global Cilium/OVN state and pod allocation events remain captured."
fi
if [ -n "$podip" ] && [ -n "$agent" ]; then
echo
echo "=== cilium-dbg bpf ct list global | grep $podip (node=$node agent=$agent) ==="
timeout 25 kubectl exec -n "$CILIUM_NS" "$agent" -c cilium-agent -- \
sh -c "cilium-dbg bpf ct list global 2>/dev/null | grep -F '$podip' || echo 'no CT entries for $podip'" 2>&1 || true
fi
cni=$(pod_on_node "$KUBEOVN_NS" app=kube-ovn-cni "$node")
if [ -n "$podip" ] && [ -n "$cni" ]; then
echo
echo "=== host netns: ip route get $podip (via kube-ovn cni-server) ==="
timeout 15 kubectl exec -n "$KUBEOVN_NS" "$cni" -c cni-server -- \
ip route get "$podip" 2>&1 || true
echo
echo "=== host netns: kernel conntrack for $podip ==="
# Under enable-host-legacy-routing the host->local-pod path is in the
# kernel netfilter conntrack table, NOT cilium BPF -- this is the
# authoritative table for the transient. Prefer the conntrack CLI; fall
# back to /proc/net/nf_conntrack when it is absent from the image.
timeout 15 kubectl exec -n "$KUBEOVN_NS" "$cni" -c cni-server -- \
sh -c "if command -v conntrack >/dev/null 2>&1; then conntrack -L 2>/dev/null | grep -F '$podip' || echo 'no conntrack entries for $podip'; else grep -F '$podip' /proc/net/nf_conntrack 2>/dev/null || echo 'no conntrack CLI; no /proc/net/nf_conntrack match for $podip'; fi" 2>&1 || true
fi
if [ -n "$central" ]; then
echo
echo "=== OVN Port_Binding / Logical_Switch_Port for $pod.$ns ==="
# kube-ovn names the OVN logical port <pod>.<namespace>.
timeout 20 kubectl exec -n "$KUBEOVN_NS" "$central" -c ovn-central -- \
ovn-sbctl --no-leader-only find port_binding "logical_port=$pod.$ns" 2>&1 || true
timeout 20 kubectl exec -n "$KUBEOVN_NS" "$central" -c ovn-central -- \
ovn-nbctl --no-leader-only find logical_switch_port "name=$pod.$ns" 2>&1 || true
fi
if [ -n "$ovs" ]; then
echo
echo "=== OVS interface ovn-installed flag + ts for iface-id=$pod.$ns (ovs pod $ovs) ==="
# kube-ovn stamps external_ids:ovn-installed (+ -ts) on the pod's OVS
# interface once ovn-controller reports the port programmed -- this is
# the barrier the CNI waits on. The OVS interface name is opaque, so
# look it up by iface-id (<pod>.<ns>), then read the flag + timestamp.
# An ovn-installed-ts set early (before physical_flow_output installed
# the local-delivery flow, see node-$node.txt) is the I-P-lag signature.
ovsif=$(timeout 20 kubectl exec -n "$KUBEOVN_NS" "$ovs" -c openvswitch -- \
ovs-vsctl --no-heading --columns=name find interface "external_ids:iface-id=$pod.$ns" 2>/dev/null \
| head -n 1 | tr -d '" ')
if [ -n "$ovsif" ]; then
echo "ovs interface = $ovsif"
timeout 20 kubectl exec -n "$KUBEOVN_NS" "$ovs" -c openvswitch -- \
ovs-vsctl get interface "$ovsif" external_ids:ovn-installed external_ids:ovn-installed-ts 2>&1 || true
else
echo "no OVS interface with external_ids:iface-id=$pod.$ns"
fi
fi
} > "$pf" 2>&1 || true
capture_pod_dataplane "$ns" "$pod" "$podip" "$node" "NotReady, Ready=$_ready" > "$pf" 2>&1 || true
capture_pod_reference "$node" "$pf"
done
log "host->pod data-plane capture complete"
}
@@ -1209,5 +1482,6 @@ capture_lb_datapath
# TMPDIR.
[ -n "$DP_ERR" ] && rm -f "$DP_ERR"
[ -n "$_POD_MEMO" ] && rm -f "$_POD_MEMO"
[ -n "$_REF_CACHE" ] && rm -rf "$_REF_CACHE"
exit 0