Files
cozystack/hack/e2e-chainsaw/_lib
myasnikovdaniil aa707f17b5 test(etcd): stop losing the metrics probe's output to the attach race (#3350)
## What this PR does

The etcd suite's metrics contract failed in CI with `metrics endpoint
did not return 200 over http on the 'metrics' port; got: If you don't
see a command prompt, try pressing enter.`

The probe runs a one-shot curl Pod and reads a `code=<status>` marker
back off its output via `kubectl run --rm --restart=Never --attach`.
`--attach` only establishes its connection once the Pod reaches Running,
so a container that exits in milliseconds — which is what curl against a
Pod IP does — writes its stdout into the void, and `--rm` then destroys
the Pod carrying the log that still held it. The step reports the
metrics endpoint as broken on a cluster that was serving it correctly
throughout.

The evidence that this was output loss rather than an endpoint failure
is that the captured output contained no `code=` at all. A genuine
connection failure would still have printed `code=000` from `-w`, plus
curl's own error text from `-fsS`, and both were being captured.
Meanwhile the probe Pod's events show a clean run, all three etcd
members were up without restarts, and every preceding assert in the
suite passed.

The fix drops `--rm`, re-reads the Pod log when the attach stream
carried no marker, and deletes the Pod explicitly afterwards. This is
the same treatment `hack/e2e-chainsaw/_lib/talos-image-cache.sh` already
applies to the identical race — that probe documents it inline; the etcd
one was simply never given it. A `--pod-running-timeout` is set for the
same reason it is there: the probe image is used nowhere else in the
tree, so it pays a cold pull on every run and kubectl's one-minute
default was a second way to end up with no marker.

The recovery cannot turn a real failure green. The marker encodes the
outcome, so a `code=503` recovered from the log still fails the step; an
attach stream that already carried a marker is never second-guessed; and
the pre-run delete blocks, so a stale Pod's log cannot be read in place
of this run's. Verified by running the probe's exact curl invocation
against a 503 and against a dead port: the first yields `code=503`, the
second `code=000` plus curl's error, and both fail. The probe still runs
exactly once — this is output recovery, not a retry.

Helper extracted to `hack/e2e-chainsaw/_lib/etcd-probe.sh` with unit
coverage in `hack/etcd-probe_test.bats`.

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved etcd metrics probe reliability by recovering probe results
when attached output is incomplete.
* Preserved meaningful probe and Kubernetes error details, including
non-success status codes.
* Improved probe pod timing and cleanup to reduce intermittent
end-to-end test failures.

* **Tests**
* Added coverage for successful recovery, failed probes, missing output,
and log retrieval errors.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-20 12:26:08 +05:00
..