kubectl logs --previous returns the immediately preceding terminated
container instance, not the original first one — after multiple restarts
the first instance is already gone. Reword the helper, the BATS and
Chainsaw failure handlers, and the diagnostics doc to match the actual
--previous semantics.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Root-causing a crash-loop failure was blocked twice over by the
diagnostics collected on failure.
`kubectl logs` shows only the current container instance, so for a
crash-looping pod the suites' `podLogs` collector captured the Nth
restart -- a fresh replay of the same startup path -- while the
evidence that explained the failure sat in the first, interrupted
instance. Chainsaw's podLogs collector has no `previous` option, so
add e2e-capture-previous-logs.sh and drive it from the global catch
and from the BATS failure hook. It discovers containers with
restartCount > 0 at runtime (init containers included, where an
interrupted bootstrap lives), orders the app suites' namespace first,
and dumps each to the job log and the snapshot tree. Containers that
never restarted are skipped, so a healthy run emits one line rather
than a page of "previous terminated container not found".
Every way that capture can come up short is named rather than
inferred, because kubectl exits 1 for a garbage-collected container,
an RBAC denial and an unreachable node alike: the container cap, a
pod list that never returned (which is not the same as nothing having
restarted), a read that timed out versus one kubectl refused and the
reason it gave, and a previous instance that produced no output. A
partial capture is kept and marked truncated rather than deleted --
it is the only copy of a log already gone from the cluster.
The crust-gather snapshot was killed by the Chainsaw op containing it.
A script op is bounded by timeouts.exec (5m), and the catch's own
wall-clock wrappers summed to 960s, so the script was SIGKILLed
partway through -- taking out the process group mid-write and skipping
the data-plane capture entirely. Give each catch op an explicit
timeout above the budgets it contains, so an inner timeout fires
instead and leaves a partial, still-servable snapshot behind. The
data-plane leg drops from 600s to 300s at that call site so both legs
fit one envelope; the BATS hook keeps the wider budget, having no
containing op.
Set crust-gather --duration explicitly as well. It is the collection
phase's own budget, it defaults to 60s, and on elapse it abandons the
collection and skips its finish step, leaving a truncated tree with no
indication anything is missing. The outer wall-clock has to exceed it
because the API discovery that runs first is not covered by that
budget at all -- with an unhealthy aggregated APIService, the state a
failed run is usually in, discovery alone can consume the whole
timeout before a single object is written. Send each collector's
output to a log beside its snapshot instead of /dev/null, with a
complete/truncated/failed verdict, so that question is answerable
from the artifact.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The v0.2.15 syntax/semantics traps were tribal knowledge scattered across PR
review notes (the number-literal caveat lived only in #2826's body, which
evaporates on merge). Consolidate them into a durable subsection so the next
author finds them before hitting each one fresh: filter-as-list conditions,
no number-literals/backticks in JMESPath, error: fails hard on an unknown
GVK, no conditional Test-level skip, step-level-only finally, bounded/blocking
cleanup surfacing latent teardown bugs, projection-style failure diffs, and
the catch-script cwd.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Reconcile the Chainsaw e2e migration with ~500 commits of main drift:
- keep main's run-kubernetes.sh helpers (kubectl_wait_retry, tenant
drain, LINSTOR pool-free wait, SC-fallback probe) under the branch's
_lib/ rename with the Chainsaw finally/snapshot deltas re-applied
- resolve modify/delete on 10 drifted hack/e2e-apps/*.bats as deleted;
their post-branch assertion drift is re-ported into the Chainsaw
suites in follow-up commits
- drop the 4 new main-side bats (kubernetes-oidc-*, securitygroup,
serviceexposure) from _lib/ relocation; ported as Chainsaw suites in
follow-up commits
- docs/agents/e2e-testing.md: merge main's plan-job TIA wording with
the Chainsaw suite phrasing
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
- scope create-github-app-token to the current repo with read-only
contents permission (was org-wide, inheriting the app's blanket
installation permissions); both token steps only read releases/assets
- fail-fast the Talos image upload when the disk is missing
(if-no-files-found: error) so drift fails in build-talos, not later
as a confusing e2e download error
- download the flux install script to a tempfile before executing it,
so a failed curl is not masked by the piped `sudo bash` exiting 0
- fix the regular-PR artifact-source comment (Talos disk comes from
build-talos, pr.patch from finalize) and the TIA scope wording in
docs/agents/e2e-testing.md
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The Chainsaw app suite runs via `chainsaw test` directly, bypassing
hack/cozytest.sh — whose EXIT trap captures a crust-gather host snapshot
per failed BATS test. As a result a failing Chainsaw run uploaded a
cozyreport.tgz with zero per-test cluster snapshots, the richest diagnostic
for failures like the gateway/etcd/bucket/openbao ones on this branch.
Add the Chainsaw analog: a global spec.error.catch in .chainsaw.yaml that
collects a bounded host snapshot into _out/cozyreport/snapshots/<test>/host
on any failure (same --exclude-kind Secret flags and timeout -k 30 300 as
cozytest.sh), keyed by the $test.metadata.name binding with a mkdir-guard so
repeat catch invocations don't re-collect. Wire COZY_SNAPSHOT_NAME into the
kubernetes-latest/previous suites so the existing tenant snapshot from
run-kubernetes.sh co-locates under the same snapshots/<test>/ dir. cozyreport.sh
already folds snapshots/ into the uploaded artifact.
The e2e-testing guide and the sandbox Dockerfile comment are updated to record
the convention.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Rewrite docs/agents/e2e-testing.md for the Chainsaw suite: declarative
asserts replace the until-get/wait backstops, Chainsaw cleanup replaces
test-level EXIT traps, catch blocks replace the `|| { ...; false; }` idiom,
and the TIA/diagnostics sections describe suites + `make test-chainsaw
CHAINSAW_SUITES`. Move the BATS->Chainsaw bullet out of 'in-flight' since it
is now the standard.
Fix references the migration left stale: hack/e2e-apps/ -> hack/e2e-chainsaw/
in the gemini styleguide and the nats/etcd backup READMEs. The
backup-postgres.bats / backup-foundationdb.bats files cited by the postgres
examples, the postgres README, and the foundationdb controller comment never
existed -- drop the false 'CI e2e drives this' claims and describe those
backup flows as the manual reference flows they are.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Reconcile the parallel-matrix PR build (#2983) with main, which since this branch
diverged landed a competing single-VM build restructure (#2939) and
test-impact-analysis E2E (#2559). Per maintainer decision, #2983's matrix build
supersedes #2939:
- .github/workflows/pull-requests.yaml: keep #2983's
plan/checks/build(matrix)/build-talos/finalize jobs; keep main's newer
resolve_assets/e2e (carrying #2559 test-impact E2E). The build->e2e artifact
interface (pr-patch + talos-image) is identical on both sides, so e2e depends
on finalize/build-talos with no input rewiring. main's top-level permissions
and SHA-pinned action refs applied across all jobs.
- All other files merged cleanly (only pull-requests.yaml conflicted).
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Removes the `Release E2E` workflow
(`.github/workflows/release-e2e.yaml`), which ran the full bats suite on
every release tag.
**Why it's redundant.** `release-e2e.yaml` was a non-blocking alarm —
its own header states *"a failure here is an alarm, not a release
blocker."* The coverage it provided is already delivered by the
release-cut PR: the "Prepare release" commit bakes image digests into
`packages/core/` (installer, platform, testing) and other packages, and
`select-e2e.sh`'s full-suite pattern escalates any `packages/core/`
change to the **full** suite. So the release PR already runs the whole
suite against the commit the tag will point at.
**Why now.** The workflow has never been green since it was added
(2026-05-02) — every tag run (`v1.5.0-rc.2`, `-rc.3`, `v1.5.0`) failed
at the build step. Even after #2973 fixed the registry login earlier
today, the next run still failed at *Build Talos image*. A
perpetually-red, non-gating duplicate is pure noise.
**Changes**
- Delete `.github/workflows/release-e2e.yaml`.
- Drop `release-e2e` from the full-suite pattern in
`hack/select-e2e.sh`.
- Repoint the workflow-trigger unit test in `hack/select-e2e_test.bats`
to `pull-requests.yaml` (the remaining E2E workflow), preserving
"workflow edit → full suite" coverage. All 11 `select-e2e_test.bats`
cases pass.
- Update `docs/agents/e2e-testing.md` to describe the release-cut PR as
where the full suite runs.
Refs #2397.
### Release note
```release-note
ci(tests): remove the redundant Release E2E workflow; the full E2E suite already runs on the release-cut PR
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Removed automated release end-to-end testing workflow
* Updated E2E test selection logic to reflect testing infrastructure
changes
* **Documentation**
* Updated E2E testing procedures documentation to reflect current
testing approach
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The "Pin the e2e management cluster Kubernetes version" item is no longer
in-flight — it is merged and live in hack/e2e-prepare-cluster.bats
(--kubernetes-version v1.33.12). Remove it from the "not yet the merged
standard" section; its rationale lives in the bats comment.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The "What TIA does and does not do" section described the old single
Build job and the `detect-changes` paths-filter. Update it for the new
`plan` job (docs gate + build matrix), note that image builds are now
diff-scoped by hack/build-matrix.sh (distinct from TIA's app-loop trim),
and point the "widen the skip" advice at the plan docs-gate.
Refs #2937
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The Release E2E workflow (`release-e2e.yaml`) ran the full bats suite on
every release tag as a non-blocking alarm — its own header notes "a
failure here is an alarm, not a release blocker." It duplicates coverage
the release-cut PR already provides: the "Prepare release" commit bakes
image digests into `packages/core/` (and other packages), which TIA
escalates to the full suite, so the release PR exercises the whole suite
against the commit the tag will point at.
It has also never been green since it was added (2026-05-02): every tag
run failed at the build step. Even after #2973 fixed the registry login
today, the next run still failed (Build Talos image). Removing a
perpetually-red, non-gating duplicate removes noise without losing
coverage.
Removes the workflow, drops `release-e2e` from `select-e2e.sh`'s
full-suite pattern, repoints its unit test to `pull-requests.yaml`, and
updates the e2e-testing doc.
Refs #2397
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Capture the e2e stabilization conventions established over the last few
months as a routed agent doc: no retries on deterministic CI steps,
event-driven backstops over fixed sleeps, no EXIT/RETURN traps,
fail-fast HelmRelease-readiness gating, the status.history remediation
guard, first-class failure diagnostics, and a reviewer checklist. Also
documents Test-Impact Analysis (TIA) and its real scope — it trims only
the per-app loop, runs after install, and most PRs escalate to the full
suite. Wire it into AGENTS.md so every agent reads it before e2e/CI work.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Aleksei Sviridkin