admin/cozystack
1
0 Fork 0
mirror of https://github.com/cozystack/cozystack.git synced 2026-11-08 21:14:56 +09:00
Code Issues Packages Projects Releases Wiki Activity
Files
1b52832f37251b2d40b9fb917669de776cabbbcd
cozystack/hack/select-e2e_test.bats
T
Aleksei Sviridkin 9a0c8c75f3 fix(ci): close the silent-green gaps in e2e selection and the helm unit sweep (#3670) 
## What this PR does

Four fixes in e2e test selection and the helm unit sweep. Relates to
#3650 #3652 #3655 #3646.

select-e2e built both dependency indexes as `$(build_owners_index | sort
-u)`, so it read sort's exit status and never yq's (#3650). A broken yq
left the index empty, every path fell through to escalation, and CI ran
the full suite on every PR. The output was identical to "no owners
matched", so it looked like the conservatism the selector is supposed to
apply.

The suite list one line up had the same shape and a worse result.
`$(find ... | sed | sort)` reports sort, never find, and errors went to
/dev/null. Every escalation prints that list, so a failed find made
"escalate to the full suite" emit a blank line, which both e2e lanes
read as "skip Chainsaw" before posting `E2E Tests` green. The outcome
that means run everything delivered a run of nothing.

find is not the only way to get an empty list, and the escalations are
not its only reader, so the check sits where the list is built. The
`examples/backups/<app>/` rule escalates nothing and still
membership-tests against that list: with it empty, an edit to a backup
harness that should run its suite reports nothing to run, on a path no
escalation-side guard would ever see. An empty list is a broken
enumeration, so the script refuses to answer at all. Both lanes run the
step under `bash -e`, so the non-zero exit fails the job. I checked both
workflows: neither overrides `shell:`, both capture the selector as a
plain assignment.

Fourteen tests used `[ "$(echo "$output" | wc -w)" -gt 5 ]` to mean
"everything was selected" (#3652). There are 21 suites, so that passes
on any six, and partial escalation stayed green. They compare against
the whole list now, derived the way the script's own full-suite branch
derives it. Drop one suite from that branch and thirteen of them go red.
All thirteen passed under the threshold.

The fourteenth is the cilium case, and equality shows it was never a
full-suite path at all. It resolves through the dependency graph, which
reaches 19 of the 21 suites. The two it misses are kuberture and
securitygroup, whose sources are not `*-application`, and the mapping
covers only those plus external-dns by name. Both controllers declare
`cozystack.networking`, so a CNI change that breaks either one runs
neither suite. That gap is #3665. The test is renamed for what it
measures and pins the 19, so closing #3665 turns it red at the right
moment.

Editing `hack/e2e-chainsaw/backup/chainsaw-test.yaml.disabled` selected
the full suite (#3655). The suffix was ignored when deriving the suite
name, the intersection against existing suites emptied, and the
empty-selection safety net escalated. A disabled suite runs nowhere, so
its edits are the one file class that cannot regress a test, and they
were buying the most expensive outcome the selector has.

The selector now has a fourth outcome, a failed step rather than a
selection, so the contract enumerations in the script header, in
`docs/agents/e2e-testing.md` and in both workflow comments say so.

The helm unit sweep never visited `packages/tests` (#3646).
cozy-lib-tests, the whole test coverage for a helper library that 34
charts consume by symlink, ran only because
`packages/library/cozy-lib/Makefile` delegates its `test` target to it.
Remove that one line and the suite goes from one execution to zero with
the sweep still exiting 0 and still printing "All Helm unit tests
passed". Every check the sweep makes runs inside a directory the loop
already reached, so a package that never enters the loop takes all of
them with it, and what is left is `tests_found`, which counts suites
across the whole run rather than per package. The directory is on the
list now: 76 executed suites become 77, at 0.1s to 0.3s over six runs
against a sweep of roughly 31 seconds. The delegation stays for `make -C
packages/library/cozy-lib test` ergonomics, so the suite runs twice.
Neither run is load-bearing alone.

### Testing

`hack/select-e2e_test.bats` goes from 27 tests to 32,
`hack/helm-unit-tests.bats` is new. Each commit is green on its own: 27,
30, 32, 32.

The new tests stub the failing component and assert the exit status plus
the line naming what broke: a broken yq, a broken find, an empty suite
list across all three diff classes, the disabled-suite path, and a guard
that the disabled rule drops only that path and not the rest of the
diff. The yq test asserts exit 0 on purpose. A missing dependency graph
still has a correct answer, the full suite, and a missing suite list
does not. Pinning both halves keeps the header and the docs from
drifting into claiming otherwise.

Two suites fail on my macOS box and fail identically at `origin/main`,
so neither comes from this branch:
`hack/migration-seaweedfs-db-adopt.bats`, where a container running as
the host UID cannot write the bind-mounted temp dir under Colima, and
`hack/seaweedfs-naming-audit.bats`, which I confirmed red at the merge
base but did not dig into. Every other `hack/*.bats` was run
individually and is green. `make unit-tests` stops at the first failing
file and both of those sort ahead of `select-e2e_test.bats`, so on macOS
the suites this PR touches have to be run directly.

### Named improvements, not done here

- The `.disabled` rule keys on the file rather than the parked
directory, so it is both wider and narrower than the property it wants.
A support file under `hack/e2e-chainsaw/backup/` still buys a full run,
and a `.disabled` file dropped inside a live suite directory now selects
nothing where it used to select that suite. The property covering both
is one level up: a suite directory holding no live `chainsaw-test.yaml`
is inert, testable with the same `grep -Fxq` membership check the
`examples/backups/*/*` arm already uses. It is not a free win, since a
half-written suite directory escalates today and would then select
nothing. A cheaper half step is narrowing the glob to
`hack/e2e-chainsaw/*/chainsaw-test.yaml*.disabled`, because case globs
cross `/` and the shipped pattern also matches nested paths.
- `packages/tests/**` is not classified by the selector and reaches the
full suite through the unclassified fall-through instead of a rule.
Tracked in #3667.
- `full_suite_list()` in the bats file cannot fail. cozytest.sh's awk
rewrites a lone `}` into `return 0; }`, so a broken find inside the
helper would return an empty expected set instead of a red test. Nothing
false can pass today, because the script hard-fails on an empty suite
list before any assert runs, but that helper is what every equality
assert measures against.
- The yq check runs before the changed-file list is read, so with a
broken graph even a docs-only diff escalates. Measured: a broken yq plus
a `docs/x.md` diff returns 21 suites at head against 0 at the merge
base. Only reachable in an already broken state, but the graph is
consulted at the component lookup and in the closure walk, so moving the
check to first use would charge nothing to docs, dashboards and
inert-meta diffs.

### Screenshots

No UI changes.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` file by file
against the diff: four files under `hack/`, one under `docs/agents/`,
and comments in the two e2e workflows. Nothing under `hack/` was moved
or renamed and no make-target contract changed, so the ccp entry does
not fire. It gates on `hack/package.mk` and `hack/common-envs.mk`,
neither of which is in the diff. The node prerequisites in
`hack/e2e-prepare-cluster.bats` that ansible-cozystack mirrors by hand
are untouched, and so is `.github/workflows/tags.yaml`. `docs/agents/`
is internal contributor documentation, not a website source. No package
was added, renamed or removed, and no `values.schema.json` or version
enum changed, so the website and provider entries do not fire either.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(ci): select-e2e no longer reports an empty selection when it cannot enumerate the Chainsaw suites, which made an escalation to the full suite indistinguishable from "skip Chainsaw" and left the E2E gate green with nothing run. A failure to list the suites is now named on stderr and fails the step, a yq failure is named and still escalates to the full suite, and an edit to a disabled Chainsaw suite is inert instead of escalating to a full run.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved end-to-end test selection for disabled, empty, or partially
changed test suites.
* Test selection now safely escalates to the full suite when suite
discovery or dependency analysis fails.
  * Helm unit-test sweeps now include tests under `packages/tests`.

* **Tests**
* Added coverage for suite discovery failures, disabled suites, command
failures, and exact test selection.
  * Added Helm test-sweep coverage, including failure handling.

* **Documentation**
  * Clarified test-impact analysis rules and suite-selection outcomes.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-08 20:47:06 +03:00

500 lines
21 KiB
Bash
Raw Blame History

#!/usr/bin/env bats
# EXIT-TRAP DEBT: 15 -- see hack/bats-no-exit-trap.bats; lower it as the traps go, delete it at zero.
# -----------------------------------------------------------------------------
# Unit tests for hack/select-e2e.sh
#
# cozytest.sh's awk parser recognizes only @test blocks and a bare `}` on its
# own line; there is no bats `run` or `$status`. Each test runs as a shell
# function under `set -eu -x`, so assertions are direct shell tests that exit
# non-zero on failure. setup()/teardown() are not honored — each test creates
# and cleans its own scratch dir.
#
# Run with: hack/cozytest.sh hack/select-e2e_test.bats
# -----------------------------------------------------------------------------
# Assert that a selection is the WHOLE suite list, not merely a long one.
#
# These asserts used to read `[ "$(echo "$output" | wc -w)" -gt 5 ]`, which is
# satisfied by any selection of six or more suites. The regression worth
# catching on every escalation path is partial escalation — a selector bug that
# picks most suites but not all — and a threshold cannot see it: with 21 suites
# in the tree, dropping fifteen of them still passes. Equality can.
#
# The expected set is derived the way the script's full-suite branch derives it
# rather than pinned as a literal, so adding or disabling a Chainsaw suite does
# not need an edit in fourteen places here.
#
# A helper rather than an inline one-liner because cozytest.sh runs each @test
# under `set -x`: a bare failing `[ ... ]` prints the two values already
# expanded, but not which side is which, and reading a 21-item diff off a trace
# line is exactly the moment a test stops being worth having.
full_suite_list() {
find hack/e2e-chainsaw -mindepth 2 -maxdepth 2 -name chainsaw-test.yaml \
| sed -e 's,^hack/e2e-chainsaw/,,' -e 's,/chainsaw-test\.yaml$,,' | sort | paste -sd ' ' -
}
assert_selection() {
if [ "$2" != "$3" ]; then
echo "$1" >&2
echo " want: $3" >&2
echo " got: $2" >&2
exit 1
fi
}
assert_full_suite() {
assert_selection "expected the full Chainsaw suite" "$1" "$(full_suite_list)"
}
@test "single app diff selects only that suite" {
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
cp -r packages/core/platform/sources "$tmp/sources"
echo "packages/apps/postgres/values.yaml" > "$tmp/diff"
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
[ "$output" = "postgres" ]
}
@test "operator diff selects all dependent app suites" {
# postgres-operator is depended on by postgres-application, harbor-application
# (Harbor uses postgres as its backing DB), and monitoring-application (Grafana
# DB). monitoring has no chainsaw suite so it's filtered out by the selector.
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
cp -r packages/core/platform/sources "$tmp/sources"
echo "packages/system/postgres-operator/values.yaml" > "$tmp/diff"
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
echo "$output" | grep -wq postgres
echo "$output" | grep -wq harbor
if echo "$output" | grep -wq kafka; then
echo "operator diff must not trigger full suite; got: $output" >&2
exit 1
fi
}
@test "engine-dependency change does not fan out via the ordering edge" {
# cert-manager is a dependency of cozystack-engine, and every app declares
# dependsOn cozystack-engine purely as an INSTALL-ORDERING edge (the app's
# *-rd HelmRelease waits for the ApplicationDefinition CRD). That edge must
# not propagate test selection: a cert-manager change selects only its
# genuine direct dependents (postgres, harbor, ...), never unrelated apps
# like kafka that reach cert-manager solely through the engine.
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
cp -r packages/core/platform/sources "$tmp/sources"
echo "packages/system/cert-manager/values.yaml" > "$tmp/diff"
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
echo "$output" | grep -wq postgres
echo "$output" | grep -wq harbor
if echo "$output" | grep -wq kafka; then
echo "cert-manager change must not fan out via engine; got: $output" >&2
exit 1
fi
}
@test "a CNI change selects every suite the graph can reach" {
# Named for what it measures rather than for the full suite it used to claim.
# `packages/system/cilium` is owned by cozystack.networking and resolves
# through the dependency graph, not through full_suite_pattern, so the
# selection is "every suite reachable from networking" — which is 19 of the
# 21 that exist, not all of them.
#
# The two it cannot reach are kuberture and securitygroup. select-e2e.sh maps
# a source to suites only for *-application names, plus external-dns by
# name; cozystack.kuberture and cozystack.securitygroup-controller are
# neither, so the walk reaches them and the filter drops them. Both declare
# cozystack.networking as a dependency, so this is the selector under-
# selecting, not the graph being right — a CNI change that breaks either
# controller runs neither of their suites. Tracked in #3665. Subtracted here
# rather than papered over: closing that gap turns this test red, which is
# the correct moment to notice — the two names below are a measurement of
# today's selector, not a typo. The previous `wc -w -gt 5` assert passed on
# 19 and said nothing.
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
cp -r packages/core/platform/sources "$tmp/sources"
echo "packages/system/cilium/values.yaml" > "$tmp/diff"
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
reachable=$(full_suite_list | tr ' ' '\n' | grep -vxE 'kuberture|securitygroup' | paste -sd ' ' -)
assert_selection "expected every graph-reachable suite" "$output" "$reachable"
}
@test "library change triggers full suite" {
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
cp -r packages/core/platform/sources "$tmp/sources"
echo "packages/library/cozy-lib/templates/_helpers.tpl" > "$tmp/diff"
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
assert_full_suite "$output"
}
@test "docs-only diff selects nothing" {
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
cp -r packages/core/platform/sources "$tmp/sources"
echo "docs/README.md" > "$tmp/diff"
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
[ -z "$output" ]
}
@test "kubernetes-application maps to the four kubernetes suites" {
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
cp -r packages/core/platform/sources "$tmp/sources"
echo "packages/apps/kubernetes/values.yaml" > "$tmp/diff"
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
echo "$output" | grep -q "kubernetes-latest"
echo "$output" | grep -q "kubernetes-previous"
# The OIDC render-side suites exercise the same kubernetes app chart, so a
# chart-only change must select them too.
echo "$output" | grep -q "kubernetes-oidc-system"
echo "$output" | grep -q "kubernetes-oidc-customconfig"
}
@test "dashboards-only diff selects nothing (path is plural)" {
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
cp -r packages/core/platform/sources "$tmp/sources"
echo "dashboards/gpu/gpu-fleet.json" > "$tmp/diff"
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
[ -z "$output" ]
}
@test "shared E2E helper script triggers full suite" {
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
cp -r packages/core/platform/sources "$tmp/sources"
echo "hack/e2e-chainsaw/_lib/run-kubernetes.sh" > "$tmp/diff"
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
assert_full_suite "$output"
}
@test "chainsaw config change triggers full suite" {
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
cp -r packages/core/platform/sources "$tmp/sources"
echo "hack/e2e-chainsaw/.chainsaw.yaml" > "$tmp/diff"
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
assert_full_suite "$output"
}
@test "install bats triggers full suite" {
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
cp -r packages/core/platform/sources "$tmp/sources"
echo "hack/e2e-install-cozystack.bats" > "$tmp/diff"
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
assert_full_suite "$output"
}
@test "per-suite edit selects only that suite, never escalates" {
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
cp -r packages/core/platform/sources "$tmp/sources"
echo "hack/e2e-chainsaw/redis/chainsaw-test.yaml" > "$tmp/diff"
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
[ "$output" = "redis" ]
}
@test "pull-requests workflow change triggers full suite" {
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
cp -r packages/core/platform/sources "$tmp/sources"
echo ".github/workflows/pull-requests.yaml" > "$tmp/diff"
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
assert_full_suite "$output"
}
@test "backup example harness edit selects its app suite" {
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
cp -r packages/core/platform/sources "$tmp/sources"
echo "examples/backups/postgres/run-all.sh" > "$tmp/diff"
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
[ "$output" = "postgres" ]
}
@test "backup example without a matching suite selects nothing" {
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
cp -r packages/core/platform/sources "$tmp/sources"
echo "examples/backups/no-such-app/run.sh" > "$tmp/diff"
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources") || true
[ -z "$output" ]
}
# --- #3392: every path is classified; unclassified escalates -----------------
#
# The bug these cover: an unrecognised path used to select nothing, both lanes
# read an empty selection as "skip Chainsaw", and the required "E2E Tests"
# status was then posted green with no suite run. Each test below pins one side
# of the classification — escalate, or skip by an explicit rule.
@test "hack/*.mk triggers full suite (build flags of every image)" {
tmp=$(mktemp -d)
cp -r packages/core/platform/sources "$tmp/sources"
echo "hack/common-envs.mk" > "$tmp/diff"
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
assert_full_suite "$output"
rm -rf "$tmp"
}
@test "the fork e2e workflow triggers full suite" {
tmp=$(mktemp -d)
cp -r packages/core/platform/sources "$tmp/sources"
echo ".github/workflows/e2e-fork.yaml" > "$tmp/diff"
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
assert_full_suite "$output"
rm -rf "$tmp"
}
@test "pkg/ triggers full suite (shipped Go code)" {
tmp=$(mktemp -d)
cp -r packages/core/platform/sources "$tmp/sources"
echo "pkg/cozystack/registry.go" > "$tmp/diff"
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
assert_full_suite "$output"
rm -rf "$tmp"
}
@test "go.mod triggers full suite" {
tmp=$(mktemp -d)
cp -r packages/core/platform/sources "$tmp/sources"
echo "go.mod" > "$tmp/diff"
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
assert_full_suite "$output"
rm -rf "$tmp"
}
@test "hack/lib helper triggers full suite" {
tmp=$(mktemp -d)
cp -r packages/core/platform/sources "$tmp/sources"
echo "hack/lib/image-refs.sh" > "$tmp/diff"
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
assert_full_suite "$output"
rm -rf "$tmp"
}
@test "an unclassified path escalates instead of selecting nothing" {
tmp=$(mktemp -d)
cp -r packages/core/platform/sources "$tmp/sources"
echo "brand-new-top-level/thing.conf" > "$tmp/diff"
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources" 2>/dev/null)
assert_full_suite "$output"
rm -rf "$tmp"
}
@test "inert repo meta selects nothing" {
tmp=$(mktemp -d)
cp -r packages/core/platform/sources "$tmp/sources"
printf '%s\n' .gitignore LICENSE .pre-commit-config.yaml > "$tmp/diff"
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
[ -z "$output" ]
rm -rf "$tmp"
}
@test "a non-e2e workflow selects nothing, an e2e one still escalates" {
tmp=$(mktemp -d)
cp -r packages/core/platform/sources "$tmp/sources"
# .github/ is inert as a directory, but the escalation for the workflows
# that run the suite is checked first and must win.
echo ".github/workflows/tags.yaml" > "$tmp/diff"
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
[ -z "$output" ]
echo ".github/workflows/e2e-tag.yaml" > "$tmp/diff"
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
assert_full_suite "$output"
rm -rf "$tmp"
}
@test "a README inside an escalating tree stays inert" {
tmp=$(mktemp -d)
cp -r packages/core/platform/sources "$tmp/sources"
# packages/core/ escalates, but *.md is matched before that so a doc edit
# under it does not burn a full run.
echo "packages/core/installer/README.md" > "$tmp/diff"
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
[ -z "$output" ]
rm -rf "$tmp"
}
@test "an inert path alongside a real one does not mask the selection" {
tmp=$(mktemp -d)
cp -r packages/core/platform/sources "$tmp/sources"
printf '%s\n' .gitignore packages/apps/postgres/values.yaml > "$tmp/diff"
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
[ "$output" = "postgres" ]
rm -rf "$tmp"
}
# The classification above is only total if every line reaches the loop that
# applies it. POSIX read assigns the last line and returns non-zero when the
# input has no trailing newline, so a plain `while read` drops it, and the
# fall-through that escalates an unrecognised path lives inside the body that
# drop skips. `git diff --name-only` always terminates its output, so the bug is
# invisible to a caller that pipes it; a caller assembling the list itself sees
# it immediately. Both cases below are red without `|| [ -n "$file" ]`.
#
# Cleanup here, as in every test added with this change, is the last statement
# of the body rather than a `trap ... EXIT`: that trap replaces the one the bats
# binary installs for its own bookkeeping, and a test failing under it can print
# no TAP line at all, which is the opposite of what a regression pin is for.
# Last, not before the assertion, so that `set -e` leaves the scratch directory
# behind on failure for inspection (docs/agents/e2e-testing.md §3).
@test "editing a disabled chainsaw suite selects nothing" {
# A .disabled suite runs nowhere, so its edits are the one file class that
# provably cannot regress a test — and before this rule they bought the
# most expensive outcome the selector has. The suffix was ignored when
# deriving the suite name, the derived name matched no existing suite, the
# final intersection came back empty, and the empty-selection safety net
# escalated to the full run.
tmp=$(mktemp -d)
cp -r packages/core/platform/sources "$tmp/sources"
echo "hack/e2e-chainsaw/backup/chainsaw-test.yaml.disabled" > "$tmp/diff"
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
[ -z "$output" ]
rm -rf "$tmp"
}
@test "a disabled suite alongside a live one does not mask the selection" {
# The inert rule must drop the disabled path only, not the whole diff: the
# cheap way to make the test above pass is a rule that returns early, and
# that would silently unselect real work committed in the same change.
tmp=$(mktemp -d)
cp -r packages/core/platform/sources "$tmp/sources"
printf '%s\n' hack/e2e-chainsaw/backup/chainsaw-test.yaml.disabled \
packages/apps/postgres/values.yaml > "$tmp/diff"
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
[ "$output" = "postgres" ]
rm -rf "$tmp"
}
@test "a broken yq is named rather than passed off as an empty graph" {
# Both indexes are one yq each, read as `$(build_owners_index | sort -u)`,
# so the pipeline reports sort's status and set -e never sees yq's. A
# missing binary or a malformed PackageSource then yields an empty index,
# every path falls through to escalation, and the run is a full suite that
# looks exactly like intended conservatism. Nothing distinguishes it from
# "no owners matched", so nobody is told the selector stopped working.
#
# The escalation is the right outcome and is asserted here too; what this
# pins is that it comes with a line naming yq.
#
# The exit status is asserted as ZERO on purpose, and it is the difference
# between this case and the two below it. What a broken yq costs is the
# dependency graph, and the full suite is a correct answer without one, so
# the script still answers. When the SUITE LIST is what broke there is no
# correct answer left to give and the script exits non-zero instead. Both
# halves are pinned so the header and docs cannot drift into claiming this
# one reddens the gate — it does not; it is conservative and quiet.
tmp=$(mktemp -d)
cp -r packages/core/platform/sources "$tmp/sources"
mkdir "$tmp/bin"
printf '#!/bin/sh\necho "yq: broken fixture" >&2\nexit 1\n' > "$tmp/bin/yq"
chmod +x "$tmp/bin/yq"
echo "packages/apps/postgres/values.yaml" > "$tmp/diff"
rc=0
output=$(PATH="$tmp/bin:$PATH" hack/select-e2e.sh "$tmp/diff" "$tmp/sources" 2>"$tmp/err") || rc=$?
if [ "$rc" -ne 0 ]; then
echo "a broken yq must still answer with the full suite, not exit $rc" >&2
cat "$tmp/err" >&2
exit 1
fi
# Matched on the script's own prefix, not on the word alone: the stub writes
# to stderr the way a real yq does, so a bare `grep yq` passes on the
# fixture's noise and pins nothing.
if ! grep -q 'select-e2e:.*yq' "$tmp/err"; then
echo "a yq failure must be reported by select-e2e itself; stderr was:" >&2
cat "$tmp/err" >&2
exit 1
fi
assert_full_suite "$output"
rm -rf "$tmp"
}
@test "a broken find is reported rather than yielding an empty suite list" {
# The suite list is built as `$(find ... | sed | sort)`, which reports
# sort's status and never find's — the same blindness as the yq indexes
# above, in a worse place: every escalation prints that list, so a silent
# failure here turns "run everything" into a run of nothing.
tmp=$(mktemp -d)
cp -r packages/core/platform/sources "$tmp/sources"
mkdir "$tmp/bin"
printf '#!/bin/sh\necho "find: broken fixture" >&2\nexit 1\n' > "$tmp/bin/find"
chmod +x "$tmp/bin/find"
echo "go.mod" > "$tmp/diff"
rc=0
output=$(PATH="$tmp/bin:$PATH" hack/select-e2e.sh "$tmp/diff" "$tmp/sources" 2>"$tmp/err") || rc=$?
if [ "$rc" -eq 0 ]; then
echo "a broken find must fail the run; it exited 0 with output: '$output'" >&2
exit 1
fi
if ! grep -q 'select-e2e:.*find' "$tmp/err"; then
echo "a find failure must be reported by select-e2e itself; stderr was:" >&2
cat "$tmp/err" >&2
exit 1
fi
rm -rf "$tmp"
}
@test "an empty suite list is fatal whatever the diff classifies as" {
# Checked once where the list is built, not at the escalation branches,
# because an empty list corrupts more than those. The escalations would
# print it, and a blank selection is what both lanes read as "skip
# Chainsaw" before posting the required status green — the outcome meaning
# "run everything" delivering a run of nothing. But the
# examples/backups/<app>/ rule escalates nothing and still consults the
# list as a membership test, so with the list empty a backup-harness edit
# that should run its suite reports nothing to run instead — and that one
# is indistinguishable from a legitimately empty selection.
#
# Run from a tree whose hack/e2e-chainsaw exists but holds no
# chainsaw-test.yaml, so find succeeds and matches nothing — the state a
# moved directory or a wrong working directory produces. All three diff
# classes are exercised: one that escalates, one that selects by
# membership, and one that legitimately selects nothing. The middle two
# exited 0 while the guard lived at the escalation branches.
tmp=$(mktemp -d)
script="$PWD/hack/select-e2e.sh"
cp -r packages/core/platform/sources "$tmp/sources"
mkdir -p "$tmp/tree/hack/e2e-chainsaw"
for d in go.mod examples/backups/postgres/run-all.sh docs/x.md; do
echo "$d" > "$tmp/diff"
rc=0
output=$(cd "$tmp/tree" && "$script" "$tmp/diff" "$tmp/sources" 2>"$tmp/err") || rc=$?
if [ "$rc" -eq 0 ]; then
echo "a broken suite enumeration must fail for '$d', not print '$output' and exit 0" >&2
cat "$tmp/err" >&2
exit 1
fi
if ! grep -q 'suite enumeration is broken' "$tmp/err"; then
echo "the refusal must say why for '$d'; stderr was:" >&2
cat "$tmp/err" >&2
exit 1
fi
done
rm -rf "$tmp"
}
@test "an unterminated last line is still classified" {
tmp=$(mktemp -d)
cp -r packages/core/platform/sources "$tmp/sources"
printf '%s' packages/apps/postgres/values.yaml > "$tmp/diff"
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
[ "$output" = "postgres" ]
rm -rf "$tmp"
}
@test "an unterminated unclassified last line still escalates" {
tmp=$(mktemp -d)
cp -r packages/core/platform/sources "$tmp/sources"
# The unclassified path is the one the drop would eat, so without the guard
# the escalation never fires and the selection comes back empty — the exact
# silent-green this classification exists to remove.
printf 'docs/x.md\n%s' brand-new/thing.conf > "$tmp/diff"
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
assert_full_suite "$output"
rm -rf "$tmp"
}
Reference in new issue View Git Blame Copy Permalink