Editing hack/e2e-chainsaw/backup/chainsaw-test.yaml.disabled selects the
whole Chainsaw suite. The per-suite rule reads the suite name off the
directory and ignores the suffix, so the change selects "backup"; the
final intersection against the suites that actually exist finds no such
suite and empties; and the safety net for an empty selection escalates to
the full run.
A disabled suite is registered nowhere and executed by nothing, which
makes its edits the one file class that provably cannot regress a test.
The mapping had it backwards: they bought the most expensive outcome the
selector has. Rare in practice, and paid on every commit by anyone
grooming or re-enabling such a suite.
Classify *.disabled as inert ahead of the per-suite rule, and record it
alongside the other rules that select nothing so the header stays an
accurate enumeration. The safety net keeps its job for paths no rule has
looked at, which is what it was added for.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Three holes on the same seam: the script builds a list, never looks at
whether the command that built it succeeded, and then decides from the
empty result.
The dependency indexes are built as `$(build_owners_index | sort -u)`,
where the function is a single yq. A pipeline carries the status of its
last command, so the script sees sort's zero and `set -e` never fires: a
missing binary or a malformed source leaves the index empty, every changed
path falls through to escalation, and CI runs the full Chainsaw suite on
every pull request. The direction is safe and that is the problem — an
empty index and "no owners matched" print the same thing, so the symptom
is a full run, which reads as the conservatism the selector is supposed to
apply. Selection can stay switched off indefinitely with nobody signalled.
Check each half's status on its own and name yq once on stderr before
taking the fallback the empty index would have reached anyway.
The suite list one line up has the identical shape and a worse
consequence: `$(find ... | sed | sort)` reports sort, never find, and
errors went to /dev/null. That list is what every escalation prints. So a
find that failed produced an empty list, and "escalate to the full suite"
then emitted a blank line — which both e2e lanes read as "skip Chainsaw"
before posting the required "E2E Tests" status green. The one outcome that
means run everything delivered a run of nothing, on the pull request that
earned a full run. Capture find on its own, check it, let its errors
through.
A failing find is not the only way to end up with an empty list, and the
escalations are not its only reader, so the check sits once where the list
is built. The examples/backups/<app>/ rule escalates nothing and still
membership-tests against that list: with it empty the test fails, the app
is not selected, and an edit to a backup harness that should run its suite
reports nothing to run — output identical to a legitimately empty
selection, on a path no escalation guard would ever see. An empty list is
a broken enumeration, a moved directory or a wrong working directory, not
a project without tests. Refuse to answer at all. Both lanes run the step
under `bash -e`, so a non-zero exit fails the job rather than falling
through to the empty selection this is here to prevent.
Each is covered by a test that stubs the failing component and asserts
both the exit status and that the script names what broke.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Three of the four workflows on the escalation list cannot be exercised
by the run they trigger. `e2e-fork` runs on `workflow_run`, which always
takes the default-branch copy; `e2e-tag` runs on `workflow_call` from a
tag push; `nightly` runs on a schedule. Only `pull-requests.yaml` is
executed from the pull request's own head, so only there does escalating
test the edit rather than the tree around it.
That argument existed in the reference doc and was removed by this
branch rather than answered, and the list it argued against is what
replaced it. Measured over six months of main, 26 commits touch one of
the three with no other escalating path; each now pays a full Chainsaw
run it used to skip, on a suite whose flake rate is the reason people
avoid running it for nothing.
Keeping them is defensible: the rule stays one idea rather than two, and
dropping them leaves those files inert, which reads as an oversight
rather than a decision. What was not defensible was paying it silently.
The trade now sits next to the pattern that makes it, with the number,
so the next person weighing it starts from measurement instead of
rediscovering the mechanism. The reference doc points there rather than
carrying its own copy.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Four places said an empty selection means the inert pattern matched, and
sent the reader to `inert_config_pattern` to see which paths those are.
Documentation, dashboards and `*.md` are dropped before that pattern is
ever consulted, and a backups example whose app has no suite is dropped
where backup examples are matched. A reader checking the pattern for any
of them finds nothing and has to reconstruct the order by reading the
script. Name the rules that produce an empty selection instead, and keep
the claim to the one the lanes rely on: empty is a decision some rule
reached, not a path nothing looked at.
The same bullet claimed every path is classified. One is not. The inert
pattern makes `.github/` inert as a whole directory and the workflows
that run the suite are exempted from it by name, so a workflow added
under `.github/workflows/` reads as inert and the fall-through never
sees it. Today's enumeration is correct and I checked it against `rg -l
test-chainsaw`, so this is not a live hole, but the sentence saying it
cannot exist is what makes the reminder next to that list look
decorative. Record it as the guard it is.
Separately, the fork lane's rationale for appending a newline described
a mechanism this branch removes: the reader now guards its own loop, so
the terminator is no longer what keeps the last path alive. Say that it
is now the text-file contract and a second line of defence across two
files, which is a reason to keep it and a different reason than the one
written there.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Four places on this branch describe an empty selection as "classified
inert", against a list of three outcomes that every path is said to land
in. There is a fourth: an edit under examples/backups/<app>/ where the
app has no Chainsaw suite selects nothing, decided inline where backup
examples are matched rather than through the inert list. The rule is
deliberate and explained where it lives; it just is not one of the three
the summary enumerates.
The property the two lanes actually depend on is weaker and true: an
empty selection means some rule decided the change cannot affect a
suite, never that nothing looked at the path. State that instead, in the
header that defines the classes and in the privileged lane's comment
that consumes the result.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The classifier this branch introduces is only total if every line
reaches the loop that applies it. POSIX read assigns the final line and
then returns non-zero when the input does not end in a newline, so a
plain `while read` drops it. The fall-through that escalates an
unrecognised path lives inside the loop body, so the dropped line is
never classified at all and the selection comes back empty. Both lanes
read empty as "skip Chainsaw" and post the required status green.
The shape that matters is a list whose only unclassified path is last:
`docs/x.md` followed by an unrecognised path with no terminator selects
nothing, where the same list with a terminator escalates to all 21
suites. The safety net this branch is built around is bypassed by
exactly the input it exists to catch.
No caller triggers it today, because `git diff --name-only` terminates
its output and the fork lane appends a newline explicitly. That makes it
caller discipline documented in one of two callers, which the next
caller will not read. Guard it where the assumption lives.
Both new cases are red without the guard and the twenty-five existing
ones stay green, so they pin this behaviour rather than the loop's
general shape. They clean up inline instead of through an EXIT trap:
that trap replaces the one the bats binary installs for its own
bookkeeping, and a test failing under it can print no TAP line at all,
which is the opposite of what a regression pin is for.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
`select-e2e.sh` silently selected nothing for a changed path it did not
recognise outside `packages/`. Both e2e lanes read an empty selection as
"skip Chainsaw" and then post the required "E2E Tests" status green, so
an unrecognised path produced a green merge gate with no suite run. The
worked example is `hack/common-envs.mk`, which supplies the tag, push
and output flags of every image in the tree and matched nothing:
`full_suite_pattern` covered `hack/*.sh` and `hack/*.bats` but not
`hack/*.mk`. `pkg/`, `tools/`, `go.mod`, `hack/lib/` and
`hack/buildkitd.toml` were in the same position, as was
`.github/workflows/e2e-fork.yaml` while `pull-requests.yaml` was
already escalated.
A blanket "empty means full suite" fallback would be the wrong fix: it
escalates the inert paths the design deliberately skips. Instead every
path now lands in one of three classes by an explicit rule — the
full-suite list for shared build inputs and the e2e harness, an inert
list for paths that cannot affect what e2e exercises, and the
PackageSource graph for `packages/(apps|system|extra)/`. A path
matching none of them escalates and logs why, so the next path someone
adds fails safe instead of failing open.
The two lists are checked in that order, so a specific escalation beats
a broad inert directory: `.github/` is inert, the workflows that run
the suite are not. `*.md` and `docs/`/`dashboards/` stay ahead of both,
so a README under `packages/core/` does not inherit that tree's
escalation.
Ten tests cover both sides of the classification. Mutation-checked:
reverting the fall-through fails the unclassified-path test, dropping
`.github/` from the inert list fails the non-e2e-workflow test, and
swapping the two list checks fails the existing pull-requests.yaml
test. Note the per-entry escalations are belt-and-braces — removing
`hack/*.mk` from the pattern still escalates via the fall-through,
which is the point of having it.
Closes#3392
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Ungate postgres-2-backup-roundtrip: instead of an opt-in env flag it now
runs whenever TIA selects the postgres suite (postgres changes, and every
release cut), like the other chainsaw tests. The blocker was network
placement, not capability: the S3 client is the barman-cloud sidecar
inside the Postgres Pod, and the isolated e2e tenant's Cilium egress
allowlist blocks it from tenant-root's seaweedfs — so the Test runs in
tenant-root (same-tenant egress) against the in-cluster S3 endpoint,
trusting its self-signed cert via the copied seaweedfs-system-ca-cert.
This is exactly the recipe validated end to end on a real cluster.
Also map examples/backups/<app>/ edits to that app's suite in
hack/select-e2e.sh (the round-trip executes those scripts as its
harness; previously such edits were silently ignored), with bats
coverage for both the mapped and the no-suite case.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Main added securitygroup.bats (#2922 membership model) and
serviceexposure.bats (#3081 ExposureClass/ServiceExposure) after the
Chainsaw migration branch was cut.
- securitygroup: two Tests — the aggregated-API projection round-trip
(backing CNP marker label, empty-value membership endpointSelector,
lineage fromApp + membership fromSG peers, 1:1 rule translation, the
view hiding storage internals, unmarked-CNP invisibility via an error
op, delete propagation) and the controller membership-label
stamp/clear lifecycle. Only the attachments annotation stays a jq
script (it is a JSON string). Chainsaw auto-cleanup covers every
fixture even on mid-test failure — the bats had no teardown hook and
leaked its fixtures.
- serviceexposure: the externalIPs backend resolve/Ready/assignedIPs
path with the no-MetalLB-pool negative as an error op (ordered after
resolvedBackend so it cannot false-pass), and the finalizer-removal
test collapsed onto the delete op itself, which waits for full
disappearance.
- selection: kubernetes-application now also maps to the two OIDC
render-side suites (they exercise the same app chart — a gap main's
bats selection had too); the new dirs enter all_apps discovery
automatically. Selector unit tests extended and green (13/13 under
cozytest).
- README: suites list updated.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Reconcile the Chainsaw e2e migration with ~500 commits of main drift:
- keep main's run-kubernetes.sh helpers (kubectl_wait_retry, tenant
drain, LINSTOR pool-free wait, SC-fallback probe) under the branch's
_lib/ rename with the Chainsaw finally/snapshot deltas re-applied
- resolve modify/delete on 10 drifted hack/e2e-apps/*.bats as deleted;
their post-branch assertion drift is re-ported into the Chainsaw
suites in follow-up commits
- drop the 4 new main-side bats (kubernetes-oidc-*, securitygroup,
serviceexposure) from _lib/ relocation; ported as Chainsaw suites in
follow-up commits
- docs/agents/e2e-testing.md: merge main's plan-job TIA wording with
the Chainsaw suite phrasing
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## Problem
On an unlucky Flux reconcile order, a `*-application` PackageSource that
ships a `system/*-rd` component can install before the
`applicationdefinitions.cozystack.io` CRD exists. Every `*-rd` chart
renders an `ApplicationDefinition` (`cozystack.io/v1alpha1`); that CRD
is installed by the `application-definition-crd` component of
`cozystack.cozystack-engine`. When an `*-rd` HelmRelease renders first,
it hard-fails with `no matches for kind "ApplicationDefinition" in
version "cozystack.io/v1alpha1"`, and that cascades into a broad install
timeout — surfacing even on changes that have nothing to do with
ordering, because it is a latent race.
## Fix
Add `- cozystack.cozystack-engine` to the `dependsOn` of every
`*-application` source that ships a `system/*-rd` component (29
sources). `gateway-application` already carried this edge — it was added
earlier for the TenantGateway CRD, the same bug class, but only for that
one source. A new invariant test fails if any source with a
`system/*-rd` component is missing the edge.
## Why
The ordering was previously implicit, relying on Flux happening to
reconcile `cozystack-engine` before the `*-application` sources. Making
the edge explicit removes the race deterministically. No dependency
cycle is introduced: `cozystack-engine` depends only on `networking` and
`cert-manager` (plus keycloak in its oidc variant), none of which reach
back to any `*-application`.
## Test
- New generic guard `hack/check-applicationdefinition-crd-ordering.bats`
(auto-discovered by `make bats-unit-tests`, runs in CI via `make
unit-tests`): parses every source and fails if any PackageSource variant
with a `system/*-rd` component lacks the `cozystack.cozystack-engine`
edge — so future sources are covered, not just today's set.
- `helm unittest packages/core/platform`: 13 suites / 66 tests pass; the
source edits do not disturb existing assertions.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved dependency ordering across several platform packages to
ensure related components install in the correct sequence.
* Prevented end-to-end test selection from being overly broad when only
ordering-related dependencies change.
* **Tests**
* Added checks to catch missing dependency requirements and to verify
test selection behavior stays accurate.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Replaces the per-app BATS E2E suite (hack/e2e-apps/*.bats) with declarative
Kyverno Chainsaw tests under hack/e2e-chainsaw/, extending this branch's pilot
(postgres + bucket) to all 18 app suites.
- Declarative ports assert on status/conditions and concrete fields; the
repeated "timeout ... until kubectl get" + "kubectl wait" boilerplate
collapses into a single polling assert with structured failure diffs and
automatic events/describe/podLogs capture on failure.
- Inherently imperative suites keep their logic in script steps: openbao
init/unseal, kuberture split-horizon probes, vminstance, the gateway
admission/impersonation cases, and kubernetes-latest/previous, which wrap
the relocated hack/e2e-chainsaw/_lib/run-kubernetes.sh verbatim.
- Gateway tests derive the tenant apex from the namespace host label at
runtime, so they are host-independent.
CI: the e2e job now runs "chainsaw test hack/e2e-chainsaw/" via a new
test-chainsaw target and uploads the JUnit chainsaw-report.xml; the chainsaw
binary is added to the e2e-sandbox image. install-cozystack and test-openapi
stay BATS. The per-app 3-retry loop is dropped, since assertion polling
replaces the fixed-timeout flakiness it papered over.
Verified against a dev cluster (DB/app, storage and VM suites pass). Service
port assertions use the ports[*].port projection form because Chainsaw
v0.2.15 mis-evaluates JMESPath number-literal port filters.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Every *-application source declares dependsOn cozystack.cozystack-engine so
its *-rd HelmRelease waits for the ApplicationDefinition CRD to register
before it reconciles. That is a universal install-ordering edge, not a
behavioral one.
The test-impact selector reused the same dependsOn graph to pick which
e2e app suites to run for a diff. Because the engine transitively depends
on postgres-operator, keycloak and cert-manager, a change to any of those
fanned out through the engine to every app, defeating test-impact analysis.
Exclude cozystack.cozystack-engine as a propagation hub in the reverse-dep
walk so the engine stays reachable but no longer pushes selection to its
app-dependents. A change to the engine itself still triggers the full suite
via the existing no-app-descendants safety net.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The Release E2E workflow (`release-e2e.yaml`) ran the full bats suite on
every release tag as a non-blocking alarm — its own header notes "a
failure here is an alarm, not a release blocker." It duplicates coverage
the release-cut PR already provides: the "Prepare release" commit bakes
image digests into `packages/core/` (and other packages), which TIA
escalates to the full suite, so the release PR exercises the whole suite
against the commit the tag will point at.
It has also never been green since it was added (2026-05-02): every tag
run failed at the build step. Even after #2973 fixed the registry login
today, the next run still failed (Build Talos image). Removing a
perpetually-red, non-gating duplicate removes noise without losing
coverage.
Removes the workflow, drops `release-e2e` from `select-e2e.sh`'s
full-suite pattern, repoints its unit test to `pull-requests.yaml`, and
updates the e2e-testing doc.
Refs #2397
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
- Use grep -Fxq instead of -w when intersecting with all_apps so apps
with hyphenated names cannot prefix-match each other.
- Collapse build_owners_index and build_reverse_deps into single yq
invocations with -N. Output is identical to the per-file loop and
about 30x faster on the current 91 PackageSource files.
- Use paste -sd ' ' - on the two full-suite emit paths for consistency
with the post-walk join at line 147.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Three changes that compose into a single E2E policy:
1. **Test-impact analysis selector** (`hack/select-e2e.sh`).
Reads `packages/core/platform/sources/*.yaml` (the PackageSource
dependency graph also rendered by `cozypkg dot`) and emits the bats
files affected by a PR diff. Library/core/networking/CI changes,
shared `hack/*.sh` helpers, the install bats, and the
`pull-requests` / `release-e2e` workflows trigger the full suite.
Doc/dashboard/`*.md` diffs select nothing. Per-app bats edits
select only that app. Anything inside `packages/` with no graph
entry, or a system source whose graph has no `*-application`
descendants, falls back to the full suite (conservative).
2. **Default-on TIA in `pull-requests.yaml`** with `full-e2e`
opt-out. Drops the previous `e2e-tia` opt-in shape — the
selector's behaviour has been validated, so making it the default
gives PR runtime savings out of the box. PRs that need the full
suite (e.g. cross-cutting refactors that the selector can't
reason about) carry the `full-e2e` label and get the unchanged
pre-TIA flow. Docs-only PRs (selector emits nothing) skip the
`Run E2E tests` step entirely.
3. **`release-e2e.yaml`** runs the full suite on every tag push
(`v*`). Independent of `tags.yaml`'s release publish flow — a
failure here is an alarm, not a release blocker. Closes the
coverage gap left by making PR runs use TIA: the full suite is
now exercised against shipped code at tag-cut time.
Bug fixes vs. the original `feat/select-e2e-script` draft:
- `dashboard/` -> `dashboards/` (path is plural in this repo).
- `hack/e2e-apps/<name>.sh` and `hack/<name>.bats` were silently
ignored; they now trigger the full suite (test infrastructure
affecting all apps).
- A path matching the graph but yielding zero `*-application`
descendants now falls back to the full suite instead of silently
emitting nothing (system-only changes are no longer silently
skipped).
- Per-app bats edits are matched before the full-suite pattern, so
editing one bats file no longer escalates to the full suite.
- Removed the dead `skip_pattern` variable (PCRE syntax that
`grep -qE` doesn't support; it was never matched).
Tests: `hack/select-e2e_test.bats` covers all 11 cases (the original
6 plus 5 for the bug fixes above).
Workflow inputs are passed via env vars rather than direct `${{ }}`
expansion into shell commands, per workflow injection guidance.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
Aleksei Sviridkin