Ungate postgres-2-backup-roundtrip: instead of an opt-in env flag it now
runs whenever TIA selects the postgres suite (postgres changes, and every
release cut), like the other chainsaw tests. The blocker was network
placement, not capability: the S3 client is the barman-cloud sidecar
inside the Postgres Pod, and the isolated e2e tenant's Cilium egress
allowlist blocks it from tenant-root's seaweedfs — so the Test runs in
tenant-root (same-tenant egress) against the in-cluster S3 endpoint,
trusting its self-signed cert via the copied seaweedfs-system-ca-cert.
This is exactly the recipe validated end to end on a real cluster.
Also map examples/backups/<app>/ edits to that app's suite in
hack/select-e2e.sh (the round-trip executes those scripts as its
harness; previously such edits were silently ignored), with bats
coverage for both the mapped and the no-suite case.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Main added securitygroup.bats (#2922 membership model) and
serviceexposure.bats (#3081 ExposureClass/ServiceExposure) after the
Chainsaw migration branch was cut.
- securitygroup: two Tests — the aggregated-API projection round-trip
(backing CNP marker label, empty-value membership endpointSelector,
lineage fromApp + membership fromSG peers, 1:1 rule translation, the
view hiding storage internals, unmarked-CNP invisibility via an error
op, delete propagation) and the controller membership-label
stamp/clear lifecycle. Only the attachments annotation stays a jq
script (it is a JSON string). Chainsaw auto-cleanup covers every
fixture even on mid-test failure — the bats had no teardown hook and
leaked its fixtures.
- serviceexposure: the externalIPs backend resolve/Ready/assignedIPs
path with the no-MetalLB-pool negative as an error op (ordered after
resolvedBackend so it cannot false-pass), and the finalizer-removal
test collapsed onto the delete op itself, which waits for full
disappearance.
- selection: kubernetes-application now also maps to the two OIDC
render-side suites (they exercise the same app chart — a gap main's
bats selection had too); the new dirs enter all_apps discovery
automatically. Selector unit tests extended and green (13/13 under
cozytest).
- README: suites list updated.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Reconcile the Chainsaw e2e migration with ~500 commits of main drift:
- keep main's run-kubernetes.sh helpers (kubectl_wait_retry, tenant
drain, LINSTOR pool-free wait, SC-fallback probe) under the branch's
_lib/ rename with the Chainsaw finally/snapshot deltas re-applied
- resolve modify/delete on 10 drifted hack/e2e-apps/*.bats as deleted;
their post-branch assertion drift is re-ported into the Chainsaw
suites in follow-up commits
- drop the 4 new main-side bats (kubernetes-oidc-*, securitygroup,
serviceexposure) from _lib/ relocation; ported as Chainsaw suites in
follow-up commits
- docs/agents/e2e-testing.md: merge main's plan-job TIA wording with
the Chainsaw suite phrasing
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## Problem
On an unlucky Flux reconcile order, a `*-application` PackageSource that
ships a `system/*-rd` component can install before the
`applicationdefinitions.cozystack.io` CRD exists. Every `*-rd` chart
renders an `ApplicationDefinition` (`cozystack.io/v1alpha1`); that CRD
is installed by the `application-definition-crd` component of
`cozystack.cozystack-engine`. When an `*-rd` HelmRelease renders first,
it hard-fails with `no matches for kind "ApplicationDefinition" in
version "cozystack.io/v1alpha1"`, and that cascades into a broad install
timeout — surfacing even on changes that have nothing to do with
ordering, because it is a latent race.
## Fix
Add `- cozystack.cozystack-engine` to the `dependsOn` of every
`*-application` source that ships a `system/*-rd` component (29
sources). `gateway-application` already carried this edge — it was added
earlier for the TenantGateway CRD, the same bug class, but only for that
one source. A new invariant test fails if any source with a
`system/*-rd` component is missing the edge.
## Why
The ordering was previously implicit, relying on Flux happening to
reconcile `cozystack-engine` before the `*-application` sources. Making
the edge explicit removes the race deterministically. No dependency
cycle is introduced: `cozystack-engine` depends only on `networking` and
`cert-manager` (plus keycloak in its oidc variant), none of which reach
back to any `*-application`.
## Test
- New generic guard `hack/check-applicationdefinition-crd-ordering.bats`
(auto-discovered by `make bats-unit-tests`, runs in CI via `make
unit-tests`): parses every source and fails if any PackageSource variant
with a `system/*-rd` component lacks the `cozystack.cozystack-engine`
edge — so future sources are covered, not just today's set.
- `helm unittest packages/core/platform`: 13 suites / 66 tests pass; the
source edits do not disturb existing assertions.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved dependency ordering across several platform packages to
ensure related components install in the correct sequence.
* Prevented end-to-end test selection from being overly broad when only
ordering-related dependencies change.
* **Tests**
* Added checks to catch missing dependency requirements and to verify
test selection behavior stays accurate.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Replaces the per-app BATS E2E suite (hack/e2e-apps/*.bats) with declarative
Kyverno Chainsaw tests under hack/e2e-chainsaw/, extending this branch's pilot
(postgres + bucket) to all 18 app suites.
- Declarative ports assert on status/conditions and concrete fields; the
repeated "timeout ... until kubectl get" + "kubectl wait" boilerplate
collapses into a single polling assert with structured failure diffs and
automatic events/describe/podLogs capture on failure.
- Inherently imperative suites keep their logic in script steps: openbao
init/unseal, kuberture split-horizon probes, vminstance, the gateway
admission/impersonation cases, and kubernetes-latest/previous, which wrap
the relocated hack/e2e-chainsaw/_lib/run-kubernetes.sh verbatim.
- Gateway tests derive the tenant apex from the namespace host label at
runtime, so they are host-independent.
CI: the e2e job now runs "chainsaw test hack/e2e-chainsaw/" via a new
test-chainsaw target and uploads the JUnit chainsaw-report.xml; the chainsaw
binary is added to the e2e-sandbox image. install-cozystack and test-openapi
stay BATS. The per-app 3-retry loop is dropped, since assertion polling
replaces the fixed-timeout flakiness it papered over.
Verified against a dev cluster (DB/app, storage and VM suites pass). Service
port assertions use the ports[*].port projection form because Chainsaw
v0.2.15 mis-evaluates JMESPath number-literal port filters.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Every *-application source declares dependsOn cozystack.cozystack-engine so
its *-rd HelmRelease waits for the ApplicationDefinition CRD to register
before it reconciles. That is a universal install-ordering edge, not a
behavioral one.
The test-impact selector reused the same dependsOn graph to pick which
e2e app suites to run for a diff. Because the engine transitively depends
on postgres-operator, keycloak and cert-manager, a change to any of those
fanned out through the engine to every app, defeating test-impact analysis.
Exclude cozystack.cozystack-engine as a propagation hub in the reverse-dep
walk so the engine stays reachable but no longer pushes selection to its
app-dependents. A change to the engine itself still triggers the full suite
via the existing no-app-descendants safety net.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The Release E2E workflow (`release-e2e.yaml`) ran the full bats suite on
every release tag as a non-blocking alarm — its own header notes "a
failure here is an alarm, not a release blocker." It duplicates coverage
the release-cut PR already provides: the "Prepare release" commit bakes
image digests into `packages/core/` (and other packages), which TIA
escalates to the full suite, so the release PR exercises the whole suite
against the commit the tag will point at.
It has also never been green since it was added (2026-05-02): every tag
run failed at the build step. Even after #2973 fixed the registry login
today, the next run still failed (Build Talos image). Removing a
perpetually-red, non-gating duplicate removes noise without losing
coverage.
Removes the workflow, drops `release-e2e` from `select-e2e.sh`'s
full-suite pattern, repoints its unit test to `pull-requests.yaml`, and
updates the e2e-testing doc.
Refs #2397
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
CI invokes hack/cozytest.sh, a project-local pure-shell harness, not
bats. cozytest does not honor setup()/teardown() and does not provide
bats' run / $status / $output, so the original tests failed with
"TMPDIR: parameter not set" under set -u.
Inline tmpdir creation and an EXIT trap for cleanup in each test.
Replace `run` + `$output` with direct command substitution, and
replace `! grep` (which set -e ignores) with an explicit if-then-exit.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Three changes that compose into a single E2E policy:
1. **Test-impact analysis selector** (`hack/select-e2e.sh`).
Reads `packages/core/platform/sources/*.yaml` (the PackageSource
dependency graph also rendered by `cozypkg dot`) and emits the bats
files affected by a PR diff. Library/core/networking/CI changes,
shared `hack/*.sh` helpers, the install bats, and the
`pull-requests` / `release-e2e` workflows trigger the full suite.
Doc/dashboard/`*.md` diffs select nothing. Per-app bats edits
select only that app. Anything inside `packages/` with no graph
entry, or a system source whose graph has no `*-application`
descendants, falls back to the full suite (conservative).
2. **Default-on TIA in `pull-requests.yaml`** with `full-e2e`
opt-out. Drops the previous `e2e-tia` opt-in shape — the
selector's behaviour has been validated, so making it the default
gives PR runtime savings out of the box. PRs that need the full
suite (e.g. cross-cutting refactors that the selector can't
reason about) carry the `full-e2e` label and get the unchanged
pre-TIA flow. Docs-only PRs (selector emits nothing) skip the
`Run E2E tests` step entirely.
3. **`release-e2e.yaml`** runs the full suite on every tag push
(`v*`). Independent of `tags.yaml`'s release publish flow — a
failure here is an alarm, not a release blocker. Closes the
coverage gap left by making PR runs use TIA: the full suite is
now exercised against shipped code at tag-cut time.
Bug fixes vs. the original `feat/select-e2e-script` draft:
- `dashboard/` -> `dashboards/` (path is plural in this repo).
- `hack/e2e-apps/<name>.sh` and `hack/<name>.bats` were silently
ignored; they now trigger the full suite (test infrastructure
affecting all apps).
- A path matching the graph but yielding zero `*-application`
descendants now falls back to the full suite instead of silently
emitting nothing (system-only changes are no longer silently
skipped).
- Per-app bats edits are matched before the full-suite pattern, so
editing one bats file no longer escalates to the full suite.
- Removed the dead `skip_pattern` variable (PCRE syntax that
`grep -qE` doesn't support; it was never matched).
Tests: `hack/select-e2e_test.bats` covers all 11 cases (the original
6 plus 5 for the bug fixes above).
Workflow inputs are passed via env vars rather than direct `${{ }}`
expansion into shell commands, per workflow injection guidance.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
