test(e2e): migrate app suite from BATS to Chainsaw and wire into CI

Replaces the per-app BATS E2E suite (hack/e2e-apps/*.bats) with declarative
Kyverno Chainsaw tests under hack/e2e-chainsaw/, extending this branch's pilot
(postgres + bucket) to all 18 app suites.

- Declarative ports assert on status/conditions and concrete fields; the
  repeated "timeout ... until kubectl get" + "kubectl wait" boilerplate
  collapses into a single polling assert with structured failure diffs and
  automatic events/describe/podLogs capture on failure.
- Inherently imperative suites keep their logic in script steps: openbao
  init/unseal, kuberture split-horizon probes, vminstance, the gateway
  admission/impersonation cases, and kubernetes-latest/previous, which wrap
  the relocated hack/e2e-chainsaw/_lib/run-kubernetes.sh verbatim.
- Gateway tests derive the tenant apex from the namespace host label at
  runtime, so they are host-independent.

CI: the e2e job now runs "chainsaw test hack/e2e-chainsaw/" via a new
test-chainsaw target and uploads the JUnit chainsaw-report.xml; the chainsaw
binary is added to the e2e-sandbox image. install-cozystack and test-openapi
stay BATS. The per-app 3-retry loop is dropped, since assertion polling
replaces the fixed-timeout flakiness it papered over.

Verified against a dev cluster (DB/app, storage and VM suites pass). Service
port assertions use the ports[*].port projection form because Chainsaw
v0.2.15 mis-evaluates JMESPath number-literal port filters.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
This commit is contained in:
Myasnikov DaniilandClaude Opus 4.8 committed 2026-06-25 13:42:34 +05:00
1 parent 22ab9379f7
commit e22d84f2c8
69 files changed
+2981 -2101

No files matched your search

+51 -34
View File
@@ -328,10 +328,17 @@ jobs:
cd /tmp/$SANDBOX_NAME
make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME test-openapi
# Run E2E tests. With `full-e2e` label the selector is skipped, its
# outputs are empty, the != 'true' guard lets this step run, and the
# full bats list is used. Without the label, the selector decides;
# an empty selection (docs-only) skips this step entirely.
# ▸ Run E2E tests (Chainsaw, scoped by Test Impact Analysis)
#
# The "Select E2E tests" step above picks the suites impacted by the PR
# diff. With the `full-e2e` label that step is skipped, SELECTED_APPS is
# empty, and we fall back to the whole suite; an empty selection on a
# docs-only PR skips this step entirely (the skip != 'true' guard).
#
# Chainsaw discovers each named suite under hack/e2e-chainsaw/ and polls
# every assertion to its own timeout, so the old per-app 3-retry loop is
# gone — a failure is reported as a structured assertion diff, with
# cluster-state diagnostics dumped below on a non-zero exit.
- name: Run E2E tests
id: e2e_tests
if: ${{ steps.select.outputs.skip != 'true' }}
@@ -341,40 +348,50 @@ jobs:
run: |
cd /tmp/$SANDBOX_NAME
if [ "$FULL_E2E" = "true" ] || [ -z "$SELECTED_APPS" ]; then
apps_list=$(ls hack/e2e-apps/*.bats | xargs -n1 basename | cut -d. -f1)
suites="" # empty => test-chainsaw runs the whole suite
echo "Running full chainsaw suite"
else
apps_list="$SELECTED_APPS"
suites="$SELECTED_APPS"
echo "Running selected chainsaw suites: $suites"
fi
failed_tests=""
for app in $apps_list; do
echo "::group::Testing $app"
if make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME test-apps-$app; then
echo "::endgroup::"
echo "✅ Test $app completed successfully"
else
echo "::endgroup::"
echo "❌ Test $app failed (no retry — see diagnostics below)"
failed_tests="$failed_tests $app"
echo "::group::Diagnostics for $app"
docker exec "$SANDBOX_NAME" sh -c 'kubectl get hr -A -o wide 2>&1 | tail -50' || true
docker exec "$SANDBOX_NAME" sh -c 'kubectl get events -A --sort-by=.lastTimestamp 2>&1 | tail -30' || true
# COSI state at the moment of failure — the CRDs ship no printer
# columns, so pull the readiness fields explicitly (cozyreport
# captures the same at end-of-job, but bucket states move between
# a test failing and the report being taken).
docker exec "$SANDBOX_NAME" sh -c '
kubectl get bucketclaims.objectstorage.k8s.io -A -o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name,READY:.status.bucketReady,BUCKET:.status.bucketName" 2>&1
kubectl get buckets.objectstorage.k8s.io -o custom-columns="NAME:.metadata.name,READY:.status.bucketReady,ID:.status.bucketID,CLAIMNS:.spec.bucketClaim.namespace" 2>&1
kubectl get bucketaccesses.objectstorage.k8s.io -A -o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name,GRANTED:.status.accessGranted,CLAIM:.spec.bucketClaimName" 2>&1
' || true
echo "::endgroup::"
fi
done
if [ -n "$failed_tests" ]; then
echo "❌ Failed tests:$failed_tests"
if ! make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME test-chainsaw CHAINSAW_SUITES="$suites"; then
echo "❌ Chainsaw E2E failed (see the assertion diffs above and diagnostics below)"
echo "::group::Diagnostics: HelmRelease status"
docker exec "$SANDBOX_NAME" sh -c 'kubectl get hr -A -o wide 2>&1 | tail -50' || true
echo "::endgroup::"
echo "::group::Diagnostics: recent events"
docker exec "$SANDBOX_NAME" sh -c 'kubectl get events -A --sort-by=.lastTimestamp 2>&1 | tail -30' || true
echo "::endgroup::"
# COSI state at the moment of failure — the CRDs ship no printer
# columns, so pull the readiness fields explicitly (cozyreport
# captures the same at end-of-job, but bucket states move between
# a test failing and the report being taken).
echo "::group::Diagnostics: COSI bucket state"
docker exec "$SANDBOX_NAME" sh -c '
kubectl get bucketclaims.objectstorage.k8s.io -A -o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name,READY:.status.bucketReady,BUCKET:.status.bucketName" 2>&1
kubectl get buckets.objectstorage.k8s.io -o custom-columns="NAME:.metadata.name,READY:.status.bucketReady,ID:.status.bucketID,CLAIMNS:.spec.bucketClaim.namespace" 2>&1
kubectl get bucketaccesses.objectstorage.k8s.io -A -o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name,GRANTED:.status.accessGranted,CLAIM:.spec.bucketClaimName" 2>&1
' || true
echo "::endgroup::"
exit 1
fi
echo "✅ All E2E tests passed"
echo "✅ All selected E2E tests passed"
- name: Collect chainsaw report
if: always()
run: |
cd /tmp/$SANDBOX_NAME
mkdir -p _out
docker cp "$SANDBOX_NAME":/workspace/hack/e2e-chainsaw/chainsaw-report.xml \
_out/chainsaw-report.xml || true
- name: Upload chainsaw report
if: always()
uses: actions/upload-artifact@v4
with:
name: chainsaw-report
path: /tmp/${{ env.SANDBOX_NAME }}/_out/chainsaw-report.xml
if-no-files-found: ignore
# ▸ Collect debug information (always runs)
- name: Collect report