mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-22 20:53:52 +09:00
test(e2e): migrate app suite from BATS to Chainsaw and wire into CI
Replaces the per-app BATS E2E suite (hack/e2e-apps/*.bats) with declarative Kyverno Chainsaw tests under hack/e2e-chainsaw/, extending this branch's pilot (postgres + bucket) to all 18 app suites. - Declarative ports assert on status/conditions and concrete fields; the repeated "timeout ... until kubectl get" + "kubectl wait" boilerplate collapses into a single polling assert with structured failure diffs and automatic events/describe/podLogs capture on failure. - Inherently imperative suites keep their logic in script steps: openbao init/unseal, kuberture split-horizon probes, vminstance, the gateway admission/impersonation cases, and kubernetes-latest/previous, which wrap the relocated hack/e2e-chainsaw/_lib/run-kubernetes.sh verbatim. - Gateway tests derive the tenant apex from the namespace host label at runtime, so they are host-independent. CI: the e2e job now runs "chainsaw test hack/e2e-chainsaw/" via a new test-chainsaw target and uploads the JUnit chainsaw-report.xml; the chainsaw binary is added to the e2e-sandbox image. install-cozystack and test-openapi stay BATS. The per-app 3-retry loop is dropped, since assertion polling replaces the fixed-timeout flakiness it papered over. Verified against a dev cluster (DB/app, storage and VM suites pass). Service port assertions use the ports[*].port projection form because Chainsaw v0.2.15 mis-evaluates JMESPath number-literal port filters. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
This commit is contained in:
1 parent
22ab9379f7
commit
e22d84f2c8
69 files changed
+2981
-2101
No files matched your search
@@ -328,10 +328,17 @@ jobs:
|
||||
cd /tmp/$SANDBOX_NAME
|
||||
make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME test-openapi
|
||||
|
||||
# Run E2E tests. With `full-e2e` label the selector is skipped, its
|
||||
# outputs are empty, the != 'true' guard lets this step run, and the
|
||||
# full bats list is used. Without the label, the selector decides;
|
||||
# an empty selection (docs-only) skips this step entirely.
|
||||
# ▸ Run E2E tests (Chainsaw, scoped by Test Impact Analysis)
|
||||
#
|
||||
# The "Select E2E tests" step above picks the suites impacted by the PR
|
||||
# diff. With the `full-e2e` label that step is skipped, SELECTED_APPS is
|
||||
# empty, and we fall back to the whole suite; an empty selection on a
|
||||
# docs-only PR skips this step entirely (the skip != 'true' guard).
|
||||
#
|
||||
# Chainsaw discovers each named suite under hack/e2e-chainsaw/ and polls
|
||||
# every assertion to its own timeout, so the old per-app 3-retry loop is
|
||||
# gone — a failure is reported as a structured assertion diff, with
|
||||
# cluster-state diagnostics dumped below on a non-zero exit.
|
||||
- name: Run E2E tests
|
||||
id: e2e_tests
|
||||
if: ${{ steps.select.outputs.skip != 'true' }}
|
||||
@@ -341,40 +348,50 @@ jobs:
|
||||
run: |
|
||||
cd /tmp/$SANDBOX_NAME
|
||||
if [ "$FULL_E2E" = "true" ] || [ -z "$SELECTED_APPS" ]; then
|
||||
apps_list=$(ls hack/e2e-apps/*.bats | xargs -n1 basename | cut -d. -f1)
|
||||
suites="" # empty => test-chainsaw runs the whole suite
|
||||
echo "Running full chainsaw suite"
|
||||
else
|
||||
apps_list="$SELECTED_APPS"
|
||||
suites="$SELECTED_APPS"
|
||||
echo "Running selected chainsaw suites: $suites"
|
||||
fi
|
||||
failed_tests=""
|
||||
for app in $apps_list; do
|
||||
echo "::group::Testing $app"
|
||||
if make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME test-apps-$app; then
|
||||
echo "::endgroup::"
|
||||
echo "✅ Test $app completed successfully"
|
||||
else
|
||||
echo "::endgroup::"
|
||||
echo "❌ Test $app failed (no retry — see diagnostics below)"
|
||||
failed_tests="$failed_tests $app"
|
||||
echo "::group::Diagnostics for $app"
|
||||
docker exec "$SANDBOX_NAME" sh -c 'kubectl get hr -A -o wide 2>&1 | tail -50' || true
|
||||
docker exec "$SANDBOX_NAME" sh -c 'kubectl get events -A --sort-by=.lastTimestamp 2>&1 | tail -30' || true
|
||||
# COSI state at the moment of failure — the CRDs ship no printer
|
||||
# columns, so pull the readiness fields explicitly (cozyreport
|
||||
# captures the same at end-of-job, but bucket states move between
|
||||
# a test failing and the report being taken).
|
||||
docker exec "$SANDBOX_NAME" sh -c '
|
||||
kubectl get bucketclaims.objectstorage.k8s.io -A -o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name,READY:.status.bucketReady,BUCKET:.status.bucketName" 2>&1
|
||||
kubectl get buckets.objectstorage.k8s.io -o custom-columns="NAME:.metadata.name,READY:.status.bucketReady,ID:.status.bucketID,CLAIMNS:.spec.bucketClaim.namespace" 2>&1
|
||||
kubectl get bucketaccesses.objectstorage.k8s.io -A -o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name,GRANTED:.status.accessGranted,CLAIM:.spec.bucketClaimName" 2>&1
|
||||
' || true
|
||||
echo "::endgroup::"
|
||||
fi
|
||||
done
|
||||
if [ -n "$failed_tests" ]; then
|
||||
echo "❌ Failed tests:$failed_tests"
|
||||
if ! make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME test-chainsaw CHAINSAW_SUITES="$suites"; then
|
||||
echo "❌ Chainsaw E2E failed (see the assertion diffs above and diagnostics below)"
|
||||
echo "::group::Diagnostics: HelmRelease status"
|
||||
docker exec "$SANDBOX_NAME" sh -c 'kubectl get hr -A -o wide 2>&1 | tail -50' || true
|
||||
echo "::endgroup::"
|
||||
echo "::group::Diagnostics: recent events"
|
||||
docker exec "$SANDBOX_NAME" sh -c 'kubectl get events -A --sort-by=.lastTimestamp 2>&1 | tail -30' || true
|
||||
echo "::endgroup::"
|
||||
# COSI state at the moment of failure — the CRDs ship no printer
|
||||
# columns, so pull the readiness fields explicitly (cozyreport
|
||||
# captures the same at end-of-job, but bucket states move between
|
||||
# a test failing and the report being taken).
|
||||
echo "::group::Diagnostics: COSI bucket state"
|
||||
docker exec "$SANDBOX_NAME" sh -c '
|
||||
kubectl get bucketclaims.objectstorage.k8s.io -A -o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name,READY:.status.bucketReady,BUCKET:.status.bucketName" 2>&1
|
||||
kubectl get buckets.objectstorage.k8s.io -o custom-columns="NAME:.metadata.name,READY:.status.bucketReady,ID:.status.bucketID,CLAIMNS:.spec.bucketClaim.namespace" 2>&1
|
||||
kubectl get bucketaccesses.objectstorage.k8s.io -A -o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name,GRANTED:.status.accessGranted,CLAIM:.spec.bucketClaimName" 2>&1
|
||||
' || true
|
||||
echo "::endgroup::"
|
||||
exit 1
|
||||
fi
|
||||
echo "✅ All E2E tests passed"
|
||||
echo "✅ All selected E2E tests passed"
|
||||
|
||||
- name: Collect chainsaw report
|
||||
if: always()
|
||||
run: |
|
||||
cd /tmp/$SANDBOX_NAME
|
||||
mkdir -p _out
|
||||
docker cp "$SANDBOX_NAME":/workspace/hack/e2e-chainsaw/chainsaw-report.xml \
|
||||
_out/chainsaw-report.xml || true
|
||||
|
||||
- name: Upload chainsaw report
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: chainsaw-report
|
||||
path: /tmp/${{ env.SANDBOX_NAME }}/_out/chainsaw-report.xml
|
||||
if-no-files-found: ignore
|
||||
|
||||
# ▸ Collect debug information (always runs)
|
||||
- name: Collect report
|
||||
|
||||
Reference in new issue
Block a user