Non-blocking review notes from #2826:
- Add `# shellcheck shell=bash` to the four sourced `_lib/*.sh` helpers
(etcd-cleanup, remediation-guard, run-kubernetes, talos-image-cache) so
shellcheck stops flagging SC2148 (no shell directive) — they are only ever
`. sourced`, so there is no runtime interpreter to infer from a shebang.
- Add a `# TODO(chainsaw>0.2.15)` marker at the postgres Service asserts
explaining the `(ports[0].port)`/`(ports[*].port)` projection is a v0.2.15
number-literal-filter workaround (and that `ports[0]` checks only the first
port), so it gets reverted on a chainsaw bump; cross-references the same
workaround in kafka/mariadb/harbor and the e2e-testing.md gotchas section.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The in-sandbox Talos image mirror (#3244) is unreachable by the tenant
workers it serves, so the kubernetes-* e2e tests fail deterministically at
the 12m node-join wait instead of relieving the public-factory flake (#3231).
Root cause (from #3254 by @lexfrei, verified against the failing run's
crust-gather snapshot): tenant namespaces run a default-deny Cilium egress
(allow-internal/external-communication + the <tenant>-egress CCNP) that
permits the world entity and the tenant tree but NOT an arbitrary kube-system
Service, so a worker's CDI importer resolves talos-image-cache.kube-system.svc
yet its TCP connect to the ClusterIP is silently dropped and no disk imports.
#3244's gate never caught this because it probed the server over localhost
inside the serve pod, which passes regardless of cross-pod reachability.
Port of #3254 to the Chainsaw layout (hack/e2e-chainsaw/_lib/):
- ship a tightly-scoped CiliumClusterwideNetworkPolicy (importer pods in
tenant-test -> mirror pod in kube-system), applied at point-of-use once
Cilium CRDs exist; excluded from the pre-Cilium install apply via yq.
- replace the readiness probe with a throwaway pod in tenant-test labelled
cdi.kubevirt.io=importer that curls the ClusterIP with a range request and
must get 206 - the exact egress path a real importer faces; tenants use the
mirror only when that end-to-end check passes, else fall back to the public
factory, so it can only help, never make CI worse.
- unit test pinning the manifest split, probe/policy label agreement, the
strict 206 gate, and the --overrides JSON builder.
Also adds (a2) DataVolume import-stage diagnostics to the node-join failure
block in run-kubernetes.sh (importer logs + a ClusterIP re-probe via
talos_image_cache_diagnose) so the import-stage failure mode is legible.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Fold the in-sandbox Talos image cache work into the Chainsaw layout:
the new shared helper relocates to hack/e2e-chainsaw/_lib/
talos-image-cache.sh and run-kubernetes.sh sources it from there (the
ouroboros DNS-check no-retry rework merges clean). The OIDC bats are
resolved as deleted again; their Chainsaw fixtures deliberately skip
the talos.imageFactoryURL injection — the helper only emits it when
the cache is up, and the zero-worker render-side tests instantiate no
DataVolume imports for it to redirect.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Main's #3233 landed mid-reconciliation and touched the two OIDC bats
again: nodeGroups (and storageClass) dropped from the render-side test
CRs — the schema default {} now renders one zero-replica
MachineDeployment via CAPI defaulting, so the tests provision zero
worker DataVolumes (#3231). Resolve the modify/delete as deleted and
mirror the fixture change (and its rationale) into the ported
kubernetes-oidc-system / kubernetes-oidc-customconfig Chainsaw suites.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
- redis: assert both StatefulSet-templated PVCs Bound (replicas: 2
provisions two, only ordinal 0 was checked)
- gateway: || true each delete inside the three cleanup() traps — they
run under set -eu, so one transient delete failure skipped the
remaining cleanup commands
- bucket, backup(.disabled): guard the trap's kill against an unset
pf_pid (script can exit before the port-forward starts)
Not addressed on purpose: the sort(ports[*].port) suggestions on
clickhouse/kafka — sort() is among the JMESPath forms chainsaw v0.2.15
(still the latest release) mis-evaluates, which is why the suites use
the ordered (ports[*].port) projection; services keep spec order.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Main added securitygroup.bats (#2922 membership model) and
serviceexposure.bats (#3081 ExposureClass/ServiceExposure) after the
Chainsaw migration branch was cut.
- securitygroup: two Tests — the aggregated-API projection round-trip
(backing CNP marker label, empty-value membership endpointSelector,
lineage fromApp + membership fromSG peers, 1:1 rule translation, the
view hiding storage internals, unmarked-CNP invisibility via an error
op, delete propagation) and the controller membership-label
stamp/clear lifecycle. Only the attachments annotation stays a jq
script (it is a JSON string). Chainsaw auto-cleanup covers every
fixture even on mid-test failure — the bats had no teardown hook and
leaked its fixtures.
- serviceexposure: the externalIPs backend resolve/Ready/assignedIPs
path with the no-MetalLB-pool negative as an error op (ordered after
resolvedBackend so it cannot false-pass), and the finalizer-removal
test collapsed onto the delete op itself, which waits for full
disappearance.
- selection: kubernetes-application now also maps to the two OIDC
render-side suites (they exercise the same app chart — a gap main's
bats selection had too); the new dirs enter all_apps discovery
automatically. Selector unit tests extended and green (13/13 under
cozytest).
- README: suites list updated.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Main added kubernetes-oidc-system.bats and
kubernetes-oidc-customconfig.bats after the Chainsaw migration branch
was cut; without a port the 260 lines of coverage would vanish with
the hack/e2e-apps/ deletion.
Both are light cluster e2e (apply CR -> assert rendered objects; no
Kamaji bringup, minReplicas: 0), exactly Chainsaw's declarative shape:
- System mode: KamajiControlPlane carries --authentication-config +
the chart-owned volume/mount; the AuthenticationConfiguration Secret
carries the cozy-realm issuer (regex, script step) + per-cluster
audience; per-cluster KeycloakClient/KeycloakClientScope land in the
cozy realm
- CustomConfig: the flag is still wired; the Secret carries the
tenant-supplied issuer and NOT the cozy realm (fail-loud negation);
no Keycloak objects and no pre-rendered OIDC kubeconfig Secret,
expressed as error ops ordered after the positive asserts so an
unreconciled HR cannot false-pass them
The port also sheds two latent bats bugs: cozytest defines no skip()
(the Keycloak guard would have died with exit 127 had it ever fired)
and the api-resources group probe exits 0 even when the group is
absent — the chart capability-gates Keycloak rendering and CI always
ships the operator, so the asserts are unconditional here.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
- qdrant: add the post-delete PVC-reclaim guard (#3059) — delete the CR
and poll the ordinal-anchored data PVC to absence via the error op
(retries transient API errors internally), with a PVC/jobs/pods dump
on failure; Chainsaw auto-cleanup alone would never notice a leaked
StatefulSet-templated PVC
- kuberture: make the split-horizon negative log assertions fail loud —
under set -e a pipeline beginning with ! suppresses errexit, so both
leak checks silently passed (the same vacuous-negation bug main fixed
in 12fe3256f)
- foundationdb: retry the one-shot generations.reconciled read for up
to 2m — the operator stamps it after the post-bounce uptime gate, so
it can trail the fdbcli-derived health fields the previous assert
converged on (a79f8009f)
- kubernetes-latest: add the version-independent SC-fallback-default
Test (#2872 B1) calling verify_storageclass_fallback_default from the
shared lib; a separate Test doc so it runs regardless of the heavy
bringup Test's outcome, safe under the suite-wide parallel: 1 while
it temporarily mutates cluster-scoped StorageClasses
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Fold main's post-branch bucket/harbor bats drift into the Chainsaw
suites:
- BucketClaim bucketReady budgets tightened to 2m (bucket had the 5m
default, harbor a stale 10m with the reversed 'can lag on a loaded
runner' rationale): the objectstorage-controller requeues the claim
until the backend Bucket is ready (0f1cc548b), so readiness converges
in tens of seconds and a claim not ready after 2m is frozen — the
tight bound surfaces the propagation regression instead of masking
it (63d14e748)
- the bucket viewer-upload soft-xfail is promoted back to a hard
failure: accessPolicy=readonly enforcement landed with
seaweedfs-cosi-driver v0.3.1 (#1921, eab9b034f), so a successful
readonly upload is now a regression, exactly what the suite's own
TODO called for
- both catch blocks gain the backend view (cluster-scoped Buckets +
cozy-objectstorage-controller pods) where a frozen claim actually
surfaces
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Main's mariadb.bats gained "Create single-replica MariaDB" after the
Chainsaw port was cut: replication is guarded on replicas>1
(c7e8f4d97), so replicas=1 must render a CR the operator webhook
admits and provision the bare <name> Service without the
-primary/-secondary split. Mirrors the bats assertions (svc :3306 +
endpoints, StatefulSet replicas=1, metrics svc/endpoints/deployment)
and dumps the rendered k8s.mariadb.com CR in catch so a webhook
rejection is legible.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Main added cozystack-ingress-hostname-policy (d89ad7069) with 12 bats
cases after the Chainsaw port was cut; this appends them as one Test.
- VAP/binding installation asserted declaratively (binding must carry
Deny)
- rejects and accepts grouped by derivation needs: apex-derived rejects
(foreign-apex host, dot-boundary prefix-adjacent, foreign tls host,
wildcard matching the root apex), static catch-all rejects (hostless
rule, empty-string host, defaultBackend), own-apex accepts (in-apex
rule+tls, own-apex wildcard), and static external-domain accepts as a
declarative apply with auto-cleanup
- hostnames derived at runtime: the tenant apex from the test
namespace's namespace.cozystack.io/host label (suite convention) and
the platform ROOT apex from the same label on tenant-root, replacing
the bats' hardcoded test.example.org/example.org; evil${apex}
preserves the dot-boundary semantics of eviltest.example.org, and the
external-domain probes use the RFC 2606 .invalid TLD so they can
never collide with a real cluster apex
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The suite was ported from the pre-#2859 etcd.bats and asserted
etcd.aenix.io/v1alpha1 objects that no longer exist on main — every
Test hard-failed at the first assert on an unknown GVK.
- drop the etcd-empty-backup and etcd-backup-schedule scenarios with
their fixtures: the chart no longer renders EtcdBackupSchedule or
etcd-s3-creds (backups moved to the generic backups API), and
legacy-values tolerance is pinned by helm-unittest, not e2e
- rewrite the remaining scenario into etcd-1-contracts, mirroring
main's contracts test: gate on status.readyMembers=3 (Available=True
flips at seed-member discovery and races replica-count contracts),
assert the operator-owned member Pod shape (strict selector,
container name, named metrics port), WorkloadMonitor
operational/availableReplicas, and script the three contracts that
need imperative probes: the /scale subresource, the plaintext-HTTP
metrics probe against the Pod IP, and the on-demand defrag Job
- add etcd-2-backup-roundtrip driving examples/backups/etcd/run-all.sh,
gated on ETCD_E2E_S3_ROUNDTRIP=1 exactly like main (kind CI has no
publicly-trusted S3 endpoint the snapshot agent could verify)
- upgrade catch blocks to the v1alpha2 GVKs plus EtcdMember/
EtcdSnapshot/BackupJob describes and the snapshot-Job Pod logs
- strengthen _lib/etcd-cleanup.sh with main's etcd_drain deltas: probe
the Etcd CR itself (it can linger Terminating and block the next
singleton apply) and use the err-sentinel pattern so a transient API
error is never misread as "deleted"; add etcd_backup_cleanup wrapping
examples/backups/etcd/cleanup.sh for the round-trip's teardown
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
- nightly.yaml still looped over hack/e2e-apps/*.bats with the removed
test-apps-% target; switch it to the single chainsaw invocation
(empty CHAINSAW_SUITES = full suite) and upload the JUnit report
- pin the chainsaw-report upload-artifact step to the same SHA the
rest of the workflows use (scorecard/zizmor gate)
- repoint hack/run-kubernetes-drain_test.bats at the relocated
hack/e2e-chainsaw/_lib/run-kubernetes.sh
- replicate cozytest.sh's NotReady-pod data-plane capture
(hack/e2e-capture-dataplane.sh) in the Chainsaw global error.catch so
chainsaw-suite failures keep diagnostic parity with the BATS lanes
- refresh stale bats path comments
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Reconcile the Chainsaw e2e migration with ~500 commits of main drift:
- keep main's run-kubernetes.sh helpers (kubectl_wait_retry, tenant
drain, LINSTOR pool-free wait, SC-fallback probe) under the branch's
_lib/ rename with the Chainsaw finally/snapshot deltas re-applied
- resolve modify/delete on 10 drifted hack/e2e-apps/*.bats as deleted;
their post-branch assertion drift is re-ported into the Chainsaw
suites in follow-up commits
- drop the 4 new main-side bats (kubernetes-oidc-*, securitygroup,
serviceexposure) from _lib/ relocation; ported as Chainsaw suites in
follow-up commits
- docs/agents/e2e-testing.md: merge main's plan-job TIA wording with
the Chainsaw suite phrasing
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The etcd suite still failed every run after the clean-after-each rework:
etcd-empty-backup (first to run) passed, but etcd-basic and etcd-backup-schedule
wedged at EtcdCluster WaitingForFirstQuorum until the chart's helm --wait timed
out (InstallFailed).
Root cause, from the failed run's crust-gather snapshot: the StatefulSet's
data-etcd-{0,1,2} PVCs are retained on uninstall and the drain never reclaimed
them, so each later scenario rebound the first scenario's volumes (identical PV
UIDs and a single creationTimestamp across all three scenarios) and booted the
previous etcd's data dir and member identity — never forming a fresh first
quorum. The pods were individually Ready (StatefulSet readyReplicas: 3) with all
certs valid, which is why it looked like an operator/network issue rather than
stale storage.
Delete the data PVCs (app.kubernetes.io/managed-by=etcd-operator) in
etcd_cleanup and wait for them to clear alongside the HR and DataStore, so each
scenario starts from empty storage.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The viewer (readonly) S3 credential can still upload: the seaweedfs-cosi-driver
does not yet honor accessPolicy=readonly (cozystack/cozystack#1921, blocked
upstream on kubernetes-sigs/container-object-storage-interface#244). The whole
cozystack chain is correct — Bucket.spec.users.viewer.readonly: true binds the
viewer BucketAccess to the tenant-root-readonly BucketAccessClass whose
parameters.accessPolicy is readonly — but the driver grants the identity write
access anyway (verified live on dev10). The BATS original asserted the same
denial; its retries and toothless gate hid the failure, which this migration
surfaces.
Keep exercising the path and verifying admin read/write + viewer read, but emit
a ::warning:: instead of failing when the viewer can upload, so the suite is not
blocked on an unimplemented upstream feature. TODO(#1921): restore the hard
failure once the driver enforces read-only access.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
- mongodb/redis: assert status.readyReplicas instead of status.replicas, so the
StatefulSet check verifies pods are actually Ready (matches openbao/qdrant).
- kuberture: poll for the asynchronously-populated target annotation before
asserting the annotation shape (it was a one-shot test under set -e that could
race on a slow cluster), and guard the probe-Pod phase capture so the
describe/logs diagnostics still run when the Pod never materializes.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The three etcd scenarios share the singleton 'etcd' release and relied on the
next scenario's ensure-clean to drain the previous one. Chainsaw runs Tests in
name order (not file order), and v0.2.15 has no test-level finally, so the
last-run scenario leaked its etcd plus the wedged Kamaji DataStore into every
later suite on full/release runs (lexfrei review B3).
Make each scenario self-contained instead: a single run step pre-cleans any
leftover etcd, installs and asserts, then drains in a finally that always runs
(the step is always reached, so teardown fires on success and failure alike).
Extract the shared drain to _lib/etcd-cleanup.sh, removing the ~25-line block
that was copied across all three scenarios.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The Chainsaw port capped the harbor-test-registry BucketClaim bucketReady wait
at the default 5m, half the 600s the BATS original used. seaweedfs/COSI
provisioning can lag several minutes on a loaded runner, so the dependent
harbor-test-system HelmRelease keeps its 10m budget while the BucketClaim it
depends on was capped at 5m — a latent false-flake. Restore 10m
(docs/agents/e2e-testing.md section 6).
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The openbao suite's finally deleted PVC data-openbao-test-0 while the
StatefulSet pod still mounted it; the kubernetes.io/pvc-protection finalizer
blocked removal and the delete hit its 5m timeout, failing the suite on every
run (confirmed in CI runs 28084122948 and 28025182013). Delete the OpenBAO CR
first so the HelmRelease/StatefulSet/pod tear down and release the PVC, then
remove the now-unmounted PVC — mirroring the BATS teardown order.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
run_kubernetes_test applies the tenant Kubernetes CR imperatively (kubectl), so
Chainsaw's auto-cleanup — which only tracks resources it apply/create-d — never
reclaims it. On success the script's tail deleted it, but on failure nothing did:
cozy_cleanup() used to be invoked by hack/cozytest.sh's trap, and the Chainsaw
harness has no equivalent caller, so a failed kubernetes-latest/previous run
leaked the tenant cluster (tens of GiB of worker PVCs) into the shared
tenant-test namespace and starved later suites.
Add a `finally` block to both kubernetes suites that runs cozy_cleanup. `finally`
always executes, after the `catch` phase, so the ordering matches the BATS
harness: pass -> cleanup; fail -> crust-gather (tenant trap + host catch) ->
cleanup. cozy_cleanup stays best-effort (|| true) so a slow tenant teardown never
flips a passing test red.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The Chainsaw app suite runs via `chainsaw test` directly, bypassing
hack/cozytest.sh — whose EXIT trap captures a crust-gather host snapshot
per failed BATS test. As a result a failing Chainsaw run uploaded a
cozyreport.tgz with zero per-test cluster snapshots, the richest diagnostic
for failures like the gateway/etcd/bucket/openbao ones on this branch.
Add the Chainsaw analog: a global spec.error.catch in .chainsaw.yaml that
collects a bounded host snapshot into _out/cozyreport/snapshots/<test>/host
on any failure (same --exclude-kind Secret flags and timeout -k 30 300 as
cozytest.sh), keyed by the $test.metadata.name binding with a mkdir-guard so
repeat catch invocations don't re-collect. Wire COZY_SNAPSHOT_NAME into the
kubernetes-latest/previous suites so the existing tenant snapshot from
run-kubernetes.sh co-locates under the same snapshots/<test>/ dir. cozyreport.sh
already folds snapshots/ into the uploaded artifact.
The e2e-testing guide and the sandbox Dockerfile comment are updated to record
the convention.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Convert hack/e2e-apps/backup.bats.disabled to a Chainsaw test under
hack/e2e-chainsaw/backup/. It stays opt-in: the chainsaw-test.yaml.disabled
filename keeps it out of chainsaw's default discovery and out of
hack/select-e2e.sh enumeration, matching the previous disabled state.
The declarative readiness chain (Bucket -> VirtualMachine -> BackupJob) becomes
assert operations; the imperative Velero-backup + S3 verification stays a single
script step, mirroring the bucket suite. README documents the opt-in run command
and the backup-infra preconditions.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The three etcd Chainsaw scenarios share the singleton release name 'etcd' and
the cluster-scoped DataStore/<namespace> the etcd chart provisions for Kamaji.
On teardown Kamaji leaks its kamaji.clastix.io/TenantControlPlane finalizer on
that DataStore — once the backing etcd is gone the reconcile that would drop it
cannot complete — so the DataStore wedges in Terminating and the etcd
HelmRelease uninstall hangs. In CI this errored every etcd case: the first
scenario's stuck teardown blocked the next scenario's install (HelmRelease
never Ready, "DataStore status: Terminating", etcd pods refusing :2381 because
the CA issuer had no key material).
Make teardown explicit instead of relying on Chainsaw auto-cleanup (which was
waiting on the wedged uninstall): set skipDelete on each Test and have every
scenario's ensure-clean drain the previous one — delete the etcd and
force-clear the orphaned finalizer (no TenantControlPlane uses this
per-namespace test DataStore) in a loop until the HelmRelease and DataStore are
both gone. Each scenario then installs from a clean slate.
Verified on a dev cluster: all three scenarios pass back-to-back where they
previously cascaded.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Replaces the per-app BATS E2E suite (hack/e2e-apps/*.bats) with declarative
Kyverno Chainsaw tests under hack/e2e-chainsaw/, extending this branch's pilot
(postgres + bucket) to all 18 app suites.
- Declarative ports assert on status/conditions and concrete fields; the
repeated "timeout ... until kubectl get" + "kubectl wait" boilerplate
collapses into a single polling assert with structured failure diffs and
automatic events/describe/podLogs capture on failure.
- Inherently imperative suites keep their logic in script steps: openbao
init/unseal, kuberture split-horizon probes, vminstance, the gateway
admission/impersonation cases, and kubernetes-latest/previous, which wrap
the relocated hack/e2e-chainsaw/_lib/run-kubernetes.sh verbatim.
- Gateway tests derive the tenant apex from the namespace host label at
runtime, so they are host-independent.
CI: the e2e job now runs "chainsaw test hack/e2e-chainsaw/" via a new
test-chainsaw target and uploads the JUnit chainsaw-report.xml; the chainsaw
binary is added to the e2e-sandbox image. install-cozystack and test-openapi
stay BATS. The per-app 3-retry loop is dropped, since assertion polling
replaces the fixed-timeout flakiness it papered over.
Verified against a dev cluster (DB/app, storage and VM suites pass). Service
port assertions use the ports[*].port projection form because Chainsaw
v0.2.15 mis-evaluates JMESPath number-literal port filters.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The (conditions[?type == 'Ready'])[0] indexed-filter idiom fails in
chainsaw v0.2.15 with 'field not found in the input object'. Use the
filter-as-list form instead. Verified: both tests pass end-to-end
against a live cluster.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Port two representative BATS E2E tests (postgres, bucket) to Kyverno
Chainsaw as a pilot for replacing the bash-based suite. Declarative
asserts replace the timeout/until polling boilerplate, catch blocks
add automatic failure diagnostics, and script steps cover the
imperative probes (psql, mc S3 access checks).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
