Files
cozystack/hack/e2e-chainsaw
myasnikovdaniil f061739ce2 ci(e2e): raise the E2E job limit from 215 to 265 minutes (#5288)
## What this PR does

Full chainsaw run doesn't fit in 215 minutes anymore. Last four nightly
runs were cancelled at the limit, and PRs that select full suite end
same way while every suite that got to run passed (#5283 for example).

This raises limit for all four jobs that run the suite (pull-requests,
e2e-fork, e2e-tag, nightly) from 215 to 265, so they stay on one figure.
Comments that state the cap follow it. Making e2e itself shorter is
separate work.

Part of #5261, `Collect report` ceiling in fork workflow from there is
not covered here.

### Screenshots

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
ci(e2e): E2E job limit raised from 215 to 265 minutes.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Chores**
* Increased the time limit for end-to-end test runs from 215 to 265
minutes across applicable workflows. This allows longer test runs to
complete before the overall job is stopped.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 09:16:13 +03:00
..

Chainsaw E2E tests

The Cozystack E2E app suite, written for Kyverno Chainsaw — a declarative, Kubernetes-native E2E test framework (CNCF, part of the Kyverno project; successor to KUTTL).

This directory replaces the per-app BATS suite that used to live in hack/e2e-apps/*.bats. The CI e2e job runs every suite under hack/e2e-chainsaw/ through hack/e2e-chainsaw-run.sh after make install-cozystack; cluster bootstrap (hack/e2e-prepare-cluster-container.bats, hack/e2e-install-cozystack.bats) and the OpenAPI checks (hack/e2e-test-openapi.bats) remain BATS.

Layout

hack/e2e-chainsaw/
├── .chainsaw.yaml          # global Configuration (namespace, timeouts, JUnit report)
├── _lib/                   # shell reused by the script-heavy suites
│   ├── run-kubernetes.sh   #   tenant Kubernetes bringup + LB/NFS/ouroboros checks
│   └── remediation-guard.sh
├── <app>/
│   ├── chainsaw-test.yaml  # one or more Test definitions
│   └── <app>.yaml          # the manifests the test applies
└── ...

A suite is a directory directly under hack/e2e-chainsaw/ in or below which Chainsaw finds a Test. Three layouts do that, and the selector (hack/select-e2e.sh) and the leak-guard allowlist both recognise the same three:

  • chainsaw-test.yaml, the usual one, or chainsaw-test.yml when the .yaml is absent;
  • numbered step files such as 01-install.yaml, which Chainsaw turns into a Test on its own when neither of the above exists;
  • either of those in a subdirectory of the suite, since Chainsaw walks every subdirectory of the directory it is handed.

A parked chainsaw-test.yaml.disabled is none of these: the directory is then not a suite, is selected by nothing and cannot be allowlisted.

Suites: postgres, bucket, mariadb, mongodb, redis, valkey, qdrant, clickhouse, kafka, kafka-metadata, rabbitmq, etcd, openbao, harbor, foundationdb, external-dns, kuberture, vminstance, gateway, opensearch, monitoring, kubernetes-latest, kubernetes-previous, securitygroup, seaweedfs, computeplane, observability-proxy. kubernetes-proxmox is parked as chainsaw-test.yaml.disabled and runs only in .github/workflows/e2e-proxmox.yaml, which needs a Proxmox hypervisor.

What Chainsaw buys over the BATS suite

  • Readable tests — a list of resources and expected states, not shell control flow. The timeout N sh -ec "until kubectl get ..." + kubectl wait pair that appeared ~100 times collapses into a single assert that polls existence and state together.
  • Distinguishable failure modes — "object never created", "created but never ready", and "wrong status" all surface as a structured diff instead of an opaque shell timeout.
  • Automatic diagnostics — catch blocks dump events, describe, and pod logs on any failure (the BATS suite only did this in harbor.bats, by hand).
  • Automatic cleanup — resources Chainsaw applied are deleted (and waited on) in reverse order; finally covers operator-created leftovers.
  • CI-ready output — a JUnit chainsaw-report.xml is emitted for the test summary.

Test anatomy

Each suite is a directory with one or more Test docs (in one of the layouts above, usually a chainsaw-test.yaml) plus the manifests it applies. A step has try (operations: apply, assert, error, script, delete, …), optional catch (diagnostics on failure), and optional finally (teardown regardless of outcome).

Declarative suites assert on status.conditions and concrete fields. Inherently imperative suites keep their logic in script steps:

  • openbao — bao operator init/unseal via kubectl exec.
  • kuberture — jq on Service annotations + external-dns split-horizon log assertions.
  • vminstance — a server-side dry-run eviction of the VM's launcher pod, checked against the VMI's LiveMigratable condition.
  • gateway — admission-rejection cases that need kubectl --as impersonation or error-message greps; the tenant apex is read from the namespace namespace.cozystack.io/host label at runtime so they pass on any host.
  • kubernetes-latest/previous — wrap _lib/run-kubernetes.sh (Kamaji bringup, port-forward, LB/NFS/ouroboros). The script cds to the repo root (the test dir is hack/e2e-chainsaw/<suite>, so repo root is ../../..).

Conventions / gotchas (Chainsaw v0.2.15)

  • Condition asserts use the filter-as-list form, not the documented [0] index:
    status:
      (conditions[?type == 'Ready']):
      - status: "True"
    
    The (conditions[?type == 'Ready'])[0]: {status: "True"} idiom fails with "field not found in the input object".
  • A reading of a failure diff: the "actual" side is a projection onto the expected fields — status: {} means the JMESPath expression failed, not that status is empty. Read the error line above the diff first.
  • error with a bare resource ref (name only) asserts the resource does not exist (used for "PVC reclaimed", "no backup schedule", "child has no Gateway HR").
  • script runs in the test directory with $NAMESPACE set to the config namespace (tenant-test).

Running locally

Requires a cluster with Cozystack installed and a tenant-test namespace (the environment hack/e2e-install-cozystack.bats produces). bucket and seaweedfs additionally need mc, nc, and jq on the host; kuberture/openbao/foundationdb/securitygroup/computeplane/kubernetes-latest/kubernetes-previous/vminstance need jq; observability-proxy needs curl and nc.

# install chainsaw
go install github.com/kyverno/chainsaw@latest   # or: brew install kyverno/chainsaw/chainsaw

# run all suites (config picked up from .chainsaw.yaml)
cd hack/e2e-chainsaw
chainsaw test

# run a single suite
chainsaw test postgres/

kubernetes-latest/kubernetes-previous each bring up a full tenant Kubernetes cluster (~25 min, real worker nodes) and vminstance boots a nested VM — expect a long run for the full set.

CI integration

The e2e job in .github/workflows/pull-requests.yaml runs make -C packages/core/testing test-chainsaw (which execs hack/e2e-chainsaw-run.sh inside the e2e sandbox) and uploads chainsaw-report.xml. The runner calls chainsaw test once per suite directory, fails a suite whose pods outlive its cleanup by more than the default timeouts.cleanup (LEAKED-PODS suite=<dir> in the log; see docs/agents/e2e-testing.md §8 and _lib/leak-allowlist.txt for known, filed leaks), and merges the per-suite reports into that one file. A local chainsaw test as shown above runs the same tests without the leak check. The sandbox image (packages/core/testing/images/e2e-sandbox/Dockerfile) ships the chainsaw binary alongside kubectl/helm/mc/jq/nc, built from source with a patch that keeps an assert polling through transient API errors such as an etcd leader change; a released chainsaw run locally fails the assert on the first such error instead.