mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-13 08:05:19 +09:00
ZverGuy
ebb5a9cec4
Unit tests mock the API server and the store, so nothing shows that the console's nginx, the proxy's TokenReview and SubjectAccessReview and a real VictoriaLogs agree. The suite drives a viewer of one tenant through the console against a seeded store: it reads its own namespace, is refused a sibling's, cannot widen the scope with client filters, an AccountID header or dot segments, and keeps a tail open past a minute. The root store is off in e2e, so the suite runs its own VictoriaLogs under the root store's Service name rather than waiting on Fluent Bit. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com>
85 lines
7.0 KiB
Markdown
85 lines
7.0 KiB
Markdown
# Chainsaw E2E tests
|
|
|
|
The Cozystack E2E app suite, written for [Kyverno Chainsaw](https://github.com/kyverno/chainsaw) — a declarative, Kubernetes-native E2E test framework (CNCF, part of the Kyverno project; successor to KUTTL).
|
|
|
|
This directory replaces the per-app BATS suite that used to live in `hack/e2e-apps/*.bats`. The CI `e2e` job runs every suite under `hack/e2e-chainsaw/` through `hack/e2e-chainsaw-run.sh` after `make install-cozystack`; cluster bootstrap (`hack/e2e-prepare-cluster-container.bats`, `hack/e2e-install-cozystack.bats`) and the OpenAPI checks (`hack/e2e-test-openapi.bats`) remain BATS.
|
|
|
|
## Layout
|
|
|
|
```
|
|
hack/e2e-chainsaw/
|
|
├── .chainsaw.yaml # global Configuration (namespace, timeouts, JUnit report)
|
|
├── _lib/ # shell reused by the script-heavy suites
|
|
│ ├── run-kubernetes.sh # tenant Kubernetes bringup + LB/NFS/ouroboros checks
|
|
│ └── remediation-guard.sh
|
|
├── <app>/
|
|
│ ├── chainsaw-test.yaml # one or more Test definitions
|
|
│ └── <app>.yaml # the manifests the test applies
|
|
└── ...
|
|
```
|
|
|
|
A suite is a directory directly under `hack/e2e-chainsaw/` in or below which Chainsaw finds a Test. Three layouts do that, and the selector (`hack/select-e2e.sh`) and the leak-guard allowlist both recognise the same three:
|
|
|
|
- `chainsaw-test.yaml`, the usual one, or `chainsaw-test.yml` when the `.yaml` is absent;
|
|
- numbered step files such as `01-install.yaml`, which Chainsaw turns into a Test on its own when neither of the above exists;
|
|
- either of those in a subdirectory of the suite, since Chainsaw walks every subdirectory of the directory it is handed.
|
|
|
|
A parked `chainsaw-test.yaml.disabled` is none of these: the directory is then not a suite, is selected by nothing and cannot be allowlisted.
|
|
|
|
Suites: `postgres`, `bucket`, `mariadb`, `mongodb`, `redis`, `valkey`, `qdrant`, `clickhouse`, `kafka`, `kafka-metadata`, `rabbitmq`, `etcd`, `openbao`, `harbor`, `foundationdb`, `external-dns`, `kuberture`, `vminstance`, `gateway`, `opensearch`, `monitoring`, `kubernetes-latest`, `kubernetes-previous`, `securitygroup`, `seaweedfs`, `computeplane`, `observability-proxy`. `kubernetes-proxmox` is parked as `chainsaw-test.yaml.disabled` and runs only in `.github/workflows/e2e-proxmox.yaml`, which needs a Proxmox hypervisor.
|
|
|
|
## What Chainsaw buys over the BATS suite
|
|
|
|
- **Readable tests** — a list of resources and expected states, not shell control flow. The `timeout N sh -ec "until kubectl get ..."` + `kubectl wait` pair that appeared ~100 times collapses into a single `assert` that polls existence and state together.
|
|
- **Distinguishable failure modes** — "object never created", "created but never ready", and "wrong status" all surface as a structured diff instead of an opaque shell timeout.
|
|
- **Automatic diagnostics** — `catch` blocks dump events, `describe`, and pod logs on any failure (the BATS suite only did this in `harbor.bats`, by hand).
|
|
- **Automatic cleanup** — resources Chainsaw applied are deleted (and waited on) in reverse order; `finally` covers operator-created leftovers.
|
|
- **CI-ready output** — a JUnit `chainsaw-report.xml` is emitted for the test summary.
|
|
|
|
## Test anatomy
|
|
|
|
Each suite is a directory with one or more `Test` docs (in one of the layouts above, usually a `chainsaw-test.yaml`) plus the manifests it applies. A step has `try` (operations: `apply`, `assert`, `error`, `script`, `delete`, …), optional `catch` (diagnostics on failure), and optional `finally` (teardown regardless of outcome).
|
|
|
|
Declarative suites assert on `status.conditions` and concrete fields. Inherently imperative suites keep their logic in `script` steps:
|
|
|
|
- **openbao** — `bao operator init`/`unseal` via `kubectl exec`.
|
|
- **kuberture** — `jq` on Service annotations + external-dns split-horizon log assertions.
|
|
- **vminstance** — a server-side dry-run eviction of the VM's launcher pod, checked against the VMI's `LiveMigratable` condition.
|
|
- **gateway** — admission-rejection cases that need `kubectl --as` impersonation or error-message greps; the tenant apex is read from the namespace `namespace.cozystack.io/host` label at runtime so they pass on any host.
|
|
- **kubernetes-latest/previous** — wrap `_lib/run-kubernetes.sh` (Kamaji bringup, port-forward, LB/NFS/ouroboros). The script `cd`s to the repo root (the test dir is `hack/e2e-chainsaw/<suite>`, so repo root is `../../..`).
|
|
|
|
## Conventions / gotchas (Chainsaw v0.2.15)
|
|
|
|
- Condition asserts use the **filter-as-list** form, not the documented `[0]` index:
|
|
```yaml
|
|
status:
|
|
(conditions[?type == 'Ready']):
|
|
- status: "True"
|
|
```
|
|
The `(conditions[?type == 'Ready'])[0]: {status: "True"}` idiom fails with "field not found in the input object".
|
|
- A reading of a failure diff: the **"actual" side is a projection onto the expected fields** — `status: {}` means the JMESPath expression failed, not that status is empty. Read the error line above the diff first.
|
|
- `error` with a bare resource ref (name only) asserts the resource **does not exist** (used for "PVC reclaimed", "no backup schedule", "child has no Gateway HR").
|
|
- `script` runs in the **test directory** with `$NAMESPACE` set to the config namespace (`tenant-test`).
|
|
|
|
## Running locally
|
|
|
|
Requires a cluster with Cozystack installed and a `tenant-test` namespace (the environment `hack/e2e-install-cozystack.bats` produces). `bucket` and `seaweedfs` additionally need `mc`, `nc`, and `jq` on the host; `kuberture`/`openbao`/`foundationdb`/`securitygroup`/`computeplane`/`kubernetes-latest`/`kubernetes-previous`/`vminstance` need `jq`; `observability-proxy` needs `curl` and `nc`.
|
|
|
|
```bash
|
|
# install chainsaw
|
|
go install github.com/kyverno/chainsaw@latest # or: brew install kyverno/chainsaw/chainsaw
|
|
|
|
# run all suites (config picked up from .chainsaw.yaml)
|
|
cd hack/e2e-chainsaw
|
|
chainsaw test
|
|
|
|
# run a single suite
|
|
chainsaw test postgres/
|
|
```
|
|
|
|
`kubernetes-latest`/`kubernetes-previous` each bring up a full tenant Kubernetes cluster (~25 min, real worker nodes) and `vminstance` boots a nested VM — expect a long run for the full set.
|
|
|
|
## CI integration
|
|
|
|
The `e2e` job in `.github/workflows/pull-requests.yaml` runs `make -C packages/core/testing test-chainsaw` (which execs `hack/e2e-chainsaw-run.sh` inside the e2e sandbox) and uploads `chainsaw-report.xml`. The runner calls `chainsaw test` once per suite directory, fails a suite whose pods outlive its cleanup by more than the default `timeouts.cleanup` (`LEAKED-PODS suite=<dir>` in the log; see `docs/agents/e2e-testing.md` §8 and `_lib/leak-allowlist.txt` for known, filed leaks), and merges the per-suite reports into that one file. A local `chainsaw test` as shown above runs the same tests without the leak check. The sandbox image (`packages/core/testing/images/e2e-sandbox/Dockerfile`) ships the `chainsaw` binary alongside `kubectl`/`helm`/`mc`/`jq`/`nc`, built from source with a patch that keeps an assert polling through transient API errors such as an etcd leader change; a released `chainsaw` run locally fails the assert on the first such error instead.
|