Commit Graph
31 Commits
Author SHA1 Message Date
Aleksei Sviridkin 813c618086 fix(e2e): keep chainsaw asserts polling through transient API errors
Chainsaw v0.2.15 ends an assert or error operation on the first API
error other than NotFound. kube-apiserver answers a GET with 500 when
etcd changes its leader, so a leader change that lands inside a poll
fails the step long before its own timeout, on whichever suite happens
to be polling.

The sandbox image now builds v0.2.15 from source with the fix proposed
upstream in kyverno/chainsaw#2761, applied unchanged, until a chainsaw
release carries it. The build runs the tests of the patched packages
and a test that drives chainsaw built from the patched tree against a
fake apiserver: a run of 500s must not fail an assert, and a
persistent one must fail it at its timeout rather than at the first
error.

Closes: #4720

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-11 11:04:12 +03:00
IvanHunters ba46b9fc60 ci(testing): download the flux installer with --fail in the e2e-sandbox image
The flux install piped `curl -sSL` into bash, so on a 4xx/5xx the error
page was piped into the shell and curl's exit code was lost. Download the
installer to a file with `curl -fsSL` first, then run it, so an HTTP error
fails the build step instead of feeding bash a bad script.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-26 14:53:58 +03:00
IvanHunters 03c6c13443 ci(testing): pin flux version in the e2e-sandbox image
The e2e-sandbox image fetched flux through `install.sh` with no version,
so the build resolved the installer's floating "latest". Add an
ARG FLUX_VERSION, matching the neighbouring COZYHR_VERSION and
CRUST_GATHER_VERSION pins, and pass it to install.sh so the sandbox's
flux CLI is fixed and reproducible like the rest of the image's tools.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-26 14:53:58 +03:00
renovate[bot] a7dfbe6f8c chore(deps): update container image & action digests
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 16:55:04 +03:00
Aleksei Sviridkin e66f134564 fix(e2e-sandbox): install mc from its GitHub release
dl.min.io now answers 410 Gone for every community mc build, since
MinIO archived the project, so the sandbox image no longer builds.
The final mc release is still published as a GitHub release asset.
Pin it and verify the per-arch checksum, because the old install
trusted whatever the URL served.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 16:55:02 +03:00
myasnikovdaniil 99a28d82c2 test(e2e): migrate app suite from BATS to Chainsaw and wire into CI (#2826)
## What this PR does

Migrates the E2E **app** test suite from BATS to [Kyverno
Chainsaw](https://github.com/kyverno/chainsaw) — a declarative,
Kubernetes-native E2E framework (CNCF, part of the Kyverno project;
successor to KUTTL) — and wires it into CI as the replacement for the
per-app BATS loop. This grows the original pilot (postgres + bucket)
into the full suite.

All 22 app suites are ported under `hack/e2e-chainsaw/` and the
`hack/e2e-apps/*.bats` files are removed:

`postgres`, `bucket`, `mariadb`, `mongodb`, `redis`, `qdrant`,
`clickhouse`, `kafka`, `etcd`, `openbao`, `harbor`, `foundationdb`,
`external-dns`, `kuberture`, `vminstance`, `gateway`,
`kubernetes-latest`, `kubernetes-previous`, `kubernetes-oidc-system`,
`kubernetes-oidc-customconfig`, `securitygroup`, `serviceexposure`.

**Approach**

- **Declarative** suites assert on `status.conditions` and concrete
fields. The `timeout N sh -ec "until kubectl get ..."` + `kubectl wait`
pair that appeared ~100 times collapses into a single `assert` that
polls existence and state together, with a structured diff on failure
and automatic `events`/`describe`/`podLogs` capture via `catch`
(previously only `harbor.bats` did this, by hand).
- **Imperative** suites keep their logic in `script` steps: `openbao`
init/unseal, `kuberture` external-dns split-horizon probes,
`vminstance`, the `gateway` admission/impersonation cases, and
`kubernetes-latest`/`previous`, which wrap the relocated
`hack/e2e-chainsaw/_lib/run-kubernetes.sh` verbatim (Kamaji bring-up,
LB/NFS/ouroboros checks).
- `gateway` tests derive the tenant apex from the namespace
`namespace.cozystack.io/host` label at runtime, so they are
host-independent.

**CI**

- The `e2e` job now runs `chainsaw test hack/e2e-chainsaw/` via a new
`test-chainsaw` target and uploads the JUnit `chainsaw-report.xml`.
- The `chainsaw` binary is added to the e2e-sandbox image.
- `install-cozystack` and `test-openapi` stay BATS (cluster bootstrap +
OpenAPI checks).
- The per-app 3-retry loop is dropped — assertion polling replaces the
fixed-timeout flakiness it papered over.

**Validation**

Ran against a development cluster: the DB/app, storage, and VM suites
pass; the three suites that depend on platform features not present on
that cluster (`gateway`, `kuberture`, `external-dns`) and the two
heavyweight `kubernetes-*` suites are exercised by this PR's CI run on a
freshly installed platform.

Note for reviewers: service-port asserts use the `(ports[*].port)`
projection form rather than a number-literal filter (`` ports[?port ==
`N`] ``), because Chainsaw v0.2.15 mis-evaluates JMESPath number-literal
comparisons.

**2026-07-09: reconciled with main (~520 commits of drift)**

- etcd suite re-ported to the v1alpha2 operator contracts (#2859):
`readyMembers=3` gate, pod-label/`/scale`/WorkloadMonitor/metrics/defrag
contracts, plus the `ETCD_E2E_S3_ROUNDTRIP`-gated backup round-trip
driving `examples/backups/etcd`
- 12 new `ingress-hostname-policy` gateway cases ported (apex derived at
runtime from the tenant-root namespace label)
- post-branch bats drift folded in: `mariadb-single` webhook guard,
kafka `c1.small` presets, bucket/harbor 2m BucketClaim fail-fast, bucket
readonly-denial promoted to a hard fail (cosi-driver v0.3.1), qdrant
PVC-reclaim guard (#3059), kuberture fail-loud negations,
SC-fallback-default test (#2872 B1)
- 4 suites that only exist on main since the branch was cut are ported:
`kubernetes-oidc-system`, `kubernetes-oidc-customconfig`,
`securitygroup`, `serviceexposure`
- CI seams: `nightly.yaml` converted to the chainsaw invocation,
`run-kubernetes.sh` reconciled (Talos/CABPT waits, tenant drain, LINSTOR
pool wait, talos-image-cache under `_lib/`), `e2e-capture-dataplane.sh`
replicated in the Chainsaw global catch

### Release note

```release-note
fix(kubernetes): tenant Kubernetes teardown no longer hangs when the cluster has no working nodes — the pre-delete hook now bounds its wait for the in-tenant HelmReleases and force-clears their Flux finalizers on timeout (#3271)
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* End-to-end testing is now driven by a unified Chainsaw suite (platform
apps, Kubernetes, networking, security groups, service exposure).
* Test Impact Analysis selects affected suites, with an option to run
the full suite.
* CI produces a JUnit report artifact and collects suite-scoped
diagnostics on failure.
* **Bug Fixes**
* CI E2E now fails fast and emits clearer Kubernetes/HelmRelease and
sandbox bucket state diagnostics.
* Improved teardown reliability with bounded wait + finalizer handling
to avoid hangs.
* **Documentation**
* Updated E2E testing docs and local guidance to reflect the new
Chainsaw-based workflow and suite paths.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-12 11:58:08 +05:00
renovate[bot] 5b9b420ffe chore(deps): update container image & action digests 2026-07-10 05:29:23 +00:00
Myasnikov DaniilandClaude Opus 4.8 2940093102 test(e2e): capture per-failed-test crust-gather snapshots in Chainsaw harness
The Chainsaw app suite runs via `chainsaw test` directly, bypassing
hack/cozytest.sh — whose EXIT trap captures a crust-gather host snapshot
per failed BATS test. As a result a failing Chainsaw run uploaded a
cozyreport.tgz with zero per-test cluster snapshots, the richest diagnostic
for failures like the gateway/etcd/bucket/openbao ones on this branch.

Add the Chainsaw analog: a global spec.error.catch in .chainsaw.yaml that
collects a bounded host snapshot into _out/cozyreport/snapshots/<test>/host
on any failure (same --exclude-kind Secret flags and timeout -k 30 300 as
cozytest.sh), keyed by the $test.metadata.name binding with a mkdir-guard so
repeat catch invocations don't re-collect. Wire COZY_SNAPSHOT_NAME into the
kubernetes-latest/previous suites so the existing tenant snapshot from
run-kubernetes.sh co-locates under the same snapshots/<test>/ dir. cozyreport.sh
already folds snapshots/ into the uploaded artifact.

The e2e-testing guide and the sandbox Dockerfile comment are updated to record
the convention.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-25 13:43:50 +05:00
Myasnikov DaniilandClaude Opus 4.8 e22d84f2c8 test(e2e): migrate app suite from BATS to Chainsaw and wire into CI
Replaces the per-app BATS E2E suite (hack/e2e-apps/*.bats) with declarative
Kyverno Chainsaw tests under hack/e2e-chainsaw/, extending this branch's pilot
(postgres + bucket) to all 18 app suites.

- Declarative ports assert on status/conditions and concrete fields; the
  repeated "timeout ... until kubectl get" + "kubectl wait" boilerplate
  collapses into a single polling assert with structured failure diffs and
  automatic events/describe/podLogs capture on failure.
- Inherently imperative suites keep their logic in script steps: openbao
  init/unseal, kuberture split-horizon probes, vminstance, the gateway
  admission/impersonation cases, and kubernetes-latest/previous, which wrap
  the relocated hack/e2e-chainsaw/_lib/run-kubernetes.sh verbatim.
- Gateway tests derive the tenant apex from the namespace host label at
  runtime, so they are host-independent.

CI: the e2e job now runs "chainsaw test hack/e2e-chainsaw/" via a new
test-chainsaw target and uploads the JUnit chainsaw-report.xml; the chainsaw
binary is added to the e2e-sandbox image. install-cozystack and test-openapi
stay BATS. The per-app 3-retry loop is dropped, since assertion polling
replaces the fixed-timeout flakiness it papered over.

Verified against a dev cluster (DB/app, storage and VM suites pass). Service
port assertions use the ports[*].port projection form because Chainsaw
v0.2.15 mis-evaluates JMESPath number-literal port filters.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-25 13:42:34 +05:00
Aleksei Sviridkin e3ed42a608 chore(testing): refresh e2e-sandbox tools and base to current upstream
Rebuilds the e2e test sandbox on current upstream to clear the published
advisories (containerd, helm, go-git, moby/spdystream, Go stdlib) carried by the
stale tool binaries and base layers:

- base: ubuntu 22.04 -> 24.04; netcat -> netcat-openbsd, since `netcat` is a
  candidate-less virtual package on 24.04
- kubectl: 1.33.2 -> 1.33.13, staying on the 1.33 line that the e2e control
  plane pins at v1.33.12, so client/server skew is unchanged
- talosctl: 1.10.4 -> 1.13.5, aligning the client with the v1.13.x Talos nodes
  the sandbox provisions
- helm: 3.18.3 -> 3.21.2 (latest helm 3); the e2e flow drives the v3 CLI, so it
  stays on the 3.x line rather than moving to the breaking 4.x
- yq: 4.44.3 -> 4.53.3
- values.yaml: re-pinned to the rebuilt multi-arch digest

crust-gather and cozyhr already track their latest releases; flux and mc install
the current release at build time.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-23 02:23:39 +03:00
Myasnikov Daniil 95f42aac31 fix(e2e-sandbox): pin crust-gather to v0.15.2 (tag needs leading v)
The crust-gather install.sh substitutes the version verbatim into both
the release-tag path and the asset filename:
  releases/download/<ver>/kubectl-crust-gather_<ver>_<target>.tar.gz
The release tag is 'v0.15.2', so passing '0.15.2' 404s on the tag path
(curl exit 22) and fails the image build. The 'v0.15.2' asset variant
exists too, so the leading v satisfies both. Verified HTTP 200.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-18 18:24:41 +05:00
Myasnikov DaniilandClaude Opus 4.8 50e5f94132 test(e2e): per-test crust-gather snapshots + cleanup-on-exit via cozytest.sh
cozytest.sh runs each .bats as one invocation, exits on the first failing @test, and never invokes bats setup/teardown. Add a single EXIT-trap hook: on failure, snapshot the host AND every nested tenant cluster (tenantkubeconfig-*) with crust-gather into per-app dirs BEFORE cleanup — so each failure is independently inspectable (crust-gather serve) instead of one end-of-suite host-only dump; then ALWAYS run the file's cozy_cleanup() if defined.

run-kubernetes.sh gains cozy_cleanup() (delete tenant clusters, wait for PVCs to free): a failed kubernetes test otherwise leaves tens of GiB of worker-VM PVCs in tenant-test, exhausting the shared tenant-quota and cascade-failing every storage-heavy app after it. The kubernetes .bats now source run-kubernetes.sh at file scope so the parent-shell trap can reach the hook.

Ship pinned crust-gather 0.15.2 in the e2e-sandbox image; fold per-test snapshots into cozyreport.tgz.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-18 18:07:52 +05:00
Myasnikov DaniilandClaude Opus 4.7 45937d6d13 fix(e2e-sandbox): use curl -fsSL when downloading mc
The minio client was downloaded without -L, so when dl.min.io served
a 302 redirect, curl wrote the HTML body ("<a href=...>Found</a>") to
/usr/local/bin/mc, chmod +x'd it, and shipped that as the binary. When
the bucket E2E test ran "mc alias set ...", the shell tried to interpret
the HTML as a script and errored with:

    /usr/local/bin/mc: 1: cannot open a: No such file

(dash reading "<a" as an input redirect from a file named "a").

The image is rebuilt per CI run from cached layers; the broken mc layer
was cached and reused. Adding -fsSL fixes the download and the textual
change busts the cache so the next build pulls a real ELF binary.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-05-20 10:40:32 +05:00
Andrei Kvapil b864c04069 Update cozyhr v1.6.1
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-01-27 16:44:47 +01:00
Timofei Larkin bf1928c96f [testing] Add aliases and autocomplete
## What this PR does

Adds a `k=kubectl` alias and bash completion for kubectl to the
e2e-testing sandbox container to maintainers have an easier time
exec'ing into the CI container when something needs to be debugged.

### Release note

```release-note
[testing] Add k=kubectl alias and enable kubectl completion in the CI
container.
```

Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2026-01-05 16:29:39 +03:00
Andrei Kvapil cbc998b2ec Rename cozypkg to cozyhr
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2025-12-25 16:54:22 +01:00
IvanHunters 730584bd15 [seaweed] add tests
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2025-07-28 16:18:07 +03:00
Andrei Kvapil 1f240387f9 [dx] fix: exclude ps from self destructing enviroments check
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2025-07-02 13:37:15 +03:00
Andrei Kvapil 39daa3a38a [dx] better check for processes in self destructing enviroments
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2025-07-02 09:54:15 +03:00
Andrei Kvapil 52714f5cce [tests] Introduce self destructing environments
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2025-07-02 03:42:14 +03:00
Andrei Kvapil ae13b58d5f [tests] Upd Kubernetes v1.33
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2025-06-23 15:37:57 +02:00
Andrei Kvapil 74633ad699 Update cozypkg v1.1.0
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2025-06-16 18:11:27 +02:00
Andrei Kvapil 908c75927e [platform] Use cozypkg instead of helm
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2025-06-13 19:02:15 +02:00
Ahmad Murzahmatov 9698f3c9f4 commit 7a7512da30 removed genisoimage package installation from Dockerfile which leds to test fail due to the fact that genisoimage is missing and runner enable to create image. issue reference - https://github.com/cozystack/cozystack/actions/runs/15084476654/job/42406141954. restored genisoimage package installation in Dockerfile
Signed-off-by: Ahmad Murzahmatov <gwynbleidd2106@yandex.com>
2025-05-20 03:54:03 +06:00
nbykov0 7a7512da30 core/testing: multiarch support
Signed-off-by: nbykov0 <166552198+nbykov0@users.noreply.github.com>
2025-05-12 02:50:11 +03:00
kklinch0 1d639fda0d [tests] fix tests and add retry for monitoring hr tests
Signed-off-by: kklinch0 <kklinch0@gmail.com>
2025-04-10 15:23:31 +03:00
Andrei Kvapil 6980dc59c5 Add workflow to run e2e tests using GitHub CI
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2025-04-03 15:02:52 +02:00
Andrei Kvapil 007ebd8c9c update Talos Linux v1.8.4 (#544)
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>

Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2024-12-27 14:33:17 +01:00
klinch0andAndrei Kvapil 7eb8864f42 Add tests refactor (#451)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

## Release Notes

- **New Features**
- Introduced a new script for managing Helm releases in Kubernetes,
including installation and status monitoring.
- Added a configuration file for tenant settings, enabling monitoring
and SeaweedFS.
- Enhanced PostgreSQL initialization script to manage database roles and
privileges dynamically.
  - Added a new local pre-commit hook for version map checks.

- **Bug Fixes**
  - Updated pre-commit hooks for consistent formatting.

- **Tests**
- Improved testing capabilities for applications in a Kubernetes
environment with new Makefile targets.
  - Enhanced Docker image with tools for YAML and JSON processing.
- Updated testing image to the latest version for improved performance.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Co-authored-by: Andrei Kvapil <kvapss@gmail.com>
2024-10-22 17:14:04 +02:00
Andrei Kvapil 9357ad4754 Prepare release v0.12.0 (#301)
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2024-08-21 13:06:07 +02:00
Andrei Kvapil 9be774ad30 Add e2e testing sandbox (#295)
This PR introduces new functionality for running e2e-tests in
k8s-cluster.

`make test` from a root invokes deploying of new sandbox for testing
cozystack.

from `packages/core/testing`:

`make test` - runs the end-to-end tests.
`make exec` - opens an interactive shell in the sandbox container.
`make login` - downloads the kubeconfig into a temporary directory and
runs a shell with the sandbox environment; mirrord must be installed.
`make proxy` - enables a SOCKS5 proxy; mirrord and gost must be
installed.

Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2024-08-20 09:07:43 +02:00