Unit tests mock the API server and the store, so nothing shows that the
console's nginx, the proxy's TokenReview and SubjectAccessReview and a
real VictoriaLogs agree. The suite drives a viewer of one tenant through
the console against a seeded store: it reads its own namespace, is
refused a sibling's, cannot widen the scope with client filters, an
AccountID header or dot segments, and keeps a tail open past a minute.
The root store is off in e2e, so the suite runs its own VictoriaLogs
under the root store's Service name rather than waiting on Fluent Bit.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
lg_suite_has_test dropped every directory with `! -type d`, but the
selector keeps a chainsaw-test.y*ml whatever its type, so Chainsaw can
fail loudly on a directory by that name. An allowlisted suite in that
shape failed validation and stopped the run before Chainsaw reported the
real error. Use the selector's own `\( ! -type d -o -name
chainsaw-test.yaml -o -name chainsaw-test.yml \)`, which still drops a
directory with a step-file name.
The step-file regex behind the find glob was never reached by a test:
install-01.yaml is already filtered by the glob. 1a-install.yaml passes
the glob and not the regex, so it pins the regex; a directory named
chainsaw-test.yaml and one named 01-install.yaml pin the type test.
The README said the runner recognises the same three layouts. It
recognises none: it hands every top-level directory to Chainsaw.
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
lg_allowlist_validate accepted only <suite>/chainsaw-test.y*ml, while the
selectors already see step-file suites and a Test nested below the suite
directory. A suite in one of those layouts could not be allowlisted at
all: the run stopped before the first suite. Validate the allowlist with
the rule hack/select-e2e.sh enumerates suites with, spelled the same way
on both sides and cross-referenced so they move together.
Describe the three layouts once in the e2e-chainsaw README, point the
agent doc at it, and list kubernetes-proxmox as the parked suite.
Closes#4659
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
Chainsaw v0.2.15 ends an assert or error operation on the first API
error other than NotFound. kube-apiserver answers a GET with 500 when
etcd changes its leader, so a leader change that lands inside a poll
fails the step long before its own timeout, on whichever suite happens
to be polling.
The sandbox image now builds v0.2.15 from source with the fix proposed
upstream in kyverno/chainsaw#2761, applied unchanged, until a chainsaw
release carries it. The build runs the tests of the patched packages
and a test that drives chainsaw built from the patched tree against a
fake apiserver: a run of 500s must not fail an assert, and a
persistent one must fail it at its timeout rather than at the first
error.
Closes: #4720
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Nightly and the rc gate still booted the management nodes as QEMU
guests, the substrate the merge-gating lanes left behind. On it the
tenant Kubernetes suites have missed their node-join deadline on most
nights since 2026-10-02, each failure spends the whole deadline, and
the job reaches its timeout before the alphabetically later suites
run. Over the same days those suites passed on every PR run that
selected them, where the nodes are Talos containers and a tenant
worker sits one virtualization level shallower.
Both lanes now use the PR lanes' container substrate. Nightly's e2e no
longer waits for the nocloud disk, which it does not boot; the disk is
still built and published. A manual e2e-tag dispatch against a tag
that predates the container lane fails early and names the ref to
dispatch from instead.
DRBD, the replicated StorageClass and the Cozystack Talos node image
lose their last e2e coverage with this. The docs and comments that
pointed at nightly and the rc gate as that coverage now say so.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The module had render tests only. Nothing checked that turning it on
in a tenant yields a working cluster: the Kamaji control plane, a
worker pool shaped by the module's allowlist, an admin kubeconfig the
tenant can neither list nor read, the pinned cert-manager and
ingress-nginx addons running inside the cluster, and a teardown that
removes the in-cluster addons before the control plane and leaves
nothing behind.
The suite runs the module in its own child tenant so the shared
tenant-test is never toggled. It supplies its own pool because the
default md0 asks for the replicated StorageClass, which the container
lane does not have. On failure it prints the worker image source,
since the pool allowlist drops talos and a ComputePlane worker cannot
use the in-sandbox image cache.
The computeplane source depends on the kubernetes application and
runs its own ingress-nginx, so an ingress-nginx change now selects
this suite too. It also owns apps/kubernetes-nodes, which used to
reach no suite and run everything; kubernetes-nodes-application now
maps to both kubernetes suites and computeplane, so a pool chart
change keeps the suites that create KubernetesNodes natively.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Assisted-by: LLM
## What this PR does
Part of #4422.
The e2e suite now asks KubeVirt whether it would let a VM launcher pod
be evicted, without evicting it. Until now nothing in e2e evicted a
launcher pod, so the drain behaviour set up in #4403 was known only from
reading KubeVirt. Once the `vminstance` VM is running, the suite sends a
dry-run Eviction (`dryRun=All`) for its virt-launcher pod and checks the
answer. Nothing gets deleted.
A dry run still goes through the KubeVirt eviction webhook, which is
registered with `sideEffects: NoneOnDryRun` in v1.9.0. With
`LiveMigrateIfPossible` its answer follows the VMI's `LiveMigratable`
condition. That condition depends on storage, so the test reads it
instead of assuming it:
- `LiveMigratable=True`: the eviction must be refused with the webhook's
`Eviction triggered evacuation of VMI "<ns>/<name>"` message (HTTP 429).
A drain retries on that while the VM migrates away.
- `LiveMigratable=False`: the eviction must be allowed. The old
`LiveMigrate` value refused this case with "is not live-migratable".
KubeVirt v1.9.0 creates no PodDisruptionBudget for a VMI, so no PDB can
refuse it either.
- Anything else, including a missing condition, fails the test.
The allowed answer alone proves little, because the webhook also lets an
eviction through under `None`. So the test also asserts
`evictionStrategy: LiveMigrateIfPossible` in the VMI spec, where
virt-api copies the cluster value at creation. After the dry run it
checks that the launcher pod is the same object and not terminating, and
that the VMI has no `evacuationNodeName`.
On the PR lane only the `False` branch runs. Post-install prep pins
`StorageProfile/local` to RWO because DRBD is off there, and an RWO disk
makes the VMI non-migratable. Nightly and tag runs use `replicated`,
where I expect the `True` branch, but I have not seen this suite run on
that lane yet. A real drain and a workload update after a KubeVirt bump
are still open in #4422.
I ran the script against a stub `kubectl`. Both expected answers pass,
and it fails on every wrong answer I tried, such as a refusal for a
non-migratable VMI, a PDB-style 429, a missing condition, two launcher
pods, or a replaced pod after the dry run. `chainsaw lint` (v0.2.15) and
shellcheck pass.
### Screenshots
Not a UI change.
### Downstream repositories
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
The diff touches only `hack/e2e-chainsaw/vminstance/` and the suite
README, and nothing in the trigger map covers either.
### Release note
```release-note
test(vminstance): the e2e suite now dry-run evicts the VM launcher pod and checks the KubeVirt eviction decision against the VMI's LiveMigratable condition.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Tests**
* Added end-to-end coverage for dry-run eviction of a running virtual
machine’s launcher pod. The test checks the VM’s live-migration policy
and `LiveMigratable` condition, expecting eviction to be refused when
the VM is live-migratable and accepted otherwise.
* Verified that the dry run leaves the launcher pod unchanged and does
not record an evacuation node on the VM.
* **Documentation**
* Documented the test’s behavior and the local `jq` requirement for
running it.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Monitoring OIDC lost its live-cluster coverage when the two pre-Chainsaw
BATS suites were removed without a port. Render tests pin each mode on a
fresh install, but not admission by cozystack-api, the Deployment the
Grafana operator builds from the CR, or what an in-place mode change leaves
behind on a running release.
The suite drives one tenant-test Monitoring release through System,
CustomConfig with an inline config, and CustomConfig with a secretRef,
so a single stack bring-up covers all three and the upgrades between
them: the cozy-realm client, scope and client Secret are pruned when
System mode is left, and the inline generic_oauth block is gone once the
secretRef mount arrives.
The monitoring package sources now map to this suite, so a change to
system/monitoring, extra/monitoring, monitoring-rd or grafana-operator
selects it alone instead of escalating to the full run, and the selector
test that relied on cozystack.monitoring reaching no suite seeds a
source that does not.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
No suite evicts a virt-launcher pod, so the platform's eviction
behaviour is known only from reading KubeVirt. A server-side dry-run
eviction goes through the same KubeVirt admission webhook as a drain
and returns its decision without deleting anything.
The test reads the VMI's LiveMigratable condition rather than assuming
it, because the answer depends on the storage class: under
LiveMigrateIfPossible a migratable VMI is refused with the evacuation
message, and a non-migratable one is let through. It also pins the
evictionStrategy that virt-api stamped into the VMI spec, since the
webhook passes an eviction under strategy None as well, and checks that
the dry run left the launcher pod and the VMI untouched.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Every existing case installs SeaweedFS at one limit and stays there, so
nothing walks the transition an operator makes when lowering it: each
volume at or above the new value leaves the writable set and the bucket
writes again only after a grow. The failure is a PUT that times out
while every release stays Ready, so the suite ends on a write into a
bucket filled between the two limits.
The suite runs its own instance, created directly so the limit can move
without racing the tenant module, and asserts on the live master that
the value set on the resource arrives with the growth counts intact.
seaweedfs changes now select the suite, which the selector's mapping
and its tests are updated for.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The agent guide, the suite README and a workflow comment said the E2E
step is a single chainsaw invocation; it is now one per suite, and a
suite whose pods outlive its cleanup fails. Say how a pod is charged
to a suite, what the budget is, and how the allowlist is kept honest.
The workflows' failure line now points at LEAKED-PODS as well, since a
run that failed only on a leak has no assertion diff to point at.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
No suite created an OpenSearchCluster anywhere in the tree, so the whole
class of failure where the chart renders correctly and the cluster then
refuses the workload had no coverage at all. Rendering proves the
manifests; only a cluster proves that securityadmin authenticates
against the CA the chart handed the operator and that the security
configuration actually applies.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Convention 7 still sent readers to the node-join carve-out, while
section 1 of the same file -- rewritten on this branch -- says the
script, the marker and the soft_red output no longer exist. Restated as
what the deadline now is.
hack/e2e-chainsaw/README.md still listed kubernetes-oidc-system and
kubernetes-oidc-customconfig among the suites, and select-install.sh's
suite_to_source() still mapped both. The mapping is harmless today
because the round-trip pin walks existing directories, but select-e2e.sh
says the two functions have to stay in step, so a stale half is a trap
for whoever reads that promise.
sandbox-runner-headroom.bats keeps reading the QEMU sandbox file for
every lane, container lanes included: `prepare-env` also matches
`prepare-env-container`, and the compose file gives its nodes the same
8 CPU / 24576 MiB by design. The figures are therefore right for both
substrates, but only because the two agree -- and that they agree is
pinned in container-lane-capacity_test.bats rather than here. Said so in
the test, so the next reader knows what would have to change if the
container lane is ever sized differently.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: LLM
Two files enumerated the BATS suites and both stopped one short.
The global catch in .chainsaw.yaml describes itself as the Chainsaw
analog of the EXIT-trap host snapshot in hack/cozytest.sh, and said
that runner "only covers the BATS install/openapi tests". The
README said cluster bootstrap and the OpenAPI checks remain BATS and
named only two files. Both miss e2e-prepare-cluster.bats, which
packages/core/testing/Makefile drives through the same runner, and
which is the suite whose failures leave the sandbox in the state
hardest to reconstruct afterwards.
docs/agents/e2e-testing.md already listed all three, so the count was
answerable from the tree; it just disagreed with itself depending on
which file you opened.
The .chainsaw.yaml paragraph is reflowed because the correction is
longer than the line it replaces; no other wording changed.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Commit 3dbc7d366 removed ExposureClass/ServiceExposure (api/network, the
controller, and the CRDs shipped in cozystack-controller/definitions) in
favor of the native LoadBalancer path, but the chainsaw port that landed
the next day (dd2f89d20) brought the serviceexposure suite along. The
suite creates network.cozystack.io/v1alpha1 fixtures against an API that
no longer exists, so every full-suite run (any PR touching core paths,
and every release cut) fails its two tests with "no matches for
network.cozystack.io/v1alpha1".
Remove the dead suite and its wiring: the suite_to_source case and its
core-platform bats test in select-install (the "-" mechanism stays for
the next core-platform-owned suite), and the README suite list.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Main added securitygroup.bats (#2922 membership model) and
serviceexposure.bats (#3081 ExposureClass/ServiceExposure) after the
Chainsaw migration branch was cut.
- securitygroup: two Tests — the aggregated-API projection round-trip
(backing CNP marker label, empty-value membership endpointSelector,
lineage fromApp + membership fromSG peers, 1:1 rule translation, the
view hiding storage internals, unmarked-CNP invisibility via an error
op, delete propagation) and the controller membership-label
stamp/clear lifecycle. Only the attachments annotation stays a jq
script (it is a JSON string). Chainsaw auto-cleanup covers every
fixture even on mid-test failure — the bats had no teardown hook and
leaked its fixtures.
- serviceexposure: the externalIPs backend resolve/Ready/assignedIPs
path with the no-MetalLB-pool negative as an error op (ordered after
resolvedBackend so it cannot false-pass), and the finalizer-removal
test collapsed onto the delete op itself, which waits for full
disappearance.
- selection: kubernetes-application now also maps to the two OIDC
render-side suites (they exercise the same app chart — a gap main's
bats selection had too); the new dirs enter all_apps discovery
automatically. Selector unit tests extended and green (13/13 under
cozytest).
- README: suites list updated.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Replaces the per-app BATS E2E suite (hack/e2e-apps/*.bats) with declarative
Kyverno Chainsaw tests under hack/e2e-chainsaw/, extending this branch's pilot
(postgres + bucket) to all 18 app suites.
- Declarative ports assert on status/conditions and concrete fields; the
repeated "timeout ... until kubectl get" + "kubectl wait" boilerplate
collapses into a single polling assert with structured failure diffs and
automatic events/describe/podLogs capture on failure.
- Inherently imperative suites keep their logic in script steps: openbao
init/unseal, kuberture split-horizon probes, vminstance, the gateway
admission/impersonation cases, and kubernetes-latest/previous, which wrap
the relocated hack/e2e-chainsaw/_lib/run-kubernetes.sh verbatim.
- Gateway tests derive the tenant apex from the namespace host label at
runtime, so they are host-independent.
CI: the e2e job now runs "chainsaw test hack/e2e-chainsaw/" via a new
test-chainsaw target and uploads the JUnit chainsaw-report.xml; the chainsaw
binary is added to the e2e-sandbox image. install-cozystack and test-openapi
stay BATS. The per-app 3-retry loop is dropped, since assertion polling
replaces the fixed-timeout flakiness it papered over.
Verified against a dev cluster (DB/app, storage and VM suites pass). Service
port assertions use the ports[*].port projection form because Chainsaw
v0.2.15 mis-evaluates JMESPath number-literal port filters.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The (conditions[?type == 'Ready'])[0] indexed-filter idiom fails in
chainsaw v0.2.15 with 'field not found in the input object'. Use the
filter-as-list form instead. Verified: both tests pass end-to-end
against a live cluster.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Port two representative BATS E2E tests (postgres, bucket) to Kyverno
Chainsaw as a pilot for replacing the bash-based suite. Declarative
asserts replace the timeout/until polling boilerplate, catch blocks
add automatic failure diagnostics, and script steps cover the
imperative probes (psql, mc S3 access checks).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
ZverGuy