Commit Graph
4 Commits
Author SHA1 Message Date
Aleksei Sviridkin a7ac37be65 test(e2e): let HR-Ready waits outlast the install timeout of the release they wait on (#4471)
## What this PR does

Stacked on #4468, which holds the raised backup waits to a start margin;
review that one first. This PR's own diff is the six commits on top of
it.

Most HR-Ready waits in the E2E suites gave up before the release they
were waiting on could fail its install. cozystack-api gives every
release it generates a 600s install timeout, and the Chainsaw asserts in
front of those releases sat at 5 or 6 minutes. When an install genuinely
failed, the suite reported a bare assert timeout against a release that
was still installing, and the reason the install failed, which the
release records when its own timeout fires, was never reached. Red runs
failed just as often as they should; they only said less.

I raised every HR-Ready wait above the install timeout of its own
release, by the 60s start margin #4468 introduces:

- Chainsaw asserts on releases cozystack-api generates go to 11m, across
bucket, clickhouse, etcd, external-dns, foundationdb, harbor, mongodb,
openbao, opensearch, postgres, qdrant, rabbitmq, redis, valkey and
vminstance.
- Asserts on nested `<name>-system` releases take their floor from the
template that renders them: 11m for external-dns, openbao and qdrant
(10m `spec.timeout`), 16m for `harbor-test-system` (15m). `harbor-test`
sat exactly on its 600s floor, which expires first because the assert's
clock starts before the install's; it goes to 11m too.
- The kuberture Package release takes its 10m from cozystack-operator
and goes to 11m.
- In the backup walkthroughs, the bucket waits go from 300s to 660s in
clickhouse, kafka, mariadb, mongodb, postgres, rabbitmq and redis, and
the two postgres waits go from 360s to 660s. The kafka bucket wait
passed no budget and silently took the helper's 300s default, which is
why earlier scans for literal budgets missed it. Each Chainsaw op that
runs a walkthrough grows by exactly what its waits gained (clickhouse
78m, kafka 46m, mariadb 48m, mongodb 42m, postgres 66m, rabbitmq 48m,
redis 36m), so a failure at any later wait keeps the headroom it had and
is still reported by the script instead of being cut off by the op.

Two waits stay as they are, on purpose:

- kafka: `kafka-rd` sets a 7h install timeout, which no budget inside
the 215-minute E2E job can clear. The kafka assert and the Kafka waits
in the kafka and kafka-metadata walkthroughs keep their numbers, and the
assert now says why in-line.
- `vm-instance-test`: `vm-instance-rd` disables the install wait, so its
release goes Ready as soon as helm has applied and there is no floor to
clear. Earlier lists of this problem included it by mistake.

The cost falls on red runs only: a suite whose release never goes Ready
now spends up to six more minutes before it reports, and in exchange the
report carries the release's own reason. Green runs are unchanged.
Suites run one after another in a single job capped at 215 minutes, so a
systemic failure that reddens many suites now reaches that cap sooner,
and a job cancelled at the cap loses its log and `cozyreport.tgz`. I
took that trade because a failed install that says why is worth more
than a red run that ends a few minutes earlier.

`docs/agents/e2e-testing.md` stopped being true in several places once
the waits were raised: it described most of the tree as under the floor,
mariadb as the only suite raised, the walkthrough bucket waits as under
it, and harbor's asserts as sitting on and under theirs. I rewrote those
sentences, and the one that pointed at "every wait above the band" of
5-6m, kept the start-margin wording from #4468, and added that a
walkthrough wait passing no budget takes the helper's default. The
comments in `hack/mariadb-first-boot-waits.bats` that quoted the old op
and bucket figures are restated; its existing checks and constants are
unchanged, because every op grew by exactly its bucket delta and each
remainder still sits at exactly its constant. I also added postgres to
the table that file holds the walkthroughs to, so its raised waits and
its op cannot drift back apart unnoticed, and a test that holds each
table suite's bucket wait to the bucket release's floor, read from the
api-server default with `bucket-rd` checked for an override; the op
check alone could not catch a bucket wait dropping back, since a lower
wait only widens the slack. redis stays out: its walkthrough waits on no
redis HelmRelease, only on the bucket and on the sentinel master, so the
table's floor check has nothing to read there.

How I checked it, since E2E cannot run locally:

- I re-ran the enumeration from the sizing section of
`docs/agents/e2e-testing.md` over every
`hack/e2e-chainsaw/*/chainsaw-test.yaml` and every `wait_hr_ready` call
under `examples/backups/`, literal or default, and held each against its
own release's floor. Every `Ready=True` assert and every `wait_hr_ready`
call a suite drives now clears its floor, except the kafka ones named
above. The opt-in etcd round-trip uses bare `kubectl wait hr` calls at
300s instead of that helper; they are listed in #4470. The `Ready=False`
assert on the rejected redis upgrade is not a readiness wait and is left
out.
- `make bats-unit-mariadb-first-boot-waits` passes, 31 of 31. For the
new postgres row, putting the op back to 50m fails only the postgres
slack test, and a postgres wait at 659s fails only the postgres floor
test. Setting any one table suite's bucket wait to 659s fails only the
bucket test, naming only that suite, and an install-timeout override on
`bucket-rd` fails it too. I ran the mutations under plain `bats` so a
fail-fast runner could not hide a second failure.
- `hack/select-e2e.sh` over this diff selects bucket, clickhouse, etcd,
external-dns, foundationdb, harbor, kafka, kuberture, mariadb, mongodb,
openbao, opensearch, postgres, qdrant, rabbitmq, redis, valkey and
vminstance.

Out of scope, and tracked in #4470: the tenant addon waits in
`hack/e2e-chainsaw/_lib/run-kubernetes.sh` (an edit there escalates the
whole E2E run), the redis and kafka-metadata `wait_hr_ready` helpers
that print no reason on timeout, and the opt-in etcd round-trip's bare
`kubectl wait` calls.

Known and left for later, none of them changed here:

- Only the mariadb Chainsaw asserts and the walkthroughs in the guard's
table are held by a test. The other raised asserts, and the redis and
kafka bucket waits, rest on literals with in-line reasons; #4470 covers
extending the guard.
- The Kafka waits in `examples/backups/kafka/04-create-kafka.sh`,
`07-restore-to-copy.sh` and `kafka-metadata/run-all.sh` stay at 300s
against the 7h floor without an in-line reason at the call site. The
doc's rule asks that of Chainsaw asserts only, and the kafka Chainsaw
assert carries it.

The postgres waits are the ones #3748 is about. Their budget now clears
the install-timeout floor and a test holds it there; what a CNPG first
boot costs on this platform is still unmeasured, so that issue stays
open.

Closes #3771

Closes #3753

### Screenshots

Not a UI change.

### Downstream repositories

I walked the trigger map in `docs/agents/contributing.md` against the
diff. It touches E2E suites, backup walkthrough scripts, a unit-lane
bats file and an agent doc, none of which any downstream repository
restates.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(tests): HelmRelease readiness waits in the E2E suites and backup walkthroughs now outlast the install timeout of the release they wait on, so a failed install reports its own reason instead of a bare timeout.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Tests**
* Extended readiness and backup round-trip test timeouts across multiple
services to accommodate slower startup and provisioning.
* Added checks to verify that bucket readiness waits meet configured
installation-timeout requirements.

* **Documentation**
* Updated testing guidance to clarify how readiness waits relate to
release-specific timeouts and overall test budgets.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-02 12:15:58 +03:00
Aleksei Sviridkin 49cefe0dff fix(tests): size HR-Ready asserts on nested and Package releases
A nested <name>-system release takes its install timeout from the
template that renders it, not from cozystack-api, and a Package release
takes it from cozystack-operator. The external-dns, openbao and qdrant
templates set 10m and harbor's sets 15m; the operator gives 10m. Their
asserts sat at 5m or 10m, so a failed install reported as a bare
timeout. Raise each above its own floor: 11m, and 16m for
harbor-test-system. harbor-test was on its 600s floor exactly, which
expires first because its clock starts before the install's; it goes
to 11m with the generated parents here.

kafka keeps its 6m: kafka-rd sets a 7h install timeout that no budget
inside the E2E job cap can clear, and the assert now says so.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 17:40:21 +03:00
Myasnikov Daniil 7fe713580d test(e2e): drop readiness asserts that HelmRelease Ready already covers
The qdrant and external-dns apps render their workload in a nested
-system HelmRelease, and both suites already assert that HelmRelease
Ready, which helm-controller reports only once the workload is up.
Asserting readyReplicas on the same Deployment or StatefulSet again
adds nothing, so those asserts keep only the instance-label pin.

The vminstance PVC asserts follow a DataVolume Ready assert, and a
DataVolume is not Ready before its PVC is Bound.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-01 18:03:25 +05:00
Myasnikov DaniilandClaude Opus 4.8 e22d84f2c8 test(e2e): migrate app suite from BATS to Chainsaw and wire into CI
Replaces the per-app BATS E2E suite (hack/e2e-apps/*.bats) with declarative
Kyverno Chainsaw tests under hack/e2e-chainsaw/, extending this branch's pilot
(postgres + bucket) to all 18 app suites.

- Declarative ports assert on status/conditions and concrete fields; the
  repeated "timeout ... until kubectl get" + "kubectl wait" boilerplate
  collapses into a single polling assert with structured failure diffs and
  automatic events/describe/podLogs capture on failure.
- Inherently imperative suites keep their logic in script steps: openbao
  init/unseal, kuberture split-horizon probes, vminstance, the gateway
  admission/impersonation cases, and kubernetes-latest/previous, which wrap
  the relocated hack/e2e-chainsaw/_lib/run-kubernetes.sh verbatim.
- Gateway tests derive the tenant apex from the namespace host label at
  runtime, so they are host-independent.

CI: the e2e job now runs "chainsaw test hack/e2e-chainsaw/" via a new
test-chainsaw target and uploads the JUnit chainsaw-report.xml; the chainsaw
binary is added to the e2e-sandbox image. install-cozystack and test-openapi
stay BATS. The per-app 3-retry loop is dropped, since assertion polling
replaces the fixed-timeout flakiness it papered over.

Verified against a dev cluster (DB/app, storage and VM suites pass). Service
port assertions use the ports[*].port projection form because Chainsaw
v0.2.15 mis-evaluates JMESPath number-literal port filters.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-25 13:42:34 +05:00