From 1e7ba63c57c5237c67e284fecbbf021cb0bbae2a Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Thu, 12 Nov 2026 16:52:11 +0300 Subject: [PATCH] docs(agents): define testing and authoring conventions Require tests for behavior changes without inventing failures for prose or unchanged upstream updates. Keep new image build filenames explicit and preserve the existing lookup policy. Correct the assertion guidance: Bats can fail on a final negation, while cozytest appends a successful return. Separate numeric validation from text coercion and account for null parent maps in Helm values. Validation: markdownlint (MD013 excluded) and git diff --check pass. Fixes: 5137d98e7119 ("test(bats): refuse a negated assertion across every suite") Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- AGENTS.md | 2 +- docs/agents/e2e-testing.md | 6 +++--- docs/agents/overview.md | 41 ++++++++++++++++++++++++++++++++++++-- 3 files changed, 43 insertions(+), 6 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 6f26a69a5..3e74ff133 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -36,7 +36,7 @@ This file provides structured guidance for AI coding assistants and agents worki - **E2E tests and E2E CI** (e.g., "write/fix an e2e test", "stabilize a flaky test", "add a bats test", "change the e2e workflow", "why did e2e fail") - Read: [`e2e-testing.md`](./docs/agents/e2e-testing.md) - - Action: Read the entire file and follow the conventions exactly — no new retries on deterministic steps, event-driven backstops before every `kubectl wait`, no test-level `EXIT`/`RETURN` traps (a self-contained one inside a Chainsaw `script` step or an explicit subshell is fine — see the doc), no assertion beginning with `!` (errexit exempts an inverted status, so it cannot fail — write `if cmd; then echo "FAIL: …"; false; fi`), fail-fast on HelmRelease readiness + - Action: Read the entire file and follow the conventions exactly — no new retries on deterministic steps, event-driven backstops before every `kubectl wait`, no test-level `EXIT`/`RETURN` traps (a self-contained one inside a Chainsaw `script` step or an explicit subshell is fine — see the doc), no assertion beginning with `!` (errexit exempts an inverted status, so it can be silently ignored — write `if cmd; then echo "FAIL: …"; false; fi`), fail-fast on HelmRelease readiness - **General questions about contributing** - Read: [`contributing.md`](./docs/agents/contributing.md) diff --git a/docs/agents/e2e-testing.md b/docs/agents/e2e-testing.md index 9ad31d967..243acdecc 100644 --- a/docs/agents/e2e-testing.md +++ b/docs/agents/e2e-testing.md @@ -181,9 +181,9 @@ Consequences to keep in mind when touching either workflow: ### 11. Never negate a BATS assertion with `!` -An assertion spelled `! cmd` inside an `@test` body cannot fail, whichever runner executes it. POSIX exempts a command whose return value is inverted with `!` from `set -e`, so the failing status is dropped where it is produced. `hack/cozytest.sh` then appends `return 0` to every test function, so it never resurfaces as the test's own status. The `bats` binary looks stricter and mostly is not: its ERR trap carries the identical exemption, and `bash -c 'set -eE; trap "echo TRAP" ERR; ! true; echo REACHED'` prints `REACHED`, runs no trap, and exits zero. The one shape `bats` does catch is a negated assertion standing **last** in the body, where the inverted status is what the function returns — put any statement below it and `bats` passes the test too. This is the same family as the ban on test-level `EXIT`/`RETURN` traps, and unlike that one it is not settled by preferring the stricter runner, because there is no stricter runner to prefer. +An assertion spelled `! cmd` inside an `@test` body can silently pass. POSIX exempts a command whose return value is inverted with `!` from `set -e`, so a failing inverted status does not stop the test body. `hack/cozytest.sh` then appends `return 0` to every test function, so it never resurfaces as the test's own status. The `bats` binary looks stricter and mostly is not: its ERR trap carries the identical exemption, and `bash -c 'set -eE; trap "echo TRAP" ERR; ! true; echo REACHED'` prints `REACHED`, runs no trap, and exits zero. The one shape `bats` does catch is a negated assertion standing **last** in the body, where the inverted status is what the function returns — put any statement below it and `bats` passes the test too. This is the same family as the ban on test-level `EXIT`/`RETURN` traps, and unlike that one it is not settled by preferring the stricter runner, because there is no stricter runner to prefer. -The exemption is also wider than the "negated pipeline" phrasing found in older comments: a bare `! true`, a negated `[ … ]`, one inside a `for` body, and one following `;`, `&&` or `||` all pass. Write the absence so that it fails — `if cmd; then echo "FAIL: "; false; fi` — and let the `false` carry the verdict. `hack/bats-no-negated-assert.bats` enforces this across every `hack/**/*.bats`, subdirectories included. Inverted **conditions** (`if !`, `elif !`, `while !`, `until !`) are untouched, because `set -e` is not what decides a condition. `[ ! -f x ]` is untouched for a different reason and stays a normal, failing assertion: the `!` there is an operator of `test`, not negation of a command, so nothing exempts its status. The scan is lexical, with the limits that implies: a statement assembled at runtime is invisible to it, and the form is counted inside a heredoc or a fixture string where no command exists — which reports a file holding nothing rather than passing one that does, and is why that guard's own fixtures build the token from a separate `printf` argument. +The exemption applies to any inverted command, not just pipelines: `! true`, negated `[ … ]`, negation inside a `for` body, and negation following `;`, `&&` or `||` do not trigger errexit. A later successful statement can hide their failure. Write the absence so that it fails — `if cmd; then echo "FAIL: "; false; fi` — and let the `false` carry the verdict. `hack/bats-no-negated-assert.bats` enforces this across every `hack/**/*.bats`, subdirectories included. Inverted **conditions** (`if !`, `elif !`, `while !`, `until !`) are untouched, because `set -e` is not what decides a condition. `[ ! -f x ]` is untouched for a different reason and stays a normal, failing assertion: the `!` there is an operator of `test`, not negation of a command, so nothing exempts its status. The scan is lexical, with the limits that implies: a statement assembled at runtime is invisible to it, and the form is counted inside a heredoc or a fixture string where no command exists — which reports a file holding nothing rather than passing one that does, and is why that guard's own fixtures build the token from a separate `printf` argument. ## Chainsaw v0.2.15 gotchas @@ -209,7 +209,7 @@ The suite is pinned to Chainsaw **v0.2.15** (the latest release as of May 2026); 7. Failure path attaches scoped diagnostics via a `catch:` block, never a silent pass. The node-join deadline is no exception, and no longer has one: with the soft-red marker gone it fails its suite like any other deadline and its diagnostics run from the same `catch:`. 8. If it touches HelmRelease install/upgrade behavior, add the `status.history` guard: fail the run on a `failed` or `uninstalled` Snapshot of a release whose install strategy is `RetryOnFailure`, fail it on a parent whose strategy is anything else, and report the footprint of every other release without failing. What is reusable today is the predicate and the jsonpath in `hack/e2e-chainsaw/_lib/remediation-guard.sh`; the policy that weighs the footprint against `.spec.install.strategy.name`, judges an empty history against the release's own readiness, and keeps each read's exit status apart from its output lives in `cozy_guard_helmrelease` in `hack/e2e-chainsaw/_lib/run-kubernetes.sh` — read it before rebuilding any of that for another suite. If the suite has a failure path that exits before the guard, give that path `cozy_report_helmrelease_remediation` as well, or the runs the guard exists for are the runs it does not see. 9. A test that creates a tenant `Kubernetes` cluster waits for its release to reach `Ready` **before the assertions it ends on**, with a `--timeout` the guard can read as positive and no longer than a day. Reads earlier in the run are expected and are not what this bounds. The mechanism, how to size the budget and what the guard does with it are under "Do not mask cleanup or teardown failures". One thing to carry from there rather than look up: the naming is part of the rule, not a detail of the check, so an end-assertion added under any name outside the `cozy_assert_*` / `cozy_switch_and_assert_*` convention is invisible to it and passes before the wait without reddening anything — as long as one recognised call site is left behind it. Rename every recognised call and the guard reds instead, on finding none at all, which is the one shape of this mistake it does catch. -10. No BATS assertion begins with `!` — an inverted status is exempt from `set -e` and the assertion cannot fail; use `if cmd; then echo "FAIL: …"; false; fi`. +10. No BATS assertion begins with `!` — an inverted status is exempt from `set -e` and can be silently ignored; use `if cmd; then echo "FAIL: …"; false; fi`. ## The container lane: what the srv nodes are, and what that costs diff --git a/docs/agents/overview.md b/docs/agents/overview.md index 9ab67c41d..74580a370 100644 --- a/docs/agents/overview.md +++ b/docs/agents/overview.md @@ -16,7 +16,7 @@ The platform exposes infrastructure services via the Kubernetes API with ready-m ## Code Layout -``` +```text . ├── packages/ # Main directory for cozystack packages │ ├── core/ # Core platform logic charts (installer, platform) @@ -50,7 +50,7 @@ The platform exposes infrastructure services via the Kubernetes API with ready-m Every package is a Helm chart following the umbrella chart pattern: -``` +```text packages/// ├── Chart.yaml # Chart definition and parameter docs ├── Makefile # Development workflow targets @@ -110,13 +110,35 @@ A chart also declares the limits on its own resource name there, with `## @name - **E2E tests:** Kyverno Chainsaw suites in `hack/e2e-chainsaw/` (one directory per app), run with `chainsaw test`. Cluster bootstrap (`hack/e2e-install-cozystack.bats`) and the OpenAPI checks (`hack/e2e-test-openapi.bats`) remain BATS. Conventions for writing and stabilising them — and the CI that runs them — live in [`e2e-testing.md`](./e2e-testing.md). - **Go tests:** standard `go test`, with Ginkgo/Gomega for controllers. +### Test first, failure path first + +These rules apply to changes with testable behavior. Documentation-only edits and unmodified upstream updates need the relevant checks, not an artificial failing test. First-party adaptations made during a dependency bump remain covered by these rules. + +1. Write the test before changing the implementation and watch it fail for the intended reason, not on a typo or a missing fixture. Then make it green, then refactor under it. +2. Cover relevant failure paths before the happy path: empty or null values, invalid input, a number where a string is expected, a missing resource, a timeout or an API error. Assert the resulting message, exit status and post-failure state required by the operation's contract. +3. Fix regressions in the implementation. Do not skip, weaken or delete a test, or silence a linter, merely to get a green run. When the intended contract changes, update or replace the obsolete assertions and retain coverage of applicable failure cases. +4. A stub needs a failing test for the intended behavior during development. Finish that behavior and make the test pass before merge; do not make the test accept an unintended not-implemented result just to get green CI. +5. A bug found along the way that is out of the PR's scope gets an issue and a test pinning the current broken behaviour, marked `FIXME(#N)`. The PR that fixes the bug flips the test to pin the fixed behaviour. + +Do not negate a BATS assertion with `!`: it can silently pass. Bats catches a failing negation only when it is the last command in the test body; `hack/cozytest.sh` appends a successful return and misses it there too. See [§11 of `e2e-testing.md`](./e2e-testing.md#11-never-negate-a-bats-assertion-with-). + +### Linters + +Fix every linter finding before pushing, the cosmetic ones included. When a rule genuinely conflicts with the project, change the linter configuration and state the reason there, instead of suppressing the finding inline. + +Markdown follows these markdownlint rules: every code fence names a language (`text` when nothing else fits), table separators have spaces around the pipes (`| --- |`), and there is a blank line before and after every list and after every heading. Line length (MD013) is the exception: prose stays one line per paragraph, as [Prose Formatting](../../AGENTS.md#prose-formatting) explains. + ## Conventions ### Helm Charts + - Follow **umbrella chart** pattern for system components - Include upstream charts in `charts/` directory (vendored, not referenced) - Override configuration in root `values.yaml` - Use `values.schema.json` for input validation and dashboard UI rendering +- An unquoted numeric value (`name: 123`) reaches a template as a number: `len` on it aborts the render with `len of type float64` (`int64` for a value given with `--set`) before any `fail` message is reached, and `if not $v` treats `0` as unset. For fields validated as text that permit numeric inputs, reject missing or null values with `kindIs "invalid"`, then coerce with `printf "%v"` and check emptiness with `eq $v ""`. Coercing null first produces the non-empty string ``. +- For integer fields, keep string inputs on their lexical validation path. Validate numeric inputs for type, integrality and bounds before conversion, then format the validated integer in decimal. Generic `printf "%v"` can emit exponent notation for a float (`4200000000` becomes `4.2e+09`), so it is not integer normalization. +- For nested values, validate or normalize each parent map before reading its children; checking only the leaf cannot prevent a nil-pointer error from a missing or null parent. **No `lookup` that copies another object into the render.** A `lookup` reads cluster state at render time, and nothing re-renders the HelmRelease when that state changes, so whatever it copied into the output (a bucket name, an endpoint, a CA, a list of StorageClasses) stays stale until something unrelated upgrades the release. Lookups of this kind already in the charts have caused failures, and several have since been moved to values, so they are not precedent: do not add one of this kind unless a maintainer has authorized that specific one. @@ -124,7 +146,18 @@ A `lookup` is fine when a repeated render gives the same output and carries noth To use a fact about the cluster instead, read it from values: `_cluster` and `_namespace` in the `cozystack-values` Secret, or `valuesFrom` on a specific Secret, whose content change is a values change and so upgrades the release. Ask whether an API is served with `.Capabilities.APIVersions.Has`, since a `lookup` on an unserved group aborts the whole render. Otherwise check at run time in a hook or Job, or let a controller own the object. +### Container images + +For a new container image built from source maintained in this repository, name the build definition `Containerfile` and pass it to `docker buildx build` with `--file` in the package `Makefile`. Keep existing `Dockerfile`s and vendored build definitions unchanged. + +### Examples and fixtures + +Illustrative addresses and names in docs, examples and test fixtures come from the reserved documentation ranges, never from a routable value that merely looks realistic: IPv4 from RFC 5737 (`192.0.2.0/24`, `198.51.100.0/24`, `203.0.113.0/24`), IPv6 from RFC 3849 (`2001:db8::/32`), domains from RFC 2606 and RFC 6761 (`example.com`, `.example`, `.test`, `.invalid`), ASNs from RFC 5398 (`64496`–`64511`, `65536`–`65551`), MAC addresses from RFC 7042 (`00:00:5E:00:53:00`–`00:00:5E:00:53:FF`). Where no reserved range exists, use an obviously synthetic placeholder such as ``. + +When a test depends on a protocol-defined address's behavior, use that value, such as `127.0.0.1` or `localhost` for a local test server. Keep network tests isolated; this exception does not permit arbitrary live endpoints. + ### Go Code + - Follow standard **Go conventions** and idioms - Use **controller-runtime** patterns for Kubernetes controllers - Prefer **kubebuilder** for API definitions and controllers @@ -170,6 +203,7 @@ Before working on these areas, read the relevant doc first: ## Things Agents Should Not Do ### Never Edit These + - Do not modify files in `/vendor/` (Go dependencies) - Do not edit generated files: `zz_generated.*.go` - Do not change `go.mod`/`go.sum` manually (use `go get`) @@ -177,13 +211,16 @@ Before working on these areas, read the relevant doc first: - Do not modify image digests in `values.yaml` (generated by build) ### Version Control + - Do not commit built artifacts from `_out` - Do not commit test artifacts or temporary files ### Git Operations + - Do not force push to main/master - Do not update git config - Do not perform destructive operations without explicit request ### Core Components + - Do not modify `packages/core/platform/` without understanding migration impact