Commit Graph
9 Commits
Author SHA1 Message Date
Myasnikov Daniil cc942fed68 test(tests): run unit suites with Bats and retain POSIX coverage
The shell translator rewrites test bodies and loses Bats reporting.
The unit lane needs complete TAP results and JUnit reports without
changing how live-cluster suites run.

Sourced production helpers still need a dash pass, and unit test bodies
need nounset without exporting shell options into subprocesses.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 11:04:00 +03:00
Aleksei Sviridkin 8cb1023788 fix(e2e): fail on a failed release of cilium and csi as on the parent
The history guard covered only the parent Kubernetes HelmRelease. A
failed action on one of the addon releases it installs left no mark
the suite inspected, so a run surfaced only the downstream symptom,
typically the node-join deadline, with nothing naming the release
behind it. The tenant CNI and CSI are the sharp cases, since they
carry a cluster-wide precondition.

The guard now reads every addon release of the parent, selected from
the cluster by name prefix so a release added later is covered too.
A "failed" or "uninstalled" Snapshot fails the run on a release whose
install strategy is RetryOnFailure: the parent, which the aggregated
apiserver stamps that way, and cilium and csi. The same footprint on
the other addons, which pair the default strategy with retries: -1,
is printed as a note and leaves the run green.

The footprint is not split into a teardown and a retried failure.
In helm-controller v1.5.0, the version the platform installs, and
unchanged in v1.5.1, a release with an active retry strategy never
remediates: atomic_release.go returns an upgrade for a failed
release before it reaches the remediation switch (lines 465-475), and
the two other paths that uninstall, a changed release target and
deletion, both call ClearHistory afterwards. On a RetryOnFailure
release an "uninstalled" Snapshot is therefore unreachable, and the
"failed" one is the only signal. A retry that later recovered still
fails the run, by design, with one blind spot the statuses cannot
close: a failed install that the very next attempt recovers is
marked superseded by Helm, because there is no deployed release to
supersede instead, and its history reads like an ordinary upgrade.

The guard runs after the assertions, while every deadline a failed
addon breaks ends the script earlier, so the same classification is
also printed without a verdict from the node-join collector and from
the exit handler. That handler is now armed at the tenant CR apply,
so the management-cluster waits before the tenant answers are
covered too. Every read is bounded and never folded into "no
history", and the parent must read back RetryOnFailure: an empty
value there means the jsonpath stopped resolving, and any other value
means the apiserver stamp is gone, either of which would turn its
footprint into a note.

The history read takes its jsonpath from HELMRELEASE_HISTORY_JSONPATH,
and the upstream-type pin accepts the bounded wrapper the read goes
through.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-03 06:59:31 +03:00
Aleksei Sviridkin 97cb2781db test(e2e): pin the release-history read against the upstream type
The test that claimed to pin the HelmRelease v2 history shape wrote a
HelmRelease document containing the field names it expected, then read
that same document back with the same jsonpath. It established that yq
can read a fixture the test itself composed.

Its own comment described the failure it was meant to catch: a future
Flux release renames the field, the jsonpath matches nothing, the guard
reports no remediation cycle, and real cycles pass the e2e assertion. A
rename moves neither the fixture nor the expression, so the test stays
green through exactly that event. The comment also named where the truth
lives, github.com/fluxcd/helm-controller/api/v2, and pinned a copy of
the author's expectations instead.

Replace it with a Go test that supplies neither half. The expression is
read out of the shell library that ships it, and the object is built
from the upstream type at the version go.mod holds, then serialized and
matched the way kubectl does. A renamed or retyped field is a compile
error or a failed match; a guard that stops reading history fails the
lookup with a message saying so. Both directions were confirmed by
mutation.

Locating the expression by searching the file is not enough on its own.
Inline in a long script, locating it means deciding which occurrences
are code and which are prose, and every way of getting that wrong fails
green: an example quoted near a broken read satisfies the pin while the
guard reads nothing.

So the expression moves into a named assignment in remediation-guard.sh,
which the guard already sources, and the test sources that library and
takes the value the shell is left holding. Deciding which text is the
assignment is then not the test's problem: an `export`, an indent, a
second assignment later in the file all resolve the way the guard will
resolve them, because the thing resolving them is the shell.

Sourcing borrows the environment along with the shell, so the snippet
unsets the name before it sources. `go test` hands the child its own
environment, and a shell that had exported this name would answer in
place of a library that stopped assigning it: the value comes back
non-empty, the empty check stays quiet, and the pin passes over a guard
that reads nothing. That is this file's own subject reaching it through
the environment rather than through the source.

Moving the expression out of the call site puts a step between what is
pinned and what runs, so the call site is checked too, and the check
starts from the call rather than from the text: the local the guard is
handed must be assigned exactly once, by an anchored assignment whose
jsonpath is the expansion of the shared name. That a well-formed read
exists somewhere in the file is a weaker claim than it reads as, since
a second assignment to the same local, or a read filling another one,
satisfies it while the guard runs a literal nothing here pins. The
expansion and not the name: kubectl echoes a braceless template back
verbatim, so a read that dropped the `$` would return its own text,
clear the script's empty-history backstop, and carry no status the cycle
check matches. The library is checked by sourcing rather than by
reading, so an assignment the shell would run is resolved by the shell
instead of by a guess about which text the shell would pick. Each rule
is pinned by its own test, so a regression names which one it opened
rather than only that extraction broke.

How kubectl's printer treats an absent path is its own configuration
rather than part of this contract, so it is left unasserted: guessing at
it would put the expectation and the fixture back in the same hands.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-22 23:41:35 +03:00
Aleksei Sviridkin ceccac27b0 test(hack): clean up bats temp dirs without EXIT traps
A `trap ... EXIT` inside an `@test` replaces the one the bats binary
installs for its own bookkeeping, so a test that then fails prints no
TAP line at all. Not `not ok`, nothing: the run ends with a warning
that N of M tests executed, and anyone judging it by grepping for
`not ok` reads a green suite. Cleanup moves to the last line of each
body instead. Both runners set -e, so a failed test leaves its scratch
directory behind, which is what you want to look at anyway.

hack/promote-retag_test.bats was failing this way already, reporting
five passes and no failures while six of its eleven tests printed
nothing. Those six invoke the script for real, and it requires
sha256sum; they pin PATH to the stub directory plus /usr/bin and /bin,
which is where coreutils installs that binary and where macOS does not,
so the failure was of the platform rather than of the code and CI never
saw it. The stub directory already supplied yq and skopeo for the same
reason and now supplies sha256sum too, resolved from absolute
candidates because the stub itself is first on PATH.

Two file headers described the trap arrangement they no longer use and
are corrected rather than left to outlive it.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-05 16:03:57 +03:00
Myasnikov DaniilandClaude Opus 4.8 e22d84f2c8 test(e2e): migrate app suite from BATS to Chainsaw and wire into CI
Replaces the per-app BATS E2E suite (hack/e2e-apps/*.bats) with declarative
Kyverno Chainsaw tests under hack/e2e-chainsaw/, extending this branch's pilot
(postgres + bucket) to all 18 app suites.

- Declarative ports assert on status/conditions and concrete fields; the
  repeated "timeout ... until kubectl get" + "kubectl wait" boilerplate
  collapses into a single polling assert with structured failure diffs and
  automatic events/describe/podLogs capture on failure.
- Inherently imperative suites keep their logic in script steps: openbao
  init/unseal, kuberture split-horizon probes, vminstance, the gateway
  admission/impersonation cases, and kubernetes-latest/previous, which wrap
  the relocated hack/e2e-chainsaw/_lib/run-kubernetes.sh verbatim.
- Gateway tests derive the tenant apex from the namespace host label at
  runtime, so they are host-independent.

CI: the e2e job now runs "chainsaw test hack/e2e-chainsaw/" via a new
test-chainsaw target and uploads the JUnit chainsaw-report.xml; the chainsaw
binary is added to the e2e-sandbox image. install-cozystack and test-openapi
stay BATS. The per-app 3-retry loop is dropped, since assertion polling
replaces the fixed-timeout flakiness it papered over.

Verified against a dev cluster (DB/app, storage and VM suites pass). Service
port assertions use the ports[*].port projection form because Chainsaw
v0.2.15 mis-evaluates JMESPath number-literal port filters.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-25 13:42:34 +05:00
Aleksei Sviridkin 39b8f0252b test(hack): rename remediation-guard bats test to match what it pins
The test body asserts .status.history[].status extraction, but the
test name still referenced the old installFailures counter (leftover
from when the guard used that field before switching to status.history
to avoid ClearFailures zeroing the counters on successful reconcile).

Address review feedback from coderabbitai on
hack/remediation-guard.bats:84: rename so grep for what the test
actually pins matches the test name.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-04-16 22:32:24 +03:00
Aleksei Sviridkin 1ddd2aaea9 fix(hack): detect remediation via status.history, not transient counters
Flux helm-controller's ClearFailures() zeroes installFailures and
upgradeFailures on every successful reconciliation (see the upstream
HelmReleaseStatus method). The previous guard ran after the HelmRelease
was Ready, at which point the counters were always 0 - the assertion
was vacuous and would have passed against a reverted fix.

Switch to .status.history, which retains per-revision release
Snapshots that survive a subsequent successful reconciliation. A
remediation cycle leaves behind a Snapshot with status=uninstalled
(the install-remediation code path) or status=failed (Helm release
failure that remediation then uninstalled). Either one signals the
race actually fired.

Rewrite the bats unit tests to cover: empty history, deployed-only,
deployed+superseded (happy path - not detected), single failed,
single uninstalled, uninstalled-then-deployed, and deployed-then-failed
(all detected). The pinned-shape test feeds a realistic HR status
snippet through yq the same way run-kubernetes.sh does via kubectl
-o jsonpath.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-04-16 21:28:11 +03:00
Aleksei Sviridkin 03426fbd71 test(hack): pin HelmRelease v2 status shape used by remediation guard
run-kubernetes.sh extracts .status.installFailures and
.status.upgradeFailures via kubectl -o jsonpath. If a future flux
release renames the counters, kubectl returns empty, the guard reports
no cycle, and e2e silently misses real remediation loops.

Add a bats unit test that feeds a pinned HelmRelease v2 status snippet
through the same jsonpath and asserts the extraction still yields the
expected values. Also leave a pointer comment in run-kubernetes.sh so a
future flux bump surfaces the version coupling in review.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-04-16 21:13:06 +03:00
Aleksei Sviridkin f87834a3ed fix(hack): group e2e remediation guard conditions correctly
Shell && and || have equal precedence and left-to-right associativity,
so the previous guard parsed as (((A && B) || C) && D) and silently
passed on the canonical failure mode: install_failures=1 with an empty
upgrade_failures.

Extract the check into helmrelease_has_remediation_cycle() in a
dedicated helper sourced from run-kubernetes.sh, and add unit tests
under hack/remediation-guard.bats that pin the expected behavior for
every combination of empty, zero, and positive counter values.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-04-16 20:48:47 +03:00