mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 20:23:39 +09:00
The history guard covered only the parent Kubernetes HelmRelease. A failed action on one of the addon releases it installs left no mark the suite inspected, so a run surfaced only the downstream symptom, typically the node-join deadline, with nothing naming the release behind it. The tenant CNI and CSI are the sharp cases, since they carry a cluster-wide precondition. The guard now reads every addon release of the parent, selected from the cluster by name prefix so a release added later is covered too. A "failed" or "uninstalled" Snapshot fails the run on a release whose install strategy is RetryOnFailure: the parent, which the aggregated apiserver stamps that way, and cilium and csi. The same footprint on the other addons, which pair the default strategy with retries: -1, is printed as a note and leaves the run green. The footprint is not split into a teardown and a retried failure. In helm-controller v1.5.0, the version the platform installs, and unchanged in v1.5.1, a release with an active retry strategy never remediates: atomic_release.go returns an upgrade for a failed release before it reaches the remediation switch (lines 465-475), and the two other paths that uninstall, a changed release target and deletion, both call ClearHistory afterwards. On a RetryOnFailure release an "uninstalled" Snapshot is therefore unreachable, and the "failed" one is the only signal. A retry that later recovered still fails the run, by design, with one blind spot the statuses cannot close: a failed install that the very next attempt recovers is marked superseded by Helm, because there is no deployed release to supersede instead, and its history reads like an ordinary upgrade. The guard runs after the assertions, while every deadline a failed addon breaks ends the script earlier, so the same classification is also printed without a verdict from the node-join collector and from the exit handler. That handler is now armed at the tenant CR apply, so the management-cluster waits before the tenant answers are covered too. Every read is bounded and never folded into "no history", and the parent must read back RetryOnFailure: an empty value there means the jsonpath stopped resolving, and any other value means the apiserver stamp is gone, either of which would turn its footprint into a note. The history read takes its jsonpath from HELMRELEASE_HISTORY_JSONPATH, and the upstream-type pin accepts the bounded wrapper the read goes through. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
68 lines
3.6 KiB
Bash
68 lines
3.6 KiB
Bash
# shellcheck shell=bash
|
|
# Helpers for asserting that a Flux HelmRelease did not fall into an
|
|
# install/upgrade remediation cycle during an e2e run.
|
|
#
|
|
# Background: Flux helm-controller's ClearFailures() zeroes
|
|
# .status.installFailures / .status.upgradeFailures on every successful
|
|
# reconciliation (see the upstream ClearFailures method on
|
|
# HelmReleaseStatus). That makes those counters useless for a guard that
|
|
# runs after the HelmRelease has reached Ready - the values are always 0.
|
|
#
|
|
# What survives a successful reconciliation is .status.history, a bounded
|
|
# list of release Snapshots. Each Snapshot carries a status field that
|
|
# tracks the Helm release state: deployed, superseded, failed, uninstalled,
|
|
# and so on. A remediation cycle leaves the footprint behind: a snapshot
|
|
# with status "uninstalled" (from install/upgrade remediation) or "failed"
|
|
# (a Helm action that failed). Those stay in history after a subsequent
|
|
# successful reinstall or retry, with one exception: a failed install that
|
|
# the next upgrade recovers is marked superseded by Helm, so its Snapshot
|
|
# no longer reads "failed" and nothing here can see it.
|
|
#
|
|
# "Bounded" is literal. The controller truncates history on every in-sync
|
|
# reconcile, and again before it retries or remediates a failed release:
|
|
# on a release whose strategy is a retry it keeps the newest
|
|
# five Snapshots, and on any other it keeps those down to the previous
|
|
# deployed or superseded one, falling back to that same five when history
|
|
# holds no such Snapshot to cut at. A footprint therefore outlives the
|
|
# reconcile that follows it, not an arbitrary number of them. Reading
|
|
# history right after the actions that wrote it, which is what an e2e run
|
|
# does, sees it; reading it after the release has moved on may not.
|
|
#
|
|
# helmrelease_has_remediation_cycle takes a newline-delimited list of
|
|
# snapshot statuses (whatever the caller extracted via kubectl -o jsonpath
|
|
# or equivalent) and returns 0 (detected) when any entry is "failed" or
|
|
# "uninstalled", 1 otherwise. Empty input is treated as "no history yet,
|
|
# no cycle observed". Whether a detected footprint fails a run is the
|
|
# caller's decision, because it depends on the release's own strategy:
|
|
# see cozy_guard_helmrelease in run-kubernetes.sh.
|
|
|
|
# The jsonpath a caller hands to kubectl to read those statuses, one per
|
|
# line. It lives here rather than inline at the call site so that it is a
|
|
# named assignment in a library that sources cleanly on its own, which is
|
|
# what lets internal/fluxcontract source this file and run the resulting
|
|
# value against the upstream Flux type: renaming a field upstream fails that
|
|
# test instead of turning the read into a silent empty string. How this
|
|
# assignment may be written is pinned there rather than restated here.
|
|
# Locally that test runs under `make test-controllers` rather than
|
|
# `make unit-tests`; in CI both are steps of one job.
|
|
# shellcheck disable=SC2034 # used by callers that source this file
|
|
HELMRELEASE_HISTORY_JSONPATH='{range .status.history[*]}{.status}{"\n"}{end}'
|
|
|
|
helmrelease_has_remediation_cycle() {
|
|
local statuses
|
|
statuses="$1"
|
|
if [ -z "${statuses}" ]; then
|
|
return 1
|
|
fi
|
|
# printf + grep over the pipe, rather than a heredoc plus while read.
|
|
# printf %s treats the status string as a literal payload, so any stray
|
|
# $ in a future caller's input does not trigger shell expansion. grep
|
|
# returns 0 iff at least one line matches the allowlist, which is
|
|
# exactly the contract the caller wants, so we can return its exit
|
|
# status directly.
|
|
if printf '%s\n' "${statuses}" | grep --extended-regexp --quiet '^(failed|uninstalled)$'; then
|
|
return 0
|
|
fi
|
|
return 1
|
|
}
|