The checklist entry had grown into a derivation and buried the rule it
states. The rule stays in the checklist and the derivation moves to a
section of its own.
The derivation under-budgeted in two ways. It stated the 10m install
timeout as universal, while releases an app chart renders itself take
their template's spec.install.timeout, then its spec.timeout, or
helm-controller's 5m where neither is set. And it restated which kinds
kstatus gates on, which is easy to get wrong, so it now points at the
upstream table instead.
The harbor comments now name the floor each of their asserts actually
sits against, and the opensearch comment stops citing a 5-6m standard
the checklist no longer states.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
The verify-statefulset-rolled step folded two outcomes with two owners into
one assert: that the operator rewrote the nodes template onto the subPath
mount plan, and that the StatefulSet finished rolling onto it. When the roll
stalled, the timeout could not say which half did not happen. The failure
surfaced only as the status mismatch, while the volume projections in the
diff read as absent even though they were present -- the projection trap
e2e-testing.md documents, where the "actual" side of a Chainsaw diff is a
projection and the real cause is the error line above it, not the diff.
Split into two asserts so each candidate fails on its own line: the template
rewrite on a reconcile-scale budget, the roll on the pod-lifetime one. The
nodePool StatefulSet is OnDelete (opensearch-k8s-operator v2.8.0), so the
roll is the operator's rolling-restart reconciler's to drive, not the
StatefulSet controller's -- it deletes the pod itself, and currentRevision
advances only once that reconciler patches it. A failure on the first assert
is the operator not writing the volumes; a failure on the second with the
first green is the roll not completing.
The catch captures the StatefulSet as YAML and as a describe and the
operator's own log, because updateRevision, currentRevision, generation and
updatedReplicas are what separate the two halves -- describe omits them --
and the restart decision is logged in the operator, not in the cluster pods.
The roll-gate comment ties each log signature to a v2.8.0 source line: the
reconciler builds its OpenSearch client at rollingRestart.go:138, before it
logs "Starting rolling restart" at :155, and that client pings the cluster,
so the absence of the log line means an early gate held silently or the
client ping failed loudly -- both, not just the first.
The stall itself is fixed in #4317: a single-replica cluster is left
permanently yellow by an index whose replica cannot be assigned. This
change only makes the suite name which half of the roll a future failure
stalled on.
Refs #4259
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
The roll onto the new mount plan timed out on some runs and passed on
others with identical content. What differed was cluster health at the
moment the listener moved: on one data node an index that asks for a
replica stays yellow for good, and the operator restarts a pod only once
its health gate opens. The security audit log creates such an index on
its first audited event, and whether that had happened yet was down to
timing.
Create the index in the suite so the roll always has to get through a
yellow cluster, and read the health back so the step cannot keep passing
while it proves nothing.
The operator logs that health held a restart back, never what the health
was, and cluster health is not a Kubernetes object. The catch block now
reads it from the node, together with the shard table and the allocation
explanation.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Assisted-by: LLM
No suite created an OpenSearchCluster anywhere in the tree, so the whole
class of failure where the chart renders correctly and the cluster then
refuses the workload had no coverage at all. Rendering proves the
manifests; only a cluster proves that securityadmin authenticates
against the CA the chart handed the operator and that the security
configuration actually applies.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Aleksei Sviridkin