The Sentinels were left on redis:6.2.6-alpine when the operator moved to
the freshworks-oss fork, so that its new compiled-in default could not
move them on upgrade. 6.2 is end of life, and every instance now runs a
Sentinel one or two major versions behind the Redis it monitors.
The rollback risk behind that pin, a newer engine rewriting dump.rdb in
a format the older one cannot load, does not apply to Sentinel. It
stores no RDB or AOF, only sentinel.conf on an emptyDir that the
config-copy init container rewrites from the operator's ConfigMap on
every start, so moving back to an older image starts from a clean
config. Taking the image from the same version map as the data pods also
means one image per node instead of two.
The e2e comment that relied on busybox's timeout exit code in the
Sentinel container is updated, since that container now carries
coreutils.
Signed-off-by: Yasar Rahman <yasar.rehman@gmail.com>
Assisted-by: LLM
The CA-projection step caps each kubectl at 30s, but more capped reads
run than its 5m op can cover, so a degraded apiserver could spend the
ceiling and leave nothing but "signal: killed". The reads now share a
step budget held under the op, and a read that hits its bound or finds
the budget spent says which read it was.
Two exec checks reported the 15s redis-cli cap for any failure, sending
a reader after a slow redis when the exec was refused or the container
was missing. They now report the exit status and what the exec
printed. The step also runs under bash with pipefail: under dash a
failed Secret read took base64's zero status and surfaced as a wrong
source name. The private-key check reads the projection already
fetched, since a second read that failed made it pass.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The backup round-trips drive examples/backups, whose bucket waits gave
up at 300s and whose postgres waits at 360s, under the 600s install
timeout cozystack-api gives each of those releases. The kafka example's
bucket wait passed no budget and took the helper's 300s default. A
failed install there reported only that the wait ran out.
Raise each to 660s, and grow the Chainsaw op around every walkthrough
by what its waits gained, so a failure at any later wait keeps the
headroom it had and is still reported by the script rather than cut
off by the op. The kafka and kafka-metadata Kafka waits stay: their
7h install timeout cannot be cleared inside the E2E job cap.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
cozystack-api gives every release it generates a 600s install timeout,
and these asserts gave up at 5-6m. A failed install was therefore
reported as a bare assert timeout while the release was still
installing and had not yet recorded why it failed.
Raise them to 11m, above that floor, so the release's own reason is on
record before Chainsaw gives up. The cost is up to six more minutes on
a red run of each suite; a green run is unchanged. vm-instance-test
stays: vm-instance-rd disables the install wait, so its release goes
Ready as soon as helm has applied.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The redis-tls topology check read INFO replication from both pods once
and failed the suite when a replica reported role:slave without
master_link_status:up. That value is legitimately in motion there.
Redis reports the link up only in REPL_STATE_CONNECTED, while the
operator's readiness script admits a replica on two different
predicates -- master_host off 127.0.0.1, and sync not in the transfer
phase -- so every handshake state between the two is ready and
unlinked, and a dropped link re-enters that window. Ready replicas
carry no evidence about the link at all, which is more than the step's
own comment allowed for and makes its claim of determinism wrong.
Poll the topology to a 60s deadline instead, and on expiry report each
pod's observed state along with how long it waited. Nothing models the
link declaratively -- the RedisFailover CRD defines no status, so there
is no condition to assert on and the poll carries the marker the e2e
conventions require.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The Redis driver shipped no round-trip backup/restore e2e, unlike its
ClickHouse, MariaDB and RabbitMQ siblings, so the backup to S3 and restore
data path had no automated coverage. The existing demo backed up through the
platform cozy-default BackupClass, whose endpoint is the shared system
bucket; in the e2e environment that bucket's advertised external address is
not routable, so a cozy-default round-trip cannot run there.
Rework examples/backups/redis to provision its OWN Bucket + Redis strategy +
BackupClass (mirroring the MariaDB and RabbitMQ demos), projecting the
bucket's keys and the seaweedfs CA into the Secrets the strategy references
and honouring an S3_ENDPOINT override so the backup Pod targets the
in-cluster seaweedfs. Add a redis-2-backup-roundtrip Chainsaw test that
drives run-all.sh in tenant-root and asserts the sentinel round-trips
through object storage in place and into a copy.
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
The two negative dials in the chainsaw suite, the plaintext dial at a
TLS instance and the TLS dial at the instance whose flip was rejected,
decided by whether PONG appeared on stdout with stderr discarded. A
missing pod, a 403 on exec, a wrong container name or the outer cap
firing print no PONG either, so each of them passed as the refusal
being asserted, and the flip case is the check that a rejected flip
left TLS off.
Keep what the dial printed and pass only on the refusal each listener
actually gives, verified in the shipped images: a TLS listener drops a
plaintext client at AUTH and redis-cli reports it as an I/O error; a
plaintext listener holds a ClientHello in silence, because the inline
parser's newline scan stops at a NUL in the handshake header, so the
inner cap expires and kubectl's exit-code trailer carries the 124.
That trailer is what separates the cap firing inside the pod from the
wrapper's cap firing outside it. A listener that answers PONG fails
the check, and anything else fails as a state the check did not read,
with the dial's output in the message.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Everything the TLS unit tests can only render now runs on a live
cluster. A second fixture with tls.enabled asserts that cert-manager
issues the chain, that the patched operator brings two Redis replicas
and the Sentinel Deployment to Ready through its --tls probes, that the
operator publishes its CA-only Secret under the widened ClusterRole, and
that the CA-extraction controller hands the tenant exactly ca.crt
through core.cozystack.io/tenantsecrets while the key-bearing Secrets
and the operator's copy stay off that path. The last check completes a
handshake through the rfr- service name with the same CA the tenant
receives.
A third case applies tls.enabled to a running plaintext release and
asserts the refusal end to end: the HelmRelease upgrade fails with the
schema's own message, the StatefulSet keeps its revision and both
replicas Ready, plaintext still answers and no TLS listener appears.
The TLS case also asserts the topology the operator built rather than
stopping at readyReplicas: exactly one master, a replica whose link is
up, and sentinels that moved off the bootstrap address. Ready alone
witnesses the bootstrap, since a pod still replicating from the initial
127.0.0.1 never gets there, but the readiness script exits 0 for
anything reporting role:master, so it stops witnessing once the
bootstrap is over.
Both PING calls note where the pinger credentials come from: the ACL user
the operator writes into every redis.conf it generates, and the one its
own liveness probe uses. Nothing in this tree defines it, so a reader
grepping for it here finds nothing and reads the calls as broken.
Every redis-cli call runs under the container's own timeout, and both
script operations carry an explicit one. The inner cap is the part
that matters. The flip case dials TLS at a plaintext listener that
will never send a ServerHello, and redis-cli has no handshake
deadline of its own. Chainsaw does kill the shell when its exec
budget runs out, but it captures the script's stdout and stderr
through pipes, and Go's exec then waits for every holder of the write
end to close it; the orphaned kubectl is one, so the suite would go
silent until the job ceiling instead of failing. Expiry reads as the
refusal for the negative check and as a named failure for the PINGs.
Both engine images provide /usr/bin/timeout: busybox on the sentinel,
coreutils on the data pods.
The negative half of the tenant-secret pair requires the error to say
NotFound. A 403, a timeout or an unserved CRD would otherwise be read
as proof that the Secret is not served, which is the direction that
fails silently.
The dashboard Role is asserted on the deployed object, not only in the
chart unit tests: it must reach the auth Secret and nothing else, which
is the name-granting half of the pair the label-driven projection
completes. Every kubectl in both scripts runs under a wall-clock bound,
because the per-command cap inside a container cannot fire when the exec
stream itself never establishes, and a kubectl left running holds the
script's stdout open. The plaintext StatefulSet in the flip case waits
5m rather than 2m: it measured 98s of that budget on a real run, being
the third failover the group brings up in one namespace.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- redis: assert both StatefulSet-templated PVCs Bound (replicas: 2
provisions two, only ordinal 0 was checked)
- gateway: || true each delete inside the three cleanup() traps — they
run under set -eu, so one transient delete failure skipped the
remaining cleanup commands
- bucket, backup(.disabled): guard the trap's kill against an unset
pf_pid (script can exit before the port-forward starts)
Not addressed on purpose: the sort(ports[*].port) suggestions on
clickhouse/kafka — sort() is among the JMESPath forms chainsaw v0.2.15
(still the latest release) mis-evaluates, which is why the suites use
the ordered (ports[*].port) projection; services keep spec order.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
- mongodb/redis: assert status.readyReplicas instead of status.replicas, so the
StatefulSet check verifies pods are actually Ready (matches openbao/qdrant).
- kuberture: poll for the asynchronously-populated target annotation before
asserting the annotation shape (it was a one-shot test under set -e that could
race on a slow cluster), and guard the probe-Pod phase capture so the
describe/logs diagnostics still run when the Pod never materializes.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Replaces the per-app BATS E2E suite (hack/e2e-apps/*.bats) with declarative
Kyverno Chainsaw tests under hack/e2e-chainsaw/, extending this branch's pilot
(postgres + bucket) to all 18 app suites.
- Declarative ports assert on status/conditions and concrete fields; the
repeated "timeout ... until kubectl get" + "kubectl wait" boilerplate
collapses into a single polling assert with structured failure diffs and
automatic events/describe/podLogs capture on failure.
- Inherently imperative suites keep their logic in script steps: openbao
init/unseal, kuberture split-horizon probes, vminstance, the gateway
admission/impersonation cases, and kubernetes-latest/previous, which wrap
the relocated hack/e2e-chainsaw/_lib/run-kubernetes.sh verbatim.
- Gateway tests derive the tenant apex from the namespace host label at
runtime, so they are host-independent.
CI: the e2e job now runs "chainsaw test hack/e2e-chainsaw/" via a new
test-chainsaw target and uploads the JUnit chainsaw-report.xml; the chainsaw
binary is added to the e2e-sandbox image. install-cozystack and test-openapi
stay BATS. The per-app 3-retry loop is dropped, since assertion polling
replaces the fixed-timeout flakiness it papered over.
Verified against a dev cluster (DB/app, storage and VM suites pass). Service
port assertions use the ports[*].port projection form because Chainsaw
v0.2.15 mis-evaluates JMESPath number-literal port filters.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Yasar Rahman