The redis-tls topology check read INFO replication from both pods once
and failed the suite when a replica reported role:slave without
master_link_status:up. That value is legitimately in motion there.
Redis reports the link up only in REPL_STATE_CONNECTED, while the
operator's readiness script admits a replica on two different
predicates -- master_host off 127.0.0.1, and sync not in the transfer
phase -- so every handshake state between the two is ready and
unlinked, and a dropped link re-enters that window. Ready replicas
carry no evidence about the link at all, which is more than the step's
own comment allowed for and makes its claim of determinism wrong.
Poll the topology to a 60s deadline instead, and on expiry report each
pod's observed state along with how long it waited. Nothing models the
link declaratively -- the RedisFailover CRD defines no status, so there
is no condition to assert on and the poll carries the marker the e2e
conventions require.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The Redis driver shipped no round-trip backup/restore e2e, unlike its
ClickHouse, MariaDB and RabbitMQ siblings, so the backup to S3 and restore
data path had no automated coverage. The existing demo backed up through the
platform cozy-default BackupClass, whose endpoint is the shared system
bucket; in the e2e environment that bucket's advertised external address is
not routable, so a cozy-default round-trip cannot run there.
Rework examples/backups/redis to provision its OWN Bucket + Redis strategy +
BackupClass (mirroring the MariaDB and RabbitMQ demos), projecting the
bucket's keys and the seaweedfs CA into the Secrets the strategy references
and honouring an S3_ENDPOINT override so the backup Pod targets the
in-cluster seaweedfs. Add a redis-2-backup-roundtrip Chainsaw test that
drives run-all.sh in tenant-root and asserts the sentinel round-trips
through object storage in place and into a copy.
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
The two negative dials in the chainsaw suite, the plaintext dial at a
TLS instance and the TLS dial at the instance whose flip was rejected,
decided by whether PONG appeared on stdout with stderr discarded. A
missing pod, a 403 on exec, a wrong container name or the outer cap
firing print no PONG either, so each of them passed as the refusal
being asserted, and the flip case is the check that a rejected flip
left TLS off.
Keep what the dial printed and pass only on the refusal each listener
actually gives, verified in the shipped images: a TLS listener drops a
plaintext client at AUTH and redis-cli reports it as an I/O error; a
plaintext listener holds a ClientHello in silence, because the inline
parser's newline scan stops at a NUL in the handshake header, so the
inner cap expires and kubectl's exit-code trailer carries the 124.
That trailer is what separates the cap firing inside the pod from the
wrapper's cap firing outside it. A listener that answers PONG fails
the check, and anything else fails as a state the check did not read,
with the dial's output in the message.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Everything the TLS unit tests can only render now runs on a live
cluster. A second fixture with tls.enabled asserts that cert-manager
issues the chain, that the patched operator brings two Redis replicas
and the Sentinel Deployment to Ready through its --tls probes, that the
operator publishes its CA-only Secret under the widened ClusterRole, and
that the CA-extraction controller hands the tenant exactly ca.crt
through core.cozystack.io/tenantsecrets while the key-bearing Secrets
and the operator's copy stay off that path. The last check completes a
handshake through the rfr- service name with the same CA the tenant
receives.
A third case applies tls.enabled to a running plaintext release and
asserts the refusal end to end: the HelmRelease upgrade fails with the
schema's own message, the StatefulSet keeps its revision and both
replicas Ready, plaintext still answers and no TLS listener appears.
The TLS case also asserts the topology the operator built rather than
stopping at readyReplicas: exactly one master, a replica whose link is
up, and sentinels that moved off the bootstrap address. Ready alone
witnesses the bootstrap, since a pod still replicating from the initial
127.0.0.1 never gets there, but the readiness script exits 0 for
anything reporting role:master, so it stops witnessing once the
bootstrap is over.
Both PING calls note where the pinger credentials come from: the ACL user
the operator writes into every redis.conf it generates, and the one its
own liveness probe uses. Nothing in this tree defines it, so a reader
grepping for it here finds nothing and reads the calls as broken.
Every redis-cli call runs under the container's own timeout, and both
script operations carry an explicit one. The inner cap is the part
that matters. The flip case dials TLS at a plaintext listener that
will never send a ServerHello, and redis-cli has no handshake
deadline of its own. Chainsaw does kill the shell when its exec
budget runs out, but it captures the script's stdout and stderr
through pipes, and Go's exec then waits for every holder of the write
end to close it; the orphaned kubectl is one, so the suite would go
silent until the job ceiling instead of failing. Expiry reads as the
refusal for the negative check and as a named failure for the PINGs.
Both engine images provide /usr/bin/timeout: busybox on the sentinel,
coreutils on the data pods.
The negative half of the tenant-secret pair requires the error to say
NotFound. A 403, a timeout or an unserved CRD would otherwise be read
as proof that the Secret is not served, which is the direction that
fails silently.
The dashboard Role is asserted on the deployed object, not only in the
chart unit tests: it must reach the auth Secret and nothing else, which
is the name-granting half of the pair the label-driven projection
completes. Every kubectl in both scripts runs under a wall-clock bound,
because the per-command cap inside a container cannot fire when the exec
stream itself never establishes, and a kubectl left running holds the
script's stdout open. The plaintext StatefulSet in the flip case waits
5m rather than 2m: it measured 98s of that budget on a real run, being
the third failover the group brings up in one namespace.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- redis: assert both StatefulSet-templated PVCs Bound (replicas: 2
provisions two, only ordinal 0 was checked)
- gateway: || true each delete inside the three cleanup() traps — they
run under set -eu, so one transient delete failure skipped the
remaining cleanup commands
- bucket, backup(.disabled): guard the trap's kill against an unset
pf_pid (script can exit before the port-forward starts)
Not addressed on purpose: the sort(ports[*].port) suggestions on
clickhouse/kafka — sort() is among the JMESPath forms chainsaw v0.2.15
(still the latest release) mis-evaluates, which is why the suites use
the ordered (ports[*].port) projection; services keep spec order.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
- mongodb/redis: assert status.readyReplicas instead of status.replicas, so the
StatefulSet check verifies pods are actually Ready (matches openbao/qdrant).
- kuberture: poll for the asynchronously-populated target annotation before
asserting the annotation shape (it was a one-shot test under set -e that could
race on a slow cluster), and guard the probe-Pod phase capture so the
describe/logs diagnostics still run when the Pod never materializes.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Replaces the per-app BATS E2E suite (hack/e2e-apps/*.bats) with declarative
Kyverno Chainsaw tests under hack/e2e-chainsaw/, extending this branch's pilot
(postgres + bucket) to all 18 app suites.
- Declarative ports assert on status/conditions and concrete fields; the
repeated "timeout ... until kubectl get" + "kubectl wait" boilerplate
collapses into a single polling assert with structured failure diffs and
automatic events/describe/podLogs capture on failure.
- Inherently imperative suites keep their logic in script steps: openbao
init/unseal, kuberture split-horizon probes, vminstance, the gateway
admission/impersonation cases, and kubernetes-latest/previous, which wrap
the relocated hack/e2e-chainsaw/_lib/run-kubernetes.sh verbatim.
- Gateway tests derive the tenant apex from the namespace host label at
runtime, so they are host-independent.
CI: the e2e job now runs "chainsaw test hack/e2e-chainsaw/" via a new
test-chainsaw target and uploads the JUnit chainsaw-report.xml; the chainsaw
binary is added to the e2e-sandbox image. install-cozystack and test-openapi
stay BATS. The per-app 3-retry loop is dropped, since assertion polling
replaces the fixed-timeout flakiness it papered over.
Verified against a dev cluster (DB/app, storage and VM suites pass). Service
port assertions use the ports[*].port projection form because Chainsaw
v0.2.15 mis-evaluates JMESPath number-literal port filters.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Aleksei Sviridkin