A Chainsaw `script` op runs its shell under exec.CommandContext with
stdout and stderr wired to in-process buffers. When the op timeout
fires it SIGKILLs the shell process alone, not its descendants, and
cmd.Wait() does not return until every process that inherited the pipe
fds closes them. An mc that keeps retrying an S3 call after the shell
is gone holds the pipe open, so the op runs to mc's own exhaustion
instead of stopping at its ceiling. This is why the bucket suite's
verify-every-bucket-takes-a-write step ran 606s under a 300s ceiling.
verify-s3-access shares that shape, so wrap each S3 and apiserver read
in its own `timeout`: a hung read dies at a known budget and the op
timeout stays a backstop. The 30s kubectl cap matches the terminal `k`
wrapper the redis suite carries. The two `mc cp` PUTs take 45s, the
budget verify-every-bucket-takes-a-write already gives a write, because
a working PUT can still wait while the master grows a volume for a
freshly created bucket; 15s left too little headroom on that grow and
could turn a slow-but-healthy write red on an unrelated PR. The 15s cap
on the mc reads has no terminal precedent: the backup-access preflight
caps `mc cp` at 15s too, but inside a retry loop where a 124 costs one
attempt rather than the step, so it is not one. Land each secret's
encoded value in a file before base64 -d so a timed-out kubectl read
(exit 124) stops the step under set -e rather than masking as an
empty-credential mc alias set further down.
The readonly-upload deny probe branched on mc directly (if mc cp ...),
which suspends set -e, so a timeout there (exit 124) was
indistinguishable from the access-denied non-zero the probe expects: a
hung deny path would exit 124, take the false branch, and reach a green
cleanup, the exact window this change bounds. Capture the status
instead: exit 0 is the readonly regression, 124 is a timed-out probe,
both fail; any other non-zero is the expected denial.
The suite's own catch read (the `kubectl get pods` in namespace
cozy-objectstorage-controller) carried neither an op timeout nor an
inner cap, so it inherits timeouts.exec (5m) and can hold the pipe the
same way; give it a 30s inner cap and a 1m op timeout so the rule the
rest of the file follows holds for it too.
The .chainsaw.yaml notes still read the op ceiling as a reliable cut,
the old SIGKILL as a de-facto cap and a smaller timeout as guaranteeing
a kill; correct them, since the shell-only kill plus the pipe wait are
why an inner cap on every read is required rather than optional.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
A volume server started with -max 0 derives the slots it has left from
the free space on its disk, less the room its unfilled volumes still
claim, divided by the master's volume size limit, and each collection
claims a batch of those slots on every volume server the replication
spans. The number of buckets an instance can hold therefore has a
ceiling well below what the free bytes suggest, and crossing it leaves
every COSI object reporting ready while the S3 PUT times out.
The existing case creates one bucket, which sits inside that ceiling at
any sizing, so nothing here told a working ceiling from an exhausted
one. This case creates three beside the platform's own backup bucket and
requires each to accept an object, which is the only signal the
exhaustion produces.
Three rather than two because the count has to clear the ceiling whether
or not cozy-backups and the unnamed default collection have grown yet,
which this suite cannot observe.
BucketAccess reconciles separately from the claim it belongs to, and the
write step reads the Secret that access produces, so readiness of the
release and the claim is not enough to enter it. Without the access
assertions the step can fail on a missing Secret, which looks the same
in the report as the capacity failure it is here to measure.
The catch reads the master's slot accounting as well as its log. A grow
failure in the log does not say whether the shipped limit is too coarse
for the PVC or an earlier collection in the same run still holds the
budget, and the per-server Max against Volumes does. The read runs
inside the master, whose image ships curl, so it needs nothing on the
host and adds no second fixed port to the reason this suite cannot run
in parallel. That reason now covers two cases rather than one, which the
shared config says.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- redis: assert both StatefulSet-templated PVCs Bound (replicas: 2
provisions two, only ordinal 0 was checked)
- gateway: || true each delete inside the three cleanup() traps — they
run under set -eu, so one transient delete failure skipped the
remaining cleanup commands
- bucket, backup(.disabled): guard the trap's kill against an unset
pf_pid (script can exit before the port-forward starts)
Not addressed on purpose: the sort(ports[*].port) suggestions on
clickhouse/kafka — sort() is among the JMESPath forms chainsaw v0.2.15
(still the latest release) mis-evaluates, which is why the suites use
the ordered (ports[*].port) projection; services keep spec order.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Fold main's post-branch bucket/harbor bats drift into the Chainsaw
suites:
- BucketClaim bucketReady budgets tightened to 2m (bucket had the 5m
default, harbor a stale 10m with the reversed 'can lag on a loaded
runner' rationale): the objectstorage-controller requeues the claim
until the backend Bucket is ready (0f1cc548b), so readiness converges
in tens of seconds and a claim not ready after 2m is frozen — the
tight bound surfaces the propagation regression instead of masking
it (63d14e748)
- the bucket viewer-upload soft-xfail is promoted back to a hard
failure: accessPolicy=readonly enforcement landed with
seaweedfs-cosi-driver v0.3.1 (#1921, eab9b034f), so a successful
readonly upload is now a regression, exactly what the suite's own
TODO called for
- both catch blocks gain the backend view (cluster-scoped Buckets +
cozy-objectstorage-controller pods) where a frozen claim actually
surfaces
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The viewer (readonly) S3 credential can still upload: the seaweedfs-cosi-driver
does not yet honor accessPolicy=readonly (cozystack/cozystack#1921, blocked
upstream on kubernetes-sigs/container-object-storage-interface#244). The whole
cozystack chain is correct — Bucket.spec.users.viewer.readonly: true binds the
viewer BucketAccess to the tenant-root-readonly BucketAccessClass whose
parameters.accessPolicy is readonly — but the driver grants the identity write
access anyway (verified live on dev10). The BATS original asserted the same
denial; its retries and toothless gate hid the failure, which this migration
surfaces.
Keep exercising the path and verifying admin read/write + viewer read, but emit
a ::warning:: instead of failing when the viewer can upload, so the suite is not
blocked on an unimplemented upstream feature. TODO(#1921): restore the hard
failure once the driver enforces read-only access.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The (conditions[?type == 'Ready'])[0] indexed-filter idiom fails in
chainsaw v0.2.15 with 'field not found in the input object'. Use the
filter-as-list form instead. Verified: both tests pass end-to-end
against a live cluster.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Port two representative BATS E2E tests (postgres, bucket) to Kyverno
Chainsaw as a pilot for replacing the bash-based suite. Declarative
asserts replace the timeout/until polling boilerplate, catch blocks
add automatic failure diagnostics, and script steps cover the
imperative probes (psql, mc S3 access checks).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Alexey Artamonov