Commit Graph
11 Commits
Author SHA1 Message Date
IvanHunters 1ca603a7b2 test(e2e): wait for the S3 gateway to accept a key before using it (#4694)
## What this PR does

The bucket e2e suite fails now and then because the S3 gateway rejects a
key that COSI already reports as granted. The error is `The access key
ID you provided does not exist in our records.` from `mc alias set`. In
job 109130131391 the BucketAccess assert finished at 21:23:43.57 and the
first `mc alias set` failed at 21:23:44.82. In job 109396174094 cap1 and
cap2 were accepted, and cap3 was rejected two seconds into the step.

Granted access is not the same as a usable key. In seaweedfs-cosi-driver
v0.3.1, `DriverGrantBucketAccess` writes the new key into
`/etc/iam/identity.json` on the filer and returns. The S3 gateway in
SeaweedFS 4.45 reloads that file later, from a filer metadata
subscription (`weed/s3api/auth_credentials_subscribe.go`). In our chart
the driver writes through `seaweedfs-filer` and the two gateways follow
`seaweedfs-filer-client`, so the change may have to pass through the
other filer replica first.

The suite now waits until the gateway accepts the key before it writes.
Every alias in both write steps goes through `await_key`, which uses the
same port-forward as the writes and so asks the same gateway pod. It
waits only while the answer is the InvalidAccessKeyId text, for at most
60s. Any other error fails at once, and so does an attempt cut off at
its 15s cap. When the window runs out, the step fails with a message
that names the BucketAccess.

One probe costs at most 77s, so both steps go from a 5m to an 8m op
timeout. The inner budgets now sum to 409s and 441s, and the budget
comments say so.

I checked the probe locally against a fake `mc`:

```text
case=ok rc=0 attempts=1 elapsed=0s
  Added `a` successfully.
case=lag rc=0 attempts=3 elapsed=5s
  Added `a` successfully.
case=other rc=1 attempts=1 elapsed=0s
  FAIL: mc alias set for BucketAccess bucket-test-admin failed (exit 1) with an error other than the key not being loaded yet
case=never rc=1 attempts=31 elapsed=60s
  FAIL: S3 gateway still rejects the key of BucketAccess bucket-test-admin after 60s of probing (the gateway never loaded it, or the driver never wrote it)
```

A hanging `mc` fails on its first attempt after 15s with exit 124.
`chainsaw lint test` from v0.2.15 accepts the file.

The read-only upload check gets fixed along the way. Before, an unloaded
`ro-user` key also made the upload fail, and that failure passed as the
expected denial.

One thing I left as it is: `await_key` is copied into both steps,
because chainsaw steps share no shell, and moving it to `_lib/` would
run every suite on each edit.

Closes #4580

### Screenshots

Not applicable, no UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
test(e2e): the bucket suite waits for the S3 gateway to accept a new key before writing with it, instead of trusting BucketAccess readiness.
```
2026-10-05 14:07:16 +03:00
Aleksei Sviridkin b306abde76 test(e2e): wait for the S3 gateway to accept a key before using it
BucketAccess turns accessGranted as soon as the COSI driver has written
the new key into the filer's IAM file. The S3 gateway loads that file
later, from a filer metadata subscription, and with two filer replicas
the change can reach the filer a gateway follows only after the other
replica passes it on. Both bucket suites call mc alias set right after
the assert, so a key the gateway has not loaded yet is rejected with
InvalidAccessKeyId and the suite goes red.

Probe the key through the same port-forward the writes use. Only the
InvalidAccessKeyId text is waited on, for at most 60s; any other error
or a timed-out attempt fails at once, and the window expiring fails
with a message naming the BucketAccess. The two steps get an 8m op
ceiling so the inner budgets still sit under it.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
Assisted-by: LLM
2026-10-02 05:47:15 +03:00
Aleksei Sviridkin 9e6cb7d0cc fix(tests): let HR-Ready asserts outlive the install they wait on
cozystack-api gives every release it generates a 600s install timeout,
and these asserts gave up at 5-6m. A failed install was therefore
reported as a bare assert timeout while the release was still
installing and had not yet recorded why it failed.

Raise them to 11m, above that floor, so the release's own reason is on
record before Chainsaw gives up. The cost is up to six more minutes on
a red run of each suite; a green run is unchanged. vm-instance-test
stays: vm-instance-rd disables the install wait, so its release goes
Ready as soon as helm has applied.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 17:40:21 +03:00
Alexey Artamonov ac92b6b95c fix(e2e): cap the bucket verify S3 reads so the step honors its timeout
A Chainsaw `script` op runs its shell under exec.CommandContext with
stdout and stderr wired to in-process buffers. When the op timeout
fires it SIGKILLs the shell process alone, not its descendants, and
cmd.Wait() does not return until every process that inherited the pipe
fds closes them. An mc that keeps retrying an S3 call after the shell
is gone holds the pipe open, so the op runs to mc's own exhaustion
instead of stopping at its ceiling. This is why the bucket suite's
verify-every-bucket-takes-a-write step ran 606s under a 300s ceiling.

verify-s3-access shares that shape, so wrap each S3 and apiserver read
in its own `timeout`: a hung read dies at a known budget and the op
timeout stays a backstop. The 30s kubectl cap matches the terminal `k`
wrapper the redis suite carries. The two `mc cp` PUTs take 45s, the
budget verify-every-bucket-takes-a-write already gives a write, because
a working PUT can still wait while the master grows a volume for a
freshly created bucket; 15s left too little headroom on that grow and
could turn a slow-but-healthy write red on an unrelated PR. The 15s cap
on the mc reads has no terminal precedent: the backup-access preflight
caps `mc cp` at 15s too, but inside a retry loop where a 124 costs one
attempt rather than the step, so it is not one. Land each secret's
encoded value in a file before base64 -d so a timed-out kubectl read
(exit 124) stops the step under set -e rather than masking as an
empty-credential mc alias set further down.

The readonly-upload deny probe branched on mc directly (if mc cp ...),
which suspends set -e, so a timeout there (exit 124) was
indistinguishable from the access-denied non-zero the probe expects: a
hung deny path would exit 124, take the false branch, and reach a green
cleanup, the exact window this change bounds. Capture the status
instead: exit 0 is the readonly regression, 124 is a timed-out probe,
both fail; any other non-zero is the expected denial.

The suite's own catch read (the `kubectl get pods` in namespace
cozy-objectstorage-controller) carried neither an op timeout nor an
inner cap, so it inherits timeouts.exec (5m) and can hold the pipe the
same way; give it a 30s inner cap and a 1m op timeout so the rule the
rest of the file follows holds for it too.

The .chainsaw.yaml notes still read the op ceiling as a reliable cut,
the old SIGKILL as a de-facto cap and a smaller timeout as guaranteeing
a kill; correct them, since the shell-only kill plus the pipe wait are
why an inner cap on every read is required rather than optional.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-22 15:16:25 +03:00
Aleksei Sviridkin c038cf25d3 test(seaweedfs): cover the bucket count a volume server can serve
A volume server started with -max 0 derives the slots it has left from
the free space on its disk, less the room its unfilled volumes still
claim, divided by the master's volume size limit, and each collection
claims a batch of those slots on every volume server the replication
spans. The number of buckets an instance can hold therefore has a
ceiling well below what the free bytes suggest, and crossing it leaves
every COSI object reporting ready while the S3 PUT times out.

The existing case creates one bucket, which sits inside that ceiling at
any sizing, so nothing here told a working ceiling from an exhausted
one. This case creates three beside the platform's own backup bucket and
requires each to accept an object, which is the only signal the
exhaustion produces.

Three rather than two because the count has to clear the ceiling whether
or not cozy-backups and the unnamed default collection have grown yet,
which this suite cannot observe.

BucketAccess reconciles separately from the claim it belongs to, and the
write step reads the Secret that access produces, so readiness of the
release and the claim is not enough to enter it. Without the access
assertions the step can fail on a missing Secret, which looks the same
in the report as the capacity failure it is here to measure.

The catch reads the master's slot accounting as well as its log. A grow
failure in the log does not say whether the shipped limit is too coarse
for the PVC or an earlier collection in the same run still holds the
budget, and the per-server Max against Volumes does. The read runs
inside the master, whose image ships curl, so it needs nothing on the
host and adds no second fixed port to the reason this suite cannot run
in parallel. That reason now covers two cases rather than one, which the
shared config says.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-16 23:16:14 +03:00
Myasnikov Daniil 4e0cfd994d test(e2e): render COSI catch output as YAML
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-22 17:27:02 +05:00
Myasnikov DaniilandClaude Fable 5 1f9555a3c6 test(e2e): resolve outstanding bot review findings
- redis: assert both StatefulSet-templated PVCs Bound (replicas: 2
  provisions two, only ordinal 0 was checked)
- gateway: || true each delete inside the three cleanup() traps — they
  run under set -eu, so one transient delete failure skipped the
  remaining cleanup commands
- bucket, backup(.disabled): guard the trap's kill against an unset
  pf_pid (script can exit before the port-forward starts)

Not addressed on purpose: the sort(ports[*].port) suggestions on
clickhouse/kafka — sort() is among the JMESPath forms chainsaw v0.2.15
(still the latest release) mis-evaluates, which is why the suites use
the ordered (ports[*].port) projection; services keep spec order.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-08 19:13:01 +05:00
Myasnikov DaniilandClaude Fable 5 d6ac62a6d9 test(bucket,harbor): fail fast on frozen BucketClaims; enforce readonly denial
Fold main's post-branch bucket/harbor bats drift into the Chainsaw
suites:

- BucketClaim bucketReady budgets tightened to 2m (bucket had the 5m
  default, harbor a stale 10m with the reversed 'can lag on a loaded
  runner' rationale): the objectstorage-controller requeues the claim
  until the backend Bucket is ready (0f1cc548b), so readiness converges
  in tens of seconds and a claim not ready after 2m is frozen — the
  tight bound surfaces the propagation regression instead of masking
  it (63d14e748)
- the bucket viewer-upload soft-xfail is promoted back to a hard
  failure: accessPolicy=readonly enforcement landed with
  seaweedfs-cosi-driver v0.3.1 (#1921, eab9b034f), so a successful
  readonly upload is now a regression, exactly what the suite's own
  TODO called for
- both catch blocks gain the backend view (cluster-scoped Buckets +
  cozy-objectstorage-controller pods) where a frozen claim actually
  surfaces

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-08 18:15:56 +05:00
Myasnikov DaniilandClaude Opus 4.8 9e2347efaf test(e2e): xfail bucket readonly-denial until driver enforces it
The viewer (readonly) S3 credential can still upload: the seaweedfs-cosi-driver
does not yet honor accessPolicy=readonly (cozystack/cozystack#1921, blocked
upstream on kubernetes-sigs/container-object-storage-interface#244). The whole
cozystack chain is correct — Bucket.spec.users.viewer.readonly: true binds the
viewer BucketAccess to the tenant-root-readonly BucketAccessClass whose
parameters.accessPolicy is readonly — but the driver grants the identity write
access anyway (verified live on dev10). The BATS original asserted the same
denial; its retries and toothless gate hid the failure, which this migration
surfaces.

Keep exercising the path and verifying admin read/write + viewer read, but emit
a ::warning:: instead of failing when the viewer can upload, so the suite is not
blocked on an unimplemented upstream feature. TODO(#1921): restore the hard
failure once the driver enforces read-only access.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-25 13:43:51 +05:00
Myasnikov Daniil 22ab9379f7 test(e2e): fix chainsaw condition assertions to filter-as-list form
The (conditions[?type == 'Ready'])[0] indexed-filter idiom fails in
chainsaw v0.2.15 with 'field not found in the input object'. Use the
filter-as-list form instead. Verified: both tests pass end-to-end
against a live cluster.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-25 13:41:18 +05:00
Myasnikov Daniil 9b56a9e4c5 test(e2e): add chainsaw pilot with postgres and bucket tests
Port two representative BATS E2E tests (postgres, bucket) to Kyverno
Chainsaw as a pilot for replacing the bash-based suite. Declarative
asserts replace the timeout/until polling boilerplate, catch blocks
add automatic failure diagnostics, and script steps cover the
imperative probes (psql, mc S3 access checks).

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-25 13:41:18 +05:00