Aleksei Sviridkin
33cf6972bb
## What this PR does The qdrant e2e suite now runs with TLS on and checks the tenant trust anchor end to end. Before this, the fixture left `tls` unset with `external: false`, so the chart rendered no CA and no `TenantProjection`, and nothing in e2e reached the controller or `tenantsecrets`. Setting `tls.enabled: true` turns TLS on without the LoadBalancer that `external: true` would allocate. The first new step checks that the anchor is published. The `qdrant-test-ca` sentinel has to report `Ready=True/Projected`, and `qdrant-test.tenant-ca` has to carry both the `tenant-ca` and `tenantresource=true` labels. The projection must hold only `ca.crt` with the same fingerprint as the chart's CA, and `tenantsecrets` must serve it. The key-bearing `qdrant-test-ca` and `qdrant-test-tls` must answer NotFound there, and any other error fails the step. The second step checks that the anchor is what decides trust. It reads the anchor through `tenantsecrets`, port-forwards the qdrant Service and runs `openssl s_client` with that file as the only CA, hostname check included. A CA with the same subject and another key must fail, and so must the runner's default trust store. A handshake failure counts only when the output shows a verification error, so a refused connection does not pass as a rejection. Plaintext HTTP must get no response. Curl exits 7 and 28 fail the step, and a second verified handshake afterwards proves the forward was still alive. TLS does not break the existing steps, because none of them talk to qdrant. `create-qdrant`, `wait-helmreleases-ready`, `verify-statefulset-and-storage` and `verify-pvc-reclaim-on-delete` only read Kubernetes objects: no `http://`, no port 6333 or 6334, no curl or exec, and the suite has no backup round trip. The release now also renders two Issuers, two Certificates and the sentinel. The chart switches its probes to HTTPS with `enable_tls`, so `readyReplicas: 1` now covers a pod serving TLS. Local checks: `chainsaw lint test` (v0.2.15) says the file is valid. `helm template` with the fixture's spec renders 2 Certificates, 2 Issuers and 1 TenantProjection, and none without TLS. I ran the handshake logic against a local `openssl s_server` with a CA and leaf shaped like the chart's. The anchor passes, the wrong CA and the default store fail on verification, a wrong hostname fails, plaintext gets `000`, and a closed port is not counted as a rejection. A sentinel with a missing source (`SourceNotFound`) is not tested here, the reconciler unit tests cover it. Review notes I left as they are: - Through `kubectl port-forward` the local socket is always accepted, so curl exits 7 and 28 only catch a dead forward process. A false pass would need qdrant to answer plaintext and a pod-side blip on that one request at the same time. - qdrant may reset the socket on the plaintext request, and some kubectl versions drop the whole port-forward on a pod-side error. Then the handshake after it turns the step red, not green. If CI shows `lost connection to pod` there, the plaintext check should get its own port-forward. - The suite's `catch` has no `describe` of the `TenantProjection`. The sentinel assert prints its conditions when it fails, but a red Secret assert or script after it would not. - The vendored chart's `servicemonitor.yaml` hardcodes `scheme: http`, so metrics scraping fails for any qdrant with TLS on. That was already the case before this PR, and the suite does not check metrics. Tracked in #4703. cert-manager does not set owner references here, so `qdrant-test-ca` and `qdrant-test-tls` stay in the namespace with their private keys after the app is deleted. That is tracked in #4704. The nats half of #3706 is tracked in #4702. Closes #3706 ### Screenshots No UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The diff touches only `hack/e2e-chainsaw/qdrant/`, which no entry in the trigger map covers. ### Release note ```release-note test(qdrant): e2e now runs qdrant with TLS on and checks that a client trusting only the tenant-published CA reaches it, while a wrong or missing anchor and plaintext HTTP are refused. ```
Chainsaw E2E tests
The Cozystack E2E app suite, written for Kyverno Chainsaw — a declarative, Kubernetes-native E2E test framework (CNCF, part of the Kyverno project; successor to KUTTL).
This directory replaces the per-app BATS suite that used to live in hack/e2e-apps/*.bats. The CI e2e job runs every suite under hack/e2e-chainsaw/ through hack/e2e-chainsaw-run.sh after make install-cozystack; cluster bootstrap (hack/e2e-prepare-cluster.bats, hack/e2e-install-cozystack.bats) and the OpenAPI checks (hack/e2e-test-openapi.bats) remain BATS.
Layout
hack/e2e-chainsaw/
├── .chainsaw.yaml # global Configuration (namespace, timeouts, JUnit report)
├── _lib/ # shell reused by the script-heavy suites
│ ├── run-kubernetes.sh # tenant Kubernetes bringup + LB/NFS/ouroboros checks
│ └── remediation-guard.sh
├── <app>/
│ ├── chainsaw-test.yaml # one or more Test definitions
│ └── <app>.yaml # the manifests the test applies
└── ...
Suites: postgres, bucket, mariadb, mongodb, redis, valkey, qdrant, clickhouse, kafka, rabbitmq, etcd, openbao, harbor, foundationdb, external-dns, kuberture, vminstance, gateway, opensearch, monitoring, kubernetes-latest, kubernetes-previous, securitygroup, seaweedfs.
What Chainsaw buys over the BATS suite
- Readable tests — a list of resources and expected states, not shell control flow. The
timeout N sh -ec "until kubectl get ..."+kubectl waitpair that appeared ~100 times collapses into a singleassertthat polls existence and state together. - Distinguishable failure modes — "object never created", "created but never ready", and "wrong status" all surface as a structured diff instead of an opaque shell timeout.
- Automatic diagnostics —
catchblocks dump events,describe, and pod logs on any failure (the BATS suite only did this inharbor.bats, by hand). - Automatic cleanup — resources Chainsaw applied are deleted (and waited on) in reverse order;
finallycovers operator-created leftovers. - CI-ready output — a JUnit
chainsaw-report.xmlis emitted for the test summary.
Test anatomy
Each suite is a directory with a chainsaw-test.yaml (one or more Test docs) plus the manifests it applies. A step has try (operations: apply, assert, error, script, delete, …), optional catch (diagnostics on failure), and optional finally (teardown regardless of outcome).
Declarative suites assert on status.conditions and concrete fields. Inherently imperative suites keep their logic in script steps:
- openbao —
bao operator init/unsealviakubectl exec. - kuberture —
jqon Service annotations + external-dns split-horizon log assertions. - gateway — admission-rejection cases that need
kubectl --asimpersonation or error-message greps; the tenant apex is read from the namespacenamespace.cozystack.io/hostlabel at runtime so they pass on any host. - kubernetes-latest/previous — wrap
_lib/run-kubernetes.sh(Kamaji bringup, port-forward, LB/NFS/ouroboros). The scriptcds to the repo root (the test dir ishack/e2e-chainsaw/<suite>, so repo root is../../..).
Conventions / gotchas (Chainsaw v0.2.15)
- Condition asserts use the filter-as-list form, not the documented
[0]index:Thestatus: (conditions[?type == 'Ready']): - status: "True"(conditions[?type == 'Ready'])[0]: {status: "True"}idiom fails with "field not found in the input object". - A reading of a failure diff: the "actual" side is a projection onto the expected fields —
status: {}means the JMESPath expression failed, not that status is empty. Read the error line above the diff first. errorwith a bare resource ref (name only) asserts the resource does not exist (used for "PVC reclaimed", "no backup schedule", "child has no Gateway HR").scriptruns in the test directory with$NAMESPACEset to the config namespace (tenant-test).
Running locally
Requires a cluster with Cozystack installed and a tenant-test namespace (the environment hack/e2e-install-cozystack.bats produces). bucket and seaweedfs additionally need mc, nc, and jq on the host; kuberture/openbao/harbor need jq.
# install chainsaw
go install github.com/kyverno/chainsaw@latest # or: brew install kyverno/chainsaw/chainsaw
# run all suites (config picked up from .chainsaw.yaml)
cd hack/e2e-chainsaw
chainsaw test
# run a single suite
chainsaw test postgres/
kubernetes-latest/kubernetes-previous each bring up a full tenant Kubernetes cluster (~25 min, real worker nodes) and vminstance boots a nested VM — expect a long run for the full set.
CI integration
The e2e job in .github/workflows/pull-requests.yaml runs make -C packages/core/testing test-chainsaw (which execs hack/e2e-chainsaw-run.sh inside the e2e sandbox) and uploads chainsaw-report.xml. The runner calls chainsaw test once per suite directory, fails a suite whose pods outlive its cleanup by more than the default timeouts.cleanup (LEAKED-PODS suite=<dir> in the log; see docs/agents/e2e-testing.md §8 and _lib/leak-allowlist.txt for known, filed leaks), and merges the per-suite reports into that one file. A local chainsaw test as shown above runs the same tests without the leak check. The sandbox image (packages/core/testing/images/e2e-sandbox/Dockerfile) ships the chainsaw binary alongside kubectl/helm/mc/jq/nc.