Files
cozystack/hack/e2e-chainsaw/qdrant/chainsaw-test.yaml
T
Aleksei Sviridkin 33cf6972bb test(qdrant): exercise the tenant trust anchor end to end (#4701)
## What this PR does

The qdrant e2e suite now runs with TLS on and checks the tenant trust
anchor end to end. Before this, the fixture left `tls` unset with
`external: false`, so the chart rendered no CA and no
`TenantProjection`, and nothing in e2e reached the controller or
`tenantsecrets`. Setting `tls.enabled: true` turns TLS on without the
LoadBalancer that `external: true` would allocate.

The first new step checks that the anchor is published. The
`qdrant-test-ca` sentinel has to report `Ready=True/Projected`, and
`qdrant-test.tenant-ca` has to carry both the `tenant-ca` and
`tenantresource=true` labels. The projection must hold only `ca.crt`
with the same fingerprint as the chart's CA, and `tenantsecrets` must
serve it. The key-bearing `qdrant-test-ca` and `qdrant-test-tls` must
answer NotFound there, and any other error fails the step.

The second step checks that the anchor is what decides trust. It reads
the anchor through `tenantsecrets`, port-forwards the qdrant Service and
runs `openssl s_client` with that file as the only CA, hostname check
included. A CA with the same subject and another key must fail, and so
must the runner's default trust store. A handshake failure counts only
when the output shows a verification error, so a refused connection does
not pass as a rejection. Plaintext HTTP must get no response. Curl exits
7 and 28 fail the step, and a second verified handshake afterwards
proves the forward was still alive.

TLS does not break the existing steps, because none of them talk to
qdrant. `create-qdrant`, `wait-helmreleases-ready`,
`verify-statefulset-and-storage` and `verify-pvc-reclaim-on-delete` only
read Kubernetes objects: no `http://`, no port 6333 or 6334, no curl or
exec, and the suite has no backup round trip. The release now also
renders two Issuers, two Certificates and the sentinel. The chart
switches its probes to HTTPS with `enable_tls`, so `readyReplicas: 1`
now covers a pod serving TLS.

Local checks: `chainsaw lint test` (v0.2.15) says the file is valid.
`helm template` with the fixture's spec renders 2 Certificates, 2
Issuers and 1 TenantProjection, and none without TLS. I ran the
handshake logic against a local `openssl s_server` with a CA and leaf
shaped like the chart's. The anchor passes, the wrong CA and the default
store fail on verification, a wrong hostname fails, plaintext gets
`000`, and a closed port is not counted as a rejection.

A sentinel with a missing source (`SourceNotFound`) is not tested here,
the reconciler unit tests cover it.

Review notes I left as they are:

- Through `kubectl port-forward` the local socket is always accepted, so
curl exits 7 and 28 only catch a dead forward process. A false pass
would need qdrant to answer plaintext and a pod-side blip on that one
request at the same time.
- qdrant may reset the socket on the plaintext request, and some kubectl
versions drop the whole port-forward on a pod-side error. Then the
handshake after it turns the step red, not green. If CI shows `lost
connection to pod` there, the plaintext check should get its own
port-forward.
- The suite's `catch` has no `describe` of the `TenantProjection`. The
sentinel assert prints its conditions when it fails, but a red Secret
assert or script after it would not.
- The vendored chart's `servicemonitor.yaml` hardcodes `scheme: http`,
so metrics scraping fails for any qdrant with TLS on. That was already
the case before this PR, and the suite does not check metrics. Tracked
in #4703.

cert-manager does not set owner references here, so `qdrant-test-ca` and
`qdrant-test-tls` stay in the namespace with their private keys after
the app is deleted. That is tracked in #4704. The nats half of #3706 is
tracked in #4702.

Closes #3706

### Screenshots

No UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff touches only `hack/e2e-chainsaw/qdrant/`, which no entry in the
trigger map covers.

### Release note

```release-note
test(qdrant): e2e now runs qdrant with TLS on and checks that a client trusting only the tenant-published CA reaches it, while a wrong or missing anchor and plaintext HTTP are refused.
```
2026-10-05 14:14:36 +03:00

397 lines
18 KiB
YAML

# Chainsaw port of hack/e2e-apps/qdrant.bats ("Create Qdrant").
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: qdrant
spec:
catch:
- events: {}
- describe:
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
name: qdrant-test
# The app chart wraps the workload in a nested HelmRelease named
# `<release>-system`, and the chart behind it stamps
# app.kubernetes.io/instance from its own release name. So the Qdrant pods
# carry `qdrant-test-system`, which the unsuffixed value never matched.
- podLogs:
selector: app.kubernetes.io/instance=qdrant-test-system
tail: 50
# The unsuffixed value stays, as a second collector rather than as the one
# that was replaced: the parent chart's PVC-cleanup Job
# (templates/hooks/cleanup-pvc.yaml) labels its pod template with the PARENT
# release name, so it is the one thing the old selector did reach, and the
# delete below happens inside a `try:`, so the Job is in scope while the test
# is still running.
#
# What this reaches, and when, so the capture is read for what it is: the hook
# exits non-zero when a delete or a listing failed, and
# `hook-delete-policy: hook-succeeded` does not fire for a hook that failed --
# so on exactly the runs worth investigating the pod is still there, and this
# is the account of what the cleanup could not remove. It also reaches a hook
# that is still running, where the uninstall hangs and Helm is still waiting.
# On a clean uninstall the hook succeeds and its pod is collected away, so an
# empty capture here is the ordinary case rather than a stale selector -- the
# instance-label listing below is what tells those two apart. The
# `get jobs,pods` in the cleanup step below lists the objects but reads no log.
- podLogs:
selector: app.kubernetes.io/instance=qdrant-test
tail: 50
# podLogs cannot report that it matched nothing, so an empty capture above and
# a selector that has gone stale look identical. Print the instance label of
# every pod in the namespace instead. Deliberately no selector is repeated
# here: a copy would only prove itself right, and would keep matching after
# the collector's own selector drifted.
#
# What is and is not guarded, so the next reader does not assume more:
# the `-system` value above is also pinned on a resource in a `steps:` assert
# below, so the chart renaming that label turns a step red -- that is the
# drift this pin exists to catch. It does NOT catch a wrong value typed into the
# selector here: the assert pins the label on the resource, this pins it in
# the collector, and they are two literals. It also catches nothing when the
# run fails before reaching that assert. The census below covers neither, and
# is not meant to -- it answers what the labels actually are.
- script:
# The helper bounds each kubectl attempt with --request-timeout, which on
# a degraded cluster costs about five times the flag; one call is ~75s at
# worst, and this op timeout sits above that so an inner budget is what
# fires. A SIGKILL at the op boundary would take the census out mid-write
# and yield nothing, where a kubectl that gives up leaves an error to
# print. The helper carries the rest of the rationale and is unit-tested
# in hack/pod-label-census_test.bats.
timeout: 2m
content: |
set -eu
cd ../../..
. hack/e2e-chainsaw/_lib/pod-label-census.sh
pod_label_census "$NAMESPACE"
steps:
- name: create-qdrant
try:
- apply:
file: qdrant.yaml
- name: wait-helmreleases-ready
try:
# 11m: above the 600s install timeout cozystack-api gives this release;
# see "Sizing an HR-Ready budget" in docs/agents/e2e-testing.md.
- assert:
timeout: 11m
resource:
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: qdrant-test
status:
(conditions[?type == 'Ready']):
- status: "True"
# 11m: above the 10m spec.timeout that
# packages/apps/qdrant/templates/qdrant.yaml gives this release;
# see "Sizing an HR-Ready budget" in docs/agents/e2e-testing.md.
- assert:
timeout: 11m
resource:
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: qdrant-test-system
status:
(conditions[?type == 'Ready']):
- status: "True"
- name: verify-statefulset-and-storage
try:
- assert:
timeout: 2m
resource:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: qdrant-test
# Pins the label the catch collector selects on. Asserts are subset
# matches, so this costs nothing and turns a silent drift of the
# instance label into a red step instead of an empty capture. Readiness
# is covered by the qdrant-test-system HelmRelease above.
spec:
selector:
matchLabels:
app.kubernetes.io/instance: qdrant-test-system
# Stays: the reclaim step below only proves the PVC was reclaimed if it was
# seen here first, an `error` on a PVC that never existed passes as well.
- assert:
timeout: 2m
resource:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: qdrant-storage-qdrant-test-0
status:
phase: Bound
# The trust-anchor chain from the sentinel to the tenantsecrets read path:
# the chart renders a TenantProjection beside its cert-manager CA, the
# cacert controller lifts ca.crt into the key-free <release>.tenant-ca, and
# the lineage webhook marks it tenantresource=true because the
# ApplicationDefinition selects the tenant-ca label. That last verdict is
# computed from spec.secrets, and nothing short of a cluster proves it.
- name: verify-tenant-anchor-published
try:
- assert:
timeout: 2m
resource:
apiVersion: internal.cozystack.io/v1alpha1
kind: TenantProjection
metadata:
name: qdrant-test-ca
status:
(conditions[?type == 'Ready']):
- status: "True"
reason: Projected
- assert:
timeout: 2m
resource:
apiVersion: v1
kind: Secret
metadata:
name: qdrant-test.tenant-ca
labels:
internal.cozystack.io/tenant-ca: "true"
internal.cozystack.io/tenantresource: "true"
type: Opaque
# Deterministic once the asserts above pass, so no retry.
- script:
timeout: 2m
content: |
set -eu
secret_keys() {
kubectl -n "$NAMESPACE" get secret "$1" \
-o go-template='{{if .data}}{{range $k, $v := .data}}{{$k}}{{"\n"}}{{end}}{{end}}'
}
# Exactly one key, and it is ca.crt. The source holds the CA private
# key under tls.key, so a second key here is that key leaving.
keys=$(secret_keys qdrant-test.tenant-ca | sort | tr '\n' ' ')
if [ "$keys" != "ca.crt " ]; then
echo "projection must hold exactly ca.crt, holds: $keys" >&2
exit 1
fi
# It lifted the right certificate. Compared by fingerprint, because
# the controller re-encodes ca.crt from the DER it parsed.
src=$(kubectl -n "$NAMESPACE" get secret qdrant-test-ca -o jsonpath='{.data.ca\.crt}' | base64 -d)
proj=$(kubectl -n "$NAMESPACE" get secret qdrant-test.tenant-ca -o jsonpath='{.data.ca\.crt}' | base64 -d)
if [ -z "$src" ]; then
echo "qdrant-test-ca has no ca.crt — the declared source name or key is wrong" >&2
exit 1
fi
src_fp=$(echo "$src" | openssl x509 -noout -fingerprint -sha256)
proj_fp=$(echo "$proj" | openssl x509 -noout -fingerprint -sha256)
if [ "$src_fp" != "$proj_fp" ]; then
echo "projection does not carry the chart's CA certificate" >&2
exit 1
fi
if echo "$proj" | grep -qi 'BEGIN .*PRIVATE KEY'; then
echo "projection carries PEM private key material" >&2
exit 1
fi
# The delivery path a tenant uses. Everything above reads raw
# Secrets, which only an admin can do.
anchor=$(kubectl -n "$NAMESPACE" get tenantsecret qdrant-test.tenant-ca \
-o jsonpath='{.data.ca\.crt}' | base64 -d)
if ! echo "$anchor" | grep -q 'BEGIN CERTIFICATE'; then
echo "the trust anchor is not served through core.cozystack.io/tenantsecrets" >&2
exit 1
fi
# The negative that gives the positive its meaning: the key-bearing
# cert-manager Secrets must not be served the same way. Each must
# hold tls.key first, so a renamed or emptied Secret cannot 404 for
# the wrong reason and pass. NotFound is the only acceptable
# failure: an RBAC denial or an unavailable aggregated apiserver also
# fails the read, and that is not "safely absent".
for s in qdrant-test-ca qdrant-test-tls; do
if ! secret_keys "$s" | grep -qx 'tls.key'; then
echo "$s no longer carries tls.key — this check has stopped proving anything" >&2
exit 1
fi
if err=$(kubectl -n "$NAMESPACE" get tenantsecret "$s" 2>&1 >/dev/null); then
echo "key-bearing $s is being served as a tenant secret" >&2
exit 1
fi
case "$err" in
*NotFound*) echo "ok $s: key-bearing and not served" ;;
*)
echo "tenantsecret $s did not answer NotFound, so this check proved nothing: $err" >&2
exit 1
;;
esac
done
# What the anchor is for: a client that trusts only what the tenant was
# handed reaches qdrant over TLS, and the same client fails closed when the
# anchor is wrong or missing, or when it falls back to plaintext.
- name: verify-endpoint-trusts-only-the-anchor
try:
- script:
# Inner budgets: the port-forward wait (30s) plus four handshakes and
# one plaintext request at 20s each, about 130s at worst, so an inner
# timeout fires before this ceiling does.
timeout: 3m
content: |
set -eu
workdir=$(mktemp -d)
trap '[ -n "${pf_pid:-}" ] && kill "$pf_pid" 2>/dev/null || true; rm -rf "$workdir"' EXIT
# Read through tenantsecrets, the tenant's own path, not the raw
# Secret an admin would read.
kubectl -n "$NAMESPACE" get tenantsecret qdrant-test.tenant-ca \
-o jsonpath='{.data.ca\.crt}' | base64 -d > "$workdir/anchor.crt"
# A wrong anchor with the real CA's subject and a different key, so
# a pass here cannot come from matching on the name.
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \
-keyout "$workdir/wrong.key" -out "$workdir/wrong.crt" \
-subj /CN=qdrant-test-ca -days 1 >/dev/null 2>&1
kubectl -n "$NAMESPACE" port-forward service/qdrant-test 6333:6333 >/dev/null 2>&1 &
pf_pid=$!
# TODO(e2e-replace-fixed-timeouts): nothing models a port-forward
# becoming ready; the socket opening is the only signal.
timeout 30 sh -ec 'until nc -z localhost 6333; do sleep 1; done'
# -verify_return_error turns a chain that does not build into a
# non-zero exit, and -verify_hostname checks the SAN the chart
# issues for the in-cluster Service name.
handshake() {
timeout 20 openssl s_client -connect 127.0.0.1:6333 \
-servername qdrant-test -verify_hostname qdrant-test \
-verify_return_error "$@" </dev/null >"$workdir/handshake.log" 2>&1
}
# A negative counts only when verification is what failed. A refused
# connection or a hang also exits non-zero, and reading that as
# "rejected" would pass while proving nothing.
expect_verify_failure() {
what=$1
shift
if handshake "$@"; then
echo "qdrant verified against $what; the anchor is not what decides trust" >&2
cat "$workdir/handshake.log" >&2
exit 1
fi
if ! grep -q 'verify error' "$workdir/handshake.log"; then
echo "the handshake against $what failed, but not on verification:" >&2
cat "$workdir/handshake.log" >&2
exit 1
fi
echo "ok: $what is rejected"
}
if ! handshake -CAfile "$workdir/anchor.crt"; then
echo "qdrant does not verify against the anchor published to the tenant" >&2
cat "$workdir/handshake.log" >&2
exit 1
fi
echo "ok: the published anchor verifies qdrant-test:6333"
expect_verify_failure "a CA with the same subject and another key" -CAfile "$workdir/wrong.crt"
# No anchor at all: the runner's default trust store only.
expect_verify_failure "the default trust store"
# Last, because it is the one request that is not TLS: the listener
# must not answer plaintext HTTP, or a client without the anchor
# could skip verification entirely. 000 means no HTTP response, but
# curl also prints it when nothing is listening, so a refused
# connection (7) or a timeout (28) proves nothing and fails here.
rc=0
code=$(curl --silent --output /dev/null --write-out '%{http_code}' --max-time 20 \
http://127.0.0.1:6333/readyz) || rc=$?
case "$rc" in
7|28)
echo "plaintext request never reached qdrant (curl exit $rc), so this check proved nothing" >&2
exit 1
;;
esac
if [ "$code" != "000" ]; then
echo "qdrant answered plaintext HTTP with status $code" >&2
exit 1
fi
# And the listener was still there to refuse it: the same forward
# still carries a verified handshake afterwards.
if ! handshake -CAfile "$workdir/anchor.crt"; then
echo "the port-forward died around the plaintext request, so its rejection proves nothing" >&2
cat "$workdir/handshake.log" >&2
exit 1
fi
echo "ok: plaintext HTTP gets no response (curl exit $rc)"
- name: verify-pvc-reclaim-on-delete
try:
# Issue #3059: Kubernetes never GCs StatefulSet-templated PVCs; the
# chart's post-delete cleanup hook Job must reclaim the data PVC
# asynchronously during uninstall (ordinal-anchored so a sibling's PVC is
# never deleted, 2587458bd/ae7c23240). The `error` op polls for absence
# and retries transient API errors internally, so a blip is never misread
# as "reclaimed" (2m mirrors the bats ~120s budget).
- delete:
ref:
apiVersion: apps.cozystack.io/v1alpha1
kind: Qdrant
name: test
- error:
timeout: 2m
resource:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: qdrant-storage-qdrant-test-0
catch:
- script:
# Same budgeting as the census in the test-wide catch, sized for this
# script: it makes up to three bounded calls, so ~240s at worst rather
# than the census's ~75s, and the op timeout is 5m rather than 2m to
# stay above that. Above the inner budgets either way, so an inner one
# is what fires instead of a SIGKILL that would yield nothing.
timeout: 5m
content: |
# `set -u` but NOT `set -e`: every lookup below is best-effort, the
# way the sibling redis script in the harbor suite already is. Under
# `-e` a failing `get pvc` would abort the script before the report
# runs, so the degraded cluster -- the only case this catch exists
# for -- is exactly the one that would produce no diagnosis at all.
set -u
cd ../../..
. hack/e2e-chainsaw/_lib/pod-label-census.sh
kubectl -n "$NAMESPACE" --request-timeout=15s get pvc -o wide || true
# `qdrant-test` here, unsuffixed. This runs after the parent Qdrant is
# deleted, and what it looks for is the parent chart's PVC-cleanup Job
# (templates/hooks/cleanup-pvc.yaml), which is labelled with the PARENT
# release name. The nested `-system` release owns the Qdrant workload,
# not this Job. The test-wide catch reads that Job's log on the same
# unsuffixed value; this reports whether the objects are there at all.
#
# The three outcomes are told apart by the shared helper rather than
# by a second copy of that branching: an empty `kubectl get -l` exits
# zero, so a stale selector here would print nothing and read as
# "cleanup finished", and kubectl also exits zero while logging to
# stderr, which folded into the value would print as a leftover
# object. One call feeds the report; the `-o wide` re-listing below is
# extra detail on objects already known to exist, not a second attempt
# at the question.
sel=app.kubernetes.io/instance=qdrant-test
errf=$(mktemp)
if found=$(kubectl -n "$NAMESPACE" --request-timeout=15s get jobs,pods -l "$sel" -o name 2>"$errf"); then
rc=0
else
rc=1
fi
lookup_report "$rc" "$found" "$(cat "$errf")" \
"cleanup lookup FAILED — it did not run, which is not the same as 'nothing left':" \
"cleanup selector '$sel' matched no objects: either cleanup completed, or the parent chart's label changed" \
"cleanup objects still present:"
if [ "$rc" -eq 0 ] && [ -n "$found" ]; then
kubectl -n "$NAMESPACE" --request-timeout=15s get jobs,pods -l "$sel" -o wide || true
fi
rm -f "$errf"