With watch granted, kubectl delete waits for the claims to be gone, and
its default for that wait is a week. A claim held by a pod that will not
stop, or a stuck detach, would hold the post-delete hook until
helm-controller's 300s uninstall budget ran out and leave the
HelmRelease stuck uninstalling.
The PVC deletes in the clickhouse, harbor, mariadb and qdrant hooks now
wait 120s at most and report the claims still terminating by name, and
each Job carries activeDeadlineSeconds 270 with a 10s termination grace
period, the numbers packages/apps/tenant/templates/cleanup-job.yaml
documents against that budget, so a kubectl that never returns cannot
hold the hook either.
The mariadb Role no longer grants watch on the named Secrets: they carry
no finalizer, so the delete never reaches the watch, and a resourceNames
rule does not authorize a collection watch anyway.
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
Assisted-by: LLM
kubectl delete waits for the object to disappear unless told otherwise,
and that wait is a watch. The post-delete hook Roles of harbor,
clickhouse, mariadb and qdrant granted get, list and delete only, so
every uninstall logged a forbidden watch, the wait was skipped and the
hook reported success before the PVCs were actually gone.
The rules now carry watch for every resource the hooks delete, including
the named Secrets the mariadb hook removes.
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
## What this PR does
Converges qdrant onto the key-free CA trust-anchor contract (#2814), the
same way #3340 does it for nats. The chart declares a `TenantProjection`
sentinel naming its cert-manager CA Secret. The CA-extraction controller
reads that declaration and publishes a key-free copy, `ca.crt` and
nothing else, as `<release>.tenant-ca`. The qdrant ResourceDefinition
selects that projection by label so it shows up in the tenant's secret
list. No key-bearing Secret gains a tenant-visible label.
The sentinel is gated on the chart's TLS condition. Engines whose
operator mints a CA unconditionally do not need that, but qdrant only
has a CA while TLS is on, and an ungated sentinel would sit
`Ready=False`/`SourceNotFound` on every plaintext release.
This was held while the CA-extraction controller was still open. That
controller landed in #3407, so the sentinel is honoured as soon as this
merges and the hold is gone.
Tests pin the invariants that otherwise fail silently. The sentinel
carries no `ownerReferences`, so the lineage walk falls through to the
Flux release label. Neither key-bearing Certificate stamps a
tenant-visible label on its Secret. The two objects that decide which
Secrets a tenant may actually read, the dashboard Role with its binding
and the ResourceDefinition, are pinned whole rather than checked for
known-bad names: a rule granting secrets with no `resourceNames` is
unrestricted rather than narrow, and a label selector can reach a
key-bearing Secret while every name in the file stays correct, so
neither adds a name for a probe to notice.
Part of #2814.
### Follow-ups
What this PR does not do, and where each belongs instead.
- nats converges the same contract in #3340 and has no
`dashboard-resourcemap_test.yaml`, so the dashboard Role pin belongs
there too. Tracked in #3701 along with postgres, which already carries
the label selector with no guard behind it.
- The qdrant e2e fixture runs plaintext
(`hack/e2e-chainsaw/qdrant/qdrant.yaml` sets `external: false`), so
nothing in CI walks the path from the sentinel to `tenantsecrets`.
postgres has a `verify-tenant-ca-projection` step to copy.
- `hack/check-qdrant-rd-secrets.bats` is engine-agnostic apart from two
literals, the chart path and the expected Secret set. The third copy of
it is the point to parameterise over `packages/system/*-rd/cozyrds/`
instead of copying again.
### Downstream repositories
Walked the trigger map in `docs/agents/contributing.md` against the
diff, file by file. Nothing needs a manual follow-up, but not for the
reason a values-only walk would give: the website reference page for an
app is regenerated from that package's `README.md`, and this PR does
change it. `qdrant` is in the app list hardcoded in the website
`Makefile`, so the docs bot picks the new section up on the next stable
tag and there is nothing to open by hand. The Terraform provider's
schema is hand-written against `values.schema.json`, which is untouched,
and the provider offers no typed access to the new projection, the same
position it holds for every other Secret it does not model. No package
is added, renamed or removed, and the `cozyrds` file is edited in place
rather than renamed, so the ccp dependency-contract anchor still
resolves.
User-facing documentation for consuming the trust anchor ships with the
chart README, matching #3340.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
### Release note
```release-note
feat(qdrant): expose the CA certificate to tenants as a key-free `<release>.tenant-ca` Secret
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Tenant environments can securely access the Qdrant TLS CA certificate
without receiving the private key.
* TLS CA access is automatically enabled when TLS or external access is
configured.
* **Documentation**
* Expanded TLS guidance explains certificate validation and provides
commands for retrieving the tenant CA certificate.
* **Tests**
* Added coverage for TLS projections, secret visibility, dashboard
resources, and secure fail-closed secret exposure.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The qdrant and harbor post-delete hooks guarded every delete with
`|| echo "WARNING: ..." >&2`, set no `-e`, and ended on an
unconditional completion message, so the container exited 0 on
every path. Helm read that as a successful hook and the
hook-succeeded delete policy removed the Job together with the pod
that carried the warning, leaving an uninstall that reclaimed
nothing indistinguishable from one that had nothing to reclaim.
Count the failures instead and set the exit code at the end, so a
failed delete does not skip the deletes that follow it. An object
that is already gone stays a success: every delete keeps
--ignore-not-found, which exits 0 in that case. This is the shape
packages/apps/mariadb already uses for the same problem.
Both hooks also consumed the output of `kubectl get` directly --
harbor by iterating it, qdrant by piping it into grep, whose status
the pipeline then reports. A listing that failed therefore arrived
as an empty list and the sweep was skipped rather than reported.
Each listing is now captured before it is used, so a listing that
failed and one that returned nothing are different events.
A hook that exits non-zero keeps its Job and pods, which is the
point: they hold the log. Give the Job one retry rather than the
Kubernetes default of six or none at all. Every delete is
idempotent, so a second attempt re-issues an API call rather than
re-running any logic, and one short backoff is what separates a
transient apiserver error from a real one -- these failures become
visible for the first time here, and a signal that fires on noise
stops being read, including on the day it is right. Six is the far
end of the same mistake: ten minutes of backoff across seven pods
outlasts the hook timeout, and Helm reports that timeout instead of
the hook's own error.
The qdrant Chainsaw suite explained its post-delete pod collector
by quoting the guard string removed here and concluding the pod is
always collected away. That is now backwards: a failed cleanup is
exactly when the pod survives and the collector is the evidence.
Rewrite the note, and the sentence in the harbor suite pointing at
it.
hack/app-cleanup-hooks_test.bats renders both scripts and runs them
against a fake kubectl. There are eight failure paths across the two
hooks -- six in harbor, two in qdrant -- and each has a case that
goes red when that one path alone is reverted to a swallowed
warning. Message assertions are anchored on the hook's own `ERROR: `
prefix, because the fake kubectl names the object in its own
diagnostic too and an unanchored match is satisfied by the fixture.
The helm-unittest suites pin the retry limit and the hook delete
policy: adding hook-failed to that annotation hands the Job back to
Helm to delete on failure and destroys the evidence this change
exists to keep, and until these asserts that reversal left both
suites green.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Two objects decide which Secrets a tenant may read: the dashboard Role
with its binding, and the ResourceDefinition. Adding qdrant-<name>-ca
to either hands every tenant subject the CA private key — enough to
issue certificates for anything — and nothing failed when that
happened.
Each layer is pinned whole rather than checked for known-bad names,
because the ways this breaks add no name to notice: a rule granting
secrets with no resourceNames restricts nothing, and a label selector
can reach a key-bearing Secret while every name stays correct. The
files carry the reasoning, including what the pinning does not settle.
The ResourceDefinition is checked against the rendered chart rather
than the file on disk, since the template globs a directory and a
second file there ships the same way.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A qdrant release with TLS on mints a self-signed CA whose Secret also
holds the CA private key, so tenants had no way to verify the server
without being handed a signing key.
The chart now declares a TenantProjection sentinel naming that Secret.
The CA-extraction controller reads it and publishes a key-free copy,
ca.crt only, as <release>.tenant-ca; the qdrant ResourceDefinition
selects that projection by label so it surfaces in the tenant's secret
list. No key-bearing Secret gains a tenant-visible label.
The sentinel is gated on the chart's TLS condition. Unlike engines whose
operator mints a CA unconditionally, qdrant only has one while TLS is
enabled, and an ungated sentinel would sit Ready=False/SourceNotFound on
every plaintext release.
Tests pin the two invariants that fail silently: the sentinel carries no
ownerReferences, so the lineage walk falls through to the Flux release
label, and neither key-bearing Certificate stamps a tenant-visible label
on its Secret.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The post-delete hook selected PVCs with an unanchored name prefix
(qdrant-storage-<release>-). Because release names are qdrant-<app>, deleting
release "qdrant-test" matched not only its own PVC
(qdrant-storage-qdrant-test-0) but also a sibling "qdrant-test-2"'s PVC
(qdrant-storage-qdrant-test-2-0) in the same namespace — irreversibly deleting
the sibling's data (or wedging it in Terminating via pvc-protection).
Anchor the ordinal with grep -E "...-[0-9]+$" so only the release's own
data PVCs match. A release name that is a prefix of a sibling no longer
collides.
- cleanup-pvc.yaml: grep -E with anchored "-[0-9]+$".
- tests: assert the anchored selector and add a notMatchRegex guard against
regressing to the unanchored prefix.
- e2e: gate the post-delete wait on a NotFound result specifically, so a
transient API/auth error is not misread as "PVC reclaimed".
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
The previous approach routed a release-unique label through
qdrant.persistence.additionalLabels, which the upstream chart renders into
the StatefulSet's volumeClaimTemplates[].metadata.labels. That field is
immutable on an existing StatefulSet, so for any already-deployed Qdrant the
API server would reject the upgrade and the <release>-system HelmRelease
(retries: -1) would retry-fail forever — permanently wedging reconciliation.
Label changes on a volumeClaimTemplate also never apply to PVCs that already
exist, so the cleanup hook could not reclaim those instances anyway.
Switch the post-delete hook to select PVCs by the deterministic StatefulSet
naming prefix (qdrant-storage-<release>-) instead. PVC names derive purely
from the release name, so this needs no StatefulSet change (upgrade-safe) and
also reclaims PVCs from instances created before the hook existed.
- qdrant.yaml: drop the additionalLabels passthrough entirely.
- hooks/cleanup-pvc.yaml: delete by name prefix via kubectl get -o name | grep.
- tests: assert the name-prefix command instead of the label passthrough.
- e2e: assert qdrant-storage-<name>-0 is gone after delete (async-waited).
BREAKING CHANGE: deleting a Qdrant app now permanently removes its data PVC
and the vector data it holds.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
The StatefulSet data PVC (qdrant-storage-<release>-0) was left orphaned
after a Qdrant app CR was deleted. The vendored upstream chart does not
support persistentVolumeClaimRetentionPolicy, so add a post-delete PVC
cleanup hook (modeled on the MariaDB hook) and pass persistence.additionalLabels
through the HelmRelease so the data PVC carries a release-unique
app.kubernetes.io/instance label the hook selects on. The cleanup Job runs
non-root with a read-only root fs and dropped capabilities; failures are
logged but never block namespace teardown.
Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io>
RSA keys are unnecessarily large for internal PKI. ECDSA P-256 provides
equivalent security with smaller key material, faster TLS handshakes,
and smaller certificates — beneficial for high-frequency p2p connections.
CA retains rotationPolicy: Never (root CA stability).
Leaf uses rotationPolicy: Always (new key on every renewal).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
cert-manager rotates the leaf cert approximately every 11 months
(renewBefore: 720h, duration: 8760h). Qdrant reads tls.cert/key/ca_cert
at process start and does not watch for file changes, so without a
restart trigger TLS would silently break at the renewal boundary.
Add a targeted Reloader annotation (secret.reloader.stakater.com/reload)
pointing at the specific TLS Secret. This uses the targeted form, not the
auto-watch form, to limit scope to the TLS Secret only. Reloader is
installed cluster-side at packages/system/reloader.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Qdrant's upstream chart constructs bootstrap/uri using short-form pod
hostnames (<release>-N.<release>-headless). The existing wildcard SAN
only covered FQDN form, causing TLS handshake failures in multi-replica
clusters when ca_cert peer verification is enabled.
Add a range loop that emits three SAN variants per replica:
- short form: <release>-N.<release>-headless
- svc form: <release>-N.<release>-headless.<ns>.svc
- FQDN form: <release>-N.<release>-headless.<ns>.svc.<cluster-domain>
Scale-up beyond initial replicas requires cert re-issue; Reloader
(see next commit) will restart pods to pick up the new cert.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Yan Bondarenko