Commit Graph
1551 Commits
Author SHA1 Message Date
Aleksei Sviridkin 19eaff6a30 test(cozystack-basics): pin which policy each route binding names
A ValidatingAdmissionPolicyBinding whose policyName points at another
policy renders without complaint and leaves its own policy inert. The
suite pinned validationActions on the bindings but never policyName, so
pointing the GRPCRoute binding at the HTTPRoute policy kept all 56 cases
green while the GRPCRoute control did nothing.

uncovered() assigned to the same variable names its caller reads back
when printing the mismatch, so a future call site with the arguments
swapped would print a diagnostic naming the wrong side.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-03 22:17:29 +03:00
Aleksei Sviridkin 70a675fd6b feat(cozystack-basics): gate GRPCRoute hostnames in tenant namespaces
The route hostname policy covered HTTPRoute and TLSRoute. GRPCRoute
carries spec.hostnames too, and it can reach a tenant Gateway: the
port-443 listeners pin allowedRoutes.kinds to HTTPRoute and TLSRoute,
but the port-80 listener leaves kinds unset, and an unset list resolves
to the kinds the listener protocol supports, which for HTTP includes
GRPCRoute. A GRPCRoute in a tenant namespace could therefore claim any
hostname on the plaintext listener with nothing checking it.

Tenants hold no gateway.networking.k8s.io RBAC, so this is the same
defence-in-depth against a chart bug that the other two rules are, not a
tenant-user control. The new rule is the HTTPRoute rule with grpcroutes
in place of httproutes and the single served version the bundle
provides; the coverage guard pins that list to the bundle like the
others.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-03 22:17:28 +03:00
Aleksei Sviridkin 6ddae65b45 test(cozystack-basics): cover the Gateway listener policy in the guard
cozystack-gateway-hostname-policy names the gateways versions literally,
the same way the two route policies do, and the set it must cover lives
in the vendored bundle and moves on `make update` there. It matched the
served set already; the guard now says so on every run instead of
leaving one of the three hostname policies unpinned.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-03 22:17:04 +03:00
Aleksei Sviridkin 9311a48f6a test(cozystack-basics): make the version-coverage guard prove it still compares
The guard added alongside the TLSRoute version fix only ever ran against a
tree that already satisfied it. A passing check that is checking nothing
looks exactly like a passing check that is checking something, so a later
refactor could empty out the comparison and the suite would stay green
forever. That is the same shape as the negated assertions that cannot fail
elsewhere in this repository: the assertion survives, the guarantee does not.

Drive the checker with synthetic input instead of only with the live tree,
the way hack/bats-no-exit-trap.bats and hack/md-no-hardwrap.bats feed their
checkers a broken sample and require a complaint. Two layers need it
separately. The comparison itself moves into uncovered() so a fixture can
call it with a served set the named set does not cover. Above that,
assert_covers runs in a subshell with both readers replaced, so the wiring,
the empty-set guards and the exit status are exercised too -- covering only
the inner function leaves the outer one free to become a no-op while every
other case in the file stays green, because the live-tree cases are its only
callers and the tree they read is correct.

Each fixture requires the complaint it expects rather than any non-zero exit.
A renamed helper exits 127 and an unbound variable under set -u exits 1, and
either would read as a successful rejection if only the status were checked;
a rejection for the wrong reason is the same false comfort as no fixture, and
harder to notice because the red looks earned.

Each way of breaking the guard now fails at its own case. Emptying
uncovered(), reducing assert_covers to a no-op, and deleting the empty-set
guards each turn a different fixture red while the live-tree cases stay
green, and changing only the wording of assert_covers' complaint -- which
still rejects, just for an unstated reason -- turns the omission fixture red
on the message rather than passing on the exit status.

Select a resourceRules entry by comparing each resource name for equality,
and match versions as fixed strings rather than patterns. yq's contains()
compares array elements by substring, so a rule scoped to a subresource
would have satisfied a search for the parent: a rule naming tlsroutes/status
answers a query for tlsroutes and donates its apiVersions to the set the
guard checks coverage against. That direction is the wrong one for a
fail-closed check -- it makes coverage succeed where the real rule falls
short. No rule in this policy is scoped that way today.

Also pin apiGroups on the TLSRoute document in the chart suite. The HTTPRoute
document has always had that assertion; the asymmetry meant the case named
after both kinds only checked the group for one of them.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-09-03 22:17:04 +03:00
Aleksei Sviridkin 7ba22369d4 fix(cozystack-basics): match every served TLSRoute version in the hostname policy
The tenant route hostname policy names v1alpha2 alone while the vendored
Gateway API bundle serves tlsroutes at v1, v1alpha2 and v1alpha3: three
served versions against one named. The gap is not new. The v1.5.1 bundle
this package shipped before and the v1.6.1 one it ships now serve the
same three versions, with the same two marked deprecated, so nothing in
this change created or widened it -- a bundle bump is simply when
someone last opened the file.

Coverage of the two unnamed versions rested entirely on request
conversion. matchConstraints.matchPolicy is unset on both rules in this
file and so defaults to Equivalent, which rewrites a request submitted
under any served version into the one the rule names before the policy
sees it. That is why there is no bypass to demonstrate, and equally why
the arrangement is brittle: the conversion has nothing left to convert
into once no named version is served, and a rule that selects nothing is
not caught by failurePolicy: Fail, because nothing failed -- there was
simply no match.

Name all three served versions, which is the shape the HTTPRoute rule
directly above already uses. The defect is a version list drifting from
the bundle it must track, so a longer list on its own would not catch
the next drift: add a check that reads the served versions out of the
vendored bundle and fails when a policy does not cover every one of
them, in either direction. Both rules get an exact-value assertion in
the chart's own suite, so the coverage check and the pinned value fail
for different reasons and neither stands alone.

The header comment being replaced claimed TLSRoute stays at v1alpha2
through Gateway API v1.4-v1.5 and that its promotion to v1 was still
upstream-only. TLSRoute has been served at v1 since v1.5.0, so that was
already false, and false in the direction that hid this gap: a reader
who believed it had no reason to look at the version list at all.

Widening the rule then falsified every remaining place that restated
which versions it matches -- the template's own header, the
security-model layer list in packages/extra/gateway/README.md, and the
name of the test case that asserts the value. Rather than resync three
copies of a list that will move again, they now say which kinds are
gated and point at the rule and its guard for the versions.

A comment in the same test file now states only the behaviour it
guards: an earlier form of the CEL used `... ? true : (...)`, which
allowed routes in any tenant-* namespace whose host label was missing
or scrubbed.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-09-03 22:17:01 +03:00
Aleksei Sviridkin 4758718d66 fix(mariadb): separate the chart-chosen Secret suffixes with a dot
The chart asks cert-manager for two Secrets and names them after the
release: "<release>-ca-tls" for the CA and "<release>-tls" for the leaf.
The release prefix is "mariadb-", so an application named foo produces
mariadb-foo-ca-tls, and an application named foo-ca produces
mariadb-foo-ca-tls as well. One release's CA private key and another
release's server private key claim the same object. Two sibling
applications in one tenant namespace are enough to hit it: whichever
cert-manager writes last owns the Secret, and each instance then serves
or trusts material issued for the other.

The suffix set is what makes it ambiguous. It holds both "-tls" and
"-ca-tls", and one is a tail of the other, so a hyphen leaves nothing to
say where the release name ends. A dot does: an application name is
validated as a DNS-1035 label and the release prefix as ^[a-z0-9-]*$, so
neither can contain one, and "<release>.ca-tls" and "<release>.tls" split
uniquely. Secret names are DNS-1123 subdomains, which admit the dot.

Only the two suffixes this chart chooses move. The Certificate and Issuer
object names keep the hyphen, because their suffixes are not a pair where
one is a tail of the other, and the operator-managed
"<release>-ca-bundle" that the trust-anchor projection reads is not this
chart's to rename.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-03 20:13:17 +03:00
Andrey Kolkov e94fcf8219 feat(backups): RabbitMQ definitions backup via a dedicated strategy driver (#3990)
## What this PR does

Adds backup/restore support for the managed `RabbitMQ` application.
RabbitMQ's cluster operator ships no backup CRD and RabbitMQ has no
consistent online export of message payloads, so this backs up broker
**definitions** (vhosts, users, permissions, queues, exchanges,
bindings, policies, parameters) over the management HTTP API
(`GET`/`POST /api/definitions`) and stores them in the system bucket.
Message payloads and point-in-time recovery are intentionally out of
scope; a full-data backup would be a Velero volume snapshot, which is a
separate strategy.

It is delivered as a **first-class
`strategy.backups.cozystack.io/Rabbitmq` driver**, following the
Altinity pattern (a dedicated driver that renders a one-shot Job for an
HTTP-API engine):

- `reconcileRabbitmq` / `reconcileRabbitmqRestore` render the strategy's
`PodTemplateSpec` — `curl` `GET`/`POST /api/definitions`,
pushing/pulling the JSON to S3 with `curl --aws-sigv4` — as a
`batch/v1.Job`, covering both restore flows: in-place (into the source
broker) and to-copy (into a separate, differently-named RabbitMQ in the
same namespace, authenticating with the target's own default-user
credentials and reading the source's object key).
- A **Go-read precondition** gates each run on the `RabbitmqCluster`
reporting `AllReplicasReady` (read through a small `rabbitmqtypes`
partial type), so a not-yet-ready broker surfaces a precise
`Ready=False` wait on the BackupJob/RestoreJob rather than a Job that
starts and fails against an unreachable management API.
- The produced `Backup` records the stored object's location on
`status.artifact.uri`, rendered from the strategy's
`spec.artifactURITemplate`. The `Backup` CRD has no status subresource,
so this persists on Create, mirroring the Etcd driver.

Dispatch is wired in both controllers (plus
`supportedBackupStrategyKinds`), `rabbitmqtypes` is registered in the
scheme, and RBAC grants read on `rabbitmqclusters.rabbitmq.com` (the Job
itself is already covered by the existing `batch/jobs` grant). It ships
`cozy-default-rabbitmq` (a `$bucketName`-gated `Rabbitmq` strategy) and
routes `apps.cozystack.io/RabbitMQ` to it in the `cozy-default`
BackupClass, which also enrols it in the `DefaultObjectsGate`. The
RabbitMQ application chart is left untouched — it had no prior backup
wiring, and the strategy carries the S3 coordinates itself.

Unit tests cover the driver's pure helpers (applicationRef validation,
cluster-name derivation, the readiness-message builder, artifact-URI
rendering, parameter round-trip) and the reconcile precondition gate
(defers with `Ready=False` and no Job until the cluster is ready, then
proceeds). A runnable example under `examples/backups/rabbitmq/` and an
honest-roundtrip Chainsaw e2e under `hack/e2e-chainsaw/rabbitmq/` seed a
sentinel vhost, back it up, drop it, and assert it round-trips through
object storage on both restore flows.

The Job pod reuses the existing `platform-migrations` image (curl +
ca-certificates) under a new
`backupStrategyController.rabbitmqBackupClientImage` key, stamped in
lockstep with `chBackupClientImage` in `packages/core/platform/Makefile`
so the repository stays pinned at a single digest
(`hack/image-pin-consistency.bats`). A dedicated slim image built by
this package is a follow-up.

**Validation:** `go build`/`go vet`/`gofmt`/`go test
./internal/backupcontroller/... ./internal/template/...`, deepcopy +
Rabbitmq CRD generation, `helm template` proving the CRD-ordering gate
both on and off, `helm lint`, and `helm unittest` (18/18). The backup +
in-place + to-copy roundtrip was verified end-to-end on a dev cluster (a
real management-API export and a real S3 `--aws-sigv4` upload/download
against SeaweedFS), so the driver's core contract is exercised on a real
cluster; the in-CI Chainsaw run is the remaining automated evidence. The
object-cleanup, terminating-namespace teardown and readiness-deadline
paths added during review are unit-tested (including `Reconcile`-level
finalizer retention and the terminating-namespace release) but not yet
cluster-exercised. (Note found while testing: RabbitMQ 4.2 is OOMKilled
on the `nano` preset, so the example/e2e use `c1.small`.)

### Screenshots

Not applicable — no UI changes.

### Downstream repositories

Walked the trigger map against the diff. This is an additive backup
strategy plus a new strategy CRD; no downstream repository is
mechanically forced by the change. A documentation page for RabbitMQ
backup on [cozystack/website](https://github.com/cozystack/website)
would be a reasonable follow-up, but it is not required by this diff —
leaving the boxes unticked for a human to decide rather than ticking "no
downstream" to clear the checklist.

### Release note

```release-note
feat(backups): add RabbitMQ definitions backup/restore via a dedicated strategy.backups.cozystack.io/Rabbitmq driver; it gates each run on the RabbitmqCluster being ready and records the stored object on Backup.status.artifact.uri.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added RabbitMQ backup and restore support for definitions, including
vhosts, users, queues, exchanges, bindings, policies, and related
settings.
  * Added in-place and restore-to-copy workflows with readiness checks.
  * Added scheduled and on-demand backup examples.
  * Integrated RabbitMQ backups with the default backup configuration.
* Backups store definitions only; message payloads and point-in-time
recovery are not included.
* **Documentation**
  * Added a complete RabbitMQ backup and restore demonstration guide.
* **Tests**
* Added end-to-end coverage for readiness and backup/restore round
trips.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-03 16:03:46 +04:00
myasnikovdaniil 2b99853e6d fix(networking): prevent webhook admission outages during namespace churn (#3997)
## What this PR does

Moves ingress-nginx and kubeovn-webhook pod deny rules to source-side
cilium policies, and keeps a target-side `ingressDeny` on the webhook
pods beside them, so enforcement does not depend on which component
performs the service DNAT. Pod access to both webhooks stays denied,
ingress-nginx keeps exception for konnectivity agent (now keyed on its
ServiceAccount instead of `k8s-app` label that any pod can set on
itself), and external `world` deny stays as static target rule.

Source-side rules deny only backend port. That covers ClusterIP traffic
too, because cilium translates service to its backends before it
enforces egress policy, and that ordering exists because
`packages/system/cilium/values.yaml` sets `kubeProxyReplacement: true`
against upstream default of `"false"`. The value is operator-reachable,
so the target-side rule is what keeps the deny working if it is ever
flipped, and two render-time beacons make a flip visible instead of
silent. Naming the frontend with `toServices` is not possible, and
templates now say why, because the construct looks correct and fails
silently: cilium resolves `toServices` only in allow rules
(`pkg/policy/k8s/service.go` walks `spec.Egress`, never
`spec.EgressDeny`), so such `egressDeny` gets non-nil empty L3 selector
from `pkg/policy/utils/parserules.go`, and
`pkg/policy/types/policyentry.go` treats that as wildcard when rule
carries `toPorts`.

This does not remove the namespace-churn map writes. The target-side
rule carries a cluster-spanning peer, so those writes come back. What
changes is that they stop being load-bearing, because whichever end is
degraded the other still denies. Cost is policy-map cardinality on the
two webhook endpoints, and residual risk is an endpoint near
`MaxEntries` getting the inverted add-before-delete order
(`pkg/endpoint/bpf.go:1168-1177`).

Both policies also change kind and name: namespaced
`CiliumNetworkPolicy/ingress-nginx-admission-webhook` and
`CiliumNetworkPolicy/kube-ovn-webhook` become cluster-scoped
`CiliumClusterwideNetworkPolicy/ingress-nginx-admission-webhook-c5d94713a379`
and `kube-ovn-webhook-c2bdbc146112`. Anything keyed on the old
namespaced objects goes quiet with no error.

## Tests

Package suites pin the deny shape per package, one rule per spec with a
destination selector cilium can actually resolve, and pin the peer shape
of the target-side rule against the two substitutions that would deny
the API server (`fromEntities: [all]`, and dropping the peer). Checked
by mutation, all of those go red. `matchRegexRaw` and `notMatchRegexRaw`
are not used, helm-unittest 1.0.3 ignores both.

`packages/system/kubeovn-webhook` had no `test` target in Makefile, and
`hack/helm-unit-tests.sh` finds suites by probing `make -n test` per
package, so all three of its suites never ran in CI, including port
agreement guard that exists to catch a drift failing open. Added the
target, plus `.PHONY: test` to it and to cilium.

E2e churn case proves webhooks stay available. Second case proves they
are still unreachable from pods, and that deny is confined to webhook
port. Both now gate every verdict on the admission endpoints having
ready addresses before, during and after, on controller pod identity and
restart count being equal across the window, and on a positive control
where the same pods' own 443 must stay reachable. So a dead or flapping
backend fails as setup failure instead of reporting a deny that was
never enforced. First run of these cases reported a false verdict for
exactly that reason.

## Not verified

Original outage in #3983 was not reproduced in isolation, and its
symptom argues against the mechanism this PR was written around.
`connect: no route to host` is `EHOSTUNREACH`, and a policy denial
cannot produce it, since a deny drops after `connect()` has already
returned and the caller sees a timeout. On a ClusterIP that errno is
what cilium's socket LB returns with no usable backend, and socket LB is
active in host netns because `pkg/kpr/kpr.go:47-49` forces it whenever
kube-proxy replacement is set. Namespace deletion churns EndpointSlices
and service backend maps as well as identities, which is directly on
that path. So this change is justified as removing a structural weakness
and adding a topology-independent deny, not as a confirmed root cause
fix. What would settle it: `cilium-dbg monitor --type drop` on the
controller's node during the window, agent log checked for the
map-overflow warning, and the admission Service's ready backend count
across those seconds.

`world` deny that issue named as its first suspect is kept, on `cilium
monitor` evidence recorded when kubeovn-webhook policy was written - api
server arrives with identity `host` on same node and `kube-apiserver`
from another, never as `world`. That is one topology, a kube-ovn-chained
management cluster, and it is not re-derivable on cilium-primary
variants or on a Kamaji guest cluster without a run.

Whether cilium without kube-proxy replacement is a supported
configuration is a maintainer decision and is open. If it is not
supported, a `fail()` in the cilium chart is the right shape and should
follow this PR.

Refs #3983.

### Screenshots

Not applicable.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(networking): enforce the Ingress and Pod admission webhook deny on both the caller and the webhook endpoint, so it no longer depends on Cilium performing service load-balancing. Both policies become cluster-scoped: CiliumNetworkPolicy/ingress-nginx-admission-webhook and CiliumNetworkPolicy/kube-ovn-webhook are replaced by CiliumClusterwideNetworkPolicy/ingress-nginx-admission-webhook-c5d94713a379 and kube-ovn-webhook-c2bdbc146112.
```

## Summary by CodeRabbit

* **Bug Fixes**
* Improved admission webhook reliability during namespace deletion and
churn.
* Preserved webhook protection while allowing unrelated secure traffic
to continue.
* Corrected webhook access controls for both ingress-nginx and kube-ovn.

* **New Features**
* Added cluster-wide policy support with safer backend-port enforcement
and automatic capability detection.
  * Added safeguards for policy naming and configuration overrides.

* **Tests**
* Expanded automated coverage for webhook availability, access
restrictions, policy rendering, and end-to-end suite selection.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-03 14:10:45 +05:00
Aleksei Sviridkin 02b7651b90 test(mariadb-rd): pin the rendered grant surface whole
The exclude checks read .spec.secrets.exclude[].resourceNames[]?, which
sees only name-shaped entries. An exclude entry selecting by label was
therefore invisible to every one of them, and adding one revokes the
tenant's connection string and its trust anchor at once: matchName
returns true for every name when resourceNames is nil, and exclude is
evaluated before include and wins. The test that existed to stop exactly
that edit let it through.

The empty-selector scan keyed on the literal token matchLabels, so an
empty matchExpressions and a bare {} passed it anywhere in the file,
including under services, which the comment claimed the same pass
covered. Both are legal: the selector type inlines metav1.LabelSelector.

One more comment stated the mechanism backwards — an exclude entry that
lost resourceNames makes the backstop total rather than voiding it — and
the check under it required only that some entry carry the key, so an
added entry without one went unnoticed.

Comparing spec.secrets and spec.services whole closes all of it. The
comparison is against the rendered chart rather than the source file,
because templates/cozyrd.yaml globs cozyrds/* and ships every match:
reading one file leaves a second one, dotfiles included, granting
whatever it likes. A document count fails closed on that.

The two per-name tests are kept ahead of the comparison, subsumed but
more legible, because the runner stops at the first failure. The
block-extraction helper is gone: the cozytest runner injects return 0
before a column-zero brace, so its status was discarded there.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-03 01:38:57 +03:00
Aleksei Sviridkin ce2cabb2d6 test(mariadb): verify the tenant-ca projection end to end
The unit suite pins what the chart renders — the sentinel, its source
name, its key — from the same assumptions the chart itself encodes, so
the two agree by construction and neither can catch the other being
wrong. Only a live mariadb-operator says whether <release>-ca-bundle is
the right name, and only a live apiserver says whether the lineage
webhook promotes the projection to something a tenant can read. Both
converged engines already carry this step; mariadb had none.

The fixture sets no tls.issuer, which is the case worth covering rather
than a gap: TLS is always served and the projection template is ungated,
so the operator reconciles the source here too.

A negative satisfied by an absent label is also satisfied by a webhook
that stamped nothing all run, so the credentials Secret leads as a
positive control. For the same reason the key-bearing names are checked
for private key material before their absence from the registry is
required, the count of names that reached that check is asserted
non-zero, and the read must answer NotFound specifically: an RBAC denial
or an unavailable aggregated apiserver also makes it fail, and reading
that as "safely absent" is how the check would stop discriminating
without ever going red.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-03 01:38:55 +03:00
Aleksei Sviridkin 14af86aac0 test(mariadb): pin the tenant-ca sentinel instead of the declared source
The file asserted that the ApplicationDefinition declares the CA source
itself -- sourceSecretName, sourceKey -- and that the operator's ca-bundle
is granted by name in secrets.include. The chart renders none of that any
more. The trust anchor now reaches the tenant as the key-free
"<release>.tenant-ca" projection: a TenantProjection sentinel in the chart
names the extraction source, and the ApplicationDefinition selects the
result on internal.cozystack.io/tenant-ca. Three assertions therefore
pinned fields the file no longer contains, and a fourth pinned a by-name
grant this design deliberately withholds.

Pin the grant surface instead, the way nats-rd and mongodb-rd do: compare
the whole value of secrets.include against the credentials Secret plus the
tenant-ca selector, so neither an added name nor an added label slips past
a per-dimension check, and assert that neither the credentials Secret nor
the projection is named in exclude, which is evaluated first and wins. The
source declaration stays pinned where it is rendered, by
packages/apps/mariadb/tests/tenant_projection_test.yaml, which compares
sourceSecretName against an exact key-free name -- a stronger guard than a
grep here could make, since this file never sees the chart.

Every assertion guarding a key-bearing Secret is kept.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-03 01:38:53 +03:00
Aleksei Sviridkin 69d5ce0644 test(mariadb): assert the CA error names its Secret, and correct the ledger
The backup CA case is called "checks the CA file exists and names it" but
only guarded the first half. Deleting the Secret name from the error left
the suite green, so the reason MYSQL_CA_SECRET is passed to the CronJob at
all — telling the operator which Secret to go look at, since the client
reports a missing CA without naming a file or its source — was unpinned.

The cozyrds suite header also counted its own coverage wrong. It claimed
two live tests and three placeholders; there are seven and four. The four
exclude guards are live and went unmentioned, and the case reading the
source key is inert like its neighbours but carried no placeholder marker.
That header is the file's honest-coverage statement, so an understated
ledger is the one error it cannot afford.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-03 01:38:52 +03:00
Aleksei Sviridkin 6cb1914ec0 test(mariadb): assert the guards abort, not just that they are spelled
Three assertions matched text that survived the mutation they exist to
catch.

The backup CA guard was pinned by three separate substrings: the two test
expressions and the message. Flipping its exit status to 0, or short
circuiting the condition with `false &&`, leaves all three intact and the
suite green — while the backup runs on with --ssl-ca pointing at a file
that is missing or empty, and in the exit-0 case reports success having
dumped nothing. Match the whole construct through its exit instead.

The exclude entries were checked by name without checking they still sit
under resourceNames. A nil resourceNames matches every name, so dropping
that key turns the backstop into a match-everything selector while each
name is still present in the block. Assert the key.

The leaf certificate pinned renewBefore but not duration. cert-manager
rejects a duration that does not exceed renewBefore, so pinning one of
the pair lets the other drift into a certificate that renders but cannot
be applied.

Each is verified by the mutation it was blind to: exit 1 to exit 0, a
short-circuited condition, a renamed resourceNames key, and an 8760h
duration shrunk to 1h all now fail the suite.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-03 01:38:51 +03:00
Aleksei Sviridkin 66d2c9b399 test(mariadb-rd): scope the exclude guards to the exclude block
The backstop guards grepped the whole file for each name. Both lists sit
at the same indentation, so an entry in `include` satisfied a guard that
exists to prove the name is in `exclude` — and moving a name between them
is the single most dangerous edit this file allows, since exclude is what
keeps a Secret away from the tenant.

Moving `-ca`, the operator's CA private key, from exclude to include left
all ten guards green. So did moving `-root`. Extract the exclude block and
match inside it; both mutations now fail on the name that moved, and the
suite is unchanged otherwise.

Also correct two descriptions the TLS work outdated. The cleanup hook's
Role comment still said the rule covered two operator-generated Secrets;
it now covers four, and two of them are issued by cert-manager, which the
block above already explains. The runtime message called them all
operator-generated for the same reason.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-03 01:38:51 +03:00
Aleksei Sviridkin 18a8bf296d test(mariadb): scope the CA-bundle guard and name the CR field it checks
The guard asserting that the trust anchor is surfaced to the tenant
grepped the whole file. The bundle is also named in a comment and in the
caCert declaration, so either of those could satisfy it while the
secrets.include entry it exists to protect was gone. Scope it to the
include block, the way the sibling key-bearing-Secret guard already does.
Verified by deleting the include entry with a matching string left
elsewhere in the file: the guard now fails, where before it passed.

The suite header described a comment as stating that spec.tls.issuer is
true. There is no such field on the MariaDB CR — issuer is a chart value,
and what the chart renders is spec.tls.enabled, which is what the rest of
that sentence already describes. Name the field the assertions actually
check.

Also drop the claim that an 11.4 client verifies by default. The default
that changed in 11.4 is the --ssl-verify-server-cert flag, not whether
verification happens: without --ssl-ca that client has no trust anchor,
and falls back to an anti-tampering check derived from the password hash
which establishes neither the chain nor the hostname, and which cannot be
derived at all for a passwordless user. Passing both flags is what gives
chain and hostname verification, on either client version. The documented
command was already correct; the sentence around it invited dropping the
flag that carries it.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-03 01:38:51 +03:00
Aleksei Sviridkin da2f5c33eb fix(mariadb-rd): restore the release variable in the CA source name
The CA source name was rewritten to a literal release prefix on the
premise that no release variable exists in the template context. That
premise came from the lineage webhook, which renders a different field:
resourceNames in the secrets selectors, where the context really is only
kind, name and namespace.

caCert.sourceSecretName is rendered by the CA-extraction controller
instead, and that one supplies a release variable explicitly. It also
parses with missingkey=error, so an undefined key would fail loudly
rather than resolve to an empty string — the silent collapse the rewrite
was meant to prevent could not have happened.

Restore `{{ .release }}-ca-bundle`. It is also the better spelling on its
own: the literal hardcodes an assumption about the release prefix and
would point at a Secret that does not exist if that prefix ever changed,
while the variable is the release name by construction. The reference
implementation for this field spells it the same way.

Drop the guard that forbade the variable. The controller's own tests pin
that spelling as correct, so the guard would have started failing the
canonical form the moment the field became live.

Keep the test pinning the source name, and say in it which renderer this
field belongs to, so the two contexts are not conflated again.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-03 01:38:50 +03:00
Aleksei Sviridkin 5a27ee4e88 fix(mariadb-rd): exclude the exporter config Secret from tenant promotion
The exclude backstop enumerated the internal credential Secrets but
omitted `-metrics-config`. The operator writes it for every instance —
`MetricsConfigSecretKeyRef` at the pinned 25.10.2 fixes the name as
`<name>-metrics-config` — and its `exporter.cnf` carries

    [client]
    user=...
    password=...

in plaintext. The exporter user is granted SELECT, PROCESS, REPLICATION
CLIENT, REPLICA MONITOR and SLAVE MONITOR on `*.*`, so that password
reads every table in every database.

It is not a hypothetical Secret: the chart sets `metrics.enabled: true`
unconditionally, so it exists on every ready instance. The include side
selects by label with no name constraint, and a nil `resourceNames`
matches everything, so the exclude list is what keeps it away from the
tenant. Exclude beats include, so naming it there is sufficient.

This is inside the enumerable part of the gap, not the part the comment
already disclaims: that carve-out covers Secrets a future operator adds
or creates mid-rotation, and this one exists today under a stable name.

Also complete a sentence in the suite that lost the words naming which
issuer it describes.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-03 01:38:49 +03:00
Aleksei Sviridkin f2bce8f57d fix(mariadb-rd): name the CA source Secret with a key the context defines
The CA source was spelled `{{ .release }}-ca-bundle`. The webhook builds
the only context these templates are rendered against, and it carries
kind, name and namespace — no release. The template resolves to nothing,
so the name would come out as `-ca-bundle` and match no Secret.

The exclude entry a few lines down already names the same Secret the way
that works, `mariadb-{{ .name }}-ca-bundle`: the release prefix is a
literal and the instance name is the templated part. Spell the source the
same. One Secret described two ways in one file meant one of them was
wrong, and it was this one.

The declaration is inert until the extraction controller lands, so this
changes nothing that runs today. That is also why it could sit here
unnoticed: the test pinned the broken spelling and passed on it, which
would have made the name wrong exactly when it first started being read.
Pin the working spelling instead, and add a guard that no undefined key
is templated into a Secret name.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-03 01:38:49 +03:00
Aleksei Sviridkin 81c44574d0 fix(mariadb): scope the Secret guards to the secrets section
The awk that was supposed to read secrets.include reset on any two-space
key and then re-entered on the next four-space include:, so it swept
services.include as well. It passes today only because no Service happens
to end in -tls — the exact case the comment promised it handled would
have misfired on a Service name. The exclude reader had the same shape.

Gate both on the secrets: section header instead of on indentation alone.
Verified in both directions: a Service named -tls no longer trips the
guard, and a Secret named -tls under secrets.include still does.

Also correct which CA the backup client verifies against: the bundle
carries the chart's CA too under the cert-manager issuer, which is what
makes the unconditional --ssl-ca right in both configurations.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-03 01:38:48 +03:00
Aleksei Sviridkin a55dc5ff19 feat(mariadb): name the dangerous Secrets in the ApplicationDefinition exclude
The include selector matches by label and carries no name constraint, so
any Secret of this instance that acquires the tenant-ca label is promoted
to the tenant with its data copied verbatim. The label on the source is
policy-guarded; the one on the destination is not. Exclude is evaluated
first and short-circuits, so naming the dangerous Secrets closes the part
of that gap we can enumerate.

Listed by group with the reason each is dangerous: the private keys from
both issuers, the internal database credentials, the backup S3 keys and
restic password, and the registry pull secret.

The guarantee is only as wide as the list. Matching is by exact name with
no globs, so a Secret added by a future operator version, or one created
transiently during a rotation, is not covered — the protection that holds
in general is still that only the extraction controller writes the label.
mariadb-operator has no rotation-snapshot Secret of the kind that makes
this worse elsewhere; its Secret names are fixed.

The existing guard against a key-bearing name is rescoped to
secrets.include while here, since such a name under exclude is the
backstop working rather than a leak.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-03 01:38:47 +03:00
Aleksei Sviridkin e164050cc1 docs(mariadb): state what the guards and the docs actually cover
Four claims of mine were broader than the thing they described.

The bats header said it pins the invariants deciding whether a private
key can reach a tenant. Three of its tests touch the declaration that the
API does not accept yet, so they pin an intended spelling and nothing
more; if the field lands under another name they stay green while the
feature does nothing. Say which tests are live and which are placeholders.

The certificate comment said the operator skips reconciliation entirely.
It skips issuance, which is why SANs go unvalidated, but it still reads
the Secret for the pod annotation hash and status.

The README mapped the client's verification default onto the version
enum, which selects the server image. The default belongs to the client
library, so an 11.4 client verifies against a 10.6 server.

The backup guard is described as catching a missing or unusable bundle.
On a fresh managed-TLS install the bundle can be present and non-empty
while holding only the operator's client CA, and the failure lands at
verification instead. Note the window, and note in the README that a new
instance stays down until cert-manager issues.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-03 01:38:46 +03:00
Aleksei Sviridkin f0b77de1e2 docs(mariadb): correct the empty-selector blast radius and order the teardown
Two claims of mine were wrong about mechanism.

The empty-selector warning said such a selector would project every Secret
in the namespace. The lineage webhook computes those labels only for
objects whose ownership resolves to the application, so the reach is the
instance's own Secrets. Still the thing to prevent — that set contains
-ca-tls and -tls — but the stated reason should match the code, since it
is what a reader checks the guard against.

The teardown step told the user to delete both Secrets with no ordering.
The server keypair stays mounted in the running pods until the operator
reissues from its own CA and rolls the StatefulSet, so following that
literally leaves the next pod unable to start. Wait for the rollout
first.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-03 01:38:45 +03:00
Aleksei Sviridkin 2be135a821 fix(mariadb): pin the verification default to the version it changed in
The note said the hostname check is off on 10.11 and on across 11.x. The
default actually changed in 11.4. Measured all four versions this chart
offers: 10.6 and 10.11 leave it off, 11.4 and 11.8 enable it. Name the
versions rather than the series, since the reader is choosing between
exactly those.

Scope the key-bearing-Secret guard to the secrets block as well. It
scanned the whole file, so a Service whose name ends in -tls would have
tripped it while a Secret entry with a trailing comment slipped past —
the wrong answer in both directions for the check that decides whether a
private key can reach a tenant.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-03 01:38:45 +03:00
Aleksei Sviridkin 3de6181da0 feat(mariadb): surface the operator CA bundle as a tenant resource
The dashboard Role already let the tenant read <release>-ca-bundle, but
the ApplicationDefinition did not list it, so it was never marked as a
tenant resource and never appeared through the API. The documentation
tells clients to fetch exactly that Secret, which worked with kubectl and
not in the interface the platform actually gives tenants.

Name it in secrets.include, the same way kafka names its CA certificates.
This works against the current API, unlike the label selector beside it,
which stays for the projection an unmerged controller will publish — so
the trust anchor is reachable now rather than only after that lands.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-03 01:38:42 +03:00
Aleksei Sviridkin 38671c3ff0 test(mariadb): catch a bare empty label selector, not just the inline one
The guard against an empty selector matched only "matchLabels: {}". A
bare "matchLabels:" with nothing nested under it is the same empty
selector — it compiles to match-everything and would project every
Secret in the namespace to the tenant — and it slipped through. Check
that a bare selector is followed by an indented label as well.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-03 01:38:41 +03:00
Aleksei Sviridkin 85c3ff1419 test(mariadb): require the tenant CA label inside the selector
The check accepted the label anywhere in the ApplicationDefinition, so a
stray occurrence would satisfy it while the selector that actually
exposes the projection stayed empty or wrong. Scope it to the
matchLabels block it has to appear in.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-03 01:38:39 +03:00
Aleksei Sviridkin 38e2c4fea4 test(mariadb): close a quoting hole in the ApplicationDefinition guard
The check rejecting key-bearing Secrets from the tenant include list
matched only unquoted list entries, while its sibling check was already
quote-agnostic. A quoted entry would have passed both the guard and the
comment claiming it could not.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-03 01:38:38 +03:00
Aleksei Sviridkin 18215895fd fix(mariadb): apply TLS enforcement regardless of who owns the CA
tls.required was only emitted when the chart managed the certificates,
so setting it on an instance using the operator's own CA was accepted by
the schema and then dropped on the way to the CR — no enforcement, no
error, no warning. For a setting whose entire purpose is refusing
unprotected connections, silently ignoring it is the worst available
outcome.

The coupling was artificial: in the operator API required sits beside
enabled with no dependency on either certificate reference, and
enforcement works against operator-issued certificates just as well.
Emit it in every configuration and let it mean what it says.

Also loosen the two negative guards in the ApplicationDefinition check,
which required a literal closing quote and exact brace spacing and so
would have missed an unquoted key-bearing source.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-03 01:38:36 +03:00
Aleksei Sviridkin 4339eb4631 test(mariadb): pin the ApplicationDefinition trust-anchor invariants
The ApplicationDefinition is where a mistake decides whether a private
key can reach a tenant, and it is the one file in this change the chart
test suite cannot reach: helm-unittest only covers rendered templates,
and the *-rd packages expose no test target.

Add a shell check in the style of the existing kafka one, which the
Makefile discovers automatically. It pins that the CA source is the
operator's key-free bundle rather than either key-bearing Secret, that
the tenant selector is the specific label rather than an empty selector
that would match every Secret in the namespace, and that no key-bearing
Secret is listed by name.

Type required as a pointer while here. As a plain bool it rendered a
default of false in the generated documentation while the description
promised true, and the zero value could not represent an explicit false
for consumers going through the generated struct — which is the value
the documented migration path depends on.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-03 01:38:35 +03:00
Aleksei Sviridkin 3cf9431f1e test(redis): fail the negative dial probes on an unread state
The two negative dials in the chainsaw suite, the plaintext dial at a
TLS instance and the TLS dial at the instance whose flip was rejected,
decided by whether PONG appeared on stdout with stderr discarded. A
missing pod, a 403 on exec, a wrong container name or the outer cap
firing print no PONG either, so each of them passed as the refusal
being asserted, and the flip case is the check that a rejected flip
left TLS off.

Keep what the dial printed and pass only on the refusal each listener
actually gives, verified in the shipped images: a TLS listener drops a
plaintext client at AUTH and redis-cli reports it as an I/O error; a
plaintext listener holds a ClientHello in silence, because the inline
parser's newline scan stops at a NUL in the handshake header, so the
inner cap expires and kubectl's exit-code trailer carries the 124.
That trailer is what separates the cap firing inside the pod from the
wrapper's cap firing outside it. A listener that answers PONG fails
the check, and anything else fails as a state the check did not read,
with the dial's output in the message.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-02 20:48:52 +03:00
Andrey Kolkov 6eba4d9fa7 test(backups): use .svc FQDN for rabbitmq backup e2e S3 endpoint
The reworked rabbitmq backup roundtrip reached the upload step and failed
TLS verification: `curl (60) no alternative certificate subject name matches
target hostname 'seaweedfs-s3.tenant-root'`. The demo Bucket, in-cluster
endpoint resolution and the copied SeaweedFS CA all worked; the remaining
gap is the hostname. The seaweedfs filer serving cert's SANs are
`*.tenant-root.svc` and `*.tenant-root.svc.cluster.local`, which the 2-label
`seaweedfs-s3.tenant-root` matches neither; the 3-label `.svc` FQDN does (and
still resolves via search-domain expansion). Point S3_ENDPOINT at
`https://seaweedfs-s3.tenant-root.svc:8333`.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-02 21:25:21 +04:00
Myasnikov Daniil 686a81a491 docs(networking): stop claiming the churn coupling is removed
Restoring the target-side pod deny put a cluster-spanning peer selector back on
the webhook endpoint, which is the exact thing the source-side rules were added
to take off it. The comments still said the coupling was "structural and
removed", and the release note still said deleting a namespace no longer
rewrites that endpoint's policy map. Both were true of the previous revision
and are not true of this one.

What ships now is two denies that fail in different directions: the source-side
one keeps enforcing while the webhook endpoint's policy is being recomputed or
has overflowed its map, the target-side one keeps enforcing where Cilium is not
the service load balancer, and neither subsumes the other. So the coupling is
not eliminated, it is made non-load-bearing -- whichever end is degraded the
other still denies, and the map writes are writes rather than an outage.
Eliminating it outright would mean dropping the target-side rule and accepting
a silent fail-open, which is the worse trade.

The chainsaw case's header claimed the deletion "briefly made fail-closed
admission webhooks unreachable". That is the unconfirmed hypothesis, and the
errno in the report contradicts it: EHOSTUNREACH cannot come from a policy
denial, which drops after connect() has returned. It now says what the case
actually asserts -- both webhooks still admit across a namespace deletion --
rather than claiming to reproduce a cause.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-09-02 20:52:06 +05:00
Myasnikov Daniil 0335421769 test(e2e): check every webhook registration, and name the controller Service
Three narrowing fixes to the two webhook cases.

The availability precondition resolved only the FIRST registration of each
webhook name (awk ... exit) while every matching registration fires on the
applies below, so an unrelated flaky ingress-nginx release would have aborted
an apply and been reported as a churn-induced outage. It now waits on all of
them and says which one it is waiting for. Split into require_service_up plus a
loop, indexed with sed rather than piped into `while read`, because a
pipeline's body runs in a subshell under dash and a `return` from it would not
reach the caller.

That split immediately produced its own bug, caught by the scenario harness:
POSIX functions share one scope, so the callee's `label="$1"` overwrote the
caller's copy and the second registration reported itself as "(1/2) (2/2)".
The callee now uses distinct names.

The positive control's Service was picked with `grep -v -- '-admission$' |
head -1`, which relies on ingress-nginx-controller sorting before
ingress-nginx-controller-metrics. Deterministic but undocumented, and the
control has to be the Service that actually exposes the https port, so
-metrics is now excluded explicitly.

Also drops a stale sentence: the churn pod's comment claimed the old
target-side policies coupled the webhook endpoint to identity churn, which
overstates a map write as an outage and is moot now that the target-side rule
is back. The pod is there because an empty namespace allocates no identity, so
without it the case would exercise nothing it names.

Eleven scenarios through both scripts under dash with a stubbed kubectl, all as
specified, the new multi-registration case included.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-09-02 20:45:22 +05:00
Aleksei Sviridkin 25a0618184 test(redis): exercise TLS end to end in the chainsaw suite
Everything the TLS unit tests can only render now runs on a live
cluster. A second fixture with tls.enabled asserts that cert-manager
issues the chain, that the patched operator brings two Redis replicas
and the Sentinel Deployment to Ready through its --tls probes, that the
operator publishes its CA-only Secret under the widened ClusterRole, and
that the CA-extraction controller hands the tenant exactly ca.crt
through core.cozystack.io/tenantsecrets while the key-bearing Secrets
and the operator's copy stay off that path. The last check completes a
handshake through the rfr- service name with the same CA the tenant
receives.

A third case applies tls.enabled to a running plaintext release and
asserts the refusal end to end: the HelmRelease upgrade fails with the
schema's own message, the StatefulSet keeps its revision and both
replicas Ready, plaintext still answers and no TLS listener appears.

The TLS case also asserts the topology the operator built rather than
stopping at readyReplicas: exactly one master, a replica whose link is
up, and sentinels that moved off the bootstrap address. Ready alone
witnesses the bootstrap, since a pod still replicating from the initial
127.0.0.1 never gets there, but the readiness script exits 0 for
anything reporting role:master, so it stops witnessing once the
bootstrap is over.

Both PING calls note where the pinger credentials come from: the ACL user
the operator writes into every redis.conf it generates, and the one its
own liveness probe uses. Nothing in this tree defines it, so a reader
grepping for it here finds nothing and reads the calls as broken.

Every redis-cli call runs under the container's own timeout, and both
script operations carry an explicit one. The inner cap is the part
that matters. The flip case dials TLS at a plaintext listener that
will never send a ServerHello, and redis-cli has no handshake
deadline of its own. Chainsaw does kill the shell when its exec
budget runs out, but it captures the script's stdout and stderr
through pipes, and Go's exec then waits for every holder of the write
end to close it; the orphaned kubectl is one, so the suite would go
silent until the job ceiling instead of failing. Expiry reads as the
refusal for the negative check and as a named failure for the PINGs.
Both engine images provide /usr/bin/timeout: busybox on the sentinel,
coreutils on the data pods.

The negative half of the tenant-secret pair requires the error to say
NotFound. A 403, a timeout or an unserved CRD would otherwise be read
as proof that the Secret is not served, which is the direction that
fails silently.

The dashboard Role is asserted on the deployed object, not only in the
chart unit tests: it must reach the auth Secret and nothing else, which
is the name-granting half of the pair the label-driven projection
completes. Every kubectl in both scripts runs under a wall-clock bound,
because the per-command cap inside a container cannot fire when the exec
stream itself never establishes, and a kubectl left running holds the
script's stdout open. The plaintext StatefulSet in the flip case waits
5m rather than 2m: it measured 98s of that budget on a real run, being
the third failover the group brings up in one namespace.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-02 17:58:06 +03:00
Aleksei Sviridkin d0cab491f3 feat(redis): move the tenant trust anchor onto the platform projection
The tenant read path was two grants on Secret names: the operator's
CA-only <release>-ca-cert in the dashboard Role, and the same name in the
ApplicationDefinition's include list. Replace both with the projection
the previous commit declares.

A name grant conveys whatever occupies the name. The operator holds
create and update on Secrets cluster-wide, so what sits at
<release>-ca-cert is the operator's choice rather than the platform's,
and RBAC has no key-level filter to catch it if that ever stops being
just ca.crt. <release>.tenant-ca is written only by the CA-extraction
controller, which copies one named key and refuses PEM private key
material outright. That is the difference between a name that happens to
hold a certificate today and provenance the platform vouches for, and it
is the reason postgres, mongodb, nats and qdrant all reach the tenant
this way — mongodb's operator publishes a CA Secret of its own too, and
that one is excluded rather than granted.

Also drop spec.caCert from the ApplicationDefinition. The
ApplicationDefinition CRD does not define that field and Flux applies
these server-side, so the apiserver rejects the whole object with "field
not declared in schema" and the redis-rd release never goes ready. It was
carried deliberately as a merge-order gate for a controller that reads
it; that controller landed as the TenantProjection reconciler, which
reads a sentinel in the app chart instead, so the gate has nothing left
to wait for.

hack/check-redis-rd-secrets.bats pins the whole rendered Secret surface
rather than probing for known-bad names, alongside the existing guards
for nats, qdrant and mongodb: an include with neither resourceNames nor
matchLabels restricts nothing, and a second matchLabels entry selecting
a label cert-manager also stamps on the key-bearing Secrets would grant
the CA private key while every name in the file stayed correct.

The connection example passes the generated password as well. Password
generation is on by default, so the command as printed connected and then
answered NOAUTH on the first real command.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-02 17:58:06 +03:00
Aleksei Sviridkin cf3cb0f133 test(values): reject a boolean default on a string-typed schema property
cozyvalues-gen quotes a default only when the declared type is the literal
`string` or `quantity`; a named alias, including one declared `@enum
{string}`, falls through to a YAML re-parse of the bare value where the 1.1
boolean words resolve. A chart spelling out a `"no"` default under such a
type emits `"default": false` into a string-typed, enum-constrained property
and `+kubebuilder:default:=false` onto the matching Go field. Templates read
values.yaml rather than the schema, so the chart suites stay green while the
shipped schema and CRD carry a default the field rejects.

Vendored schemas under packages/*/*/charts/ are left out of the scan:
cozyvalues-gen never writes them, they change only through `make update`,
and a violation there would have no fix inside this tree.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-02 17:58:05 +03:00
Myasnikov Daniil cbd0e78cad test(e2e): stop the webhook cases passing for the wrong reason
Both new cases could report a verdict they had not observed, and the first CI
run of them did exactly that against a crash-looping ingress-nginx controller.

admission-webhooks-survive-namespace-churn died in setup: the admission Service
had no ready backend, so the first apply -- which passes through that same
fail-closed webhook -- failed with "no route to host" before any churn ran, and
the failure read as the outage under test. It now asserts both webhooks are up
BEFORE the churn, resolving their coordinates from the webhook registrations
themselves rather than from a guessed release name, and says so in its own
words when they are not. Proven against the pre-change script with a stubbed
kubectl: with a dead admission Service the old script proceeded into the churn
(delete namespace, two applies, delete namespace, two dry-runs); the new one
stops and names the cause without touching the churn namespace.

admission-webhook-deny-still-enforced measured a corpse. Its `kubectl wait` did
observe both pods Ready, but the controller was flapping on a ~100s cycle and
the probe ran against a dead backend 19 seconds later. "Denied by policy" and
"no backend to route to" are the same observation from inside the probe, so the
two deny assertions would have reported success against a cluster with no
policy installed at all, and the port-scoping assertion reported a policy bug
that did not exist. The deny verdicts are now gated on the admission Service
having ready backends before the probe, on every poll while it runs, and after
it finishes, and on the controller pods neither restarting nor being replaced
across that window -- EndpointSlice readiness rather than pod readiness,
because a `kubectl wait` is what a flapping pod satisfies transiently. The
probe reports reachable/unreachable per destination and always exits 0; this
script holds the endpoint evidence and is the only place that can attribute a
refused connect, so anything undecidable now fails as a setup failure instead
of as a verdict.

Ten scenarios driven through both scripts under dash with a stubbed kubectl,
all behaving as specified: a dead backend, a backend dying mid-probe, a
controller restarting mid-probe, a probe that never completed, an unregistered
webhook, the deny lost, the deny widened to every 443 destination, and the
healthy path. The restart-mid-probe case is the one that used to produce a
confident false pass.

Two more things the same rewrite fixes. The step budget was 6m against inner
budgets summing to 420s, so a slow-but-healthy cluster was killed with no
verdict and no describe output; both steps now carry a timeout above their own
sum, per the rule in hack/e2e-chainsaw/.chainsaw.yaml, and the `kubectl wait`
whose 180s dominated that sum is gone. The churn namespace now carries a
running pod, because a Cilium identity is a set of endpoint labels including
k8s:io.kubernetes.pod.namespace: an empty namespace allocates no identity and
its deletion drives none of the UpdateIdentities path the old target-side
policies coupled the webhook endpoint to. And both server-dry-run Pods carry a
restricted-compliant securityContext -- never scheduled, but a Pod that
PodSecurity would reject aborts the check under set -eu and reads as the
webhook being down.

Refs #3983.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-09-02 19:34:51 +05:00
Andrey Kolkov 86fe8e5a7e feat(valkey): add managed Valkey service (#3380)
## What this PR does

Adds a **managed Valkey** service to the catalog, alongside managed
Redis.

> **Stacked on #3406** (redis-operator migration to freshworks-oss).
This PR targets that branch and depends on it; review/merge #3406 first.

### Why

Redis relicensed at 8.x to a source-available family (AGPLv3 / SSPL /
RSALv2), and the managed Redis app currently ships `v8: 8.4.0` with no
BSD-licensed option ≤ 7.2. That is a licensing risk for CNCF Incubation.
[Valkey](https://valkey.io) is the BSD-3-Clause fork of Redis 7.2.4,
governed by the Linux Foundation, and a drop-in replacement for Redis.
This is the same move Argo CD made —
[`cncf/foundation#750`](https://github.com/cncf/foundation/issues/750)
was closed with "no need for exception" after they migrated to Valkey.

### How it works

Valkey runs on the same operator as managed Redis. #3406 migrates that
operator from the archived `spotahome/redis-operator` to the
actively-maintained **freshworks-oss/redis-operator** fork, which adds a
native `spec.engine: Valkey` switch. With it, the operator runs
`valkey-server` / `valkey-cli` natively (in the launch command and in
its liveness / readiness / shutdown probes), so the chart uses the
official `valkey/valkey` image directly — **no Redis-compatibility shim
image**.

The app mirrors managed Redis: same shape/values (`replicas`,
`resources`/`resourcesPreset`, `size`, `storageClass`, `external`,
`version`, `authEnabled`), Sentinel-based HA, metrics exporter,
`WorkloadMonitor`s, dashboard resource map, and `ApplicationDefinition`
(`kind: Valkey`).

Versions offered: `v8 → 8.1.8` (default) and `v7 → 7.2.13` (the
BSD-3-Clause Redis 7.2 line, most conservative drop-in).

### How it was verified

- `helm template` renders cleanly for `v8`, `v7`, and `external: true`;
the `RedisFailover` carries `spec.engine: Valkey` and `image:
valkey/valkey:<version>`. `helm lint` shows only the pre-existing
icon-URL / `cozy-lib` warnings that managed Redis emits too.
- `helm template` of `valkey-rd` renders a valid `ApplicationDefinition`
(`kind: Valkey`, `plural: valkeys`, `category: PaaS`).
- Generated artifacts (`values.schema.json`, `README.md`, `types.go`,
`zz_generated.deepcopy.go`, the ApplicationDefinition `openAPISchema`)
reproduce with the pinned `cozyvalues-gen` v1.6.0 and `controller-gen`
v0.16.4 — root `make generate` and per-app generate both produce no
drift; `go build ./valkey/...` passes.
- A live-cluster e2e was not run from this environment; the Chainsaw
suite `hack/e2e-chainsaw/valkey/` (mirroring the redis suite) is
included and `hack/select-e2e.sh` maps the `valkey-application` source
to it, so it runs in CI once the freshworks operator image is built.

### Notes

- The Valkey logo (`packages/apps/valkey/logos/valkey.svg`) is a
placeholder in Valkey's brand colors, pending the official brand asset.
- A Grafana `db/valkey` dashboard was intentionally left out of this
initial PoC; it can follow.

### Screenshots

N/A — no UI changes.

### Downstream repositories

This adds a new managed-app kind, which likely warrants follow-ups in
`cozystack/website` (user-facing docs for the new service) and possibly
`cozystack/terraform-provider-cozystack`. No follow-up PRs are opened
yet — leaving these for a maintainer to decide, per the template
guidance.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
feat(valkey): add managed Valkey service — a BSD-3-Clause, Linux Foundation-governed alternative to managed Redis, running on the freshworks-oss redis-operator via native spec.engine: Valkey (versions 8.1.8 and 7.2.13)
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added managed Valkey service support with high availability and
configurable replicas.
* Supports Valkey versions 7 and 8, persistent storage, resource
presets, authentication, and optional external access.
  * Added monitoring metrics and dashboard integration.
  * Added platform package integration and tenant administrator access.
* **Documentation**
* Added Valkey configuration guidance and storage-class immutability
documentation.
* **Tests**
* Added end-to-end coverage for deployment readiness, storage, and
replicas.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-02 18:18:42 +04:00
Myasnikov Daniil f73b54fee9 test(e2e): assert the webhook deny is enforced and scoped to its port
The churn case added alongside it proves the webhooks stay AVAILABLE. Nothing
proved they are still UNREACHABLE from pods, which is the property the
policies exist for, and no test anywhere checked that the deny is confined to
the webhook port -- which is why a rule that denied all TCP/443 egress
cluster-wide passed every existing assertion.

Probe from tenant-root, where the host cluster's ingress-nginx lives and where
allow-internal-communication permits pods to reach it, so the deny is the only
thing that can block the webhook port; from the isolated tenant-test the probe
would pass for the wrong reason. Three assertions against the same pods and
the same tenant egress allowances, differing only in port: the controller's
own HTTPS port must stay reachable, the admission Service ClusterIP must not
be, and the backend port must not be. The first is the one that separates a
scoped deny from a wildcard.

Uses busybox:musl, already staged on every node by hack/e2e-prepull-images.sh,
with a securityContext that satisfies the restricted PodSecurity profile.
Service names, ports and the backend port are discovered at runtime rather
than hardcoded, since the release name differs between installs.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-09-02 19:15:45 +05:00
Myasnikov Daniil ad961f6de1 test(e2e): cover webhook admission after namespace churn
Exercise ingress and pod admission immediately after deleting a tenant
namespace, and route ingress-nginx changes to the gateway suite.

Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-09-02 19:15:44 +05:00
Andrey Kolkov 329ef9cf3f test(backups): make rabbitmq backup e2e self-contained via demo bucket
The rabbitmq backup roundtrip was the only backup e2e routing through the
platform cozy-default system bucket, whose advertised endpoint in the CI
sandbox is the unroutable s3.example.org placeholder, so the backup Job
failed with "Could not resolve host: s3.example.org". Mirror the
clickhouse/mariadb backup examples: provision a demo Bucket, project its
access keys into rabbitmq-backup-creds and copy the seaweedfs CA into
rabbitmq-backup-ca, and render a self-contained Rabbitmq strategy +
rabbitmq-default BackupClass whose curl uploads to the in-cluster seaweedfs
endpoint (S3_ENDPOINT override, CURL_CA_BUNDLE for the self-signed cert).

The in-Pod script is kept identical to the shipped cozy-default-rabbitmq
strategy, which is left unchanged; the demo only reuses its client image.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-02 15:37:30 +04:00
myasnikovdaniil 67b4e23ca2 test(tests): cut the noise in unit and e2e job logs (#3973)
## What this PR does

Unit test job log is 238,794 lines and 31MB, and 233,444 of those are
`set -x` trace from `hack/cozytest.sh`. The one failing assertion sits
at line 238,573. Nothing in that stream is needed there, `run_one`
already tees raw output to `$log` and the fail handler dumps all of it
for whichever test fails, so dropping the live stream loses output of
passing tests only.

`COZYTEST_TRACE` defaults to 1, so e2e call sites in
`packages/core/testing/Makefile` keep the live stream. For a long e2e
suite that stream is the only progress signal, and the only record at
all when a step timeout or a dead runner kills the job before the fail
handler runs. `bats-unit-tests` sets 0. Running the runner by hand stays
verbose, that is the case where a human is actually watching.

Second thing, previous-instance log capture now orders by restart
recency. Container cap (`COZY_PREVLOG_MAX=12`) is spent in namespace
order and "unrelated" meant only "in another namespace", time was never
a key. So a container that restarted once during bring-up competed on
equal footing with one that crash-looped during the test, and won
whenever the pod list reached it first. Two measured e2e jobs spent 1034
and 1545 lines on `virt-api` and `cdi-apiserver`, both restarted once at
install time by `Unable to create certwatcher`, hours before any app
suite ran. On a live stand 178 containers carry a restart and their
timestamps span 24 days, so the cap binds hard enough for order to
decide everything.

This orders, it does not filter. Install-time restart is the explanation
when install is what broke, and the capture has no reference point for
"when the test began" that would make a cutoff honest. Namespace stays
primary key, because a stale restart in the namespace under test beats a
fresh one three namespaces away. So it fixes cap displacement, not raw
volume of a single run.

`lastState.terminated.finishedAt` joins the row. The guard on it is
three deep and the innermost one is load bearing, which I got wrong
first time round and review caught: `finishedAt` is a metav1.Time behind
`json:",omitempty"`, omitempty cannot omit a struct, and the zero value
marshals to `null`. The kubelet does synthesize a terminated state with
no time, ContainerStatusUnknown on a pod deleted mid-run, which also
bumps restartCount and so clears the restart filter. kubectl renders
that leaf as the literal `<no value>`, and `<` sorts above every digit,
so under the reverse sort that row landed FIRST and displaced the most
recent real restart. Exactly the inversion the ordering exists to
prevent, plus a header reading `previous instance ended <no value>`.
Fixed at both layers because they fail independently: the template
guards the leaf so the string cannot be produced, and the sort
normalises any field that is not shaped like a timestamp to empty so the
ordering holds even if some other producer emits one.

Worth naming how that slipped through. I probed the template against a
live cluster, saw no `<no value>`, and read it as the guard being
correct. It only meant no container on that cluster was in that state.

Reading the field also fixes a latent bug it would otherwise have
caused: `read -r ... restarts` folds a trailing field into the last
variable, so the dump header would print
`restarts=1|2026-08-22T09:06:00Z`. Now it names the time, which is the
first thing telling a reader whether the dump predates the test.

### What quiet mode keeps and what it drops

Two things are dropped, and the second one came out of review rather
than out of me.

First, xtrace and stdout of tests that pass. You cannot diff a passing
test's trace against the failing one inside the same job log, and a
warning printed by a test that passed is gone.

Second, a test that is KILLED rather than failed loses its trace as
well. The fail handler runs only after the pipeline returns, so a hang
reaped by the job timeout never dumps anything, and quiet mode has by
then dropped the live stream. Verbose mode never noticed, because the
trace was already printed. Measured: verbose leaves the line the test
blocked on, quiet leaves the `╭` and nothing else. So this one is a real
regression on the unit lane, not a pre-existing gap.

An INT/TERM trap is the obvious repair and it does not work. POSIX sh
defers a trap until the current foreground command returns, so
installing one stops the runner dying on the signal at all: a suite hung
on `sleep 30` took 30s to exit with the trap against 2s under the
default disposition. GitHub escalates INT, then TERM, then KILL, so the
deferred handler gets SIGKILLed before it ever runs, and the grace
period goes on waiting for the hang. Built it, measured it, reverted it,
and the measurement is in the comment now so the next reader does not
repeat it. Making it work needs the test pipeline moved into the
background with `wait`, which is the interruptible one, and that is
surgery on the path every suite in this repo takes for a partial trace
in a rare case.

What is left for that case is the escape hatch below, which for this
lane costs 48s and no cluster. The same argument is why e2e keeps the
stream on: re-running there is an hour and a live cluster, so the record
has to be written as it goes.

Everything else stays. The `╭`/`╰` pair per test with elapsed time, so
which tests ran, in what order and how long each took is all still
there. Full trace, stdout and stderr of the failing test, same bytes as
before, because `run_one` tees to `$log` and the fail handler dumps it,
and that path is untouched. Anything printed outside `run_one` as well,
file level code, `.bats` parse errors, and the `--- running <file> ---`
markers. A hang still shows up as `╭` with no matching `╰`. And `make
unit-tests` stops at the first failing test anyway, so there was never
more than one dump to read.

Measured on a suite that is actually red (`hack/ghcr-mirror_test.bats`
fails on my machine because it reads a live cluster): verbose 1472
lines, quiet 63 lines, and the post-failure dump is 23 lines and
byte-identical in both modes.

Getting the trace back, both paths checked:

```
COZYTEST_TRACE=1 make unit-tests     # env prefix, 29200 trace lines
make unit-tests COZYTEST_TRACE=1     # make cmdline, same
hack/cozytest.sh hack/foo.bats       # by hand, verbose by default
```

This lane needs no cluster and runs locally in 48s, so anyone who wants
the trace just reruns it. That is exactly why the same argument does not
extend to e2e, and why the default stays 1 there.

Nothing else consumed that stream. Unit lane stdout goes only to the job
log, `cozyreport.sh` folds in crust-gather snapshots and the trap writes
those to disk independently, and the unit lane produces none of them.

### Verification

Measured on this PR's own run: unit job went from 238,794 lines and 31MB
with 233,444 trace lines, to 5,720 lines and 580KB with 0 trace lines
and 1548 tests OK. Those two totals are not strictly like for like, the
old run was red and this one is green, so the honest comparator is
233,444 trace lines down to 0.

Quiet output is byte-for-byte the non-trace subset of verbose, and the
fail handler dump is unaffected (checked with a deliberately failing
suite). Verbose output unchanged against main, byte-identical sorted on
`hack/bats-no-exit-trap.bats`. Other suites are not self-stable run to
run (temp names, epochs, stdout/stderr interleaving) so that is the
ceiling on that check, not a finding.

Eight tests on the capture and seven on the switch. Every fix is
mutation proven: reverting the template leaf guard, removing the sort
key normalisation, dropping the read variable, dropping `-s`, silencing
an e2e recipe, removing the fail handler dump, flipping the default, and
making the switch a no-op are each caught. Three of those guards did not
bite when first written, which review also found: both stability
fixtures were pre-sorted ascending, which is what sort's last-resort
whole-line compare reproduces with no `-s` at all, and no test fed a
six-field row through the whole script, so deleting the read variable
left the suite green at 39/39. Whole unit lane, 83 files, 1503 passed
and 1 failed. The failure is `hack/ghcr-mirror_test.bats`, it reads a
live cluster and fails identically under main's own runner.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map file by file against the diff. Nothing under
`hack/` is moved or renamed, no make target changes what it does
(`bats-unit-tests` runs the same files to the same verdict, only
quieter), and `hack/package.mk`, `hack/common-envs.mk`,
`hack/update-crd.sh` and `hack/e2e-prepare-cluster.bats` are untouched,
which is what ccp, external-apps-example, talm and ansible-cozystack
couple to. No package `values.yaml`, `values.schema.json`, CRD,
namespace, label, annotation or metric name is involved.

- [x] No downstream repository is affected by this change

### Release note

```release-note
test(tests): bats unit test trace is now opt-in via COZYTEST_TRACE and quiet by default in `make unit-tests`, and the e2e previous-instance log capture now picks the most recently restarted containers first
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Improvements**
- Test runs can suppress live trace output for passing tests while
preserving full diagnostics for failures.
  - Trace visibility can be customized through the test configuration.
- Previous-instance logs are prioritized by the most recent container
termination within the preferred namespace.
  - Captured log headers now show when the previous instance ended.
  - Undated logs are retained and placed after dated entries.

- **Tests**
- Added coverage for trace controls, failure handling, and recent-log
ordering.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-02 10:44:23 +05:00
Andrey KolkovandClaude Opus 4.8 ba32df69c0 feat(backups): add RabbitMQ definitions backup via a dedicated strategy driver
RabbitMQ's cluster operator ships no backup CRD and RabbitMQ has no
consistent online export of message payloads, so back up broker
DEFINITIONS (vhosts, users, permissions, queues, exchanges, bindings,
policies, parameters) over the management HTTP API and store them in the
system bucket. Message payloads and PITR are out of scope - a full-data
backup is a Velero volume snapshot, a separate strategy.

Add a first-class strategy.backups.cozystack.io/Rabbitmq driver, following
the Altinity pattern (a dedicated driver that renders a one-shot Job for an
HTTP-API engine):

  - reconcileRabbitmq / reconcileRabbitmqRestore render the strategy's
    PodTemplateSpec (curl GET/POST /api/definitions, push/pull to S3 with
    --aws-sigv4) as a batch/v1.Job, for both in-place and to-copy restore.
  - A Go-read precondition gates each run on the RabbitmqCluster reporting
    AllReplicasReady (via the rabbitmqtypes partial type), surfacing a
    precise Ready=False wait instead of a Job that starts and fails.
  - The produced Backup records the object location on status.artifact.uri
    (rendered from spec.artifactURITemplate; the Backup CRD has no status
    subresource, so it persists on Create, as in the Etcd driver).

Wire dispatch in both controllers + supportedBackupStrategyKinds, register
rabbitmqtypes in the scheme, and grant read on rabbitmqclusters. Ship
cozy-default-rabbitmq (a $bucketName-gated Rabbitmq strategy) and route
apps.cozystack.io/RabbitMQ to it in the cozy-default BackupClass. The
application chart is untouched.

Add unit tests for the driver's pure helpers and the precondition gate, a
runnable example, and an honest-roundtrip Chainsaw e2e that seeds a
sentinel vhost, backs it up, drops it, and asserts it round-trips through
object storage on both restore flows.

The Job pod reuses the platform-migrations image (curl + ca-certificates)
under a new rabbitmqBackupClientImage key, stamped in lockstep with
chBackupClientImage so the repository stays pinned at one digest; a
dedicated slim image is a follow-up.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-01 18:21:10 +04:00
Aleksei Sviridkin c94a1da5bb feat(mongodb): add TLS support via PSMDB operator-managed certificates (#2692)
## What this PR does

Enables operator-managed TLS for MongoDB and stops handing tenants any
Secret that carries private key material.

- Percona Server for MongoDB serves TLS unconditionally (`mode:
preferTLS`), so this PR drops the misleading `tls.enabled` toggle rather
than pretending the chart controls it.
- The tenant no longer receives the operator's CA Secret
(`<release>-ca-cert`, which holds `ca.key`) or the leaf
(`<release>-ssl`). Both were previously reachable, and either one is a
private key in tenant hands, the CA one is a full trust-chain
compromise.
- The chart declares a `TenantProjection` sentinel naming its CA source
Secret (`<release>-ssl`, the operator-managed leaf, which carries
`ca.crt`), and the ResourceDefinition selects the resulting key-free
`<release>.tenant-ca` projection by the engine-agnostic
`internal.cozystack.io/tenant-ca` label. The CA-extraction controller
reads the sentinel and publishes the projection holding only `ca.crt`.
The tenant gets what it needs to verify the server and nothing else.

### Why the include also carries an exclude list

The trust-anchor selector matches by label with no name constraint, so
any Secret in the namespace that acquired that label would be promoted
with its data copied verbatim. The enumerable dangerous set is therefore
excluded explicitly: the CA, both leaves, the rotation snapshots
(`-ssl-old`, `-ssl-internal-old`, same server keys as the leaves, and
unlike them they carry no owner reference, so an interrupted rotation
can leave them behind after the release is gone), the system-user
passwords, the at-rest encryption key, the keyfile, the object-storage
credentials, and both forms of the operator's internal Secret.

That backstop is bounded, and the comment says so: matching is by exact
name with no globs, so a Secret a future operator version introduces, or
a transient object an issuer creates mid-renewal, stays in scope of the
label-only include. The per-user Secrets cannot be enumerated at all,
since their names carry the username. The primary defence remains that
the controller stamps only its own projection; this closes the
enumerable set, not the whole surface.

### Release note

```release-note
feat(mongodb): stop exposing TLS private key material to tenants and publish the CA as a key-free trust anchor
```
2026-09-01 14:16:24 +04:00
Aleksei Sviridkin edd767ea0f feat(nats): publish ca.crt-only trust anchor to tenants (#3340)
## What this PR does

Converges nats onto the key-free CA trust-anchor contract. A TLS-enabled
nats release renders a `TenantProjection` sentinel naming the chart's
cert-manager CA Secret as its source, and the nats ResourceDefinition
selects the resulting `<release>.tenant-ca` object by label. The
projection is built by the CA extraction controller in
`internal/controller/cacert`, which parses the source certificate and
re-encodes it into a single `ca.crt` key, so a tenant that enables TLS
gets something to verify the server against without ever getting a
private key. No key-bearing Secret is granted to a tenant, by name or by
label.

`packages/apps/nats/templates/certmanager.yaml` is not touched. An
earlier draft of this change stamped publish labels onto the CA
`Certificate`'s `secretTemplate`; the controller that shipped consumes a
sentinel object instead, so the chart now declares its source rather
than marking it. The functional change is one new template and one
`matchLabels` entry in the ResourceDefinition.

The sentinel is gated on the chart's effective TLS state, which is
tri-state: left unset it follows `external`. That gate is the single
divergence from the postgres implementation of the same contract, and it
exists because the nats CA is only issued while TLS resolves true. An
ungated sentinel would sit `Ready=False/SourceNotFound` forever on every
plaintext release.

The CA extraction controller is already on main, so this is live on
merge rather than dormant until a follow-up lands. The
`do-not-merge/hold` label predates that and no longer guards anything.

## Verified against a running cluster

The premise this rests on is that cert-manager populates `ca.crt` on the
Secret of a `selfSigned`, `isCA: true` Certificate. Nothing in this
repository witnesses it: `hack/e2e-chainsaw/` has no nats suite, and
nats appears in no e2e workflow, so every unit test here encodes the
same assumption the chart encodes and the two cannot catch each other
being wrong. I closed it against a live cluster instead, running
cert-manager v1.20.2, the version this repo ships, on two independent
nats releases:

- `nats-<name>-ca`, the chart's CA Secret, holds `ca.crt`, `tls.crt` and
`tls.key`. The declared source key exists.
- `nats-<name>.tenant-ca`, the projection, holds exactly one key,
`ca.crt`.
- The projection's SHA-256 certificate fingerprint equals the source's,
so it carries that CA and not some other certificate.
- The projection carries `internal.cozystack.io/tenant-ca: "true"` and
`internal.cozystack.io/tenantresource: "true"`. The second is stamped by
the lineage webhook, computed from the ResourceDefinition selector this
PR adds, so its presence means the whole chain held.
- `kubectl get tenantsecret nats-<name>-ca` returns NotFound. The
key-bearing Secret is not served through the tenant API.

What is still missing is an automated regression witness for that
premise. Adding one means a new chainsaw suite for nats, which has no
e2e coverage of any kind today, and that is a larger change than this
one.

## What the tests pin

61 helm-unittest cases and 3 bats cases. The sentinel's absence of
`ownerReferences` (an owner would short-circuit the lineage walk before
it reaches the Flux label the attribution depends on), the length of
`spec.projections` as well as its single entry (the controller refuses a
multi-entry sentinel, so a second entry is a silent loss of the anchor
rather than a leak), both implied legs of the tri-state TLS gate, and
`notExists` on either visibility label for both key-bearing
Certificates.

The bats guard covers the two grant surfaces nothing else asserts on.
The ResourceDefinition is rendered, by the `nats-rd` chart inlining
`cozyrds/*`, but that chart has neither a tests directory nor the `test`
target `hack/helm-unit-tests.sh` looks for, and no
`packages/system/*-rd` package does. The dashboard Role is rendered by a
chart that has both, and no suite under `packages/apps/nats/tests/`
asserts on it.

Both are compared whole rather than filtered down to the entries that
mention a Secret, because every such filter admits the spelling it did
not anticipate. A name whitelist passes an added label selector, and
`controller.cert-manager.io/fao` is stamped on both key-bearing Secrets,
so a selector on it is a one-line grant of the CA private key. On the
Role side, a rule with no `resourceNames` names no Secret, a rule with
`resources: ["*"]` names no resource, and a ClusterRole is not a Role,
while all three grant read access to the CA key. Comparing the rendered
output whole needs no filter and leaves no spelling to find.

## Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff, file by file. No trigger fires, and no follow-up PR is owed.

`packages/apps/nats/templates/tenant-projection.yaml` is a new template
and `packages/apps/nats/tests/` holds chart tests. No package is added,
renamed or removed; `values.yaml` and `values.schema.json` are
untouched, so neither the hand-written Terraform schema, its version
enums, nor its restated defaults have anything to track. `kind`,
`plural` and `release.prefix` are unchanged, and no output Secret or
Service is renamed, so the provider's concatenated name lookups still
resolve.

`packages/system/nats-rd/cozyrds/nats.yaml` is edited in place rather
than renamed, so the ccp external-app skill's dependency-contract anchor
still resolves. The provider offers no typed access to the new
projection, which is the position it already holds for every Secret it
does not model, and not a break.

`packages/apps/nats/README.md` is covered by the one automated path that
does exist: `nats` is present in the website `Makefile` `APPS` list, so
the docs bot regenerates the managed-apps reference page from this
`README.md` on the next stable tag. This is the silent-skip failure mode
the trigger map warns about, so I checked the list rather than assuming;
an app missing from it gets no page at all, which is what happened to
`opensearch`.

`hack/check-nats-rd-secrets.bats` is a new file, not a move or a rename,
and it changes no make target's contract: `Makefile` globs `hack/*.bats`
into `BATS_UNIT_FILES` automatically. The ccp skills anchor on
`hack/package.mk` and `hack/common-envs.mk`, neither of which is
touched.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

Part of #2814.

### Release note

```release-note
feat(nats): expose the CA certificate to tenants as a key-free `<release>.tenant-ca` Secret
```
2026-09-01 13:11:22 +04:00
Aleksei Sviridkin 65e700486b feat(qdrant): publish ca.crt-only trust anchor to tenants (#3341)
## What this PR does

Converges qdrant onto the key-free CA trust-anchor contract (#2814), the
same way #3340 does it for nats. The chart declares a `TenantProjection`
sentinel naming its cert-manager CA Secret. The CA-extraction controller
reads that declaration and publishes a key-free copy, `ca.crt` and
nothing else, as `<release>.tenant-ca`. The qdrant ResourceDefinition
selects that projection by label so it shows up in the tenant's secret
list. No key-bearing Secret gains a tenant-visible label.

The sentinel is gated on the chart's TLS condition. Engines whose
operator mints a CA unconditionally do not need that, but qdrant only
has a CA while TLS is on, and an ungated sentinel would sit
`Ready=False`/`SourceNotFound` on every plaintext release.

This was held while the CA-extraction controller was still open. That
controller landed in #3407, so the sentinel is honoured as soon as this
merges and the hold is gone.

Tests pin the invariants that otherwise fail silently. The sentinel
carries no `ownerReferences`, so the lineage walk falls through to the
Flux release label. Neither key-bearing Certificate stamps a
tenant-visible label on its Secret. The two objects that decide which
Secrets a tenant may actually read, the dashboard Role with its binding
and the ResourceDefinition, are pinned whole rather than checked for
known-bad names: a rule granting secrets with no `resourceNames` is
unrestricted rather than narrow, and a label selector can reach a
key-bearing Secret while every name in the file stays correct, so
neither adds a name for a probe to notice.

Part of #2814.

### Follow-ups

What this PR does not do, and where each belongs instead.

- nats converges the same contract in #3340 and has no
`dashboard-resourcemap_test.yaml`, so the dashboard Role pin belongs
there too. Tracked in #3701 along with postgres, which already carries
the label selector with no guard behind it.
- The qdrant e2e fixture runs plaintext
(`hack/e2e-chainsaw/qdrant/qdrant.yaml` sets `external: false`), so
nothing in CI walks the path from the sentinel to `tenantsecrets`.
postgres has a `verify-tenant-ca-projection` step to copy.
- `hack/check-qdrant-rd-secrets.bats` is engine-agnostic apart from two
literals, the chart path and the expected Secret set. The third copy of
it is the point to parameterise over `packages/system/*-rd/cozyrds/`
instead of copying again.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff, file by file. Nothing needs a manual follow-up, but not for the
reason a values-only walk would give: the website reference page for an
app is regenerated from that package's `README.md`, and this PR does
change it. `qdrant` is in the app list hardcoded in the website
`Makefile`, so the docs bot picks the new section up on the next stable
tag and there is nothing to open by hand. The Terraform provider's
schema is hand-written against `values.schema.json`, which is untouched,
and the provider offers no typed access to the new projection, the same
position it holds for every other Secret it does not model. No package
is added, renamed or removed, and the `cozyrds` file is edited in place
rather than renamed, so the ccp dependency-contract anchor still
resolves.

User-facing documentation for consuming the trust anchor ships with the
chart README, matching #3340.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
feat(qdrant): expose the CA certificate to tenants as a key-free `<release>.tenant-ca` Secret
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Tenant environments can securely access the Qdrant TLS CA certificate
without receiving the private key.
* TLS CA access is automatically enabled when TLS or external access is
configured.

* **Documentation**
* Expanded TLS guidance explains certificate validation and provides
commands for retrieving the tenant CA certificate.

* **Tests**
* Added coverage for TLS projections, secret visibility, dashboard
resources, and secure fail-closed secret exposure.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-01 13:10:37 +04:00
Aleksei Sviridkin 8b6867dc47 fix(mongodb): correct the trust-anchor prose and two e2e assertions
The name-clash note claimed the shared TLS helper documents
<release>-ca-cert as the conventional key-free anchor. It documents the
opposite: the canonical name is <release>.tenant-ca, and it names this
engine as the reason, because MongoDB already claims <release>-ca-cert
with a key-bearing Secret. There is no clash to warn about and the chart
follows the convention rather than deviating from it. The paragraph
above already says <release>-ca-cert holds the CA private key, so drop
the note instead of inverting it.

Guard the ownerReference assertions against an absent list. length() of
null is a type error rather than a failed comparison, and the harness
returns it as an error, which aborts the poll instead of retrying — so
the timeout never applies while the operator is still patching the
reference on.

Restore the two-variable form of the fingerprint comparison. Inlining
both command substitutions into the test swallows a non-zero exit: with
no openssl on PATH both sides are empty, they compare equal, and the
check passes having compared nothing.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-01 11:20:33 +03:00
Aleksei Sviridkin 01cfb5fda0 fix(mongodb): correct what a missing extraction controller does
The README said the sentinel would sit inert with nothing else
affected. The CRD is a template of the controller's own chart, so a
cluster without the controller has no TenantProjection kind and the
release fails to render at all. The caveat also guarded a state that
cannot occur, since the controller is a mandatory platform component —
drop it rather than restate it, matching the postgres chart, and keep
the paragraph explaining the choice of source Secret.

Enumerate all three key-bearing Secrets in the registry negative. Two of
the three were listed, which leaves the third free to acquire the label
and leak a key, the exact reasoning the step above it already spells
out.

Collect TenantProjection status on failure. The controller reports
SourceNotFound and its siblings there, and without it a failed
projection surfaces only as an absent Secret with no reason given.

Rename the source-shape step, which no longer looks at the CA Secret,
and cross-reference the two halves of the leaf Secret's name.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-01 11:20:33 +03:00
Aleksei Sviridkin c0b2ddcdf7 test(mongodb): assert the published trust anchor end-to-end
The existing steps only assert that key-bearing Secrets are NOT promoted
to the tenant, and every one of those is satisfied by an absent label —
they hold just as well if nothing was ever published. Add the positive:
the projection exists, carries exactly ca.crt, matches the source
certificate by fingerprint, and is reachable through the tenantsecrets
registry rather than only as a raw Secret.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-01 11:20:33 +03:00
Aleksei Sviridkin 29d2955241 fix(mongodb): declare the trust anchor with a TenantProjection
A tenant verifying a MongoDB endpoint needs the CA certificate, and
every Secret that holds one also holds a private key: the operator's
<release>-ca-cert carries the CA key, the leaf <release>-ssl the server
key. Granting either would hand over the key along with the anchor.

Render a TenantProjection naming the source, so the CA-extraction
controller can publish the key-free <release>.tenant-ca, and select that
projection by label from the managed-service definition. The sentinel
carries no ownerReferences: it is resolved back to its release through
the label Flux stamps, and an owner reference short-circuits that walk
and silently costs the tenant its anchor.

The sentinel is ungated. The operator issues the certificate chain
whatever the chart asks for, so the source exists for every release and
there is no plaintext leg to skip.

The source is <release>-ssl rather than <release>-ca-cert. It is the
name this chart itself sets in spec.secrets.ssl, so the reference cannot
drift from the CR; the operator writes it on both its cert-manager path
and the self-signed fallback it uses when cert-manager is absent, where
no <release>-ca-cert exists at all; and it holds the merged old and new
CA through a rotation, where <release>-ca-cert holds only the new one
and would stop verifying pods still serving the previous leaf.

Withhold the key-bearing Secrets from the tenant explicitly as well. The
label selector matches on no name, so a single mis-stamp would promote a
private key; the exclude list is evaluated first and closes the
enumerable set.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-01 11:20:32 +03:00