mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-22 04:33:36 +09:00
daf733ad9c1cf8e7bedffca080884caadc88a468
1468
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
c693ed36fb |
fix(postgres): force client-side apply so the autoscaler seed survives
Under active read-replica autoscaling the chart seeds spec.instances at a constant max(replicas, effectiveMin) rather than omitting it (an omitted field defaults to 1, which the CNPG quorum webhook rejects, and — under server-side apply — the phase-2 omit prunes the field whenever KEDA has not yet written /scale, collapsing an idle-at-floor cluster to 1 and shedding its replica PVCs). Keeping the constant only works if Flux does not force-own and revert it. The postgres ApplicationDefinition therefore carries release.cozystack.io/helm-server-side-apply: "false", so the tenant HelmRelease applies client-side: helm-controller patches the CNPG Cluster from the previous-vs-new rendered manifest without consulting live state, the unchanged constant yields no patch, and KEDA's higher live /scale count is preserved. The constant also equals the disabled/dryRun/transition render (instances: replicas) whenever replicas sits at/above the floor, so those flips are patch-free too. Correct the merge-semantics comments across db.yaml, _autoscaling.tpl and values.yaml to describe the two-way previous-vs-new patch precisely (not a "three-way no-op"), and document the residual floor-change hazard: editing replicas / minReplicas / quorum.maxSyncReplicas moves the seed, so that one reconcile does patch spec.instances to the new seed. Add autoscaling.serverAddress so a monitoring:true tenant can point the ScaledObject at a vmselect that carries both the CNPG metrics and kube_pod_labels. Regenerated README/schema/types/cozyrds. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
826f08972f |
fix(apps/postgres): make autoscaler fail-safe, dip-free, and observable
Address review findings on the KEDA read-replica autoscaler where the
implementation diverged from the merged design proposal.
Fail-safe read load. Gate the `or vector(0)` floor on the existence of ANY CNPG
instance pod (`count(kube_pod_labels{...instance_role=~".+"}) > 0`) — primary or
replica — so the query is empty ONLY when no CNPG pod labels exist at all (a
broken kube-state-metrics pipeline), not during initial provisioning when the
sole replica-side pod is CNPG's join Job (jobRole, not instanceRole). Also pin
`ignoreNullValues: "false"` on the trigger: KEDA's prometheus scaler defaults it
true, turning an EMPTY result into metric 0 (not an error) — the HPA would then
desire 0 and clamp to the floor, i.e. silently scale a broken-pipeline cluster
DOWN. With it false an empty result is a scaler error, so the HPA HOLDS and
DatabaseAutoscalerScalerErrors fires; gating on any instance role keeps a healthy
cluster (including a long basebackup) from tripping that alert.
Constant seed = max(replicas, effectiveMin), so enable/dryRun/transition flips
are a merge no-op with no dip and never undercut the quorum floor.
Fail closed when the quorum floor exceeds maxReplicas (was: effectiveMax silently
raised both bounds, unbounded for a large maxSyncReplicas).
Dry-run / transition keep the HPA (unidirectional pause pair, not the HPA-deleting
full pause) but both SelectPolicy=Disabled, so desired normalizes to current — the
recommendation is read from the read-load metric, not a diverging desired count.
Observability. DatabaseAutoscalerAtMax keys on desired >= maxReplicas, NOT the
ScalingLimited condition (which is also true at the floor, so it fired on every
idle cluster with wrong "raise maxReplicas" advice). Dashboard read-load panels
are per-cluster (label_cnpg_io_cluster) with a CPU panel for ReadCPUUtilization
and the exact clamp in descriptions; ScaleStuck's Pending intersection is scoped
to CNPG pods; alerts.yaml narrative comments moved to Go-template comments so they
do not ship into the PrometheusRule.
Guard the seeding invariant: TestBuildHelmReleaseSpecNoForceOrDriftDetection.
Regenerated README/schema/types.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
|
||
|
|
8769232997 |
docs(apps/postgres): correct autoscaling merge-safety notes
The disable case (3) note claimed disabling snaps instances back to .Values.replicas, but the client-side three-way merge compares the newly rendered value against the previously applied one, not the live count. When replicas == effectiveMin (the default) it is a no-op and KEDA's live count is preserved (silent over-provisioning), so it does not snap back. Reword case (3) and the "differs from the live one" framing to match the actual merge semantics. Re-point the no-Force/no-SSA invariant citation from internal/controller/applicationdefinition_helmreconciler.go (which holds no apply-strategy code) to internal/operator/package_reconciler.go buildHelmReleaseSpec, where the HelmRelease spec is actually built. Comment/doc only; helm unittest 52/52 unchanged. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
b40cf82642 |
fix(postgres): seed instances on both paths, add input guards, and harden alerts
Address IvanHunters' deep review round on #3954: - [MAJOR] Enabling autoscaling directly on an existing non-quorum database no longer collapses it to 1 instance and deletes every replica PVC. The chart now SEEDS spec.instances at the quorum floor (effectiveMin) on BOTH the quorum and non-quorum active paths instead of omitting the field in the non-quorum case; the constant seed is a three-way-merge no-op so KEDA still owns the runtime count via /scale, and the enable dip is capped at the floor (>=2) rather than collapsing to 1. Documents the client-side-3-way-merge (not SSA) assumption that the no-op relies on. - [MINOR] Fail closed on autoscaling.minReplicas > autoscaling.maxReplicas instead of silently resolving the ceiling upward (which defeated maxReplicas as a cost knob). The stray maxReplicas: 0 case now rejects rather than clamps. - [MINOR] The disable-wedge guard, its values note and its test no longer emit the invalid `kubectl scale --subresource=scale` (scale has no --subresource); use `kubectl scale cluster.postgresql.cnpg.io/<name> --replicas=N`. - [MINOR] Correct the enablement note: raising minReplicas/maxSyncReplicas on a live cluster is NOT harmless when KEDA's live count exceeds the new floor (it decommissions surplus replicas + PVCs); prescribe pin-first for raises too. - [MINOR] Add DatabaseAutoscalerKedaOperatorDown (absent(keda_build_info)): when the operator is unhealthy a ScaledObject never becomes an HPA and no other alert can fire. Intersect DatabaseAutoscalerScaleStuck with a Pending pod so a healthy long basebackup (Running pod, desired>current >15m) no longer false-fires. - [MINOR] Keep the KEDA CRDs on package uninstall (crds.additionalAnnotations resource-policy: keep via the shim) so removing the keda package doesn't cascade-delete every tenant ScaledObject. - [MINOR] cozy-lib helper: reject a non-numeric bound (int "two" -> 0 -> a scale-to-zero ScaledObject) and render an explicit pollingInterval/cooldownPeriod of 0 (`if not kindIs invalid`, not `with`). - [MINOR] Note the read-load fail-open: a broken metrics join reads as idle and silently pins autoscaling at the floor under real load. New/updated tests cover the non-quorum seed, the min>max guard, the corrected kubectl message, the operator-down + Pending-intersected alerts, the CRD keep policy, and the non-numeric-bound helper guard. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
1ebc021491 |
style(postgres): strip the autoscaling db.yaml comments from the rendered manifest
The autoscaling instance-count explanations were plain `#` YAML comments, so they
shipped verbatim into every rendered CNPG Cluster manifest on every customer
cluster (unlike the {{/* */}} comments elsewhere in the chart). Convert them to
Go-template comments so the rationale stays in-source but is stripped at render,
matching the treatment already applied to the apps.yaml clusterDomain comment.
Also document (as a stripped comment) that WorkloadMonitor.spec.replicas tracks
the quorum floor, not KEDA's live autoscaled target, under active autoscaling.
Addresses the deep cozy-review notes on #3954.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
|
||
|
|
5a5047ac23 |
fix(dashboards): bind the autoscaling dashboard to a datasource variable
The database-autoscaling dashboard hardcoded "datasource": "VictoriaMetrics" on
its namespace template variable, but cozystack provisions VictoriaMetrics
datasources as vm-shortterm / vm-longterm — there is no datasource named
"VictoriaMetrics", so on import the variable query failed ("datasource not
found"), the namespace variable never populated, and every panel (which filters
on namespace=~"$namespace") showed no data. Switch to the repo-standard pattern
used by db/cloudnativepg and the other dashboards: a `datasource`-typed template
variable (ds_prometheus) referenced as { type: prometheus, uid: ${ds_prometheus} }
from the namespace variable and every panel/target (the panels previously carried
no datasource at all and only worked by falling back to the default).
Also reframe the disable-wedge guard message to lead with the invariant (replicas
must exceed quorum.maxSyncReplicas) so a pure-quorum user who never enabled
autoscaling isn't told to "disable autoscaling"; the disable path is kept as a
parenthetical cause.
Addresses the adversarial branch-review findings on #3954.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
|
||
|
|
6e3da63586 |
fix(postgres): guard the disable-wedge, role-scope the freeze count, and harden edges
Address the remaining IvanHunters review points on #3954: - [MAJOR] Disabling autoscaling wedged the HelmRelease under synchronous quorum. An autoscaled + maxSyncReplicas>=1 cluster runs with instances seeded from effectiveMin while replicas stays at its default, so flipping autoscaling off rendered instances: <replicas> next to the quorum and CNPG's webhook rejected it ("maxSyncReplicas must be lower than the number of instances"). Add a symmetric render-time guard on the static-count path (covers off/dryRun/ transition): fail with an actionable message when replicas <= maxSyncReplicas. - [MINOR] Role-scope the freeze brake's currentInstances count (label_cnpg_io_instance_role=~".+") so lingering CNPG join/bootstrap Job pods (cnpg.io/cluster but jobRole, not instanceRole) don't inflate it and nudge desired up while the brake is meant to hold it. - [MINOR] Default maxReplicationLagSeconds to 0 (brake opt-in): only the non-braking path is validated live, so ship the validated default and let the freeze be enabled explicitly until its branch is exercised under load. - [MINOR] Document the floor-change and non-quorum-disable dips (same three-way- merge hazard) and the pin-replicas-first mitigation. - [NIT] Coerce the cozy-lib helper's `paused` (eq "true") so a caller passing the string "false" does not pause the ScaledObject. - [NIT] Trim the keda.clusterDomain injection comment so it no longer ships ~8 verbatim lines into the cozystack-values Secret on every cluster. Regenerate types/schema/README/postgres-rd. New tests cover the disable-wedge guard, the role-scoped freeze count, and the string-"false" paused coercion. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
70e4087432 |
fix(postgres): guard empty monitoring namespace; clarify alert scope and helper doc
Address the branch-review nits: - Fail closed when the monitoring namespace is unset. serverAddress builds vmselect-shortterm.<ns-monitoring>.svc, and an empty ns-monitoring rendered the malformed vmselect-shortterm..svc:8481, surfacing late as a KEDA scaler error. The platform always injects it, but a hand-rendered/misconfigured tenant now gets an actionable render-time failure mirroring the KEDA-CRD and target guards. Tested. - Document that the database-autoscaling PrometheusRule matches every KEDA HPA / ScaledObject error (the underlying series carry no owning-app label), which is correct only while database autoscaling is KEDA's sole consumer in cozystack; note to scope the rules if a non-database ScaledObject is ever added. - Correct the cozy-lib.keda.scaledObject usage example: `paused` should reflect dry-run OR migration transition, not dry-run alone, so a second engine copying the example does not drop the transition pause. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
9cf4a171f0 |
fix(postgres): stop coercing numeric-zero autoscaling bounds and floor each lag gate
Two review notes from IvanHunters: - effectiveMin/effectiveMax wrapped minReplicas/maxReplicas/maxSyncReplicas/ maxReplicationLagSeconds in Sprig `default`, which treats a numeric 0 as empty and silently substitutes the fallback (e.g. maxReplicas: 0 became 6, discarding the tenant's value). values.yaml/schema already default these when omitted, so read them directly; an explicit out-of-range value now flows into max() and is clamped to the quorum floor rather than swapped for a different number. Add a test that maxReplicas: 0 clamps to effectiveMin instead of becoming 6. - The replication-lag brake only floored the product $braking with `or vector(0)`, so if a CNPG build stops emitting cnpg_pg_replication_lag or cnpg_collector_wal_records the individual gate resolved to an empty vector. Floor each gate ($lagHigh, $writing) individually so the fail-open decision is explicit and per-gate: the brake engages only on positively-observed lag AND writing, and a missing signal degrades to "scale normally" (safer than freezing on absent data) rather than relying on the emptiness of the product. Assert both gates carry the floor. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
a7b5077da2 |
fix(postgres): floor the lag-brake frozen term so a KSM gap can't empty the query
The read-load query's base term is floored with `or vector(0)`, but the replication-lag brake's frozen additive term (count(kube_pod_labels) * target) was not. PromQL `+` is a set intersection, so if count(kube_pod_labels) is momentarily empty — a kube-state-metrics scrape gap or KSM rollout — the frozen term is empty and the whole `base + frozen*braking` sum collapses to empty, defeating the base term's own graceful degradation. KEDA's prometheus scaler (ignoreNullValues: true by default) then reads that as 0 and pins desired to minReplicaCount, ignoring real read load exactly when metrics are flaky. This hits the default config (maxReplicationLagSeconds: 30, brake active). Floor the frozen summand with `or vector(0)` so base + 0 = base, and assert the floor in the lag-brake unittest. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
573213ef8b |
test(postgres): assert the standby-only read-load selector in the query
Cover the load-bearing replica selector that both read-load queries depend on: the join must restrict to standby pods via label_cnpg_io_instance_role="replica", the exact key kube-state-metrics exposes for CNPG's cnpg.io/instance_role. A wrong key (e.g. a camelCase label_cnpg_io_instanceRole) would match no series, floor the read term to 0, and silently pin desired to the minimum - the suite previously never asserted the selector at all. Verified live on dev9 (CNPG 1.30.0) that the snake_case key is the one present in kube_pod_labels and the camelCase form does not exist. Adds the assertion for both the ReadConnections and ReadCPUUtilization paths. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
842a59a957 |
fix(postgres): correct the migration-dip docs and pin the KEDA metricType
Two review findings: - The enablement note claimed the two-phase `transition` flow avoids the count dip when enabling autoscaling on an existing cluster. It does not: helm-controller upgrades with Helm's three-way merge (not SSA), which keys off the old-vs-new manifest diff, not KEDA's live /scale value. At phase 2 the chart stops rendering the staged static count, so the field is deleted (no quorum -> CNPG defaults to 1, a brief collapse) or changed (quorum -> rebased to effectiveMin), then the warm KEDA restores it. Rewrite the docs to state the flow reduces (KEDA is pre-warmed, no cold-start) but does not eliminate the dip, and spell out the suspend + scale-subresource pin that actually makes the handoff dip-free. This is the SSA-vs-three-way-merge handoff the design proposal left for the PoC to settle. - Pin metricType: AverageValue on the Prometheus trigger in cozy-lib.keda.scaledObject. The desired = ceil(query / threshold) identity holds only for an AverageValue external metric; it was relying on KEDA's default. Make it explicit and assert it so a re-vendor or default change cannot silently break the arithmetic. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
87b26d6595 |
fix(postgres): only seed spec.instances when synchronous quorum needs it
Narrow the previous unconditional seed to a conditional one, keeping design
proposal §3 (chart omits the field, HPA is the sole writer) intact for the
default path. Without synchronous quorum (maxSyncReplicas=0) CNPG's defaulted
instances=1 is admissible, so the field is omitted exactly as §3 specifies and
KEDA owns it via /scale with nothing contested. Only when maxSyncReplicas>=1 -
where an omitted/defaulted instances=1 fails the admission webhook
("maxSyncReplicas must be lower than the number of instances") - does the chart
seed the quorum floor (effectiveMin) so a fresh Cluster is admitted; the constant
re-render keeps three-way merge from reverting KEDA's higher live value.
This confines the deviation to the case §3 did not consider (sync quorum), so no
design-proposal amendment is needed for the documented path. Verified live on a
dev cluster: the quorum render (instances=2, maxSyncReplicas=1) and the
non-quorum render (instances omitted) are both admitted by the CNPG webhook. Adds
a render test for the non-quorum omit path alongside the quorum seed tests.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
|
||
|
|
610bcb5df9 |
fix(postgres): seed spec.instances at the quorum floor under autoscaling
Under active autoscaling the chart previously omitted .spec.instances entirely,
letting the KEDA HPA own it via /scale. But CNPG defaults an absent
.spec.instances to 1, and its admission webhook rejects a Cluster where
maxSyncReplicas >= instances ("maxSyncReplicas must be lower than the number of
instances"). So a fresh database created with autoscaling.enabled: true AND
synchronous quorum (maxSyncReplicas >= 1) was rejected at admission before KEDA
could ever scale it to effectiveMin (= maxSyncReplicas+1) — the whole tenant
HelmRelease failed. The default maxSyncReplicas: 0 path was unaffected, which is
why it went unnoticed. Verified live: a rendered Cluster with maxSyncReplicas: 1
and an omitted instances is rejected by the dev-cluster webhook; with the seeded
floor it is admitted.
Render instances: effectiveMin under active autoscaling instead of omitting it,
so a fresh Cluster starts at a valid, quorum-satisfying count. Because the chart
emits the same constant floor on every reconcile, Helm's three-way merge is a
no-op on the field and never reverts KEDA's higher live value, so the HPA still
owns the runtime count. Update the enablement docs (a new DB can enable directly
even with synchronous replication; existing DBs above the floor rebase to it
unless the two-phase transition flow is used) and add render tests for the fresh
create + maxSyncReplicas>=1 case.
Addresses the fresh-create-with-quorum admission gap found in branch review.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
|
||
|
|
284f5e5543 |
fix(postgres): fail closed when autoscaling is enabled without KEDA, guard target
autoscaling.enabled is a tenant-facing knob, but KEDA is an optional platform package that is not installed by default. A tenant enabling autoscaling on a cluster without KEDA would render a ScaledObject against an unknown kind and the whole database HelmRelease would fail with an opaque `no matches for kind "ScaledObject"`. Guard the ScaledObject template on `.Capabilities.APIVersions.Has "keda.sh/v1alpha1"` and stop with an actionable message instead. Skipping the render is deliberately NOT an option: db.yaml already drops the static .spec.instances under active autoscaling, so a missing ScaledObject would let CNPG collapse the cluster to 1 instance. Document the KEDA prerequisite on the `enabled` field (README/schema). Also reject a non-positive autoscaling.target at render time: it is a plain integer now, but the desired-replica formula divides the read load by it, so 0 is undefined and negatives invert scaling. cozyvalues-gen has no minimum directive, so the bound is a template guard. Add unittests for both guards (a dedicated suite renders without the KEDA CRD so the fail-closed path is exercised; the main suite now advertises keda.sh/v1alpha1 via capabilities). Addresses IvanHunters (KEDA-absent) and CodeRabbit (non-positive target) review feedback on #3954. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
f8fc2d99bf |
fix(postgres): make autoscaling.target an integer and drop the dead root metric
Two review findings on the generated API: - autoscaling.target was a resource.Quantity, which accepts suffixed values like `150m`. The chart splices target straight into the PromQL query and the KEDA threshold, where `150m` means 150 minutes to PromQL and is not a usable KEDA float threshold. Retype it as a dimensionless integer so the schema rejects suffixed input - helm unittest's schema validation now enforces it, and the existing autoscaling suite exercises the integer contract. - ConfigSpec carried a root `metric` field (and spec.metric in the postgres-rd schema) that nothing reads - the chart consumes only autoscaling.metric - so setting it silently did nothing. Remove the dead root param, keeping the ReadMetric enum that autoscaling.metric references. Regenerate types.go, zz_generated.deepcopy.go, values.schema.json, README.md and the postgres-rd schema; update the autoscaling tests to the integer target. Addresses CodeRabbit review feedback on #3954. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
ec06dafd25 |
feat(postgres): fold the replication-lag brake into the autoscaling query
Wire autoscaling.maxReplicationLagSeconds into the read-load query: while the max standby lag has exceeded the threshold within the cooldown window and the primary is actively writing WAL, the query returns currentInstances*target, pinning desired to the current count and freezing scaling in both directions (safer than pinning maxReplicas, which would still allow scale-down under lag). The lag term uses max_over_time as a hysteresis band; the write gate (WAL-record rate > 0) keeps an idle primary from tripping it. Metrics verified live on CNPG 1.30.0: cnpg_pg_replication_lag and cnpg_collector_wal_records exist and carry namespace/pod (the proposal's cnpg_pg_stat_replication_sent_diff_bytes does not). Also floor the read load at zero via `or vector(0)` so an idle cluster returns target rather than an empty result. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
d4c76fadb4 |
feat(postgres): two-phase autoscaling migration and read-CPU metric
Make enabling autoscaling on an existing database count-preserving, and add a second read-load signal: - autoscaling.transition (migration phase 1): keep .spec.instances rendered as a floor (from replicas, which the operator stages to the live count) AND pause the ScaledObject, so KEDA stands up without contending with Flux for the field. Flip transition off (phase 2) to drop the static field and hand the count to KEDA — suspend the HelmRelease across that apply, since helm-controller uses Helm's three-way merge (not SSA) and would otherwise race CNPG's default-to-1. The procedure is documented on the values field. - autoscaling.metric (ReadConnections | ReadCPUUtilization): the query helper emits either the active-connections sum or the postgres-container CPU sum, both + target. active = enabled && !dryRun && !transition; the ScaledObject is paused whenever dryRun or transition is set. Covered by helm unittest (phase-1 holds the staged count + pauses; CPU metric query) and regenerated schema/README/Go types/postgres-rd. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
0335276b61 |
feat(postgres): render a KEDA ScaledObject for read-replica autoscaling
Implement the KEDA-based read-replica autoscaler for postgres per the merged design proposal (cozystack/community#44): - cozy-lib.keda.scaledObject: engine-agnostic helper rendering a KEDA ScaledObject (scale target, effective bounds, prometheus trigger, scale-down behavior, optional pause), fail-closed on bad input. - postgres: an `autoscaling` values block (enabled / minReplicas / maxReplicas / target / maxReplicationLagSeconds / dryRun). Under active autoscaling the chart drops the static .spec.instances so Flux and the KEDA-managed HPA no longer contend, and renders a ScaledObject whose minReplicaCount folds in the synchronous-quorum floor (max(minReplicas, maxSyncReplicas+1, 2)); when the floor exceeds maxReplicas, quorum wins. dry-run keeps the static count and pauses the ScaledObject. The WorkloadMonitor tracks the quorum floor instead of a stale static value. Covered by helm unittest (cozy-lib helper: valid render, pause, three fail-closed guards; postgres: instances dropped, quorum-floored bounds, quorum-over-max, dry-run pause). Regenerated values.schema.json, README, Go types and postgres-rd. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
3a369fed99 |
feat(redis): add opt-in TLS for Redis and Sentinel (#2729)
## What this PR does `tls.enabled` puts Redis and Sentinel on a TLS-only listener with a per-release cert-manager chain, and hands the tenant the CA certificate as `redis-<name>.tenant-ca`. Off unless set. redis-operator v3.3.5 has no TLS support. The work is ported upstream in freshworks-oss/redis-operator#85, which is open and not in a release, so this carries it as three patches over the v3.3.5 tarball. `images/redis-operator/patches/tls.diff` is the source half, cut from that PR's commit `2ad2ed8d1`, which the patch header pins; it is applied by the glob the image Dockerfile already runs, after the Dockerfile strips the tarball's `manifests`, `charts`, `example` and `.github`, so the image builds from source alone and the upstream tests that read the CRD YAML skip instead of failing against the stale v3.3.5 copy. `patches/crd-tls.diff` adds `spec.tls` to the vendored CRD, and with it the CEL rule that fixes `tls.enabled` at creation; without the schema block the apiserver prunes the field on admission and the patched binary waits for something it never sees, and the rule cannot exist without the block it references. `patches/rbac-tls.diff` grants Create, Get and Update on Secrets, so the operator can publish the CA-only Secret it addresses by name. That is the whole delta, and it is narrower than the ported operator's own chart asks for: those three are the only Secret calls in the patched source, and cert-manager Certificates get no grant at all, because the operator only touches one under `spec.tls.certManager` and this chart pins `certificateSecret` mode instead. One change in `tls.diff` is not TLS and takes effect with TLS off, named in the patch header: `MakeSlaveOfWithPort` connects to the replica on its own port rather than on the master's. It changes a dial, not a pod template, so shipping the rebuilt image rolls nothing; the plaintext pod templates stay byte-identical to v3.3.5. The image build registration this branch used to carry (`make -C packages/system/redis-operator image` in the root `build:` target) landed on main with the Valkey service (#3380), so the rebuilt operator image is produced by CI from main as well; the digest in `values.yaml` is still refreshed by release preparation, not here. The chart patches re-apply at the end of `update:`, the same way reloader, grafana-operator, ingress-nginx, seaweedfs and kamaji carry theirs. Otherwise `rm -rf charts` takes TLS back out on the next `make update` and nothing fails. `make update` reproduces the committed tree byte for byte. TLS is chosen when the instance is created. The config gets `port 0` and the listener moves to `tls-port`, so TLS replaces plaintext rather than running beside it, and a running instance cannot change mode: a replica restarted onto the TLS listener cannot replicate from a master still on plaintext, and Sentinels already on TLS would fail over to a stale replica. The RedisFailover schema in the carried CRD patch rejects a change of `tls.enabled`, which on an existing instance surfaces as a failed HelmRelease upgrade while the instance keeps serving what it did; the README points at creating a new instance and migrating. That is also why TLS is not inferred from `external`: an externally reachable instance is the one most likely to have clients an upgrade must not drop. Existing releases keep their plaintext listener and are not restarted by the rebuilt operator image. The tenant gets `ca.crt` and nothing else, through `tenantsecrets`. The chart's own Secrets take a dot between the release and the suffix (`<release>.ca-tls`, `<release>.tls`, `<release>.ca-cert`): the suffixes are dot-free, so the split is unique and release `foo`'s CA Secret can no longer be release `foo-ca`'s leaf, the collision the hyphenated form had. The chart renders a `TenantProjection` pointing at `<release>.ca-tls`, and the CA-extraction controller re-encodes the one named key from the parsed certificate, so nothing that is not a certificate gets through. Same path as postgres, mongodb, nats and qdrant. No Secret of this release is granted to the tenant by name beyond the generated password, and that includes `<release>.ca-cert`, the CA-only copy the operator publishes for itself. It holds no key today, but the operator has Create and Update on Secrets cluster-wide, so what sits at that name is the operator's call rather than the platform's, and RBAC cannot filter by key. `hack/check-redis-rd-secrets.bats` pins the whole rendered Secret surface instead of a list of known-bad names, because an include with no `resourceNames` and no `matchLabels` restricts nothing. The chainsaw suite for redis gains two cases. A TLS fixture asserts the cert-manager chain is issued, the TLS-probed Redis replicas and Sentinel come up, the operator publishes its CA-only Secret under the widened ClusterRole, the tenant receives exactly `ca.crt` through `tenantsecrets` while the key-bearing Secrets stay off that path, and a `redis-cli --tls` PING through the service name completes with the same CA. A second case applies `tls.enabled` to a running plaintext release and asserts the refusal: the HelmRelease upgrade fails with the schema's message, the StatefulSet keeps its revision with both replicas Ready, plaintext still answers and no TLS listener appears. Certificate renewal rolls the pods from inside the operator. Redis and Sentinel read the certificate once at startup, so the patched operator stamps a hash of the TLS Secret's `tls.crt` and `ca.crt` into the pod template of both the Redis StatefulSet and the Sentinel Deployment on every reconcile, as `redis-failover.freshworks.com/tls-secret-hash`. A renewal changes the hash; the operator's own roller replaces the Redis pods one at a time and the Deployment controller rolls Sentinel. README names the fallback: if the roll cannot run, the deadline is the old certificate's expiry, past which the operator's own TLS connections fail along with the clients. `tls.authClients` maps to `tls-auth-clients` and defaults to `no`. The platform mints no client certificate for the tenant and the CA key is out of reach, so `yes` is only usable when the client certificate comes from whoever also controls the CA. Changing the value on a running release restarts the Redis and Sentinel pods: Redis reads the directive from its config file only at startup, so the operator folds the effective value into the certificate hash that drives the rollout, and the change lands when it is made rather than at the next unrelated restart. Follow-ups in the carried operator patch go upstream first, to freshworks-oss/redis-operator#85 or a successor there, and are picked up here afterwards; the carried patch does not grow Go that upstream has not seen. `tlsConfigFor` fetches the Secret on every call, tens of GETs per reconcile. The renewal roll is static analysis so far: the live check, `cmctl renew` on a running release while watching the `rfr-*` and `rfs-*` pods roll, is still owed and belongs with the e2e follow-up. Two facts about the shipped image rather than the patch. `packages/system/redis-operator/values.yaml` keeps the digest of the image built before the patch until release preparation rebuilds it, so on `main` between this merge and the next release-prep, `tls.enabled` renders the chain and `spec.tls` while the running operator ignores them and serves plaintext. And the pinned v3.3.5 tarball carries GO-2026-5026 in `golang.org/x/net` v0.53.0, reached through `PodService.UpdatePodLabels`; the port neither introduces nor fixes it, upstream `master` is already on v0.55.0, so it lands with the next tag bump rather than here. Outside the patch: `cozyvalues-gen` renders `{}` in the README Value column for `tls.authClients`: a child property without a default inherits the parent object's `{}`, and `make generate` reproduces it byte for byte, so it is a generator bug rather than something to hand-edit here. ### Screenshots No UI changes. ### Downstream repositories - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [x] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: cozystack/terraform-provider-cozystack#37 - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: I walked the trigger map against the diff file by file. terraform-provider-cozystack is reached, tracked as cozystack/terraform-provider-cozystack#37 rather than a speculative PR: `values.schema.json` gains a `tls` object, and the provider hand-writes a schema, model and expand/flatten pair per Kind. It already models `tls` for postgres, kafka, nats and qdrant, not for redis. Redis needs different wording from those four, whose descriptions say to omit the block to follow `external`, which is the one thing redis does not do. Nothing else is reached. The website regenerates this package's reference page from `README.md` on a stable tag, so the README changes need no PR there. No package added, renamed or removed. `packages/core/platform/values.yaml`, variant and bundle composition and release asset names are untouched, and so are `hack/package.mk`, `hack/common-envs.mk` and `hack/update-crd.sh`, which is what ccp and external-apps-example read. The two new files under `hack/` are bats unit tests. No node prerequisite, installer value, telemetry metric or cozy-proxy label changes. ### Release note ```release-note feat(redis): add opt-in TLS for Redis and Sentinel. Set `tls.enabled: true` to serve both on a TLS-only listener backed by a per-release cert-manager chain; the tenant reads the CA certificate as the `<release>.tenant-ca` tenant secret. TLS is off unless set and is not inferred from `external`, so existing releases keep serving plaintext and are not restarted by the rebuilt operator image. TLS is chosen at creation: a running instance cannot switch modes, the change is rejected and surfaces as a failed HelmRelease upgrade, so add TLS by creating a new instance and migrating to it. Requires cert-manager, which the platform's `system` bundle installs on every variant. A renewed certificate rolls the Redis and Sentinel pods through the operator. TLS support is carried as a downstream patch over redis-operator v3.3.5 until freshworks-oss/redis-operator#85 lands in a release. ``` |
||
|
|
d800526696 |
feat(mariadb): add TLS support via cert-manager (#2680)
## What this PR does Gives MariaDB a chart-managed certificate chain, makes TLS enforcement explicit, and hands the tenant a key-free trust anchor instead of a private key. - `tls.issuer` selects who issues the server certificate: `operator` (the CA mariadb-operator manages itself, the default and today's behaviour) or `cert-manager` (the instance gets its own CA, and this is what covers the external hostname). The field is named for the issuer rather than for on/off because TLS is served under both — unlike the similarly spelled field in some other charts, which really is a switch. - `tls.required` refuses plaintext connections. It is opt-in and defaults to false: enforcing TLS before the tenant can obtain the CA would be backwards, and a platform upgrade must not sever connections that work today. - Neither issuer is inferred from `external`. Existing instances keep the CA they have and change nothing on upgrade. - The tenant no longer receives `<release>-ca-tls` or `<release>-tls` — the CA private key and the server private key. It receives the operator's `ca.crt`-only bundle, which works today, and the ResourceDefinition additionally declares that bundle as the source for the CA-extraction controller. ### Fixes along the way - The TLS backup path could never have worked: it passed `--ssl-mode=VERIFY_IDENTITY`, a MySQL-8 flag the MariaDB client rejects during argument parsing. Replaced with `--ssl-ca` + `--ssl-verify-server-cert`, and the test now asserts the dialect rather than the presence of a substring. - The leaf certificate carried `clientAuth`. Because `REQUIRE X509` accepts any certificate signed by a trusted CA, the server's own key was usable as a client credential. Removed. - The CA and server private keys survived deletion of the instance: cert-manager runs with `enableCertificateOwnerRef: false`, so removing the `Certificate` left the Secrets behind. Both are now cleaned up by the existing post-delete hook. - The Secrets are labelled for the operator's watch predicate, without which a renewed leaf would not reach the pods until a full resync. ### Why the include carries an exclude list The trust-anchor selector matches by label with no name constraint, so the enumerable dangerous set is excluded explicitly — both issuers' private keys, the internal credentials, the backup keys, and the metrics configuration, which the operator writes unconditionally with credentials in plain text for a user holding `SELECT` on everything. The backstop is bounded and the comment says so: matching is by exact name with no globs, so a Secret introduced by a future operator version, or a transient object created mid-renewal, stays in scope of the label-only include. The primary defence remains that the controller stamps only its own projection. ## Known limitations 1. **Merge is blocked on #3299 — and the blocking is mechanical, not just procedural.** The CRD on `main` does not carry `spec.caCert`, and under Flux's Server-Side Apply the API server rejects a field the schema does not declare (`field not declared in schema`); under the older client-side apply the same field was silently dropped, so installing this branch without #3299 fails outright. An earlier revision of this description claimed the field would be silently pruned; that was wrong. The tenant still gets a usable trust anchor once merged in order, through the operator's bundle. 2. **The deprecated backup CronJob cannot start at all**, for reasons that predate this branch: its `secretKeyRef` names a Secret and key the chart never renders. That is filed separately as #3347 and deliberately not fixed here, since repairing it would start scheduled dumps on production instances from a TLS change. The tests that pin the backup TLS flags therefore pin a path that does not currently execute — the suite header says so. 3. **The operator's `clusterName` is hardcoded**, so on a cluster with a non-default domain the SANs and the name the operator dials diverge. Pre-existing, raised separately. 4. **No e2e coverage for the managed path.** Unit coverage pins the rendered chain, the SAN set, the exclusion set and the guards; the end-to-end handshake is not asserted in CI. ### On review coverage Two independent review passes were run repeatedly against this branch — seven rounds, converging from a security finding, through comment accuracy, to guard strength, and finally to nothing new. A third, model-based pass reviewed the branch through `6cb1bd236` and its findings were fixed; it could not be run against the final revision because of an external account quota lasting until 25 July. The delta since that revision is substantial and includes runtime code, not only tests and docs. Recording this so the coverage is not overstated: if the quota returns before this is ready to merge, that pass is worth running again. ### Downstream repositories `packages/apps/mariadb/values.schema.json` gains the `tls` object, so `cozystack/terraform-provider-cozystack` needs the field modelled for `cozystack_mariadb`; tracked in cozystack/terraform-provider-cozystack#40. ### Release note ```release-note feat(mariadb): chart-managed TLS via cert-manager, opt-in enforcement, and the CA published to tenants as a key-free trust anchor ``` |
||
|
|
487cafac3e |
feat(platform): let a tenant pick its Gateway class from an allowlist
Tenant Gateways were pinned to the bundled Cilium class. Give the platform a default in gateway.className, the set a tenant may choose from in gateway.tenantSelectableClasses, and the set whose provider terminates TLS upstream of the Gateway in gateway.edgeTerminatedClasses. A tenant picks its own with tenant.spec.gatewayClass, read only when it owns a Gateway; an inheriting tenant runs on its ancestor's. The selectable set is an allowlist rather than free choice because Tenant is a tenant-writable object while a GatewayClass is cluster-scoped: without it a tenant could attach its Gateway to a class an admin keeps for internal-only traffic. Naming anything outside the set fails that tenant's own gateway release and reaches no other tenant. Where TLS ends is a property of the class, so the gateway chart derives certMode from whichever class the tenant landed on instead of taking it as a separate input. The two therefore cannot be configured out of step. The publishing tenant cannot cross that line by itself. The cluster-wide ACME issuers point their HTTP-01 solver at its Gateway and are rendered from the platform default alone, so a mismatch would pin the solver to a listener that Gateway does not have. Those issuers now drop the sectionName pin when the default class is edge-terminated, letting the challenge attach by hostname instead. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <3811295@gmail.com> |
||
|
|
b24ed36f00 |
test(mariadb): bind the uninstall Secret names to the delete that removes them
Both cleanup suites asserted the delete as "(?s)kubectl delete secret -n \S+.*<name>". Under (?s) the dot matches a newline, so the gap runs past the end of the command: any earlier "kubectl delete secret" satisfies the opening and the names may then sit on any later command in the script. The pattern never established that they are arguments of the delete at all. Moving the two cert-manager Secrets off the delete onto a "kubectl get", leaving the password delete above them untouched, keeps both suites green while the CA private key and the server private key outlive the release in a namespace the tenant shares with its other applications — the one outcome this hook exists to prevent. The comments beside the assertions already claimed the verb was pinned, so a reader trusting them had no reason to re-check; they stand unchanged because the patterns now do what they say. Pin the whole invocation with its argument list instead, spelling out the shell's line continuations rather than stepping over them. Whitespace and backslashes are the only separators the pattern admits, and no command can be written in those, so a match cannot leave the command it began in. That mutation now reddens both suites, as does dropping one name from the list, as does swapping the verb for a read. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
3fd032e28b |
docs(mariadb): say the CA Secret is excluded by name, not unnamed
The clause claimed the ApplicationDefinition names neither Secret. It names this one: "mariadb-<name>.ca-tls" sits in its secrets.exclude list, and the bats guard pins it there. What is true is narrower — neither is named as the source the anchor is published from — so say that and say where the name does appear, or the sentence invites a reader to conclude the ApplicationDefinition has no reference to the Secret at all. Rewrap the SAN-list paragraph's one 124-column line back into the block it sits in, which is otherwise at 78. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
55a4046541 |
test(mariadb): pin the schema rejection of a misspelled tls.issuer
The issuer resolver maps anything that is not exactly "cert-manager" to operator issuance, so a typo fails open: the instance keeps the operator's CA and nothing at render time reports that the value was not understood. The enum is the only thing that refuses it, and one kubebuilder marker generates that enum into both the chart's values.schema.json and the ResourceDefinition's copy, so a single lost marker opens both guards at once. The suite already pins the rejection for the sibling field on this object and left this one, the direction that fails quietly, uncovered. Removing the enum from the schema fails this case and no other. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
2e759c7008 |
docs(mariadb): correct what names the trust anchor and what lineage resolves
Two claims the chart makes about itself stopped being true when the anchor moved to a projection. certmanager.yaml said the ApplicationDefinition names the operator's key-free "<release>-ca-bundle" as the publication source. It does not: the TenantProjection this chart renders names it, and the ApplicationDefinition carries no name for it at all — it selects the published anchor by label. The ApplicationDefinition's own include comment then listed the two cert-manager Secrets among those the lineage webhook resolves to this instance. The webhook resolves an object through ownerReferences, or failing that the Flux helm label, and returns nothing when it has neither. A cert-manager Secret has neither while enableCertificateOwnerRef stays false, which is the upstream default this platform does not override, and is the same fact the uninstall hook relies on when it deletes those Secrets itself. What does resolve is the operator's own "-ca", "-server-cert", "-client-cert" and "-ca-bundle". The two names stay in exclude, because one setting flip is all it takes to make the sentence true. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
4758718d66 |
fix(mariadb): separate the chart-chosen Secret suffixes with a dot
The chart asks cert-manager for two Secrets and names them after the release: "<release>-ca-tls" for the CA and "<release>-tls" for the leaf. The release prefix is "mariadb-", so an application named foo produces mariadb-foo-ca-tls, and an application named foo-ca produces mariadb-foo-ca-tls as well. One release's CA private key and another release's server private key claim the same object. Two sibling applications in one tenant namespace are enough to hit it: whichever cert-manager writes last owns the Secret, and each instance then serves or trusts material issued for the other. The suffix set is what makes it ambiguous. It holds both "-tls" and "-ca-tls", and one is a tail of the other, so a hyphen leaves nothing to say where the release name ends. A dot does: an application name is validated as a DNS-1035 label and the release prefix as ^[a-z0-9-]*$, so neither can contain one, and "<release>.ca-tls" and "<release>.tls" split uniquely. Secret names are DNS-1123 subdomains, which admit the dot. Only the two suffixes this chart chooses move. The Certificate and Issuer object names keep the hyphen, because their suffixes are not a pair where one is a tail of the other, and the operator-managed "<release>-ca-bundle" that the trust-anchor projection reads is not this chart's to rename. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
e94fcf8219 |
feat(backups): RabbitMQ definitions backup via a dedicated strategy driver (#3990)
## What this PR does Adds backup/restore support for the managed `RabbitMQ` application. RabbitMQ's cluster operator ships no backup CRD and RabbitMQ has no consistent online export of message payloads, so this backs up broker **definitions** (vhosts, users, permissions, queues, exchanges, bindings, policies, parameters) over the management HTTP API (`GET`/`POST /api/definitions`) and stores them in the system bucket. Message payloads and point-in-time recovery are intentionally out of scope; a full-data backup would be a Velero volume snapshot, which is a separate strategy. It is delivered as a **first-class `strategy.backups.cozystack.io/Rabbitmq` driver**, following the Altinity pattern (a dedicated driver that renders a one-shot Job for an HTTP-API engine): - `reconcileRabbitmq` / `reconcileRabbitmqRestore` render the strategy's `PodTemplateSpec` — `curl` `GET`/`POST /api/definitions`, pushing/pulling the JSON to S3 with `curl --aws-sigv4` — as a `batch/v1.Job`, covering both restore flows: in-place (into the source broker) and to-copy (into a separate, differently-named RabbitMQ in the same namespace, authenticating with the target's own default-user credentials and reading the source's object key). - A **Go-read precondition** gates each run on the `RabbitmqCluster` reporting `AllReplicasReady` (read through a small `rabbitmqtypes` partial type), so a not-yet-ready broker surfaces a precise `Ready=False` wait on the BackupJob/RestoreJob rather than a Job that starts and fails against an unreachable management API. - The produced `Backup` records the stored object's location on `status.artifact.uri`, rendered from the strategy's `spec.artifactURITemplate`. The `Backup` CRD has no status subresource, so this persists on Create, mirroring the Etcd driver. Dispatch is wired in both controllers (plus `supportedBackupStrategyKinds`), `rabbitmqtypes` is registered in the scheme, and RBAC grants read on `rabbitmqclusters.rabbitmq.com` (the Job itself is already covered by the existing `batch/jobs` grant). It ships `cozy-default-rabbitmq` (a `$bucketName`-gated `Rabbitmq` strategy) and routes `apps.cozystack.io/RabbitMQ` to it in the `cozy-default` BackupClass, which also enrols it in the `DefaultObjectsGate`. The RabbitMQ application chart is left untouched — it had no prior backup wiring, and the strategy carries the S3 coordinates itself. Unit tests cover the driver's pure helpers (applicationRef validation, cluster-name derivation, the readiness-message builder, artifact-URI rendering, parameter round-trip) and the reconcile precondition gate (defers with `Ready=False` and no Job until the cluster is ready, then proceeds). A runnable example under `examples/backups/rabbitmq/` and an honest-roundtrip Chainsaw e2e under `hack/e2e-chainsaw/rabbitmq/` seed a sentinel vhost, back it up, drop it, and assert it round-trips through object storage on both restore flows. The Job pod reuses the existing `platform-migrations` image (curl + ca-certificates) under a new `backupStrategyController.rabbitmqBackupClientImage` key, stamped in lockstep with `chBackupClientImage` in `packages/core/platform/Makefile` so the repository stays pinned at a single digest (`hack/image-pin-consistency.bats`). A dedicated slim image built by this package is a follow-up. **Validation:** `go build`/`go vet`/`gofmt`/`go test ./internal/backupcontroller/... ./internal/template/...`, deepcopy + Rabbitmq CRD generation, `helm template` proving the CRD-ordering gate both on and off, `helm lint`, and `helm unittest` (18/18). The backup + in-place + to-copy roundtrip was verified end-to-end on a dev cluster (a real management-API export and a real S3 `--aws-sigv4` upload/download against SeaweedFS), so the driver's core contract is exercised on a real cluster; the in-CI Chainsaw run is the remaining automated evidence. The object-cleanup, terminating-namespace teardown and readiness-deadline paths added during review are unit-tested (including `Reconcile`-level finalizer retention and the terminating-namespace release) but not yet cluster-exercised. (Note found while testing: RabbitMQ 4.2 is OOMKilled on the `nano` preset, so the example/e2e use `c1.small`.) ### Screenshots Not applicable — no UI changes. ### Downstream repositories Walked the trigger map against the diff. This is an additive backup strategy plus a new strategy CRD; no downstream repository is mechanically forced by the change. A documentation page for RabbitMQ backup on [cozystack/website](https://github.com/cozystack/website) would be a reasonable follow-up, but it is not required by this diff — leaving the boxes unticked for a human to decide rather than ticking "no downstream" to clear the checklist. ### Release note ```release-note feat(backups): add RabbitMQ definitions backup/restore via a dedicated strategy.backups.cozystack.io/Rabbitmq driver; it gates each run on the RabbitmqCluster being ready and records the stored object on Backup.status.artifact.uri. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added RabbitMQ backup and restore support for definitions, including vhosts, users, queues, exchanges, bindings, policies, and related settings. * Added in-place and restore-to-copy workflows with readiness checks. * Added scheduled and on-demand backup examples. * Integrated RabbitMQ backups with the default backup configuration. * Backups store definitions only; message payloads and point-in-time recovery are not included. * **Documentation** * Added a complete RabbitMQ backup and restore demonstration guide. * **Tests** * Added end-to-end coverage for readiness and backup/restore round trips. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
2b99853e6d |
fix(networking): prevent webhook admission outages during namespace churn (#3997)
## What this PR does Moves ingress-nginx and kubeovn-webhook pod deny rules to source-side cilium policies, and keeps a target-side `ingressDeny` on the webhook pods beside them, so enforcement does not depend on which component performs the service DNAT. Pod access to both webhooks stays denied, ingress-nginx keeps exception for konnectivity agent (now keyed on its ServiceAccount instead of `k8s-app` label that any pod can set on itself), and external `world` deny stays as static target rule. Source-side rules deny only backend port. That covers ClusterIP traffic too, because cilium translates service to its backends before it enforces egress policy, and that ordering exists because `packages/system/cilium/values.yaml` sets `kubeProxyReplacement: true` against upstream default of `"false"`. The value is operator-reachable, so the target-side rule is what keeps the deny working if it is ever flipped, and two render-time beacons make a flip visible instead of silent. Naming the frontend with `toServices` is not possible, and templates now say why, because the construct looks correct and fails silently: cilium resolves `toServices` only in allow rules (`pkg/policy/k8s/service.go` walks `spec.Egress`, never `spec.EgressDeny`), so such `egressDeny` gets non-nil empty L3 selector from `pkg/policy/utils/parserules.go`, and `pkg/policy/types/policyentry.go` treats that as wildcard when rule carries `toPorts`. This does not remove the namespace-churn map writes. The target-side rule carries a cluster-spanning peer, so those writes come back. What changes is that they stop being load-bearing, because whichever end is degraded the other still denies. Cost is policy-map cardinality on the two webhook endpoints, and residual risk is an endpoint near `MaxEntries` getting the inverted add-before-delete order (`pkg/endpoint/bpf.go:1168-1177`). Both policies also change kind and name: namespaced `CiliumNetworkPolicy/ingress-nginx-admission-webhook` and `CiliumNetworkPolicy/kube-ovn-webhook` become cluster-scoped `CiliumClusterwideNetworkPolicy/ingress-nginx-admission-webhook-c5d94713a379` and `kube-ovn-webhook-c2bdbc146112`. Anything keyed on the old namespaced objects goes quiet with no error. ## Tests Package suites pin the deny shape per package, one rule per spec with a destination selector cilium can actually resolve, and pin the peer shape of the target-side rule against the two substitutions that would deny the API server (`fromEntities: [all]`, and dropping the peer). Checked by mutation, all of those go red. `matchRegexRaw` and `notMatchRegexRaw` are not used, helm-unittest 1.0.3 ignores both. `packages/system/kubeovn-webhook` had no `test` target in Makefile, and `hack/helm-unit-tests.sh` finds suites by probing `make -n test` per package, so all three of its suites never ran in CI, including port agreement guard that exists to catch a drift failing open. Added the target, plus `.PHONY: test` to it and to cilium. E2e churn case proves webhooks stay available. Second case proves they are still unreachable from pods, and that deny is confined to webhook port. Both now gate every verdict on the admission endpoints having ready addresses before, during and after, on controller pod identity and restart count being equal across the window, and on a positive control where the same pods' own 443 must stay reachable. So a dead or flapping backend fails as setup failure instead of reporting a deny that was never enforced. First run of these cases reported a false verdict for exactly that reason. ## Not verified Original outage in #3983 was not reproduced in isolation, and its symptom argues against the mechanism this PR was written around. `connect: no route to host` is `EHOSTUNREACH`, and a policy denial cannot produce it, since a deny drops after `connect()` has already returned and the caller sees a timeout. On a ClusterIP that errno is what cilium's socket LB returns with no usable backend, and socket LB is active in host netns because `pkg/kpr/kpr.go:47-49` forces it whenever kube-proxy replacement is set. Namespace deletion churns EndpointSlices and service backend maps as well as identities, which is directly on that path. So this change is justified as removing a structural weakness and adding a topology-independent deny, not as a confirmed root cause fix. What would settle it: `cilium-dbg monitor --type drop` on the controller's node during the window, agent log checked for the map-overflow warning, and the admission Service's ready backend count across those seconds. `world` deny that issue named as its first suspect is kept, on `cilium monitor` evidence recorded when kubeovn-webhook policy was written - api server arrives with identity `host` on same node and `kube-apiserver` from another, never as `world`. That is one topology, a kube-ovn-chained management cluster, and it is not re-derivable on cilium-primary variants or on a Kamaji guest cluster without a run. Whether cilium without kube-proxy replacement is a supported configuration is a maintainer decision and is open. If it is not supported, a `fail()` in the cilium chart is the right shape and should follow this PR. Refs #3983. ### Screenshots Not applicable. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(networking): enforce the Ingress and Pod admission webhook deny on both the caller and the webhook endpoint, so it no longer depends on Cilium performing service load-balancing. Both policies become cluster-scoped: CiliumNetworkPolicy/ingress-nginx-admission-webhook and CiliumNetworkPolicy/kube-ovn-webhook are replaced by CiliumClusterwideNetworkPolicy/ingress-nginx-admission-webhook-c5d94713a379 and kube-ovn-webhook-c2bdbc146112. ``` ## Summary by CodeRabbit * **Bug Fixes** * Improved admission webhook reliability during namespace deletion and churn. * Preserved webhook protection while allowing unrelated secure traffic to continue. * Corrected webhook access controls for both ingress-nginx and kube-ovn. * **New Features** * Added cluster-wide policy support with safer backend-port enforcement and automatic capability detection. * Added safeguards for policy naming and configuration overrides. * **Tests** * Expanded automated coverage for webhook availability, access restrictions, policy rendering, and end-to-end suite selection. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
9ece9dcbc3 |
test(mariadb): cover every issuer state the projection claims to ignore
The case titled "renders regardless of tls.issuer" set issuer: operator, which is the chart default, so both sides of its claim were the same side. Hiding the sentinel behind the cert-manager gate — the one edit the title rules out — left it green. Being ungated is the projection's entire contract, so each state is now its own case: the default, operator, cert-manager, and an explicit null, which the issuer helper treats as unset rather than as a spelling of the default. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
e17c898724 |
test(mariadb): make the backup CA-volume negative able to fail
The notContains element omitted optional: true, and contains compares each list element whole rather than as a subset, so the element it described equalled nothing the template can render. Mounting the server key Secret as the backup CA volume — the exact substitution the assertion names — left it green, and only the neighbouring positive caught the mutation. The comment two cases below already described this trap as handled, which is how it survived being reintroduced. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
de443de487 |
test(mariadb): bind the uninstall assertions to the delete verb
Both suites matched only the Secret name list in the hook's script, so replacing kubectl delete with kubectl get left all 106 cases green while the CA private key and the server private key survived the release. The verb is the claim these tests exist to make; the names alone are just text that happens to appear in a shell script. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
a79a6d4aa4 |
docs(mariadb): say that switching issuer rolls the StatefulSet
The migration procedure was headed "the order that avoids downtime" and closed by claiming step 3 is the only step that can refuse a connection. Step 1 restarts the instance: under cert-manager the CR gains serverCertSecretRef and serverCASecretRef, so the name of the Secret projected into the pods changes, and a pod-template change rolls the StatefulSet. The chart supports single-replica instances, and for those a rolling restart is plain downtime. The document already knew this in the other direction — the going-back section warns that deleting the Secret before the operator has reissued and rolled the StatefulSet leaves the next pod unable to start, and prints the rollout-status command — so the two halves contradicted each other. Both claims are now scoped to what they are true of: the order still matters, because it keeps enforcement behind the client migration, and step 3 remains the only step that refuses a client by policy rather than by restarting the server under it. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
57784714d9 |
fix(mariadb): state the real reason the cleanup hook may read cert-manager Secrets
The rule sold itself as least privilege while granting -ca-tls and -tls under every tls.issuer, including operator, where neither Secret is ever rendered. The comment gave no reason for that, which reads as an oversight in a rule whose whole point is scope. Unconditional is what the template's own logic supports, so the reason is now written rather than the grant narrowed. The delete above is unconditional too and leans on --ignore-not-found: a release issued by cert-manager and later switched to tls.issuer: operator still has both Secrets on the cluster, and uninstall renders the Role from the values current at that moment. Gating the names on the issuer would drop the grant for exactly that release, failing the delete on RBAC and leaving the CA and server private keys in a namespace the tenant shares with its other applications. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
eebfdc023c |
test(mariadb): point the projection test at the suite that covers the certificates
The comment sent the reader to certmanager_test.yaml, which this chart does not have. Its cert-manager coverage — the rendering of the CA and leaf Certificates under each tls.issuer value — lives in tls_test.yaml. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
a14bff8c6a |
test(mariadb): guard key-bearing Certificates with the tenant labels
The publication guards asserted notExists on the labels internal.cozystack.io/publish-ca-cert and its -release variant. Neither name exists anywhere in the tree. The chart renders only k8s.mariadb.com/watch into secretTemplate.labels, so a notExists on a path no template can produce holds under every value, and the two cases titled for a CA publication source could not fail for what they named. The labels that decide tenant visibility are tenant-ca, which the ApplicationDefinition's secrets.include selects on, and tenantresource, which tenantsecrets serves on. Both are now asserted absent on each key-bearing Certificate. Exclude-by-name in the ApplicationDefinition stays the backstop behind them, not a substitute. The documents are addressed by the Secret each Certificate writes plus an explicit kind rather than by documentIndex, because an index follows the slot: reordering the template would slide a labelled key-bearing Certificate out from under the assertion and pass green. metadata.name cannot serve, as the CA Certificate and the CA Issuer share the name <release>-ca. The operator-issuer case promised a label check its body never made; it asserts only that the template renders nothing. Under that issuer no Certificate exists to label, so zero documents is the whole statement available, and the title now says that instead. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
8a6f4884cb |
fix(mariadb): publish the tenant CA anchor through a projection, not by name
The dashboard Role and the ApplicationDefinition granted the operator's key-free mariadb-<name>-ca-bundle Secret directly by name. Every other converged engine (mongodb, postgres, nats, qdrant) instead renders a TenantProjection that lifts ca.crt into the canonical <release>.tenant-ca object, reached through core.cozystack.io/tenantsecrets rather than a per-Secret grant. Add templates/tenant-projection.yaml sourcing from the operator's ca-bundle, drop the name grant from the dashboard Role and from the ApplicationDefinition's secrets.include (the existing tenant-ca matchLabels selector now has something to match), and remove the ApplicationDefinition's caCert field, which named a field the ApplicationDefinition CRD does not define and would have rejected the whole object under server-side apply. Update the README and the pinned RBAC tests to the new tenant-ca name. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
69d5ce0644 |
test(mariadb): assert the CA error names its Secret, and correct the ledger
The backup CA case is called "checks the CA file exists and names it" but only guarded the first half. Deleting the Secret name from the error left the suite green, so the reason MYSQL_CA_SECRET is passed to the CronJob at all — telling the operator which Secret to go look at, since the client reports a missing CA without naming a file or its source — was unpinned. The cozyrds suite header also counted its own coverage wrong. It claimed two live tests and three placeholders; there are seven and four. The four exclude guards are live and went unmentioned, and the case reading the source key is inert like its neighbours but carried no placeholder marker. That header is the file's honest-coverage statement, so an understated ledger is the one error it cannot afford. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
6cb1914ec0 |
test(mariadb): assert the guards abort, not just that they are spelled
Three assertions matched text that survived the mutation they exist to catch. The backup CA guard was pinned by three separate substrings: the two test expressions and the message. Flipping its exit status to 0, or short circuiting the condition with `false &&`, leaves all three intact and the suite green — while the backup runs on with --ssl-ca pointing at a file that is missing or empty, and in the exit-0 case reports success having dumped nothing. Match the whole construct through its exit instead. The exclude entries were checked by name without checking they still sit under resourceNames. A nil resourceNames matches every name, so dropping that key turns the backstop into a match-everything selector while each name is still present in the block. Assert the key. The leaf certificate pinned renewBefore but not duration. cert-manager rejects a duration that does not exceed renewBefore, so pinning one of the pair lets the other drift into a certificate that renders but cannot be applied. Each is verified by the mutation it was blind to: exit 1 to exit 0, a short-circuited condition, a renamed resourceNames key, and an 8760h duration shrunk to 1h all now fail the suite. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
66d2c9b399 |
test(mariadb-rd): scope the exclude guards to the exclude block
The backstop guards grepped the whole file for each name. Both lists sit at the same indentation, so an entry in `include` satisfied a guard that exists to prove the name is in `exclude` — and moving a name between them is the single most dangerous edit this file allows, since exclude is what keeps a Secret away from the tenant. Moving `-ca`, the operator's CA private key, from exclude to include left all ten guards green. So did moving `-root`. Extract the exclude block and match inside it; both mutations now fail on the name that moved, and the suite is unchanged otherwise. Also correct two descriptions the TLS work outdated. The cleanup hook's Role comment still said the rule covered two operator-generated Secrets; it now covers four, and two of them are issued by cert-manager, which the block above already explains. The runtime message called them all operator-generated for the same reason. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
18a8bf296d |
test(mariadb): scope the CA-bundle guard and name the CR field it checks
The guard asserting that the trust anchor is surfaced to the tenant grepped the whole file. The bundle is also named in a comment and in the caCert declaration, so either of those could satisfy it while the secrets.include entry it exists to protect was gone. Scope it to the include block, the way the sibling key-bearing-Secret guard already does. Verified by deleting the include entry with a matching string left elsewhere in the file: the guard now fails, where before it passed. The suite header described a comment as stating that spec.tls.issuer is true. There is no such field on the MariaDB CR — issuer is a chart value, and what the chart renders is spec.tls.enabled, which is what the rest of that sentence already describes. Name the field the assertions actually check. Also drop the claim that an 11.4 client verifies by default. The default that changed in 11.4 is the --ssl-verify-server-cert flag, not whether verification happens: without --ssl-ca that client has no trust anchor, and falls back to an anti-tampering check derived from the password hash which establishes neither the chain nor the hostname, and which cannot be derived at all for a passwordless user. Passing both flags is what gives chain and hostname verification, on either client version. The documented command was already correct; the sentence around it invited dropping the flag that carries it. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
92109b2a1b |
docs(mariadb): correct two claims in the TLS security rationale
Both comments justify a security decision, and both overstated it. The exclude list described `-ca`, `-server-cert` and `-client-cert` as what the operator creates "when it issues its own" certificates. Only `-server-cert` is conditional. The chart never sets the client refs, so at the pinned 25.10.2 `WithCA`/`WithCert` gate on those refs being nil and the operator issues the client leg — and `-ca`, which carries the CA private key — under both issuers. The list was already right; the comment invited a future reader to prune two live private-key names as inapplicable under cert-manager. The watch label was described as the difference between picking up a cert-manager renewal and waiting for the next full resync. The operator requeues every five minutes whenever TLS is on, so the label buys promptness, not correctness. Say that, and note the operator sets the same label on Certificates it builds itself, which is why the chart mirrors it here. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
5a27ee4e88 |
fix(mariadb-rd): exclude the exporter config Secret from tenant promotion
The exclude backstop enumerated the internal credential Secrets but
omitted `-metrics-config`. The operator writes it for every instance —
`MetricsConfigSecretKeyRef` at the pinned 25.10.2 fixes the name as
`<name>-metrics-config` — and its `exporter.cnf` carries
[client]
user=...
password=...
in plaintext. The exporter user is granted SELECT, PROCESS, REPLICATION
CLIENT, REPLICA MONITOR and SLAVE MONITOR on `*.*`, so that password
reads every table in every database.
It is not a hypothetical Secret: the chart sets `metrics.enabled: true`
unconditionally, so it exists on every ready instance. The include side
selects by label with no name constraint, and a nil `resourceNames`
matches everything, so the exclude list is what keeps it away from the
tenant. Exclude beats include, so naming it there is sufficient.
This is inside the enumerable part of the gap, not the part the comment
already disclaims: that carve-out covers Secrets a future operator adds
or creates mid-rotation, and this one exists today under a stable name.
Also complete a sentence in the suite that lost the words naming which
issuer it describes.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
|
fbb4595067 |
test(mariadb): say that the backup cases pin a path that does not run
The eight cases covering the backup client's TLS flags read as coverage of a live path. They are not: that CronJob cannot start, because it reads the root password from a Secret name and key the chart does not render, and the mismatch predates this branch. Keep the cases — the flag defect was real, and they are what will stop --ssl-mode returning whenever the job is repaired or dropped — but say in the header what they actually pin, so the suite does not quietly claim a path it never exercises. Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
81c44574d0 |
fix(mariadb): scope the Secret guards to the secrets section
The awk that was supposed to read secrets.include reset on any two-space key and then re-entered on the next four-space include:, so it swept services.include as well. It passes today only because no Service happens to end in -tls — the exact case the comment promised it handled would have misfired on a Service name. The exclude reader had the same shape. Gate both on the secrets: section header instead of on indentation alone. Verified in both directions: a Service named -tls no longer trips the guard, and a Secret named -tls under secrets.include still does. Also correct which CA the backup client verifies against: the bundle carries the chart's CA too under the cert-manager issuer, which is what makes the unconditional --ssl-ca right in both configurations. Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
1018c874c9 |
docs(mariadb): retire the tri-state vocabulary and qualify the backup claim
Renaming the field left the suite describing the design it replaced: titles about keeping TLS off, comments about a tri-state falling through to external inheritance, one case named for a boolean value the enum no longer has. The suite is where the contract is read, so it was still teaching the old one. The migration note also claimed the backup client's new verification is a live change to running jobs. It is not: that CronJob reads the root password from a Secret name the chart does not render, so it cannot start in the first place. Say that, and point at the supported backup flow instead of implying the deprecated one works. Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
a0d506eeff |
docs(mariadb): name the other charts' field correctly in the issuer note
The note explaining why this chart spells the field `tls.issuer` called the other charts' switch `tls.issuer` as well, which inverted it: it read as though those charts use this same key to mean on/off. They spell it `tls.enabled`. Restore that name and hedge the claim to "some other charts". The key is not uniformly a switch elsewhere either — one chart uses it to select whether a hostname is added to a certificate its engine serves regardless — so an unqualified "other charts" overstates how settled the meaning is. Link the epic tracking the convergence of the TLS surface, so the divergence the note describes points somewhere instead of only being asserted. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
5a9c3dd79f |
feat(mariadb): name the TLS field for the issuer instead of for on/off
tls.enabled read as a switch and was not one — TLS is served under every value, and what the field actually chose was which authority issues the server certificate. The same key means on/off in other charts here, so a tenant carrying that habit across would have read this exactly backwards. Replace it with tls.issuer: operator | cert-manager, defaulting to operator. The value now states the behaviour instead of needing prose to undo the name: operator is what runs today, so the fact that nothing changes on upgrade is visible in the default rather than argued in a comment. Being an enum, a typo is rejected by the schema, which a bool could not do. The explanation the old name required is gone with it — the helper docblock and the CR comment lose the paragraphs that existed only to say this was not a switch, and the README states the split in a table. Not marked breaking: the field is introduced by this series and does not exist on the target branch, so there is no released spelling to migrate from. Breaking is measured against what is merged, not against the intermediate history of the branch. Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
e164050cc1 |
docs(mariadb): state what the guards and the docs actually cover
Four claims of mine were broader than the thing they described. The bats header said it pins the invariants deciding whether a private key can reach a tenant. Three of its tests touch the declaration that the API does not accept yet, so they pin an intended spelling and nothing more; if the field lands under another name they stay green while the feature does nothing. Say which tests are live and which are placeholders. The certificate comment said the operator skips reconciliation entirely. It skips issuance, which is why SANs go unvalidated, but it still reads the Secret for the pod annotation hash and status. The README mapped the client's verification default onto the version enum, which selects the server image. The default belongs to the client library, so an 11.4 client verifies against a 10.6 server. The backup guard is described as catching a missing or unusable bundle. On a fresh managed-TLS install the bundle can be present and non-empty while holding only the operator's client CA, and the failure lands at verification instead. Note the window, and note in the README that a new instance stays down until cert-manager issues. Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
2e4ed43b88 |
docs(mariadb): shorten the TLS field descriptions to fit where they render
These strings flow into the ApplicationDefinition schema and become the help text beside the fields in the console. They had grown into paragraphs of rationale — 704 and 613 characters against 48 for the neighbouring external — which is the wrong length for a form label and duplicated what the README already explains at the right length. Say what the field does and that it is opt-in; the reasoning stays in the README, which is where a reader goes for it. Signed-off-by: Aleksei Sviridkin <f@lex.la> |