mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-22 04:33:36 +09:00
e89c39cb962dde735b423456c7d0dc3afd017b95
6099
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
e89c39cb96 |
test(cozystack-api): cover the annotation-to-ReleaseConfig wiring
Complete() reads release.cozystack.io/* annotations into ReleaseConfig once per ApplicationDefinition, but nothing exercised the assignments that join them. A mutation dropping release.HelmServerSideApply = serverSideApply left the suite green while shipping a silent nil: every Postgres HelmRelease would keep helm-controller's server-side default, Helm would force-own the rendered spec.instances seed, and the next reconcile would revert KEDA's live /scale count - the highest-blast-radius line in the feature, untested. Extract the per-CRD assembly into buildResourceFromCRD so the wiring is unit-testable without a live discovery client, and add TestBuildResourceFromCRD covering the apply-strategy annotation (present -> false, absent -> nil), the disable-wait sibling on the same loop, and the healthCheckExprs pass-through. Reverting either the SSA or the healthCheckExprs assignment now turns the test red. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
e54d728087 |
fix(postgres): stop an unready ScaledObject from wedging the release
A Postgres app with autoscaling.enabled but no resolvable metrics endpoint (a monitoring: true tenant who did not point serverAddress at a vmselect carrying kube_pod_labels) renders a ScaledObject that stays Ready=False reason=TriggerError forever. helm-controller's default wait gates the whole HelmRelease on every rendered resource's kstatus, so that one unready ScaledObject drives the release to UpgradeFailed and it retries forever - wedging an otherwise-healthy database and blocking every later chart change to it, while the tenant cannot even read the ScaledObject to see why (observed on dev3). Autoscaling health is a degraded-but-alive state surfaced by the DatabaseAutoscaler* alerts, not a release-readiness gate. Add a spec.release.healthCheckExprs entry on the postgres ApplicationDefinition that treats the ScaledObject as always-ready for release readiness (current: "true"); the CNPG Cluster and every other resource keep their built-in kstatus wait, so the database's own health still gates the release. Setting any healthCheckExprs couples WaitStrategy to poller (config.ResolveWaitStrategy); that the poller wait stays equivalent to the default for the kstatus-checked Cluster is verified on cozystack-pr-test, not here (render-blind). The cozyrd unittest pins the expr so a regeneration cannot drop it. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
3a4bade677 |
docs(postgres): document the kind-wide scope of the client-side apply flip
The release.cozystack.io/helm-server-side-apply annotation moves every Postgres HelmRelease to client-side apply, including releases that never enable autoscaling. IvanHunters flagged that this platform-wide side effect was unexplained and that non-autoscaled releases silently lose SSA drift re-assertion (client-side diffs previous-vs-new render only, never live state; driftDetection is configured nowhere in the repo). Record why the scope is deliberate rather than narrowing it. helm-controller v1.5.0 resolves an unset Upgrade.ServerSideApply through "auto" from the last release's apply method, so gating client-side on "autoscaling active" would force an SSA->client-side ownership handoff on the live spec.instances the moment a tenant enables autoscaling - the exact live-field handoff the seed design was chosen to avoid. Keeping the apply method stable for the kind's whole lifecycle keeps enable/dry-run/transition/disable clean two-way merges with no apply-method transition. The drift-re-assertion cost on never-autoscaled releases is called out as an accepted trade-off, with a per-instance apply strategy as the follow-up if it is ever needed. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
678d22f209 |
docs(postgres): move the serverAddress namespace-pin note out of a template comment
Closes IvanHunters' MINOR on scaledobject.yaml:97. The API-owner-gate marker was a Go template comment, stripped at render, so the ownership question and the pin's residual surface were recorded where nobody reads them again. State the constraint on the serverAddress field doc (propagated into values.schema.json, README and the cozyrds openAPISchema) including the accepted residual: tenant-root is always in the allow list because the derived default points there, so a tenant can still aim the operator at a root Service as a blind in-cluster probe (no response body reaches the tenant), which the egress NetworkPolicy does not narrow. The API-owner sign-off is tracked on the PR, not in a stripped comment. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
1192922a88 |
docs(keda): document the teardown order for the ScaledObject finalizer
Closes IvanHunters' MINOR on keda/values.yaml:19. The resource-policy: keep comment reasons about an admin removing the keda package but covers only the CRDs, not the finalizer.keda.sh KEDA stamps on every ScaledObject: with the operator gone a tenant deleting their Postgres app gets a ScaledObject stuck Terminating and a HelmRelease uninstall that hangs, recoverable only by an admin finalizer patch. Note that autoscaling must be disabled on every tenant Postgres before the package is removed, so the operator clears the finalizers while it is still up. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
07f1daa22d |
docs(postgres): correct the replicas "restored on disable" claim
Closes IvanHunters' MINOR on values.yaml:58. The replicas description (propagated into values.schema.json, README and the cozyrds openAPISchema) said the count is "restored on disable". Under client-side apply it is not: the two-way merge writes instances only when the rendered value changes, so a cluster KEDA grew past replicas stays there after autoscaling.enabled: false with no autoscaler left to reduce it — the sibling dryRun doc already states the true rule, so the two contradicted each other. Reword both the @param and the @typedef to say the live count is not auto-restored and that the operator lowers replicas to reclaim the grown capacity; regenerated the derived artifacts. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
7e664d269b |
test(postgres-rd): pin the client-side apply annotation on the cozyrds AppDefinition
Closes IvanHunters' MAJOR on cozyrds/postgres.yaml:12 (coverage half). Dropping the release.cozystack.io/helm-server-side-apply annotation left the whole suite green: the Go builder test pins the plumbing that consumes it but nothing pinned the annotation at the point it is set, so the single load-bearing invariant of the feature — client-side apply, without which server-side apply reverts KEDA's live /scale on every reconcile — was unguarded. Add a helm-unittest that renders the ApplicationDefinition and asserts the annotation, plus a test target so hack/helm-unit-tests.sh runs it; removing the annotation now goes red. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
568ccf4835 |
fix(keda): gate the operator CiliumNetworkPolicy on the cilium.io/v2 API
Closes IvanHunters' MAJOR on keda/values.yaml:60. The shim hardcodes networkPolicy.flavor: cilium and the vendored templates gate only on networkPolicy.enabled + flavor==cilium, never on the cilium.io/v2 CRD being served. keda is offered on every variant from the shared system bundle, so on a variant that ships no Cilium the CiliumNetworkPolicy is unmappable and the whole HelmRelease fails to install. Add a .Capabilities.APIVersions.Has "cilium.io/v2" check to the three CiliumNetworkPolicy templates so the package installs on any CNI; where Cilium is not served the egress backstop is simply absent and the chart's serverAddress namespace-pin remains the control. The vendored edit is captured as patches/ so make update re-applies it instead of silently reverting. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
3a09c3e9dc |
fix(postgres): guard dry-run/transition against the paused HPA min-clamp
Closes IvanHunters' MAJOR on scaledobject.yaml:39. The render guard rejected replicas > maxReplicas but not the symmetric low side. Under dry-run/transition the chart renders the static instances=replicas while the ScaledObject carries minReplicaCount=effectiveMin with both scale directions paused. The pause disables the HPA's scaling policies, not its min-replica clamp, so when replicas < effectiveMin the HPA raises the live count to the floor on its first reconcile — a scale-up in a mode documented as "actuates neither way" — and because the rendered instances value never changes, the client-side two-way patch never pulls it back, leaving permanent extra standbys and PVCs. Fail closed and tell the operator to stage replicas to the floor first. Under dry-run/transition this floor guard is strictly broader than the db.yaml quorum wedge (any replicas <= maxSyncReplicas is also < effectiveMin), so it supersedes it there; the wedge stays exercised on the disabled path. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
ccf6165645 |
fix(keda): scope autoscaler alerts per-cluster and fix ScaleBlocked false-fire
ScaleStuck and ScaleBlocked joined on namespace alone, assuming one autoscaled database per namespace — but a tenant may run several Postgres apps in one namespace. A Pending pod of one CNPG cluster then falsely tripped a healthy sibling's ScaleStuck, and ScaleBlocked's namespace-wide pod count masked a real block on one cluster. Derive a `cluster` label from the keda-hpa-<cluster> HPA name with label_replace and join the pod terms on (namespace, cluster). ScaleBlocked also cried wolf on a healthy multi-step scale-up of a single large database: CNPG materialises replicas one basebackup at a time (each often >15m), so the instance count legitimately lags a desired jump of +2 with no Pending pod, and the old rule fired with the actively-harmful "lower maxReplicas" advice. Count only instance pods (a transient join Job pod otherwise inflates the count and masks a block) and suppress via `unless` while a join Job is provisioning a replica; a genuine admission block creates neither. Make the desired > current comparison an explicit on(namespace,horizontalpodautoscaler) join. Note in the egress NetworkPolicy values that its `cluster` entity does not restrict cross-namespace egress (the chart's namespace-pin does) and that off-cluster / link-local IMDS denial depends on CNI enforcement. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
3516bbf314 |
docs(postgres): correct lag-brake floor and floor-raise shed comments
The lag-brake frozen summand's `or vector(0)` comment claimed it prevents a silent pin-to-floor when the pod-label count is momentarily empty under load. That state is unreachable: the frozen count selects the same kube_pod_labels series the base load term joins on, so base and frozen are empty or non-empty together and the floor never changes the result. State that it is a no-op marker. The enablement note framed raising the quorum floor / minReplicas as purely upward, but the merge rebases to the newly-rendered seed vs the previous seed, never to KEDA's live count — so raising the floor (or lowering replicas) to a seed still below a load-scaled live count sheds the surplus replicas and PVCs in one step before the HPA re-clones. Add the HAZARD note. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
187f18a167 |
fix(postgres): harden autoscaler serverAddress guard and cap maxReplicas
The serverAddress SSRF guard had two holes beyond the external-domain suffix already closed. It extracted the host by truncating at the first `:` or `/`, which a `http://<svc>.<ns>.svc:port@evil.example.com/` userinfo payload defeats: the truncated text passes the `.svc` check while net/url dials the host after the `@`. And it validated only the host shape, never the namespace, so a tenant could point the cluster-privileged KEDA operator at a sibling tenant's or a system namespace's Service (the egress NetworkPolicy's `cluster` entity does not stop in-cluster egress). Parse the URL authority properly (scheme, path/query/fragment, userinfo to the last `@`, then port) and pin the host's namespace to the tenant's own subtree via cozy-lib.rbac.allParentTenantsAndThis. The namespace pin narrows what a tenant may set and needs API-owner sign-off (noted at the guard). Separately, nothing bounded maxReplicas: the schema carries no maximum and the aggregated apiserver does not range-validate the CR, so a tenant could set maxReplicas: 100000 and drive KEDA to scale CNPG toward that count. Add a render guard rejecting a maxReplicas above a sane read-replica ceiling. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
d06354993e |
docs(postgres): correct the WorkloadMonitor replicas rationale
The comment claimed setting replicas to effectiveMin prevents a false under-provisioning alarm when KEDA idles the cluster to its floor. But WorkloadMonitorReconciler gates Status.Operational solely on availableReplicas < minReplicas and never reads spec.replicas, so with minReplicas: 1 the field is inert for health today. Keep the value (it is the honest desired-floor target per the field doc) but state what actually drives the alert, so the rationale is not read as describing behaviour the controller does not have. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
2bf37f8260 |
fix(postgres): anchor autoscaler SSRF guard to the cluster domain
The serverAddress in-cluster check accepted an unanchored trailing domain (`\.svc(\.[a-zA-Z0-9.-]+)?$`), so a crafted host like x.svc.attacker.com — a `.svc` label followed by any external domain — passed the guard. A tenant could then point the cluster-privileged KEDA operator off-cluster (the egress NetworkPolicy is the only remaining barrier, and it is inert where the CNI does not enforce CiliumNetworkPolicy). Anchor the optional suffix to this cluster's domain (from _cluster.cluster-domain, injected by the platform from networking.clusterDomain): a legitimate vmselect is <service>.<namespace>.svc or ...svc.<clusterDomain>, nothing else. The prior tests only exercised non-.svc hosts and gave false confidence; add a case for the .svc.<external> bypass. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
7cdfe7a197 |
feat(keda): add the quota-blocked alert and restrict operator egress
Close two follow-ups the design proposal names for this PR: - DatabaseAutoscalerScaleBlocked: the Pending-independent companion to DatabaseAutoscalerScaleStuck. A ResourceQuota (or any admission webhook) refuses the new standby pod at admission, so no pod object is created and ScaleStuck's Pending intersection misses it. Key instead on desired_replicas > count(CNPG pods in the namespace): a healthy basebackup's pod exists (Running) so the count already equals desired, while an un-admitted pod leaves the count short. Correct the stale ScaleStuck comment that claimed it covered the quota case. - Enable the vendored KEDA egress NetworkPolicy (flavor cilium): egress is toEntities [kube-apiserver, cluster] only — apiserver and in-cluster endpoints (CoreDNS, vmselect, CNPG) stay reachable, world (off-cluster) is denied — so the operator cannot be used as an SSRF relay off-cluster. The kube-apiserver entity covers the control plane however it is addressed, so it does not starve the operator's apiserver access the way a raw-CIDR policy could. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
6083a9fa1d |
feat(postgres): guard replicas>maxReplicas and pin serverAddress to in-cluster
Close two render-guard follow-ups the design proposal names for this PR: - Fail closed when replicas exceeds autoscaling.maxReplicas. The stock HPA's currentReplicas > maxReplicaCount branch writes /scale = maxReplicaCount unconditionally on the first reconcile — even under the both-direction pause of dryRun/transition — so a resting/seed count above the ceiling would shed the surplus standbys and their PVCs in one step, past the scale-down pacing. Mirrors the existing effectiveMin > maxReplicas guard. - Refuse an autoscaling.serverAddress whose host is not an in-cluster Service DNS name (…svc[.<domain>]). The cluster-privileged KEDA operator fetches it every poll, so an external/IP address is an SSRF surface (cloud metadata endpoint, another tenant); the query body is already namespace-scoped, this restricts the destination. Defense-in-depth pairs with the operator egress policy (keda package). Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
dde7c5099b |
docs(postgres): correct the apply-strategy annotation comment to "two-way"
The ApplicationDefinition annotation comment called client-side apply a "three-way-merge", contradicting the corrected wording everywhere else in the change (config.go, _autoscaling.tpl, db.yaml, values.yaml all say the client-side patch diffs the previous vs new render and does not consult live state). For a CRD, helm-controller v1.5.0 leaves threeWayMergeForUnstructured at its default, so Helm takes the two-way jsonpatch.CreateMergePatch(original, modified) path. This comment ships on the live ApplicationDefinition, so a maintainer reading it in isolation would otherwise get the wrong mental model of the safety property. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
daf733ad9c |
fix(keda): narrow autoscaler RBAC and scope the operator-down alert
Restrict the KEDA operator ClusterRole: set permissions.operator.restrict.secret so it drops cluster-wide secret list/watch, and pin rbac.scaledRefKinds to postgresql.cnpg.io/clusters so it no longer holds a wildcard */scale across every group. Gate DatabaseAutoscalerKedaOperatorDown on the presence of keda-hpa-* HPA series so a tenant running its own Monitoring app (which never scrapes keda_build_info, root-only) does not fire it forever about a healthy platform component. Make the two negative containsDocument matchers in the bundles-default test non-vacuous (any: true) so "KEDA is not installed by default" is actually asserted. Drop the unused pollingInterval/cooldownPeriod params from the cozy-lib keda helper. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
c693ed36fb |
fix(postgres): force client-side apply so the autoscaler seed survives
Under active read-replica autoscaling the chart seeds spec.instances at a constant max(replicas, effectiveMin) rather than omitting it (an omitted field defaults to 1, which the CNPG quorum webhook rejects, and — under server-side apply — the phase-2 omit prunes the field whenever KEDA has not yet written /scale, collapsing an idle-at-floor cluster to 1 and shedding its replica PVCs). Keeping the constant only works if Flux does not force-own and revert it. The postgres ApplicationDefinition therefore carries release.cozystack.io/helm-server-side-apply: "false", so the tenant HelmRelease applies client-side: helm-controller patches the CNPG Cluster from the previous-vs-new rendered manifest without consulting live state, the unchanged constant yields no patch, and KEDA's higher live /scale count is preserved. The constant also equals the disabled/dryRun/transition render (instances: replicas) whenever replicas sits at/above the floor, so those flips are patch-free too. Correct the merge-semantics comments across db.yaml, _autoscaling.tpl and values.yaml to describe the two-way previous-vs-new patch precisely (not a "three-way no-op"), and document the residual floor-change hazard: editing replicas / minReplicas / quorum.maxSyncReplicas moves the seed, so that one reconcile does patch spec.instances to the new seed. Add autoscaling.serverAddress so a monitoring:true tenant can point the ScaledObject at a vmselect that carries both the CNPG metrics and kube_pod_labels. Regenerated README/schema/types/cozyrds. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
27d7dbb140 |
feat(api): add per-app helm apply-strategy override annotation
Add release.cozystack.io/helm-server-side-apply on the ApplicationDefinition, threaded through ReleaseConfig into every emitted HelmRelease's Install/Upgrade.ServerSideApply. Empty leaves the helm-controller default (server-side apply on v1.5.0); "false"/"disabled" forces client-side apply. The postgres read-replica autoscaler needs this: KEDA's HPA writes the CNPG Cluster's /scale subresource (spec.instances) while the chart renders a constant seed. Under helm-controller's default server-side apply the rendered field is force-owned and reverts KEDA's live value on every reconcile. Under client-side apply helm-controller patches the CRD from the previous-vs-new rendered manifest without consulting live state, so the unchanged constant yields no patch and the /scale value survives (verified against Helm v4 createPatch: unstructured/CRD objects take the two-way jsonpatch.CreateMergePatch(original, modified) path unless threeWayMergeForUnstructured is set, which helm-controller v1.5.0 leaves at its default). Guarded by TestConvertApplicationToHelmRelease_ServerSideApplyOverride on the tenant-release builder (convertApplicationToHelmRelease), and by TestParseHelmServerSideApplyAnnotation. Removes a stale package_reconciler_test guard that asserted a false invariant on the wrong (platform Package) builder. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
826f08972f |
fix(apps/postgres): make autoscaler fail-safe, dip-free, and observable
Address review findings on the KEDA read-replica autoscaler where the
implementation diverged from the merged design proposal.
Fail-safe read load. Gate the `or vector(0)` floor on the existence of ANY CNPG
instance pod (`count(kube_pod_labels{...instance_role=~".+"}) > 0`) — primary or
replica — so the query is empty ONLY when no CNPG pod labels exist at all (a
broken kube-state-metrics pipeline), not during initial provisioning when the
sole replica-side pod is CNPG's join Job (jobRole, not instanceRole). Also pin
`ignoreNullValues: "false"` on the trigger: KEDA's prometheus scaler defaults it
true, turning an EMPTY result into metric 0 (not an error) — the HPA would then
desire 0 and clamp to the floor, i.e. silently scale a broken-pipeline cluster
DOWN. With it false an empty result is a scaler error, so the HPA HOLDS and
DatabaseAutoscalerScalerErrors fires; gating on any instance role keeps a healthy
cluster (including a long basebackup) from tripping that alert.
Constant seed = max(replicas, effectiveMin), so enable/dryRun/transition flips
are a merge no-op with no dip and never undercut the quorum floor.
Fail closed when the quorum floor exceeds maxReplicas (was: effectiveMax silently
raised both bounds, unbounded for a large maxSyncReplicas).
Dry-run / transition keep the HPA (unidirectional pause pair, not the HPA-deleting
full pause) but both SelectPolicy=Disabled, so desired normalizes to current — the
recommendation is read from the read-load metric, not a diverging desired count.
Observability. DatabaseAutoscalerAtMax keys on desired >= maxReplicas, NOT the
ScalingLimited condition (which is also true at the floor, so it fired on every
idle cluster with wrong "raise maxReplicas" advice). Dashboard read-load panels
are per-cluster (label_cnpg_io_cluster) with a CPU panel for ReadCPUUtilization
and the exact clamp in descriptions; ScaleStuck's Pending intersection is scoped
to CNPG pods; alerts.yaml narrative comments moved to Go-template comments so they
do not ship into the PrometheusRule.
Guard the seeding invariant: TestBuildHelmReleaseSpecNoForceOrDriftDetection.
Regenerated README/schema/types.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
|
||
|
|
8769232997 |
docs(apps/postgres): correct autoscaling merge-safety notes
The disable case (3) note claimed disabling snaps instances back to .Values.replicas, but the client-side three-way merge compares the newly rendered value against the previously applied one, not the live count. When replicas == effectiveMin (the default) it is a no-op and KEDA's live count is preserved (silent over-provisioning), so it does not snap back. Reword case (3) and the "differs from the live one" framing to match the actual merge semantics. Re-point the no-Force/no-SSA invariant citation from internal/controller/applicationdefinition_helmreconciler.go (which holds no apply-strategy code) to internal/operator/package_reconciler.go buildHelmReleaseSpec, where the HelmRelease spec is actually built. Comment/doc only; helm unittest 52/52 unchanged. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
b40cf82642 |
fix(postgres): seed instances on both paths, add input guards, and harden alerts
Address IvanHunters' deep review round on #3954: - [MAJOR] Enabling autoscaling directly on an existing non-quorum database no longer collapses it to 1 instance and deletes every replica PVC. The chart now SEEDS spec.instances at the quorum floor (effectiveMin) on BOTH the quorum and non-quorum active paths instead of omitting the field in the non-quorum case; the constant seed is a three-way-merge no-op so KEDA still owns the runtime count via /scale, and the enable dip is capped at the floor (>=2) rather than collapsing to 1. Documents the client-side-3-way-merge (not SSA) assumption that the no-op relies on. - [MINOR] Fail closed on autoscaling.minReplicas > autoscaling.maxReplicas instead of silently resolving the ceiling upward (which defeated maxReplicas as a cost knob). The stray maxReplicas: 0 case now rejects rather than clamps. - [MINOR] The disable-wedge guard, its values note and its test no longer emit the invalid `kubectl scale --subresource=scale` (scale has no --subresource); use `kubectl scale cluster.postgresql.cnpg.io/<name> --replicas=N`. - [MINOR] Correct the enablement note: raising minReplicas/maxSyncReplicas on a live cluster is NOT harmless when KEDA's live count exceeds the new floor (it decommissions surplus replicas + PVCs); prescribe pin-first for raises too. - [MINOR] Add DatabaseAutoscalerKedaOperatorDown (absent(keda_build_info)): when the operator is unhealthy a ScaledObject never becomes an HPA and no other alert can fire. Intersect DatabaseAutoscalerScaleStuck with a Pending pod so a healthy long basebackup (Running pod, desired>current >15m) no longer false-fires. - [MINOR] Keep the KEDA CRDs on package uninstall (crds.additionalAnnotations resource-policy: keep via the shim) so removing the keda package doesn't cascade-delete every tenant ScaledObject. - [MINOR] cozy-lib helper: reject a non-numeric bound (int "two" -> 0 -> a scale-to-zero ScaledObject) and render an explicit pollingInterval/cooldownPeriod of 0 (`if not kindIs invalid`, not `with`). - [MINOR] Note the read-load fail-open: a broken metrics join reads as idle and silently pins autoscaling at the floor under real load. New/updated tests cover the non-quorum seed, the min>max guard, the corrected kubectl message, the operator-down + Pending-intersected alerts, the CRD keep policy, and the non-numeric-bound helper guard. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
1ebc021491 |
style(postgres): strip the autoscaling db.yaml comments from the rendered manifest
The autoscaling instance-count explanations were plain `#` YAML comments, so they
shipped verbatim into every rendered CNPG Cluster manifest on every customer
cluster (unlike the {{/* */}} comments elsewhere in the chart). Convert them to
Go-template comments so the rationale stays in-source but is stripped at render,
matching the treatment already applied to the apps.yaml clusterDomain comment.
Also document (as a stripped comment) that WorkloadMonitor.spec.replicas tracks
the quorum floor, not KEDA's live autoscaled target, under active autoscaling.
Addresses the deep cozy-review notes on #3954.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
|
||
|
|
5a5047ac23 |
fix(dashboards): bind the autoscaling dashboard to a datasource variable
The database-autoscaling dashboard hardcoded "datasource": "VictoriaMetrics" on
its namespace template variable, but cozystack provisions VictoriaMetrics
datasources as vm-shortterm / vm-longterm — there is no datasource named
"VictoriaMetrics", so on import the variable query failed ("datasource not
found"), the namespace variable never populated, and every panel (which filters
on namespace=~"$namespace") showed no data. Switch to the repo-standard pattern
used by db/cloudnativepg and the other dashboards: a `datasource`-typed template
variable (ds_prometheus) referenced as { type: prometheus, uid: ${ds_prometheus} }
from the namespace variable and every panel/target (the panels previously carried
no datasource at all and only worked by falling back to the default).
Also reframe the disable-wedge guard message to lead with the invariant (replicas
must exceed quorum.maxSyncReplicas) so a pure-quorum user who never enabled
autoscaling isn't told to "disable autoscaling"; the disable path is kept as a
parenthetical cause.
Addresses the adversarial branch-review findings on #3954.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
|
||
|
|
6e3da63586 |
fix(postgres): guard the disable-wedge, role-scope the freeze count, and harden edges
Address the remaining IvanHunters review points on #3954: - [MAJOR] Disabling autoscaling wedged the HelmRelease under synchronous quorum. An autoscaled + maxSyncReplicas>=1 cluster runs with instances seeded from effectiveMin while replicas stays at its default, so flipping autoscaling off rendered instances: <replicas> next to the quorum and CNPG's webhook rejected it ("maxSyncReplicas must be lower than the number of instances"). Add a symmetric render-time guard on the static-count path (covers off/dryRun/ transition): fail with an actionable message when replicas <= maxSyncReplicas. - [MINOR] Role-scope the freeze brake's currentInstances count (label_cnpg_io_instance_role=~".+") so lingering CNPG join/bootstrap Job pods (cnpg.io/cluster but jobRole, not instanceRole) don't inflate it and nudge desired up while the brake is meant to hold it. - [MINOR] Default maxReplicationLagSeconds to 0 (brake opt-in): only the non-braking path is validated live, so ship the validated default and let the freeze be enabled explicitly until its branch is exercised under load. - [MINOR] Document the floor-change and non-quorum-disable dips (same three-way- merge hazard) and the pin-replicas-first mitigation. - [NIT] Coerce the cozy-lib helper's `paused` (eq "true") so a caller passing the string "false" does not pause the ScaledObject. - [NIT] Trim the keda.clusterDomain injection comment so it no longer ships ~8 verbatim lines into the cozystack-values Secret on every cluster. Regenerate types/schema/README/postgres-rd. New tests cover the disable-wedge guard, the role-scoped freeze count, and the string-"false" paused coercion. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
e8609ac00d |
fix(keda): ship the autoscaling dashboard via grafana-operator, not a dead ConfigMap
The database-autoscaling dashboard shipped from the keda package as a ConfigMap labeled grafana_dashboard: "1" for a Grafana sidecar that cozystack does not run (no k8s-sidecar anywhere; every dashboard is provisioned by grafana-operator from the grafana-dashboards image via GrafanaDashboard CRs). It was enabled by default and never appeared in any Grafana. Move the JSON to dashboards/db/database-autoscaling.json and register db/database-autoscaling in packages/system/monitoring/dashboards.list, exactly how db/cloudnativepg and the other managed-db dashboards ship, and drop the keda ConfigMap, its default-on toggle, and its wrong-mechanism test. Addresses IvanHunters [MAJOR] review feedback on #3954. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
70e4087432 |
fix(postgres): guard empty monitoring namespace; clarify alert scope and helper doc
Address the branch-review nits: - Fail closed when the monitoring namespace is unset. serverAddress builds vmselect-shortterm.<ns-monitoring>.svc, and an empty ns-monitoring rendered the malformed vmselect-shortterm..svc:8481, surfacing late as a KEDA scaler error. The platform always injects it, but a hand-rendered/misconfigured tenant now gets an actionable render-time failure mirroring the KEDA-CRD and target guards. Tested. - Document that the database-autoscaling PrometheusRule matches every KEDA HPA / ScaledObject error (the underlying series carry no owning-app label), which is correct only while database autoscaling is KEDA's sole consumer in cozystack; note to scope the rules if a non-database ScaledObject is ever added. - Correct the cozy-lib.keda.scaledObject usage example: `paused` should reflect dry-run OR migration transition, not dry-run alone, so a second engine copying the example does not drop the transition pause. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
9cf4a171f0 |
fix(postgres): stop coercing numeric-zero autoscaling bounds and floor each lag gate
Two review notes from IvanHunters: - effectiveMin/effectiveMax wrapped minReplicas/maxReplicas/maxSyncReplicas/ maxReplicationLagSeconds in Sprig `default`, which treats a numeric 0 as empty and silently substitutes the fallback (e.g. maxReplicas: 0 became 6, discarding the tenant's value). values.yaml/schema already default these when omitted, so read them directly; an explicit out-of-range value now flows into max() and is clamped to the quorum floor rather than swapped for a different number. Add a test that maxReplicas: 0 clamps to effectiveMin instead of becoming 6. - The replication-lag brake only floored the product $braking with `or vector(0)`, so if a CNPG build stops emitting cnpg_pg_replication_lag or cnpg_collector_wal_records the individual gate resolved to an empty vector. Floor each gate ($lagHigh, $writing) individually so the fail-open decision is explicit and per-gate: the brake engages only on positively-observed lag AND writing, and a missing signal degrades to "scale normally" (safer than freezing on absent data) rather than relying on the emptiness of the product. Assert both gates carry the floor. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
24fe1f802a |
test(cozy-lib): cover the non-map scaleTargetRef guard
Close the last uncovered rejection path of cozy-lib.keda.scaledObject: a non-map scaleTargetRef now has a failedTemplate assertion, so all twelve fail-closed guards are executable-documented. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
a92cb5b5f8 |
test(cozy-lib): cover the min/max required guards of keda.scaledObject
Complete the helper's rejection-path coverage: add the two remaining fail-closed assertions (minReplicaCount / maxReplicaCount required) so all eleven guards are executable-documented. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
2f294c65cd |
test(cozy-lib): cover every fail-closed guard of the keda.scaledObject helper
The helper validates ~10 required inputs but the suite only exercised three rejection paths (non-dict arg, missing scaleTargetRef.name, inverted bounds). Add a parameterised fixture that builds a fully-valid argument dict and drops exactly one required field per case, and assert the fail message for name, scaleTargetRef.kind, scaleTargetRef.apiVersion, query, threshold and serverAddress. The helper's rejection contract is now executable-documented in full rather than sampled. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
42a533fc50 |
fix(platform): inject networking.clusterDomain into the KEDA package
KEDA's clusterDomain is a vendored-subchart value baked into the metrics-apiserver gRPC target (keda-operator.<ns>.svc.<clusterDomain>:9666) and the operator's serving-cert SANs at render time. A chart template cannot derive a subchart value from _cluster, so the keda package hardcoded cozy.local — which silently breaks (NXDOMAIN, every DB HPA FailedGetExternalMetric) on any cluster whose domain is not the default. Inject the authoritative networking.clusterDomain into the shared cozystack-values Secret under the keda subchart key so the value tracks the platform config; the keda package keeps cozy.local as a standalone fallback. The key is ignored by packages without a keda subchart (no app/extra schema sets root additionalProperties:false). Add platform unittests asserting the Secret carries keda.clusterDomain and that it tracks a non-default networking.clusterDomain. Addresses the hardcoded-clusterDomain portability gap found in branch review. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
a7b5077da2 |
fix(postgres): floor the lag-brake frozen term so a KSM gap can't empty the query
The read-load query's base term is floored with `or vector(0)`, but the replication-lag brake's frozen additive term (count(kube_pod_labels) * target) was not. PromQL `+` is a set intersection, so if count(kube_pod_labels) is momentarily empty — a kube-state-metrics scrape gap or KSM rollout — the frozen term is empty and the whole `base + frozen*braking` sum collapses to empty, defeating the base term's own graceful degradation. KEDA's prometheus scaler (ignoreNullValues: true by default) then reads that as 0 and pins desired to minReplicaCount, ignoring real read load exactly when metrics are flaky. This hits the default config (maxReplicationLagSeconds: 30, brake active). Floor the frozen summand with `or vector(0)` so base + 0 = base, and assert the floor in the lag-brake unittest. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
573213ef8b |
test(postgres): assert the standby-only read-load selector in the query
Cover the load-bearing replica selector that both read-load queries depend on: the join must restrict to standby pods via label_cnpg_io_instance_role="replica", the exact key kube-state-metrics exposes for CNPG's cnpg.io/instance_role. A wrong key (e.g. a camelCase label_cnpg_io_instanceRole) would match no series, floor the read term to 0, and silently pin desired to the minimum - the suite previously never asserted the selector at all. Verified live on dev9 (CNPG 1.30.0) that the snake_case key is the one present in kube_pod_labels and the camelCase form does not exist. Adds the assertion for both the ReadConnections and ReadCPUUtilization paths. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
842a59a957 |
fix(postgres): correct the migration-dip docs and pin the KEDA metricType
Two review findings: - The enablement note claimed the two-phase `transition` flow avoids the count dip when enabling autoscaling on an existing cluster. It does not: helm-controller upgrades with Helm's three-way merge (not SSA), which keys off the old-vs-new manifest diff, not KEDA's live /scale value. At phase 2 the chart stops rendering the staged static count, so the field is deleted (no quorum -> CNPG defaults to 1, a brief collapse) or changed (quorum -> rebased to effectiveMin), then the warm KEDA restores it. Rewrite the docs to state the flow reduces (KEDA is pre-warmed, no cold-start) but does not eliminate the dip, and spell out the suspend + scale-subresource pin that actually makes the handoff dip-free. This is the SSA-vs-three-way-merge handoff the design proposal left for the PoC to settle. - Pin metricType: AverageValue on the Prometheus trigger in cozy-lib.keda.scaledObject. The desired = ceil(query / threshold) identity holds only for an AverageValue external metric; it was relying on KEDA's default. Make it explicit and assert it so a re-vendor or default change cannot silently break the arithmetic. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
87b26d6595 |
fix(postgres): only seed spec.instances when synchronous quorum needs it
Narrow the previous unconditional seed to a conditional one, keeping design
proposal §3 (chart omits the field, HPA is the sole writer) intact for the
default path. Without synchronous quorum (maxSyncReplicas=0) CNPG's defaulted
instances=1 is admissible, so the field is omitted exactly as §3 specifies and
KEDA owns it via /scale with nothing contested. Only when maxSyncReplicas>=1 -
where an omitted/defaulted instances=1 fails the admission webhook
("maxSyncReplicas must be lower than the number of instances") - does the chart
seed the quorum floor (effectiveMin) so a fresh Cluster is admitted; the constant
re-render keeps three-way merge from reverting KEDA's higher live value.
This confines the deviation to the case §3 did not consider (sync quorum), so no
design-proposal amendment is needed for the documented path. Verified live on a
dev cluster: the quorum render (instances=2, maxSyncReplicas=1) and the
non-quorum render (instances omitted) are both admitted by the CNPG webhook. Adds
a render test for the non-quorum omit path alongside the quorum seed tests.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
|
||
|
|
610bcb5df9 |
fix(postgres): seed spec.instances at the quorum floor under autoscaling
Under active autoscaling the chart previously omitted .spec.instances entirely,
letting the KEDA HPA own it via /scale. But CNPG defaults an absent
.spec.instances to 1, and its admission webhook rejects a Cluster where
maxSyncReplicas >= instances ("maxSyncReplicas must be lower than the number of
instances"). So a fresh database created with autoscaling.enabled: true AND
synchronous quorum (maxSyncReplicas >= 1) was rejected at admission before KEDA
could ever scale it to effectiveMin (= maxSyncReplicas+1) — the whole tenant
HelmRelease failed. The default maxSyncReplicas: 0 path was unaffected, which is
why it went unnoticed. Verified live: a rendered Cluster with maxSyncReplicas: 1
and an omitted instances is rejected by the dev-cluster webhook; with the seeded
floor it is admitted.
Render instances: effectiveMin under active autoscaling instead of omitting it,
so a fresh Cluster starts at a valid, quorum-satisfying count. Because the chart
emits the same constant floor on every reconcile, Helm's three-way merge is a
no-op on the field and never reverts KEDA's higher live value, so the HPA still
owns the runtime count. Update the enablement docs (a new DB can enable directly
even with synchronous replication; existing DBs above the floor rebase to it
unless the two-phase transition flow is used) and add render tests for the fresh
create + maxSyncReplicas>=1 case.
Addresses the fresh-create-with-quorum admission gap found in branch review.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
|
||
|
|
284f5e5543 |
fix(postgres): fail closed when autoscaling is enabled without KEDA, guard target
autoscaling.enabled is a tenant-facing knob, but KEDA is an optional platform package that is not installed by default. A tenant enabling autoscaling on a cluster without KEDA would render a ScaledObject against an unknown kind and the whole database HelmRelease would fail with an opaque `no matches for kind "ScaledObject"`. Guard the ScaledObject template on `.Capabilities.APIVersions.Has "keda.sh/v1alpha1"` and stop with an actionable message instead. Skipping the render is deliberately NOT an option: db.yaml already drops the static .spec.instances under active autoscaling, so a missing ScaledObject would let CNPG collapse the cluster to 1 instance. Document the KEDA prerequisite on the `enabled` field (README/schema). Also reject a non-positive autoscaling.target at render time: it is a plain integer now, but the desired-replica formula divides the read load by it, so 0 is undefined and negatives invert scaling. cozyvalues-gen has no minimum directive, so the bound is a template guard. Add unittests for both guards (a dedicated suite renders without the KEDA CRD so the fail-closed path is exercised; the main suite now advertises keda.sh/v1alpha1 via capabilities). Addresses IvanHunters (KEDA-absent) and CodeRabbit (non-positive target) review feedback on #3954. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
ec123649bf |
fix(keda): use the KEDA 2.20 metric name in the scaler-errors alert
The DatabaseAutoscalerScalerErrors rule queried keda_scaler_errors_total, which KEDA 2.20 no longer exposes (verified against a live 2.20.2 operator: it emits keda_scaled_object_errors_total for per-ScaledObject errors and keda_scaler_detail_errors_total for per-scaler errors). The alert would have silently never fired. Switch to keda_scaled_object_errors_total and add a unittest assertion pinning the metric name so a rename regresses loudly. Addresses CodeRabbit review feedback on #3954. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
f8fc2d99bf |
fix(postgres): make autoscaling.target an integer and drop the dead root metric
Two review findings on the generated API: - autoscaling.target was a resource.Quantity, which accepts suffixed values like `150m`. The chart splices target straight into the PromQL query and the KEDA threshold, where `150m` means 150 minutes to PromQL and is not a usable KEDA float threshold. Retype it as a dimensionless integer so the schema rejects suffixed input - helm unittest's schema validation now enforces it, and the existing autoscaling suite exercises the integer contract. - ConfigSpec carried a root `metric` field (and spec.metric in the postgres-rd schema) that nothing reads - the chart consumes only autoscaling.metric - so setting it silently did nothing. Remove the dead root param, keeping the ReadMetric enum that autoscaling.metric references. Regenerate types.go, zz_generated.deepcopy.go, values.schema.json, README.md and the postgres-rd schema; update the autoscaling tests to the integer target. Addresses CodeRabbit review feedback on #3954. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
ec06dafd25 |
feat(postgres): fold the replication-lag brake into the autoscaling query
Wire autoscaling.maxReplicationLagSeconds into the read-load query: while the max standby lag has exceeded the threshold within the cooldown window and the primary is actively writing WAL, the query returns currentInstances*target, pinning desired to the current count and freezing scaling in both directions (safer than pinning maxReplicas, which would still allow scale-down under lag). The lag term uses max_over_time as a hysteresis band; the write gate (WAL-record rate > 0) keeps an idle primary from tripping it. Metrics verified live on CNPG 1.30.0: cnpg_pg_replication_lag and cnpg_collector_wal_records exist and carry namespace/pod (the proposal's cnpg_pg_stat_replication_sent_diff_bytes does not). Also floor the read load at zero via `or vector(0)` so an idle cluster returns target rather than an empty result. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
ac82ce186c |
fix(keda): pin cozy.local domain and raise operator API throttle
The vendored KEDA chart defaults clusterDomain to cluster.local, but cozystack runs on cozy.local. KEDA's metrics-apiserver dials the operator metrics gRPC service at keda-operator.<ns>.svc.<clusterDomain>:9666 and the operator mints that service's serving-cert SANs from the same domain, so the default resolves to NXDOMAIN and every HPA hangs on FailedGetExternalMetric. Pin keda.clusterDomain=cozy.local. Also raise the operator's client-go throttle to 50 QPS / 100 burst: at the upstream 20/30 a single-replica operator loses its leader lease under scale-subresource polling and crash-loops (the client rate limiter starves the lease renewal). Both were observed live on a dev cluster; guard them in the package unittest. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
d4c76fadb4 |
feat(postgres): two-phase autoscaling migration and read-CPU metric
Make enabling autoscaling on an existing database count-preserving, and add a second read-load signal: - autoscaling.transition (migration phase 1): keep .spec.instances rendered as a floor (from replicas, which the operator stages to the live count) AND pause the ScaledObject, so KEDA stands up without contending with Flux for the field. Flip transition off (phase 2) to drop the static field and hand the count to KEDA — suspend the HelmRelease across that apply, since helm-controller uses Helm's three-way merge (not SSA) and would otherwise race CNPG's default-to-1. The procedure is documented on the values field. - autoscaling.metric (ReadConnections | ReadCPUUtilization): the query helper emits either the active-connections sum or the postgres-container CPU sum, both + target. active = enabled && !dryRun && !transition; the ScaledObject is paused whenever dryRun or transition is set. Covered by helm unittest (phase-1 holds the staged count + pauses; CPU metric query) and regenerated schema/README/Go types/postgres-rd. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
237053191e |
feat(keda): ship database-autoscaling alerts and Grafana dashboard
Add cozystack-owned observability for managed-database read-replica autoscaling, rendered by the keda umbrella (not the vendored subchart): - PrometheusRule (templates/alerts.yaml), scoped to the KEDA-generated HPAs (keda-hpa-*): DatabaseAutoscalerScaleStuck (desired > current for 15m — stuck scale-up with no automatic rollback), DatabaseAutoscalerAtMax (ScalingLimited=true for 30m — at max or quorum-raised ceiling), and DatabaseAutoscalerScalerErrors (KEDA cannot evaluate the trigger — never scale blind). - A Grafana dashboard ConfigMap (grafana_dashboard sidecar label) with desired-vs-current replicas, active read connections, replication lag, and scaling-limited panels. Both gated by databaseAutoscaling.alerts.enabled / .dashboard.enabled (default true). Metric names are calibrated on a live cluster per the proposal's PoC; the rule and panel shapes are what ships. Covered by helm unittest (PrometheusRule with the three alerts + disable; dashboard ConfigMap label + disable). Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
7f8f2c1d86 |
feat(keda): add KEDA as an optional platform component
Vendor the upstream KEDA chart (kedacore/keda 2.20.2) as a new optional system package and wire it into the platform, so database read-replica autoscaling (which renders KEDA ScaledObjects) has a backend. - packages/system/keda: umbrella cozy-keda + vendored chart (Makefile pins KEDA_CHART_VERSION with a lockstep assert), shim values enabling the operator / metrics-server / webhook ServiceMonitors under the platform Prometheus. - core/platform: PackageSource cozystack.keda (namespace cozy-keda, dependsOn networking + prometheus-operator-crds; cert-manager is not needed, KEDA self-generates its webhook TLS) plus an optional bundle entry, opt-in via bundles.enabledPackages. It is not a dependsOn of any default package, so the platform HelmRelease never blocks on it. - KEDA claims the cluster-wide external.metrics.k8s.io APIService (distinct from metrics-server's metrics.k8s.io — no conflict). Covered by helm unittest: platform bundle enable/disable/opt-out, the PackageSource dependsOn, and the vendored chart's external-metrics APIService + shim-enabled ServiceMonitor. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
0335276b61 |
feat(postgres): render a KEDA ScaledObject for read-replica autoscaling
Implement the KEDA-based read-replica autoscaler for postgres per the merged design proposal (cozystack/community#44): - cozy-lib.keda.scaledObject: engine-agnostic helper rendering a KEDA ScaledObject (scale target, effective bounds, prometheus trigger, scale-down behavior, optional pause), fail-closed on bad input. - postgres: an `autoscaling` values block (enabled / minReplicas / maxReplicas / target / maxReplicationLagSeconds / dryRun). Under active autoscaling the chart drops the static .spec.instances so Flux and the KEDA-managed HPA no longer contend, and renders a ScaledObject whose minReplicaCount folds in the synchronous-quorum floor (max(minReplicas, maxSyncReplicas+1, 2)); when the floor exceeds maxReplicas, quorum wins. dry-run keeps the static count and pauses the ScaledObject. The WorkloadMonitor tracks the quorum floor instead of a stale static value. Covered by helm unittest (cozy-lib helper: valid render, pause, three fail-closed guards; postgres: instances dropped, quorum-floored bounds, quorum-over-max, dry-run pause). Regenerated values.schema.json, README, Go types and postgres-rd. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
3a369fed99 |
feat(redis): add opt-in TLS for Redis and Sentinel (#2729)
## What this PR does `tls.enabled` puts Redis and Sentinel on a TLS-only listener with a per-release cert-manager chain, and hands the tenant the CA certificate as `redis-<name>.tenant-ca`. Off unless set. redis-operator v3.3.5 has no TLS support. The work is ported upstream in freshworks-oss/redis-operator#85, which is open and not in a release, so this carries it as three patches over the v3.3.5 tarball. `images/redis-operator/patches/tls.diff` is the source half, cut from that PR's commit `2ad2ed8d1`, which the patch header pins; it is applied by the glob the image Dockerfile already runs, after the Dockerfile strips the tarball's `manifests`, `charts`, `example` and `.github`, so the image builds from source alone and the upstream tests that read the CRD YAML skip instead of failing against the stale v3.3.5 copy. `patches/crd-tls.diff` adds `spec.tls` to the vendored CRD, and with it the CEL rule that fixes `tls.enabled` at creation; without the schema block the apiserver prunes the field on admission and the patched binary waits for something it never sees, and the rule cannot exist without the block it references. `patches/rbac-tls.diff` grants Create, Get and Update on Secrets, so the operator can publish the CA-only Secret it addresses by name. That is the whole delta, and it is narrower than the ported operator's own chart asks for: those three are the only Secret calls in the patched source, and cert-manager Certificates get no grant at all, because the operator only touches one under `spec.tls.certManager` and this chart pins `certificateSecret` mode instead. One change in `tls.diff` is not TLS and takes effect with TLS off, named in the patch header: `MakeSlaveOfWithPort` connects to the replica on its own port rather than on the master's. It changes a dial, not a pod template, so shipping the rebuilt image rolls nothing; the plaintext pod templates stay byte-identical to v3.3.5. The image build registration this branch used to carry (`make -C packages/system/redis-operator image` in the root `build:` target) landed on main with the Valkey service (#3380), so the rebuilt operator image is produced by CI from main as well; the digest in `values.yaml` is still refreshed by release preparation, not here. The chart patches re-apply at the end of `update:`, the same way reloader, grafana-operator, ingress-nginx, seaweedfs and kamaji carry theirs. Otherwise `rm -rf charts` takes TLS back out on the next `make update` and nothing fails. `make update` reproduces the committed tree byte for byte. TLS is chosen when the instance is created. The config gets `port 0` and the listener moves to `tls-port`, so TLS replaces plaintext rather than running beside it, and a running instance cannot change mode: a replica restarted onto the TLS listener cannot replicate from a master still on plaintext, and Sentinels already on TLS would fail over to a stale replica. The RedisFailover schema in the carried CRD patch rejects a change of `tls.enabled`, which on an existing instance surfaces as a failed HelmRelease upgrade while the instance keeps serving what it did; the README points at creating a new instance and migrating. That is also why TLS is not inferred from `external`: an externally reachable instance is the one most likely to have clients an upgrade must not drop. Existing releases keep their plaintext listener and are not restarted by the rebuilt operator image. The tenant gets `ca.crt` and nothing else, through `tenantsecrets`. The chart's own Secrets take a dot between the release and the suffix (`<release>.ca-tls`, `<release>.tls`, `<release>.ca-cert`): the suffixes are dot-free, so the split is unique and release `foo`'s CA Secret can no longer be release `foo-ca`'s leaf, the collision the hyphenated form had. The chart renders a `TenantProjection` pointing at `<release>.ca-tls`, and the CA-extraction controller re-encodes the one named key from the parsed certificate, so nothing that is not a certificate gets through. Same path as postgres, mongodb, nats and qdrant. No Secret of this release is granted to the tenant by name beyond the generated password, and that includes `<release>.ca-cert`, the CA-only copy the operator publishes for itself. It holds no key today, but the operator has Create and Update on Secrets cluster-wide, so what sits at that name is the operator's call rather than the platform's, and RBAC cannot filter by key. `hack/check-redis-rd-secrets.bats` pins the whole rendered Secret surface instead of a list of known-bad names, because an include with no `resourceNames` and no `matchLabels` restricts nothing. The chainsaw suite for redis gains two cases. A TLS fixture asserts the cert-manager chain is issued, the TLS-probed Redis replicas and Sentinel come up, the operator publishes its CA-only Secret under the widened ClusterRole, the tenant receives exactly `ca.crt` through `tenantsecrets` while the key-bearing Secrets stay off that path, and a `redis-cli --tls` PING through the service name completes with the same CA. A second case applies `tls.enabled` to a running plaintext release and asserts the refusal: the HelmRelease upgrade fails with the schema's message, the StatefulSet keeps its revision with both replicas Ready, plaintext still answers and no TLS listener appears. Certificate renewal rolls the pods from inside the operator. Redis and Sentinel read the certificate once at startup, so the patched operator stamps a hash of the TLS Secret's `tls.crt` and `ca.crt` into the pod template of both the Redis StatefulSet and the Sentinel Deployment on every reconcile, as `redis-failover.freshworks.com/tls-secret-hash`. A renewal changes the hash; the operator's own roller replaces the Redis pods one at a time and the Deployment controller rolls Sentinel. README names the fallback: if the roll cannot run, the deadline is the old certificate's expiry, past which the operator's own TLS connections fail along with the clients. `tls.authClients` maps to `tls-auth-clients` and defaults to `no`. The platform mints no client certificate for the tenant and the CA key is out of reach, so `yes` is only usable when the client certificate comes from whoever also controls the CA. Changing the value on a running release restarts the Redis and Sentinel pods: Redis reads the directive from its config file only at startup, so the operator folds the effective value into the certificate hash that drives the rollout, and the change lands when it is made rather than at the next unrelated restart. Follow-ups in the carried operator patch go upstream first, to freshworks-oss/redis-operator#85 or a successor there, and are picked up here afterwards; the carried patch does not grow Go that upstream has not seen. `tlsConfigFor` fetches the Secret on every call, tens of GETs per reconcile. The renewal roll is static analysis so far: the live check, `cmctl renew` on a running release while watching the `rfr-*` and `rfs-*` pods roll, is still owed and belongs with the e2e follow-up. Two facts about the shipped image rather than the patch. `packages/system/redis-operator/values.yaml` keeps the digest of the image built before the patch until release preparation rebuilds it, so on `main` between this merge and the next release-prep, `tls.enabled` renders the chain and `spec.tls` while the running operator ignores them and serves plaintext. And the pinned v3.3.5 tarball carries GO-2026-5026 in `golang.org/x/net` v0.53.0, reached through `PodService.UpdatePodLabels`; the port neither introduces nor fixes it, upstream `master` is already on v0.55.0, so it lands with the next tag bump rather than here. Outside the patch: `cozyvalues-gen` renders `{}` in the README Value column for `tls.authClients`: a child property without a default inherits the parent object's `{}`, and `make generate` reproduces it byte for byte, so it is a generator bug rather than something to hand-edit here. ### Screenshots No UI changes. ### Downstream repositories - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [x] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: cozystack/terraform-provider-cozystack#37 - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: I walked the trigger map against the diff file by file. terraform-provider-cozystack is reached, tracked as cozystack/terraform-provider-cozystack#37 rather than a speculative PR: `values.schema.json` gains a `tls` object, and the provider hand-writes a schema, model and expand/flatten pair per Kind. It already models `tls` for postgres, kafka, nats and qdrant, not for redis. Redis needs different wording from those four, whose descriptions say to omit the block to follow `external`, which is the one thing redis does not do. Nothing else is reached. The website regenerates this package's reference page from `README.md` on a stable tag, so the README changes need no PR there. No package added, renamed or removed. `packages/core/platform/values.yaml`, variant and bundle composition and release asset names are untouched, and so are `hack/package.mk`, `hack/common-envs.mk` and `hack/update-crd.sh`, which is what ccp and external-apps-example read. The two new files under `hack/` are bats unit tests. No node prerequisite, installer value, telemetry metric or cozy-proxy label changes. ### Release note ```release-note feat(redis): add opt-in TLS for Redis and Sentinel. Set `tls.enabled: true` to serve both on a TLS-only listener backed by a per-release cert-manager chain; the tenant reads the CA certificate as the `<release>.tenant-ca` tenant secret. TLS is off unless set and is not inferred from `external`, so existing releases keep serving plaintext and are not restarted by the rebuilt operator image. TLS is chosen at creation: a running instance cannot switch modes, the change is rejected and surfaces as a failed HelmRelease upgrade, so add TLS by creating a new instance and migrating to it. Requires cert-manager, which the platform's `system` bundle installs on every variant. A renewed certificate rolls the Redis and Sentinel pods through the operator. TLS support is carried as a downstream patch over redis-operator v3.3.5 until freshworks-oss/redis-operator#85 lands in a release. ``` |
||
|
|
d800526696 |
feat(mariadb): add TLS support via cert-manager (#2680)
## What this PR does Gives MariaDB a chart-managed certificate chain, makes TLS enforcement explicit, and hands the tenant a key-free trust anchor instead of a private key. - `tls.issuer` selects who issues the server certificate: `operator` (the CA mariadb-operator manages itself, the default and today's behaviour) or `cert-manager` (the instance gets its own CA, and this is what covers the external hostname). The field is named for the issuer rather than for on/off because TLS is served under both — unlike the similarly spelled field in some other charts, which really is a switch. - `tls.required` refuses plaintext connections. It is opt-in and defaults to false: enforcing TLS before the tenant can obtain the CA would be backwards, and a platform upgrade must not sever connections that work today. - Neither issuer is inferred from `external`. Existing instances keep the CA they have and change nothing on upgrade. - The tenant no longer receives `<release>-ca-tls` or `<release>-tls` — the CA private key and the server private key. It receives the operator's `ca.crt`-only bundle, which works today, and the ResourceDefinition additionally declares that bundle as the source for the CA-extraction controller. ### Fixes along the way - The TLS backup path could never have worked: it passed `--ssl-mode=VERIFY_IDENTITY`, a MySQL-8 flag the MariaDB client rejects during argument parsing. Replaced with `--ssl-ca` + `--ssl-verify-server-cert`, and the test now asserts the dialect rather than the presence of a substring. - The leaf certificate carried `clientAuth`. Because `REQUIRE X509` accepts any certificate signed by a trusted CA, the server's own key was usable as a client credential. Removed. - The CA and server private keys survived deletion of the instance: cert-manager runs with `enableCertificateOwnerRef: false`, so removing the `Certificate` left the Secrets behind. Both are now cleaned up by the existing post-delete hook. - The Secrets are labelled for the operator's watch predicate, without which a renewed leaf would not reach the pods until a full resync. ### Why the include carries an exclude list The trust-anchor selector matches by label with no name constraint, so the enumerable dangerous set is excluded explicitly — both issuers' private keys, the internal credentials, the backup keys, and the metrics configuration, which the operator writes unconditionally with credentials in plain text for a user holding `SELECT` on everything. The backstop is bounded and the comment says so: matching is by exact name with no globs, so a Secret introduced by a future operator version, or a transient object created mid-renewal, stays in scope of the label-only include. The primary defence remains that the controller stamps only its own projection. ## Known limitations 1. **Merge is blocked on #3299 — and the blocking is mechanical, not just procedural.** The CRD on `main` does not carry `spec.caCert`, and under Flux's Server-Side Apply the API server rejects a field the schema does not declare (`field not declared in schema`); under the older client-side apply the same field was silently dropped, so installing this branch without #3299 fails outright. An earlier revision of this description claimed the field would be silently pruned; that was wrong. The tenant still gets a usable trust anchor once merged in order, through the operator's bundle. 2. **The deprecated backup CronJob cannot start at all**, for reasons that predate this branch: its `secretKeyRef` names a Secret and key the chart never renders. That is filed separately as #3347 and deliberately not fixed here, since repairing it would start scheduled dumps on production instances from a TLS change. The tests that pin the backup TLS flags therefore pin a path that does not currently execute — the suite header says so. 3. **The operator's `clusterName` is hardcoded**, so on a cluster with a non-default domain the SANs and the name the operator dials diverge. Pre-existing, raised separately. 4. **No e2e coverage for the managed path.** Unit coverage pins the rendered chain, the SAN set, the exclusion set and the guards; the end-to-end handshake is not asserted in CI. ### On review coverage Two independent review passes were run repeatedly against this branch — seven rounds, converging from a security finding, through comment accuracy, to guard strength, and finally to nothing new. A third, model-based pass reviewed the branch through `6cb1bd236` and its findings were fixed; it could not be run against the final revision because of an external account quota lasting until 25 July. The delta since that revision is substantial and includes runtime code, not only tests and docs. Recording this so the coverage is not overstated: if the quota returns before this is ready to merge, that pass is worth running again. ### Downstream repositories `packages/apps/mariadb/values.schema.json` gains the `tls` object, so `cozystack/terraform-provider-cozystack` needs the field modelled for `cozystack_mariadb`; tracked in cozystack/terraform-provider-cozystack#40. ### Release note ```release-note feat(mariadb): chart-managed TLS via cert-manager, opt-in enforcement, and the CA published to tenants as a key-free trust anchor ``` |
||
|
|
18d02d813a |
fix(cacert): make the comments match the reconciler (#4064)
## What this PR does Comments in the CA-extraction package claim guards the code does not have. No behaviour change; one test is renamed to match the property it pins. - `ManagedLabel` said the reconciler only ever writes over a Secret carrying that label. It does not. `isOurProjection` decides that from the Secret type and the owner reference, and never looks at the label, which any writer in the namespace can strip. - `managedByCozystackLabel` said the reconciler never writes it. It deletes it whenever the selectors digest changes, which is what gets the projection re-admitted by the lineage webhook. - `projectionSuffix` said the dotted canonical name cannot collide with an operator-owned object. A release the apps API created cannot carry a dot, since application names are DNS-1035 labels and `release.prefix` is restricted to `^[a-z0-9-]*$`. The guarantee stops there: the controller reads the release from the `helm.toolkit.fluxcd.io/name` label, where a dot is legal, so a hand-written HelmRelease sits outside it. On the validated path the dot can only come from the suffix an operator or chart appends, which nothing validates at all, and since a canonical name ends in `.tenant-ca` and carries that one dot, a suffix ending in `.tenant-ca` is the only one that reaches a canonical name. The comment says all of that now. - The test pinning that name claimed the property for any engine suffix, in its doc and in its own name, while its loop samples only dot-free ones. It is now `TestProjectionNameCannotCollideWithADotFreeEngineSuffix`, and its doc says what the loop covers. ### Screenshots Not a UI change. ### Downstream repositories Comments and one test name in two Go files. No package, values key, schema field, CRD, metric, release asset or `hack/` target changes, so nothing in the trigger map fires. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Clarified certificate reconciliation behavior, including label handling, projection naming, and source certificate scenarios. * Documented naming guarantees and their API boundaries more precisely. * **Tests** * Updated test descriptions and failure messaging to clarify coverage for dot-free engine suffixes. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |