Commit Graph
13 Commits
Author SHA1 Message Date
Alexey Artamonov 2d1e425339 docs(postgres): correct the KEDA-guard rationale and note the gradual client-side rollout
The KEDA-missing fail-closed guard's comment still described the replaced
drop-instances design ("db.yaml drops spec.instances ... CNPG collapses to 1");
under the shipped seed design db.yaml seeds instances at max(replicas,
effectiveMin), so a skipped ScaledObject leaves a fixed-size cluster with no HPA
- autoscaling silently absent, not a collapse. Correct the rationale. Also note
on the cozyrds apply-strategy annotation that the client-side switch rolls out
per-app as each Application is next written through the aggregated API, not
fleet-wide at upgrade (the ApplicationDefinition reconciler syncs only
chartRef/valuesFrom/labels onto a live HelmRelease). Follow-up from #3954 (#4187).

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-09 21:36:54 +03:00
Alexey Artamonov 9511f56307 fix(postgres): require service.namespace.svc shape in the autoscaling SSRF host guard
The serverAddress host regex accepted any dotted tail ending in .svc, so a
single-label host like foo.svc passed the in-cluster-DNS check. It then reached
the namespace-pin, whose extraction regex left the unresolvable host unchanged
and rejected it as a namespace outside the tenant subtree - the right outcome by
the wrong path, with a misleading message. Anchor the host to two DNS labels
before .svc so it is refused at the host check with the actionable message, and
drop the stray dot the old label class allowed.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-08 18:06:34 +03:00
Alexey Artamonov 678d22f209 docs(postgres): move the serverAddress namespace-pin note out of a template comment
Closes IvanHunters' MINOR on scaledobject.yaml:97. The API-owner-gate marker was a Go
template comment, stripped at render, so the ownership question and the pin's residual
surface were recorded where nobody reads them again. State the constraint on the
serverAddress field doc (propagated into values.schema.json, README and the cozyrds
openAPISchema) including the accepted residual: tenant-root is always in the allow list
because the derived default points there, so a tenant can still aim the operator at a root
Service as a blind in-cluster probe (no response body reaches the tenant), which the egress
NetworkPolicy does not narrow. The API-owner sign-off is tracked on the PR, not in a
stripped comment.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-07 21:03:15 +03:00
Alexey Artamonov 3a09c3e9dc fix(postgres): guard dry-run/transition against the paused HPA min-clamp
Closes IvanHunters' MAJOR on scaledobject.yaml:39. The render guard rejected
replicas > maxReplicas but not the symmetric low side. Under dry-run/transition the
chart renders the static instances=replicas while the ScaledObject carries
minReplicaCount=effectiveMin with both scale directions paused. The pause disables
the HPA's scaling policies, not its min-replica clamp, so when replicas < effectiveMin
the HPA raises the live count to the floor on its first reconcile — a scale-up in a
mode documented as "actuates neither way" — and because the rendered instances value
never changes, the client-side two-way patch never pulls it back, leaving permanent
extra standbys and PVCs. Fail closed and tell the operator to stage replicas to the
floor first. Under dry-run/transition this floor guard is strictly broader than the
db.yaml quorum wedge (any replicas <= maxSyncReplicas is also < effectiveMin), so it
supersedes it there; the wedge stays exercised on the disabled path.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-07 20:48:11 +03:00
Alexey Artamonov 187f18a167 fix(postgres): harden autoscaler serverAddress guard and cap maxReplicas
The serverAddress SSRF guard had two holes beyond the external-domain suffix
already closed. It extracted the host by truncating at the first `:` or `/`,
which a `http://<svc>.<ns>.svc:port@evil.example.com/` userinfo payload defeats:
the truncated text passes the `.svc` check while net/url dials the host after the
`@`. And it validated only the host shape, never the namespace, so a tenant could
point the cluster-privileged KEDA operator at a sibling tenant's or a system
namespace's Service (the egress NetworkPolicy's `cluster` entity does not stop
in-cluster egress). Parse the URL authority properly (scheme, path/query/fragment,
userinfo to the last `@`, then port) and pin the host's namespace to the tenant's
own subtree via cozy-lib.rbac.allParentTenantsAndThis. The namespace pin narrows
what a tenant may set and needs API-owner sign-off (noted at the guard).

Separately, nothing bounded maxReplicas: the schema carries no maximum and the
aggregated apiserver does not range-validate the CR, so a tenant could set
maxReplicas: 100000 and drive KEDA to scale CNPG toward that count. Add a render
guard rejecting a maxReplicas above a sane read-replica ceiling.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-07 15:29:06 +03:00
Alexey Artamonov 2bf37f8260 fix(postgres): anchor autoscaler SSRF guard to the cluster domain
The serverAddress in-cluster check accepted an unanchored trailing domain
(`\.svc(\.[a-zA-Z0-9.-]+)?$`), so a crafted host like x.svc.attacker.com — a
`.svc` label followed by any external domain — passed the guard. A tenant could
then point the cluster-privileged KEDA operator off-cluster (the egress
NetworkPolicy is the only remaining barrier, and it is inert where the CNI does
not enforce CiliumNetworkPolicy).

Anchor the optional suffix to this cluster's domain (from _cluster.cluster-domain,
injected by the platform from networking.clusterDomain): a legitimate vmselect is
<service>.<namespace>.svc or ...svc.<clusterDomain>, nothing else. The prior tests
only exercised non-.svc hosts and gave false confidence; add a case for the
.svc.<external> bypass.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-07 10:20:24 +03:00
Alexey Artamonov 6083a9fa1d feat(postgres): guard replicas>maxReplicas and pin serverAddress to in-cluster
Close two render-guard follow-ups the design proposal names for this PR:

- Fail closed when replicas exceeds autoscaling.maxReplicas. The stock HPA's
  currentReplicas > maxReplicaCount branch writes /scale = maxReplicaCount
  unconditionally on the first reconcile — even under the both-direction pause of
  dryRun/transition — so a resting/seed count above the ceiling would shed the
  surplus standbys and their PVCs in one step, past the scale-down pacing. Mirrors
  the existing effectiveMin > maxReplicas guard.
- Refuse an autoscaling.serverAddress whose host is not an in-cluster Service DNS
  name (…svc[.<domain>]). The cluster-privileged KEDA operator fetches it every
  poll, so an external/IP address is an SSRF surface (cloud metadata endpoint,
  another tenant); the query body is already namespace-scoped, this restricts the
  destination. Defense-in-depth pairs with the operator egress policy (keda package).

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-04 23:52:57 +03:00
Alexey Artamonov 826f08972f fix(apps/postgres): make autoscaler fail-safe, dip-free, and observable
Address review findings on the KEDA read-replica autoscaler where the
implementation diverged from the merged design proposal.

Fail-safe read load. Gate the `or vector(0)` floor on the existence of ANY CNPG
instance pod (`count(kube_pod_labels{...instance_role=~".+"}) > 0`) — primary or
replica — so the query is empty ONLY when no CNPG pod labels exist at all (a
broken kube-state-metrics pipeline), not during initial provisioning when the
sole replica-side pod is CNPG's join Job (jobRole, not instanceRole). Also pin
`ignoreNullValues: "false"` on the trigger: KEDA's prometheus scaler defaults it
true, turning an EMPTY result into metric 0 (not an error) — the HPA would then
desire 0 and clamp to the floor, i.e. silently scale a broken-pipeline cluster
DOWN. With it false an empty result is a scaler error, so the HPA HOLDS and
DatabaseAutoscalerScalerErrors fires; gating on any instance role keeps a healthy
cluster (including a long basebackup) from tripping that alert.

Constant seed = max(replicas, effectiveMin), so enable/dryRun/transition flips
are a merge no-op with no dip and never undercut the quorum floor.

Fail closed when the quorum floor exceeds maxReplicas (was: effectiveMax silently
raised both bounds, unbounded for a large maxSyncReplicas).

Dry-run / transition keep the HPA (unidirectional pause pair, not the HPA-deleting
full pause) but both SelectPolicy=Disabled, so desired normalizes to current — the
recommendation is read from the read-load metric, not a diverging desired count.

Observability. DatabaseAutoscalerAtMax keys on desired >= maxReplicas, NOT the
ScalingLimited condition (which is also true at the floor, so it fired on every
idle cluster with wrong "raise maxReplicas" advice). Dashboard read-load panels
are per-cluster (label_cnpg_io_cluster) with a CPU panel for ReadCPUUtilization
and the exact clamp in descriptions; ScaleStuck's Pending intersection is scoped
to CNPG pods; alerts.yaml narrative comments moved to Go-template comments so they
do not ship into the PrometheusRule.

Guard the seeding invariant: TestBuildHelmReleaseSpecNoForceOrDriftDetection.
Regenerated README/schema/types.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-04 23:19:25 +03:00
Alexey Artamonov b40cf82642 fix(postgres): seed instances on both paths, add input guards, and harden alerts
Address IvanHunters' deep review round on #3954:

- [MAJOR] Enabling autoscaling directly on an existing non-quorum database no
  longer collapses it to 1 instance and deletes every replica PVC. The chart now
  SEEDS spec.instances at the quorum floor (effectiveMin) on BOTH the quorum and
  non-quorum active paths instead of omitting the field in the non-quorum case;
  the constant seed is a three-way-merge no-op so KEDA still owns the runtime
  count via /scale, and the enable dip is capped at the floor (>=2) rather than
  collapsing to 1. Documents the client-side-3-way-merge (not SSA) assumption
  that the no-op relies on.

- [MINOR] Fail closed on autoscaling.minReplicas > autoscaling.maxReplicas instead
  of silently resolving the ceiling upward (which defeated maxReplicas as a cost
  knob). The stray maxReplicas: 0 case now rejects rather than clamps.

- [MINOR] The disable-wedge guard, its values note and its test no longer emit the
  invalid `kubectl scale --subresource=scale` (scale has no --subresource); use
  `kubectl scale cluster.postgresql.cnpg.io/<name> --replicas=N`.

- [MINOR] Correct the enablement note: raising minReplicas/maxSyncReplicas on a
  live cluster is NOT harmless when KEDA's live count exceeds the new floor (it
  decommissions surplus replicas + PVCs); prescribe pin-first for raises too.

- [MINOR] Add DatabaseAutoscalerKedaOperatorDown (absent(keda_build_info)): when
  the operator is unhealthy a ScaledObject never becomes an HPA and no other alert
  can fire. Intersect DatabaseAutoscalerScaleStuck with a Pending pod so a healthy
  long basebackup (Running pod, desired>current >15m) no longer false-fires.

- [MINOR] Keep the KEDA CRDs on package uninstall (crds.additionalAnnotations
  resource-policy: keep via the shim) so removing the keda package doesn't
  cascade-delete every tenant ScaledObject.

- [MINOR] cozy-lib helper: reject a non-numeric bound (int "two" -> 0 -> a
  scale-to-zero ScaledObject) and render an explicit pollingInterval/cooldownPeriod
  of 0 (`if not kindIs invalid`, not `with`).

- [MINOR] Note the read-load fail-open: a broken metrics join reads as idle and
  silently pins autoscaling at the floor under real load.

New/updated tests cover the non-quorum seed, the min>max guard, the corrected
kubectl message, the operator-down + Pending-intersected alerts, the CRD keep
policy, and the non-numeric-bound helper guard.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-04 23:19:25 +03:00
Alexey Artamonov 70e4087432 fix(postgres): guard empty monitoring namespace; clarify alert scope and helper doc
Address the branch-review nits:

- Fail closed when the monitoring namespace is unset. serverAddress builds
  vmselect-shortterm.<ns-monitoring>.svc, and an empty ns-monitoring rendered the
  malformed vmselect-shortterm..svc:8481, surfacing late as a KEDA scaler error.
  The platform always injects it, but a hand-rendered/misconfigured tenant now
  gets an actionable render-time failure mirroring the KEDA-CRD and target guards.
  Tested.

- Document that the database-autoscaling PrometheusRule matches every KEDA HPA /
  ScaledObject error (the underlying series carry no owning-app label), which is
  correct only while database autoscaling is KEDA's sole consumer in cozystack;
  note to scope the rules if a non-database ScaledObject is ever added.

- Correct the cozy-lib.keda.scaledObject usage example: `paused` should reflect
  dry-run OR migration transition, not dry-run alone, so a second engine copying
  the example does not drop the transition pause.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-04 23:18:47 +03:00
Alexey Artamonov 284f5e5543 fix(postgres): fail closed when autoscaling is enabled without KEDA, guard target
autoscaling.enabled is a tenant-facing knob, but KEDA is an optional platform
package that is not installed by default. A tenant enabling autoscaling on a
cluster without KEDA would render a ScaledObject against an unknown kind and the
whole database HelmRelease would fail with an opaque `no matches for kind
"ScaledObject"`. Guard the ScaledObject template on
`.Capabilities.APIVersions.Has "keda.sh/v1alpha1"` and stop with an actionable
message instead. Skipping the render is deliberately NOT an option: db.yaml
already drops the static .spec.instances under active autoscaling, so a missing
ScaledObject would let CNPG collapse the cluster to 1 instance. Document the
KEDA prerequisite on the `enabled` field (README/schema).

Also reject a non-positive autoscaling.target at render time: it is a plain
integer now, but the desired-replica formula divides the read load by it, so 0
is undefined and negatives invert scaling. cozyvalues-gen has no minimum
directive, so the bound is a template guard.

Add unittests for both guards (a dedicated suite renders without the KEDA CRD so
the fail-closed path is exercised; the main suite now advertises keda.sh/v1alpha1
via capabilities).

Addresses IvanHunters (KEDA-absent) and CodeRabbit (non-positive target) review
feedback on #3954.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-04 23:18:47 +03:00
Alexey Artamonov d4c76fadb4 feat(postgres): two-phase autoscaling migration and read-CPU metric
Make enabling autoscaling on an existing database count-preserving, and
add a second read-load signal:

- autoscaling.transition (migration phase 1): keep .spec.instances
  rendered as a floor (from replicas, which the operator stages to the
  live count) AND pause the ScaledObject, so KEDA stands up without
  contending with Flux for the field. Flip transition off (phase 2) to
  drop the static field and hand the count to KEDA — suspend the
  HelmRelease across that apply, since helm-controller uses Helm's
  three-way merge (not SSA) and would otherwise race CNPG's default-to-1.
  The procedure is documented on the values field.
- autoscaling.metric (ReadConnections | ReadCPUUtilization): the query
  helper emits either the active-connections sum or the postgres-container
  CPU sum, both + target.

active = enabled && !dryRun && !transition; the ScaledObject is paused
whenever dryRun or transition is set. Covered by helm unittest (phase-1
holds the staged count + pauses; CPU metric query) and regenerated
schema/README/Go types/postgres-rd.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-04 23:18:47 +03:00
Alexey Artamonov 0335276b61 feat(postgres): render a KEDA ScaledObject for read-replica autoscaling
Implement the KEDA-based read-replica autoscaler for postgres per the
merged design proposal (cozystack/community#44):

- cozy-lib.keda.scaledObject: engine-agnostic helper rendering a KEDA
  ScaledObject (scale target, effective bounds, prometheus trigger,
  scale-down behavior, optional pause), fail-closed on bad input.
- postgres: an `autoscaling` values block (enabled / minReplicas /
  maxReplicas / target / maxReplicationLagSeconds / dryRun). Under active
  autoscaling the chart drops the static .spec.instances so Flux and the
  KEDA-managed HPA no longer contend, and renders a ScaledObject whose
  minReplicaCount folds in the synchronous-quorum floor
  (max(minReplicas, maxSyncReplicas+1, 2)); when the floor exceeds
  maxReplicas, quorum wins. dry-run keeps the static count and pauses the
  ScaledObject. The WorkloadMonitor tracks the quorum floor instead of a
  stale static value.

Covered by helm unittest (cozy-lib helper: valid render, pause, three
fail-closed guards; postgres: instances dropped, quorum-floored bounds,
quorum-over-max, dry-run pause). Regenerated values.schema.json, README,
Go types and postgres-rd.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-04 23:18:47 +03:00