Commit Graph
19 Commits
Author SHA1 Message Date
Aleksei Sviridkin 520af57905 chore: drop private cluster, host and personal names from the tree
Private infrastructure names, a real hostname, a commented-out
internal mirror block, an internal product name, and a reviewer's
name in a comment do not belong in a public repository; drop them
and replace what needs a stand-in with neutral placeholders.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-07 04:38:06 +03:00
Yan Bondarenko 0cf6e8c98b fix(keda): use severity informational instead of info in the at-max alert
Alerta validates the severity label against its alarm model, which has
no info in it, so the DatabaseAutoscalerAtMax notification is refused
the same way the piraeus warn alerts were. informational is the value
the model carries for this level, and the one the other rules in the
tree already use.

Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
Assisted-by: LLM
2026-09-30 13:53:06 +03:00
Aleksei Sviridkin dae5d5c5ae fix(build): name the repository of the keda and kuberture pins
Both packages pinned their images by writing tag@digest into the tag
field alone and left the repository to the vendored chart's defaults.
Tooling that reads refs from the package values saw a bare
"2.20.2@sha256:..." with no image to resolve, so the release gate
could not tell whether these images serve arm64. The values now
repeat the chart's registry and repository beside the pin; the
rendered manifests are unchanged.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-27 07:57:31 +03:00
Alexey Artamonov 49e8aa58f9 fix(keda): pin job=kube-state-metrics in the ScaleStuck Pending join
The DatabaseAutoscalerScaleStuck Pending-intersection term selected
kube_pod_status_phase and kube_pod_labels without job="kube-state-metrics",
unlike every other selector in this file (the rule's own HPA terms, all of
ScaleBlocked, AtMax). Each VMRule runs against a tenant's own vmselect; if that
vmselect ever carries a second series for either metric under a different job
(a tenant-run KSM alongside federated/remote-written series), the on(namespace,pod)
match becomes ambiguous and the whole rule errors out silently while its siblings
keep working. Pin the job label on both selectors so the join stays one-to-one.
The alerts unittest now asserts the pin on both terms so a revert goes red.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-08 16:35:57 +03:00
Alexey Artamonov a43e3e2709 fix(keda): digest-pin the three KEDA images
The shim rendered the operator, metrics-apiserver and admission-webhooks images
tag-only, and the values comment claimed a @sha256 pin was impossible without
patching the subchart. That is wrong: the vendored template renders
repository:{{ tag | default appVersion }}, and repository:tag@sha256:<digest> is a
valid OCI reference where the digest wins at resolution, so the pin goes straight
into the tag field of these shim values with no subchart change.

Pin all three to the multi-arch image-index digests for the 2.20.2 tag on ghcr.io,
matching how every other cozystack system image is pinned, and replace the
incorrect note. Guard the operator image keeps a digest with a helm-unittest so a
re-vendor that drops the override goes red rather than silently shipping a floating
tag. Bump the digests in lockstep with KEDA_CHART_VERSION on re-vendor.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-08 10:48:06 +03:00
Alexey Artamonov 1192922a88 docs(keda): document the teardown order for the ScaledObject finalizer
Closes IvanHunters' MINOR on keda/values.yaml:19. The resource-policy: keep comment
reasons about an admin removing the keda package but covers only the CRDs, not the
finalizer.keda.sh KEDA stamps on every ScaledObject: with the operator gone a tenant
deleting their Postgres app gets a ScaledObject stuck Terminating and a HelmRelease
uninstall that hangs, recoverable only by an admin finalizer patch. Note that autoscaling
must be disabled on every tenant Postgres before the package is removed, so the operator
clears the finalizers while it is still up.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-07 21:03:15 +03:00
Alexey Artamonov 568ccf4835 fix(keda): gate the operator CiliumNetworkPolicy on the cilium.io/v2 API
Closes IvanHunters' MAJOR on keda/values.yaml:60. The shim hardcodes
networkPolicy.flavor: cilium and the vendored templates gate only on
networkPolicy.enabled + flavor==cilium, never on the cilium.io/v2 CRD being served.
keda is offered on every variant from the shared system bundle, so on a variant that
ships no Cilium the CiliumNetworkPolicy is unmappable and the whole HelmRelease fails
to install. Add a .Capabilities.APIVersions.Has "cilium.io/v2" check to the three
CiliumNetworkPolicy templates so the package installs on any CNI; where Cilium is not
served the egress backstop is simply absent and the chart's serverAddress namespace-pin
remains the control. The vendored edit is captured as patches/ so make update re-applies
it instead of silently reverting.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-07 20:56:18 +03:00
Alexey Artamonov ccf6165645 fix(keda): scope autoscaler alerts per-cluster and fix ScaleBlocked false-fire
ScaleStuck and ScaleBlocked joined on namespace alone, assuming one autoscaled
database per namespace — but a tenant may run several Postgres apps in one
namespace. A Pending pod of one CNPG cluster then falsely tripped a healthy
sibling's ScaleStuck, and ScaleBlocked's namespace-wide pod count masked a real
block on one cluster. Derive a `cluster` label from the keda-hpa-<cluster> HPA
name with label_replace and join the pod terms on (namespace, cluster).

ScaleBlocked also cried wolf on a healthy multi-step scale-up of a single large
database: CNPG materialises replicas one basebackup at a time (each often >15m),
so the instance count legitimately lags a desired jump of +2 with no Pending pod,
and the old rule fired with the actively-harmful "lower maxReplicas" advice. Count
only instance pods (a transient join Job pod otherwise inflates the count and
masks a block) and suppress via `unless` while a join Job is provisioning a
replica; a genuine admission block creates neither. Make the desired > current
comparison an explicit on(namespace,horizontalpodautoscaler) join.

Note in the egress NetworkPolicy values that its `cluster` entity does not
restrict cross-namespace egress (the chart's namespace-pin does) and that
off-cluster / link-local IMDS denial depends on CNI enforcement.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-07 15:29:06 +03:00
Alexey Artamonov 7cdfe7a197 feat(keda): add the quota-blocked alert and restrict operator egress
Close two follow-ups the design proposal names for this PR:

- DatabaseAutoscalerScaleBlocked: the Pending-independent companion to
  DatabaseAutoscalerScaleStuck. A ResourceQuota (or any admission webhook) refuses
  the new standby pod at admission, so no pod object is created and ScaleStuck's
  Pending intersection misses it. Key instead on desired_replicas > count(CNPG pods
  in the namespace): a healthy basebackup's pod exists (Running) so the count already
  equals desired, while an un-admitted pod leaves the count short. Correct the stale
  ScaleStuck comment that claimed it covered the quota case.
- Enable the vendored KEDA egress NetworkPolicy (flavor cilium): egress is
  toEntities [kube-apiserver, cluster] only — apiserver and in-cluster endpoints
  (CoreDNS, vmselect, CNPG) stay reachable, world (off-cluster) is denied — so the
  operator cannot be used as an SSRF relay off-cluster. The kube-apiserver entity
  covers the control plane however it is addressed, so it does not starve the
  operator's apiserver access the way a raw-CIDR policy could.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-04 23:52:57 +03:00
Alexey Artamonov daf733ad9c fix(keda): narrow autoscaler RBAC and scope the operator-down alert
Restrict the KEDA operator ClusterRole: set permissions.operator.restrict.secret
so it drops cluster-wide secret list/watch, and pin rbac.scaledRefKinds to
postgresql.cnpg.io/clusters so it no longer holds a wildcard */scale across every
group.

Gate DatabaseAutoscalerKedaOperatorDown on the presence of keda-hpa-* HPA series
so a tenant running its own Monitoring app (which never scrapes keda_build_info,
root-only) does not fire it forever about a healthy platform component. Make the
two negative containsDocument matchers in the bundles-default test non-vacuous
(any: true) so "KEDA is not installed by default" is actually asserted. Drop the
unused pollingInterval/cooldownPeriod params from the cozy-lib keda helper.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-04 23:19:25 +03:00
Alexey Artamonov 826f08972f fix(apps/postgres): make autoscaler fail-safe, dip-free, and observable
Address review findings on the KEDA read-replica autoscaler where the
implementation diverged from the merged design proposal.

Fail-safe read load. Gate the `or vector(0)` floor on the existence of ANY CNPG
instance pod (`count(kube_pod_labels{...instance_role=~".+"}) > 0`) — primary or
replica — so the query is empty ONLY when no CNPG pod labels exist at all (a
broken kube-state-metrics pipeline), not during initial provisioning when the
sole replica-side pod is CNPG's join Job (jobRole, not instanceRole). Also pin
`ignoreNullValues: "false"` on the trigger: KEDA's prometheus scaler defaults it
true, turning an EMPTY result into metric 0 (not an error) — the HPA would then
desire 0 and clamp to the floor, i.e. silently scale a broken-pipeline cluster
DOWN. With it false an empty result is a scaler error, so the HPA HOLDS and
DatabaseAutoscalerScalerErrors fires; gating on any instance role keeps a healthy
cluster (including a long basebackup) from tripping that alert.

Constant seed = max(replicas, effectiveMin), so enable/dryRun/transition flips
are a merge no-op with no dip and never undercut the quorum floor.

Fail closed when the quorum floor exceeds maxReplicas (was: effectiveMax silently
raised both bounds, unbounded for a large maxSyncReplicas).

Dry-run / transition keep the HPA (unidirectional pause pair, not the HPA-deleting
full pause) but both SelectPolicy=Disabled, so desired normalizes to current — the
recommendation is read from the read-load metric, not a diverging desired count.

Observability. DatabaseAutoscalerAtMax keys on desired >= maxReplicas, NOT the
ScalingLimited condition (which is also true at the floor, so it fired on every
idle cluster with wrong "raise maxReplicas" advice). Dashboard read-load panels
are per-cluster (label_cnpg_io_cluster) with a CPU panel for ReadCPUUtilization
and the exact clamp in descriptions; ScaleStuck's Pending intersection is scoped
to CNPG pods; alerts.yaml narrative comments moved to Go-template comments so they
do not ship into the PrometheusRule.

Guard the seeding invariant: TestBuildHelmReleaseSpecNoForceOrDriftDetection.
Regenerated README/schema/types.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-04 23:19:25 +03:00
Alexey Artamonov b40cf82642 fix(postgres): seed instances on both paths, add input guards, and harden alerts
Address IvanHunters' deep review round on #3954:

- [MAJOR] Enabling autoscaling directly on an existing non-quorum database no
  longer collapses it to 1 instance and deletes every replica PVC. The chart now
  SEEDS spec.instances at the quorum floor (effectiveMin) on BOTH the quorum and
  non-quorum active paths instead of omitting the field in the non-quorum case;
  the constant seed is a three-way-merge no-op so KEDA still owns the runtime
  count via /scale, and the enable dip is capped at the floor (>=2) rather than
  collapsing to 1. Documents the client-side-3-way-merge (not SSA) assumption
  that the no-op relies on.

- [MINOR] Fail closed on autoscaling.minReplicas > autoscaling.maxReplicas instead
  of silently resolving the ceiling upward (which defeated maxReplicas as a cost
  knob). The stray maxReplicas: 0 case now rejects rather than clamps.

- [MINOR] The disable-wedge guard, its values note and its test no longer emit the
  invalid `kubectl scale --subresource=scale` (scale has no --subresource); use
  `kubectl scale cluster.postgresql.cnpg.io/<name> --replicas=N`.

- [MINOR] Correct the enablement note: raising minReplicas/maxSyncReplicas on a
  live cluster is NOT harmless when KEDA's live count exceeds the new floor (it
  decommissions surplus replicas + PVCs); prescribe pin-first for raises too.

- [MINOR] Add DatabaseAutoscalerKedaOperatorDown (absent(keda_build_info)): when
  the operator is unhealthy a ScaledObject never becomes an HPA and no other alert
  can fire. Intersect DatabaseAutoscalerScaleStuck with a Pending pod so a healthy
  long basebackup (Running pod, desired>current >15m) no longer false-fires.

- [MINOR] Keep the KEDA CRDs on package uninstall (crds.additionalAnnotations
  resource-policy: keep via the shim) so removing the keda package doesn't
  cascade-delete every tenant ScaledObject.

- [MINOR] cozy-lib helper: reject a non-numeric bound (int "two" -> 0 -> a
  scale-to-zero ScaledObject) and render an explicit pollingInterval/cooldownPeriod
  of 0 (`if not kindIs invalid`, not `with`).

- [MINOR] Note the read-load fail-open: a broken metrics join reads as idle and
  silently pins autoscaling at the floor under real load.

New/updated tests cover the non-quorum seed, the min>max guard, the corrected
kubectl message, the operator-down + Pending-intersected alerts, the CRD keep
policy, and the non-numeric-bound helper guard.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-04 23:19:25 +03:00
Alexey Artamonov e8609ac00d fix(keda): ship the autoscaling dashboard via grafana-operator, not a dead ConfigMap
The database-autoscaling dashboard shipped from the keda package as a ConfigMap
labeled grafana_dashboard: "1" for a Grafana sidecar that cozystack does not run
(no k8s-sidecar anywhere; every dashboard is provisioned by grafana-operator from
the grafana-dashboards image via GrafanaDashboard CRs). It was enabled by default
and never appeared in any Grafana. Move the JSON to dashboards/db/database-autoscaling.json
and register db/database-autoscaling in packages/system/monitoring/dashboards.list,
exactly how db/cloudnativepg and the other managed-db dashboards ship, and drop the
keda ConfigMap, its default-on toggle, and its wrong-mechanism test.

Addresses IvanHunters [MAJOR] review feedback on #3954.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-04 23:19:25 +03:00
Alexey Artamonov 70e4087432 fix(postgres): guard empty monitoring namespace; clarify alert scope and helper doc
Address the branch-review nits:

- Fail closed when the monitoring namespace is unset. serverAddress builds
  vmselect-shortterm.<ns-monitoring>.svc, and an empty ns-monitoring rendered the
  malformed vmselect-shortterm..svc:8481, surfacing late as a KEDA scaler error.
  The platform always injects it, but a hand-rendered/misconfigured tenant now
  gets an actionable render-time failure mirroring the KEDA-CRD and target guards.
  Tested.

- Document that the database-autoscaling PrometheusRule matches every KEDA HPA /
  ScaledObject error (the underlying series carry no owning-app label), which is
  correct only while database autoscaling is KEDA's sole consumer in cozystack;
  note to scope the rules if a non-database ScaledObject is ever added.

- Correct the cozy-lib.keda.scaledObject usage example: `paused` should reflect
  dry-run OR migration transition, not dry-run alone, so a second engine copying
  the example does not drop the transition pause.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-04 23:18:47 +03:00
Alexey Artamonov 42a533fc50 fix(platform): inject networking.clusterDomain into the KEDA package
KEDA's clusterDomain is a vendored-subchart value baked into the metrics-apiserver
gRPC target (keda-operator.<ns>.svc.<clusterDomain>:9666) and the operator's
serving-cert SANs at render time. A chart template cannot derive a subchart value
from _cluster, so the keda package hardcoded cozy.local — which silently breaks
(NXDOMAIN, every DB HPA FailedGetExternalMetric) on any cluster whose domain is
not the default. Inject the authoritative networking.clusterDomain into the shared
cozystack-values Secret under the keda subchart key so the value tracks the
platform config; the keda package keeps cozy.local as a standalone fallback. The
key is ignored by packages without a keda subchart (no app/extra schema sets root
additionalProperties:false). Add platform unittests asserting the Secret carries
keda.clusterDomain and that it tracks a non-default networking.clusterDomain.

Addresses the hardcoded-clusterDomain portability gap found in branch review.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-04 23:18:47 +03:00
Alexey Artamonov ec123649bf fix(keda): use the KEDA 2.20 metric name in the scaler-errors alert
The DatabaseAutoscalerScalerErrors rule queried keda_scaler_errors_total, which
KEDA 2.20 no longer exposes (verified against a live 2.20.2 operator: it emits
keda_scaled_object_errors_total for per-ScaledObject errors and
keda_scaler_detail_errors_total for per-scaler errors). The alert would have
silently never fired. Switch to keda_scaled_object_errors_total and add a
unittest assertion pinning the metric name so a rename regresses loudly.

Addresses CodeRabbit review feedback on #3954.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-04 23:18:47 +03:00
Alexey Artamonov ac82ce186c fix(keda): pin cozy.local domain and raise operator API throttle
The vendored KEDA chart defaults clusterDomain to cluster.local, but
cozystack runs on cozy.local. KEDA's metrics-apiserver dials the operator
metrics gRPC service at keda-operator.<ns>.svc.<clusterDomain>:9666 and the
operator mints that service's serving-cert SANs from the same domain, so the
default resolves to NXDOMAIN and every HPA hangs on FailedGetExternalMetric.
Pin keda.clusterDomain=cozy.local.

Also raise the operator's client-go throttle to 50 QPS / 100 burst: at the
upstream 20/30 a single-replica operator loses its leader lease under
scale-subresource polling and crash-loops (the client rate limiter starves
the lease renewal). Both were observed live on a dev cluster; guard them in
the package unittest.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-04 23:18:47 +03:00
Alexey Artamonov 237053191e feat(keda): ship database-autoscaling alerts and Grafana dashboard
Add cozystack-owned observability for managed-database read-replica
autoscaling, rendered by the keda umbrella (not the vendored subchart):

- PrometheusRule (templates/alerts.yaml), scoped to the KEDA-generated
  HPAs (keda-hpa-*): DatabaseAutoscalerScaleStuck (desired > current for
  15m — stuck scale-up with no automatic rollback), DatabaseAutoscalerAtMax
  (ScalingLimited=true for 30m — at max or quorum-raised ceiling), and
  DatabaseAutoscalerScalerErrors (KEDA cannot evaluate the trigger — never
  scale blind).
- A Grafana dashboard ConfigMap (grafana_dashboard sidecar label) with
  desired-vs-current replicas, active read connections, replication lag,
  and scaling-limited panels.

Both gated by databaseAutoscaling.alerts.enabled / .dashboard.enabled
(default true). Metric names are calibrated on a live cluster per the
proposal's PoC; the rule and panel shapes are what ships. Covered by helm
unittest (PrometheusRule with the three alerts + disable; dashboard
ConfigMap label + disable).

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-04 23:18:47 +03:00
Alexey Artamonov 7f8f2c1d86 feat(keda): add KEDA as an optional platform component
Vendor the upstream KEDA chart (kedacore/keda 2.20.2) as a new optional
system package and wire it into the platform, so database read-replica
autoscaling (which renders KEDA ScaledObjects) has a backend.

- packages/system/keda: umbrella cozy-keda + vendored chart (Makefile
  pins KEDA_CHART_VERSION with a lockstep assert), shim values enabling
  the operator / metrics-server / webhook ServiceMonitors under the
  platform Prometheus.
- core/platform: PackageSource cozystack.keda (namespace cozy-keda,
  dependsOn networking + prometheus-operator-crds; cert-manager is not
  needed, KEDA self-generates its webhook TLS) plus an optional bundle
  entry, opt-in via bundles.enabledPackages. It is not a dependsOn of any
  default package, so the platform HelmRelease never blocks on it.
- KEDA claims the cluster-wide external.metrics.k8s.io APIService
  (distinct from metrics-server's metrics.k8s.io — no conflict).

Covered by helm unittest: platform bundle enable/disable/opt-out, the
PackageSource dependsOn, and the vendored chart's external-metrics
APIService + shim-enabled ServiceMonitor.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-04 23:18:47 +03:00