Private infrastructure names, a real hostname, a commented-out
internal mirror block, an internal product name, and a reviewer's
name in a comment do not belong in a public repository; drop them
and replace what needs a stand-in with neutral placeholders.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Alerta validates the severity label against its alarm model, which has
no info in it, so the DatabaseAutoscalerAtMax notification is refused
the same way the piraeus warn alerts were. informational is the value
the model carries for this level, and the one the other rules in the
tree already use.
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
Assisted-by: LLM
Both packages pinned their images by writing tag@digest into the tag
field alone and left the repository to the vendored chart's defaults.
Tooling that reads refs from the package values saw a bare
"2.20.2@sha256:..." with no image to resolve, so the release gate
could not tell whether these images serve arm64. The values now
repeat the chart's registry and repository beside the pin; the
rendered manifests are unchanged.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The DatabaseAutoscalerScaleStuck Pending-intersection term selected
kube_pod_status_phase and kube_pod_labels without job="kube-state-metrics",
unlike every other selector in this file (the rule's own HPA terms, all of
ScaleBlocked, AtMax). Each VMRule runs against a tenant's own vmselect; if that
vmselect ever carries a second series for either metric under a different job
(a tenant-run KSM alongside federated/remote-written series), the on(namespace,pod)
match becomes ambiguous and the whole rule errors out silently while its siblings
keep working. Pin the job label on both selectors so the join stays one-to-one.
The alerts unittest now asserts the pin on both terms so a revert goes red.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The shim rendered the operator, metrics-apiserver and admission-webhooks images
tag-only, and the values comment claimed a @sha256 pin was impossible without
patching the subchart. That is wrong: the vendored template renders
repository:{{ tag | default appVersion }}, and repository:tag@sha256:<digest> is a
valid OCI reference where the digest wins at resolution, so the pin goes straight
into the tag field of these shim values with no subchart change.
Pin all three to the multi-arch image-index digests for the 2.20.2 tag on ghcr.io,
matching how every other cozystack system image is pinned, and replace the
incorrect note. Guard the operator image keeps a digest with a helm-unittest so a
re-vendor that drops the override goes red rather than silently shipping a floating
tag. Bump the digests in lockstep with KEDA_CHART_VERSION on re-vendor.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Closes IvanHunters' MINOR on keda/values.yaml:19. The resource-policy: keep comment
reasons about an admin removing the keda package but covers only the CRDs, not the
finalizer.keda.sh KEDA stamps on every ScaledObject: with the operator gone a tenant
deleting their Postgres app gets a ScaledObject stuck Terminating and a HelmRelease
uninstall that hangs, recoverable only by an admin finalizer patch. Note that autoscaling
must be disabled on every tenant Postgres before the package is removed, so the operator
clears the finalizers while it is still up.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Closes IvanHunters' MAJOR on keda/values.yaml:60. The shim hardcodes
networkPolicy.flavor: cilium and the vendored templates gate only on
networkPolicy.enabled + flavor==cilium, never on the cilium.io/v2 CRD being served.
keda is offered on every variant from the shared system bundle, so on a variant that
ships no Cilium the CiliumNetworkPolicy is unmappable and the whole HelmRelease fails
to install. Add a .Capabilities.APIVersions.Has "cilium.io/v2" check to the three
CiliumNetworkPolicy templates so the package installs on any CNI; where Cilium is not
served the egress backstop is simply absent and the chart's serverAddress namespace-pin
remains the control. The vendored edit is captured as patches/ so make update re-applies
it instead of silently reverting.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
ScaleStuck and ScaleBlocked joined on namespace alone, assuming one autoscaled
database per namespace — but a tenant may run several Postgres apps in one
namespace. A Pending pod of one CNPG cluster then falsely tripped a healthy
sibling's ScaleStuck, and ScaleBlocked's namespace-wide pod count masked a real
block on one cluster. Derive a `cluster` label from the keda-hpa-<cluster> HPA
name with label_replace and join the pod terms on (namespace, cluster).
ScaleBlocked also cried wolf on a healthy multi-step scale-up of a single large
database: CNPG materialises replicas one basebackup at a time (each often >15m),
so the instance count legitimately lags a desired jump of +2 with no Pending pod,
and the old rule fired with the actively-harmful "lower maxReplicas" advice. Count
only instance pods (a transient join Job pod otherwise inflates the count and
masks a block) and suppress via `unless` while a join Job is provisioning a
replica; a genuine admission block creates neither. Make the desired > current
comparison an explicit on(namespace,horizontalpodautoscaler) join.
Note in the egress NetworkPolicy values that its `cluster` entity does not
restrict cross-namespace egress (the chart's namespace-pin does) and that
off-cluster / link-local IMDS denial depends on CNI enforcement.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Close two follow-ups the design proposal names for this PR:
- DatabaseAutoscalerScaleBlocked: the Pending-independent companion to
DatabaseAutoscalerScaleStuck. A ResourceQuota (or any admission webhook) refuses
the new standby pod at admission, so no pod object is created and ScaleStuck's
Pending intersection misses it. Key instead on desired_replicas > count(CNPG pods
in the namespace): a healthy basebackup's pod exists (Running) so the count already
equals desired, while an un-admitted pod leaves the count short. Correct the stale
ScaleStuck comment that claimed it covered the quota case.
- Enable the vendored KEDA egress NetworkPolicy (flavor cilium): egress is
toEntities [kube-apiserver, cluster] only — apiserver and in-cluster endpoints
(CoreDNS, vmselect, CNPG) stay reachable, world (off-cluster) is denied — so the
operator cannot be used as an SSRF relay off-cluster. The kube-apiserver entity
covers the control plane however it is addressed, so it does not starve the
operator's apiserver access the way a raw-CIDR policy could.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Restrict the KEDA operator ClusterRole: set permissions.operator.restrict.secret
so it drops cluster-wide secret list/watch, and pin rbac.scaledRefKinds to
postgresql.cnpg.io/clusters so it no longer holds a wildcard */scale across every
group.
Gate DatabaseAutoscalerKedaOperatorDown on the presence of keda-hpa-* HPA series
so a tenant running its own Monitoring app (which never scrapes keda_build_info,
root-only) does not fire it forever about a healthy platform component. Make the
two negative containsDocument matchers in the bundles-default test non-vacuous
(any: true) so "KEDA is not installed by default" is actually asserted. Drop the
unused pollingInterval/cooldownPeriod params from the cozy-lib keda helper.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Address review findings on the KEDA read-replica autoscaler where the
implementation diverged from the merged design proposal.
Fail-safe read load. Gate the `or vector(0)` floor on the existence of ANY CNPG
instance pod (`count(kube_pod_labels{...instance_role=~".+"}) > 0`) — primary or
replica — so the query is empty ONLY when no CNPG pod labels exist at all (a
broken kube-state-metrics pipeline), not during initial provisioning when the
sole replica-side pod is CNPG's join Job (jobRole, not instanceRole). Also pin
`ignoreNullValues: "false"` on the trigger: KEDA's prometheus scaler defaults it
true, turning an EMPTY result into metric 0 (not an error) — the HPA would then
desire 0 and clamp to the floor, i.e. silently scale a broken-pipeline cluster
DOWN. With it false an empty result is a scaler error, so the HPA HOLDS and
DatabaseAutoscalerScalerErrors fires; gating on any instance role keeps a healthy
cluster (including a long basebackup) from tripping that alert.
Constant seed = max(replicas, effectiveMin), so enable/dryRun/transition flips
are a merge no-op with no dip and never undercut the quorum floor.
Fail closed when the quorum floor exceeds maxReplicas (was: effectiveMax silently
raised both bounds, unbounded for a large maxSyncReplicas).
Dry-run / transition keep the HPA (unidirectional pause pair, not the HPA-deleting
full pause) but both SelectPolicy=Disabled, so desired normalizes to current — the
recommendation is read from the read-load metric, not a diverging desired count.
Observability. DatabaseAutoscalerAtMax keys on desired >= maxReplicas, NOT the
ScalingLimited condition (which is also true at the floor, so it fired on every
idle cluster with wrong "raise maxReplicas" advice). Dashboard read-load panels
are per-cluster (label_cnpg_io_cluster) with a CPU panel for ReadCPUUtilization
and the exact clamp in descriptions; ScaleStuck's Pending intersection is scoped
to CNPG pods; alerts.yaml narrative comments moved to Go-template comments so they
do not ship into the PrometheusRule.
Guard the seeding invariant: TestBuildHelmReleaseSpecNoForceOrDriftDetection.
Regenerated README/schema/types.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Address IvanHunters' deep review round on #3954:
- [MAJOR] Enabling autoscaling directly on an existing non-quorum database no
longer collapses it to 1 instance and deletes every replica PVC. The chart now
SEEDS spec.instances at the quorum floor (effectiveMin) on BOTH the quorum and
non-quorum active paths instead of omitting the field in the non-quorum case;
the constant seed is a three-way-merge no-op so KEDA still owns the runtime
count via /scale, and the enable dip is capped at the floor (>=2) rather than
collapsing to 1. Documents the client-side-3-way-merge (not SSA) assumption
that the no-op relies on.
- [MINOR] Fail closed on autoscaling.minReplicas > autoscaling.maxReplicas instead
of silently resolving the ceiling upward (which defeated maxReplicas as a cost
knob). The stray maxReplicas: 0 case now rejects rather than clamps.
- [MINOR] The disable-wedge guard, its values note and its test no longer emit the
invalid `kubectl scale --subresource=scale` (scale has no --subresource); use
`kubectl scale cluster.postgresql.cnpg.io/<name> --replicas=N`.
- [MINOR] Correct the enablement note: raising minReplicas/maxSyncReplicas on a
live cluster is NOT harmless when KEDA's live count exceeds the new floor (it
decommissions surplus replicas + PVCs); prescribe pin-first for raises too.
- [MINOR] Add DatabaseAutoscalerKedaOperatorDown (absent(keda_build_info)): when
the operator is unhealthy a ScaledObject never becomes an HPA and no other alert
can fire. Intersect DatabaseAutoscalerScaleStuck with a Pending pod so a healthy
long basebackup (Running pod, desired>current >15m) no longer false-fires.
- [MINOR] Keep the KEDA CRDs on package uninstall (crds.additionalAnnotations
resource-policy: keep via the shim) so removing the keda package doesn't
cascade-delete every tenant ScaledObject.
- [MINOR] cozy-lib helper: reject a non-numeric bound (int "two" -> 0 -> a
scale-to-zero ScaledObject) and render an explicit pollingInterval/cooldownPeriod
of 0 (`if not kindIs invalid`, not `with`).
- [MINOR] Note the read-load fail-open: a broken metrics join reads as idle and
silently pins autoscaling at the floor under real load.
New/updated tests cover the non-quorum seed, the min>max guard, the corrected
kubectl message, the operator-down + Pending-intersected alerts, the CRD keep
policy, and the non-numeric-bound helper guard.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The database-autoscaling dashboard shipped from the keda package as a ConfigMap
labeled grafana_dashboard: "1" for a Grafana sidecar that cozystack does not run
(no k8s-sidecar anywhere; every dashboard is provisioned by grafana-operator from
the grafana-dashboards image via GrafanaDashboard CRs). It was enabled by default
and never appeared in any Grafana. Move the JSON to dashboards/db/database-autoscaling.json
and register db/database-autoscaling in packages/system/monitoring/dashboards.list,
exactly how db/cloudnativepg and the other managed-db dashboards ship, and drop the
keda ConfigMap, its default-on toggle, and its wrong-mechanism test.
Addresses IvanHunters [MAJOR] review feedback on #3954.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Address the branch-review nits:
- Fail closed when the monitoring namespace is unset. serverAddress builds
vmselect-shortterm.<ns-monitoring>.svc, and an empty ns-monitoring rendered the
malformed vmselect-shortterm..svc:8481, surfacing late as a KEDA scaler error.
The platform always injects it, but a hand-rendered/misconfigured tenant now
gets an actionable render-time failure mirroring the KEDA-CRD and target guards.
Tested.
- Document that the database-autoscaling PrometheusRule matches every KEDA HPA /
ScaledObject error (the underlying series carry no owning-app label), which is
correct only while database autoscaling is KEDA's sole consumer in cozystack;
note to scope the rules if a non-database ScaledObject is ever added.
- Correct the cozy-lib.keda.scaledObject usage example: `paused` should reflect
dry-run OR migration transition, not dry-run alone, so a second engine copying
the example does not drop the transition pause.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
KEDA's clusterDomain is a vendored-subchart value baked into the metrics-apiserver
gRPC target (keda-operator.<ns>.svc.<clusterDomain>:9666) and the operator's
serving-cert SANs at render time. A chart template cannot derive a subchart value
from _cluster, so the keda package hardcoded cozy.local — which silently breaks
(NXDOMAIN, every DB HPA FailedGetExternalMetric) on any cluster whose domain is
not the default. Inject the authoritative networking.clusterDomain into the shared
cozystack-values Secret under the keda subchart key so the value tracks the
platform config; the keda package keeps cozy.local as a standalone fallback. The
key is ignored by packages without a keda subchart (no app/extra schema sets root
additionalProperties:false). Add platform unittests asserting the Secret carries
keda.clusterDomain and that it tracks a non-default networking.clusterDomain.
Addresses the hardcoded-clusterDomain portability gap found in branch review.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The DatabaseAutoscalerScalerErrors rule queried keda_scaler_errors_total, which
KEDA 2.20 no longer exposes (verified against a live 2.20.2 operator: it emits
keda_scaled_object_errors_total for per-ScaledObject errors and
keda_scaler_detail_errors_total for per-scaler errors). The alert would have
silently never fired. Switch to keda_scaled_object_errors_total and add a
unittest assertion pinning the metric name so a rename regresses loudly.
Addresses CodeRabbit review feedback on #3954.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The vendored KEDA chart defaults clusterDomain to cluster.local, but
cozystack runs on cozy.local. KEDA's metrics-apiserver dials the operator
metrics gRPC service at keda-operator.<ns>.svc.<clusterDomain>:9666 and the
operator mints that service's serving-cert SANs from the same domain, so the
default resolves to NXDOMAIN and every HPA hangs on FailedGetExternalMetric.
Pin keda.clusterDomain=cozy.local.
Also raise the operator's client-go throttle to 50 QPS / 100 burst: at the
upstream 20/30 a single-replica operator loses its leader lease under
scale-subresource polling and crash-loops (the client rate limiter starves
the lease renewal). Both were observed live on a dev cluster; guard them in
the package unittest.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Add cozystack-owned observability for managed-database read-replica
autoscaling, rendered by the keda umbrella (not the vendored subchart):
- PrometheusRule (templates/alerts.yaml), scoped to the KEDA-generated
HPAs (keda-hpa-*): DatabaseAutoscalerScaleStuck (desired > current for
15m — stuck scale-up with no automatic rollback), DatabaseAutoscalerAtMax
(ScalingLimited=true for 30m — at max or quorum-raised ceiling), and
DatabaseAutoscalerScalerErrors (KEDA cannot evaluate the trigger — never
scale blind).
- A Grafana dashboard ConfigMap (grafana_dashboard sidecar label) with
desired-vs-current replicas, active read connections, replication lag,
and scaling-limited panels.
Both gated by databaseAutoscaling.alerts.enabled / .dashboard.enabled
(default true). Metric names are calibrated on a live cluster per the
proposal's PoC; the rule and panel shapes are what ships. Covered by helm
unittest (PrometheusRule with the three alerts + disable; dashboard
ConfigMap label + disable).
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Vendor the upstream KEDA chart (kedacore/keda 2.20.2) as a new optional
system package and wire it into the platform, so database read-replica
autoscaling (which renders KEDA ScaledObjects) has a backend.
- packages/system/keda: umbrella cozy-keda + vendored chart (Makefile
pins KEDA_CHART_VERSION with a lockstep assert), shim values enabling
the operator / metrics-server / webhook ServiceMonitors under the
platform Prometheus.
- core/platform: PackageSource cozystack.keda (namespace cozy-keda,
dependsOn networking + prometheus-operator-crds; cert-manager is not
needed, KEDA self-generates its webhook TLS) plus an optional bundle
entry, opt-in via bundles.enabledPackages. It is not a dependsOn of any
default package, so the platform HelmRelease never blocks on it.
- KEDA claims the cluster-wide external.metrics.k8s.io APIService
(distinct from metrics-server's metrics.k8s.io — no conflict).
Covered by helm unittest: platform bundle enable/disable/opt-out, the
PackageSource dependsOn, and the vendored chart's external-metrics
APIService + shim-enabled ServiceMonitor.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>