LastFailedArchiveTime compares the last failed archival with the last
successful one, so it resolves as soon as any segment gets through. A
cluster whose archiving fails intermittently, or runs slower than it
writes WAL, clears that alert every time one upload succeeds while the
queue of segments waiting on the Postgres volume keeps growing, until
the volume is full and the database stops. That alert is also only a
warning.
A healthy archiver empties archive_status within seconds of each WAL
switch. A queue that has not been empty once in 30 minutes means point
in time recovery is falling behind whatever the archiver is reporting,
so that is what pages, as critical. Idle clusters and replicas have an
empty queue and stay quiet.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Bump the vendored cloudnative-pg chart 0.27.1 -> 0.29.0 (operator + CRDs
1.28.1-patched-to-1.28.2 -> 1.30.0) and drop the hand-maintained
cloudnative-pg-1.28.2.patch entirely — 0.29.0 is a published chart, so no
version-alignment patch is needed.
Why this is required
--------------------
Database read-replica autoscaling (community proposal
design-proposals/database-horizontal-autoscaling) drives the CNPG Cluster
scale subresource with a stock HPA (via KEDA). That only works when the
subresource serves status.selector, which cloudnative-pg#8996 added and
shipped in CNPG 1.28.4 / 1.29.2 / 1.30.0. cozystack shipped 1.28.2, whose
scale subresource has no selector, so the HPA fails with InvalidSelector.
Why 1.30.0 and not 1.28.4
-------------------------
The floor is >= 1.28.4, but the upstream chart repo publishes no chart for
1.28.4 (the 1.28 line stops at chart 0.27.1 / appVersion 1.28.1), so 1.28.4
could only be reached by hand-patching the 0.27.1 chart's operator image and
CRDs up to the backport. A live PoC on a dev cluster proved that path is
broken: the 1.28.4 operator running under the 0.27.1 (1.28.1) chart
templates crash-loops on a missing RBAC grant it needs but the older chart
does not provide —
endpointslices.discovery.k8s.io is forbidden: ... cannot list resource
"endpointslices"
— so status.selector is never populated and the HPA still cannot scale. The
hand-patch would have to port the operator's RBAC (and any other template
delta) from 1.28.1 to 1.28.4 by hand, with no published chart to diff
against. 1.30.0 (chart 0.29.0) is the only published chart that carries the
selector together with matching RBAC/CRDs/templates, and the 1.28 line is
EOL (2026-06-30), so 1.30.0 also keeps us on a supported release. cozystack
uses the barman-cloud plugin (not native barman, removed in 1.31) and no
CNPG Poolers, so the 1.30 deprecations and the known pooler bug do not
apply.
Update the version-guard unittest to 1.30.0 and assert the clusters CRD
scale subresource carries labelSelectorPath: .status.selector.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Raises the CloudNativePG operator image and its CRDs together to 1.28.2, which carries the PVC resize-deadlock fix (cloudnative-pg#9980 / #9981): after a simultaneous resources+size change the operator deletes the sole primary Pod, leaves the PVC in the resizing class, and never recreates the Pod, wedging the cluster.
Upstream publishes a chart only per minor .0/.1, so there is no 1.28.2 chart on the 1.28 line. Following #3526/#3528, patches/cloudnative-pg-1.28.2.patch (applied by make update) raises the vendored chart's appVersion and CRDs from 1.28.1 to 1.28.2 in lockstep; the operator image follows appVersion, so no image.tag pin. Verified #3479-safe: no CRD status field changes between 1.28.1 and 1.28.2 (only the extensions spec grows). Backportable to release-1.6.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Address review feedback from gemini-code-assist on
packages/system/postgres-operator/templates/webhook-ready-hook.yaml:115.
The wait container had no resource requests or limits, so schedulers
treated it as BestEffort and downstream quota enforcement had no signal.
Set small requests (10m CPU, 32Mi memory) and conservative limits
(100m CPU, 64Mi memory) matching the actual footprint of the kubectl
polling loop.
Add a matching unittest assertion so the values stay in sync if anyone
touches the template.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on
packages/system/postgres-operator/templates/webhook-ready-hook.yaml:83.
The Job targets the cnpg-webhook-service/services/proxy subresource, which
is namespaced and lives in the release namespace. A namespaced Role and
RoleBinding grant the exact permission needed without creating global
RBAC for a namespaced probe, which is the principle of least privilege.
Also update the kind assertions in tests/webhook-ready-hook_test.yaml so
the unittest suite tracks the new Role/RoleBinding objects.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The cnpg admission webhook's controller pod passes readinessProbe as soon
as the local HTTPS server binds to :9443. The HelmRelease marks itself
Ready right after that via helm install --wait. But the Service
cnpg-webhook-service needs its EndpointSlice populated and the data plane
(kube-proxy / cilium) programmed before kube-apiserver can reach the
webhook through the Service ClusterIP. That gap is short but not zero,
and any HelmRelease that depends on postgres-operator (cozy-keycloak,
tenant Postgres apps) can fire its own install inside the window and hit
Internal error occurred: failed calling webhook "mcluster.cnpg.io":
failed to call webhook: Post "https://cnpg-webhook-service.cozy-postgres-operator.svc:443/...":
dial tcp <svc-ip>:443: connect: connection refused
which fails the install of the downstream release. Seen on cozystack/cozystack#2470
E2E run 24862782568.
Add a post-install,post-upgrade Helm hook Job that blocks the release
from reporting Ready until the webhook answers /readyz through the
apiserver service proxy. Apiserver proxy routes the call over the same
Service IP → EndpointSlice → pod path the admission webhook uses, so
once it responds, the webhook admission path is also working.
RBAC is minimal: a dedicated ServiceAccount with a ClusterRole that only
grants get on services/proxy scoped to https:cnpg-webhook-service:webhook-server.
The Job times out after 120s with 60 attempts at 2s intervals — longer
than any data-plane programming delay seen on E2E, but bounded.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Alexey Artamonov