The per-pod subquery stepped once a minute and took the latest sample
at each step, so with a 30-second scrape it looked at half the samples.
A queue that emptied only in a skipped sample did not count, and the
rule could page on a cluster that was keeping up. While a recreated pod
overlapped its old series, taking the maximum also hid the new
series' zero. The minimum of each raw series, then the minimum per pod
name, sees every sample and still keeps the window across a new
instance label.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The scraped series carries the pod IP in its instance label, so a
recreated primary starts a new series, and min_over_time over it saw
only the samples since the restart. The rule could then page ten
minutes after the pod came back with a queue that was still draining,
not after the 30 minutes its description promises. Grouping by pod name
first keeps one series per pod across the restart, and the description
now says what happens to a pod name that is newer than the window.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
LastFailedArchiveTime compares the last failed archival with the last
successful one, so it resolves as soon as any segment gets through. A
cluster whose archiving fails intermittently, or runs slower than it
writes WAL, clears that alert every time one upload succeeds while the
queue of segments waiting on the Postgres volume keeps growing, until
the volume is full and the database stops. That alert is also only a
warning.
A healthy archiver empties archive_status within seconds of each WAL
switch. A queue that has not been empty once in 30 minutes means point
in time recovery is falling behind whatever the archiver is reporting,
so that is what pages, as critical. Idle clusters and replicas have an
empty queue and stay quiet.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Reword the values.yaml comment so it does not call appVersion 1.30.0 a
"published chart": the published chart version is 0.29.0 and it carries
appVersion 1.30.0. Keeping the two distinct stops a maintainer from running
`helm pull --version 1.30.0`.
Addresses CodeRabbit review feedback on #3951.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The explanatory comment in values.yaml still described the removed
patches/cloudnative-pg-1.28.2.patch mechanism and the old 0.27.1/1.28.1
versions, and pointed at an "edit the patch" workflow that no longer exists.
Rewrite it to describe the native 0.29.0/1.30.0 chart bump (operator and CRDs
aligned by construction, no patch step) while preserving the still-valid
guidance: do not pin image.tag, keep the operator and CRDs in lockstep, since a
mismatched pin prunes CRD status fields and fails backups cluster-wide.
Addresses review feedback on #3951 (IvanHunters, CodeRabbit).
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Bump the vendored cloudnative-pg chart 0.27.1 -> 0.29.0 (operator + CRDs
1.28.1-patched-to-1.28.2 -> 1.30.0) and drop the hand-maintained
cloudnative-pg-1.28.2.patch entirely — 0.29.0 is a published chart, so no
version-alignment patch is needed.
Why this is required
--------------------
Database read-replica autoscaling (community proposal
design-proposals/database-horizontal-autoscaling) drives the CNPG Cluster
scale subresource with a stock HPA (via KEDA). That only works when the
subresource serves status.selector, which cloudnative-pg#8996 added and
shipped in CNPG 1.28.4 / 1.29.2 / 1.30.0. cozystack shipped 1.28.2, whose
scale subresource has no selector, so the HPA fails with InvalidSelector.
Why 1.30.0 and not 1.28.4
-------------------------
The floor is >= 1.28.4, but the upstream chart repo publishes no chart for
1.28.4 (the 1.28 line stops at chart 0.27.1 / appVersion 1.28.1), so 1.28.4
could only be reached by hand-patching the 0.27.1 chart's operator image and
CRDs up to the backport. A live PoC on a dev cluster proved that path is
broken: the 1.28.4 operator running under the 0.27.1 (1.28.1) chart
templates crash-loops on a missing RBAC grant it needs but the older chart
does not provide —
endpointslices.discovery.k8s.io is forbidden: ... cannot list resource
"endpointslices"
— so status.selector is never populated and the HPA still cannot scale. The
hand-patch would have to port the operator's RBAC (and any other template
delta) from 1.28.1 to 1.28.4 by hand, with no published chart to diff
against. 1.30.0 (chart 0.29.0) is the only published chart that carries the
selector together with matching RBAC/CRDs/templates, and the 1.28 line is
EOL (2026-06-30), so 1.30.0 also keeps us on a supported release. cozystack
uses the barman-cloud plugin (not native barman, removed in 1.31) and no
CNPG Poolers, so the 1.30 deprecations and the known pooler bug do not
apply.
Update the version-guard unittest to 1.30.0 and assert the clusters CRD
scale subresource carries labelSelectorPath: .status.selector.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Raises the CloudNativePG operator image and its CRDs together to 1.28.2, which carries the PVC resize-deadlock fix (cloudnative-pg#9980 / #9981): after a simultaneous resources+size change the operator deletes the sole primary Pod, leaves the PVC in the resizing class, and never recreates the Pod, wedging the cluster.
Upstream publishes a chart only per minor .0/.1, so there is no 1.28.2 chart on the 1.28 line. Following #3526/#3528, patches/cloudnative-pg-1.28.2.patch (applied by make update) raises the vendored chart's appVersion and CRDs from 1.28.1 to 1.28.2 in lockstep; the operator image follows appVersion, so no image.tag pin. Verified #3479-safe: no CRD status field changes between 1.28.1 and 1.28.2 (only the extensions spec grows). Backportable to release-1.6.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Address review feedback from coderabbitai on
packages/system/postgres-operator/values.yaml:30.
The old comment's 5-minute ImagePullBackOff scenario conflicted with
the 180s activeDeadlineSeconds that the default maxAttempts/sleepSeconds
resolve to, so the numbers could not both be taken at face value.
Rewrite the comment to state the actual deadline math and frame the two
gates as an AND with activeDeadlineSeconds being the shorter one under
defaults, so readers understand why backoffLimit has little headroom
without an accompanying maxAttempts/sleepSeconds bump.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on
packages/system/postgres-operator/templates/webhook-ready-hook.yaml:115.
The wait container had no resource requests or limits, so schedulers
treated it as BestEffort and downstream quota enforcement had no signal.
Set small requests (10m CPU, 32Mi memory) and conservative limits
(100m CPU, 64Mi memory) matching the actual footprint of the kubectl
polling loop.
Add a matching unittest assertion so the values stay in sync if anyone
touches the template.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on
packages/system/postgres-operator/templates/webhook-ready-hook.yaml:83.
The Job targets the cnpg-webhook-service/services/proxy subresource, which
is namespaced and lives in the release namespace. A namespaced Role and
RoleBinding grant the exact permission needed without creating global
RBAC for a namespaced probe, which is the principle of least privilege.
Also update the kind assertions in tests/webhook-ready-hook_test.yaml so
the unittest suite tracks the new Role/RoleBinding objects.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The cnpg admission webhook's controller pod passes readinessProbe as soon
as the local HTTPS server binds to :9443. The HelmRelease marks itself
Ready right after that via helm install --wait. But the Service
cnpg-webhook-service needs its EndpointSlice populated and the data plane
(kube-proxy / cilium) programmed before kube-apiserver can reach the
webhook through the Service ClusterIP. That gap is short but not zero,
and any HelmRelease that depends on postgres-operator (cozy-keycloak,
tenant Postgres apps) can fire its own install inside the window and hit
Internal error occurred: failed calling webhook "mcluster.cnpg.io":
failed to call webhook: Post "https://cnpg-webhook-service.cozy-postgres-operator.svc:443/...":
dial tcp <svc-ip>:443: connect: connection refused
which fails the install of the downstream release. Seen on cozystack/cozystack#2470
E2E run 24862782568.
Add a post-install,post-upgrade Helm hook Job that blocks the release
from reporting Ready until the webhook answers /readyz through the
apiserver service proxy. Apiserver proxy routes the call over the same
Service IP → EndpointSlice → pod path the admission webhook uses, so
once it responds, the webhook admission path is also working.
RBAC is minimal: a dedicated ServiceAccount with a ClusterRole that only
grants get on services/proxy scoped to https:cnpg-webhook-service:webhook-server.
The Job times out after 120s with 60 attempts at 2s intervals — longer
than any data-plane programming delay seen on E2E, but bounded.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Remove extra closing parenthesis in the CNPGClusterOffline alert expression
that causes vmalert pods to crash with "bad prometheus expr" error.
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
Move common-envs.mk and package.mk from scripts/ to hack/ directory.
Update all Makefile includes to use new paths. Remove unused
issue-flux-certificates.sh script.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced CloudNativePG Operator configuration with new options for
cluster-wide monitoring and namespace control
- Added support for IP family configuration in service settings
- Increased flexibility for concurrent reconciliation processes
- **Version Updates**
- Upgraded CloudNativePG Operator from version 1.24.0 to 1.25.0
- Updated Helm chart version from 0.22.0 to 0.23.0
- **Configuration Improvements**
- Introduced new options for namespace override and cluster-wide event
observation
- Added maximum concurrent reconciles setting
- Expanded service networking configuration capabilities
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
fix regression introduced in
https://github.com/aenix-io/cozystack/pull/558
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Refactor**
- Updated label formatting in PostgreSQL operator default alerts
configuration
- Enhanced alert template generation to dynamically include multiple
alert configurations from separate files
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added comprehensive monitoring and alerting rules for PostgreSQL
instances.
- Introduced alerts for:
- Long-running transactions
- Backend waiting times
- Transaction ID age
- Replication lag
- Archiving failures
- Deadlock conflicts
- Replication status
- New resource: `PrometheusRule` named `cnpg-default-alerts`.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This change is aimed at improving the development experience.
- The option `make delete` has been added.
- Added check for `NAME` and `NAMESPACE` variables
- Now, any package (not just system ones) can include options such as
make show, make diff, make apply.
- Applications from packages/extra require explicit specification of the
`NAMESPACE`.
- Applications from packages/apps require explicit specification of both
`NAME` and `NAMESPACE`.
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Alexey Artamonov