fix(backups): give the barman-cloud sidecar an explicit CPU limit (#5280)

## What this PR does

The barman-cloud sidecar of Postgres backups is deliberately left
without a CPU limit, so that a throttled sidecar slows WAL archiving
down instead of failing it. In a tenant with `resourceQuotas` set that
intent does not hold: `packages/apps/tenant/templates/quota.yaml`
creates the `tenant-range-limits` LimitRange, whose default CPU limit is
`250m`, and the sidecar receives it. Full backups and WAL segments are
then gzip-compressed on a quarter of a core. On a production tenant the
first full backup of a 1.6 GB database took 30 minutes, uploading at
about 0.5 MB/s with the sidecar pinned at its 250m limit, while a small
database on the same destination finished in 35 seconds.

This sets an explicit `limits.cpu: "1"` on the sidecar, in the two
places that build its configuration: `barmanSidecarConfiguration` in the
backup driver (platform-managed ObjectStores) and
`cozy-lib.barman.sidecarConfiguration` (ObjectStores rendered by the
postgres and keycloak charts). This mirrors how the memory request and
limit there already override the LimitRange's 128Mi default. One core
removes the 250m ceiling without letting the sidecar starve the
instance. Inside a tenant it counts against the `limits.cpu` quota, one
core per instance pod; the comments now say so.

The sidecar is injected when the instance pod is created, so existing
instances pick the new limit up at their next restart.

Tests: the Go test that asserted the absence of a CPU limit now asserts
one core, and so does
`packages/apps/postgres/tests/barman_resources_test.yaml`. Both fail
with the implementation reverted and pass with it. `go test
./internal/backupcontroller/`, `helm unittest packages/apps/postgres`
(220) and `helm unittest packages/system/keycloak` (121) pass.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff (the backup driver, `cozy-lib`'s
barman template, two tests): no app, value, schema, default, CRD,
annotation or tooling change, so none of the listed repositories
restates anything touched here.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(backups): the barman-cloud sidecar of Postgres backups now has an explicit one-core CPU limit instead of inheriting the tenant LimitRange's 250m default, which made full backups and WAL archiving several times slower. Each instance pod of a tenant with a CPU quota now counts 750m more against `limits.cpu`; check the headroom before instance pods are recreated. Existing ObjectStores pick the limit up at their next backup (platform-managed) or HelmRelease upgrade (chart-rendered), and running pods at their next restart.
```
This commit is contained in:
Aleksei Sviridkin authored and GitHub committed 2026-11-13 14:54:51 +03:00
commit 7c46e1e384
4 files changed
+21 -15

No files matched your search

@@ -2718,9 +2718,12 @@ func buildBarmanPlugin(objectStoreName, serverName string) cnpgtypes.PluginConfi
// high-water mark on a 38 MB database is 254 MiB, so 128Mi cannot hold it. A
// tenant that leaves resourceQuotas empty has no LimitRange, and there the
// sidecar had no requests and no limit at all. 256Mi holds the measured working
// set and 1Gi is four times the measurement. No CPU limit is set, matching the
// entityOperator precedent in packages/apps/kafka: a throttled sidecar stalls
// WAL archiving instead of failing it.
// set and 1Gi is four times the measurement. The CPU limit is explicit for the
// same reason: left unset, that LimitRange gives the sidecar its 250m default,
// so gzip compression of full backups and WAL segments runs on a quarter of a
// core (a 1.6 GB database took 30 minutes to back up). One core keeps the
// sidecar from starving the instance while removing that ceiling; it counts
// against the tenant's limits.cpu budget, one core per instance pod.
//
// Checksum: since botocore ~1.36 the default (when_supported) attaches a
// flexible checksum to every PutObject, which non-AWS S3-compatible backends
@@ -2740,6 +2743,7 @@ func barmanSidecarConfiguration() *cnpgtypes.InstanceSidecarConfiguration {
corev1.ResourceMemory: resource.MustParse("256Mi"),
},
Limits: corev1.ResourceList{
corev1.ResourceCPU: resource.MustParse("1"),
corev1.ResourceMemory: resource.MustParse("1Gi"),
},
},
@@ -2156,8 +2156,11 @@ func TestApplyClusterPluginBackup_PatchesExistingCluster(t *testing.T) {
sc.Resources.Requests.Cpu().MilliValue() != 100 {
t.Errorf("ObjectStore leaves Barman subject to undersized tenant resource defaults: %+v", sc.Resources)
}
if _, ok := sc.Resources.Limits[corev1.ResourceCPU]; ok {
t.Errorf("ObjectStore caps sidecar CPU, which throttles WAL archiving instead of failing it: %+v", sc.Resources.Limits)
// Without an explicit CPU limit the tenant LimitRange caps the sidecar at
// its 250m default, and every backup and WAL upload compresses on a
// quarter of a core.
if sc.Resources.Limits.Cpu().MilliValue() != 1000 {
t.Errorf("ObjectStore leaves the sidecar CPU limit to the tenant LimitRange default: %+v", sc.Resources.Limits)
}
// The ObjectStore must be owner-referenced to the Cluster so Kubernetes GC
// removes it when the Cluster is deleted (no orphan in the platform flow,
@@ -13,7 +13,7 @@ set:
s3CredentialsSecret:
name: postgres-backup
tests:
- it: sets resources for native backup without inheriting the tenant 128Mi limit
- it: sets resources for native backup without inheriting the tenant 128Mi and 250m limits
set:
backup.enabled: true
asserts:
@@ -24,15 +24,11 @@ tests:
cpu: 100m
memory: 256Mi
limits:
cpu: "1"
memory: 1Gi
documentSelector:
path: kind
value: ObjectStore
- notExists:
path: spec.instanceSidecarConfiguration.resources.limits.cpu
documentSelector:
path: kind
value: ObjectStore
- equal:
path: spec.configuration.s3Credentials.accessKeyId.name
value: postgres-backup
@@ -13,10 +13,12 @@ LimitRange, which is cozy-keycloak and any tenant that leaves resourceQuotas
empty, gave the sidecar no requests and no limit at all, so nothing reserved
memory for it and nothing bounded it. The same request and limit settle both:
256Mi holds the measured working set and 1Gi is four times the measurement.
No CPU limit is set, matching the entityOperator precedent in
packages/apps/kafka: a throttled sidecar stalls WAL archiving instead of
failing it. Inside a tenant the limit counts against the ResourceQuota
limits.memory budget, 1Gi per instance pod.
The CPU limit is explicit for the same reason: left unset, that LimitRange
gives the sidecar its 250m default, so gzip compression of full backups and
WAL segments runs on a quarter of a core (a 1.6 GB database took 30 minutes
to back up). One core keeps the sidecar from starving the instance while
removing that ceiling. Inside a tenant both limits count against the
ResourceQuota budget, one core and 1Gi per instance pod.
Checksum: since botocore ~1.36 (early 2025) the default
RequestChecksumCalculation is when_supported, which attaches a flexible checksum
@@ -39,6 +41,7 @@ instanceSidecarConfiguration:
cpu: 100m
memory: 256Mi
limits:
cpu: "1"
memory: 1Gi
env:
- name: AWS_REQUEST_CHECKSUM_CALCULATION