mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 12:13:55 +09:00
fix(backups): give the barman-cloud sidecar an explicit CPU limit (#5280)
## What this PR does The barman-cloud sidecar of Postgres backups is deliberately left without a CPU limit, so that a throttled sidecar slows WAL archiving down instead of failing it. In a tenant with `resourceQuotas` set that intent does not hold: `packages/apps/tenant/templates/quota.yaml` creates the `tenant-range-limits` LimitRange, whose default CPU limit is `250m`, and the sidecar receives it. Full backups and WAL segments are then gzip-compressed on a quarter of a core. On a production tenant the first full backup of a 1.6 GB database took 30 minutes, uploading at about 0.5 MB/s with the sidecar pinned at its 250m limit, while a small database on the same destination finished in 35 seconds. This sets an explicit `limits.cpu: "1"` on the sidecar, in the two places that build its configuration: `barmanSidecarConfiguration` in the backup driver (platform-managed ObjectStores) and `cozy-lib.barman.sidecarConfiguration` (ObjectStores rendered by the postgres and keycloak charts). This mirrors how the memory request and limit there already override the LimitRange's 128Mi default. One core removes the 250m ceiling without letting the sidecar starve the instance. Inside a tenant it counts against the `limits.cpu` quota, one core per instance pod; the comments now say so. The sidecar is injected when the instance pod is created, so existing instances pick the new limit up at their next restart. Tests: the Go test that asserted the absence of a CPU limit now asserts one core, and so does `packages/apps/postgres/tests/barman_resources_test.yaml`. Both fail with the implementation reverted and pass with it. `go test ./internal/backupcontroller/`, `helm unittest packages/apps/postgres` (220) and `helm unittest packages/system/keycloak` (121) pass. ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map against the diff (the backup driver, `cozy-lib`'s barman template, two tests): no app, value, schema, default, CRD, annotation or tooling change, so none of the listed repositories restates anything touched here. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(backups): the barman-cloud sidecar of Postgres backups now has an explicit one-core CPU limit instead of inheriting the tenant LimitRange's 250m default, which made full backups and WAL archiving several times slower. Each instance pod of a tenant with a CPU quota now counts 750m more against `limits.cpu`; check the headroom before instance pods are recreated. Existing ObjectStores pick the limit up at their next backup (platform-managed) or HelmRelease upgrade (chart-rendered), and running pods at their next restart. ```
This commit is contained in:
4 files changed
+21
-15
No files matched your search
@@ -2718,9 +2718,12 @@ func buildBarmanPlugin(objectStoreName, serverName string) cnpgtypes.PluginConfi
|
||||
// high-water mark on a 38 MB database is 254 MiB, so 128Mi cannot hold it. A
|
||||
// tenant that leaves resourceQuotas empty has no LimitRange, and there the
|
||||
// sidecar had no requests and no limit at all. 256Mi holds the measured working
|
||||
// set and 1Gi is four times the measurement. No CPU limit is set, matching the
|
||||
// entityOperator precedent in packages/apps/kafka: a throttled sidecar stalls
|
||||
// WAL archiving instead of failing it.
|
||||
// set and 1Gi is four times the measurement. The CPU limit is explicit for the
|
||||
// same reason: left unset, that LimitRange gives the sidecar its 250m default,
|
||||
// so gzip compression of full backups and WAL segments runs on a quarter of a
|
||||
// core (a 1.6 GB database took 30 minutes to back up). One core keeps the
|
||||
// sidecar from starving the instance while removing that ceiling; it counts
|
||||
// against the tenant's limits.cpu budget, one core per instance pod.
|
||||
//
|
||||
// Checksum: since botocore ~1.36 the default (when_supported) attaches a
|
||||
// flexible checksum to every PutObject, which non-AWS S3-compatible backends
|
||||
@@ -2740,6 +2743,7 @@ func barmanSidecarConfiguration() *cnpgtypes.InstanceSidecarConfiguration {
|
||||
corev1.ResourceMemory: resource.MustParse("256Mi"),
|
||||
},
|
||||
Limits: corev1.ResourceList{
|
||||
corev1.ResourceCPU: resource.MustParse("1"),
|
||||
corev1.ResourceMemory: resource.MustParse("1Gi"),
|
||||
},
|
||||
},
|
||||
|
||||
@@ -2156,8 +2156,11 @@ func TestApplyClusterPluginBackup_PatchesExistingCluster(t *testing.T) {
|
||||
sc.Resources.Requests.Cpu().MilliValue() != 100 {
|
||||
t.Errorf("ObjectStore leaves Barman subject to undersized tenant resource defaults: %+v", sc.Resources)
|
||||
}
|
||||
if _, ok := sc.Resources.Limits[corev1.ResourceCPU]; ok {
|
||||
t.Errorf("ObjectStore caps sidecar CPU, which throttles WAL archiving instead of failing it: %+v", sc.Resources.Limits)
|
||||
// Without an explicit CPU limit the tenant LimitRange caps the sidecar at
|
||||
// its 250m default, and every backup and WAL upload compresses on a
|
||||
// quarter of a core.
|
||||
if sc.Resources.Limits.Cpu().MilliValue() != 1000 {
|
||||
t.Errorf("ObjectStore leaves the sidecar CPU limit to the tenant LimitRange default: %+v", sc.Resources.Limits)
|
||||
}
|
||||
// The ObjectStore must be owner-referenced to the Cluster so Kubernetes GC
|
||||
// removes it when the Cluster is deleted (no orphan in the platform flow,
|
||||
|
||||
@@ -13,7 +13,7 @@ set:
|
||||
s3CredentialsSecret:
|
||||
name: postgres-backup
|
||||
tests:
|
||||
- it: sets resources for native backup without inheriting the tenant 128Mi limit
|
||||
- it: sets resources for native backup without inheriting the tenant 128Mi and 250m limits
|
||||
set:
|
||||
backup.enabled: true
|
||||
asserts:
|
||||
@@ -24,15 +24,11 @@ tests:
|
||||
cpu: 100m
|
||||
memory: 256Mi
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: 1Gi
|
||||
documentSelector:
|
||||
path: kind
|
||||
value: ObjectStore
|
||||
- notExists:
|
||||
path: spec.instanceSidecarConfiguration.resources.limits.cpu
|
||||
documentSelector:
|
||||
path: kind
|
||||
value: ObjectStore
|
||||
- equal:
|
||||
path: spec.configuration.s3Credentials.accessKeyId.name
|
||||
value: postgres-backup
|
||||
|
||||
@@ -13,10 +13,12 @@ LimitRange, which is cozy-keycloak and any tenant that leaves resourceQuotas
|
||||
empty, gave the sidecar no requests and no limit at all, so nothing reserved
|
||||
memory for it and nothing bounded it. The same request and limit settle both:
|
||||
256Mi holds the measured working set and 1Gi is four times the measurement.
|
||||
No CPU limit is set, matching the entityOperator precedent in
|
||||
packages/apps/kafka: a throttled sidecar stalls WAL archiving instead of
|
||||
failing it. Inside a tenant the limit counts against the ResourceQuota
|
||||
limits.memory budget, 1Gi per instance pod.
|
||||
The CPU limit is explicit for the same reason: left unset, that LimitRange
|
||||
gives the sidecar its 250m default, so gzip compression of full backups and
|
||||
WAL segments runs on a quarter of a core (a 1.6 GB database took 30 minutes
|
||||
to back up). One core keeps the sidecar from starving the instance while
|
||||
removing that ceiling. Inside a tenant both limits count against the
|
||||
ResourceQuota budget, one core and 1Gi per instance pod.
|
||||
|
||||
Checksum: since botocore ~1.36 (early 2025) the default
|
||||
RequestChecksumCalculation is when_supported, which attaches a flexible checksum
|
||||
@@ -39,6 +41,7 @@ instanceSidecarConfiguration:
|
||||
cpu: 100m
|
||||
memory: 256Mi
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: 1Gi
|
||||
env:
|
||||
- name: AWS_REQUEST_CHECKSUM_CALCULATION
|
||||
|
||||
Reference in new issue
Block a user