fix(seaweedfs, cozy-lib): cap the Go heap so components survive their memory limit (#4234)

<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Every SeaweedFS component is a Go binary running under a cgroup memory
limit, and none of them set `GOMEMLIMIT`. The collector then works from
`GOGC` alone, which is a target relative to the live heap with no
absolute ceiling, so a burst of allocation grows the heap past the limit
and the kernel kills the container.

This is not a leak. On the cluster where it showed up, the S3 gateway
held 6.4 GiB of `heap_sys` against an 8 GiB limit while `heap_inuse` was
389 MiB: the container was killed over memory the runtime was holding
and not using. Ten kills in seven days, each one an interruption of the
object store every tenant writes its backups to. Raising the limit does
not help, because nothing bounds the growth; it only moves the ceiling
the heap climbs to.

This PR adds a `cozy-lib.resources.goMemLimit` helper next to the
existing `javaHeap` one and wires it into the master, volume, filer and
s3 components and into every volume pool and zone, each sized from the
memory limit already computed for it. Pools need their own value because
the subchart merges `volume` into every entry of `volumes`: without one,
a `t1.nano` pool would inherit the value sized for the default volume's
512Mi limit, above its own 128Mi. The value is 75% of the limit, the
same fraction `javaHeap` uses for the JVM; the remaining quarter covers
what `GOMEMLIMIT` does not account for, goroutine stacks and mmap'd
regions among them.

The lifecycle admin and worker run the same binary under limits the
parent chart does not set (256Mi and 512Mi in
`packages/system/seaweedfs/values.yaml`), so their value is pinned next
to those limits at the same 75%: 192MiB and 384MiB.

A limit under 4/3 Mi (1398102 bytes) floors to `0MiB`, which Go accepts
and then collects almost continuously, so the helper fails the render
instead.

The upstream chart already supports `extraEnvironmentVars` on every one
of these components, so nothing had to change there. No values key is
added, so this needs no configuration and changes no public schema.

Tested with `helm unittest`: a suite on the seaweedfs package asserts
the value tracks the memory limit rather than a fixed string, for the
default volume, Simple pools, MultiZone zones and zone pools; the system
seaweedfs suite asserts the admin and worker env; and a helper suite in
`cozy-lib-tests` pins the unit grammar and the rejection of a zero
value. Reverting each fix makes its suite fail.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(seaweedfs): set GOMEMLIMIT on master, volume (and each pool and zone), filer, s3 and the lifecycle admin and worker so the Go runtime collects before the container hits its memory limit instead of being OOM-killed
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* SeaweedFS master, volume, pool, zonal volume, filer, and S3 workloads
now receive a Go memory limit set to 75% of their configured memory
limit.
* Pool workloads use their own resource settings when provided, falling
back to the default volume preset when needed.
* SeaweedFS admin and worker components now have Go memory limits
aligned with their configured memory limits.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
Aleksei Sviridkin authored and GitHub committed 2026-11-13 17:53:25 +03:00
commit df42104b13
8 files changed
+339 -7

No files matched your search

@@ -248,13 +248,19 @@ spec:
{{- /* An explicit null is dropped before schema validation, so neither
bound sees it, and the master exits on the empty flag it renders. */}}
volumeSizeLimitMB: {{ required "master.volumeSizeLimitMB must be set" .Values.master.volumeSizeLimitMB }}
resources: {{- include "cozy-lib.resources.defaultingSanitize" (list .Values.master.resourcesPreset .Values.master.resources $) | nindent 10 }}
{{- $masterResources := include "cozy-lib.resources.defaultingSanitize" (list .Values.master.resourcesPreset .Values.master.resources $) | fromYaml }}
resources: {{- toYaml $masterResources | nindent 10 }}
extraEnvironmentVars:
GOMEMLIMIT: {{ include "cozy-lib.resources.goMemLimit" $masterResources | quote }}
volume:
{{ if eq .Values.topology "MultiZone" }}
enabled: false
{{- end }}
replicas: {{ .Values.volume.replicas }}
resources: {{- include "cozy-lib.resources.defaultingSanitize" (list .Values.volume.resourcesPreset .Values.volume.resources $) | nindent 10 }}
{{- $volumeResources := include "cozy-lib.resources.defaultingSanitize" (list .Values.volume.resourcesPreset .Values.volume.resources $) | fromYaml }}
resources: {{- toYaml $volumeResources | nindent 10 }}
extraEnvironmentVars:
GOMEMLIMIT: {{ include "cozy-lib.resources.goMemLimit" $volumeResources | quote }}
dataDirs:
- name: data1
type: "persistentVolumeClaim"
@@ -273,7 +279,13 @@ spec:
{{- range $poolName, $pool := .Values.volume.pools }}
{{ $poolName }}:
replicas: {{ ternary $pool.replicas $.Values.volume.replicas (hasKey $pool "replicas") }}
resources: {{- include "cozy-lib.resources.defaultingSanitize" (list ($pool.resourcesPreset | default $.Values.volume.resourcesPreset) (default dict $pool.resources) $) | nindent 12 }}
{{- $poolResources := include "cozy-lib.resources.defaultingSanitize" (list ($pool.resourcesPreset | default $.Values.volume.resourcesPreset) (default dict $pool.resources) $) | fromYaml }}
resources: {{- toYaml $poolResources | nindent 12 }}
{{- /* The subchart merges volume into every entry of volumes, so a
pool that does not set its own value runs on the default
volume's, sized for a different limit. */}}
extraEnvironmentVars:
GOMEMLIMIT: {{ include "cozy-lib.resources.goMemLimit" $poolResources | quote }}
dataDirs:
- name: data1
type: "persistentVolumeClaim"
@@ -289,7 +301,9 @@ spec:
{{- range $zoneName, $zone := .Values.volume.zones }}
{{ $zoneName }}:
replicas: {{ ternary $zone.replicas $.Values.volume.replicas (hasKey $zone "replicas") }}
resources: {{- include "cozy-lib.resources.defaultingSanitize" (list $.Values.volume.resourcesPreset $.Values.volume.resources $) | nindent 12 }}
resources: {{- toYaml $volumeResources | nindent 12 }}
extraEnvironmentVars:
GOMEMLIMIT: {{ include "cozy-lib.resources.goMemLimit" $volumeResources | quote }}
dataDirs:
- name: data1
type: "persistentVolumeClaim"
@@ -314,7 +328,13 @@ spec:
{{- range $poolName, $pool := (dig "pools" dict $zone) }}
{{ $zoneName }}-{{ $poolName }}:
replicas: {{ ternary $pool.replicas (ternary $zone.replicas $.Values.volume.replicas (hasKey $zone "replicas")) (hasKey $pool "replicas") }}
resources: {{- include "cozy-lib.resources.defaultingSanitize" (list ($pool.resourcesPreset | default $.Values.volume.resourcesPreset) (default dict $pool.resources) $) | nindent 12 }}
{{- $poolResources := include "cozy-lib.resources.defaultingSanitize" (list ($pool.resourcesPreset | default $.Values.volume.resourcesPreset) (default dict $pool.resources) $) | fromYaml }}
resources: {{- toYaml $poolResources | nindent 12 }}
{{- /* The subchart merges volume into every entry of volumes, so a
pool that does not set its own value runs on the default
volume's, sized for a different limit. */}}
extraEnvironmentVars:
GOMEMLIMIT: {{ include "cozy-lib.resources.goMemLimit" $poolResources | quote }}
dataDirs:
- name: data1
type: "persistentVolumeClaim"
@@ -345,7 +365,10 @@ spec:
replicas: {{ .Values.filer.replicas }}
s3:
domainName: {{ .Values.host | default (printf "s3.%s" $host) }}
resources: {{- include "cozy-lib.resources.defaultingSanitize" (list .Values.filer.resourcesPreset .Values.filer.resources $) | nindent 10 }}
{{- $filerResources := include "cozy-lib.resources.defaultingSanitize" (list .Values.filer.resourcesPreset .Values.filer.resources $) | fromYaml }}
resources: {{- toYaml $filerResources | nindent 10 }}
extraEnvironmentVars:
GOMEMLIMIT: {{ include "cozy-lib.resources.goMemLimit" $filerResources | quote }}
s3:
replicas: {{ .Values.s3.replicas }}
ingress:
@@ -381,7 +404,10 @@ spec:
{{- if not $wildcardSecret }}
secretName: {{ .Release.Name }}-s3-ingress-tls
{{- end }}
resources: {{- include "cozy-lib.resources.defaultingSanitize" (list .Values.s3.resourcesPreset .Values.s3.resources $) | nindent 10 }}
{{- $s3Resources := include "cozy-lib.resources.defaultingSanitize" (list .Values.s3.resourcesPreset .Values.s3.resources $) | fromYaml }}
resources: {{- toYaml $s3Resources | nindent 10 }}
extraEnvironmentVars:
GOMEMLIMIT: {{ include "cozy-lib.resources.goMemLimit" $s3Resources | quote }}
admin:
enabled: {{ .Values.lifecycle.enabled }}
secret:
@@ -0,0 +1,162 @@
suite: seaweedfs components carry a GOMEMLIMIT derived from their memory limit
templates:
- templates/seaweedfs.yaml
# Every SeaweedFS component is a Go binary. Without GOMEMLIMIT the runtime
# collects on a relative target and keeps no absolute ceiling, so a burst of
# allocation grows the heap past the cgroup limit and the kernel kills the
# container. Each component sizes its own value from its own memory limit,
# so the assertions below track the limit rather than a fixed string.
tests:
- it: derives GOMEMLIMIT from the default preset on every component
documentSelector:
path: kind
value: HelmRelease
set:
topology: Simple
replicationFactor: 2
_namespace:
host: example.org
ingress: tenant-root
_cluster:
issuer-name: letsencrypt-prod
solver: http01
asserts:
- equal:
path: spec.values.seaweedfs.master.resources.limits.memory
value: 512Mi
- equal:
path: spec.values.seaweedfs.master.extraEnvironmentVars.GOMEMLIMIT
value: 384MiB
- equal:
path: spec.values.seaweedfs.volume.extraEnvironmentVars.GOMEMLIMIT
value: 384MiB
- equal:
path: spec.values.seaweedfs.filer.extraEnvironmentVars.GOMEMLIMIT
value: 384MiB
- equal:
path: spec.values.seaweedfs.s3.extraEnvironmentVars.GOMEMLIMIT
value: 384MiB
- it: follows the memory limit when a component is resized
documentSelector:
path: kind
value: HelmRelease
set:
topology: Simple
replicationFactor: 2
_namespace:
host: example.org
ingress: tenant-root
_cluster:
issuer-name: letsencrypt-prod
solver: http01
s3:
resources:
memory: 8Gi
volume:
resources:
memory: 2Gi
asserts:
- equal:
path: spec.values.seaweedfs.s3.resources.limits.memory
value: 8Gi
- equal:
path: spec.values.seaweedfs.s3.extraEnvironmentVars.GOMEMLIMIT
value: 6144MiB
- equal:
path: spec.values.seaweedfs.volume.extraEnvironmentVars.GOMEMLIMIT
value: 1536MiB
- equal:
path: spec.values.seaweedfs.master.extraEnvironmentVars.GOMEMLIMIT
value: 384MiB
- it: keeps GOMEMLIMIT below the cgroup limit it protects
documentSelector:
path: kind
value: HelmRelease
set:
topology: Simple
replicationFactor: 2
_namespace:
host: example.org
ingress: tenant-root
_cluster:
issuer-name: letsencrypt-prod
solver: http01
s3:
resources:
memory: 1Gi
asserts:
- equal:
path: spec.values.seaweedfs.s3.resources.limits.memory
value: 1Gi
- equal:
path: spec.values.seaweedfs.s3.extraEnvironmentVars.GOMEMLIMIT
value: 768MiB
# The subchart merges volume into every entry of volumes, so a pool that
# did not carry its own value would run on the default volume's, sized for
# a larger limit than its own.
- it: sizes a Simple pool from its own limit, not the default volume's
documentSelector:
path: kind
value: HelmRelease
set:
topology: Simple
replicationFactor: 2
_namespace:
host: example.org
ingress: tenant-root
_cluster:
issuer-name: letsencrypt-prod
solver: http01
volume:
pools:
small:
diskType: ssd
resourcesPreset: t1.nano
asserts:
- equal:
path: spec.values.seaweedfs.volumes.small.resources.limits.memory
value: 128Mi
- equal:
path: spec.values.seaweedfs.volumes.small.extraEnvironmentVars.GOMEMLIMIT
value: 96MiB
- equal:
path: spec.values.seaweedfs.volume.extraEnvironmentVars.GOMEMLIMIT
value: 384MiB
- it: sizes zones and zone pools from their own limits in MultiZone
documentSelector:
path: kind
value: HelmRelease
set:
topology: MultiZone
replicationFactor: 1
_namespace:
host: example.org
ingress: tenant-root
_cluster:
issuer-name: letsencrypt-prod
solver: http01
volume:
resources:
memory: 2Gi
zones:
a:
pools:
small:
diskType: ssd
resourcesPreset: t1.nano
asserts:
- equal:
path: spec.values.seaweedfs.volumes.a.extraEnvironmentVars.GOMEMLIMIT
value: 1536MiB
- equal:
path: spec.values.seaweedfs.volumes.a-small.resources.limits.memory
value: 128Mi
- equal:
path: spec.values.seaweedfs.volumes.a-small.extraEnvironmentVars.GOMEMLIMIT
value: 96MiB
@@ -202,6 +202,26 @@
{{- printf `-Xms%dm -Xmx%dm` $xmsMi $xmxMi }}
{{- end }}
{{- /*
goMemLimit takes a sanitized .Values.resources and returns a GOMEMLIMIT
value set to 75% of the memory limit. Without it the Go runtime collects
on a relative target (GOGC) with no absolute ceiling, so a burst of
allocation grows the heap past the cgroup limit and the kernel kills the
container. The 25% headroom covers what GOMEMLIMIT does not account for:
goroutine stacks and any mmap'd region. Accepts only sanitized resource
maps.
*/}}
{{- define "cozy-lib.resources.goMemLimit" }}
{{- $memoryLimitInt := include "cozy-lib.resources.toFloat" .limits.memory | float64 | int64 }}
{{- /* 4194304 is 4Mi */}}
{{- $limitMiB := div (mul $memoryLimitInt 3) 4194304 }}
{{- /* Go accepts 0, and then collects almost without pause. */}}
{{- if lt $limitMiB 1 }}
{{- fail (printf "ERROR: a memory limit of %s leaves no room for a Go heap; GOMEMLIMIT needs a limit of at least 1398102 bytes (4/3 Mi) to come out at 1MiB." .limits.memory) }}
{{- end }}
{{- printf `%dMiB` $limitMiB }}
{{- end }}
{{- define "cozy-lib.resources.flatten" -}}
{{- $out := dict -}}
{{- $res := include "cozy-lib.resources.sanitize" . | fromYaml -}}
@@ -57,6 +57,36 @@ tests:
path: spec.template.spec.containers[0].command[2]
pattern: '-admin=seaweedfs-admin\.tenant-root:23646\.33646 '
# Both are the weed binary under a fixed memory limit the parent chart does
# not set, so their GOMEMLIMIT is pinned here rather than derived: without
# one the heap has no ceiling and a large lifecycle pass grows it past the
# limit.
- it: caps the admin heap below its memory limit
template: charts/seaweedfs/templates/admin/admin-statefulset.yaml
set: *enabled
asserts:
- equal:
path: spec.template.spec.containers[0].resources.limits.memory
value: 256Mi
- contains:
path: spec.template.spec.containers[0].env
content:
name: GOMEMLIMIT
value: 192MiB
- it: caps the worker heap below its memory limit
template: charts/seaweedfs/templates/worker/worker-deployment.yaml
set: *enabled
asserts:
- equal:
path: spec.template.spec.containers[0].resources.limits.memory
value: 512Mi
- contains:
path: spec.template.spec.containers[0].env
content:
name: GOMEMLIMIT
value: 384MiB
# The worker executes every job type it registers for. Lifecycle is the only
# one this release asks for: vacuum, balance and erasure coding move data and
# are not something enabling lifecycle should start doing as a side effect.
+8
View File
@@ -349,6 +349,11 @@ seaweedfs:
limits:
cpu: "500m"
memory: "256Mi"
# 75% of the limit above, as cozy-lib.resources.goMemLimit derives it for
# the components whose resources the parent chart sets. Without it the
# heap has no absolute ceiling and grows past the limit.
extraEnvironmentVars:
GOMEMLIMIT: "192MiB"
worker:
enabled: false
# Vacuum, balance and erasure coding move data; enabling lifecycle must
@@ -378,6 +383,9 @@ seaweedfs:
limits:
cpu: "1"
memory: "512Mi"
# 75% of the limit above; see admin.extraEnvironmentVars.
extraEnvironmentVars:
GOMEMLIMIT: "384MiB"
certificates:
commonName: "SeaweedFS CA"
ipAddresses: []
@@ -0,0 +1,8 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ .Release.Name }}-gomemlimit
data:
{{- range $name, $resources := .Values.sanitizedResources }}
{{ $name }}: {{ include "cozy-lib.resources.goMemLimit" $resources | quote }}
{{- end }}
@@ -0,0 +1,62 @@
# ./tests/resources_gomemlimit_test.yaml
suite: resources goMemLimit helper
# The helper turns a memory limit into a GOMEMLIMIT the Go runtime accepts,
# at 75% of the limit. Two properties matter and neither is visible from a
# single value: the result carries the MiB suffix Go parses, and it stays
# strictly under the limit it protects, whatever unit the quantity came in.
# A value that came out at or above the limit would leave the runtime with
# no headroom and the container would still be killed.
templates:
- templates/tests/resources-gomemlimit.yaml
tests:
- it: takes 75% of the limit and keeps the unit Go parses
values:
- resources_gomemlimit_values.yaml
release:
name: myrelease
namespace: default
asserts:
- equal:
path: data.binaryGi
value: "6144MiB"
- equal:
path: data.binaryMi
value: "384MiB"
# A decimal quantity is smaller than its binary lookalike: 1G is
# 1000000000 bytes, so 75% of it lands below 768MiB.
- equal:
path: data.decimalG
value: "715MiB"
- equal:
path: data.bareBytes
value: "768MiB"
# The smallest limit that still yields a usable value: 75% of
# 1398102 bytes (4/3 Mi, rounded up) floors to 1MiB.
- equal:
path: data.smallestUsable
value: "1MiB"
# Integer division floors, so a limit one byte under 4/3 Mi comes out
# at 0MiB. Go accepts that and then collects almost without pause,
# slowing the component instead of protecting it, so the render stops
# instead.
- it: refuses a limit too small to leave a Go heap
values:
- resources_gomemlimit_values.yaml
set:
sanitizedResources:
belowOneMiB:
limits:
memory: "1398101"
asserts:
- failedTemplate:
errorPattern: "leaves no room for a Go heap"
@@ -0,0 +1,16 @@
sanitizedResources:
binaryGi:
limits:
memory: "8Gi"
binaryMi:
limits:
memory: "512Mi"
decimalG:
limits:
memory: "1G"
bareBytes:
limits:
memory: "1073741824"
smallestUsable:
limits:
memory: "1398102"