Files
cozystack/packages/system/metallb/values.yaml
T
Myasnikov Daniil c9ba7a3cca fix(platform): give node DaemonSets memory requests and limits
The Talos userspace OOM handler (v1.12+) ranks cgroups with

  memory_max.hasValue() ? 0.0 :
    {Besteffort: 1.0, Burstable: 0.5, ...}[class] * memory_current

and discards every cgroup scoring zero, so only a memory limit takes a
pod out of the victim set. Selection is decoupled from the trigger: a
tenant workload thrashing against its own limit drives node memory PSI,
and a limit-free system DaemonSet is killed for it. metallb-speaker and
linstor-satellite were the observed victims.

Set requests and limits on the node DaemonSets that had none. Requests
come from 14d peak working set measured on a 3-node dev cluster; limits
are deliberately loose, sized as a runaway backstop rather than a fitted
ceiling, because several of these scale with cluster size that a dev
cluster understates and because undersizing converts a rare
pressure-driven kill into a deterministic one. A limit costs nothing
until it is hit -- only the request reserves.

memory.max lands on the POD cgroup only when EVERY container in the pod
carries a limit, so sidecars and init containers are covered too; one
limit-free container returns the whole pod to the victim set.

Two DaemonSets are not chart-rendered and needed their own seam:
linstor-satellite via the LinstorSatelliteConfiguration podTemplate, and
virt-handler via KubeVirt customizeComponents.patches, since virt-operator
owns that DaemonSet. Note linstor-controller stays deliberately limit-free
-- it is a JVM that must not be OOM-killed mid-operation, and cluster_test
pins that.

Two things the multus manifest needed beyond the values change.
customize-deployment.patch is the source for that template, so the
limits are carried there rather than only in the generated file, which
`make update` would discard. And the reason kube-multus had no memory
limit was that this patch DELETED upstream's 50Mi rather than raising
it -- 50Mi being well under the 230Mi measured peak -- which is what
removed memory.max from the pod cgroup in the first place.

Not covered, because upstream hardcodes the containers with no resources
knob: the four frr-k8s cp-* init containers, the kube-ovn hostpath-init
and install-cni init containers, and cozy-proxy, whose vendored chart
exposes no resources value at all. hami and kilo are left alone as
optional components with no usage measurements behind them. All of these
still get a limit from the default LimitRange the operator maintains in
every system namespace.

Every limit is pinned by a helm-unittest assertion, and every assertion
was mutation-tested: changing the rendered value turns its suite red.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-13 16:47:52 +05:00

125 lines
5.6 KiB
YAML

# Notes for operators on bumps that crossed v0.15.x → v0.16.x:
#
# - BGP backend: upstream defaults flipped from FRR mode to FRR-K8s. This chart
# ships frr-k8s as a sibling DaemonSet (cozy-metallb/metallb-frr-k8s). Speaker
# talks to it via in-cluster API; existing BGPPeer / BGPAdvertisement objects
# keep working unchanged. To revert speaker to legacy FRR mode (deprecated
# upstream), set BOTH of these here:
# metallb.frrk8s.enabled: false
# metallb.speaker.frr.enabled: true
# This switches the speaker DaemonSet to the four-container layout (speaker,
# frr, reloader, frr-metrics) and METALLB_BGP_TYPE=frr. The frr-k8s subchart
# stays installed but idle — Helm evaluates a `condition:` on a dep against
# the declaring chart's own values context (here: the vendored metallb
# chart), so a grandparent override from the cozy-metallb umbrella cannot
# suppress that dep via the condition. The metallb-frr-k8s DaemonSet and
# metallb-frr-k8s-statuscleaner Deployment are still rendered with zero
# replicas worth of effect on the chosen BGP path. The frr-k8s metrics ports
# (9140/9141) remain host-network and the cilium-networkpolicy denylist
# covers them; under this escape hatch the additional legacy FRR metrics
# port 9121 on the speaker pod becomes active and the same denylist covers
# it. If you need a fully-clean revert, deinstall the chart and reinstall
# from a prior cozystack release that still ran v0.15.x.
# - Metrics endpoints: kube-rbac-proxy is gone. Speaker and controller now serve
# /metrics directly on HTTPS at port 9120 (was HTTP on 7472). Any external
# scrape config that targeted 7472 must move to 9120 and add a TLS skip / CA.
# - TLS minimum version: upstream removed controller.tlsMinVersion in favor of
# a global tls.minVersion. The new default (empty string) resolves to TLS
# 1.3 per the upstream chart README; the previous chart pinned TLS 1.2.
# Prometheus scrapers without TLS 1.3 support fail until you override:
# metallb.tls.minVersion: VersionTLS12
# - Host-network ports added by the frr-k8s DaemonSet: 9140 (controller metrics),
# 9141 (FRR metrics), plus 19443 from the frr-k8s statuscleaner Deployment.
# The cozystack cilium-networkpolicy denies world ingress to all of these.
metallb:
crds:
enabled: true
controller:
image:
repository: ghcr.io/cozystack/cozystack/metallb-controller
tag: v1.6.0@sha256:cb7321c5674509048dfbea7938378b84013d1370eca5ace7f9665fc79b4a8a05
speaker:
image:
repository: ghcr.io/cozystack/cozystack/metallb-speaker
tag: v1.6.0@sha256:61ab7e2381f0897607b15df7cb5ba4d51a947c261c1c197fcda3b79c7a280016
# A memory limit is what sets memory.max on the pod cgroup, and the Talos
# userspace OOM handler (v1.12+) only ever kills cgroups that lack it. The
# speaker ran BestEffort, so it was picked as the victim whenever a tenant
# workload drove node memory PSI — the pressure and the victim being
# unrelated by design. See the same block on every node DaemonSet below.
#
# Requests come from observed usage (14d peak 97Mi on a 3-node dev cluster);
# limits are deliberately loose, sized as a runaway backstop rather than a
# fitted ceiling, because undersizing here converts a rare pressure-driven
# kill into a deterministic one.
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
memory: 512Mi
# The vendored metallb chart's values.yaml leaves `frr-k8s.prometheus:` as
# a YAML block containing only commented examples — it parses to null.
# Helm-controller's deep-merge then writes that null over the frr-k8s
# subchart's own `prometheus:` map (which carries the schema-driving
# defaults for serviceMonitor / podMonitor). The next render of
# `frr-k8s/templates/service-monitor.yaml` dereferences nil at
# `.Values.prometheus.serviceMonitor.enabled` and aborts the install
# with "nil pointer evaluating interface {}.serviceMonitor".
#
# Pin the merge target here so the subchart always sees a real (false)
# prometheus block. This matches the corresponding test-fixture
# workaround in tests/metallb_test.yaml and unblocks the bump; the
# underlying upstream bug is filed at metallb/frr-k8s (the schema
# placement issue tracked alongside it).
frr-k8s:
prometheus:
serviceMonitor:
enabled: false
podMonitor:
enabled: false
# Same reasoning as speaker.resources above. frr-k8s is the sibling node
# DaemonSet that actually speaks BGP, so it shares the speaker's exposure.
# 14d peaks on a 3-node dev cluster: controller ~124Mi, frr 43Mi,
# frr-metrics 49Mi, frr-status 31Mi, reloader 7Mi.
#
# The four cp-* init containers are NOT covered here: the upstream chart
# hardcodes them with no resources knob, so the pod-level memory.max still
# depends on the default LimitRange the operator maintains in every system
# namespace. Lifting that needs an upstream change in metallb/frr-k8s.
frrk8s:
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
memory: 512Mi
frr:
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
memory: 512Mi
reloader:
resources:
requests:
cpu: 10m
memory: 16Mi
limits:
memory: 128Mi
frrMetrics:
resources:
requests:
cpu: 10m
memory: 32Mi
limits:
memory: 256Mi
frrStatus:
resources:
requests:
cpu: 10m
memory: 32Mi
limits:
memory: 256Mi