# System Component Memory Limits Cozystack defaults container memory in eligible system namespaces and sets explicit requests and limits on the node DaemonSets. This page explains why that is not merely resource hygiene, how to tune it, and what it deliberately does not cover. ## Why a memory limit, and not a PriorityClass Talos Linux v1.12 introduced a userspace OOM handler, enabled by default, that reacts to memory pressure before the kernel OOM killer does. Its only input from the machine is its own config document; everything else it reads straight out of cgroupfs, and it holds no Kubernetes client at all. `PriorityClass` — `system-node-critical` included — therefore has no bearing on which cgroup it picks. Setting a priority on a system component to stop these kills looks like a fix and changes nothing about them. Take that claim narrowly, because priority is far from inert elsewhere: it still drives scheduler preemption, kubelet eviction ordering, and the `oom_score_adj` kubelet hands the kernel OOM killer. None of those three is what kills these pods. The one ordering input this handler has is the Kubernetes **QoS class**, inferred from the cgroup path rather than from any pod object — a different axis from priority, so a `system-node-critical` pod with no memory limit is an ordinary candidate like any other. The handler scores each pod cgroup with a CEL expression. The default in Talos v1.14.2, the version Cozystack currently ships for host nodes, is: ``` memory_max.hasValue() ? 0.0 : {Besteffort: 1.0, Burstable: 0.5, Guaranteed: 0.0, Podruntime: 0.0, System: 0.0}[class] * double(memory_current.orValue(0u)) ``` Any cgroup scoring zero is dropped from the candidate set entirely. A pod whose containers all carry a memory limit has `memory.max` set on its pod cgroup, scores `0.0`, and can never be selected. A pod without one stays a candidate no matter how little memory it is using. Three consequences follow, and all three are easy to get wrong: - Only a **limit** grants immunity. A memory **request** merely moves the pod from BestEffort to Burstable, which under the default `strictCgroupClassOrdering: true` means "killed second" rather than "not killed" — Burstable cgroups are considered only once no BestEffort one is eligible, and the score then breaks ties within the class. That setting arrived in v1.13.4; on v1.13.0 through v1.13.3 the score alone decided, so a large Burstable pod could outrank a small BestEffort one. - **Every** container in the pod needs a limit, init containers included. Kubelet only sets pod-level `memory.max` when all of them have one, so a single limit-free sidecar puts the whole pod back in the candidate set. - CPU limits are irrelevant here. The ranking expression is given the cgroup's path, its QoS class, and `memory.max`, `memory.current` and `memory.peak` — no CPU information of any kind. Victim selection is also decoupled from the trigger. The cgroup that caused the pressure and the cgroup that gets `SIGKILL`ed are unrelated by design. In practice the pods carrying limits are overwhelmingly tenant workloads — managed applications are sized through resource presets, and a tenant namespace with `resourceQuotas` configured gets a default limit of its own — while system components carried none until the change this page describes. That inverts the intended order: whatever fires the trigger, `metallb-speaker` or `linstor-satellite`, using a few dozen megabytes and entirely uninvolved, is the one killed for it, repeatedly, until the pressure subsides. On Talos v1.13.x a tenant workload thrashing against its own multi-gigabyte limit is enough to fire it, through the node-wide PSI clause described [below](#when-the-trigger-itself-is-the-problem). ## What Cozystack does **A default LimitRange in eligible system namespaces.** The `cozystack-operator` maintains a `LimitRange` named `cozystack-system-defaults` in each namespace it reconciles whose name does not begin with `tenant-`, defaulting container memory for anything that declares none. It covers containers whose upstream chart exposes no `resources` knob. Namespaces with a foreign memory policy or an unresolved safety hold are excluded. **Explicit requests and limits on node DaemonSets.** Charts additionally set real values on the DaemonSets that run on every node — the cilium agent and its init containers, metallb speaker, the frr-k8s controller and its sidecars, the linstor satellite along with plunger and drbd-logger, virt-handler, fluent-bit, node-exporter, multus and its init container, velero node-agent, and all three kubevirt-csi-node containers. Requests there are fitted to observed usage, which is a real signal for the scheduler; the LimitRange cannot supply that, because it applies one number to every container in the namespace and so has to keep its default request deliberately tiny. kube-ovn is absent from that list because its vendored chart already sets both. The operator's LimitRange stops at the `tenant-` prefix, so tenant namespaces never receive it. A tenant namespace gets a default of its own only from the tenant chart's `tenant-range-limits`, which is rendered only when `resourceQuotas` is configured on the tenant and is empty by default. A tenant workload left without a memory limit therefore stays an eviction candidate, which is the upstream design working as intended and is what restores the ordering described above. ## Tuning Two installer values, both empty by default so the operator's own defaults apply: | Value | Operator flag | Default | |---|---|---| | `cozystackOperator.systemNamespaceMemoryLimit` | `--system-namespace-memory-limit` | `32Gi` | | `cozystackOperator.systemNamespaceMemoryRequest` | `--system-namespace-memory-request` | `32Mi` | The limit is a ceiling, not a reservation, and the default is deliberately far above any real working set. The handler discards a cgroup for carrying a `memory.max` at all, whatever its value, so what the default buys is immunity rather than a fitted ceiling. That is also why it is set so high: the scan described below sees *declared requests*, never a container's actual resident set, so a ceiling low enough to bind is a ceiling that can convert a rare pressure-driven kill into a deterministic one for any component whose real usage nobody measured. Raising it is close to free; lowering it is where the risk lives, and it is worth doing per component in that component's own chart before doing it fleet-wide here. **The limit must stay above the largest memory request in any system namespace.** A defaulted limit below a container's own request is rejected at admission, and the pod simply will not start. List the requests before lowering it — the operator marks exactly the namespaces it treats as system with `cozystack.io/system=true`, the same condition under which it creates the LimitRange: ```bash for ns in $(kubectl get ns -l cozystack.io/system=true -o jsonpath='{.items[*].metadata.name}'); do kubectl get pods -n "$ns" -o json | jq -r --arg ns "$ns" ' .items[] | [.spec.containers[], (.spec.initContainers // [])[]][] | select(.resources.requests.memory != null) | "\($ns)\t\(.name)\t\(.resources.requests.memory)"' done | sort -u ``` **Keep the request small.** The operator always pairs the default limit with a default request, because a `LimitRange` that sets `default` without `defaultRequest` makes each container's request equal its limit and reserves the whole ceiling at schedule time. Note that leaving `systemNamespaceMemoryRequest` empty does not produce that state: an empty installer value omits the flag, which selects the operator's own `32Mi`. The knob is there to be raised or lowered, and clearing it is not a way to switch the request off. The operator also refuses to start when the request exceeds the limit. A `LimitRange` whose `defaultRequest` is above its `default` is rejected by the API server, which would wedge namespace reconciliation for every system package, so the check happens at startup rather than at apply time. Setting the limit to `0` disables the feature, so the knob is reversible. Leaving it empty is not the same thing: that selects the operator's `32Gi` default. On the next Package reconcile after the feature is disabled, the operator lists LimitRanges cluster-wide by `app.kubernetes.io/managed-by: cozystack-package-controller` and removes every object it owns, including leftovers in namespaces no active Package targets. A same-named LimitRange without that ownership label is left alone. The same ownership boundary applies while the feature is enabled: a foreign object already holding the `cozystack-system-defaults` name is neither overwritten nor adopted, even if it contains only CPU or storage policy, so that namespace receives no Cozystack memory default until the name is free. ## Withholding and acknowledging a default **An unsafe default stays withheld until a cluster administrator acknowledges it.** Before applying, the operator scans Deployments, StatefulSets, DaemonSets, CronJobs, Jobs and Pods, including init containers, for requests above the configured limit without an explicit limit. A visible blocker withdraws the managed LimitRange and records a hold on the Namespace. Removing a Pod or template does not release the hold: the same empty snapshot can occur between deletion and recreation by another controller. Lowering a previously applied default also creates a hold, even if no blocker is visible. LimitRanger persists its default on admitted Pods, so a stored limit cannot prove that the workload's source declares one. The operator remembers the previous limit independently of the LimitRange and does not infer permission to tighten from a Pod's `limits` field or `kubernetes.io/limit-ranger` annotation. The cluster-scoped Namespace carries `operator.cozystack.io/system-memory-state`, a JSON object with `version`, `lastLimit`, and, when held, `hold`, `target` and `reason`. This is operator-managed safety state. It survives controller restart, removal of the LimitRange, removal of the Package target, and disabling the feature. Do not remove or edit it to clear a hold: doing so discards the evidence protecting future Pod admission. Invalid state blocks new defaults until a cluster administrator repairs it. Inspect the hold and the required acknowledgement: ```bash kubectl get namespace -o json | jq -r ' .metadata.annotations["operator.cozystack.io/system-memory-state"] | fromjson | {reason, target, acknowledgement: (.hold + ":" + .target)}' ``` Before acknowledging, inspect the source of every affected workload, including custom resources and static-pod manifests that this scanner cannot read. Give the affected container an explicit limit at least as large as its request, lower its request only when appropriate, or raise the configured default. A currently running Pod's defaulted limit is not that source. Also check actual memory usage: admission safety does not make a small runtime ceiling safe. Copy the exact `acknowledgement` value reported for the current target into a separate annotation: ```bash kubectl annotate namespace \ operator.cozystack.io/system-memory-ack=':' --overwrite ``` This requires permission to patch the cluster-scoped Namespace; a tenant's namespaced RoleBinding does not grant it. The operator grants no additional permission for acknowledgement. A token authorizes only its hold generation and canonical target quantity. Changing the configured target rotates the hold, even when returning to an earlier value. The operator consumes a matching token once; if a concrete request-above-default blocker still exists, the token is rejected and removed, and the hold stays. Acknowledgement never overrides a foreign LimitRange. After successful acknowledgement and a clean scan, the configured default is applied. Existing Pods keep their resources; the new default reaches later admissions. Package reconciles run after ordinary events and are additionally scheduled every five minutes, so acknowledgement and workload changes also progress on a quiet cluster, subject to controller queue and API availability. The reason and required token are visible on the Namespace and in operator logs: ```bash kubectl -n cozy-system logs deploy/cozystack-operator | grep "withholding the default container memory limit" ``` A hold remains after disable/re-enable, and changing the configured value while held requires acknowledging the new token. The operator also removes its LimitRange when no active Package targets the namespace, considering all Packages sharing it; the Namespace safety state remains for a later reinstall. Setting the limit to zero removes managed defaults even if their safety state is invalid. **A namespace another LimitRange already governs is left alone.** If a system namespace — `kube-system` included — already carries a LimitRange that says anything about memory, the operator writes nothing there and withdraws anything it wrote earlier. Any memory-bearing field counts, at either the `Container` or the `Pod` scope: a second `default` leaves the effective ceiling to the order `LimitRanger` happens to iterate them, a `max` below this default or a `min` above the default request rejects the pod outright, and a `maxLimitRequestRatio` rejects it too, since a 32Gi default against a 32Mi default request is a ratio of 1024:1. The `Pod` scope is included because the collision there is not decidable in advance at all — whether a per-container default breaches a pod-wide bound depends on how many containers a not-yet-created pod will have. LimitRanges that bound only cpu or storage say nothing about memory and coexist normally. Note that withdrawing hands the namespace to the administrator's policy in full: if that LimitRange defaults container memory below some container's own request, the resulting admission rejection is theirs to resolve, and it is one that would have occurred without Cozystack in the picture at all. A LimitRange only mutates at admission. Existing pods keep running without limits until they restart, so the protection lands progressively as workloads roll rather than the moment the setting is applied. ## Verifying Confirm a pod cgroup actually carries `memory.max` — this is the property that matters, not the QoS class. Talos runs kubelet with the `cgroupfs` driver on a unified cgroup v2 hierarchy, so a pod cgroup is a directory named for the pod UID, dashes and all: ```bash POD_UID=$(kubectl get pod -n -o jsonpath='{.metadata.uid}') NODE=$(kubectl get pod -n -o jsonpath='{.spec.nodeName}') NODE_IP=$(kubectl get node "$NODE" -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}') talosctl -n "$NODE_IP" read "/sys/fs/cgroup/kubepods/burstable/pod$POD_UID/memory.max" ``` A byte count rather than the literal `max` means the pod is out of the victim set. Note that `talosctl -n` takes a machine address rather than a Kubernetes node name, which is why the address is looked up instead of being reused from `nodeName`. Mind the QoS segment of that path, and do not extrapolate it. BestEffort and Burstable pods live under `kubepods/besteffort/` and `kubepods/burstable/`, but there is no `guaranteed` directory: kubelet creates QoS-level cgroups for those two classes only, so a Guaranteed pod sits directly at `kubepods/pod`. To avoid guessing, list the pod directories two levels down instead: ```bash talosctl -n "$NODE_IP" ls -d 2 -t d /sys/fs/cgroup/kubepods ``` `talosctl read` wants an `os:admin` talosconfig and a single node per call. Reading the same file through `kubectl debug node/$NODE --profile=sysadmin` under `/host/sys/fs/cgroup` also works, but it depends on the kubelet, on scheduling, and on a namespace that admits a privileged pod — all shakier than the Talos API in precisely the situation that makes you run this check. Inspect what the handler has actually killed: ```bash talosctl -n get oomactions -o yaml ``` Each entry carries the score the victim was selected on, the command lines of the processes in it, and a dump of the trigger context it fired under. Ask for `-o yaml`: the default table shows only a score, and its `Time` column reads a field the spec never sets, so it renders empty. This is a ring buffer of the last 50 actions held in memory, so it does not survive a `machined` restart and is no substitute for an alert — but it is still the better of the two signals here, being structured and complete where the logs are neither. Once every eligible pod in a namespace carries a limit, the healthy signature is the controller still triggering under pressure and finding nothing it is allowed to kill: ```bash talosctl -n logs controller-runtime | grep OOMController ``` Both halves of that signature — `OOM controller triggered`, then `no eligible cgroup to kill` — are emitted at the default log level, so neither needs a debug flag. Note that the structured fields are encoded as JSON rather than logfmt, so the second line reads `no eligible cgroup to kill {"component": "controller-runtime", "controller": "runtime.OOMController", "ranked": 0}`. Grep for `"ranked"` or for the message text; `ranked=` matches nothing and reads as though the handler were not running. The trigger firing is not itself a fault. `talosctl dmesg` carries the same lines and is the more common reflex, but it is the weaker source: that path strips log levels and timestamps, truncates at 976 bytes, and suppresses the first four occurrences of each ranking error — precisely the lines worth having if scoring ever misbehaves. ## What this does not cover **Kubernetes static pods.** `kube-apiserver`, `kube-controller-manager` and `kube-scheduler` are managed by Talos rather than by any chart. A `cozystack-system-defaults` LimitRange does reach `kube-system`, since `cozystack-scheduler` installs there, but it cannot touch them: a LimitRange defaults resources at API-server admission, and kubelet builds static pods straight from files on disk without ever passing through it. Sizing those is a Talos machine-config matter. Their mirror pods pass admission. An already persisted mirror Pod with an oversized request and no limit creates a hold, and lowering the previous default requires acknowledgement. A mirror Pod rejected before persistence provides no scan evidence; its absence does not mean the static manifest is safe. **Future Pods outside the scanned templates.** A custom controller can change its desired request without producing a Deployment, StatefulSet, DaemonSet, CronJob or Job. If its first replacement Pod is rejected at admission, there is no new Pod for this scanner to inspect. An ownerless ReplicaSet at zero replicas has the same gap. A `RequestsOnly` VPA can likewise introduce an unseen request after defaulting. The hold prevents automatic lowering and automatic recovery after an observed blocker, but does not validate arbitrary future custom-resource changes. Configure explicit resources for those workloads and inspect their admission errors. **A handful of vendored containers with no upstream `resources` knob** — the four frr-k8s `cp-*` init containers, the kube-ovn `hostpath-init` and `install-cni` init containers, and `cozy-proxy`, whose chart exposes no resources value at all. These are covered by the namespace LimitRange and nothing else. That is enough for OOM immunity, since the defaulted limit is what puts `memory.max` on the cgroup, but their request is then the generic default rather than a figure fitted to measured usage, so the scheduler gets a weaker signal for them than for the DaemonSets above. Lifting that needs changes upstream. **Optional components** `hami` and `kilo` carry no chart-level values, on the grounds that inventing numbers for components with no usage measurements behind them is worse than the blanket default. ## When the trigger itself is the problem Giving system components limits stops them being *victims*. It does not stop the handler *triggering*, and a node under genuine sustained pressure will keep firing. The v1.14.2 default trigger is: ``` (multiply_qos_vectors(d_qos_memory_full_total, {System: 8.0, Podruntime: 4.0}) > 3000.0 && multiply_qos_vectors(qos_memory_full_avg10, {System: 1.0, Podruntime: 1.0}) > 5.0 && time_since_trigger > duration("5s")) ``` It fires only on memory stalls in the `System` and `Podruntime` cgroups, so pressure confined to pods does not trip it. It reached that shape over four changes: [siderolabs/talos#12602](https://github.com/siderolabs/talos/pull/12602) replaced the original global-PSI trigger with the per-QoS form in v1.13.0, [#12632](https://github.com/siderolabs/talos/pull/12632) added the `qos_memory_full_avg10 > 5.0` conjunct to make it less sensitive, [#13725](https://github.com/siderolabs/talos/pull/13725) added the `5s` cooldown in v1.13.6, and [#13895](https://github.com/siderolabs/talos/pull/13895) dropped a second clause, `memory_full_avg10 > 75.0 && time_since_trigger > duration("10s")`, in v1.14.0. That clause was a cause-blind global-PSI backstop: a single workload thrashing inside its own cgroup limit could drive root `memory_full` above 75 while the node had free RAM. Any node still running a v1.13 release carries it, tenant Kubernetes workers included. A tenant worker's machine config is rendered by the `kubernetes-nodes` chart, which accepts no extra documents, so the workaround below is for host nodes only. On a v1.13 host node the backstop can be removed without upgrading, through an `OOMConfig` machine-config document that sets the v1.14 trigger — at the cost of the last-resort guard before the kernel OOM killer takes over. The document also accepts `cgroupRankingExpression`, `strictCgroupClassOrdering` and `sampleInterval`; anything left out keeps its default, so overriding the trigger alone is enough here: ```yaml apiVersion: v1alpha1 kind: OOMConfig triggerExpression: |- (multiply_qos_vectors(d_qos_memory_full_total, {System: 8.0, Podruntime: 4.0}) > 3000.0 && multiply_qos_vectors(qos_memory_full_avg10, {System: 1.0, Podruntime: 1.0}) > 5.0 && time_since_trigger > duration("5s")) ``` Prefer fixing the workload that is generating the pressure. Persistent triggering on v1.14 means the system and runtime cgroups keep stalling for memory, and capping the pod that is growing into the node's memory addresses the cause rather than the symptom. On v1.13 it can also mean a pod sitting at its own memory ceiling and reclaiming constantly, which raising that pod's limit addresses.