Andrei Kvapil
587eeb09f0
<!-- Thank you for making a contribution! --> ## What this PR does Cilium 1.19 init containers `mount-cgroup`, `apply-sysctl-overwrites` and `clean-cilium-state` call `nsenter` to join the host cgroup/mount namespace. On any distribution that loads the `cri-containerd.apparmor.d` AppArmor profile by default for containerd workloads (Ubuntu 22.04+, Debian with AppArmor), the kernel denies the operation and the audit log reports it as a blocked `ptrace` operation: ```text apparmor="DENIED" operation="ptrace" class="ptrace" profile="cri-containerd.apparmor.d" comm="nsenter" requested_mask="read" denied_mask="read" peer="unconfined" ``` Symptom on non-Talos clusters: after a cilium rollout the new pod stays in `Init:CrashLoopBackOff`, and the rest of the platform cascades into `dependency not ready` until cilium is healthy on every node. This PR opts the affected containers out of the `cri-containerd.apparmor.d` profile via deprecated per-container `container.apparmor.security.beta.kubernetes.io/<name>: unconfined` annotations. Four containers are annotated: `cilium-agent`, `clean-cilium-state`, `mount-cgroup`, `apply-sysctl-overwrites`. `mount-bpf-fs` is intentionally excluded because it renders with its own `securityContext.privileged: true`, and the kernel does not apply AppArmor profiles to privileged containers. ### Why pod annotations instead of `podSecurityContext.appArmorProfile` The upstream Cilium chart already sets `podSecurityContext.appArmorProfile.type: Unconfined` (the modern k8s >= 1.30 API), but containerd's CRI implementation silently ignores this field today and keeps applying `cri-containerd.apparmor.d` — see kubernetes/kubernetes#125069 (still open). Only the deprecated per-container annotations take effect end-to-end on current k3s / kubeadm + containerd setups. ### Why a dedicated values file limited to non-Talos variants On Talos (`isp-full` bundle) `values-talos.yaml` sets `cgroup.autoMount.enabled: false`, so the upstream chart does not render the `mount-cgroup` init container. The kube-apiserver rejects a DaemonSet that carries a per-container AppArmor annotation for a container that is not in the pod spec: ``` DaemonSet.apps "cilium" is invalid: spec.template.annotations[ container.apparmor.security.beta.kubernetes.io/mount-cgroup]: Invalid value: "mount-cgroup": container not found ``` To keep the Talos path untouched and only add the annotations where they are both safe and useful, this PR puts the `cilium.podAnnotations` block in a new `packages/system/cilium/values-apparmor.yaml` and wires it into only the non-Talos PackageSource variants (`cilium-generic`, `kubeovn-cilium-generic`). Those variants always run with `cgroup.autoMount.enabled: true` (explicit bundle override in `packages/core/platform/templates/bundles/system.yaml:83-86`), so every annotated container really exists in the rendered DaemonSet. On Talos the annotations are omitted entirely — that is safe because the Talos kernel does not load the AppArmor LSM anyway. ### Vendored chart patch to avoid duplicate keys on unsupported k8s < 1.30 On k8s < 1.30 the upstream cilium chart template emits the same annotation keys from its own `semverCompare "<1.30.0"` branch in `cilium-agent/daemonset.yaml`. Letting both sources write to the annotations mapping would produce a YAML map with duplicate keys. Cozystack's supported k8s matrix starts at 1.30 (`packages/apps/kubernetes/files/versions.yaml`), so the duplicate rendering is never observed in practice, but to keep the rendered manifest clean on any k8s version this PR also adds a `perl -i -0pe` multi-line delete to `packages/system/cilium/Makefile` that strips the entire upstream `{{- if not .Values.securityContext.privileged }} / {{- if semverCompare "<1.30.0" }}` block from the vendored template. A fail-fast `grep` guard after the `perl` invocation turns a silently-failed patch (e.g. if upstream reformats the block) into a loud error. The same patch is applied to the currently vendored copy so the repository state matches what `make update` produces. This follows the existing sed-patch pattern already used for `values.yaml` (`Used in iptables`, `SYS_MODULE`). ### Scope - `packages/system/cilium/values-apparmor.yaml` — new file containing the four `cilium.podAnnotations` - `packages/system/cilium/values.yaml` — unchanged apart from reverting the previously-added annotations block (the annotations now live in `values-apparmor.yaml`) - `packages/system/cilium/Makefile` — extends the `update:` target with a `perl` multi-line delete (plus a `grep` fail-fast guard) that drops the entire upstream AppArmor annotation block on every vendoring refresh - `packages/system/cilium/charts/cilium/templates/cilium-agent/daemonset.yaml` — the vendored file with that block already removed to match the Makefile patch - `packages/core/platform/sources/networking.yaml` — adds `values-apparmor.yaml` to the `cilium-generic` and `kubeovn-cilium-generic` variants only ### Compatibility across supported platforms - **Talos Linux** (`isp-full` bundle, `kubeovn-cilium` variant) — AppArmor LSM is not loaded, `cgroup.autoMount.enabled: false`, `values-apparmor.yaml` is not included. Zero annotations in the rendered DaemonSet. No behavioural change. Verified with `helm template` + `kubectl apply --dry-run=server` against a live k3s v1.35.2 cluster: no errors, no AppArmor lines. - **Ubuntu 22.04 / 24.04, Debian with AppArmor** (`isp-full-generic` bundle, `kubeovn-cilium-generic` / `cilium-generic` variants) — the four annotations are present in the rendered DaemonSet, kubelet starts the listed containers with the unconfined profile, `nsenter` succeeds. Verified with `helm template` + `kubectl apply --dry-run=server` against a live k3s v1.35.2 cluster: no errors, exactly four `container.apparmor.security.beta.kubernetes.io/*: unconfined` lines. Verified on a fresh k3s + Ubuntu 24.04 deployment end-to-end. - **RHEL / Rocky / AlmaLinux / Fedora** — use SELinux, not AppArmor. When deployed via `*-generic` variants these platforms pick up the annotations; kubelet silently ignores AppArmor annotations when the LSM is not loaded, SELinux policy is unaffected. - **k8s < 1.30 and k8s >= 1.30** — both now render the same four annotations (on non-Talos), sourced exclusively from `cilium.podAnnotations`. Verified via `helm template --kube-version 1.29.0` and `--kube-version 1.35.0` → exactly 4 annotations with no duplicates. ### Release note ```release-note [cilium] Opt four cilium-agent containers (cilium-agent, clean-cilium-state, mount-cgroup, apply-sysctl-overwrites) out of the cri-containerd.apparmor.d AppArmor profile via per-container annotations, applied only on non-Talos cilium variants (cilium-generic, kubeovn-cilium-generic). This unblocks cilium rollouts on Ubuntu 22.04+, Debian and other distributions that load this profile for containerd by default, where the kernel previously denied nsenter's namespace entry (reported as a blocked ptrace operation) and the cilium agent landed in Init:CrashLoopBackOff. Talos variants are untouched. ```
Cozystack
Cozystack is a free PaaS platform and framework for building clouds.
Cozystack is a CNCF Sandbox Level Project that was originally built and sponsored by Ænix.
With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.
Use Cozystack to build your own cloud or provide a cost-effective development environment.
Use-Cases
-
Using Cozystack to build a public cloud
You can use Cozystack as a backend for a public cloud -
Using Cozystack to build a private cloud
You can use Cozystack as a platform to build a private cloud powered by Infrastructure-as-Code approach -
Using Cozystack as a Kubernetes distribution
You can use Cozystack as a Kubernetes distribution for Bare Metal
Documentation
The documentation is located on the cozystack.io website.
Read the Getting Started section for a quick start.
If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.
Versioning
Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.
Contributions
Contributions are highly appreciated and very welcomed!
In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.
You can express your intention to on the fix on your own. Commits are used to generate the changelog, and their author will be referenced in it.
If you have Feature Requests please use the Discussion's Feature Request section.
Community
You are welcome to join our Telegram group and come to our weekly community meetings. Add them to your Google Calendar or iCal for convenience.
License
Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.
Commercial Support
A list of companies providing commercial support for this project can be found on official site.