[cilium] Opt-out of cri-containerd.apparmor.d for nsenter init containers (#2370)

<!-- Thank you for making a contribution! -->

## What this PR does

Cilium 1.19 init containers `mount-cgroup`, `apply-sysctl-overwrites`
and `clean-cilium-state` call `nsenter` to join the host cgroup/mount
namespace. On any distribution that loads the
`cri-containerd.apparmor.d` AppArmor profile by default for containerd
workloads (Ubuntu 22.04+, Debian with AppArmor), the kernel denies the
operation and the audit log reports it as a blocked `ptrace` operation:

```text
apparmor="DENIED" operation="ptrace" class="ptrace"
  profile="cri-containerd.apparmor.d" comm="nsenter"
  requested_mask="read" denied_mask="read" peer="unconfined"
```

Symptom on non-Talos clusters: after a cilium rollout the new pod stays
in `Init:CrashLoopBackOff`, and the rest of the platform cascades into
`dependency not ready` until cilium is healthy on every node.

This PR opts the affected containers out of the
`cri-containerd.apparmor.d` profile via deprecated per-container
`container.apparmor.security.beta.kubernetes.io/<name>: unconfined`
annotations. Four containers are annotated: `cilium-agent`,
`clean-cilium-state`, `mount-cgroup`, `apply-sysctl-overwrites`.
`mount-bpf-fs` is intentionally excluded because it renders with its own
`securityContext.privileged: true`, and the kernel does not apply
AppArmor profiles to privileged containers.

### Why pod annotations instead of `podSecurityContext.appArmorProfile`

The upstream Cilium chart already sets
`podSecurityContext.appArmorProfile.type: Unconfined` (the modern k8s >=
1.30 API), but containerd's CRI implementation silently ignores this
field today and keeps applying `cri-containerd.apparmor.d` — see
kubernetes/kubernetes#125069 (still open). Only the deprecated
per-container annotations take effect end-to-end on current k3s /
kubeadm + containerd setups.

### Why a dedicated values file limited to non-Talos variants

On Talos (`isp-full` bundle) `values-talos.yaml` sets
`cgroup.autoMount.enabled: false`, so the upstream chart does not render
the `mount-cgroup` init container. The kube-apiserver rejects a
DaemonSet that carries a per-container AppArmor annotation for a
container that is not in the pod spec:

```
DaemonSet.apps "cilium" is invalid: spec.template.annotations[
container.apparmor.security.beta.kubernetes.io/mount-cgroup]:
Invalid value: "mount-cgroup": container not found
```

To keep the Talos path untouched and only add the annotations where they
are both safe and useful, this PR puts the `cilium.podAnnotations` block
in a new `packages/system/cilium/values-apparmor.yaml` and wires it into
only the non-Talos PackageSource variants (`cilium-generic`,
`kubeovn-cilium-generic`). Those variants always run with
`cgroup.autoMount.enabled: true` (explicit bundle override in
`packages/core/platform/templates/bundles/system.yaml:83-86`), so every
annotated container really exists in the rendered DaemonSet.

On Talos the annotations are omitted entirely — that is safe because the
Talos kernel does not load the AppArmor LSM anyway.

### Vendored chart patch to avoid duplicate keys on unsupported k8s <
1.30

On k8s < 1.30 the upstream cilium chart template emits the same
annotation keys from its own `semverCompare "<1.30.0"` branch in
`cilium-agent/daemonset.yaml`. Letting both sources write to the
annotations mapping would produce a YAML map with duplicate keys.
Cozystack's supported k8s matrix starts at 1.30
(`packages/apps/kubernetes/files/versions.yaml`), so the duplicate
rendering is never observed in practice, but to keep the rendered
manifest clean on any k8s version this PR also adds a `perl -i -0pe`
multi-line delete to `packages/system/cilium/Makefile` that strips the
entire upstream `{{- if not .Values.securityContext.privileged }} / {{-
if semverCompare "<1.30.0" }}` block from the vendored template. A
fail-fast `grep` guard after the `perl` invocation turns a
silently-failed patch (e.g. if upstream reformats the block) into a loud
error. The same patch is applied to the currently vendored copy so the
repository state matches what `make update` produces. This follows the
existing sed-patch pattern already used for `values.yaml` (`Used in
iptables`, `SYS_MODULE`).

### Scope

- `packages/system/cilium/values-apparmor.yaml` — new file containing
the four `cilium.podAnnotations`
- `packages/system/cilium/values.yaml` — unchanged apart from reverting
the previously-added annotations block (the annotations now live in
`values-apparmor.yaml`)
- `packages/system/cilium/Makefile` — extends the `update:` target with
a `perl` multi-line delete (plus a `grep` fail-fast guard) that drops
the entire upstream AppArmor annotation block on every vendoring refresh
-
`packages/system/cilium/charts/cilium/templates/cilium-agent/daemonset.yaml`
— the vendored file with that block already removed to match the
Makefile patch
- `packages/core/platform/sources/networking.yaml` — adds
`values-apparmor.yaml` to the `cilium-generic` and
`kubeovn-cilium-generic` variants only

### Compatibility across supported platforms

- **Talos Linux** (`isp-full` bundle, `kubeovn-cilium` variant) —
AppArmor LSM is not loaded, `cgroup.autoMount.enabled: false`,
`values-apparmor.yaml` is not included. Zero annotations in the rendered
DaemonSet. No behavioural change. Verified with `helm template` +
`kubectl apply --dry-run=server` against a live k3s v1.35.2 cluster: no
errors, no AppArmor lines.
- **Ubuntu 22.04 / 24.04, Debian with AppArmor** (`isp-full-generic`
bundle, `kubeovn-cilium-generic` / `cilium-generic` variants) — the four
annotations are present in the rendered DaemonSet, kubelet starts the
listed containers with the unconfined profile, `nsenter` succeeds.
Verified with `helm template` + `kubectl apply --dry-run=server` against
a live k3s v1.35.2 cluster: no errors, exactly four
`container.apparmor.security.beta.kubernetes.io/*: unconfined` lines.
Verified on a fresh k3s + Ubuntu 24.04 deployment end-to-end.
- **RHEL / Rocky / AlmaLinux / Fedora** — use SELinux, not AppArmor.
When deployed via `*-generic` variants these platforms pick up the
annotations; kubelet silently ignores AppArmor annotations when the LSM
is not loaded, SELinux policy is unaffected.
- **k8s < 1.30 and k8s >= 1.30** — both now render the same four
annotations (on non-Talos), sourced exclusively from
`cilium.podAnnotations`. Verified via `helm template --kube-version
1.29.0` and `--kube-version 1.35.0` → exactly 4 annotations with no
duplicates.

### Release note

```release-note
[cilium] Opt four cilium-agent containers (cilium-agent, clean-cilium-state, mount-cgroup, apply-sysctl-overwrites) out of the cri-containerd.apparmor.d AppArmor profile via per-container annotations, applied only on non-Talos cilium variants (cilium-generic, kubeovn-cilium-generic). This unblocks cilium rollouts on Ubuntu 22.04+, Debian and other distributions that load this profile for containerd by default, where the kernel previously denied nsenter's namespace entry (reported as a blocked ptrace operation) and the cilium agent landed in Init:CrashLoopBackOff. Talos variants are untouched.
```

packages/core/platform/sources/networking.yaml

@@ -66,6 +66,7 @@ spec:
       path: system/cilium
       valuesFiles:
       - values.yaml
+      - values-apparmor.yaml
       install:
         privileged: true
         namespace: cozy-cilium
@@ -117,6 +118,7 @@ spec:
       path: system/cilium
       valuesFiles:
       - values.yaml
+      - values-apparmor.yaml
       - values-kubeovn.yaml
       install:
         privileged: true

packages/system/cilium/Makefile

@@ -12,6 +12,15 @@ update:
 	helm repo update cilium
 	helm pull cilium/cilium --untar --untardir charts --version 1.19
 	$(SED_INPLACE) -e '/Used in iptables/d' -e '/SYS_MODULE/d' charts/cilium/values.yaml
+	# Drop the whole k8s<1.30 AppArmor annotations block from the cilium-agent
+	# DaemonSet. Cozystack manages these annotations through cilium.podAnnotations
+	# in values.yaml on every k8s version, so keeping the upstream block would
+	# produce duplicate mapping keys on k8s<1.30.
+	perl -i -0pe 's|\n        \{\{- if not \.Values\.securityContext\.privileged \}\}\n        \{\{- if semverCompare "<1\.30\.0".*?\n        \{\{- end \}\}\n        \{\{- end \}\}\n        \{\{- end \}\}||s' \
+		charts/cilium/templates/cilium-agent/daemonset.yaml
+	@! grep -q 'container\.apparmor\.security\.beta\.kubernetes\.io' \
+		charts/cilium/templates/cilium-agent/daemonset.yaml || \
+		{ echo 'ERROR: perl patch did not remove the upstream AppArmor block from cilium-agent/daemonset.yaml (upstream template format may have changed)' >&2; exit 1; }
 	version=$$(awk '$$1 == "version:" {print $$2}' charts/cilium/Chart.yaml) && \
 	$(SED_INPLACE) "s/ARG VERSION=.*/ARG VERSION=v$${version}/" images/cilium/Dockerfile
 

packages/system/cilium/charts/cilium/templates/cilium-agent/daemonset.yaml

@@ -57,19 +57,6 @@ spec:
         # ensure pods roll when configmap updates
         cilium.io/cilium-configmap-checksum: {{ include (print $.Template.BasePath "/cilium-configmap.yaml") . | sha256sum | quote }}
         {{- end }}
-        {{- if not .Values.securityContext.privileged }}
-        {{- if semverCompare "<1.30.0" (printf "%d.%d.0" (semver .Capabilities.KubeVersion.Version).Major (semver .Capabilities.KubeVersion.Version).Minor) }}
-        # Set app AppArmor's profile to "unconfined". The value of this annotation
-        # can be modified as long users know which profiles they have available
-        # in AppArmor.
-        container.apparmor.security.beta.kubernetes.io/cilium-agent: "unconfined"
-        container.apparmor.security.beta.kubernetes.io/clean-cilium-state: "unconfined"
-        {{- if .Values.cgroup.autoMount.enabled }}
-        container.apparmor.security.beta.kubernetes.io/mount-cgroup: "unconfined"
-        container.apparmor.security.beta.kubernetes.io/apply-sysctl-overwrites: "unconfined"
-        {{- end }}
-        {{- end }}
-        {{- end }}
         {{- with .Values.podAnnotations }}
         {{- toYaml . | nindent 8 }}
         {{- end }}

packages/system/cilium/values-apparmor.yaml

@@ -0,0 +1,20 @@
+cilium:
+  # Opt out of the cri-containerd.apparmor.d profile for containers that invoke
+  # nsenter to join the host cgroup/mount namespace. The kernel reports the
+  # denial as a blocked "ptrace" operation. Required on Ubuntu 22.04+ and
+  # other distros that load this AppArmor profile by default, where the
+  # denial otherwise puts cilium-agent into Init:CrashLoopBackOff.
+  # The deprecated annotations are used because k8s >= 1.30 pod-level
+  # appArmorProfile: Unconfined is not honoured by containerd CRI today
+  # (kubernetes/kubernetes#125069).
+  # Only applied on non-Talos cilium variants where cgroup.autoMount.enabled is
+  # true — on Talos mount-cgroup is not rendered and the kube-apiserver would
+  # reject an annotation referencing a non-existent container.
+  # mount-bpf-fs is intentionally excluded: it renders with its own
+  # securityContext.privileged=true, and the kernel does not apply AppArmor
+  # profiles to privileged containers.
+  podAnnotations:
+    container.apparmor.security.beta.kubernetes.io/cilium-agent: unconfined
+    container.apparmor.security.beta.kubernetes.io/clean-cilium-state: unconfined
+    container.apparmor.security.beta.kubernetes.io/mount-cgroup: unconfined
+    container.apparmor.security.beta.kubernetes.io/apply-sysctl-overwrites: unconfined