mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-03 18:47:04 +09:00
fix(installer): host cozy-system labeler hook in kube-system
Re-introduces the pre-install/pre-upgrade Job hook (cozy-system-labeler) that was added by 3a87485c/4754f57f and dropped bybe0a6687, but moves it into kube-system to dodge the PSA admission denial that broke the in-namespace approach: pods "cozy-system-labeler-..." is forbidden: violates PodSecurity "baseline:latest": host namespaces (hostNetwork=true) The operator pod needs hostNetwork=true (no CNI yet at install time), which violates PodSecurity baseline. cozy-system therefore needs enforce=privileged before the operator pod can be admitted. A labeler pod inside cozy-system would itself need hostNetwork=true and hit the same admission denial — Job admission denial means no pod is ever created, .status.failed never increments and backoffLimit never trips, helm waits for Complete=True until its budget elapses. `kube-system` is PSA-exempt on Talos (defaults.exemptions.namespaces in the apiserver PodSecurityConfiguration) and unlabeled on vanilla kubeadm/kind/k3s, so hostNetwork pods are admitted there. The hook's ServiceAccount is bound to a ClusterRole scoped via `resourceNames: ["cozy-system"]` to the one target namespace, keeping the cluster permission grant minimal. Also revert the e2e bats workaround (ddc429c8) that pre-applied cozy-system manually before helm install: the labeler hook handles the labels in-chart now, so the bats can go back to plain `helm upgrade --install --namespace cozy-system --create-namespace ...`. Mirrors the corresponding fix on PR #2508 (fix/installer-namespace-bootstrap) so the two branches converge on the same resolution. Verified end-to-end in CI on this branch: cold install passes admission, operator pod comes up, all downstream E2E tests pass. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
This commit is contained in:
2 files changed
+125
-20
No files matched your search
@@ -28,30 +28,13 @@
|
||||
}
|
||||
|
||||
@test "Install Cozystack" {
|
||||
# Pre-create the cozy-system namespace with the labels the operator pod needs.
|
||||
# The chart no longer ships a Namespace resource (helm v3's --create-namespace
|
||||
# collides with chart-defined Namespace). On Talos clusters that enforce PSA
|
||||
# baseline-by-default, the cozystack-operator pod (hostNetwork=true) is
|
||||
# rejected unless the namespace is labelled enforce=privileged BEFORE the
|
||||
# pod is scheduled. A pre-install hook can't bootstrap that label because
|
||||
# the hook itself runs in the same namespace and faces the same PSA gate.
|
||||
# So apply the namespace from the caller (this bats / production install docs)
|
||||
# — same pattern as kube-prometheus-stack, argo-cd, cert-manager.
|
||||
kubectl apply -f - <<'EOF'
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: cozy-system
|
||||
labels:
|
||||
cozystack.io/system: "true"
|
||||
cozystack.io/deletion-protected: "true"
|
||||
pod-security.kubernetes.io/enforce: privileged
|
||||
EOF
|
||||
|
||||
# Install cozy-installer chart (operator installs CRDs on startup via --install-crds).
|
||||
# The chart's pre-install hook patches PSA + cozystack labels onto cozy-system
|
||||
# after --create-namespace bootstraps it.
|
||||
helm upgrade installer packages/core/installer \
|
||||
--install \
|
||||
--namespace cozy-system \
|
||||
--create-namespace \
|
||||
--set cozystackOperator.helmReleaseInterval=30s \
|
||||
--wait \
|
||||
--timeout 2m
|
||||
|
||||
@@ -0,0 +1,122 @@
|
||||
{{/*
|
||||
Pre-install/upgrade hook that labels the cozy-system namespace with the
|
||||
PodSecurity and cozystack identity labels.
|
||||
|
||||
Two chicken-and-egg constraints to thread:
|
||||
|
||||
1. helm v3 cannot adopt a namespace pre-created by --create-namespace
|
||||
because it lacks helm meta annotations; without --create-namespace,
|
||||
helm fails to write its release-secret because the namespace does
|
||||
not exist yet. The chart no longer ships a Namespace cozy-system
|
||||
resource; callers run helm with --create-namespace and the namespace
|
||||
is born bare (no labels).
|
||||
|
||||
2. The operator pod needs hostNetwork=true (no CNI yet at install time),
|
||||
which violates PodSecurity baseline. So cozy-system needs
|
||||
enforce=privileged before the operator pod is admitted. But a labeler
|
||||
pod inside cozy-system would itself need hostNetwork=true and hit
|
||||
the same admission denial. The hook therefore lives in kube-system,
|
||||
which is PSA-exempt on Talos (defaults.exemptions.namespaces in the
|
||||
apiserver PodSecurityConfiguration) and unlabeled on vanilla
|
||||
kubeadm/kind/k3s. The Job's SA is bound to a ClusterRole scoped via
|
||||
resourceNames to "cozy-system" only, so the cluster permission grant
|
||||
is minimal.
|
||||
*/}}
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: cozy-system-labeler
|
||||
namespace: kube-system
|
||||
annotations:
|
||||
helm.sh/hook: pre-install,pre-upgrade
|
||||
helm.sh/hook-weight: "-200"
|
||||
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: cozy-system-labeler
|
||||
annotations:
|
||||
helm.sh/hook: pre-install,pre-upgrade
|
||||
helm.sh/hook-weight: "-180"
|
||||
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["namespaces"]
|
||||
resourceNames: ["cozy-system"]
|
||||
verbs: ["get", "patch"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: cozy-system-labeler
|
||||
annotations:
|
||||
helm.sh/hook: pre-install,pre-upgrade
|
||||
helm.sh/hook-weight: "-160"
|
||||
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: cozy-system-labeler
|
||||
namespace: kube-system
|
||||
roleRef:
|
||||
kind: ClusterRole
|
||||
name: cozy-system-labeler
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
---
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: cozy-system-labeler
|
||||
namespace: kube-system
|
||||
annotations:
|
||||
helm.sh/hook: pre-install,pre-upgrade
|
||||
helm.sh/hook-weight: "-100"
|
||||
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
|
||||
spec:
|
||||
backoffLimit: 6
|
||||
template:
|
||||
spec:
|
||||
serviceAccountName: cozy-system-labeler
|
||||
restartPolicy: OnFailure
|
||||
# The hook runs BEFORE Cilium / kube-ovn / any CNI is installed, so the
|
||||
# cluster's nodes are NotReady-NoSchedule and have no pod network. Mirror
|
||||
# the cozystack-operator deployment's scheduling: hostNetwork=true so the
|
||||
# pod doesn't need CNI, plus the same tolerations the operator uses.
|
||||
hostNetwork: true
|
||||
tolerations:
|
||||
- key: "node.kubernetes.io/not-ready"
|
||||
operator: "Exists"
|
||||
- key: "node.kubernetes.io/unreachable"
|
||||
operator: "Exists"
|
||||
- key: "node.cilium.io/agent-not-ready"
|
||||
operator: "Exists"
|
||||
- key: "node.cloudprovider.kubernetes.io/uninitialized"
|
||||
operator: "Exists"
|
||||
containers:
|
||||
- name: kubectl
|
||||
image: alpine/k8s:1.32.0
|
||||
imagePullPolicy: IfNotPresent
|
||||
{{- if eq .Values.cozystackOperator.variant "talos" }}
|
||||
env:
|
||||
# Talos KubePrism endpoint — same as cozystack-operator uses to reach
|
||||
# the apiserver before CNI is up.
|
||||
- name: KUBERNETES_SERVICE_HOST
|
||||
value: "localhost"
|
||||
- name: KUBERNETES_SERVICE_PORT
|
||||
value: "7445"
|
||||
{{- else if eq .Values.cozystackOperator.variant "generic" }}
|
||||
env:
|
||||
- name: KUBERNETES_SERVICE_HOST
|
||||
value: {{ required "cozystack.apiServerHost is required in generic mode" .Values.cozystack.apiServerHost | quote }}
|
||||
- name: KUBERNETES_SERVICE_PORT
|
||||
value: {{ .Values.cozystack.apiServerPort | quote }}
|
||||
{{- end }}
|
||||
command:
|
||||
- kubectl
|
||||
- label
|
||||
- --overwrite
|
||||
- namespace
|
||||
- cozy-system
|
||||
- cozystack.io/system=true
|
||||
- pod-security.kubernetes.io/enforce=privileged
|
||||
Reference in new issue
Block a user