From 542ab899c2dff4df64eaf660293f70d901ec8607 Mon Sep 17 00:00:00 2001 From: IvanHunters Date: Mon, 17 Aug 2026 13:44:09 +0300 Subject: [PATCH] fix(kubernetes): break bootstrap-token Job out of Helm hook to unblock fresh installs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The bootstrap-token-tenant Job was wired as a Helm post-install/post-upgrade hook. post-* hooks fire only after Helm's wait phase resolves, and the tenant kubernetes release wait blocks on MachineDeployment Ready, which blocks on nodes registering, which needs the kubelet CSR path — for which the very bootstrap-token this Job produces has to already exist in the tenant kube-system. Fresh installs and any HR reconcile that finds MD in a non-Ready state deadlock; HR.upgradeFailures climbs while post-upgrade hooks never get their turn to run. pre-install/pre-upgrade would not help either: they fire before non-hook templates apply, so -admin-kubeconfig (materialised by Kamaji after KamajiControlPlane is applied) is not yet available. Move the ServiceAccount and Job to the main install phase, matching what talos-reconcile already does in this chart. Helm applies them together with the KamajiControlPlane and talos-secrets; the Job Pod stays pending until Kamaji publishes the admin kubeconfig Secret (backoffLimit covers the retry window) and then completes before MachineDeployment can converge. Helm's wait treats the Job as another resource to reach terminal state and does not gate it on MD. Extend the existing template test to pin the absence of helm.sh/hook on both the ServiceAccount and the Job, so this cannot regress silently. Signed-off-by: IvanHunters --- .../talos/bootstrap-token-tenant-job.yaml | 27 ++++++++++--------- .../tests/talos_templates_test.yaml | 15 +++++++++++ 2 files changed, 30 insertions(+), 12 deletions(-) diff --git a/packages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yaml b/packages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yaml index b2f23e7c6..79daaa432 100644 --- a/packages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yaml +++ b/packages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yaml @@ -8,10 +8,21 @@ kube-system, which the management cluster can only reach via the Kamaji-issued admin kubeconfig stored as -admin-kubeconfig. - The Job runs as a Helm post-install/post-upgrade hook so it executes - after the KamajiControlPlane has reconciled and the admin kubeconfig - Secret exists. It is idempotent (kubectl apply) and re-runs harmlessly - on every upgrade. + Why not a Helm post-install/post-upgrade hook (the earlier shape): + post-* hooks execute only after Helm's wait phase resolves. On the + tenant `kubernetes` release the wait blocks on MachineDeployment Ready, + which blocks on nodes registering, which blocks on kubelet CSR + authentication, which needs the bootstrap-token this Job produces — + a hard deadlock on every fresh install and on every HR reconcile that + finds MD in a non-Ready state. pre-* hooks are not usable either: they + fire before non-hook templates apply, so -admin-kubeconfig + (materialised by Kamaji after KamajiControlPlane is applied) is not + yet available. The workable shape is a regular resource in the chart: + it lands together with the KamajiControlPlane and the talos-secrets, + its Pod stays pending until Kamaji publishes the admin kubeconfig + Secret (backoffLimit covers the retry window), then runs and completes + before MachineDeployment can converge. Helm's wait treats the Job as + another resource to reach terminal state and does not gate it on MD. */}} --- apiVersion: v1 @@ -21,10 +32,6 @@ metadata: namespace: {{ .Release.Namespace }} labels: {{- include "kubernetes.labels" . | nindent 4 }} - annotations: - "helm.sh/hook": post-install,post-upgrade - "helm.sh/hook-weight": "0" - "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded --- apiVersion: batch/v1 kind: Job @@ -33,10 +40,6 @@ metadata: namespace: {{ .Release.Namespace }} labels: {{- include "kubernetes.labels" . | nindent 4 }} - annotations: - "helm.sh/hook": post-install,post-upgrade - "helm.sh/hook-weight": "10" - "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded spec: # backoffLimit + restartPolicy: OnFailure gives the Job up to 11 attempts at # the tenant apiserver with exponential backoff (capped at 6 min between diff --git a/packages/apps/kubernetes/tests/talos_templates_test.yaml b/packages/apps/kubernetes/tests/talos_templates_test.yaml index 5dcc9f9d5..92783fa2f 100644 --- a/packages/apps/kubernetes/tests/talos_templates_test.yaml +++ b/packages/apps/kubernetes/tests/talos_templates_test.yaml @@ -362,6 +362,21 @@ tests: path: metadata.name value: test-k8s-bootstrap-token-tenant documentIndex: 1 + # Main-phase, not a hook: NONE of the bootstrap-token resources may + # carry helm.sh/hook. The Job must run at t0 alongside the tenant + # KamajiControlPlane so that the bootstrap-token- Secret lands + # in the tenant kube-system before MachineDeployment starts + # scaling. post-install/post-upgrade hooks would gate this on + # Helm's wait phase, which itself waits on MachineDeployment + # Ready, which waits on nodes to register, which waits on the + # very token this Job produces — a deadlock. Pin the absence on + # both documents (0 SA, 1 Job). + - notExists: + path: metadata.annotations["helm.sh/hook"] + documentIndex: 0 + - notExists: + path: metadata.annotations["helm.sh/hook"] + documentIndex: 1 - it: talos-pki emits two cert-manager Certificates and two Issuers templates: