fix(apps/kubernetes): release-own the OIDC SA/Role/RoleBinding + pre-render the kubeconfig Secret

Self-review fixes for four linked lifecycle issues:

  - The post-install Job created `<release>-oidc-kubeconfig` via
    `kubectl apply` from inside the script; the resulting Secret had
    no Helm release labels and was orphaned on `helm uninstall`. Now
    a release-owned placeholder Secret is rendered up front in
    `oidc-kubeconfig-secret.yaml` and the Job patches its `data`
    field at runtime (the tenant CA still has to come from the
    Kamaji-issued admin kubeconfig, which appears asynchronously).
  - The ServiceAccount/Role/RoleBinding for the bootstrap Job were
    hook resources with `helm.sh/hook-delete-policy: before-hook-creation`,
    so they leaked on `helm uninstall`. They are now regular release
    resources (no hook annotations) and are reaped by `helm uninstall`
    alongside everything else. Both the post-install bootstrap Job
    and the pre-delete cleanup Job find a working SA.
  - The previous design tried to add `hook-succeeded` to the SA, but
    that would delete the SA right after the bootstrap hook
    succeeded — leaving the pre-delete cleanup Job with no SA to
    schedule. Release-owned resources sidestep this race entirely.
  - The Role had `verbs: ["create"]` on `secrets` without
    `resourceNames` (Kubernetes RBAC cannot scope `create` by name).
    With the placeholder Secret pre-rendered the Job only needs
    `get`/`patch` on the specific name — the broad `create` rule is
    removed.

Also tightens the script: `set -euo pipefail` (was `set -eu`, which
silently swallowed `kubectl get` failures inside command
substitutions) and uses `kubectl patch --type=merge` against the
existing Secret instead of `kubectl create --dry-run | kubectl apply`.

Adds three helm-unittest cases for the placeholder Secret across all
three modes.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>

packages/apps/kubernetes/templates/oidc-kubeconfig-secret.yaml

@@ -0,0 +1,31 @@
+{{- /*
+  Pre-render the OIDC kubeconfig Secret as a chart-owned resource so
+  `helm uninstall` cleans it up. The bootstrap Job patches the `data`
+  field at runtime with the actual kubeconfig (it cannot be rendered
+  at template time because the tenant CA is generated asynchronously
+  by Kamaji).
+
+  Emitted only for `mode: System`; in `CustomConfig` mode the chart
+  does not know the issuer/clientId — the operator distributes the
+  OIDC kubeconfig out-of-band from their own IdP configuration.
+
+  The placeholder content keeps the Secret addressable by the
+  dashboard immediately on install; the bootstrap Job swaps it for a
+  real kubeconfig as soon as the Kamaji-issued admin kubeconfig is
+  available.
+*/}}
+{{- if eq .Values.oidc.mode "System" }}
+apiVersion: v1
+kind: Secret
+metadata:
+  name: {{ .Release.Name }}-oidc-kubeconfig
+  namespace: {{ .Release.Namespace }}
+  labels:
+    {{- include "kubernetes.labels" . | nindent 4 }}
+type: Opaque
+stringData:
+  kubeconfig: |
+    # awaiting OIDC bootstrap Job — the tenant CA is generated by
+    # Kamaji asynchronously; the Job populates this Secret as soon as
+    # the admin kubeconfig appears.
+{{- end }}

packages/apps/kubernetes/templates/oidc-rbac-job.yaml

@@ -10,14 +10,26 @@
     2. Remove ClusterRoleBindings labelled by the release that are no
        longer in `users[]` — toggling a user out of the list revokes
        their kubectl access on the next reconcile.
-    3. For `mode: System`, also write a ready-to-use OIDC kubeconfig
-       Secret in the management cluster's tenant namespace
-       (`<release>-oidc-kubeconfig`) wired to `kubectl oidc-login` so
-       a user only needs to run `kubectl --kubeconfig … get pods` to
-       trigger the browser flow.
+    3. For `mode: System`, patch the pre-rendered
+       `<release>-oidc-kubeconfig` Secret with a ready-to-use
+       kubeconfig wired to `kubectl oidc-login` (the tenant CA is
+       generated by Kamaji asynchronously, so the Job extracts it
+       from the admin kubeconfig at runtime).
 
-  Rendered only when `spec.oidc.mode` is not None. Skipped entirely
-  for `mode: None`.
+  Lifecycle:
+    - ServiceAccount, Role, RoleBinding are RELEASE-OWNED (no
+      `helm.sh/hook` annotations). They are created on `helm install`
+      and deleted on `helm uninstall`, so both the post-install
+      bootstrap hook AND the pre-delete cleanup hook find a working
+      SA. This avoids the race where a hook-scoped SA is deleted
+      before pre-delete fires.
+    - The bootstrap Job itself stays a hook so `helm upgrade`
+      re-runs it on every reconcile (picks up `users[]` changes).
+    - The Secret it patches is pre-rendered in
+      `oidc-kubeconfig-secret.yaml` and is also release-owned, so
+      `helm uninstall` cleans it up.
+
+  Rendered only when `spec.oidc.mode` is not None.
 
   Pod is labelled `policy.cozystack.io/allow-to-apiserver: "true"` so
   the Cilium policy lets it reach the tenant apiserver through
@@ -34,10 +46,6 @@ kind: ServiceAccount
 metadata:
   name: {{ .Release.Name }}-oidc-bootstrap
   namespace: {{ .Release.Namespace }}
-  annotations:
-    "helm.sh/hook": post-install,post-upgrade
-    "helm.sh/hook-weight": "0"
-    "helm.sh/hook-delete-policy": before-hook-creation
   labels:
     {{- include "kubernetes.labels" . | nindent 4 }}
 ---
@@ -46,36 +54,20 @@ kind: Role
 metadata:
   name: {{ .Release.Name }}-oidc-bootstrap
   namespace: {{ .Release.Namespace }}
-  annotations:
-    "helm.sh/hook": post-install,post-upgrade
-    "helm.sh/hook-weight": "1"
-    "helm.sh/hook-delete-policy": before-hook-creation
   labels:
     {{- include "kubernetes.labels" . | nindent 4 }}
 rules:
-  - apiGroups: [""]
-    resources: ["secrets"]
-    resourceNames:
-      - {{ .Release.Name }}-admin-kubeconfig
-    verbs: ["get"]
   - apiGroups: [""]
     resources: ["secrets"]
     resourceNames:
       - {{ .Release.Name }}-oidc-kubeconfig
-    verbs: ["get", "create", "update", "patch"]
-  - apiGroups: [""]
-    resources: ["secrets"]
-    verbs: ["create"]
+    verbs: ["get", "patch"]
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: RoleBinding
 metadata:
   name: {{ .Release.Name }}-oidc-bootstrap
   namespace: {{ .Release.Namespace }}
-  annotations:
-    "helm.sh/hook": post-install,post-upgrade
-    "helm.sh/hook-weight": "2"
-    "helm.sh/hook-delete-policy": before-hook-creation
   labels:
     {{- include "kubernetes.labels" . | nindent 4 }}
 roleRef:
@@ -151,7 +143,7 @@ spec:
             - /bin/sh
             - -c
             - |
-              set -eu
+              set -euo pipefail
               echo "Waiting for the tenant admin kubeconfig to appear..."
               deadline=$(( $(date +%s) + 600 ))
               until [ -s "$KUBECONFIG_TENANT" ]; do
@@ -192,9 +184,10 @@ spec:
               {{- end }}
 
               echo "Pruning stale ClusterRoleBindings labelled by this release..."
-              for crb in $(kubectl --kubeconfig "$KUBECONFIG_TENANT" get clusterrolebindings \
-                  --selector=app.kubernetes.io/managed-by=cozystack-oidc,app.kubernetes.io/instance=$RELEASE \
-                  --output=jsonpath='{.items[*].metadata.name}'); do
+              EXISTING=$(kubectl --kubeconfig "$KUBECONFIG_TENANT" get clusterrolebindings \
+                --selector=app.kubernetes.io/managed-by=cozystack-oidc,app.kubernetes.io/instance=$RELEASE \
+                --output=jsonpath='{.items[*].metadata.name}')
+              for crb in $EXISTING; do
                 if ! grep --line-regexp --quiet --fixed-strings "$crb" "$DESIRED_FILE"; then
                   echo "  removing $crb"
                   kubectl --kubeconfig "$KUBECONFIG_TENANT" delete clusterrolebinding "$crb" --ignore-not-found
@@ -203,11 +196,11 @@ spec:
               rm --force "$DESIRED_FILE"
 
               if [ "$OIDC_MODE" = "System" ]; then
-                echo "Writing the OIDC kubeconfig Secret on the management cluster..."
+                echo "Patching the OIDC kubeconfig Secret on the management cluster..."
                 CA_DATA=$(kubectl --kubeconfig "$KUBECONFIG_TENANT" config view --raw \
                   --output=jsonpath='{.clusters[0].cluster.certificate-authority-data}')
                 if [ -z "$CA_DATA" ]; then
-                  echo "WARNING: tenant admin kubeconfig has no certificate-authority-data; OIDC kubeconfig will not be written" >&2
+                  echo "WARNING: tenant admin kubeconfig has no certificate-authority-data; leaving OIDC kubeconfig Secret as the install-time placeholder" >&2
                 else
                   KUBECONFIG_FILE=$(mktemp)
                   cat > "$KUBECONFIG_FILE" <<EOF
@@ -240,11 +233,12 @@ spec:
                   cluster: $RELEASE
                   user: $RELEASE-oidc
               EOF
-                  kubectl --namespace "$RELEASE_NAMESPACE" create secret generic \
+                  KUBECONFIG_B64=$(base64 -w 0 < "$KUBECONFIG_FILE")
+                  PATCH=$(printf '{"data":{"kubeconfig":"%s"},"stringData":null}' "$KUBECONFIG_B64")
+                  kubectl --namespace "$RELEASE_NAMESPACE" patch secret \
                     "$RELEASE-oidc-kubeconfig" \
-                    --from-file=kubeconfig="$KUBECONFIG_FILE" \
-                    --dry-run=client --output=yaml \
-                    | kubectl apply --filename -
+                    --type=merge \
+                    --patch "$PATCH"
                   rm --force "$KUBECONFIG_FILE"
                 fi
               fi
@@ -265,10 +259,11 @@ spec:
           emptyDir: {}
 {{- /*
   Pre-delete cleanup: remove the release's ClusterRoleBindings inside
-  the tenant cluster before the cluster itself is torn down. The
-  tenant cluster is going away in seconds, so this is best-effort —
-  if it cannot reach the apiserver the orphan CRBs disappear with the
-  control plane.
+  the tenant cluster before the cluster itself is torn down. Reuses
+  the release-owned ServiceAccount above (NOT a hook resource, so it
+  is still present when this hook fires). Best-effort — if the tenant
+  apiserver is unreachable the orphan CRBs disappear with the control
+  plane anyway.
 */}}
 ---
 apiVersion: batch/v1
@@ -325,7 +320,7 @@ spec:
             - /bin/sh
             - -c
             - |
-              set -eu
+              set -euo pipefail
               if [ ! -s "$KUBECONFIG_TENANT" ]; then
                 echo "tenant kubeconfig not present; nothing to clean up"
                 exit 0

packages/apps/kubernetes/tests/oidc_test.yaml

@@ -61,6 +61,15 @@ tests:
       - hasDocuments:
           count: 0
 
+  - it: emits no OIDC kubeconfig placeholder Secret when oidc.mode is None
+    release:
+      name: test
+      namespace: tenant-test
+    template: templates/oidc-kubeconfig-secret.yaml
+    asserts:
+      - hasDocuments:
+          count: 0
+
   ###############################################
   # mode: System — happy path                   #
   ###############################################
@@ -135,7 +144,7 @@ tests:
           value: oidc-audience-mapper
         documentIndex: 0
       - equal:
-          path: spec.protocolMappers[0].config["included.custom.audience"]
+          path: spec.protocolMappers[0].config["included.client.audience"]
           value: tenant-foo-kubernetes-app
         documentIndex: 0
 
@@ -186,13 +195,13 @@ tests:
     asserts:
       - contains:
           path: spec.apiServer.extraArgs
-          content: "--authentication-config=/etc/kubernetes/cozy-oidc/config.yaml"
+          content: "--authentication-config=/etc/kubernetes/authentication-config/config.yaml"
         documentIndex: 1
       - contains:
           path: spec.apiServer.extraVolumeMounts
           content:
-            name: cozy-oidc-authn-config
-            mountPath: /etc/kubernetes/cozy-oidc
+            name: authentication-config
+            mountPath: /etc/kubernetes/authentication-config
             readOnly: true
         documentIndex: 1
       - lengthEqual:
@@ -201,13 +210,35 @@ tests:
         documentIndex: 1
       - equal:
           path: spec.deployment.extraVolumes[2].name
-          value: cozy-oidc-authn-config
+          value: authentication-config
         documentIndex: 1
       - equal:
           path: spec.deployment.extraVolumes[2].secret.secretName
           value: kubernetes-app-oidc-authn-config
         documentIndex: 1
 
+  - it: System mode pre-renders the OIDC kubeconfig Secret as a release-owned placeholder
+    release:
+      name: kubernetes-app
+      namespace: tenant-foo
+    template: templates/oidc-kubeconfig-secret.yaml
+    set:
+      _cluster.oidc-enabled: "true"
+      _cluster.root-host: example.org
+      oidc.mode: System
+    asserts:
+      - isKind:
+          of: Secret
+      - equal:
+          path: metadata.name
+          value: kubernetes-app-oidc-kubeconfig
+      - equal:
+          path: metadata.labels["app.kubernetes.io/managed-by"]
+          value: Helm
+      - matchRegex:
+          path: stringData.kubeconfig
+          pattern: "awaiting OIDC bootstrap Job"
+
   - it: System mode hard-fails when platform OIDC is off
     release:
       name: kubernetes-app
@@ -339,6 +370,18 @@ tests:
           value: my-byo-authn
         documentIndex: 1
 
+  - it: CustomConfig mode does not pre-render the OIDC kubeconfig Secret
+    release:
+      name: kubernetes-app
+      namespace: tenant-foo
+    template: templates/oidc-kubeconfig-secret.yaml
+    set:
+      oidc.mode: CustomConfig
+      oidc.customConfig.config: "..."
+    asserts:
+      - hasDocuments:
+          count: 0
+
   - it: CustomConfig mode does not render any Keycloak objects
     release:
       name: kubernetes-app