feat(keycloak): add optional KMS-encrypting database proxy (#3003)

## What this PR does

Adds an optional, flag-gated **column-level PII encryption** feature to
the `keycloak` system package. When `encryption.enabled=true`, a
`keycloak-kms-proxy` Deployment + Service is placed on the wire between
the Keycloak StatefulSet and its CloudNativePG database, and Keycloak's
`KC_DB_URL_HOST`/`KC_DB_URL_PORT` are repointed at the proxy. The proxy
transparently encrypts Keycloak PII columns (username, email, first/last
name, `pii-*` attributes, credential secrets) before they reach
PostgreSQL and decrypts them on reads; the data-encryption key is
wrapped by a KMS and never lives inside the database.

The proxy lives in its own repository:
https://github.com/cozystack/keycloak-kms-proxy (image
`ghcr.io/cozystack/keycloak-kms-proxy:v0.1.0`).

**KMS backend** is selectable via `encryption.kms.backend`:
- `static` (default): a self-generated 32-byte KEK kept in a Secret —
works on any cluster, suitable for evaluation.
- `vault-transit`: the KEK lives in HashiCorp Vault (recommended for
production; supports KEK rotation).

The feature is **off by default**, so existing installs are unaffected —
`KC_DB_URL_HOST` keeps sourcing from the CNPG `keycloak-db-app` secret.

### Changes
- `values.yaml`: new `encryption` block.
- `templates/proxy.yaml` (new): gated Deployment + Service (+
self-generated KEK Secret for the static backend), hardened pod
(nonroot, read-only rootfs, dropped caps, seccomp).
- `templates/sts.yaml`: conditional DB-endpoint override when encryption
is enabled.
- `tests/encryption_test.yaml` (new): helm-unittest covering on/off and
both KMS backends.

`helm unittest .` → 11/11 pass.

### Follow-ups (not in this draft)
- Backend TLS re-origination to the CNPG endpoint + CA validation.
- Liquibase bootstrap-window handling (`KKP_LENIENT`).
- Offline backfill Job for migrating existing data.
- HA via a shared DEK set (currently single replica).

### Screenshots

N/A — no UI changes.

### Release note

```release-note
feat(keycloak): add optional KMS-encrypting database proxy for column-level PII encryption (off by default; static KEK or Vault Transit)
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added optional transparent encryption for Keycloak database sensitive
fields
* Users can configure encryption scope and select key management backend
options

* **Tests**
* Added comprehensive test coverage validating encryption feature across
multiple configuration scenarios

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

packages/system/keycloak/templates/proxy.yaml

@@ -0,0 +1,163 @@
+{{- if .Values.encryption.enabled }}
+{{- $clusterDomain := (index .Values._cluster "cluster-domain") | default "cozy.local" }}
+{{- $kms := .Values.encryption.kms }}
+{{- /*
+  Fail fast on an unsupported backend. Both the KEK Secret below and the env
+  wiring further down key off $backend, so an invalid value (e.g. a "vault"
+  typo) can never render a Deployment that mounts KKP_KEK from a Secret that
+  was never created — which would silently break the whole IdP.
+*/}}
+{{- $backend := ($kms.backend | default "static") }}
+{{- if and (ne $backend "static") (ne $backend "vault-transit") }}
+{{- fail (printf "encryption.kms.backend must be 'static' or 'vault-transit', got %q" $backend) }}
+{{- end }}
+{{- if eq $backend "static" }}
+{{- /*
+  Static KMS: a self-generated 32-byte KEK kept in a Secret. Generated once
+  and preserved across upgrades via lookup (same pattern as the admin
+  credentials in sts.yaml). For production prefer kms.backend=vault-transit.
+*/}}
+{{- $existing := lookup "v1" "Secret" .Release.Namespace "keycloak-kms-proxy-kek" }}
+{{- $kek := randAlphaNum 32 | b64enc }}
+{{- if and $existing $existing.data }}
+{{- $kek = index $existing.data "kek" | default $kek }}
+{{- end }}
+apiVersion: v1
+kind: Secret
+metadata:
+  name: keycloak-kms-proxy-kek
+  labels:
+    app: keycloak-kms-proxy
+data:
+  kek: {{ $kek | quote }}
+---
+{{- end }}
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: keycloak-kms-proxy
+  labels:
+    app: keycloak-kms-proxy
+spec:
+  # Single replica: HA needs a shared wrapped DEK set (KKP_DEKSET_FILE) so
+  # every replica encrypts deterministic columns with the same key. Tracked
+  # as a follow-up.
+  replicas: 1
+  selector:
+    matchLabels:
+      app: keycloak-kms-proxy
+  template:
+    metadata:
+      labels:
+        app: keycloak-kms-proxy
+    spec:
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 65532
+        fsGroup: 65532
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: proxy
+          image: {{ .Values.encryption.image }}
+          imagePullPolicy: IfNotPresent
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop:
+                - ALL
+          ports:
+            - name: postgres
+              containerPort: 5432
+            - name: metrics
+              containerPort: 9090
+          env:
+            - name: KKP_LISTEN_ADDR
+              value: ":5432"
+            - name: KKP_METRICS_ADDR
+              value: ":9090"
+            # NOTE: the proxy re-originates the connection to keycloak-db-rw and
+            # negotiates TLS in-band, but does not yet pin the CNPG CA. Backend
+            # CA validation (sslrootcert) is a tracked follow-up before this is
+            # production-ready; see the keycloak-kms-proxy project.
+            - name: KKP_BACKEND_ADDR
+              value: "keycloak-db-rw.{{ .Release.Namespace }}.svc.{{ $clusterDomain }}:5432"
+            # The proxy terminates SCRAM on both legs. Keycloak still presents
+            # the CNPG-managed credential (KC_DB_USERNAME/KC_DB_PASSWORD come
+            # from keycloak-db-app), so the upstream and backend credentials
+            # are the same CNPG role.
+            - name: KKP_UPSTREAM_USER
+              valueFrom:
+                secretKeyRef:
+                  name: keycloak-db-app
+                  key: username
+            - name: KKP_UPSTREAM_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: keycloak-db-app
+                  key: password
+            - name: KKP_BACKEND_USER
+              valueFrom:
+                secretKeyRef:
+                  name: keycloak-db-app
+                  key: username
+            - name: KKP_BACKEND_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: keycloak-db-app
+                  key: password
+            - name: KKP_FIELDS
+              value: {{ .Values.encryption.fields | quote }}
+            - name: KKP_LENIENT
+              value: {{ .Values.encryption.lenient | quote }}
+            {{- if eq $backend "vault-transit" }}
+            {{- $vault := $kms.vault | default dict }}
+            - name: KKP_VAULT_ADDR
+              value: {{ required "encryption.kms.vault.address is required when kms.backend=vault-transit" $vault.address | quote }}
+            - name: KKP_VAULT_KEY_NAME
+              value: {{ required "encryption.kms.vault.keyName is required when kms.backend=vault-transit" $vault.keyName | quote }}
+            - name: KKP_VAULT_MOUNT
+              value: {{ $vault.mount | default "transit" | quote }}
+            - name: KKP_VAULT_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: {{ required "encryption.kms.vault.tokenSecretName is required when kms.backend=vault-transit" $vault.tokenSecretName }}
+                  key: token
+            {{- else }}
+            - name: KKP_KEK
+              valueFrom:
+                secretKeyRef:
+                  name: keycloak-kms-proxy-kek
+                  key: kek
+            {{- end }}
+          {{- with .Values.encryption.resources }}
+          resources:
+            {{- toYaml . | nindent 12 }}
+          {{- end }}
+          readinessProbe:
+            tcpSocket:
+              port: postgres
+            periodSeconds: 10
+          livenessProbe:
+            tcpSocket:
+              port: postgres
+            periodSeconds: 15
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: keycloak-kms-proxy
+  labels:
+    app: keycloak-kms-proxy
+spec:
+  selector:
+    app: keycloak-kms-proxy
+  ports:
+    - name: postgres
+      port: 5432
+      targetPort: postgres
+    - name: metrics
+      port: 9090
+      targetPort: metrics
+{{- end }}

packages/system/keycloak/templates/sts.yaml

@@ -156,24 +156,27 @@ spec:
                   key: password
             - name: KC_DB
               value: "postgres"
+            {{- if .Values.encryption.enabled }}
+            # Encryption on: route through the keycloak-kms-proxy, which dials
+            # the real CNPG cluster. Auth (username/password/dbname) still comes
+            # from the CNPG-managed keycloak-db-app secret below and passes
+            # through the proxy unchanged.
+            - name: KC_DB_URL_HOST
+              value: keycloak-kms-proxy.{{ .Release.Namespace }}.svc.{{ $clusterDomain }}
+            - name: KC_DB_URL_PORT
+              value: "5432"
+            {{- else }}
             - name: KC_DB_URL_HOST
-              {{- if .Values.db.host }}
-              value: {{ .Values.db.host | quote }}
-              {{- else }}
               valueFrom:
                 secretKeyRef:
                   name: keycloak-db-app
                   key: "host"
-              {{- end }}
             - name: KC_DB_URL_PORT
-              {{- if .Values.db.port }}
-              value: {{ .Values.db.port | quote }}
-              {{- else }}
               valueFrom:
                 secretKeyRef:
                   name: keycloak-db-app
                   key: "port"
-              {{- end }}
+            {{- end }}
             - name: KC_DB_USERNAME
               valueFrom:
                 secretKeyRef:
@@ -189,10 +192,6 @@ spec:
                 secretKeyRef:
                   name: keycloak-db-app
                   key: "dbname"
-            {{- with .Values.db.urlProperties }}
-            - name: KC_DB_URL_PROPERTIES
-              value: {{ . | quote }}
-            {{- end }}
             - name: KC_FEATURES
               value: "docker"
             - name: KC_HOSTNAME
@@ -206,15 +205,10 @@ spec:
             {{- with .Values.extraEnv }}
             {{- toYaml . | nindent 12 }}
             {{- end }}
-          {{- if or $themes .Values.extraVolumeMounts }}
+          {{- if $themes }}
           volumeMounts:
-            {{- if $themes }}
             - name: themes
               mountPath: /opt/keycloak/themes
-            {{- end }}
-            {{- with .Values.extraVolumeMounts }}
-            {{- toYaml . | nindent 12 }}
-            {{- end }}
           {{- end }}
           ports:
             - name: http
@@ -243,15 +237,10 @@ spec:
             periodSeconds: 10
             timeoutSeconds: 5
             failureThreshold: 3
-      {{- if or $themes .Values.extraVolumes }}
+      {{- if $themes }}
       volumes:
-        {{- if $themes }}
         - name: themes
           emptyDir:
             sizeLimit: 256Mi
-        {{- end }}
-        {{- with .Values.extraVolumes }}
-        {{- toYaml . | nindent 8 }}
-        {{- end }}
       {{- end }}
       terminationGracePeriodSeconds: 60

packages/system/keycloak/tests/encryption_test.yaml

@@ -0,0 +1,222 @@
+suite: keycloak optional KMS-encrypting DB proxy
+templates:
+  - templates/proxy.yaml
+  - templates/sts.yaml
+
+release:
+  name: keycloak
+  namespace: cozy-keycloak
+
+tests:
+  - it: does not render the proxy when encryption is disabled (default)
+    template: templates/proxy.yaml
+    set:
+      _cluster:
+        root-host: example.org
+        cluster-domain: cozy.local
+    asserts:
+      - hasDocuments:
+          count: 0
+
+  - it: keeps Keycloak pointed at the CNPG secret when encryption is disabled
+    template: templates/sts.yaml
+    documentSelector:
+      path: kind
+      value: StatefulSet
+    set:
+      _cluster:
+        root-host: example.org
+        cluster-domain: cozy.local
+    asserts:
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: KC_DB_URL_HOST
+            valueFrom:
+              secretKeyRef:
+                name: keycloak-db-app
+                key: "host"
+
+  - it: renders three documents (KEK Secret + Deployment + Service) when enabled
+    template: templates/proxy.yaml
+    set:
+      encryption.enabled: true
+      _cluster:
+        root-host: example.org
+        cluster-domain: cozy.local
+    asserts:
+      - hasDocuments:
+          count: 3
+
+  - it: deploys the proxy wired to the static KEK and the CNPG backend
+    template: templates/proxy.yaml
+    documentSelector:
+      path: kind
+      value: Deployment
+    set:
+      encryption.enabled: true
+      _cluster:
+        root-host: example.org
+        cluster-domain: cozy.local
+    asserts:
+      - equal:
+          path: metadata.name
+          value: keycloak-kms-proxy
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: KKP_KEK
+            valueFrom:
+              secretKeyRef:
+                name: keycloak-kms-proxy-kek
+                key: kek
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: KKP_BACKEND_ADDR
+            value: keycloak-db-rw.cozy-keycloak.svc.cozy.local:5432
+
+  - it: generates the static KEK Secret
+    template: templates/proxy.yaml
+    documentSelector:
+      path: kind
+      value: Secret
+    set:
+      encryption.enabled: true
+      _cluster:
+        root-host: example.org
+        cluster-domain: cozy.local
+    asserts:
+      - equal:
+          path: metadata.name
+          value: keycloak-kms-proxy-kek
+
+  - it: exposes the proxy on a postgres Service
+    template: templates/proxy.yaml
+    documentSelector:
+      path: kind
+      value: Service
+    set:
+      encryption.enabled: true
+      _cluster:
+        root-host: example.org
+        cluster-domain: cozy.local
+    asserts:
+      - equal:
+          path: metadata.name
+          value: keycloak-kms-proxy
+      - equal:
+          path: spec.ports[0].port
+          value: 5432
+
+  - it: repoints Keycloak at the proxy Service when encryption is enabled
+    template: templates/sts.yaml
+    documentSelector:
+      path: kind
+      value: StatefulSet
+    set:
+      encryption.enabled: true
+      _cluster:
+        root-host: example.org
+        cluster-domain: cozy.local
+    asserts:
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: KC_DB_URL_HOST
+            value: keycloak-kms-proxy.cozy-keycloak.svc.cozy.local
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: KC_DB_URL_PORT
+            value: "5432"
+
+  - it: uses the Vault Transit backend when selected
+    template: templates/proxy.yaml
+    documentSelector:
+      path: kind
+      value: Deployment
+    set:
+      encryption.enabled: true
+      encryption.kms.backend: vault-transit
+      encryption.kms.vault.address: https://vault.vault.svc:8200
+      encryption.kms.vault.keyName: keycloak
+      encryption.kms.vault.tokenSecretName: keycloak-vault-token
+      _cluster:
+        root-host: example.org
+        cluster-domain: cozy.local
+    asserts:
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: KKP_VAULT_ADDR
+            value: https://vault.vault.svc:8200
+
+  - it: does not generate a static KEK Secret for the Vault backend
+    template: templates/proxy.yaml
+    set:
+      encryption.enabled: true
+      encryption.kms.backend: vault-transit
+      encryption.kms.vault.address: https://vault.vault.svc:8200
+      encryption.kms.vault.keyName: keycloak
+      encryption.kms.vault.tokenSecretName: keycloak-vault-token
+      _cluster:
+        root-host: example.org
+        cluster-domain: cozy.local
+    asserts:
+      - hasDocuments:
+          count: 2
+
+  - it: rejects an unsupported kms backend at render time
+    template: templates/proxy.yaml
+    set:
+      encryption.enabled: true
+      encryption.kms.backend: vault
+      _cluster:
+        root-host: example.org
+        cluster-domain: cozy.local
+    asserts:
+      - failedTemplate:
+          errorPattern: "encryption.kms.backend must be 'static' or 'vault-transit'"
+
+  - it: requires vault.address for the vault-transit backend
+    template: templates/proxy.yaml
+    set:
+      encryption.enabled: true
+      encryption.kms.backend: vault-transit
+      encryption.kms.vault.keyName: keycloak
+      encryption.kms.vault.tokenSecretName: keycloak-vault-token
+      _cluster:
+        root-host: example.org
+        cluster-domain: cozy.local
+    asserts:
+      - failedTemplate:
+          errorPattern: "encryption.kms.vault.address is required"
+
+  - it: requires vault.keyName for the vault-transit backend
+    template: templates/proxy.yaml
+    set:
+      encryption.enabled: true
+      encryption.kms.backend: vault-transit
+      encryption.kms.vault.address: https://vault.vault.svc:8200
+      encryption.kms.vault.tokenSecretName: keycloak-vault-token
+      _cluster:
+        root-host: example.org
+        cluster-domain: cozy.local
+    asserts:
+      - failedTemplate:
+          errorPattern: "encryption.kms.vault.keyName is required"
+
+  - it: requires vault.tokenSecretName for the vault-transit backend
+    template: templates/proxy.yaml
+    set:
+      encryption.enabled: true
+      encryption.kms.backend: vault-transit
+      encryption.kms.vault.address: https://vault.vault.svc:8200
+      encryption.kms.vault.keyName: keycloak
+      _cluster:
+        root-host: example.org
+        cluster-domain: cozy.local
+    asserts:
+      - failedTemplate:
+          errorPattern: "encryption.kms.vault.tokenSecretName is required"

packages/system/keycloak/values.yaml

@@ -62,28 +62,57 @@ extraEnv: []
 #       name: my-secret
 #       key: my-key
 
-# Database connection. By default Keycloak connects directly to the chart's
-# CNPG cluster using the keycloak-db-app secret. Override these to route the
-# connection through an external endpoint (e.g. a connection proxy such as
-# PgBouncer) without the chart needing to know about it.
-db:
-  # DB host. Empty = use the keycloak-db-app secret's "host" key.
-  host: ""
-  # DB port. Empty = use the keycloak-db-app secret's "port" key.
-  port: ""
-  # Extra JDBC URL properties, appended verbatim as KC_DB_URL_PROPERTIES, e.g.
-  #   "?sslmode=verify-full&prepareThreshold=0&sslrootcert=/etc/keycloak-db-ca/ca.crt"
-  # Empty = KC_DB_URL_PROPERTIES is not set.
-  urlProperties: ""
-
-# Extra volumes and volume mounts on the Keycloak container (in addition to
-# themes). Useful for mounting a CA so KC_DB_URL_PROPERTIES can reference an
-# sslrootcert when routing the DB connection through a proxy.
-extraVolumes: []
-# - name: keycloak-db-ca
-#   secret:
-#     secretName: keycloak-db-ca
-extraVolumeMounts: []
-# - name: keycloak-db-ca
-#   mountPath: /etc/keycloak-db-ca
-#   readOnly: true
+# Transparent column-level PII encryption for the Keycloak database.
+# When enabled, a keycloak-kms-proxy Deployment is placed on the wire
+# between Keycloak and its CloudNativePG database: Keycloak's DB endpoint
+# is repointed at the proxy, which encrypts designated PII columns on
+# writes and decrypts them on reads. The data-encryption key is wrapped by
+# a KMS, so it never lives inside PostgreSQL.
+# Project: https://github.com/cozystack/keycloak-kms-proxy
+#
+# OPERATIONAL CAVEATS — read before enabling:
+#   * Availability downgrade. The proxy runs as a single replica (HA needs a
+#     shared wrapped DEK set, tracked as a follow-up). Keycloak itself runs HA
+#     (2 replicas); with encryption on, both replicas funnel their DB traffic
+#     through this one pod, so any proxy restart — node drain, rollout, image
+#     pull, OOM — briefly drops the database connection for the entire IdP.
+#   * Existing data is NOT migrated automatically. Flipping enabled:true on a
+#     populated database leaves the pre-existing PII as plaintext; logins that
+#     match on the deterministic columns (username/email) then fail until the
+#     offline backfill has run. Disabling later is just as unsafe — Keycloak
+#     would point back at CNPG and read ciphertext directly. Off-by-default only
+#     protects FRESH installs. To enable on an existing install, run the backfill
+#     procedure first:
+#     https://github.com/cozystack/keycloak-kms-proxy/blob/main/docs/migration-existing-keycloak.md
+#   * Static KEK = single point of permanent data loss. With kms.backend=static
+#     the KEK is generated once and kept in the keycloak-kms-proxy-kek Secret
+#     (in etcd, next to the DB credentials — "static" trades some at-rest
+#     protection for zero external dependencies). If that Secret is ever lost,
+#     every previously-encrypted value becomes permanently undecryptable, so it
+#     MUST be part of the backup set. For production prefer kms.backend=vault-transit.
+encryption:
+  enabled: false
+  image: ghcr.io/cozystack/keycloak-kms-proxy:v0.1.0
+  # Encrypted field set: "" = full default set (username, email, names,
+  # PII attributes, credentials), "email-only", or "disabled" (passthrough).
+  fields: ""
+  # Bootstrap window only: downgrades fail-loud rules to passthrough so the
+  # Keycloak Liquibase migrations can complete. MUST be false in steady state.
+  lenient: false
+  kms:
+    # "static" — a self-generated 32-byte KEK kept in a Secret (works on any
+    # cluster, suitable for evaluation). "vault-transit" — the KEK lives in
+    # HashiCorp Vault (recommended for production; supports KEK rotation).
+    backend: static
+    vault:
+      address: ""       # e.g. https://vault.vault.svc:8200
+      keyName: ""       # Transit key name
+      mount: transit    # Transit mount path
+      # Existing Secret holding the Vault token under key "token".
+      tokenSecretName: ""
+  resources:
+    requests:
+      memory: 128Mi
+      cpu: 50m
+    limits:
+      memory: 256Mi