The aggregation-layer section said System-mode OIDC users have no UID for the published header to carry, and that mapping a claim for them was somebody else's change. Both are now false: the chart maps the sub claim. An operator upgrading a running cluster needs to know the identity of those users gained a field. Tenant RBAC never matched on it, but audit records and kubectl auth whoami show it, and an in-tenant webhook or extension server that read the empty value now reads the subject. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
23 KiB
OIDC for tenant Kubernetes clusters (Phase 1)
Cozystack tenant Kubernetes clusters opt in to OIDC authentication on their
kube-apiserver through a flat selector on the Kubernetes CR. This document
covers the operator-facing surface: what the modes mean, what the chart
provisions on either end, and how to give a user kubectl access.
The architectural rationale lives in cozystack/community#24 — in particular why per-tenant Keycloak realms are deliberately deferred to Phase 2.
Selector
apiVersion: apps.cozystack.io/v1alpha1
kind: Kubernetes
metadata:
name: prod
namespace: tenant-acme
spec:
oidc:
mode: System # System | CustomConfig | None (default)
users:
- email: alice@acme.example
role: admin # admin → cluster-admin
- email: bob@acme.example
role: view # view → view
Three modes:
- None — the only user-facing path is the static
<release>-admin-kubeconfigSecret (the Kamaji-mintedsuper-admin.svckubeconfig). This is the default; existing clusters render byte-identical to before. - System — the apiserver trusts the platform Keycloak realm (
cozyby default, set viaauthentication.oidc.realmName) via a per-cluster public client. Authenticates the users already in that realm (the one cozystack provisions). Zero-config default. - CustomConfig — the apiserver trusts a tenant-supplied OIDC issuer
directly.
cozyis not in the path. Use for BYO IdPs (Okta, Auth0, a customer's own Keycloak).
The users[] map is independent of the mode and drives per-user
ClusterRoleBindings inside the tenant cluster.
System mode
What the chart provisions, in the management cluster:
- KeycloakClient named
<namespace>-<release>in thecozyrealm —public: true, PKCE required, redirect URIs locked tolocalhost:8000andlocalhost:18000(the kubelogin /kubectl oidc-logindefaults). - KeycloakClientScope named
<namespace>-<release>-audiencecarrying anoidc-audience-mapperthat pinsid_token.audto the per-cluster clientId. This is the per-cluster isolation primitive: a token minted for cluster A is rejected by cluster B's apiserver. - A Secret
<release>-oidc-authn-configcarrying a structuredapiserver.config.k8s.io/v1beta1AuthenticationConfigurationwith the cozy realm issuer and the per-cluster audience. - A
--authentication-config=flag, mount, and volume on theKamajiControlPlanereferencing the Secret above. - A bootstrap Job (Helm
post-install,post-upgradehook) that applies oneClusterRoleBindingperusers[]entry inside the tenant cluster and writes a ready-to-use OIDC kubeconfig Secret on the management side (see below). - A
<release>-oidc-kubeconfigSecret in the tenant namespace carrying a kubeconfig with akubectl oidc-loginexec block, exposed to the dashboard viapackages/system/kubernetes-rd.
The structured authentication-config form (rather than the legacy
--oidc-* flags) is intentional: it accepts multiple issuers in a list,
inline private-CA PEM, and future Phase-2 issuers extend the same Secret
instead of fighting the chart shape.
CustomConfig mode
The tenant supplies the entire AuthenticationConfiguration. Two paths:
spec:
oidc:
mode: CustomConfig
customConfig:
config: |
apiVersion: apiserver.config.k8s.io/v1beta1
kind: AuthenticationConfiguration
jwt:
- issuer:
url: https://idp.acme.example
certificateAuthority: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
audiences:
- cozystack-prod
claimMappings:
username:
claim: email
prefix: ""
groups:
claim: groups
prefix: ""
users:
- email: alice@acme.example
role: admin
…or via an out-of-band Secret the operator has already created in the tenant namespace:
spec:
oidc:
mode: CustomConfig
customConfig:
secretRef:
name: acme-byo-authn-config # has key config.yaml
config and secretRef.name are mutually exclusive; the chart fails the
render if both — or neither — are set.
No Keycloak objects land in cozy; the chart writes only the
AuthenticationConfiguration Secret (or mounts the operator's) and the
RBAC Job. The <release>-oidc-kubeconfig helper Secret is NOT written
in CustomConfig mode: the issuer and clientId are inside the
operator-supplied config and are not knowable to the chart. Distribute
the OIDC kubeconfig out-of-band.
Aggregated API servers
Authenticating a user is only half of the path. A call that lands on an aggregated API server — an APIService backed by an extension server rather than by the core apiserver — reaches it over the aggregation layer, which forwards the caller's identity in request headers. The UID travels in X-Remote-Uid, which the aggregation layer sends whenever the RemoteRequestHeaderUID gate is on, and the extension server trusts that header only when the tenant kube-apiserver has published it under requestheader-uid-headers in the extension-apiserver-authentication ConfigMap. Without that key the extension server ignores the header, and the call is served as if the caller had no UID, even though kubectl against the core apiserver works fine. Anything on the far side that reads the UID, an authorizer or an audit record, sees an empty value.
Carrying the UID across that hop is a property of the aggregation layer, not of OIDC: a caller authenticated by any means loses their UID the same way. The chart nonetheless renders these flags only when spec.oidc.mode is not None, which keeps the blast radius to clusters opting into a new feature. A cluster that has never left mode: None therefore still loses the UID at the extension server.
Be clear about who this helps. The flag forwards a UID the caller already has; it does not create one. ServiceAccount tokens carry a UID, and so do other identities that set one, and for those the aggregation hop starts working. An OIDC user gets one only from a claim mapping: in spec.oidc.mode: System the AuthenticationConfiguration the chart generates maps uid to the sub claim, the Keycloak account UUID, so those users reach an aggregated API server under a stable identifier. sub rather than the username claim, because the platform realm lets a user change the address email carries while the subject stays where it is. In CustomConfig mode the configuration is yours: map uid and the header carries what you mapped; leave it out and upstream resolves an empty UID (plugin/pkg/authenticator/token/oidc/oidc.go, getUID, read at v0.35.0), with nothing for the header to carry.
On a cluster that was running before the mapping existed, those users gain a UID where they had none. Tenant RBAC does not notice — a role binding matches a username or a group, never a UID — but an audit record on the tenant apiserver now carries user.uid, kubectl auth whoami shows it, and an admission webhook or extension server inside the tenant that read the empty value now reads the subject.
Whenever spec.oidc.mode is not None, the chart renders the flags that publish it, alongside --authentication-config:
spec.version |
rendered |
|---|---|
v1.31 |
nothing — the flag does not exist upstream, and an unknown flag stops the apiserver from starting |
v1.32 |
--requestheader-uid-headers=X-Remote-Uid plus --feature-gates=RemoteRequestHeaderUID=true |
v1.33 and newer |
--requestheader-uid-headers=X-Remote-Uid |
The gate is rendered on v1.32 only, where RemoteRequestHeaderUID is still alpha and the apiserver rejects the flag while the gate is off. From v1.33 the gate is beta and on by default, so rendering it would buy nothing and would collide with a gate list of your own.
The version in that table is the effective one, not the image tag. --emulated-version in controlPlane.apiServer.extraArgs moves it: the gate resolves through the versioned spec at the emulated version, so --emulated-version=1.32 puts RemoteRequestHeaderUID back to alpha-and-off on a v1.35 cluster, and the chart follows by rendering the gate alongside the header flag. Below 1.32 the gate has no spec at all, featureGate.Set() reports it PreAlpha and refuses to enable it, so the chart renders neither flag.
Two values the chart ignores are still not harmless. An element naming another component, --emulated-version=wardle=1.2, says nothing about the tenant kube-apiserver, but the apiserver rejects a component it has not registered, component not registered: wardle, and does not start. A value whose minor is above your spec.version is ignored too, because emulation only moves the effective version down, and there the apiserver checks against a range rather than a ceiling: it rejects one above its own binary or below the floor it still supports, naming the range it accepts. In both cases the control plane will not start whatever the chart renders.
Adding either flag through controlPlane.apiServer.extraArgs by hand is no longer necessary, and an existing cluster that does keeps working: the chart skips its own copy when your entry already carries X-Remote-Uid or already enables the gate.
Where your entry does not already satisfy the requirement, the chart works around it rather than failing. On v1.32 a --feature-gates entry of yours that says nothing about RemoteRequestHeaderUID gets a second entry rendered after it, carrying your terms plus RemoteRequestHeaderUID=true. That decides the gate whether the tenant control plane collapses duplicate flags into the last one or hands both to the apiserver, which merges them in the order it reads them; carrying your terms into the merged entry is what makes the first reading safe.
If you turned the gate off on purpose, with RemoteRequestHeaderUID=false or a blanket AllAlpha=false on v1.32 and AllBeta=false from v1.33, the chart renders neither flag and the render is byte-identical to the one before this feature existed. You keep losing the UID at the aggregation layer, which is what the opt-out asks for. On a cluster that is already running, adding the opt-out and removing it again each need one more edit alongside, below.
The running control plane is a separate matter, and on v1.33 and newer this needs care. The tenant control plane rewrites the apiserver's arguments from scratch only when the number of extraArgs entries changes. Otherwise it starts from the arguments already on the running Deployment and lays the new list over them, so a flag that has left extraArgs stays behind. That is behaviour of the Kamaji build this repo pins in packages/system/kamaji rather than a property of Kubernetes, and upstream has since changed it. It bites in both directions:
- Adding the opt-out adds one entry while the chart drops one, so the count does not change, and the
--requestheader-uid-headers=X-Remote-Uidalready on the running Deployment stays there beside your now-disabled gate. - Removing the opt-out entry to turn the feature back on drops one entry while the chart adds one. The count again does not change, so your
RemoteRequestHeaderUID=falseorAllBeta=falsestays on the Deployment beside the header flag the chart now renders, although your CR no longer shows it. --emulated-version=1.31has the same shape both ways onv1.33andv1.34. Adding it leaves the header flag behind at an emulated version where the gate is off; removing it leaves the emulation behind beside the header flag the chart now renders. Fromv1.35that value is below the oldest version the apiserver can emulate, and it refuses to start on it in any case.
The apiserver refuses to start in each of these combinations.
Change the entry count in the same edit to force the rewrite, whichever way you are going: any second flag you were going to add or drop anyway will do, or set spec.oidc.mode: None, let it converge, and turn it back on. On v1.32 the count changes by itself as long as the chart renders the gate too, because it then drops or adds two entries rather than one; where the gate is already yours, the chart renders the header flag alone and the rule above applies as it does on v1.33. That migration is the whole mitigation, and it stays necessary for as long as the pinned build keys the rewrite on the entry count; the fix belongs to that pin rather than to this chart, and is tracked in cozystack/cozystack#3541. The paths that need no operator edit are clear: turning the feature on for a default tenant moves the count, and a tenant already carrying the opt-out renders byte-identically before and after.
The render still fails where a uid-headers list of your own describes a kube-apiserver that already refuses to start: a list that omits X-Remote-Uid, a list with an element that is empty once trimmed, which a trailing comma leaves behind, and a list alongside a gate setting that turns RemoteRequestHeaderUID off. The chart cannot rescue any of them by withholding its own flag, because the flag the apiserver rejects is yours. The error message names the fix, which beats a crash-looping control plane.
Outside those shapes, nothing that renders today stops rendering: these flags are new, so every other extraArgs shape reaching this code was rendering before it existed. Setups whose gate is genuinely already on are left alone, whether you named it explicitly or switched it on with a blanket AllAlpha=true.
The gate check reads your entry the way the apiserver does.
- An element may name a component, because
--feature-gatesis a colon-separated multimap:kube:RemoteRequestHeaderUID=falseis the same opinion asRemoteRequestHeaderUID=false, while an element naming another component is not about the tenant kube-apiserver and is ignored, on the same terms as an unregistered component in--emulated-versionabove. - The chart writes its own term in whichever spelling you used, because the apiserver rejects a value mixing the bare and
kube:forms. - Past that it is one
Name=valueelement at a time, spaces trimmed on both sides, and the value resolved the waystrconv.ParseBoolresolves it, so1,tandTmean the same astrue, and0,fandFthe same asfalse. - Blanket settings count, because the apiserver applies them to every gate you have not named explicitly. On
v1.32a--feature-gates=AllAlpha=truegenuinely switchesRemoteRequestHeaderUIDon, so the chart treats it as yours and leaves it alone; fromv1.33the mirror image applies, andAllBeta=falsereads as your opt-out. - Naming the gate wins either way:
AllAlpha=true,RemoteRequestHeaderUID=falseis an opt-out onv1.32, andAllBeta=false,RemoteRequestHeaderUID=trueleaves the gate on fromv1.33, because an explicit entry beats a blanket one whichever order they appear in. - Gate names are compared exactly, as the apiserver looks them up, so
remoterequestheaderuid=trueis not an opinion aboutRemoteRequestHeaderUID; the apiserver would reject that spelling as an unrecognized gate anyway. - An entry with an empty value is not an entry as far as the chart is concerned: it reads the last entry that carries a value.
Do not rely on an empty value to cancel an earlier entry, and this has nothing to do with OIDC: the tenant control plane collapses extraArgs by flag name, again a property of the pinned build rather than of Kubernetes, and writes an empty value back as a bare flag, so a trailing --requestheader-uid-headers= reaches the apiserver as --requestheader-uid-headers, which it refuses to start on. Drop the entry you do not want instead of emptying it.
Quotes stop the chart from reading your entry at all. --requestheader-uid-headers and --emulated-version are both CSV-parsed by the apiserver, which strips a surrounding pair of double quotes before it sees the value, and the chart does not reproduce that. Rather than act on a value it may be reading differently, it renders neither flag, exactly as it did before these flags existed. That applies to the entry the chart reads, the last one on that flag that carries a value; quotes on an earlier entry that a later one replaces change nothing. Drop the quotes if you want the chart to read the entry.
The uid-headers list is read without trimming, because the apiserver does not trim it either: --requestheader-uid-headers=X-Remote-Uid , Other does not contain X-Remote-Uid as far as the apiserver is concerned, and the chart treats it the same way rather than accepting a value the apiserver will reject. The emptiness check is the one that trims, again because the apiserver's does: in X-Remote-Uid, the second element is empty once trimmed, and the apiserver refuses the whole list with empty value in "requestheader-uid-headers" although X-Remote-Uid is in it.
On v1.31 the chart renders neither flag, and the UID guards above do not run, so a uid-headers or --feature-gates entry of yours passes through untouched. The unrelated --oidc-* and --authentication-config collision guards still apply there as everywhere else. Aggregated API servers cannot see the caller's UID on v1.31; move the cluster to v1.32 or newer if you need them to. Writing --requestheader-uid-headers yourself does not work around it — the flag does not exist on v1.31, and an unknown flag stops the apiserver from starting.
Users and RBAC
users[] is a flat list. Each entry produces a single
ClusterRoleBinding inside the tenant cluster, labelled
app.kubernetes.io/managed-by=cozystack-oidc and
app.kubernetes.io/instance=<release>. The CRB name is a
deterministic hash of <release>-<email>, so the same email always
maps to the same binding.
role: |
ClusterRole bound |
|---|---|
admin |
cluster-admin |
view |
view |
The CRB User: subject is the literal email value; it must match the
email claim emitted by the issuer. The chart-generated OIDC
kubeconfig requests --oidc-extra-scope=email in the
kubectl oidc-login exec block, so the token includes email
regardless of whether the per-cluster client lists email in its
default client scopes. The email scope itself is a built-in OIDC
scope available in every conformant issuer (Keycloak included), so no
extra realm-side configuration is required for System. For
CustomConfig, verify your BYO issuer emits the email claim when
the client requests the email scope; every conformant OIDC provider
does.
Toggling a user out of users[] revokes their access on the next chart
reconcile — the bootstrap Job prunes any CRBs labelled by the release
that are no longer in the desired list.
The static admin kubeconfig stays as the documented break-glass path regardless of mode.
How a user logs in (System mode)
The user installs kubectl oidc-login once:
kubectl krew install oidc-login
The operator hands them the OIDC kubeconfig. The Secret name follows
the Helm release pattern kubernetes-<cluster>-oidc-kubeconfig (the
cozystack-api prefixes the cluster name with kubernetes- when it
materialises the HelmRelease):
kubectl --namespace tenant-acme get secret kubernetes-prod-oidc-kubeconfig \
--output=jsonpath='{.data.kubeconfig}' | base64 -d > prod.kubeconfig
First kubectl --kubeconfig prod.kubeconfig get … call triggers the
Keycloak browser flow on localhost:8000 (then 18000 as fallback) and
caches the token locally. Subsequent calls are silent until the token
expires.
Failure modes
-
mode: Systemwithout the platform-level OIDC feature — chart render hard-fails:spec.oidc.mode: System requires the platform-level OIDC feature (authentication.oidc.enabled) — enable it in cozystack-values, or use mode: CustomConfig for a tenant-supplied issuer. -
CustomConfigwith an unreachable issuer or wrong claim mappings — the apiserver rejects tokens at request time; the admin kubeconfig keeps working as break-glass. -
emailVerifiedon Keycloak users is a prescriptive requirement, not a chart-enforced one. The chart emits aclaimValidationRulesentry keyed ongroups(see next bullet), but not one keyed onemail_verified. The layered guarantees you rely on instead:- Provision users with
emailVerified: true(viaKeycloakRealmUseror the Keycloak UI's email-verify flow) so no unverified identity ever holds the email you name inusers[].email. - The
cozyrealm keeps Keycloak's defaultduplicateEmails: false, so a second account cannot claim an already-registered address to impersonate an existing operator. - As a k8s side-effect, if the issuer explicitly emits
email_verified: falseon a token the apiserver rejects it withoidc: email not verified— but a missing claim is treated as verified. Adding a CELclaimValidationRulesentry (!has(claims.email_verified) || claims.email_verified == true) to the rendered System-mode config would elevate item 3 to a hard gate; that is a reasonable follow-up hardening but is out of scope for Phase 1.
- Provision users with
-
Cross-tenant isolation gate. The rendered
AuthenticationConfigurationcarries a CELclaimValidationRulesentry on thegroupsclaim that rejects any token whose caller is not a member of at least one of this tenant's four Keycloak groups:<namespace>-view <namespace>-use <namespace>-admin <namespace>-super-adminThe tenant chart (
packages/apps/tenant/templates/keycloakgroups.yaml) provisions these groups in the sharedcozyrealm. Without this gate, any authenticated cozy-realm user could pointkubectl oidc-loginat another tenant's cluster, land atsystem:authenticated, and enumerate the discovery surface (kubectl api-resources, OpenAPI schemata) — RBAC would default-deny named-resource access, but discovery still leaks. Symmetric with the Grafana-sideallowed_groupsgate on the Monitoring chart'sauth.generic_oauthblock (seedocs/oidc-grafana.md). CustomConfig mode does NOT get this gate injected — the tenant's own AuthenticationConfiguration is authoritative and the tenant is responsible for their own claim-side guards. -
kubectlwithout theoidc-loginplugin — the exec block errors out client-side; install the plugin.
What is NOT in Phase 1
- Per-tenant Keycloak realms. Phase 2 candidate; tracked in cozystack/community#24 (Option B — retained as the strongest isolation answer; Option A is Keycloak Organizations).
- Federating an external IdP into
cozy. Out of scope. - Cross-cluster SSO inside one tenant. Each cluster has its own audience — that is the per-cluster isolation primitive.
- Custom credential plugin / RFC 8693 token exchange. Possible future optimisation; not required for the per-cluster client + audience model.