cozystack/packages/apps/kubernetes

Managed Kubernetes Service

Managed Kubernetes in Cozystack

Whenever you want to deploy a custom containerized application in Cozystack, it's best to deploy it to a managed Kubernetes cluster.

Cozystack deploys and manages Kubernetes-as-a-service as standalone applications within each tenant’s isolated environment. In Cozystack, such clusters are named tenant Kubernetes clusters, while the base Cozystack cluster is called a management or root cluster. Tenant clusters are fully separated from the management cluster and are intended for deploying tenant-specific or customer-developed applications.

Within a tenant cluster, users can take advantage of LoadBalancer services and easily provision physical volumes as needed.
The control-plane operates within containers, while the worker nodes are deployed as virtual machines, all seamlessly managed by the application.

Kubernetes version in tenant clusters is independent of Kubernetes in the management cluster. Users can select the supported patch versions from 1.31 to 1.35.

Why Use a Managed Kubernetes Cluster?

Kubernetes has emerged as the industry standard, providing a unified and accessible API, primarily utilizing YAML for configuration. This means that teams can easily understand and work with Kubernetes, streamlining infrastructure management.

Kubernetes leverages robust software design patterns, enabling continuous recovery in any scenario through the reconciliation method. Additionally, it ensures seamless scaling across a multitude of servers, addressing the challenges posed by complex and outdated APIs found in traditional virtualization platforms. This managed service eliminates the need for developing custom solutions or modifying source code, saving valuable time and effort.

The Managed Kubernetes Service in Cozystack offers a streamlined solution for efficiently managing server workloads.

Starting Work

Once the tenant Kubernetes cluster is ready, you can get a kubeconfig file to work with it. It can be done via UI or a kubectl request:

• Open the Cozystack dashboard, switch to your tenant, find and open the application page. Copy one of the config files from the Secrets section.

• Run the following command (using the management cluster kubeconfig):

  kubectl get secret -n tenant-<name> kubernetes-<clusterName>-admin-kubeconfig -o go-template='{{ printf "%s\n" (index .data "admin.conf" | base64decode) }}' > admin.conf
  

There are several kubeconfig options available:

• admin.conf — The standard kubeconfig for accessing your new cluster. You can create additional Kubernetes users using this configuration.
• admin.svc — Same token as admin.conf, but with the API server address set to the internal service name. Use it for applications running inside the cluster that need API access.
• super-admin.conf — Similar to admin.conf, but with extended administrative permissions. Intended for troubleshooting and cluster maintenance tasks.
• super-admin.svc — Same as super-admin.conf, but pointing to the internal API server address.

Implementation Details

A tenant Kubernetes cluster in Cozystack is essentially Kubernetes-in-Kubernetes. Deploying it involves the following components:

• Kamaji Control Plane: Kamaji is an open-source project that facilitates the deployment of Kubernetes control planes as pods within a root cluster. Each control plane pod includes essential components like kube-apiserver, controller-manager, and scheduler, allowing for efficient multi-tenancy and resource utilization.

• Etcd Cluster: A dedicated etcd cluster is deployed using the cozystack etcd-operator (etcd-operator.cozystack.io/v1alpha2). It provides reliable and scalable key-value storage for the Kubernetes control plane.

• Worker Nodes: Virtual Machines are provisioned to serve as worker nodes using KubeVirt. These nodes are configured to join the tenant Kubernetes cluster, enabling the deployment and management of workloads. Worker node disks automatically detect and match the underlying volume's block size (blockSize.matchVolume) to ensure compatibility with storage backends that use non-512-byte sectors, such as LINSTOR DRBD with 4Kn drives.

• Cluster API: Cozystack is using the Kubernetes Cluster API to provision the components of a cluster.

This architecture ensures isolated, scalable, and efficient tenant Kubernetes environments.

See the reference for components utilized in this service:

• Kamaji Control Plane
• Kamaji — Cluster API
• github.com/clastix/kamaji
• KubeVirt
• github.com/kubevirt/kubevirt
• github.com/cozystack/etcd-operator
• Kubernetes Cluster API
• github.com/kubernetes-sigs/cluster-api-provider-kubevirt
• github.com/kubevirt/csi-driver

Breaking Changes

• ephemeralStorage renamed to diskSize (v1.4): The nodeGroups[name].ephemeralStorage field has been renamed to nodeGroups[name].diskSize. Existing clusters are migrated transparently by platform migration 41 during the pre-upgrade hook — no manual action is required. Newly written values should use diskSize. Existing VMs are rolling-updated via CAPI when the new values are applied. With the Talos worker rollover in this release the field now sizes the single system disk (Talos OS image streamed from factory.talos.dev, kubelet state, containerd image cache, local-path PVCs) — the pre-Talos disk-kubelet PVC layout was removed. State on that single disk persists across same-VM reboots (virt-launcher restart, guest reboot, node failure); VM replacement by CAPI (e.g. nodeGroup field change, MachineHealthCheck remediation) provisions a fresh disk.

• Air-gapped tenant workers (Phase 1 Talos rollover): tenant worker bootstrap moves from Ubuntu containerDisk + kubeadm to a Talos OS disk image fetched over HTTP by CDI and a .../installer/... installer reference. Both sources are now overridable — set talos.imageFactoryURL (default https://factory.talos.dev) to a self-hosted Image Factory, a caching mirror, or an internal HTTP file server, and talos.installerRepository (default factory.talos.dev/installer) to a mirrored registry. The Helm-rendered *-patch-containerd Secret that previously plumbed the platform-wide registries.mirrors config to tenant workers (via the kubeadm template's containerd certs.d mount) has no consumer in the Talos machineconfig and was removed in this release; in-guest container-image pulls (pods pulling from registries inside the tenant cluster) still honour no per-tenant registries.mirrors override until Phase 2 restores it via machine.registries.mirrors knobs. So a strict-egress tenant can now mirror the Talos OS image and installer, but rate-limited in-guest registry pulls remain a Phase 2 follow-up — file an issue if you depend on this and it is not yet landed.

• Worker MachineHealthCheck remediation is now enabled by default (Phase 1 Talos rollover): the worker MHC maxUnhealthy moved from a hard-coded 0 (remediation effectively off) to a configurable nodeHealthCheck.maxUnhealthy defaulting to "50%". CAPI now auto-remediates (deletes and replaces) unhealthy worker Machines while at least 50% of a group stays healthy. "50%" deliberately leaves headroom for transient unhealthy nodes during the kubeadm→Talos rollover and slow first boots from factory.talos.dev; set nodeHealthCheck.maxUnhealthy: "0%" to keep the previous remediation-off behaviour until your fleet is stable on Talos workers.

• GitOps-managed tenant Kubernetes CRs must bump spec.version in Git (Phase 1 Talos rollover): platform migration 46 patches live kuberneteses.apps.cozystack.io objects from v1.30 to v1.31 (v1.30 left the Talos↔Kubernetes support matrix). When the tenant Kubernetes CR is reconciled from Git (Flux/Argo), the next source reconcile re-applies version: v1.30 over migration 46's patch, which then trips the chart's _versions.tpl guard and the HelmRelease fails. Update spec.version to v1.31 (or newer) in your Git source before/with the platform upgrade. Tenants managed via the API or dashboard need no action — migration 46 handles them.

• Remote-accessible LINSTOR StorageClasses are auto-propagated to tenants (v1.5): infra-cluster LINSTOR StorageClasses whose linstor.csi.linbit.com/allowRemoteVolumeAccess is not "false" are created inside each tenant under the same name (provisioned by csi.kubevirt.io). Upgrade caveat: delete any manually created tenant StorageClass whose name collides with a propagated infra class (e.g. a hand-made replicated) before upgrading, or the propagated class cannot be created and the tenant CSI release will not converge. Infra classes that must stay node-local need allowRemoteVolumeAccess: "false" set explicitly — an absent annotation is treated as remote-accessible. Propagation is evaluated at HelmRelease render time, so classes added or removed on the infra cluster after a tenant exists propagate only on that tenant's next reconcile.

• Worker pools split into KubernetesNodes (Phase 2): spec.nodeGroups, spec.nodeHealthCheck and spec.maxNodeProvisionTime are removed from the Kubernetes CR — worker pools are now separate KubernetesNodes resources (one HelmRelease per pool; see the kubernetes-nodes chart), on which spec.cluster and spec.storageClass are immutable (self == oldSelf). Existing pools are adopted automatically on upgrade by platform migration 56 with no worker-VM churn. Consequences: (1) the Kubernetes CR's WorkloadsReady condition is now control-plane-only — worker health is reported on each KubernetesNodes resource, so a dashboard or kubectl wait watching the parent no longer reflects worker readiness; (2) enabling addons.ingressNginx now requires a pool carrying roles: [ingress-nginx] (the implicit default md0 is gone — without such a pool the controller DaemonSet schedules nothing); (3) do not edit Kubernetes CRs during a platform upgrade — a mid-window parent render can transiently fail Helm ownership validation while migration 56 re-annotates worker objects, and self-heals once the new chart artifact lands; (4) when deleting a cluster, delete its KubernetesNodes pools before the parent Kubernetes resource, so each pool's pre-delete unpin hook can run while the tenant apiserver and the CAPI CRDs still exist — deleting the parent first can leave the pool release reconciling CAPI objects against a cluster that is being torn down (in the pre-split monolith the pools were part of the parent release and died with it; they are now separate resources with their own lifecycle); (5) the removed fields (nodeGroups, nodeHealthCheck, maxNodeProvisionTime) are still accepted and stored on an already-upgraded Kubernetes CR — the adoption migration deliberately leaves them in place rather than rewriting parent values mid-upgrade — but they have no effect: editing them (for example scaling via spec.nodeGroups.<pool>.minReplicas) does nothing, and the API returns an admission warning pointing you at the KubernetesNodes resources. A cluster name is also capped at 32 characters so every pool's KubernetesNodes release name fits the 53-character Helm limit. ComputePlane clusters (the computeplane module) are affected the same way: their worker pools are rendered as separate per-pool KubernetesNodes releases too (materialised directly by the module rather than adopted), so the same pool-name length limit applies to their nodeGroups keys.

• controlPlane.apiServer.extraArgs refuses flags the tenant control plane owns (Kamaji 26.9.4-edge): the tenant control plane now sets a list of kube-apiserver flags itself and drops an extraArgs entry naming one, where the previous build let the entry win. The entry stops taking effect once the new build reconciles the control plane, whatever the chart renders. Two of those flags have a field the control plane renders them from, and the chart moves an entry written as --flag=value into it while the field is empty: --enable-admission-plugins into controlPlane.apiServer.admissionControllers, --kubelet-preferred-address-types into controlPlane.kubelet.preferredAddressTypes. The KamajiControlPlane CRD accepts only the plugin names it lists, so a plugin outside that list that the apiserver leaves off by default can no longer be enabled, and a moved name outside it fails when the KamajiControlPlane is applied, with an error that names it, rather than at render. The chart refuses to render any other entry naming one of the flags, so a tenant carrying one stops reconciling until the entry is removed; the description of controlPlane.apiServer.extraArgs lists the flags. The same Kamaji update passes every extraArgs entry to the apiserver, so repeats of a flag no longer collapse to the last one: naming the kube feature gates both bare and kube:-prefixed across entries, or giving two --emulated-version entries that name kube, now stops the apiserver from starting. The chart refuses neither, in any spec.oidc.mode: both reach the apiserver as written. Keep one spelling for kube gate terms and one kube emulated version.

The top-level storageClass field is annotated as immutable in the chart schema — see docs/storage-immutability.md for the contract and which consumers enforce it. The per-pool worker storageClass now lives on the KubernetesNodes resource (see the kubernetes-nodes chart).

Proxmox substrate: credentials, privileges and tenant scope

On substrate: proxmox two components authenticate to the hypervisor, and both run in the management cluster, in the tenant's namespace: the CSI controller (<release>-pcsi-controller) and the cloud-controller-manager (<release>-pccm). The tenant cluster gets the CSI node plugin, the CSIDriver object and the storage classes, none of which carry a credential. proxmox.csi.credentialsSecretName and proxmox.ccm.credentialsSecretName name Secrets in the management cluster with the keys url, token_id, token_secret and region; the tenant cannot read them, and nothing copies them into the tenant cluster.

Keeping the token out of the tenant is half of the isolation. The tenant is cluster-admin of its own cluster, and the controllers act on what the tenant's objects tell them: the CSI controller attaches the disk a PV's volumeHandle names to the VM a node's ID names, and deletes the disk a PV names, with no check that either belongs to this cluster. Under the privileges the upstream charts document — granted on / — one tenant's objects can make the controller attach, detach or delete another tenant's disks. The per-cluster volume owner id (vm-<id>-pvc-…) makes leftovers attributable; it does not scope anything.

What contains it is one Proxmox ACL per tenant, on a resource pool. Create a pool for the tenant on the hypervisor, set proxmox.pool to its name on every KubernetesNodes pool of the tenant's cluster — capmox adds each worker VM to it as it is cloned, replacements included, so the ACL follows every VM without being granted again — and give the tenant's tokens their privileges on that pool and on a storage dedicated to the tenant:

Component	Privileges	Path
CSI controller	VM.Audit, VM.Config.Disk	/pool/<proxmox.pool>
CSI controller	Datastore.Allocate, Datastore.AllocateSpace, Datastore.Audit	/storage/<the tenant's storage>: Datastore.Allocate on a storage reaches every volume on it
CCM	VM.Audit, VM.GuestAgent.Audit	/pool/<proxmox.pool>: a NotReady node whose VM the token cannot see is reported as gone, and the CCM deletes it
CCM	Sys.Audit	/

Grant these to the user that owns the tokens as well as to the tokens themselves: Proxmox caps a privilege-separated token at its user's permissions, so a token whose user holds nothing has nothing, whatever its own ACL says. pveum user token permissions <user> <token> --path /pool/<proxmox.pool> shows the set that actually applies.

A storage shared between tenants cannot be scoped: Datastore.Allocate on it is what deletes volumes and also what lets any volume on it be attached. Without proxmox.pool there is no VM-level scope either: a token that reaches the tenant's VMs then reaches every VM on the hypervisor. What the pool-and-storage scope still leaves open is the tenant's own disks, which the tenant can attach, detach and delete through the controller — the same reach a cluster-admin of the tenant has anyway.

proxmox.insecure defaults to false: the controllers verify the hypervisor's certificate. A stock Proxmox VE install serves a self-signed one, so either set insecure: true for it or install a certificate the management cluster trusts.

Parameters

Common Parameters

Name	Description	Type	Value
storageClass	Default StorageClass inside the tenant cluster. Remote-accessible LINSTOR classes are auto-propagated to the tenant under the same name.	string	replicated

Application-specific Parameters

Name	Description	Type	Value
version	Kubernetes major.minor version to deploy	string	v1.35
host	External hostname for Kubernetes cluster. Defaults to <cluster-name>.<tenant-host> if empty.	string	""

Substrate

Name	Description	Type	Value
substrate	Which infrastructure provider backs this cluster's worker VMs. kubevirt runs them inside this cluster; proxmox runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Must match the substrate of every KubernetesNodes pool attached to this cluster: the pools reference this cluster's infrastructure object by kind, and a mismatch leaves their Machines unreconciled. Switching an existing cluster is not supported; create a new one.	string	kubevirt
proxmox	Proxmox substrate settings.	object	{}
proxmox.allowedNodes	Proxmox nodes capmox may place VMs on. Empty means every node in the Proxmox cluster.	[]string	[]
proxmox.dnsServers	Nameservers written into each worker's network config. Required by the ProxmoxCluster schema.	[]string	[]
proxmox.insecure	Skip verification of the Proxmox API server certificate in the CCM and the CSI controller, both of which run in the management cluster. Default false: the certificate is verified unless an operator turns that off. A stock Proxmox VE install serves a self-signed one, so set true for it, or install a certificate the management cluster trusts.	bool	false
proxmox.ipv4Config	Address pool for workers. capmox assigns static addresses and has no DHCP mode, so this is required rather than optional.	object	{}
proxmox.ipv4Config.addresses	Ranges or CIDRs, e.g. ["10.0.0.120-10.0.0.170"]. capmox turns these into an InClusterIPPool, so they must not overlap any DHCP range on the same L2 or two workers will answer to one address.	[]string	[]
proxmox.ipv4Config.gateway	Default gateway for the workers.	string	""
proxmox.ipv4Config.prefix	Netmask prefix length.	int	24
proxmox.ccm	Proxmox cloud-controller-manager settings.	object	{}
proxmox.ccm.credentialsSecretName	Secret in THIS namespace holding the controller's Proxmox API credentials under the keys url, token_id, token_secret and region. May name the same Secret as csi.credentialsSecretName; it is a separate knob so the two can hold separate tokens, since this one only reads VM inventory while the driver attaches and detaches disks.	string	""
proxmox.csi	Proxmox CSI driver settings.	object	{}
proxmox.csi.enabled	Install the driver. Without it a Proxmox-backed tenant has no way to provision PersistentVolumes at all.	bool	true
proxmox.csi.credentialsSecretName	Secret in THIS namespace holding the driver's Proxmox API credentials under the keys url, token_id, token_secret and region. The parent chart composes them into the controller's config file here, in the management cluster; nothing reaches the tenant, and no token is written into a rendered manifest. Distinct from the capmox credentials on purpose: the driver attaches and detaches disks, which is a different blast radius from creating VMs.	string	""
proxmox.csi.storageClasses	StorageClasses to create in the tenant.	[]object	[]
proxmox.csi.storageClasses[i].name	StorageClass name inside the tenant cluster.	string	""
proxmox.csi.storageClasses[i].storage	Proxmox storage id the volumes are created on (as it appears in pvesm status).	string	""
proxmox.csi.storageClasses[i].fstype	Filesystem the driver formats volumes with: ext4 or xfs.	string	""
proxmox.csi.storageClasses[i].ssd	Mark volumes as SSD-backed, which changes the discard/cache defaults Proxmox applies.	bool	false
proxmox.csi.storageClasses[i].reclaimPolicy	Delete or Retain.	string	""

Cluster Addons

Name	Description	Type	Value
addons	Cluster addons configuration.	object	{}
addons.certManager	Cert-manager addon.	object	{}
addons.certManager.enabled	Enable cert-manager.	bool	false
addons.certManager.valuesOverride	Custom Helm values overrides.	object	{}
addons.cilium	Cilium CNI plugin. Always installed as the tenant cluster's CNI; this section only overrides its Helm values.	object	{}
addons.cilium.valuesOverride	Custom Helm values overrides.	object	{}
addons.gatewayAPI	Gateway API addon.	object	{}
addons.gatewayAPI.enabled	Enable Gateway API.	bool	false
addons.ingressNginx	Ingress-NGINX controller.	object	{}
addons.ingressNginx.enabled	Enable the controller (requires nodes labeled ingress-nginx).	bool	false
addons.ingressNginx.exposeMethod	Method to expose the controller. Allowed values: Proxied, LoadBalancer.	string	Proxied
addons.ingressNginx.hosts	Domains routed to this tenant cluster when exposeMethod is Proxied.	[]string	[]
addons.ingressNginx.valuesOverride	Custom Helm values overrides.	object	{}
addons.gpuOperator	NVIDIA GPU Operator.	object	{}
addons.gpuOperator.enabled	Enable GPU Operator.	bool	false
addons.gpuOperator.valuesOverride	Custom Helm values overrides.	object	{}
addons.hami	HAMi GPU virtualization middleware.	object	{}
addons.hami.enabled	Enable HAMi (requires GPU Operator).	bool	false
addons.hami.valuesOverride	Custom Helm values overrides.	object	{}
addons.monitoringAgents	Monitoring agents.	object	{}
addons.monitoringAgents.enabled	Enable monitoring agents.	bool	false
addons.monitoringAgents.valuesOverride	Custom Helm values overrides.	object	{}
addons.verticalPodAutoscaler	Vertical Pod Autoscaler. Installed and removed together with addons.monitoringAgents.enabled; this section only overrides its Helm values.	object	{}
addons.verticalPodAutoscaler.valuesOverride	Custom Helm values overrides.	object	{}
addons.velero	Velero backup/restore addon.	object	{}
addons.velero.enabled	Enable Velero.	bool	false
addons.velero.valuesOverride	Custom Helm values overrides.	object	{}
addons.coredns	CoreDNS addon. Always installed as the tenant cluster's DNS; this section only overrides its Helm values.	object	{}
addons.coredns.valuesOverride	Custom Helm values overrides.	object	{}
addons.ouroboros	Hairpin-NAT fix for ingress-nginx with PROXY-protocol.	object	{}
addons.ouroboros.enabled	Enable ouroboros. Requires addons.ingressNginx.enabled (chart-render fail otherwise). Only useful when PROXY-protocol is wired on the tenant ingress-nginx via valuesOverride.	bool	false
addons.ouroboros.valuesOverride	Custom Helm values overrides. Operator-key wins over cozystack defaults.	object	{}

Kubernetes Control Plane Configuration

Name	Description	Type	Value
controlPlane	Kubernetes control-plane configuration.	object	{}
controlPlane.replicas	Number of control-plane replicas.	int	2
controlPlane.apiServer	API Server configuration.	object	{}
controlPlane.apiServer.resources	CPU and memory resources for API Server.	object	{}
controlPlane.apiServer.resources.cpu	CPU available.	quantity	""
controlPlane.apiServer.resources.memory	Memory (RAM) available.	quantity	""
controlPlane.apiServer.resourcesPreset	Preset if resources omitted.	string	c1.medium
controlPlane.apiServer.extraArgs	Extra command-line flags appended to the tenant kube-apiserver, passed through to KamajiControlPlane spec.apiServer.extraArgs. For OIDC use spec.oidc.mode — this passthrough is the escape hatch for unrelated apiserver flags. The tenant control plane sets each of the following flags itself and drops an entry naming one before the apiserver sees it. The chart moves an --enable-admission-plugins= entry into controlPlane.apiServer.admissionControllers and a --kubelet-preferred-address-types= entry into controlPlane.kubelet.preferredAddressTypes while that field is empty, and refuses any other entry naming one of them: --advertise-address, --client-ca-file, --enable-admission-plugins, --service-cluster-ip-range, --kubelet-client-certificate, --kubelet-client-key, --kubelet-preferred-address-types, --proxy-client-cert-file, --proxy-client-key-file, --requestheader-allowed-names, --requestheader-client-ca-file, --secure-port, --service-account-key-file, --service-account-signing-key-file, --tls-cert-file, --tls-private-key-file, --etcd-servers, --etcd-cafile, --etcd-certfile, --etcd-keyfile, --etcd-prefix, --etcd-compaction-interval, --egress-selector-config-file. When spec.oidc.mode is not None the chart also renders OIDC flags here, reads what you have written on them, and refuses some combinations outright; docs/oidc-tenant.md carries which flags, which combinations, and why. Do NOT add legacy --oidc-* flags here when spec.oidc.mode is not None; the chart injects --authentication-config and the apiserver refuses to boot with both. Empty by default (no change to current behavior).	[]string	[]
controlPlane.apiServer.admissionControllers	Admission plugins the tenant kube-apiserver enables, passed through to KamajiControlPlane spec.admissionControllers, which the control plane renders as --enable-admission-plugins. A non-empty list replaces the control plane's default set instead of adding to it. The kube-apiserver enables these plugins in addition to its own default ones, so leaving a plugin out does not turn it off; to turn one off, name it in --disable-admission-plugins in controlPlane.apiServer.extraArgs and keep it out of this list, since the apiserver refuses to start with a plugin in both, and the control plane's default set names several. Empty by default, which keeps that default set, taken from the TenantControlPlane CRD: CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, LimitRanger, MutatingAdmissionWebhook, NamespaceLifecycle, PersistentVolumeClaimResize, Priority, ResourceQuota, RuntimeClass, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. NodeRestriction is not in it.	[]string	[]
controlPlane.apiServer.extraVolumes	Extra volumes added to the control-plane Deployment, passed through to KamajiControlPlane spec.deployment.extraVolumes. Use to mount a ConfigMap or Secret holding an AuthenticationConfiguration file referenced by extraArgs. Each item is a core/v1 Volume, but because the control-plane pod runs on the management cluster only configMap and secret sources are allowed (host-reaching sources like hostPath/csi and token-projection via projected are rejected); each volume must have a unique, non-empty name and exactly one source. The names talos-ca and talos-tls-cert are reserved by the chart. Empty by default.	[]object	[]
controlPlane.apiServer.extraVolumeMounts	Extra volume mounts added to the tenant kube-apiserver container, passed through to KamajiControlPlane spec.apiServer.extraVolumeMounts. Each name must reference a volume declared in extraVolumes; the chart-managed talos secret volumes cannot be mounted. Each item is a core/v1 VolumeMount. Empty by default.	[]object	[]
controlPlane.controllerManager	Controller Manager configuration.	object	{}
controlPlane.controllerManager.resources	CPU and memory resources for Controller Manager.	object	{}
controlPlane.controllerManager.resources.cpu	CPU available.	quantity	""
controlPlane.controllerManager.resources.memory	Memory (RAM) available.	quantity	""
controlPlane.controllerManager.resourcesPreset	Preset if resources omitted.	string	t1.micro
controlPlane.scheduler	Scheduler configuration.	object	{}
controlPlane.scheduler.resources	CPU and memory resources for Scheduler.	object	{}
controlPlane.scheduler.resources.cpu	CPU available.	quantity	""
controlPlane.scheduler.resources.memory	Memory (RAM) available.	quantity	""
controlPlane.scheduler.resourcesPreset	Preset if resources omitted.	string	t1.micro
controlPlane.konnectivity	Konnectivity configuration.	object	{}
controlPlane.konnectivity.server	Konnectivity Server configuration.	object	{}
controlPlane.konnectivity.server.resources	CPU and memory resources for Konnectivity.	object	{}
controlPlane.konnectivity.server.resources.cpu	CPU available.	quantity	""
controlPlane.konnectivity.server.resources.memory	Memory (RAM) available.	quantity	""
controlPlane.konnectivity.server.resourcesPreset	Preset if resources omitted.	string	t1.micro
controlPlane.kubelet	Kubelet connection settings.	object	{}
controlPlane.kubelet.preferredAddressTypes	Node address types the tenant kube-apiserver tries, in order, when it connects to a kubelet, passed through to KamajiControlPlane spec.kubelet.preferredAddressTypes, which the control plane renders as --kubelet-preferred-address-types. The KamajiControlPlane CRD rejects a repeated name. Empty by default, which renders InternalIP, ExternalIP, or the list of a --kubelet-preferred-address-types= entry in controlPlane.apiServer.extraArgs when there is one.	[]string	[]
images	Optional image overrides for air-gapped or rate-limited registries.	object	{}
images.waitForKubeconfig	Image used by the wait-for-kubeconfig init container. Empty falls back to images/busybox.tag.	string	""
images.kubectl	Image used by the bootstrap-token tenant Job (kubectl). Empty falls back to images/kubectl.tag.	string	""
images.talosCsrSigner	Image used by the talos-csr-signer sidecar in the Kamaji control plane. Empty falls back to images/talos-csr-signer.tag.	string	""

Talos Worker Image

Name	Description	Type	Value
talos	Talos worker image configuration.	object	{}
talos.version	Talos release used for worker OS image and installer. Must satisfy the chart's Talos<->Kubernetes support matrix against the chosen version.	string	v1.13.6
talos.schematicID	Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here.	string	ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515
talos.imageFactoryURL	Base URL of the Talos Image Factory that serves the worker OS disk image (the openstack-amd64.raw.xz raw artifact streamed in by CDI over HTTP). Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash.	string	https://factory.talos.dev
talos.installerRepository	OCI repository prefix for the Talos installer image used by the in-guest talos-reconcile upgrade Job. Resolved as <installerRepository>/<schematicID>:<version>. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash.	string	factory.talos.dev/installer
talos.registryMirrors	Talos machine.registries.mirrors passthrough for worker nodes: a map of upstream registry host to { endpoints: [ ... ] }. Empty by default, so workers pull container images (the Talos kubelet image included) directly from the upstream registry. Point a host such as ghcr.io at an in-cluster pull-through mirror for air-gapped, rate-limited, or flaky-egress environments so a worker's boot does not depend on live public egress. Talos still falls back to the upstream registry unless a host also sets skipFallback: true, so a mirror alone does not enforce air-gap.	object	{}

OIDC

Name	Description	Type	Value
oidc	OIDC authentication and per-user RBAC for the tenant kube-apiserver. See docs/oidc-tenant.md for the operator guide.	object	{}
oidc.mode	Identity mode. None: no OIDC, only the static admin kubeconfig works. System: trust the platform Keycloak realm (cozy by default) via a per-cluster public client with audience binding; zero-config default. CustomConfig: trust a tenant-supplied issuer directly (BYO); cozy is not in the path.	string	None
oidc.customConfig	Tenant-supplied AuthenticationConfiguration; consumed only when mode: CustomConfig.	object	{}
oidc.customConfig.config	Inline AuthenticationConfiguration YAML (apiserver.config.k8s.io/v1beta1). The chart writes the value verbatim into a Secret mounted at /etc/kubernetes/authentication-config/config.yaml on the kube-apiserver. Mutually exclusive with secretRef.name.	string	""
oidc.customConfig.secretRef	Reference to an existing Secret in the tenant namespace carrying the AuthenticationConfiguration.	object	{}
oidc.customConfig.secretRef.name	Name of an existing Secret in the tenant (release) namespace whose config.yaml key holds an apiserver.config.k8s.io/v1beta1 AuthenticationConfiguration. Mutually exclusive with customConfig.config.	string	""
oidc.users	Users granted access to the tenant cluster; each entry produces one ClusterRoleBinding inside the tenant cluster. Works for both System and CustomConfig modes.	[]object	[]
oidc.users[i].email	Email address matched against the email claim from the issuer. Used verbatim as the User: subject in the ClusterRoleBinding inside the tenant cluster. The email claim is a built-in OIDC scope requested explicitly by the chart-generated kubectl oidc-login kubeconfig; every conformant OIDC provider (cozy realm included) emits it.	string	""
oidc.users[i].role	Role to bind: admin maps to ClusterRole/cluster-admin, view maps to ClusterRole/view.	string	{}

Parameter examples and reference

resources and resourcesPreset

resources sets explicit CPU and memory configurations for each replica. Every resource it leaves unset is taken from the preset defined in resourcesPreset.

resources:
  cpu: 4000m
  memory: 4Gi

resourcesPreset sets named CPU and memory configurations for each replica. This setting is ignored if the corresponding resources value is set.

Preset name	CPU	memory
nano	250m	128Mi
micro	500m	256Mi
small	1	512Mi
medium	1	1Gi
large	2	2Gi
xlarge	4	4Gi
2xlarge	8	8Gi

instanceType Resources

The following instanceType resources are provided by Cozystack:

Name	vCPUs	Memory
cx1.2xlarge	8	16Gi
cx1.2xlarge1gi	8	16Gi
cx1.4xlarge	16	32Gi
cx1.4xlarge1gi	16	32Gi
cx1.8xlarge	32	64Gi
cx1.8xlarge1gi	32	64Gi
cx1.large	2	4Gi
cx1.large1gi	2	4Gi
cx1.medium	1	2Gi
cx1.medium1gi	1	2Gi
cx1.xlarge	4	8Gi
cx1.xlarge1gi	4	8Gi
d1.2xlarge	8	32Gi
d1.2xmedium	2	4Gi
d1.4xlarge	16	64Gi
d1.8xlarge	32	128Gi
d1.large	2	8Gi
d1.medium	1	4Gi
d1.micro	1	1Gi
d1.nano	1	512Mi
d1.small	1	2Gi
d1.xlarge	4	16Gi
gn1.2xlarge	8	32Gi
gn1.4xlarge	16	64Gi
gn1.8xlarge	32	128Gi
gn1.xlarge	4	16Gi
m1.2xlarge	8	64Gi
m1.2xlarge1gi	8	64Gi
m1.4xlarge	16	128Gi
m1.4xlarge1gi	16	128Gi
m1.8xlarge	32	256Gi
m1.8xlarge1gi	32	256Gi
m1.large	2	16Gi
m1.large1gi	2	16Gi
m1.xlarge	4	32Gi
m1.xlarge1gi	4	32Gi
n1.2xlarge	16	32Gi
n1.4xlarge	32	64Gi
n1.8xlarge	64	128Gi
n1.large	4	8Gi
n1.medium	4	4Gi
n1.xlarge	8	16Gi
o1.2xlarge	8	32Gi
o1.4xlarge	16	64Gi
o1.8xlarge	32	128Gi
o1.large	2	8Gi
o1.medium	1	4Gi
o1.micro	1	1Gi
o1.nano	1	512Mi
o1.small	1	2Gi
o1.xlarge	4	16Gi
rt1.2xlarge	8	32Gi
rt1.4xlarge	16	64Gi
rt1.8xlarge	32	128Gi
rt1.large	2	8Gi
rt1.medium	1	4Gi
rt1.micro	1	1Gi
rt1.small	1	2Gi
rt1.xlarge	4	16Gi
u1.2xlarge	8	32Gi
u1.2xmedium	2	4Gi
u1.4xlarge	16	64Gi
u1.8xlarge	32	128Gi
u1.large	2	8Gi
u1.medium	1	4Gi
u1.micro	1	1Gi
u1.nano	1	512Mi
u1.small	1	2Gi
u1.xlarge	4	16Gi

U Series: Universal

The U Series is quite neutral and provides resources for general purpose applications.

U is the abbreviation for "Universal", hinting at the universal attitude towards workloads.

VMs of instance types will share physical CPU cores on a time-slice basis with other VMs.

U Series Characteristics

Specific characteristics of this series are:

• Burstable CPU performance - The workload has a baseline compute performance but is permitted to burst beyond this baseline, if excess compute resources are available.
• vCPU-To-Memory Ratio (1:4) - A vCPU-to-Memory ratio of 1:4, for less noise per node.

O Series: Overcommitted

The O Series is based on the U Series, with the only difference being that memory is overcommitted.

O is the abbreviation for "Overcommitted".

O Series Characteristics

Specific characteristics of this series are:

• Burstable CPU performance - The workload has a baseline compute performance but is permitted to burst beyond this baseline, if excess compute resources are available.
• Overcommitted Memory - Memory is over-committed in order to achieve a higher workload density.
• vCPU-To-Memory Ratio (1:4) - A vCPU-to-Memory ratio of 1:4, for less noise per node.

CX Series: Compute Exclusive

The CX Series provides exclusive compute resources for compute intensive applications.

CX is the abbreviation of "Compute Exclusive".

The exclusive resources are given to the compute threads of the VM. In order to ensure this, some additional cores (depending on the number of disks and NICs) will be requested to offload the IO threading from cores dedicated to the workload. In addition, in this series, the NUMA topology of the used cores is provided to the VM.

CX Series Characteristics

Specific characteristics of this series are:

• Hugepages - Hugepages are used in order to improve memory performance.
• Dedicated CPU - Physical cores are exclusively assigned to every vCPU in order to provide fixed and high compute guarantees to the workload.
• Isolated emulator threads - Hypervisor emulator threads are isolated from the vCPUs in order to reduce emaulation related impact on the workload.
• vNUMA - Physical NUMA topology is reflected in the guest in order to optimize guest sided cache utilization.
• vCPU-To-Memory Ratio (1:2) - A vCPU-to-Memory ratio of 1:2.

M Series: Memory

The M Series provides resources for memory intensive applications.

M is the abbreviation of "Memory".

M Series Characteristics

Specific characteristics of this series are:

• Hugepages - Hugepages are used in order to improve memory performance.
• Burstable CPU performance - The workload has a baseline compute performance but is permitted to burst beyond this baseline, if excess compute resources are available.
• vCPU-To-Memory Ratio (1:8) - A vCPU-to-Memory ratio of 1:8, for much less noise per node.

RT Series: RealTime

The RT Series provides resources for realtime applications, like Oslat.

RT is the abbreviation for "realtime".

This series of instance types requires nodes capable of running realtime applications.

RT Series Characteristics

Specific characteristics of this series are:

• Hugepages - Hugepages are used in order to improve memory performance.
• Dedicated CPU - Physical cores are exclusively assigned to every vCPU in order to provide fixed and high compute guarantees to the workload.
• Isolated emulator threads - Hypervisor emulator threads are isolated from the vCPUs in order to reduce emaulation related impact on the workload.
• vCPU-To-Memory Ratio (1:4) - A vCPU-to-Memory ratio of 1:4 starting from the medium size.