## What this PR does Cleanup after the Kubernetes app was split into `Kubernetes` (control plane and addons) and `KubernetesNodes` (one release per worker pool): what the split left behind in values, docs, templates and tests. ### Kubernetes The `Kubernetes` application has no `talos` field any more. Worker pools are `KubernetesNodes` resources and take their Talos release, image, installer and registry mirrors from their own `talos` and `osImage`, so the cluster's copies did nothing. The last one, `talos.version`, fed a duplicate of the Talos to Kubernetes support-matrix check. The pool chart already runs that check against the Talos release each pool actually boots, including one picked through an `osImage` override; the cluster's copy compared the Kubernetes version against a release no node runs. A stored spec that still carries `talos.*` keeps rendering, because the values schema does not close the object and the API does not validate `spec` against it, and a unit test pins that. The pool's check now has render coverage. Its own suite pins the pairings it must accept, and a bats test renders the refusals through `talos.version`, `osImage.factory.version` and `osImage.builtin.version` with schema validation skipped, since the schema offers no Kubernetes version the matrix refuses. I fixed the README and field descriptions that described behavior the chart no longer has, or never had: a dedicated etcd cluster, a `blockSize.matchVolume` setting that does not exist, a default `md0` pool, the host default, the substrate mismatch behavior, and rollout phase names. The README's Breaking Changes section becomes a pointer to the per-release upgrade notes on cozystack.io. Unused helpers and stale comments are removed; the chart renders the same bytes. ### KubernetesNodes Five field descriptions and the `talos` block told users to keep them in sync with parent fields that do not exist. NOTES warned about colliding with a parent-managed `md0` pool, although the parent renders no pools and `md0` is the normal name of a pool. The installer image was credited to an upgrade Job that never runs one. The values descriptions, README intro and NOTES now say what the chart renders today, including which fields have no effect on `substrate: proxmox`: `storageClass`, `diskSize`, `instanceType`, `talos.imageFactoryURL` (beyond its format check) and `talos.imageFactoryCA`. The render error for a pool whose MachineDeployment belongs to another release told users to remove the pool from the parent's `nodeGroups`, which cannot clear the conflict. It now names a remedy that works. Split-era history is gone from Helm and test comments, along with two `storageClass` fallbacks that can only return the value they fall back from, a constant variable, and two `.helmignore` rules that match nothing. The tenant pod and service CIDRs are literals in both charts, and nothing checked that they agree. A new bats test renders both charts and fails when they differ. No rendered manifest of the pool chart changes. I rendered it for the snapshot values, GPU, kubelet-override, factory/CA, storageClass variants and three Proxmox variants on main and on every commit, and the output is byte-identical. The snapshot and TalosConfigTemplate hash tests pass unchanged, so no worker rolls. YAML comments inside the hashed machine config and machine template are left as they are for that reason. ### The rule `docs/agents/overview.md` gains a rule so this does not come back with the next change: app READMEs and field descriptions describe current behavior, a change users must act on goes to the website's upgrade page, and a description does not ask the reader to keep it in sync with another chart. ```release-note [kubernetes] The Kubernetes application no longer has a `talos` field. Talos settings for workers, and the Talos to Kubernetes compatibility check, live on each KubernetesNodes pool. Existing clusters that still store `talos` values keep working; the values are ignored. KubernetesNodes field descriptions, README and install notes now describe the worker pool as it works today, including which fields have no effect on Proxmox pools. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Configuration** * Worker OS image, installer, and registry-mirror settings are configured per worker pool; these settings are no longer available at the cluster level. * Kubernetes and Talos version compatibility is checked for each worker pool. A worker Kubernetes version cannot be ahead of its parent cluster. * Worker pools support KubeVirt and Proxmox, with pool-specific sizing and disk settings. * Rendering is refused when a pool uses a different substrate from its cluster or conflicts with an existing worker pool owned by another release. * The default cluster hostname now includes the `kubernetes-` prefix. * Existing cluster configurations that still include Talos settings receive a warning; those settings are no longer used. * **Documentation** * Updated guidance for cluster and worker-pool settings, storage, upgrades, and version compatibility. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Managed Kubernetes Service
Managed Kubernetes in Cozystack
Whenever you want to deploy a custom containerized application in Cozystack, it's best to deploy it to a managed Kubernetes cluster.
Cozystack deploys and manages Kubernetes-as-a-service as standalone applications within each tenant’s isolated environment. In Cozystack, such clusters are named tenant Kubernetes clusters, while the base Cozystack cluster is called a management or root cluster. Tenant clusters are fully separated from the management cluster and are intended for deploying tenant-specific or customer-developed applications.
Within a tenant cluster, users can take advantage of LoadBalancer services and easily provision physical volumes as needed. The control plane runs in containers in the management cluster, and the worker nodes are virtual machines, grouped into pools that are separate KubernetesNodes resources.
Kubernetes version in tenant clusters is independent of Kubernetes in the management cluster. Users choose the Kubernetes minor version, from 1.31 to 1.35, and the chart deploys the patch release it pins for that minor.
Why Use a Managed Kubernetes Cluster?
Kubernetes has emerged as the industry standard, providing a unified and accessible API, primarily utilizing YAML for configuration. This means that teams can easily understand and work with Kubernetes, streamlining infrastructure management.
Kubernetes leverages robust software design patterns, enabling continuous recovery in any scenario through the reconciliation method. Additionally, it ensures seamless scaling across a multitude of servers, addressing the challenges posed by complex and outdated APIs found in traditional virtualization platforms. This managed service eliminates the need for developing custom solutions or modifying source code, saving valuable time and effort.
The Managed Kubernetes Service in Cozystack offers a streamlined solution for efficiently managing server workloads.
Starting Work
Once the tenant Kubernetes cluster is ready, you can get a kubeconfig file to work with it.
It can be done via UI or a kubectl request:
-
Open the Cozystack dashboard, switch to your tenant, find and open the application page. Copy one of the config files from the Secrets section.
-
Run the following command (using the management cluster kubeconfig):
kubectl get secret -n tenant-<name> kubernetes-<clusterName>-admin-kubeconfig -o go-template='{{ printf "%s\n" (index .data "admin.conf" | base64decode) }}' > admin.conf
There are several kubeconfig options available:
admin.conf— The standard kubeconfig for accessing your new cluster. You can create additional Kubernetes users using this configuration.admin.svc— Same token asadmin.conf, but with the API server address set to the internal service name. Use it for applications running inside the cluster that need API access.super-admin.conf— Similar toadmin.conf, but with extended administrative permissions. Intended for troubleshooting and cluster maintenance tasks.super-admin.svc— Same assuper-admin.conf, but pointing to the internal API server address.
Implementation Details
A tenant Kubernetes cluster in Cozystack is essentially Kubernetes-in-Kubernetes. Deploying it involves the following components:
-
Kamaji Control Plane: Kamaji is an open-source project that facilitates the deployment of Kubernetes control planes as pods within a root cluster. Each control plane pod includes essential components like
kube-apiserver,controller-manager, andscheduler, allowing for efficient multi-tenancy and resource utilization. -
Etcd: the control plane keeps its state in the tenant's etcd, which this application does not deploy. It is the etcd the Tenant runs when
etcdis enabled on it or on one of its parent tenants, built with the cozystack etcd-operator, and every tenant Kubernetes cluster that uses it shares it. Until that etcd exists, the application creates only a<release>-awaiting-etcdConfigMap that says so, and provisions the cluster once the etcd is there. -
Worker Nodes: worker nodes are virtual machines running Talos Linux that join the tenant cluster. They come in pools, and each pool is a separate
KubernetesNodesresource that names this cluster and sets its own size, disk, Talos image and scaling bounds; this application creates no worker nodes itself. Withsubstrate: kubevirtthe virtual machines run on KubeVirt in the management cluster, and withsubstrate: proxmoxon an external Proxmox VE cluster. -
Cluster API: Cozystack is using the Kubernetes Cluster API to provision the components of a cluster.
This architecture ensures isolated, scalable, and efficient tenant Kubernetes environments.
See the reference for components utilized in this service:
- Kamaji Control Plane
- Kamaji — Cluster API
- github.com/clastix/kamaji
- KubeVirt
- github.com/kubevirt/kubevirt
- github.com/cozystack/etcd-operator
- Kubernetes Cluster API
- github.com/kubernetes-sigs/cluster-api-provider-kubevirt
- github.com/kubevirt/csi-driver
The top-level storageClass is the default StorageClass inside the tenant cluster and cannot be changed after the cluster is created; see storageClass immutability for the contract and which consumers enforce it. Each worker pool sets the StorageClass of its own disks on its KubernetesNodes resource.
Upgrading
Changes that need action when Cozystack is upgraded are described in the upgrade notes of each release, for example Upgrading from v1.6 to v1.7.
Proxmox substrate: credentials, privileges and tenant scope
On substrate: proxmox two components authenticate to the hypervisor, and both run in the management cluster, in the tenant's namespace: the CSI controller (<release>-pcsi-controller) and the cloud-controller-manager (<release>-pccm). The tenant cluster gets the CSI node plugin, the CSIDriver object and the storage classes, none of which carry a credential. proxmox.csi.credentialsSecretName and proxmox.ccm.credentialsSecretName name Secrets in the management cluster with the keys url, token_id, token_secret and region; the tenant cannot read them, and nothing copies them into the tenant cluster.
Keeping the token out of the tenant is half of the isolation. The tenant is cluster-admin of its own cluster, and the controllers act on what the tenant's objects tell them: the CSI controller attaches the disk a PV's volumeHandle names to the VM a node's ID names, and deletes the disk a PV names, with no check that either belongs to this cluster. Under the privileges the upstream charts document — granted on / — one tenant's objects can make the controller attach, detach or delete another tenant's disks. The per-cluster volume owner id (vm-<id>-pvc-…) makes leftovers attributable; it does not scope anything.
What contains it is one Proxmox ACL per tenant, on a resource pool. Create a pool for the tenant on the hypervisor, set proxmox.pool to its name on every KubernetesNodes pool of the tenant's cluster — capmox adds each worker VM to it as it is cloned, replacements included, so the ACL follows every VM without being granted again — and give the tenant's tokens their privileges on that pool and on a storage dedicated to the tenant:
| Component | Privileges | Path |
|---|---|---|
| CSI controller | VM.Audit, VM.Config.Disk |
/pool/<proxmox.pool> |
| CSI controller | Datastore.Allocate, Datastore.AllocateSpace, Datastore.Audit |
/storage/<the tenant's storage>: Datastore.Allocate on a storage reaches every volume on it |
| CCM | VM.Audit, VM.GuestAgent.Audit |
/pool/<proxmox.pool>: a NotReady node whose VM the token cannot see is reported as gone, and the CCM deletes it |
| CCM | Sys.Audit |
/ |
Grant these to the user that owns the tokens as well as to the tokens themselves: Proxmox caps a privilege-separated token at its user's permissions, so a token whose user holds nothing has nothing, whatever its own ACL says. pveum user token permissions <user> <token> --path /pool/<proxmox.pool> shows the set that actually applies.
A storage shared between tenants cannot be scoped: Datastore.Allocate on it is what deletes volumes and also what lets any volume on it be attached. Without proxmox.pool there is no VM-level scope either: a token that reaches the tenant's VMs then reaches every VM on the hypervisor. What the pool-and-storage scope still leaves open is the tenant's own disks, which the tenant can attach, detach and delete through the controller — the same reach a cluster-admin of the tenant has anyway.
proxmox.insecure defaults to false: the controllers verify the hypervisor's certificate. A stock Proxmox VE install serves a self-signed one, so either set insecure: true for it or install a certificate the management cluster trusts.
Parameters
Common Parameters
| Name | Description | Type | Value |
|---|---|---|---|
storageClass |
Default StorageClass inside the tenant cluster. Remote-accessible LINSTOR classes are auto-propagated to the tenant under the same name. | string |
replicated |
Application-specific Parameters
| Name | Description | Type | Value |
|---|---|---|---|
version |
Kubernetes major.minor version to deploy | string |
v1.35 |
host |
External hostname for Kubernetes cluster. Defaults to kubernetes-<cluster-name>.<tenant-host> if empty. |
string |
"" |
Substrate
| Name | Description | Type | Value |
|---|---|---|---|
substrate |
Which infrastructure provider backs this cluster's worker VMs. kubevirt runs them inside this cluster; proxmox runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Every KubernetesNodes pool of this cluster must set the same substrate: a pool set to the other one refuses to render once this cluster exists. Switching an existing cluster is not supported; create a new one. |
string |
kubevirt |
proxmox |
Proxmox substrate settings. | object |
{} |
proxmox.allowedNodes |
Proxmox nodes capmox may place VMs on. Empty means every node in the Proxmox cluster. | []string |
[] |
proxmox.dnsServers |
Nameservers written into each worker's network config. Required by the ProxmoxCluster schema. | []string |
[] |
proxmox.insecure |
Skip verification of the Proxmox API server certificate in the CCM and the CSI controller, both of which run in the management cluster. Default false: the certificate is verified unless an operator turns that off. A stock Proxmox VE install serves a self-signed one, so set true for it, or install a certificate the management cluster trusts. | bool |
false |
proxmox.ipv4Config |
Address pool for workers. capmox assigns static addresses and has no DHCP mode, so this is required rather than optional. | object |
{} |
proxmox.ipv4Config.addresses |
Ranges or CIDRs, e.g. ["10.0.0.120-10.0.0.170"]. capmox turns these into an InClusterIPPool, so they must not overlap any DHCP range on the same L2 or two workers will answer to one address. |
[]string |
[] |
proxmox.ipv4Config.gateway |
Default gateway for the workers. | string |
"" |
proxmox.ipv4Config.prefix |
Netmask prefix length. | int |
24 |
proxmox.ccm |
Proxmox cloud-controller-manager settings. | object |
{} |
proxmox.ccm.credentialsSecretName |
Secret in THIS namespace holding the controller's Proxmox API credentials under the keys url, token_id, token_secret and region. May name the same Secret as csi.credentialsSecretName; it is a separate knob so the two can hold separate tokens, since this one only reads VM inventory while the driver attaches and detaches disks. |
string |
"" |
proxmox.csi |
Proxmox CSI driver settings. | object |
{} |
proxmox.csi.enabled |
Install the driver. Without it a Proxmox-backed tenant has no way to provision PersistentVolumes at all. | bool |
true |
proxmox.csi.credentialsSecretName |
Secret in THIS namespace holding the driver's Proxmox API credentials under the keys url, token_id, token_secret and region. The parent chart composes them into the controller's config file here, in the management cluster; nothing reaches the tenant, and no token is written into a rendered manifest. Distinct from the capmox credentials on purpose: the driver attaches and detaches disks, which is a different blast radius from creating VMs. |
string |
"" |
proxmox.csi.storageClasses |
StorageClasses to create in the tenant. | []object |
[] |
proxmox.csi.storageClasses[i].name |
StorageClass name inside the tenant cluster. | string |
"" |
proxmox.csi.storageClasses[i].storage |
Proxmox storage id the volumes are created on (as it appears in pvesm status). |
string |
"" |
proxmox.csi.storageClasses[i].fstype |
Filesystem the driver formats volumes with: ext4 or xfs. | string |
"" |
proxmox.csi.storageClasses[i].ssd |
Mark volumes as SSD-backed, which changes the discard/cache defaults Proxmox applies. | bool |
false |
proxmox.csi.storageClasses[i].reclaimPolicy |
Delete or Retain. | string |
"" |
Cluster Addons
| Name | Description | Type | Value |
|---|---|---|---|
addons |
Cluster addons configuration. | object |
{} |
addons.certManager |
Cert-manager addon. | object |
{} |
addons.certManager.enabled |
Enable cert-manager. | bool |
false |
addons.certManager.valuesOverride |
Custom Helm values overrides. | object |
{} |
addons.cilium |
Cilium CNI plugin. Always installed as the tenant cluster's CNI; this section only overrides its Helm values. | object |
{} |
addons.cilium.valuesOverride |
Custom Helm values overrides. | object |
{} |
addons.gatewayAPI |
Gateway API addon. | object |
{} |
addons.gatewayAPI.enabled |
Enable Gateway API. | bool |
false |
addons.ingressNginx |
Ingress-NGINX controller. | object |
{} |
addons.ingressNginx.enabled |
Enable the controller (requires nodes labeled ingress-nginx). |
bool |
false |
addons.ingressNginx.exposeMethod |
Method to expose the controller. Allowed values: Proxied, LoadBalancer. |
string |
Proxied |
addons.ingressNginx.hosts |
Domains routed to this tenant cluster when exposeMethod is Proxied. |
[]string |
[] |
addons.ingressNginx.valuesOverride |
Custom Helm values overrides. | object |
{} |
addons.gpuOperator |
NVIDIA GPU Operator. | object |
{} |
addons.gpuOperator.enabled |
Enable GPU Operator. | bool |
false |
addons.gpuOperator.valuesOverride |
Custom Helm values overrides. | object |
{} |
addons.hami |
HAMi GPU virtualization middleware. | object |
{} |
addons.hami.enabled |
Enable HAMi (requires GPU Operator). | bool |
false |
addons.hami.valuesOverride |
Custom Helm values overrides. | object |
{} |
addons.monitoringAgents |
Monitoring agents. | object |
{} |
addons.monitoringAgents.enabled |
Enable monitoring agents. | bool |
false |
addons.monitoringAgents.valuesOverride |
Custom Helm values overrides. | object |
{} |
addons.verticalPodAutoscaler |
Vertical Pod Autoscaler. Installed and removed together with addons.monitoringAgents.enabled; this section only overrides its Helm values. | object |
{} |
addons.verticalPodAutoscaler.valuesOverride |
Custom Helm values overrides. | object |
{} |
addons.velero |
Velero backup/restore addon. | object |
{} |
addons.velero.enabled |
Enable Velero. | bool |
false |
addons.velero.valuesOverride |
Custom Helm values overrides. | object |
{} |
addons.coredns |
CoreDNS addon. Always installed as the tenant cluster's DNS; this section only overrides its Helm values. | object |
{} |
addons.coredns.valuesOverride |
Custom Helm values overrides. | object |
{} |
addons.ouroboros |
Hairpin-NAT fix for ingress-nginx with PROXY-protocol. | object |
{} |
addons.ouroboros.enabled |
Enable ouroboros. Requires addons.ingressNginx.enabled (chart-render fail otherwise). Only useful when PROXY-protocol is wired on the tenant ingress-nginx via valuesOverride. | bool |
false |
addons.ouroboros.valuesOverride |
Custom Helm values overrides. Operator-key wins over cozystack defaults. | object |
{} |
Kubernetes Control Plane Configuration
| Name | Description | Type | Value |
|---|---|---|---|
controlPlane |
Kubernetes control-plane configuration. | object |
{} |
controlPlane.replicas |
Number of control-plane replicas. | int |
2 |
controlPlane.apiServer |
API Server configuration. | object |
{} |
controlPlane.apiServer.resources |
CPU and memory resources for API Server. | object |
{} |
controlPlane.apiServer.resources.cpu |
CPU available. | quantity |
"" |
controlPlane.apiServer.resources.memory |
Memory (RAM) available. | quantity |
"" |
controlPlane.apiServer.resourcesPreset |
Preset if resources omitted. |
string |
c1.medium |
controlPlane.apiServer.extraArgs |
Extra command-line flags appended to the tenant kube-apiserver, passed through to KamajiControlPlane spec.apiServer.extraArgs. For OIDC use spec.oidc.mode — this passthrough is the escape hatch for unrelated apiserver flags. The tenant control plane sets each of the following flags itself and drops an entry naming one before the apiserver sees it. The chart moves an --enable-admission-plugins= entry into controlPlane.apiServer.admissionControllers and a --kubelet-preferred-address-types= entry into controlPlane.kubelet.preferredAddressTypes while that field is empty, and refuses any other entry naming one of them: --advertise-address, --client-ca-file, --enable-admission-plugins, --service-cluster-ip-range, --kubelet-client-certificate, --kubelet-client-key, --kubelet-preferred-address-types, --proxy-client-cert-file, --proxy-client-key-file, --requestheader-allowed-names, --requestheader-client-ca-file, --secure-port, --service-account-key-file, --service-account-signing-key-file, --tls-cert-file, --tls-private-key-file, --etcd-servers, --etcd-cafile, --etcd-certfile, --etcd-keyfile, --etcd-prefix, --etcd-compaction-interval, --egress-selector-config-file. When spec.oidc.mode is not None the chart also renders OIDC flags here, reads what you have written on them, and refuses some combinations outright; the Aggregated API servers section of the Kubernetes OIDC guide on cozystack.io carries which flags, which combinations, and why. Do NOT add legacy --oidc-* flags here when spec.oidc.mode is not None; the chart injects --authentication-config and the apiserver refuses to boot with both. Empty by default. |
[]string |
[] |
controlPlane.apiServer.admissionControllers |
Admission plugins the tenant kube-apiserver enables, passed through to KamajiControlPlane spec.admissionControllers, which the control plane renders as --enable-admission-plugins. A non-empty list replaces the control plane's default set instead of adding to it. The kube-apiserver enables these plugins in addition to its own default ones, so leaving a plugin out does not turn it off; to turn one off, name it in --disable-admission-plugins in controlPlane.apiServer.extraArgs and keep it out of this list, since the apiserver refuses to start with a plugin in both, and the control plane's default set names several. Empty by default, which keeps that default set, taken from the TenantControlPlane CRD: CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, LimitRanger, MutatingAdmissionWebhook, NamespaceLifecycle, PersistentVolumeClaimResize, Priority, ResourceQuota, RuntimeClass, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. NodeRestriction is not in it. |
[]string |
[] |
controlPlane.apiServer.extraVolumes |
Extra volumes added to the control-plane Deployment, passed through to KamajiControlPlane spec.deployment.extraVolumes. Use to mount a ConfigMap or Secret holding an AuthenticationConfiguration file referenced by extraArgs. Each item is a core/v1 Volume, but because the control-plane pod runs on the management cluster only configMap and secret sources are allowed (host-reaching sources like hostPath/csi and token-projection via projected are rejected); each volume must have a unique, non-empty name and exactly one source. The names talos-ca and talos-tls-cert are reserved by the chart. Empty by default. |
[]object |
[] |
controlPlane.apiServer.extraVolumeMounts |
Extra volume mounts added to the tenant kube-apiserver container, passed through to KamajiControlPlane spec.apiServer.extraVolumeMounts. Each name must reference a volume declared in extraVolumes; the chart-managed talos secret volumes cannot be mounted. Each item is a core/v1 VolumeMount. Empty by default. |
[]object |
[] |
controlPlane.controllerManager |
Controller Manager configuration. | object |
{} |
controlPlane.controllerManager.resources |
CPU and memory resources for Controller Manager. | object |
{} |
controlPlane.controllerManager.resources.cpu |
CPU available. | quantity |
"" |
controlPlane.controllerManager.resources.memory |
Memory (RAM) available. | quantity |
"" |
controlPlane.controllerManager.resourcesPreset |
Preset if resources omitted. |
string |
t1.micro |
controlPlane.scheduler |
Scheduler configuration. | object |
{} |
controlPlane.scheduler.resources |
CPU and memory resources for Scheduler. | object |
{} |
controlPlane.scheduler.resources.cpu |
CPU available. | quantity |
"" |
controlPlane.scheduler.resources.memory |
Memory (RAM) available. | quantity |
"" |
controlPlane.scheduler.resourcesPreset |
Preset if resources omitted. |
string |
t1.micro |
controlPlane.konnectivity |
Konnectivity configuration. | object |
{} |
controlPlane.konnectivity.server |
Konnectivity Server configuration. | object |
{} |
controlPlane.konnectivity.server.resources |
CPU and memory resources for Konnectivity. | object |
{} |
controlPlane.konnectivity.server.resources.cpu |
CPU available. | quantity |
"" |
controlPlane.konnectivity.server.resources.memory |
Memory (RAM) available. | quantity |
"" |
controlPlane.konnectivity.server.resourcesPreset |
Preset if resources omitted. |
string |
t1.micro |
controlPlane.kubelet |
Kubelet connection settings. | object |
{} |
controlPlane.kubelet.preferredAddressTypes |
Node address types the tenant kube-apiserver tries, in order, when it connects to a kubelet, passed through to KamajiControlPlane spec.kubelet.preferredAddressTypes, which the control plane renders as --kubelet-preferred-address-types. The KamajiControlPlane CRD rejects a repeated name. Empty by default, which renders InternalIP, ExternalIP, or the list of a --kubelet-preferred-address-types= entry in controlPlane.apiServer.extraArgs when there is one. |
[]string |
[] |
OIDC
| Name | Description | Type | Value |
|---|---|---|---|
oidc |
OIDC authentication and per-user RBAC for the tenant kube-apiserver. See https://cozystack.io/docs/v1.6/kubernetes/oidc-authentication/ for the operator guide. | object |
{} |
oidc.mode |
Identity mode. None: no OIDC, only the static admin kubeconfig works. System: trust the platform Keycloak realm (cozy by default) via a per-cluster public client with audience binding; zero-config default. CustomConfig: trust a tenant-supplied issuer directly (BYO); cozy is not in the path. |
string |
None |
oidc.customConfig |
Tenant-supplied AuthenticationConfiguration; consumed only when mode: CustomConfig. |
object |
{} |
oidc.customConfig.config |
Inline AuthenticationConfiguration YAML (apiserver.config.k8s.io/v1beta1). The chart writes the value verbatim into a Secret mounted at /etc/kubernetes/authentication-config/config.yaml on the kube-apiserver. Mutually exclusive with secretRef.name. |
string |
"" |
oidc.customConfig.secretRef |
Reference to an existing Secret in the tenant namespace carrying the AuthenticationConfiguration. |
object |
{} |
oidc.customConfig.secretRef.name |
Name of an existing Secret in the tenant (release) namespace whose config.yaml key holds an apiserver.config.k8s.io/v1beta1 AuthenticationConfiguration. Mutually exclusive with customConfig.config. |
string |
"" |
oidc.users |
Users granted access to the tenant cluster; each entry produces one ClusterRoleBinding inside the tenant cluster. Works for both System and CustomConfig modes. |
[]object |
[] |
oidc.users[i].email |
Email address matched against the email claim from the issuer. Used verbatim as the User: subject in the ClusterRoleBinding inside the tenant cluster. The email claim is a built-in OIDC scope requested explicitly by the chart-generated kubectl oidc-login kubeconfig; every conformant OIDC provider (cozy realm included) emits it. |
string |
"" |
oidc.users[i].role |
Role to bind: admin maps to ClusterRole/cluster-admin, view maps to ClusterRole/view. |
string |
{} |
Parameter examples and reference
resources and resourcesPreset
resources sets explicit CPU and memory configurations for each replica.
Every resource it leaves unset is taken from the preset defined in resourcesPreset.
resources:
cpu: 4000m
memory: 4Gi
resourcesPreset sets named CPU and memory configurations for each replica.
This setting is ignored if the corresponding resources value is set.
| Preset name | CPU | memory |
|---|---|---|
nano |
250m |
128Mi |
micro |
500m |
256Mi |
small |
1 |
512Mi |
medium |
1 |
1Gi |
large |
2 |
2Gi |
xlarge |
4 |
4Gi |
2xlarge |
8 |
8Gi |
instanceType Resources
The following instanceType resources are provided by Cozystack:
| Name | vCPUs | Memory |
|---|---|---|
cx1.2xlarge |
8 | 16Gi |
cx1.2xlarge1gi |
8 | 16Gi |
cx1.4xlarge |
16 | 32Gi |
cx1.4xlarge1gi |
16 | 32Gi |
cx1.8xlarge |
32 | 64Gi |
cx1.8xlarge1gi |
32 | 64Gi |
cx1.large |
2 | 4Gi |
cx1.large1gi |
2 | 4Gi |
cx1.medium |
1 | 2Gi |
cx1.medium1gi |
1 | 2Gi |
cx1.xlarge |
4 | 8Gi |
cx1.xlarge1gi |
4 | 8Gi |
d1.2xlarge |
8 | 32Gi |
d1.2xmedium |
2 | 4Gi |
d1.4xlarge |
16 | 64Gi |
d1.8xlarge |
32 | 128Gi |
d1.large |
2 | 8Gi |
d1.medium |
1 | 4Gi |
d1.micro |
1 | 1Gi |
d1.nano |
1 | 512Mi |
d1.small |
1 | 2Gi |
d1.xlarge |
4 | 16Gi |
gn1.2xlarge |
8 | 32Gi |
gn1.4xlarge |
16 | 64Gi |
gn1.8xlarge |
32 | 128Gi |
gn1.xlarge |
4 | 16Gi |
m1.2xlarge |
8 | 64Gi |
m1.2xlarge1gi |
8 | 64Gi |
m1.4xlarge |
16 | 128Gi |
m1.4xlarge1gi |
16 | 128Gi |
m1.8xlarge |
32 | 256Gi |
m1.8xlarge1gi |
32 | 256Gi |
m1.large |
2 | 16Gi |
m1.large1gi |
2 | 16Gi |
m1.xlarge |
4 | 32Gi |
m1.xlarge1gi |
4 | 32Gi |
n1.2xlarge |
16 | 32Gi |
n1.4xlarge |
32 | 64Gi |
n1.8xlarge |
64 | 128Gi |
n1.large |
4 | 8Gi |
n1.medium |
4 | 4Gi |
n1.xlarge |
8 | 16Gi |
o1.2xlarge |
8 | 32Gi |
o1.4xlarge |
16 | 64Gi |
o1.8xlarge |
32 | 128Gi |
o1.large |
2 | 8Gi |
o1.medium |
1 | 4Gi |
o1.micro |
1 | 1Gi |
o1.nano |
1 | 512Mi |
o1.small |
1 | 2Gi |
o1.xlarge |
4 | 16Gi |
rt1.2xlarge |
8 | 32Gi |
rt1.4xlarge |
16 | 64Gi |
rt1.8xlarge |
32 | 128Gi |
rt1.large |
2 | 8Gi |
rt1.medium |
1 | 4Gi |
rt1.micro |
1 | 1Gi |
rt1.small |
1 | 2Gi |
rt1.xlarge |
4 | 16Gi |
u1.2xlarge |
8 | 32Gi |
u1.2xmedium |
2 | 4Gi |
u1.4xlarge |
16 | 64Gi |
u1.8xlarge |
32 | 128Gi |
u1.large |
2 | 8Gi |
u1.medium |
1 | 4Gi |
u1.micro |
1 | 1Gi |
u1.nano |
1 | 512Mi |
u1.small |
1 | 2Gi |
u1.xlarge |
4 | 16Gi |
U Series: Universal
The U Series is quite neutral and provides resources for general purpose applications.
U is the abbreviation for "Universal", hinting at the universal attitude towards workloads.
VMs of instance types will share physical CPU cores on a time-slice basis with other VMs.
U Series Characteristics
Specific characteristics of this series are:
- Burstable CPU performance - The workload has a baseline compute performance but is permitted to burst beyond this baseline, if excess compute resources are available.
- vCPU-To-Memory Ratio (1:4) - A vCPU-to-Memory ratio of 1:4, for less noise per node.
O Series: Overcommitted
The O Series is based on the U Series, with the only difference being that memory is overcommitted.
O is the abbreviation for "Overcommitted".
O Series Characteristics
Specific characteristics of this series are:
- Burstable CPU performance - The workload has a baseline compute performance but is permitted to burst beyond this baseline, if excess compute resources are available.
- Overcommitted Memory - Memory is over-committed in order to achieve a higher workload density.
- vCPU-To-Memory Ratio (1:4) - A vCPU-to-Memory ratio of 1:4, for less noise per node.
CX Series: Compute Exclusive
The CX Series provides exclusive compute resources for compute intensive applications.
CX is the abbreviation of "Compute Exclusive".
The exclusive resources are given to the compute threads of the VM. In order to ensure this, some additional cores (depending on the number of disks and NICs) will be requested to offload the IO threading from cores dedicated to the workload. In addition, in this series, the NUMA topology of the used cores is provided to the VM.
CX Series Characteristics
Specific characteristics of this series are:
- Hugepages - Hugepages are used in order to improve memory performance.
- Dedicated CPU - Physical cores are exclusively assigned to every vCPU in order to provide fixed and high compute guarantees to the workload.
- Isolated emulator threads - Hypervisor emulator threads are isolated from the vCPUs in order to reduce emaulation related impact on the workload.
- vNUMA - Physical NUMA topology is reflected in the guest in order to optimize guest sided cache utilization.
- vCPU-To-Memory Ratio (1:2) - A vCPU-to-Memory ratio of 1:2.
M Series: Memory
The M Series provides resources for memory intensive applications.
M is the abbreviation of "Memory".
M Series Characteristics
Specific characteristics of this series are:
- Hugepages - Hugepages are used in order to improve memory performance.
- Burstable CPU performance - The workload has a baseline compute performance but is permitted to burst beyond this baseline, if excess compute resources are available.
- vCPU-To-Memory Ratio (1:8) - A vCPU-to-Memory ratio of 1:8, for much less noise per node.
RT Series: RealTime
The RT Series provides resources for realtime applications, like Oslat.
RT is the abbreviation for "realtime".
This series of instance types requires nodes capable of running realtime applications.
RT Series Characteristics
Specific characteristics of this series are:
- Hugepages - Hugepages are used in order to improve memory performance.
- Dedicated CPU - Physical cores are exclusively assigned to every vCPU in order to provide fixed and high compute guarantees to the workload.
- Isolated emulator threads - Hypervisor emulator threads are isolated from the vCPUs in order to reduce emaulation related impact on the workload.
- vCPU-To-Memory Ratio (1:4) - A vCPU-to-Memory ratio of 1:4 starting from the medium size.