Files
cozystack/packages/apps/kubernetes
myasnikovdaniil aa36d3fad6 fix(kubernetes)!: remove the talos block and other leftovers of the worker pool split (#5191)
## What this PR does

Cleanup after the Kubernetes app was split into `Kubernetes` (control
plane and addons) and `KubernetesNodes` (one release per worker pool):
what the split left behind in values, docs, templates and tests.

### Kubernetes

The `Kubernetes` application has no `talos` field any more. Worker pools
are `KubernetesNodes` resources and take their Talos release, image,
installer and registry mirrors from their own `talos` and `osImage`, so
the cluster's copies did nothing. The last one, `talos.version`, fed a
duplicate of the Talos to Kubernetes support-matrix check. The pool
chart already runs that check against the Talos release each pool
actually boots, including one picked through an `osImage` override; the
cluster's copy compared the Kubernetes version against a release no node
runs. A stored spec that still carries `talos.*` keeps rendering,
because the values schema does not close the object and the API does not
validate `spec` against it, and a unit test pins that.

The pool's check now has render coverage. Its own suite pins the
pairings it must accept, and a bats test renders the refusals through
`talos.version`, `osImage.factory.version` and `osImage.builtin.version`
with schema validation skipped, since the schema offers no Kubernetes
version the matrix refuses.

I fixed the README and field descriptions that described behavior the
chart no longer has, or never had: a dedicated etcd cluster, a
`blockSize.matchVolume` setting that does not exist, a default `md0`
pool, the host default, the substrate mismatch behavior, and rollout
phase names. The README's Breaking Changes section becomes a pointer to
the per-release upgrade notes on cozystack.io. Unused helpers and stale
comments are removed; the chart renders the same bytes.

### KubernetesNodes

Five field descriptions and the `talos` block told users to keep them in
sync with parent fields that do not exist. NOTES warned about colliding
with a parent-managed `md0` pool, although the parent renders no pools
and `md0` is the normal name of a pool. The installer image was credited
to an upgrade Job that never runs one. The values descriptions, README
intro and NOTES now say what the chart renders today, including which
fields have no effect on `substrate: proxmox`: `storageClass`,
`diskSize`, `instanceType`, `talos.imageFactoryURL` (beyond its format
check) and `talos.imageFactoryCA`.

The render error for a pool whose MachineDeployment belongs to another
release told users to remove the pool from the parent's `nodeGroups`,
which cannot clear the conflict. It now names a remedy that works.
Split-era history is gone from Helm and test comments, along with two
`storageClass` fallbacks that can only return the value they fall back
from, a constant variable, and two `.helmignore` rules that match
nothing.

The tenant pod and service CIDRs are literals in both charts, and
nothing checked that they agree. A new bats test renders both charts and
fails when they differ.

No rendered manifest of the pool chart changes. I rendered it for the
snapshot values, GPU, kubelet-override, factory/CA, storageClass
variants and three Proxmox variants on main and on every commit, and the
output is byte-identical. The snapshot and TalosConfigTemplate hash
tests pass unchanged, so no worker rolls. YAML comments inside the
hashed machine config and machine template are left as they are for that
reason.

### The rule

`docs/agents/overview.md` gains a rule so this does not come back with
the next change: app READMEs and field descriptions describe current
behavior, a change users must act on goes to the website's upgrade page,
and a description does not ask the reader to keep it in sync with
another chart.

```release-note
[kubernetes] The Kubernetes application no longer has a `talos` field. Talos settings for workers, and the Talos to Kubernetes compatibility check, live on each KubernetesNodes pool. Existing clusters that still store `talos` values keep working; the values are ignored. KubernetesNodes field descriptions, README and install notes now describe the worker pool as it works today, including which fields have no effect on Proxmox pools.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Configuration**
* Worker OS image, installer, and registry-mirror settings are
configured per worker pool; these settings are no longer available at
the cluster level.
* Kubernetes and Talos version compatibility is checked for each worker
pool. A worker Kubernetes version cannot be ahead of its parent cluster.
* Worker pools support KubeVirt and Proxmox, with pool-specific sizing
and disk settings.
* Rendering is refused when a pool uses a different substrate from its
cluster or conflicts with an existing worker pool owned by another
release.
  * The default cluster hostname now includes the `kubernetes-` prefix.
* Existing cluster configurations that still include Talos settings
receive a warning; those settings are no longer used.

* **Documentation**
* Updated guidance for cluster and worker-pool settings, storage,
upgrades, and version compatibility.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-14 00:34:44 +05:00
..
2024-08-12 14:47:11 +02:00
2025-09-24 12:32:37 +02:00

Managed Kubernetes Service

Managed Kubernetes in Cozystack

Whenever you want to deploy a custom containerized application in Cozystack, it's best to deploy it to a managed Kubernetes cluster.

Cozystack deploys and manages Kubernetes-as-a-service as standalone applications within each tenant’s isolated environment. In Cozystack, such clusters are named tenant Kubernetes clusters, while the base Cozystack cluster is called a management or root cluster. Tenant clusters are fully separated from the management cluster and are intended for deploying tenant-specific or customer-developed applications.

Within a tenant cluster, users can take advantage of LoadBalancer services and easily provision physical volumes as needed. The control plane runs in containers in the management cluster, and the worker nodes are virtual machines, grouped into pools that are separate KubernetesNodes resources.

Kubernetes version in tenant clusters is independent of Kubernetes in the management cluster. Users choose the Kubernetes minor version, from 1.31 to 1.35, and the chart deploys the patch release it pins for that minor.

Why Use a Managed Kubernetes Cluster?

Kubernetes has emerged as the industry standard, providing a unified and accessible API, primarily utilizing YAML for configuration. This means that teams can easily understand and work with Kubernetes, streamlining infrastructure management.

Kubernetes leverages robust software design patterns, enabling continuous recovery in any scenario through the reconciliation method. Additionally, it ensures seamless scaling across a multitude of servers, addressing the challenges posed by complex and outdated APIs found in traditional virtualization platforms. This managed service eliminates the need for developing custom solutions or modifying source code, saving valuable time and effort.

The Managed Kubernetes Service in Cozystack offers a streamlined solution for efficiently managing server workloads.

Starting Work

Once the tenant Kubernetes cluster is ready, you can get a kubeconfig file to work with it. It can be done via UI or a kubectl request:

  • Open the Cozystack dashboard, switch to your tenant, find and open the application page. Copy one of the config files from the Secrets section.

  • Run the following command (using the management cluster kubeconfig):

    kubectl get secret -n tenant-<name> kubernetes-<clusterName>-admin-kubeconfig -o go-template='{{ printf "%s\n" (index .data "admin.conf" | base64decode) }}' > admin.conf
    

There are several kubeconfig options available:

  • admin.conf — The standard kubeconfig for accessing your new cluster. You can create additional Kubernetes users using this configuration.
  • admin.svc — Same token as admin.conf, but with the API server address set to the internal service name. Use it for applications running inside the cluster that need API access.
  • super-admin.conf — Similar to admin.conf, but with extended administrative permissions. Intended for troubleshooting and cluster maintenance tasks.
  • super-admin.svc — Same as super-admin.conf, but pointing to the internal API server address.

Implementation Details

A tenant Kubernetes cluster in Cozystack is essentially Kubernetes-in-Kubernetes. Deploying it involves the following components:

  • Kamaji Control Plane: Kamaji is an open-source project that facilitates the deployment of Kubernetes control planes as pods within a root cluster. Each control plane pod includes essential components like kube-apiserver, controller-manager, and scheduler, allowing for efficient multi-tenancy and resource utilization.

  • Etcd: the control plane keeps its state in the tenant's etcd, which this application does not deploy. It is the etcd the Tenant runs when etcd is enabled on it or on one of its parent tenants, built with the cozystack etcd-operator, and every tenant Kubernetes cluster that uses it shares it. Until that etcd exists, the application creates only a <release>-awaiting-etcd ConfigMap that says so, and provisions the cluster once the etcd is there.

  • Worker Nodes: worker nodes are virtual machines running Talos Linux that join the tenant cluster. They come in pools, and each pool is a separate KubernetesNodes resource that names this cluster and sets its own size, disk, Talos image and scaling bounds; this application creates no worker nodes itself. With substrate: kubevirt the virtual machines run on KubeVirt in the management cluster, and with substrate: proxmox on an external Proxmox VE cluster.

  • Cluster API: Cozystack is using the Kubernetes Cluster API to provision the components of a cluster.

This architecture ensures isolated, scalable, and efficient tenant Kubernetes environments.

See the reference for components utilized in this service:

The top-level storageClass is the default StorageClass inside the tenant cluster and cannot be changed after the cluster is created; see storageClass immutability for the contract and which consumers enforce it. Each worker pool sets the StorageClass of its own disks on its KubernetesNodes resource.

Upgrading

Changes that need action when Cozystack is upgraded are described in the upgrade notes of each release, for example Upgrading from v1.6 to v1.7.

Proxmox substrate: credentials, privileges and tenant scope

On substrate: proxmox two components authenticate to the hypervisor, and both run in the management cluster, in the tenant's namespace: the CSI controller (<release>-pcsi-controller) and the cloud-controller-manager (<release>-pccm). The tenant cluster gets the CSI node plugin, the CSIDriver object and the storage classes, none of which carry a credential. proxmox.csi.credentialsSecretName and proxmox.ccm.credentialsSecretName name Secrets in the management cluster with the keys url, token_id, token_secret and region; the tenant cannot read them, and nothing copies them into the tenant cluster.

Keeping the token out of the tenant is half of the isolation. The tenant is cluster-admin of its own cluster, and the controllers act on what the tenant's objects tell them: the CSI controller attaches the disk a PV's volumeHandle names to the VM a node's ID names, and deletes the disk a PV names, with no check that either belongs to this cluster. Under the privileges the upstream charts document — granted on / — one tenant's objects can make the controller attach, detach or delete another tenant's disks. The per-cluster volume owner id (vm-<id>-pvc-…) makes leftovers attributable; it does not scope anything.

What contains it is one Proxmox ACL per tenant, on a resource pool. Create a pool for the tenant on the hypervisor, set proxmox.pool to its name on every KubernetesNodes pool of the tenant's cluster — capmox adds each worker VM to it as it is cloned, replacements included, so the ACL follows every VM without being granted again — and give the tenant's tokens their privileges on that pool and on a storage dedicated to the tenant:

Component Privileges Path
CSI controller VM.Audit, VM.Config.Disk /pool/<proxmox.pool>
CSI controller Datastore.Allocate, Datastore.AllocateSpace, Datastore.Audit /storage/<the tenant's storage>: Datastore.Allocate on a storage reaches every volume on it
CCM VM.Audit, VM.GuestAgent.Audit /pool/<proxmox.pool>: a NotReady node whose VM the token cannot see is reported as gone, and the CCM deletes it
CCM Sys.Audit /

Grant these to the user that owns the tokens as well as to the tokens themselves: Proxmox caps a privilege-separated token at its user's permissions, so a token whose user holds nothing has nothing, whatever its own ACL says. pveum user token permissions <user> <token> --path /pool/<proxmox.pool> shows the set that actually applies.

A storage shared between tenants cannot be scoped: Datastore.Allocate on it is what deletes volumes and also what lets any volume on it be attached. Without proxmox.pool there is no VM-level scope either: a token that reaches the tenant's VMs then reaches every VM on the hypervisor. What the pool-and-storage scope still leaves open is the tenant's own disks, which the tenant can attach, detach and delete through the controller — the same reach a cluster-admin of the tenant has anyway.

proxmox.insecure defaults to false: the controllers verify the hypervisor's certificate. A stock Proxmox VE install serves a self-signed one, so either set insecure: true for it or install a certificate the management cluster trusts.

Parameters

Common Parameters

Name Description Type Value
storageClass Default StorageClass inside the tenant cluster. Remote-accessible LINSTOR classes are auto-propagated to the tenant under the same name. string replicated

Application-specific Parameters

Name Description Type Value
version Kubernetes major.minor version to deploy string v1.35
host External hostname for Kubernetes cluster. Defaults to kubernetes-<cluster-name>.<tenant-host> if empty. string ""

Substrate

Name Description Type Value
substrate Which infrastructure provider backs this cluster's worker VMs. kubevirt runs them inside this cluster; proxmox runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Every KubernetesNodes pool of this cluster must set the same substrate: a pool set to the other one refuses to render once this cluster exists. Switching an existing cluster is not supported; create a new one. string kubevirt
proxmox Proxmox substrate settings. object {}
proxmox.allowedNodes Proxmox nodes capmox may place VMs on. Empty means every node in the Proxmox cluster. []string []
proxmox.dnsServers Nameservers written into each worker's network config. Required by the ProxmoxCluster schema. []string []
proxmox.insecure Skip verification of the Proxmox API server certificate in the CCM and the CSI controller, both of which run in the management cluster. Default false: the certificate is verified unless an operator turns that off. A stock Proxmox VE install serves a self-signed one, so set true for it, or install a certificate the management cluster trusts. bool false
proxmox.ipv4Config Address pool for workers. capmox assigns static addresses and has no DHCP mode, so this is required rather than optional. object {}
proxmox.ipv4Config.addresses Ranges or CIDRs, e.g. ["10.0.0.120-10.0.0.170"]. capmox turns these into an InClusterIPPool, so they must not overlap any DHCP range on the same L2 or two workers will answer to one address. []string []
proxmox.ipv4Config.gateway Default gateway for the workers. string ""
proxmox.ipv4Config.prefix Netmask prefix length. int 24
proxmox.ccm Proxmox cloud-controller-manager settings. object {}
proxmox.ccm.credentialsSecretName Secret in THIS namespace holding the controller's Proxmox API credentials under the keys url, token_id, token_secret and region. May name the same Secret as csi.credentialsSecretName; it is a separate knob so the two can hold separate tokens, since this one only reads VM inventory while the driver attaches and detaches disks. string ""
proxmox.csi Proxmox CSI driver settings. object {}
proxmox.csi.enabled Install the driver. Without it a Proxmox-backed tenant has no way to provision PersistentVolumes at all. bool true
proxmox.csi.credentialsSecretName Secret in THIS namespace holding the driver's Proxmox API credentials under the keys url, token_id, token_secret and region. The parent chart composes them into the controller's config file here, in the management cluster; nothing reaches the tenant, and no token is written into a rendered manifest. Distinct from the capmox credentials on purpose: the driver attaches and detaches disks, which is a different blast radius from creating VMs. string ""
proxmox.csi.storageClasses StorageClasses to create in the tenant. []object []
proxmox.csi.storageClasses[i].name StorageClass name inside the tenant cluster. string ""
proxmox.csi.storageClasses[i].storage Proxmox storage id the volumes are created on (as it appears in pvesm status). string ""
proxmox.csi.storageClasses[i].fstype Filesystem the driver formats volumes with: ext4 or xfs. string ""
proxmox.csi.storageClasses[i].ssd Mark volumes as SSD-backed, which changes the discard/cache defaults Proxmox applies. bool false
proxmox.csi.storageClasses[i].reclaimPolicy Delete or Retain. string ""

Cluster Addons

Name Description Type Value
addons Cluster addons configuration. object {}
addons.certManager Cert-manager addon. object {}
addons.certManager.enabled Enable cert-manager. bool false
addons.certManager.valuesOverride Custom Helm values overrides. object {}
addons.cilium Cilium CNI plugin. Always installed as the tenant cluster's CNI; this section only overrides its Helm values. object {}
addons.cilium.valuesOverride Custom Helm values overrides. object {}
addons.gatewayAPI Gateway API addon. object {}
addons.gatewayAPI.enabled Enable Gateway API. bool false
addons.ingressNginx Ingress-NGINX controller. object {}
addons.ingressNginx.enabled Enable the controller (requires nodes labeled ingress-nginx). bool false
addons.ingressNginx.exposeMethod Method to expose the controller. Allowed values: Proxied, LoadBalancer. string Proxied
addons.ingressNginx.hosts Domains routed to this tenant cluster when exposeMethod is Proxied. []string []
addons.ingressNginx.valuesOverride Custom Helm values overrides. object {}
addons.gpuOperator NVIDIA GPU Operator. object {}
addons.gpuOperator.enabled Enable GPU Operator. bool false
addons.gpuOperator.valuesOverride Custom Helm values overrides. object {}
addons.hami HAMi GPU virtualization middleware. object {}
addons.hami.enabled Enable HAMi (requires GPU Operator). bool false
addons.hami.valuesOverride Custom Helm values overrides. object {}
addons.monitoringAgents Monitoring agents. object {}
addons.monitoringAgents.enabled Enable monitoring agents. bool false
addons.monitoringAgents.valuesOverride Custom Helm values overrides. object {}
addons.verticalPodAutoscaler Vertical Pod Autoscaler. Installed and removed together with addons.monitoringAgents.enabled; this section only overrides its Helm values. object {}
addons.verticalPodAutoscaler.valuesOverride Custom Helm values overrides. object {}
addons.velero Velero backup/restore addon. object {}
addons.velero.enabled Enable Velero. bool false
addons.velero.valuesOverride Custom Helm values overrides. object {}
addons.coredns CoreDNS addon. Always installed as the tenant cluster's DNS; this section only overrides its Helm values. object {}
addons.coredns.valuesOverride Custom Helm values overrides. object {}
addons.ouroboros Hairpin-NAT fix for ingress-nginx with PROXY-protocol. object {}
addons.ouroboros.enabled Enable ouroboros. Requires addons.ingressNginx.enabled (chart-render fail otherwise). Only useful when PROXY-protocol is wired on the tenant ingress-nginx via valuesOverride. bool false
addons.ouroboros.valuesOverride Custom Helm values overrides. Operator-key wins over cozystack defaults. object {}

Kubernetes Control Plane Configuration

Name Description Type Value
controlPlane Kubernetes control-plane configuration. object {}
controlPlane.replicas Number of control-plane replicas. int 2
controlPlane.apiServer API Server configuration. object {}
controlPlane.apiServer.resources CPU and memory resources for API Server. object {}
controlPlane.apiServer.resources.cpu CPU available. quantity ""
controlPlane.apiServer.resources.memory Memory (RAM) available. quantity ""
controlPlane.apiServer.resourcesPreset Preset if resources omitted. string c1.medium
controlPlane.apiServer.extraArgs Extra command-line flags appended to the tenant kube-apiserver, passed through to KamajiControlPlane spec.apiServer.extraArgs. For OIDC use spec.oidc.mode — this passthrough is the escape hatch for unrelated apiserver flags. The tenant control plane sets each of the following flags itself and drops an entry naming one before the apiserver sees it. The chart moves an --enable-admission-plugins= entry into controlPlane.apiServer.admissionControllers and a --kubelet-preferred-address-types= entry into controlPlane.kubelet.preferredAddressTypes while that field is empty, and refuses any other entry naming one of them: --advertise-address, --client-ca-file, --enable-admission-plugins, --service-cluster-ip-range, --kubelet-client-certificate, --kubelet-client-key, --kubelet-preferred-address-types, --proxy-client-cert-file, --proxy-client-key-file, --requestheader-allowed-names, --requestheader-client-ca-file, --secure-port, --service-account-key-file, --service-account-signing-key-file, --tls-cert-file, --tls-private-key-file, --etcd-servers, --etcd-cafile, --etcd-certfile, --etcd-keyfile, --etcd-prefix, --etcd-compaction-interval, --egress-selector-config-file. When spec.oidc.mode is not None the chart also renders OIDC flags here, reads what you have written on them, and refuses some combinations outright; the Aggregated API servers section of the Kubernetes OIDC guide on cozystack.io carries which flags, which combinations, and why. Do NOT add legacy --oidc-* flags here when spec.oidc.mode is not None; the chart injects --authentication-config and the apiserver refuses to boot with both. Empty by default. []string []
controlPlane.apiServer.admissionControllers Admission plugins the tenant kube-apiserver enables, passed through to KamajiControlPlane spec.admissionControllers, which the control plane renders as --enable-admission-plugins. A non-empty list replaces the control plane's default set instead of adding to it. The kube-apiserver enables these plugins in addition to its own default ones, so leaving a plugin out does not turn it off; to turn one off, name it in --disable-admission-plugins in controlPlane.apiServer.extraArgs and keep it out of this list, since the apiserver refuses to start with a plugin in both, and the control plane's default set names several. Empty by default, which keeps that default set, taken from the TenantControlPlane CRD: CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, LimitRanger, MutatingAdmissionWebhook, NamespaceLifecycle, PersistentVolumeClaimResize, Priority, ResourceQuota, RuntimeClass, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. NodeRestriction is not in it. []string []
controlPlane.apiServer.extraVolumes Extra volumes added to the control-plane Deployment, passed through to KamajiControlPlane spec.deployment.extraVolumes. Use to mount a ConfigMap or Secret holding an AuthenticationConfiguration file referenced by extraArgs. Each item is a core/v1 Volume, but because the control-plane pod runs on the management cluster only configMap and secret sources are allowed (host-reaching sources like hostPath/csi and token-projection via projected are rejected); each volume must have a unique, non-empty name and exactly one source. The names talos-ca and talos-tls-cert are reserved by the chart. Empty by default. []object []
controlPlane.apiServer.extraVolumeMounts Extra volume mounts added to the tenant kube-apiserver container, passed through to KamajiControlPlane spec.apiServer.extraVolumeMounts. Each name must reference a volume declared in extraVolumes; the chart-managed talos secret volumes cannot be mounted. Each item is a core/v1 VolumeMount. Empty by default. []object []
controlPlane.controllerManager Controller Manager configuration. object {}
controlPlane.controllerManager.resources CPU and memory resources for Controller Manager. object {}
controlPlane.controllerManager.resources.cpu CPU available. quantity ""
controlPlane.controllerManager.resources.memory Memory (RAM) available. quantity ""
controlPlane.controllerManager.resourcesPreset Preset if resources omitted. string t1.micro
controlPlane.scheduler Scheduler configuration. object {}
controlPlane.scheduler.resources CPU and memory resources for Scheduler. object {}
controlPlane.scheduler.resources.cpu CPU available. quantity ""
controlPlane.scheduler.resources.memory Memory (RAM) available. quantity ""
controlPlane.scheduler.resourcesPreset Preset if resources omitted. string t1.micro
controlPlane.konnectivity Konnectivity configuration. object {}
controlPlane.konnectivity.server Konnectivity Server configuration. object {}
controlPlane.konnectivity.server.resources CPU and memory resources for Konnectivity. object {}
controlPlane.konnectivity.server.resources.cpu CPU available. quantity ""
controlPlane.konnectivity.server.resources.memory Memory (RAM) available. quantity ""
controlPlane.konnectivity.server.resourcesPreset Preset if resources omitted. string t1.micro
controlPlane.kubelet Kubelet connection settings. object {}
controlPlane.kubelet.preferredAddressTypes Node address types the tenant kube-apiserver tries, in order, when it connects to a kubelet, passed through to KamajiControlPlane spec.kubelet.preferredAddressTypes, which the control plane renders as --kubelet-preferred-address-types. The KamajiControlPlane CRD rejects a repeated name. Empty by default, which renders InternalIP, ExternalIP, or the list of a --kubelet-preferred-address-types= entry in controlPlane.apiServer.extraArgs when there is one. []string []

OIDC

Name Description Type Value
oidc OIDC authentication and per-user RBAC for the tenant kube-apiserver. See https://cozystack.io/docs/v1.6/kubernetes/oidc-authentication/ for the operator guide. object {}
oidc.mode Identity mode. None: no OIDC, only the static admin kubeconfig works. System: trust the platform Keycloak realm (cozy by default) via a per-cluster public client with audience binding; zero-config default. CustomConfig: trust a tenant-supplied issuer directly (BYO); cozy is not in the path. string None
oidc.customConfig Tenant-supplied AuthenticationConfiguration; consumed only when mode: CustomConfig. object {}
oidc.customConfig.config Inline AuthenticationConfiguration YAML (apiserver.config.k8s.io/v1beta1). The chart writes the value verbatim into a Secret mounted at /etc/kubernetes/authentication-config/config.yaml on the kube-apiserver. Mutually exclusive with secretRef.name. string ""
oidc.customConfig.secretRef Reference to an existing Secret in the tenant namespace carrying the AuthenticationConfiguration. object {}
oidc.customConfig.secretRef.name Name of an existing Secret in the tenant (release) namespace whose config.yaml key holds an apiserver.config.k8s.io/v1beta1 AuthenticationConfiguration. Mutually exclusive with customConfig.config. string ""
oidc.users Users granted access to the tenant cluster; each entry produces one ClusterRoleBinding inside the tenant cluster. Works for both System and CustomConfig modes. []object []
oidc.users[i].email Email address matched against the email claim from the issuer. Used verbatim as the User: subject in the ClusterRoleBinding inside the tenant cluster. The email claim is a built-in OIDC scope requested explicitly by the chart-generated kubectl oidc-login kubeconfig; every conformant OIDC provider (cozy realm included) emits it. string ""
oidc.users[i].role Role to bind: admin maps to ClusterRole/cluster-admin, view maps to ClusterRole/view. string {}

Parameter examples and reference

resources and resourcesPreset

resources sets explicit CPU and memory configurations for each replica. Every resource it leaves unset is taken from the preset defined in resourcesPreset.

resources:
  cpu: 4000m
  memory: 4Gi

resourcesPreset sets named CPU and memory configurations for each replica. This setting is ignored if the corresponding resources value is set.

Preset name CPU memory
nano 250m 128Mi
micro 500m 256Mi
small 1 512Mi
medium 1 1Gi
large 2 2Gi
xlarge 4 4Gi
2xlarge 8 8Gi

instanceType Resources

The following instanceType resources are provided by Cozystack:

Name vCPUs Memory
cx1.2xlarge 8 16Gi
cx1.2xlarge1gi 8 16Gi
cx1.4xlarge 16 32Gi
cx1.4xlarge1gi 16 32Gi
cx1.8xlarge 32 64Gi
cx1.8xlarge1gi 32 64Gi
cx1.large 2 4Gi
cx1.large1gi 2 4Gi
cx1.medium 1 2Gi
cx1.medium1gi 1 2Gi
cx1.xlarge 4 8Gi
cx1.xlarge1gi 4 8Gi
d1.2xlarge 8 32Gi
d1.2xmedium 2 4Gi
d1.4xlarge 16 64Gi
d1.8xlarge 32 128Gi
d1.large 2 8Gi
d1.medium 1 4Gi
d1.micro 1 1Gi
d1.nano 1 512Mi
d1.small 1 2Gi
d1.xlarge 4 16Gi
gn1.2xlarge 8 32Gi
gn1.4xlarge 16 64Gi
gn1.8xlarge 32 128Gi
gn1.xlarge 4 16Gi
m1.2xlarge 8 64Gi
m1.2xlarge1gi 8 64Gi
m1.4xlarge 16 128Gi
m1.4xlarge1gi 16 128Gi
m1.8xlarge 32 256Gi
m1.8xlarge1gi 32 256Gi
m1.large 2 16Gi
m1.large1gi 2 16Gi
m1.xlarge 4 32Gi
m1.xlarge1gi 4 32Gi
n1.2xlarge 16 32Gi
n1.4xlarge 32 64Gi
n1.8xlarge 64 128Gi
n1.large 4 8Gi
n1.medium 4 4Gi
n1.xlarge 8 16Gi
o1.2xlarge 8 32Gi
o1.4xlarge 16 64Gi
o1.8xlarge 32 128Gi
o1.large 2 8Gi
o1.medium 1 4Gi
o1.micro 1 1Gi
o1.nano 1 512Mi
o1.small 1 2Gi
o1.xlarge 4 16Gi
rt1.2xlarge 8 32Gi
rt1.4xlarge 16 64Gi
rt1.8xlarge 32 128Gi
rt1.large 2 8Gi
rt1.medium 1 4Gi
rt1.micro 1 1Gi
rt1.small 1 2Gi
rt1.xlarge 4 16Gi
u1.2xlarge 8 32Gi
u1.2xmedium 2 4Gi
u1.4xlarge 16 64Gi
u1.8xlarge 32 128Gi
u1.large 2 8Gi
u1.medium 1 4Gi
u1.micro 1 1Gi
u1.nano 1 512Mi
u1.small 1 2Gi
u1.xlarge 4 16Gi

U Series: Universal

The U Series is quite neutral and provides resources for general purpose applications.

U is the abbreviation for "Universal", hinting at the universal attitude towards workloads.

VMs of instance types will share physical CPU cores on a time-slice basis with other VMs.

U Series Characteristics

Specific characteristics of this series are:

  • Burstable CPU performance - The workload has a baseline compute performance but is permitted to burst beyond this baseline, if excess compute resources are available.
  • vCPU-To-Memory Ratio (1:4) - A vCPU-to-Memory ratio of 1:4, for less noise per node.

O Series: Overcommitted

The O Series is based on the U Series, with the only difference being that memory is overcommitted.

O is the abbreviation for "Overcommitted".

O Series Characteristics

Specific characteristics of this series are:

  • Burstable CPU performance - The workload has a baseline compute performance but is permitted to burst beyond this baseline, if excess compute resources are available.
  • Overcommitted Memory - Memory is over-committed in order to achieve a higher workload density.
  • vCPU-To-Memory Ratio (1:4) - A vCPU-to-Memory ratio of 1:4, for less noise per node.

CX Series: Compute Exclusive

The CX Series provides exclusive compute resources for compute intensive applications.

CX is the abbreviation of "Compute Exclusive".

The exclusive resources are given to the compute threads of the VM. In order to ensure this, some additional cores (depending on the number of disks and NICs) will be requested to offload the IO threading from cores dedicated to the workload. In addition, in this series, the NUMA topology of the used cores is provided to the VM.

CX Series Characteristics

Specific characteristics of this series are:

  • Hugepages - Hugepages are used in order to improve memory performance.
  • Dedicated CPU - Physical cores are exclusively assigned to every vCPU in order to provide fixed and high compute guarantees to the workload.
  • Isolated emulator threads - Hypervisor emulator threads are isolated from the vCPUs in order to reduce emaulation related impact on the workload.
  • vNUMA - Physical NUMA topology is reflected in the guest in order to optimize guest sided cache utilization.
  • vCPU-To-Memory Ratio (1:2) - A vCPU-to-Memory ratio of 1:2.

M Series: Memory

The M Series provides resources for memory intensive applications.

M is the abbreviation of "Memory".

M Series Characteristics

Specific characteristics of this series are:

  • Hugepages - Hugepages are used in order to improve memory performance.
  • Burstable CPU performance - The workload has a baseline compute performance but is permitted to burst beyond this baseline, if excess compute resources are available.
  • vCPU-To-Memory Ratio (1:8) - A vCPU-to-Memory ratio of 1:8, for much less noise per node.

RT Series: RealTime

The RT Series provides resources for realtime applications, like Oslat.

RT is the abbreviation for "realtime".

This series of instance types requires nodes capable of running realtime applications.

RT Series Characteristics

Specific characteristics of this series are:

  • Hugepages - Hugepages are used in order to improve memory performance.
  • Dedicated CPU - Physical cores are exclusively assigned to every vCPU in order to provide fixed and high compute guarantees to the workload.
  • Isolated emulator threads - Hypervisor emulator threads are isolated from the vCPUs in order to reduce emaulation related impact on the workload.
  • vCPU-To-Memory Ratio (1:4) - A vCPU-to-Memory ratio of 1:4 starting from the medium size.