mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 12:13:55 +09:00
docs(kubernetes): describe the cluster as it is and drop the history
The README told users two things the chart does not do: that it deploys a dedicated etcd cluster (the control plane uses the tenant's etcd, which the Tenant deploys), and that worker disks match the volume's block size through blockSize.matchVolume, a setting that exists nowhere in the tree. It also described worker nodes as KubeVirt VMs managed by this application, while they are KubernetesNodes pools on KubeVirt or Proxmox, and offered "patch versions" where the user picks a minor. The Breaking Changes section was a release log: entries from several releases, most of them about fields that live on KubernetesNodes now, and each one written as a change rather than as the current behaviour. What a user has to do on upgrade belongs on the per-release upgrade page of the website, so the README points there and keeps only the storageClass immutability note, rewritten as current behaviour. The field descriptions are what the dashboard, the API schema and a rejected name show the user, and several were written against how the application used to work. The name limit was explained by a default md0 pool that no longer exists, while the 32-character cap leaves room for any three-character pool name. The host default left out the kubernetes- prefix the release name carries. A substrate mismatch with a pool was said to leave Machines unreconciled, while the pool refuses to render. The OIDC type carried a roadmap phase in its title, and extraArgs promised "no change to current behavior". The name-cap test looked for "md0" in the rejection message as its sign that the chart's explanation arrives; it looks for the release-name pattern the new explanation gives. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
This commit is contained in:
6 files changed
+78
-97
No files matched your search
Generated
+3
-3
@@ -25,10 +25,10 @@ type ConfigSpec struct {
|
||||
// Kubernetes major.minor version to deploy
|
||||
// +kubebuilder:default:="v1.35"
|
||||
Version Version `json:"version"`
|
||||
// External hostname for Kubernetes cluster. Defaults to `<cluster-name>.<tenant-host>` if empty.
|
||||
// External hostname for Kubernetes cluster. Defaults to `kubernetes-<cluster-name>.<tenant-host>` if empty.
|
||||
// +kubebuilder:default:=""
|
||||
Host string `json:"host"`
|
||||
// Which infrastructure provider backs this cluster's worker VMs. `kubevirt` runs them inside this cluster; `proxmox` runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Must match the `substrate` of every KubernetesNodes pool attached to this cluster: the pools reference this cluster's infrastructure object by kind, and a mismatch leaves their Machines unreconciled. Switching an existing cluster is not supported; create a new one.
|
||||
// Which infrastructure provider backs this cluster's worker VMs. `kubevirt` runs them inside this cluster; `proxmox` runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Every KubernetesNodes pool of this cluster must set the same `substrate`: a pool set to the other one refuses to render once this cluster exists. Switching an existing cluster is not supported; create a new one.
|
||||
// +kubebuilder:default:="kubevirt"
|
||||
// +kubebuilder:validation:XValidation:rule="self == oldSelf",message="substrate is immutable"
|
||||
Substrate string `json:"substrate"`
|
||||
@@ -50,7 +50,7 @@ type APIServer struct {
|
||||
// Admission plugins the tenant kube-apiserver enables, passed through to KamajiControlPlane `spec.admissionControllers`, which the control plane renders as `--enable-admission-plugins`. A non-empty list replaces the control plane's default set instead of adding to it. The kube-apiserver enables these plugins in addition to its own default ones, so leaving a plugin out does not turn it off; to turn one off, name it in `--disable-admission-plugins` in `controlPlane.apiServer.extraArgs` and keep it out of this list, since the apiserver refuses to start with a plugin in both, and the control plane's default set names several. Empty by default, which keeps that default set, taken from the TenantControlPlane CRD: CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, LimitRanger, MutatingAdmissionWebhook, NamespaceLifecycle, PersistentVolumeClaimResize, Priority, ResourceQuota, RuntimeClass, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. `NodeRestriction` is not in it.
|
||||
// +kubebuilder:default:={}
|
||||
AdmissionControllers []AdmissionController `json:"admissionControllers,omitempty"`
|
||||
// Extra command-line flags appended to the tenant kube-apiserver, passed through to KamajiControlPlane `spec.apiServer.extraArgs`. For OIDC use `spec.oidc.mode` — this passthrough is the escape hatch for unrelated apiserver flags. The tenant control plane sets each of the following flags itself and drops an entry naming one before the apiserver sees it. The chart moves an `--enable-admission-plugins=` entry into `controlPlane.apiServer.admissionControllers` and a `--kubelet-preferred-address-types=` entry into `controlPlane.kubelet.preferredAddressTypes` while that field is empty, and refuses any other entry naming one of them: `--advertise-address`, `--client-ca-file`, `--enable-admission-plugins`, `--service-cluster-ip-range`, `--kubelet-client-certificate`, `--kubelet-client-key`, `--kubelet-preferred-address-types`, `--proxy-client-cert-file`, `--proxy-client-key-file`, `--requestheader-allowed-names`, `--requestheader-client-ca-file`, `--secure-port`, `--service-account-key-file`, `--service-account-signing-key-file`, `--tls-cert-file`, `--tls-private-key-file`, `--etcd-servers`, `--etcd-cafile`, `--etcd-certfile`, `--etcd-keyfile`, `--etcd-prefix`, `--etcd-compaction-interval`, `--egress-selector-config-file`. When `spec.oidc.mode` is not `None` the chart also renders OIDC flags here, reads what you have written on them, and refuses some combinations outright; the Aggregated API servers section of the Kubernetes OIDC guide on cozystack.io carries which flags, which combinations, and why. Do NOT add legacy `--oidc-*` flags here when `spec.oidc.mode` is not `None`; the chart injects `--authentication-config` and the apiserver refuses to boot with both. Empty by default (no change to current behavior).
|
||||
// Extra command-line flags appended to the tenant kube-apiserver, passed through to KamajiControlPlane `spec.apiServer.extraArgs`. For OIDC use `spec.oidc.mode` — this passthrough is the escape hatch for unrelated apiserver flags. The tenant control plane sets each of the following flags itself and drops an entry naming one before the apiserver sees it. The chart moves an `--enable-admission-plugins=` entry into `controlPlane.apiServer.admissionControllers` and a `--kubelet-preferred-address-types=` entry into `controlPlane.kubelet.preferredAddressTypes` while that field is empty, and refuses any other entry naming one of them: `--advertise-address`, `--client-ca-file`, `--enable-admission-plugins`, `--service-cluster-ip-range`, `--kubelet-client-certificate`, `--kubelet-client-key`, `--kubelet-preferred-address-types`, `--proxy-client-cert-file`, `--proxy-client-key-file`, `--requestheader-allowed-names`, `--requestheader-client-ca-file`, `--secure-port`, `--service-account-key-file`, `--service-account-signing-key-file`, `--tls-cert-file`, `--tls-private-key-file`, `--etcd-servers`, `--etcd-cafile`, `--etcd-certfile`, `--etcd-keyfile`, `--etcd-prefix`, `--etcd-compaction-interval`, `--egress-selector-config-file`. When `spec.oidc.mode` is not `None` the chart also renders OIDC flags here, reads what you have written on them, and refuses some combinations outright; the Aggregated API servers section of the Kubernetes OIDC guide on cozystack.io carries which flags, which combinations, and why. Do NOT add legacy `--oidc-*` flags here when `spec.oidc.mode` is not `None`; the chart injects `--authentication-config` and the apiserver refuses to boot with both. Empty by default.
|
||||
// +kubebuilder:default:={}
|
||||
ExtraArgs []string `json:"extraArgs,omitempty"`
|
||||
// Extra volume mounts added to the tenant kube-apiserver container, passed through to KamajiControlPlane `spec.apiServer.extraVolumeMounts`. Each `name` must reference a volume declared in `extraVolumes`; the chart-managed talos secret volumes cannot be mounted. Each item is a core/v1 VolumeMount. Empty by default.
|
||||
|
||||
@@ -8,11 +8,9 @@ Cozystack deploys and manages Kubernetes-as-a-service as standalone applications
|
||||
In Cozystack, such clusters are named tenant Kubernetes clusters, while the base Cozystack cluster is called a management or root cluster.
|
||||
Tenant clusters are fully separated from the management cluster and are intended for deploying tenant-specific or customer-developed applications.
|
||||
|
||||
Within a tenant cluster, users can take advantage of LoadBalancer services and easily provision physical volumes as needed.
|
||||
The control-plane operates within containers, while the worker nodes are deployed as virtual machines, all seamlessly managed by the application.
|
||||
Within a tenant cluster, users can take advantage of LoadBalancer services and easily provision physical volumes as needed. The control plane runs in containers in the management cluster, and the worker nodes are virtual machines, grouped into pools that are separate `KubernetesNodes` resources.
|
||||
|
||||
Kubernetes version in tenant clusters is independent of Kubernetes in the management cluster.
|
||||
Users can select the supported patch versions from 1.31 to 1.35.
|
||||
Kubernetes version in tenant clusters is independent of Kubernetes in the management cluster. Users choose the Kubernetes minor version, from 1.31 to 1.35, and the chart deploys the patch release it pins for that minor.
|
||||
|
||||
## Why Use a Managed Kubernetes Cluster?
|
||||
|
||||
@@ -58,14 +56,9 @@ Deploying it involves the following components:
|
||||
Each control plane pod includes essential components like `kube-apiserver`, `controller-manager`, and `scheduler`,
|
||||
allowing for efficient multi-tenancy and resource utilization.
|
||||
|
||||
- **Etcd Cluster**: A dedicated etcd cluster is deployed using the cozystack [etcd-operator](https://github.com/cozystack/etcd-operator) (`etcd-operator.cozystack.io/v1alpha2`).
|
||||
It provides reliable and scalable key-value storage for the Kubernetes control plane.
|
||||
- **Etcd**: the control plane keeps its state in the tenant's etcd, which this application does not deploy. It is the etcd the Tenant runs when `etcd` is enabled on it or on one of its parent tenants, built with the cozystack [etcd-operator](https://github.com/cozystack/etcd-operator), and every tenant Kubernetes cluster that uses it shares it. Until that etcd exists, the application creates only a `<release>-awaiting-etcd` ConfigMap that says so, and provisions the cluster once the etcd is there.
|
||||
|
||||
- **Worker Nodes**: Virtual Machines are provisioned to serve as worker nodes using KubeVirt.
|
||||
These nodes are configured to join the tenant Kubernetes cluster, enabling the deployment and management of workloads.
|
||||
Worker node disks automatically detect and match the underlying volume's block size
|
||||
(`blockSize.matchVolume`) to ensure compatibility with storage backends that use
|
||||
non-512-byte sectors, such as LINSTOR DRBD with 4Kn drives.
|
||||
- **Worker Nodes**: worker nodes are virtual machines running Talos Linux that join the tenant cluster. They come in pools, and each pool is a separate `KubernetesNodes` resource that names this cluster and sets its own size, disk, Talos image and scaling bounds; this application creates no worker nodes itself. With `substrate: kubevirt` the virtual machines run on KubeVirt in the management cluster, and with `substrate: proxmox` on an external Proxmox VE cluster.
|
||||
|
||||
- **Cluster API**: Cozystack is using the [Kubernetes Cluster API](https://cluster-api.sigs.k8s.io/) to provision the components of a cluster.
|
||||
|
||||
@@ -83,23 +76,11 @@ See the reference for components utilized in this service:
|
||||
- [github.com/kubernetes-sigs/cluster-api-provider-kubevirt](https://github.com/kubernetes-sigs/cluster-api-provider-kubevirt)
|
||||
- [github.com/kubevirt/csi-driver](https://github.com/kubevirt/csi-driver)
|
||||
|
||||
## Breaking Changes
|
||||
The top-level `storageClass` is the default StorageClass inside the tenant cluster and cannot be changed after the cluster is created; see [storageClass immutability](https://cozystack.io/docs/v1/operations/storage-class-immutability/) for the contract and which consumers enforce it. Each worker pool sets the StorageClass of its own disks on its `KubernetesNodes` resource.
|
||||
|
||||
- **`ephemeralStorage` renamed to `diskSize`** (v1.4): The `nodeGroups[name].ephemeralStorage` field has been renamed to `nodeGroups[name].diskSize`. Existing clusters are migrated transparently by platform migration 41 during the pre-upgrade hook — no manual action is required. Newly written values should use `diskSize`. Existing VMs are rolling-updated via CAPI when the new values are applied. With the Talos worker rollover in this release the field now sizes the **single system disk** (Talos OS image streamed from `factory.talos.dev`, kubelet state, containerd image cache, local-path PVCs) — the pre-Talos `disk-kubelet` PVC layout was removed. State on that single disk persists across same-VM reboots (virt-launcher restart, guest reboot, node failure); VM replacement by CAPI (e.g. nodeGroup field change, MachineHealthCheck remediation) provisions a fresh disk.
|
||||
## Upgrading
|
||||
|
||||
- **Air-gapped tenant workers (Phase 1 Talos rollover)**: tenant worker bootstrap moves from Ubuntu containerDisk + kubeadm to a Talos OS disk image fetched over HTTP by CDI and a `.../installer/...` installer reference. Both sources are now overridable on each worker pool, which is a KubernetesNodes resource: set its `talos.imageFactoryURL` (default `https://factory.talos.dev`) to a self-hosted Image Factory, a caching mirror, or an internal HTTP file server, and its `talos.installerRepository` (default `factory.talos.dev/installer`) to a mirrored registry. The `Kubernetes` resource itself has no `talos` field. The Helm-rendered `*-patch-containerd` Secret that previously plumbed the platform-wide `registries.mirrors` config to tenant workers (via the kubeadm template's containerd certs.d mount) has no consumer in the Talos machineconfig, so the Secret is removed together with the platform `registries` value that fed it. A Package that still sets `registries` keeps installing, the key is ignored. Registry mirrors for tenant workers, **in-guest container-image pulls** (pods pulling from registries inside the tenant cluster) included, are configured per worker pool through `talos.registryMirrors` of the `KubernetesNodes` resource.
|
||||
|
||||
- **Worker MachineHealthCheck remediation is now enabled by default** (Phase 1 Talos rollover): the worker MHC `maxUnhealthy` moved from a hard-coded `0` (remediation effectively **off**) to a configurable `nodeHealthCheck.maxUnhealthy` defaulting to `"50%"`. CAPI now auto-remediates (deletes and replaces) unhealthy worker Machines while at least 50% of a group stays healthy. `"50%"` deliberately leaves headroom for transient unhealthy nodes during the kubeadm→Talos rollover and slow first boots from `factory.talos.dev`; set `nodeHealthCheck.maxUnhealthy: "0%"` to keep the previous remediation-off behaviour until your fleet is stable on Talos workers.
|
||||
|
||||
- **GitOps-managed tenant `Kubernetes` CRs must bump `spec.version` in Git** (Phase 1 Talos rollover): platform migration 46 patches live `kuberneteses.apps.cozystack.io` objects from `v1.30` to `v1.31` (v1.30 left the Talos↔Kubernetes support matrix). When the tenant `Kubernetes` CR is reconciled from Git (Flux/Argo), the next source reconcile re-applies `version: v1.30` over migration 46's patch, which then trips the chart's `_versions.tpl` guard and the HelmRelease fails. Update `spec.version` to `v1.31` (or newer) in your Git source before/with the platform upgrade. Tenants managed via the API or dashboard need no action — migration 46 handles them.
|
||||
|
||||
- **Remote-accessible LINSTOR StorageClasses are auto-propagated to tenants** (v1.5): infra-cluster LINSTOR StorageClasses whose `linstor.csi.linbit.com/allowRemoteVolumeAccess` is not `"false"` are created inside each tenant under the same name (provisioned by `csi.kubevirt.io`). **Upgrade caveat:** delete any *manually created* tenant StorageClass whose name collides with a propagated infra class (e.g. a hand-made `replicated`) before upgrading, or the propagated class cannot be created and the tenant CSI release will not converge. Infra classes that must stay node-local need `allowRemoteVolumeAccess: "false"` set explicitly — an absent annotation is treated as remote-accessible. Propagation is evaluated at HelmRelease render time, so classes added or removed on the infra cluster after a tenant exists propagate only on that tenant's next reconcile.
|
||||
|
||||
- **Worker pools split into `KubernetesNodes` (Phase 2)**: `spec.nodeGroups`, `spec.nodeHealthCheck` and `spec.maxNodeProvisionTime` are removed from the `Kubernetes` CR — worker pools are now separate `KubernetesNodes` resources (one HelmRelease per pool; see the `kubernetes-nodes` chart), on which `spec.cluster` and `spec.storageClass` are immutable (`self == oldSelf`). Existing pools are adopted automatically on upgrade by platform migration 56 with no worker-VM churn. Consequences: (1) the `Kubernetes` CR's `WorkloadsReady` condition is now **control-plane-only** — worker health is reported on each `KubernetesNodes` resource, so a dashboard or `kubectl wait` watching the parent no longer reflects worker readiness; (2) enabling `addons.ingressNginx` now requires a pool carrying `roles: [ingress-nginx]` (the implicit default `md0` is gone — without such a pool the controller DaemonSet schedules nothing); (3) do **not** edit `Kubernetes` CRs during a platform upgrade — a mid-window parent render can transiently fail Helm ownership validation while migration 56 re-annotates worker objects, and self-heals once the new chart artifact lands; (4) when deleting a cluster, delete its `KubernetesNodes` pools **before** the parent `Kubernetes` resource, so each pool's pre-delete unpin hook can run while the tenant apiserver and the CAPI CRDs still exist — deleting the parent first can leave the pool release reconciling CAPI objects against a cluster that is being torn down (in the pre-split monolith the pools were part of the parent release and died with it; they are now separate resources with their own lifecycle); (5) the removed fields (`nodeGroups`, `nodeHealthCheck`, `maxNodeProvisionTime`) are still **accepted and stored** on an already-upgraded `Kubernetes` CR — the adoption migration deliberately leaves them in place rather than rewriting parent values mid-upgrade — but they have **no effect**: editing them (for example scaling via `spec.nodeGroups.<pool>.minReplicas`) does nothing, and the API returns an admission warning pointing you at the `KubernetesNodes` resources. A cluster name is also capped at 32 characters so every pool's `KubernetesNodes` release name fits the 53-character Helm limit. ComputePlane clusters (the `computeplane` module) are affected the same way: their worker pools are rendered as separate per-pool `KubernetesNodes` releases too (materialised directly by the module rather than adopted), so the same pool-name length limit applies to their `nodeGroups` keys.
|
||||
|
||||
- **`controlPlane.apiServer.extraArgs` refuses flags the tenant control plane owns** (Kamaji 26.9.4-edge): the tenant control plane now sets a list of kube-apiserver flags itself and drops an `extraArgs` entry naming one, where the previous build let the entry win. The entry stops taking effect once the new build reconciles the control plane, whatever the chart renders. Two of those flags have a field the control plane renders them from, and the chart moves an entry written as `--flag=value` into it while the field is empty: `--enable-admission-plugins` into `controlPlane.apiServer.admissionControllers`, `--kubelet-preferred-address-types` into `controlPlane.kubelet.preferredAddressTypes`. The KamajiControlPlane CRD accepts only the plugin names it lists, so a plugin outside that list that the apiserver leaves off by default can no longer be enabled, and a moved name outside it fails when the KamajiControlPlane is applied, with an error that names it, rather than at render. The chart refuses to render any other entry naming one of the flags, so a tenant carrying one stops reconciling until the entry is removed; the description of `controlPlane.apiServer.extraArgs` lists the flags. The same Kamaji update passes every `extraArgs` entry to the apiserver, so repeats of a flag no longer collapse to the last one: naming the kube feature gates both bare and `kube:`-prefixed across entries, or giving two `--emulated-version` entries that name kube, now stops the apiserver from starting. The chart refuses neither, in any `spec.oidc.mode`: both reach the apiserver as written. Keep one spelling for kube gate terms and one kube emulated version.
|
||||
|
||||
> The top-level `storageClass` field is annotated as immutable in the chart schema — see [storageClass immutability](https://cozystack.io/docs/v1/operations/storage-class-immutability/) for the contract and which consumers enforce it. The per-pool worker `storageClass` now lives on the `KubernetesNodes` resource (see the `kubernetes-nodes` chart).
|
||||
Changes that need action when Cozystack is upgraded are described in the upgrade notes of each release, for example [Upgrading from v1.6 to v1.7](https://cozystack.io/docs/v1.7/operations/cluster/upgrade-to-v1.7/).
|
||||
|
||||
## Proxmox substrate: credentials, privileges and tenant scope
|
||||
|
||||
@@ -133,36 +114,36 @@ A storage shared between tenants cannot be scoped: `Datastore.Allocate` on it is
|
||||
|
||||
### Application-specific Parameters
|
||||
|
||||
| Name | Description | Type | Value |
|
||||
| --------- | ---------------------------------------------------------------------------------------------- | -------- | ------- |
|
||||
| `version` | Kubernetes major.minor version to deploy | `string` | `v1.35` |
|
||||
| `host` | External hostname for Kubernetes cluster. Defaults to `<cluster-name>.<tenant-host>` if empty. | `string` | `""` |
|
||||
| Name | Description | Type | Value |
|
||||
| --------- | --------------------------------------------------------------------------------------------------------- | -------- | ------- |
|
||||
| `version` | Kubernetes major.minor version to deploy | `string` | `v1.35` |
|
||||
| `host` | External hostname for Kubernetes cluster. Defaults to `kubernetes-<cluster-name>.<tenant-host>` if empty. | `string` | `""` |
|
||||
|
||||
|
||||
### Substrate
|
||||
|
||||
| Name | Description | Type | Value |
|
||||
| --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ---------- |
|
||||
| `substrate` | Which infrastructure provider backs this cluster's worker VMs. `kubevirt` runs them inside this cluster; `proxmox` runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Must match the `substrate` of every KubernetesNodes pool attached to this cluster: the pools reference this cluster's infrastructure object by kind, and a mismatch leaves their Machines unreconciled. Switching an existing cluster is not supported; create a new one. | `string` | `kubevirt` |
|
||||
| `proxmox` | Proxmox substrate settings. | `object` | `{}` |
|
||||
| `proxmox.allowedNodes` | Proxmox nodes capmox may place VMs on. Empty means every node in the Proxmox cluster. | `[]string` | `[]` |
|
||||
| `proxmox.dnsServers` | Nameservers written into each worker's network config. Required by the ProxmoxCluster schema. | `[]string` | `[]` |
|
||||
| `proxmox.insecure` | Skip verification of the Proxmox API server certificate in the CCM and the CSI controller, both of which run in the management cluster. Default false: the certificate is verified unless an operator turns that off. A stock Proxmox VE install serves a self-signed one, so set true for it, or install a certificate the management cluster trusts. | `bool` | `false` |
|
||||
| `proxmox.ipv4Config` | Address pool for workers. capmox assigns static addresses and has no DHCP mode, so this is required rather than optional. | `object` | `{}` |
|
||||
| `proxmox.ipv4Config.addresses` | Ranges or CIDRs, e.g. `["10.0.0.120-10.0.0.170"]`. capmox turns these into an InClusterIPPool, so they must not overlap any DHCP range on the same L2 or two workers will answer to one address. | `[]string` | `[]` |
|
||||
| `proxmox.ipv4Config.gateway` | Default gateway for the workers. | `string` | `""` |
|
||||
| `proxmox.ipv4Config.prefix` | Netmask prefix length. | `int` | `24` |
|
||||
| `proxmox.ccm` | Proxmox cloud-controller-manager settings. | `object` | `{}` |
|
||||
| `proxmox.ccm.credentialsSecretName` | Secret in THIS namespace holding the controller's Proxmox API credentials under the keys `url`, `token_id`, `token_secret` and `region`. May name the same Secret as `csi.credentialsSecretName`; it is a separate knob so the two can hold separate tokens, since this one only reads VM inventory while the driver attaches and detaches disks. | `string` | `""` |
|
||||
| `proxmox.csi` | Proxmox CSI driver settings. | `object` | `{}` |
|
||||
| `proxmox.csi.enabled` | Install the driver. Without it a Proxmox-backed tenant has no way to provision PersistentVolumes at all. | `bool` | `true` |
|
||||
| `proxmox.csi.credentialsSecretName` | Secret in THIS namespace holding the driver's Proxmox API credentials under the keys `url`, `token_id`, `token_secret` and `region`. The parent chart composes them into the controller's config file here, in the management cluster; nothing reaches the tenant, and no token is written into a rendered manifest. Distinct from the capmox credentials on purpose: the driver attaches and detaches disks, which is a different blast radius from creating VMs. | `string` | `""` |
|
||||
| `proxmox.csi.storageClasses` | StorageClasses to create in the tenant. | `[]object` | `[]` |
|
||||
| `proxmox.csi.storageClasses[i].name` | StorageClass name inside the tenant cluster. | `string` | `""` |
|
||||
| `proxmox.csi.storageClasses[i].storage` | Proxmox storage id the volumes are created on (as it appears in `pvesm status`). | `string` | `""` |
|
||||
| `proxmox.csi.storageClasses[i].fstype` | Filesystem the driver formats volumes with: ext4 or xfs. | `string` | `""` |
|
||||
| `proxmox.csi.storageClasses[i].ssd` | Mark volumes as SSD-backed, which changes the discard/cache defaults Proxmox applies. | `bool` | `false` |
|
||||
| `proxmox.csi.storageClasses[i].reclaimPolicy` | Delete or Retain. | `string` | `""` |
|
||||
| Name | Description | Type | Value |
|
||||
| --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ---------- |
|
||||
| `substrate` | Which infrastructure provider backs this cluster's worker VMs. `kubevirt` runs them inside this cluster; `proxmox` runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Every KubernetesNodes pool of this cluster must set the same `substrate`: a pool set to the other one refuses to render once this cluster exists. Switching an existing cluster is not supported; create a new one. | `string` | `kubevirt` |
|
||||
| `proxmox` | Proxmox substrate settings. | `object` | `{}` |
|
||||
| `proxmox.allowedNodes` | Proxmox nodes capmox may place VMs on. Empty means every node in the Proxmox cluster. | `[]string` | `[]` |
|
||||
| `proxmox.dnsServers` | Nameservers written into each worker's network config. Required by the ProxmoxCluster schema. | `[]string` | `[]` |
|
||||
| `proxmox.insecure` | Skip verification of the Proxmox API server certificate in the CCM and the CSI controller, both of which run in the management cluster. Default false: the certificate is verified unless an operator turns that off. A stock Proxmox VE install serves a self-signed one, so set true for it, or install a certificate the management cluster trusts. | `bool` | `false` |
|
||||
| `proxmox.ipv4Config` | Address pool for workers. capmox assigns static addresses and has no DHCP mode, so this is required rather than optional. | `object` | `{}` |
|
||||
| `proxmox.ipv4Config.addresses` | Ranges or CIDRs, e.g. `["10.0.0.120-10.0.0.170"]`. capmox turns these into an InClusterIPPool, so they must not overlap any DHCP range on the same L2 or two workers will answer to one address. | `[]string` | `[]` |
|
||||
| `proxmox.ipv4Config.gateway` | Default gateway for the workers. | `string` | `""` |
|
||||
| `proxmox.ipv4Config.prefix` | Netmask prefix length. | `int` | `24` |
|
||||
| `proxmox.ccm` | Proxmox cloud-controller-manager settings. | `object` | `{}` |
|
||||
| `proxmox.ccm.credentialsSecretName` | Secret in THIS namespace holding the controller's Proxmox API credentials under the keys `url`, `token_id`, `token_secret` and `region`. May name the same Secret as `csi.credentialsSecretName`; it is a separate knob so the two can hold separate tokens, since this one only reads VM inventory while the driver attaches and detaches disks. | `string` | `""` |
|
||||
| `proxmox.csi` | Proxmox CSI driver settings. | `object` | `{}` |
|
||||
| `proxmox.csi.enabled` | Install the driver. Without it a Proxmox-backed tenant has no way to provision PersistentVolumes at all. | `bool` | `true` |
|
||||
| `proxmox.csi.credentialsSecretName` | Secret in THIS namespace holding the driver's Proxmox API credentials under the keys `url`, `token_id`, `token_secret` and `region`. The parent chart composes them into the controller's config file here, in the management cluster; nothing reaches the tenant, and no token is written into a rendered manifest. Distinct from the capmox credentials on purpose: the driver attaches and detaches disks, which is a different blast radius from creating VMs. | `string` | `""` |
|
||||
| `proxmox.csi.storageClasses` | StorageClasses to create in the tenant. | `[]object` | `[]` |
|
||||
| `proxmox.csi.storageClasses[i].name` | StorageClass name inside the tenant cluster. | `string` | `""` |
|
||||
| `proxmox.csi.storageClasses[i].storage` | Proxmox storage id the volumes are created on (as it appears in `pvesm status`). | `string` | `""` |
|
||||
| `proxmox.csi.storageClasses[i].fstype` | Filesystem the driver formats volumes with: ext4 or xfs. | `string` | `""` |
|
||||
| `proxmox.csi.storageClasses[i].ssd` | Mark volumes as SSD-backed, which changes the discard/cache defaults Proxmox applies. | `bool` | `false` |
|
||||
| `proxmox.csi.storageClasses[i].reclaimPolicy` | Delete or Retain. | `string` | `""` |
|
||||
|
||||
|
||||
### Cluster Addons
|
||||
@@ -205,37 +186,37 @@ A storage shared between tenants cannot be scoped: `Datastore.Allocate` on it is
|
||||
|
||||
### Kubernetes Control Plane Configuration
|
||||
|
||||
| Name | Description | Type | Value |
|
||||
| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ----------- |
|
||||
| `controlPlane` | Kubernetes control-plane configuration. | `object` | `{}` |
|
||||
| `controlPlane.replicas` | Number of control-plane replicas. | `int` | `2` |
|
||||
| `controlPlane.apiServer` | API Server configuration. | `object` | `{}` |
|
||||
| `controlPlane.apiServer.resources` | CPU and memory resources for API Server. | `object` | `{}` |
|
||||
| `controlPlane.apiServer.resources.cpu` | CPU available. | `quantity` | `""` |
|
||||
| `controlPlane.apiServer.resources.memory` | Memory (RAM) available. | `quantity` | `""` |
|
||||
| `controlPlane.apiServer.resourcesPreset` | Preset if `resources` omitted. | `string` | `c1.medium` |
|
||||
| `controlPlane.apiServer.extraArgs` | Extra command-line flags appended to the tenant kube-apiserver, passed through to KamajiControlPlane `spec.apiServer.extraArgs`. For OIDC use `spec.oidc.mode` — this passthrough is the escape hatch for unrelated apiserver flags. The tenant control plane sets each of the following flags itself and drops an entry naming one before the apiserver sees it. The chart moves an `--enable-admission-plugins=` entry into `controlPlane.apiServer.admissionControllers` and a `--kubelet-preferred-address-types=` entry into `controlPlane.kubelet.preferredAddressTypes` while that field is empty, and refuses any other entry naming one of them: `--advertise-address`, `--client-ca-file`, `--enable-admission-plugins`, `--service-cluster-ip-range`, `--kubelet-client-certificate`, `--kubelet-client-key`, `--kubelet-preferred-address-types`, `--proxy-client-cert-file`, `--proxy-client-key-file`, `--requestheader-allowed-names`, `--requestheader-client-ca-file`, `--secure-port`, `--service-account-key-file`, `--service-account-signing-key-file`, `--tls-cert-file`, `--tls-private-key-file`, `--etcd-servers`, `--etcd-cafile`, `--etcd-certfile`, `--etcd-keyfile`, `--etcd-prefix`, `--etcd-compaction-interval`, `--egress-selector-config-file`. When `spec.oidc.mode` is not `None` the chart also renders OIDC flags here, reads what you have written on them, and refuses some combinations outright; the Aggregated API servers section of the Kubernetes OIDC guide on cozystack.io carries which flags, which combinations, and why. Do NOT add legacy `--oidc-*` flags here when `spec.oidc.mode` is not `None`; the chart injects `--authentication-config` and the apiserver refuses to boot with both. Empty by default (no change to current behavior). | `[]string` | `[]` |
|
||||
| `controlPlane.apiServer.admissionControllers` | Admission plugins the tenant kube-apiserver enables, passed through to KamajiControlPlane `spec.admissionControllers`, which the control plane renders as `--enable-admission-plugins`. A non-empty list replaces the control plane's default set instead of adding to it. The kube-apiserver enables these plugins in addition to its own default ones, so leaving a plugin out does not turn it off; to turn one off, name it in `--disable-admission-plugins` in `controlPlane.apiServer.extraArgs` and keep it out of this list, since the apiserver refuses to start with a plugin in both, and the control plane's default set names several. Empty by default, which keeps that default set, taken from the TenantControlPlane CRD: CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, LimitRanger, MutatingAdmissionWebhook, NamespaceLifecycle, PersistentVolumeClaimResize, Priority, ResourceQuota, RuntimeClass, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. `NodeRestriction` is not in it. | `[]string` | `[]` |
|
||||
| `controlPlane.apiServer.extraVolumes` | Extra volumes added to the control-plane Deployment, passed through to KamajiControlPlane `spec.deployment.extraVolumes`. Use to mount a ConfigMap or Secret holding an AuthenticationConfiguration file referenced by `extraArgs`. Each item is a core/v1 Volume, but because the control-plane pod runs on the management cluster only `configMap` and `secret` sources are allowed (host-reaching sources like `hostPath`/`csi` and token-projection via `projected` are rejected); each volume must have a unique, non-empty name and exactly one source. The names `talos-ca` and `talos-tls-cert` are reserved by the chart. Empty by default. | `[]object` | `[]` |
|
||||
| `controlPlane.apiServer.extraVolumeMounts` | Extra volume mounts added to the tenant kube-apiserver container, passed through to KamajiControlPlane `spec.apiServer.extraVolumeMounts`. Each `name` must reference a volume declared in `extraVolumes`; the chart-managed talos secret volumes cannot be mounted. Each item is a core/v1 VolumeMount. Empty by default. | `[]object` | `[]` |
|
||||
| `controlPlane.controllerManager` | Controller Manager configuration. | `object` | `{}` |
|
||||
| `controlPlane.controllerManager.resources` | CPU and memory resources for Controller Manager. | `object` | `{}` |
|
||||
| `controlPlane.controllerManager.resources.cpu` | CPU available. | `quantity` | `""` |
|
||||
| `controlPlane.controllerManager.resources.memory` | Memory (RAM) available. | `quantity` | `""` |
|
||||
| `controlPlane.controllerManager.resourcesPreset` | Preset if `resources` omitted. | `string` | `t1.micro` |
|
||||
| `controlPlane.scheduler` | Scheduler configuration. | `object` | `{}` |
|
||||
| `controlPlane.scheduler.resources` | CPU and memory resources for Scheduler. | `object` | `{}` |
|
||||
| `controlPlane.scheduler.resources.cpu` | CPU available. | `quantity` | `""` |
|
||||
| `controlPlane.scheduler.resources.memory` | Memory (RAM) available. | `quantity` | `""` |
|
||||
| `controlPlane.scheduler.resourcesPreset` | Preset if `resources` omitted. | `string` | `t1.micro` |
|
||||
| `controlPlane.konnectivity` | Konnectivity configuration. | `object` | `{}` |
|
||||
| `controlPlane.konnectivity.server` | Konnectivity Server configuration. | `object` | `{}` |
|
||||
| `controlPlane.konnectivity.server.resources` | CPU and memory resources for Konnectivity. | `object` | `{}` |
|
||||
| `controlPlane.konnectivity.server.resources.cpu` | CPU available. | `quantity` | `""` |
|
||||
| `controlPlane.konnectivity.server.resources.memory` | Memory (RAM) available. | `quantity` | `""` |
|
||||
| `controlPlane.konnectivity.server.resourcesPreset` | Preset if `resources` omitted. | `string` | `t1.micro` |
|
||||
| `controlPlane.kubelet` | Kubelet connection settings. | `object` | `{}` |
|
||||
| `controlPlane.kubelet.preferredAddressTypes` | Node address types the tenant kube-apiserver tries, in order, when it connects to a kubelet, passed through to KamajiControlPlane `spec.kubelet.preferredAddressTypes`, which the control plane renders as `--kubelet-preferred-address-types`. The KamajiControlPlane CRD rejects a repeated name. Empty by default, which renders `InternalIP`, `ExternalIP`, or the list of a `--kubelet-preferred-address-types=` entry in `controlPlane.apiServer.extraArgs` when there is one. | `[]string` | `[]` |
|
||||
| Name | Description | Type | Value |
|
||||
| --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ----------- |
|
||||
| `controlPlane` | Kubernetes control-plane configuration. | `object` | `{}` |
|
||||
| `controlPlane.replicas` | Number of control-plane replicas. | `int` | `2` |
|
||||
| `controlPlane.apiServer` | API Server configuration. | `object` | `{}` |
|
||||
| `controlPlane.apiServer.resources` | CPU and memory resources for API Server. | `object` | `{}` |
|
||||
| `controlPlane.apiServer.resources.cpu` | CPU available. | `quantity` | `""` |
|
||||
| `controlPlane.apiServer.resources.memory` | Memory (RAM) available. | `quantity` | `""` |
|
||||
| `controlPlane.apiServer.resourcesPreset` | Preset if `resources` omitted. | `string` | `c1.medium` |
|
||||
| `controlPlane.apiServer.extraArgs` | Extra command-line flags appended to the tenant kube-apiserver, passed through to KamajiControlPlane `spec.apiServer.extraArgs`. For OIDC use `spec.oidc.mode` — this passthrough is the escape hatch for unrelated apiserver flags. The tenant control plane sets each of the following flags itself and drops an entry naming one before the apiserver sees it. The chart moves an `--enable-admission-plugins=` entry into `controlPlane.apiServer.admissionControllers` and a `--kubelet-preferred-address-types=` entry into `controlPlane.kubelet.preferredAddressTypes` while that field is empty, and refuses any other entry naming one of them: `--advertise-address`, `--client-ca-file`, `--enable-admission-plugins`, `--service-cluster-ip-range`, `--kubelet-client-certificate`, `--kubelet-client-key`, `--kubelet-preferred-address-types`, `--proxy-client-cert-file`, `--proxy-client-key-file`, `--requestheader-allowed-names`, `--requestheader-client-ca-file`, `--secure-port`, `--service-account-key-file`, `--service-account-signing-key-file`, `--tls-cert-file`, `--tls-private-key-file`, `--etcd-servers`, `--etcd-cafile`, `--etcd-certfile`, `--etcd-keyfile`, `--etcd-prefix`, `--etcd-compaction-interval`, `--egress-selector-config-file`. When `spec.oidc.mode` is not `None` the chart also renders OIDC flags here, reads what you have written on them, and refuses some combinations outright; the Aggregated API servers section of the Kubernetes OIDC guide on cozystack.io carries which flags, which combinations, and why. Do NOT add legacy `--oidc-*` flags here when `spec.oidc.mode` is not `None`; the chart injects `--authentication-config` and the apiserver refuses to boot with both. Empty by default. | `[]string` | `[]` |
|
||||
| `controlPlane.apiServer.admissionControllers` | Admission plugins the tenant kube-apiserver enables, passed through to KamajiControlPlane `spec.admissionControllers`, which the control plane renders as `--enable-admission-plugins`. A non-empty list replaces the control plane's default set instead of adding to it. The kube-apiserver enables these plugins in addition to its own default ones, so leaving a plugin out does not turn it off; to turn one off, name it in `--disable-admission-plugins` in `controlPlane.apiServer.extraArgs` and keep it out of this list, since the apiserver refuses to start with a plugin in both, and the control plane's default set names several. Empty by default, which keeps that default set, taken from the TenantControlPlane CRD: CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, LimitRanger, MutatingAdmissionWebhook, NamespaceLifecycle, PersistentVolumeClaimResize, Priority, ResourceQuota, RuntimeClass, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. `NodeRestriction` is not in it. | `[]string` | `[]` |
|
||||
| `controlPlane.apiServer.extraVolumes` | Extra volumes added to the control-plane Deployment, passed through to KamajiControlPlane `spec.deployment.extraVolumes`. Use to mount a ConfigMap or Secret holding an AuthenticationConfiguration file referenced by `extraArgs`. Each item is a core/v1 Volume, but because the control-plane pod runs on the management cluster only `configMap` and `secret` sources are allowed (host-reaching sources like `hostPath`/`csi` and token-projection via `projected` are rejected); each volume must have a unique, non-empty name and exactly one source. The names `talos-ca` and `talos-tls-cert` are reserved by the chart. Empty by default. | `[]object` | `[]` |
|
||||
| `controlPlane.apiServer.extraVolumeMounts` | Extra volume mounts added to the tenant kube-apiserver container, passed through to KamajiControlPlane `spec.apiServer.extraVolumeMounts`. Each `name` must reference a volume declared in `extraVolumes`; the chart-managed talos secret volumes cannot be mounted. Each item is a core/v1 VolumeMount. Empty by default. | `[]object` | `[]` |
|
||||
| `controlPlane.controllerManager` | Controller Manager configuration. | `object` | `{}` |
|
||||
| `controlPlane.controllerManager.resources` | CPU and memory resources for Controller Manager. | `object` | `{}` |
|
||||
| `controlPlane.controllerManager.resources.cpu` | CPU available. | `quantity` | `""` |
|
||||
| `controlPlane.controllerManager.resources.memory` | Memory (RAM) available. | `quantity` | `""` |
|
||||
| `controlPlane.controllerManager.resourcesPreset` | Preset if `resources` omitted. | `string` | `t1.micro` |
|
||||
| `controlPlane.scheduler` | Scheduler configuration. | `object` | `{}` |
|
||||
| `controlPlane.scheduler.resources` | CPU and memory resources for Scheduler. | `object` | `{}` |
|
||||
| `controlPlane.scheduler.resources.cpu` | CPU available. | `quantity` | `""` |
|
||||
| `controlPlane.scheduler.resources.memory` | Memory (RAM) available. | `quantity` | `""` |
|
||||
| `controlPlane.scheduler.resourcesPreset` | Preset if `resources` omitted. | `string` | `t1.micro` |
|
||||
| `controlPlane.konnectivity` | Konnectivity configuration. | `object` | `{}` |
|
||||
| `controlPlane.konnectivity.server` | Konnectivity Server configuration. | `object` | `{}` |
|
||||
| `controlPlane.konnectivity.server.resources` | CPU and memory resources for Konnectivity. | `object` | `{}` |
|
||||
| `controlPlane.konnectivity.server.resources.cpu` | CPU available. | `quantity` | `""` |
|
||||
| `controlPlane.konnectivity.server.resources.memory` | Memory (RAM) available. | `quantity` | `""` |
|
||||
| `controlPlane.konnectivity.server.resourcesPreset` | Preset if `resources` omitted. | `string` | `t1.micro` |
|
||||
| `controlPlane.kubelet` | Kubelet connection settings. | `object` | `{}` |
|
||||
| `controlPlane.kubelet.preferredAddressTypes` | Node address types the tenant kube-apiserver tries, in order, when it connects to a kubelet, passed through to KamajiControlPlane `spec.kubelet.preferredAddressTypes`, which the control plane renders as `--kubelet-preferred-address-types`. The KamajiControlPlane CRD rejects a repeated name. Empty by default, which renders `InternalIP`, `ExternalIP`, or the list of a `--kubelet-preferred-address-types=` entry in `controlPlane.apiServer.extraArgs` when there is one. | `[]string` | `[]` |
|
||||
|
||||
|
||||
### OIDC
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
"type": "object",
|
||||
"x-cozystack-name": {
|
||||
"type": "string",
|
||||
"description": "Cluster name. Every worker pool is a separate KubernetesNodes release named `kubernetes-nodes-\u003cname\u003e-\u003cpool\u003e`, so the name must leave room for the default `md0` pool inside the 53-character Helm release name limit.",
|
||||
"description": "Cluster name. Each worker pool is a KubernetesNodes release named `kubernetes-nodes-\u003cname\u003e-\u003cpool\u003e`, which has to fit the 53-character Helm release name limit, so a cluster name of at most 32 characters leaves at least three characters for the pool name.",
|
||||
"maxLength": 32
|
||||
},
|
||||
"properties": {
|
||||
@@ -34,12 +34,12 @@
|
||||
]
|
||||
},
|
||||
"host": {
|
||||
"description": "External hostname for Kubernetes cluster. Defaults to `\u003ccluster-name\u003e.\u003ctenant-host\u003e` if empty.",
|
||||
"description": "External hostname for Kubernetes cluster. Defaults to `kubernetes-\u003ccluster-name\u003e.\u003ctenant-host\u003e` if empty.",
|
||||
"type": "string",
|
||||
"default": ""
|
||||
},
|
||||
"substrate": {
|
||||
"description": "Which infrastructure provider backs this cluster's worker VMs. `kubevirt` runs them inside this cluster; `proxmox` runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Must match the `substrate` of every KubernetesNodes pool attached to this cluster: the pools reference this cluster's infrastructure object by kind, and a mismatch leaves their Machines unreconciled. Switching an existing cluster is not supported; create a new one.",
|
||||
"description": "Which infrastructure provider backs this cluster's worker VMs. `kubevirt` runs them inside this cluster; `proxmox` runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Every KubernetesNodes pool of this cluster must set the same `substrate`: a pool set to the other one refuses to render once this cluster exists. Switching an existing cluster is not supported; create a new one.",
|
||||
"type": "string",
|
||||
"default": "kubevirt",
|
||||
"x-kubernetes-validations": [
|
||||
@@ -506,7 +506,7 @@
|
||||
}
|
||||
},
|
||||
"extraArgs": {
|
||||
"description": "Extra command-line flags appended to the tenant kube-apiserver, passed through to KamajiControlPlane `spec.apiServer.extraArgs`. For OIDC use `spec.oidc.mode` — this passthrough is the escape hatch for unrelated apiserver flags. The tenant control plane sets each of the following flags itself and drops an entry naming one before the apiserver sees it. The chart moves an `--enable-admission-plugins=` entry into `controlPlane.apiServer.admissionControllers` and a `--kubelet-preferred-address-types=` entry into `controlPlane.kubelet.preferredAddressTypes` while that field is empty, and refuses any other entry naming one of them: `--advertise-address`, `--client-ca-file`, `--enable-admission-plugins`, `--service-cluster-ip-range`, `--kubelet-client-certificate`, `--kubelet-client-key`, `--kubelet-preferred-address-types`, `--proxy-client-cert-file`, `--proxy-client-key-file`, `--requestheader-allowed-names`, `--requestheader-client-ca-file`, `--secure-port`, `--service-account-key-file`, `--service-account-signing-key-file`, `--tls-cert-file`, `--tls-private-key-file`, `--etcd-servers`, `--etcd-cafile`, `--etcd-certfile`, `--etcd-keyfile`, `--etcd-prefix`, `--etcd-compaction-interval`, `--egress-selector-config-file`. When `spec.oidc.mode` is not `None` the chart also renders OIDC flags here, reads what you have written on them, and refuses some combinations outright; the Aggregated API servers section of the Kubernetes OIDC guide on cozystack.io carries which flags, which combinations, and why. Do NOT add legacy `--oidc-*` flags here when `spec.oidc.mode` is not `None`; the chart injects `--authentication-config` and the apiserver refuses to boot with both. Empty by default (no change to current behavior).",
|
||||
"description": "Extra command-line flags appended to the tenant kube-apiserver, passed through to KamajiControlPlane `spec.apiServer.extraArgs`. For OIDC use `spec.oidc.mode` — this passthrough is the escape hatch for unrelated apiserver flags. The tenant control plane sets each of the following flags itself and drops an entry naming one before the apiserver sees it. The chart moves an `--enable-admission-plugins=` entry into `controlPlane.apiServer.admissionControllers` and a `--kubelet-preferred-address-types=` entry into `controlPlane.kubelet.preferredAddressTypes` while that field is empty, and refuses any other entry naming one of them: `--advertise-address`, `--client-ca-file`, `--enable-admission-plugins`, `--service-cluster-ip-range`, `--kubelet-client-certificate`, `--kubelet-client-key`, `--kubelet-preferred-address-types`, `--proxy-client-cert-file`, `--proxy-client-key-file`, `--requestheader-allowed-names`, `--requestheader-client-ca-file`, `--secure-port`, `--service-account-key-file`, `--service-account-signing-key-file`, `--tls-cert-file`, `--tls-private-key-file`, `--etcd-servers`, `--etcd-cafile`, `--etcd-certfile`, `--etcd-keyfile`, `--etcd-prefix`, `--etcd-compaction-interval`, `--egress-selector-config-file`. When `spec.oidc.mode` is not `None` the chart also renders OIDC flags here, reads what you have written on them, and refuses some combinations outright; the Aggregated API servers section of the Kubernetes OIDC guide on cozystack.io carries which flags, which combinations, and why. Do NOT add legacy `--oidc-*` flags here when `spec.oidc.mode` is not `None`; the chart injects `--authentication-config` and the apiserver refuses to boot with both. Empty by default.",
|
||||
"type": "array",
|
||||
"default": [],
|
||||
"items": {
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
## @name {string} - Cluster name. Every worker pool is a separate KubernetesNodes release named `kubernetes-nodes-<name>-<pool>`, so the name must leave room for the default `md0` pool inside the 53-character Helm release name limit.
|
||||
## @name {string} - Cluster name. Each worker pool is a KubernetesNodes release named `kubernetes-nodes-<name>-<pool>`, which has to fit the 53-character Helm release name limit, so a cluster name of at most 32 characters leaves at least three characters for the pool name.
|
||||
## @maxLength 32
|
||||
|
||||
##
|
||||
@@ -79,13 +79,13 @@ storageClass: replicated
|
||||
version: "v1.35"
|
||||
|
||||
|
||||
## @param {string} host - External hostname for Kubernetes cluster. Defaults to `<cluster-name>.<tenant-host>` if empty.
|
||||
## @param {string} host - External hostname for Kubernetes cluster. Defaults to `kubernetes-<cluster-name>.<tenant-host>` if empty.
|
||||
host: ""
|
||||
|
||||
##
|
||||
## @section Substrate
|
||||
|
||||
## @param {string} substrate="kubevirt" - Which infrastructure provider backs this cluster's worker VMs. `kubevirt` runs them inside this cluster; `proxmox` runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Must match the `substrate` of every KubernetesNodes pool attached to this cluster: the pools reference this cluster's infrastructure object by kind, and a mismatch leaves their Machines unreconciled. Switching an existing cluster is not supported; create a new one.
|
||||
## @param {string} substrate="kubevirt" - Which infrastructure provider backs this cluster's worker VMs. `kubevirt` runs them inside this cluster; `proxmox` runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Every KubernetesNodes pool of this cluster must set the same `substrate`: a pool set to the other one refuses to render once this cluster exists. Switching an existing cluster is not supported; create a new one.
|
||||
## @immutable
|
||||
substrate: kubevirt
|
||||
|
||||
@@ -288,7 +288,7 @@ addons:
|
||||
## @typedef {struct} APIServer - API Server configuration.
|
||||
## @field {Resources} resources - CPU and memory resources for API Server.
|
||||
## @field {ResourcesPreset} resourcesPreset="c1.medium" - Preset if `resources` omitted.
|
||||
## @field {[]string} extraArgs - Extra command-line flags appended to the tenant kube-apiserver, passed through to KamajiControlPlane `spec.apiServer.extraArgs`. For OIDC use `spec.oidc.mode` — this passthrough is the escape hatch for unrelated apiserver flags. The tenant control plane sets each of the following flags itself and drops an entry naming one before the apiserver sees it. The chart moves an `--enable-admission-plugins=` entry into `controlPlane.apiServer.admissionControllers` and a `--kubelet-preferred-address-types=` entry into `controlPlane.kubelet.preferredAddressTypes` while that field is empty, and refuses any other entry naming one of them: `--advertise-address`, `--client-ca-file`, `--enable-admission-plugins`, `--service-cluster-ip-range`, `--kubelet-client-certificate`, `--kubelet-client-key`, `--kubelet-preferred-address-types`, `--proxy-client-cert-file`, `--proxy-client-key-file`, `--requestheader-allowed-names`, `--requestheader-client-ca-file`, `--secure-port`, `--service-account-key-file`, `--service-account-signing-key-file`, `--tls-cert-file`, `--tls-private-key-file`, `--etcd-servers`, `--etcd-cafile`, `--etcd-certfile`, `--etcd-keyfile`, `--etcd-prefix`, `--etcd-compaction-interval`, `--egress-selector-config-file`. When `spec.oidc.mode` is not `None` the chart also renders OIDC flags here, reads what you have written on them, and refuses some combinations outright; the Aggregated API servers section of the Kubernetes OIDC guide on cozystack.io carries which flags, which combinations, and why. Do NOT add legacy `--oidc-*` flags here when `spec.oidc.mode` is not `None`; the chart injects `--authentication-config` and the apiserver refuses to boot with both. Empty by default (no change to current behavior).
|
||||
## @field {[]string} extraArgs - Extra command-line flags appended to the tenant kube-apiserver, passed through to KamajiControlPlane `spec.apiServer.extraArgs`. For OIDC use `spec.oidc.mode` — this passthrough is the escape hatch for unrelated apiserver flags. The tenant control plane sets each of the following flags itself and drops an entry naming one before the apiserver sees it. The chart moves an `--enable-admission-plugins=` entry into `controlPlane.apiServer.admissionControllers` and a `--kubelet-preferred-address-types=` entry into `controlPlane.kubelet.preferredAddressTypes` while that field is empty, and refuses any other entry naming one of them: `--advertise-address`, `--client-ca-file`, `--enable-admission-plugins`, `--service-cluster-ip-range`, `--kubelet-client-certificate`, `--kubelet-client-key`, `--kubelet-preferred-address-types`, `--proxy-client-cert-file`, `--proxy-client-key-file`, `--requestheader-allowed-names`, `--requestheader-client-ca-file`, `--secure-port`, `--service-account-key-file`, `--service-account-signing-key-file`, `--tls-cert-file`, `--tls-private-key-file`, `--etcd-servers`, `--etcd-cafile`, `--etcd-certfile`, `--etcd-keyfile`, `--etcd-prefix`, `--etcd-compaction-interval`, `--egress-selector-config-file`. When `spec.oidc.mode` is not `None` the chart also renders OIDC flags here, reads what you have written on them, and refuses some combinations outright; the Aggregated API servers section of the Kubernetes OIDC guide on cozystack.io carries which flags, which combinations, and why. Do NOT add legacy `--oidc-*` flags here when `spec.oidc.mode` is not `None`; the chart injects `--authentication-config` and the apiserver refuses to boot with both. Empty by default.
|
||||
## @field {[]AdmissionController} admissionControllers - Admission plugins the tenant kube-apiserver enables, passed through to KamajiControlPlane `spec.admissionControllers`, which the control plane renders as `--enable-admission-plugins`. A non-empty list replaces the control plane's default set instead of adding to it. The kube-apiserver enables these plugins in addition to its own default ones, so leaving a plugin out does not turn it off; to turn one off, name it in `--disable-admission-plugins` in `controlPlane.apiServer.extraArgs` and keep it out of this list, since the apiserver refuses to start with a plugin in both, and the control plane's default set names several. Empty by default, which keeps that default set, taken from the TenantControlPlane CRD: CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, LimitRanger, MutatingAdmissionWebhook, NamespaceLifecycle, PersistentVolumeClaimResize, Priority, ResourceQuota, RuntimeClass, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. `NodeRestriction` is not in it.
|
||||
## @field {[]object} extraVolumes - Extra volumes added to the control-plane Deployment, passed through to KamajiControlPlane `spec.deployment.extraVolumes`. Use to mount a ConfigMap or Secret holding an AuthenticationConfiguration file referenced by `extraArgs`. Each item is a core/v1 Volume, but because the control-plane pod runs on the management cluster only `configMap` and `secret` sources are allowed (host-reaching sources like `hostPath`/`csi` and token-projection via `projected` are rejected); each volume must have a unique, non-empty name and exactly one source. The names `talos-ca` and `talos-tls-cert` are reserved by the chart. Empty by default.
|
||||
## @field {[]object} extraVolumeMounts - Extra volume mounts added to the tenant kube-apiserver container, passed through to KamajiControlPlane `spec.apiServer.extraVolumeMounts`. Each `name` must reference a volume declared in `extraVolumes`; the chart-managed talos secret volumes cannot be mounted. Each item is a core/v1 VolumeMount. Empty by default.
|
||||
@@ -366,7 +366,7 @@ controlPlane:
|
||||
## @field {string} [config] - Inline AuthenticationConfiguration YAML (`apiserver.config.k8s.io/v1beta1`). The chart writes the value verbatim into a Secret mounted at `/etc/kubernetes/authentication-config/config.yaml` on the kube-apiserver. Mutually exclusive with `secretRef.name`.
|
||||
## @field {OIDCSecretRef} [secretRef] - Reference to an existing Secret in the tenant namespace carrying the `AuthenticationConfiguration`.
|
||||
|
||||
## @typedef {struct} OIDC - OIDC selector for the tenant cluster (Phase 1 — per-cluster audience binding; no per-tenant Keycloak realm).
|
||||
## @typedef {struct} OIDC - OIDC selector for the tenant cluster. There is no per-tenant Keycloak realm: `System` mode uses the platform realm with a public client and an audience of its own per cluster.
|
||||
## @field {OIDCMode} mode - Identity mode. `None`: no OIDC, only the static admin kubeconfig works. `System`: trust the platform Keycloak realm (`cozy` by default) via a per-cluster public client with audience binding; zero-config default. `CustomConfig`: trust a tenant-supplied issuer directly (BYO); `cozy` is not in the path.
|
||||
## @field {OIDCCustomConfig} [customConfig] - Tenant-supplied AuthenticationConfiguration; consumed only when `mode: CustomConfig`.
|
||||
## @field {[]OIDCUser} [users] - Users granted access to the tenant cluster; each entry produces one ClusterRoleBinding inside the tenant cluster. Works for both `System` and `CustomConfig` modes.
|
||||
|
||||
+1
-1
@@ -32,7 +32,7 @@ spec:
|
||||
singular: kubernetes
|
||||
plural: kuberneteses
|
||||
openAPISchema: |-
|
||||
{"title":"Chart Values","type":"object","x-cozystack-name":{"type":"string","description":"Cluster name. Every worker pool is a separate KubernetesNodes release named `kubernetes-nodes-<name>-<pool>`, so the name must leave room for the default `md0` pool inside the 53-character Helm release name limit.","maxLength":32},"properties":{"storageClass":{"description":"Default StorageClass inside the tenant cluster. Remote-accessible LINSTOR classes are auto-propagated to the tenant under the same name.","type":"string","default":"replicated","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"version":{"description":"Kubernetes major.minor version to deploy","type":"string","default":"v1.35","enum":["v1.35","v1.34","v1.33","v1.32","v1.31"]},"host":{"description":"External hostname for Kubernetes cluster. Defaults to `<cluster-name>.<tenant-host>` if empty.","type":"string","default":""},"substrate":{"description":"Which infrastructure provider backs this cluster's worker VMs. `kubevirt` runs them inside this cluster; `proxmox` runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Must match the `substrate` of every KubernetesNodes pool attached to this cluster: the pools reference this cluster's infrastructure object by kind, and a mismatch leaves their Machines unreconciled. Switching an existing cluster is not supported; create a new one.","type":"string","default":"kubevirt","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"substrate is immutable"}]},"proxmox":{"description":"Proxmox substrate settings.","type":"object","default":{},"required":["ipv4Config"],"properties":{"allowedNodes":{"description":"Proxmox nodes capmox may place VMs on. Empty means every node in the Proxmox cluster.","type":"array","default":[],"items":{"type":"string"}},"ccm":{"description":"Proxmox cloud-controller-manager settings.","type":"object","default":{},"required":["credentialsSecretName"],"properties":{"credentialsSecretName":{"description":"Secret in THIS namespace holding the controller's Proxmox API credentials under the keys `url`, `token_id`, `token_secret` and `region`. May name the same Secret as `csi.credentialsSecretName`; it is a separate knob so the two can hold separate tokens, since this one only reads VM inventory while the driver attaches and detaches disks.","type":"string","default":""}}},"csi":{"description":"Proxmox CSI driver settings.","type":"object","default":{},"required":["credentialsSecretName","enabled"],"properties":{"credentialsSecretName":{"description":"Secret in THIS namespace holding the driver's Proxmox API credentials under the keys `url`, `token_id`, `token_secret` and `region`. The parent chart composes them into the controller's config file here, in the management cluster; nothing reaches the tenant, and no token is written into a rendered manifest. Distinct from the capmox credentials on purpose: the driver attaches and detaches disks, which is a different blast radius from creating VMs.","type":"string","default":""},"enabled":{"description":"Install the driver. Without it a Proxmox-backed tenant has no way to provision PersistentVolumes at all.","type":"boolean","default":true},"storageClasses":{"description":"StorageClasses to create in the tenant.","type":"array","default":[],"items":{"type":"object","required":["name","storage"],"properties":{"fstype":{"description":"Filesystem the driver formats volumes with: ext4 or xfs.","type":"string"},"name":{"description":"StorageClass name inside the tenant cluster.","type":"string"},"reclaimPolicy":{"description":"Delete or Retain.","type":"string"},"ssd":{"description":"Mark volumes as SSD-backed, which changes the discard/cache defaults Proxmox applies.","type":"boolean"},"storage":{"description":"Proxmox storage id the volumes are created on (as it appears in `pvesm status`).","type":"string"}}}}}},"dnsServers":{"description":"Nameservers written into each worker's network config. Required by the ProxmoxCluster schema.","type":"array","default":[],"items":{"type":"string"}},"insecure":{"description":"Skip verification of the Proxmox API server certificate in the CCM and the CSI controller, both of which run in the management cluster. Default false: the certificate is verified unless an operator turns that off. A stock Proxmox VE install serves a self-signed one, so set true for it, or install a certificate the management cluster trusts.","type":"boolean","default":false},"ipv4Config":{"description":"Address pool for workers. capmox assigns static addresses and has no DHCP mode, so this is required rather than optional.","type":"object","default":{},"required":["gateway","prefix"],"properties":{"addresses":{"description":"Ranges or CIDRs, e.g. `[\"10.0.0.120-10.0.0.170\"]`. capmox turns these into an InClusterIPPool, so they must not ovLine truncated
|
||||
{"title":"Chart Values","type":"object","x-cozystack-name":{"type":"string","description":"Cluster name. Each worker pool is a KubernetesNodes release named `kubernetes-nodes-<name>-<pool>`, which has to fit the 53-character Helm release name limit, so a cluster name of at most 32 characters leaves at least three characters for the pool name.","maxLength":32},"properties":{"storageClass":{"description":"Default StorageClass inside the tenant cluster. Remote-accessible LINSTOR classes are auto-propagated to the tenant under the same name.","type":"string","default":"replicated","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"version":{"description":"Kubernetes major.minor version to deploy","type":"string","default":"v1.35","enum":["v1.35","v1.34","v1.33","v1.32","v1.31"]},"host":{"description":"External hostname for Kubernetes cluster. Defaults to `kubernetes-<cluster-name>.<tenant-host>` if empty.","type":"string","default":""},"substrate":{"description":"Which infrastructure provider backs this cluster's worker VMs. `kubevirt` runs them inside this cluster; `proxmox` runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Every KubernetesNodes pool of this cluster must set the same `substrate`: a pool set to the other one refuses to render once this cluster exists. Switching an existing cluster is not supported; create a new one.","type":"string","default":"kubevirt","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"substrate is immutable"}]},"proxmox":{"description":"Proxmox substrate settings.","type":"object","default":{},"required":["ipv4Config"],"properties":{"allowedNodes":{"description":"Proxmox nodes capmox may place VMs on. Empty means every node in the Proxmox cluster.","type":"array","default":[],"items":{"type":"string"}},"ccm":{"description":"Proxmox cloud-controller-manager settings.","type":"object","default":{},"required":["credentialsSecretName"],"properties":{"credentialsSecretName":{"description":"Secret in THIS namespace holding the controller's Proxmox API credentials under the keys `url`, `token_id`, `token_secret` and `region`. May name the same Secret as `csi.credentialsSecretName`; it is a separate knob so the two can hold separate tokens, since this one only reads VM inventory while the driver attaches and detaches disks.","type":"string","default":""}}},"csi":{"description":"Proxmox CSI driver settings.","type":"object","default":{},"required":["credentialsSecretName","enabled"],"properties":{"credentialsSecretName":{"description":"Secret in THIS namespace holding the driver's Proxmox API credentials under the keys `url`, `token_id`, `token_secret` and `region`. The parent chart composes them into the controller's config file here, in the management cluster; nothing reaches the tenant, and no token is written into a rendered manifest. Distinct from the capmox credentials on purpose: the driver attaches and detaches disks, which is a different blast radius from creating VMs.","type":"string","default":""},"enabled":{"description":"Install the driver. Without it a Proxmox-backed tenant has no way to provision PersistentVolumes at all.","type":"boolean","default":true},"storageClasses":{"description":"StorageClasses to create in the tenant.","type":"array","default":[],"items":{"type":"object","required":["name","storage"],"properties":{"fstype":{"description":"Filesystem the driver formats volumes with: ext4 or xfs.","type":"string"},"name":{"description":"StorageClass name inside the tenant cluster.","type":"string"},"reclaimPolicy":{"description":"Delete or Retain.","type":"string"},"ssd":{"description":"Mark volumes as SSD-backed, which changes the discard/cache defaults Proxmox applies.","type":"boolean"},"storage":{"description":"Proxmox storage id the volumes are created on (as it appears in `pvesm status`).","type":"string"}}}}}},"dnsServers":{"description":"Nameservers written into each worker's network config. Required by the ProxmoxCluster schema.","type":"array","default":[],"items":{"type":"string"}},"insecure":{"description":"Skip verification of the Proxmox API server certificate in the CCM and the CSI controller, both of which run in the management cluster. Default false: the certificate is verified unless an operator turns that off. A stock Proxmox VE install serves a self-signed one, so set true for it, or install a certificate the management cluster trusts.","type":"boolean","default":false},"ipv4Config":{"description":"Address pool for workers. capmox assigns static addresses and has no DHCP mode, so this is required rather than optional.","type":"object","default":{},"required":["gateway","prefix"],"properties":{"addresses":{"description":"Ranges or CIDRs, e.g. `[\"10.0.0.120-10.0.0.170\"]`. capmox turns these into an InClusterIPPool, so they must not overlLine truncated
|
||||
release:
|
||||
prefix: kubernetes-
|
||||
labels:
|
||||
|
||||
@@ -36,7 +36,7 @@ func TestNameCapComesFromTheShippedSchema(t *testing.T) {
|
||||
cap int
|
||||
explains string
|
||||
}{
|
||||
{"Kubernetes", "kubernetes-", "../../../../packages/system/kubernetes-rd/cozyrds/kubernetes.yaml", 32, "md0"},
|
||||
{"Kubernetes", "kubernetes-", "../../../../packages/system/kubernetes-rd/cozyrds/kubernetes.yaml", 32, "kubernetes-nodes-<name>-<pool>"},
|
||||
{"Kafka", "kafka-", "../../../../packages/system/kafka-rd/cozyrds/kafka.yaml", 44, "KRaft"},
|
||||
} {
|
||||
t.Run(tc.kind, func(t *testing.T) {
|
||||
|
||||
Reference in new issue
Block a user