Files
cozystack/packages/core/platform/sources/kubernetes-application.yaml
T
Marian Koreniuk 93c89c8bf6 feat(kubernetes): add the Proxmox CSI driver, with its controller in this cluster
Tenant clusters on the proxmox substrate get persistent volumes backed by the
hypervisor's storage. The driver is sergelogvinov/proxmox-csi-plugin, vendored
as a chart that reproduces byte-for-byte from `helm pull`.

It is deployed in halves, and which half goes where is the whole design. The
controller authenticates to Proxmox, so it runs HERE, in the tenant's namespace
of the management cluster, and drives the tenant over its admin kubeconfig —
the same shape templates/csi/deploy.yaml already uses for kubevirt-csi. The
tenant gets the node plugin, the CSIDriver object and the storage classes; its
DaemonSet mounts no credentials.

Putting the controller in the tenant would hand the token to the tenant. The
resource definition lists the admin kubeconfig in secrets.include, so the
tenant is cluster-admin of its own cluster and can read any Secret there, and
the chart writes the driver's config.yaml — token included — wherever it is
installed. The privileges the driver documents are granted on `/`, so that
token reaches every VM and every volume on the Proxmox cluster, not just this
tenant's.

Three consequences of the split are handled here rather than discovered later:

  - The vendored chart has no switch for its controller, so the tenant release
    sets replicaCount: 0. The Deployment object still renders; no Pod does.
  - Its Secret template is gated on a non-empty config.clusters, so an empty
    list is what keeps the credential out.
  - CSIStorageCapacity is published by the provisioner sidecar, which now runs
    here and owns those objects through a Pod that does not exist in the
    tenant. Left on, CSIDriver.storageCapacity=true and the scheduler refuses
    every Pod with "node(s) did not have enough free storage". The chart
    hardcodes that field, so a kustomize postRenderer patches it rather than
    forking the chart.

The controller follows proxmox.csi.enabled, like the tenant release: with the
driver switched off there is nothing for it to serve and no token to hold.

Volumes are owned per tenant. The driver names them vm-<controllerVmID>-pvc-…
and defaults the id to 9999; on shared storage that made every tenant's volumes
indistinguishable to an operator and to a VM-level ACL. Each cluster derives its
own from its release name.

What this does not do is make one tenant unable to reach another's volumes: a
tenant can still create a static PV naming a foreign volumeHandle. That needs a
per-tenant Proxmox scope — a dedicated storage and a VM ACL per tenant — which
is a deployment requirement rather than a chart change.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-29 21:04:05 +02:00

80 lines
3.0 KiB
YAML

---
apiVersion: cozystack.io/v1alpha1
kind: PackageSource
metadata:
name: cozystack.kubernetes-application
spec:
sourceRef:
kind: OCIRepository
name: cozystack-packages
namespace: cozy-system
path: /
variants:
- name: kubevirt
dependsOn:
- cozystack.networking
- cozystack.capi-operator
- cozystack.capi-provider-bootstrap-kubeadm
- cozystack.capi-provider-core
- cozystack.capi-provider-cp-kamaji
- cozystack.capi-provider-infra-kubevirt
# cozystack-engine ships the ApplicationDefinition CRD rendered by the *-rd component below.
- cozystack.cozystack-engine
libraries:
- name: cozy-lib
path: library/cozy-lib
components:
- name: kubernetes-ingress-nginx
path: system/ingress-nginx
- name: kubernetes-cert-manager-crds
path: system/cert-manager-crds
- name: kubernetes-volumesnapshot-crd
path: system/vsnap-crd
- name: kubernetes-velero
path: system/velero
- name: kubernetes-gateway-api-crds
path: system/gateway-api-crds
- name: kubernetes-victoria-metrics-operator
path: system/victoria-metrics-operator
- name: kubernetes-kubevirt-csi-node
path: system/kubevirt-csi-node
# The node half of the storage driver for the proxmox substrate: the
# DaemonSet, the CSIDriver object and the storage classes. Published
# alongside the KubeVirt one rather than in a variant of its own,
# because a second variant would rename every tenant addon artifact.
# The controller half and the cloud-controller-manager are not here:
# they hold the hypervisor token, so they run in the management cluster
# as plain Deployments rendered by the kubernetes chart itself.
- name: kubernetes-proxmox-csi
path: system/proxmox-csi
- name: kubernetes-vertical-pod-autoscaler-crds
path: system/vertical-pod-autoscaler-crds
- name: kubernetes-coredns
path: system/coredns
- name: kubernetes-cert-manager
path: system/cert-manager
- name: kubernetes-monitoring-agents
path: system/monitoring-agents
- name: kubernetes-cilium
path: system/cilium
- name: kubernetes-gpu-operator
path: system/gpu-operator
- name: kubernetes-hami
path: system/hami
- name: kubernetes-vertical-pod-autoscaler
path: system/vertical-pod-autoscaler
- name: kubernetes-prometheus-operator-crds
path: system/prometheus-operator-crds
- name: kubernetes-metrics-server
path: system/metrics-server
- name: kubernetes-ouroboros
path: system/ouroboros
- name: kubernetes
path: apps/kubernetes
libraries: ["cozy-lib"]
- name: kubernetes-rd
path: system/kubernetes-rd
install:
namespace: cozy-system
releaseName: kubernetes-rd