mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-08 21:14:56 +09:00
Aleksei Sviridkin
6580d2eeb0
## What this PR does **Series behind #3481** — three separable pull requests: 1. #4087 — the Cluster API providers (capmox + in-cluster IPAM) 2. #4088 — the `substrate` switch in `kubernetes` and `kubernetes-nodes` 3. #4089 — the Proxmox CSI driver and cloud-controller-manager #4088 has merged, so this diff is the eight commits on top of it: `proxmox.pool`, `csi-proxmox`, `ccm-proxmox`, `delete.yaml`, the tests, the e2e suite, one README sentence on token privileges, and the reconcile Job's replace-on-immutable step for existing clusters. The `substrate` value this gates on is on `main`. ### One chain, not two components Storage and the cloud-controller-manager are one chain, not two components: ``` kubelet --cloud-provider=external → taint node.cloudprovider.kubernetes.io/uninitialized → CCM applies topology.kubernetes.io/region|zone + providerID → CSI node plugin starts → PVC binds ``` The CSI node plugin reads region and zone off its own Node and exits fatally without them: ``` F main.go:126] Failed to get region or zone for node: …, region: , zone: , see documentation about topology labels ``` Those labels come from the cloud-controller-manager, and the CCM ignores a Node until the kubelet's `--cloud-provider=external` has tainted it. Shipping the CSI driver alone gives a tenant a StorageClass, a healthy controller, and a node plugin in CrashLoopBackOff. The CSI driver has no equivalent knob for this. Its `features.provider` accepts `default` and `capmox` and validates neither, so an unrecognised value falls through to the default path — the key is therefore not set at all. The cloud-controller-manager is the opposite case: `capmox` is a real mode there and is set, because the default one writes a Node providerID of `proxmox://<node>/<vmid>` while capmox has already written `proxmox://<SMBIOS UUID>` on the Machine. So the CCM has no `enabled` switch and its credentials reference is required: on this substrate, a cluster without a CCM is a cluster where every worker keeps the `uninitialized` taint and nothing schedules. A chart that cannot name the Secret should stop rather than render that. The CSI release depends on cilium only; the CCM is a Deployment of this chart, not a HelmRelease, so nothing in the tenant can depend on it — the node plugin waits on the labels the CCM applies, which is the ordering that matters. ### Packaging | Package | Chart | App | |---|---|---| | `packages/system/proxmox-csi` | proxmox-csi-plugin 0.5.12 | v0.20.0 | The vendored chart is the tenant-side half only. It **drops the upstream `values.talos.yaml` preset**, which pins the workloads onto `node-role.kubernetes.io/control-plane` and `node.cloudprovider.kubernetes.io/platform: nocloud`: a Kamaji tenant has no control-plane nodes at all — its control plane is a Deployment in the management cluster — so that preset leaves the components unschedulable. It is published as a component of the existing `kubevirt` variant of `kubernetes-application` rather than in a new variant: a second variant would rename the artifacts of all eighteen tenant addons, and exactly one component differs between the substrates. The cloud-controller-manager is not a vendored chart. It is a Deployment the `kubernetes` chart renders itself (`templates/ccm-proxmox/controller.yaml`), because it runs here, in the management cluster, and there is nothing of it to install in the tenant. Credentials never reach the tenant. Each controller names a Secret in the tenant's namespace of this cluster (`proxmox.csi.credentialsSecretName`, `proxmox.ccm.credentialsSecretName`), an init container composes the config file from it into a memory-backed emptyDir at start-up, and the `kubernetes` release stores only the Secret's name. They are separate knobs from the capmox credentials, and from each other, so an operator can scope them: attaching disks is a different blast radius from creating VMs, and the CCM only reads inventory. `proxmox.insecure` defaults to `false`; a stand whose Proxmox serves a self-signed certificate sets it to `true`. `proxmox.csi.enabled: false` switches off the controller as well as the tenant release, and needs no CSI token. ### Where the hypervisor token lives, and why it decides the shape Both controllers run in the tenant's namespace of **this** cluster and drive the tenant over its admin kubeconfig. Only the CSI node plugin goes into the tenant, and it mounts no credentials. Installing them into the tenant would hand the token to the tenant. The resource definition lists the admin kubeconfig in `secrets.include`, so a tenant is cluster-admin of its own cluster, and the vendored chart writes the driver's `config.yaml` — token included — wherever it is installed. The privileges the driver documents are granted on `/`, so such a token reaches every VM and every volume on the Proxmox cluster. The KubeVirt path already has the shape that works, in `templates/csi/deploy.yaml`, and the upstream components support it: the CSI controller's `--kubeconfig` is documented for running out of cluster. Three consequences of the split are handled rather than left to be discovered: the vendored chart has no switch for its controller, so the tenant release sets `replicaCount: 0`; its Secret template is gated on a non-empty `config.clusters`, so an empty list keeps the credential out; and `CSIStorageCapacity` is published by the provisioner sidecar, which now owns those objects through a Pod that does not exist in the tenant, so `CSIDriver.storageCapacity` is patched to false by a kustomize postRenderer — the chart hardcodes it, and left true the scheduler refuses every Pod with "node(s) did not have enough free storage". Volumes are owned per cluster. The driver names them `vm-<controllerVmID>-pvc-…` and owns every volume as VMID 9999 unless `controllerVmID` is set, which leaves different clusters' volumes indistinguishable on shared storage. Each cluster sets its own, derived from its tenant namespace and release name, so two tenants' clusters with the same name still get different owners. **This does not make one tenant unable to reach another's volumes.** A tenant can still create a static PV naming a foreign `volumeHandle`, and the controller will attach it, detach it or delete it: the driver checks neither the VMID a node ID names nor the volume a handle names against the cluster it serves. Closing that needs a per-tenant Proxmox scope, and the charts can now produce one: `proxmox.pool` on the `kubernetes-nodes` chart puts every worker VM into a Proxmox resource pool as capmox clones it, replacements included, so one ACL on `/pool/<name>` follows every VM without being granted again. The README states the scope where an operator configuring the chart reads: a storage of the tenant's own (`Datastore.Allocate` on a storage reaches every volume on it), `VM.Audit`/`VM.Config.Disk` on that pool for the CSI controller, `VM.Audit` and `VM.GuestAgent.Audit` on it for the CCM — a NotReady Node whose VM the token cannot see is reported as gone and deleted — and what that scope still leaves open: the tenant's own disks, which a cluster-admin of the tenant could reach anyway. ### Deleting a cluster frees its disks The pre-delete hook deletes the CSI driver in Step 2, and a volume is freed only by the driver answering DeleteVolume, so a Step 1b runs first: it deletes the tenant's claims (`--wait=false`; a claim in use carries `kubernetes.io/pvc-protection`, so it clears once nothing mounts it), then the Pods holding each claim, then waits for the PVs rather than the PVCs. The Pods are found by listing every Pod with the claims it mounts and matching in the shell: a JSONPath filter on the claim name fails for a whole namespace as soon as one Pod there mounts two claims, and would find no Pod at all. The claims go first because a claim being deleted is one the scheduler places no new Pod on, so a Pod that a StatefulSet brings back cannot mount it again. Without the step the disks stayed on the hypervisor with the tenant's data in them and no PV object left to name their owner. A tenant apiserver that does not answer is told apart from a tenant with no claims: the first goes through the hook's backstop and shows in its summary, the second is the clean case. The probe is bounded with `timeout`, not `--request-timeout`, which the repo's hook check refuses because a non-zero value diverts an in-cluster kubectl call to localhost:8080. On the KubeVirt path two things change. The pre-delete hook's Step 5 checks that the DataVolume CRD exists before deleting DataVolumes, so a hook that runs where CDI is not installed no longer fails after every earlier step has run; the hook runs only at deletion. The worker pool's talos-reconcile Job changes at upgrade: its Role's TalosConfigTemplate rights are narrowed to the pool's own template and gain a named `delete`, and its script gains the replace step, which moves the Job's content-hash name. Every existing worker pool therefore gets a new reconcile Job when it upgrades. Where a pool's live TalosConfigTemplate still matches the render, that Job changes nothing; where it has drifted, the Job now deletes and recreates it instead of failing. Machines already running keep the bootstrap data they joined with; Machines created afterwards boot from the current template. ### Existing clusters: the TalosConfigTemplate CABPT will not let us patch CABPT's validating webhook rejects any change to `TalosConfigTemplate.spec` (`TalosConfigTemplate.Spec is immutable`). The kubelet flag this PR adds changes that spec, so on a cluster created before it the reconcile Job's `kubectl apply` fails with `is immutable` on every run, and a pool whose `ProxmoxMachineTemplate` also changed (say, `proxmox.pool` set) rolls its workers off the old template: they join without `--cloud-provider=external`, the CCM leaves them unmanaged, and the CSI node plugin exits on the missing zone label. The last commit makes the Job replace the object when apply reports the spec immutable: delete, apply again, log both. Existing Machines keep their bootstrap data; new ones are built from the replaced template. Pinned by `talos_tct_immutable_test.yaml`, and the RBAC the Job runs with covers exactly that delete (`resourceNames` of its own template). ### Evidence ``` $ KUBECONFIG=<tenant> kubectl get node -o custom-columns=… kubernetes-px3-md0-rp4s5-rgr48 prox-home prox-home proxmox://b903f905-7b4f-4048-b1b6-3e8f58530d8f $ kubectl -n tenant-proxmox3 get deploy kubernetes-px3-pcsi-controller 1/1 kubernetes-px3-pccm 1/1 $ KUBECONFIG=<tenant> kubectl get pods -n csi-proxmox csi-plugin-node 3/3 Running (the controller Deployment has 0 replicas) $ KUBECONFIG=<tenant> kubectl get secret -A | grep -c token_secret 0 $ KUBECONFIG=<tenant> kubectl get pvc csi-probe csi-probe Bound pvc-… 1Gi RWO proxmox ``` That the volume is real, not merely `Bound` in the API, and that it is this tenant's: ``` $ qm config <vmid> scsi1: main-pool:vm-798497-pvc-…,size=1G ``` Deleting the PVC frees the disk from the storage; the e2e probe checks that too. ### Known limitation The CCM logs every five minutes on PVE 9: ``` error get ha-groups 500 cannot index groups: ha groups have been migrated to rules ``` Proxmox 9 changed the HA-groups API. Node initialization is unaffected — labels and providerID are applied — but the CCM image v0.15.0's HA-groups lookup is not compatible with PVE 9. ### Testing `make unit-tests` — exit 0. `apps/kubernetes` 28 suites / 338 tests; `apps/kubernetes-nodes` 33 suites / 182 tests / 12 snapshots. Each of the eight commits passes both suites on its own. What the new asserts pin, each shown to go red on the mutation it exists for: the tenant release carries no credentials and no `valuesFrom`; the controllers run here with the Secret mounted and the tenant as their target; `csi.enabled: false` renders no CSI controller and needs no CSI token; the kubelet runs with `cloud-provider: external` on proxmox and without it on kubevirt; Step 1b and the cleanup Role's admin-kubeconfig entry render on proxmox and not on kubevirt, and Step 1b deletes the claims before the Pods; `proxmox.pool` reaches the `ProxmoxMachineTemplate` and an empty one renders no `pool`; each tenant gets its own volume owner id; the CCM runs in capmox mode; the CCM tolerates exactly two taints. `proxmox.ccm`, `proxmox.csi` and `proxmox.insecure` are optional in the schema — each has a default, and the render enforces what the substrate needs — so a `Kubernetes` object written against `main`'s `proxmox` block stays valid; the API review gate reports no sizeable change at any of the commits. The e2e suite ships `.disabled` and no CI lane runs it, so the substrate has no automated coverage until a runner with the `proxmox` label exists. What is in the tree is runnable: the fixture is rendered by `run-proxmox.sh` with explicit defaults (GNU `envsubst` does not expand `${VAR:-default}`, so the defaults live in the script and the render refuses a fixture with a variable left in it — pinned by `hack/proxmox-e2e-fixture_test.bats`), the runner reads the `.conf` kubeconfig a runner outside the management cluster can reach, the StorageClass and the PVC probe read the same storage variable, the probe checks the disk is gone after the claim is deleted, `keep_tenant` is honoured by the suite's own teardown, the workflow fails on an unset stand variable instead of falling back to a hardcoded address, and its schedule runs only where `vars.PROXMOX_NIGHTLY` is set, since the job needs a self-hosted runner with the `proxmox` label. The fixture takes the stand's resource pool through `COZY_PVE_POOL`. ### Screenshots Not applicable — no UI change. ### Downstream repositories I walked the trigger map in `docs/agents/contributing.md` against this diff, file by file. What it matches is below. **No box is ticked**, because every follow-up it points at depends on a decision that is not mine to make: #3481 asks whether Proxmox becomes a supported platform at all, and writing its documentation or its Terraform schema before that is answered would be work built on an assumption. I have not filed speculative PRs or issues in those repositories for the same reason. If the direction is accepted, I open them; a human should decide which. **cozystack/terraform-provider-cozystack — matches, no follow-up yet.** - `packages/apps/kubernetes/values.schema.json` gains `proxmox.ccm.credentialsSecretName`, `proxmox.csi` (`enabled`, `credentialsSecretName`, `storageClasses[]`) and `proxmox.insecure`, with defaults in `values.yaml`. Same coupling as the substrate PR: hand-written schema, model and expand/flatten, and defaults the provider restates and sends itself. **cozystack/website — unclear, deliberately left unticked.** This PR adds one package under `packages/system/`, not under `packages/apps/` or `packages/extra/`, so the app-list trigger does not fire. Whether `proxmox-csi` counts as a "platform component" for `guides/platform-stack/_index.md` is a judgement about that page's scope: it is not part of the platform stack — it is an addon installed *into* a tenant cluster, like the KubeVirt CSI driver, which has no entry there either. I read that as no, but I am not confident enough to decide it for you, so the box stays empty. The README section on token privileges and per-tenant scope is the kind of text `operations/` pages carry; that follows the decision #3481 asks for. **Everything else: no match.** No app package added or renamed; no version enum, `kind`, `plural` or `release.prefix` change; `packages/core/platform/values.yaml` untouched; `packages/core/platform/sources/kubernetes-application.yaml` gains one component but is not a renamed `*-rd` file (ccp); `hack/` gains `e2e-chainsaw/_lib/run-proxmox.sh`, a suite directory shipped `.disabled`, and `proxmox-e2e-fixture_test.bats`, which runs with the other bats files; no `ApplicationDefinition` semantics or `release.chartRef.kind` enum change (external-apps-example); no node prerequisites; no telemetry metric; no commit-convention change. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(kubernetes): add the Proxmox CSI driver and cloud-controller-manager for tenant clusters on the `proxmox` substrate, giving them PersistentVolumes backed by Proxmox storage. The cloud-controller-manager is not optional there: worker kubelets run with `--cloud-provider=external`, so without it every node keeps the `uninitialized` taint and the CSI node plugin cannot start. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## New Features - Added Proxmox VE as an infrastructure option alongside KubeVirt for Kubernetes clusters and worker pools. - Added configuration for Proxmox networking, DNS, VM templates, storage, node placement, credentials, IPv4 allocation, and storage classes. - Added Proxmox cloud-controller-manager and CSI integrations. - Added validation for required Proxmox settings and unsupported combinations. ## Bug Fixes - Improved recovery when Talos configuration templates cannot be updated because they are immutable. ## Documentation - Documented Proxmox configuration, requirements, and supported parameters. ## Tests - Added coverage for Proxmox rendering, sizing, integrations, validation, and KubeVirt compatibility. <!-- end of auto-generated comment: release notes by coderabbit.ai --> ### Verified on a live stand Run against three throwaway tenants on Proxmox VE 9.2, each built and destroyed: - every Secret in the tenant cluster scanned for the token — none found, while the PVC round-trip works; - a PVC bound, mounted, written and read back, its disk visible on the hypervisor as `vm-798497-pvc-…` rather than `vm-9999-…`; - `helm uninstall` with the volume still mounted by a running Pod, after which the disk is gone from the storage with nothing done by hand. What that run pinned: the CCM's capmox mode, capacity-aware scheduling off with `CSIDriver.storageCapacity` patched, and the teardown Role's admin-kubeconfig entry. A second run, on the head of this PR, used a pool per tenant and two privilege-separated tokens holding exactly the README's rows, on a throwaway tenant and then on the stand's long-lived cluster upgraded in place: - the worker VM lands in the pool as capmox clones it, and so does its replacement after the Machine is deleted — the CCM initialises the new node and the CSI attaches a disk to it with no ACL touched; - no `401` or `403` from the hypervisor in the CSI controller or the CCM over the whole run, once the roles were granted to the user as well as the token; - the PVC round-trip on both clusters, the disk gone from the storage after the claim; `helm uninstall` with a claim held by a running Pod deletes the claim before the Pod and waits for the volume to be released; - the in-place upgrade replaced the immutable template on the first apply conflict and the re-created worker came up with `--cloud-provider=external`.
Cozystack
Cozystack is a free and open-source platform and framework for building clouds.
Cozystack is a CNCF Sandbox project that was originally built and sponsored by Ænix.
With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.
Use Cozystack to build your own cloud or provide a cost-effective development environment.
Use-Cases
-
Using Cozystack to build a public cloud
You can use Cozystack as a backend for a public cloud -
Using Cozystack to build a private cloud
You can use Cozystack as a platform to build a private cloud powered by Infrastructure-as-Code approach -
Using Cozystack as a Kubernetes distribution
You can use Cozystack as a Kubernetes distribution for Bare Metal
Documentation
The documentation is located on the cozystack.io website.
Read the Getting Started section for a quick start.
If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.
Versioning
Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.
Contributions
Contributions are highly appreciated and very welcomed!
In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.
You are welcome to take on the fix yourself. Commits are used to generate the changelog, and their author will be referenced in it.
If you have a concrete feature request, open an issue. For a cross-cutting or architectural change, open a design proposal in the community repo. For questions or early ideas, use GitHub Discussions. See CONTRIBUTING.md for the full routing.
Community
You are welcome to join us in #cozystack, on either CNCF Slack (get an invite) or Kubernetes Slack (get an invite), in our Telegram group, and at our community meetings. Add them to your Google Calendar or iCal for convenience.
Project documents
| Document | What it covers |
|---|---|
| Governance | How decisions are made and who makes them |
| Contributor Ladder | How to grow from contributor to maintainer |
| Maintainers | Who maintains the project |
| Code of Conduct | What we expect of each other |
| Security Policy | Reporting a vulnerability, and which releases get fixes |
| Adopters | Who runs Cozystack in production |
| Roadmap | Where the project is going |
License
Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.
Commercial Support
A list of companies providing commercial support for this project can be found on official site.