mirror of
https://github.com/cozystack/cozystack.git
synced 2026-10-30 08:37:31 +09:00
Aleksei Sviridkin
4a2dea2d08
## What this PR does **Series behind #3481** — three separable pull requests: 1. **#4087 — Cluster API providers** (capmox + in-cluster IPAM) 2. #4088 — the `substrate` switch in `kubernetes` and `kubernetes-nodes` 3. #4089 — the Proxmox CSI driver and cloud-controller-manager #4087 and #4088 share no files and can be reviewed in parallel. #4089 builds on #4088. --- This is the provisioning layer #3481 asks to specify first: the part that creates the VMs. Neither of the other two is needed to review it. ### What it adds Two Cluster API providers, packaged the way the existing KubeVirt provider is — vendored components, `gzip -9 -n` for a reproducible artifact, a `make components` target that regenerates it from the upstream release and checks the downloads against the SHA-256 sums the Makefile pins: | Package | Version | Contract | |---|---|---| | `capi-providers-infraprovider-proxmox` | capmox v0.7.7 | v1beta1 | | `capi-providers-ipam-in-cluster` | CAIP v1.0.3 | v1beta1 | capmox v0.7.7 rather than the current v0.9.x on purpose: from v0.8 it declares contract `v1beta2`, which the CAPI core shipped here (v1.10.1) does not serve. Both join the `iaas` bundle as **opt-in** packages, where the KubeVirt infrastructure provider is unconditional. The asymmetry is deliberate: capmox drives a hypervisor outside the cluster and needs a Secret holding a Proxmox API token, which capi-operator reads before it installs anything — with no Secret the provider is never deployed at all. Enabling it by default would ask every Cozystack install that has no Proxmox to supply hypervisor credentials. ```yaml bundles: enabledPackages: - cozystack.capi-provider-infra-proxmox # brings IPAM with it - cozystack.capi-provider-ipam-in-cluster # also enableable on its own ``` Enabling capmox emits the IPAM provider automatically — capmox allocates worker addresses from an `InClusterIPPool` and has no DHCP mode, so on its own it never gets a `ProxmoxCluster` to `Ready`. IPAM stays independently enableable, since assigning addresses is a generic Cluster API concern. Naming capmox in `enabledPackages` while disabling IPAM stops the render. The cluster would catch that combination on its own — the capmox `PackageSource` depends on the IPAM `Package`, so with that `Package` absent capmox goes `Ready=False` with `DependenciesNotReady` and no `HelmRelease` is created — but it would not say why. Failing the render names the two conflicting list entries at the moment someone writes them, instead of leaving a condition to decode. ### Why the earlier attempt did not produce a VM #1927's blocker — "ProxmoxMachine does not produce a VM" — was not a capmox incompatibility. It was four independent causes, each of which only became visible once the previous one was cleared: 1. **`PROXMOX_URL` included `/api2/json`.** capmox appends that itself, so every call went to `/api2/json/api2/json/…` and returned 501. 2. **No IPAM provider.** `no matches for kind "InClusterIPPool"`; the `ProxmoxCluster` never reached `Ready` and every `ProxmoxMachine` sat in `WaitingForClusterInfrastructure`. 3. **`templateSelector.matchTags` is set equality, not containment** (`slices.Equal` after sort). A template carrying four tags is not selected by naming one of them: `found 0 VM templates with tags`. 4. **capmox waits for the qemu-guest-agent, which Talos does not ship.** `error waiting for agent: timed out` — cleared with `checks.skipQemuGuestAgent` and `skipCloudInitStatus`. None of these are in the provider. They are packaging and configuration, which is why the earlier work could look complete and still never boot a machine. ### Evidence Run on Proxmox VE 9.2.11 with Talos v1.13.6 and Cozystack's CAPI stack (core v1.10.1, CABPT v0.6.12, Kamaji control-plane provider v0.19.0): ``` $ kubectl get proxmoxmachine,machine -n tenant-proxmox proxmoxmachine/kubernetes-px-md0-… Ready=True VMProvisioned=True machine/kubernetes-px-md0-… Running proxmox://0ef88cc1-6aea-4563-af64-344d8060d5c0 $ qm list 106 kubernetes-px-md0-jmzds-h9npx running ``` The machine boots Talos from the nocloud ISO, reads the machineconfig capmox writes, and joins. That is the layer #69 called a hard blocker. ### What this PR does not claim - One Proxmox node, one tenant cluster, manual runs — there is no e2e coverage for Proxmox in CI, and this PR does not add any. - It does not make Proxmox a supported platform. It makes the provisioning layer reviewable, which #3481 names as the precondition for that conversation. - It does not touch `capi-providers-core`. The same drift check, pointed at that package, shows that the `startupProbe` from #2946 never reached the shipped blob; that fix changes what every new install runs and is #4483, with its own release note. - Nothing here exposes `ProxmoxCluster.spec.credentialsRef` to a tenant, and nothing should: `capmox-manager-role` grants get/list/watch/patch on Secrets cluster-wide and that field takes a namespace. Whatever renders a `ProxmoxCluster` for a tenant in the rest of the series has to keep the field out of tenant hands. ### Testing `make unit-tests` — exit 0. New suites: `packages/core/platform/tests/sources_capi_provider_proxmox_test.yaml` and `bundles_proxmox_wiring_test.yaml` (both bundle paths, the enable/disable precedence, and the refusal above). Every negated `containsDocument` in the wiring suite carries `any: true`; without it the negation passes whether or not the document is in the render. Three mutations, each reverted afterwards — emitting the capmox `Package` unconditionally, emitting the IPAM `Package` unconditionally, and dropping the `disabledPackages` check from `$proxmoxEnabled` — each turn red the tests named after the property they break, and the unmutated chart stays green (44 suites, 198 tests). `hack/capi-provider-components_test.bats` covers both new provider packages: the blob and the YAML are the same bytes; the Makefile version, the ConfigMap name and `spec.version` are one value — `applied-spec-hash` covers the provider spec and the config Secret but not the ConfigMap, so a bump that regenerates the files while `spec.version` stays behind installs the old provider silently; and the vendored files match the SHA-256 sums the Makefile pins, so the pin holds even when nobody runs `make components`. The render was compared against `main` for five configurations (defaults, `isp-full`, `isp-full-generic` with paas, an unrelated `enabledPackages`, and `disabledPackages` naming IPAM). No line is removed in any of them and the `Package` count is unchanged; the only additions are the two `PackageSource` objects, which the chart renders for every package, optional or not. ### Screenshots Not applicable — no UI change. ### Downstream repositories I walked the trigger map in `docs/agents/contributing.md` against this diff, file by file. What it matches is below. **No box is ticked**, because every follow-up it points at depends on a decision that is not mine to make: #3481 asks whether Proxmox becomes a supported platform at all, and writing its documentation or its Terraform schema before that is answered would be work built on an assumption. I have not filed speculative PRs or issues in those repositories for the same reason. If the direction is accepted, I open them; a human should decide which. **cozystack/website — matches, no follow-up yet.** - `packages/core/platform/values.yaml` changed. `content/en/docs/next/operations/configuration/platform-package.md` is a hand-written table of the `spec.components.platform.values.*` keys; this PR does not add a key, but it documents two new values that `bundles.enabledPackages` now accepts, and the table is where a reader would look for them. - Two platform components are added (`capi-providers-infraprovider-proxmox`, `capi-providers-ipam-in-cluster`) → `guides/platform-stack/_index.md`, `operations/configuration/licenses.md`. - The `iaas` bundle gains two optional packages → `operations/configuration/variants.md`. **Everything else: no match.** Nothing under `packages/apps/` or `packages/extra/` (terraform-provider, website app lists), no `packages/core/installer/values.yaml` or `networking.*`/`publishing.*` key the Ansible role sets, no `hack/` file or package-layout change (ccp, external-apps-example), no `ApplicationDefinition` semantics, no telemetry metric, no node prerequisites (talm, ansible), no commit-convention change (community). - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(cluster-api): add optional Cluster API packages for Proxmox — the capmox infrastructure provider and the in-cluster IPAM provider it requires. Neither is deployed unless named in `bundles.enabledPackages`, since capmox drives a hypervisor outside the cluster and does nothing without Proxmox API credentials. Enabling the Proxmox provider brings the IPAM provider with it. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added optional Proxmox infrastructure provider support for IaaS clusters. - Added in-cluster IP address management, usable independently or alongside Proxmox. - Proxmox deployments require a pre-created credentials Secret and automatically include in-cluster IPAM. - Added validation for incompatible Proxmox and IPAM settings. - **Tests** - Added coverage for provider dependencies, package enablement, disabling behavior, invalid configurations, and packaged provider components. <!-- end of auto-generated comment: release notes by coderabbit.ai -->