mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-09 05:25:14 +09:00
Aleksei Sviridkin
4a2dea2d08
## What this PR does **Series behind #3481** — three separable pull requests: 1. **#4087 — Cluster API providers** (capmox + in-cluster IPAM) 2. #4088 — the `substrate` switch in `kubernetes` and `kubernetes-nodes` 3. #4089 — the Proxmox CSI driver and cloud-controller-manager #4087 and #4088 share no files and can be reviewed in parallel. #4089 builds on #4088. --- This is the provisioning layer #3481 asks to specify first: the part that creates the VMs. Neither of the other two is needed to review it. ### What it adds Two Cluster API providers, packaged the way the existing KubeVirt provider is — vendored components, `gzip -9 -n` for a reproducible artifact, a `make components` target that regenerates it from the upstream release and checks the downloads against the SHA-256 sums the Makefile pins: | Package | Version | Contract | |---|---|---| | `capi-providers-infraprovider-proxmox` | capmox v0.7.7 | v1beta1 | | `capi-providers-ipam-in-cluster` | CAIP v1.0.3 | v1beta1 | capmox v0.7.7 rather than the current v0.9.x on purpose: from v0.8 it declares contract `v1beta2`, which the CAPI core shipped here (v1.10.1) does not serve. Both join the `iaas` bundle as **opt-in** packages, where the KubeVirt infrastructure provider is unconditional. The asymmetry is deliberate: capmox drives a hypervisor outside the cluster and needs a Secret holding a Proxmox API token, which capi-operator reads before it installs anything — with no Secret the provider is never deployed at all. Enabling it by default would ask every Cozystack install that has no Proxmox to supply hypervisor credentials. ```yaml bundles: enabledPackages: - cozystack.capi-provider-infra-proxmox # brings IPAM with it - cozystack.capi-provider-ipam-in-cluster # also enableable on its own ``` Enabling capmox emits the IPAM provider automatically — capmox allocates worker addresses from an `InClusterIPPool` and has no DHCP mode, so on its own it never gets a `ProxmoxCluster` to `Ready`. IPAM stays independently enableable, since assigning addresses is a generic Cluster API concern. Naming capmox in `enabledPackages` while disabling IPAM stops the render. The cluster would catch that combination on its own — the capmox `PackageSource` depends on the IPAM `Package`, so with that `Package` absent capmox goes `Ready=False` with `DependenciesNotReady` and no `HelmRelease` is created — but it would not say why. Failing the render names the two conflicting list entries at the moment someone writes them, instead of leaving a condition to decode. ### Why the earlier attempt did not produce a VM #1927's blocker — "ProxmoxMachine does not produce a VM" — was not a capmox incompatibility. It was four independent causes, each of which only became visible once the previous one was cleared: 1. **`PROXMOX_URL` included `/api2/json`.** capmox appends that itself, so every call went to `/api2/json/api2/json/…` and returned 501. 2. **No IPAM provider.** `no matches for kind "InClusterIPPool"`; the `ProxmoxCluster` never reached `Ready` and every `ProxmoxMachine` sat in `WaitingForClusterInfrastructure`. 3. **`templateSelector.matchTags` is set equality, not containment** (`slices.Equal` after sort). A template carrying four tags is not selected by naming one of them: `found 0 VM templates with tags`. 4. **capmox waits for the qemu-guest-agent, which Talos does not ship.** `error waiting for agent: timed out` — cleared with `checks.skipQemuGuestAgent` and `skipCloudInitStatus`. None of these are in the provider. They are packaging and configuration, which is why the earlier work could look complete and still never boot a machine. ### Evidence Run on Proxmox VE 9.2.11 with Talos v1.13.6 and Cozystack's CAPI stack (core v1.10.1, CABPT v0.6.12, Kamaji control-plane provider v0.19.0): ``` $ kubectl get proxmoxmachine,machine -n tenant-proxmox proxmoxmachine/kubernetes-px-md0-… Ready=True VMProvisioned=True machine/kubernetes-px-md0-… Running proxmox://0ef88cc1-6aea-4563-af64-344d8060d5c0 $ qm list 106 kubernetes-px-md0-jmzds-h9npx running ``` The machine boots Talos from the nocloud ISO, reads the machineconfig capmox writes, and joins. That is the layer #69 called a hard blocker. ### What this PR does not claim - One Proxmox node, one tenant cluster, manual runs — there is no e2e coverage for Proxmox in CI, and this PR does not add any. - It does not make Proxmox a supported platform. It makes the provisioning layer reviewable, which #3481 names as the precondition for that conversation. - It does not touch `capi-providers-core`. The same drift check, pointed at that package, shows that the `startupProbe` from #2946 never reached the shipped blob; that fix changes what every new install runs and is #4483, with its own release note. - Nothing here exposes `ProxmoxCluster.spec.credentialsRef` to a tenant, and nothing should: `capmox-manager-role` grants get/list/watch/patch on Secrets cluster-wide and that field takes a namespace. Whatever renders a `ProxmoxCluster` for a tenant in the rest of the series has to keep the field out of tenant hands. ### Testing `make unit-tests` — exit 0. New suites: `packages/core/platform/tests/sources_capi_provider_proxmox_test.yaml` and `bundles_proxmox_wiring_test.yaml` (both bundle paths, the enable/disable precedence, and the refusal above). Every negated `containsDocument` in the wiring suite carries `any: true`; without it the negation passes whether or not the document is in the render. Three mutations, each reverted afterwards — emitting the capmox `Package` unconditionally, emitting the IPAM `Package` unconditionally, and dropping the `disabledPackages` check from `$proxmoxEnabled` — each turn red the tests named after the property they break, and the unmutated chart stays green (44 suites, 198 tests). `hack/capi-provider-components_test.bats` covers both new provider packages: the blob and the YAML are the same bytes; the Makefile version, the ConfigMap name and `spec.version` are one value — `applied-spec-hash` covers the provider spec and the config Secret but not the ConfigMap, so a bump that regenerates the files while `spec.version` stays behind installs the old provider silently; and the vendored files match the SHA-256 sums the Makefile pins, so the pin holds even when nobody runs `make components`. The render was compared against `main` for five configurations (defaults, `isp-full`, `isp-full-generic` with paas, an unrelated `enabledPackages`, and `disabledPackages` naming IPAM). No line is removed in any of them and the `Package` count is unchanged; the only additions are the two `PackageSource` objects, which the chart renders for every package, optional or not. ### Screenshots Not applicable — no UI change. ### Downstream repositories I walked the trigger map in `docs/agents/contributing.md` against this diff, file by file. What it matches is below. **No box is ticked**, because every follow-up it points at depends on a decision that is not mine to make: #3481 asks whether Proxmox becomes a supported platform at all, and writing its documentation or its Terraform schema before that is answered would be work built on an assumption. I have not filed speculative PRs or issues in those repositories for the same reason. If the direction is accepted, I open them; a human should decide which. **cozystack/website — matches, no follow-up yet.** - `packages/core/platform/values.yaml` changed. `content/en/docs/next/operations/configuration/platform-package.md` is a hand-written table of the `spec.components.platform.values.*` keys; this PR does not add a key, but it documents two new values that `bundles.enabledPackages` now accepts, and the table is where a reader would look for them. - Two platform components are added (`capi-providers-infraprovider-proxmox`, `capi-providers-ipam-in-cluster`) → `guides/platform-stack/_index.md`, `operations/configuration/licenses.md`. - The `iaas` bundle gains two optional packages → `operations/configuration/variants.md`. **Everything else: no match.** Nothing under `packages/apps/` or `packages/extra/` (terraform-provider, website app lists), no `packages/core/installer/values.yaml` or `networking.*`/`publishing.*` key the Ansible role sets, no `hack/` file or package-layout change (ccp, external-apps-example), no `ApplicationDefinition` semantics, no telemetry metric, no node prerequisites (talm, ansible), no commit-convention change (community). - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(cluster-api): add optional Cluster API packages for Proxmox — the capmox infrastructure provider and the in-cluster IPAM provider it requires. Neither is deployed unless named in `bundles.enabledPackages`, since capmox drives a hypervisor outside the cluster and does nothing without Proxmox API credentials. Enabling the Proxmox provider brings the IPAM provider with it. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added optional Proxmox infrastructure provider support for IaaS clusters. - Added in-cluster IP address management, usable independently or alongside Proxmox. - Proxmox deployments require a pre-created credentials Secret and automatically include in-cluster IPAM. - Added validation for incompatible Proxmox and IPAM settings. - **Tests** - Added coverage for provider dependencies, package enablement, disabling behavior, invalid configurations, and packaged provider components. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Cozystack
Cozystack is a free and open-source platform and framework for building clouds.
Cozystack is a CNCF Sandbox project that was originally built and sponsored by Ænix.
With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.
Use Cozystack to build your own cloud or provide a cost-effective development environment.
Use-Cases
-
Using Cozystack to build a public cloud
You can use Cozystack as a backend for a public cloud -
Using Cozystack to build a private cloud
You can use Cozystack as a platform to build a private cloud powered by Infrastructure-as-Code approach -
Using Cozystack as a Kubernetes distribution
You can use Cozystack as a Kubernetes distribution for Bare Metal
Documentation
The documentation is located on the cozystack.io website.
Read the Getting Started section for a quick start.
If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.
Versioning
Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.
Contributions
Contributions are highly appreciated and very welcomed!
In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.
You are welcome to take on the fix yourself. Commits are used to generate the changelog, and their author will be referenced in it.
If you have a concrete feature request, open an issue. For a cross-cutting or architectural change, open a design proposal in the community repo. For questions or early ideas, use GitHub Discussions. See CONTRIBUTING.md for the full routing.
Community
You are welcome to join us in #cozystack, on either CNCF Slack (get an invite) or Kubernetes Slack (get an invite), in our Telegram group, and at our community meetings. Add them to your Google Calendar or iCal for convenience.
Project documents
| Document | What it covers |
|---|---|
| Governance | How decisions are made and who makes them |
| Contributor Ladder | How to grow from contributor to maintainer |
| Maintainers | Who maintains the project |
| Code of Conduct | What we expect of each other |
| Security Policy | Reporting a vulnerability, and which releases get fixes |
| Adopters | Who runs Cozystack in production |
| Roadmap | Where the project is going |
License
Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.
Commercial Support
A list of companies providing commercial support for this project can be found on official site.