mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-03 02:27:01 +09:00
myasnikovdaniil
3748352eda
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does Tenant worker boots from talos OS disk image that CDI streams over HTTP from image factory, and it does that once per worker. Same bytes N times, and N chances for a fetch to fail. In closed or rate-limited environment that is wrong shape. This PR adds opt-in catalog and per-pool selector that reads from it. `packages/system/kubernetes-worker-image` imports one golden DataVolume into `cozy-public` per `(schematicID, version)`. Disabled by default, enable by adding `cozystack.kubernetes-worker-image` to `bundles.enabledPackages`. Its knobs (`imageFactoryURL`, `storageClass`, `storage`, `images`) are set through `spec.components.platform.values.kubernetesWorkerImage` on cozystack-platform Package. That needed optional package helper to learn components argument, which it did not have, so every optional package registered through it could name variant and nothing else. Nine other packages get same path for free. `KubernetesNodes` gets `osImage` field with two mutually exclusive forms: - `osImage.builtin` boots pool workers by CDI-cloning golden from catalog. Clone is storage layer CSI copy, so OS image never crosses pod network and factory answers once for whole fleet instead of once per worker. - `osImage.factory` keeps HTTP import and lets pool pin own `imageFactoryURL`, `schematicID` and `version`. This is bring-your-own path for operator running own factory or caching mirror, and it is what pool gets when `osImage` is omitted. In-guest talos installer written by reconcile job follows same resolution, so pool that pins release boots it and upgrades into it instead of being flipped back to cluster default halfway. This is pool half of what #3950 asks for. That issue wants platform level defaults for two channels, worker OS image and in-guest registry mirrors. Here image channel gets per-pool source and shared cache, platform default plumbing is left to the issue. ### Removed e2e workaround Suites used to deploy `talos-image-cache` Deployment and point tenant CRs at it, on theory that public factory stalling mid-transfer is what makes workers miss node-join deadline. Work on #3513 settled it differently, what separates run that joins from run that stalls is host `kvm_amd` `vgif` flag, and stall reproduces on runs where cache was up and serving. So cache, its manifest, helper and test suite are gone and suites clone golden instead. That also puts clone path under e2e, unit tests alone never reached it. Stacked on #4007 which removes other half of same band-aid on same evidence. Base of this PR points at that branch, so diff here is only this feature. Merge #4007 first. ### Existing workers do not roll With `osImage` omitted rendered `KubevirtMachineTemplate` is byte identical to what chart produced before. KMT is named by hash of its own content, so any drift would re-image every worker VM in fleet on next upgrade. Stored render snapshot is untouched by this PR, that is the evidence and not the claim. ### What fails at render Clone that would silently degrade into something slower or wrong is refused instead of rendered. Golden absent, or its import in terminal `Failed` phase. Pool `storageClass` different from golden one, because CDI cant CSI-clone across storage classes and falls back to host assisted copy over pod network, which is exactly the transfer `osImage.builtin` removes. Empty class on either side resolves against cluster default class before comparing, so pool at `storageClass: ""` is checked too. `diskSize` below golden storage, because clone target cannot be smaller than source. Talos release the matrix lists as unsupported, whether pool reached it through `talos.version` or through `osImage.*` override. Matrix passes Talos minor it does not list, same as before, so what it promises is that known-bad pairing is refused, not that every pairing is known. Golden that is only mid-import is not an error. CDI holds worker disk until source populates, and failing there would abort whole pool HelmRelease for length of a download. Same reasoning now covers pool that already cloned: absent or `Failed` golden fails render only for pool still waiting for its first disk, otherwise reconcile of healthy pool would abort over source it no longer reads. Catalog side refuses duplicate `(schematicID, version)` entries, entries whose schematic or version cannot produce name pool can reconstruct, and edit to `storage`, `storageClass` or source URL of entry whose golden already exists. Last one is not patchable in place, so without guard it lands as rejected patch on next upgrade. Golden that declares no StorageClass is read from its PVC, not from whatever default is in force today, because those differ once cluster default moves. Cluster with two default classes resolves same way Kubernetes does, newest wins. Pool that already cloned is identified by annotation naming exactly that pool, not by disk name prefix, because pool `md0-gpu` produces disks carrying `md0` prefix. Three drift guards live in `hack/check-worker-image-catalog-defaults.bats`, because values that have to agree sit in files nobody edits together. One-sided talos bump trips all three. ### Testing - `helm unittest`: kubernetes-nodes 117, kubernetes 145, kubernetes-worker-image 14, core/platform 143, render snapshot unchanged - full `make unit-tests`: 82 charts, 1742 helm-unittest cases, 1485 bats cases, exit 0 - mutation pass over every guard term this diff adds: each term deleted in turn, suite required to notice. Four terms are not caught by `helm unittest`, and the count an earlier revision of this description carried was wrong. One is the second support-matrix call, pinned instead by the greps in `hack/check-worker-image-catalog-defaults.bats:96-110`. The other three were reported by @IvanHunters and re-confirmed here by re-running his mutations: dropping the written `kubernetes-worker-image.cozystack.io/pool` annotation leaves the suite green (only the reader is asserted), replacing the legacy `storageclass.beta.kubernetes.io/is-default-class` disjunct with `false` leaves it green, and forcing the `if $goldenStorage` presence term open leaves it green while forcing it shut reddens two cases. All three are test-coverage gaps rather than live defects, and they are deliberately left open: this round closes only the two findings that change behaviour - `go build` and `go vet` on root and `api/apps/v1alpha1` - root `make generate` and `pre-commit` clean ### Verified on a live cluster The clone path was exercised end to end on a throwaway three-node Talos stand (OCI, Talos v1.13.6, LINSTOR/ZFS, `replicated` StorageClass) installed from this branch's `packages/` tree. This covers what neither `helm template` nor the unit suites can answer. - The catalog wires up from `bundles.enabledPackages` alone: `PackageSource cozystack.kubernetes-worker-image` reconciled, its HelmRelease installed, and the golden `talos-worker-<schematic>-v1.13.6` imported to `Succeeded` in 108s (6Gi, RWX, `replicated`). - A pool on `osImage.builtin` clones it on the storage layer. Both worker DataVolumes went `CSICloneInProgress` then `Succeeded` in 43s with `Running=False reason=Completed message="Clone Complete"`, carrying `source.pvc` and no `source.http`, and no importer or upload Pod appeared in the tenant namespace. - The clone yields a bootable Talos root disk. Both worker VMs reached `Running`/`Ready`, joined the tenant cluster and went `Ready` about 66s after creation, and the tenant's cilium, coredns, csi, ingress-nginx and metrics-server addons all installed on them. - Access modes match across the hop: golden `ReadWriteMany` 6Gi to worker `ReadWriteMany` 20Gi on the same `replicated` class, grown by CDI. - The written annotations are correct in practice: each worker disk carries `kubernetes-worker-image.cozystack.io/golden` naming the golden and `.../pool` naming `kubernetes-wimg-md0`. - helm-controller converges on a builtin pool: forcing a reconcile left the content-hashed `KubevirtMachineTemplate` name unchanged and the VMI and DataVolume UIDs identical, so a reconcile does not roll the pool. - Cross-namespace clone RBAC is in place as shipped: `create datavolumes --subresource=source -n cozy-public` resolves to `yes` for a tenant ServiceAccount through the `cdi-clone-dv` RoleBinding, and CDI runs with `cloneStrategyOverride: csi-clone`. Both behavioural findings were reproduced on the same stand. A `KubernetesNodes` carrying `talos.version: v1.13` was admitted by the API with the value intact and its HelmRelease went `InstallFailed` naming the format; a subsequent unrelated edit (`maxReplicas` 1 to 3) was accepted by the CR and reached `.spec.values` of the HelmRelease while it stayed `InstallFailed`, so the edit is taken everywhere and applied nowhere. And a pool on `osImage.builtin` with `talos.imageFactoryURL: ""` -- a value the API preserves rather than defaulting away -- renders and clones normally with the guard scoped, which is the fix in this round. ### Screenshots Not applicable, no UI change. ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [x] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: https://github.com/cozystack/terraform-provider-cozystack/issues/39 - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note feat(kubernetes): worker pool can boot from shared golden talos image instead of importing OS image over HTTP per worker. Enable optional `cozystack.kubernetes-worker-image` package to seed one golden DataVolume per (schematicID, version) in `cozy-public`, configure it through `kubernetesWorkerImage` in cozystack-platform values, then set `osImage.builtin` on `KubernetesNodes` pool to clone it. `osImage.factory` keeps HTTP path and adds per-pool overrides for `imageFactoryURL`, `schematicID` and `version`. Pools that leave `osImage` unset render as before and their workers do not roll. The render also refuses a `talos.schematicID`, `talos.version` or `imageFactoryURL` -- and their `osImage.*` overrides -- whose shape cannot produce a working image reference: a plain lowercase alphanumeric schematic, a full `vMAJOR.MINOR.PATCH` release, and a plain `http(s)` URL. Every refused shape already produced an unbootable worker before this change, but it rendered, so a pool holding one moves from Ready-with-no-workers to a failed HelmRelease. The values schema types all three as bare strings, so the rejection lands on the HelmRelease rather than on the CR: the CR is admitted with the bad value, and later edits to unrelated fields on that pool are accepted and reach the HelmRelease values without being applied. `imageFactoryURL` is checked only on the paths that build a source URL from it, so a pool on `osImage.builtin` may leave it blank. ```
Cozystack
Cozystack is a free platform and framework for building clouds.
Cozystack is a CNCF Sandbox Level Project that was originally built and sponsored by Ænix.
With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.
Use Cozystack to build your own cloud or provide a cost-effective development environment.
Use-Cases
-
Using Cozystack to build a public cloud
You can use Cozystack as a backend for a public cloud -
Using Cozystack to build a private cloud
You can use Cozystack as a platform to build a private cloud powered by Infrastructure-as-Code approach -
Using Cozystack as a Kubernetes distribution
You can use Cozystack as a Kubernetes distribution for Bare Metal
Documentation
The documentation is located on the cozystack.io website.
Read the Getting Started section for a quick start.
If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.
Versioning
Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.
Contributions
Contributions are highly appreciated and very welcomed!
In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.
You can express your intention to on the fix on your own. Commits are used to generate the changelog, and their author will be referenced in it.
If you have a concrete feature request, open an issue. For a cross-cutting or architectural change, open a design proposal in the community repo. For questions or early ideas, use GitHub Discussions. See CONTRIBUTING.md for the full routing.
Community
You are welcome to join our Telegram group and come to our weekly community meetings. Add them to your Google Calendar or iCal for convenience.
License
Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.
Commercial Support
A list of companies providing commercial support for this project can be found on official site.