Aleksei Sviridkin
8b861668b8
## What this PR does Pre-release builds now publish every image as an amd64 and arm64 index, and a release fails if anything it ships is single-arch. Main, release lines and pull requests keep building amd64 only. It builds on #4498, which made every image buildable for both architectures. Part of #1961. **This PR is on hold until everything under "Merge after" has landed.** Merged earlier, the new gate fails the next rc on the images those PRs fix. ### How a pre-release is built A pre-release tag gets a second job, for arm64, next to the existing release job. That covers `-rc.N`, `-beta.N` and `-alpha.N` tags. It runs on the CNCF arm64 pool (`oracle-vm-24cpu-96gb-arm64`), builds the same images with `PLATFORM=linux/arm64`, and pushes them as `<tag>-arm64`. `prepare-release` waits for it, and if the arm64 job fails the rc fails with it. There is no amd64-only fallback. Stable tags do not run it: promotion copies the rc digests with `skopeo copy --multi-arch all`, so a stable release inherits the indexes. After the amd64 build, the stitch joins each amd64 image with its arm64 twin into one index. It lives in `hack/stitch-multiarch.sh`. Every tag of the amd64 image moves to the index, the component-versioned ones included. The script then rewrites the digests in the tree and republishes the packages artifact and the installer chart pinned on them. It reads refs through `hack/lib/image-refs.sh`, and it fails if an old digest survives anywhere outside `charts/`. ### The gate A new check fails the rc when a pinned digest in the tree is not an index with both linux/amd64 and linux/arm64. It lives in `hack/verify-multiarch.sh`. It checks first-party and third-party refs alike. Images that are amd64 by nature go into `hack/multiarch-allowlist`, one repository per line, each with a mandatory reason. Today that is only the e2e sandbox. An entry that matches nothing is reported as stale. The static gate only sees digest-pinned refs in the tree. A tag-only ref, an image from a vendored chart default, or one an operator starts at runtime is invisible to it. So the rc e2e also audits every image its nodes pulled, with the same check and the same allowlist. Neither covers a package the e2e never installs. ### Nightly A new nightly workflow builds main for arm64. It keeps the arm64 build cache warm, since the rc job only reads it. It also prints the single-arch refs that would block the next rc. Nothing it builds is published. ### Smaller changes - `CACHE_TAG` in `hack/common-envs.mk` moves the default cache ref, so the arm64 build keeps a cache of its own. - `MATRIX_ARCH=arm64` makes `hack/build-matrix.sh` leave out the amd64-only e2e sandbox. - The kamaji provider image is pushed under the build's `IMAGE_TAG` like every other image. Its Makefile set `IMAGE_TAG` itself, so concurrent builds overwrote each other's component tag. Closes #4503. - `hack/nightly-mirror.sh` verifies a copy against the raw manifest digest. Plain `skopeo inspect` resolves an index to one platform's child. - matchbox is rebuilt for both platforms in the amd64 release job. The arm64 leg skips the talos package, so there is no arm64 half to stitch, and its Dockerfile only copies files, so no QEMU is needed. Releases then network-boot arm64 machines too. Closes #4524. The stitch also rewrites the kamaji ref inside `files/components.gz`, recompressed with `gzip -n` so the bytes are reproducible, so the kamaji control-plane provider ships multi-arch too. keda and kuberture now name the repository next to their pinned digest, so the gate can resolve them; their rendered manifests do not change. ### Merge after - #4552 builds the Talos installer, matchbox, Harbor and the Velero KubeVirt plugin for arm64, and adds flux-plunger, keycloak-operator, kilo and migration-controller to the root `build:` list. It replaces #4507, #4515, #4525 and #4528. - #4549 moves ingress-nginx, cozy-proxy, cozystack-scheduler and keycloak-kms-proxy to their multi-arch releases, which are already published. The rest of the stack is merged: #4485, #4498, #4512, #4517, #4521 and #4522 here, and the multi-arch PRs in ingress-nginx-with-protobuf-exporter, cozy-proxy, cozystack-scheduler and keycloak-kms-proxy. ### Verification The bats suites pass: stitch, verify-multiarch, build-matrix, common-envs, nightly-mirror and the release contracts. The first three also pass under `hack/cozytest.sh` with dash. Each new test was red before its implementation, and the verify-multiarch and stitch tests each have a mutation that turns them red. I ran `verify-multiarch --report` against main. It lists the first-party refs the stitch will fix, plus the third-party and special cases above. The workflows have not run yet. The arm64 job's tools and duration, the stitch against real registries, and the audit are checked for the first time on the nightly and on the next rc. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: Nothing under `hack/` is moved or renamed, and no make target changes its default behaviour: `CACHE_TAG` and `MATRIX_ARCH` are opt-in. The satellite repositories listed under "Merge after" are prerequisites, not follow-ups this change forces on them. ### Release note ```release-note ci(release): pre-release builds publish every image as an amd64 and arm64 multi-arch index, and a release fails if any image it ships or pulls in e2e lacks either architecture. Stable releases inherit the indexes through promotion. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Release candidates now build amd64 and arm64 images in parallel and combine eligible images into verified multi-architecture indexes before publication. * A nightly arm64 image build is available for testing and supplies a build cache for release-candidate builds. * **Bug Fixes** * End-to-end checks audit whether pulled images support both architectures. Audit failures block prerelease checks, while stable releases continue with a warning. * End-to-end test artifacts now include collected image references and multi-architecture audit results. * **Documentation** * Updated release and image guidance covers multi-architecture builds, verification checks, and troubleshooting, including arm64 build and stitching failures. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Cozystack
Cozystack is a free and open-source platform and framework for building clouds.
Cozystack is a CNCF Sandbox project that was originally built and sponsored by Ænix.
With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.
Use Cozystack to build your own cloud or provide a cost-effective development environment.
Use-Cases
-
Using Cozystack to build a public cloud
You can use Cozystack as a backend for a public cloud -
Using Cozystack to build a private cloud
You can use Cozystack as a platform to build a private cloud powered by Infrastructure-as-Code approach -
Using Cozystack as a Kubernetes distribution
You can use Cozystack as a Kubernetes distribution for Bare Metal
Documentation
The documentation is located on the cozystack.io website.
Read the Getting Started section for a quick start.
If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.
Versioning
Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.
Contributions
Contributions are highly appreciated and very welcomed!
In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.
You are welcome to take on the fix yourself. Commits are used to generate the changelog, and their author will be referenced in it.
If you have a concrete feature request, open an issue. For a cross-cutting or architectural change, open a design proposal in the community repo. For questions or early ideas, use GitHub Discussions. See CONTRIBUTING.md for the full routing.
Community
You are welcome to join us in #cozystack, on either CNCF Slack (get an invite) or Kubernetes Slack (get an invite), in our Telegram group, and at our community meetings. Add them to your Google Calendar or iCal for convenience.
Project documents
| Document | What it covers |
|---|---|
| Governance | How decisions are made and who makes them |
| Contributor Ladder | How to grow from contributor to maintainer |
| Maintainers | Who maintains the project |
| Code of Conduct | What we expect of each other |
| Security Policy | Reporting a vulnerability, and which releases get fixes |
| Adopters | Who runs Cozystack in production |
| Roadmap | Where the project is going |
License
Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.
Commercial Support
A list of companies providing commercial support for this project can be found on official site.