Aleksei Sviridkin
54159736bc
## What this PR does Bumps the Talos image Cozystack builds from v1.13.6 to v1.14.2 to get a newer DRBD. v1.13.6 ships DRBD 9.3.2, v1.13.7 and later and v1.14.0/v1.14.1 ship 9.3.3, and v1.14.2 is the first release with 9.3.4. ZFS moves from 2.4.3 to 2.4.4. On 9.3.2 I've seen a peer reboot leave the DRBD sender thread spinning on a failed send (EPIPE). What followed was RCU stalls, processes stuck in D-state, a hanging `drbdsetup disconnect` and two-phase-commit timeouts. The [9.3.4 ChangeLog](https://github.com/LINBIT/drbd/blob/drbd-9.3.4/ChangeLog) has fixes for a sender thread pinning a CPU while its connection is down, for two-phase-commit issues and a hanging `drbdadm disconnect`, and for two nodes ending UpToDate with different data after a reconnect. None of them are in 9.3.3. I matched the trace to those entries by their descriptions only, not against the commits. The base installer image had to change. Talos v1.14 no longer publishes `ghcr.io/siderolabs/installer`, so `gen-profiles.sh` now uses `ghcr.io/siderolabs/installer-base`, same as upstream's own v1.14 imager profiles. I built the amd64 installer from the new profile with `imager:v1.14.2` locally and it builds without errors. PR CI will not boot the new image. It builds the installer and matchbox images, but the e2e runs on the upstream `talos:v1.13.5` container from `hack/e2e-compose.yaml`. The first run that boots the new kernel, DRBD and ZFS is the nightly after merge, which builds the nocloud disk from these profiles and runs it in QEMU. On Talos v1.14.2 the `drbd_transport_tcp` module is no longer loaded on demand. After upgrading a node from v1.13.6, `drbd` was loaded and the transport module was not, `drbdsetup new-peer` failed with "Failed to create transport (drbd_transport_xxx module missing?)", and every DRBD resource on the node stayed in Connecting. Listing the module in `machine.kernel.modules` fixed it without a reboot. The cause is [siderolabs/pkgs#1565](https://github.com/siderolabs/pkgs/pull/1565), in Talos since v1.14.0: the kernel is built with an empty modprobe path, so it no longer loads modules on `request_module()`, which is how DRBD asks for its transport. Any module that used to load that way now has to be listed explicitly (reported upstream as siderolabs/talos#14501). Outside DRBD this includes `dm-thin-pool` and `dm-multipath`, which the v1.14.2 kernel builds as modules, so LVM-thin or multipath users on Talos need them listed too. The e2e node config now loads `drbd_transport_tcp` explicitly, and the talm preset (cozystack/talm#252) and the install docs (cozystack/website#720) get the same line. The module ships in the drbd extension on v1.13 too, so the extra line is safe before the upgrade. The order matters for host nodes. Either update the cozystack preset in the talm project to a version with cozystack/talm#252 and re-render and apply the node config, or add `drbd_transport_tcp` to `machine.kernel.modules` by hand. Only then upgrade the node to v1.14.2. A node upgraded first loses DRBD replication until the module is added. The host network policy now also denies port 2383 to `world`. Talos v1.14 moved etcd's `/metrics`, `/health` and gRPC-gateway JSON API from 2379 to a separate listener on 2383, and the policy only blocked 2379 and 2380. Without this, the etcd JSON API on v1.14 control-plane nodes would be reachable from outside, protected only by client mTLS. Two smaller changes come with the bump. The multus `talos-cni-plugins-checked-against` marker moves to v1.14.2, because the pkgs release behind Talos v1.14.2 still pins CNI plugins v1.9.1, same as multus. The system memory limits doc now describes the v1.14 OOM trigger, since v1.14.0 dropped its global memory PSI clause ([siderolabs/talos#13895](https://github.com/siderolabs/talos/pull/13895)). Before upgrading hosts, operators should know a few things from the [v1.14.0 release notes](https://github.com/siderolabs/talos/releases/tag/v1.14.0) and the Talos compatibility code: - Talos v1.14 accepts a host upgrade from v1.12.0 or later. It supports Kubernetes 1.32 through 1.37, while v1.13 still accepted 1.31, so a cluster on Kubernetes 1.31 has to upgrade Kubernetes first. - etcd's HTTP endpoints (`/metrics`, `/health`) moved from port 2379 to 2383. The Cozystack etcd scrape proxy reads a separate metrics listener on `127.0.0.1:2381`, which this change does not move. Anything else that scraped or health-checked etcd on 2379 has to switch to 2383, and a firewall outside the cluster that blocked 2379 should block 2383 too. - etcd and kube-apiserver now require TLS 1.3, so a client that can only speak TLS 1.2 stops connecting. - Workload isolation (`SecurityProfileConfig`) stays off on upgraded clusters, but `talosctl gen config` turns it on for new ones. I haven't tested Cozystack with it enabled. - `talosctl apply-config --mode=reboot` is gone. ### Screenshots Not a UI change. ### Downstream repositories - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: cozystack/website#720 - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [x] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: cozystack/talm#252 - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The website PR refreshes the `next` version pins for v1.14.2 and adds the module to the Talos install pages. The talm PR adds it to the cozystack preset. ### Release note ```release-note feat(talos): Bump Talos to v1.14.2, which ships DRBD 9.3.4 and ZFS 2.4.4. DRBD 9.3.4 fixes a sender thread spinning on a dead connection, two-phase-commit hangs and a data divergence after a reconnect. Before upgrading a node, add `drbd_transport_tcp` to its Talos `machine.kernel.modules` next to `drbd`: Talos v1.14.2 no longer loads it on demand, and without it DRBD cannot connect to any peer. We recommend skipping the v1.13 line and upgrading nodes straight to v1.14.2. Talos accepts a direct upgrade from v1.12.0 or later. Talos v1.14 moves etcd's HTTP metrics and health endpoints from port 2379 to 2383, and the Cozystack host network policy now blocks 2383 from outside the cluster. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Updates** * Upgraded host node support to Talos v1.14.2 across machine images, with refreshed firmware and storage extensions. * Updated memory-pressure guidance for Talos v1.14, including version-specific behavior and configuration recommendations. * Added protection for Talos v1.14’s etcd HTTP endpoint on port 2383. * Updated cluster preparation to load the TCP transport module required for DRBD. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Cozystack
Cozystack is a free and open-source platform and framework for building clouds.
Cozystack is a CNCF Sandbox project that was originally built and sponsored by Ænix.
With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.
Use Cozystack to build your own cloud or provide a cost-effective development environment.
Use-Cases
-
Using Cozystack to build a public cloud
You can use Cozystack as a backend for a public cloud -
Using Cozystack to build a private cloud
You can use Cozystack as a platform to build a private cloud powered by Infrastructure-as-Code approach -
Using Cozystack as a Kubernetes distribution
You can use Cozystack as a Kubernetes distribution for Bare Metal
Documentation
The documentation is located on the cozystack.io website.
Read the Getting Started section for a quick start.
If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.
Versioning
Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.
Contributions
Contributions are highly appreciated and very welcomed!
In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.
You are welcome to take on the fix yourself. Commits are used to generate the changelog, and their author will be referenced in it.
If you have a concrete feature request, open an issue. For a cross-cutting or architectural change, open a design proposal in the community repo. For questions or early ideas, use GitHub Discussions. See CONTRIBUTING.md for the full routing.
Community
You are welcome to join us in #cozystack, on either CNCF Slack (get an invite) or Kubernetes Slack (get an invite), in our Telegram group, and at our community meetings. Add them to your Google Calendar or iCal for convenience.
Project documents
| Document | What it covers |
|---|---|
| Governance | How decisions are made and who makes them |
| Contributor Ladder | How to grow from contributor to maintainer |
| Maintainers | Who maintains the project |
| Code of Conduct | What we expect of each other |
| Security Policy | Reporting a vulnerability, and which releases get fixes |
| Adopters | Who runs Cozystack in production |
| Roadmap | Where the project is going |
License
Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.
Commercial Support
A list of companies providing commercial support for this project can be found on official site.