<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Fixes merged to `main` reach the release lines the backport batch runs for automatically, so do not open backport PRs by hand: https://github.com/cozystack/cozystack/blob/main/docs/release/backports.md --> ## What this PR does Backports seaweedfs/seaweedfs#11216 ("topology: fix fatal concurrent map read/write on VolumeLayout.crowded", released in 4.46) as a patch on the in-tree 4.45 image, next to `reader-cache-prefetch-window.diff`. In 4.45, `(*VolumeLayout).SetVolumeCrowded` mutates the `crowded` map under `accessLock.RLock()`, while `(*VolumeLayout).GetWritableVolumeCount` reads the same map under `RLock()` on the `Assign` path. Two read-lock holders, one writing and one reading the map, make the Go runtime abort the master with `fatal error: concurrent map read and map write`. This cannot be recovered, so the process exits and S3 requests that need an assign fail until a master is back. The fix takes the write lock in `SetVolumeCrowded`; every other mutation of `crowded` already holds it. `patches/volume-layout-crowded-lock.diff` is the upstream squash commit 5d5ea18287cfbf9e274f4b78be40a0ae220cb55d (files under `weed/topology/`), regenerated against the 4.45 source so it applies without offset. Its content is unchanged, including the regression test `TestSetVolumeCrowdedNoRaceWithGetWritableVolumeCount`, which the Dockerfile's existing `go test` step runs on every build. #### Seen in production On a production platform running Cozystack v1.7.0 (in-tree SeaweedFS 4.45), with a dedicated S3 cluster used as the Velero backup target: - Since the upgrade to 4.45 on 2026-11-05, all 3 masters crash every night with exit code 2 under the backup load, between 01:00 and 03:00 UTC: 2 crashes on 2026-11-06, about 23 on 2026-11-13, 33 to 43 restarts per master over 8 days. - The stack is always the same: ``` fatal error: concurrent map read and map write ... github.com/seaweedfs/seaweedfs/weed/topology.(*VolumeLayout).GetWritableVolumeCount(...) weed/topology/volume_layout.go:834 github.com/seaweedfs/seaweedfs/weed/server.(*MasterServer).Assign(...) weed/server/master_grpc_server_assign.go:97 ``` - Clients see HTTP 500 `InternalError` from the S3 gateway, and the nightly backups end `Failed` or `PartiallyFailed` (4 `Failed` and 3 `PartiallyFailed` on the night of 2026-11-12 to 13). Upstream issue: seaweedfs/seaweedfs#11211. #### Testing - `git apply --check` of both patches against the 4.45 source archive (`SOURCE_SHA256` in the Dockerfile matches). - Regression test on 4.45 with only the test applied, `CGO_ENABLED=0` as in the image build: `fatal error: concurrent map writes`, FAIL. With the fix: `go test ./weed/topology/` passes, and the new test passes under `-race` too. - `docker buildx build images/seaweedfs --platform linux/amd64` builds, running the patched packages' tests. - `make test` in `packages/system/seaweedfs`: 107 helm unit tests, the naming-guard check and the image/chart version check pass. #### Lifecycle The patch is dropped once the vendored chart reaches 4.46 or later, for example with #4879 (4.48 bump). From that release on it no longer applies, and `make update-image` fails until it is removed, as the Dockerfile header says. This PR is kept small so that it does not wait for #4879. #### Backport This is a `fix` with no CRD, migration or API change, and `release-1.7` carries the same `images/seaweedfs/` as `main`, so the backport batch should take it onto `release-1.7` with a clean cherry-pick for the next v1.7.x. Clusters on v1.7 are the ones that crash today. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The diff touches only `packages/system/seaweedfs/images/seaweedfs/` (a Go patch and a Dockerfile comment): no package added or renamed, no values, schema, CRD, `hack/` or installer change, so no entry of the trigger map applies. ### Release note ```release-note fix(seaweedfs): stop SeaweedFS masters crashing with "concurrent map read and map write" under write load, which made S3 answer 500 (backport of seaweedfs/seaweedfs#11216 onto 4.45). ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * Improved reliability when updating crowded-volume state concurrently with writable-volume queries. * **Documentation** * Added notes about the upstream fix and when the local patch can be removed. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Cozystack
Cozystack is a free and open-source platform and framework for building clouds.
Cozystack is a CNCF Sandbox project that was originally built and sponsored by Ænix.
With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.
Use Cozystack to build your own cloud or provide a cost-effective development environment.
Use-Cases
-
Using Cozystack to build a public cloud
You can use Cozystack as a backend for a public cloud -
Using Cozystack to build a private cloud
You can use Cozystack as a platform to build a private cloud powered by Infrastructure-as-Code approach -
Using Cozystack to build your own platform
You can install only the Cozystack components you need on an existing Kubernetes cluster
Documentation
The documentation is located on the cozystack.io website.
Read the Getting Started section for a quick start.
If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.
Versioning
Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.
Contributions
Contributions are highly appreciated and very welcomed!
In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.
You are welcome to take on the fix yourself. Commits are used to generate the changelog, and their author will be referenced in it.
If you have a concrete feature request, open an issue. For a cross-cutting or architectural change, open a design proposal in the community repo. For questions or early ideas, use GitHub Discussions. See CONTRIBUTING.md for the full routing.
Community
You are welcome to join us in #cozystack, on either CNCF Slack (get an invite) or Kubernetes Slack (get an invite), in our Telegram group, and at our community meetings. Add them to your Google Calendar or iCal for convenience.
Project documents
| Document | What it covers |
|---|---|
| Governance | How decisions are made and who makes them |
| Contributor Ladder | How to grow from contributor to maintainer |
| Maintainers | Who maintains the project |
| Code of Conduct | What we expect of each other |
| Security Policy | Reporting a vulnerability, and which releases get fixes |
| Adopters | Who runs Cozystack in production |
| Roadmap | Where the project is going |
License
Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.
Commercial Support
A list of companies providing commercial support for this project can be found on official site.
