mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 12:13:55 +09:00
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Fixes merged to `main` reach the release lines the backport batch runs for automatically, so do not open backport PRs by hand: https://github.com/cozystack/cozystack/blob/main/docs/release/backports.md --> ## What this PR does Backports seaweedfs/seaweedfs#11216 ("topology: fix fatal concurrent map read/write on VolumeLayout.crowded", released in 4.46) as a patch on the in-tree 4.45 image, next to `reader-cache-prefetch-window.diff`. In 4.45, `(*VolumeLayout).SetVolumeCrowded` mutates the `crowded` map under `accessLock.RLock()`, while `(*VolumeLayout).GetWritableVolumeCount` reads the same map under `RLock()` on the `Assign` path. Two read-lock holders, one writing and one reading the map, make the Go runtime abort the master with `fatal error: concurrent map read and map write`. This cannot be recovered, so the process exits and S3 requests that need an assign fail until a master is back. The fix takes the write lock in `SetVolumeCrowded`; every other mutation of `crowded` already holds it. `patches/volume-layout-crowded-lock.diff` is the upstream squash commit 5d5ea18287cfbf9e274f4b78be40a0ae220cb55d (files under `weed/topology/`), regenerated against the 4.45 source so it applies without offset. Its content is unchanged, including the regression test `TestSetVolumeCrowdedNoRaceWithGetWritableVolumeCount`, which the Dockerfile's existing `go test` step runs on every build. #### Seen in production On a production platform running Cozystack v1.7.0 (in-tree SeaweedFS 4.45), with a dedicated S3 cluster used as the Velero backup target: - Since the upgrade to 4.45 on 2026-11-05, all 3 masters crash every night with exit code 2 under the backup load, between 01:00 and 03:00 UTC: 2 crashes on 2026-11-06, about 23 on 2026-11-13, 33 to 43 restarts per master over 8 days. - The stack is always the same: ``` fatal error: concurrent map read and map write ... github.com/seaweedfs/seaweedfs/weed/topology.(*VolumeLayout).GetWritableVolumeCount(...) weed/topology/volume_layout.go:834 github.com/seaweedfs/seaweedfs/weed/server.(*MasterServer).Assign(...) weed/server/master_grpc_server_assign.go:97 ``` - Clients see HTTP 500 `InternalError` from the S3 gateway, and the nightly backups end `Failed` or `PartiallyFailed` (4 `Failed` and 3 `PartiallyFailed` on the night of 2026-11-12 to 13). Upstream issue: seaweedfs/seaweedfs#11211. #### Testing - `git apply --check` of both patches against the 4.45 source archive (`SOURCE_SHA256` in the Dockerfile matches). - Regression test on 4.45 with only the test applied, `CGO_ENABLED=0` as in the image build: `fatal error: concurrent map writes`, FAIL. With the fix: `go test ./weed/topology/` passes, and the new test passes under `-race` too. - `docker buildx build images/seaweedfs --platform linux/amd64` builds, running the patched packages' tests. - `make test` in `packages/system/seaweedfs`: 107 helm unit tests, the naming-guard check and the image/chart version check pass. #### Lifecycle The patch is dropped once the vendored chart reaches 4.46 or later, for example with #4879 (4.48 bump). From that release on it no longer applies, and `make update-image` fails until it is removed, as the Dockerfile header says. This PR is kept small so that it does not wait for #4879. #### Backport This is a `fix` with no CRD, migration or API change, and `release-1.7` carries the same `images/seaweedfs/` as `main`, so the backport batch should take it onto `release-1.7` with a clean cherry-pick for the next v1.7.x. Clusters on v1.7 are the ones that crash today. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The diff touches only `packages/system/seaweedfs/images/seaweedfs/` (a Go patch and a Dockerfile comment): no package added or renamed, no values, schema, CRD, `hack/` or installer change, so no entry of the trigger map applies. ### Release note ```release-note fix(seaweedfs): stop SeaweedFS masters crashing with "concurrent map read and map write" under write load, which made S3 answer 500 (backport of seaweedfs/seaweedfs#11216 onto 4.45). ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * Improved reliability when updating crowded-volume state concurrently with writable-volume queries. * **Documentation** * Added notes about the upstream fix and when the local patch can be removed. <!-- end of auto-generated comment: release notes by coderabbit.ai -->