fix(seaweedfs): stop the master crashing on the crowded volume map (#5308)

<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Fixes merged to `main` reach the release lines the backport batch runs
for automatically, so do not open backport PRs by hand:
https://github.com/cozystack/cozystack/blob/main/docs/release/backports.md
-->

## What this PR does

Backports seaweedfs/seaweedfs#11216 ("topology: fix fatal concurrent map
read/write on VolumeLayout.crowded", released in 4.46) as a patch on the
in-tree 4.45 image, next to `reader-cache-prefetch-window.diff`.

In 4.45, `(*VolumeLayout).SetVolumeCrowded` mutates the `crowded` map
under `accessLock.RLock()`, while
`(*VolumeLayout).GetWritableVolumeCount` reads the same map under
`RLock()` on the `Assign` path. Two read-lock holders, one writing and
one reading the map, make the Go runtime abort the master with `fatal
error: concurrent map read and map write`. This cannot be recovered, so
the process exits and S3 requests that need an assign fail until a
master is back. The fix takes the write lock in `SetVolumeCrowded`;
every other mutation of `crowded` already holds it.

`patches/volume-layout-crowded-lock.diff` is the upstream squash commit
5d5ea18287cfbf9e274f4b78be40a0ae220cb55d (files under `weed/topology/`),
regenerated against the 4.45 source so it applies without offset. Its
content is unchanged, including the regression test
`TestSetVolumeCrowdedNoRaceWithGetWritableVolumeCount`, which the
Dockerfile's existing `go test` step runs on every build.

#### Seen in production

On a production platform running Cozystack v1.7.0 (in-tree SeaweedFS
4.45), with a dedicated S3 cluster used as the Velero backup target:

- Since the upgrade to 4.45 on 2026-11-05, all 3 masters crash every
night with exit code 2 under the backup load, between 01:00 and 03:00
UTC: 2 crashes on 2026-11-06, about 23 on 2026-11-13, 33 to 43 restarts
per master over 8 days.
- The stack is always the same:
  ```
  fatal error: concurrent map read and map write
  ...

github.com/seaweedfs/seaweedfs/weed/topology.(*VolumeLayout).GetWritableVolumeCount(...)
      weed/topology/volume_layout.go:834
  github.com/seaweedfs/seaweedfs/weed/server.(*MasterServer).Assign(...)
      weed/server/master_grpc_server_assign.go:97
  ```
- Clients see HTTP 500 `InternalError` from the S3 gateway, and the
nightly backups end `Failed` or `PartiallyFailed` (4 `Failed` and 3
`PartiallyFailed` on the night of 2026-11-12 to 13).

Upstream issue: seaweedfs/seaweedfs#11211.

#### Testing

- `git apply --check` of both patches against the 4.45 source archive
(`SOURCE_SHA256` in the Dockerfile matches).
- Regression test on 4.45 with only the test applied, `CGO_ENABLED=0` as
in the image build: `fatal error: concurrent map writes`, FAIL. With the
fix: `go test ./weed/topology/` passes, and the new test passes under
`-race` too.
- `docker buildx build images/seaweedfs --platform linux/amd64` builds,
running the patched packages' tests.
- `make test` in `packages/system/seaweedfs`: 107 helm unit tests, the
naming-guard check and the image/chart version check pass.

#### Lifecycle

The patch is dropped once the vendored chart reaches 4.46 or later, for
example with #4879 (4.48 bump). From that release on it no longer
applies, and `make update-image` fails until it is removed, as the
Dockerfile header says. This PR is kept small so that it does not wait
for #4879.

#### Backport

This is a `fix` with no CRD, migration or API change, and `release-1.7`
carries the same `images/seaweedfs/` as `main`, so the backport batch
should take it onto `release-1.7` with a clean cherry-pick for the next
v1.7.x. Clusters on v1.7 are the ones that crash today.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff touches only `packages/system/seaweedfs/images/seaweedfs/` (a
Go patch and a Dockerfile comment): no package added or renamed, no
values, schema, CRD, `hack/` or installer change, so no entry of the
trigger map applies.

### Release note

```release-note
fix(seaweedfs): stop SeaweedFS masters crashing with "concurrent map read and map write" under write load, which made S3 answer 500 (backport of seaweedfs/seaweedfs#11216 onto 4.45).
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved reliability when updating crowded-volume state concurrently
with writable-volume queries.
* **Documentation**
* Added notes about the upstream fix and when the local patch can be
removed.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
Aleksei Sviridkin authored and GitHub committed 2026-11-13 14:52:45 +03:00
commit 30c4232be1
2 files changed
+96

No files matched your search

@@ -16,6 +16,15 @@
# seaweedfs/seaweedfs#11674 after 4.48; drop the patch on the first vendored
# release that carries it.
#
# patches/volume-layout-crowded-lock.diff makes SetVolumeCrowded take the write
# lock of the volume layout. Upstream 4.45 mutates the crowded map under the
# read lock, while Assign reads it under the same read lock through
# GetWritableVolumeCount, so under a steady stream of writes the master dies on
# a fatal "concurrent map read and map write", which recover cannot catch, and
# S3 answers 500 until it is back. Backport of seaweedfs/seaweedfs#11216,
# released in 4.46; drop the patch on the first vendored release that carries
# it.
#
# `make update` re-pins VERSION from the vendored chart, SOURCE_SHA256 from the
# source archive of that tag, UPSTREAM_DIGEST from the upstream image of that
# tag and the builder from upstream's docker/Dockerfile.go_build, and checks
@@ -0,0 +1,87 @@
diff --git a/weed/topology/crowded_writable_test.go b/weed/topology/crowded_writable_test.go
--- a/weed/topology/crowded_writable_test.go
+++ b/weed/topology/crowded_writable_test.go
@@ -1,6 +1,7 @@
package topology
import (
+ "sync"
"testing"
"time"
@@ -73,3 +74,52 @@ func TestCrowdedCountStillMatchesWritables(t *testing.T) {
t.Errorf("writable=%d crowded=%d, want 1 and 1", writable, crowded)
}
}
+
+// SetVolumeCrowded mutates the crowded map while GetWritableVolumeCount reads
+// it on the Assign hot path. Under RLock both could run at once and the Go
+// runtime aborts with "concurrent map read and map write". Run with -race to
+// reproduce the original bug; the write lock makes this pass.
+func TestSetVolumeCrowdedNoRaceWithGetWritableVolumeCount(t *testing.T) {
+ rp, _ := super_block.NewReplicaPlacementFromString("000")
+ vl := NewVolumeLayout(rp, needle.EMPTY_TTL, types.HardDriveType, 10000, false)
+ // GetWritableVolumeCount iterates vl.writables, so give it something to
+ // read while the crowded map is being mutated.
+ vl.writables = []needle.VolumeId{1, 2, 3}
+
+ var wg sync.WaitGroup
+ stop := make(chan struct{})
+ for range 8 {
+ wg.Add(1)
+ go func() {
+ defer wg.Done()
+ for {
+ select {
+ case <-stop:
+ return
+ default:
+ vl.SetVolumeCrowded(needle.VolumeId(2))
+ }
+ }
+ }()
+ }
+ for range 8 {
+ wg.Add(1)
+ go func() {
+ defer wg.Done()
+ for {
+ select {
+ case <-stop:
+ return
+ default:
+ vl.GetWritableVolumeCount()
+ }
+ }
+ }()
+ }
+
+ // The race detector needs only a brief window to flag a concurrent
+ // map read/write; let the goroutines hammer the lock briefly.
+ time.Sleep(50 * time.Millisecond)
+ close(stop)
+ wg.Wait()
+}
diff --git a/weed/topology/volume_layout.go b/weed/topology/volume_layout.go
--- a/weed/topology/volume_layout.go
+++ b/weed/topology/volume_layout.go
@@ -1019,11 +1019,14 @@ func (vl *VolumeLayout) setVolumeCrowded(vid needle.VolumeId) {
}
func (vl *VolumeLayout) SetVolumeCrowded(vid needle.VolumeId) {
- // since delete is guarded by accessLock.Lock(),
- // and is always called in sequential order,
- // RLock() should be safe enough
- vl.accessLock.RLock()
- defer vl.accessLock.RUnlock()
+ // setVolumeCrowded mutates the crowded map, and GetWritableVolumeCount
+ // reads it under RLock on the Assign hot path. Two concurrent RLock
+ // holders with one writing the map triggers a fatal
+ // "concurrent map read and map write". Take the write lock: this path
+ // is a low-frequency single consumer driven by the crowded-volume
+ // event loop, and every other mutation of crowded already holds Lock().
+ vl.accessLock.Lock()
+ defer vl.accessLock.Unlock()
vl.setVolumeCrowded(vid)
}