mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 20:23:39 +09:00
fix(seaweedfs): stop the master crashing on the crowded volume map (#5308)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Fixes merged to `main` reach the release lines the backport batch runs for automatically, so do not open backport PRs by hand: https://github.com/cozystack/cozystack/blob/main/docs/release/backports.md --> ## What this PR does Backports seaweedfs/seaweedfs#11216 ("topology: fix fatal concurrent map read/write on VolumeLayout.crowded", released in 4.46) as a patch on the in-tree 4.45 image, next to `reader-cache-prefetch-window.diff`. In 4.45, `(*VolumeLayout).SetVolumeCrowded` mutates the `crowded` map under `accessLock.RLock()`, while `(*VolumeLayout).GetWritableVolumeCount` reads the same map under `RLock()` on the `Assign` path. Two read-lock holders, one writing and one reading the map, make the Go runtime abort the master with `fatal error: concurrent map read and map write`. This cannot be recovered, so the process exits and S3 requests that need an assign fail until a master is back. The fix takes the write lock in `SetVolumeCrowded`; every other mutation of `crowded` already holds it. `patches/volume-layout-crowded-lock.diff` is the upstream squash commit 5d5ea18287cfbf9e274f4b78be40a0ae220cb55d (files under `weed/topology/`), regenerated against the 4.45 source so it applies without offset. Its content is unchanged, including the regression test `TestSetVolumeCrowdedNoRaceWithGetWritableVolumeCount`, which the Dockerfile's existing `go test` step runs on every build. #### Seen in production On a production platform running Cozystack v1.7.0 (in-tree SeaweedFS 4.45), with a dedicated S3 cluster used as the Velero backup target: - Since the upgrade to 4.45 on 2026-11-05, all 3 masters crash every night with exit code 2 under the backup load, between 01:00 and 03:00 UTC: 2 crashes on 2026-11-06, about 23 on 2026-11-13, 33 to 43 restarts per master over 8 days. - The stack is always the same: ``` fatal error: concurrent map read and map write ... github.com/seaweedfs/seaweedfs/weed/topology.(*VolumeLayout).GetWritableVolumeCount(...) weed/topology/volume_layout.go:834 github.com/seaweedfs/seaweedfs/weed/server.(*MasterServer).Assign(...) weed/server/master_grpc_server_assign.go:97 ``` - Clients see HTTP 500 `InternalError` from the S3 gateway, and the nightly backups end `Failed` or `PartiallyFailed` (4 `Failed` and 3 `PartiallyFailed` on the night of 2026-11-12 to 13). Upstream issue: seaweedfs/seaweedfs#11211. #### Testing - `git apply --check` of both patches against the 4.45 source archive (`SOURCE_SHA256` in the Dockerfile matches). - Regression test on 4.45 with only the test applied, `CGO_ENABLED=0` as in the image build: `fatal error: concurrent map writes`, FAIL. With the fix: `go test ./weed/topology/` passes, and the new test passes under `-race` too. - `docker buildx build images/seaweedfs --platform linux/amd64` builds, running the patched packages' tests. - `make test` in `packages/system/seaweedfs`: 107 helm unit tests, the naming-guard check and the image/chart version check pass. #### Lifecycle The patch is dropped once the vendored chart reaches 4.46 or later, for example with #4879 (4.48 bump). From that release on it no longer applies, and `make update-image` fails until it is removed, as the Dockerfile header says. This PR is kept small so that it does not wait for #4879. #### Backport This is a `fix` with no CRD, migration or API change, and `release-1.7` carries the same `images/seaweedfs/` as `main`, so the backport batch should take it onto `release-1.7` with a clean cherry-pick for the next v1.7.x. Clusters on v1.7 are the ones that crash today. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The diff touches only `packages/system/seaweedfs/images/seaweedfs/` (a Go patch and a Dockerfile comment): no package added or renamed, no values, schema, CRD, `hack/` or installer change, so no entry of the trigger map applies. ### Release note ```release-note fix(seaweedfs): stop SeaweedFS masters crashing with "concurrent map read and map write" under write load, which made S3 answer 500 (backport of seaweedfs/seaweedfs#11216 onto 4.45). ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * Improved reliability when updating crowded-volume state concurrently with writable-volume queries. * **Documentation** * Added notes about the upstream fix and when the local patch can be removed. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
2 files changed
+96
No files matched your search
@@ -16,6 +16,15 @@
|
||||
# seaweedfs/seaweedfs#11674 after 4.48; drop the patch on the first vendored
|
||||
# release that carries it.
|
||||
#
|
||||
# patches/volume-layout-crowded-lock.diff makes SetVolumeCrowded take the write
|
||||
# lock of the volume layout. Upstream 4.45 mutates the crowded map under the
|
||||
# read lock, while Assign reads it under the same read lock through
|
||||
# GetWritableVolumeCount, so under a steady stream of writes the master dies on
|
||||
# a fatal "concurrent map read and map write", which recover cannot catch, and
|
||||
# S3 answers 500 until it is back. Backport of seaweedfs/seaweedfs#11216,
|
||||
# released in 4.46; drop the patch on the first vendored release that carries
|
||||
# it.
|
||||
#
|
||||
# `make update` re-pins VERSION from the vendored chart, SOURCE_SHA256 from the
|
||||
# source archive of that tag, UPSTREAM_DIGEST from the upstream image of that
|
||||
# tag and the builder from upstream's docker/Dockerfile.go_build, and checks
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
diff --git a/weed/topology/crowded_writable_test.go b/weed/topology/crowded_writable_test.go
|
||||
--- a/weed/topology/crowded_writable_test.go
|
||||
+++ b/weed/topology/crowded_writable_test.go
|
||||
@@ -1,6 +1,7 @@
|
||||
package topology
|
||||
|
||||
import (
|
||||
+ "sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -73,3 +74,52 @@ func TestCrowdedCountStillMatchesWritables(t *testing.T) {
|
||||
t.Errorf("writable=%d crowded=%d, want 1 and 1", writable, crowded)
|
||||
}
|
||||
}
|
||||
+
|
||||
+// SetVolumeCrowded mutates the crowded map while GetWritableVolumeCount reads
|
||||
+// it on the Assign hot path. Under RLock both could run at once and the Go
|
||||
+// runtime aborts with "concurrent map read and map write". Run with -race to
|
||||
+// reproduce the original bug; the write lock makes this pass.
|
||||
+func TestSetVolumeCrowdedNoRaceWithGetWritableVolumeCount(t *testing.T) {
|
||||
+ rp, _ := super_block.NewReplicaPlacementFromString("000")
|
||||
+ vl := NewVolumeLayout(rp, needle.EMPTY_TTL, types.HardDriveType, 10000, false)
|
||||
+ // GetWritableVolumeCount iterates vl.writables, so give it something to
|
||||
+ // read while the crowded map is being mutated.
|
||||
+ vl.writables = []needle.VolumeId{1, 2, 3}
|
||||
+
|
||||
+ var wg sync.WaitGroup
|
||||
+ stop := make(chan struct{})
|
||||
+ for range 8 {
|
||||
+ wg.Add(1)
|
||||
+ go func() {
|
||||
+ defer wg.Done()
|
||||
+ for {
|
||||
+ select {
|
||||
+ case <-stop:
|
||||
+ return
|
||||
+ default:
|
||||
+ vl.SetVolumeCrowded(needle.VolumeId(2))
|
||||
+ }
|
||||
+ }
|
||||
+ }()
|
||||
+ }
|
||||
+ for range 8 {
|
||||
+ wg.Add(1)
|
||||
+ go func() {
|
||||
+ defer wg.Done()
|
||||
+ for {
|
||||
+ select {
|
||||
+ case <-stop:
|
||||
+ return
|
||||
+ default:
|
||||
+ vl.GetWritableVolumeCount()
|
||||
+ }
|
||||
+ }
|
||||
+ }()
|
||||
+ }
|
||||
+
|
||||
+ // The race detector needs only a brief window to flag a concurrent
|
||||
+ // map read/write; let the goroutines hammer the lock briefly.
|
||||
+ time.Sleep(50 * time.Millisecond)
|
||||
+ close(stop)
|
||||
+ wg.Wait()
|
||||
+}
|
||||
diff --git a/weed/topology/volume_layout.go b/weed/topology/volume_layout.go
|
||||
--- a/weed/topology/volume_layout.go
|
||||
+++ b/weed/topology/volume_layout.go
|
||||
@@ -1019,11 +1019,14 @@ func (vl *VolumeLayout) setVolumeCrowded(vid needle.VolumeId) {
|
||||
}
|
||||
|
||||
func (vl *VolumeLayout) SetVolumeCrowded(vid needle.VolumeId) {
|
||||
- // since delete is guarded by accessLock.Lock(),
|
||||
- // and is always called in sequential order,
|
||||
- // RLock() should be safe enough
|
||||
- vl.accessLock.RLock()
|
||||
- defer vl.accessLock.RUnlock()
|
||||
+ // setVolumeCrowded mutates the crowded map, and GetWritableVolumeCount
|
||||
+ // reads it under RLock on the Assign hot path. Two concurrent RLock
|
||||
+ // holders with one writing the map triggers a fatal
|
||||
+ // "concurrent map read and map write". Take the write lock: this path
|
||||
+ // is a low-frequency single consumer driven by the crowded-volume
|
||||
+ // event loop, and every other mutation of crowded already holds Lock().
|
||||
+ vl.accessLock.Lock()
|
||||
+ defer vl.accessLock.Unlock()
|
||||
|
||||
vl.setVolumeCrowded(vid)
|
||||
}
|
||||
Reference in new issue
Block a user