mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-05 11:37:31 +09:00
Andrei Kvapil
6c2dba9b21
## What this PR does Adds a fifth piraeus-server patch that retries `drbdadm secondary` after `mkfs` when DRBD reports the device as held open by an external process. ### Problem On Talos (and any system with active block-device probing — udev, multipathd, etc.) the kernel uevent for a freshly-promoted DRBD device causes an external probe to `open()` the device for superblock inspection. If LINSTOR satellite issues `drbdadm secondary` while the probe still holds the fd, DRBD returns `(-12) Device is held open by someone` and aborts initialization with `StorageException: Failed to become secondary again after creating filesystem`. The satellite never reports the final `UpToDate` event to the controller, so the controller view stays in `SyncTarget(N%) / Inconsistent / DELETING` permanently — even though the DRBD kernel state on the satellite is `Established/UpToDate`. This blocks cleanup of orphan PVs in `Released` phase and accumulates with every PVC creation. Observed in production on Talos 1.10 / LINSTOR 1.33.2: 4 occurrences in 24 hours across 3 nodes for different PVCs. Manual recovery requires `drbdadm down/up` on the Primary to force the satellite to re-report state. ### Fix New `secondaryAfterMkfs()` wrapper in `DrbdAdm.java`: - 7 attempts of `drbdadm secondary` with exponential backoff (250 ms → 2 s, capped) - Retry only triggers when stderr contains `Device is held open by someone`; any other failure fails fast as before - Existing `secondary()` contract is unchanged — only `DrbdPrimary.close()` (the post-mkfs demotion path) is rerouted to the new wrapper The retry window covers ~10 s in the worst case, which comfortably exceeds typical probe durations (probes release the fd within milliseconds). ### Related upstream - [LINBIT/linstor-server#268](https://github.com/LINBIT/linstor-server/issues/268) — open since 2022, identical symptom, no upstream fix - [LINBIT/drbd#74](https://github.com/LINBIT/drbd/issues/74) — same EBUSY pattern with multipathd holding the fd ### Validation - `git apply *.diff` on a fresh `v1.33.2` checkout applies all five patches cleanly in alphabetical order - Java brace/paren/bracket balance verified on the patched file - `helm template packages/system/linstor` renders without errors ### Release note ```release-note fix(linstor): retry `drbdadm secondary` after mkfs on transient `Device is held open by someone` errors. Prevents orphan PVs in `Released` phase from accumulating when an external block-device probe (Talos block-controller, udev, multipathd) wins a race against the post-mkfs demotion. Fail-fast behavior preserved for any other secondary failure. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Implemented automatic retry logic for device secondary state transitions after filesystem creation. System now gracefully handles scenarios where devices are temporarily held open by other processes. * **Documentation** * Added documentation for the new device state transition patch. <!-- end of auto-generated comment: release notes by coderabbit.ai -->