## What this PR does
Cozystack's Talos configuration removes the systemd DRBD shutdown guard
without a replacement. Add `talos.gracefulShutdown.enabled`, disabled by
default, to release unused Secondary DRBD resources before the satellite
stops so their backing devices, including ZFS zvols, can be released
during normal shutdown. The hook skips Primary resources and open
devices, rechecks each candidate, and aborts further actions on
ambiguous status, command failure or timeout. It uses no force,
demotion, Kubernetes API calls or additional permissions.
The hook is available only when both Talos and DRBD are enabled. Keep
the value in
`spec.components.linstor.values.talos.gracefulShutdown.enabled` on the
`cozystack.linstor` package. The chart embeds the script in its existing
`cozystack-talos` satellite configuration, which Piraeus merges into the
generated DaemonSets. No custom satellite image is required. Default
renders remain unchanged for Talos, generic Linux and DRBD-less Talos.
This is opt-in because preStop also runs on ordinary satellite restarts.
Unused Secondary replicas disconnect until their replacement satellite
reattaches them, and Piraeus has one DaemonSet per node without a
cluster-wide rollout barrier. Simultaneous restarts can therefore pause
I/O. Enable this only with sequential satellite/node maintenance and
replication checks between nodes. The README covers the 60-second hook
budget, pod versus node shutdown grace periods, rollback behavior and
the unsupported forced-reboot/broken-kubelet paths. Related upstream
discussion:
https://github.com/piraeusdatastore/piraeus-operator/issues/860. The
reported outage from unconditional forced demotion in
https://github.com/piraeusdatastore/piraeus-operator/issues/860#issuecomment-5635881307
is why this hook never demotes a Primary, even when its devices appear
closed.
Validation: 22 Helm tests and 20 behavioral Python tests pass. The same
20 tests pass inside the existing Cozystack satellite image. Two of them
pin the rendered command form: `python3 -c <script> --execute` reaches
`main()` with `["--execute"]` because CPython sets `sys.argv[0]` to
`-c`, and a script-name placeholder before `--execute` is rejected.
Failure-path tests cover nonzero status/down exits, bounded stderr,
partial timeout diagnostics, and stopping before the next resource. Full
default renders match the base commit for all three supported substrate
variants. The rendered script survives strategic merging against Piraeus
2.10.2 and 2.11.0 satellite manifests; the base chart fails the new
feature's regression tests.
Runtime validation on Talos 1.13.6 used the script from commit
`b8c86f2e70d029166939ff29eab8a45ac191e34b` through per-node satellite
configurations: a normal node reboot with drain completed without force
or an external reset, and the node returned with its ZFS pool online and
DRBD replicas UpToDate. That version is installed on all three nodes of
the tested cluster after sequential satellite rollouts and replication
checks. The later diagnostic-only update in
`527570d8079cba7c91b3c345050cab1b73a290a8` is covered by the local and
exact-image tests above but has not been rolled out to that cluster.
This does not validate forced reboot, a failed kubelet, or simultaneous
satellite restarts. GitHub Actions for this fork require maintainer
approval; local checks do not replace that CI.
### Screenshots
Not applicable.
### Downstream repositories
The diff adds an opt-in system-package value and its
implementation/tests. It does not change app schemas, platform variants,
node prerequisites, shared tooling contracts or an existing documented
workaround.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(linstor): add an opt-in Talos satellite preStop hook that releases unused Secondary DRBD resources without force. Enable it with talos.gracefulShutdown.enabled for sequential node maintenance; existing installations keep their current behavior by default.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added optional graceful shutdown support for LINSTOR satellites
running on Talos.
* When enabled, unused Secondary DRBD resources are released before
termination, with safeguards for active resources and bounded shutdown
timing.
* Added a 90-second termination grace period and configuration
validation for the new setting.
* Added documentation covering configuration, operational requirements,
and shutdown behavior.
* **Tests**
* Added Helm and Python coverage for shutdown behavior, validation,
timeouts, and failure scenarios.
* The test target now runs both Helm and Python unit tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
drbd_stall_act disconnects one peer with "drbdadm disconnect res:peer"
but then runs "drbdadm connect res", which walks every connection of
the resource. The peers that are still connected make drbdsetup fail
with "Device has a net-config (use disconnect first)" (exit code 10),
the helper only logs "WARN: action failed" and the stalled peer stays
disconnected.
Reconnect the same res:peer pair that was just disconnected.
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
DRBD auto-promotes a resource to Primary during the provisioning path
(e.g. mkfs.ext4 against a freshly created replica), but auto-demote
sometimes fails to fire on close. The resource then stays Primary on
a node that has no consumer, and CSI on the legitimate target node
fails NodePublishVolume with "failed to set source device readwrite".
The only recovery is an explicit drbdadm secondary on the stuck node.
Add a per-satellite plunger handler that detects local Primaries
where every device has open == false and demotes them. Both the
detection and the action are local to the satellite, so no peer
coordination is needed. open:no makes drbdadm secondary safe; if a
real consumer is about to arrive, auto-promote will redo it.
Upstream tracker: piraeusdatastore/piraeus-operator#942
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Add patches for piraeus-server:
- adjust-on-resfile-change: handle resource file changes
- allow-toggle-disk-retry: enable disk operation retries
- force-metadata-check-on-disk-add: verify metadata on disk addition
Update plunger-satellite script and values.yaml for new build.
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Sometimes DRBD devices get stuck in "Connecting" state, probably due to some
race conditions. This scriptlet provides a workaround for such situations.
Signed-off-by: Denis Seleznev <kto.3decb@gmail.com>
Do not restart the container if something goes wrong while plunger tries
to fix the linstor state.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved error notifications during system operations to ensure any
issues are promptly reported.
- **Documentation**
- Updated descriptive information on secondary volume management to help
clarify operational conditions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: Denis Seleznev <kto.3decb@gmail.com>
I just noticed that many devices are still attached to `/`, they are not
deleted but still blocking DRBD operations
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved detection of inactive loop devices by broadening the
filtering criteria, ensuring more accurate identification in various
scenarios.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
The Linstor satellite creates problems with admin privileges, so the
plunger needs the same privileges to fix those problems.
Also, use the native `losetup`. The Linstor image has a wrapper with an
additional function that we do not need here.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Refactor**
- Improved the management of unused loop devices with clearer feedback
and refined error handling.
- **New Features**
- Enhanced container configuration by adding elevated system
permissions, allowing the container to perform higher-level operations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced new management scripts for automating controller
maintenance tasks and satellite operations, including automatic cleanup
and reconnection of resources.
- Added dynamic configuration templates to streamline container image
management and refine deployment settings for enhanced security and
monitoring.
- Rolled out updated configurations for satellite deployments, improving
network performance and resource handling.
- New YAML configurations for Linstor satellites have been added,
enhancing TLS management and pod specifications.
- **Revert**
- Removed legacy configuration resources to consolidate and simplify
system management.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Aleksei Sviridkin