Aleksei Sviridkin
a3470391e2
## What this PR does Restoring a PVC from a LINSTOR VolumeSnapshot can leave one replica with an empty volume that DRBD marks UpToDate. A pod that lands on that replica sees an empty filesystem. linstor-csi restores the snapshot on one node and autoplaces the other replicas. LINSTOR v1.33.3 copies `Linstor/Drbd/InitialUptoDateOn` from the source volume into the restored volume definition and leaves the definition without `DRBD_INITIALIZED`. When autoplace puts a replica on the node that name points to, that empty new volume wins the initial UpToDate race, gets a random current UUID and a fresh filesystem, and goes StandAlone against the restored replica. LINBIT fixed this in v1.34.0. This PR backports that commit, LINBIT/linstor-server@9d8cdf5, to our v1.33.3 build as a patch. The README entry says to drop the patch with the move to v1.34.0 or later. Tested on a three-node cluster with a thick ZFS pool and `placementCount: 3`. Before the change the reproduction failed every time. With the patch I ran 5 restores, 4 of them with the source volume's winner on a different node than the restore. I also restored a snapshot taken before the change, and a 10Gi volume. Every time all replicas came up connected and UpToDate, a pod on each node read the same checksum, and the backing zvols were byte-identical when read with `O_DIRECT`. The resync after a restore moves about the written data, around 40 s for 1Gi and 134 s for 7.5 GiB on 10Gi. CSI clone PVC-to-PVC was checked too and is not affected. There is no e2e test. Every e2e lane runs without DRBD (`COZY_LINSTOR_DRBD_ENABLED=false`, StorageClass `local`), so a suite for replicated restores would never run in CI. Volumes restored before this fix are not repaired by it. How to find and fix them is in [this comment](https://github.com/cozystack/cozystack/issues/4862#issuecomment-6106661441). Known limitation, shared by upstream v1.34 since that code did not change: the autoplaced replicas skip the initial sync and rely on the source volume's day0 UUID still being in the restored replica's history. If that ever fails, DRBD goes StandAlone with "unrelated data" instead of syncing, which is visible, not silent. I could not reproduce it. Closes #4862 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(linstor): a PVC restored from a VolumeSnapshot no longer gets a replica with an empty volume marked UpToDate. Volumes restored before this release may still have such a replica; https://github.com/cozystack/cozystack/issues/4862#issuecomment-6106661441 explains how to check and repair them. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Restored snapshots with diskful DRBD resources are now recognized as initialized by default, helping them resume without unnecessary metadata recreation. * When force-initial-sync is enabled, restored volumes continue through the existing metadata-recreation path. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Talos satellite graceful shutdown
Cozystack's Talos configuration removes the systemd drbd-shutdown-guard. The optional satellite preStop hook releases unused Secondary DRBD resources so their backing devices, including ZFS zvols, can be released during normal shutdown.
| Parameter | Default | Effect |
|---|---|---|
talos.gracefulShutdown.enabled |
false |
Add the satellite preStop hook when both talos.enabled and drbd.enabled are also true. |
The default satellite lifecycle is unchanged. To enable the hook, set these values for the linstor component of the cozystack.linstor package:
talos:
gracefulShutdown:
enabled: true
Keep the setting in the package's declarative configuration at spec.components.linstor.values.talos.gracefulShutdown.enabled. The LINSTOR Helm chart embeds hack/satellite-pre-stop.py in LinstorSatelliteConfiguration/cozystack-talos; Piraeus converts its podTemplate to a strategic merge patch for each selected satellite DaemonSet. Package reconciliation and upgrades therefore retain the hook while the value remains enabled. Editing a generated Pod or DaemonSet directly does not persist through reconciliation. The hook uses Python and drbdsetup already supplied by the Cozystack satellite image.
The hook runs on every graceful satellite container stop, including ordinary pod restarts. It skips all Primary resources and any resource with an open device, rechecks each candidate, then calls drbdsetup down without force. Unknown or malformed status, command failure and timeout stop further actions. It does not demote resources, call the Kubernetes API, or change access permissions.
Enable it only when satellite and node restarts are performed sequentially, with replication and quorum checked between nodes. Piraeus creates a separate DaemonSet for each node; this chart does not serialize their rollouts. A shared configuration or image update can therefore stop unused Secondary replicas on several nodes at once and pause I/O until they return. Changing or disabling the setting can trigger a pod rollout, and the old pod still runs its existing hook.
Drain workloads before a planned node shutdown. The hook has a 60-second work budget, with each status command limited to 4 seconds and each down command to 10 seconds, and requests a 90-second pod termination grace period. The kubelet's node-shutdown budget can terminate the hook earlier; the pod grace period does not override that budget. Forced reboot, failed kubelet shutdown, still-open devices and unfinished replication are not fixed by this hook.
Related upstream discussion: Piraeus orderly shutdown on Talos.
Run make test in this directory with Helm, helm-unittest and Python 3.11 or later. The tests cover rendering with each feature flag, resource selection and rechecks, bounded errors and timeouts, dry-run behavior, and logging in the shared pod PID namespace.