fix(linstor): update piraeus-operator to v2.12.1 (#5166)

## What this PR does

Updates the LINSTOR stack to the current stable piraeus-operator
release: operator v2.12.1, LINSTOR server 1.35.3 and LINSTOR CSI
v1.13.2, the last two built here with our patches. This started as a fix
for the NFS server image hanging up the host console on bare-metal
storage nodes (Ubuntu 24.04, Cozystack 1.6.5 and 1.7.0). That needed
nfs-server v1.10.6 or later. Operators also asked for per-volume IO
limits, which need linstor-csi v1.11.3 or later and piraeus-operator
v2.10.8 or later, so the update goes to the current release rather than
the minimum.

The first commit fixes a problem the update exposed. The operator runs
the LINSTOR database migration in a `run-migration` init container, and
this package pinned only the main controller container to our build.
With the operator update, that init container ran the operator's default
LINSTOR 1.35.3 and migrated the database before the controller image
changed. The 1.33.3 controller then crash-looped until the linstor
release caught up. Both init containers that run the LINSTOR server
image (`run-migration` and the satellite's `setup-lvm-configuration`)
now use the platform image, with a test pinning it. This protects later
upgrades where the operator default is newer than our build. It does not
help this upgrade: the pin ships in the linstor release, and Flux
reconciles piraeus-operator first.

Patches:

- linstor-csi: the Protocol C override for dual attach is dropped. Since
v1.12 the driver asks LINSTOR to manage dual-primary, and LINSTOR 1.35
sets protocol C on the connections that need it. Upstream closed the
same change as superseded (piraeusdatastore/linstor-csi#435). The
relocate-after-clone patch is carried forward.
- LINSTOR server: six of the nine patches are in upstream by 1.35.3 and
are removed. `fix-k8s-create-already-exists`,
`retry-adjust-after-stale-bitmap` and `native-protoc` are carried
forward. LINSTOR 1.34 requires Java 21, so the image moves from bookworm
to trixie.

Behaviour that changes for users:

- No downgrade. LINSTOR 1.35 migrates the database and drops the `SEC_*`
tables, so a cluster that has run this version cannot go back to 1.33.
- The LINSTOR controller is down for a short time during this upgrade.
piraeus-operator is reconciled before the linstor release, so the
database is migrated to 1.35 while the old controller still runs. That
controller crash-loops until the linstor release rolls out, and if the
rollout fails there is no way back.
- A volume on a Protocol A or B resource group that was attached to two
nodes while the CSI driver was updated (a live migration in progress)
keeps the Protocol C override the old driver set on its resource
definition. The new driver does not remove it. Data is not affected, the
resource just stays on protocol C. Such resource definitions have the
`Aux/csi-protocol-override` property. To remove the override, run
`linstor resource-definition set-property <rd> DrbdOptions/Net/protocol`
and `linstor resource-definition set-property --aux <rd>
csi-protocol-override`.
- The CSIDriver moves to `fsGroupPolicy: File`, so `fsGroup` is also
applied to ReadWriteMany filesystem volumes.
- The operator manages Secrets only in its own namespace, and the CSI
controller loses cluster-wide read access to Secrets. Nothing in this
repository references snapshotter Secrets.
- The csi-health-monitor sidecar is removed.
- LINSTOR's default evict timeout goes from 1h to 12h; this package does
not set it.
- Volume group snapshots turn on only when
`groupsnapshot.storage.k8s.io/v1` is served. The snapshot controller
here still serves v1alpha1, so they stay off; #5239 changes that.
- IO limits can be set through `qos.linbit.com/{rbps,wbps,riops,wiops}`
StorageClass parameters. Enforcing them needs NRI enabled on the nodes
and the nri-volume-qos plugin, neither of which this PR adds.

Tested on a three-node Talos cluster, upgrading from the current main
packages to this branch:

- LINSTOR database migrated to 1.35.3, no faulty resources, no error
reports.
- Data written before the upgrade is intact, checked by checksum.
- A snapshot taken before the upgrade restores, and all three replicas
come up `UpToDate`; cloning and new snapshots work.
- KubeVirt live migration works for a ReplicatedBlock volume, including
one on a Protocol A resource group: LINSTOR switches the connection to
protocol C for the migration and restores A afterwards.
- A StorageClass with QoS parameters provisions, and the limits reach
the resource definition and the VolumeAttachment metadata.
- ZFS userspace in the satellite is now 2.4.4, matching the kernel
module.

Not tested: an arm64 build.

### Screenshots

No UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(linstor): update piraeus-operator to v2.12.1, LINSTOR to 1.35.3 and LINSTOR CSI to v1.13.2. Fixes the NFS server breaking console login on storage nodes and adds IO QoS StorageClass parameters. LINSTOR 1.35 migrates its database, and a downgrade to an earlier version is not possible after the upgrade.
```
This commit is contained in:
Aleksei Sviridkin authored and GitHub committed 2026-11-13 15:15:43 +03:00
commit 900baf2215
32 files changed
+511 -4252

No files matched your search

+2 -2
View File
@@ -4,11 +4,11 @@ export NAMESPACE=cozy-$(NAME)
include ../../../hack/common-envs.mk
include ../../../hack/package.mk
LINSTOR_VERSION ?= 1.33.3
LINSTOR_VERSION ?= 1.35.3
# The golden-image clone guard in packages/apps/vm-disk/templates/dv.yaml
# reads allowRemoteVolumeAccess and layerList the way this release does and
# has to be re-checked against the new one.
LINSTOR_CSI_VERSION ?= v1.11.2
LINSTOR_CSI_VERSION ?= v1.13.2
test:
helm unittest .
@@ -1,8 +1,8 @@
FROM --platform=$BUILDPLATFORM golang:1.27@sha256:3680233e3204827fbdc66088528ae6d4b3d034f51d03a99d454f6de034888244 AS builder
ARG GOPROXY
ARG VERSION=v1.11.2
ARG LINSTOR_WAIT_UNTIL_VERSION=v0.3.1
ARG VERSION=v1.13.2
ARG LINSTOR_WAIT_UNTIL_VERSION=v1.0.1
ARG TARGETARCH
ARG TARGETOS
@@ -15,13 +15,6 @@ RUN git apply /patches/*.diff
RUN for attempt in 1 2 3; do go mod download && break; [ "$attempt" = 3 ] && exit 1; sleep $((attempt * 10)); done
# Upstream pins grpc v1.81.0, which uses http2.TrailerPrefix; on Go 1.27
# golang.org/x/net builds http2 on top of net/http and no longer defines it.
# v1.83.2 also carries the fix for GO-2026-6443, a server panic reachable
# through the CSI gRPC endpoint.
RUN go get google.golang.org/grpc@v1.83.2 && \
go mod tidy
# go build never compiles a _test.go, so nothing else runs the tests of the
# packages the patches touch. Each package is tested from its own directory:
# that finds the module it belongs to, and an empty list runs nothing.
@@ -1,38 +1,37 @@
diff --git a/pkg/client/linstor.go b/pkg/client/linstor.go
index ac4651d..46d565d 100644
--- a/pkg/client/linstor.go
+++ b/pkg/client/linstor.go
@@ -462,6 +462,14 @@ func (s *Linstor) Clone(ctx context.Context, vol, src *volume.Info, params *volu
@@ -498,6 +498,14 @@
return err
}
+ if params.RelocateAfterClone {
+ logger.Debug("relocate resources to optimal nodes")
+
+ if err := s.relocateResources(ctx, vol.ID, rGroup.Name); err != nil {
+ if err := s.relocateResources(ctx, vol.ResourceName, rGroup.Name); err != nil {
+ logger.WithError(err).Warn("resource relocation failed, volume is still usable")
+ }
+ }
+
logger.Debug("reconcile extra properties")
err = s.client.ResourceDefinitions.Modify(ctx, vol.ID, lapi.GenericPropsModify{OverrideProps: vol.Properties})
@@ -1297,6 +1305,14 @@ func (s *Linstor) VolFromSnap(ctx context.Context, snap *volume.Snapshot, vol *v
err = s.client.ResourceDefinitions.Modify(ctx, vol.ResourceName, lapi.GenericPropsModify{OverrideProps: vol.Properties})
@@ -1614,6 +1622,14 @@
return err
}
+ if snapParams != nil && snapParams.RelocateAfterRestore {
+ logger.Debug("relocate resources to optimal nodes")
+
+ if err := s.relocateResources(ctx, vol.ID, rGroup.Name); err != nil {
+ if err := s.relocateResources(ctx, vol.ResourceName, rGroup.Name); err != nil {
+ logger.WithError(err).Warn("resource relocation failed, volume is still usable")
+ }
+ }
+
logger.Debug("reconcile extra properties")
err = s.client.ResourceDefinitions.Modify(ctx, vol.ID, lapi.GenericPropsModify{OverrideProps: vol.Properties})
@@ -1487,9 +1503,8 @@ func (s *Linstor) reconcileSnapshotResources(ctx context.Context, snapshot *volu
err = s.client.ResourceDefinitions.Modify(ctx, vol.ResourceName, lapi.GenericPropsModify{OverrideProps: vol.Properties})
@@ -1818,9 +1834,8 @@
return fmt.Errorf("snapshot '%s' not deployed on any node", snap.Name)
}
@@ -44,7 +43,7 @@ index ac4651d..46d565d 100644
for _, snapNode := range snap.Nodes {
if err := s.NodeAvailable(ctx, snapNode); err != nil {
logger.WithField("selected node candidate", snapNode).WithError(err).Debug("node is not available")
@@ -1497,13 +1512,23 @@ func (s *Linstor) reconcileSnapshotResources(ctx context.Context, snapshot *volu
@@ -1828,15 +1843,25 @@
}
if slices.Contains(preferredNodes, snapNode) {
@@ -58,8 +57,8 @@ index ac4651d..46d565d 100644
}
+
+ available = append(available, snapNode)
+ }
+
}
+ // Pick a random node from preferred candidates, falling back to any available node.
+ // Randomization distributes restore load across snapshot nodes, preventing all
+ // clones of the same source from concentrating on a single node.
@@ -71,13 +70,20 @@ index ac4651d..46d565d 100644
+ var selectedNode string
+ if len(candidates) > 0 {
+ selectedNode = candidates[rand.Intn(len(candidates))]
}
+ }
+
if selectedNode == "" {
@@ -1696,6 +1721,114 @@ func (s *Linstor) reconcileResourcePlacement(ctx context.Context, vol *volume.In
return nil
}
return fmt.Errorf("no node [%s] available for snapshot '%s'", strings.Join(snap.Nodes, ","), snap.Name)
}
@@ -2030,6 +2055,114 @@
err = volumeScheduler.Create(ctx, vol.ResourceName, params, topologies)
if err != nil {
return err
+ }
+
+ return nil
+}
+
+const propertyRelocationTriggered = "Aux/csi-relocation-triggered"
+
+// relocateResources migrates replicas from their current nodes to the nodes that
@@ -181,19 +187,13 @@ index ac4651d..46d565d 100644
+ logger.WithError(err).Warn("migrate-disk failed, skipping this pair")
+ continue
+ }
+ }
+
+ return nil
+}
+
// FindSnapsByID searches the snapshot in the backend
func (s *Linstor) FindSnapsByID(ctx context.Context, id string) ([]*volume.Snapshot, error) {
snapshotId, err := volume.ParseSnapshotId(id)
}
return nil
diff --git a/pkg/volume/parameter.go b/pkg/volume/parameter.go
index 39acd95..54d1dfb 100644
--- a/pkg/volume/parameter.go
+++ b/pkg/volume/parameter.go
@@ -50,6 +50,7 @@ const (
@@ -51,6 +51,7 @@
nfsservicename
nfssquash
nfsrecoveryvolumesize
@@ -201,24 +201,24 @@ index 39acd95..54d1dfb 100644
)
// Parameters configuration for linstor volumes.
@@ -118,6 +119,8 @@ type Parameters struct {
// NfsRecoveryVolumeSize sets the volume size (in bytes) of the recovery volume used by the NFS server.
// Defaults to 300MiB.
NfsRecoveryVolumeBytes int64
@@ -122,6 +123,8 @@
// IOQoSLimits holds the IO QoS limits parsed from the qos.linbit.com/* parameters. They are stored on
// the LINSTOR resource definition and passed to the nri-volume-qos plugin via VolumeAttachment metadata.
IOQoSLimits meta.Limits
+ // RelocateAfterClone triggers asynchronous relocation of replicas to optimal nodes after cloning.
+ RelocateAfterClone bool
}
const DefaultDisklessStoragePoolName = "DfltDisklessStorPool"
@@ -140,6 +143,7 @@ func NewParameters(params map[string]string, topologyPrefix string) (Parameters,
@@ -144,6 +147,7 @@
NfsServiceName: "linstor-csi-nfs",
NfsSquash: "no_root_squash",
NfsRecoveryVolumeBytes: 300 * 1024 * 1024,
+ RelocateAfterClone: true,
}
for k, v := range params {
@@ -287,6 +291,13 @@ func NewParameters(params map[string]string, topologyPrefix string) (Parameters,
limits, err := meta.ParseLimits(params)
@@ -298,6 +302,13 @@
}
p.NfsRecoveryVolumeBytes = s.Value()
@@ -233,10 +233,9 @@ index 39acd95..54d1dfb 100644
}
diff --git a/pkg/volume/paramkey_enumer.go b/pkg/volume/paramkey_enumer.go
index 5474963..2eb8a7c 100644
--- a/pkg/volume/paramkey_enumer.go
+++ b/pkg/volume/paramkey_enumer.go
@@ -7,11 +7,11 @@ import (
@@ -7,11 +7,11 @@
"strings"
)
@@ -251,7 +250,7 @@ index 5474963..2eb8a7c 100644
func (i paramKey) String() string {
if i < 0 || i >= paramKey(len(_paramKeyIndex)-1) {
@@ -50,9 +50,10 @@ func _paramKeyNoOp() {
@@ -50,9 +50,10 @@
_ = x[nfsservicename-(23)]
_ = x[nfssquash-(24)]
_ = x[nfsrecoveryvolumesize-(25)]
@@ -262,17 +261,24 @@ index 5474963..2eb8a7c 100644
+var _paramKeyValues = []paramKey{allowremotevolumeaccess, autoplace, clientlist, disklessonremaining, disklessstoragepool, donotplacewithregex, encryption, fsopts, layerlist, mountopts, nodelist, placementcount, placementpolicy, replicasondifferent, replicasonsame, sizekib, storagepool, postmountxfsopts, resourcegroup, usepvcname, overprovision, xreplicasondifferent, nfsconfigtemplate, nfsservicename, nfssquash, nfsrecoveryvolumesize, relocateafterclone}
var _paramKeyNameToValueMap = map[string]paramKey{
_paramKeyName[0:23]: allowremotevolumeaccess,
@@ -107,6 +108,8 @@ var _paramKeyNameToValueMap = map[string]paramKey{
_paramKeyName[0:23]: allowremotevolumeaccess,
@@ -81,6 +82,7 @@
_paramKeyName[310:324]: nfsservicename,
_paramKeyName[324:333]: nfssquash,
_paramKeyName[333:354]: nfsrecoveryvolumesize,
+ _paramKeyName[354:372]: relocateafterclone,
}
var _paramKeyLowerNameToValueMap = map[string]paramKey{
@@ -110,6 +112,7 @@
_paramKeyLowerName[310:324]: nfsservicename,
_paramKeyLowerName[324:333]: nfssquash,
_paramKeyName[333:354]: nfsrecoveryvolumesize,
_paramKeyLowerName[333:354]: nfsrecoveryvolumesize,
+ _paramKeyName[354:372]: relocateafterclone,
+ _paramKeyLowerName[354:372]: relocateafterclone,
}
var _paramKeyNames = []string{
@@ -136,6 +139,7 @@ var _paramKeyNames = []string{
@@ -139,6 +142,7 @@
_paramKeyName[310:324],
_paramKeyName[324:333],
_paramKeyName[333:354],
@@ -281,18 +287,17 @@ index 5474963..2eb8a7c 100644
// paramKeyString retrieves an enum value from the enum constants string name.
diff --git a/pkg/volume/snapshot_params.go b/pkg/volume/snapshot_params.go
index d167cb8..50b70fb 100644
--- a/pkg/volume/snapshot_params.go
+++ b/pkg/volume/snapshot_params.go
@@ -35,6 +35,7 @@ type SnapshotParameters struct {
LinstorTargetUrl string `json:"linstor-target-url,omitempty"`
LinstorTargetClusterID string `json:"linstor-target-cluster-id,omitempty"`
LinstorTargetStoragePool string `json:"linstor-target-storage-pool,omitempty"`
+ RelocateAfterRestore bool `json:"relocate-after-restore,omitempty"`
@@ -38,6 +38,7 @@
LinstorTargetUrl string `json:"linstor-target-url,omitempty"`
LinstorTargetClusterID string `json:"linstor-target-cluster-id,omitempty"`
LinstorTargetStoragePool string `json:"linstor-target-storage-pool,omitempty"`
+ RelocateAfterRestore bool `json:"relocate-after-restore,omitempty"`
}
func NewSnapshotParameters(params, secrets map[string]string) (*SnapshotParameters, error) {
@@ -91,6 +92,13 @@ func NewSnapshotParameters(params, secrets map[string]string) (*SnapshotParamete
@@ -116,6 +117,13 @@
p.LinstorTargetClusterID = v
case "/linstor-target-storage-pool":
p.LinstorTargetStoragePool = v
@@ -1,132 +0,0 @@
diff --git a/pkg/client/linstor.go b/pkg/client/linstor.go
index ac4651d..9d75c79 100644
--- a/pkg/client/linstor.go
+++ b/pkg/client/linstor.go
@@ -653,11 +653,35 @@ func (s *Linstor) Attach(ctx context.Context, volId, node string, rwxBlock bool)
}
if otherResInUse > 0 && rwxBlock {
- rdPropsModify := lapi.GenericPropsModify{OverrideProps: map[string]string{
+ rdProps := map[string]string{
linstor.PropertyAllowTwoPrimaries: "yes",
- }}
+ }
- err = s.client.ResourceDefinitions.Modify(ctx, volId, rdPropsModify)
+ // DRBD requires Protocol C whenever allow-two-primaries is enabled.
+ // If the resource is configured with Protocol A or B (typically
+ // through its resource-group), drbdadm adjust on the satellites
+ // fails with "Protocol C required" (errno 139) and live migration
+ // cannot proceed. Override Protocol to C on the resource-definition
+ // itself so it applies to every connection (including diskless
+ // peers, e.g. a TieBreaker). The marker lets Detach revert exactly
+ // the override we installed without disturbing operator-set props.
+ proto, protoErr := s.getEffectiveDrbdProtocol(ctx, volId)
+ if protoErr != nil {
+ s.log.WithError(protoErr).WithField("volume", volId).
+ Warn("failed to determine effective DRBD protocol; skipping Protocol=C override for dual-attach")
+ } else if proto == "A" || proto == "B" {
+ s.log.WithFields(logrus.Fields{
+ "volume": volId,
+ "protocol": proto,
+ }).Info("installing Protocol=C override on resource-definition for dual-attach")
+
+ rdProps[linstor.PropertyDrbdNetProtocol] = "C"
+ rdProps[linstor.PropertyCsiProtocolOverride] = "yes"
+ }
+
+ err = s.client.ResourceDefinitions.Modify(ctx, volId, lapi.GenericPropsModify{
+ OverrideProps: rdProps,
+ })
if err != nil {
return "", err
}
@@ -774,11 +798,26 @@ func (s *Linstor) Detach(ctx context.Context, volId, node string) error {
}
if resInUse == 1 {
- rdPropsModify := lapi.GenericPropsModify{DeleteProps: []string{
- linstor.PropertyAllowTwoPrimaries,
- }}
+ deleteProps := []string{linstor.PropertyAllowTwoPrimaries}
+
+ // Drop the Protocol=C override only if Attach installed it (gated by
+ // our marker), so an operator-set Protocol property on the
+ // resource-definition is preserved.
+ rd, getErr := s.client.ResourceDefinitions.Get(ctx, volId)
+ if getErr != nil {
+ log.WithError(getErr).Warn("failed to fetch resource-definition props; skipping Protocol=C override removal")
+ } else if rd.Props[linstor.PropertyCsiProtocolOverride] != "" {
+ log.Info("removing Protocol=C override from resource-definition")
- err = s.client.ResourceDefinitions.Modify(ctx, volId, rdPropsModify)
+ deleteProps = append(deleteProps,
+ linstor.PropertyDrbdNetProtocol,
+ linstor.PropertyCsiProtocolOverride,
+ )
+ }
+
+ err = s.client.ResourceDefinitions.Modify(ctx, volId, lapi.GenericPropsModify{
+ DeleteProps: deleteProps,
+ })
if err != nil {
return nil404(err)
}
@@ -805,6 +844,33 @@ func (s *Linstor) Detach(ctx context.Context, volId, node string) error {
return nil404(s.client.Resources.Delete(ctx, volId, node))
}
+// getEffectiveDrbdProtocol returns the DRBD network protocol ("A", "B" or "C")
+// that LINSTOR will hand to drbdadm for the given resource. Resource-definition
+// properties take precedence; otherwise the value is inherited from the
+// resource-group. An empty string is returned when neither level sets the
+// property (LINSTOR's compiled-in default is "C").
+func (s *Linstor) getEffectiveDrbdProtocol(ctx context.Context, volId string) (string, error) {
+ rd, err := s.client.ResourceDefinitions.Get(ctx, volId)
+ if err != nil {
+ return "", err
+ }
+
+ if v, ok := rd.Props[linstor.PropertyDrbdNetProtocol]; ok {
+ return v, nil
+ }
+
+ if rd.ResourceGroupName == "" {
+ return "", nil
+ }
+
+ rg, err := s.client.ResourceGroups.Get(ctx, rd.ResourceGroupName)
+ if err != nil {
+ return "", err
+ }
+
+ return rg.Props[linstor.PropertyDrbdNetProtocol], nil
+}
+
// CapacityBytes returns the amount of free space in the storage pool specified by the params and topology.
func (s *Linstor) CapacityBytes(ctx context.Context, storagePools []string, overProvision *float64, segments map[string]string) (int64, error) {
log := s.log.WithField("storage-pools", storagePools).WithField("segments", segments)
diff --git a/pkg/linstor/const.go b/pkg/linstor/const.go
index 9a5f79c..c8bc9c3 100644
--- a/pkg/linstor/const.go
+++ b/pkg/linstor/const.go
@@ -44,6 +44,19 @@ const (
// PropertyAllowTwoPrimaries is DRBD option to allow second primary. Mainly used for live-migration.
PropertyAllowTwoPrimaries = lc.NamespcDrbdNetOptions + "/allow-two-primaries"
+ // PropertyDrbdNetProtocol is the DRBD network replication protocol option
+ // (A = async, B = semi-sync, C = sync). DRBD requires Protocol C whenever
+ // allow-two-primaries is enabled, otherwise drbdadm adjust fails with
+ // "Protocol C required" (errno 139).
+ PropertyDrbdNetProtocol = lc.NamespcDrbdNetOptions + "/protocol"
+
+ // PropertyCsiProtocolOverride marks a resource-connection where this driver
+ // has installed a temporary Protocol=C override to make a Protocol-A/B
+ // resource compatible with allow-two-primaries during live migration. The
+ // marker lets us safely remove only the overrides we set, without touching
+ // connection-level overrides installed by the operator.
+ PropertyCsiProtocolOverride = lc.NamespcAuxiliary + "/csi-protocol-override"
+
// CreatedForTemporaryDisklessAttach marks a resource as temporary, i.e. it should be removed after it is no longer
// needed.
CreatedForTemporaryDisklessAttach = "temporary-diskless-attach"
@@ -1,9 +1,10 @@
ARG DISTRO=bookworm@sha256:f37a335e82bca302e955fa39f9dfe28f1be618f016f8a2b56318e5a5111afc26
# LINSTOR 1.35 builds and runs on Java 21, which bookworm does not ship.
ARG DISTRO=trixie@sha256:9c6e381421d0f48d1b8133e573f33c185ee16cbf505eda9c4a8a4791b25ea8a6
ARG LINSTOR_VERSION
# ------------------------------------------------------------------------------
# Build linstor-server from source
FROM --platform=$BUILDPLATFORM debian:bookworm@sha256:f37a335e82bca302e955fa39f9dfe28f1be618f016f8a2b56318e5a5111afc26 AS builder
FROM --platform=$BUILDPLATFORM debian:trixie@sha256:9c6e381421d0f48d1b8133e573f33c185ee16cbf505eda9c4a8a4791b25ea8a6 AS builder
ARG LINSTOR_VERSION
ARG VERSION=${LINSTOR_VERSION}
@@ -13,10 +14,10 @@ ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update \
&& apt-get -y upgrade \
&& apt-get -y install build-essential git default-jdk-headless python3-all debhelper wget unzip && \
wget https://services.gradle.org/distributions/gradle-8.9-bin.zip -O /tmp/gradle.zip && \
wget https://services.gradle.org/distributions/gradle-8.14.2-bin.zip -O /tmp/gradle.zip && \
unzip -d /opt /tmp/gradle.zip && \
rm /tmp/gradle.zip && \
ln -s /opt/gradle-8.9/bin/gradle /usr/local/bin/gradle
ln -s /opt/gradle-8.14.2/bin/gradle /usr/local/bin/gradle
RUN git clone https://github.com/LINBIT/linstor-server.git /linstor-server
WORKDIR /linstor-server
@@ -1,36 +1,22 @@
# LINSTOR Server Patches
Custom patches for piraeus-server (linstor-server) v1.33.3.
Custom patches for piraeus-server (linstor-server) v1.35.3.
- **allow-toggle-disk-retry.diff** — Backport maintainer implementation of toggle-disk retry/abort
- Source PR/comment: [#475](https://github.com/LINBIT/linstor-server/pull/475), [maintainer note](https://github.com/LINBIT/linstor-server/pull/475#issuecomment-3949630419)
- Backported from upstream commit: [`3d97f71c9`](https://github.com/LINBIT/linstor-server/commit/3d97f71c95a493588d3d521c63eac4d846935fb3)
- **fix-duplicate-tcp-ports.diff** — Preserve DRBD TCP ports during toggle-disk and avoid redundant `ensureStackDataExists()`
- Source PR/review: [#476](https://github.com/LINBIT/linstor-server/pull/476), [review suggestion](https://github.com/LINBIT/linstor-server/pull/476#discussion_r3007725079)
- Backported from commits: [`79d6375c5`](https://github.com/kvaps/linstor-server/commit/79d6375c55d6181b35a7b7f0fe8dbdfb86e126cd), [`bcc89902f`](https://github.com/kvaps/linstor-server/commit/bcc89902f4f61ac1589dd07ebb7f5aae1935370d)
- **fix-luks-header-size.diff** — Account for LUKS2 `--offset`, metadata/keyslots sizing, and device `optimal_io_size`
- Source PR/comment: [#472](https://github.com/LINBIT/linstor-server/pull/472), [maintainer note](https://github.com/LINBIT/linstor-server/pull/472#issuecomment-3949687603)
- Backported from commits: [`ccc85fbd2`](https://github.com/LINBIT/linstor-server/commit/ccc85fbd2c65f0b97c52403fa80f1efdb886ec4e), [`71b601554`](https://github.com/LINBIT/linstor-server/commit/71b601554f41bcb50cd5bd06989c5b0d3a814acd)
- Note: upstream commit [`3d0402a0c`](https://github.com/LINBIT/linstor-server/commit/3d0402a0c25f0a4b57b380321f10e89982f26e7a) is already included in `v1.33.1`
- **fix-k8s-create-already-exists.diff** — Adopt the existing object when a create in the Kubernetes CRD backend returns 409 AlreadyExists. The HTTP client retries a POST after a 5xx or a dropped connection, so a create the apiserver already persisted comes back as 409; a single-object transaction has no rollback entry, the key is dropped from memory while the CR stays, and every later write of that key fails until the controller restarts. Seen as a PV stuck in `Released` because linstor-csi could not set `BalanceResourcesEnabled` on the resource definition.
- Source PR: [#531](https://github.com/LINBIT/linstor-server/pull/531)
- Backported from commit: [`8badf8f05`](https://github.com/lexfrei/linstor-server/commit/8badf8f0526ff3b1b391857e4a046ed46b95a79c)
- **retry-adjust-after-stale-bitmap.diff** — Retry `drbdadm adjust` after detaching a stale local bitmap state
- Source PR: [#491](https://github.com/LINBIT/linstor-server/pull/491)
- Backported from commit: [`51ae50a84`](https://github.com/kvaps/linstor-server/commit/51ae50a84dcb98093f543b819652c750a94d96c9)
- **retry-secondary-after-mkfs.diff** — Retry `drbdadm secondary` after mkfs when DRBD reports the device as held open by an external probe (Talos block-controller, udev, multipathd, etc.). Without this retry, a transient `Device is held open by someone` aborts resource initialization, leaves the satellite in an intermediate state, and prevents the controller from receiving the final `UpToDate` event — orphan PVs in `Released` then cannot be cleaned up.
- Related upstream issue: [#268](https://github.com/LINBIT/linstor-server/issues/268)
- Related upstream issue: [drbd #74](https://github.com/LINBIT/drbd/issues/74) (same EBUSY pattern with multipathd)
- **fix-min-io-probe-device-wait.diff** — Wait for temporary probe devices before reading pool I/O properties; retry failures and fall back to a probe when ZFS volumes have no usable device path.
- Source PR: [#528](https://github.com/LINBIT/linstor-server/pull/528), fixes [#527](https://github.com/LINBIT/linstor-server/issues/527)
- Adapted for v1.33.3; inactive LVM volumes retain the existing no-probe behavior.
- Rebased onto v1.35.3, where `DrbdAdm` gained the `DFLT_WINDOWS_TIMEOUT_MS` constant; `DrbdadmAdjust` still receives the failed `OutputData` through `ExtCmdFailedException`, so its own retry on "Resize not allowed during resync" keeps working on top of this one.
- **native-protoc.diff** — Download the protoc build for the build host's architecture instead of always `linux-x86_64`. The builder stage runs on `$BUILDPLATFORM`, so on a native arm64 builder the x86_64 protoc cannot execute and `./gradlew generateJava` fails with `Syntax error: ")" unexpected`.
- Not proposed upstream yet.
- **snapshot-restore-drbd-initialized.diff** — Mark the volume definitions of a restored snapshot as DRBD-initialized, unless the restored resource definition carries `DrbdOptions/ForceInitialSyncPermanent`, which keeps the existing path that recreates metadata, as for a backup. Without the flag the initial UpToDate race is still open on a restored volume, and `Linstor/Drbd/InitialUptoDateOn`, copied from the source volume through the snapshot, names whichever node won it for the source. When linstor-csi autoplaces a replica on that node, the empty new volume wins the race, gets mkfs and goes StandAlone against the restored replica, so a pod attached there sees an empty filesystem. With the flag set, nothing wins the race, and an autoplaced replica resyncs from the restored one like any replica added to an existing volume.
- Backported from upstream commit: [`9d8cdf511`](https://github.com/LINBIT/linstor-server/commit/9d8cdf5118338fc72a34bb99799e452e031ea8f6), first released in v1.34.0; the `CHANGELOG.md` hunk is left out
- Drop when LINSTOR is upgraded to v1.34.0 or later
- On ZFS and thin pools the autoplaced replicas still skip the initial sync and rely on the source volume's day0 UUID being in the restored replica's history, upstream included; if it is not, DRBD reports unrelated data and goes StandAlone rather than syncing
- Fixes [cozystack/cozystack#4862](https://github.com/cozystack/cozystack/issues/4862)
Dropped on the move from v1.33.3, because v1.35.3 already carries them:
- `allow-toggle-disk-retry.diff` — upstream [`3d97f71c9`](https://github.com/LINBIT/linstor-server/commit/3d97f71c95a493588d3d521c63eac4d846935fb3), v1.34.0
- `fix-luks-header-size.diff` — upstream [`ccc85fbd2`](https://github.com/LINBIT/linstor-server/commit/ccc85fbd2c65f0b97c52403fa80f1efdb886ec4e) and [`71b601554`](https://github.com/LINBIT/linstor-server/commit/71b601554f41bcb50cd5bd06989c5b0d3a814acd), v1.34.0
- `snapshot-restore-drbd-initialized.diff` — upstream [`9d8cdf511`](https://github.com/LINBIT/linstor-server/commit/9d8cdf5118338fc72a34bb99799e452e031ea8f6), v1.34.0
- `fix-duplicate-tcp-ports.diff` — upstream [`521c6d211`](https://github.com/LINBIT/linstor-server/commit/521c6d21121dbbcc26cf6eda723c24a5b34889be) (from [#476](https://github.com/LINBIT/linstor-server/pull/476)), v1.34.0
- `retry-secondary-after-mkfs.diff` — upstream [`677789c02`](https://github.com/LINBIT/linstor-server/commit/677789c028ac), v1.34.0
- `fix-min-io-probe-device-wait.diff` — upstream [`6e5557839`](https://github.com/LINBIT/linstor-server/commit/6e5557839c911c7a1db240dcb6fc348383a74266) (from [#528](https://github.com/LINBIT/linstor-server/pull/528)), v1.35.1
@@ -1,261 +0,0 @@
diff --git a/controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlRscToggleDiskApiCallHandler.java b/controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlRscToggleDiskApiCallHandler.java
index d93a18014..a944cb809 100644
--- a/controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlRscToggleDiskApiCallHandler.java
+++ b/controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlRscToggleDiskApiCallHandler.java
@@ -111,6 +111,14 @@ import reactor.util.function.Tuple2;
@Singleton
public class CtrlRscToggleDiskApiCallHandler implements CtrlSatelliteConnectionListener
{
+ private enum ToggleDiskAction
+ {
+ NOOP,
+ NORMAL,
+ ABORT,
+ RETRY
+ }
+
private final AccessContext apiCtx;
private final ScopeRunner scopeRunner;
private final BackgroundRunner backgroundRunner;
@@ -386,69 +394,33 @@ public class CtrlRscToggleDiskApiCallHandler implements CtrlSatelliteConnectionL
"Toggle Disk on %s/%s %s", nodeNameStr, rscNameStr, removeDisk ? "removing disk" : "adding disk");
Resource rsc = ctrlApiDataLoader.loadRsc(nodeName, rscName, true);
+ ResourceDefinition rscDfn = rsc.getResourceDefinition();
- if (hasDiskAddRequested(rsc))
- {
- throw new ApiRcException(ApiCallRcImpl.simpleEntry(
- ApiConsts.FAIL_RSC_BUSY,
- "Addition of disk to resource already requested",
- true
- ));
- }
- if (hasDiskRemoveRequested(rsc))
- {
- throw new ApiRcException(ApiCallRcImpl.simpleEntry(
- ApiConsts.FAIL_RSC_BUSY,
- "Removal of disk from resource already requested",
- true
- ));
- }
+ ToggleDiskAction action = determineToggleDiskAction(rsc, removeDisk);
+ errorReporter.logDebug("Toggle Disk action: %s", action);
- if (!removeDisk && !ctrlVlmCrtApiHelper.isDiskless(rsc))
+ switch (action)
{
- throw new ApiRcException(ApiCallRcImpl.simpleEntry(
- ApiConsts.WARN_RSC_ALREADY_HAS_DISK,
- "Resource already has disk",
- true
- ));
- }
- if (removeDisk && ctrlVlmCrtApiHelper.isDiskless(rsc))
- {
- throw new ApiRcException(ApiCallRcImpl.simpleEntry(
- ApiConsts.WARN_RSC_ALREADY_DISKLESS,
- "Resource already diskless",
- true
- ));
+ case NOOP:
+ return handleNoopAction(rsc, removeDisk);
+ case RETRY:
+ return handleRetryAction(rsc, removeDisk, toggleIntoTiebreakerRef, context);
+ case ABORT:
+ clearToggleDiskFlags(rsc);
+ errorReporter.logInfo(
+ "Aborting previous toggle disk transition, starting new transition to %s",
+ removeDisk ? "diskless" : "diskful"
+ );
+ break;
+ case NORMAL:
+ break;
+ default:
+ throw new ImplementationError("Unhandled case: " + action);
}
- ResourceDefinition rscDfn = rsc.getResourceDefinition();
- AccessContext peerCtx = peerAccCtx.get();
- if (removeDisk)
- {
- // Prevent removal of the last disk
- int haveDiskCount = countDisksAndIsOnline(rscDfn);
- if (haveDiskCount <= 1)
- {
- throw new ApiRcException(ApiCallRcImpl.simpleEntry(
- ApiConsts.FAIL_INSUFFICIENT_REPLICA_COUNT,
- "Cannot remove the disk from the only online resource with a disk",
- true
- ));
- }
+ validateToggleDiskPreconditions(rsc, removeDisk);
- if (!LayerUtils.hasLayer(getLayerData(peerCtx, rsc), DeviceLayerKind.DRBD))
- {
- throw new ApiRcException(ApiCallRcImpl.simpleEntry(
- ApiConsts.FAIL_INVLD_LAYER_STACK,
- "Toggle disk is only supported in combination with DRBD",
- true
- ));
- }
- }
- else
- {
- ensureAllPeersHavePeerSlotLeft(rscDfn);
- }
+ AccessContext peerCtx = peerAccCtx.get();
// Save the requested storage pool in the resource properties.
// This does not cause the storage pool to be used automatically.
@@ -628,10 +600,10 @@ public class CtrlRscToggleDiskApiCallHandler implements CtrlSatelliteConnectionL
ctrlTransactionHelper.commit();
- String action = removeDisk ? "Removal of disk from" : "Addition of disk to";
+ String actionStr = removeDisk ? "Removal of disk from" : "Addition of disk to";
responses.addEntry(ApiCallRcImpl.simpleEntry(
ApiConsts.MODIFIED,
- action + " resource '" + rscDfn.getName().displayValue + "' " +
+ actionStr + " resource '" + rscDfn.getName().displayValue + "' " +
"on node '" + rsc.getNode().getName().displayValue + "' registered"
));
@@ -651,6 +623,40 @@ public class CtrlRscToggleDiskApiCallHandler implements CtrlSatelliteConnectionL
);
}
+ private Flux<ApiCallRc> handleNoopAction(Resource rscRef, boolean removeDiskRef)
+ {
+ String state = removeDiskRef ? "diskless" : "diskful";
+ ApiCallRcImpl responses = new ApiCallRcImpl();
+ responses.addEntry(ApiCallRcImpl.simpleEntry(
+ ApiConsts.INFO_NOOP,
+ "Resource '" + rscRef.getResourceDefinition().getName().displayValue + "' on node '" +
+ rscRef.getNode().getName().displayValue + "' is already " + state
+ ));
+ return Flux.just(responses);
+ }
+
+ private Flux<ApiCallRc> handleRetryAction(
+ Resource rscRef,
+ boolean removeDisk,
+ boolean toggleIntoTiebreakerRef,
+ ResponseContext context
+ )
+ {
+ String direction = removeDisk ? "diskless" : "diskful";
+ ApiCallRcImpl responses = new ApiCallRcImpl();
+ NodeName nodeName = rscRef.getNode().getName();
+ ResourceName rscName = rscRef.getResourceDefinition().getName();
+ responses.addEntry(ApiCallRcImpl.simpleEntry(
+ ApiConsts.INFO_NOOP,
+ "Retrying toggle disk to " + direction + " for resource '" + rscName.displayValue +
+ "' on node '" + nodeName.displayValue + "'"
+ ));
+
+ return Flux
+ .<ApiCallRc>just(responses)
+ .concatWith(updateAndAdjustDisk(nodeName, rscName, removeDisk, toggleIntoTiebreakerRef, context));
+ }
+
private long getVlmDfnSizePrivileged(VolumeDefinition vlmDfnRef)
{
try
@@ -781,6 +787,96 @@ public class CtrlRscToggleDiskApiCallHandler implements CtrlSatelliteConnectionL
}
}
+ private ToggleDiskAction determineToggleDiskAction(Resource rsc, boolean removeDisk)
+ {
+ boolean isDiskless = ctrlVlmCrtApiHelper.isDiskless(rsc);
+ boolean diskAddRequested = hasDiskAddRequested(rsc);
+ boolean diskRemoveRequested = hasDiskRemoveRequested(rsc);
+
+ ToggleDiskAction action;
+ if (isDiskless)
+ {
+ if (diskAddRequested)
+ {
+ action = removeDisk ? ToggleDiskAction.ABORT : ToggleDiskAction.RETRY;
+ }
+ else if (diskRemoveRequested)
+ {
+ action = removeDisk ? ToggleDiskAction.RETRY : ToggleDiskAction.ABORT;
+ }
+ else
+ {
+ action = removeDisk ? ToggleDiskAction.NOOP : ToggleDiskAction.NORMAL;
+ }
+ }
+ else
+ {
+ if (diskRemoveRequested)
+ {
+ action = removeDisk ? ToggleDiskAction.RETRY : ToggleDiskAction.ABORT;
+ }
+ else
+ {
+ action = removeDisk ? ToggleDiskAction.NORMAL : ToggleDiskAction.NOOP;
+ }
+ }
+
+ return action;
+ }
+
+ private void validateToggleDiskPreconditions(Resource rsc, boolean removeDisk)
+ {
+ ResourceDefinition rscDfn = rsc.getResourceDefinition();
+ if (removeDisk)
+ {
+ validateDiskRemovalAllowed(rsc, rscDfn);
+ }
+ else
+ {
+ ensureAllPeersHavePeerSlotLeft(rscDfn);
+ }
+ }
+
+ private void clearToggleDiskFlags(Resource rsc)
+ {
+ try
+ {
+ rsc.getStateFlags().disableFlags(
+ apiCtx,
+ Resource.Flags.DISK_ADD_REQUESTED,
+ Resource.Flags.DISK_ADDING,
+ Resource.Flags.DISK_REMOVE_REQUESTED,
+ Resource.Flags.DISK_REMOVING
+ );
+ }
+ catch (AccessDeniedException | DatabaseException exc)
+ {
+ throw new ImplementationError(exc);
+ }
+ }
+
+ private void validateDiskRemovalAllowed(Resource rsc, ResourceDefinition rscDfn)
+ {
+ int haveDiskCount = countDisksAndIsOnline(rscDfn);
+ if (haveDiskCount <= 1)
+ {
+ throw new ApiRcException(ApiCallRcImpl.simpleEntry(
+ ApiConsts.FAIL_INSUFFICIENT_REPLICA_COUNT,
+ "Cannot remove the disk from the only online resource with a disk",
+ true
+ ));
+ }
+
+ if (!LayerUtils.hasLayer(getLayerData(peerAccCtx.get(), rsc), DeviceLayerKind.DRBD))
+ {
+ throw new ApiRcException(ApiCallRcImpl.simpleEntry(
+ ApiConsts.FAIL_INVLD_LAYER_STACK,
+ "Toggle disk is only supported in combination with DRBD",
+ true
+ ));
+ }
+ }
+
/**
* Although we need to rebuild the layerData as the layerList might have changed, if we do not
* deactivate (i.e. down) the current resource, we need to make sure that deleting DrbdRscData
@@ -1,137 +0,0 @@
diff --git a/controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlRscToggleDiskApiCallHandler.java b/controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlRscToggleDiskApiCallHandler.java
index d93a18014..01cfbbacf 100644
--- a/controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlRscToggleDiskApiCallHandler.java
+++ b/controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlRscToggleDiskApiCallHandler.java
@@ -37,6 +37,7 @@ import com.linbit.linstor.core.objects.StorPool;
import com.linbit.linstor.core.objects.Volume;
import com.linbit.linstor.core.objects.VolumeDefinition;
import com.linbit.linstor.core.objects.utils.MixedStorPoolHelper;
+import com.linbit.linstor.core.types.TcpPortNumber;
import com.linbit.linstor.dbdrivers.DatabaseException;
import com.linbit.linstor.event.EventWaiter;
import com.linbit.linstor.event.ObjectIdentifier;
@@ -85,6 +86,7 @@ import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
import java.util.Set;
+import java.util.TreeSet;
import org.reactivestreams.Publisher;
import reactor.core.publisher.Flux;
@@ -575,12 +577,13 @@ public class CtrlRscToggleDiskApiCallHandler implements CtrlSatelliteConnectionL
/*
* We also have to remove the currently diskless DrbdRscData and free up the node-id as now we must
- * use the shared resource's node-id
+ * use the shared resource's node-id. We still need to preserve TCP ports though.
*/
+ copyDrbdTcpPortsIfExists(rsc, payload);
}
else
{
- copyDrbdNodeIdIfExists(rsc, payload);
+ copyDrbdSettings(rsc, payload);
}
/*
* rebuilds the layerdata in case we just removed it..
@@ -782,10 +785,20 @@ public class CtrlRscToggleDiskApiCallHandler implements CtrlSatelliteConnectionL
}
/**
- * Although we need to rebuild the layerData as the layerList might have changed, if we do not
- * deactivate (i.e. down) the current resource, we need to make sure that deleting DrbdRscData
- * and recreating a new DrbdRscData ends up with the same node-id as before.
+ * Copies DRBD settings (node-id and TCP ports) from the existing DrbdRscData into the payload
+ * before removeLayerData() deletes them. This ensures that recreated DrbdRscData ends up with
+ * the same node-id and TCP ports as before.
+ *
+ * TCP ports must be preserved because if the satellite misses the update (e.g. due to controller
+ * restart or connectivity issues), it will keep the old ports while peers receive the new ones,
+ * causing DRBD connections to fail with StandAlone state.
*/
+ private void copyDrbdSettings(Resource rsc, LayerPayload payload) throws ImplementationError
+ {
+ copyDrbdNodeIdIfExists(rsc, payload);
+ copyDrbdTcpPortsIfExists(rsc, payload);
+ }
+
private void copyDrbdNodeIdIfExists(Resource rsc, LayerPayload payload) throws ImplementationError
{
Set<AbsRscLayerObject<Resource>> drbdRscDataSet = LayerRscUtils.getRscDataByLayer(
@@ -804,6 +817,28 @@ public class CtrlRscToggleDiskApiCallHandler implements CtrlSatelliteConnectionL
}
}
+ private void copyDrbdTcpPortsIfExists(Resource rsc, LayerPayload payload) throws ImplementationError
+ {
+ Set<AbsRscLayerObject<Resource>> drbdRscDataSet = LayerRscUtils.getRscDataByLayer(
+ getLayerData(apiCtx, rsc),
+ DeviceLayerKind.DRBD
+ );
+ if (!drbdRscDataSet.isEmpty())
+ {
+ DrbdRscData<Resource> drbdRscData = (DrbdRscData<Resource>) drbdRscDataSet.iterator().next();
+ Collection<TcpPortNumber> tcpPorts = drbdRscData.getTcpPortList();
+ if (tcpPorts != null && !tcpPorts.isEmpty())
+ {
+ Set<Integer> portInts = new TreeSet<>();
+ for (TcpPortNumber port : tcpPorts)
+ {
+ portInts.add(port.value);
+ }
+ payload.drbdRsc.tcpPorts = portInts;
+ }
+ }
+ }
+
private List<DeviceLayerKind> removeLayerData(Resource rscRef)
{
List<DeviceLayerKind> layerList;
@@ -1058,15 +1093,15 @@ public class CtrlRscToggleDiskApiCallHandler implements CtrlSatelliteConnectionL
/*
* We also have to remove the possible meta-children of previous StorageRscData.
* LayerData will be recreated with ensureStackDataExists.
- * However, we still need to remember our node-id if we had / have DRBD in the list
+ * However, we still need to remember our DRBD settings if we had / have DRBD in the list
*/
- copyDrbdNodeIdIfExists(rsc, payload);
+ copyDrbdSettings(rsc, payload);
layerList = removeLayerData(rsc);
}
else
{
markDiskAdded(rsc);
- ctrlLayerStackHelper.resetStoragePools(rsc);
+ ctrlLayerStackHelper.resetStoragePools(rsc, false);
}
ctrlLayerStackHelper.ensureStackDataExists(rsc, layerList, payload);
diff --git a/controller/src/main/java/com/linbit/linstor/layer/resource/CtrlRscLayerDataFactory.java b/controller/src/main/java/com/linbit/linstor/layer/resource/CtrlRscLayerDataFactory.java
index 3538b380c..f9733b6f1 100644
--- a/controller/src/main/java/com/linbit/linstor/layer/resource/CtrlRscLayerDataFactory.java
+++ b/controller/src/main/java/com/linbit/linstor/layer/resource/CtrlRscLayerDataFactory.java
@@ -263,6 +263,11 @@ public class CtrlRscLayerDataFactory
}
public void resetStoragePools(Resource rscRef)
+ {
+ resetStoragePools(rscRef, true);
+ }
+
+ public void resetStoragePools(Resource rscRef, boolean callEnsureStackDataExistsRef)
{
try
{
@@ -276,8 +281,10 @@ public class CtrlRscLayerDataFactory
rscDataToProcess.addAll(rscData.getChildren());
}
-
- ensureStackDataExists(rscRef, null, new LayerPayload());
+ if (callEnsureStackDataExistsRef)
+ {
+ ensureStackDataExists(rscRef, null, new LayerPayload());
+ }
}
catch (AccessDeniedException exc)
{
@@ -1,570 +0,0 @@
diff --git a/satellite/src/main/java/com/linbit/linstor/layer/storage/AbsStorageProvider.java b/satellite/src/main/java/com/linbit/linstor/layer/storage/AbsStorageProvider.java
--- a/satellite/src/main/java/com/linbit/linstor/layer/storage/AbsStorageProvider.java
+++ b/satellite/src/main/java/com/linbit/linstor/layer/storage/AbsStorageProvider.java
@@ -2081,7 +2081,6 @@ public abstract class AbsStorageProvider<
)
throws IOException, AccessDeniedException
{
- final StorPoolName storPoolObjName = storPoolObj.getName();
if (storDevicePath != null)
{
errorReporter.logDebug("updateMinIoSize: Have storDevicePath \"%s\"", storDevicePath);
@@ -2094,35 +2093,18 @@ public abstract class AbsStorageProvider<
"updateMinIoSize: Block device path is \"%s\"",
blockDevicePath.toString()
);
- final long minIoSize = BlockSizeInfo.getBlockSize(blockDevicePath);
-
- boolean updateValue = true;
-
- final String propKey = StorageConstants.NAMESPACE_INTERNAL + '/' +
- StorageConstants.BLK_DEV_MIN_IO_SIZE;
- final Props storPoolProps = storPoolObj.getProps(storDriverAccCtx);
- final @Nullable String currentPropValue = storPoolProps.getProp(propKey);
- if (currentPropValue != null)
- {
- try
- {
- final long currentPropMinIoSize = Long.parseLong(currentPropValue);
- updateValue = currentPropMinIoSize != minIoSize;
- }
- catch (NumberFormatException ignored)
- {
- }
- }
-
- if (updateValue)
- {
- final String propValue = Long.toString(minIoSize);
- errorReporter.logDebug(
- "Storage pool \"%s\": Set property \"%s\" = \"%s\"",
- storPoolObjName.displayValue, propKey, propValue
- );
- propsChange.changeStorPoolProp(storPoolObj, propKey, propValue);
- }
+ updatePropIfNeeded(
+ propsChange,
+ storPoolObj,
+ StorageConstants.NAMESPACE_INTERNAL + '/' + StorageConstants.BLK_DEV_MIN_IO_SIZE,
+ Long.toString(BlockSizeInfo.getBlockSize(blockDevicePath))
+ );
+ updatePropIfNeeded(
+ propsChange,
+ storPoolObj,
+ StorageConstants.NAMESPACE_INTERNAL + '/' + StorageConstants.BLK_DEV_OPT_IO_SIZE,
+ Long.toString(BlockSizeInfo.getOptimalIoSize(blockDevicePath))
+ );
}
else
{
@@ -2130,6 +2112,28 @@ public abstract class AbsStorageProvider<
}
}
+ private void updatePropIfNeeded(
+ LocalPropsChangePojo propsChangeRef,
+ StorPool storPoolObjRef,
+ String propKeyRef,
+ String propValueRef
+ )
+ throws AccessDeniedException
+ {
+ final Props storPoolProps = storPoolObjRef.getProps(storDriverAccCtx);
+ final @Nullable String currentPropValue = storPoolProps.getProp(propKeyRef);
+ if (currentPropValue == null || !currentPropValue.equals(propValueRef))
+ {
+ errorReporter.logDebug(
+ "Storage pool \"%s\": Set property \"%s\" = \"%s\"",
+ storPoolObjRef.getName().displayValue,
+ propKeyRef,
+ propValueRef
+ );
+ propsChangeRef.changeStorPoolProp(storPoolObjRef, propKeyRef, propValueRef);
+ }
+ }
+
@SuppressWarnings("unchecked")
@Override
public void updateAllocatedSize(VlmProviderObject<Resource> vlmDataRef)
diff --git a/satellite/src/main/java/com/linbit/linstor/layer/storage/utils/BlockSizeInfo.java b/satellite/src/main/java/com/linbit/linstor/layer/storage/utils/BlockSizeInfo.java
--- a/satellite/src/main/java/com/linbit/linstor/layer/storage/utils/BlockSizeInfo.java
+++ b/satellite/src/main/java/com/linbit/linstor/layer/storage/utils/BlockSizeInfo.java
@@ -3,8 +3,11 @@ package com.linbit.linstor.layer.storage.utils;
import com.linbit.utils.MathUtils;
import com.linbit.utils.SymbolicLinkResolver;
+import static com.linbit.linstor.layer.storage.BlockSizeConsts.DFLT_OPT_IO_SIZE;
import static com.linbit.linstor.layer.storage.BlockSizeConsts.DFLT_IO_SIZE;
+import static com.linbit.linstor.layer.storage.BlockSizeConsts.MAX_OPT_IO_SIZE;
import static com.linbit.linstor.layer.storage.BlockSizeConsts.MAX_IO_SIZE;
+import static com.linbit.linstor.layer.storage.BlockSizeConsts.MIN_OPT_IO_SIZE;
import static com.linbit.linstor.layer.storage.BlockSizeConsts.MIN_IO_SIZE;
import java.io.FileInputStream;
@@ -13,6 +16,8 @@ import java.nio.file.Path;
public class BlockSizeInfo
{
+ private static final int NUMBER_BUFFER_SIZE = 32;
+
/**
* Determines the blocksize, aka minimum I/O size, for the specified backing storage path.
*
@@ -51,7 +56,7 @@ public class BlockSizeInfo
final Path infoSourceName = blockDevice.getFileName();
final Path infoSource = Path.of("/sys/block", infoSourceName.toString(), "queue/physical_block_size");
- final byte[] data = new byte[32];
+ final byte[] data = new byte[NUMBER_BUFFER_SIZE];
try (final FileInputStream fileIn = new FileInputStream(infoSource.toString()))
{
final int readCount = fileIn.read(data);
@@ -75,4 +80,38 @@ public class BlockSizeInfo
}
return blockSize;
}
+
+ public static long getOptimalIoSize(final Path storageObj)
+ {
+ long optIoSize = DFLT_OPT_IO_SIZE;
+ try
+ {
+ final Path blockDevice = SymbolicLinkResolver.resolveSymLink(storageObj);
+ final Path infoSourceName = blockDevice.getFileName();
+ final Path infoSource = Path.of("/sys/block", infoSourceName.toString(), "queue/optimal_io_size");
+
+ final byte[] data = new byte[NUMBER_BUFFER_SIZE];
+ try (final FileInputStream fileIn = new FileInputStream(infoSource.toString()))
+ {
+ final int readCount = fileIn.read(data);
+ if (readCount > 0)
+ {
+ String numberStr = new String(data, 0, readCount);
+ numberStr = numberStr.trim();
+ try
+ {
+ final long unboundedOptIoSize = Long.parseLong(numberStr);
+ optIoSize = MathUtils.bounds(MIN_OPT_IO_SIZE, unboundedOptIoSize, MAX_OPT_IO_SIZE);
+ }
+ catch (NumberFormatException ignored)
+ {
+ }
+ }
+ }
+ }
+ catch (IOException ignored)
+ {
+ }
+ return optIoSize;
+ }
}
diff --git a/server/src/main/java/com/linbit/SizeConv.java b/server/src/main/java/com/linbit/SizeConv.java
--- a/server/src/main/java/com/linbit/SizeConv.java
+++ b/server/src/main/java/com/linbit/SizeConv.java
@@ -16,6 +16,7 @@ public class SizeConv
public enum SizeUnit
{
UNIT_B,
+ UNIT_SECTORS,
UNIT_KiB,
UNIT_MiB,
UNIT_GiB,
@@ -41,6 +42,9 @@ public class SizeConv
case UNIT_B:
factor = FACTOR_B;
break;
+ case UNIT_SECTORS:
+ factor = FACTOR_SECTORS;
+ break;
case UNIT_KiB:
factor = FACTOR_KiB;
break;
@@ -111,6 +115,9 @@ public class SizeConv
case "b":
unit = SizeUnit.UNIT_B;
break;
+ case "s":
+ unit = SizeUnit.UNIT_SECTORS;
+ break;
case "k":
// fall-through
case "kb":
@@ -217,6 +224,9 @@ public class SizeConv
// Factor 1
public static final BigInteger FACTOR_B = BigInteger.valueOf(1L);
+ // Factor 512
+ public static final BigInteger FACTOR_SECTORS = BigInteger.valueOf(512L);
+
// Factor 1,024
// Naming convention exception: SI unit capitalization rules
@SuppressWarnings("checkstyle:constantname")
diff --git a/server/src/main/java/com/linbit/linstor/layer/luks/LuksLayerSizeCalculator.java b/server/src/main/java/com/linbit/linstor/layer/luks/LuksLayerSizeCalculator.java
--- a/server/src/main/java/com/linbit/linstor/layer/luks/LuksLayerSizeCalculator.java
+++ b/server/src/main/java/com/linbit/linstor/layer/luks/LuksLayerSizeCalculator.java
@@ -1,27 +1,62 @@
package com.linbit.linstor.layer.luks;
+import com.linbit.SizeConv;
+import com.linbit.SizeConv.SizeUnit;
import com.linbit.exceptions.InvalidSizeException;
+import com.linbit.linstor.PriorityProps;
+import com.linbit.linstor.annotation.Nullable;
+import com.linbit.linstor.api.ApiConsts;
+import com.linbit.linstor.core.objects.AbsResource;
+import com.linbit.linstor.core.objects.AbsVolume;
+import com.linbit.linstor.core.objects.Resource;
+import com.linbit.linstor.core.objects.ResourceDefinition;
+import com.linbit.linstor.core.objects.ResourceGroup;
+import com.linbit.linstor.core.objects.Snapshot;
+import com.linbit.linstor.core.objects.SnapshotVolume;
+import com.linbit.linstor.core.objects.StorPool;
+import com.linbit.linstor.core.objects.Volume;
+import com.linbit.linstor.core.objects.VolumeDefinition;
import com.linbit.linstor.dbdrivers.DatabaseException;
import com.linbit.linstor.layer.AbsLayerSizeCalculator;
+import com.linbit.linstor.netcom.Peer;
+import com.linbit.linstor.propscon.InvalidKeyException;
+import com.linbit.linstor.propscon.ReadOnlyProps;
import com.linbit.linstor.security.AccessDeniedException;
+import com.linbit.linstor.storage.StorageConstants;
import com.linbit.linstor.storage.data.adapter.luks.LuksVlmData;
import com.linbit.linstor.storage.interfaces.categories.resource.VlmProviderObject;
import com.linbit.linstor.storage.kinds.DeviceLayerKind;
import com.linbit.linstor.storage.kinds.ExtTools;
import com.linbit.linstor.storage.kinds.ExtToolsInfo;
import com.linbit.linstor.storage.kinds.ExtToolsInfo.Version;
+import com.linbit.linstor.utils.layer.LayerVlmUtils;
+import com.linbit.utils.ShellUtils;
+import com.linbit.utils.SignedAlign;
import javax.inject.Inject;
import javax.inject.Singleton;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
@Singleton
public class LuksLayerSizeCalculator extends AbsLayerSizeCalculator<LuksVlmData<?>>
{
+ public static final String LUKS2_OPT_METADATA_SIZE = "--luks2-metadata-size";
+ public static final String LUKS2_OPT_KEYSLOTS_SIZE = "--luks2-keyslots-size";
+ public static final String LUKS2_OPT_OFFSET = "--offset";
+ private static final String LUKS2_OPT_ALIGN_PAYLOAD = "--align-payload";
+ private static final long DFLT_LUKS2_METADATA_SIZE_IN_BYTES = 16L << 10;
+ private static final long DFLT_ALIGNMENT_1MIB_IN_BYTES = 1L << 20;
+ private static final long LUKS1_HEADER_SIZE_IN_KIB = 2L << 10;
+ private static final long LUKS2_HEADER_SIZE_IN_KIB = 16L << 10;
+ private static final long LUKS2_HEADER_SIZE_IN_BYTES = LUKS2_HEADER_SIZE_IN_KIB << 10;
- // linstor calculates in KiB
- private static final int MIB = 1024;
- private static final int LUKS1_HEADER_SIZE = 2 * MIB;
- private static final int LUKS2_HEADER_SIZE = 16 * MIB;
+ private static final Pattern PATTERN_SIZE = Pattern.compile("(\\d+)([kKmMgGtT]i?[bB]?|[sS]|)");
@Inject
public LuksLayerSizeCalculator(AbsLayerSizeCalculatorInit initRef)
@@ -74,30 +109,266 @@ public class LuksLayerSizeCalculator extends AbsLayerSizeCalculator<LuksVlmData<
}
private long getLuksHeaderSize(VlmProviderObject<?> vlmDataRef)
- throws AccessDeniedException
+ throws AccessDeniedException, InvalidSizeException
{
- ExtToolsInfo cryptSetupInfo = vlmDataRef.getRscLayerObject()
+ @Nullable Peer peer = vlmDataRef.getRscLayerObject()
.getAbsResource()
.getNode()
- .getPeer(sysCtx)
- .getExtToolsManager()
+ .getPeer(sysCtx);
+ if (peer == null)
+ {
+ throw new InvalidSizeException(
+ "Could not calculate size of LUKS volume, since cryptsetup's version could not be determined",
+ null
+ );
+ }
+ ExtToolsInfo cryptSetupInfo = peer.getExtToolsManager()
.getExtToolInfo(ExtTools.CRYPT_SETUP);
- long luksHeaderSize;
+
+ final long luksHeaderSize;
if (cryptSetupInfo != null && cryptSetupInfo.isSupported())
{
if (cryptSetupInfo.hasVersionOrHigher(new Version(2, 1)))
{
- luksHeaderSize = LUKS2_HEADER_SIZE;
+ luksHeaderSize = calcLuks2HeaderSize(vlmDataRef);
}
else
{
- luksHeaderSize = LUKS1_HEADER_SIZE;
+ luksHeaderSize = LUKS1_HEADER_SIZE_IN_KIB;
}
}
else
{
- luksHeaderSize = -1;
+ throw new InvalidSizeException(
+ "Could not calculate size of LUKS volume, since cryptsetup's version could not be determined",
+ null
+ );
}
return luksHeaderSize;
}
+
+ private long calcLuks2HeaderSize(VlmProviderObject<?> vlmDataRef) throws AccessDeniedException
+ {
+ PriorityProps prioProps = getPrioProps(vlmDataRef);
+
+ final @Nullable String userOptProp = prioProps.getProp(
+ ApiConsts.KEY_STOR_DRIVER_LUKS_FORMAT_OPTIONS,
+ ApiConsts.NAMESPC_STORAGE_DRIVER
+ );
+ List<String> userOptions = userOptProp != null ?
+ ShellUtils.shellSplit(userOptProp) :
+ new LinkedList<>();
+
+ final long alignedHeaderSizeInBytes;
+ final long unalignedHeaderSizeInBytes;
+ @Nullable Long cryptsetupOffsetInBytes = getLongOptionValue(
+ userOptions,
+ LUKS2_OPT_OFFSET,
+ SizeUnit.UNIT_SECTORS
+ );
+ if (cryptsetupOffsetInBytes != null)
+ {
+ alignedHeaderSizeInBytes = cryptsetupOffsetInBytes;
+ }
+ else
+ {
+ @Nullable Long cryptsetupLuks2KeyslotsSize = getLongOptionValue(
+ userOptions,
+ LUKS2_OPT_KEYSLOTS_SIZE
+ );
+ if (cryptsetupLuks2KeyslotsSize == null)
+ {
+ unalignedHeaderSizeInBytes = LUKS2_HEADER_SIZE_IN_BYTES;
+ }
+ else
+ {
+ @Nullable Long cryptsetupLuks2MetadataSize = getLongOptionValue(
+ userOptions,
+ LUKS2_OPT_METADATA_SIZE
+ );
+ cryptsetupLuks2MetadataSize = cryptsetupLuks2MetadataSize == null ?
+ DFLT_LUKS2_METADATA_SIZE_IN_BYTES :
+ cryptsetupLuks2MetadataSize;
+
+ unalignedHeaderSizeInBytes = 2 * cryptsetupLuks2MetadataSize + cryptsetupLuks2KeyslotsSize;
+ }
+
+ long alignment = getAlignment(vlmDataRef, userOptions);
+ alignedHeaderSizeInBytes = new SignedAlign(alignment).ceiling(unalignedHeaderSizeInBytes);
+ }
+ return SizeConv.convert(
+ alignedHeaderSizeInBytes,
+ SizeUnit.UNIT_B,
+ SizeUnit.UNIT_KiB
+ );
+ }
+
+ private long getAlignment(VlmProviderObject<?> vlmDataRef, List<String> userOptions) throws AccessDeniedException
+ {
+ @Nullable Long cryptsetupAlignPayloadInBytes = getLongOptionValue(
+ userOptions,
+ LUKS2_OPT_ALIGN_PAYLOAD,
+ SizeUnit.UNIT_SECTORS
+ );
+
+ long alignment = DFLT_ALIGNMENT_1MIB_IN_BYTES;
+ if (cryptsetupAlignPayloadInBytes != null)
+ {
+ alignment = cryptsetupAlignPayloadInBytes;
+ }
+ else
+ {
+ final long maxOptIoSize = getMaxOptIoSize(vlmDataRef);
+ alignment = Math.max(alignment, maxOptIoSize);
+ }
+ return alignment;
+ }
+
+ private long getMaxOptIoSize(VlmProviderObject<?> vlmDataRef) throws InvalidKeyException, AccessDeniedException
+ {
+ long ret = 0;
+ Set<StorPool> storPoolSet = LayerVlmUtils.getStorPoolSet(vlmDataRef, sysCtx);
+ for (StorPool sp : storPoolSet)
+ {
+ @Nullable String strValue = sp.getProps(sysCtx)
+ .getProp(
+ StorageConstants.BLK_DEV_OPT_IO_SIZE,
+ StorageConstants.NAMESPACE_INTERNAL
+ );
+ if (strValue != null)
+ {
+ try
+ {
+ long parsed = Long.parseLong(strValue);
+ ret = Math.max(parsed, ret);
+ }
+ catch (NumberFormatException ignored)
+ {
+ errorReporter.logWarning(
+ "LuksHeaderSize: Failed to parse '%s' from prop %s. Defaulting to 0 opt_io_size " +
+ "(no recommendation/hint)",
+ strValue,
+ StorageConstants.NAMESPACE_INTERNAL + "/" + StorageConstants.BLK_DEV_OPT_IO_SIZE
+ );
+ }
+ }
+ }
+ return ret;
+ }
+
+ private @Nullable Long getLongOptionValue(List<String> userOptionsRef, String optRef)
+ {
+ return getLongOptionValue(userOptionsRef, optRef, SizeUnit.UNIT_B);
+ }
+
+ @SuppressWarnings("checkstyle:magicnumber")
+ private @Nullable Long getLongOptionValue(List<String> userOptionsRef, String optRef, SizeUnit dfltSizeUnit)
+ {
+ @Nullable Long ret = null;
+ @Nullable String val = findLastValue(userOptionsRef, optRef);
+ if (val != null && !val.isBlank())
+ {
+ Matcher matcher = PATTERN_SIZE.matcher(val);
+ if (matcher.matches())
+ {
+ try
+ {
+ ret = Long.parseLong(matcher.group(1));
+ String unit = matcher.group(2);
+ SizeUnit sizeUnit;
+ if (unit.isBlank())
+ {
+ sizeUnit = dfltSizeUnit;
+ }
+ else
+ {
+ boolean forcePowerOfTwo = unit.length() == 1 || unit.length() == 3;
+ sizeUnit = SizeUnit.parse(unit, forcePowerOfTwo);
+ }
+
+ ret = SizeConv.convert(ret, sizeUnit, SizeUnit.UNIT_B);
+ }
+ catch (NumberFormatException ignored)
+ {
+ errorReporter.logWarning(
+ "LuksHeaderSize: Failed to parse '%s' from option '%s %s'.",
+ matcher.group(1),
+ optRef,
+ val
+ );
+ }
+ }
+ }
+ return ret;
+ }
+
+ private @Nullable String findLastValue(List<String> userOptionsRef, String optRef)
+ {
+ Iterator<String> it = userOptionsRef.iterator();
+ @Nullable String val = null;
+ while (it.hasNext())
+ {
+ String opt = it.next();
+ if (opt.equals(optRef))
+ {
+ if (it.hasNext())
+ {
+ val = it.next();
+ }
+ else
+ {
+ val = null;
+ }
+ }
+ else if (opt.startsWith(optRef + "="))
+ {
+ val = opt.substring(optRef.length() + 1);
+ if (val.isBlank())
+ {
+ val = null;
+ }
+ }
+ }
+ return val;
+ }
+
+ private PriorityProps getPrioProps(VlmProviderObject<?> vlmDataRef) throws AccessDeniedException
+ {
+ final AbsVolume<?> vlm = vlmDataRef.getVolume();
+ final AbsResource<?> rsc = vlm.getAbsResource();
+ final ResourceDefinition rscDfn = vlm.getResourceDefinition();
+ final VolumeDefinition vlmDfn = vlm.getVolumeDefinition();
+ final ResourceGroup rscGrp = rscDfn.getResourceGroup();
+
+ final ReadOnlyProps vlmProps;
+ final ReadOnlyProps rscProps;
+ if (vlm instanceof Volume)
+ {
+ vlmProps = ((Volume) vlm).getProps(sysCtx);
+ rscProps = ((Resource) rsc).getProps(sysCtx);
+ }
+ else
+ {
+ vlmProps = ((SnapshotVolume) vlm).getVlmProps(sysCtx);
+ rscProps = ((Snapshot) rsc).getRscProps(sysCtx);
+ }
+
+ final PriorityProps prioProps = new PriorityProps(
+ vlmProps,
+ rscProps
+ );
+ for (StorPool storPool : LayerVlmUtils.getStorPoolSet(vlmDataRef, sysCtx))
+ {
+ prioProps.addProps(storPool.getProps(sysCtx));
+ }
+ prioProps.addProps(
+ rsc.getNode().getProps(sysCtx),
+ vlmDfn.getProps(sysCtx),
+ rscDfn.getProps(sysCtx),
+ rscGrp.getVolumeGroupProps(sysCtx, vlmDfn.getVolumeNumber()),
+ rscGrp.getProps(sysCtx),
+ stltProps
+ );
+ return prioProps;
+ }
}
diff --git a/server/src/main/java/com/linbit/linstor/layer/storage/BlockSizeConsts.java b/server/src/main/java/com/linbit/linstor/layer/storage/BlockSizeConsts.java
--- a/server/src/main/java/com/linbit/linstor/layer/storage/BlockSizeConsts.java
+++ b/server/src/main/java/com/linbit/linstor/layer/storage/BlockSizeConsts.java
@@ -13,4 +13,9 @@ public class BlockSizeConsts
// Default value for the minimum_io_size value of non-storage layers
public static final long DFLT_SPECIAL_IO_SIZE = (1L << 12);
+
+ // optimal_io_size may be 0 to indicate no recommendation.
+ public static final long MIN_OPT_IO_SIZE = 0;
+ public static final long DFLT_OPT_IO_SIZE = 0;
+ public static final long MAX_OPT_IO_SIZE = Long.MAX_VALUE;
}
diff --git a/server/src/main/java/com/linbit/linstor/storage/StorageConstants.java b/server/src/main/java/com/linbit/linstor/storage/StorageConstants.java
--- a/server/src/main/java/com/linbit/linstor/storage/StorageConstants.java
+++ b/server/src/main/java/com/linbit/linstor/storage/StorageConstants.java
@@ -11,6 +11,7 @@ public class StorageConstants
public static final String NAMESPACE_NVME = ApiConsts.NAMESPC_STORAGE_DRIVER + "/NVME";
public static final String NAMESPACE_INTERNAL = NAMESPACE_STOR_DRIVER + "/internal/";
+ public static final String BLK_DEV_OPT_IO_SIZE = "optIoSize";
public static final String BLK_DEV_MIN_IO_SIZE = "minIoSize";
public static final String BLK_DEV_MIN_IO_SIZE_AUTO = "minIoSizeAuto";
public static final String BLK_DEV_MAX_BIO_SIZE = "maxBioSize";
@@ -1,660 +0,0 @@
diff --git a/satellite/src/main/java/com/linbit/linstor/layer/storage/AbsStorageProvider.java b/satellite/src/main/java/com/linbit/linstor/layer/storage/AbsStorageProvider.java
--- a/satellite/src/main/java/com/linbit/linstor/layer/storage/AbsStorageProvider.java
+++ b/satellite/src/main/java/com/linbit/linstor/layer/storage/AbsStorageProvider.java
@@ -163,6 +163,8 @@
}
private static final long DFLT_WAIT_UNTIL_DEVICE_CREATED_TIMEOUT_IN_MS = 5000;
+ private static final int PROBE_VLM_ATTEMPTS = 3;
+ private static final long PROBE_VLM_RETRY_DELAY_IN_MS = 1000;
protected static final int DFLT_STRIPES = 1;
public static final long SIZE_OF_NOT_FOUND_STOR_POOL = ApiConsts.VAL_STOR_POOL_SPACE_NOT_FOUND;
@@ -184,6 +186,7 @@
protected final Set<String> changedStoragePoolStrings = new HashSet<>();
private final String typeDescr;
private final FileSystemWatch fsWatch;
+ private final Object probeVlmLock = new Object();
protected final DeviceProviderKind kind;
private final DrbdInvalidateUtils drbdInvalidateUtils;
@@ -2022,61 +2025,126 @@
final StorPoolName storPoolObjName = storPoolObj.getName();
errorReporter.logDebug("ENTER updateMinIoSize method: Storage pool \"%s\"", storPoolObjName.displayValue);
Collection<VlmProviderObject<Resource>> vlmProviderList = storPoolObj.getVolumes(storDriverAccCtx);
+ boolean haveInfo = false;
if (vlmProviderList != null && vlmProviderList.size() >= 1)
{
errorReporter.logDebug("updateMinIoSize: Have vlmProviderList with %d items", vlmProviderList.size());
Iterator<VlmProviderObject<Resource>> vlmProviderIter = vlmProviderList.iterator();
- boolean haveInfo = false;
while (vlmProviderIter.hasNext() && !haveInfo)
{
VlmProviderObject<Resource> vlmProvider = vlmProviderIter.next();
errorReporter.logDebug("updateMinIoSize: Have vlmProvider");
final String storDevicePath = vlmProvider.getDevicePath();
+ if (storDevicePath != null)
+ {
+ try
+ {
+ updateMinIoSizeByDevice(storPoolObj, storDevicePath, propsChange);
+ haveInfo = true;
+ }
+ catch (IOException ignored)
+ {
+ }
+ }
+ }
+ }
+ // An inactive shared LVM volume can be owned by another node; do not probe it by creating a volume.
+ if (!haveInfo && this instanceof ProbeVlmStorageProvider &&
+ (vlmProviderList == null || vlmProviderList.isEmpty() ||
+ kind == DeviceProviderKind.ZFS || kind == DeviceProviderKind.ZFS_THIN))
+ {
+ updateMinIoSizeByProbeVlm((ProbeVlmStorageProvider) this, storPoolObj, propsChange);
+ }
+ errorReporter.logDebug("EXIT updateMinIoSize method");
+ }
+
+ private void updateMinIoSizeByProbeVlm(
+ ProbeVlmStorageProvider storPrv,
+ StorPool storPoolObj,
+ LocalPropsChangePojo propsChange
+ )
+ throws AccessDeniedException
+ {
+ // Providers use a fixed probe name. Keep creation, probing and cleanup in one critical section.
+ synchronized (probeVlmLock)
+ {
+ boolean haveInfo = false;
+ boolean interrupted = false;
+ String lastFailure = "";
+ int attempts = 0;
+ while (attempts < PROBE_VLM_ATTEMPTS && !haveInfo && !interrupted)
+ {
+ if (attempts > 0)
+ {
+ try
+ {
+ Thread.sleep(PROBE_VLM_RETRY_DELAY_IN_MS);
+ }
+ catch (InterruptedException exc)
+ {
+ Thread.currentThread().interrupt();
+ lastFailure = "interrupted before the next probe attempt";
+ break;
+ }
+ }
+ attempts++;
+ boolean cleaned = false;
try
{
- updateMinIoSizeByDevice(storPoolObj, storDevicePath, propsChange);
- haveInfo = true;
- }
- catch (IOException ignored)
- {
- }
- }
- }
- else
- {
- errorReporter.logDebug("updateMinIoSize: Don't have vlmProviderList, using temporary volumes");
- if (this instanceof ProbeVlmStorageProvider)
- {
- final ProbeVlmStorageProvider storPrv = (ProbeVlmStorageProvider) this;
- try
- {
- final String storDevicePath = storPrv.createTmpProbeVlm(storPoolObj);
+ @Nullable String storDevicePath = storPrv.createTmpProbeVlm(storPoolObj);
+ if (storDevicePath == null)
+ {
+ lastFailure = "temporary volume has no device path";
+ }
+ else
+ {
+ waitUntilDeviceCreated(storDevicePath, getWaitTimeoutAfterCreate(storPoolObj));
+ updateMinIoSizeByDevice(storPoolObj, storDevicePath, propsChange);
+ haveInfo = true;
+ }
+ }
+ catch (StorageException | IOException exc)
+ {
+ lastFailure = exc.getMessage();
+ interrupted = exc.getCause() instanceof InterruptedException;
+ errorReporter.logDebug("Temporary volume probe attempt %d failed: %s", attempts, lastFailure);
+ }
+ finally
+ {
try
{
- updateMinIoSizeByDevice(storPoolObj, storDevicePath, propsChange);
- }
- catch (IOException ignored)
- {
- }
- }
- catch (StorageException ignored)
- {
- errorReporter.logDebug("updateMinIoSize: Temporary volume creation failed");
- }
- finally
- {
- try
- {
storPrv.deleteTmpProbeVlm(storPoolObj);
- }
- catch (StorageException ignored)
- {
- errorReporter.logDebug("updateMinIoSize: Temporary volume deletion failed");
- }
- }
- }
- }
- errorReporter.logDebug("EXIT updateMinIoSize method");
+ cleaned = true;
+ }
+ catch (StorageException exc)
+ {
+ interrupted |= exc.getCause() instanceof InterruptedException;
+ errorReporter.logWarning(
+ "Failed to delete temporary probe volume for storage pool '%s': %s",
+ storPoolObj.getName().displayValue,
+ exc.getMessage()
+ );
+ }
+ }
+ if (interrupted)
+ {
+ Thread.currentThread().interrupt();
+ }
+ if (!cleaned)
+ {
+ break;
+ }
+ }
+ if (!haveInfo)
+ {
+ errorReporter.logWarning(
+ "Could not determine block device information for storage pool '%s' after %d probe attempts: %s",
+ storPoolObj.getName().displayValue,
+ attempts,
+ lastFailure
+ );
+ }
+ }
}
private void updateMinIoSizeByDevice(
diff --git a/src/test/java/com/linbit/fsevent/ProbeVolumeDeviceWaitTest.java b/src/test/java/com/linbit/fsevent/ProbeVolumeDeviceWaitTest.java
new file mode 100644
--- /dev/null
+++ b/src/test/java/com/linbit/fsevent/ProbeVolumeDeviceWaitTest.java
@@ -0,0 +1,467 @@
+package com.linbit.fsevent;
+
+import com.linbit.Platform;
+import com.linbit.linstor.core.identifier.StorPoolName;
+import com.linbit.linstor.core.objects.Resource;
+import com.linbit.linstor.core.objects.StorPool;
+import com.linbit.linstor.core.pojos.LocalPropsChangePojo;
+import com.linbit.linstor.layer.storage.AbsStorageProvider.AbsStorageProviderInit;
+import com.linbit.linstor.layer.storage.BlockSizeConsts;
+import com.linbit.linstor.layer.storage.lvm.LvmProvider;
+import com.linbit.linstor.layer.storage.zfs.ZfsThinProvider;
+import com.linbit.linstor.logging.ErrorReporter;
+import com.linbit.linstor.propscon.Props;
+import com.linbit.linstor.storage.StorageConstants;
+import com.linbit.linstor.storage.StorageException;
+import com.linbit.linstor.storage.interfaces.categories.resource.VlmProviderObject;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.rules.Timeout;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assume.assumeTrue;
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.mockingDetails;
+import static org.mockito.Mockito.when;
+
+public class ProbeVolumeDeviceWaitTest
+{
+ @Rule
+ public final TemporaryFolder directory = new TemporaryFolder();
+
+ @Rule
+ public final Timeout timeout = Timeout.seconds(15);
+
+ private FileSystemWatch watch;
+ private Path device;
+ private Path link;
+ private StorPool pool;
+ private ProbeProvider provider;
+ private ErrorReporter reporter;
+ private ScheduledExecutorService executor;
+
+ @Before
+ public void setUp() throws Exception
+ {
+ assumeTrue(Platform.isLinux());
+ device = directory.newFile("zd-test").toPath();
+ link = directory.getRoot().toPath().resolve("probe");
+ executor = Executors.newSingleThreadScheduledExecutor();
+ reporter = mock(ErrorReporter.class);
+ watch = new FileSystemWatch(reporter);
+ watch.start();
+ pool = mock(StorPool.class);
+ when(pool.getName()).thenReturn(new StorPoolName("data"));
+ when(pool.getVolumes(null)).thenReturn(Collections.emptyList());
+ when(pool.getProps(null)).thenReturn(mock(Props.class));
+ provider = new ProbeProvider(providerInit());
+ }
+
+ @After
+ public void tearDown() throws Exception
+ {
+ if (executor != null)
+ {
+ executor.shutdownNow();
+ assertTrue(executor.awaitTermination(2, TimeUnit.SECONDS));
+ }
+ if (watch != null)
+ {
+ watch.shutdown(false);
+ watch.awaitShutdown(2000);
+ watch.cancelAllWatchKeys();
+ }
+ }
+
+ @Test
+ public void delayedDevicePublishesPropertiesBeforeCleanup() throws Exception
+ {
+ provider.delayMillis = 50;
+ assertPublished(probe());
+ assertCleaned(1);
+ assertEquals(0, warningCount());
+ }
+
+ @Test
+ public void existingProbeDevicePublishesProperties() throws Exception
+ {
+ assertPublished(probe());
+ assertCleaned(1);
+ }
+
+ @Test
+ public void missingDeviceTimesOutAndCleansEveryAttempt() throws Exception
+ {
+ provider.delayMillis = -1;
+ provider.timeoutMillis = 20;
+ assertTrue(probe().isEmpty());
+ assertCleaned(3);
+ assertEquals(1, warningCount());
+ }
+
+ @Test
+ public void failedCreationStillCleansEveryAttempt() throws Exception
+ {
+ provider.failCreateTimes = Integer.MAX_VALUE;
+ assertTrue(probe().isEmpty());
+ assertCleaned(3);
+ assertEquals(1, warningCount());
+ }
+
+ @Test
+ public void transientCreationFailureIsRetried() throws Exception
+ {
+ provider.failCreateTimes = 1;
+ provider.delayMillis = 50;
+ assertPublished(probe());
+ assertCleaned(2);
+ assertEquals(0, warningCount());
+ }
+
+ @Test
+ public void nullProbePathDoesNotPublishProperties() throws Exception
+ {
+ provider.nullPath = true;
+ assertTrue(probe().isEmpty());
+ assertCleaned(3);
+ assertEquals(1, warningCount());
+ }
+
+ @Test
+ public void nullFirstPathDoesNotSkipUsableVolume() throws Exception
+ {
+ Files.createSymbolicLink(link, device);
+ doReturn(List.of(volume(null), volume(link))).when(pool).getVolumes(null);
+ assertPublished(probe());
+ assertEquals(0, provider.createCalls);
+ assertEquals(0, provider.deleteCalls);
+ }
+
+ @Test
+ public void missingFirstPathDoesNotSkipUsableVolume() throws Exception
+ {
+ Files.createSymbolicLink(link, device);
+ doReturn(List.of(volume(link.resolveSibling("missing")), volume(link))).when(pool).getVolumes(null);
+ assertPublished(probe());
+ assertEquals(0, provider.createCalls);
+ assertEquals(0, provider.deleteCalls);
+ }
+
+ @Test
+ public void allNullPathsFallBackToProbe() throws Exception
+ {
+ doReturn(List.of(volume(null))).when(pool).getVolumes(null);
+ provider.delayMillis = 50;
+ assertPublished(probe());
+ assertCleaned(1);
+ }
+
+ @Test
+ public void allMissingPathsFallBackToProbe() throws Exception
+ {
+ doReturn(List.of(
+ volume(link.resolveSibling("missing-a")), volume(link.resolveSibling("missing-b"))
+ )).when(pool).getVolumes(null);
+ provider.delayMillis = 50;
+ assertPublished(probe());
+ assertCleaned(1);
+ }
+
+ @Test
+ public void cleanupFailureAfterSuccessIsReported() throws Exception
+ {
+ provider.failDelete = true;
+ assertPublished(probe());
+ assertEquals(1, provider.createCalls);
+ assertEquals(1, provider.deleteCalls);
+ assertEquals(1, warningCount());
+ }
+
+ @Test
+ public void cleanupFailurePreventsAnotherCreationAttempt() throws Exception
+ {
+ provider.delayMillis = -1;
+ provider.timeoutMillis = 20;
+ provider.failDelete = true;
+ assertTrue(probe().isEmpty());
+ assertEquals(1, provider.createCalls);
+ assertEquals(1, provider.deleteCalls);
+ assertEquals(2, warningCount());
+ }
+
+ @Test
+ public void interruptedDeviceWaitCleansUpAndPreservesInterrupt() throws Exception
+ {
+ provider.delayMillis = -1;
+ Thread callingThread = Thread.currentThread();
+ executor.schedule(callingThread::interrupt, 50, TimeUnit.MILLISECONDS);
+ try
+ {
+ assertTrue(probe().isEmpty());
+ assertTrue(callingThread.isInterrupted());
+ assertCleaned(1);
+ assertEquals(1, warningCount());
+ }
+ finally
+ {
+ Thread.interrupted();
+ }
+ }
+
+ @Test
+ public void interruptedRetryDelayDoesNotCreateAnotherProbe() throws Exception
+ {
+ provider.failCreateTimes = Integer.MAX_VALUE;
+ Thread callingThread = Thread.currentThread();
+ executor.schedule(callingThread::interrupt, 100, TimeUnit.MILLISECONDS);
+ try
+ {
+ assertTrue(probe().isEmpty());
+ assertTrue(callingThread.isInterrupted());
+ assertCleaned(1);
+ assertEquals(1, warningCount());
+ }
+ finally
+ {
+ Thread.interrupted();
+ }
+ }
+
+ @Test
+ public void interruptedCleanupPreservesInterruptAndStopsRetries() throws Exception
+ {
+ provider.failCreateTimes = Integer.MAX_VALUE;
+ provider.interruptDelete = true;
+ try
+ {
+ assertTrue(probe().isEmpty());
+ assertTrue(Thread.currentThread().isInterrupted());
+ assertEquals(1, provider.createCalls);
+ assertEquals(1, provider.deleteCalls);
+ }
+ finally
+ {
+ Thread.interrupted();
+ }
+ }
+
+ @Test
+ public void concurrentProbesDoNotShareTheTemporaryVolume() throws Exception
+ {
+ provider.creationDelayMillis = 200;
+ ExecutorService callers = Executors.newFixedThreadPool(2);
+ CountDownLatch ready = new CountDownLatch(2);
+ CountDownLatch start = new CountDownLatch(1);
+ Callable<LocalPropsChangePojo> call = () ->
+ {
+ ready.countDown();
+ start.await();
+ return probe();
+ };
+ try
+ {
+ Future<LocalPropsChangePojo> first = callers.submit(call);
+ Future<LocalPropsChangePojo> second = callers.submit(call);
+ assertTrue(ready.await(2, TimeUnit.SECONDS));
+ start.countDown();
+ assertFalse(first.get(10, TimeUnit.SECONDS).isEmpty());
+ assertFalse(second.get(10, TimeUnit.SECONDS).isEmpty());
+ assertEquals(1, provider.peakActiveProbes.get());
+ assertCleaned(2);
+ }
+ finally
+ {
+ callers.shutdownNow();
+ assertTrue(callers.awaitTermination(2, TimeUnit.SECONDS));
+ }
+ }
+
+ @Test
+ public void inactiveLvmVolumesDoNotCreateProbe() throws Exception
+ {
+ doReturn(List.of(volume(null))).when(pool).getVolumes(null);
+ LvmProvider lvm = new LvmProvider(providerInit())
+ {
+ @Override
+ public String createTmpProbeVlm(StorPool ignored)
+ {
+ throw new AssertionError("Inactive shared LVM volumes must not trigger a probe volume");
+ }
+
+ @Override
+ public void deleteTmpProbeVlm(StorPool ignored)
+ {
+ throw new AssertionError("No probe volume was created");
+ }
+ };
+ LocalPropsChangePojo changes = new LocalPropsChangePojo();
+ lvm.updateMinIoSize(pool, changes);
+ assertTrue(changes.isEmpty());
+ }
+
+ private AbsStorageProviderInit providerInit()
+ {
+ return new AbsStorageProviderInit(
+ reporter, null, null, null, null, null, null, null, null, null, watch, null, null, null
+ );
+ }
+
+ private LocalPropsChangePojo probe() throws Exception
+ {
+ LocalPropsChangePojo changes = new LocalPropsChangePojo();
+ provider.updateMinIoSize(pool, changes);
+ return changes;
+ }
+
+ private void assertPublished(LocalPropsChangePojo changesRef)
+ {
+ String namespace = StorageConstants.NAMESPACE_INTERNAL + '/';
+ assertEquals(Map.of(
+ namespace + StorageConstants.BLK_DEV_MIN_IO_SIZE, Long.toString(BlockSizeConsts.DFLT_IO_SIZE),
+ namespace + StorageConstants.BLK_DEV_OPT_IO_SIZE, Long.toString(BlockSizeConsts.DFLT_OPT_IO_SIZE)
+ ), changesRef.changedStorPoolProps.get(pool.getName()));
+ }
+
+ private void assertCleaned(int attemptsRef)
+ {
+ assertEquals(attemptsRef, provider.createCalls);
+ assertEquals(attemptsRef, provider.deleteCalls);
+ assertFalse(Files.exists(link));
+ }
+
+ @SuppressWarnings("unchecked")
+ private VlmProviderObject<Resource> volume(Path pathRef)
+ {
+ VlmProviderObject<Resource> volume = mock(VlmProviderObject.class);
+ when(volume.getDevicePath()).thenReturn(pathRef == null ? null : pathRef.toString());
+ return volume;
+ }
+
+ private long warningCount()
+ {
+ return mockingDetails(reporter).getInvocations().stream()
+ .filter(invocation -> invocation.getMethod().getName().equals("logWarning"))
+ .count();
+ }
+
+ private class ProbeProvider extends ZfsThinProvider
+ {
+ private long delayMillis;
+ private long creationDelayMillis;
+ private long timeoutMillis = 4000;
+ private int failCreateTimes;
+ private boolean nullPath;
+ private boolean failDelete;
+ private boolean interruptDelete;
+ private int createCalls;
+ private int deleteCalls;
+ private ScheduledFuture<?> create;
+ private final AtomicInteger activeProbes = new AtomicInteger();
+ private final AtomicInteger peakActiveProbes = new AtomicInteger();
+
+ ProbeProvider(AbsStorageProviderInit initRef)
+ {
+ super(initRef);
+ }
+
+ @Override
+ protected long getWaitTimeoutAfterCreate(StorPool ignored)
+ {
+ return timeoutMillis;
+ }
+
+ @Override
+ public String createTmpProbeVlm(StorPool ignored) throws StorageException
+ {
+ peakActiveProbes.accumulateAndGet(activeProbes.incrementAndGet(), Math::max);
+ createCalls++;
+ if (creationDelayMillis > 0)
+ {
+ try
+ {
+ Thread.sleep(creationDelayMillis);
+ }
+ catch (InterruptedException exc)
+ {
+ throw new StorageException("Test probe creation interrupted", exc);
+ }
+ }
+ if (failCreateTimes > 0)
+ {
+ failCreateTimes--;
+ throw new StorageException("Probe creation failed");
+ }
+ if (nullPath)
+ {
+ return null;
+ }
+ if (delayMillis == 0)
+ {
+ try
+ {
+ Files.createSymbolicLink(link, device);
+ }
+ catch (IOException exc)
+ {
+ throw new StorageException("Test device creation failed", exc);
+ }
+ }
+ else if (delayMillis > 0)
+ {
+ create = executor.schedule(() -> Files.createSymbolicLink(link, device),
+ delayMillis, TimeUnit.MILLISECONDS);
+ }
+ return link.toString();
+ }
+
+ @Override
+ public void deleteTmpProbeVlm(StorPool ignored) throws StorageException
+ {
+ deleteCalls++;
+ if (interruptDelete)
+ {
+ throw new StorageException("Probe cleanup interrupted", new InterruptedException());
+ }
+ if (failDelete)
+ {
+ throw new StorageException("Probe cleanup failed");
+ }
+ try
+ {
+ if (create != null && !create.cancel(false))
+ {
+ create.get();
+ }
+ Files.deleteIfExists(link);
+ activeProbes.decrementAndGet();
+ }
+ catch (Exception exc)
+ {
+ throw new StorageException("Test device cleanup failed", exc);
+ }
+ }
+ }
+}
@@ -1,8 +1,8 @@
diff --git a/build.gradle b/build.gradle
index 75b7f64..d2dfec9 100644
index 9df6c840cdc0..9894a436712c 100644
--- a/build.gradle
+++ b/build.gradle
@@ -37,6 +37,9 @@ configurations {
@@ -34,6 +34,9 @@ configurations {
}
def protobufVersion = '31.1'
@@ -12,7 +12,7 @@ index 75b7f64..d2dfec9 100644
def protoc = file("${projectDir}/tools/protoc-" + protobufVersion + '/bin/protoc')
def logbackConfig = "logback.xml"
def jvmOpts = ["-Xms32M"]
@@ -577,7 +580,7 @@ tasks.register('buildTestCommands', Exec) {
@@ -650,7 +653,7 @@ tasks.register('buildTestCommands', Exec) {
tasks.register('unzipProtoc', Copy) {
dependsOn(downloadProtoc)
@@ -21,7 +21,7 @@ index 75b7f64..d2dfec9 100644
def outputDir = file("tools/protoc-" + protobufVersion)
from zipTree(zipFile)
@@ -591,7 +594,7 @@ tasks.register('unzipProtoc', Copy) {
@@ -664,7 +667,7 @@ tasks.register('unzipProtoc', Copy) {
tasks.register('downloadProtoc') {
outputs.dir "tools"
@@ -30,7 +30,7 @@ index 75b7f64..d2dfec9 100644
outputs.file protozip
doLast {
@@ -599,7 +602,7 @@ tasks.register('downloadProtoc') {
@@ -672,7 +675,7 @@ tasks.register('downloadProtoc') {
mkdir "tools"
println "downloading protoc..."
new URL('https://github.com/google/protobuf/releases/download/v'
@@ -1,28 +1,27 @@
diff --git a/satellite/src/main/java/com/linbit/linstor/layer/drbd/utils/DrbdAdm.java b/satellite/src/main/java/com/linbit/linstor/layer/drbd/utils/DrbdAdm.java
index 5627d1be8..ece191292 100644
index 0a19f875b702..d44e19b70d88 100644
--- a/satellite/src/main/java/com/linbit/linstor/layer/drbd/utils/DrbdAdm.java
+++ b/satellite/src/main/java/com/linbit/linstor/layer/drbd/utils/DrbdAdm.java
@@ -42,6 +42,9 @@ import java.util.Arrays;
import java.util.List;
@@ -45,6 +45,8 @@ import java.util.List;
import java.util.Set;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.TimeUnit;
+import java.nio.charset.StandardCharsets;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
import java.util.stream.Collectors;
@Singleton
@@ -56,6 +58,9 @@ public class DrbdAdm
public static final int WAIT_CONNECT_RES_TIME = 10;
private static final long DOWN_WAIT_TIMEOUT_SEC = 5;
@@ -68,6 +70,9 @@ public class DrbdAdm
private static final long SECONDARY_RETRY_INITIAL_BACKOFF_MS = 250L;
private static final long SECONDARY_RETRY_MAX_BACKOFF_MS = 2_000L;
private static final String DEVICE_HELD_OPEN_ERR_MSG = "Device is held open by someone";
+ private static final long FORCE_DETACH_RETRY_WAIT_MS = 250;
+ private static final String BITMAP_LEAK_ERR_MSG = "already has a bitmap, this should not happen";
+ private static final Pattern BITMAP_LEAK_MINOR_PATTERN = Pattern.compile("\\bminor\\s+(\\d+)\\b");
private final ExtCmdFactory extCmdFactory;
private final AccessContext sysCtx;
@@ -131,8 +136,38 @@ public class DrbdAdm
private final StltConfigAccessor stltCfgAccessor;
@@ -143,8 +148,38 @@ public class DrbdAdm
// command.add(resName);
command.add(drbdRscData.getSuffixedResourceName());
// execute(Arrays.asList("drbdsetup", "show", drbdRscData.getSuffixedResourceName()));
@@ -35,7 +34,7 @@ index 5627d1be8..ece191292 100644
+ ExtCmd extCmd = extCmdFactory.create();
+ if (Platform.isWindows())
+ {
+ extCmd.setTimeout(TimeoutType.WAIT, 5 * 60 * 1000);
+ extCmd.setTimeout(TimeoutType.WAIT, DFLT_WINDOWS_TIMEOUT_MS);
+ }
+
+ OutputData outputData = extCmd.pipeExec(ProcessBuilder.Redirect.from(nullDevice), commandArr);
@@ -63,8 +62,8 @@ index 5627d1be8..ece191292 100644
drbdRscData.setAdjustRequired(false);
}
@@ -805,6 +840,75 @@ public class DrbdAdm
}
@@ -955,6 +990,75 @@ public class DrbdAdm
return ret;
}
+ static boolean isBitmapLeakOnAttach(OutputData outputData)
@@ -1,104 +0,0 @@
diff --git a/satellite/src/main/java/com/linbit/linstor/layer/drbd/utils/DrbdAdm.java b/satellite/src/main/java/com/linbit/linstor/layer/drbd/utils/DrbdAdm.java
index 71c1df6..f91903b 100644
--- a/satellite/src/main/java/com/linbit/linstor/layer/drbd/utils/DrbdAdm.java
+++ b/satellite/src/main/java/com/linbit/linstor/layer/drbd/utils/DrbdAdm.java
@@ -62,6 +62,10 @@ public class DrbdAdm
private static final long FORCE_DETACH_RETRY_WAIT_MS = 250;
private static final String BITMAP_LEAK_ERR_MSG = "already has a bitmap, this should not happen";
private static final Pattern BITMAP_LEAK_MINOR_PATTERN = Pattern.compile("\\bminor\\s+(\\d+)\\b");
+ private static final int SECONDARY_RETRY_MAX_ATTEMPTS = 7;
+ private static final long SECONDARY_RETRY_INITIAL_BACKOFF_MS = 250L;
+ private static final long SECONDARY_RETRY_MAX_BACKOFF_MS = 2000L;
+ private static final String DEVICE_HELD_OPEN_ERR_MSG = "Device is held open by someone";
private final ExtCmdFactory extCmdFactory;
private final AccessContext sysCtx;
@@ -945,6 +949,79 @@ public class DrbdAdm
return true;
}
+ /**
+ * Switches a resource to secondary mode after mkfs, retrying when DRBD reports the device as held
+ * open by another process. On Talos and other systems with active block-device probing
+ * (machined/udev), the kernel uevent for a freshly-promoted device may cause an external probe
+ * to open the device just before LINSTOR demotes it back to Secondary, producing
+ * "Device is held open by someone" (errno -12) and aborting resource initialization.
+ * Probes typically release the fd within milliseconds, so a short retry with exponential backoff
+ * is sufficient to win the race without changing the contract of {@link #secondary}.
+ */
+ public void secondaryAfterMkfs(DrbdRscData<Resource> drbdRscData) throws ExtCmdFailedException
+ {
+ String[] commandArr = new String[] {
+ DRBDADM_UTIL, "-vvv", "secondary", drbdRscData.getSuffixedResourceName()
+ };
+
+ File nullDevice = new File(Platform.nullDevice());
+
+ OutputData lastOutput = null;
+ for (int attempt = 1; attempt <= SECONDARY_RETRY_MAX_ATTEMPTS; attempt++)
+ {
+ try
+ {
+ ExtCmd extCmd = extCmdFactory.create();
+ if (Platform.isWindows())
+ {
+ extCmd.setTimeout(TimeoutType.WAIT, 5 * 60 * 1000);
+ }
+
+ lastOutput = extCmd.pipeExec(ProcessBuilder.Redirect.from(nullDevice), commandArr);
+ if (lastOutput.exitCode == 0)
+ {
+ return;
+ }
+ if (!isDeviceHeldOpenError(lastOutput) || attempt == SECONDARY_RETRY_MAX_ATTEMPTS)
+ {
+ throw new ExtCmdFailedException(commandArr, lastOutput);
+ }
+ }
+ catch (ChildProcessTimeoutException timeoutExc)
+ {
+ throw new ExtCmdFailedException(commandArr, timeoutExc);
+ }
+ catch (IOException ioExc)
+ {
+ throw new ExtCmdFailedException(commandArr, ioExc);
+ }
+
+ long sleepMs = Math.min(
+ SECONDARY_RETRY_INITIAL_BACKOFF_MS * (1L << (attempt - 1)),
+ SECONDARY_RETRY_MAX_BACKOFF_MS
+ );
+ try
+ {
+ Thread.sleep(sleepMs);
+ }
+ catch (InterruptedException ie)
+ {
+ Thread.currentThread().interrupt();
+ throw new ExtCmdFailedException(commandArr, lastOutput);
+ }
+ }
+ }
+
+ static boolean isDeviceHeldOpenError(OutputData outputData)
+ {
+ if (outputData == null || outputData.stderrData == null)
+ {
+ return false;
+ }
+ String stderr = new String(outputData.stderrData, StandardCharsets.UTF_8);
+ return stderr.contains(DEVICE_HELD_OPEN_ERR_MSG);
+ }
+
public static class DrbdPrimary implements AutoCloseable
{
private final DrbdAdm drbdAdm;
@@ -963,7 +1040,7 @@ public class DrbdAdm
@Override
public void close() throws ExtCmdFailedException
{
- drbdAdm.secondary(drbdRscData);
+ drbdAdm.secondaryAfterMkfs(drbdRscData);
}
}
}
@@ -1,67 +0,0 @@
diff --git a/controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlSnapshotRestoreApiCallHandler.java b/controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlSnapshotRestoreApiCallHandler.java
index a639c27cc..4dabc1a83 100644
--- a/controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlSnapshotRestoreApiCallHandler.java
+++ b/controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlSnapshotRestoreApiCallHandler.java
@@ -434,26 +434,39 @@ public class CtrlSnapshotRestoreApiCallHandler
}
}
- /*
- * When a snapshot was received as a backup, it is marked to re-create its local metadata. Before the
- * snapshot was sent the new DRBD_INITIALIZED flag and the VlmDfn property InitialUpToDateOn were
+ /**
+ * <p>Sets the {@value InternalApiConsts#KEY_LINSTOR_DRBD_INITIAL_UPTODATE_ON} property to the nodeName
+ * of the first Snapshot (which has to have a disk) of the given SnapshotDefinition.</p>
+ *
+ * <p>If this SnapshotDefinition is a backup, we also unset the {@link VolumeDefinition.Flags#DRBD_INITIALIZED}
+ * flag. Otherwise we ensure the same flag is set.</p>
+ *
+ * <p>Backup path: When a snapshot was received as a backup, it is marked to re-create its local metadata.
+ * Before the snapshot was sent the new DRBD_INITIALIZED flag and the VlmDfn property InitialUpToDateOn were
* cleared. If DRBD_INITIALZED would be set, no peer would even read the property (on purpose).
* Without the property, no peer would think of itself as a winner, thus the DRBD resource would never
* become UpToDate.
* To avoid this, this method sets the InitialUpToDateOn property to let the first resource win the
- * race, so it can become UpToDate and therefore also pull the other peers eventually into UpToDate state.
+ * race, so it can become UpToDate and therefore also pull the other peers eventually into UpToDate state. </p>
*
- * If the given SnapDfn had also non-backup snapshots, those should be able to keep their metadata.
+ * <p>If the given SnapDfn had also non-backup snapshots, those should be able to keep their metadata.
* In that case we do not have to do anything. If however all snapshots are descendants from backups
* we must pick one and decide the UpToDate race upfront so that the winner can set its local DRBD
- * as UpToDate, which also triggers the controller to set the DRBD_INITIALZED flag eventually.
+ * as UpToDate, which also triggers the controller to set the DRBD_INITIALZED flag eventually.</p>
+ *
+ * <p>Non-Backup path: When a regular snapshot (i.e. not from a backup) is restored, we can or even should
+ * assume that the backing snapshot LVs have correct metadata which do not need be be initialized. When
+ * a SnapshotVolumeDefinition is created it unfortunately does not properly store the VolumeDefinition's flags,
+ * therefore the restored VolumeDefinition will also not have the DRBD_INITIALIZED flag set, although it
+ * refers to a device with healthy and UpToDate metadata. To fix this issue we simply set the flag in this case.</p>
*/
private void checkDrbdInitializedState(ResourceDefinition rscDfnRef) throws AccessDeniedException
{
AccessContext accCtx = peerAccCtx.get();
- if (DrbdLayerUtils.isForceInitialSyncSet(accCtx, rscDfnRef) && rscDfnRef.getDiskfulCount(accCtx) > 0)
+ if (rscDfnRef.getDiskfulCount(accCtx) > 0)
{
String winnerNodeName = rscDfnRef.getDiskfulResources(accCtx).get(0).getNode().getName().value;
+ boolean uninitializeDrbd = DrbdLayerUtils.isForceInitialSyncSet(accCtx, rscDfnRef);
Iterator<VolumeDefinition> vlmDfnIt = rscDfnRef.iterateVolumeDfn(accCtx);
while (vlmDfnIt.hasNext())
{
@@ -462,7 +475,14 @@ public class CtrlSnapshotRestoreApiCallHandler
{
vlmDfn.getProps(accCtx)
.setProp(InternalApiConsts.KEY_LINSTOR_DRBD_INITIAL_UPTODATE_ON, winnerNodeName);
- vlmDfn.getFlags().disableFlags(accCtx, VolumeDefinition.Flags.DRBD_INITIALIZED);
+ if (uninitializeDrbd)
+ {
+ vlmDfn.getFlags().disableFlags(accCtx, VolumeDefinition.Flags.DRBD_INITIALIZED);
+ }
+ else
+ {
+ vlmDfn.getFlags().enableFlags(accCtx, VolumeDefinition.Flags.DRBD_INITIALIZED);
+ }
}
catch (InvalidKeyException | InvalidValueException exc)
{
@@ -36,6 +36,12 @@ spec:
enabled: true
podTemplate:
spec:
{{- /* run-migration upgrades the database in place: left on the
operator's default image it migrates ahead of the controller,
irreversibly. */}}
initContainers:
- name: run-migration
image: {{ .Values.piraeusServer.image.repository }}:{{ .Values.piraeusServer.image.tag }}
containers:
- name: linstor-controller
image: {{ .Values.piraeusServer.image.repository }}:{{ .Values.piraeusServer.image.tag }}
@@ -11,6 +11,9 @@ spec:
spec:
# host-network is recommended by Piraeus while it is not default in the upstream
hostNetwork: true
initContainers:
- name: setup-lvm-configuration
image: {{ .Values.piraeusServer.image.repository }}:{{ .Values.piraeusServer.image.tag }}
containers:
- name: linstor-satellite
image: {{ .Values.piraeusServer.image.repository }}:{{ .Values.piraeusServer.image.tag }}
@@ -34,7 +34,7 @@ tests:
# Until there is, the re-check on a bump is a human obligation, stated in
# templates/cluster.yaml beside the list.
#
# The list was last verified complete against piraeus-operator v2.10.2:
# The list was last verified complete against piraeus-operator v2.12.1:
# eleven upstream flags plus --strict-topology.
- it: the restated argument list is complete
template: templates/cluster.yaml
@@ -0,0 +1,40 @@
suite: every LINSTOR server container runs the platform image
# piraeus-operator ships its own default for the LINSTOR server image and
# applies it to every container this package leaves unset. The init
# containers matter most: run-migration upgrades the LINSTOR database in
# place, so an operator default newer than the platform build migrates the
# schema ahead of the controller, which then refuses to start, and the
# migration cannot be undone. Pinning them to the same image as the main
# containers makes the database move only together with the controller.
templates:
- templates/cluster.yaml
- templates/satellites-cozy.yaml
set:
piraeusServer:
image:
repository: registry.example/piraeus-server
tag: v0.0.0-test
tests:
- it: the controller migration runs the platform image
template: templates/cluster.yaml
asserts:
- equal:
path: spec.controller.podTemplate.spec.initContainers[?(@.name=="run-migration")].image
value: registry.example/piraeus-server:v0.0.0-test
- equal:
path: spec.controller.podTemplate.spec.containers[?(@.name=="linstor-controller")].image
value: registry.example/piraeus-server:v0.0.0-test
- it: the satellite LVM setup runs the platform image
template: templates/satellites-cozy.yaml
asserts:
- equal:
path: spec.podTemplate.spec.initContainers[?(@.name=="setup-lvm-configuration")].image
value: registry.example/piraeus-server:v0.0.0-test
- equal:
path: spec.podTemplate.spec.containers[?(@.name=="linstor-satellite")].image
value: registry.example/piraeus-server:v0.0.0-test
@@ -151,13 +151,17 @@ spec:
*SecretName using the issuer configured here.
properties:
group:
description: Group of the resource being referred to.
description: |-
Group of the issuer being referred to.
Defaults to 'cert-manager.io'.
type: string
kind:
description: Kind of the resource being referred to.
description: |-
Kind of the issuer being referred to.
Defaults to 'Issuer'.
type: string
name:
description: Name of the resource being referred to.
description: Name of the issuer being referred to.
type: string
required:
- name
@@ -201,6 +205,11 @@ spec:
type: object
x-kubernetes-map-type: atomic
x-kubernetes-preserve-unknown-fields: true
replicas:
description: Number of desired pods. Defaults to 1.
format: int32
minimum: 0
type: integer
type: object
csiController:
description: CSIController controls the deployment of the CSI Controller
@@ -320,13 +329,17 @@ spec:
issuer configured here.
properties:
group:
description: Group of the resource being referred to.
description: |-
Group of the issuer being referred to.
Defaults to 'cert-manager.io'.
type: string
kind:
description: Kind of the resource being referred to.
description: |-
Kind of the issuer being referred to.
Defaults to 'Issuer'.
type: string
name:
description: Name of the resource being referred to.
description: Name of the issuer being referred to.
type: string
required:
- name
@@ -345,6 +358,18 @@ spec:
* Store credentials for accessing remotes for backups.
See https://linbit.com/drbd-user-guide/linstor-guide-1_0-en/#s-encrypt_commands for more information.
type: string
maxConcurrentEvacuations:
description: |-
MaxConcurrentEvacuations limits how many Satellites are evacuated at the same time when using the
"Evacuate" deletion policy.
This applies both to Satellites removed because their node no longer matches the cluster, and to nodes
removed through ClusterAPI. While the limit is reached, additional Satellites wait for a free slot before
their volumes are moved and, for ClusterAPI managed nodes, before the node is allowed to drain.
A value of 0 (the default) places no limit on the number of concurrent evacuations.
format: int32
minimum: 0
type: integer
nfsServer:
description: NFSServer controls the deployment of the LINSTOR CSI
NFS Server DaemonSet.
@@ -568,9 +593,10 @@ spec:
operator:
description: |-
Operator represents a key's relationship to the value.
Valid operators are Exists and Equal. Defaults to Equal.
Valid operators are Exists, Equal, Lt, and Gt. Defaults to Equal.
Exists is equivalent to wildcard for value, so that a pod can
tolerate all taints of a particular category.
Lt and Gt perform numeric comparisons (requires feature gate TaintTolerationComparisonOperators).
type: string
tolerationSeconds:
description: |-
@@ -1054,13 +1080,17 @@ spec:
issuer configured here.
properties:
group:
description: Group of the resource being referred to.
description: |-
Group of the issuer being referred to.
Defaults to 'cert-manager.io'.
type: string
kind:
description: Kind of the resource being referred to.
description: |-
Kind of the issuer being referred to.
Defaults to 'Issuer'.
type: string
name:
description: Name of the resource being referred to.
description: Name of the issuer being referred to.
type: string
required:
- name
@@ -1321,6 +1351,15 @@ spec:
x-kubernetes-list-map-keys:
- name
x-kubernetes-list-type: map
resourceNameSuffixSeparator:
description: |-
ResourceNameSuffixSeparator is inserted between the base resource name and the Satellite name.
Defaults to ".".
enum:
- .
- '-'
type: string
storagePools:
description: StoragePools is a list of storage pools to configure
on the node.
@@ -1347,18 +1386,66 @@ spec:
lvmPool:
description: Configures a LVM Volume Group as storage pool.
properties:
volumeGroup:
externalLocking:
description: |-
ExternalLocking skips LINSTOR's internal locking for the shared VG, relying on an external
lock manager such as lvmlockd on the hosts instead.
type: boolean
physicalVolumeCreateArguments:
description: |-
PhysicalVolumeCreateArguments are arguments to pass to "pvcreate".
This has no effect on an existing VG, it only applies when the VG initially gets created.
items:
type: string
type: array
sharedSpace:
description: |-
SharedSpace marks the VG as shared between multiple nodes, identified by the given name.
All storage pools backed by the same physical storage must use the same shared space name.
A shared VG must be set up manually on the hosts, it cannot be created from source devices.
type: string
volumeGroup:
description: VolumeGroup is the name of the Volume Group
(VG) to use.
type: string
volumeGroupCreateArguments:
description: |-
VolumeGroupCreateArguments are arguments to pass to "vgcreate".
This has no effect on an existing VG, it only applies when the VG initially gets created.
items:
type: string
type: array
type: object
lvmThinPool:
description: Configures a LVM Thin Pool as storage pool.
properties:
logicalVolumeCreateArguments:
description: |-
LogicalVolumeCreateArguments are arguments to pass to "lvcreate".
This has no effect on an existing thinpool, it only applies when the thinpool initially gets created.
items:
type: string
type: array
physicalVolumeCreateArguments:
description: |-
PhysicalVolumeCreateArguments are arguments to pass to "pvcreate".
This has no effect on an existing VG, it only applies when the VG initially gets created.
items:
type: string
type: array
thinPool:
description: ThinPool is the name of the thinpool LV (without
VG prefix).
type: string
volumeGroup:
type: string
volumeGroupCreateArguments:
description: |-
VolumeGroupCreateArguments are arguments to pass to "vgcreate".
This has no effect on an existing VG, it only applies when the VG initially gets created.
items:
type: string
type: array
type: object
name:
description: Name of the storage pool in linstor.
@@ -1447,6 +1534,16 @@ spec:
zPool:
description: ZPool is the name of the ZFS zpool.
type: string
zpoolCreateArguments:
description: |-
ZpoolCreateArguments are arguments to pass to "zpool create".
This has no effect on an existing zpool, it only applies when the zpool initially gets created.
If not set, "-o failmode=continue" is automatically added to ensure a failed zpool does not stop replication
to and from other nodes in the cluster.
items:
type: string
type: array
type: object
zfsThinPool:
description: Configures a ZFS system based storage pool, allocating
@@ -1455,6 +1552,16 @@ spec:
zPool:
description: ZPool is the name of the ZFS zpool.
type: string
zpoolCreateArguments:
description: |-
ZpoolCreateArguments are arguments to pass to "zpool create".
This has no effect on an existing zpool, it only applies when the zpool initially gets created.
If not set, "-o failmode=continue" is automatically added to ensure a failed zpool does not stop replication
to and from other nodes in the cluster.
items:
type: string
type: array
type: object
required:
- name
@@ -1762,13 +1869,17 @@ spec:
issuer configured here.
properties:
group:
description: Group of the resource being referred to.
description: |-
Group of the issuer being referred to.
Defaults to 'cert-manager.io'.
type: string
kind:
description: Kind of the resource being referred to.
description: |-
Kind of the issuer being referred to.
Defaults to 'Issuer'.
type: string
name:
description: Name of the resource being referred to.
description: Name of the issuer being referred to.
type: string
required:
- name
@@ -1925,6 +2036,15 @@ spec:
repository:
description: Repository used to pull workload images.
type: string
resourceNameSuffixSeparator:
description: |-
ResourceNameSuffixSeparator is inserted between the base resource name and the Satellite name.
Defaults to ".".
enum:
- .
- '-'
type: string
storagePools:
description: StoragePools is a list of storage pools to configure
on the node.
@@ -1951,18 +2071,66 @@ spec:
lvmPool:
description: Configures a LVM Volume Group as storage pool.
properties:
volumeGroup:
externalLocking:
description: |-
ExternalLocking skips LINSTOR's internal locking for the shared VG, relying on an external
lock manager such as lvmlockd on the hosts instead.
type: boolean
physicalVolumeCreateArguments:
description: |-
PhysicalVolumeCreateArguments are arguments to pass to "pvcreate".
This has no effect on an existing VG, it only applies when the VG initially gets created.
items:
type: string
type: array
sharedSpace:
description: |-
SharedSpace marks the VG as shared between multiple nodes, identified by the given name.
All storage pools backed by the same physical storage must use the same shared space name.
A shared VG must be set up manually on the hosts, it cannot be created from source devices.
type: string
volumeGroup:
description: VolumeGroup is the name of the Volume Group
(VG) to use.
type: string
volumeGroupCreateArguments:
description: |-
VolumeGroupCreateArguments are arguments to pass to "vgcreate".
This has no effect on an existing VG, it only applies when the VG initially gets created.
items:
type: string
type: array
type: object
lvmThinPool:
description: Configures a LVM Thin Pool as storage pool.
properties:
logicalVolumeCreateArguments:
description: |-
LogicalVolumeCreateArguments are arguments to pass to "lvcreate".
This has no effect on an existing thinpool, it only applies when the thinpool initially gets created.
items:
type: string
type: array
physicalVolumeCreateArguments:
description: |-
PhysicalVolumeCreateArguments are arguments to pass to "pvcreate".
This has no effect on an existing VG, it only applies when the VG initially gets created.
items:
type: string
type: array
thinPool:
description: ThinPool is the name of the thinpool LV (without
VG prefix).
type: string
volumeGroup:
type: string
volumeGroupCreateArguments:
description: |-
VolumeGroupCreateArguments are arguments to pass to "vgcreate".
This has no effect on an existing VG, it only applies when the VG initially gets created.
items:
type: string
type: array
type: object
name:
description: Name of the storage pool in linstor.
@@ -2051,6 +2219,16 @@ spec:
zPool:
description: ZPool is the name of the ZFS zpool.
type: string
zpoolCreateArguments:
description: |-
ZpoolCreateArguments are arguments to pass to "zpool create".
This has no effect on an existing zpool, it only applies when the zpool initially gets created.
If not set, "-o failmode=continue" is automatically added to ensure a failed zpool does not stop replication
to and from other nodes in the cluster.
items:
type: string
type: array
type: object
zfsThinPool:
description: Configures a ZFS system based storage pool, allocating
@@ -2059,6 +2237,16 @@ spec:
zPool:
description: ZPool is the name of the ZFS zpool.
type: string
zpoolCreateArguments:
description: |-
ZpoolCreateArguments are arguments to pass to "zpool create".
This has no effect on an existing zpool, it only applies when the zpool initially gets created.
If not set, "-o failmode=continue" is automatically added to ensure a failed zpool does not stop replication
to and from other nodes in the cluster.
items:
type: string
type: array
type: object
required:
- name
@@ -1,3 +1,7 @@
apiVersion: v2
name: cozy-piraeus-operator
version: 0.0.0 # Placeholder, the actual version will be automatically set during the build process
dependencies:
- name: piraeus
version: "*"
repository: file://charts/piraeus
@@ -3,8 +3,8 @@ name: piraeus
description: |
The Piraeus Operator manages software defined storage clusters using LINSTOR in Kubernetes.
type: application
version: 2.10.2
appVersion: "v2.10.2"
version: 2.12.1
appVersion: "v2.12.1"
maintainers:
- name: Piraeus Datastore
url: https://piraeus.io
@@ -3,6 +3,7 @@ apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "piraeus-operator.fullname" . }}-image-config
namespace: {{ .Release.Namespace }}
labels:
{{- include "piraeus-operator.labels" . | nindent 4 }}
data:
@@ -17,38 +18,38 @@ data:
# quay.io/piraeusdatastore/piraeus-server:v1.24.2
components:
linstor-controller:
tag: v1.32.3
tag: v1.35.3
image: piraeus-server
linstor-satellite:
tag: v1.32.3
tag: v1.35.3
image: piraeus-server
linstor-csi:
tag: v1.10.3
tag: v1.13.2
image: piraeus-csi
nfs-server:
tag: v1.10.3
tag: v1.13.2
image: piraeus-csi-nfs-server
drbd-reactor:
tag: v1.10.0
tag: v1.12.0
image: drbd-reactor
ha-controller:
tag: v1.3.1
tag: v1.3.4
image: piraeus-ha-controller
drbd-shutdown-guard:
tag: v1.1.1
tag: v1.1.2
image: drbd-shutdown-guard
ktls-utils:
tag: v1.2.1
tag: v1.5.0
image: ktls-utils
linstor-affinity-controller:
tag: v1.3.0
tag: v1.5.0
image: linstor-affinity-controller
drbd-module-loader:
tag: v9.2.16
tag: v9.3.4
# The special "match" attribute is used to select an image based on the node's reported OS.
# The operator will first check the k8s node's ".status.nodeInfo.osImage" field, and compare it against the list
# here. If one matches, that specific image name will be used instead of the fallback image.
image: drbd9-noble # Fallback image: chose a recent kernel, which can hopefully compile whatever config is actually in use
image: drbd9-resolute # Fallback image: chose a recent kernel, which can hopefully compile whatever config is actually in use
match:
- osImage: Red Hat Enterprise Linux 8\.
image: drbd9-almalinux8
@@ -84,6 +85,10 @@ data:
image: drbd9-jammy
- osImage: Ubuntu 24\.04
image: drbd9-noble
- osImage: Ubuntu 26\.04
image: drbd9-resolute
- osImage: Debian GNU/Linux 13
image: drbd9-trixie
- osImage: Debian GNU/Linux 12
image: drbd9-bookworm
- osImage: Debian GNU/Linux 11
@@ -93,25 +98,22 @@ data:
base: registry.k8s.io/sig-storage
components:
csi-attacher:
tag: v4.10.0
tag: v4.13.0
image: csi-attacher
csi-livenessprobe:
tag: v2.17.0
tag: v2.20.0
image: livenessprobe
csi-provisioner:
tag: v6.1.0
tag: v6.4.0
image: csi-provisioner
csi-snapshotter:
tag: v8.4.0
tag: v8.7.0
image: csi-snapshotter
csi-resizer:
tag: v2.0.0
tag: v2.3.0
image: csi-resizer
csi-external-health-monitor-controller:
tag: v0.16.0
image: csi-external-health-monitor-controller
csi-node-driver-registrar:
tag: v2.15.0
tag: v2.18.0
image: csi-node-driver-registrar
{{- range $idx, $value := .Values.imageConfigOverride }}
{{ add $idx 1 }}_helm_override.yaml: |
File diff suppressed because it is too large. Load diff
@@ -3,6 +3,7 @@ apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "piraeus-operator.fullname" . }}-controller-manager
namespace: {{ .Release.Namespace }}
labels:
{{- include "piraeus-operator.labels" . | nindent 4 }}
spec:
@@ -70,6 +71,9 @@ spec:
- mountPath: /tmp/k8s-webhook-server/serving-certs
name: cert
readOnly: true
{{- with .Values.operator.extraVolumeMounts }}
{{- toYaml . | nindent 8 }}
{{- end }}
{{- if .Values.kubeRbacProxy.enabled }}
- args:
- --secure-listen-address=0.0.0.0:8443
@@ -91,8 +95,11 @@ spec:
securityContext:
{{- toYaml .Values.kubeRbacProxy.securityContext | nindent 12}}
volumeMounts:
- mountPath: /etc/tls
name: cert
- mountPath: /etc/tls
name: cert
{{- with .Values.kubeRbacProxy.extraVolumeMounts }}
{{- toYaml . | nindent 8 }}
{{- end }}
{{- end }}
securityContext:
runAsNonRoot: true
@@ -100,6 +107,10 @@ spec:
{{ toYaml . | nindent 8 }}
{{- end }}
serviceAccountName: {{ include "piraeus-operator.serviceAccountName" . }}
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
terminationGracePeriodSeconds: 10
priorityClassName: {{ .Values.priorityClassName | default "system-cluster-critical" }}
nodeSelector:
@@ -113,3 +124,6 @@ spec:
secret:
defaultMode: 420
secretName: {{ include "piraeus-operator.certifcateName" . }}
{{- with .Values.extraVolumes }}
{{- toYaml . | nindent 6 }}
{{- end }}
@@ -0,0 +1,15 @@
{{- /* Normalize extraObjects to a list, easier to loop over */ -}}
{{- $extraObjects := .Values.extraManifests | default (list) -}}
{{- if kindIs "map" $extraObjects -}}
{{- $extraObjects = values $extraObjects -}}
{{- end -}}
{{- range $extraObjects }}
---
{{- if kindIs "map" . }}
{{- tpl (toYaml .) $ | nindent 0 }}
{{- else if kindIs "string" . }}
{{- tpl . $ | nindent 0 }}
{{- end }}
{{- end }}
@@ -2,6 +2,7 @@ apiVersion: v1
kind: Service
metadata:
name: {{ include "piraeus-operator.fullname" . }}-controller-manager-metrics-service
namespace: {{ .Release.Namespace }}
labels:
{{- include "piraeus-operator.labels" . | nindent 4 }}
spec:
@@ -15,7 +15,6 @@ rules:
- events
- persistentvolumes
- pods
- secrets
- serviceaccounts
- services
verbs:
@@ -59,6 +58,13 @@ rules:
- pods/eviction
verbs:
- create
- apiGroups:
- ""
resources:
- pods/exec
verbs:
- create
- get
- apiGroups:
- apiextensions.k8s.io
resources:
@@ -71,6 +77,12 @@ rules:
- patch
- update
- watch
- apiGroups:
- apiextensions.k8s.io
resources:
- customresourcedefinitions/status
verbs:
- patch
- apiGroups:
- apps
resources:
@@ -235,7 +247,6 @@ rules:
- snapshot.storage.k8s.io
resources:
- volumesnapshotclasses
- volumesnapshots
verbs:
- get
- list
@@ -257,6 +268,15 @@ rules:
verbs:
- patch
- update
- apiGroups:
- snapshot.storage.k8s.io
resources:
- volumesnapshots
verbs:
- get
- list
- update
- watch
- apiGroups:
- storage.k8s.io
resources:
@@ -283,6 +303,7 @@ rules:
- storage.k8s.io
resources:
- storageclasses
- volumeattributesclasses
verbs:
- get
- list
@@ -306,8 +327,30 @@ rules:
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: {{ include "piraeus-operator.fullname" . }}-controller-manager
namespace: {{ .Release.Namespace }}
labels:
{{- include "piraeus-operator.labels" . | nindent 4 }}
rules:
- apiGroups:
- ""
resources:
- secrets
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: {{ include "piraeus-operator.fullname" . }}-leader-election
namespace: {{ .Release.Namespace }}
labels:
{{- include "piraeus-operator.labels" . | nindent 4 }}
rules:
@@ -17,6 +17,22 @@ subjects:
namespace: {{ .Release.Namespace }}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: {{ include "piraeus-operator.fullname" . }}-controller-manager
namespace: {{ .Release.Namespace }}
labels:
{{- include "piraeus-operator.labels" . | nindent 4 }}
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: {{ include "piraeus-operator.fullname" . }}-controller-manager
subjects:
- kind: ServiceAccount
name: {{ include "piraeus-operator.serviceAccountName" . }}
namespace: {{ .Release.Namespace }}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: {{ include "piraeus-operator.fullname" . }}-controller-manager
@@ -4,6 +4,7 @@ apiVersion: v1
kind: ServiceAccount
metadata:
name: {{ include "piraeus-operator.serviceAccountName" . }}
namespace: {{ .Release.Namespace }}
labels:
{{- include "piraeus-operator.labels" . | nindent 4 }}
{{ end }}
@@ -9,6 +9,7 @@ apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: {{ include "piraeus-operator.fullname" . }}
namespace: {{ .Release.Namespace }}
labels:
{{- include "piraeus-operator.labels" . | nindent 4 }}
spec:
@@ -36,6 +37,7 @@ apiVersion: v1
kind: Secret
metadata:
name: {{ include "piraeus-operator.certifcateName" . }}
namespace: {{ .Release.Namespace }}
labels:
{{- include "piraeus-operator.labels" . | nindent 4 }}
type: kubernetes.io/tls
@@ -2,6 +2,7 @@ apiVersion: v1
kind: Service
metadata:
name: {{ include "piraeus-operator.fullname" . }}-webhook-service
namespace: {{ .Release.Namespace }}
labels:
{{- include "piraeus-operator.labels" . | nindent 4 }}
spec:
@@ -29,6 +29,12 @@ operator:
# cpu: 100m
# memory: 128Mi
## Additional volume mounts
extraVolumeMounts: [ ]
# - name: clusterapi-kubeconfig
# mountPath: /etc/operator
# readOnly: true
kubeRbacProxy:
enabled: false
image:
@@ -54,6 +60,16 @@ kubeRbacProxy:
# cpu: 100m
# memory: 128Mi
## Additional volume mounts
extraVolumeMounts: [ ]
## Additional volumes
extraVolumes: []
# - name: clusterapi-kubeconfig
# secret:
# secretName: super-secret
webhook:
timeoutSeconds: 2
failurePolicy: Fail
@@ -111,3 +127,30 @@ imageConfigOverride: []
# tag: my-custom-tag
# Results in the image example.com/piraeus/linstor-csi:my-custom-tag being used.
# See templates/config.yaml for available components.
## Extra manifests to deploy. Can be of type dict or list.
## If dict, keys are ignored and only values are used.
## Items contained within extraObjects can be defined as dict or string and are passed through tpl.
extraManifests: null
# - apiVersion: v1
# kind: ConfigMap
# metadata:
# labels:
# name: piraeus-extra
# data:
# extra-data: "value"
#
# can also be defined as a string, useful for templating field names
# - |
# apiVersion: v1
# kind: Secret
# type: Opaque
# metadata:
# name: super-secret
# labels:
# {{- range $key, $value := .Values.commonLabels }}
# {{ $key }}: {{ $value }}
# {{- end }}
# data:
# plaintext: Zm9vYmFy
# templated: '{{ print "foobar" | upper | b64enc }}'