## What this PR does
Restoring a PVC from a LINSTOR VolumeSnapshot can leave one replica with
an empty volume that DRBD marks UpToDate. A pod that lands on that
replica sees an empty filesystem.
linstor-csi restores the snapshot on one node and autoplaces the other
replicas. LINSTOR v1.33.3 copies `Linstor/Drbd/InitialUptoDateOn` from
the source volume into the restored volume definition and leaves the
definition without `DRBD_INITIALIZED`. When autoplace puts a replica on
the node that name points to, that empty new volume wins the initial
UpToDate race, gets a random current UUID and a fresh filesystem, and
goes StandAlone against the restored replica.
LINBIT fixed this in v1.34.0. This PR backports that commit,
LINBIT/linstor-server@9d8cdf5, to our v1.33.3 build as a patch. The
README entry says to drop the patch with the move to v1.34.0 or later.
Tested on a three-node cluster with a thick ZFS pool and
`placementCount: 3`. Before the change the reproduction failed every
time. With the patch I ran 5 restores, 4 of them with the source
volume's winner on a different node than the restore. I also restored a
snapshot taken before the change, and a 10Gi volume. Every time all
replicas came up connected and UpToDate, a pod on each node read the
same checksum, and the backing zvols were byte-identical when read with
`O_DIRECT`. The resync after a restore moves about the written data,
around 40 s for 1Gi and 134 s for 7.5 GiB on 10Gi. CSI clone PVC-to-PVC
was checked too and is not affected.
There is no e2e test. Every e2e lane runs without DRBD
(`COZY_LINSTOR_DRBD_ENABLED=false`, StorageClass `local`), so a suite
for replicated restores would never run in CI.
Volumes restored before this fix are not repaired by it. How to find and
fix them is in [this
comment](https://github.com/cozystack/cozystack/issues/4862#issuecomment-6106661441).
Known limitation, shared by upstream v1.34 since that code did not
change: the autoplaced replicas skip the initial sync and rely on the
source volume's day0 UUID still being in the restored replica's history.
If that ever fails, DRBD goes StandAlone with "unrelated data" instead
of syncing, which is visible, not silent. I could not reproduce it.
Closes#4862
### Screenshots
Not a UI change.
### Downstream repositories
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(linstor): a PVC restored from a VolumeSnapshot no longer gets a replica with an empty volume marked UpToDate. Volumes restored before this release may still have such a replica; https://github.com/cozystack/cozystack/issues/4862#issuecomment-6106661441 explains how to check and repair them.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Restored snapshots with diskful DRBD resources are now recognized as
initialized by default, helping them resume without unnecessary metadata
recreation.
* When force-initial-sync is enabled, restored volumes continue through
the existing metadata-recreation path.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
linstor-csi restores a snapshot on one node and autoplaces the other
replicas. In LINSTOR v1.33.3 the restored volume definition has no
DRBD_INITIALIZED flag, so the initial UpToDate race is still open, and
Linstor/Drbd/InitialUptoDateOn, copied from the source volume through
the snapshot, names the node that won it for the source. When a
replica is autoplaced on that node, the empty new volume wins the
race: it gets a random current UUID and a fresh filesystem, and goes
StandAlone against the restored replica. A pod attached there sees an
empty volume.
Backport the upstream fix, released in LINSTOR v1.34.0, which sets the
flag on a restore unless the resource definition asks for new metadata
through ForceInitialSyncPermanent, as a backup does. Nothing wins the
race any more, and an autoplaced replica resyncs from the restored one
like any replica added to an existing volume.
Link: https://github.com/LINBIT/linstor-server/commit/9d8cdf5118338fc72a34bb99799e452e031ea8f6Closes: #4862
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
When CDI clones a golden image into a StorageClass other than the one
the golden PVC lives in, its csi-clone path, and its snapshot path once
it gets a VolumeSnapshotClass of the provisioner, check the two classes
for a common driver, a matching volume mode and room to expand, and
never compare the class names. A target class of the same provisioner
therefore gets no host-assisted fallback and the clone is left to the
CSI driver. LINSTOR creates that clone next to the source data, so on a
target class that does not allow remote volume access from every node,
such as a node-local one, the temporary clone PVC stays unbound with
nothing surfaced to the user.
Refuse such a disk at render time, mirroring the planner's inputs: the
CDI override or the target StorageProfile's strategy, the snapshot class
CDI would pick, the golden's driver as its PV records it, the target's
provisioner, the volume modes and the target's expansion setting, and
refuse only a LINSTOR target whose remote access policy, parsed the way
linstor-csi parses it, is not plainly "from anywhere". A snapshot class
both StorageProfiles name but that does not exist is refused as well,
because CDI then fails to plan the clone instead of falling back. An
empty storageClass is resolved the way CDI resolves it for a kubevirt
DataVolume. A disk whose DataVolume or PVC already exists is exempt,
since CDI can garbage-collect a completed DataVolume. Combinations that
fall back to a host-assisted copy, targets reachable from any node, and
anything else the lookups cannot find still render.
This changes behaviour for a new disk in a LINSTOR class without remote
access from every node, and for one whose StorageProfiles name a
snapshot class that does not exist: both rendered before and are refused
now. The way out is the golden image's class, or the copy clone
strategy.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->
## What this PR does
One transient error from proxy.golang.org is enough to fail an image
build or the `Unit & controller tests` job today. The error is an HTTP/2
stream reset (`stream error ... INTERNAL_ERROR`), and nothing retries or
falls back after it. This PR adds three things: a direct fallback, a
bounded retry and a module cache.
Image builds now fall back to fetching modules from their origin when
the proxy fails. They get `GOPROXY=https://proxy.golang.org|direct` as a
build arg from `hack/common-envs.mk`, and every Go builder stage
declares `ARG GOPROXY`. With a pipe Go goes direct on any proxy error.
With Go's default comma it does that only on a 404 or 410. The value
comes from `IMAGE_GOPROXY` and not from the host's `GOPROXY`, because
buildkit can't reach a mirror on localhost or a `file://` proxy. Every
workflow job that runs Go on the runner sets the same value as env.
Explicit module downloads in Dockerfiles are retried three times with a
growing pause. The loop is inline, in one form that the contract test
pins. I tried a shared script from a named build context first, but
buildx refuses any `--build-context` for the four Dockerfiles still
pinned to the `dockerfile:1.2` frontend. The images also build from
different contexts, so no single file can be copied in. Raising those
pins changes how those images build and needs a real build to check, so
I left it for a follow-up. Two alpine stages, token-proxy and
shadowbox's `ss-server`, left the download to `go build`. The alpine
golang images have no git, so the direct fallback can't fetch for them,
and they now get the same download step. The runner jobs use
`hack/go-mod-download.sh` with the same bound.
The unit test job now starts from a module cache instead of downloading
everything. It restores `GOMODCACHE` with `actions/cache`, keyed on
every `go.sum` in the tree, and then prefetches all tracked modules
through the retry, so a cache miss still works. A cache saved by a PR
run is visible only to that PR, but one saved on the default branch can
be restored by any PR
([docs](https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching#restrictions-for-accessing-a-cache)).
That is why the build cache warmer on main writes the same key before
its image build. This way the first run of a PR, which is the one that
flakes, starts warm. The warmer restores only an exact key, so old
module versions don't pile up, and its Go steps can't fail the image
build. Both runner labels are Linux, so the key matches. Both also check
out under `/home/ubuntu/_work`, so `go env GOMODCACHE` should give the
same cache path, but I have not seen it on the warmer's runner yet. The
release jobs in `tags.yaml` and `promote-rc.yaml` get the GOPROXY env
only, no cache.
`hack/go-module-fetch-contract.bats` covers all three. Each test is red
on main, and I checked each guard with a mutation.
Closes#4591
### Screenshots
<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->
### Downstream repositories
<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.
Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.
IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:
1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
ci(build): Go module downloads in image builds and CI jobs now fall back to direct fetching and retry on transient proxy.golang.org errors. The unit test job reuses a module cache written from main.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Reliability**
- Go module downloads now fall back to direct access when the public
proxy encounters errors.
- Builds and automated checks retry failed module downloads, helping
them recover from temporary network issues.
- **Performance**
- CI caches Go modules to reduce repeated downloads and speed up builds
and tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The image build ran go build only, and go build never compiles a
_test.go, so nothing ran the tests of the code the patches change. Run
go test in the builder stage on the packages the patches touch. The list
is read from the patches themselves, so a new patch is covered without
an edit to the Dockerfile.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A transient proxy.golang.org HTTP/2 stream reset fails an image build
outright: Go's default GOPROXY falls back to direct only on a 404 or
410. Image builds now get GOPROXY=https://proxy.golang.org|direct as a
build arg from hack/common-envs.mk, declared by every Go builder stage,
so any proxy error falls back to fetching from the origin. The value
comes from IMAGE_GOPROXY rather than the host's GOPROXY, which may
name a mirror that buildkit cannot reach. The alpine golang images
carry no git, so in their stages a module hosted on GitHub cannot be
fetched directly and the retry is the cover.
Explicit `go mod download` steps retry three times with a growing
pause. The loop is inline, in one form the contract test pins: the
images build from different contexts, so no shared script can be
copied in, and a named build context in the shared buildx arguments
makes buildx refuse every Dockerfile pinned to a frontend older than
dockerfile:1.4.
The two alpine stages that left the download to `go build` get the
same download step, because without git neither the fallback nor a
retry covered them.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
linstor-server's build always downloads the linux-x86_64 protoc, and
the builder stage runs on the build platform. The release now builds
the arm64 images on a native arm64 runner, where that protoc cannot
execute and generateJava fails. A patch picks the protoc build for the
host's architecture.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Package images can now be built for amd64 and arm64 in one `make image`,
and a published build is multi-arch unless `PLATFORM` says otherwise.
This is the build half of arm64 support from #1961 and #903. CI still
publishes amd64 only until it gets a native arm64 leg, which comes in a
separate PR.
Builder stages now run on the build platform and compile for the target
one, so Go never runs under emulation. Under emulation it is slow and it
crashes: the migration-controller Dockerfile already records a "bad
pointer in Go heap" from exactly that. On top of this, `PLATFORM`
defaults to `linux/amd64,linux/arm64`. A build on an arm64 workstation
can't publish an arm64-only digest any more. That happened once already,
the cilium pin was arm64-only until 3f36a1b45 rebuilt it by hand.
Two images were only right because they were emulated. On a native
builder they would ship the wrong architecture. token-proxy ran `go
build` with no GOARCH, and kube-ovn calls upstream `make build-go`,
which hardcodes `GOARCH=amd64`. Both are fixed now. The new bats test
wants every compiling stage on `$BUILDPLATFORM` and actually using
`TARGETARCH`. A bare `ARG TARGETARCH` does not count, this is how
token-proxy got through.
Some things stay amd64. talos and testing pin it with `override`,
because matchbox serves amd64 Talos assets and the sandbox runs
`qemu-system-x86_64`. `LOAD=1` builds for the host only, since the
classic docker image store can't load an index. Every CI build job pins
`PLATFORM: linux/amd64`, so CI output does not change.
This changes local builds. A `make image` on the default docker driver
with the classic image store now fails with "Multi-platform build is not
supported for the docker driver", where before it built for the host.
Use a `docker-container` builder (`BUILDER=<name>`), or pass
`PLATFORM=linux/amd64` (or `LOAD=1`) to build one architecture.
I checked it on an arm64 host with a docker-container builder.
cozystack-api, dashboard, kubeovn, platform-migrations, cilium and
capi-providers-cpprovider build and push as amd64+arm64 indexes.
Binaries pulled from each platform are x86-64 and aarch64:
cozystack-api, token-proxy, the kamaji provider manager, cilium-agent
and all four kube-ovn binaries. piraeus-server builds for both
platforms, but the push to my test registry failed on registry 500s. The
capi-providers-cpprovider inline cache works with a multi-platform push.
Not covered is running the arm64 images on an arm64 cluster, CI has
none.
This is stacked on #4485 and targets its branch, so the diff here is
only the two commits on top.
### Screenshots
No UI changes.
### Downstream repositories
- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#711
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
feat(build): package images build for linux/amd64 and linux/arm64 by default, with Go builders cross-compiling on the build platform; a local `make image` now needs a docker-container builder, or PLATFORM=linux/amd64 / LOAD=1 for a single architecture. CI still publishes amd64.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Build Improvements**
* Container image builds now target both amd64 and arm64 by default when
building without loading images locally.
* CI and release builds are pinned to amd64; Talos and testing images
remain amd64-only.
* Cross-platform builds now compile on the builder’s native platform
while targeting the requested architecture.
* **Documentation**
* Clarified platform defaults and architecture requirements for local
and CI builds.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The fabric8 client retries a POST after a 5xx or a dropped
connection, so a create that the apiserver already persisted can come
back as 409 AlreadyExists. A single-object transaction has no
rollback entry, so the controller drops the key from memory while the
CR stays in the apiserver, and the caching client never re-reads it:
every later write of that key fails the same way until the controller
restarts.
This left a CDI scratch PV stuck in Released, because linstor-csi
starts every volume delete by setting BalanceResourcesEnabled on the
resource definition and that write never succeeded.
Treat 409 on create as success when the object exists: take its
resourceVersion and replace it with the intended content.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The remaining Go builder stages, plus the kube-ovn, console and
piraeus-server builders, ran on the target platform. A multi-platform
build therefore ran the whole compiler under QEMU: slow, and for Go
unreliable, as the migration-controller Dockerfile already records a
"found bad pointer in Go heap" crash from exactly that. The builders
now run on $BUILDPLATFORM and produce the target architecture from
TARGETARCH.
Two builders only produced the right architecture because they were
emulated. token-proxy ran `go build` with no GOOS/GOARCH, and kube-ovn
calls upstream `make build-go`, which hardcodes GOARCH=amd64 even on an
arm64 build. On a native builder both would have shipped host binaries
inside the other architecture's image, and nothing fails until a node
of that architecture starts the container. token-proxy now passes the
target explicitly; kube-ovn picks build-go or its arm64 twin
build-go-arm from TARGETARCH.
The console builder emits static JS and linstor-server's debian/control
declares every package Architecture: all, so both run natively without
a target.
A bats test requires every stage that compiles to sit on a BUILDPLATFORM
FROM and to read TARGETARCH or TARGETPLATFORM on a RUN or ENV line. A
bare ARG declaration does not count, which is what let token-proxy
through.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Go 1.26.4 carries standard-library advisories in net/http, net/url,
crypto/tls, encoding/asn1 and html/template that govulncheck reports
as reachable from the first-party binaries. The shipped stdlib comes
from the builder image, so the builder tags move together with the
go directive; the digests are the multi-arch indexes that golang:1.27,
1.27-alpine and 1.27-bookworm resolve to at 1.27.1.
api/apps/v1alpha1 keeps go 1.26.4. It is a published types-only
module, no binary here links its stdlib, and raising its directive
would force every consumer onto Go 1.27.
linstor-csi v1.11.2 pins grpc v1.81.0, which no longer builds on
Go 1.27: golang.org/x/net drops http2.TrailerPrefix there. Its image
now raises grpc to v1.83.2 before building, as the kamaji image does
for its advisories; the upstream unit tests pass on it.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Cozystack's Talos configuration removes the systemd DRBD shutdown guard
without a replacement. Add `talos.gracefulShutdown.enabled`, disabled by
default, to release unused Secondary DRBD resources before the satellite
stops so their backing devices, including ZFS zvols, can be released
during normal shutdown. The hook skips Primary resources and open
devices, rechecks each candidate, and aborts further actions on
ambiguous status, command failure or timeout. It uses no force,
demotion, Kubernetes API calls or additional permissions.
The hook is available only when both Talos and DRBD are enabled. Keep
the value in
`spec.components.linstor.values.talos.gracefulShutdown.enabled` on the
`cozystack.linstor` package. The chart embeds the script in its existing
`cozystack-talos` satellite configuration, which Piraeus merges into the
generated DaemonSets. No custom satellite image is required. Default
renders remain unchanged for Talos, generic Linux and DRBD-less Talos.
This is opt-in because preStop also runs on ordinary satellite restarts.
Unused Secondary replicas disconnect until their replacement satellite
reattaches them, and Piraeus has one DaemonSet per node without a
cluster-wide rollout barrier. Simultaneous restarts can therefore pause
I/O. Enable this only with sequential satellite/node maintenance and
replication checks between nodes. The README covers the 60-second hook
budget, pod versus node shutdown grace periods, rollback behavior and
the unsupported forced-reboot/broken-kubelet paths. Related upstream
discussion:
https://github.com/piraeusdatastore/piraeus-operator/issues/860. The
reported outage from unconditional forced demotion in
https://github.com/piraeusdatastore/piraeus-operator/issues/860#issuecomment-5635881307
is why this hook never demotes a Primary, even when its devices appear
closed.
Validation: 22 Helm tests and 20 behavioral Python tests pass. The same
20 tests pass inside the existing Cozystack satellite image. Two of them
pin the rendered command form: `python3 -c <script> --execute` reaches
`main()` with `["--execute"]` because CPython sets `sys.argv[0]` to
`-c`, and a script-name placeholder before `--execute` is rejected.
Failure-path tests cover nonzero status/down exits, bounded stderr,
partial timeout diagnostics, and stopping before the next resource. Full
default renders match the base commit for all three supported substrate
variants. The rendered script survives strategic merging against Piraeus
2.10.2 and 2.11.0 satellite manifests; the base chart fails the new
feature's regression tests.
Runtime validation on Talos 1.13.6 used the script from commit
`b8c86f2e70d029166939ff29eab8a45ac191e34b` through per-node satellite
configurations: a normal node reboot with drain completed without force
or an external reset, and the node returned with its ZFS pool online and
DRBD replicas UpToDate. That version is installed on all three nodes of
the tested cluster after sequential satellite rollouts and replication
checks. The later diagnostic-only update in
`527570d8079cba7c91b3c345050cab1b73a290a8` is covered by the local and
exact-image tests above but has not been rolled out to that cluster.
This does not validate forced reboot, a failed kubelet, or simultaneous
satellite restarts. GitHub Actions for this fork require maintainer
approval; local checks do not replace that CI.
### Screenshots
Not applicable.
### Downstream repositories
The diff adds an opt-in system-package value and its
implementation/tests. It does not change app schemas, platform variants,
node prerequisites, shared tooling contracts or an existing documented
workaround.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(linstor): add an opt-in Talos satellite preStop hook that releases unused Secondary DRBD resources without force. Enable it with talos.gracefulShutdown.enabled for sequential node maintenance; existing installations keep their current behavior by default.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added optional graceful shutdown support for LINSTOR satellites
running on Talos.
* When enabled, unused Secondary DRBD resources are released before
termination, with safeguards for active resources and bounded shutdown
timing.
* Added a 90-second termination grace period and configuration
validation for the new setting.
* Added documentation covering configuration, operational requirements,
and shutdown behavior.
* **Tests**
* Added Helm and Python coverage for shutdown behavior, validation,
timeouts, and failure scenarios.
* The test target now runs both Helm and Python unit tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
drbd_stall_act disconnects one peer with "drbdadm disconnect res:peer"
but then runs "drbdadm connect res", which walks every connection of
the resource. The peers that are still connected make drbdsetup fail
with "Device has a net-config (use disconnect first)" (exit code 10),
the helper only logs "WARN: action failed" and the stalled peer stays
disconnected.
Reconnect the same res:peer pair that was just disconnected.
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
The first paragraph asserted what external-provisioner sends without the
flag; it is now read off the wire on a three-node dev stand. A
WaitForFirstConsumer CreateVolume carries requisite as the single segment
the scheduler chose with the flag set, and all three node segments
without it, the chosen one merely first in preferred. Same run confirmed
piraeus re-derives the csi-controller Deployment once when the flag lands
on an existing LinstorCluster, and that the satellite DaemonSets do not
move.
The `replicated` note claimed less than the facts support. That class
binds Immediate, and external-provisioner scopes strict topology to late
binding, so the flag never reaches that path -- which is a stronger
statement than the PV merely being remotely accessible, and does not
depend on reasoning about placement at all.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: LLM
A `#` comment in a rendered template is output, not source: the eight
lines explaining --strict-topology were applied to the live
LinstorCluster and came back on every `kubectl get`. Moved into a
template comment, which is stripped at render.
The test comment beside the argument-count assertion claimed the count
was "a tripwire that forces a re-check" on a piraeus-operator bump. It
cannot be. The assertion reads the chart's own rendered args, and those
do not move when upstream adds a flag -- the count stays 12 and the
suite stays green. It catches an edit to this chart, which is the case
that needs it least. Say what it does catch, and say where the upstream
half actually has to be checked: the tree vendors the piraeus chart but
not the operator resources carrying those defaults, so there is nothing
in-tree to diff against and the re-check on a bump stays a human
obligation. The list is recorded as verified against v2.10.2.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: LLM
drbd.enabled=false removes the drbd-logger sidecar, and that is all it
removes. The satellite DaemonSet also carries two DRBD-only
initContainers that piraeus-operator contributes unconditionally --
drbd-module-loader, which cannot load a module the substrate does not
have, and drbd-shutdown-guard -- and the only configuration that deletes
them is satellites-talos.yaml, gated on talos.enabled.
So with talos.enabled=false and drbd.enabled=false the chart rendered a
satellite that can never reach Ready: exactly the state the flag exists
to prevent, reached by setting the flag. Not hypothetical -- the
isp-full-generic bundle sets talos.enabled=false
(packages/core/platform/templates/bundles/system.yaml).
Fail at render time and name both flags. The alternative, moving the two
initContainer deletions and their volumes out from under talos.enabled,
is the real fix for generic Linux and is a larger change than this lane
needs; it is tracked in cozystack/cozystack#4092, and the message points
there. Refusing is what keeps an operator from getting no signal at all
in the meantime. values.yaml states the requirement beside the flag.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: LLM
Without --strict-topology, external-provisioner passes EVERY topology
segment as `requisite` and merely *prefers* the node the scheduler
picked, so when that node is out of space linstor-csi may legally
provision the volume somewhere else. For a node-pinned volume
(allowRemoteVolumeAccess: false, the `local` class that is the cluster
default) the result is a PV whose node affinity contradicts the pod's
other volumes, and the pod is then unschedulable forever against
pv "pvc-..." node affinity doesn't match node "srv1": no matching NodeSelectorTerms
with nothing erroring anywhere.
It bites hardest on CDI imports, because an importer pod mounts two
volumes: the disk, and a ~21 GiB scratch volume CDI creates only AFTER
the pod is scheduled. The scheduler therefore sizes the node against the
disk alone, can place two importers where both disks plus both scratch
volumes do not fit, and the second scratch lands elsewhere. Capacity
tracking cannot help -- the scratch volume does not exist when placement
is decided, and CSIStorageCapacity agreed with LINSTOR to within 5% when
this was measured.
Reproduced on 2026-08-25 by cordoning two of three nodes so both
importers landed on one constrained to 45 GiB: enough for two 20 GiB
disks, not for their scratch. With the flag the scratch PVC stays
Pending and the pod reports `1 node(s) did not have enough free
storage`, then binds and completes once space frees. A silent permanent
deadlock becomes a legible, self-correcting wait.
The full argument list is restated because `args` has no patchMergeKey,
so a strategic merge replaces rather than appends. That makes the list a
drift hazard on every piraeus-operator bump, and the note beside it says
so -- the count assertion in the test guards this chart's own list and
cannot see a flag the operator adds upstream.
Verified for node-pinned volumes. For `replicated` the PV is remotely
accessible so the constraint is satisfiable wherever the pod lands, and
that case is reasoned rather than measured -- hence the switch.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
On a substrate where the DRBD kernel module is unavailable the
drbd-logger sidecar exits immediately, the satellite pod is therefore
never Ready, and piraeus consequently registers zero nodes: LINSTOR
reports none, linstor-csi-node hangs in Init polling for a node that
does not exist, and every PVC stays Pending behind a cluster that looks
healthy. So this flag is the difference between working storage and no
storage at all there, not a cosmetic reduction in logging.
The concrete case is Talos nodes running as containers. DRBD's resource
registry is global per kernel, so three containers sharing one kernel
cannot represent three DRBD nodes -- the second new-resource for a given
name silently aliases the first instead of failing, which is worse than
an error because LINSTOR then records a replica it does not have. It
applies to kind equally; it is a property of one kernel, not of Talos.
Omitting the sidecar from Cozystack's own pod template is not enough,
because piraeus strategically merges satellite configurations and a
container another configuration contributes survives the omission. So a
LinstorSatelliteConfiguration deletes it from the effective DaemonSet as
well.
What the flag does NOT do is touch a StorageClass, and the comment says
so explicitly. Cozystack does not own them -- an operator writes them by
hand from the install guide -- so a class carrying layerList: "drbd
storage" still exists after setting this false and volumes on it will
never provision. The operator-facing half of that warning goes beside
the examples they copy, in cozystack/website#675.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The Talos userspace OOM handler (v1.12+) ranks cgroups with
memory_max.hasValue() ? 0.0 :
{Besteffort: 1.0, Burstable: 0.5, ...}[class] * memory_current
and discards every cgroup scoring zero, so only a memory limit takes a
pod out of the victim set. Selection is decoupled from the trigger: a
tenant workload thrashing against its own limit drives node memory PSI,
and a limit-free system DaemonSet is killed for it. metallb-speaker and
linstor-satellite were the observed victims.
Set requests and limits on the node DaemonSets that had none. Requests
come from 14d peak working set measured on a 3-node dev cluster; limits
are deliberately loose, sized as a runaway backstop rather than a fitted
ceiling, because several of these scale with cluster size that a dev
cluster understates and because undersizing converts a rare
pressure-driven kill into a deterministic one. A limit costs nothing
until it is hit -- only the request reserves.
memory.max lands on the POD cgroup only when EVERY container in the pod
carries a limit, so sidecars and init containers are covered too; one
limit-free container returns the whole pod to the victim set.
Two DaemonSets are not chart-rendered and needed their own seam:
linstor-satellite via the LinstorSatelliteConfiguration podTemplate, and
virt-handler via KubeVirt customizeComponents.patches, since virt-operator
owns that DaemonSet. Note linstor-controller stays deliberately limit-free
-- it is a JVM that must not be OOM-killed mid-operation, and cluster_test
pins that.
Two things the multus manifest needed beyond the values change.
customize-deployment.patch is the source for that template, so the
limits are carried there rather than only in the generated file, which
`make update` would discard. And the reason kube-multus had no memory
limit was that this patch DELETED upstream's 50Mi rather than raising
it -- 50Mi being well under the 230Mi measured peak -- which is what
removed memory.max from the pod cgroup in the first place.
Not covered, because upstream hardcodes the containers with no resources
knob: the four frr-k8s cp-* init containers, the kube-ovn hostpath-init
and install-cni init containers, and cozy-proxy, whose vendored chart
exposes no resources value at all. hami and kilo are left alone as
optional components with no usage measurements behind them. All of these
still get a limit from the default LimitRange the operator maintains in
every system namespace.
Every limit is pinned by a helm-unittest assertion, and every assertion
was mutation-tested: changing the rendered value turns its suite red.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Rebuilds the cozystack piraeus-server image off linstor-server v1.33.3 (latest
1.33 patch) and the linstor-csi image off v1.11.2, both multi-arch, and pins the
new digests in values.yaml. Picks up the published security fixes plus the
upstream 1.33.3 DRBD fixes (clone status check, shrink trigger). linstor-csi
v1.11.2 go.mod requires Go >= 1.26, which the csi builder image (golang:1.26)
already satisfies.
All five piraeus-server patches re-apply cleanly against v1.33.3 in the
Dockerfile glob order, and both linstor-csi patches re-apply against v1.11.2.
Verified on a dev cluster: a rolling restart of all satellites/controller/csi to
1.33.3 / v1.11.2 kept every node Online with zero faulty resources; a replicated
PVC provisioned, mkfs'd, and reached UpToDate across 3 nodes with no DRBD
"device held open" abort (the secondary-after-mkfs patch handled the path). The
csi v1.11.2 "no RWX without DRBD layer" change is a no-op for cozystack, which
always uses the DRBD layer.
Adaptations:
- Makefile: LINSTOR_VERSION 1.33.2 -> 1.33.3, LINSTOR_CSI_VERSION v1.10.6 -> v1.11.2; add test target
- images/linstor-csi/Dockerfile: ARG VERSION v1.10.6 -> v1.11.2
- values.yaml: pin both rebuilt multi-arch image digests
- tests/linstor_test.yaml: new helm-unittest image pins
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Under node CPU contention the host `linstor-controller` JVM cannot
answer its `GET /health` liveness probe within the piraeus-operator
default timing (`timeoutSeconds: 1`, `failureThreshold: 3`). The kubelet
then liveness-kills the container (exit 137, reason `Error`, not
`OOMKilled`). The resulting crash-loop leaves the `linstor-controller`
Service with no ready endpoints, `linstor-csi` gets `EPERM`, and
RWX-Filesystem `DataVolume`s never bind — breaking tenant Kubernetes /
NFS provisioning.
The operator-generated controller pod sets no `resources` (so no CPU
request and no scheduling/CPU guarantee) and ships the tight default
probe. This change, on the `linstor-controller` container in
`spec.controller.podTemplate`:
- adds `resources.requests` (cpu `250m`, memory `700Mi`) — requests
only, no limits: a CPU request guarantees CPU shares so the JVM keeps
answering `/health` under contention, while omitting limits avoids
throttling the JVM or OOM-killing it. The memory value is a conservative
starting estimate; tune it to observed RSS.
- relaxes the liveness timing to `timeoutSeconds: 5` and
`failureThreshold: 6` so a JVM GC pause or CPU contention does not trip
a liveness kill.
piraeus-operator merges `spec.controller.podTemplate` into the generated
Deployment as a strategic merge patch, so the liveness override carries
only the two timing fields; the operator's `httpGet` target (`GET
/health` on the api port) is left intact to avoid pinning a wrong path
or scheme. Both knobs are exposed under `controller.*` in `values.yaml`
for tuning.
This is a latent chart weakness amplified by load, not a regression: on
the same branch, image, and chart, a green run had the controller stable
at `restartCount=0`, while a CPU-loaded run on identical code
crash-looped. Same code, different outcome under load.
### Screenshots
N/A — no UI changes.
### Release note
```release-note
fix(linstor): give the linstor-controller CPU/memory requests and a more tolerant liveness probe so it no longer crash-loops under node CPU contention, which previously left the controller Service with no endpoints and blocked RWX volume provisioning
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added default controller resource requests for the Linstor controller.
* Added configurable liveness probe timing settings for the controller
container.
* **Tests**
* Added coverage to verify the controller pod template applies the
expected resource requests and liveness probe timing values.
* Confirmed the controller image remains correctly pinned while the new
settings are applied.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Under node CPU contention the linstor-controller JVM cannot answer its GET /health liveness probe within the piraeus-operator defaults (timeoutSeconds 1, failureThreshold 3). The kubelet then liveness-kills the container; the resulting crash-loop leaves the linstor-controller Service with no ready endpoints, linstor-csi gets EPERM, and RWX-Filesystem DataVolumes never bind, breaking tenant provisioning.
The operator-generated controller pod sets no resources (no CPU request, hence no scheduling/CPU guarantee) and ships the tight default probe. Add CPU/memory requests and relax the liveness timing via the controller podTemplate, which piraeus-operator merges into the Deployment as a strategic merge patch. The probe override carries only the timing fields; the httpGet target (GET /health on the api port) is left to the operator to avoid pinning a wrong path or scheme. Requests-only (no limits) keeps the JVM from being throttled or OOM-killed. The knobs are exposed under controller.* for tuning to observed RSS.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Several first-party images still built their Go binaries with EOL or
behind-latest toolchains (golang 1.20, 1.20.6, 1.23.4, 1.24, 1.25), so a
plain rebuild kept shipping standard-library versions that the current
1.26 toolchain fixes. Raise every Go builder image (FROM and the
builder_image ARG defaults) to golang:1.26 across cluster-autoscaler,
kubevirt-cloud-provider, redis-operator, multus-cni, keycloak-operator,
objectstorage, kilo and linstor-csi.
This is the build-time counterpart to raising the go directive in the
first-party Go modules: the toolchain that compiles the binary, not the
go.mod minimum, determines the standard-library version baked into the
image.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
PR #2711 switched PR builds to push only :pr-<N>-<sha> tags (PUBLISH_FLOATING=0) to stop concurrent-push 409 conflicts, but left --cache-from pointing at $(REGISTRY)/<img>:latest. PR builds set REGISTRY to the per-CI registry (iad.ocir.io) and never publish :latest there, so every PR build's registry cache lookup 404s and rebuilds cold.
Add a CACHE_REGISTRY knob (default ghcr.io/cozystack/cozystack, where release builds do publish :latest) and point every --cache-from at it. Push targets ($(REGISTRY)) and the unique per-PR tags are unchanged, so the #2711 anti-conflict behaviour is preserved while PR builds regain a warm cache from the last release.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Block accidental DELETE on critical platform objects (tenant-root,
cozy-system, LinstorCluster, CRDs, ClusterIssuers, cozystack-version
ConfigMap, cozystack-packages OCIRepository, tenant-root HelmRelease)
via a label-scoped ValidatingAdmissionPolicy. No webhook DaemonSet,
Service, TLS, or image — the API server evaluates the CEL expression
in-process. Requires Kubernetes 1.30+.
Bypass for legitimate deletes is two steps:
kubectl label <kind> <name> platform.cozystack.io/no-delete-
kubectl delete <kind> <name>
CRDs installed dynamically by the operator get the label stamped in
internal/crdinstall/install.go before the apply call.
Release note:
Added deletion-protection guardrail (ValidatingAdmissionPolicy) blocking
DELETE on cozystack platform objects labeled platform.cozystack.io/no-delete=true.
Bypass by removing the label first.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Every package Makefile tagged images via `settag` which, on a non-release
commit, collapsed to `:latest`. Concurrent PR builds therefore raced on every
image's `:latest` manifest in the OCIR registry. Two outliers (metallb,
redis-operator) used hardcoded version tags and collided across PRs even when
`settag` would not have.
Replace `settag` and the `TAG=latest` fallback with an `image-tags` macro
driven by three env vars:
- IMAGE_TAG — build-unique tag, always pushed
- PUBLISH_VERSIONED — also push :<component-version>
- PUBLISH_FLOATING — also push :latest
PR CI sets IMAGE_TAG=pr-<N>-<sha> with both PUBLISH_* off, so every push is
unique across runs. Release CI sets IMAGE_TAG=<git-tag> with both PUBLISH_*
on, preserving the previous release output and adding a uniform :<git-tag>
tag across every image. Local builds default to IMAGE_TAG=dev with PUSH=0
already off.
Image consumers reference digests pinned into values.yaml by each Makefile,
so the tag is decorative for cluster-side pulls; the build-unique tag is
sufficient for E2E.
Drops the redundant `:<component-version>-<git-tag>` form (cilium, linstor,
matchbox, ubuntu-container-disk, kubevirt-*, cluster-autoscaler) — nothing
in the repo or downstream values.yaml consumed it. The same information is
preserved via the new uniform `:<git-tag>` plus the existing
`:<component-version>` on release.
ubuntu-container-disk builds a separate image per Kubernetes version in a
loop. Switched its primary tag to `:<k8s-ver>-<IMAGE_TAG>` so loop
iterations within a single build do not push to the same tag.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Adds a fifth piraeus-server patch that retries `drbdadm secondary` after
`mkfs` when DRBD reports the device as held open by an external process.
### Problem
On Talos (and any system with active block-device probing — udev,
multipathd, etc.) the kernel uevent for a freshly-promoted DRBD device
causes an external probe to `open()` the device for superblock
inspection. If LINSTOR satellite issues `drbdadm secondary` while the
probe still holds the fd, DRBD returns `(-12) Device is held open by
someone` and aborts initialization with `StorageException: Failed to
become secondary again after creating filesystem`.
The satellite never reports the final `UpToDate` event to the
controller, so the controller view stays in `SyncTarget(N%) /
Inconsistent / DELETING` permanently — even though the DRBD kernel state
on the satellite is `Established/UpToDate`. This blocks cleanup of
orphan PVs in `Released` phase and accumulates with every PVC creation.
Observed in production on Talos 1.10 / LINSTOR 1.33.2: 4 occurrences in
24 hours across 3 nodes for different PVCs. Manual recovery requires
`drbdadm down/up` on the Primary to force the satellite to re-report
state.
### Fix
New `secondaryAfterMkfs()` wrapper in `DrbdAdm.java`:
- 7 attempts of `drbdadm secondary` with exponential backoff (250 ms → 2
s, capped)
- Retry only triggers when stderr contains `Device is held open by
someone`; any other failure fails fast as before
- Existing `secondary()` contract is unchanged — only
`DrbdPrimary.close()` (the post-mkfs demotion path) is rerouted to the
new wrapper
The retry window covers ~10 s in the worst case, which comfortably
exceeds typical probe durations (probes release the fd within
milliseconds).
### Related upstream
-
[LINBIT/linstor-server#268](https://github.com/LINBIT/linstor-server/issues/268)
— open since 2022, identical symptom, no upstream fix
- [LINBIT/drbd#74](https://github.com/LINBIT/drbd/issues/74) — same
EBUSY pattern with multipathd holding the fd
### Validation
- `git apply *.diff` on a fresh `v1.33.2` checkout applies all five
patches cleanly in alphabetical order
- Java brace/paren/bracket balance verified on the patched file
- `helm template packages/system/linstor` renders without errors
### Release note
```release-note
fix(linstor): retry `drbdadm secondary` after mkfs on transient `Device is held open by someone` errors. Prevents orphan PVs in `Released` phase from accumulating when an external block-device probe (Talos block-controller, udev, multipathd) wins a race against the post-mkfs demotion. Fail-fast behavior preserved for any other secondary failure.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Implemented automatic retry logic for device secondary state
transitions after filesystem creation. System now gracefully handles
scenarios where devices are temporarily held open by other processes.
* **Documentation**
* Added documentation for the new device state transition patch.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
DRBD auto-promotes a resource to Primary during the provisioning path
(e.g. mkfs.ext4 against a freshly created replica), but auto-demote
sometimes fails to fire on close. The resource then stays Primary on
a node that has no consumer, and CSI on the legitimate target node
fails NodePublishVolume with "failed to set source device readwrite".
The only recovery is an explicit drbdadm secondary on the stuck node.
Add a per-satellite plunger handler that detects local Primaries
where every device has open == false and demotes them. Both the
detection and the action are local to the satellite, so no peer
coordination is needed. open:no makes drbdadm secondary safe; if a
real consumer is about to arrive, auto-promote will redo it.
Upstream tracker: piraeusdatastore/piraeus-operator#942
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
LINSTOR satellite aborts resource initialization with "Device is held open
by someone" (errno -12) when an external probe (Talos block-controller,
udev, multipathd) opens the freshly-promoted DRBD device just before the
post-mkfs demotion. The controller view then stays in SyncTarget /
Inconsistent / DELETING indefinitely, blocking cleanup of orphan PVs in
Released phase.
Add a retry wrapper secondaryAfterMkfs() with exponential backoff
(7 attempts, 250ms -> 2s) that triggers only on this specific error
message. DrbdPrimary.close() now uses the wrapper so the retry only
applies to the post-mkfs demotion path; the existing public secondary()
contract is untouched.
Related: LINBIT/linstor-server#268, LINBIT/drbd#74
Signed-off-by: IvanHunters <ivan.okhotnikov@aenix.io>
Bump linstor-csi to v1.10.6 and add an out-of-tree patch that overrides
DrbdOptions/Net/protocol to C on the resource-definition whenever a
volume is attached to a second node with allow-two-primaries (the
live-migration flow), and reverts it on detach.
Without the patch, any KubeVirt live migration of a VM whose volume
sits on a Protocol-A/B resource (for example a "replicated-async"
StorageClass) ends up in a permanent evacuation loop: drbdadm adjust
on the satellite rejects allow-two-primaries with "Protocol C
required" (errno 139), the migration fails, and KubeVirt retries
until manual intervention.
The override is set at the resource-definition level so it covers
every connection, including diskless TieBreaker peers, and is tagged
with an Aux marker so Detach reverts only the override we installed.
The 001 relocate-after-clone patch was regenerated against v1.10.6
(context shift only, no logic change).
Verified end-to-end on dev5: live migration of a KubeVirt VM whose
PVC sits on a Protocol-A resource-group now succeeds in a single
Attach.
Patch is upstreamed as draft PR piraeusdatastore/linstor-csi#435.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Make the 5-minute timeout self-documenting by setting periodSeconds: 10
explicitly rather than relying on the Kubernetes default.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
The default Kubernetes startup probe allows only 30 seconds (3 retries
× 10s) for LINSTOR satellites to become ready. This is insufficient on
nodes with slow storage initialization, causing unnecessary pod restarts.
Raise failureThreshold to 30, giving satellites up to 300 seconds (5
minutes) to complete startup.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
All existing patches (allow-toggle-disk-retry, fix-duplicate-tcp-ports,
fix-luks-header-size, retry-adjust-after-stale-bitmap) apply cleanly on
the new version without modifications.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
The crct10dif kernel crypto module is no longer available in Talos
v1.12.6 (kernel 6.18.18). DRBD auto-verify selects crct10dif by
default, causing peer connections to fail with VERIFYAlgNotAvail.
Setting verify-alg explicitly to crc32c at the controller level
ensures all resources use an available algorithm.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>