Aleksei Sviridkin 52aa837821 fix(kubernetes): force the kubevirt-csi temp NFS unmount after a timeout (#4584)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The kubevirt-csi node plugin now unmounts its temporary NFS mount with a
timeout, and retries with `umount -f` when the timeout runs out.

For an NFS publish the plugin mounts the export root into an
`nfs-init-*` temp dir to create `/data`. It unmounted that dir in a
defer with a plain `umount`, which has no deadline. Before this change,
if the NFS server was gone by then, the publish could hang on that
`umount` forever and the temp mount stayed.

The force-unmount helper in `k8s.io/mount-utils` doesn't work for this
case. `UnmountWithForce` retries with `-f` only after its first `umount`
returns. That `umount` forks the `umount.nfs` helper. When the timeout
kills `umount`, the helper stays stuck on the dead server and keeps the
output pipe open, and `CombinedOutput` waits for that pipe, so the `-f`
retry never runs. The plugin now runs `umount` itself with `WaitDelay`
set, so the pipe is closed a second after the kill, and then runs
`umount -f` under its own timeout. A "not mounted" answer counts as
done, same as in mount-utils. `umount2` detaches the mount before it
blocks on tearing down the superblock, so the `-f` may find nothing
left.

Some umounts can't be killed at all, for example one stuck in
uninterruptible sleep. So the whole cleanup also waits at most 30s, well
inside the 2m deadline kubelet sets for NodePublishVolume (`csiTimeout`
in `pkg/volume/csi/csi_plugin.go`). Once the budget runs out the publish
goes on, and the unmount keeps running in the background and logs its
result when it ends. The temp dir is removed by that background unmount
once it succeeds, so an unmount that finishes late still cleans up after
itself. A mounter other than the exec-based one keeps its plain
`Unmount`, under the same budget.

The temp mount now uses `nosharecache`. A forced unmount runs
`nfs_umount_begin`, which kills every RPC in flight on the superblock,
and NFS shares a superblock between mounts of the same export with the
same options. Without this option, `-f` on the temp mount could fail I/O
on the pod mount this same publish creates, and on other pods on the
node that mount the same volume. The cost is a separate cache: pods that
already have `/data` mounted may not see what the migration moves in
until their attribute cache expires. The migration is a one-time
backfill, so I'm fine with that.

The leak only lives until the container restarts, because `/tmp` in the
node plugin container has no hostPath and the mounts go away with the
container's mount namespace. So this change stops leaked mounts from
piling up inside one container. For the same reason a sweep of
`nfs-init-*` dirs at startup would never find anything, so I did not add
one.

Two cases stay as they were. An `umount` that fails fast, for example
with `target is busy`, gets no force retry, is logged, and the dir stays
because `rmdir` refuses a mount point. And since the temp mount is a
hard mount, a server that disappears before the unmount still hangs the
mount and migration calls earlier in this path. That second case is
#4581.

The tests run the real exec-based mounter against a fake `umount` on
`PATH`. The fake covers a plain unmount that succeeds, one that fails
fast and gets no `-f`, and one that hangs while an orphaned child holds
its output pipe. The last one has to end in `umount -f` inside the
budget. A "not mounted" from `-f` counts as done, and any other `-f`
failure is reported and keeps the dir. Fake mounters cover the
background path: giving up on a stalled unmount, removing the dir after
a late unmount, and logging the late result. Other tests check
`nosharecache` on the temp mount and the temp dir after a failed mount.
I haven't reproduced this against a real NFS server that went away, so
it is not checked whether `umount -f` in the almalinux image gets as far
as `umount2(MNT_FORCE)` there. The tests are in a new file, so this PR
does not overlap with #4562.

Closes #4561

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

No UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(kubernetes): the kubevirt-csi node plugin no longer hangs an NFS volume publish forever when unmounting its temporary mount after the NFS server went away; the unmount now times out and is retried with force
```




<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Temporary NFS mounts use isolated caching to reduce interference with
pod mounts.
* Cleanup waits up to 30 seconds for unmounting. If the limit is
reached, the publish operation returns an error while unmounting
continues in the background.
* Temporary directories are removed after a successful unmount or a
failed mount, and retained if unmounting fails.
* When a standard unmount times out, cleanup retries with a forced
unmount when available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-07 06:03:37 +03:00
2025-10-08 09:43:34 +05:00
2024-02-08 12:04:32 +01:00

Cozystack Cozystack

GitHub Release Apache-2.0 License Commit activity OpenSSF Best Practices OpenSSF Scorecard CLOMonitor Support

Cozystack

Cozystack is a free and open-source platform and framework for building clouds.

Cozystack is a CNCF Sandbox project that was originally built and sponsored by Ænix.

With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.

Use Cozystack to build your own cloud or provide a cost-effective development environment.

Cozystack user interface

Use-Cases

Documentation

The documentation is located on the cozystack.io website.

Read the Getting Started section for a quick start.

If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.

Versioning

Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.

Contributions

Contributions are highly appreciated and very welcomed!

In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.

You are welcome to take on the fix yourself. Commits are used to generate the changelog, and their author will be referenced in it.

If you have a concrete feature request, open an issue. For a cross-cutting or architectural change, open a design proposal in the community repo. For questions or early ideas, use GitHub Discussions. See CONTRIBUTING.md for the full routing.

Community

You are welcome to join us in #cozystack, on either CNCF Slack (get an invite) or Kubernetes Slack (get an invite), in our Telegram group, and at our community meetings. Add them to your Google Calendar or iCal for convenience.

Project documents

Document What it covers
Governance How decisions are made and who makes them
Contributor Ladder How to grow from contributor to maintainer
Maintainers Who maintains the project
Code of Conduct What we expect of each other
Security Policy Reporting a vulnerability, and which releases get fixes
Adopters Who runs Cozystack in production
Roadmap Where the project is going

License

Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.

Commercial Support

A list of companies providing commercial support for this project can be found on official site.

Languages
Go 52%
Shell 34%
TypeScript 6.8%
Go Template 4.7%
Makefile 1.3%
Other 1.1%