Re: ublk: consequences of a dead server are only failing on the last /dev/ublkcN release

[Ming Lei <tom.leiming@gmail.com>]

Hello,

On Fri, Sep 18, 2026 at 12:29:22PM +0200, Christian Brauner wrote:
> Hi,
> 
> Chris scanned a new series of mine for me and he reported some
> interesting behavior that I dug into.
> 
> A ublk request that was handed to the server is only failed once the
> last reference to /dev/ublkcN is dropped. So holding the file open keeps
> the request alive.
> 
> This causes any waiter to be stuck in D state and blocks STOP_DEV.
> 
> I'm attaching two reproducers:
> 
> (1) ublk_inherited_fd_test.c   the general case, no vfork involved
> (2) vfork_ublk_test.c          the holder of the reference is the waiter itself
> 
> Re (1):
> 
> A single-threaded ublk server fork()s a helper. The helper inherits the
> fdtable including /dev/ublkcN. Now another process opens /dev/ublkbN and
> writes 4 kb then fsync()s. The servers gets a WRITE and _exit()s without
> completing it - say it crashes:
> 
>   - The writer sleeps in folio_wait_writeback() in D state.
>   - STOP_DEV issued from another process never returns.
> 
>     The control command runs on an io-wq worker of that process, which
>     calls ublk_stop_dev() -> del_gendisk() -> bdev_mark_dead() ->
>     filemap_write_and_wait_range() and ends up waiting on the page that
>     won't be written back.
> 
>     Said process can't exit and io_wq_put_and_exit() waits for the
>     worker.
> 
> This can only be undone if the helper is SIGKILLed:
> 
>   # ublk0 up, server 119
>   # helper 123 forked by the server, holds the inherited fds
>   # WRITER fsync
>   # server 119 died with the WRITE in hand
>   # 2s after the server died:
>   # writer pid 124, state D:
>   #   folio_wait_bit_common
>   #   folio_wait_writeback
>   #   __filemap_fdatawait_range
>   #   file_write_and_wait_range
>   #   blkdev_fsync
>   #   do_fsync
>   # STOP_DEV from pid 125 has not returned after 5s:
>   # STOP_DEV issuer thread 126:
>   #   folio_wait_bit_common
>   #   folio_wait_writeback
>   #   __filemap_fdatawait_range
>   #   filemap_write_and_wait_range
>   #   bdev_mark_dead
>   #   blk_report_disk_dead
>   #   __del_gendisk
>   #   del_gendisk
>   #   ublk_stop_dev_unlocked.part.0
>   #   ublk_ctrl_uring_cmd
>   #   io_uring_cmd
>   #   io_wq_submit_work
>   #   io_worker_handle_work
>   #   io_wq_worker
>   # killing helper 123
>   # WRITER fsync returned -1 errno 5
>   # STOP_DEV returned after the helper died
> 
> What happens to a device after a server crash depends on whoever else
> holds /dev/ublkcN open: A forked helper, a process that got the fd via
> SCM_RIGHTS, or a vfork child.
> 
> I'm not sure whether that's intended semantics but it surely has
> confusion potential and should probably be at least documented.

It is attributed to ublk driver implementation and kernel:

1) ublk driver uses two-stage tear-down: 

- uring commands are canceled when exiting io_uring

- block requests are aborted when ublk server is down, which can only be signaled
by closing /dev/ublkcN; this stage depends on canceling uring commands.

Not get other idea for deciding if ublk server can be thought as being down.

But yes, it should be documented.

> 
> I suppose failing outstanding requests on server crash is intentionally
> not done for recover reasons or whatever. But maybe STOP_DEV should be
> made to work.

STOP_DEV needs to freeze request queue, which has to drain inflight
request.

> 
> Re (2):
> 
> vfork() makes it really ugly. Say a single-threaded ublk server vfork()s
> a child. The child opens /dev/ublkbN, writes some stuff and closes the
> fd. Since close(2) runs bdev_release() -> sync_blockdev() it waits for
> the write.

ublk driver shouldn't access to /dev/ublkbN, #2 is similar with that, so it
shouldn't be supported.

> 
> The write gets dispatched to the server's task work. But the server
> sleeps in wait_for_vfork_done() until the child execs or exits.
> 
> But the child cannot exec or exit because close(2) doesn't return.
> 
> ublk makes this more hairy though. Even a SIGKILL sent to the server aka
> the parent won't help. The vfork()ed child hangs on the close(2).
> 
> So that stays in D state forever with an undeletable device:
> 
>     server, after SIGKILL: gone
>     child:
>       State:  D (disk sleep)
>       folio_wait_bit_common
>       folio_wait_writeback
>       __filemap_fdatawait_range
>       filemap_write_and_wait_range
>       bdev_release
>       blkdev_release
>       __fput
>       fput_close_sync
>       __x64_sys_close
> 
> Unrelated, but also a potential issue:
> 
> The commit 7fc4da6a304b ("ublk: scan partition in async way") made
> partition scans run from a workqueue that holds disk->open_mutex while
> the server serves its reads and START_DEV returns before it's finished.
> 
> Any server that treats START_DEV's return as a signal that the disk is
> ready and doesn't serve requests hangs every open() of /dev/ublkbN on
> uninterruptible on open_mutex. You can combine that with (1) and (2).

START_DEV doesn't promise partitions scanning done when it is returned.

nvme multipath has similar usage too.


Thanks, 
Ming