Re: [BUG] ublk: ublk server hangs in D state during STOP_DEV

[Ming Lei <ming.lei@redhat.com>]

On Sun, Jan 18, 2026 at 01:03:19AM +0800, huang-jl wrote:
> > > - Upon receiving SIGINT, our ublk server will sends UBLK_U_CMD_STOP_DEV to the
> > > driver.
> > 
> > Can you share how your server sends STOP_DEV when receiving SIGINT?
> > 
> > If it prevents normal IO command handling, ublk_stop_dev() will cause deadlock.
> > 
> > For example, follows the preferred IO handling in ublk server:
> > 
> > prepare UBLK_IO_FETCH_REQ uring_cmds;
> > while (1) {
> > io_uring_enter(submission & wait event);
> > }
> > 
> > If you send STOP_DEV command inside the above loop, you will get the
> > deadlock, because inflight and new IOs can't be handled any more.
> 
> My ublk server has two threads:
> 
> - The main thread: opens /dev/ublk-control and create an io uring. It handles
>   all control uring cmd (e.g., ADD_DEV, START_DEV, STOP_DEV).
> 
> - A worker thread: also creates an io uring. It will send UBLK_U_IO_FETCH_REQ
>   and UBLK_U_IO_COMMIT_AND_FETCH_REQ uring cmd, and handle IO request.
> 
> Main thread will first ADD and START device, then listens for SIGINT signal.
> Upon receiving SIGINT, main thread issues a STOP_DEV cmd.
> The worker thread continues running independently and is not directly affected
> by this signal.
> 
> (Implementation detail: I am using Rust and the Tokio async runtime. The SIGINT
> is handled within the userspace rather than directly inside the signal handler.)
> 
> > If that is the case, you may remove the sending of STOP_DEV simply in your
> > implementation, and let ublk server exit. Then delete the ublk device before
> > ublk server is exiting to kernel, or let your monitor process remove the
> > ublk device, which becomes DEAD state after ublk server exits.
> > 
> > Otherwise, you may have to investigate why ublk server can't handle IO
> > command after sending STOP_DEV command.
> 
> Based on my previous post [1], **the ublk server gets SIGKILL after sending
> STOP_DEV command**, so it cannot handle IO request.
> 
> In detail:
> 
> 1. Flush Kernel Thread: A kworker/flush thread attempts to write dirty pages
> to the ublk device. It acquires a folio lock but then enters a sleep state
> in wbt_wait() due to Writeback Throttling (WBT).
> 
> 2. STOP_DEV Command: The ublk server issues a STOP_DEV command. This uring cmd
> is processed by an uring kernel thread.
> 
> 3. Resource Conflict: When handle STOP_DEV, the uring kernel thread also tries
> to flush dirty pages. It needs to acquire the same folio lock held by the flush
> kworker to submit the I/O.
> 
> 4. Termination Signal: At this point, the ublk server receives a SIGKILL.

It is _not_ related with WBT or write cache.

When your monitor sends SIGKILL, do_exit() is actually called, but
io_uring_cancel_generic() blocks on io_wq_put_and_exit() because STOP_DEV
can't be completed, then do_exit() can't move on, and finally the ublk cancel
function can't be called, then ublk char device can't be closed.

Looks something which might be improved in future, but it highly depends on
(complicated) io_uring's cancel code path, and I believe there isn't
any way for ublk to get notified when do_exit() happens from `kill -9`.

> 
> Conclusion:
> This creates a circular dependency: the flush kworker cannot be woken from WBT
> because the ublk server is no longer processing I/O requests.
> Simultaneously, the ublk server cannot exit because it is waiting for the
> STOP_DEV command to complete. However, the uring kernel thread remains stuck
> waiting for the folio lock held by the flush kworker.
> 
> Should we avoid having the ublk server issue STOP_DEV command by it self? As it
> is highly prone to deadlocks if a SIGKILL occurs during the shutdown sequence.

So far there are at least two ways:

1) don't provide signal handler for SIGINT, or simply call exit() in the
signal handler, and ublk block device is removed automatically when ublk
server is done in case of !USER_RECOVERY.

OR

2) provide signal SIGINT handler to stop device, meantime not send SIGKILL
in your monitor process. This way works just fine if your ublk server is
well implemented, which often requires you to handle timeout in the server
implementation.


Thanks,
Ming