ublk: slow recovery process when io queue depth is low

ublk: slow recovery process when io queue depth is low

[Yoav Cohen]

Hi,

I'm running ublk on Ubuntu Kenrel 6.8.0-56-generic.
I notice that if I'm running an IO commands that causing the IO queues to be full (at least the ublk hw queue) it seems like some of the IO's are done 30 seconds~ after they where submitted.
Enlarging the IO queues fixed the issue, and I'm pretty sure the 30 seconds magic number may be result of blk_mq_tag_set timeout filed default(see blk_mq_init_allocated_queue where it set to 30 * HZ)


Can you guys explain the behaviour? 


Thank you

Re: ublk: slow recovery process when io queue depth is low

[Jens Axboe]

On 4/17/25 4:06 AM, Yoav Cohen wrote:
> Hi,
> 
> I'm running ublk on Ubuntu Kenrel 6.8.0-56-generic. I notice that if
> I'm running an IO commands that causing the IO queues to be full (at
> least the ublk hw queue) it seems like some of the IO's are done 30
> seconds~ after they where submitted. Enlarging the IO queues fixed the
> issue, and I'm pretty sure the 30 seconds magic number may be result
> of blk_mq_tag_set timeout filed default(see
> blk_mq_init_allocated_queue where it set to 30 * HZ)
> 
> 
> Can you guys explain the behaviour? 

Please try a more recent kernel - 6.8 is both pretty old in terms of
ublk, and more importantly, it's also not a supported stable release.
That means that even if we could fix any issues in 6.8, since there's no
further release of that, it's a dead end.

The only one you can talk to about distro kernel issues is the distro.
And I'm guessing you're using Ubuntu - they do have closer-to-mainline
kernels available, that'd be a much better base to use.

-- 
Jens Axboe

Re: ublk: slow recovery process when io queue depth is low

[Uday Shankar]

On Thu, Apr 17, 2025 at 10:06:36AM +0000, Yoav Cohen wrote:
> Hi,
> 
> I'm running ublk on Ubuntu Kenrel 6.8.0-56-generic.
> I notice that if I'm running an IO commands that causing the IO queues to be full (at least the ublk hw queue) it seems like some of the IO's are done 30 seconds~ after they where submitted.
> Enlarging the IO queues fixed the issue, and I'm pretty sure the 30 seconds magic number may be result of blk_mq_tag_set timeout filed default(see blk_mq_init_allocated_queue where it set to 30 * HZ)

Correct, that's where the 30s comes from.

> Can you guys explain the behaviour? 

The behavior you describe was actually just fixed yesterday by [1]; see
that commit message for a more detailed explanation. You might also be
interested in [2], which is a test that recreates exactly the behavior
you describe. Before the fix in [1], the test saw an I/O take ~30s to
complete, but after the fix, the I/O completes quickly.

[1] https://lore.kernel.org/linux-block/20250416035444.99569-6-ming.lei@redhat.com/
[2] https://lore.kernel.org/linux-block/20250416035444.99569-9-ming.lei@redhat.com/