Re: Races in sbitmap batched wakeups

[Ming Lei <ming.lei@redhat.com>]

On Thu, Jun 16, 2022 at 07:21:02PM +0200, Jan Kara wrote:
> Hello!
> 
> I've been debugging some customer reports of tasks hanging (forever)
> waiting for free tags when in fact all tags are free. After looking into it
> for some time I think I know what it happening. First, keep in mind that
> it concerns a device which uses shared tags. There are 127 tags available
> and the number of active queues using these tags is easily 40 or more. So
> number of tags available for each device is rather small. Now I'm not sure
> how batched wakeups can ever work in such situations, but maybe I'm missing
> something.
> 
> So take for example a situation where two tags are available for a device,
> they are both currently used. Now a process comes into blk_mq_get_tag() and
> wants to allocate tag and goes to sleep. Now how can it ever be woken up if
> wake_batch is 4? If the two IOs complete, sbitmap will get two wakeups but
> that's not enough to trigger the batched wakeup to really wakeup the
> waiter...

commit 180dccb0dba4 ("blk-mq: fix tag_get wait task can't be awakened")
is supposed for addressing this kind of issue.

> 
> Even if we have say 4 tags available so in theory there should be enough
> wakeups to fill the batch, there can be the following problem. So 4 tags
> are in use, two processes come to blk_mq_get_tag() and sleep, one on wait
> queue 0, one on wait queue 1. Now four IOs complete so
> sbitmap_queue_wake_up() gets called 4 times and the fourth call decrements
> wait_cnt to 0 so it ends up calling wake_up_nr(wq0, 4). Fine, one of the
> waiters is woken up but the other one is still sleeping in wq1 and there
> are not enough wakeups to fill the batch and wake it up? This is
> essentially because we have lost three wakeups on wq0 because it didn't
> have enough waiters to wake...

But the following completions will wake up the waiter in wq1, given
there are more in-flight.

Thanks,
Ming