Re: blk-mq request allocation stalls

[Jens Axboe <axboe@kernel.dk>]

On 01/12/2015 10:53 AM, Keith Busch wrote:
> On Mon, 12 Jan 2015, Jens Axboe wrote:
>> On 01/12/2015 10:04 AM, Bart Van Assche wrote:
>>> The tag state after having stopped multipathd (systemctl stop
>>> multipathd) is as follows:
>>> # dmsetup table /dev/dm-0
>>> 0 256000 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1
>>> service-time 0 2 2 8:48 1 1 8:32 1 1
>>> # ls -l /dev/sd[cd]
>>> brw-rw---- 1 root disk 8, 32 Jan 12 17:47 /dev/sdc
>>> brw-rw---- 1 root disk 8, 48 Jan 12 17:47 /dev/sdd
>>> # for d in sdc sdd dm-0; do echo ==== $d; (cd /sys/block/$d/mq &&
>>>    find|cut -c3-|grep active|xargs grep -aH ''); done
>>> ==== sdc
>>> 0/active:10
>>> 1/active:14
>>> 2/active:7
>>> 3/active:13
>>> 4/active:6
>>> 5/active:10
>>> ==== sdd
>>> 0/active:17
>>> 1/active:8
>>> 2/active:9
>>> 3/active:13
>>> 4/active:5
>>> 5/active:10
>>> ==== dm-0
>>> -bash: cd: /sys/block/dm-0/mq: No such file or directory
>>
>> OK, so it's definitely leaking, but only partially - the requests are
>> freed, yet the active count isn't decremented. I wonder if we're
>> losing that flag along the way. It's numbered high enough that a cast
>> to int will drop it, perhaps the cmd_flags is being copied/passed
>> around as an int and not the appropriate u64? We've had bugs like that
>> before.
>
> Is the nr_active count correct prior to starting the mkfs test? Trying
> to see if someone is calling "blk_mq_alloc_tag_set()" twice on the same
> set. It might be good to add a WARN if this is detected anyway.

That might be a good debug aid, I agree. But the above doesn't look like 
it's corrupted. If you add the values, you get 60 and 62 for the two 
cases, which seems to indicate that we did bump the values correctly, 
but for some reason we never did the decrement on completion. Hence we 
stabilize around the queue depth of the device, which will be 62 +/- a 
bit due to the sharing.

I'm not familiar with how rq based dm works. We clone the original 
request (which has the RQ_MQ_INFLIGHT flag set), then we issue the 
clone(s) to the underlying device(s)? And when that completes, we 
complete the original? That would work fine with the flag on the 
original request. Maybe I'm missing something, and I'll let more 
knowledgeable people discuss that.

-- 
Jens Axboe