Re: for-next hangs on test srp/012

[Bart Van Assche <bvanassche@acm.org>]

On Tue, 2018-12-11 at 17:02 -0700, Jens Axboe wrote:
> On 12/11/18 3:58 PM, Bart Van Assche wrote:
> > Hi Jens,
> > 
> > If I run the following subset of blktests:
> > 
> >   while :; do ./check -q srp && ./check -q nvmeof-mp; done
> > 
> > against today's for-next branch (commit dd2bf2df85a7) then after some
> > time the following hang is reported:
> > 
> > INFO: task fio:14869 blocked for more than 120 seconds.
> >       Not tainted 4.20.0-rc6-dbg+ #1
> > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > fio             D25272 14869  14195 0x00000000
> > Call Trace:
> >  __schedule+0x401/0xe50
> >  schedule+0x4e/0xd0
> >  io_schedule+0x21/0x50
> >  blk_mq_get_tag+0x46d/0x640
> >  blk_mq_get_request+0x7c0/0xa00
> >  blk_mq_make_request+0x241/0xa70
> >  generic_make_request+0x411/0x950
> >  submit_bio+0x9b/0x250
> >  blkdev_direct_IO+0x7fb/0x870
> >  generic_file_direct_write+0x119/0x210
> >  __generic_file_write_iter+0x11c/0x280
> >  blkdev_write_iter+0x13c/0x220
> >  aio_write+0x204/0x310
> >  io_submit_one+0x9c6/0xe70
> >  __x64_sys_io_submit+0x115/0x340
> >  do_syscall_64+0x71/0x210
> >  entry_SYSCALL_64_after_hwframe+0x49/0xbe
> > 
> > When that hang occurs my list-pending-block-requests script does not show
> > any pending requests:
> > 
> > # list-pending-block-requests        
> > dm-0
> > loop0
> > loop1
> > loop2
> > loop3
> > loop4
> > loop5
> > loop6
> > loop7
> > nullb0
> > nullb1
> > sda
> > sdb
> > sdc
> > sdd
> > vda
> > vdb
> > 
> > Enabling fail_if_no_path mode did not resolve the hang so I don't think
> > that the root cause is in any of the dm drivers used in this test:
> > 
> > # dmsetup ls | while read dm rest; do dmsetup message $dm 0 fail_if_no_path; done; dmsetup remove_all; dmsetup table
> > 360014056e756c6c62300000000000000: 0 65536 multipath 0 1 alua 1 1 service-time 0 1 2 8:16 1 1
> > 
> > The same test passes against kernel v4.20-rc6.
> 
> What device is this being run on?

Older versions of the srp and nvmeof-mp tests used the brd block device.
Today these tests use null_blk with memory_backed set to 1. See also
configure_null_blk() in common/multipath-over-rdma. null_blk is accessed
by ib_srpt. The dm-mpath driver is stacked on top of the ib_srp instance
that communicates with ib_srpt driver. The ib_srp and ib_srpt drivers
communicate with each other over the loopback functionality of the
rdma_rxe driver.

Bart.