Re: [PATCH blktests 0/3] Add NVMeOF multipath tests

Bart Van Assche
2018-09-28T05:47:30Z

On Tue, 2018-09-18 at 17:18 -0700, Omar Sandoval wrote:
> On Tue, Sep 18, 2018 at 05:02:47PM -0700, Bart Van Assche wrote:
> > On 9/18/18 4:24 PM, Omar Sandoval wrote:
> > > On Tue, Sep 18, 2018 at 02:20:59PM -0700, Bart Van Assche wrote:
> > > > Can you have a look at the updated master branch of
> > > > https://github.com/bvanassche/blktests? That code should no longer fail if
> > > > unloading the nvme kernel module fails. Please note that you will need
> > > > kernel v4.18 to test these scripts - a KASAN complaint appears if I run
> > > > these tests against kernel v4.19-rc4.
> > >
> > > Thanks, these pass now. Is it expected that my nvme device gets a
> > > multipath device configured after running these tests?
> > >
> > > $ lsblk
> > > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
> > > vda 254:0 0 16G 0 disk
> > > └─vda1 254:1 0 16G 0 part /
> > > vdb 254:16 0 8G 0 disk
> > > vdc 254:32 0 8G 0 disk
> > > vdd 254:48 0 8G 0 disk
> > > nvme0n1 259:0 0 8G 0 disk
> > > └─mpatha 253:0 0 8G 0 mpath
> >
> > No, all multipath devices that were created during a test should be removed
> > before that test finishes. I will look into this.
> >
> > > Also, can you please fix:
> > >
> > > _have_kernel_option NVME_MULTIPATH && exit 1
> > >
> > > to not exit on failure? It should use SKIP_REASON and return 1. You
> > > might need to add something like _dont_have_kernel_option to properly
> > > handle the case where the config is not found.
> >
> > OK, I will change this.
> >
> > > Side note which isn't a blocker for merging is that there's a lot of
> > > duplicated code between these helpers and the srp helpers. How hard
> > > would it be to refactor that?
> >
> > Are you perhaps referring to the code that is shared between the
> > tests/srp/rc tests/nvmeof-mp/rc shell scripts?
>
> Yes, those.
>
> > The hardest part is probably
> > to chose a location where to store these functions. Should I create a file
> > with common code under common/, under tests/srp/, under tests/nvmeof-mp/ or
> > perhaps somewhere else?
>
> Just put it under common.

Hi Omar,

All feedback mentioned above has been addressed. The following pull request has
been updated: https://github.com/osandov/blktests/pull/33. Please let me know
if you want me to post these patches on the linux-block mailing list.

Note: neither the upstream kernel v4.18 nor v4.19-rc4 are stable enough to pass
all nvmeof-mp tests if kernel debugging options like KASAN are enabled.
Additionally, the NVMe device_add_disk() race condition often causes multipathd
to refuse to consider /dev/nvme... devices. The output on my test setup is as
follows (all tests pass):

# ./check -q nvmeof-mp
nvmeof-mp/001 (Log in and log out) [passed]
runtime 1.528s ... 1.909s
nvmeof-mp/002 (File I/O on top of multipath concurrently with logout and login (mq)) [
passed]time 38.968s ...
runtime 38.968s ... 38.571s
nvmeof-mp/004 (File I/O on top of multipath concurrently with logout and login (sq-on-
nvmeof-mp/004 (File I/O on top of multipath concurrently with logout and login (sq-on-
mq)) [passed]38.632s ...
runtime 38.632s ... 37.529s
nvmeof-mp/005 (Direct I/O with large transfer sizes and bs=4M) [passed]
runtime 13.382s ... 13.684s
nvmeof-mp/006 (Direct I/O with large transfer sizes and bs=8M) [passed]
runtime 13.511s ... 13.480s
nvmeof-mp/009 (Buffered I/O with large transfer sizes and bs=4M) [passed]
runtime 13.665s ... 13.763s
nvmeof-mp/010 (Buffered I/O with large transfer sizes and bs=8M) [passed]
runtime 13.442s ... 13.900s
nvmeof-mp/011 (Block I/O on top of multipath concurrently with logout and login) [pass
ed] runtime 37.988s ...
runtime 37.988s ... 37.945s
nvmeof-mp/012 (dm-mpath on top of multiple I/O schedulers) [passed]
runtime 21.659s ... 21.733s

Thanks,

Bart.