multipath target and non-request-stackable devices

multipath target and non-request-stackable devices

[Drew Hastings]

[-- Attachment #1.1: Type: text/plain, Size: 1559 bytes --]

Firstly, thanks for the hard work you guys are doing on the dm drivers.

I'm curious to know if I correctly understand the limitations of the
multipath target. I'm using kernel 4.19.0-rc5

When attempting to create a device from two NVMEs connected over nvme_rdma
/ nvmf, I get the following message:

[43012.002418] device-mapper: table: table load rejected: including
non-request-stackable devices
[43012.004067] device-mapper: table: unable to determine table type

Here's an example request that will fail:

dmsetup create test_path --table "0 1562824368 multipath 0 0 2 1
round-robin 0 1 1 /dev/nvme1n1 1 round-robin 0 1 1 /dev/nvme2n1 1"

This will work:

dmsetup create test_path --table "0 1562824368 multipath 0 0 2 1
round-robin 0 1 1 /dev/sdc 1 round-robin 0 1 1 /dev/sdb 1"

After looking at dm-table.c, it seems like this may be an issue of these
underlying devices not being compatible with the multipath driver. I had
noticed this same error when trying "non-physical" devices in other tests,
such as trying to multipath other virtual devices.

Is there anything that can be done to do the (relatively "basic") multipath
that detects IO errors and switches to the other device? It seems like
fundamentally multipath's error detection works in the same way raid1
implements IO error detection of the underlying devices, so why is it that
multipath seems to have this limitation but raid1 does not?

Is md-multipath.c still in use? It seems like it might not have this same
restriction on the underlying device.

Thank you so much for your time!

[-- Attachment #1.2: Type: text/html, Size: 2042 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: multipath target and non-request-stackable devices

[Mike Snitzer]

On Wed, Oct 31 2018 at  5:32am -0400,
Drew Hastings <dhastings@crucialwebhost.com> wrote:

>    Firstly, thanks for the hard work you guys are doing on the dm drivers.
>    I'm curious to know if I correctly understand the limitations of the
>    multipath target. I'm using kernel 4.19.0-rc5
>    When attempting to create a device from two NVMEs connected over nvme_rdma
>    / nvmf, I get the following message:
> 
>    [43012.002418] device-mapper: table: table load rejected: including
>    non-request-stackable devices
>    [43012.004067] device-mapper: table: unable to determine table type
>    Here's an example request that will fail:
>    dmsetup create test_path --table "0 1562824368 multipath 0 0 2 1
>    round-robin 0 1 1 /dev/nvme1n1 1 round-robin 0 1 1 /dev/nvme2n1 1"

dm-multipath works fine ontop of nvme.  Both blktests and mptest have
test coverage for dm-multipath on nvme:
https://github.com/osandov/blktests 
https://github.com/snitm/mptest

That said, there are 2 different modes that can be used when creating
the dm-multipath device,  The default is request-based ("mq") and there
is also bio-based ("bio"), e.g.:
 queue_mode mq
or
 queue_mode bio

It should be noted that in 4.20 request-based DM _only_ supports blk-mq
("queue_mode mq").  Using "rq" just means "mq" (and vice-versa) as of
4.20.

The error you've encountered shouldn't occur given that nvme devices use
blk-mq for their request_queue.  It could be that 4.19-rc5 had a bug
that was fixed but nothing springs to mind.

But I do recall hitting such an error during development and testing.
And dm_table_determine_type() still isn't as "clean" as I'd like...

>    This will work:
>    dmsetup create test_path --table "0 1562824368 multipath 0 0 2 1
>    round-robin 0 1 1 /dev/sdc 1 round-robin 0 1 1 /dev/sdb 1"
>    After looking at dm-table.c, it seems like this may be an issue of these
>    underlying devices not being compatible with the multipath driver. I had
>    noticed this same error when trying "non-physical" devices in other tests,
>    such as trying to multipath other virtual devices.
>    Is there anything that can be done to do the (relatively "basic")
>    multipath that detects IO errors and switches to the other device? It
>    seems like fundamentally multipath's error detection works in the same way
>    raid1 implements IO error detection of the underlying devices, so why is
>    it that multipath seems to have this limitation but raid1 does not?

request-based DM can only be stacked on request-based devices.  MD raid1
is bio-based, bio-based can stack on anything.

But again: nvme uses blk-mq.

>    Is md-multipath.c still in use? It seems like it might not have this same
>    restriction on the underlying device.

I don't know of anyone who uses MD multipath.

Mike

Re: multipath target and non-request-stackable devices

[Drew Hastings]

[-- Attachment #1.1: Type: text/plain, Size: 346 bytes --]

Perfect, thank you!

For what it's worth, the error did not happen with regular, locally
attached NVME devices. It only occurred with NVMEoF devices, with the
chelsio driver shipped with the kernel. You seemed to discuss this a bit in
https://www.redhat.com/archives/dm-devel/2016-June/msg00430.html

Anyway, this solves it for me. Thanks again!

[-- Attachment #1.2: Type: text/html, Size: 521 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]