From: sagi@grimberg.me (Sagi Grimberg)
Subject: [dm-devel] NVMeoF multi-path setup
Date: Wed, 13 Jul 2016 13:19:34 +0300 [thread overview]
Message-ID: <578615B6.8030702@grimberg.me> (raw)
In-Reply-To: <20160630225207.GB22293@redhat.com>
On 01/07/16 01:52, Mike Snitzer wrote:
> On Thu, Jun 30 2016 at 5:57pm -0400,
> Ming Lin <mlin@kernel.org> wrote:
>
>> On Thu, 2016-06-30@14:08 -0700, Ming Lin wrote:
>>> Hi Mike,
>>>
>>> I'm trying to test NVMeoF multi-path.
>>>
>>> root at host:~# lsmod |grep dm_multipath
>>> dm_multipath 24576 0
>>> root at host:~# ps aux |grep multipath
>>> root 13183 0.0 0.1 238452 4972 ? SLl 13:41 0:00
>>> /sbin/multipathd
>>>
>>> I have nvme0 and nvme1 that are 2 paths to the same NVMe subsystem.
>>>
>>> root at host:/sys/class/nvme# grep . nvme*/address
>>> nvme0/address:traddr=192.168.3.2,trsvcid=1023
>>> nvme1/address:traddr=192.168.2.2,trsvcid=1023
>>>
>>> root at host:/sys/class/nvme# grep . nvme*/subsysnqn
>>> nvme0/subsysnqn:nqn.testiqn
>>> nvme1/subsysnqn:nqn.testiqn
>>>
>>> root at host:~# /lib/udev/scsi_id --export --whitelisted -d /dev/nvme1n1
>>> ID_SCSI=1
>>> ID_VENDOR=NVMe
>>> ID_VENDOR_ENC=NVMe\x20\x20\x20\x20
>>> ID_MODEL=Linux
>>> ID_MODEL_ENC=Linux
>>> ID_REVISION=0-rc
>>> ID_TYPE=disk
>>> ID_SERIAL=SNVMe_Linux
>>> ID_SERIAL_SHORT=
>>> ID_SCSI_SERIAL=1122334455667788
>>>
>>> root at host:~# /lib/udev/scsi_id --export --whitelisted -d /dev/nvme0n1
>>> ID_SCSI=1
>>> ID_VENDOR=NVMe
>>> ID_VENDOR_ENC=NVMe\x20\x20\x20\x20
>>> ID_MODEL=Linux
>>> ID_MODEL_ENC=Linux
>>> ID_REVISION=0-rc
>>> ID_TYPE=disk
>>> ID_SERIAL=SNVMe_Linux
>>> ID_SERIAL_SHORT=
>>> ID_SCSI_SERIAL=1122334455667788
>>>
>>> But seems multipathd didn't recognize these 2 devices.
>>>
>>> What else I'm missing?
>>
>> There are two problems:
>>
>> 1. there is no "/block/" in the path
>>
>> /sys/devices/virtual/nvme-fabrics/block/nvme0/nvme0n1
>
> You clarified that it is:
> /sys/devices/virtual/nvme-fabrics/ctl/nvme0/nvme0n1
>
> Do you have CONFIG_BLK_DEV_NVME_SCSI enabled?
Indeed, for dm-multipath we need CONFIG_BLK_DEV_NVME_SCSI on.
Another thing I noticed was that for nvme we need to manually
set the timeout value because nvme devices don't expose
device/timeout sysfs file. This causes dm-multipath to take
a 200 seconds default (not a huge problem because we
have keep alive in fabrics too).
prev parent reply other threads:[~2016-07-13 10:19 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-06-30 21:08 NVMeoF multi-path setup Ming Lin
2016-06-30 21:57 ` Ming Lin
2016-06-30 22:19 ` Ming Lin
2016-06-30 22:52 ` Mike Snitzer
2016-06-30 22:57 ` Mike Snitzer
2016-06-30 23:14 ` Keith Busch
2016-07-13 10:19 ` Sagi Grimberg [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=578615B6.8030702@grimberg.me \
--to=sagi@grimberg.me \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).