[SPDK] how to use spdk's nvme over fabrics

[SPDK] how to use spdk's nvme over fabrics

[Wang, Teng]

[-- Attachment #1: Type: text/plain, Size: 232 bytes --]

Hi all,

Is there any detailed instruction or sample code for using SPDK's NVMe over fabrics?
How to set up the experiment? I would really appreciate if anyone give me some
Instructions or materials on this.

Thanks,
Teng


[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 2265 bytes --]

Re: [SPDK] how to use spdk's nvme over fabrics

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 2265 bytes --]

The guide linked below is our best attempt at documentation so far. I do really need to write more. The general outline is the following, assuming you have two systems (initiator and target) connected via RDMA NICs:


1)      On the target system, compile SPDK. Follow the guide linked below to start the SPDK NVMe-oF target application. A fully documented example configuration file is in <spdk repo>/etc/nvmf/nvmf.conf.in � use that as the starting point.

2)      On the initiator system, make sure you are running Linux kernel 4.8 and have nvme-cli installed. This is outside of SPDK, so we don’t provide instructions on how to do this. By far, the easiest way is to just install Fedora 24/25 on the initiator. It has a suitable kernel by default and the nvme-cli tool is packaged, so you have zero additional configuration to do.

3)      The nvme-cli tool is capable of configuring the Linux kernel’s built-in NVMe-oF initiator to connect to a target. You simply type something like “nvme connect �t rdma �a <ip> -s <port> -n <nqn>”, where the values match the ones you put into the SPDK NVMe-oF target’s configuration file. The initiator will then connect and a block device will appear on the initiator system at /dev/nvmeX.

Only step #1 is SPDK-specific. Step #2 and #3 are the same if you are using the Linux kernel’s NVMe-oF target � that’s why we don’t really document those today.

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Lei Zhu
Sent: Tuesday, November 29, 2016 7:54 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] 答复: how to use spdk's nvme over fabrics

Hi,

You can follow this guide:
NVMe over Fabrics Target Getting Started Guide
http://www.spdk.io/spdk/doc/nvmf_getting_started.html

发件人: SPDK [mailto:spdk-bounces(a)lists.01.org] 代表 Wang, Teng
发送时间: 2016年11月30日 6:45
收件人: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>
主题: [SPDK] how to use spdk's nvme over fabrics

Hi all,

Is there any detailed instruction or sample code for using SPDK's NVMe over fabrics?
How to set up the experiment? I would really appreciate if anyone give me some
Instructions or materials on this.

Thanks,
Teng


[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 9796 bytes --]

Re: [SPDK] how to use spdk's nvme over fabrics

[Kariuki, John K]

[-- Attachment #1: Type: text/plain, Size: 3853 bytes --]

Teng
Here are some notes that I took on my system when starting the nvmf target. I put steps 1 � 3 and 5 in a script that I run whenever my system reboots.
1.      Load ib_rdma_modules
sudo modprobe ib_cm
sudo modprobe ib_core
sudo modprobe ib_ucm
sudo modprobe ib_umad
sudo modprobe ib_uverbs
sudo modprobe iw_cm
sudo modprobe rdma_cm
sudo modprobe rdma_ucm
2.      Detect Mellanox NICs
sudo modprobe mlx5_core
sudo modprobe mlx5_ib
3.      Assign NICs ip addresses
a.      sudo ifconfig eth1 192.168.100.8 netmask 255.255.255.0 up
b.      sudo ifconfig eth2 192.168.100.9 netmask 255.255.255.0 up
4.      Start the nvmf target
cd ~/spdk/app/nvmf_tgt
sudo ./nvmf_tgt -c nvmf.conf.in &

My configuration file has the following Subsystems

[Subsystem1]
  NQN nqn.2016-06.io.spdk:cnode1
  Core 0
  Mode Direct
  Listen RDMA 192.168.100.8:4420
  Host nqn.2016-06.io.spdk:init
  NVMe 0000:81:00.0

# Multiple subsystems are allowed.
[Subsystem2]
  NQN nqn.2016-06.io.spdk:cnode2
  Core 0
  Mode Direct
  Listen RDMA 192.168.100.9:4420
  Host nqn.2016-06.io.spdk:init
 NVMe 0000:86:00.0


5.      On the client load nvme-rdma kernel module
sudo modprobe -v nvme-rdma
6.      Use the cliet to discover the nvme target
sudo nvme discover -t rdma -a 192.168.100.8 -s 4420


From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
Sent: Wednesday, November 30, 2016 9:57 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] how to use spdk's nvme over fabrics

The guide linked below is our best attempt at documentation so far. I do really need to write more. The general outline is the following, assuming you have two systems (initiator and target) connected via RDMA NICs:


1)      On the target system, compile SPDK. Follow the guide linked below to start the SPDK NVMe-oF target application. A fully documented example configuration file is in <spdk repo>/etc/nvmf/nvmf.conf.in � use that as the starting point.

2)      On the initiator system, make sure you are running Linux kernel 4.8 and have nvme-cli installed. This is outside of SPDK, so we don’t provide instructions on how to do this. By far, the easiest way is to just install Fedora 24/25 on the initiator. It has a suitable kernel by default and the nvme-cli tool is packaged, so you have zero additional configuration to do.

3)      The nvme-cli tool is capable of configuring the Linux kernel’s built-in NVMe-oF initiator to connect to a target. You simply type something like “nvme connect �t rdma �a <ip> -s <port> -n <nqn>”, where the values match the ones you put into the SPDK NVMe-oF target’s configuration file. The initiator will then connect and a block device will appear on the initiator system at /dev/nvmeX.

Only step #1 is SPDK-specific. Step #2 and #3 are the same if you are using the Linux kernel’s NVMe-oF target � that’s why we don’t really document those today.

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Lei Zhu
Sent: Tuesday, November 29, 2016 7:54 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] 答复: how to use spdk's nvme over fabrics

Hi,

You can follow this guide:
NVMe over Fabrics Target Getting Started Guide
http://www.spdk.io/spdk/doc/nvmf_getting_started.html

发件人: SPDK [mailto:spdk-bounces(a)lists.01.org] 代表 Wang, Teng
发送时间: 2016年11月30日 6:45
收件人: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>
主题: [SPDK] how to use spdk's nvme over fabrics

Hi all,

Is there any detailed instruction or sample code for using SPDK's NVMe over fabrics?
How to set up the experiment? I would really appreciate if anyone give me some
Instructions or materials on this.

Thanks,
Teng


[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 21289 bytes --]

Re: [SPDK] how to use spdk's nvme over fabrics

[Wang, Teng]

[-- Attachment #1: Type: text/plain, Size: 4320 bytes --]

Hi Lei, Ben and John,

Thanks so much for your instructions! I heard that
intel is going to release its own SPDK initiator, may
I know the approximate time for this release?

Thanks,
Teng

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Kariuki, John K
Sent: Wednesday, November 30, 2016 2:21 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] how to use spdk's nvme over fabrics

Teng
Here are some notes that I took on my system when starting the nvmf target. I put steps 1 – 3 and 5 in a script that I run whenever my system reboots.
1.      Load ib_rdma_modules
sudo modprobe ib_cm
sudo modprobe ib_core
sudo modprobe ib_ucm
sudo modprobe ib_umad
sudo modprobe ib_uverbs
sudo modprobe iw_cm
sudo modprobe rdma_cm
sudo modprobe rdma_ucm
2.      Detect Mellanox NICs
sudo modprobe mlx5_core
sudo modprobe mlx5_ib
3.      Assign NICs ip addresses
a.      sudo ifconfig eth1 192.168.100.8 netmask 255.255.255.0 up
b.      sudo ifconfig eth2 192.168.100.9 netmask 255.255.255.0 up
4.      Start the nvmf target
cd ~/spdk/app/nvmf_tgt
sudo ./nvmf_tgt -c nvmf.conf.in &

My configuration file has the following Subsystems

[Subsystem1]
  NQN nqn.2016-06.io.spdk:cnode1
  Core 0
  Mode Direct
  Listen RDMA 192.168.100.8:4420
  Host nqn.2016-06.io.spdk:init
  NVMe 0000:81:00.0

# Multiple subsystems are allowed.
[Subsystem2]
  NQN nqn.2016-06.io.spdk:cnode2
  Core 0
  Mode Direct
  Listen RDMA 192.168.100.9:4420
  Host nqn.2016-06.io.spdk:init
 NVMe 0000:86:00.0


5.      On the client load nvme-rdma kernel module
sudo modprobe -v nvme-rdma
6.      Use the cliet to discover the nvme target
sudo nvme discover -t rdma -a 192.168.100.8 -s 4420


From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
Sent: Wednesday, November 30, 2016 9:57 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] how to use spdk's nvme over fabrics

The guide linked below is our best attempt at documentation so far. I do really need to write more. The general outline is the following, assuming you have two systems (initiator and target) connected via RDMA NICs:


1)     On the target system, compile SPDK. Follow the guide linked below to start the SPDK NVMe-oF target application. A fully documented example configuration file is in <spdk repo>/etc/nvmf/nvmf.conf.in – use that as the starting point.

2)     On the initiator system, make sure you are running Linux kernel 4.8 and have nvme-cli installed. This is outside of SPDK, so we don’t provide instructions on how to do this. By far, the easiest way is to just install Fedora 24/25 on the initiator. It has a suitable kernel by default and the nvme-cli tool is packaged, so you have zero additional configuration to do.

3)     The nvme-cli tool is capable of configuring the Linux kernel’s built-in NVMe-oF initiator to connect to a target. You simply type something like “nvme connect –t rdma –a <ip> -s <port> -n <nqn>”, where the values match the ones you put into the SPDK NVMe-oF target’s configuration file. The initiator will then connect and a block device will appear on the initiator system at /dev/nvmeX.

Only step #1 is SPDK-specific. Step #2 and #3 are the same if you are using the Linux kernel’s NVMe-oF target – that’s why we don’t really document those today.

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Lei Zhu
Sent: Tuesday, November 29, 2016 7:54 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] 答复: how to use spdk's nvme over fabrics

Hi,

You can follow this guide:
NVMe over Fabrics Target Getting Started Guide
http://www.spdk.io/spdk/doc/nvmf_getting_started.html

发件人: SPDK [mailto:spdk-bounces(a)lists.01.org] 代表 Wang, Teng
发送时间: 2016年11月30日 6:45
收件人: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>
主题: [SPDK] how to use spdk's nvme over fabrics

Hi all,

Is there any detailed instruction or sample code for using SPDK's NVMe over fabrics?
How to set up the experiment? I would really appreciate if anyone give me some
Instructions or materials on this.

Thanks,
Teng


[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 27540 bytes --]

Re: [SPDK] how to use spdk's nvme over fabrics

[Kirubakaran Kaliannan]

[-- Attachment #1: Type: text/plain, Size: 3026 bytes --]

Hi,



I have a question regarding this,



1.       Do the target to be on the same version (linux kernel 4.8) as of
initiator. ?  or is there any minimum version I have to be in. ? I am using
3.18 linux kernel

2.       Since the support is through the Mellanox, do we need to have the
same OFED version to be installed on both initiator and target ?



Thanks

-kiru



*From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Walker,
Benjamin
*Sent:* Wednesday, November 30, 2016 10:27 PM
*To:* Storage Performance Development Kit
*Subject:* Re: [SPDK] how to use spdk's nvme over fabrics



The guide linked below is our best attempt at documentation so far. I do
really need to write more. The general outline is the following, assuming
you have two systems (initiator and target) connected via RDMA NICs:



1)      On the target system, compile SPDK. Follow the guide linked below
to start the SPDK NVMe-oF target application. A fully documented example
configuration file is in <spdk repo>/etc/nvmf/nvmf.conf.in �C use that as
the starting point.

2)      On the initiator system, make sure you are running Linux kernel 4.8
and have nvme-cli installed. This is outside of SPDK, so we don’t provide
instructions on how to do this. By far, the easiest way is to just install
Fedora 24/25 on the initiator. It has a suitable kernel by default and the
nvme-cli tool is packaged, so you have zero additional configuration to do.

3)      The nvme-cli tool is capable of configuring the Linux kernel’s
built-in NVMe-oF initiator to connect to a target. You simply type
something like “nvme connect �Ct rdma �Ca <ip> -s <port> -n <nqn>”, where
the values match the ones you put into the SPDK NVMe-oF target’s
configuration file. The initiator will then connect and a block device will
appear on the initiator system at /dev/nvmeX.



Only step #1 is SPDK-specific. Step #2 and #3 are the same if you are using
the Linux kernel’s NVMe-oF target �C that’s why we don’t really document
those today.



*From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01.org>] *On
Behalf Of *Lei Zhu
*Sent:* Tuesday, November 29, 2016 7:54 PM
*To:* Storage Performance Development Kit <spdk(a)lists.01.org>
*Subject:* [SPDK] 答复: how to use spdk's nvme over fabrics



*Hi,*



*You can follow this guide:*

*NVMe over Fabrics Target Getting Started Guide*

http://www.spdk.io/spdk/doc/nvmf_getting_started.html



*发件人**:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01.org>]
*代表 *Wang, Teng
*发送时间:* 2016年11月30日 6:45
*收件人:* spdk(a)lists.01.org
*主题:* [SPDK] how to use spdk's nvme over fabrics



Hi all,



Is there any detailed instruction or sample code for using SPDK's NVMe over
fabrics?

How to set up the experiment? I would really appreciate if anyone give me
some

Instructions or materials on this.



Thanks,

Teng

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 10209 bytes --]

Re: [SPDK] how to use spdk's nvme over fabrics

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 4658 bytes --]

The NVMe-oF initiator will be released by the end of this month.

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Wang, Teng
Sent: Wednesday, November 30, 2016 7:07 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] how to use spdk's nvme over fabrics

Hi Lei, Ben and John,

Thanks so much for your instructions! I heard that
intel is going to release its own SPDK initiator, may
I know the approximate time for this release?

Thanks,
Teng

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Kariuki, John K
Sent: Wednesday, November 30, 2016 2:21 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] how to use spdk's nvme over fabrics

Teng
Here are some notes that I took on my system when starting the nvmf target. I put steps 1 – 3 and 5 in a script that I run whenever my system reboots.
1.      Load ib_rdma_modules
sudo modprobe ib_cm
sudo modprobe ib_core
sudo modprobe ib_ucm
sudo modprobe ib_umad
sudo modprobe ib_uverbs
sudo modprobe iw_cm
sudo modprobe rdma_cm
sudo modprobe rdma_ucm
2.      Detect Mellanox NICs
sudo modprobe mlx5_core
sudo modprobe mlx5_ib
3.      Assign NICs ip addresses
a.      sudo ifconfig eth1 192.168.100.8 netmask 255.255.255.0 up
b.      sudo ifconfig eth2 192.168.100.9 netmask 255.255.255.0 up
4.      Start the nvmf target
cd ~/spdk/app/nvmf_tgt
sudo ./nvmf_tgt -c nvmf.conf.in &

My configuration file has the following Subsystems

[Subsystem1]
  NQN nqn.2016-06.io.spdk:cnode1
  Core 0
  Mode Direct
  Listen RDMA 192.168.100.8:4420
  Host nqn.2016-06.io.spdk:init
  NVMe 0000:81:00.0

# Multiple subsystems are allowed.
[Subsystem2]
  NQN nqn.2016-06.io.spdk:cnode2
  Core 0
  Mode Direct
  Listen RDMA 192.168.100.9:4420
  Host nqn.2016-06.io.spdk:init
 NVMe 0000:86:00.0


5.      On the client load nvme-rdma kernel module
sudo modprobe -v nvme-rdma
6.      Use the cliet to discover the nvme target
sudo nvme discover -t rdma -a 192.168.100.8 -s 4420


From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
Sent: Wednesday, November 30, 2016 9:57 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] how to use spdk's nvme over fabrics

The guide linked below is our best attempt at documentation so far. I do really need to write more. The general outline is the following, assuming you have two systems (initiator and target) connected via RDMA NICs:


1)      On the target system, compile SPDK. Follow the guide linked below to start the SPDK NVMe-oF target application. A fully documented example configuration file is in <spdk repo>/etc/nvmf/nvmf.conf.in – use that as the starting point.

2)      On the initiator system, make sure you are running Linux kernel 4.8 and have nvme-cli installed. This is outside of SPDK, so we don’t provide instructions on how to do this. By far, the easiest way is to just install Fedora 24/25 on the initiator. It has a suitable kernel by default and the nvme-cli tool is packaged, so you have zero additional configuration to do.

3)      The nvme-cli tool is capable of configuring the Linux kernel’s built-in NVMe-oF initiator to connect to a target. You simply type something like “nvme connect –t rdma –a <ip> -s <port> -n <nqn>”, where the values match the ones you put into the SPDK NVMe-oF target’s configuration file. The initiator will then connect and a block device will appear on the initiator system at /dev/nvmeX.

Only step #1 is SPDK-specific. Step #2 and #3 are the same if you are using the Linux kernel’s NVMe-oF target – that’s why we don’t really document those today.

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Lei Zhu
Sent: Tuesday, November 29, 2016 7:54 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] 答复: how to use spdk's nvme over fabrics

Hi,

You can follow this guide:
NVMe over Fabrics Target Getting Started Guide
http://www.spdk.io/spdk/doc/nvmf_getting_started.html

发件人: SPDK [mailto:spdk-bounces(a)lists.01.org] 代表 Wang, Teng
发送时间: 2016年11月30日 6:45
收件人: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>
主题: [SPDK] how to use spdk's nvme over fabrics

Hi all,

Is there any detailed instruction or sample code for using SPDK's NVMe over fabrics?
How to set up the experiment? I would really appreciate if anyone give me some
Instructions or materials on this.

Thanks,
Teng


[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 27952 bytes --]