Is SRP over RoCE possible?

Is SRP over RoCE possible?

[Robert LeBlanc]

I'm trying to get SRP working on my ConnectX-4-LX cards to see if it
has the same problem as iSER does. When I try to connect the initiator
to the target I get that opensm is not running. I tried all the umad
devices without any luck. I found that someone had to modify the
source to get it to work with RDMACM [0] and then another thread that
specified that an option had to be passed to the ib_srpt module, but I
don't see that option in modinfo and the Readme mentioned in the
thread is not there[1]. If SRP should work, can someone help me figure
out what I'm doing wrong here.

Thank you,
Robert LeBlanc

# targetcli ls
o- / .........................................................................................................................
[...]
 o- backstores ..............................................................................................................
[...]
 | o- block ..................................................................................................
[Storage Objects: 0]
 | o- fileio .................................................................................................
[Storage Objects: 0]
 | o- pscsi ..................................................................................................
[Storage Objects: 0]
 | o- ramdisk ................................................................................................
[Storage Objects: 1]
 |   o- test1 ..............................................................................................
[(100.0GiB) activated]
 o- iscsi ............................................................................................................
[Targets: 1]
 | o- iqn.2016-12.com.betterservers
.....................................................................................
[TPGs: 1]
 |   o- tpg1 ..................................................................................................
[gen-acls, no-auth]
 |     o- acls ..........................................................................................................
[ACLs: 0]
 |     o- luns ..........................................................................................................
[LUNs: 1]
 |     | o- lun0
..................................................................................................
[ramdisk/test1]
 |     o- portals
....................................................................................................
[Portals: 1]
 |       o- 0.0.0.0:3260
...................................................................................................
[iser]
 o- loopback .........................................................................................................
[Targets: 0]
 o- srpt .............................................................................................................
[Targets: 1]
   o- ib.fe800000000000000ec47afffe89f719
...........................................................................
[no-gen-acls]
     o- acls ............................................................................................................
[ACLs: 1]
     | o- ib.fe800000000000000ec47afffe880731
....................................................................
[Mapped LUNs: 1]
     |   o- mapped_lun0
.................................................................................
[lun0 ramdisk/test1 (rw)]
     o- luns ............................................................................................................
[LUNs: 1]
       o- lun0 ....................................................................................................
[ramdisk/test1]

# ibsrpdm -c
SM LID is 0, maybe no opensm is running
Querying SRP targets failed

# modinfo ib_srpt
filename:
/lib/modules/4.10.0-rc4+/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko
license:        Dual BSD/GPL
description:    InfiniBand SCSI RDMA Protocol target v2.0.0 (2011-02-14)
author:         Vu Pham and Bart Van Assche
depends:        ib_core,ib_cm
intree:         Y
vermagic:       4.10.0-rc4+ SMP mod_unload
parm:           srp_max_req_size:Maximum size of SRP request messages
in bytes. (int)
parm:           srpt_srq_size:Shared receive queue (SRQ) size. (int)
parm:           srpt_service_guid:Using this value for ioc_guid,
id_ext, and cm_listen_id instead of using the node_guid of the first
HCA.

[0] http://www.spinics.net/lists/linux-rdma/msg09083.html
[1] https://sourceforge.net/p/scst/mailman/message/34316586/
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Is SRP over RoCE possible?

[Bart Van Assche]

On Tue, 2017-02-14 at 11:24 -0700, Robert LeBlanc wrote:
> I'm trying to get SRP working on my ConnectX-4-LX cards to see if it
> has the same problem as iSER does. When I try to connect the initiator
> to the target I get that opensm is not running. I tried all the umad
> devices without any luck. I found that someone had to modify the
> source to get it to work with RDMACM [0] and then another thread that
> specified that an option had to be passed to the ib_srpt module, but I
> don't see that option in modinfo and the Readme mentioned in the
> thread is not there[1]. If SRP should work, can someone help me figure
> out what I'm doing wrong here.

Hello Robert,

This is not yet possible with the upstream SRP drivers. I'm waiting for
the standardization of SRP over RoCE to add such support to the upstream
SRP drivers. But this is already possible today with the following
drivers:
- At the initiator side: https://github.com/bvanassche/ib_srp-backport
- At the target side: the SCST core + the SCST ib_srpt target driver and
  something like the following in /etc/modprobe.d/ib_srpt.conf:
  options ib_srpt rdma_cm_port=5000
- Since neither ibsrpdm nor srp_daemon support SRP over RoCE, login has
  to be performed by writing a login string into
  /sys/class/infiniband_srp/*/add_target. The format of that login string
  is id_ext=$ioc_guid,ioc_guid=$ioc_guid,dest=$dest where ioc_guid is the
  following parameter from the target system:
  /sys/module/ib_srpt/parameters/srpt_service_guid
  and where dest is [<IPv4 address>|<IPv6 address>]:<SRP RDMA/CM port number>

BTW, I have used these drivers to test and improve the SoftRoCE driver.

Bart.--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Is SRP over RoCE possible?

[Robert LeBlanc]

Thanks, I'll give it a whirl.

Robert LeBlanc
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1


On Tue, Feb 14, 2017 at 11:45 AM, Bart Van Assche
<Bart.VanAssche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org> wrote:
> On Tue, 2017-02-14 at 11:24 -0700, Robert LeBlanc wrote:
>> I'm trying to get SRP working on my ConnectX-4-LX cards to see if it
>> has the same problem as iSER does. When I try to connect the initiator
>> to the target I get that opensm is not running. I tried all the umad
>> devices without any luck. I found that someone had to modify the
>> source to get it to work with RDMACM [0] and then another thread that
>> specified that an option had to be passed to the ib_srpt module, but I
>> don't see that option in modinfo and the Readme mentioned in the
>> thread is not there[1]. If SRP should work, can someone help me figure
>> out what I'm doing wrong here.
>
> Hello Robert,
>
> This is not yet possible with the upstream SRP drivers. I'm waiting for
> the standardization of SRP over RoCE to add such support to the upstream
> SRP drivers. But this is already possible today with the following
> drivers:
> - At the initiator side: https://github.com/bvanassche/ib_srp-backport
> - At the target side: the SCST core + the SCST ib_srpt target driver and
>   something like the following in /etc/modprobe.d/ib_srpt.conf:
>   options ib_srpt rdma_cm_port=5000
> - Since neither ibsrpdm nor srp_daemon support SRP over RoCE, login has
>   to be performed by writing a login string into
>   /sys/class/infiniband_srp/*/add_target. The format of that login string
>   is id_ext=$ioc_guid,ioc_guid=$ioc_guid,dest=$dest where ioc_guid is the
>   following parameter from the target system:
>   /sys/module/ib_srpt/parameters/srpt_service_guid
>   and where dest is [<IPv4 address>|<IPv6 address>]:<SRP RDMA/CM port number>
>
> BTW, I have used these drivers to test and improve the SoftRoCE driver.
>
> Bart.
> Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer:
>
> This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system.
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Is SRP over RoCE possible?

[Robert LeBlanc]

Trying to add "options ib_srpt rdma_cm_port=5000" gives me this error.

[   25.318411] ib_srpt: unknown parameter 'rdma_cm_port' ignored

This is on 4.10-rc4.
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1


On Tue, Feb 14, 2017 at 11:45 AM, Bart Van Assche
<Bart.VanAssche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org> wrote:
> On Tue, 2017-02-14 at 11:24 -0700, Robert LeBlanc wrote:
>> I'm trying to get SRP working on my ConnectX-4-LX cards to see if it
>> has the same problem as iSER does. When I try to connect the initiator
>> to the target I get that opensm is not running. I tried all the umad
>> devices without any luck. I found that someone had to modify the
>> source to get it to work with RDMACM [0] and then another thread that
>> specified that an option had to be passed to the ib_srpt module, but I
>> don't see that option in modinfo and the Readme mentioned in the
>> thread is not there[1]. If SRP should work, can someone help me figure
>> out what I'm doing wrong here.
>
> Hello Robert,
>
> This is not yet possible with the upstream SRP drivers. I'm waiting for
> the standardization of SRP over RoCE to add such support to the upstream
> SRP drivers. But this is already possible today with the following
> drivers:
> - At the initiator side: https://github.com/bvanassche/ib_srp-backport
> - At the target side: the SCST core + the SCST ib_srpt target driver and
>   something like the following in /etc/modprobe.d/ib_srpt.conf:
>   options ib_srpt rdma_cm_port=5000
> - Since neither ibsrpdm nor srp_daemon support SRP over RoCE, login has
>   to be performed by writing a login string into
>   /sys/class/infiniband_srp/*/add_target. The format of that login string
>   is id_ext=$ioc_guid,ioc_guid=$ioc_guid,dest=$dest where ioc_guid is the
>   following parameter from the target system:
>   /sys/module/ib_srpt/parameters/srpt_service_guid
>   and where dest is [<IPv4 address>|<IPv6 address>]:<SRP RDMA/CM port number>
>
> BTW, I have used these drivers to test and improve the SoftRoCE driver.
>
> Bart.
> Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer:
>
> This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system.
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Is SRP over RoCE possible?

[Bart Van Assche]

On Tue, 2017-02-14 at 13:34 -0700, Robert LeBlanc wrote:
> Trying to add "options ib_srpt rdma_cm_port=5000" gives me this error.
> 
> [   25.318411] ib_srpt: unknown parameter 'rdma_cm_port' ignored

Does this mean that you are not familiar with SCST? I referred explicitly
to the SCST ib_srpt kernel module in my e-mail.

Bart.--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Is SRP over RoCE possible?

[Robert LeBlanc]

On Tue, Feb 14, 2017 at 1:36 PM, Bart Van Assche
<Bart.VanAssche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org> wrote:
> On Tue, 2017-02-14 at 13:34 -0700, Robert LeBlanc wrote:
>> Trying to add "options ib_srpt rdma_cm_port=5000" gives me this error.
>>
>> [   25.318411] ib_srpt: unknown parameter 'rdma_cm_port' ignored
>
> Does this mean that you are not familiar with SCST? I referred explicitly
> to the SCST ib_srpt kernel module in my e-mail.

I haven't used SCST in almost a decade, favoring LIO. Using SCST may
compromise the goal of running SRP which is if I can reproduce the
iSER hang that I've been experiencing with SRP. That would help
pinpoint if the bug is in iSER, ib_core, or the Mellanox drivers. With
compiling SCST modules, I'm not sure how much that will invalidate my
thought process. I'm still going to give it a try.

Thanks,
Robert

----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Is SRP over RoCE possible?

[Robert LeBlanc]

On Tue, Feb 14, 2017 at 1:52 PM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote:
> On Tue, Feb 14, 2017 at 1:36 PM, Bart Van Assche
> <Bart.VanAssche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org> wrote:
>> On Tue, 2017-02-14 at 13:34 -0700, Robert LeBlanc wrote:
>>> Trying to add "options ib_srpt rdma_cm_port=5000" gives me this error.
>>>
>>> [   25.318411] ib_srpt: unknown parameter 'rdma_cm_port' ignored
>>
>> Does this mean that you are not familiar with SCST? I referred explicitly
>> to the SCST ib_srpt kernel module in my e-mail.
>
> I haven't used SCST in almost a decade, favoring LIO. Using SCST may
> compromise the goal of running SRP which is if I can reproduce the
> iSER hang that I've been experiencing with SRP. That would help
> pinpoint if the bug is in iSER, ib_core, or the Mellanox drivers. With
> compiling SCST modules, I'm not sure how much that will invalidate my
> thought process. I'm still going to give it a try.
>
> Thanks,
> Robert

I got things running with SRP and SCST and I could not get the hang
with SRP. Thanks for your help.

----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Is SRP over RoCE possible?

[Bart Van Assche]

On Tue, 2017-02-14 at 16:43 -0700, Robert LeBlanc wrote:
> I got things running with SRP and SCST and I could not get the hang
> with SRP. Thanks for your help.

Hello Robert,

Thank you for the testing and for the feedback to the linux-rdma mailing
list. If switching to the NVMe-over-Fabrics protocol is not a valid
alternative on your setup I think this means that it would be valuable
to have SRP-over-RoCE support upstream ...

Bart.--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Is SRP over RoCE possible?

[Robert LeBlanc]

On Tue, Feb 14, 2017 at 4:51 PM, Bart Van Assche
<Bart.VanAssche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org> wrote:
> On Tue, 2017-02-14 at 16:43 -0700, Robert LeBlanc wrote:
>> I got things running with SRP and SCST and I could not get the hang
>> with SRP. Thanks for your help.
>
> Hello Robert,
>
> Thank you for the testing and for the feedback to the linux-rdma mailing
> list. If switching to the NVMe-over-Fabrics protocol is not a valid
> alternative on your setup I think this means that it would be valuable
> to have SRP-over-RoCE support upstream ...

I certainly like to have as many options as possible, so I'd sure like
to see it.

I was under the impression that the backing device for
NVMe-over-Fabrics had to be an NVMe device. We are using ZFS for our
backing stores, if it could work with that I need to really
investigate that option. I'll have to do some more research on the
subject and get myself more familiar with it instead of just assuming
things based on the name.

Thank you,
Robert LeBlanc

----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Is SRP over RoCE possible?

[Bart Van Assche]

On Tue, 2017-02-14 at 18:10 -0700, Robert LeBlanc wrote:
> I was under the impression that the backing device for
> NVMe-over-Fabrics had to be an NVMe device. We are using ZFS for our
> backing stores, if it could work with that I need to really
> investigate that option. I'll have to do some more research on the
> subject and get myself more familiar with it instead of just assuming
> things based on the name.

Hello Robert,

Are you using ZFS on Solaris or on Linux? In the last case, are you aware
that the Linux NVMeoF target driver supports all block devices? That
includes regular block devices SCSI devices and also the loop device. The
last option allows to export a file through the NVMeOF target driver.

Bart.

Re: Is SRP over RoCE possible?

[Robert LeBlanc]

On Tue, Feb 14, 2017 at 9:32 PM, Bart Van Assche
<Bart.VanAssche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org> wrote:
> Hello Robert,
>
> Are you using ZFS on Solaris or on Linux? In the last case, are you aware
> that the Linux NVMeoF target driver supports all block devices? That
> includes regular block devices SCSI devices and also the loop device. The
> last option allows to export a file through the NVMeOF target driver.

We are using ZFS on Linux. I did some reading and got NVMeoF working
on our RoCE cards using a loop device. It was surprisingly easy and I
didn't have any connection problems or D state issues. I'd like to
know how the performance compares to iSER, but this D state problem
prevents getting data on the iSER side.

Thanks for helping me understand that NVMeoF can use a regular block device.

Robert LeBlanc

----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Is SRP over RoCE possible?

[Robert LeBlanc]

Oh, I think you are saying that SCST (I was reading SCSI) has it's own
ib_srp kernel driver. I've been using LIO, let me look into that.
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1


On Tue, Feb 14, 2017 at 1:34 PM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote:
> Trying to add "options ib_srpt rdma_cm_port=5000" gives me this error.
>
> [   25.318411] ib_srpt: unknown parameter 'rdma_cm_port' ignored
>
> This is on 4.10-rc4.
> ----------------
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
>
>
> On Tue, Feb 14, 2017 at 11:45 AM, Bart Van Assche
> <Bart.VanAssche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org> wrote:
>> On Tue, 2017-02-14 at 11:24 -0700, Robert LeBlanc wrote:
>>> I'm trying to get SRP working on my ConnectX-4-LX cards to see if it
>>> has the same problem as iSER does. When I try to connect the initiator
>>> to the target I get that opensm is not running. I tried all the umad
>>> devices without any luck. I found that someone had to modify the
>>> source to get it to work with RDMACM [0] and then another thread that
>>> specified that an option had to be passed to the ib_srpt module, but I
>>> don't see that option in modinfo and the Readme mentioned in the
>>> thread is not there[1]. If SRP should work, can someone help me figure
>>> out what I'm doing wrong here.
>>
>> Hello Robert,
>>
>> This is not yet possible with the upstream SRP drivers. I'm waiting for
>> the standardization of SRP over RoCE to add such support to the upstream
>> SRP drivers. But this is already possible today with the following
>> drivers:
>> - At the initiator side: https://github.com/bvanassche/ib_srp-backport
>> - At the target side: the SCST core + the SCST ib_srpt target driver and
>>   something like the following in /etc/modprobe.d/ib_srpt.conf:
>>   options ib_srpt rdma_cm_port=5000
>> - Since neither ibsrpdm nor srp_daemon support SRP over RoCE, login has
>>   to be performed by writing a login string into
>>   /sys/class/infiniband_srp/*/add_target. The format of that login string
>>   is id_ext=$ioc_guid,ioc_guid=$ioc_guid,dest=$dest where ioc_guid is the
>>   following parameter from the target system:
>>   /sys/module/ib_srpt/parameters/srpt_service_guid
>>   and where dest is [<IPv4 address>|<IPv6 address>]:<SRP RDMA/CM port number>
>>
>> BTW, I have used these drivers to test and improve the SoftRoCE driver.
>>
>> Bart.
>> Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer:
>>
>> This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system.
>>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html