Loopback with RXE?

Loopback with RXE?

[Yi Bairen]

Hi folks,

Seems I can’t get a Linux 4.9 (Debian 9.3) VM working with single box SoftRoCE loopback. I installed rdma-core 16. The rdma_client & rdma_server in the same box seems working. However, when I use ib_write_bw in the same box, the QPs can’t be connected through the same rxe device.

I’d like to know if rxe loopback is known to be broken, and how to fix it.
 
Best,
Bairen Yi

Re: Loopback with RXE?

[Majd Dibbiny]

> On Feb 14, 2018, at 7:13 PM, Yi Bairen <byron@rbird.tech> wrote:
> 
> Hi folks,
> 
> Seems I can’t get a Linux 4.9 (Debian 9.3) VM working with single box SoftRoCE loopback. I installed rdma-core 16. The rdma_client & rdma_server in the same box seems working. However, when I use ib_write_bw in the same box, the QPs can’t be connected through the same rxe device.
> 
What is the error that you see?
Can you provide command lines and output?
> I’d like to know if rxe loopback is known to be broken, and how to fix it.
> 
> Best,
> Bairen Yi\x13��칻\x1c�&�~�&�\x18��+-��ݶ\x17��w��˛���m�b��kvf���^n�r���z�\x1a��h����&��\x1e�G���h�\x03(�階�ݢj"��\x1a�^[m�����z�ޖ���f���h���~�m�

Re: Loopback with RXE?

[Yuval Shaia]

On Wed, Feb 14, 2018 at 05:13:07PM +0000, Yi Bairen wrote:
> Hi folks,
> 
> Seems I can’t get a Linux 4.9 (Debian 9.3) VM working with single box SoftRoCE loopback. I installed rdma-core 16. The rdma_client & rdma_server in the same box seems working. However, when I use ib_write_bw in the same box, the QPs can’t be connected through the same rxe device.

If the QPs connection is problematic then how come rdma_* is working?

> 
> I’d like to know if rxe loopback is known to be broken, and how to fix it.
>  
> Best,
> Bairen Yi
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Loopback with RXE?

[Yi Bairen]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 3553 bytes --]

Hi,

> On 15 Feb 2018, at 03:25, Yuval Shaia <yuval.shaia@oracle.com> wrote:
> 
> On Wed, Feb 14, 2018 at 05:13:07PM +0000, Yi Bairen wrote:
>> Hi folks,
>> 
>> Seems I can’t get a Linux 4.9 (Debian 9.3) VM working with single box SoftRoCE loopback. I installed rdma-core 16. The rdma_client & rdma_server in the same box seems working. However, when I use ib_write_bw in the same box, the QPs can’t be connected through the same rxe device.
> 
> If the QPs connection is problematic then how come rdma_* is working?

The rdma_client and rdma_server do work with `end 0`. 

The ib_write_bw seems to hang. If I kill the server side, the client does not respond. If I kill the client side, it shows the following log:

$ ib_write_bw

************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF		Device         : rxe0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 CQ Moderation   : 100
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 1
 Max inline data : 0[B]
 rdma_cm QPs	 : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x0014 PSN 0x3787ca RKey 0x00052c VAddr 0x007f3dd4c42000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:159:65:13:34
 remote address: LID 0000 QPN 0x0015 PSN 0x5c8ba9 RKey 0x000658 VAddr 0x007f30e74b3000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:159:65:13:34
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
ethernet_read_keys: Couldn't read remote address
 Unable to read to socket/rdam_cm
 Failed to exchange data between server and clients

 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
ethernet_read_keys: Couldn't read remote address
 Unable to read to socket/rdam_cm
 Failed to exchange data between server and clients

My RXE config:

$ sudo rxe_cfg start
  Name     Link  Driver      Speed  NMTU  IPv4_addr  RDEV  RMTU
  docker0  no    bridge
  eth0     yes   virtio_net                          rxe0  1024  (3)
  eth1     yes   virtio_net                          rxe1  1024  (3)

dmesg:

[14441.336707] rdma_rxe: loaded
[14441.375108] Loading iSCSI transport class v2.0-870.
[14441.380930] rdma_rxe: set rxe0 active
[14441.381824] rdma_rxe: added rxe0 to eth0
[14441.389567] iscsi: registered transport (iser)
[14441.403595] RPC: Registered named UNIX socket transport module.
[14441.404618] RPC: Registered udp transport module.
[14441.405390] RPC: Registered tcp transport module.
[14441.406057] RPC: Registered tcp NFSv4.1 backchannel transport module.
[14441.415816] RPC: Registered rdma transport module.
[14441.416689] RPC: Registered rdma backchannel transport module.
[14441.419979] rdma_rxe: set rxe1 active
[14441.420657] rdma_rxe: added rxe1 to eth1
[14470.422836] detected loopback device
[14484.278022] rdma_rxe: qp#18 moved to error state

> 
>> 
>> I’d like to know if rxe loopback is known to be broken, and how to fix it.
>> 
>> Best,
>> Bairen Yi

Bairen Yi

N‹§²æìr¸›yúèšØb²X¬¶Ç§vØ^–)Þº{.nÇ+‰·¥Š{±­ÙšŠ{ayº\x1dʇڙë,j\a­¢f£¢·hš‹»öì\x17/oSc¾™Ú³9˜uÀ¦æå‰È&jw¨®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿïêäz¹Þ–Šàþf£¢·hšˆ§~ˆmš

Re: Loopback with RXE?

[Majd Dibbiny]

> On Feb 17, 2018, at 3:03 PM, Yi Bairen <byron@rbird.tech> wrote:
> 
> Hi,
> 
>> On 15 Feb 2018, at 03:25, Yuval Shaia <yuval.shaia@oracle.com> wrote:
>> 
>> On Wed, Feb 14, 2018 at 05:13:07PM +0000, Yi Bairen wrote:
>>> Hi folks,
>>> 
>>> Seems I can’t get a Linux 4.9 (Debian 9.3) VM working with single box SoftRoCE loopback. I installed rdma-core 16. The rdma_client & rdma_server in the same box seems working. However, when I use ib_write_bw in the same box, the QPs can’t be connected through the same rxe device.
>> 
>> If the QPs connection is problematic then how come rdma_* is working?
> 
> The rdma_client and rdma_server do work with `end 0`. 
> 
> The ib_write_bw seems to hang. If I kill the server side, the client does not respond. If I kill the client side, it shows the following log:
> 
> $ ib_write_bw
Did you try to specify a gid index with -x param? For example ib_write_bw -x 0.

Without gid I would expect init2rtr to fail.. which doesn’t seem the case though ..
> 
> ************************************
> * Waiting for client to connect... *
> ************************************
> ---------------------------------------------------------------------------------------
>                    RDMA_Write BW Test
> Dual-port       : OFF        Device         : rxe0
> Number of qps   : 1        Transport type : IB
> Connection type : RC        Using SRQ      : OFF
> CQ Moderation   : 100
> Mtu             : 1024[B]
> Link type       : Ethernet
> GID index       : 1
> Max inline data : 0[B]
> rdma_cm QPs     : OFF
> Data ex. method : Ethernet
> ---------------------------------------------------------------------------------------
> local address: LID 0000 QPN 0x0014 PSN 0x3787ca RKey 0x00052c VAddr 0x007f3dd4c42000
> GID: 00:00:00:00:00:00:00:00:00:00:255:255:159:65:13:34
> remote address: LID 0000 QPN 0x0015 PSN 0x5c8ba9 RKey 0x000658 VAddr 0x007f30e74b3000
> GID: 00:00:00:00:00:00:00:00:00:00:255:255:159:65:13:34
> ---------------------------------------------------------------------------------------
> #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
> ethernet_read_keys: Couldn't read remote address
> Unable to read to socket/rdam_cm
> Failed to exchange data between server and clients
> 
> #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
> ethernet_read_keys: Couldn't read remote address
> Unable to read to socket/rdam_cm
> Failed to exchange data between server and clients
> 
> My RXE config:
> 
> $ sudo rxe_cfg start
>  Name     Link  Driver      Speed  NMTU  IPv4_addr  RDEV  RMTU
>  docker0  no    bridge
>  eth0     yes   virtio_net                          rxe0  1024  (3)
>  eth1     yes   virtio_net                          rxe1  1024  (3)
> 
> dmesg:
> 
> [14441.336707] rdma_rxe: loaded
> [14441.375108] Loading iSCSI transport class v2.0-870.
> [14441.380930] rdma_rxe: set rxe0 active
> [14441.381824] rdma_rxe: added rxe0 to eth0
> [14441.389567] iscsi: registered transport (iser)
> [14441.403595] RPC: Registered named UNIX socket transport module.
> [14441.404618] RPC: Registered udp transport module.
> [14441.405390] RPC: Registered tcp transport module.
> [14441.406057] RPC: Registered tcp NFSv4.1 backchannel transport module.
> [14441.415816] RPC: Registered rdma transport module.
> [14441.416689] RPC: Registered rdma backchannel transport module.
> [14441.419979] rdma_rxe: set rxe1 active
> [14441.420657] rdma_rxe: added rxe1 to eth1
> [14470.422836] detected loopback device
> [14484.278022] rdma_rxe: qp#18 moved to error state
> 
>> 
>>> 
>>> I’d like to know if rxe loopback is known to be broken, and how to fix it.
>>> 
>>> Best,
>>> Bairen Yi
> 
> Bairen Yi
> 
> N�����r��y���b�X��ǧv�^�)޺{.n�+����{��ٚ�{ay�\x1dʇڙ�,j\a��f���h���z�\x1e�w���\f���j:+v���w�j�m����\a����zZ+�����ݢj"��!�i

Re: Loopback with RXE?

[Bairen Yi]

Hi folks,

Just a quick update, upgrading to kernel v4.14 solves my issue.

It looks like rdma-core 16 in Debian stretch-backports does not work with kernel v4.9 in Debian stretch.

And yep, matching is important :P

Best,
Bairen

> On 18 Feb 2018, at 05:23, Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> wrote:
> 
>> 
>> On Feb 17, 2018, at 3:03 PM, Yi Bairen <byron-DzDj5FMp9MpyDzI6CaY1VQ@public.gmane.org> wrote:
>> 
>> Hi,
>> 
>>> On 15 Feb 2018, at 03:25, Yuval Shaia <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> wrote:
>>> 
>>> On Wed, Feb 14, 2018 at 05:13:07PM +0000, Yi Bairen wrote:
>>>> Hi folks,
>>>> 
>>>> Seems I can’t get a Linux 4.9 (Debian 9.3) VM working with single box SoftRoCE loopback. I installed rdma-core 16. The rdma_client & rdma_server in the same box seems working. However, when I use ib_write_bw in the same box, the QPs can’t be connected through the same rxe device.
>>> 
>>> If the QPs connection is problematic then how come rdma_* is working?
>> 
>> The rdma_client and rdma_server do work with `end 0`. 
>> 
>> The ib_write_bw seems to hang. If I kill the server side, the client does not respond. If I kill the client side, it shows the following log:
>> 
>> $ ib_write_bw
> Did you try to specify a gid index with -x param? For example ib_write_bw -x 0.
> 
> Without gid I would expect init2rtr to fail.. which doesn’t seem the case though ..
>> 
>> ************************************
>> * Waiting for client to connect... *
>> ************************************
>> ---------------------------------------------------------------------------------------
>>                   RDMA_Write BW Test
>> Dual-port       : OFF        Device         : rxe0
>> Number of qps   : 1        Transport type : IB
>> Connection type : RC        Using SRQ      : OFF
>> CQ Moderation   : 100
>> Mtu             : 1024[B]
>> Link type       : Ethernet
>> GID index       : 1
>> Max inline data : 0[B]
>> rdma_cm QPs     : OFF
>> Data ex. method : Ethernet
>> ---------------------------------------------------------------------------------------
>> local address: LID 0000 QPN 0x0014 PSN 0x3787ca RKey 0x00052c VAddr 0x007f3dd4c42000
>> GID: 00:00:00:00:00:00:00:00:00:00:255:255:159:65:13:34
>> remote address: LID 0000 QPN 0x0015 PSN 0x5c8ba9 RKey 0x000658 VAddr 0x007f30e74b3000
>> GID: 00:00:00:00:00:00:00:00:00:00:255:255:159:65:13:34
>> ---------------------------------------------------------------------------------------
>> #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
>> ethernet_read_keys: Couldn't read remote address
>> Unable to read to socket/rdam_cm
>> Failed to exchange data between server and clients
>> 
>> #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
>> ethernet_read_keys: Couldn't read remote address
>> Unable to read to socket/rdam_cm
>> Failed to exchange data between server and clients
>> 
>> My RXE config:
>> 
>> $ sudo rxe_cfg start
>> Name     Link  Driver      Speed  NMTU  IPv4_addr  RDEV  RMTU
>> docker0  no    bridge
>> eth0     yes   virtio_net                          rxe0  1024  (3)
>> eth1     yes   virtio_net                          rxe1  1024  (3)
>> 
>> dmesg:
>> 
>> [14441.336707] rdma_rxe: loaded
>> [14441.375108] Loading iSCSI transport class v2.0-870.
>> [14441.380930] rdma_rxe: set rxe0 active
>> [14441.381824] rdma_rxe: added rxe0 to eth0
>> [14441.389567] iscsi: registered transport (iser)
>> [14441.403595] RPC: Registered named UNIX socket transport module.
>> [14441.404618] RPC: Registered udp transport module.
>> [14441.405390] RPC: Registered tcp transport module.
>> [14441.406057] RPC: Registered tcp NFSv4.1 backchannel transport module.
>> [14441.415816] RPC: Registered rdma transport module.
>> [14441.416689] RPC: Registered rdma backchannel transport module.
>> [14441.419979] rdma_rxe: set rxe1 active
>> [14441.420657] rdma_rxe: added rxe1 to eth1
>> [14470.422836] detected loopback device
>> [14484.278022] rdma_rxe: qp#18 moved to error state
>> 
>>> 
>>>> 
>>>> I’d like to know if rxe loopback is known to be broken, and how to fix it.
>>>> 
>>>> Best,
>>>> Bairen Yi
>> 
>> Bairen Yi
>> 
>> N�����r��y���b�X��ǧv�^�)޺{.n�+����{��ٚ�{ay�\x1dʇڙ�,j\a��f���h���z�\x1e�w���\f���j:+v���w�j�m����\a����zZ+�����ݢj"��!�i

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html