RoCE passive side failures on 4.4-rc5

RoCE passive side failures on 4.4-rc5

[Or Gerlitz]

Guys,

Using 4.4-rc5+ [1] and **not** applying any of the patches I sent today,
I noted that RoCE passive side isn't working (rdma-cm, ibv_rc_pingpong 
works).

I have two nodes in ConnectX3 VPI config (port1 IB and port2 Eth), the 
one with the 4.4-rc5 kernel can act as both (rping) client/server for IB 
links but only (rping) client for RoCE.

I tried both inter-node and loopback runs, in all cases, the client side 
getsCM
reject with reason 28, see [2], tried both iser and rping. Eth (ICMP, 
TCP) works OK.

Or.

[1] Linus tree up to commit a5e90b1 "Merge branch 'fixes' of 
git://ftp.arm.linux.org.uk/~rmk/linux-arm"

[2] $ rping -d -v -c -a 192.168.31.17 -C 1
verbose
client
count 1
created cm_id 0x60cd90
cma_event type RDMA_CM_EVENT_ADDR_RESOLVED cma_id 0x60cd90 (parent)
cma_event type RDMA_CM_EVENT_ROUTE_RESOLVED cma_id 0x60cd90 (parent)
rdma_resolve_addr - rdma_resolve_route successful
created pd 0x60cfe0
created channel 0x608510
created cq 0x607f50
created qp 0x60d000
rping_setup_buffers called on cb 0x606010
allocated & registered buffers...
cq_thread started.
wait for CONNECTED state 10
cma_event type RDMA_CM_EVENT_REJECTED cma_id 0x60cd90 (parent)
connect error -1
rping_free_buffers called on cb 0x606010
cma event RDMA_CM_EVENT_REJECTED, error 28

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RoCE passive side failures on 4.4-rc5

[Sagi Grimberg]

> Guys,
>
> Using 4.4-rc5+ [1] and **not** applying any of the patches I sent today,
> I noted that RoCE passive side isn't working (rdma-cm, ibv_rc_pingpong
> works).

I'm using 4.4-rc2+ and I have RoCE working.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RoCE passive side failures on 4.4-rc5

[Or Gerlitz]

On 12/17/2015 4:18 PM, Sagi Grimberg wrote:
> I'm using 4.4-rc2+ and I have RoCE working. 

I tried 4.4-rc2 and I see the same problem. I will make my .config
available to you and Moni so you can try it out on your systems Sunday.

I see this over both CX2 and CX3-pro, both are in VPI config.

BTW getting the 28 (consumer defined) reject reason is something I saw 
coming
and going from time to time in various weird cases over the past years.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RoCE passive side failures on 4.4-rc5

[Or Gerlitz]

On 12/17/2015 3:58 PM, Or Gerlitz wrote:
> Using 4.4-rc5+ [1] and **not** applying any of the patches I sent today,
> I noted that RoCE passive side isn't working (rdma-cm, ibv_rc_pingpong 
> works).
>
> I have two nodes in ConnectX3 VPI config (port1 IB and port2 Eth), the 
> one with the 4.4-rc5 kernel can act as both (rping) client/server for 
> IB links but only (rping) client for RoCE.
>
> I tried both inter-node and loopback runs, in all cases, the client 
> side getsCM
> reject with reason 28, see [2], tried both iser and rping. Eth (ICMP, 
> TCP) works OK.

OK, small progress, when the force Eth link type on my IB port (using 
mlx4 sysfs), things work.

You should be able to reproduce it on your non-VPI systems the other way 
around, by
forcing IB link type on one of the Eth ports and see the failure.

I Saw the same behavior with both 4.4-rc2 and 4.4-rc5

Or.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RoCE passive side failures on 4.4-rc5

[Matan Barak]

On Sun, Dec 20, 2015 at 9:29 AM, Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> wrote:
> On 12/17/2015 3:58 PM, Or Gerlitz wrote:
>>
>> Using 4.4-rc5+ [1] and **not** applying any of the patches I sent today,
>> I noted that RoCE passive side isn't working (rdma-cm, ibv_rc_pingpong
>> works).
>>
>> I have two nodes in ConnectX3 VPI config (port1 IB and port2 Eth), the one
>> with the 4.4-rc5 kernel can act as both (rping) client/server for IB links
>> but only (rping) client for RoCE.
>>
>> I tried both inter-node and loopback runs, in all cases, the client side
>> getsCM
>> reject with reason 28, see [2], tried both iser and rping. Eth (ICMP, TCP)
>> works OK.
>
>
> OK, small progress, when the force Eth link type on my IB port (using mlx4
> sysfs), things work.
>
> You should be able to reproduce it on your non-VPI systems the other way
> around, by
> forcing IB link type on one of the Eth ports and see the failure.
>
> I Saw the same behavior with both 4.4-rc2 and 4.4-rc5
>
> Or.

I've posted a patch that fixes that, please take a look at [1].

Regards,
Matan

[1] https://www.mail-archive.com/linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org/msg30777.html

>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RoCE passive side failures on 4.4-rc5

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 1511 bytes --]

On 12/21/2015 10:08 AM, Matan Barak wrote:
> On Sun, Dec 20, 2015 at 9:29 AM, Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> wrote:
>> On 12/17/2015 3:58 PM, Or Gerlitz wrote:
>>>
>>> Using 4.4-rc5+ [1] and **not** applying any of the patches I sent today,
>>> I noted that RoCE passive side isn't working (rdma-cm, ibv_rc_pingpong
>>> works).
>>>
>>> I have two nodes in ConnectX3 VPI config (port1 IB and port2 Eth), the one
>>> with the 4.4-rc5 kernel can act as both (rping) client/server for IB links
>>> but only (rping) client for RoCE.
>>>
>>> I tried both inter-node and loopback runs, in all cases, the client side
>>> getsCM
>>> reject with reason 28, see [2], tried both iser and rping. Eth (ICMP, TCP)
>>> works OK.
>>
>>
>> OK, small progress, when the force Eth link type on my IB port (using mlx4
>> sysfs), things work.
>>
>> You should be able to reproduce it on your non-VPI systems the other way
>> around, by
>> forcing IB link type on one of the Eth ports and see the failure.
>>
>> I Saw the same behavior with both 4.4-rc2 and 4.4-rc5
>>
>> Or.
> 
> I've posted a patch that fixes that, please take a look at [1].
> 
> Regards,
> Matan
> 
> [1] https://www.mail-archive.com/linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org/msg30777.html

I've been seeing this too in my 4.4-rc testing, so I'll have test
results today.


-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
              GPG KeyID: 0E572FDD



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]