rsockets with RoCE

rsockets with RoCE

[Pradeep Satyanarayana]

Hi Sean,

I attempted to use rsockets with ConnectX-EN adapters and the client 
receives a "Connection refused" message. I debugged this a bit further 
and see that the client is actually receiving an IB_CM_REJ_RECEIVED with 
reason being 28 i.e. "Consumer Reject". Could this be because of the 
difference in how GIDs are obtained in the case of RoCE? Have you 
attempted rsockets on RoCE at all?

Incidentally this problem also highlights another issue we have 
encountered previously. The only way to debug issues like this in RDMA 
CM is to modify the kernel module. Currently there is no way we can turn 
on (and off) debug and get insights into the connection set up process. 
Are you open to considering patches to enhance debug with RDMA CM, 
particularly during the address resolution and connection setup process. 
That way it should not impact the performance during data transfer.

Thanks
Pradeep

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: rsockets with RoCE

[Hefty, Sean]

> I attempted to use rsockets with ConnectX-EN adapters and the client
> receives a "Connection refused" message. I debugged this a bit further
> and see that the client is actually receiving an IB_CM_REJ_RECEIVED with
> reason being 28 i.e. "Consumer Reject". Could this be because of the
> difference in how GIDs are obtained in the case of RoCE? Have you
> attempted rsockets on RoCE at all?

I have not tested this on RoCE.  Do other librdmacm examples (e.g. ucmatose or rdma_client/server) work in your configuration?

> Incidentally this problem also highlights another issue we have
> encountered previously. The only way to debug issues like this in RDMA
> CM is to modify the kernel module. Currently there is no way we can turn
> on (and off) debug and get insights into the connection set up process.
> Are you open to considering patches to enhance debug with RDMA CM,
> particularly during the address resolution and connection setup process.
> That way it should not impact the performance during data transfer.

No objection.  The rdma_cm shouldn't be considered speed path anyway.  Btw, the IB CM exports some counters which can sometimes be helpful in debugging, though, those only report a count of which messages have been sent/received.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rsockets with RoCE

[Pradeep Satyanarayana]

On 06/27/2012 04:54 PM, Hefty, Sean wrote:
>> I attempted to use rsockets with ConnectX-EN adapters and the client
>> receives a "Connection refused" message. I debugged this a bit further
>> and see that the client is actually receiving an IB_CM_REJ_RECEIVED with
>> reason being 28 i.e. "Consumer Reject". Could this be because of the
>> difference in how GIDs are obtained in the case of RoCE? Have you
>> attempted rsockets on RoCE at all?
> I have not tested this on RoCE.  Do other librdmacm examples (e.g. ucmatose or rdma_client/server) work in your configuration?

I have attempted running the utilities in the rsocket/examples/ 
directory like ucmatose and rping. ucamtose fails identically:
cmatose: event: RDMA_CM_EVENT_REJECTED, error: 28

and rping fails as follows:

rdma_resolve_route: No such device
waiting for addr/route resolution state 10

>
>> Incidentally this problem also highlights another issue we have
>> encountered previously. The only way to debug issues like this in RDMA
>> CM is to modify the kernel module. Currently there is no way we can turn
>> on (and off) debug and get insights into the connection set up process.
>> Are you open to considering patches to enhance debug with RDMA CM,
>> particularly during the address resolution and connection setup process.
>> That way it should not impact the performance during data transfer.
> No objection.  The rdma_cm shouldn't be considered speed path anyway.  Btw, the IB CM exports some counters which can sometimes be helpful in debugging, though, those only report a count of which messages have been sent/received.

I have not used this before. How does one read these counters?

Thanks
Pradeep

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: rsockets with RoCE

[Hefty, Sean]

> > No objection.  The rdma_cm shouldn't be considered speed path anyway.  Btw, the IB CM exports some
> counters which can sometimes be helpful in debugging, though, those only report a count of which
> messages have been sent/received.
> 
> I have not used this before. How does one read these counters?

Look in /sys/class/infiniband_cm, under the device/port subdirectories
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html