udaddy.c question

udaddy.c question

[Steve Wise]

Hello rdma wizards,

I'm trying to understand how IB UD service works with the rdmacm in 
linux.  According to the IBTA specification, I should be able to create 
a single UD QP and exchange messages with any number of peers as long as 
I know their qkeys.  Yet when I look at udaddy.c (and even mckey.c), it 
appears that the UD service in the rdmacm is really connection oriented 
and a QP is required for each pair of processes exchanging data.  I 
think this is just the way udaddy.c was implemented, but I want to know 
if it is possible with librdmacm to create a single cm_id with a single 
QP, bind it to an IP address/port, and then send/recv to multiple peers 
(via unicast).  Is this possible?

Thanks!

Steve.


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: udaddy.c question

[Sean Hefty]

>I'm trying to understand how IB UD service works with the rdmacm in
>linux.  According to the IBTA specification, I should be able to create
>a single UD QP and exchange messages with any number of peers as long as
>I know their qkeys.  Yet when I look at udaddy.c (and even mckey.c), it
>appears that the UD service in the rdmacm is really connection oriented
>and a QP is required for each pair of processes exchanging data.  I
>think this is just the way udaddy.c was implemented, but I want to know
>if it is possible with librdmacm to create a single cm_id with a single
>QP, bind it to an IP address/port, and then send/recv to multiple peers
>(via unicast).  Is this possible?

It's intended to be possible.  :)  The rdma_cm uses a single qkey be default in
order to handle mixed UD and multicast traffic.  You could override this by
exchanging whatever qkey you want via private data.  But you must discover the
remote qpn.

To discover the remote qpn, you use the normal rdma calls (resolve_addr /
resolve_route / connect or listen / accept).  You need to do this for each
remote QP that you want to exchange data with.  You don't need a QP associated
with the rdma_cm_id's when resolving the remote qpn, or even need to keep the
rdma_cm_id's around afterwards.  Actually, I don't think there's any reason why
you can't create the UD QPs using ibv_create_qp, and only use the rdma cm to
discover the QPNs. 

The udaddy code was derived from ucmatose, which is why it looks a lot like
connection oriented code.

- Sean

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: udaddy.c question

[Steve Wise]

Sean Hefty wrote:
>> I'm trying to understand how IB UD service works with the rdmacm in
>> linux.  According to the IBTA specification, I should be able to create
>> a single UD QP and exchange messages with any number of peers as long as
>> I know their qkeys.  Yet when I look at udaddy.c (and even mckey.c), it
>> appears that the UD service in the rdmacm is really connection oriented
>> and a QP is required for each pair of processes exchanging data.  I
>> think this is just the way udaddy.c was implemented, but I want to know
>> if it is possible with librdmacm to create a single cm_id with a single
>> QP, bind it to an IP address/port, and then send/recv to multiple peers
>> (via unicast).  Is this possible?
>>     
>
> It's intended to be possible.  :)  The rdma_cm uses a single qkey be default in
> order to handle mixed UD and multicast traffic.  You could override this by
> exchanging whatever qkey you want via private data.  But you must discover the
> remote qpn.
>
> To discover the remote qpn, you use the normal rdma calls (resolve_addr /
> resolve_route / connect or listen / accept).  You need to do this for each
> remote QP that you want to exchange data with.  You don't need a QP associated
> with the rdma_cm_id's when resolving the remote qpn, or even need to keep the
> rdma_cm_id's around afterwards.  Actually, I don't think there's any reason why
> you can't create the UD QPs using ibv_create_qp, and only use the rdma cm to
> discover the QPNs. 
>
> The udaddy code was derived from ucmatose, which is why it looks a lot like
> connection oriented code.
>
> - Sean
>   

What if I don't want to use the qpn discovery mechanism from librdmacm?  
IE I have have an out of band protocol for exchanging qkeys -and- qpns.  
Can I avoid the connect on the active side, and the listen/accept on the 
server side?




--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: udaddy.c question

[Sean Hefty]

>What if I don't want to use the qpn discovery mechanism from librdmacm?
>IE I have have an out of band protocol for exchanging qkeys -and- qpns.
>Can I avoid the connect on the active side, and the listen/accept on the
>server side?

If you exchange the qpn and qkeys out of band, then I'm not sure what the rdma
cm buys you. 

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html