Converting from RC to UC

Converting from RC to UC

[Alan Cook]

I currently have a working RDMA InfiniBand based client-server application using
Reliable Connection (RC) service.  I would like to change the application to use
Unreliable Connection (UC) service.  I am not performing RDMA reads, so I should
be able to use UC.

I was under the impression that the only change that would be required would be
changing the queue pair type from IBV_QPT_RC to IBV_QPT_UC.  After making this
change on both the client and server applications, I receive an Invalid
Parameter error from rdma_accept() on the server side of the application.  The
client side reports no errors (other than the rejected server connection).

Internet searches have not been fruitful, as it seems most people are using RC
rather than UC.

What else needs to be changed in the setup for switching from RC to UC?

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Converting from RC to UC

[Jason Gunthorpe]

On Fri, Mar 11, 2011 at 06:29:19PM +0000, Alan Cook wrote:
> I currently have a working RDMA InfiniBand based client-server application using
> Reliable Connection (RC) service.  I would like to change the application to use
> Unreliable Connection (UC) service.  I am not performing RDMA reads, so I should
> be able to use UC.
> 
> I was under the impression that the only change that would be required would be
> changing the queue pair type from IBV_QPT_RC to IBV_QPT_UC.  After making this
> change on both the client and server applications, I receive an Invalid
> Parameter error from rdma_accept() on the server side of the application.  The
> client side reports no errors (other than the rejected server connection).
> 
> Internet searches have not been fruitful, as it seems most people are using RC
> rather than UC.
> 
> What else needs to be changed in the setup for switching from RC to UC?

I think what you are hitting is some code is calling ibv_modify_qp
with an invalid qp_attr_mask for one of the state transitions. UC and
RC have different requirements. Assuming you are not calling
ibv_modify_qp in your app it is probably a kernel bug?

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Converting from RC to UC

[Alan Cook]

Jason Gunthorpe <jgunthorpe@...> writes:
> I think what you are hitting is some code is calling ibv_modify_qp
> with an invalid qp_attr_mask for one of the state transitions. UC and
> RC have different requirements. Assuming you are not calling
> ibv_modify_qp in your app it is probably a kernel bug?
> 

No calls are made to ibv_modify_qp.  The application is failing on the connect,
so there has not been any application specific operations performed over
InfiniBand yet (other than the connection setup).  The application itself is
quite basic: eight application specific messages to be passed over InfiniBand,
along with RDMA write messages.  It can be thought of as a glorified scp 
operation.

I am using the vanilla 2.6.27.57 kernel.  If the problem is a kernel bug, I
would expect the kernel to not support UC at all, given the simplicity of the
application.

The only change made to the application for switching from RC to UC was to
change the queue type, which does not appear to work.  It seems that there are
other setup operations performed for the RC server which are not compatible with
a UC server, but no calls return errors until rdma_accept().  What other
settings/setup would be causing the issue that would not return an error?

I have found no examples for setting up a basic IB application using UC.  All
examples are with RC.  Does anyone know of any examples for UC that they could
point me towards?



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Converting from RC to UC

[Roland Dreier]

On Fri, Mar 11, 2011 at 10:29 AM, Alan Cook
<acook-osxm6dDZNBBZx8iatJs59jGjJy/sRE9J@public.gmane.org> wrote:
> I currently have a working RDMA InfiniBand based client-server application using
> Reliable Connection (RC) service.  I would like to change the application to use
> Unreliable Connection (UC) service.  I am not performing RDMA reads, so I should
> be able to use UC.
>
> I was under the impression that the only change that would be required would be
> changing the queue pair type from IBV_QPT_RC to IBV_QPT_UC.

How are you setting up the connection?

My understanding is that the RDMA CM (ie librdmacm) does not handle UC,
so if you are using that, it shouldn't work.  Sean could confirm this.

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Converting from RC to UC

[Jason Gunthorpe]

On Fri, Mar 11, 2011 at 02:52:24PM -0500, Alan Cook wrote:
> I am using the vanilla 2.6.27.57 kernel. If the problem is a kernel
> bug, I would expect the kernel to not support UC at all, given the
> simplicity of the application.

That is quite likely, actually. You are using the RDMA CM and I have
never heard of anyone using UC that way.

Raw verbs UC definitely works.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Converting from RC to UC

[Alan Cook]

Roland Dreier <roland@...> writes:
> 
> How are you setting up the connection?
> 
> My understanding is that the RDMA CM (ie librdmacm) does not handle UC,
> so if you are using that, it shouldn't work.  Sean could confirm this.
> 
>  - R.

I am using RDMA CM.  I have not seen any documentation that states it does not
support UC, although I have not seen any documentation that says it does either.
 I have seen documentation regarding RDMA CM that talks about a connection not
actually being established between queue pairs when using unreliable datagram
(UD), but nothing regarding support for UC.

If it is the case that RDMA CM is not supported, could someone point me at a
resource that describes setting up an IB RDMA connection not using RDMA CM?

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Converting from RC to UC

[Alan Cook]

Jason Gunthorpe <jgunthorpe@...> writes:
> 
> That is quite likely, actually. You are using the RDMA CM and I have
> never heard of anyone using UC that way.
> 
> Raw verbs UC definitely works.
> 
> Jason

I will look into switching to using raw verbs.  Thanks everyone for the
assistance.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Converting from RC to UC

[Jason Gunthorpe]

On Fri, Mar 11, 2011 at 09:55:00PM +0000, Alan Cook wrote:
> Roland Dreier <roland@...> writes:
> > 
> > How are you setting up the connection?
> > 
> > My understanding is that the RDMA CM (ie librdmacm) does not handle UC,
> > so if you are using that, it shouldn't work.  Sean could confirm this.
> > 
> >  - R.
> 
> I am using RDMA CM.  I have not seen any documentation that states it does not
> support UC, although I have not seen any documentation that says it does either.
>  I have seen documentation regarding RDMA CM that talks about a connection not
> actually being established between queue pairs when using unreliable datagram
> (UD), but nothing regarding support for UC.
> 
> If it is the case that RDMA CM is not supported, could someone point me at a
> resource that describes setting up an IB RDMA connection not using RDMA CM?

If you can tolerate it you might be better off just fixing the kernel,
the change is probably only one or two additional if statements to
support RC vs UC. Unfortunately I looked around for a mins for you but
I did not see an obvious error. Though perhaps the RTS transition in
the IB CM is at fault?

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Converting from RC to UC

[Or Gerlitz]

  Alan Cook wrote:
> Roland Dreier<roland@...>  writes:
>> How are you setting up the connection? My understanding is that the RDMA CM (ie librdmacm) does not handle UC, so if you are using that, it shouldn't work.  Sean could confirm this.
>
> I am using RDMA CM.  I have not seen any documentation that states it does not support UC, although I have not seen any documentation that says it does either.   I have seen documentation regarding RDMA CM that talks about a connection not  actually being established between queue pairs when using unreliable datagram (UD), but nothing regarding support for UC.

Alan, you haven't specified if you app is user space or kernel space 
based,  I assume the former, I believe that it either just works or can 
be just made to work... e.g maybe a minor error on your side, or maybe 
something small to change/complete on the rdma-cm side...

I'm copying the maintainer of the kernel rdma-cm and librdmacm, Sean - 
should there any problem for librdmacm consumers to use UC at their apps?

Or.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Converting from RC to UC

[Alan Cook]

Or Gerlitz <ogerlitz@...> writes:

> 
> Alan, you haven't specified if you app is user space or kernel space 
> based,  I assume the former, I believe that it either just works or can 
> be just made to work... e.g maybe a minor error on your side, or maybe 
> something small to change/complete on the rdma-cm side...
> 
> I'm copying the maintainer of the kernel rdma-cm and librdmacm, Sean - 
> should there any problem for librdmacm consumers to use UC at their apps?
> 
> Or.
> 

My application is running in user space, but could run in kernel space if need
be.  I would assume that the error is on my side, but I am unable to find any
documentation on setting up a UC service as opposed to an RC service.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Converting from RC to UC

[frank zago]

On 03/13/2011 08:08 PM, Alan Cook wrote:
> Or Gerlitz <ogerlitz@...> writes:
> 
>>
>> Alan, you haven't specified if you app is user space or kernel space 
>> based,  I assume the former, I believe that it either just works or can 
>> be just made to work... e.g maybe a minor error on your side, or maybe 
>> something small to change/complete on the rdma-cm side...
>>
>> I'm copying the maintainer of the kernel rdma-cm and librdmacm, Sean - 
>> should there any problem for librdmacm consumers to use UC at their apps?
>>
>> Or.
>>
> 
> My application is running in user space, but could run in kernel space if need
> be.  I would assume that the error is on my side, but I am unable to find any
> documentation on setting up a UC service as opposed to an RC service.
> 

You can look at perftest. A few tests support UC (but none with rdma cm support).

Frank.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Converting from RC to UC

[Hefty, Sean]

> My understanding is that the RDMA CM (ie librdmacm) does not handle UC,
> so if you are using that, it shouldn't work.  Sean could confirm this.

This is correct for the current version of the code.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html