initiator depth and responder resources in a connect request.

initiator depth and responder resources in a connect request.

[Steve Wise]

Hey Sean,

Dumb question:  What exactly do the rdma_conn_param fields initiator_depth and responder_resources mean in an incoming 
connection request?  Does it mean the peer's values for these two resources, or is responder_resources set to match the 
peer's initiator_depth and initiator_depth set to match the peer's responder_resources?  IE Are they swapped so the app 
can turn around and use them in the rdma_accept()?

(hope that question makes sense)


Steve.



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: initiator depth and responder resources in a connect request.

[Hefty, Sean]

> Dumb question:  What exactly do the rdma_conn_param fields initiator_depth
> and responder_resources mean in an incoming
> connection request?  Does it mean the peer's values for these two
> resources, or is responder_resources set to match the
> peer's initiator_depth and initiator_depth set to match the peer's
> responder_resources?  IE Are they swapped so the app
> can turn around and use them in the rdma_accept()?

The values are swapped.  See the man page for rdma_get_cm_event and let me know if it's not clear.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: initiator depth and responder resources in a connect request.

[Steve Wise]

On 03/14/2011 10:28 AM, Hefty, Sean wrote:
>> Dumb question:  What exactly do the rdma_conn_param fields initiator_depth
>> and responder_resources mean in an incoming
>> connection request?  Does it mean the peer's values for these two
>> resources, or is responder_resources set to match the
>> peer's initiator_depth and initiator_depth set to match the peer's
>> responder_resources?  IE Are they swapped so the app
>> can turn around and use them in the rdma_accept()?
> The values are swapped.  See the man page for rdma_get_cm_event and let me know if it's not clear.

Thanks.  Yes.  That man page is clear.

I think the rdma_accept() page has a bug though:

----
        responder_resources
               The  maximum  number  of outstanding RDMA read and atomic opera-
               tions that the local side will  accept  from  the  remote  side.
               Applies  only  to  RDMA_PS_TCP.  This value must be less than or
               equal to the local RDMA device attribute max_qp_rd_atom and  the
               responder_resources value reported in the connect request event.
----

"and the responder_resources value" should be "and greater than or equal to the responder_resources value".


Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: initiator depth and responder resources in a connect request.

[Hefty, Sean]

> "and the responder_resources value" should be "and greater than or equal to
> the responder_resources value".

thanks - I added the following change to the man page:

-max_qp_rd_atom and the responder_resources value reported in the connect
-request event.
+max_qp_rd_atom, but preferably greater than or equal to the responder_resources
+value reported in the connect request event.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: initiator depth and responder resources in a connect request.

[Steve Wise]

On 03/14/2011 04:33 PM, Hefty, Sean wrote:
>> "and the responder_resources value" should be "and greater than or equal to
>> the responder_resources value".
> thanks - I added the following change to the man page:
>
> -max_qp_rd_atom and the responder_resources value reported in the connect
> -request event.
> +max_qp_rd_atom, but preferably greater than or equal to the responder_resources
> +value reported in the connect request event.
>

For iwarp, its a fatal connection termination if the responder resources (aka IRD) is overflowed by the peer.  Is it not 
fatal for an IB connection?




> - Sean
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: initiator depth and responder resources in a connect request.

[Hefty, Sean]

> > -max_qp_rd_atom and the responder_resources value reported in the connect
> > -request event.
> > +max_qp_rd_atom, but preferably greater than or equal to the
> responder_resources
> > +value reported in the connect request event.
> >
> 
> For iwarp, its a fatal connection termination if the responder resources
> (aka IRD) is overflowed by the peer.  Is it not
> fatal for an IB connection?

Yes - it's fatal.  With rdma_accept, the user _can_ specify a lower number of responder_resources than what the client side requested in their initiator_depth.  The client must then determine if the responder_resources that were offered is sufficient (at least with IB).  This is why I added 'preferably' to the documentation.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: initiator depth and responder resources in a connect request.

[Steve Wise]

On 03/14/2011 04:40 PM, Hefty, Sean wrote:
>>> -max_qp_rd_atom and the responder_resources value reported in the connect
>>> -request event.
>>> +max_qp_rd_atom, but preferably greater than or equal to the
>> responder_resources
>>> +value reported in the connect request event.
>>>
>> For iwarp, its a fatal connection termination if the responder resources
>> (aka IRD) is overflowed by the peer.  Is it not
>> fatal for an IB connection?
> Yes - it's fatal.  With rdma_accept, the user _can_ specify a lower number of responder_resources than what the client side requested in their initiator_depth.  The client must then determine if the responder_resources that were offered is sufficient (at least with IB).  This is why I added 'preferably' to the documentation.

When you say 'client must then determine', do you mean the client application?  So the IB CM really just passes along 
these values and doesn't really enforce any semantics?

The reason I'm asking is we're reviewing the IETF ID that extends MPA connection setup to allow negotiating ORD/IRD.  
And I think we should not diverge from the way its done in IB.  IE: the RDMACM semantics should be the same IMO.



> - Sean
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: initiator depth and responder resources in a connect request.

[Hefty, Sean]

> When you say 'client must then determine', do you mean the client
> application?  So the IB CM really just passes along
> these values and doesn't really enforce any semantics?

Yes and yes.  (The ib cm is probably a bit brain dead on this, but the librdmacm does check for this error.)  From the IB spec (sections intentionally reversed) see the two areas marked *****:

12.7.30 INITIATOR DEPTH
The maximum number of outstanding RDMA Read/Atomic operations the
sender will have to the remote QP/EEC. This value may be zero. The
maximum number that the HCA can support for a QP/EEC can be determined
using the Query HCA verb. See section 11.2.1.2 Query HCA.
InfiniBandTM Architecture Release 1.2.1 Communication Management November 2007

*****The recipient of the REQ message should try to choose a number of local
Responder Resources that is greater than or equal to the Initiator Depth
in the REQ message. If it is unwilling or unable to do so, it may send a
REP message containing fewer Responder Resources than the Initiator
Depth in the REQ message.*****

12.7.29 RESPONDER RESOURCES
The maximum number of outstanding RDMA Read/Atomic operations the
sender will support from the remote QP/EEC. This value may be zero. The
maximum number that the HCA can support for a QP/EEC can be determined
using the Query HCA verb. See section 11.2.1.2 Query HCA.

The recipient of the REQ message shall choose a local Initiator Depth that
does not exceed the Responder Resources offered in the REQ. If the recipient
of the REQ message is unwilling or unable to do so, it shall send a
REJ message to discontinue the connection establishment.

*****The recipient of the REP message shall decide whether the Responder
Resources offered in the REP are sufficient for the Initiator Depth the recipient
of the REP wishes to use.***** If not, it shall send a REJ message to
discontinue the connection establishment.


> The reason I'm asking is we're reviewing the IETF ID that extends MPA
> connection setup to allow negotiating ORD/IRD.
> And I think we should not diverge from the way its done in IB.  IE: the
> RDMACM semantics should be the same IMO.

What does iwarp do now?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: initiator depth and responder resources in a connect request.

[Steve Wise]

On 03/14/2011 05:12 PM, Hefty, Sean wrote:
>> When you say 'client must then determine', do you mean the client
>> application?  So the IB CM really just passes along
>> these values and doesn't really enforce any semantics?
> Yes and yes.  (The ib cm is probably a bit brain dead on this, but the librdmacm does check for this error.)  From the IB spec (sections intentionally reversed) see the two areas marked *****:
>
> 12.7.30 INITIATOR DEPTH
> The maximum number of outstanding RDMA Read/Atomic operations the
> sender will have to the remote QP/EEC. This value may be zero. The
> maximum number that the HCA can support for a QP/EEC can be determined
> using the Query HCA verb. See section 11.2.1.2 Query HCA.
> InfiniBandTM Architecture Release 1.2.1 Communication Management November 2007
>
> *****The recipient of the REQ message should try to choose a number of local
> Responder Resources that is greater than or equal to the Initiator Depth
> in the REQ message. If it is unwilling or unable to do so, it may send a
> REP message containing fewer Responder Resources than the Initiator
> Depth in the REQ message.*****
>
> 12.7.29 RESPONDER RESOURCES
> The maximum number of outstanding RDMA Read/Atomic operations the
> sender will support from the remote QP/EEC. This value may be zero. The
> maximum number that the HCA can support for a QP/EEC can be determined
> using the Query HCA verb. See section 11.2.1.2 Query HCA.
>
> The recipient of the REQ message shall choose a local Initiator Depth that
> does not exceed the Responder Resources offered in the REQ. If the recipient
> of the REQ message is unwilling or unable to do so, it shall send a
> REJ message to discontinue the connection establishment.
>
> *****The recipient of the REP message shall decide whether the Responder
> Resources offered in the REP are sufficient for the Initiator Depth the recipient
> of the REP wishes to use.***** If not, it shall send a REJ message to
> discontinue the connection establishment.
>
>
>> The reason I'm asking is we're reviewing the IETF ID that extends MPA
>> connection setup to allow negotiating ORD/IRD.
>> And I think we should not diverge from the way its done in IB.  IE: the
>> RDMACM semantics should be the same IMO.
> What does iwarp do now?

The current IETF specs leave it up to the applications/ULPs to deal with it.  Neither ORD or IRD are exchanged as part 
of MPA or other iWARP protocol negotiations.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: initiator depth and responder resources in a connect request.

[Jason Gunthorpe]

On Mon, Mar 14, 2011 at 04:57:52PM -0500, Steve Wise wrote:
> On 03/14/2011 04:40 PM, Hefty, Sean wrote:
> >>>-max_qp_rd_atom and the responder_resources value reported in the connect
> >>>-request event.
> >>>+max_qp_rd_atom, but preferably greater than or equal to the
> >>responder_resources
> >>>+value reported in the connect request event.
> >>>
> >>For iwarp, its a fatal connection termination if the responder resources
> >>(aka IRD) is overflowed by the peer.  Is it not
> >>fatal for an IB connection?
> >Yes - it's fatal.  With rdma_accept, the user _can_ specify a lower number of responder_resources than what the client side requested in their initiator_depth.  The client must then determine if the responder_resources that were offered is sufficient (at least with IB).  This is why I added 'preferably' to the documentation.
> 
> When you say 'client must then determine', do you mean the client
> application?  So the IB CM really just passes along these values and
> doesn't really enforce any semantics?
> 
> The reason I'm asking is we're reviewing the IETF ID that extends
> MPA connection setup to allow negotiating ORD/IRD.  And I think we
> should not diverge from the way its done in IB.  IE: the RDMACM
> semantics should be the same IMO.

Not sure how this is presented by RDMA CM, but the way it works on
the wire in IB is..
 - Client sends "client's max outgoing RR usage" and "client's max
   incoming RR available" [which is min(ULP request,HCA limit)]
 - Server computes:
    server's max outgoing RR usage = min(client's max
                              incoming RR available,ULP limit,HCA
			      limit)
    server's max incoming RR available = min(client's max outoing RR
                           usage,ULP limit,HCA limit)
   and sends it back.
 - Client should sanity check:
    server's max outgoing RR usage <= client's max incoming RR available
    server's max incoming RR available >= client's max outoing RR usage
   And the ULP should confirm it can function within the RR limit.
 - The four values are encoded into two pairs in the CM messages, the
   client pair is in the REQ and the server pair is in the REP.
   Both sides end up learning all four values, but only the server's
   values are used.

Basically, the equations:
 server max outgoing RR usage = client max incoming RR available =
     min(server ULP, client ULP, server HCA, client HCA);
 client max outgoing RR usage = server max incoming RR available =
     min(server ULP, client ULP, server HCA, client HCA);
are being computed. Each side's ULP should have the opportunity to
specify the maximum number of RRs it will consume at once when posting
send WR's, to avoid consuming global RRs in the HCA.

FWIW, I'm not sure this actually works in Linux, last time I looked
it only worked if all values were the same, which they generally are
since nearly everything seems wired to 4.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html