Problem running rping over Intel adapters

Problem running rping over Intel adapters

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 2096 bytes --]

We ran into this problem when testing rping over Intel/QLogic hardware:

[root@rdmaperf3 ~]# rping -s -a 172.31.2.103 -v
wait for CONNECTED state 10
connect error -1
cma event RDMA_CM_EVENT_REJECTED, error 28
[root@rdmaperf3 ~]#


[root@rdmaperf8 ~]# rping -c -a 172.31.2.103 -v -C 5
cma event RDMA_CM_EVENT_CONNECT_ERROR, error -1
wait for CONNECTED state 4
connect error -1
[root@rdmaperf8 ~]#

Turns out this is because of a couple things:

1) rping, on the client side, clears the conn_params for the newly to be
attempted connection, then sets:

	conn_param.responder_resources = 1;
	conn_param.initiator_depth = 1;
	conn_param.retry_count = 10;

On the accept side, rping clears the conn_params and then sets just the
responder_resources and initiator_depth, without even checking the
incoming requested conn_param values from the incoming cm_id.  So, OK,
you can get away with that since this is a simple test program, but
still not "best programming practices".  However, the important part
here is the retry_count of 10.  That won't work on Intel/QLogic hardware.

2) the qib driver enforces a maximum of 7 for retry_count.  I don't see
anything in the spec that specifies a maximum for this entry, and in
particular I know it doesn't call out for 7 to mean infinite retries
like it does for rnr_retry_count.

I don't think the spec really cares how we solve this, and I don't think
there is a hard limit of 7 for the retry_count like the qib driver
enforces.  On the other hand, the spec doesn't call out a limit on the
retry_count but I would assume each driver has the option to implement
their own "reasonable, implementation defined" limit in a case like this.

So, do we make qib more liberal in its acceptance of retry_count or do
we fix rping to use a smaller number?  Matters not to me...

-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
              GPG KeyID: 0E572FDD
	      http://people.redhat.com/dledford

Infiniband specific RPMs available at
	      http://people.redhat.com/dledford/Infiniband


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 895 bytes --]

RE: Problem running rping over Intel adapters

[Hefty, Sean]

> 1) rping, on the client side, clears the conn_params for the newly to be
> attempted connection, then sets:
> 
> 	conn_param.responder_resources = 1;
> 	conn_param.initiator_depth = 1;
> 	conn_param.retry_count = 10;


> 2) the qib driver enforces a maximum of 7 for retry_count.  I don't see
> anything in the spec that specifies a maximum for this entry, and in
> particular I know it doesn't call out for 7 to mean infinite retries
> like it does for rnr_retry_count.

On IB, retry count is a 3-bit value.  See the CM REQ message.

The kernel rdma cm should probably check for a valid value and reduce it accordingly.

- Sean

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem running rping over Intel adapters

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 1050 bytes --]

On 10/3/2012 5:50 PM, Hefty, Sean wrote:
>> 1) rping, on the client side, clears the conn_params for the newly to be
>> attempted connection, then sets:
>>
>> 	conn_param.responder_resources = 1;
>> 	conn_param.initiator_depth = 1;
>> 	conn_param.retry_count = 10;
> 
> 
>> 2) the qib driver enforces a maximum of 7 for retry_count.  I don't see
>> anything in the spec that specifies a maximum for this entry, and in
>> particular I know it doesn't call out for 7 to mean infinite retries
>> like it does for rnr_retry_count.
> 
> On IB, retry count is a 3-bit value.  See the CM REQ message.
> 
> The kernel rdma cm should probably check for a valid value and reduce it accordingly.

Then it's also fair to say that right now rping passes a known invalid
value and needs fixed ;-)


-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
              GPG KeyID: 0E572FDD
	      http://people.redhat.com/dledford

Infiniband specific RPMs available at
	      http://people.redhat.com/dledford/Infiniband


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 895 bytes --]

RE: Problem running rping over Intel adapters

[Hefty, Sean]

> >> 1) rping, on the client side, clears the conn_params for the newly to be
> >> attempted connection, then sets:
> >>
> >> 	conn_param.responder_resources = 1;
> >> 	conn_param.initiator_depth = 1;
> >> 	conn_param.retry_count = 10;
> >
> >
> >> 2) the qib driver enforces a maximum of 7 for retry_count.  I don't see
> >> anything in the spec that specifies a maximum for this entry, and in
> >> particular I know it doesn't call out for 7 to mean infinite retries
> >> like it does for rnr_retry_count.
> >
> > On IB, retry count is a 3-bit value.  See the CM REQ message.
> >
> > The kernel rdma cm should probably check for a valid value and reduce it
> accordingly.
> 
> Then it's also fair to say that right now rping passes a known invalid
> value and needs fixed ;-)

For IB, but I wasn't sure about iWarp.  It doesn't look like it's used there though.

If not, I'll apply a patch to rping, plus submit a fix for the kernel to validate the value.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem running rping over Intel adapters

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 1427 bytes --]

On 10/3/2012 6:02 PM, Hefty, Sean wrote:
>>>> 1) rping, on the client side, clears the conn_params for the 
>>>> newly to be attempted connection, then sets:
>>>> 
>>>> conn_param.responder_resources = 1; conn_param.initiator_depth 
>>>> = 1; conn_param.retry_count = 10;
>>> 
>>> 
>>>> 2) the qib driver enforces a maximum of 7 for retry_count.  I 
>>>> don't see anything in the spec that specifies a maximum for 
>>>> this entry, and in particular I know it doesn't call out for 7 
>>>> to mean infinite retries like it does for rnr_retry_count.
>>> 
>>> On IB, retry count is a 3-bit value.  See the CM REQ message.
>>> 
>>> The kernel rdma cm should probably check for a valid value and 
>>> reduce it
>> accordingly.
>> 
>> Then it's also fair to say that right now rping passes a known 
>> invalid value and needs fixed ;-)
> 
> For IB, but I wasn't sure about iWarp.  It doesn't look like it's 
> used there though.

I would hope it's not used on iWARP....it's running over it's
own link layer where this is neither defined nor applicable.

> If not, I'll apply a patch to rping, plus submit a fix for the
> kernel to validate the value.




-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
              GPG KeyID: 0E572FDD
	      http://people.redhat.com/dledford

Infiniband specific RPMs available at
	      http://people.redhat.com/dledford/Infiniband


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 895 bytes --]

Re: Problem running rping over Intel adapters

[Steve Wise]

On 10/3/2012 5:02 PM, Hefty, Sean wrote:
>>>> 1) rping, on the client side, clears the conn_params for the newly to be
>>>> attempted connection, then sets:
>>>>
>>>> 	conn_param.responder_resources = 1;
>>>> 	conn_param.initiator_depth = 1;
>>>> 	conn_param.retry_count = 10;
>>>
>>>> 2) the qib driver enforces a maximum of 7 for retry_count.  I don't see
>>>> anything in the spec that specifies a maximum for this entry, and in
>>>> particular I know it doesn't call out for 7 to mean infinite retries
>>>> like it does for rnr_retry_count.
>>> On IB, retry count is a 3-bit value.  See the CM REQ message.
>>>
>>> The kernel rdma cm should probably check for a valid value and reduce it
>> accordingly.
>>
>> Then it's also fair to say that right now rping passes a known invalid
>> value and needs fixed ;-)
> For IB, but I wasn't sure about iWarp.  It doesn't look like it's used there though.
>
> If not, I'll apply a patch to rping, plus submit a fix for the kernel to validate the value.
> --

Not used by iwarp drivers...

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem running rping over Intel adapters

[Or Gerlitz]

On 04/10/2012 03:47, Steve Wise wrote:
>
> Not used by iwarp drivers...

Which one, the retry counter or the RNR retry counter?

Or.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem running rping over Intel adapters

[Steve Wise]

On 10/4/2012 12:41 AM, Or Gerlitz wrote:
> On 04/10/2012 03:47, Steve Wise wrote:
>>
>> Not used by iwarp drivers...
>
> Which one, the retry counter or the RNR retry counter?
>

rnr_retry_count has no meaning in iWARP.  There is no RNR message in the 
iwarp protocol.  For some iwarp devices (cxgb3 for example), an ingress 
send with no recv buffer available results in immediate connection 
termination.   Some devices, (eg cxgb4), the ingress SEND is buffered in 
the adapter until a recv buf is posted.  Both behaviors are allowed in 
the iwarp specs.

retry_count, I assume, is for connection establishment?  For the Chelsio 
devices, this is all up to the TCP stack and thus this counter is not 
used.  If there is no TCP listener, then the connection establishment 
will fail immediately.  If the node is not reachable, then normal TCP 
retransmit rules apply, but once TCP gives up, then the rdma connection 
establishment will fail.

Also, the RDMA_CM and IW_CM don't even pass these two counters down to 
the drivers' connect function.  See struct iw_cm_conn_param and 
cma.c/cma_connect_iw() iwcm.c/iw_cm_connect().


Steve.

> Or.
>
> -- 
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Problem running rping over Intel adapters

[Hefty, Sean]

> retry_count, I assume, is for connection establishment?  For the Chelsio
> devices, this is all up to the TCP stack and thus this counter is not
> used.  If there is no TCP listener, then the connection establishment
> will fail immediately.  If the node is not reachable, then normal TCP
> retransmit rules apply, but once TCP gives up, then the rdma connection
> establishment will fail.

On IB, this is a transport level retry of a lost request.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem running rping over Intel adapters

[Steve Wise]

On 10/4/2012 10:01 AM, Hefty, Sean wrote:
>> retry_count, I assume, is for connection establishment?  For the Chelsio
>> devices, this is all up to the TCP stack and thus this counter is not
>> used.  If there is no TCP listener, then the connection establishment
>> will fail immediately.  If the node is not reachable, then normal TCP
>> retransmit rules apply, but once TCP gives up, then the rdma connection
>> establishment will fail.
> On IB, this is a transport level retry of a lost request.
Ah, ok.  Regardless, TCP handles all that for IWARP and we currently 
don't provide the rdma application any way to manipulate the TCP behavior.

Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Problem running rping over Intel adapters

[Hefty, Sean]

> 1) rping, on the client side, clears the conn_params for the newly to be
> attempted connection, then sets:
> 
> 	conn_param.responder_resources = 1;
> 	conn_param.initiator_depth = 1;
> 	conn_param.retry_count = 10;
> 
> On the accept side, rping clears the conn_params and then sets just the
> responder_resources and initiator_depth, without even checking the
> incoming requested conn_param values from the incoming cm_id.  So, OK,
> you can get away with that since this is a simple test program, but
> still not "best programming practices".  However, the important part
> here is the retry_count of 10.  That won't work on Intel/QLogic hardware.

I pushed in the following fix to rping.  Thanks

---

rping: Reduce retry_count to fit in 3-bits

From: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

retry_count is a 3 bit value on IB, reduce it from
10 to 7.

A value of 10 prevents rping from working over the Intel
IB HCA.  Problem reported by Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>

The retry_count is also not set when calling rdma_accept.
Rather than passing different values into rdma_accept than
what was specified by the remote side, use the values given
in the connection request.

Signed-off-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 examples/rping.c |    9 ++-------
 1 files changed, 2 insertions(+), 7 deletions(-)

diff --git a/examples/rping.c b/examples/rping.c
index 785338e..32bd70a 100644
--- a/examples/rping.c
+++ b/examples/rping.c
@@ -342,16 +342,11 @@ error:
 
 static int rping_accept(struct rping_cb *cb)
 {
-	struct rdma_conn_param conn_param;
 	int ret;
 
 	DEBUG_LOG("accepting client connection request\n");
 
-	memset(&conn_param, 0, sizeof conn_param);
-	conn_param.responder_resources = 1;
-	conn_param.initiator_depth = 1;
-
-	ret = rdma_accept(cb->child_cm_id, &conn_param);
+	ret = rdma_accept(cb->child_cm_id, NULL);
 	if (ret) {
 		perror("rdma_accept");
 		return ret;
@@ -975,7 +970,7 @@ static int rping_connect_client(struct rping_cb *cb)
 	memset(&conn_param, 0, sizeof conn_param);
 	conn_param.responder_resources = 1;
 	conn_param.initiator_depth = 1;
-	conn_param.retry_count = 10;
+	conn_param.retry_count = 7;
 
 	ret = rdma_connect(cb->cm_id, &conn_param);
 	if (ret) {


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Problem running rping over Intel adapters

[Marciniszyn, Mike]

The IBTA spec clearly binds the IB HCA to a max of 7:

C9-142: For an HCA requester using Reliable Connection service, to prevent
the requester from retrying the request forever, the requester shall
maintain a 3 bit retry counter which is used to count the number of times
a particular request packet has been retried and timed out. This counter
shall be decremented each time the transport timer expires for a given request
packet. The counter shall be re-loaded whenever a given outstanding
request is cleared.

The language round modify QP probably validates a test on the Retry count:

C11-9: If any of the QP attributes to be modified are invalid or the requested
state transition is invalid, none of the QP attributes shall be modified.
An immediate error shall be returned and the QP state shall remain
unchanged.

A brief look at Mellanox code indicates that the "invalid" value is trusted and or'ed into the modify mechanism.

Mike

> -----Original Message-----
> From: Hefty, Sean
> Sent: Wednesday, October 03, 2012 6:39 PM
> To: Doug Ledford; Marciniszyn, Mike; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Subject: RE: Problem running rping over Intel adapters
> 
> > 1) rping, on the client side, clears the conn_params for the newly to
> > be attempted connection, then sets:
> >
> > 	conn_param.responder_resources = 1;
> > 	conn_param.initiator_depth = 1;
> > 	conn_param.retry_count = 10;
> >
> > On the accept side, rping clears the conn_params and then sets just
> > the responder_resources and initiator_depth, without even checking the
> > incoming requested conn_param values from the incoming cm_id.  So, OK,
> > you can get away with that since this is a simple test program, but
> > still not "best programming practices".  However, the important part
> > here is the retry_count of 10.  That won't work on Intel/QLogic hardware.
> 
> I pushed in the following fix to rping.  Thanks
> 
> ---
> 
> rping: Reduce retry_count to fit in 3-bits
> 
> From: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> 
> retry_count is a 3 bit value on IB, reduce it from
> 10 to 7.
> 
> A value of 10 prevents rping from working over the Intel IB HCA.  Problem
> reported by Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> 
> The retry_count is also not set when calling rdma_accept.
> Rather than passing different values into rdma_accept than what was specified
> by the remote side, use the values given in the connection request.
> 
> Signed-off-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> ---
>  examples/rping.c |    9 ++-------
>  1 files changed, 2 insertions(+), 7 deletions(-)
> 
> diff --git a/examples/rping.c b/examples/rping.c index 785338e..32bd70a
> 100644
> --- a/examples/rping.c
> +++ b/examples/rping.c
> @@ -342,16 +342,11 @@ error:
> 
>  static int rping_accept(struct rping_cb *cb)  {
> -	struct rdma_conn_param conn_param;
>  	int ret;
> 
>  	DEBUG_LOG("accepting client connection request\n");
> 
> -	memset(&conn_param, 0, sizeof conn_param);
> -	conn_param.responder_resources = 1;
> -	conn_param.initiator_depth = 1;
> -
> -	ret = rdma_accept(cb->child_cm_id, &conn_param);
> +	ret = rdma_accept(cb->child_cm_id, NULL);
>  	if (ret) {
>  		perror("rdma_accept");
>  		return ret;
> @@ -975,7 +970,7 @@ static int rping_connect_client(struct rping_cb *cb)
>  	memset(&conn_param, 0, sizeof conn_param);
>  	conn_param.responder_resources = 1;
>  	conn_param.initiator_depth = 1;
> -	conn_param.retry_count = 10;
> +	conn_param.retry_count = 7;
> 
>  	ret = rdma_connect(cb->cm_id, &conn_param);
>  	if (ret) {
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html