Linux NFS development
 help / color / mirror / Atom feed
* [PATCH REPOST] svcrdma: Limit ORD based on client's advertised IRD
@ 2008-05-19 19:17 Tom Tucker
       [not found] ` <1211224653.31725.62.camel-SMNkleLxa3ZimH42XvhXlA@public.gmane.org>
  0 siblings, 1 reply; 2+ messages in thread
From: Tom Tucker @ 2008-05-19 19:17 UTC (permalink / raw)
  To: Bruce Fields; +Cc: Talpey, Thomas, linux-nfs

svcrdma: Limit ORD based on client's advertised IRD
    
When adapters have differing IRD limits, the RDMA transport will fail to
connect properly. The RDMA transport should use the client's advertised
inbound read limit when computing its outbound read limit. For iWARP
transports, there is currently no standard for exchanging IRD/ORD
during connection establishment so the 'responder_resources' field in the
connect event is the local device's limit. The RDMA transport can be
configured to use a smaller ORD by writing the desired number to the
/proc/sys/sunrpc/svc_rdma/max_outbound_read_requests file.
    
Signed-off-by: Tom Tucker <tom@opengridcomputing.com>
---
Re-posted per Tom's suggestions. There was also a cast missing in the min
macro that caused a warning.

 net/sunrpc/xprtrdma/svc_rdma_transport.c |   16 +++++++++++++---
 1 files changed, 13 insertions(+), 3 deletions(-)

diff --git a/net/sunrpc/xprtrdma/svc_rdma_transport.c b/net/sunrpc/xprtrdma/svc_rdma_transport.c
index 68908b5..e1ae326 100644
--- a/net/sunrpc/xprtrdma/svc_rdma_transport.c
+++ b/net/sunrpc/xprtrdma/svc_rdma_transport.c
@@ -580,7 +580,7 @@ int svc_rdma_post_recv(struct svcxprt_rdma *xprt)
  * will call the recvfrom method on the listen xprt which will accept the new
  * connection.
  */
-static void handle_connect_req(struct rdma_cm_id *new_cma_id)
+static void handle_connect_req(struct rdma_cm_id *new_cma_id, size_t client_ird)
 {
 	struct svcxprt_rdma *listen_xprt = new_cma_id->context;
 	struct svcxprt_rdma *newxprt;
@@ -597,6 +597,9 @@ static void handle_connect_req(struct rdma_cm_id *new_cma_id)
 	dprintk("svcrdma: Creating newxprt=%p, cm_id=%p, listenxprt=%p\n",
 		newxprt, newxprt->sc_cm_id, listen_xprt);
 
+	/* Save client advertised inbound read limit for use later in accept. */
+	newxprt->sc_ord = client_ird;
+
 	/* Set the local and remote addresses in the transport */
 	sa = (struct sockaddr *)&newxprt->sc_cm_id->route.addr.dst_addr;
 	svc_xprt_set_remote(&newxprt->sc_xprt, sa, svc_addr_len(sa));
@@ -633,7 +636,8 @@ static int rdma_listen_handler(struct rdma_cm_id *cma_id,
 	case RDMA_CM_EVENT_CONNECT_REQUEST:
 		dprintk("svcrdma: Connect request on cma_id=%p, xprt = %p, "
 			"event=%d\n", cma_id, cma_id->context, event->event);
-		handle_connect_req(cma_id);
+		handle_connect_req(cma_id,
+				   event->param.conn.responder_resources);
 		break;
 
 	case RDMA_CM_EVENT_ESTABLISHED:
@@ -807,8 +811,14 @@ static struct svc_xprt *svc_rdma_accept(struct svc_xprt *xprt)
 				   (size_t)svcrdma_max_requests);
 	newxprt->sc_sq_depth = RPCRDMA_SQ_DEPTH_MULT * newxprt->sc_max_requests;
 
+	/* 
+	 * Limit ORD based on client limit, local device limit, and
+	 * configured svcrdma limit.
+	 */
 	newxprt->sc_ord =  min((size_t)devattr.max_qp_rd_atom,
-			       (size_t)svcrdma_ord);
+			       (size_t)newxprt->sc_ord);
+	newxprt->sc_ord = min((size_t)svcrdma_ord,
+			      (size_t)newxprt->sc_ord);
 
 	newxprt->sc_pd = ib_alloc_pd(newxprt->sc_cm_id->device);
 	if (IS_ERR(newxprt->sc_pd)) {


^ permalink raw reply related	[flat|nested] 2+ messages in thread

* Re: [PATCH REPOST] svcrdma: Limit ORD based on client's advertised IRD
       [not found] ` <1211224653.31725.62.camel-SMNkleLxa3ZimH42XvhXlA@public.gmane.org>
@ 2008-05-19 19:35   ` Talpey, Thomas
  0 siblings, 0 replies; 2+ messages in thread
From: Talpey, Thomas @ 2008-05-19 19:35 UTC (permalink / raw)
  To: Tom Tucker; +Cc: Bruce Fields, linux-nfs

At 03:17 PM 5/19/2008, Tom Tucker wrote:
>-static void handle_connect_req(struct rdma_cm_id *new_cma_id)
>+static void handle_connect_req(struct rdma_cm_id *new_cma_id, size_t client_ird)

Wow! That'll do it - 4 billion RDMA reads here we come!

Looks good to me with one tiny suggestion:

> 	newxprt->sc_ord =  min((size_t)devattr.max_qp_rd_atom,
>-			       (size_t)svcrdma_ord);
>+			       (size_t)newxprt->sc_ord);
>+	newxprt->sc_ord = min((size_t)svcrdma_ord,
>+			      (size_t)newxprt->sc_ord);
> 

You can code these a tiny bit more cleanly as min_t(size_t, foo, bar)
and thereby avoid all four casts.

Tom.


^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, other threads:[~2008-05-19 19:35 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2008-05-19 19:17 [PATCH REPOST] svcrdma: Limit ORD based on client's advertised IRD Tom Tucker
     [not found] ` <1211224653.31725.62.camel-SMNkleLxa3ZimH42XvhXlA@public.gmane.org>
2008-05-19 19:35   ` Talpey, Thomas

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox