* [PATCH] RDMA: Increasing RPCRDMA_MAX_DATA_SEGS @ 2011-07-21 17:49 Steve Dickson 2011-07-21 21:41 ` J. Bruce Fields 2011-07-22 8:19 ` Max Matveev 0 siblings, 2 replies; 6+ messages in thread From: Steve Dickson @ 2011-07-21 17:49 UTC (permalink / raw) To: Linux NFS Mailing list Our performance team has noticed that increasing RPCRDMA_MAX_DATA_SEGS from 8 to 64 significantly increases throughput when using the RDMA transport. Signed-off-by: Steve Dickson <steved@redhat.com> --- net/sunrpc/xprtrdma/xprt_rdma.h | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index cae761a..5d1cfe5 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -109,7 +109,7 @@ struct rpcrdma_ep { */ /* temporary static scatter/gather max */ -#define RPCRDMA_MAX_DATA_SEGS (8) /* max scatter/gather */ +#define RPCRDMA_MAX_DATA_SEGS (64) /* max scatter/gather */ #define RPCRDMA_MAX_SEGS (RPCRDMA_MAX_DATA_SEGS + 2) /* head+tail = 2 */ #define MAX_RPCRDMAHDR (\ /* max supported RPC/RDMA header */ \ -- 1.7.6 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH] RDMA: Increasing RPCRDMA_MAX_DATA_SEGS 2011-07-21 17:49 [PATCH] RDMA: Increasing RPCRDMA_MAX_DATA_SEGS Steve Dickson @ 2011-07-21 21:41 ` J. Bruce Fields 2011-07-22 1:42 ` Trond Myklebust 2011-07-22 8:19 ` Max Matveev 1 sibling, 1 reply; 6+ messages in thread From: J. Bruce Fields @ 2011-07-21 21:41 UTC (permalink / raw) To: Steve Dickson; +Cc: Linux NFS Mailing list, tom, tmtalpey On Thu, Jul 21, 2011 at 01:49:02PM -0400, Steve Dickson wrote: > Our performance team has noticed that increasing > RPCRDMA_MAX_DATA_SEGS from 8 to 64 significantly > increases throughput when using the RDMA transport. The main risk that I can see being that we have on the stack in two places: rpcrdma_register_fmr_external(struct rpcrdma_mr_seg *seg, ... { ... u64 physaddrs[RPCRDMA_MAX_DATA_SEGS]; rpcrdma_register_default_external(struct rpcrdma_mr_seg *seg, ... { ... struct ib_phys_buf ipb[RPCRDMA_MAX_DATA_SEGS]; Where ip_phys_buf is 16 bytes. So that's 512 bytes in the first case, 1024 in the second. This is called from rpciod--what are our rules about allocating memory from rpciod? --b. > > Signed-off-by: Steve Dickson <steved@redhat.com> > --- > net/sunrpc/xprtrdma/xprt_rdma.h | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h > index cae761a..5d1cfe5 100644 > --- a/net/sunrpc/xprtrdma/xprt_rdma.h > +++ b/net/sunrpc/xprtrdma/xprt_rdma.h > @@ -109,7 +109,7 @@ struct rpcrdma_ep { > */ > > /* temporary static scatter/gather max */ > -#define RPCRDMA_MAX_DATA_SEGS (8) /* max scatter/gather */ > +#define RPCRDMA_MAX_DATA_SEGS (64) /* max scatter/gather */ > #define RPCRDMA_MAX_SEGS (RPCRDMA_MAX_DATA_SEGS + 2) /* head+tail = 2 */ > #define MAX_RPCRDMAHDR (\ > /* max supported RPC/RDMA header */ \ > -- > 1.7.6 > > -- > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] RDMA: Increasing RPCRDMA_MAX_DATA_SEGS 2011-07-21 21:41 ` J. Bruce Fields @ 2011-07-22 1:42 ` Trond Myklebust 2011-07-22 1:55 ` J. Bruce Fields 0 siblings, 1 reply; 6+ messages in thread From: Trond Myklebust @ 2011-07-22 1:42 UTC (permalink / raw) To: J. Bruce Fields; +Cc: Steve Dickson, Linux NFS Mailing list, tom, tmtalpey On Thu, 2011-07-21 at 17:41 -0400, J. Bruce Fields wrote: > On Thu, Jul 21, 2011 at 01:49:02PM -0400, Steve Dickson wrote: > > Our performance team has noticed that increasing > > RPCRDMA_MAX_DATA_SEGS from 8 to 64 significantly > > increases throughput when using the RDMA transport. > > The main risk that I can see being that we have on the stack in two > places: > > rpcrdma_register_fmr_external(struct rpcrdma_mr_seg *seg, ... > { > ... > u64 physaddrs[RPCRDMA_MAX_DATA_SEGS]; > > rpcrdma_register_default_external(struct rpcrdma_mr_seg *seg, ... > { > ... > struct ib_phys_buf ipb[RPCRDMA_MAX_DATA_SEGS]; > > Where ip_phys_buf is 16 bytes. > > So that's 512 bytes in the first case, 1024 in the second. This is > called from rpciod--what are our rules about allocating memory from > rpciod? Is that allocated on the stack? We should always try to avoid 1024-byte allocations on the stack, since that eats up a full 1/8th (or 1/4 in the case of 4k stacks) of the total stack space. If, OTOH, that memory is being allocated dynamically, then the rule is "don't let rpciod sleep". Cheers Trond -- Trond Myklebust Linux NFS client maintainer NetApp Trond.Myklebust@netapp.com www.netapp.com ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] RDMA: Increasing RPCRDMA_MAX_DATA_SEGS 2011-07-22 1:42 ` Trond Myklebust @ 2011-07-22 1:55 ` J. Bruce Fields 0 siblings, 0 replies; 6+ messages in thread From: J. Bruce Fields @ 2011-07-22 1:55 UTC (permalink / raw) To: Trond Myklebust; +Cc: Steve Dickson, Linux NFS Mailing list, tom, tmtalpey On Thu, Jul 21, 2011 at 09:42:04PM -0400, Trond Myklebust wrote: > On Thu, 2011-07-21 at 17:41 -0400, J. Bruce Fields wrote: > > On Thu, Jul 21, 2011 at 01:49:02PM -0400, Steve Dickson wrote: > > > Our performance team has noticed that increasing > > > RPCRDMA_MAX_DATA_SEGS from 8 to 64 significantly > > > increases throughput when using the RDMA transport. > > > > The main risk that I can see being that we have on the stack in two > > places: > > > > rpcrdma_register_fmr_external(struct rpcrdma_mr_seg *seg, ... > > { > > ... > > u64 physaddrs[RPCRDMA_MAX_DATA_SEGS]; > > > > rpcrdma_register_default_external(struct rpcrdma_mr_seg *seg, ... > > { > > ... > > struct ib_phys_buf ipb[RPCRDMA_MAX_DATA_SEGS]; > > > > Where ip_phys_buf is 16 bytes. > > > > So that's 512 bytes in the first case, 1024 in the second. This is > > called from rpciod--what are our rules about allocating memory from > > rpciod? > > Is that allocated on the stack? We should always try to avoid 1024-byte > allocations on the stack, since that eats up a full 1/8th (or 1/4 in the > case of 4k stacks) of the total stack space. Right, it's on the stack, so I was wondering what we should do instead.... > If, OTOH, that memory is being allocated dynamically, then the rule is > "don't let rpciod sleep". OK, so, looking around, the buf_alloc methods might provide examples to follow for dynamic allocation here? --b. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] RDMA: Increasing RPCRDMA_MAX_DATA_SEGS 2011-07-21 17:49 [PATCH] RDMA: Increasing RPCRDMA_MAX_DATA_SEGS Steve Dickson 2011-07-21 21:41 ` J. Bruce Fields @ 2011-07-22 8:19 ` Max Matveev 2011-07-25 15:18 ` Steve Dickson 1 sibling, 1 reply; 6+ messages in thread From: Max Matveev @ 2011-07-22 8:19 UTC (permalink / raw) To: Steve Dickson; +Cc: Linux NFS Mailing list On Thu, 21 Jul 2011 13:49:02 -0400, Steve Dickson wrote: steved> Our performance team has noticed that increasing steved> RPCRDMA_MAX_DATA_SEGS from 8 to 64 significantly steved> increases throughput when using the RDMA transport. Did they try new client with old server and vice versa? Both read and write? max ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] RDMA: Increasing RPCRDMA_MAX_DATA_SEGS 2011-07-22 8:19 ` Max Matveev @ 2011-07-25 15:18 ` Steve Dickson 0 siblings, 0 replies; 6+ messages in thread From: Steve Dickson @ 2011-07-25 15:18 UTC (permalink / raw) To: Max Matveev; +Cc: Linux NFS Mailing list, Ben England Sorry for the delayed response... I took a day off.. On 07/22/2011 04:19 AM, Max Matveev wrote: > On Thu, 21 Jul 2011 13:49:02 -0400, Steve Dickson wrote: > > steved> Our performance team has noticed that increasing > steved> RPCRDMA_MAX_DATA_SEGS from 8 to 64 significantly > steved> increases throughput when using the RDMA transport. > > Did they try new client with old server and vice versa? > Both read and write? I believe it was done on the server side, but I've cc-ed the person who did the testing.... steved. ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2011-07-25 15:18 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2011-07-21 17:49 [PATCH] RDMA: Increasing RPCRDMA_MAX_DATA_SEGS Steve Dickson 2011-07-21 21:41 ` J. Bruce Fields 2011-07-22 1:42 ` Trond Myklebust 2011-07-22 1:55 ` J. Bruce Fields 2011-07-22 8:19 ` Max Matveev 2011-07-25 15:18 ` Steve Dickson
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).