From: "Steve Wise" <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
To: 'Chuck Lever'
<chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Subject: RE: [PATCH 10/10] svcrdma: Switch CQs from IB_POLL_SOFTIRQ to IB_POLL_WORKQUEUE
Date: Thu, 28 Apr 2016 10:59:47 -0500 [thread overview]
Message-ID: <00ec01d1a166$fd134650$f739d2f0$@opengridcomputing.com> (raw)
In-Reply-To: <20160428151550.13068.24199.stgit-Hs+gFlyCn65vLzlybtyyYzGyq/o6K9yX@public.gmane.org>
> -----Original Message-----
> From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-
> owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Chuck Lever
> Sent: Thursday, April 28, 2016 10:16 AM
> To: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Subject: [PATCH 10/10] svcrdma: Switch CQs from IB_POLL_SOFTIRQ to
> IB_POLL_WORKQUEUE
>
> Spread NFSD completion handling across CPUs, and replace
> BH-friendly spin locking with plain spin locks.
>
> iozone -i0 -i1 -s128m -y1k -az -I -N
>
> Microseconds/op Mode. Output is in microseconds per operation.
>
> Before:
> KB reclen write rewrite read reread
> 131072 1 51 51 43 43
> 131072 2 53 52 42 43
> 131072 4 53 52 43 43
> 131072 8 55 54 44 44
> 131072 16 62 59 49 47
> 131072 32 72 69 53 53
> 131072 64 92 87 66 66
> 131072 128 144 130 94 93
> 131072 256 225 216 146 145
> 131072 512 485 474 251 251
> 131072 1024 573 540 514 512
> 131072 2048 1007 941 624 618
> 131072 4096 1672 1699 976 969
> 131072 8192 3179 3158 1660 1649
> 131072 16384 5836 5659 3062 3041
>
> After:
> KB reclen write rewrite read reread
> 131072 1 54 54 43 43
> 131072 2 55 55 43 43
> 131072 4 56 57 44 45
> 131072 8 59 58 45 45
> 131072 16 64 62 47 47
> 131072 32 76 74 54 54
> 131072 64 96 91 67 66
> 131072 128 148 133 97 97
> 131072 256 229 227 148 147
> 131072 512 488 445 252 255
> 131072 1024 582 534 511 540
> 131072 2048 998 988 614 620
> 131072 4096 1685 1679 946 965
> 131072 8192 3113 3048 1650 1644
> 131072 16384 6010 5745 3046 3053
>
> NFS READ is roughly the same, NFS WRITE is marginally worse.
>
> Before:
> GETATTR:
> 242 ops (0%)
> avg bytes sent per op: 127
> avg bytes received per op: 112
> backlog wait: 0.000000
> RTT: 0.041322
> total execute time: 0.049587 (milliseconds)
>
> After:
> GETATTR:
> 242 ops (0%)
> avg bytes sent per op: 127
> avg bytes received per op: 112
> backlog wait: 0.000000
> RTT: 0.045455
> total execute time: 0.053719 (milliseconds)
>
> Small op latency increased by 4usec.
>
Hey Chuck, in what scenario or under what type of load do you expect this change to help performance? I guess it would help as you scale out the number of clients and thus the number of CQs in use? Do you do any measurements along these lines?
Stevo
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
next prev parent reply other threads:[~2016-04-28 15:59 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-04-28 15:14 [PATCH 00/10] NFS/RDMA server updates proposed for v4.7 Chuck Lever
[not found] ` <20160428150915.13068.94602.stgit-Hs+gFlyCn65vLzlybtyyYzGyq/o6K9yX@public.gmane.org>
2016-04-28 15:14 ` [PATCH 01/10] svcrdma: Support IPv6 with NFS/RDMA Chuck Lever
2016-04-28 15:14 ` [PATCH 02/10] svcrdma: Do not add XDR padding to xdr_buf page vector Chuck Lever
2016-04-28 15:14 ` [PATCH 03/10] svcrdma: svc_rdma_put_context() is invoked twice in Send error path Chuck Lever
2016-04-28 15:15 ` [PATCH 04/10] svcrdma: Remove superfluous line from rdma_read_chunks() Chuck Lever
2016-04-28 15:15 ` [PATCH 05/10] svcrdma: Post Receives only for forward channel requests Chuck Lever
2016-04-28 15:15 ` [PATCH 06/10] svcrdma: Drain QP before freeing svcrdma_xprt Chuck Lever
2016-04-28 15:15 ` [PATCH 07/10] svcrdma: Eliminate code duplication in svc_rdma_recvfrom() Chuck Lever
2016-04-28 15:15 ` [PATCH 08/10] svcrdma: Generalize svc_rdma_xdr_decode_req() Chuck Lever
2016-04-28 15:15 ` [PATCH 09/10] svcrdma: Simplify the check for backward direction replies Chuck Lever
2016-04-28 15:15 ` [PATCH 10/10] svcrdma: Switch CQs from IB_POLL_SOFTIRQ to IB_POLL_WORKQUEUE Chuck Lever
[not found] ` <20160428151550.13068.24199.stgit-Hs+gFlyCn65vLzlybtyyYzGyq/o6K9yX@public.gmane.org>
2016-04-28 15:59 ` Steve Wise [this message]
2016-04-28 16:15 ` Chuck Lever
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='00ec01d1a166$fd134650$f739d2f0$@opengridcomputing.com' \
--to=swise-7bpotxp6k4+p2yhjcf5u+vpxobypeauw@public.gmane.org \
--cc=chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org \
--cc=linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
--cc=linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox