From: "J. Bruce Fields" <bfields@fieldses.org>
To: Yan Burman <yanb@mellanox.com>
Cc: Wendy Cheng <s.wendy.cheng@gmail.com>,
"Atchley, Scott" <atchleyes@ornl.gov>,
Tom Tucker <tom@opengridcomputing.com>,
"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
"linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>,
Or Gerlitz <ogerlitz@mellanox.com>
Subject: Re: NFS over RDMA benchmark
Date: Sun, 28 Apr 2013 10:42:48 -0400 [thread overview]
Message-ID: <20130428144248.GA2037@fieldses.org> (raw)
In-Reply-To: <0EE9A1CDC8D6434DB00095CD7DB873462CF9A820@MTLDAG01.mtl.com>
On Sun, Apr 28, 2013 at 06:28:16AM +0000, Yan Burman wrote:
> > > > > > > >> On Wed, Apr 17, 2013 at 7:36 AM, Yan Burman
> > > > > > > >> <yanb@mellanox.com>
> > > > > > > >>> I've been trying to do some benchmarks for NFS over RDMA
> > > > > > > >>> and I seem to
> > > > > > > only get about half of the bandwidth that the HW can give me.
> > > > > > > >>> My setup consists of 2 servers each with 16 cores, 32Gb of
> > > > > > > >>> memory, and
> > > > > > > Mellanox ConnectX3 QDR card over PCI-e gen3.
> > > > > > > >>> These servers are connected to a QDR IB switch. The
> > > > > > > >>> backing storage on
> > > > > > > the server is tmpfs mounted with noatime.
> > > > > > > >>> I am running kernel 3.5.7.
> > > > > > > >>>
> > > > > > > >>> When running ib_send_bw, I get 4.3-4.5 GB/sec for block sizes 4-
> > 512K.
> > > > > > > >>> When I run fio over rdma mounted nfs, I get 260-2200MB/sec
> > > > > > > >>> for the
> > > > > > > same block sizes (4-512K). running over IPoIB-CM, I get 200-
> > 980MB/sec.
...
> > > > > > I am trying to get maximum performance from a single server - I
> > > > > > used 2
> > > > > processes in fio test - more than 2 did not show any performance boost.
> > > > > > I tried running fio from 2 different PCs on 2 different files,
> > > > > > but the sum of
> > > > > the two is more or less the same as running from single client PC.
> > > > > >
> > > > > > What I did see is that server is sweating a lot more than the
> > > > > > clients and
> > > > > more than that, it has 1 core (CPU5) in 100% softirq tasklet:
> > > > > > cat /proc/softirqs
...
> > > > Perf top for the CPU with high tasklet count gives:
> > > >
> > > > samples pcnt RIP function DSO
...
> > > > 2787.00 24.1% ffffffff81062a00 mutex_spin_on_owner
> > /root/vmlinux
...
> > Googling around.... I think we want:
> >
> > perf record -a --call-graph
> > (give it a chance to collect some samples, then ^C)
> > perf report --call-graph --stdio
> >
>
> Sorry it took me a while to get perf to show the call trace (did not enable frame pointers in kernel and struggled with perf options...), but what I get is:
> 36.18% nfsd [kernel.kallsyms] [k] mutex_spin_on_owner
> |
> --- mutex_spin_on_owner
> |
> |--99.99%-- __mutex_lock_slowpath
> | mutex_lock
> | |
> | |--85.30%-- generic_file_aio_write
That's the inode i_mutex.
> | | do_sync_readv_writev
> | | do_readv_writev
> | | vfs_writev
> | | nfsd_vfs_write
> | | nfsd_write
> | | nfsd3_proc_write
> | | nfsd_dispatch
> | | svc_process_common
> | | svc_process
> | | nfsd
> | | kthread
> | | kernel_thread_helper
> | |
> | --14.70%-- svc_send
That's the xpt_mutex (ensuring rpc replies aren't interleaved).
> | svc_process
> | nfsd
> | kthread
> | kernel_thread_helper
> --0.01%-- [...]
>
> 9.63% nfsd [kernel.kallsyms] [k] _raw_spin_lock_irqsave
> |
> --- _raw_spin_lock_irqsave
> |
> |--43.97%-- alloc_iova
And that (and __free_iova below) looks like iova_rbtree_lock.
--b.
> | intel_alloc_iova
> | __intel_map_single
> | intel_map_page
> | |
> | |--60.47%-- svc_rdma_sendto
> | | svc_send
> | | svc_process
> | | nfsd
> | | kthread
> | | kernel_thread_helper
> | |
> | |--30.10%-- rdma_read_xdr
> | | svc_rdma_recvfrom
> | | svc_recv
> | | nfsd
> | | kthread
> | | kernel_thread_helper
> | |
> | |--6.69%-- svc_rdma_post_recv
> | | send_reply
> | | svc_rdma_sendto
> | | svc_send
> | | svc_process
> | | nfsd
> | | kthread
> | | kernel_thread_helper
> | |
> | --2.74%-- send_reply
> | svc_rdma_sendto
> | svc_send
> | svc_process
> | nfsd
> | kthread
> | kernel_thread_helper
> |
> |--37.52%-- __free_iova
> | flush_unmaps
> | add_unmap
> | intel_unmap_page
> | |
> | |--97.18%-- svc_rdma_put_frmr
> | | sq_cq_reap
> | | dto_tasklet_func
> | | tasklet_action
> | | __do_softirq
> | | call_softirq
> | | do_softirq
> | | |
> | | |--97.40%-- irq_exit
> | | | |
> | | | |--99.85%-- do_IRQ
> | | | | ret_from_intr
> | | | | |
> | | | | |--40.74%-- generic_file_buffered_write
> | | | | | __generic_file_aio_write
> | | | | | generic_file_aio_write
> | | | | | do_sync_readv_writev
> | | | | | do_readv_writev
> | | | | | vfs_writev
> | | | | | nfsd_vfs_write
> | | | | | nfsd_write
> | | | | | nfsd3_proc_write
> | | | | | nfsd_dispatch
> | | | | | svc_process_common
> | | | | | svc_process
> | | | | | nfsd
> | | | | | kthread
> | | | | | kernel_thread_helper
> | | | | |
> | | | | |--25.21%-- __mutex_lock_slowpath
> | | | | | mutex_lock
> | | | | | |
> | | | | | |--94.84%-- generic_file_aio_write
> | | | | | | do_sync_readv_writev
> | | | | | | do_readv_writev
> | | | | | | vfs_writev
> | | | | | | nfsd_vfs_write
> | | | | | | nfsd_write
> | | | | | | nfsd3_proc_write
> | | | | | | nfsd_dispatch
> | | | | | | svc_process_common
> | | | | | | svc_process
> | | | | | | nfsd
> | | | | | | kthread
> | | | | | | kernel_thread_helper
> | | | | | |
>
> The entire trace is almost 1MB, so send me an off-list message if you want it.
>
> Yan
>
next prev parent reply other threads:[~2013-04-28 14:42 UTC|newest]
Thread overview: 41+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-04-17 14:36 NFS over RDMA benchmark Yan Burman
2013-04-17 17:15 ` Wendy Cheng
2013-04-17 17:32 ` Atchley, Scott
2013-04-17 18:06 ` Wendy Cheng
2013-04-18 12:47 ` Yan Burman
2013-04-18 16:16 ` Wendy Cheng
2013-04-23 21:06 ` J. Bruce Fields
2013-04-24 12:35 ` Yan Burman
2013-04-24 15:05 ` J. Bruce Fields
2013-04-24 15:26 ` J. Bruce Fields
2013-04-24 16:27 ` Wendy Cheng
2013-04-24 18:04 ` Wendy Cheng
2013-04-24 18:26 ` Tom Talpey
2013-04-25 17:18 ` Wendy Cheng
2013-04-25 19:01 ` Phil Pishioneri
2013-04-25 20:14 ` Tom Talpey
2013-04-25 20:04 ` Tom Talpey
2013-04-25 21:17 ` Tom Tucker
2013-04-25 21:58 ` Wendy Cheng
2013-04-25 22:26 ` Wendy Cheng
2013-04-28 6:28 ` Yan Burman
2013-04-28 14:42 ` J. Bruce Fields [this message]
2013-04-29 5:34 ` Wendy Cheng
2013-04-29 12:16 ` Yan Burman
2013-04-29 13:05 ` Tom Tucker
2013-04-29 13:07 ` Tom Tucker
2013-04-30 5:09 ` Yan Burman
2013-04-30 13:05 ` Tom Talpey
2013-04-30 14:23 ` Yan Burman
2013-04-30 14:44 ` Tom Talpey
2013-04-30 14:20 ` Tom Talpey
2013-04-30 14:38 ` Yan Burman
2013-04-30 18:58 ` Tom Tucker
[not found] ` <CALsNU1MsjH5=p4Wtj2aJ5+odC7y7-5oTGhrzOL-=15pXaYYUZw@mail.gmail.com>
[not found] ` <CABgxfbFhZTBO81WC5BcRRfQB_YBjE4N=sfS+G9eAzaFHYC_dWw@mail.gmail.com>
2013-06-20 14:56 ` Or Gerlitz
2013-04-30 16:24 ` Wendy Cheng
2013-04-30 13:38 ` J. Bruce Fields
2013-04-19 2:27 ` Peng Tao
2013-04-22 11:07 ` Yan Burman
[not found] <51703280.03e9440a.06a6.3f9f@mx.google.com>
2013-04-18 19:15 ` Wendy Cheng
2013-04-19 1:03 ` Atchley, Scott
2013-04-19 3:35 ` Spencer
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20130428144248.GA2037@fieldses.org \
--to=bfields@fieldses.org \
--cc=atchleyes@ornl.gov \
--cc=linux-nfs@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=ogerlitz@mellanox.com \
--cc=s.wendy.cheng@gmail.com \
--cc=tom@opengridcomputing.com \
--cc=yanb@mellanox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox