From: Tom Tucker <tom@opengridcomputing.com>
To: Yan Burman <yanb@mellanox.com>
Cc: Wendy Cheng <s.wendy.cheng@gmail.com>,
"J. Bruce Fields" <bfields@fieldses.org>,
"Atchley, Scott" <atchleyes@ornl.gov>,
"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
"linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>,
Or Gerlitz <ogerlitz@mellanox.com>
Subject: Re: NFS over RDMA benchmark
Date: Mon, 29 Apr 2013 08:07:24 -0500 [thread overview]
Message-ID: <517E708C.1010705@opengridcomputing.com> (raw)
In-Reply-To: <517E701F.1010807@opengridcomputing.com>
On 4/29/13 8:05 AM, Tom Tucker wrote:
> On 4/29/13 7:16 AM, Yan Burman wrote:
>>
>>> -----Original Message-----
>>> From: Wendy Cheng [mailto:s.wendy.cheng@gmail.com]
>>> Sent: Monday, April 29, 2013 08:35
>>> To: J. Bruce Fields
>>> Cc: Yan Burman; Atchley, Scott; Tom Tucker; linux-rdma@vger.kernel.org;
>>> linux-nfs@vger.kernel.org; Or Gerlitz
>>> Subject: Re: NFS over RDMA benchmark
>>>
>>> On Sun, Apr 28, 2013 at 7:42 AM, J. Bruce Fields
>>> <bfields@fieldses.org> wrote:
>>>
>>>>> On Wed, Apr 17, 2013 at 7:36 AM, Yan Burman
>>>>> When running ib_send_bw, I get 4.3-4.5 GB/sec for block sizes 4-512K.
>>>>> When I run fio over rdma mounted nfs, I get 260-2200MB/sec for the
>>>>> same block sizes (4-512K). running over IPoIB-CM, I get
>>>>> 200-980MB/sec.
>>>> ...
>>> [snip]
>>>
>>>>> 36.18% nfsd [kernel.kallsyms] [k] mutex_spin_on_owner
>>>> That's the inode i_mutex.
>>>>
>>>>> 14.70%-- svc_send
>>>> That's the xpt_mutex (ensuring rpc replies aren't interleaved).
>>>>
>>>>> 9.63% nfsd [kernel.kallsyms] [k]
>>>>> _raw_spin_lock_irqsave
>>>>>
>>>> And that (and __free_iova below) looks like iova_rbtree_lock.
>>>>
>>>>
>>> Let's revisit your command:
>>>
>>> "FIO arguments: --rw=randread --bs=4k --numjobs=2 --iodepth=128 --
>>> ioengine=libaio --size=100000k --prioclass=1 --prio=0 --cpumask=255
>>> --loops=25 --direct=1 --invalidate=1 --fsync_on_close=1
>>> --randrepeat=1 --
>>> norandommap --group_reporting --exitall --buffered=0"
>>>
>> I tried block sizes from 4-512K.
>> 4K does not give 2.2GB bandwidth - optimal bandwidth is achieved
>> around 128-256K block size
>>
>>> * inode's i_mutex:
>>> If increasing process/file count didn't help, maybe increase "iodepth"
>>> (say 512 ?) could offset the i_mutex overhead a little bit ?
>>>
>> I tried with different iodepth parameters, but found no improvement
>> above iodepth 128.
>>
>>> * xpt_mutex:
>>> (no idea)
>>>
>>> * iova_rbtree_lock
>>> DMA mapping fragmentation ? I have not studied whether NFS-RDMA
>>> routines such as "svc_rdma_sendto()" could do better but maybe
>>> sequential
>>> IO (instead of "randread") could help ? Bigger block size (instead
>>> of 4K) can
>>> help ?
>>>
>
> I think the biggest issue is that max_payload for TCP is 2MB but only
> 256k for RDMA.
Sorry, I mean 1MB...
>
>> I am trying to simulate real load (more or less), that is the reason
>> I use randread. Anyhow, read does not result in better performance.
>> It's probably because backing storage is tmpfs...
>>
>> Yan
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
next prev parent reply other threads:[~2013-04-29 13:07 UTC|newest]
Thread overview: 41+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-04-17 14:36 NFS over RDMA benchmark Yan Burman
2013-04-17 17:15 ` Wendy Cheng
2013-04-17 17:32 ` Atchley, Scott
2013-04-17 18:06 ` Wendy Cheng
2013-04-18 12:47 ` Yan Burman
2013-04-18 16:16 ` Wendy Cheng
2013-04-23 21:06 ` J. Bruce Fields
2013-04-24 12:35 ` Yan Burman
2013-04-24 15:05 ` J. Bruce Fields
2013-04-24 15:26 ` J. Bruce Fields
2013-04-24 16:27 ` Wendy Cheng
2013-04-24 18:04 ` Wendy Cheng
2013-04-24 18:26 ` Tom Talpey
2013-04-25 17:18 ` Wendy Cheng
2013-04-25 19:01 ` Phil Pishioneri
2013-04-25 20:14 ` Tom Talpey
2013-04-25 20:04 ` Tom Talpey
2013-04-25 21:17 ` Tom Tucker
2013-04-25 21:58 ` Wendy Cheng
2013-04-25 22:26 ` Wendy Cheng
2013-04-28 6:28 ` Yan Burman
2013-04-28 14:42 ` J. Bruce Fields
2013-04-29 5:34 ` Wendy Cheng
2013-04-29 12:16 ` Yan Burman
2013-04-29 13:05 ` Tom Tucker
2013-04-29 13:07 ` Tom Tucker [this message]
2013-04-30 5:09 ` Yan Burman
2013-04-30 13:05 ` Tom Talpey
2013-04-30 14:23 ` Yan Burman
2013-04-30 14:44 ` Tom Talpey
2013-04-30 14:20 ` Tom Talpey
2013-04-30 14:38 ` Yan Burman
2013-04-30 18:58 ` Tom Tucker
[not found] ` <CALsNU1MsjH5=p4Wtj2aJ5+odC7y7-5oTGhrzOL-=15pXaYYUZw@mail.gmail.com>
[not found] ` <CABgxfbFhZTBO81WC5BcRRfQB_YBjE4N=sfS+G9eAzaFHYC_dWw@mail.gmail.com>
2013-06-20 14:56 ` Or Gerlitz
2013-04-30 16:24 ` Wendy Cheng
2013-04-30 13:38 ` J. Bruce Fields
2013-04-19 2:27 ` Peng Tao
2013-04-22 11:07 ` Yan Burman
[not found] <51703280.03e9440a.06a6.3f9f@mx.google.com>
2013-04-18 19:15 ` Wendy Cheng
2013-04-19 1:03 ` Atchley, Scott
2013-04-19 3:35 ` Spencer
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=517E708C.1010705@opengridcomputing.com \
--to=tom@opengridcomputing.com \
--cc=atchleyes@ornl.gov \
--cc=bfields@fieldses.org \
--cc=linux-nfs@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=ogerlitz@mellanox.com \
--cc=s.wendy.cheng@gmail.com \
--cc=yanb@mellanox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox