Linux NFS development
 help / color / mirror / Atom feed
From: Tom Talpey <tom@talpey.com>
To: Yan Burman <yanb@mellanox.com>
Cc: "J. Bruce Fields" <bfields@fieldses.org>,
	Wendy Cheng <s.wendy.cheng@gmail.com>,
	"Atchley, Scott" <atchleyes@ornl.gov>,
	Tom Tucker <tom@opengridcomputing.com>,
	"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
	"linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>,
	Or Gerlitz <ogerlitz@mellanox.com>
Subject: Re: NFS over RDMA benchmark
Date: Tue, 30 Apr 2013 09:05:06 -0400	[thread overview]
Message-ID: <517FC182.3030703@talpey.com> (raw)
In-Reply-To: <0EE9A1CDC8D6434DB00095CD7DB873462CF9C90C@MTLDAG01.mtl.com>

On 4/30/2013 1:09 AM, Yan Burman wrote:
>
>
>> -----Original Message-----
>> From: J. Bruce Fields [mailto:bfields@fieldses.org]
>> Sent: Sunday, April 28, 2013 17:43
>> To: Yan Burman
>> Cc: Wendy Cheng; Atchley, Scott; Tom Tucker; linux-rdma@vger.kernel.org;
>> linux-nfs@vger.kernel.org; Or Gerlitz
>> Subject: Re: NFS over RDMA benchmark
>>
>> On Sun, Apr 28, 2013 at 06:28:16AM +0000, Yan Burman wrote:
>>>>>>>>>>> On Wed, Apr 17, 2013 at 7:36 AM, Yan Burman
>>>>>>>>>>> <yanb@mellanox.com>
>>>>>>>>>>>> I've been trying to do some benchmarks for NFS over
>>>>>>>>>>>> RDMA and I seem to
>>>>>>>>> only get about half of the bandwidth that the HW can give me.
>>>>>>>>>>>> My setup consists of 2 servers each with 16 cores,
>>>>>>>>>>>> 32Gb of memory, and
>>>>>>>>> Mellanox ConnectX3 QDR card over PCI-e gen3.
>>>>>>>>>>>> These servers are connected to a QDR IB switch. The
>>>>>>>>>>>> backing storage on
>>>>>>>>> the server is tmpfs mounted with noatime.
>>>>>>>>>>>> I am running kernel 3.5.7.
>>>>>>>>>>>>
>>>>>>>>>>>> When running ib_send_bw, I get 4.3-4.5 GB/sec for
>>>>>>>>>>>> block sizes 4-
>>>> 512K.
>>>>>>>>>>>> When I run fio over rdma mounted nfs, I get
>>>>>>>>>>>> 260-2200MB/sec for the
>>>>>>>>> same block sizes (4-512K). running over IPoIB-CM, I get
>>>>>>>>> 200-
>>>> 980MB/sec.
>> ...
>>>>>>>> I am trying to get maximum performance from a single server
>>>>>>>> - I used 2
>>>>>>> processes in fio test - more than 2 did not show any performance
>> boost.
>>>>>>>> I tried running fio from 2 different PCs on 2 different
>>>>>>>> files, but the sum of
>>>>>>> the two is more or less the same as running from single client PC.
>>>>>>>>
>
> I finally got up to 4.1GB/sec bandwidth with RDMA (ipoib-CM bandwidth is also way higher now).
> For some reason when I had intel IOMMU enabled, the performance dropped significantly.
> I now get up to ~95K IOPS and 4.1GB/sec bandwidth.

Excellent, but is that 95K IOPS a typo? At 4KB, that's less than 400MBps.

What is the client CPU percentage you see under this workload, and
how different are the NFS/RDMA and NFS/IPoIB overheads?

> Now I will take care of the issue that I am running only at 40Gbit/s instead of 56Gbit/s, but that is another unrelated problem (I suspect I have a cable issue).
>
> This is still strange, since ib_send_bw with intel iommu enabled did get up to 4.5GB/sec, so why did intel iommu affect only nfs code?

You'll need to do more profiling to track that down. I would suspect
that ib_send_bw is using some sort of direct hardware access, bypassing
the IOMMU management and possibly performing no dynamic memory registration.

The NFS/RDMA code goes via the standard kernel DMA API, and correctly
registers/deregisters memory on a per-i/o basis in order to provide
storage data integrity. Perhaps there are overheads in the IOMMU
management which can be addressed.

  reply	other threads:[~2013-04-30 13:05 UTC|newest]

Thread overview: 41+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2013-04-17 14:36 NFS over RDMA benchmark Yan Burman
2013-04-17 17:15 ` Wendy Cheng
2013-04-17 17:32   ` Atchley, Scott
2013-04-17 18:06     ` Wendy Cheng
2013-04-18 12:47       ` Yan Burman
2013-04-18 16:16         ` Wendy Cheng
2013-04-23 21:06         ` J. Bruce Fields
2013-04-24 12:35           ` Yan Burman
2013-04-24 15:05             ` J. Bruce Fields
2013-04-24 15:26               ` J. Bruce Fields
2013-04-24 16:27                 ` Wendy Cheng
2013-04-24 18:04                   ` Wendy Cheng
2013-04-24 18:26                     ` Tom Talpey
2013-04-25 17:18                       ` Wendy Cheng
2013-04-25 19:01                         ` Phil Pishioneri
2013-04-25 20:14                           ` Tom Talpey
2013-04-25 20:04                         ` Tom Talpey
2013-04-25 21:17                           ` Tom Tucker
2013-04-25 21:58                             ` Wendy Cheng
2013-04-25 22:26                               ` Wendy Cheng
2013-04-28  6:28                 ` Yan Burman
2013-04-28 14:42                   ` J. Bruce Fields
2013-04-29  5:34                     ` Wendy Cheng
2013-04-29 12:16                       ` Yan Burman
2013-04-29 13:05                         ` Tom Tucker
2013-04-29 13:07                           ` Tom Tucker
2013-04-30  5:09                     ` Yan Burman
2013-04-30 13:05                       ` Tom Talpey [this message]
2013-04-30 14:23                         ` Yan Burman
2013-04-30 14:44                           ` Tom Talpey
2013-04-30 14:20                       ` Tom Talpey
2013-04-30 14:38                         ` Yan Burman
2013-04-30 18:58                           ` Tom Tucker
     [not found]                             ` <CALsNU1MsjH5=p4Wtj2aJ5+odC7y7-5oTGhrzOL-=15pXaYYUZw@mail.gmail.com>
     [not found]                               ` <CABgxfbFhZTBO81WC5BcRRfQB_YBjE4N=sfS+G9eAzaFHYC_dWw@mail.gmail.com>
2013-06-20 14:56                                 ` Or Gerlitz
2013-04-30 16:24                       ` Wendy Cheng
2013-04-30 13:38                     ` J. Bruce Fields
2013-04-19  2:27 ` Peng Tao
2013-04-22 11:07   ` Yan Burman
     [not found] <51703280.03e9440a.06a6.3f9f@mx.google.com>
2013-04-18 19:15 ` Wendy Cheng
2013-04-19  1:03   ` Atchley, Scott
2013-04-19  3:35     ` Spencer

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=517FC182.3030703@talpey.com \
    --to=tom@talpey.com \
    --cc=atchleyes@ornl.gov \
    --cc=bfields@fieldses.org \
    --cc=linux-nfs@vger.kernel.org \
    --cc=linux-rdma@vger.kernel.org \
    --cc=ogerlitz@mellanox.com \
    --cc=s.wendy.cheng@gmail.com \
    --cc=tom@opengridcomputing.com \
    --cc=yanb@mellanox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox