From: azher@hep.caltech.edu (Azher Mughal)
Subject: NVMe scalability issue
Date: Mon, 01 Jun 2015 16:28:44 -0700 [thread overview]
Message-ID: <556CEAAC.6040206@hep.caltech.edu> (raw)
In-Reply-To: <alpine.LNX.2.00.1506012254500.15930@localhost.lm.intel.com>
I ran some tests last year before SC using 8 drives in a SuperMicro
server. Please see attached. OS was CentOS 6.5 I think.
-Azher
On 6/1/2015 4:02 PM, Keith Busch wrote:
> On Mon, 1 Jun 2015, Ming Lin wrote:
>> Hi list,
>>
>> I'm playing with 8 high performance NVMe devices on a 4 sockets server.
>> Each device can get 730K 4k read IOPS.
>>
>> Kernel: 4.1-rc3
>> fio test shows it doesn't scale well with 4 or more devices.
>> I wonder any possible direction to improve it.
>
> There was a demo at SC'14 with a heck of a lot more NVMe drives than
> that,
> and performance scaled quite linearly. Are your devices sharing PCI-e
> lanes?
>
> You could try setting "cpus_allowed" on each job to the CPU's on the
> socket local to the nvme device. That should get a measurable
> improvement,
> and if your irq's are appropriately affinitized.
>
> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-nvme
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 8drives-dd-SC9.PNG
Type: image/png
Size: 83980 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-nvme/attachments/20150601/814c0441/attachment-0001.png>
next prev parent reply other threads:[~2015-06-01 23:28 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-06-01 22:52 NVMe scalability issue Ming Lin
2015-06-01 23:02 ` Keith Busch
2015-06-01 23:24 ` Ming Lin
2015-06-02 3:30 ` Keith Busch
2015-06-02 17:24 ` Ming Lin
2015-06-02 18:22 ` Jens Axboe
2015-06-02 20:55 ` Ming Lin
2015-06-01 23:28 ` Azher Mughal [this message]
2015-06-02 7:58 ` Matias Bjørling
2015-06-02 19:03 ` Andrey Kuzmin
2015-06-02 19:09 ` Jens Axboe
2015-06-02 19:11 ` Andrey Kuzmin
2015-06-02 19:14 ` Jens Axboe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=556CEAAC.6040206@hep.caltech.edu \
--to=azher@hep.caltech.edu \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.