From: keith.busch@intel.com (Keith Busch)
Subject: bad IOPS when running multiple btest/fio in parallel
Date: Fri, 12 Oct 2018 08:39:21 -0600 [thread overview]
Message-ID: <20181012143921.GA15177@localhost.localdomain> (raw)
In-Reply-To: <1539319463418.78556@marvell.com>
On Fri, Oct 12, 2018@04:44:22AM +0000, Yao Lin wrote:
> Today I changed to a much simpler setup and the same issue persists.
>
> Directly connect 2 PCs (identical hardware) with a pair of 100G rNICs. Create a null block device on the target PC and configure it as the NVMeOF target. So, there is no switch or SSD in this setup. And this is a single FIO, not the 4 FIO in parallel I mentioned earlier.
>
> Start fio test against that null block device from the host, the best IOPS is 1550K. That's the best IOPS after I try out many different QD, # of job, and CPU affinity setting. Run the same fio test on the target, I get 2250K IOPS (it jumps to 3650K when I increased the number of threads). ?
>
> So it seems to me that Linux NVMe stack is quite good and can support 100Gb/s + throughput. But the same can not be said of the NVMeOF stack. Any tuning possible?
You're sure it's the software stack? Need to check your CPU utilization to
see if that's a possibility.
next prev parent reply other threads:[~2018-10-12 14:39 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-10-12 4:44 bad IOPS when running multiple btest/fio in parallel Yao Lin
2018-10-12 14:39 ` Keith Busch [this message]
2018-10-12 15:37 ` [EXT] " Yao Lin
2018-10-12 15:49 ` Bart Van Assche
2018-10-12 16:02 ` [EXT] " Yao Lin
2018-10-15 7:50 ` Sagi Grimberg
-- strict thread matches above, loose matches on Subject: below --
2018-10-10 21:52 Yao Lin
2018-10-15 7:55 ` Sagi Grimberg
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20181012143921.GA15177@localhost.localdomain \
--to=keith.busch@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox