The perf tool allocates 1 queue pair per thread (worker_fn is the thread entry point in the code) and there is 1 thread per core. Our findings on all of the devices we've tested has been that using multiple queue pairs does not increase the total performance of the SSD. The full quoted SSD hardware performance can be achieved with a single queue pair. This finding, however, is completely device specific. Different SSD controllers may have different behavior.

Using multiple queue pairs is a big advantage for the host software (i.e. SPDK) though. Using 1 queue pair per thread allows the host software to submit I/O in parallel with no locks.

On Thu, 2016-06-16 at 14:24 +0000, Li Tianxiang wrote:
Hi，

I noticed that the perf  app.  only  allocates one qpair.

using : 
"ns_ctx->u.nvme.qpair = spdk_nvme_ctrlr_alloc_io_qpair(ns_ctx->entry->u.nvme.ctrlr, 0);"

Does more  qpair help to the  performance?

And P3700  has about 30 submission  queues default.


________________________________________
发件人: SPDK <spdk-bounces@lists.01.org> 代表 Walker, Benjamin <benjamin.walker@intel.com>
发送时间: 2016年6月15日 23:29
收件人: Storage Performance Development Kit; sbradshaw@micron.com; Robert.Cleveland@skhms.com
主题: Re: [SPDK] SPDK fio benchmark support

The perf application should scale linearly with CPU cores - so if you get 1.6 million IOPS with 2 SSDs on 1 core, you should get 3.2 million IOPS with 4 SSDs on 2 cores. Since you aren't, there must be something else impacting the performance. Can you provide the following information:

1) The Xeon E5 SKU you are using
2) The number of CPU sockets and which socket the devices are attached to (all 4 need to be attached to the same socket that you are running perf on).
3) The command line parameters you are passing to the perf tool.

Thanks,
Ben

-----Original Message-----
From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Raj (Rajinikanth) Pandurangan
Sent: Wednesday, June 15, 2016 3:16 PM
To: Storage Performance Development Kit <spdk@lists.01.org>; sbradshaw@micron.com; Robert.Cleveland@skhms.com
Subject: Re: [SPDK] SPDK fio benchmark support

Hello Ben,

Thanks for the details.

I have Intel Xeon E5 system and able to saturate 2 Samsung NVMe SSDs (~1.6 million) on 1 core.   IOPS are same even when I had 4 SSDs on 1 core.  But I expect the IOPS to go up when I introduce extra cores with 4 SSDs and it did not.

Thanks,
-Raj P

-----Original Message-----
From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Walker, Benjamin
Sent: Wednesday, June 15, 2016 1:33 PM
To: Raj (Rajinikanth) Pandurangan; sbradshaw@micron.com; spdk@lists.01.org; Robert.Cleveland@skhms.com
Subject: Re: [SPDK] SPDK fio benchmark support

The fio plugin for SPDK is new and was only tested for a single job. We'd welcome patches to make it span more cores when given more jobs if someone is interested in doing the work here. At some point, I'm sure we'll get to it ourselves, but it's lower priority than the other work we're doing right now (NVMf target). The perf example we provide definitely does scale to many cores though, and it can optionally use libaio as its backend, so it is a great alternative to using fio that works today with no extra effort.

As far as expected performance scaling across cores with our perf tool, are you seeing the maximum quoted IOPS for all of the device(s) on the system with only a single core? If you are, adding more cores won't improve performance. If you are not getting the performance you expect from the hardware I'd love to know the details of the devices and the platform.

To provide a quick example from my own system (and none of the following should be treated as official benchmarking numbers for any reason), I have a Haswell Xeon E7 with 8 Intel P3700 NVMe SSDs attached. Running the perf tool on my system yields about 3 million 4k read IOPs on 1 core, which is the equivalent of 6 or 7 of my SSDs. Moving to two cores gives me the full hardware spec IOPS for 8 SSDs. If I only test against 4 SSDs I get the same performance (the maximum that the SSDs can provide) no matter how many cores I give the tool.

Thanks,
Ben

-----Original Message-----
From: Raj (Rajinikanth) Pandurangan [mailto:rajini.pandu@samsung.com]
Sent: Wednesday, June 15, 2016 11:52 AM
To: Walker, Benjamin <benjamin.walker@intel.com>; sbradshaw@micron.com; spdk@lists.01.org; Robert.Cleveland@skhms.com
Cc: Raj (Rajinikanth) Pandurangan <rajini.pandu@samsung.com>
Subject: RE: [SPDK] SPDK fio benchmark support

Hello All,

1. I tried to run SPDK fio with multiple nvme drives.  But for some reason, I don't see more than 1 core being used even when I increased numjobs?

Here is my config file.
=============================================================

[global]
ioengine=/spdk-master_temp_experiment/examples/nvme/fio_plugin/fio_plugin
group_reporting=1
direct=1
verify=0
time_based=1
ramp_time=0
runtime=10


[test1]
iodepth=128
rw=randread
bs=4k
#filename=0000.09.00.0/1:0000.07.00.0/1:0000.0a.00.0/1
filename=0000.09.00.0/1
numjobs=2

[test2]
iodepth=128
rw=randread
bs=4k
numjobs=1
filename=0000.08.00.0/1
================================================================

Has anyone succeeded using multiple cores?

2. I have also tried with 'perf' and I noticed that even though multiple cores were being used but performance numbers didn't scale.  Seemed like similar IOPS when I used single core.

Any insights would really be appreciated.

Thanks,
-Raj P

-----Original Message-----
From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Walker, Benjamin
Sent: Friday, February 26, 2016 2:30 PM
To: sbradshaw@micron.com; spdk@ml01.01.org; Robert.Cleveland@skhms.com
Subject: Re: [SPDK] SPDK fio benchmark support

As a follow up to this, I pushed some improvements to SPDK's setup scripts so that we now more easily support vfio. That makes it a lot easier to switch back and forth between the unvme driver and the SPDK driver. We'd happily accept more patches to improve vfio if anyone has suggestions.

Thanks all.

On Thu, 2016-02-18 at 18:59 +0000, Sam Bradshaw (sbradshaw) wrote:
Hi Robert,

We built a fio ioengine plugin for benchmarking SPDK as well as our own userspace NVMe driver.  Source and documentation are available here:  https://github.com/MicronSSD/unvme

(SPDK fio plugin is ioengine/spdk_fio.c)

If you have any questions on how to use it or interpret benchmark results, feel free to ask.

-Sam

From: SPDK [mailto:spdk-bounces@ml01.01.org] On Behalf Of Robert Cleveland
Sent: Thursday, February 18, 2016 10:24 AM
To: spdk@lists.01.org
Subject: [SPDK] SPDK fio benchmark support

Hello all,

Has anyone done any work to make it easy to benchmark polling mode driver with something like FIO?

Thanks,
Robert Cleveland
The information contained in this e-mail is considered confidential of SK hynix memory solutions Inc. and intended only for the persons addressed or copied in this e-mail. Any unauthorized use, dissemination of the information, or copying of this message is strictly prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete the original and any copies of this email.

_______________________________________________
SPDK mailing list
SPDK@lists.01.org<mailto:SPDK@lists.01.org>
https://lists.01.org/mailman/listinfo/spdk

_______________________________________________
SPDK mailing list
SPDK@lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK@lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK@lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK@lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK@lists.01.org
https://lists.01.org/mailman/listinfo/spdk