Re: [SPDK] SPDK fio benchmark support

[Walker, Benjamin <benjamin.walker at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 7968 bytes --]

Glad you figured it out. If you find any problems while testing with Samsung SSDs, patches are most
welcome!

On Thu, 2016-06-16 at 01:39 +0000, Raj (Rajinikanth) Pandurangan wrote:
> Apparently all 4 SSDs were sharing same PCIe that's why we were not scaling.  It's been fixed and
> able to scale now.
> 
> Thanks for your help.
> -Raj P
> 
> 
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
> Sent: Wednesday, June 15, 2016 4:30 PM
> To: Storage Performance Development Kit; sbradshaw(a)micron.com; Robert.Cleveland(a)skhms.com
> Subject: Re: [SPDK] SPDK fio benchmark support
> 
> The perf application should scale linearly with CPU cores - so if you get 1.6 million IOPS with 2
> SSDs on 1 core, you should get 3.2 million IOPS with 4 SSDs on 2 cores. Since you aren't, there
> must be something else impacting the performance. Can you provide the following information:
> 
> 1) The Xeon E5 SKU you are using
> 2) The number of CPU sockets and which socket the devices are attached to (all 4 need to be
> attached to the same socket that you are running perf on).
> 3) The command line parameters you are passing to the perf tool.
> 
> Thanks,
> Ben
> 
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Raj (Rajinikanth) Pandurangan
> Sent: Wednesday, June 15, 2016 3:16 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>; sbradshaw(a)micron.com; Robert.Clevelan
> d(a)skhms.com
> Subject: Re: [SPDK] SPDK fio benchmark support
> 
> Hello Ben,
> 
> Thanks for the details.
> 
> I have Intel Xeon E5 system and able to saturate 2 Samsung NVMe SSDs (~1.6 million) on 1
> core.   IOPS are same even when I had 4 SSDs on 1 core.  But I expect the IOPS to go up when I
> introduce extra cores with 4 SSDs and it did not.
> 
> Thanks,
> -Raj P
> 
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
> Sent: Wednesday, June 15, 2016 1:33 PM
> To: Raj (Rajinikanth) Pandurangan; sbradshaw(a)micron.com; spdk(a)lists.01.org; Robert.Cleveland(a)skhms
> .com
> Subject: Re: [SPDK] SPDK fio benchmark support
> 
> The fio plugin for SPDK is new and was only tested for a single job. We'd welcome patches to make
> it span more cores when given more jobs if someone is interested in doing the work here. At some
> point, I'm sure we'll get to it ourselves, but it's lower priority than the other work we're doing
> right now (NVMf target). The perf example we provide definitely does scale to many cores though,
> and it can optionally use libaio as its backend, so it is a great alternative to using fio that
> works today with no extra effort.
> 
> As far as expected performance scaling across cores with our perf tool, are you seeing the maximum
> quoted IOPS for all of the device(s) on the system with only a single core? If you are, adding
> more cores won't improve performance. If you are not getting the performance you expect from the
> hardware I'd love to know the details of the devices and the platform.
> 
> To provide a quick example from my own system (and none of the following should be treated as
> official benchmarking numbers for any reason), I have a Haswell Xeon E7 with 8 Intel P3700 NVMe
> SSDs attached. Running the perf tool on my system yields about 3 million 4k read IOPs on 1 core,
> which is the equivalent of 6 or 7 of my SSDs. Moving to two cores gives me the full hardware spec
> IOPS for 8 SSDs. If I only test against 4 SSDs I get the same performance (the maximum that the
> SSDs can provide) no matter how many cores I give the tool.
> 
> Thanks,
> Ben
> 
> -----Original Message-----
> From: Raj (Rajinikanth) Pandurangan [mailto:rajini.pandu(a)samsung.com] 
> Sent: Wednesday, June 15, 2016 11:52 AM
> To: Walker, Benjamin <benjamin.walker(a)intel.com>; sbradshaw(a)micron.com; spdk(a)lists.01.org; Robert.
> Cleveland(a)skhms.com
> Cc: Raj (Rajinikanth) Pandurangan <rajini.pandu(a)samsung.com>
> Subject: RE: [SPDK] SPDK fio benchmark support
> 
> Hello All,
> 
> 1. I tried to run SPDK fio with multiple nvme drives.  But for some reason, I don't see more than
> 1 core being used even when I increased numjobs?
> 
> Here is my config file.
> =============================================================
> 
> [global]
> ioengine=/spdk-master_temp_experiment/examples/nvme/fio_plugin/fio_plugin
> group_reporting=1
> direct=1
> verify=0
> time_based=1
> ramp_time=0
> runtime=10
> 
> 
> [test1]
> iodepth=128
> rw=randread
> bs=4k
> #filename=0000.09.00.0/1:0000.07.00.0/1:0000.0a.00.0/1
> filename=0000.09.00.0/1
> numjobs=2
> 
> [test2]
> iodepth=128
> rw=randread
> bs=4k
> numjobs=1
> filename=0000.08.00.0/1
> ================================================================
> 
> Has anyone succeeded using multiple cores?
> 
> 2. I have also tried with 'perf' and I noticed that even though multiple cores were being used but
> performance numbers didn't scale.  Seemed like similar IOPS when I used single core.
> 
> Any insights would really be appreciated.
> 
> Thanks,
> -Raj P
> 
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
> Sent: Friday, February 26, 2016 2:30 PM
> To: sbradshaw(a)micron.com; spdk(a)ml01.01.org; Robert.Cleveland(a)skhms.com
> Subject: Re: [SPDK] SPDK fio benchmark support
> 
> As a follow up to this, I pushed some improvements to SPDK's setup scripts so that we now more
> easily support vfio. That makes it a lot easier to switch back and forth between the unvme driver
> and the SPDK driver. We'd happily accept more patches to improve vfio if anyone has suggestions.
> 
> Thanks all.
> 
> On Thu, 2016-02-18 at 18:59 +0000, Sam Bradshaw (sbradshaw) wrote:
> Hi Robert,
> 
> We built a fio ioengine plugin for benchmarking SPDK as well as our own userspace NVMe
> driver.  Source and documentation are available here:  https://github.com/MicronSSD/unvme
> 
> (SPDK fio plugin is ioengine/spdk_fio.c)
> 
> If you have any questions on how to use it or interpret benchmark results, feel free to ask.
> 
> -Sam
> 
> From: SPDK [mailto:spdk-bounces(a)ml01.01.org] On Behalf Of Robert Cleveland
> Sent: Thursday, February 18, 2016 10:24 AM
> To: spdk(a)lists.01.org
> Subject: [SPDK] SPDK fio benchmark support
> 
> Hello all,
> 
> Has anyone done any work to make it easy to benchmark polling mode driver with something like FIO?
> 
> Thanks,
> Robert Cleveland
> The information contained in this e-mail is considered confidential of SK hynix memory solutions
> Inc. and intended only for the persons addressed or copied in this e-mail. Any unauthorized use,
> dissemination of the information, or copying of this message is strictly prohibited. If you are
> not the intended recipient, please contact the sender immediately and permanently delete the
> original and any copies of this email.
> 
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
> https://lists.01.org/mailman/listinfo/spdk
> 
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk