Glad you figured it out. If you find any problems while testing with Samsung SSDs, patches are most welcome! On Thu, 2016-06-16 at 01:39 +0000, Raj (Rajinikanth) Pandurangan wrote: > Apparently all 4 SSDs were sharing same PCIe that's why we were not scaling.  It's been fixed and > able to scale now. > > Thanks for your help. > -Raj P > > > -----Original Message----- > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin > Sent: Wednesday, June 15, 2016 4:30 PM > To: Storage Performance Development Kit; sbradshaw(a)micron.com; Robert.Cleveland(a)skhms.com > Subject: Re: [SPDK] SPDK fio benchmark support > > The perf application should scale linearly with CPU cores - so if you get 1.6 million IOPS with 2 > SSDs on 1 core, you should get 3.2 million IOPS with 4 SSDs on 2 cores. Since you aren't, there > must be something else impacting the performance. Can you provide the following information: > > 1) The Xeon E5 SKU you are using > 2) The number of CPU sockets and which socket the devices are attached to (all 4 need to be > attached to the same socket that you are running perf on). > 3) The command line parameters you are passing to the perf tool. > > Thanks, > Ben > > -----Original Message----- > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Raj (Rajinikanth) Pandurangan > Sent: Wednesday, June 15, 2016 3:16 PM > To: Storage Performance Development Kit ; sbradshaw(a)micron.com; Robert.Clevelan > d(a)skhms.com > Subject: Re: [SPDK] SPDK fio benchmark support > > Hello Ben, > > Thanks for the details. > > I have Intel Xeon E5 system and able to saturate 2 Samsung NVMe SSDs (~1.6 million) on 1 > core.   IOPS are same even when I had 4 SSDs on 1 core.  But I expect the IOPS to go up when I > introduce extra cores with 4 SSDs and it did not. > > Thanks, > -Raj P > > -----Original Message----- > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin > Sent: Wednesday, June 15, 2016 1:33 PM > To: Raj (Rajinikanth) Pandurangan; sbradshaw(a)micron.com; spdk(a)lists.01.org; Robert.Cleveland(a)skhms > .com > Subject: Re: [SPDK] SPDK fio benchmark support > > The fio plugin for SPDK is new and was only tested for a single job. We'd welcome patches to make > it span more cores when given more jobs if someone is interested in doing the work here. At some > point, I'm sure we'll get to it ourselves, but it's lower priority than the other work we're doing > right now (NVMf target). The perf example we provide definitely does scale to many cores though, > and it can optionally use libaio as its backend, so it is a great alternative to using fio that > works today with no extra effort. > > As far as expected performance scaling across cores with our perf tool, are you seeing the maximum > quoted IOPS for all of the device(s) on the system with only a single core? If you are, adding > more cores won't improve performance. If you are not getting the performance you expect from the > hardware I'd love to know the details of the devices and the platform. > > To provide a quick example from my own system (and none of the following should be treated as > official benchmarking numbers for any reason), I have a Haswell Xeon E7 with 8 Intel P3700 NVMe > SSDs attached. Running the perf tool on my system yields about 3 million 4k read IOPs on 1 core, > which is the equivalent of 6 or 7 of my SSDs. Moving to two cores gives me the full hardware spec > IOPS for 8 SSDs. If I only test against 4 SSDs I get the same performance (the maximum that the > SSDs can provide) no matter how many cores I give the tool. > > Thanks, > Ben > > -----Original Message----- > From: Raj (Rajinikanth) Pandurangan [mailto:rajini.pandu(a)samsung.com]  > Sent: Wednesday, June 15, 2016 11:52 AM > To: Walker, Benjamin ; sbradshaw(a)micron.com; spdk(a)lists.01.org; Robert. > Cleveland(a)skhms.com > Cc: Raj (Rajinikanth) Pandurangan > Subject: RE: [SPDK] SPDK fio benchmark support > > Hello All, > > 1. I tried to run SPDK fio with multiple nvme drives.  But for some reason, I don't see more than > 1 core being used even when I increased numjobs? > > Here is my config file. > ============================================================= > > [global] > ioengine=/spdk-master_temp_experiment/examples/nvme/fio_plugin/fio_plugin > group_reporting=1 > direct=1 > verify=0 > time_based=1 > ramp_time=0 > runtime=10 > > > [test1] > iodepth=128 > rw=randread > bs=4k > #filename=0000.09.00.0/1:0000.07.00.0/1:0000.0a.00.0/1 > filename=0000.09.00.0/1 > numjobs=2 > > [test2] > iodepth=128 > rw=randread > bs=4k > numjobs=1 > filename=0000.08.00.0/1 > ================================================================ > > Has anyone succeeded using multiple cores? > > 2. I have also tried with 'perf' and I noticed that even though multiple cores were being used but > performance numbers didn't scale.  Seemed like similar IOPS when I used single core. > > Any insights would really be appreciated. > > Thanks, > -Raj P > > -----Original Message----- > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin > Sent: Friday, February 26, 2016 2:30 PM > To: sbradshaw(a)micron.com; spdk(a)ml01.01.org; Robert.Cleveland(a)skhms.com > Subject: Re: [SPDK] SPDK fio benchmark support > > As a follow up to this, I pushed some improvements to SPDK's setup scripts so that we now more > easily support vfio. That makes it a lot easier to switch back and forth between the unvme driver > and the SPDK driver. We'd happily accept more patches to improve vfio if anyone has suggestions. > > Thanks all. > > On Thu, 2016-02-18 at 18:59 +0000, Sam Bradshaw (sbradshaw) wrote: > Hi Robert, > > We built a fio ioengine plugin for benchmarking SPDK as well as our own userspace NVMe > driver.  Source and documentation are available here:  https://github.com/MicronSSD/unvme > > (SPDK fio plugin is ioengine/spdk_fio.c) > > If you have any questions on how to use it or interpret benchmark results, feel free to ask. > > -Sam > > From: SPDK [mailto:spdk-bounces(a)ml01.01.org] On Behalf Of Robert Cleveland > Sent: Thursday, February 18, 2016 10:24 AM > To: spdk(a)lists.01.org > Subject: [SPDK] SPDK fio benchmark support > > Hello all, > > Has anyone done any work to make it easy to benchmark polling mode driver with something like FIO? > > Thanks, > Robert Cleveland > The information contained in this e-mail is considered confidential of SK hynix memory solutions > Inc. and intended only for the persons addressed or copied in this e-mail. Any unauthorized use, > dissemination of the information, or copying of this message is strictly prohibited. If you are > not the intended recipient, please contact the sender immediately and permanently delete the > original and any copies of this email. > > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk