From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============8689666166168422117==" MIME-Version: 1.0 From: Walker, Benjamin Subject: Re: [SPDK] SPDK fio benchmark support Date: Thu, 16 Jun 2016 15:44:31 +0000 Message-ID: <1466091871.26925.43.camel@intel.com> In-Reply-To: FEB349DD341C264E8FE626BE236D26ED1C48B22F@SSIEXCH-MB3.ssi.samsung.com List-ID: To: spdk@lists.01.org --===============8689666166168422117== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Glad you figured it out. If you find any problems while testing with Samsun= g SSDs, patches are most welcome! On Thu, 2016-06-16 at 01:39 +0000, Raj (Rajinikanth) Pandurangan wrote: > Apparently all 4 SSDs were sharing same PCIe that's why we were not scali= ng.=C2=A0=C2=A0It's been fixed and > able to scale now. > = > Thanks for your help. > -Raj P > = > = > -----Original Message----- > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benj= amin > Sent: Wednesday, June 15, 2016 4:30 PM > To: Storage Performance Development Kit; sbradshaw(a)micron.com; Robert.C= leveland(a)skhms.com > Subject: Re: [SPDK] SPDK fio benchmark support > = > The perf application should scale linearly with CPU cores - so if you get= 1.6 million IOPS with 2 > SSDs on 1 core, you should get 3.2 million IOPS with 4 SSDs on 2 cores. S= ince you aren't, there > must be something else impacting the performance. Can you provide the fol= lowing information: > = > 1) The Xeon E5 SKU you are using > 2) The number of CPU sockets and which socket the devices are attached to= (all 4 need to be > attached to the same socket that you are running perf on). > 3) The command line parameters you are passing to the perf tool. > = > Thanks, > Ben > = > -----Original Message----- > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Raj (Rajinik= anth) Pandurangan > Sent: Wednesday, June 15, 2016 3:16 PM > To: Storage Performance Development Kit ; sbradshaw(= a)micron.com; Robert.Clevelan > d(a)skhms.com > Subject: Re: [SPDK] SPDK fio benchmark support > = > Hello Ben, > = > Thanks for the details. > = > I have Intel Xeon E5 system and able to saturate 2 Samsung NVMe SSDs (~1.= 6 million) on 1 > core.=C2=A0=C2=A0=C2=A0IOPS are same even when I had 4 SSDs on 1 core.=C2= =A0=C2=A0But I expect the IOPS to go up when I > introduce extra cores with 4 SSDs and it did not. > = > Thanks, > -Raj P > = > -----Original Message----- > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benj= amin > Sent: Wednesday, June 15, 2016 1:33 PM > To: Raj (Rajinikanth) Pandurangan; sbradshaw(a)micron.com; spdk(a)lists.0= 1.org; Robert.Cleveland(a)skhms > .com > Subject: Re: [SPDK] SPDK fio benchmark support > = > The fio plugin for SPDK is new and was only tested for a single job. We'd= welcome patches to make > it span more cores when given more jobs if someone is interested in doing= the work here. At some > point, I'm sure we'll get to it ourselves, but it's lower priority than t= he other work we're doing > right now (NVMf target). The perf example we provide definitely does scal= e to many cores though, > and it can optionally use libaio as its backend, so it is a great alterna= tive to using fio that > works today with no extra effort. > = > As far as expected performance scaling across cores with our perf tool, a= re you seeing the maximum > quoted IOPS for all of the device(s) on the system with only a single cor= e? If you are, adding > more cores won't improve performance. If you are not getting the performa= nce you expect from the > hardware I'd love to know the details of the devices and the platform. > = > To provide a quick example from my own system (and none of the following = should be treated as > official benchmarking numbers for any reason), I have a Haswell Xeon E7 w= ith 8 Intel P3700 NVMe > SSDs attached. Running the perf tool on my system yields about 3 million = 4k read IOPs on 1 core, > which is the equivalent of 6 or 7 of my SSDs. Moving to two cores gives m= e the full hardware spec > IOPS for 8 SSDs. If I only test against 4 SSDs I get the same performance= (the maximum that the > SSDs can provide) no matter how many cores I give the tool. > = > Thanks, > Ben > = > -----Original Message----- > From: Raj (Rajinikanth) Pandurangan [mailto:rajini.pandu(a)samsung.com]= =C2=A0 > Sent: Wednesday, June 15, 2016 11:52 AM > To: Walker, Benjamin ; sbradshaw(a)micron.co= m; spdk(a)lists.01.org; Robert. > Cleveland(a)skhms.com > Cc: Raj (Rajinikanth) Pandurangan > Subject: RE: [SPDK] SPDK fio benchmark support > = > Hello All, > = > 1. I tried to run SPDK fio with multiple nvme drives.=C2=A0=C2=A0But for = some reason, I don't see more than > 1 core being used even when I increased numjobs? > = > Here is my config file. > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > = > [global] > ioengine=3D/spdk-master_temp_experiment/examples/nvme/fio_plugin/fio_plug= in > group_reporting=3D1 > direct=3D1 > verify=3D0 > time_based=3D1 > ramp_time=3D0 > runtime=3D10 > = > = > [test1] > iodepth=3D128 > rw=3Drandread > bs=3D4k > #filename=3D0000.09.00.0/1:0000.07.00.0/1:0000.0a.00.0/1 > filename=3D0000.09.00.0/1 > numjobs=3D2 > = > [test2] > iodepth=3D128 > rw=3Drandread > bs=3D4k > numjobs=3D1 > filename=3D0000.08.00.0/1 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > = > Has anyone succeeded using multiple cores? > = > 2. I have also tried with 'perf' and I noticed that even though multiple = cores were being used but > performance numbers didn't scale.=C2=A0=C2=A0Seemed like similar IOPS whe= n I used single core. > = > Any insights would really be appreciated. > = > Thanks, > -Raj P > = > -----Original Message----- > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benj= amin > Sent: Friday, February 26, 2016 2:30 PM > To: sbradshaw(a)micron.com; spdk(a)ml01.01.org; Robert.Cleveland(a)skhms.= com > Subject: Re: [SPDK] SPDK fio benchmark support > = > As a follow up to this, I pushed some improvements to SPDK's setup script= s so that we now more > easily support vfio. That makes it a lot easier to switch back and forth = between the unvme driver > and the SPDK driver. We'd happily accept more patches to improve vfio if = anyone has suggestions. > = > Thanks all. > = > On Thu, 2016-02-18 at 18:59 +0000, Sam Bradshaw (sbradshaw) wrote: > Hi Robert, > = > We built a fio ioengine plugin for benchmarking SPDK as well as our own u= serspace NVMe > driver.=C2=A0=C2=A0Source and documentation are available here:=C2=A0=C2= =A0https://github.com/MicronSSD/unvme > = > (SPDK fio plugin is ioengine/spdk_fio.c) > = > If you have any questions on how to use it or interpret benchmark results= , feel free to ask. > = > -Sam > = > From: SPDK [mailto:spdk-bounces(a)ml01.01.org] On Behalf Of Robert Clevel= and > Sent: Thursday, February 18, 2016 10:24 AM > To: spdk(a)lists.01.org > Subject: [SPDK] SPDK fio benchmark support > = > Hello all, > = > Has anyone done any work to make it easy to benchmark polling mode driver= with something like FIO? > = > Thanks, > Robert Cleveland > The information contained in this e-mail is considered confidential of SK= hynix memory solutions > Inc. and intended only for the persons addressed or copied in this e-mail= . Any unauthorized use, > dissemination of the information, or copying of this message is strictly = prohibited. If you are > not the intended recipient, please contact the sender immediately and per= manently delete the > original and any copies of this email. > = > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > = > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk --===============8689666166168422117==--