From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============8553764618605629319==" MIME-Version: 1.0 From: Walker, Benjamin Subject: =?utf-8?q?Re=3A_=5BSPDK=5D_=E7=AD=94=E5=A4=8D=3A_=E7=AD=94=E5=A4=8D=3A_SP?= =?utf-8?q?DK_fio_benchmark_support?= Date: Mon, 20 Jun 2016 16:34:55 +0000 Message-ID: <1466440495.26925.133.camel@intel.com> In-Reply-To: CANvN+e==J_4tcPzqLpSJMvkLcmDaU2DE-sJsHDhUZqZVoAa5+w@mail.gmail.com List-ID: To: spdk@lists.01.org --===============8553764618605629319== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Sat, 2016-06-18 at 22:09 +0300, Andrey Kuzmin wrote: > On Fri, Jun 17, 2016 at 7:59 PM, Walker, Benjamin wrote: > > On Fri, 2016-06-17 at 00:57 +0000, Li Tianxiang wrote: > > > Thx, > = > > > > If there is any general benchmarks such as fio, the result could be mor= e=C2=A0 > > convincible. > > And=C2=A0the mentioned fio plug-in needs to add multi jobs support. > = > Is there some inherent reason that fio is a more convincing test than our= perf tool?=C2=A0 > Because fio has de-facto became an industry-standard benchmark=C2=A0in th= e recent years, and thus is > definitely more=C2=A0convincing than any single-vendor tool. Furthermore,= in flexibility/coverage fio > delivers what one would expect from a full-fledged storage benchmark, whi= le perf is essentially an > spdk stress tool (no offense meant). > = > > I'm typically > > convinced by benchmarks that are open and easy to understand. Our perf = tool is ~1100 lines of > > code > > and fio contains ~71k. I'm a big fan of fio as a benchmarking tool and = I recommend its use, but > > isn't it much easier to audit exactly what our perf tool is doing to bu= ild trust? > > = > > At some point we'll definitely add multi-core support to our fio plugin= though,=C2=A0 > While debatable with a single device where one can argue, as you do above= , that single thread > suffices (although my mileage is different), it's clearly a must for mult= iple device case. > =C2=A0 > > and if someone > > volunteers to do it sooner we'd happily accept the patch. > > = > > > > > > ----- > > > It could be interesting if=C2=A0NVMe CMD sets could manage the mapping > > > between Qpair and real parallelism. > > = > > If by "real parallelism" you mean actual NAND channels, it's impossible= to expose those directly > > and > > still present a block interface.=C2=A0 > Non-FTL SSDs do expose the internal SSD composition and thus allow the in= terested/capable > application to fully control/exploit the internal SSD parallelism. I encourage you or other interested parties to consider implementing the Li= ghtNVM commands as a header file (at include/spdk/lightnvm.h), doing any minor enabling work so = that SPDK understands alternate command sets (may be no work at all), and submitting a patch back= to us. That would make SPDK a great platform for FTL research and we'd support that effort. The cu= rrent team that works on SPDK is focused on production deployments of block devices though, so the o= nly avenue I see right now for this to get done is through community contributions. > = > Regards, > Andrey > = > =C2=A0 > > The rewrite rules are totally different. > > = > > > ----- > > > > > > > > > =E5=8F=91=E4=BB=B6=E4=BA=BA: SPDK =E4= =BB=A3=E8=A1=A8 Walker, Benjamin > > > =E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4: 2016=E5=B9=B46=E6=9C=8816=E6=97= =A5 15:40 > > > =E6=94=B6=E4=BB=B6=E4=BA=BA: spdk(a)lists.01.org > > > =E4=B8=BB=E9=A2=98: Re: [SPDK] =E7=AD=94=E5=A4=8D: SPDK fio benchmark= support > > > =C2=A0 > > > The perf tool allocates 1 queue pair per thread (worker_fn is the thr= ead entry point in the > > code) > > > and there is 1 thread per core. Our findings on all of the devices we= 've tested has been that > > > using multiple queue pairs does not increase the total performance of= the SSD. The full quoted > > SSD > > > hardware performance can be achieved with a single queue pair. This f= inding, however, is > > > completely device specific. Different SSD controllers may have differ= ent behavior. > > > > > > Using multiple queue pairs is a big advantage for the host software (= i.e. SPDK) though. Using > > 1 > > > queue pair per thread allows the host software to submit I/O in paral= lel with no locks. > > > > > > On Thu, 2016-06-16 at 14:24 +0000, Li Tianxiang wrote: > > > > Hi=EF=BC=8C > > > > > > > > I noticed that the perf=C2=A0=C2=A0app.=C2=A0=C2=A0only=C2=A0=C2=A0= allocates one qpair. > > > > > > > > using :=C2=A0 > > > > "ns_ctx->u.nvme.qpair =3D spdk_nvme_ctrlr_alloc_io_qpair(ns_ctx->en= try->u.nvme.ctrlr, 0);" > > > > > > > > Does more=C2=A0=C2=A0qpair help to the=C2=A0=C2=A0performance? > > > > > > > > And P3700=C2=A0=C2=A0has about 30 submission=C2=A0=C2=A0queues defa= ult. > > > > > > > > > > > > ________________________________________ > > > > =E5=8F=91=E4=BB=B6=E4=BA=BA: SPDK =E4= =BB=A3=E8=A1=A8 Walker, Benjamin > > > > =E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4: 2016=E5=B9=B46=E6=9C=8815=E6= =97=A5 23:29 > > > > =E6=94=B6=E4=BB=B6=E4=BA=BA: Storage Performance Development Kit; s= bradshaw(a)micron.com; Robert.Cleveland(a)skhms.com > > > > =E4=B8=BB=E9=A2=98: Re: [SPDK] SPDK fio benchmark support > > > > > > > > The perf application should scale linearly with CPU cores - so if y= ou get 1.6 million IOPS > > with > > > > 2 SSDs on 1 core, you should get 3.2 million IOPS with 4 SSDs on 2 = cores. Since you aren't, > > > > there must be something else impacting the performance. Can you pro= vide the following > > > > information: > > > > > > > > 1) The Xeon E5 SKU you are using > > > > 2) The number of CPU sockets and which socket the devices are attac= hed to (all 4 need to be > > > > attached to the same socket that you are running perf on). > > > > 3) The command line parameters you are passing to the perf tool. > > > > > > > > Thanks, > > > > Ben > > > > > > > > -----Original Message----- > > > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Raj (R= ajinikanth) Pandurangan > > > > Sent: Wednesday, June 15, 2016 3:16 PM > > > > To: Storage Performance Development Kit ; sbra= dshaw(a)micron.com; > > Robert.Clevel > > > > and(a)skhms.com > > > > Subject: Re: [SPDK] SPDK fio benchmark support > > > > > > > > Hello Ben, > > > > > > > > Thanks for the details. > > > > > > > > I have Intel Xeon E5 system and able to saturate 2 Samsung NVMe SSD= s (~1.6 million) on 1 > > > > core.=C2=A0=C2=A0=C2=A0IOPS are same even when I had 4 SSDs on 1 co= re.=C2=A0=C2=A0But I expect the IOPS to go up when > > I > > > > introduce extra cores with 4 SSDs and it did not. > > > > > > > > Thanks, > > > > -Raj P > > > > > > > > -----Original Message----- > > > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker= , Benjamin > > > > Sent: Wednesday, June 15, 2016 1:33 PM > > > > To: Raj (Rajinikanth) Pandurangan; sbradshaw(a)micron.com; spdk(a)l= ists.01.org; Robert.Cleveland > > @skh > > > > ms.com > > > > Subject: Re: [SPDK] SPDK fio benchmark support > > > > > > > > The fio plugin for SPDK is new and was only tested for a single job= . We'd welcome patches to > > > > make it span more cores when given more jobs if someone is interest= ed in doing the work > > here. At > > > > some point, I'm sure we'll get to it ourselves, but it's lower prio= rity than the other work > > > > we're doing right now (NVMf target). The perf example we provide de= finitely does scale to > > many > > > > cores though, and it can optionally use libaio as its backend, so i= t is a great alternative > > to > > > > using fio that works today with no extra effort. > > > > > > > > As far as expected performance scaling across cores with our perf t= ool, are you seeing the > > > > maximum quoted IOPS for all of the device(s) on the system with onl= y a single core? If you > > are, > > > > adding more cores won't improve performance. If you are not getting= the performance you > > expect > > > > from the hardware I'd love to know the details of the devices and t= he platform. > > > > > > > > To provide a quick example from my own system (and none of the foll= owing should be treated > > as > > > > official benchmarking numbers for any reason), I have a Haswell Xeo= n E7 with 8 Intel P3700 > > NVMe > > > > SSDs attached. Running the perf tool on my system yields about 3 mi= llion 4k read IOPs on 1 > > core, > > > > which is the equivalent of 6 or 7 of my SSDs. Moving to two cores g= ives me the full hardware > > > > spec IOPS for 8 SSDs. If I only test against 4 SSDs I get the same = performance (the maximum > > that > > > > the SSDs can provide) no matter how many cores I give the tool. > > > > > > > > Thanks, > > > > Ben > > > > > > > > -----Original Message----- > > > > From: Raj (Rajinikanth) Pandurangan [mailto:rajini.pandu(a)samsung.= com] > > > > Sent: Wednesday, June 15, 2016 11:52 AM > > > > To: Walker, Benjamin ; sbradshaw(a)mic= ron.com; spdk(a)lists.01.org; > > Rober > > > > t.Cleveland(a)skhms.com > > > > Cc: Raj (Rajinikanth) Pandurangan > > > > Subject: RE: [SPDK] SPDK fio benchmark support > > > > > > > > Hello All, > > > > > > > > 1. I tried to run SPDK fio with multiple nvme drives.=C2=A0=C2=A0Bu= t for some reason, I don't see more > > > > than 1 core being used even when I increased numjobs? > > > > > > > > Here is my config file. > > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > > > > > > > [global] > > > > ioengine=3D/spdk-master_temp_experiment/examples/nvme/fio_plugin/fi= o_plugin > > > > group_reporting=3D1 > > > > direct=3D1 > > > > verify=3D0 > > > > time_based=3D1 > > > > ramp_time=3D0 > > > > runtime=3D10 > > > > > > > > > > > > [test1] > > > > iodepth=3D128 > > > > rw=3Drandread > > > > bs=3D4k > > > > #filename=3D0000.09.00.0/1:0000.07.00.0/1:0000.0a.00.0/1 > > > > filename=3D0000.09.00.0/1 > > > > numjobs=3D2 > > > > > > > > [test2] > > > > iodepth=3D128 > > > > rw=3Drandread > > > > bs=3D4k > > > > numjobs=3D1 > > > > filename=3D0000.08.00.0/1 > > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > > > > > > > Has anyone succeeded using multiple cores? > > > > > > > > 2. I have also tried with 'perf' and I noticed that even though mul= tiple cores were being > > used > > > > but performance numbers didn't scale.=C2=A0=C2=A0Seemed like simila= r IOPS when I used single core. > > > > > > > > Any insights would really be appreciated. > > > > > > > > Thanks, > > > > -Raj P > > > > > > > > -----Original Message----- > > > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker= , Benjamin > > > > Sent: Friday, February 26, 2016 2:30 PM > > > > To: sbradshaw(a)micron.com; spdk(a)ml01.01.org; Robert.Cleveland(a)= skhms.com > > > > Subject: Re: [SPDK] SPDK fio benchmark support > > > > > > > > As a follow up to this, I pushed some improvements to SPDK's setup = scripts so that we now > > more > > > > easily support vfio. That makes it a lot easier to switch back and = forth between the unvme > > > > driver and the SPDK driver. We'd happily accept more patches to imp= rove vfio if anyone has > > > > suggestions. > > > > > > > > Thanks all. > > > > > > > > On Thu, 2016-02-18 at 18:59 +0000, Sam Bradshaw (sbradshaw) wrote: > > > > Hi Robert, > > > > > > > > We built a fio ioengine plugin for benchmarking SPDK as well as our= own userspace NVMe > > > > driver.=C2=A0=C2=A0Source and documentation are available here:=C2= =A0=C2=A0https://github.com/MicronSSD/unvme > > > > > > > > (SPDK fio plugin is ioengine/spdk_fio.c) > > > > > > > > If you have any questions on how to use it or interpret benchmark r= esults, feel free to ask. > > > > > > > > -Sam > > > > > > > > From: SPDK [mailto:spdk-bounces(a)ml01.01.org] On Behalf Of Robert = Cleveland > > > > Sent: Thursday, February 18, 2016 10:24 AM > > > > To: spdk(a)lists.01.org > > > > Subject: [SPDK] SPDK fio benchmark support > > > > > > > > Hello all, > > > > > > > > Has anyone done any work to make it easy to benchmark polling mode = driver with something > > like > > > > FIO? > > > > > > > > Thanks, > > > > Robert Cleveland > > > > The information contained in this e-mail is considered confidential= of SK hynix memory > > solutions > > > > Inc. and intended only for the persons addressed or copied in this = e-mail. Any unauthorized > > use, > > > > dissemination of the information, or copying of this message is str= ictly prohibited. If you > > are > > > > not the intended recipient, please contact the sender immediately a= nd permanently delete the > > > > original and any copies of this email. > > > > > > > > _______________________________________________ > > > > SPDK mailing list > > > > SPDK(a)lists.01.org > > > > https://lists.01.org/mailman/listinfo/spdk > > > > > > > > _______________________________________________ > > > > SPDK mailing list > > > > SPDK(a)lists.01.org > > > > https://lists.01.org/mailman/listinfo/spdk > > > > _______________________________________________ > > > > SPDK mailing list > > > > SPDK(a)lists.01.org > > > > https://lists.01.org/mailman/listinfo/spdk > > > > _______________________________________________ > > > > SPDK mailing list > > > > SPDK(a)lists.01.org > > > > https://lists.01.org/mailman/listinfo/spdk > > > > _______________________________________________ > > > > SPDK mailing list > > > > SPDK(a)lists.01.org > > > > https://lists.01.org/mailman/listinfo/spdk > > > > _______________________________________________ > > > > SPDK mailing list > > > > SPDK(a)lists.01.org > > > > https://lists.01.org/mailman/listinfo/spdk > > > _______________________________________________ > > > SPDK mailing list > > > SPDK(a)lists.01.org > > > https://lists.01.org/mailman/listinfo/spdk > > _______________________________________________ > > SPDK mailing list > > SPDK(a)lists.01.org > > https://lists.01.org/mailman/listinfo/spdk > >=20 --===============8553764618605629319==--