From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============5647221763339449234==" MIME-Version: 1.0 From: =?utf-8?q?=D0=9C=D0=B0=D1=80=D0=BE=D0=B2_=D0=90=D0=BB=D0=B5=D0=BA=D1=81?= =?utf-8?q?=D0=B5=D0=B9_=3Calekseymmm_at_mail=2Eru=3E?= Subject: Re: [SPDK] FIO NVMe Performance Results Date: Mon, 05 Nov 2018 11:45:29 +0300 Message-ID: <1541407529.940345495@f379.i.mail.ru> In-Reply-To: F009CE4E1CB4E047B169243B6A3189273C119F88@SHSMSX101.ccr.corp.intel.com List-ID: To: spdk@lists.01.org --===============5647221763339449234== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Ok, Do you=C2=A0 see some performance issues using general bdev fio plugin = with you configuration ? Best Regards Alex >=D0=9F=D0=BE=D0=BD=D0=B5=D0=B4=D0=B5=D0=BB=D1=8C=D0=BD=D0=B8=D0=BA, 5 =D0= =BD=D0=BE=D1=8F=D0=B1=D1=80=D1=8F 2018, 11:39 +03:00 =D0=BE=D1=82 "Cao, Gan= g" : > >Yes. This is for the bdev version of SPDK FIO plugin and you can assign th= e conf file as the parameter in the FIO configuration. > >Previously, it's another SPDK FIO plugin for the user space NVMe driver on= ly. > >These are two different FIO plugins. > >Thanks, >Gang > >-----Original Message----- >From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of =D0=9C=D0=B0= =D1=80=D0=BE=D0=B2 =D0=90=D0=BB=D0=B5=D0=BA=D1=81=D0=B5=D0=B9 >Sent: Monday, November 5, 2018 4:32 PM >To: Storage Performance Development Kit < spdk(a)lists.01.org > >Subject: Re: [SPDK] FIO NVMe Performance Results > >Sorry for incomplete message I have sent before > >Hello Gang. > >I just curious, why would you run fio with SPDK using this conf file, with= is a bit different from what is=C2=A0 suggested in here=C2=A0 https://git= hub.com/spdk/spdk/tree/master/examples/bdev/fio_plugin =C2=A0. > >I mean may be you should create conf file for spdk like this one : > >nvme.conf : >[Nvme] >=C2=A0 TransportId "trtype:PCIe traddr:0000.1c.00.0" Nvme0 > >and provide it in fio config. > >fio.ini : >[global] >ioengine=3Dspdk_bdev >spdk_conf=3D....../nvme.conf >rw=3Drandrw >rwmixread=3D100 >numjobs=3D4 >iodepth=3D32 >bs=3D4k >direct=3D1 >thread=3D1 >time_based=3D1 >ramp_time=3D0 >runtime=3D10 >group_reporting=3D1 >unified_rw_reporting=3D1 >exitall=3D1 >randrepeat=3D0 >norandommap=3D1 > >[0000:06:00.0] >filename=3DNvme0n1 > >and run like this > >#=C2=A0LD_PRELOAD=3Dexamples/bdev/fio_plugin/fio_plugin fio examples/bdev/= fio_plugin/fio.ini > >Just to note. The recommended version of fio for SPDK is 3.3 > >Best Regards=C2=A0 >Alex > > >>=D0=9F=D0=BE=D0=BD=D0=B5=D0=B4=D0=B5=D0=BB=D1=8C=D0=BD=D0=B8=D0=BA, 5 = =D0=BD=D0=BE=D1=8F=D0=B1=D1=80=D1=8F 2018, 9:53 +03:00 =D0=BE=D1=82 "Cao, G= ang" < gang.cao(a)intel.com >: >> >>Thanks. This is interesting especially for your finding on queue depth 1 = case. We are trying to reproduce this issue locally same as your configurat= ion. >> >>Thanks, >>Gang >> >>-----Original Message----- >>From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Gruher, Jose= ph R >>Sent: Monday, November 5, 2018 1:32 PM >>To: Storage Performance Development Kit < spdk(a)lists.01.org > >>Subject: Re: [SPDK] FIO NVMe Performance Results >> >>Interesting catch. It seems like FIO is actually not accurately reportin= g the number of jobs in the report out for the SPDK case. If I disable the= group_reporting parameter I do get a printout for four separate jobs in th= e SPDK case, so I think it is really running four jobs there, and just summ= arizing incorrectly. I also see four separate CPU cores go to 100% when I = run that FIO test case, indicating four jobs are active. >> >>Also, I don't think would explain the difference in performance. Using a= n aggregate queue depth of 32 versus 128 shouldn't lead to a drop in perfor= mance from 377K to 13K. = >> >>I retested SPDK with multiple separate job descriptions as you suggested = anyway, using four separate jobs instead of relying on the numjobs=3D4 para= meter, and the results are below. In that SPDK test with four separate FIO= job statements, each with queue depth 32, I still only achieved 12.5K IOPS= . = >> >>I then re-ran the kernel NVMe driver test case with only a single job for= comparison, and even with only 1 job and queue depth 32 it still achieves = 254K IOPS. >> >>We might be on the right track with a problem around queue depth however.= If I run the kernel NVMe driver case with just 1 job and queue depth of o= nly 1, I then get 11.2K IOPS, which is around the performance I get in the = SPDK test case. Perhaps the SPDK case is not actually queuing IOs on the d= evice for some reason? >> >> >>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#' >>[global] >>rw=3Drandrw >>rwmixread=3D100 >>numjobs=3D1 >>iodepth=3D32 >>bs=3D4k >>direct=3D1 >>thread=3D1 >>time_based=3D1 >>ramp_time=3D0 >>runtime=3D10 >>ioengine=3Dlibaio >>group_reporting=3D1 >>unified_rw_reporting=3D1 >>exitall=3D1 >>randrepeat=3D0 >>norandommap=3D1 >>[nvme2n1] >>filename=3D/dev/nvme2n1 >> >> >>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini >>nvme2n1: (g=3D0): rw=3Drandrw, bs=3D(R) 4096B-4096B, (W) 4096B-4096B, (T)= 4096B-4096B, ioengine=3Dlibaio, iodepth=3D32 >>fio-3.11 >>Starting 1 thread >>Jobs: 1 (f=3D1): [r(1)][100.0%][r=3D1025MiB/s][r=3D262k IOPS][eta 00m:00s] >>nvme2n1: (groupid=3D0, jobs=3D1): err=3D 0: pid=3D2422: Mon Nov 5 05:19:= 13 2018 >>=C2=A0=C2=A0mixed: IOPS=3D254k, BW=3D992MiB/s (1040MB/s)(9922MiB/10001mse= c) >>=C2=A0=C2=A0=C2=A0=C2=A0slat (nsec): min=3D1185, max=3D319052, avg=3D1611= .64, stdev=3D808.40 >>=C2=A0=C2=A0=C2=A0=C2=A0clat (usec): min=3D2, max=3D4743, avg=3D124.02, s= tdev=3D121.00 >>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0lat (usec): min=3D6, max=3D4744, avg=3D125.= 68, stdev=3D121.02 >>=C2=A0=C2=A0=C2=A0=C2=A0clat percentiles (usec): >>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0| 1.00th=3D[ 9], 5.00th=3D[ 56], 10.= 00th=3D[ 61], 20.00th=3D[ 73], >>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0| 30.00th=3D[ 80], 40.00th=3D[ 91], 50.= 00th=3D[ 110], 60.00th=3D[ 118], >>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0| 70.00th=3D[ 130], 80.00th=3D[ 157], 90.= 00th=3D[ 202], 95.00th=3D[ 253], >>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0| 99.00th=3D[ 408], 99.50th=3D[ 498], 99.= 90th=3D[ 1795], 99.95th=3D[ 3064], >>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0| 99.99th=3D[ 3785] >>=C2=A0=C2=A0=C2=A0bw ( KiB/s): min=3D612240, max=3D1078792, per=3D100.00= %, avg=3D1015996.40, stdev=3D106032.96, samples=3D20 >>=C2=A0=C2=A0=C2=A0iops : min=3D153060, max=3D269698, avg=3D253998.= 90, stdev=3D26508.40, samples=3D20 >>=C2=A0=C2=A0lat (usec) : 4=3D0.01%, 10=3D1.47%, 20=3D1.19%, 50=3D0.17%,= 100=3D42.03% >>=C2=A0=C2=A0lat (usec) : 250=3D49.95%, 500=3D4.70%, 750=3D0.28%, 1000= =3D0.06% >>=C2=A0=C2=A0lat (msec) : 2=3D0.06%, 4=3D0.09%, 10=3D0.01% >>=C2=A0=C2=A0cpu : usr=3D21.45%, sys=3D51.11%, ctx=3D530348, majf= =3D0, minf=3D139 >>=C2=A0=C2=A0IO depths : 1=3D0.1%, 2=3D0.1%, 4=3D0.1%, 8=3D0.1%, 16=3D0= .1%, 32=3D100.0%, >=3D64=3D0.0% >>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0submit : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%,= 16=3D0.0%, 32=3D0.0%, 64=3D0.0%, >=3D64=3D0.0% >>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0complete : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%,= 16=3D0.0%, 32=3D0.1%, 64=3D0.0%, >=3D64=3D0.0% >>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0issued rwts: total=3D2540023,0,0,0 short=3D= 0,0,0,0 dropped=3D0,0,0,0 >>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0latency : target=3D0, window=3D0, percent= ile=3D100.00%, depth=3D32 >>Run status group 0 (all jobs): >>=C2=A0=C2=A0MIXED: bw=3D992MiB/s (1040MB/s), 992MiB/s-992MiB/s (1040MB/s-= 1040MB/s), io=3D9922MiB (10.4GB), run=3D10001-10001msec Disk stats (read/wr= ite): >>=C2=A0=C2=A0nvme2n1: ios=3D2514356/0, merge=3D0/0, ticks=3D400/0, in_queu= e=3D297612, util=3D99.06% >> >> >>don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#' >>[global] >>rw=3Drandrw >>rwmixread=3D100 >>numjobs=3D1 >>iodepth=3D32 >>bs=3D4k >>direct=3D1 >>thread=3D1 >>time_based=3D1 >>ramp_time=3D0 >>runtime=3D10 >>ioengine=3D/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_plugin >>group_reporting=3D1 >>unified_rw_reporting=3D1 >>exitall=3D1 >>randrepeat=3D0 >>norandommap=3D1 >>[0000.1c.00.0-1] >>filename=3Dtrtype=3DPCIe traddr=3D0000.1c.00.0 ns=3D1 [0000.1c.00.0-2] fi= lename=3Dtrtype=3DPCIe traddr=3D0000.1c.00.0 ns=3D1 [0000.1c.00.0-3] filena= me=3Dtrtype=3DPCIe traddr=3D0000.1c.00.0 ns=3D1 [0000.1c.00.0-4] filename= =3Dtrtype=3DPCIe traddr=3D0000.1c.00.0 ns=3D1 >> >> >>don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini >>0000.1c.00.0-1: (g=3D0): rw=3Drandrw, bs=3D(R) 4096B-4096B, (W) 4096B-409= 6B, (T) 4096B-4096B, ioengine=3Dspdk, iodepth=3D32 >>0000.1c.00.0-2: (g=3D0): rw=3Drandrw, bs=3D(R) 4096B-4096B, (W) 4096B-409= 6B, (T) 4096B-4096B, ioengine=3Dspdk, iodepth=3D32 >>0000.1c.00.0-3: (g=3D0): rw=3Drandrw, bs=3D(R) 4096B-4096B, (W) 4096B-409= 6B, (T) 4096B-4096B, ioengine=3Dspdk, iodepth=3D32 >>0000.1c.00.0-4: (g=3D0): rw=3Drandrw, bs=3D(R) 4096B-4096B, (W) 4096B-409= 6B, (T) 4096B-4096B, ioengine=3Dspdk, iodepth=3D32 >>fio-3.11 >>Starting 4 threads >>Starting SPDK v18.10 / DPDK 18.08.0 initialization... >>[ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512 --file-prefix=3Dspdk= _pid4270 ] >>EAL: Detected 36 lcore(s) >>EAL: Detected 2 NUMA nodes >>EAL: No free hugepages reported in hugepages-1048576kB >>EAL: Probing VFIO support... >>EAL: PCI device 0000:1c:00.0 on NUMA socket 0 >>EAL: probe driver: 8086:a54 spdk_nvme >>Jobs: 1 (f=3D0): [_(3),f(1)][100.0%][eta 00m:00s] >>0000.1c.00.0-1: (groupid=3D0, jobs=3D4): err=3D 0: pid=3D4310: Mon Nov 5= 05:16:03 2018 >>=C2=A0=C2=A0mixed: IOPS=3D12.5k, BW=3D48.0MiB/s (51.4MB/s)(490MiB/10010ms= ec) >>=C2=A0=C2=A0=C2=A0=C2=A0slat (nsec): min=3D98, max=3D15092, avg=3D120.02,= stdev=3D90.07 >>=C2=A0=C2=A0=C2=A0=C2=A0clat (usec): min=3D55, max=3D19757, avg=3D10207.2= 2, stdev=3D884.06 >>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0lat (usec): min=3D60, max=3D19757, avg=3D10= 207.34, stdev=3D884.06 >>=C2=A0=C2=A0=C2=A0=C2=A0clat percentiles (usec): >>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0| 1.00th=3D[ 8979], 5.00th=3D[ 9110], 10.= 00th=3D[ 9372], 20.00th=3D[ 9503], >>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0| 30.00th=3D[ 9765], 40.00th=3D[ 9896], 50.= 00th=3D[10159], 60.00th=3D[10290], >>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0| 70.00th=3D[10552], 80.00th=3D[10814], 90.= 00th=3D[11207], 95.00th=3D[11600], >>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0| 99.00th=3D[12911], 99.50th=3D[14353], 99.= 90th=3D[17695], 99.95th=3D[18744], >>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0| 99.99th=3D[19006] >>=C2=A0=C2=A0=C2=A0bw ( KiB/s): min=3D12008, max=3D12976, per=3D25.00%, a= vg=3D12535.70, stdev=3D306.81, samples=3D80 >>=C2=A0=C2=A0=C2=A0iops : min=3D 3002, max=3D 3244, avg=3D3133.85, = stdev=3D76.77, samples=3D80 >>=C2=A0=C2=A0lat (usec) : 100=3D0.01%, 250=3D0.01%, 500=3D0.01%, 750=3D0= .01%, 1000=3D0.01% >>=C2=A0=C2=A0lat (msec) : 2=3D0.01%, 4=3D0.02%, 10=3D45.13%, 20=3D54.83% >>=C2=A0=C2=A0cpu : usr=3D100.01%, sys=3D0.00%, ctx=3D27, majf=3D0= , minf=3D0 >>=C2=A0=C2=A0IO depths : 1=3D0.1%, 2=3D0.1%, 4=3D0.1%, 8=3D0.1%, 16=3D0= .1%, 32=3D99.9%, >=3D64=3D0.0% >>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0submit : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%,= 16=3D0.0%, 32=3D0.0%, 64=3D0.0%, >=3D64=3D0.0% >>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0complete : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%,= 16=3D0.0%, 32=3D0.1%, 64=3D0.0%, >=3D64=3D0.0% >>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0issued rwts: total=3D125506,0,0,0 short=3D0= ,0,0,0 dropped=3D0,0,0,0 >>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0latency : target=3D0, window=3D0, percent= ile=3D100.00%, depth=3D32 >>Run status group 0 (all jobs): >>=C2=A0=C2=A0MIXED: bw=3D48.0MiB/s (51.4MB/s), 48.0MiB/s-48.0MiB/s (51.4MB= /s-51.4MB/s), io=3D490MiB (514MB), run=3D10010-10010msec >> >> >>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#' >>[global] >>rw=3Drandrw >>rwmixread=3D100 >>numjobs=3D1 >>iodepth=3D1 >>bs=3D4k >>direct=3D1 >>thread=3D1 >>time_based=3D1 >>ramp_time=3D0 >>runtime=3D10 >>ioengine=3Dlibaio >>group_reporting=3D1 >>unified_rw_reporting=3D1 >>exitall=3D1 >>randrepeat=3D0 >>norandommap=3D1 >>[nvme2n1] >>filename=3D/dev/nvme2n1 >> >> >>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini >>nvme2n1: (g=3D0): rw=3Drandrw, bs=3D(R) 4096B-4096B, (W) 4096B-4096B, (T)= 4096B-4096B, ioengine=3Dlibaio, iodepth=3D1 >>fio-3.11 >>Starting 1 thread >>Jobs: 1 (f=3D1): [r(1)][100.0%][r=3D43.6MiB/s][r=3D11.2k IOPS][eta 00m:00= s] >>nvme2n1: (groupid=3D0, jobs=3D1): err=3D 0: pid=3D2525: Mon Nov 5 05:22:= 57 2018 >>=C2=A0=C2=A0mixed: IOPS=3D11.2k, BW=3D43.6MiB/s (45.7MB/s)(436MiB/10001ms= ec) >>=C2=A0=C2=A0=C2=A0=C2=A0slat (nsec): min=3D4848, max=3D208595, avg=3D5126= .33, stdev=3D771.27 >>=C2=A0=C2=A0=C2=A0=C2=A0clat (nsec): min=3D1845, max=3D3405.8k, avg=3D830= 04.99, stdev=3D33577.36 >>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0lat (usec): min=3D15, max=3D3411, avg=3D88.= 26, stdev=3D33.59 >>=C2=A0=C2=A0=C2=A0=C2=A0clat percentiles (usec): >>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0| 1.00th=3D[ 15], 5.00th=3D[ 58], 10.= 00th=3D[ 59], 20.00th=3D[ 59], >>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0| 30.00th=3D[ 60], 40.00th=3D[ 77], 50.= 00th=3D[ 77], 60.00th=3D[ 78], >>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0| 70.00th=3D[ 111], 80.00th=3D[ 113], 90.= 00th=3D[ 113], 95.00th=3D[ 115], >>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0| 99.00th=3D[ 186], 99.50th=3D[ 186], 99.= 90th=3D[ 190], 99.95th=3D[ 192], >>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0| 99.99th=3D[ 375] >>=C2=A0=C2=A0=C2=A0bw ( KiB/s): min=3D43816, max=3D45016, per=3D99.98%, a= vg=3D44639.58, stdev=3D312.88, samples=3D19 >>=C2=A0=C2=A0=C2=A0iops : min=3D10954, max=3D11254, avg=3D11159.89,= stdev=3D78.22, samples=3D19 >>=C2=A0=C2=A0lat (usec) : 2=3D0.01%, 20=3D2.83%, 50=3D0.01%, 100=3D64.57= %, 250=3D32.59% >>=C2=A0=C2=A0lat (usec) : 500=3D0.01%, 750=3D0.01% >>=C2=A0=C2=A0lat (msec) : 2=3D0.01%, 4=3D0.01% >>=C2=A0=C2=A0cpu : usr=3D4.02%, sys=3D10.33%, ctx=3D111630, majf= =3D0, minf=3D12 >>=C2=A0=C2=A0IO depths : 1=3D100.0%, 2=3D0.0%, 4=3D0.0%, 8=3D0.0%, 16= =3D0.0%, 32=3D0.0%, >=3D64=3D0.0% >>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0submit : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%,= 16=3D0.0%, 32=3D0.0%, 64=3D0.0%, >=3D64=3D0.0% >>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0complete : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%,= 16=3D0.0%, 32=3D0.0%, 64=3D0.0%, >=3D64=3D0.0% >>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0issued rwts: total=3D111631,0,0,0 short=3D0= ,0,0,0 dropped=3D0,0,0,0 >>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0latency : target=3D0, window=3D0, percent= ile=3D100.00%, depth=3D1 >>Run status group 0 (all jobs): >>=C2=A0=C2=A0MIXED: bw=3D43.6MiB/s (45.7MB/s), 43.6MiB/s-43.6MiB/s (45.7MB= /s-45.7MB/s), io=3D436MiB (457MB), run=3D10001-10001msec Disk stats (read/w= rite): >>=C2=A0=C2=A0nvme2n1: ios=3D110275/0, merge=3D0/0, ticks=3D0/0, in_queue= =3D8596, util=3D86.22% >> >> >>Thanks, >>Joe >> >>> -----Original Message----- >>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang >>> Sent: Sunday, November 4, 2018 8:40 PM >>> To: Storage Performance Development Kit < spdk(a)lists.01.org > >>> Subject: Re: [SPDK] FIO NVMe Performance Results >>> = >>> For the SPDK plugin, from the below result, looks like only 1 job = >>> started with >>> 32 queue depth: >>> = >>> Jobs: 1 (f=3D0): [f(1),_(3)][100.0%][eta 00m:00s] <<<< >>> = >>> Try to update the FIO configuration file as >>> = >>> [job1] >>> filename=3Dtrtype=3DPCIe traddr=3D0000.1c.00.0 ns=3D1 [job2] = >>> filename=3Dtrtype=3DPCIe >>> traddr=3D0000.1c.00.0 ns=3D1 .... >>> = >>> The kernel FIO is with 4 jobs and each job of 32 queue depth. For = >>> SPDK, you can configure as the same, or just configure two jobs and = >>> each job of 64 queue depth. The overall queue depth to the device is sa= me of 128. >>> = >>> Thanks, >>> Gang >>> = >>> -----Original Message----- >>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Gruher, = >>> Joseph R >>> Sent: Monday, November 5, 2018 10:49 AM >>> To: Storage Performance Development Kit < spdk(a)lists.01.org > >>> Subject: [SPDK] FIO NVMe Performance Results >>> = >>> Hi folks- >>> = >>> I'm testing SPDK 18.10 on Ubuntu 18.04 with kernel 4.18.16. I used = >>> FIO with the kernel NVMe driver to measure the performance of a local = >>> (PCIe >>> attached) Intel P4500 NVMe device on a 4KB random read workload and = >>> obtained 477K IOPS, roughly in line with the drive spec. Then I = >>> tested the same drive with the SPDK FIO plugin and only achieved 13K = >>> IOPS. The FIO test files and the results are pasted below. Any ideas = >>> where I'm going wrong here? >>> = >>> Thanks! >>> = >>> = >>> don(a)donst201:~/fio/single/rr$ ls >>> /sys/devices/pci0000:17/0000:17:02.0/0000:1c:00.0/nvme/nvme1/ >>> address cntlid dev device firmware_rev model nvme2n1 power = >>> rescan_controller reset_controller serial state subsysnqn = >>> subsystem transport uevent >>> = >>> = >>> don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#' >>> [global] >>> rw=3Drandrw >>> rwmixread=3D100 >>> numjobs=3D4 >>> iodepth=3D32 >>> bs=3D4k >>> direct=3D1 >>> thread=3D1 >>> time_based=3D1 >>> ramp_time=3D0 >>> runtime=3D10 >>> ioengine=3Dlibaio >>> group_reporting=3D1 >>> unified_rw_reporting=3D1 >>> exitall=3D1 >>> randrepeat=3D0 >>> norandommap=3D1 >>> [nvme2n1] >>> filename=3D/dev/nvme2n1 >>> = >>> = >>> don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini >>> nvme2n1: (g=3D0): rw=3Drandrw, bs=3D(R) 4096B-4096B, (W) 4096B-4096B, (= T) = >>> 4096B-4096B, ioengine=3Dlibaio, iodepth=3D32 ... >>> fio-3.11 >>> Starting 4 threads >>> Jobs: 4 (f=3D4): [r(4)][100.0%][r=3D1874MiB/s][r=3D480k IOPS][eta 00m:0= 0s] >>> nvme2n1: (groupid=3D0, jobs=3D4): err=3D 0: pid=3D2575: Mon Nov 5 02:2= 6:25 2018 >>> mixed: IOPS=3D477k, BW=3D1862MiB/s (1952MB/s)(18.2GiB/10001msec) >>> slat (nsec): min=3D1295, max=3D333088, avg=3D2216.35, stdev=3D927.89 >>> clat (nsec): min=3D506, max=3D5818.8k, avg=3D265924.16, stdev=3D2237= 61.59 >>> lat (usec): min=3D6, max=3D5821, avg=3D268.19, stdev=3D223.76 >>> clat percentiles (usec): >>> | 1.00th=3D[ 13], 5.00th=3D[ 74], 10.00th=3D[ 87], 20.00th= =3D[ 115], >>> | 30.00th=3D[ 139], 40.00th=3D[ 167], 50.00th=3D[ 204], 60.00th= =3D[ 247], >>> | 70.00th=3D[ 306], 80.00th=3D[ 388], 90.00th=3D[ 523], 95.00th= =3D[ 660], >>> | 99.00th=3D[ 988], 99.50th=3D[ 1156], 99.90th=3D[ 2343], 99.95th= =3D[ 3195], >>> | 99.99th=3D[ 4424] >>> bw ( KiB/s): min=3D448192, max=3D483104, per=3D25.00%, avg=3D476597= .10, = >>> stdev=3D6748.78, samples=3D80 >>> iops : min=3D112048, max=3D120776, avg=3D119149.27, stdev=3D1= 687.20, >>> samples=3D80 >>> lat (nsec) : 750=3D0.01% >>> lat (usec) : 10=3D0.52%, 20=3D1.25%, 50=3D0.25%, 100=3D13.38%, 250= =3D45.13% >>> lat (usec) : 500=3D28.23%, 750=3D8.08%, 1000=3D2.22% >>> lat (msec) : 2=3D0.82%, 4=3D0.10%, 10=3D0.02% >>> cpu : usr=3D14.08%, sys=3D36.02%, ctx=3D1930207, majf=3D0, m= inf=3D697 >>> IO depths : 1=3D0.1%, 2=3D0.1%, 4=3D0.1%, 8=3D0.1%, 16=3D0.1%, 32= =3D100.0%, >>> >=3D64=3D0.0% >>> submit : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.0%, = 64=3D0.0%, >>> >=3D64=3D0.0% >>> complete : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.1%, = 64=3D0.0%, >>> >=3D64=3D0.0% >>> issued rwts: total=3D4766095,0,0,0 short=3D0,0,0,0 dropped=3D0,0,0= ,0 >>> latency : target=3D0, window=3D0, percentile=3D100.00%, depth=3D= 32 >>> Run status group 0 (all jobs): >>> MIXED: bw=3D1862MiB/s (1952MB/s), 1862MiB/s-1862MiB/s (1952MB/s- = >>> 1952MB/s), io=3D18.2GiB (19.5GB), run=3D10001-10001msec Disk stats >>> (read/write): >>> nvme2n1: ios=3D4709190/0, merge=3D0/0, ticks=3D4096/0, in_queue=3D138= 8752, = >>> util=3D100.00% >>> = >>> = >>> don(a)donst201:~/fio/single/rr$ sudo >>> /home/don/install/spdk/spdk/scripts/setup.sh >>> Active mountpoints on /dev/nvme0n1, so not binding PCI dev = >>> 0000:03:00.0 >>> 0000:1c:00.0 (8086 0a54): nvme -> uio_pci_generic >>> 0000:1d:00.0 (8086 0a54): nvme -> uio_pci_generic >>> 0000:5e:00.0 (8086 0a54): nvme -> uio_pci_generic >>> 0000:5f:00.0 (8086 0a54): nvme -> uio_pci_generic >>> 0000:62:00.0 (8086 0a54): nvme -> uio_pci_generic >>> 0000:63:00.0 (8086 0a54): nvme -> uio_pci_generic >>> 0000:64:00.0 (8086 0a54): nvme -> uio_pci_generic >>> 0000:65:00.0 (8086 0a54): nvme -> uio_pci_generic >>> 0000:da:00.0 (8086 0a54): nvme -> uio_pci_generic >>> 0000:db:00.0 (8086 0a54): nvme -> uio_pci_generic >>> 0000:dc:00.0 (8086 0a54): nvme -> uio_pci_generic >>> 0000:dd:00.0 (8086 0a54): nvme -> uio_pci_generic >>> 0000:e0:00.0 (8086 0a54): nvme -> uio_pci_generic >>> 0000:e1:00.0 (8086 0a54): nvme -> uio_pci_generic >>> 0000:e2:00.0 (8086 0a54): nvme -> uio_pci_generic >>> 0000:e3:00.0 (8086 0a54): nvme -> uio_pci_generic >>> 0000:00:04.0 (8086 2021): ioatdma -> uio_pci_generic >>> 0000:00:04.1 (8086 2021): ioatdma -> uio_pci_generic >>> 0000:00:04.2 (8086 2021): ioatdma -> uio_pci_generic >>> 0000:00:04.3 (8086 2021): ioatdma -> uio_pci_generic >>> 0000:00:04.4 (8086 2021): ioatdma -> uio_pci_generic >>> 0000:00:04.5 (8086 2021): ioatdma -> uio_pci_generic >>> 0000:00:04.6 (8086 2021): ioatdma -> uio_pci_generic >>> 0000:00:04.7 (8086 2021): ioatdma -> uio_pci_generic >>> 0000:80:04.0 (8086 2021): ioatdma -> uio_pci_generic >>> 0000:80:04.1 (8086 2021): ioatdma -> uio_pci_generic >>> 0000:80:04.2 (8086 2021): ioatdma -> uio_pci_generic >>> 0000:80:04.3 (8086 2021): ioatdma -> uio_pci_generic >>> 0000:80:04.4 (8086 2021): ioatdma -> uio_pci_generic >>> 0000:80:04.5 (8086 2021): ioatdma -> uio_pci_generic >>> 0000:80:04.6 (8086 2021): ioatdma -> uio_pci_generic >>> 0000:80:04.7 (8086 2021): ioatdma -> uio_pci_generic >>> = >>> = >>> don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#' >>> [global] >>> rw=3Drandrw >>> rwmixread=3D100 >>> numjobs=3D4 >>> iodepth=3D32 >>> bs=3D4k >>> direct=3D1 >>> thread=3D1 >>> time_based=3D1 >>> ramp_time=3D0 >>> runtime=3D10 >>> ioengine=3D/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_plug >>> in >>> group_reporting=3D1 >>> unified_rw_reporting=3D1 >>> exitall=3D1 >>> randrepeat=3D0 >>> norandommap=3D1 >>> [0000.1c.00.0] >>> filename=3Dtrtype=3DPCIe traddr=3D0000.1c.00.0 ns=3D1 >>> = >>> = >>> don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini >>> 0000.1c.00.0: (g=3D0): rw=3Drandrw, bs=3D(R) 4096B-4096B, (W) 4096B-409= 6B, = >>> (T) 4096B-4096B, ioengine=3Dspdk, iodepth=3D32 ... >>> fio-3.11 >>> Starting 4 threads >>> Starting SPDK v18.10 / DPDK 18.08.0 initialization... >>> [ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512 --file- >>> prefix=3Dspdk_pid3668 ] >>> EAL: Detected 36 lcore(s) >>> EAL: Detected 2 NUMA nodes >>> EAL: No free hugepages reported in hugepages-1048576kB >>> EAL: Probing VFIO support... >>> EAL: PCI device 0000:1c:00.0 on NUMA socket 0 >>> EAL: probe driver: 8086:a54 spdk_nvme >>> Jobs: 1 (f=3D0): [f(1),_(3)][100.0%][eta 00m:00s] >>> 0000.1c.00.0: (groupid=3D0, jobs=3D4): err=3D 0: pid=3D3709: Mon Nov 5 = >>> 02:28:29 >>> 2018 >>> mixed: IOPS=3D13.2k, BW=3D51.6MiB/s (54.1MB/s)(517MiB/10011msec) >>> slat (nsec): min=3D109, max=3D15344, avg=3D127.91, stdev=3D73.44 >>> clat (usec): min=3D187, max=3D18715, avg=3D9683.80, stdev=3D650.54 >>> lat (usec): min=3D193, max=3D18715, avg=3D9683.93, stdev=3D650.53 >>> clat percentiles (usec): >>> | 1.00th=3D[ 8717], 5.00th=3D[ 8848], 10.00th=3D[ 8979], 20.00th= =3D[ 9110], >>> | 30.00th=3D[ 9241], 40.00th=3D[ 9372], 50.00th=3D[ 9503], 60.00th= =3D[ 9765], >>> | 70.00th=3D[10028], 80.00th=3D[10159], 90.00th=3D[10552], 95.00th= =3D[10945], >>> | 99.00th=3D[11338], 99.50th=3D[11338], 99.90th=3D[11731], 99.95th= =3D[13960], >>> | 99.99th=3D[17695] >>> bw ( KiB/s): min=3D13080, max=3D13304, per=3D25.00%, avg=3D13214.70= , = >>> stdev=3D55.30, samples=3D80 >>> iops : min=3D 3270, max=3D 3326, avg=3D3303.65, stdev=3D13.84= , samples=3D80 >>> lat (usec) : 250=3D0.01%, 500=3D0.01%, 750=3D0.01%, 1000=3D0.01% >>> lat (msec) : 2=3D0.01%, 4=3D0.02%, 10=3D70.85%, 20=3D29.11% >>> cpu : usr=3D100.01%, sys=3D0.00%, ctx=3D23, majf=3D0, minf= =3D0 >>> IO depths : 1=3D0.1%, 2=3D0.1%, 4=3D0.1%, 8=3D0.1%, 16=3D0.1%, 32= =3D99.9%, >>> >=3D64=3D0.0% >>> submit : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.0%, = 64=3D0.0%, >>> >=3D64=3D0.0% >>> complete : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.1%, = 64=3D0.0%, >>> >=3D64=3D0.0% >>> issued rwts: total=3D132300,0,0,0 short=3D0,0,0,0 dropped=3D0,0,0,0 >>> latency : target=3D0, window=3D0, percentile=3D100.00%, depth=3D= 32 >>> Run status group 0 (all jobs): >>> MIXED: bw=3D51.6MiB/s (54.1MB/s), 51.6MiB/s-51.6MiB/s (54.1MB/s- = >>> 54.1MB/s), io=3D517MiB (542MB), run=3D10011-10011msec >>> = >>> _______________________________________________ >>> SPDK mailing list >>> SPDK(a)lists.01.org >>> https://lists.01.org/mailman/listinfo/spdk >>> _______________________________________________ >>> SPDK mailing list >>> SPDK(a)lists.01.org >>> https://lists.01.org/mailman/listinfo/spdk >>_______________________________________________ >>SPDK mailing list >> SPDK(a)lists.01.org >> https://lists.01.org/mailman/listinfo/spdk >>_______________________________________________ >>SPDK mailing list >> SPDK(a)lists.01.org >> https://lists.01.org/mailman/listinfo/spdk > > >-- = >=D0=9C=D0=B0=D1=80=D0=BE=D0=B2 =D0=90=D0=BB=D0=B5=D0=BA=D1=81=D0=B5=D0=B9 >_______________________________________________ >SPDK mailing list >SPDK(a)lists.01.org >https://lists.01.org/mailman/listinfo/spdk >_______________________________________________ >SPDK mailing list >SPDK(a)lists.01.org >https://lists.01.org/mailman/listinfo/spdk -- = =D0=9C=D0=B0=D1=80=D0=BE=D0=B2 =D0=90=D0=BB=D0=B5=D0=BA=D1=81=D0=B5=D0=B9 --===============5647221763339449234==--