From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============2901104388327956709==" MIME-Version: 1.0 From: =?utf-8?q?=D0=9C=D0=B0=D1=80=D0=BE=D0=B2_=D0=90=D0=BB=D0=B5=D0=BA=D1=81?= =?utf-8?q?=D0=B5=D0=B9_=3Calekseymmm_at_mail=2Eru=3E?= Subject: Re: [SPDK] FIO NVMe Performance Results Date: Mon, 05 Nov 2018 11:27:48 +0300 Message-ID: <1541406468.898808197@f431.i.mail.ru> In-Reply-To: F009CE4E1CB4E047B169243B6A3189273C119DD0@SHSMSX101.ccr.corp.intel.com List-ID: To: spdk@lists.01.org --===============2901104388327956709== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hello Gang. I just curious, why would you run fio with SPDK using this conf file, with = is a bit different from what is=C2=A0 suggested in here=C2=A0 https://githu= b.com/spdk/spdk/tree/master/examples/bdev/fio_plugin =C2=A0. I mean may be you should create conf file for spdk like this one : nvme.conf : [Nvme] =C2=A0 TransportId "trtype:PCIe traddr:0000.1c.00.0" Nvme0 and provide it in fio config. fio.ini : >=D0=9F=D0=BE=D0=BD=D0=B5=D0=B4=D0=B5=D0=BB=D1=8C=D0=BD=D0=B8=D0=BA, 5 =D0= =BD=D0=BE=D1=8F=D0=B1=D1=80=D1=8F 2018, 9:53 +03:00 =D0=BE=D1=82 "Cao, Gang= " : > >Thanks. This is interesting especially for your finding on queue depth 1 c= ase. We are trying to reproduce this issue locally same as your configurati= on. > >Thanks, >Gang > >-----Original Message----- >From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Gruher, Josep= h R >Sent: Monday, November 5, 2018 1:32 PM >To: Storage Performance Development Kit < spdk(a)lists.01.org > >Subject: Re: [SPDK] FIO NVMe Performance Results > >Interesting catch. It seems like FIO is actually not accurately reporting= the number of jobs in the report out for the SPDK case. If I disable the = group_reporting parameter I do get a printout for four separate jobs in the= SPDK case, so I think it is really running four jobs there, and just summa= rizing incorrectly. I also see four separate CPU cores go to 100% when I r= un that FIO test case, indicating four jobs are active. > >Also, I don't think would explain the difference in performance. Using an= aggregate queue depth of 32 versus 128 shouldn't lead to a drop in perform= ance from 377K to 13K. = > >I retested SPDK with multiple separate job descriptions as you suggested a= nyway, using four separate jobs instead of relying on the numjobs=3D4 param= eter, and the results are below. In that SPDK test with four separate FIO = job statements, each with queue depth 32, I still only achieved 12.5K IOPS. = > >I then re-ran the kernel NVMe driver test case with only a single job for = comparison, and even with only 1 job and queue depth 32 it still achieves 2= 54K IOPS. > >We might be on the right track with a problem around queue depth however. = If I run the kernel NVMe driver case with just 1 job and queue depth of on= ly 1, I then get 11.2K IOPS, which is around the performance I get in the S= PDK test case. Perhaps the SPDK case is not actually queuing IOs on the de= vice for some reason? > > >don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#' >[global] >rw=3Drandrw >rwmixread=3D100 >numjobs=3D1 >iodepth=3D32 >bs=3D4k >direct=3D1 >thread=3D1 >time_based=3D1 >ramp_time=3D0 >runtime=3D10 >ioengine=3Dlibaio >group_reporting=3D1 >unified_rw_reporting=3D1 >exitall=3D1 >randrepeat=3D0 >norandommap=3D1 >[nvme2n1] >filename=3D/dev/nvme2n1 > > >don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini >nvme2n1: (g=3D0): rw=3Drandrw, bs=3D(R) 4096B-4096B, (W) 4096B-4096B, (T) = 4096B-4096B, ioengine=3Dlibaio, iodepth=3D32 >fio-3.11 >Starting 1 thread >Jobs: 1 (f=3D1): [r(1)][100.0%][r=3D1025MiB/s][r=3D262k IOPS][eta 00m:00s] >nvme2n1: (groupid=3D0, jobs=3D1): err=3D 0: pid=3D2422: Mon Nov 5 05:19:1= 3 2018 >=C2=A0=C2=A0mixed: IOPS=3D254k, BW=3D992MiB/s (1040MB/s)(9922MiB/10001msec) >=C2=A0=C2=A0=C2=A0=C2=A0slat (nsec): min=3D1185, max=3D319052, avg=3D1611.= 64, stdev=3D808.40 >=C2=A0=C2=A0=C2=A0=C2=A0clat (usec): min=3D2, max=3D4743, avg=3D124.02, st= dev=3D121.00 >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0lat (usec): min=3D6, max=3D4744, avg=3D125.6= 8, stdev=3D121.02 >=C2=A0=C2=A0=C2=A0=C2=A0clat percentiles (usec): >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0| 1.00th=3D[ 9], 5.00th=3D[ 56], 10.0= 0th=3D[ 61], 20.00th=3D[ 73], >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0| 30.00th=3D[ 80], 40.00th=3D[ 91], 50.0= 0th=3D[ 110], 60.00th=3D[ 118], >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0| 70.00th=3D[ 130], 80.00th=3D[ 157], 90.0= 0th=3D[ 202], 95.00th=3D[ 253], >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0| 99.00th=3D[ 408], 99.50th=3D[ 498], 99.9= 0th=3D[ 1795], 99.95th=3D[ 3064], >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0| 99.99th=3D[ 3785] >=C2=A0=C2=A0=C2=A0bw ( KiB/s): min=3D612240, max=3D1078792, per=3D100.00%= , avg=3D1015996.40, stdev=3D106032.96, samples=3D20 >=C2=A0=C2=A0=C2=A0iops : min=3D153060, max=3D269698, avg=3D253998.9= 0, stdev=3D26508.40, samples=3D20 >=C2=A0=C2=A0lat (usec) : 4=3D0.01%, 10=3D1.47%, 20=3D1.19%, 50=3D0.17%, = 100=3D42.03% >=C2=A0=C2=A0lat (usec) : 250=3D49.95%, 500=3D4.70%, 750=3D0.28%, 1000=3D= 0.06% >=C2=A0=C2=A0lat (msec) : 2=3D0.06%, 4=3D0.09%, 10=3D0.01% >=C2=A0=C2=A0cpu : usr=3D21.45%, sys=3D51.11%, ctx=3D530348, majf= =3D0, minf=3D139 >=C2=A0=C2=A0IO depths : 1=3D0.1%, 2=3D0.1%, 4=3D0.1%, 8=3D0.1%, 16=3D0.= 1%, 32=3D100.0%, >=3D64=3D0.0% >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0submit : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, = 16=3D0.0%, 32=3D0.0%, 64=3D0.0%, >=3D64=3D0.0% >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0complete : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, = 16=3D0.0%, 32=3D0.1%, 64=3D0.0%, >=3D64=3D0.0% >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0issued rwts: total=3D2540023,0,0,0 short=3D0= ,0,0,0 dropped=3D0,0,0,0 >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0latency : target=3D0, window=3D0, percenti= le=3D100.00%, depth=3D32 >Run status group 0 (all jobs): >=C2=A0=C2=A0MIXED: bw=3D992MiB/s (1040MB/s), 992MiB/s-992MiB/s (1040MB/s-1= 040MB/s), io=3D9922MiB (10.4GB), run=3D10001-10001msec Disk stats (read/wri= te): >=C2=A0=C2=A0nvme2n1: ios=3D2514356/0, merge=3D0/0, ticks=3D400/0, in_queue= =3D297612, util=3D99.06% > > >don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#' >[global] >rw=3Drandrw >rwmixread=3D100 >numjobs=3D1 >iodepth=3D32 >bs=3D4k >direct=3D1 >thread=3D1 >time_based=3D1 >ramp_time=3D0 >runtime=3D10 >ioengine=3D/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_plugin >group_reporting=3D1 >unified_rw_reporting=3D1 >exitall=3D1 >randrepeat=3D0 >norandommap=3D1 >[0000.1c.00.0-1] >filename=3Dtrtype=3DPCIe traddr=3D0000.1c.00.0 ns=3D1 [0000.1c.00.0-2] fil= ename=3Dtrtype=3DPCIe traddr=3D0000.1c.00.0 ns=3D1 [0000.1c.00.0-3] filenam= e=3Dtrtype=3DPCIe traddr=3D0000.1c.00.0 ns=3D1 [0000.1c.00.0-4] filename=3D= trtype=3DPCIe traddr=3D0000.1c.00.0 ns=3D1 > > >don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini >0000.1c.00.0-1: (g=3D0): rw=3Drandrw, bs=3D(R) 4096B-4096B, (W) 4096B-4096= B, (T) 4096B-4096B, ioengine=3Dspdk, iodepth=3D32 >0000.1c.00.0-2: (g=3D0): rw=3Drandrw, bs=3D(R) 4096B-4096B, (W) 4096B-4096= B, (T) 4096B-4096B, ioengine=3Dspdk, iodepth=3D32 >0000.1c.00.0-3: (g=3D0): rw=3Drandrw, bs=3D(R) 4096B-4096B, (W) 4096B-4096= B, (T) 4096B-4096B, ioengine=3Dspdk, iodepth=3D32 >0000.1c.00.0-4: (g=3D0): rw=3Drandrw, bs=3D(R) 4096B-4096B, (W) 4096B-4096= B, (T) 4096B-4096B, ioengine=3Dspdk, iodepth=3D32 >fio-3.11 >Starting 4 threads >Starting SPDK v18.10 / DPDK 18.08.0 initialization... >[ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512 --file-prefix=3Dspdk_= pid4270 ] >EAL: Detected 36 lcore(s) >EAL: Detected 2 NUMA nodes >EAL: No free hugepages reported in hugepages-1048576kB >EAL: Probing VFIO support... >EAL: PCI device 0000:1c:00.0 on NUMA socket 0 >EAL: probe driver: 8086:a54 spdk_nvme >Jobs: 1 (f=3D0): [_(3),f(1)][100.0%][eta 00m:00s] >0000.1c.00.0-1: (groupid=3D0, jobs=3D4): err=3D 0: pid=3D4310: Mon Nov 5 = 05:16:03 2018 >=C2=A0=C2=A0mixed: IOPS=3D12.5k, BW=3D48.0MiB/s (51.4MB/s)(490MiB/10010mse= c) >=C2=A0=C2=A0=C2=A0=C2=A0slat (nsec): min=3D98, max=3D15092, avg=3D120.02, = stdev=3D90.07 >=C2=A0=C2=A0=C2=A0=C2=A0clat (usec): min=3D55, max=3D19757, avg=3D10207.22= , stdev=3D884.06 >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0lat (usec): min=3D60, max=3D19757, avg=3D102= 07.34, stdev=3D884.06 >=C2=A0=C2=A0=C2=A0=C2=A0clat percentiles (usec): >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0| 1.00th=3D[ 8979], 5.00th=3D[ 9110], 10.0= 0th=3D[ 9372], 20.00th=3D[ 9503], >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0| 30.00th=3D[ 9765], 40.00th=3D[ 9896], 50.0= 0th=3D[10159], 60.00th=3D[10290], >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0| 70.00th=3D[10552], 80.00th=3D[10814], 90.0= 0th=3D[11207], 95.00th=3D[11600], >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0| 99.00th=3D[12911], 99.50th=3D[14353], 99.9= 0th=3D[17695], 99.95th=3D[18744], >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0| 99.99th=3D[19006] >=C2=A0=C2=A0=C2=A0bw ( KiB/s): min=3D12008, max=3D12976, per=3D25.00%, av= g=3D12535.70, stdev=3D306.81, samples=3D80 >=C2=A0=C2=A0=C2=A0iops : min=3D 3002, max=3D 3244, avg=3D3133.85, s= tdev=3D76.77, samples=3D80 >=C2=A0=C2=A0lat (usec) : 100=3D0.01%, 250=3D0.01%, 500=3D0.01%, 750=3D0.= 01%, 1000=3D0.01% >=C2=A0=C2=A0lat (msec) : 2=3D0.01%, 4=3D0.02%, 10=3D45.13%, 20=3D54.83% >=C2=A0=C2=A0cpu : usr=3D100.01%, sys=3D0.00%, ctx=3D27, majf=3D0,= minf=3D0 >=C2=A0=C2=A0IO depths : 1=3D0.1%, 2=3D0.1%, 4=3D0.1%, 8=3D0.1%, 16=3D0.= 1%, 32=3D99.9%, >=3D64=3D0.0% >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0submit : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, = 16=3D0.0%, 32=3D0.0%, 64=3D0.0%, >=3D64=3D0.0% >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0complete : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, = 16=3D0.0%, 32=3D0.1%, 64=3D0.0%, >=3D64=3D0.0% >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0issued rwts: total=3D125506,0,0,0 short=3D0,= 0,0,0 dropped=3D0,0,0,0 >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0latency : target=3D0, window=3D0, percenti= le=3D100.00%, depth=3D32 >Run status group 0 (all jobs): >=C2=A0=C2=A0MIXED: bw=3D48.0MiB/s (51.4MB/s), 48.0MiB/s-48.0MiB/s (51.4MB/= s-51.4MB/s), io=3D490MiB (514MB), run=3D10010-10010msec > > >don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#' >[global] >rw=3Drandrw >rwmixread=3D100 >numjobs=3D1 >iodepth=3D1 >bs=3D4k >direct=3D1 >thread=3D1 >time_based=3D1 >ramp_time=3D0 >runtime=3D10 >ioengine=3Dlibaio >group_reporting=3D1 >unified_rw_reporting=3D1 >exitall=3D1 >randrepeat=3D0 >norandommap=3D1 >[nvme2n1] >filename=3D/dev/nvme2n1 > > >don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini >nvme2n1: (g=3D0): rw=3Drandrw, bs=3D(R) 4096B-4096B, (W) 4096B-4096B, (T) = 4096B-4096B, ioengine=3Dlibaio, iodepth=3D1 >fio-3.11 >Starting 1 thread >Jobs: 1 (f=3D1): [r(1)][100.0%][r=3D43.6MiB/s][r=3D11.2k IOPS][eta 00m:00s] >nvme2n1: (groupid=3D0, jobs=3D1): err=3D 0: pid=3D2525: Mon Nov 5 05:22:5= 7 2018 >=C2=A0=C2=A0mixed: IOPS=3D11.2k, BW=3D43.6MiB/s (45.7MB/s)(436MiB/10001mse= c) >=C2=A0=C2=A0=C2=A0=C2=A0slat (nsec): min=3D4848, max=3D208595, avg=3D5126.= 33, stdev=3D771.27 >=C2=A0=C2=A0=C2=A0=C2=A0clat (nsec): min=3D1845, max=3D3405.8k, avg=3D8300= 4.99, stdev=3D33577.36 >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0lat (usec): min=3D15, max=3D3411, avg=3D88.2= 6, stdev=3D33.59 >=C2=A0=C2=A0=C2=A0=C2=A0clat percentiles (usec): >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0| 1.00th=3D[ 15], 5.00th=3D[ 58], 10.0= 0th=3D[ 59], 20.00th=3D[ 59], >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0| 30.00th=3D[ 60], 40.00th=3D[ 77], 50.0= 0th=3D[ 77], 60.00th=3D[ 78], >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0| 70.00th=3D[ 111], 80.00th=3D[ 113], 90.0= 0th=3D[ 113], 95.00th=3D[ 115], >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0| 99.00th=3D[ 186], 99.50th=3D[ 186], 99.9= 0th=3D[ 190], 99.95th=3D[ 192], >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0| 99.99th=3D[ 375] >=C2=A0=C2=A0=C2=A0bw ( KiB/s): min=3D43816, max=3D45016, per=3D99.98%, av= g=3D44639.58, stdev=3D312.88, samples=3D19 >=C2=A0=C2=A0=C2=A0iops : min=3D10954, max=3D11254, avg=3D11159.89, = stdev=3D78.22, samples=3D19 >=C2=A0=C2=A0lat (usec) : 2=3D0.01%, 20=3D2.83%, 50=3D0.01%, 100=3D64.57%= , 250=3D32.59% >=C2=A0=C2=A0lat (usec) : 500=3D0.01%, 750=3D0.01% >=C2=A0=C2=A0lat (msec) : 2=3D0.01%, 4=3D0.01% >=C2=A0=C2=A0cpu : usr=3D4.02%, sys=3D10.33%, ctx=3D111630, majf= =3D0, minf=3D12 >=C2=A0=C2=A0IO depths : 1=3D100.0%, 2=3D0.0%, 4=3D0.0%, 8=3D0.0%, 16=3D= 0.0%, 32=3D0.0%, >=3D64=3D0.0% >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0submit : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, = 16=3D0.0%, 32=3D0.0%, 64=3D0.0%, >=3D64=3D0.0% >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0complete : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, = 16=3D0.0%, 32=3D0.0%, 64=3D0.0%, >=3D64=3D0.0% >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0issued rwts: total=3D111631,0,0,0 short=3D0,= 0,0,0 dropped=3D0,0,0,0 >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0latency : target=3D0, window=3D0, percenti= le=3D100.00%, depth=3D1 >Run status group 0 (all jobs): >=C2=A0=C2=A0MIXED: bw=3D43.6MiB/s (45.7MB/s), 43.6MiB/s-43.6MiB/s (45.7MB/= s-45.7MB/s), io=3D436MiB (457MB), run=3D10001-10001msec Disk stats (read/wr= ite): >=C2=A0=C2=A0nvme2n1: ios=3D110275/0, merge=3D0/0, ticks=3D0/0, in_queue=3D= 8596, util=3D86.22% > > >Thanks, >Joe > >> -----Original Message----- >> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang >> Sent: Sunday, November 4, 2018 8:40 PM >> To: Storage Performance Development Kit < spdk(a)lists.01.org > >> Subject: Re: [SPDK] FIO NVMe Performance Results >> = >> For the SPDK plugin, from the below result, looks like only 1 job = >> started with >> 32 queue depth: >> = >> Jobs: 1 (f=3D0): [f(1),_(3)][100.0%][eta 00m:00s] <<<< >> = >> Try to update the FIO configuration file as >> = >> [job1] >> filename=3Dtrtype=3DPCIe traddr=3D0000.1c.00.0 ns=3D1 [job2] = >> filename=3Dtrtype=3DPCIe >> traddr=3D0000.1c.00.0 ns=3D1 .... >> = >> The kernel FIO is with 4 jobs and each job of 32 queue depth. For = >> SPDK, you can configure as the same, or just configure two jobs and = >> each job of 64 queue depth. The overall queue depth to the device is sam= e of 128. >> = >> Thanks, >> Gang >> = >> -----Original Message----- >> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Gruher, = >> Joseph R >> Sent: Monday, November 5, 2018 10:49 AM >> To: Storage Performance Development Kit < spdk(a)lists.01.org > >> Subject: [SPDK] FIO NVMe Performance Results >> = >> Hi folks- >> = >> I'm testing SPDK 18.10 on Ubuntu 18.04 with kernel 4.18.16. I used = >> FIO with the kernel NVMe driver to measure the performance of a local = >> (PCIe >> attached) Intel P4500 NVMe device on a 4KB random read workload and = >> obtained 477K IOPS, roughly in line with the drive spec. Then I = >> tested the same drive with the SPDK FIO plugin and only achieved 13K = >> IOPS. The FIO test files and the results are pasted below. Any ideas = >> where I'm going wrong here? >> = >> Thanks! >> = >> = >> don(a)donst201:~/fio/single/rr$ ls >> /sys/devices/pci0000:17/0000:17:02.0/0000:1c:00.0/nvme/nvme1/ >> address cntlid dev device firmware_rev model nvme2n1 power = >> rescan_controller reset_controller serial state subsysnqn = >> subsystem transport uevent >> = >> = >> don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#' >> [global] >> rw=3Drandrw >> rwmixread=3D100 >> numjobs=3D4 >> iodepth=3D32 >> bs=3D4k >> direct=3D1 >> thread=3D1 >> time_based=3D1 >> ramp_time=3D0 >> runtime=3D10 >> ioengine=3Dlibaio >> group_reporting=3D1 >> unified_rw_reporting=3D1 >> exitall=3D1 >> randrepeat=3D0 >> norandommap=3D1 >> [nvme2n1] >> filename=3D/dev/nvme2n1 >> = >> = >> don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini >> nvme2n1: (g=3D0): rw=3Drandrw, bs=3D(R) 4096B-4096B, (W) 4096B-4096B, (T= ) = >> 4096B-4096B, ioengine=3Dlibaio, iodepth=3D32 ... >> fio-3.11 >> Starting 4 threads >> Jobs: 4 (f=3D4): [r(4)][100.0%][r=3D1874MiB/s][r=3D480k IOPS][eta 00m:00= s] >> nvme2n1: (groupid=3D0, jobs=3D4): err=3D 0: pid=3D2575: Mon Nov 5 02:26= :25 2018 >> mixed: IOPS=3D477k, BW=3D1862MiB/s (1952MB/s)(18.2GiB/10001msec) >> slat (nsec): min=3D1295, max=3D333088, avg=3D2216.35, stdev=3D927.89 >> clat (nsec): min=3D506, max=3D5818.8k, avg=3D265924.16, stdev=3D22376= 1.59 >> lat (usec): min=3D6, max=3D5821, avg=3D268.19, stdev=3D223.76 >> clat percentiles (usec): >> | 1.00th=3D[ 13], 5.00th=3D[ 74], 10.00th=3D[ 87], 20.00th= =3D[ 115], >> | 30.00th=3D[ 139], 40.00th=3D[ 167], 50.00th=3D[ 204], 60.00th= =3D[ 247], >> | 70.00th=3D[ 306], 80.00th=3D[ 388], 90.00th=3D[ 523], 95.00th= =3D[ 660], >> | 99.00th=3D[ 988], 99.50th=3D[ 1156], 99.90th=3D[ 2343], 99.95th= =3D[ 3195], >> | 99.99th=3D[ 4424] >> bw ( KiB/s): min=3D448192, max=3D483104, per=3D25.00%, avg=3D476597.= 10, = >> stdev=3D6748.78, samples=3D80 >> iops : min=3D112048, max=3D120776, avg=3D119149.27, stdev=3D16= 87.20, >> samples=3D80 >> lat (nsec) : 750=3D0.01% >> lat (usec) : 10=3D0.52%, 20=3D1.25%, 50=3D0.25%, 100=3D13.38%, 250= =3D45.13% >> lat (usec) : 500=3D28.23%, 750=3D8.08%, 1000=3D2.22% >> lat (msec) : 2=3D0.82%, 4=3D0.10%, 10=3D0.02% >> cpu : usr=3D14.08%, sys=3D36.02%, ctx=3D1930207, majf=3D0, mi= nf=3D697 >> IO depths : 1=3D0.1%, 2=3D0.1%, 4=3D0.1%, 8=3D0.1%, 16=3D0.1%, 32= =3D100.0%, >> >=3D64=3D0.0% >> submit : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.0%, 6= 4=3D0.0%, >> >=3D64=3D0.0% >> complete : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.1%, 6= 4=3D0.0%, >> >=3D64=3D0.0% >> issued rwts: total=3D4766095,0,0,0 short=3D0,0,0,0 dropped=3D0,0,0,0 >> latency : target=3D0, window=3D0, percentile=3D100.00%, depth=3D32 >> Run status group 0 (all jobs): >> MIXED: bw=3D1862MiB/s (1952MB/s), 1862MiB/s-1862MiB/s (1952MB/s- = >> 1952MB/s), io=3D18.2GiB (19.5GB), run=3D10001-10001msec Disk stats >> (read/write): >> nvme2n1: ios=3D4709190/0, merge=3D0/0, ticks=3D4096/0, in_queue=3D1388= 752, = >> util=3D100.00% >> = >> = >> don(a)donst201:~/fio/single/rr$ sudo >> /home/don/install/spdk/spdk/scripts/setup.sh >> Active mountpoints on /dev/nvme0n1, so not binding PCI dev = >> 0000:03:00.0 >> 0000:1c:00.0 (8086 0a54): nvme -> uio_pci_generic >> 0000:1d:00.0 (8086 0a54): nvme -> uio_pci_generic >> 0000:5e:00.0 (8086 0a54): nvme -> uio_pci_generic >> 0000:5f:00.0 (8086 0a54): nvme -> uio_pci_generic >> 0000:62:00.0 (8086 0a54): nvme -> uio_pci_generic >> 0000:63:00.0 (8086 0a54): nvme -> uio_pci_generic >> 0000:64:00.0 (8086 0a54): nvme -> uio_pci_generic >> 0000:65:00.0 (8086 0a54): nvme -> uio_pci_generic >> 0000:da:00.0 (8086 0a54): nvme -> uio_pci_generic >> 0000:db:00.0 (8086 0a54): nvme -> uio_pci_generic >> 0000:dc:00.0 (8086 0a54): nvme -> uio_pci_generic >> 0000:dd:00.0 (8086 0a54): nvme -> uio_pci_generic >> 0000:e0:00.0 (8086 0a54): nvme -> uio_pci_generic >> 0000:e1:00.0 (8086 0a54): nvme -> uio_pci_generic >> 0000:e2:00.0 (8086 0a54): nvme -> uio_pci_generic >> 0000:e3:00.0 (8086 0a54): nvme -> uio_pci_generic >> 0000:00:04.0 (8086 2021): ioatdma -> uio_pci_generic >> 0000:00:04.1 (8086 2021): ioatdma -> uio_pci_generic >> 0000:00:04.2 (8086 2021): ioatdma -> uio_pci_generic >> 0000:00:04.3 (8086 2021): ioatdma -> uio_pci_generic >> 0000:00:04.4 (8086 2021): ioatdma -> uio_pci_generic >> 0000:00:04.5 (8086 2021): ioatdma -> uio_pci_generic >> 0000:00:04.6 (8086 2021): ioatdma -> uio_pci_generic >> 0000:00:04.7 (8086 2021): ioatdma -> uio_pci_generic >> 0000:80:04.0 (8086 2021): ioatdma -> uio_pci_generic >> 0000:80:04.1 (8086 2021): ioatdma -> uio_pci_generic >> 0000:80:04.2 (8086 2021): ioatdma -> uio_pci_generic >> 0000:80:04.3 (8086 2021): ioatdma -> uio_pci_generic >> 0000:80:04.4 (8086 2021): ioatdma -> uio_pci_generic >> 0000:80:04.5 (8086 2021): ioatdma -> uio_pci_generic >> 0000:80:04.6 (8086 2021): ioatdma -> uio_pci_generic >> 0000:80:04.7 (8086 2021): ioatdma -> uio_pci_generic >> = >> = >> don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#' >> [global] >> rw=3Drandrw >> rwmixread=3D100 >> numjobs=3D4 >> iodepth=3D32 >> bs=3D4k >> direct=3D1 >> thread=3D1 >> time_based=3D1 >> ramp_time=3D0 >> runtime=3D10 >> ioengine=3D/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_plug >> in >> group_reporting=3D1 >> unified_rw_reporting=3D1 >> exitall=3D1 >> randrepeat=3D0 >> norandommap=3D1 >> [0000.1c.00.0] >> filename=3Dtrtype=3DPCIe traddr=3D0000.1c.00.0 ns=3D1 >> = >> = >> don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini >> 0000.1c.00.0: (g=3D0): rw=3Drandrw, bs=3D(R) 4096B-4096B, (W) 4096B-4096= B, = >> (T) 4096B-4096B, ioengine=3Dspdk, iodepth=3D32 ... >> fio-3.11 >> Starting 4 threads >> Starting SPDK v18.10 / DPDK 18.08.0 initialization... >> [ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512 --file- >> prefix=3Dspdk_pid3668 ] >> EAL: Detected 36 lcore(s) >> EAL: Detected 2 NUMA nodes >> EAL: No free hugepages reported in hugepages-1048576kB >> EAL: Probing VFIO support... >> EAL: PCI device 0000:1c:00.0 on NUMA socket 0 >> EAL: probe driver: 8086:a54 spdk_nvme >> Jobs: 1 (f=3D0): [f(1),_(3)][100.0%][eta 00m:00s] >> 0000.1c.00.0: (groupid=3D0, jobs=3D4): err=3D 0: pid=3D3709: Mon Nov 5 = >> 02:28:29 >> 2018 >> mixed: IOPS=3D13.2k, BW=3D51.6MiB/s (54.1MB/s)(517MiB/10011msec) >> slat (nsec): min=3D109, max=3D15344, avg=3D127.91, stdev=3D73.44 >> clat (usec): min=3D187, max=3D18715, avg=3D9683.80, stdev=3D650.54 >> lat (usec): min=3D193, max=3D18715, avg=3D9683.93, stdev=3D650.53 >> clat percentiles (usec): >> | 1.00th=3D[ 8717], 5.00th=3D[ 8848], 10.00th=3D[ 8979], 20.00th= =3D[ 9110], >> | 30.00th=3D[ 9241], 40.00th=3D[ 9372], 50.00th=3D[ 9503], 60.00th= =3D[ 9765], >> | 70.00th=3D[10028], 80.00th=3D[10159], 90.00th=3D[10552], 95.00th= =3D[10945], >> | 99.00th=3D[11338], 99.50th=3D[11338], 99.90th=3D[11731], 99.95th= =3D[13960], >> | 99.99th=3D[17695] >> bw ( KiB/s): min=3D13080, max=3D13304, per=3D25.00%, avg=3D13214.70, = >> stdev=3D55.30, samples=3D80 >> iops : min=3D 3270, max=3D 3326, avg=3D3303.65, stdev=3D13.84,= samples=3D80 >> lat (usec) : 250=3D0.01%, 500=3D0.01%, 750=3D0.01%, 1000=3D0.01% >> lat (msec) : 2=3D0.01%, 4=3D0.02%, 10=3D70.85%, 20=3D29.11% >> cpu : usr=3D100.01%, sys=3D0.00%, ctx=3D23, majf=3D0, minf=3D0 >> IO depths : 1=3D0.1%, 2=3D0.1%, 4=3D0.1%, 8=3D0.1%, 16=3D0.1%, 32= =3D99.9%, >> >=3D64=3D0.0% >> submit : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.0%, 6= 4=3D0.0%, >> >=3D64=3D0.0% >> complete : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.1%, 6= 4=3D0.0%, >> >=3D64=3D0.0% >> issued rwts: total=3D132300,0,0,0 short=3D0,0,0,0 dropped=3D0,0,0,0 >> latency : target=3D0, window=3D0, percentile=3D100.00%, depth=3D32 >> Run status group 0 (all jobs): >> MIXED: bw=3D51.6MiB/s (54.1MB/s), 51.6MiB/s-51.6MiB/s (54.1MB/s- = >> 54.1MB/s), io=3D517MiB (542MB), run=3D10011-10011msec >> = >> _______________________________________________ >> SPDK mailing list >> SPDK(a)lists.01.org >> https://lists.01.org/mailman/listinfo/spdk >> _______________________________________________ >> SPDK mailing list >> SPDK(a)lists.01.org >> https://lists.01.org/mailman/listinfo/spdk >_______________________________________________ >SPDK mailing list >SPDK(a)lists.01.org >https://lists.01.org/mailman/listinfo/spdk >_______________________________________________ >SPDK mailing list >SPDK(a)lists.01.org >https://lists.01.org/mailman/listinfo/spdk -- = =D0=9C=D0=B0=D1=80=D0=BE=D0=B2 =D0=90=D0=BB=D0=B5=D0=BA=D1=81=D0=B5=D0=B9 --===============2901104388327956709==--