From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============6436113331615179505==" MIME-Version: 1.0 From: Harris, James R Subject: Re: [SPDK] FIO NVMe Performance Results Date: Mon, 05 Nov 2018 21:56:29 +0000 Message-ID: In-Reply-To: EA913ED399BBA34AA4EAC2EDC24CDD009C2166C9@FMSMSX105.amr.corp.intel.com List-ID: To: spdk@lists.01.org --===============6436113331615179505== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi Joe, This is all just local PCIe NVMe =E2=80=93 no NVMe-oF? Can you make sure y= ou don=E2=80=99t have any other SPDK processes running while you=E2=80=99re= doing this? I=E2=80=99m concerned that fio is polling on the same CPU cor= e as an SPDK process, and your performance degradation is due to timeslice = sharing. -Jim On 11/5/18, 2:42 PM, "SPDK on behalf of Howell, Seth" wrote: I think that the uio_pci_generic vs. vfio thing is unrelated. The only = reason I brought it up was because it could have been indicative of a large= r configuration difference. However, I ran my test with and without vfio su= pport just to be sure and got the higher performance numbers both times. = As a side note: It's probably a good thing to have enabled in general when running spdk= benchmarks because that is the recommended configuration on any physical m= achine. = You can enable it in grub by adding the following to the GRUB_CMDLINE_L= INUX line of /etc/default.grub: intel_iommu=3Don and then running grub2-mkconfig. = -----Original Message----- From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Gruher, Jo= seph R Sent: Monday, November 5, 2018 1:39 PM To: Storage Performance Development Kit Subject: Re: [SPDK] FIO NVMe Performance Results = Attempting to fall back to earlier revisions has not yielded any improv= ements for me. All tests use the same P4500 physical disk. FIO files are = as shown before, running a 4 thread, queue depth 32 per thread, 4KB random = read workload. = SPDK 18.10, kernel 4.18.16, FIO 3.11 =3D 12.9k IOPS SPDK 18.10, kernel = 4.16.14, FIO 3.11 =3D 11.9k IOPS SPDK 18.04, kernel 4.18.16, FIO 3.11 =3D 1= 2.9k IOPS SPDK 18.04, kernel 4.16.14, FIO 3.11 =3D 13.0k IOPS SPDK 18.04, k= ernel 4.16.16, FIO 3.3 =3D 13.1k IOPS = Kernel NVMe driver, kernel 4.16.16, FIO 3.3 =3D 477k IOPS Kernel NVMe d= river, kernel 4.18.16, FIO 3.11 =3D 477k IOPS = I am curious if the comment below about uio_pci_generic instead of vfio= can lead us anywhere, or maybe there's some OS setting or BIOS setting tha= t needs to be toggled... = For the folks working inside Intel, I can also give you direct access t= o the test system, if that would help. = Thanks, Joe = > -----Original Message----- > From: Gruher, Joseph R > Sent: Monday, November 5, 2018 11:05 AM > To: Storage Performance Development Kit > Subject: RE: [SPDK] FIO NVMe Performance Results > = > I've had good FIO NVMe results in the past we well, with SPDK 18.04 o= n = > Ubuntu 18.04, with kernel 4.16.14. That was on a different, but = > similar, hardware platform. > = > I suspect something has changed now in SPDK 18.10 and/or kernel = > 4.18.16. I can try going back to earlier versions of each to see if = I = > can recreate earlier good results and find where things went wrong. > = > We're not running in a VM - this is a bare metal Xeon server with = > Skylake CPUs. > = > Thanks, > Joe > = > > -----Original Message----- > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Howell= , = > > Seth > > Sent: Monday, November 5, 2018 10:32 AM > > To: Storage Performance Development Kit > > Subject: Re: [SPDK] FIO NVMe Performance Results > > > > Hi Joseph, > > I attempted to verify your results on my system. I am using fedora = > > 27 with kernel version 4.17.3 with a p3700 drive. I cloned fio from = > > https://github.com/axboe/fio and checked out the appropriate tag. I = > > got appx 408k iops. > > Typically we run spdk with the iommu enabled. I noticed that your = > > devices are binding to uio_pci_generic instead of vfio. This = > > shouldn't make any difference at all, but I just want to make sure = > > you aren't running in a VM or something like that and passing = > > through your NVMe drives and that's why you aren't using the IOMMU. = > > That could explain a deeper configuration difference that could hel= p = > > us know where to look for > the disparity. > > > > Please see the contents of my configuration files and output below: > > > > Test.ini: > > [global] > > rw=3Drandrw > > rwmixread=3D100 > > numjobs=3D4 > > iodepth=3D32 > > bs=3D4k > > direct=3D1 > > thread=3D1 > > time_based=3D1 > > ramp_time=3D0 > > runtime=3D10 > > ioengine=3Dspdk_bdev > > spdk_conf=3D./nvme.conf > > group_reporting=3D1 > > unified_rw_reporting=3D1 > > exitall=3D1 > > randrepeat=3D0 > > norandommap=3D1 > > [0000.81.00.0] > > filename=3DNvme0n1 > > > > nvme.conf: > > [Nvme] > > TransportId "trtype:PCIe traddr:0000.81.00.0" Nvme0 > > > > Sudo > > > LD_PRELOAD=3D/home/sethhowe/Desktop/Development_Folders/spdk/exam > > ples/bdev/fio_plugin/fio_plugin ./fio ./test.ini >./out,txt 2>&1 > > > > Out.txt: > > 0000.81.00.0: (g=3D0): rw=3Drandrw, bs=3D(R) 4096B-4096B, (W) 4096B= -4096B, > > (T) 4096B-4096B, ioengine=3Dspdk_bdev, iodepth=3D32 ... > > fio-3.11 > > Starting 4 threads > > Starting SPDK v18.10 / DPDK 18.08.0 initialization... > > [ DPDK EAL parameters: fio --no-shconf -c 0x1 > > --file-prefix=3Dspdk_pid782339 ] > > EAL: Detected 16 lcore(s) > > EAL: Detected 2 NUMA nodes > > EAL: No free hugepages reported in hugepages-1048576kB > > EAL: Probing VFIO support... > > EAL: VFIO support initialized > > EAL: PCI device 0000:81:00.0 on NUMA socket 1 > > EAL: probe driver: 8086:953 spdk_nvme > > > > 0000.81.00.0: (groupid=3D0, jobs=3D4): err=3D 0: pid=3D782359: Mon = Nov 5 > > 10:58:59 > > 2018 > > mixed: IOPS=3D410k, BW=3D1600MiB/s (1678MB/s)(15.6GiB/10001msec) > > slat (nsec): min=3D179, max=3D19985, avg=3D245.97, stdev=3D122.= 96 > > clat (usec): min=3D8, max=3D1821, avg=3D311.82, stdev=3D167.14 > > lat (usec): min=3D8, max=3D1822, avg=3D312.06, stdev=3D167.14 > > clat percentiles (usec): > > | 1.00th=3D[ 24], 5.00th=3D[ 48], 10.00th=3D[ 78], 20.= 00th=3D[ 141], > > | 30.00th=3D[ 200], 40.00th=3D[ 258], 50.00th=3D[ 314], 60.= 00th=3D[ 371], > > | 70.00th=3D[ 429], 80.00th=3D[ 486], 90.00th=3D[ 545], 95.= 00th=3D[ 570], > > | 99.00th=3D[ 594], 99.50th=3D[ 594], 99.90th=3D[ 627], 99.= 95th=3D[ 652], > > | 99.99th=3D[ 676] > > bw ( KiB/s): min=3D407264, max=3D416040, per=3D25.01%, avg=3D40= 9742.55, = > > stdev=3D1337.38, samples=3D76 > > iops : min=3D101816, max=3D104010, avg=3D102435.61, stdev= =3D334.41, > > samples=3D76 > > lat (usec) : 10=3D0.01%, 20=3D0.46%, 50=3D4.88%, 100=3D8.41%, 2= 50=3D25.29% > > lat (usec) : 500=3D43.15%, 750=3D17.81%, 1000=3D0.01% > > lat (msec) : 2=3D0.01% > > cpu : usr=3D99.78%, sys=3D0.00%, ctx=3D26, majf=3D0, min= f=3D0 > > IO depths : 1=3D0.1%, 2=3D0.1%, 4=3D0.1%, 8=3D0.1%, 16=3D0.1%,= 32=3D99.9%, > > >=3D64=3D0.0% > > submit : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.= 0%, 64=3D0.0%, > > >=3D64=3D0.0% > > complete : 0=3D0.0%, 4=3D100.0%, 8=3D0.1%, 16=3D0.1%, 32=3D0.= 1%, = > > 64=3D0.0%, > > >=3D64=3D0.0% > > issued rwts: total=3D4096867,0,0,0 short=3D0,0,0,0 dropped=3D0= ,0,0,0 > > latency : target=3D0, window=3D0, percentile=3D100.00%, dept= h=3D32 > > > > Run status group 0 (all jobs): > > MIXED: bw=3D1600MiB/s (1678MB/s), 1600MiB/s-1600MiB/s (1678MB/s- = > > 1678MB/s), io=3D15.6GiB (16.8GB), run=3D10001-10001msec > > > > Thanks, > > > > Seth > > > > > > > > -----Original Message----- > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey = > > Kuzmin > > Sent: Monday, November 5, 2018 9:48 AM > > To: Storage Performance Development Kit > > Subject: Re: [SPDK] FIO NVMe Performance Results > > > > On Mon, Nov 5, 2018, 18:29 Gruher, Joseph R = > > > > wrote: > > > > > My FIO version is v3.11. > > > > > > I tried the bdev FIO plugin as suggested, but performance is abou= t = > > > the same as the other SPDK FIO plugin, I observed only 10.4K IOPS = > > > in a quick 4KB random read test. See configuration and results b= elow. > > > > > > > As 12 ms median read latency for an nvme ssd is far beyond anything = > > reasonable, why don't you just perf this run, to see where the time = > > is being spent? > > > > Regards, > > Andrey > > > > > > > > > > > don(a)donst201:~/fio/single/rr$ cat nvme.conf [Nvme] TransportId = > > > "trtype:PCIe traddr:0000.1c.00.0" Nvme0 > > > > > > > > > don(a)donst201:~/fio/single/rr$ cat 1c-bdev.ini |grep -v '#' > > > [global] > > > rw=3Drandrw > > > rwmixread=3D100 > > > numjobs=3D4 > > > iodepth=3D32 > > > bs=3D4k > > > direct=3D1 > > > thread=3D1 > > > time_based=3D1 > > > ramp_time=3D0 > > > runtime=3D10 > > > ioengine=3Dspdk_bdev > > > spdk_conf=3D./nvme.conf > > > group_reporting=3D1 > > > unified_rw_reporting=3D1 > > > exitall=3D1 > > > randrepeat=3D0 > > > norandommap=3D1 > > > [0000.1c.00.0] > > > filename=3DNvme0n1 > > > > > > > > > don(a)donst201:~/fio/single/rr$ sudo > > > > > > LD_PRELOAD=3D/home/don/install/spdk/spdk/examples/bdev/fio_plugin/fio= _p > > l > > > ugin > > > fio 1c-bdev.ini > > > 0000.1c.00.0: (g=3D0): rw=3Drandrw, bs=3D(R) 4096B-4096B, (W) = > > > 4096B-4096B, > > > (T) 4096B-4096B, ioengine=3Dspdk_bdev, iodepth=3D32 ... > > > fio-3.11 > > > Starting 4 threads > > > Starting SPDK v18.10 / DPDK 18.08.0 initialization... > > > [ DPDK EAL parameters: fio --no-shconf -c 0x1 > > > --file-prefix=3Dspdk_pid4865 ] > > > EAL: Detected 36 lcore(s) > > > EAL: Detected 2 NUMA nodes > > > EAL: No free hugepages reported in hugepages-1048576kB > > > EAL: Probing VFIO support... > > > EAL: PCI device 0000:03:00.0 on NUMA socket 0 > > > EAL: probe driver: 8086:f1a5 spdk_nvme > > > EAL: PCI device 0000:1c:00.0 on NUMA socket 0 > > > EAL: probe driver: 8086:a54 spdk_nvme > > > EAL: PCI device 0000:1d:00.0 on NUMA socket 0 > > > EAL: probe driver: 8086:a54 spdk_nvme > > > EAL: PCI device 0000:5e:00.0 on NUMA socket 0 > > > EAL: probe driver: 8086:a54 spdk_nvme > > > EAL: PCI device 0000:5f:00.0 on NUMA socket 0 > > > EAL: probe driver: 8086:a54 spdk_nvme > > > EAL: PCI device 0000:62:00.0 on NUMA socket 0 > > > EAL: probe driver: 8086:a54 spdk_nvme > > > EAL: PCI device 0000:63:00.0 on NUMA socket 0 > > > EAL: probe driver: 8086:a54 spdk_nvme > > > EAL: PCI device 0000:64:00.0 on NUMA socket 0 > > > EAL: probe driver: 8086:a54 spdk_nvme > > > EAL: PCI device 0000:65:00.0 on NUMA socket 0 > > > EAL: probe driver: 8086:a54 spdk_nvme > > > EAL: PCI device 0000:da:00.0 on NUMA socket 1 > > > EAL: probe driver: 8086:a54 spdk_nvme > > > EAL: PCI device 0000:db:00.0 on NUMA socket 1 > > > EAL: probe driver: 8086:a54 spdk_nvme > > > EAL: PCI device 0000:dc:00.0 on NUMA socket 1 > > > EAL: probe driver: 8086:a54 spdk_nvme > > > EAL: PCI device 0000:dd:00.0 on NUMA socket 1 > > > EAL: probe driver: 8086:a54 spdk_nvme > > > EAL: PCI device 0000:e0:00.0 on NUMA socket 1 > > > EAL: probe driver: 8086:a54 spdk_nvme > > > EAL: PCI device 0000:e1:00.0 on NUMA socket 1 > > > EAL: probe driver: 8086:a54 spdk_nvme > > > EAL: PCI device 0000:e2:00.0 on NUMA socket 1 > > > EAL: probe driver: 8086:a54 spdk_nvme > > > EAL: PCI device 0000:e3:00.0 on NUMA socket 1 > > > EAL: probe driver: 8086:a54 spdk_nvme > > > Jobs: 4 (f=3D4): [r(4)][100.0%][r=3D45.7MiB/s][r=3D11.7k IOPS][et= a = > > > 00m:00s] > > > 0000.1c.00.0: (groupid=3D0, jobs=3D4): err=3D 0: pid=3D4906: Mon = Nov 5 > > > 15:23:47 > > > 2018 > > > mixed: IOPS=3D10.4k, BW=3D40.5MiB/s (42.5MB/s)(406MiB/10012msec) > > > slat (nsec): min=3D201, max=3D15831, avg=3D238.11, stdev=3D14= 8.78 > > > clat (usec): min=3D124, max=3D31244, avg=3D12330.69, stdev=3D= 1697.98 > > > lat (usec): min=3D130, max=3D31244, avg=3D12330.93, stdev=3D= 1697.98 > > > clat percentiles (usec): > > > | 1.00th=3D[ 9634], 5.00th=3D[10159], 10.00th=3D[10552], 2= 0.00th=3D[10945], > > > | 30.00th=3D[11338], 40.00th=3D[11731], 50.00th=3D[12125], > > 60.00th=3D[12518], > > > | 70.00th=3D[13042], 80.00th=3D[13566], 90.00th=3D[14353], > > 95.00th=3D[15008], > > > | 99.00th=3D[16712], 99.50th=3D[18744], 99.90th=3D[27395], > > 99.95th=3D[29754], > > > | 99.99th=3D[30278] > > > bw ( KiB/s): min=3D 8784, max=3D11952, per=3D24.99%, avg=3D10= 375.53, = > > > stdev=3D929.68, samples=3D80 > > > iops : min=3D 2196, max=3D 2988, avg=3D2593.85, stdev= =3D232.41, > > > samples=3D80 > > > lat (usec) : 250=3D0.01%, 500=3D0.01%, 750=3D0.01%, 1000=3D0.= 01% > > > lat (msec) : 2=3D0.01%, 4=3D0.02%, 10=3D2.71%, 20=3D96.91%, 5= 0=3D0.33% > > > cpu : usr=3D99.99%, sys=3D0.01%, ctx=3D28, majf=3D0, m= inf=3D0 > > > IO depths : 1=3D0.1%, 2=3D0.1%, 4=3D0.1%, 8=3D0.1%, 16=3D0.1= %, 32=3D99.9%, > > > >=3D64=3D0.0% > > > submit : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D= 0.0%, 64=3D0.0%, > > > >=3D64=3D0.0% > > > complete : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D= 0.1%, = > > > 64=3D0.0%, > > > >=3D64=3D0.0% > > > issued rwts: total=3D103907,0,0,0 short=3D0,0,0,0 dropped=3D= 0,0,0,0 > > > latency : target=3D0, window=3D0, percentile=3D100.00%, de= pth=3D32 > > > Run status group 0 (all jobs): > > > MIXED: bw=3D40.5MiB/s (42.5MB/s), 40.5MiB/s-40.5MiB/s = > > > (42.5MB/s-42.5MB/s), io=3D406MiB (426MB), run=3D10012-10012msec > > > > > > > > > Thanks, > > > Joe > > > > > > > -----Original Message----- > > > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of ??= ??? > > ??????? > > > > Sent: Monday, November 5, 2018 12:45 AM > > > > To: Storage Performance Development Kit > > > > Subject: Re: [SPDK] FIO NVMe Performance Results > > > > > > > > Ok, Do you see some performance issues using general bdev fio = > > > > plugin > > > with > > > > you configuration ? > > > > > > > > Best Regards > > > > Alex > > > > > > > > > > > > >=D0=9F=D0=BE=D0=BD=D0=B5=D0=B4=D0=B5=D0=BB=D1=8C=D0=BD=D0=B8= =D0=BA, 5 =D0=BD=D0=BE=D1=8F=D0=B1=D1=80=D1=8F 2018, 11:39 +03:00 =D0=BE= =D1=82 "Cao, Gang" > > > > : > > > > > > > > > >Yes. This is for the bdev version of SPDK FIO plugin and you = > > > > >can assign > > > the > > > > conf file as the parameter in the FIO configuration. > > > > > > > > > >Previously, it's another SPDK FIO plugin for the user space = > > > > >NVMe driver > > > > only. > > > > > > > > > >These are two different FIO plugins. > > > > > > > > > >Thanks, > > > > >Gang > > > > > > > > > >-----Original Message----- > > > > >From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of = > > > > >=D0=9C=D0=B0=D1=80=D0=BE=D0=B2 =D0=90=D0=BB=D0=B5=D0=BA=D1=81= =D0=B5=D0=B9 > > > > >Sent: Monday, November 5, 2018 4:32 PM > > > > >To: Storage Performance Development Kit < spdk(a)lists.01.org > > > > > >Subject: Re: [SPDK] FIO NVMe Performance Results > > > > > > > > > >Sorry for incomplete message I have sent before > > > > > > > > > >Hello Gang. > > > > > > > > > >I just curious, why would you run fio with SPDK using this con= f = > > > > >file, > > > with is a > > > > bit different from what is suggested in here > > https://github.com/spdk/spdk/tree/master/examples/bdev/fio_plugin > > > > . > > > > > > > > > >I mean may be you should create conf file for spdk like this o= ne : > > > > > > > > > >nvme.conf : > > > > >[Nvme] > > > > > TransportId "trtype:PCIe traddr:0000.1c.00.0" Nvme0 > > > > > > > > > >and provide it in fio config. > > > > > > > > > >fio.ini : > > > > >[global] > > > > >ioengine=3Dspdk_bdev > > > > >spdk_conf=3D....../nvme.conf > > > > >rw=3Drandrw > > > > >rwmixread=3D100 > > > > >numjobs=3D4 > > > > >iodepth=3D32 > > > > >bs=3D4k > > > > >direct=3D1 > > > > >thread=3D1 > > > > >time_based=3D1 > > > > >ramp_time=3D0 > > > > >runtime=3D10 > > > > >group_reporting=3D1 > > > > >unified_rw_reporting=3D1 > > > > >exitall=3D1 > > > > >randrepeat=3D0 > > > > >norandommap=3D1 > > > > > > > > > >[0000:06:00.0] > > > > >filename=3DNvme0n1 > > > > > > > > > >and run like this > > > > > > > > > ># LD_PRELOAD=3Dexamples/bdev/fio_plugin/fio_plugin fio = > > > > >examples/bdev/fio_plugin/fio.ini > > > > > > > > > >Just to note. The recommended version of fio for SPDK is 3.3 > > > > > > > > > >Best Regards > > > > >Alex > > > > > > > > > > > > > > >>=D0=9F=D0=BE=D0=BD=D0=B5=D0=B4=D0=B5=D0=BB=D1=8C=D0=BD=D0=B8= =D0=BA, 5 =D0=BD=D0=BE=D1=8F=D0=B1=D1=80=D1=8F 2018, 9:53 +03:00 =D0=BE=D1= =82 "Cao, Gang" < > > > > gang.cao(a)intel.com >: > > > > >> > > > > >>Thanks. This is interesting especially for your finding on = > > > > >>queue depth > > > 1 > > > > case. We are trying to reproduce this issue locally same as you= r = > > > > configuration. > > > > >> > > > > >>Thanks, > > > > >>Gang > > > > >> > > > > >>-----Original Message----- > > > > >>From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of = > > > > >>Gruher, Joseph R > > > > >>Sent: Monday, November 5, 2018 1:32 PM > > > > >>To: Storage Performance Development Kit < spdk(a)lists.01.or= g > > > > > >>Subject: Re: [SPDK] FIO NVMe Performance Results > > > > >> > > > > >>Interesting catch. It seems like FIO is actually not = > > > > >>accurately > > > reporting the > > > > number of jobs in the report out for the SPDK case. If I = > > > > disable the group_reporting parameter I do get a printout for = > > > > four separate jobs in > > > the > > > > SPDK case, so I think it is really running four jobs there, and = > > > > just > > > summarizing > > > > incorrectly. I also see four separate CPU cores go to 100% whe= n = > > > > I run > > > that > > > > FIO test case, indicating four jobs are active. > > > > >> > > > > >>Also, I don't think would explain the difference in performan= ce. > > > Using an > > > > aggregate queue depth of 32 versus 128 shouldn't lead to a drop = > > > > in performance from 377K to 13K. > > > > >> > > > > >>I retested SPDK with multiple separate job descriptions as you > > > suggested > > > > anyway, using four separate jobs instead of relying on the > > > > numjobs=3D4 parameter, and the results are below. In that SPDK = > > > > test with four > > > separate > > > > FIO job statements, each with queue depth 32, I still only = > > > > achieved 12.5K IOPS. > > > > >> > > > > >>I then re-ran the kernel NVMe driver test case with only a = > > > > >>single job > > > for > > > > comparison, and even with only 1 job and queue depth 32 it stil= l = > > > > achieves 254K IOPS. > > > > >> > > > > >>We might be on the right track with a problem around queue = > > > > >>depth > > > > however. If I run the kernel NVMe driver case with just 1 job = > > > > and queue depth of only 1, I then get 11.2K IOPS, which is = > > > > around the performance > > > I get > > > > in the SPDK test case. Perhaps the SPDK case is not actually = > > > > queuing > > > IOs on > > > > the device for some reason? > > > > >> > > > > >> > > > > >>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#' > > > > >>[global] > > > > >>rw=3Drandrw > > > > >>rwmixread=3D100 > > > > >>numjobs=3D1 > > > > >>iodepth=3D32 > > > > >>bs=3D4k > > > > >>direct=3D1 > > > > >>thread=3D1 > > > > >>time_based=3D1 > > > > >>ramp_time=3D0 > > > > >>runtime=3D10 > > > > >>ioengine=3Dlibaio > > > > >>group_reporting=3D1 > > > > >>unified_rw_reporting=3D1 > > > > >>exitall=3D1 > > > > >>randrepeat=3D0 > > > > >>norandommap=3D1 > > > > >>[nvme2n1] > > > > >>filename=3D/dev/nvme2n1 > > > > >> > > > > >> > > > > >>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini > > > > >>nvme2n1: (g=3D0): rw=3Drandrw, bs=3D(R) 4096B-4096B, (W) = > > > > >>4096B-4096B, > > > > >>(T) 4096B-4096B, ioengine=3Dlibaio, iodepth=3D32 > > > > >>fio-3.11 > > > > >>Starting 1 thread > > > > >>Jobs: 1 (f=3D1): [r(1)][100.0%][r=3D1025MiB/s][r=3D262k IOPS]= [eta = > > > > >>00m:00s] > > > > >>nvme2n1: (groupid=3D0, jobs=3D1): err=3D 0: pid=3D2422: Mon N= ov 5 > > > > >>05:19:13 > > > > >>2018 > > > > >> mixed: IOPS=3D254k, BW=3D992MiB/s > (1040MB/s)(9922MiB/10001msec) > > > > >> slat (nsec): min=3D1185, max=3D319052, avg=3D1611.64, std= ev=3D808.40 > > > > >> clat (usec): min=3D2, max=3D4743, avg=3D124.02, stdev=3D1= 21.00 > > > > >> lat (usec): min=3D6, max=3D4744, avg=3D125.68, stdev=3D1= 21.02 > > > > >> clat percentiles (usec): > > > > >> | 1.00th=3D[ 9], 5.00th=3D[ 56], 10.00th=3D[ 61= ], 20.00th=3D[ > > > 73], > > > > >> | 30.00th=3D[ 80], 40.00th=3D[ 91], 50.00th=3D[ 110= ], 60.00th=3D[ > > > 118], > > > > >> | 70.00th=3D[ 130], 80.00th=3D[ 157], 90.00th=3D[ 202= ], = > > > > >>95.00th=3D[ 253], > > > > >> | 99.00th=3D[ 408], 99.50th=3D[ 498], 99.90th=3D[ 1795= ], = > > > > >>99.95th=3D[ 3064], > > > > >> | 99.99th=3D[ 3785] > > > > >> bw ( KiB/s): min=3D612240, max=3D1078792, per=3D100.00%, = > > > > >>avg=3D1015996.40, > > > > stdev=3D106032.96, samples=3D20 > > > > >> iops : min=3D153060, max=3D269698, avg=3D253998.90, > > stdev=3D26508.40, > > > > samples=3D20 > > > > >> lat (usec) : 4=3D0.01%, 10=3D1.47%, 20=3D1.19%, 50=3D0.17= %, 100=3D42.03% > > > > >> lat (usec) : 250=3D49.95%, 500=3D4.70%, 750=3D0.28%, 1000= =3D0.06% > > > > >> lat (msec) : 2=3D0.06%, 4=3D0.09%, 10=3D0.01% > > > > >> cpu : usr=3D21.45%, sys=3D51.11%, ctx=3D530348, ma= jf=3D0, minf=3D139 > > > > >> IO depths : 1=3D0.1%, 2=3D0.1%, 4=3D0.1%, 8=3D0.1%, 16= =3D0.1%, > 32=3D100.0%, > > > > >=3D64=3D0.0% > > > > >> submit : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 3= 2=3D0.0%, > 64=3D0.0%, > > > > >=3D64=3D0.0% > > > > >> complete : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 3= 2=3D0.1%, = > > > > >> 64=3D0.0%, > > > > >>>=3D64=3D0.0% > > > > >> issued rwts: total=3D2540023,0,0,0 short=3D0,0,0,0 dropp= ed=3D0,0,0,0 > > > > >> latency : target=3D0, window=3D0, percentile=3D100.00%= , depth=3D32 > > > > >>Run status group 0 (all jobs): > > > > >> MIXED: bw=3D992MiB/s (1040MB/s), 992MiB/s-992MiB/s (1040MB/= s- > > > > 1040MB/s), io=3D9922MiB (10.4GB), run=3D10001-10001msec Disk st= ats > > > > (read/write): > > > > >> nvme2n1: ios=3D2514356/0, merge=3D0/0, ticks=3D400/0, > > in_queue=3D297612, > > > > >>util=3D99.06% > > > > >> > > > > >> > > > > >>don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#' > > > > >>[global] > > > > >>rw=3Drandrw > > > > >>rwmixread=3D100 > > > > >>numjobs=3D1 > > > > >>iodepth=3D32 > > > > >>bs=3D4k > > > > >>direct=3D1 > > > > >>thread=3D1 > > > > >>time_based=3D1 > > > > >>ramp_time=3D0 > > > > >>runtime=3D10 > > > > > > >>ioengine=3D/home/don/install/spdk/spdk/examples/nvme/fio_plugin/f= io_ > > > > >>pl > > > > ug > > > > >>in > > > > >>group_reporting=3D1 > > > > >>unified_rw_reporting=3D1 > > > > >>exitall=3D1 > > > > >>randrepeat=3D0 > > > > >>norandommap=3D1 > > > > >>[0000.1c.00.0-1] > > > > >>filename=3Dtrtype=3DPCIe traddr=3D0000.1c.00.0 ns=3D1 [0000.1= c.00.0-2] = > > > > >>filename=3Dtrtype=3DPCIe traddr=3D0000.1c.00.0 ns=3D1 [0000.1= c.00.0-3] = > > > > >>filename=3Dtrtype=3DPCIe traddr=3D0000.1c.00.0 ns=3D1 [0000.1= c.00.0-4] = > > > > >>filename=3Dtrtype=3DPCIe traddr=3D0000.1c.00.0 ns=3D1 > > > > >> > > > > >> > > > > >>don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini > > > > >>0000.1c.00.0-1: (g=3D0): rw=3Drandrw, bs=3D(R) 4096B-4096B, (= W) = > > > > >>4096B-4096B, > > > > >>(T) 4096B-4096B, ioengine=3Dspdk, iodepth=3D32 > > > > >>0000.1c.00.0-2: (g=3D0): rw=3Drandrw, bs=3D(R) 4096B-4096B, (= W) = > > > > >>4096B-4096B, > > > > >>(T) 4096B-4096B, ioengine=3Dspdk, iodepth=3D32 > > > > >>0000.1c.00.0-3: (g=3D0): rw=3Drandrw, bs=3D(R) 4096B-4096B, (= W) = > > > > >>4096B-4096B, > > > > >>(T) 4096B-4096B, ioengine=3Dspdk, iodepth=3D32 > > > > >>0000.1c.00.0-4: (g=3D0): rw=3Drandrw, bs=3D(R) 4096B-4096B, (= W) = > > > > >>4096B-4096B, > > > > >>(T) 4096B-4096B, ioengine=3Dspdk, iodepth=3D32 > > > > >>fio-3.11 > > > > >>Starting 4 threads > > > > >>Starting SPDK v18.10 / DPDK 18.08.0 initialization... > > > > >>[ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512 > > > > >>--file-prefix=3Dspdk_pid4270 ] > > > > >>EAL: Detected 36 lcore(s) > > > > >>EAL: Detected 2 NUMA nodes > > > > >>EAL: No free hugepages reported in hugepages-1048576kB > > > > >>EAL: Probing VFIO support... > > > > >>EAL: PCI device 0000:1c:00.0 on NUMA socket 0 > > > > >>EAL: probe driver: 8086:a54 spdk_nvme > > > > >>Jobs: 1 (f=3D0): [_(3),f(1)][100.0%][eta 00m:00s] > > > > >>0000.1c.00.0-1: (groupid=3D0, jobs=3D4): err=3D 0: pid=3D4310= : Mon Nov > > > > >>5 > > > > >>05:16:03 2018 > > > > >> mixed: IOPS=3D12.5k, BW=3D48.0MiB/s (51.4MB/s)(490MiB/10010= msec) > > > > >> slat (nsec): min=3D98, max=3D15092, avg=3D120.02, stdev= =3D90.07 > > > > >> clat (usec): min=3D55, max=3D19757, avg=3D10207.22, stdev= =3D884.06 > > > > >> lat (usec): min=3D60, max=3D19757, avg=3D10207.34, stdev= =3D884.06 > > > > >> clat percentiles (usec): > > > > >> | 1.00th=3D[ 8979], 5.00th=3D[ 9110], 10.00th=3D[ 9372= ], = > > > > >>20.00th=3D[ 9503], > > > > >> | 30.00th=3D[ 9765], 40.00th=3D[ 9896], 50.00th=3D[10159= ], = > > > > >>60.00th=3D[10290], > > > > >> | 70.00th=3D[10552], 80.00th=3D[10814], 90.00th=3D[11207= ], = > > > > >>95.00th=3D[11600], > > > > >> | 99.00th=3D[12911], 99.50th=3D[14353], 99.90th=3D[17695= ], = > > > > >>99.95th=3D[18744], > > > > >> | 99.99th=3D[19006] > > > > >> bw ( KiB/s): min=3D12008, max=3D12976, per=3D25.00%, = > > > > >>avg=3D12535.70, > > > > stdev=3D306.81, samples=3D80 > > > > >> iops : min=3D 3002, max=3D 3244, avg=3D3133.85, std= ev=3D76.77, > > > samples=3D80 > > > > >> lat (usec) : 100=3D0.01%, 250=3D0.01%, 500=3D0.01%, 750= =3D0.01%, > > 1000=3D0.01% > > > > >> lat (msec) : 2=3D0.01%, 4=3D0.02%, 10=3D45.13%, 20=3D54.8= 3% > > > > >> cpu : usr=3D100.01%, sys=3D0.00%, ctx=3D27, majf= =3D0, minf=3D0 > > > > >> IO depths : 1=3D0.1%, 2=3D0.1%, 4=3D0.1%, 8=3D0.1%, 16= =3D0.1%, 32=3D99.9%, > > > > >=3D64=3D0.0% > > > > >> submit : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 3= 2=3D0.0%, > 64=3D0.0%, > > > > >=3D64=3D0.0% > > > > >> complete : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 3= 2=3D0.1%, = > > > > >> 64=3D0.0%, > > > > >>>=3D64=3D0.0% > > > > >> issued rwts: total=3D125506,0,0,0 short=3D0,0,0,0 droppe= d=3D0,0,0,0 > > > > >> latency : target=3D0, window=3D0, percentile=3D100.00%= , depth=3D32 > > > > >>Run status group 0 (all jobs): > > > > >> MIXED: bw=3D48.0MiB/s (51.4MB/s), 48.0MiB/s-48.0MiB/s = > > > > >>(51.4MB/s-51.4MB/s), io=3D490MiB (514MB), run=3D10010-10010ms= ec > > > > >> > > > > >> > > > > >>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#' > > > > >>[global] > > > > >>rw=3Drandrw > > > > >>rwmixread=3D100 > > > > >>numjobs=3D1 > > > > >>iodepth=3D1 > > > > >>bs=3D4k > > > > >>direct=3D1 > > > > >>thread=3D1 > > > > >>time_based=3D1 > > > > >>ramp_time=3D0 > > > > >>runtime=3D10 > > > > >>ioengine=3Dlibaio > > > > >>group_reporting=3D1 > > > > >>unified_rw_reporting=3D1 > > > > >>exitall=3D1 > > > > >>randrepeat=3D0 > > > > >>norandommap=3D1 > > > > >>[nvme2n1] > > > > >>filename=3D/dev/nvme2n1 > > > > >> > > > > >> > > > > >>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini > > > > >>nvme2n1: (g=3D0): rw=3Drandrw, bs=3D(R) 4096B-4096B, (W) = > > > > >>4096B-4096B, > > > > >>(T) 4096B-4096B, ioengine=3Dlibaio, iodepth=3D1 > > > > >>fio-3.11 > > > > >>Starting 1 thread > > > > >>Jobs: 1 (f=3D1): [r(1)][100.0%][r=3D43.6MiB/s][r=3D11.2k IOPS= ][eta = > > > > >>00m:00s] > > > > >>nvme2n1: (groupid=3D0, jobs=3D1): err=3D 0: pid=3D2525: Mon N= ov 5 > > > > >>05:22:57 > > > > >>2018 > > > > >> mixed: IOPS=3D11.2k, BW=3D43.6MiB/s (45.7MB/s)(436MiB/10001= msec) > > > > >> slat (nsec): min=3D4848, max=3D208595, avg=3D5126.33, std= ev=3D771.27 > > > > >> clat (nsec): min=3D1845, max=3D3405.8k, avg=3D83004.99, > stdev=3D33577.36 > > > > >> lat (usec): min=3D15, max=3D3411, avg=3D88.26, stdev=3D3= 3.59 > > > > >> clat percentiles (usec): > > > > >> | 1.00th=3D[ 15], 5.00th=3D[ 58], 10.00th=3D[ 59= ], 20.00th=3D[ > > > 59], > > > > >> | 30.00th=3D[ 60], 40.00th=3D[ 77], 50.00th=3D[ 77= ], 60.00th=3D[ > > > 78], > > > > >> | 70.00th=3D[ 111], 80.00th=3D[ 113], 90.00th=3D[ 113= ], = > > > > >>95.00th=3D[ 115], > > > > >> | 99.00th=3D[ 186], 99.50th=3D[ 186], 99.90th=3D[ 190= ], = > > > > >>99.95th=3D[ 192], > > > > >> | 99.99th=3D[ 375] > > > > >> bw ( KiB/s): min=3D43816, max=3D45016, per=3D99.98%, = > > > > >>avg=3D44639.58, > > > > stdev=3D312.88, samples=3D19 > > > > >> iops : min=3D10954, max=3D11254, avg=3D11159.89, st= dev=3D78.22, > > > > samples=3D19 > > > > >> lat (usec) : 2=3D0.01%, 20=3D2.83%, 50=3D0.01%, 100=3D64.= 57%, > 250=3D32.59% > > > > >> lat (usec) : 500=3D0.01%, 750=3D0.01% > > > > >> lat (msec) : 2=3D0.01%, 4=3D0.01% > > > > >> cpu : usr=3D4.02%, sys=3D10.33%, ctx=3D111630, maj= f=3D0, minf=3D12 > > > > >> IO depths : 1=3D100.0%, 2=3D0.0%, 4=3D0.0%, 8=3D0.0%, 16= =3D0.0%, > 32=3D0.0%, > > > > >=3D64=3D0.0% > > > > >> submit : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 3= 2=3D0.0%, > 64=3D0.0%, > > > > >=3D64=3D0.0% > > > > >> complete : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 3= 2=3D0.0%, = > > > > >> 64=3D0.0%, > > > > >>>=3D64=3D0.0% > > > > >> issued rwts: total=3D111631,0,0,0 short=3D0,0,0,0 droppe= d=3D0,0,0,0 > > > > >> latency : target=3D0, window=3D0, percentile=3D100.00%= , depth=3D1 > > > > >>Run status group 0 (all jobs): > > > > >> MIXED: bw=3D43.6MiB/s (45.7MB/s), 43.6MiB/s-43.6MiB/s > (45.7MB/s- > > > > 45.7MB/s), io=3D436MiB (457MB), run=3D10001-10001msec Disk stats > > > > (read/write): > > > > >> nvme2n1: ios=3D110275/0, merge=3D0/0, ticks=3D0/0, in_queue= =3D8596, = > > > > >>util=3D86.22% > > > > >> > > > > >> > > > > >>Thanks, > > > > >>Joe > > > > >> > > > > >>> -----Original Message----- > > > > >>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf O= f = > > > > >>> Cao, Gang > > > > >>> Sent: Sunday, November 4, 2018 8:40 PM > > > > >>> To: Storage Performance Development Kit < spdk(a)lists.01.= org = > > > > >>> > > > > > >>> Subject: Re: [SPDK] FIO NVMe Performance Results > > > > >>> > > > > >>> For the SPDK plugin, from the below result, looks like only = > > > > >>> 1 job started with > > > > >>> 32 queue depth: > > > > >>> > > > > >>> Jobs: 1 (f=3D0): [f(1),_(3)][100.0%][eta 00m:00s] <<<< > > > > >>> > > > > >>> Try to update the FIO configuration file as > > > > >>> > > > > >>> [job1] > > > > >>> filename=3Dtrtype=3DPCIe traddr=3D0000.1c.00.0 ns=3D1 [job2= ] = > > > > >>> filename=3Dtrtype=3DPCIe > > > > >>> traddr=3D0000.1c.00.0 ns=3D1 .... > > > > >>> > > > > >>> The kernel FIO is with 4 jobs and each job of 32 queue dept= h. > > > > >>> For SPDK, you can configure as the same, or just configure = > > > > >>> two jobs and each job of 64 queue depth. The overall queue = > > > > >>> depth to the device is > > > > same of 128. > > > > >>> > > > > >>> Thanks, > > > > >>> Gang > > > > >>> > > > > >>> -----Original Message----- > > > > >>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf O= f = > > > > >>> Gruher, Joseph R > > > > >>> Sent: Monday, November 5, 2018 10:49 AM > > > > >>> To: Storage Performance Development Kit < spdk(a)lists.01.= org = > > > > >>> > > > > > >>> Subject: [SPDK] FIO NVMe Performance Results > > > > >>> > > > > >>> Hi folks- > > > > >>> > > > > >>> I'm testing SPDK 18.10 on Ubuntu 18.04 with kernel 4.18.16.= = > > > > >>> I used FIO with the kernel NVMe driver to measure the = > > > > >>> performance of a local (PCIe > > > > >>> attached) Intel P4500 NVMe device on a 4KB random read = > > > > >>> workload and obtained 477K IOPS, roughly in line with the = > > > > >>> drive > spec. > > > > >>> Then I tested the same drive with the SPDK FIO plugin and = > > > > >>> only achieved 13K IOPS. The FIO test files and the results = > > > > >>> are pasted below. Any ideas where I'm going wrong here? > > > > >>> > > > > >>> Thanks! > > > > >>> > > > > >>> > > > > >>> don(a)donst201:~/fio/single/rr$ ls = > > > > >>> /sys/devices/pci0000:17/0000:17:02.0/0000:1c:00.0/nvme/nvme1 > > > > >>> / address cntlid dev device firmware_rev model nvme2n= 1 = > > > > >>> power rescan_controller reset_controller serial state = > > > > >>> subsysnqn subsystem transport uevent > > > > >>> > > > > >>> > > > > >>> don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#' > > > > >>> [global] > > > > >>> rw=3Drandrw > > > > >>> rwmixread=3D100 > > > > >>> numjobs=3D4 > > > > >>> iodepth=3D32 > > > > >>> bs=3D4k > > > > >>> direct=3D1 > > > > >>> thread=3D1 > > > > >>> time_based=3D1 > > > > >>> ramp_time=3D0 > > > > >>> runtime=3D10 > > > > >>> ioengine=3Dlibaio > > > > >>> group_reporting=3D1 > > > > >>> unified_rw_reporting=3D1 > > > > >>> exitall=3D1 > > > > >>> randrepeat=3D0 > > > > >>> norandommap=3D1 > > > > >>> [nvme2n1] > > > > >>> filename=3D/dev/nvme2n1 > > > > >>> > > > > >>> > > > > >>> don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini > > > > >>> nvme2n1: (g=3D0): rw=3Drandrw, bs=3D(R) 4096B-4096B, (W) = > > > > >>> 4096B-4096B, > > > > >>> (T) 4096B-4096B, ioengine=3Dlibaio, iodepth=3D32 ... > > > > >>> fio-3.11 > > > > >>> Starting 4 threads > > > > >>> Jobs: 4 (f=3D4): [r(4)][100.0%][r=3D1874MiB/s][r=3D480k IOP= S][eta = > > > > >>> 00m:00s] > > > > >>> nvme2n1: (groupid=3D0, jobs=3D4): err=3D 0: pid=3D2575: Mon= Nov 5 > > > > >>> 02:26:25 > > > > 2018 > > > > >>> mixed: IOPS=3D477k, BW=3D1862MiB/s > > (1952MB/s)(18.2GiB/10001msec) > > > > >>> slat (nsec): min=3D1295, max=3D333088, avg=3D2216.35, s= tdev=3D927.89 > > > > >>> clat (nsec): min=3D506, max=3D5818.8k, avg=3D265924.16, > > stdev=3D223761.59 > > > > >>> lat (usec): min=3D6, max=3D5821, avg=3D268.19, stdev= =3D223.76 > > > > >>> clat percentiles (usec): > > > > >>> | 1.00th=3D[ 13], 5.00th=3D[ 74], 10.00th=3D[ = 87], 20.00th=3D[ > > > 115], > > > > >>> | 30.00th=3D[ 139], 40.00th=3D[ 167], 50.00th=3D[ 2= 04], = > > > > >>> 60.00th=3D[ > > > 247], > > > > >>> | 70.00th=3D[ 306], 80.00th=3D[ 388], 90.00th=3D[ 5= 23], = > > > > >>> 95.00th=3D[ > > > 660], > > > > >>> | 99.00th=3D[ 988], 99.50th=3D[ 1156], 99.90th=3D[ 23= 43], = > > > > >>> 99.95th=3D[ > > > 3195], > > > > >>> | 99.99th=3D[ 4424] > > > > >>> bw ( KiB/s): min=3D448192, max=3D483104, per=3D25.00%, = > > > > >>> avg=3D476597.10, stdev=3D6748.78, samples=3D80 > > > > >>> iops : min=3D112048, max=3D120776, avg=3D119149.2= 7, > > > stdev=3D1687.20, > > > > >>> samples=3D80 > > > > >>> lat (nsec) : 750=3D0.01% > > > > >>> lat (usec) : 10=3D0.52%, 20=3D1.25%, 50=3D0.25%, 100=3D= 13.38%, > > 250=3D45.13% > > > > >>> lat (usec) : 500=3D28.23%, 750=3D8.08%, 1000=3D2.22% > > > > >>> lat (msec) : 2=3D0.82%, 4=3D0.10%, 10=3D0.02% > > > > >>> cpu : usr=3D14.08%, sys=3D36.02%, ctx=3D1930207,= majf=3D0, > > > minf=3D697 > > > > >>> IO depths : 1=3D0.1%, 2=3D0.1%, 4=3D0.1%, 8=3D0.1%, 16= =3D0.1%, > > 32=3D100.0%, > > > > >>> >=3D64=3D0.0% > > > > >>> submit : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%,= 32=3D0.0%, > > 64=3D0.0%, > > > > >>> >=3D64=3D0.0% > > > > >>> complete : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%,= 32=3D0.1%, = > > > > >>> 64=3D0.0%, > > > > >>> >=3D64=3D0.0% > > > > >>> issued rwts: total=3D4766095,0,0,0 short=3D0,0,0,0 dro= pped=3D0,0,0,0 > > > > >>> latency : target=3D0, window=3D0, percentile=3D100.0= 0%, depth=3D32 > > > > >>> Run status group 0 (all jobs): > > > > >>> MIXED: bw=3D1862MiB/s (1952MB/s), 1862MiB/s-1862MiB/s > > (1952MB/s- > > > > >>> 1952MB/s), io=3D18.2GiB (19.5GB), run=3D10001-10001msec Dis= k = > > > > >>> stats > > > > >>> (read/write): > > > > >>> nvme2n1: ios=3D4709190/0, merge=3D0/0, ticks=3D4096/0, > > > > in_queue=3D1388752, > > > > >>> util=3D100.00% > > > > >>> > > > > >>> > > > > >>> don(a)donst201:~/fio/single/rr$ sudo = > > > > >>> /home/don/install/spdk/spdk/scripts/setup.sh > > > > >>> Active mountpoints on /dev/nvme0n1, so not binding PCI dev > > > > >>> 0000:03:00.0 > > > > >>> 0000:1c:00.0 (8086 0a54): nvme -> uio_pci_generic > > > > >>> 0000:1d:00.0 (8086 0a54): nvme -> uio_pci_generic > > > > >>> 0000:5e:00.0 (8086 0a54): nvme -> uio_pci_generic > > > > >>> 0000:5f:00.0 (8086 0a54): nvme -> uio_pci_generic > > > > >>> 0000:62:00.0 (8086 0a54): nvme -> uio_pci_generic > > > > >>> 0000:63:00.0 (8086 0a54): nvme -> uio_pci_generic > > > > >>> 0000:64:00.0 (8086 0a54): nvme -> uio_pci_generic > > > > >>> 0000:65:00.0 (8086 0a54): nvme -> uio_pci_generic > > > > >>> 0000:da:00.0 (8086 0a54): nvme -> uio_pci_generic > > > > >>> 0000:db:00.0 (8086 0a54): nvme -> uio_pci_generic > > > > >>> 0000:dc:00.0 (8086 0a54): nvme -> uio_pci_generic > > > > >>> 0000:dd:00.0 (8086 0a54): nvme -> uio_pci_generic > > > > >>> 0000:e0:00.0 (8086 0a54): nvme -> uio_pci_generic > > > > >>> 0000:e1:00.0 (8086 0a54): nvme -> uio_pci_generic > > > > >>> 0000:e2:00.0 (8086 0a54): nvme -> uio_pci_generic > > > > >>> 0000:e3:00.0 (8086 0a54): nvme -> uio_pci_generic > > > > >>> 0000:00:04.0 (8086 2021): ioatdma -> uio_pci_generic > > > > >>> 0000:00:04.1 (8086 2021): ioatdma -> uio_pci_generic > > > > >>> 0000:00:04.2 (8086 2021): ioatdma -> uio_pci_generic > > > > >>> 0000:00:04.3 (8086 2021): ioatdma -> uio_pci_generic > > > > >>> 0000:00:04.4 (8086 2021): ioatdma -> uio_pci_generic > > > > >>> 0000:00:04.5 (8086 2021): ioatdma -> uio_pci_generic > > > > >>> 0000:00:04.6 (8086 2021): ioatdma -> uio_pci_generic > > > > >>> 0000:00:04.7 (8086 2021): ioatdma -> uio_pci_generic > > > > >>> 0000:80:04.0 (8086 2021): ioatdma -> uio_pci_generic > > > > >>> 0000:80:04.1 (8086 2021): ioatdma -> uio_pci_generic > > > > >>> 0000:80:04.2 (8086 2021): ioatdma -> uio_pci_generic > > > > >>> 0000:80:04.3 (8086 2021): ioatdma -> uio_pci_generic > > > > >>> 0000:80:04.4 (8086 2021): ioatdma -> uio_pci_generic > > > > >>> 0000:80:04.5 (8086 2021): ioatdma -> uio_pci_generic > > > > >>> 0000:80:04.6 (8086 2021): ioatdma -> uio_pci_generic > > > > >>> 0000:80:04.7 (8086 2021): ioatdma -> uio_pci_generic > > > > >>> > > > > >>> > > > > >>> don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#' > > > > >>> [global] > > > > >>> rw=3Drandrw > > > > >>> rwmixread=3D100 > > > > >>> numjobs=3D4 > > > > >>> iodepth=3D32 > > > > >>> bs=3D4k > > > > >>> direct=3D1 > > > > >>> thread=3D1 > > > > >>> time_based=3D1 > > > > >>> ramp_time=3D0 > > > > >>> runtime=3D10 > > > > >>> > > > > > > ioengine=3D/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio= _pl > > > > >>> ug > > > > >>> in > > > > >>> group_reporting=3D1 > > > > >>> unified_rw_reporting=3D1 > > > > >>> exitall=3D1 > > > > >>> randrepeat=3D0 > > > > >>> norandommap=3D1 > > > > >>> [0000.1c.00.0] > > > > >>> filename=3Dtrtype=3DPCIe traddr=3D0000.1c.00.0 ns=3D1 > > > > >>> > > > > >>> > > > > >>> don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini > > > > >>> 0000.1c.00.0: (g=3D0): rw=3Drandrw, bs=3D(R) 4096B-4096B, (= W) = > > > > >>> 4096B-4096B, > > > > >>> (T) 4096B-4096B, ioengine=3Dspdk, iodepth=3D32 ... > > > > >>> fio-3.11 > > > > >>> Starting 4 threads > > > > >>> Starting SPDK v18.10 / DPDK 18.08.0 initialization... > > > > >>> [ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512 --file- > > > > >>> prefix=3Dspdk_pid3668 ] > > > > >>> EAL: Detected 36 lcore(s) > > > > >>> EAL: Detected 2 NUMA nodes > > > > >>> EAL: No free hugepages reported in hugepages-1048576kB > > > > >>> EAL: Probing VFIO support... > > > > >>> EAL: PCI device 0000:1c:00.0 on NUMA socket 0 > > > > >>> EAL: probe driver: 8086:a54 spdk_nvme > > > > >>> Jobs: 1 (f=3D0): [f(1),_(3)][100.0%][eta 00m:00s] > > > > >>> 0000.1c.00.0: (groupid=3D0, jobs=3D4): err=3D 0: pid=3D3709= : Mon Nov > > > > >>> 5 > > > > >>> 02:28:29 > > > > >>> 2018 > > > > >>> mixed: IOPS=3D13.2k, BW=3D51.6MiB/s > (54.1MB/s)(517MiB/10011msec) > > > > >>> slat (nsec): min=3D109, max=3D15344, avg=3D127.91, stde= v=3D73.44 > > > > >>> clat (usec): min=3D187, max=3D18715, avg=3D9683.80, std= ev=3D650.54 > > > > >>> lat (usec): min=3D193, max=3D18715, avg=3D9683.93, std= ev=3D650.53 > > > > >>> clat percentiles (usec): > > > > >>> | 1.00th=3D[ 8717], 5.00th=3D[ 8848], 10.00th=3D[ 89= 79], = > > > > >>> 20.00th=3D[ > > > 9110], > > > > >>> | 30.00th=3D[ 9241], 40.00th=3D[ 9372], 50.00th=3D[ 95= 03], = > > > > >>> 60.00th=3D[ > > > 9765], > > > > >>> | 70.00th=3D[10028], 80.00th=3D[10159], 90.00th=3D[105= 52], > > > > 95.00th=3D[10945], > > > > >>> | 99.00th=3D[11338], 99.50th=3D[11338], 99.90th=3D[117= 31], > > > > 99.95th=3D[13960], > > > > >>> | 99.99th=3D[17695] > > > > >>> bw ( KiB/s): min=3D13080, max=3D13304, per=3D25.00%, = > > > > >>> avg=3D13214.70, stdev=3D55.30, samples=3D80 > > > > >>> iops : min=3D 3270, max=3D 3326, avg=3D3303.65, s= tdev=3D13.84, > > > > samples=3D80 > > > > >>> lat (usec) : 250=3D0.01%, 500=3D0.01%, 750=3D0.01%, 100= 0=3D0.01% > > > > >>> lat (msec) : 2=3D0.01%, 4=3D0.02%, 10=3D70.85%, 20=3D29= .11% > > > > >>> cpu : usr=3D100.01%, sys=3D0.00%, ctx=3D23, majf= =3D0, minf=3D0 > > > > >>> IO depths : 1=3D0.1%, 2=3D0.1%, 4=3D0.1%, 8=3D0.1%, 16= =3D0.1%, > 32=3D99.9%, > > > > >>> >=3D64=3D0.0% > > > > >>> submit : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%,= 32=3D0.0%, > > 64=3D0.0%, > > > > >>> >=3D64=3D0.0% > > > > >>> complete : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%,= 32=3D0.1%, = > > > > >>> 64=3D0.0%, > > > > >>> >=3D64=3D0.0% > > > > >>> issued rwts: total=3D132300,0,0,0 short=3D0,0,0,0 drop= ped=3D0,0,0,0 > > > > >>> latency : target=3D0, window=3D0, percentile=3D100.0= 0%, depth=3D32 > > > > >>> Run status group 0 (all jobs): > > > > >>> MIXED: bw=3D51.6MiB/s (54.1MB/s), 51.6MiB/s-51.6MiB/s > > (54.1MB/s- > > > > >>> 54.1MB/s), io=3D517MiB (542MB), run=3D10011-10011msec > > > > >>> > > > > >>> _______________________________________________ > > > > >>> SPDK mailing list > > > > >>> SPDK(a)lists.01.org > > > > >>> https://lists.01.org/mailman/listinfo/spdk > > > > >>> _______________________________________________ > > > > >>> SPDK mailing list > > > > >>> SPDK(a)lists.01.org > > > > >>> https://lists.01.org/mailman/listinfo/spdk > > > > >>_______________________________________________ > > > > >>SPDK mailing list > > > > >> SPDK(a)lists.01.org > > > > >> https://lists.01.org/mailman/listinfo/spdk > > > > >>_______________________________________________ > > > > >>SPDK mailing list > > > > >> SPDK(a)lists.01.org > > > > >> https://lists.01.org/mailman/listinfo/spdk > > > > > > > > > > > > > > >-- > > > > >=D0=9C=D0=B0=D1=80=D0=BE=D0=B2 =D0=90=D0=BB=D0=B5=D0=BA=D1=81= =D0=B5=D0=B9 > > > > >_______________________________________________ > > > > >SPDK mailing list > > > > >SPDK(a)lists.01.org > > > > >https://lists.01.org/mailman/listinfo/spdk > > > > >_______________________________________________ > > > > >SPDK mailing list > > > > >SPDK(a)lists.01.org > > > > >https://lists.01.org/mailman/listinfo/spdk > > > > > > > > > > > > -- > > > > =D0=9C=D0=B0=D1=80=D0=BE=D0=B2 =D0=90=D0=BB=D0=B5=D0=BA=D1=81= =D0=B5=D0=B9 > > > > _______________________________________________ > > > > SPDK mailing list > > > > SPDK(a)lists.01.org > > > > https://lists.01.org/mailman/listinfo/spdk > > > _______________________________________________ > > > SPDK mailing list > > > SPDK(a)lists.01.org > > > https://lists.01.org/mailman/listinfo/spdk > > > > > -- > > > > Regards, > > Andrey > > _______________________________________________ > > SPDK mailing list > > SPDK(a)lists.01.org > > https://lists.01.org/mailman/listinfo/spdk > > _______________________________________________ > > SPDK mailing list > > SPDK(a)lists.01.org > > https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk = --===============6436113331615179505==--