From: Harris, James R <james.r.harris at intel.com>
To: spdk@lists.01.org
Subject: Re: [SPDK] FIO NVMe Performance Results
Date: Mon, 05 Nov 2018 21:56:29 +0000 [thread overview]
Message-ID: <E5866F0D-BCD3-4D82-8FC3-CCF883E62517@intel.com> (raw)
In-Reply-To: EA913ED399BBA34AA4EAC2EDC24CDD009C2166C9@FMSMSX105.amr.corp.intel.com
[-- Attachment #1: Type: text/plain, Size: 46705 bytes --]
Hi Joe,
This is all just local PCIe NVMe – no NVMe-oF? Can you make sure you don’t have any other SPDK processes running while you’re doing this? I’m concerned that fio is polling on the same CPU core as an SPDK process, and your performance degradation is due to timeslice sharing.
-Jim
On 11/5/18, 2:42 PM, "SPDK on behalf of Howell, Seth" <spdk-bounces(a)lists.01.org on behalf of seth.howell(a)intel.com> wrote:
I think that the uio_pci_generic vs. vfio thing is unrelated. The only reason I brought it up was because it could have been indicative of a larger configuration difference. However, I ran my test with and without vfio support just to be sure and got the higher performance numbers both times.
As a side note:
It's probably a good thing to have enabled in general when running spdk benchmarks because that is the recommended configuration on any physical machine.
You can enable it in grub by adding the following to the GRUB_CMDLINE_LINUX line of /etc/default.grub: intel_iommu=on
and then running grub2-mkconfig.
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Gruher, Joseph R
Sent: Monday, November 5, 2018 1:39 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] FIO NVMe Performance Results
Attempting to fall back to earlier revisions has not yielded any improvements for me. All tests use the same P4500 physical disk. FIO files are as shown before, running a 4 thread, queue depth 32 per thread, 4KB random read workload.
SPDK 18.10, kernel 4.18.16, FIO 3.11 = 12.9k IOPS SPDK 18.10, kernel 4.16.14, FIO 3.11 = 11.9k IOPS SPDK 18.04, kernel 4.18.16, FIO 3.11 = 12.9k IOPS SPDK 18.04, kernel 4.16.14, FIO 3.11 = 13.0k IOPS SPDK 18.04, kernel 4.16.16, FIO 3.3 = 13.1k IOPS
Kernel NVMe driver, kernel 4.16.16, FIO 3.3 = 477k IOPS Kernel NVMe driver, kernel 4.18.16, FIO 3.11 = 477k IOPS
I am curious if the comment below about uio_pci_generic instead of vfio can lead us anywhere, or maybe there's some OS setting or BIOS setting that needs to be toggled...
For the folks working inside Intel, I can also give you direct access to the test system, if that would help.
Thanks,
Joe
> -----Original Message-----
> From: Gruher, Joseph R
> Sent: Monday, November 5, 2018 11:05 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: RE: [SPDK] FIO NVMe Performance Results
>
> I've had good FIO NVMe results in the past we well, with SPDK 18.04 on
> Ubuntu 18.04, with kernel 4.16.14. That was on a different, but
> similar, hardware platform.
>
> I suspect something has changed now in SPDK 18.10 and/or kernel
> 4.18.16. I can try going back to earlier versions of each to see if I
> can recreate earlier good results and find where things went wrong.
>
> We're not running in a VM - this is a bare metal Xeon server with
> Skylake CPUs.
>
> Thanks,
> Joe
>
> > -----Original Message-----
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Howell,
> > Seth
> > Sent: Monday, November 5, 2018 10:32 AM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: Re: [SPDK] FIO NVMe Performance Results
> >
> > Hi Joseph,
> > I attempted to verify your results on my system. I am using fedora
> > 27 with kernel version 4.17.3 with a p3700 drive. I cloned fio from
> > https://github.com/axboe/fio and checked out the appropriate tag. I
> > got appx 408k iops.
> > Typically we run spdk with the iommu enabled. I noticed that your
> > devices are binding to uio_pci_generic instead of vfio. This
> > shouldn't make any difference at all, but I just want to make sure
> > you aren't running in a VM or something like that and passing
> > through your NVMe drives and that's why you aren't using the IOMMU.
> > That could explain a deeper configuration difference that could help
> > us know where to look for
> the disparity.
> >
> > Please see the contents of my configuration files and output below:
> >
> > Test.ini:
> > [global]
> > rw=randrw
> > rwmixread=100
> > numjobs=4
> > iodepth=32
> > bs=4k
> > direct=1
> > thread=1
> > time_based=1
> > ramp_time=0
> > runtime=10
> > ioengine=spdk_bdev
> > spdk_conf=./nvme.conf
> > group_reporting=1
> > unified_rw_reporting=1
> > exitall=1
> > randrepeat=0
> > norandommap=1
> > [0000.81.00.0]
> > filename=Nvme0n1
> >
> > nvme.conf:
> > [Nvme]
> > TransportId "trtype:PCIe traddr:0000.81.00.0" Nvme0
> >
> > Sudo
> >
> LD_PRELOAD=/home/sethhowe/Desktop/Development_Folders/spdk/exam
> > ples/bdev/fio_plugin/fio_plugin ./fio ./test.ini >./out,txt 2>&1
> >
> > Out.txt:
> > 0000.81.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B,
> > (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=32 ...
> > fio-3.11
> > Starting 4 threads
> > Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> > [ DPDK EAL parameters: fio --no-shconf -c 0x1
> > --file-prefix=spdk_pid782339 ]
> > EAL: Detected 16 lcore(s)
> > EAL: Detected 2 NUMA nodes
> > EAL: No free hugepages reported in hugepages-1048576kB
> > EAL: Probing VFIO support...
> > EAL: VFIO support initialized
> > EAL: PCI device 0000:81:00.0 on NUMA socket 1
> > EAL: probe driver: 8086:953 spdk_nvme
> >
> > 0000.81.00.0: (groupid=0, jobs=4): err= 0: pid=782359: Mon Nov 5
> > 10:58:59
> > 2018
> > mixed: IOPS=410k, BW=1600MiB/s (1678MB/s)(15.6GiB/10001msec)
> > slat (nsec): min=179, max=19985, avg=245.97, stdev=122.96
> > clat (usec): min=8, max=1821, avg=311.82, stdev=167.14
> > lat (usec): min=8, max=1822, avg=312.06, stdev=167.14
> > clat percentiles (usec):
> > | 1.00th=[ 24], 5.00th=[ 48], 10.00th=[ 78], 20.00th=[ 141],
> > | 30.00th=[ 200], 40.00th=[ 258], 50.00th=[ 314], 60.00th=[ 371],
> > | 70.00th=[ 429], 80.00th=[ 486], 90.00th=[ 545], 95.00th=[ 570],
> > | 99.00th=[ 594], 99.50th=[ 594], 99.90th=[ 627], 99.95th=[ 652],
> > | 99.99th=[ 676]
> > bw ( KiB/s): min=407264, max=416040, per=25.01%, avg=409742.55,
> > stdev=1337.38, samples=76
> > iops : min=101816, max=104010, avg=102435.61, stdev=334.41,
> > samples=76
> > lat (usec) : 10=0.01%, 20=0.46%, 50=4.88%, 100=8.41%, 250=25.29%
> > lat (usec) : 500=43.15%, 750=17.81%, 1000=0.01%
> > lat (msec) : 2=0.01%
> > cpu : usr=99.78%, sys=0.00%, ctx=26, majf=0, minf=0
> > IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
> > >=64=0.0%
> > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> > >=64=0.0%
> > complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.1%, 32=0.1%,
> > 64=0.0%,
> > >=64=0.0%
> > issued rwts: total=4096867,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > latency : target=0, window=0, percentile=100.00%, depth=32
> >
> > Run status group 0 (all jobs):
> > MIXED: bw=1600MiB/s (1678MB/s), 1600MiB/s-1600MiB/s (1678MB/s-
> > 1678MB/s), io=15.6GiB (16.8GB), run=10001-10001msec
> >
> > Thanks,
> >
> > Seth
> >
> >
> >
> > -----Original Message-----
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey
> > Kuzmin
> > Sent: Monday, November 5, 2018 9:48 AM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: Re: [SPDK] FIO NVMe Performance Results
> >
> > On Mon, Nov 5, 2018, 18:29 Gruher, Joseph R
> > <joseph.r.gruher(a)intel.com>
> > wrote:
> >
> > > My FIO version is v3.11.
> > >
> > > I tried the bdev FIO plugin as suggested, but performance is about
> > > the same as the other SPDK FIO plugin, I observed only 10.4K IOPS
> > > in a quick 4KB random read test. See configuration and results below.
> > >
> >
> > As 12 ms median read latency for an nvme ssd is far beyond anything
> > reasonable, why don't you just perf this run, to see where the time
> > is being spent?
> >
> > Regards,
> > Andrey
> >
> > >
> > >
> > > don(a)donst201:~/fio/single/rr$ cat nvme.conf [Nvme] TransportId
> > > "trtype:PCIe traddr:0000.1c.00.0" Nvme0
> > >
> > >
> > > don(a)donst201:~/fio/single/rr$ cat 1c-bdev.ini |grep -v '#'
> > > [global]
> > > rw=randrw
> > > rwmixread=100
> > > numjobs=4
> > > iodepth=32
> > > bs=4k
> > > direct=1
> > > thread=1
> > > time_based=1
> > > ramp_time=0
> > > runtime=10
> > > ioengine=spdk_bdev
> > > spdk_conf=./nvme.conf
> > > group_reporting=1
> > > unified_rw_reporting=1
> > > exitall=1
> > > randrepeat=0
> > > norandommap=1
> > > [0000.1c.00.0]
> > > filename=Nvme0n1
> > >
> > >
> > > don(a)donst201:~/fio/single/rr$ sudo
> > >
> >
> LD_PRELOAD=/home/don/install/spdk/spdk/examples/bdev/fio_plugin/fio_p
> > l
> > > ugin
> > > fio 1c-bdev.ini
> > > 0000.1c.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > 4096B-4096B,
> > > (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=32 ...
> > > fio-3.11
> > > Starting 4 threads
> > > Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> > > [ DPDK EAL parameters: fio --no-shconf -c 0x1
> > > --file-prefix=spdk_pid4865 ]
> > > EAL: Detected 36 lcore(s)
> > > EAL: Detected 2 NUMA nodes
> > > EAL: No free hugepages reported in hugepages-1048576kB
> > > EAL: Probing VFIO support...
> > > EAL: PCI device 0000:03:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:f1a5 spdk_nvme
> > > EAL: PCI device 0000:1c:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:1d:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:5e:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:5f:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:62:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:63:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:64:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:65:00.0 on NUMA socket 0
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:da:00.0 on NUMA socket 1
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:db:00.0 on NUMA socket 1
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:dc:00.0 on NUMA socket 1
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:dd:00.0 on NUMA socket 1
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:e0:00.0 on NUMA socket 1
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:e1:00.0 on NUMA socket 1
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:e2:00.0 on NUMA socket 1
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > EAL: PCI device 0000:e3:00.0 on NUMA socket 1
> > > EAL: probe driver: 8086:a54 spdk_nvme
> > > Jobs: 4 (f=4): [r(4)][100.0%][r=45.7MiB/s][r=11.7k IOPS][eta
> > > 00m:00s]
> > > 0000.1c.00.0: (groupid=0, jobs=4): err= 0: pid=4906: Mon Nov 5
> > > 15:23:47
> > > 2018
> > > mixed: IOPS=10.4k, BW=40.5MiB/s (42.5MB/s)(406MiB/10012msec)
> > > slat (nsec): min=201, max=15831, avg=238.11, stdev=148.78
> > > clat (usec): min=124, max=31244, avg=12330.69, stdev=1697.98
> > > lat (usec): min=130, max=31244, avg=12330.93, stdev=1697.98
> > > clat percentiles (usec):
> > > | 1.00th=[ 9634], 5.00th=[10159], 10.00th=[10552], 20.00th=[10945],
> > > | 30.00th=[11338], 40.00th=[11731], 50.00th=[12125],
> > 60.00th=[12518],
> > > | 70.00th=[13042], 80.00th=[13566], 90.00th=[14353],
> > 95.00th=[15008],
> > > | 99.00th=[16712], 99.50th=[18744], 99.90th=[27395],
> > 99.95th=[29754],
> > > | 99.99th=[30278]
> > > bw ( KiB/s): min= 8784, max=11952, per=24.99%, avg=10375.53,
> > > stdev=929.68, samples=80
> > > iops : min= 2196, max= 2988, avg=2593.85, stdev=232.41,
> > > samples=80
> > > lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
> > > lat (msec) : 2=0.01%, 4=0.02%, 10=2.71%, 20=96.91%, 50=0.33%
> > > cpu : usr=99.99%, sys=0.01%, ctx=28, majf=0, minf=0
> > > IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
> > > >=64=0.0%
> > > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> > > >=64=0.0%
> > > complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > > 64=0.0%,
> > > >=64=0.0%
> > > issued rwts: total=103907,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > latency : target=0, window=0, percentile=100.00%, depth=32
> > > Run status group 0 (all jobs):
> > > MIXED: bw=40.5MiB/s (42.5MB/s), 40.5MiB/s-40.5MiB/s
> > > (42.5MB/s-42.5MB/s), io=406MiB (426MB), run=10012-10012msec
> > >
> > >
> > > Thanks,
> > > Joe
> > >
> > > > -----Original Message-----
> > > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of ?????
> > ???????
> > > > Sent: Monday, November 5, 2018 12:45 AM
> > > > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > > > Subject: Re: [SPDK] FIO NVMe Performance Results
> > > >
> > > > Ok, Do you see some performance issues using general bdev fio
> > > > plugin
> > > with
> > > > you configuration ?
> > > >
> > > > Best Regards
> > > > Alex
> > > >
> > > >
> > > > >Понедельник, 5 ноября 2018, 11:39 +03:00 от "Cao, Gang"
> > > > <gang.cao(a)intel.com>:
> > > > >
> > > > >Yes. This is for the bdev version of SPDK FIO plugin and you
> > > > >can assign
> > > the
> > > > conf file as the parameter in the FIO configuration.
> > > > >
> > > > >Previously, it's another SPDK FIO plugin for the user space
> > > > >NVMe driver
> > > > only.
> > > > >
> > > > >These are two different FIO plugins.
> > > > >
> > > > >Thanks,
> > > > >Gang
> > > > >
> > > > >-----Original Message-----
> > > > >From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of
> > > > >Маров Алексей
> > > > >Sent: Monday, November 5, 2018 4:32 PM
> > > > >To: Storage Performance Development Kit < spdk(a)lists.01.org >
> > > > >Subject: Re: [SPDK] FIO NVMe Performance Results
> > > > >
> > > > >Sorry for incomplete message I have sent before
> > > > >
> > > > >Hello Gang.
> > > > >
> > > > >I just curious, why would you run fio with SPDK using this conf
> > > > >file,
> > > with is a
> > > > bit different from what is suggested in here
> > https://github.com/spdk/spdk/tree/master/examples/bdev/fio_plugin
> > > > .
> > > > >
> > > > >I mean may be you should create conf file for spdk like this one :
> > > > >
> > > > >nvme.conf :
> > > > >[Nvme]
> > > > > TransportId "trtype:PCIe traddr:0000.1c.00.0" Nvme0
> > > > >
> > > > >and provide it in fio config.
> > > > >
> > > > >fio.ini :
> > > > >[global]
> > > > >ioengine=spdk_bdev
> > > > >spdk_conf=....../nvme.conf
> > > > >rw=randrw
> > > > >rwmixread=100
> > > > >numjobs=4
> > > > >iodepth=32
> > > > >bs=4k
> > > > >direct=1
> > > > >thread=1
> > > > >time_based=1
> > > > >ramp_time=0
> > > > >runtime=10
> > > > >group_reporting=1
> > > > >unified_rw_reporting=1
> > > > >exitall=1
> > > > >randrepeat=0
> > > > >norandommap=1
> > > > >
> > > > >[0000:06:00.0]
> > > > >filename=Nvme0n1
> > > > >
> > > > >and run like this
> > > > >
> > > > ># LD_PRELOAD=examples/bdev/fio_plugin/fio_plugin fio
> > > > >examples/bdev/fio_plugin/fio.ini
> > > > >
> > > > >Just to note. The recommended version of fio for SPDK is 3.3
> > > > >
> > > > >Best Regards
> > > > >Alex
> > > > >
> > > > >
> > > > >>Понедельник, 5 ноября 2018, 9:53 +03:00 от "Cao, Gang" <
> > > > gang.cao(a)intel.com >:
> > > > >>
> > > > >>Thanks. This is interesting especially for your finding on
> > > > >>queue depth
> > > 1
> > > > case. We are trying to reproduce this issue locally same as your
> > > > configuration.
> > > > >>
> > > > >>Thanks,
> > > > >>Gang
> > > > >>
> > > > >>-----Original Message-----
> > > > >>From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of
> > > > >>Gruher, Joseph R
> > > > >>Sent: Monday, November 5, 2018 1:32 PM
> > > > >>To: Storage Performance Development Kit < spdk(a)lists.01.org >
> > > > >>Subject: Re: [SPDK] FIO NVMe Performance Results
> > > > >>
> > > > >>Interesting catch. It seems like FIO is actually not
> > > > >>accurately
> > > reporting the
> > > > number of jobs in the report out for the SPDK case. If I
> > > > disable the group_reporting parameter I do get a printout for
> > > > four separate jobs in
> > > the
> > > > SPDK case, so I think it is really running four jobs there, and
> > > > just
> > > summarizing
> > > > incorrectly. I also see four separate CPU cores go to 100% when
> > > > I run
> > > that
> > > > FIO test case, indicating four jobs are active.
> > > > >>
> > > > >>Also, I don't think would explain the difference in performance.
> > > Using an
> > > > aggregate queue depth of 32 versus 128 shouldn't lead to a drop
> > > > in performance from 377K to 13K.
> > > > >>
> > > > >>I retested SPDK with multiple separate job descriptions as you
> > > suggested
> > > > anyway, using four separate jobs instead of relying on the
> > > > numjobs=4 parameter, and the results are below. In that SPDK
> > > > test with four
> > > separate
> > > > FIO job statements, each with queue depth 32, I still only
> > > > achieved 12.5K IOPS.
> > > > >>
> > > > >>I then re-ran the kernel NVMe driver test case with only a
> > > > >>single job
> > > for
> > > > comparison, and even with only 1 job and queue depth 32 it still
> > > > achieves 254K IOPS.
> > > > >>
> > > > >>We might be on the right track with a problem around queue
> > > > >>depth
> > > > however. If I run the kernel NVMe driver case with just 1 job
> > > > and queue depth of only 1, I then get 11.2K IOPS, which is
> > > > around the performance
> > > I get
> > > > in the SPDK test case. Perhaps the SPDK case is not actually
> > > > queuing
> > > IOs on
> > > > the device for some reason?
> > > > >>
> > > > >>
> > > > >>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
> > > > >>[global]
> > > > >>rw=randrw
> > > > >>rwmixread=100
> > > > >>numjobs=1
> > > > >>iodepth=32
> > > > >>bs=4k
> > > > >>direct=1
> > > > >>thread=1
> > > > >>time_based=1
> > > > >>ramp_time=0
> > > > >>runtime=10
> > > > >>ioengine=libaio
> > > > >>group_reporting=1
> > > > >>unified_rw_reporting=1
> > > > >>exitall=1
> > > > >>randrepeat=0
> > > > >>norandommap=1
> > > > >>[nvme2n1]
> > > > >>filename=/dev/nvme2n1
> > > > >>
> > > > >>
> > > > >>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
> > > > >>nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > >>4096B-4096B,
> > > > >>(T) 4096B-4096B, ioengine=libaio, iodepth=32
> > > > >>fio-3.11
> > > > >>Starting 1 thread
> > > > >>Jobs: 1 (f=1): [r(1)][100.0%][r=1025MiB/s][r=262k IOPS][eta
> > > > >>00m:00s]
> > > > >>nvme2n1: (groupid=0, jobs=1): err= 0: pid=2422: Mon Nov 5
> > > > >>05:19:13
> > > > >>2018
> > > > >> mixed: IOPS=254k, BW=992MiB/s
> (1040MB/s)(9922MiB/10001msec)
> > > > >> slat (nsec): min=1185, max=319052, avg=1611.64, stdev=808.40
> > > > >> clat (usec): min=2, max=4743, avg=124.02, stdev=121.00
> > > > >> lat (usec): min=6, max=4744, avg=125.68, stdev=121.02
> > > > >> clat percentiles (usec):
> > > > >> | 1.00th=[ 9], 5.00th=[ 56], 10.00th=[ 61], 20.00th=[
> > > 73],
> > > > >> | 30.00th=[ 80], 40.00th=[ 91], 50.00th=[ 110], 60.00th=[
> > > 118],
> > > > >> | 70.00th=[ 130], 80.00th=[ 157], 90.00th=[ 202],
> > > > >>95.00th=[ 253],
> > > > >> | 99.00th=[ 408], 99.50th=[ 498], 99.90th=[ 1795],
> > > > >>99.95th=[ 3064],
> > > > >> | 99.99th=[ 3785]
> > > > >> bw ( KiB/s): min=612240, max=1078792, per=100.00%,
> > > > >>avg=1015996.40,
> > > > stdev=106032.96, samples=20
> > > > >> iops : min=153060, max=269698, avg=253998.90,
> > stdev=26508.40,
> > > > samples=20
> > > > >> lat (usec) : 4=0.01%, 10=1.47%, 20=1.19%, 50=0.17%, 100=42.03%
> > > > >> lat (usec) : 250=49.95%, 500=4.70%, 750=0.28%, 1000=0.06%
> > > > >> lat (msec) : 2=0.06%, 4=0.09%, 10=0.01%
> > > > >> cpu : usr=21.45%, sys=51.11%, ctx=530348, majf=0, minf=139
> > > > >> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%,
> 32=100.0%,
> > > > >=64=0.0%
> > > > >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> 64=0.0%,
> > > > >=64=0.0%
> > > > >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > > > >> 64=0.0%,
> > > > >>>=64=0.0%
> > > > >> issued rwts: total=2540023,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > > >> latency : target=0, window=0, percentile=100.00%, depth=32
> > > > >>Run status group 0 (all jobs):
> > > > >> MIXED: bw=992MiB/s (1040MB/s), 992MiB/s-992MiB/s (1040MB/s-
> > > > 1040MB/s), io=9922MiB (10.4GB), run=10001-10001msec Disk stats
> > > > (read/write):
> > > > >> nvme2n1: ios=2514356/0, merge=0/0, ticks=400/0,
> > in_queue=297612,
> > > > >>util=99.06%
> > > > >>
> > > > >>
> > > > >>don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
> > > > >>[global]
> > > > >>rw=randrw
> > > > >>rwmixread=100
> > > > >>numjobs=1
> > > > >>iodepth=32
> > > > >>bs=4k
> > > > >>direct=1
> > > > >>thread=1
> > > > >>time_based=1
> > > > >>ramp_time=0
> > > > >>runtime=10
> > > >
> > >>ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_
> > > > >>pl
> > > > ug
> > > > >>in
> > > > >>group_reporting=1
> > > > >>unified_rw_reporting=1
> > > > >>exitall=1
> > > > >>randrepeat=0
> > > > >>norandommap=1
> > > > >>[0000.1c.00.0-1]
> > > > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-2]
> > > > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-3]
> > > > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-4]
> > > > >>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
> > > > >>
> > > > >>
> > > > >>don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
> > > > >>0000.1c.00.0-1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > >>4096B-4096B,
> > > > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > > > >>0000.1c.00.0-2: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > >>4096B-4096B,
> > > > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > > > >>0000.1c.00.0-3: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > >>4096B-4096B,
> > > > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > > > >>0000.1c.00.0-4: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > >>4096B-4096B,
> > > > >>(T) 4096B-4096B, ioengine=spdk, iodepth=32
> > > > >>fio-3.11
> > > > >>Starting 4 threads
> > > > >>Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> > > > >>[ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512
> > > > >>--file-prefix=spdk_pid4270 ]
> > > > >>EAL: Detected 36 lcore(s)
> > > > >>EAL: Detected 2 NUMA nodes
> > > > >>EAL: No free hugepages reported in hugepages-1048576kB
> > > > >>EAL: Probing VFIO support...
> > > > >>EAL: PCI device 0000:1c:00.0 on NUMA socket 0
> > > > >>EAL: probe driver: 8086:a54 spdk_nvme
> > > > >>Jobs: 1 (f=0): [_(3),f(1)][100.0%][eta 00m:00s]
> > > > >>0000.1c.00.0-1: (groupid=0, jobs=4): err= 0: pid=4310: Mon Nov
> > > > >>5
> > > > >>05:16:03 2018
> > > > >> mixed: IOPS=12.5k, BW=48.0MiB/s (51.4MB/s)(490MiB/10010msec)
> > > > >> slat (nsec): min=98, max=15092, avg=120.02, stdev=90.07
> > > > >> clat (usec): min=55, max=19757, avg=10207.22, stdev=884.06
> > > > >> lat (usec): min=60, max=19757, avg=10207.34, stdev=884.06
> > > > >> clat percentiles (usec):
> > > > >> | 1.00th=[ 8979], 5.00th=[ 9110], 10.00th=[ 9372],
> > > > >>20.00th=[ 9503],
> > > > >> | 30.00th=[ 9765], 40.00th=[ 9896], 50.00th=[10159],
> > > > >>60.00th=[10290],
> > > > >> | 70.00th=[10552], 80.00th=[10814], 90.00th=[11207],
> > > > >>95.00th=[11600],
> > > > >> | 99.00th=[12911], 99.50th=[14353], 99.90th=[17695],
> > > > >>99.95th=[18744],
> > > > >> | 99.99th=[19006]
> > > > >> bw ( KiB/s): min=12008, max=12976, per=25.00%,
> > > > >>avg=12535.70,
> > > > stdev=306.81, samples=80
> > > > >> iops : min= 3002, max= 3244, avg=3133.85, stdev=76.77,
> > > samples=80
> > > > >> lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%,
> > 1000=0.01%
> > > > >> lat (msec) : 2=0.01%, 4=0.02%, 10=45.13%, 20=54.83%
> > > > >> cpu : usr=100.01%, sys=0.00%, ctx=27, majf=0, minf=0
> > > > >> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
> > > > >=64=0.0%
> > > > >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> 64=0.0%,
> > > > >=64=0.0%
> > > > >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > > > >> 64=0.0%,
> > > > >>>=64=0.0%
> > > > >> issued rwts: total=125506,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > > >> latency : target=0, window=0, percentile=100.00%, depth=32
> > > > >>Run status group 0 (all jobs):
> > > > >> MIXED: bw=48.0MiB/s (51.4MB/s), 48.0MiB/s-48.0MiB/s
> > > > >>(51.4MB/s-51.4MB/s), io=490MiB (514MB), run=10010-10010msec
> > > > >>
> > > > >>
> > > > >>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
> > > > >>[global]
> > > > >>rw=randrw
> > > > >>rwmixread=100
> > > > >>numjobs=1
> > > > >>iodepth=1
> > > > >>bs=4k
> > > > >>direct=1
> > > > >>thread=1
> > > > >>time_based=1
> > > > >>ramp_time=0
> > > > >>runtime=10
> > > > >>ioengine=libaio
> > > > >>group_reporting=1
> > > > >>unified_rw_reporting=1
> > > > >>exitall=1
> > > > >>randrepeat=0
> > > > >>norandommap=1
> > > > >>[nvme2n1]
> > > > >>filename=/dev/nvme2n1
> > > > >>
> > > > >>
> > > > >>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
> > > > >>nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > >>4096B-4096B,
> > > > >>(T) 4096B-4096B, ioengine=libaio, iodepth=1
> > > > >>fio-3.11
> > > > >>Starting 1 thread
> > > > >>Jobs: 1 (f=1): [r(1)][100.0%][r=43.6MiB/s][r=11.2k IOPS][eta
> > > > >>00m:00s]
> > > > >>nvme2n1: (groupid=0, jobs=1): err= 0: pid=2525: Mon Nov 5
> > > > >>05:22:57
> > > > >>2018
> > > > >> mixed: IOPS=11.2k, BW=43.6MiB/s (45.7MB/s)(436MiB/10001msec)
> > > > >> slat (nsec): min=4848, max=208595, avg=5126.33, stdev=771.27
> > > > >> clat (nsec): min=1845, max=3405.8k, avg=83004.99,
> stdev=33577.36
> > > > >> lat (usec): min=15, max=3411, avg=88.26, stdev=33.59
> > > > >> clat percentiles (usec):
> > > > >> | 1.00th=[ 15], 5.00th=[ 58], 10.00th=[ 59], 20.00th=[
> > > 59],
> > > > >> | 30.00th=[ 60], 40.00th=[ 77], 50.00th=[ 77], 60.00th=[
> > > 78],
> > > > >> | 70.00th=[ 111], 80.00th=[ 113], 90.00th=[ 113],
> > > > >>95.00th=[ 115],
> > > > >> | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 190],
> > > > >>99.95th=[ 192],
> > > > >> | 99.99th=[ 375]
> > > > >> bw ( KiB/s): min=43816, max=45016, per=99.98%,
> > > > >>avg=44639.58,
> > > > stdev=312.88, samples=19
> > > > >> iops : min=10954, max=11254, avg=11159.89, stdev=78.22,
> > > > samples=19
> > > > >> lat (usec) : 2=0.01%, 20=2.83%, 50=0.01%, 100=64.57%,
> 250=32.59%
> > > > >> lat (usec) : 500=0.01%, 750=0.01%
> > > > >> lat (msec) : 2=0.01%, 4=0.01%
> > > > >> cpu : usr=4.02%, sys=10.33%, ctx=111630, majf=0, minf=12
> > > > >> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%,
> 32=0.0%,
> > > > >=64=0.0%
> > > > >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> 64=0.0%,
> > > > >=64=0.0%
> > > > >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> > > > >> 64=0.0%,
> > > > >>>=64=0.0%
> > > > >> issued rwts: total=111631,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > > >> latency : target=0, window=0, percentile=100.00%, depth=1
> > > > >>Run status group 0 (all jobs):
> > > > >> MIXED: bw=43.6MiB/s (45.7MB/s), 43.6MiB/s-43.6MiB/s
> (45.7MB/s-
> > > > 45.7MB/s), io=436MiB (457MB), run=10001-10001msec Disk stats
> > > > (read/write):
> > > > >> nvme2n1: ios=110275/0, merge=0/0, ticks=0/0, in_queue=8596,
> > > > >>util=86.22%
> > > > >>
> > > > >>
> > > > >>Thanks,
> > > > >>Joe
> > > > >>
> > > > >>> -----Original Message-----
> > > > >>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of
> > > > >>> Cao, Gang
> > > > >>> Sent: Sunday, November 4, 2018 8:40 PM
> > > > >>> To: Storage Performance Development Kit < spdk(a)lists.01.org
> > > > >>> >
> > > > >>> Subject: Re: [SPDK] FIO NVMe Performance Results
> > > > >>>
> > > > >>> For the SPDK plugin, from the below result, looks like only
> > > > >>> 1 job started with
> > > > >>> 32 queue depth:
> > > > >>>
> > > > >>> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s] <<<<
> > > > >>>
> > > > >>> Try to update the FIO configuration file as
> > > > >>>
> > > > >>> [job1]
> > > > >>> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [job2]
> > > > >>> filename=trtype=PCIe
> > > > >>> traddr=0000.1c.00.0 ns=1 ....
> > > > >>>
> > > > >>> The kernel FIO is with 4 jobs and each job of 32 queue depth.
> > > > >>> For SPDK, you can configure as the same, or just configure
> > > > >>> two jobs and each job of 64 queue depth. The overall queue
> > > > >>> depth to the device is
> > > > same of 128.
> > > > >>>
> > > > >>> Thanks,
> > > > >>> Gang
> > > > >>>
> > > > >>> -----Original Message-----
> > > > >>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of
> > > > >>> Gruher, Joseph R
> > > > >>> Sent: Monday, November 5, 2018 10:49 AM
> > > > >>> To: Storage Performance Development Kit < spdk(a)lists.01.org
> > > > >>> >
> > > > >>> Subject: [SPDK] FIO NVMe Performance Results
> > > > >>>
> > > > >>> Hi folks-
> > > > >>>
> > > > >>> I'm testing SPDK 18.10 on Ubuntu 18.04 with kernel 4.18.16.
> > > > >>> I used FIO with the kernel NVMe driver to measure the
> > > > >>> performance of a local (PCIe
> > > > >>> attached) Intel P4500 NVMe device on a 4KB random read
> > > > >>> workload and obtained 477K IOPS, roughly in line with the
> > > > >>> drive
> spec.
> > > > >>> Then I tested the same drive with the SPDK FIO plugin and
> > > > >>> only achieved 13K IOPS. The FIO test files and the results
> > > > >>> are pasted below. Any ideas where I'm going wrong here?
> > > > >>>
> > > > >>> Thanks!
> > > > >>>
> > > > >>>
> > > > >>> don(a)donst201:~/fio/single/rr$ ls
> > > > >>> /sys/devices/pci0000:17/0000:17:02.0/0000:1c:00.0/nvme/nvme1
> > > > >>> / address cntlid dev device firmware_rev model nvme2n1
> > > > >>> power rescan_controller reset_controller serial state
> > > > >>> subsysnqn subsystem transport uevent
> > > > >>>
> > > > >>>
> > > > >>> don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
> > > > >>> [global]
> > > > >>> rw=randrw
> > > > >>> rwmixread=100
> > > > >>> numjobs=4
> > > > >>> iodepth=32
> > > > >>> bs=4k
> > > > >>> direct=1
> > > > >>> thread=1
> > > > >>> time_based=1
> > > > >>> ramp_time=0
> > > > >>> runtime=10
> > > > >>> ioengine=libaio
> > > > >>> group_reporting=1
> > > > >>> unified_rw_reporting=1
> > > > >>> exitall=1
> > > > >>> randrepeat=0
> > > > >>> norandommap=1
> > > > >>> [nvme2n1]
> > > > >>> filename=/dev/nvme2n1
> > > > >>>
> > > > >>>
> > > > >>> don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
> > > > >>> nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > >>> 4096B-4096B,
> > > > >>> (T) 4096B-4096B, ioengine=libaio, iodepth=32 ...
> > > > >>> fio-3.11
> > > > >>> Starting 4 threads
> > > > >>> Jobs: 4 (f=4): [r(4)][100.0%][r=1874MiB/s][r=480k IOPS][eta
> > > > >>> 00m:00s]
> > > > >>> nvme2n1: (groupid=0, jobs=4): err= 0: pid=2575: Mon Nov 5
> > > > >>> 02:26:25
> > > > 2018
> > > > >>> mixed: IOPS=477k, BW=1862MiB/s
> > (1952MB/s)(18.2GiB/10001msec)
> > > > >>> slat (nsec): min=1295, max=333088, avg=2216.35, stdev=927.89
> > > > >>> clat (nsec): min=506, max=5818.8k, avg=265924.16,
> > stdev=223761.59
> > > > >>> lat (usec): min=6, max=5821, avg=268.19, stdev=223.76
> > > > >>> clat percentiles (usec):
> > > > >>> | 1.00th=[ 13], 5.00th=[ 74], 10.00th=[ 87], 20.00th=[
> > > 115],
> > > > >>> | 30.00th=[ 139], 40.00th=[ 167], 50.00th=[ 204],
> > > > >>> 60.00th=[
> > > 247],
> > > > >>> | 70.00th=[ 306], 80.00th=[ 388], 90.00th=[ 523],
> > > > >>> 95.00th=[
> > > 660],
> > > > >>> | 99.00th=[ 988], 99.50th=[ 1156], 99.90th=[ 2343],
> > > > >>> 99.95th=[
> > > 3195],
> > > > >>> | 99.99th=[ 4424]
> > > > >>> bw ( KiB/s): min=448192, max=483104, per=25.00%,
> > > > >>> avg=476597.10, stdev=6748.78, samples=80
> > > > >>> iops : min=112048, max=120776, avg=119149.27,
> > > stdev=1687.20,
> > > > >>> samples=80
> > > > >>> lat (nsec) : 750=0.01%
> > > > >>> lat (usec) : 10=0.52%, 20=1.25%, 50=0.25%, 100=13.38%,
> > 250=45.13%
> > > > >>> lat (usec) : 500=28.23%, 750=8.08%, 1000=2.22%
> > > > >>> lat (msec) : 2=0.82%, 4=0.10%, 10=0.02%
> > > > >>> cpu : usr=14.08%, sys=36.02%, ctx=1930207, majf=0,
> > > minf=697
> > > > >>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%,
> > 32=100.0%,
> > > > >>> >=64=0.0%
> > > > >>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> > 64=0.0%,
> > > > >>> >=64=0.0%
> > > > >>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > > > >>> 64=0.0%,
> > > > >>> >=64=0.0%
> > > > >>> issued rwts: total=4766095,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > > >>> latency : target=0, window=0, percentile=100.00%, depth=32
> > > > >>> Run status group 0 (all jobs):
> > > > >>> MIXED: bw=1862MiB/s (1952MB/s), 1862MiB/s-1862MiB/s
> > (1952MB/s-
> > > > >>> 1952MB/s), io=18.2GiB (19.5GB), run=10001-10001msec Disk
> > > > >>> stats
> > > > >>> (read/write):
> > > > >>> nvme2n1: ios=4709190/0, merge=0/0, ticks=4096/0,
> > > > in_queue=1388752,
> > > > >>> util=100.00%
> > > > >>>
> > > > >>>
> > > > >>> don(a)donst201:~/fio/single/rr$ sudo
> > > > >>> /home/don/install/spdk/spdk/scripts/setup.sh
> > > > >>> Active mountpoints on /dev/nvme0n1, so not binding PCI dev
> > > > >>> 0000:03:00.0
> > > > >>> 0000:1c:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:1d:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:5e:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:5f:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:62:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:63:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:64:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:65:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:da:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:db:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:dc:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:dd:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:e0:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:e1:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:e2:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:e3:00.0 (8086 0a54): nvme -> uio_pci_generic
> > > > >>> 0000:00:04.0 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:00:04.1 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:00:04.2 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:00:04.3 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:00:04.4 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:00:04.5 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:00:04.6 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:00:04.7 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:80:04.0 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:80:04.1 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:80:04.2 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:80:04.3 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:80:04.4 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:80:04.5 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:80:04.6 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>> 0000:80:04.7 (8086 2021): ioatdma -> uio_pci_generic
> > > > >>>
> > > > >>>
> > > > >>> don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
> > > > >>> [global]
> > > > >>> rw=randrw
> > > > >>> rwmixread=100
> > > > >>> numjobs=4
> > > > >>> iodepth=32
> > > > >>> bs=4k
> > > > >>> direct=1
> > > > >>> thread=1
> > > > >>> time_based=1
> > > > >>> ramp_time=0
> > > > >>> runtime=10
> > > > >>>
> > > >
> > ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_pl
> > > > >>> ug
> > > > >>> in
> > > > >>> group_reporting=1
> > > > >>> unified_rw_reporting=1
> > > > >>> exitall=1
> > > > >>> randrepeat=0
> > > > >>> norandommap=1
> > > > >>> [0000.1c.00.0]
> > > > >>> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
> > > > >>>
> > > > >>>
> > > > >>> don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
> > > > >>> 0000.1c.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W)
> > > > >>> 4096B-4096B,
> > > > >>> (T) 4096B-4096B, ioengine=spdk, iodepth=32 ...
> > > > >>> fio-3.11
> > > > >>> Starting 4 threads
> > > > >>> Starting SPDK v18.10 / DPDK 18.08.0 initialization...
> > > > >>> [ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512 --file-
> > > > >>> prefix=spdk_pid3668 ]
> > > > >>> EAL: Detected 36 lcore(s)
> > > > >>> EAL: Detected 2 NUMA nodes
> > > > >>> EAL: No free hugepages reported in hugepages-1048576kB
> > > > >>> EAL: Probing VFIO support...
> > > > >>> EAL: PCI device 0000:1c:00.0 on NUMA socket 0
> > > > >>> EAL: probe driver: 8086:a54 spdk_nvme
> > > > >>> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s]
> > > > >>> 0000.1c.00.0: (groupid=0, jobs=4): err= 0: pid=3709: Mon Nov
> > > > >>> 5
> > > > >>> 02:28:29
> > > > >>> 2018
> > > > >>> mixed: IOPS=13.2k, BW=51.6MiB/s
> (54.1MB/s)(517MiB/10011msec)
> > > > >>> slat (nsec): min=109, max=15344, avg=127.91, stdev=73.44
> > > > >>> clat (usec): min=187, max=18715, avg=9683.80, stdev=650.54
> > > > >>> lat (usec): min=193, max=18715, avg=9683.93, stdev=650.53
> > > > >>> clat percentiles (usec):
> > > > >>> | 1.00th=[ 8717], 5.00th=[ 8848], 10.00th=[ 8979],
> > > > >>> 20.00th=[
> > > 9110],
> > > > >>> | 30.00th=[ 9241], 40.00th=[ 9372], 50.00th=[ 9503],
> > > > >>> 60.00th=[
> > > 9765],
> > > > >>> | 70.00th=[10028], 80.00th=[10159], 90.00th=[10552],
> > > > 95.00th=[10945],
> > > > >>> | 99.00th=[11338], 99.50th=[11338], 99.90th=[11731],
> > > > 99.95th=[13960],
> > > > >>> | 99.99th=[17695]
> > > > >>> bw ( KiB/s): min=13080, max=13304, per=25.00%,
> > > > >>> avg=13214.70, stdev=55.30, samples=80
> > > > >>> iops : min= 3270, max= 3326, avg=3303.65, stdev=13.84,
> > > > samples=80
> > > > >>> lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
> > > > >>> lat (msec) : 2=0.01%, 4=0.02%, 10=70.85%, 20=29.11%
> > > > >>> cpu : usr=100.01%, sys=0.00%, ctx=23, majf=0, minf=0
> > > > >>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%,
> 32=99.9%,
> > > > >>> >=64=0.0%
> > > > >>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> > 64=0.0%,
> > > > >>> >=64=0.0%
> > > > >>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
> > > > >>> 64=0.0%,
> > > > >>> >=64=0.0%
> > > > >>> issued rwts: total=132300,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > > > >>> latency : target=0, window=0, percentile=100.00%, depth=32
> > > > >>> Run status group 0 (all jobs):
> > > > >>> MIXED: bw=51.6MiB/s (54.1MB/s), 51.6MiB/s-51.6MiB/s
> > (54.1MB/s-
> > > > >>> 54.1MB/s), io=517MiB (542MB), run=10011-10011msec
> > > > >>>
> > > > >>> _______________________________________________
> > > > >>> SPDK mailing list
> > > > >>> SPDK(a)lists.01.org
> > > > >>> https://lists.01.org/mailman/listinfo/spdk
> > > > >>> _______________________________________________
> > > > >>> SPDK mailing list
> > > > >>> SPDK(a)lists.01.org
> > > > >>> https://lists.01.org/mailman/listinfo/spdk
> > > > >>_______________________________________________
> > > > >>SPDK mailing list
> > > > >> SPDK(a)lists.01.org
> > > > >> https://lists.01.org/mailman/listinfo/spdk
> > > > >>_______________________________________________
> > > > >>SPDK mailing list
> > > > >> SPDK(a)lists.01.org
> > > > >> https://lists.01.org/mailman/listinfo/spdk
> > > > >
> > > > >
> > > > >--
> > > > >Маров Алексей
> > > > >_______________________________________________
> > > > >SPDK mailing list
> > > > >SPDK(a)lists.01.org
> > > > >https://lists.01.org/mailman/listinfo/spdk
> > > > >_______________________________________________
> > > > >SPDK mailing list
> > > > >SPDK(a)lists.01.org
> > > > >https://lists.01.org/mailman/listinfo/spdk
> > > >
> > > >
> > > > --
> > > > Маров Алексей
> > > > _______________________________________________
> > > > SPDK mailing list
> > > > SPDK(a)lists.01.org
> > > > https://lists.01.org/mailman/listinfo/spdk
> > > _______________________________________________
> > > SPDK mailing list
> > > SPDK(a)lists.01.org
> > > https://lists.01.org/mailman/listinfo/spdk
> > >
> > --
> >
> > Regards,
> > Andrey
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
next reply other threads:[~2018-11-05 21:56 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-11-05 21:56 Harris, James R [this message]
-- strict thread matches above, loose matches on Subject: below --
2018-11-06 10:31 [SPDK] FIO NVMe Performance Results Andrey Kuzmin
2018-11-06 1:40 Cao, Gang
2018-11-05 22:19 Gruher, Joseph R
2018-11-05 21:42 Howell, Seth
2018-11-05 20:45 Gruher, Joseph R
2018-11-05 20:39 Gruher, Joseph R
2018-11-05 20:18 Andrey Kuzmin
2018-11-05 19:59 Andrey Kuzmin
2018-11-05 19:38 Gruher, Joseph R
2018-11-05 19:04 Gruher, Joseph R
2018-11-05 18:40 Andrey Kuzmin
2018-11-05 18:33 Andrey Kuzmin
2018-11-05 18:32 Howell, Seth
2018-11-05 18:27 Gruher, Joseph R
2018-11-05 16:48 Andrey Kuzmin
2018-11-05 15:29 Gruher, Joseph R
2018-11-05 8:45
2018-11-05 8:39 Cao, Gang
2018-11-05 8:32
2018-11-05 8:27
2018-11-05 6:53 Cao, Gang
2018-11-05 5:32 Gruher, Joseph R
2018-11-05 4:40 Cao, Gang
2018-11-05 2:49 Gruher, Joseph R
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=E5866F0D-BCD3-4D82-8FC3-CCF883E62517@intel.com \
--to=spdk@lists.01.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox