Re: [SPDK] FIO NVMe Performance Results

[-- Attachment #1: Type: text/plain, Size: 22647 bytes --]

Ok, Do you  see some performance issues using general bdev fio plugin with you configuration ?

Best Regards
Alex


>Понедельник,  5 ноября 2018, 11:39 +03:00 от "Cao, Gang" <gang.cao(a)intel.com>:
>
>Yes. This is for the bdev version of SPDK FIO plugin and you can assign the conf file as the parameter in the FIO configuration.
>
>Previously, it's another SPDK FIO plugin for the user space NVMe driver only.
>
>These are two different FIO plugins.
>
>Thanks,
>Gang
>
>-----Original Message-----
>From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Маров Алексей
>Sent: Monday, November 5, 2018 4:32 PM
>To: Storage Performance Development Kit < spdk(a)lists.01.org >
>Subject: Re: [SPDK] FIO NVMe Performance Results
>
>Sorry for incomplete message I have sent before
>
>Hello Gang.
>
>I just curious, why would you run fio with SPDK using this conf file, with is a bit different from what is  suggested in here   https://github.com/spdk/spdk/tree/master/examples/bdev/fio_plugin  .
>
>I mean may be you should create conf file for spdk like this one :
>
>nvme.conf :
>[Nvme]
>  TransportId "trtype:PCIe traddr:0000.1c.00.0" Nvme0
>
>and provide it in fio config.
>
>fio.ini :
>[global]
>ioengine=spdk_bdev
>spdk_conf=....../nvme.conf
>rw=randrw
>rwmixread=100
>numjobs=4
>iodepth=32
>bs=4k
>direct=1
>thread=1
>time_based=1
>ramp_time=0
>runtime=10
>group_reporting=1
>unified_rw_reporting=1
>exitall=1
>randrepeat=0
>norandommap=1
>
>[0000:06:00.0]
>filename=Nvme0n1
>
>and run like this
>
># LD_PRELOAD=examples/bdev/fio_plugin/fio_plugin fio examples/bdev/fio_plugin/fio.ini
>
>Just to note. The recommended version of fio for SPDK is 3.3
>
>Best Regards 
>Alex
>
>
>>Понедельник,  5 ноября 2018, 9:53 +03:00 от "Cao, Gang" < gang.cao(a)intel.com >:
>>
>>Thanks. This is interesting especially for your finding on queue depth 1 case. We are trying to reproduce this issue locally same as your configuration.
>>
>>Thanks,
>>Gang
>>
>>-----Original Message-----
>>From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Gruher, Joseph R
>>Sent: Monday, November 5, 2018 1:32 PM
>>To: Storage Performance Development Kit <  spdk(a)lists.01.org >
>>Subject: Re: [SPDK] FIO NVMe Performance Results
>>
>>Interesting catch.  It seems like FIO is actually not accurately reporting the number of jobs in the report out for the SPDK case.  If I disable the group_reporting parameter I do get a printout for four separate jobs in the SPDK case, so I think it is really running four jobs there, and just summarizing incorrectly.  I also see four separate CPU cores go to 100% when I run that FIO test case, indicating four jobs are active.
>>
>>Also, I don't think would explain the difference in performance.  Using an aggregate queue depth of 32 versus 128 shouldn't lead to a drop in performance from 377K to 13K. 
>>
>>I retested SPDK with multiple separate job descriptions as you suggested anyway, using four separate jobs instead of relying on the numjobs=4 parameter, and the results are below.  In that SPDK test with four separate FIO job statements, each with queue depth 32, I still only achieved 12.5K IOPS. 
>>
>>I then re-ran the kernel NVMe driver test case with only a single job for comparison, and even with only 1 job and queue depth 32 it still achieves 254K IOPS.
>>
>>We might be on the right track with a problem around queue depth however.  If I run the kernel NVMe driver case with just 1 job and queue depth of only 1, I then get 11.2K IOPS, which is around the performance I get in the SPDK test case.  Perhaps the SPDK case is not actually queuing IOs on the device for some reason?
>>
>>
>>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
>>[global]
>>rw=randrw
>>rwmixread=100
>>numjobs=1
>>iodepth=32
>>bs=4k
>>direct=1
>>thread=1
>>time_based=1
>>ramp_time=0
>>runtime=10
>>ioengine=libaio
>>group_reporting=1
>>unified_rw_reporting=1
>>exitall=1
>>randrepeat=0
>>norandommap=1
>>[nvme2n1]
>>filename=/dev/nvme2n1
>>
>>
>>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
>>nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
>>fio-3.11
>>Starting 1 thread
>>Jobs: 1 (f=1): [r(1)][100.0%][r=1025MiB/s][r=262k IOPS][eta 00m:00s]
>>nvme2n1: (groupid=0, jobs=1): err= 0: pid=2422: Mon Nov  5 05:19:13 2018
>>  mixed: IOPS=254k, BW=992MiB/s (1040MB/s)(9922MiB/10001msec)
>>    slat (nsec): min=1185, max=319052, avg=1611.64, stdev=808.40
>>    clat (usec): min=2, max=4743, avg=124.02, stdev=121.00
>>     lat (usec): min=6, max=4744, avg=125.68, stdev=121.02
>>    clat percentiles (usec):
>>     |  1.00th=[    9],  5.00th=[   56], 10.00th=[   61], 20.00th=[   73],
>>     | 30.00th=[   80], 40.00th=[   91], 50.00th=[  110], 60.00th=[  118],
>>     | 70.00th=[  130], 80.00th=[  157], 90.00th=[  202], 95.00th=[  253],
>>     | 99.00th=[  408], 99.50th=[  498], 99.90th=[ 1795], 99.95th=[ 3064],
>>     | 99.99th=[ 3785]
>>   bw (  KiB/s): min=612240, max=1078792, per=100.00%, avg=1015996.40, stdev=106032.96, samples=20
>>   iops        : min=153060, max=269698, avg=253998.90, stdev=26508.40, samples=20
>>  lat (usec)   : 4=0.01%, 10=1.47%, 20=1.19%, 50=0.17%, 100=42.03%
>>  lat (usec)   : 250=49.95%, 500=4.70%, 750=0.28%, 1000=0.06%
>>  lat (msec)   : 2=0.06%, 4=0.09%, 10=0.01%
>>  cpu          : usr=21.45%, sys=51.11%, ctx=530348, majf=0, minf=139
>>  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
>>     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>>     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
>>     issued rwts: total=2540023,0,0,0 short=0,0,0,0 dropped=0,0,0,0
>>     latency   : target=0, window=0, percentile=100.00%, depth=32
>>Run status group 0 (all jobs):
>>  MIXED: bw=992MiB/s (1040MB/s), 992MiB/s-992MiB/s (1040MB/s-1040MB/s), io=9922MiB (10.4GB), run=10001-10001msec Disk stats (read/write):
>>  nvme2n1: ios=2514356/0, merge=0/0, ticks=400/0, in_queue=297612, util=99.06%
>>
>>
>>don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
>>[global]
>>rw=randrw
>>rwmixread=100
>>numjobs=1
>>iodepth=32
>>bs=4k
>>direct=1
>>thread=1
>>time_based=1
>>ramp_time=0
>>runtime=10
>>ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_plugin
>>group_reporting=1
>>unified_rw_reporting=1
>>exitall=1
>>randrepeat=0
>>norandommap=1
>>[0000.1c.00.0-1]
>>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-2] filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-3] filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-4] filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
>>
>>
>>don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
>>0000.1c.00.0-1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=32
>>0000.1c.00.0-2: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=32
>>0000.1c.00.0-3: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=32
>>0000.1c.00.0-4: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=32
>>fio-3.11
>>Starting 4 threads
>>Starting SPDK v18.10 / DPDK 18.08.0 initialization...
>>[ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512 --file-prefix=spdk_pid4270 ]
>>EAL: Detected 36 lcore(s)
>>EAL: Detected 2 NUMA nodes
>>EAL: No free hugepages reported in hugepages-1048576kB
>>EAL: Probing VFIO support...
>>EAL: PCI device 0000:1c:00.0 on NUMA socket 0
>>EAL:   probe driver: 8086:a54 spdk_nvme
>>Jobs: 1 (f=0): [_(3),f(1)][100.0%][eta 00m:00s]
>>0000.1c.00.0-1: (groupid=0, jobs=4): err= 0: pid=4310: Mon Nov  5 05:16:03 2018
>>  mixed: IOPS=12.5k, BW=48.0MiB/s (51.4MB/s)(490MiB/10010msec)
>>    slat (nsec): min=98, max=15092, avg=120.02, stdev=90.07
>>    clat (usec): min=55, max=19757, avg=10207.22, stdev=884.06
>>     lat (usec): min=60, max=19757, avg=10207.34, stdev=884.06
>>    clat percentiles (usec):
>>     |  1.00th=[ 8979],  5.00th=[ 9110], 10.00th=[ 9372], 20.00th=[ 9503],
>>     | 30.00th=[ 9765], 40.00th=[ 9896], 50.00th=[10159], 60.00th=[10290],
>>     | 70.00th=[10552], 80.00th=[10814], 90.00th=[11207], 95.00th=[11600],
>>     | 99.00th=[12911], 99.50th=[14353], 99.90th=[17695], 99.95th=[18744],
>>     | 99.99th=[19006]
>>   bw (  KiB/s): min=12008, max=12976, per=25.00%, avg=12535.70, stdev=306.81, samples=80
>>   iops        : min= 3002, max= 3244, avg=3133.85, stdev=76.77, samples=80
>>  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
>>  lat (msec)   : 2=0.01%, 4=0.02%, 10=45.13%, 20=54.83%
>>  cpu          : usr=100.01%, sys=0.00%, ctx=27, majf=0, minf=0
>>  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
>>     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>>     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
>>     issued rwts: total=125506,0,0,0 short=0,0,0,0 dropped=0,0,0,0
>>     latency   : target=0, window=0, percentile=100.00%, depth=32
>>Run status group 0 (all jobs):
>>  MIXED: bw=48.0MiB/s (51.4MB/s), 48.0MiB/s-48.0MiB/s (51.4MB/s-51.4MB/s), io=490MiB (514MB), run=10010-10010msec
>>
>>
>>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
>>[global]
>>rw=randrw
>>rwmixread=100
>>numjobs=1
>>iodepth=1
>>bs=4k
>>direct=1
>>thread=1
>>time_based=1
>>ramp_time=0
>>runtime=10
>>ioengine=libaio
>>group_reporting=1
>>unified_rw_reporting=1
>>exitall=1
>>randrepeat=0
>>norandommap=1
>>[nvme2n1]
>>filename=/dev/nvme2n1
>>
>>
>>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
>>nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
>>fio-3.11
>>Starting 1 thread
>>Jobs: 1 (f=1): [r(1)][100.0%][r=43.6MiB/s][r=11.2k IOPS][eta 00m:00s]
>>nvme2n1: (groupid=0, jobs=1): err= 0: pid=2525: Mon Nov  5 05:22:57 2018
>>  mixed: IOPS=11.2k, BW=43.6MiB/s (45.7MB/s)(436MiB/10001msec)
>>    slat (nsec): min=4848, max=208595, avg=5126.33, stdev=771.27
>>    clat (nsec): min=1845, max=3405.8k, avg=83004.99, stdev=33577.36
>>     lat (usec): min=15, max=3411, avg=88.26, stdev=33.59
>>    clat percentiles (usec):
>>     |  1.00th=[   15],  5.00th=[   58], 10.00th=[   59], 20.00th=[   59],
>>     | 30.00th=[   60], 40.00th=[   77], 50.00th=[   77], 60.00th=[   78],
>>     | 70.00th=[  111], 80.00th=[  113], 90.00th=[  113], 95.00th=[  115],
>>     | 99.00th=[  186], 99.50th=[  186], 99.90th=[  190], 99.95th=[  192],
>>     | 99.99th=[  375]
>>   bw (  KiB/s): min=43816, max=45016, per=99.98%, avg=44639.58, stdev=312.88, samples=19
>>   iops        : min=10954, max=11254, avg=11159.89, stdev=78.22, samples=19
>>  lat (usec)   : 2=0.01%, 20=2.83%, 50=0.01%, 100=64.57%, 250=32.59%
>>  lat (usec)   : 500=0.01%, 750=0.01%
>>  lat (msec)   : 2=0.01%, 4=0.01%
>>  cpu          : usr=4.02%, sys=10.33%, ctx=111630, majf=0, minf=12
>>  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
>>     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>>     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>>     issued rwts: total=111631,0,0,0 short=0,0,0,0 dropped=0,0,0,0
>>     latency   : target=0, window=0, percentile=100.00%, depth=1
>>Run status group 0 (all jobs):
>>  MIXED: bw=43.6MiB/s (45.7MB/s), 43.6MiB/s-43.6MiB/s (45.7MB/s-45.7MB/s), io=436MiB (457MB), run=10001-10001msec Disk stats (read/write):
>>  nvme2n1: ios=110275/0, merge=0/0, ticks=0/0, in_queue=8596, util=86.22%
>>
>>
>>Thanks,
>>Joe
>>
>>> -----Original Message-----
>>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
>>> Sent: Sunday, November 4, 2018 8:40 PM
>>> To: Storage Performance Development Kit <  spdk(a)lists.01.org >
>>> Subject: Re: [SPDK] FIO NVMe Performance Results
>>> 
>>> For the SPDK plugin, from the below result, looks like only 1 job 
>>> started with
>>> 32 queue depth:
>>> 
>>> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s] <<<<
>>> 
>>> Try to update the FIO configuration file as
>>> 
>>> [job1]
>>> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [job2] 
>>> filename=trtype=PCIe
>>> traddr=0000.1c.00.0 ns=1 ....
>>> 
>>> The kernel FIO is with 4 jobs and each job of 32 queue depth. For 
>>> SPDK, you can configure as the same, or just configure two jobs and 
>>> each job of 64 queue depth. The overall queue depth to the device is same of 128.
>>> 
>>> Thanks,
>>> Gang
>>> 
>>> -----Original Message-----
>>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Gruher, 
>>> Joseph R
>>> Sent: Monday, November 5, 2018 10:49 AM
>>> To: Storage Performance Development Kit <  spdk(a)lists.01.org >
>>> Subject: [SPDK] FIO NVMe Performance Results
>>> 
>>> Hi folks-
>>> 
>>> I'm testing SPDK 18.10 on Ubuntu 18.04 with kernel 4.18.16.  I used 
>>> FIO with the kernel NVMe driver to measure the performance of a local 
>>> (PCIe
>>> attached) Intel P4500 NVMe device on a 4KB random read workload and 
>>> obtained 477K IOPS, roughly in line with the drive spec.  Then I 
>>> tested the same drive with the SPDK FIO plugin and only achieved 13K 
>>> IOPS.  The FIO test files and the results are pasted below.  Any ideas 
>>> where I'm going wrong here?
>>> 
>>> Thanks!
>>> 
>>> 
>>> don(a)donst201:~/fio/single/rr$ ls
>>> /sys/devices/pci0000:17/0000:17:02.0/0000:1c:00.0/nvme/nvme1/
>>> address  cntlid  dev  device  firmware_rev  model  nvme2n1  power 
>>> rescan_controller  reset_controller  serial  state  subsysnqn 
>>> subsystem transport  uevent
>>> 
>>> 
>>> don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
>>> [global]
>>> rw=randrw
>>> rwmixread=100
>>> numjobs=4
>>> iodepth=32
>>> bs=4k
>>> direct=1
>>> thread=1
>>> time_based=1
>>> ramp_time=0
>>> runtime=10
>>> ioengine=libaio
>>> group_reporting=1
>>> unified_rw_reporting=1
>>> exitall=1
>>> randrepeat=0
>>> norandommap=1
>>> [nvme2n1]
>>> filename=/dev/nvme2n1
>>> 
>>> 
>>> don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
>>> nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 
>>> 4096B-4096B, ioengine=libaio, iodepth=32 ...
>>> fio-3.11
>>> Starting 4 threads
>>> Jobs: 4 (f=4): [r(4)][100.0%][r=1874MiB/s][r=480k IOPS][eta 00m:00s]
>>> nvme2n1: (groupid=0, jobs=4): err= 0: pid=2575: Mon Nov  5 02:26:25 2018
>>>   mixed: IOPS=477k, BW=1862MiB/s (1952MB/s)(18.2GiB/10001msec)
>>>     slat (nsec): min=1295, max=333088, avg=2216.35, stdev=927.89
>>>    clat (nsec): min=506, max=5818.8k, avg=265924.16, stdev=223761.59
>>>      lat (usec): min=6, max=5821, avg=268.19, stdev=223.76
>>>     clat percentiles (usec):
>>>      |  1.00th=[   13],  5.00th=[   74], 10.00th=[   87], 20.00th=[  115],
>>>      | 30.00th=[  139], 40.00th=[  167], 50.00th=[  204], 60.00th=[  247],
>>>      | 70.00th=[  306], 80.00th=[  388], 90.00th=[  523], 95.00th=[  660],
>>>      | 99.00th=[  988], 99.50th=[ 1156], 99.90th=[ 2343], 99.95th=[ 3195],
>>>      | 99.99th=[ 4424]
>>>    bw (  KiB/s): min=448192, max=483104, per=25.00%, avg=476597.10, 
>>> stdev=6748.78, samples=80
>>>    iops        : min=112048, max=120776, avg=119149.27, stdev=1687.20,
>>> samples=80
>>>   lat (nsec)   : 750=0.01%
>>>   lat (usec)   : 10=0.52%, 20=1.25%, 50=0.25%, 100=13.38%, 250=45.13%
>>>   lat (usec)   : 500=28.23%, 750=8.08%, 1000=2.22%
>>>   lat (msec)   : 2=0.82%, 4=0.10%, 10=0.02%
>>>   cpu          : usr=14.08%, sys=36.02%, ctx=1930207, majf=0, minf=697
>>>   IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>> >=64=0.0%
>>>      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>> >=64=0.0%
>>>      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>> >=64=0.0%
>>>      issued rwts: total=4766095,0,0,0 short=0,0,0,0 dropped=0,0,0,0
>>>      latency   : target=0, window=0, percentile=100.00%, depth=32
>>> Run status group 0 (all jobs):
>>>   MIXED: bw=1862MiB/s (1952MB/s), 1862MiB/s-1862MiB/s (1952MB/s- 
>>> 1952MB/s), io=18.2GiB (19.5GB), run=10001-10001msec Disk stats
>>> (read/write):
>>>   nvme2n1: ios=4709190/0, merge=0/0, ticks=4096/0, in_queue=1388752, 
>>> util=100.00%
>>> 
>>> 
>>> don(a)donst201:~/fio/single/rr$ sudo
>>> /home/don/install/spdk/spdk/scripts/setup.sh
>>> Active mountpoints on /dev/nvme0n1, so not binding PCI dev 
>>> 0000:03:00.0
>>> 0000:1c:00.0 (8086 0a54): nvme -> uio_pci_generic
>>> 0000:1d:00.0 (8086 0a54): nvme -> uio_pci_generic
>>> 0000:5e:00.0 (8086 0a54): nvme -> uio_pci_generic
>>> 0000:5f:00.0 (8086 0a54): nvme -> uio_pci_generic
>>> 0000:62:00.0 (8086 0a54): nvme -> uio_pci_generic
>>> 0000:63:00.0 (8086 0a54): nvme -> uio_pci_generic
>>> 0000:64:00.0 (8086 0a54): nvme -> uio_pci_generic
>>> 0000:65:00.0 (8086 0a54): nvme -> uio_pci_generic
>>> 0000:da:00.0 (8086 0a54): nvme -> uio_pci_generic
>>> 0000:db:00.0 (8086 0a54): nvme -> uio_pci_generic
>>> 0000:dc:00.0 (8086 0a54): nvme -> uio_pci_generic
>>> 0000:dd:00.0 (8086 0a54): nvme -> uio_pci_generic
>>> 0000:e0:00.0 (8086 0a54): nvme -> uio_pci_generic
>>> 0000:e1:00.0 (8086 0a54): nvme -> uio_pci_generic
>>> 0000:e2:00.0 (8086 0a54): nvme -> uio_pci_generic
>>> 0000:e3:00.0 (8086 0a54): nvme -> uio_pci_generic
>>> 0000:00:04.0 (8086 2021): ioatdma -> uio_pci_generic
>>> 0000:00:04.1 (8086 2021): ioatdma -> uio_pci_generic
>>> 0000:00:04.2 (8086 2021): ioatdma -> uio_pci_generic
>>> 0000:00:04.3 (8086 2021): ioatdma -> uio_pci_generic
>>> 0000:00:04.4 (8086 2021): ioatdma -> uio_pci_generic
>>> 0000:00:04.5 (8086 2021): ioatdma -> uio_pci_generic
>>> 0000:00:04.6 (8086 2021): ioatdma -> uio_pci_generic
>>> 0000:00:04.7 (8086 2021): ioatdma -> uio_pci_generic
>>> 0000:80:04.0 (8086 2021): ioatdma -> uio_pci_generic
>>> 0000:80:04.1 (8086 2021): ioatdma -> uio_pci_generic
>>> 0000:80:04.2 (8086 2021): ioatdma -> uio_pci_generic
>>> 0000:80:04.3 (8086 2021): ioatdma -> uio_pci_generic
>>> 0000:80:04.4 (8086 2021): ioatdma -> uio_pci_generic
>>> 0000:80:04.5 (8086 2021): ioatdma -> uio_pci_generic
>>> 0000:80:04.6 (8086 2021): ioatdma -> uio_pci_generic
>>> 0000:80:04.7 (8086 2021): ioatdma -> uio_pci_generic
>>> 
>>> 
>>> don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
>>> [global]
>>> rw=randrw
>>> rwmixread=100
>>> numjobs=4
>>> iodepth=32
>>> bs=4k
>>> direct=1
>>> thread=1
>>> time_based=1
>>> ramp_time=0
>>> runtime=10
>>> ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_plug
>>> in
>>> group_reporting=1
>>> unified_rw_reporting=1
>>> exitall=1
>>> randrepeat=0
>>> norandommap=1
>>> [0000.1c.00.0]
>>> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
>>> 
>>> 
>>> don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
>>> 0000.1c.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, 
>>> (T) 4096B-4096B, ioengine=spdk, iodepth=32 ...
>>> fio-3.11
>>> Starting 4 threads
>>> Starting SPDK v18.10 / DPDK 18.08.0 initialization...
>>> [ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512 --file-
>>> prefix=spdk_pid3668 ]
>>> EAL: Detected 36 lcore(s)
>>> EAL: Detected 2 NUMA nodes
>>> EAL: No free hugepages reported in hugepages-1048576kB
>>> EAL: Probing VFIO support...
>>> EAL: PCI device 0000:1c:00.0 on NUMA socket 0
>>> EAL:   probe driver: 8086:a54 spdk_nvme
>>> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s]
>>> 0000.1c.00.0: (groupid=0, jobs=4): err= 0: pid=3709: Mon Nov  5 
>>> 02:28:29
>>> 2018
>>>   mixed: IOPS=13.2k, BW=51.6MiB/s (54.1MB/s)(517MiB/10011msec)
>>>     slat (nsec): min=109, max=15344, avg=127.91, stdev=73.44
>>>     clat (usec): min=187, max=18715, avg=9683.80, stdev=650.54
>>>      lat (usec): min=193, max=18715, avg=9683.93, stdev=650.53
>>>     clat percentiles (usec):
>>>      |  1.00th=[ 8717],  5.00th=[ 8848], 10.00th=[ 8979], 20.00th=[ 9110],
>>>      | 30.00th=[ 9241], 40.00th=[ 9372], 50.00th=[ 9503], 60.00th=[ 9765],
>>>      | 70.00th=[10028], 80.00th=[10159], 90.00th=[10552], 95.00th=[10945],
>>>      | 99.00th=[11338], 99.50th=[11338], 99.90th=[11731], 99.95th=[13960],
>>>      | 99.99th=[17695]
>>>    bw (  KiB/s): min=13080, max=13304, per=25.00%, avg=13214.70, 
>>> stdev=55.30, samples=80
>>>    iops        : min= 3270, max= 3326, avg=3303.65, stdev=13.84, samples=80
>>>   lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
>>>   lat (msec)   : 2=0.01%, 4=0.02%, 10=70.85%, 20=29.11%
>>>   cpu          : usr=100.01%, sys=0.00%, ctx=23, majf=0, minf=0
>>>   IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
>>> >=64=0.0%
>>>      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>> >=64=0.0%
>>>      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>> >=64=0.0%
>>>      issued rwts: total=132300,0,0,0 short=0,0,0,0 dropped=0,0,0,0
>>>      latency   : target=0, window=0, percentile=100.00%, depth=32
>>> Run status group 0 (all jobs):
>>>   MIXED: bw=51.6MiB/s (54.1MB/s), 51.6MiB/s-51.6MiB/s (54.1MB/s- 
>>> 54.1MB/s), io=517MiB (542MB), run=10011-10011msec
>>> 
>>> _______________________________________________
>>> SPDK mailing list
>>>  SPDK(a)lists.01.org
>>>  https://lists.01.org/mailman/listinfo/spdk
>>> _______________________________________________
>>> SPDK mailing list
>>>  SPDK(a)lists.01.org
>>>  https://lists.01.org/mailman/listinfo/spdk
>>_______________________________________________
>>SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>>_______________________________________________
>>SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>
>
>-- 
>Маров Алексей
>_______________________________________________
>SPDK mailing list
>SPDK(a)lists.01.org
>https://lists.01.org/mailman/listinfo/spdk
>_______________________________________________
>SPDK mailing list
>SPDK(a)lists.01.org
>https://lists.01.org/mailman/listinfo/spdk


-- 
Маров Алексей