Re: [SPDK] FIO NVMe Performance Results

[-- Attachment #1: Type: text/plain, Size: 21051 bytes --]

Sorry for incomplete message I have sent before

Hello Gang.

I just curious, why would you run fio with SPDK using this conf file, with is a bit different from what is  suggested in here  https://github.com/spdk/spdk/tree/master/examples/bdev/fio_plugin  .

I mean may be you should create conf file for spdk like this one :

nvme.conf :
[Nvme]
  TransportId "trtype:PCIe traddr:0000.1c.00.0" Nvme0

and provide it in fio config.

fio.ini :
[global]
ioengine=spdk_bdev
spdk_conf=....../nvme.conf
rw=randrw
rwmixread=100
numjobs=4
iodepth=32
bs=4k
direct=1
thread=1
time_based=1
ramp_time=0
runtime=10
group_reporting=1
unified_rw_reporting=1
exitall=1
randrepeat=0
norandommap=1

[0000:06:00.0]
filename=Nvme0n1

and run like this

# LD_PRELOAD=examples/bdev/fio_plugin/fio_plugin fio examples/bdev/fio_plugin/fio.ini

Just to note. The recommended version of fio for SPDK is 3.3

Best Regards 
Alex


>Понедельник,  5 ноября 2018, 9:53 +03:00 от "Cao, Gang" <gang.cao(a)intel.com>:
>
>Thanks. This is interesting especially for your finding on queue depth 1 case. We are trying to reproduce this issue locally same as your configuration.
>
>Thanks,
>Gang
>
>-----Original Message-----
>From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Gruher, Joseph R
>Sent: Monday, November 5, 2018 1:32 PM
>To: Storage Performance Development Kit < spdk(a)lists.01.org >
>Subject: Re: [SPDK] FIO NVMe Performance Results
>
>Interesting catch.  It seems like FIO is actually not accurately reporting the number of jobs in the report out for the SPDK case.  If I disable the group_reporting parameter I do get a printout for four separate jobs in the SPDK case, so I think it is really running four jobs there, and just summarizing incorrectly.  I also see four separate CPU cores go to 100% when I run that FIO test case, indicating four jobs are active.
>
>Also, I don't think would explain the difference in performance.  Using an aggregate queue depth of 32 versus 128 shouldn't lead to a drop in performance from 377K to 13K. 
>
>I retested SPDK with multiple separate job descriptions as you suggested anyway, using four separate jobs instead of relying on the numjobs=4 parameter, and the results are below.  In that SPDK test with four separate FIO job statements, each with queue depth 32, I still only achieved 12.5K IOPS. 
>
>I then re-ran the kernel NVMe driver test case with only a single job for comparison, and even with only 1 job and queue depth 32 it still achieves 254K IOPS.
>
>We might be on the right track with a problem around queue depth however.  If I run the kernel NVMe driver case with just 1 job and queue depth of only 1, I then get 11.2K IOPS, which is around the performance I get in the SPDK test case.  Perhaps the SPDK case is not actually queuing IOs on the device for some reason?
>
>
>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
>[global]
>rw=randrw
>rwmixread=100
>numjobs=1
>iodepth=32
>bs=4k
>direct=1
>thread=1
>time_based=1
>ramp_time=0
>runtime=10
>ioengine=libaio
>group_reporting=1
>unified_rw_reporting=1
>exitall=1
>randrepeat=0
>norandommap=1
>[nvme2n1]
>filename=/dev/nvme2n1
>
>
>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
>nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
>fio-3.11
>Starting 1 thread
>Jobs: 1 (f=1): [r(1)][100.0%][r=1025MiB/s][r=262k IOPS][eta 00m:00s]
>nvme2n1: (groupid=0, jobs=1): err= 0: pid=2422: Mon Nov  5 05:19:13 2018
>  mixed: IOPS=254k, BW=992MiB/s (1040MB/s)(9922MiB/10001msec)
>    slat (nsec): min=1185, max=319052, avg=1611.64, stdev=808.40
>    clat (usec): min=2, max=4743, avg=124.02, stdev=121.00
>     lat (usec): min=6, max=4744, avg=125.68, stdev=121.02
>    clat percentiles (usec):
>     |  1.00th=[    9],  5.00th=[   56], 10.00th=[   61], 20.00th=[   73],
>     | 30.00th=[   80], 40.00th=[   91], 50.00th=[  110], 60.00th=[  118],
>     | 70.00th=[  130], 80.00th=[  157], 90.00th=[  202], 95.00th=[  253],
>     | 99.00th=[  408], 99.50th=[  498], 99.90th=[ 1795], 99.95th=[ 3064],
>     | 99.99th=[ 3785]
>   bw (  KiB/s): min=612240, max=1078792, per=100.00%, avg=1015996.40, stdev=106032.96, samples=20
>   iops        : min=153060, max=269698, avg=253998.90, stdev=26508.40, samples=20
>  lat (usec)   : 4=0.01%, 10=1.47%, 20=1.19%, 50=0.17%, 100=42.03%
>  lat (usec)   : 250=49.95%, 500=4.70%, 750=0.28%, 1000=0.06%
>  lat (msec)   : 2=0.06%, 4=0.09%, 10=0.01%
>  cpu          : usr=21.45%, sys=51.11%, ctx=530348, majf=0, minf=139
>  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
>     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
>     issued rwts: total=2540023,0,0,0 short=0,0,0,0 dropped=0,0,0,0
>     latency   : target=0, window=0, percentile=100.00%, depth=32
>Run status group 0 (all jobs):
>  MIXED: bw=992MiB/s (1040MB/s), 992MiB/s-992MiB/s (1040MB/s-1040MB/s), io=9922MiB (10.4GB), run=10001-10001msec Disk stats (read/write):
>  nvme2n1: ios=2514356/0, merge=0/0, ticks=400/0, in_queue=297612, util=99.06%
>
>
>don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
>[global]
>rw=randrw
>rwmixread=100
>numjobs=1
>iodepth=32
>bs=4k
>direct=1
>thread=1
>time_based=1
>ramp_time=0
>runtime=10
>ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_plugin
>group_reporting=1
>unified_rw_reporting=1
>exitall=1
>randrepeat=0
>norandommap=1
>[0000.1c.00.0-1]
>filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-2] filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-3] filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [0000.1c.00.0-4] filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
>
>
>don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
>0000.1c.00.0-1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=32
>0000.1c.00.0-2: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=32
>0000.1c.00.0-3: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=32
>0000.1c.00.0-4: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=32
>fio-3.11
>Starting 4 threads
>Starting SPDK v18.10 / DPDK 18.08.0 initialization...
>[ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512 --file-prefix=spdk_pid4270 ]
>EAL: Detected 36 lcore(s)
>EAL: Detected 2 NUMA nodes
>EAL: No free hugepages reported in hugepages-1048576kB
>EAL: Probing VFIO support...
>EAL: PCI device 0000:1c:00.0 on NUMA socket 0
>EAL:   probe driver: 8086:a54 spdk_nvme
>Jobs: 1 (f=0): [_(3),f(1)][100.0%][eta 00m:00s]
>0000.1c.00.0-1: (groupid=0, jobs=4): err= 0: pid=4310: Mon Nov  5 05:16:03 2018
>  mixed: IOPS=12.5k, BW=48.0MiB/s (51.4MB/s)(490MiB/10010msec)
>    slat (nsec): min=98, max=15092, avg=120.02, stdev=90.07
>    clat (usec): min=55, max=19757, avg=10207.22, stdev=884.06
>     lat (usec): min=60, max=19757, avg=10207.34, stdev=884.06
>    clat percentiles (usec):
>     |  1.00th=[ 8979],  5.00th=[ 9110], 10.00th=[ 9372], 20.00th=[ 9503],
>     | 30.00th=[ 9765], 40.00th=[ 9896], 50.00th=[10159], 60.00th=[10290],
>     | 70.00th=[10552], 80.00th=[10814], 90.00th=[11207], 95.00th=[11600],
>     | 99.00th=[12911], 99.50th=[14353], 99.90th=[17695], 99.95th=[18744],
>     | 99.99th=[19006]
>   bw (  KiB/s): min=12008, max=12976, per=25.00%, avg=12535.70, stdev=306.81, samples=80
>   iops        : min= 3002, max= 3244, avg=3133.85, stdev=76.77, samples=80
>  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
>  lat (msec)   : 2=0.01%, 4=0.02%, 10=45.13%, 20=54.83%
>  cpu          : usr=100.01%, sys=0.00%, ctx=27, majf=0, minf=0
>  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
>     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
>     issued rwts: total=125506,0,0,0 short=0,0,0,0 dropped=0,0,0,0
>     latency   : target=0, window=0, percentile=100.00%, depth=32
>Run status group 0 (all jobs):
>  MIXED: bw=48.0MiB/s (51.4MB/s), 48.0MiB/s-48.0MiB/s (51.4MB/s-51.4MB/s), io=490MiB (514MB), run=10010-10010msec
>
>
>don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
>[global]
>rw=randrw
>rwmixread=100
>numjobs=1
>iodepth=1
>bs=4k
>direct=1
>thread=1
>time_based=1
>ramp_time=0
>runtime=10
>ioengine=libaio
>group_reporting=1
>unified_rw_reporting=1
>exitall=1
>randrepeat=0
>norandommap=1
>[nvme2n1]
>filename=/dev/nvme2n1
>
>
>don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
>nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
>fio-3.11
>Starting 1 thread
>Jobs: 1 (f=1): [r(1)][100.0%][r=43.6MiB/s][r=11.2k IOPS][eta 00m:00s]
>nvme2n1: (groupid=0, jobs=1): err= 0: pid=2525: Mon Nov  5 05:22:57 2018
>  mixed: IOPS=11.2k, BW=43.6MiB/s (45.7MB/s)(436MiB/10001msec)
>    slat (nsec): min=4848, max=208595, avg=5126.33, stdev=771.27
>    clat (nsec): min=1845, max=3405.8k, avg=83004.99, stdev=33577.36
>     lat (usec): min=15, max=3411, avg=88.26, stdev=33.59
>    clat percentiles (usec):
>     |  1.00th=[   15],  5.00th=[   58], 10.00th=[   59], 20.00th=[   59],
>     | 30.00th=[   60], 40.00th=[   77], 50.00th=[   77], 60.00th=[   78],
>     | 70.00th=[  111], 80.00th=[  113], 90.00th=[  113], 95.00th=[  115],
>     | 99.00th=[  186], 99.50th=[  186], 99.90th=[  190], 99.95th=[  192],
>     | 99.99th=[  375]
>   bw (  KiB/s): min=43816, max=45016, per=99.98%, avg=44639.58, stdev=312.88, samples=19
>   iops        : min=10954, max=11254, avg=11159.89, stdev=78.22, samples=19
>  lat (usec)   : 2=0.01%, 20=2.83%, 50=0.01%, 100=64.57%, 250=32.59%
>  lat (usec)   : 500=0.01%, 750=0.01%
>  lat (msec)   : 2=0.01%, 4=0.01%
>  cpu          : usr=4.02%, sys=10.33%, ctx=111630, majf=0, minf=12
>  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
>     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>     issued rwts: total=111631,0,0,0 short=0,0,0,0 dropped=0,0,0,0
>     latency   : target=0, window=0, percentile=100.00%, depth=1
>Run status group 0 (all jobs):
>  MIXED: bw=43.6MiB/s (45.7MB/s), 43.6MiB/s-43.6MiB/s (45.7MB/s-45.7MB/s), io=436MiB (457MB), run=10001-10001msec Disk stats (read/write):
>  nvme2n1: ios=110275/0, merge=0/0, ticks=0/0, in_queue=8596, util=86.22%
>
>
>Thanks,
>Joe
>
>> -----Original Message-----
>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
>> Sent: Sunday, November 4, 2018 8:40 PM
>> To: Storage Performance Development Kit < spdk(a)lists.01.org >
>> Subject: Re: [SPDK] FIO NVMe Performance Results
>> 
>> For the SPDK plugin, from the below result, looks like only 1 job 
>> started with
>> 32 queue depth:
>> 
>> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s] <<<<
>> 
>> Try to update the FIO configuration file as
>> 
>> [job1]
>> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1 [job2] 
>> filename=trtype=PCIe
>> traddr=0000.1c.00.0 ns=1 ....
>> 
>> The kernel FIO is with 4 jobs and each job of 32 queue depth. For 
>> SPDK, you can configure as the same, or just configure two jobs and 
>> each job of 64 queue depth. The overall queue depth to the device is same of 128.
>> 
>> Thanks,
>> Gang
>> 
>> -----Original Message-----
>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Gruher, 
>> Joseph R
>> Sent: Monday, November 5, 2018 10:49 AM
>> To: Storage Performance Development Kit < spdk(a)lists.01.org >
>> Subject: [SPDK] FIO NVMe Performance Results
>> 
>> Hi folks-
>> 
>> I'm testing SPDK 18.10 on Ubuntu 18.04 with kernel 4.18.16.  I used 
>> FIO with the kernel NVMe driver to measure the performance of a local 
>> (PCIe
>> attached) Intel P4500 NVMe device on a 4KB random read workload and 
>> obtained 477K IOPS, roughly in line with the drive spec.  Then I 
>> tested the same drive with the SPDK FIO plugin and only achieved 13K 
>> IOPS.  The FIO test files and the results are pasted below.  Any ideas 
>> where I'm going wrong here?
>> 
>> Thanks!
>> 
>> 
>> don(a)donst201:~/fio/single/rr$ ls
>> /sys/devices/pci0000:17/0000:17:02.0/0000:1c:00.0/nvme/nvme1/
>> address  cntlid  dev  device  firmware_rev  model  nvme2n1  power 
>> rescan_controller  reset_controller  serial  state  subsysnqn 
>> subsystem transport  uevent
>> 
>> 
>> don(a)donst201:~/fio/single/rr$ cat nvme2n1.ini |grep -v '#'
>> [global]
>> rw=randrw
>> rwmixread=100
>> numjobs=4
>> iodepth=32
>> bs=4k
>> direct=1
>> thread=1
>> time_based=1
>> ramp_time=0
>> runtime=10
>> ioengine=libaio
>> group_reporting=1
>> unified_rw_reporting=1
>> exitall=1
>> randrepeat=0
>> norandommap=1
>> [nvme2n1]
>> filename=/dev/nvme2n1
>> 
>> 
>> don(a)donst201:~/fio/single/rr$ sudo fio nvme2n1.ini
>> nvme2n1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 
>> 4096B-4096B, ioengine=libaio, iodepth=32 ...
>> fio-3.11
>> Starting 4 threads
>> Jobs: 4 (f=4): [r(4)][100.0%][r=1874MiB/s][r=480k IOPS][eta 00m:00s]
>> nvme2n1: (groupid=0, jobs=4): err= 0: pid=2575: Mon Nov  5 02:26:25 2018
>>   mixed: IOPS=477k, BW=1862MiB/s (1952MB/s)(18.2GiB/10001msec)
>>     slat (nsec): min=1295, max=333088, avg=2216.35, stdev=927.89
>>    clat (nsec): min=506, max=5818.8k, avg=265924.16, stdev=223761.59
>>      lat (usec): min=6, max=5821, avg=268.19, stdev=223.76
>>     clat percentiles (usec):
>>      |  1.00th=[   13],  5.00th=[   74], 10.00th=[   87], 20.00th=[  115],
>>      | 30.00th=[  139], 40.00th=[  167], 50.00th=[  204], 60.00th=[  247],
>>      | 70.00th=[  306], 80.00th=[  388], 90.00th=[  523], 95.00th=[  660],
>>      | 99.00th=[  988], 99.50th=[ 1156], 99.90th=[ 2343], 99.95th=[ 3195],
>>      | 99.99th=[ 4424]
>>    bw (  KiB/s): min=448192, max=483104, per=25.00%, avg=476597.10, 
>> stdev=6748.78, samples=80
>>    iops        : min=112048, max=120776, avg=119149.27, stdev=1687.20,
>> samples=80
>>   lat (nsec)   : 750=0.01%
>>   lat (usec)   : 10=0.52%, 20=1.25%, 50=0.25%, 100=13.38%, 250=45.13%
>>   lat (usec)   : 500=28.23%, 750=8.08%, 1000=2.22%
>>   lat (msec)   : 2=0.82%, 4=0.10%, 10=0.02%
>>   cpu          : usr=14.08%, sys=36.02%, ctx=1930207, majf=0, minf=697
>>   IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>> >=64=0.0%
>>      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>> >=64=0.0%
>>      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>> >=64=0.0%
>>      issued rwts: total=4766095,0,0,0 short=0,0,0,0 dropped=0,0,0,0
>>      latency   : target=0, window=0, percentile=100.00%, depth=32
>> Run status group 0 (all jobs):
>>   MIXED: bw=1862MiB/s (1952MB/s), 1862MiB/s-1862MiB/s (1952MB/s- 
>> 1952MB/s), io=18.2GiB (19.5GB), run=10001-10001msec Disk stats
>> (read/write):
>>   nvme2n1: ios=4709190/0, merge=0/0, ticks=4096/0, in_queue=1388752, 
>> util=100.00%
>> 
>> 
>> don(a)donst201:~/fio/single/rr$ sudo
>> /home/don/install/spdk/spdk/scripts/setup.sh
>> Active mountpoints on /dev/nvme0n1, so not binding PCI dev 
>> 0000:03:00.0
>> 0000:1c:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:1d:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:5e:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:5f:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:62:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:63:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:64:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:65:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:da:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:db:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:dc:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:dd:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:e0:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:e1:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:e2:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:e3:00.0 (8086 0a54): nvme -> uio_pci_generic
>> 0000:00:04.0 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:00:04.1 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:00:04.2 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:00:04.3 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:00:04.4 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:00:04.5 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:00:04.6 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:00:04.7 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:80:04.0 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:80:04.1 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:80:04.2 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:80:04.3 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:80:04.4 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:80:04.5 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:80:04.6 (8086 2021): ioatdma -> uio_pci_generic
>> 0000:80:04.7 (8086 2021): ioatdma -> uio_pci_generic
>> 
>> 
>> don(a)donst201:~/fio/single/rr$ cat 1c.ini |grep -v '#'
>> [global]
>> rw=randrw
>> rwmixread=100
>> numjobs=4
>> iodepth=32
>> bs=4k
>> direct=1
>> thread=1
>> time_based=1
>> ramp_time=0
>> runtime=10
>> ioengine=/home/don/install/spdk/spdk/examples/nvme/fio_plugin/fio_plug
>> in
>> group_reporting=1
>> unified_rw_reporting=1
>> exitall=1
>> randrepeat=0
>> norandommap=1
>> [0000.1c.00.0]
>> filename=trtype=PCIe traddr=0000.1c.00.0 ns=1
>> 
>> 
>> don(a)donst201:~/fio/single/rr$ sudo fio 1c.ini
>> 0000.1c.00.0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, 
>> (T) 4096B-4096B, ioengine=spdk, iodepth=32 ...
>> fio-3.11
>> Starting 4 threads
>> Starting SPDK v18.10 / DPDK 18.08.0 initialization...
>> [ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 512 --file-
>> prefix=spdk_pid3668 ]
>> EAL: Detected 36 lcore(s)
>> EAL: Detected 2 NUMA nodes
>> EAL: No free hugepages reported in hugepages-1048576kB
>> EAL: Probing VFIO support...
>> EAL: PCI device 0000:1c:00.0 on NUMA socket 0
>> EAL:   probe driver: 8086:a54 spdk_nvme
>> Jobs: 1 (f=0): [f(1),_(3)][100.0%][eta 00m:00s]
>> 0000.1c.00.0: (groupid=0, jobs=4): err= 0: pid=3709: Mon Nov  5 
>> 02:28:29
>> 2018
>>   mixed: IOPS=13.2k, BW=51.6MiB/s (54.1MB/s)(517MiB/10011msec)
>>     slat (nsec): min=109, max=15344, avg=127.91, stdev=73.44
>>     clat (usec): min=187, max=18715, avg=9683.80, stdev=650.54
>>      lat (usec): min=193, max=18715, avg=9683.93, stdev=650.53
>>     clat percentiles (usec):
>>      |  1.00th=[ 8717],  5.00th=[ 8848], 10.00th=[ 8979], 20.00th=[ 9110],
>>      | 30.00th=[ 9241], 40.00th=[ 9372], 50.00th=[ 9503], 60.00th=[ 9765],
>>      | 70.00th=[10028], 80.00th=[10159], 90.00th=[10552], 95.00th=[10945],
>>      | 99.00th=[11338], 99.50th=[11338], 99.90th=[11731], 99.95th=[13960],
>>      | 99.99th=[17695]
>>    bw (  KiB/s): min=13080, max=13304, per=25.00%, avg=13214.70, 
>> stdev=55.30, samples=80
>>    iops        : min= 3270, max= 3326, avg=3303.65, stdev=13.84, samples=80
>>   lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
>>   lat (msec)   : 2=0.01%, 4=0.02%, 10=70.85%, 20=29.11%
>>   cpu          : usr=100.01%, sys=0.00%, ctx=23, majf=0, minf=0
>>   IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
>> >=64=0.0%
>>      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>> >=64=0.0%
>>      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>> >=64=0.0%
>>      issued rwts: total=132300,0,0,0 short=0,0,0,0 dropped=0,0,0,0
>>      latency   : target=0, window=0, percentile=100.00%, depth=32
>> Run status group 0 (all jobs):
>>   MIXED: bw=51.6MiB/s (54.1MB/s), 51.6MiB/s-51.6MiB/s (54.1MB/s- 
>> 54.1MB/s), io=517MiB (542MB), run=10011-10011msec
>> 
>> _______________________________________________
>> SPDK mailing list
>>  SPDK(a)lists.01.org
>>  https://lists.01.org/mailman/listinfo/spdk
>> _______________________________________________
>> SPDK mailing list
>>  SPDK(a)lists.01.org
>>  https://lists.01.org/mailman/listinfo/spdk
>_______________________________________________
>SPDK mailing list
>SPDK(a)lists.01.org
>https://lists.01.org/mailman/listinfo/spdk
>_______________________________________________
>SPDK mailing list
>SPDK(a)lists.01.org
>https://lists.01.org/mailman/listinfo/spdk


-- 
Маров Алексей