[Qemu-devel] virtio-blk and virtio-scsi performance comparison

[Konstantin Krotov <kkv@clodo.ru>]

Hello list!

I performed tests with fio and obtained results:

*** virtio-scsi with cache=none, io=threads, blok device is md-device 
from mdadm raid1, random r/w, 32 thread from guest (debian, kernel 3.16):

fio fio1
readtest: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, 
iodepth=32
fio-2.1.11
Starting 1 process
Jobs: 1 (f=1): [m(1)] [100.0% done] [126.2MB/125.1MB/0KB /s] 
[32.3K/32.3K/0 iops] [eta 00m:00s]
readtest: (groupid=0, jobs=1): err= 0: pid=707: Wed Apr  8 07:35:01 2015
   read : io=5117.4MB, bw=125830KB/s, iops=31457, runt= 41645msec
     slat (usec): min=4, max=343, avg=11.45, stdev=10.24
     clat (usec): min=104, max=16667, avg=484.09, stdev=121.96
      lat (usec): min=112, max=16672, avg=495.90, stdev=123.67
     clat percentiles (usec):
      |  1.00th=[  302],  5.00th=[  346], 10.00th=[  374], 20.00th=[  406],
      | 30.00th=[  426], 40.00th=[  446], 50.00th=[  462], 60.00th=[  482],
      | 70.00th=[  506], 80.00th=[  540], 90.00th=[  596], 95.00th=[  732],
      | 99.00th=[  948], 99.50th=[  996], 99.90th=[ 1176], 99.95th=[ 1240],
      | 99.99th=[ 1384]
     bw (KB  /s): min=67392, max=135216, per=99.99%, avg=125813.01, 
stdev=12524.05
   write: io=5114.7MB, bw=125763KB/s, iops=31440, runt= 41645msec
     slat (usec): min=4, max=388, avg=11.85, stdev=10.47
     clat (usec): min=147, max=8968, avg=505.23, stdev=127.40
      lat (usec): min=155, max=8973, avg=517.45, stdev=128.97
     clat percentiles (usec):
      |  1.00th=[  334],  5.00th=[  370], 10.00th=[  394], 20.00th=[  426],
      | 30.00th=[  446], 40.00th=[  462], 50.00th=[  478], 60.00th=[  498],
      | 70.00th=[  524], 80.00th=[  556], 90.00th=[  628], 95.00th=[  756],
      | 99.00th=[  988], 99.50th=[ 1064], 99.90th=[ 1288], 99.95th=[ 1368],
      | 99.99th=[ 2224]
     bw (KB  /s): min=67904, max=136384, per=99.99%, avg=125746.89, 
stdev=12449.56
     lat (usec) : 250=0.05%, 500=64.27%, 750=30.80%, 1000=4.20%
     lat (msec) : 2=0.67%, 4=0.01%, 10=0.01%, 20=0.01%
   cpu          : usr=18.03%, sys=76.42%, ctx=26617, majf=0, minf=7
   IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, 
 >=64=0.0%
      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
 >=64=0.0%
      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, 
 >=64=0.0%
      issued    : total=r=1310044/w=1309348/d=0, short=r=0/w=0/d=0
      latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
    READ: io=5117.4MB, aggrb=125829KB/s, minb=125829KB/s, 
maxb=125829KB/s, mint=41645msec, maxt=41645msec
   WRITE: io=5114.7MB, aggrb=125762KB/s, minb=125762KB/s, 
maxb=125762KB/s, mint=41645msec, maxt=41645msec

Disk stats (read/write):
   sda: ios=1302885/1302192, merge=55/0, ticks=281040/321660, 
in_queue=601264, util=99.29%


same guest,
*** virtio-blk with cache=none, io=threads, blok device is md-device 
from mdadm raid1, random r/w, 32 thread from guest (debian, kernel 3.16):

  fio fio1
readtest: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, 
iodepth=32
fio-2.1.11
Starting 1 process
Jobs: 1 (f=1): [m(1)] [100.0% done] [123.7MB/123.3MB/0KB /s] 
[31.7K/31.6K/0 iops] [eta 00m:00s]
readtest: (groupid=0, jobs=1): err= 0: pid=810: Wed Apr  8 07:26:37 2015
   read : io=5117.4MB, bw=148208KB/s, iops=37051, runt= 35357msec
     slat (usec): min=2, max=2513, avg= 7.27, stdev=10.28
     clat (usec): min=104, max=10716, avg=382.30, stdev=113.38
      lat (usec): min=108, max=10719, avg=389.94, stdev=115.48
     clat percentiles (usec):
      |  1.00th=[  215],  5.00th=[  249], 10.00th=[  270], 20.00th=[  298],
      | 30.00th=[  318], 40.00th=[  338], 50.00th=[  358], 60.00th=[  386],
      | 70.00th=[  418], 80.00th=[  462], 90.00th=[  516], 95.00th=[  572],
      | 99.00th=[  756], 99.50th=[  820], 99.90th=[  996], 99.95th=[ 1176],
      | 99.99th=[ 2256]
     bw (KB  /s): min=119296, max=165456, per=99.94%, avg=148124.33, 
stdev=11834.17
   write: io=5114.7MB, bw=148129KB/s, iops=37032, runt= 35357msec
     slat (usec): min=2, max=2851, avg= 7.55, stdev=10.53
     clat (usec): min=172, max=11080, avg=461.92, stdev=137.02
      lat (usec): min=178, max=11086, avg=469.86, stdev=138.05
     clat percentiles (usec):
      |  1.00th=[  278],  5.00th=[  318], 10.00th=[  338], 20.00th=[  366],
      | 30.00th=[  390], 40.00th=[  414], 50.00th=[  438], 60.00th=[  466],
      | 70.00th=[  494], 80.00th=[  532], 90.00th=[  604], 95.00th=[  716],
      | 99.00th=[  900], 99.50th=[  980], 99.90th=[ 1336], 99.95th=[ 1704],
      | 99.99th=[ 3408]
     bw (KB  /s): min=119656, max=166680, per=99.93%, avg=148029.21, 
stdev=11824.30
     lat (usec) : 250=2.71%, 500=77.22%, 750=17.60%, 1000=2.21%
     lat (msec) : 2=0.24%, 4=0.02%, 10=0.01%, 20=0.01%
   cpu          : usr=27.92%, sys=55.44%, ctx=91283, majf=0, minf=7
   IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, 
 >=64=0.0%
      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
 >=64=0.0%
      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, 
 >=64=0.0%
      issued    : total=r=1310044/w=1309348/d=0, short=r=0/w=0/d=0
      latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
    READ: io=5117.4MB, aggrb=148207KB/s, minb=148207KB/s, 
maxb=148207KB/s, mint=35357msec, maxt=35357msec
   WRITE: io=5114.7MB, aggrb=148128KB/s, minb=148128KB/s, 
maxb=148128KB/s, mint=35357msec, maxt=35357msec

Disk stats (read/write):
   vdb: ios=1302512/1301780, merge=0/0, ticks=294828/407184, 
in_queue=701380, util=99.51%

In my tests virtio-scsi shows worse results than virtio-blk.
Host kernel 3.19-3, qemu-system-x86_64 -version
QEMU emulator version 2.0.0.

-- 
WBR
Konstantin V. Krotov