Re: [Qemu-devel] [PATCH 0/7] virtio: virtio-blk data plane

[Asias He <asias@redhat.com>]

On 11/21/2012 01:39 PM, Asias He wrote:
> On 11/20/2012 08:25 PM, Stefan Hajnoczi wrote:
>> On Tue, Nov 20, 2012 at 1:21 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>>> On Tue, Nov 20, 2012 at 10:02 AM, Asias He <asias@redhat.com> wrote:
>>>> Hello Stefan,
>>>>
>>>> On 11/15/2012 11:18 PM, Stefan Hajnoczi wrote:
>>>>> This series adds the -device virtio-blk-pci,x-data-plane=on property that
>>>>> enables a high performance I/O codepath.  A dedicated thread is used to process
>>>>> virtio-blk requests outside the global mutex and without going through the QEMU
>>>>> block layer.
>>>>>
>>>>> Khoa Huynh <khoa@us.ibm.com> reported an increase from 140,000 IOPS to 600,000
>>>>> IOPS for a single VM using virtio-blk-data-plane in July:
>>>>>
>>>>>   http://comments.gmane.org/gmane.comp.emulators.kvm.devel/94580
>>>>>
>>>>> The virtio-blk-data-plane approach was originally presented at Linux Plumbers
>>>>> Conference 2010.  The following slides contain a brief overview:
>>>>>
>>>>>   http://linuxplumbersconf.org/2010/ocw/system/presentations/651/original/Optimizing_the_QEMU_Storage_Stack.pdf
>>>>>
>>>>> The basic approach is:
>>>>> 1. Each virtio-blk device has a thread dedicated to handling ioeventfd
>>>>>    signalling when the guest kicks the virtqueue.
>>>>> 2. Requests are processed without going through the QEMU block layer using
>>>>>    Linux AIO directly.
>>>>> 3. Completion interrupts are injected via irqfd from the dedicated thread.
>>>>>
>>>>> To try it out:
>>>>>
>>>>>   qemu -drive if=none,id=drive0,cache=none,aio=native,format=raw,file=...
>>>>>        -device virtio-blk-pci,drive=drive0,scsi=off,x-data-plane=on
>>>>
>>>>
>>>> Is this the latest dataplane bits:
>>>> (git://github.com/stefanha/qemu.git virtio-blk-data-plane)
>>>>
>>>> commit 7872075c24fa01c925d4f41faa9d04ce69bf5328
>>>> Author: Stefan Hajnoczi <stefanha@redhat.com>
>>>> Date:   Wed Nov 14 15:45:38 2012 +0100
>>>>
>>>>     virtio-blk: add x-data-plane=on|off performance feature
>>>>
>>>>
>>>> With this commit on a ramdisk based box, I am seeing about 10K IOPS with
>>>> x-data-plane on and 90K IOPS with x-data-plane off.
>>>>
>>>> Any ideas?
>>>>
>>>> Command line I used:
>>>>
>>>> IMG=/dev/ram0
>>>> x86_64-softmmu/qemu-system-x86_64 \
>>>> -drive file=/root/img/sid.img,if=ide \
>>>> -drive file=${IMG},if=none,cache=none,aio=native,id=disk1 -device
>>>> virtio-blk-pci,x-data-plane=off,drive=disk1,scsi=off \
>>>> -kernel $KERNEL -append "root=/dev/sdb1 console=tty0" \
>>>> -L /tmp/qemu-dataplane/share/qemu/ -nographic -vnc :0 -enable-kvm -m
>>>> 2048 -smp 4 -cpu qemu64,+x2apic -M pc
>>>
>>> Was just about to send out the latest patch series which addresses
>>> review comments, so I have tested the latest code
>>> (61b70fef489ce51ecd18d69afb9622c110b9315c).
>>
>> Rebased onto qemu.git/master before sending out.  The commit ID is now:
>> cf6ed6406543ecc43895012a9ac9665e3753d5e8
>>
>> https://github.com/stefanha/qemu/commits/virtio-blk-data-plane
>>
>> Stefan
> 
> Ok, thanks. /me trying

Hi Stefan,

If I enable the merge in guest the IOPS for seq read/write goes up to
~400K/300K. If I disable the merge in guest the IOPS drops to ~17K/24K
for seq read/write (which is similar to the result I posted yesterday,
with merge disalbed). Could you please also share the numbers for rand
read and write in your setup?


1.(With merge enabled in guest + dataplane on)
echo noop > /sys/block/vda/queue/scheduler
echo 0 > /sys/block/vda/queue/nomerges
-------------------------------------
  read : io=0 B, bw=1575.2MB/s, iops=403453 , runt= 10396msec
  write: io=0 B, bw=1224.1MB/s, iops=313592 , runt= 13375msec
  read : io=0 B, bw=99534KB/s, iops=24883 , runt=168558msec
  write: io=0 B, bw=197695KB/s, iops=49423 , runt= 84864msec
    clat (usec): min=102 , max=395042 , avg=2167.18, stdev=4307.91
    clat (usec): min=89 , max=636874 , avg=2728.93, stdev=6777.71
    clat (usec): min=285 , max=4737.1K, avg=39939.42, stdev=87137.41
    clat (usec): min=74 , max=1408.1K, avg=18575.50, stdev=47752.13
  cpu          : usr=79.16%, sys=261.43%, ctx=1837824, majf=0, minf=58
  cpu          : usr=73.61%, sys=251.80%, ctx=1585892, majf=0, minf=7
  cpu          : usr=17.03%, sys=71.03%, ctx=6427788, majf=0, minf=6
  cpu          : usr=25.51%, sys=88.46%, ctx=5117624, majf=0, minf=1
  vda: ios=5037311/5303500, merge=3351489/3084584,
ticks=30674372/15110818, in_queue=45816533, util=97.50%
 41:      40099      40163      40110      40157   PCI-MSI-edge
virtio0-requests
 41: interrupt in total: 160529
fio --exec_prerun="echo 3 > /proc/sys/vm/drop_caches" --group_reporting
--ioscheduler=noop --thread --bs=4k --size=512MB --direct=1 --numjobs=16
--ioengine=libaio --iodepth=64 --loops=3 --ramp_time=0
--filename=/dev/vda --name=seq-read --stonewall --rw=read
--name=seq-write --stonewall --rw=write --name=rnd-read --stonewall
--rw=randread --name=rnd-write --stonewall --rw=randwrite

2.(With merge diabled in guest + dataplane on)
echo noop > /sys/block/vda/queue/scheduler
echo 2 > /sys/block/vda/queue/nomerges
-------------------------------------
  read : io=0 B, bw=69185KB/s, iops=17296 , runt=242497msec
  write: io=0 B, bw=96219KB/s, iops=24054 , runt=174365msec
  read : io=0 B, bw=90866KB/s, iops=22716 , runt=184637msec
  write: io=0 B, bw=202018KB/s, iops=50504 , runt= 83048msec
    clat (usec): min=98 , max=1719.7K, avg=57623.32, stdev=84730.86
    clat (usec): min=0 , max=1372.8K, avg=41286.47, stdev=73252.45
    clat (usec): min=82 , max=1308.8K, avg=43828.82, stdev=73483.73
    clat (usec): min=0 , max=1239.6K, avg=18445.77, stdev=49099.15
  cpu          : usr=10.12%, sys=72.64%, ctx=7942850, majf=0, minf=62
  cpu          : usr=14.72%, sys=78.99%, ctx=7358973, majf=0, minf=3
  cpu          : usr=16.34%, sys=72.60%, ctx=7394674, majf=0, minf=5
  cpu          : usr=29.69%, sys=83.41%, ctx=5262809, majf=0, minf=4
  vda: ios=8389288/8388552, merge=0/0, ticks=76013872/44425860,
in_queue=120493774, util=99.58%
 41:      89414      89456      89504      89534   PCI-MSI-edge
virtio0-requests
 41: interrupt in total: 357908
fio --exec_prerun="echo 3 > /proc/sys/vm/drop_caches" --group_reporting
--ioscheduler=noop --thread --bs=4k --size=512MB --direct=1 --numjobs=16
--ioengine=libaio --iodepth=64 --loops=3 --ramp_time=0
--filename=/dev/vda --name=seq-read --stonewall --rw=read
--name=seq-write --stonewall --rw=write --name=rnd-read --stonewall
--rw=randread --name=rnd-write --stonewall --rw=randwrite

3.(With merge enabled in guest + dataplane off)
-------------------------------------
  read : io=0 B, bw=810220KB/s, iops=202554 , runt= 20707msec
  write: io=0 B, bw=999.97MB/s, iops=255984 , runt= 16385msec
  read : io=0 B, bw=338066KB/s, iops=84516 , runt= 49627msec
  write: io=0 B, bw=455420KB/s, iops=113854 , runt= 36839msec
    clat (usec): min=0 , max=26340 , avg=5019.36, stdev=2185.04
    clat (usec): min=58 , max=21572 , avg=3972.33, stdev=1708.00
    clat (usec): min=34 , max=90185 , avg=11879.72, stdev=8054.58
    clat (usec): min=189 , max=122825 , avg=8822.87, stdev=4608.65
  cpu          : usr=44.89%, sys=141.08%, ctx=1611401, majf=0, minf=54
  cpu          : usr=58.46%, sys=177.50%, ctx=1582260, majf=0, minf=0
  cpu          : usr=21.09%, sys=63.61%, ctx=7609871, majf=0, minf=2
  cpu          : usr=28.88%, sys=73.51%, ctx=8140689, majf=0, minf=10
  vda: ios=5222932/5209798, merge=3164880/3163834,
ticks=10133305/7618081, in_queue=17773509, util=99.46%
 41:     286378     284870     285478     285759   PCI-MSI-edge
virtio0-requests
 41: interrupt in total: 1142485
fio --exec_prerun="echo 3 > /proc/sys/vm/drop_caches" --group_reporting
--ioscheduler=noop --thread --bs=4k --size=512MB --direct=1 --numjobs=16
--ioengine=libaio --iodepth=64 --loops=3 --ramp_time=0
--filename=/dev/vda --name=seq-read --stonewall --rw=read
--name=seq-write --stonewall --rw=write --name=rnd-read --stonewall
--rw=randread --name=rnd-write --stonewall --rw=randwrite


4.(With merge disabled in guest + dataplane off)
-------------------------------------
  read : io=0 B, bw=331147KB/s, iops=82786 , runt= 50664msec
  write: io=0 B, bw=431802KB/s, iops=107950 , runt= 38854msec
  read : io=0 B, bw=376424KB/s, iops=94105 , runt= 44570msec
  write: io=0 B, bw=373200KB/s, iops=93300 , runt= 44955msec
    clat (usec): min=84 , max=99635 , avg=12075.27, stdev=7899.12
    clat (usec): min=97 , max=84882 , avg=9289.49, stdev=5688.96
    clat (usec): min=56 , max=76176 , avg=10662.84, stdev=6375.63
    clat (usec): min=158 , max=112216 , avg=10762.62, stdev=6412.89
  cpu          : usr=15.16%, sys=59.67%, ctx=8007217, majf=0, minf=62
  cpu          : usr=26.45%, sys=99.42%, ctx=7477108, majf=0, minf=8
  cpu          : usr=29.52%, sys=85.40%, ctx=7889672, majf=0, minf=2
  cpu          : usr=30.71%, sys=86.54%, ctx=7893782, majf=0, minf=2
  vda: ios=8389195/8374171, merge=0/0, ticks=15202863/12550941,
in_queue=27840167, util=99.63%
 41:     179902     179182     179232     179539   PCI-MSI-edge
virtio0-requests
 41: interrupt in total: 717855
fio --exec_prerun="echo 3 > /proc/sys/vm/drop_caches" --group_reporting
--ioscheduler=noop --thread --bs=4k --size=512MB --direct=1 --numjobs=16
--ioengine=libaio --iodepth=64 --loops=3 --ramp_time=0
--filename=/dev/vda --name=seq-read --stonewall --rw=read
--name=seq-write --stonewall --rw=write --name=rnd-read --stonewall
--rw=randread --name=rnd-write --stonewall --rw=randwrite


-- 
Asias