5x slower guest disk performance with virtio disk

["Brian J. Murrell" <brian@interlinx.bc.ca>]

[-- Attachment #1: Type: text/plain, Size: 6244 bytes --]

I have a CentOS 6 host system running a CentOS 6 KVM guest and the
guest seems to get about 5x slower disk throughput than the host:

host $ sudo dd of=/dev/datavol/disk1 if=/dev/zero bs=1M oflag=direct count=50000
50000+0 records in
50000+0 records out
52428800000 bytes (52 GB) copied, 150.36 s, 349 MB/s
host $ ssh guest dd of=/dev/vdb if=/dev/zero bs=1M oflag=direct count=50000
50000+0 records in
50000+0 records out
52428800000 bytes (52 GB) copied, 731.007 s, 71.7 MB/s

This is actually a vast improvement over earlier tests where it was a
factor of 40x difference.  But even still, I thought virtio disks were
supposed to be much closer to native host speed than 5x.

/dev/datavol/disk1 is a RAIDed LV in an LVM volume group consisting of 5 disks:

$ sudo pvs
  PV         VG      Fmt  Attr PSize   PFree  
  /dev/sda3          lvm2 a-   420.16g 420.16g
  /dev/sdb   datavol lvm2 a-   465.76g 227.66g
  /dev/sdc   datavol lvm2 a-   465.76g 245.76g
  /dev/sdd   datavol lvm2 a-   465.76g 245.76g
  /dev/sde   datavol lvm2 a-   465.76g 245.76g
  /dev/sdf   datavol lvm2 a-   465.76g 245.76g
$ sudo lvdisplay --maps /dev/datavol/disk1
  --- Logical volume ---
  LV Name                /dev/datavol/disk1
  VG Name                datavol
  LV UUID                3gD1N3-ybAU-GzUO-8QBV-b6op-lsK9-GMNm3w
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                500.00 GiB
  Current LE             128000
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1
   
  --- Segments ---
  Logical extent 0 to 127999:
    Type		striped
    Stripes		5
    Stripe size		4.00 KiB
    Stripe 0:
      Physical volume	/dev/sdb
      Physical extents	5120 to 30719
    Stripe 1:
      Physical volume	/dev/sdc
      Physical extents	5120 to 30719
    Stripe 2:
      Physical volume	/dev/sdd
      Physical extents	5120 to 30719
    Stripe 3:
      Physical volume	/dev/sde
      Physical extents	5120 to 30719
    Stripe 4:
      Physical volume	/dev/sdf
      Physical extents	5120 to 30719

The KVM command line is as follows:

$ tr '\0' '\n' < /proc/9409/cmdline 
/usr/libexec/qemu-kvm
-S
-M
rhel6.0.0
-enable-kvm
-m
1024
-smp
1,sockets=1,cores=1,threads=1
-name
guest
-uuid
e2bf97e0-cfb7-444c-abc3-9efe262d8efe
-nodefconfig
-nodefaults
-chardev
socket,id=monitor,path=/var/lib/libvirt/qemu/guest.monitor,server,nowait
-mon
chardev=monitor,mode=control
-rtc
base=utc
-boot
c
-drive
file=/var/lib/libvirt/images/cdrom.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
-device
ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
-drive
file=/var/lib/libvirt/images/guest.qcow2,if=none,id=drive-virtio-disk0,boot=on,format=qcow2
-device
virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
-drive
file=/dev/datavol/disk1,if=none,id=drive-virtio-disk1,format=raw
-device
virtio-blk-pci,bus=pci.0,addr=0x6,drive=drive-virtio-disk1,id=virtio-disk1
-netdev
tap,fd=23,id=hostnet0
-device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:ec:8d:47,bus=pci.0,addr=0x3
-chardev
pty,id=serial0
-device
isa-serial,chardev=serial0
-usb
-vnc
127.0.0.1:2
-vga
cirrus
-device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5

That is started from virt-manager given the following configuration:

<domain type='kvm' id='18'>
  <name>guest</name>
  <uuid>e2bf97e0-cfb7-444c-abc3-9efe262d8efe</uuid>
  <memory>1048576</memory>
  <currentMemory>1048576</currentMemory>
  <vcpu>1</vcpu>
  <os>
    <type arch='x86_64' machine='rhel6.0.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/cdrom.iso'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <alias name='ide0-1-0'/>
      <address type='drive' controller='0' bus='1' unit='0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/guest.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/datavol/disk1'/>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
    <controller type='ide' index='0'>
      <alias name='ide0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:ec:8d:47'/>
      <source network='default'/>
      <target dev='vnet2'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/3'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/3'>
      <source path='/dev/pts/3'/>
      <target port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='5902' autoport='yes'/>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
</domain>

Any ideas why the huge performance difference?

Cheers,
b.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]