kvm + ceph performance issues

kvm + ceph performance issues

[Wolfgang Hennerbichler]

hi,

I'm running a virtual machine with the following command:

LC_ALL=C
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-1.0 -enable-kvm -m 4096 -smp
2,sockets=2,cores=1,threads=1 -name korfu_ceph -uuid
a9131b8f-d087-26f4-2ca9-018505f11838 -nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/korfu_ceph.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime
-no-shutdown -device lsi,id=scsi0,bus=pci.0,addr=0x4 -drive
file=rbd:rd/korfu:rbd_cache=1:mon_host=rd-clusternode21\:6789\;rd-clusternode22\:6789,if=none,id=drive-ide0-0-0,format=raw
-device
ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
-usb -vnc 127.0.0.1:0 -vga std -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5

kvm-version is that from ubuntu LTS: 1.0+noroms-0ubuntu14.7

When I read or write big files the system basically gets unusable
(mouse-cursor in VNC jerks across the screen, i/o is mostly on
blocking). I know it is related to ceph in a way, but also to KVM, as it
seems that there a lot of IRQ's happening (or how do you explain the
mourse cursor in VNC jerking and lagging behind time?_) . Ceph Mailing
List doesn't really help. High CPU load doesn't hurt the machine, it's
only Harddisk I/O. Oh, and the main host running kvm doesn't really
suffer, too. some i/o waiting, but not really swapping or something.

Here's my libvirt-config if it is of any help:
http://pastie.org/6411055

any hints would REALLY be appreciated...

Wolfgang

Re: kvm + ceph performance issues

[Stefan Hajnoczi]

On Thu, Mar 07, 2013 at 12:57:55PM +0100, Wolfgang Hennerbichler wrote:
> I'm running a virtual machine with the following command:
> 
> LC_ALL=C
> PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
> QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-1.0 -enable-kvm -m 4096 -smp
> 2,sockets=2,cores=1,threads=1 -name korfu_ceph -uuid
> a9131b8f-d087-26f4-2ca9-018505f11838 -nodefconfig -nodefaults -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/korfu_ceph.monitor,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime
> -no-shutdown -device lsi,id=scsi0,bus=pci.0,addr=0x4 -drive
> file=rbd:rd/korfu:rbd_cache=1:mon_host=rd-clusternode21\:6789\;rd-clusternode22\:6789,if=none,id=drive-ide0-0-0,format=raw
> -device
> ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
> -usb -vnc 127.0.0.1:0 -vga std -device
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
> 
> kvm-version is that from ubuntu LTS: 1.0+noroms-0ubuntu14.7
> 
> When I read or write big files the system basically gets unusable
> (mouse-cursor in VNC jerks across the screen, i/o is mostly on
> blocking). I know it is related to ceph in a way, but also to KVM, as it
> seems that there a lot of IRQ's happening (or how do you explain the
> mourse cursor in VNC jerking and lagging behind time?_) . Ceph Mailing
> List doesn't really help. High CPU load doesn't hurt the machine, it's
> only Harddisk I/O. Oh, and the main host running kvm doesn't really
> suffer, too. some i/o waiting, but not really swapping or something.
> 
> Here's my libvirt-config if it is of any help:
> http://pastie.org/6411055
> 
> any hints would REALLY be appreciated...

Please try using virtio-blk instead of IDE.

If the guest still jerks try using the Linux rbd block driver instead of
QEMU -drive rbd:.  I haven't used Ceph much but there should be
documentation on attaching a RADOS block device to your Linux host.
Tell QEMU to use the RADOS block device like a regular file (you are now
using the kernel driver instead of QEMU code to talk to the Ceph
cluster).

Please let us know the outcome.  If you find that virtio-blk does not
make much difference but using the kernel rbd driver does, then this
suggests there is a bug in QEMU's block/rbd.c.

Stefan

Re: kvm + ceph performance issues

[Wolfgang Hennerbichler]

On 03/08/2013 09:35 AM, Stefan Hajnoczi wrote:

> Please try using virtio-blk instead of IDE.

there is no difference if I use virtio-blk.

> If the guest still jerks try using the Linux rbd block driver instead of
> QEMU -drive rbd:.  I haven't used Ceph much but there should be
> documentation on attaching a RADOS block device to your Linux host.
> Tell QEMU to use the RADOS block device like a regular file (you are now
> using the kernel driver instead of QEMU code to talk to the Ceph
> cluster).

This doesn't work yet on the linux-kernel with format 2 images. Format 2
is capable of read/write snapshots and cooler cloning-stuff. I need this
for backup-reasons.

> Please let us know the outcome.  If you find that virtio-blk does not
> make much difference but using the kernel rbd driver does, then this
> suggests there is a bug in QEMU's block/rbd.c.

I'm pretty sure there is, other ceph users reported the same problems
that I've found. The ceph developers are looking into it.

> Stefan

Wolfgang