libvirtd + rbd - stale kvm after migrate

[Smart Weblications GmbH - Florian Wiessner <f.wiessner@smart-weblications.de>]

Hi List,


i set up a 4 node cluster using corosync, pacemaker and ceph, created a 160gb
rbd image file with qemu-img and started up one virtual machine using qemu-kvm.

The virtual machine runs fine until i isse crm node standby on the running host.
I can see the VM to migrate to another host and start up without issues, but the
vm seems to be unable to access the disk so all processes inside the VM are
hanging waiting on disk-io.

If i migrate the VM back to the host where i started the VM, it runs normally
without any problems. It seems that the other host is unable to use the rbd
image - is there anything i am missing here?

I also tried to use a file mounted on ceph as image, but then the node to which
the VM wants to migrate gets kernel-oops when trying to access the image file
and locks up :(

Any help would be highly appreciated!

ceph --version
ceph version 0.39-54-g745be30 (commit:745be30f517216474d83b9ada2f355217a984258)
virsh --version
0.9.8
qemu-system-x86_64 --version
QEMU emulator version 1.0.50, Copyright (c) 2003-2008 Fabrice Bellard
rbd --version
ceph version 0.39-54-g745be30 (commit:745be30f517216474d83b9ada2f355217a984258)

Executing  rbd  showmapped  on node01
id      pool    image   snap    device
0       rbd     ns1     -       /dev/rbd0
Executing  rbd  showmapped  on node02
id      pool    image   snap    device
0       rbd     ns1     -       /dev/rbd0
Executing  rbd  showmapped  on node03
id      pool    image   snap    device
0       rbd     ns1     -       /dev/rbd0
Executing  rbd  showmapped  on node04
id      pool    image   snap    device
0       rbd     ns1     -       /dev/rbd0

cat /etc/ceph/ceph.conf
[global]
       pid file = /var/run/ceph/$name.pid
       debug ms = 1
        auth supported = cephx
       osd journal = /data/ceph.journal
       osd_journal_size = 512
#       filestore journal writeahead = true
#       filestore journal parallel = true
        mds max = 4

[mon]
       mon data = /data/ceph/mon
[mon.0]
       host = node01
       mon addr = xxx.xxx.xxx.4:6789
[mon.1]
       host = node02
       mon addr = xxx.xxx.xxx.5:6789
[mon.2]
       host = node03
       mon addr = xxx.xxx.xxx.6:6789
[mon.3]
       host = node04
       mon addr = xxx.xxx.xxx.7:6789

[mds]
        keyring = /etc/ceph/keyring.$name
#       mds dir max commit size 32

[mds.0]
       host = node01
[mds.1]
       host = node02
[mds.2]
       host = node03
[mds.3]
       host = node04


[osd]
       sudo = true
       osd data = /data/ceph/osd
        keyring = /etc/ceph/keyring.$name
[osd.0]
       host = node01
[osd.1]
       host = node02
[osd.2]
       host = node03
[osd.3]
       host = node04

ns1.xml:
<domain type='kvm'>
  <name>ns1</name>
  <uuid>350e51e8-2fe5-274f-76c4-58b237bc0fba</uuid>
  <memory>1048576</memory>
  <currentMemory>524288</currentMemory>
  <vcpu>2</vcpu>
  <os>
    <type arch='x86_64' machine='pc-0.12'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/ceph/debian-6.0.3-amd64-netinst.iso'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' unit='0'/>
    </disk>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw'/>
      <source protocol='rbd' name='rbd/ns1'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:d2:ec:15'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='5900' autoport='no'/>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
</domain>

-- 

Mit freundlichen Grüßen,

Florian Wiessner

Smart Weblications GmbH
Martinsberger Str. 1
D-95119 Naila

fon.: +49 9282 9638 200
fax.: +49 9282 9638 205
24/7: +49 900 144 000 00 - 0,99 EUR/Min*
http://www.smart-weblications.de

--
Sitz der Gesellschaft: Naila
Geschäftsführer: Florian Wiessner
HRB-Nr.: HRB 3840 Amtsgericht Hof
*aus dem dt. Festnetz, ggf. abweichende Preise aus dem Mobilfunknetz
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html