From mboxrd@z Thu Jan 1 00:00:00 1970 From: Smart Weblications GmbH - Florian Wiessner Subject: libvirtd + rbd - stale kvm after migrate Date: Thu, 08 Dec 2011 15:47:53 +0100 Message-ID: <4EE0CE19.5060805@smart-weblications.de> Reply-To: f.wiessner@smart-weblications.de Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: QUOTED-PRINTABLE Return-path: Received: from mx04.smart-weblications.de ([188.65.144.39]:35019 "EHLO mx04.smart-weblications.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751651Ab1LHOrg (ORCPT ); Thu, 8 Dec 2011 09:47:36 -0500 Received: from office.smart-weblications.net (office.smart-weblications.net [91.204.168.193]) by mx04.smart-weblications.de (Postfix) with ESMTPA id 9E28227867FD for ; Thu, 8 Dec 2011 14:47:34 +0000 (UTC) Received: from [192.168.200.13] (callosum.office.smart-weblications.net [192.168.200.13]) by office.smart-weblications.net (Postfix) with ESMTP id 493A65B9CCD for ; Thu, 8 Dec 2011 15:47:33 +0100 (CET) Sender: ceph-devel-owner@vger.kernel.org List-ID: To: ceph-devel@vger.kernel.org Hi List, i set up a 4 node cluster using corosync, pacemaker and ceph, created a= 160gb rbd image file with qemu-img and started up one virtual machine using q= emu-kvm. The virtual machine runs fine until i isse crm node standby on the runn= ing host. I can see the VM to migrate to another host and start up without issues= , but the vm seems to be unable to access the disk so all processes inside the VM= are hanging waiting on disk-io. If i migrate the VM back to the host where i started the VM, it runs no= rmally without any problems. It seems that the other host is unable to use the= rbd image - is there anything i am missing here? I also tried to use a file mounted on ceph as image, but then the node = to which the VM wants to migrate gets kernel-oops when trying to access the imag= e file and locks up :( Any help would be highly appreciated! ceph --version ceph version 0.39-54-g745be30 (commit:745be30f517216474d83b9ada2f355217= a984258) virsh --version 0.9.8 qemu-system-x86_64 --version QEMU emulator version 1.0.50, Copyright (c) 2003-2008 Fabrice Bellard rbd --version ceph version 0.39-54-g745be30 (commit:745be30f517216474d83b9ada2f355217= a984258) Executing rbd showmapped on node01 id pool image snap device 0 rbd ns1 - /dev/rbd0 Executing rbd showmapped on node02 id pool image snap device 0 rbd ns1 - /dev/rbd0 Executing rbd showmapped on node03 id pool image snap device 0 rbd ns1 - /dev/rbd0 Executing rbd showmapped on node04 id pool image snap device 0 rbd ns1 - /dev/rbd0 cat /etc/ceph/ceph.conf [global] pid file =3D /var/run/ceph/$name.pid debug ms =3D 1 auth supported =3D cephx osd journal =3D /data/ceph.journal osd_journal_size =3D 512 # filestore journal writeahead =3D true # filestore journal parallel =3D true mds max =3D 4 [mon] mon data =3D /data/ceph/mon [mon.0] host =3D node01 mon addr =3D xxx.xxx.xxx.4:6789 [mon.1] host =3D node02 mon addr =3D xxx.xxx.xxx.5:6789 [mon.2] host =3D node03 mon addr =3D xxx.xxx.xxx.6:6789 [mon.3] host =3D node04 mon addr =3D xxx.xxx.xxx.7:6789 [mds] keyring =3D /etc/ceph/keyring.$name # mds dir max commit size 32 [mds.0] host =3D node01 [mds.1] host =3D node02 [mds.2] host =3D node03 [mds.3] host =3D node04 [osd] sudo =3D true osd data =3D /data/ceph/osd keyring =3D /etc/ceph/keyring.$name [osd.0] host =3D node01 [osd.1] host =3D node02 [osd.2] host =3D node03 [osd.3] host =3D node04 ns1.xml: ns1 350e51e8-2fe5-274f-76c4-58b237bc0fba 1048576 524288 2 hvm destroy restart restart /usr/local/bin/qemu-system-x86_64