From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ryan Harper Subject: Re: Live migration of x86-64 kvm guests broken? Date: Wed, 2 Dec 2009 12:10:28 -0600 Message-ID: <20091202181028.GS27346@us.ibm.com> References: <4B1686AB.4020206@siemens.com> <20091202164408.GQ27346@us.ibm.com> <4B16AC68.2090304@siemens.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: Ryan Harper , qemu-devel , kvm , Juan Quintela , Pierre Riteau , Liran Schour To: Jan Kiszka Return-path: Received: from e5.ny.us.ibm.com ([32.97.182.145]:43916 "EHLO e5.ny.us.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754897AbZLBSK0 (ORCPT ); Wed, 2 Dec 2009 13:10:26 -0500 Received: from d01relay06.pok.ibm.com (d01relay06.pok.ibm.com [9.56.227.116]) by e5.ny.us.ibm.com (8.14.3/8.13.1) with ESMTP id nB2HxOJd022387 for ; Wed, 2 Dec 2009 12:59:24 -0500 Received: from d01av02.pok.ibm.com (d01av02.pok.ibm.com [9.56.224.216]) by d01relay06.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id nB2IAWft1884354 for ; Wed, 2 Dec 2009 13:10:32 -0500 Received: from d01av02.pok.ibm.com (loopback [127.0.0.1]) by d01av02.pok.ibm.com (8.14.3/8.13.1/NCO v10.0 AVout) with ESMTP id nB2E0hpA020835 for ; Wed, 2 Dec 2009 09:00:44 -0500 Content-Disposition: inline In-Reply-To: <4B16AC68.2090304@siemens.com> Sender: kvm-owner@vger.kernel.org List-ID: * Jan Kiszka [2009-12-02 12:07]: > Ryan Harper wrote: > > * Jan Kiszka [2009-12-02 09:28]: > >> Hi, > >> > >> I'm facing stalled x86-64 guests after live migration when using kvm > >> (share disk images). This does not happen with x86-32 guests or when > >> disabling kvm. Both qemu and qemu-kvm git heads are affected (recent > >> vmstate fixes applied). Running I/O load during the migration (e.g. a > >> simple "ls -R /") seems to trigger it reliably. > >> > >> Can anyone confirm / comment on this? Do we fail to restore some magic > >> state that only long mode cares about? > > > > Just tested upstream qemu.git (with vmstate fixes) and I could migrate > > RHEL 5.3 64-bit guest with your light io load over localhost. I'll try > > remote hosts next. > > Hell, too many variables. It looks like the kernel modules have some if > not the ultimate impact: kvm-mod-2.6.31.6b on my 2.6.27 host works fine, > kvm-kmod stable-2.6.32 as well as master cause troubles. > > What is your host kernel and/or kvm module version? Stock 64-bit Ubuntu Karmic: % uname -a Linux symmetry 2.6.31-15-generic #50-Ubuntu SMP Tue Nov 10 14:53:52 UTC 2009 x86_64 GNU/Linux % modinfo kvm filename: /lib/modules/2.6.31-15-generic/kernel/arch/x86/kvm/kvm.ko license: GPL author: Qumranet srcversion: 82D6B673524596F9CF3E84C depends: vermagic: 2.6.31-15-generic SMP mod_unload modversions parm: oos_shadow:bool % modinfo kvm-intel filename: /lib/modules/2.6.31-15-generic/kernel/arch/x86/kvm/kvm-intel.ko license: GPL author: Qumranet srcversion: 8FEA479DFCD7F174DA7864E depends: kvm vermagic: 2.6.31-15-generic SMP mod_unload modversions parm: bypass_guest_pf:bool parm: vpid:bool parm: flexpriority:bool parm: ept:bool parm: emulate_invalid_guest_state:bool -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx ryanh@us.ibm.com From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1NFtex-0000DP-3b for qemu-devel@nongnu.org; Wed, 02 Dec 2009 13:10:39 -0500 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1NFtes-00007D-5I for qemu-devel@nongnu.org; Wed, 02 Dec 2009 13:10:38 -0500 Received: from [199.232.76.173] (port=48143 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1NFter-00006n-US for qemu-devel@nongnu.org; Wed, 02 Dec 2009 13:10:33 -0500 Received: from e9.ny.us.ibm.com ([32.97.182.139]:60854) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1NFter-0003Jo-Kq for qemu-devel@nongnu.org; Wed, 02 Dec 2009 13:10:33 -0500 Received: from d01relay03.pok.ibm.com (d01relay03.pok.ibm.com [9.56.227.235]) by e9.ny.us.ibm.com (8.14.3/8.13.1) with ESMTP id nB2I4XXn001163 for ; Wed, 2 Dec 2009 13:04:33 -0500 Received: from d01av02.pok.ibm.com (d01av02.pok.ibm.com [9.56.224.216]) by d01relay03.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id nB2IAV6W125010 for ; Wed, 2 Dec 2009 13:10:31 -0500 Received: from d01av02.pok.ibm.com (loopback [127.0.0.1]) by d01av02.pok.ibm.com (8.14.3/8.13.1/NCO v10.0 AVout) with ESMTP id nB2E0hp2020835 for ; Wed, 2 Dec 2009 09:00:43 -0500 Date: Wed, 2 Dec 2009 12:10:28 -0600 From: Ryan Harper Message-ID: <20091202181028.GS27346@us.ibm.com> References: <4B1686AB.4020206@siemens.com> <20091202164408.GQ27346@us.ibm.com> <4B16AC68.2090304@siemens.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4B16AC68.2090304@siemens.com> Subject: [Qemu-devel] Re: Live migration of x86-64 kvm guests broken? List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Jan Kiszka Cc: kvm , Juan Quintela , qemu-devel , Ryan Harper , Liran Schour , Pierre Riteau * Jan Kiszka [2009-12-02 12:07]: > Ryan Harper wrote: > > * Jan Kiszka [2009-12-02 09:28]: > >> Hi, > >> > >> I'm facing stalled x86-64 guests after live migration when using kvm > >> (share disk images). This does not happen with x86-32 guests or when > >> disabling kvm. Both qemu and qemu-kvm git heads are affected (recent > >> vmstate fixes applied). Running I/O load during the migration (e.g. a > >> simple "ls -R /") seems to trigger it reliably. > >> > >> Can anyone confirm / comment on this? Do we fail to restore some magic > >> state that only long mode cares about? > > > > Just tested upstream qemu.git (with vmstate fixes) and I could migrate > > RHEL 5.3 64-bit guest with your light io load over localhost. I'll try > > remote hosts next. > > Hell, too many variables. It looks like the kernel modules have some if > not the ultimate impact: kvm-mod-2.6.31.6b on my 2.6.27 host works fine, > kvm-kmod stable-2.6.32 as well as master cause troubles. > > What is your host kernel and/or kvm module version? Stock 64-bit Ubuntu Karmic: % uname -a Linux symmetry 2.6.31-15-generic #50-Ubuntu SMP Tue Nov 10 14:53:52 UTC 2009 x86_64 GNU/Linux % modinfo kvm filename: /lib/modules/2.6.31-15-generic/kernel/arch/x86/kvm/kvm.ko license: GPL author: Qumranet srcversion: 82D6B673524596F9CF3E84C depends: vermagic: 2.6.31-15-generic SMP mod_unload modversions parm: oos_shadow:bool % modinfo kvm-intel filename: /lib/modules/2.6.31-15-generic/kernel/arch/x86/kvm/kvm-intel.ko license: GPL author: Qumranet srcversion: 8FEA479DFCD7F174DA7864E depends: kvm vermagic: 2.6.31-15-generic SMP mod_unload modversions parm: bypass_guest_pf:bool parm: vpid:bool parm: flexpriority:bool parm: ept:bool parm: emulate_invalid_guest_state:bool -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx ryanh@us.ibm.com