[Qemu-devel] Live migration of x86-64 kvm guests broken?

[Qemu-devel] Live migration of x86-64 kvm guests broken?

[Jan Kiszka]

Hi,

I'm facing stalled x86-64 guests after live migration when using kvm
(share disk images). This does not happen with x86-32 guests or when
disabling kvm. Both qemu and qemu-kvm git heads are affected (recent
vmstate fixes applied). Running I/O load during the migration (e.g. a
simple "ls -R /") seems to trigger it reliably.

Can anyone confirm / comment on this? Do we fail to restore some magic
state that only long mode cares about?

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

[Qemu-devel] Re: Live migration of x86-64 kvm guests broken?

[Ryan Harper]

* Jan Kiszka <jan.kiszka@siemens.com> [2009-12-02 09:28]:
> Hi,
> 
> I'm facing stalled x86-64 guests after live migration when using kvm
> (share disk images). This does not happen with x86-32 guests or when
> disabling kvm. Both qemu and qemu-kvm git heads are affected (recent
> vmstate fixes applied). Running I/O load during the migration (e.g. a
> simple "ls -R /") seems to trigger it reliably.
> 
> Can anyone confirm / comment on this? Do we fail to restore some magic
> state that only long mode cares about?

Just tested upstream qemu.git (with vmstate fixes) and I could migrate
RHEL 5.3 64-bit guest with your light io load over localhost.  I'll try
remote hosts next.


-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
ryanh@us.ibm.com

[Qemu-devel] Re: Live migration of x86-64 kvm guests broken?

[Jan Kiszka]

Ryan Harper wrote:
> * Jan Kiszka <jan.kiszka@siemens.com> [2009-12-02 09:28]:
>> Hi,
>>
>> I'm facing stalled x86-64 guests after live migration when using kvm
>> (share disk images). This does not happen with x86-32 guests or when
>> disabling kvm. Both qemu and qemu-kvm git heads are affected (recent
>> vmstate fixes applied). Running I/O load during the migration (e.g. a
>> simple "ls -R /") seems to trigger it reliably.
>>
>> Can anyone confirm / comment on this? Do we fail to restore some magic
>> state that only long mode cares about?
> 
> Just tested upstream qemu.git (with vmstate fixes) and I could migrate
> RHEL 5.3 64-bit guest with your light io load over localhost.  I'll try
> remote hosts next.

Hell, too many variables. It looks like the kernel modules have some if
not the ultimate impact: kvm-mod-2.6.31.6b on my 2.6.27 host works fine,
kvm-kmod stable-2.6.32 as well as master cause troubles.

What is your host kernel and/or kvm module version?

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

[Qemu-devel] Re: Live migration of x86-64 kvm guests broken?

[Ryan Harper]

* Jan Kiszka <jan.kiszka@siemens.com> [2009-12-02 12:07]:
> Ryan Harper wrote:
> > * Jan Kiszka <jan.kiszka@siemens.com> [2009-12-02 09:28]:
> >> Hi,
> >>
> >> I'm facing stalled x86-64 guests after live migration when using kvm
> >> (share disk images). This does not happen with x86-32 guests or when
> >> disabling kvm. Both qemu and qemu-kvm git heads are affected (recent
> >> vmstate fixes applied). Running I/O load during the migration (e.g. a
> >> simple "ls -R /") seems to trigger it reliably.
> >>
> >> Can anyone confirm / comment on this? Do we fail to restore some magic
> >> state that only long mode cares about?
> > 
> > Just tested upstream qemu.git (with vmstate fixes) and I could migrate
> > RHEL 5.3 64-bit guest with your light io load over localhost.  I'll try
> > remote hosts next.
> 
> Hell, too many variables. It looks like the kernel modules have some if
> not the ultimate impact: kvm-mod-2.6.31.6b on my 2.6.27 host works fine,
> kvm-kmod stable-2.6.32 as well as master cause troubles.
> 
> What is your host kernel and/or kvm module version?

Stock 64-bit Ubuntu Karmic:
% uname -a
Linux symmetry 2.6.31-15-generic #50-Ubuntu SMP Tue Nov 10 14:53:52 UTC 2009 x86_64 GNU/Linux

% modinfo kvm
filename:
/lib/modules/2.6.31-15-generic/kernel/arch/x86/kvm/kvm.ko
license:        GPL
author:         Qumranet
srcversion:     82D6B673524596F9CF3E84C
depends:        
vermagic:       2.6.31-15-generic SMP mod_unload modversions 
parm:           oos_shadow:bool

% modinfo kvm-intel
filename: /lib/modules/2.6.31-15-generic/kernel/arch/x86/kvm/kvm-intel.ko
license:        GPL
author:         Qumranet
srcversion:     8FEA479DFCD7F174DA7864E
depends:        kvm
vermagic:       2.6.31-15-generic SMP mod_unload modversions 
parm:           bypass_guest_pf:bool
parm:           vpid:bool
parm:           flexpriority:bool
parm:           ept:bool
parm:           emulate_invalid_guest_state:bool


-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
ryanh@us.ibm.com

[Qemu-devel] Re: Live migration of x86-64 kvm guests broken?

[Jan Kiszka]

Ryan Harper wrote:
> * Jan Kiszka <jan.kiszka@siemens.com> [2009-12-02 12:07]:
>> Ryan Harper wrote:
>>> * Jan Kiszka <jan.kiszka@siemens.com> [2009-12-02 09:28]:
>>>> Hi,
>>>>
>>>> I'm facing stalled x86-64 guests after live migration when using kvm
>>>> (share disk images). This does not happen with x86-32 guests or when
>>>> disabling kvm. Both qemu and qemu-kvm git heads are affected (recent
>>>> vmstate fixes applied). Running I/O load during the migration (e.g. a
>>>> simple "ls -R /") seems to trigger it reliably.
>>>>
>>>> Can anyone confirm / comment on this? Do we fail to restore some magic
>>>> state that only long mode cares about?
>>> Just tested upstream qemu.git (with vmstate fixes) and I could migrate
>>> RHEL 5.3 64-bit guest with your light io load over localhost.  I'll try
>>> remote hosts next.
>> Hell, too many variables. It looks like the kernel modules have some if
>> not the ultimate impact: kvm-mod-2.6.31.6b on my 2.6.27 host works fine,
>> kvm-kmod stable-2.6.32 as well as master cause troubles.
>>
>> What is your host kernel and/or kvm module version?
> 
> Stock 64-bit Ubuntu Karmic:
> % uname -a
> Linux symmetry 2.6.31-15-generic #50-Ubuntu SMP Tue Nov 10 14:53:52 UTC 2009 x86_64 GNU/Linux
> 
> % modinfo kvm
> filename:
> /lib/modules/2.6.31-15-generic/kernel/arch/x86/kvm/kvm.ko
> license:        GPL
> author:         Qumranet
> srcversion:     82D6B673524596F9CF3E84C
> depends:        
> vermagic:       2.6.31-15-generic SMP mod_unload modversions 
> parm:           oos_shadow:bool
> 
> % modinfo kvm-intel
> filename: /lib/modules/2.6.31-15-generic/kernel/arch/x86/kvm/kvm-intel.ko
> license:        GPL
> author:         Qumranet
> srcversion:     8FEA479DFCD7F174DA7864E
> depends:        kvm
> vermagic:       2.6.31-15-generic SMP mod_unload modversions 
> parm:           bypass_guest_pf:bool
> parm:           vpid:bool
> parm:           flexpriority:bool
> parm:           ept:bool
> parm:           emulate_invalid_guest_state:bool
> 

So far, so consistent. Could you try kvm-kmod-2.6.32-rc7 on top of that?

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

[Qemu-devel] Re: Live migration of x86-64 kvm guests broken?

[Ryan Harper]

* Jan Kiszka <jan.kiszka@siemens.com> [2009-12-02 12:18]:
> Ryan Harper wrote:
> > * Jan Kiszka <jan.kiszka@siemens.com> [2009-12-02 12:07]:
> >> Ryan Harper wrote:
> >>> * Jan Kiszka <jan.kiszka@siemens.com> [2009-12-02 09:28]:
> >>>> Hi,
> >>>>
> >>>> I'm facing stalled x86-64 guests after live migration when using kvm
> >>>> (share disk images). This does not happen with x86-32 guests or when
> >>>> disabling kvm. Both qemu and qemu-kvm git heads are affected (recent
> >>>> vmstate fixes applied). Running I/O load during the migration (e.g. a
> >>>> simple "ls -R /") seems to trigger it reliably.
> >>>>
> >>>> Can anyone confirm / comment on this? Do we fail to restore some magic
> >>>> state that only long mode cares about?
> >>> Just tested upstream qemu.git (with vmstate fixes) and I could migrate
> >>> RHEL 5.3 64-bit guest with your light io load over localhost.  I'll try
> >>> remote hosts next.
> >> Hell, too many variables. It looks like the kernel modules have some if
> >> not the ultimate impact: kvm-mod-2.6.31.6b on my 2.6.27 host works fine,
> >> kvm-kmod stable-2.6.32 as well as master cause troubles.
> >>
> >> What is your host kernel and/or kvm module version?
> > 
> > Stock 64-bit Ubuntu Karmic:
> > % uname -a
> > Linux symmetry 2.6.31-15-generic #50-Ubuntu SMP Tue Nov 10 14:53:52 UTC 2009 x86_64 GNU/Linux
> > 
> > % modinfo kvm
> > filename:
> > /lib/modules/2.6.31-15-generic/kernel/arch/x86/kvm/kvm.ko
> > license:        GPL
> > author:         Qumranet
> > srcversion:     82D6B673524596F9CF3E84C
> > depends:        
> > vermagic:       2.6.31-15-generic SMP mod_unload modversions 
> > parm:           oos_shadow:bool
> > 
> > % modinfo kvm-intel
> > filename: /lib/modules/2.6.31-15-generic/kernel/arch/x86/kvm/kvm-intel.ko
> > license:        GPL
> > author:         Qumranet
> > srcversion:     8FEA479DFCD7F174DA7864E
> > depends:        kvm
> > vermagic:       2.6.31-15-generic SMP mod_unload modversions 
> > parm:           bypass_guest_pf:bool
> > parm:           vpid:bool
> > parm:           flexpriority:bool
> > parm:           ept:bool
> > parm:           emulate_invalid_guest_state:bool
> > 
> 
> So far, so consistent. Could you try kvm-kmod-2.6.32-rc7 on top of that?
> 

Sure, can you remind me of the kvm-kmod build magic for building that
branch?


> Jan
> 
> -- 
> Siemens AG, Corporate Technology, CT T DE IT 1
> Corporate Competence Center Embedded Linux

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
ryanh@us.ibm.com

[Qemu-devel] Re: Live migration of x86-64 kvm guests broken?

[Jan Kiszka]

Ryan Harper wrote:
> * Jan Kiszka <jan.kiszka@siemens.com> [2009-12-02 12:18]:
>> Ryan Harper wrote:
>>> * Jan Kiszka <jan.kiszka@siemens.com> [2009-12-02 12:07]:
>>>> Ryan Harper wrote:
>>>>> * Jan Kiszka <jan.kiszka@siemens.com> [2009-12-02 09:28]:
>>>>>> Hi,
>>>>>>
>>>>>> I'm facing stalled x86-64 guests after live migration when using kvm
>>>>>> (share disk images). This does not happen with x86-32 guests or when
>>>>>> disabling kvm. Both qemu and qemu-kvm git heads are affected (recent
>>>>>> vmstate fixes applied). Running I/O load during the migration (e.g. a
>>>>>> simple "ls -R /") seems to trigger it reliably.
>>>>>>
>>>>>> Can anyone confirm / comment on this? Do we fail to restore some magic
>>>>>> state that only long mode cares about?
>>>>> Just tested upstream qemu.git (with vmstate fixes) and I could migrate
>>>>> RHEL 5.3 64-bit guest with your light io load over localhost.  I'll try
>>>>> remote hosts next.
>>>> Hell, too many variables. It looks like the kernel modules have some if
>>>> not the ultimate impact: kvm-mod-2.6.31.6b on my 2.6.27 host works fine,
>>>> kvm-kmod stable-2.6.32 as well as master cause troubles.
>>>>
>>>> What is your host kernel and/or kvm module version?
>>> Stock 64-bit Ubuntu Karmic:
>>> % uname -a
>>> Linux symmetry 2.6.31-15-generic #50-Ubuntu SMP Tue Nov 10 14:53:52 UTC 2009 x86_64 GNU/Linux
>>>
>>> % modinfo kvm
>>> filename:
>>> /lib/modules/2.6.31-15-generic/kernel/arch/x86/kvm/kvm.ko
>>> license:        GPL
>>> author:         Qumranet
>>> srcversion:     82D6B673524596F9CF3E84C
>>> depends:        
>>> vermagic:       2.6.31-15-generic SMP mod_unload modversions 
>>> parm:           oos_shadow:bool
>>>
>>> % modinfo kvm-intel
>>> filename: /lib/modules/2.6.31-15-generic/kernel/arch/x86/kvm/kvm-intel.ko
>>> license:        GPL
>>> author:         Qumranet
>>> srcversion:     8FEA479DFCD7F174DA7864E
>>> depends:        kvm
>>> vermagic:       2.6.31-15-generic SMP mod_unload modversions 
>>> parm:           bypass_guest_pf:bool
>>> parm:           vpid:bool
>>> parm:           flexpriority:bool
>>> parm:           ept:bool
>>> parm:           emulate_invalid_guest_state:bool
>>>
>> So far, so consistent. Could you try kvm-kmod-2.6.32-rc7 on top of that?
>>
> 
> Sure, can you remind me of the kvm-kmod build magic for building that
> branch?
> 

If you want to build from git, check Wolfgang's nice README. But it
might be easier to just download the tarball from sourceforge.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

[Qemu-devel] Re: Live migration of x86-64 kvm guests broken?

[Ryan Harper]

* Jan Kiszka <jan.kiszka@siemens.com> [2009-12-02 12:45]:
> >>>
> >> So far, so consistent. Could you try kvm-kmod-2.6.32-rc7 on top of that?
> >>
> > 
> > Sure, can you remind me of the kvm-kmod build magic for building that
> > branch?
> > 
> 
> If you want to build from git, check Wolfgang's nice README. But it
> might be easier to just download the tarball from sourceforge.

Indeed, tarball is easiest.  Still working on top of qemu.git+vmstate
fixes.  Here's my invocation:


Source:
sudo ./x86_64-softmmu/qemu-system-x86_64 -L pc-bios -m 512 -drive
file=/media/blackstar/images/rhel5-64.raw,if=ide -net nic,model=e1000
-monitor stdio -enable-kvm

Dest:
sudo ./x86_64-softmmu/qemu-system-x86_64 -L pc-bios -m 512 -drive
file=/media/blackstar/images/rhel5-64.raw,if=ide -net nic,model=e1000
-monitor stdio -enable-kvm -incoming tcp:0:4444

  
Lemme try qemu-kvm.git

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
ryanh@us.ibm.com

[Qemu-devel] Re: Live migration of x86-64 kvm guests broken?

[Ryan Harper]

* Ryan Harper <ryanh@us.ibm.com> [2009-12-02 13:11]:
> * Jan Kiszka <jan.kiszka@siemens.com> [2009-12-02 12:45]:
> > >>>
> > >> So far, so consistent. Could you try kvm-kmod-2.6.32-rc7 on top of that?
> > >>
> > > 
> > > Sure, can you remind me of the kvm-kmod build magic for building that
> > > branch?
> > > 
> > 
> > If you want to build from git, check Wolfgang's nice README. But it
> > might be easier to just download the tarball from sourceforge.
> 
> Indeed, tarball is easiest.  Still working on top of qemu.git+vmstate
> fixes.  Here's my invocation:
> 
> 
> Source:
> sudo ./x86_64-softmmu/qemu-system-x86_64 -L pc-bios -m 512 -drive
> file=/media/blackstar/images/rhel5-64.raw,if=ide -net nic,model=e1000
> -monitor stdio -enable-kvm
> 
> Dest:
> sudo ./x86_64-softmmu/qemu-system-x86_64 -L pc-bios -m 512 -drive
> file=/media/blackstar/images/rhel5-64.raw,if=ide -net nic,model=e1000
> -monitor stdio -enable-kvm -incoming tcp:0:4444
> 
>   
> Lemme try qemu-kvm.git

Working there as well on top of 2.6.32-rc7 kmod modules and kvm-qemu.git
tip+vmstate fixes.  Any other varient?


-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
ryanh@us.ibm.com

[Qemu-devel] Re: Live migration of x86-64 kvm guests broken?

[Jan Kiszka]

Ryan Harper wrote:
> * Ryan Harper <ryanh@us.ibm.com> [2009-12-02 13:11]:
>> * Jan Kiszka <jan.kiszka@siemens.com> [2009-12-02 12:45]:
>>>>> So far, so consistent. Could you try kvm-kmod-2.6.32-rc7 on top of that?
>>>>>
>>>> Sure, can you remind me of the kvm-kmod build magic for building that
>>>> branch?
>>>>
>>> If you want to build from git, check Wolfgang's nice README. But it
>>> might be easier to just download the tarball from sourceforge.
>> Indeed, tarball is easiest.  Still working on top of qemu.git+vmstate
>> fixes.  Here's my invocation:
>>
>>
>> Source:
>> sudo ./x86_64-softmmu/qemu-system-x86_64 -L pc-bios -m 512 -drive
>> file=/media/blackstar/images/rhel5-64.raw,if=ide -net nic,model=e1000
>> -monitor stdio -enable-kvm
>>
>> Dest:
>> sudo ./x86_64-softmmu/qemu-system-x86_64 -L pc-bios -m 512 -drive
>> file=/media/blackstar/images/rhel5-64.raw,if=ide -net nic,model=e1000
>> -monitor stdio -enable-kvm -incoming tcp:0:4444
>>
>>   
>> Lemme try qemu-kvm.git
> 
> Working there as well on top of 2.6.32-rc7 kmod modules and kvm-qemu.git
> tip+vmstate fixes.  Any other varient?
> 

Interesting. I think I need to try with a more recent host kernel (not
sure if you want to downgrade your kernel :) ). Maybe it's a kvm-kmod
wrapping issue.

Thanks so far,
Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

[Qemu-devel] Re: Live migration of x86-64 kvm guests broken?

[Juan Quintela]

Jan Kiszka <jan.kiszka@siemens.com> wrote:
> Hi,
>
> I'm facing stalled x86-64 guests after live migration when using kvm
> (share disk images). This does not happen with x86-32 guests or when
> disabling kvm. Both qemu and qemu-kvm git heads are affected (recent
> vmstate fixes applied). Running I/O load during the migration (e.g. a
> simple "ls -R /") seems to trigger it reliably.
>
> Can anyone confirm / comment on this? Do we fail to restore some magic
> state that only long mode cares about?

My tests were with badblocks -svw /dev/vda

(qemu.git not qemu-kvm)  I can try if you want.

Later, Juan.