qemu-kvm-0.13.0, 2.6.37.1 - after migration, some of guests got stuck

qemu-kvm-0.13.0, 2.6.37.1 - after migration, some of guests got stuck

[Nikola Ciprich]

Hello,
I've just migrated a bunch of guests from one node to another
due to upgrading host kernel from 2.6.37 to 2.6.37.1.
After that, some of guests started consuming 100% of CPU time and their os
seems to be stuck.

Looks like all the stuck guest were running 2.6.32 with kvm paravirt enabled.
others running same as well as different kernels seem to be ok, including windows
guests.
I guess I can keep stuck guest in this state for few hours, so if somebody would be willing
to have a look at this, I'll be happy to try to help.

running kvm_stat for few seconds reports:
 kvm_exit                                    237284   19119
 kvm_entry                                   237183   19119
 kvm_emulate_insn                            123548   10088
 kvm_mmio                                     92947    7622
 kvm_exit(IO_INSTRUCTION)                     90744    7395
 kvm_pio                                      90726    7395
 kvm_exit(EXCEPTION_NMI)                      69993    5416
 kvm_page_fault                               69254    5347
 kvm_apic                                     51528    4157

host is x86_64, guests were x86_64 too..
BR
nik

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

Re: qemu-kvm-0.13.0, 2.6.37.1 - after migration, some of guests got stuck

[Philipp Hahn]

[-- Attachment #1: Type: text/plain, Size: 1929 bytes --]

Hello,

I'm no KVM core developer, so please take my advise with a grain of caution.

Am Sonntag 20 Februar 2011 20:32:35 schrieb Nikola Ciprich:
> I've just migrated a bunch of guests from one node to another
> due to upgrading host kernel from 2.6.37 to 2.6.37.1.
> After that, some of guests started consuming 100% of CPU time and their os
> seems to be stuck.
>
> Looks like all the stuck guest were running 2.6.32 with kvm paravirt
> enabled. others running same as well as different kernels seem to be ok,
> including windows guests.

I also sometimes experience problems with migrated (in my case: suspended) 
VMs, see <http://marc.info/?l=kvm&m=129484437222822&w=2>: They were either 
stuck in an interrupt storm or did not receive any interrupts any more (looks 
like some interrupts relates state gets not saved/restored properly).
For me the following did help:
1. Suspend VM to disk
2. Restart VM with the -no-kvm-irqchip Option
Perhaps you could test that as well. (An alternative would be to use qemus 
internal debugger or gdb to remote debug the VM and detect what your VMs are 
doing, especially "x/20i $rip" to dump the next 20 assembler instructions)
Please be advised, that -no-kvm-irqchip opens a new can of worms and should 
only be used to diagnose the problem. You probably should also backup the 
save file and use the -snapshot option to be able to repeatly test different 
strategies.

Sincerely
Philipp
-- 
Philipp Hahn           Open Source Software Engineer      hahn@univention.de
Univention GmbH        Linux for Your Business        fon: +49 421 22 232- 0
Mary-Somerville-Str.1  28359 Bremen                   fax: +49 421 22 232-99
                                                   http://www.univention.de/
** Besuchen Sie uns auf der CeBIT in Hannover **
** Auf dem Univention Stand D36 in Halle 2    **
** Vom 01. bis 05. März 2011                  **

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]