* Freezing Windows 2008 x64bit guest @ 2010-07-15 13:19 Christoph Adomeit 2010-07-15 13:44 ` Gleb Natapov 0 siblings, 1 reply; 22+ messages in thread From: Christoph Adomeit @ 2010-07-15 13:19 UTC (permalink / raw) To: kvm Hi, we are running several kvm servers with kvm 0.12.4 and kernel 2.6.32 (actually we run proxmox ve) These proxmox servers are running all kinds of vms rock-solid, be it linux or windows 32 platforms. But one Windows 2008 64 Bit Server Standard is freezing regularly. This happens sometimes 3 times a day, sometimes it takes 2 days until freeze. The Windows Machine is a clean fresh install. By Freezing I mean that the Machine console still shows the windows logo but does not accept any keystroke and does not answer to ping or rdp or whatever. The virtual cpu-load is then shown as 100% (by proxmox). We tried several combinations of ide or virtio, several virtual network cards (intel and realtek) and single-cpu vs multi cpu. The Problem persists. The Windows Machine is a quite fresh install but runs exchange and 10 rdp sessions. We are experienced linux-admins but we dont find any hints on the kvm servers what might be the problem, no syslogs,dmesgs, nothing. Also there are no dumps on the windows guest. Do you have any hints what might be the problem and what we could do to debug the problem and to get more information of the root cause ? Thanks Christoph ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Freezing Windows 2008 x64bit guest 2010-07-15 13:19 Freezing Windows 2008 x64bit guest Christoph Adomeit @ 2010-07-15 13:44 ` Gleb Natapov 2010-07-19 7:17 ` Harri Olin 0 siblings, 1 reply; 22+ messages in thread From: Gleb Natapov @ 2010-07-15 13:44 UTC (permalink / raw) To: Christoph Adomeit; +Cc: kvm On Thu, Jul 15, 2010 at 03:19:44PM +0200, Christoph Adomeit wrote: > Hi, > > we are running several kvm servers with kvm 0.12.4 and kernel 2.6.32 > (actually we run proxmox ve) > > These proxmox servers are running all kinds of vms rock-solid, be it > linux or windows 32 platforms. > > But one Windows 2008 64 Bit Server Standard is freezing regularly. > This happens sometimes 3 times a day, sometimes it takes 2 days > until freeze. The Windows Machine is a clean fresh install. > > By Freezing I mean that the Machine console still shows the > windows logo but does not accept any keystroke and does not answer > to ping or rdp or whatever. The virtual cpu-load is then shown > as 100% (by proxmox). > > We tried several combinations of ide or virtio, several virtual network cards (intel and realtek) and single-cpu vs multi cpu. The Problem persists. > > The Windows Machine is a quite fresh install but runs exchange and 10 rdp sessions. > > We are experienced linux-admins but we dont find any hints on the kvm servers what might be the problem, no syslogs,dmesgs, nothing. Also there are no > dumps on the windows guest. > > Do you have any hints what might be the problem and what we could do to debug the problem and to get more information of the root cause ? > When hang occurs ensure that problematic vm is the only on on the server and run kvm_stat. Send output here. Also do the following: # mount -t debugfs debugfs /sys/kernel/debug # echo kvm > /sys/kernel/debug/tracing/set_event # sleep 1 # cat /sys/kernel/debug/tracing/trace > /tmp/trace Send /tmp/trace here too, but it may be huge, so send only last 1000 lines. -- Gleb. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Freezing Windows 2008 x64bit guest 2010-07-15 13:44 ` Gleb Natapov @ 2010-07-19 7:17 ` Harri Olin 2010-07-19 7:42 ` Gleb Natapov 0 siblings, 1 reply; 22+ messages in thread From: Harri Olin @ 2010-07-19 7:17 UTC (permalink / raw) To: Gleb Natapov; +Cc: Christoph Adomeit, kvm@vger.kernel.org Gleb Natapov kirjoitti: > On Thu, Jul 15, 2010 at 03:19:44PM +0200, Christoph Adomeit wrote: >> >> But one Windows 2008 64 Bit Server Standard is freezing regularly. >> This happens sometimes 3 times a day, sometimes it takes 2 days >> until freeze. The Windows Machine is a clean fresh install. >> >> By Freezing I mean that the Machine console still shows the >> windows logo but does not accept any keystroke and does not answer >> to ping or rdp or whatever. The virtual cpu-load is then shown >> as 100% (by proxmox). Hi I think I have seen same problem occur on my Windows 2008 SBS SP2 64bit system, but a bit less often, only like once a week. Short description of system: Supermicro X7SBL-LN2 motherboard, Q8300 cpu, 8GB memory, Areca ARC-1210 RAID controller. Linux kernel 2.6.33.1, qemu-kvm-0.11.1, windows 2008 quest is the only one at the moment. I had to step down to 0.11.1 from 0.12.x as that was more unstable, windows 2008 bluescreened every second day. Also 0.11.1 did the same when I applied -rtc-td-hack, I think. (might also be -cpu host, which I removed at the same time but I think the crashing started when I added -rt-td-hack). Sample bluescreen error codes from that time: 0x0000003b (0x00000000c0000005, 0xfffff80001cb8842, 0xfffffa600 0x0000003b (0x00000000c0000005, 0xfffff80001c9c842, 0xfffffa600 0x0000000a (0x0000000000000080, 0x000000000000000c, 0x000000000 0x0000003b (0x00000000c0000005, 0xfffff80001c5d842, 0xfffffa600 Also I think I have seen qemu crashing once.. Now I haven't seen crashes but only freezes with qemu on 100% and virtual system unresponsive. qemu command line: /usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net nic,macaddr=C0:FF:12:FB:AA:01,model=e1000 -net tap -smp 4 -localtime > When hang occurs ensure that problematic vm is the only on on the server and run > kvm_stat. Send output here. Also do the following: > > # mount -t debugfs debugfs /sys/kernel/debug > # echo kvm > /sys/kernel/debug/tracing/set_event > # sleep 1 > # cat /sys/kernel/debug/tracing/trace > /tmp/trace > > Send /tmp/trace here too, but it may be huge, so send only last 1000 > lines. kvm_stat logging output during hang. logging output at http://mizar.remote.agasha.com/k/kvm/kvm_stat_hang.txt efer_reload 0 0 exits 17193383 14804 fpu_reload 2346791 2089 halt_exits 213095 0 halt_wakeup 191350 0 host_state_reload 4692238 2089 hypercalls 0 0 insn_emulation 6726836 4012 insn_emulation_fail 0 0 invlpg 217889 0 io_exits 1755022 0 irq_exits 145317 9409 irq_injections 296911 4006 irq_window 28551 1381 largepages 0 0 mmio_exits 505162 0 mmu_cache_miss 56734 0 mmu_flooded 12110 0 mmu_pde_zapped 22167 0 mmu_pte_updated 99276 0 mmu_pte_write 63788 0 mmu_recycled 0 0 And here's the trace: also readable at http://mizar.remote.agasha.com/k/kvm/kvm_trace_hang.txt qemu-system-x86-4586 [003] 1076789.993832: kvm_exit: reason ext_irq rip 0xfffff80001c7e891 qemu-system-x86-4586 [003] 1076789.993832: kvm_entry: vcpu 0 qemu-system-x86-4580 [002] 1076789.994369: kvm_set_irq: gsi 0 level 1 source 0 qemu-system-x86-4580 [002] 1076789.994370: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4580 [002] 1076789.994370: kvm_apic_accept_irq: apicid 0 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076789.994371: kvm_apic_accept_irq: apicid 1 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076789.994371: kvm_apic_accept_irq: apicid 2 vec 209 (Fixed|edge) qemu-system-x86-4586 [003] 1076789.994371: kvm_exit: reason ext_irq rip 0xfffff80001c7e891 qemu-system-x86-4580 [002] 1076789.994372: kvm_apic_accept_irq: apicid 3 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076789.994372: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4586 [003] 1076789.994372: kvm_inj_virq: irq 209 qemu-system-x86-4586 [003] 1076789.994372: kvm_entry: vcpu 0 qemu-system-x86-4588 [000] 1076789.994372: kvm_exit: reason ext_irq rip 0xfffff80001ca12ee qemu-system-x86-4580 [002] 1076789.994373: kvm_set_irq: gsi 0 level 0 source 0 qemu-system-x86-4580 [002] 1076789.994373: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4580 [002] 1076789.994373: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4588 [000] 1076789.994373: kvm_inj_virq: irq 209 qemu-system-x86-4589 [001] 1076789.994373: kvm_exit: reason ext_irq rip 0xfffff80001c9b859 qemu-system-x86-4588 [000] 1076789.994373: kvm_entry: vcpu 2 qemu-system-x86-4586 [003] 1076789.994374: kvm_exit: reason apic_access rip 0xfffff80001c9a5c4 qemu-system-x86-4589 [001] 1076789.994374: kvm_inj_virq: irq 209 qemu-system-x86-4589 [001] 1076789.994374: kvm_entry: vcpu 3 qemu-system-x86-4588 [000] 1076789.994375: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4586 [003] 1076789.994375: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4586 [003] 1076789.994376: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4589 [001] 1076789.994376: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4586 [003] 1076789.994376: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4586 [003] 1076789.994377: kvm_entry: vcpu 0 qemu-system-x86-4588 [000] 1076789.994377: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4588 [000] 1076789.994377: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4589 [001] 1076789.994377: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4589 [001] 1076789.994378: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4588 [000] 1076789.994378: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4589 [001] 1076789.994378: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4588 [000] 1076789.994379: kvm_entry: vcpu 2 qemu-system-x86-4589 [001] 1076789.994379: kvm_entry: vcpu 3 qemu-system-x86-4587 [002] 1076789.994380: kvm_exit: reason ext_irq rip 0xfffff80001c9b859 qemu-system-x86-4587 [002] 1076789.994381: kvm_inj_virq: irq 209 qemu-system-x86-4587 [002] 1076789.994382: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076789.994384: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4587 [002] 1076789.994386: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4587 [002] 1076789.994386: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4587 [002] 1076789.994386: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4587 [002] 1076789.994387: kvm_entry: vcpu 1 qemu-system-x86-4588 [000] 1076789.994457: kvm_exit: reason ext_irq rip 0xfffff80001c9ffb6 qemu-system-x86-4588 [000] 1076789.994457: kvm_entry: vcpu 2 qemu-system-x86-4589 [001] 1076789.994582: kvm_exit: reason ext_irq rip 0xfffff80001c9b859 qemu-system-x86-4588 [000] 1076789.994582: kvm_exit: reason ext_irq rip 0xfffff80001cd9121 qemu-system-x86-4589 [001] 1076789.994582: kvm_entry: vcpu 3 qemu-system-x86-4588 [000] 1076789.994583: kvm_entry: vcpu 2 qemu-system-x86-4587 [002] 1076789.994634: kvm_exit: reason ext_irq rip 0xfffff80001ca00d5 qemu-system-x86-4587 [002] 1076789.994635: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076789.994706: kvm_exit: reason ext_irq rip 0xfffff80001cda795 qemu-system-x86-4587 [002] 1076789.994706: kvm_entry: vcpu 1 qemu-system-x86-4586 [003] 1076789.994830: kvm_exit: reason ext_irq rip 0xfffff80001c7e891 qemu-system-x86-4586 [003] 1076789.994831: kvm_entry: vcpu 0 qemu-system-x86-4580 [002] 1076789.995367: kvm_set_irq: gsi 0 level 1 source 0 qemu-system-x86-4580 [002] 1076789.995367: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4580 [002] 1076789.995368: kvm_apic_accept_irq: apicid 0 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076789.995368: kvm_apic_accept_irq: apicid 1 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076789.995369: kvm_apic_accept_irq: apicid 2 vec 209 (Fixed|edge) qemu-system-x86-4586 [003] 1076789.995369: kvm_exit: reason ext_irq rip 0xfffff80001c7e891 qemu-system-x86-4580 [002] 1076789.995369: kvm_apic_accept_irq: apicid 3 vec 209 (Fixed|edge) qemu-system-x86-4586 [003] 1076789.995370: kvm_inj_virq: irq 209 qemu-system-x86-4580 [002] 1076789.995370: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4586 [003] 1076789.995370: kvm_entry: vcpu 0 qemu-system-x86-4580 [002] 1076789.995370: kvm_set_irq: gsi 0 level 0 source 0 qemu-system-x86-4588 [000] 1076789.995370: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4580 [002] 1076789.995370: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4580 [002] 1076789.995371: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4589 [001] 1076789.995371: kvm_exit: reason ext_irq rip 0xfffff80001cd8bac qemu-system-x86-4588 [000] 1076789.995371: kvm_entry: vcpu 2 qemu-system-x86-4589 [001] 1076789.995372: kvm_inj_virq: irq 209 qemu-system-x86-4586 [003] 1076789.995372: kvm_exit: reason apic_access rip 0xfffff80001c9a5c4 qemu-system-x86-4589 [001] 1076789.995372: kvm_entry: vcpu 3 qemu-system-x86-4588 [000] 1076789.995372: kvm_exit: reason interrupt_window rip 0xfffff80001ca0002 qemu-system-x86-4588 [000] 1076789.995373: kvm_inj_virq: irq 209 qemu-system-x86-4586 [003] 1076789.995373: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4588 [000] 1076789.995373: kvm_entry: vcpu 2 qemu-system-x86-4586 [003] 1076789.995373: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4589 [001] 1076789.995373: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4586 [003] 1076789.995374: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4586 [003] 1076789.995375: kvm_entry: vcpu 0 qemu-system-x86-4588 [000] 1076789.995375: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4589 [001] 1076789.995375: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4589 [001] 1076789.995375: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4589 [001] 1076789.995376: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4588 [000] 1076789.995376: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4588 [000] 1076789.995377: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4589 [001] 1076789.995377: kvm_entry: vcpu 3 qemu-system-x86-4588 [000] 1076789.995377: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4588 [000] 1076789.995378: kvm_entry: vcpu 2 qemu-system-x86-4587 [002] 1076789.995378: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4587 [002] 1076789.995379: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076789.995381: kvm_exit: reason interrupt_window rip 0xfffff80001ca0002 qemu-system-x86-4587 [002] 1076789.995381: kvm_inj_virq: irq 209 qemu-system-x86-4587 [002] 1076789.995381: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076789.995383: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4587 [002] 1076789.995385: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4587 [002] 1076789.995385: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4587 [002] 1076789.995386: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4587 [002] 1076789.995387: kvm_entry: vcpu 1 qemu-system-x86-4588 [000] 1076789.995455: kvm_exit: reason ext_irq rip 0xfffff80001cde8ec qemu-system-x86-4588 [000] 1076789.995456: kvm_entry: vcpu 2 qemu-system-x86-4589 [001] 1076789.995580: kvm_exit: reason ext_irq rip 0xfffff80001d019f1 qemu-system-x86-4589 [001] 1076789.995580: kvm_entry: vcpu 3 qemu-system-x86-4587 [002] 1076789.995632: kvm_exit: reason ext_irq rip 0xfffff80001ca138b qemu-system-x86-4587 [002] 1076789.995633: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076789.995704: kvm_exit: reason ext_irq rip 0xfffff80001d01a00 qemu-system-x86-4587 [002] 1076789.995705: kvm_entry: vcpu 1 qemu-system-x86-4586 [003] 1076789.995829: kvm_exit: reason ext_irq rip 0xfffff80001c7e891 qemu-system-x86-4586 [003] 1076789.995829: kvm_entry: vcpu 0 qemu-system-x86-4580 [002] 1076789.996365: kvm_set_irq: gsi 0 level 1 source 0 qemu-system-x86-4580 [002] 1076789.996366: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4580 [002] 1076789.996367: kvm_apic_accept_irq: apicid 0 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076789.996367: kvm_apic_accept_irq: apicid 1 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076789.996367: kvm_apic_accept_irq: apicid 2 vec 209 (Fixed|edge) qemu-system-x86-4586 [003] 1076789.996368: kvm_exit: reason ext_irq rip 0xfffff80001c7e88f qemu-system-x86-4580 [002] 1076789.996368: kvm_apic_accept_irq: apicid 3 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076789.996368: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4586 [003] 1076789.996368: kvm_inj_virq: irq 209 qemu-system-x86-4586 [003] 1076789.996369: kvm_entry: vcpu 0 qemu-system-x86-4588 [000] 1076789.996369: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4580 [002] 1076789.996369: kvm_set_irq: gsi 0 level 0 source 0 qemu-system-x86-4580 [002] 1076789.996369: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4580 [002] 1076789.996369: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4589 [001] 1076789.996370: kvm_exit: reason ext_irq rip 0xfffff80001c9b859 qemu-system-x86-4588 [000] 1076789.996370: kvm_entry: vcpu 2 qemu-system-x86-4589 [001] 1076789.996370: kvm_inj_virq: irq 209 qemu-system-x86-4586 [003] 1076789.996370: kvm_exit: reason apic_access rip 0xfffff80001c9a5c4 qemu-system-x86-4589 [001] 1076789.996371: kvm_entry: vcpu 3 qemu-system-x86-4588 [000] 1076789.996371: kvm_exit: reason interrupt_window rip 0xfffff80001ca0002 qemu-system-x86-4588 [000] 1076789.996371: kvm_inj_virq: irq 209 qemu-system-x86-4588 [000] 1076789.996372: kvm_entry: vcpu 2 qemu-system-x86-4586 [003] 1076789.996372: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4586 [003] 1076789.996372: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4589 [001] 1076789.996372: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4586 [003] 1076789.996373: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4588 [000] 1076789.996373: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4586 [003] 1076789.996373: kvm_entry: vcpu 0 qemu-system-x86-4589 [001] 1076789.996374: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4589 [001] 1076789.996374: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4589 [001] 1076789.996374: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4588 [000] 1076789.996375: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4588 [000] 1076789.996375: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4588 [000] 1076789.996375: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4589 [001] 1076789.996376: kvm_entry: vcpu 3 qemu-system-x86-4588 [000] 1076789.996376: kvm_entry: vcpu 2 qemu-system-x86-4587 [002] 1076789.996377: kvm_exit: reason ext_irq rip 0xfffff80001ca010b qemu-system-x86-4587 [002] 1076789.996378: kvm_inj_virq: irq 209 qemu-system-x86-4587 [002] 1076789.996378: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076789.996380: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4587 [002] 1076789.996382: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4587 [002] 1076789.996382: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4587 [002] 1076789.996382: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4587 [002] 1076789.996384: kvm_entry: vcpu 1 qemu-system-x86-4588 [000] 1076789.996453: kvm_exit: reason ext_irq rip 0xfffff80001cd916f qemu-system-x86-4588 [000] 1076789.996454: kvm_entry: vcpu 2 qemu-system-x86-4589 [001] 1076789.996578: kvm_exit: reason ext_irq rip 0xfffff80001cd8c58 qemu-system-x86-4589 [001] 1076789.996579: kvm_entry: vcpu 3 qemu-system-x86-4587 [002] 1076789.996630: kvm_exit: reason ext_irq rip 0xfffff80001d01a00 qemu-system-x86-4587 [002] 1076789.996632: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076789.996703: kvm_exit: reason ext_irq rip 0xfffff80001c9b859 qemu-system-x86-4587 [002] 1076789.996703: kvm_entry: vcpu 1 qemu-system-x86-4586 [003] 1076789.996827: kvm_exit: reason ext_irq rip 0xfffff80001c7e891 qemu-system-x86-4586 [003] 1076789.996828: kvm_entry: vcpu 0 qemu-system-x86-4580 [002] 1076789.997364: kvm_set_irq: gsi 0 level 1 source 0 qemu-system-x86-4580 [002] 1076789.997365: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4580 [002] 1076789.997365: kvm_apic_accept_irq: apicid 0 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076789.997366: kvm_apic_accept_irq: apicid 1 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076789.997366: kvm_apic_accept_irq: apicid 2 vec 209 (Fixed|edge) qemu-system-x86-4586 [003] 1076789.997367: kvm_exit: reason ext_irq rip 0xfffff80001c7e894 qemu-system-x86-4580 [002] 1076789.997367: kvm_apic_accept_irq: apicid 3 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076789.997367: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4586 [003] 1076789.997367: kvm_inj_virq: irq 209 qemu-system-x86-4586 [003] 1076789.997367: kvm_entry: vcpu 0 qemu-system-x86-4588 [000] 1076789.997368: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4580 [002] 1076789.997368: kvm_set_irq: gsi 0 level 0 source 0 qemu-system-x86-4580 [002] 1076789.997368: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4580 [002] 1076789.997368: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4588 [000] 1076789.997368: kvm_entry: vcpu 2 qemu-system-x86-4589 [001] 1076789.997368: kvm_exit: reason ext_irq rip 0xfffff80001c9b85d qemu-system-x86-4586 [003] 1076789.997369: kvm_exit: reason apic_access rip 0xfffff80001c9a5c4 qemu-system-x86-4589 [001] 1076789.997369: kvm_inj_virq: irq 209 qemu-system-x86-4589 [001] 1076789.997369: kvm_entry: vcpu 3 qemu-system-x86-4588 [000] 1076789.997369: kvm_exit: reason interrupt_window rip 0xfffff80001ca0002 qemu-system-x86-4588 [000] 1076789.997370: kvm_inj_virq: irq 209 qemu-system-x86-4588 [000] 1076789.997371: kvm_entry: vcpu 2 qemu-system-x86-4586 [003] 1076789.997371: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4589 [001] 1076789.997371: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4586 [003] 1076789.997371: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4586 [003] 1076789.997371: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4588 [000] 1076789.997372: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4586 [003] 1076789.997372: kvm_entry: vcpu 0 qemu-system-x86-4589 [001] 1076789.997373: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4589 [001] 1076789.997373: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4589 [001] 1076789.997373: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4588 [000] 1076789.997374: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4588 [000] 1076789.997374: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4588 [000] 1076789.997374: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4589 [001] 1076789.997374: kvm_entry: vcpu 3 qemu-system-x86-4588 [000] 1076789.997375: kvm_entry: vcpu 2 qemu-system-x86-4587 [002] 1076789.997376: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4587 [002] 1076789.997377: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076789.997378: kvm_exit: reason interrupt_window rip 0xfffff80001ca0002 qemu-system-x86-4587 [002] 1076789.997379: kvm_inj_virq: irq 209 qemu-system-x86-4587 [002] 1076789.997379: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076789.997380: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4587 [002] 1076789.997382: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4587 [002] 1076789.997383: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4587 [002] 1076789.997383: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4587 [002] 1076789.997384: kvm_entry: vcpu 1 qemu-system-x86-4588 [000] 1076789.997452: kvm_exit: reason ext_irq rip 0xfffff80001c9b859 qemu-system-x86-4588 [000] 1076789.997452: kvm_entry: vcpu 2 qemu-system-x86-4589 [001] 1076789.997576: kvm_exit: reason ext_irq rip 0xfffff80001d01a00 qemu-system-x86-4589 [001] 1076789.997577: kvm_entry: vcpu 3 qemu-system-x86-4587 [002] 1076789.997629: kvm_exit: reason ext_irq rip 0xfffff80001d01a00 qemu-system-x86-4587 [002] 1076789.997630: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076789.997701: kvm_exit: reason ext_irq rip 0xfffff80001cd8c58 qemu-system-x86-4587 [002] 1076789.997702: kvm_entry: vcpu 1 qemu-system-x86-4586 [003] 1076789.997826: kvm_exit: reason ext_irq rip 0xfffff80001c7e894 qemu-system-x86-4586 [003] 1076789.997826: kvm_entry: vcpu 0 qemu-system-x86-4580 [002] 1076789.998363: kvm_set_irq: gsi 0 level 1 source 0 qemu-system-x86-4580 [002] 1076789.998364: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4580 [002] 1076789.998364: kvm_apic_accept_irq: apicid 0 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076789.998365: kvm_apic_accept_irq: apicid 1 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076789.998365: kvm_apic_accept_irq: apicid 2 vec 209 (Fixed|edge) qemu-system-x86-4586 [003] 1076789.998365: kvm_exit: reason ext_irq rip 0xfffff80001c7e894 qemu-system-x86-4580 [002] 1076789.998366: kvm_apic_accept_irq: apicid 3 vec 209 (Fixed|edge) qemu-system-x86-4586 [003] 1076789.998366: kvm_inj_virq: irq 209 qemu-system-x86-4580 [002] 1076789.998366: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4586 [003] 1076789.998366: kvm_entry: vcpu 0 qemu-system-x86-4588 [000] 1076789.998367: kvm_exit: reason ext_irq rip 0xfffff80001cd90c2 qemu-system-x86-4580 [002] 1076789.998367: kvm_set_irq: gsi 0 level 0 source 0 qemu-system-x86-4580 [002] 1076789.998367: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4580 [002] 1076789.998367: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4589 [001] 1076789.998367: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4588 [000] 1076789.998367: kvm_inj_virq: irq 209 qemu-system-x86-4588 [000] 1076789.998368: kvm_entry: vcpu 2 qemu-system-x86-4586 [003] 1076789.998368: kvm_exit: reason apic_access rip 0xfffff80001c9a5c4 qemu-system-x86-4589 [001] 1076789.998368: kvm_entry: vcpu 3 qemu-system-x86-4588 [000] 1076789.998369: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4589 [001] 1076789.998369: kvm_exit: reason interrupt_window rip 0xfffff80001ca0002 qemu-system-x86-4586 [003] 1076789.998370: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4586 [003] 1076789.998370: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4589 [001] 1076789.998370: kvm_inj_virq: irq 209 qemu-system-x86-4586 [003] 1076789.998370: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4589 [001] 1076789.998370: kvm_entry: vcpu 3 qemu-system-x86-4588 [000] 1076789.998371: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4588 [000] 1076789.998371: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4586 [003] 1076789.998371: kvm_entry: vcpu 0 qemu-system-x86-4588 [000] 1076789.998371: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4589 [001] 1076789.998372: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4588 [000] 1076789.998372: kvm_entry: vcpu 2 qemu-system-x86-4589 [001] 1076789.998373: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4589 [001] 1076789.998374: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4589 [001] 1076789.998374: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4587 [002] 1076789.998374: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4589 [001] 1076789.998375: kvm_entry: vcpu 3 qemu-system-x86-4587 [002] 1076789.998376: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076789.998377: kvm_exit: reason interrupt_window rip 0xfffff80001ca0002 qemu-system-x86-4587 [002] 1076789.998378: kvm_inj_virq: irq 209 qemu-system-x86-4587 [002] 1076789.998378: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076789.998379: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4587 [002] 1076789.998381: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4587 [002] 1076789.998382: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4587 [002] 1076789.998382: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4587 [002] 1076789.998383: kvm_entry: vcpu 1 qemu-system-x86-4588 [000] 1076789.998450: kvm_exit: reason ext_irq rip 0xfffff80001ca136a qemu-system-x86-4588 [000] 1076789.998451: kvm_entry: vcpu 2 qemu-system-x86-4589 [001] 1076789.998575: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4589 [001] 1076789.998576: kvm_entry: vcpu 3 qemu-system-x86-4588 [000] 1076789.998576: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4588 [000] 1076789.998577: kvm_entry: vcpu 2 qemu-system-x86-4587 [002] 1076789.998628: kvm_exit: reason ext_irq rip 0xfffff80001d01a00 qemu-system-x86-4587 [002] 1076789.998629: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076789.998700: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4587 [002] 1076789.998700: kvm_entry: vcpu 1 qemu-system-x86-4586 [003] 1076789.998824: kvm_exit: reason ext_irq rip 0xfffff80001c7e891 qemu-system-x86-4586 [003] 1076789.998824: kvm_entry: vcpu 0 qemu-system-x86-4580 [002] 1076789.999361: kvm_set_irq: gsi 0 level 1 source 0 qemu-system-x86-4580 [002] 1076789.999362: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4580 [002] 1076789.999362: kvm_apic_accept_irq: apicid 0 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076789.999362: kvm_apic_accept_irq: apicid 1 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076789.999363: kvm_apic_accept_irq: apicid 2 vec 209 (Fixed|edge) qemu-system-x86-4586 [003] 1076789.999363: kvm_exit: reason ext_irq rip 0xfffff80001c7e894 qemu-system-x86-4580 [002] 1076789.999363: kvm_apic_accept_irq: apicid 3 vec 209 (Fixed|edge) qemu-system-x86-4586 [003] 1076789.999364: kvm_inj_virq: irq 209 qemu-system-x86-4580 [002] 1076789.999364: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4586 [003] 1076789.999364: kvm_entry: vcpu 0 qemu-system-x86-4588 [000] 1076789.999364: kvm_exit: reason ext_irq rip 0xfffff80001ca1311 qemu-system-x86-4580 [002] 1076789.999364: kvm_set_irq: gsi 0 level 0 source 0 qemu-system-x86-4580 [002] 1076789.999365: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4580 [002] 1076789.999365: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4588 [000] 1076789.999365: kvm_inj_virq: irq 209 qemu-system-x86-4589 [001] 1076789.999365: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4588 [000] 1076789.999366: kvm_entry: vcpu 2 qemu-system-x86-4586 [003] 1076789.999366: kvm_exit: reason apic_access rip 0xfffff80001c9a5c4 qemu-system-x86-4589 [001] 1076789.999366: kvm_entry: vcpu 3 qemu-system-x86-4588 [000] 1076789.999367: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4589 [001] 1076789.999367: kvm_exit: reason interrupt_window rip 0xfffff80001ca0002 qemu-system-x86-4586 [003] 1076789.999368: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4586 [003] 1076789.999368: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4589 [001] 1076789.999368: kvm_inj_virq: irq 209 qemu-system-x86-4586 [003] 1076789.999368: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4589 [001] 1076789.999368: kvm_entry: vcpu 3 qemu-system-x86-4588 [000] 1076789.999369: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4588 [000] 1076789.999369: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4586 [003] 1076789.999369: kvm_entry: vcpu 0 qemu-system-x86-4588 [000] 1076789.999369: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4589 [001] 1076789.999370: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4588 [000] 1076789.999370: kvm_entry: vcpu 2 qemu-system-x86-4589 [001] 1076789.999371: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4589 [001] 1076789.999372: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4589 [001] 1076789.999372: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4587 [002] 1076789.999372: kvm_exit: reason ext_irq rip 0xfffff80001c9b859 qemu-system-x86-4589 [001] 1076789.999373: kvm_entry: vcpu 3 qemu-system-x86-4587 [002] 1076789.999374: kvm_inj_virq: irq 209 qemu-system-x86-4587 [002] 1076789.999374: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076789.999376: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4587 [002] 1076789.999378: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4587 [002] 1076789.999378: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4587 [002] 1076789.999378: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4587 [002] 1076789.999379: kvm_entry: vcpu 1 qemu-system-x86-4588 [000] 1076789.999449: kvm_exit: reason ext_irq rip 0xfffff80001ca12d8 qemu-system-x86-4588 [000] 1076789.999449: kvm_entry: vcpu 2 qemu-system-x86-4589 [001] 1076789.999573: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4589 [001] 1076789.999574: kvm_entry: vcpu 3 qemu-system-x86-4587 [002] 1076789.999626: kvm_exit: reason ext_irq rip 0xfffff80001c9b859 qemu-system-x86-4587 [002] 1076789.999627: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076789.999698: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4587 [002] 1076789.999699: kvm_entry: vcpu 1 qemu-system-x86-4586 [003] 1076789.999822: kvm_exit: reason ext_irq rip 0xfffff80001c7e894 qemu-system-x86-4586 [003] 1076789.999823: kvm_entry: vcpu 0 qemu-system-x86-4580 [002] 1076790.000360: kvm_set_irq: gsi 0 level 1 source 0 qemu-system-x86-4580 [002] 1076790.000360: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4580 [002] 1076790.000361: kvm_apic_accept_irq: apicid 0 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076790.000361: kvm_apic_accept_irq: apicid 1 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076790.000362: kvm_apic_accept_irq: apicid 2 vec 209 (Fixed|edge) qemu-system-x86-4586 [003] 1076790.000362: kvm_exit: reason ext_irq rip 0xfffff80001c7e891 qemu-system-x86-4580 [002] 1076790.000362: kvm_apic_accept_irq: apicid 3 vec 209 (Fixed|edge) qemu-system-x86-4586 [003] 1076790.000363: kvm_inj_virq: irq 209 qemu-system-x86-4586 [003] 1076790.000363: kvm_entry: vcpu 0 qemu-system-x86-4580 [002] 1076790.000363: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4588 [000] 1076790.000363: kvm_exit: reason ext_irq rip 0xfffff80001cd8fa3 qemu-system-x86-4580 [002] 1076790.000364: kvm_set_irq: gsi 0 level 0 source 0 qemu-system-x86-4588 [000] 1076790.000364: kvm_inj_virq: irq 209 qemu-system-x86-4580 [002] 1076790.000364: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4589 [001] 1076790.000364: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4580 [002] 1076790.000364: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4588 [000] 1076790.000364: kvm_entry: vcpu 2 qemu-system-x86-4586 [003] 1076790.000365: kvm_exit: reason apic_access rip 0xfffff80001c9a5c4 qemu-system-x86-4589 [001] 1076790.000365: kvm_entry: vcpu 3 qemu-system-x86-4588 [000] 1076790.000366: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4589 [001] 1076790.000366: kvm_exit: reason interrupt_window rip 0xfffff80001ca0002 qemu-system-x86-4586 [003] 1076790.000366: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4586 [003] 1076790.000366: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4586 [003] 1076790.000367: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4589 [001] 1076790.000367: kvm_inj_virq: irq 209 qemu-system-x86-4589 [001] 1076790.000367: kvm_entry: vcpu 3 qemu-system-x86-4588 [000] 1076790.000367: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4588 [000] 1076790.000367: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4586 [003] 1076790.000368: kvm_entry: vcpu 0 qemu-system-x86-4588 [000] 1076790.000368: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4589 [001] 1076790.000369: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4588 [000] 1076790.000369: kvm_entry: vcpu 2 qemu-system-x86-4589 [001] 1076790.000370: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4589 [001] 1076790.000370: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4589 [001] 1076790.000371: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4587 [002] 1076790.000371: kvm_exit: reason ext_irq rip 0xfffff80001c9ff41 qemu-system-x86-4589 [001] 1076790.000372: kvm_entry: vcpu 3 qemu-system-x86-4587 [002] 1076790.000373: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076790.000374: kvm_exit: reason interrupt_window rip 0xfffff80001ca0002 qemu-system-x86-4587 [002] 1076790.000375: kvm_inj_virq: irq 209 qemu-system-x86-4587 [002] 1076790.000375: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076790.000377: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4587 [002] 1076790.000379: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4587 [002] 1076790.000379: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4587 [002] 1076790.000379: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4587 [002] 1076790.000380: kvm_entry: vcpu 1 qemu-system-x86-4588 [000] 1076790.000447: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4588 [000] 1076790.000448: kvm_entry: vcpu 2 qemu-system-x86-4589 [001] 1076790.000572: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4589 [001] 1076790.000572: kvm_entry: vcpu 3 qemu-system-x86-4587 [002] 1076790.000625: kvm_exit: reason ext_irq rip 0xfffff80001ca1666 qemu-system-x86-4587 [002] 1076790.000626: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076790.000697: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4587 [002] 1076790.000697: kvm_entry: vcpu 1 qemu-system-x86-4586 [003] 1076790.000821: kvm_exit: reason ext_irq rip 0xfffff80001c7e894 qemu-system-x86-4586 [003] 1076790.000821: kvm_entry: vcpu 0 qemu-system-x86-4580 [002] 1076790.001357: kvm_set_irq: gsi 0 level 1 source 0 qemu-system-x86-4580 [002] 1076790.001358: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4580 [002] 1076790.001359: kvm_apic_accept_irq: apicid 0 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076790.001359: kvm_apic_accept_irq: apicid 1 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076790.001359: kvm_apic_accept_irq: apicid 2 vec 209 (Fixed|edge) qemu-system-x86-4586 [003] 1076790.001360: kvm_exit: reason ext_irq rip 0xfffff80001c7e891 qemu-system-x86-4580 [002] 1076790.001360: kvm_apic_accept_irq: apicid 3 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076790.001360: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4586 [003] 1076790.001360: kvm_inj_virq: irq 209 qemu-system-x86-4586 [003] 1076790.001361: kvm_entry: vcpu 0 qemu-system-x86-4580 [002] 1076790.001361: kvm_set_irq: gsi 0 level 0 source 0 qemu-system-x86-4588 [000] 1076790.001361: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4580 [002] 1076790.001361: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4580 [002] 1076790.001361: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4589 [001] 1076790.001362: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4588 [000] 1076790.001362: kvm_entry: vcpu 2 qemu-system-x86-4586 [003] 1076790.001362: kvm_exit: reason apic_access rip 0xfffff80001c9a5c4 qemu-system-x86-4589 [001] 1076790.001362: kvm_entry: vcpu 3 qemu-system-x86-4588 [000] 1076790.001363: kvm_exit: reason interrupt_window rip 0xfffff80001ca0002 qemu-system-x86-4588 [000] 1076790.001364: kvm_inj_virq: irq 209 qemu-system-x86-4589 [001] 1076790.001364: kvm_exit: reason interrupt_window rip 0xfffff80001ca0002 qemu-system-x86-4586 [003] 1076790.001364: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4588 [000] 1076790.001364: kvm_entry: vcpu 2 qemu-system-x86-4586 [003] 1076790.001364: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4589 [001] 1076790.001364: kvm_inj_virq: irq 209 qemu-system-x86-4586 [003] 1076790.001364: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4589 [001] 1076790.001364: kvm_entry: vcpu 3 qemu-system-x86-4586 [003] 1076790.001365: kvm_entry: vcpu 0 qemu-system-x86-4588 [000] 1076790.001366: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4589 [001] 1076790.001366: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4588 [000] 1076790.001367: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4588 [000] 1076790.001367: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4589 [001] 1076790.001367: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4589 [001] 1076790.001368: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4588 [000] 1076790.001368: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4589 [001] 1076790.001368: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4587 [002] 1076790.001369: kvm_exit: reason ext_irq rip 0xfffff80001ca0010 qemu-system-x86-4588 [000] 1076790.001369: kvm_entry: vcpu 2 qemu-system-x86-4589 [001] 1076790.001369: kvm_entry: vcpu 3 qemu-system-x86-4587 [002] 1076790.001370: kvm_inj_virq: irq 209 qemu-system-x86-4587 [002] 1076790.001370: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076790.001372: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4587 [002] 1076790.001374: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4587 [002] 1076790.001374: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4587 [002] 1076790.001375: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4587 [002] 1076790.001376: kvm_entry: vcpu 1 qemu-system-x86-4588 [000] 1076790.001446: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4588 [000] 1076790.001447: kvm_entry: vcpu 2 qemu-system-x86-4587 [002] 1076790.001515: kvm_exit: reason ext_irq rip 0xfffff80001ca1680 qemu-system-x86-4587 [002] 1076790.001516: kvm_entry: vcpu 1 qemu-system-x86-4589 [001] 1076790.001571: kvm_exit: reason ext_irq rip 0xfffff80001d01a00 qemu-system-x86-4589 [001] 1076790.001571: kvm_entry: vcpu 3 qemu-system-x86-4587 [002] 1076790.001623: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4587 [002] 1076790.001625: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076790.001702: kvm_exit: reason ext_irq rip 0xfffff80001c9ffb6 qemu-system-x86-4587 [002] 1076790.001703: kvm_entry: vcpu 1 qemu-system-x86-4586 [003] 1076790.001819: kvm_exit: reason ext_irq rip 0xfffff80001c7e891 qemu-system-x86-4586 [003] 1076790.001820: kvm_entry: vcpu 0 qemu-system-x86-4588 [000] 1076790.001962: kvm_exit: reason ext_irq rip 0xfffff80001ca12e4 qemu-system-x86-4588 [000] 1076790.001964: kvm_entry: vcpu 2 qemu-system-x86-4580 [002] 1076790.002358: kvm_set_irq: gsi 0 level 1 source 0 qemu-system-x86-4580 [002] 1076790.002359: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4580 [002] 1076790.002360: kvm_apic_accept_irq: apicid 0 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076790.002360: kvm_apic_accept_irq: apicid 1 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076790.002360: kvm_apic_accept_irq: apicid 2 vec 209 (Fixed|edge) qemu-system-x86-4586 [003] 1076790.002361: kvm_exit: reason ext_irq rip 0xfffff80001c7e88f qemu-system-x86-4580 [002] 1076790.002361: kvm_apic_accept_irq: apicid 3 vec 209 (Fixed|edge) qemu-system-x86-4586 [003] 1076790.002361: kvm_inj_virq: irq 209 qemu-system-x86-4580 [002] 1076790.002361: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4586 [003] 1076790.002362: kvm_entry: vcpu 0 qemu-system-x86-4588 [000] 1076790.002362: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4580 [002] 1076790.002362: kvm_set_irq: gsi 0 level 0 source 0 qemu-system-x86-4580 [002] 1076790.002362: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4580 [002] 1076790.002363: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4588 [000] 1076790.002363: kvm_entry: vcpu 2 qemu-system-x86-4589 [001] 1076790.002363: kvm_exit: reason ext_irq rip 0xfffff80001cb7aa5 qemu-system-x86-4586 [003] 1076790.002363: kvm_exit: reason apic_access rip 0xfffff80001c9a5c4 qemu-system-x86-4589 [001] 1076790.002363: kvm_inj_virq: irq 209 qemu-system-x86-4589 [001] 1076790.002364: kvm_entry: vcpu 3 qemu-system-x86-4588 [000] 1076790.002364: kvm_exit: reason interrupt_window rip 0xfffff80001ca0002 qemu-system-x86-4588 [000] 1076790.002365: kvm_inj_virq: irq 209 qemu-system-x86-4586 [003] 1076790.002365: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4588 [000] 1076790.002365: kvm_entry: vcpu 2 qemu-system-x86-4586 [003] 1076790.002365: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4589 [001] 1076790.002365: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4586 [003] 1076790.002365: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4588 [000] 1076790.002366: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4586 [003] 1076790.002366: kvm_entry: vcpu 0 qemu-system-x86-4589 [001] 1076790.002367: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4589 [001] 1076790.002367: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4589 [001] 1076790.002368: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4589 [001] 1076790.002369: kvm_entry: vcpu 3 qemu-system-x86-4588 [000] 1076790.002369: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4588 [000] 1076790.002369: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4588 [000] 1076790.002370: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4587 [002] 1076790.002370: kvm_exit: reason ext_irq rip 0xfffff80001d019f1 qemu-system-x86-4588 [000] 1076790.002370: kvm_entry: vcpu 2 qemu-system-x86-4587 [002] 1076790.002371: kvm_inj_virq: irq 209 qemu-system-x86-4587 [002] 1076790.002372: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076790.002374: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4587 [002] 1076790.002376: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4587 [002] 1076790.002376: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4587 [002] 1076790.002376: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4587 [002] 1076790.002377: kvm_entry: vcpu 1 qemu-system-x86-4588 [000] 1076790.002447: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4588 [000] 1076790.002447: kvm_entry: vcpu 2 qemu-system-x86-4589 [001] 1076790.002569: kvm_exit: reason ext_irq rip 0xfffff80001cd8aec qemu-system-x86-4589 [001] 1076790.002570: kvm_entry: vcpu 3 qemu-system-x86-4587 [002] 1076790.002623: kvm_exit: reason ext_irq rip 0xfffff80001ca1666 qemu-system-x86-4587 [002] 1076790.002625: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076790.002694: kvm_exit: reason ext_irq rip 0xfffff80001c9b859 qemu-system-x86-4587 [002] 1076790.002695: kvm_entry: vcpu 1 qemu-system-x86-4586 [003] 1076790.002818: kvm_exit: reason ext_irq rip 0xfffff80001c7e880 qemu-system-x86-4586 [003] 1076790.002819: kvm_entry: vcpu 0 qemu-system-x86-4580 [002] 1076790.003355: kvm_set_irq: gsi 0 level 1 source 0 qemu-system-x86-4580 [002] 1076790.003356: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4580 [002] 1076790.003357: kvm_apic_accept_irq: apicid 0 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076790.003357: kvm_apic_accept_irq: apicid 1 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076790.003357: kvm_apic_accept_irq: apicid 2 vec 209 (Fixed|edge) qemu-system-x86-4586 [003] 1076790.003358: kvm_exit: reason ext_irq rip 0xfffff80001c7e894 qemu-system-x86-4580 [002] 1076790.003358: kvm_apic_accept_irq: apicid 3 vec 209 (Fixed|edge) qemu-system-x86-4586 [003] 1076790.003358: kvm_inj_virq: irq 209 qemu-system-x86-4580 [002] 1076790.003358: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4586 [003] 1076790.003359: kvm_entry: vcpu 0 qemu-system-x86-4580 [002] 1076790.003359: kvm_set_irq: gsi 0 level 0 source 0 qemu-system-x86-4588 [000] 1076790.003359: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4580 [002] 1076790.003359: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4580 [002] 1076790.003359: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4589 [001] 1076790.003360: kvm_exit: reason ext_irq rip 0xfffff80001cd91b7 qemu-system-x86-4588 [000] 1076790.003360: kvm_entry: vcpu 2 qemu-system-x86-4589 [001] 1076790.003360: kvm_inj_virq: irq 209 qemu-system-x86-4586 [003] 1076790.003360: kvm_exit: reason apic_access rip 0xfffff80001c9a5c4 qemu-system-x86-4589 [001] 1076790.003361: kvm_entry: vcpu 3 qemu-system-x86-4588 [000] 1076790.003361: kvm_exit: reason interrupt_window rip 0xfffff80001ca0002 qemu-system-x86-4588 [000] 1076790.003361: kvm_inj_virq: irq 209 qemu-system-x86-4588 [000] 1076790.003362: kvm_entry: vcpu 2 qemu-system-x86-4586 [003] 1076790.003362: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4589 [001] 1076790.003362: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4586 [003] 1076790.003362: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4586 [003] 1076790.003363: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4588 [000] 1076790.003363: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4586 [003] 1076790.003363: kvm_entry: vcpu 0 qemu-system-x86-4589 [001] 1076790.003364: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4589 [001] 1076790.003364: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4589 [001] 1076790.003365: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4588 [000] 1076790.003365: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4588 [000] 1076790.003365: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4588 [000] 1076790.003366: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4589 [001] 1076790.003366: kvm_entry: vcpu 3 qemu-system-x86-4588 [000] 1076790.003366: kvm_entry: vcpu 2 qemu-system-x86-4587 [002] 1076790.003367: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4587 [002] 1076790.003368: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076790.003369: kvm_exit: reason interrupt_window rip 0xfffff80001ca0002 qemu-system-x86-4587 [002] 1076790.003370: kvm_inj_virq: irq 209 qemu-system-x86-4587 [002] 1076790.003370: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076790.003372: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4587 [002] 1076790.003374: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4587 [002] 1076790.003374: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4587 [002] 1076790.003374: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4587 [002] 1076790.003375: kvm_entry: vcpu 1 qemu-system-x86-4588 [000] 1076790.003451: kvm_exit: reason ext_irq rip 0xfffff80001cd8d59 qemu-system-x86-4588 [000] 1076790.003453: kvm_entry: vcpu 2 qemu-system-x86-4587 [002] 1076790.003479: kvm_exit: reason ext_irq rip 0xfffff80001cb76c3 qemu-system-x86-4587 [002] 1076790.003480: kvm_entry: vcpu 1 qemu-system-x86-4589 [001] 1076790.003568: kvm_exit: reason ext_irq rip 0xfffff80001d01a00 qemu-system-x86-4589 [001] 1076790.003568: kvm_entry: vcpu 3 qemu-system-x86-4587 [002] 1076790.003621: kvm_exit: reason ext_irq rip 0xfffff80001d019f1 qemu-system-x86-4587 [002] 1076790.003623: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076790.003692: kvm_exit: reason ext_irq rip 0xfffff80001d01a00 qemu-system-x86-4587 [002] 1076790.003693: kvm_entry: vcpu 1 qemu-system-x86-4586 [003] 1076790.003817: kvm_exit: reason ext_irq rip 0xfffff80001c7e880 qemu-system-x86-4586 [003] 1076790.003817: kvm_entry: vcpu 0 qemu-system-x86-4580 [002] 1076790.004354: kvm_set_irq: gsi 0 level 1 source 0 qemu-system-x86-4580 [002] 1076790.004355: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4580 [002] 1076790.004355: kvm_apic_accept_irq: apicid 0 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076790.004356: kvm_apic_accept_irq: apicid 1 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076790.004356: kvm_apic_accept_irq: apicid 2 vec 209 (Fixed|edge) qemu-system-x86-4586 [003] 1076790.004357: kvm_exit: reason ext_irq rip 0xfffff80001c7e891 qemu-system-x86-4580 [002] 1076790.004357: kvm_apic_accept_irq: apicid 3 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076790.004357: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4586 [003] 1076790.004357: kvm_inj_virq: irq 209 qemu-system-x86-4586 [003] 1076790.004357: kvm_entry: vcpu 0 qemu-system-x86-4588 [000] 1076790.004358: kvm_exit: reason ext_irq rip 0xfffff80001cd8fc5 qemu-system-x86-4580 [002] 1076790.004358: kvm_set_irq: gsi 0 level 0 source 0 qemu-system-x86-4580 [002] 1076790.004358: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4580 [002] 1076790.004358: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4589 [001] 1076790.004358: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4588 [000] 1076790.004358: kvm_inj_virq: irq 209 qemu-system-x86-4588 [000] 1076790.004359: kvm_entry: vcpu 2 qemu-system-x86-4586 [003] 1076790.004359: kvm_exit: reason apic_access rip 0xfffff80001c9a5c4 qemu-system-x86-4589 [001] 1076790.004359: kvm_entry: vcpu 3 qemu-system-x86-4588 [000] 1076790.004360: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4589 [001] 1076790.004360: kvm_exit: reason interrupt_window rip 0xfffff80001ca0002 qemu-system-x86-4586 [003] 1076790.004361: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4586 [003] 1076790.004361: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4589 [001] 1076790.004361: kvm_inj_virq: irq 209 qemu-system-x86-4586 [003] 1076790.004361: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4589 [001] 1076790.004361: kvm_entry: vcpu 3 qemu-system-x86-4586 [003] 1076790.004362: kvm_entry: vcpu 0 qemu-system-x86-4588 [000] 1076790.004362: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4588 [000] 1076790.004362: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4589 [001] 1076790.004363: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4588 [000] 1076790.004363: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4588 [000] 1076790.004364: kvm_entry: vcpu 2 qemu-system-x86-4589 [001] 1076790.004364: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4589 [001] 1076790.004365: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4589 [001] 1076790.004365: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4587 [002] 1076790.004365: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4589 [001] 1076790.004366: kvm_entry: vcpu 3 qemu-system-x86-4587 [002] 1076790.004367: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076790.004368: kvm_exit: reason interrupt_window rip 0xfffff80001ca0002 qemu-system-x86-4587 [002] 1076790.004369: kvm_inj_virq: irq 209 qemu-system-x86-4587 [002] 1076790.004369: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076790.004371: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4587 [002] 1076790.004373: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4587 [002] 1076790.004373: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4587 [002] 1076790.004373: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4587 [002] 1076790.004374: kvm_entry: vcpu 1 qemu-system-x86-4588 [000] 1076790.004442: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4588 [000] 1076790.004442: kvm_entry: vcpu 2 qemu-system-x86-4586 [003] 1076790.004475: kvm_exit: reason ext_irq rip 0xfffff80001c7e891 qemu-system-x86-4586 [003] 1076790.004475: kvm_entry: vcpu 0 qemu-system-x86-4589 [001] 1076790.004566: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4589 [001] 1076790.004567: kvm_entry: vcpu 3 qemu-system-x86-4587 [002] 1076790.004619: kvm_exit: reason ext_irq rip 0xfffff80001c9b859 qemu-system-x86-4587 [002] 1076790.004620: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076790.004690: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4587 [002] 1076790.004691: kvm_entry: vcpu 1 qemu-system-x86-4586 [003] 1076790.004815: kvm_exit: reason ext_irq rip 0xfffff80001c7e894 qemu-system-x86-4586 [003] 1076790.004815: kvm_entry: vcpu 0 qemu-system-x86-4580 [002] 1076790.005352: kvm_set_irq: gsi 0 level 1 source 0 qemu-system-x86-4580 [002] 1076790.005353: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4580 [002] 1076790.005353: kvm_apic_accept_irq: apicid 0 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076790.005354: kvm_apic_accept_irq: apicid 1 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076790.005354: kvm_apic_accept_irq: apicid 2 vec 209 (Fixed|edge) qemu-system-x86-4586 [003] 1076790.005354: kvm_exit: reason ext_irq rip 0xfffff80001c7e891 qemu-system-x86-4580 [002] 1076790.005355: kvm_apic_accept_irq: apicid 3 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076790.005355: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4586 [003] 1076790.005355: kvm_inj_virq: irq 209 qemu-system-x86-4586 [003] 1076790.005355: kvm_entry: vcpu 0 qemu-system-x86-4588 [000] 1076790.005355: kvm_exit: reason ext_irq rip 0xfffff80001cd8caa qemu-system-x86-4580 [002] 1076790.005356: kvm_set_irq: gsi 0 level 0 source 0 qemu-system-x86-4580 [002] 1076790.005356: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4580 [002] 1076790.005356: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4589 [001] 1076790.005356: kvm_exit: reason ext_irq rip 0xfffff80001c9b859 qemu-system-x86-4588 [000] 1076790.005356: kvm_inj_virq: irq 209 qemu-system-x86-4588 [000] 1076790.005357: kvm_entry: vcpu 2 qemu-system-x86-4586 [003] 1076790.005357: kvm_exit: reason apic_access rip 0xfffff80001c9a5c4 qemu-system-x86-4589 [001] 1076790.005357: kvm_inj_virq: irq 209 qemu-system-x86-4589 [001] 1076790.005357: kvm_entry: vcpu 3 qemu-system-x86-4588 [000] 1076790.005358: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4586 [003] 1076790.005359: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4586 [003] 1076790.005359: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4589 [001] 1076790.005359: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4586 [003] 1076790.005359: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4588 [000] 1076790.005360: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4588 [000] 1076790.005360: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4586 [003] 1076790.005360: kvm_entry: vcpu 0 qemu-system-x86-4589 [001] 1076790.005360: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4588 [000] 1076790.005361: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4589 [001] 1076790.005361: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4589 [001] 1076790.005361: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4588 [000] 1076790.005362: kvm_entry: vcpu 2 qemu-system-x86-4589 [001] 1076790.005362: kvm_entry: vcpu 3 qemu-system-x86-4587 [002] 1076790.005363: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4587 [002] 1076790.005365: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076790.005366: kvm_exit: reason interrupt_window rip 0xfffff80001ca0002 qemu-system-x86-4587 [002] 1076790.005367: kvm_inj_virq: irq 209 qemu-system-x86-4587 [002] 1076790.005367: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076790.005369: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4587 [002] 1076790.005371: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4587 [002] 1076790.005371: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4587 [002] 1076790.005371: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4587 [002] 1076790.005372: kvm_entry: vcpu 1 qemu-system-x86-4588 [000] 1076790.005439: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4588 [000] 1076790.005440: kvm_entry: vcpu 2 qemu-system-x86-4589 [001] 1076790.005565: kvm_exit: reason ext_irq rip 0xfffff80001d01a00 qemu-system-x86-4589 [001] 1076790.005565: kvm_entry: vcpu 3 qemu-system-x86-4587 [002] 1076790.005617: kvm_exit: reason ext_irq rip 0xfffff80001d019e4 qemu-system-x86-4587 [002] 1076790.005619: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076790.005689: kvm_exit: reason ext_irq rip 0xfffff80001d019f1 qemu-system-x86-4587 [002] 1076790.005689: kvm_entry: vcpu 1 qemu-system-x86-4586 [003] 1076790.005813: kvm_exit: reason ext_irq rip 0xfffff80001c7e880 qemu-system-x86-4586 [003] 1076790.005814: kvm_entry: vcpu 0 qemu-system-x86-4580 [002] 1076790.006350: kvm_set_irq: gsi 0 level 1 source 0 qemu-system-x86-4580 [002] 1076790.006351: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4580 [002] 1076790.006351: kvm_apic_accept_irq: apicid 0 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076790.006352: kvm_apic_accept_irq: apicid 1 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076790.006352: kvm_apic_accept_irq: apicid 2 vec 209 (Fixed|edge) qemu-system-x86-4586 [003] 1076790.006352: kvm_exit: reason ext_irq rip 0xfffff80001c7e894 qemu-system-x86-4580 [002] 1076790.006353: kvm_apic_accept_irq: apicid 3 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076790.006353: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4586 [003] 1076790.006353: kvm_inj_virq: irq 209 qemu-system-x86-4586 [003] 1076790.006353: kvm_entry: vcpu 0 qemu-system-x86-4588 [000] 1076790.006353: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4580 [002] 1076790.006353: kvm_set_irq: gsi 0 level 0 source 0 qemu-system-x86-4580 [002] 1076790.006354: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4580 [002] 1076790.006354: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4588 [000] 1076790.006354: kvm_entry: vcpu 2 qemu-system-x86-4589 [001] 1076790.006354: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4586 [003] 1076790.006355: kvm_exit: reason apic_access rip 0xfffff80001c9a5c4 qemu-system-x86-4589 [001] 1076790.006355: kvm_entry: vcpu 3 qemu-system-x86-4588 [000] 1076790.006355: kvm_exit: reason interrupt_window rip 0xfffff80001ca0002 qemu-system-x86-4588 [000] 1076790.006356: kvm_inj_virq: irq 209 qemu-system-x86-4589 [001] 1076790.006356: kvm_exit: reason interrupt_window rip 0xfffff80001ca0002 qemu-system-x86-4586 [003] 1076790.006357: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4588 [000] 1076790.006357: kvm_entry: vcpu 2 qemu-system-x86-4586 [003] 1076790.006357: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4589 [001] 1076790.006357: kvm_inj_virq: irq 209 qemu-system-x86-4589 [001] 1076790.006357: kvm_entry: vcpu 3 qemu-system-x86-4586 [003] 1076790.006357: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4588 [000] 1076790.006358: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4586 [003] 1076790.006358: kvm_entry: vcpu 0 qemu-system-x86-4589 [001] 1076790.006359: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4588 [000] 1076790.006360: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4588 [000] 1076790.006360: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4589 [001] 1076790.006360: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4588 [000] 1076790.006360: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4589 [001] 1076790.006361: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4589 [001] 1076790.006361: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4587 [002] 1076790.006361: kvm_exit: reason ext_irq rip 0xfffff80001c9b859 qemu-system-x86-4588 [000] 1076790.006361: kvm_entry: vcpu 2 qemu-system-x86-4589 [001] 1076790.006362: kvm_entry: vcpu 3 qemu-system-x86-4587 [002] 1076790.006363: kvm_inj_virq: irq 209 qemu-system-x86-4587 [002] 1076790.006363: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076790.006365: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4587 [002] 1076790.006367: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4587 [002] 1076790.006368: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4587 [002] 1076790.006368: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4587 [002] 1076790.006369: kvm_entry: vcpu 1 qemu-system-x86-4588 [000] 1076790.006438: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4588 [000] 1076790.006438: kvm_entry: vcpu 2 qemu-system-x86-4589 [001] 1076790.006563: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4589 [001] 1076790.006563: kvm_entry: vcpu 3 qemu-system-x86-4587 [002] 1076790.006615: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4587 [002] 1076790.006617: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076790.006687: kvm_exit: reason ext_irq rip 0xfffff80001d01a00 qemu-system-x86-4587 [002] 1076790.006688: kvm_entry: vcpu 1 qemu-system-x86-4586 [003] 1076790.006812: kvm_exit: reason ext_irq rip 0xfffff80001c7e891 qemu-system-x86-4586 [003] 1076790.006812: kvm_entry: vcpu 0 qemu-system-x86-4580 [002] 1076790.007349: kvm_set_irq: gsi 0 level 1 source 0 qemu-system-x86-4580 [002] 1076790.007350: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4580 [002] 1076790.007350: kvm_apic_accept_irq: apicid 0 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076790.007351: kvm_apic_accept_irq: apicid 1 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076790.007351: kvm_apic_accept_irq: apicid 2 vec 209 (Fixed|edge) qemu-system-x86-4586 [003] 1076790.007351: kvm_exit: reason ext_irq rip 0xfffff80001c7e891 qemu-system-x86-4580 [002] 1076790.007352: kvm_apic_accept_irq: apicid 3 vec 209 (Fixed|edge) qemu-system-x86-4586 [003] 1076790.007352: kvm_inj_virq: irq 209 qemu-system-x86-4580 [002] 1076790.007352: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4586 [003] 1076790.007352: kvm_entry: vcpu 0 qemu-system-x86-4588 [000] 1076790.007352: kvm_exit: reason ext_irq rip 0xfffff80001cd90d2 qemu-system-x86-4580 [002] 1076790.007353: kvm_set_irq: gsi 0 level 0 source 0 qemu-system-x86-4580 [002] 1076790.007353: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4580 [002] 1076790.007353: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4589 [001] 1076790.007353: kvm_exit: reason ext_irq rip 0xfffff80001cde8ec qemu-system-x86-4588 [000] 1076790.007353: kvm_inj_virq: irq 209 qemu-system-x86-4588 [000] 1076790.007354: kvm_entry: vcpu 2 qemu-system-x86-4586 [003] 1076790.007354: kvm_exit: reason apic_access rip 0xfffff80001c9a5c4 qemu-system-x86-4589 [001] 1076790.007354: kvm_inj_virq: irq 209 qemu-system-x86-4589 [001] 1076790.007354: kvm_entry: vcpu 3 qemu-system-x86-4588 [000] 1076790.007355: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4586 [003] 1076790.007355: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4586 [003] 1076790.007356: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4589 [001] 1076790.007356: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4586 [003] 1076790.007356: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4588 [000] 1076790.007357: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4586 [003] 1076790.007357: kvm_entry: vcpu 0 qemu-system-x86-4588 [000] 1076790.007357: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4588 [000] 1076790.007357: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4589 [001] 1076790.007358: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4589 [001] 1076790.007358: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4589 [001] 1076790.007358: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4588 [000] 1076790.007358: kvm_entry: vcpu 2 qemu-system-x86-4589 [001] 1076790.007359: kvm_entry: vcpu 3 qemu-system-x86-4588 [000] 1076790.007436: kvm_exit: reason ext_irq rip 0xfffff80001c9b85d qemu-system-x86-4588 [000] 1076790.007437: kvm_entry: vcpu 2 qemu-system-x86-4587 [002] 1076790.007560: kvm_exit: reason ext_irq rip 0xfffff80001c9b859 qemu-system-x86-4589 [001] 1076790.007561: kvm_exit: reason ext_irq rip 0xfffff80001cda706 qemu-system-x86-4589 [001] 1076790.007562: kvm_entry: vcpu 3 qemu-system-x86-4587 [002] 1076790.007562: kvm_inj_virq: irq 209 qemu-system-x86-4587 [002] 1076790.007563: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076790.007565: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4587 [002] 1076790.007567: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4587 [002] 1076790.007567: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4587 [002] 1076790.007568: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4587 [002] 1076790.007569: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076790.007616: kvm_exit: reason ext_irq rip 0xfffff80001c9b859 qemu-system-x86-4587 [002] 1076790.007618: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076790.007686: kvm_exit: reason ext_irq rip 0xfffff80001c9ffb6 qemu-system-x86-4587 [002] 1076790.007687: kvm_entry: vcpu 1 qemu-system-x86-4586 [003] 1076790.007810: kvm_exit: reason ext_irq rip 0xfffff80001c7e894 qemu-system-x86-4586 [003] 1076790.007811: kvm_entry: vcpu 0 qemu-system-x86-4580 [002] 1076790.008348: kvm_set_irq: gsi 0 level 1 source 0 qemu-system-x86-4580 [002] 1076790.008348: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4580 [002] 1076790.008349: kvm_apic_accept_irq: apicid 0 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076790.008349: kvm_apic_accept_irq: apicid 1 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076790.008350: kvm_apic_accept_irq: apicid 2 vec 209 (Fixed|edge) qemu-system-x86-4586 [003] 1076790.008350: kvm_exit: reason ext_irq rip 0xfffff80001c7e891 qemu-system-x86-4580 [002] 1076790.008350: kvm_apic_accept_irq: apicid 3 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076790.008351: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4586 [003] 1076790.008351: kvm_inj_virq: irq 209 qemu-system-x86-4586 [003] 1076790.008351: kvm_entry: vcpu 0 qemu-system-x86-4588 [000] 1076790.008351: kvm_exit: reason ext_irq rip 0xfffff80001cd8c33 qemu-system-x86-4580 [002] 1076790.008351: kvm_set_irq: gsi 0 level 0 source 0 qemu-system-x86-4580 [002] 1076790.008351: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4580 [002] 1076790.008352: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4589 [001] 1076790.008352: kvm_exit: reason ext_irq rip 0xfffff80001c9b859 qemu-system-x86-4588 [000] 1076790.008352: kvm_inj_virq: irq 209 qemu-system-x86-4588 [000] 1076790.008353: kvm_entry: vcpu 2 qemu-system-x86-4589 [001] 1076790.008353: kvm_inj_virq: irq 209 qemu-system-x86-4586 [003] 1076790.008353: kvm_exit: reason apic_access rip 0xfffff80001c9a5c4 qemu-system-x86-4589 [001] 1076790.008353: kvm_entry: vcpu 3 qemu-system-x86-4586 [003] 1076790.008354: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4588 [000] 1076790.008354: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4586 [003] 1076790.008354: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4589 [001] 1076790.008354: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4586 [003] 1076790.008355: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4586 [003] 1076790.008356: kvm_entry: vcpu 0 qemu-system-x86-4588 [000] 1076790.008356: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4589 [001] 1076790.008356: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4589 [001] 1076790.008356: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4588 [000] 1076790.008356: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4588 [000] 1076790.008357: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4589 [001] 1076790.008357: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4588 [000] 1076790.008358: kvm_entry: vcpu 2 qemu-system-x86-4589 [001] 1076790.008358: kvm_entry: vcpu 3 qemu-system-x86-4587 [002] 1076790.008359: kvm_exit: reason ext_irq rip 0xfffff80001cd8dc3 qemu-system-x86-4587 [002] 1076790.008360: kvm_inj_virq: irq 209 qemu-system-x86-4587 [002] 1076790.008361: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076790.008362: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4587 [002] 1076790.008364: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4587 [002] 1076790.008365: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4587 [002] 1076790.008365: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4587 [002] 1076790.008366: kvm_entry: vcpu 1 qemu-system-x86-4588 [000] 1076790.008435: kvm_exit: reason ext_irq rip 0xfffff80001c9b859 qemu-system-x86-4588 [000] 1076790.008436: kvm_entry: vcpu 2 qemu-system-x86-4589 [001] 1076790.008560: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4589 [001] 1076790.008560: kvm_entry: vcpu 3 qemu-system-x86-4587 [002] 1076790.008612: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4587 [002] 1076790.008614: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076790.008684: kvm_exit: reason ext_irq rip 0xfffff80001d019e4 qemu-system-x86-4587 [002] 1076790.008685: kvm_entry: vcpu 1 qemu-system-x86-4586 [003] 1076790.008809: kvm_exit: reason ext_irq rip 0xfffff80001c7e891 qemu-system-x86-4586 [003] 1076790.008809: kvm_entry: vcpu 0 qemu-system-x86-4580 [002] 1076790.009345: kvm_set_irq: gsi 0 level 1 source 0 qemu-system-x86-4580 [002] 1076790.009346: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4580 [002] 1076790.009346: kvm_apic_accept_irq: apicid 0 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076790.009347: kvm_apic_accept_irq: apicid 1 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076790.009347: kvm_apic_accept_irq: apicid 2 vec 209 (Fixed|edge) qemu-system-x86-4586 [003] 1076790.009348: kvm_exit: reason ext_irq rip 0xfffff80001c7e894 qemu-system-x86-4580 [002] 1076790.009348: kvm_apic_accept_irq: apicid 3 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076790.009348: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4586 [003] 1076790.009348: kvm_inj_virq: irq 209 qemu-system-x86-4586 [003] 1076790.009348: kvm_entry: vcpu 0 qemu-system-x86-4588 [000] 1076790.009349: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4580 [002] 1076790.009349: kvm_set_irq: gsi 0 level 0 source 0 qemu-system-x86-4580 [002] 1076790.009349: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4580 [002] 1076790.009349: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4588 [000] 1076790.009349: kvm_entry: vcpu 2 qemu-system-x86-4589 [001] 1076790.009349: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4589 [001] 1076790.009350: kvm_entry: vcpu 3 qemu-system-x86-4586 [003] 1076790.009350: kvm_exit: reason apic_access rip 0xfffff80001c9a5c4 qemu-system-x86-4588 [000] 1076790.009350: kvm_exit: reason interrupt_window rip 0xfffff80001ca0002 qemu-system-x86-4588 [000] 1076790.009351: kvm_inj_virq: irq 209 qemu-system-x86-4589 [001] 1076790.009351: kvm_exit: reason interrupt_window rip 0xfffff80001ca0002 qemu-system-x86-4588 [000] 1076790.009351: kvm_entry: vcpu 2 qemu-system-x86-4586 [003] 1076790.009352: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4586 [003] 1076790.009352: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4589 [001] 1076790.009352: kvm_inj_virq: irq 209 qemu-system-x86-4586 [003] 1076790.009352: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4589 [001] 1076790.009352: kvm_entry: vcpu 3 qemu-system-x86-4588 [000] 1076790.009353: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4586 [003] 1076790.009353: kvm_entry: vcpu 0 qemu-system-x86-4589 [001] 1076790.009354: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4588 [000] 1076790.009354: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4588 [000] 1076790.009355: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4588 [000] 1076790.009355: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4589 [001] 1076790.009355: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4589 [001] 1076790.009356: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4589 [001] 1076790.009356: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4588 [000] 1076790.009356: kvm_entry: vcpu 2 qemu-system-x86-4587 [002] 1076790.009356: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4589 [001] 1076790.009357: kvm_entry: vcpu 3 qemu-system-x86-4587 [002] 1076790.009358: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076790.009359: kvm_exit: reason interrupt_window rip 0xfffff80001ca0002 qemu-system-x86-4587 [002] 1076790.009360: kvm_inj_virq: irq 209 qemu-system-x86-4587 [002] 1076790.009360: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076790.009362: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4587 [002] 1076790.009364: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4587 [002] 1076790.009364: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4587 [002] 1076790.009364: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4587 [002] 1076790.009365: kvm_entry: vcpu 1 qemu-system-x86-4588 [000] 1076790.009433: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4588 [000] 1076790.009434: kvm_entry: vcpu 2 qemu-system-x86-4589 [001] 1076790.009559: kvm_exit: reason ext_irq rip 0xfffff80001c9b85d qemu-system-x86-4586 [003] 1076790.009559: kvm_exit: reason ext_irq rip 0xfffff80001c7e891 qemu-system-x86-4589 [001] 1076790.009560: kvm_entry: vcpu 3 qemu-system-x86-4586 [003] 1076790.009560: kvm_entry: vcpu 0 qemu-system-x86-4587 [002] 1076790.009610: kvm_exit: reason ext_irq rip 0xfffff80001cd9112 qemu-system-x86-4587 [002] 1076790.009611: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076790.009683: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4587 [002] 1076790.009683: kvm_entry: vcpu 1 qemu-system-x86-4586 [003] 1076790.009807: kvm_exit: reason ext_irq rip 0xfffff80001c7e891 qemu-system-x86-4586 [003] 1076790.009808: kvm_entry: vcpu 0 qemu-system-x86-4580 [002] 1076790.010344: kvm_set_irq: gsi 0 level 1 source 0 qemu-system-x86-4580 [002] 1076790.010344: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4580 [002] 1076790.010345: kvm_apic_accept_irq: apicid 0 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076790.010345: kvm_apic_accept_irq: apicid 1 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076790.010346: kvm_apic_accept_irq: apicid 2 vec 209 (Fixed|edge) qemu-system-x86-4586 [003] 1076790.010346: kvm_exit: reason ext_irq rip 0xfffff80001c7e88f qemu-system-x86-4580 [002] 1076790.010346: kvm_apic_accept_irq: apicid 3 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076790.010347: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4586 [003] 1076790.010347: kvm_inj_virq: irq 209 qemu-system-x86-4586 [003] 1076790.010347: kvm_entry: vcpu 0 qemu-system-x86-4588 [000] 1076790.010347: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4580 [002] 1076790.010347: kvm_set_irq: gsi 0 level 0 source 0 qemu-system-x86-4580 [002] 1076790.010348: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4580 [002] 1076790.010348: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4588 [000] 1076790.010348: kvm_entry: vcpu 2 qemu-system-x86-4589 [001] 1076790.010348: kvm_exit: reason ext_irq rip 0xfffff80001cd8b43 qemu-system-x86-4586 [003] 1076790.010349: kvm_exit: reason apic_access rip 0xfffff80001c9a5c4 qemu-system-x86-4589 [001] 1076790.010349: kvm_inj_virq: irq 209 qemu-system-x86-4589 [001] 1076790.010349: kvm_entry: vcpu 3 qemu-system-x86-4588 [000] 1076790.010349: kvm_exit: reason interrupt_window rip 0xfffff80001ca0002 qemu-system-x86-4588 [000] 1076790.010350: kvm_inj_virq: irq 209 qemu-system-x86-4588 [000] 1076790.010350: kvm_entry: vcpu 2 qemu-system-x86-4586 [003] 1076790.010350: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4586 [003] 1076790.010351: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4589 [001] 1076790.010351: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4586 [003] 1076790.010351: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4588 [000] 1076790.010352: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4586 [003] 1076790.010352: kvm_entry: vcpu 0 qemu-system-x86-4589 [001] 1076790.010352: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4589 [001] 1076790.010352: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4589 [001] 1076790.010353: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4588 [000] 1076790.010353: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4588 [000] 1076790.010353: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4588 [000] 1076790.010354: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4589 [001] 1076790.010354: kvm_entry: vcpu 3 qemu-system-x86-4588 [000] 1076790.010355: kvm_entry: vcpu 2 qemu-system-x86-4587 [002] 1076790.010355: kvm_exit: reason ext_irq rip 0xfffff80001ca142d qemu-system-x86-4587 [002] 1076790.010356: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076790.010358: kvm_exit: reason interrupt_window rip 0xfffff80001c9b859 qemu-system-x86-4587 [002] 1076790.010358: kvm_inj_virq: irq 209 qemu-system-x86-4587 [002] 1076790.010358: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076790.010360: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4587 [002] 1076790.010362: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4587 [002] 1076790.010362: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4587 [002] 1076790.010363: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4587 [002] 1076790.010364: kvm_entry: vcpu 1 qemu-system-x86-4588 [000] 1076790.010432: kvm_exit: reason ext_irq rip 0xfffff80001c9ffbf qemu-system-x86-4588 [000] 1076790.010432: kvm_entry: vcpu 2 qemu-system-x86-4589 [001] 1076790.010557: kvm_exit: reason ext_irq rip 0xfffff80001cd90d2 qemu-system-x86-4589 [001] 1076790.010557: kvm_entry: vcpu 3 qemu-system-x86-4587 [002] 1076790.010609: kvm_exit: reason ext_irq rip 0xfffff80001d01a00 qemu-system-x86-4587 [002] 1076790.010610: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076790.010681: kvm_exit: reason ext_irq rip 0xfffff80001d01a00 qemu-system-x86-4587 [002] 1076790.010681: kvm_entry: vcpu 1 qemu-system-x86-4586 [003] 1076790.010806: kvm_exit: reason ext_irq rip 0xfffff80001c7e891 qemu-system-x86-4586 [003] 1076790.010806: kvm_entry: vcpu 0 qemu-system-x86-4580 [002] 1076790.011343: kvm_set_irq: gsi 0 level 1 source 0 qemu-system-x86-4580 [002] 1076790.011343: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4580 [002] 1076790.011344: kvm_apic_accept_irq: apicid 0 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076790.011344: kvm_apic_accept_irq: apicid 1 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076790.011345: kvm_apic_accept_irq: apicid 2 vec 209 (Fixed|edge) qemu-system-x86-4586 [003] 1076790.011345: kvm_exit: reason ext_irq rip 0xfffff80001c7e891 qemu-system-x86-4580 [002] 1076790.011345: kvm_apic_accept_irq: apicid 3 vec 209 (Fixed|edge) qemu-system-x86-4580 [002] 1076790.011345: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4586 [003] 1076790.011346: kvm_inj_virq: irq 209 qemu-system-x86-4580 [002] 1076790.011346: kvm_set_irq: gsi 0 level 0 source 0 qemu-system-x86-4586 [003] 1076790.011346: kvm_entry: vcpu 0 qemu-system-x86-4588 [000] 1076790.011346: kvm_exit: reason ext_irq rip 0xfffff80001c9b859 qemu-system-x86-4580 [002] 1076790.011346: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-4580 [002] 1076790.011347: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-4589 [001] 1076790.011347: kvm_exit: reason ext_irq rip 0xfffff80001cb7d22 qemu-system-x86-4588 [000] 1076790.011347: kvm_inj_virq: irq 209 qemu-system-x86-4588 [000] 1076790.011347: kvm_entry: vcpu 2 qemu-system-x86-4589 [001] 1076790.011347: kvm_inj_virq: irq 209 qemu-system-x86-4589 [001] 1076790.011348: kvm_entry: vcpu 3 qemu-system-x86-4586 [003] 1076790.011348: kvm_exit: reason apic_access rip 0xfffff80001c9a5c4 qemu-system-x86-4588 [000] 1076790.011349: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4589 [001] 1076790.011349: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4586 [003] 1076790.011349: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4586 [003] 1076790.011350: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4586 [003] 1076790.011350: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4588 [000] 1076790.011350: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4588 [000] 1076790.011351: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4589 [001] 1076790.011351: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4586 [003] 1076790.011351: kvm_entry: vcpu 0 qemu-system-x86-4589 [001] 1076790.011351: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4588 [000] 1076790.011351: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4589 [001] 1076790.011351: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4588 [000] 1076790.011352: kvm_entry: vcpu 2 qemu-system-x86-4589 [001] 1076790.011352: kvm_entry: vcpu 3 qemu-system-x86-4587 [002] 1076790.011354: kvm_exit: reason ext_irq rip 0xfffff80001c9b859 qemu-system-x86-4587 [002] 1076790.011355: kvm_inj_virq: irq 209 qemu-system-x86-4587 [002] 1076790.011355: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076790.011357: kvm_exit: reason apic_access rip 0xfffff80001c9af13 qemu-system-x86-4587 [002] 1076790.011359: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-4587 [002] 1076790.011359: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-4587 [002] 1076790.011360: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-4587 [002] 1076790.011361: kvm_entry: vcpu 1 qemu-system-x86-4589 [001] 1076790.011555: kvm_exit: reason ext_irq rip 0xfffff80001ca1302 qemu-system-x86-4589 [001] 1076790.011556: kvm_entry: vcpu 3 qemu-system-x86-4587 [002] 1076790.011608: kvm_exit: reason ext_irq rip 0xfffff80001d01a00 qemu-system-x86-4587 [002] 1076790.011609: kvm_entry: vcpu 1 qemu-system-x86-4587 [002] 1076790.011679: kvm_exit: reason ext_irq rip 0xfffff80001c9ff40 qemu-system-x86-4587 [002] 1076790.011680: kvm_entry: vcpu 1 qemu-system-x86-4586 [003] 1076790.011804: kvm_exit: reason ext_irq rip 0xfffff80001c7e88f qemu-system-x86-4586 [003] 1076790.011805: kvm_entry: vcpu 0 -- Harri. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Freezing Windows 2008 x64bit guest 2010-07-19 7:17 ` Harri Olin @ 2010-07-19 7:42 ` Gleb Natapov 2010-07-21 6:25 ` Harri Olin 0 siblings, 1 reply; 22+ messages in thread From: Gleb Natapov @ 2010-07-19 7:42 UTC (permalink / raw) To: Harri Olin; +Cc: Christoph Adomeit, kvm@vger.kernel.org On Mon, Jul 19, 2010 at 10:17:02AM +0300, Harri Olin wrote: > Gleb Natapov kirjoitti: > >On Thu, Jul 15, 2010 at 03:19:44PM +0200, Christoph Adomeit wrote: > >> > >>But one Windows 2008 64 Bit Server Standard is freezing regularly. > >>This happens sometimes 3 times a day, sometimes it takes 2 days > >>until freeze. The Windows Machine is a clean fresh install. > >> > >>By Freezing I mean that the Machine console still shows the > >>windows logo but does not accept any keystroke and does not > >>answer > >>to ping or rdp or whatever. The virtual cpu-load is then shown > >>as 100% (by proxmox). > > Hi > > I think I have seen same problem occur on my Windows 2008 SBS SP2 > 64bit system, but a bit less often, only like once a week. > > Short description of system: > Supermicro X7SBL-LN2 motherboard, Q8300 cpu, 8GB memory, Areca > ARC-1210 RAID controller. > > Linux kernel 2.6.33.1, qemu-kvm-0.11.1, windows 2008 quest is the > only one at the moment. > > I had to step down to 0.11.1 from 0.12.x as that was more unstable, > windows 2008 bluescreened every second day. Also 0.11.1 did the same > when I applied -rtc-td-hack, I think. (might also be -cpu host, > which I removed at the same time but I think the crashing started > when I added -rt-td-hack). > Sample bluescreen error codes from that time: > 0x0000003b (0x00000000c0000005, 0xfffff80001cb8842, 0xfffffa600 > 0x0000003b (0x00000000c0000005, 0xfffff80001c9c842, 0xfffffa600 > 0x0000000a (0x0000000000000080, 0x000000000000000c, 0x000000000 > 0x0000003b (0x00000000c0000005, 0xfffff80001c5d842, 0xfffffa600 > Also I think I have seen qemu crashing once.. > > Now I haven't seen crashes but only freezes with qemu on 100% and > virtual system unresponsive. > > qemu command line: > /usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive > file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive > file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net > nic,macaddr=C0:FF:12:FB:AA:01,model=e1000 -net tap -smp 4 -localtime > > Try with different model then e1000 please. Default driver that comes with Windows known to have problems. > >When hang occurs ensure that problematic vm is the only on on the server and run > >kvm_stat. Send output here. Also do the following: > > > ># mount -t debugfs debugfs /sys/kernel/debug > ># echo kvm > /sys/kernel/debug/tracing/set_event > ># sleep 1 > ># cat /sys/kernel/debug/tracing/trace > /tmp/trace > > > >Send /tmp/trace here too, but it may be huge, so send only last 1000 > >lines. > -- Gleb. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Freezing Windows 2008 x64bit guest 2010-07-19 7:42 ` Gleb Natapov @ 2010-07-21 6:25 ` Harri Olin 2010-07-21 8:28 ` Christoph Adomeit 2010-07-21 8:37 ` Gleb Natapov 0 siblings, 2 replies; 22+ messages in thread From: Harri Olin @ 2010-07-21 6:25 UTC (permalink / raw) To: Gleb Natapov; +Cc: Christoph Adomeit, kvm@vger.kernel.org Gleb Natapov kirjoitti: > On Mon, Jul 19, 2010 at 10:17:02AM +0300, Harri Olin wrote: >> Gleb Natapov kirjoitti: >>> On Thu, Jul 15, 2010 at 03:19:44PM +0200, Christoph Adomeit wrote: >>>> But one Windows 2008 64 Bit Server Standard is freezing regularly. >>>> This happens sometimes 3 times a day, sometimes it takes 2 days >>>> until freeze. The Windows Machine is a clean fresh install. >> I think I have seen same problem occur on my Windows 2008 SBS SP2 >> 64bit system, but a bit less often, only like once a week. >> Now I haven't seen crashes but only freezes with qemu on 100% and >> virtual system unresponsive. >> >> qemu command line: >> /usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive >> file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive >> file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net >> nic,macaddr=C0:FF:12:FB:AA:01,model=e1000 -net tap -smp 4 -localtime >> >> > Try with different model then e1000 please. Default driver that comes > with Windows known to have problems. Didn't help, changed to default realtek emulation, but system freezed again. Command line: /usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net nic,macaddr=C0:FF:12:FB:AA:01 -net tap -smp 4 -localtime This time the virtual system was not totally unresponsive somehow, system pinged just fine and I think DNS server worked too. However on console mouse moved but didn't react to clicking, ctrl-alt-del, etc. > >>> When hang occurs ensure that problematic vm is the only on on the server and run >>> kvm_stat. Send output here. Also do the following: >>> >>> # mount -t debugfs debugfs /sys/kernel/debug >>> # echo kvm > /sys/kernel/debug/tracing/set_event >>> # sleep 1 >>> # cat /sys/kernel/debug/tracing/trace > /tmp/trace >>> >>> Send /tmp/trace here too, but it may be huge, so send only last 1000 >>> lines. Stats from this hang: log style http://mizar.remote.agasha.com/k/kvm/kvm_stat_2_log.txt kvm statistics efer_reload 0 0 exits 5368720296 21258 fpu_reload 2826750721 2216 halt_exits 695286653 0 halt_wakeup 685187661 0 host_state_reload 3010399640 2216 hypercalls 0 0 insn_emulation 2765352032 6542 insn_emulation_fail 477 0 invlpg 42417110 204 io_exits 818764026 122 irq_exits 109659572 9870 irq_injections 788992032 4643 irq_window 22170277 1718 largepages 0 0 mmio_exits 1490912756 69 mmu_cache_miss 2471963 0 mmu_flooded 905690 0 mmu_pde_zapped 1821325 0 mmu_pte_updated 3256583 0 mmu_pte_write 2896199 0 mmu_recycled 259845 0 full trace: http://mizar.remote.agasha.com/k/kvm/kvm_trace_2_full.txt qemu-system-x86-5155 [002] 1248198.487147: kvm_entry: vcpu 3 qemu-system-x86-5152 [000] 1248198.487147: kvm_inj_virq: irq 209 qemu-system-x86-5152 [000] 1248198.487148: kvm_entry: vcpu 0 qemu-system-x86-5154 [001] 1248198.487148: kvm_exit: reason apic_access rip 0xfffff80001c9df13 qemu-system-x86-5155 [002] 1248198.487148: kvm_exit: reason apic_access rip 0xfffff80001c9df13 qemu-system-x86-5152 [000] 1248198.487150: kvm_exit: reason apic_access rip 0xfffff80001c9d5c4 qemu-system-x86-5154 [001] 1248198.487150: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-5154 [001] 1248198.487150: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-5155 [002] 1248198.487150: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-5155 [002] 1248198.487151: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-5154 [001] 1248198.487151: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-5155 [002] 1248198.487151: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-5152 [000] 1248198.487151: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-5152 [000] 1248198.487152: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-5154 [001] 1248198.487152: kvm_entry: vcpu 2 qemu-system-x86-5155 [002] 1248198.487152: kvm_entry: vcpu 3 qemu-system-x86-5152 [000] 1248198.487152: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-5153 [003] 1248198.487153: kvm_exit: reason ext_irq rip 0xfffff80001d04a00 qemu-system-x86-5152 [000] 1248198.487153: kvm_entry: vcpu 0 qemu-system-x86-5153 [003] 1248198.487154: kvm_inj_virq: irq 209 qemu-system-x86-5153 [003] 1248198.487155: kvm_entry: vcpu 1 qemu-system-x86-5153 [003] 1248198.487157: kvm_exit: reason apic_access rip 0xfffff80001c9df13 qemu-system-x86-5153 [003] 1248198.487159: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-5153 [003] 1248198.487159: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-5153 [003] 1248198.487160: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-5153 [003] 1248198.487161: kvm_entry: vcpu 1 qemu-system-x86-5155 [002] 1248198.487198: kvm_exit: reason ext_irq rip 0xfffff80001ca42c3 qemu-system-x86-5155 [002] 1248198.487199: kvm_entry: vcpu 3 qemu-system-x86-5153 [003] 1248198.487324: kvm_exit: reason ext_irq rip 0xfffff80001c9e859 qemu-system-x86-5153 [003] 1248198.487324: kvm_entry: vcpu 1 qemu-system-x86-5153 [003] 1248198.487407: kvm_exit: reason ext_irq rip 0xfffff80001ca2f40 qemu-system-x86-5153 [003] 1248198.487408: kvm_entry: vcpu 1 qemu-system-x86-5152 [000] 1248198.487947: kvm_exit: reason ext_irq rip 0xfffff80001d04a00 qemu-system-x86-5152 [000] 1248198.487948: kvm_entry: vcpu 0 qemu-system-x86-5154 [001] 1248198.488072: kvm_exit: reason ext_irq rip 0xfffff80001c9e859 qemu-system-x86-5154 [001] 1248198.488072: kvm_entry: vcpu 2 qemu-system-x86-5146 [003] 1248198.488141: kvm_set_irq: gsi 0 level 1 source 0 qemu-system-x86-5146 [003] 1248198.488141: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-5146 [003] 1248198.488142: kvm_apic_accept_irq: apicid 0 vec 209 (Fixed|edge) qemu-system-x86-5146 [003] 1248198.488142: kvm_apic_accept_irq: apicid 1 vec 209 (Fixed|edge) qemu-system-x86-5146 [003] 1248198.488143: kvm_apic_accept_irq: apicid 2 vec 209 (Fixed|edge) qemu-system-x86-5146 [003] 1248198.488143: kvm_apic_accept_irq: apicid 3 vec 209 (Fixed|edge) qemu-system-x86-5152 [000] 1248198.488144: kvm_exit: reason ext_irq rip 0xfffff80001ca2f40 qemu-system-x86-5146 [003] 1248198.488144: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-5146 [003] 1248198.488144: kvm_set_irq: gsi 0 level 0 source 0 qemu-system-x86-5152 [000] 1248198.488144: kvm_entry: vcpu 0 qemu-system-x86-5154 [001] 1248198.488145: kvm_exit: reason ext_irq rip 0xfffff80001d049f1 qemu-system-x86-5146 [003] 1248198.488145: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-5146 [003] 1248198.488145: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-5155 [002] 1248198.488145: kvm_exit: reason ext_irq rip 0xfffff80001d04a00 qemu-system-x86-5154 [001] 1248198.488145: kvm_inj_virq: irq 209 qemu-system-x86-5155 [002] 1248198.488145: kvm_inj_virq: irq 209 qemu-system-x86-5154 [001] 1248198.488146: kvm_entry: vcpu 2 qemu-system-x86-5152 [000] 1248198.488146: kvm_exit: reason interrupt_window rip 0xfffff80001ca3002 qemu-system-x86-5155 [002] 1248198.488146: kvm_entry: vcpu 3 qemu-system-x86-5152 [000] 1248198.488146: kvm_inj_virq: irq 209 qemu-system-x86-5152 [000] 1248198.488147: kvm_entry: vcpu 0 qemu-system-x86-5154 [001] 1248198.488147: kvm_exit: reason apic_access rip 0xfffff80001c9df13 qemu-system-x86-5155 [002] 1248198.488147: kvm_exit: reason apic_access rip 0xfffff80001c9df13 qemu-system-x86-5152 [000] 1248198.488148: kvm_exit: reason apic_access rip 0xfffff80001c9d5c4 qemu-system-x86-5154 [001] 1248198.488149: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-5154 [001] 1248198.488149: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-5155 [002] 1248198.488149: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-5155 [002] 1248198.488149: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-5154 [001] 1248198.488149: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-5155 [002] 1248198.488150: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-5152 [000] 1248198.488150: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-5152 [000] 1248198.488150: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-5152 [000] 1248198.488151: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-5154 [001] 1248198.488151: kvm_entry: vcpu 2 qemu-system-x86-5155 [002] 1248198.488152: kvm_entry: vcpu 3 qemu-system-x86-5152 [000] 1248198.488152: kvm_entry: vcpu 0 qemu-system-x86-5153 [003] 1248198.488153: kvm_exit: reason ext_irq rip 0xfffff80001d04a00 qemu-system-x86-5153 [003] 1248198.488154: kvm_inj_virq: irq 209 qemu-system-x86-5153 [003] 1248198.488154: kvm_entry: vcpu 1 qemu-system-x86-5153 [003] 1248198.488156: kvm_exit: reason apic_access rip 0xfffff80001c9df13 qemu-system-x86-5153 [003] 1248198.488158: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-5153 [003] 1248198.488158: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-5153 [003] 1248198.488159: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-5153 [003] 1248198.488160: kvm_entry: vcpu 1 qemu-system-x86-5155 [002] 1248198.488197: kvm_exit: reason ext_irq rip 0xfffff80001ca2f40 qemu-system-x86-5155 [002] 1248198.488197: kvm_entry: vcpu 3 qemu-system-x86-5153 [003] 1248198.488322: kvm_exit: reason ext_irq rip 0xfffff80001cba795 qemu-system-x86-5153 [003] 1248198.488322: kvm_entry: vcpu 1 qemu-system-x86-5153 [003] 1248198.488406: kvm_exit: reason ext_irq rip 0xfffff80001d04a00 qemu-system-x86-5153 [003] 1248198.488407: kvm_entry: vcpu 1 qemu-system-x86-5152 [000] 1248198.488945: kvm_exit: reason ext_irq rip 0xfffff80001cdc193 qemu-system-x86-5152 [000] 1248198.488946: kvm_entry: vcpu 0 qemu-system-x86-5154 [001] 1248198.489070: kvm_exit: reason ext_irq rip 0xfffff80001ca2f40 qemu-system-x86-5154 [001] 1248198.489071: kvm_entry: vcpu 2 qemu-system-x86-5146 [003] 1248198.489139: kvm_set_irq: gsi 0 level 1 source 0 qemu-system-x86-5146 [003] 1248198.489139: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-5146 [003] 1248198.489140: kvm_apic_accept_irq: apicid 0 vec 209 (Fixed|edge) qemu-system-x86-5146 [003] 1248198.489140: kvm_apic_accept_irq: apicid 1 vec 209 (Fixed|edge) qemu-system-x86-5146 [003] 1248198.489141: kvm_apic_accept_irq: apicid 2 vec 209 (Fixed|edge) qemu-system-x86-5146 [003] 1248198.489141: kvm_apic_accept_irq: apicid 3 vec 209 (Fixed|edge) qemu-system-x86-5146 [003] 1248198.489142: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-5152 [000] 1248198.489142: kvm_exit: reason ext_irq rip 0xfffff80001d04a00 qemu-system-x86-5146 [003] 1248198.489142: kvm_set_irq: gsi 0 level 0 source 0 qemu-system-x86-5146 [003] 1248198.489142: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-5152 [000] 1248198.489142: kvm_inj_virq: irq 209 qemu-system-x86-5146 [003] 1248198.489143: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-5154 [001] 1248198.489143: kvm_exit: reason ext_irq rip 0xfffff80001d04a00 qemu-system-x86-5155 [002] 1248198.489143: kvm_exit: reason ext_irq rip 0xfffff80001d049e4 qemu-system-x86-5152 [000] 1248198.489143: kvm_entry: vcpu 0 qemu-system-x86-5154 [001] 1248198.489143: kvm_inj_virq: irq 209 qemu-system-x86-5155 [002] 1248198.489143: kvm_inj_virq: irq 209 qemu-system-x86-5154 [001] 1248198.489144: kvm_entry: vcpu 2 qemu-system-x86-5155 [002] 1248198.489144: kvm_entry: vcpu 3 qemu-system-x86-5152 [000] 1248198.489145: kvm_exit: reason apic_access rip 0xfffff80001c9d5c4 qemu-system-x86-5154 [001] 1248198.489145: kvm_exit: reason apic_access rip 0xfffff80001c9df13 qemu-system-x86-5155 [002] 1248198.489145: kvm_exit: reason apic_access rip 0xfffff80001c9df13 qemu-system-x86-5152 [000] 1248198.489146: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-5152 [000] 1248198.489147: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-5154 [001] 1248198.489147: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-5154 [001] 1248198.489147: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-5155 [002] 1248198.489147: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-5152 [000] 1248198.489147: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-5155 [002] 1248198.489147: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-5154 [001] 1248198.489147: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-5155 [002] 1248198.489149: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-5152 [000] 1248198.489149: kvm_entry: vcpu 0 qemu-system-x86-5154 [001] 1248198.489150: kvm_entry: vcpu 2 qemu-system-x86-5153 [003] 1248198.489150: kvm_exit: reason ext_irq rip 0xfffff80001ca468f qemu-system-x86-5155 [002] 1248198.489151: kvm_entry: vcpu 3 qemu-system-x86-5153 [003] 1248198.489151: kvm_inj_virq: irq 209 qemu-system-x86-5153 [003] 1248198.489152: kvm_entry: vcpu 1 qemu-system-x86-5153 [003] 1248198.489153: kvm_exit: reason apic_access rip 0xfffff80001c9df13 qemu-system-x86-5153 [003] 1248198.489156: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-5153 [003] 1248198.489156: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-5153 [003] 1248198.489156: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-5153 [003] 1248198.489157: kvm_entry: vcpu 1 qemu-system-x86-5155 [002] 1248198.489195: kvm_exit: reason ext_irq rip 0xfffff80001ca2f40 qemu-system-x86-5155 [002] 1248198.489195: kvm_entry: vcpu 3 qemu-system-x86-5153 [003] 1248198.489321: kvm_exit: reason ext_irq rip 0xfffff80001ca2f40 qemu-system-x86-5154 [001] 1248198.489322: kvm_exit: reason ext_irq rip 0xfffff80001c9e859 qemu-system-x86-5153 [003] 1248198.489322: kvm_entry: vcpu 1 qemu-system-x86-5154 [001] 1248198.489323: kvm_entry: vcpu 2 qemu-system-x86-5153 [003] 1248198.489404: kvm_exit: reason ext_irq rip 0xfffff80001d04a00 qemu-system-x86-5153 [003] 1248198.489406: kvm_entry: vcpu 1 qemu-system-x86-5152 [000] 1248198.489944: kvm_exit: reason ext_irq rip 0xfffff80001d04a00 qemu-system-x86-5152 [000] 1248198.489945: kvm_entry: vcpu 0 qemu-system-x86-5154 [001] 1248198.490069: kvm_exit: reason ext_irq rip 0xfffff80001cdd795 qemu-system-x86-5154 [001] 1248198.490069: kvm_entry: vcpu 2 qemu-system-x86-5146 [003] 1248198.490137: kvm_set_irq: gsi 0 level 1 source 0 qemu-system-x86-5146 [003] 1248198.490137: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-5146 [003] 1248198.490138: kvm_apic_accept_irq: apicid 0 vec 209 (Fixed|edge) qemu-system-x86-5146 [003] 1248198.490139: kvm_apic_accept_irq: apicid 1 vec 209 (Fixed|edge) qemu-system-x86-5146 [003] 1248198.490139: kvm_apic_accept_irq: apicid 2 vec 209 (Fixed|edge) qemu-system-x86-5146 [003] 1248198.490140: kvm_apic_accept_irq: apicid 3 vec 209 (Fixed|edge) qemu-system-x86-5146 [003] 1248198.490140: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-5152 [000] 1248198.490140: kvm_exit: reason ext_irq rip 0xfffff80001d04a00 qemu-system-x86-5146 [003] 1248198.490140: kvm_set_irq: gsi 0 level 0 source 0 qemu-system-x86-5152 [000] 1248198.490141: kvm_inj_virq: irq 209 qemu-system-x86-5146 [003] 1248198.490141: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-5146 [003] 1248198.490141: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-5154 [001] 1248198.490141: kvm_exit: reason ext_irq rip 0xfffff80001d04a00 qemu-system-x86-5155 [002] 1248198.490141: kvm_exit: reason ext_irq rip 0xfffff80001ca2fbf qemu-system-x86-5152 [000] 1248198.490141: kvm_entry: vcpu 0 qemu-system-x86-5154 [001] 1248198.490142: kvm_inj_virq: irq 209 qemu-system-x86-5155 [002] 1248198.490142: kvm_entry: vcpu 3 qemu-system-x86-5154 [001] 1248198.490142: kvm_entry: vcpu 2 qemu-system-x86-5152 [000] 1248198.490143: kvm_exit: reason apic_access rip 0xfffff80001c9d5c4 qemu-system-x86-5155 [002] 1248198.490143: kvm_exit: reason interrupt_window rip 0xfffff80001ca3002 qemu-system-x86-5154 [001] 1248198.490143: kvm_exit: reason apic_access rip 0xfffff80001c9df13 qemu-system-x86-5155 [002] 1248198.490144: kvm_inj_virq: irq 209 qemu-system-x86-5155 [002] 1248198.490144: kvm_entry: vcpu 3 qemu-system-x86-5152 [000] 1248198.490144: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-5152 [000] 1248198.490145: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-5152 [000] 1248198.490145: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-5154 [001] 1248198.490145: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-5154 [001] 1248198.490146: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-5155 [002] 1248198.490146: kvm_exit: reason apic_access rip 0xfffff80001c9df13 qemu-system-x86-5154 [001] 1248198.490146: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-5152 [000] 1248198.490146: kvm_entry: vcpu 0 qemu-system-x86-5154 [001] 1248198.490147: kvm_entry: vcpu 2 qemu-system-x86-5155 [002] 1248198.490147: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-5155 [002] 1248198.490148: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-5155 [002] 1248198.490148: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-5153 [003] 1248198.490149: kvm_exit: reason ext_irq rip 0xfffff80001d04a00 qemu-system-x86-5155 [002] 1248198.490149: kvm_entry: vcpu 3 qemu-system-x86-5153 [003] 1248198.490150: kvm_inj_virq: irq 209 qemu-system-x86-5153 [003] 1248198.490150: kvm_entry: vcpu 1 qemu-system-x86-5153 [003] 1248198.490152: kvm_exit: reason apic_access rip 0xfffff80001c9df13 qemu-system-x86-5153 [003] 1248198.490154: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-5153 [003] 1248198.490154: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-5153 [003] 1248198.490154: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-5153 [003] 1248198.490155: kvm_entry: vcpu 1 qemu-system-x86-5155 [002] 1248198.490193: kvm_exit: reason ext_irq rip 0xfffff80001ca2f40 qemu-system-x86-5155 [002] 1248198.490194: kvm_entry: vcpu 3 qemu-system-x86-5153 [003] 1248198.490319: kvm_exit: reason ext_irq rip 0xfffff80001d04a00 qemu-system-x86-5153 [003] 1248198.490319: kvm_entry: vcpu 1 qemu-system-x86-5153 [003] 1248198.490402: kvm_exit: reason ext_irq rip 0xfffff80001c9e859 qemu-system-x86-5153 [003] 1248198.490404: kvm_entry: vcpu 1 qemu-system-x86-5154 [001] 1248198.491067: kvm_exit: reason ext_irq rip 0xfffff80001d04a7c qemu-system-x86-5154 [001] 1248198.491068: kvm_entry: vcpu 2 qemu-system-x86-5146 [003] 1248198.491136: kvm_set_irq: gsi 0 level 1 source 0 qemu-system-x86-5146 [003] 1248198.491137: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-5146 [003] 1248198.491138: kvm_apic_accept_irq: apicid 0 vec 209 (Fixed|edge) qemu-system-x86-5146 [003] 1248198.491138: kvm_apic_accept_irq: apicid 1 vec 209 (Fixed|edge) qemu-system-x86-5146 [003] 1248198.491139: kvm_apic_accept_irq: apicid 2 vec 209 (Fixed|edge) qemu-system-x86-5146 [003] 1248198.491139: kvm_apic_accept_irq: apicid 3 vec 209 (Fixed|edge) qemu-system-x86-5146 [003] 1248198.491140: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-5146 [003] 1248198.491140: kvm_set_irq: gsi 0 level 0 source 0 qemu-system-x86-5146 [003] 1248198.491140: kvm_pic_set_irq: chip 0 pin 0 (edge|masked) qemu-system-x86-5154 [001] 1248198.491141: kvm_exit: reason ext_irq rip 0xfffff80001d049f1 qemu-system-x86-5146 [003] 1248198.491141: kvm_ioapic_set_irq: pin 2 dst ff vec=209 (Fixed|physical|edge) qemu-system-x86-5155 [002] 1248198.491141: kvm_exit: reason ext_irq rip 0xfffff80001d04a00 qemu-system-x86-5154 [001] 1248198.491141: kvm_inj_virq: irq 209 qemu-system-x86-5155 [002] 1248198.491141: kvm_inj_virq: irq 209 qemu-system-x86-5154 [001] 1248198.491142: kvm_entry: vcpu 2 qemu-system-x86-5155 [002] 1248198.491142: kvm_entry: vcpu 3 qemu-system-x86-5154 [001] 1248198.491143: kvm_exit: reason apic_access rip 0xfffff80001c9df13 qemu-system-x86-5155 [002] 1248198.491143: kvm_exit: reason apic_access rip 0xfffff80001c9df13 qemu-system-x86-5154 [001] 1248198.491145: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-5155 [002] 1248198.491145: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-5154 [001] 1248198.491145: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-5155 [002] 1248198.491145: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-5154 [001] 1248198.491146: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-5155 [002] 1248198.491146: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-5154 [001] 1248198.491147: kvm_entry: vcpu 2 qemu-system-x86-5155 [002] 1248198.491147: kvm_entry: vcpu 3 qemu-system-x86-5153 [003] 1248198.491148: kvm_exit: reason ext_irq rip 0xfffff80001ca2fbf qemu-system-x86-5153 [003] 1248198.491149: kvm_entry: vcpu 1 qemu-system-x86-5153 [003] 1248198.491151: kvm_exit: reason interrupt_window rip 0xfffff80001ca3002 qemu-system-x86-5153 [003] 1248198.491151: kvm_inj_virq: irq 209 qemu-system-x86-5153 [003] 1248198.491152: kvm_entry: vcpu 1 qemu-system-x86-5153 [003] 1248198.491153: kvm_exit: reason apic_access rip 0xfffff80001c9df13 qemu-system-x86-5153 [003] 1248198.491155: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-5153 [003] 1248198.491155: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-5153 [003] 1248198.491156: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-5153 [003] 1248198.491157: kvm_entry: vcpu 1 qemu-system-x86-5152 [000] 1248198.491187: kvm_exit: reason ext_irq rip 0xfffff80001ca2f40 qemu-system-x86-5152 [000] 1248198.491189: kvm_entry: vcpu 0 qemu-system-x86-5152 [000] 1248198.491191: kvm_exit: reason interrupt_window rip 0xfffff80001ca3002 qemu-system-x86-5152 [000] 1248198.491192: kvm_inj_virq: irq 209 qemu-system-x86-5152 [000] 1248198.491192: kvm_entry: vcpu 0 qemu-system-x86-5152 [000] 1248198.491194: kvm_exit: reason apic_access rip 0xfffff80001c9d5c4 qemu-system-x86-5152 [000] 1248198.491196: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-5152 [000] 1248198.491196: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-5152 [000] 1248198.491197: kvm_ack_irq: irqchip IOAPIC pin 2 qemu-system-x86-5152 [000] 1248198.491198: kvm_entry: vcpu 0 qemu-system-x86-5152 [000] 1248198.491203: kvm_exit: reason apic_access rip 0xfffff80001c14092 qemu-system-x86-5152 [000] 1248198.491205: kvm_apic: apic_read APIC_ICR = 0x82f qemu-system-x86-5152 [000] 1248198.491205: kvm_mmio: mmio read len 4 gpa 0xfee00300 val 0x82f qemu-system-x86-5152 [000] 1248198.491206: kvm_entry: vcpu 0 qemu-system-x86-5152 [000] 1248198.491207: kvm_exit: reason apic_access rip 0xfffff80001c140a9 qemu-system-x86-5152 [000] 1248198.491208: kvm_mmio: mmio write len 4 gpa 0xfee00310 val 0x2000000 qemu-system-x86-5152 [000] 1248198.491209: kvm_apic: apic_write APIC_ICR2 = 0x2000000 qemu-system-x86-5152 [000] 1248198.491209: kvm_entry: vcpu 0 qemu-system-x86-5152 [000] 1248198.491210: kvm_exit: reason apic_access rip 0xfffff80001c140b0 qemu-system-x86-5152 [000] 1248198.491212: kvm_mmio: mmio write len 4 gpa 0xfee00300 val 0x82f qemu-system-x86-5152 [000] 1248198.491212: kvm_apic: apic_write APIC_ICR = 0x82f qemu-system-x86-5152 [000] 1248198.491212: kvm_apic_ipi: dst 2 vec 47 (Fixed|logical|de-assert|edge|dst) qemu-system-x86-5152 [000] 1248198.491213: kvm_apic_accept_irq: apicid 1 vec 47 (Fixed|edge) qemu-system-x86-5152 [000] 1248198.491214: kvm_entry: vcpu 0 qemu-system-x86-5153 [003] 1248198.491215: kvm_exit: reason ext_irq rip 0xfffff80001ca2f40 qemu-system-x86-5153 [003] 1248198.491215: kvm_entry: vcpu 1 qemu-system-x86-5153 [003] 1248198.491216: kvm_exit: reason interrupt_window rip 0xfffff80001ca3002 qemu-system-x86-5153 [003] 1248198.491217: kvm_inj_virq: irq 47 qemu-system-x86-5153 [003] 1248198.491218: kvm_entry: vcpu 1 qemu-system-x86-5153 [003] 1248198.491219: kvm_exit: reason apic_access rip 0xfffff80001cedc67 qemu-system-x86-5153 [003] 1248198.491220: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-5153 [003] 1248198.491220: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-5153 [003] 1248198.491222: kvm_entry: vcpu 1 qemu-system-x86-5153 [003] 1248198.491224: kvm_exit: reason cr_access rip 0xfffff80001ca6ff6 qemu-system-x86-5153 [003] 1248198.491224: kvm_cr: cr_write 3 = 0x15f46d000 qemu-system-x86-5153 [003] 1248198.491228: kvm_entry: vcpu 1 qemu-system-x86-5153 [003] 1248198.491229: kvm_exit: reason exception rip 0xfffff80001ca702c qemu-system-x86-5153 [003] 1248198.491230: kvm_entry: vcpu 1 qemu-system-x86-5153 [003] 1248198.491238: kvm_exit: reason exception rip 0xfffff80001c0f11e qemu-system-x86-5153 [003] 1248198.491239: kvm_page_fault: address ffffffffffd070f0 error_code 9 qemu-system-x86-5153 [003] 1248198.491243: kvm_mmio: mmio unsatisfied-read len 4 gpa 0xfed000f0 val 0x0 qemu-system-x86-5153 [003] 1248198.491252: kvm_mmio: mmio read len 4 gpa 0xfed000f0 val 0x993c675d qemu-system-x86-5153 [003] 1248198.491253: kvm_entry: vcpu 1 qemu-system-x86-5153 [003] 1248198.491259: kvm_exit: reason cr_access rip 0xfffff80001ca6ff6 qemu-system-x86-5153 [003] 1248198.491260: kvm_cr: cr_write 3 = 0x164599000 qemu-system-x86-5153 [003] 1248198.491262: kvm_entry: vcpu 1 qemu-system-x86-5153 [003] 1248198.491263: kvm_exit: reason exception rip 0xfffff80001ca702c qemu-system-x86-5153 [003] 1248198.491263: kvm_entry: vcpu 1 qemu-system-x86-5153 [003] 1248198.491318: kvm_exit: reason ext_irq rip 0xfffff80001cbad28 qemu-system-x86-5153 [003] 1248198.491319: kvm_entry: vcpu 1 qemu-system-x86-5153 [003] 1248198.491404: kvm_exit: reason ext_irq rip 0xfffff80001cdc130 qemu-system-x86-5153 [003] 1248198.491405: kvm_entry: vcpu 1 ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Freezing Windows 2008 x64bit guest 2010-07-21 6:25 ` Harri Olin @ 2010-07-21 8:28 ` Christoph Adomeit 2010-07-21 8:37 ` Gleb Natapov 1 sibling, 0 replies; 22+ messages in thread From: Christoph Adomeit @ 2010-07-21 8:28 UTC (permalink / raw) To: Harri Olin; +Cc: Gleb Natapov, kvm@vger.kernel.org We also just had another freeze, although we meanwhile use the realtek driver instead of e1000. Our Windows Freeze behaves a little different, vm is totally unresponsive: no ping, no mouse pointer, nothing. I have just enabled nmi debugging and i hope the next crash and the corresponding dump might help us. On Wed, Jul 21, 2010 at 09:25:31AM +0300, Harri Olin wrote: > Gleb Natapov kirjoitti: >> On Mon, Jul 19, 2010 at 10:17:02AM +0300, Harri Olin wrote: >>> Gleb Natapov kirjoitti: >>>> On Thu, Jul 15, 2010 at 03:19:44PM +0200, Christoph Adomeit wrote: >>>>> But one Windows 2008 64 Bit Server Standard is freezing regularly. >>>>> This happens sometimes 3 times a day, sometimes it takes 2 days >>>>> until freeze. The Windows Machine is a clean fresh install. >>> I think I have seen same problem occur on my Windows 2008 SBS SP2 >>> 64bit system, but a bit less often, only like once a week. >>> Now I haven't seen crashes but only freezes with qemu on 100% and >>> virtual system unresponsive. >>> >>> qemu command line: >>> /usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive >>> file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive >>> file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net >>> nic,macaddr=C0:FF:12:FB:AA:01,model=e1000 -net tap -smp 4 -localtime >>> >>> >> Try with different model then e1000 please. Default driver that comes >> with Windows known to have problems. > > Didn't help, changed to default realtek emulation, but system freezed > again. Command line: > /usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive > file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive > file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net > nic,macaddr=C0:FF:12:FB:AA:01 -net tap -smp 4 -localtime > > This time the virtual system was not totally unresponsive somehow, > system pinged just fine and I think DNS server worked too. However on > console mouse moved but didn't react to clicking, ctrl-alt-del, etc. > >> >>>> When hang occurs ensure that problematic vm is the only on on the server and run >>>> kvm_stat. Send output here. Also do the following: >>>> >>>> # mount -t debugfs debugfs /sys/kernel/debug >>>> # echo kvm > /sys/kernel/debug/tracing/set_event >>>> # sleep 1 >>>> # cat /sys/kernel/debug/tracing/trace > /tmp/trace >>>> >>>> Send /tmp/trace here too, but it may be huge, so send only last 1000 >>>> lines. > > Stats from this hang: > log style http://mizar.remote.agasha.com/k/kvm/kvm_stat_2_log.txt > > kvm statistics > > efer_reload 0 0 > exits 5368720296 21258 > fpu_reload 2826750721 2216 > halt_exits 695286653 0 > halt_wakeup 685187661 0 > host_state_reload 3010399640 2216 > hypercalls 0 0 > insn_emulation 2765352032 6542 > insn_emulation_fail 477 0 > invlpg 42417110 204 > io_exits 818764026 122 > irq_exits 109659572 9870 > irq_injections 788992032 4643 > irq_window 22170277 1718 > largepages 0 0 > mmio_exits 1490912756 69 > mmu_cache_miss 2471963 0 > mmu_flooded 905690 0 > mmu_pde_zapped 1821325 0 > mmu_pte_updated 3256583 0 > mmu_pte_write 2896199 0 > mmu_recycled 259845 0 > > > full trace: http://mizar.remote.agasha.com/k/kvm/kvm_trace_2_full.txt > > qemu-system-x86-5155 [002] 1248198.487147: kvm_entry: vcpu 3 > qemu-system-x86-5152 [000] 1248198.487147: kvm_inj_virq: irq 209 > qemu-system-x86-5152 [000] 1248198.487148: kvm_entry: vcpu 0 > qemu-system-x86-5154 [001] 1248198.487148: kvm_exit: reason > apic_access rip 0xfffff80001c9df13 > qemu-system-x86-5155 [002] 1248198.487148: kvm_exit: reason > apic_access rip 0xfffff80001c9df13 > qemu-system-x86-5152 [000] 1248198.487150: kvm_exit: reason > apic_access rip 0xfffff80001c9d5c4 > qemu-system-x86-5154 [001] 1248198.487150: kvm_mmio: mmio write len 4 > gpa 0xfee000b0 val 0x0 > qemu-system-x86-5154 [001] 1248198.487150: kvm_apic: apic_write > APIC_EOI = 0x0 > qemu-system-x86-5155 [002] 1248198.487150: kvm_mmio: mmio write len 4 > gpa 0xfee000b0 val 0x0 > qemu-system-x86-5155 [002] 1248198.487151: kvm_apic: apic_write > APIC_EOI = 0x0 > qemu-system-x86-5154 [001] 1248198.487151: kvm_ack_irq: irqchip IOAPIC > pin 2 > qemu-system-x86-5155 [002] 1248198.487151: kvm_ack_irq: irqchip IOAPIC > pin 2 > qemu-system-x86-5152 [000] 1248198.487151: kvm_mmio: mmio write len 4 > gpa 0xfee000b0 val 0x0 > qemu-system-x86-5152 [000] 1248198.487152: kvm_apic: apic_write > APIC_EOI = 0x0 > qemu-system-x86-5154 [001] 1248198.487152: kvm_entry: vcpu 2 > qemu-system-x86-5155 [002] 1248198.487152: kvm_entry: vcpu 3 > qemu-system-x86-5152 [000] 1248198.487152: kvm_ack_irq: irqchip IOAPIC > pin 2 > qemu-system-x86-5153 [003] 1248198.487153: kvm_exit: reason ext_irq > rip 0xfffff80001d04a00 > qemu-system-x86-5152 [000] 1248198.487153: kvm_entry: vcpu 0 > qemu-system-x86-5153 [003] 1248198.487154: kvm_inj_virq: irq 209 > qemu-system-x86-5153 [003] 1248198.487155: kvm_entry: vcpu 1 > qemu-system-x86-5153 [003] 1248198.487157: kvm_exit: reason > apic_access rip 0xfffff80001c9df13 > qemu-system-x86-5153 [003] 1248198.487159: kvm_mmio: mmio write len 4 > gpa 0xfee000b0 val 0x0 > qemu-system-x86-5153 [003] 1248198.487159: kvm_apic: apic_write > APIC_EOI = 0x0 > qemu-system-x86-5153 [003] 1248198.487160: kvm_ack_irq: irqchip IOAPIC > pin 2 > qemu-system-x86-5153 [003] 1248198.487161: kvm_entry: vcpu 1 > qemu-system-x86-5155 [002] 1248198.487198: kvm_exit: reason ext_irq > rip 0xfffff80001ca42c3 > qemu-system-x86-5155 [002] 1248198.487199: kvm_entry: vcpu 3 > qemu-system-x86-5153 [003] 1248198.487324: kvm_exit: reason ext_irq > rip 0xfffff80001c9e859 > qemu-system-x86-5153 [003] 1248198.487324: kvm_entry: vcpu 1 > qemu-system-x86-5153 [003] 1248198.487407: kvm_exit: reason ext_irq > rip 0xfffff80001ca2f40 > qemu-system-x86-5153 [003] 1248198.487408: kvm_entry: vcpu 1 > qemu-system-x86-5152 [000] 1248198.487947: kvm_exit: reason ext_irq > rip 0xfffff80001d04a00 > qemu-system-x86-5152 [000] 1248198.487948: kvm_entry: vcpu 0 > qemu-system-x86-5154 [001] 1248198.488072: kvm_exit: reason ext_irq > rip 0xfffff80001c9e859 > qemu-system-x86-5154 [001] 1248198.488072: kvm_entry: vcpu 2 > qemu-system-x86-5146 [003] 1248198.488141: kvm_set_irq: gsi 0 level 1 > source 0 > qemu-system-x86-5146 [003] 1248198.488141: kvm_pic_set_irq: chip 0 pin > 0 (edge|masked) > qemu-system-x86-5146 [003] 1248198.488142: kvm_apic_accept_irq: apicid > 0 vec 209 (Fixed|edge) > qemu-system-x86-5146 [003] 1248198.488142: kvm_apic_accept_irq: apicid > 1 vec 209 (Fixed|edge) > qemu-system-x86-5146 [003] 1248198.488143: kvm_apic_accept_irq: apicid > 2 vec 209 (Fixed|edge) > qemu-system-x86-5146 [003] 1248198.488143: kvm_apic_accept_irq: apicid > 3 vec 209 (Fixed|edge) > qemu-system-x86-5152 [000] 1248198.488144: kvm_exit: reason ext_irq > rip 0xfffff80001ca2f40 > qemu-system-x86-5146 [003] 1248198.488144: kvm_ioapic_set_irq: pin 2 > dst ff vec=209 (Fixed|physical|edge) > qemu-system-x86-5146 [003] 1248198.488144: kvm_set_irq: gsi 0 level 0 > source 0 > qemu-system-x86-5152 [000] 1248198.488144: kvm_entry: vcpu 0 > qemu-system-x86-5154 [001] 1248198.488145: kvm_exit: reason ext_irq > rip 0xfffff80001d049f1 > qemu-system-x86-5146 [003] 1248198.488145: kvm_pic_set_irq: chip 0 pin > 0 (edge|masked) > qemu-system-x86-5146 [003] 1248198.488145: kvm_ioapic_set_irq: pin 2 > dst ff vec=209 (Fixed|physical|edge) > qemu-system-x86-5155 [002] 1248198.488145: kvm_exit: reason ext_irq > rip 0xfffff80001d04a00 > qemu-system-x86-5154 [001] 1248198.488145: kvm_inj_virq: irq 209 > qemu-system-x86-5155 [002] 1248198.488145: kvm_inj_virq: irq 209 > qemu-system-x86-5154 [001] 1248198.488146: kvm_entry: vcpu 2 > qemu-system-x86-5152 [000] 1248198.488146: kvm_exit: reason > interrupt_window rip 0xfffff80001ca3002 > qemu-system-x86-5155 [002] 1248198.488146: kvm_entry: vcpu 3 > qemu-system-x86-5152 [000] 1248198.488146: kvm_inj_virq: irq 209 > qemu-system-x86-5152 [000] 1248198.488147: kvm_entry: vcpu 0 > qemu-system-x86-5154 [001] 1248198.488147: kvm_exit: reason > apic_access rip 0xfffff80001c9df13 > qemu-system-x86-5155 [002] 1248198.488147: kvm_exit: reason > apic_access rip 0xfffff80001c9df13 > qemu-system-x86-5152 [000] 1248198.488148: kvm_exit: reason > apic_access rip 0xfffff80001c9d5c4 > qemu-system-x86-5154 [001] 1248198.488149: kvm_mmio: mmio write len 4 > gpa 0xfee000b0 val 0x0 > qemu-system-x86-5154 [001] 1248198.488149: kvm_apic: apic_write > APIC_EOI = 0x0 > qemu-system-x86-5155 [002] 1248198.488149: kvm_mmio: mmio write len 4 > gpa 0xfee000b0 val 0x0 > qemu-system-x86-5155 [002] 1248198.488149: kvm_apic: apic_write > APIC_EOI = 0x0 > qemu-system-x86-5154 [001] 1248198.488149: kvm_ack_irq: irqchip IOAPIC > pin 2 > qemu-system-x86-5155 [002] 1248198.488150: kvm_ack_irq: irqchip IOAPIC > pin 2 > qemu-system-x86-5152 [000] 1248198.488150: kvm_mmio: mmio write len 4 > gpa 0xfee000b0 val 0x0 > qemu-system-x86-5152 [000] 1248198.488150: kvm_apic: apic_write > APIC_EOI = 0x0 > qemu-system-x86-5152 [000] 1248198.488151: kvm_ack_irq: irqchip IOAPIC > pin 2 > qemu-system-x86-5154 [001] 1248198.488151: kvm_entry: vcpu 2 > qemu-system-x86-5155 [002] 1248198.488152: kvm_entry: vcpu 3 > qemu-system-x86-5152 [000] 1248198.488152: kvm_entry: vcpu 0 > qemu-system-x86-5153 [003] 1248198.488153: kvm_exit: reason ext_irq > rip 0xfffff80001d04a00 > qemu-system-x86-5153 [003] 1248198.488154: kvm_inj_virq: irq 209 > qemu-system-x86-5153 [003] 1248198.488154: kvm_entry: vcpu 1 > qemu-system-x86-5153 [003] 1248198.488156: kvm_exit: reason > apic_access rip 0xfffff80001c9df13 > qemu-system-x86-5153 [003] 1248198.488158: kvm_mmio: mmio write len 4 > gpa 0xfee000b0 val 0x0 > qemu-system-x86-5153 [003] 1248198.488158: kvm_apic: apic_write > APIC_EOI = 0x0 > qemu-system-x86-5153 [003] 1248198.488159: kvm_ack_irq: irqchip IOAPIC > pin 2 > qemu-system-x86-5153 [003] 1248198.488160: kvm_entry: vcpu 1 > qemu-system-x86-5155 [002] 1248198.488197: kvm_exit: reason ext_irq > rip 0xfffff80001ca2f40 > qemu-system-x86-5155 [002] 1248198.488197: kvm_entry: vcpu 3 > qemu-system-x86-5153 [003] 1248198.488322: kvm_exit: reason ext_irq > rip 0xfffff80001cba795 > qemu-system-x86-5153 [003] 1248198.488322: kvm_entry: vcpu 1 > qemu-system-x86-5153 [003] 1248198.488406: kvm_exit: reason ext_irq > rip 0xfffff80001d04a00 > qemu-system-x86-5153 [003] 1248198.488407: kvm_entry: vcpu 1 > qemu-system-x86-5152 [000] 1248198.488945: kvm_exit: reason ext_irq > rip 0xfffff80001cdc193 > qemu-system-x86-5152 [000] 1248198.488946: kvm_entry: vcpu 0 > qemu-system-x86-5154 [001] 1248198.489070: kvm_exit: reason ext_irq > rip 0xfffff80001ca2f40 > qemu-system-x86-5154 [001] 1248198.489071: kvm_entry: vcpu 2 > qemu-system-x86-5146 [003] 1248198.489139: kvm_set_irq: gsi 0 level 1 > source 0 > qemu-system-x86-5146 [003] 1248198.489139: kvm_pic_set_irq: chip 0 pin > 0 (edge|masked) > qemu-system-x86-5146 [003] 1248198.489140: kvm_apic_accept_irq: apicid > 0 vec 209 (Fixed|edge) > qemu-system-x86-5146 [003] 1248198.489140: kvm_apic_accept_irq: apicid > 1 vec 209 (Fixed|edge) > qemu-system-x86-5146 [003] 1248198.489141: kvm_apic_accept_irq: apicid > 2 vec 209 (Fixed|edge) > qemu-system-x86-5146 [003] 1248198.489141: kvm_apic_accept_irq: apicid > 3 vec 209 (Fixed|edge) > qemu-system-x86-5146 [003] 1248198.489142: kvm_ioapic_set_irq: pin 2 > dst ff vec=209 (Fixed|physical|edge) > qemu-system-x86-5152 [000] 1248198.489142: kvm_exit: reason ext_irq > rip 0xfffff80001d04a00 > qemu-system-x86-5146 [003] 1248198.489142: kvm_set_irq: gsi 0 level 0 > source 0 > qemu-system-x86-5146 [003] 1248198.489142: kvm_pic_set_irq: chip 0 pin > 0 (edge|masked) > qemu-system-x86-5152 [000] 1248198.489142: kvm_inj_virq: irq 209 > qemu-system-x86-5146 [003] 1248198.489143: kvm_ioapic_set_irq: pin 2 > dst ff vec=209 (Fixed|physical|edge) > qemu-system-x86-5154 [001] 1248198.489143: kvm_exit: reason ext_irq > rip 0xfffff80001d04a00 > qemu-system-x86-5155 [002] 1248198.489143: kvm_exit: reason ext_irq > rip 0xfffff80001d049e4 > qemu-system-x86-5152 [000] 1248198.489143: kvm_entry: vcpu 0 > qemu-system-x86-5154 [001] 1248198.489143: kvm_inj_virq: irq 209 > qemu-system-x86-5155 [002] 1248198.489143: kvm_inj_virq: irq 209 > qemu-system-x86-5154 [001] 1248198.489144: kvm_entry: vcpu 2 > qemu-system-x86-5155 [002] 1248198.489144: kvm_entry: vcpu 3 > qemu-system-x86-5152 [000] 1248198.489145: kvm_exit: reason > apic_access rip 0xfffff80001c9d5c4 > qemu-system-x86-5154 [001] 1248198.489145: kvm_exit: reason > apic_access rip 0xfffff80001c9df13 > qemu-system-x86-5155 [002] 1248198.489145: kvm_exit: reason > apic_access rip 0xfffff80001c9df13 > qemu-system-x86-5152 [000] 1248198.489146: kvm_mmio: mmio write len 4 > gpa 0xfee000b0 val 0x0 > qemu-system-x86-5152 [000] 1248198.489147: kvm_apic: apic_write > APIC_EOI = 0x0 > qemu-system-x86-5154 [001] 1248198.489147: kvm_mmio: mmio write len 4 > gpa 0xfee000b0 val 0x0 > qemu-system-x86-5154 [001] 1248198.489147: kvm_apic: apic_write > APIC_EOI = 0x0 > qemu-system-x86-5155 [002] 1248198.489147: kvm_mmio: mmio write len 4 > gpa 0xfee000b0 val 0x0 > qemu-system-x86-5152 [000] 1248198.489147: kvm_ack_irq: irqchip IOAPIC > pin 2 > qemu-system-x86-5155 [002] 1248198.489147: kvm_apic: apic_write > APIC_EOI = 0x0 > qemu-system-x86-5154 [001] 1248198.489147: kvm_ack_irq: irqchip IOAPIC > pin 2 > qemu-system-x86-5155 [002] 1248198.489149: kvm_ack_irq: irqchip IOAPIC > pin 2 > qemu-system-x86-5152 [000] 1248198.489149: kvm_entry: vcpu 0 > qemu-system-x86-5154 [001] 1248198.489150: kvm_entry: vcpu 2 > qemu-system-x86-5153 [003] 1248198.489150: kvm_exit: reason ext_irq > rip 0xfffff80001ca468f > qemu-system-x86-5155 [002] 1248198.489151: kvm_entry: vcpu 3 > qemu-system-x86-5153 [003] 1248198.489151: kvm_inj_virq: irq 209 > qemu-system-x86-5153 [003] 1248198.489152: kvm_entry: vcpu 1 > qemu-system-x86-5153 [003] 1248198.489153: kvm_exit: reason > apic_access rip 0xfffff80001c9df13 > qemu-system-x86-5153 [003] 1248198.489156: kvm_mmio: mmio write len 4 > gpa 0xfee000b0 val 0x0 > qemu-system-x86-5153 [003] 1248198.489156: kvm_apic: apic_write > APIC_EOI = 0x0 > qemu-system-x86-5153 [003] 1248198.489156: kvm_ack_irq: irqchip IOAPIC > pin 2 > qemu-system-x86-5153 [003] 1248198.489157: kvm_entry: vcpu 1 > qemu-system-x86-5155 [002] 1248198.489195: kvm_exit: reason ext_irq > rip 0xfffff80001ca2f40 > qemu-system-x86-5155 [002] 1248198.489195: kvm_entry: vcpu 3 > qemu-system-x86-5153 [003] 1248198.489321: kvm_exit: reason ext_irq > rip 0xfffff80001ca2f40 > qemu-system-x86-5154 [001] 1248198.489322: kvm_exit: reason ext_irq > rip 0xfffff80001c9e859 > qemu-system-x86-5153 [003] 1248198.489322: kvm_entry: vcpu 1 > qemu-system-x86-5154 [001] 1248198.489323: kvm_entry: vcpu 2 > qemu-system-x86-5153 [003] 1248198.489404: kvm_exit: reason ext_irq > rip 0xfffff80001d04a00 > qemu-system-x86-5153 [003] 1248198.489406: kvm_entry: vcpu 1 > qemu-system-x86-5152 [000] 1248198.489944: kvm_exit: reason ext_irq > rip 0xfffff80001d04a00 > qemu-system-x86-5152 [000] 1248198.489945: kvm_entry: vcpu 0 > qemu-system-x86-5154 [001] 1248198.490069: kvm_exit: reason ext_irq > rip 0xfffff80001cdd795 > qemu-system-x86-5154 [001] 1248198.490069: kvm_entry: vcpu 2 > qemu-system-x86-5146 [003] 1248198.490137: kvm_set_irq: gsi 0 level 1 > source 0 > qemu-system-x86-5146 [003] 1248198.490137: kvm_pic_set_irq: chip 0 pin > 0 (edge|masked) > qemu-system-x86-5146 [003] 1248198.490138: kvm_apic_accept_irq: apicid > 0 vec 209 (Fixed|edge) > qemu-system-x86-5146 [003] 1248198.490139: kvm_apic_accept_irq: apicid > 1 vec 209 (Fixed|edge) > qemu-system-x86-5146 [003] 1248198.490139: kvm_apic_accept_irq: apicid > 2 vec 209 (Fixed|edge) > qemu-system-x86-5146 [003] 1248198.490140: kvm_apic_accept_irq: apicid > 3 vec 209 (Fixed|edge) > qemu-system-x86-5146 [003] 1248198.490140: kvm_ioapic_set_irq: pin 2 > dst ff vec=209 (Fixed|physical|edge) > qemu-system-x86-5152 [000] 1248198.490140: kvm_exit: reason ext_irq > rip 0xfffff80001d04a00 > qemu-system-x86-5146 [003] 1248198.490140: kvm_set_irq: gsi 0 level 0 > source 0 > qemu-system-x86-5152 [000] 1248198.490141: kvm_inj_virq: irq 209 > qemu-system-x86-5146 [003] 1248198.490141: kvm_pic_set_irq: chip 0 pin > 0 (edge|masked) > qemu-system-x86-5146 [003] 1248198.490141: kvm_ioapic_set_irq: pin 2 > dst ff vec=209 (Fixed|physical|edge) > qemu-system-x86-5154 [001] 1248198.490141: kvm_exit: reason ext_irq > rip 0xfffff80001d04a00 > qemu-system-x86-5155 [002] 1248198.490141: kvm_exit: reason ext_irq > rip 0xfffff80001ca2fbf > qemu-system-x86-5152 [000] 1248198.490141: kvm_entry: vcpu 0 > qemu-system-x86-5154 [001] 1248198.490142: kvm_inj_virq: irq 209 > qemu-system-x86-5155 [002] 1248198.490142: kvm_entry: vcpu 3 > qemu-system-x86-5154 [001] 1248198.490142: kvm_entry: vcpu 2 > qemu-system-x86-5152 [000] 1248198.490143: kvm_exit: reason > apic_access rip 0xfffff80001c9d5c4 > qemu-system-x86-5155 [002] 1248198.490143: kvm_exit: reason > interrupt_window rip 0xfffff80001ca3002 > qemu-system-x86-5154 [001] 1248198.490143: kvm_exit: reason > apic_access rip 0xfffff80001c9df13 > qemu-system-x86-5155 [002] 1248198.490144: kvm_inj_virq: irq 209 > qemu-system-x86-5155 [002] 1248198.490144: kvm_entry: vcpu 3 > qemu-system-x86-5152 [000] 1248198.490144: kvm_mmio: mmio write len 4 > gpa 0xfee000b0 val 0x0 > qemu-system-x86-5152 [000] 1248198.490145: kvm_apic: apic_write > APIC_EOI = 0x0 > qemu-system-x86-5152 [000] 1248198.490145: kvm_ack_irq: irqchip IOAPIC > pin 2 > qemu-system-x86-5154 [001] 1248198.490145: kvm_mmio: mmio write len 4 > gpa 0xfee000b0 val 0x0 > qemu-system-x86-5154 [001] 1248198.490146: kvm_apic: apic_write > APIC_EOI = 0x0 > qemu-system-x86-5155 [002] 1248198.490146: kvm_exit: reason > apic_access rip 0xfffff80001c9df13 > qemu-system-x86-5154 [001] 1248198.490146: kvm_ack_irq: irqchip IOAPIC > pin 2 > qemu-system-x86-5152 [000] 1248198.490146: kvm_entry: vcpu 0 > qemu-system-x86-5154 [001] 1248198.490147: kvm_entry: vcpu 2 > qemu-system-x86-5155 [002] 1248198.490147: kvm_mmio: mmio write len 4 > gpa 0xfee000b0 val 0x0 > qemu-system-x86-5155 [002] 1248198.490148: kvm_apic: apic_write > APIC_EOI = 0x0 > qemu-system-x86-5155 [002] 1248198.490148: kvm_ack_irq: irqchip IOAPIC > pin 2 > qemu-system-x86-5153 [003] 1248198.490149: kvm_exit: reason ext_irq > rip 0xfffff80001d04a00 > qemu-system-x86-5155 [002] 1248198.490149: kvm_entry: vcpu 3 > qemu-system-x86-5153 [003] 1248198.490150: kvm_inj_virq: irq 209 > qemu-system-x86-5153 [003] 1248198.490150: kvm_entry: vcpu 1 > qemu-system-x86-5153 [003] 1248198.490152: kvm_exit: reason > apic_access rip 0xfffff80001c9df13 > qemu-system-x86-5153 [003] 1248198.490154: kvm_mmio: mmio write len 4 > gpa 0xfee000b0 val 0x0 > qemu-system-x86-5153 [003] 1248198.490154: kvm_apic: apic_write > APIC_EOI = 0x0 > qemu-system-x86-5153 [003] 1248198.490154: kvm_ack_irq: irqchip IOAPIC > pin 2 > qemu-system-x86-5153 [003] 1248198.490155: kvm_entry: vcpu 1 > qemu-system-x86-5155 [002] 1248198.490193: kvm_exit: reason ext_irq > rip 0xfffff80001ca2f40 > qemu-system-x86-5155 [002] 1248198.490194: kvm_entry: vcpu 3 > qemu-system-x86-5153 [003] 1248198.490319: kvm_exit: reason ext_irq > rip 0xfffff80001d04a00 > qemu-system-x86-5153 [003] 1248198.490319: kvm_entry: vcpu 1 > qemu-system-x86-5153 [003] 1248198.490402: kvm_exit: reason ext_irq > rip 0xfffff80001c9e859 > qemu-system-x86-5153 [003] 1248198.490404: kvm_entry: vcpu 1 > qemu-system-x86-5154 [001] 1248198.491067: kvm_exit: reason ext_irq > rip 0xfffff80001d04a7c > qemu-system-x86-5154 [001] 1248198.491068: kvm_entry: vcpu 2 > qemu-system-x86-5146 [003] 1248198.491136: kvm_set_irq: gsi 0 level 1 > source 0 > qemu-system-x86-5146 [003] 1248198.491137: kvm_pic_set_irq: chip 0 pin > 0 (edge|masked) > qemu-system-x86-5146 [003] 1248198.491138: kvm_apic_accept_irq: apicid > 0 vec 209 (Fixed|edge) > qemu-system-x86-5146 [003] 1248198.491138: kvm_apic_accept_irq: apicid > 1 vec 209 (Fixed|edge) > qemu-system-x86-5146 [003] 1248198.491139: kvm_apic_accept_irq: apicid > 2 vec 209 (Fixed|edge) > qemu-system-x86-5146 [003] 1248198.491139: kvm_apic_accept_irq: apicid > 3 vec 209 (Fixed|edge) > qemu-system-x86-5146 [003] 1248198.491140: kvm_ioapic_set_irq: pin 2 > dst ff vec=209 (Fixed|physical|edge) > qemu-system-x86-5146 [003] 1248198.491140: kvm_set_irq: gsi 0 level 0 > source 0 > qemu-system-x86-5146 [003] 1248198.491140: kvm_pic_set_irq: chip 0 pin > 0 (edge|masked) > qemu-system-x86-5154 [001] 1248198.491141: kvm_exit: reason ext_irq > rip 0xfffff80001d049f1 > qemu-system-x86-5146 [003] 1248198.491141: kvm_ioapic_set_irq: pin 2 > dst ff vec=209 (Fixed|physical|edge) > qemu-system-x86-5155 [002] 1248198.491141: kvm_exit: reason ext_irq > rip 0xfffff80001d04a00 > qemu-system-x86-5154 [001] 1248198.491141: kvm_inj_virq: irq 209 > qemu-system-x86-5155 [002] 1248198.491141: kvm_inj_virq: irq 209 > qemu-system-x86-5154 [001] 1248198.491142: kvm_entry: vcpu 2 > qemu-system-x86-5155 [002] 1248198.491142: kvm_entry: vcpu 3 > qemu-system-x86-5154 [001] 1248198.491143: kvm_exit: reason > apic_access rip 0xfffff80001c9df13 > qemu-system-x86-5155 [002] 1248198.491143: kvm_exit: reason > apic_access rip 0xfffff80001c9df13 > qemu-system-x86-5154 [001] 1248198.491145: kvm_mmio: mmio write len 4 > gpa 0xfee000b0 val 0x0 > qemu-system-x86-5155 [002] 1248198.491145: kvm_mmio: mmio write len 4 > gpa 0xfee000b0 val 0x0 > qemu-system-x86-5154 [001] 1248198.491145: kvm_apic: apic_write > APIC_EOI = 0x0 > qemu-system-x86-5155 [002] 1248198.491145: kvm_apic: apic_write > APIC_EOI = 0x0 > qemu-system-x86-5154 [001] 1248198.491146: kvm_ack_irq: irqchip IOAPIC > pin 2 > qemu-system-x86-5155 [002] 1248198.491146: kvm_ack_irq: irqchip IOAPIC > pin 2 > qemu-system-x86-5154 [001] 1248198.491147: kvm_entry: vcpu 2 > qemu-system-x86-5155 [002] 1248198.491147: kvm_entry: vcpu 3 > qemu-system-x86-5153 [003] 1248198.491148: kvm_exit: reason ext_irq > rip 0xfffff80001ca2fbf > qemu-system-x86-5153 [003] 1248198.491149: kvm_entry: vcpu 1 > qemu-system-x86-5153 [003] 1248198.491151: kvm_exit: reason > interrupt_window rip 0xfffff80001ca3002 > qemu-system-x86-5153 [003] 1248198.491151: kvm_inj_virq: irq 209 > qemu-system-x86-5153 [003] 1248198.491152: kvm_entry: vcpu 1 > qemu-system-x86-5153 [003] 1248198.491153: kvm_exit: reason > apic_access rip 0xfffff80001c9df13 > qemu-system-x86-5153 [003] 1248198.491155: kvm_mmio: mmio write len 4 > gpa 0xfee000b0 val 0x0 > qemu-system-x86-5153 [003] 1248198.491155: kvm_apic: apic_write > APIC_EOI = 0x0 > qemu-system-x86-5153 [003] 1248198.491156: kvm_ack_irq: irqchip IOAPIC > pin 2 > qemu-system-x86-5153 [003] 1248198.491157: kvm_entry: vcpu 1 > qemu-system-x86-5152 [000] 1248198.491187: kvm_exit: reason ext_irq > rip 0xfffff80001ca2f40 > qemu-system-x86-5152 [000] 1248198.491189: kvm_entry: vcpu 0 > qemu-system-x86-5152 [000] 1248198.491191: kvm_exit: reason > interrupt_window rip 0xfffff80001ca3002 > qemu-system-x86-5152 [000] 1248198.491192: kvm_inj_virq: irq 209 > qemu-system-x86-5152 [000] 1248198.491192: kvm_entry: vcpu 0 > qemu-system-x86-5152 [000] 1248198.491194: kvm_exit: reason > apic_access rip 0xfffff80001c9d5c4 > qemu-system-x86-5152 [000] 1248198.491196: kvm_mmio: mmio write len 4 > gpa 0xfee000b0 val 0x0 > qemu-system-x86-5152 [000] 1248198.491196: kvm_apic: apic_write > APIC_EOI = 0x0 > qemu-system-x86-5152 [000] 1248198.491197: kvm_ack_irq: irqchip IOAPIC > pin 2 > qemu-system-x86-5152 [000] 1248198.491198: kvm_entry: vcpu 0 > qemu-system-x86-5152 [000] 1248198.491203: kvm_exit: reason > apic_access rip 0xfffff80001c14092 > qemu-system-x86-5152 [000] 1248198.491205: kvm_apic: apic_read > APIC_ICR = 0x82f > qemu-system-x86-5152 [000] 1248198.491205: kvm_mmio: mmio read len 4 > gpa 0xfee00300 val 0x82f > qemu-system-x86-5152 [000] 1248198.491206: kvm_entry: vcpu 0 > qemu-system-x86-5152 [000] 1248198.491207: kvm_exit: reason > apic_access rip 0xfffff80001c140a9 > qemu-system-x86-5152 [000] 1248198.491208: kvm_mmio: mmio write len 4 > gpa 0xfee00310 val 0x2000000 > qemu-system-x86-5152 [000] 1248198.491209: kvm_apic: apic_write > APIC_ICR2 = 0x2000000 > qemu-system-x86-5152 [000] 1248198.491209: kvm_entry: vcpu 0 > qemu-system-x86-5152 [000] 1248198.491210: kvm_exit: reason > apic_access rip 0xfffff80001c140b0 > qemu-system-x86-5152 [000] 1248198.491212: kvm_mmio: mmio write len 4 > gpa 0xfee00300 val 0x82f > qemu-system-x86-5152 [000] 1248198.491212: kvm_apic: apic_write > APIC_ICR = 0x82f > qemu-system-x86-5152 [000] 1248198.491212: kvm_apic_ipi: dst 2 vec 47 > (Fixed|logical|de-assert|edge|dst) > qemu-system-x86-5152 [000] 1248198.491213: kvm_apic_accept_irq: apicid > 1 vec 47 (Fixed|edge) > qemu-system-x86-5152 [000] 1248198.491214: kvm_entry: vcpu 0 > qemu-system-x86-5153 [003] 1248198.491215: kvm_exit: reason ext_irq > rip 0xfffff80001ca2f40 > qemu-system-x86-5153 [003] 1248198.491215: kvm_entry: vcpu 1 > qemu-system-x86-5153 [003] 1248198.491216: kvm_exit: reason > interrupt_window rip 0xfffff80001ca3002 > qemu-system-x86-5153 [003] 1248198.491217: kvm_inj_virq: irq 47 > qemu-system-x86-5153 [003] 1248198.491218: kvm_entry: vcpu 1 > qemu-system-x86-5153 [003] 1248198.491219: kvm_exit: reason > apic_access rip 0xfffff80001cedc67 > qemu-system-x86-5153 [003] 1248198.491220: kvm_mmio: mmio write len 4 > gpa 0xfee000b0 val 0x0 > qemu-system-x86-5153 [003] 1248198.491220: kvm_apic: apic_write > APIC_EOI = 0x0 > qemu-system-x86-5153 [003] 1248198.491222: kvm_entry: vcpu 1 > qemu-system-x86-5153 [003] 1248198.491224: kvm_exit: reason cr_access > rip 0xfffff80001ca6ff6 > qemu-system-x86-5153 [003] 1248198.491224: kvm_cr: cr_write 3 = > 0x15f46d000 > qemu-system-x86-5153 [003] 1248198.491228: kvm_entry: vcpu 1 > qemu-system-x86-5153 [003] 1248198.491229: kvm_exit: reason exception > rip 0xfffff80001ca702c > qemu-system-x86-5153 [003] 1248198.491230: kvm_entry: vcpu 1 > qemu-system-x86-5153 [003] 1248198.491238: kvm_exit: reason exception > rip 0xfffff80001c0f11e > qemu-system-x86-5153 [003] 1248198.491239: kvm_page_fault: address > ffffffffffd070f0 error_code 9 > qemu-system-x86-5153 [003] 1248198.491243: kvm_mmio: mmio > unsatisfied-read len 4 gpa 0xfed000f0 val 0x0 > qemu-system-x86-5153 [003] 1248198.491252: kvm_mmio: mmio read len 4 > gpa 0xfed000f0 val 0x993c675d > qemu-system-x86-5153 [003] 1248198.491253: kvm_entry: vcpu 1 > qemu-system-x86-5153 [003] 1248198.491259: kvm_exit: reason cr_access > rip 0xfffff80001ca6ff6 > qemu-system-x86-5153 [003] 1248198.491260: kvm_cr: cr_write 3 = > 0x164599000 > qemu-system-x86-5153 [003] 1248198.491262: kvm_entry: vcpu 1 > qemu-system-x86-5153 [003] 1248198.491263: kvm_exit: reason exception > rip 0xfffff80001ca702c > qemu-system-x86-5153 [003] 1248198.491263: kvm_entry: vcpu 1 > qemu-system-x86-5153 [003] 1248198.491318: kvm_exit: reason ext_irq > rip 0xfffff80001cbad28 > qemu-system-x86-5153 [003] 1248198.491319: kvm_entry: vcpu 1 > qemu-system-x86-5153 [003] 1248198.491404: kvm_exit: reason ext_irq > rip 0xfffff80001cdc130 > qemu-system-x86-5153 [003] 1248198.491405: kvm_entry: vcpu 1 > > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Christoph Adomeit GATWORKS GmbH Reststrauch 191 41199 Moenchengladbach Sitz: Moenchengladbach Amtsgericht Moenchengladbach, HRB 6303 Geschaeftsfuehrer: Christoph Adomeit, Hans Wilhelm Terstappen Christoph.Adomeit@gatworks.de Internetloesungen vom Feinsten Fon. +49 2166 9149-32 Fax. +49 2166 9149-10 ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Freezing Windows 2008 x64bit guest 2010-07-21 6:25 ` Harri Olin 2010-07-21 8:28 ` Christoph Adomeit @ 2010-07-21 8:37 ` Gleb Natapov 2010-07-21 9:22 ` Harri Olin 2010-07-27 21:53 ` Harri Olin 1 sibling, 2 replies; 22+ messages in thread From: Gleb Natapov @ 2010-07-21 8:37 UTC (permalink / raw) To: Harri Olin; +Cc: Christoph Adomeit, kvm@vger.kernel.org On Wed, Jul 21, 2010 at 09:25:31AM +0300, Harri Olin wrote: > Gleb Natapov kirjoitti: > >On Mon, Jul 19, 2010 at 10:17:02AM +0300, Harri Olin wrote: > >>Gleb Natapov kirjoitti: > >>>On Thu, Jul 15, 2010 at 03:19:44PM +0200, Christoph Adomeit wrote: > >>>>But one Windows 2008 64 Bit Server Standard is freezing regularly. > >>>>This happens sometimes 3 times a day, sometimes it takes 2 days > >>>>until freeze. The Windows Machine is a clean fresh install. > >>I think I have seen same problem occur on my Windows 2008 SBS SP2 > >>64bit system, but a bit less often, only like once a week. > >>Now I haven't seen crashes but only freezes with qemu on 100% and > >>virtual system unresponsive. > >> > >>qemu command line: > >> /usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive > >>file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive > >>file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net > >>nic,macaddr=C0:FF:12:FB:AA:01,model=e1000 -net tap -smp 4 -localtime > >> > >> > >Try with different model then e1000 please. Default driver that comes > >with Windows known to have problems. > > Didn't help, changed to default realtek emulation, but system > freezed again. Command line: > /usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive > file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive > file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net > nic,macaddr=C0:FF:12:FB:AA:01 -net tap -smp 4 -localtime > > This time the virtual system was not totally unresponsive somehow, > system pinged just fine and I think DNS server worked too. However > on console mouse moved but didn't react to clicking, ctrl-alt-del, > etc. > Does sendkey from monitor works? qemu-kvm-0.11.1 is very old and this is not total freeze which even harder to debug. I don't see anything extraordinary in your logs. 4643 interrupt per second for 4 cpus is normal if windows runs multimedia or other app that need hi-res timers. Does your host swapping? Is there any chance that you can try upstream qemu-kvm? -- Gleb. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Freezing Windows 2008 x64bit guest 2010-07-21 8:37 ` Gleb Natapov @ 2010-07-21 9:22 ` Harri Olin 2010-07-21 9:48 ` Gleb Natapov 2010-07-27 21:53 ` Harri Olin 1 sibling, 1 reply; 22+ messages in thread From: Harri Olin @ 2010-07-21 9:22 UTC (permalink / raw) To: Gleb Natapov; +Cc: Christoph Adomeit, kvm@vger.kernel.org Gleb Natapov wrote: > On Wed, Jul 21, 2010 at 09:25:31AM +0300, Harri Olin wrote: >> Gleb Natapov kirjoitti: >>> On Mon, Jul 19, 2010 at 10:17:02AM +0300, Harri Olin wrote: >>>> Gleb Natapov kirjoitti: >>>>> On Thu, Jul 15, 2010 at 03:19:44PM +0200, Christoph Adomeit wrote: >>>>>> But one Windows 2008 64 Bit Server Standard is freezing regularly. >>>>>> This happens sometimes 3 times a day, sometimes it takes 2 days >>>>>> until freeze. The Windows Machine is a clean fresh install. >>>> I think I have seen same problem occur on my Windows 2008 SBS SP2 >>>> 64bit system, but a bit less often, only like once a week. >>>> Now I haven't seen crashes but only freezes with qemu on 100% and >>>> virtual system unresponsive. >>>> >>>> qemu command line: >>>> /usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive >>>> file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive >>>> file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net >>>> nic,macaddr=C0:FF:12:FB:AA:01,model=e1000 -net tap -smp 4 -localtime >>>> >>>> >>> Try with different model then e1000 please. Default driver that comes >>> with Windows known to have problems. >> Didn't help, changed to default realtek emulation, but system >> freezed again. Command line: >> /usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive >> file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive >> file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net >> nic,macaddr=C0:FF:12:FB:AA:01 -net tap -smp 4 -localtime >> >> This time the virtual system was not totally unresponsive somehow, >> system pinged just fine and I think DNS server worked too. However >> on console mouse moved but didn't react to clicking, ctrl-alt-del, >> etc. >> > Does sendkey from monitor works? qemu-kvm-0.11.1 is very old and this is > not total freeze which even harder to debug. I don't see anything > extraordinary in your logs. 4643 interrupt per second for 4 cpus is > normal if windows runs multimedia or other app that need hi-res timers. > Does your host swapping? Is there any chance that you can try upstream qemu-kvm? > I haven't tested sendkey. I originally tried 0.12.2 or so but with that windows crashed more often with bluescreen, I went down to 0.11.1. This might be related to -td-rtc-hack and/or e1000 as using td-rtc-hack on 0.11.1 windows crashed too to bluescreen. System does not run any multimedia or so, on top of normal windows 2008 small business server installation there should be only nagios client and apache. Only thing that catches my eye on stats frozen vs normal is irq_exits which jumps from 200 to 10000 when frozen and irq_window which goes 10x. Host has swap partition enabled but hasn't swapped once yet, currently there's over 1GB of free. I'll update host kernel to latest stable and qemu-kvm to upstream later today. Here's stats from running system: efer_reload 0 0 exits 386474714 23272 fpu_reload 173420913 12775 halt_exits 51894921 4294 halt_wakeup 51039254 4223 host_state_reload 188645310 13420 hypercalls 0 0 insn_emulation 212729430 15545 insn_emulation_fail 324 0 invlpg 5177828 134 io_exits 23132581 161 irq_exits 7485833 225 irq_injections 59239306 4655 irq_window 2147438 153 largepages 0 0 mmio_exits 110931423 9026 mmu_cache_miss 468410 0 mmu_flooded 193153 0 mmu_pde_zapped 352178 0 mmu_pte_updated 678023 0 mmu_pte_write 587175 0 mmu_recycled 34998 0 mmu_shadow_zapped 454073 0 mmu_unsync 19084 0 nmi_injections 0 0 nmi_window 0 0 pf_fixed 25171943 120 pf_guest 6209034 2 remote_tlb_flush 3044169 23 request_irq 0 0 signal_exits 0 0 tlb_flush 8422950 149 -- Harri. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Freezing Windows 2008 x64bit guest 2010-07-21 9:22 ` Harri Olin @ 2010-07-21 9:48 ` Gleb Natapov 2010-07-21 10:09 ` Harri Olin 0 siblings, 1 reply; 22+ messages in thread From: Gleb Natapov @ 2010-07-21 9:48 UTC (permalink / raw) To: Harri Olin; +Cc: Christoph Adomeit, kvm@vger.kernel.org On Wed, Jul 21, 2010 at 12:22:45PM +0300, Harri Olin wrote: > Gleb Natapov wrote: > >On Wed, Jul 21, 2010 at 09:25:31AM +0300, Harri Olin wrote: > >>Gleb Natapov kirjoitti: > >>>On Mon, Jul 19, 2010 at 10:17:02AM +0300, Harri Olin wrote: > >>>>Gleb Natapov kirjoitti: > >>>>>On Thu, Jul 15, 2010 at 03:19:44PM +0200, Christoph Adomeit wrote: > >>>>>>But one Windows 2008 64 Bit Server Standard is freezing regularly. > >>>>>>This happens sometimes 3 times a day, sometimes it takes 2 days > >>>>>>until freeze. The Windows Machine is a clean fresh install. > >>>>I think I have seen same problem occur on my Windows 2008 SBS SP2 > >>>>64bit system, but a bit less often, only like once a week. > >>>>Now I haven't seen crashes but only freezes with qemu on 100% and > >>>>virtual system unresponsive. > >>>> > >>>>qemu command line: > >>>> /usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive > >>>>file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive > >>>>file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net > >>>>nic,macaddr=C0:FF:12:FB:AA:01,model=e1000 -net tap -smp 4 -localtime > >>>> > >>>> > >>>Try with different model then e1000 please. Default driver that comes > >>>with Windows known to have problems. > >>Didn't help, changed to default realtek emulation, but system > >>freezed again. Command line: > >>/usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive > >>file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive > >>file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net > >>nic,macaddr=C0:FF:12:FB:AA:01 -net tap -smp 4 -localtime > >> > >>This time the virtual system was not totally unresponsive somehow, > >>system pinged just fine and I think DNS server worked too. However > >>on console mouse moved but didn't react to clicking, ctrl-alt-del, > >>etc. > >> > >Does sendkey from monitor works? qemu-kvm-0.11.1 is very old and this is > >not total freeze which even harder to debug. I don't see anything > >extraordinary in your logs. 4643 interrupt per second for 4 cpus is > >normal if windows runs multimedia or other app that need hi-res timers. > >Does your host swapping? Is there any chance that you can try upstream qemu-kvm? > > > > I haven't tested sendkey. I originally tried 0.12.2 or so but with > that windows crashed more often with bluescreen, I went down to > 0.11.1. This might be related to -td-rtc-hack and/or e1000 as using > td-rtc-hack on 0.11.1 windows crashed too to bluescreen. > With your guest -td-rtc-hack does nothing anyway, so you can just drop it. Although it shouldn't cause BSOD with upstream (it had such bug in the past and I am not sure that 0.12 has the fix). And you are using HPET anyway so -td-rtc-hack should be nop for you. > System does not run any multimedia or so, on top of normal windows > 2008 small business server installation there should be only nagios > client and apache. Any application that uses hi-res timers will cause Windows to switch to 1024Hz timer rate. > > Only thing that catches my eye on stats frozen vs normal is > irq_exits which jumps from 200 to 10000 when frozen and irq_window > which goes 10x. Hm, default Windows timer rate is 64Hz, so if nothing happens in the guest you should see at least 256 irq injection per second. In your KVM start output I only see 4655 irq injection per second not 10000. And trace shows that most of them are timer interrupts (vector 209 IIRC). > > Host has swap partition enabled but hasn't swapped once yet, > currently there's over 1GB of free. And how many cpus does it have? BTW I see a lot of instruction emulations. That is strange. Looking at you trace again I see that the bulk of the emulations are caused by mmio to HPET. Please run with -no-hpet flag. > > I'll update host kernel to latest stable and qemu-kvm to upstream > later today. > > Here's stats from running system: > efer_reload 0 0 > exits 386474714 23272 > fpu_reload 173420913 12775 > halt_exits 51894921 4294 > halt_wakeup 51039254 4223 > host_state_reload 188645310 13420 > hypercalls 0 0 > insn_emulation 212729430 15545 > insn_emulation_fail 324 0 > invlpg 5177828 134 > io_exits 23132581 161 > irq_exits 7485833 225 > irq_injections 59239306 4655 > irq_window 2147438 153 > largepages 0 0 > mmio_exits 110931423 9026 > mmu_cache_miss 468410 0 > mmu_flooded 193153 0 > mmu_pde_zapped 352178 0 > mmu_pte_updated 678023 0 > mmu_pte_write 587175 0 > mmu_recycled 34998 0 > mmu_shadow_zapped 454073 0 > mmu_unsync 19084 0 > nmi_injections 0 0 > nmi_window 0 0 > pf_fixed 25171943 120 > pf_guest 6209034 2 > remote_tlb_flush 3044169 23 > request_irq 0 0 > signal_exits 0 0 > tlb_flush 8422950 149 > > > -- > Harri. -- Gleb. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Freezing Windows 2008 x64bit guest 2010-07-21 9:48 ` Gleb Natapov @ 2010-07-21 10:09 ` Harri Olin 2010-07-21 10:30 ` Gleb Natapov 0 siblings, 1 reply; 22+ messages in thread From: Harri Olin @ 2010-07-21 10:09 UTC (permalink / raw) To: Gleb Natapov; +Cc: Christoph Adomeit, kvm@vger.kernel.org Gleb Natapov wrote: > On Wed, Jul 21, 2010 at 12:22:45PM +0300, Harri Olin wrote: >> Gleb Natapov wrote: >>> On Wed, Jul 21, 2010 at 09:25:31AM +0300, Harri Olin wrote: >>>> Gleb Natapov kirjoitti: >>>>> On Mon, Jul 19, 2010 at 10:17:02AM +0300, Harri Olin wrote: >>>>>> Gleb Natapov kirjoitti: >>>>>>> On Thu, Jul 15, 2010 at 03:19:44PM +0200, Christoph Adomeit wrote: >>>>>>>> But one Windows 2008 64 Bit Server Standard is freezing regularly. >>>>>>>> This happens sometimes 3 times a day, sometimes it takes 2 days >>>>>>>> until freeze. The Windows Machine is a clean fresh install. >>>>>> I think I have seen same problem occur on my Windows 2008 SBS SP2 >>>>>> 64bit system, but a bit less often, only like once a week. >>>>>> Now I haven't seen crashes but only freezes with qemu on 100% and >>>>>> virtual system unresponsive. >>>>>> >>>>>> qemu command line: >>>>>> /usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive >>>>>> file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive >>>>>> file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net >>>>>> nic,macaddr=C0:FF:12:FB:AA:01,model=e1000 -net tap -smp 4 -localtime >>>>>> >>>>>> >>>>> Try with different model then e1000 please. Default driver that comes >>>>> with Windows known to have problems. >>>> Didn't help, changed to default realtek emulation, but system >>>> freezed again. Command line: >>>> /usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive >>>> file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive >>>> file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net >>>> nic,macaddr=C0:FF:12:FB:AA:01 -net tap -smp 4 -localtime >>>> >>>> This time the virtual system was not totally unresponsive somehow, >>>> system pinged just fine and I think DNS server worked too. However >>>> on console mouse moved but didn't react to clicking, ctrl-alt-del, >>>> etc. >>>> >>> Does sendkey from monitor works? qemu-kvm-0.11.1 is very old and this is >>> not total freeze which even harder to debug. I don't see anything >>> extraordinary in your logs. 4643 interrupt per second for 4 cpus is >>> normal if windows runs multimedia or other app that need hi-res timers. >>> Does your host swapping? Is there any chance that you can try upstream qemu-kvm? >>> >> I haven't tested sendkey. I originally tried 0.12.2 or so but with >> that windows crashed more often with bluescreen, I went down to >> 0.11.1. This might be related to -td-rtc-hack and/or e1000 as using >> td-rtc-hack on 0.11.1 windows crashed too to bluescreen. >> > With your guest -td-rtc-hack does nothing anyway, so you can just drop > it. Although it shouldn't cause BSOD with upstream (it had such bug in > the past and I am not sure that 0.12 has the fix). And you are using > HPET anyway so -td-rtc-hack should be nop for you. Ok, I think I added it because windows guest clock lagged behind a lot but didn't check if it helped. >> Only thing that catches my eye on stats frozen vs normal is >> irq_exits which jumps from 200 to 10000 when frozen and irq_window >> which goes 10x. > Hm, default Windows timer rate is 64Hz, so if nothing happens in the > guest you should see at least 256 irq injection per second. In your KVM > start output I only see 4655 irq injection per second not 10000. And > trace shows that most of them are timer interrupts (vector 209 IIRC). Frozen kvm_stat can be seen here; when I grabbed this, windows did not answer to ping. stats: http://mizar.remote.agasha.com/k/kvm/kvm_stat_hang.txt trace: http://mizar.remote.agasha.com/k/kvm/kvm_trace_hang.txt Here's another frozen state stat dump; when I got this windows did answer to ping but was otherwise unresponsive: stats: http://mizar.remote.agasha.com/k/kvm/kvm_stat_2_log.txt trace: http://mizar.remote.agasha.com/k/kvm/kvm_trace_2.txt On both times irq_exits go to 10000 but irq_injections stay at around 4600. On both times qemu process was at 100% cpu load. Note that stats quoted below are from normal operation, where everything works. >> Host has swap partition enabled but hasn't swapped once yet, >> currently there's over 1GB of free. > And how many cpus does it have? Host has one quad-core Q8300 and 8GB of memory. > BTW I see a lot of instruction emulations. That is strange. Looking at > you trace again I see that the bulk of the emulations are caused by mmio > to HPET. Please run with -no-hpet flag. I'll do that later today too. > >> I'll update host kernel to latest stable and qemu-kvm to upstream >> later today. >> >> Here's stats from running system: >> efer_reload 0 0 >> exits 386474714 23272 >> fpu_reload 173420913 12775 >> halt_exits 51894921 4294 >> halt_wakeup 51039254 4223 >> host_state_reload 188645310 13420 >> hypercalls 0 0 >> insn_emulation 212729430 15545 >> insn_emulation_fail 324 0 >> invlpg 5177828 134 >> io_exits 23132581 161 >> irq_exits 7485833 225 >> irq_injections 59239306 4655 >> irq_window 2147438 153 >> largepages 0 0 >> mmio_exits 110931423 9026 >> mmu_cache_miss 468410 0 >> mmu_flooded 193153 0 >> mmu_pde_zapped 352178 0 >> mmu_pte_updated 678023 0 >> mmu_pte_write 587175 0 >> mmu_recycled 34998 0 >> mmu_shadow_zapped 454073 0 >> mmu_unsync 19084 0 >> nmi_injections 0 0 >> nmi_window 0 0 >> pf_fixed 25171943 120 >> pf_guest 6209034 2 >> remote_tlb_flush 3044169 23 >> request_irq 0 0 >> signal_exits 0 0 >> tlb_flush 8422950 149 >> ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Freezing Windows 2008 x64bit guest 2010-07-21 10:09 ` Harri Olin @ 2010-07-21 10:30 ` Gleb Natapov 2010-07-21 10:43 ` Harri Olin 2010-07-21 13:03 ` Harri Olin 0 siblings, 2 replies; 22+ messages in thread From: Gleb Natapov @ 2010-07-21 10:30 UTC (permalink / raw) To: Harri Olin; +Cc: Christoph Adomeit, kvm@vger.kernel.org On Wed, Jul 21, 2010 at 01:09:06PM +0300, Harri Olin wrote: > Gleb Natapov wrote: > >On Wed, Jul 21, 2010 at 12:22:45PM +0300, Harri Olin wrote: > >>Gleb Natapov wrote: > >>>On Wed, Jul 21, 2010 at 09:25:31AM +0300, Harri Olin wrote: > >>>>Gleb Natapov kirjoitti: > >>>>>On Mon, Jul 19, 2010 at 10:17:02AM +0300, Harri Olin wrote: > >>>>>>Gleb Natapov kirjoitti: > >>>>>>>On Thu, Jul 15, 2010 at 03:19:44PM +0200, Christoph Adomeit wrote: > >>>>>>>>But one Windows 2008 64 Bit Server Standard is freezing regularly. > >>>>>>>>This happens sometimes 3 times a day, sometimes it takes 2 days > >>>>>>>>until freeze. The Windows Machine is a clean fresh install. > >>>>>>I think I have seen same problem occur on my Windows 2008 SBS SP2 > >>>>>>64bit system, but a bit less often, only like once a week. > >>>>>>Now I haven't seen crashes but only freezes with qemu on 100% and > >>>>>>virtual system unresponsive. > >>>>>> > >>>>>>qemu command line: > >>>>>>/usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive > >>>>>>file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive > >>>>>>file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net > >>>>>>nic,macaddr=C0:FF:12:FB:AA:01,model=e1000 -net tap -smp 4 -localtime > >>>>>> > >>>>>> > >>>>>Try with different model then e1000 please. Default driver that comes > >>>>>with Windows known to have problems. > >>>>Didn't help, changed to default realtek emulation, but system > >>>>freezed again. Command line: > >>>>/usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive > >>>>file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive > >>>>file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net > >>>>nic,macaddr=C0:FF:12:FB:AA:01 -net tap -smp 4 -localtime > >>>> > >>>>This time the virtual system was not totally unresponsive somehow, > >>>>system pinged just fine and I think DNS server worked too. However > >>>>on console mouse moved but didn't react to clicking, ctrl-alt-del, > >>>>etc. > >>>> > >>>Does sendkey from monitor works? qemu-kvm-0.11.1 is very old and this is > >>>not total freeze which even harder to debug. I don't see anything > >>>extraordinary in your logs. 4643 interrupt per second for 4 cpus is > >>>normal if windows runs multimedia or other app that need hi-res timers. > >>>Does your host swapping? Is there any chance that you can try upstream qemu-kvm? > >>> > >>I haven't tested sendkey. I originally tried 0.12.2 or so but with > >>that windows crashed more often with bluescreen, I went down to > >>0.11.1. This might be related to -td-rtc-hack and/or e1000 as using > >>td-rtc-hack on 0.11.1 windows crashed too to bluescreen. > >> > >With your guest -td-rtc-hack does nothing anyway, so you can just drop > >it. Although it shouldn't cause BSOD with upstream (it had such bug in > >the past and I am not sure that 0.12 has the fix). And you are using > >HPET anyway so -td-rtc-hack should be nop for you. > > Ok, I think I added it because windows guest clock lagged behind a > lot but didn't check if it helped. > For win2k8-64bit (and win2k3-64bit) we don't have a solution for time drift currently. And with hpet we don't have it for any windows. > >>Only thing that catches my eye on stats frozen vs normal is > >>irq_exits which jumps from 200 to 10000 when frozen and irq_window > >>which goes 10x. > >Hm, default Windows timer rate is 64Hz, so if nothing happens in the > >guest you should see at least 256 irq injection per second. In your KVM > >start output I only see 4655 irq injection per second not 10000. And > >trace shows that most of them are timer interrupts (vector 209 IIRC). > > Frozen kvm_stat can be seen here; when I grabbed this, windows did > not answer to ping. > stats: http://mizar.remote.agasha.com/k/kvm/kvm_stat_hang.txt > trace: http://mizar.remote.agasha.com/k/kvm/kvm_trace_hang.txt Not much happens here. Windows gets timer irqs at 1024Hz rate and process them. What is your host kernel? Does it have hi-res timers? > > Here's another frozen state stat dump; when I got this windows did > answer to ping but was otherwise unresponsive: > stats: http://mizar.remote.agasha.com/k/kvm/kvm_stat_2_log.txt > trace: http://mizar.remote.agasha.com/k/kvm/kvm_trace_2.txt > > On both times irq_exits go to 10000 but irq_injections stay at > around 4600. On both times qemu process was at 100% cpu load. > irq_exits are exits due to host interrupt. What "cat /proc/interrupts" show? Which counter grows at this rate there (or 1/4 of the rate but for each host cpu). -- Gleb. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Freezing Windows 2008 x64bit guest 2010-07-21 10:30 ` Gleb Natapov @ 2010-07-21 10:43 ` Harri Olin 2010-07-21 13:03 ` Harri Olin 1 sibling, 0 replies; 22+ messages in thread From: Harri Olin @ 2010-07-21 10:43 UTC (permalink / raw) To: Gleb Natapov; +Cc: Christoph Adomeit, kvm@vger.kernel.org Gleb Natapov wrote: > On Wed, Jul 21, 2010 at 01:09:06PM +0300, Harri Olin wrote: >> Gleb Natapov wrote: >> Frozen kvm_stat can be seen here; when I grabbed this, windows did >> not answer to ping. >> stats: http://mizar.remote.agasha.com/k/kvm/kvm_stat_hang.txt >> trace: http://mizar.remote.agasha.com/k/kvm/kvm_trace_hang.txt > Not much happens here. Windows gets timer irqs at 1024Hz rate and > process them. What is your host kernel? Does it have hi-res timers? Host kernel is vanilla 2.6.33.1 from kernel.org. Should have hi-res timers enabled. full dmesg at: http://mizar.remote.agasha.com/k/kvm/dmesg.txt [ 0.415380] HPET: 4 timers in total, 0 timers will be used for per-cpu timer [ 0.416004] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0 [ 0.416263] hpet0: 4 comparators, 64-bit 14.318180 MHz counter >> Here's another frozen state stat dump; when I got this windows did >> answer to ping but was otherwise unresponsive: >> stats: http://mizar.remote.agasha.com/k/kvm/kvm_stat_2_log.txt >> trace: http://mizar.remote.agasha.com/k/kvm/kvm_trace_2.txt >> >> On both times irq_exits go to 10000 but irq_injections stay at >> around 4600. On both times qemu process was at 100% cpu load. >> > irq_exits are exits due to host interrupt. What "cat /proc/interrupts" > show? Which counter grows at this rate there (or 1/4 of the rate but for > each host cpu). I have to check that next time when freeze happens. # cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 0: 1144 1056 1081 1072 IO-APIC-edge timer 1: 324 329 348 329 IO-APIC-edge i8042 3: 1 1 0 0 IO-APIC-edge 4: 0 0 1 1 IO-APIC-edge 8: 16 15 15 15 IO-APIC-edge rtc0 9: 0 0 0 0 IO-APIC-fasteoi acpi 12: 1 1 0 1 IO-APIC-edge i8042 16: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3 17: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4 18: 47695411 47681526 47693374 47680775 IO-APIC-fasteoi arcmsr, ehci_hcd:usb1, uhci_hcd:usb5, uhci_hcd:usb8 22: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb7 23: 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb2, uhci_hcd:usb6 28: 2376183 2377506 2376812 2376143 PCI-MSI-edge ahci 29: 14545152 14557669 14546472 14559769 PCI-MSI-edge eth0 NMI: 0 0 0 0 Non-maskable interrupts LOC: 1522685098 614396609 1720798830 518770603 Local timer interrupts SPU: 0 0 0 0 Spurious interrupts PMI: 0 0 0 0 Performance monitoring interrupts PND: 0 0 0 0 Performance pending work RES: 109858789 114294880 107422261 113139171 Rescheduling interrupts CAL: 228945035 210709845 246864690 232136482 Function call interrupts TLB: 1161 2146 1090 2319 TLB shootdowns TRM: 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 Threshold APIC interrupts MCE: 0 0 0 0 Machine check exceptions MCP: 4226 4226 4226 4226 Machine check polls ERR: 0 MIS: 0 Also at: http://mizar.remote.agasha.com/k/kvm/interrupts.txt -- Harri. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Freezing Windows 2008 x64bit guest 2010-07-21 10:30 ` Gleb Natapov 2010-07-21 10:43 ` Harri Olin @ 2010-07-21 13:03 ` Harri Olin 2010-07-21 13:45 ` Gleb Natapov 1 sibling, 1 reply; 22+ messages in thread From: Harri Olin @ 2010-07-21 13:03 UTC (permalink / raw) To: Gleb Natapov; +Cc: Christoph Adomeit, kvm@vger.kernel.org Gleb Natapov wrote: > On Wed, Jul 21, 2010 at 01:09:06PM +0300, Harri Olin wrote: >> Gleb Natapov wrote: >> On both times irq_exits go to 10000 but irq_injections stay at >> around 4600. On both times qemu process was at 100% cpu load. > irq_exits are exits due to host interrupt. What "cat /proc/interrupts" > show? Which counter grows at this rate there (or 1/4 of the rate but for > each host cpu). System froze again just a moment ago and there are 6100 "local timer interrupts" and around 3600 "Rescheduling interrupts" every second happening. #cat /proc/interrupts; sleep 1; cat /proc/interrupts; sleep 1; .... http://mizar.remote.agasha.com/k/kvm/interrupts.txt http://mizar.remote.agasha.com/k/kvm/interrupts1.txt quick calculations total RES: 110847929 115251972 108671867 114338799 449110567 per second: RES: 110848996 115252094 108673031 114340080 449114201 3634 RES: 110850059 115252174 108674157 114341446 449117836 3635 RES: 110851132 115252276 108675332 114342801 449121541 3705 total LOC: 1531251837 618157953 1727791012 522296117 4399496919 per second: LOC: 1531252861 618161024 1727792036 522297141 4399503062 6143 LOC: 1531253884 618164095 1727793060 522298165 4399509204 6142 LOC: 1531254907 618167166 1727794084 522299189 4399515346 6142 ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Freezing Windows 2008 x64bit guest 2010-07-21 13:03 ` Harri Olin @ 2010-07-21 13:45 ` Gleb Natapov 2010-07-21 14:05 ` Harri Olin 0 siblings, 1 reply; 22+ messages in thread From: Gleb Natapov @ 2010-07-21 13:45 UTC (permalink / raw) To: Harri Olin; +Cc: Christoph Adomeit, kvm@vger.kernel.org On Wed, Jul 21, 2010 at 04:03:28PM +0300, Harri Olin wrote: > Gleb Natapov wrote: > >On Wed, Jul 21, 2010 at 01:09:06PM +0300, Harri Olin wrote: > >>Gleb Natapov wrote: > >>On both times irq_exits go to 10000 but irq_injections stay at > >>around 4600. On both times qemu process was at 100% cpu load. > >irq_exits are exits due to host interrupt. What "cat /proc/interrupts" > >show? Which counter grows at this rate there (or 1/4 of the rate but for > >each host cpu). > > System froze again just a moment ago and there are 6100 "local timer > interrupts" and around 3600 "Rescheduling interrupts" every second > happening. > > #cat /proc/interrupts; sleep 1; cat /proc/interrupts; sleep 1; .... > http://mizar.remote.agasha.com/k/kvm/interrupts.txt > http://mizar.remote.agasha.com/k/kvm/interrupts1.txt > > quick calculations > total > RES: 110847929 115251972 108671867 114338799 449110567 per second: > RES: 110848996 115252094 108673031 114340080 449114201 3634 > RES: 110850059 115252174 108674157 114341446 449117836 3635 > RES: 110851132 115252276 108675332 114342801 449121541 3705 > > total > LOC: 1531251837 618157953 1727791012 522296117 4399496919 per second: > LOC: 1531252861 618161024 1727792036 522297141 4399503062 6143 > LOC: 1531253884 618164095 1727793060 522298165 4399509204 6142 > LOC: 1531254907 618167166 1727794084 522299189 4399515346 6142 Same qemu and command line as before? Without -no-hpet? -- Gleb. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Freezing Windows 2008 x64bit guest 2010-07-21 13:45 ` Gleb Natapov @ 2010-07-21 14:05 ` Harri Olin 0 siblings, 0 replies; 22+ messages in thread From: Harri Olin @ 2010-07-21 14:05 UTC (permalink / raw) To: Gleb Natapov; +Cc: Christoph Adomeit, kvm@vger.kernel.org Gleb Natapov wrote: > On Wed, Jul 21, 2010 at 04:03:28PM +0300, Harri Olin wrote: >> Gleb Natapov wrote: >>> On Wed, Jul 21, 2010 at 01:09:06PM +0300, Harri Olin wrote: >>>> Gleb Natapov wrote: >>>> On both times irq_exits go to 10000 but irq_injections stay at >>>> around 4600. On both times qemu process was at 100% cpu load. >>> irq_exits are exits due to host interrupt. What "cat /proc/interrupts" >>> show? Which counter grows at this rate there (or 1/4 of the rate but for >>> each host cpu). >> System froze again just a moment ago and there are 6100 "local timer >> interrupts" and around 3600 "Rescheduling interrupts" every second >> happening. >> >> #cat /proc/interrupts; sleep 1; cat /proc/interrupts; sleep 1; .... >> http://mizar.remote.agasha.com/k/kvm/interrupts.txt >> http://mizar.remote.agasha.com/k/kvm/interrupts1.txt >> >> quick calculations >> total >> RES: 110847929 115251972 108671867 114338799 449110567 per second: >> RES: 110848996 115252094 108673031 114340080 449114201 3634 >> RES: 110850059 115252174 108674157 114341446 449117836 3635 >> RES: 110851132 115252276 108675332 114342801 449121541 3705 >> >> total >> LOC: 1531251837 618157953 1727791012 522296117 4399496919 per second: >> LOC: 1531252861 618161024 1727792036 522297141 4399503062 6143 >> LOC: 1531253884 618164095 1727793060 522298165 4399509204 6142 >> LOC: 1531254907 618167166 1727794084 522299189 4399515346 6142 > > Same qemu and command line as before? Without -no-hpet? Yes those interrupt numbers area still running qemu-kvm-0.11.1 without -no-hpet. I now started the quest with -no-hpet, kvm_stat, everything working: efer_reload 0 0 exits 283535303 41437 fpu_reload 94754003 16231 halt_exits 24716918 4696 halt_wakeup 24416218 4651 host_state_reload 112571913 18552 hypercalls 0 0 insn_emulation 75833651 11377 insn_emulation_fail 83 0 invlpg 5327501 124 io_exits 84146648 13718 irq_exits 10182390 2295 irq_injections 31379031 5979 irq_window 941352 113 largepages 0 0 mmio_exits 29794 0 mmu_cache_miss 380265 11 mmu_flooded 167547 2 mmu_pde_zapped 288251 2 mmu_pte_updated 586646 2 mmu_pte_write 505897 3 mmu_recycled 40892 0 mmu_shadow_zapped 365079 4 mmu_unsync 19338 -1 nmi_injections 0 0 nmi_window 0 0 pf_fixed 20331924 1570 pf_guest 5332473 214 remote_tlb_flush 3213779 211 request_irq 0 0 signal_exits 0 0 tlb_flush 8205865 335 interrupt count calculations, system working, -no-hpet: total LOC: 1537337119 621778834 1732851735 525376978 4417344666 per sec LOC: 1537339334 621779137 1732851925 525377284 4417347680 3014 LOC: 1537339987 621779260 1732853664 525377824 4417350735 3055 LOC: 1537340126 621779388 1732855889 525378420 4417353823 3088 RES: 112093536 116293086 109808558 115510943 453706123 RES: 112093737 116293164 109808699 115511006 453706606 483 RES: 112093921 116293182 109808731 115511040 453706874 268 RES: 112093976 116293225 109808733 115511061 453706995 121 full /proc/interrupts at: http://mizar.remote.agasha.com/k/kvm/interrupts-working-no-hpet.txt command line: /usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net nic,macaddr=C0:FF:12:FB:AA:01 -net tap -smp 4 -localtime -no-hpet Still couple of hours before I can restart the host to new kernel. -- Harri. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Freezing Windows 2008 x64bit guest 2010-07-21 8:37 ` Gleb Natapov 2010-07-21 9:22 ` Harri Olin @ 2010-07-27 21:53 ` Harri Olin 2010-07-28 15:18 ` Gleb Natapov 1 sibling, 1 reply; 22+ messages in thread From: Harri Olin @ 2010-07-27 21:53 UTC (permalink / raw) To: Gleb Natapov; +Cc: Christoph Adomeit, kvm@vger.kernel.org Gleb Natapov wrote: > On Wed, Jul 21, 2010 at 09:25:31AM +0300, Harri Olin wrote: >> Gleb Natapov kirjoitti: >>> On Mon, Jul 19, 2010 at 10:17:02AM +0300, Harri Olin wrote: >>>> Gleb Natapov kirjoitti: >>>>> On Thu, Jul 15, 2010 at 03:19:44PM +0200, Christoph Adomeit wrote: >>>>>> But one Windows 2008 64 Bit Server Standard is freezing regularly. >>>>>> This happens sometimes 3 times a day, sometimes it takes 2 days >>>>>> until freeze. The Windows Machine is a clean fresh install. >>>> I think I have seen same problem occur on my Windows 2008 SBS SP2 >>>> 64bit system, but a bit less often, only like once a week. >>>> Now I haven't seen crashes but only freezes with qemu on 100% and >>>> virtual system unresponsive. > Does sendkey from monitor works? qemu-kvm-0.11.1 is very old and this is > not total freeze which even harder to debug. I don't see anything > extraordinary in your logs. 4643 interrupt per second for 4 cpus is > normal if windows runs multimedia or other app that need hi-res timers. > Does your host swapping? Is there any chance that you can try upstream qemu-kvm? I tried running qemu-kvm from git but it exhibited the same problem as 12.x that I tried before, BSODing once in a while, running kernel 2.6.34.1. sample BSOD failure details: These two with Realtec nic and qemu cpu 0x00000019 (0x0000000000000020, 0xfffff88007e65970, 0xfffff88007e65990, 0x000000000502040f) 0x00000019 (0x0000000000000020, 0xfffff88007a414c0, 0xfffff88007a414e0, 0x000000000502044c) These are with e1000 and -cpu host 0x0000003b (0x00000000c0000005, 0xfffff80001c5d842, 0xfffffa60093ddb70, 0x0000000000000000) 0x0000003b (0x00000000c0000005, 0xfffff80001cb8842, 0xfffffa600c94ab70, 0x0000000000000000) 0x0000000a (0x0000000000000080, 0x000000000000000c, 0x0000000000000001, 0xfffff80001cadefd) I'll see if I can analyze minidumps later. In addition to these there have been as many reboots that have been only logged as 'disruptive shutdown'. Right now I'm running the problematic guest under Xen 3.2.1-something from Debian to see if it works better. -- Harri. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Freezing Windows 2008 x64bit guest 2010-07-27 21:53 ` Harri Olin @ 2010-07-28 15:18 ` Gleb Natapov 2010-12-13 19:42 ` Manfred Heubach 0 siblings, 1 reply; 22+ messages in thread From: Gleb Natapov @ 2010-07-28 15:18 UTC (permalink / raw) To: Harri Olin; +Cc: Christoph Adomeit, kvm@vger.kernel.org On Wed, Jul 28, 2010 at 12:53:02AM +0300, Harri Olin wrote: > Gleb Natapov wrote: > >On Wed, Jul 21, 2010 at 09:25:31AM +0300, Harri Olin wrote: > >>Gleb Natapov kirjoitti: > >>>On Mon, Jul 19, 2010 at 10:17:02AM +0300, Harri Olin wrote: > >>>>Gleb Natapov kirjoitti: > >>>>>On Thu, Jul 15, 2010 at 03:19:44PM +0200, Christoph Adomeit wrote: > >>>>>>But one Windows 2008 64 Bit Server Standard is freezing regularly. > >>>>>>This happens sometimes 3 times a day, sometimes it takes 2 days > >>>>>>until freeze. The Windows Machine is a clean fresh install. > >>>>I think I have seen same problem occur on my Windows 2008 SBS SP2 > >>>>64bit system, but a bit less often, only like once a week. > >>>>Now I haven't seen crashes but only freezes with qemu on 100% and > >>>>virtual system unresponsive. > >Does sendkey from monitor works? qemu-kvm-0.11.1 is very old and this is > >not total freeze which even harder to debug. I don't see anything > >extraordinary in your logs. 4643 interrupt per second for 4 cpus is > >normal if windows runs multimedia or other app that need hi-res timers. > >Does your host swapping? Is there any chance that you can try upstream qemu-kvm? > > I tried running qemu-kvm from git but it exhibited the same problem > as 12.x that I tried before, BSODing once in a while, running kernel > 2.6.34.1. > That should be pretty stable config, although it would be nice if you could try running in qemy-kvm.git head. > sample BSOD failure details: > These two with Realtec nic and qemu cpu > 0x00000019 (0x0000000000000020, 0xfffff88007e65970, > 0xfffff88007e65990, 0x000000000502040f) > 0x00000019 (0x0000000000000020, 0xfffff88007a414c0, > 0xfffff88007a414e0, 0x000000000502044c) > > These are with e1000 and -cpu host > 0x0000003b (0x00000000c0000005, 0xfffff80001c5d842, > 0xfffffa60093ddb70, 0x0000000000000000) > 0x0000003b (0x00000000c0000005, 0xfffff80001cb8842, > 0xfffffa600c94ab70, 0x0000000000000000) > 0x0000000a (0x0000000000000080, 0x000000000000000c, > 0x0000000000000001, 0xfffff80001cadefd) > Can you attach screenshots of BSODs? Have you reinstalled your guests or are you running the same images you ran in 11.x? > I'll see if I can analyze minidumps later. > > In addition to these there have been as many reboots that have been > only logged as 'disruptive shutdown'. > > Right now I'm running the problematic guest under Xen > 3.2.1-something from Debian to see if it works better. > > -- > Harri. -- Gleb. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Freezing Windows 2008 x64bit guest 2010-07-28 15:18 ` Gleb Natapov @ 2010-12-13 19:42 ` Manfred Heubach 2010-12-13 20:12 ` Dor Laor 0 siblings, 1 reply; 22+ messages in thread From: Manfred Heubach @ 2010-12-13 19:42 UTC (permalink / raw) To: kvm Gleb Natapov <gleb <at> redhat.com> writes: > > On Wed, Jul 28, 2010 at 12:53:02AM +0300, Harri Olin wrote: > > Gleb Natapov wrote: > > >On Wed, Jul 21, 2010 at 09:25:31AM +0300, Harri Olin wrote: > > >>Gleb Natapov kirjoitti: > > >>>On Mon, Jul 19, 2010 at 10:17:02AM +0300, Harri Olin wrote: > > >>>>Gleb Natapov kirjoitti: > > >>>>>On Thu, Jul 15, 2010 at 03:19:44PM +0200, Christoph Adomeit wrote: > > >>>>>>But one Windows 2008 64 Bit Server Standard is freezing regularly. > > >>>>>>This happens sometimes 3 times a day, sometimes it takes 2 days > > >>>>>>until freeze. The Windows Machine is a clean fresh install. > > >>>>I think I have seen same problem occur on my Windows 2008 SBS SP2 > > >>>>64bit system, but a bit less often, only like once a week. > > >>>>Now I haven't seen crashes but only freezes with qemu on 100% and > > >>>>virtual system unresponsive. > > >Does sendkey from monitor works? qemu-kvm-0.11.1 is very old and this is > > >not total freeze which even harder to debug. I don't see anything > > >extraordinary in your logs. 4643 interrupt per second for 4 cpus is > > >normal if windows runs multimedia or other app that need hi-res timers. > > >Does your host swapping? Is there any chance that you can try upstream qemu-kvm? > > > > I tried running qemu-kvm from git but it exhibited the same problem > > as 12.x that I tried before, BSODing once in a while, running kernel > > 2.6.34.1. > > > That should be pretty stable config, although it would be nice if you > could try running in qemy-kvm.git head. > > > sample BSOD failure details: > > These two with Realtec nic and qemu cpu > > 0x00000019 (0x0000000000000020, 0xfffff88007e65970, > > 0xfffff88007e65990, 0x000000000502040f) > > 0x00000019 (0x0000000000000020, 0xfffff88007a414c0, > > 0xfffff88007a414e0, 0x000000000502044c) > > > > These are with e1000 and -cpu host > > 0x0000003b (0x00000000c0000005, 0xfffff80001c5d842, > > 0xfffffa60093ddb70, 0x0000000000000000) > > 0x0000003b (0x00000000c0000005, 0xfffff80001cb8842, > > 0xfffffa600c94ab70, 0x0000000000000000) > > 0x0000000a (0x0000000000000080, 0x000000000000000c, > > 0x0000000000000001, 0xfffff80001cadefd) > > > Can you attach screenshots of BSODs? Have you reinstalled your guests or > are you running the same images you ran in 11.x? > > > I'll see if I can analyze minidumps later. > > > > In addition to these there have been as many reboots that have been > > only logged as 'disruptive shutdown'. > > > > Right now I'm running the problematic guest under Xen > > 3.2.1-something from Debian to see if it works better. > > > > -- > > Harri. > Hello, is there a solution for that problem? I'm experiencing the same problems ever since I installed SBS 2008 on KVM. I was running the host with Ubuntu 10.04 but upgraded to 10.10 - mainly because of performance problems which were solved by the upgrade. After the upgrade the system became extremly unstable. It was crashing as soon as disk io and network io load was growing. 100% reproduceable with windows server backup to an iscsi volume. i had virtio drivers for storage and network installed (redhat/fedora 1.1.11). At each BSOD I had the following line in the log of the guest: virtio_ioport_write: unexpected address 0x13 value 0x1 I changed the network interface back to e1000. What I experience now (and I had that a the very beginning before i switched to virtio network) are freezes. The guest doesn't respond anymore (doesn't answer to pings and doesn't interact via mouse/keyboard anymore). Host CPU usage of the kvm process is 100% on as many cores as there are virtual cpus (in this case 4). I'm a bit frustrated about this. I have 2 windows 2003 32bit, 1 windows xp and 3 linux guests (2x 32bit, 1x64 bit). They are all running without any problems (except that the windows xp guest cannot boot without an ntldr cd image). Only the SBS2008 guest regulary freezes. The host system has 2 Intel Xeon 5504, Intel Chipset 5500, Adaptec Raid 5805, 24 GB DDR3 RAM. I know there is a lack of detailed information right now. I first need to know if anybody is working on this or has similar problems. I can deliver minidumps, and any debugging information you need. I don't want to give up now. We will switch to Hyper-V if we cannot solve this, because we need a stable virtualization plattform for Windows Guests. I would like to use KVM it is so much more flexibel. Best regards Manfred ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Freezing Windows 2008 x64bit guest 2010-12-13 19:42 ` Manfred Heubach @ 2010-12-13 20:12 ` Dor Laor 2010-12-13 20:44 ` Vadim Rozenfeld 0 siblings, 1 reply; 22+ messages in thread From: Dor Laor @ 2010-12-13 20:12 UTC (permalink / raw) To: Manfred Heubach; +Cc: kvm, Vadim Rozenfeld On 12/13/2010 09:42 PM, Manfred Heubach wrote: > > > Gleb Natapov<gleb<at> redhat.com> writes: > >> >> On Wed, Jul 28, 2010 at 12:53:02AM +0300, Harri Olin wrote: >>> Gleb Natapov wrote: >>>> On Wed, Jul 21, 2010 at 09:25:31AM +0300, Harri Olin wrote: >>>>> Gleb Natapov kirjoitti: >>>>>> On Mon, Jul 19, 2010 at 10:17:02AM +0300, Harri Olin wrote: >>>>>>> Gleb Natapov kirjoitti: >>>>>>>> On Thu, Jul 15, 2010 at 03:19:44PM +0200, Christoph Adomeit wrote: >>>>>>>>> But one Windows 2008 64 Bit Server Standard is freezing regularly. >>>>>>>>> This happens sometimes 3 times a day, sometimes it takes 2 days >>>>>>>>> until freeze. The Windows Machine is a clean fresh install. >>>>>>> I think I have seen same problem occur on my Windows 2008 SBS SP2 >>>>>>> 64bit system, but a bit less often, only like once a week. >>>>>>> Now I haven't seen crashes but only freezes with qemu on 100% and >>>>>>> virtual system unresponsive. >>>> Does sendkey from monitor works? qemu-kvm-0.11.1 is very old and this is >>>> not total freeze which even harder to debug. I don't see anything >>>> extraordinary in your logs. 4643 interrupt per second for 4 cpus is >>>> normal if windows runs multimedia or other app that need hi-res timers. >>>> Does your host swapping? Is there any chance that you can try upstream > qemu-kvm? >>> >>> I tried running qemu-kvm from git but it exhibited the same problem >>> as 12.x that I tried before, BSODing once in a while, running kernel >>> 2.6.34.1. >>> >> That should be pretty stable config, although it would be nice if you >> could try running in qemy-kvm.git head. >> >>> sample BSOD failure details: >>> These two with Realtec nic and qemu cpu >>> 0x00000019 (0x0000000000000020, 0xfffff88007e65970, >>> 0xfffff88007e65990, 0x000000000502040f) >>> 0x00000019 (0x0000000000000020, 0xfffff88007a414c0, >>> 0xfffff88007a414e0, 0x000000000502044c) >>> >>> These are with e1000 and -cpu host >>> 0x0000003b (0x00000000c0000005, 0xfffff80001c5d842, >>> 0xfffffa60093ddb70, 0x0000000000000000) >>> 0x0000003b (0x00000000c0000005, 0xfffff80001cb8842, >>> 0xfffffa600c94ab70, 0x0000000000000000) >>> 0x0000000a (0x0000000000000080, 0x000000000000000c, >>> 0x0000000000000001, 0xfffff80001cadefd) >>> >> Can you attach screenshots of BSODs? Have you reinstalled your guests or >> are you running the same images you ran in 11.x? >> >>> I'll see if I can analyze minidumps later. >>> >>> In addition to these there have been as many reboots that have been >>> only logged as 'disruptive shutdown'. >>> >>> Right now I'm running the problematic guest under Xen >>> 3.2.1-something from Debian to see if it works better. >>> >>> -- >>> Harri. >> > Hello, > > is there a solution for that problem? I'm experiencing the same problems ever > since I installed SBS 2008 on KVM. > > I was running the host with Ubuntu 10.04 but upgraded to 10.10 - mainly because > of performance problems which were solved by the upgrade. > > After the upgrade the system became extremly unstable. It was crashing as soon > as disk io and network io load was growing. 100% reproduceable with windows > server backup to an iscsi volume. > > i had virtio drivers for storage and network installed (redhat/fedora 1.1.11). Which fedora/rhel release is that? What's the windows virtio driver version? Have you tried using virt-manager/virhs instead of raw cmdline? About e1000, some windows comes with buggy driver and an update e1000 from Intel fixes some issues. > At each BSOD I had the following line in the log of the guest: > > virtio_ioport_write: unexpected address 0x13 value 0x1 > > I changed the network interface back to e1000. What I experience now (and I had > that a the very beginning before i switched to virtio network) are freezes. The > guest doesn't respond anymore (doesn't answer to pings and doesn't interact via > mouse/keyboard anymore). Host CPU usage of the kvm process is 100% on as many > cores as there are virtual cpus (in this case 4). > > I'm a bit frustrated about this. I have 2 windows 2003 32bit, 1 windows xp and 3 > linux guests (2x 32bit, 1x64 bit). They are all running without any problems > (except that the windows xp guest cannot boot without an ntldr cd image). Only > the SBS2008 guest regulary freezes. > > The host system has 2 Intel Xeon 5504, Intel Chipset 5500, Adaptec Raid 5805, 24 > GB DDR3 RAM. > > I know there is a lack of detailed information right now. I first need to know > if anybody is working on this or has similar problems. I can deliver minidumps, > and any debugging information you need. > > I don't want to give up now. We will switch to Hyper-V if we cannot solve this, > because we need a stable virtualization plattform for Windows Guests. I would > like to use KVM it is so much more flexibel. > > Best regards > Manfred > > > > > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Freezing Windows 2008 x64bit guest 2010-12-13 20:12 ` Dor Laor @ 2010-12-13 20:44 ` Vadim Rozenfeld 2010-12-14 23:57 ` AW: " Manfred Heubach 0 siblings, 1 reply; 22+ messages in thread From: Vadim Rozenfeld @ 2010-12-13 20:44 UTC (permalink / raw) To: dlaor; +Cc: Manfred Heubach, kvm On Mon, 2010-12-13 at 22:12 +0200, Dor Laor wrote: > On 12/13/2010 09:42 PM, Manfred Heubach wrote: > > > > > > Gleb Natapov<gleb<at> redhat.com> writes: > > > >> > >> On Wed, Jul 28, 2010 at 12:53:02AM +0300, Harri Olin wrote: > >>> Gleb Natapov wrote: > >>>> On Wed, Jul 21, 2010 at 09:25:31AM +0300, Harri Olin wrote: > >>>>> Gleb Natapov kirjoitti: > >>>>>> On Mon, Jul 19, 2010 at 10:17:02AM +0300, Harri Olin wrote: > >>>>>>> Gleb Natapov kirjoitti: > >>>>>>>> On Thu, Jul 15, 2010 at 03:19:44PM +0200, Christoph Adomeit wrote: > >>>>>>>>> But one Windows 2008 64 Bit Server Standard is freezing regularly. > >>>>>>>>> This happens sometimes 3 times a day, sometimes it takes 2 days > >>>>>>>>> until freeze. The Windows Machine is a clean fresh install. > >>>>>>> I think I have seen same problem occur on my Windows 2008 SBS SP2 > >>>>>>> 64bit system, but a bit less often, only like once a week. > >>>>>>> Now I haven't seen crashes but only freezes with qemu on 100% and > >>>>>>> virtual system unresponsive. > >>>> Does sendkey from monitor works? qemu-kvm-0.11.1 is very old and this is > >>>> not total freeze which even harder to debug. I don't see anything > >>>> extraordinary in your logs. 4643 interrupt per second for 4 cpus is > >>>> normal if windows runs multimedia or other app that need hi-res timers. > >>>> Does your host swapping? Is there any chance that you can try upstream > > qemu-kvm? > >>> > >>> I tried running qemu-kvm from git but it exhibited the same problem > >>> as 12.x that I tried before, BSODing once in a while, running kernel > >>> 2.6.34.1. > >>> > >> That should be pretty stable config, although it would be nice if you > >> could try running in qemy-kvm.git head. > >> > >>> sample BSOD failure details: > >>> These two with Realtec nic and qemu cpu > >>> 0x00000019 (0x0000000000000020, 0xfffff88007e65970, > >>> 0xfffff88007e65990, 0x000000000502040f) > >>> 0x00000019 (0x0000000000000020, 0xfffff88007a414c0, > >>> 0xfffff88007a414e0, 0x000000000502044c) > >>> > >>> These are with e1000 and -cpu host > >>> 0x0000003b (0x00000000c0000005, 0xfffff80001c5d842, > >>> 0xfffffa60093ddb70, 0x0000000000000000) > >>> 0x0000003b (0x00000000c0000005, 0xfffff80001cb8842, > >>> 0xfffffa600c94ab70, 0x0000000000000000) > >>> 0x0000000a (0x0000000000000080, 0x000000000000000c, > >>> 0x0000000000000001, 0xfffff80001cadefd) > >>> > >> Can you attach screenshots of BSODs? Have you reinstalled your guests or > >> are you running the same images you ran in 11.x? > >> > >>> I'll see if I can analyze minidumps later. > >>> > >>> In addition to these there have been as many reboots that have been > >>> only logged as 'disruptive shutdown'. > >>> > >>> Right now I'm running the problematic guest under Xen > >>> 3.2.1-something from Debian to see if it works better. > >>> > >>> -- > >>> Harri. > >> > > Hello, > > > > is there a solution for that problem? I'm experiencing the same problems ever > > since I installed SBS 2008 on KVM. > > > > I was running the host with Ubuntu 10.04 but upgraded to 10.10 - mainly because > > of performance problems which were solved by the upgrade. > > > > After the upgrade the system became extremly unstable. It was crashing as soon > > as disk io and network io load was growing. 100% reproduceable with windows > > server backup to an iscsi volume. > > > > i had virtio drivers for storage and network installed (redhat/fedora 1.1.11). > > Which fedora/rhel release is that? > What's the windows virtio driver version? > > Have you tried using virt-manager/virhs instead of raw cmdline? > About e1000, some windows comes with buggy driver and an update e1000 > from Intel fixes some issues. > > > > At each BSOD I had the following line in the log of the guest: > > > > virtio_ioport_write: unexpected address 0x13 value 0x1 > > > > I changed the network interface back to e1000. What I experience now (and I had > > that a the very beginning before i switched to virtio network) are freezes. The > > guest doesn't respond anymore (doesn't answer to pings and doesn't interact via > > mouse/keyboard anymore). Host CPU usage of the kvm process is 100% on as many > > cores as there are virtual cpus (in this case 4). > > Sounds like an interrupt storm to me. Can you try to ping your VM? Anyway the best way to start debugging a stalled system is just to crash it with BSOD. For doing it you will need: - enable NMICrashDump (please see http://support.microsoft.com/kb/927069 for more information - enable Kernel Memory Dump (actually Complete is much better, but it can be too big) http://support.microsoft.com/kb/969028 - you only will need to type "nmi 0" in the qemu monitor to crash the system, when the system hangs next time. Best regards, Vadim. > > I'm a bit frustrated about this. I have 2 windows 2003 32bit, 1 windows xp and 3 > > linux guests (2x 32bit, 1x64 bit). They are all running without any problems > > (except that the windows xp guest cannot boot without an ntldr cd image). Only > > the SBS2008 guest regulary freezes. > > > > The host system has 2 Intel Xeon 5504, Intel Chipset 5500, Adaptec Raid 5805, 24 > > GB DDR3 RAM. > > > > I know there is a lack of detailed information right now. I first need to know > > if anybody is working on this or has similar problems. I can deliver minidumps, > > and any debugging information you need. > > > > I don't want to give up now. We will switch to Hyper-V if we cannot solve this, > > because we need a stable virtualization plattform for Windows Guests. I would > > like to use KVM it is so much more flexibel. > > > > Best regards > > Manfred > > > > > > > > > > -- > > To unsubscribe from this list: send the line "unsubscribe kvm" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 22+ messages in thread
* AW: Freezing Windows 2008 x64bit guest 2010-12-13 20:44 ` Vadim Rozenfeld @ 2010-12-14 23:57 ` Manfred Heubach 2010-12-15 10:48 ` Vadim Rozenfeld 0 siblings, 1 reply; 22+ messages in thread From: Manfred Heubach @ 2010-12-14 23:57 UTC (permalink / raw) To: Vadim Rozenfeld, dlaor@redhat.com; +Cc: kvm@vger.kernel.org Vadim Rozenfeld <vrozenfe <at> redhat.com> writes: > > On Mon, 2010-12-13 at 22:12 +0200, Dor Laor wrote: > > On 12/13/2010 09:42 PM, Manfred Heubach wrote: > > > > > > I was running the host with Ubuntu 10.04 but upgraded to 10.10 - mainly because > > > of performance problems which were solved by the upgrade. > > > > > > After the upgrade the system became extremly unstable. It was crashing as soon > > > as disk io and network io load was growing. 100% reproduceable with windows > > > server backup to an iscsi volume. > > > > > > i had virtio drivers for storage and network installed (redhat/fedora 1.1.11). > > > > Which fedora/rhel release is that? The host is Ubuntu 10.10 x64 The drivers are from http://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/ 1.1.11-0 released on 17-Aug-2010 - are there any newer drivers? > > What's the windows virtio driver version? The virtio storage version shown in Windows is 6.0.0.10 > > > > Have you tried using virt-manager/virhs instead of raw cmdline? I'm starting it with libvirt/virsh cmd-line copied from the log (and some log entries): LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.12 -enable-kvm -m 8192 -smp 4,sockets=4,cores=1,threads=1 -name sbs2008 -uuid 933c2ef2-e5b0-0b39-db60-016b5d226534 -nodefaults -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/sbs2008.monitor,server,nowait -mon chardev=monitor,mode=readline -rtc base=localtime -boot c -drive file=/var/lib/libvirt/images/olscanner/virtio-win-1.1.11-0.iso,if=none,media=cdrom, id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/var/lib/libvirt/images/sbs2008/sbs2008.img,if=none,id=drive-virtio-disk0, boot=on,format=qcow2 -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -drive file=/dev/volg1/sbsdata,if=none,id=drive-virtio-disk1,format=raw,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1 -drive file=/dev/volg1/wsus,if=none,id=drive-virtio-disk2,format=raw,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x7,drive=drive-virtio-disk2,id=virtio-disk2 -device e1000,vlan=0,id=net0,mac=52:54:00:8a:bc:c9,bus=pci.0,addr=0x6 -net tap,fd=107,vlan=0,name=hostnet0 -chardev pty,id=serial0 -device isa-serial,chardev=serial0 -usb -device usb-tablet,id=input0 -vnc 0.0.0.0:0 -k de -vga std -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 07:12:02.715: debug : qemudInitCpuAffinity:2423 : Setting CPU affinity 07:12:02.717: debug : qemuSecurityDACSetProcessLabel:547 : Dropping privileges of VM to 105:114 char device redirected to /dev/pts/0 pci_add_option_rom: failed to find romfile "pxe-e1000.bin" > > About e1000, some windows comes with buggy driver and an update e1000 > > from Intel fixes some issues. > > I'm running latest drivers from Intel: 8.3.15.0 > > > > > At each BSOD I had the following line in the log of the guest: > > > > > > virtio_ioport_write: unexpected address 0x13 value 0x1 > > > > > > I changed the network interface back to e1000. What I experience now (and I had > > > that a the very beginning before i switched to virtio network) are freezes. The > > > guest doesn't respond anymore (doesn't answer to pings and doesn't interact via > > > mouse/keyboard anymore). Host CPU usage of the kvm process is 100% on as many > > > cores as there are virtual cpus (in this case 4). I had a crash today but no logentry on the host - but that could be because I had to restart syslog (ran out of diskspace after turning on debug logging ob libvirtd - didn't think that it would generate 6 GB of logs per day :-) > > > > Sounds like an interrupt storm to me. Can you try to ping your VM? No responds to ping. > Anyway the best way to start debugging a stalled system is just to crash > it with BSOD. For doing it you will need: > - enable NMICrashDump (please see http://support.microsoft.com/kb/927069 > for more information > - enable Kernel Memory Dump (actually Complete is much better, but it > can be too big) http://support.microsoft.com/kb/969028 > - you only will need to type "nmi 0" in the qemu monitor to crash the > system, when the system hangs next time. I prepared this. When the system crashed today I didn't have the complete memory dump ready - so I only have a minidump. The intersting point is that the system today crashed with a BSOD and didn't freeze. The result of dumpchk.exe is as follows: Microsoft (R) Windows Debugger Version 6.12.0002.633 AMD64 Copyright (c) Microsoft Corporation. All rights reserved. Loading Dump File [c:\Windows\Minidump\Mini121410-01.dmp] Mini Kernel Dump File: Only registers and stack trace are available Symbol search path is: SRV*http://msdl.microsoft.com/download/symbols Executable search path is: Windows Server 2008/Windows Vista Kernel Version 6002 (Service Pack 2) MP (4 procs) Free x64 Product: LanManNt, suite: SmallBusiness TerminalServer SmallBusinessRestricted SingleUserTS Built by: 6002.18267.amd64fre.vistasp2_gdr.100608-0458 Machine Name: Kernel base = 0xfffff800`01e16000 PsLoadedModuleList = 0xfffff800`01fdadd0 Debug session time: Tue Dec 14 18:41:20.058 2010 (UTC + 1:00) System Uptime: 1 days 11:16:05.967 Loading Kernel Symbols ............................................................... ............................................................ Loading User Symbols Loading unloaded module list ..... ******************************************************************************* * * * Bugcheck Analysis * * * ******************************************************************************* Use !analyze -v to get detailed debugging information. BugCheck 3B, {c0000005, fffff80001e6a842, fffffa6004efcb70, 0} Probably caused by : Npfs.SYS ( Npfs!NpSetAttributeInList+9a ) Followup: MachineOwner --------- ----- 64 bit Kernel Mini Dump Analysis DUMP_HEADER64: MajorVersion 0000000f MinorVersion 00001772 KdSecondaryVersion 00000000 DirectoryTableBase 00000001`e5b51000 PfnDataBase fffff800`0203d250 PsLoadedModuleList fffff800`01fdadd0 PsActiveProcessHead fffff800`01fba440 MachineImageType 00008664 NumberProcessors 00000004 BugCheckCode 0000003b BugCheckParameter1 00000000`c0000005 BugCheckParameter2 fffff800`01e6a842 BugCheckParameter3 fffffa60`04efcb70 BugCheckParameter4 00000000`00000000 KdDebuggerDataBlock fffff800`01f87f20 ProductType 00000002 SuiteMask 00000131 WriterStatus 00000000 MiniDumpFields 00000cff TRIAGE_DUMP64: ServicePackBuild 00000200 SizeOfDump 00040000 ValidOffset 0003fffc ContextOffset 00000348 ExceptionOffset 00000f00 MmOffset 00002080 UnloadedDriversOffset 000020d0 PrcbOffset 000021f0 ProcessOffset 00005d10 ThreadOffset 000060f8 CallStackOffset 00006548 SizeOfCallStack 00001bf8 DriverListOffset 00008470 DriverCount 0000007c StringPoolOffset 0000ca30 StringPoolSize 00002950 BrokenDriverOffset 00000000 TriageOptions ffffffff TopOfStack fffffa60`04efc1b8 BStoreOffset 00000000 SizeOfBStore 00000000 LimitOfBStore 00000000`00000000 DebuggerDataOffset 00008140 DebuggerDataSize 00000330 DataBlocksOffset 0000f380 DataBlocksCount 00000008 fffff800`01fdb720 - fffff800`01fdb79f at offset 0000f400 fffff800`01fbf898 - fffff800`01fbf89b at offset 0000f480 fffff800`01fbf894 - fffff800`01fbf897 at offset 0000f484 fffff800`0203d078 - fffff800`0203d07b at offset 0000f488 fffff800`0203d09c - fffff800`0203d09f at offset 0000f48c fffff800`01e6f000 - fffff800`01e6ffff at offset 0000f490 fffff800`02019000 - fffff800`02019fff at offset 00010490 fffff800`01e70000 - fffff800`01e70fff at offset 00011490 Max offset 12490, 2e2f8 from end of file Windows Server 2008/Windows Vista Kernel Version 6002 (Service Pack 2) MP (4 procs) Free x64 Product: LanManNt, suite: SmallBusiness TerminalServer SmallBusinessRestricted SingleUserTS Built by: 6002.18267.amd64fre.vistasp2_gdr.100608-0458 Machine Name: Kernel base = 0xfffff800`01e16000 PsLoadedModuleList = 0xfffff800`01fdadd0 Debug session time: Tue Dec 14 18:41:20.058 2010 (UTC + 1:00) System Uptime: 1 days 11:16:05.967 start end module name fffff800`01e16000 fffff800`0232d000 nt Tue Jun 08 16:59:47 2010 (4C0E5AE3) fffff800`0232d000 fffff800`02373000 hal Sat Apr 11 09:04:56 2009 (49E04118) fffff960`000b0000 fffff960`00364000 win32k Tue Aug 31 16:57:22 2010 (4C7D1852) fffff960`004d0000 fffff960`004ee000 dxg Sat Apr 11 07:09:45 2009 (49E02619) fffff960`00680000 fffff960`0068a000 TSDDD Sat Jan 19 07:42:04 2008 (47919BBC) fffff960`008c0000 fffff960`008c9000 framebuf unavailable (00000000) fffff960`00a80000 fffff960`00ab0000 RDPDD unavailable (00000000) fffffa60`00604000 fffffa60`0060e000 kdcom Sat Apr 11 09:07:57 2009 (49E041CD) fffffa60`0060e000 fffffa60`00649000 mcupdate_GenuineIntel Sat Apr 11 09:05:41 2009 (49E04145) fffffa60`00649000 fffffa60`0065d000 PSHED Sat Apr 11 09:08:17 2009 (49E041E1) fffffa60`0065d000 fffffa60`006ba000 CLFS Sat Apr 11 06:54:21 2009 (49E0227D) fffffa60`006ba000 fffffa60`0076c000 CI Sat Apr 11 09:08:09 2009 (49E041D9) fffffa60`0076c000 fffffa60`0077e000 HIDCLASS Sat Apr 11 07:39:32 2009 (49E02D14) fffffa60`0077e000 fffffa60`00785b80 HIDPARSE Sat Jan 19 07:33:51 2008 (479199CF) fffffa60`00788000 fffffa60`007d8000 msrpc Sat Apr 11 07:32:13 2009 (49E02B5D) fffffa60`007d8000 fffffa60`007f5000 tdx Sat Apr 11 07:43:00 2009 (49E02DE4) fffffa60`007f5000 fffffa60`00800000 mouhid Sat Jan 19 07:28:10 2008 (4791987A) fffffa60`00800000 fffffa60`00809000 hidusb Sat Apr 11 07:39:32 2009 (49E02D14) fffffa60`0080a000 fffffa60`009cd000 NDIS Sat Apr 11 07:43:15 2009 (49E02DF3) fffffa60`009cd000 fffffa60`009e1000 volmgr Sat Apr 11 07:34:49 2009 (49E02BF9) fffffa60`009e1000 fffffa60`009f2000 Npfs Sat Apr 11 06:54:22 2009 (49E0227E) fffffa60`009f2000 fffffa60`009fb000 rasacd Sat Jan 19 07:37:30 2008 (47919AAA) fffffa60`00a02000 fffffa60`00a5b000 NETIO Sat Apr 11 07:43:34 2009 (49E02E06) fffffa60`00a5b000 fffffa60`00b35000 Wdf01000 Sat Jan 19 07:33:27 2008 (479199B7) fffffa60`00b35000 fffffa60`00b43000 WDFLDR Sat Jan 19 07:32:33 2008 (47919981) fffffa60`00b43000 fffffa60`00b99000 acpi Sat Apr 11 07:03:26 2009 (49E0249E) fffffa60`00b99000 fffffa60`00ba2000 WMILIB Sat Jan 19 07:33:45 2008 (479199C9) fffffa60`00ba2000 fffffa60`00bac000 msisadrv Sat Jan 19 07:02:50 2008 (4791928A) fffffa60`00bac000 fffffa60`00bdc000 pci Sat Apr 11 07:03:33 2009 (49E024A5) fffffa60`00bdc000 fffffa60`00bf1000 partmgr Sat Apr 11 07:34:51 2009 (49E02BFB) fffffa60`00bf1000 fffffa60`00bfc000 Msfs Sat Jan 19 06:53:55 2008 (47919073) fffffa60`00c00000 fffffa60`00c09000 rdpencdd Sat Jan 19 07:42:03 2008 (47919BBB) fffffa60`00c0d000 fffffa60`00c73000 volmgrx Sat Apr 11 07:35:14 2009 (49E02C12) fffffa60`00c73000 fffffa60`00c7b000 intelide Sat Jan 19 07:28:56 2008 (479198A8) fffffa60`00c7b000 fffffa60`00c8b000 PCIIDEX Sat Apr 11 07:34:22 2009 (49E02BDE) fffffa60`00c8b000 fffffa60`00c9e000 mountmgr Sat Jan 19 07:28:01 2008 (47919871) fffffa60`00c9e000 fffffa60`00ca6000 atapi Sat Apr 11 07:34:19 2009 (49E02BDB) fffffa60`00ca6000 fffffa60`00cca000 ataport Sat Apr 11 07:34:25 2009 (49E02BE1) fffffa60`00cca000 fffffa60`00cd3000 viostor Tue Aug 10 15:58:06 2010 (4C615AEE) fffffa60`00cd3000 fffffa60`00d30000 storport Sat Apr 11 07:34:45 2009 (49E02BF5) fffffa60`00d30000 fffffa60`00d77000 fltmgr Sat Apr 11 06:54:38 2009 (49E0228E) fffffa60`00d77000 fffffa60`00da2000 quota Sat Apr 11 06:59:38 2009 (49E023BA) fffffa60`00da2000 fffffa60`00db7000 datascrn Sat Apr 11 06:59:35 2009 (49E023B7) fffffa60`00db7000 fffffa60`00df0000 msiscsi Sat Apr 11 07:36:09 2009 (49E02C49) fffffa60`00df0000 fffffa60`00dfe000 vga Sat Jan 19 07:32:21 2008 (47919975) fffffa60`00e00000 fffffa60`00e0c000 ndistapi Sat Jan 19 07:37:22 2008 (47919AA2) fffffa60`00e0d000 fffffa60`00e94000 ksecdd Mon Jun 15 15:15:18 2009 (4A364966) fffffa60`00e94000 fffffa60`00eaa000 i8042prt Sat Jan 19 07:28:08 2008 (47919878) fffffa60`00eaa000 fffffa60`00eb8000 kbdclass Sat Jan 19 07:28:05 2008 (47919875) fffffa60`00eb8000 fffffa60`00ec4000 mouclass Sat Jan 19 07:28:05 2008 (47919875) fffffa60`00ec4000 fffffa60`00ed1000 fdc Sat Jan 19 07:28:45 2008 (4791989D) fffffa60`00ed1000 fffffa60`00eed000 cdrom Sat Apr 11 07:34:39 2009 (49E02BEF) fffffa60`00eed000 fffffa60`00ef9000 usbuhci Sat Jan 19 07:33:56 2008 (479199D4) fffffa60`00ef9000 fffffa60`00f3f000 USBPORT Sat Apr 11 07:39:39 2009 (49E02D1B) fffffa60`00f3f000 fffffa60`00f4d000 vgapnp Sat Jan 19 07:32:22 2008 (47919976) fffffa60`00f4d000 fffffa60`00f72000 VIDEOPRT Sat Jan 19 07:32:25 2008 (47919979) fffffa60`00f72000 fffffa60`00f82000 watchdog Sat Apr 11 07:09:16 2009 (49E025FC) fffffa60`00f82000 fffffa60`00fa5a00 E1G6032E Wed Sep 24 01:09:49 2008 (48D9773D) fffffa60`00fa6000 fffffa60`00fb9000 intelppm Sat Jan 19 06:52:45 2008 (4791902D) fffffa60`00fb9000 fffffa60`00fcc000 blbdrive Sat Jan 19 06:58:37 2008 (4791918D) fffffa60`00fcc000 fffffa60`00fd9000 TDI Sat Apr 11 07:44:14 2009 (49E02E2E) fffffa60`00fd9000 fffffa60`00ffc000 rasl2tp Sat Apr 11 07:43:37 2009 (49E02E09) fffffa60`01003000 fffffa60`01179000 tcpip Wed Jun 16 16:51:56 2010 (4C18E50C) fffffa60`01179000 fffffa60`011a5000 fwpkclnt Sat Apr 11 07:42:44 2009 (49E02DD4) fffffa60`011a5000 fffffa60`011d1000 CLASSPNP Sat Apr 11 07:34:15 2009 (49E02BD7) fffffa60`011d1000 fffffa60`011db000 crcdisk Sat Jan 19 07:30:12 2008 (479198F4) fffffa60`011f3000 fffffa60`011fc000 tunmp Sat Jan 19 07:36:30 2008 (47919A6E) fffffa60`01209000 fffffa60`01389000 Ntfs Sat Apr 11 06:55:38 2009 (49E022CA) fffffa60`01389000 fffffa60`013cd000 volsnap Sat Apr 11 07:35:33 2009 (49E02C25) fffffa60`013cd000 fffffa60`013d5000 spldr Mon Jan 12 22:51:29 2009 (496BBB61) fffffa60`013d5000 fffffa60`013e7000 mup Sat Apr 11 06:54:47 2009 (49E02297) fffffa60`013e7000 fffffa60`013fb000 disk Sat Apr 11 07:34:38 2009 (49E02BEE) fffffa60`03600000 fffffa60`03631000 ndiswan Sat Apr 11 07:43:39 2009 (49E02E0B) fffffa60`03631000 fffffa60`03641000 raspppoe Sat Apr 11 07:43:33 2009 (49E02E05) fffffa60`03641000 fffffa60`0365f000 raspptp Sat Apr 11 07:43:38 2009 (49E02E0A) fffffa60`0365f000 fffffa60`03677000 rassstp Sat Apr 11 07:43:46 2009 (49E02E12) fffffa60`03677000 fffffa60`03711000 rdpdr Sat Apr 11 07:49:54 2009 (49E02F82) fffffa60`03711000 fffffa60`03724000 termdd Sat Apr 11 07:48:13 2009 (49E02F1D) fffffa60`03724000 fffffa60`03725480 swenum Thu Nov 02 10:37:33 2006 (4549BC5D) fffffa60`03726000 fffffa60`0375a000 ks Sat Apr 11 07:33:51 2009 (49E02BBF) fffffa60`0375a000 fffffa60`03765000 mssmbios Sat Jan 19 07:02:54 2008 (4791928E) fffffa60`03765000 fffffa60`03775000 umbus Sat Jan 19 07:34:16 2008 (479199E8) fffffa60`03775000 fffffa60`037bd000 usbhub Sat Apr 11 07:39:52 2009 (49E02D28) fffffa60`037bd000 fffffa60`037d1000 NDProxy Sat Jan 19 07:37:26 2008 (47919AA6) fffffa60`037d1000 fffffa60`037e1000 dfs Sat Jan 19 06:54:14 2008 (47919086) fffffa60`037e1000 fffffa60`037eb000 Fs_Rec Sat Jan 19 06:53:41 2008 (47919065) fffffa60`037eb000 fffffa60`037f4000 Null Thu Nov 02 10:37:15 2006 (4549BC4B) fffffa60`037f4000 fffffa60`037fd000 RDPCDD Sat Jan 19 07:42:04 2008 (47919BBC) fffffa60`037fd000 fffffa60`037fee00 USBD Sat Jan 19 07:33:53 2008 (479199D1) fffffa60`03c04000 fffffa60`03c1f000 smb Sat Apr 11 07:42:19 2009 (49E02DBB) fffffa60`03c1f000 fffffa60`03c8a000 afd Sat Apr 11 07:44:21 2009 (49E02E35) fffffa60`03c8a000 fffffa60`03cce000 netbt Sat Apr 11 07:42:31 2009 (49E02DC7) fffffa60`03cce000 fffffa60`03cec000 pacer Sat Apr 11 07:42:56 2009 (49E02DE0) fffffa60`03cec000 fffffa60`03cfb000 netbios Sat Jan 19 07:36:35 2008 (47919A73) fffffa60`03cfb000 fffffa60`03d0e000 monitor Sat Jan 19 07:32:34 2008 (47919982) fffffa60`03d18000 fffffa60`03d33000 wanarp Sat Apr 11 07:43:38 2009 (49E02E0A) fffffa60`03d33000 fffffa60`03d80000 rdbss Sat Apr 11 06:55:21 2009 (49E022B9) fffffa60`03d80000 fffffa60`03d8c000 nsiproxy Sat Jan 19 07:36:45 2008 (47919A7D) fffffa60`03d8c000 fffffa60`03da9000 dfsc Sat Apr 11 06:54:44 2009 (49E02294) fffffa60`03da9000 fffffa60`03dc5000 cdfs Sat Jan 19 06:53:45 2008 (47919069) fffffa60`03dc5000 fffffa60`03dd3000 crashdmp Sat Apr 11 07:34:31 2009 (49E02BE7) fffffa60`03dd3000 fffffa60`03ddd000 dump_diskdump Sat Apr 11 07:34:30 2009 (49E02BE6) fffffa60`03ddd000 fffffa60`03de6000 dump_viostor Tue Aug 10 15:58:06 2010 (4C615AEE) fffffa60`03de6000 fffffa60`03df2000 Dxapi Sat Jan 19 07:08:00 2008 (479193C0) fffffa60`04204000 fffffa60`04226000 luafv Sat Jan 19 06:59:06 2008 (479191AA) fffffa60`04226000 fffffa60`0423a000 lltdio Sat Jan 19 07:35:48 2008 (47919A44) fffffa60`0423a000 fffffa60`04252000 rspndr Sat Jan 19 07:35:48 2008 (47919A44) fffffa60`04252000 fffffa60`042ec000 spsys Tue Mar 10 18:16:43 2009 (49B6A07B) fffffa60`042ec000 fffffa60`0430a000 bowser Sat Jan 19 06:54:51 2008 (479190AB) fffffa60`0430a000 fffffa60`04324000 mpsdrv Sat Jan 19 07:35:28 2008 (47919A30) fffffa60`04324000 fffffa60`0434d000 mrxsmb Tue Feb 23 12:32:22 2010 (4B83BCC6) fffffa60`0434d000 fffffa60`04396000 mrxsmb10 Tue Feb 23 12:32:21 2010 (4B83BCC5) fffffa60`04396000 fffffa60`043b5000 mrxsmb20 Tue Feb 23 12:32:20 2010 (4B83BCC4) fffffa60`043b5000 fffffa60`043c0000 asyncmac Sat Jan 19 07:37:27 2008 (47919AA7) fffffa60`043c0000 fffffa60`043e9000 srvnet Mon Sep 06 17:33:48 2010 (4C8509DC) fffffa60`04806000 fffffa60`04838000 srv2 Mon Sep 06 17:33:50 2010 (4C8509DE) fffffa60`04838000 fffffa60`048cc000 srv Mon Sep 06 17:34:12 2010 (4C8509F4) fffffa60`048cc000 fffffa60`0496f000 HTTP Sat Feb 20 22:30:05 2010 (4B80545D) fffffa60`0500e000 fffffa60`050c4000 peauth Mon Oct 23 13:57:00 2006 (453CAE0C) fffffa60`050c4000 fffffa60`050cf000 secdrv Wed Sep 13 15:18:38 2006 (4508052E) fffffa60`050cf000 fffffa60`050df000 tcpipreg Tue Dec 08 18:55:32 2009 (4B1E9314) fffffa60`050df000 fffffa60`050ec000 tdtcp Sat Jan 19 07:42:03 2008 (47919BBB) fffffa60`050ec000 fffffa60`050fa000 tssecsrv Sat Jan 19 07:42:10 2008 (47919BC2) fffffa60`050fa000 fffffa60`05136000 RDPWD Sat Apr 11 07:48:33 2009 (49E02F31) Unloaded modules: fffffa60`011db000 fffffa60`011e9000 crashdmp.sys Timestamp: unavailable (00000000) Checksum: 00000000 ImageSize: 0000E000 fffffa60`011e9000 fffffa60`011f3000 dump_storpor Timestamp: unavailable (00000000) Checksum: 00000000 ImageSize: 0000A000 fffffa60`01200000 fffffa60`01209000 dump_viostor Timestamp: unavailable (00000000) Checksum: 00000000 ImageSize: 00009000 fffffa60`03cfb000 fffffa60`03d18000 serial.sys Timestamp: unavailable (00000000) Checksum: 00000000 ImageSize: 0001D000 fffffa60`0076c000 fffffa60`00788000 sacdrv.sys Timestamp: unavailable (00000000) Checksum: 00000000 ImageSize: 0001C000 ************************************************************************** * * * Bugcheck Analysis * * * ************************************************************************** Use !analyze -v to get detailed debugging information. BugCheck 3B, {c0000005, fffff80001e6a842, fffffa6004efcb70, 0} Probably caused by : Npfs.SYS ( Npfs!NpSetAttributeInList+9a ) Followup: MachineOwner --------- Finished dump check best regards Manfred ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: AW: Freezing Windows 2008 x64bit guest 2010-12-14 23:57 ` AW: " Manfred Heubach @ 2010-12-15 10:48 ` Vadim Rozenfeld 0 siblings, 0 replies; 22+ messages in thread From: Vadim Rozenfeld @ 2010-12-15 10:48 UTC (permalink / raw) To: Manfred Heubach; +Cc: dlaor@redhat.com, kvm@vger.kernel.org On Wed, 2010-12-15 at 00:57 +0100, Manfred Heubach wrote: > Vadim Rozenfeld <vrozenfe <at> redhat.com> writes: > > > > > On Mon, 2010-12-13 at 22:12 +0200, Dor Laor wrote: > > > On 12/13/2010 09:42 PM, Manfred Heubach wrote: > > > > > > > > I was running the host with Ubuntu 10.04 but upgraded to 10.10 - mainly > because > > > > of performance problems which were solved by the upgrade. > > > > > > > > After the upgrade the system became extremly unstable. It was crashing as soon > > > > as disk io and network io load was growing. 100% reproduceable with windows > > > > server backup to an iscsi volume. > > > > > > > > i had virtio drivers for storage and network installed (redhat/fedora 1.1.11). > > > > > > Which fedora/rhel release is that? > > > The host is Ubuntu 10.10 x64 > > The drivers are from > http://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/ 1.1.11-0 > released on 17-Aug-2010 - are there any newer drivers? Yes, they are the most recent drivers at fedoraproject. Our rhel6 drivers are almost the same, except for a few non-critical, WHQL related bug fixes. However, there is one fix related to Large Send Offload (LSO) problem which was fixed in rhel6, but I don't see the relevant changes in fedora kvmnet driver sources http://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/src/ > > > > > What's the windows virtio driver version? > > The virtio storage version shown in Windows is 6.0.0.10 > > > > > > > Have you tried using virt-manager/virhs instead of raw cmdline? > > I'm starting it with libvirt/virsh > > cmd-line copied from the log (and some log entries): > > LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin > QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.12 -enable-kvm -m 8192 -smp > 4,sockets=4,cores=1,threads=1 -name sbs2008 -uuid > 933c2ef2-e5b0-0b39-db60-016b5d226534 -nodefaults -chardev > socket,id=monitor,path=/var/lib/libvirt/qemu/sbs2008.monitor,server,nowait -mon > chardev=monitor,mode=readline -rtc base=localtime -boot c -drive > file=/var/lib/libvirt/images/olscanner/virtio-win-1.1.11-0.iso,if=none,media=cdrom, > id=drive-ide0-1-0,readonly=on,format=raw > -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive > file=/var/lib/libvirt/images/sbs2008/sbs2008.img,if=none,id=drive-virtio-disk0, > boot=on,format=qcow2 -device > virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 > -drive > file=/dev/volg1/sbsdata,if=none,id=drive-virtio-disk1,format=raw,cache=none > -device > virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1 > -drive file=/dev/volg1/wsus,if=none,id=drive-virtio-disk2,format=raw,cache=none > -device > virtio-blk-pci,bus=pci.0,addr=0x7,drive=drive-virtio-disk2,id=virtio-disk2 > -device e1000,vlan=0,id=net0,mac=52:54:00:8a:bc:c9,bus=pci.0,addr=0x6 -net > tap,fd=107,vlan=0,name=hostnet0 -chardev pty,id=serial0 -device > isa-serial,chardev=serial0 -usb -device usb-tablet,id=input0 -vnc 0.0.0.0:0 -k > de -vga std -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 > 07:12:02.715: debug : qemudInitCpuAffinity:2423 : Setting CPU affinity > 07:12:02.717: debug : qemuSecurityDACSetProcessLabel:547 : Dropping privileges > of VM to 105:114 > char device redirected to /dev/pts/0 > pci_add_option_rom: failed to find romfile "pxe-e1000.bin" > > > > > About e1000, some windows comes with buggy driver and an update e1000 > > > from Intel fixes some issues. > > > > > I'm running latest drivers from Intel: 8.3.15.0 > > > > > > > > At each BSOD I had the following line in the log of the guest: > > > > > > > > virtio_ioport_write: unexpected address 0x13 value 0x1 Seems to be the result of calling PapaNdis_OnBugCheck function > > > > > > > > I changed the network interface back to e1000. What I experience now (and > I had > > > > that a the very beginning before i switched to virtio network) are > freezes. The > > > > guest doesn't respond anymore (doesn't answer to pings and doesn't > interact via > > > > mouse/keyboard anymore). Host CPU usage of the kvm process is 100% on as many > > > > cores as there are virtual cpus (in this case 4). > > I had a crash today but no logentry on the host - but that could be because I > had to restart syslog (ran out of diskspace after turning on debug logging ob > libvirtd - didn't think that it would generate 6 GB of logs per day :-) > > > > > > > Sounds like an interrupt storm to me. Can you try to ping your VM? > > No responds to ping. > > > Anyway the best way to start debugging a stalled system is just to crash > > it with BSOD. For doing it you will need: > > - enable NMICrashDump (please see http://support.microsoft.com/kb/927069 > > for more information > > - enable Kernel Memory Dump (actually Complete is much better, but it > > can be too big) http://support.microsoft.com/kb/969028 > > - you only will need to type "nmi 0" in the qemu monitor to crash the > > system, when the system hangs next time. > > I prepared this. When the system crashed today I didn't have the complete > memory dump ready - so I only have a minidump. The intersting point is that > the system today crashed with a BSOD and didn't freeze. > > The result of dumpchk.exe is as follows: > > > Microsoft (R) Windows Debugger Version 6.12.0002.633 AMD64 > Copyright (c) Microsoft Corporation. All rights reserved. > > > Loading Dump File [c:\Windows\Minidump\Mini121410-01.dmp] > Mini Kernel Dump File: Only registers and stack trace are available > > Symbol search path is: SRV*http://msdl.microsoft.com/download/symbols > Executable search path is: > Windows Server 2008/Windows Vista Kernel Version 6002 (Service Pack 2) MP (4 > procs) Free x64 > Product: LanManNt, suite: SmallBusiness TerminalServer SmallBusinessRestricted > SingleUserTS > Built by: 6002.18267.amd64fre.vistasp2_gdr.100608-0458 > Machine Name: > Kernel base = 0xfffff800`01e16000 PsLoadedModuleList = 0xfffff800`01fdadd0 > Debug session time: Tue Dec 14 18:41:20.058 2010 (UTC + 1:00) > System Uptime: 1 days 11:16:05.967 > Loading Kernel Symbols > ............................................................... > ............................................................ > Loading User Symbols > Loading unloaded module list > ..... > ******************************************************************************* > * * > * Bugcheck Analysis * > * * > ******************************************************************************* > > Use !analyze -v to get detailed debugging information. > > BugCheck 3B, {c0000005, fffff80001e6a842, fffffa6004efcb70, 0} > > Probably caused by : Npfs.SYS ( Npfs!NpSetAttributeInList+9a ) It is named pipe file system. > > Followup: MachineOwner > --------- > > ----- 64 bit Kernel Mini Dump Analysis > > DUMP_HEADER64: > MajorVersion 0000000f > MinorVersion 00001772 > KdSecondaryVersion 00000000 > DirectoryTableBase 00000001`e5b51000 > PfnDataBase fffff800`0203d250 > PsLoadedModuleList fffff800`01fdadd0 > PsActiveProcessHead fffff800`01fba440 > MachineImageType 00008664 > NumberProcessors 00000004 > BugCheckCode 0000003b > BugCheckParameter1 00000000`c0000005 > BugCheckParameter2 fffff800`01e6a842 > BugCheckParameter3 fffffa60`04efcb70 > BugCheckParameter4 00000000`00000000 > KdDebuggerDataBlock fffff800`01f87f20 > ProductType 00000002 > SuiteMask 00000131 > WriterStatus 00000000 > MiniDumpFields 00000cff > > TRIAGE_DUMP64: > ServicePackBuild 00000200 > SizeOfDump 00040000 > ValidOffset 0003fffc > ContextOffset 00000348 > ExceptionOffset 00000f00 > MmOffset 00002080 > UnloadedDriversOffset 000020d0 > PrcbOffset 000021f0 > ProcessOffset 00005d10 > ThreadOffset 000060f8 > CallStackOffset 00006548 > SizeOfCallStack 00001bf8 > DriverListOffset 00008470 > DriverCount 0000007c > StringPoolOffset 0000ca30 > StringPoolSize 00002950 > BrokenDriverOffset 00000000 > TriageOptions ffffffff > TopOfStack fffffa60`04efc1b8 > BStoreOffset 00000000 > SizeOfBStore 00000000 > LimitOfBStore 00000000`00000000 > DebuggerDataOffset 00008140 > DebuggerDataSize 00000330 > DataBlocksOffset 0000f380 > DataBlocksCount 00000008 > fffff800`01fdb720 - fffff800`01fdb79f at offset 0000f400 > fffff800`01fbf898 - fffff800`01fbf89b at offset 0000f480 > fffff800`01fbf894 - fffff800`01fbf897 at offset 0000f484 > fffff800`0203d078 - fffff800`0203d07b at offset 0000f488 > fffff800`0203d09c - fffff800`0203d09f at offset 0000f48c > fffff800`01e6f000 - fffff800`01e6ffff at offset 0000f490 > fffff800`02019000 - fffff800`02019fff at offset 00010490 > fffff800`01e70000 - fffff800`01e70fff at offset 00011490 > Max offset 12490, 2e2f8 from end of file > > > Windows Server 2008/Windows Vista Kernel Version 6002 (Service Pack 2) MP (4 > procs) Free x64 > Product: LanManNt, suite: SmallBusiness TerminalServer SmallBusinessRestricted > SingleUserTS > Built by: 6002.18267.amd64fre.vistasp2_gdr.100608-0458 > Machine Name: > Kernel base = 0xfffff800`01e16000 PsLoadedModuleList = 0xfffff800`01fdadd0 > Debug session time: Tue Dec 14 18:41:20.058 2010 (UTC + 1:00) > System Uptime: 1 days 11:16:05.967 > start end module name > fffff800`01e16000 fffff800`0232d000 nt Tue Jun 08 16:59:47 2010 (4C0E5AE3) > fffff800`0232d000 fffff800`02373000 hal Sat Apr 11 09:04:56 2009 (49E04118) > fffff960`000b0000 fffff960`00364000 win32k Tue Aug 31 16:57:22 2010 (4C7D1852) > fffff960`004d0000 fffff960`004ee000 dxg Sat Apr 11 07:09:45 2009 (49E02619) > fffff960`00680000 fffff960`0068a000 TSDDD Sat Jan 19 07:42:04 2008 (47919BBC) > fffff960`008c0000 fffff960`008c9000 framebuf unavailable (00000000) > fffff960`00a80000 fffff960`00ab0000 RDPDD unavailable (00000000) > fffffa60`00604000 fffffa60`0060e000 kdcom Sat Apr 11 09:07:57 2009 (49E041CD) > fffffa60`0060e000 fffffa60`00649000 mcupdate_GenuineIntel Sat Apr 11 > 09:05:41 2009 (49E04145) > fffffa60`00649000 fffffa60`0065d000 PSHED Sat Apr 11 09:08:17 2009 (49E041E1) > fffffa60`0065d000 fffffa60`006ba000 CLFS Sat Apr 11 06:54:21 2009 (49E0227D) > fffffa60`006ba000 fffffa60`0076c000 CI Sat Apr 11 09:08:09 2009 (49E041D9) > fffffa60`0076c000 fffffa60`0077e000 HIDCLASS Sat Apr 11 07:39:32 2009 (49E02D14) > fffffa60`0077e000 fffffa60`00785b80 HIDPARSE Sat Jan 19 07:33:51 2008 (479199CF) > fffffa60`00788000 fffffa60`007d8000 msrpc Sat Apr 11 07:32:13 2009 (49E02B5D) > fffffa60`007d8000 fffffa60`007f5000 tdx Sat Apr 11 07:43:00 2009 (49E02DE4) > fffffa60`007f5000 fffffa60`00800000 mouhid Sat Jan 19 07:28:10 2008 (4791987A) > fffffa60`00800000 fffffa60`00809000 hidusb Sat Apr 11 07:39:32 2009 (49E02D14) > fffffa60`0080a000 fffffa60`009cd000 NDIS Sat Apr 11 07:43:15 2009 (49E02DF3) > fffffa60`009cd000 fffffa60`009e1000 volmgr Sat Apr 11 07:34:49 2009 (49E02BF9) > fffffa60`009e1000 fffffa60`009f2000 Npfs Sat Apr 11 06:54:22 2009 (49E0227E) > fffffa60`009f2000 fffffa60`009fb000 rasacd Sat Jan 19 07:37:30 2008 (47919AAA) > fffffa60`00a02000 fffffa60`00a5b000 NETIO Sat Apr 11 07:43:34 2009 (49E02E06) > fffffa60`00a5b000 fffffa60`00b35000 Wdf01000 Sat Jan 19 07:33:27 2008 (479199B7) > fffffa60`00b35000 fffffa60`00b43000 WDFLDR Sat Jan 19 07:32:33 2008 (47919981) > fffffa60`00b43000 fffffa60`00b99000 acpi Sat Apr 11 07:03:26 2009 (49E0249E) > fffffa60`00b99000 fffffa60`00ba2000 WMILIB Sat Jan 19 07:33:45 2008 (479199C9) > fffffa60`00ba2000 fffffa60`00bac000 msisadrv Sat Jan 19 07:02:50 2008 (4791928A) > fffffa60`00bac000 fffffa60`00bdc000 pci Sat Apr 11 07:03:33 2009 (49E024A5) > fffffa60`00bdc000 fffffa60`00bf1000 partmgr Sat Apr 11 07:34:51 2009 (49E02BFB) > fffffa60`00bf1000 fffffa60`00bfc000 Msfs Sat Jan 19 06:53:55 2008 (47919073) > fffffa60`00c00000 fffffa60`00c09000 rdpencdd Sat Jan 19 07:42:03 2008 (47919BBB) > fffffa60`00c0d000 fffffa60`00c73000 volmgrx Sat Apr 11 07:35:14 2009 (49E02C12) > fffffa60`00c73000 fffffa60`00c7b000 intelide Sat Jan 19 07:28:56 2008 (479198A8) > fffffa60`00c7b000 fffffa60`00c8b000 PCIIDEX Sat Apr 11 07:34:22 2009 (49E02BDE) > fffffa60`00c8b000 fffffa60`00c9e000 mountmgr Sat Jan 19 07:28:01 2008 (47919871) > fffffa60`00c9e000 fffffa60`00ca6000 atapi Sat Apr 11 07:34:19 2009 (49E02BDB) > fffffa60`00ca6000 fffffa60`00cca000 ataport Sat Apr 11 07:34:25 2009 (49E02BE1) > fffffa60`00cca000 fffffa60`00cd3000 viostor Tue Aug 10 15:58:06 2010 (4C615AEE) > fffffa60`00cd3000 fffffa60`00d30000 storport Sat Apr 11 07:34:45 2009 (49E02BF5) > fffffa60`00d30000 fffffa60`00d77000 fltmgr Sat Apr 11 06:54:38 2009 (49E0228E) > fffffa60`00d77000 fffffa60`00da2000 quota Sat Apr 11 06:59:38 2009 (49E023BA) > fffffa60`00da2000 fffffa60`00db7000 datascrn Sat Apr 11 06:59:35 2009 (49E023B7) > fffffa60`00db7000 fffffa60`00df0000 msiscsi Sat Apr 11 07:36:09 2009 (49E02C49) > fffffa60`00df0000 fffffa60`00dfe000 vga Sat Jan 19 07:32:21 2008 (47919975) > fffffa60`00e00000 fffffa60`00e0c000 ndistapi Sat Jan 19 07:37:22 2008 (47919AA2) > fffffa60`00e0d000 fffffa60`00e94000 ksecdd Mon Jun 15 15:15:18 2009 (4A364966) > fffffa60`00e94000 fffffa60`00eaa000 i8042prt Sat Jan 19 07:28:08 2008 (47919878) > fffffa60`00eaa000 fffffa60`00eb8000 kbdclass Sat Jan 19 07:28:05 2008 (47919875) > fffffa60`00eb8000 fffffa60`00ec4000 mouclass Sat Jan 19 07:28:05 2008 (47919875) > fffffa60`00ec4000 fffffa60`00ed1000 fdc Sat Jan 19 07:28:45 2008 (4791989D) > fffffa60`00ed1000 fffffa60`00eed000 cdrom Sat Apr 11 07:34:39 2009 (49E02BEF) > fffffa60`00eed000 fffffa60`00ef9000 usbuhci Sat Jan 19 07:33:56 2008 (479199D4) > fffffa60`00ef9000 fffffa60`00f3f000 USBPORT Sat Apr 11 07:39:39 2009 (49E02D1B) > fffffa60`00f3f000 fffffa60`00f4d000 vgapnp Sat Jan 19 07:32:22 2008 (47919976) > fffffa60`00f4d000 fffffa60`00f72000 VIDEOPRT Sat Jan 19 07:32:25 2008 (47919979) > fffffa60`00f72000 fffffa60`00f82000 watchdog Sat Apr 11 07:09:16 2009 (49E025FC) > fffffa60`00f82000 fffffa60`00fa5a00 E1G6032E Wed Sep 24 01:09:49 2008 (48D9773D) > fffffa60`00fa6000 fffffa60`00fb9000 intelppm Sat Jan 19 06:52:45 2008 (4791902D) > fffffa60`00fb9000 fffffa60`00fcc000 blbdrive Sat Jan 19 06:58:37 2008 (4791918D) > fffffa60`00fcc000 fffffa60`00fd9000 TDI Sat Apr 11 07:44:14 2009 (49E02E2E) > fffffa60`00fd9000 fffffa60`00ffc000 rasl2tp Sat Apr 11 07:43:37 2009 (49E02E09) > fffffa60`01003000 fffffa60`01179000 tcpip Wed Jun 16 16:51:56 2010 (4C18E50C) > fffffa60`01179000 fffffa60`011a5000 fwpkclnt Sat Apr 11 07:42:44 2009 (49E02DD4) > fffffa60`011a5000 fffffa60`011d1000 CLASSPNP Sat Apr 11 07:34:15 2009 (49E02BD7) > fffffa60`011d1000 fffffa60`011db000 crcdisk Sat Jan 19 07:30:12 2008 (479198F4) > fffffa60`011f3000 fffffa60`011fc000 tunmp Sat Jan 19 07:36:30 2008 (47919A6E) > fffffa60`01209000 fffffa60`01389000 Ntfs Sat Apr 11 06:55:38 2009 (49E022CA) > fffffa60`01389000 fffffa60`013cd000 volsnap Sat Apr 11 07:35:33 2009 (49E02C25) > fffffa60`013cd000 fffffa60`013d5000 spldr Mon Jan 12 22:51:29 2009 (496BBB61) > fffffa60`013d5000 fffffa60`013e7000 mup Sat Apr 11 06:54:47 2009 (49E02297) > fffffa60`013e7000 fffffa60`013fb000 disk Sat Apr 11 07:34:38 2009 (49E02BEE) > fffffa60`03600000 fffffa60`03631000 ndiswan Sat Apr 11 07:43:39 2009 (49E02E0B) > fffffa60`03631000 fffffa60`03641000 raspppoe Sat Apr 11 07:43:33 2009 (49E02E05) > fffffa60`03641000 fffffa60`0365f000 raspptp Sat Apr 11 07:43:38 2009 (49E02E0A) > fffffa60`0365f000 fffffa60`03677000 rassstp Sat Apr 11 07:43:46 2009 (49E02E12) > fffffa60`03677000 fffffa60`03711000 rdpdr Sat Apr 11 07:49:54 2009 (49E02F82) > fffffa60`03711000 fffffa60`03724000 termdd Sat Apr 11 07:48:13 2009 (49E02F1D) > fffffa60`03724000 fffffa60`03725480 swenum Thu Nov 02 10:37:33 2006 (4549BC5D) > fffffa60`03726000 fffffa60`0375a000 ks Sat Apr 11 07:33:51 2009 (49E02BBF) > fffffa60`0375a000 fffffa60`03765000 mssmbios Sat Jan 19 07:02:54 2008 (4791928E) > fffffa60`03765000 fffffa60`03775000 umbus Sat Jan 19 07:34:16 2008 (479199E8) > fffffa60`03775000 fffffa60`037bd000 usbhub Sat Apr 11 07:39:52 2009 (49E02D28) > fffffa60`037bd000 fffffa60`037d1000 NDProxy Sat Jan 19 07:37:26 2008 (47919AA6) > fffffa60`037d1000 fffffa60`037e1000 dfs Sat Jan 19 06:54:14 2008 (47919086) > fffffa60`037e1000 fffffa60`037eb000 Fs_Rec Sat Jan 19 06:53:41 2008 (47919065) > fffffa60`037eb000 fffffa60`037f4000 Null Thu Nov 02 10:37:15 2006 (4549BC4B) > fffffa60`037f4000 fffffa60`037fd000 RDPCDD Sat Jan 19 07:42:04 2008 (47919BBC) > fffffa60`037fd000 fffffa60`037fee00 USBD Sat Jan 19 07:33:53 2008 (479199D1) > fffffa60`03c04000 fffffa60`03c1f000 smb Sat Apr 11 07:42:19 2009 (49E02DBB) > fffffa60`03c1f000 fffffa60`03c8a000 afd Sat Apr 11 07:44:21 2009 (49E02E35) > fffffa60`03c8a000 fffffa60`03cce000 netbt Sat Apr 11 07:42:31 2009 (49E02DC7) > fffffa60`03cce000 fffffa60`03cec000 pacer Sat Apr 11 07:42:56 2009 (49E02DE0) > fffffa60`03cec000 fffffa60`03cfb000 netbios Sat Jan 19 07:36:35 2008 (47919A73) > fffffa60`03cfb000 fffffa60`03d0e000 monitor Sat Jan 19 07:32:34 2008 (47919982) > fffffa60`03d18000 fffffa60`03d33000 wanarp Sat Apr 11 07:43:38 2009 (49E02E0A) > fffffa60`03d33000 fffffa60`03d80000 rdbss Sat Apr 11 06:55:21 2009 (49E022B9) > fffffa60`03d80000 fffffa60`03d8c000 nsiproxy Sat Jan 19 07:36:45 2008 (47919A7D) > fffffa60`03d8c000 fffffa60`03da9000 dfsc Sat Apr 11 06:54:44 2009 (49E02294) > fffffa60`03da9000 fffffa60`03dc5000 cdfs Sat Jan 19 06:53:45 2008 (47919069) > fffffa60`03dc5000 fffffa60`03dd3000 crashdmp Sat Apr 11 07:34:31 2009 (49E02BE7) > fffffa60`03dd3000 fffffa60`03ddd000 dump_diskdump Sat Apr 11 07:34:30 2009 > (49E02BE6) > fffffa60`03ddd000 fffffa60`03de6000 dump_viostor Tue Aug 10 15:58:06 2010 > (4C615AEE) > fffffa60`03de6000 fffffa60`03df2000 Dxapi Sat Jan 19 07:08:00 2008 (479193C0) > fffffa60`04204000 fffffa60`04226000 luafv Sat Jan 19 06:59:06 2008 (479191AA) > fffffa60`04226000 fffffa60`0423a000 lltdio Sat Jan 19 07:35:48 2008 (47919A44) > fffffa60`0423a000 fffffa60`04252000 rspndr Sat Jan 19 07:35:48 2008 (47919A44) > fffffa60`04252000 fffffa60`042ec000 spsys Tue Mar 10 18:16:43 2009 (49B6A07B) > fffffa60`042ec000 fffffa60`0430a000 bowser Sat Jan 19 06:54:51 2008 (479190AB) > fffffa60`0430a000 fffffa60`04324000 mpsdrv Sat Jan 19 07:35:28 2008 (47919A30) > fffffa60`04324000 fffffa60`0434d000 mrxsmb Tue Feb 23 12:32:22 2010 (4B83BCC6) > fffffa60`0434d000 fffffa60`04396000 mrxsmb10 Tue Feb 23 12:32:21 2010 (4B83BCC5) > fffffa60`04396000 fffffa60`043b5000 mrxsmb20 Tue Feb 23 12:32:20 2010 (4B83BCC4) > fffffa60`043b5000 fffffa60`043c0000 asyncmac Sat Jan 19 07:37:27 2008 (47919AA7) > fffffa60`043c0000 fffffa60`043e9000 srvnet Mon Sep 06 17:33:48 2010 (4C8509DC) > fffffa60`04806000 fffffa60`04838000 srv2 Mon Sep 06 17:33:50 2010 (4C8509DE) > fffffa60`04838000 fffffa60`048cc000 srv Mon Sep 06 17:34:12 2010 (4C8509F4) > fffffa60`048cc000 fffffa60`0496f000 HTTP Sat Feb 20 22:30:05 2010 (4B80545D) > fffffa60`0500e000 fffffa60`050c4000 peauth Mon Oct 23 13:57:00 2006 (453CAE0C) > fffffa60`050c4000 fffffa60`050cf000 secdrv Wed Sep 13 15:18:38 2006 (4508052E) > fffffa60`050cf000 fffffa60`050df000 tcpipreg Tue Dec 08 18:55:32 2009 (4B1E9314) > fffffa60`050df000 fffffa60`050ec000 tdtcp Sat Jan 19 07:42:03 2008 (47919BBB) > fffffa60`050ec000 fffffa60`050fa000 tssecsrv Sat Jan 19 07:42:10 2008 (47919BC2) > fffffa60`050fa000 fffffa60`05136000 RDPWD Sat Apr 11 07:48:33 2009 (49E02F31) > > Unloaded modules: > fffffa60`011db000 fffffa60`011e9000 crashdmp.sys > Timestamp: unavailable (00000000) > Checksum: 00000000 > ImageSize: 0000E000 > fffffa60`011e9000 fffffa60`011f3000 dump_storpor > Timestamp: unavailable (00000000) > Checksum: 00000000 > ImageSize: 0000A000 > fffffa60`01200000 fffffa60`01209000 dump_viostor > Timestamp: unavailable (00000000) > Checksum: 00000000 > ImageSize: 00009000 > fffffa60`03cfb000 fffffa60`03d18000 serial.sys > Timestamp: unavailable (00000000) > Checksum: 00000000 > ImageSize: 0001D000 > fffffa60`0076c000 fffffa60`00788000 sacdrv.sys > Timestamp: unavailable (00000000) > Checksum: 00000000 > ImageSize: 0001C000 > ************************************************************************** > * * > * Bugcheck Analysis * > * * > ************************************************************************** > > Use !analyze -v to get detailed debugging information. > > BugCheck 3B, {c0000005, fffff80001e6a842, fffffa6004efcb70, 0} > > Probably caused by : Npfs.SYS ( Npfs!NpSetAttributeInList+9a ) > > Followup: MachineOwner > --------- > > Finished dump check > > > > best regards > Manfred > NrybXǧv^){.n+h\x17ܨ}Ơz&j:+v\azZ++zfh~iz\x1ew?&)ߢ^[f ^ permalink raw reply [flat|nested] 22+ messages in thread
end of thread, other threads:[~2010-12-15 10:49 UTC | newest] Thread overview: 22+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2010-07-15 13:19 Freezing Windows 2008 x64bit guest Christoph Adomeit 2010-07-15 13:44 ` Gleb Natapov 2010-07-19 7:17 ` Harri Olin 2010-07-19 7:42 ` Gleb Natapov 2010-07-21 6:25 ` Harri Olin 2010-07-21 8:28 ` Christoph Adomeit 2010-07-21 8:37 ` Gleb Natapov 2010-07-21 9:22 ` Harri Olin 2010-07-21 9:48 ` Gleb Natapov 2010-07-21 10:09 ` Harri Olin 2010-07-21 10:30 ` Gleb Natapov 2010-07-21 10:43 ` Harri Olin 2010-07-21 13:03 ` Harri Olin 2010-07-21 13:45 ` Gleb Natapov 2010-07-21 14:05 ` Harri Olin 2010-07-27 21:53 ` Harri Olin 2010-07-28 15:18 ` Gleb Natapov 2010-12-13 19:42 ` Manfred Heubach 2010-12-13 20:12 ` Dor Laor 2010-12-13 20:44 ` Vadim Rozenfeld 2010-12-14 23:57 ` AW: " Manfred Heubach 2010-12-15 10:48 ` Vadim Rozenfeld
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox