Freezing Windows 2008 x64bit guest

public inbox for kvm@vger.kernel.org
 help / color / mirror / Atom feed

* Freezing Windows 2008 x64bit guest
@ 2010-07-15 13:19 Christoph Adomeit
  2010-07-15 13:44 ` Gleb Natapov
  0 siblings, 1 reply; 22+ messages in thread
From: Christoph Adomeit @ 2010-07-15 13:19 UTC (permalink / raw)
  To: kvm

Hi,

we are running several kvm servers with kvm 0.12.4 and kernel 2.6.32
(actually we run proxmox ve)

These proxmox servers are running all kinds of vms rock-solid, be it
linux or windows 32 platforms.

But one Windows 2008 64 Bit Server Standard is freezing regularly.
This happens sometimes 3 times a day, sometimes it takes 2 days
until freeze. The Windows Machine is a clean fresh install.

By Freezing I mean that the Machine console still shows the 
windows logo but does not accept any keystroke and does not answer
to ping or rdp or whatever. The virtual cpu-load is then shown
as 100% (by proxmox).

We tried several combinations of ide or virtio, several virtual network cards (intel and realtek) and single-cpu vs multi cpu. The Problem persists.

The Windows Machine is a quite fresh install but runs exchange and 10 rdp sessions.

We are experienced linux-admins but we dont find any hints on the kvm servers what might be the problem, no syslogs,dmesgs, nothing. Also there are no 
dumps on the windows guest.

Do you have any hints what might be the problem and what we could do to debug the problem and to get more information of the root cause ?

Thanks
  Christoph

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: Freezing Windows 2008 x64bit guest
  2010-07-15 13:19 Freezing Windows 2008 x64bit guest Christoph Adomeit
@ 2010-07-15 13:44 ` Gleb Natapov
  2010-07-19  7:17   ` Harri Olin
  0 siblings, 1 reply; 22+ messages in thread
From: Gleb Natapov @ 2010-07-15 13:44 UTC (permalink / raw)
  To: Christoph Adomeit; +Cc: kvm

On Thu, Jul 15, 2010 at 03:19:44PM +0200, Christoph Adomeit wrote:
> Hi,
> 
> we are running several kvm servers with kvm 0.12.4 and kernel 2.6.32
> (actually we run proxmox ve)
> 
> These proxmox servers are running all kinds of vms rock-solid, be it
> linux or windows 32 platforms.
> 
> But one Windows 2008 64 Bit Server Standard is freezing regularly.
> This happens sometimes 3 times a day, sometimes it takes 2 days
> until freeze. The Windows Machine is a clean fresh install.
> 
> By Freezing I mean that the Machine console still shows the 
> windows logo but does not accept any keystroke and does not answer
> to ping or rdp or whatever. The virtual cpu-load is then shown
> as 100% (by proxmox).
> 
> We tried several combinations of ide or virtio, several virtual network cards (intel and realtek) and single-cpu vs multi cpu. The Problem persists.
> 
> The Windows Machine is a quite fresh install but runs exchange and 10 rdp sessions.
> 
> We are experienced linux-admins but we dont find any hints on the kvm servers what might be the problem, no syslogs,dmesgs, nothing. Also there are no 
> dumps on the windows guest.
> 
> Do you have any hints what might be the problem and what we could do to debug the problem and to get more information of the root cause ?
> 
When hang occurs ensure that problematic vm is the only on on the server and run
kvm_stat. Send output here. Also do the following:

# mount -t debugfs debugfs /sys/kernel/debug
# echo kvm > /sys/kernel/debug/tracing/set_event
# sleep 1
# cat /sys/kernel/debug/tracing/trace > /tmp/trace

Send /tmp/trace here too, but it may be huge, so send only last 1000
lines.

--
			Gleb.

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: Freezing Windows 2008 x64bit guest
  2010-07-15 13:44 ` Gleb Natapov
@ 2010-07-19  7:17   ` Harri Olin
  2010-07-19  7:42     ` Gleb Natapov
  0 siblings, 1 reply; 22+ messages in thread
From: Harri Olin @ 2010-07-19  7:17 UTC (permalink / raw)
  To: Gleb Natapov; +Cc: Christoph Adomeit, kvm@vger.kernel.org

Gleb Natapov kirjoitti:
> On Thu, Jul 15, 2010 at 03:19:44PM +0200, Christoph Adomeit wrote:
>>
>> But one Windows 2008 64 Bit Server Standard is freezing regularly.
>> This happens sometimes 3 times a day, sometimes it takes 2 days
>> until freeze. The Windows Machine is a clean fresh install.
>>
>> By Freezing I mean that the Machine console still shows the 
>> windows logo but does not accept any keystroke and does not answer
>> to ping or rdp or whatever. The virtual cpu-load is then shown
>> as 100% (by proxmox).

Hi

I think I have seen same problem occur on my Windows 2008 SBS SP2 64bit 
system, but a bit less often, only like once a week.

Short description of system:
Supermicro X7SBL-LN2 motherboard, Q8300 cpu, 8GB memory, Areca ARC-1210 
RAID controller.

Linux kernel 2.6.33.1, qemu-kvm-0.11.1, windows 2008 quest is the only 
one at the moment.

I had to step down to 0.11.1 from 0.12.x as that was more unstable, 
windows 2008 bluescreened every second day. Also 0.11.1 did the same 
when I applied -rtc-td-hack, I think. (might also be -cpu host, which I 
removed at the same time but I think the crashing started when I added 
-rt-td-hack).
Sample bluescreen error codes from that time:
0x0000003b (0x00000000c0000005, 0xfffff80001cb8842, 0xfffffa600
0x0000003b (0x00000000c0000005, 0xfffff80001c9c842, 0xfffffa600
0x0000000a (0x0000000000000080, 0x000000000000000c, 0x000000000
0x0000003b (0x00000000c0000005, 0xfffff80001c5d842, 0xfffffa600
Also I think I have seen qemu crashing once..

Now I haven't seen crashes but only freezes with qemu on 100% and 
virtual system unresponsive.

qemu command line:
   /usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive 
file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive 
file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net 
nic,macaddr=C0:FF:12:FB:AA:01,model=e1000 -net tap -smp 4 -localtime


> When hang occurs ensure that problematic vm is the only on on the server and run
> kvm_stat. Send output here. Also do the following:
> 
> # mount -t debugfs debugfs /sys/kernel/debug
> # echo kvm > /sys/kernel/debug/tracing/set_event
> # sleep 1
> # cat /sys/kernel/debug/tracing/trace > /tmp/trace
> 
> Send /tmp/trace here too, but it may be huge, so send only last 1000
> lines.

kvm_stat logging output during hang. logging output at 
http://mizar.remote.agasha.com/k/kvm/kvm_stat_hang.txt

  efer_reload                  0        0
  exits                 17193383	   14804
  fpu_reload             2346791     2089
  halt_exits              213095        0
  halt_wakeup             191350        0
  host_state_reload      4692238     2089
  hypercalls                   0        0
  insn_emulation         6726836     4012
  insn_emulation_fail          0        0
  invlpg                  217889        0
  io_exits               1755022        0
  irq_exits               145317     9409
  irq_injections          296911     4006
  irq_window               28551     1381
  largepages                   0        0
  mmio_exits              505162        0
  mmu_cache_miss           56734        0
  mmu_flooded              12110        0
  mmu_pde_zapped           22167        0
  mmu_pte_updated          99276        0
  mmu_pte_write            63788        0
  mmu_recycled                 0        0

And here's the trace:
also readable at http://mizar.remote.agasha.com/k/kvm/kvm_trace_hang.txt

  qemu-system-x86-4586  [003] 1076789.993832: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e891
  qemu-system-x86-4586  [003] 1076789.993832: kvm_entry: vcpu 0
  qemu-system-x86-4580  [002] 1076789.994369: kvm_set_irq: gsi 0 level 1 
source 0
  qemu-system-x86-4580  [002] 1076789.994370: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4580  [002] 1076789.994370: kvm_apic_accept_irq: 
apicid 0 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076789.994371: kvm_apic_accept_irq: 
apicid 1 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076789.994371: kvm_apic_accept_irq: 
apicid 2 vec 209 (Fixed|edge)
  qemu-system-x86-4586  [003] 1076789.994371: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e891
  qemu-system-x86-4580  [002] 1076789.994372: kvm_apic_accept_irq: 
apicid 3 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076789.994372: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4586  [003] 1076789.994372: kvm_inj_virq: irq 209
  qemu-system-x86-4586  [003] 1076789.994372: kvm_entry: vcpu 0
  qemu-system-x86-4588  [000] 1076789.994372: kvm_exit: reason ext_irq 
rip 0xfffff80001ca12ee
  qemu-system-x86-4580  [002] 1076789.994373: kvm_set_irq: gsi 0 level 0 
source 0
  qemu-system-x86-4580  [002] 1076789.994373: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4580  [002] 1076789.994373: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4588  [000] 1076789.994373: kvm_inj_virq: irq 209
  qemu-system-x86-4589  [001] 1076789.994373: kvm_exit: reason ext_irq 
rip 0xfffff80001c9b859
  qemu-system-x86-4588  [000] 1076789.994373: kvm_entry: vcpu 2
  qemu-system-x86-4586  [003] 1076789.994374: kvm_exit: reason 
apic_access rip 0xfffff80001c9a5c4
  qemu-system-x86-4589  [001] 1076789.994374: kvm_inj_virq: irq 209
  qemu-system-x86-4589  [001] 1076789.994374: kvm_entry: vcpu 3
  qemu-system-x86-4588  [000] 1076789.994375: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4586  [003] 1076789.994375: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4586  [003] 1076789.994376: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4589  [001] 1076789.994376: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4586  [003] 1076789.994376: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4586  [003] 1076789.994377: kvm_entry: vcpu 0
  qemu-system-x86-4588  [000] 1076789.994377: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4588  [000] 1076789.994377: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4589  [001] 1076789.994377: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4589  [001] 1076789.994378: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4588  [000] 1076789.994378: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4589  [001] 1076789.994378: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4588  [000] 1076789.994379: kvm_entry: vcpu 2
  qemu-system-x86-4589  [001] 1076789.994379: kvm_entry: vcpu 3
  qemu-system-x86-4587  [002] 1076789.994380: kvm_exit: reason ext_irq 
rip 0xfffff80001c9b859
  qemu-system-x86-4587  [002] 1076789.994381: kvm_inj_virq: irq 209
  qemu-system-x86-4587  [002] 1076789.994382: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076789.994384: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4587  [002] 1076789.994386: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4587  [002] 1076789.994386: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4587  [002] 1076789.994386: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4587  [002] 1076789.994387: kvm_entry: vcpu 1
  qemu-system-x86-4588  [000] 1076789.994457: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ffb6
  qemu-system-x86-4588  [000] 1076789.994457: kvm_entry: vcpu 2
  qemu-system-x86-4589  [001] 1076789.994582: kvm_exit: reason ext_irq 
rip 0xfffff80001c9b859
  qemu-system-x86-4588  [000] 1076789.994582: kvm_exit: reason ext_irq 
rip 0xfffff80001cd9121
  qemu-system-x86-4589  [001] 1076789.994582: kvm_entry: vcpu 3
  qemu-system-x86-4588  [000] 1076789.994583: kvm_entry: vcpu 2
  qemu-system-x86-4587  [002] 1076789.994634: kvm_exit: reason ext_irq 
rip 0xfffff80001ca00d5
  qemu-system-x86-4587  [002] 1076789.994635: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076789.994706: kvm_exit: reason ext_irq 
rip 0xfffff80001cda795
  qemu-system-x86-4587  [002] 1076789.994706: kvm_entry: vcpu 1
  qemu-system-x86-4586  [003] 1076789.994830: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e891
  qemu-system-x86-4586  [003] 1076789.994831: kvm_entry: vcpu 0
  qemu-system-x86-4580  [002] 1076789.995367: kvm_set_irq: gsi 0 level 1 
source 0
  qemu-system-x86-4580  [002] 1076789.995367: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4580  [002] 1076789.995368: kvm_apic_accept_irq: 
apicid 0 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076789.995368: kvm_apic_accept_irq: 
apicid 1 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076789.995369: kvm_apic_accept_irq: 
apicid 2 vec 209 (Fixed|edge)
  qemu-system-x86-4586  [003] 1076789.995369: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e891
  qemu-system-x86-4580  [002] 1076789.995369: kvm_apic_accept_irq: 
apicid 3 vec 209 (Fixed|edge)
  qemu-system-x86-4586  [003] 1076789.995370: kvm_inj_virq: irq 209
  qemu-system-x86-4580  [002] 1076789.995370: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4586  [003] 1076789.995370: kvm_entry: vcpu 0
  qemu-system-x86-4580  [002] 1076789.995370: kvm_set_irq: gsi 0 level 0 
source 0
  qemu-system-x86-4588  [000] 1076789.995370: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4580  [002] 1076789.995370: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4580  [002] 1076789.995371: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4589  [001] 1076789.995371: kvm_exit: reason ext_irq 
rip 0xfffff80001cd8bac
  qemu-system-x86-4588  [000] 1076789.995371: kvm_entry: vcpu 2
  qemu-system-x86-4589  [001] 1076789.995372: kvm_inj_virq: irq 209
  qemu-system-x86-4586  [003] 1076789.995372: kvm_exit: reason 
apic_access rip 0xfffff80001c9a5c4
  qemu-system-x86-4589  [001] 1076789.995372: kvm_entry: vcpu 3
  qemu-system-x86-4588  [000] 1076789.995372: kvm_exit: reason 
interrupt_window rip 0xfffff80001ca0002
  qemu-system-x86-4588  [000] 1076789.995373: kvm_inj_virq: irq 209
  qemu-system-x86-4586  [003] 1076789.995373: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4588  [000] 1076789.995373: kvm_entry: vcpu 2
  qemu-system-x86-4586  [003] 1076789.995373: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4589  [001] 1076789.995373: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4586  [003] 1076789.995374: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4586  [003] 1076789.995375: kvm_entry: vcpu 0
  qemu-system-x86-4588  [000] 1076789.995375: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4589  [001] 1076789.995375: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4589  [001] 1076789.995375: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4589  [001] 1076789.995376: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4588  [000] 1076789.995376: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4588  [000] 1076789.995377: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4589  [001] 1076789.995377: kvm_entry: vcpu 3
  qemu-system-x86-4588  [000] 1076789.995377: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4588  [000] 1076789.995378: kvm_entry: vcpu 2
  qemu-system-x86-4587  [002] 1076789.995378: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4587  [002] 1076789.995379: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076789.995381: kvm_exit: reason 
interrupt_window rip 0xfffff80001ca0002
  qemu-system-x86-4587  [002] 1076789.995381: kvm_inj_virq: irq 209
  qemu-system-x86-4587  [002] 1076789.995381: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076789.995383: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4587  [002] 1076789.995385: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4587  [002] 1076789.995385: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4587  [002] 1076789.995386: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4587  [002] 1076789.995387: kvm_entry: vcpu 1
  qemu-system-x86-4588  [000] 1076789.995455: kvm_exit: reason ext_irq 
rip 0xfffff80001cde8ec
  qemu-system-x86-4588  [000] 1076789.995456: kvm_entry: vcpu 2
  qemu-system-x86-4589  [001] 1076789.995580: kvm_exit: reason ext_irq 
rip 0xfffff80001d019f1
  qemu-system-x86-4589  [001] 1076789.995580: kvm_entry: vcpu 3
  qemu-system-x86-4587  [002] 1076789.995632: kvm_exit: reason ext_irq 
rip 0xfffff80001ca138b
  qemu-system-x86-4587  [002] 1076789.995633: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076789.995704: kvm_exit: reason ext_irq 
rip 0xfffff80001d01a00
  qemu-system-x86-4587  [002] 1076789.995705: kvm_entry: vcpu 1
  qemu-system-x86-4586  [003] 1076789.995829: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e891
  qemu-system-x86-4586  [003] 1076789.995829: kvm_entry: vcpu 0
  qemu-system-x86-4580  [002] 1076789.996365: kvm_set_irq: gsi 0 level 1 
source 0
  qemu-system-x86-4580  [002] 1076789.996366: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4580  [002] 1076789.996367: kvm_apic_accept_irq: 
apicid 0 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076789.996367: kvm_apic_accept_irq: 
apicid 1 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076789.996367: kvm_apic_accept_irq: 
apicid 2 vec 209 (Fixed|edge)
  qemu-system-x86-4586  [003] 1076789.996368: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e88f
  qemu-system-x86-4580  [002] 1076789.996368: kvm_apic_accept_irq: 
apicid 3 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076789.996368: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4586  [003] 1076789.996368: kvm_inj_virq: irq 209
  qemu-system-x86-4586  [003] 1076789.996369: kvm_entry: vcpu 0
  qemu-system-x86-4588  [000] 1076789.996369: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4580  [002] 1076789.996369: kvm_set_irq: gsi 0 level 0 
source 0
  qemu-system-x86-4580  [002] 1076789.996369: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4580  [002] 1076789.996369: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4589  [001] 1076789.996370: kvm_exit: reason ext_irq 
rip 0xfffff80001c9b859
  qemu-system-x86-4588  [000] 1076789.996370: kvm_entry: vcpu 2
  qemu-system-x86-4589  [001] 1076789.996370: kvm_inj_virq: irq 209
  qemu-system-x86-4586  [003] 1076789.996370: kvm_exit: reason 
apic_access rip 0xfffff80001c9a5c4
  qemu-system-x86-4589  [001] 1076789.996371: kvm_entry: vcpu 3
  qemu-system-x86-4588  [000] 1076789.996371: kvm_exit: reason 
interrupt_window rip 0xfffff80001ca0002
  qemu-system-x86-4588  [000] 1076789.996371: kvm_inj_virq: irq 209
  qemu-system-x86-4588  [000] 1076789.996372: kvm_entry: vcpu 2
  qemu-system-x86-4586  [003] 1076789.996372: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4586  [003] 1076789.996372: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4589  [001] 1076789.996372: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4586  [003] 1076789.996373: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4588  [000] 1076789.996373: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4586  [003] 1076789.996373: kvm_entry: vcpu 0
  qemu-system-x86-4589  [001] 1076789.996374: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4589  [001] 1076789.996374: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4589  [001] 1076789.996374: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4588  [000] 1076789.996375: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4588  [000] 1076789.996375: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4588  [000] 1076789.996375: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4589  [001] 1076789.996376: kvm_entry: vcpu 3
  qemu-system-x86-4588  [000] 1076789.996376: kvm_entry: vcpu 2
  qemu-system-x86-4587  [002] 1076789.996377: kvm_exit: reason ext_irq 
rip 0xfffff80001ca010b
  qemu-system-x86-4587  [002] 1076789.996378: kvm_inj_virq: irq 209
  qemu-system-x86-4587  [002] 1076789.996378: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076789.996380: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4587  [002] 1076789.996382: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4587  [002] 1076789.996382: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4587  [002] 1076789.996382: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4587  [002] 1076789.996384: kvm_entry: vcpu 1
  qemu-system-x86-4588  [000] 1076789.996453: kvm_exit: reason ext_irq 
rip 0xfffff80001cd916f
  qemu-system-x86-4588  [000] 1076789.996454: kvm_entry: vcpu 2
  qemu-system-x86-4589  [001] 1076789.996578: kvm_exit: reason ext_irq 
rip 0xfffff80001cd8c58
  qemu-system-x86-4589  [001] 1076789.996579: kvm_entry: vcpu 3
  qemu-system-x86-4587  [002] 1076789.996630: kvm_exit: reason ext_irq 
rip 0xfffff80001d01a00
  qemu-system-x86-4587  [002] 1076789.996632: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076789.996703: kvm_exit: reason ext_irq 
rip 0xfffff80001c9b859
  qemu-system-x86-4587  [002] 1076789.996703: kvm_entry: vcpu 1
  qemu-system-x86-4586  [003] 1076789.996827: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e891
  qemu-system-x86-4586  [003] 1076789.996828: kvm_entry: vcpu 0
  qemu-system-x86-4580  [002] 1076789.997364: kvm_set_irq: gsi 0 level 1 
source 0
  qemu-system-x86-4580  [002] 1076789.997365: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4580  [002] 1076789.997365: kvm_apic_accept_irq: 
apicid 0 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076789.997366: kvm_apic_accept_irq: 
apicid 1 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076789.997366: kvm_apic_accept_irq: 
apicid 2 vec 209 (Fixed|edge)
  qemu-system-x86-4586  [003] 1076789.997367: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e894
  qemu-system-x86-4580  [002] 1076789.997367: kvm_apic_accept_irq: 
apicid 3 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076789.997367: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4586  [003] 1076789.997367: kvm_inj_virq: irq 209
  qemu-system-x86-4586  [003] 1076789.997367: kvm_entry: vcpu 0
  qemu-system-x86-4588  [000] 1076789.997368: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4580  [002] 1076789.997368: kvm_set_irq: gsi 0 level 0 
source 0
  qemu-system-x86-4580  [002] 1076789.997368: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4580  [002] 1076789.997368: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4588  [000] 1076789.997368: kvm_entry: vcpu 2
  qemu-system-x86-4589  [001] 1076789.997368: kvm_exit: reason ext_irq 
rip 0xfffff80001c9b85d
  qemu-system-x86-4586  [003] 1076789.997369: kvm_exit: reason 
apic_access rip 0xfffff80001c9a5c4
  qemu-system-x86-4589  [001] 1076789.997369: kvm_inj_virq: irq 209
  qemu-system-x86-4589  [001] 1076789.997369: kvm_entry: vcpu 3
  qemu-system-x86-4588  [000] 1076789.997369: kvm_exit: reason 
interrupt_window rip 0xfffff80001ca0002
  qemu-system-x86-4588  [000] 1076789.997370: kvm_inj_virq: irq 209
  qemu-system-x86-4588  [000] 1076789.997371: kvm_entry: vcpu 2
  qemu-system-x86-4586  [003] 1076789.997371: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4589  [001] 1076789.997371: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4586  [003] 1076789.997371: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4586  [003] 1076789.997371: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4588  [000] 1076789.997372: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4586  [003] 1076789.997372: kvm_entry: vcpu 0
  qemu-system-x86-4589  [001] 1076789.997373: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4589  [001] 1076789.997373: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4589  [001] 1076789.997373: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4588  [000] 1076789.997374: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4588  [000] 1076789.997374: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4588  [000] 1076789.997374: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4589  [001] 1076789.997374: kvm_entry: vcpu 3
  qemu-system-x86-4588  [000] 1076789.997375: kvm_entry: vcpu 2
  qemu-system-x86-4587  [002] 1076789.997376: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4587  [002] 1076789.997377: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076789.997378: kvm_exit: reason 
interrupt_window rip 0xfffff80001ca0002
  qemu-system-x86-4587  [002] 1076789.997379: kvm_inj_virq: irq 209
  qemu-system-x86-4587  [002] 1076789.997379: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076789.997380: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4587  [002] 1076789.997382: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4587  [002] 1076789.997383: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4587  [002] 1076789.997383: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4587  [002] 1076789.997384: kvm_entry: vcpu 1
  qemu-system-x86-4588  [000] 1076789.997452: kvm_exit: reason ext_irq 
rip 0xfffff80001c9b859
  qemu-system-x86-4588  [000] 1076789.997452: kvm_entry: vcpu 2
  qemu-system-x86-4589  [001] 1076789.997576: kvm_exit: reason ext_irq 
rip 0xfffff80001d01a00
  qemu-system-x86-4589  [001] 1076789.997577: kvm_entry: vcpu 3
  qemu-system-x86-4587  [002] 1076789.997629: kvm_exit: reason ext_irq 
rip 0xfffff80001d01a00
  qemu-system-x86-4587  [002] 1076789.997630: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076789.997701: kvm_exit: reason ext_irq 
rip 0xfffff80001cd8c58
  qemu-system-x86-4587  [002] 1076789.997702: kvm_entry: vcpu 1
  qemu-system-x86-4586  [003] 1076789.997826: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e894
  qemu-system-x86-4586  [003] 1076789.997826: kvm_entry: vcpu 0
  qemu-system-x86-4580  [002] 1076789.998363: kvm_set_irq: gsi 0 level 1 
source 0
  qemu-system-x86-4580  [002] 1076789.998364: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4580  [002] 1076789.998364: kvm_apic_accept_irq: 
apicid 0 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076789.998365: kvm_apic_accept_irq: 
apicid 1 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076789.998365: kvm_apic_accept_irq: 
apicid 2 vec 209 (Fixed|edge)
  qemu-system-x86-4586  [003] 1076789.998365: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e894
  qemu-system-x86-4580  [002] 1076789.998366: kvm_apic_accept_irq: 
apicid 3 vec 209 (Fixed|edge)
  qemu-system-x86-4586  [003] 1076789.998366: kvm_inj_virq: irq 209
  qemu-system-x86-4580  [002] 1076789.998366: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4586  [003] 1076789.998366: kvm_entry: vcpu 0
  qemu-system-x86-4588  [000] 1076789.998367: kvm_exit: reason ext_irq 
rip 0xfffff80001cd90c2
  qemu-system-x86-4580  [002] 1076789.998367: kvm_set_irq: gsi 0 level 0 
source 0
  qemu-system-x86-4580  [002] 1076789.998367: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4580  [002] 1076789.998367: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4589  [001] 1076789.998367: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4588  [000] 1076789.998367: kvm_inj_virq: irq 209
  qemu-system-x86-4588  [000] 1076789.998368: kvm_entry: vcpu 2
  qemu-system-x86-4586  [003] 1076789.998368: kvm_exit: reason 
apic_access rip 0xfffff80001c9a5c4
  qemu-system-x86-4589  [001] 1076789.998368: kvm_entry: vcpu 3
  qemu-system-x86-4588  [000] 1076789.998369: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4589  [001] 1076789.998369: kvm_exit: reason 
interrupt_window rip 0xfffff80001ca0002
  qemu-system-x86-4586  [003] 1076789.998370: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4586  [003] 1076789.998370: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4589  [001] 1076789.998370: kvm_inj_virq: irq 209
  qemu-system-x86-4586  [003] 1076789.998370: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4589  [001] 1076789.998370: kvm_entry: vcpu 3
  qemu-system-x86-4588  [000] 1076789.998371: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4588  [000] 1076789.998371: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4586  [003] 1076789.998371: kvm_entry: vcpu 0
  qemu-system-x86-4588  [000] 1076789.998371: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4589  [001] 1076789.998372: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4588  [000] 1076789.998372: kvm_entry: vcpu 2
  qemu-system-x86-4589  [001] 1076789.998373: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4589  [001] 1076789.998374: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4589  [001] 1076789.998374: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4587  [002] 1076789.998374: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4589  [001] 1076789.998375: kvm_entry: vcpu 3
  qemu-system-x86-4587  [002] 1076789.998376: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076789.998377: kvm_exit: reason 
interrupt_window rip 0xfffff80001ca0002
  qemu-system-x86-4587  [002] 1076789.998378: kvm_inj_virq: irq 209
  qemu-system-x86-4587  [002] 1076789.998378: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076789.998379: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4587  [002] 1076789.998381: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4587  [002] 1076789.998382: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4587  [002] 1076789.998382: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4587  [002] 1076789.998383: kvm_entry: vcpu 1
  qemu-system-x86-4588  [000] 1076789.998450: kvm_exit: reason ext_irq 
rip 0xfffff80001ca136a
  qemu-system-x86-4588  [000] 1076789.998451: kvm_entry: vcpu 2
  qemu-system-x86-4589  [001] 1076789.998575: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4589  [001] 1076789.998576: kvm_entry: vcpu 3
  qemu-system-x86-4588  [000] 1076789.998576: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4588  [000] 1076789.998577: kvm_entry: vcpu 2
  qemu-system-x86-4587  [002] 1076789.998628: kvm_exit: reason ext_irq 
rip 0xfffff80001d01a00
  qemu-system-x86-4587  [002] 1076789.998629: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076789.998700: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4587  [002] 1076789.998700: kvm_entry: vcpu 1
  qemu-system-x86-4586  [003] 1076789.998824: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e891
  qemu-system-x86-4586  [003] 1076789.998824: kvm_entry: vcpu 0
  qemu-system-x86-4580  [002] 1076789.999361: kvm_set_irq: gsi 0 level 1 
source 0
  qemu-system-x86-4580  [002] 1076789.999362: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4580  [002] 1076789.999362: kvm_apic_accept_irq: 
apicid 0 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076789.999362: kvm_apic_accept_irq: 
apicid 1 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076789.999363: kvm_apic_accept_irq: 
apicid 2 vec 209 (Fixed|edge)
  qemu-system-x86-4586  [003] 1076789.999363: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e894
  qemu-system-x86-4580  [002] 1076789.999363: kvm_apic_accept_irq: 
apicid 3 vec 209 (Fixed|edge)
  qemu-system-x86-4586  [003] 1076789.999364: kvm_inj_virq: irq 209
  qemu-system-x86-4580  [002] 1076789.999364: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4586  [003] 1076789.999364: kvm_entry: vcpu 0
  qemu-system-x86-4588  [000] 1076789.999364: kvm_exit: reason ext_irq 
rip 0xfffff80001ca1311
  qemu-system-x86-4580  [002] 1076789.999364: kvm_set_irq: gsi 0 level 0 
source 0
  qemu-system-x86-4580  [002] 1076789.999365: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4580  [002] 1076789.999365: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4588  [000] 1076789.999365: kvm_inj_virq: irq 209
  qemu-system-x86-4589  [001] 1076789.999365: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4588  [000] 1076789.999366: kvm_entry: vcpu 2
  qemu-system-x86-4586  [003] 1076789.999366: kvm_exit: reason 
apic_access rip 0xfffff80001c9a5c4
  qemu-system-x86-4589  [001] 1076789.999366: kvm_entry: vcpu 3
  qemu-system-x86-4588  [000] 1076789.999367: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4589  [001] 1076789.999367: kvm_exit: reason 
interrupt_window rip 0xfffff80001ca0002
  qemu-system-x86-4586  [003] 1076789.999368: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4586  [003] 1076789.999368: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4589  [001] 1076789.999368: kvm_inj_virq: irq 209
  qemu-system-x86-4586  [003] 1076789.999368: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4589  [001] 1076789.999368: kvm_entry: vcpu 3
  qemu-system-x86-4588  [000] 1076789.999369: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4588  [000] 1076789.999369: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4586  [003] 1076789.999369: kvm_entry: vcpu 0
  qemu-system-x86-4588  [000] 1076789.999369: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4589  [001] 1076789.999370: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4588  [000] 1076789.999370: kvm_entry: vcpu 2
  qemu-system-x86-4589  [001] 1076789.999371: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4589  [001] 1076789.999372: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4589  [001] 1076789.999372: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4587  [002] 1076789.999372: kvm_exit: reason ext_irq 
rip 0xfffff80001c9b859
  qemu-system-x86-4589  [001] 1076789.999373: kvm_entry: vcpu 3
  qemu-system-x86-4587  [002] 1076789.999374: kvm_inj_virq: irq 209
  qemu-system-x86-4587  [002] 1076789.999374: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076789.999376: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4587  [002] 1076789.999378: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4587  [002] 1076789.999378: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4587  [002] 1076789.999378: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4587  [002] 1076789.999379: kvm_entry: vcpu 1
  qemu-system-x86-4588  [000] 1076789.999449: kvm_exit: reason ext_irq 
rip 0xfffff80001ca12d8
  qemu-system-x86-4588  [000] 1076789.999449: kvm_entry: vcpu 2
  qemu-system-x86-4589  [001] 1076789.999573: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4589  [001] 1076789.999574: kvm_entry: vcpu 3
  qemu-system-x86-4587  [002] 1076789.999626: kvm_exit: reason ext_irq 
rip 0xfffff80001c9b859
  qemu-system-x86-4587  [002] 1076789.999627: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076789.999698: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4587  [002] 1076789.999699: kvm_entry: vcpu 1
  qemu-system-x86-4586  [003] 1076789.999822: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e894
  qemu-system-x86-4586  [003] 1076789.999823: kvm_entry: vcpu 0
  qemu-system-x86-4580  [002] 1076790.000360: kvm_set_irq: gsi 0 level 1 
source 0
  qemu-system-x86-4580  [002] 1076790.000360: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4580  [002] 1076790.000361: kvm_apic_accept_irq: 
apicid 0 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076790.000361: kvm_apic_accept_irq: 
apicid 1 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076790.000362: kvm_apic_accept_irq: 
apicid 2 vec 209 (Fixed|edge)
  qemu-system-x86-4586  [003] 1076790.000362: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e891
  qemu-system-x86-4580  [002] 1076790.000362: kvm_apic_accept_irq: 
apicid 3 vec 209 (Fixed|edge)
  qemu-system-x86-4586  [003] 1076790.000363: kvm_inj_virq: irq 209
  qemu-system-x86-4586  [003] 1076790.000363: kvm_entry: vcpu 0
  qemu-system-x86-4580  [002] 1076790.000363: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4588  [000] 1076790.000363: kvm_exit: reason ext_irq 
rip 0xfffff80001cd8fa3
  qemu-system-x86-4580  [002] 1076790.000364: kvm_set_irq: gsi 0 level 0 
source 0
  qemu-system-x86-4588  [000] 1076790.000364: kvm_inj_virq: irq 209
  qemu-system-x86-4580  [002] 1076790.000364: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4589  [001] 1076790.000364: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4580  [002] 1076790.000364: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4588  [000] 1076790.000364: kvm_entry: vcpu 2
  qemu-system-x86-4586  [003] 1076790.000365: kvm_exit: reason 
apic_access rip 0xfffff80001c9a5c4
  qemu-system-x86-4589  [001] 1076790.000365: kvm_entry: vcpu 3
  qemu-system-x86-4588  [000] 1076790.000366: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4589  [001] 1076790.000366: kvm_exit: reason 
interrupt_window rip 0xfffff80001ca0002
  qemu-system-x86-4586  [003] 1076790.000366: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4586  [003] 1076790.000366: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4586  [003] 1076790.000367: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4589  [001] 1076790.000367: kvm_inj_virq: irq 209
  qemu-system-x86-4589  [001] 1076790.000367: kvm_entry: vcpu 3
  qemu-system-x86-4588  [000] 1076790.000367: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4588  [000] 1076790.000367: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4586  [003] 1076790.000368: kvm_entry: vcpu 0
  qemu-system-x86-4588  [000] 1076790.000368: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4589  [001] 1076790.000369: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4588  [000] 1076790.000369: kvm_entry: vcpu 2
  qemu-system-x86-4589  [001] 1076790.000370: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4589  [001] 1076790.000370: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4589  [001] 1076790.000371: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4587  [002] 1076790.000371: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff41
  qemu-system-x86-4589  [001] 1076790.000372: kvm_entry: vcpu 3
  qemu-system-x86-4587  [002] 1076790.000373: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076790.000374: kvm_exit: reason 
interrupt_window rip 0xfffff80001ca0002
  qemu-system-x86-4587  [002] 1076790.000375: kvm_inj_virq: irq 209
  qemu-system-x86-4587  [002] 1076790.000375: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076790.000377: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4587  [002] 1076790.000379: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4587  [002] 1076790.000379: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4587  [002] 1076790.000379: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4587  [002] 1076790.000380: kvm_entry: vcpu 1
  qemu-system-x86-4588  [000] 1076790.000447: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4588  [000] 1076790.000448: kvm_entry: vcpu 2
  qemu-system-x86-4589  [001] 1076790.000572: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4589  [001] 1076790.000572: kvm_entry: vcpu 3
  qemu-system-x86-4587  [002] 1076790.000625: kvm_exit: reason ext_irq 
rip 0xfffff80001ca1666
  qemu-system-x86-4587  [002] 1076790.000626: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076790.000697: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4587  [002] 1076790.000697: kvm_entry: vcpu 1
  qemu-system-x86-4586  [003] 1076790.000821: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e894
  qemu-system-x86-4586  [003] 1076790.000821: kvm_entry: vcpu 0
  qemu-system-x86-4580  [002] 1076790.001357: kvm_set_irq: gsi 0 level 1 
source 0
  qemu-system-x86-4580  [002] 1076790.001358: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4580  [002] 1076790.001359: kvm_apic_accept_irq: 
apicid 0 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076790.001359: kvm_apic_accept_irq: 
apicid 1 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076790.001359: kvm_apic_accept_irq: 
apicid 2 vec 209 (Fixed|edge)
  qemu-system-x86-4586  [003] 1076790.001360: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e891
  qemu-system-x86-4580  [002] 1076790.001360: kvm_apic_accept_irq: 
apicid 3 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076790.001360: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4586  [003] 1076790.001360: kvm_inj_virq: irq 209
  qemu-system-x86-4586  [003] 1076790.001361: kvm_entry: vcpu 0
  qemu-system-x86-4580  [002] 1076790.001361: kvm_set_irq: gsi 0 level 0 
source 0
  qemu-system-x86-4588  [000] 1076790.001361: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4580  [002] 1076790.001361: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4580  [002] 1076790.001361: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4589  [001] 1076790.001362: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4588  [000] 1076790.001362: kvm_entry: vcpu 2
  qemu-system-x86-4586  [003] 1076790.001362: kvm_exit: reason 
apic_access rip 0xfffff80001c9a5c4
  qemu-system-x86-4589  [001] 1076790.001362: kvm_entry: vcpu 3
  qemu-system-x86-4588  [000] 1076790.001363: kvm_exit: reason 
interrupt_window rip 0xfffff80001ca0002
  qemu-system-x86-4588  [000] 1076790.001364: kvm_inj_virq: irq 209
  qemu-system-x86-4589  [001] 1076790.001364: kvm_exit: reason 
interrupt_window rip 0xfffff80001ca0002
  qemu-system-x86-4586  [003] 1076790.001364: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4588  [000] 1076790.001364: kvm_entry: vcpu 2
  qemu-system-x86-4586  [003] 1076790.001364: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4589  [001] 1076790.001364: kvm_inj_virq: irq 209
  qemu-system-x86-4586  [003] 1076790.001364: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4589  [001] 1076790.001364: kvm_entry: vcpu 3
  qemu-system-x86-4586  [003] 1076790.001365: kvm_entry: vcpu 0
  qemu-system-x86-4588  [000] 1076790.001366: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4589  [001] 1076790.001366: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4588  [000] 1076790.001367: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4588  [000] 1076790.001367: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4589  [001] 1076790.001367: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4589  [001] 1076790.001368: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4588  [000] 1076790.001368: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4589  [001] 1076790.001368: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4587  [002] 1076790.001369: kvm_exit: reason ext_irq 
rip 0xfffff80001ca0010
  qemu-system-x86-4588  [000] 1076790.001369: kvm_entry: vcpu 2
  qemu-system-x86-4589  [001] 1076790.001369: kvm_entry: vcpu 3
  qemu-system-x86-4587  [002] 1076790.001370: kvm_inj_virq: irq 209
  qemu-system-x86-4587  [002] 1076790.001370: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076790.001372: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4587  [002] 1076790.001374: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4587  [002] 1076790.001374: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4587  [002] 1076790.001375: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4587  [002] 1076790.001376: kvm_entry: vcpu 1
  qemu-system-x86-4588  [000] 1076790.001446: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4588  [000] 1076790.001447: kvm_entry: vcpu 2
  qemu-system-x86-4587  [002] 1076790.001515: kvm_exit: reason ext_irq 
rip 0xfffff80001ca1680
  qemu-system-x86-4587  [002] 1076790.001516: kvm_entry: vcpu 1
  qemu-system-x86-4589  [001] 1076790.001571: kvm_exit: reason ext_irq 
rip 0xfffff80001d01a00
  qemu-system-x86-4589  [001] 1076790.001571: kvm_entry: vcpu 3
  qemu-system-x86-4587  [002] 1076790.001623: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4587  [002] 1076790.001625: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076790.001702: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ffb6
  qemu-system-x86-4587  [002] 1076790.001703: kvm_entry: vcpu 1
  qemu-system-x86-4586  [003] 1076790.001819: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e891
  qemu-system-x86-4586  [003] 1076790.001820: kvm_entry: vcpu 0
  qemu-system-x86-4588  [000] 1076790.001962: kvm_exit: reason ext_irq 
rip 0xfffff80001ca12e4
  qemu-system-x86-4588  [000] 1076790.001964: kvm_entry: vcpu 2
  qemu-system-x86-4580  [002] 1076790.002358: kvm_set_irq: gsi 0 level 1 
source 0
  qemu-system-x86-4580  [002] 1076790.002359: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4580  [002] 1076790.002360: kvm_apic_accept_irq: 
apicid 0 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076790.002360: kvm_apic_accept_irq: 
apicid 1 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076790.002360: kvm_apic_accept_irq: 
apicid 2 vec 209 (Fixed|edge)
  qemu-system-x86-4586  [003] 1076790.002361: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e88f
  qemu-system-x86-4580  [002] 1076790.002361: kvm_apic_accept_irq: 
apicid 3 vec 209 (Fixed|edge)
  qemu-system-x86-4586  [003] 1076790.002361: kvm_inj_virq: irq 209
  qemu-system-x86-4580  [002] 1076790.002361: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4586  [003] 1076790.002362: kvm_entry: vcpu 0
  qemu-system-x86-4588  [000] 1076790.002362: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4580  [002] 1076790.002362: kvm_set_irq: gsi 0 level 0 
source 0
  qemu-system-x86-4580  [002] 1076790.002362: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4580  [002] 1076790.002363: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4588  [000] 1076790.002363: kvm_entry: vcpu 2
  qemu-system-x86-4589  [001] 1076790.002363: kvm_exit: reason ext_irq 
rip 0xfffff80001cb7aa5
  qemu-system-x86-4586  [003] 1076790.002363: kvm_exit: reason 
apic_access rip 0xfffff80001c9a5c4
  qemu-system-x86-4589  [001] 1076790.002363: kvm_inj_virq: irq 209
  qemu-system-x86-4589  [001] 1076790.002364: kvm_entry: vcpu 3
  qemu-system-x86-4588  [000] 1076790.002364: kvm_exit: reason 
interrupt_window rip 0xfffff80001ca0002
  qemu-system-x86-4588  [000] 1076790.002365: kvm_inj_virq: irq 209
  qemu-system-x86-4586  [003] 1076790.002365: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4588  [000] 1076790.002365: kvm_entry: vcpu 2
  qemu-system-x86-4586  [003] 1076790.002365: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4589  [001] 1076790.002365: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4586  [003] 1076790.002365: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4588  [000] 1076790.002366: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4586  [003] 1076790.002366: kvm_entry: vcpu 0
  qemu-system-x86-4589  [001] 1076790.002367: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4589  [001] 1076790.002367: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4589  [001] 1076790.002368: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4589  [001] 1076790.002369: kvm_entry: vcpu 3
  qemu-system-x86-4588  [000] 1076790.002369: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4588  [000] 1076790.002369: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4588  [000] 1076790.002370: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4587  [002] 1076790.002370: kvm_exit: reason ext_irq 
rip 0xfffff80001d019f1
  qemu-system-x86-4588  [000] 1076790.002370: kvm_entry: vcpu 2
  qemu-system-x86-4587  [002] 1076790.002371: kvm_inj_virq: irq 209
  qemu-system-x86-4587  [002] 1076790.002372: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076790.002374: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4587  [002] 1076790.002376: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4587  [002] 1076790.002376: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4587  [002] 1076790.002376: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4587  [002] 1076790.002377: kvm_entry: vcpu 1
  qemu-system-x86-4588  [000] 1076790.002447: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4588  [000] 1076790.002447: kvm_entry: vcpu 2
  qemu-system-x86-4589  [001] 1076790.002569: kvm_exit: reason ext_irq 
rip 0xfffff80001cd8aec
  qemu-system-x86-4589  [001] 1076790.002570: kvm_entry: vcpu 3
  qemu-system-x86-4587  [002] 1076790.002623: kvm_exit: reason ext_irq 
rip 0xfffff80001ca1666
  qemu-system-x86-4587  [002] 1076790.002625: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076790.002694: kvm_exit: reason ext_irq 
rip 0xfffff80001c9b859
  qemu-system-x86-4587  [002] 1076790.002695: kvm_entry: vcpu 1
  qemu-system-x86-4586  [003] 1076790.002818: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e880
  qemu-system-x86-4586  [003] 1076790.002819: kvm_entry: vcpu 0
  qemu-system-x86-4580  [002] 1076790.003355: kvm_set_irq: gsi 0 level 1 
source 0
  qemu-system-x86-4580  [002] 1076790.003356: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4580  [002] 1076790.003357: kvm_apic_accept_irq: 
apicid 0 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076790.003357: kvm_apic_accept_irq: 
apicid 1 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076790.003357: kvm_apic_accept_irq: 
apicid 2 vec 209 (Fixed|edge)
  qemu-system-x86-4586  [003] 1076790.003358: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e894
  qemu-system-x86-4580  [002] 1076790.003358: kvm_apic_accept_irq: 
apicid 3 vec 209 (Fixed|edge)
  qemu-system-x86-4586  [003] 1076790.003358: kvm_inj_virq: irq 209
  qemu-system-x86-4580  [002] 1076790.003358: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4586  [003] 1076790.003359: kvm_entry: vcpu 0
  qemu-system-x86-4580  [002] 1076790.003359: kvm_set_irq: gsi 0 level 0 
source 0
  qemu-system-x86-4588  [000] 1076790.003359: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4580  [002] 1076790.003359: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4580  [002] 1076790.003359: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4589  [001] 1076790.003360: kvm_exit: reason ext_irq 
rip 0xfffff80001cd91b7
  qemu-system-x86-4588  [000] 1076790.003360: kvm_entry: vcpu 2
  qemu-system-x86-4589  [001] 1076790.003360: kvm_inj_virq: irq 209
  qemu-system-x86-4586  [003] 1076790.003360: kvm_exit: reason 
apic_access rip 0xfffff80001c9a5c4
  qemu-system-x86-4589  [001] 1076790.003361: kvm_entry: vcpu 3
  qemu-system-x86-4588  [000] 1076790.003361: kvm_exit: reason 
interrupt_window rip 0xfffff80001ca0002
  qemu-system-x86-4588  [000] 1076790.003361: kvm_inj_virq: irq 209
  qemu-system-x86-4588  [000] 1076790.003362: kvm_entry: vcpu 2
  qemu-system-x86-4586  [003] 1076790.003362: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4589  [001] 1076790.003362: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4586  [003] 1076790.003362: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4586  [003] 1076790.003363: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4588  [000] 1076790.003363: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4586  [003] 1076790.003363: kvm_entry: vcpu 0
  qemu-system-x86-4589  [001] 1076790.003364: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4589  [001] 1076790.003364: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4589  [001] 1076790.003365: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4588  [000] 1076790.003365: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4588  [000] 1076790.003365: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4588  [000] 1076790.003366: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4589  [001] 1076790.003366: kvm_entry: vcpu 3
  qemu-system-x86-4588  [000] 1076790.003366: kvm_entry: vcpu 2
  qemu-system-x86-4587  [002] 1076790.003367: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4587  [002] 1076790.003368: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076790.003369: kvm_exit: reason 
interrupt_window rip 0xfffff80001ca0002
  qemu-system-x86-4587  [002] 1076790.003370: kvm_inj_virq: irq 209
  qemu-system-x86-4587  [002] 1076790.003370: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076790.003372: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4587  [002] 1076790.003374: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4587  [002] 1076790.003374: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4587  [002] 1076790.003374: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4587  [002] 1076790.003375: kvm_entry: vcpu 1
  qemu-system-x86-4588  [000] 1076790.003451: kvm_exit: reason ext_irq 
rip 0xfffff80001cd8d59
  qemu-system-x86-4588  [000] 1076790.003453: kvm_entry: vcpu 2
  qemu-system-x86-4587  [002] 1076790.003479: kvm_exit: reason ext_irq 
rip 0xfffff80001cb76c3
  qemu-system-x86-4587  [002] 1076790.003480: kvm_entry: vcpu 1
  qemu-system-x86-4589  [001] 1076790.003568: kvm_exit: reason ext_irq 
rip 0xfffff80001d01a00
  qemu-system-x86-4589  [001] 1076790.003568: kvm_entry: vcpu 3
  qemu-system-x86-4587  [002] 1076790.003621: kvm_exit: reason ext_irq 
rip 0xfffff80001d019f1
  qemu-system-x86-4587  [002] 1076790.003623: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076790.003692: kvm_exit: reason ext_irq 
rip 0xfffff80001d01a00
  qemu-system-x86-4587  [002] 1076790.003693: kvm_entry: vcpu 1
  qemu-system-x86-4586  [003] 1076790.003817: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e880
  qemu-system-x86-4586  [003] 1076790.003817: kvm_entry: vcpu 0
  qemu-system-x86-4580  [002] 1076790.004354: kvm_set_irq: gsi 0 level 1 
source 0
  qemu-system-x86-4580  [002] 1076790.004355: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4580  [002] 1076790.004355: kvm_apic_accept_irq: 
apicid 0 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076790.004356: kvm_apic_accept_irq: 
apicid 1 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076790.004356: kvm_apic_accept_irq: 
apicid 2 vec 209 (Fixed|edge)
  qemu-system-x86-4586  [003] 1076790.004357: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e891
  qemu-system-x86-4580  [002] 1076790.004357: kvm_apic_accept_irq: 
apicid 3 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076790.004357: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4586  [003] 1076790.004357: kvm_inj_virq: irq 209
  qemu-system-x86-4586  [003] 1076790.004357: kvm_entry: vcpu 0
  qemu-system-x86-4588  [000] 1076790.004358: kvm_exit: reason ext_irq 
rip 0xfffff80001cd8fc5
  qemu-system-x86-4580  [002] 1076790.004358: kvm_set_irq: gsi 0 level 0 
source 0
  qemu-system-x86-4580  [002] 1076790.004358: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4580  [002] 1076790.004358: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4589  [001] 1076790.004358: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4588  [000] 1076790.004358: kvm_inj_virq: irq 209
  qemu-system-x86-4588  [000] 1076790.004359: kvm_entry: vcpu 2
  qemu-system-x86-4586  [003] 1076790.004359: kvm_exit: reason 
apic_access rip 0xfffff80001c9a5c4
  qemu-system-x86-4589  [001] 1076790.004359: kvm_entry: vcpu 3
  qemu-system-x86-4588  [000] 1076790.004360: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4589  [001] 1076790.004360: kvm_exit: reason 
interrupt_window rip 0xfffff80001ca0002
  qemu-system-x86-4586  [003] 1076790.004361: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4586  [003] 1076790.004361: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4589  [001] 1076790.004361: kvm_inj_virq: irq 209
  qemu-system-x86-4586  [003] 1076790.004361: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4589  [001] 1076790.004361: kvm_entry: vcpu 3
  qemu-system-x86-4586  [003] 1076790.004362: kvm_entry: vcpu 0
  qemu-system-x86-4588  [000] 1076790.004362: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4588  [000] 1076790.004362: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4589  [001] 1076790.004363: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4588  [000] 1076790.004363: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4588  [000] 1076790.004364: kvm_entry: vcpu 2
  qemu-system-x86-4589  [001] 1076790.004364: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4589  [001] 1076790.004365: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4589  [001] 1076790.004365: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4587  [002] 1076790.004365: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4589  [001] 1076790.004366: kvm_entry: vcpu 3
  qemu-system-x86-4587  [002] 1076790.004367: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076790.004368: kvm_exit: reason 
interrupt_window rip 0xfffff80001ca0002
  qemu-system-x86-4587  [002] 1076790.004369: kvm_inj_virq: irq 209
  qemu-system-x86-4587  [002] 1076790.004369: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076790.004371: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4587  [002] 1076790.004373: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4587  [002] 1076790.004373: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4587  [002] 1076790.004373: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4587  [002] 1076790.004374: kvm_entry: vcpu 1
  qemu-system-x86-4588  [000] 1076790.004442: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4588  [000] 1076790.004442: kvm_entry: vcpu 2
  qemu-system-x86-4586  [003] 1076790.004475: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e891
  qemu-system-x86-4586  [003] 1076790.004475: kvm_entry: vcpu 0
  qemu-system-x86-4589  [001] 1076790.004566: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4589  [001] 1076790.004567: kvm_entry: vcpu 3
  qemu-system-x86-4587  [002] 1076790.004619: kvm_exit: reason ext_irq 
rip 0xfffff80001c9b859
  qemu-system-x86-4587  [002] 1076790.004620: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076790.004690: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4587  [002] 1076790.004691: kvm_entry: vcpu 1
  qemu-system-x86-4586  [003] 1076790.004815: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e894
  qemu-system-x86-4586  [003] 1076790.004815: kvm_entry: vcpu 0
  qemu-system-x86-4580  [002] 1076790.005352: kvm_set_irq: gsi 0 level 1 
source 0
  qemu-system-x86-4580  [002] 1076790.005353: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4580  [002] 1076790.005353: kvm_apic_accept_irq: 
apicid 0 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076790.005354: kvm_apic_accept_irq: 
apicid 1 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076790.005354: kvm_apic_accept_irq: 
apicid 2 vec 209 (Fixed|edge)
  qemu-system-x86-4586  [003] 1076790.005354: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e891
  qemu-system-x86-4580  [002] 1076790.005355: kvm_apic_accept_irq: 
apicid 3 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076790.005355: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4586  [003] 1076790.005355: kvm_inj_virq: irq 209
  qemu-system-x86-4586  [003] 1076790.005355: kvm_entry: vcpu 0
  qemu-system-x86-4588  [000] 1076790.005355: kvm_exit: reason ext_irq 
rip 0xfffff80001cd8caa
  qemu-system-x86-4580  [002] 1076790.005356: kvm_set_irq: gsi 0 level 0 
source 0
  qemu-system-x86-4580  [002] 1076790.005356: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4580  [002] 1076790.005356: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4589  [001] 1076790.005356: kvm_exit: reason ext_irq 
rip 0xfffff80001c9b859
  qemu-system-x86-4588  [000] 1076790.005356: kvm_inj_virq: irq 209
  qemu-system-x86-4588  [000] 1076790.005357: kvm_entry: vcpu 2
  qemu-system-x86-4586  [003] 1076790.005357: kvm_exit: reason 
apic_access rip 0xfffff80001c9a5c4
  qemu-system-x86-4589  [001] 1076790.005357: kvm_inj_virq: irq 209
  qemu-system-x86-4589  [001] 1076790.005357: kvm_entry: vcpu 3
  qemu-system-x86-4588  [000] 1076790.005358: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4586  [003] 1076790.005359: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4586  [003] 1076790.005359: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4589  [001] 1076790.005359: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4586  [003] 1076790.005359: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4588  [000] 1076790.005360: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4588  [000] 1076790.005360: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4586  [003] 1076790.005360: kvm_entry: vcpu 0
  qemu-system-x86-4589  [001] 1076790.005360: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4588  [000] 1076790.005361: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4589  [001] 1076790.005361: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4589  [001] 1076790.005361: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4588  [000] 1076790.005362: kvm_entry: vcpu 2
  qemu-system-x86-4589  [001] 1076790.005362: kvm_entry: vcpu 3
  qemu-system-x86-4587  [002] 1076790.005363: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4587  [002] 1076790.005365: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076790.005366: kvm_exit: reason 
interrupt_window rip 0xfffff80001ca0002
  qemu-system-x86-4587  [002] 1076790.005367: kvm_inj_virq: irq 209
  qemu-system-x86-4587  [002] 1076790.005367: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076790.005369: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4587  [002] 1076790.005371: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4587  [002] 1076790.005371: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4587  [002] 1076790.005371: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4587  [002] 1076790.005372: kvm_entry: vcpu 1
  qemu-system-x86-4588  [000] 1076790.005439: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4588  [000] 1076790.005440: kvm_entry: vcpu 2
  qemu-system-x86-4589  [001] 1076790.005565: kvm_exit: reason ext_irq 
rip 0xfffff80001d01a00
  qemu-system-x86-4589  [001] 1076790.005565: kvm_entry: vcpu 3
  qemu-system-x86-4587  [002] 1076790.005617: kvm_exit: reason ext_irq 
rip 0xfffff80001d019e4
  qemu-system-x86-4587  [002] 1076790.005619: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076790.005689: kvm_exit: reason ext_irq 
rip 0xfffff80001d019f1
  qemu-system-x86-4587  [002] 1076790.005689: kvm_entry: vcpu 1
  qemu-system-x86-4586  [003] 1076790.005813: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e880
  qemu-system-x86-4586  [003] 1076790.005814: kvm_entry: vcpu 0
  qemu-system-x86-4580  [002] 1076790.006350: kvm_set_irq: gsi 0 level 1 
source 0
  qemu-system-x86-4580  [002] 1076790.006351: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4580  [002] 1076790.006351: kvm_apic_accept_irq: 
apicid 0 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076790.006352: kvm_apic_accept_irq: 
apicid 1 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076790.006352: kvm_apic_accept_irq: 
apicid 2 vec 209 (Fixed|edge)
  qemu-system-x86-4586  [003] 1076790.006352: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e894
  qemu-system-x86-4580  [002] 1076790.006353: kvm_apic_accept_irq: 
apicid 3 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076790.006353: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4586  [003] 1076790.006353: kvm_inj_virq: irq 209
  qemu-system-x86-4586  [003] 1076790.006353: kvm_entry: vcpu 0
  qemu-system-x86-4588  [000] 1076790.006353: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4580  [002] 1076790.006353: kvm_set_irq: gsi 0 level 0 
source 0
  qemu-system-x86-4580  [002] 1076790.006354: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4580  [002] 1076790.006354: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4588  [000] 1076790.006354: kvm_entry: vcpu 2
  qemu-system-x86-4589  [001] 1076790.006354: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4586  [003] 1076790.006355: kvm_exit: reason 
apic_access rip 0xfffff80001c9a5c4
  qemu-system-x86-4589  [001] 1076790.006355: kvm_entry: vcpu 3
  qemu-system-x86-4588  [000] 1076790.006355: kvm_exit: reason 
interrupt_window rip 0xfffff80001ca0002
  qemu-system-x86-4588  [000] 1076790.006356: kvm_inj_virq: irq 209
  qemu-system-x86-4589  [001] 1076790.006356: kvm_exit: reason 
interrupt_window rip 0xfffff80001ca0002
  qemu-system-x86-4586  [003] 1076790.006357: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4588  [000] 1076790.006357: kvm_entry: vcpu 2
  qemu-system-x86-4586  [003] 1076790.006357: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4589  [001] 1076790.006357: kvm_inj_virq: irq 209
  qemu-system-x86-4589  [001] 1076790.006357: kvm_entry: vcpu 3
  qemu-system-x86-4586  [003] 1076790.006357: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4588  [000] 1076790.006358: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4586  [003] 1076790.006358: kvm_entry: vcpu 0
  qemu-system-x86-4589  [001] 1076790.006359: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4588  [000] 1076790.006360: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4588  [000] 1076790.006360: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4589  [001] 1076790.006360: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4588  [000] 1076790.006360: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4589  [001] 1076790.006361: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4589  [001] 1076790.006361: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4587  [002] 1076790.006361: kvm_exit: reason ext_irq 
rip 0xfffff80001c9b859
  qemu-system-x86-4588  [000] 1076790.006361: kvm_entry: vcpu 2
  qemu-system-x86-4589  [001] 1076790.006362: kvm_entry: vcpu 3
  qemu-system-x86-4587  [002] 1076790.006363: kvm_inj_virq: irq 209
  qemu-system-x86-4587  [002] 1076790.006363: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076790.006365: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4587  [002] 1076790.006367: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4587  [002] 1076790.006368: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4587  [002] 1076790.006368: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4587  [002] 1076790.006369: kvm_entry: vcpu 1
  qemu-system-x86-4588  [000] 1076790.006438: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4588  [000] 1076790.006438: kvm_entry: vcpu 2
  qemu-system-x86-4589  [001] 1076790.006563: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4589  [001] 1076790.006563: kvm_entry: vcpu 3
  qemu-system-x86-4587  [002] 1076790.006615: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4587  [002] 1076790.006617: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076790.006687: kvm_exit: reason ext_irq 
rip 0xfffff80001d01a00
  qemu-system-x86-4587  [002] 1076790.006688: kvm_entry: vcpu 1
  qemu-system-x86-4586  [003] 1076790.006812: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e891
  qemu-system-x86-4586  [003] 1076790.006812: kvm_entry: vcpu 0
  qemu-system-x86-4580  [002] 1076790.007349: kvm_set_irq: gsi 0 level 1 
source 0
  qemu-system-x86-4580  [002] 1076790.007350: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4580  [002] 1076790.007350: kvm_apic_accept_irq: 
apicid 0 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076790.007351: kvm_apic_accept_irq: 
apicid 1 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076790.007351: kvm_apic_accept_irq: 
apicid 2 vec 209 (Fixed|edge)
  qemu-system-x86-4586  [003] 1076790.007351: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e891
  qemu-system-x86-4580  [002] 1076790.007352: kvm_apic_accept_irq: 
apicid 3 vec 209 (Fixed|edge)
  qemu-system-x86-4586  [003] 1076790.007352: kvm_inj_virq: irq 209
  qemu-system-x86-4580  [002] 1076790.007352: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4586  [003] 1076790.007352: kvm_entry: vcpu 0
  qemu-system-x86-4588  [000] 1076790.007352: kvm_exit: reason ext_irq 
rip 0xfffff80001cd90d2
  qemu-system-x86-4580  [002] 1076790.007353: kvm_set_irq: gsi 0 level 0 
source 0
  qemu-system-x86-4580  [002] 1076790.007353: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4580  [002] 1076790.007353: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4589  [001] 1076790.007353: kvm_exit: reason ext_irq 
rip 0xfffff80001cde8ec
  qemu-system-x86-4588  [000] 1076790.007353: kvm_inj_virq: irq 209
  qemu-system-x86-4588  [000] 1076790.007354: kvm_entry: vcpu 2
  qemu-system-x86-4586  [003] 1076790.007354: kvm_exit: reason 
apic_access rip 0xfffff80001c9a5c4
  qemu-system-x86-4589  [001] 1076790.007354: kvm_inj_virq: irq 209
  qemu-system-x86-4589  [001] 1076790.007354: kvm_entry: vcpu 3
  qemu-system-x86-4588  [000] 1076790.007355: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4586  [003] 1076790.007355: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4586  [003] 1076790.007356: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4589  [001] 1076790.007356: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4586  [003] 1076790.007356: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4588  [000] 1076790.007357: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4586  [003] 1076790.007357: kvm_entry: vcpu 0
  qemu-system-x86-4588  [000] 1076790.007357: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4588  [000] 1076790.007357: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4589  [001] 1076790.007358: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4589  [001] 1076790.007358: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4589  [001] 1076790.007358: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4588  [000] 1076790.007358: kvm_entry: vcpu 2
  qemu-system-x86-4589  [001] 1076790.007359: kvm_entry: vcpu 3
  qemu-system-x86-4588  [000] 1076790.007436: kvm_exit: reason ext_irq 
rip 0xfffff80001c9b85d
  qemu-system-x86-4588  [000] 1076790.007437: kvm_entry: vcpu 2
  qemu-system-x86-4587  [002] 1076790.007560: kvm_exit: reason ext_irq 
rip 0xfffff80001c9b859
  qemu-system-x86-4589  [001] 1076790.007561: kvm_exit: reason ext_irq 
rip 0xfffff80001cda706
  qemu-system-x86-4589  [001] 1076790.007562: kvm_entry: vcpu 3
  qemu-system-x86-4587  [002] 1076790.007562: kvm_inj_virq: irq 209
  qemu-system-x86-4587  [002] 1076790.007563: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076790.007565: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4587  [002] 1076790.007567: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4587  [002] 1076790.007567: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4587  [002] 1076790.007568: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4587  [002] 1076790.007569: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076790.007616: kvm_exit: reason ext_irq 
rip 0xfffff80001c9b859
  qemu-system-x86-4587  [002] 1076790.007618: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076790.007686: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ffb6
  qemu-system-x86-4587  [002] 1076790.007687: kvm_entry: vcpu 1
  qemu-system-x86-4586  [003] 1076790.007810: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e894
  qemu-system-x86-4586  [003] 1076790.007811: kvm_entry: vcpu 0
  qemu-system-x86-4580  [002] 1076790.008348: kvm_set_irq: gsi 0 level 1 
source 0
  qemu-system-x86-4580  [002] 1076790.008348: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4580  [002] 1076790.008349: kvm_apic_accept_irq: 
apicid 0 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076790.008349: kvm_apic_accept_irq: 
apicid 1 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076790.008350: kvm_apic_accept_irq: 
apicid 2 vec 209 (Fixed|edge)
  qemu-system-x86-4586  [003] 1076790.008350: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e891
  qemu-system-x86-4580  [002] 1076790.008350: kvm_apic_accept_irq: 
apicid 3 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076790.008351: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4586  [003] 1076790.008351: kvm_inj_virq: irq 209
  qemu-system-x86-4586  [003] 1076790.008351: kvm_entry: vcpu 0
  qemu-system-x86-4588  [000] 1076790.008351: kvm_exit: reason ext_irq 
rip 0xfffff80001cd8c33
  qemu-system-x86-4580  [002] 1076790.008351: kvm_set_irq: gsi 0 level 0 
source 0
  qemu-system-x86-4580  [002] 1076790.008351: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4580  [002] 1076790.008352: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4589  [001] 1076790.008352: kvm_exit: reason ext_irq 
rip 0xfffff80001c9b859
  qemu-system-x86-4588  [000] 1076790.008352: kvm_inj_virq: irq 209
  qemu-system-x86-4588  [000] 1076790.008353: kvm_entry: vcpu 2
  qemu-system-x86-4589  [001] 1076790.008353: kvm_inj_virq: irq 209
  qemu-system-x86-4586  [003] 1076790.008353: kvm_exit: reason 
apic_access rip 0xfffff80001c9a5c4
  qemu-system-x86-4589  [001] 1076790.008353: kvm_entry: vcpu 3
  qemu-system-x86-4586  [003] 1076790.008354: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4588  [000] 1076790.008354: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4586  [003] 1076790.008354: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4589  [001] 1076790.008354: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4586  [003] 1076790.008355: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4586  [003] 1076790.008356: kvm_entry: vcpu 0
  qemu-system-x86-4588  [000] 1076790.008356: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4589  [001] 1076790.008356: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4589  [001] 1076790.008356: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4588  [000] 1076790.008356: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4588  [000] 1076790.008357: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4589  [001] 1076790.008357: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4588  [000] 1076790.008358: kvm_entry: vcpu 2
  qemu-system-x86-4589  [001] 1076790.008358: kvm_entry: vcpu 3
  qemu-system-x86-4587  [002] 1076790.008359: kvm_exit: reason ext_irq 
rip 0xfffff80001cd8dc3
  qemu-system-x86-4587  [002] 1076790.008360: kvm_inj_virq: irq 209
  qemu-system-x86-4587  [002] 1076790.008361: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076790.008362: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4587  [002] 1076790.008364: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4587  [002] 1076790.008365: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4587  [002] 1076790.008365: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4587  [002] 1076790.008366: kvm_entry: vcpu 1
  qemu-system-x86-4588  [000] 1076790.008435: kvm_exit: reason ext_irq 
rip 0xfffff80001c9b859
  qemu-system-x86-4588  [000] 1076790.008436: kvm_entry: vcpu 2
  qemu-system-x86-4589  [001] 1076790.008560: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4589  [001] 1076790.008560: kvm_entry: vcpu 3
  qemu-system-x86-4587  [002] 1076790.008612: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4587  [002] 1076790.008614: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076790.008684: kvm_exit: reason ext_irq 
rip 0xfffff80001d019e4
  qemu-system-x86-4587  [002] 1076790.008685: kvm_entry: vcpu 1
  qemu-system-x86-4586  [003] 1076790.008809: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e891
  qemu-system-x86-4586  [003] 1076790.008809: kvm_entry: vcpu 0
  qemu-system-x86-4580  [002] 1076790.009345: kvm_set_irq: gsi 0 level 1 
source 0
  qemu-system-x86-4580  [002] 1076790.009346: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4580  [002] 1076790.009346: kvm_apic_accept_irq: 
apicid 0 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076790.009347: kvm_apic_accept_irq: 
apicid 1 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076790.009347: kvm_apic_accept_irq: 
apicid 2 vec 209 (Fixed|edge)
  qemu-system-x86-4586  [003] 1076790.009348: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e894
  qemu-system-x86-4580  [002] 1076790.009348: kvm_apic_accept_irq: 
apicid 3 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076790.009348: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4586  [003] 1076790.009348: kvm_inj_virq: irq 209
  qemu-system-x86-4586  [003] 1076790.009348: kvm_entry: vcpu 0
  qemu-system-x86-4588  [000] 1076790.009349: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4580  [002] 1076790.009349: kvm_set_irq: gsi 0 level 0 
source 0
  qemu-system-x86-4580  [002] 1076790.009349: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4580  [002] 1076790.009349: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4588  [000] 1076790.009349: kvm_entry: vcpu 2
  qemu-system-x86-4589  [001] 1076790.009349: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4589  [001] 1076790.009350: kvm_entry: vcpu 3
  qemu-system-x86-4586  [003] 1076790.009350: kvm_exit: reason 
apic_access rip 0xfffff80001c9a5c4
  qemu-system-x86-4588  [000] 1076790.009350: kvm_exit: reason 
interrupt_window rip 0xfffff80001ca0002
  qemu-system-x86-4588  [000] 1076790.009351: kvm_inj_virq: irq 209
  qemu-system-x86-4589  [001] 1076790.009351: kvm_exit: reason 
interrupt_window rip 0xfffff80001ca0002
  qemu-system-x86-4588  [000] 1076790.009351: kvm_entry: vcpu 2
  qemu-system-x86-4586  [003] 1076790.009352: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4586  [003] 1076790.009352: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4589  [001] 1076790.009352: kvm_inj_virq: irq 209
  qemu-system-x86-4586  [003] 1076790.009352: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4589  [001] 1076790.009352: kvm_entry: vcpu 3
  qemu-system-x86-4588  [000] 1076790.009353: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4586  [003] 1076790.009353: kvm_entry: vcpu 0
  qemu-system-x86-4589  [001] 1076790.009354: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4588  [000] 1076790.009354: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4588  [000] 1076790.009355: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4588  [000] 1076790.009355: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4589  [001] 1076790.009355: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4589  [001] 1076790.009356: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4589  [001] 1076790.009356: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4588  [000] 1076790.009356: kvm_entry: vcpu 2
  qemu-system-x86-4587  [002] 1076790.009356: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4589  [001] 1076790.009357: kvm_entry: vcpu 3
  qemu-system-x86-4587  [002] 1076790.009358: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076790.009359: kvm_exit: reason 
interrupt_window rip 0xfffff80001ca0002
  qemu-system-x86-4587  [002] 1076790.009360: kvm_inj_virq: irq 209
  qemu-system-x86-4587  [002] 1076790.009360: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076790.009362: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4587  [002] 1076790.009364: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4587  [002] 1076790.009364: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4587  [002] 1076790.009364: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4587  [002] 1076790.009365: kvm_entry: vcpu 1
  qemu-system-x86-4588  [000] 1076790.009433: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4588  [000] 1076790.009434: kvm_entry: vcpu 2
  qemu-system-x86-4589  [001] 1076790.009559: kvm_exit: reason ext_irq 
rip 0xfffff80001c9b85d
  qemu-system-x86-4586  [003] 1076790.009559: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e891
  qemu-system-x86-4589  [001] 1076790.009560: kvm_entry: vcpu 3
  qemu-system-x86-4586  [003] 1076790.009560: kvm_entry: vcpu 0
  qemu-system-x86-4587  [002] 1076790.009610: kvm_exit: reason ext_irq 
rip 0xfffff80001cd9112
  qemu-system-x86-4587  [002] 1076790.009611: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076790.009683: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4587  [002] 1076790.009683: kvm_entry: vcpu 1
  qemu-system-x86-4586  [003] 1076790.009807: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e891
  qemu-system-x86-4586  [003] 1076790.009808: kvm_entry: vcpu 0
  qemu-system-x86-4580  [002] 1076790.010344: kvm_set_irq: gsi 0 level 1 
source 0
  qemu-system-x86-4580  [002] 1076790.010344: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4580  [002] 1076790.010345: kvm_apic_accept_irq: 
apicid 0 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076790.010345: kvm_apic_accept_irq: 
apicid 1 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076790.010346: kvm_apic_accept_irq: 
apicid 2 vec 209 (Fixed|edge)
  qemu-system-x86-4586  [003] 1076790.010346: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e88f
  qemu-system-x86-4580  [002] 1076790.010346: kvm_apic_accept_irq: 
apicid 3 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076790.010347: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4586  [003] 1076790.010347: kvm_inj_virq: irq 209
  qemu-system-x86-4586  [003] 1076790.010347: kvm_entry: vcpu 0
  qemu-system-x86-4588  [000] 1076790.010347: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4580  [002] 1076790.010347: kvm_set_irq: gsi 0 level 0 
source 0
  qemu-system-x86-4580  [002] 1076790.010348: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4580  [002] 1076790.010348: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4588  [000] 1076790.010348: kvm_entry: vcpu 2
  qemu-system-x86-4589  [001] 1076790.010348: kvm_exit: reason ext_irq 
rip 0xfffff80001cd8b43
  qemu-system-x86-4586  [003] 1076790.010349: kvm_exit: reason 
apic_access rip 0xfffff80001c9a5c4
  qemu-system-x86-4589  [001] 1076790.010349: kvm_inj_virq: irq 209
  qemu-system-x86-4589  [001] 1076790.010349: kvm_entry: vcpu 3
  qemu-system-x86-4588  [000] 1076790.010349: kvm_exit: reason 
interrupt_window rip 0xfffff80001ca0002
  qemu-system-x86-4588  [000] 1076790.010350: kvm_inj_virq: irq 209
  qemu-system-x86-4588  [000] 1076790.010350: kvm_entry: vcpu 2
  qemu-system-x86-4586  [003] 1076790.010350: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4586  [003] 1076790.010351: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4589  [001] 1076790.010351: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4586  [003] 1076790.010351: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4588  [000] 1076790.010352: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4586  [003] 1076790.010352: kvm_entry: vcpu 0
  qemu-system-x86-4589  [001] 1076790.010352: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4589  [001] 1076790.010352: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4589  [001] 1076790.010353: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4588  [000] 1076790.010353: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4588  [000] 1076790.010353: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4588  [000] 1076790.010354: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4589  [001] 1076790.010354: kvm_entry: vcpu 3
  qemu-system-x86-4588  [000] 1076790.010355: kvm_entry: vcpu 2
  qemu-system-x86-4587  [002] 1076790.010355: kvm_exit: reason ext_irq 
rip 0xfffff80001ca142d
  qemu-system-x86-4587  [002] 1076790.010356: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076790.010358: kvm_exit: reason 
interrupt_window rip 0xfffff80001c9b859
  qemu-system-x86-4587  [002] 1076790.010358: kvm_inj_virq: irq 209
  qemu-system-x86-4587  [002] 1076790.010358: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076790.010360: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4587  [002] 1076790.010362: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4587  [002] 1076790.010362: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4587  [002] 1076790.010363: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4587  [002] 1076790.010364: kvm_entry: vcpu 1
  qemu-system-x86-4588  [000] 1076790.010432: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ffbf
  qemu-system-x86-4588  [000] 1076790.010432: kvm_entry: vcpu 2
  qemu-system-x86-4589  [001] 1076790.010557: kvm_exit: reason ext_irq 
rip 0xfffff80001cd90d2
  qemu-system-x86-4589  [001] 1076790.010557: kvm_entry: vcpu 3
  qemu-system-x86-4587  [002] 1076790.010609: kvm_exit: reason ext_irq 
rip 0xfffff80001d01a00
  qemu-system-x86-4587  [002] 1076790.010610: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076790.010681: kvm_exit: reason ext_irq 
rip 0xfffff80001d01a00
  qemu-system-x86-4587  [002] 1076790.010681: kvm_entry: vcpu 1
  qemu-system-x86-4586  [003] 1076790.010806: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e891
  qemu-system-x86-4586  [003] 1076790.010806: kvm_entry: vcpu 0
  qemu-system-x86-4580  [002] 1076790.011343: kvm_set_irq: gsi 0 level 1 
source 0
  qemu-system-x86-4580  [002] 1076790.011343: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4580  [002] 1076790.011344: kvm_apic_accept_irq: 
apicid 0 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076790.011344: kvm_apic_accept_irq: 
apicid 1 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076790.011345: kvm_apic_accept_irq: 
apicid 2 vec 209 (Fixed|edge)
  qemu-system-x86-4586  [003] 1076790.011345: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e891
  qemu-system-x86-4580  [002] 1076790.011345: kvm_apic_accept_irq: 
apicid 3 vec 209 (Fixed|edge)
  qemu-system-x86-4580  [002] 1076790.011345: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4586  [003] 1076790.011346: kvm_inj_virq: irq 209
  qemu-system-x86-4580  [002] 1076790.011346: kvm_set_irq: gsi 0 level 0 
source 0
  qemu-system-x86-4586  [003] 1076790.011346: kvm_entry: vcpu 0
  qemu-system-x86-4588  [000] 1076790.011346: kvm_exit: reason ext_irq 
rip 0xfffff80001c9b859
  qemu-system-x86-4580  [002] 1076790.011346: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-4580  [002] 1076790.011347: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-4589  [001] 1076790.011347: kvm_exit: reason ext_irq 
rip 0xfffff80001cb7d22
  qemu-system-x86-4588  [000] 1076790.011347: kvm_inj_virq: irq 209
  qemu-system-x86-4588  [000] 1076790.011347: kvm_entry: vcpu 2
  qemu-system-x86-4589  [001] 1076790.011347: kvm_inj_virq: irq 209
  qemu-system-x86-4589  [001] 1076790.011348: kvm_entry: vcpu 3
  qemu-system-x86-4586  [003] 1076790.011348: kvm_exit: reason 
apic_access rip 0xfffff80001c9a5c4
  qemu-system-x86-4588  [000] 1076790.011349: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4589  [001] 1076790.011349: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4586  [003] 1076790.011349: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4586  [003] 1076790.011350: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4586  [003] 1076790.011350: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4588  [000] 1076790.011350: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4588  [000] 1076790.011351: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4589  [001] 1076790.011351: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4586  [003] 1076790.011351: kvm_entry: vcpu 0
  qemu-system-x86-4589  [001] 1076790.011351: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4588  [000] 1076790.011351: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4589  [001] 1076790.011351: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4588  [000] 1076790.011352: kvm_entry: vcpu 2
  qemu-system-x86-4589  [001] 1076790.011352: kvm_entry: vcpu 3
  qemu-system-x86-4587  [002] 1076790.011354: kvm_exit: reason ext_irq 
rip 0xfffff80001c9b859
  qemu-system-x86-4587  [002] 1076790.011355: kvm_inj_virq: irq 209
  qemu-system-x86-4587  [002] 1076790.011355: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076790.011357: kvm_exit: reason 
apic_access rip 0xfffff80001c9af13
  qemu-system-x86-4587  [002] 1076790.011359: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-4587  [002] 1076790.011359: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-4587  [002] 1076790.011360: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-4587  [002] 1076790.011361: kvm_entry: vcpu 1
  qemu-system-x86-4589  [001] 1076790.011555: kvm_exit: reason ext_irq 
rip 0xfffff80001ca1302
  qemu-system-x86-4589  [001] 1076790.011556: kvm_entry: vcpu 3
  qemu-system-x86-4587  [002] 1076790.011608: kvm_exit: reason ext_irq 
rip 0xfffff80001d01a00
  qemu-system-x86-4587  [002] 1076790.011609: kvm_entry: vcpu 1
  qemu-system-x86-4587  [002] 1076790.011679: kvm_exit: reason ext_irq 
rip 0xfffff80001c9ff40
  qemu-system-x86-4587  [002] 1076790.011680: kvm_entry: vcpu 1
  qemu-system-x86-4586  [003] 1076790.011804: kvm_exit: reason ext_irq 
rip 0xfffff80001c7e88f
  qemu-system-x86-4586  [003] 1076790.011805: kvm_entry: vcpu 0

-- 
Harri.

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: Freezing Windows 2008 x64bit guest
  2010-07-19  7:17   ` Harri Olin
@ 2010-07-19  7:42     ` Gleb Natapov
  2010-07-21  6:25       ` Harri Olin
  0 siblings, 1 reply; 22+ messages in thread
From: Gleb Natapov @ 2010-07-19  7:42 UTC (permalink / raw)
  To: Harri Olin; +Cc: Christoph Adomeit, kvm@vger.kernel.org

On Mon, Jul 19, 2010 at 10:17:02AM +0300, Harri Olin wrote:
> Gleb Natapov kirjoitti:
> >On Thu, Jul 15, 2010 at 03:19:44PM +0200, Christoph Adomeit wrote:
> >>
> >>But one Windows 2008 64 Bit Server Standard is freezing regularly.
> >>This happens sometimes 3 times a day, sometimes it takes 2 days
> >>until freeze. The Windows Machine is a clean fresh install.
> >>
> >>By Freezing I mean that the Machine console still shows the
> >>windows logo but does not accept any keystroke and does not
> >>answer
> >>to ping or rdp or whatever. The virtual cpu-load is then shown
> >>as 100% (by proxmox).
> 
> Hi
> 
> I think I have seen same problem occur on my Windows 2008 SBS SP2
> 64bit system, but a bit less often, only like once a week.
> 
> Short description of system:
> Supermicro X7SBL-LN2 motherboard, Q8300 cpu, 8GB memory, Areca
> ARC-1210 RAID controller.
> 
> Linux kernel 2.6.33.1, qemu-kvm-0.11.1, windows 2008 quest is the
> only one at the moment.
> 
> I had to step down to 0.11.1 from 0.12.x as that was more unstable,
> windows 2008 bluescreened every second day. Also 0.11.1 did the same
> when I applied -rtc-td-hack, I think. (might also be -cpu host,
> which I removed at the same time but I think the crashing started
> when I added -rt-td-hack).
> Sample bluescreen error codes from that time:
> 0x0000003b (0x00000000c0000005, 0xfffff80001cb8842, 0xfffffa600
> 0x0000003b (0x00000000c0000005, 0xfffff80001c9c842, 0xfffffa600
> 0x0000000a (0x0000000000000080, 0x000000000000000c, 0x000000000
> 0x0000003b (0x00000000c0000005, 0xfffff80001c5d842, 0xfffffa600
> Also I think I have seen qemu crashing once..
> 
> Now I haven't seen crashes but only freezes with qemu on 100% and
> virtual system unresponsive.
> 
> qemu command line:
>   /usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive
> file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive
> file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net
> nic,macaddr=C0:FF:12:FB:AA:01,model=e1000 -net tap -smp 4 -localtime
> 
> 
Try with different model then e1000 please. Default driver that comes
with Windows known to have problems.

> >When hang occurs ensure that problematic vm is the only on on the server and run
> >kvm_stat. Send output here. Also do the following:
> >
> ># mount -t debugfs debugfs /sys/kernel/debug
> ># echo kvm > /sys/kernel/debug/tracing/set_event
> ># sleep 1
> ># cat /sys/kernel/debug/tracing/trace > /tmp/trace
> >
> >Send /tmp/trace here too, but it may be huge, so send only last 1000
> >lines.
> 

--
			Gleb.

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: Freezing Windows 2008 x64bit guest
  2010-07-19  7:42     ` Gleb Natapov
@ 2010-07-21  6:25       ` Harri Olin
  2010-07-21  8:28         ` Christoph Adomeit
  2010-07-21  8:37         ` Gleb Natapov
  0 siblings, 2 replies; 22+ messages in thread
From: Harri Olin @ 2010-07-21  6:25 UTC (permalink / raw)
  To: Gleb Natapov; +Cc: Christoph Adomeit, kvm@vger.kernel.org

Gleb Natapov kirjoitti:
> On Mon, Jul 19, 2010 at 10:17:02AM +0300, Harri Olin wrote:
>> Gleb Natapov kirjoitti:
>>> On Thu, Jul 15, 2010 at 03:19:44PM +0200, Christoph Adomeit wrote:
>>>> But one Windows 2008 64 Bit Server Standard is freezing regularly.
>>>> This happens sometimes 3 times a day, sometimes it takes 2 days
>>>> until freeze. The Windows Machine is a clean fresh install.
>> I think I have seen same problem occur on my Windows 2008 SBS SP2
>> 64bit system, but a bit less often, only like once a week.
>> Now I haven't seen crashes but only freezes with qemu on 100% and
>> virtual system unresponsive.
>>
>> qemu command line:
>>   /usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive
>> file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive
>> file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net
>> nic,macaddr=C0:FF:12:FB:AA:01,model=e1000 -net tap -smp 4 -localtime
>>
>>
> Try with different model then e1000 please. Default driver that comes
> with Windows known to have problems.

Didn't help, changed to default realtek emulation, but system freezed 
again. Command line:
/usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive 
file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive 
file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net 
nic,macaddr=C0:FF:12:FB:AA:01 -net tap -smp 4 -localtime

This time the virtual system was not totally unresponsive somehow, 
system pinged just fine and I think DNS server worked too. However on 
console mouse moved but didn't react to clicking, ctrl-alt-del, etc.

> 
>>> When hang occurs ensure that problematic vm is the only on on the server and run
>>> kvm_stat. Send output here. Also do the following:
>>>
>>> # mount -t debugfs debugfs /sys/kernel/debug
>>> # echo kvm > /sys/kernel/debug/tracing/set_event
>>> # sleep 1
>>> # cat /sys/kernel/debug/tracing/trace > /tmp/trace
>>>
>>> Send /tmp/trace here too, but it may be huge, so send only last 1000
>>> lines.

Stats from this hang:
log style http://mizar.remote.agasha.com/k/kvm/kvm_stat_2_log.txt

kvm statistics

  efer_reload                  0       0
  exits               5368720296   21258
  fpu_reload          2826750721    2216
  halt_exits           695286653       0
  halt_wakeup          685187661       0
  host_state_reload   3010399640    2216
  hypercalls                   0       0
  insn_emulation      2765352032    6542
  insn_emulation_fail        477       0
  invlpg                42417110     204
  io_exits             818764026     122
  irq_exits            109659572    9870
  irq_injections       788992032    4643
  irq_window            22170277    1718
  largepages                   0       0
  mmio_exits          1490912756      69
  mmu_cache_miss         2471963       0
  mmu_flooded             905690       0
  mmu_pde_zapped         1821325       0
  mmu_pte_updated        3256583       0
  mmu_pte_write          2896199       0
  mmu_recycled            259845       0


full trace: http://mizar.remote.agasha.com/k/kvm/kvm_trace_2_full.txt

  qemu-system-x86-5155  [002] 1248198.487147: kvm_entry: vcpu 3
  qemu-system-x86-5152  [000] 1248198.487147: kvm_inj_virq: irq 209
  qemu-system-x86-5152  [000] 1248198.487148: kvm_entry: vcpu 0
  qemu-system-x86-5154  [001] 1248198.487148: kvm_exit: reason 
apic_access rip 0xfffff80001c9df13
  qemu-system-x86-5155  [002] 1248198.487148: kvm_exit: reason 
apic_access rip 0xfffff80001c9df13
  qemu-system-x86-5152  [000] 1248198.487150: kvm_exit: reason 
apic_access rip 0xfffff80001c9d5c4
  qemu-system-x86-5154  [001] 1248198.487150: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5154  [001] 1248198.487150: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5155  [002] 1248198.487150: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5155  [002] 1248198.487151: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5154  [001] 1248198.487151: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5155  [002] 1248198.487151: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5152  [000] 1248198.487151: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5152  [000] 1248198.487152: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5154  [001] 1248198.487152: kvm_entry: vcpu 2
  qemu-system-x86-5155  [002] 1248198.487152: kvm_entry: vcpu 3
  qemu-system-x86-5152  [000] 1248198.487152: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5153  [003] 1248198.487153: kvm_exit: reason ext_irq 
rip 0xfffff80001d04a00
  qemu-system-x86-5152  [000] 1248198.487153: kvm_entry: vcpu 0
  qemu-system-x86-5153  [003] 1248198.487154: kvm_inj_virq: irq 209
  qemu-system-x86-5153  [003] 1248198.487155: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.487157: kvm_exit: reason 
apic_access rip 0xfffff80001c9df13
  qemu-system-x86-5153  [003] 1248198.487159: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5153  [003] 1248198.487159: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5153  [003] 1248198.487160: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5153  [003] 1248198.487161: kvm_entry: vcpu 1
  qemu-system-x86-5155  [002] 1248198.487198: kvm_exit: reason ext_irq 
rip 0xfffff80001ca42c3
  qemu-system-x86-5155  [002] 1248198.487199: kvm_entry: vcpu 3
  qemu-system-x86-5153  [003] 1248198.487324: kvm_exit: reason ext_irq 
rip 0xfffff80001c9e859
  qemu-system-x86-5153  [003] 1248198.487324: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.487407: kvm_exit: reason ext_irq 
rip 0xfffff80001ca2f40
  qemu-system-x86-5153  [003] 1248198.487408: kvm_entry: vcpu 1
  qemu-system-x86-5152  [000] 1248198.487947: kvm_exit: reason ext_irq 
rip 0xfffff80001d04a00
  qemu-system-x86-5152  [000] 1248198.487948: kvm_entry: vcpu 0
  qemu-system-x86-5154  [001] 1248198.488072: kvm_exit: reason ext_irq 
rip 0xfffff80001c9e859
  qemu-system-x86-5154  [001] 1248198.488072: kvm_entry: vcpu 2
  qemu-system-x86-5146  [003] 1248198.488141: kvm_set_irq: gsi 0 level 1 
source 0
  qemu-system-x86-5146  [003] 1248198.488141: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-5146  [003] 1248198.488142: kvm_apic_accept_irq: 
apicid 0 vec 209 (Fixed|edge)
  qemu-system-x86-5146  [003] 1248198.488142: kvm_apic_accept_irq: 
apicid 1 vec 209 (Fixed|edge)
  qemu-system-x86-5146  [003] 1248198.488143: kvm_apic_accept_irq: 
apicid 2 vec 209 (Fixed|edge)
  qemu-system-x86-5146  [003] 1248198.488143: kvm_apic_accept_irq: 
apicid 3 vec 209 (Fixed|edge)
  qemu-system-x86-5152  [000] 1248198.488144: kvm_exit: reason ext_irq 
rip 0xfffff80001ca2f40
  qemu-system-x86-5146  [003] 1248198.488144: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-5146  [003] 1248198.488144: kvm_set_irq: gsi 0 level 0 
source 0
  qemu-system-x86-5152  [000] 1248198.488144: kvm_entry: vcpu 0
  qemu-system-x86-5154  [001] 1248198.488145: kvm_exit: reason ext_irq 
rip 0xfffff80001d049f1
  qemu-system-x86-5146  [003] 1248198.488145: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-5146  [003] 1248198.488145: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-5155  [002] 1248198.488145: kvm_exit: reason ext_irq 
rip 0xfffff80001d04a00
  qemu-system-x86-5154  [001] 1248198.488145: kvm_inj_virq: irq 209
  qemu-system-x86-5155  [002] 1248198.488145: kvm_inj_virq: irq 209
  qemu-system-x86-5154  [001] 1248198.488146: kvm_entry: vcpu 2
  qemu-system-x86-5152  [000] 1248198.488146: kvm_exit: reason 
interrupt_window rip 0xfffff80001ca3002
  qemu-system-x86-5155  [002] 1248198.488146: kvm_entry: vcpu 3
  qemu-system-x86-5152  [000] 1248198.488146: kvm_inj_virq: irq 209
  qemu-system-x86-5152  [000] 1248198.488147: kvm_entry: vcpu 0
  qemu-system-x86-5154  [001] 1248198.488147: kvm_exit: reason 
apic_access rip 0xfffff80001c9df13
  qemu-system-x86-5155  [002] 1248198.488147: kvm_exit: reason 
apic_access rip 0xfffff80001c9df13
  qemu-system-x86-5152  [000] 1248198.488148: kvm_exit: reason 
apic_access rip 0xfffff80001c9d5c4
  qemu-system-x86-5154  [001] 1248198.488149: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5154  [001] 1248198.488149: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5155  [002] 1248198.488149: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5155  [002] 1248198.488149: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5154  [001] 1248198.488149: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5155  [002] 1248198.488150: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5152  [000] 1248198.488150: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5152  [000] 1248198.488150: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5152  [000] 1248198.488151: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5154  [001] 1248198.488151: kvm_entry: vcpu 2
  qemu-system-x86-5155  [002] 1248198.488152: kvm_entry: vcpu 3
  qemu-system-x86-5152  [000] 1248198.488152: kvm_entry: vcpu 0
  qemu-system-x86-5153  [003] 1248198.488153: kvm_exit: reason ext_irq 
rip 0xfffff80001d04a00
  qemu-system-x86-5153  [003] 1248198.488154: kvm_inj_virq: irq 209
  qemu-system-x86-5153  [003] 1248198.488154: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.488156: kvm_exit: reason 
apic_access rip 0xfffff80001c9df13
  qemu-system-x86-5153  [003] 1248198.488158: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5153  [003] 1248198.488158: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5153  [003] 1248198.488159: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5153  [003] 1248198.488160: kvm_entry: vcpu 1
  qemu-system-x86-5155  [002] 1248198.488197: kvm_exit: reason ext_irq 
rip 0xfffff80001ca2f40
  qemu-system-x86-5155  [002] 1248198.488197: kvm_entry: vcpu 3
  qemu-system-x86-5153  [003] 1248198.488322: kvm_exit: reason ext_irq 
rip 0xfffff80001cba795
  qemu-system-x86-5153  [003] 1248198.488322: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.488406: kvm_exit: reason ext_irq 
rip 0xfffff80001d04a00
  qemu-system-x86-5153  [003] 1248198.488407: kvm_entry: vcpu 1
  qemu-system-x86-5152  [000] 1248198.488945: kvm_exit: reason ext_irq 
rip 0xfffff80001cdc193
  qemu-system-x86-5152  [000] 1248198.488946: kvm_entry: vcpu 0
  qemu-system-x86-5154  [001] 1248198.489070: kvm_exit: reason ext_irq 
rip 0xfffff80001ca2f40
  qemu-system-x86-5154  [001] 1248198.489071: kvm_entry: vcpu 2
  qemu-system-x86-5146  [003] 1248198.489139: kvm_set_irq: gsi 0 level 1 
source 0
  qemu-system-x86-5146  [003] 1248198.489139: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-5146  [003] 1248198.489140: kvm_apic_accept_irq: 
apicid 0 vec 209 (Fixed|edge)
  qemu-system-x86-5146  [003] 1248198.489140: kvm_apic_accept_irq: 
apicid 1 vec 209 (Fixed|edge)
  qemu-system-x86-5146  [003] 1248198.489141: kvm_apic_accept_irq: 
apicid 2 vec 209 (Fixed|edge)
  qemu-system-x86-5146  [003] 1248198.489141: kvm_apic_accept_irq: 
apicid 3 vec 209 (Fixed|edge)
  qemu-system-x86-5146  [003] 1248198.489142: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-5152  [000] 1248198.489142: kvm_exit: reason ext_irq 
rip 0xfffff80001d04a00
  qemu-system-x86-5146  [003] 1248198.489142: kvm_set_irq: gsi 0 level 0 
source 0
  qemu-system-x86-5146  [003] 1248198.489142: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-5152  [000] 1248198.489142: kvm_inj_virq: irq 209
  qemu-system-x86-5146  [003] 1248198.489143: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-5154  [001] 1248198.489143: kvm_exit: reason ext_irq 
rip 0xfffff80001d04a00
  qemu-system-x86-5155  [002] 1248198.489143: kvm_exit: reason ext_irq 
rip 0xfffff80001d049e4
  qemu-system-x86-5152  [000] 1248198.489143: kvm_entry: vcpu 0
  qemu-system-x86-5154  [001] 1248198.489143: kvm_inj_virq: irq 209
  qemu-system-x86-5155  [002] 1248198.489143: kvm_inj_virq: irq 209
  qemu-system-x86-5154  [001] 1248198.489144: kvm_entry: vcpu 2
  qemu-system-x86-5155  [002] 1248198.489144: kvm_entry: vcpu 3
  qemu-system-x86-5152  [000] 1248198.489145: kvm_exit: reason 
apic_access rip 0xfffff80001c9d5c4
  qemu-system-x86-5154  [001] 1248198.489145: kvm_exit: reason 
apic_access rip 0xfffff80001c9df13
  qemu-system-x86-5155  [002] 1248198.489145: kvm_exit: reason 
apic_access rip 0xfffff80001c9df13
  qemu-system-x86-5152  [000] 1248198.489146: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5152  [000] 1248198.489147: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5154  [001] 1248198.489147: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5154  [001] 1248198.489147: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5155  [002] 1248198.489147: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5152  [000] 1248198.489147: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5155  [002] 1248198.489147: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5154  [001] 1248198.489147: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5155  [002] 1248198.489149: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5152  [000] 1248198.489149: kvm_entry: vcpu 0
  qemu-system-x86-5154  [001] 1248198.489150: kvm_entry: vcpu 2
  qemu-system-x86-5153  [003] 1248198.489150: kvm_exit: reason ext_irq 
rip 0xfffff80001ca468f
  qemu-system-x86-5155  [002] 1248198.489151: kvm_entry: vcpu 3
  qemu-system-x86-5153  [003] 1248198.489151: kvm_inj_virq: irq 209
  qemu-system-x86-5153  [003] 1248198.489152: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.489153: kvm_exit: reason 
apic_access rip 0xfffff80001c9df13
  qemu-system-x86-5153  [003] 1248198.489156: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5153  [003] 1248198.489156: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5153  [003] 1248198.489156: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5153  [003] 1248198.489157: kvm_entry: vcpu 1
  qemu-system-x86-5155  [002] 1248198.489195: kvm_exit: reason ext_irq 
rip 0xfffff80001ca2f40
  qemu-system-x86-5155  [002] 1248198.489195: kvm_entry: vcpu 3
  qemu-system-x86-5153  [003] 1248198.489321: kvm_exit: reason ext_irq 
rip 0xfffff80001ca2f40
  qemu-system-x86-5154  [001] 1248198.489322: kvm_exit: reason ext_irq 
rip 0xfffff80001c9e859
  qemu-system-x86-5153  [003] 1248198.489322: kvm_entry: vcpu 1
  qemu-system-x86-5154  [001] 1248198.489323: kvm_entry: vcpu 2
  qemu-system-x86-5153  [003] 1248198.489404: kvm_exit: reason ext_irq 
rip 0xfffff80001d04a00
  qemu-system-x86-5153  [003] 1248198.489406: kvm_entry: vcpu 1
  qemu-system-x86-5152  [000] 1248198.489944: kvm_exit: reason ext_irq 
rip 0xfffff80001d04a00
  qemu-system-x86-5152  [000] 1248198.489945: kvm_entry: vcpu 0
  qemu-system-x86-5154  [001] 1248198.490069: kvm_exit: reason ext_irq 
rip 0xfffff80001cdd795
  qemu-system-x86-5154  [001] 1248198.490069: kvm_entry: vcpu 2
  qemu-system-x86-5146  [003] 1248198.490137: kvm_set_irq: gsi 0 level 1 
source 0
  qemu-system-x86-5146  [003] 1248198.490137: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-5146  [003] 1248198.490138: kvm_apic_accept_irq: 
apicid 0 vec 209 (Fixed|edge)
  qemu-system-x86-5146  [003] 1248198.490139: kvm_apic_accept_irq: 
apicid 1 vec 209 (Fixed|edge)
  qemu-system-x86-5146  [003] 1248198.490139: kvm_apic_accept_irq: 
apicid 2 vec 209 (Fixed|edge)
  qemu-system-x86-5146  [003] 1248198.490140: kvm_apic_accept_irq: 
apicid 3 vec 209 (Fixed|edge)
  qemu-system-x86-5146  [003] 1248198.490140: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-5152  [000] 1248198.490140: kvm_exit: reason ext_irq 
rip 0xfffff80001d04a00
  qemu-system-x86-5146  [003] 1248198.490140: kvm_set_irq: gsi 0 level 0 
source 0
  qemu-system-x86-5152  [000] 1248198.490141: kvm_inj_virq: irq 209
  qemu-system-x86-5146  [003] 1248198.490141: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-5146  [003] 1248198.490141: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-5154  [001] 1248198.490141: kvm_exit: reason ext_irq 
rip 0xfffff80001d04a00
  qemu-system-x86-5155  [002] 1248198.490141: kvm_exit: reason ext_irq 
rip 0xfffff80001ca2fbf
  qemu-system-x86-5152  [000] 1248198.490141: kvm_entry: vcpu 0
  qemu-system-x86-5154  [001] 1248198.490142: kvm_inj_virq: irq 209
  qemu-system-x86-5155  [002] 1248198.490142: kvm_entry: vcpu 3
  qemu-system-x86-5154  [001] 1248198.490142: kvm_entry: vcpu 2
  qemu-system-x86-5152  [000] 1248198.490143: kvm_exit: reason 
apic_access rip 0xfffff80001c9d5c4
  qemu-system-x86-5155  [002] 1248198.490143: kvm_exit: reason 
interrupt_window rip 0xfffff80001ca3002
  qemu-system-x86-5154  [001] 1248198.490143: kvm_exit: reason 
apic_access rip 0xfffff80001c9df13
  qemu-system-x86-5155  [002] 1248198.490144: kvm_inj_virq: irq 209
  qemu-system-x86-5155  [002] 1248198.490144: kvm_entry: vcpu 3
  qemu-system-x86-5152  [000] 1248198.490144: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5152  [000] 1248198.490145: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5152  [000] 1248198.490145: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5154  [001] 1248198.490145: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5154  [001] 1248198.490146: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5155  [002] 1248198.490146: kvm_exit: reason 
apic_access rip 0xfffff80001c9df13
  qemu-system-x86-5154  [001] 1248198.490146: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5152  [000] 1248198.490146: kvm_entry: vcpu 0
  qemu-system-x86-5154  [001] 1248198.490147: kvm_entry: vcpu 2
  qemu-system-x86-5155  [002] 1248198.490147: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5155  [002] 1248198.490148: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5155  [002] 1248198.490148: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5153  [003] 1248198.490149: kvm_exit: reason ext_irq 
rip 0xfffff80001d04a00
  qemu-system-x86-5155  [002] 1248198.490149: kvm_entry: vcpu 3
  qemu-system-x86-5153  [003] 1248198.490150: kvm_inj_virq: irq 209
  qemu-system-x86-5153  [003] 1248198.490150: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.490152: kvm_exit: reason 
apic_access rip 0xfffff80001c9df13
  qemu-system-x86-5153  [003] 1248198.490154: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5153  [003] 1248198.490154: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5153  [003] 1248198.490154: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5153  [003] 1248198.490155: kvm_entry: vcpu 1
  qemu-system-x86-5155  [002] 1248198.490193: kvm_exit: reason ext_irq 
rip 0xfffff80001ca2f40
  qemu-system-x86-5155  [002] 1248198.490194: kvm_entry: vcpu 3
  qemu-system-x86-5153  [003] 1248198.490319: kvm_exit: reason ext_irq 
rip 0xfffff80001d04a00
  qemu-system-x86-5153  [003] 1248198.490319: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.490402: kvm_exit: reason ext_irq 
rip 0xfffff80001c9e859
  qemu-system-x86-5153  [003] 1248198.490404: kvm_entry: vcpu 1
  qemu-system-x86-5154  [001] 1248198.491067: kvm_exit: reason ext_irq 
rip 0xfffff80001d04a7c
  qemu-system-x86-5154  [001] 1248198.491068: kvm_entry: vcpu 2
  qemu-system-x86-5146  [003] 1248198.491136: kvm_set_irq: gsi 0 level 1 
source 0
  qemu-system-x86-5146  [003] 1248198.491137: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-5146  [003] 1248198.491138: kvm_apic_accept_irq: 
apicid 0 vec 209 (Fixed|edge)
  qemu-system-x86-5146  [003] 1248198.491138: kvm_apic_accept_irq: 
apicid 1 vec 209 (Fixed|edge)
  qemu-system-x86-5146  [003] 1248198.491139: kvm_apic_accept_irq: 
apicid 2 vec 209 (Fixed|edge)
  qemu-system-x86-5146  [003] 1248198.491139: kvm_apic_accept_irq: 
apicid 3 vec 209 (Fixed|edge)
  qemu-system-x86-5146  [003] 1248198.491140: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-5146  [003] 1248198.491140: kvm_set_irq: gsi 0 level 0 
source 0
  qemu-system-x86-5146  [003] 1248198.491140: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-5154  [001] 1248198.491141: kvm_exit: reason ext_irq 
rip 0xfffff80001d049f1
  qemu-system-x86-5146  [003] 1248198.491141: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-5155  [002] 1248198.491141: kvm_exit: reason ext_irq 
rip 0xfffff80001d04a00
  qemu-system-x86-5154  [001] 1248198.491141: kvm_inj_virq: irq 209
  qemu-system-x86-5155  [002] 1248198.491141: kvm_inj_virq: irq 209
  qemu-system-x86-5154  [001] 1248198.491142: kvm_entry: vcpu 2
  qemu-system-x86-5155  [002] 1248198.491142: kvm_entry: vcpu 3
  qemu-system-x86-5154  [001] 1248198.491143: kvm_exit: reason 
apic_access rip 0xfffff80001c9df13
  qemu-system-x86-5155  [002] 1248198.491143: kvm_exit: reason 
apic_access rip 0xfffff80001c9df13
  qemu-system-x86-5154  [001] 1248198.491145: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5155  [002] 1248198.491145: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5154  [001] 1248198.491145: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5155  [002] 1248198.491145: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5154  [001] 1248198.491146: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5155  [002] 1248198.491146: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5154  [001] 1248198.491147: kvm_entry: vcpu 2
  qemu-system-x86-5155  [002] 1248198.491147: kvm_entry: vcpu 3
  qemu-system-x86-5153  [003] 1248198.491148: kvm_exit: reason ext_irq 
rip 0xfffff80001ca2fbf
  qemu-system-x86-5153  [003] 1248198.491149: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.491151: kvm_exit: reason 
interrupt_window rip 0xfffff80001ca3002
  qemu-system-x86-5153  [003] 1248198.491151: kvm_inj_virq: irq 209
  qemu-system-x86-5153  [003] 1248198.491152: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.491153: kvm_exit: reason 
apic_access rip 0xfffff80001c9df13
  qemu-system-x86-5153  [003] 1248198.491155: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5153  [003] 1248198.491155: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5153  [003] 1248198.491156: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5153  [003] 1248198.491157: kvm_entry: vcpu 1
  qemu-system-x86-5152  [000] 1248198.491187: kvm_exit: reason ext_irq 
rip 0xfffff80001ca2f40
  qemu-system-x86-5152  [000] 1248198.491189: kvm_entry: vcpu 0
  qemu-system-x86-5152  [000] 1248198.491191: kvm_exit: reason 
interrupt_window rip 0xfffff80001ca3002
  qemu-system-x86-5152  [000] 1248198.491192: kvm_inj_virq: irq 209
  qemu-system-x86-5152  [000] 1248198.491192: kvm_entry: vcpu 0
  qemu-system-x86-5152  [000] 1248198.491194: kvm_exit: reason 
apic_access rip 0xfffff80001c9d5c4
  qemu-system-x86-5152  [000] 1248198.491196: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5152  [000] 1248198.491196: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5152  [000] 1248198.491197: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5152  [000] 1248198.491198: kvm_entry: vcpu 0
  qemu-system-x86-5152  [000] 1248198.491203: kvm_exit: reason 
apic_access rip 0xfffff80001c14092
  qemu-system-x86-5152  [000] 1248198.491205: kvm_apic: apic_read 
APIC_ICR = 0x82f
  qemu-system-x86-5152  [000] 1248198.491205: kvm_mmio: mmio read len 4 
gpa 0xfee00300 val 0x82f
  qemu-system-x86-5152  [000] 1248198.491206: kvm_entry: vcpu 0
  qemu-system-x86-5152  [000] 1248198.491207: kvm_exit: reason 
apic_access rip 0xfffff80001c140a9
  qemu-system-x86-5152  [000] 1248198.491208: kvm_mmio: mmio write len 4 
gpa 0xfee00310 val 0x2000000
  qemu-system-x86-5152  [000] 1248198.491209: kvm_apic: apic_write 
APIC_ICR2 = 0x2000000
  qemu-system-x86-5152  [000] 1248198.491209: kvm_entry: vcpu 0
  qemu-system-x86-5152  [000] 1248198.491210: kvm_exit: reason 
apic_access rip 0xfffff80001c140b0
  qemu-system-x86-5152  [000] 1248198.491212: kvm_mmio: mmio write len 4 
gpa 0xfee00300 val 0x82f
  qemu-system-x86-5152  [000] 1248198.491212: kvm_apic: apic_write 
APIC_ICR = 0x82f
  qemu-system-x86-5152  [000] 1248198.491212: kvm_apic_ipi: dst 2 vec 47 
(Fixed|logical|de-assert|edge|dst)
  qemu-system-x86-5152  [000] 1248198.491213: kvm_apic_accept_irq: 
apicid 1 vec 47 (Fixed|edge)
  qemu-system-x86-5152  [000] 1248198.491214: kvm_entry: vcpu 0
  qemu-system-x86-5153  [003] 1248198.491215: kvm_exit: reason ext_irq 
rip 0xfffff80001ca2f40
  qemu-system-x86-5153  [003] 1248198.491215: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.491216: kvm_exit: reason 
interrupt_window rip 0xfffff80001ca3002
  qemu-system-x86-5153  [003] 1248198.491217: kvm_inj_virq: irq 47
  qemu-system-x86-5153  [003] 1248198.491218: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.491219: kvm_exit: reason 
apic_access rip 0xfffff80001cedc67
  qemu-system-x86-5153  [003] 1248198.491220: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5153  [003] 1248198.491220: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5153  [003] 1248198.491222: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.491224: kvm_exit: reason cr_access 
rip 0xfffff80001ca6ff6
  qemu-system-x86-5153  [003] 1248198.491224: kvm_cr: cr_write 3 = 
0x15f46d000
  qemu-system-x86-5153  [003] 1248198.491228: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.491229: kvm_exit: reason exception 
rip 0xfffff80001ca702c
  qemu-system-x86-5153  [003] 1248198.491230: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.491238: kvm_exit: reason exception 
rip 0xfffff80001c0f11e
  qemu-system-x86-5153  [003] 1248198.491239: kvm_page_fault: address 
ffffffffffd070f0 error_code 9
  qemu-system-x86-5153  [003] 1248198.491243: kvm_mmio: mmio 
unsatisfied-read len 4 gpa 0xfed000f0 val 0x0
  qemu-system-x86-5153  [003] 1248198.491252: kvm_mmio: mmio read len 4 
gpa 0xfed000f0 val 0x993c675d
  qemu-system-x86-5153  [003] 1248198.491253: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.491259: kvm_exit: reason cr_access 
rip 0xfffff80001ca6ff6
  qemu-system-x86-5153  [003] 1248198.491260: kvm_cr: cr_write 3 = 
0x164599000
  qemu-system-x86-5153  [003] 1248198.491262: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.491263: kvm_exit: reason exception 
rip 0xfffff80001ca702c
  qemu-system-x86-5153  [003] 1248198.491263: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.491318: kvm_exit: reason ext_irq 
rip 0xfffff80001cbad28
  qemu-system-x86-5153  [003] 1248198.491319: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.491404: kvm_exit: reason ext_irq 
rip 0xfffff80001cdc130
  qemu-system-x86-5153  [003] 1248198.491405: kvm_entry: vcpu 1


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: Freezing Windows 2008 x64bit guest
  2010-07-21  6:25       ` Harri Olin
@ 2010-07-21  8:28         ` Christoph Adomeit
  2010-07-21  8:37         ` Gleb Natapov
  1 sibling, 0 replies; 22+ messages in thread
From: Christoph Adomeit @ 2010-07-21  8:28 UTC (permalink / raw)
  To: Harri Olin; +Cc: Gleb Natapov, kvm@vger.kernel.org

We also just had another freeze, although we meanwhile use the
realtek driver instead of e1000.

Our Windows Freeze behaves a little different, vm is totally
unresponsive: no ping, no mouse pointer, nothing.

I have just enabled nmi debugging and i hope the next crash
and the corresponding dump might help us.


On Wed, Jul 21, 2010 at 09:25:31AM +0300, Harri Olin wrote:
> Gleb Natapov kirjoitti:
>> On Mon, Jul 19, 2010 at 10:17:02AM +0300, Harri Olin wrote:
>>> Gleb Natapov kirjoitti:
>>>> On Thu, Jul 15, 2010 at 03:19:44PM +0200, Christoph Adomeit wrote:
>>>>> But one Windows 2008 64 Bit Server Standard is freezing regularly.
>>>>> This happens sometimes 3 times a day, sometimes it takes 2 days
>>>>> until freeze. The Windows Machine is a clean fresh install.
>>> I think I have seen same problem occur on my Windows 2008 SBS SP2
>>> 64bit system, but a bit less often, only like once a week.
>>> Now I haven't seen crashes but only freezes with qemu on 100% and
>>> virtual system unresponsive.
>>>
>>> qemu command line:
>>>   /usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive
>>> file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive
>>> file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net
>>> nic,macaddr=C0:FF:12:FB:AA:01,model=e1000 -net tap -smp 4 -localtime
>>>
>>>
>> Try with different model then e1000 please. Default driver that comes
>> with Windows known to have problems.
>
> Didn't help, changed to default realtek emulation, but system freezed  
> again. Command line:
> /usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive  
> file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive  
> file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net  
> nic,macaddr=C0:FF:12:FB:AA:01 -net tap -smp 4 -localtime
>
> This time the virtual system was not totally unresponsive somehow,  
> system pinged just fine and I think DNS server worked too. However on  
> console mouse moved but didn't react to clicking, ctrl-alt-del, etc.
>
>>
>>>> When hang occurs ensure that problematic vm is the only on on the server and run
>>>> kvm_stat. Send output here. Also do the following:
>>>>
>>>> # mount -t debugfs debugfs /sys/kernel/debug
>>>> # echo kvm > /sys/kernel/debug/tracing/set_event
>>>> # sleep 1
>>>> # cat /sys/kernel/debug/tracing/trace > /tmp/trace
>>>>
>>>> Send /tmp/trace here too, but it may be huge, so send only last 1000
>>>> lines.
>
> Stats from this hang:
> log style http://mizar.remote.agasha.com/k/kvm/kvm_stat_2_log.txt
>
> kvm statistics
>
>  efer_reload                  0       0
>  exits               5368720296   21258
>  fpu_reload          2826750721    2216
>  halt_exits           695286653       0
>  halt_wakeup          685187661       0
>  host_state_reload   3010399640    2216
>  hypercalls                   0       0
>  insn_emulation      2765352032    6542
>  insn_emulation_fail        477       0
>  invlpg                42417110     204
>  io_exits             818764026     122
>  irq_exits            109659572    9870
>  irq_injections       788992032    4643
>  irq_window            22170277    1718
>  largepages                   0       0
>  mmio_exits          1490912756      69
>  mmu_cache_miss         2471963       0
>  mmu_flooded             905690       0
>  mmu_pde_zapped         1821325       0
>  mmu_pte_updated        3256583       0
>  mmu_pte_write          2896199       0
>  mmu_recycled            259845       0
>
>
> full trace: http://mizar.remote.agasha.com/k/kvm/kvm_trace_2_full.txt
>
>  qemu-system-x86-5155  [002] 1248198.487147: kvm_entry: vcpu 3
>  qemu-system-x86-5152  [000] 1248198.487147: kvm_inj_virq: irq 209
>  qemu-system-x86-5152  [000] 1248198.487148: kvm_entry: vcpu 0
>  qemu-system-x86-5154  [001] 1248198.487148: kvm_exit: reason  
> apic_access rip 0xfffff80001c9df13
>  qemu-system-x86-5155  [002] 1248198.487148: kvm_exit: reason  
> apic_access rip 0xfffff80001c9df13
>  qemu-system-x86-5152  [000] 1248198.487150: kvm_exit: reason  
> apic_access rip 0xfffff80001c9d5c4
>  qemu-system-x86-5154  [001] 1248198.487150: kvm_mmio: mmio write len 4  
> gpa 0xfee000b0 val 0x0
>  qemu-system-x86-5154  [001] 1248198.487150: kvm_apic: apic_write  
> APIC_EOI = 0x0
>  qemu-system-x86-5155  [002] 1248198.487150: kvm_mmio: mmio write len 4  
> gpa 0xfee000b0 val 0x0
>  qemu-system-x86-5155  [002] 1248198.487151: kvm_apic: apic_write  
> APIC_EOI = 0x0
>  qemu-system-x86-5154  [001] 1248198.487151: kvm_ack_irq: irqchip IOAPIC 
> pin 2
>  qemu-system-x86-5155  [002] 1248198.487151: kvm_ack_irq: irqchip IOAPIC 
> pin 2
>  qemu-system-x86-5152  [000] 1248198.487151: kvm_mmio: mmio write len 4  
> gpa 0xfee000b0 val 0x0
>  qemu-system-x86-5152  [000] 1248198.487152: kvm_apic: apic_write  
> APIC_EOI = 0x0
>  qemu-system-x86-5154  [001] 1248198.487152: kvm_entry: vcpu 2
>  qemu-system-x86-5155  [002] 1248198.487152: kvm_entry: vcpu 3
>  qemu-system-x86-5152  [000] 1248198.487152: kvm_ack_irq: irqchip IOAPIC 
> pin 2
>  qemu-system-x86-5153  [003] 1248198.487153: kvm_exit: reason ext_irq  
> rip 0xfffff80001d04a00
>  qemu-system-x86-5152  [000] 1248198.487153: kvm_entry: vcpu 0
>  qemu-system-x86-5153  [003] 1248198.487154: kvm_inj_virq: irq 209
>  qemu-system-x86-5153  [003] 1248198.487155: kvm_entry: vcpu 1
>  qemu-system-x86-5153  [003] 1248198.487157: kvm_exit: reason  
> apic_access rip 0xfffff80001c9df13
>  qemu-system-x86-5153  [003] 1248198.487159: kvm_mmio: mmio write len 4  
> gpa 0xfee000b0 val 0x0
>  qemu-system-x86-5153  [003] 1248198.487159: kvm_apic: apic_write  
> APIC_EOI = 0x0
>  qemu-system-x86-5153  [003] 1248198.487160: kvm_ack_irq: irqchip IOAPIC 
> pin 2
>  qemu-system-x86-5153  [003] 1248198.487161: kvm_entry: vcpu 1
>  qemu-system-x86-5155  [002] 1248198.487198: kvm_exit: reason ext_irq  
> rip 0xfffff80001ca42c3
>  qemu-system-x86-5155  [002] 1248198.487199: kvm_entry: vcpu 3
>  qemu-system-x86-5153  [003] 1248198.487324: kvm_exit: reason ext_irq  
> rip 0xfffff80001c9e859
>  qemu-system-x86-5153  [003] 1248198.487324: kvm_entry: vcpu 1
>  qemu-system-x86-5153  [003] 1248198.487407: kvm_exit: reason ext_irq  
> rip 0xfffff80001ca2f40
>  qemu-system-x86-5153  [003] 1248198.487408: kvm_entry: vcpu 1
>  qemu-system-x86-5152  [000] 1248198.487947: kvm_exit: reason ext_irq  
> rip 0xfffff80001d04a00
>  qemu-system-x86-5152  [000] 1248198.487948: kvm_entry: vcpu 0
>  qemu-system-x86-5154  [001] 1248198.488072: kvm_exit: reason ext_irq  
> rip 0xfffff80001c9e859
>  qemu-system-x86-5154  [001] 1248198.488072: kvm_entry: vcpu 2
>  qemu-system-x86-5146  [003] 1248198.488141: kvm_set_irq: gsi 0 level 1  
> source 0
>  qemu-system-x86-5146  [003] 1248198.488141: kvm_pic_set_irq: chip 0 pin 
> 0 (edge|masked)
>  qemu-system-x86-5146  [003] 1248198.488142: kvm_apic_accept_irq: apicid 
> 0 vec 209 (Fixed|edge)
>  qemu-system-x86-5146  [003] 1248198.488142: kvm_apic_accept_irq: apicid 
> 1 vec 209 (Fixed|edge)
>  qemu-system-x86-5146  [003] 1248198.488143: kvm_apic_accept_irq: apicid 
> 2 vec 209 (Fixed|edge)
>  qemu-system-x86-5146  [003] 1248198.488143: kvm_apic_accept_irq: apicid 
> 3 vec 209 (Fixed|edge)
>  qemu-system-x86-5152  [000] 1248198.488144: kvm_exit: reason ext_irq  
> rip 0xfffff80001ca2f40
>  qemu-system-x86-5146  [003] 1248198.488144: kvm_ioapic_set_irq: pin 2  
> dst ff vec=209 (Fixed|physical|edge)
>  qemu-system-x86-5146  [003] 1248198.488144: kvm_set_irq: gsi 0 level 0  
> source 0
>  qemu-system-x86-5152  [000] 1248198.488144: kvm_entry: vcpu 0
>  qemu-system-x86-5154  [001] 1248198.488145: kvm_exit: reason ext_irq  
> rip 0xfffff80001d049f1
>  qemu-system-x86-5146  [003] 1248198.488145: kvm_pic_set_irq: chip 0 pin 
> 0 (edge|masked)
>  qemu-system-x86-5146  [003] 1248198.488145: kvm_ioapic_set_irq: pin 2  
> dst ff vec=209 (Fixed|physical|edge)
>  qemu-system-x86-5155  [002] 1248198.488145: kvm_exit: reason ext_irq  
> rip 0xfffff80001d04a00
>  qemu-system-x86-5154  [001] 1248198.488145: kvm_inj_virq: irq 209
>  qemu-system-x86-5155  [002] 1248198.488145: kvm_inj_virq: irq 209
>  qemu-system-x86-5154  [001] 1248198.488146: kvm_entry: vcpu 2
>  qemu-system-x86-5152  [000] 1248198.488146: kvm_exit: reason  
> interrupt_window rip 0xfffff80001ca3002
>  qemu-system-x86-5155  [002] 1248198.488146: kvm_entry: vcpu 3
>  qemu-system-x86-5152  [000] 1248198.488146: kvm_inj_virq: irq 209
>  qemu-system-x86-5152  [000] 1248198.488147: kvm_entry: vcpu 0
>  qemu-system-x86-5154  [001] 1248198.488147: kvm_exit: reason  
> apic_access rip 0xfffff80001c9df13
>  qemu-system-x86-5155  [002] 1248198.488147: kvm_exit: reason  
> apic_access rip 0xfffff80001c9df13
>  qemu-system-x86-5152  [000] 1248198.488148: kvm_exit: reason  
> apic_access rip 0xfffff80001c9d5c4
>  qemu-system-x86-5154  [001] 1248198.488149: kvm_mmio: mmio write len 4  
> gpa 0xfee000b0 val 0x0
>  qemu-system-x86-5154  [001] 1248198.488149: kvm_apic: apic_write  
> APIC_EOI = 0x0
>  qemu-system-x86-5155  [002] 1248198.488149: kvm_mmio: mmio write len 4  
> gpa 0xfee000b0 val 0x0
>  qemu-system-x86-5155  [002] 1248198.488149: kvm_apic: apic_write  
> APIC_EOI = 0x0
>  qemu-system-x86-5154  [001] 1248198.488149: kvm_ack_irq: irqchip IOAPIC 
> pin 2
>  qemu-system-x86-5155  [002] 1248198.488150: kvm_ack_irq: irqchip IOAPIC 
> pin 2
>  qemu-system-x86-5152  [000] 1248198.488150: kvm_mmio: mmio write len 4  
> gpa 0xfee000b0 val 0x0
>  qemu-system-x86-5152  [000] 1248198.488150: kvm_apic: apic_write  
> APIC_EOI = 0x0
>  qemu-system-x86-5152  [000] 1248198.488151: kvm_ack_irq: irqchip IOAPIC 
> pin 2
>  qemu-system-x86-5154  [001] 1248198.488151: kvm_entry: vcpu 2
>  qemu-system-x86-5155  [002] 1248198.488152: kvm_entry: vcpu 3
>  qemu-system-x86-5152  [000] 1248198.488152: kvm_entry: vcpu 0
>  qemu-system-x86-5153  [003] 1248198.488153: kvm_exit: reason ext_irq  
> rip 0xfffff80001d04a00
>  qemu-system-x86-5153  [003] 1248198.488154: kvm_inj_virq: irq 209
>  qemu-system-x86-5153  [003] 1248198.488154: kvm_entry: vcpu 1
>  qemu-system-x86-5153  [003] 1248198.488156: kvm_exit: reason  
> apic_access rip 0xfffff80001c9df13
>  qemu-system-x86-5153  [003] 1248198.488158: kvm_mmio: mmio write len 4  
> gpa 0xfee000b0 val 0x0
>  qemu-system-x86-5153  [003] 1248198.488158: kvm_apic: apic_write  
> APIC_EOI = 0x0
>  qemu-system-x86-5153  [003] 1248198.488159: kvm_ack_irq: irqchip IOAPIC 
> pin 2
>  qemu-system-x86-5153  [003] 1248198.488160: kvm_entry: vcpu 1
>  qemu-system-x86-5155  [002] 1248198.488197: kvm_exit: reason ext_irq  
> rip 0xfffff80001ca2f40
>  qemu-system-x86-5155  [002] 1248198.488197: kvm_entry: vcpu 3
>  qemu-system-x86-5153  [003] 1248198.488322: kvm_exit: reason ext_irq  
> rip 0xfffff80001cba795
>  qemu-system-x86-5153  [003] 1248198.488322: kvm_entry: vcpu 1
>  qemu-system-x86-5153  [003] 1248198.488406: kvm_exit: reason ext_irq  
> rip 0xfffff80001d04a00
>  qemu-system-x86-5153  [003] 1248198.488407: kvm_entry: vcpu 1
>  qemu-system-x86-5152  [000] 1248198.488945: kvm_exit: reason ext_irq  
> rip 0xfffff80001cdc193
>  qemu-system-x86-5152  [000] 1248198.488946: kvm_entry: vcpu 0
>  qemu-system-x86-5154  [001] 1248198.489070: kvm_exit: reason ext_irq  
> rip 0xfffff80001ca2f40
>  qemu-system-x86-5154  [001] 1248198.489071: kvm_entry: vcpu 2
>  qemu-system-x86-5146  [003] 1248198.489139: kvm_set_irq: gsi 0 level 1  
> source 0
>  qemu-system-x86-5146  [003] 1248198.489139: kvm_pic_set_irq: chip 0 pin 
> 0 (edge|masked)
>  qemu-system-x86-5146  [003] 1248198.489140: kvm_apic_accept_irq: apicid 
> 0 vec 209 (Fixed|edge)
>  qemu-system-x86-5146  [003] 1248198.489140: kvm_apic_accept_irq: apicid 
> 1 vec 209 (Fixed|edge)
>  qemu-system-x86-5146  [003] 1248198.489141: kvm_apic_accept_irq: apicid 
> 2 vec 209 (Fixed|edge)
>  qemu-system-x86-5146  [003] 1248198.489141: kvm_apic_accept_irq: apicid 
> 3 vec 209 (Fixed|edge)
>  qemu-system-x86-5146  [003] 1248198.489142: kvm_ioapic_set_irq: pin 2  
> dst ff vec=209 (Fixed|physical|edge)
>  qemu-system-x86-5152  [000] 1248198.489142: kvm_exit: reason ext_irq  
> rip 0xfffff80001d04a00
>  qemu-system-x86-5146  [003] 1248198.489142: kvm_set_irq: gsi 0 level 0  
> source 0
>  qemu-system-x86-5146  [003] 1248198.489142: kvm_pic_set_irq: chip 0 pin 
> 0 (edge|masked)
>  qemu-system-x86-5152  [000] 1248198.489142: kvm_inj_virq: irq 209
>  qemu-system-x86-5146  [003] 1248198.489143: kvm_ioapic_set_irq: pin 2  
> dst ff vec=209 (Fixed|physical|edge)
>  qemu-system-x86-5154  [001] 1248198.489143: kvm_exit: reason ext_irq  
> rip 0xfffff80001d04a00
>  qemu-system-x86-5155  [002] 1248198.489143: kvm_exit: reason ext_irq  
> rip 0xfffff80001d049e4
>  qemu-system-x86-5152  [000] 1248198.489143: kvm_entry: vcpu 0
>  qemu-system-x86-5154  [001] 1248198.489143: kvm_inj_virq: irq 209
>  qemu-system-x86-5155  [002] 1248198.489143: kvm_inj_virq: irq 209
>  qemu-system-x86-5154  [001] 1248198.489144: kvm_entry: vcpu 2
>  qemu-system-x86-5155  [002] 1248198.489144: kvm_entry: vcpu 3
>  qemu-system-x86-5152  [000] 1248198.489145: kvm_exit: reason  
> apic_access rip 0xfffff80001c9d5c4
>  qemu-system-x86-5154  [001] 1248198.489145: kvm_exit: reason  
> apic_access rip 0xfffff80001c9df13
>  qemu-system-x86-5155  [002] 1248198.489145: kvm_exit: reason  
> apic_access rip 0xfffff80001c9df13
>  qemu-system-x86-5152  [000] 1248198.489146: kvm_mmio: mmio write len 4  
> gpa 0xfee000b0 val 0x0
>  qemu-system-x86-5152  [000] 1248198.489147: kvm_apic: apic_write  
> APIC_EOI = 0x0
>  qemu-system-x86-5154  [001] 1248198.489147: kvm_mmio: mmio write len 4  
> gpa 0xfee000b0 val 0x0
>  qemu-system-x86-5154  [001] 1248198.489147: kvm_apic: apic_write  
> APIC_EOI = 0x0
>  qemu-system-x86-5155  [002] 1248198.489147: kvm_mmio: mmio write len 4  
> gpa 0xfee000b0 val 0x0
>  qemu-system-x86-5152  [000] 1248198.489147: kvm_ack_irq: irqchip IOAPIC 
> pin 2
>  qemu-system-x86-5155  [002] 1248198.489147: kvm_apic: apic_write  
> APIC_EOI = 0x0
>  qemu-system-x86-5154  [001] 1248198.489147: kvm_ack_irq: irqchip IOAPIC 
> pin 2
>  qemu-system-x86-5155  [002] 1248198.489149: kvm_ack_irq: irqchip IOAPIC 
> pin 2
>  qemu-system-x86-5152  [000] 1248198.489149: kvm_entry: vcpu 0
>  qemu-system-x86-5154  [001] 1248198.489150: kvm_entry: vcpu 2
>  qemu-system-x86-5153  [003] 1248198.489150: kvm_exit: reason ext_irq  
> rip 0xfffff80001ca468f
>  qemu-system-x86-5155  [002] 1248198.489151: kvm_entry: vcpu 3
>  qemu-system-x86-5153  [003] 1248198.489151: kvm_inj_virq: irq 209
>  qemu-system-x86-5153  [003] 1248198.489152: kvm_entry: vcpu 1
>  qemu-system-x86-5153  [003] 1248198.489153: kvm_exit: reason  
> apic_access rip 0xfffff80001c9df13
>  qemu-system-x86-5153  [003] 1248198.489156: kvm_mmio: mmio write len 4  
> gpa 0xfee000b0 val 0x0
>  qemu-system-x86-5153  [003] 1248198.489156: kvm_apic: apic_write  
> APIC_EOI = 0x0
>  qemu-system-x86-5153  [003] 1248198.489156: kvm_ack_irq: irqchip IOAPIC 
> pin 2
>  qemu-system-x86-5153  [003] 1248198.489157: kvm_entry: vcpu 1
>  qemu-system-x86-5155  [002] 1248198.489195: kvm_exit: reason ext_irq  
> rip 0xfffff80001ca2f40
>  qemu-system-x86-5155  [002] 1248198.489195: kvm_entry: vcpu 3
>  qemu-system-x86-5153  [003] 1248198.489321: kvm_exit: reason ext_irq  
> rip 0xfffff80001ca2f40
>  qemu-system-x86-5154  [001] 1248198.489322: kvm_exit: reason ext_irq  
> rip 0xfffff80001c9e859
>  qemu-system-x86-5153  [003] 1248198.489322: kvm_entry: vcpu 1
>  qemu-system-x86-5154  [001] 1248198.489323: kvm_entry: vcpu 2
>  qemu-system-x86-5153  [003] 1248198.489404: kvm_exit: reason ext_irq  
> rip 0xfffff80001d04a00
>  qemu-system-x86-5153  [003] 1248198.489406: kvm_entry: vcpu 1
>  qemu-system-x86-5152  [000] 1248198.489944: kvm_exit: reason ext_irq  
> rip 0xfffff80001d04a00
>  qemu-system-x86-5152  [000] 1248198.489945: kvm_entry: vcpu 0
>  qemu-system-x86-5154  [001] 1248198.490069: kvm_exit: reason ext_irq  
> rip 0xfffff80001cdd795
>  qemu-system-x86-5154  [001] 1248198.490069: kvm_entry: vcpu 2
>  qemu-system-x86-5146  [003] 1248198.490137: kvm_set_irq: gsi 0 level 1  
> source 0
>  qemu-system-x86-5146  [003] 1248198.490137: kvm_pic_set_irq: chip 0 pin 
> 0 (edge|masked)
>  qemu-system-x86-5146  [003] 1248198.490138: kvm_apic_accept_irq: apicid 
> 0 vec 209 (Fixed|edge)
>  qemu-system-x86-5146  [003] 1248198.490139: kvm_apic_accept_irq: apicid 
> 1 vec 209 (Fixed|edge)
>  qemu-system-x86-5146  [003] 1248198.490139: kvm_apic_accept_irq: apicid 
> 2 vec 209 (Fixed|edge)
>  qemu-system-x86-5146  [003] 1248198.490140: kvm_apic_accept_irq: apicid 
> 3 vec 209 (Fixed|edge)
>  qemu-system-x86-5146  [003] 1248198.490140: kvm_ioapic_set_irq: pin 2  
> dst ff vec=209 (Fixed|physical|edge)
>  qemu-system-x86-5152  [000] 1248198.490140: kvm_exit: reason ext_irq  
> rip 0xfffff80001d04a00
>  qemu-system-x86-5146  [003] 1248198.490140: kvm_set_irq: gsi 0 level 0  
> source 0
>  qemu-system-x86-5152  [000] 1248198.490141: kvm_inj_virq: irq 209
>  qemu-system-x86-5146  [003] 1248198.490141: kvm_pic_set_irq: chip 0 pin 
> 0 (edge|masked)
>  qemu-system-x86-5146  [003] 1248198.490141: kvm_ioapic_set_irq: pin 2  
> dst ff vec=209 (Fixed|physical|edge)
>  qemu-system-x86-5154  [001] 1248198.490141: kvm_exit: reason ext_irq  
> rip 0xfffff80001d04a00
>  qemu-system-x86-5155  [002] 1248198.490141: kvm_exit: reason ext_irq  
> rip 0xfffff80001ca2fbf
>  qemu-system-x86-5152  [000] 1248198.490141: kvm_entry: vcpu 0
>  qemu-system-x86-5154  [001] 1248198.490142: kvm_inj_virq: irq 209
>  qemu-system-x86-5155  [002] 1248198.490142: kvm_entry: vcpu 3
>  qemu-system-x86-5154  [001] 1248198.490142: kvm_entry: vcpu 2
>  qemu-system-x86-5152  [000] 1248198.490143: kvm_exit: reason  
> apic_access rip 0xfffff80001c9d5c4
>  qemu-system-x86-5155  [002] 1248198.490143: kvm_exit: reason  
> interrupt_window rip 0xfffff80001ca3002
>  qemu-system-x86-5154  [001] 1248198.490143: kvm_exit: reason  
> apic_access rip 0xfffff80001c9df13
>  qemu-system-x86-5155  [002] 1248198.490144: kvm_inj_virq: irq 209
>  qemu-system-x86-5155  [002] 1248198.490144: kvm_entry: vcpu 3
>  qemu-system-x86-5152  [000] 1248198.490144: kvm_mmio: mmio write len 4  
> gpa 0xfee000b0 val 0x0
>  qemu-system-x86-5152  [000] 1248198.490145: kvm_apic: apic_write  
> APIC_EOI = 0x0
>  qemu-system-x86-5152  [000] 1248198.490145: kvm_ack_irq: irqchip IOAPIC 
> pin 2
>  qemu-system-x86-5154  [001] 1248198.490145: kvm_mmio: mmio write len 4  
> gpa 0xfee000b0 val 0x0
>  qemu-system-x86-5154  [001] 1248198.490146: kvm_apic: apic_write  
> APIC_EOI = 0x0
>  qemu-system-x86-5155  [002] 1248198.490146: kvm_exit: reason  
> apic_access rip 0xfffff80001c9df13
>  qemu-system-x86-5154  [001] 1248198.490146: kvm_ack_irq: irqchip IOAPIC 
> pin 2
>  qemu-system-x86-5152  [000] 1248198.490146: kvm_entry: vcpu 0
>  qemu-system-x86-5154  [001] 1248198.490147: kvm_entry: vcpu 2
>  qemu-system-x86-5155  [002] 1248198.490147: kvm_mmio: mmio write len 4  
> gpa 0xfee000b0 val 0x0
>  qemu-system-x86-5155  [002] 1248198.490148: kvm_apic: apic_write  
> APIC_EOI = 0x0
>  qemu-system-x86-5155  [002] 1248198.490148: kvm_ack_irq: irqchip IOAPIC 
> pin 2
>  qemu-system-x86-5153  [003] 1248198.490149: kvm_exit: reason ext_irq  
> rip 0xfffff80001d04a00
>  qemu-system-x86-5155  [002] 1248198.490149: kvm_entry: vcpu 3
>  qemu-system-x86-5153  [003] 1248198.490150: kvm_inj_virq: irq 209
>  qemu-system-x86-5153  [003] 1248198.490150: kvm_entry: vcpu 1
>  qemu-system-x86-5153  [003] 1248198.490152: kvm_exit: reason  
> apic_access rip 0xfffff80001c9df13
>  qemu-system-x86-5153  [003] 1248198.490154: kvm_mmio: mmio write len 4  
> gpa 0xfee000b0 val 0x0
>  qemu-system-x86-5153  [003] 1248198.490154: kvm_apic: apic_write  
> APIC_EOI = 0x0
>  qemu-system-x86-5153  [003] 1248198.490154: kvm_ack_irq: irqchip IOAPIC 
> pin 2
>  qemu-system-x86-5153  [003] 1248198.490155: kvm_entry: vcpu 1
>  qemu-system-x86-5155  [002] 1248198.490193: kvm_exit: reason ext_irq  
> rip 0xfffff80001ca2f40
>  qemu-system-x86-5155  [002] 1248198.490194: kvm_entry: vcpu 3
>  qemu-system-x86-5153  [003] 1248198.490319: kvm_exit: reason ext_irq  
> rip 0xfffff80001d04a00
>  qemu-system-x86-5153  [003] 1248198.490319: kvm_entry: vcpu 1
>  qemu-system-x86-5153  [003] 1248198.490402: kvm_exit: reason ext_irq  
> rip 0xfffff80001c9e859
>  qemu-system-x86-5153  [003] 1248198.490404: kvm_entry: vcpu 1
>  qemu-system-x86-5154  [001] 1248198.491067: kvm_exit: reason ext_irq  
> rip 0xfffff80001d04a7c
>  qemu-system-x86-5154  [001] 1248198.491068: kvm_entry: vcpu 2
>  qemu-system-x86-5146  [003] 1248198.491136: kvm_set_irq: gsi 0 level 1  
> source 0
>  qemu-system-x86-5146  [003] 1248198.491137: kvm_pic_set_irq: chip 0 pin 
> 0 (edge|masked)
>  qemu-system-x86-5146  [003] 1248198.491138: kvm_apic_accept_irq: apicid 
> 0 vec 209 (Fixed|edge)
>  qemu-system-x86-5146  [003] 1248198.491138: kvm_apic_accept_irq: apicid 
> 1 vec 209 (Fixed|edge)
>  qemu-system-x86-5146  [003] 1248198.491139: kvm_apic_accept_irq: apicid 
> 2 vec 209 (Fixed|edge)
>  qemu-system-x86-5146  [003] 1248198.491139: kvm_apic_accept_irq: apicid 
> 3 vec 209 (Fixed|edge)
>  qemu-system-x86-5146  [003] 1248198.491140: kvm_ioapic_set_irq: pin 2  
> dst ff vec=209 (Fixed|physical|edge)
>  qemu-system-x86-5146  [003] 1248198.491140: kvm_set_irq: gsi 0 level 0  
> source 0
>  qemu-system-x86-5146  [003] 1248198.491140: kvm_pic_set_irq: chip 0 pin 
> 0 (edge|masked)
>  qemu-system-x86-5154  [001] 1248198.491141: kvm_exit: reason ext_irq  
> rip 0xfffff80001d049f1
>  qemu-system-x86-5146  [003] 1248198.491141: kvm_ioapic_set_irq: pin 2  
> dst ff vec=209 (Fixed|physical|edge)
>  qemu-system-x86-5155  [002] 1248198.491141: kvm_exit: reason ext_irq  
> rip 0xfffff80001d04a00
>  qemu-system-x86-5154  [001] 1248198.491141: kvm_inj_virq: irq 209
>  qemu-system-x86-5155  [002] 1248198.491141: kvm_inj_virq: irq 209
>  qemu-system-x86-5154  [001] 1248198.491142: kvm_entry: vcpu 2
>  qemu-system-x86-5155  [002] 1248198.491142: kvm_entry: vcpu 3
>  qemu-system-x86-5154  [001] 1248198.491143: kvm_exit: reason  
> apic_access rip 0xfffff80001c9df13
>  qemu-system-x86-5155  [002] 1248198.491143: kvm_exit: reason  
> apic_access rip 0xfffff80001c9df13
>  qemu-system-x86-5154  [001] 1248198.491145: kvm_mmio: mmio write len 4  
> gpa 0xfee000b0 val 0x0
>  qemu-system-x86-5155  [002] 1248198.491145: kvm_mmio: mmio write len 4  
> gpa 0xfee000b0 val 0x0
>  qemu-system-x86-5154  [001] 1248198.491145: kvm_apic: apic_write  
> APIC_EOI = 0x0
>  qemu-system-x86-5155  [002] 1248198.491145: kvm_apic: apic_write  
> APIC_EOI = 0x0
>  qemu-system-x86-5154  [001] 1248198.491146: kvm_ack_irq: irqchip IOAPIC 
> pin 2
>  qemu-system-x86-5155  [002] 1248198.491146: kvm_ack_irq: irqchip IOAPIC 
> pin 2
>  qemu-system-x86-5154  [001] 1248198.491147: kvm_entry: vcpu 2
>  qemu-system-x86-5155  [002] 1248198.491147: kvm_entry: vcpu 3
>  qemu-system-x86-5153  [003] 1248198.491148: kvm_exit: reason ext_irq  
> rip 0xfffff80001ca2fbf
>  qemu-system-x86-5153  [003] 1248198.491149: kvm_entry: vcpu 1
>  qemu-system-x86-5153  [003] 1248198.491151: kvm_exit: reason  
> interrupt_window rip 0xfffff80001ca3002
>  qemu-system-x86-5153  [003] 1248198.491151: kvm_inj_virq: irq 209
>  qemu-system-x86-5153  [003] 1248198.491152: kvm_entry: vcpu 1
>  qemu-system-x86-5153  [003] 1248198.491153: kvm_exit: reason  
> apic_access rip 0xfffff80001c9df13
>  qemu-system-x86-5153  [003] 1248198.491155: kvm_mmio: mmio write len 4  
> gpa 0xfee000b0 val 0x0
>  qemu-system-x86-5153  [003] 1248198.491155: kvm_apic: apic_write  
> APIC_EOI = 0x0
>  qemu-system-x86-5153  [003] 1248198.491156: kvm_ack_irq: irqchip IOAPIC 
> pin 2
>  qemu-system-x86-5153  [003] 1248198.491157: kvm_entry: vcpu 1
>  qemu-system-x86-5152  [000] 1248198.491187: kvm_exit: reason ext_irq  
> rip 0xfffff80001ca2f40
>  qemu-system-x86-5152  [000] 1248198.491189: kvm_entry: vcpu 0
>  qemu-system-x86-5152  [000] 1248198.491191: kvm_exit: reason  
> interrupt_window rip 0xfffff80001ca3002
>  qemu-system-x86-5152  [000] 1248198.491192: kvm_inj_virq: irq 209
>  qemu-system-x86-5152  [000] 1248198.491192: kvm_entry: vcpu 0
>  qemu-system-x86-5152  [000] 1248198.491194: kvm_exit: reason  
> apic_access rip 0xfffff80001c9d5c4
>  qemu-system-x86-5152  [000] 1248198.491196: kvm_mmio: mmio write len 4  
> gpa 0xfee000b0 val 0x0
>  qemu-system-x86-5152  [000] 1248198.491196: kvm_apic: apic_write  
> APIC_EOI = 0x0
>  qemu-system-x86-5152  [000] 1248198.491197: kvm_ack_irq: irqchip IOAPIC 
> pin 2
>  qemu-system-x86-5152  [000] 1248198.491198: kvm_entry: vcpu 0
>  qemu-system-x86-5152  [000] 1248198.491203: kvm_exit: reason  
> apic_access rip 0xfffff80001c14092
>  qemu-system-x86-5152  [000] 1248198.491205: kvm_apic: apic_read  
> APIC_ICR = 0x82f
>  qemu-system-x86-5152  [000] 1248198.491205: kvm_mmio: mmio read len 4  
> gpa 0xfee00300 val 0x82f
>  qemu-system-x86-5152  [000] 1248198.491206: kvm_entry: vcpu 0
>  qemu-system-x86-5152  [000] 1248198.491207: kvm_exit: reason  
> apic_access rip 0xfffff80001c140a9
>  qemu-system-x86-5152  [000] 1248198.491208: kvm_mmio: mmio write len 4  
> gpa 0xfee00310 val 0x2000000
>  qemu-system-x86-5152  [000] 1248198.491209: kvm_apic: apic_write  
> APIC_ICR2 = 0x2000000
>  qemu-system-x86-5152  [000] 1248198.491209: kvm_entry: vcpu 0
>  qemu-system-x86-5152  [000] 1248198.491210: kvm_exit: reason  
> apic_access rip 0xfffff80001c140b0
>  qemu-system-x86-5152  [000] 1248198.491212: kvm_mmio: mmio write len 4  
> gpa 0xfee00300 val 0x82f
>  qemu-system-x86-5152  [000] 1248198.491212: kvm_apic: apic_write  
> APIC_ICR = 0x82f
>  qemu-system-x86-5152  [000] 1248198.491212: kvm_apic_ipi: dst 2 vec 47  
> (Fixed|logical|de-assert|edge|dst)
>  qemu-system-x86-5152  [000] 1248198.491213: kvm_apic_accept_irq: apicid 
> 1 vec 47 (Fixed|edge)
>  qemu-system-x86-5152  [000] 1248198.491214: kvm_entry: vcpu 0
>  qemu-system-x86-5153  [003] 1248198.491215: kvm_exit: reason ext_irq  
> rip 0xfffff80001ca2f40
>  qemu-system-x86-5153  [003] 1248198.491215: kvm_entry: vcpu 1
>  qemu-system-x86-5153  [003] 1248198.491216: kvm_exit: reason  
> interrupt_window rip 0xfffff80001ca3002
>  qemu-system-x86-5153  [003] 1248198.491217: kvm_inj_virq: irq 47
>  qemu-system-x86-5153  [003] 1248198.491218: kvm_entry: vcpu 1
>  qemu-system-x86-5153  [003] 1248198.491219: kvm_exit: reason  
> apic_access rip 0xfffff80001cedc67
>  qemu-system-x86-5153  [003] 1248198.491220: kvm_mmio: mmio write len 4  
> gpa 0xfee000b0 val 0x0
>  qemu-system-x86-5153  [003] 1248198.491220: kvm_apic: apic_write  
> APIC_EOI = 0x0
>  qemu-system-x86-5153  [003] 1248198.491222: kvm_entry: vcpu 1
>  qemu-system-x86-5153  [003] 1248198.491224: kvm_exit: reason cr_access  
> rip 0xfffff80001ca6ff6
>  qemu-system-x86-5153  [003] 1248198.491224: kvm_cr: cr_write 3 =  
> 0x15f46d000
>  qemu-system-x86-5153  [003] 1248198.491228: kvm_entry: vcpu 1
>  qemu-system-x86-5153  [003] 1248198.491229: kvm_exit: reason exception  
> rip 0xfffff80001ca702c
>  qemu-system-x86-5153  [003] 1248198.491230: kvm_entry: vcpu 1
>  qemu-system-x86-5153  [003] 1248198.491238: kvm_exit: reason exception  
> rip 0xfffff80001c0f11e
>  qemu-system-x86-5153  [003] 1248198.491239: kvm_page_fault: address  
> ffffffffffd070f0 error_code 9
>  qemu-system-x86-5153  [003] 1248198.491243: kvm_mmio: mmio  
> unsatisfied-read len 4 gpa 0xfed000f0 val 0x0
>  qemu-system-x86-5153  [003] 1248198.491252: kvm_mmio: mmio read len 4  
> gpa 0xfed000f0 val 0x993c675d
>  qemu-system-x86-5153  [003] 1248198.491253: kvm_entry: vcpu 1
>  qemu-system-x86-5153  [003] 1248198.491259: kvm_exit: reason cr_access  
> rip 0xfffff80001ca6ff6
>  qemu-system-x86-5153  [003] 1248198.491260: kvm_cr: cr_write 3 =  
> 0x164599000
>  qemu-system-x86-5153  [003] 1248198.491262: kvm_entry: vcpu 1
>  qemu-system-x86-5153  [003] 1248198.491263: kvm_exit: reason exception  
> rip 0xfffff80001ca702c
>  qemu-system-x86-5153  [003] 1248198.491263: kvm_entry: vcpu 1
>  qemu-system-x86-5153  [003] 1248198.491318: kvm_exit: reason ext_irq  
> rip 0xfffff80001cbad28
>  qemu-system-x86-5153  [003] 1248198.491319: kvm_entry: vcpu 1
>  qemu-system-x86-5153  [003] 1248198.491404: kvm_exit: reason ext_irq  
> rip 0xfffff80001cdc130
>  qemu-system-x86-5153  [003] 1248198.491405: kvm_entry: vcpu 1
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Christoph Adomeit
GATWORKS GmbH
Reststrauch 191
41199 Moenchengladbach
Sitz: Moenchengladbach
Amtsgericht Moenchengladbach, HRB 6303
Geschaeftsfuehrer:
Christoph Adomeit, Hans Wilhelm Terstappen

Christoph.Adomeit@gatworks.de     Internetloesungen vom Feinsten
Fon. +49 2166 9149-32                      Fax. +49 2166 9149-10

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: Freezing Windows 2008 x64bit guest
  2010-07-21  6:25       ` Harri Olin
  2010-07-21  8:28         ` Christoph Adomeit
@ 2010-07-21  8:37         ` Gleb Natapov
  2010-07-21  9:22           ` Harri Olin
  2010-07-27 21:53           ` Harri Olin
  1 sibling, 2 replies; 22+ messages in thread
From: Gleb Natapov @ 2010-07-21  8:37 UTC (permalink / raw)
  To: Harri Olin; +Cc: Christoph Adomeit, kvm@vger.kernel.org

On Wed, Jul 21, 2010 at 09:25:31AM +0300, Harri Olin wrote:
> Gleb Natapov kirjoitti:
> >On Mon, Jul 19, 2010 at 10:17:02AM +0300, Harri Olin wrote:
> >>Gleb Natapov kirjoitti:
> >>>On Thu, Jul 15, 2010 at 03:19:44PM +0200, Christoph Adomeit wrote:
> >>>>But one Windows 2008 64 Bit Server Standard is freezing regularly.
> >>>>This happens sometimes 3 times a day, sometimes it takes 2 days
> >>>>until freeze. The Windows Machine is a clean fresh install.
> >>I think I have seen same problem occur on my Windows 2008 SBS SP2
> >>64bit system, but a bit less often, only like once a week.
> >>Now I haven't seen crashes but only freezes with qemu on 100% and
> >>virtual system unresponsive.
> >>
> >>qemu command line:
> >>  /usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive
> >>file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive
> >>file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net
> >>nic,macaddr=C0:FF:12:FB:AA:01,model=e1000 -net tap -smp 4 -localtime
> >>
> >>
> >Try with different model then e1000 please. Default driver that comes
> >with Windows known to have problems.
> 
> Didn't help, changed to default realtek emulation, but system
> freezed again. Command line:
> /usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive
> file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive
> file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net
> nic,macaddr=C0:FF:12:FB:AA:01 -net tap -smp 4 -localtime
> 
> This time the virtual system was not totally unresponsive somehow,
> system pinged just fine and I think DNS server worked too. However
> on console mouse moved but didn't react to clicking, ctrl-alt-del,
> etc.
> 
Does sendkey from monitor works? qemu-kvm-0.11.1 is very old and this is
not total freeze which even harder to debug. I don't see anything
extraordinary in your logs. 4643 interrupt per second for 4 cpus is
normal if windows runs multimedia or other app that need hi-res timers.
Does your host swapping? Is there any chance that you can try upstream qemu-kvm?


--
			Gleb.

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: Freezing Windows 2008 x64bit guest
  2010-07-21  8:37         ` Gleb Natapov
@ 2010-07-21  9:22           ` Harri Olin
  2010-07-21  9:48             ` Gleb Natapov
  2010-07-27 21:53           ` Harri Olin
  1 sibling, 1 reply; 22+ messages in thread
From: Harri Olin @ 2010-07-21  9:22 UTC (permalink / raw)
  To: Gleb Natapov; +Cc: Christoph Adomeit, kvm@vger.kernel.org

Gleb Natapov wrote:
> On Wed, Jul 21, 2010 at 09:25:31AM +0300, Harri Olin wrote:
>> Gleb Natapov kirjoitti:
>>> On Mon, Jul 19, 2010 at 10:17:02AM +0300, Harri Olin wrote:
>>>> Gleb Natapov kirjoitti:
>>>>> On Thu, Jul 15, 2010 at 03:19:44PM +0200, Christoph Adomeit wrote:
>>>>>> But one Windows 2008 64 Bit Server Standard is freezing regularly.
>>>>>> This happens sometimes 3 times a day, sometimes it takes 2 days
>>>>>> until freeze. The Windows Machine is a clean fresh install.
>>>> I think I have seen same problem occur on my Windows 2008 SBS SP2
>>>> 64bit system, but a bit less often, only like once a week.
>>>> Now I haven't seen crashes but only freezes with qemu on 100% and
>>>> virtual system unresponsive.
>>>>
>>>> qemu command line:
>>>>  /usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive
>>>> file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive
>>>> file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net
>>>> nic,macaddr=C0:FF:12:FB:AA:01,model=e1000 -net tap -smp 4 -localtime
>>>>
>>>>
>>> Try with different model then e1000 please. Default driver that comes
>>> with Windows known to have problems.
>> Didn't help, changed to default realtek emulation, but system
>> freezed again. Command line:
>> /usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive
>> file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive
>> file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net
>> nic,macaddr=C0:FF:12:FB:AA:01 -net tap -smp 4 -localtime
>>
>> This time the virtual system was not totally unresponsive somehow,
>> system pinged just fine and I think DNS server worked too. However
>> on console mouse moved but didn't react to clicking, ctrl-alt-del,
>> etc.
>>
> Does sendkey from monitor works? qemu-kvm-0.11.1 is very old and this is
> not total freeze which even harder to debug. I don't see anything
> extraordinary in your logs. 4643 interrupt per second for 4 cpus is
> normal if windows runs multimedia or other app that need hi-res timers.
> Does your host swapping? Is there any chance that you can try upstream qemu-kvm?
> 

I haven't tested sendkey. I originally tried 0.12.2 or so but with that 
windows crashed more often with bluescreen, I went down to 0.11.1. This 
might be related to -td-rtc-hack and/or e1000 as using td-rtc-hack on 
0.11.1 windows crashed too to bluescreen.

System does not run any multimedia or so, on top of normal windows 2008 
small business server installation there should be only nagios client 
and apache.

Only thing that catches my eye on stats frozen vs normal is irq_exits 
which jumps from 200 to 10000 when frozen and irq_window which goes 10x.

Host has swap partition enabled but hasn't swapped once yet, currently 
there's over 1GB of free.

I'll update host kernel to latest stable and qemu-kvm to upstream later 
today.

Here's stats from running system:
  efer_reload                  0       0
  exits                386474714   23272
  fpu_reload           173420913   12775
  halt_exits            51894921    4294
  halt_wakeup           51039254    4223
  host_state_reload    188645310   13420
  hypercalls                   0       0
  insn_emulation       212729430   15545
  insn_emulation_fail        324       0
  invlpg                 5177828     134
  io_exits              23132581     161
  irq_exits              7485833     225
  irq_injections        59239306    4655
  irq_window             2147438     153
  largepages                   0       0
  mmio_exits           110931423    9026
  mmu_cache_miss          468410       0
  mmu_flooded             193153       0
  mmu_pde_zapped          352178       0
  mmu_pte_updated         678023       0
  mmu_pte_write           587175       0
  mmu_recycled             34998       0
  mmu_shadow_zapped       454073       0
  mmu_unsync               19084       0
  nmi_injections               0       0
  nmi_window                   0       0
  pf_fixed              25171943     120
  pf_guest               6209034       2
  remote_tlb_flush       3044169      23
  request_irq                  0       0
  signal_exits                 0       0
  tlb_flush              8422950     149


-- 
Harri.

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: Freezing Windows 2008 x64bit guest
  2010-07-21  9:22           ` Harri Olin
@ 2010-07-21  9:48             ` Gleb Natapov
  2010-07-21 10:09               ` Harri Olin
  0 siblings, 1 reply; 22+ messages in thread
From: Gleb Natapov @ 2010-07-21  9:48 UTC (permalink / raw)
  To: Harri Olin; +Cc: Christoph Adomeit, kvm@vger.kernel.org

On Wed, Jul 21, 2010 at 12:22:45PM +0300, Harri Olin wrote:
> Gleb Natapov wrote:
> >On Wed, Jul 21, 2010 at 09:25:31AM +0300, Harri Olin wrote:
> >>Gleb Natapov kirjoitti:
> >>>On Mon, Jul 19, 2010 at 10:17:02AM +0300, Harri Olin wrote:
> >>>>Gleb Natapov kirjoitti:
> >>>>>On Thu, Jul 15, 2010 at 03:19:44PM +0200, Christoph Adomeit wrote:
> >>>>>>But one Windows 2008 64 Bit Server Standard is freezing regularly.
> >>>>>>This happens sometimes 3 times a day, sometimes it takes 2 days
> >>>>>>until freeze. The Windows Machine is a clean fresh install.
> >>>>I think I have seen same problem occur on my Windows 2008 SBS SP2
> >>>>64bit system, but a bit less often, only like once a week.
> >>>>Now I haven't seen crashes but only freezes with qemu on 100% and
> >>>>virtual system unresponsive.
> >>>>
> >>>>qemu command line:
> >>>> /usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive
> >>>>file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive
> >>>>file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net
> >>>>nic,macaddr=C0:FF:12:FB:AA:01,model=e1000 -net tap -smp 4 -localtime
> >>>>
> >>>>
> >>>Try with different model then e1000 please. Default driver that comes
> >>>with Windows known to have problems.
> >>Didn't help, changed to default realtek emulation, but system
> >>freezed again. Command line:
> >>/usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive
> >>file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive
> >>file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net
> >>nic,macaddr=C0:FF:12:FB:AA:01 -net tap -smp 4 -localtime
> >>
> >>This time the virtual system was not totally unresponsive somehow,
> >>system pinged just fine and I think DNS server worked too. However
> >>on console mouse moved but didn't react to clicking, ctrl-alt-del,
> >>etc.
> >>
> >Does sendkey from monitor works? qemu-kvm-0.11.1 is very old and this is
> >not total freeze which even harder to debug. I don't see anything
> >extraordinary in your logs. 4643 interrupt per second for 4 cpus is
> >normal if windows runs multimedia or other app that need hi-res timers.
> >Does your host swapping? Is there any chance that you can try upstream qemu-kvm?
> >
> 
> I haven't tested sendkey. I originally tried 0.12.2 or so but with
> that windows crashed more often with bluescreen, I went down to
> 0.11.1. This might be related to -td-rtc-hack and/or e1000 as using
> td-rtc-hack on 0.11.1 windows crashed too to bluescreen.
> 
With your guest -td-rtc-hack does nothing anyway, so you can just drop
it. Although it shouldn't cause BSOD with upstream (it had such bug in
the past and I am not sure that 0.12 has the fix). And you are using
HPET anyway so -td-rtc-hack should be nop for you.

> System does not run any multimedia or so, on top of normal windows
> 2008 small business server installation there should be only nagios
> client and apache.
Any application that uses hi-res timers will cause Windows to switch to
1024Hz timer rate.

> 
> Only thing that catches my eye on stats frozen vs normal is
> irq_exits which jumps from 200 to 10000 when frozen and irq_window
> which goes 10x.
Hm, default Windows timer rate is 64Hz, so if nothing happens in the
guest you should see at least 256 irq injection per second. In your KVM
start output I only see 4655 irq injection per second not 10000. And
trace shows that most of them are timer interrupts (vector 209 IIRC).

> 
> Host has swap partition enabled but hasn't swapped once yet,
> currently there's over 1GB of free.
And how many cpus does it have? 

BTW I see a lot of instruction emulations. That is strange. Looking at
you trace again I see that the bulk of the emulations are caused by mmio
to HPET. Please run with -no-hpet flag.

> 
> I'll update host kernel to latest stable and qemu-kvm to upstream
> later today.
> 
> Here's stats from running system:
>  efer_reload                  0       0
>  exits                386474714   23272
>  fpu_reload           173420913   12775
>  halt_exits            51894921    4294
>  halt_wakeup           51039254    4223
>  host_state_reload    188645310   13420
>  hypercalls                   0       0
>  insn_emulation       212729430   15545
>  insn_emulation_fail        324       0
>  invlpg                 5177828     134
>  io_exits              23132581     161
>  irq_exits              7485833     225
>  irq_injections        59239306    4655
>  irq_window             2147438     153
>  largepages                   0       0
>  mmio_exits           110931423    9026
>  mmu_cache_miss          468410       0
>  mmu_flooded             193153       0
>  mmu_pde_zapped          352178       0
>  mmu_pte_updated         678023       0
>  mmu_pte_write           587175       0
>  mmu_recycled             34998       0
>  mmu_shadow_zapped       454073       0
>  mmu_unsync               19084       0
>  nmi_injections               0       0
>  nmi_window                   0       0
>  pf_fixed              25171943     120
>  pf_guest               6209034       2
>  remote_tlb_flush       3044169      23
>  request_irq                  0       0
>  signal_exits                 0       0
>  tlb_flush              8422950     149
> 
> 
> -- 
> Harri.

--
			Gleb.

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: Freezing Windows 2008 x64bit guest
  2010-07-21  9:48             ` Gleb Natapov
@ 2010-07-21 10:09               ` Harri Olin
  2010-07-21 10:30                 ` Gleb Natapov
  0 siblings, 1 reply; 22+ messages in thread
From: Harri Olin @ 2010-07-21 10:09 UTC (permalink / raw)
  To: Gleb Natapov; +Cc: Christoph Adomeit, kvm@vger.kernel.org

Gleb Natapov wrote:
> On Wed, Jul 21, 2010 at 12:22:45PM +0300, Harri Olin wrote:
>> Gleb Natapov wrote:
>>> On Wed, Jul 21, 2010 at 09:25:31AM +0300, Harri Olin wrote:
>>>> Gleb Natapov kirjoitti:
>>>>> On Mon, Jul 19, 2010 at 10:17:02AM +0300, Harri Olin wrote:
>>>>>> Gleb Natapov kirjoitti:
>>>>>>> On Thu, Jul 15, 2010 at 03:19:44PM +0200, Christoph Adomeit wrote:
>>>>>>>> But one Windows 2008 64 Bit Server Standard is freezing regularly.
>>>>>>>> This happens sometimes 3 times a day, sometimes it takes 2 days
>>>>>>>> until freeze. The Windows Machine is a clean fresh install.
>>>>>> I think I have seen same problem occur on my Windows 2008 SBS SP2
>>>>>> 64bit system, but a bit less often, only like once a week.
>>>>>> Now I haven't seen crashes but only freezes with qemu on 100% and
>>>>>> virtual system unresponsive.
>>>>>>
>>>>>> qemu command line:
>>>>>> /usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive
>>>>>> file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive
>>>>>> file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net
>>>>>> nic,macaddr=C0:FF:12:FB:AA:01,model=e1000 -net tap -smp 4 -localtime
>>>>>>
>>>>>>
>>>>> Try with different model then e1000 please. Default driver that comes
>>>>> with Windows known to have problems.
>>>> Didn't help, changed to default realtek emulation, but system
>>>> freezed again. Command line:
>>>> /usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive
>>>> file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive
>>>> file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net
>>>> nic,macaddr=C0:FF:12:FB:AA:01 -net tap -smp 4 -localtime
>>>>
>>>> This time the virtual system was not totally unresponsive somehow,
>>>> system pinged just fine and I think DNS server worked too. However
>>>> on console mouse moved but didn't react to clicking, ctrl-alt-del,
>>>> etc.
>>>>
>>> Does sendkey from monitor works? qemu-kvm-0.11.1 is very old and this is
>>> not total freeze which even harder to debug. I don't see anything
>>> extraordinary in your logs. 4643 interrupt per second for 4 cpus is
>>> normal if windows runs multimedia or other app that need hi-res timers.
>>> Does your host swapping? Is there any chance that you can try upstream qemu-kvm?
>>>
>> I haven't tested sendkey. I originally tried 0.12.2 or so but with
>> that windows crashed more often with bluescreen, I went down to
>> 0.11.1. This might be related to -td-rtc-hack and/or e1000 as using
>> td-rtc-hack on 0.11.1 windows crashed too to bluescreen.
>>
> With your guest -td-rtc-hack does nothing anyway, so you can just drop
> it. Although it shouldn't cause BSOD with upstream (it had such bug in
> the past and I am not sure that 0.12 has the fix). And you are using
> HPET anyway so -td-rtc-hack should be nop for you.

Ok, I think I added it because windows guest clock lagged behind a lot 
but didn't check if it helped.

>> Only thing that catches my eye on stats frozen vs normal is
>> irq_exits which jumps from 200 to 10000 when frozen and irq_window
>> which goes 10x.
> Hm, default Windows timer rate is 64Hz, so if nothing happens in the
> guest you should see at least 256 irq injection per second. In your KVM
> start output I only see 4655 irq injection per second not 10000. And
> trace shows that most of them are timer interrupts (vector 209 IIRC).

Frozen kvm_stat can be seen here; when I grabbed this, windows did not 
answer to ping.
stats: http://mizar.remote.agasha.com/k/kvm/kvm_stat_hang.txt
trace: http://mizar.remote.agasha.com/k/kvm/kvm_trace_hang.txt

Here's another frozen state stat dump; when I got this windows did 
answer to ping but was otherwise unresponsive:
stats: http://mizar.remote.agasha.com/k/kvm/kvm_stat_2_log.txt
trace: http://mizar.remote.agasha.com/k/kvm/kvm_trace_2.txt

On both times irq_exits go to 10000 but irq_injections stay at around 
4600. On both times qemu process was at 100% cpu load.

Note that stats quoted below are from normal operation, where everything 
works.

>> Host has swap partition enabled but hasn't swapped once yet,
>> currently there's over 1GB of free.
> And how many cpus does it have? 

Host has one quad-core Q8300 and 8GB of memory.

> BTW I see a lot of instruction emulations. That is strange. Looking at
> you trace again I see that the bulk of the emulations are caused by mmio
> to HPET. Please run with -no-hpet flag.

I'll do that later today too.

> 
>> I'll update host kernel to latest stable and qemu-kvm to upstream
>> later today.
>>
>> Here's stats from running system:
>>  efer_reload                  0       0
>>  exits                386474714   23272
>>  fpu_reload           173420913   12775
>>  halt_exits            51894921    4294
>>  halt_wakeup           51039254    4223
>>  host_state_reload    188645310   13420
>>  hypercalls                   0       0
>>  insn_emulation       212729430   15545
>>  insn_emulation_fail        324       0
>>  invlpg                 5177828     134
>>  io_exits              23132581     161
>>  irq_exits              7485833     225
>>  irq_injections        59239306    4655
>>  irq_window             2147438     153
>>  largepages                   0       0
>>  mmio_exits           110931423    9026
>>  mmu_cache_miss          468410       0
>>  mmu_flooded             193153       0
>>  mmu_pde_zapped          352178       0
>>  mmu_pte_updated         678023       0
>>  mmu_pte_write           587175       0
>>  mmu_recycled             34998       0
>>  mmu_shadow_zapped       454073       0
>>  mmu_unsync               19084       0
>>  nmi_injections               0       0
>>  nmi_window                   0       0
>>  pf_fixed              25171943     120
>>  pf_guest               6209034       2
>>  remote_tlb_flush       3044169      23
>>  request_irq                  0       0
>>  signal_exits                 0       0
>>  tlb_flush              8422950     149
>>

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: Freezing Windows 2008 x64bit guest
  2010-07-21 10:09               ` Harri Olin
@ 2010-07-21 10:30                 ` Gleb Natapov
  2010-07-21 10:43                   ` Harri Olin
  2010-07-21 13:03                   ` Harri Olin
  0 siblings, 2 replies; 22+ messages in thread
From: Gleb Natapov @ 2010-07-21 10:30 UTC (permalink / raw)
  To: Harri Olin; +Cc: Christoph Adomeit, kvm@vger.kernel.org

On Wed, Jul 21, 2010 at 01:09:06PM +0300, Harri Olin wrote:
> Gleb Natapov wrote:
> >On Wed, Jul 21, 2010 at 12:22:45PM +0300, Harri Olin wrote:
> >>Gleb Natapov wrote:
> >>>On Wed, Jul 21, 2010 at 09:25:31AM +0300, Harri Olin wrote:
> >>>>Gleb Natapov kirjoitti:
> >>>>>On Mon, Jul 19, 2010 at 10:17:02AM +0300, Harri Olin wrote:
> >>>>>>Gleb Natapov kirjoitti:
> >>>>>>>On Thu, Jul 15, 2010 at 03:19:44PM +0200, Christoph Adomeit wrote:
> >>>>>>>>But one Windows 2008 64 Bit Server Standard is freezing regularly.
> >>>>>>>>This happens sometimes 3 times a day, sometimes it takes 2 days
> >>>>>>>>until freeze. The Windows Machine is a clean fresh install.
> >>>>>>I think I have seen same problem occur on my Windows 2008 SBS SP2
> >>>>>>64bit system, but a bit less often, only like once a week.
> >>>>>>Now I haven't seen crashes but only freezes with qemu on 100% and
> >>>>>>virtual system unresponsive.
> >>>>>>
> >>>>>>qemu command line:
> >>>>>>/usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive
> >>>>>>file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive
> >>>>>>file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net
> >>>>>>nic,macaddr=C0:FF:12:FB:AA:01,model=e1000 -net tap -smp 4 -localtime
> >>>>>>
> >>>>>>
> >>>>>Try with different model then e1000 please. Default driver that comes
> >>>>>with Windows known to have problems.
> >>>>Didn't help, changed to default realtek emulation, but system
> >>>>freezed again. Command line:
> >>>>/usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive
> >>>>file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive
> >>>>file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net
> >>>>nic,macaddr=C0:FF:12:FB:AA:01 -net tap -smp 4 -localtime
> >>>>
> >>>>This time the virtual system was not totally unresponsive somehow,
> >>>>system pinged just fine and I think DNS server worked too. However
> >>>>on console mouse moved but didn't react to clicking, ctrl-alt-del,
> >>>>etc.
> >>>>
> >>>Does sendkey from monitor works? qemu-kvm-0.11.1 is very old and this is
> >>>not total freeze which even harder to debug. I don't see anything
> >>>extraordinary in your logs. 4643 interrupt per second for 4 cpus is
> >>>normal if windows runs multimedia or other app that need hi-res timers.
> >>>Does your host swapping? Is there any chance that you can try upstream qemu-kvm?
> >>>
> >>I haven't tested sendkey. I originally tried 0.12.2 or so but with
> >>that windows crashed more often with bluescreen, I went down to
> >>0.11.1. This might be related to -td-rtc-hack and/or e1000 as using
> >>td-rtc-hack on 0.11.1 windows crashed too to bluescreen.
> >>
> >With your guest -td-rtc-hack does nothing anyway, so you can just drop
> >it. Although it shouldn't cause BSOD with upstream (it had such bug in
> >the past and I am not sure that 0.12 has the fix). And you are using
> >HPET anyway so -td-rtc-hack should be nop for you.
> 
> Ok, I think I added it because windows guest clock lagged behind a
> lot but didn't check if it helped.
> 
For win2k8-64bit (and win2k3-64bit) we don't have a solution for time
drift currently. And with hpet we don't have it for any windows.

> >>Only thing that catches my eye on stats frozen vs normal is
> >>irq_exits which jumps from 200 to 10000 when frozen and irq_window
> >>which goes 10x.
> >Hm, default Windows timer rate is 64Hz, so if nothing happens in the
> >guest you should see at least 256 irq injection per second. In your KVM
> >start output I only see 4655 irq injection per second not 10000. And
> >trace shows that most of them are timer interrupts (vector 209 IIRC).
> 
> Frozen kvm_stat can be seen here; when I grabbed this, windows did
> not answer to ping.
> stats: http://mizar.remote.agasha.com/k/kvm/kvm_stat_hang.txt
> trace: http://mizar.remote.agasha.com/k/kvm/kvm_trace_hang.txt
Not much happens here. Windows gets timer irqs at 1024Hz rate and
process them. What is your host kernel? Does it have hi-res timers?

> 
> Here's another frozen state stat dump; when I got this windows did
> answer to ping but was otherwise unresponsive:
> stats: http://mizar.remote.agasha.com/k/kvm/kvm_stat_2_log.txt
> trace: http://mizar.remote.agasha.com/k/kvm/kvm_trace_2.txt
> 
> On both times irq_exits go to 10000 but irq_injections stay at
> around 4600. On both times qemu process was at 100% cpu load.
> 
irq_exits are exits due to host interrupt. What "cat /proc/interrupts"
show? Which counter grows at this rate there (or 1/4 of the rate but for
each host cpu).

--
			Gleb.

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: Freezing Windows 2008 x64bit guest
  2010-07-21 10:30                 ` Gleb Natapov
@ 2010-07-21 10:43                   ` Harri Olin
  2010-07-21 13:03                   ` Harri Olin
  1 sibling, 0 replies; 22+ messages in thread
From: Harri Olin @ 2010-07-21 10:43 UTC (permalink / raw)
  To: Gleb Natapov; +Cc: Christoph Adomeit, kvm@vger.kernel.org

Gleb Natapov wrote:
> On Wed, Jul 21, 2010 at 01:09:06PM +0300, Harri Olin wrote:
>> Gleb Natapov wrote:
>> Frozen kvm_stat can be seen here; when I grabbed this, windows did
>> not answer to ping.
>> stats: http://mizar.remote.agasha.com/k/kvm/kvm_stat_hang.txt
>> trace: http://mizar.remote.agasha.com/k/kvm/kvm_trace_hang.txt
> Not much happens here. Windows gets timer irqs at 1024Hz rate and
> process them. What is your host kernel? Does it have hi-res timers?

Host kernel is vanilla 2.6.33.1 from kernel.org. Should have hi-res 
timers enabled.
full dmesg at: http://mizar.remote.agasha.com/k/kvm/dmesg.txt

[    0.415380] HPET: 4 timers in total, 0 timers will be used for 
per-cpu timer
[    0.416004] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0
[    0.416263] hpet0: 4 comparators, 64-bit 14.318180 MHz counter

>> Here's another frozen state stat dump; when I got this windows did
>> answer to ping but was otherwise unresponsive:
>> stats: http://mizar.remote.agasha.com/k/kvm/kvm_stat_2_log.txt
>> trace: http://mizar.remote.agasha.com/k/kvm/kvm_trace_2.txt
>>
>> On both times irq_exits go to 10000 but irq_injections stay at
>> around 4600. On both times qemu process was at 100% cpu load.
>>
> irq_exits are exits due to host interrupt. What "cat /proc/interrupts"
> show? Which counter grows at this rate there (or 1/4 of the rate but for
> each host cpu).

I have to check that next time when freeze happens.

# cat /proc/interrupts
            CPU0       CPU1       CPU2       CPU3
   0:       1144       1056       1081       1072   IO-APIC-edge      timer
   1:        324        329        348        329   IO-APIC-edge      i8042
   3:          1          1          0          0   IO-APIC-edge
   4:          0          0          1          1   IO-APIC-edge
   8:         16         15         15         15   IO-APIC-edge      rtc0
   9:          0          0          0          0   IO-APIC-fasteoi   acpi
  12:          1          1          0          1   IO-APIC-edge      i8042
  16:          0          0          0          0   IO-APIC-fasteoi 
uhci_hcd:usb3
  17:          0          0          0          0   IO-APIC-fasteoi 
uhci_hcd:usb4
  18:   47695411   47681526   47693374   47680775   IO-APIC-fasteoi 
arcmsr, ehci_hcd:usb1, uhci_hcd:usb5, uhci_hcd:usb8
  22:          0          0          0          0   IO-APIC-fasteoi 
uhci_hcd:usb7
  23:          0          0          0          0   IO-APIC-fasteoi 
ehci_hcd:usb2, uhci_hcd:usb6
  28:    2376183    2377506    2376812    2376143   PCI-MSI-edge      ahci
  29:   14545152   14557669   14546472   14559769   PCI-MSI-edge      eth0
NMI:          0          0          0          0   Non-maskable interrupts
LOC: 1522685098  614396609 1720798830  518770603   Local timer interrupts
SPU:          0          0          0          0   Spurious interrupts
PMI:          0          0          0          0   Performance 
monitoring interrupts
PND:          0          0          0          0   Performance pending work
RES:  109858789  114294880  107422261  113139171   Rescheduling interrupts
CAL:  228945035  210709845  246864690  232136482   Function call interrupts
TLB:       1161       2146       1090       2319   TLB shootdowns
TRM:          0          0          0          0   Thermal event interrupts
THR:          0          0          0          0   Threshold APIC interrupts
MCE:          0          0          0          0   Machine check exceptions
MCP:       4226       4226       4226       4226   Machine check polls
ERR:          0
MIS:          0

Also at: http://mizar.remote.agasha.com/k/kvm/interrupts.txt

-- 
Harri.

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: Freezing Windows 2008 x64bit guest
  2010-07-21 10:30                 ` Gleb Natapov
  2010-07-21 10:43                   ` Harri Olin
@ 2010-07-21 13:03                   ` Harri Olin
  2010-07-21 13:45                     ` Gleb Natapov
  1 sibling, 1 reply; 22+ messages in thread
From: Harri Olin @ 2010-07-21 13:03 UTC (permalink / raw)
  To: Gleb Natapov; +Cc: Christoph Adomeit, kvm@vger.kernel.org

Gleb Natapov wrote:
> On Wed, Jul 21, 2010 at 01:09:06PM +0300, Harri Olin wrote:
>> Gleb Natapov wrote:
>> On both times irq_exits go to 10000 but irq_injections stay at
>> around 4600. On both times qemu process was at 100% cpu load.
> irq_exits are exits due to host interrupt. What "cat /proc/interrupts"
> show? Which counter grows at this rate there (or 1/4 of the rate but for
> each host cpu).

System froze again just a moment ago and there are 6100 "local timer 
interrupts" and around 3600 "Rescheduling interrupts" every second 
happening.

#cat /proc/interrupts; sleep 1; cat /proc/interrupts; sleep 1; ....
http://mizar.remote.agasha.com/k/kvm/interrupts.txt
http://mizar.remote.agasha.com/k/kvm/interrupts1.txt

quick calculations
                                              total
RES: 110847929 115251972 108671867 114338799 449110567 	per second:
RES: 110848996 115252094 108673031 114340080 449114201 	3634
RES: 110850059 115252174 108674157 114341446 449117836 	3635
RES: 110851132 115252276 108675332 114342801 449121541 	3705
						
                                                total		
LOC: 1531251837 618157953 1727791012 522296117 4399496919 per second:
LOC: 1531252861 618161024 1727792036 522297141 4399503062 6143
LOC: 1531253884 618164095 1727793060 522298165 4399509204 6142
LOC: 1531254907 618167166 1727794084 522299189 4399515346 6142

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: Freezing Windows 2008 x64bit guest
  2010-07-21 13:03                   ` Harri Olin
@ 2010-07-21 13:45                     ` Gleb Natapov
  2010-07-21 14:05                       ` Harri Olin
  0 siblings, 1 reply; 22+ messages in thread
From: Gleb Natapov @ 2010-07-21 13:45 UTC (permalink / raw)
  To: Harri Olin; +Cc: Christoph Adomeit, kvm@vger.kernel.org

On Wed, Jul 21, 2010 at 04:03:28PM +0300, Harri Olin wrote:
> Gleb Natapov wrote:
> >On Wed, Jul 21, 2010 at 01:09:06PM +0300, Harri Olin wrote:
> >>Gleb Natapov wrote:
> >>On both times irq_exits go to 10000 but irq_injections stay at
> >>around 4600. On both times qemu process was at 100% cpu load.
> >irq_exits are exits due to host interrupt. What "cat /proc/interrupts"
> >show? Which counter grows at this rate there (or 1/4 of the rate but for
> >each host cpu).
> 
> System froze again just a moment ago and there are 6100 "local timer
> interrupts" and around 3600 "Rescheduling interrupts" every second
> happening.
> 
> #cat /proc/interrupts; sleep 1; cat /proc/interrupts; sleep 1; ....
> http://mizar.remote.agasha.com/k/kvm/interrupts.txt
> http://mizar.remote.agasha.com/k/kvm/interrupts1.txt
> 
> quick calculations
>                                              total
> RES: 110847929 115251972 108671867 114338799 449110567 	per second:
> RES: 110848996 115252094 108673031 114340080 449114201 	3634
> RES: 110850059 115252174 108674157 114341446 449117836 	3635
> RES: 110851132 115252276 108675332 114342801 449121541 	3705
> 						
>                                                total		
> LOC: 1531251837 618157953 1727791012 522296117 4399496919 per second:
> LOC: 1531252861 618161024 1727792036 522297141 4399503062 6143
> LOC: 1531253884 618164095 1727793060 522298165 4399509204 6142
> LOC: 1531254907 618167166 1727794084 522299189 4399515346 6142

Same qemu and command line as before? Without -no-hpet?

--
			Gleb.

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: Freezing Windows 2008 x64bit guest
  2010-07-21 13:45                     ` Gleb Natapov
@ 2010-07-21 14:05                       ` Harri Olin
  0 siblings, 0 replies; 22+ messages in thread
From: Harri Olin @ 2010-07-21 14:05 UTC (permalink / raw)
  To: Gleb Natapov; +Cc: Christoph Adomeit, kvm@vger.kernel.org

Gleb Natapov wrote:
 > On Wed, Jul 21, 2010 at 04:03:28PM +0300, Harri Olin wrote:
 >> Gleb Natapov wrote:
 >>> On Wed, Jul 21, 2010 at 01:09:06PM +0300, Harri Olin wrote:
 >>>> Gleb Natapov wrote:
 >>>> On both times irq_exits go to 10000 but irq_injections stay at
 >>>> around 4600. On both times qemu process was at 100% cpu load.
 >>> irq_exits are exits due to host interrupt. What "cat /proc/interrupts"
 >>> show? Which counter grows at this rate there (or 1/4 of the rate 
but for
 >>> each host cpu).
 >> System froze again just a moment ago and there are 6100 "local timer
 >> interrupts" and around 3600 "Rescheduling interrupts" every second
 >> happening.
 >>
 >> #cat /proc/interrupts; sleep 1; cat /proc/interrupts; sleep 1; ....
 >> http://mizar.remote.agasha.com/k/kvm/interrupts.txt
 >> http://mizar.remote.agasha.com/k/kvm/interrupts1.txt
 >>
 >> quick calculations
 >>                                              total
 >> RES: 110847929 115251972 108671867 114338799 449110567 	per second:
 >> RES: 110848996 115252094 108673031 114340080 449114201 	3634
 >> RES: 110850059 115252174 108674157 114341446 449117836 	3635
 >> RES: 110851132 115252276 108675332 114342801 449121541 	3705
 >> 						
 >>                                                total		
 >> LOC: 1531251837 618157953 1727791012 522296117 4399496919 per second:
 >> LOC: 1531252861 618161024 1727792036 522297141 4399503062 6143
 >> LOC: 1531253884 618164095 1727793060 522298165 4399509204 6142
 >> LOC: 1531254907 618167166 1727794084 522299189 4399515346 6142
 >
 > Same qemu and command line as before? Without -no-hpet?

Yes those interrupt numbers area still running qemu-kvm-0.11.1 without 
-no-hpet.

I now started the quest with -no-hpet, kvm_stat, everything working:

  efer_reload                  0       0
  exits                283535303   41437
  fpu_reload            94754003   16231
  halt_exits            24716918    4696
  halt_wakeup           24416218    4651
  host_state_reload    112571913   18552
  hypercalls                   0       0
  insn_emulation        75833651   11377
  insn_emulation_fail         83       0
  invlpg                 5327501     124
  io_exits              84146648   13718
  irq_exits             10182390    2295
  irq_injections        31379031    5979
  irq_window              941352     113
  largepages                   0       0
  mmio_exits               29794       0
  mmu_cache_miss          380265      11
  mmu_flooded             167547       2
  mmu_pde_zapped          288251       2
  mmu_pte_updated         586646       2
  mmu_pte_write           505897       3
  mmu_recycled             40892       0
  mmu_shadow_zapped       365079       4
  mmu_unsync               19338      -1
  nmi_injections               0       0
  nmi_window                   0       0
  pf_fixed              20331924    1570
  pf_guest               5332473     214
  remote_tlb_flush       3213779     211
  request_irq                  0       0
  signal_exits                 0       0
  tlb_flush              8205865     335


interrupt count calculations, system working, -no-hpet:

                                                total
LOC: 1537337119 621778834 1732851735 525376978 4417344666 per sec
LOC: 1537339334 621779137 1732851925 525377284 4417347680 3014
LOC: 1537339987 621779260 1732853664 525377824 4417350735 3055
LOC: 1537340126 621779388 1732855889 525378420 4417353823 3088

RES: 112093536 116293086 109808558 115510943 453706123
RES: 112093737 116293164 109808699 115511006 453706606 483
RES: 112093921 116293182 109808731 115511040 453706874 268
RES: 112093976 116293225 109808733 115511061 453706995 121

full /proc/interrupts at: 
http://mizar.remote.agasha.com/k/kvm/interrupts-working-no-hpet.txt

command line:
/usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive 
file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive 
file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net 
nic,macaddr=C0:FF:12:FB:AA:01 -net tap -smp 4 -localtime -no-hpet

Still couple of hours before I can restart the host to new kernel.

-- 
Harri.

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: Freezing Windows 2008 x64bit guest
  2010-07-21  8:37         ` Gleb Natapov
  2010-07-21  9:22           ` Harri Olin
@ 2010-07-27 21:53           ` Harri Olin
  2010-07-28 15:18             ` Gleb Natapov
  1 sibling, 1 reply; 22+ messages in thread
From: Harri Olin @ 2010-07-27 21:53 UTC (permalink / raw)
  To: Gleb Natapov; +Cc: Christoph Adomeit, kvm@vger.kernel.org

Gleb Natapov wrote:
> On Wed, Jul 21, 2010 at 09:25:31AM +0300, Harri Olin wrote:
>> Gleb Natapov kirjoitti:
>>> On Mon, Jul 19, 2010 at 10:17:02AM +0300, Harri Olin wrote:
>>>> Gleb Natapov kirjoitti:
>>>>> On Thu, Jul 15, 2010 at 03:19:44PM +0200, Christoph Adomeit wrote:
>>>>>> But one Windows 2008 64 Bit Server Standard is freezing regularly.
>>>>>> This happens sometimes 3 times a day, sometimes it takes 2 days
>>>>>> until freeze. The Windows Machine is a clean fresh install.
>>>> I think I have seen same problem occur on my Windows 2008 SBS SP2
>>>> 64bit system, but a bit less often, only like once a week.
>>>> Now I haven't seen crashes but only freezes with qemu on 100% and
>>>> virtual system unresponsive.
> Does sendkey from monitor works? qemu-kvm-0.11.1 is very old and this is
> not total freeze which even harder to debug. I don't see anything
> extraordinary in your logs. 4643 interrupt per second for 4 cpus is
> normal if windows runs multimedia or other app that need hi-res timers.
> Does your host swapping? Is there any chance that you can try upstream qemu-kvm?

I tried running qemu-kvm from git but it exhibited the same problem as 
12.x that I tried before, BSODing once in a while, running kernel 2.6.34.1.

sample BSOD failure details:
These two with Realtec nic and qemu cpu
0x00000019 (0x0000000000000020, 0xfffff88007e65970, 0xfffff88007e65990, 
0x000000000502040f)
0x00000019 (0x0000000000000020, 0xfffff88007a414c0, 0xfffff88007a414e0, 
0x000000000502044c)

These are with e1000 and -cpu host
0x0000003b (0x00000000c0000005, 0xfffff80001c5d842, 0xfffffa60093ddb70, 
0x0000000000000000)
0x0000003b (0x00000000c0000005, 0xfffff80001cb8842, 0xfffffa600c94ab70, 
0x0000000000000000)
0x0000000a (0x0000000000000080, 0x000000000000000c, 0x0000000000000001, 
0xfffff80001cadefd)

I'll see if I can analyze minidumps later.

In addition to these there have been as many reboots that have been only 
logged as 'disruptive shutdown'.

Right now I'm running the problematic guest under Xen 3.2.1-something 
from Debian to see if it works better.

-- 
Harri.

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: Freezing Windows 2008 x64bit guest
  2010-07-27 21:53           ` Harri Olin
@ 2010-07-28 15:18             ` Gleb Natapov
  2010-12-13 19:42               ` Manfred Heubach
  0 siblings, 1 reply; 22+ messages in thread
From: Gleb Natapov @ 2010-07-28 15:18 UTC (permalink / raw)
  To: Harri Olin; +Cc: Christoph Adomeit, kvm@vger.kernel.org

On Wed, Jul 28, 2010 at 12:53:02AM +0300, Harri Olin wrote:
> Gleb Natapov wrote:
> >On Wed, Jul 21, 2010 at 09:25:31AM +0300, Harri Olin wrote:
> >>Gleb Natapov kirjoitti:
> >>>On Mon, Jul 19, 2010 at 10:17:02AM +0300, Harri Olin wrote:
> >>>>Gleb Natapov kirjoitti:
> >>>>>On Thu, Jul 15, 2010 at 03:19:44PM +0200, Christoph Adomeit wrote:
> >>>>>>But one Windows 2008 64 Bit Server Standard is freezing regularly.
> >>>>>>This happens sometimes 3 times a day, sometimes it takes 2 days
> >>>>>>until freeze. The Windows Machine is a clean fresh install.
> >>>>I think I have seen same problem occur on my Windows 2008 SBS SP2
> >>>>64bit system, but a bit less often, only like once a week.
> >>>>Now I haven't seen crashes but only freezes with qemu on 100% and
> >>>>virtual system unresponsive.
> >Does sendkey from monitor works? qemu-kvm-0.11.1 is very old and this is
> >not total freeze which even harder to debug. I don't see anything
> >extraordinary in your logs. 4643 interrupt per second for 4 cpus is
> >normal if windows runs multimedia or other app that need hi-res timers.
> >Does your host swapping? Is there any chance that you can try upstream qemu-kvm?
> 
> I tried running qemu-kvm from git but it exhibited the same problem
> as 12.x that I tried before, BSODing once in a while, running kernel
> 2.6.34.1.
> 
That should be pretty stable config, although it would be nice if you
could try running in qemy-kvm.git head.

> sample BSOD failure details:
> These two with Realtec nic and qemu cpu
> 0x00000019 (0x0000000000000020, 0xfffff88007e65970,
> 0xfffff88007e65990, 0x000000000502040f)
> 0x00000019 (0x0000000000000020, 0xfffff88007a414c0,
> 0xfffff88007a414e0, 0x000000000502044c)
> 
> These are with e1000 and -cpu host
> 0x0000003b (0x00000000c0000005, 0xfffff80001c5d842,
> 0xfffffa60093ddb70, 0x0000000000000000)
> 0x0000003b (0x00000000c0000005, 0xfffff80001cb8842,
> 0xfffffa600c94ab70, 0x0000000000000000)
> 0x0000000a (0x0000000000000080, 0x000000000000000c,
> 0x0000000000000001, 0xfffff80001cadefd)
> 
Can you attach screenshots of BSODs? Have you reinstalled your guests or
are you running the same images you ran in 11.x?

> I'll see if I can analyze minidumps later.
> 
> In addition to these there have been as many reboots that have been
> only logged as 'disruptive shutdown'.
> 
> Right now I'm running the problematic guest under Xen
> 3.2.1-something from Debian to see if it works better.
> 
> -- 
> Harri.

--
			Gleb.

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: Freezing Windows 2008 x64bit guest
  2010-07-28 15:18             ` Gleb Natapov
@ 2010-12-13 19:42               ` Manfred Heubach
  2010-12-13 20:12                 ` Dor Laor
  0 siblings, 1 reply; 22+ messages in thread
From: Manfred Heubach @ 2010-12-13 19:42 UTC (permalink / raw)
  To: kvm

Gleb Natapov <gleb <at> redhat.com> writes:

> 
> On Wed, Jul 28, 2010 at 12:53:02AM +0300, Harri Olin wrote:
> > Gleb Natapov wrote:
> > >On Wed, Jul 21, 2010 at 09:25:31AM +0300, Harri Olin wrote:
> > >>Gleb Natapov kirjoitti:
> > >>>On Mon, Jul 19, 2010 at 10:17:02AM +0300, Harri Olin wrote:
> > >>>>Gleb Natapov kirjoitti:
> > >>>>>On Thu, Jul 15, 2010 at 03:19:44PM +0200, Christoph Adomeit wrote:
> > >>>>>>But one Windows 2008 64 Bit Server Standard is freezing regularly.
> > >>>>>>This happens sometimes 3 times a day, sometimes it takes 2 days
> > >>>>>>until freeze. The Windows Machine is a clean fresh install.
> > >>>>I think I have seen same problem occur on my Windows 2008 SBS SP2
> > >>>>64bit system, but a bit less often, only like once a week.
> > >>>>Now I haven't seen crashes but only freezes with qemu on 100% and
> > >>>>virtual system unresponsive.
> > >Does sendkey from monitor works? qemu-kvm-0.11.1 is very old and this is
> > >not total freeze which even harder to debug. I don't see anything
> > >extraordinary in your logs. 4643 interrupt per second for 4 cpus is
> > >normal if windows runs multimedia or other app that need hi-res timers.
> > >Does your host swapping? Is there any chance that you can try upstream
qemu-kvm?
> > 
> > I tried running qemu-kvm from git but it exhibited the same problem
> > as 12.x that I tried before, BSODing once in a while, running kernel
> > 2.6.34.1.
> > 
> That should be pretty stable config, although it would be nice if you
> could try running in qemy-kvm.git head.
> 
> > sample BSOD failure details:
> > These two with Realtec nic and qemu cpu
> > 0x00000019 (0x0000000000000020, 0xfffff88007e65970,
> > 0xfffff88007e65990, 0x000000000502040f)
> > 0x00000019 (0x0000000000000020, 0xfffff88007a414c0,
> > 0xfffff88007a414e0, 0x000000000502044c)
> > 
> > These are with e1000 and -cpu host
> > 0x0000003b (0x00000000c0000005, 0xfffff80001c5d842,
> > 0xfffffa60093ddb70, 0x0000000000000000)
> > 0x0000003b (0x00000000c0000005, 0xfffff80001cb8842,
> > 0xfffffa600c94ab70, 0x0000000000000000)
> > 0x0000000a (0x0000000000000080, 0x000000000000000c,
> > 0x0000000000000001, 0xfffff80001cadefd)
> > 
> Can you attach screenshots of BSODs? Have you reinstalled your guests or
> are you running the same images you ran in 11.x?
> 
> > I'll see if I can analyze minidumps later.
> > 
> > In addition to these there have been as many reboots that have been
> > only logged as 'disruptive shutdown'.
> > 
> > Right now I'm running the problematic guest under Xen
> > 3.2.1-something from Debian to see if it works better.
> > 
> > -- 
> > Harri.
>
 Hello,

is there a solution for that problem? I'm experiencing the same problems ever
since I installed SBS 2008 on KVM.

I was running the host with Ubuntu 10.04 but upgraded to 10.10 - mainly because
of performance problems which were solved by the upgrade.

After the upgrade the system became extremly unstable. It was crashing as soon
as disk io and network io load was growing. 100% reproduceable with windows
server backup to an iscsi volume.

i had virtio drivers for storage and network installed (redhat/fedora 1.1.11).
At each BSOD I had the following line in the log of the guest:

 virtio_ioport_write: unexpected address 0x13 value 0x1

I changed the network interface back to e1000. What I experience now (and I had
that a the very beginning before i switched to virtio network) are freezes. The
guest doesn't respond anymore (doesn't answer to pings and doesn't interact via
mouse/keyboard anymore). Host CPU usage of the kvm process is 100% on as many
cores as there are virtual cpus (in this case 4).

I'm a bit frustrated about this. I have 2 windows 2003 32bit, 1 windows xp and 3
linux guests (2x 32bit, 1x64 bit). They are all running without any problems
(except that the windows xp guest cannot boot without an ntldr cd image). Only
the SBS2008 guest regulary freezes.

The host system has 2 Intel Xeon 5504, Intel Chipset 5500, Adaptec Raid 5805, 24
GB DDR3 RAM.

I know there is a lack of detailed information right now. I first need to know
if anybody is working on this or has similar problems. I can deliver minidumps,
and any debugging information you need.

I don't want to give up now. We will switch to Hyper-V if we cannot solve this,
because we need a stable virtualization plattform for Windows Guests. I would
like to use KVM it is so much more flexibel.

Best regards
Manfred

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: Freezing Windows 2008 x64bit guest
  2010-12-13 19:42               ` Manfred Heubach
@ 2010-12-13 20:12                 ` Dor Laor
  2010-12-13 20:44                   ` Vadim Rozenfeld
  0 siblings, 1 reply; 22+ messages in thread
From: Dor Laor @ 2010-12-13 20:12 UTC (permalink / raw)
  To: Manfred Heubach; +Cc: kvm, Vadim Rozenfeld

On 12/13/2010 09:42 PM, Manfred Heubach wrote:
>
>
> Gleb Natapov<gleb<at>  redhat.com>  writes:
>
>>
>> On Wed, Jul 28, 2010 at 12:53:02AM +0300, Harri Olin wrote:
>>> Gleb Natapov wrote:
>>>> On Wed, Jul 21, 2010 at 09:25:31AM +0300, Harri Olin wrote:
>>>>> Gleb Natapov kirjoitti:
>>>>>> On Mon, Jul 19, 2010 at 10:17:02AM +0300, Harri Olin wrote:
>>>>>>> Gleb Natapov kirjoitti:
>>>>>>>> On Thu, Jul 15, 2010 at 03:19:44PM +0200, Christoph Adomeit wrote:
>>>>>>>>> But one Windows 2008 64 Bit Server Standard is freezing regularly.
>>>>>>>>> This happens sometimes 3 times a day, sometimes it takes 2 days
>>>>>>>>> until freeze. The Windows Machine is a clean fresh install.
>>>>>>> I think I have seen same problem occur on my Windows 2008 SBS SP2
>>>>>>> 64bit system, but a bit less often, only like once a week.
>>>>>>> Now I haven't seen crashes but only freezes with qemu on 100% and
>>>>>>> virtual system unresponsive.
>>>> Does sendkey from monitor works? qemu-kvm-0.11.1 is very old and this is
>>>> not total freeze which even harder to debug. I don't see anything
>>>> extraordinary in your logs. 4643 interrupt per second for 4 cpus is
>>>> normal if windows runs multimedia or other app that need hi-res timers.
>>>> Does your host swapping? Is there any chance that you can try upstream
> qemu-kvm?
>>>
>>> I tried running qemu-kvm from git but it exhibited the same problem
>>> as 12.x that I tried before, BSODing once in a while, running kernel
>>> 2.6.34.1.
>>>
>> That should be pretty stable config, although it would be nice if you
>> could try running in qemy-kvm.git head.
>>
>>> sample BSOD failure details:
>>> These two with Realtec nic and qemu cpu
>>> 0x00000019 (0x0000000000000020, 0xfffff88007e65970,
>>> 0xfffff88007e65990, 0x000000000502040f)
>>> 0x00000019 (0x0000000000000020, 0xfffff88007a414c0,
>>> 0xfffff88007a414e0, 0x000000000502044c)
>>>
>>> These are with e1000 and -cpu host
>>> 0x0000003b (0x00000000c0000005, 0xfffff80001c5d842,
>>> 0xfffffa60093ddb70, 0x0000000000000000)
>>> 0x0000003b (0x00000000c0000005, 0xfffff80001cb8842,
>>> 0xfffffa600c94ab70, 0x0000000000000000)
>>> 0x0000000a (0x0000000000000080, 0x000000000000000c,
>>> 0x0000000000000001, 0xfffff80001cadefd)
>>>
>> Can you attach screenshots of BSODs? Have you reinstalled your guests or
>> are you running the same images you ran in 11.x?
>>
>>> I'll see if I can analyze minidumps later.
>>>
>>> In addition to these there have been as many reboots that have been
>>> only logged as 'disruptive shutdown'.
>>>
>>> Right now I'm running the problematic guest under Xen
>>> 3.2.1-something from Debian to see if it works better.
>>>
>>> --
>>> Harri.
>>
>   Hello,
>
> is there a solution for that problem? I'm experiencing the same problems ever
> since I installed SBS 2008 on KVM.
>
> I was running the host with Ubuntu 10.04 but upgraded to 10.10 - mainly because
> of performance problems which were solved by the upgrade.
>
> After the upgrade the system became extremly unstable. It was crashing as soon
> as disk io and network io load was growing. 100% reproduceable with windows
> server backup to an iscsi volume.
>
> i had virtio drivers for storage and network installed (redhat/fedora 1.1.11).

Which fedora/rhel release is that?
What's the windows virtio driver version?

Have you tried using virt-manager/virhs instead of raw cmdline?
About e1000, some windows comes with buggy driver and an update e1000 
from Intel fixes some issues.


> At each BSOD I had the following line in the log of the guest:
>
>   virtio_ioport_write: unexpected address 0x13 value 0x1
>
> I changed the network interface back to e1000. What I experience now (and I had
> that a the very beginning before i switched to virtio network) are freezes. The
> guest doesn't respond anymore (doesn't answer to pings and doesn't interact via
> mouse/keyboard anymore). Host CPU usage of the kvm process is 100% on as many
> cores as there are virtual cpus (in this case 4).
>
> I'm a bit frustrated about this. I have 2 windows 2003 32bit, 1 windows xp and 3
> linux guests (2x 32bit, 1x64 bit). They are all running without any problems
> (except that the windows xp guest cannot boot without an ntldr cd image). Only
> the SBS2008 guest regulary freezes.
>
> The host system has 2 Intel Xeon 5504, Intel Chipset 5500, Adaptec Raid 5805, 24
> GB DDR3 RAM.
>
> I know there is a lack of detailed information right now. I first need to know
> if anybody is working on this or has similar problems. I can deliver minidumps,
> and any debugging information you need.
>
> I don't want to give up now. We will switch to Hyper-V if we cannot solve this,
> because we need a stable virtualization plattform for Windows Guests. I would
> like to use KVM it is so much more flexibel.
>
> Best regards
> Manfred
>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: Freezing Windows 2008 x64bit guest
  2010-12-13 20:12                 ` Dor Laor
@ 2010-12-13 20:44                   ` Vadim Rozenfeld
  2010-12-14 23:57                     ` AW: " Manfred Heubach
  0 siblings, 1 reply; 22+ messages in thread
From: Vadim Rozenfeld @ 2010-12-13 20:44 UTC (permalink / raw)
  To: dlaor; +Cc: Manfred Heubach, kvm

On Mon, 2010-12-13 at 22:12 +0200, Dor Laor wrote:
> On 12/13/2010 09:42 PM, Manfred Heubach wrote:
> >
> >
> > Gleb Natapov<gleb<at>  redhat.com>  writes:
> >
> >>
> >> On Wed, Jul 28, 2010 at 12:53:02AM +0300, Harri Olin wrote:
> >>> Gleb Natapov wrote:
> >>>> On Wed, Jul 21, 2010 at 09:25:31AM +0300, Harri Olin wrote:
> >>>>> Gleb Natapov kirjoitti:
> >>>>>> On Mon, Jul 19, 2010 at 10:17:02AM +0300, Harri Olin wrote:
> >>>>>>> Gleb Natapov kirjoitti:
> >>>>>>>> On Thu, Jul 15, 2010 at 03:19:44PM +0200, Christoph Adomeit wrote:
> >>>>>>>>> But one Windows 2008 64 Bit Server Standard is freezing regularly.
> >>>>>>>>> This happens sometimes 3 times a day, sometimes it takes 2 days
> >>>>>>>>> until freeze. The Windows Machine is a clean fresh install.
> >>>>>>> I think I have seen same problem occur on my Windows 2008 SBS SP2
> >>>>>>> 64bit system, but a bit less often, only like once a week.
> >>>>>>> Now I haven't seen crashes but only freezes with qemu on 100% and
> >>>>>>> virtual system unresponsive.
> >>>> Does sendkey from monitor works? qemu-kvm-0.11.1 is very old and this is
> >>>> not total freeze which even harder to debug. I don't see anything
> >>>> extraordinary in your logs. 4643 interrupt per second for 4 cpus is
> >>>> normal if windows runs multimedia or other app that need hi-res timers.
> >>>> Does your host swapping? Is there any chance that you can try upstream
> > qemu-kvm?
> >>>
> >>> I tried running qemu-kvm from git but it exhibited the same problem
> >>> as 12.x that I tried before, BSODing once in a while, running kernel
> >>> 2.6.34.1.
> >>>
> >> That should be pretty stable config, although it would be nice if you
> >> could try running in qemy-kvm.git head.
> >>
> >>> sample BSOD failure details:
> >>> These two with Realtec nic and qemu cpu
> >>> 0x00000019 (0x0000000000000020, 0xfffff88007e65970,
> >>> 0xfffff88007e65990, 0x000000000502040f)
> >>> 0x00000019 (0x0000000000000020, 0xfffff88007a414c0,
> >>> 0xfffff88007a414e0, 0x000000000502044c)
> >>>
> >>> These are with e1000 and -cpu host
> >>> 0x0000003b (0x00000000c0000005, 0xfffff80001c5d842,
> >>> 0xfffffa60093ddb70, 0x0000000000000000)
> >>> 0x0000003b (0x00000000c0000005, 0xfffff80001cb8842,
> >>> 0xfffffa600c94ab70, 0x0000000000000000)
> >>> 0x0000000a (0x0000000000000080, 0x000000000000000c,
> >>> 0x0000000000000001, 0xfffff80001cadefd)
> >>>
> >> Can you attach screenshots of BSODs? Have you reinstalled your guests or
> >> are you running the same images you ran in 11.x?
> >>
> >>> I'll see if I can analyze minidumps later.
> >>>
> >>> In addition to these there have been as many reboots that have been
> >>> only logged as 'disruptive shutdown'.
> >>>
> >>> Right now I'm running the problematic guest under Xen
> >>> 3.2.1-something from Debian to see if it works better.
> >>>
> >>> --
> >>> Harri.
> >>
> >   Hello,
> >
> > is there a solution for that problem? I'm experiencing the same problems ever
> > since I installed SBS 2008 on KVM.
> >
> > I was running the host with Ubuntu 10.04 but upgraded to 10.10 - mainly because
> > of performance problems which were solved by the upgrade.
> >
> > After the upgrade the system became extremly unstable. It was crashing as soon
> > as disk io and network io load was growing. 100% reproduceable with windows
> > server backup to an iscsi volume.
> >
> > i had virtio drivers for storage and network installed (redhat/fedora 1.1.11).
> 
> Which fedora/rhel release is that?
> What's the windows virtio driver version?
> 
> Have you tried using virt-manager/virhs instead of raw cmdline?
> About e1000, some windows comes with buggy driver and an update e1000 
> from Intel fixes some issues.
> 
> 
> > At each BSOD I had the following line in the log of the guest:
> >
> >   virtio_ioport_write: unexpected address 0x13 value 0x1
> >
> > I changed the network interface back to e1000. What I experience now (and I had
> > that a the very beginning before i switched to virtio network) are freezes. The
> > guest doesn't respond anymore (doesn't answer to pings and doesn't interact via
> > mouse/keyboard anymore). Host CPU usage of the kvm process is 100% on as many
> > cores as there are virtual cpus (in this case 4).
> >
Sounds like an interrupt storm to me. Can you try to ping your VM?
Anyway the best way to start debugging a stalled system is just to crash
it with BSOD. For doing it you will need:
- enable NMICrashDump (please see http://support.microsoft.com/kb/927069
for more information
- enable Kernel Memory Dump (actually Complete is much better, but it
can be too big)  http://support.microsoft.com/kb/969028
- you only will need to type "nmi 0" in the qemu monitor to crash the
system, when the system hangs next time.
Best regards,
Vadim. 
> > I'm a bit frustrated about this. I have 2 windows 2003 32bit, 1 windows xp and 3
> > linux guests (2x 32bit, 1x64 bit). They are all running without any problems
> > (except that the windows xp guest cannot boot without an ntldr cd image). Only
> > the SBS2008 guest regulary freezes.
> >
> > The host system has 2 Intel Xeon 5504, Intel Chipset 5500, Adaptec Raid 5805, 24
> > GB DDR3 RAM.
> >
> > I know there is a lack of detailed information right now. I first need to know
> > if anybody is working on this or has similar problems. I can deliver minidumps,
> > and any debugging information you need.
> >
> > I don't want to give up now. We will switch to Hyper-V if we cannot solve this,
> > because we need a stable virtualization plattform for Windows Guests. I would
> > like to use KVM it is so much more flexibel.
> >
> > Best regards
> > Manfred
> >
> >
> >
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe kvm" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 



^ permalink raw reply	[flat|nested] 22+ messages in thread

* AW: Freezing Windows 2008 x64bit guest
  2010-12-13 20:44                   ` Vadim Rozenfeld
@ 2010-12-14 23:57                     ` Manfred Heubach
  2010-12-15 10:48                       ` Vadim Rozenfeld
  0 siblings, 1 reply; 22+ messages in thread
From: Manfred Heubach @ 2010-12-14 23:57 UTC (permalink / raw)
  To: Vadim Rozenfeld, dlaor@redhat.com; +Cc: kvm@vger.kernel.org

Vadim Rozenfeld <vrozenfe <at> redhat.com> writes:

>
> On Mon, 2010-12-13 at 22:12 +0200, Dor Laor wrote:
> > On 12/13/2010 09:42 PM, Manfred Heubach wrote:
> > >
> > > I was running the host with Ubuntu 10.04 but upgraded to 10.10 - mainly
because
> > > of performance problems which were solved by the upgrade.
> > >
> > > After the upgrade the system became extremly unstable. It was crashing as soon
> > > as disk io and network io load was growing. 100% reproduceable with windows
> > > server backup to an iscsi volume.
> > >
> > > i had virtio drivers for storage and network installed (redhat/fedora 1.1.11).
> >
> > Which fedora/rhel release is that?


The host is Ubuntu 10.10 x64

The drivers are from
http://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/ 1.1.11-0
released on 17-Aug-2010 - are there any newer drivers?


> > What's the windows virtio driver version?

The virtio storage version shown in Windows is 6.0.0.10

> >
> > Have you tried using virt-manager/virhs instead of raw cmdline?

I'm starting it with libvirt/virsh

cmd-line copied from the log (and some log entries):

LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.12 -enable-kvm -m 8192 -smp
4,sockets=4,cores=1,threads=1 -name sbs2008 -uuid
933c2ef2-e5b0-0b39-db60-016b5d226534 -nodefaults -chardev
socket,id=monitor,path=/var/lib/libvirt/qemu/sbs2008.monitor,server,nowait -mon
chardev=monitor,mode=readline -rtc base=localtime -boot c -drive
file=/var/lib/libvirt/images/olscanner/virtio-win-1.1.11-0.iso,if=none,media=cdrom,
id=drive-ide0-1-0,readonly=on,format=raw
-device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive
file=/var/lib/libvirt/images/sbs2008/sbs2008.img,if=none,id=drive-virtio-disk0,
boot=on,format=qcow2 -device
virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
-drive
file=/dev/volg1/sbsdata,if=none,id=drive-virtio-disk1,format=raw,cache=none
-device
virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1
-drive file=/dev/volg1/wsus,if=none,id=drive-virtio-disk2,format=raw,cache=none
-device
virtio-blk-pci,bus=pci.0,addr=0x7,drive=drive-virtio-disk2,id=virtio-disk2
-device e1000,vlan=0,id=net0,mac=52:54:00:8a:bc:c9,bus=pci.0,addr=0x6 -net
tap,fd=107,vlan=0,name=hostnet0 -chardev pty,id=serial0 -device
isa-serial,chardev=serial0 -usb -device usb-tablet,id=input0 -vnc 0.0.0.0:0 -k
de -vga std -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3
07:12:02.715: debug : qemudInitCpuAffinity:2423 : Setting CPU affinity
07:12:02.717: debug : qemuSecurityDACSetProcessLabel:547 : Dropping privileges
of VM to 105:114
char device redirected to /dev/pts/0
pci_add_option_rom: failed to find romfile "pxe-e1000.bin"


> > About e1000, some windows comes with buggy driver and an update e1000
> > from Intel fixes some issues.
> >

I'm running latest drivers from Intel: 8.3.15.0

> >
> > > At each BSOD I had the following line in the log of the guest:
> > >
> > >   virtio_ioport_write: unexpected address 0x13 value 0x1
> > >
> > > I changed the network interface back to e1000. What I experience now (and
I had
> > > that a the very beginning before i switched to virtio network) are
freezes. The
> > > guest doesn't respond anymore (doesn't answer to pings and doesn't
interact via
> > > mouse/keyboard anymore). Host CPU usage of the kvm process is 100% on as many
> > > cores as there are virtual cpus (in this case 4).

I had a crash today but no logentry on the host - but that could be because I
had to restart syslog (ran out of diskspace after turning on debug logging ob
libvirtd - didn't think that it would generate 6 GB of logs per day :-)

> > >
> Sounds like an interrupt storm to me. Can you try to ping your VM?

No responds to ping.

> Anyway the best way to start debugging a stalled system is just to crash
> it with BSOD. For doing it you will need:
> - enable NMICrashDump (please see http://support.microsoft.com/kb/927069
> for more information
> - enable Kernel Memory Dump (actually Complete is much better, but it
> can be too big)  http://support.microsoft.com/kb/969028
> - you only will need to type "nmi 0" in the qemu monitor to crash the
> system, when the system hangs next time.

I prepared this. When the system crashed today I didn't have the complete
memory dump ready - so I only have a minidump. The intersting point is that
the system today crashed with a BSOD and didn't freeze.

The result of dumpchk.exe is as follows:


Microsoft (R) Windows Debugger Version 6.12.0002.633 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [c:\Windows\Minidump\Mini121410-01.dmp]
Mini Kernel Dump File: Only registers and stack trace are available

Symbol search path is: SRV*http://msdl.microsoft.com/download/symbols
Executable search path is:
Windows Server 2008/Windows Vista Kernel Version 6002 (Service Pack 2) MP (4
procs) Free x64
Product: LanManNt, suite: SmallBusiness TerminalServer SmallBusinessRestricted
SingleUserTS
Built by: 6002.18267.amd64fre.vistasp2_gdr.100608-0458
Machine Name:
Kernel base = 0xfffff800`01e16000 PsLoadedModuleList = 0xfffff800`01fdadd0
Debug session time: Tue Dec 14 18:41:20.058 2010 (UTC + 1:00)
System Uptime: 1 days 11:16:05.967
Loading Kernel Symbols
...............................................................
............................................................
Loading User Symbols
Loading unloaded module list
.....
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck 3B, {c0000005, fffff80001e6a842, fffffa6004efcb70, 0}

Probably caused by : Npfs.SYS ( Npfs!NpSetAttributeInList+9a )

Followup: MachineOwner
---------

----- 64 bit Kernel Mini Dump Analysis

DUMP_HEADER64:
MajorVersion        0000000f
MinorVersion        00001772
KdSecondaryVersion  00000000
DirectoryTableBase  00000001`e5b51000
PfnDataBase         fffff800`0203d250
PsLoadedModuleList  fffff800`01fdadd0
PsActiveProcessHead fffff800`01fba440
MachineImageType    00008664
NumberProcessors    00000004
BugCheckCode        0000003b
BugCheckParameter1  00000000`c0000005
BugCheckParameter2  fffff800`01e6a842
BugCheckParameter3  fffffa60`04efcb70
BugCheckParameter4  00000000`00000000
KdDebuggerDataBlock fffff800`01f87f20
ProductType         00000002
SuiteMask           00000131
WriterStatus        00000000
MiniDumpFields      00000cff

TRIAGE_DUMP64:
ServicePackBuild      00000200
SizeOfDump            00040000
ValidOffset           0003fffc
ContextOffset         00000348
ExceptionOffset       00000f00
MmOffset              00002080
UnloadedDriversOffset 000020d0
PrcbOffset            000021f0
ProcessOffset         00005d10
ThreadOffset          000060f8
CallStackOffset       00006548
SizeOfCallStack       00001bf8
DriverListOffset      00008470
DriverCount           0000007c
StringPoolOffset      0000ca30
StringPoolSize        00002950
BrokenDriverOffset    00000000
TriageOptions         ffffffff
TopOfStack            fffffa60`04efc1b8
BStoreOffset          00000000
SizeOfBStore          00000000
LimitOfBStore         00000000`00000000
DebuggerDataOffset    00008140
DebuggerDataSize      00000330
DataBlocksOffset      0000f380
DataBlocksCount       00000008
  fffff800`01fdb720 - fffff800`01fdb79f at offset 0000f400
  fffff800`01fbf898 - fffff800`01fbf89b at offset 0000f480
  fffff800`01fbf894 - fffff800`01fbf897 at offset 0000f484
  fffff800`0203d078 - fffff800`0203d07b at offset 0000f488
  fffff800`0203d09c - fffff800`0203d09f at offset 0000f48c
  fffff800`01e6f000 - fffff800`01e6ffff at offset 0000f490
  fffff800`02019000 - fffff800`02019fff at offset 00010490
  fffff800`01e70000 - fffff800`01e70fff at offset 00011490
  Max offset 12490, 2e2f8 from end of file


Windows Server 2008/Windows Vista Kernel Version 6002 (Service Pack 2) MP (4
procs) Free x64
Product: LanManNt, suite: SmallBusiness TerminalServer SmallBusinessRestricted
SingleUserTS
Built by: 6002.18267.amd64fre.vistasp2_gdr.100608-0458
Machine Name:
Kernel base = 0xfffff800`01e16000 PsLoadedModuleList = 0xfffff800`01fdadd0
Debug session time: Tue Dec 14 18:41:20.058 2010 (UTC + 1:00)
System Uptime: 1 days 11:16:05.967
start             end                 module name
fffff800`01e16000 fffff800`0232d000   nt        Tue Jun 08 16:59:47 2010 (4C0E5AE3)
fffff800`0232d000 fffff800`02373000   hal       Sat Apr 11 09:04:56 2009 (49E04118)
fffff960`000b0000 fffff960`00364000   win32k    Tue Aug 31 16:57:22 2010 (4C7D1852)
fffff960`004d0000 fffff960`004ee000   dxg       Sat Apr 11 07:09:45 2009 (49E02619)
fffff960`00680000 fffff960`0068a000   TSDDD     Sat Jan 19 07:42:04 2008 (47919BBC)
fffff960`008c0000 fffff960`008c9000   framebuf  unavailable (00000000)
fffff960`00a80000 fffff960`00ab0000   RDPDD     unavailable (00000000)
fffffa60`00604000 fffffa60`0060e000   kdcom     Sat Apr 11 09:07:57 2009 (49E041CD)
fffffa60`0060e000 fffffa60`00649000   mcupdate_GenuineIntel  Sat Apr 11
09:05:41 2009 (49E04145)
fffffa60`00649000 fffffa60`0065d000   PSHED     Sat Apr 11 09:08:17 2009 (49E041E1)
fffffa60`0065d000 fffffa60`006ba000   CLFS      Sat Apr 11 06:54:21 2009 (49E0227D)
fffffa60`006ba000 fffffa60`0076c000   CI        Sat Apr 11 09:08:09 2009 (49E041D9)
fffffa60`0076c000 fffffa60`0077e000   HIDCLASS  Sat Apr 11 07:39:32 2009 (49E02D14)
fffffa60`0077e000 fffffa60`00785b80   HIDPARSE  Sat Jan 19 07:33:51 2008 (479199CF)
fffffa60`00788000 fffffa60`007d8000   msrpc     Sat Apr 11 07:32:13 2009 (49E02B5D)
fffffa60`007d8000 fffffa60`007f5000   tdx       Sat Apr 11 07:43:00 2009 (49E02DE4)
fffffa60`007f5000 fffffa60`00800000   mouhid    Sat Jan 19 07:28:10 2008 (4791987A)
fffffa60`00800000 fffffa60`00809000   hidusb    Sat Apr 11 07:39:32 2009 (49E02D14)
fffffa60`0080a000 fffffa60`009cd000   NDIS      Sat Apr 11 07:43:15 2009 (49E02DF3)
fffffa60`009cd000 fffffa60`009e1000   volmgr    Sat Apr 11 07:34:49 2009 (49E02BF9)
fffffa60`009e1000 fffffa60`009f2000   Npfs      Sat Apr 11 06:54:22 2009 (49E0227E)
fffffa60`009f2000 fffffa60`009fb000   rasacd    Sat Jan 19 07:37:30 2008 (47919AAA)
fffffa60`00a02000 fffffa60`00a5b000   NETIO     Sat Apr 11 07:43:34 2009 (49E02E06)
fffffa60`00a5b000 fffffa60`00b35000   Wdf01000  Sat Jan 19 07:33:27 2008 (479199B7)
fffffa60`00b35000 fffffa60`00b43000   WDFLDR    Sat Jan 19 07:32:33 2008 (47919981)
fffffa60`00b43000 fffffa60`00b99000   acpi      Sat Apr 11 07:03:26 2009 (49E0249E)
fffffa60`00b99000 fffffa60`00ba2000   WMILIB    Sat Jan 19 07:33:45 2008 (479199C9)
fffffa60`00ba2000 fffffa60`00bac000   msisadrv  Sat Jan 19 07:02:50 2008 (4791928A)
fffffa60`00bac000 fffffa60`00bdc000   pci       Sat Apr 11 07:03:33 2009 (49E024A5)
fffffa60`00bdc000 fffffa60`00bf1000   partmgr   Sat Apr 11 07:34:51 2009 (49E02BFB)
fffffa60`00bf1000 fffffa60`00bfc000   Msfs      Sat Jan 19 06:53:55 2008 (47919073)
fffffa60`00c00000 fffffa60`00c09000   rdpencdd  Sat Jan 19 07:42:03 2008 (47919BBB)
fffffa60`00c0d000 fffffa60`00c73000   volmgrx   Sat Apr 11 07:35:14 2009 (49E02C12)
fffffa60`00c73000 fffffa60`00c7b000   intelide  Sat Jan 19 07:28:56 2008 (479198A8)
fffffa60`00c7b000 fffffa60`00c8b000   PCIIDEX   Sat Apr 11 07:34:22 2009 (49E02BDE)
fffffa60`00c8b000 fffffa60`00c9e000   mountmgr  Sat Jan 19 07:28:01 2008 (47919871)
fffffa60`00c9e000 fffffa60`00ca6000   atapi     Sat Apr 11 07:34:19 2009 (49E02BDB)
fffffa60`00ca6000 fffffa60`00cca000   ataport   Sat Apr 11 07:34:25 2009 (49E02BE1)
fffffa60`00cca000 fffffa60`00cd3000   viostor   Tue Aug 10 15:58:06 2010 (4C615AEE)
fffffa60`00cd3000 fffffa60`00d30000   storport  Sat Apr 11 07:34:45 2009 (49E02BF5)
fffffa60`00d30000 fffffa60`00d77000   fltmgr    Sat Apr 11 06:54:38 2009 (49E0228E)
fffffa60`00d77000 fffffa60`00da2000   quota     Sat Apr 11 06:59:38 2009 (49E023BA)
fffffa60`00da2000 fffffa60`00db7000   datascrn  Sat Apr 11 06:59:35 2009 (49E023B7)
fffffa60`00db7000 fffffa60`00df0000   msiscsi   Sat Apr 11 07:36:09 2009 (49E02C49)
fffffa60`00df0000 fffffa60`00dfe000   vga       Sat Jan 19 07:32:21 2008 (47919975)
fffffa60`00e00000 fffffa60`00e0c000   ndistapi  Sat Jan 19 07:37:22 2008 (47919AA2)
fffffa60`00e0d000 fffffa60`00e94000   ksecdd    Mon Jun 15 15:15:18 2009 (4A364966)
fffffa60`00e94000 fffffa60`00eaa000   i8042prt  Sat Jan 19 07:28:08 2008 (47919878)
fffffa60`00eaa000 fffffa60`00eb8000   kbdclass  Sat Jan 19 07:28:05 2008 (47919875)
fffffa60`00eb8000 fffffa60`00ec4000   mouclass  Sat Jan 19 07:28:05 2008 (47919875)
fffffa60`00ec4000 fffffa60`00ed1000   fdc       Sat Jan 19 07:28:45 2008 (4791989D)
fffffa60`00ed1000 fffffa60`00eed000   cdrom     Sat Apr 11 07:34:39 2009 (49E02BEF)
fffffa60`00eed000 fffffa60`00ef9000   usbuhci   Sat Jan 19 07:33:56 2008 (479199D4)
fffffa60`00ef9000 fffffa60`00f3f000   USBPORT   Sat Apr 11 07:39:39 2009 (49E02D1B)
fffffa60`00f3f000 fffffa60`00f4d000   vgapnp    Sat Jan 19 07:32:22 2008 (47919976)
fffffa60`00f4d000 fffffa60`00f72000   VIDEOPRT  Sat Jan 19 07:32:25 2008 (47919979)
fffffa60`00f72000 fffffa60`00f82000   watchdog  Sat Apr 11 07:09:16 2009 (49E025FC)
fffffa60`00f82000 fffffa60`00fa5a00   E1G6032E  Wed Sep 24 01:09:49 2008 (48D9773D)
fffffa60`00fa6000 fffffa60`00fb9000   intelppm  Sat Jan 19 06:52:45 2008 (4791902D)
fffffa60`00fb9000 fffffa60`00fcc000   blbdrive  Sat Jan 19 06:58:37 2008 (4791918D)
fffffa60`00fcc000 fffffa60`00fd9000   TDI       Sat Apr 11 07:44:14 2009 (49E02E2E)
fffffa60`00fd9000 fffffa60`00ffc000   rasl2tp   Sat Apr 11 07:43:37 2009 (49E02E09)
fffffa60`01003000 fffffa60`01179000   tcpip     Wed Jun 16 16:51:56 2010 (4C18E50C)
fffffa60`01179000 fffffa60`011a5000   fwpkclnt  Sat Apr 11 07:42:44 2009 (49E02DD4)
fffffa60`011a5000 fffffa60`011d1000   CLASSPNP  Sat Apr 11 07:34:15 2009 (49E02BD7)
fffffa60`011d1000 fffffa60`011db000   crcdisk   Sat Jan 19 07:30:12 2008 (479198F4)
fffffa60`011f3000 fffffa60`011fc000   tunmp     Sat Jan 19 07:36:30 2008 (47919A6E)
fffffa60`01209000 fffffa60`01389000   Ntfs      Sat Apr 11 06:55:38 2009 (49E022CA)
fffffa60`01389000 fffffa60`013cd000   volsnap   Sat Apr 11 07:35:33 2009 (49E02C25)
fffffa60`013cd000 fffffa60`013d5000   spldr     Mon Jan 12 22:51:29 2009 (496BBB61)
fffffa60`013d5000 fffffa60`013e7000   mup       Sat Apr 11 06:54:47 2009 (49E02297)
fffffa60`013e7000 fffffa60`013fb000   disk      Sat Apr 11 07:34:38 2009 (49E02BEE)
fffffa60`03600000 fffffa60`03631000   ndiswan   Sat Apr 11 07:43:39 2009 (49E02E0B)
fffffa60`03631000 fffffa60`03641000   raspppoe  Sat Apr 11 07:43:33 2009 (49E02E05)
fffffa60`03641000 fffffa60`0365f000   raspptp   Sat Apr 11 07:43:38 2009 (49E02E0A)
fffffa60`0365f000 fffffa60`03677000   rassstp   Sat Apr 11 07:43:46 2009 (49E02E12)
fffffa60`03677000 fffffa60`03711000   rdpdr     Sat Apr 11 07:49:54 2009 (49E02F82)
fffffa60`03711000 fffffa60`03724000   termdd    Sat Apr 11 07:48:13 2009 (49E02F1D)
fffffa60`03724000 fffffa60`03725480   swenum    Thu Nov 02 10:37:33 2006 (4549BC5D)
fffffa60`03726000 fffffa60`0375a000   ks        Sat Apr 11 07:33:51 2009 (49E02BBF)
fffffa60`0375a000 fffffa60`03765000   mssmbios  Sat Jan 19 07:02:54 2008 (4791928E)
fffffa60`03765000 fffffa60`03775000   umbus     Sat Jan 19 07:34:16 2008 (479199E8)
fffffa60`03775000 fffffa60`037bd000   usbhub    Sat Apr 11 07:39:52 2009 (49E02D28)
fffffa60`037bd000 fffffa60`037d1000   NDProxy   Sat Jan 19 07:37:26 2008 (47919AA6)
fffffa60`037d1000 fffffa60`037e1000   dfs       Sat Jan 19 06:54:14 2008 (47919086)
fffffa60`037e1000 fffffa60`037eb000   Fs_Rec    Sat Jan 19 06:53:41 2008 (47919065)
fffffa60`037eb000 fffffa60`037f4000   Null      Thu Nov 02 10:37:15 2006 (4549BC4B)
fffffa60`037f4000 fffffa60`037fd000   RDPCDD    Sat Jan 19 07:42:04 2008 (47919BBC)
fffffa60`037fd000 fffffa60`037fee00   USBD      Sat Jan 19 07:33:53 2008 (479199D1)
fffffa60`03c04000 fffffa60`03c1f000   smb       Sat Apr 11 07:42:19 2009 (49E02DBB)
fffffa60`03c1f000 fffffa60`03c8a000   afd       Sat Apr 11 07:44:21 2009 (49E02E35)
fffffa60`03c8a000 fffffa60`03cce000   netbt     Sat Apr 11 07:42:31 2009 (49E02DC7)
fffffa60`03cce000 fffffa60`03cec000   pacer     Sat Apr 11 07:42:56 2009 (49E02DE0)
fffffa60`03cec000 fffffa60`03cfb000   netbios   Sat Jan 19 07:36:35 2008 (47919A73)
fffffa60`03cfb000 fffffa60`03d0e000   monitor   Sat Jan 19 07:32:34 2008 (47919982)
fffffa60`03d18000 fffffa60`03d33000   wanarp    Sat Apr 11 07:43:38 2009 (49E02E0A)
fffffa60`03d33000 fffffa60`03d80000   rdbss     Sat Apr 11 06:55:21 2009 (49E022B9)
fffffa60`03d80000 fffffa60`03d8c000   nsiproxy  Sat Jan 19 07:36:45 2008 (47919A7D)
fffffa60`03d8c000 fffffa60`03da9000   dfsc      Sat Apr 11 06:54:44 2009 (49E02294)
fffffa60`03da9000 fffffa60`03dc5000   cdfs      Sat Jan 19 06:53:45 2008 (47919069)
fffffa60`03dc5000 fffffa60`03dd3000   crashdmp  Sat Apr 11 07:34:31 2009 (49E02BE7)
fffffa60`03dd3000 fffffa60`03ddd000   dump_diskdump  Sat Apr 11 07:34:30 2009
(49E02BE6)
fffffa60`03ddd000 fffffa60`03de6000   dump_viostor  Tue Aug 10 15:58:06 2010
(4C615AEE)
fffffa60`03de6000 fffffa60`03df2000   Dxapi     Sat Jan 19 07:08:00 2008 (479193C0)
fffffa60`04204000 fffffa60`04226000   luafv     Sat Jan 19 06:59:06 2008 (479191AA)
fffffa60`04226000 fffffa60`0423a000   lltdio    Sat Jan 19 07:35:48 2008 (47919A44)
fffffa60`0423a000 fffffa60`04252000   rspndr    Sat Jan 19 07:35:48 2008 (47919A44)
fffffa60`04252000 fffffa60`042ec000   spsys     Tue Mar 10 18:16:43 2009 (49B6A07B)
fffffa60`042ec000 fffffa60`0430a000   bowser    Sat Jan 19 06:54:51 2008 (479190AB)
fffffa60`0430a000 fffffa60`04324000   mpsdrv    Sat Jan 19 07:35:28 2008 (47919A30)
fffffa60`04324000 fffffa60`0434d000   mrxsmb    Tue Feb 23 12:32:22 2010 (4B83BCC6)
fffffa60`0434d000 fffffa60`04396000   mrxsmb10  Tue Feb 23 12:32:21 2010 (4B83BCC5)
fffffa60`04396000 fffffa60`043b5000   mrxsmb20  Tue Feb 23 12:32:20 2010 (4B83BCC4)
fffffa60`043b5000 fffffa60`043c0000   asyncmac  Sat Jan 19 07:37:27 2008 (47919AA7)
fffffa60`043c0000 fffffa60`043e9000   srvnet    Mon Sep 06 17:33:48 2010 (4C8509DC)
fffffa60`04806000 fffffa60`04838000   srv2      Mon Sep 06 17:33:50 2010 (4C8509DE)
fffffa60`04838000 fffffa60`048cc000   srv       Mon Sep 06 17:34:12 2010 (4C8509F4)
fffffa60`048cc000 fffffa60`0496f000   HTTP      Sat Feb 20 22:30:05 2010 (4B80545D)
fffffa60`0500e000 fffffa60`050c4000   peauth    Mon Oct 23 13:57:00 2006 (453CAE0C)
fffffa60`050c4000 fffffa60`050cf000   secdrv    Wed Sep 13 15:18:38 2006 (4508052E)
fffffa60`050cf000 fffffa60`050df000   tcpipreg  Tue Dec 08 18:55:32 2009 (4B1E9314)
fffffa60`050df000 fffffa60`050ec000   tdtcp     Sat Jan 19 07:42:03 2008 (47919BBB)
fffffa60`050ec000 fffffa60`050fa000   tssecsrv  Sat Jan 19 07:42:10 2008 (47919BC2)
fffffa60`050fa000 fffffa60`05136000   RDPWD     Sat Apr 11 07:48:33 2009 (49E02F31)

Unloaded modules:
fffffa60`011db000 fffffa60`011e9000   crashdmp.sys
    Timestamp: unavailable (00000000)
    Checksum:  00000000
    ImageSize:  0000E000
fffffa60`011e9000 fffffa60`011f3000   dump_storpor
    Timestamp: unavailable (00000000)
    Checksum:  00000000
    ImageSize:  0000A000
fffffa60`01200000 fffffa60`01209000   dump_viostor
    Timestamp: unavailable (00000000)
    Checksum:  00000000
    ImageSize:  00009000
fffffa60`03cfb000 fffffa60`03d18000   serial.sys
    Timestamp: unavailable (00000000)
    Checksum:  00000000
    ImageSize:  0001D000
fffffa60`0076c000 fffffa60`00788000   sacdrv.sys
    Timestamp: unavailable (00000000)
    Checksum:  00000000
    ImageSize:  0001C000
**************************************************************************
*                                                                        *
*                   Bugcheck Analysis                                    *
*                                                                        *
**************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck 3B, {c0000005, fffff80001e6a842, fffffa6004efcb70, 0}

Probably caused by : Npfs.SYS ( Npfs!NpSetAttributeInList+9a )

Followup: MachineOwner
---------

Finished dump check



best regards
Manfred

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: AW: Freezing Windows 2008 x64bit guest
  2010-12-14 23:57                     ` AW: " Manfred Heubach
@ 2010-12-15 10:48                       ` Vadim Rozenfeld
  0 siblings, 0 replies; 22+ messages in thread
From: Vadim Rozenfeld @ 2010-12-15 10:48 UTC (permalink / raw)
  To: Manfred Heubach; +Cc: dlaor@redhat.com, kvm@vger.kernel.org

On Wed, 2010-12-15 at 00:57 +0100, Manfred Heubach wrote:
> Vadim Rozenfeld <vrozenfe <at> redhat.com> writes:
> 
> >
> > On Mon, 2010-12-13 at 22:12 +0200, Dor Laor wrote:
> > > On 12/13/2010 09:42 PM, Manfred Heubach wrote:
> > > >
> > > > I was running the host with Ubuntu 10.04 but upgraded to 10.10 - mainly
> because
> > > > of performance problems which were solved by the upgrade.
> > > >
> > > > After the upgrade the system became extremly unstable. It was crashing as soon
> > > > as disk io and network io load was growing. 100% reproduceable with windows
> > > > server backup to an iscsi volume.
> > > >
> > > > i had virtio drivers for storage and network installed (redhat/fedora 1.1.11).
> > >
> > > Which fedora/rhel release is that?
> 
> 
> The host is Ubuntu 10.10 x64
> 
> The drivers are from
> http://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/ 1.1.11-0
> released on 17-Aug-2010 - are there any newer drivers?
Yes, they are the most recent drivers at fedoraproject. Our rhel6
drivers are almost the same, except for a few non-critical, WHQL related
bug fixes.
However, there is one fix related to Large Send Offload (LSO) problem
which was fixed in rhel6, but I don't see the relevant changes in fedora
kvmnet driver sources
http://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/src/
 
> 
> 
> > > What's the windows virtio driver version?
> 
> The virtio storage version shown in Windows is 6.0.0.10
> 
> > >
> > > Have you tried using virt-manager/virhs instead of raw cmdline?
> 
> I'm starting it with libvirt/virsh
> 
> cmd-line copied from the log (and some log entries):
> 
> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
> QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.12 -enable-kvm -m 8192 -smp
> 4,sockets=4,cores=1,threads=1 -name sbs2008 -uuid
> 933c2ef2-e5b0-0b39-db60-016b5d226534 -nodefaults -chardev
> socket,id=monitor,path=/var/lib/libvirt/qemu/sbs2008.monitor,server,nowait -mon
> chardev=monitor,mode=readline -rtc base=localtime -boot c -drive
> file=/var/lib/libvirt/images/olscanner/virtio-win-1.1.11-0.iso,if=none,media=cdrom,
> id=drive-ide0-1-0,readonly=on,format=raw
> -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive
> file=/var/lib/libvirt/images/sbs2008/sbs2008.img,if=none,id=drive-virtio-disk0,
> boot=on,format=qcow2 -device
> virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
> -drive
> file=/dev/volg1/sbsdata,if=none,id=drive-virtio-disk1,format=raw,cache=none
> -device
> virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1
> -drive file=/dev/volg1/wsus,if=none,id=drive-virtio-disk2,format=raw,cache=none
> -device
> virtio-blk-pci,bus=pci.0,addr=0x7,drive=drive-virtio-disk2,id=virtio-disk2
> -device e1000,vlan=0,id=net0,mac=52:54:00:8a:bc:c9,bus=pci.0,addr=0x6 -net
> tap,fd=107,vlan=0,name=hostnet0 -chardev pty,id=serial0 -device
> isa-serial,chardev=serial0 -usb -device usb-tablet,id=input0 -vnc 0.0.0.0:0 -k
> de -vga std -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3
> 07:12:02.715: debug : qemudInitCpuAffinity:2423 : Setting CPU affinity
> 07:12:02.717: debug : qemuSecurityDACSetProcessLabel:547 : Dropping privileges
> of VM to 105:114
> char device redirected to /dev/pts/0
> pci_add_option_rom: failed to find romfile "pxe-e1000.bin"
> 
> 
> > > About e1000, some windows comes with buggy driver and an update e1000
> > > from Intel fixes some issues.
> > >
> 
> I'm running latest drivers from Intel: 8.3.15.0
> 
> > >
> > > > At each BSOD I had the following line in the log of the guest:
> > > >
> > > >   virtio_ioport_write: unexpected address 0x13 value 0x1
Seems to be the result of calling PapaNdis_OnBugCheck function

> > > >
> > > > I changed the network interface back to e1000. What I experience now (and
> I had
> > > > that a the very beginning before i switched to virtio network) are
> freezes. The
> > > > guest doesn't respond anymore (doesn't answer to pings and doesn't
> interact via
> > > > mouse/keyboard anymore). Host CPU usage of the kvm process is 100% on as many
> > > > cores as there are virtual cpus (in this case 4).
> 
> I had a crash today but no logentry on the host - but that could be because I
> had to restart syslog (ran out of diskspace after turning on debug logging ob
> libvirtd - didn't think that it would generate 6 GB of logs per day :-)
> 
> > > >
> > Sounds like an interrupt storm to me. Can you try to ping your VM?
> 
> No responds to ping.
> 
> > Anyway the best way to start debugging a stalled system is just to crash
> > it with BSOD. For doing it you will need:
> > - enable NMICrashDump (please see http://support.microsoft.com/kb/927069
> > for more information
> > - enable Kernel Memory Dump (actually Complete is much better, but it
> > can be too big)  http://support.microsoft.com/kb/969028
> > - you only will need to type "nmi 0" in the qemu monitor to crash the
> > system, when the system hangs next time.
> 
> I prepared this. When the system crashed today I didn't have the complete
> memory dump ready - so I only have a minidump. The intersting point is that
> the system today crashed with a BSOD and didn't freeze.
> 
> The result of dumpchk.exe is as follows:
> 
> 
> Microsoft (R) Windows Debugger Version 6.12.0002.633 AMD64
> Copyright (c) Microsoft Corporation. All rights reserved.
> 
> 
> Loading Dump File [c:\Windows\Minidump\Mini121410-01.dmp]
> Mini Kernel Dump File: Only registers and stack trace are available
> 
> Symbol search path is: SRV*http://msdl.microsoft.com/download/symbols
> Executable search path is:
> Windows Server 2008/Windows Vista Kernel Version 6002 (Service Pack 2) MP (4
> procs) Free x64
> Product: LanManNt, suite: SmallBusiness TerminalServer SmallBusinessRestricted
> SingleUserTS
> Built by: 6002.18267.amd64fre.vistasp2_gdr.100608-0458
> Machine Name:
> Kernel base = 0xfffff800`01e16000 PsLoadedModuleList = 0xfffff800`01fdadd0
> Debug session time: Tue Dec 14 18:41:20.058 2010 (UTC + 1:00)
> System Uptime: 1 days 11:16:05.967
> Loading Kernel Symbols
> ...............................................................
> ............................................................
> Loading User Symbols
> Loading unloaded module list
> .....
> *******************************************************************************
> *                                                                             *
> *                        Bugcheck Analysis                                    *
> *                                                                             *
> *******************************************************************************
> 
> Use !analyze -v to get detailed debugging information.
> 
> BugCheck 3B, {c0000005, fffff80001e6a842, fffffa6004efcb70, 0}
> 
> Probably caused by : Npfs.SYS ( Npfs!NpSetAttributeInList+9a )
It is named pipe file system. 
> 
> Followup: MachineOwner
> ---------
> 
> ----- 64 bit Kernel Mini Dump Analysis
> 
> DUMP_HEADER64:
> MajorVersion        0000000f
> MinorVersion        00001772
> KdSecondaryVersion  00000000
> DirectoryTableBase  00000001`e5b51000
> PfnDataBase         fffff800`0203d250
> PsLoadedModuleList  fffff800`01fdadd0
> PsActiveProcessHead fffff800`01fba440
> MachineImageType    00008664
> NumberProcessors    00000004
> BugCheckCode        0000003b
> BugCheckParameter1  00000000`c0000005
> BugCheckParameter2  fffff800`01e6a842
> BugCheckParameter3  fffffa60`04efcb70
> BugCheckParameter4  00000000`00000000
> KdDebuggerDataBlock fffff800`01f87f20
> ProductType         00000002
> SuiteMask           00000131
> WriterStatus        00000000
> MiniDumpFields      00000cff
> 
> TRIAGE_DUMP64:
> ServicePackBuild      00000200
> SizeOfDump            00040000
> ValidOffset           0003fffc
> ContextOffset         00000348
> ExceptionOffset       00000f00
> MmOffset              00002080
> UnloadedDriversOffset 000020d0
> PrcbOffset            000021f0
> ProcessOffset         00005d10
> ThreadOffset          000060f8
> CallStackOffset       00006548
> SizeOfCallStack       00001bf8
> DriverListOffset      00008470
> DriverCount           0000007c
> StringPoolOffset      0000ca30
> StringPoolSize        00002950
> BrokenDriverOffset    00000000
> TriageOptions         ffffffff
> TopOfStack            fffffa60`04efc1b8
> BStoreOffset          00000000
> SizeOfBStore          00000000
> LimitOfBStore         00000000`00000000
> DebuggerDataOffset    00008140
> DebuggerDataSize      00000330
> DataBlocksOffset      0000f380
> DataBlocksCount       00000008
>   fffff800`01fdb720 - fffff800`01fdb79f at offset 0000f400
>   fffff800`01fbf898 - fffff800`01fbf89b at offset 0000f480
>   fffff800`01fbf894 - fffff800`01fbf897 at offset 0000f484
>   fffff800`0203d078 - fffff800`0203d07b at offset 0000f488
>   fffff800`0203d09c - fffff800`0203d09f at offset 0000f48c
>   fffff800`01e6f000 - fffff800`01e6ffff at offset 0000f490
>   fffff800`02019000 - fffff800`02019fff at offset 00010490
>   fffff800`01e70000 - fffff800`01e70fff at offset 00011490
>   Max offset 12490, 2e2f8 from end of file
> 
> 
> Windows Server 2008/Windows Vista Kernel Version 6002 (Service Pack 2) MP (4
> procs) Free x64
> Product: LanManNt, suite: SmallBusiness TerminalServer SmallBusinessRestricted
> SingleUserTS
> Built by: 6002.18267.amd64fre.vistasp2_gdr.100608-0458
> Machine Name:
> Kernel base = 0xfffff800`01e16000 PsLoadedModuleList = 0xfffff800`01fdadd0
> Debug session time: Tue Dec 14 18:41:20.058 2010 (UTC + 1:00)
> System Uptime: 1 days 11:16:05.967
> start             end                 module name
> fffff800`01e16000 fffff800`0232d000   nt        Tue Jun 08 16:59:47 2010 (4C0E5AE3)
> fffff800`0232d000 fffff800`02373000   hal       Sat Apr 11 09:04:56 2009 (49E04118)
> fffff960`000b0000 fffff960`00364000   win32k    Tue Aug 31 16:57:22 2010 (4C7D1852)
> fffff960`004d0000 fffff960`004ee000   dxg       Sat Apr 11 07:09:45 2009 (49E02619)
> fffff960`00680000 fffff960`0068a000   TSDDD     Sat Jan 19 07:42:04 2008 (47919BBC)
> fffff960`008c0000 fffff960`008c9000   framebuf  unavailable (00000000)
> fffff960`00a80000 fffff960`00ab0000   RDPDD     unavailable (00000000)
> fffffa60`00604000 fffffa60`0060e000   kdcom     Sat Apr 11 09:07:57 2009 (49E041CD)
> fffffa60`0060e000 fffffa60`00649000   mcupdate_GenuineIntel  Sat Apr 11
> 09:05:41 2009 (49E04145)
> fffffa60`00649000 fffffa60`0065d000   PSHED     Sat Apr 11 09:08:17 2009 (49E041E1)
> fffffa60`0065d000 fffffa60`006ba000   CLFS      Sat Apr 11 06:54:21 2009 (49E0227D)
> fffffa60`006ba000 fffffa60`0076c000   CI        Sat Apr 11 09:08:09 2009 (49E041D9)
> fffffa60`0076c000 fffffa60`0077e000   HIDCLASS  Sat Apr 11 07:39:32 2009 (49E02D14)
> fffffa60`0077e000 fffffa60`00785b80   HIDPARSE  Sat Jan 19 07:33:51 2008 (479199CF)
> fffffa60`00788000 fffffa60`007d8000   msrpc     Sat Apr 11 07:32:13 2009 (49E02B5D)
> fffffa60`007d8000 fffffa60`007f5000   tdx       Sat Apr 11 07:43:00 2009 (49E02DE4)
> fffffa60`007f5000 fffffa60`00800000   mouhid    Sat Jan 19 07:28:10 2008 (4791987A)
> fffffa60`00800000 fffffa60`00809000   hidusb    Sat Apr 11 07:39:32 2009 (49E02D14)
> fffffa60`0080a000 fffffa60`009cd000   NDIS      Sat Apr 11 07:43:15 2009 (49E02DF3)
> fffffa60`009cd000 fffffa60`009e1000   volmgr    Sat Apr 11 07:34:49 2009 (49E02BF9)
> fffffa60`009e1000 fffffa60`009f2000   Npfs      Sat Apr 11 06:54:22 2009 (49E0227E)
> fffffa60`009f2000 fffffa60`009fb000   rasacd    Sat Jan 19 07:37:30 2008 (47919AAA)
> fffffa60`00a02000 fffffa60`00a5b000   NETIO     Sat Apr 11 07:43:34 2009 (49E02E06)
> fffffa60`00a5b000 fffffa60`00b35000   Wdf01000  Sat Jan 19 07:33:27 2008 (479199B7)
> fffffa60`00b35000 fffffa60`00b43000   WDFLDR    Sat Jan 19 07:32:33 2008 (47919981)
> fffffa60`00b43000 fffffa60`00b99000   acpi      Sat Apr 11 07:03:26 2009 (49E0249E)
> fffffa60`00b99000 fffffa60`00ba2000   WMILIB    Sat Jan 19 07:33:45 2008 (479199C9)
> fffffa60`00ba2000 fffffa60`00bac000   msisadrv  Sat Jan 19 07:02:50 2008 (4791928A)
> fffffa60`00bac000 fffffa60`00bdc000   pci       Sat Apr 11 07:03:33 2009 (49E024A5)
> fffffa60`00bdc000 fffffa60`00bf1000   partmgr   Sat Apr 11 07:34:51 2009 (49E02BFB)
> fffffa60`00bf1000 fffffa60`00bfc000   Msfs      Sat Jan 19 06:53:55 2008 (47919073)
> fffffa60`00c00000 fffffa60`00c09000   rdpencdd  Sat Jan 19 07:42:03 2008 (47919BBB)
> fffffa60`00c0d000 fffffa60`00c73000   volmgrx   Sat Apr 11 07:35:14 2009 (49E02C12)
> fffffa60`00c73000 fffffa60`00c7b000   intelide  Sat Jan 19 07:28:56 2008 (479198A8)
> fffffa60`00c7b000 fffffa60`00c8b000   PCIIDEX   Sat Apr 11 07:34:22 2009 (49E02BDE)
> fffffa60`00c8b000 fffffa60`00c9e000   mountmgr  Sat Jan 19 07:28:01 2008 (47919871)
> fffffa60`00c9e000 fffffa60`00ca6000   atapi     Sat Apr 11 07:34:19 2009 (49E02BDB)
> fffffa60`00ca6000 fffffa60`00cca000   ataport   Sat Apr 11 07:34:25 2009 (49E02BE1)
> fffffa60`00cca000 fffffa60`00cd3000   viostor   Tue Aug 10 15:58:06 2010 (4C615AEE)
> fffffa60`00cd3000 fffffa60`00d30000   storport  Sat Apr 11 07:34:45 2009 (49E02BF5)
> fffffa60`00d30000 fffffa60`00d77000   fltmgr    Sat Apr 11 06:54:38 2009 (49E0228E)
> fffffa60`00d77000 fffffa60`00da2000   quota     Sat Apr 11 06:59:38 2009 (49E023BA)
> fffffa60`00da2000 fffffa60`00db7000   datascrn  Sat Apr 11 06:59:35 2009 (49E023B7)
> fffffa60`00db7000 fffffa60`00df0000   msiscsi   Sat Apr 11 07:36:09 2009 (49E02C49)
> fffffa60`00df0000 fffffa60`00dfe000   vga       Sat Jan 19 07:32:21 2008 (47919975)
> fffffa60`00e00000 fffffa60`00e0c000   ndistapi  Sat Jan 19 07:37:22 2008 (47919AA2)
> fffffa60`00e0d000 fffffa60`00e94000   ksecdd    Mon Jun 15 15:15:18 2009 (4A364966)
> fffffa60`00e94000 fffffa60`00eaa000   i8042prt  Sat Jan 19 07:28:08 2008 (47919878)
> fffffa60`00eaa000 fffffa60`00eb8000   kbdclass  Sat Jan 19 07:28:05 2008 (47919875)
> fffffa60`00eb8000 fffffa60`00ec4000   mouclass  Sat Jan 19 07:28:05 2008 (47919875)
> fffffa60`00ec4000 fffffa60`00ed1000   fdc       Sat Jan 19 07:28:45 2008 (4791989D)
> fffffa60`00ed1000 fffffa60`00eed000   cdrom     Sat Apr 11 07:34:39 2009 (49E02BEF)
> fffffa60`00eed000 fffffa60`00ef9000   usbuhci   Sat Jan 19 07:33:56 2008 (479199D4)
> fffffa60`00ef9000 fffffa60`00f3f000   USBPORT   Sat Apr 11 07:39:39 2009 (49E02D1B)
> fffffa60`00f3f000 fffffa60`00f4d000   vgapnp    Sat Jan 19 07:32:22 2008 (47919976)
> fffffa60`00f4d000 fffffa60`00f72000   VIDEOPRT  Sat Jan 19 07:32:25 2008 (47919979)
> fffffa60`00f72000 fffffa60`00f82000   watchdog  Sat Apr 11 07:09:16 2009 (49E025FC)
> fffffa60`00f82000 fffffa60`00fa5a00   E1G6032E  Wed Sep 24 01:09:49 2008 (48D9773D)
> fffffa60`00fa6000 fffffa60`00fb9000   intelppm  Sat Jan 19 06:52:45 2008 (4791902D)
> fffffa60`00fb9000 fffffa60`00fcc000   blbdrive  Sat Jan 19 06:58:37 2008 (4791918D)
> fffffa60`00fcc000 fffffa60`00fd9000   TDI       Sat Apr 11 07:44:14 2009 (49E02E2E)
> fffffa60`00fd9000 fffffa60`00ffc000   rasl2tp   Sat Apr 11 07:43:37 2009 (49E02E09)
> fffffa60`01003000 fffffa60`01179000   tcpip     Wed Jun 16 16:51:56 2010 (4C18E50C)
> fffffa60`01179000 fffffa60`011a5000   fwpkclnt  Sat Apr 11 07:42:44 2009 (49E02DD4)
> fffffa60`011a5000 fffffa60`011d1000   CLASSPNP  Sat Apr 11 07:34:15 2009 (49E02BD7)
> fffffa60`011d1000 fffffa60`011db000   crcdisk   Sat Jan 19 07:30:12 2008 (479198F4)
> fffffa60`011f3000 fffffa60`011fc000   tunmp     Sat Jan 19 07:36:30 2008 (47919A6E)
> fffffa60`01209000 fffffa60`01389000   Ntfs      Sat Apr 11 06:55:38 2009 (49E022CA)
> fffffa60`01389000 fffffa60`013cd000   volsnap   Sat Apr 11 07:35:33 2009 (49E02C25)
> fffffa60`013cd000 fffffa60`013d5000   spldr     Mon Jan 12 22:51:29 2009 (496BBB61)
> fffffa60`013d5000 fffffa60`013e7000   mup       Sat Apr 11 06:54:47 2009 (49E02297)
> fffffa60`013e7000 fffffa60`013fb000   disk      Sat Apr 11 07:34:38 2009 (49E02BEE)
> fffffa60`03600000 fffffa60`03631000   ndiswan   Sat Apr 11 07:43:39 2009 (49E02E0B)
> fffffa60`03631000 fffffa60`03641000   raspppoe  Sat Apr 11 07:43:33 2009 (49E02E05)
> fffffa60`03641000 fffffa60`0365f000   raspptp   Sat Apr 11 07:43:38 2009 (49E02E0A)
> fffffa60`0365f000 fffffa60`03677000   rassstp   Sat Apr 11 07:43:46 2009 (49E02E12)
> fffffa60`03677000 fffffa60`03711000   rdpdr     Sat Apr 11 07:49:54 2009 (49E02F82)
> fffffa60`03711000 fffffa60`03724000   termdd    Sat Apr 11 07:48:13 2009 (49E02F1D)
> fffffa60`03724000 fffffa60`03725480   swenum    Thu Nov 02 10:37:33 2006 (4549BC5D)
> fffffa60`03726000 fffffa60`0375a000   ks        Sat Apr 11 07:33:51 2009 (49E02BBF)
> fffffa60`0375a000 fffffa60`03765000   mssmbios  Sat Jan 19 07:02:54 2008 (4791928E)
> fffffa60`03765000 fffffa60`03775000   umbus     Sat Jan 19 07:34:16 2008 (479199E8)
> fffffa60`03775000 fffffa60`037bd000   usbhub    Sat Apr 11 07:39:52 2009 (49E02D28)
> fffffa60`037bd000 fffffa60`037d1000   NDProxy   Sat Jan 19 07:37:26 2008 (47919AA6)
> fffffa60`037d1000 fffffa60`037e1000   dfs       Sat Jan 19 06:54:14 2008 (47919086)
> fffffa60`037e1000 fffffa60`037eb000   Fs_Rec    Sat Jan 19 06:53:41 2008 (47919065)
> fffffa60`037eb000 fffffa60`037f4000   Null      Thu Nov 02 10:37:15 2006 (4549BC4B)
> fffffa60`037f4000 fffffa60`037fd000   RDPCDD    Sat Jan 19 07:42:04 2008 (47919BBC)
> fffffa60`037fd000 fffffa60`037fee00   USBD      Sat Jan 19 07:33:53 2008 (479199D1)
> fffffa60`03c04000 fffffa60`03c1f000   smb       Sat Apr 11 07:42:19 2009 (49E02DBB)
> fffffa60`03c1f000 fffffa60`03c8a000   afd       Sat Apr 11 07:44:21 2009 (49E02E35)
> fffffa60`03c8a000 fffffa60`03cce000   netbt     Sat Apr 11 07:42:31 2009 (49E02DC7)
> fffffa60`03cce000 fffffa60`03cec000   pacer     Sat Apr 11 07:42:56 2009 (49E02DE0)
> fffffa60`03cec000 fffffa60`03cfb000   netbios   Sat Jan 19 07:36:35 2008 (47919A73)
> fffffa60`03cfb000 fffffa60`03d0e000   monitor   Sat Jan 19 07:32:34 2008 (47919982)
> fffffa60`03d18000 fffffa60`03d33000   wanarp    Sat Apr 11 07:43:38 2009 (49E02E0A)
> fffffa60`03d33000 fffffa60`03d80000   rdbss     Sat Apr 11 06:55:21 2009 (49E022B9)
> fffffa60`03d80000 fffffa60`03d8c000   nsiproxy  Sat Jan 19 07:36:45 2008 (47919A7D)
> fffffa60`03d8c000 fffffa60`03da9000   dfsc      Sat Apr 11 06:54:44 2009 (49E02294)
> fffffa60`03da9000 fffffa60`03dc5000   cdfs      Sat Jan 19 06:53:45 2008 (47919069)
> fffffa60`03dc5000 fffffa60`03dd3000   crashdmp  Sat Apr 11 07:34:31 2009 (49E02BE7)
> fffffa60`03dd3000 fffffa60`03ddd000   dump_diskdump  Sat Apr 11 07:34:30 2009
> (49E02BE6)
> fffffa60`03ddd000 fffffa60`03de6000   dump_viostor  Tue Aug 10 15:58:06 2010
> (4C615AEE)
> fffffa60`03de6000 fffffa60`03df2000   Dxapi     Sat Jan 19 07:08:00 2008 (479193C0)
> fffffa60`04204000 fffffa60`04226000   luafv     Sat Jan 19 06:59:06 2008 (479191AA)
> fffffa60`04226000 fffffa60`0423a000   lltdio    Sat Jan 19 07:35:48 2008 (47919A44)
> fffffa60`0423a000 fffffa60`04252000   rspndr    Sat Jan 19 07:35:48 2008 (47919A44)
> fffffa60`04252000 fffffa60`042ec000   spsys     Tue Mar 10 18:16:43 2009 (49B6A07B)
> fffffa60`042ec000 fffffa60`0430a000   bowser    Sat Jan 19 06:54:51 2008 (479190AB)
> fffffa60`0430a000 fffffa60`04324000   mpsdrv    Sat Jan 19 07:35:28 2008 (47919A30)
> fffffa60`04324000 fffffa60`0434d000   mrxsmb    Tue Feb 23 12:32:22 2010 (4B83BCC6)
> fffffa60`0434d000 fffffa60`04396000   mrxsmb10  Tue Feb 23 12:32:21 2010 (4B83BCC5)
> fffffa60`04396000 fffffa60`043b5000   mrxsmb20  Tue Feb 23 12:32:20 2010 (4B83BCC4)
> fffffa60`043b5000 fffffa60`043c0000   asyncmac  Sat Jan 19 07:37:27 2008 (47919AA7)
> fffffa60`043c0000 fffffa60`043e9000   srvnet    Mon Sep 06 17:33:48 2010 (4C8509DC)
> fffffa60`04806000 fffffa60`04838000   srv2      Mon Sep 06 17:33:50 2010 (4C8509DE)
> fffffa60`04838000 fffffa60`048cc000   srv       Mon Sep 06 17:34:12 2010 (4C8509F4)
> fffffa60`048cc000 fffffa60`0496f000   HTTP      Sat Feb 20 22:30:05 2010 (4B80545D)
> fffffa60`0500e000 fffffa60`050c4000   peauth    Mon Oct 23 13:57:00 2006 (453CAE0C)
> fffffa60`050c4000 fffffa60`050cf000   secdrv    Wed Sep 13 15:18:38 2006 (4508052E)
> fffffa60`050cf000 fffffa60`050df000   tcpipreg  Tue Dec 08 18:55:32 2009 (4B1E9314)
> fffffa60`050df000 fffffa60`050ec000   tdtcp     Sat Jan 19 07:42:03 2008 (47919BBB)
> fffffa60`050ec000 fffffa60`050fa000   tssecsrv  Sat Jan 19 07:42:10 2008 (47919BC2)
> fffffa60`050fa000 fffffa60`05136000   RDPWD     Sat Apr 11 07:48:33 2009 (49E02F31)
> 
> Unloaded modules:
> fffffa60`011db000 fffffa60`011e9000   crashdmp.sys
>     Timestamp: unavailable (00000000)
>     Checksum:  00000000
>     ImageSize:  0000E000
> fffffa60`011e9000 fffffa60`011f3000   dump_storpor
>     Timestamp: unavailable (00000000)
>     Checksum:  00000000
>     ImageSize:  0000A000
> fffffa60`01200000 fffffa60`01209000   dump_viostor
>     Timestamp: unavailable (00000000)
>     Checksum:  00000000
>     ImageSize:  00009000
> fffffa60`03cfb000 fffffa60`03d18000   serial.sys
>     Timestamp: unavailable (00000000)
>     Checksum:  00000000
>     ImageSize:  0001D000
> fffffa60`0076c000 fffffa60`00788000   sacdrv.sys
>     Timestamp: unavailable (00000000)
>     Checksum:  00000000
>     ImageSize:  0001C000
> **************************************************************************
> *                                                                        *
> *                   Bugcheck Analysis                                    *
> *                                                                        *
> **************************************************************************
> 
> Use !analyze -v to get detailed debugging information.
> 
> BugCheck 3B, {c0000005, fffff80001e6a842, fffffa6004efcb70, 0}
> 
> Probably caused by : Npfs.SYS ( Npfs!NpSetAttributeInList+9a )
> 
> Followup: MachineOwner
> ---------
> 
> Finished dump check
> 
> 
> 
> best regards
> Manfred
> NrybXǧv^)޺{.n+h\x17ܨ}Ơz&j:+v\azZ++zfh~iz\x1ew?&)ߢ^[f



^ permalink raw reply	[flat|nested] 22+ messages in thread

end of thread, other threads:[~2010-12-15 10:49 UTC | newest]

Thread overview: 22+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2010-07-15 13:19 Freezing Windows 2008 x64bit guest Christoph Adomeit
2010-07-15 13:44 ` Gleb Natapov
2010-07-19  7:17   ` Harri Olin
2010-07-19  7:42     ` Gleb Natapov
2010-07-21  6:25       ` Harri Olin
2010-07-21  8:28         ` Christoph Adomeit
2010-07-21  8:37         ` Gleb Natapov
2010-07-21  9:22           ` Harri Olin
2010-07-21  9:48             ` Gleb Natapov
2010-07-21 10:09               ` Harri Olin
2010-07-21 10:30                 ` Gleb Natapov
2010-07-21 10:43                   ` Harri Olin
2010-07-21 13:03                   ` Harri Olin
2010-07-21 13:45                     ` Gleb Natapov
2010-07-21 14:05                       ` Harri Olin
2010-07-27 21:53           ` Harri Olin
2010-07-28 15:18             ` Gleb Natapov
2010-12-13 19:42               ` Manfred Heubach
2010-12-13 20:12                 ` Dor Laor
2010-12-13 20:44                   ` Vadim Rozenfeld
2010-12-14 23:57                     ` AW: " Manfred Heubach
2010-12-15 10:48                       ` Vadim Rozenfeld

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox