From mboxrd@z Thu Jan  1 00:00:00 1970
From: Harri Olin <harri.olin@gmail.com>
Subject: Re: Freezing Windows 2008 x64bit guest
Date: Wed, 21 Jul 2010 09:25:31 +0300
Message-ID: <4C4692DB.1000703@gmail.com>
References: <20100715131944.GA24978@mp3.niederrhein.de> <20100715134441.GQ4689@redhat.com> <4C43FBEE.6000409@gmail.com> <20100719074207.GA4689@redhat.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="ISO-8859-1"; format=flowed
Content-Transfer-Encoding: 7bit
Cc: Christoph Adomeit <Christoph.Adomeit@gatworks.de>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>
To: Gleb Natapov <gleb@redhat.com>
Return-path: <kvm-owner@vger.kernel.org>
Received: from emh07.mail.saunalahti.fi ([62.142.5.117]:58462 "EHLO
	emh07.mail.saunalahti.fi" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
	with ESMTP id S1753591Ab0GUGZQ (ORCPT <rfc822;kvm@vger.kernel.org>);
	Wed, 21 Jul 2010 02:25:16 -0400
In-Reply-To: <20100719074207.GA4689@redhat.com>
Sender: kvm-owner@vger.kernel.org
List-ID: <kvm.vger.kernel.org>

Gleb Natapov kirjoitti:
> On Mon, Jul 19, 2010 at 10:17:02AM +0300, Harri Olin wrote:
>> Gleb Natapov kirjoitti:
>>> On Thu, Jul 15, 2010 at 03:19:44PM +0200, Christoph Adomeit wrote:
>>>> But one Windows 2008 64 Bit Server Standard is freezing regularly.
>>>> This happens sometimes 3 times a day, sometimes it takes 2 days
>>>> until freeze. The Windows Machine is a clean fresh install.
>> I think I have seen same problem occur on my Windows 2008 SBS SP2
>> 64bit system, but a bit less often, only like once a week.
>> Now I haven't seen crashes but only freezes with qemu on 100% and
>> virtual system unresponsive.
>>
>> qemu command line:
>>   /usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive
>> file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive
>> file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net
>> nic,macaddr=C0:FF:12:FB:AA:01,model=e1000 -net tap -smp 4 -localtime
>>
>>
> Try with different model then e1000 please. Default driver that comes
> with Windows known to have problems.

Didn't help, changed to default realtek emulation, but system freezed 
again. Command line:
/usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive 
file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive 
file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net 
nic,macaddr=C0:FF:12:FB:AA:01 -net tap -smp 4 -localtime

This time the virtual system was not totally unresponsive somehow, 
system pinged just fine and I think DNS server worked too. However on 
console mouse moved but didn't react to clicking, ctrl-alt-del, etc.

> 
>>> When hang occurs ensure that problematic vm is the only on on the server and run
>>> kvm_stat. Send output here. Also do the following:
>>>
>>> # mount -t debugfs debugfs /sys/kernel/debug
>>> # echo kvm > /sys/kernel/debug/tracing/set_event
>>> # sleep 1
>>> # cat /sys/kernel/debug/tracing/trace > /tmp/trace
>>>
>>> Send /tmp/trace here too, but it may be huge, so send only last 1000
>>> lines.

Stats from this hang:
log style http://mizar.remote.agasha.com/k/kvm/kvm_stat_2_log.txt

kvm statistics

  efer_reload                  0       0
  exits               5368720296   21258
  fpu_reload          2826750721    2216
  halt_exits           695286653       0
  halt_wakeup          685187661       0
  host_state_reload   3010399640    2216
  hypercalls                   0       0
  insn_emulation      2765352032    6542
  insn_emulation_fail        477       0
  invlpg                42417110     204
  io_exits             818764026     122
  irq_exits            109659572    9870
  irq_injections       788992032    4643
  irq_window            22170277    1718
  largepages                   0       0
  mmio_exits          1490912756      69
  mmu_cache_miss         2471963       0
  mmu_flooded             905690       0
  mmu_pde_zapped         1821325       0
  mmu_pte_updated        3256583       0
  mmu_pte_write          2896199       0
  mmu_recycled            259845       0


full trace: http://mizar.remote.agasha.com/k/kvm/kvm_trace_2_full.txt

  qemu-system-x86-5155  [002] 1248198.487147: kvm_entry: vcpu 3
  qemu-system-x86-5152  [000] 1248198.487147: kvm_inj_virq: irq 209
  qemu-system-x86-5152  [000] 1248198.487148: kvm_entry: vcpu 0
  qemu-system-x86-5154  [001] 1248198.487148: kvm_exit: reason 
apic_access rip 0xfffff80001c9df13
  qemu-system-x86-5155  [002] 1248198.487148: kvm_exit: reason 
apic_access rip 0xfffff80001c9df13
  qemu-system-x86-5152  [000] 1248198.487150: kvm_exit: reason 
apic_access rip 0xfffff80001c9d5c4
  qemu-system-x86-5154  [001] 1248198.487150: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5154  [001] 1248198.487150: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5155  [002] 1248198.487150: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5155  [002] 1248198.487151: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5154  [001] 1248198.487151: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5155  [002] 1248198.487151: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5152  [000] 1248198.487151: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5152  [000] 1248198.487152: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5154  [001] 1248198.487152: kvm_entry: vcpu 2
  qemu-system-x86-5155  [002] 1248198.487152: kvm_entry: vcpu 3
  qemu-system-x86-5152  [000] 1248198.487152: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5153  [003] 1248198.487153: kvm_exit: reason ext_irq 
rip 0xfffff80001d04a00
  qemu-system-x86-5152  [000] 1248198.487153: kvm_entry: vcpu 0
  qemu-system-x86-5153  [003] 1248198.487154: kvm_inj_virq: irq 209
  qemu-system-x86-5153  [003] 1248198.487155: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.487157: kvm_exit: reason 
apic_access rip 0xfffff80001c9df13
  qemu-system-x86-5153  [003] 1248198.487159: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5153  [003] 1248198.487159: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5153  [003] 1248198.487160: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5153  [003] 1248198.487161: kvm_entry: vcpu 1
  qemu-system-x86-5155  [002] 1248198.487198: kvm_exit: reason ext_irq 
rip 0xfffff80001ca42c3
  qemu-system-x86-5155  [002] 1248198.487199: kvm_entry: vcpu 3
  qemu-system-x86-5153  [003] 1248198.487324: kvm_exit: reason ext_irq 
rip 0xfffff80001c9e859
  qemu-system-x86-5153  [003] 1248198.487324: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.487407: kvm_exit: reason ext_irq 
rip 0xfffff80001ca2f40
  qemu-system-x86-5153  [003] 1248198.487408: kvm_entry: vcpu 1
  qemu-system-x86-5152  [000] 1248198.487947: kvm_exit: reason ext_irq 
rip 0xfffff80001d04a00
  qemu-system-x86-5152  [000] 1248198.487948: kvm_entry: vcpu 0
  qemu-system-x86-5154  [001] 1248198.488072: kvm_exit: reason ext_irq 
rip 0xfffff80001c9e859
  qemu-system-x86-5154  [001] 1248198.488072: kvm_entry: vcpu 2
  qemu-system-x86-5146  [003] 1248198.488141: kvm_set_irq: gsi 0 level 1 
source 0
  qemu-system-x86-5146  [003] 1248198.488141: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-5146  [003] 1248198.488142: kvm_apic_accept_irq: 
apicid 0 vec 209 (Fixed|edge)
  qemu-system-x86-5146  [003] 1248198.488142: kvm_apic_accept_irq: 
apicid 1 vec 209 (Fixed|edge)
  qemu-system-x86-5146  [003] 1248198.488143: kvm_apic_accept_irq: 
apicid 2 vec 209 (Fixed|edge)
  qemu-system-x86-5146  [003] 1248198.488143: kvm_apic_accept_irq: 
apicid 3 vec 209 (Fixed|edge)
  qemu-system-x86-5152  [000] 1248198.488144: kvm_exit: reason ext_irq 
rip 0xfffff80001ca2f40
  qemu-system-x86-5146  [003] 1248198.488144: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-5146  [003] 1248198.488144: kvm_set_irq: gsi 0 level 0 
source 0
  qemu-system-x86-5152  [000] 1248198.488144: kvm_entry: vcpu 0
  qemu-system-x86-5154  [001] 1248198.488145: kvm_exit: reason ext_irq 
rip 0xfffff80001d049f1
  qemu-system-x86-5146  [003] 1248198.488145: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-5146  [003] 1248198.488145: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-5155  [002] 1248198.488145: kvm_exit: reason ext_irq 
rip 0xfffff80001d04a00
  qemu-system-x86-5154  [001] 1248198.488145: kvm_inj_virq: irq 209
  qemu-system-x86-5155  [002] 1248198.488145: kvm_inj_virq: irq 209
  qemu-system-x86-5154  [001] 1248198.488146: kvm_entry: vcpu 2
  qemu-system-x86-5152  [000] 1248198.488146: kvm_exit: reason 
interrupt_window rip 0xfffff80001ca3002
  qemu-system-x86-5155  [002] 1248198.488146: kvm_entry: vcpu 3
  qemu-system-x86-5152  [000] 1248198.488146: kvm_inj_virq: irq 209
  qemu-system-x86-5152  [000] 1248198.488147: kvm_entry: vcpu 0
  qemu-system-x86-5154  [001] 1248198.488147: kvm_exit: reason 
apic_access rip 0xfffff80001c9df13
  qemu-system-x86-5155  [002] 1248198.488147: kvm_exit: reason 
apic_access rip 0xfffff80001c9df13
  qemu-system-x86-5152  [000] 1248198.488148: kvm_exit: reason 
apic_access rip 0xfffff80001c9d5c4
  qemu-system-x86-5154  [001] 1248198.488149: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5154  [001] 1248198.488149: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5155  [002] 1248198.488149: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5155  [002] 1248198.488149: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5154  [001] 1248198.488149: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5155  [002] 1248198.488150: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5152  [000] 1248198.488150: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5152  [000] 1248198.488150: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5152  [000] 1248198.488151: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5154  [001] 1248198.488151: kvm_entry: vcpu 2
  qemu-system-x86-5155  [002] 1248198.488152: kvm_entry: vcpu 3
  qemu-system-x86-5152  [000] 1248198.488152: kvm_entry: vcpu 0
  qemu-system-x86-5153  [003] 1248198.488153: kvm_exit: reason ext_irq 
rip 0xfffff80001d04a00
  qemu-system-x86-5153  [003] 1248198.488154: kvm_inj_virq: irq 209
  qemu-system-x86-5153  [003] 1248198.488154: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.488156: kvm_exit: reason 
apic_access rip 0xfffff80001c9df13
  qemu-system-x86-5153  [003] 1248198.488158: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5153  [003] 1248198.488158: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5153  [003] 1248198.488159: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5153  [003] 1248198.488160: kvm_entry: vcpu 1
  qemu-system-x86-5155  [002] 1248198.488197: kvm_exit: reason ext_irq 
rip 0xfffff80001ca2f40
  qemu-system-x86-5155  [002] 1248198.488197: kvm_entry: vcpu 3
  qemu-system-x86-5153  [003] 1248198.488322: kvm_exit: reason ext_irq 
rip 0xfffff80001cba795
  qemu-system-x86-5153  [003] 1248198.488322: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.488406: kvm_exit: reason ext_irq 
rip 0xfffff80001d04a00
  qemu-system-x86-5153  [003] 1248198.488407: kvm_entry: vcpu 1
  qemu-system-x86-5152  [000] 1248198.488945: kvm_exit: reason ext_irq 
rip 0xfffff80001cdc193
  qemu-system-x86-5152  [000] 1248198.488946: kvm_entry: vcpu 0
  qemu-system-x86-5154  [001] 1248198.489070: kvm_exit: reason ext_irq 
rip 0xfffff80001ca2f40
  qemu-system-x86-5154  [001] 1248198.489071: kvm_entry: vcpu 2
  qemu-system-x86-5146  [003] 1248198.489139: kvm_set_irq: gsi 0 level 1 
source 0
  qemu-system-x86-5146  [003] 1248198.489139: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-5146  [003] 1248198.489140: kvm_apic_accept_irq: 
apicid 0 vec 209 (Fixed|edge)
  qemu-system-x86-5146  [003] 1248198.489140: kvm_apic_accept_irq: 
apicid 1 vec 209 (Fixed|edge)
  qemu-system-x86-5146  [003] 1248198.489141: kvm_apic_accept_irq: 
apicid 2 vec 209 (Fixed|edge)
  qemu-system-x86-5146  [003] 1248198.489141: kvm_apic_accept_irq: 
apicid 3 vec 209 (Fixed|edge)
  qemu-system-x86-5146  [003] 1248198.489142: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-5152  [000] 1248198.489142: kvm_exit: reason ext_irq 
rip 0xfffff80001d04a00
  qemu-system-x86-5146  [003] 1248198.489142: kvm_set_irq: gsi 0 level 0 
source 0
  qemu-system-x86-5146  [003] 1248198.489142: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-5152  [000] 1248198.489142: kvm_inj_virq: irq 209
  qemu-system-x86-5146  [003] 1248198.489143: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-5154  [001] 1248198.489143: kvm_exit: reason ext_irq 
rip 0xfffff80001d04a00
  qemu-system-x86-5155  [002] 1248198.489143: kvm_exit: reason ext_irq 
rip 0xfffff80001d049e4
  qemu-system-x86-5152  [000] 1248198.489143: kvm_entry: vcpu 0
  qemu-system-x86-5154  [001] 1248198.489143: kvm_inj_virq: irq 209
  qemu-system-x86-5155  [002] 1248198.489143: kvm_inj_virq: irq 209
  qemu-system-x86-5154  [001] 1248198.489144: kvm_entry: vcpu 2
  qemu-system-x86-5155  [002] 1248198.489144: kvm_entry: vcpu 3
  qemu-system-x86-5152  [000] 1248198.489145: kvm_exit: reason 
apic_access rip 0xfffff80001c9d5c4
  qemu-system-x86-5154  [001] 1248198.489145: kvm_exit: reason 
apic_access rip 0xfffff80001c9df13
  qemu-system-x86-5155  [002] 1248198.489145: kvm_exit: reason 
apic_access rip 0xfffff80001c9df13
  qemu-system-x86-5152  [000] 1248198.489146: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5152  [000] 1248198.489147: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5154  [001] 1248198.489147: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5154  [001] 1248198.489147: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5155  [002] 1248198.489147: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5152  [000] 1248198.489147: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5155  [002] 1248198.489147: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5154  [001] 1248198.489147: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5155  [002] 1248198.489149: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5152  [000] 1248198.489149: kvm_entry: vcpu 0
  qemu-system-x86-5154  [001] 1248198.489150: kvm_entry: vcpu 2
  qemu-system-x86-5153  [003] 1248198.489150: kvm_exit: reason ext_irq 
rip 0xfffff80001ca468f
  qemu-system-x86-5155  [002] 1248198.489151: kvm_entry: vcpu 3
  qemu-system-x86-5153  [003] 1248198.489151: kvm_inj_virq: irq 209
  qemu-system-x86-5153  [003] 1248198.489152: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.489153: kvm_exit: reason 
apic_access rip 0xfffff80001c9df13
  qemu-system-x86-5153  [003] 1248198.489156: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5153  [003] 1248198.489156: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5153  [003] 1248198.489156: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5153  [003] 1248198.489157: kvm_entry: vcpu 1
  qemu-system-x86-5155  [002] 1248198.489195: kvm_exit: reason ext_irq 
rip 0xfffff80001ca2f40
  qemu-system-x86-5155  [002] 1248198.489195: kvm_entry: vcpu 3
  qemu-system-x86-5153  [003] 1248198.489321: kvm_exit: reason ext_irq 
rip 0xfffff80001ca2f40
  qemu-system-x86-5154  [001] 1248198.489322: kvm_exit: reason ext_irq 
rip 0xfffff80001c9e859
  qemu-system-x86-5153  [003] 1248198.489322: kvm_entry: vcpu 1
  qemu-system-x86-5154  [001] 1248198.489323: kvm_entry: vcpu 2
  qemu-system-x86-5153  [003] 1248198.489404: kvm_exit: reason ext_irq 
rip 0xfffff80001d04a00
  qemu-system-x86-5153  [003] 1248198.489406: kvm_entry: vcpu 1
  qemu-system-x86-5152  [000] 1248198.489944: kvm_exit: reason ext_irq 
rip 0xfffff80001d04a00
  qemu-system-x86-5152  [000] 1248198.489945: kvm_entry: vcpu 0
  qemu-system-x86-5154  [001] 1248198.490069: kvm_exit: reason ext_irq 
rip 0xfffff80001cdd795
  qemu-system-x86-5154  [001] 1248198.490069: kvm_entry: vcpu 2
  qemu-system-x86-5146  [003] 1248198.490137: kvm_set_irq: gsi 0 level 1 
source 0
  qemu-system-x86-5146  [003] 1248198.490137: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-5146  [003] 1248198.490138: kvm_apic_accept_irq: 
apicid 0 vec 209 (Fixed|edge)
  qemu-system-x86-5146  [003] 1248198.490139: kvm_apic_accept_irq: 
apicid 1 vec 209 (Fixed|edge)
  qemu-system-x86-5146  [003] 1248198.490139: kvm_apic_accept_irq: 
apicid 2 vec 209 (Fixed|edge)
  qemu-system-x86-5146  [003] 1248198.490140: kvm_apic_accept_irq: 
apicid 3 vec 209 (Fixed|edge)
  qemu-system-x86-5146  [003] 1248198.490140: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-5152  [000] 1248198.490140: kvm_exit: reason ext_irq 
rip 0xfffff80001d04a00
  qemu-system-x86-5146  [003] 1248198.490140: kvm_set_irq: gsi 0 level 0 
source 0
  qemu-system-x86-5152  [000] 1248198.490141: kvm_inj_virq: irq 209
  qemu-system-x86-5146  [003] 1248198.490141: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-5146  [003] 1248198.490141: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-5154  [001] 1248198.490141: kvm_exit: reason ext_irq 
rip 0xfffff80001d04a00
  qemu-system-x86-5155  [002] 1248198.490141: kvm_exit: reason ext_irq 
rip 0xfffff80001ca2fbf
  qemu-system-x86-5152  [000] 1248198.490141: kvm_entry: vcpu 0
  qemu-system-x86-5154  [001] 1248198.490142: kvm_inj_virq: irq 209
  qemu-system-x86-5155  [002] 1248198.490142: kvm_entry: vcpu 3
  qemu-system-x86-5154  [001] 1248198.490142: kvm_entry: vcpu 2
  qemu-system-x86-5152  [000] 1248198.490143: kvm_exit: reason 
apic_access rip 0xfffff80001c9d5c4
  qemu-system-x86-5155  [002] 1248198.490143: kvm_exit: reason 
interrupt_window rip 0xfffff80001ca3002
  qemu-system-x86-5154  [001] 1248198.490143: kvm_exit: reason 
apic_access rip 0xfffff80001c9df13
  qemu-system-x86-5155  [002] 1248198.490144: kvm_inj_virq: irq 209
  qemu-system-x86-5155  [002] 1248198.490144: kvm_entry: vcpu 3
  qemu-system-x86-5152  [000] 1248198.490144: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5152  [000] 1248198.490145: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5152  [000] 1248198.490145: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5154  [001] 1248198.490145: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5154  [001] 1248198.490146: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5155  [002] 1248198.490146: kvm_exit: reason 
apic_access rip 0xfffff80001c9df13
  qemu-system-x86-5154  [001] 1248198.490146: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5152  [000] 1248198.490146: kvm_entry: vcpu 0
  qemu-system-x86-5154  [001] 1248198.490147: kvm_entry: vcpu 2
  qemu-system-x86-5155  [002] 1248198.490147: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5155  [002] 1248198.490148: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5155  [002] 1248198.490148: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5153  [003] 1248198.490149: kvm_exit: reason ext_irq 
rip 0xfffff80001d04a00
  qemu-system-x86-5155  [002] 1248198.490149: kvm_entry: vcpu 3
  qemu-system-x86-5153  [003] 1248198.490150: kvm_inj_virq: irq 209
  qemu-system-x86-5153  [003] 1248198.490150: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.490152: kvm_exit: reason 
apic_access rip 0xfffff80001c9df13
  qemu-system-x86-5153  [003] 1248198.490154: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5153  [003] 1248198.490154: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5153  [003] 1248198.490154: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5153  [003] 1248198.490155: kvm_entry: vcpu 1
  qemu-system-x86-5155  [002] 1248198.490193: kvm_exit: reason ext_irq 
rip 0xfffff80001ca2f40
  qemu-system-x86-5155  [002] 1248198.490194: kvm_entry: vcpu 3
  qemu-system-x86-5153  [003] 1248198.490319: kvm_exit: reason ext_irq 
rip 0xfffff80001d04a00
  qemu-system-x86-5153  [003] 1248198.490319: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.490402: kvm_exit: reason ext_irq 
rip 0xfffff80001c9e859
  qemu-system-x86-5153  [003] 1248198.490404: kvm_entry: vcpu 1
  qemu-system-x86-5154  [001] 1248198.491067: kvm_exit: reason ext_irq 
rip 0xfffff80001d04a7c
  qemu-system-x86-5154  [001] 1248198.491068: kvm_entry: vcpu 2
  qemu-system-x86-5146  [003] 1248198.491136: kvm_set_irq: gsi 0 level 1 
source 0
  qemu-system-x86-5146  [003] 1248198.491137: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-5146  [003] 1248198.491138: kvm_apic_accept_irq: 
apicid 0 vec 209 (Fixed|edge)
  qemu-system-x86-5146  [003] 1248198.491138: kvm_apic_accept_irq: 
apicid 1 vec 209 (Fixed|edge)
  qemu-system-x86-5146  [003] 1248198.491139: kvm_apic_accept_irq: 
apicid 2 vec 209 (Fixed|edge)
  qemu-system-x86-5146  [003] 1248198.491139: kvm_apic_accept_irq: 
apicid 3 vec 209 (Fixed|edge)
  qemu-system-x86-5146  [003] 1248198.491140: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-5146  [003] 1248198.491140: kvm_set_irq: gsi 0 level 0 
source 0
  qemu-system-x86-5146  [003] 1248198.491140: kvm_pic_set_irq: chip 0 
pin 0 (edge|masked)
  qemu-system-x86-5154  [001] 1248198.491141: kvm_exit: reason ext_irq 
rip 0xfffff80001d049f1
  qemu-system-x86-5146  [003] 1248198.491141: kvm_ioapic_set_irq: pin 2 
dst ff vec=209 (Fixed|physical|edge)
  qemu-system-x86-5155  [002] 1248198.491141: kvm_exit: reason ext_irq 
rip 0xfffff80001d04a00
  qemu-system-x86-5154  [001] 1248198.491141: kvm_inj_virq: irq 209
  qemu-system-x86-5155  [002] 1248198.491141: kvm_inj_virq: irq 209
  qemu-system-x86-5154  [001] 1248198.491142: kvm_entry: vcpu 2
  qemu-system-x86-5155  [002] 1248198.491142: kvm_entry: vcpu 3
  qemu-system-x86-5154  [001] 1248198.491143: kvm_exit: reason 
apic_access rip 0xfffff80001c9df13
  qemu-system-x86-5155  [002] 1248198.491143: kvm_exit: reason 
apic_access rip 0xfffff80001c9df13
  qemu-system-x86-5154  [001] 1248198.491145: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5155  [002] 1248198.491145: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5154  [001] 1248198.491145: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5155  [002] 1248198.491145: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5154  [001] 1248198.491146: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5155  [002] 1248198.491146: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5154  [001] 1248198.491147: kvm_entry: vcpu 2
  qemu-system-x86-5155  [002] 1248198.491147: kvm_entry: vcpu 3
  qemu-system-x86-5153  [003] 1248198.491148: kvm_exit: reason ext_irq 
rip 0xfffff80001ca2fbf
  qemu-system-x86-5153  [003] 1248198.491149: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.491151: kvm_exit: reason 
interrupt_window rip 0xfffff80001ca3002
  qemu-system-x86-5153  [003] 1248198.491151: kvm_inj_virq: irq 209
  qemu-system-x86-5153  [003] 1248198.491152: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.491153: kvm_exit: reason 
apic_access rip 0xfffff80001c9df13
  qemu-system-x86-5153  [003] 1248198.491155: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5153  [003] 1248198.491155: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5153  [003] 1248198.491156: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5153  [003] 1248198.491157: kvm_entry: vcpu 1
  qemu-system-x86-5152  [000] 1248198.491187: kvm_exit: reason ext_irq 
rip 0xfffff80001ca2f40
  qemu-system-x86-5152  [000] 1248198.491189: kvm_entry: vcpu 0
  qemu-system-x86-5152  [000] 1248198.491191: kvm_exit: reason 
interrupt_window rip 0xfffff80001ca3002
  qemu-system-x86-5152  [000] 1248198.491192: kvm_inj_virq: irq 209
  qemu-system-x86-5152  [000] 1248198.491192: kvm_entry: vcpu 0
  qemu-system-x86-5152  [000] 1248198.491194: kvm_exit: reason 
apic_access rip 0xfffff80001c9d5c4
  qemu-system-x86-5152  [000] 1248198.491196: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5152  [000] 1248198.491196: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5152  [000] 1248198.491197: kvm_ack_irq: irqchip 
IOAPIC pin 2
  qemu-system-x86-5152  [000] 1248198.491198: kvm_entry: vcpu 0
  qemu-system-x86-5152  [000] 1248198.491203: kvm_exit: reason 
apic_access rip 0xfffff80001c14092
  qemu-system-x86-5152  [000] 1248198.491205: kvm_apic: apic_read 
APIC_ICR = 0x82f
  qemu-system-x86-5152  [000] 1248198.491205: kvm_mmio: mmio read len 4 
gpa 0xfee00300 val 0x82f
  qemu-system-x86-5152  [000] 1248198.491206: kvm_entry: vcpu 0
  qemu-system-x86-5152  [000] 1248198.491207: kvm_exit: reason 
apic_access rip 0xfffff80001c140a9
  qemu-system-x86-5152  [000] 1248198.491208: kvm_mmio: mmio write len 4 
gpa 0xfee00310 val 0x2000000
  qemu-system-x86-5152  [000] 1248198.491209: kvm_apic: apic_write 
APIC_ICR2 = 0x2000000
  qemu-system-x86-5152  [000] 1248198.491209: kvm_entry: vcpu 0
  qemu-system-x86-5152  [000] 1248198.491210: kvm_exit: reason 
apic_access rip 0xfffff80001c140b0
  qemu-system-x86-5152  [000] 1248198.491212: kvm_mmio: mmio write len 4 
gpa 0xfee00300 val 0x82f
  qemu-system-x86-5152  [000] 1248198.491212: kvm_apic: apic_write 
APIC_ICR = 0x82f
  qemu-system-x86-5152  [000] 1248198.491212: kvm_apic_ipi: dst 2 vec 47 
(Fixed|logical|de-assert|edge|dst)
  qemu-system-x86-5152  [000] 1248198.491213: kvm_apic_accept_irq: 
apicid 1 vec 47 (Fixed|edge)
  qemu-system-x86-5152  [000] 1248198.491214: kvm_entry: vcpu 0
  qemu-system-x86-5153  [003] 1248198.491215: kvm_exit: reason ext_irq 
rip 0xfffff80001ca2f40
  qemu-system-x86-5153  [003] 1248198.491215: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.491216: kvm_exit: reason 
interrupt_window rip 0xfffff80001ca3002
  qemu-system-x86-5153  [003] 1248198.491217: kvm_inj_virq: irq 47
  qemu-system-x86-5153  [003] 1248198.491218: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.491219: kvm_exit: reason 
apic_access rip 0xfffff80001cedc67
  qemu-system-x86-5153  [003] 1248198.491220: kvm_mmio: mmio write len 4 
gpa 0xfee000b0 val 0x0
  qemu-system-x86-5153  [003] 1248198.491220: kvm_apic: apic_write 
APIC_EOI = 0x0
  qemu-system-x86-5153  [003] 1248198.491222: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.491224: kvm_exit: reason cr_access 
rip 0xfffff80001ca6ff6
  qemu-system-x86-5153  [003] 1248198.491224: kvm_cr: cr_write 3 = 
0x15f46d000
  qemu-system-x86-5153  [003] 1248198.491228: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.491229: kvm_exit: reason exception 
rip 0xfffff80001ca702c
  qemu-system-x86-5153  [003] 1248198.491230: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.491238: kvm_exit: reason exception 
rip 0xfffff80001c0f11e
  qemu-system-x86-5153  [003] 1248198.491239: kvm_page_fault: address 
ffffffffffd070f0 error_code 9
  qemu-system-x86-5153  [003] 1248198.491243: kvm_mmio: mmio 
unsatisfied-read len 4 gpa 0xfed000f0 val 0x0
  qemu-system-x86-5153  [003] 1248198.491252: kvm_mmio: mmio read len 4 
gpa 0xfed000f0 val 0x993c675d
  qemu-system-x86-5153  [003] 1248198.491253: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.491259: kvm_exit: reason cr_access 
rip 0xfffff80001ca6ff6
  qemu-system-x86-5153  [003] 1248198.491260: kvm_cr: cr_write 3 = 
0x164599000
  qemu-system-x86-5153  [003] 1248198.491262: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.491263: kvm_exit: reason exception 
rip 0xfffff80001ca702c
  qemu-system-x86-5153  [003] 1248198.491263: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.491318: kvm_exit: reason ext_irq 
rip 0xfffff80001cbad28
  qemu-system-x86-5153  [003] 1248198.491319: kvm_entry: vcpu 1
  qemu-system-x86-5153  [003] 1248198.491404: kvm_exit: reason ext_irq 
rip 0xfffff80001cdc130
  qemu-system-x86-5153  [003] 1248198.491405: kvm_entry: vcpu 1