Re: Freezing Windows 2008 x64bit guest

[Harri Olin <harri.olin@gmail.com>]

Gleb Natapov wrote:
> On Wed, Jul 21, 2010 at 12:22:45PM +0300, Harri Olin wrote:
>> Gleb Natapov wrote:
>>> On Wed, Jul 21, 2010 at 09:25:31AM +0300, Harri Olin wrote:
>>>> Gleb Natapov kirjoitti:
>>>>> On Mon, Jul 19, 2010 at 10:17:02AM +0300, Harri Olin wrote:
>>>>>> Gleb Natapov kirjoitti:
>>>>>>> On Thu, Jul 15, 2010 at 03:19:44PM +0200, Christoph Adomeit wrote:
>>>>>>>> But one Windows 2008 64 Bit Server Standard is freezing regularly.
>>>>>>>> This happens sometimes 3 times a day, sometimes it takes 2 days
>>>>>>>> until freeze. The Windows Machine is a clean fresh install.
>>>>>> I think I have seen same problem occur on my Windows 2008 SBS SP2
>>>>>> 64bit system, but a bit less often, only like once a week.
>>>>>> Now I haven't seen crashes but only freezes with qemu on 100% and
>>>>>> virtual system unresponsive.
>>>>>>
>>>>>> qemu command line:
>>>>>> /usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive
>>>>>> file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive
>>>>>> file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net
>>>>>> nic,macaddr=C0:FF:12:FB:AA:01,model=e1000 -net tap -smp 4 -localtime
>>>>>>
>>>>>>
>>>>> Try with different model then e1000 please. Default driver that comes
>>>>> with Windows known to have problems.
>>>> Didn't help, changed to default realtek emulation, but system
>>>> freezed again. Command line:
>>>> /usr/local/qemu-kvm-0.11.1/bin/qemu-system-x86_64 -drive
>>>> file=/dev/rigelvg/w2k8system,cache=none,boot=on -drive
>>>> file=/dev/rigelvg/w2k8data,cache=none -m 6144 -vnc :1 -net
>>>> nic,macaddr=C0:FF:12:FB:AA:01 -net tap -smp 4 -localtime
>>>>
>>>> This time the virtual system was not totally unresponsive somehow,
>>>> system pinged just fine and I think DNS server worked too. However
>>>> on console mouse moved but didn't react to clicking, ctrl-alt-del,
>>>> etc.
>>>>
>>> Does sendkey from monitor works? qemu-kvm-0.11.1 is very old and this is
>>> not total freeze which even harder to debug. I don't see anything
>>> extraordinary in your logs. 4643 interrupt per second for 4 cpus is
>>> normal if windows runs multimedia or other app that need hi-res timers.
>>> Does your host swapping? Is there any chance that you can try upstream qemu-kvm?
>>>
>> I haven't tested sendkey. I originally tried 0.12.2 or so but with
>> that windows crashed more often with bluescreen, I went down to
>> 0.11.1. This might be related to -td-rtc-hack and/or e1000 as using
>> td-rtc-hack on 0.11.1 windows crashed too to bluescreen.
>>
> With your guest -td-rtc-hack does nothing anyway, so you can just drop
> it. Although it shouldn't cause BSOD with upstream (it had such bug in
> the past and I am not sure that 0.12 has the fix). And you are using
> HPET anyway so -td-rtc-hack should be nop for you.

Ok, I think I added it because windows guest clock lagged behind a lot 
but didn't check if it helped.

>> Only thing that catches my eye on stats frozen vs normal is
>> irq_exits which jumps from 200 to 10000 when frozen and irq_window
>> which goes 10x.
> Hm, default Windows timer rate is 64Hz, so if nothing happens in the
> guest you should see at least 256 irq injection per second. In your KVM
> start output I only see 4655 irq injection per second not 10000. And
> trace shows that most of them are timer interrupts (vector 209 IIRC).

Frozen kvm_stat can be seen here; when I grabbed this, windows did not 
answer to ping.
stats: http://mizar.remote.agasha.com/k/kvm/kvm_stat_hang.txt
trace: http://mizar.remote.agasha.com/k/kvm/kvm_trace_hang.txt

Here's another frozen state stat dump; when I got this windows did 
answer to ping but was otherwise unresponsive:
stats: http://mizar.remote.agasha.com/k/kvm/kvm_stat_2_log.txt
trace: http://mizar.remote.agasha.com/k/kvm/kvm_trace_2.txt

On both times irq_exits go to 10000 but irq_injections stay at around 
4600. On both times qemu process was at 100% cpu load.

Note that stats quoted below are from normal operation, where everything 
works.

>> Host has swap partition enabled but hasn't swapped once yet,
>> currently there's over 1GB of free.
> And how many cpus does it have? 

Host has one quad-core Q8300 and 8GB of memory.

> BTW I see a lot of instruction emulations. That is strange. Looking at
> you trace again I see that the bulk of the emulations are caused by mmio
> to HPET. Please run with -no-hpet flag.

I'll do that later today too.

> 
>> I'll update host kernel to latest stable and qemu-kvm to upstream
>> later today.
>>
>> Here's stats from running system:
>>  efer_reload                  0       0
>>  exits                386474714   23272
>>  fpu_reload           173420913   12775
>>  halt_exits            51894921    4294
>>  halt_wakeup           51039254    4223
>>  host_state_reload    188645310   13420
>>  hypercalls                   0       0
>>  insn_emulation       212729430   15545
>>  insn_emulation_fail        324       0
>>  invlpg                 5177828     134
>>  io_exits              23132581     161
>>  irq_exits              7485833     225
>>  irq_injections        59239306    4655
>>  irq_window             2147438     153
>>  largepages                   0       0
>>  mmio_exits           110931423    9026
>>  mmu_cache_miss          468410       0
>>  mmu_flooded             193153       0
>>  mmu_pde_zapped          352178       0
>>  mmu_pte_updated         678023       0
>>  mmu_pte_write           587175       0
>>  mmu_recycled             34998       0
>>  mmu_shadow_zapped       454073       0
>>  mmu_unsync               19084       0
>>  nmi_injections               0       0
>>  nmi_window                   0       0
>>  pf_fixed              25171943     120
>>  pf_guest               6209034       2
>>  remote_tlb_flush       3044169      23
>>  request_irq                  0       0
>>  signal_exits                 0       0
>>  tlb_flush              8422950     149
>>