Re: XP machine freeze

[Marcelo Tosatti <mtosatti@redhat.com>]

On Mon, Mar 16, 2015 at 04:10:40PM +0100, Saso Slavicic wrote:
> Hi,
> 
> I'm fairly experienced with KVM (Centos 5/6), running about a dozen servers
> with 20-30 different (Linux & MS platform) systems.
> I have one Windows XP machine that acts very strangely - it freezes. I get
> ping timeout for the VM from my monitoring and the machine spins 2 or 3
> cores using all the cpu. Now the interesting thing that happens is that once
> you open the console, it suddenly starts working again. You can see the
> clock catching up as it was frozen in time and everything works normally
> once the timer catches up. It usually happens probably about once a month,
> although it happened yesterday and today again.
> 
> This machine is on Centos 6, qemu-kvm-0.12.1.2-2.448.el6_6, kernel
> 2.6.32-504.3.3.el6.x86_64.
> I was able to do some debugging when the machine was frozen, so I got some
> things to work with:
> 
> # virsh qemu-monitor-command --hmp DBserver 'info cpus'
> * CPU #0: pc=0x0000000080501fdd thread_id=32595
>   CPU #1: pc=0x00000000806e7a9b thread_id=32596
>   CPU #2: pc=0x00000000ba2da162 (halted) thread_id=32597
>   CPU #3: pc=0x00000000ba2da162 (halted) thread_id=32598
> 
> Now, in both yesterday's and today's event the CPU0 was stopped at
> 0x0000000080501fdd. I've disassembled the function and got this:
> 
>  0x0000000080501fb5:  int3
>  0x0000000080501fb6:  mov    %edi,%edi
>  0x0000000080501fb8:  push   %ebp
>  0x0000000080501fb9:  mov    %esp,%ebp
>  0x0000000080501fbb:  push   %esi
>  0x0000000080501fbc:  mov    %fs:0x20,%eax
>  0x0000000080501fc2:  mov    0x8(%ebp),%ecx
>  0x0000000080501fc5:  lea    -0x1(%ecx),%esi
>  0x0000000080501fc8:  test   %esi,%ecx
>  0x0000000080501fca:  lea    0x7ec(%eax),%edx
>  0x0000000080501fd0:  pop    %esi
>  0x0000000080501fd1:  je     0x80501fdd
>  0x0000000080501fd3:  lea    0x7a0(%eax),%edx
>  0x0000000080501fd9:  jmp    0x80501fdd
>  *0x0000000080501fdb:  pause
>  0x0000000080501fdd:  cmpl   $0x0,(%edx)
>  0x0000000080501fe0:  jne    0x80501fdb
>  0x0000000080501fe2:  pop    %ebp
>  0x0000000080501fe3:  ret    $0x4
>  0x0000000080501fe6:  int3
> 
> Mov %edi,%edi is clearly the start of some function. From what I've been
> able to understand, the code fetches _KPRCB structure (%fs:0x20) and then
> does a spinlock between fdb and fe0 checking for PacketBarrier (?) in EDX
> (0xffdff8c0). Now, $pc always shows fdd address, shouldn't it jump between
> fdb and fe0, it seems as if it was stuck at fdd?
> 
> # virsh qemu-monitor-command --hmp DBserver 'info registers'
>  EAX=ffdff120 EBX=c06ddf58 ECX=0000000e EDX=ffdff8c0
>  ESI=be6e3921 EDI=c06ddf60 EBP=ba4ff708 ESP=ba4ff708
>  EIP=80501fdd EFL=00000202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
>  ES =0023 00000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
>  CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
>  SS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
>  DS =0023 00000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
>  FS =0030 ffdff000 00001fff 00c09300 DPL=0 DS   [-WA]
>  GS =0000 00000000 000fffff 00000000
>  LDT=0000 00000000 000fffff 00000000
>  TR =0028 80042000 000020ab 00008b00 DPL=0 TSS32-busy
>  GDT=     8003f000 000003ff
>  IDT=     8003f400 000007ff
>  CR0=8001003b CR2=dbbec000 CR3=0b3c0020 CR4=000006f8
>  DR0=00000000 DR1=00000000 DR2=00000000 DR3=00000000
>  DR6=ffff0ff0 DR7=00000400
>  FCW=027f FSW=0020 [ST=0] FTW=00 MXCSR=00001fa0
>  FPR0=8053632b003c1658 c048 FPR1=e1e0c048bf80f6ab 76f8
>  FPR2=e1e0000000000000 0023 FPR3=0b017c30003c1658 0000
>  FPR4=0000003bba1a7604 1e64 FPR5=0007268c00000000 003b
>  FPR6=000002020000001b 2684 FPR7=e3e0a9b4e1b50de4 ca0b
>  XMM00=0000000000a1fc95000000000020027f
> XMM01=0000ffff00001fa000001c4c00000001
>  XMM02=000000000000c0488053632b003c1658
> XMM03=00000000000076f8e1e0c048bf80f6ab
>  XMM04=0000000000000023e1e0000000000000
> XMM05=00000000000000000b017c30003c1658
>  XMM06=0000000000001e640000003bba1a7604
> XMM07=000000000000003b0007268c00000000
> 
> Clearly, the address in EDX is not 0:
> 
> [root@linux ~]# virsh qemu-monitor-command --hmp DBserver 'x/1xb 0xFFDFF8C0'
> 00000000ffdff8c0: 0x0e
> 
> [root@linux ~]# virt-manager
> 
> [root@linux ~]# virsh qemu-monitor-command --hmp DBserver 'x/1xb 0xFFDFF8C0'
> 00000000ffdff8c0: 0x00
> 
> However as soon as the VM console is opened and machine starts, the address
> in EDX is set to 0 and the loop is broken.
> Does anybody recognize what function that is? What could possibly happen
> that opening the console and moving the mouse a little, unfreezes the
> machine?
> VM has .81 virtio drivers from Fedora repo at the moment.

Generate a Windows dump? 

https://support.microsoft.com/en-us/kb/254649

https://support.microsoft.com/en-us/kb/972110
Step 7: Generate a complete crash dump file or a kernel crash dump file
by using an NMI on a Windows-based system

(you can inject NMIs via QEMU monitor).

> 
> The configuration of the machine is pretty standard:
> 
> <!--
> WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
> OVERWRITTEN AND LOST. Changes to this xml configuration should be made
> using:
>   virsh edit DBserver
> or other application using the libvirt API.
> -->
> 
>  <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
>   <name>DBserver</name>
>   <uuid>e42b4cf2-7264-515f-4d24-6267eaa24be8</uuid>
>   <memory unit='KiB'>3145728</memory>
>   <currentMemory unit='KiB'>3145728</currentMemory>
>   <vcpu placement='static'>4</vcpu>
>   <os>
>     <type arch='x86_64' machine='rhel6.6.0'>hvm</type>
>     <boot dev='hd'/>
>   </os>
>   <features>
>     <acpi/>
>     <apic/>
>     <pae/>
>   </features>
>   <cpu>
>     <topology sockets='1' cores='4' threads='4'/>
>   </cpu>
>   <clock offset='localtime'>
>     <timer name='rtc' tickpolicy='catchup'/>
>   </clock>
>   <on_poweroff>destroy</on_poweroff>
>   <on_reboot>restart</on_reboot>
>   <on_crash>restart</on_crash>
>   <devices>
>     <emulator>/usr/libexec/qemu-kvm</emulator>
>     <disk type='block' device='disk'>
>       <driver name='qemu' type='raw' cache='none' io='native'/>
>       <source dev='/dev/drbd1'/>
>       <target dev='vda' bus='virtio'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> function='0x0'/>
>     </disk>
>     <disk type='block' device='disk'>
>       <driver name='qemu' type='raw' cache='none' io='native'/>
>       <source
> dev='/dev/disk/by-id/usb-WD_Ext_HDD_1021_574D415A4138353838383731-0:0'/>
>       <target dev='vdb' bus='virtio'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x04'
> function='0x0'/>
>     </disk>
>     <disk type='file' device='cdrom'>
>       <driver name='qemu' type='raw'/>
>       <target dev='hdc' bus='ide'/>
>       <readonly/>
>       <address type='drive' controller='0' bus='1' target='0' unit='0'/>
>     </disk>
>     <controller type='usb' index='0' model='ich9-ehci1'>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x05'
> function='0x7'/>
>     </controller>
>     <controller type='usb' index='0' model='ich9-uhci1'>
>       <master startport='0'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x05'
> function='0x0' multifunction='on'/>
>     </controller>
>     <controller type='usb' index='0' model='ich9-uhci2'>
>       <master startport='2'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x05'
> function='0x1'/>
>     </controller>
>     <controller type='usb' index='0' model='ich9-uhci3'>
>       <master startport='4'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x05'
> function='0x2'/>
>     </controller>
>     <controller type='ide' index='0'>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> function='0x1'/>
>     </controller>
>     <interface type='bridge'>
>       <mac address='52:54:00:a6:92:ca'/>
>       <source bridge='br0'/>
>       <model type='virtio'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
> function='0x0'/>
>     </interface>
>     <serial type='pty'>
>       <target port='0'/>
>     </serial>
>     <console type='pty'>
>       <target type='serial' port='0'/>
>     </console>
>     <input type='mouse' bus='ps2'/>
>     <graphics type='vnc' port='-1' autoport='yes'/>
>     <video>
>       <model type='vga' vram='9216' heads='1'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> function='0x0'/>
>     </video>
>     <memballoon model='virtio'>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x07'
> function='0x0'/>
>     </memballoon>
>   </devices>
>   <qemu:commandline>
>     <qemu:arg value='-set'/>
>     <qemu:arg value='device.virtio-disk0.x-data-plane=on'/>
>   </qemu:commandline>
>  </domain>
> 
> The above config is already changed as I've first experimented with removing
> usb tablet (and installing vmware mouse drivers), turning 'x-data-plane on'
> and so on, hoping to solve the problem...Is there anything else I can check
> the next time the machine freezes?
> 
> Regards,
> Saso Slavicic
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html