Re: Xen host crash

[Andrew Cooper <andrew.cooper3@citrix.com>]

[-- Attachment #1.1: Type: text/plain, Size: 4369 bytes --]

On 29/08/13 17:38, Rushikesh Jadhav wrote:
> This is a spinlock deadlock, resulting in the NMI watchdog timing out
> and killing the host.  Do you have Stack and register dump for PCPU6 ?
>
>
>     ~Andrew
>
>
> Hi Andrew, here is the stack trace and register dump for PCPU6 & PCPU7
>
> PCPU6 host state:
> RIP:    e008:[<ffff828c8010e310>]
> RFLAGS: 0000000000000002
> rax: 0000000000000004   rbx: 0000000000000001   rcx: ffff828c803629cc
> rdx: ffff828c8036286c   rsi: ffff828c803628dc   rdi: 00000000ffffffff
> rbp: 0000000000000082   rsp: ffff83247fd88e10   r8:  0000000000000001
> r9:  0000000000000001   r10: 00000000fffffffc   r11: 0000000000000001
> r12: 0000000000000001   r13: ffff832270af39a0   r14: 0000000000000002
> r15: 0000000000000009
> cr0: 0000000080050033   cr4: 00000000000026f0
> cr3: 000000205a12e000   cr2: fffff880005c5000
> ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
>
> current:DOM185 VCPU6 (ffff830047ee8000)
> stack context:DOM185 VCPU6 (ffff830047ee8000)
> idle VCPU:ffff83007ea5e000
>
> Stack at 0xffff83247fd88e10: 
>  ffff83247fd88e00:                                     8010eeb7
> ffff828c 801fb1d0 ffff828c
>  ffff83247fd88e20: 80127b36 ffff828c 00000028 00000030 7fd88f18
> ffff8324 7fd88e48 ffff8324
>  ffff83247fd88e40: 00000001 00000000 00000002 00000000 00000002
> 00000000 801f423d ffff828c
>  ffff83247fd88e60: 00000000 00000000 801f45b3 ffff828c 00000000
> 00000000 00000096 00000000
>  ffff83247fd88e80: 8011b7da ffff828c 000000e5 00000000 00000000
> 00000000 8019b4ab ffff828c
>  ffff83247fd88ea0: 7fd8ff20 ffff8324 80100000 ffff828c 8015eb75
> ffff828c 7fd88f58 ffff8324
>  ffff83247fd88ec0: 7fd8ff28 ffff8324 7fd88f58 ffff8324 00000002
> 00000000 7fd8ff28 ffff8324
>  ffff83247fd88ee0: 7fd88f58 ffff8324 00000002 00000000 8015eeba
> ffff828c 7fd8ff28 ffff8324
>  ffff83247fd88f00: 7fd8ff28 ffff8324 7fd88f58 ffff8324 801567a1
> ffff828c 00000000 00000000
>  ffff83247fd88f20: 00000006 00000000 7fd88f58 ffff8324 8015f37f
> ffff828c 00000000 00000000
>  ffff83247fd88f40: 339e8000 ffff8322 7fd8ff28 ffff8324 801d6877
> ffff828c 00000009 00000000
>  ffff83247fd88f60: 00000002 00000000 70af39a0 ffff8322 00000001
> 00000000 7fd8ff28 ffff8324
>  ffff83247fd88f80: 339e8000 ffff8322 00ff00ff 00ff00ff 0000ffff
> 0000ffff 339e8018 ffff8322
>  ffff83247fd88fa0: 00000002 00000000 00000000 00000000 0000000f
> 00000000 47ee8000 ffff8300
>  ffff83247fd88fc0: 00366807 00000000 60e72e90 ffff8319 00000000
> 00000002 8011ab02 ffff828c
>  ffff83247fd88fe0: 0000e008 00000000 00000246 00000000 7fd8fd70
> ffff8324 00000000 00000000
>
> Code:
>  da e8 df 91 01 00 e9 19 fd ff ff 90 90 90 90 90 90 90 90 90 90 <4c>
> 89 3f 4c 89 77 08 4c 89 6f 10 
>
> Call Trace:
>  [ffff828c8010e310] dump_domains+0x4d0
>   ffff828c8010eeb7  kexec_crash+0x57
>   ffff828c80127b36  panic+0x136
>   ffff828c8011b7da  __print_symbol+0x8a
>   ffff828c8019b4ab  vmx_asm_vmexit_handler+0x6b
>   ffff828c80100000  __per_cpu_shift+0x800ffff4
>   ffff828c8015eb75  show_stack+0x155
>   ffff828c8015eeba  fatal_trap+0x6a
>   ffff828c801567a1  nmi_watchdog_tick+0x131
>   ffff828c8015f37f  do_nmi+0xbf
>   ffff828c801d6877  handle_ist_exception+0x52
>   ffff828c8011ab02  _spin_lock+0x12
>
>   PCPU6 guest state:
> DOMAIN185 VCPU3
> RIP:    0000:[<fffff800016caee0>]
> RFLAGS: 0000000000010206
> rax: 0000000000000000   rbx: ffffffffffffffff   rcx: fffffa6002998000
> rdx: 0000000000000100   rsi: fffff6fd30014d00   rdi: 0000000000000010
> rbp: 0000000000000010   rsp: fffffa6001bc6c48   r8:  0000000000000000
> r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
> r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
> r15: 0000000000000000
> cr0: 0000000080050033   cr4: 00000000000026f0
> cr3: 000000205a12e000   cr2: fffff880005c5000
> ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: 0000
>
> VCPU pause flags: 0 arch flags 0x1
>
> current on PCPU6
> struct vcpu at ffff830047ee8000
>
> Stack unavailable.
>

Ok.

The problematic spinlock is at address 0xffff831960e72e90, which is
sadly a dynamically allocated one so cant be traced back to a symbol
using the symbol table.

Having said that, you are using Xen 3.4 which is ages out of date, and
in fact, probably using XenServer 5.6SP2 (so shouldn't be using
xen-devel anyway).

I suggest you upgrade to something less ancient.

~Andrew

[-- Attachment #1.2: Type: text/html, Size: 12309 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel