Re: Xen 3.2.1-rc5: FATAL PAGE FAULT

["Christopher S. Aker" <caker@theshore.net>]

Keir Fraser wrote:
> On 22/4/08 20:39, "Christopher S. Aker" <caker@theshore.net> wrote:
> 
>>> My best guess just now, seeing as noone else has reported ever seeing this,
>>> is that maybe you have a bad driver or hardware corrupting memory? Obviously
>>> that's a bit of a stab in the dark though.
>>>
>>> Have you seen this particular type of crash on multiple different machines?
>>> If so, are they different types of machine?
>> Two machines thus far, both are of identical software and hardware
>> configurations.
> 
> Have you been running this type of workload on a variety of hardware, or are
> you limited in the range of types of hardware that you're testing on? This
> might indicate whether it is significant that you have only seen the crash
> on a single hardware type.

Make that three machines.  They're all of the same config.  This 
identical hardware config runs fine under non-Xen.  It also only occurs 
when a domain is being destroyed, so I wouldn't suspect this is a driver 
issue or memory corruption given the pattern.  Xen is most suspect, in 
my mind.

Will you provide me with some debugging code that'll make these 
occurrences more useful in tracking down the problem the next time it 
triggers?

(XEN) Pagetable walk from 00000000c16e3f30:
(XEN)  L4[0x000] = 00000002bfe8d027 00000000000258e3
(XEN)  L3[0x003] = 646c696843206120 ffffffffffffffff
(XEN) domain_crash_sync called from entry.S
(XEN) Domain 84 (vcpu#2) crashed on cpu#1:
(XEN) ----[ Xen-3.2.1-rc1  x86_64  debug=y  Not tainted ]----
(XEN) CPU:    1
(XEN) RIP:    0061:[<00000000c0101347>]
(XEN) RFLAGS: 0000000000010246   CONTEXT: guest
(XEN) rax: 0000000000000000   rbx: 00000000deadbeef   rcx: 00000000deadbeef
(XEN) rdx: 00000000deadbeef   rsi: 00000000deadbeef   rdi: 00000000c7006030
(XEN) rbp: 00000000c16e3fac   rsp: 00000000c16e3f38   r8:  0000000000000000
(XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 00000000000026b0
(XEN) cr3: 000000060f4c8000   cr2: 00000000c0101347
(XEN) ds: 007b   es: 007b   fs: 0000   gs: 0000   ss: 0069   cs: 0061
(XEN) Guest stack trace from esp=c16e3f38:
(XEN)  Fault while accessing guest memory.
(XEN) ----[ Xen-3.2.1-rc1  x86_64  debug=y  Not tainted ]----
(XEN) CPU:    5
(XEN) RIP:    e008:[<ffff828c8013dee4>] put_page_type+0x17/0x107
(XEN) RFLAGS: 0000000000210282   CONTEXT: hypervisor
(XEN) rax: 000006162f512f98   rbx: ffff889a2f512f98   rcx: 6765746143206568
(XEN) rdx: 00000026f4620797   rsi: 00000000002bfe8d   rdi: ffff889a2f512f98
(XEN) rbp: ffff8300cfde7cb8   rsp: ffff8300cfde7c98   r8:  0000000000000000
(XEN) r9:  00000000deadbeef   r10: ffff828c801c5bf0   r11: 0000000000000000
(XEN) r12: 0000000000000001   r13: ffff889a2f512f98   r14: ffff8300cee88100
(XEN) r15: ffff8300cee88118   cr0: 000000008005003b   cr4: 00000000000026b0
(XEN) cr3: 000000062ffdf000   cr2: ffff889a2f512fb0
(XEN) ds: 007b   es: 007b   fs: 0000   gs: 0033   ss: 0000   cs: e008
(XEN) Xen stack trace from rsp=ffff8300cfde7c98:
(XEN)    ffff8300cfde7ca8 ffff889a2f512f98 0000000000000001 
00000000002bfe8d
(XEN)    ffff8300cfde7cd8 ffff828c8013b409 ffff8300cee88100 
ffff8302bfe8d008
(XEN)    ffff8300cfde7d08 ffff828c8013c06d ffff8300cee88100 
ffff828406dfc608
(XEN)    0000000068000001 ffff8300cee890f8 ffff8300cfde7d38 
ffff828c8013de5a
(XEN)    0000000060000001 0000000068000000 ffff828406dfc608 
ffff8300cee890f8
(XEN)    ffff8300cfde7d68 ffff828c8013df63 ffff828406dfc608 
ffff828406dfc608
(XEN)    ffff828406dfc608 ffff8300cee88100 ffff8300cfde7db8 
ffff828c80131680
(XEN)    0000000088000000 0000000080000000 ffff8300cfde7f28 
ffff8300cee88100
(XEN)    ffff8300cee88100 00000000b4dfb508 0000000000000000 
0000000000000000
(XEN)    ffff8300cfde7dd8 ffff828c80131a94 ffff8300cee88100 
0000000000000000
(XEN)    ffff8300cfde7e08 ffff828c80105638 ffff8300cfde7e08 
ffff828c8014601a
(XEN)    00000000b4dfb508 fffffffffffffff3 ffff8300cfde7f08 
ffff828c8010479f
(XEN)    0000000000000001 0000000000000000 0000000000000001 
0000000000000000
(XEN)    ffff8300cfde7e68 0000000000200286 0000000500000002 
082ebba4b7b80054
(XEN)    0836d2a401dfb538 b7ddfc50b7b8f68c b7aa53e400000001 
00000001b7a2ecdc
(XEN)    080facafb4dfb568 081361e0082f17c0 080797e7b775bf0c 
00000000b775bf28
(XEN)    b7dda02c00000060 b76f084c00000000 0805946cb4dfb588 
b7dda02cb76f084c
(XEN)    b7ddd6a000000000 00000002b765eeac a5dba1eea5dba1ee 
0000001f00000000
(XEN)    0000000000000010 ffff8300cee3c100 0000000000000000 
0000000000000000
(XEN)    0000000000000000 0000000000000000 00007cff302180b7 
ffff828c801bdd50
(XEN) Xen call trace:
(XEN)    [<ffff828c8013dee4>] put_page_type+0x17/0x107
(XEN)    [<ffff828c8013b409>] put_page_from_l3e+0x3f/0x4e
(XEN)    [<ffff828c8013c06d>] free_l3_table+0x78/0xc4
(XEN)    [<ffff828c8013de5a>] free_page_type+0x1d4/0x247
(XEN)    [<ffff828c8013df63>] put_page_type+0x96/0x107
(XEN)    [<ffff828c80131680>] relinquish_memory+0xce/0x262
(XEN)    [<ffff828c80131a94>] domain_relinquish_resources+0xd1/0x1b0
(XEN)    [<ffff828c80105638>] domain_kill+0x77/0x164
(XEN)    [<ffff828c8010479f>] do_domctl+0x4dd/0xc1e
(XEN)    [<ffff828c801bdd50>] compat_tracing_off+0xb/0x64
(XEN)
(XEN) Pagetable walk from ffff889a2f512fb0:
(XEN)  L4[0x111] = 0000000000000000 ffffffffffffffff
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 5:
(XEN) FATAL PAGE FAULT
(XEN) [error_code=0000]
(XEN) Faulting linear address: ffff889a2f512fb0
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...

Thanks,
-Chris