From: "Christopher S. Aker" <caker@theshore.net>
To: Keir Fraser <keir.fraser@eu.citrix.com>
Cc: xen devel <xen-devel@lists.xensource.com>
Subject: Re: Xen 3.2.1-rc5: FATAL PAGE FAULT
Date: Tue, 22 Apr 2008 14:19:24 -0400 [thread overview]
Message-ID: <480E2C2C.303@theshore.net> (raw)
In-Reply-To: <C41ABC81.1EC6A%keir.fraser@eu.citrix.com>
Keir Fraser wrote:
> That's going to be rather hard to answer without finding a more reliable
> repro of the bug, and then adding some hypervisor tracing.
Here are two more Xen traces with this problem. These always appear to
occur after we're forced to destroy a domain. The first trace is a
DoubleDump<tm> and has something new in the second dump...
http://www.theshore.net/~caker/xen/build-1.11/
I still don't have a method to reproduce, but since we're hitting this
with some frequency, would it be worth it to stick in some extra
debugging now?
====== First trace ======
----[ Xen-3.2.1-rc5 x86_64 debug=y Not tainted ]----
CPU: 1
RIP: e008:[<ffff828c8013dee4>] put_page_type+0x17/0x107
RFLAGS: 0000000000210286 CONTEXT: hypervisor
rax: 00001da2f4162bf0 rbx: ffffa026f4162bf0 rcx: 0000000080000000
rdx: 000000bdac808de6 rsi: 0000000000402fe3 rdi: ffffa026f4162bf0
rbp: ffff8300cf13fbf8 rsp: ffff8300cf13fbd8 r8: 0000000000000000
r9: 00000000deadbeef r10: ffff828c801c5bf0 r11: 0000000000000000
r12: 0000000000000000 r13: ffffa026f4162bf0 r14: 0000000000402fe3
r15: ffff82840a077b78 cr0: 000000008005003b cr4: 00000000000026b0
cr3: 000000062ffdd000 cr2: ffffa026f4162c08
ds: 007b es: 007b fs: 0000 gs: 0033 ss: 0000 cs: e008
Xen stack trace from rsp=ffff8300cf13fbd8:
0000000000000002 ffffa026f4162bf0 0000000000000000 ffff8300cee48100
ffff8300cf13fc18 ffff828c8013b3bb 0000000000200202 ffff830402fe3000
ffff8300cf13fc58 ffff828c8013bfcd 00000000cee48100 ffff8300cee48100
ffff82840a077b78 000000004c000001 ffff8300cee48100 ffff8300cee48118
ffff8300cf13fc88 ffff828c8013de4a 0000000044000001 000000004c000000
ffff82840a077b78 ffff8300cee48100 ffff8300cf13fcb8 ffff828c8013df63
00007cff30ec0337 ffff82840a077b78 0000000000000003 00000000004011a4
ffff8300cf13fcd8 ffff828c8013b409 ffff8300cf13fd68 ffff8304011a4018
ffff8300cf13fd08 ffff828c8013c06d ffff8300cee48100 ffff82840a02c1a0
0000000068000001 ffff8300cee490f8 ffff8300cf13fd38 ffff828c8013de5a
0000000060000001 0000000068000000 ffff82840a02c1a0 ffff8300cee490f8
ffff8300cf13fd68 ffff828c8013df63 ffff82840a02c1a0 ffff82840a02c1a0
ffff82840a02c1a0 ffff8300cee48100 ffff8300cf13fdb8 ffff828c80131680
0000000088000000 0000000080000000 ffff8300cf13ff28 ffff8300cee48100
ffff8300cee48100 00000000b4dfc508 0000000000000000 0000000000000000
ffff8300cf13fdd8 ffff828c80131a94 ffff8300cee48100 0000000000000000
ffff8300cf13fe08 ffff828c80105638 ffff82840f448b58 ffff8300cf13fe28
00000000b4dfc508 fffffffffffffff3 ffff8300cf13ff08 ffff828c8010479f
00000000000000fb ffff8300cee3a130 ffff8300cf13fe68 ffff828c8011c746
0000000000200282 ffff8300ceefe118 0000000500000002 083010acb7ab000a
Xen call trace:
[<ffff828c8013dee4>] put_page_type+0x17/0x107
[<ffff828c8013b3bb>] put_page_from_l2e+0x3f/0x4e
[<ffff828c8013bfcd>] free_l2_table+0xa6/0xce
[<ffff828c8013de4a>] free_page_type+0x1c4/0x247
[<ffff828c8013df63>] put_page_type+0x96/0x107
[<ffff828c8013b409>] put_page_from_l3e+0x3f/0x4e
[<ffff828c8013c06d>] free_l3_table+0x78/0xc4
[<ffff828c8013de5a>] free_page_type+0x1d4/0x247
[<ffff828c8013df63>] put_page_type+0x96/0x107
[<ffff828c80131680>] relinquish_memory+0xce/0x262
[<ffff828c80131a94>] domain_relinquish_resources+0xd1/0x1b0
[<ffff828c80105638>] domain_kill+0x77/0x164
[<ffff828c8010479f>] do_domctl+0x4dd/0xc1e
[<ffff828c801bdd50>] compat_tracing_off+0xb/0x64
Pagetable walk from ffffa026f4162c08:
L4[0x140] = 0000000000000000 ffffffffffffffff
****************************************
Panic on CPU 1:
FATAL PAGE FAULT
[error_code=0000]
Faulting linear address: ffffa026f4162c08
****************************************
Reboot in five seconds...
...3 seconds later, this occurred...
Assertion '__cpus_subset(&(cpumask), &(cpu_online_map), 32)' failed at
smp.c:84
----[ Xen-3.2.1-rc5 x86_64 debug=y Not tainted ]----
CPU: 0
RIP: e008:[<ffff828c80145c68>] send_IPI_mask_flat+0x29/0x9c
RFLAGS: 0000000000010002 CONTEXT: hypervisor
rax: 00000000fffffffe rbx: ffff8300cee3c100 rcx: 0000000000000003
rdx: 0000000000000040 rsi: 00000000000000fc rdi: 0000000000000004
rbp: ffff828c80237be8 rsp: ffff828c80237bd0 r8: ffff828c8024c780
r9: 0000000000000002 r10: 00000000deadbeef r11: 0000000000000000
r12: 0000000000000004 r13: 00000000000000fc r14: 0000000000000010
r15: 00001485db7a5091 cr0: 000000008005003b cr4: 00000000000026b0
cr3: 00000003ff15a000 cr2: 00000000e3015078
ds: 007b es: 007b fs: 00d8 gs: 0000 ss: 0000 cs: e008
Xen stack trace from rsp=ffff828c80237bd0:
ffff8300cee3c100 0000000000000086 0000000000000000 ffff828c80237c08
ffff828c8014601a ffff8300cee30f00 0000000000000004 ffff828c80237c38
ffff828c80114da0 0000000000000004 ffff828c80137fe0 0000000000000004
ffff828c8025951c ffff828c80237c68 ffff828c80119b18 ffff828c80237c98
ffff828c80137ac2 ffff8300cee3c100 ffff8300cfdd4100 ffff828c80237c98
ffff828c80107409 00000000c0621300 ffff8300cfdd4100 ffff8300cee30f00
0000000000000000 ffff828c80237ca8 ffff828c801075c9 ffff828c80237cd8
ffff828c80137fe0 ffff828c80259500 ffff828c8025951c 0000000000000098
ffff828c80237d38 ffff828c80237d28 ffff828c80137ac2 0000000000000082
0000000000000000 ffff828c80237d18 0000000000000009 00000000ffffffff
ffff828c801ebb60 ffff828c8020e100 00001485db7a5091 00007d737fdc82a7
ffff828c801336e6 00001485db7a5091 ffff828c8020e100 ffff828c801ebb60
00000000ffffffff ffff828c80237de8 0000000000000009 0000000000000000
00000000deadbeef 0000000000000000 0000000000000000 000000007d9b040e
000000007d8a4358 000000000000290c 00000000001e8480 00000000000003e8
0000009800000000 ffff828c8012ac48 000000000000e008 0000000000000216
ffff828c80237de8 0000000000000000 00001485db7a5091 ffff828c80237e08
ffff828c80146257 ffff828c80237f28 ffff828c8020e534 ffff828c80237e28
ffff828c80145b9a ffff828c80237f28 ffff828c8020e534 ffff828c80237e38
ffff828c80146312 00007d737fdc8197 ffff828c801347a0 00001485db7a5091
Xen call trace:
[<ffff828c80145c68>] send_IPI_mask_flat+0x29/0x9c
[<ffff828c8014601a>] smp_send_event_check_mask+0x3e/0x40
[<ffff828c80114da0>] csched_vcpu_wake+0x242/0x259
[<ffff828c80119b18>] vcpu_wake+0x12d/0x248
[<ffff828c80107409>] evtchn_set_pending+0xe5/0x15c
[<ffff828c801075c9>] send_guest_pirq+0x61/0x63
[<ffff828c80137fe0>] __do_IRQ_guest+0x19c/0x1b2
[<ffff828c80137ac2>] do_IRQ+0x5a/0x1a7
[<ffff828c801336e6>] common_interrupt+0x26/0x30
[<ffff828c8012ac48>] __udelay+0x30/0x48
[<ffff828c80146257>] smp_send_stop+0x39/0x67
[<ffff828c80145b9a>] machine_restart+0x4f/0xc5
[<ffff828c80146312>] smp_call_function_interrupt+0x79/0xa7
[<ffff828c801347a0>] call_function_interrupt+0x30/0x40
[<ffff828c8012c73b>] default_idle+0x2f/0x34
[<ffff828c8012c7ff>] idle_loop+0x70/0x77
****************************************
Panic on CPU 0:
Assertion '__cpus_subset(&(cpumask), &(cpu_online_map), 32)' failed at
smp.c:84
****************************************
Reboot in five seconds...
====== Second trace ======
----[ Xen-3.2.1-rc5 x86_64 debug=y Not tainted ]----
CPU: 0
RIP: e008:[<ffff828c8013dee4>] put_page_type+0x17/0x107
RFLAGS: 0000000000210286 CONTEXT: hypervisor
rax: 00000a51169fd050 rbx: ffff8cd5169fd050 rcx: 0000000080000000
rdx: 0000004206f73202 rsi: 00000000004041e1 rdi: ffff8cd5169fd050
rbp: ffff828c80237bf8 rsp: ffff828c80237bd8 r8: 0000000000000000
r9: 00000000deadbeef r10: ffff828c801c5bf0 r11: 0000000000000000
r12: 0000000000000000 r13: ffff8cd5169fd050 r14: 00000000004041e1
r15: ffff82840a0a4b28 cr0: 000000008005003b cr4: 00000000000026b0
cr3: 000000062ffd9000 cr2: ffff8cd5169fd068
ds: 007b es: 007b fs: 0000 gs: 0033 ss: 0000 cs: e008
Xen stack trace from rsp=ffff828c80237bd8:
ffff828409df5d01 ffff8cd5169fd050 0000000000000000 ffff8300ceea0100
ffff828c80237c18 ffff828c8013b3bb 0000000400000004 ffff8304041e1000
ffff828c80237c58 ffff828c8013bfcd 00000003f2f24027 ffff8300ceea0100
ffff82840a0a4b28 0000000048000001 ffff8300ceea0100 ffff8300ceea0118
ffff828c80237c88 ffff828c8013de4a 0000000040000001 0000000048000000
ffff82840a0a4b28 ffff8300ceea0100 ffff828c80237cb8 ffff828c8013df63
0000000000000000 ffff82840a0a4b28 0000000000000000 0000000000402dd4
ffff828c80237cd8 ffff828c8013b409 ffff8300ceea0100 ffff830402dd4000
ffff828c80237d08 ffff828c8013c06d ffff8300ceea0100 ffff82840a072920
0000000068000001 ffff8300ceea10f8 ffff828c80237d38 ffff828c8013de5a
0000000060000001 0000000068000000 ffff82840a072920 ffff8300ceea10f8
ffff828c80237d68 ffff828c8013df63 ffff82840a072920 ffff82840a072920
ffff82840a072920 ffff8300ceea0100 ffff828c80237db8 ffff828c80131680
0000000088000000 0000000080000000 ffff828c80237f28 ffff8300ceea0100
ffff8300ceea0100 00000000b2cf9868 0000000000000000 0000000000000000
ffff828c80237dd8 ffff828c80131a94 ffff8300ceea0100 0000000000000000
ffff828c80237e08 ffff828c80105638 ffff828c80237e18 ffff828c80114da0
00000000b2cf9868 fffffffffffffff3 ffff828c80237f08 ffff828c8010479f
ffff828c80237e48 ffff8300cee36130 0000000000000000 000078cdfb20f27f
0000000000200282 0000000000000000 0000000500000002 081d66ecb7af0010
Xen call trace:
[<ffff828c8013dee4>] put_page_type+0x17/0x107
[<ffff828c8013b3bb>] put_page_from_l2e+0x3f/0x4e
[<ffff828c8013bfcd>] free_l2_table+0xa6/0xce
[<ffff828c8013de4a>] free_page_type+0x1c4/0x247
[<ffff828c8013df63>] put_page_type+0x96/0x107
[<ffff828c8013b409>] put_page_from_l3e+0x3f/0x4e
[<ffff828c8013c06d>] free_l3_table+0x78/0xc4
[<ffff828c8013de5a>] free_page_type+0x1d4/0x247
[<ffff828c8013df63>] put_page_type+0x96/0x107
[<ffff828c80131680>] relinquish_memory+0xce/0x262
[<ffff828c80131a94>] domain_relinquish_resources+0xd1/0x1b0
[<ffff828c80105638>] domain_kill+0x77/0x164
[<ffff828c8010479f>] do_domctl+0x4dd/0xc1e
[<ffff828c801bdd50>] compat_tracing_off+0xb/0x64
Pagetable walk from ffff8cd5169fd068:
L4[0x119] = 0000000000000000 ffffffffffffffff
****************************************
Panic on CPU 0:
FATAL PAGE FAULT
[error_code=0000]
Faulting linear address: ffff8cd5169fd068
****************************************
Reboot in five seconds...
-Chris
next prev parent reply other threads:[~2008-04-22 18:19 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2008-04-03 4:34 Xen 3.2.1-rc1: FATAL PAGE FAULT Christopher S. Aker
2008-04-03 14:04 ` Christopher S. Aker
2008-04-03 15:55 ` Keir Fraser
2008-04-22 18:19 ` Christopher S. Aker [this message]
2008-04-22 18:46 ` Xen 3.2.1-rc5: " Keir Fraser
2008-04-22 19:39 ` Christopher S. Aker
2008-04-22 20:21 ` Keir Fraser
2008-04-28 14:02 ` Christopher S. Aker
2008-04-28 14:44 ` Keir Fraser
2008-04-28 15:00 ` Christopher S. Aker
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=480E2C2C.303@theshore.net \
--to=caker@theshore.net \
--cc=keir.fraser@eu.citrix.com \
--cc=xen-devel@lists.xensource.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.