* xm mem-set causes kernel panic
@ 2006-05-03 19:01 Krysan, Susan
0 siblings, 0 replies; 7+ messages in thread
From: Krysan, Susan @ 2006-05-03 19:01 UTC (permalink / raw)
To: xen-devel
Cc: Subrahmanian, Raj, Vessey, Bruce A, Puthiyaparambil, Aravindh,
Carb, Brian A
[-- Attachment #1: Type: text/plain, Size: 483 bytes --]
When I boot the ES7000 with 32 GB of RAM, dom0 is allocated almost all of it. I issued an xm mem-set 0 512 command to decrease dom0's memory to 512 MB. Using xentop, I see the amount of memory allocated to dom0 slowly decrease. The kernel panic occurs when dom0's memory is around 1 GB. The serial console output is attached. Using SLES10 Beta 11 upgraded to xen changeset 9903.
<<serport2.txt>>
Thanks,
> Sue Krysan
> Linux Systems Group
> Unisys Corporation
>
>
[-- Attachment #2: serport2.txt --]
[-- Type: text/plain, Size: 59886 bytes --]
Kernel panic - not syncing: Out of memory and no killable processes...
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
f801cd57c>{blk_congestion_wait+132} <ffffffff8013fa4d>{autoremove_wake_function+0}
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
<ffffffff80159715>{try_to_free_pages+342} <ffffffff80154cdf>{__alloc_pages+422}
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
<ffffffff8023c4c2>{balloon_process+904} <ffffffff8023c13a>{balloon_process+0}
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
<ffffffff8013bf43>{run_workqueue+153} <ffffffff8013c685>{worker_thread+0}
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
<ffffffff8013c78e>{worker_thread+265} <ffffffff801236c3>{__wake_up_common+62}
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
<ffffffff80124910>{default_wake_function+0} <ffffffff8013f913>{kthread+212}
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
<ffffffff8013c685>{worker_thread+0} <ffffffff8010b8c6>{child_rip+8}
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
<ffffffff8013c685>{worker_thread+0} <ffffffff8013f83f>{kthread+0}
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
<ffffffff8010b8be>{child_rip+0}
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
<0>Kernel panic - not syncing: Aiee, killing interrupt handler!
<0>general protection fault: 0000 [12] SMP
CPU 15
Modules linked in: ipv6 bridge button battery ac dm_mod e1000 ext3 jbd reiserfs fan thermal processor sg mptspi mptscsih mptbase piix sd_mod scsi_mod
Pid: 4687, comm: python Tainted: GF 2.6.16-xen #1
RIP: e030:[<ffffffff8011326b>] <ffffffff8011326b>{smp_really_stop_cpu+9}
RSP: e02b:ffff88000491bea8 EFLAGS: 00010002
RAX: ffffffffff5783c0 RBX: 0000000000000000 RCX: 0000000000000001
RDX: ffffffffff578000 RSI: 0000000000000000 RDI: 0000000000000000
RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000020 R11: 0000000000000000 R12: ffffffff80113262
R13: 0000000000000000 R14: ffff8807d087b7d8 R15: ffff8807d087b7d8
FS: 00002aae5150ffc0(0000) GS:ffffffff803fb780(0000) knlGS:0000000000000000
CS: e033 DS: 0000 ES: 0000
Process python (pid: 4687, threadinfo ffff8807d087a000, task ffff8807d12eb040)
Stack: 0000000000000000 ffffffff801134c8 ffff8807d6a56140 0000000000000000
000000000000012e ffffffff8014ee2d ffffffff80405b80 0000000000009700
000000000000012e ffff8807d6a56140
Call Trace: <IRQ> <ffffffff801134c8>{smp_call_function_interrupt+79}
<ffffffff8014ee2d>{handle_IRQ_event+78} <ffffffff8014ef10>{__do_IRQ+153}
<ffffffff8010d661>{do_IRQ+60} <ffffffff8023834a>{evtchn_do_upcall+134}
<ffffffff80113262>{smp_really_stop_cpu+0} <ffffffff8010b672>{do_hypervisor_callback+30} <EOI>
<ffffffff80113262>{smp_really_stop_cpu+0} <ffffffff801132d2>{__smp_call_function+100}
<ffffffff801132d0>{__smp_call_function+98} <ffffffff80113262>{smp_really_stop_cpu+0}
<ffffffff8011331b>{smp_send_stop+38} <ffffffff8012b885>{panic+141}
<ffffffff8015909f>{shrink_slab+317} <ffffffff8013038d>{getnstimeofday+16}
<ffffffff8014b7ec>{cpuset_excl_nodes_overlap+30} <ffffffff80153300>{out_of_memory+276}
<ffffffff80154d56>{__alloc_pages+541} <ffffffff80109fc0>{__switch_to+740}
<ffffffff80167143>{read_swap_cache_async+66} <ffffffff8015b333>{swapin_readahead+78}
<ffffffff802ceb52>{.text.lock.spinlock+103} <ffffffff8015d670>{__handle_mm_fault+2324}
<ffffffff8016d3cb>{kmem_cache_alloc+20} <ffffffff801180d0>{do_page_fault+1938}
<ffffffff801248ff>{try_to_wake_up+852} <ffffffff8010b5c7>{error_exit+0}
<ffffffff801db6e0>{__put_user_4+32} <ffffffff80128e40>{mm_release+102}
<ffffffff8012d3d2>{exit_mm+22} <ffffffff8012eaa8>{do_exit+520}
<ffffffff8012f1f3>{sys_exit_group+0} <ffffffff80138379>{get_signal_to_deliver+1652}
<ffffffff8010a1f7>{do_signal+109} <ffffffff80124910>{default_wake_function+0}
<ffffffff8010af9d>{sysret_signal+56} <ffffffff8010b271>{ptregscall_common+61}
Code: f4 eb fd 41 55 49 89 fd 48 c7 c7 c8 c9 3c 80 41 54 55 48 89
RIP <ffffffff8011326b>{smp_really_stop_cpu+9} RSP <ffff88000491bea8>
Badness in do_unblank_screen at drivers/char/vt.c:2830
Call Trace: <IRQ> <ffffffff80228114>{do_unblank_screen+76}
<ffffffff8011775b>{bust_spinlocks+28} <ffffffff8010bbbd>{oops_end+21}
<ffffffff8010c25b>{die+58} <ffffffff8010d04a>{do_general_protection+291}
<ffffffff80113262>{smp_really_stop_cpu+0} <ffffffff8010b5c7>{error_exit+0}
<ffffffff80113262>{smp_really_stop_cpu+0} <ffffffff8011326b>{smp_really_stop_cpu+9}
<ffffffff8011326b>{smp_really_stop_cpu+9} <ffffffff801134c8>{smp_call_function_interrupt+79}
<ffffffff8014ee2d>{handle_IRQ_event+78} <ffffffff8014ef10>{__do_IRQ+153}
<ffffffff8010d661>{do_IRQ+60} <ffffffff8023834a>{evtchn_do_upcall+134}
<ffffffff80113262>{smp_really_stop_cpu+0} <ffffffff8010b672>{do_hypervisor_callback+30} <EOI>
<ffffffff80113262>{smp_really_stop_cpu+0} <ffffffff801132d2>{__smp_call_function+100}
<ffffffff801132d0>{__smp_call_function+98} <ffffffff80113262>{smp_really_stop_cpu+0}
<ffffffff8011331b>{smp_send_stop+38} <ffffffff8012b885>{panic+141}
<ffffffff8015909f>{shrink_slab+317} <ffffffff8013038d>{getnstimeofday+16}
<ffffffff8014b7ec>{cpuset_excl_nodes_overlap+30} <ffffffff80153300>{out_of_memory+276}
<ffffffff80154d56>{__alloc_pages+541} <ffffffff80109fc0>{__switch_to+740}
<ffffffff80167143>{read_swap_cache_async+66} <ffffffff8015b333>{swapin_readahead+78}
<ffffffff802ceb52>{.text.lock.spinlock+103} <ffffffff8015d670>{__handle_mm_fault+2324}
<ffffffff8016d3cb>{kmem_cache_alloc+20} <ffffffff801180d0>{do_page_fault+1938}
<ffffffff801248ff>{try_to_wake_up+852} <ffffffff8010b5c7>{error_exit+0}
<ffffffff801db6e0>{__put_user_4+32} <ffffffff80128e40>{mm_release+102}
<ffffffff8012d3d2>{exit_mm+22} <ffffffff8012eaa8>{do_exit+520}
<ffffffff8012f1f3>{sys_exit_group+0} <ffffffff80138379>{get_signal_to_deliver+1652}
<ffffffff8010a1f7>{do_signal+109} <ffffffff80124910>{default_wake_function+0}
<ffffffff8010af9d>{sysret_signal+56} <ffffffff8010b271>{ptregscall_common+61}
<0>Kernel panic - not syncing: Aiee, killing interrupt handler!
<0>general protection fault: 0000 [13] SMP
CPU 7
Modules linked in: ipv6 bridge button battery ac dm_mod e1000 ext3 jbd reiserfs fan thermal processor sg mptspi mptscsih mptbase piix sd_mod scsi_mod
Pid: 5967, comm: gnome-panel Tainted: GF 2.6.16-xen #1
RIP: e030:[<ffffffff8011326b>] <ffffffff8011326b>{smp_really_stop_cpu+9}
RSP: e02b:ffff8807d6a7bea8 EFLAGS: 00010002
RAX: ffffffffff5781c0 RBX: 0000000000000000 RCX: 0000000000000001
RDX: ffffffffff578000 RSI: 0000000000000000 RDI: 0000000000000000
RBP: 0000000000000000 R08: ffff880024041368 R09: ffff8807c57e7b48
R10: 0000000000000007 R11: 0000000000000246 R12: ffffffff80113262
R13: 0000000000000000 R14: ffff8807c57e78f8 R15: ffff8807c57e78f8
FS: 00002ae36fe73e30(0000) GS:ffffffff803fb380(0000) knlGS:0000000000000000
CS: e033 DS: 0000 ES: 0000
Process gnome-panel (pid: 5967, threadinfo ffff8807c57e6000, task ffff8807d4817180)
Stack: 0000000000000000 ffffffff801134c8 ffff880000044380 0000000000000000
0000000000000116 ffffffff8014ee2d ffffffff80404f80 0000000000008b00
0000000000000116 ffff880000044380
Call Trace: <IRQ> <ffffffff801134c8>{smp_call_function_interrupt+79}
<ffffffff8014ee2d>{handle_IRQ_event+78} <ffffffff8014ef10>{__do_IRQ+153}
<ffffffff8010d661>{do_IRQ+60} <ffffffff8023834a>{evtchn_do_upcall+134}
<ffffffff8010b672>{do_hypervisor_callback+30} <EOI>
<ffffffff8010722a>{hypercall_page+554} <ffffffff8010722a>{hypercall_page+554}
<ffffffff80237bfd>{force_evtchn_callback+10} <ffffffff801585f6>{shrink_zone+1774}
<ffffffff802cd0ab>{thread_return+0} <ffffffff80134fb6>{lock_timer_base+27}
<ffffffff80135028>{try_to_del_timer_sync+81} <ffffffff802cd47c>{io_schedule_timeout+50}
<ffffffff801cd57c>{blk_congestion_wait+132} <ffffffff8013fa4d>{autoremove_wake_function+0}
<ffffffff80159715>{try_to_free_pages+342} <ffffffff8015335c>{out_of_memory+368}
<ffffffff80154cdf>{__alloc_pages+422} <ffffffff8015667c>{__do_page_cache_readahead+245}
<ffffffff8015016c>{sync_page+0} <ffffffff802cd4b9>{io_schedule+50}
<ffffffff802cdcfd>{__wait_on_bit_lock+91} <ffffffff80152893>{filemap_nopage+323}
<ffffffff8015d2e3>{__handle_mm_fault+1415} <ffffffff801812b2>{__user_walk_fd+65}
<ffffffff801180d0>{do_page_fault+1938} <ffffffff8018aa7b>{mntput_no_expire+25}
<ffffffff8014dd09>{audit_syscall_exit+747} <ffffffff8010b5c7>{error_exit+0}
Code: f4 eb fd 41 55 49 89 fd 48 c7 c7 c8 c9 3c 80 41 54 55 48 89
RIP <ffffffff8011326b>{smp_really_stop_cpu+9} RSP <ffff8807d6a7bea8>
<0>Kernel panic - not syncing: Aiee, killing interrupt handler!
general protection fault: 0000 [14] SMP
CPU 11
Modules linked in: ipv6 bridge button battery ac dm_mod e1000 ext3 jbd reiserfs fan thermal processor sg mptspi mptscsih mptbase piix sd_mod scsi_mod
Pid: 0, comm: swapper Tainted: GF 2.6.16-xen #1
RIP: e030:[<ffffffff8011326b>] <ffffffff8011326b>{smp_really_stop_cpu+9}
RSP: e02b:ffff88002c0cbea8 EFLAGS: 00010002
RAX: ffffffffff5782c0 RBX: 0000000000000000 RCX: 0000000000000001
RDX: ffffffffff578000 RSI: 0000000000000002 RDI: 0000000000000000
RBP: 0000000000000000 R08: 0000000100073353 R09: 000000000000000b
R10: 00000001000733b4 R11: 0000000000000246 R12: ffffffff80113262
R13: 0000000000000000 R14: ffff88000499fe58 R15: ffff88000499fe58
FS: 00002b8d6e4e6ca0(0000) GS:ffffffff803fb580(0000) knlGS:0000000000000000
CS: e033 DS: 002b ES: 002b
Process swapper (pid: 0, threadinfo ffff88000499e000, task ffff8807d6abb180)
Stack: ffffffff80113262 ffffffff801134c8 ffff8807d6a569c0 0000000000000000
0000000000000122 ffffffff8014ee2d ffffffff80405580 0000000000009100
0000000000000122 ffff8807d6a569c0
Call Trace: <IRQ> <ffffffff80113262>{smp_really_stop_cpu+0}
<ffffffff801134c8>{smp_call_function_interrupt+79} <ffffffff8014ee2d>{handle_IRQ_event+78}
<ffffffff8014ef10>{__do_IRQ+153} <ffffffff8010d661>{do_IRQ+60}
<ffffffff8023834a>{evtchn_do_upcall+134} <ffffffff8010b672>{do_hypervisor_callback+30} <EOI>
<ffffffff801073aa>{hypercall_page+938} <ffffffff801073aa>{hypercall_page+938}
<ffffffff801095de>{xen_idle+132} <ffffffff80109933>{cpu_idle+217}
Code: f4 eb fd 41 55 49 89 fd 48 c7 c7 c8 c9 3c 80 41 54 55 48 89
RIP <ffffffff8011326b>{smp_really_stop_cpu+9} RSP <ffff88002c0cbea8>
Badness in do_unblank_screen at drivers/char/vt.c:2830
Call Trace: <IRQ> <ffffffff80228114>{do_unblank_screen+76}
<ffffffff8011775b>{bust_spinlocks+28} <ffffffff8010bbbd>{oops_end+21}
<ffffffff8010c25b>{die+58} <ffffffff8010d04a>{do_general_protection+291}
<ffffffff80113262>{smp_really_stop_cpu+0} <ffffffff8010b5c7>{error_exit+0}
<ffffffff80113262>{smp_really_stop_cpu+0} <ffffffff8011326b>{smp_really_stop_cpu+9}
<ffffffff8011326b>{smp_really_stop_cpu+9} <ffffffff80113262>{smp_really_stop_cpu+0}
<ffffffff801134c8>{smp_call_function_interrupt+79} <ffffffff8014ee2d>{handle_IRQ_event+78}
<ffffffff8014ef10>{__do_IRQ+153} <ffffffff8010d661>{do_IRQ+60}
<ffffffff8023834a>{evtchn_do_upcall+134} <ffffffff8010b672>{do_hypervisor_callback+30} <EOI>
<ffffffff801073aa>{hypercall_page+938} <ffffffff801073aa>{hypercall_page+938}
<ffffffff801095de>{xen_idle+132} <ffffffff80109933>{cpu_idle+217}
<0>Kernel panic - not syncing: Aiee, killing interrupt handler!
<0>general protection fault: 0000 [15] SMP
CPU 10
Modules linked in: ipv6 bridge button battery ac dm_mod e1000 ext3 jbd reiserfs fan thermal processor sg mptspi mptscsih mptbase piix sd_mod scsi_mod
Pid: 5972, comm: nautilus Tainted: GF 2.6.16-xen #1
RIP: e030:[<ffffffff8011326b>] <ffffffff8011326b>{smp_really_stop_cpu+9}
RSP: e02b:ffff88002c0bfea8 EFLAGS: 00010002
RAX: ffffffffff578280 RBX: 0000000000000000 RCX: 0000000000000001
RDX: ffffffffff578000 RSI: 0000000000000000 RDI: 0000000000000000
RBP: 0000000000000000 R08: ffff8800240db608 R09: ffff8807c6ca1b48
R10: 000000000000001e R11: 0000000000000246 R12: ffffffff80113262
R13: 0000000000000000 R14: ffff8807c6ca18f8 R15: ffff8807c6ca18f8
FS: 00002aede0373370(0000) GS:ffffffff803fb500(0000) knlGS:0000000000000000
CS: e033 DS: 0000 ES: 0000
Process nautilus (pid: 5972, threadinfo ffff8807c6ca0000, task ffff8807d5120820)
Stack: 0000000000000000 ffffffff801134c8 ffff8807d6a56bc0 0000000000000000
000000000000011f ffffffff8014ee2d ffffffff80405400 0000000000008f80
000000000000011f ffff8807d6a56bc0
Call Trace: <IRQ> <ffffffff801134c8>{smp_call_function_interrupt+79}
<ffffffff8014ee2d>{handle_IRQ_event+78} <ffffffff8014ef10>{__do_IRQ+153}
<ffffffff8010d661>{do_IRQ+60} <ffffffff8023834a>{evtchn_do_upcall+134}
<ffffffff8010b672>{do_hypervisor_callback+30} <EOI>
<ffffffff8010722a>{hypercall_page+554} <ffffffff8010722a>{hypercall_page+554}
<ffffffff80237bfd>{force_evtchn_callback+10} <ffffffff801585f6>{shrink_zone+1774}
<ffffffff802cd0ab>{thread_return+0} <ffffffff80134fb6>{lock_timer_base+27}
<ffffffff80135028>{try_to_del_timer_sync+81} <ffffffff802cd47c>{io_schedule_timeout+50}
<ffffffff801cd57c>{blk_congestion_wait+132} <ffffffff8013fa4d>{autoremove_wake_function+0}
<ffffffff80159715>{try_to_free_pages+342} <ffffffff8015335c>{out_of_memory+368}
<ffffffff80154cdf>{__alloc_pages+422} <ffffffff8015667c>{__do_page_cache_readahead+245}
<ffffffff8015016c>{sync_page+0} <ffffffff802cd4b9>{io_schedule+50}
<ffffffff802cdcfd>{__wait_on_bit_lock+91} <ffffffff80152893>{filemap_nopage+323}
<ffffffff8015d2e3>{__handle_mm_fault+1415} <ffffffff801812b2>{__user_walk_fd+65}
<ffffffff801180d0>{do_page_fault+1938} <ffffffff8018aa7b>{mntput_no_expire+25}
<ffffffff8014dd09>{audit_syscall_exit+747} <ffffffff8010b5c7>{error_exit+0}
Code: f4 eb fd 41 55 49 89 fd 48 c7 c7 c8 c9 3c 80 41 54 55 48 89
RIP <ffffffff8011326b>{smp_really_stop_cpu+9} RSP <ffff88002c0bfea8>
Badness in do_unblank_screen at drivers/char/vt.c:2830
Call Trace: <IRQ> <ffffffff80228114>{do_unblank_screen+76}
<ffffffff8011775b>{bust_spinlocks+28} <ffffffff8010bbbd>{oops_end+21}
<ffffffff8010c25b>{die+58} <ffffffff8010d04a>{do_general_protection+291}
<ffffffff80113262>{smp_really_stop_cpu+0} <ffffffff8010b5c7>{error_exit+0}
<ffffffff80113262>{smp_really_stop_cpu+0} <ffffffff8011326b>{smp_really_stop_cpu+9}
<ffffffff8011326b>{smp_really_stop_cpu+9} <ffffffff801134c8>{smp_call_function_interrupt+79}
<ffffffff8014ee2d>{handle_IRQ_event+78} <ffffffff8014ef10>{__do_IRQ+153}
<ffffffff8010d661>{do_IRQ+60} <ffffffff8023834a>{evtchn_do_upcall+134}
<ffffffff8010b672>{do_hypervisor_callback+30} <EOI>
<ffffffff8010722a>{hypercall_page+554} <ffffffff8010722a>{hypercall_page+554}
<ffffffff80237bfd>{force_evtchn_callback+10} <ffffffff801585f6>{shrink_zone+1774}
<ffffffff802cd0ab>{thread_return+0} <ffffffff80134fb6>{lock_timer_base+27}
<ffffffff80135028>{try_to_del_timer_sync+81} <ffffffff802cd47c>{io_schedule_timeout+50}
<ffffffff801cd57c>{blk_congestion_wait+132} <ffffffff8013fa4d>{autoremove_wake_function+0}
<ffffffff80159715>{try_to_free_pages+342} <ffffffff8015335c>{out_of_memory+368}
<ffffffff80154cdf>{__alloc_pages+422} <ffffffff8015667c>{__do_page_cache_readahead+245}
<ffffffff8015016c>{sync_page+0} <ffffffff802cd4b9>{io_schedule+50}
<ffffffff802cdcfd>{__wait_on_bit_lock+91} <ffffffff80152893>{filemap_nopage+323}
<ffffffff8015d2e3>{__handle_mm_fault+1415} <ffffffff801812b2>{__user_walk_fd+65}
<ffffffff801180d0>{do_page_fault+1938} <ffffffff8018aa7b>{mntput_no_expire+25}
<ffffffff8014dd09>{audit_syscall_exit+747} <ffffffff8010b5c7>{error_exit+0}
<0>Kernel panic - not syncing: Aiee, killing interrupt handler!
<0>general protection fault: 0000 [16] SMP
CPU 2
Modules linked in: ipv6 bridge button battery ac dm_mod e1000 ext3 jbd reiserfs fan thermal processor sg mptspi mptscsih mptbase piix sd_mod scsi_mod
Pid: 5632, comm: master Tainted: GF 2.6.16-xen #1
RIP: e030:[<ffffffff8011326b>] <ffffffff8011326b>{smp_really_stop_cpu+9}
RSP: e02b:ffff88000007bea8 EFLAGS: 00010002
RAX: ffffffffff578080 RBX: 0000000000000000 RCX: 0000000000000001
RDX: ffffffffff578000 RSI: 0000000000000000 RDI: 0000000000000000
RBP: 0000000000000000 R08: ffff880023ff0fc8 R09: ffff8807cf869ae8
R10: 000000000000001d R11: 0000000000000246 R12: ffffffff80113262
R13: 0000000000000000 R14: ffff8807cf869898 R15: ffff8807cf869898
FS: 00002abbf927f370(0000) GS:ffffffff803fb100(0000) knlGS:0000000000000000
CS: e033 DS: 0000 ES: 0000
Process master (pid: 5632, threadinfo ffff8807cf868000, task ffff8807d4d0b040)
Stack: 0000000000000100 ffffffff801134c8 ffff880000044e00 0000000000000000
0000000000000107 ffffffff8014ee2d ffffffff80404800 0000000000008380
0000000000000107 ffff880000044e00
Call Trace: <IRQ> <ffffffff801134c8>{smp_call_function_interrupt+79}
<ffffffff8014ee2d>{handle_IRQ_event+78} <ffffffff8014ef10>{__do_IRQ+153}
<ffffffff8010d661>{do_IRQ+60} <ffffffff8023834a>{evtchn_do_upcall+134}
<ffffffff8010b672>{do_hypervisor_callback+30} <EOI>
<ffffffff8010722a>{hypercall_page+554} <ffffffff8010722a>{hypercall_page+554}
<ffffffff80237bfd>{force_evtchn_callback+10} <ffffffff80158e15>{shrink_zone+3853}
<ffffffff802cd0ab>{thread_return+0} <ffffffff80134fb6>{lock_timer_base+27}
<ffffffff80135028>{try_to_del_timer_sync+81} <ffffffff802cd47c>{io_schedule_timeout+50}
<ffffffff801cd57c>{blk_congestion_wait+132} <ffffffff8013fa4d>{autoremove_wake_function+0}
<ffffffff80159715>{try_to_free_pages+342} <ffffffff80154cdf>{__alloc_pages+422}
<ffffffff802c8ed4>{unix_write_space+69} <ffffffff80167143>{read_swap_cache_async+66}
<ffffffff8015b333>{swapin_readahead+78} <ffffffff8015d670>{__handle_mm_fault+2324}
<ffffffff8015499e>{get_page_from_freelist+561} <ffffffff801180d0>{do_page_fault+1938}
<ffffffff80135028>{try_to_del_timer_sync+81} <ffffffff8013503d>{del_timer_sync+12}
<ffffffff8015407e>{free_hot_cold_page+234} <ffffffff8010b5c7>{error_exit+0}
<ffffffff802c644c>{unix_poll+0} <ffffffff801dac33>{copy_user_generic+147}
<ffffffff80183b84>{sys_select+788} <ffffffff8010b05c>{tracesys+167}
Code: f4 eb fd 41 55 49 89 fd 48 c7 c7 c8 c9 3c 80 41 54 55 48 89
RIP <ffffffff8011326b>{smp_really_stop_cpu+9} RSP <ffff88000007bea8>
<0>Kernel panic - not syncing: Aiee, killing interrupt handler!
<0>general protection fault: 0000 [17] SMP
CPU 30
Modules linked in: ipv6 bridge button battery ac dm_mod e1000 ext3 jbd reiserfs fan thermal processor sg mptspi mptscsih mptbase piix sd_mod scsi_mod
Pid: 6296, comm: mono Tainted: GF 2.6.16-xen #1
RIP: e030:[<ffffffff8011326b>] <ffffffff8011326b>{smp_really_stop_cpu+9}
RSP: e02b:ffff880000cf3ea8 EFLAGS: 00010002
RAX: ffffffffff578780 RBX: 0000000000000000 RCX: 0000000000000001
RDX: ffffffffff578000 RSI: 0000000000000000 RDI: 0000000000000000
RBP: 0000000000000000 R08: ffff8807bea56000 R09: 000000000000001e
R10: ffff8807d4bf7b70 R11: 0000000000000246 R12: ffffffff80113262
R13: 0000000000000000 R14: ffff8807bea579e8 R15: ffff8807bea579e8
FS: 00002b660bc42ce0(0063) GS:ffffffff803fbf00(0000) knlGS:0000000000000000
CS: e033 DS: 0000 ES: 0000
Process mono (pid: 6296, threadinfo ffff8807bea56000, task ffff8807bd5db8a0)
Stack: 0000000000000100 ffffffff801134c8 ffff880000cbfd80 0000000000000000
000000000000015b ffffffff8014ee2d ffffffff80407200 000000000000ad80
000000000000015b ffff880000cbfd80
Call Trace: <IRQ> <ffffffff801134c8>{smp_call_function_interrupt+79}
<ffffffff8014ee2d>{handle_IRQ_event+78} <ffffffff8014ef10>{__do_IRQ+153}
<ffffffff8010d661>{do_IRQ+60} <ffffffff8023834a>{evtchn_do_upcall+134}
<ffffffff8010b672>{do_hypervisor_callback+30} <EOI>
<ffffffff8010722a>{hypercall_page+554} <ffffffff8010722a>{hypercall_page+554}
<ffffffff80237bfd>{force_evtchn_callback+10} <ffffffff802cd129>{thread_return+126}
<ffffffff801daa07>{find_next_bit+85} <ffffffff8012408f>{move_tasks+183}
<ffffffff80134fb6>{lock_timer_base+27} <ffffffff802cdc55>{schedule_timeout+138}
<ffffffff80134b67>{process_timeout+0} <ffffffff802cd47c>{io_schedule_timeout+50}
<ffffffff801cd56e>{blk_congestion_wait+118} <ffffffff801564bf>{pdflush_operation+71}
<ffffffff8013fa4d>{autoremove_wake_function+0} <ffffffff801597ba>{try_to_free_pages+507}
<ffffffff8015335c>{out_of_memory+368} <ffffffff80154cdf>{__alloc_pages+422}
<ffffffff80167143>{read_swap_cache_async+66} <ffffffff8015b333>{swapin_readahead+78}
<ffffffff8015d670>{__handle_mm_fault+2324} <ffffffff801180d0>{do_page_fault+1938}
<ffffffff802ce0ad>{schedule_hrtimer+49} <ffffffff80142162>{hrtimer_nanosleep+76}
<ffffffff8014da85>{audit_syscall_exit+103} <ffffffff8010b5c7>{error_exit+0}
Code: f4 eb fd 41 55 49 89 fd 48 c7 c7 c8 c9 3c 80 41 54 55 48 89
RIP <ffffffff8011326b>{smp_really_stop_cpu+9} RSP <ffff880000cf3ea8>
<0>Kernel panic - not syncing: Aiee, killing interrupt handler!
<0>general protection fault: 0000 [18] SMP
CPU 28
Modules linked in: ipv6 bridge button battery ac dm_mod e1000 ext3 jbd reiserfs fan thermal processor sg mptspi mptscsih mptbase piix sd_mod scsi_mod
Pid: 5346, comm: xinetd Tainted: GF 2.6.16-xen #1
RIP: e030:[<ffffffff8011326b>] <ffffffff8011326b>{smp_really_stop_cpu+9}
RSP: e02b:ffff880000cbbea8 EFLAGS: 00010002
RAX: ffffffffff578700 RBX: 0000000000000000 RCX: 0000000000000001
RDX: ffffffffff578000 RSI: 0000000000000000 RDI: 0000000000000000
RBP: 0000000000000000 R08: 00000000ffffffff R09: ffff8807d0109a28
R10: 0000000000000000 R11: 0000000000000246 R12: ffffffff80113262
R13: 0000000000000000 R14: ffff8807d01097d8 R15: ffff8807d01097d8
FS: 00002b46375cfae0(0000) GS:ffffffff803fbe00(0000) knlGS:0000000000000000
CS: e033 DS: 0000 ES: 0000
Process xinetd (pid: 5346, threadinfo ffff8807d0108000, task ffff8807d6886140)
Stack: 0000000000000040 ffffffff801134c8 ffff8800041a9300 0000000000000000
0000000000000155 ffffffff8014ee2d ffffffff80406f00 000000000000aa80
0000000000000155 ffff8800041a9300
Call Trace: <IRQ> <ffffffff801134c8>{smp_call_function_interrupt+79}
<ffffffff8014ee2d>{handle_IRQ_event+78} <ffffffff8014ef10>{__do_IRQ+153}
<ffffffff8010d661>{do_IRQ+60} <ffffffff8023834a>{evtchn_do_upcall+134}
<ffffffff8010b672>{do_hypervisor_callback+30} <EOI>
<ffffffff8010722a>{hypercall_page+554} <ffffffff8010722a>{hypercall_page+554}
<ffffffff80237bfd>{force_evtchn_callback+10} <ffffffff801585f6>{shrink_zone+1774}
<ffffffff802cd0ab>{thread_return+0} <ffffffff80134fb6>{lock_timer_base+27}
<ffffffff80135028>{try_to_del_timer_sync+81} <ffffffff8013503d>{del_timer_sync+12}
<ffffffff802cdc5d>{schedule_timeout+146} <ffffffff80134b67>{process_timeout+0}
<ffffffff802cd47c>{io_schedule_timeout+50} <ffffffff801cd57c>{blk_congestion_wait+132}
<ffffffff8013fa4d>{autoremove_wake_function+0} <ffffffff80159715>{try_to_free_pages+342}
<ffffffff8015335c>{out_of_memory+368} <ffffffff80154cdf>{__alloc_pages+422}
<ffffffff80167143>{read_swap_cache_async+66} <ffffffff8015b333>{swapin_readahead+78}
<ffffffff8015d670>{__handle_mm_fault+2324} <ffffffff801d9298>{__up_read+19}
<ffffffff801180d0>{do_page_fault+1938} <ffffffff802cd0ab>{thread_return+0}
<ffffffff8010b5c7>{error_exit+0} <ffffffff8010a400>{do_signal+630}
<ffffffff8010a38e>{do_signal+516} <ffffffff8014da85>{audit_syscall_exit+103}
<ffffffff8010af9d>{sysret_signal+56} <ffffffff8010b271>{ptregscall_common+61}
Code: f4 eb fd 41 55 49 89 fd 48 c7 c7 c8 c9 3c 80 41 54 55 48 89
RIP <ffffffff8011326b>{smp_really_stop_cpu+9} RSP <ffff880000cbbea8>
<0>Kernel panic - not syncing: Aiee, killing interrupt handler!
<0>general protection fault: 0000 [19] SMP
CPU 18
Modules linked in: ipv6 bridge button battery ac dm_mod e1000 ext3 jbd reiserfs fan thermal processor sg mptspi mptscsih mptbase piix sd_mod scsi_mod
Pid: 6305, comm: gnome-power-man Tainted: GF 2.6.16-xen #1
RIP: e030:[<ffffffff8011326b>] <ffffffff8011326b>{smp_really_stop_cpu+9}
RSP: e02b:ffff880004963ea8 EFLAGS: 00010002
RAX: ffffffffff578480 RBX: 0000000000000000 RCX: 0000000000000001
RDX: ffffffffff578000 RSI: 0000000000000000 RDI: 0000000000000000
RBP: 0000000000000000 R08: ffff880023bec1f8 R09: ffff8807c0925b48
R10: 0000000000000020 R11: 0000000000000246 R12: ffffffff80113262
R13: 0000000000000000 R14: ffff8807c09258f8 R15: ffff8807c09258f8
FS: 00002afcbc5701f0(0000) GS:ffffffff803fb900(0000) knlGS:0000000000000000
CS: e033 DS: 0000 ES: 0000
Process gnome-power-man (pid: 6305, threadinfo ffff8807c0924000, task ffff8807be9db8a0)
Stack: 0000000000000000 ffffffff801134c8 ffff88002c0f1980 0000000000000000
0000000000000137 ffffffff8014ee2d ffffffff80406000 0000000000009b80
0000000000000137 ffff88002c0f1980
Call Trace: <IRQ> <ffffffff801134c8>{smp_call_function_interrupt+79}
<ffffffff8014ee2d>{handle_IRQ_event+78} <ffffffff8014ef10>{__do_IRQ+153}
<ffffffff8010d661>{do_IRQ+60} <ffffffff8023834a>{evtchn_do_upcall+134}
<ffffffff8010b672>{do_hypervisor_callback+30} <EOI>
<ffffffff8010722a>{hypercall_page+554} <ffffffff8010722a>{hypercall_page+554}
<ffffffff80237bfd>{force_evtchn_callback+10} <ffffffff801585f6>{shrink_zone+1774}
<ffffffff802cd0ab>{thread_return+0} <ffffffff80134fb6>{lock_timer_base+27}
<ffffffff80135028>{try_to_del_timer_sync+81} <ffffffff802cdc5d>{schedule_timeout+146}
<ffffffff80134b67>{process_timeout+0} <ffffffff801cd57c>{blk_congestion_wait+132}
<ffffffff8013fa4d>{autoremove_wake_function+0} <ffffffff80159715>{try_to_free_pages+342}
<ffffffff80154cdf>{__alloc_pages+422} <ffffffff8015667c>{__do_page_cache_readahead+245}
<ffffffff8015016c>{sync_page+0} <ffffffff802cd4b9>{io_schedule+50}
<ffffffff802cdcfd>{__wait_on_bit_lock+91} <ffffffff80152893>{filemap_nopage+323}
<ffffffff8015d2e3>{__handle_mm_fault+1415} <ffffffff801180d0>{do_page_fault+1938}
<ffffffff8016d9b0>{__kmalloc+69} <ffffffff8014da85>{audit_syscall_exit+103}
<ffffffff8010b5c7>{error_exit+0}
Code: f4 eb fd 41 55 49 89 fd 48 c7 c7 c8 c9 3c 80 41 54 55 48 89
RIP <ffffffff8011326b>{smp_really_stop_cpu+9} RSP <ffff880004963ea8>
<0>Kernel panic - not syncing: Aiee, killing interrupt handler!
<0>general protection fault: 0000 [20] SMP
CPU 16
Modules linked in: ipv6 bridge button battery ac dm_mod e1000 ext3 jbd reiserfs fan thermal processor sg mptspi mptscsih mptbase piix sd_mod scsi_mod
Pid: 0, comm: swapper Tainted: GF 2.6.16-xen #1
RIP: e030:[<ffffffff8011326b>] <ffffffff8011326b>{smp_really_stop_cpu+9}
RSP: e02b:ffff880004947ea8 EFLAGS: 00010002
RAX: ffffffffff578400 RBX: 0000000000000000 RCX: 0000000000000001
RDX: ffffffffff578000 RSI: 0000000000000002 RDI: 0000000000000000
RBP: 0000000000000000 R08: 0000000100073335 R09: 0000000000000010
R10: 0000000100073398 R11: 0000000000000246 R12: ffffffff80113262
R13: 0000000000000000 R14: ffff880000c4de58 R15: ffff880000c4de58
FS: 00002b76e9a3c1b0(0000) GS:ffffffff803fb800(0000) knlGS:0000000000000000
CS: e033 DS: 002b ES: 002b
Process swapper (pid: 0, threadinfo ffff880000c4c000, task ffff8800049a97e0)
Stack: ffffffff80113262 ffffffff801134c8 ffff88002c0f1dc0 0000000000000000
0000000000000131 ffffffff8014ee2d ffffffff80405d00 0000000000009880
0000000000000131 ffff88002c0f1dc0
Call Trace: <IRQ> <ffffffff80113262>{smp_really_stop_cpu+0}
<ffffffff801134c8>{smp_call_function_interrupt+79} <ffffffff8014ee2d>{handle_IRQ_event+78}
<ffffffff8014ef10>{__do_IRQ+153} <ffffffff8010d661>{do_IRQ+60}
<ffffffff8023834a>{evtchn_do_upcall+134} <ffffffff8010b672>{do_hypervisor_callback+30} <EOI>
<ffffffff801073aa>{hypercall_page+938} <ffffffff801073aa>{hypercall_page+938}
<ffffffff801095de>{xen_idle+132} <ffffffff80109933>{cpu_idle+217}
Code: f4 eb fd 41 55 49 89 fd 48 c7 c7 c8 c9 3c 80 41 54 55 48 89
RIP <ffffffff8011326b>{smp_really_stop_cpu+9} RSP <ffff880004947ea8>
Badness in do_unblank_screen at drivers/char/vt.c:2830
Call Trace: <IRQ> <ffffffff80228114>{do_unblank_screen+76}
<ffffffff8011775b>{bust_spinlocks+28} <ffffffff8010bbbd>{oops_end+21}
<ffffffff8010c25b>{die+58} <ffffffff8010d04a>{do_general_protection+291}
<ffffffff80113262>{smp_really_stop_cpu+0} <ffffffff8010b5c7>{error_exit+0}
<ffffffff80113262>{smp_really_stop_cpu+0} <ffffffff8011326b>{smp_really_stop_cpu+9}
<ffffffff8011326b>{smp_really_stop_cpu+9} <ffffffff80113262>{smp_really_stop_cpu+0}
<ffffffff801134c8>{smp_call_function_interrupt+79} <ffffffff8014ee2d>{handle_IRQ_event+78}
<ffffffff8014ef10>{__do_IRQ+153} <ffffffff8010d661>{do_IRQ+60}
<ffffffff8023834a>{evtchn_do_upcall+134} <ffffffff8010b672>{do_hypervisor_callback+30} <EOI>
<ffffffff801073aa>{hypercall_page+938} <ffffffff801073aa>{hypercall_page+938}
<ffffffff801095de>{xen_idle+132} <ffffffff80109933>{cpu_idle+217}
<0>Kernel panic - not syncing: Aiee, killing interrupt handler!
<0>general protection fault: 0000 [21] SMP
CPU 20
Modules linked in: ipv6 bridge button battery ac dm_mod e1000 ext3 jbd reiserfs fan thermal processor sg mptspi mptscsih mptbase piix sd_mod scsi_mod
Pid: 6283, comm: gnome-panel Tainted: GF 2.6.16-xen #1
RIP: e030:[<ffffffff8011326b>] <ffffffff8011326b>{smp_really_stop_cpu+9}
RSP: e02b:ffff880004197ea8 EFLAGS: 00010002
RAX: ffffffffff578500 RBX: 0000000000000000 RCX: 0000000000000001
RDX: ffffffffff578000 RSI: 0000000000000000 RDI: 0000000000000000
RBP: 0000000000000000 R08: ffff8807c2b699d8 R09: 0000000000000010
R10: ffff88002c13e2d8 R11: 0000000000000246 R12: ffffffff80113262
R13: 0000000000000000 R14: ffff8807c2b699d8 R15: ffff8807c2b699d8
FS: 00002b2e6d783e30(0000) GS:ffffffff803fba00(0000) knlGS:0000000000000000
CS: e033 DS: 0000 ES: 0000
Process gnome-panel (pid: 6283, threadinfo ffff8807c2b68000, task ffff8807c261b8a0)
Stack: 0000000000000000 ffffffff801134c8 ffff88002c0f1540 0000000000000000
000000000000013d ffffffff8014ee2d ffffffff80406300 0000000000009e80
000000000000013d ffff88002c0f1540
Call Trace: <IRQ> <ffffffff801134c8>{smp_call_function_interrupt+79}
<ffffffff8014ee2d>{handle_IRQ_event+78} <ffffffff8014ef10>{__do_IRQ+153}
<ffffffff8010d661>{do_IRQ+60} <ffffffff8023834a>{evtchn_do_upcall+134}
<ffffffff8010b672>{do_hypervisor_callback+30} <EOI>
<ffffffff8010722a>{hypercall_page+554} <ffffffff8010722a>{hypercall_page+554}
<ffffffff80237bfd>{force_evtchn_callback+10} <ffffffff80158e84>{shrink_zone+3964}
<ffffffff802cd0ab>{thread_return+0} <ffffffff80134fb6>{lock_timer_base+27}
<ffffffff80135028>{try_to_del_timer_sync+81} <ffffffff802cd47c>{io_schedule_timeout+50}
<ffffffff801cd57c>{blk_congestion_wait+132} <ffffffff8013fa4d>{autoremove_wake_function+0}
<ffffffff80159715>{try_to_free_pages+342} <ffffffff8015016c>{sync_page+0}
<ffffffff80154cdf>{__alloc_pages+422} <ffffffff80167143>{read_swap_cache_async+66}
<ffffffff8015b333>{swapin_readahead+78} <ffffffff8015d670>{__handle_mm_fault+2324}
<ffffffff801180d0>{do_page_fault+1938} <ffffffff801830b3>{poll_freewait+64}
<ffffffff8014da85>{audit_syscall_exit+103} <ffffffff8010b5c7>{error_exit+0}
Code: f4 eb fd 41 55 49 89 fd 48 c7 c7 c8 c9 3c 80 41 54 55 48 89
RIP <ffffffff8011326b>{smp_really_stop_cpu+9} RSP <ffff880004197ea8>
<0>Kernel panic - not syncing: Aiee, killing interrupt handler!
<1>Unable to handle kernel NULL pointer dereference at 0000000000000001 RIP:
[<0000000000000001>]
PGD 7d230f067 PUD 7d2b0e067 PMD 0
Oops: 0010 [22] SMP
CPU 29
Modules linked in: ipv6 bridge button battery ac dm_mod e1000 ext3 jbd reiserfs fan thermal processor sg mptspi mptscsih mptbase piix sd_mod scsi_mod
Pid: 5941, comm: gpg-agent Tainted: GF 2.6.16-xen #1
RIP: e030:[<0000000000000001>] [<0000000000000001>]
RSP: e02b:ffff880000ce7eb0 EFLAGS: 00010006
RAX: ffff8807c7873fd8 RBX: ffffffff8011011e RCX: 0000000000000158
RDX: 0000000000001735 RSI: 0000000000000000 RDI: ffffffff8011011e
RBP: 0000000000000000 R08: ffff88002405d7c8 R09: ffff8807c7873c28
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000001
R13: 0000000000000000 R14: ffff8807c78739d8 R15: ffff8807c78739d8
FS: 00002b5688465fc0(0000) GS:ffffffff803fbe80(0000) knlGS:0000000000000000
CS: e033 DS: 0000 ES: 0000
Process gpg-agent (pid: 5941, threadinfo ffff8807c7872000, task ffff8800006448a0)
Stack: ffffffff801134c8 ffff880000cbffc0 0000000000000000 0000000000000158
ffffffff8014ee2d ffffffff80407080 000000000000ac00 0000000000000158
ffff880000cbffc0 ffffffff804070a8
Call Trace: <IRQ> <ffffffff801134c8>{smp_call_function_interrupt+79}
<ffffffff8014ee2d>{handle_IRQ_event+78} <ffffffff8014ef10>{__do_IRQ+153}
<ffffffff8010d661>{do_IRQ+60} <ffffffff8023834a>{evtchn_do_upcall+134}
<ffffffff8010b672>{do_hypervisor_callback+30} <EOI>
<ffffffff8010722a>{hypercall_page+554} <ffffffff8010722a>{hypercall_page+554}
<ffffffff80237bfd>{force_evtchn_callback+10} <ffffffff801585f6>{shrink_zone+1774}
<ffffffff802cd0ab>{thread_return+0} <ffffffff8019f6c7>{mb_cache_shrink_fn+36}
<ffffffff80159715>{try_to_free_pages+342} <ffffffff8015335c>{out_of_memory+368}
<ffffffff80154cdf>{__alloc_pages+422} <ffffffff80167143>{read_swap_cache_async+66}
<ffffffff8015b333>{swapin_readahead+78} <ffffffff8015d670>{__handle_mm_fault+2324}
<ffffffff801180d0>{do_page_fault+1938} <ffffffff8014da85>{audit_syscall_exit+103}
<ffffffff8010b5c7>{error_exit+0}
Code: Bad RIP value.
RIP [<0000000000000001>] RSP <ffff880000ce7eb0>
CR2: 0000000000000001
<0>Kernel panic - not syncing: Aiee, killing interrupt handler!
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
general protection fault: 0000 [23] SMP
CPU 24
Modules linked in: ipv6 bridge button battery ac dm_mod e1000 ext3 jbd reiserfs fan thermal processor sg mptspi mptscsih mptbase piix sd_mod scsi_mod
Pid: 5900, comm: gnome-session Tainted: GF 2.6.16-xen #1
RIP: e030:[<ffffffff8011326b>] <ffffffff8011326b>{smp_really_stop_cpu+9}
RSP: e02b:ffff8800041ebea8 EFLAGS: 00010002
RAX: ffffffffff578600 RBX: 0000000000000000 RCX: 0000000000000001
RDX: ffffffffff578000 RSI: 0000000000000000 RDI: 0000000000000000
RBP: 0000000000000000 R08: ffffffff80333000 R09: 00000000007d72ca
R10: ffff8800089ee000 R11: 0000000000000000 R12: ffffffff80113262
R13: 0000000000000000 R14: ffff8807cfb29b28 R15: ffff8807cfb29b28
FS: 00002b2d4633aa60(0000) GS:ffffffff803fbc00(0000) knlGS:0000000000000000
CS: e033 DS: 0000 ES: 0000
Process gnome-session (pid: 5900, threadinfo ffff8807cfb28000, task ffff8807d3d7f180)
Stack: ffff8800010e6000 ffffffff801134c8 ffff8800041a9b80 0000000000000000
0000000000000149 ffffffff8014ee2d ffffffff80406900 000000000000a480
0000000000000149 ffff8800041a9b80
Call Trace: <IRQ> <ffffffff801134c8>{smp_call_function_interrupt+79}
<ffffffff8014ee2d>{handle_IRQ_event+78} <ffffffff8014ef10>{__do_IRQ+153}
<ffffffff8010d661>{do_IRQ+60} <ffffffff8023834a>{evtchn_do_upcall+134}
<ffffffff8010b672>{do_hypervisor_callback+30} <EOI>
<ffffffff80117379>{show_mem+132} <ffffffff8011732c>{show_mem+55}
<ffffffff80153221>{out_of_memory+53} <ffffffff80154d56>{__alloc_pages+541}
<ffffffff8015667c>{__do_page_cache_readahead+245} <ffffffff8015016c>{sync_page+0}
<ffffffff802cd4b9>{io_schedule+50} <ffffffff802cdcfd>{__wait_on_bit_lock+91}
<ffffffff80152893>{filemap_nopage+323} <ffffffff8015d2e3>{__handle_mm_fault+1415}
<ffffffff801180d0>{do_page_fault+1938} <ffffffff801830b3>{poll_freewait+64}
<ffffffff8014da85>{audit_syscall_exit+103} <ffffffff8010b5c7>{error_exit+0}
Code: f4 eb fd 41 55 49 89 fd 48 c7 c7 c8 c9 3c 80 41 54 55 48 89
RIP <ffffffff8011326b>{smp_really_stop_cpu+9} RSP <ffff8800041ebea8>
<0>Kernel panic - not syncing: Aiee, killing interrupt handler!
<0>general protection fault: 0000 [24] SMP
CPU 31
Modules linked in: ipv6 bridge button battery ac dm_mod e1000 ext3 jbd reiserfs fan thermal processor sg mptspi mptscsih mptbase piix sd_mod scsi_mod
Pid: 0, comm: swapper Tainted: GF 2.6.16-xen #1
RIP: e030:[<ffffffff8011326b>] <ffffffff8011326b>{smp_really_stop_cpu+9}
RSP: e02b:ffff880000cffea8 EFLAGS: 00010002
RAX: ffffffffff5787c0 RBX: 0000000000000000 RCX: 0000000000000001
RDX: ffffffffff578000 RSI: 0000000000000002 RDI: 0000000000000000
RBP: 0000000000000000 R08: 000000010007333b R09: 000000000000001f
R10: 0000000100073384 R11: 0000000000000246 R12: ffffffff80113262
R13: 0000000000000000 R14: ffff880000055e58 R15: ffff880000055e58
FS: 00002b2e6d783e30(0000) GS:ffffffff803fbf80(0000) knlGS:0000000000000000
CS: e033 DS: 002b ES: 002b
Process swapper (pid: 0, threadinfo ffff880000054000, task ffff880000c74100)
Stack: ffffffff80113262 ffffffff801134c8 ffff880000cbfb80 0000000000000000
000000000000015e ffffffff8014ee2d ffffffff80407380 000000000000af00
000000000000015e ffff880000cbfb80
Call Trace: <IRQ> <ffffffff80113262>{smp_really_stop_cpu+0}
<ffffffff801134c8>{smp_call_function_interrupt+79} <ffffffff8014ee2d>{handle_IRQ_event+78}
<ffffffff8014ef10>{__do_IRQ+153} <ffffffff8010d661>{do_IRQ+60}
<ffffffff8023834a>{evtchn_do_upcall+134} <ffffffff8010b672>{do_hypervisor_callback+30} <EOI>
<ffffffff801073aa>{hypercall_page+938} <ffffffff801073aa>{hypercall_page+938}
<ffffffff801095de>{xen_idle+132} <ffffffff80109933>{cpu_idle+217}
Code: f4 eb fd 41 55 49 89 fd 48 c7 c7 c8 c9 3c 80 41 54 55 48 89
RIP <ffffffff8011326b>{smp_really_stop_cpu+9} RSP <ffff880000cffea8>
<0>Kernel panic - not syncing: Aiee, killing interrupt handler!
<0>general protection fault: 0000 [25] SMP
CPU 25
Modules linked in: ipv6 bridge button battery ac dm_mod e1000 ext3 jbd reiserfs fan thermal processor sg mptspi mptscsih mptbase piix sd_mod scsi_mod
Pid: 6086, comm: gnome-terminal Tainted: GF 2.6.16-xen #1
RIP: e030:[<ffffffff8011326b>] <ffffffff8011326b>{smp_really_stop_cpu+9}
RSP: e02b:ffff8800041f7ea8 EFLAGS: 00010002
RAX: ffffffffff578640 RBX: 0000000000000000 RCX: 0000000000000001
RDX: ffffffffff578000 RSI: 0000000000000000 RDI: 0000000000000000
RBP: 0000000000000000 R08: ffff880023f3c0d8 R09: ffff8807bde15c28
R10: 0000000000000002 R11: 0000000000000246 R12: ffffffff80113262
R13: 0000000000000000 R14: ffff8807bde159d8 R15: ffff8807bde159d8
FS: 00002ae071177fc0(0000) GS:ffffffff803fbc80(0000) knlGS:0000000000000000
CS: e033 DS: 0000 ES: 0000
Process gnome-terminal (pid: 6086, threadinfo ffff8807bde14000, task ffff880000710040)
Stack: 0000000000000040 ffffffff801134c8 ffff8800041a9980 0000000000000000
000000000000014c ffffffff8014ee2d ffffffff80406a80 000000000000a600
000000000000014c ffff8800041a9980
Call Trace: <IRQ> <ffffffff801134c8>{smp_call_function_interrupt+79}
<ffffffff8014ee2d>{handle_IRQ_event+78} <ffffffff8014ef10>{__do_IRQ+153}
<ffffffff8010d661>{do_IRQ+60} <ffffffff8023834a>{evtchn_do_upcall+134}
<ffffffff8010b672>{do_hypervisor_callback+30} <EOI>
<ffffffff8010722a>{hypercall_page+554} <ffffffff8010722a>{hypercall_page+554}
<ffffffff80237bfd>{force_evtchn_callback+10} <ffffffff801585f6>{shrink_zone+1774}
<ffffffff802cd0ab>{thread_return+0} <ffffffff801cba4c>{generic_make_request+339}
<ffffffff80134fb6>{lock_timer_base+27} <ffffffff80135028>{try_to_del_timer_sync+81}
<ffffffff802cd47c>{io_schedule_timeout+50} <ffffffff801cd57c>{blk_congestion_wait+132}
<ffffffff8013fa4d>{autoremove_wake_function+0} <ffffffff80159715>{try_to_free_pages+342}
<ffffffff8013fa56>{autoremove_wake_function+9} <ffffffff80154cdf>{__alloc_pages+422}
<ffffffff80167143>{read_swap_cache_async+66} <ffffffff8015b333>{swapin_readahead+78}
<ffffffff8015d670>{__handle_mm_fault+2324} <ffffffff801180d0>{do_page_fault+1938}
<ffffffff801830b3>{poll_freewait+64} <ffffffff8014da85>{audit_syscall_exit+103}
<ffffffff8010b5c7>{error_exit+0}
Code: f4 eb fd 41 55 49 89 fd 48 c7 c7 c8 c9 3c 80 41 54 55 48 89
RIP <ffffffff8011326b>{smp_really_stop_cpu+9} RSP <ffff8800041f7ea8>
<0>Kernel panic - not syncing: Aiee, killing interrupt handler!
<0>general protection fault: 0000 [26] SMP
CPU 22
Modules linked in: ipv6 bridge button battery ac dm_mod e1000 ext3 jbd reiserfs fan thermal processor sg mptspi mptscsih mptbase piix sd_mod scsi_mod
Pid: 6275, comm: gnome-settings- Tainted: GF 2.6.16-xen #1
RIP: e030:[<ffffffff8011326b>] <ffffffff8011326b>{smp_really_stop_cpu+9}
RSP: e02b:ffff8800041d3ea8 EFLAGS: 00010002
RAX: ffffffffff578580 RBX: 0000000000000000 RCX: 0000000000000001
RDX: ffffffffff578000 RSI: 0000000000000000 RDI: 0000000000000000
RBP: 0000000000000000 R08: ffff880023b0b1c8 R09: ffff8807bc82fb48
R10: 0000000000000012 R11: 0000000000000246 R12: ffffffff80113262
R13: 0000000000000000 R14: ffff8807bc82f8f8 R15: ffff8807bc82f8f8
FS: 00002aae5150ffc0(0000) GS:ffffffff803fbb00(0000) knlGS:0000000000000000
CS: e033 DS: 0000 ES: 0000
Process gnome-settings- (pid: 6275, threadinfo ffff8807bc82e000, task ffff8807d0f0e140)
Stack: 0000000000000040 ffffffff801134c8 ffff8800041a9fc0 0000000000000000
0000000000000143 ffffffff8014ee2d ffffffff80406600 000000000000a180
0000000000000143 ffff8800041a9fc0
Call Trace: <IRQ> <ffffffff801134c8>{smp_call_function_interrupt+79}
<ffffffff8014ee2d>{handle_IRQ_event+78} <ffffffff8014ef10>{__do_IRQ+153}
<ffffffff8010d661>{do_IRQ+60} <ffffffff8023834a>{evtchn_do_upcall+134}
<ffffffff8010b672>{do_hypervisor_callback+30} <EOI>
<ffffffff8010722a>{hypercall_page+554} <ffffffff8010722a>{hypercall_page+554}
<ffffffff80237bfd>{force_evtchn_callback+10} <ffffffff801585f6>{shrink_zone+1774}
<ffffffff802cd0ab>{thread_return+0} <ffffffff80113fc1>{xen_send_IPI_mask+0}
<ffffffff80134fb6>{lock_timer_base+27} <ffffffff80135028>{try_to_del_timer_sync+81}
<ffffffff801cd57c>{blk_congestion_wait+132} <ffffffff8013fa4d>{autoremove_wake_function+0}
<ffffffff80159715>{try_to_free_pages+342} <ffffffff8015335c>{out_of_memory+368}
<ffffffff80154cdf>{__alloc_pages+422} <ffffffff8015667c>{__do_page_cache_readahead+245}
<ffffffff8015016c>{sync_page+0} <ffffffff802cd4b9>{io_schedule+50}
<ffffffff802cdcfd>{__wait_on_bit_lock+91} <ffffffff80152893>{filemap_nopage+323}
<ffffffff8015d2e3>{__handle_mm_fault+1415} <ffffffff801180d0>{do_page_fault+1938}
<ffffffff801830b3>{poll_freewait+64} <ffffffff8014da85>{audit_syscall_exit+103}
<ffffffff8010b5c7>{error_exit+0}
Code: f4 eb fd 41 55 49 89 fd 48 c7 c7 c8 c9 3c 80 41 54 55 48 89
RIP <ffffffff8011326b>{smp_really_stop_cpu+9} RSP <ffff8800041d3ea8>
<0>Kernel panic - not syncing: Aiee, killing interrupt handler!
<0>general protection fault: 0000 [27] SMP
CPU 17
Modules linked in: ipv6 bridge button battery ac dm_mod e1000 ext3 jbd reiserfs fan thermal processor sg mptspi mptscsih mptbase piix sd_mod scsi_mod
Pid: 0, comm: swapper Tainted: GF 2.6.16-xen #1
RIP: e030:[<ffffffff8011326b>] <ffffffff8011326b>{smp_really_stop_cpu+9}
RSP: e02b:ffff880004953ea8 EFLAGS: 00010002
RAX: ffffffffff578440 RBX: 0000000000000000 RCX: 0000000000000001
RDX: ffffffffff578000 RSI: 0000000000000002 RDI: 0000000000000000
RBP: 0000000000000000 R08: 000000010007334e R09: 0000000000000011
R10: 00000001000733af R11: 0000000000000246 R12: ffffffff80113262
R13: 0000000000000000 R14: ffff880000c51e58 R15: ffff880000c51e58
FS: 00002b103f96a490(0000) GS:ffffffff803fb880(0000) knlGS:0000000000000000
CS: e033 DS: 002b ES: 002b
Process swapper (pid: 0, threadinfo ffff880000c50000, task ffff8800049a90c0)
Stack: ffffffff80113262 ffffffff801134c8 ffff88002c0f1bc0 0000000000000000
0000000000000134 ffffffff8014ee2d ffffffff80405e80 0000000000009a00
0000000000000134 ffff88002c0f1bc0
Call Trace: <IRQ> <ffffffff80113262>{smp_really_stop_cpu+0}
<ffffffff801134c8>{smp_call_function_interrupt+79} <ffffffff8014ee2d>{handle_IRQ_event+78}
<ffffffff8014ef10>{__do_IRQ+153} <ffffffff8010d661>{do_IRQ+60}
<ffffffff8023834a>{evtchn_do_upcall+134} <ffffffff8010b672>{do_hypervisor_callback+30} <EOI>
<ffffffff801073aa>{hypercall_page+938} <ffffffff801073aa>{hypercall_page+938}
<ffffffff801095de>{xen_idle+132} <ffffffff80109933>{cpu_idle+217}
Code: f4 eb fd 41 55 49 89 fd 48 c7 c7 c8 c9 3c 80 41 54 55 48 89
RIP <ffffffff8011326b>{smp_really_stop_cpu+9} RSP <ffff880004953ea8>
<0>Kernel panic - not syncing: Aiee, killing interrupt handler!
<0>general protection fault: 0000 [28] SMP
CPU 21
Modules linked in: ipv6 bridge button battery ac dm_mod e1000 ext3 jbd reiserfs fan thermal processor sg mptspi mptscsih mptbase piix sd_mod scsi_mod
Pid: 0, comm: swapper Tainted: GF 2.6.16-xen #1
RIP: e030:[<ffffffff8011326b>] <ffffffff8011326b>{smp_really_stop_cpu+9}
RSP: e02b:ffff8800041a3ea8 EFLAGS: 00010002
RAX: ffffffffff578540 RBX: 0000000000000000 RCX: 0000000000000001
RDX: ffffffffff578000 RSI: 0000000000000002 RDI: 0000000000000000
RBP: 0000000000000000 R08: 000000010007333b R09: 0000000000000015
R10: 000000010007339e R11: 0000000000000246 R12: ffffffff80113262
R13: 0000000000000000 R14: ffff880000c5be58 R15: ffff880000c5be58
FS: 00002aab92f75d80(0000) GS:ffffffff803fba80(0000) knlGS:0000000000000000
CS: e033 DS: 002b ES: 002b
Process swapper (pid: 0, threadinfo ffff880000c5a000, task ffff8800049b0140)
Stack: ffffffff80113262 ffffffff801134c8 ffff88002c0f1340 0000000000000000
0000000000000140 ffffffff8014ee2d ffffffff80406480 000000000000a000
0000000000000140 ffff88002c0f1340
Call Trace: <IRQ> <ffffffff80113262>{smp_really_stop_cpu+0}
<ffffffff801134c8>{smp_call_function_interrupt+79} <ffffffff8014ee2d>{handle_IRQ_event+78}
<ffffffff8014ef10>{__do_IRQ+153} <ffffffff8010d661>{do_IRQ+60}
<ffffffff8023834a>{evtchn_do_upcall+134} <ffffffff8010b672>{do_hypervisor_callback+30} <EOI>
<ffffffff801073aa>{hypercall_page+938} <ffffffff801073aa>{hypercall_page+938}
<ffffffff801095de>{xen_idle+132} <ffffffff80109933>{cpu_idle+217}
Code: f4 eb fd 41 55 49 89 fd 48 c7 c7 c8 c9 3c 80 41 54 55 48 89
RIP <ffffffff8011326b>{smp_really_stop_cpu+9} RSP <ffff8800041a3ea8>
<0>Kernel panic - not syncing: Aiee, killing interrupt handler!
<0>general protection fault: 0000 [29] SMP
CPU 19
Modules linked in: ipv6 bridge button battery ac dm_mod e1000 ext3 jbd reiserfs fan thermal processor sg mptspi mptscsih mptbase piix sd_mod scsi_mod
Pid: 5667, comm: gdm Tainted: GF 2.6.16-xen #1
RIP: e030:[<ffffffff8011326b>] <ffffffff8011326b>{smp_really_stop_cpu+9}
RSP: e02b:ffff88000418bea8 EFLAGS: 00010002
RAX: ffffffffff5784c0 RBX: 0000000000000000 RCX: 0000000000000001
RDX: ffffffffff578000 RSI: 0000000000000000 RDI: 0000000000000000
RBP: 0000000000000000 R08: ffff880023f2d6c8 R09: ffff8807ceb73b48
R10: 0000000000000020 R11: 0000000000000246 R12: ffffffff80113262
R13: 0000000000000000 R14: ffff8807ceb738f8 R15: ffff8807ceb738f8
FS: 00002aab92f75d80(0000) GS:ffffffff803fb980(0000) knlGS:0000000000000000
CS: e033 DS: 0000 ES: 0000
Process gdm (pid: 5667, threadinfo ffff8807ceb72000, task ffff8807d69d97e0)
Stack: 0000000000000000 ffffffff801134c8 ffff88002c0f1780 0000000000000000
000000000000013a ffffffff8014ee2d ffffffff80406180 0000000000009d00
000000000000013a ffff88002c0f1780
Call Trace: <IRQ> <ffffffff801134c8>{smp_call_function_interrupt+79}
<ffffffff8014ee2d>{handle_IRQ_event+78} <ffffffff8014ef10>{__do_IRQ+153}
<ffffffff8010d661>{do_IRQ+60} <ffffffff8023834a>{evtchn_do_upcall+134}
<ffffffff8010b672>{do_hypervisor_callback+30} <EOI>
<ffffffff8010722a>{hypercall_page+554} <ffffffff8010722a>{hypercall_page+554}
<ffffffff80237bfd>{force_evtchn_callback+10} <ffffffff80158e15>{shrink_zone+3853}
<ffffffff802cd0ab>{thread_return+0} <ffffffff80113fc1>{xen_send_IPI_mask+0}
<ffffffff80237cf1>{notify_remote_via_irq+44} <ffffffff80134fb6>{lock_timer_base+27}
<ffffffff801236c3>{__wake_up_common+62} <ffffffff8019f6c7>{mb_cache_shrink_fn+36}
<ffffffff80159715>{try_to_free_pages+342} <ffffffff8015335c>{out_of_memory+368}
<ffffffff80154cdf>{__alloc_pages+422} <ffffffff8015667c>{__do_page_cache_readahead+245}
<ffffffff8015016c>{sync_page+0} <ffffffff802cd4b9>{io_schedule+50}
<ffffffff802cdcfd>{__wait_on_bit_lock+91} <ffffffff80152893>{filemap_nopage+323}
<ffffffff8015d2e3>{__handle_mm_fault+1415} <ffffffff801180d0>{do_page_fault+1938}
<ffffffff8014da85>{audit_syscall_exit+103} <ffffffff8010b5c7>{error_exit+0}
Code: f4 eb fd 41 55 49 89 fd 48 c7 c7 c8 c9 3c 80 41 54 55 48 89
RIP <ffffffff8011326b>{smp_really_stop_cpu+9} RSP <ffff88000418bea8>
<0>Kernel panic - not syncing: Aiee, killing interrupt handler!
<0>general protection fault: 0000 [30] SMP
CPU 26
Modules linked in: ipv6 bridge button battery ac dm_mod e1000 ext3 jbd reiserfs fan thermal processor sg mptspi mptscsih mptbase piix sd_mod scsi_mod
Pid: 0, comm: swapper Tainted: GF 2.6.16-xen #1
RIP: e030:[<ffffffff8011326b>] <ffffffff8011326b>{smp_really_stop_cpu+9}
RSP: e02b:ffff880000ca3ea8 EFLAGS: 00010002
RAX: ffffffffff578680 RBX: 0000000000000000 RCX: 0000000000000001
RDX: ffffffffff578000 RSI: 0000000000000002 RDI: 0000000000000000
RBP: 0000000000000000 R08: 000000010007333b R09: 000000000000001a
R10: 000000010007339f R11: 0000000000000246 R12: ffffffff80113262
R13: 0000000000000000 R14: ffff880000c6be58 R15: ffff880000c6be58
FS: 00002b2e6d783e30(0000) GS:ffffffff803fbd00(0000) knlGS:0000000000000000
CS: e033 DS: 002b ES: 002b
Process swapper (pid: 0, threadinfo ffff880000c6a000, task ffff8800049bb7a0)
Stack: ffffffff80113262 ffffffff801134c8 ffff8800041a9740 0000000000000000
000000000000014f ffffffff8014ee2d ffffffff80406c00 000000000000a780
000000000000014f ffff8800041a9740
Call Trace: <IRQ> <ffffffff80113262>{smp_really_stop_cpu+0}
<ffffffff801134c8>{smp_call_function_interrupt+79} <ffffffff8014ee2d>{handle_IRQ_event+78}
<ffffffff8014ef10>{__do_IRQ+153} <ffffffff8010d661>{do_IRQ+60}
<ffffffff8023834a>{evtchn_do_upcall+134} <ffffffff8010b672>{do_hypervisor_callback+30} <EOI>
<ffffffff801073aa>{hypercall_page+938} <ffffffff801073aa>{hypercall_page+938}
<ffffffff801095de>{xen_idle+132} <ffffffff80109933>{cpu_idle+217}
Code: f4 eb fd 41 55 49 89 fd 48 c7 c7 c8 c9 3c 80 41 54 55 48 89
RIP <ffffffff8011326b>{smp_really_stop_cpu+9} RSP <ffff880000ca3ea8>
<0>Kernel panic - not syncing: Aiee, killing interrupt handler!
general protection fault: 0000 [31] SMP
CPU 23
Modules linked in: ipv6 bridge button battery ac dm_mod e1000 ext3 jbd reiserfs fan thermal processor sg mptspi mptscsih mptbase piix sd_mod scsi_mod
Pid: 3568, comm: klogd Tainted: GF 2.6.16-xen #1
RIP: e030:[<ffffffff8011326b>] <ffffffff8011326b>{smp_really_stop_cpu+9}
RSP: e02b:ffff8800041dfea8 EFLAGS: 00010002
RAX: ffffffffff5785c0 RBX: 0000000000000000 RCX: 0000000000000001
RDX: ffffffffff578000 RSI: 0000000000000000 RDI: 0000000000000000
RBP: 0000000000000000 R08: ffff88002401ec98 R09: ffff8807d4619b48
R10: 0000000000000002 R11: 0000000000000246 R12: ffffffff80113262
R13: 0000000000000000 R14: ffff8807d46198f8 R15: ffff8807d46198f8
FS: 00002b5863f0d6d0(0000) GS:ffffffff803fbb80(0000) knlGS:0000000000000000
CS: e033 DS: 0000 ES: 0000
Process klogd (pid: 3568, threadinfo ffff8807d4618000, task ffff8807d56f4820)
Stack: 0000000000000040 ffffffff801134c8 ffff8800041a9dc0 0000000000000000
0000000000000146 ffffffff8014ee2d ffffffff80406780 000000000000a300
0000000000000146 ffff8800041a9dc0
Call Trace: <IRQ> <ffffffff801134c8>{smp_call_function_interrupt+79}
<ffffffff8014ee2d>{handle_IRQ_event+78} <ffffffff8014ef10>{__do_IRQ+153}
<ffffffff8010d661>{do_IRQ+60} <ffffffff8023834a>{evtchn_do_upcall+134}
<ffffffff8010b672>{do_hypervisor_callback+30} <EOI>
<ffffffff8010722a>{hypercall_page+554} <ffffffff8010722a>{hypercall_page+554}
<ffffffff8015208c>{__remove_from_page_cache+28} <ffffffff80237bfd>{force_evtchn_callback+10}
<ffffffff801585f6>{shrink_zone+1774} <ffffffff80113fc1>{xen_send_IPI_mask+0}
<ffffffff80113fc1>{xen_send_IPI_mask+0} <ffffffff80237cf1>{notify_remote_via_irq+44}
<ffffffff801236c3>{__wake_up_common+62} <ffffffff8019f6c7>{mb_cache_shrink_fn+36}
<ffffffff80159715>{try_to_free_pages+342} <ffffffff8015335c>{out_of_memory+368}
<ffffffff80154cdf>{__alloc_pages+422} <ffffffff8015667c>{__do_page_cache_readahead+245}
<ffffffff8015016c>{sync_page+0} <ffffffff802cd4b9>{io_schedule+50}
<ffffffff802cdcfd>{__wait_on_bit_lock+91} <ffffffff80152893>{filemap_nopage+323}
<ffffffff8015d2e3>{__handle_mm_fault+1415} <ffffffff801180d0>{do_page_fault+1938}
<ffffffff8013fa4d>{autoremove_wake_function+0} <ffffffff801611cd>{unmap_region+274}
<ffffffff8010b5c7>{error_exit+0}
Code: f4 eb fd 41 55 49 89 fd 48 c7 c7 c8 c9 3c 80 41 54 55 48 89
RIP <ffffffff8011326b>{smp_really_stop_cpu+9} RSP <ffff8800041dfea8>
<0>Kernel panic - not syncing: Aiee, killing interrupt handler!
general protection fault: 0000 [32] SMP
CPU 27
Modules linked in: ipv6 bridge button battery ac dm_mod e1000 ext3 jbd reiserfs fan thermal processor sg mptspi mptscsih mptbase piix sd_mod scsi_mod
Pid: 6498, comm: xentop Tainted: GF 2.6.16-xen #1
RIP: e030:[<ffffffff8011326b>] <ffffffff8011326b>{smp_really_stop_cpu+9}
RSP: e02b:ffff880000cafea8 EFLAGS: 00010002
RAX: ffffffffff5786c0 RBX: 0000000000000000 RCX: 0000000000000001
RDX: ffffffffff578000 RSI: 0000000000000000 RDI: 0000000000000000
RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8807c1e524a8
R10: ffffffff80172daf R11: 0000000000000246 R12: ffffffff80113262
R13: 0000000000000000 R14: ffff88078f01f8f8 R15: ffff88078f01f8f8
FS: 00002b8a13c69b70(0000) GS:ffffffff803fbd80(0000) knlGS:0000000000000000
CS: e033 DS: 0000 ES: 0000
Process xentop (pid: 6498, threadinfo ffff88078f01e000, task ffff8807be9db180)
Stack: 0000000000000040 ffffffff801134c8 ffff8800041a9540 0000000000000000
0000000000000152 ffffffff8014ee2d ffffffff80406d80 000000000000a900
0000000000000152 ffff8800041a9540
Call Trace: <IRQ> <ffffffff801134c8>{smp_call_function_interrupt+79}
<ffffffff8014ee2d>{handle_IRQ_event+78} <ffffffff8014ef10>{__do_IRQ+153}
<ffffffff8010d661>{do_IRQ+60} <ffffffff8023834a>{evtchn_do_upcall+134}
<ffffffff8010b672>{do_hypervisor_callback+30} <EOI>
<ffffffff80172daf>{try_to_free_buffers+96} <ffffffff8010722a>{hypercall_page+554}
<ffffffff8010722a>{hypercall_page+554} <ffffffff80237bfd>{force_evtchn_callback+10}
<ffffffff80158e15>{shrink_zone+3853} <ffffffff802cd0ab>{thread_return+0}
<ffffffff80113fc1>{xen_send_IPI_mask+0} <ffffffff80237cf1>{notify_remote_via_irq+44}
<ffffffff80134fb6>{lock_timer_base+27} <ffffffff80135028>{try_to_del_timer_sync+81}
<ffffffff802cd47c>{io_schedule_timeout+50} <ffffffff801cd57c>{blk_congestion_wait+132}
<ffffffff8013fa4d>{autoremove_wake_function+0} <ffffffff80159715>{try_to_free_pages+342}
<ffffffff80154cdf>{__alloc_pages+422} <ffffffff8015667c>{__do_page_cache_readahead+245}
<ffffffff8015016c>{sync_page+0} <ffffffff802cd4b9>{io_schedule+50}
<ffffffff802cdcfd>{__wait_on_bit_lock+91} <ffffffff80152893>{filemap_nopage+323}
<ffffffff8015d2e3>{__handle_mm_fault+1415} <ffffffff801180d0>{do_page_fault+1938}
<ffffffff80183bb7>{__pollwait+0} <ffffffff8014da85>{audit_syscall_exit+103}
<ffffffff8010b5c7>{error_exit+0}
Code: f4 eb fd 41 55 49 89 fd 48 c7 c7 c8 c9 3c 80 41 54 55 48 89
RIP <ffffffff8011326b>{smp_really_stop_cpu+9} RSP <ffff880000cafea8>
<0>Kernel panic - not syncing: Aiee, killing interrupt handler!
Badness in do_unblank_screen at drivers/char/vt.c:2830
Call Trace: <IRQ> <ffffffff80228114>{do_unblank_screen+76}
<ffffffff8011775b>{bust_spinlocks+28} <ffffffff8012b880>{panic+136}
<ffffffff8012e91c>{do_exit+124} <ffffffff801183cc>{do_page_fault+2702}
<ffffffff8010e483>{monotonic_clock+53} <ffffffff801daa07>{find_next_bit+85}
<ffffffff8012408f>{move_tasks+183} <ffffffff8011011e>{xen_panic_event+30}
<ffffffff8010b5c7>{error_exit+0} <ffffffff8011011e>{xen_panic_event+30}
<ffffffff8011011e>{xen_panic_event+30} <ffffffff801134c8>{smp_call_function_interrupt+79}
<ffffffff8014ee2d>{handle_IRQ_event+78} <ffffffff8014ef10>{__do_IRQ+153}
<ffffffff8010d661>{do_IRQ+60} <ffffffff8023834a>{evtchn_do_upcall+134}
<ffffffff8010b672>{do_hypervisor_callback+30} <EOI>
<ffffffff8010722a>{hypercall_page+554} <ffffffff8010722a>{hypercall_page+554}
<ffffffff80237bfd>{force_evtchn_callback+10} <ffffffff801585f6>{shrink_zone+1774}
<ffffffff802cd0ab>{thread_return+0} <ffffffff8019f6c7>{mb_cache_shrink_fn+36}
<ffffffff80159715>{try_to_free_pages+342} <ffffffff8015335c>{out_of_memory+368}
<ffffffff80154cdf>{__alloc_pages+422} <ffffffff80167143>{read_swap_cache_async+66}
<ffffffff8015b333>{swapin_readahead+78} <ffffffff8015d670>{__handle_mm_fault+2324}
<ffffffff801180d0>{do_page_fault+1938} <ffffffff8014da85>{audit_syscall_exit+103}
<ffffffff8010b5c7>{error_exit+0}
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
[-- Attachment #3: Type: text/plain, Size: 138 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* RE: xm mem-set causes kernel panic
@ 2006-05-03 20:18 Ian Pratt
0 siblings, 0 replies; 7+ messages in thread
From: Ian Pratt @ 2006-05-03 20:18 UTC (permalink / raw)
To: Krysan, Susan, xen-devel
Cc: Subrahmanian, Raj, Vessey, Bruce A, Puthiyaparambil, Aravindh,
Carb, Brian A
> When I boot the ES7000 with 32 GB of RAM, dom0 is allocated
> almost all of it. I issued an xm mem-set 0 512 command to
> decrease dom0's memory to 512 MB. Using xentop, I see the
> amount of memory allocated to dom0 slowly decrease. The
> kernel panic occurs when dom0's memory is around 1 GB. The
> serial console output is attached. Using SLES10 Beta 11
> upgraded to xen changeset 9903.
Can you repro this on -unstable? What about with a PAE hypervisor?
Thanks,
Ian
^ permalink raw reply [flat|nested] 7+ messages in thread
* RE: xm mem-set causes kernel panic
@ 2006-05-03 20:58 Krysan, Susan
2006-05-03 21:29 ` Andrew Theurer
0 siblings, 1 reply; 7+ messages in thread
From: Krysan, Susan @ 2006-05-03 20:58 UTC (permalink / raw)
To: Ian Pratt, xen-devel
Cc: Subrahmanian, Raj, Vessey, Bruce A, Puthiyaparambil, Aravindh,
Carb, Brian A
We did run using unstable. We installed sles10 beta11 but we pulled and built xen-unstable changeset 9903 from xensource repository. We have not tried with PAE hypervisor. It is not our focus. How desperately would you need us to try PAE?
Thanks,
> Sue Krysan
> Linux Systems Group
> Unisys Corporation
>
-----Original Message-----
From: Ian Pratt [mailto:m+Ian.Pratt@cl.cam.ac.uk]
Sent: Wednesday, May 03, 2006 4:19 PM
To: Krysan, Susan; xen-devel@lists.xensource.com
Cc: Subrahmanian, Raj; Vessey, Bruce A; Puthiyaparambil, Aravindh; Carb,
Brian A; ian.pratt@cl.cam.ac.uk
Subject: RE: [Xen-devel] xm mem-set causes kernel panic
> When I boot the ES7000 with 32 GB of RAM, dom0 is allocated
> almost all of it. I issued an xm mem-set 0 512 command to
> decrease dom0's memory to 512 MB. Using xentop, I see the
> amount of memory allocated to dom0 slowly decrease. The
> kernel panic occurs when dom0's memory is around 1 GB. The
> serial console output is attached. Using SLES10 Beta 11
> upgraded to xen changeset 9903.
Can you repro this on -unstable? What about with a PAE hypervisor?
Thanks,
Ian
^ permalink raw reply [flat|nested] 7+ messages in thread
* RE: xm mem-set causes kernel panic
@ 2006-05-03 21:15 Ian Pratt
2006-05-04 10:03 ` Keir Fraser
0 siblings, 1 reply; 7+ messages in thread
From: Ian Pratt @ 2006-05-03 21:15 UTC (permalink / raw)
To: Krysan, Susan, xen-devel
Cc: Subrahmanian, Raj, Vessey, Bruce A, Puthiyaparambil, Aravindh,
Carb, Brian A
> We did run using unstable. We installed sles10 beta11 but we
> pulled and built xen-unstable changeset 9903 from xensource
> repository. We have not tried with PAE hypervisor. It is
> not our focus. How desperately would you need us to try PAE?
Interesting. If you boot with dom0_mem=512 and then start a 15GB guest,
do you see problems shrinking the guest's memory?
I'd certainly be interested in hearing whether the problem exists on
PAE.
Ian
> -----Original Message-----
> From: Ian Pratt [mailto:m+Ian.Pratt@cl.cam.ac.uk]
> Sent: Wednesday, May 03, 2006 4:19 PM
> To: Krysan, Susan; xen-devel@lists.xensource.com
> Cc: Subrahmanian, Raj; Vessey, Bruce A; Puthiyaparambil,
> Aravindh; Carb, Brian A; ian.pratt@cl.cam.ac.uk
> Subject: RE: [Xen-devel] xm mem-set causes kernel panic
>
>
> > When I boot the ES7000 with 32 GB of RAM, dom0 is allocated
> almost all
> > of it. I issued an xm mem-set 0 512 command to decrease
> dom0's memory
> > to 512 MB. Using xentop, I see the amount of memory
> allocated to dom0
> > slowly decrease. The kernel panic occurs when dom0's
> memory is around
> > 1 GB. The serial console output is attached. Using SLES10 Beta 11
> > upgraded to xen changeset 9903.
>
> Can you repro this on -unstable? What about with a PAE hypervisor?
>
> Thanks,
> Ian
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: xm mem-set causes kernel panic
2006-05-03 20:58 xm mem-set causes kernel panic Krysan, Susan
@ 2006-05-03 21:29 ` Andrew Theurer
0 siblings, 0 replies; 7+ messages in thread
From: Andrew Theurer @ 2006-05-03 21:29 UTC (permalink / raw)
To: Krysan, Susan
Cc: Ian Pratt, xen-devel, Puthiyaparambil, Aravindh,
Subrahmanian, Raj, Vessey, Bruce A, Carb, Brian A
Krysan, Susan wrote:
> We did run using unstable. We installed sles10 beta11 but we pulled and built xen-unstable changeset 9903 from xensource repository. We have not tried with PAE hypervisor. It is not our focus. How desperately would you need us to try PAE?
>
I have done this on 32GB x86_64 as well and have pretty much the same
problem. I think the problem is that the static memory allocations in
the kernel for things like mem_map, page_struct etc, are quite large for
32GB, and do not shrink (or do they?) when balooning to 512 MB. This
leaves almost no usable memory when balooning down, and you get OOM. If
you know you don't need 32G for dom0, I would try setting dom0_mem to
something much lower, like 1GB.
-Andrew Theurer
> Thanks,
>
>
>> Sue Krysan
>> Linux Systems Group
>> Unisys Corporation
>>
>>
>
>
> -----Original Message-----
> From: Ian Pratt [mailto:m+Ian.Pratt@cl.cam.ac.uk]
> Sent: Wednesday, May 03, 2006 4:19 PM
> To: Krysan, Susan; xen-devel@lists.xensource.com
> Cc: Subrahmanian, Raj; Vessey, Bruce A; Puthiyaparambil, Aravindh; Carb,
> Brian A; ian.pratt@cl.cam.ac.uk
> Subject: RE: [Xen-devel] xm mem-set causes kernel panic
>
>
>
>> When I boot the ES7000 with 32 GB of RAM, dom0 is allocated
>> almost all of it. I issued an xm mem-set 0 512 command to
>> decrease dom0's memory to 512 MB. Using xentop, I see the
>> amount of memory allocated to dom0 slowly decrease. The
>> kernel panic occurs when dom0's memory is around 1 GB. The
>> serial console output is attached. Using SLES10 Beta 11
>> upgraded to xen changeset 9903.
>>
>
> Can you repro this on -unstable? What about with a PAE hypervisor?
>
> Thanks,
> Ian
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: xm mem-set causes kernel panic
2006-05-03 21:15 Ian Pratt
@ 2006-05-04 10:03 ` Keir Fraser
0 siblings, 0 replies; 7+ messages in thread
From: Keir Fraser @ 2006-05-04 10:03 UTC (permalink / raw)
To: Ian Pratt
Cc: xen-devel, Krysan, Susan, Puthiyaparambil, Aravindh,
Subrahmanian, Raj, Vessey, Bruce A, Carb, Brian A
On 3 May 2006, at 22:15, Ian Pratt wrote:
>> We did run using unstable. We installed sles10 beta11 but we
>> pulled and built xen-unstable changeset 9903 from xensource
>> repository. We have not tried with PAE hypervisor. It is
>> not our focus. How desperately would you need us to try PAE?
>
> Interesting. If you boot with dom0_mem=512 and then start a 15GB guest,
> do you see problems shrinking the guest's memory?
>
> I'd certainly be interested in hearing whether the problem exists on
> PAE.
I don't think this behaviour is surprising. The kernel is probably
taking about 1% of memory to track memory usage. So trying to balloon
down to 1-2% of your initial memory reservation is rather harsh.
The 'fix' here is to stop users being able to shoot their feet off, by
setting a lower bound on ballooning that is related to the initial
memory size of the guest, and/or quiesce the balloon driver when
low-memory and out-of-memory code paths are executed in the guest.
-- Keir
^ permalink raw reply [flat|nested] 7+ messages in thread
* RE: xm mem-set causes kernel panic
@ 2006-05-04 15:49 Puthiyaparambil, Aravindh
0 siblings, 0 replies; 7+ messages in thread
From: Puthiyaparambil, Aravindh @ 2006-05-04 15:49 UTC (permalink / raw)
To: Keir Fraser, Ian Pratt
Cc: Carb, Brian A, xen-devel, Krysan, Susan, Subrahmanian, Raj,
Vessey, Bruce A
> I don't think this behaviour is surprising. The kernel is probably
> taking about 1% of memory to track memory usage. So trying to balloon
> down to 1-2% of your initial memory reservation is rather harsh.
I agree.
> The 'fix' here is to stop users being able to shoot their feet off, by
> setting a lower bound on ballooning that is related to the initial
> memory size of the guest, and/or quiesce the balloon driver when
> low-memory and out-of-memory code paths are executed in the guest.
We will look into this and send in a patch.
Cheers,
Aravindh
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2006-05-04 15:49 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2006-05-03 20:58 xm mem-set causes kernel panic Krysan, Susan
2006-05-03 21:29 ` Andrew Theurer
-- strict thread matches above, loose matches on Subject: below --
2006-05-04 15:49 Puthiyaparambil, Aravindh
2006-05-03 21:15 Ian Pratt
2006-05-04 10:03 ` Keir Fraser
2006-05-03 20:18 Ian Pratt
2006-05-03 19:01 Krysan, Susan
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.