From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jeremy Fitzhardinge Subject: Re: 2.6.35.7 pvops crash Date: Fri, 19 Nov 2010 09:30:01 -0800 Message-ID: <4CE6B419.5050607@goop.org> References: Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Shaun Reitan Cc: xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org On 11/19/2010 09:15 AM, Shaun Reitan wrote: > OK if I'm posting this to the wrong ML I'm sorry. I'm not sure if i > should be posting this to -devel or to -user, or maybe not to you guys > at all (maybe the kernel ML i should be sending this too?). Here's a > snip i was able to grab, sorry i didn't get it all. xen-devel is probably the right place for this, but its a bit unclear what's going on here. Unfortunately your report is missing the most interesting information. Can you go into more detail about what's going on here? This is a 32bit kernel, but what are the other details? What kernel version is it (is it a distro kernel?) dom0? domU? Which version of Xen? What was the system doing at the time? Thanks, J > > > <0> 00000000 d1df9a78 e8ddbb34 00000000 00000000 ec362580 c0105bd4 > 00400000 > <0> 00000000 00000001 c0000000 e424e010 ec362580 c2050b60 c147080c > cd5aeff8 > Call Trace: > [] ? unmap_vmas+0x2dd/0x7e0 > [] ? check_events+0x8/0xc > [] ? xen_restore_fl_direct_end+0x0/0x1 > [] ? xen_mc_flush+0xe0/0x190 > [] ? check_events+0x8/0xc > [] ? exit_mmap+0x8c/0x140 > [] ? mmput+0x3d/0xd0 > [] ? exit_mm+0xd3/0xf0 > [] ? _raw_spin_lock_irq+0x10/0x20 > [] ? do_exit+0x167/0x6e0 > [] ? _raw_spin_lock_irqsave+0x27/0x40 > [] ? kmsg_dump+0x76/0x140 > [] ? printk+0x1b/0x20 > [] ? do_invalid_op+0x0/0x90 > [] ? oops_end+0x9f/0xa0 > [] ? do_invalid_op+0x7f/0x90 > [] ? swap_count_continued+0x1bf/0x1d0 > [] ? xen_restore_fl_direct_end+0x0/0x1 > [] ? _raw_spin_unlock_irqrestore+0x11/0x20 > [] ? __rcu_process_callbacks+0x6c/0x2d0 > [] ? rcu_process_callbacks+0x35/0x40 > [] ? __do_softirq+0xb6/0x120 > [] ? xen_force_evtchn_callback+0x17/0x30 > [] ? pte_pfn_to_mfn+0x6d/0x80 > [] ? xen_set_pte+0x19/0x30 > [] ? error_code+0x66/0x6c > [] ? do_invalid_op+0x0/0x90 > [] ? swap_count_continued+0x1bf/0x1d0 > [] ? swap_entry_free+0x127/0x150 > [] ? free_swap_and_cache+0x27/0xd0 > [] ? unmap_vmas+0x561/0x7e0 > [] ? exit_mmap+0x8c/0x140 > [] ? mmput+0x3d/0xd0 > [] ? flush_old_exec+0x238/0x570 > [] ? do_sync_read+0x0/0xf0 > [] ? load_elf_binary+0x2d9/0x1810 > [] ? __alloc_pages_nodemask+0xeb/0x560 > [] ? security_vm_enough_memory_mm+0x1f/0x40 > [] ? __raw_callee_save_xen_pte_val+0x6/0x8 > [] ? vm_normal_page+0x2b/0xd0 > [] ? strrchr+0xb/0x20 > [] ? load_misc_binary+0x87/0x3d0 > [] ? __get_user_pages+0xe9/0x3e0 > [] ? get_arg_page+0x52/0xb0 > [] ? search_binary_handler+0x9b/0x230 > [] ? do_execve+0x1f3/0x250 > [] ? sys_execve+0x33/0x70 > [] ? ptregs_execve+0x12/0x18 > [] ? sysenter_do_call+0x12/0x28 > Code: b6 00 00 00 00 8b 54 24 08 8b 44 24 04 89 51 04 89 01 c3 66 90 > 64 a1 44 db 7d c0 85 c0 75 0c 64 c7 05 44 db 7d c0 01 00 00 00 c3 <0f> > 0b eb fe 90 90 90 90 90 90 fa fb 50 9d 9c 58 cf fb 0f 35 0f > EIP: [] paravirt_enter_lazy_mmu+0x16/0x20 SS:ESP 0069:e8ddba94 > ---[ end trace 684584b81fd0c02c ]--- > Fixing recursive fault but reboot is needed! > BUG: scheduling while atomic: httpd/14418/0x00000001 > Modules linked in: > Pid: 14418, comm: httpd Tainted: G D 2.6.35.7-1 #4 > Call Trace: > [] ? schedule+0x550/0x610 > [] ? vprintk+0x193/0x3a0 > [] ? do_exit+0x624/0x6e0 > [] ? _raw_spin_lock_irqsave+0x27/0x40 > [] ? _raw_spin_trylock+0x5/0x10 > [] ? kmsg_dump+0x76/0x140 > [] ? printk+0x1b/0x20 > [] ? do_invalid_op+0x0/0x90 > [] ? oops_end+0x9f/0xa0 > [] ? do_invalid_op+0x7f/0x90 > [] ? paravirt_enter_lazy_mmu+0x16/0x20 > [] ? tick_program_event+0x2a/0x40 > [] ? hrtimer_interrupt+0x147/0x240 > [] ? check_events+0x8/0xc > [] ? xen_force_evtchn_callback+0x17/0x30 > [] ? xen_force_evtchn_callback+0x17/0x30 > [] ? xen_force_evtchn_callback+0x17/0x30 > [] ? check_events+0x8/0xc > [] ? error_code+0x66/0x6c > [] ? do_invalid_op+0x0/0x90 > [] ? paravirt_enter_lazy_mmu+0x16/0x20 > [] ? unmap_vmas+0x2dd/0x7e0 > [] ? check_events+0x8/0xc > [] ? xen_restore_fl_direct_end+0x0/0x1 > [] ? xen_mc_flush+0xe0/0x190 > [] ? check_events+0x8/0xc > [] ? exit_mmap+0x8c/0x140 > [] ? mmput+0x3d/0xd0 > [] ? exit_mm+0xd3/0xf0 > [] ? _raw_spin_lock_irq+0x10/0x20 > [] ? do_exit+0x167/0x6e0 > [] ? _raw_spin_lock_irqsave+0x27/0x40 > [] ? kmsg_dump+0x76/0x140 > [] ? printk+0x1b/0x20 > [] ? do_invalid_op+0x0/0x90 > [] ? oops_end+0x9f/0xa0 > [] ? do_invalid_op+0x7f/0x90 > [] ? swap_count_continued+0x1bf/0x1d0 > [] ? xen_restore_fl_direct_end+0x0/0x1 > [] ? _raw_spin_unlock_irqrestore+0x11/0x20 > [] ? __rcu_process_callbacks+0x6c/0x2d0 > [] ? rcu_process_callbacks+0x35/0x40 > [] ? __do_softirq+0xb6/0x120 > [] ? xen_force_evtchn_callback+0x17/0x30 > [] ? pte_pfn_to_mfn+0x6d/0x80 > [] ? xen_set_pte+0x19/0x30 > [] ? error_code+0x66/0x6c > [] ? do_invalid_op+0x0/0x90 > [] ? swap_count_continued+0x1bf/0x1d0 > [] ? swap_entry_free+0x127/0x150 > [] ? free_swap_and_cache+0x27/0xd0 > [] ? unmap_vmas+0x561/0x7e0 > [] ? exit_mmap+0x8c/0x140 > [] ? mmput+0x3d/0xd0 > [] ? flush_old_exec+0x238/0x570 > [] ? do_sync_read+0x0/0xf0 > [] ? load_elf_binary+0x2d9/0x1810 > [] ? __alloc_pages_nodemask+0xeb/0x560 > [] ? security_vm_enough_memory_mm+0x1f/0x40 > [] ? __raw_callee_save_xen_pte_val+0x6/0x8 > [] ? vm_normal_page+0x2b/0xd0 > [] ? strrchr+0xb/0x20 > [] ? load_misc_binary+0x87/0x3d0 > [] ? __get_user_pages+0xe9/0x3e0 > [] ? get_arg_page+0x52/0xb0 > [] ? search_binary_handler+0x9b/0x230 > [] ? do_execve+0x1f3/0x250 > [] ? sys_execve+0x33/0x70 > [] ? ptregs_execve+0x12/0x18 > [] ? sysenter_do_call+0x12/0x28 > ------------[ cut here ]------------ > kernel BUG at arch/x86/mm/highmem_32.c:45! > invalid opcode: 0000 [#3] SMP > last sysfs file: /sys/devices/vif-0/net/eth0/broadcast > Modules linked in: > > Pid: 31745, comm: mysqld Tainted: G D 2.6.35.7-1 #4 / > EIP: 0061:[] EFLAGS: 00010282 CPU: 4 > EIP is at kmap_atomic_prot+0x137/0x140 > EAX: 5b489063 EBX: db489067 ECX: c17f86c0 EDX: 80000004 > ESI: f57ff000 EDI: 0000001c EBP: 0000001c ESP: df171df4 > DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069 > Process mysqld (pid: 31745, ti=df170000 task=eb4b0030 task.ti=df170000) > Stack: > c20349c0 c05ef7c7 f57ff000 00000298 00000053 00000163 80000000 80000000 > <0> df171eb8 0000001c 0000001c c012328c 00000163 80000000 b174ada0 > c017505b > <0> 00000000 c17f86c0 c891a978 0000001c 022cda20 c17f86c0 00000106 > c891a974 > Call Trace: > [] ? io_schedule+0x77/0xa0 > [] ? kmap_atomic+0x1c/0x30 > [] ? file_read_actor+0xcb/0x100 > [] ? generic_file_aio_read+0x2ca/0x720 > [] ? do_sync_read+0xb5/0xf0 > [] ? rcu_process_callbacks+0x35/0x40 > [] ? security_file_permission+0xc/0x10 > [] ? rw_verify_area+0x66/0xd0 > [] ? vfs_read+0xa1/0x180 > [] ? do_sync_read+0x0/0xf0 > [] ? sys_read+0x41/0x70 > [] ? sysenter_do_call+0x12/0x28 > Code: e0 0c 29 44 24 08 8b 44 24 08 83 c4 1c 5b 5e 5f 5d c3 8b 74 24 > 14 8b 7c 24 18 23 35 d8 0c 77 c0 23 3d dc 0c 77 c0 e9 7a ff ff ff <0f> > 0b eb fe 90 8d 74 26 00 53 83 ec 08 8b 1d 10 05 80 c0 8b 0d > EIP: [] kmap_atomic_prot+0x137/0x140 SS:ESP 0069:df171df4 > ---[ end trace 684584b81fd0c02d ]--- > note: mysqld[31745] exited with preempt_count 1 > ------------[ cut here ]------------ > kernel BUG at arch/x86/mm/highmem_32.c:45! > invalid opcode: 0000 [#4] SMP > last sysfs file: /sys/devices/vif-0/net/eth0/broadcast > Modules linked in: > > Pid: 14396, comm: leechprotect Tainted: G D 2.6.35.7-1 #4 / > EIP: 0061:[] EFLAGS: 00210282 CPU: 4 > EIP is at kmap_atomic_prot+0x137/0x140 > EAX: 5b489063 EBX: db489067 ECX: c18b3bc0 EDX: 80000004 > ESI: f57ff000 EDI: 00001000 EBP: 000046fd ESP: ce9a3df4 > DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069 > Process leechprotect (pid: 14396, ti=ce9a2000 task=c4ecf7f0 > task.ti=ce9a2000) > Stack: > c05f1676 08cf2440 f57ff000 00000298 00000053 00000163 80000000 80000000 > <0> ce9a3eb8 00001000 000046fd c012328c 00000163 80000000 08cf2440 > c017505b > <0> 00000000 c18b3bc0 e0dc8ef0 00001000 000046fd c18b3bc0 00000000 > e0dc8eec > Call Trace: > [] ? error_code+0x66/0x6c > [] ? kmap_atomic+0x1c/0x30 > [] ? file_read_actor+0xcb/0x100 > [] ? generic_file_aio_read+0x2ca/0x720 > [] ? dquot_file_open+0x0/0x40 > [] ? dput+0x1e/0x120 > [] ? do_sync_read+0xb5/0xf0 > [] ? xen_force_evtchn_callback+0x17/0x30 > [] ? check_events+0x8/0xc > [] ? xen_force_evtchn_callback+0x17/0x30 > [] ? security_file_permission+0xc/0x10 > [] ? rw_verify_area+0x66/0xd0 > [] ? check_events+0x8/0xc > [] ? vfs_read+0xa1/0x180 > [] ? do_sync_read+0x0/0xf0 > [] ? sys_read+0x41/0x70 > [] ? sysenter_do_call+0x12/0x28 > Code: e0 0c 29 44 24 08 8b 44 24 08 83 c4 1c 5b 5e 5f 5d c3 8b 74 24 > 14 8b 7c 24 18 23 35 d8 0c 77 c0 23 3d dc 0c 77 c0 e9 7a ff ff ff <0f> > 0b eb fe 90 8d 74 26 00 53 83 ec 08 8b 1d 10 05 80 c0 8b 0d > EIP: [] kmap_atomic_prot+0x137/0x140 SS:ESP 0069:ce9a3df4 > ---[ end trace 684584b81fd0c02e ]--- > note: leechprotect[14396] exited with preempt_count 1 > ------------[ cut here ]------------ > kernel BUG at arch/x86/mm/highmem_32.c:45! > invalid opcode: 0000 [#5] SMP > last sysfs file: /sys/devices/vif-0/net/eth0/broadcast > Modules linked in: > > Pid: 13782, comm: cpanellogd Tainted: G D 2.6.35.7-1 #4 / > EIP: 0061:[] EFLAGS: 00010282 CPU: 4 > EIP is at kmap_atomic_prot+0x137/0x140 > EAX: 5b489063 EBX: db489067 ECX: c1ca5300 EDX: 80000004 > ESI: f57ff000 EDI: ccbaa500 EBP: eb4b3d14 ESP: eb4b3ca0 > DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069 > Process cpanellogd (pid: 13782, ti=eb4b2000 task=d17257f0 > task.ti=eb4b2000) > Stack: > c01c2576 dcec2e3c f57ff000 00000298 00000053 00000163 80000000 80000000 > <0> 00001000 ccbaa500 eb4b3d14 c012328c 00000163 80000000 00000000 > c01c3b0e > <0> eaca9b38 00000001 eb4b3d14 00000000 c1ca5300 dcec2e3c 00001000 > 00000000 > Call Trace: > [] ? __find_get_block_slow+0x46/0x140 > [] ? kmap_atomic+0x1c/0x30 > [] ? __block_prepare_write+0x23e/0x380 > [] ? block_write_begin_newtrunc+0x56/0xd0 > [] ? ext3_get_block+0x0/0x100 > [] ? block_write_begin+0x53/0xc0 > [] ? ext3_get_block+0x0/0x100 > [] ? journal_start+0x7c/0xa0 > [] ? ext3_write_begin+0x146/0x220 > [] ? ext3_get_block+0x0/0x100 > [] ? generic_file_buffered_write+0xe2/0x200 > [] ? __generic_file_aio_write+0x26b/0x520 > [] ? __dentry_open+0x17f/0x230 > [] ? nameidata_to_filp+0x47/0x50 > [] ? dquot_file_open+0x0/0x40 > [] ? do_last+0x55c/0x5a0 > [] ? generic_file_aio_write+0x52/0xb0 > [] ? do_sync_write+0xb5/0xf0 > [] ? cp_new_stat64+0xee/0x100 > [] ? security_file_permission+0xc/0x10 > [] ? rw_verify_area+0x66/0xd0 > [] ? vfs_write+0xa3/0x180 > [] ? do_fcntl+0x396/0x470 > [] ? do_sync_write+0x0/0xf0 > [] ? sys_write+0x41/0x70 > [] ? sysenter_do_call+0x12/0x28 > Code: e0 0c 29 44 24 08 8b 44 24 08 83 c4 1c 5b 5e 5f 5d c3 8b 74 24 > 14 8b 7c 24 18 23 35 d8 0c 77 c0 23 3d dc 0c 77 c0 e9 7a ff ff ff <0f> > 0b eb fe 90 8d 74 26 00 53 83 ec 08 8b 1d 10 05 80 c0 8b 0d > EIP: [] kmap_atomic_prot+0x137/0x140 SS:ESP 0069:eb4b3ca0 > ---[ end trace 684584b81fd0c02f ]--- > > -- > ShaunR > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >