From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx127.postini.com [74.125.245.127]) by kanga.kvack.org (Postfix) with SMTP id 981A36B0062 for ; Wed, 11 Jan 2012 13:08:09 -0500 (EST) Received: from /spool/local by e28smtp06.in.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 11 Jan 2012 23:38:06 +0530 Received: from d28av04.in.ibm.com (d28av04.in.ibm.com [9.184.220.66]) by d28relay05.in.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id q0BI81FP4239592 for ; Wed, 11 Jan 2012 23:38:02 +0530 Received: from d28av04.in.ibm.com (loopback [127.0.0.1]) by d28av04.in.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id q0BI80vh011376 for ; Thu, 12 Jan 2012 05:08:01 +1100 Message-ID: <4F0DCFFC.5040805@linux.vnet.ibm.com> Date: Wed, 11 Jan 2012 23:37:56 +0530 From: "Srivatsa S. Bhat" MIME-Version: 1.0 Subject: Several bugs in latest kernel Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: mgorman@suse.de Cc: Al Viro , Tejun Heo , Linus Torvalds , linux-mm@kvack.org, linux-kernel , Pekka Enberg , Peter Zijlstra , "mingo@elte.hu" , "akpm@linux-foundation.org" Hi, I was running the latest kernel and not doing anything in particular. Eventually the machine locked up hard and due to my config setting (panic on hard-lockup), I got a kernel panic. Looks like there are several issues involved. Here is the log: [ 7314.423828] ------------[ cut here ]------------ [ 7314.427769] kernel BUG at mm/slab.c:3111! [ 7314.427769] invalid opcode: 0000 [#1] SMP [ 7314.427769] CPU 3 [ 7314.427769] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod bnx2 ioatdma tpm_tis tpm cdc_ether usbnet i2c_i801 iTCO_wdt mii i7core_edac i2c_core dca edac_core iTCO_vendor_support rtc_cmos tpm_bios shpchp pci_hotplug button pcspkr serio_raw sg uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon [ 7314.427769] [ 7314.427769] Pid: 6699, comm: cron Tainted: G W 3.2.0-0.0.0.28.36b5ec9-default #3 IBM IBM System x -[7870C4Q]-/68Y8033 [ 7314.427769] RIP: 0010:[] [] cache_alloc_refill+0x1e9/0x290 [ 7314.427769] RSP: 0018:ffff8808c881bc48 EFLAGS: 00010046 [ 7314.427769] RAX: 000000000000000f RBX: ffff8808ca66b000 RCX: 0000000000000018 [ 7314.427769] RDX: ffff8808c7e2d040 RSI: ffff8808c8f60040 RDI: 0000000000000024 [ 7314.427769] RBP: ffff8808c881bc88 R08: ffff8808ff802510 R09: ffff8808ff802520 [ 7314.427769] R10: dead000000200200 R11: dead000000100100 R12: 0000000000000024 [ 7314.427769] R13: ffff8808ff800880 R14: ffff8808ff802500 R15: 0000000000000000 [ 7314.427769] FS: 00007fdcd8f54780(0000) GS:ffff8808ffcc0000(0000) knlGS:0000000000000000 [ 7314.427769] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7314.427769] CR2: ffffffffff600400 CR3: 00000008c6e95000 CR4: 00000000000006e0 [ 7314.427769] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7314.427769] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7314.427769] Process cron (pid: 6699, threadinfo ffff8808c881a000, task ffff8808c68a0380) [ 7314.427769] Stack: [ 7314.427769] ffffffff81785cf1 00000000000412d0 ffff8808ff802540 ffff8808ff800880 [ 7314.427769] ffff8808ff800880 0000000000000100 00000000000000d0 00000000000000d0 [ 7314.427769] ffff8808c881bcd8 ffffffff8115c7e7 ffff8808c881bd26 ffffffff81230418 [ 7314.427769] Call Trace: [ 7314.427769] [] __kmalloc+0x327/0x330 [ 7314.427769] [] ? aa_get_name+0x58/0x100 [ 7314.427769] [] aa_get_name+0x58/0x100 [ 7314.427769] [] ? cap_bprm_set_creds+0x239/0x2a0 [ 7314.427769] [] apparmor_bprm_set_creds+0x112/0x580 [ 7314.427769] [] ? __lock_release+0x7e/0x170 [ 7314.427769] [] ? might_fault+0x4e/0xa0 [ 7314.427769] [] security_bprm_set_creds+0xe/0x10 [ 7314.427769] [] prepare_binprm+0xca/0x140 [ 7314.427769] [] do_execve_common+0x204/0x320 [ 7314.427769] [] do_execve+0x3a/0x40 [ 7314.427769] [] sys_execve+0x49/0x70 [ 7314.427769] [] stub_execve+0x6c/0xc0 [ 7314.427769] Code: 08 49 89 76 10 eb a6 0f 1f 00 49 8b 76 20 41 c7 86 90 00 00 00 01 00 00 00 49 39 f1 74 97 8b 46 20 41 3b 45 18 0f 82 02 ff ff ff <0f> 0b eb fe 0f 1f 00 41 39 c4 41 89 c7 45 0f 46 fc e9 ab fe ff [ 7314.427769] RIP [] cache_alloc_refill+0x1e9/0x290 [ 7314.427769] RSP [ 7314.427769] ---[ end trace c15ebd724b0d27b5 ]--- [ 7314.427769] BUG: sleeping function called from invalid context at kernel/rwsem.c:21 [ 7314.427769] in_atomic(): 1, irqs_disabled(): 1, pid: 6699, name: cron [ 7314.427769] INFO: lockdep is turned off. [ 7314.427769] irq event stamp: 1056 [ 7314.427769] hardirqs last enabled at (1055): [] kmem_cache_alloc+0x225/0x2d0 [ 7314.427769] hardirqs last disabled at (1056): [] __kmalloc+0xa7/0x330 [ 7314.427769] softirqs last enabled at (642): [] unix_sock_destructor+0x80/0xf0 [ 7314.427769] softirqs last disabled at (640): [] unix_sock_destructor+0x69/0xf0 [ 7314.427769] Pid: 6699, comm: cron Tainted: G D W 3.2.0-0.0.0.28.36b5ec9-default #3 [ 7314.427769] Call Trace: [ 7314.427769] [] __might_sleep+0x152/0x1f0 [ 7314.427769] [] down_read+0x1f/0x60 [ 7314.427769] [] exit_signals+0x1f/0x140 [ 7314.427769] [] ? blocking_notifier_call_chain+0x11/0x20 [ 7314.427769] [] do_exit+0xb2/0x480 [ 7314.427769] [] oops_end+0xe4/0xf0 [ 7314.427769] [] die+0x56/0x90 [ 7314.427769] [] do_trap+0x148/0x160 [ 7314.427769] [] ? atomic_notifier_call_chain+0x11/0x20 [ 7314.427769] [] do_invalid_op+0x90/0xb0 [ 7314.427769] [] ? cache_alloc_refill+0x1e9/0x290 [ 7314.427769] [] ? __lock_acquire+0x301/0x520 [ 7314.427769] [] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 7314.427769] [] ? restore_args+0x30/0x30 [ 7314.427769] [] invalid_op+0x1b/0x20 [ 7314.427769] [] ? cache_alloc_refill+0x1e9/0x290 [ 7314.427769] [] ? cache_alloc_refill+0x85/0x290 [ 7314.427769] [] __kmalloc+0x327/0x330 [ 7314.427769] [] ? aa_get_name+0x58/0x100 [ 7314.427769] [] aa_get_name+0x58/0x100 [ 7314.427769] [] ? cap_bprm_set_creds+0x239/0x2a0 [ 7314.427769] [] apparmor_bprm_set_creds+0x112/0x580 [ 7314.427769] [] ? __lock_release+0x7e/0x170 [ 7314.427769] [] ? might_fault+0x4e/0xa0 [ 7314.427769] [] security_bprm_set_creds+0xe/0x10 [ 7314.427769] [] prepare_binprm+0xca/0x140 [ 7314.427769] [] do_execve_common+0x204/0x320 [ 7314.427769] [] do_execve+0x3a/0x40 [ 7314.427769] [] sys_execve+0x49/0x70 [ 7314.427769] [] stub_execve+0x6c/0xc0 [ 7314.427769] note: cron[6699] exited with preempt_count 1 [ 7314.981405] BUG: scheduling while atomic: cron/6699/0x10000002 [ 7314.987495] INFO: lockdep is turned off. [ 7314.987497] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod bnx2 ioatdma tpm_tis tpm cdc_ether usbnet i2c_i801 iTCO_wdt mii i7core_edac i2c_core dca edac_core iTCO_vendor_support rtc_cmos tpm_bios shpchp pci_hotplug button pcspkr serio_raw sg uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon [ 7314.987531] Pid: 6699, comm: cron Tainted: G D W 3.2.0-0.0.0.28.36b5ec9-default #3 [ 7314.987533] Call Trace: [ 7314.987538] [] __schedule_bug+0x97/0xa0 [ 7314.987542] [] __schedule+0x705/0x9a0 [ 7314.987546] [] ? _raw_spin_unlock+0x26/0x40 [ 7314.987550] [] ? zap_pte_range+0x84/0x3b0 [ 7314.987554] [] ? zap_pte_range+0x1b5/0x3b0 [ 7314.987559] [] ? __atomic_notifier_call_chain+0xa6/0x130 [ 7314.987564] [] __cond_resched+0x25/0x40 [ 7314.987567] [] _cond_resched+0x2d/0x40 [ 7314.987571] [] unmap_page_range+0x25e/0x300 [ 7314.987575] [] unmap_vmas+0xcc/0x150 [ 7314.987580] [] exit_mmap+0x8d/0x120 [ 7314.987584] [] ? exit_mm+0xfa/0x140 [ 7314.987587] [] mmput+0x6c/0x150 [ 7314.987591] [] exit_mm+0x10a/0x140 [ 7314.987594] [] ? _raw_spin_unlock_irq+0x2b/0x50 [ 7314.987599] [] ? tty_audit_exit+0x23/0xa0 [ 7314.987603] [] do_exit+0x153/0x480 [ 7314.987606] [] oops_end+0xe4/0xf0 [ 7314.987610] [] die+0x56/0x90 [ 7314.987613] [] do_trap+0x148/0x160 [ 7314.987617] [] ? atomic_notifier_call_chain+0x11/0x20 [ 7314.987622] [] do_invalid_op+0x90/0xb0 [ 7314.987626] [] ? cache_alloc_refill+0x1e9/0x290 [ 7314.987630] [] ? __lock_acquire+0x301/0x520 [ 7314.987634] [] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 7314.987638] [] ? restore_args+0x30/0x30 [ 7314.987641] [] invalid_op+0x1b/0x20 [ 7314.987646] [] ? cache_alloc_refill+0x1e9/0x290 [ 7314.987650] [] ? cache_alloc_refill+0x85/0x290 [ 7314.987654] [] __kmalloc+0x327/0x330 [ 7314.987658] [] ? aa_get_name+0x58/0x100 [ 7314.987661] [] aa_get_name+0x58/0x100 [ 7314.987665] [] ? cap_bprm_set_creds+0x239/0x2a0 [ 7314.987669] [] apparmor_bprm_set_creds+0x112/0x580 [ 7314.987673] [] ? __lock_release+0x7e/0x170 [ 7314.987677] [] ? might_fault+0x4e/0xa0 [ 7314.987681] [] security_bprm_set_creds+0xe/0x10 [ 7314.987685] [] prepare_binprm+0xca/0x140 [ 7314.987689] [] do_execve_common+0x204/0x320 [ 7314.987694] [] do_execve+0x3a/0x40 [ 7314.987697] [] sys_execve+0x49/0x70 [ 7314.987701] [] stub_execve+0x6c/0xc0 [ 7320.364127] Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 13 [ 7320.364127] Pid: 85, comm: kworker/13:1 Tainted: G D W 3.2.0-0.0.0.28.36b5ec9-default #3 [ 7320.364127] Call Trace: [ 7320.364127] [] panic+0x9f/0x1e5 [ 7320.364127] [] ? sched_clock_local+0x25/0x90 [ 7320.364127] [] watchdog_overflow_callback+0xb1/0xc0 [ 7320.364127] [] __perf_event_overflow+0xa5/0x2d0 [ 7320.364127] [] ? perf_event_update_userpage+0x3c/0x280 [ 7320.364127] [] ? x86_perf_event_set_period+0xdf/0x170 [ 7320.364127] [] perf_event_overflow+0x14/0x20 [ 7320.364127] [] intel_pmu_handle_irq+0x173/0x350 [ 7320.364127] [] perf_event_nmi_handler+0x19/0x20 [ 7320.364127] [] nmi_handle+0xbe/0x1d0 [ 7320.364127] [] ? nmi_handle+0x4b/0x1d0 [ 7320.364127] [] default_do_nmi+0x63/0x270 [ 7320.364127] [] do_nmi+0xa8/0xc0 [ 7320.364127] [] nmi+0x20/0x39 [ 7320.364127] [] ? read_persistent_clock+0x30/0x30 [ 7320.364127] <> [] ? delay_tsc+0x78/0xd0 [ 7320.364127] [] __delay+0xa/0x10 [ 7320.364127] [] do_raw_spin_lock+0xab/0x150 [ 7320.364127] [] _raw_spin_lock+0x44/0x50 [ 7320.364127] [] ? __drain_alien_cache+0x60/0x100 [ 7320.364127] [] __drain_alien_cache+0x60/0x100 [ 7320.364127] [] cache_reap+0x172/0x260 [ 7320.364127] [] process_one_work+0x1fb/0x4f0 [ 7320.364127] [] ? process_one_work+0x138/0x4f0 [ 7320.364127] [] ? worker_thread+0x60/0x420 [ 7320.364127] [] ? drain_freelist+0xd0/0xd0 [ 7320.364127] [] worker_thread+0x183/0x420 [ 7320.364127] [] ? manage_workers+0x120/0x120 [ 7320.364127] [] kthread+0x9e/0xb0 [ 7320.364127] [] kernel_thread_helper+0x4/0x10 [ 7320.364127] [] ? retint_restore_args+0x13/0x13 [ 7320.364127] [] ? __init_kthread_worker+0x70/0x70 [ 7320.364127] [] ? gs_change+0x13/0x13 Regards, Srivatsa S. Bhat IBM Linux Technology Center -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: email@kvack.org