From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1759405AbZBMIbi (ORCPT ); Fri, 13 Feb 2009 03:31:38 -0500 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1751496AbZBMIbZ (ORCPT ); Fri, 13 Feb 2009 03:31:25 -0500 Received: from mx2.mail.elte.hu ([157.181.151.9]:50665 "EHLO mx2.mail.elte.hu" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750774AbZBMIbY (ORCPT ); Fri, 13 Feb 2009 03:31:24 -0500 Date: Fri, 13 Feb 2009 09:31:06 +0100 From: Ingo Molnar To: Andrew Morton Cc: Vegard Nossum , linux-kernel@vger.kernel.org, linux-usb@vger.kernel.org, Jens Axboe , linux-scsi@vger.kernel.org Subject: Re: 2.6.29-rc3: BUG: NMI Watchdog detected LOCKUP Message-ID: <20090213083106.GD4606@elte.hu> References: <19f34abd0902080221w662635f0h51875a125b156535@mail.gmail.com> <20090212161908.2cc2045c.akpm@linux-foundation.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20090212161908.2cc2045c.akpm@linux-foundation.org> User-Agent: Mutt/1.5.18 (2008-05-17) X-ELTE-VirusStatus: clean X-ELTE-SpamScore: -1.5 X-ELTE-SpamLevel: X-ELTE-SpamCheck: no X-ELTE-SpamVersion: ELTE 2.0 X-ELTE-SpamCheck-Details: score=-1.5 required=5.9 tests=BAYES_00 autolearn=no SpamAssassin version=3.2.3 -1.5 BAYES_00 BODY: Bayesian spam probability is 0 to 1% [score: 0.0000] Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org * Andrew Morton wrote: > > [ 3344.972984] Call Trace: > > [ 3344.972984] [] __make_request+0x3e/0x412 > > [ 3344.972984] [] generic_make_request+0x279/0x2c3 > > [ 3344.972984] [] ? radix_tree_tag_set+0x6b/0xce > > [ 3344.972984] [] submit_bio+0xc6/0xcf > > [ 3344.972984] [] ? unlock_page+0x22/0x26 > > [ 3344.972984] [] swap_writepage+0xa2/0xac > > [ 3344.972984] [] shrink_page_list+0x3a7/0x67b > > [ 3344.972984] [] ? finish_task_switch+0x68/0x88 > > [ 3344.972984] [] ? __cpus_empty+0x9/0xb > > [ 3344.972984] [] ? flush_tlb_page+0x66/0x83 > > [ 3344.972984] [] ? thread_return+0x3d/0xc6 > > [ 3344.972984] [] shrink_list+0x29d/0x59f > > [ 3344.972984] [] ? get_dirty_limits+0x22/0x24a > > [ 3344.972984] [] shrink_zone+0x281/0x32b > > [ 3344.972984] [] ? __up_read+0x92/0x9c > > [ 3344.972984] [] ? shrink_slab+0x146/0x158 > > [ 3344.972984] [] try_to_free_pages+0x23d/0x38f > > [ 3344.972984] [] ? isolate_pages_global+0x0/0x219 > > [ 3344.972984] [] __alloc_pages_internal+0x292/0x43d > > [ 3344.972984] [] alloc_pages_current+0xb9/0xc2 > > [ 3344.972984] [] alloc_slab_page+0x19/0x69 > > [ 3344.972984] [] new_slab+0x49/0x1cc > > [ 3344.972984] [] ? rb_insert_color+0xbd/0xe6 > > [ 3344.972984] [] __slab_alloc+0x1f3/0x36c > > [ 3344.972984] [] ? __alloc_skb+0x42/0x130 > > [ 3344.972984] [] ? __alloc_skb+0x42/0x130 > > [ 3344.972984] [] kmem_cache_alloc_node+0x69/0xa2 > > [ 3344.972984] [] __alloc_skb+0x42/0x130 > > [ 3344.972984] [] sock_alloc_send_skb+0xa1/0x200 > > [ 3344.972984] [] ? security_socket_getpeersec_dgram+0x11/0x3 > > [ 3344.972984] [] unix_stream_sendmsg+0x138/0x2b5 > > [ 3344.972984] [] __sock_sendmsg+0x59/0x62 > > [ 3344.972984] [] sock_aio_write+0xe8/0xf8 > > [ 3344.972984] [] do_sync_write+0xe7/0x12d > > [ 3344.972984] [] ? autoremove_wake_function+0x0/0x38 > > [ 3344.972984] [] ? selinux_file_permission+0xbd/0xc6 > > [ 3344.972984] [] ? security_file_permission+0x11/0x13 > > [ 3344.972984] [] vfs_write+0xbe/0x105 > > [ 3344.972984] [] sys_write+0x47/0x6f > > [ 3344.972984] [] sysenter_dispatch+0x7/0x27 > > [ 3344.972984] Code: 01 00 00 f0 66 0f c1 17 38 f2 74 06 f3 90 8a 17 eb f6 c9 c > > [ 3344.972984] BUG: NMI Watchdog detected LOCKUP<4>---[ end trace 820f38a7b2441- > > [ 3344.972984] on CPU0, ip ffffffff81491f6c, registers: > > And then the block layer died. Looks like it was trying to take the > queue lock. Probably against the recently-offlined device. > > I'd say that either someone forgot to release the lock on an error > path. Or the structure was freed, but the kernel still tries to use it. Should run with CONFIG_PROVE_LOCKING=y for more precise forensics about precisely who did that and when. Or if this was with lockdep enabled already, some other modes of failure should be considered too. (such as memory corruption) Ingo