From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
To: Kamalesh Babulal <kamalesh@linux.vnet.ibm.com>
Cc: Stephen Rothwell <sfr@canb.auug.org.au>,
linux-next@vger.kernel.org, LKML <linux-kernel@vger.kernel.org>,
dm-devel@redhat.com, tglx@linutronix.de, mel@csn.ul.ie
Subject: Re: [BUG] linux-next: 20081209 - kernel bug at __rcu_process_callbacks, while booting up
Date: Wed, 10 Dec 2008 06:54:14 -0800 [thread overview]
Message-ID: <20081210145414.GA6945@linux.vnet.ibm.com> (raw)
In-Reply-To: <20081210115721.GB6107@linux.vnet.ibm.com>
On Wed, Dec 10, 2008 at 05:27:21PM +0530, Kamalesh Babulal wrote:
> Hi,
>
> Kernel bug is hit while booting up the next-20081208/09 kernels over
> the x86_64 box. The IP is pointing to 0x0 and its stuck at
> __rcu_process_callbacks.
Kernel config?
Thanx, Paul
> Activating logical volumes
> 4 logical volume(s) in volume group "VolGroup00"BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
> IP: [<0000000000000000>] 0x0
> PGD 3e8ad067 PUD 3e8ac067 PMD 0
> Oops: 0010 [#1] SMP
> last sysfs file: /sys/block/dm-1/removable
> CPU 3
> Modules linked in:
> Pid: 0, comm: swapper Not tainted 2.6.28-rc7-next-20081209-autotest #1
> RIP: 0010:[<0000000000000000>] [<0000000000000000>] 0x0
> RSP: 0018:ffff88003fa73ef0 EFLAGS: 00010286
> RAX: ffff88003eae0500 RBX: ffff880001047040 RCX: ffffffff80268f66
> RDX: 0000000000000000 RSI: ffffe20000fab800 RDI: ffff88003eae0500
> RBP: 0000000000000000 R08: ffff880001047040 R09: ffffffff80853950
> R10: 0000000000000038 R11: ffffffff8032222d R12: 0000000000000005
> R13: 0000000000000038 R14: 0000000000000009 R15: 0000000000000018
> FS: 000000000066d870(0000) GS:ffff88003f803f00(0000) knlGS:0000000000000000
> CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
> CR2: 0000000000000000 CR3: 000000003e8a9000 CR4: 00000000000006e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process swapper (pid: 0, threadinfo ffff88003fa64000, task ffff88003f9dde00)
> Stack:
> ffffffff80268f66 ffff88003fa73ef8 0000000000000100 0000000000000001
> ffffffff807400b8 0000000000000038 ffffffff80269001 0000000000000018
> ffffffff8023b891 ffffffff8020e79e 0000000000000046 ffff88003fa73f80
> Call Trace:
> <IRQ> <0> [<ffffffff80268f66>] __rcu_process_callbacks+0x14e/0x1c3
> [<ffffffff80269001>] rcu_process_callbacks+0x26/0x4a
> [<ffffffff8023b891>] __do_softirq+0x76/0x136
> [<ffffffff8020e79e>] profile_pc+0x21/0x5b
> [<ffffffff8020d03c>] call_softirq+0x1c/0x28
> [<ffffffff8020e091>] do_softirq+0x2c/0x6c
> [<ffffffff8021e018>] smp_apic_timer_interrupt+0x93/0xac
> [<ffffffff8020ca73>] apic_timer_interrupt+0x13/0x20
> <EOI> <0>Code: Bad RIP value.
> RIP [<0000000000000000>] 0x0
> RSP <ffff88003fa73ef0>
> CR2: 0000000000000000
> ---[ end trace 69ecde41a682e571 ]---
> Kernel panic - not syncing: Fatal exception in interrupt
> Pid: 0, comm: swapper Tainted: G D 2.6.28-rc7-next-20081209-autotest #1
> Call Trace:
> Mounting root fi <IRQ> lesystem.
> [<ffffffff80236e7d>] panic+0x86/0x144
> [<ffffffff8020ca73>] apic_timer_interrupt+0x13/0x20
> [<ffffffff8051fd57>] oops_end+0x61/0xad
> [<ffffffff8051fd96>] oops_end+0xa0/0xad
> [<ffffffff805215ac>] do_page_fault+0x748/0x801
> [<ffffffff8051f3af>] page_fault+0x1f/0x30
> [<ffffffff8032222d>] selinux_cred_free+0x0/0x14
> BUG: unable to handle kernel NULL pointer dereference at 0000000000000002
> IP: [<ffffffff802926da>] kmem_cache_alloc+0x6a/0x99
> PGD 3e8ad067 PUD 3e8ac067 PMD 0
> Oops: 0000 [#2] SMP
> last sysfs file: /sys/block/ram7/removable
> CPU 2
> Modules linked in:
> Pid: 1, comm: init Tainted: G D 2.6.28-rc7-next-20081209-autotest #1
> RIP: 0010:[<ffffffff802926da>] [<ffffffff802926da>] kmem_cache_alloc+0x6a/0x99
> RSP: 0018:ffff88003f9d7db8 EFLAGS: 00010002
> RAX: 0000000000000000 RBX: 0000000000000246 RCX: 0000000000000001
> RDX: ffff880001037800 RSI: 0000000000000002 RDI: ffffffff80854f70
> RBP: 0000000000000020 R08: 0000000000000000 R09: ffffffff80859a80
> R10: 0000000000000001 R11: ffff88003f426018 R12: ffffffff80854f70
> R13: ffffffff80253ab1 R14: 0000000000000080 R15: ffffffff802b4141
> FS: 000000000066d870(0063) GS:ffff88003f803c80(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 0000000000000002 CR3: 000000003e8a9000 CR4: 00000000000006e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process init (pid: 1, threadinfo ffff88003f9d6000, task ffff88003f9d8000)
> Stack:
> 00000000ffffffff ffff88003f426000 ffffffff80859a80 ffffffff80859a80
> 0000000000000000 ffffffff80253ab1 00000000006f1db0 0000000000000000
> ffff88003eb6a2f8 ffff88003f9d7e48 0000000000000000 01ff88003e97c400
> Call Trace:
> [<ffffffff80253ab1>] ? smp_call_function_many+0xdb/0x220
> [<ffffffff802704ca>] ? do_writepages+0x23/0x30
> [<ffffffff802b4141>] ? invalidate_bh_lru+0x0/0x42
> [<ffffffff80253c16>] ? smp_call_function+0x20/0x24
> [<ffffffff8023b483>] ? on_each_cpu+0x10/0x22
> [<ffffffff802b919b>] ? kill_bdev+0x1b/0x30
> [<ffffffff802b9892>] ? __blkdev_put+0x54/0x151
> [<ffffffff8029899f>] ? __fput+0xe7/0x1b4
> [<ffffffff802961ea>] ? filp_close+0x5e/0x66
> [<ffffffff8029627f>] ? sys_close+0x8d/0xd1
> [<ffffffff8020bf5b>] ? system_call_fastpath+0x16/0x1b
> Code: 18 03 00 00 48 8b 32 44 8b 72 18 48 85 f6 75 18 49 89 d0 89 ee 4c 89 e9 83 ca ff 4c 89 e7 e8 57 f8 ff ff 48 89 c6 eb 0a 8b 42 14 <48> 8b 04 c6 48 89 02 53 9d 31 c0 c1 ed 0f 48 85 f6 0f 95 c0 85
> RIP [<ffffffff802926da>] kmem_cache_alloc+0x6a/0x99
> RSP <ffff88003f9d7db8>
> CR2: 0000000000000002
> ---[ end trace 69ecde41a682e571 ]---
> Kernel panic - not syncing: Attempted to kill init!
> Pid: 1, comm: init Tainted: G D 2.6.28-rc7-next-20081209-autotest #1
> Call Trace:
> [<ffffffff80236e7d>] panic+0x86/0x144
> [<ffffffff802abbfb>] mntput_no_expire+0x1e/0x139
> [<ffffffff802961ea>] filp_close+0x5e/0x66
> [<ffffffff8023870d>] exit_fs+0x35/0x46
> [<ffffffff80239b7a>] do_exit+0x75/0x766
> [<ffffffff8051fd9e>] oops_end+0xa8/0xad
> [<ffffffff805215ac>] do_page_fault+0x748/0x801
> [<ffffffff80253ab1>] smp_call_function_many+0xdb/0x220
> [<ffffffff802b4141>] invalidate_bh_lru+0x0/0x42
> [<ffffffff8051f3af>] page_fault+0x1f/0x30
> [<ffffffff802b4141>] invalidate_bh_lru+0x0/0x42
> [<ffffffff80253ab1>] smp_call_function_many+0xdb/0x220
> [<ffffffff802926da>] kmem_cache_alloc+0x6a/0x99
> [<ffffffff80253ab1>] smp_call_function_many+0xdb/0x220
> [<ffffffff802704ca>] do_writepages+0x23/0x30
> [<ffffffff802b4141>] invalidate_bh_lru+0x0/0x42
> [<ffffffff80253c16>] smp_call_function+0x20/0x24
> [<ffffffff8023b483>] on_each_cpu+0x10/0x22
> [<ffffffff802b919b>] kill_bdev+0x1b/0x30
> [<ffffffff802b9892>] __blkdev_put+0x54/0x151
> [<ffffffff8029899f>] __fput+0xe7/0x1b4
> [<ffffffff802961ea>] filp_close+0x5e/0x66
> [<ffffffff8029627f>] sys_close+0x8d/0xd1
> [<ffffffff8020bf5b>] system_call_fastpath+0x16/0x1b
> BUG: unable to handle kernel NULL pointer dereference at 0000000000000002
> IP: [<ffffffff802926da>] kmem_cache_alloc+0x6a/0x99
> PGD 0
> Oops: 0000 [#3] SMP
> last sysfs file: /sys/block/ram7/removable
> CPU 2
> Modules linked in:
> Pid: 1, comm: init Tainted: G D 2.6.28-rc7-next-20081209-autotest #1
> RIP: 0010:[<ffffffff802926da>] [<ffffffff802926da>] kmem_cache_alloc+0x6a/0x99
> RSP: 0018:ffff88003f9d7a58 EFLAGS: 00010002
> RAX: 0000000000000000 RBX: 0000000000000246 RCX: 0000000000000001
> RDX: ffff880001037800 RSI: 0000000000000002 RDI: ffffffff80854f70
> RBP: 0000000000000020 R08: ffffffff8052a5c0 R09: ffffffff80859a80
> R10: 0000000000000000 R11: ffffffff8067c887 R12: ffffffff80854f70
> R13: ffffffff80253ab1 R14: 0000000000000080 R15: ffffffff80211dd3
> FS: 000000000066d870(0000) GS:ffff88003f803c80(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 0000000000000002 CR3: 0000000000201000 CR4: 00000000000006e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process init (pid: 1, threadinfo ffff88003f9d6000, task ffff88003f9d8000)
> Stack:
> 00000000ffffffff ffff88003f9d8000 ffffffff80859a80 ffffffff80859a80
> 0000000000000000 ffffffff80253ab1 ffffffff8052a5c0 ffff88003fa3bfc0
> 0000000000000000 ffffffff8020e525 0000000000000010 000000003e943800
> Call Trace:
> [<ffffffff80253ab1>] ? smp_call_function_many+0xdb/0x220
> [<ffffffff8020e525>] ? dump_trace+0x223/0x232
> [<ffffffff80253c16>] ? smp_call_function+0x20/0x24
> [<ffffffff8021ccc1>] ? native_smp_send_stop+0x1a/0x26
> [<ffffffff80236e91>] ? panic+0x9a/0x144
> [<ffffffff802abbfb>] ? mntput_no_expire+0x1e/0x139
> [<ffffffff802961ea>] ? filp_close+0x5e/0x66
> [<ffffffff8023870d>] ? exit_fs+0x35/0x46
> [<ffffffff80239b7a>] ? do_exit+0x75/0x766
> [<ffffffff8051fd9e>] ? oops_end+0xa8/0xad
> [<ffffffff805215ac>] ? do_page_fault+0x748/0x801
> [<ffffffff80253ab1>] ? smp_call_function_many+0xdb/0x220
> [<ffffffff802b4141>] ? invalidate_bh_lru+0x0/0x42
> [<ffffffff8051f3af>] ? page_fault+0x1f/0x30
> [<ffffffff802b4141>] ? invalidate_bh_lru+0x0/0x42
> [<ffffffff80253ab1>] ? smp_call_function_many+0xdb/0x220
> [<ffffffff802926da>] ? kmem_cache_alloc+0x6a/0x99
> [<ffffffff80253ab1>] ? smp_call_function_many+0xdb/0x220
> [<ffffffff802704ca>] ? do_writepages+0x23/0x30
> [<ffffffff802b4141>] ? invalidate_bh_lru+0x0/0x42
> [<ffffffff80253c16>] ? smp_call_function+0x20/0x24
> [<ffffffff8023b483>] ? on_each_cpu+0x10/0x22
> [<ffffffff802b919b>] ? kill_bdev+0x1b/0x30
> [<ffffffff802b9892>] ? __blkdev_put+0x54/0x151
> [<ffffffff8029899f>] ? __fput+0xe7/0x1b4
> [<ffffffff802961ea>] ? filp_close+0x5e/0x66
> [<ffffffff8029627f>] ? sys_close+0x8d/0xd1
> [<ffffffff8020bf5b>] ? system_call_fastpath+0x16/0x1b
> Code: 18 03 00 00 48 8b 32 44 8b 72 18 48 85 f6 75 18 49 89 d0 89 ee 4c 89 e9 83 ca ff 4c 89 e7 e8 57 f8 ff ff 48 89 c6 eb 0a 8b 42 14 <48> 8b 04 c6 48 89 02 53 9d 31 c0 c1 ed 0f 48 85 f6 0f 95 c0 85
> RIP [<ffffffff802926da>] kmem_cache_alloc+0x6a/0x99
> RSP <ffff88003f9d7a58>
> CR2: 0000000000000002
> ---[ end trace 69ecde41a682e571 ]---
> Fixing recursive fault but reboot is needed!
> [<ffffffff80268f66>] __rcu_process_callbacks+0x14e/0x1c3
> [<ffffffff80268f66>] __rcu_process_callbacks+0x14e/0x1c3
> [<ffffffff80269001>] rcu_process_callbacks+0x26/0x4a
> [<ffffffff8023b891>] __do_softirq+0x76/0x136
> [<ffffffff8020e79e>] profile_pc+0x21/0x5b
> [<ffffffff8020d03c>] call_softirq+0x1c/0x28
> [<ffffffff8020e091>] do_softirq+0x2c/0x6c
> [<ffffffff8021e018>] smp_apic_timer_interrupt+0x93/0xac
> [<ffffffff8020ca73>] apic_timer_interrupt+0x13/0x20
> <EOI>
>
> --
> Thanks & Regards,
> Kamalesh Babulal,
> Linux Technology Center,
> IBM, ISTL.
next prev parent reply other threads:[~2008-12-10 14:54 UTC|newest]
Thread overview: 26+ messages / expand[flat|nested] mbox.gz Atom feed top
2008-12-09 7:52 linux-next: Tree for December 9 Stephen Rothwell
2008-12-09 8:53 ` Al Viro
2008-12-09 13:57 ` byteorder headers on parisc Alexey Dobriyan
2008-12-09 23:12 ` Harvey Harrison
2008-12-09 23:34 ` linux-next: Tree for December 9 Andrew Morton
2008-12-10 0:06 ` Kay Sievers
2008-12-10 0:15 ` Andrew Morton
2008-12-10 0:31 ` Kay Sievers
2008-12-10 1:30 ` Andrew Morton
2008-12-10 1:36 ` Kay Sievers
2008-12-10 1:42 ` NeilBrown
2008-12-10 1:58 ` Andrew Morton
2008-12-10 19:16 ` Andrew Morton
2008-12-10 22:15 ` Neil Brown
2008-12-10 11:57 ` [BUG] linux-next: 20081209 - kernel bug at __rcu_process_callbacks, while booting up Kamalesh Babulal
2008-12-10 14:54 ` Paul E. McKenney [this message]
2008-12-10 16:30 ` Kamalesh Babulal
2008-12-10 17:53 ` Paul E. McKenney
2008-12-10 18:09 ` Kamalesh Babulal
2008-12-10 18:33 ` Paul E. McKenney
2008-12-12 19:40 ` Kamalesh Babulal
2008-12-12 22:16 ` Paul E. McKenney
2008-12-16 14:30 ` Kamalesh Babulal
2008-12-16 14:37 ` Jens Axboe
2008-12-17 11:12 ` Kamalesh Babulal
2008-12-17 11:18 ` Jens Axboe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20081210145414.GA6945@linux.vnet.ibm.com \
--to=paulmck@linux.vnet.ibm.com \
--cc=dm-devel@redhat.com \
--cc=kamalesh@linux.vnet.ibm.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-next@vger.kernel.org \
--cc=mel@csn.ul.ie \
--cc=sfr@canb.auug.org.au \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.