Re: 2.6.24-rc6-mm1 Kernel panics at different functions ()

[Kamalesh Babulal <kamalesh@linux.vnet.ibm.com>]

Andrew Morton wrote:
> On Thu, 27 Dec 2007 14:19:54 +0530 Kamalesh Babulal <kamalesh@linux.vnet.ibm.com> wrote:
> 
>> Hi Andrew,
>>
>> The 2.6.24-rc6-mm1 kernel with hotfix x86-fix-system-gate-related-crash.patch applied
>> panics while booting on a x86_64 box
>>
>> Unable to handle kernel NULL pointer dereference at 0000000000000046 RIP: 
>>  [<ffffffff80369a0b>] rb_erase+0xe7/0x2a3
>> PGD 17ff65067 PUD 17f1c7067 PMD 0 
>> Oops: 0000 [1] SMP 
>> last sysfs file: /sys/devices/pci0000:00/0000:00:0a.0/0000:02:04.0/host0/target0:0:6/0:0:6:0/type
>> CPU 0 
>> Modules linked in:
>> Pid: 0, comm: swapper Not tainted 2.6.24-rc6-mm1-autokern1 #1
>> RIP: 0010:[<ffffffff80369a0b>]  [<ffffffff80369a0b>] rb_erase+0xe7/0x2a3
>> RSP: 0000:ffffffff80650e00  EFLAGS: 00010002
>> RAX: ffff8101fe9568c8 RBX: ffff8100010062a8 RCX: ffff8101fe9568b0
>> RDX: ffff8101fe9568c8 RSI: 0000000000000046 RDI: 0000000000000000
>> RBP: ffffffff80650e10 R08: ffff8101fe9568c8 R09: 0000000000000086
>> R10: 0000000000000000 R11: 00000000000001e8 R12: ffff8100010062b8
>> R13: 0000000000000002 R14: ffff810001006260 R15: 0000000000000001
>> FS:  0000000000000000(0000) GS:ffffffff805dc000(0000) knlGS:00000000f31ffbb0
>> CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
>> CR2: 0000000000000046 CR3: 000000017f0ab000 CR4: 00000000000006e0
>> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Process swapper (pid: 0, threadinfo ffffffff805f6000, task ffffffff805a2080)
>> Stack:  ffff8100010062a8 ffff8101fe9568b0 ffffffff80650e40 ffffffff8024be16
>>  ffffffff80369d65 ffffffff80369d65 ffff8101fe9568b0 ffff8100010062a8
>>  ffffffff80650eb0 ffffffff8024c1d5 ffffffffb88cc28e 0000000006e73eff
>> Call Trace:
>>  <IRQ>  [<ffffffff8024be16>] __remove_hrtimer+0x2e/0x3c
>>  [<ffffffff80369d65>] __down_read_trylock+0x16/0x42
>>  [<ffffffff80369d65>] __down_read_trylock+0x16/0x42
>>  [<ffffffff8024c1d5>] hrtimer_run_queues+0x130/0x191
>>  [<ffffffff8023fd09>] run_timer_softirq+0x28/0x1a7
>>  [<ffffffff8023c018>] __do_softirq+0x55/0xc2
>>  [<ffffffff8020c73c>] call_softirq+0x1c/0x28
>>  [<ffffffff8020e719>] do_softirq+0x32/0x9d
>>  [<ffffffff8023c0dd>] irq_exit+0x3f/0x41
>>  [<ffffffff8021ff85>] smp_apic_timer_interrupt+0x92/0xa7
>>  [<ffffffff8020c1e6>] apic_timer_interrupt+0x66/0x70
>>  <EOI>  [<ffffffff802095f5>] default_idle+0x36/0x5e
>>  [<ffffffff802095f0>] default_idle+0x31/0x5e
>>  [<ffffffff802095bf>] default_idle+0x0/0x5e
>>  [<ffffffff802096b6>] cpu_idle+0x90/0xb2
>>  [<ffffffff804b0126>] rest_init+0x5a/0x5c
>>  [<ffffffff806017ee>] start_kernel+0x2b8/0x2c4
>>  [<ffffffff8060112b>] _sinittext+0x12b/0x132
>>
>>
> 
> It does seem to be mostly hrtimer-related.  But surely the hrtimer system
> is initialised by the time tis happens.
> 
> The usual refrain: is it possible to run a bisection search?
Hi Andrew,

While doing the git bisect, following panic was seen

Unable to handle kernel paging request at 000000000000401e RIP: 
 [<ffffffff80232ec8>] load_balance_monitor+0x15e/0x2a4
PGD 0 
Oops: 0000 [1] SMP 
last sysfs file: /devices/pci0000:00/0000:00:0a.0/0000:02:04.0/host0/target0:0:6/0:0:6:0/type
CPU 1 
Modules linked in:
Pid: 15, comm: load_balance_mo Not tainted 2.6.24-rc6-mm1-autokern1 #1
RIP: 0010:[<ffffffff80232ec8>]  [<ffffffff80232ec8>] load_balance_monitor+0x15e/0x2a4
RSP: 0000:ffff81007ffb7eb0  EFLAGS: 00010297
RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000
RDX: 000000000000401e RSI: ffff81007ffb7ed8 RDI: 0000000000000000
RBP: ffff81007ffb7f20 R08: ffff81007ffb6000 R09: ffff81007ffb6000
R10: ffff81007ffb6000 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000003 R14: 0000000000000800 R15: ffff8101fe997f00
FS:  0000000000000000(0000) GS:ffff8100e3b10000(0000) knlGS:00000000f73e1bb0
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 000000000000401e CR3: 0000000000201000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process load_balance_mo (pid: 15, threadinfo ffff81007ffb6000, task ffff81007ff94790)
Stack:  0000000000002000 0000000000000000 ffff810001009cc0 00000001e3b29d90
 0000008000000000 000000000000000f ffff81007f0be780 000000000000000f
 000000017ffb7f20 0000000000000000 00000000fffffffc ffffffffffffffff
Call Trace:
 [<ffffffff80232d6a>] load_balance_monitor+0x0/0x2a4
 [<ffffffff80247830>] kthread+0x3d/0x63
 [<ffffffff8020c2b8>] child_rip+0xa/0x12
 [<ffffffff802477f3>] kthread+0x0/0x63
 [<ffffffff8020c2ae>] child_rip+0x0/0x12


Code: 48 8b 04 c2 48 8b 10 48 01 55 98 e8 ce 40 12 00 83 f8 07 41 
RIP  [<ffffffff80232ec8>] load_balance_monitor+0x15e/0x2a4
 RSP <ffff81007ffb7eb0>
CR2: 000000000000401e


The git-sched.patch is causing this panic, and i am searching for the patch causing the
hrtimer-related panic.
 
-- 
Thanks & Regards,
Kamalesh Babulal,
Linux Technology Center,
IBM, ISTL.