Re: 2.6.24-rc3-mm1 - Kernel Panic on IO-APIC

Re: 2.6.24-rc3-mm1 - Kernel Panic on IO-APIC

[Andrew Morton]

On Wed, 21 Nov 2007 11:41:23 +0530 Kamalesh Babulal <kamalesh@linux.vnet.ibm.com> wrote:

> Hi Andrew,
> 
> Kernel panic's across different architectures like powerpc, x86_64, 

powerpc complains about IO-APICs??

> Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes)
> Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes)
> Mount-cache hash table entries: 256
> SMP alternatives: switching to UP code
> ACPI: Core revision 20070126
> ..MP-BIOS bug: 8254 timer not connected to IO-APIC
> Kernel panic - not syncing: IO-APIC + timer doesn't work! Try using the 'noapic' kernel parameter

ACPI or x86 breakage, I guess.

Did 'noapic' work?

Re: 2.6.24-rc3-mm1 - Kernel Panic on IO-APIC

[Kamalesh Babulal]

Andrew Morton wrote:
> On Wed, 21 Nov 2007 11:41:23 +0530 Kamalesh Babulal <kamalesh@linux.vnet.ibm.com> wrote:
> 
>> Hi Andrew,
>>
>> Kernel panic's across different architectures like powerpc, x86_64, 
> 
> powerpc complains about IO-APICs??
> 
>> Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes)
>> Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes)
>> Mount-cache hash table entries: 256
>> SMP alternatives: switching to UP code
>> ACPI: Core revision 20070126
>> ..MP-BIOS bug: 8254 timer not connected to IO-APIC
>> Kernel panic - not syncing: IO-APIC + timer doesn't work! Try using the 'noapic' kernel parameter
> 
> ACPI or x86 breakage, I guess.
> 
> Did 'noapic' work?
Hi Andrew,

Passing noapic works, but the kernel oops's 

[   97.161103] Unable to handle kernel NULL pointer dereference at 0000000000000009 RIP:
[   97.193973]  [<ffffffff802341df>] cpu_to_allnodes_group+0x69/0x7c
[   97.245359] PGD 0
[   97.257611] Oops: 0000 [1] SMP
[   97.276638] last sysfs file:
[   97.294417] CPU 0
[   97.306620] Modules linked in:
[   97.325066] Pid: 1, comm: swapper Not tainted 2.6.24-rc3-mm1 #1
[   97.360514] RIP: 0010:[<ffffffff802341df>]  [<ffffffff802341df>] cpu_to_allnodes_group+0x69/0x7c
[   97.413287] RSP: 0000:ffff81012fabb650  EFLAGS: 00010286
[   97.445363] RAX: ffffffff809bb060 RBX: ffff81012fabb650 RCX: 00000000000000ff
[   97.488378] RDX: 0000000000000001 RSI: 000000000000013e RDI: 0000000000000100
[   97.531413] RBP: ffff81012fabb680 R08: ffff81012fa88180 R09: 0000000000000000
[   97.574428] R10: 0000000000000000 R11: 0000000000000000 R12: ffff810001005f50
[   97.617394] R13: 0000000000000000 R14: ffff81012fa88180 R15: ffff810001005f40
[   97.660421] FS:  0000000000000000(0000) GS:ffffffff806c3000(0000) knlGS:0000000000000000
[   97.709327] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[   97.743995] CR2: 0000000000000009 CR3: 0000000000201000 CR4: 00000000000006a0
[   97.787021] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   97.830053] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[   97.873036] Process swapper (pid: 1, threadinfo FFFF81012FABA000, task FFFF81012FAB8040)
[   97.921993] Stack:  0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   97.971056]  ffff810001005f40 ffff81012fabb700 ffff81012fabbdf0 ffffffff80235487
[   98.016420]  0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   98.060324] Call Trace:
[   98.076657]  [<ffffffff80235487>] build_sched_domains+0x1e1/0xc19
[   98.113383]  [<ffffffff8025072a>] __kernel_text_address+0x22/0x30
[   98.150173]  [<ffffffff8025b127>] check_chain_key+0x9c/0x15f
[   98.184355]  [<ffffffff8025d544>] mark_lock+0x3b/0x5b3
[   98.215406]  [<ffffffff8025db06>] mark_held_locks+0x4a/0x6a
[   98.249027]  [<ffffffff80284f4a>] get_page_from_freelist+0x42a/0x77d
[   98.287362]  [<ffffffff8025dda1>] trace_hardirqs_on+0x198/0x1c3
[   98.323123]  [<ffffffff8028527a>] get_page_from_freelist+0x75a/0x77d
[   98.361429]  [<ffffffff8025d544>] mark_lock+0x3b/0x5b3
[   98.392427]  [<ffffffff8025b127>] check_chain_key+0x9c/0x15f
[   98.426621]  [<ffffffff8034d01a>] number+0x115/0x21f
[   98.456594]  [<ffffffff8025072a>] __kernel_text_address+0x22/0x30
[   98.493362]  [<ffffffff8020d80c>] dump_trace+0x248/0x25d
[   98.525493]  [<ffffffff8025b127>] check_chain_key+0x9c/0x15f
[   98.559678]  [<ffffffff8025f03f>] __lock_acquire+0xdee/0xf06
[   98.593868]  [<ffffffff8025b127>] check_chain_key+0x9c/0x15f
[   98.628038]  [<ffffffff8025b127>] check_chain_key+0x9c/0x15f
[   98.662225]  [<ffffffff8025b127>] check_chain_key+0x9c/0x15f
[   98.696370]  [<ffffffff8025f03f>] __lock_acquire+0xdee/0xf06
[   98.730563]  [<ffffffff8025b127>] check_chain_key+0x9c/0x15f
[   98.764689]  [<ffffffff8025d544>] mark_lock+0x3b/0x5b3
[   98.795767]  [<ffffffff8025db06>] mark_held_locks+0x4a/0x6a
[   98.829432]  [<ffffffff8034d01a>] number+0x115/0x21f
[   98.859460]  [<ffffffff804ca267>] kprobe_flush_task+0x63/0xa9
[   98.894166]  [<ffffffff8034ddbd>] vsnprintf+0x58f/0x5d5
[   98.925739]  [<ffffffff8034de6b>] sprintf+0x68/0x6a
[   98.955257]  [<ffffffff8025f1c9>] lock_acquire+0x72/0xe0
[   98.987363]  [<ffffffff8025ca45>] lock_acquired+0x57/0x1d4
[   99.020446]  [<ffffffff8025f430>] lock_release+0x67/0x21a
[   99.053079]  [<ffffffff8025b127>] check_chain_key+0x9c/0x15f
[   99.087261]  [<ffffffff8025d544>] mark_lock+0x3b/0x5b3
[   99.118328]  [<ffffffff8025d544>] mark_lock+0x3b/0x5b3
[   99.149394]  [<ffffffff80236330>] arch_init_sched_domains+0x27/0x69
[   99.187217]  [<ffffffff802a5a5f>] dbg_redzone2+0x2a/0x52
[   99.219320]  [<ffffffff802a656b>] cache_alloc_debugcheck_after+0x16e/0x1cb
[   99.260779]  [<ffffffff802a8083>] kmem_cache_alloc+0x15e/0x182
[   99.295944]  [<ffffffff80236365>] arch_init_sched_domains+0x5c/0x69
[   99.333768]  [<ffffffff8098e501>] sched_init_smp+0x27/0x113
[   99.367400]  [<ffffffff8034ff35>] __bitmap_weight+0x78/0x8d
[   99.401090]  [<ffffffff8097e631>] kernel_init+0x12d/0x315
[   99.433718]  [<ffffffff804c6f57>] _spin_unlock_irq+0x2b/0x30
[   99.467842]  [<ffffffff8025dda1>] trace_hardirqs_on+0x198/0x1c3
[   99.503534]  [<ffffffff8025dda1>] trace_hardirqs_on+0x198/0x1c3
[   99.539251]  [<ffffffff8020d1b8>] child_rip+0xa/0x12
[   99.569234]  [<ffffffff8020c8cf>] restore_args+0x0/0x30
[   99.600845]  [<ffffffff8097e504>] kernel_init+0x0/0x315
[   99.632426]  [<ffffffff8020d1ae>] child_rip+0x0/0x12
[   99.662455]
[   99.671637] INFO: lockdep is turned off.
[   99.695385]
[   99.695385] Code: 48 03 42 08 49 89 04 24 48 83 c4 20 89 c8 5b 41 5c c9 c3 55
[   99.750603] RIP  [<ffffffff802341df>] cpu_to_allnodes_group+0x69/0x7c
[   99.789632]  RSP <ffff81012fabb650>

-- 
Thanks & Regards,
Kamalesh Babulal,
Linux Technology Center,
IBM, ISTL.

Re: 2.6.24-rc3-mm1 - Kernel Panic on IO-APIC

[Andrew Morton]

On Wed, 21 Nov 2007 14:52:26 +0530 Kamalesh Babulal <kamalesh@linux.vnet.ibm.com> wrote:

> Andrew Morton wrote:
> > On Wed, 21 Nov 2007 11:41:23 +0530 Kamalesh Babulal <kamalesh@linux.vnet.ibm.com> wrote:
> > 
> >> Hi Andrew,
> >>
> >> Kernel panic's across different architectures like powerpc, x86_64, 
> > 
> > powerpc complains about IO-APICs??
> > 
> >> Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes)
> >> Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes)
> >> Mount-cache hash table entries: 256
> >> SMP alternatives: switching to UP code
> >> ACPI: Core revision 20070126
> >> ..MP-BIOS bug: 8254 timer not connected to IO-APIC
> >> Kernel panic - not syncing: IO-APIC + timer doesn't work! Try using the 'noapic' kernel parameter
> > 
> > ACPI or x86 breakage, I guess.
> > 
> > Did 'noapic' work?
> Hi Andrew,
> 
> Passing noapic works,

OK.

>  but the kernel oops's 
> 
> [   97.161103] Unable to handle kernel NULL pointer dereference at 0000000000000009 RIP:
> [   97.193973]  [<ffffffff802341df>] cpu_to_allnodes_group+0x69/0x7c
> [   97.245359] PGD 0
> [   97.257611] Oops: 0000 [1] SMP
> [   97.276638] last sysfs file:
> [   97.294417] CPU 0
> [   97.306620] Modules linked in:
> [   97.325066] Pid: 1, comm: swapper Not tainted 2.6.24-rc3-mm1 #1
> [   97.360514] RIP: 0010:[<ffffffff802341df>]  [<ffffffff802341df>] cpu_to_allnodes_group+0x69/0x7c
> [   97.413287] RSP: 0000:ffff81012fabb650  EFLAGS: 00010286
> [   97.445363] RAX: ffffffff809bb060 RBX: ffff81012fabb650 RCX: 00000000000000ff
> [   97.488378] RDX: 0000000000000001 RSI: 000000000000013e RDI: 0000000000000100
> [   97.531413] RBP: ffff81012fabb680 R08: ffff81012fa88180 R09: 0000000000000000
> [   97.574428] R10: 0000000000000000 R11: 0000000000000000 R12: ffff810001005f50
> [   97.617394] R13: 0000000000000000 R14: ffff81012fa88180 R15: ffff810001005f40
> [   97.660421] FS:  0000000000000000(0000) GS:ffffffff806c3000(0000) knlGS:0000000000000000
> [   97.709327] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
> [   97.743995] CR2: 0000000000000009 CR3: 0000000000201000 CR4: 00000000000006a0
> [   97.787021] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [   97.830053] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [   97.873036] Process swapper (pid: 1, threadinfo FFFF81012FABA000, task FFFF81012FAB8040)
> [   97.921993] Stack:  0000000000000000 0000000000000000 0000000000000000 0000000000000000
> [   97.971056]  ffff810001005f40 ffff81012fabb700 ffff81012fabbdf0 ffffffff80235487
> [   98.016420]  0000000000000000 0000000000000000 0000000000000000 0000000000000000
> [   98.060324] Call Trace:
> [   98.076657]  [<ffffffff80235487>] build_sched_domains+0x1e1/0xc19
> [   98.113383]  [<ffffffff8025072a>] __kernel_text_address+0x22/0x30
> [   98.150173]  [<ffffffff8025b127>] check_chain_key+0x9c/0x15f
> [   98.184355]  [<ffffffff8025d544>] mark_lock+0x3b/0x5b3
> [   98.215406]  [<ffffffff8025db06>] mark_held_locks+0x4a/0x6a
> [   98.249027]  [<ffffffff80284f4a>] get_page_from_freelist+0x42a/0x77d
> [   98.287362]  [<ffffffff8025dda1>] trace_hardirqs_on+0x198/0x1c3
> [   98.323123]  [<ffffffff8028527a>] get_page_from_freelist+0x75a/0x77d
> [   98.361429]  [<ffffffff8025d544>] mark_lock+0x3b/0x5b3
> [   98.392427]  [<ffffffff8025b127>] check_chain_key+0x9c/0x15f
> [   98.426621]  [<ffffffff8034d01a>] number+0x115/0x21f
> [   98.456594]  [<ffffffff8025072a>] __kernel_text_address+0x22/0x30
> [   98.493362]  [<ffffffff8020d80c>] dump_trace+0x248/0x25d
> [   98.525493]  [<ffffffff8025b127>] check_chain_key+0x9c/0x15f
> [   98.559678]  [<ffffffff8025f03f>] __lock_acquire+0xdee/0xf06
> [   98.593868]  [<ffffffff8025b127>] check_chain_key+0x9c/0x15f
> [   98.628038]  [<ffffffff8025b127>] check_chain_key+0x9c/0x15f
> [   98.662225]  [<ffffffff8025b127>] check_chain_key+0x9c/0x15f
> [   98.696370]  [<ffffffff8025f03f>] __lock_acquire+0xdee/0xf06
> [   98.730563]  [<ffffffff8025b127>] check_chain_key+0x9c/0x15f
> [   98.764689]  [<ffffffff8025d544>] mark_lock+0x3b/0x5b3
> [   98.795767]  [<ffffffff8025db06>] mark_held_locks+0x4a/0x6a
> [   98.829432]  [<ffffffff8034d01a>] number+0x115/0x21f
> [   98.859460]  [<ffffffff804ca267>] kprobe_flush_task+0x63/0xa9
> [   98.894166]  [<ffffffff8034ddbd>] vsnprintf+0x58f/0x5d5
> [   98.925739]  [<ffffffff8034de6b>] sprintf+0x68/0x6a
> [   98.955257]  [<ffffffff8025f1c9>] lock_acquire+0x72/0xe0
> [   98.987363]  [<ffffffff8025ca45>] lock_acquired+0x57/0x1d4
> [   99.020446]  [<ffffffff8025f430>] lock_release+0x67/0x21a
> [   99.053079]  [<ffffffff8025b127>] check_chain_key+0x9c/0x15f
> [   99.087261]  [<ffffffff8025d544>] mark_lock+0x3b/0x5b3
> [   99.118328]  [<ffffffff8025d544>] mark_lock+0x3b/0x5b3
> [   99.149394]  [<ffffffff80236330>] arch_init_sched_domains+0x27/0x69
> [   99.187217]  [<ffffffff802a5a5f>] dbg_redzone2+0x2a/0x52
> [   99.219320]  [<ffffffff802a656b>] cache_alloc_debugcheck_after+0x16e/0x1cb
> [   99.260779]  [<ffffffff802a8083>] kmem_cache_alloc+0x15e/0x182
> [   99.295944]  [<ffffffff80236365>] arch_init_sched_domains+0x5c/0x69
> [   99.333768]  [<ffffffff8098e501>] sched_init_smp+0x27/0x113
> [   99.367400]  [<ffffffff8034ff35>] __bitmap_weight+0x78/0x8d
> [   99.401090]  [<ffffffff8097e631>] kernel_init+0x12d/0x315
> [   99.433718]  [<ffffffff804c6f57>] _spin_unlock_irq+0x2b/0x30
> [   99.467842]  [<ffffffff8025dda1>] trace_hardirqs_on+0x198/0x1c3
> [   99.503534]  [<ffffffff8025dda1>] trace_hardirqs_on+0x198/0x1c3
> [   99.539251]  [<ffffffff8020d1b8>] child_rip+0xa/0x12
> [   99.569234]  [<ffffffff8020c8cf>] restore_args+0x0/0x30
> [   99.600845]  [<ffffffff8097e504>] kernel_init+0x0/0x315
> [   99.632426]  [<ffffffff8020d1ae>] child_rip+0x0/0x12
> [   99.662455]
> [   99.671637] INFO: lockdep is turned off.
> [   99.695385]
> [   99.695385] Code: 48 03 42 08 49 89 04 24 48 83 c4 20 89 c8 5b 41 5c c9 c3 55
> [   99.750603] RIP  [<ffffffff802341df>] cpu_to_allnodes_group+0x69/0x7c
> [   99.789632]  RSP <ffff81012fabb650>

urgh, mess.  Enabling frame pointers might help here.

But we're cc'ing the right guy ;)

Re: 2.6.24-rc3-mm1 - Kernel Panic on IO-APIC

[Kamalesh Babulal]

Andrew Morton wrote:
> On Wed, 21 Nov 2007 14:52:26 +0530 Kamalesh Babulal <kamalesh@linux.vnet.ibm.com> wrote:
> 
>> Andrew Morton wrote:
>>> On Wed, 21 Nov 2007 11:41:23 +0530 Kamalesh Babulal <kamalesh@linux.vnet.ibm.com> wrote:
>>>
>>>> Hi Andrew,
>>>>
>>>> Kernel panic's across different architectures like powerpc, x86_64, 
>>> powerpc complains about IO-APICs??
>>>
>>>> Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes)
>>>> Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes)
>>>> Mount-cache hash table entries: 256
>>>> SMP alternatives: switching to UP code
>>>> ACPI: Core revision 20070126
>>>> ..MP-BIOS bug: 8254 timer not connected to IO-APIC
>>>> Kernel panic - not syncing: IO-APIC + timer doesn't work! Try using the 'noapic' kernel parameter
>>> ACPI or x86 breakage, I guess.
>>>
>>> Did 'noapic' work?
>> Hi Andrew,
>>
>> Passing noapic works,
> 
> OK.
> 
>>  but the kernel oops's 
>>
>> [   97.161103] Unable to handle kernel NULL pointer dereference at 0000000000000009 RIP:
>> [   97.193973]  [<ffffffff802341df>] cpu_to_allnodes_group+0x69/0x7c
>> [   97.245359] PGD 0
>> [   97.257611] Oops: 0000 [1] SMP
>> [   97.276638] last sysfs file:
>> [   97.294417] CPU 0
>> [   97.306620] Modules linked in:
>> [   97.325066] Pid: 1, comm: swapper Not tainted 2.6.24-rc3-mm1 #1
>> [   97.360514] RIP: 0010:[<ffffffff802341df>]  [<ffffffff802341df>] cpu_to_allnodes_group+0x69/0x7c
>> [   97.413287] RSP: 0000:ffff81012fabb650  EFLAGS: 00010286
>> [   97.445363] RAX: ffffffff809bb060 RBX: ffff81012fabb650 RCX: 00000000000000ff
>> [   97.488378] RDX: 0000000000000001 RSI: 000000000000013e RDI: 0000000000000100
>> [   97.531413] RBP: ffff81012fabb680 R08: ffff81012fa88180 R09: 0000000000000000
>> [   97.574428] R10: 0000000000000000 R11: 0000000000000000 R12: ffff810001005f50
>> [   97.617394] R13: 0000000000000000 R14: ffff81012fa88180 R15: ffff810001005f40
>> [   97.660421] FS:  0000000000000000(0000) GS:ffffffff806c3000(0000) knlGS:0000000000000000
>> [   97.709327] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
>> [   97.743995] CR2: 0000000000000009 CR3: 0000000000201000 CR4: 00000000000006a0
>> [   97.787021] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> [   97.830053] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> [   97.873036] Process swapper (pid: 1, threadinfo FFFF81012FABA000, task FFFF81012FAB8040)
>> [   97.921993] Stack:  0000000000000000 0000000000000000 0000000000000000 0000000000000000
>> [   97.971056]  ffff810001005f40 ffff81012fabb700 ffff81012fabbdf0 ffffffff80235487
>> [   98.016420]  0000000000000000 0000000000000000 0000000000000000 0000000000000000
>> [   98.060324] Call Trace:
>> [   98.076657]  [<ffffffff80235487>] build_sched_domains+0x1e1/0xc19
>> [   98.113383]  [<ffffffff8025072a>] __kernel_text_address+0x22/0x30
>> [   98.150173]  [<ffffffff8025b127>] check_chain_key+0x9c/0x15f
>> [   98.184355]  [<ffffffff8025d544>] mark_lock+0x3b/0x5b3
>> [   98.215406]  [<ffffffff8025db06>] mark_held_locks+0x4a/0x6a
>> [   98.249027]  [<ffffffff80284f4a>] get_page_from_freelist+0x42a/0x77d
>> [   98.287362]  [<ffffffff8025dda1>] trace_hardirqs_on+0x198/0x1c3
>> [   98.323123]  [<ffffffff8028527a>] get_page_from_freelist+0x75a/0x77d
>> [   98.361429]  [<ffffffff8025d544>] mark_lock+0x3b/0x5b3
>> [   98.392427]  [<ffffffff8025b127>] check_chain_key+0x9c/0x15f
>> [   98.426621]  [<ffffffff8034d01a>] number+0x115/0x21f
>> [   98.456594]  [<ffffffff8025072a>] __kernel_text_address+0x22/0x30
>> [   98.493362]  [<ffffffff8020d80c>] dump_trace+0x248/0x25d
>> [   98.525493]  [<ffffffff8025b127>] check_chain_key+0x9c/0x15f
>> [   98.559678]  [<ffffffff8025f03f>] __lock_acquire+0xdee/0xf06
>> [   98.593868]  [<ffffffff8025b127>] check_chain_key+0x9c/0x15f
>> [   98.628038]  [<ffffffff8025b127>] check_chain_key+0x9c/0x15f
>> [   98.662225]  [<ffffffff8025b127>] check_chain_key+0x9c/0x15f
>> [   98.696370]  [<ffffffff8025f03f>] __lock_acquire+0xdee/0xf06
>> [   98.730563]  [<ffffffff8025b127>] check_chain_key+0x9c/0x15f
>> [   98.764689]  [<ffffffff8025d544>] mark_lock+0x3b/0x5b3
>> [   98.795767]  [<ffffffff8025db06>] mark_held_locks+0x4a/0x6a
>> [   98.829432]  [<ffffffff8034d01a>] number+0x115/0x21f
>> [   98.859460]  [<ffffffff804ca267>] kprobe_flush_task+0x63/0xa9
>> [   98.894166]  [<ffffffff8034ddbd>] vsnprintf+0x58f/0x5d5
>> [   98.925739]  [<ffffffff8034de6b>] sprintf+0x68/0x6a
>> [   98.955257]  [<ffffffff8025f1c9>] lock_acquire+0x72/0xe0
>> [   98.987363]  [<ffffffff8025ca45>] lock_acquired+0x57/0x1d4
>> [   99.020446]  [<ffffffff8025f430>] lock_release+0x67/0x21a
>> [   99.053079]  [<ffffffff8025b127>] check_chain_key+0x9c/0x15f
>> [   99.087261]  [<ffffffff8025d544>] mark_lock+0x3b/0x5b3
>> [   99.118328]  [<ffffffff8025d544>] mark_lock+0x3b/0x5b3
>> [   99.149394]  [<ffffffff80236330>] arch_init_sched_domains+0x27/0x69
>> [   99.187217]  [<ffffffff802a5a5f>] dbg_redzone2+0x2a/0x52
>> [   99.219320]  [<ffffffff802a656b>] cache_alloc_debugcheck_after+0x16e/0x1cb
>> [   99.260779]  [<ffffffff802a8083>] kmem_cache_alloc+0x15e/0x182
>> [   99.295944]  [<ffffffff80236365>] arch_init_sched_domains+0x5c/0x69
>> [   99.333768]  [<ffffffff8098e501>] sched_init_smp+0x27/0x113
>> [   99.367400]  [<ffffffff8034ff35>] __bitmap_weight+0x78/0x8d
>> [   99.401090]  [<ffffffff8097e631>] kernel_init+0x12d/0x315
>> [   99.433718]  [<ffffffff804c6f57>] _spin_unlock_irq+0x2b/0x30
>> [   99.467842]  [<ffffffff8025dda1>] trace_hardirqs_on+0x198/0x1c3
>> [   99.503534]  [<ffffffff8025dda1>] trace_hardirqs_on+0x198/0x1c3
>> [   99.539251]  [<ffffffff8020d1b8>] child_rip+0xa/0x12
>> [   99.569234]  [<ffffffff8020c8cf>] restore_args+0x0/0x30
>> [   99.600845]  [<ffffffff8097e504>] kernel_init+0x0/0x315
>> [   99.632426]  [<ffffffff8020d1ae>] child_rip+0x0/0x12
>> [   99.662455]
>> [   99.671637] INFO: lockdep is turned off.
>> [   99.695385]
>> [   99.695385] Code: 48 03 42 08 49 89 04 24 48 83 c4 20 89 c8 5b 41 5c c9 c3 55
>> [   99.750603] RIP  [<ffffffff802341df>] cpu_to_allnodes_group+0x69/0x7c
>> [   99.789632]  RSP <ffff81012fabb650>
> 
> urgh, mess.  Enabling frame pointers might help here.
> 
> But we're cc'ing the right guy ;)
> 

The kernel was compiled with frame pointers enabled.
-- 
Thanks & Regards,
Kamalesh Babulal,
Linux Technology Center,
IBM, ISTL.

Re: 2.6.24-rc3-mm1 - Kernel Panic on IO-APIC

[Len Brown]

On Wednesday 21 November 2007 01:18, Andrew Morton wrote:
> On Wed, 21 Nov 2007 11:41:23 +0530 Kamalesh Babulal <kamalesh@linux.vnet.ibm.com> wrote:

> > SMP alternatives: switching to UP code
> > ACPI: Core revision 20070126
> > ..MP-BIOS bug: 8254 timer not connected to IO-APIC

did previous kernels print this too?

> > Kernel panic - not syncing: IO-APIC + timer doesn't work! Try using the 'noapic' kernel parameter
> 
> ACPI or x86 breakage, I guess.

If you suspect ACPI breakage, then try "acpi=off" or "acpi=noirq".

thanks,
-Len

Re: 2.6.24-rc3-mm1 - Kernel Panic on IO-APIC

[Torsten Kaiser]

On Nov 21, 2007 10:29 AM, Andrew Morton <akpm@linux-foundation.org> wrote:
> On Wed, 21 Nov 2007 14:52:26 +0530 Kamalesh Babulal <kamalesh@linux.vnet.ibm.com> wrote:
>
> > Andrew Morton wrote:
> > > On Wed, 21 Nov 2007 11:41:23 +0530 Kamalesh Babulal <kamalesh@linux.vnet.ibm.com> wrote:
> > >> ACPI: Core revision 20070126
> > >> ..MP-BIOS bug: 8254 timer not connected to IO-APIC
> > >> Kernel panic - not syncing: IO-APIC + timer doesn't work! Try using the 'noapic' kernel parameter

I seen an identical error.

> > > ACPI or x86 breakage, I guess.
> > >
> > > Did 'noapic' work?
> >
> > Passing noapic works,
>
> OK.

Not for me. I get a similar oops, but then the kernel panics

> >  but the kernel oops's
> >
> > [   97.161103] Unable to handle kernel NULL pointer dereference at 0000000000000009 RIP:
> > [   97.193973]  [<ffffffff802341df>] cpu_to_allnodes_group+0x69/0x7c
[snip]
> urgh, mess.  Enabling frame pointers might help here.

CONFIG_FRAME_POINTER=y

The oops/panic that happens with noapic:
[   35.866758] Initializing CPU#3
[   35.868769] Stuck ??
[   35.874043] Inquiring remote APIC #3...
[   35.877896] ... APIC #3 ID: 03000000
[   35.881523] ... APIC #3 VERSION: 80050010
[   35.885587] ... APIC #3 SPIV: 000001ff
[   35.889390] Brought up 1 CPUs
[   35.892375] Unable to handle kernel NULL pointer dereference at
0000000000000009 RIP:
[   35.897868]  [<ffffffff8022fc5b>] cpu_to_allnodes_group+0x4b/0x60
[   35.906464] PGD 0
[   35.908523] Oops: 0000 [1] SMP
[   35.911757] last sysfs file:
[   35.914740] CPU 0
[   35.916798] Modules linked in:
[   35.919990] Pid: 1, comm: swapper Not tainted 2.6.24-rc3-mm1 #2
[   35.925914] RIP: 0010:[<ffffffff8022fc5b>]  [<ffffffff8022fc5b>]
cpu_to_allnodes_group+0x4b/0x60
[   35.934734] RSP: 0000:ffff81011ff2bdb0  EFLAGS: 00010282
[   35.940053] RAX: ffffffff8084d870 RBX: ffff810001005810 RCX: 0000000000000004
[   35.947188] RDX: 0000000000000001 RSI: ffff81011ff26f68 RDI: ffff81011ff2bdb0
[   35.954323] RBP: ffff81011ff2bdd0 R08: 2222222222222222 R09: 0000000000000000
[   35.961457] R10: ffff81007ff1c200 R11: 0000000000000200 R12: ffff810001005800
[   35.968592] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[   35.975727] FS:  0000000000000000(0000) GS:ffffffff807d4000(0000)
knlGS:0000000000000000
[   35.983951] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[   35.989701] CR2: 0000000000000009 CR3: 0000000000201000 CR4: 00000000000006a0
[   35.996836] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   36.003971] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[   36.011105] Process swapper (pid: 1, threadinfo FFFF81011FF2A000,
task FFFF81007FF2A000)
[   36.019191] Stack:  0000000000000000 ffffffff807e8f98
0000000000000000 ffff810001005800
[   36.027373]  ffff81011ff2be80 ffffffff80230580 ffffffff8084d640
ffffffff8084d6e0
[   36.034922]  ffffffff8084d780 ffffffff8084d800 ffffffffffffffff
ffff81011ff26f68
[   36.042247] Call Trace:
[   36.044929]  [<ffffffff80230580>] build_sched_domains+0x460/0x820
[   36.051701]  [<ffffffff805cf489>] mutex_lock_nested+0x199/0x2e0
[   36.057624]  [<ffffffff80230991>] arch_init_sched_domains+0x51/0x60
[   36.063895]  [<ffffffff8080e422>] sched_init_smp+0x22/0xe0
[   36.069385]  [<ffffffff80806825>] smp_cpus_done+0x25/0x30
[   36.074791]  [<ffffffff807fb739>] kernel_init+0x109/0x350
[   36.080196]  [<ffffffff8025a34f>] trace_hardirqs_on+0xbf/0x160
[   36.086032]  [<ffffffff805d03d2>] trace_hardirqs_on_thunk+0x35/0x3a
[   36.092303]  [<ffffffff8025a34f>] trace_hardirqs_on+0xbf/0x160
[   36.098141]  [<ffffffff8020cbc8>] child_rip+0xa/0x12
[   36.103113]  [<ffffffff8020c2df>] restore_args+0x0/0x30
[   36.108345]  [<ffffffff807fb630>] kernel_init+0x0/0x350
[   36.113750]  [<ffffffff8020cbbe>] child_rip+0x0/0x12
[   36.118722]
[   36.120236] INFO: lockdep is turned off.
[   36.124170]
[   36.124170] Code: 48 03 42 08 48 89 03 48 83 c4 18 89 c8 5b c9 c3
0f 1f 44 00
[   36.133640] RIP  [<ffffffff8022fc5b>] cpu_to_allnodes_group+0x4b/0x60
[   36.140116]  RSP <ffff81011ff2bdb0>
[   36.143619] CR2: 0000000000000009
[   36.146952] Kernel panic - not syncing: Attempted to kill init!

(gdb) list *0xffffffff8022fc5b
0xffffffff8022fc5b is in cpu_to_allnodes_group (kernel/sched.c:6073).
6068
6069            cpus_and(nodemask, nodemask, *cpu_map);
6070            group = first_cpu(nodemask);
6071
6072            if (sg)
6073                    *sg = &per_cpu(sched_group_allnodes, group);
6074            return group;
6075    }
6076
6077    static void init_numa_sched_groups_power(struct sched_group *group_head)

Hope this stack trace is better.

Torsten

Re: 2.6.24-rc3-mm1 - Kernel Panic on IO-APIC

[Torsten Kaiser]

On Nov 21, 2007 8:22 PM, Len Brown <lenb@kernel.org> wrote:
> On Wednesday 21 November 2007 01:18, Andrew Morton wrote:
> > On Wed, 21 Nov 2007 11:41:23 +0530 Kamalesh Babulal <kamalesh@linux.vnet.ibm.com> wrote:
>
> > > SMP alternatives: switching to UP code
> > > ACPI: Core revision 20070126
> > > ..MP-BIOS bug: 8254 timer not connected to IO-APIC
>
> did previous kernels print this too?

Not since my last BIOS upgrade.

This is from what dmesg's I still had laying around:
2.6.22-rc6-mm1: No
2.6.23-rc1-mm1, 2.6.23-rc2-mm1, 2.6.23-rc3-mm1: Yes
2.6.23-rc3-mm1 after BIOS upgrade: No
2.6.23-rc4-mm1...2.6.24-rc2-mm1: No

> > > Kernel panic - not syncing: IO-APIC + timer doesn't work! Try using the 'noapic' kernel parameter
> >
> > ACPI or x86 breakage, I guess.
>
> If you suspect ACPI breakage, then try "acpi=off" or "acpi=noirq".

ACPI doesn't look guilty.
acpi=noirq:
[   39.905884] Freeing SMP alternatives: 28k freed
[   39.910674] ACPI: Core revision 20070126
[   39.916542] ACPI: setting ELCR to 0e20 (from 0c20)
[   39.921855] ExtINT not setup in hardware but reported by MP table
[   39.928244] ..MP-BIOS bug: 8254 timer not connected to IO-APIC
[   39.934586] Kernel panic - not syncing: IO-APIC + timer doesn't
work! Try using the 'noapic' kernel parameter
[   39.934587]

acpi=off:
[    0.000000] Freeing SMP alternatives: 28k freed
[    0.000000] ExtINT not setup in hardware but reported by MP table
[    0.000000] ..MP-BIOS bug: 8254 timer not connected to IO-APIC
[    0.000000] Kernel panic - not syncing: IO-APIC + timer doesn't
work! Try using the 'noapic' kernel parameter
[    0.000000]

Torsten

Re: 2.6.24-rc3-mm1 - Kernel Panic on IO-APIC

[Kirill A. Shutemov]

[-- Attachment #1: Type: text/plain, Size: 782 bytes --]

On [Wed, 21.11.2007 20:33], Torsten Kaiser wrote:
> On Nov 21, 2007 10:29 AM, Andrew Morton <akpm@linux-foundation.org> wrote:
> > On Wed, 21 Nov 2007 14:52:26 +0530 Kamalesh Babulal <kamalesh@linux.vnet.ibm.com> wrote:
> >
> > > Andrew Morton wrote:
> > > > On Wed, 21 Nov 2007 11:41:23 +0530 Kamalesh Babulal <kamalesh@linux.vnet.ibm.com> wrote:
> > > >> ACPI: Core revision 20070126
> > > >> ..MP-BIOS bug: 8254 timer not connected to IO-APIC
> > > >> Kernel panic - not syncing: IO-APIC + timer doesn't work! Try using the 'noapic' kernel parameter
> 
> I seen an identical error.

This bug is also reproducible with qemu.

-- 
Regards,  Kirill A. Shutemov
 + Belarus, Minsk
 + Velesys LLC, http://www.velesys.com/
 + ALT Linux Team, http://www.altlinux.com/

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: 2.6.24-rc3-mm1 - Kernel Panic on IO-APIC

[Alexey Dobriyan]

On Tue, Nov 20, 2007 at 10:18:39PM -0800, Andrew Morton wrote:
> On Wed, 21 Nov 2007 11:41:23 +0530 Kamalesh Babulal <kamalesh@linux.vnet.ibm.com> wrote:
> 
> > Hi Andrew,
> > 
> > Kernel panic's across different architectures like powerpc, x86_64, 
> 
> powerpc complains about IO-APICs??
> 
> > Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes)
> > Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes)
> > Mount-cache hash table entries: 256
> > SMP alternatives: switching to UP code
> > ACPI: Core revision 20070126

Hmm. same date here. It's Asus P5B-E motheboard

> > ..MP-BIOS bug: 8254 timer not connected to IO-APIC
> > Kernel panic - not syncing: IO-APIC + timer doesn't work! Try using the 'noapic' kernel parameter
> 
> ACPI or x86 breakage, I guess.
> 
> Did 'noapic' work?

No! The box freezes somewhere after "Freeing unused kernel memory"...

Bisection points to git-x86.patch, though.

git-bisect start
# good: [f05092637dc0d9a3f2249c9b283b973e6e96b7d2] Linux 2.6.24-rc3
git-bisect good f05092637dc0d9a3f2249c9b283b973e6e96b7d2
# bad: [46c8c396d2c87b786a7fac615c289f85a18e53ce] w1-build-fix
git-bisect bad 46c8c396d2c87b786a7fac615c289f85a18e53ce
# bad: [4e22f4852c48e1eddfe04299e78c0456164abe86] frv-move-dma-macros-to-scatterlisth-for-consistency
git-bisect bad 4e22f4852c48e1eddfe04299e78c0456164abe86
# bad: [4e22f4852c48e1eddfe04299e78c0456164abe86] frv-move-dma-macros-to-scatterlisth-for-consistency
git-bisect bad 4e22f4852c48e1eddfe04299e78c0456164abe86
# good: [d5135f31313af2be37d8ccb71e2a42f8e221d8c4] ide-mm-ide-disk-extend-timeout-for-pio-out-commands
git-bisect good d5135f31313af2be37d8ccb71e2a42f8e221d8c4
# good: [6be815e83f506f4c39a46cf59014e29a95c5e6c4] iommu-sg-merging-call-blk_queue_segment_boundary-in-__scsi_alloc_queue
git-bisect good 6be815e83f506f4c39a46cf59014e29a95c5e6c4
# good: [6be815e83f506f4c39a46cf59014e29a95c5e6c4] iommu-sg-merging-call-blk_queue_segment_boundary-in-__scsi_alloc_queue
git-bisect good 6be815e83f506f4c39a46cf59014e29a95c5e6c4
# bad: [c792db6d06114a85e33a27c89e9e979f11b951c4] slub-fix-coding-style-violations
git-bisect bad c792db6d06114a85e33a27c89e9e979f11b951c4
# bad: [c792db6d06114a85e33a27c89e9e979f11b951c4] slub-fix-coding-style-violations
git-bisect bad c792db6d06114a85e33a27c89e9e979f11b951c4
# bad: [76f3939b76ff557f73720b57a16716196f04e407] x86_64-make-sparsemem-vmemmap-the-default-memory-model-v2
git-bisect bad 76f3939b76ff557f73720b57a16716196f04e407
# good: [b8ba611566d8799a979b190d4bb14305ca64ee0e] sis-fb-driver-_ioctl32_conversion-functions-do-not-exist-in-recent-kernels
git-bisect good b8ba611566d8799a979b190d4bb14305ca64ee0e
# good: [e34995928859308d2abef1709332e2b12d36db2f] git-ipwireless_cs
git-bisect good e34995928859308d2abef1709332e2b12d36db2f
# bad: [f520abbbe11bc8253714bcd34aaaf19bdf82189e] git-x86-identify_cpu-fix
git-bisect bad f520abbbe11bc8253714bcd34aaaf19bdf82189e


I honestly tried fresh #mm from x86 tree -- the one which ends at commit
70be766db1105c0fc9aed8e954d0c343c1eda067 "x86: Add the RDC machine
specific reboot fixup". FWIW, commit "x86: validate against ACPI motherboard
resources" is innocent. After "x86: make stack size configurable" damn thing
wouldn't build and applying fixets from -mm doesn't help at 3AM.

Again, it's late here, I'll recheck today.

Re: 2.6.24-rc3-mm1 - Kernel Panic on IO-APIC

[Rik van Riel]

On Tue, 20 Nov 2007 22:18:39 -0800
Andrew Morton <akpm@linux-foundation.org> wrote:

> > ..MP-BIOS bug: 8254 timer not connected to IO-APIC
> > Kernel panic - not syncing: IO-APIC + timer doesn't work! Try using the 'noapic' kernel parameter
> 
> ACPI or x86 breakage, I guess.
> 
> Did 'noapic' work?

I got the same bug as above, 'noapic' gets past that point and right to the
next oops.  I'm posting it here because this one is different from the others
in the thread, yet looks vaguely related:

Unable to handle kernel NULL pointer dereference at 0000000000000021 RIP:
 [<ffffffff8108382a>] refresh_zone_stat_thresholds+0x6d/0x90
PGD 0
Oops: 0002 [1] SMP
last sysfs file:
CPU 0
Modules linked in:
Pid: 1, comm: swapper Not tainted 2.6.24-rc3-mm1 #2
RIP: 0010:[<ffffffff8108382a>]  [<ffffffff8108382a>] refresh_zone_stat_thresholds+0x6d/0x90
RSP: 0000:ffff81007fb59ec0  EFLAGS: 00010293
RAX: 0000000000000000 RBX: 0000000000000004 RCX: 0000000000000001
RDX: 0000000000000001 RSI: ffffffff8146fb38 RDI: 0000000000000001
RBP: ffff81000000c000 R08: 0000000000000000 R09: 0000000000000000
R10: ffff81007fb59e60 R11: 0000000000000028 R12: ffffffff814d4558
R13: 0000000000000000 R14: ffffffff814b62c0 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffffffff813d9000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000021 CR3: 0000000000201000 CR4: 00000000000006a0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper (pid: 1, threadinfo FFFF81007FB58000, task FFFF81007FB56000)
Stack:  0000000000000000 0000000000000000 0000000000000000 ffffffff814a3839
 0000000000000000 ffffffff8148e626 ffff81007fb56000 ffffffff8126d36a
 0000000000000000 ffffffffffffffff ffffffff8105786b 0000000000000000
Call Trace:
 [<ffffffff814a3839>] setup_vmstat+0x6/0x40
 [<ffffffff8148e626>] kernel_init+0x169/0x2d8
 [<ffffffff8126d36a>] trace_hardirqs_on_thunk+0x35/0x3a
 [<ffffffff8105786b>] trace_hardirqs_on+0x115/0x138
 [<ffffffff8100ce48>] child_rip+0xa/0x12
 [<ffffffff8100c55f>] restore_args+0x0/0x30
 [<ffffffff8148e4bd>] kernel_init+0x0/0x2d8
 [<ffffffff8100ce3e>] child_rip+0x0/0x12

INFO: lockdep is turned off.

-- 
All Rights Reversed

Re: 2.6.24-rc3-mm1 - Kernel Panic on IO-APIC

[Andrew Morton]

On Mon, 26 Nov 2007 14:39:43 -0500
Rik van Riel <riel@redhat.com> wrote:

> On Tue, 20 Nov 2007 22:18:39 -0800
> Andrew Morton <akpm@linux-foundation.org> wrote:
> 
> > > ..MP-BIOS bug: 8254 timer not connected to IO-APIC
> > > Kernel panic - not syncing: IO-APIC + timer doesn't work! Try using the 'noapic' kernel parameter
> > 
> > ACPI or x86 breakage, I guess.
> > 
> > Did 'noapic' work?
> 
> I got the same bug as above, 'noapic' gets past that point 

We still don't know what caused this, afaik.

> and right to the
> next oops.  I'm posting it here because this one is different from the others
> in the thread, yet looks vaguely related:
> 
> Unable to handle kernel NULL pointer dereference at 0000000000000021 RIP:
>  [<ffffffff8108382a>] refresh_zone_stat_thresholds+0x6d/0x90
> PGD 0
> Oops: 0002 [1] SMP
> last sysfs file:
> CPU 0
> Modules linked in:
> Pid: 1, comm: swapper Not tainted 2.6.24-rc3-mm1 #2
> RIP: 0010:[<ffffffff8108382a>]  [<ffffffff8108382a>] refresh_zone_stat_thresholds+0x6d/0x90
> RSP: 0000:ffff81007fb59ec0  EFLAGS: 00010293
> RAX: 0000000000000000 RBX: 0000000000000004 RCX: 0000000000000001
> RDX: 0000000000000001 RSI: ffffffff8146fb38 RDI: 0000000000000001
> RBP: ffff81000000c000 R08: 0000000000000000 R09: 0000000000000000
> R10: ffff81007fb59e60 R11: 0000000000000028 R12: ffffffff814d4558
> R13: 0000000000000000 R14: ffffffff814b62c0 R15: 0000000000000000
> FS:  0000000000000000(0000) GS:ffffffff813d9000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
> CR2: 0000000000000021 CR3: 0000000000201000 CR4: 00000000000006a0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process swapper (pid: 1, threadinfo FFFF81007FB58000, task FFFF81007FB56000)
> Stack:  0000000000000000 0000000000000000 0000000000000000 ffffffff814a3839
>  0000000000000000 ffffffff8148e626 ffff81007fb56000 ffffffff8126d36a
>  0000000000000000 ffffffffffffffff ffffffff8105786b 0000000000000000
> Call Trace:
>  [<ffffffff814a3839>] setup_vmstat+0x6/0x40
>  [<ffffffff8148e626>] kernel_init+0x169/0x2d8
>  [<ffffffff8126d36a>] trace_hardirqs_on_thunk+0x35/0x3a
>  [<ffffffff8105786b>] trace_hardirqs_on+0x115/0x138
>  [<ffffffff8100ce48>] child_rip+0xa/0x12
>  [<ffffffff8100c55f>] restore_args+0x0/0x30
>  [<ffffffff8148e4bd>] kernel_init+0x0/0x2d8
>  [<ffffffff8100ce3e>] child_rip+0x0/0x12
> 
> INFO: lockdep is turned off.

hm.  This smells like a startup ordering problem, but everything which
refresh_zone_stat_thresholds() should be set up by the time we run
initcalls.  Maybe the zone lists are bad?

Re: 2.6.24-rc3-mm1 - Kernel Panic on IO-APIC

[Ingo Molnar]

* Andrew Morton <akpm@linux-foundation.org> wrote:

> On Mon, 26 Nov 2007 14:39:43 -0500
> Rik van Riel <riel@redhat.com> wrote:
> 
> > On Tue, 20 Nov 2007 22:18:39 -0800
> > Andrew Morton <akpm@linux-foundation.org> wrote:
> > 
> > > > ..MP-BIOS bug: 8254 timer not connected to IO-APIC
> > > > Kernel panic - not syncing: IO-APIC + timer doesn't work! Try using the 'noapic' kernel parameter
> > > 
> > > ACPI or x86 breakage, I guess.
> > > 
> > > Did 'noapic' work?
> > 
> > I got the same bug as above, 'noapic' gets past that point 
> 
> We still don't know what caused this, afaik.

yes. Is it a regression? If yes, could someone try to bisect it so that 
we can fix it? If it's caused by x86.git then the 'mm' branch of the x86 
git tree can be used for bisection:

   git://git.kernel.org/pub/scm/linux/kernel/git/x86/linux-2.6-x86.git

it's supposed to build and boot fine at every bisection point. The 
bisection run can be cut significantly by narrowing the bisection to the 
arch/x86 changes only:

  git-bisect start arch/x86 include/asm-x86/

(and if it finds a nonsensical commit, i.e. the breakage is not caused 
by the x86 commits, save the "git-bisect log" output into a file, 
restart the git bisection and use "git-bisect replay" to insert all the 
test points into a fuller bisection run - this saves quite some time.)

	Ingo

Re: 2.6.24-rc3-mm1 - Kernel Panic on IO-APIC

[Rik van Riel]

On Mon, 26 Nov 2007 12:33:19 -0800
Andrew Morton <akpm@linux-foundation.org> wrote:

> > Unable to handle kernel NULL pointer dereference at 0000000000000021 RIP:
> >  [<ffffffff8108382a>] refresh_zone_stat_thresholds+0x6d/0x90
> > PGD 0
> > Oops: 0002 [1] SMP
> > last sysfs file:
> > CPU 0
> > Modules linked in:
> > Pid: 1, comm: swapper Not tainted 2.6.24-rc3-mm1 #2
> > RIP: 0010:[<ffffffff8108382a>]  [<ffffffff8108382a>] refresh_zone_stat_thresholds+0x6d/0x90
> > RSP: 0000:ffff81007fb59ec0  EFLAGS: 00010293
> > RAX: 0000000000000000 RBX: 0000000000000004 RCX: 0000000000000001
> > RDX: 0000000000000001 RSI: ffffffff8146fb38 RDI: 0000000000000001
> > RBP: ffff81000000c000 R08: 0000000000000000 R09: 0000000000000000
> > R10: ffff81007fb59e60 R11: 0000000000000028 R12: ffffffff814d4558
> > R13: 0000000000000000 R14: ffffffff814b62c0 R15: 0000000000000000
> > FS:  0000000000000000(0000) GS:ffffffff813d9000(0000) knlGS:0000000000000000
> > CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
> > CR2: 0000000000000021 CR3: 0000000000201000 CR4: 00000000000006a0
> > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > Process swapper (pid: 1, threadinfo FFFF81007FB58000, task FFFF81007FB56000)
> > Stack:  0000000000000000 0000000000000000 0000000000000000 ffffffff814a3839
> >  0000000000000000 ffffffff8148e626 ffff81007fb56000 ffffffff8126d36a
> >  0000000000000000 ffffffffffffffff ffffffff8105786b 0000000000000000
> > Call Trace:
> >  [<ffffffff814a3839>] setup_vmstat+0x6/0x40
> >  [<ffffffff8148e626>] kernel_init+0x169/0x2d8
> >  [<ffffffff8126d36a>] trace_hardirqs_on_thunk+0x35/0x3a
> >  [<ffffffff8105786b>] trace_hardirqs_on+0x115/0x138
> >  [<ffffffff8100ce48>] child_rip+0xa/0x12
> >  [<ffffffff8100c55f>] restore_args+0x0/0x30
> >  [<ffffffff8148e4bd>] kernel_init+0x0/0x2d8
> >  [<ffffffff8100ce3e>] child_rip+0x0/0x12
> > 
> > INFO: lockdep is turned off.
> 
> hm.  This smells like a startup ordering problem, but everything which
> refresh_zone_stat_thresholds() should be set up by the time we run
> initcalls.  Maybe the zone lists are bad?

Or the CPU array. Look at the oops Kamalesh got a few mails upthread...

I guess I'll have to start a bisect - can't port the VM code to a kernel
that doesn't boot...

-- 
All Rights Reversed

Re: 2.6.24-rc3-mm1 - Kernel Panic on IO-APIC

[Christoph Lameter]

On Mon, 26 Nov 2007, Andrew Morton wrote:

> hm.  This smells like a startup ordering problem, but everything which
> refresh_zone_stat_thresholds() should be set up by the time we run
> initcalls.  Maybe the zone lists are bad?

refresh_zone_stat_thresholds goes through each zone and updates
the stat threshold for every per cpu structure in each zone.

So this could be a processor marked online where the pcp structures have 
not been allocated or a zone NULL pointer.

Re: 2.6.24-rc3-mm1 - Kernel Panic on IO-APIC

[Jiri Slaby]

On 11/26/2007 09:45 PM, Ingo Molnar wrote:
> * Andrew Morton <akpm@linux-foundation.org> wrote:
> 
>> On Mon, 26 Nov 2007 14:39:43 -0500
>> Rik van Riel <riel@redhat.com> wrote:
>>
>>> On Tue, 20 Nov 2007 22:18:39 -0800
>>> Andrew Morton <akpm@linux-foundation.org> wrote:
>>>
>>>>> ..MP-BIOS bug: 8254 timer not connected to IO-APIC
>>>>> Kernel panic - not syncing: IO-APIC + timer doesn't work! Try using the 'noapic' kernel parameter
>>>> ACPI or x86 breakage, I guess.
>>>>
>>>> Did 'noapic' work?
>>> I got the same bug as above, 'noapic' gets past that point 
>> We still don't know what caused this, afaik.
> 
> yes. Is it a regression? If yes, could someone try to bisect it so that 
> we can fix it? If it's caused by x86.git then the 'mm' branch of the x86 
> git tree can be used for bisection:
> 
>    git://git.kernel.org/pub/scm/linux/kernel/git/x86/linux-2.6-x86.git

I did, but it's hard, if you don't know the BAD point. HEAD boots fine and 'x86:
randomize brk' too (the top of git-x86.patch). Andrew, how do you pull it, git
#mm doesn't fit to the ids from the patch.

Maybe if you can emit a broken-out with the fresh pull to test?

regards,
-- 
Jiri Slaby (jirislaby@gmail.com)
Faculty of Informatics, Masaryk University

Re: 2.6.24-rc3-mm1 - Kernel Panic on IO-APIC

[Andrew Morton]

On Mon, 26 Nov 2007 23:08:33 +0100
Jiri Slaby <jirislaby@gmail.com> wrote:

> On 11/26/2007 09:45 PM, Ingo Molnar wrote:
> > * Andrew Morton <akpm@linux-foundation.org> wrote:
> > 
> >> On Mon, 26 Nov 2007 14:39:43 -0500
> >> Rik van Riel <riel@redhat.com> wrote:
> >>
> >>> On Tue, 20 Nov 2007 22:18:39 -0800
> >>> Andrew Morton <akpm@linux-foundation.org> wrote:
> >>>
> >>>>> ..MP-BIOS bug: 8254 timer not connected to IO-APIC
> >>>>> Kernel panic - not syncing: IO-APIC + timer doesn't work! Try using the 'noapic' kernel parameter
> >>>> ACPI or x86 breakage, I guess.
> >>>>
> >>>> Did 'noapic' work?
> >>> I got the same bug as above, 'noapic' gets past that point 
> >> We still don't know what caused this, afaik.
> > 
> > yes. Is it a regression? If yes, could someone try to bisect it so that 
> > we can fix it? If it's caused by x86.git then the 'mm' branch of the x86 
> > git tree can be used for bisection:
> > 
> >    git://git.kernel.org/pub/scm/linux/kernel/git/x86/linux-2.6-x86.git
> 
> I did, but it's hard, if you don't know the BAD point. HEAD boots fine and 'x86:
> randomize brk' too (the top of git-x86.patch).

So the bug wasn't in git-x86 in 2.6.24-rc3-mm1.

But it might be in there now, as some patches got moved over.

Or it could be git-acpi.  Or lots of other things.

> Andrew, how do you pull it, git
> #mm doesn't fit to the ids from the patch.

The -mm git tree reimports the plain git-foo.patch files back into a new
git tree, so the commit IDs won't line up.

The way to find the culprit patch in 2.6.24-rc3-mm1 is
http://www.zip.com.au/~akpm/linux/patches/stuff/bisecting-mm-trees.txt.  It
will be quite quick.

> Maybe if you can emit a broken-out with the fresh pull to test?

http://userweb.kernel.org/~akpm/mmotm/ is current.  But it probably won't
compile.  I'd suggest bisecting 2.6.24-rc3-mm1 would be easier.  

Re: 2.6.24-rc3-mm1 - Kernel Panic on IO-APIC

[Jiri Slaby]

On 11/26/2007 11:17 PM, Andrew Morton wrote:
> http://userweb.kernel.org/~akpm/mmotm/ is current.  But it probably won't
> compile.  I'd suggest bisecting 2.6.24-rc3-mm1 would be easier.  

Yes, I've bisected this and it pointed to git-x86.patch + 2 pushed fixes from
series, Then tried x86 git, but its HEAD was OK.

Re: 2.6.24-rc3-mm1 - Kernel Panic on IO-APIC

[Jiri Slaby]

On 11/26/2007 11:17 PM, Andrew Morton wrote:
>> Maybe if you can emit a broken-out with the fresh pull to test?
> 
> http://userweb.kernel.org/~akpm/mmotm/ is current.  But it probably won't
> compile. 

Yes it did :). And it worked. Both in qemu and on my desktop...

qemu output at:
http://www.fi.muni.cz/~xslaby/sklad/qemu-output.txt

thanks,
-- 
Jiri Slaby (jirislaby@gmail.com)
Faculty of Informatics, Masaryk University

Re: 2.6.24-rc3-mm1 - Kernel Panic on IO-APIC

[Andrew Morton]

On Tue, 27 Nov 2007 00:14:17 +0100
Jiri Slaby <jirislaby@gmail.com> wrote:

> On 11/26/2007 11:17 PM, Andrew Morton wrote:
> >> Maybe if you can emit a broken-out with the fresh pull to test?
> > 
> > http://userweb.kernel.org/~akpm/mmotm/ is current.  But it probably won't
> > compile. 
> 
> Yes it did :). And it worked. Both in qemu and on my desktop...

boggle.  Let's slap 2.6.25 on it and take the rest of the year off.

> qemu output at:
> http://www.fi.muni.cz/~xslaby/sklad/qemu-output.txt

Thanks for testing.

Re: 2.6.24-rc3-mm1 - Kernel Panic on IO-APIC

[Rik van Riel]

On Mon, 26 Nov 2007 15:28:32 -0800
Andrew Morton <akpm@linux-foundation.org> wrote:

> On Tue, 27 Nov 2007 00:14:17 +0100
> Jiri Slaby <jirislaby@gmail.com> wrote:
> 
> > On 11/26/2007 11:17 PM, Andrew Morton wrote:
> > >> Maybe if you can emit a broken-out with the fresh pull to test?
> > > 
> > > http://userweb.kernel.org/~akpm/mmotm/ is current.  But it probably won't
> > > compile. 
> > 
> > Yes it did :). And it worked. Both in qemu and on my desktop...
> 
> boggle.  Let's slap 2.6.25 on it and take the rest of the year off.

No worries, the mmotm compiling issue seems to have been fixed:

  CC [M]  drivers/scsi/libsas/sas_ata.o
drivers/scsi/libsas/sas_ata.c:39: error: field ‘rphy’ has incomplete type
drivers/scsi/libsas/sas_ata.c: In function ‘sas_discover_sata’:
drivers/scsi/libsas/sas_ata.c:773: error: implicit declaration of function ‘ata_sas_rphy_alloc’
drivers/scsi/libsas/sas_ata.c:775: error: dereferencing pointer to incomplete type
drivers/scsi/libsas/sas_ata.c:775: warning: assignment makes pointer from integer without a cast
drivers/scsi/libsas/sas_ata.c:781: error: dereferencing pointer to incomplete type
drivers/scsi/libsas/sas_ata.c:782: error: dereferencing pointer to incomplete type
drivers/scsi/libsas/sas_ata.c:784: warning: type defaults to ‘int’ in declaration of ‘__mptr’
drivers/scsi/libsas/sas_ata.c:784: warning: initialization from incompatible pointer type
drivers/scsi/libsas/sas_ata.c:791: error: implicit declaration of function ‘ata_sas_rphy_add’
drivers/scsi/libsas/sas_ata.c:807: error: implicit declaration of function ‘ata_sas_rphy_delete’
drivers/scsi/libsas/sas_ata.c:809: error: implicit declaration of function ‘ata_sas_rphy_free’
make[3]: *** [drivers/scsi/libsas/sas_ata.o] Error 1
make[2]: *** [drivers/scsi/libsas] Error 2
make[1]: *** [drivers/scsi] Error 2
make: *** [drivers] Error 2

So much for continuing the bisect with that tree, to find the
cause of the second bug :)

Guess I'll extract an x86 tree changeset first, to place into
the 2.6.23-rc3-mm1 broken out tree and work from there...

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan