Boot failure on the powerstation with 2.6.30 latest

Boot failure on the powerstation with 2.6.30 latest

[James Bottomley]

2.6.30-rc8 worked fine ... unless this is a known problem, I suppose I
can begin bisecting.

The boot log of the hang is:

Please wait, loading kernel...
   Elf64 kernel loaded...
Loading ramdisk...
ramdisk loaded at 02500000, size: 8280 Kbytes
OF stdout device is: /ht/isa@8/serial@2f8
Preparing to boot Linux version 2.6.30 (jejb@claymoor) (gcc version 4.3.3 (Debian 4.3.3-10) ) #1 SMP Mon Jun 22 09:59:35 CDT 2009
command line: root=/dev/sda3 ro console=ttyS0,19200n1 
memory layout at init:
  alloc_bottom : 0000000002d16000
  alloc_top    : 0000000030000000
  alloc_top_hi : 0000000080000000
  rmo_top      : 0000000030000000
  ram_top      : 0000000080000000
instantiating rtas at 0x000000002fff5000... done
boot cpu hw idx 0000000000000000
starting cpu hw idx 0000000000000001... done
starting cpu hw idx 0000000000000002... done
starting cpu hw idx 0000000000000003... done
copying OF device tree...
Building dt strings...
Building dt structure...
Device tree strings 0x0000000003117000 -> 0x0000000003117640
Device tree struct  0x0000000003118000 -> 0x000000000311b000
Calling quiesce...
returning from prom_init

So it looks like some type of early boot failure or handoff in head_64

James

Re: Boot failure on the powerstation with 2.6.30 latest

[Brian King]

James,

I was running into a similar hang on one of my Power boxes as well.
Reverting c868d550115b9ccc0027c67265b9520790f05601 allowed by system
to boot. It looks like that patch injected a bug where we can end up
waiting on an uninitialized mutex:

[c0000000009f3c30] c00000000052c7dc .mutex_lock+0x34/0x50
[c0000000009f3cb0] c00000000008b190 .get_online_cpus+0x3c/0x74
[c0000000009f3d40] c000000000146cd0 .kmem_cache_create+0xcc/0x548
[c0000000009f3e50] c000000000032ae0 .pgtable_cache_init+0x28/0x6c
[c0000000009f3ee0] c000000000780960 .start_kernel+0x1ec/0x520
[c0000000009f3f90] c0000000000083d8 .start_here_common+0x1c/0x44

The mutex gets initialized in cpu_hotplug_init, which doesn't get called until
after pgtable_cache_init.

-Brian

James Bottomley wrote:
> 2.6.30-rc8 worked fine ... unless this is a known problem, I suppose I
> can begin bisecting.
> 
> The boot log of the hang is:
> 
> Please wait, loading kernel...
>    Elf64 kernel loaded...
> Loading ramdisk...
> ramdisk loaded at 02500000, size: 8280 Kbytes
> OF stdout device is: /ht/isa@8/serial@2f8
> Preparing to boot Linux version 2.6.30 (jejb@claymoor) (gcc version 4.3.3 (Debian 4.3.3-10) ) #1 SMP Mon Jun 22 09:59:35 CDT 2009
> command line: root=/dev/sda3 ro console=ttyS0,19200n1 
> memory layout at init:
>   alloc_bottom : 0000000002d16000
>   alloc_top    : 0000000030000000
>   alloc_top_hi : 0000000080000000
>   rmo_top      : 0000000030000000
>   ram_top      : 0000000080000000
> instantiating rtas at 0x000000002fff5000... done
> boot cpu hw idx 0000000000000000
> starting cpu hw idx 0000000000000001... done
> starting cpu hw idx 0000000000000002... done
> starting cpu hw idx 0000000000000003... done
> copying OF device tree...
> Building dt strings...
> Building dt structure...
> Device tree strings 0x0000000003117000 -> 0x0000000003117640
> Device tree struct  0x0000000003118000 -> 0x000000000311b000
> Calling quiesce...
> returning from prom_init
> 
> So it looks like some type of early boot failure or handoff in head_64
> 
> James
> 
> 
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/linuxppc-dev


-- 
Brian King
Linux on Power Virtualization
IBM Linux Technology Center

Re: Boot failure on the powerstation with 2.6.30 latest

[James Bottomley]

On Mon, 2009-06-22 at 11:11 -0500, Brian King wrote:
> James,
> 
> I was running into a similar hang on one of my Power boxes as well.
> Reverting c868d550115b9ccc0027c67265b9520790f05601 allowed by system
> to boot. It looks like that patch injected a bug where we can end up
> waiting on an uninitialized mutex:
> 
> [c0000000009f3c30] c00000000052c7dc .mutex_lock+0x34/0x50
> [c0000000009f3cb0] c00000000008b190 .get_online_cpus+0x3c/0x74
> [c0000000009f3d40] c000000000146cd0 .kmem_cache_create+0xcc/0x548
> [c0000000009f3e50] c000000000032ae0 .pgtable_cache_init+0x28/0x6c
> [c0000000009f3ee0] c000000000780960 .start_kernel+0x1ec/0x520
> [c0000000009f3f90] c0000000000083d8 .start_here_common+0x1c/0x44
> 
> The mutex gets initialized in cpu_hotplug_init, which doesn't get called until
> after pgtable_cache_init.

Actually, no, reverting that one doesn't fix it.

A full run of git bisect turns up this commit as the culprit; I'll make
a fuss on lkml:

83b519e8b9572c319c8e0c615ee5dd7272856090 is first bad commit
commit 83b519e8b9572c319c8e0c615ee5dd7272856090
Author: Pekka Enberg <penberg@cs.helsinki.fi>
Date:   Wed Jun 10 19:40:04 2009 +0300

    slab: setup allocators earlier in the boot sequence

James

Re: Boot failure on the powerstation with 2.6.30 latest

[Benjamin Herrenschmidt]

On Mon, 2009-06-22 at 11:11 -0500, Brian King wrote:
> James,
> 
> I was running into a similar hang on one of my Power boxes as well.
> Reverting c868d550115b9ccc0027c67265b9520790f05601 allowed by system
> to boot. It looks like that patch injected a bug where we can end up
> waiting on an uninitialized mutex:
> 
> [c0000000009f3c30] c00000000052c7dc .mutex_lock+0x34/0x50
> [c0000000009f3cb0] c00000000008b190 .get_online_cpus+0x3c/0x74
> [c0000000009f3d40] c000000000146cd0 .kmem_cache_create+0xcc/0x548
> [c0000000009f3e50] c000000000032ae0 .pgtable_cache_init+0x28/0x6c
> [c0000000009f3ee0] c000000000780960 .start_kernel+0x1ec/0x520
> [c0000000009f3f90] c0000000000083d8 .start_here_common+0x1c/0x44
> 
> The mutex gets initialized in cpu_hotplug_init, which doesn't get called until
> after pgtable_cache_init.

Ah good, I didn't have a chance to track that one down yet.

So the problem here is that we must do pgtable_cache_init there because
vmalloc is initialized right after, which relies on allocating page
tables and that will need kmem caches on some archs.

So I suspect we need to sort out this mutex, either initializing it from
elsewhere, moving cpu_hotplug_init() earlier, or avoiding it when the
kernel state isn't SYSTEM_RUNNING, I haven't looked in details yet.

Cheers,
Ben.

> -Brian
> 
> James Bottomley wrote:
> > 2.6.30-rc8 worked fine ... unless this is a known problem, I suppose I
> > can begin bisecting.
> > 
> > The boot log of the hang is:
> > 
> > Please wait, loading kernel...
> >    Elf64 kernel loaded...
> > Loading ramdisk...
> > ramdisk loaded at 02500000, size: 8280 Kbytes
> > OF stdout device is: /ht/isa@8/serial@2f8
> > Preparing to boot Linux version 2.6.30 (jejb@claymoor) (gcc version 4.3.3 (Debian 4.3.3-10) ) #1 SMP Mon Jun 22 09:59:35 CDT 2009
> > command line: root=/dev/sda3 ro console=ttyS0,19200n1 
> > memory layout at init:
> >   alloc_bottom : 0000000002d16000
> >   alloc_top    : 0000000030000000
> >   alloc_top_hi : 0000000080000000
> >   rmo_top      : 0000000030000000
> >   ram_top      : 0000000080000000
> > instantiating rtas at 0x000000002fff5000... done
> > boot cpu hw idx 0000000000000000
> > starting cpu hw idx 0000000000000001... done
> > starting cpu hw idx 0000000000000002... done
> > starting cpu hw idx 0000000000000003... done
> > copying OF device tree...
> > Building dt strings...
> > Building dt structure...
> > Device tree strings 0x0000000003117000 -> 0x0000000003117640
> > Device tree struct  0x0000000003118000 -> 0x000000000311b000
> > Calling quiesce...
> > returning from prom_init
> > 
> > So it looks like some type of early boot failure or handoff in head_64
> > 
> > James
> > 
> > 
> > _______________________________________________
> > Linuxppc-dev mailing list
> > Linuxppc-dev@lists.ozlabs.org
> > https://lists.ozlabs.org/listinfo/linuxppc-dev
> 
> 

Re: Boot failure on the powerstation with 2.6.30 latest

[Benjamin Herrenschmidt]

> Actually, no, reverting that one doesn't fix it.
> 
> A full run of git bisect turns up this commit as the culprit; I'll make
> a fuss on lkml:

I haven't had the full log of that boot failure, but reverting the patch
Brian suggested won't work well indeed, as I said, from the moment slab
is initialized, page table allocations will use kmem caches which are
initialized by pgtable_cache_init().

So the problem does indeed seem to be another fallover of moving the
allocator initialization earlier.

I'm working from home today but I'll see if I can get somebody in the
office to wire up the powerstation (got disconnected for some reason
last week) for me so I can have a look.

The mutex issue Brian noticed will definitely break _any_ kmem_cache
operation anyway, so that's one bug that need fixing at least (well,
provided Brian analysis is right, I didn't have a chance to look myself
yet :-)

Cheers,
Ben.

> 83b519e8b9572c319c8e0c615ee5dd7272856090 is first bad commit
> commit 83b519e8b9572c319c8e0c615ee5dd7272856090
> Author: Pekka Enberg <penberg@cs.helsinki.fi>
> Date:   Wed Jun 10 19:40:04 2009 +0300
> 
>     slab: setup allocators earlier in the boot sequence
> 
> James
>