Initrd and 2.6.33 curious behaviour

Initrd and 2.6.33 curious behaviour

[Robert Jarzmik]

I pulled the 2.6.33 kernel and tried my mainboard with it (mioa701). My kernel
boots fine, but initrd load always fails.

After digging a bit in arch/arm/mm/init.c, I find that :
 - phys_initrd_start = 0xa0508000
 - phys_initrd_size = 3 759 480
 - initrd_start = 0 and initrd_end = 0

I'm a bit puzzled why initrd_start is null. I would expect it to be the virtual
address where the initrd is to be found.

Am I the only one with initrd problems on 2.6.34 kernels ?

For the record, the bootloader is the same I had before (haret), and the ATAGS
list passed to the kernel is correct. The initrd is the same I've been using
since 2.6.29 kernel. I have not tested since 2.6.32 I think.

Cheers.

--
Robert

Initrd and 2.6.33 curious behaviour

[Russell King - ARM Linux]

On Fri, May 21, 2010 at 10:27:05PM +0200, Robert Jarzmik wrote:
> I pulled the 2.6.33 kernel and tried my mainboard with it (mioa701). My
> kernel boots fine, but initrd load always fails.

Please post all the kernel messages.

> After digging a bit in arch/arm/mm/init.c, I find that :
>  - phys_initrd_start = 0xa0508000
>  - phys_initrd_size = 3 759 480
>  - initrd_start = 0 and initrd_end = 0

It's unclear where you're checking this.  The virtual pointers are only
setup if we're satisfied that the phys_* are within RAM, and we can
exclusively reserve the region.

If we can't reserve the region, that means something's overwritten it
during kernel boot and it's become unusable.

Initrd and 2.6.33 curious behaviour

[Robert Jarzmik]

Russell King - ARM Linux <linux@arm.linux.org.uk> writes:

> On Fri, May 21, 2010 at 10:27:05PM +0200, Robert Jarzmik wrote:
>> I pulled the 2.6.33 kernel and tried my mainboard with it (mioa701). My
>> kernel boots fine, but initrd load always fails.
>
> Please post all the kernel messages.

If only I could ...
I have no serial console, only the screen, and until initrd is loaded, no
network connection. The mioa701 is a smartphone, with no keyboard, and only
bluetooth and USB as a communication mean.

The only thing I can see is the console output on the framebuffer, which is damn
small.

I was thinking that if at least 1 person could load the initrd, then I would
have no other choice but investigate. But if all people have the same problem,
there would be someone with a development board who will have a full console
output and find the issue easier.

>> After digging a bit in arch/arm/mm/init.c, I find that :
>>  - phys_initrd_start = 0xa0508000
>>  - phys_initrd_size = 3 759 480
>>  - initrd_start = 0 and initrd_end = 0
>
> It's unclear where you're checking this.  The virtual pointers are only
> setup if we're satisfied that the phys_* are within RAM, and we can
> exclusively reserve the region.
I added printk in do_mounts_rd.c, in the first lines of rd_load_image().
I had to remove the "static" statement from phys_initrd_* declaration in init.c,
and added extern declarations to reach the phys_initrd_* variables.

> If we can't reserve the region, that means something's overwritten it
> during kernel boot and it's become unusable.
When you say "overwritten", what is the criteria ? (ie. is it at least one byte
is non-zero, checksum failed, ...).

Or alternatively, is there a clever way of storing some data relative to
reserve_bootmem_node() or __reserve() to be printed at the tail of kernel log
(ie. in do_mounts_rd()) which would help me investigate ?

Cheers.

--
Robert

Initrd and 2.6.33 curious behaviour

[Robert Jarzmik]

Russell King - ARM Linux <linux@arm.linux.org.uk> writes:

> On Fri, May 21, 2010 at 10:27:05PM +0200, Robert Jarzmik wrote:
>> I pulled the 2.6.33 kernel and tried my mainboard with it (mioa701). My
>> kernel boots fine, but initrd load always fails.
>
> Please post all the kernel messages.

If only I could ...
I have no serial console, only the screen, and until initrd is loaded, no
network connection. The mioa701 is a smartphone, with no keyboard, and only
bluetooth and USB as a communication mean.

The only thing I can see is the console output on the framebuffer, which is damn
small.

I was thinking that if at least 1 person could load the initrd, then I would
have no other choice but investigate. But if all people have the same problem,
there would be someone with a development board who will have a full console
output and find the issue easier.

>> After digging a bit in arch/arm/mm/init.c, I find that :
>>  - phys_initrd_start = 0xa0508000
>>  - phys_initrd_size = 3 759 480
>>  - initrd_start = 0 and initrd_end = 0
>
> It's unclear where you're checking this.  The virtual pointers are only
> setup if we're satisfied that the phys_* are within RAM, and we can
> exclusively reserve the region.
I added printk in do_mounts_rd.c, in the first lines of rd_load_image().
I had to remove the "static" statement from phys_initrd_* declaration in init.c,
and added extern declarations to reach the phys_initrd_* variables.

> If we can't reserve the region, that means something's overwritten it
> during kernel boot and it's become unusable.
When you say "overwritten", what is the criteria ? (ie. is it at least one byte
is non-zero, checksum failed, ...).

Or alternatively, is there a clever way of storing some data relative to
reserve_bootmem_node() or __reserve() to be printed at the tail of kernel log
(ie. in do_mounts_rd()) which would help me investigate ?

Cheers.

--
Robert

Initrd and 2.6.33 curious behaviour

[Russell King - ARM Linux]

On Sat, May 22, 2010 at 11:37:43AM +0200, Robert Jarzmik wrote:
> Or alternatively, is there a clever way of storing some data relative to
> reserve_bootmem_node() or __reserve() to be printed at the tail of kernel log
> (ie. in do_mounts_rd()) which would help me investigate ?

You could boot with 'quiet' so that the kernel shuts up most of the
noisy kernel messages.

Initrd and 2.6.33 curious behaviour

[Robert Jarzmik]

Russell King - ARM Linux <linux@arm.linux.org.uk> writes:
> You could boot with 'quiet' so that the kernel shuts up most of the
> noisy kernel messages.

With "quiet" option, the kernel console outputs only 1 line :
"Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(1,0)"

--
Robert

Initrd and 2.6.33 curious behaviour

[Russell King - ARM Linux]

On Sun, May 23, 2010 at 02:23:33AM +0200, Robert Jarzmik wrote:
> Russell King - ARM Linux <linux@arm.linux.org.uk> writes:
> > You could boot with 'quiet' so that the kernel shuts up most of the
> > noisy kernel messages.
> 
> With "quiet" option, the kernel console outputs only 1 line :
> "Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(1,0)"

Right, so none of the errors in the ARM MM initialization are being
triggered, so the initrd image isn't being corrupted.

The reason that initrd_start is zero at rd_load_ramdisk is that in
populate_rootfs(), the initrd is written into the ramfs root as a
/initrd.image file.  Then the original initrd image is freed, resulting
in both initrd_start and initrd_end being zeroed.

Do you have CONFIG_BLK_DEV_RAM enabled?  Try some debugging (using
pr_err() and booting with 'quiet' so you can see the output) in
populate_rootfs().

Initrd and 2.6.33 curious behaviour

[Robert Jarzmik]

Russell King - ARM Linux <linux@arm.linux.org.uk> writes:

> On Sun, May 23, 2010 at 02:23:33AM +0200, Robert Jarzmik wrote:
> The reason that initrd_start is zero at rd_load_ramdisk is that in
> populate_rootfs(), the initrd is written into the ramfs root as a
> /initrd.image file.  Then the original initrd image is freed, resulting
> in both initrd_start and initrd_end being zeroed.
I understand better now.

> Do you have CONFIG_BLK_DEV_RAM enabled?  Try some debugging (using
> pr_err() and booting with 'quiet' so you can see the output) in
> populate_rootfs().
CONFIG_BLK_DEV_RAM is enabled. I'll add some debugging now and will try again.

Thanks for the help.

--
Robert

Initrd and 2.6.33 curious behaviour

[Robert Jarzmik]

Robert Jarzmik <robert.jarzmik@free.fr> writes:
> CONFIG_BLK_DEV_RAM is enabled. I'll add some debugging now and will try again.
>
OK, found out the culprit.
My bootloader(haret) is to be blamed : the first 6 bytes of initrd as the kernel
sees them are totally different from the actual 6 first bytes of initrd file.

The long story is that when haret stops wince MMU, it "reorders" the pages to
create a physically contiguous kernel and initrd, from the scattered pages
allocated in wince. The relocation algorithm is faulty, as it doesn't cope with
the cases where there is a double dependency (ie. a kernel page should be moved
to the page occupied by a initrd page, and the initrd page should be moved to
where resides the kernel page).

It's funny how people want to solve the hanoi towers problem without a spare
slot ...

Cheers.

--
Robert

Initrd and 2.6.33 curious behaviour

[Robert Jarzmik]

I have made some progress.

I have traced that :
 - after the bootloader has disabled the MMU, disabled the caches, quiesced
 the DMAs and masked the interrupts, and just before the final jump to the
 kernel first instruction :
   => the first u32 at physical address 0xa0508000 is 1f:8b:08:08
   => this is the correct begining of my initrd.gz

In function setup_arch(), just after paging_init(mdesc), I read the value again
with a "rjk = *((unsigned int *)(phys_to_virt(0xa0508000)));", and it has
changed to 10:00:00:a0.

Now, I would need some help to trace this memory location before
paging_init(). My current understanding is that before paging_init(), the MMU is
set by __create_page_tables(). The mapping doesn't cover the initrd area.
If I was to be able to watch this area, I would need to :
 - amend __create_page_tables(), and add a mapping for the 0xa0508000 area (1MB
 is enough)
 - read the value of the u32 at physical address 0xa0508000 from this mapping
 - printk the stored value at the end of setup_arch()

The questions I have are :
 - is this the right approach ?
 - which virtual address space can I use for my mapping

--
Robert