[U-Boot] 83xx fails to boot with moderately sized kernels

[Ira W. Snyder <iws@ovro.caltech.edu>]

Hello all,

I am using vanilla U-Boot on an MPC8349EMDS board with 256MB of RAM. I have
been tracking down a kernel hang very early during boot, and have traced it
back to a U-Boot bug. The bug has to do with the way CONFIG_SYS_BOOTMAPSZ
is set.

Please read through this thread for more lots more information:
http://lists.ozlabs.org/pipermail/linuxppc-dev/2010-September/085524.html

On most (all?) 83xx boards, CONFIG_SYS_BOOTMAPSZ is set to 8MB. The comment
says:

/*
 * For booting Linux, the board info and command line data
 * have to be in the first 8 MB of memory, since this is
 * the maximum mapped by the Linux kernel during initialization.
 */
#define CONFIG_SYS_BOOTMAPSZ	(8 << 20)	/* Initial Memory map for Linux*/

I am booting with a FIT image containing a kernel, FDT, and ramdisk.
U-Boot's bootm command makes sure that the FDT is relocated such that it is
located at an address lower than CONFIG_SYS_BOOTMAPSZ. The ramdisk is
placed at the end of memory. The relevant U-Boot messages:

   Booting using the fdt blob at 0x226a278
   Uncompressing Kernel Image ... OK
   Loading Ramdisk to 0fe9f000, end 0ff75699 ... OK
   Loading Device Tree to 007f8000, end 007ff78f ... OK

You'll notice the ramdisk was loaded@the end of RAM (near 256MB), while
the FDT was loaded near the end of CONFIG_SYS_BOOTMAPSZ (near 8MB).

Newer Linux kernels are getting pretty large. In fact, even a fairly
minimal kernel is large enough such that the uncompressed kernel code + BSS
is >8MB. When this happens, Linux overwrites the FDT and the boot hangs
silently.

U-Boot cannot detect this, since it does not load ELF images. It does not
know the BSS size.

Below is the size of my two kernels, toggling only the CONFIG_PROVE_LOCKING
feature on or off. As stated above, the rest of the kernel is relatively
minimal.

With CONFIG_PROVE_LOCKING:
c0369000 A __bss_start
c08a0c48 A __bss_stop

0xc08a0c48 - 0xc0369000 == 5471304 bytes BSS

Kernel size (uncompressed):
3573068 vmlinux.bin

Total (code + bss): 3573068 + 5471304 == 9044372 (8.625MB)

Without CONFIG_PROVE_LOCKING:
c0363000 A __bss_start
c0786c08 A __bss_stop

0xc0786c08 - 0xc0363000 == 4340744 bytes BSS

Kernel size (uncompressed):
3548492 vmlinux.bin

Total (code + bss): 3548492 + 4340744 == 7889236 (7.523MB)

So, just adding CONFIG_PROVE_LOCKING=y made a 1MB difference, most of it in
the bss section.

Now, as the U-Boot output shows (with CONFIG_SYS_BOOTMAPSZ == 8MB):

   Booting using the fdt blob at 0x226a278
   Uncompressing Kernel Image ... OK
   Loading Ramdisk to 0fe9f000, end 0ff75699 ... OK
   Loading Device Tree to 007f8000, end 007ff78f ... OK

Now, since the Linux kernel is loaded to address 0x0, this means that after
it is running and has initialized it's BSS, the end of the kernel will be
at address: 0x786154. The working kernel is just barely squeaking by with
about 450KB of space between the kernel and FDT.

Note that on PowerPC, Linux initializes it's BSS very, very early. The BSS
is initialized before the FDT is parsed.

Nothing that Timur or I can find indicate that CONFIG_SYS_BOOTMAPSZ is
actually limited to 8MB, as the comment states. Timur and Kumar chatted,
and realized that they had seen this bug before, on 85xx. They raised the
value of CONFIG_SYS_BOOTMAPSZ to 16MB without issue.

Reading through the Linux code, I cannot find any reason why this should be
limited to 8MB, or even 16MB. Should it be limited to CONFIG_MAX_MEM_MAPPED?

I'm worried that others either have hit this bug, or will hit it soon. By
building lots of drivers into the kernel, I'm sure that even the 16MB
boundary could be overrun.

Does anyone know the true maximum value for CONFIG_SYS_BOOTMAPSZ on Linux
(if one even exists)?

Thanks,
Ira