[U-Boot] Where to put a large bootloader-supplied device tree on ARM ?

[U-Boot] Where to put a large bootloader-supplied device tree on ARM ?

[Rob Herring]

[adding u-boot list]

On 07/12/2012 01:52 AM, Mitch Bradley wrote:
> On 7/8/2012 6:30 PM, Nicolas Pitre wrote:
>> On Fri, 6 Jul 2012, Mitch Bradley wrote:
>>
>>> On 7/6/2012 3:23 PM, David VomLehn (dvomlehn) wrote:
>>>> The kernel *must* go where it is linked, but the FDT contains only relative
>>>> references and is thus free to go anywhere. The same is true of ramdisks,
>>>> which
>>>> are usually placed after the kernel.
>>
>> The kernel must go where it is linked *only* if you are using the 
>> 'Image' output.  When using 'zImage' you can put the kernel anywhere in 
>> memory, or in the first 128MB of RAM if CONFIG_AUTO_ZRELADDR is used.
>>
>>> Right, but the kernel image is compressed, so after decompression it expands
>>> into the area just after it.  Also, the .bss segment is in that vicinity.
>>
>> To be exact, the compressed kernel moves itself out of the region where 
>> the decompressed kernel will end up before doing the decompression, but 
>> only if necessary.  So it is a good idea to load zImage away from the 
>> decompressed kernel area to avoid this extra move and save some fraction 
>> of a second on boot time.
>>
>>> There's some code in arch/arm/boot/compressed/head.S to relocate
>>> device tree blobs, but it requires CONFIG_ARM_APPENDED_DTB which
>>> is not recommended - arch/arm/Kconfig recommends using the
>>> documented boot protocol istead .
>>
>> This is in case a DTB is appended to zImage.  When the DTB is detected, 
>> the moving of zImage out of the decompressed area must take care of 
>> moving the DTB as well.
>>
>>> Documentation/arm/Booting says
>>> to put the dtb "in a region of memory where the kernel decompressor
>>> will not overwrite it", further recommending the first 16KiB.
>>>
>>> As noted, the first 16KiB loses if the dtb is too large.  And
>>> "where the kernel decompressor will not overwrite it" says what
>>> won't work, not what will.  It appears that the decompressor works
>>> out its addresses dynamically, so there's no hard prescription even
>>> for what to avoid.
>>
>> A good rule of thumb is to take the size of the decompressed kernel and 
>> multiply this by 3.  Rounding up is also fine.  So for example if your 
>> arch/arm/boot/Image is 5MB, then putting anciliary data such as a 
>> ramdisk or a large DTB from 16MB into RAM or above should be fine.
>>
>>> For now, I'm putting the initrd at the end of memory and the dtb
>>> below that.  That seems to work, but I'm unsure whether or not
>>> I'm just "getting lucky".
>>
>> That's also perfectly fine.
> 
> 
> Alas, that worked for machines with 512 MiB of main memory, but failed
> on 1 GiB machines.  My guess is that, when the initrd and dtb are near
> the top of a 1 GiB memory, the virtual address gets too near the top of
> the kernel's 1 GiB of virtual space (which starts at 0xc0000000),
> perhaps colliding with the VMALLOC space.
> 
> Putting them just below the 128 MiB boundary seems to work.

Interesting. I think this is also a problem on u-boot just waiting to
happen. u-boot locates itself at the end of RAM and likes to copy the
fdt and initrd to just below that. Any machine with 1G+ is going to hit
this. I avoided it because I limited u-boot to 512MB on highbank.

Rob

> 
>>
>>
>> Nicolas
>>
> 
> _______________________________________________
> devicetree-discuss mailing list
> devicetree-discuss at lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/devicetree-discuss
> 

[U-Boot] Where to put a large bootloader-supplied device tree on ARM ?

[Albert ARIBAUD]

Hi Rob,

On Thu, 12 Jul 2012 15:34:17 -0500, Rob Herring <robherring2@gmail.com> wrote:
> [adding u-boot list]
> 
> On 07/12/2012 01:52 AM, Mitch Bradley wrote:
> > On 7/8/2012 6:30 PM, Nicolas Pitre wrote:
> >> On Fri, 6 Jul 2012, Mitch Bradley wrote:
> >>
> >>> On 7/6/2012 3:23 PM, David VomLehn (dvomlehn) wrote:
> >>>> The kernel *must* go where it is linked, but the FDT contains only relative
> >>>> references and is thus free to go anywhere. The same is true of ramdisks,
> >>>> which
> >>>> are usually placed after the kernel.
> >>
> >> The kernel must go where it is linked *only* if you are using the 
> >> 'Image' output.  When using 'zImage' you can put the kernel anywhere in 
> >> memory, or in the first 128MB of RAM if CONFIG_AUTO_ZRELADDR is used.
> >>
> >>> Right, but the kernel image is compressed, so after decompression it expands
> >>> into the area just after it.  Also, the .bss segment is in that vicinity.
> >>
> >> To be exact, the compressed kernel moves itself out of the region where 
> >> the decompressed kernel will end up before doing the decompression, but 
> >> only if necessary.  So it is a good idea to load zImage away from the 
> >> decompressed kernel area to avoid this extra move and save some fraction 
> >> of a second on boot time.
> >>
> >>> There's some code in arch/arm/boot/compressed/head.S to relocate
> >>> device tree blobs, but it requires CONFIG_ARM_APPENDED_DTB which
> >>> is not recommended - arch/arm/Kconfig recommends using the
> >>> documented boot protocol istead .
> >>
> >> This is in case a DTB is appended to zImage.  When the DTB is detected, 
> >> the moving of zImage out of the decompressed area must take care of 
> >> moving the DTB as well.
> >>
> >>> Documentation/arm/Booting says
> >>> to put the dtb "in a region of memory where the kernel decompressor
> >>> will not overwrite it", further recommending the first 16KiB.
> >>>
> >>> As noted, the first 16KiB loses if the dtb is too large.  And
> >>> "where the kernel decompressor will not overwrite it" says what
> >>> won't work, not what will.  It appears that the decompressor works
> >>> out its addresses dynamically, so there's no hard prescription even
> >>> for what to avoid.
> >>
> >> A good rule of thumb is to take the size of the decompressed kernel and 
> >> multiply this by 3.  Rounding up is also fine.  So for example if your 
> >> arch/arm/boot/Image is 5MB, then putting anciliary data such as a 
> >> ramdisk or a large DTB from 16MB into RAM or above should be fine.
> >>
> >>> For now, I'm putting the initrd at the end of memory and the dtb
> >>> below that.  That seems to work, but I'm unsure whether or not
> >>> I'm just "getting lucky".
> >>
> >> That's also perfectly fine.
> > 
> > 
> > Alas, that worked for machines with 512 MiB of main memory, but failed
> > on 1 GiB machines.  My guess is that, when the initrd and dtb are near
> > the top of a 1 GiB memory, the virtual address gets too near the top of
> > the kernel's 1 GiB of virtual space (which starts at 0xc0000000),
> > perhaps colliding with the VMALLOC space.
> > 
> > Putting them just below the 128 MiB boundary seems to work.
> 
> Interesting. I think this is also a problem on u-boot just waiting to
> happen. u-boot locates itself at the end of RAM and likes to copy the
> fdt and initrd to just below that. Any machine with 1G+ is going to hit
> this. I avoided it because I limited u-boot to 512MB on highbank.

If I'm not mistaken, yes U-Boot loads itself as high as it can, and I don't
know about the FDT, but no, U-Boot does not "like" to load initrd "just
below that": it loads initrd where the boot commands tell it to, and the
boot commands are written by board developers. Nothing in U-Boot forces
initrd to be loaded as high as possible.

That leaves the question of the FDT, though -- I'm not familiar enough
with it (yet) to tell if it is always located just under U-Boot or if
its placement is controllable by board commands.

Amicalement,
-- 
Albert.

[U-Boot] Where to put a large bootloader-supplied device tree on ARM ?

[Wolfgang Denk]

Dear Albert ARIBAUD,

In message <20120712233801.0411daa7@lilith> you wrote:
> 
> If I'm not mistaken, yes U-Boot loads itself as high as it can, and I don't
> know about the FDT, but no, U-Boot does not "like" to load initrd "just
> below that": it loads initrd where the boot commands tell it to, and the
> boot commands are written by board developers. Nothing in U-Boot forces
> initrd to be loaded as high as possible.
> 
> That leaves the question of the FDT, though -- I'm not familiar enough
> with it (yet) to tell if it is always located just under U-Boot or if
> its placement is controllable by board commands.

You can always just set the "fdt_high" and "initrd_high" evironment
variables to restrict the positioning of initrd and FDT in RAM; see
the README for details.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
"He was so narrow minded he could see through  a  keyhole  with  both
eyes ..."

[U-Boot] Where to put a large bootloader-supplied device tree on ARM ?

[Rob Herring]

On 07/12/2012 04:47 PM, Wolfgang Denk wrote:
> Dear Albert ARIBAUD,
> 
> In message <20120712233801.0411daa7@lilith> you wrote:
>>
>> If I'm not mistaken, yes U-Boot loads itself as high as it can, and I don't
>> know about the FDT, but no, U-Boot does not "like" to load initrd "just
>> below that": it loads initrd where the boot commands tell it to, and the
>> boot commands are written by board developers. Nothing in U-Boot forces
>> initrd to be loaded as high as possible.

u-boot loads the initrd where you tell it, then bootm relocates it for
some reason.

>>
>> That leaves the question of the FDT, though -- I'm not familiar enough
>> with it (yet) to tell if it is always located just under U-Boot or if
>> its placement is controllable by board commands.
> 
> You can always just set the "fdt_high" and "initrd_high" evironment
> variables to restrict the positioning of initrd and FDT in RAM; see
> the README for details.

Yes, I'm aware of all this, but this is not the default behavior and the
default behavior will not work in this case. It also has another bug
related to the relocation that I fixed:

http://www.mail-archive.com/u-boot at lists.denx.de/msg86475.html

Rob

[U-Boot] Where to put a large bootloader-supplied device tree on ARM ?

[Albert ARIBAUD]

Hi Rob,

On Thu, 12 Jul 2012 20:28:22 -0500, Rob Herring <robherring2@gmail.com> wrote:
> On 07/12/2012 04:47 PM, Wolfgang Denk wrote:
> > Dear Albert ARIBAUD,
> > 
> > In message <20120712233801.0411daa7@lilith> you wrote:
> >>
> >> If I'm not mistaken, yes U-Boot loads itself as high as it can, and I don't
> >> know about the FDT, but no, U-Boot does not "like" to load initrd "just
> >> below that": it loads initrd where the boot commands tell it to, and the
> >> boot commands are written by board developers. Nothing in U-Boot forces
> >> initrd to be loaded as high as possible.
> 
> u-boot loads the initrd where you tell it, then bootm relocates it for
> some reason.
> >>
> >> That leaves the question of the FDT, though -- I'm not familiar enough
> >> with it (yet) to tell if it is always located just under U-Boot or if
> >> its placement is controllable by board commands.
> > 
> > You can always just set the "fdt_high" and "initrd_high" evironment
> > variables to restrict the positioning of initrd and FDT in RAM; see
> > the README for details.
> 
> Yes, I'm aware of all this, but this is not the default behavior and the
> default behavior will not work in this case. It also has another bug
> related to the relocation that I fixed:
> 
> http://www.mail-archive.com/u-boot at lists.denx.de/msg86475.html
> 
> Rob

As for the default behavior, this becomes a board config issue, not a
U-Boot issue.

Regarding the patch, as a bugfix it will go in u-boot-arm/master
for 2012.07.

With all this, do we have the overall issue covered?

Amicalement,
-- 
Albert.