[U-Boot] Default initrd and fdt load behaviour on ARM (Was: Re: [linux-sunxi] Re: Not able to boot ramdisk on cubietruck)

[U-Boot] Default initrd and fdt load behaviour on ARM (Was: Re: [linux-sunxi] Re: Not able to boot ramdisk on cubietruck)

[Ian Campbell]

Adding u-boot list since I think this is a general issue/question.

On Sun, 2014-03-09 at 20:00 +0100, Olliver Schinagl wrote:
> On 03/09/14 16:18, tyler.baker at linaro.org wrote:
> > On Sunday, March 9, 2014 8:06:27 AM UTC-7, tyler... at linaro.org wrote:
> >> Hello,
> >>
> >> I am trying to boot the cubietruck with a minimal ramdisk. However, it
> >> seems to hang at "Starting kernel..." whenever I pass bootz a
> >> ramdisk load address.
[...]
> > Turl help me solve this in IRC. Needed to setenv initrd_high 
> > '0xffffffff' in case anyone else runs into this.

> The http://linux-sunxi.org/Mainline_Kernel_Howto did mention this ;) but 
> thanks for helping remind people!

Actually it mentions fdt_high but not initrd_high.

But what is the reason for the default behaviour of bootz doing things
which do not conform to linux/Documentation/arm/Booting and therefore is
not going to boot?

The docs recommend that DTB and initrd go just after the 128MB boundary.
If either are loaded "too high" then the kernel will crash on boot,
often in a tricky to diagnose way (I've chased it down more than once).
It's especially annoying when one has carefully loaded everything at a
suitable address in the first place ;-)

Should the global default be changed to either 0xffffffff (no
relocation) or to start-of-ram+256MB (which should satisfy the kernels
requirements without needing logic changes to handle "just after the
128MB boundary" as a concept).

Or is it intended that each board should opt-in to a sensible default
via their default environment?

Does bootm suffer the same? I suspect it does, at least for fdt (since
initrd has a load addr in the u-boot header).

Ian.

[U-Boot] Default initrd and fdt load behaviour on ARM (Was: Re: [linux-sunxi] Re: Not able to boot ramdisk on cubietruck)

[Tom Rini]

On Tue, Mar 11, 2014 at 10:38:05AM +0000, Ian Campbell wrote:
> Adding u-boot list since I think this is a general issue/question.
> 
> On Sun, 2014-03-09 at 20:00 +0100, Olliver Schinagl wrote:
> > On 03/09/14 16:18, tyler.baker at linaro.org wrote:
> > > On Sunday, March 9, 2014 8:06:27 AM UTC-7, tyler... at linaro.org wrote:
> > >> Hello,
> > >>
> > >> I am trying to boot the cubietruck with a minimal ramdisk. However, it
> > >> seems to hang at "Starting kernel..." whenever I pass bootz a
> > >> ramdisk load address.
> [...]
> > > Turl help me solve this in IRC. Needed to setenv initrd_high 
> > > '0xffffffff' in case anyone else runs into this.
> 
> > The http://linux-sunxi.org/Mainline_Kernel_Howto did mention this ;) but 
> > thanks for helping remind people!
> 
> Actually it mentions fdt_high but not initrd_high.
> 
> But what is the reason for the default behaviour of bootz doing things
> which do not conform to linux/Documentation/arm/Booting and therefore is
> not going to boot?

Because U-Boot is more than just ARM, and ARM boards not enforcing /
getting the correct behavour via fdt_high / initrd_high or bootm_low /
bootm_mapsize / bootm_size / CONFIG_SYS_BOOTMAPSZ.

> The docs recommend that DTB and initrd go just after the 128MB boundary.
> If either are loaded "too high" then the kernel will crash on boot,
> often in a tricky to diagnose way (I've chased it down more than once).
> It's especially annoying when one has carefully loaded everything at a
> suitable address in the first place ;-)

I have as well, sadly.

> Should the global default be changed to either 0xffffffff (no
> relocation) or to start-of-ram+256MB (which should satisfy the kernels
> requirements without needing logic changes to handle "just after the
> 128MB boundary" as a concept).
> 
> Or is it intended that each board should opt-in to a sensible default
> via their default environment?
> 
> Does bootm suffer the same? I suspect it does, at least for fdt (since
> initrd has a load addr in the u-boot header).

SoCs should set a sane and correct set of values, using either of the
available mechanisms.  I think Tegra gets this all correct via
CONFIG_SYS_BOOTMAPSZ and I've been meaning to whack the various TI parts
as part of the generic distro work.

And keep in mind that when we set fdt/initrd_high to a non-0xffffffff
value we're setting a maximum that's still checked vs how much we have,
so setting base+512MB is probably a best default for cases where we may
run on boards with varying amounts of DDR.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20140311/5d725124/attachment.pgp>

[U-Boot] Default initrd and fdt load behaviour on ARM (Was: Re: [linux-sunxi] Re: Not able to boot ramdisk on cubietruck)

[Ian Campbell]

On Tue, 2014-03-11 at 09:48 -0400, Tom Rini wrote:
> On Tue, Mar 11, 2014 at 10:38:05AM +0000, Ian Campbell wrote:
> > But what is the reason for the default behaviour of bootz doing things
> > which do not conform to linux/Documentation/arm/Booting and therefore is
> > not going to boot?
> 
> Because U-Boot is more than just ARM, and ARM boards not enforcing /
> getting the correct behavour via fdt_high / initrd_high or bootm_low /
> bootm_mapsize / bootm_size / CONFIG_SYS_BOOTMAPSZ.

So you consider this to be a board level problem, rather than an arch
level one? (IOW we can't just fix this for all arm boards at once)

[...]
> > Should the global default be changed to either 0xffffffff (no
> > relocation) or to start-of-ram+256MB (which should satisfy the kernels
> > requirements without needing logic changes to handle "just after the
> > 128MB boundary" as a concept).
> > 
> > Or is it intended that each board should opt-in to a sensible default
> > via their default environment?
> > 
> > Does bootm suffer the same? I suspect it does, at least for fdt (since
> > initrd has a load addr in the u-boot header).
> 
> SoCs should set a sane and correct set of values, using either of the
> available mechanisms.  I think Tegra gets this all correct via
> CONFIG_SYS_BOOTMAPSZ and I've been meaning to whack the various TI parts
> as part of the generic distro work.

I took a look at BOOTMAPSZ on sunxi and it is using (16 << 20) which
seems sensible, if a bit small, so perhaps something else is going
wrong.

Tegra uses 256MB, might be worth giving that a go I guess.

> And keep in mind that when we set fdt/initrd_high to a non-0xffffffff
> value we're setting a maximum that's still checked vs how much we have,
> so setting base+512MB is probably a best default for cases where we may
> run on boards with varying amounts of DDR.

I'm not sure I follow -- why is +512MB best?

Thanks,
Ian.

[U-Boot] Default initrd and fdt load behaviour on ARM (Was: Re: [linux-sunxi] Re: Not able to boot ramdisk on cubietruck)

[Tom Rini]

On Tue, Mar 11, 2014 at 03:21:54PM +0000, Ian Campbell wrote:
> On Tue, 2014-03-11 at 09:48 -0400, Tom Rini wrote:
> > On Tue, Mar 11, 2014 at 10:38:05AM +0000, Ian Campbell wrote:
> > > But what is the reason for the default behaviour of bootz doing things
> > > which do not conform to linux/Documentation/arm/Booting and therefore is
> > > not going to boot?
> > 
> > Because U-Boot is more than just ARM, and ARM boards not enforcing /
> > getting the correct behavour via fdt_high / initrd_high or bootm_low /
> > bootm_mapsize / bootm_size / CONFIG_SYS_BOOTMAPSZ.
> 
> So you consider this to be a board level problem, rather than an arch
> level one? (IOW we can't just fix this for all arm boards at once)

I'm not 100% sure.  An issue is that lowmem isn't always 512MB, it may
be 768MB.  That also may not really matter for this, I'm not sure.  But
part of the problem is that AFAICS no one provides a 100% correct
example today.  With some testing around now, for TI platforms (DDR
starts at 0x80000000 it would be:
/*
 * We setup defaults based on constraints from the Linux kernel, which
 * should also be safe elsewhere.  We have the default load at 32MB into
 * DDR (for the kernel), FDT above 128MB (the maximum location for the
 * end of the kernel), and the ramdisk 512KB above that (allowing for
 * hopefully never seen large trees).  We say all of this must be within
 * the first 512MB as that will always be within the kernel lowmem and
 * thus visible via bootm_size.
 */
#define CONFIG_EXTRA_ENV_SETTINGS \
-       "loadaddr=0x80200000\0" \
-       "fdtaddr=0x80F80000\0" \
-       "fdt_high=0xffffffff\0" \
+       "loadaddr=0x82000000\0" \
+       "fdtaddr=0x88000000\0" \
+       "rdaddr=0x88080000\0" \
+       "bootm_size=0x20000000\0" \

All of that said, I could be convinced that we could try and make
common/image.c::getenv_bootm_size() smarter about defaults but we need
to take a lot of care there, and allow people to opt-out as that's
common code and how much lowmem we have is configurable a lot.

> [...]
> > > Should the global default be changed to either 0xffffffff (no
> > > relocation) or to start-of-ram+256MB (which should satisfy the kernels
> > > requirements without needing logic changes to handle "just after the
> > > 128MB boundary" as a concept).
> > > 
> > > Or is it intended that each board should opt-in to a sensible default
> > > via their default environment?
> > > 
> > > Does bootm suffer the same? I suspect it does, at least for fdt (since
> > > initrd has a load addr in the u-boot header).
> > 
> > SoCs should set a sane and correct set of values, using either of the
> > available mechanisms.  I think Tegra gets this all correct via
> > CONFIG_SYS_BOOTMAPSZ and I've been meaning to whack the various TI parts
> > as part of the generic distro work.
> 
> I took a look at BOOTMAPSZ on sunxi and it is using (16 << 20) which
> seems sensible, if a bit small, so perhaps something else is going
> wrong.
> 
> Tegra uses 256MB, might be worth giving that a go I guess.

Doing some playing around, BOOTMAPSZ isn't a complete solution, you need
bootm_low also set.  In fact, I found the best results with just doing
botom_low but there may be cases where you also want BOOTMAPSZ.

> > And keep in mind that when we set fdt/initrd_high to a non-0xffffffff
> > value we're setting a maximum that's still checked vs how much we have,
> > so setting base+512MB is probably a best default for cases where we may
> > run on boards with varying amounts of DDR.
> 
> I'm not sure I follow -- why is +512MB best?

I'm a fan of setting things to the maximum allowed, to allow for future
growth.  This would also allow for pretty giant ramdisks to be passed
which is unlikely but not impossible (I need to debug X and I can't use
nfs/sd/etc for root, let me assemble a big ramdisk with everything I
need).  Getting back to why this would be left up to the SoC/board
level, if for some reason there's a reason to not go up quite that high,
they can tweak it.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20140311/e29b6ef9/attachment.pgp>

[U-Boot] Default initrd and fdt load behaviour on ARM (Was: Re: [linux-sunxi] Re: Not able to boot ramdisk on cubietruck)

[Ian Campbell]

On Tue, 2014-03-11 at 13:17 -0400, Tom Rini wrote:
> On Tue, Mar 11, 2014 at 03:21:54PM +0000, Ian Campbell wrote:
> > On Tue, 2014-03-11 at 09:48 -0400, Tom Rini wrote:
> > > On Tue, Mar 11, 2014 at 10:38:05AM +0000, Ian Campbell wrote:
> > > > But what is the reason for the default behaviour of bootz doing things
> > > > which do not conform to linux/Documentation/arm/Booting and therefore is
> > > > not going to boot?
> > > 
> > > Because U-Boot is more than just ARM, and ARM boards not enforcing /
> > > getting the correct behavour via fdt_high / initrd_high or bootm_low /
> > > bootm_mapsize / bootm_size / CONFIG_SYS_BOOTMAPSZ.
> > 
> > So you consider this to be a board level problem, rather than an arch
> > level one? (IOW we can't just fix this for all arm boards at once)
> 
> I'm not 100% sure.  An issue is that lowmem isn't always 512MB, it may
> be 768MB.

I think in reality it can vary substantially between the two (or even
outside it) based on the kernel's .config (so outside u-boot's control).

>   That also may not really matter for this, I'm not sure.  But
> part of the problem is that AFAICS no one provides a 100% correct
> example today.  With some testing around now, for TI platforms (DDR
> starts at 0x80000000 it would be:
> /*
>  * We setup defaults based on constraints from the Linux kernel, which
>  * should also be safe elsewhere.  We have the default load at 32MB into
>  * DDR (for the kernel), FDT above 128MB (the maximum location for the
>  * end of the kernel), and the ramdisk 512KB above that (allowing for
>  * hopefully never seen large trees).  We say all of this must be within
>  * the first 512MB as that will always be within the kernel lowmem and
>  * thus visible via bootm_size.
>  */
> #define CONFIG_EXTRA_ENV_SETTINGS \
> -       "loadaddr=0x80200000\0" \
> -       "fdtaddr=0x80F80000\0" \
> -       "fdt_high=0xffffffff\0" \
> +       "loadaddr=0x82000000\0" \
> +       "fdtaddr=0x88000000\0" \
> +       "rdaddr=0x88080000\0" \
> +       "bootm_size=0x20000000\0" \
> 
> All of that said, I could be convinced that we could try and make
> common/image.c::getenv_bootm_size() smarter about defaults but we need
> to take a lot of care there, and allow people to opt-out as that's
> common code and how much lowmem we have is configurable a lot.

> > [...]
> > > > Should the global default be changed to either 0xffffffff (no
> > > > relocation) or to start-of-ram+256MB (which should satisfy the kernels
> > > > requirements without needing logic changes to handle "just after the
> > > > 128MB boundary" as a concept).
> > > > 
> > > > Or is it intended that each board should opt-in to a sensible default
> > > > via their default environment?
> > > > 
> > > > Does bootm suffer the same? I suspect it does, at least for fdt (since
> > > > initrd has a load addr in the u-boot header).
> > > 
> > > SoCs should set a sane and correct set of values, using either of the
> > > available mechanisms.  I think Tegra gets this all correct via
> > > CONFIG_SYS_BOOTMAPSZ and I've been meaning to whack the various TI parts
> > > as part of the generic distro work.
> > 
> > I took a look at BOOTMAPSZ on sunxi and it is using (16 << 20) which
> > seems sensible, if a bit small, so perhaps something else is going
> > wrong.
> > 
> > Tegra uses 256MB, might be worth giving that a go I guess.
> 
> Doing some playing around, BOOTMAPSZ isn't a complete solution, you need
> bootm_low also set.  In fact, I found the best results with just doing
> botom_low but there may be cases where you also want BOOTMAPSZ.

I'll put fiddling with this for sunxi on my todo list. Although I kind
of hope someone else will beat me to it ;-)

> > > And keep in mind that when we set fdt/initrd_high to a non-0xffffffff
> > > value we're setting a maximum that's still checked vs how much we have,
> > > so setting base+512MB is probably a best default for cases where we may
> > > run on boards with varying amounts of DDR.
> > 
> > I'm not sure I follow -- why is +512MB best?
> 
> I'm a fan of setting things to the maximum allowed, to allow for future
> growth.  This would also allow for pretty giant ramdisks to be passed
> which is unlikely but not impossible (I need to debug X and I can't use
> nfs/sd/etc for root, let me assemble a big ramdisk with everything I
> need).

That makes sense.

On the other hand the "just above 128MB" advice is even better since
that will Just Work and allow images to stretch to the end of RAM if
necessary. But I suppose that is a far bigger rework on the u-boot side
of things, to flip from "maximum address" allocation to "starting from"
allocation. I don't think I want to get into that...

>   Getting back to why this would be left up to the SoC/board
> level, if for some reason there's a reason to not go up quite that high,
> they can tweak it.

Ian.

[U-Boot] Default initrd and fdt load behaviour on ARM (Was: Re: [linux-sunxi] Re: Not able to boot ramdisk on cubietruck)

[Tom Rini]

On Wed, Mar 12, 2014 at 01:04:38PM +0000, Ian Campbell wrote:
> On Tue, 2014-03-11 at 13:17 -0400, Tom Rini wrote:
> > On Tue, Mar 11, 2014 at 03:21:54PM +0000, Ian Campbell wrote:
> > > On Tue, 2014-03-11 at 09:48 -0400, Tom Rini wrote:
> > > > On Tue, Mar 11, 2014 at 10:38:05AM +0000, Ian Campbell wrote:
> > > > > But what is the reason for the default behaviour of bootz doing things
> > > > > which do not conform to linux/Documentation/arm/Booting and therefore is
> > > > > not going to boot?
> > > > 
> > > > Because U-Boot is more than just ARM, and ARM boards not enforcing /
> > > > getting the correct behavour via fdt_high / initrd_high or bootm_low /
> > > > bootm_mapsize / bootm_size / CONFIG_SYS_BOOTMAPSZ.
> > > 
> > > So you consider this to be a board level problem, rather than an arch
> > > level one? (IOW we can't just fix this for all arm boards at once)
> > 
> > I'm not 100% sure.  An issue is that lowmem isn't always 512MB, it may
> > be 768MB.
> 
> I think in reality it can vary substantially between the two (or even
> outside it) based on the kernel's .config (so outside u-boot's control).

Yeah, I think you're right.  Which is why this is a board/SoC level
problem to provide "good enough" defaults.

> >   That also may not really matter for this, I'm not sure.  But
> > part of the problem is that AFAICS no one provides a 100% correct
> > example today.  With some testing around now, for TI platforms (DDR
> > starts at 0x80000000 it would be:
> > /*
> >  * We setup defaults based on constraints from the Linux kernel, which
> >  * should also be safe elsewhere.  We have the default load at 32MB into
> >  * DDR (for the kernel), FDT above 128MB (the maximum location for the
> >  * end of the kernel), and the ramdisk 512KB above that (allowing for
> >  * hopefully never seen large trees).  We say all of this must be within
> >  * the first 512MB as that will always be within the kernel lowmem and
> >  * thus visible via bootm_size.
> >  */
> > #define CONFIG_EXTRA_ENV_SETTINGS \
> > -       "loadaddr=0x80200000\0" \
> > -       "fdtaddr=0x80F80000\0" \
> > -       "fdt_high=0xffffffff\0" \
> > +       "loadaddr=0x82000000\0" \
> > +       "fdtaddr=0x88000000\0" \
> > +       "rdaddr=0x88080000\0" \
> > +       "bootm_size=0x20000000\0" \
> > 
> > All of that said, I could be convinced that we could try and make
> > common/image.c::getenv_bootm_size() smarter about defaults but we need
> > to take a lot of care there, and allow people to opt-out as that's
> > common code and how much lowmem we have is configurable a lot.
> 
> > > [...]
> > > > > Should the global default be changed to either 0xffffffff (no
> > > > > relocation) or to start-of-ram+256MB (which should satisfy the kernels
> > > > > requirements without needing logic changes to handle "just after the
> > > > > 128MB boundary" as a concept).
> > > > > 
> > > > > Or is it intended that each board should opt-in to a sensible default
> > > > > via their default environment?
> > > > > 
> > > > > Does bootm suffer the same? I suspect it does, at least for fdt (since
> > > > > initrd has a load addr in the u-boot header).
> > > > 
> > > > SoCs should set a sane and correct set of values, using either of the
> > > > available mechanisms.  I think Tegra gets this all correct via
> > > > CONFIG_SYS_BOOTMAPSZ and I've been meaning to whack the various TI parts
> > > > as part of the generic distro work.
> > > 
> > > I took a look at BOOTMAPSZ on sunxi and it is using (16 << 20) which
> > > seems sensible, if a bit small, so perhaps something else is going
> > > wrong.
> > > 
> > > Tegra uses 256MB, might be worth giving that a go I guess.
> > 
> > Doing some playing around, BOOTMAPSZ isn't a complete solution, you need
> > bootm_low also set.  In fact, I found the best results with just doing
> > botom_low but there may be cases where you also want BOOTMAPSZ.
> 
> I'll put fiddling with this for sunxi on my todo list. Although I kind
> of hope someone else will beat me to it ;-)

Having whacked at it for the TI parts, I strongly suspect sunxi can also
get away with defaulting to including bootm_low (which yes, also covers
bootz) set to 512MB or 256MB and possibly also...

> > > > And keep in mind that when we set fdt/initrd_high to a non-0xffffffff
> > > > value we're setting a maximum that's still checked vs how much we have,
> > > > so setting base+512MB is probably a best default for cases where we may
> > > > run on boards with varying amounts of DDR.
> > > 
> > > I'm not sure I follow -- why is +512MB best?
> > 
> > I'm a fan of setting things to the maximum allowed, to allow for future
> > growth.  This would also allow for pretty giant ramdisks to be passed
> > which is unlikely but not impossible (I need to debug X and I can't use
> > nfs/sd/etc for root, let me assemble a big ramdisk with everything I
> > need).
> 
> That makes sense.
> 
> On the other hand the "just above 128MB" advice is even better since
> that will Just Work and allow images to stretch to the end of RAM if
> necessary. But I suppose that is a far bigger rework on the u-boot side
> of things, to flip from "maximum address" allocation to "starting from"
> allocation. I don't think I want to get into that...

Nope, you've got it backwards.  If set set bootm_low to 512MB then we
allow for ramdisks to grow to the end of usable memory (and since it's
overridable if you have a case with a kernel of 1GB lowmem and a 768MB
ramdisk, you can adjust and be fine).  The starting point for these
should be just above 128MB (unless you know you have to work with 128MB
or smaller boards) since that is a real limit on usable ARM kernels,
without major rework.

But, "starting from" allocation would just be loading from low and
setting fdt/initrd_high to 0xffffffff so we don't move it.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20140312/885e0b2c/attachment.pgp>