[U-Boot] [RFC] ARM: Start using fdt_high for relocation

[U-Boot] [RFC] ARM: Start using fdt_high for relocation

[Tom Rini]

Hey all,

I've been thinking.  We've had a thread on i.MX platforms about fdt
being overwritten and needing to be moved to another address.  And I've
also had an internal problem about fdt being overwritten.  So, how about
as a rule of thumb we start setting fdt_high (in configs) to
memory-start + 512MiB, as that's the lowmem limit we should always have
available.  This will fix the problem of BSS overwriting the DT, which
is the problem we won't catch in normal bootm/bootz usage.  Thoughts?

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20131206/26377e53/attachment.pgp>

[U-Boot] [RFC] ARM: Start using fdt_high for relocation

[Albert ARIBAUD]

Hi Tom,

On Fri, 6 Dec 2013 10:48:50 -0500, Tom Rini <trini@ti.com> wrote:

> Hey all,
> 
> I've been thinking.  We've had a thread on i.MX platforms about fdt
> being overwritten and needing to be moved to another address.  And I've
> also had an internal problem about fdt being overwritten.  So, how about
> as a rule of thumb we start setting fdt_high (in configs) to
> memory-start + 512MiB, as that's the lowmem limit we should always have
> available.  This will fix the problem of BSS overwriting the DT, which
> is the problem we won't catch in normal bootm/bootz usage.  Thoughts?

Not sure I'm getting the issue clear, and I would like to avoid (me and
others) having to switch back and forth between threads. Can you sketch
the failure scenario in a couple of lines?

Amicalement,
-- 
Albert.

[U-Boot] [RFC] ARM: Start using fdt_high for relocation

[Tom Rini]

On Fri, Dec 06, 2013 at 05:59:37PM +0100, Albert ARIBAUD wrote:
> Hi Tom,
> 
> On Fri, 6 Dec 2013 10:48:50 -0500, Tom Rini <trini@ti.com> wrote:
> 
> > Hey all,
> > 
> > I've been thinking.  We've had a thread on i.MX platforms about fdt
> > being overwritten and needing to be moved to another address.  And I've
> > also had an internal problem about fdt being overwritten.  So, how about
> > as a rule of thumb we start setting fdt_high (in configs) to
> > memory-start + 512MiB, as that's the lowmem limit we should always have
> > available.  This will fix the problem of BSS overwriting the DT, which
> > is the problem we won't catch in normal bootm/bootz usage.  Thoughts?
> 
> Not sure I'm getting the issue clear, and I would like to avoid (me and
> others) having to switch back and forth between threads. Can you sketch
> the failure scenario in a couple of lines?

Sure.  Lets take am335x_evm builds (so Beaglebone Black/White, etc).
If you start enabling all of the tracing options in the kernel (function
tracing, graphs, etc), you get an uncompressed kernel and BSS that will
use up the first ~16MiB of DDR.  We default to placing the DT at about
15MiB into memory.  So the kernel runs, clears BSS and eats the DT.
System now hangs, and depending on debug options set you may or may not
see anything at all from the kernel.  U-Boot couldn't detect this
failure because we don't know how big the kernel BSS is, only how big
the zImage is (and where it is) and how big the fdt is and where it is.
No overlaps, go ahead and run.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20131206/8437870f/attachment.pgp>

[U-Boot] [RFC] ARM: Start using fdt_high for relocation

[Albert ARIBAUD]

Hi Tom,

On Fri, 6 Dec 2013 12:18:13 -0500, Tom Rini <trini@ti.com> wrote:

> On Fri, Dec 06, 2013 at 05:59:37PM +0100, Albert ARIBAUD wrote:
> > Hi Tom,
> > 
> > On Fri, 6 Dec 2013 10:48:50 -0500, Tom Rini <trini@ti.com> wrote:
> > 
> > > Hey all,
> > > 
> > > I've been thinking.  We've had a thread on i.MX platforms about fdt
> > > being overwritten and needing to be moved to another address.  And I've
> > > also had an internal problem about fdt being overwritten.  So, how about
> > > as a rule of thumb we start setting fdt_high (in configs) to
> > > memory-start + 512MiB, as that's the lowmem limit we should always have
> > > available.  This will fix the problem of BSS overwriting the DT, which
> > > is the problem we won't catch in normal bootm/bootz usage.  Thoughts?
> > 
> > Not sure I'm getting the issue clear, and I would like to avoid (me and
> > others) having to switch back and forth between threads. Can you sketch
> > the failure scenario in a couple of lines?
> 
> Sure.  Lets take am335x_evm builds (so Beaglebone Black/White, etc).
> If you start enabling all of the tracing options in the kernel (function
> tracing, graphs, etc), you get an uncompressed kernel and BSS that will
> use up the first ~16MiB of DDR.  We default to placing the DT at about
> 15MiB into memory.  So the kernel runs, clears BSS and eats the DT.
> System now hangs, and depending on debug options set you may or may not
> see anything at all from the kernel.  U-Boot couldn't detect this
> failure because we don't know how big the kernel BSS is, only how big
> the zImage is (and where it is) and how big the fdt is and where it is.
> No overlaps, go ahead and run.

Thanks.

The only issue I have with the RFC is that the +512 MiB value will only
work with targets which have more than 512 MiB DDR, right? But since
you're suggesting this should be set in configs, you are only suggesting
+512 MiB, and any target could actually specify a lower value as long as
it's greater than or eqal to 16 MiB. Correct?

Amicalement,
-- 
Albert.

[U-Boot] [RFC] ARM: Start using fdt_high for relocation

[Tom Rini]

On Fri, Dec 06, 2013 at 08:28:04PM +0100, Albert ARIBAUD wrote:
> Hi Tom,
> 
> On Fri, 6 Dec 2013 12:18:13 -0500, Tom Rini <trini@ti.com> wrote:
> 
> > On Fri, Dec 06, 2013 at 05:59:37PM +0100, Albert ARIBAUD wrote:
> > > Hi Tom,
> > > 
> > > On Fri, 6 Dec 2013 10:48:50 -0500, Tom Rini <trini@ti.com> wrote:
> > > 
> > > > Hey all,
> > > > 
> > > > I've been thinking.  We've had a thread on i.MX platforms about fdt
> > > > being overwritten and needing to be moved to another address.  And I've
> > > > also had an internal problem about fdt being overwritten.  So, how about
> > > > as a rule of thumb we start setting fdt_high (in configs) to
> > > > memory-start + 512MiB, as that's the lowmem limit we should always have
> > > > available.  This will fix the problem of BSS overwriting the DT, which
> > > > is the problem we won't catch in normal bootm/bootz usage.  Thoughts?
> > > 
> > > Not sure I'm getting the issue clear, and I would like to avoid (me and
> > > others) having to switch back and forth between threads. Can you sketch
> > > the failure scenario in a couple of lines?
> > 
> > Sure.  Lets take am335x_evm builds (so Beaglebone Black/White, etc).
> > If you start enabling all of the tracing options in the kernel (function
> > tracing, graphs, etc), you get an uncompressed kernel and BSS that will
> > use up the first ~16MiB of DDR.  We default to placing the DT at about
> > 15MiB into memory.  So the kernel runs, clears BSS and eats the DT.
> > System now hangs, and depending on debug options set you may or may not
> > see anything at all from the kernel.  U-Boot couldn't detect this
> > failure because we don't know how big the kernel BSS is, only how big
> > the zImage is (and where it is) and how big the fdt is and where it is.
> > No overlaps, go ahead and run.
> 
> Thanks.
> 
> The only issue I have with the RFC is that the +512 MiB value will only
> work with targets which have more than 512 MiB DDR, right? But since
> you're suggesting this should be set in configs, you are only suggesting
> +512 MiB, and any target could actually specify a lower value as long as
> it's greater than or eqal to 16 MiB. Correct?

fdt_high is only an upper bound on what we may relocate to (setting
aside the magic value of 0xffffffff which means no relocation).  I just
confirmed this too:
U-Boot# bdi
...
boot_params = 0x80000100
DRAM bank   = 0x00000000
-> start    = 0x80000000
-> size     = 0x40000000
...
U-Boot# setenv fdt_high 0xe0000000
...
U-Boot# bootz $loadaddr - $fdtaddr
Kernel image @ 0x80200000 [ 0x000000 - 0x3e5068 ]
## Flattened Device Tree blob at 80f80000
   Booting using the fdt blob at 0x80f80000
   Loading Device Tree to bf62a000, end bf630d9a ... OK

Of course, 0xbf62a000 is a bad address in that it won't be seen by the
kernel, but it was in the higher-than-we-have-memory-at limit I set.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20131206/947a51e0/attachment.pgp>

[U-Boot] [RFC] ARM: Start using fdt_high for relocation

[Stephen Warren]

On 12/06/2013 01:31 PM, Tom Rini wrote:
> On Fri, Dec 06, 2013 at 08:28:04PM +0100, Albert ARIBAUD wrote:
>> Hi Tom,
>>
>> On Fri, 6 Dec 2013 12:18:13 -0500, Tom Rini <trini@ti.com> wrote:
>>
>>> On Fri, Dec 06, 2013 at 05:59:37PM +0100, Albert ARIBAUD wrote:
>>>> Hi Tom,
>>>>
>>>> On Fri, 6 Dec 2013 10:48:50 -0500, Tom Rini <trini@ti.com> wrote:
>>>>
>>>>> Hey all,
>>>>>
>>>>> I've been thinking.  We've had a thread on i.MX platforms about fdt
>>>>> being overwritten and needing to be moved to another address.  And I've
>>>>> also had an internal problem about fdt being overwritten.  So, how about
>>>>> as a rule of thumb we start setting fdt_high (in configs) to
>>>>> memory-start + 512MiB, as that's the lowmem limit we should always have
>>>>> available.  This will fix the problem of BSS overwriting the DT, which
>>>>> is the problem we won't catch in normal bootm/bootz usage.  Thoughts?
>>>>
>>>> Not sure I'm getting the issue clear, and I would like to avoid (me and
>>>> others) having to switch back and forth between threads. Can you sketch
>>>> the failure scenario in a couple of lines?
>>>
>>> Sure.  Lets take am335x_evm builds (so Beaglebone Black/White, etc).
>>> If you start enabling all of the tracing options in the kernel (function
>>> tracing, graphs, etc), you get an uncompressed kernel and BSS that will
>>> use up the first ~16MiB of DDR.  We default to placing the DT at about
>>> 15MiB into memory.  So the kernel runs, clears BSS and eats the DT.
>>> System now hangs, and depending on debug options set you may or may not
>>> see anything at all from the kernel.  U-Boot couldn't detect this
>>> failure because we don't know how big the kernel BSS is, only how big
>>> the zImage is (and where it is) and how big the fdt is and where it is.
>>> No overlaps, go ahead and run.
>>
>> Thanks.
>>
>> The only issue I have with the RFC is that the +512 MiB value will only
>> work with targets which have more than 512 MiB DDR, right? But since
>> you're suggesting this should be set in configs, you are only suggesting
>> +512 MiB, and any target could actually specify a lower value as long as
>> it's greater than or eqal to 16 MiB. Correct?
> 
> fdt_high is only an upper bound on what we may relocate to (setting
> aside the magic value of 0xffffffff which means no relocation).  I just
> confirmed this too:
> U-Boot# bdi
> ...
> boot_params = 0x80000100
> DRAM bank   = 0x00000000
> -> start    = 0x80000000
> -> size     = 0x40000000
> ...
> U-Boot# setenv fdt_high 0xe0000000
> ...
> U-Boot# bootz $loadaddr - $fdtaddr
> Kernel image @ 0x80200000 [ 0x000000 - 0x3e5068 ]
> ## Flattened Device Tree blob at 80f80000
>    Booting using the fdt blob at 0x80f80000
>    Loading Device Tree to bf62a000, end bf630d9a ... OK
> 
> Of course, 0xbf62a000 is a bad address in that it won't be seen by the
> kernel, but it was in the higher-than-we-have-memory-at limit I set.

I haven't really followed this thread since I just noticed it
tangentially, but on Tegra...

Since commit 7f1b767aea94 "ARM: tegra: define CONFIG_SYS_BOOTMAPSZ"
all Tegra boards define BOOTMAPSZ to influence where the DTB gets
relocated. I wonder what's the benefit of using BOOTMAPSZ vs. defining
a default value for $fdt_high?

And couldn't you just move $fdt_addr_r high enough in RAM so even if
the DTB wasn't relocated it'd never overlap BSS? Tegra's
$kernel_addr_r and $fdt_addr_r are likely set up that way already,
although I didn't check.

[U-Boot] [RFC] ARM: Start using fdt_high for relocation

[Tom Rini]

On Fri, Dec 06, 2013 at 01:44:22PM -0700, Stephen Warren wrote:
> On 12/06/2013 01:31 PM, Tom Rini wrote:
> > On Fri, Dec 06, 2013 at 08:28:04PM +0100, Albert ARIBAUD wrote:
> >> Hi Tom,
> >>
> >> On Fri, 6 Dec 2013 12:18:13 -0500, Tom Rini <trini@ti.com> wrote:
> >>
> >>> On Fri, Dec 06, 2013 at 05:59:37PM +0100, Albert ARIBAUD wrote:
> >>>> Hi Tom,
> >>>>
> >>>> On Fri, 6 Dec 2013 10:48:50 -0500, Tom Rini <trini@ti.com> wrote:
> >>>>
> >>>>> Hey all,
> >>>>>
> >>>>> I've been thinking.  We've had a thread on i.MX platforms about fdt
> >>>>> being overwritten and needing to be moved to another address.  And I've
> >>>>> also had an internal problem about fdt being overwritten.  So, how about
> >>>>> as a rule of thumb we start setting fdt_high (in configs) to
> >>>>> memory-start + 512MiB, as that's the lowmem limit we should always have
> >>>>> available.  This will fix the problem of BSS overwriting the DT, which
> >>>>> is the problem we won't catch in normal bootm/bootz usage.  Thoughts?
> >>>>
> >>>> Not sure I'm getting the issue clear, and I would like to avoid (me and
> >>>> others) having to switch back and forth between threads. Can you sketch
> >>>> the failure scenario in a couple of lines?
> >>>
> >>> Sure.  Lets take am335x_evm builds (so Beaglebone Black/White, etc).
> >>> If you start enabling all of the tracing options in the kernel (function
> >>> tracing, graphs, etc), you get an uncompressed kernel and BSS that will
> >>> use up the first ~16MiB of DDR.  We default to placing the DT at about
> >>> 15MiB into memory.  So the kernel runs, clears BSS and eats the DT.
> >>> System now hangs, and depending on debug options set you may or may not
> >>> see anything at all from the kernel.  U-Boot couldn't detect this
> >>> failure because we don't know how big the kernel BSS is, only how big
> >>> the zImage is (and where it is) and how big the fdt is and where it is.
> >>> No overlaps, go ahead and run.
> >>
> >> Thanks.
> >>
> >> The only issue I have with the RFC is that the +512 MiB value will only
> >> work with targets which have more than 512 MiB DDR, right? But since
> >> you're suggesting this should be set in configs, you are only suggesting
> >> +512 MiB, and any target could actually specify a lower value as long as
> >> it's greater than or eqal to 16 MiB. Correct?
> > 
> > fdt_high is only an upper bound on what we may relocate to (setting
> > aside the magic value of 0xffffffff which means no relocation).  I just
> > confirmed this too:
> > U-Boot# bdi
> > ...
> > boot_params = 0x80000100
> > DRAM bank   = 0x00000000
> > -> start    = 0x80000000
> > -> size     = 0x40000000
> > ...
> > U-Boot# setenv fdt_high 0xe0000000
> > ...
> > U-Boot# bootz $loadaddr - $fdtaddr
> > Kernel image @ 0x80200000 [ 0x000000 - 0x3e5068 ]
> > ## Flattened Device Tree blob at 80f80000
> >    Booting using the fdt blob at 0x80f80000
> >    Loading Device Tree to bf62a000, end bf630d9a ... OK
> > 
> > Of course, 0xbf62a000 is a bad address in that it won't be seen by the
> > kernel, but it was in the higher-than-we-have-memory-at limit I set.
> 
> I haven't really followed this thread since I just noticed it
> tangentially, but on Tegra...
> 
> Since commit 7f1b767aea94 "ARM: tegra: define CONFIG_SYS_BOOTMAPSZ"
> all Tegra boards define BOOTMAPSZ to influence where the DTB gets
> relocated. I wonder what's the benefit of using BOOTMAPSZ vs. defining
> a default value for $fdt_high?

I believe they both get the same thing done but SYS_BOOTMAPSZ also
influences relocation of ramdisks (and in these cases we disable ramdisk
relocation with initrd_high=0xffffffff).

> And couldn't you just move $fdt_addr_r high enough in RAM so even if
> the DTB wasn't relocated it'd never overlap BSS? Tegra's
> $kernel_addr_r and $fdt_addr_r are likely set up that way already,
> although I didn't check.

There's two problems here.  The first problem is that we have between
256MiB and 1GiB of DDR on the platform, but we could just design for the
smallest case.  The second problem is, what's big enough?  You've got
32MiB (tegra30) which I would hope is enough (and I suggested as much in
Dennis' thread) for kernel + BSS, but how big is a multi platform kernel
with a few big features going to get?  Or do we say that's an
unreasonable out of box use case?

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20131206/501509bc/attachment.pgp>

[U-Boot] [RFC] ARM: Start using fdt_high for relocation

[Stephen Warren]

On 12/06/2013 02:04 PM, Tom Rini wrote:
...
> There's two problems here.  The first problem is that we have
> between 256MiB and 1GiB of DDR on the platform, but we could just
> design for the smallest case.  The second problem is, what's big
> enough?  You've got 32MiB (tegra30) which I would hope is enough
> (and I suggested as much in Dennis' thread) for kernel + BSS, but
> how big is a multi platform kernel with a few big features going to
> get?  Or do we say that's an unreasonable out of box use case?

Is there any limit on .data/.bss size like there is .text (due to the
limited range of relative jump encoding), or would data accesses just
fall back to a relative load of the absolute address, and hence be
unbounded.

FWIW, I see the following sizes currently:

tegra_defconfig

.text	005862c4
.data	0005eb68
.bss	00055bf0

multi_v7_defconfig

.text	004a5560
.data	00092600
.bss	00046014

(I think multi_v7_defconfig doesn't yet have that many drivers
enabled, and when it does presumably they'd be modules)

... so BSS isn't terribly large at the moment.

[U-Boot] [RFC] ARM: Start using fdt_high for relocation

[Wolfgang Denk]

Dear Tom,

In message <20131206154850.GW420@bill-the-cat> you wrote:
> 
> I've been thinking.  We've had a thread on i.MX platforms about fdt
> being overwritten and needing to be moved to another address.  And I've
> also had an internal problem about fdt being overwritten.  So, how about
> as a rule of thumb we start setting fdt_high (in configs) to
> memory-start + 512MiB, as that's the lowmem limit we should always have
> available.  This will fix the problem of BSS overwriting the DT, which
> is the problem we won't catch in normal bootm/bootz usage.  Thoughts?

Maybe I'm missing something - but what about systems with less than
1 GB RAM?  Actually only a small percentage of devices we see here
have 512 MB or more...

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
In the bathtub of history the truth is harder to hold than the  soap,
and much more difficult to find ...     - Terry Pratchett, _Sourcery_