Boot interface for device trees on ARM

Boot interface for device trees on ARM

[Jeremy Kerr]

Hi all,

As we're getting closer to device tree support on ARM, I'd like to get some 
input on our proposed boot interface.

Basically, I'd like to define how we pass the device tree from the bootloader 
to the kernel.

My current method of doing this is through a new atag. It looks like this:

	/* flattened device tree blob pointer */
	#define ATAG_DEVTREE	0x5441000a

	struct tag_devtree {
		__u32 start;	/* physical start address */
		__u32 size;	/* size of dtb image in bytes */
	};

With ATAG_DEVTREE, we keep the existing boot interface the same (ie, machine 
number in r1, atags pointer r2).

Some notes about this scheme:

 + We can easily keep compatibility with the existing boot interface; both DT 
and non-DT kernels will be supported if a bootloader uses this.

 - It's a little more complex, as the bootloader has to initialise the atags 
structure.

 - If we end up in a situation where most machines are DT-enabled, then we'll 
be carrying a seldom-used structure (ie, a mostly-empty atags block) just to 
provide one pointer to the kernel.

 - We are now potentially carrying data in two different places - atags and 
the device tree. For example, the physical memory layout and kernel command 
line may be present in both.

Nicolas Pitre has suggested that we make it simpler, and specify the device 
tree blob directly instead (and remove the atags). In this case, r2 would 
point to the device tree blob, and r1 would be ignored.

Fortunately, both structures (atags list and device tree blob) begin with a 
magic number, so it is trivial to determine whether the pointer is to an atags 
list or a device tree blob.

Some notes about this scheme:

 - This would break compatibility with the existing boot interface: 
bootloaders that expect a DT kernel will not be able to boot a non-DT kernel. 
However, does this matter? Once the machine support (ie, bootloader and 
kernel) is done, we don't expect to have to enable both methods.

 + A simpler boot interface, so less to do (and get wrong) in the bootloader

 + We don't have two potential sources of boot information

Although I have been using the atag for a while, I have not pushed it to an 
upstream (either qemu or the kernel), as I would like to get a firm decision 
on the best method before making any commitment.

Comments and questions most welcome.

Cheers,


Jeremy

Boot interface for device trees on ARM

[Nicolas Pitre]

On Tue, 18 May 2010, Jeremy Kerr wrote:

> Hi all,
> 
> As we're getting closer to device tree support on ARM, I'd like to get some 
> input on our proposed boot interface.
> 
> Basically, I'd like to define how we pass the device tree from the bootloader 
> to the kernel.
> 
> My current method of doing this is through a new atag. It looks like this:
> 
> 	/* flattened device tree blob pointer */
> 	#define ATAG_DEVTREE	0x5441000a
> 
> 	struct tag_devtree {
> 		__u32 start;	/* physical start address */
> 		__u32 size;	/* size of dtb image in bytes */
> 	};
> 
> With ATAG_DEVTREE, we keep the existing boot interface the same (ie, machine 
> number in r1, atags pointer r2).
> 
> Some notes about this scheme:
> 
>  + We can easily keep compatibility with the existing boot interface; both DT 
> and non-DT kernels will be supported if a bootloader uses this.
> 
>  - It's a little more complex, as the bootloader has to initialise the atags 
> structure.
> 
>  - If we end up in a situation where most machines are DT-enabled, then we'll 
> be carrying a seldom-used structure (ie, a mostly-empty atags block) just to 
> provide one pointer to the kernel.
> 
>  - We are now potentially carrying data in two different places - atags and 
> the device tree. For example, the physical memory layout and kernel command 
> line may be present in both.
> 
> Nicolas Pitre has suggested that we make it simpler, and specify the device 
> tree blob directly instead (and remove the atags). In this case, r2 would 
> point to the device tree blob, and r1 would be ignored.

This is almost what I suggested, except for ignoring r1.  More on this 
below.

> Fortunately, both structures (atags list and device tree blob) begin with a 
> magic number, so it is trivial to determine whether the pointer is to an atags 
> list or a device tree blob.
> 
> Some notes about this scheme:
> 
>  - This would break compatibility with the existing boot interface: 
> bootloaders that expect a DT kernel will not be able to boot a non-DT kernel. 
> However, does this matter? Once the machine support (ie, bootloader and 
> kernel) is done, we don't expect to have to enable both methods.

I think that, for the moment, it is best if the bootloader on already 
existing subarchitectures where DT is introduced still preserve the 
already existing ability to boot using ATAGs.  This allows for the 
testing and validation of the DT concept against the legacy ATAG method 
more easily.

On new subarchitectures, it might make sense to go with DT from the 
start instead of creating setup code for every single machine.  In that 
case the bootloader for those machines would only need to care about DT 
and forget about ATAGs.

>  + A simpler boot interface, so less to do (and get wrong) in the bootloader
> 
>  + We don't have two potential sources of boot information

Those last two are IMHO the biggest reasons for not having both ATAGs 
and DT at the same time.  Otherwise the confusion about which one is 
authoritative, which one has precedence over the other, and/or whether 
the information should be obtained from one structure if it is missing 
from the other will simply bite us eventually for sure, as bootloader 
writers will get sloppy/lazy and have it wrong.  I strongly suggest that 
we should specify that the kernel must consider either ATAGs _or_ a 
device tree, and that the bootloader must pass only one of them.

[ I also insist on the ability for the DT info to be extractable and 
  updatable at the bootloader level, and not hardcoded into the
  bootloader itself. But that's another topic for discussion. ]

> Although I have been using the atag for a while, I have not pushed it to an 
> upstream (either qemu or the kernel), as I would like to get a firm decision 
> on the best method before making any commitment.
> 
> Comments and questions most welcome.

My suggestion is to have the DT support to be considered just as another 
machine _within_ each subarchitecture.  This means that a machine ID 
could be registered for DT on PXA, another for DT on OMAP, another for 
DT on Dove, etc.  This way, the DT support can be developed in parallel 
to the existing machine support code.  So if for example you want to 
test DT for Kirkwood then you may boot the kernel passing the ID for DT 
on Kirkwood into r1 and provide the DT corresponding to, say, a 
SheevaPlug.  Or you may decide to boot the same kernel binary and use 
the legacy SheevaPlug machine ID instead.  In theory both methods should 
be equivalent, baring any bugs.

Why one DT machine ID per subarchitecture?  Simply because a significant 
part of the DT handling code will have to be subarchitecture specific 
anyway.  The timer hardware, the GPIO configuration and muxing, SOC 
specific platform data handling, power management config, and many other 
things are simply too different from one SOC family to another and 
trying to have a single global DT support code to rule them all is 
insane.  At least with the concept of a "virtual" machine definition for 
DT per subarchitecture, the problem can easily be split and just fits 
naturally into the existing model on ARM.

This means that, over time, the machine ID registration would simply 
transition from a per machine thing to a per subarchitecture / SOC 
family thing.  And once the DT support is introduced for a given SOC 
family, then new machines using that SOC should be able to reuse 
the existing kernel binary for that SOC simply by providing a new DT 
data for the existing kernel to consume.

[ There is also the issue of being able to support multiple SOC families 
  within the same kernel binary, but that's something that could be done 
  with or without the device tree, and has issues of its own that the DT 
  cannot solve. Hence this is orthogonal to DT and a topic for yet another
  discussion. ]


Nicolas

Boot interface for device trees on ARM

[Jeremy Kerr]

Hi Nicolas,

> I think that, for the moment, it is best if the bootloader on already 
> existing subarchitectures where DT is introduced still preserve the 
> already existing ability to boot using ATAGs.  This allows for the 
> testing and validation of the DT concept against the legacy ATAG method 
> more easily.

Just to clarify - by "still preserve the existing ability to use ATAGs" you 
mean only for non-DT boot, right? This proposal still does not require 
ATAG_DEVTREE?

> Why one DT machine ID per subarchitecture?  Simply because a significant
> part of the DT handling code will have to be subarchitecture specific
> anyway.  The timer hardware, the GPIO configuration and muxing, SOC
> specific platform data handling, power management config, and many other
> things are simply too different from one SOC family to another and
> trying to have a single global DT support code to rule them all is
> insane.

The code for DT boot will be still subarch-specific, but I don't think we need 
IDs for that. There is enough information in the device tree to select the 
subarch-specific code to use for early init, without needing to parameterise 
every element of the machine. The machine-level "compatible" property allows 
us to do this.

Therefore, I don't think we need the machine ID at all: once the DT is 
available, we can use that for any machine-specific stuff. Even though we're 
not *configuring* it from the device tree, we can *select* it from there 
instead.

Cheers,


Jeremy

Boot interface for device trees on ARM

[David Gibson]

On Tue, May 18, 2010 at 01:24:43PM +0800, Jeremy Kerr wrote:
> Hi Nicolas,
> 
> > I think that, for the moment, it is best if the bootloader on already 
> > existing subarchitectures where DT is introduced still preserve the 
> > already existing ability to boot using ATAGs.  This allows for the 
> > testing and validation of the DT concept against the legacy ATAG method 
> > more easily.
> 
> Just to clarify - by "still preserve the existing ability to use ATAGs" you 
> mean only for non-DT boot, right? This proposal still does not require 
> ATAG_DEVTREE?
> 
> > Why one DT machine ID per subarchitecture?  Simply because a significant
> > part of the DT handling code will have to be subarchitecture specific
> > anyway.  The timer hardware, the GPIO configuration and muxing, SOC
> > specific platform data handling, power management config, and many other
> > things are simply too different from one SOC family to another and
> > trying to have a single global DT support code to rule them all is
> > insane.
> 
> The code for DT boot will be still subarch-specific, but I don't
> think we need IDs for that. There is enough information in the
> device tree to select the subarch-specific code to use for early
> init, without needing to parameterise every element of the
> machine. The machine-level "compatible" property allows us to do
> this.

That's right.  On PowerPC currently we don't have any real concept of
sub-architecture, but we do have platform level code to cover exactly
the sorts of messy things you mention, and it is selected on the basis
of the device tree's top-level compatible property.  Well, usually on
the compatible property, the platform probe routines can use other
information if, for example, firmware has screwed up the compatible
property.

One of the nice features that makes using the flat device tree quite
robust, is that even when firmware (or whoever) totally and utterly
stuffs up the device tree, there's nearly always *something* in there
that's sufficient to identify the board (by accident, if nothing
else).  A suitable platform probe routine can look for that, and claw
its way back to sanity from there (in the worst case, it could even
substitute an entire new corrected device tree, though I don't think
that's been necessary yet).

The only reason you'd need a subarchitecture number or equivalent is
if you need to do things differently in the very, very early asm boot
code.  We may need a minimal form of this on PowerPC if we ever
support multiple MMU families in the same kernel binary.

> Therefore, I don't think we need the machine ID at all: once the DT is 
> available, we can use that for any machine-specific stuff. Even though we're 
> not *configuring* it from the device tree, we can *select* it from there 
> instead.

Yes, absolutely.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Boot interface for device trees on ARM

[Nicolas Pitre]

On Tue, 18 May 2010, David Gibson wrote:

> On Tue, May 18, 2010 at 01:24:43PM +0800, Jeremy Kerr wrote:
> 
> > Nicolas Pitre wrote:
> > 
> > > Why one DT machine ID per subarchitecture?  Simply because a significant
> > > part of the DT handling code will have to be subarchitecture specific
> > > anyway.  The timer hardware, the GPIO configuration and muxing, SOC
> > > specific platform data handling, power management config, and many other
> > > things are simply too different from one SOC family to another and
> > > trying to have a single global DT support code to rule them all is
> > > insane.
> > 
> > The code for DT boot will be still subarch-specific, but I don't
> > think we need IDs for that. There is enough information in the
> > device tree to select the subarch-specific code to use for early
> > init, without needing to parameterise every element of the
> > machine. The machine-level "compatible" property allows us to do
> > this.
> 
> That's right.  On PowerPC currently we don't have any real concept of
> sub-architecture, but we do have platform level code to cover exactly
> the sorts of messy things you mention, and it is selected on the basis
> of the device tree's top-level compatible property.

Agreed.  However this is not PPC we're discussing here.  This is about 
ARM which has been structuring its wildly different subarchitectures 
around another model for over 15 years.

> The only reason you'd need a subarchitecture number or equivalent is
> if you need to do things differently in the very, very early asm boot
> code.  We may need a minimal form of this on PowerPC if we ever
> support multiple MMU families in the same kernel binary.

Exact.  For example, on ARM the machine ID is also used to figure out 
the MMU mapping needed to be able to simply be able to debug the very 
early assembly boot stage when there isn't even a stack available. While 
this info is stored in the machine record, it is actually 
subarchitecture specific and already half-digested for easy usage by 
that initial MMU setup.  I just don't want to imagine what the 
equivalent functionality with DT would look like.

> > Therefore, I don't think we need the machine ID at all: once the DT is 
> > available, we can use that for any machine-specific stuff. Even though we're 
> > not *configuring* it from the device tree, we can *select* it from there 
> > instead.
> 
> Yes, absolutely.

Please see above why I disagree.


Nicolas

Boot interface for device trees on ARM

[Jason McMullan]

With respect to machine IDs, why not have our cake and eat it too?

(NOTE: This proposal assumes that the boot code has already
       determined whether ATAGs or a DT has been passed to
       the kernel, and that ATAG and DT machine IDs are in
       the same namespace)

1) We already have machine names in the mach-types file, and we
can propose that those are used as the top-level 'compatible'
tags for the DTs.

...
   compatible = "openrd\000kirkwood\000armv6\000arm"
...


2) We can trivially generate a name-to-machine ID hash table from
the mach-types file (CRC32 of the name, or some other hash)

3) Allocate per-SoC machine IDs for DT names, and a
global generic machine ID as a catch-all:

dt_arm        MACH_DT_ARM       DT_ARM       xxxx
dt_armv6      MACH_DT_ARMV6     DT_ARMV6     xxxx
dt_kirkwood   MACH_DT_KIRKWOOD  DT_KIRKWOOD  xxxx
dt_at91       MACH_DT_AT91      DT_AT91      xxxx

4) Each machine type that cares about DTs could register a
callback function, that would be placed in a linker-assembled
table mapping machine IDs (or the name hashes) to callbacks:

void at91_setup_from_dt(struct device_tree_node *dt)
{
  ...
}

DECLARE_DT_CALLBACK(at91, at91_setup_from_dt)

5) The 'arm' callback would handle common-code devices,
such as NOR flash, memory areas, etc. and would be called
after the machine-specific callbacks.

6) So, for the openrd case above, the DT parser callback
would call on, for each object in the DT:

  a) The MACH_DT_OPENRD callback, if it exists. If not, or
     if it returns -ENOTSUP:
  b) The MACH_DT_KIRKWOOD callback, if it exists. If not, or
     if it returns -ENOTSUP:
  c) The MACH_DT_ARMV6 callback, if it exists. If not, or
     if it returns -ENOTSUP:
  d) The MACH_DT_ARM callback, if it exists. If not, or
     if it returns -ENOTSUP:

-- 
Jason S. McMullan
Netronome Systems, Inc.

Boot interface for device trees on ARM

[David Gibson]

On Tue, May 18, 2010 at 10:06:38AM -0400, Jason McMullan wrote:
> With respect to machine IDs, why not have our cake and eat it too?
> 
> (NOTE: This proposal assumes that the boot code has already
>        determined whether ATAGs or a DT has been passed to
>        the kernel, and that ATAG and DT machine IDs are in
>        the same namespace)
> 
> 1) We already have machine names in the mach-types file, and we
> can propose that those are used as the top-level 'compatible'
> tags for the DTs.
> 
> ...
>    compatible = "openrd\000kirkwood\000armv6\000arm"
> ...

That sounds like a good idea.  Except that you should probably prefix
the ARM machine name with something (e.g. "linux,arm-machine-XXXX"),
both to meet the normal OF "vendor,type" convention for compatible
values, and to make sure to avoid any conflicts wth things of other
architectures.

Oh, also, dtc supports nicer syntax for NULL separated stringlists
like that, you can go:
	compatible = "openrd", "kirkwood", "armv6", "arm";

> 2) We can trivially generate a name-to-machine ID hash table from
> the mach-types file (CRC32 of the name, or some other hash)

gperf...

The rest I'm not so sure about, I don't know arch/arm well enough.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Boot interface for device trees on ARM

[David Gibson]

On Tue, May 18, 2010 at 08:24:06AM -0400, Nicolas Pitre wrote:
> On Tue, 18 May 2010, David Gibson wrote:
> > On Tue, May 18, 2010 at 01:24:43PM +0800, Jeremy Kerr wrote:
> > > Nicolas Pitre wrote:
[snip]
> > The only reason you'd need a subarchitecture number or equivalent is
> > if you need to do things differently in the very, very early asm boot
> > code.  We may need a minimal form of this on PowerPC if we ever
> > support multiple MMU families in the same kernel binary.
> 
> Exact.  For example, on ARM the machine ID is also used to figure out 
> the MMU mapping needed to be able to simply be able to debug the very 
> early assembly boot stage when there isn't even a stack available. While 
> this info is stored in the machine record, it is actually 
> subarchitecture specific and already half-digested for easy usage by 
> that initial MMU setup.  I just don't want to imagine what the 
> equivalent functionality with DT would look like.

Well, it wouldn't be *that* bad - you'd need a minimal asm-only tree
walker to find and look up the compatible property.  Quite possible
but, yes, fairly awkward.

Yeah, if you have to make really early MMU decisions based on
subarchitecture it probbaly makes sense to have a simple ID for
those.  So one thing to consider here is that we've contemplated
adding an "MMU family" ID to the device tree blob header for this
purpose on PowerPC.  If we did such a thing, it could also be used on
ARM for a similar purpose.  Then all the relevant machine information
is in one block, and it's just a simple fixed dereference to extract
the MMU type from the early asm.

I'd have to talk to Ben et al, but I think we'd be happy enough to
create a new dtb v18 which included that field, as well as a couple of
other small improvements to the header and blob internals.  ARM could
standardize on that as the minimum acceptable dtb version.

> > > Therefore, I don't think we need the machine ID at all: once the DT is 
> > > available, we can use that for any machine-specific stuff. Even though we're 
> > > not *configuring* it from the device tree, we can *select* it from there 
> > > instead.
> > 
> > Yes, absolutely.
> 
> Please see above why I disagree.

Well, I see your point for the early MMU stuff.  However, I'm betting
there's other, later platform related setup which could just as easily
be selected bsaed on the device tree, rather than on a magic ID.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Boot interface for device trees on ARM

[Nicolas Pitre]

On Wed, 19 May 2010, David Gibson wrote:

> On Tue, May 18, 2010 at 08:24:06AM -0400, Nicolas Pitre wrote:
> > On Tue, 18 May 2010, David Gibson wrote:
> > > On Tue, May 18, 2010 at 01:24:43PM +0800, Jeremy Kerr wrote:
> > > > Nicolas Pitre wrote:
> [snip]
> > > The only reason you'd need a subarchitecture number or equivalent is
> > > if you need to do things differently in the very, very early asm boot
> > > code.  We may need a minimal form of this on PowerPC if we ever
> > > support multiple MMU families in the same kernel binary.
> > 
> > Exact.  For example, on ARM the machine ID is also used to figure out 
> > the MMU mapping needed to be able to simply be able to debug the very 
> > early assembly boot stage when there isn't even a stack available. While 
> > this info is stored in the machine record, it is actually 
> > subarchitecture specific and already half-digested for easy usage by 
> > that initial MMU setup.  I just don't want to imagine what the 
> > equivalent functionality with DT would look like.
> 
> Well, it wouldn't be *that* bad - you'd need a minimal asm-only tree
> walker to find and look up the compatible property.  Quite possible
> but, yes, fairly awkward.
> 
> Yeah, if you have to make really early MMU decisions based on
> subarchitecture it probbaly makes sense to have a simple ID for
> those.  So one thing to consider here is that we've contemplated
> adding an "MMU family" ID to the device tree blob header for this
> purpose on PowerPC.  If we did such a thing, it could also be used on
> ARM for a similar purpose.  Then all the relevant machine information
> is in one block, and it's just a simple fixed dereference to extract
> the MMU type from the early asm.
> 
> I'd have to talk to Ben et al, but I think we'd be happy enough to
> create a new dtb v18 which included that field, as well as a couple of
> other small improvements to the header and blob internals.  ARM could
> standardize on that as the minimum acceptable dtb version.

I still don't get it.  Why on earth would you insist on replacing the 
already existing ID value that is _already_ passed to the kernel by all 
ARM bootloaders with something evidently far more complex?

Please let's step back a bit.

DT is a mean to help making the code more generic and reduce the amount 
of machine specific code, not to increase it in other ways.  While the 
ID passed in r1 makes perfect sense to me as it is simple and straight 
forward, using the DT for this provides absolutely no advantage.

Yet we still need the ID into r1 to distinguish between a DT boot from a 
legacy boot.  So it is much simpler to always have the ID to select 
whether a specific machine is being booted, or if the DT should later be 
used to configure the kernel for a specific machine.  No special code 
path is needed, etc.

Moving the ID into the DT just for the sake of it is not useful.


Nicolas

Boot interface for device trees on ARM

[David Gibson]

On Tue, May 18, 2010 at 09:28:38PM -0400, Nicolas Pitre wrote:
> On Wed, 19 May 2010, David Gibson wrote:
> 
> > On Tue, May 18, 2010 at 08:24:06AM -0400, Nicolas Pitre wrote:
> > > On Tue, 18 May 2010, David Gibson wrote:
> > > > On Tue, May 18, 2010 at 01:24:43PM +0800, Jeremy Kerr wrote:
> > > > > Nicolas Pitre wrote:
> > [snip]
> > > > The only reason you'd need a subarchitecture number or equivalent is
> > > > if you need to do things differently in the very, very early asm boot
> > > > code.  We may need a minimal form of this on PowerPC if we ever
> > > > support multiple MMU families in the same kernel binary.
> > > 
> > > Exact.  For example, on ARM the machine ID is also used to figure out 
> > > the MMU mapping needed to be able to simply be able to debug the very 
> > > early assembly boot stage when there isn't even a stack available. While 
> > > this info is stored in the machine record, it is actually 
> > > subarchitecture specific and already half-digested for easy usage by 
> > > that initial MMU setup.  I just don't want to imagine what the 
> > > equivalent functionality with DT would look like.
> > 
> > Well, it wouldn't be *that* bad - you'd need a minimal asm-only tree
> > walker to find and look up the compatible property.  Quite possible
> > but, yes, fairly awkward.
> > 
> > Yeah, if you have to make really early MMU decisions based on
> > subarchitecture it probbaly makes sense to have a simple ID for
> > those.  So one thing to consider here is that we've contemplated
> > adding an "MMU family" ID to the device tree blob header for this
> > purpose on PowerPC.  If we did such a thing, it could also be used on
> > ARM for a similar purpose.  Then all the relevant machine information
> > is in one block, and it's just a simple fixed dereference to extract
> > the MMU type from the early asm.
> > 
> > I'd have to talk to Ben et al, but I think we'd be happy enough to
> > create a new dtb v18 which included that field, as well as a couple of
> > other small improvements to the header and blob internals.  ARM could
> > standardize on that as the minimum acceptable dtb version.
> 
> I still don't get it.  Why on earth would you insist on replacing the 
> already existing ID value that is _already_ passed to the kernel by all 
> ARM bootloaders with something evidently far more complex?

Um.. no.  It's not more complex at all.  We're talking about this ID
in the dt blob's header, not in the device tree itself.  As I said,
it's one dereference to get to the ID, from the DT address which is
being passed in.  i.e. *one* more instruction than having it passed in
register.

There are two advantages, both small, but contrary to your assertion
it's not significantly more complex, so they don't need to be big.
	1) It means the DT blob contains *all* the information you
need about the machine.  If you have multi-stage bootloaders, you can
copy that blob and know you've got everything.
	2) It means we could have information with a similar purpose
in the same place for both PowerPC and ARM (and maybe others in
future).

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Boot interface for device trees on ARM

[Grant Likely]

On Wed, May 19, 2010 at 12:50 AM, David Gibson
<david@gibson.dropbear.id.au> wrote:
> On Tue, May 18, 2010 at 09:28:38PM -0400, Nicolas Pitre wrote:
>> On Wed, 19 May 2010, David Gibson wrote:
>>
>> > On Tue, May 18, 2010 at 08:24:06AM -0400, Nicolas Pitre wrote:
>> > > On Tue, 18 May 2010, David Gibson wrote:
>> > > > On Tue, May 18, 2010 at 01:24:43PM +0800, Jeremy Kerr wrote:
>> > > > > Nicolas Pitre wrote:
>> > [snip]
>> > > > The only reason you'd need a subarchitecture number or equivalent is
>> > > > if you need to do things differently in the very, very early asm boot
>> > > > code. ?We may need a minimal form of this on PowerPC if we ever
>> > > > support multiple MMU families in the same kernel binary.
>> > >
>> > > Exact. ?For example, on ARM the machine ID is also used to figure out
>> > > the MMU mapping needed to be able to simply be able to debug the very
>> > > early assembly boot stage when there isn't even a stack available. While
>> > > this info is stored in the machine record, it is actually
>> > > subarchitecture specific and already half-digested for easy usage by
>> > > that initial MMU setup. ?I just don't want to imagine what the
>> > > equivalent functionality with DT would look like.
>> >
>> > Well, it wouldn't be *that* bad - you'd need a minimal asm-only tree
>> > walker to find and look up the compatible property. ?Quite possible
>> > but, yes, fairly awkward.
>> >
>> > Yeah, if you have to make really early MMU decisions based on
>> > subarchitecture it probbaly makes sense to have a simple ID for
>> > those. ?So one thing to consider here is that we've contemplated
>> > adding an "MMU family" ID to the device tree blob header for this
>> > purpose on PowerPC. ?If we did such a thing, it could also be used on
>> > ARM for a similar purpose. ?Then all the relevant machine information
>> > is in one block, and it's just a simple fixed dereference to extract
>> > the MMU type from the early asm.
>> >
>> > I'd have to talk to Ben et al, but I think we'd be happy enough to
>> > create a new dtb v18 which included that field, as well as a couple of
>> > other small improvements to the header and blob internals. ?ARM could
>> > standardize on that as the minimum acceptable dtb version.
>>
>> I still don't get it. ?Why on earth would you insist on replacing the
>> already existing ID value that is _already_ passed to the kernel by all
>> ARM bootloaders with something evidently far more complex?
>
> Um.. no. ?It's not more complex at all. ?We're talking about this ID
> in the dt blob's header, not in the device tree itself. ?As I said,
> it's one dereference to get to the ID, from the DT address which is
> being passed in. ?i.e. *one* more instruction than having it passed in
> register.
>
> There are two advantages, both small, but contrary to your assertion
> it's not significantly more complex, so they don't need to be big.
> ? ? ? ?1) It means the DT blob contains *all* the information you
> need about the machine. ?If you have multi-stage bootloaders, you can
> copy that blob and know you've got everything.
> ? ? ? ?2) It means we could have information with a similar purpose
> in the same place for both PowerPC and ARM (and maybe others in
> future).

Regardless, I think Nicolas is right.  While there is value in common
methods between architectures, I don't think this case warrants it
when the existing method works fine and changing it is
non-backwards-compatible.

g.

Boot interface for device trees on ARM

[Jamie Lokier]

David Gibson wrote:
> On Tue, May 18, 2010 at 08:24:06AM -0400, Nicolas Pitre wrote:
> > On Tue, 18 May 2010, David Gibson wrote:
> > > On Tue, May 18, 2010 at 01:24:43PM +0800, Jeremy Kerr wrote:
> > > > Nicolas Pitre wrote:
> [snip]
> > > The only reason you'd need a subarchitecture number or equivalent is
> > > if you need to do things differently in the very, very early asm boot
> > > code.  We may need a minimal form of this on PowerPC if we ever
> > > support multiple MMU families in the same kernel binary.
> > 
> > Exact.  For example, on ARM the machine ID is also used to figure out 
> > the MMU mapping needed to be able to simply be able to debug the very 
> > early assembly boot stage when there isn't even a stack available. While 
> > this info is stored in the machine record, it is actually 
> > subarchitecture specific and already half-digested for easy usage by 
> > that initial MMU setup.  I just don't want to imagine what the 
> > equivalent functionality with DT would look like.
> 
> Well, it wouldn't be *that* bad - you'd need a minimal asm-only tree
> walker to find and look up the compatible property.  Quite possible
> but, yes, fairly awkward.

I'm not entirely clear, is the DT intended to replace the command line
for saying things like "console=XXX"?

Real example: I have a device where the bootloader decides which
serial port will be the diagnostic boot console, if any, based on a
specially wired serial cable detected at boot time, and it passes the
decision to the kernel.

In that case, does the console selection need to be easily accessible
to that early asm code, for early printks?

-- Jamie

Boot interface for device trees on ARM

[David Gibson]

On Wed, May 19, 2010 at 02:41:18AM +0100, Jamie Lokier wrote:
> David Gibson wrote:
> > On Tue, May 18, 2010 at 08:24:06AM -0400, Nicolas Pitre wrote:
> > > On Tue, 18 May 2010, David Gibson wrote:
> > > > On Tue, May 18, 2010 at 01:24:43PM +0800, Jeremy Kerr wrote:
> > > > > Nicolas Pitre wrote:
> > [snip]
> > > > The only reason you'd need a subarchitecture number or equivalent is
> > > > if you need to do things differently in the very, very early asm boot
> > > > code.  We may need a minimal form of this on PowerPC if we ever
> > > > support multiple MMU families in the same kernel binary.
> > > 
> > > Exact.  For example, on ARM the machine ID is also used to figure out 
> > > the MMU mapping needed to be able to simply be able to debug the very 
> > > early assembly boot stage when there isn't even a stack available. While 
> > > this info is stored in the machine record, it is actually 
> > > subarchitecture specific and already half-digested for easy usage by 
> > > that initial MMU setup.  I just don't want to imagine what the 
> > > equivalent functionality with DT would look like.
> > 
> > Well, it wouldn't be *that* bad - you'd need a minimal asm-only tree
> > walker to find and look up the compatible property.  Quite possible
> > but, yes, fairly awkward.
> 
> I'm not entirely clear, is the DT intended to replace the command line
> for saying things like "console=XXX"?

Um.. well, in a DT based system, the bootloader typically passes the
command line to the kernel via the device tree (it goes in the
bootargs property of /chosen).

However, at least on PowerPC the default console is also specified
directly in the DT (via the linux,stdout-path property in /chosen), so
putting console= on the command line isn't usually necessary (if given
it will override the linux,stdout-path property for most purposes).

> Real example: I have a device where the bootloader decides which
> serial port will be the diagnostic boot console, if any, based on a
> specially wired serial cable detected at boot time, and it passes the
> decision to the kernel.

Setting linux,stdout-path in the device tree sounds like the right
thing to do here.

> In that case, does the console selection need to be easily accessible
> to that early asm code, for early printks?

Well, typically I wouldn't expect printk() to occur until you're in C
code at least, but I don't know about the ARM code.  Once you're in C
code it shouldn't be significantly harder to consult the device tree
than to parse the command line.

On PowerPC we do have a super-early debug printing mechanism, but when
that's enabled all the specifics are compile-time selected (it's
basically only recommended for use during bringup).

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Boot interface for device trees on ARM

[Grant Likely]

On Tue, May 18, 2010 at 7:41 PM, Jamie Lokier <jamie@shareable.org> wrote:
> David Gibson wrote:
>> On Tue, May 18, 2010 at 08:24:06AM -0400, Nicolas Pitre wrote:
>> > On Tue, 18 May 2010, David Gibson wrote:
>> > > On Tue, May 18, 2010 at 01:24:43PM +0800, Jeremy Kerr wrote:
>> > > > Nicolas Pitre wrote:
>> [snip]
>> > > The only reason you'd need a subarchitecture number or equivalent is
>> > > if you need to do things differently in the very, very early asm boot
>> > > code. ?We may need a minimal form of this on PowerPC if we ever
>> > > support multiple MMU families in the same kernel binary.
>> >
>> > Exact. ?For example, on ARM the machine ID is also used to figure out
>> > the MMU mapping needed to be able to simply be able to debug the very
>> > early assembly boot stage when there isn't even a stack available. While
>> > this info is stored in the machine record, it is actually
>> > subarchitecture specific and already half-digested for easy usage by
>> > that initial MMU setup. ?I just don't want to imagine what the
>> > equivalent functionality with DT would look like.
>>
>> Well, it wouldn't be *that* bad - you'd need a minimal asm-only tree
>> walker to find and look up the compatible property. ?Quite possible
>> but, yes, fairly awkward.
>
> I'm not entirely clear, is the DT intended to replace the command line
> for saying things like "console=XXX"?

It doesn't replace the kernel parameters line, but the kernel
parameters line can be passed to the kernel via a property in the
/chosen node.  Firmware is allowed to modify the DT (ie. to change the
kernel parameters) before passing it to the kernel.

Cheers,
g.

-- 
Grant Likely, B.Sc., P.Eng.
Secret Lab Technologies Ltd.

Boot interface for device trees on ARM

[Mitch Bradley]

Nicolas Pitre wrote:
> ...
> Exact.  For example, on ARM the machine ID is also used to figure out 
> the MMU mapping needed to be able to simply be able to debug the very 
> early assembly boot stage when there isn't even a stack available. While 
> this info is stored in the machine record, it is actually 
> subarchitecture specific and already half-digested for easy usage by 
> that initial MMU setup.  I just don't want to imagine what the 
> equivalent functionality with DT would look like.
>
>
>   

Under what circumstances would the Linux startup code not have a stack - 
or not be able to set up one easily?

Boot interface for device trees on ARM

[Jeremy Kerr]

Nicolas,

> Exact.  For example, on ARM the machine ID is also used to figure out
> the MMU mapping needed to be able to simply be able to debug the very
> early assembly boot stage when there isn't even a stack available.

I get the impression that this is the only thing that we need the io_pg_offset 
for - setting up the debug page mapping, for very early printk output.

If this is the case, I would much rather have it as a compile-time constant 
when doing this early debug; this is what I have been doing for DT boot in my 
tree (however, I would like to move it to something more configurable).

In the cases where we need to debug the pre-C stuff (ie, machine bringup), I 
think we can live with a compile-time constant, rather than having to define a 
special 'exception' in the DT boot interface to handle this.

Or is there another early use of the mdesc that I'm not aware of?

Cheers,


Jeremy

Boot interface for device trees on ARM

[Nicolas Pitre]

On Tue, 18 May 2010, Jeremy Kerr wrote:

> Hi Nicolas,
> 
> > I think that, for the moment, it is best if the bootloader on already 
> > existing subarchitectures where DT is introduced still preserve the 
> > already existing ability to boot using ATAGs.  This allows for the 
> > testing and validation of the DT concept against the legacy ATAG method 
> > more easily.
> 
> Just to clarify - by "still preserve the existing ability to use ATAGs" you 
> mean only for non-DT boot, right?

Exact.  Once a particular SOC family has no non-DT support anymore (due 
to being entirely new, or because people get really enthusiastic and 
fade out legacy machine specific init code completely) then and only 
then it might be logical to remove ATAG from the concerned bootloaders.


> This proposal still does not require ATAG_DEVTREE?

No.

> > Why one DT machine ID per subarchitecture?  Simply because a significant
> > part of the DT handling code will have to be subarchitecture specific
> > anyway.  The timer hardware, the GPIO configuration and muxing, SOC
> > specific platform data handling, power management config, and many other
> > things are simply too different from one SOC family to another and
> > trying to have a single global DT support code to rule them all is
> > insane.
> 
> The code for DT boot will be still subarch-specific, but I don't think we need 
> IDs for that. There is enough information in the device tree to select the 
> subarch-specific code to use for early init, without needing to parameterise 
> every element of the machine.

You can't do that without shifting the World Order on ARM.  And _that_ 
is going to create resistance for merging DT support.

> The machine-level "compatible" property allows 
> us to do this.
> 
> Therefore, I don't think we need the machine ID at all: once the DT is 
> available, we can use that for any machine-specific stuff. Even though we're 
> not *configuring* it from the device tree, we can *select* it from there 
> instead.

This is just added complexity, especially in the early boot code which 
for now simply has to compare r1 against well known constants.  We have 
that ID passed by the bootloader already.  I don't see the advantage of 
ignoring it and relying solely on DT for that.

Furthermore, if you're interested only in, say, OMAP5 (let's say you're 
doing U-Boot for it) then you need no bother about anything else than 
passing the ID number for OMAP5_DEVICE_TREE into r1 and manage 
*everything else* in the device tree.


Nicolas

Boot interface for device trees on ARM

[Grant Likely]

On Tue, May 18, 2010 at 5:57 AM, Nicolas Pitre <nico@fluxnic.net> wrote:
> On Tue, 18 May 2010, Jeremy Kerr wrote:
>
>> Hi Nicolas,
>>
>> > I think that, for the moment, it is best if the bootloader on already
>> > existing subarchitectures where DT is introduced still preserve the
>> > already existing ability to boot using ATAGs. ?This allows for the
>> > testing and validation of the DT concept against the legacy ATAG method
>> > more easily.
>>
>> Just to clarify - by "still preserve the existing ability to use ATAGs" you
>> mean only for non-DT boot, right?
>
> Exact. ?Once a particular SOC family has no non-DT support anymore (due
> to being entirely new, or because people get really enthusiastic and
> fade out legacy machine specific init code completely) then and only
> then it might be logical to remove ATAG from the concerned bootloaders.
>
>
>> This proposal still does not require ATAG_DEVTREE?
>
> No.

Hmmm...  I misunderstood then.  I don't agree that this is the best way forward.

Doing it this way means a non-compatible break in the interface.  It
means that the bootloader needs to know what interface the kernel is
expecting for boot; information that is not readily available from the
image type.  The user then needs to tell the boot loader which
interface to use rather than a backwards compatible addition of a blob
of data.

You mention below "shifting the World Order on ARM" and it creating
resistance for merging DT support.  Isn't this much the same thing as
it creates a non backwards compatible change in the way bootloaders
pass data to the kernel.  The cutover in powerpc from the old
interface to the new caused no end of confusion and people who could
no longer get their systems to boot.  On PowerPC is was necessary
because the old method was completely broken, but ATAGs are clean,
simple and well implemented.

It also means teaching every boot loader two separate methods for
booting and exposing those differences to the user.

>> The code for DT boot will be still subarch-specific, but I don't think we need
>> IDs for that. There is enough information in the device tree to select the
>> subarch-specific code to use for early init, without needing to parameterise
>> every element of the machine.
>
> You can't do that without shifting the World Order on ARM. ?And _that_
> is going to create resistance for merging DT support.
>
>> The machine-level "compatible" property allows
>> us to do this.
>>
>> Therefore, I don't think we need the machine ID at all: once the DT is
>> available, we can use that for any machine-specific stuff. Even though we're
>> not *configuring* it from the device tree, we can *select* it from there
>> instead.
>
> This is just added complexity, especially in the early boot code which
> for now simply has to compare r1 against well known constants. ?We have
> that ID passed by the bootloader already. ?I don't see the advantage of
> ignoring it and relying solely on DT for that.
>
> Furthermore, if you're interested only in, say, OMAP5 (let's say you're
> doing U-Boot for it) then you need no bother about anything else than
> passing the ID number for OMAP5_DEVICE_TREE into r1 and manage
> *everything else* in the device tree.
>
>
> Nicolas
> _______________________________________________
> devicetree-discuss mailing list
> devicetree-discuss at lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/devicetree-discuss
>



-- 
Grant Likely, B.Sc., P.Eng.
Secret Lab Technologies Ltd.

Boot interface for device trees on ARM

[Jamie Lokier]

Grant Likely wrote:
> On Tue, May 18, 2010 at 5:57 AM, Nicolas Pitre <nico@fluxnic.net> wrote:
> > On Tue, 18 May 2010, Jeremy Kerr wrote:
> >
> >> Hi Nicolas,
> >>
> >> > I think that, for the moment, it is best if the bootloader on already
> >> > existing subarchitectures where DT is introduced still preserve the
> >> > already existing ability to boot using ATAGs. ?This allows for the
> >> > testing and validation of the DT concept against the legacy ATAG method
> >> > more easily.
> >>
> >> Just to clarify - by "still preserve the existing ability to use ATAGs" you
> >> mean only for non-DT boot, right?
> >
> > Exact. ?Once a particular SOC family has no non-DT support anymore (due
> > to being entirely new, or because people get really enthusiastic and
> > fade out legacy machine specific init code completely) then and only
> > then it might be logical to remove ATAG from the concerned bootloaders.
> >
> >
> >> This proposal still does not require ATAG_DEVTREE?
> >
> > No.
> 
> Hmmm...  I misunderstood then.  I don't agree that this is the best way forward.
> 
> Doing it this way means a non-compatible break in the interface.  It
> means that the bootloader needs to know what interface the kernel is
> expecting for boot; information that is not readily available from the
> image type.  The user then needs to tell the boot loader which
> interface to use rather than a backwards compatible addition of a blob
> of data.

Also the other way around: Sometimes you want to install the same
kernel on systems with old and new bootloaders, without touching the
bootloaders (due to that not being powerfail-safe, say).  The kernel
needs to know if it's passed a DT from a newer bootloader or not.

And sometimes you'd like to install a newer, tested kernel (that uses
DTs) on systems with old bootloaders.

> You mention below "shifting the World Order on ARM" and it creating
> resistance for merging DT support.  Isn't this much the same thing as
> it creates a non backwards compatible change in the way bootloaders
> pass data to the kernel.  The cutover in powerpc from the old
> interface to the new caused no end of confusion and people who could
> no longer get their systems to boot.  On PowerPC is was necessary
> because the old method was completely broken, but ATAGs are clean,
> simple and well implemented.

You can't always update the boot loader.  Sometimes you're stuck with
what's there for the life of a device.  Either it's ROM, or it's too
risky to modify in place.

> It also means teaching every boot loader two separate methods for
> booting and exposing those differences to the user.

Embedded devices usually don't have any way for the "user" to choose
from a boot menu ;-)

ATAG_DEVTREE sounds good to me for mix'n'match systems.

New systems that always use DTs could use _just_ ATAG_DEVTREE, to
avoid questions of conflicting info.  They could also settle on a
fixed R1 value meaning "devtree platform".

Or if a fixed "devtree platform" R1 is used, R2 could point directly
at the DT, no atag list in that specific case.

-- Jamie

Boot interface for device trees on ARM

[Grant Likely]

Hi Jamie,

On Wed, May 19, 2010 at 10:45 AM, Jamie Lokier <jamie@shareable.org> wrote:
> Grant Likely wrote:
>> Doing it this way means a non-compatible break in the interface. ?It
>> means that the bootloader needs to know what interface the kernel is
>> expecting for boot; information that is not readily available from the
>> image type. ?The user then needs to tell the boot loader which
>> interface to use rather than a backwards compatible addition of a blob
>> of data.
>
> Also the other way around: Sometimes you want to install the same
> kernel on systems with old and new bootloaders, without touching the
> bootloaders (due to that not being powerfail-safe, say). ?The kernel
> needs to know if it's passed a DT from a newer bootloader or not.
>
> And sometimes you'd like to install a newer, tested kernel (that uses
> DTs) on systems with old bootloaders.

Which will be an issue regardless of which approach is chosen.  Either
way, this case does require the use of a boot shim or wrapper.

>> You mention below "shifting the World Order on ARM" and it creating
>> resistance for merging DT support. ?Isn't this much the same thing as
>> it creates a non backwards compatible change in the way bootloaders
>> pass data to the kernel. ?The cutover in powerpc from the old
>> interface to the new caused no end of confusion and people who could
>> no longer get their systems to boot. ?On PowerPC is was necessary
>> because the old method was completely broken, but ATAGs are clean,
>> simple and well implemented.
>
> You can't always update the boot loader. ?Sometimes you're stuck with
> what's there for the life of a device. ?Either it's ROM, or it's too
> risky to modify in place.

Yes.  Though this is a bit of a side issue when talking about the boot
interface.  We're still going to have cases where the bootloader
passes bad data and we may need a shim or wrapper (not part of
firmware) to adapt or fix it up.  Just like with booting new kernels
on old (no-dt-support) firmware.

There is a larger issue w.r.t. what the overall boot architecture is,
and what recommendations distributions (like Ubuntu) should make to
device manufactures to ensure firmware is reliable and can boot a
distribution OS image.  That's a topic for a different email.

>> It also means teaching every boot loader two separate methods for
>> booting and exposing those differences to the user.
>
> Embedded devices usually don't have any way for the "user" to choose
> from a boot menu ;-)
>
> ATAG_DEVTREE sounds good to me for mix'n'match systems.
>
> New systems that always use DTs could use _just_ ATAG_DEVTREE, to
> avoid questions of conflicting info.

Yes, this is trivial to do.

> ?They could also settle on a
> fixed R1 value meaning "devtree platform".
>
> Or if a fixed "devtree platform" R1 is used, R2 could point directly
> at the DT, no atag list in that specific case.

My sticking point is I want there to be exactly one method of passing
the device tree blob.  I don't see any advantage in having more than
one mechanism for passing the dtb pointer.  Whether it be ATAG or bare
DT pointer, we should be able to choose a single method now that is
suitable for the foreseeable future.

Thanks,
g.

-- 
Grant Likely, B.Sc., P.Eng.
Secret Lab Technologies Ltd.

Boot interface for device trees on ARM

[M. Warner Losh]

In message: <AANLkTinzTKkSIwGemGJ-gI0DSD-ANOSePkdwFZhMwFnM@mail.gmail.com>
            Grant Likely <grant.likely@secretlab.ca> writes:
: > ?They could also settle on a
: > fixed R1 value meaning "devtree platform".
: >
: > Or if a fixed "devtree platform" R1 is used, R2 could point directly
: > at the DT, no atag list in that specific case.
: 
: My sticking point is I want there to be exactly one method of passing
: the device tree blob.  I don't see any advantage in having more than
: one mechanism for passing the dtb pointer.  Whether it be ATAG or bare
: DT pointer, we should be able to choose a single method now that is
: suitable for the foreseeable future.

The gain from having a shortcut is small (maybe a dozen instructions
at most saved).  I'd rather see an interface compatible with old ATAG
boot loaders that would allow for easy selection, inside the kernel,
of which information to believe.

The biggest concern that I have with this is to make sure that it is
unambiguous what the kernel should do with the data.  Eg, the DT takes
precedence over the rest of the ATAG list, but the two are required to
be 100% consistent (with all the mappings between ATAG and DT well
defined).  The only time things should vary is when DT is more
expressive than ATAG, and even then kernels relying on the old
interface should do a sensible thing...

Warner

Boot interface for device trees on ARM

[Grant Likely]

On Mon, May 17, 2010 at 10:34 PM, Nicolas Pitre <nico@fluxnic.net> wrote:
> On Tue, 18 May 2010, Jeremy Kerr wrote:
>> Some notes about this scheme:
>>
>> ?- This would break compatibility with the existing boot interface:
>> bootloaders that expect a DT kernel will not be able to boot a non-DT kernel.
>> However, does this matter? Once the machine support (ie, bootloader and
>> kernel) is done, we don't expect to have to enable both methods.
>
> I think that, for the moment, it is best if the bootloader on already
> existing subarchitectures where DT is introduced still preserve the
> already existing ability to boot using ATAGs. ?This allows for the
> testing and validation of the DT concept against the legacy ATAG method
> more easily.

I think we've got an agreement!  :-)

>
> On new subarchitectures, it might make sense to go with DT from the
> start instead of creating setup code for every single machine. ?In that
> case the bootloader for those machines would only need to care about DT
> and forget about ATAGs.
>
>> ?+ A simpler boot interface, so less to do (and get wrong) in the bootloader
>>
>> ?+ We don't have two potential sources of boot information
>
> Those last two are IMHO the biggest reasons for not having both ATAGs
> and DT at the same time. ?Otherwise the confusion about which one is
> authoritative, which one has precedence over the other, and/or whether
> the information should be obtained from one structure if it is missing
> from the other will simply bite us eventually for sure, as bootloader
> writers will get sloppy/lazy and have it wrong. ?I strongly suggest that
> we should specify that the kernel must consider either ATAGs _or_ a
> device tree, and that the bootloader must pass only one of them.

I still disagree on this point.  I think it will cause less confusion
to only have a single method for passing the dtb, but that is a debate
that we don't need to solve immediately since we've got a way forward
on passing the dtb now.

> [ I also insist on the ability for the DT info to be extractable and
> ?updatable at the bootloader level, and not hardcoded into the
> ?bootloader itself. But that's another topic for discussion. ]

Yet an important one.  I fully support you on this.

>> Although I have been using the atag for a while, I have not pushed it to an
>> upstream (either qemu or the kernel), as I would like to get a firm decision
>> on the best method before making any commitment.
>>
>> Comments and questions most welcome.
>
> My suggestion is to have the DT support to be considered just as another
> machine _within_ each subarchitecture. ?This means that a machine ID
> could be registered for DT on PXA, another for DT on OMAP, another for
> DT on Dove, etc. ?This way, the DT support can be developed in parallel
> to the existing machine support code. ?So if for example you want to
> test DT for Kirkwood then you may boot the kernel passing the ID for DT
> on Kirkwood into r1 and provide the DT corresponding to, say, a
> SheevaPlug. ?Or you may decide to boot the same kernel binary and use
> the legacy SheevaPlug machine ID instead. ?In theory both methods should
> be equivalent, baring any bugs.
>
> Why one DT machine ID per subarchitecture? ?Simply because a significant
> part of the DT handling code will have to be subarchitecture specific
> anyway. ?The timer hardware, the GPIO configuration and muxing, SOC
> specific platform data handling, power management config, and many other
> things are simply too different from one SOC family to another and
> trying to have a single global DT support code to rule them all is
> insane. ?At least with the concept of a "virtual" machine definition for
> DT per subarchitecture, the problem can easily be split and just fits
> naturally into the existing model on ARM.
>
> This means that, over time, the machine ID registration would simply
> transition from a per machine thing to a per subarchitecture / SOC
> family thing. ?And once the DT support is introduced for a given SOC
> family, then new machines using that SOC should be able to reuse
> the existing kernel binary for that SOC simply by providing a new DT
> data for the existing kernel to consume.

I think this sounds sane.  Per-subarchitecture ids are far more
scalable that per-machine ids.

side question: Do recent ARM SoCs provide any facility to reliably
detected the silicon at run time?  ie. like the processor (unique to
core) and system (unique to SoC) id registers on PowerPC.

> [ There is also the issue of being able to support multiple SOC families
> ?within the same kernel binary, but that's something that could be done
> ?with or without the device tree, and has issues of its own that the DT
> ?cannot solve. Hence this is orthogonal to DT and a topic for yet another
> ?discussion. ]

agreed.

g.

Boot interface for device trees on ARM

[Russell King - ARM Linux]

On Wed, May 19, 2010 at 05:57:55AM -0600, Grant Likely wrote:
> side question: Do recent ARM SoCs provide any facility to reliably
> detected the silicon at run time?  ie. like the processor (unique to
> core) and system (unique to SoC) id registers on PowerPC.

No.

Every SoC is effectively unique in all aspects.  There is no common
physical memory layout, no common interrupt controller, no common
timer, etc.  The base addresses for everything is also SoC specific,
so it's a catch-22 situation - you need to know what the SoC is to
be able to find out where various registers and memory is located.

Boot interface for device trees on ARM

[Nicolas Pitre]

[ Complete boot description and proposal added at the end
  for those interested. ]

On Wed, 19 May 2010, Grant Likely wrote:

> On Mon, May 17, 2010 at 10:34 PM, Nicolas Pitre <nico@fluxnic.net> wrote:
> > On Tue, 18 May 2010, Jeremy Kerr wrote:
> >> Some notes about this scheme:
> >>
> >> ?- This would break compatibility with the existing boot interface:
> >> bootloaders that expect a DT kernel will not be able to boot a non-DT kernel.
> >> However, does this matter? Once the machine support (ie, bootloader and
> >> kernel) is done, we don't expect to have to enable both methods.
> >
> > I think that, for the moment, it is best if the bootloader on already
> > existing subarchitectures where DT is introduced still preserve the
> > already existing ability to boot using ATAGs. ?This allows for the
> > testing and validation of the DT concept against the legacy ATAG method
> > more easily.
> 
> I think we've got an agreement!  :-)

Good.

> > On new subarchitectures, it might make sense to go with DT from the
> > start instead of creating setup code for every single machine. ?In that
> > case the bootloader for those machines would only need to care about DT
> > and forget about ATAGs.
> >
> >> ?+ A simpler boot interface, so less to do (and get wrong) in the bootloader
> >>
> >> ?+ We don't have two potential sources of boot information
> >
> > Those last two are IMHO the biggest reasons for not having both ATAGs
> > and DT at the same time. ?Otherwise the confusion about which one is
> > authoritative, which one has precedence over the other, and/or whether
> > the information should be obtained from one structure if it is missing
> > from the other will simply bite us eventually for sure, as bootloader
> > writers will get sloppy/lazy and have it wrong. ?I strongly suggest that
> > we should specify that the kernel must consider either ATAGs _or_ a
> > device tree, and that the bootloader must pass only one of them.
> 
> I still disagree on this point.  I think it will cause less confusion
> to only have a single method for passing the dtb, but that is a debate
> that we don't need to solve immediately since we've got a way forward
> on passing the dtb now.

I don't understand you here.  We still are not in agreement on the 
method to pass the DTB yet, but I for sure don't want more than one 
method to pass it to the kernel.

There is a proposal to use the ATAG facility to encapsulate a pointer to 
the DT data.  I think this is unnecessary.

I'd much prefer if r2 contained either a pointer to the ATAG list, or a 
pointer to the DTB.  That's much cleaner and simpler.  As Jeremy pointed 
out, both structures include a magic number allowing to distinguish one 
from the other, so there is no backward compatibility issues.

> >> This proposal still does not require ATAG_DEVTREE?
> >
> > No.
>
> Hmmm...  I misunderstood then.  I don't agree that this is the best
> way forward
>
> Doing it this way means a non-compatible break in the interface.  It
> means that the bootloader needs to know what interface the kernel is
> expecting for boot; information that is not readily available from the
> image type.

I don't follow your point here.

It is not up to the bootloader to "adjust" to the kernel.  But rather 
for the kernel to cope with the bootloader's provided information.  If 
the bootloader passes a specific machine ID with the ATAG list then the 
kernel will use that, and if the bootloader passes a DT machine ID with 
a DT blob then the kernel will use that.  You just have to configure 
your kernel with both "machine types" at the same time.

> The user then needs to tell the boot loader which
> interface to use rather than a backwards compatible addition of a blob
> of data.

This is certainly not a show stopper, right?  If you planned for your 
bootloader to _already_ support both the ATAGs and the DT at the same 
time, it is not a big deal to have a config option that the user can 
change to select between "legacy" and "DT" boot methods.  Since we still 
want to preserve the ability to have a DT enabled kernel to boot using 
the legacy method, you will need a way to tell the bootloader which 
machine ID to use in that case anyway.

> You mention below "shifting the World Order on ARM" and it creating
> resistance for merging DT support.  Isn't this much the same thing as
> it creates a non backwards compatible change in the way bootloaders
> pass data to the kernel.  The cutover in powerpc from the old
> interface to the new caused no end of confusion and people who could
> no longer get their systems to boot.  On PowerPC is was necessary
> because the old method was completely broken, but ATAGs are clean,
> simple and well implemented.

I don't dispute the ATAG implementation.  I don't believe it is a good 
thing to carry the ATAG and DT as a combined boot requirement going 
forward.

On one hand, some people are claiming that the machine ID should be 
stuffed in the device tree and whatever is passed into r1 should be 
ignored.  On the other hand, you are saying that ATAGs should be kept 
forever as legacy cruft just to provide a reference to the DT data.  To 
me those are incompatible design objectives.

> It also means teaching every boot loader two separate methods for
> booting and exposing those differences to the user.

That's overstating the problem.  First, any bootloader wanting to use DT 
_will_ have to implement such a method.  With your proposal, all those 
bootloaders will _also_ have to know about ATAGs anyway.  So I don't see 
what your point is.

Let me start it all over again.

Here's how I think this should be handled on ARM.  Sorry PPC folks, but 
there is 15 years of ARM legacy to consider here, and what has been done 
on PPC might or might not be practical here -- please keep that in mind.  
I'm also going to use the present tense so to make this proposal as 
unambiguous as possible.

Booting Linux on ARM
====================

Two methods for passing boot information to the Linux kernel on ARM are 
possible:

1) the legacy method

2) the DT (device tree) method

The legacy method
-----------------

The legacy method for booting Linux on ARM requires that a unique 
machine ID be registered with RMK's machine ID database.  It can be 
viewed online at http://www.arm.linux.org.uk/developer/machines/.  
Before branching into the kernel image, the CPU register r1 must be 
initialized by the bootloader with the appropriate machine ID value for 
the exact hardware the kernel is booted on.  That is a hard requirement.

The kernel must also contain code specific to that machine ID in order 
to perform early setup of appropriate resources, create mappings for 
basic peripherals such as hardware timers, register relevant platform 
devices, etc.  That code can be found in those files containing the 
MACHINE_START and MACHINE_END macros, typically in arch/arm/mach-*/*.c.  
This is also a hard requirement.

Optionally (but strongly recommended), a tagged list (aka ATAGs) may be 
created in memory to pass extra information such as the size and 
location for RAM banks, size and location for a ramdisk, the kernel 
cmdline string, etc.  When not provided, the kernel will rely on default 
values which may or may not be sufficient to boot the kernel, or fall 
back to the information provided by a built-in kernel cmdline string 
determined by CONFIG_CMDLINE.

The location of the ATAG list is somewhat problematic.  Traditionally it 
has been stored at an offset of 0x100 from start of memory.  But some 
proposals were pushed forward to use r2 to indicate the location of the 
ATAG list.  Because it is impossible to determine with certainty whether 
or not the bootloader does actually initialize r2 with that information, 
this requirement was never enforced, which would otherwise cause 
backward compatibility issues.  The ATAG location is therefore stored 
into the machine record structure and therefore hardcoded at compile 
time.

When the kernel boots, it looks up into a table of all the machine 
records that have been compiled into the kernel for the one that 
corresponds with the machine ID passed into r1.  This lookup is 
performed in the very early boot stage from assembly code (with no stack 
available) to set up basic MMU stuff that is necessary for the kernel to 
be debuggable until the full fledged MMU support code takes over later 
during the boot.  Later on, the machine record is used to call init 
functions to initialize IO mappings, IRQs, timers, and so on as 
described above.  While the init_machine field in the machine record is 
obviously machine specific, all the other fields in the machine record 
end up being pretty much the same across all machines within the same 
SOC family.

The DT method
-------------

The DT method requires that a unique ID be registered for each SOC 
family for the DT purpose within the same ID space.  But instead of 
having one ID for each possible machine, only one ID per SOC family is 
required.  So the bootloader simply has to pass into r1 the ID 
corresponding to DT instead of the specific machine ID.  This is a hard 
requirement.

With this, the kernel can remain largely backward compatible with the 
legacy boot method, requiring _no_ change to the existing code, as the 
ID is sufficient to distinguish between both boot types.  The machine 
record remains largely relevant even for a DT boot as the majority of 
its content is SOC specific anyway, and given a per SOC ID for DT usage 
means that the early boot facilities are still usable as is even in the 
DT context.  And then the init_machine method in the machine record is 
naturally used to parse the device tree and do its work on multiple 
machines' behalf instead of relying on compiled-in static data for a 
specific machine.

The bootloader must also store in memory the DT data and pass its 
location via r2.  The r2 initialization therefore becomes a hard 
requirement when a DT is used. The boot_params field in the DT machine 
record could be set to -1 to disable ATAG parsing when a DT machine ID 
is used, and look into the DT data instead.

Backward Compatibility Considerations
-------------------------------------

Now let's have a look at what happens when we mix this with bootloaders.

If a kernel is too old, or was not configured with the "machine support" 
for DT, then only one thing may happen when a bootloader passes a DT 
machine ID to the kernel.  The early boot code will fail to find the 
machine record for the provided ID and refuse to boot.  If you have 
CONFIG_DEBUG_LL and are attempting to boot on the right SOC, then you'll 
probably get the error message on a serial port, along with a list of 
the actual machines and their IDs supported by that kernel.  In other 
words, this is not a new failure mode. Two solutions: either reconfigure 
your kernel to add DT support for your SOC, or configure your bootloader 
to use the legacy boot method.  And having a bootloader that can do DT 
only for a machine that has only legacy support in the kernel is 
senseless, so that case needs no be considered any longer.

The opposite is the kernel being configured for DT _only_, and the 
bootloader being too old, or misconfigured, so it passes a non-DT 
machine ID to the kernel.  In this case the same failure mode as above 
will be observed: kernel failing to find the matching machine record and 
halting (after attempting to display the error and the available machine 
supported). Solution: reconfigure your kernel with legacy support for 
that machine.

If the goal is to experiment with DT support in the kernel, and the 
bootloader does not support DT, then some shim can be prepended to the 
kernel image to fix things up.  Most bootloaders already have the 
ability to load arbitrary binary data in memory, and branch to arbitrary 
location too.  So it should be pretty easy to tftp the kernel, then tftp 
the DT data, to finally branch into the kernel with appropriate values 
in r1 and r2.

Conclusion
----------

I hope I've illustrated clearly why 1) keeping the requirement for 
passing a machine ID into r1, even for DT, is useful, and 2) why 
dropping ATAGs when using DT is simpler, cleaner, and has no relevant 
backward compatibility issues.  With #1 the ability to seamlessly use 
the existing early debugging code is preserved, and the introduction of 
DT fits perfectly well in the existing machine support model on ARM. Yet 
the new boot requirements introduced with DT are only for "machine" IDS 
that are still not in use, meaning there is no backward compatibilities 
introduced at all.  With #2 it will be possible to simplify bootloader 
support for new platforms being DT enabled without any legacy needs, and 
when the kernel is configured only for DT support then it will be 
possible to configure out the ATAG support code entirely.


Nicolas

Boot interface for device trees on ARM

[Jamie Lokier]

Nicolas Pitre wrote:
> It is not up to the bootloader to "adjust" to the kernel.  But rather 
> for the kernel to cope with the bootloader's provided information.  If 
> the bootloader passes a specific machine ID with the ATAG list then the 
> kernel will use that, and if the bootloader passes a DT machine ID with 
> a DT blob then the kernel will use that.  You just have to configure 
> your kernel with both "machine types" at the same time.

Scenario:

You upgrade your systems to a new DT-capable kernel and DT-capable
bootloader.  It works great.  You ship new instances of your device with this.

Once they're in the field, someone reports a bug that doesn't happen
with the older device instances.  It's not a bug you can reproduce,
but you suspect the newer of kernel.  So you remote-update some of the
newly shipped devices with an old, pre-DT kernel binary that's been
stable on the older devices, to see if the bug goes away.

Problem: The newer shipped devices have a DT-capable bootloader, and
it can't boot old kernels, because they don't understand the DT format.

You could remote-downgrade the bootloaders, but that's risky.  You
could try building the old kernel with DT support, but that adds
another variable to your testing.

Anticipating this well in advance, you of course built your DT-capable
bootloader with the ability to boot old and new style kernels...

-- Jamie

Boot interface for device trees on ARM

[Nicolas Pitre]

On Wed, 19 May 2010, Jamie Lokier wrote:

> Nicolas Pitre wrote:
> > It is not up to the bootloader to "adjust" to the kernel.  But rather 
> > for the kernel to cope with the bootloader's provided information.  If 
> > the bootloader passes a specific machine ID with the ATAG list then the 
> > kernel will use that, and if the bootloader passes a DT machine ID with 
> > a DT blob then the kernel will use that.  You just have to configure 
> > your kernel with both "machine types" at the same time.
> 
> Scenario:
> 
> You upgrade your systems to a new DT-capable kernel and DT-capable
> bootloader.  It works great.  You ship new instances of your device with this.
> 
> Once they're in the field, someone reports a bug that doesn't happen
> with the older device instances.  It's not a bug you can reproduce,
> but you suspect the newer of kernel.  So you remote-update some of the
> newly shipped devices with an old, pre-DT kernel binary that's been
> stable on the older devices, to see if the bug goes away.
> 
> Problem: The newer shipped devices have a DT-capable bootloader, and
> it can't boot old kernels, because they don't understand the DT format.
> 
> You could remote-downgrade the bootloaders, but that's risky.  You
> could try building the old kernel with DT support, but that adds
> another variable to your testing.
> 
> Anticipating this well in advance, you of course built your DT-capable
> bootloader with the ability to boot old and new style kernels...

Exact.  Quoting myself:

|I think that, for the moment, it is best if the bootloader on already
|existing subarchitectures where DT is introduced still preserve the
|already existing ability to boot using ATAGs.  This allows for the
|testing and validation of the DT concept against the legacy ATAG method
|more easily.
|
|On new subarchitectures, it might make sense to go with DT from the
|start instead of creating setup code for every single machine.  In that
|case the bootloader for those machines would only need to care about DT
|and forget about ATAGs.


Nicolas

Boot interface for device trees on ARM

[John Rigby]

Wow, no responses for almost two days, does that mean consensus has
been reached?

On Wed, May 19, 2010 at 2:22 PM, Nicolas Pitre <nico@fluxnic.net> wrote:
> On Wed, 19 May 2010, Jamie Lokier wrote:
>
>> Nicolas Pitre wrote:
>> > It is not up to the bootloader to "adjust" to the kernel. ?But rather
>> > for the kernel to cope with the bootloader's provided information. ?If
>> > the bootloader passes a specific machine ID with the ATAG list then the
>> > kernel will use that, and if the bootloader passes a DT machine ID with
>> > a DT blob then the kernel will use that. ?You just have to configure
>> > your kernel with both "machine types" at the same time.
>>
>> Scenario:
>>
>> You upgrade your systems to a new DT-capable kernel and DT-capable
>> bootloader. ?It works great. ?You ship new instances of your device with this.
>>
>> Once they're in the field, someone reports a bug that doesn't happen
>> with the older device instances. ?It's not a bug you can reproduce,
>> but you suspect the newer of kernel. ?So you remote-update some of the
>> newly shipped devices with an old, pre-DT kernel binary that's been
>> stable on the older devices, to see if the bug goes away.
>>
>> Problem: The newer shipped devices have a DT-capable bootloader, and
>> it can't boot old kernels, because they don't understand the DT format.
>>
>> You could remote-downgrade the bootloaders, but that's risky. ?You
>> could try building the old kernel with DT support, but that adds
>> another variable to your testing.
>>
>> Anticipating this well in advance, you of course built your DT-capable
>> bootloader with the ability to boot old and new style kernels...
>
> Exact. ?Quoting myself:
>
> |I think that, for the moment, it is best if the bootloader on already
> |existing subarchitectures where DT is introduced still preserve the
> |already existing ability to boot using ATAGs. ?This allows for the
> |testing and validation of the DT concept against the legacy ATAG method
> |more easily.
> |
> |On new subarchitectures, it might make sense to go with DT from the
> |start instead of creating setup code for every single machine. ?In that
> |case the bootloader for those machines would only need to care about DT
> |and forget about ATAGs.
>
>
> Nicolas
> _______________________________________________
> devicetree-discuss mailing list
> devicetree-discuss at lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/devicetree-discuss
>

Boot interface for device trees on ARM

[Jamie Bennett]

On Fri, 2010-05-21 at 10:24 -0600, John Rigby wrote:
> Wow, no responses for almost two days, does that mean consensus has
> been reached?

Consensus on device-tree enabled ARM hardware, surely not ;)

Regards,
Jamie.

Boot interface for device trees on ARM

[Russell King - ARM Linux]

On Fri, May 21, 2010 at 10:24:27AM -0600, John Rigby wrote:
> Wow, no responses for almost two days, does that mean consensus has
> been reached?

Probably not; I've not read much of the thread so far yet - I've been
mostly avoiding the ARM mailing list since the merge window has opened!

Boot interface for device trees on ARM

[Grant Likely]

On Fri, May 21, 2010 at 10:24 AM, John Rigby <jcrigby@gmail.com> wrote:
> Wow, no responses for almost two days, does that mean consensus has
> been reached?

No, it just means the merge window was open; not to mention the UDS hangover.

g.

>
> On Wed, May 19, 2010 at 2:22 PM, Nicolas Pitre <nico@fluxnic.net> wrote:
>> On Wed, 19 May 2010, Jamie Lokier wrote:
>>
>>> Nicolas Pitre wrote:
>>> > It is not up to the bootloader to "adjust" to the kernel. ?But rather
>>> > for the kernel to cope with the bootloader's provided information. ?If
>>> > the bootloader passes a specific machine ID with the ATAG list then the
>>> > kernel will use that, and if the bootloader passes a DT machine ID with
>>> > a DT blob then the kernel will use that. ?You just have to configure
>>> > your kernel with both "machine types" at the same time.
>>>
>>> Scenario:
>>>
>>> You upgrade your systems to a new DT-capable kernel and DT-capable
>>> bootloader. ?It works great. ?You ship new instances of your device with this.
>>>
>>> Once they're in the field, someone reports a bug that doesn't happen
>>> with the older device instances. ?It's not a bug you can reproduce,
>>> but you suspect the newer of kernel. ?So you remote-update some of the
>>> newly shipped devices with an old, pre-DT kernel binary that's been
>>> stable on the older devices, to see if the bug goes away.
>>>
>>> Problem: The newer shipped devices have a DT-capable bootloader, and
>>> it can't boot old kernels, because they don't understand the DT format.
>>>
>>> You could remote-downgrade the bootloaders, but that's risky. ?You
>>> could try building the old kernel with DT support, but that adds
>>> another variable to your testing.
>>>
>>> Anticipating this well in advance, you of course built your DT-capable
>>> bootloader with the ability to boot old and new style kernels...
>>
>> Exact. ?Quoting myself:
>>
>> |I think that, for the moment, it is best if the bootloader on already
>> |existing subarchitectures where DT is introduced still preserve the
>> |already existing ability to boot using ATAGs. ?This allows for the
>> |testing and validation of the DT concept against the legacy ATAG method
>> |more easily.
>> |
>> |On new subarchitectures, it might make sense to go with DT from the
>> |start instead of creating setup code for every single machine. ?In that
>> |case the bootloader for those machines would only need to care about DT
>> |and forget about ATAGs.
>>
>>
>> Nicolas
>> _______________________________________________
>> devicetree-discuss mailing list
>> devicetree-discuss at lists.ozlabs.org
>> https://lists.ozlabs.org/listinfo/devicetree-discuss
>>
> _______________________________________________
> devicetree-discuss mailing list
> devicetree-discuss at lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/devicetree-discuss
>



-- 
Grant Likely, B.Sc., P.Eng.
Secret Lab Technologies Ltd.

Boot interface for device trees on ARM

[Grant Likely]

On Wed, May 19, 2010 at 11:52 AM, Nicolas Pitre <nico@fluxnic.net> wrote:
> [ Complete boot description and proposal added at the end
> ?for those interested. ]

Alright, after much thinking, I'm okay with this approach.  None of my
remaining objections are showstoppers.

I've still got a few comments/questions.  I'll restrict them to the
full proposal to avoid confusion.

> Booting Linux on ARM
> ====================
>
> Two methods for passing boot information to the Linux kernel on ARM are
> possible:
>
> 1) the legacy method
>
> 2) the DT (device tree) method
>
> The legacy method
> -----------------
[...]
> The DT method
> -------------
>
> The DT method requires that a unique ID be registered for each SOC
> family for the DT purpose within the same ID space. ?But instead of
> having one ID for each possible machine, only one ID per SOC family is
> required. ?So the bootloader simply has to pass into r1 the ID
> corresponding to DT instead of the specific machine ID. ?This is a hard
> requirement.
>
> With this, the kernel can remain largely backward compatible with the
> legacy boot method, requiring _no_ change to the existing code, as the
> ID is sufficient to distinguish between both boot types. ?The machine
> record remains largely relevant even for a DT boot as the majority of
> its content is SOC specific anyway, and given a per SOC ID for DT usage
> means that the early boot facilities are still usable as is even in the
> DT context. ?And then the init_machine method in the machine record is
> naturally used to parse the device tree and do its work on multiple
> machines' behalf instead of relying on compiled-in static data for a
> specific machine.

There will still be instances of machine-specific setup code that
needs to be chosen at boot (based on the top level 'compatible'
property), but init_machine() appears to be early enough to handle
this.

hmmm... however, things the device tree blob and the initrd both need
to be marked as bootmem at paging_init() time, but init_machine()
doesn't run until later.  There will still need to be some hooks for
doing early DT processing, but none of that should be either board or
SoC specific.

> The bootloader must also store in memory the DT data and pass its
> location via r2. ?The r2 initialization therefore becomes a hard
> requirement when a DT is used. The boot_params field in the DT machine
> record could be set to -1 to disable ATAG parsing when a DT machine ID
> is used, and look into the DT data instead.
>
> Backward Compatibility Considerations
> -------------------------------------
>
> Now let's have a look at what happens when we mix this with bootloaders.
>
> If a kernel is too old, or was not configured with the "machine support"
> for DT, then only one thing may happen when a bootloader passes a DT
> machine ID to the kernel. ?The early boot code will fail to find the
> machine record for the provided ID and refuse to boot. ?If you have
> CONFIG_DEBUG_LL and are attempting to boot on the right SOC, then you'll
> probably get the error message on a serial port, along with a list of
> the actual machines and their IDs supported by that kernel. ?In other
> words, this is not a new failure mode. Two solutions: either reconfigure
> your kernel to add DT support for your SOC, or configure your bootloader
> to use the legacy boot method. ?And having a bootloader that can do DT
> only for a machine that has only legacy support in the kernel is
> senseless, so that case needs no be considered any longer.
>
> The opposite is the kernel being configured for DT _only_, and the
> bootloader being too old, or misconfigured, so it passes a non-DT
> machine ID to the kernel. ?In this case the same failure mode as above
> will be observed: kernel failing to find the matching machine record and
> halting (after attempting to display the error and the available machine
> supported). Solution: reconfigure your kernel with legacy support for
> that machine.
>
> If the goal is to experiment with DT support in the kernel, and the
> bootloader does not support DT, then some shim can be prepended to the
> kernel image to fix things up. ?Most bootloaders already have the
> ability to load arbitrary binary data in memory, and branch to arbitrary
> location too. ?So it should be pretty easy to tftp the kernel, then tftp
> the DT data, to finally branch into the kernel with appropriate values
> in r1 and r2.

A couple of other options here for experimenting is to carry a dtb as
a payload in the zImage wrapper, or to link dtbs directly into the
kernel proper and point to it from the machine structure.  Doing so
would allow even existing machines to use the same DT setup code.

My remaining objection is that the DT-only approach demands firmware
not just pass a dtb blob, but to also modify it.  The ATAG+DT approach
allows current firmware projects to continue passing data in the same
way it does now (less impact on firmware developers to add DT
support).

However, I'm not going to dwell on this point any longer because we
already have at least one firmware project (U-Boot) that understands
the device tree, and other firmware projects can use the shim approach
as a stop gap until native support is added.

It may make for a more painful transition, but you're probably right
in that it is better in the long run.

Go ahead and consider me "agreed".  :-)

g.

-- 
Grant Likely, B.Sc., P.Eng.
Secret Lab Technologies Ltd.

Boot interface for device trees on ARM

[John Rigby]

On Fri, Jun 4, 2010 at 2:01 PM, Grant Likely <grant.likely@secretlab.ca> wrote:

...

>
> Go ahead and consider me "agreed". ?:-)
>
> g.
>

And there was much rejoicing in the land.

Boot interface for device trees on ARM

[Jon Loeliger]

>
> > Go ahead and consider me "agreed". =A0:-)
> >
> > g.
> 
> And there was much rejoicing in the land.

Really.  Let the record show that Grant is agreeable. :-)

jdl

Boot interface for device trees on ARM

[Grant Likely]

On Fri, Jun 4, 2010 at 2:37 PM, Jon Loeliger <jdl@jdl.com> wrote:
>>
>> > Go ahead and consider me "agreed". =A0:-)
>> >
>> > g.
>>
>> And there was much rejoicing in the land.
>
> Really. ?Let the record show that Grant is agreeable. :-)

hey now, let's not get carried away.

g.

Boot interface for device trees on ARM

[Jeremy Kerr]

All,

> > With this, the kernel can remain largely backward compatible with the
> > legacy boot method, requiring _no_ change to the existing code, as the
> > ID is sufficient to distinguish between both boot types.  The machine
> > record remains largely relevant even for a DT boot as the majority of
> > its content is SOC specific anyway, and given a per SOC ID for DT usage
> > means that the early boot facilities are still usable as is even in the
> > DT context.  And then the init_machine method in the machine record is
> > naturally used to parse the device tree and do its work on multiple
> > machines' behalf instead of relying on compiled-in static data for a
> > specific machine.
> 
> There will still be instances of machine-specific setup code that
> needs to be chosen at boot (based on the top level 'compatible'
> property), but init_machine() appears to be early enough to handle
> this.
> 
> hmmm... however, things the device tree blob and the initrd both need
> to be marked as bootmem at paging_init() time, but init_machine()
> doesn't run until later.  There will still need to be some hooks for
> doing early DT processing, but none of that should be either board or
> SoC specific.

If we're planning to keep the machine IDs around (even if they are now per-
SoC), I'd like to know what would be left using them. The only thing that I 
can see that we currently use is io_pg_offset for the DEBUG_LL builds, and 
that isn't a convincing case to keep them.

Cheers,


Jeremy

Boot interface for device trees on ARM

[Nicolas Pitre]

On Sat, 5 Jun 2010, Jeremy Kerr wrote:

> All,
> 
> > > With this, the kernel can remain largely backward compatible with the
> > > legacy boot method, requiring _no_ change to the existing code, as the
> > > ID is sufficient to distinguish between both boot types.  The machine
> > > record remains largely relevant even for a DT boot as the majority of
> > > its content is SOC specific anyway, and given a per SOC ID for DT usage
> > > means that the early boot facilities are still usable as is even in the
> > > DT context.  And then the init_machine method in the machine record is
> > > naturally used to parse the device tree and do its work on multiple
> > > machines' behalf instead of relying on compiled-in static data for a
> > > specific machine.
> > 
> > There will still be instances of machine-specific setup code that
> > needs to be chosen at boot (based on the top level 'compatible'
> > property), but init_machine() appears to be early enough to handle
> > this.
> > 
> > hmmm... however, things the device tree blob and the initrd both need
> > to be marked as bootmem at paging_init() time, but init_machine()
> > doesn't run until later.  There will still need to be some hooks for
> > doing early DT processing, but none of that should be either board or
> > SoC specific.
> 
> If we're planning to keep the machine IDs around (even if they are now per-
> SoC), I'd like to know what would be left using them. The only thing that I 
> can see that we currently use is io_pg_offset for the DEBUG_LL builds, and 
> that isn't a convincing case to keep them.

Why not?

Kernel infrastructure backward compatibility means that you need to keep 
struct machine_desc instances around.  Granted, for the DT case, many of 
the members could be NULL or initialized with dummy stubs.  But that is 
very cheap to keep around, and that allows the same kernel binary to be 
able to use both the DT boot and the legacy boot.

In 10 years we might realize that non DT setups are no longer relevant 
and then all we'll have to do is rip out struct machine_desc and ignore 
the r1 value from the bootloader.  Ignoring that now means a flag day. 
And nobody wants a flag day forced upon them.


Nicolas

Boot interface for device trees on ARM

[Grant Likely]

On Fri, Jun 4, 2010 at 8:29 PM, Nicolas Pitre <nico@fluxnic.net> wrote:
> On Sat, 5 Jun 2010, Jeremy Kerr wrote:
>> > hmmm... however, things the device tree blob and the initrd both need
>> > to be marked as bootmem at paging_init() time, but init_machine()
>> > doesn't run until later. ?There will still need to be some hooks for
>> > doing early DT processing, but none of that should be either board or
>> > SoC specific.
>>
>> If we're planning to keep the machine IDs around (even if they are now per-
>> SoC), I'd like to know what would be left using them. The only thing that I
>> can see that we currently use is io_pg_offset for the DEBUG_LL builds, and
>> that isn't a convincing case to keep them.
>
> Why not?
>
> Kernel infrastructure backward compatibility means that you need to keep
> struct machine_desc instances around. ?Granted, for the DT case, many of
> the members could be NULL or initialized with dummy stubs. ?But that is
> very cheap to keep around, and that allows the same kernel binary to be
> able to use both the DT boot and the legacy boot.

I agree with Nicolas on this point.  He's convinced me that machine id
without ATAGs is sufficient to deal with the backwards compatibility
issues.  I have no problem with making machine_id in r1 part of the
boot requirements.

That being said; each DT must still be fully-formed.  Every DT still
needs a top level compatible property to uniquely identify the
machine.  If it turns out that Linux can ignore the machine ID and
rely solely on the DT data, then so be it.  I actually don't care much
since then it is just an internal Linux kernel implementation detail.

g.

-- 
Grant Likely, B.Sc., P.Eng.
Secret Lab Technologies Ltd.

Boot interface for device trees on ARM

[Jeremy Kerr]

Hi Nicolas,

> > If we're planning to keep the machine IDs around (even if they are now
> > per- SoC), I'd like to know what would be left using them. The only
> > thing that I can see that we currently use is io_pg_offset for the
> > DEBUG_LL builds, and that isn't a convincing case to keep them.
> 
> Why not?

Because it's something we can justifiably replace with a single compile-time 
constant, since it's only for low-level debug. I don't see that as a reason 
for keeping these machine numbers around if that's all they're used for.

However...

> Kernel infrastructure backward compatibility means that you need to keep
> struct machine_desc instances around.  Granted, for the DT case, many of
> the members could be NULL or initialized with dummy stubs.  But that is
> very cheap to keep around, and that allows the same kernel binary to be
> able to use both the DT boot and the legacy boot.

... this is a good point; if it helps with the transition, then this is 
probably a decent reason to keep it in the interface.

I'd like to keep my "only use the device tree" value (MACH_TYPE_DT = 
0xffffffff) though, so we don't needlessly allocate machine numbers where they 
aren't actually used.

Cheers,


Jeremy

Boot interface for device trees on ARM

[Nicolas Pitre]

On Wed, 9 Jun 2010, Jeremy Kerr wrote:

> Hi Nicolas,
> 
> > > If we're planning to keep the machine IDs around (even if they are now
> > > per- SoC), I'd like to know what would be left using them. The only
> > > thing that I can see that we currently use is io_pg_offset for the
> > > DEBUG_LL builds, and that isn't a convincing case to keep them.
> > 
> > Why not?
> 
> Because it's something we can justifiably replace with a single compile-time 
> constant, since it's only for low-level debug. I don't see that as a reason 
> for keeping these machine numbers around if that's all they're used for.

it's not only for that.  As I said below:

> > Kernel infrastructure backward compatibility means that you need to keep
> > struct machine_desc instances around.  Granted, for the DT case, many of
> > the members could be NULL or initialized with dummy stubs.  But that is
> > very cheap to keep around, and that allows the same kernel binary to be
> > able to use both the DT boot and the legacy boot.
> 
> ... this is a good point; if it helps with the transition, then this is 
> probably a decent reason to keep it in the interface.
> 
> I'd like to keep my "only use the device tree" value (MACH_TYPE_DT = 
> 0xffffffff) though, so we don't needlessly allocate machine numbers where they 
> aren't actually used.

There isn't such thing as "only use the device tree".  To keep the low 
level code _unmodified_ and _useful_ you'll have to allocate a number 
per machine class otherwise you won't be able to support more than one 
machine class in the same kernel binary at the same time.  You can't 
just say that those machine IDs aren't used.

I don't see the point of crippling that existing support just for the 
sake of using 0xffffffff, if it is only to save you a few machine ID 
registrations.  Because let's be honest, we won't DT-ify old machine 
targets anytime soon.  So only the recent and future machine classes 
will require a number, and that's probably on the order of 2 or 3 new 
IDs per year (meaning less than 5 minutes on RMK's automatic 
registration website).  Certainly much much less than the current 2000+ 
IDs that have been already allocated to date.


Nicolas

Boot interface for device trees on ARM

[Grant Likely]

On Mon, May 17, 2010 at 8:54 PM, Jeremy Kerr <jeremy.kerr@canonical.com> wrote:
> Hi all,
>
> As we're getting closer to device tree support on ARM, I'd like to get some
> input on our proposed boot interface.
>
> Basically, I'd like to define how we pass the device tree from the bootloader
> to the kernel.
>
> My current method of doing this is through a new atag. It looks like this:
>
> ? ? ? ?/* flattened device tree blob pointer */
> ? ? ? ?#define ATAG_DEVTREE ? ?0x5441000a
>
> ? ? ? ?struct tag_devtree {
> ? ? ? ? ? ? ? ?__u32 start; ? ?/* physical start address */
> ? ? ? ? ? ? ? ?__u32 size; ? ? /* size of dtb image in bytes */
> ? ? ? ?};
>
> With ATAG_DEVTREE, we keep the existing boot interface the same (ie, machine
> number in r1, atags pointer r2).

For the reference of those new to this discussion, here is a link to
the previous thread and the link to the draft boot interface document.

http://www.mail-archive.com/devicetree-discuss at lists.ozlabs.org/msg00884.html
http://devicetree.org/Boot_Environment

[...]

> ?- We are now potentially carrying data in two different places - atags and
> the device tree. For example, the physical memory layout and kernel command
> line may be present in both.

I've tried to explicitly addresses this in the draft document.  the DT
blob always supersedes ATAG information.

Cheers,
g.

-- 
Grant Likely, B.Sc., P.Eng.
Secret Lab Technologies Ltd.