RFC: Host-endian device tree format

RFC: Host-endian device tree format

[Dave Martin]

Hi all,

Apologies if this has been discussed before--- I don't see an obvious
rebuttal in my quick searching past threads, so I'll just quickly ask
the question:

Could we use host-endianness for the device tree binary blob?

The fdt binary format seems to lend itself strongly to a host-endian
format: it begins with a word-sized magic number, and when parsing the
device tree every elements type and size is known before that element
is read; furthermore, every element is size-aligned.  Therefore, I
(naively?) presume that a little-endian format should "just work",
simply by flipping the endianness of every element and discarding all
the __be32_to_cpu() type stuff when reading the fdt at boot time.  The
magic number ensures that there's no risk of accidentally reading the
fst in the wrong byte order.

Although it would be necessary to augment the fdt tools to cope with
this, offline tools appear to be the right place to put this
intelligence.  For the kernel to flip the byte order of everything in
the fdt blob every time the system boots just seems unnecessary when
the required endianness is statically known.

Since fdt is new on ARM we have an opportunity to change determine the
binary format now without breaking anything in the field -- we don't
necessarily have to adopt the host-endian format on any other
architectures if it's not desirable.

Any thoughts?

Cheers
---Dave

Re: RFC: Host-endian device tree format

[Grant Likely]

[-- Attachment #1.1: Type: text/plain, Size: 2274 bytes --]

On Jan 19, 2011 3:43 AM, "Dave Martin" <dave.martin-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org> wrote:
>
> Hi all,
>
> Apologies if this has been discussed before--- I don't see an obvious
> rebuttal in my quick searching past threads, so I'll just quickly ask
> the question:
>
> Could we use host-endianness for the device tree binary blob?

Hi David.

This has been discussed thoroughly and rejected. The dtb isn't so much
bigendian as it is network byte order. While we /could/ switch it for arm,
it creates all kinds of problems with tools and working with .dtb files on
dissimilar platforms.  Plus all the property values must remain in BE
because all the bindings are already defined that way in the OpenFirmware
specifications.

Besides, there is no real advantage to changing it: it already works as-is
on le architectures.  :-)

g.

>
> The fdt binary format seems to lend itself strongly to a host-endian
> format: it begins with a word-sized magic number, and when parsing the
> device tree every elements type and size is known before that element
> is read; furthermore, every element is size-aligned.  Therefore, I
> (naively?) presume that a little-endian format should "just work",
> simply by flipping the endianness of every element and discarding all
> the __be32_to_cpu() type stuff when reading the fdt at boot time.  The
> magic number ensures that there's no risk of accidentally reading the
> fst in the wrong byte order.
>
> Although it would be necessary to augment the fdt tools to cope with
> this, offline tools appear to be the right place to put this
> intelligence.  For the kernel to flip the byte order of everything in
> the fdt blob every time the system boots just seems unnecessary when
> the required endianness is statically known.
>
> Since fdt is new on ARM we have an opportunity to change determine the
> binary format now without breaking anything in the field -- we don't
> necessarily have to adopt the host-endian format on any other
> architectures if it's not desirable.
>
> Any thoughts?
>
> Cheers
> ---Dave
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[-- Attachment #1.2: Type: text/html, Size: 2924 bytes --]

[-- Attachment #2: Type: text/plain, Size: 192 bytes --]

_______________________________________________
devicetree-discuss mailing list
devicetree-discuss-uLR06cmDAlY/bJ5BZ2RsiQ@public.gmane.org
https://lists.ozlabs.org/listinfo/devicetree-discuss

Re: RFC: Host-endian device tree format

[Dave Martin]

Hi,

On Wed, Jan 19, 2011 at 1:41 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
>
> On Jan 19, 2011 3:43 AM, "Dave Martin" <dave.martin@linaro.org> wrote:
>>
>> Hi all,
>>
>> Apologies if this has been discussed before--- I don't see an obvious
>> rebuttal in my quick searching past threads, so I'll just quickly ask
>> the question:
>>
>> Could we use host-endianness for the device tree binary blob?
>
> Hi David.
>
> This has been discussed thoroughly and rejected.

I thought this may have been the case...

> The dtb isn't so much bigendian as it is network byte order.

(For me, "network byte order" is a euphemism ... <insert
unconstructive rant here>)

However I digress... it sounds like you are familiar with the
arguments I would make, and I'm not trying to get into an endianness
war ;)

> While we /could/ switch it for arm,
> it creates all kinds of problems with tools and working with .dtb files on
> dissimilar platforms.  Plus all the property values must remain in BE
> because all the bindings are already defined that way in the OpenFirmware
> specifications.

OK, agreed -- I guess I might have specified it differently :)  But
since the specs are already written and can't be supplemented without
causing disruption, I guess attempting changes may be more trouble
than it's worth.

>
> Besides, there is no real advantage to changing it: it already works as-is
> on le architectures.  :-)

I'm happy to go along with it if nobody sees a benefit in changing
it... the status quo is what it is.

Just wanted to know whether anyone cared / whether an opportunity was
being missed, given that the newness of this stuff on ARM could
present an opportunity.

If this is not the case then that's fine by me.

Cheers
---Dave

Re: RFC: Host-endian device tree format

[Nicolas Pitre]

On Wed, 19 Jan 2011, Dave Martin wrote:
> On Wed, Jan 19, 2011 at 1:41 PM, Grant Likely <grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org> wrote:
> > The dtb isn't so much bigendian as it is network byte order.
> 
> (For me, "network byte order" is a euphemism ... <insert
> unconstructive rant here>)

I concur.

> However I digress... it sounds like you are familiar with the
> arguments I would make, and I'm not trying to get into an endianness
> war ;)

I think this is the same issue as for filesystems.  Having different 
endianness in use might only lead to confusion when a piece of data can 
be externally provided.  And since DT was created on PPC first then that 
naturally decided on the endianness for it.


Nicolas

Re: RFC: Host-endian device tree format

[Grant Likely]

On Wed, Jan 19, 2011 at 8:41 AM, Nicolas Pitre <nicolas.pitre-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org> wrote:
> On Wed, 19 Jan 2011, Dave Martin wrote:
>> On Wed, Jan 19, 2011 at 1:41 PM, Grant Likely <grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org> wrote:
>> > The dtb isn't so much bigendian as it is network byte order.
>>
>> (For me, "network byte order" is a euphemism ... <insert
>> unconstructive rant here>)
>
> I concur.

meh. it's all about conventions.  When talking about external data, it
is far more important for everyone to agree on the same representation
than it is to tailor to each platforms preferences.  That said, rant
away!  I love a good tangent.

>> However I digress... it sounds like you are familiar with the
>> arguments I would make, and I'm not trying to get into an endianness
>> war ;)
>
> I think this is the same issue as for filesystems.  Having different
> endianness in use might only lead to confusion when a piece of data can
> be externally provided.

exactly.

>  And since DT was created on PPC first then that
> naturally decided on the endianness for it.

SPARC, actually, for the device tree data (via OpenFirmware), but the
dtb representation was indeed a powerpc invention.

g.

-- 
Grant Likely, B.Sc., P.Eng.
Secret Lab Technologies Ltd.

Re: RFC: Host-endian device tree format

[Nicolas Pitre]

On Wed, 19 Jan 2011, Grant Likely wrote:

> On Wed, Jan 19, 2011 at 8:41 AM, Nicolas Pitre <nicolas.pitre-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org> wrote:
> > On Wed, 19 Jan 2011, Dave Martin wrote:
> >> On Wed, Jan 19, 2011 at 1:41 PM, Grant Likely <grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org> wrote:
> >> > The dtb isn't so much bigendian as it is network byte order.
> >>
> >> (For me, "network byte order" is a euphemism ... <insert
> >> unconstructive rant here>)
> >
> > I concur.
> 
> meh. it's all about conventions.  When talking about external data, it
> is far more important for everyone to agree on the same representation
> than it is to tailor to each platforms preferences.

Exactly what I said, using different words.

> That said, rant
> away!  I love a good tangent.

The actual problem is not that DT chose to use big endian ordering, but 
rather that BE ordering was invented in the first place.  ;-)


Nicolas

Re: RFC: Host-endian device tree format

[Grant Likely]

On Wed, Jan 19, 2011 at 9:05 AM, Nicolas Pitre <nicolas.pitre-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org> wrote:
> On Wed, 19 Jan 2011, Grant Likely wrote:
>
>> On Wed, Jan 19, 2011 at 8:41 AM, Nicolas Pitre <nicolas.pitre-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org> wrote:
>> > On Wed, 19 Jan 2011, Dave Martin wrote:
>> >> On Wed, Jan 19, 2011 at 1:41 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
>> >> > The dtb isn't so much bigendian as it is network byte order.
>> >>
>> >> (For me, "network byte order" is a euphemism ... <insert
>> >> unconstructive rant here>)
>> >
>> > I concur.
>>
>> meh. it's all about conventions.  When talking about external data, it
>> is far more important for everyone to agree on the same representation
>> than it is to tailor to each platforms preferences.
>
> Exactly what I said, using different words.
>
>> That said, rant
>> away!  I love a good tangent.
>
> The actual problem is not that DT chose to use big endian ordering, but
> rather that BE ordering was invented in the first place.  ;-)

hahahahaha

g.


-- 
Grant Likely, B.Sc., P.Eng.
Secret Lab Technologies Ltd.

Re: RFC: Host-endian device tree format

[Dave Martin]

On Wed, Jan 19, 2011 at 3:52 PM, Grant Likely <grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org> wrote:
> On Wed, Jan 19, 2011 at 8:41 AM, Nicolas Pitre <nicolas.pitre-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org> wrote:
>> On Wed, 19 Jan 2011, Dave Martin wrote:
>>> On Wed, Jan 19, 2011 at 1:41 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
>>> > The dtb isn't so much bigendian as it is network byte order.
>>>
>>> (For me, "network byte order" is a euphemism ... <insert
>>> unconstructive rant here>)
>>
>> I concur.
>
> meh. it's all about conventions.  When talking about external data, it
> is far more important for everyone to agree on the same representation
> than it is to tailor to each platforms preferences.  That said, rant
> away!  I love a good tangent.

I guess a good example of what I mean is ELF - the endianness of an
ELF image is discoverable, and cross tools do exist -- but because ELF
images are strongly bound to their target platform, running the image
takes precedence over the simplicity of the tools: so the images are
specified in such a way that they can always be host-endian for the
machine they run on, without creating format ambiguities.

Of course, it's stretching things rather a lot to claim that fdt
parsing is performance critical ... or that ELF cross tools work well
in all host/target combinations ... rather it's a niggle.

So, I'm happy as-is.  It was just tempting to rock the boat a bit to
make sure everyone has their sea legs :)

Cheers
---Dave

Re: RFC: Host-endian device tree format

[Segher Boessenkool]

> The actual problem is not that DT chose to use big endian ordering, but
> rather that BE ordering was invented in the first place.  ;-)

Blame the Greeks, they started it, approximately 500 B.C.


Segher

Re: RFC: Host-endian device tree format

[Scott Wood]

On Wed, 19 Jan 2011 10:43:02 +0000
Dave Martin <dave.martin@linaro.org> wrote:

> Hi all,
> 
> Apologies if this has been discussed before--- I don't see an obvious
> rebuttal in my quick searching past threads, so I'll just quickly ask
> the question:
> 
> Could we use host-endianness for the device tree binary blob?
> 
> The fdt binary format seems to lend itself strongly to a host-endian
> format: it begins with a word-sized magic number, and when parsing the
> device tree every elements type and size is known before that element
> is read; furthermore, every element is size-aligned.  Therefore, I
> (naively?) presume that a little-endian format should "just work",
> simply by flipping the endianness of every element and discarding all
> the __be32_to_cpu() type stuff when reading the fdt at boot time.  The
> magic number ensures that there's no risk of accidentally reading the
> fst in the wrong byte order.

It's simpler and faster to unconditionally swap a word as you load it,
especially on instruction sets with swapping support, than it is to
dynamically decide whether to swap.  It would be nice if GCC would let
data structures be annotated with endianness, so that it would
automatically swap when needed, though.

Look at the I/O accessors in drivers/dma/fsldma.h -- that complexity
wouldn't have been necessary if Freescale hadn't "fixed" the endianness
when sticking the DMA engine in a new chip family.

What about bi-endian hardware?  Is it possible you might ever want to
boot a kernel in one endianness when the device tree was generated
assuming the opposite choice?

-Scott

Re: RFC: Host-endian device tree format

[Mikael Pettersson]

Scott Wood writes:
 > What about bi-endian hardware?  Is it possible you might ever want to
 > boot a kernel in one endianness when the device tree was generated
 > assuming the opposite choice?

There are ARMs out there, for instance IXP4XXs, that
are bi-endian, and people can and do switch endianess
when booting their kernels.

Re: RFC: Host-endian device tree format

[David Gibson]

On Wed, Jan 19, 2011 at 04:09:39PM +0000, Dave Martin wrote:
> On Wed, Jan 19, 2011 at 3:52 PM, Grant Likely <grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org> wrote:
> > On Wed, Jan 19, 2011 at 8:41 AM, Nicolas Pitre <nicolas.pitre-QSEj5FYQhm5QFI55V6+gNQ@public.gmane.orgg> wrote:
> >> On Wed, 19 Jan 2011, Dave Martin wrote:
> >>> On Wed, Jan 19, 2011 at 1:41 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
> >>> > The dtb isn't so much bigendian as it is network byte order.
> >>>
> >>> (For me, "network byte order" is a euphemism ... <insert
> >>> unconstructive rant here>)
> >>
> >> I concur.
> >
> > meh. it's all about conventions.  When talking about external data, it
> > is far more important for everyone to agree on the same representation
> > than it is to tailor to each platforms preferences.  That said, rant
> > away!  I love a good tangent.
> 
> I guess a good example of what I mean is ELF - the endianness of an
> ELF image is discoverable, and cross tools do exist -- but because ELF
> images are strongly bound to their target platform, running the image
> takes precedence over the simplicity of the tools: so the images are
> specified in such a way that they can always be host-endian for the
> machine they run on, without creating format ambiguities.

Yes, there's no technical impediment to making a little-endian dtb
format.  In fact dtc and libfdt were written with this possibility in
mind, to the extent that there are hooks / functions in the right
places to make this reasonably straightforward.

The fact that there are actually two questions here - the endianness
of the framing dtb format, and the endianness of the property values
themselves complicates the issue.

> Of course, it's stretching things rather a lot to claim that fdt
> parsing is performance critical ... or that ELF cross tools work well
> in all host/target combinations ... rather it's a niggle.

But, there is just no point in creating a variant format.  The
performance cost is completely negligible (even using the rather
stupid conversion macros from libfdt which gcc generates horrible code
for).

The simplicity of having only one variant for tools to deal with wins,
no contest.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Re: RFC: Host-endian device tree format

[Dave Martin]

On Wed, Jan 19, 2011 at 11:47 PM, David Gibson
<david-xT8FGy+AXnRB3Ne2BGzF6laj5H9X9Tb+@public.gmane.org> wrote:
> On Wed, Jan 19, 2011 at 04:09:39PM +0000, Dave Martin wrote:
>> On Wed, Jan 19, 2011 at 3:52 PM, Grant Likely <grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org> wrote:
>> > On Wed, Jan 19, 2011 at 8:41 AM, Nicolas Pitre <nicolas.pitre@linaro.org> wrote:
>> >> On Wed, 19 Jan 2011, Dave Martin wrote:
>> >>> On Wed, Jan 19, 2011 at 1:41 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
>> >>> > The dtb isn't so much bigendian as it is network byte order.
>> >>>
>> >>> (For me, "network byte order" is a euphemism ... <insert
>> >>> unconstructive rant here>)
>> >>
>> >> I concur.
>> >
>> > meh. it's all about conventions.  When talking about external data, it
>> > is far more important for everyone to agree on the same representation
>> > than it is to tailor to each platforms preferences.  That said, rant
>> > away!  I love a good tangent.
>>
>> I guess a good example of what I mean is ELF - the endianness of an
>> ELF image is discoverable, and cross tools do exist -- but because ELF
>> images are strongly bound to their target platform, running the image
>> takes precedence over the simplicity of the tools: so the images are
>> specified in such a way that they can always be host-endian for the
>> machine they run on, without creating format ambiguities.
>
> Yes, there's no technical impediment to making a little-endian dtb
> format.  In fact dtc and libfdt were written with this possibility in
> mind, to the extent that there are hooks / functions in the right
> places to make this reasonably straightforward.
>
> The fact that there are actually two questions here - the endianness
> of the framing dtb format, and the endianness of the property values
> themselves complicates the issue.
>
>> Of course, it's stretching things rather a lot to claim that fdt
>> parsing is performance critical ... or that ELF cross tools work well
>> in all host/target combinations ... rather it's a niggle.
>
> But, there is just no point in creating a variant format.  The
> performance cost is completely negligible (even using the rather
> stupid conversion macros from libfdt which gcc generates horrible code
> for).
>
> The simplicity of having only one variant for tools to deal with wins,
> no contest.

All good arguments, everyone-- thanks for the background.

Like I said, I'm happy for things to stay the way they are.

Cheers
---Dave