[Buildroot] endian issue

[Buildroot] endian issue

[Marcel]

Hi,

I'm using an Atmel sam9g45 using buildroot with linux 2.6.33.

I currently have most of my things working but run into an issue which is 
endian related.

My driver outputs it's data in big-endian mode (ADC data). This is fine with 
me.I added a package to buildroot with my own software that talks to the 
driver. This package than transfers the results either over ethernet, usb or 
rs232. This all works great as long as I don't perform any calculations in the 
package that I added. When I do so, all data will be messed up and I'm sure 
it's related to endiannes.
I think my package is compiled in little-endian mode and I'm quite sure this 
is the issue. I did some byte swapping to proof it's the issue and this byte 
swapping makes things fine. 

Is there any way to compile my package in big-endian mode from buildroot ?
Or is there another way I should force this ?

Best regards,
Marcel

[Buildroot] endian issue

[Grant Edwards]

On 2010-05-15, Marcel <korgull@home.nl> wrote:

> I'm using an Atmel sam9g45 using buildroot with linux 2.6.33.
>
> I currently have most of my things working but run into an issue
> which is endian related.
>
> My driver outputs it's data in big-endian mode (ADC data).

Oops. The 9g45 is a little-endian part.

> This is fine with me.

Apparently not, since you're complaining about how it breaks things.  ;)

> I added a package to buildroot with my own software that talks to the
> driver. This package than transfers the results either over ethernet,
> usb or rs232. This all works great as long as I don't perform any
> calculations in the package that I added.

That's because your processor is running in little-endian mode.  Doing
calculations on big-endian data using a little-endian part (or vice
versa) doesn't work.

> When I do so, all data will be messed up and I'm sure it's related to
> endiannes.
n
Indeed it is.

> I think my package is compiled in little-endian mode

I would hope so, since you're running it on a little-endian part.

> and I'm quite sure this is the issue.

The issue is your driver.  Change your driver to return data in the
format expected by your processor (that's little-endian in the case of
the 9g45). Then use standard macros (hton(), htons(), htonl()) to
convert data to "network order" before you ship it out via Ethernet or
USB.

[Don't ask me why Atmel chose little-endian when they designed the
AT91 peripherals.  For anything with a network interface, big-endian
makes a lot more sense to me.  But that debate has been going on for
40 years...]

> I did some byte swapping to proof it's the issue and this byte
> swapping makes things fine. 
>
> Is there any way to compile my package in big-endian mode from
> buildroot ?

Yes.

But doing so will result in object files that don't work.  In fact
you won't even be able to link them.

> Or is there another way I should force this ?

No, you shouldn't force this.  You should fix your driver.  Really.

-- 
Grant

[Buildroot] endian issue

[Lionel Landwerlin]

Le samedi 15 mai 2010 ? 15:47 +0200, Marcel a ?crit :
> Hi,
> 
> I'm using an Atmel sam9g45 using buildroot with linux 2.6.33.
> 
> I currently have most of my things working but run into an issue which is 
> endian related.
> 
> My driver outputs it's data in big-endian mode (ADC data). This is fine with 
> me.I added a package to buildroot with my own software that talks to the 
> driver. This package than transfers the results either over ethernet, usb or 
> rs232. This all works great as long as I don't perform any calculations in the 
> package that I added. When I do so, all data will be messed up and I'm sure 
> it's related to endiannes.
> I think my package is compiled in little-endian mode and I'm quite sure this 
> is the issue. I did some byte swapping to proof it's the issue and this byte 
> swapping makes things fine. 

The endianness configuration of your processor isn't something you can
usually change 'on-the-fly'. It's usually set up early in the boot
process.
So you have to choose whether you want to compile all your system in big
or little endian, you can select that from the buildroot architecture
configuration (arm -> little, armeb -> big, for example).

> 
> Is there any way to compile my package in big-endian mode from buildroot ?
> Or is there another way I should force this ?
> 

You can't select that for 1 package, it's for the whole system or
nothing.
Otherwise, the smartest approch would be to make endian detection (at
compile time or at running time) to adapt your processing algorithm.

Regards,

--
Lionel Landwerlin

[Buildroot] endian issue

[Grant Edwards]

On 2010-05-15, Lionel Landwerlin <llandwerlin@gmail.com> wrote:
> Le samedi 15 mai 2010 ?? 15:47 +0200, Marcel a ??crit :
>> 
>> I'm using an Atmel sam9g45 using buildroot with linux 2.6.33.
>> 
>> I currently have most of my things working but run into an issue which is 
>> endian related.
>
> The endianness configuration of your processor isn't something you
> can usually change 'on-the-fly'. It's usually set up early in the
> boot process.

While the endian configuration of the ARM9 core can be changed, the
AT91 peripherals are little-endian.  In theory, he should be able to
run the ARM9 core in big-endian mode, but a running uC core without
any working peripherals is surprisingly useless.

> So you have to choose whether you want to compile all your system in
> big or little endian, you can select that from the buildroot
> architecture configuration (arm -> little, armeb -> big, for
> example).

In this case, he has to choose little-endian.

[I've never seen an ARM-based uController that had peripherals with
configurable endianess -- are there any?]

>> Is there any way to compile my package in big-endian mode from
>> buildroot? Or is there another way I should force this?
>
> You can't select that for 1 package, it's for the whole system or
> nothing. Otherwise, the smartest approch would be to make endian
> detection (at compile time or at running time) to adapt your
> processing algorithm.

Yep, the OP's should driver should return data in host-order.

Otherwise, he'll have to suffer the guilt of knowing that a few years
from now some poor sod who inherits the code will have a stroke from
the effort required to resist the urge to track down the OP and slap
him silly.

-- 
Grant

[Buildroot] endian issue

[Marcel]

On Saturday 15 May 2010 04:43:29 pm Grant Edwards wrote:
> On 2010-05-15, Lionel Landwerlin <llandwerlin@gmail.com> wrote:
> > Le samedi 15 mai 2010 ?? 15:47 +0200, Marcel a ??crit :
> >> I'm using an Atmel sam9g45 using buildroot with linux 2.6.33.
> >> 
> >> I currently have most of my things working but run into an issue which
> >> is endian related.
> > 
> > The endianness configuration of your processor isn't something you
> > can usually change 'on-the-fly'. It's usually set up early in the
> > boot process.
> 
> While the endian configuration of the ARM9 core can be changed, the
> AT91 peripherals are little-endian.  In theory, he should be able to
> run the ARM9 core in big-endian mode, but a running uC core without
> any working peripherals is surprisingly useless.
> 
> > So you have to choose whether you want to compile all your system in
> > big or little endian, you can select that from the buildroot
> > architecture configuration (arm -> little, armeb -> big, for
> > example).
> 
> In this case, he has to choose little-endian.

That's very clear. 

> [I've never seen an ARM-based uController that had peripherals with
> configurable endianess -- are there any?]
> 
> >> Is there any way to compile my package in big-endian mode from
> >> buildroot? Or is there another way I should force this?
> > 
> > You can't select that for 1 package, it's for the whole system or
> > nothing. Otherwise, the smartest approch would be to make endian
> > detection (at compile time or at running time) to adapt your
> > processing algorithm.
> 
> Yep, the OP's should driver should return data in host-order.
> 
> Otherwise, he'll have to suffer the guilt of knowing that a few years
> from now some poor sod who inherits the code will have a stroke from
> the effort required to resist the urge to track down the OP and slap
> him silly.

If I wish to do that, how do I detect the endianness of a sytem and do I need 
to reformat my data in my drivers for this ? If so, isn't that a pure waste of 
cpu cycles for the sake of reusable code ?
If I can do this without any speed sacrifice than I will do it, if not....than 
it's simply not an option for this system.

Best regards,
Marcel

[Buildroot] endian issue

[Thomas Petazzoni]

On Sat, 15 May 2010 17:26:57 +0200
Marcel <korgull@home.nl> wrote:

> If I wish to do that, how do I detect the endianness of a sytem and
> do I need to reformat my data in my drivers for this ? If so, isn't
> that a pure waste of cpu cycles for the sake of reusable code ?
> If I can do this without any speed sacrifice than I will do it, if
> not....than it's simply not an option for this system.

If your driver is in the kernel, tust use the kernel functions
be16_to_cpu(), be32_to_cpu(), etc. They will automatically do the
appropriate conversion depending on the CPU endianess.

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] endian issue

[Lionel Landwerlin]

Le samedi 15 mai 2010 ? 17:26 +0200, Marcel a ?crit :
> On Saturday 15 May 2010 04:43:29 pm Grant Edwards wrote:
> > On 2010-05-15, Lionel Landwerlin <llandwerlin@gmail.com> wrote:
> > > Le samedi 15 mai 2010 ?? 15:47 +0200, Marcel a ??crit :
> > >> I'm using an Atmel sam9g45 using buildroot with linux 2.6.33.
> > >> 
> > >> I currently have most of my things working but run into an issue which
> > >> is endian related.
> > > 
> > > The endianness configuration of your processor isn't something you
> > > can usually change 'on-the-fly'. It's usually set up early in the
> > > boot process.
> > 
> > While the endian configuration of the ARM9 core can be changed, the
> > AT91 peripherals are little-endian.  In theory, he should be able to
> > run the ARM9 core in big-endian mode, but a running uC core without
> > any working peripherals is surprisingly useless.
> > 
> > > So you have to choose whether you want to compile all your system in
> > > big or little endian, you can select that from the buildroot
> > > architecture configuration (arm -> little, armeb -> big, for
> > > example).
> > 
> > In this case, he has to choose little-endian.
> 
> That's very clear. 
> 
> > [I've never seen an ARM-based uController that had peripherals with
> > configurable endianess -- are there any?]
> > 
> > >> Is there any way to compile my package in big-endian mode from
> > >> buildroot? Or is there another way I should force this?
> > > 
> > > You can't select that for 1 package, it's for the whole system or
> > > nothing. Otherwise, the smartest approch would be to make endian
> > > detection (at compile time or at running time) to adapt your
> > > processing algorithm.
> > 
> > Yep, the OP's should driver should return data in host-order.
> > 
> > Otherwise, he'll have to suffer the guilt of knowing that a few years
> > from now some poor sod who inherits the code will have a stroke from
> > the effort required to resist the urge to track down the OP and slap
> > him silly.
> 
> If I wish to do that, how do I detect the endianness of a sytem and do I need 
> to reformat my data in my drivers for this ? If so, isn't that a pure waste of 
> cpu cycles for the sake of reusable code ?
> If I can do this without any speed sacrifice than I will do it, if not....than 
> it's simply not an option for this system.
> 
> Best regards,
> Marcel
> 

A basic example of detection (copy pasta from here
http://www.allegro.cc/forums/print-thread/592785)

=>

Uint32 Value32;
Uint8 *VPtr = (Uint8 *)&Value32;

VPtr[0] = VPtr[1] = VPtr[2] = 0; VPtr[3] = 1;

if(Value32 == 1)
   printf("I'm big endian\n");
else
   printf("I'm little endian\n");

<=

Some packages make this detection within configure script using such
code (which is problematic with cross compilation because you can't run
target code on the host) and then use some #ifdef BIG_ENDIAN etc...
Of course doing endian translation costs more than doing nothing.

--
Lionel Landwerlin

[Buildroot] endian issue

[Grant Edwards]

On 2010-05-15, Marcel <korgull@home.nl> wrote:

> If I wish to do that, how do I detect the endianness of a sytem

For user space:

  /usr/include/endian.h

For kernel code:

 /usr/src/linux/include/linux/byteorder.h

> and do I need to reformat my data in my drivers for this?

That's up to you to decide.

Based on your original posting, I was under the impression you needed
to do calculations on your data on the 9g45. If you need to be able to
do calculations on your data on the 9g45, then that data has to be
little-endian.

If you need to do calculations on big-endian input data and generate a
big-endian output stream, and the 9g45 doesn't have the throughput to
do the byte-swapping, then you probably need to choose a different
(probably big-endian) part.  Many parts that are aimed more squarely
at network stuff are big-endian. Perhaps one of the Intel IXP network
processors (but not the PXA parts) or Motorola PPC chips [you can
check with your local distributor to find out the nom du jour for
company/product-line].

If the peripheral produces big-endian data, you don't need to do any
computations with that data, and you want to ship out big-endian data,
then, no you don't have a problem.  Just make sure that you put in
plenty of warning comments so that later when somebody does decide to
add some computations on the 9g45 they have an easier time figuring
out why it doesn't work.

Better yet, since wise maintainers generally ignore comments, you
could use an opaque data type for the data since it's just a blob of
bytes and not a numeric value.  A structure containing a single field
that's an array of 4 bytes should make a good opaque 32-bit type. That
way whoever has to maintain your code won't fall into the trap you'd be
setting if you declare something as a numeric type when it really isn't.

> If so, isn't that a pure waste of cpu cycles for the sake of reusable
> code?

If you didn't want to do calculations on data on the 9g45, what
exactly was the problem you were asking about in your OP?

> If I can do this without any speed sacrifice than I will do it, if
> not....than it's simply not an option for this system.

Sorry, I have no way of knowing what your requirements are nor of
knowing the cost of producing little-endian data in your driver.

Perhaps you can fix the board layout to correctly connect whatever
peripheral is providing big-endian data?  Then you'd only have to do
the byte-swapping once: after you've done the calculations on the 9g45
and before you ship it out via Ethernet or USB.

-- 
Grant

[Buildroot] endian issue

[Marcel]

On Saturday 15 May 2010 06:45:50 pm Grant Edwards wrote:
> On 2010-05-15, Marcel <korgull@home.nl> wrote:
> > If I wish to do that, how do I detect the endianness of a sytem
> 
> For user space:
> 
>   /usr/include/endian.h
> 
> For kernel code:
> 
>  /usr/src/linux/include/linux/byteorder.h
> 
> > and do I need to reformat my data in my drivers for this?
> 
> That's up to you to decide.
> 
> Based on your original posting, I was under the impression you needed
> to do calculations on your data on the 9g45. If you need to be able to
> do calculations on your data on the 9g45, then that data has to be
> little-endian.
> 
> If you need to do calculations on big-endian input data and generate a
> big-endian output stream, and the 9g45 doesn't have the throughput to
> do the byte-swapping, then you probably need to choose a different
> (probably big-endian) part.  Many parts that are aimed more squarely
> at network stuff are big-endian. Perhaps one of the Intel IXP network
> processors (but not the PXA parts) or Motorola PPC chips [you can
> check with your local distributor to find out the nom du jour for
> company/product-line].
> 
> If the peripheral produces big-endian data, you don't need to do any
> computations with that data, and you want to ship out big-endian data,
> then, no you don't have a problem.  Just make sure that you put in
> plenty of warning comments so that later when somebody does decide to
> add some computations on the 9g45 they have an easier time figuring
> out why it doesn't work.
> 
> Better yet, since wise maintainers generally ignore comments, you
> could use an opaque data type for the data since it's just a blob of
> bytes and not a numeric value.  A structure containing a single field
> that's an array of 4 bytes should make a good opaque 32-bit type. That
> way whoever has to maintain your code won't fall into the trap you'd be
> setting if you declare something as a numeric type when it really isn't.
> 
> > If so, isn't that a pure waste of cpu cycles for the sake of reusable
> > code?
> 
> If you didn't want to do calculations on data on the 9g45, what
> exactly was the problem you were asking about in your OP?
> 
> > If I can do this without any speed sacrifice than I will do it, if
> > not....than it's simply not an option for this system.
> 
> Sorry, I have no way of knowing what your requirements are nor of
> knowing the cost of producing little-endian data in your driver.
> 
> Perhaps you can fix the board layout to correctly connect whatever
> peripheral is providing big-endian data?  Then you'd only have to do
> the byte-swapping once: after you've done the calculations on the 9g45
> and before you ship it out via Ethernet or USB.

Well, I need to do quite a lot of calculation so I decided it's wise to make 
my driver output the data in little-endian format, than perform the 
calculations and at the very end convert if to big endian before moving it out 
via ethernet or USB.
My calculations involve quite a lot of matrix multiplying in floating point so 
conversions in there wouldn't be nice either. So, the only good choice would 
be to output the data from my driver in little-endian format.

There's no cost of cycles in my driver whether I output little or big endian 
data. The data comes in two bytes from a 16bit ADC and can be stored in any 
order in my array. I basically read a stream from this ADC. Having both big 
and little supported in the driver may cost some overhead which I don't  want.

However since this driver is also used on another platform I will use the 
suggested methods from the other posts to make the driver independent of the 
host endian format and try to reduce the overhead to a minimum.

Thanks for all the great replies. 

Best regards,
Marcel

[Buildroot] endian issue

[Marcel]

On Saturday 15 May 2010 04:19:42 pm Lionel Landwerlin wrote:
> The endianness configuration of your processor isn't something you can
> usually change 'on-the-fly'. It's usually set up early in the boot
> process.
> So you have to choose whether you want to compile all your system in big
> or little endian, you can select that from the buildroot architecture
> configuration (arm -> little, armeb -> big, for example).

That's clear. 
My board is being developed from an environment I got from in-circuit.
arm is selected (not armeb) and indeed the packages are compiled in little 
endian format. 

Are there any additional items I should be aware of when switching from arm to 
armeb ?
 
> > Is there any way to compile my package in big-endian mode from buildroot
> > ? Or is there another way I should force this ?
> 
> You can't select that for 1 package, it's for the whole system or
> nothing.
> Otherwise, the smartest approch would be to make endian detection (at
> compile time or at running time) to adapt your processing algorithm.

Well, the biggest issue I have is that my data that gets send out to the host 
PC must be in big-endian format. My drivers work this way too so the best 
would be if everything is big-endian so that I can transfer data from my 
drivers, through the package to the host PC without touching the data.
I could leave everything in little endian format but I would than have to 
reformat my data before sending it out. Although this is not all that difficult 
I don't like to waste time in that process.

Thanks for your reply,
Marcel