[Buildroot] Question about 64Bit kernel and 32Bit applications

[Buildroot] Question about 64Bit kernel and 32Bit applications

[Ronny Meeus]

Hello

I'm using buildroot to build an environment for MIPS (Cavium).
For this I use a 64 bit toolchain provided by Cavium.

My buildroot configuration is as follows:
* Buildroot 2012.05
* Target Architecture: MIPS (big endian)
* Target Architecture Variant: mips 64
* Target ABI: n64
* glibc

Building and deploying the kernel / busybox in this configuration
works without issues.

I want to deploy on this configuration "legacy" applications that need
to run in 32 bit mode.
So in fact I want to have a mixture of 32 and 64 bit
(busybox,dropbear,...) applications to be present on the board.

Is this supported in buildroot? If it is, how can this be configured?

What I already did is changing the .mk files in the package directory
to pass the correct options to build 32 bit variants to the individual
libraries (libUIO / xenomai-forge / libfuse) I want to use.
This works, but I consider it as a workaround since I only have 32 bit
variants of these libraries. To be correct, I should have both a 32
and a 64 bit version of each of these libraries.
I also need to adapt all the packages I want to include, which is
something I would like to configure at buildroot level.

This is what I changed for libfuse for example:
diff --git a/package/libfuse/libfuse.mk b/package/libfuse/libfuse.mk
--- a/package/libfuse/libfuse.mk
+++ b/package/libfuse/libfuse.mk
@@ -13,7 +13,8 @@ LIBFUSE_CONF_OPT= --disable-nls \
                --disable-example \
                --disable-kernel-module \
                --enable-lib \
-               --enable-util
+               --enable-util \
+               CFLAGS="-mabi=n32 -march=octeon2"
LDFLAGS="-Wl,-melf32btsmipn32" CPPFLAGS="-mabi=n32 -march=octeon2"


On the short term, it would also be fine to have user-land compiled
completely in 32 bit mode while the kernel is compiled in 64 bit mode.
If this is supported in some way, I would be fine for the moment but
in the future we want to run a mixture of both 32 and 64 bit apps.

---
Ronny

[Buildroot] Question about 64Bit kernel and 32Bit applications

[Thomas Petazzoni]

Ronny,

On Tue, 2 Oct 2012 15:57:43 +0200, Ronny Meeus wrote:

> I'm using buildroot to build an environment for MIPS (Cavium).
> For this I use a 64 bit toolchain provided by Cavium.

Just curious, is this toolchain publicly available? I would really like
to have the opportunity to test it and integrate it as an external
toolchain in Buildroot.

> Building and deploying the kernel / busybox in this configuration
> works without issues.
> 
> I want to deploy on this configuration "legacy" applications that need
> to run in 32 bit mode.
> So in fact I want to have a mixture of 32 and 64 bit
> (busybox,dropbear,...) applications to be present on the board.
> 
> Is this supported in buildroot? If it is, how can this be configured?

No, generating a userland with a mix of libraries with different ABIs
is not supported.

And my opinion is that we should not try to support this. Unless I'm
missing something, it is a really bizarre use case, and adding support
for it would create an horrible additional complexity in Buildroot.

The change you did for libfuse cannot be integrated: it is MIPS
specific, and we also have no way of specifying on a per-package basis
whether it should be built for 32 bits or 64 bits.

Best regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] Question about 64Bit kernel and 32Bit applications

[Thomas Petazzoni]

Ronny,

Please keep the list Cc'ed for Buildroot related discussions.

On Tue, 2 Oct 2012 19:43:46 +0200, Ronny Meeus wrote:

> The toolchain is available from cnusers.org.
> You need to register first but once registered (can take 1 day or so), go to:
> Downloads -> Downloads for registered users -> Octeon Software
> development kit -> SDK 2.3
> This is the complete octeon SDK including an installed toochain, Linux
> source code, example code, uboot etc for the Cavium reference boards.
> This is the direct link:
> http://www.cnusers.org/index.php?option=com_remository&Itemid=32&func=fileinfo&id=165

Ok, thanks, I'll have a look. However, if registration is required,
integrating the toolchain in Buildroot will be difficult.

> > And my opinion is that we should not try to support this. Unless I'm
> > missing something, it is a really bizarre use case, and adding support
> > for it would create an horrible additional complexity in Buildroot.
> 
> In my opinion this usecase is not so bizarre. New applications can be
> compiled in 64Bit mode while legacy applications (coming from a 32Bit
> machine) can be compiled with the -mabi=n32 compiler option. If we
> would not do this and just compile in 64Bit mode, the impact in the
> application can be big (size of several types changes from 4 to 8
> bytes, impact on the size of data-types, alignment issues etc).
> 
> What about the support to compile the userland with a different ABI
> than the kernel?

Sure. Do you know what changes we should do to make this possible?

> > The change you did for libfuse cannot be integrated: it is MIPS
> > specific, and we also have no way of specifying on a per-package basis
> > whether it should be built for 32 bits or 64 bits.
> 
> I just included this code to give you an indication of the changes I
> needed to do ...

Sure, that's the way I took your code, I just highlighted the reason
why we couldn't accept this type of modification.

Thanks!

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] Question about 64Bit kernel and 32Bit applications

[Thomas De Schampheleire]

Hi,

On Tue, Oct 2, 2012 at 7:49 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Ronny,
>
> Please keep the list Cc'ed for Buildroot related discussions.
>
> On Tue, 2 Oct 2012 19:43:46 +0200, Ronny Meeus wrote:
>
>> The toolchain is available from cnusers.org.
>> You need to register first but once registered (can take 1 day or so), go to:
>> Downloads -> Downloads for registered users -> Octeon Software
>> development kit -> SDK 2.3
>> This is the complete octeon SDK including an installed toochain, Linux
>> source code, example code, uboot etc for the Cavium reference boards.
>> This is the direct link:
>> http://www.cnusers.org/index.php?option=com_remository&Itemid=32&func=fileinfo&id=165
>
> Ok, thanks, I'll have a look. However, if registration is required,
> integrating the toolchain in Buildroot will be difficult.
>
>> > And my opinion is that we should not try to support this. Unless I'm
>> > missing something, it is a really bizarre use case, and adding support
>> > for it would create an horrible additional complexity in Buildroot.
>>
>> In my opinion this usecase is not so bizarre. New applications can be
>> compiled in 64Bit mode while legacy applications (coming from a 32Bit
>> machine) can be compiled with the -mabi=n32 compiler option. If we
>> would not do this and just compile in 64Bit mode, the impact in the
>> application can be big (size of several types changes from 4 to 8
>> bytes, impact on the size of data-types, alignment issues etc).

(note: I'm working together with Ronny so my opinions are biased)

I also think that this use case is valid: on a 64 bit systems you may
need to support 32-bit legacy applications. The end goal obviously is
to fix the applications so that they truly support 64-bit, but this is
not a quick fix. In the short term, a mixed approach would be needed.
It's true that this may be a new use case for Buildroot, and I also
understand and agree that we shouldn't support this if it makes the
implementation ugly or overcomplicated. However, I hope that it's
possible with a clean and simple implementation, which ideally is also
integrateable in mainline Buildroot. If not possible, we'll have to
keep it in our local tree.

>>
>> What about the support to compile the userland with a different ABI
>> than the kernel?
>
> Sure. Do you know what changes we should do to make this possible?
>
>> > The change you did for libfuse cannot be integrated: it is MIPS
>> > specific, and we also have no way of specifying on a per-package basis
>> > whether it should be built for 32 bits or 64 bits.

In fact, if we are to support a mixed 32/64 bit system, there are
several levels:
- specify for some packages/libraries whether it should be built for
this or that bitness/ABI but only build one combination in a given
environment.
- build certain libraries for more than one bitness/ABI combination in
the same environment, e.g. 32 and 64-bit. This way, the same library
can be used by both 32- as 64-bit applications.
- build all libraries for more than one bitness/ABI if a given global
option is selected

The first option is probably the easiest. One could envision a global
option that specifies the deviant CFLAGS and LDFLAGS to use, and
another option to list the packages that should use these deviant
flags.
Or another solution would be inspired by the local.mk feature: in this
case one could create a single file that specifies entries like
<PKG>_DEVIANT_FLAGS="..."
The impact of such a solution in the central infrastructure is limited.

>>
>> I just included this code to give you an indication of the changes I
>> needed to do ...
>
> Sure, that's the way I took your code, I just highlighted the reason
> why we couldn't accept this type of modification.
>

Aside from the selection of bitness/ABI for certain packages built by
buildroot, there is the question of libraries part of the (external)
sysroot. The C library would obviously need to be present on the
rootfs in both 32- and 64-bit versions.

Best regards,
Thomas

[Buildroot] Question about 64Bit kernel and 32Bit applications

[Ronny Meeus]

On Tue, Oct 2, 2012 at 7:49 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Ronny,
>
> Please keep the list Cc'ed for Buildroot related discussions.
>
> On Tue, 2 Oct 2012 19:43:46 +0200, Ronny Meeus wrote:
>
>> The toolchain is available from cnusers.org.
>> You need to register first but once registered (can take 1 day or so), go to:
>> Downloads -> Downloads for registered users -> Octeon Software
>> development kit -> SDK 2.3
>> This is the complete octeon SDK including an installed toochain, Linux
>> source code, example code, uboot etc for the Cavium reference boards.
>> This is the direct link:
>> http://www.cnusers.org/index.php?option=com_remository&Itemid=32&func=fileinfo&id=165
>
> Ok, thanks, I'll have a look. However, if registration is required,
> integrating the toolchain in Buildroot will be difficult.
>
>> > And my opinion is that we should not try to support this. Unless I'm
>> > missing something, it is a really bizarre use case, and adding support
>> > for it would create an horrible additional complexity in Buildroot.
>>
>> In my opinion this usecase is not so bizarre. New applications can be
>> compiled in 64Bit mode while legacy applications (coming from a 32Bit
>> machine) can be compiled with the -mabi=n32 compiler option. If we
>> would not do this and just compile in 64Bit mode, the impact in the
>> application can be big (size of several types changes from 4 to 8
>> bytes, impact on the size of data-types, alignment issues etc).
>>
>> What about the support to compile the userland with a different ABI
>> than the kernel?
>
> Sure. Do you know what changes we should do to make this possible?

What I have done is the following:
- pass following flags to the compiler (CC and CPP), for 32Bit
applications/libraries: -mabi=n32 -march=octeon2
- pass following flags to the linker, for 32Bit
applications/libraries:  -melf32btsmipn32
- copy the lib32 and lib64 directories of the sysroot directory of the
toolchain to buildroot and the target/output directory.
- do not pass the -mtune, -mabi and -m64 in
"toolchain/toolchain-external/ext-toolchain-wrapper.c"

With the last step I take the assumption that the default mode for the
toolchain is 64Bit.

I do not know how exactly to fit it into buildroot.
One suggestion could be to use the current buildroot flags for the
application space and introduce a new set of flags for the kernel.
This is under the assumption that all libraries/packages work will for
the n32 ABI.

In the meantime I tried to compile with the target ABI set to n32 and
all code seems to after manually copying following directories:
cp -R /tmp/tools/mips64-octeon-linux-gnu/sys-root/usr/lib64
/repo/meeusr/buildroot-cavium-5/output/host/usr/mips-unknown-linux-gnu/sysroot/usr/
cp -R /tmp/tools/mips64-octeon-linux-gnu/sys-root/usr/lib32
/repo/meeusr/buildroot-cavium-5/output/host/usr/mips-unknown-linux-gnu/sysroot/usr/
cp -R /tmp/tools/mips64-octeon-linux-gnu/sys-root/lib
/repo/meeusr/buildroot-cavium-5/output/host/usr/mips-unknown-linux-gnu/sysroot
cp -R /tmp/tools/mips64-octeon-linux-gnu/sys-root/lib32
/repo/meeusr/buildroot-cavium-5/output/host/usr/mips-unknown-linux-gnu/sysroot
cp -R /tmp/tools/mips64-octeon-linux-gnu/sys-root/lib64
/repo/meeusr/buildroot-cavium-5/output/host/usr/mips-unknown-linux-gnu/sysroot
(the toolchain is also installed in /tmp/tools)

After doing this, I'm able to build everything and I see that all
applications/libraries are in the correct ABI (N32).
Even the kernel is correctly compiled (64Bit). After some grepping in
the kernel code I found:

$ grep -r "\-mabi\=64" *
arch/mips/Makefile:cflags-y             += -mabi=64
drivers/staging/rt2860/config.mk:
-mabi=64 $(WFLAGS)

So it looks like the model were the complete userspace is compiled
with n32 works correctly.
[I still had to copy the lib32 directory myself to the "target/output"
directory make the system boot.]

>> > The change you did for libfuse cannot be integrated: it is MIPS
>> > specific, and we also have no way of specifying on a per-package basis
>> > whether it should be built for 32 bits or 64 bits.
>>
>> I just included this code to give you an indication of the changes I
>> needed to do ...
>
> Sure, that's the way I took your code, I just highlighted the reason
> why we couldn't accept this type of modification.
>
> Thanks!
>
> Thomas
> --
> Thomas Petazzoni, Free Electrons
> Kernel, drivers, real-time and embedded Linux
> development, consulting, training and support.
> http://free-electrons.com

[Buildroot] Question about 64Bit kernel and 32Bit applications

[Ronny Meeus]

Another solution would be to have 2 buildroot environments, 1 for the
32bit applications and 1 for the 64bit applications.
For example the 64bit could be "the master" one. In this a directory
of the slave buildroot can be specified and this master will merge the
32bit applications together with its libraries from the slave
output/target into its own target/output and remake the rootfs.

This might not be a nice solution, but I think it will work.

It can be considered similar to BR2_TOOLCHAIN_EXTRA_EXTERNAL_LIBS
where a copy is made of certain things into the root-fs.

On Tue, Oct 2, 2012 at 9:21 PM, Ronny Meeus <ronny.meeus@gmail.com> wrote:
> On Tue, Oct 2, 2012 at 7:49 PM, Thomas Petazzoni
> <thomas.petazzoni@free-electrons.com> wrote:
>> Ronny,
>>
>> Please keep the list Cc'ed for Buildroot related discussions.
>>
>> On Tue, 2 Oct 2012 19:43:46 +0200, Ronny Meeus wrote:
>>
>>> The toolchain is available from cnusers.org.
>>> You need to register first but once registered (can take 1 day or so), go to:
>>> Downloads -> Downloads for registered users -> Octeon Software
>>> development kit -> SDK 2.3
>>> This is the complete octeon SDK including an installed toochain, Linux
>>> source code, example code, uboot etc for the Cavium reference boards.
>>> This is the direct link:
>>> http://www.cnusers.org/index.php?option=com_remository&Itemid=32&func=fileinfo&id=165
>>
>> Ok, thanks, I'll have a look. However, if registration is required,
>> integrating the toolchain in Buildroot will be difficult.
>>
>>> > And my opinion is that we should not try to support this. Unless I'm
>>> > missing something, it is a really bizarre use case, and adding support
>>> > for it would create an horrible additional complexity in Buildroot.
>>>
>>> In my opinion this usecase is not so bizarre. New applications can be
>>> compiled in 64Bit mode while legacy applications (coming from a 32Bit
>>> machine) can be compiled with the -mabi=n32 compiler option. If we
>>> would not do this and just compile in 64Bit mode, the impact in the
>>> application can be big (size of several types changes from 4 to 8
>>> bytes, impact on the size of data-types, alignment issues etc).
>>>
>>> What about the support to compile the userland with a different ABI
>>> than the kernel?
>>
>> Sure. Do you know what changes we should do to make this possible?
>
> What I have done is the following:
> - pass following flags to the compiler (CC and CPP), for 32Bit
> applications/libraries: -mabi=n32 -march=octeon2
> - pass following flags to the linker, for 32Bit
> applications/libraries:  -melf32btsmipn32
> - copy the lib32 and lib64 directories of the sysroot directory of the
> toolchain to buildroot and the target/output directory.
> - do not pass the -mtune, -mabi and -m64 in
> "toolchain/toolchain-external/ext-toolchain-wrapper.c"
>
> With the last step I take the assumption that the default mode for the
> toolchain is 64Bit.
>
> I do not know how exactly to fit it into buildroot.
> One suggestion could be to use the current buildroot flags for the
> application space and introduce a new set of flags for the kernel.
> This is under the assumption that all libraries/packages work will for
> the n32 ABI.
>
> In the meantime I tried to compile with the target ABI set to n32 and
> all code seems to after manually copying following directories:
> cp -R /tmp/tools/mips64-octeon-linux-gnu/sys-root/usr/lib64
> /repo/meeusr/buildroot-cavium-5/output/host/usr/mips-unknown-linux-gnu/sysroot/usr/
> cp -R /tmp/tools/mips64-octeon-linux-gnu/sys-root/usr/lib32
> /repo/meeusr/buildroot-cavium-5/output/host/usr/mips-unknown-linux-gnu/sysroot/usr/
> cp -R /tmp/tools/mips64-octeon-linux-gnu/sys-root/lib
> /repo/meeusr/buildroot-cavium-5/output/host/usr/mips-unknown-linux-gnu/sysroot
> cp -R /tmp/tools/mips64-octeon-linux-gnu/sys-root/lib32
> /repo/meeusr/buildroot-cavium-5/output/host/usr/mips-unknown-linux-gnu/sysroot
> cp -R /tmp/tools/mips64-octeon-linux-gnu/sys-root/lib64
> /repo/meeusr/buildroot-cavium-5/output/host/usr/mips-unknown-linux-gnu/sysroot
> (the toolchain is also installed in /tmp/tools)
>
> After doing this, I'm able to build everything and I see that all
> applications/libraries are in the correct ABI (N32).
> Even the kernel is correctly compiled (64Bit). After some grepping in
> the kernel code I found:
>
> $ grep -r "\-mabi\=64" *
> arch/mips/Makefile:cflags-y             += -mabi=64
> drivers/staging/rt2860/config.mk:
> -mabi=64 $(WFLAGS)
>
> So it looks like the model were the complete userspace is compiled
> with n32 works correctly.
> [I still had to copy the lib32 directory myself to the "target/output"
> directory make the system boot.]
>
>>> > The change you did for libfuse cannot be integrated: it is MIPS
>>> > specific, and we also have no way of specifying on a per-package basis
>>> > whether it should be built for 32 bits or 64 bits.
>>>
>>> I just included this code to give you an indication of the changes I
>>> needed to do ...
>>
>> Sure, that's the way I took your code, I just highlighted the reason
>> why we couldn't accept this type of modification.
>>
>> Thanks!
>>
>> Thomas
>> --
>> Thomas Petazzoni, Free Electrons
>> Kernel, drivers, real-time and embedded Linux
>> development, consulting, training and support.
>> http://free-electrons.com

[Buildroot] Question about 64Bit kernel and 32Bit applications

[Ronny Meeus]

Thomas,

to be able to use the Cavium toolchain I had to do some manual things.
Since now the buildroot will be used by more people I want to get rid
of these manual actions.
I will explain what needs to be done...

This is what is included in the Cavium toolchain:

$ ls -l /repo/meeusr/buildroot-cavium-6/output/host/opt/ext-toolchain/mips64-octeon-linux-gnu/sys-root/
total 28
drwxr-xr-x  2 meeusr we2 4096 Sep 11 15:34 etc/
drwxr-xr-x  2 meeusr we2 4096 Feb 24  2012 lib/
drwxr-xr-x  3 meeusr we2 4096 Sep 11 15:34 lib32/
drwxr-xr-x  3 meeusr we2 4096 Sep 11 15:34 lib64/
drwxr-xr-x  2 meeusr we2 4096 Sep 11 15:34 sbin/
drwxr-xr-x  5 meeusr we2 4096 Sep 11 15:34 uclibc/
drwxr-xr-x 13 meeusr we2 4096 Sep 11 15:34 usr/

The lib directory is empty:

$ ls -l /repo/meeusr/buildroot-cavium-6/output/host/opt/ext-toolchain/mips64-octeon-linux-gnu/sys-root/lib
total 0


Because of this I get following error in buildroot:

mkdir -p /repo/meeusr/buildroot-cavium-6/output/host/opt/ext-toolchain
gzip -d -c /repo/meeusr/buildroot-cavium-6/dl/OCTEON-SDK-2.3-tools.tgz | \

/repo/meeusr/buildroot-cavium-6/output/host/usr/bin/tar
--strip-components=1 --exclude='usr/lib/locale/*' -C
/repo/meeusr/buildroot-cavium-6/output/host/opt/ext-toolchain  -xf -
Checking external toolchain settings
Incorrect selection of the C library
make: *** [/repo/meeusr/buildroot-cavium-6/output/stamps/ext-toolchain-checked]
Error 255


To be able to use the external toolchain I need to create a link from
lib to lib32.

$ rmdir /repo/meeusr/buildroot-cavium-6/output/host/opt/ext-toolchain/mips64-octeon-linux-gnu/sys-root/lib
$ ln -s lib32 /repo/meeusr/buildroot-cavium-6/output/host/opt/ext-toolchain/mips64-octeon-linux-gnu/sys-root/lib
$ ls -l /repo/meeusr/buildroot-cavium-6/output/host/opt/ext-toolchain/mips64-octeon-linux-gnu/sys-root/
total 24
drwxr-xr-x  2 meeusr we2 4096 Sep 11 15:34 etc/
lrwxrwxrwx  1 meeusr we2    5 Oct  3 16:25 lib -> lib32/
drwxr-xr-x  3 meeusr we2 4096 Oct  3 16:25 lib32/
drwxr-xr-x  3 meeusr we2 4096 Sep 11 15:34 lib64/
drwxr-xr-x  2 meeusr we2 4096 Sep 11 15:34 sbin/
drwxr-xr-x  5 meeusr we2 4096 Sep 11 15:34 uclibc/
drwxr-xr-x 13 meeusr we2 4096 Sep 11 15:34 usr/

After doing this buildroot is happy and starts to compile busybox.
During the linking of bosybox I get:

/repo/meeusr/buildroot-cavium-6/output/host/opt/ext-toolchain/bin/../lib/gcc/mips64-octeon-linux-gnu/4.3.3/../../../../mips64-octeon-linux-gnu/bin/ld:
cannot find /lib32/libc.so.6 inside
/repo/meeusr/buildroot-cavium-6/output/host/usr/mips-unknown-linux-gnu/sysroot
collect2: ld returned 1 exit status
make[1]: *** [busybox_unstripped] Error 1
make[1]: Leaving directory
`/repo/meeusr/buildroot-cavium-6/output/build/busybox-1.20.1'

When I create a link lib32 to lib in
output/host/usr/mips-unknown-linux-gnu/sysroot

$ ls -l output/host/usr/mips-unknown-linux-gnu/sysroot
total 16
drwxr-xr-x 2 meeusr we2 4096 Oct  3 16:16 bin/
drwxr-xr-x 4 meeusr we2 4096 Oct  3 16:34 lib/
lrwxrwxrwx 1 meeusr we2    3 Oct  3 16:34 lib32 -> lib/
drwxr-xr-x 3 meeusr we2 4096 Oct  3 16:27 repo/
drwxr-xr-x 5 meeusr we2 4096 Oct  3 16:16 usr/

The problem is still not solved.


I start all over again, but this time creating a copy instead of
working with links.
make clean
make
[failure: Incorrect selection of the C library]
rmdir /repo/meeusr/buildroot-cavium-6/output/host/opt/ext-toolchain/mips64-octeon-linux-gnu/sys-root/lib
cp -R /repo/meeusr/buildroot-cavium-6/output/host/opt/ext-toolchain/mips64-octeon-linux-gnu/sys-root/lib32
/repo/meeusr/buildroot-cavium-6/output/host/opt/ext-toolchain/mips64-octeon-linux-gnu/sys-root/lib
make
[now failure (Incorrect selection of the C library) is again gone]
[now link failure in busybox]

$ cp -R /repo/meeusr/buildroot-cavium-6/output/host/opt/ext-toolchain/mips64-octeon-linux-gnu/sys-root/lib32
 /repo/meeusr/buildroot-cavium-6/output/host/usr/mips-unknown-linux-gnu/sysroot

Now I have a different error during the link of busybox:
cannot find /usr/lib32/libc_nonshared.a inside
/repo/meeusr/buildroot-cavium-6/output/host/usr/mips-unknown-linux-gnu/sysroot]

Now I copy usr/lib32:
$ cp -R /repo/meeusr/buildroot-cavium-6/output/host/opt/ext-toolchain/mips64-octeon-linux-gnu/sys-root/usr/lib32
/repo/meeusr/buildroot-cavium-6/output/host/usr/mips-unknown-linux-gnu/sysroot/usr

After this the compilation of busybox and the rest of the modules is
working well.
To be able to use the resulting image on the target, I also need to
copy the lib32 directory into the target/output.

So it looks like if the structure of the external toolchain is
organised with lib32 and lib64 buildroot is not able to handle this
correctly.


Any suggestions on how to solve this issue would be welcome.

Thanks.

On Tue, Oct 2, 2012 at 10:46 PM, Ronny Meeus <ronny.meeus@gmail.com> wrote:
> Another solution would be to have 2 buildroot environments, 1 for the
> 32bit applications and 1 for the 64bit applications.
> For example the 64bit could be "the master" one. In this a directory
> of the slave buildroot can be specified and this master will merge the
> 32bit applications together with its libraries from the slave
> output/target into its own target/output and remake the rootfs.
>
> This might not be a nice solution, but I think it will work.
>
> It can be considered similar to BR2_TOOLCHAIN_EXTRA_EXTERNAL_LIBS
> where a copy is made of certain things into the root-fs.
>
> On Tue, Oct 2, 2012 at 9:21 PM, Ronny Meeus <ronny.meeus@gmail.com> wrote:
>> On Tue, Oct 2, 2012 at 7:49 PM, Thomas Petazzoni
>> <thomas.petazzoni@free-electrons.com> wrote:
>>> Ronny,
>>>
>>> Please keep the list Cc'ed for Buildroot related discussions.
>>>
>>> On Tue, 2 Oct 2012 19:43:46 +0200, Ronny Meeus wrote:
>>>
>>>> The toolchain is available from cnusers.org.
>>>> You need to register first but once registered (can take 1 day or so), go to:
>>>> Downloads -> Downloads for registered users -> Octeon Software
>>>> development kit -> SDK 2.3
>>>> This is the complete octeon SDK including an installed toochain, Linux
>>>> source code, example code, uboot etc for the Cavium reference boards.
>>>> This is the direct link:
>>>> http://www.cnusers.org/index.php?option=com_remository&Itemid=32&func=fileinfo&id=165
>>>
>>> Ok, thanks, I'll have a look. However, if registration is required,
>>> integrating the toolchain in Buildroot will be difficult.
>>>
>>>> > And my opinion is that we should not try to support this. Unless I'm
>>>> > missing something, it is a really bizarre use case, and adding support
>>>> > for it would create an horrible additional complexity in Buildroot.
>>>>
>>>> In my opinion this usecase is not so bizarre. New applications can be
>>>> compiled in 64Bit mode while legacy applications (coming from a 32Bit
>>>> machine) can be compiled with the -mabi=n32 compiler option. If we
>>>> would not do this and just compile in 64Bit mode, the impact in the
>>>> application can be big (size of several types changes from 4 to 8
>>>> bytes, impact on the size of data-types, alignment issues etc).
>>>>
>>>> What about the support to compile the userland with a different ABI
>>>> than the kernel?
>>>
>>> Sure. Do you know what changes we should do to make this possible?
>>
>> What I have done is the following:
>> - pass following flags to the compiler (CC and CPP), for 32Bit
>> applications/libraries: -mabi=n32 -march=octeon2
>> - pass following flags to the linker, for 32Bit
>> applications/libraries:  -melf32btsmipn32
>> - copy the lib32 and lib64 directories of the sysroot directory of the
>> toolchain to buildroot and the target/output directory.
>> - do not pass the -mtune, -mabi and -m64 in
>> "toolchain/toolchain-external/ext-toolchain-wrapper.c"
>>
>> With the last step I take the assumption that the default mode for the
>> toolchain is 64Bit.
>>
>> I do not know how exactly to fit it into buildroot.
>> One suggestion could be to use the current buildroot flags for the
>> application space and introduce a new set of flags for the kernel.
>> This is under the assumption that all libraries/packages work will for
>> the n32 ABI.
>>
>> In the meantime I tried to compile with the target ABI set to n32 and
>> all code seems to after manually copying following directories:
>> cp -R /tmp/tools/mips64-octeon-linux-gnu/sys-root/usr/lib64
>> /repo/meeusr/buildroot-cavium-5/output/host/usr/mips-unknown-linux-gnu/sysroot/usr/
>> cp -R /tmp/tools/mips64-octeon-linux-gnu/sys-root/usr/lib32
>> /repo/meeusr/buildroot-cavium-5/output/host/usr/mips-unknown-linux-gnu/sysroot/usr/
>> cp -R /tmp/tools/mips64-octeon-linux-gnu/sys-root/lib
>> /repo/meeusr/buildroot-cavium-5/output/host/usr/mips-unknown-linux-gnu/sysroot
>> cp -R /tmp/tools/mips64-octeon-linux-gnu/sys-root/lib32
>> /repo/meeusr/buildroot-cavium-5/output/host/usr/mips-unknown-linux-gnu/sysroot
>> cp -R /tmp/tools/mips64-octeon-linux-gnu/sys-root/lib64
>> /repo/meeusr/buildroot-cavium-5/output/host/usr/mips-unknown-linux-gnu/sysroot
>> (the toolchain is also installed in /tmp/tools)
>>
>> After doing this, I'm able to build everything and I see that all
>> applications/libraries are in the correct ABI (N32).
>> Even the kernel is correctly compiled (64Bit). After some grepping in
>> the kernel code I found:
>>
>> $ grep -r "\-mabi\=64" *
>> arch/mips/Makefile:cflags-y             += -mabi=64
>> drivers/staging/rt2860/config.mk:
>> -mabi=64 $(WFLAGS)
>>
>> So it looks like the model were the complete userspace is compiled
>> with n32 works correctly.
>> [I still had to copy the lib32 directory myself to the "target/output"
>> directory make the system boot.]
>>
>>>> > The change you did for libfuse cannot be integrated: it is MIPS
>>>> > specific, and we also have no way of specifying on a per-package basis
>>>> > whether it should be built for 32 bits or 64 bits.
>>>>
>>>> I just included this code to give you an indication of the changes I
>>>> needed to do ...
>>>
>>> Sure, that's the way I took your code, I just highlighted the reason
>>> why we couldn't accept this type of modification.
>>>
>>> Thanks!
>>>
>>> Thomas
>>> --
>>> Thomas Petazzoni, Free Electrons
>>> Kernel, drivers, real-time and embedded Linux
>>> development, consulting, training and support.
>>> http://free-electrons.com

[Buildroot] Question about 64Bit kernel and 32Bit applications

[Arnout Vandecappelle]

On 02/10/12 22:46, Ronny Meeus wrote:
> Another solution would be to have 2 buildroot environments, 1 for the
> 32bit applications and 1 for the 64bit applications.
> For example the 64bit could be "the master" one. In this a directory
> of the slave buildroot can be specified and this master will merge the
> 32bit applications together with its libraries from the slave
> output/target into its own target/output and remake the rootfs.
>
> This might not be a nice solution, but I think it will work.
>
> It can be considered similar to BR2_TOOLCHAIN_EXTRA_EXTERNAL_LIBS
> where a copy is made of certain things into the root-fs.

  First of all, I'm not at all convinced that the multilib usecase is valid
for buildroot.  I mean, obviously it's a valid usecase, but if you need
that I don't think that buildroot should be your weapon of choice.

  That said, if you (temporarily) do need a mixed 32/64-bit system, I think
this approach is the most appropriate: make two separate buildroot builds,
and merge them together at the end.  The merge can be done in the post-build
script.  The post-build script could even call the'make' for the other part
  of the system...  The tricky part is that the libraries from the 'slave'
environment will have to be moved into a multilib directory (i.e. lib32 or
lib64).

  I don't immediately see a need for changes in the buildroot infrastructure.

  Regards,
  Arnout
-- 
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] Question about 64Bit kernel and 32Bit applications

[Thomas De Schampheleire]

On Thu, Oct 4, 2012 at 11:08 AM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On 02/10/12 22:46, Ronny Meeus wrote:
>>
>> Another solution would be to have 2 buildroot environments, 1 for the
>> 32bit applications and 1 for the 64bit applications.
>> For example the 64bit could be "the master" one. In this a directory
>> of the slave buildroot can be specified and this master will merge the
>> 32bit applications together with its libraries from the slave
>> output/target into its own target/output and remake the rootfs.
>>
>> This might not be a nice solution, but I think it will work.
>>
>> It can be considered similar to BR2_TOOLCHAIN_EXTRA_EXTERNAL_LIBS
>> where a copy is made of certain things into the root-fs.
>
>
>  First of all, I'm not at all convinced that the multilib usecase is valid
> for buildroot.  I mean, obviously it's a valid usecase, but if you need
> that I don't think that buildroot should be your weapon of choice.
>
>  That said, if you (temporarily) do need a mixed 32/64-bit system, I think
> this approach is the most appropriate: make two separate buildroot builds,
> and merge them together at the end.  The merge can be done in the post-build
> script.  The post-build script could even call the'make' for the other part
>  of the system...  The tricky part is that the libraries from the 'slave'
> environment will have to be moved into a multilib directory (i.e. lib32 or
> lib64).
>
>  I don't immediately see a need for changes in the buildroot infrastructure.
>

The downside with this approach is that there will be a lot of
duplicate builds. You only need 32-bit versions of the libraries. All
the rest of userspace would be 64-bit only. To achieve that, you'd
have two separate defconfigs, one with only libraries, 32-bit, and one
with everything, 64-bit.
However, still with that configuration a lot is duplicated: all the
host builds like host-automake, host-autoconf, ... would be created in
each buildroot instance.

I like following approaches better:
- 64-bit kernel, entire 32-bit userspace
- 64-bit kernel, mostly 64-bit userspace, but selected libraries built
only in 32-bit (e.g. using a local.mk-like file to specify)
- 64-bit kernel, 64-bit applications, 64+32bit libraries (true multilib).

In all of these cases, you will probably need to look into the
toolchain extraction, as you'll need the correct sysroot lib
directories (possibly multiple).

Best regards,
Thomas

[Buildroot] Question about 64Bit kernel and 32Bit applications

[Arnout Vandecappelle]

On 04/10/12 12:56, Thomas De Schampheleire wrote:
> On Thu, Oct 4, 2012 at 11:08 AM, Arnout Vandecappelle<arnout@mind.be>  wrote:
>>   That said, if you (temporarily) do need a mixed 32/64-bit system, I think
>> this approach is the most appropriate: make two separate buildroot builds,
>> and merge them together at the end.  The merge can be done in the post-build
>> script.  The post-build script could even call the'make' for the other part
>>   of the system...  The tricky part is that the libraries from the 'slave'
>> environment will have to be moved into a multilib directory (i.e. lib32 or
>> lib64).
>>
>>   I don't immediately see a need for changes in the buildroot infrastructure.
>>
>
> The downside with this approach is that there will be a lot of
> duplicate builds. You only need 32-bit versions of the libraries. All
> the rest of userspace would be 64-bit only. To achieve that, you'd
> have two separate defconfigs, one with only libraries, 32-bit, and one
> with everything, 64-bit.
> However, still with that configuration a lot is duplicated: all the
> host builds like host-automake, host-autoconf, ... would be created in
> each buildroot instance.
>
> I like following approaches better:
> - 64-bit kernel, entire 32-bit userspace
> - 64-bit kernel, mostly 64-bit userspace, but selected libraries built
> only in 32-bit (e.g. using a local.mk-like file to specify)

  Actually, you can put the following in local.mk:

$(BUILD_DIR)/foo-$(FOO_VERSION)/.stamp_configured: TARGET_CFLAGS = $(TARGET_CFLAGS) -m32

(and similarly for .stamp_built etc.)

> - 64-bit kernel, 64-bit applications, 64+32bit libraries (true multilib).

  That would probably mean you need to do something like the host-pkg
infrastructure.


> In all of these cases, you will probably need to look into the
> toolchain extraction, as you'll need the correct sysroot lib
> directories (possibly multiple).

  And you either have to make it depend on an external toolchain that has
to be multilib (so you somehow have to check that the external toolchain is
indeed multilib), or you have to add the possibility to generate a multilib
buildroot / ct-ng toolchain.

  I think the toolchain aspect is the biggest problem.  Toolchains are already
the biggest mess in buildroot, I'd hate to add to their complexity.

  Regards,
  Arnout
-- 
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F