Building and using a second toolchain

Building and using a second toolchain

[Mike Crowe]

I'm trying to compile a vendor bootloader for an AArch64-based chip. This
bootloader requires both a 32-bit "arm" compiler and a 64-bit "aarch64"
compiler to build. The rest of the system only requires the aarch64
compiler, so I'm using arch-armv8.inc in my machine configuration.

Since this is for a bootloader, the 32-bit compiler can probably be the
equivalent of gcc-cross-initial, and of course I need binutils-cross too.

Inspired by native.bbclass, I produced binutils-cross-arm and
gcc-cross-initial-arm recipes that rewrite lots of TARGET_ variables and
then copied the handful of require lines from the standard binutils-cross
and gcc-cross-initial recipes. This seems to work well - the binaries end
up in sysroots-components as would be expected.

I then added gcc-cross-initial-arm to DEPENDS for the bootloader recipe.
Unfortunately, things seem to go wrong when populating the recipe-specific
sysroot as the secondary toolchain is not present. I get messages like:

 WARNING: bootloader-7 do_prepare_recipe_sysroot: Manifest .../tmp-glibc/sstate-control/manifest-x86_64_aarch64-gcc-cross-initial-arm.populate_sysroot not found?
 WARNING: bootloader-7 do_prepare_recipe_sysroot: Manifest .../tmp-glibc/sstate-control/manifest-x86_64_aarch64-binutils-cross-arm.populate_sysroot not found?

These manifest files do exist though:

 tmp-glibc/sstate-control/manifest-x86_64_arm-gcc-cross-initial-arm.populate_sysroot
 tmp-glibc/sstate-control/manifest-x86_64_arm-binutils-cross-arm.populate_sysroot

These filenames appear to come from staging.bbclass and are generated from:

 manifest-${BUILD_ARCH}_${TARGET_ARCH}-%s.populate_sysroot

It thought that the simplest path to making this work would be to ensure
that the manifest files generated during the build have the names that all
other recipes will expect, but this means using the wrong TARGET_ARCH in my
binutils-cross-arm and gcc-cross-initial-arm recipes which I can't do
without also overriding TARGET_SYS and PN at least which definitely isn't
pretty.

I remember reading posts about doing this on the list in the past, but my
search engine skills aren't up to the job of finding anything. Am I missing
an easier way to make this work?

Thanks.

Mike.

Re: Building and using a second toolchain

[Richard Purdie]

On Thu, 2017-09-07 at 12:13 +0100, Mike Crowe wrote:
> It thought that the simplest path to making this work would be to ensure
> that the manifest files generated during the build have the names that all
> other recipes will expect, but this means using the wrong TARGET_ARCH in my
> binutils-cross-arm and gcc-cross-initial-arm recipes which I can't do
> without also overriding TARGET_SYS and PN at least which definitely isn't
> pretty.
> 
> I remember reading posts about doing this on the list in the past, but my
> search engine skills aren't up to the job of finding anything. Am I missing
> an easier way to make this work?

Have you considered using multilib? You could have a 32 bit multilib
defined for the 64 bit system, then just use its 32 bit compiler?

Cheers,

Richard

Re: Building and using a second toolchain

[Mike Crowe]

On Thursday 07 September 2017 at 13:21:09 +0100, Richard Purdie wrote:
> On Thu, 2017-09-07 at 12:13 +0100, Mike Crowe wrote:
> > It thought that the simplest path to making this work would be to ensure
> > that the manifest files generated during the build have the names that all
> > other recipes will expect, but this means using the wrong TARGET_ARCH in my
> > binutils-cross-arm and gcc-cross-initial-arm recipes which I can't do
> > without also overriding TARGET_SYS and PN at least which definitely isn't
> > pretty.
> > 
> > I remember reading posts about doing this on the list in the past, but my
> > search engine skills aren't up to the job of finding anything. Am I missing
> > an easier way to make this work?
> 
> Have you considered using multilib? You could have a 32 bit multilib
> defined for the 64 bit system, then just use its 32 bit compiler?

I'd persuaded myself that multilib was the x86-style -m32/-m64 thing on one
compiler binary and it appeared that ARM didn't do that.

But, now I've read up on
http://www.yoctoproject.org/docs/current/dev-manual/dev-manual.html#combining-multiple-versions-library-files-into-one-image
and played around I see that oe-core multilib support does generate
multiple compiler binaries.

I'm able to generate a second compiler with a machine file containing:

 require conf/machine/include/arm/arch-armv8.inc
 require conf/multilib.conf
 MULTILIBS = "multilib:lib32"
 DEFAULTTUNE_virtclass-multilib-lib32 = "armv7at-neon"

However, what I really want to use is:

 DEFAULTTUNE_virtclass-multilib-lib32 = "cortexa15t"

but this fails in the sanity checker with:

    Toolchain tunings invalid:
    Tuning 'cortexa15t' has no defined features, and cannot be used.

and if I add

 require conf/machine/include/tune-cortexa15.inc

then I get oodles of duplicate-inclusion warnings and I no longer seem to
be generating an aarch64 compiler.

I'm not really sure where the split between arch and tune is supposed to be
in that include directory, so whilst I could probably hack it to work, I
doubt it would be acceptable to you.

So, I went back to using armv7at-neon and managed to generate a compiler.
In my bootloader recipe I added:

 DEPENDS_aarch64_append = "virtual/lib32-arm-oemllib32-linux-gnueabi-gcc"

which results in the compiler ending up in recipe-sysroot-native, but also
ends up with the associated libs32-recipe-sysroot installed outside the work
directory in:

 work/${MACHINE}-oemllib32-linux-gnueabi/bootloader/7-r0/lib32-recipe-sysroot/

which doesn't appear to break anything until I run the clean task on
bootloader and it doesn't get cleaned up. This means that subsequent builds
fail because the files in lib32-recipe-sysroot already exist. :(

(I'm not up-to-date with oe-core master, but I looked through the changes
that I don't have yet and couldn't spot anything that looked like it would
address these problems. If you prefer, I can try to reproduce both with the
current tip of master.)

Your suggestion has been a great help, but it looks like I'm not quite
there yet.

Thanks.

Mike.

Re: Building and using a second toolchain

[Richard Purdie]

On Thu, 2017-09-07 at 18:00 +0100, Mike Crowe wrote:
> On Thursday 07 September 2017 at 13:21:09 +0100, Richard Purdie wrote:
> > 
> > defined for the 64 bit system, then just use its 32 bit compiler?
> I'd persuaded myself that multilib was the x86-style -m32/-m64 thing on one
> compiler binary and it appeared that ARM didn't do that.
> 
> But, now I've read up on
> http://www.yoctoproject.org/docs/current/dev-manual/dev-manual.html#combining-multiple-versions-library-files-into-one-image
> and played around I see that oe-core multilib support does generate
> multiple compiler binaries.
> 
> I'm able to generate a second compiler with a machine file containing:
> 
>  require conf/machine/include/arm/arch-armv8.inc

This is a less walked path on arm than it is in the x86 world. For x86,
tune-corei7 includes tune-i586 for example so it 'stacks' neatly and
you have the various compatible tune options available.

I suspect a glitch in the inheritance order of the arm tune files in
other to make this work. That said, is there much difference in
practise between armv7at-neon and cortexa15t?

>  require conf/multilib.conf
>  MULTILIBS = "multilib:lib32"
>  DEFAULTTUNE_virtclass-multilib-lib32 = "armv7at-neon"
> 
> However, what I really want to use is:
> 
>  DEFAULTTUNE_virtclass-multilib-lib32 = "cortexa15t"
> 
> but this fails in the sanity checker with:
> 
>     Toolchain tunings invalid:
>     Tuning 'cortexa15t' has no defined features, and cannot be used.
> 
> and if I add
> 
>  require conf/machine/include/tune-cortexa15.inc
> 
> then I get oodles of duplicate-inclusion warnings and I no longer seem to
> be generating an aarch64 compiler.
> 
> I'm not really sure where the split between arch and tune is supposed to be
> in that include directory, so whilst I could probably hack it to work, I
> doubt it would be acceptable to you.



> 
> So, I went back to using armv7at-neon and managed to generate a compiler.
> In my bootloader recipe I added:
> 
>  DEPENDS_aarch64_append = "virtual/lib32-arm-oemllib32-linux-gnueabi-gcc"
> 
> which results in the compiler ending up in recipe-sysroot-native, but also
> ends up with the associated libs32-recipe-sysroot installed outside the work
> directory in:
> 
>  work/${MACHINE}-oemllib32-linux-gnueabi/bootloader/7-r0/lib32-recipe-sysroot/
> 
> which doesn't appear to break anything until I run the clean task on
> bootloader and it doesn't get cleaned up. This means that subsequent builds
> fail because the files in lib32-recipe-sysroot already exist. :(
> 
> (I'm not up-to-date with oe-core master, but I looked through the changes
> that I don't have yet and couldn't spot anything that looked like it would
> address these problems. If you prefer, I can try to reproduce both with the
> current tip of master.)
> 
> Your suggestion has been a great help, but it looks like I'm not quite
> there yet.

Glad its at least sort of there. You're actually using the multilib in
a different way to the one it'd normally get used as you're trying to
put both into one build and its only really designed for one or the
other.

What would be "normal" is to build "lib32-bootloader" which is the
lib32 BBCLASSEXTEND'd variant of bootloader. If the 64 bit version
doesn't make sense you can make that raise SkipRecipe for the nonsense
variants.

I'm hoping if you do that the sysroot issue goes away. 

Cheers,

Richard

Re: Building and using a second toolchain

[Mike Crowe]

On Thursday 07 September 2017 at 22:28:56 +0100, Richard Purdie wrote:
> On Thu, 2017-09-07 at 18:00 +0100, Mike Crowe wrote:
> > On Thursday 07 September 2017 at 13:21:09 +0100, Richard Purdie wrote:
> > > 
> > > defined for the 64 bit system, then just use its 32 bit compiler?
> > I'd persuaded myself that multilib was the x86-style -m32/-m64 thing on one
> > compiler binary and it appeared that ARM didn't do that.
> > 
> > But, now I've read up on
> > http://www.yoctoproject.org/docs/current/dev-manual/dev-manual.html#combining-multiple-versions-library-files-into-one-image
> > and played around I see that oe-core multilib support does generate
> > multiple compiler binaries.
> > 
> > I'm able to generate a second compiler with a machine file containing:
> > 
> >  require conf/machine/include/arm/arch-armv8.inc

> I suspect a glitch in the inheritance order of the arm tune files in
> other to make this work. That said, is there much difference in
> practise between armv7at-neon and cortexa15t?

Probably not, but I wanted to make sure I was using exactly the same
options as our vendor when compiling the bootloader. Having said that, I
can always tweak CFLAGS by hand so this isn't really a showstopper.

> Glad its at least sort of there. You're actually using the multilib in
> a different way to the one it'd normally get used as you're trying to
> put both into one build and its only really designed for one or the
> other.

Ah, well that's going to be a problem then. I probably didn't make it
clear, but this is a single bootloader build that uses both compilers. Some
parts get built with the 32-bit compiler and some with the 64-bit compiler,
and then the whole thing gets combined into a single binary. Splitting it
into three separate recipes would mean making larger changes to the vendor
build system than I'd really like.

> What would be "normal" is to build "lib32-bootloader" which is the
> lib32 BBCLASSEXTEND'd variant of bootloader. If the 64 bit version
> doesn't make sense you can make that raise SkipRecipe for the nonsense
> variants.

I was successfully building a few lib32- recipes in order to test the
compiler and, as you say, they all consistently use the lib32-named work
directory.

I can think of two ways round the problem:

1. Add a do_prepare_sysroot_append that moves the lib32- sysroot into the
correct work directory.

2. The simplest solution is to add a do_clean_append to my bootloader
recipe that also cleans up the problem sysroot directory.

The first is probably preferable and I'll have a go at doing that. Neither
is ideal, but anything else looks like it's going to complicate the
multilib class code greatly and possibly break the intended use of the
secondary toolchain.

Thanks for all your help.

Mike.

Re: Building and using a second toolchain

[Richard Purdie]

On Fri, 2017-09-08 at 07:04 +0100, Mike Crowe wrote:
> On Thursday 07 September 2017 at 22:28:56 +0100, Richard Purdie
> wrote:
> > 
> > On Thu, 2017-09-07 at 18:00 +0100, Mike Crowe wrote:
> > I suspect a glitch in the inheritance order of the arm tune files
> > in
> > other to make this work. That said, is there much difference in
> > practise between armv7at-neon and cortexa15t?
> Probably not, but I wanted to make sure I was using exactly the same
> options as our vendor when compiling the bootloader. Having said
> that, I
> can always tweak CFLAGS by hand so this isn't really a showstopper.

Fair enough, one issue at a time I guess! :)

> > Glad its at least sort of there. You're actually using the multilib
> > in
> > a different way to the one it'd normally get used as you're trying
> > to
> > put both into one build and its only really designed for one or the
> > other.
> Ah, well that's going to be a problem then. I probably didn't make it
> clear, but this is a single bootloader build that uses both
> compilers. Some
> parts get built with the 32-bit compiler and some with the 64-bit
> compiler,
> and then the whole thing gets combined into a single binary.
> Splitting it
> into three separate recipes would mean making larger changes to the
> vendor
> build system than I'd really like.
> 
> > 
> > What would be "normal" is to build "lib32-bootloader" which is the
> > lib32 BBCLASSEXTEND'd variant of bootloader. If the 64 bit version
> > doesn't make sense you can make that raise SkipRecipe for the
> > nonsense
> > variants.
> I was successfully building a few lib32- recipes in order to test the
> compiler and, as you say, they all consistently use the lib32-named
> work
> directory.
> 
> I can think of two ways round the problem:
> 
> 1. Add a do_prepare_sysroot_append that moves the lib32- sysroot into
> the
> correct work directory.
> 
> 2. The simplest solution is to add a do_clean_append to my bootloader
> recipe that also cleans up the problem sysroot directory.
> 
> The first is probably preferable and I'll have a go at doing that.
> Neither
> is ideal, but anything else looks like it's going to complicate the
> multilib class code greatly and possibly break the intended use of
> the
> secondary toolchain.

I mentioned the issues above so you understand how you're differing
from the "standard" way multilib is used. I'm hoping there is some
simple way you can get it working for you. If I remember rightly, there
are differently named sysroots simply to avoid file conflict issues
with the sysroots overlapping (e.g. sets of header files, the usr/share
data and so on). We allow for that overlapping on the target device, we
don't really deal with that in the sysroots.

Option 3 may be to have the core staging.bbclass clean code iterate
over the sysroots, it sounds like its creating them, just not cleaning
them up, which is kind of a bug. "clean" is a slow path so even a tiny
performance hit there for this likely isn't an issue...

Cheers,

Richard

Re: Building and using a second toolchain

[Mike Crowe]

Recap: I have a bootloader recipe that uses both the primary AArch64
cross-toolchain and an AArch32 cross-toolchain. I arrange for those
toolchains to be available by setting MULTILIBS for the machine and adding
the multilib compiler to DEPENDS for the recipe. However, this means that
the AArch32 sysroot for the recipe is not cleaned up correctly by do_clean
since it is outside WORKDIR.

On Friday 08 September 2017 at 07:53:41 +0100, Richard Purdie wrote:
> Option 3 may be to have the core staging.bbclass clean code iterate
> over the sysroots, it sounds like its creating them, just not cleaning
> them up, which is kind of a bug. "clean" is a slow path so even a tiny
> performance hit there for this likely isn't an issue...

Sorry for the delay, but I'm finally looking at this again.

I've read through staging.bbclass and I'm not sure what the best way to
implement what you suggest is.

The simplest way would be to build up a deduplicated list of sysroots from
the destsysroot assignments in extend_recipe_sysroot. If this list differs
from RECIPE_SYSROOT then stick the result in a variable and add a function
to CLEANFUNCS to perform the cleanup. Unfortunately, this adds code to the
normal build path which, based on your comment above, is probably not
acceptable.

The alternative would be to always add a function to CLEANFUNCS which runs
the same loop as extend_recipe_sysroot to identify the sysroots and remove
them (the primary ones should have already been deleted by do_clean at that
point.) This should have no effect on the normal build path, but the code
will probably not be updated if extend_recipe_sysroot changes in the
future.

Or, something else.

Do you have any further advice?

Thanks.

Mike.