odd endianness toolchains for crosstool

odd endianness toolchains for crosstool

[Jason A. Donenfeld]

Hey Arnd,

I'm again experimenting with switching to your crosstool toolchains for
WireGuard's CI. I've hit a few snags in the process:

- For powerpc, gcc needs to be built with `--enable-secureplt
  --with-long-double-64` in order for musl to run.
- Need powerpc64le compiler (-mabi=elfv2).
- Need mipsel compiler.
- Need aarch64_be compiler.
- Need armeb compiler.
- Need mips64el compiler.

While the existing compilers can all produce code for the wrong endian,
they hit trouble when trying to link against libgcc. So generally a
separate full toolchain is supplied for the less common endians.

I have had success with arm, arm64, mips, x86_64, i386, m68k. If you're
up for adding the above compilers to the collection, I'd be able to
complete the transition, and then look into adding a few more
architectures.

Jason

Re: odd endianness toolchains for crosstool

[Arnd Bergmann]

On Mon, Apr 25, 2022 at 3:39 AM Jason A. Donenfeld <Jason@zx2c4.com> wrote:
>
> Hey Arnd,
>
> I'm again experimenting with switching to your crosstool toolchains for
> WireGuard's CI. I've hit a few snags in the process:
>
> - For powerpc, gcc needs to be built with `--enable-secureplt
>   --with-long-double-64` in order for musl to run.
> - Need powerpc64le compiler (-mabi=elfv2).
> - Need mipsel compiler.
> - Need aarch64_be compiler.
> - Need armeb compiler.
> - Need mips64el compiler.
>
> While the existing compilers can all produce code for the wrong endian,
> they hit trouble when trying to link against libgcc. So generally a
> separate full toolchain is supplied for the less common endians.

Hi Jason,

I'm definitely interested in improving the user space testing,  and I agree
we need the powerpc64le and mipsel targets at the minimum for that.

The situation on my end is that I'm planning to migrate my main workstation
(which I'm building the compilers on) to an arm64 machine soon, and
will then need to set it all up again. I don't really want to change much before
then to avoid changing things twice.

I've added Nick to Cc, as he's experimenting with a clang based toolchain
that we can put on kernel.org along with the gcc toolchains, and that
would probably include a musl based sysroot roughly the same set of
architectures that you are testing on already. Possibly we could reuse the
same user space between clang and gcc.

> I have had success with arm, arm64, mips, x86_64, i386, m68k. If you're
> up for adding the above compilers to the collection, I'd be able to
> complete the transition, and then look into adding a few more
> architectures.

I've also looked at other projects that do qemu based testing, everyone
seems to be missing one or two architectures out of a common set,
https://tinyurl.com/linux-architectures is where I keep my data.

The most common subset of architectures that get tested as far
as I can tell is x86, arm64, powerpc, arm, riscv, mips, s390: those
are the ones that support all the important pieces (gcc, clang, musl,
glibc, qemu, debian and buildroot) and that have the most users.
The exceptions to this are I think:

* your wireguard tests are missing riscv and s390 tests, those should
  be easy to add. You do support m68k, which the others are missing
* kunit is missing mips tests, but has alpha and sparc
* tuxrun[1] adds sparc64, sh4 and armv5 (softfp), could add rv32
* Günter's linux-build-test[2] has all the above, plus microblaze,
  nios2, parisc, shbe, and xtensa

What I'd really like to see is to have the necessary information for
building and running the most common subset of these in one place
in the kernel tree where at least wireguard, kunit and tuxrun can
share the setup for qemu. One idea I had was to encode the
platform specific qemu command line options using Kconfig
dependencies in a way that "make O=obj ARCH=foo defconfig zImage;
obj/run-qemu" results in a booting kernel on a lot of the typical
defconfigs for supported architectures.

       Arnd

[1] https://gitlab.com/Linaro/tuxrun/-/blob/master/tuxrun/devices/qemu.py
[2] https://github.com/groeck/linux-build-test/tree/master/rootfs

Re: odd endianness toolchains for crosstool

[Jason A. Donenfeld]

Hi Arnd,

On Mon, Apr 25, 2022 at 10:46:45AM +0200, Arnd Bergmann wrote:
> The situation on my end is that I'm planning to migrate my main workstation
> (which I'm building the compilers on) to an arm64 machine soon, and
> will then need to set it all up again. I don't really want to change much before
> then to avoid changing things twice.

Ahh, okay, so probably crosstool won't be viable for me for a while. Are
your existing scripts fairly reproducible and easy? I suppose I could
just build my own if I can't find another project supplying light
compilers.

> I've added Nick to Cc, as he's experimenting with a clang based toolchain
> that we can put on kernel.org along with the gcc toolchains, and that
> would probably include a musl based sysroot roughly the same set of
> architectures that you are testing on already. Possibly we could reuse the
> same user space between clang and gcc.

I personally have no use for compilers with user spaces. My test harness
builds musl as part of it. It's really the quickest part of the entire
harness to build. I also probably won't switch things over to clang;
Google has the resources to do that themselves. Basically all I need is
the boring nolibc compilers for a few extra platforms, and for the ppc
one to build with the mentioned flags.

> I've also looked at other projects that do qemu based testing, everyone
> seems to be missing one or two architectures out of a common set,
> https://tinyurl.com/linux-architectures is where I keep my data.

If the compilers are there, and they can build a working musl, and QEMU
will boot it, and I can work out a minimal kernel .config that doesn't
take a long time to compile, then it'll get included in the CI. So in
theory, I should be able to expand the portfolio of architectures I'm
using.

> building and running the most common subset of these in one place
> in the kernel tree where at least wireguard, kunit and tuxrun can
> share the setup for qemu.

I have little interest in that kind of abstraction unfortunately.
WireGuard's test suite is optimized for building minimal kernels that
exercise its functionality. Tons of emphasis is placed on those kernels
being minimal. And the userspaces too. The whole thing needs to compile
fast and boot fast. Perhaps megacorps can afford to throw massive
clusters at this, but I'm trying to test every single commit across 10
or so trees on every architecture that I can, from one box. Plus, I use
this when developing on my laptop. If it takes forever to build and if I
can't iterate fast on it, then it's simply not useful for me (or for
other various contributors who have been able to quickly pick up the
WireGuard test suite for development without a lot of futzing). So for
that reason, I don't see a lot of promise in making a monster
abstraction layer that does all the things. Plus, the more people who
invent the wheel (which is not even a particularly interesting wheel)
the more odd configurations and bugs we'll surface.

Jason

Re: odd endianness toolchains for crosstool

[Arnd Bergmann]

On Mon, Apr 25, 2022 at 1:43 PM Jason A. Donenfeld <Jason@zx2c4.com> wrote:
> On Mon, Apr 25, 2022 at 10:46:45AM +0200, Arnd Bergmann wrote:
> > The situation on my end is that I'm planning to migrate my main workstation
> > (which I'm building the compilers on) to an arm64 machine soon, and
> > will then need to set it all up again. I don't really want to change much before
> > then to avoid changing things twice.
>
> Ahh, okay, so probably crosstool won't be viable for me for a while. Are
> your existing scripts fairly reproducible and easy? I suppose I could
> just build my own if I can't find another project supplying light
> compilers.

The scripts are fairly solid, but the original git tree is no longer available
and my version has a couple of local changes with a bit of a dirty history
from adding support for cross-compiling the compilers themselves to
do the canadian-cross arm64 and ppc64le hosted ones.

There is another fork of the same tree on
https://github.com/nathanchance/buildall

The main issue with building distributable binaries is to actually build
them on an older rootfs to avoid linking against a newer glibc, and
to ensure the dependencies for gcc are statically linked. Without that,
the output is too distro specific.

> > I've added Nick to Cc, as he's experimenting with a clang based toolchain
> > that we can put on kernel.org along with the gcc toolchains, and that
> > would probably include a musl based sysroot roughly the same set of
> > architectures that you are testing on already. Possibly we could reuse the
> > same user space between clang and gcc.
>
> I personally have no use for compilers with user spaces. My test harness
> builds musl as part of it. It's really the quickest part of the entire
> harness to build. I also probably won't switch things over to clang;
> Google has the resources to do that themselves. Basically all I need is
> the boring nolibc compilers for a few extra platforms, and for the ppc
> one to build with the mentioned flags.

I suppose the only thing you are missing is libgcc (or libcompiler-rt)
for those platforms. I had a closer look into what is or can be included
here, and I see that my builds include multiple versions on some of
the  architectures, but not on others:

aarch64-linux/lib/gcc/aarch64-linux/11.1.0/libgcc.a
alpha-linux/lib/gcc/alpha-linux/11.1.0/libgcc.a
arc-linux/lib/gcc/arc-linux/11.1.0/libgcc.a
arc-linux/lib/gcc/arc-linux/11.1.0/hs/libgcc.a
arc-linux/lib/gcc/arc-linux/11.1.0/archs/libgcc.a
arc-linux/lib/gcc/arc-linux/11.1.0/hs38/libgcc.a
arc-linux/lib/gcc/arc-linux/11.1.0/hs38_linux/libgcc.a
arc-linux/lib/gcc/arc-linux/11.1.0/arc700/libgcc.a
arc-linux/lib/gcc/arc-linux/11.1.0/nps400/libgcc.a
arm-linux-gnueabi/lib/gcc/arm-linux-gnueabi/11.1.0/libgcc.a
...
powerpc-linux/lib/gcc/powerpc-linux/11.1.0/libgcc.a
powerpc-linux/lib/gcc/powerpc-linux/11.1.0/64/libgcc.a
powerpc-linux/lib/gcc/powerpc-linux/11.1.0/64/le/libgcc.a
powerpc-linux/lib/gcc/powerpc-linux/11.1.0/le/libgcc.a
powerpc-linux/lib/gcc/powerpc-linux/11.1.0/32/le/libgcc.a
powerpc64-linux/lib/gcc/powerpc64-linux/11.1.0/libgcc.a
powerpc64-linux/lib/gcc/powerpc64-linux/11.1.0/32/libgcc.a
powerpc64-linux/lib/gcc/powerpc64-linux/11.1.0/32/le/libgcc.a
powerpc64-linux/lib/gcc/powerpc64-linux/11.1.0/le/libgcc.a
powerpc64-linux/lib/gcc/powerpc64-linux/11.1.0/64/le/libgcc.a
mips-linux/lib/gcc/mips-linux/11.1.0/libgcc.a
mips-linux/lib/gcc/mips-linux/11.1.0/n32/libgcc.a
mips-linux/lib/gcc/mips-linux/11.1.0/64/libgcc.a
mips64-linux/lib/gcc/mips64-linux/11.1.0/libgcc.a
mips64-linux/lib/gcc/mips64-linux/11.1.0/32/libgcc.a
mips64-linux/lib/gcc/mips64-linux/11.1.0/64/libgcc.a

So on powerpc, there are already both big-endian and
little-endian binaries, but arm and mips only have one of the
two. I asked our local compiler experts, and they suggested
that one can add further multiarch-configs like the one
in gcc/config/arm/t-aprofile to allow building for a different
subset of the hundreds of possible configurations.

the t-aprofile builds libgcc for a couple of combinations of
cpu architecture level and FPU ABIs, but they are all the
same endianess. gcc/config/rs6000/t-linux64lebe is the
corresponding file for powerpc that enables all combinations
of 32/64 and be/le.

> > I've also looked at other projects that do qemu based testing, everyone
> > seems to be missing one or two architectures out of a common set,
> > https://tinyurl.com/linux-architectures is where I keep my data.
>
> If the compilers are there, and they can build a working musl, and QEMU
> will boot it, and I can work out a minimal kernel .config that doesn't
> take a long time to compile, then it'll get included in the CI. So in
> theory, I should be able to expand the portfolio of architectures I'm
> using.

Adding riscv and s390 should indeed be fairly simple to add,
and you can probably just take a look at what ktest does for them.

You have a good point about the minimal kernel config, which makes
sense for testing a single thing, but of course others generally want
to test a 'defconfig' build that is closer to what users would actually
run.

> > building and running the most common subset of these in one place
> > in the kernel tree where at least wireguard, kunit and tuxrun can
> > share the setup for qemu.
>
> I have little interest in that kind of abstraction unfortunately.

Ok, fair enough.

      Arnd

Re: odd endianness toolchains for crosstool

[Jason A. Donenfeld]

Hey Arnd,

On Mon, Apr 25, 2022 at 04:55:20PM +0200, Arnd Bergmann wrote:
> I suppose the only thing you are missing is libgcc (or libcompiler-rt)
> for those platforms. I had a closer look into what is or can be included
> here, and I see that my builds include multiple versions on some of
> the  architectures, but not on others:
> 
> aarch64-linux/lib/gcc/aarch64-linux/11.1.0/libgcc.a
> alpha-linux/lib/gcc/alpha-linux/11.1.0/libgcc.a
> arc-linux/lib/gcc/arc-linux/11.1.0/libgcc.a
> arc-linux/lib/gcc/arc-linux/11.1.0/hs/libgcc.a
> arc-linux/lib/gcc/arc-linux/11.1.0/archs/libgcc.a
> arc-linux/lib/gcc/arc-linux/11.1.0/hs38/libgcc.a
> arc-linux/lib/gcc/arc-linux/11.1.0/hs38_linux/libgcc.a
> arc-linux/lib/gcc/arc-linux/11.1.0/arc700/libgcc.a
> arc-linux/lib/gcc/arc-linux/11.1.0/nps400/libgcc.a
> arm-linux-gnueabi/lib/gcc/arm-linux-gnueabi/11.1.0/libgcc.a
> ...
> powerpc-linux/lib/gcc/powerpc-linux/11.1.0/libgcc.a
> powerpc-linux/lib/gcc/powerpc-linux/11.1.0/64/libgcc.a
> powerpc-linux/lib/gcc/powerpc-linux/11.1.0/64/le/libgcc.a
> powerpc-linux/lib/gcc/powerpc-linux/11.1.0/le/libgcc.a
> powerpc-linux/lib/gcc/powerpc-linux/11.1.0/32/le/libgcc.a
> powerpc64-linux/lib/gcc/powerpc64-linux/11.1.0/libgcc.a
> powerpc64-linux/lib/gcc/powerpc64-linux/11.1.0/32/libgcc.a
> powerpc64-linux/lib/gcc/powerpc64-linux/11.1.0/32/le/libgcc.a
> powerpc64-linux/lib/gcc/powerpc64-linux/11.1.0/le/libgcc.a
> powerpc64-linux/lib/gcc/powerpc64-linux/11.1.0/64/le/libgcc.a
> mips-linux/lib/gcc/mips-linux/11.1.0/libgcc.a
> mips-linux/lib/gcc/mips-linux/11.1.0/n32/libgcc.a
> mips-linux/lib/gcc/mips-linux/11.1.0/64/libgcc.a
> mips64-linux/lib/gcc/mips64-linux/11.1.0/libgcc.a
> mips64-linux/lib/gcc/mips64-linux/11.1.0/32/libgcc.a
> mips64-linux/lib/gcc/mips64-linux/11.1.0/64/libgcc.a
> 
> So on powerpc, there are already both big-endian and
> little-endian binaries, but arm and mips only have one of the
> two. I asked our local compiler experts, and they suggested
> that one can add further multiarch-configs like the one
> in gcc/config/arm/t-aprofile to allow building for a different
> subset of the hundreds of possible configurations.
> 
> the t-aprofile builds libgcc for a couple of combinations of
> cpu architecture level and FPU ABIs, but they are all the
> same endianess. gcc/config/rs6000/t-linux64lebe is the
> corresponding file for powerpc that enables all combinations
> of 32/64 and be/le.

Right, exactly. So if you simply add little endian libgcc to ppc64, ppc,
mips, and mips64, and add big endian libgcc to arm and arm64, then we'd
be set. (And also, build ppc32 with --enable-secureplt
--with-long-double-64.)

Jason

Re: odd endianness toolchains for crosstool

[Arnd Bergmann]

On Mon, Apr 25, 2022 at 5:20 PM Jason A. Donenfeld <Jason@zx2c4.com> wrote:

> Right, exactly. So if you simply add little endian libgcc to ppc64, ppc,
> mips, and mips64, and add big endian libgcc to arm and arm64, then we'd
> be set.

I see that gcc-11.3 was just released, so I should probably try
building that in the
near future. I'll see if I can manage to come up with multiarch
configs for that,
and maybe even get those merged upstream so I don't have to carry patches
against the gcc source tree (all the other builds are for unmodified sources).

I can probably do that before migrating to the new machine, but I can't
promise how quickly I find time to start.

> (And also, build ppc32 with --enable-secureplt --with-long-double-64.)

Can you explain what those are about? Is this related to the ELFv1
vs ELFv2 difference or something else? Is this needed in both the
ppc32 and ppc64 compilers that each come with both targets?

      Arnd

Re: odd endianness toolchains for crosstool

[Jason A. Donenfeld]

Hi Arnd,

On Mon, Apr 25, 2022 at 05:39:34PM +0200, Arnd Bergmann wrote:
> I can probably do that before migrating to the new machine, but I can't
> promise how quickly I find time to start.

Oh awesome. Will keep my eye out for it.

> > (And also, build ppc32 with --enable-secureplt --with-long-double-64.)
> 
> Can you explain what those are about? Is this related to the ELFv1
> vs ELFv2 difference or something else? Is this needed in both the
> ppc32 and ppc64 compilers that each come with both targets?

For 32-bit, it is required. From
<https://wiki.musl-libc.org/supported-platforms.html>:

    powerpc (needs gcc built with --enable-secureplt
    --with-long-double-64, and -Wl,--secure-plt to link dynamic
    binaries.)

And from the INSTALL file it says:

    * PowerPC
        * Compiler toolchain must provide 64-bit long double, not IBM
          double-double or IEEE quad
        * For dynamic linking, compiler toolchain must be configured for
          "secure PLT" variant

For 64-bit, I'm not sure, but I know that at least -mabi=elfv2 is required,
and I think --with-long-double-64 too, according to documentation:

    * PowerPC64
        * Both little and big endian variants are supported
        * Compiler toolchain must provide 64-bit long double, not IBM
          double-double or IEEE quad
        * Compiler toolchain must use the new (ELFv2) ABI regardless of
          whether it is for little or big endian

Taken from <https://git.musl-libc.org/cgit/musl/tree/INSTALL#n35>.

Jason

Re: odd endianness toolchains for crosstool

[Arnd Bergmann]

On Mon, Apr 25, 2022 at 5:53 PM Jason A. Donenfeld <Jason@zx2c4.com> wrote:
>
> Hi Arnd,
>
> On Mon, Apr 25, 2022 at 05:39:34PM +0200, Arnd Bergmann wrote:
> > I can probably do that before migrating to the new machine, but I can't
> > promise how quickly I find time to start.
>
> Oh awesome. Will keep my eye out for it.
>
> > > (And also, build ppc32 with --enable-secureplt --with-long-double-64.)
> >
> > Can you explain what those are about? Is this related to the ELFv1
> > vs ELFv2 difference or something else? Is this needed in both the
> > ppc32 and ppc64 compilers that each come with both targets?
>
> For 32-bit, it is required. From
> <https://wiki.musl-libc.org/supported-platforms.html>:
>
>     powerpc (needs gcc built with --enable-secureplt
>     --with-long-double-64, and -Wl,--secure-plt to link dynamic
>     binaries.)
>
> And from the INSTALL file it says:
>
>     * PowerPC
>         * Compiler toolchain must provide 64-bit long double, not IBM
>           double-double or IEEE quad
>         * For dynamic linking, compiler toolchain must be configured for
>           "secure PLT" variant
>
> For 64-bit, I'm not sure, but I know that at least -mabi=elfv2 is required,
> and I think --with-long-double-64 too, according to documentation:
>
>     * PowerPC64
>         * Both little and big endian variants are supported
>         * Compiler toolchain must provide 64-bit long double, not IBM
>           double-double or IEEE quad
>         * Compiler toolchain must use the new (ELFv2) ABI regardless of
>           whether it is for little or big endian

Ok, I see. For all I can tell, the toolchain I built already uses both
--with-long-double-64
and --enable-secureplt, as those seemt to be the default for Linux.

Regarding the the ELF ABI, I'm not sure how to check, but I think it
only does ELFv1, which is the default for big-endian glibc.

       Arnd

Re: odd endianness toolchains for crosstool

[Jason A. Donenfeld]

Hi Arnd,

On Mon, Apr 25, 2022 at 06:15:07PM +0200, Arnd Bergmann wrote:
> On Mon, Apr 25, 2022 at 5:53 PM Jason A. Donenfeld <Jason@zx2c4.com> wrote:
> >
> > Hi Arnd,
> >
> > On Mon, Apr 25, 2022 at 05:39:34PM +0200, Arnd Bergmann wrote:
> > > I can probably do that before migrating to the new machine, but I can't
> > > promise how quickly I find time to start.
> >
> > Oh awesome. Will keep my eye out for it.
> >
> > > > (And also, build ppc32 with --enable-secureplt --with-long-double-64.)
> > >
> > > Can you explain what those are about? Is this related to the ELFv1
> > > vs ELFv2 difference or something else? Is this needed in both the
> > > ppc32 and ppc64 compilers that each come with both targets?
> >
> > For 32-bit, it is required. From
> > <https://wiki.musl-libc.org/supported-platforms.html>:
> >
> >     powerpc (needs gcc built with --enable-secureplt
> >     --with-long-double-64, and -Wl,--secure-plt to link dynamic
> >     binaries.)
> >
> > And from the INSTALL file it says:
> >
> >     * PowerPC
> >         * Compiler toolchain must provide 64-bit long double, not IBM
> >           double-double or IEEE quad
> >         * For dynamic linking, compiler toolchain must be configured for
> >           "secure PLT" variant
> >
> > For 64-bit, I'm not sure, but I know that at least -mabi=elfv2 is required,
> > and I think --with-long-double-64 too, according to documentation:
> >
> >     * PowerPC64
> >         * Both little and big endian variants are supported
> >         * Compiler toolchain must provide 64-bit long double, not IBM
> >           double-double or IEEE quad
> >         * Compiler toolchain must use the new (ELFv2) ABI regardless of
> >           whether it is for little or big endian
> 
> Ok, I see. For all I can tell, the toolchain I built already uses both
> --with-long-double-64
> and --enable-secureplt, as those seemt to be the default for Linux.

For ppc32? I'm unable to produce working executables with the toolchain.
And looking at the target info, -msecure-plt is missing, while
-mlong-double-64 is there:

$ ./powerpc-linux-gcc -Q --help=target | grep long-double
  -mlong-double-                        64
$ ./powerpc-linux-gcc -Q --help=target | grep msecure-plt
  -msecure-plt                          [disabled]

For ppc64, I see the same. I'll try to look into it more though.

> Regarding the the ELF ABI, I'm not sure how to check, but I think it
> only does ELFv1, which is the default for big-endian glibc.

Yes, it only is doing ELFv1 right now. musl checks this in
<https://git.musl-libc.org/cgit/musl/tree/configure#n689> with this:

    trycppif "_CALL_ELF == 2" "$t" || fail "$0: error: unsupported powerpc64 ABI"

Jason

Re: odd endianness toolchains for crosstool

[Jason A. Donenfeld]

On Mon, Apr 25, 2022 at 07:01:57PM +0200, Jason A. Donenfeld wrote:
> For ppc32? I'm unable to produce working executables with the toolchain.
> And looking at the target info, -msecure-plt is missing, while
> -mlong-double-64 is there:
> 
> $ ./powerpc-linux-gcc -Q --help=target | grep long-double
>   -mlong-double-                        64
> $ ./powerpc-linux-gcc -Q --help=target | grep msecure-plt
>   -msecure-plt                          [disabled]

And looking at the actual sections of the binary, indeed the .plt
section is RWX, which means it's not getting -msecure-plt as it should.

Re: odd endianness toolchains for crosstool

[Arnd Bergmann]

On Mon, Apr 25, 2022 at 7:01 PM Jason A. Donenfeld <Jason@zx2c4.com> wrote:
> On Mon, Apr 25, 2022 at 06:15:07PM +0200, Arnd Bergmann wrote:
> > On Mon, Apr 25, 2022 at 5:53 PM Jason A. Donenfeld <Jason@zx2c4.com> wrote:
> >
> > Ok, I see. For all I can tell, the toolchain I built already uses both
> > --with-long-double-64
> > and --enable-secureplt, as those seemt to be the default for Linux.
>
> For ppc32? I'm unable to produce working executables with the toolchain.
> And looking at the target info, -msecure-plt is missing, while
> -mlong-double-64 is there:
>
> $ ./powerpc-linux-gcc -Q --help=target | grep long-double
>   -mlong-double-                        64
> $ ./powerpc-linux-gcc -Q --help=target | grep msecure-plt
>   -msecure-plt                          [disabled]

Ah, my mistake, I looked at the distro provided compiler instead of my own :(

> For ppc64, I see the same. I'll try to look into it more though.
>
> > Regarding the the ELF ABI, I'm not sure how to check, but I think it
> > only does ELFv1, which is the default for big-endian glibc.
>
> Yes, it only is doing ELFv1 right now. musl checks this in
> <https://git.musl-libc.org/cgit/musl/tree/configure#n689> with this:
>
>     trycppif "_CALL_ELF == 2" "$t" || fail "$0: error: unsupported powerpc64 ABI"

Does it work if you pass -mabi=elfv2? This seems to be ignored here as well:

$ powerpc-linux-gcc  -mlittle-endian -mabi=elfv2 -xc -c /dev/null  -o /tmp/a.o
$ file /tmp/a.o
/tmp/a.o: ELF 32-bit LSB relocatable, PowerPC or cisco 4500, version 1
(SYSV), not stripped

        Arnd

Re: odd endianness toolchains for crosstool

[Jason A. Donenfeld]

Hi Arnd,

On Mon, Apr 25, 2022 at 7:40 PM Arnd Bergmann <arnd@arndb.de> wrote:
> > For ppc64, I see the same. I'll try to look into it more though.
> >
> > > Regarding the the ELF ABI, I'm not sure how to check, but I think it
> > > only does ELFv1, which is the default for big-endian glibc.
> >
> > Yes, it only is doing ELFv1 right now. musl checks this in
> > <https://git.musl-libc.org/cgit/musl/tree/configure#n689> with this:
> >
> >     trycppif "_CALL_ELF == 2" "$t" || fail "$0: error: unsupported powerpc64 ABI"
>
> Does it work if you pass -mabi=elfv2? This seems to be ignored here as well:
>
> $ powerpc-linux-gcc  -mlittle-endian -mabi=elfv2 -xc -c /dev/null  -o /tmp/a.o
> $ file /tmp/a.o
> /tmp/a.o: ELF 32-bit LSB relocatable, PowerPC or cisco 4500, version 1
> (SYSV), not stripped

You're mixing things up I think. -mabi=elfv2 is what powerpc64 needs,
but I don't think powerpc32 has it.

If you try to add -mabi=elfv2 to your existing powerpc64 compiler, it
gets upset when linking to libgcc.

Jason

Re: odd endianness toolchains for crosstool

[Rob Landley]

On 4/24/22 20:39, Jason A. Donenfeld wrote:
> Hey Arnd,
> 
> I'm again experimenting with switching to your crosstool toolchains for
> WireGuard's CI. I've hit a few snags in the process:
> 
> - For powerpc, gcc needs to be built with `--enable-secureplt
>   --with-long-double-64` in order for musl to run.
> - Need powerpc64le compiler (-mabi=elfv2).
> - Need mipsel compiler.
> - Need aarch64_be compiler.
> - Need armeb compiler.
> - Need mips64el compiler.

https://landley.net/toybox/faq.html#cross2
https://landley.net/toybox/downloads/binaries/toolchains/latest/

They're all musl based. I use them to build little bootable systems under qemu
with a 300 line bash script:

https://landley.net/toybox/faq.html#mkroot
https://landley.net/toybox/downloads/binaries/mkroot/latest/

Ala:

https://github.com/landley/toybox/blob/master/scripts/mkroot.sh

Rob

Re: odd endianness toolchains for crosstool

[Jason A. Donenfeld]

Hey Arnd,

On Mon, Apr 25, 2022 at 03:39:27AM +0200, Jason A. Donenfeld wrote:
> Hey Arnd,
> 
> I'm again experimenting with switching to your crosstool toolchains for
> WireGuard's CI. I've hit a few snags in the process:
> 
> - For powerpc, gcc needs to be built with `--enable-secureplt
>   --with-long-double-64` in order for musl to run.
> - Need powerpc64le compiler (-mabi=elfv2).
> - Need mipsel compiler.
> - Need aarch64_be compiler.
> - Need armeb compiler.
> - Need mips64el compiler.
> 
> While the existing compilers can all produce code for the wrong endian,
> they hit trouble when trying to link against libgcc. So generally a
> separate full toolchain is supplied for the less common endians.
> 
> I have had success with arm, arm64, mips, x86_64, i386, m68k. If you're
> up for adding the above compilers to the collection, I'd be able to
> complete the transition, and then look into adding a few more
> architectures.
> 
> Jason

I decided to give it another look, seeing if I could replace my musl.cc
compilers with your crosstool ones.

The actual changes required weren't so bad:

    https://git.zx2c4.com/wireguard-linux/commit/?h=update-toolchain

But there's not universal success:

    x86_64 - good
    i386 - good
    arm - good
    armeb - MISSING
    aarch64 - good
    aarch64_be - MISSING
    mips - BROKEN (doesn't like -EB)
    mipsel - MISSING
    mips64 - BROKEN (doesn't like -EB)
    mips64el - MISSING
    powerpc64 - BROKEN (wrong powerpc ABI)
    powerpc64le - MISSING
    powerpc - BROKEN (builds but some binaries segfault)
    m68k - good
    riscv64 - good
    riscv32 - good
    s390 - BROKEN (should be called "s390x" instead)
    um - kinda broken (but not crosstool's problem)

To try these, I've been running:

    ARCH=aarch64 make -C tools/testing/selftests/wireguard/qemu -j$(nproc)

or similar, against this tree:

    $ git clone -b update-toolchain https://git.zx2c4.com/wireguard-linux/

So it looks like it's not quite there, but not bad either. Just FYI in
case you're interested.

Jason

Re: odd endianness toolchains for crosstool

[Arnd Bergmann]

On Thu, Aug 29, 2024, at 17:51, Jason A. Donenfeld wrote:
> On Mon, Apr 25, 2022 at 03:39:27AM +0200, Jason A. Donenfeld wrote:
>
>
> I decided to give it another look, seeing if I could replace my musl.cc
> compilers with your crosstool ones.
>
> The actual changes required weren't so bad:
>
>     https://git.zx2c4.com/wireguard-linux/commit/?h=update-toolchain
>
> But there's not universal success:
>
>     x86_64 - good
>     i386 - good
>     arm - good
>     armeb - MISSING
>     aarch64 - good
>     aarch64_be - MISSING
>     mips - BROKEN (doesn't like -EB)
>     mipsel - MISSING
>     mips64 - BROKEN (doesn't like -EB)
>     mips64el - MISSING
>     powerpc64 - BROKEN (wrong powerpc ABI)
>     powerpc64le - MISSING
>     powerpc - BROKEN (builds but some binaries segfault)
>     m68k - good
>     riscv64 - good
>     riscv32 - good
>     s390 - BROKEN (should be called "s390x" instead)
>     um - kinda broken (but not crosstool's problem)
>
> To try these, I've been running:
>
>     ARCH=aarch64 make -C tools/testing/selftests/wireguard/qemu -j$(nproc)
>
> or similar, against this tree:
>
>     $ git clone -b update-toolchain https://git.zx2c4.com/wireguard-linux/
>
> So it looks like it's not quite there, but not bad either. Just FYI in
> case you're interested.

I wonder if the ones you list as missing all work with Nathan's clang
builds from https://mirrors.edge.kernel.org/pub/tools/llvm/ instead.

As far as I can tell, the main missing bit here is libgcc, which
is not always built along with gcc for all possible targets.
The llvm replacement for libgcc is https://compiler-rt.llvm.org/,
and you may have to build that in addition to musl when you try it.

I don't know if compiler-rt also works with gcc, but if it does,
that should fix most of the ones that you report as failing above.
The only one that won't work at all is um because the x86 toolchain
is already unable to build a kernel for that.

      Arnd

Re: odd endianness toolchains for crosstool

[Nathan Chancellor]

On Thu, Aug 29, 2024 at 10:25:32PM +0200, Arnd Bergmann wrote:
> On Thu, Aug 29, 2024, at 17:51, Jason A. Donenfeld wrote:
> > On Mon, Apr 25, 2022 at 03:39:27AM +0200, Jason A. Donenfeld wrote:
> >
> >
> > I decided to give it another look, seeing if I could replace my musl.cc
> > compilers with your crosstool ones.
> >
> > The actual changes required weren't so bad:
> >
> >     https://git.zx2c4.com/wireguard-linux/commit/?h=update-toolchain
> >
> > But there's not universal success:
> >
> >     x86_64 - good
> >     i386 - good
> >     arm - good
> >     armeb - MISSING
> >     aarch64 - good
> >     aarch64_be - MISSING
> >     mips - BROKEN (doesn't like -EB)
> >     mipsel - MISSING
> >     mips64 - BROKEN (doesn't like -EB)
> >     mips64el - MISSING
> >     powerpc64 - BROKEN (wrong powerpc ABI)
> >     powerpc64le - MISSING
> >     powerpc - BROKEN (builds but some binaries segfault)
> >     m68k - good
> >     riscv64 - good
> >     riscv32 - good
> >     s390 - BROKEN (should be called "s390x" instead)
> >     um - kinda broken (but not crosstool's problem)
> >
> > To try these, I've been running:
> >
> >     ARCH=aarch64 make -C tools/testing/selftests/wireguard/qemu -j$(nproc)
> >
> > or similar, against this tree:
> >
> >     $ git clone -b update-toolchain https://git.zx2c4.com/wireguard-linux/
> >
> > So it looks like it's not quite there, but not bad either. Just FYI in
> > case you're interested.
> 
> I wonder if the ones you list as missing all work with Nathan's clang
> builds from https://mirrors.edge.kernel.org/pub/tools/llvm/ instead.
> 
> As far as I can tell, the main missing bit here is libgcc, which
> is not always built along with gcc for all possible targets.
> The llvm replacement for libgcc is https://compiler-rt.llvm.org/,
> and you may have to build that in addition to musl when you try it.
> 
> I don't know if compiler-rt also works with gcc, but if it does,
> that should fix most of the ones that you report as failing above.
> The only one that won't work at all is um because the x86 toolchain
> is already unable to build a kernel for that.

I doubt my toolchains would fix this issue. As far as I am aware, only
the native compiler runtime is included, not any of the cross ones, as
the kernel itself does not use them. However, I suspect it would not be
too difficult to cross build compiler-rt standalone and stitch it into
my builds separately if desired.

FWIW, I have used your toolchains to build UML but maybe that is
environment dependent?

Cheers,
Nathan

Re: odd endianness toolchains for crosstool

[Arnd Bergmann]

On Tue, Sep 3, 2024, at 01:23, Nathan Chancellor wrote:
> On Thu, Aug 29, 2024 at 10:25:32PM +0200, Arnd Bergmann wrote:
>> On Thu, Aug 29, 2024, at 17:51, Jason A. Donenfeld wrote:
>> 
>> I wonder if the ones you list as missing all work with Nathan's clang
>> builds from https://mirrors.edge.kernel.org/pub/tools/llvm/ instead.
>> 
>> As far as I can tell, the main missing bit here is libgcc, which
>> is not always built along with gcc for all possible targets.
>> The llvm replacement for libgcc is https://compiler-rt.llvm.org/,
>> and you may have to build that in addition to musl when you try it.
>> 
>> I don't know if compiler-rt also works with gcc, but if it does,
>> that should fix most of the ones that you report as failing above.
>> The only one that won't work at all is um because the x86 toolchain
>> is already unable to build a kernel for that.
>
> I doubt my toolchains would fix this issue. As far as I am aware, only
> the native compiler runtime is included, not any of the cross ones, as
> the kernel itself does not use them. However, I suspect it would not be
> too difficult to cross build compiler-rt standalone and stitch it into
> my builds separately if desired.

Right, that's what I meant: building compiler-rt for all the
supported targets on clang should not be that hard, it would be
just another step on top of building musl. Doing the same for
gcc may be a different story as libgcc is not usually built as
a standalone library and compiler-rt may require building with
clang rather than gcc.

> FWIW, I have used your toolchains to build UML but maybe that is
> environment dependent?

Did you do that on x86? I only tried on arm64, where it is built
as a nolibc toolchain. Cross-building native x86 toolchains on arm64
is always tricky and I had expected that this would also be
a nolibc version, but it's possible that I accidentally built it
against the x86 glibc as well.

      Arnd