Re: [scarthgap] recipe's sysroot misconfiguration in multilib

Re: [scarthgap] recipe's sysroot misconfiguration in multilib

[Richard Purdie]

On Wed, 2025-03-26 at 19:47 +0100, Joao Marcos Costa wrote:
> I have been trying to set up a multilib environment to generate 
> libraries built with Control Flow Integrity support. The gcc toolchain 
> is already patched to support this custom multilib ABI, and it is indeed 
> capable of generating binaries with the CFI flags.
> 
> I am, however, facing this strange issue where a header is not found in 
> the multilib recipe-sysroot (i.e., libcfi-recipe-sysroot) of a given 
> package (e.g., zstd), and do_compile task fails:
> 
> libcfi-zstd/1.5.5/libcfi-recipe-sysroot/usr/include/c++/13.2.0/cstdint:38:10: 
> fatal error: bits/c++config.h: No such file or directory
> >    38 | #include <bits/c++config.h>
> >       |          ^~~~~~~~~~~~~~~~~~
> > compilation terminated.
> > make: *** [Makefile:248: Options.o] Error 1
> 
> The strange part is the header is there:
> libcfi-recipe-sysroot/usr/include/c++/13.2.0/riscv64-freedomusdkmllibcfi-linux/bits/c++config.h
> 
> Another strange aspect in this very same context is: the 
> recipe-sysroot-native seems misconfigured and incomplete. For reference, 
> let me use libcfi-zstd/zstd again:
> 
> zstd/1.5.5$ ls recipe-sysroot-native/sysroot-providers/ -l | wc -l
> 66
> zstd/1.5.5$ du -sh recipe-sysroot-native/
> 457M	recipe-sysroot-native/
> 
> libcfi-zstd/1.5.5$ ls recipe-sysroot-native/sysroot-providers/ -l | wc -l
> 27
> libcfi-zstd/1.5.5$ du -sh recipe-sysroot-native/
> 194M	recipe-sysroot-native/
> 
> Unless I'm missing something, shouldn't I have roughly the same values here?
> 
> Besides my TUNE_*, DEFAULTTUNE, BASE_LIB values set up, I also added
> 'GCCMULTILIB = "--enable-multilib"' to gcc-cross.inc, to enable multilib 
> for the cross toolchain. Could I be missing something, maybe with
> recipes-devtools/gcc/gcc-multilib-config.inc?

I'm finding it odd that you have to add --enable-multilib since in
general with gcc-cross, we use separate cross toolchains for each
multilib. This kind of implies you're configuring the setup differently
to how the system would expect.

The recipe-sysroot-native/installeddeps/ directory will show you what
is installed in each and might help narrow down the size difference.

Cheers,

Richard

Re: [OE-core] [scarthgap] recipe's sysroot misconfiguration in multilib

[Joao Marcos Costa]

Hello, Richard

Thanks for your reply

On 3/26/25 19:55, Richard Purdie via lists.openembedded.org wrote:
> On Wed, 2025-03-26 at 19:47 +0100, Joao Marcos Costa wrote:
>> I have been trying to set up a multilib environment to generate
>> libraries built with Control Flow Integrity support. The gcc toolchain
>> is already patched to support this custom multilib ABI, and it is indeed
>> capable of generating binaries with the CFI flags.
>>
>> I am, however, facing this strange issue where a header is not found in
>> the multilib recipe-sysroot (i.e., libcfi-recipe-sysroot) of a given
>> package (e.g., zstd), and do_compile task fails:
>>
(...)
>>
>> Besides my TUNE_*, DEFAULTTUNE, BASE_LIB values set up, I also added
>> 'GCCMULTILIB = "--enable-multilib"' to gcc-cross.inc, to enable multilib
>> for the cross toolchain. Could I be missing something, maybe with
>> recipes-devtools/gcc/gcc-multilib-config.inc?
> 
> I'm finding it odd that you have to add --enable-multilib since in
> general with gcc-cross, we use separate cross toolchains for each
> multilib. This kind of implies you're configuring the setup differently
> to how the system would expect.

If I don't, my cross toolchain is configured with "--disable-multilib". 
To confirm this, I ran gcc with "-v" from within the recipe's native 
sysroot, and the output is:

Target: riscv64-freedomusdkmllibcfi-linux
Configured with: 
../../../../../../work-shared/gcc-internal-13.2.0-r0/git/configure 
--build=x86_64-linux --host=x86_64-linux 
--target=riscv64-freedomusdkmllibcfi-linux --prefix=//host/usr 
--exec_prefix=//host/usr 
--bindir=//host/usr/bin/riscv64-freedomusdkmllibcfi-linux 
--sbindir=//host/usr/bin/riscv64-freedomusdkmllibcfi-linux 
--libexecdir=//host/usr/libexec/riscv64-freedomusdkmllibcfi-linux 
--datadir=//host/usr/share --sysconfdir=//host/etc 
--sharedstatedir=//host/com --localstatedir=//host/var 
--libdir=//host/usr/lib/riscv64-freedomusdkmllibcfi-linux 
--includedir=//host/usr/include --oldincludedir=//host/usr/include 
--infodir=//host/usr/share/info --mandir=//host/usr/share/man 
--disable-silent-rules --disable-dependency-tracking 
--with-libtool-sysroot=//host --enable-clocale=generic --with-gnu-ld 
--enable-shared --enable-languages=c,c++,fortran --enable-threads=posix 
--disable-multilib --enable-default-pie --enable-c99 --enable-long-long 
--enable-(...)

As a consequence, the build fails even before it gets to libcfi-zstd:

...libcfi-glibc-internal/2.37/recipe-sysroot-native/usr/bin/riscv64-freedomusdkmllibcfi-linux/../../libexec/riscv64-freedomusdkmllibcfi-linux/gcc/riscv64-freedomusdkmllibcfi-linux/13.2.0/ld.bfd: 
cannot find -lgcc: No such file or directory
| collect2: error: ld returned 1 exit status
| make[2]: *** [Makefile:1293: 
...libcfi-glibc-internal/2.37/build-riscv64-freedomusdkmllibcfi-linux/elf/librtld.map] 
Error 1


> 
> The recipe-sysroot-native/installeddeps/ directory will show you what
> is installed in each and might help narrow down the size difference.
> 

Oddly enough, both installeddeps seem to contain (at least) the same 
number of items:

$ ls -l zstd/1.5.5/recipe-sysroot-native/installeddeps/ | wc -l
74
$ ls 
../riscv64cfi-freedomusdkmllibcfi-linux/libcfi-zstd/1.5.5/recipe-sysroot-native/installeddeps/ 
-l | wc -l
75

But, to be honest, my comparison didn't go any further than comparing 
this line count.

> Cheers,
> 
> Richard

Thanks!

-- 
Best regards,
João Marcos Costa

Re: [OE-core] [scarthgap] recipe's sysroot misconfiguration in multilib

[Khem Raj]

On Wed, Mar 26, 2025 at 12:15 PM Joao Marcos Costa
<joaomarcos.costa@bootlin.com> wrote:
>
> Hello, Richard
>
> Thanks for your reply
>
> On 3/26/25 19:55, Richard Purdie via lists.openembedded.org wrote:
> > On Wed, 2025-03-26 at 19:47 +0100, Joao Marcos Costa wrote:
> >> I have been trying to set up a multilib environment to generate
> >> libraries built with Control Flow Integrity support. The gcc toolchain
> >> is already patched to support this custom multilib ABI, and it is indeed
> >> capable of generating binaries with the CFI flags.
> >>
> >> I am, however, facing this strange issue where a header is not found in
> >> the multilib recipe-sysroot (i.e., libcfi-recipe-sysroot) of a given
> >> package (e.g., zstd), and do_compile task fails:
> >>
> (...)
> >>
> >> Besides my TUNE_*, DEFAULTTUNE, BASE_LIB values set up, I also added
> >> 'GCCMULTILIB = "--enable-multilib"' to gcc-cross.inc, to enable multilib
> >> for the cross toolchain. Could I be missing something, maybe with
> >> recipes-devtools/gcc/gcc-multilib-config.inc?
> >
> > I'm finding it odd that you have to add --enable-multilib since in
> > general with gcc-cross, we use separate cross toolchains for each
> > multilib. This kind of implies you're configuring the setup differently
> > to how the system would expect.
>
> If I don't, my cross toolchain is configured with "--disable-multilib".
> To confirm this, I ran gcc with "-v" from within the recipe's native
> sysroot, and the output is:
>
> Target: riscv64-freedomusdkmllibcfi-linux
> Configured with:
> ../../../../../../work-shared/gcc-internal-13.2.0-r0/git/configure
> --build=x86_64-linux --host=x86_64-linux
> --target=riscv64-freedomusdkmllibcfi-linux --prefix=//host/usr
> --exec_prefix=//host/usr
> --bindir=//host/usr/bin/riscv64-freedomusdkmllibcfi-linux
> --sbindir=//host/usr/bin/riscv64-freedomusdkmllibcfi-linux
> --libexecdir=//host/usr/libexec/riscv64-freedomusdkmllibcfi-linux
> --datadir=//host/usr/share --sysconfdir=//host/etc
> --sharedstatedir=//host/com --localstatedir=//host/var
> --libdir=//host/usr/lib/riscv64-freedomusdkmllibcfi-linux
> --includedir=//host/usr/include --oldincludedir=//host/usr/include
> --infodir=//host/usr/share/info --mandir=//host/usr/share/man
> --disable-silent-rules --disable-dependency-tracking
> --with-libtool-sysroot=//host --enable-clocale=generic --with-gnu-ld
> --enable-shared --enable-languages=c,c++,fortran --enable-threads=posix
> --disable-multilib --enable-default-pie --enable-c99 --enable-long-long
> --enable-(...)
>
> As a consequence, the build fails even before it gets to libcfi-zstd:
>
> ...libcfi-glibc-internal/2.37/recipe-sysroot-native/usr/bin/riscv64-freedomusdkmllibcfi-linux/../../libexec/riscv64-freedomusdkmllibcfi-linux/gcc/riscv64-freedomusdkmllibcfi-linux/13.2.0/ld.bfd:
> cannot find -lgcc: No such file or directory
> | collect2: error: ld returned 1 exit status
> | make[2]: *** [Makefile:1293:

OE's notion of multilib is at system level, which means it will need
full runtimes for the chosen multilibs, GCC builds the compiler
runtime and has its own
directory naming structure to place them which may not be inline with
what OE toolchain is doing. For hosted targets we do not build a
toolchain with all
possible multilibs for a given architecture which technically is
possible with gcc. our mutlilibs are a small subset of that and that
is what toolchain will recognise
provided its placed in right locations in sysroot.

> ...libcfi-glibc-internal/2.37/build-riscv64-freedomusdkmllibcfi-linux/elf/librtld.map]
> Error 1
>
>
> >
> > The recipe-sysroot-native/installeddeps/ directory will show you what
> > is installed in each and might help narrow down the size difference.
> >
>
> Oddly enough, both installeddeps seem to contain (at least) the same
> number of items:
>
> $ ls -l zstd/1.5.5/recipe-sysroot-native/installeddeps/ | wc -l
> 74
> $ ls
> ../riscv64cfi-freedomusdkmllibcfi-linux/libcfi-zstd/1.5.5/recipe-sysroot-native/installeddeps/
> -l | wc -l
> 75
>
> But, to be honest, my comparison didn't go any further than comparing
> this line count.
>
> > Cheers,
> >
> > Richard
>
> Thanks!
>
> --
> Best regards,
> João Marcos Costa

Re: [OE-core] [scarthgap] recipe's sysroot misconfiguration in multilib

[Joao Marcos Costa]

Hello,

On 3/26/25 20:26, Khem Raj via lists.openembedded.org wrote:
> On Wed, Mar 26, 2025 at 12:15 PM Joao Marcos Costa
> <joaomarcos.costa@bootlin.com> wrote:
>>
>> Hello, Richard
>>
>> Thanks for your reply
>>
>> On 3/26/25 19:55, Richard Purdie via lists.openembedded.org wrote:
>>> On Wed, 2025-03-26 at 19:47 +0100, Joao Marcos Costa wrote:
>>>> I have been trying to set up a multilib environment to generate
>>>> libraries built with Control Flow Integrity support. The gcc toolchain
>>>> is already patched to support this custom multilib ABI, and it is indeed
>>>> capable of generating binaries with the CFI flags.
>>>>
>>>> I am, however, facing this strange issue where a header is not found in
>>>> the multilib recipe-sysroot (i.e., libcfi-recipe-sysroot) of a given
>>>> package (e.g., zstd), and do_compile task fails:
>>>>
>> (...)
>>>>
>>>> Besides my TUNE_*, DEFAULTTUNE, BASE_LIB values set up, I also added
>>>> 'GCCMULTILIB = "--enable-multilib"' to gcc-cross.inc, to enable multilib
>>>> for the cross toolchain. Could I be missing something, maybe with
>>>> recipes-devtools/gcc/gcc-multilib-config.inc?
>>>
>>> I'm finding it odd that you have to add --enable-multilib since in
>>> general with gcc-cross, we use separate cross toolchains for each
>>> multilib. This kind of implies you're configuring the setup differently
>>> to how the system would expect.
>>
>> If I don't, my cross toolchain is configured with "--disable-multilib".
>> To confirm this, I ran gcc with "-v" from within the recipe's native
>> sysroot, and the output is:
>>
>> Target: riscv64-freedomusdkmllibcfi-linux
>> Configured with:
>> ../../../../../../work-shared/gcc-internal-13.2.0-r0/git/configure
>> --build=x86_64-linux --host=x86_64-linux
>> --target=riscv64-freedomusdkmllibcfi-linux --prefix=//host/usr
>> --exec_prefix=//host/usr
>> --bindir=//host/usr/bin/riscv64-freedomusdkmllibcfi-linux
>> --sbindir=//host/usr/bin/riscv64-freedomusdkmllibcfi-linux
>> --libexecdir=//host/usr/libexec/riscv64-freedomusdkmllibcfi-linux
>> --datadir=//host/usr/share --sysconfdir=//host/etc
>> --sharedstatedir=//host/com --localstatedir=//host/var
>> --libdir=//host/usr/lib/riscv64-freedomusdkmllibcfi-linux
>> --includedir=//host/usr/include --oldincludedir=//host/usr/include
>> --infodir=//host/usr/share/info --mandir=//host/usr/share/man
>> --disable-silent-rules --disable-dependency-tracking
>> --with-libtool-sysroot=//host --enable-clocale=generic --with-gnu-ld
>> --enable-shared --enable-languages=c,c++,fortran --enable-threads=posix
>> --disable-multilib --enable-default-pie --enable-c99 --enable-long-long
>> --enable-(...)
>>
>> As a consequence, the build fails even before it gets to libcfi-zstd:
>>
>> ...libcfi-glibc-internal/2.37/recipe-sysroot-native/usr/bin/riscv64-freedomusdkmllibcfi-linux/../../libexec/riscv64-freedomusdkmllibcfi-linux/gcc/riscv64-freedomusdkmllibcfi-linux/13.2.0/ld.bfd:
>> cannot find -lgcc: No such file or directory
>> | collect2: error: ld returned 1 exit status
>> | make[2]: *** [Makefile:1293:
> 
> OE's notion of multilib is at system level, which means it will need
> full runtimes for the chosen multilibs, GCC builds the compiler
> runtime and has its own
> directory naming structure to place them which may not be inline with
> what OE toolchain is doing. For hosted targets we do not build a
> toolchain with all
> possible multilibs for a given architecture which technically is
> possible with gcc. our mutlilibs are a small subset of that and that
> is what toolchain will recognise
> provided its placed in right locations in sysroot.
> 

Thanks, Khem.

After your reply and Richard's, I checked with a Poky build (arm64 + 
lib32 multilib), and doing the same test (i.e., gcc -v) I see both 
compilers (64-bit and 32-bit) were configured with "--disable-multilib". 
I will get rid of that GCCMULTILIB then.

Please tell me if I got your point correctly: each multilib ABI (lib32, 
lib64, libfoo, libbar) will have it's respective toolchain built, and as 
long as the toolchain's naming matches what OE expects in terms of 
sysroot structure, everything will work transparently.

In case I got you right, how can I tell what exactly OE expects of the 
sysroot locations and directory names? I ask this specially because in 
my arm64 + lib32 test, the target sysroots are:

lib32-zstd/1.5.6/lib32-recipe-sysroot/usr/lib
zstd/1.5.6/recipe-sysroot/usr/lib (with a symlink inside to lib64)
zstd/1.5.6/recipe-sysroot/usr/lib64

and to be fair, I was expecting the very opposite. Something like:
lib32-zstd/1.5.6/lib32-recipe-sysroot/usr/lib32
zstd/1.5.6/recipe-sysroot/usr/lib

In other words, the target architecture libdir would remain the same 
(i.e. usr/lib) and the multilib variant would be usr/lib32).

Thanks once more.

-- 
Best regards,
João Marcos Costa

Re: [OE-core] [scarthgap] recipe's sysroot misconfiguration in multilib

[Khem Raj]

On Wed, Mar 26, 2025 at 1:54 PM Joao Marcos Costa via
lists.openembedded.org
<joaomarcos.costa=bootlin.com@lists.openembedded.org> wrote:
>
> Hello,
>
> On 3/26/25 20:26, Khem Raj via lists.openembedded.org wrote:
> > On Wed, Mar 26, 2025 at 12:15 PM Joao Marcos Costa
> > <joaomarcos.costa@bootlin.com> wrote:
> >>
> >> Hello, Richard
> >>
> >> Thanks for your reply
> >>
> >> On 3/26/25 19:55, Richard Purdie via lists.openembedded.org wrote:
> >>> On Wed, 2025-03-26 at 19:47 +0100, Joao Marcos Costa wrote:
> >>>> I have been trying to set up a multilib environment to generate
> >>>> libraries built with Control Flow Integrity support. The gcc toolchain
> >>>> is already patched to support this custom multilib ABI, and it is indeed
> >>>> capable of generating binaries with the CFI flags.
> >>>>
> >>>> I am, however, facing this strange issue where a header is not found in
> >>>> the multilib recipe-sysroot (i.e., libcfi-recipe-sysroot) of a given
> >>>> package (e.g., zstd), and do_compile task fails:
> >>>>
> >> (...)
> >>>>
> >>>> Besides my TUNE_*, DEFAULTTUNE, BASE_LIB values set up, I also added
> >>>> 'GCCMULTILIB = "--enable-multilib"' to gcc-cross.inc, to enable multilib
> >>>> for the cross toolchain. Could I be missing something, maybe with
> >>>> recipes-devtools/gcc/gcc-multilib-config.inc?
> >>>
> >>> I'm finding it odd that you have to add --enable-multilib since in
> >>> general with gcc-cross, we use separate cross toolchains for each
> >>> multilib. This kind of implies you're configuring the setup differently
> >>> to how the system would expect.
> >>
> >> If I don't, my cross toolchain is configured with "--disable-multilib".
> >> To confirm this, I ran gcc with "-v" from within the recipe's native
> >> sysroot, and the output is:
> >>
> >> Target: riscv64-freedomusdkmllibcfi-linux
> >> Configured with:
> >> ../../../../../../work-shared/gcc-internal-13.2.0-r0/git/configure
> >> --build=x86_64-linux --host=x86_64-linux
> >> --target=riscv64-freedomusdkmllibcfi-linux --prefix=//host/usr
> >> --exec_prefix=//host/usr
> >> --bindir=//host/usr/bin/riscv64-freedomusdkmllibcfi-linux
> >> --sbindir=//host/usr/bin/riscv64-freedomusdkmllibcfi-linux
> >> --libexecdir=//host/usr/libexec/riscv64-freedomusdkmllibcfi-linux
> >> --datadir=//host/usr/share --sysconfdir=//host/etc
> >> --sharedstatedir=//host/com --localstatedir=//host/var
> >> --libdir=//host/usr/lib/riscv64-freedomusdkmllibcfi-linux
> >> --includedir=//host/usr/include --oldincludedir=//host/usr/include
> >> --infodir=//host/usr/share/info --mandir=//host/usr/share/man
> >> --disable-silent-rules --disable-dependency-tracking
> >> --with-libtool-sysroot=//host --enable-clocale=generic --with-gnu-ld
> >> --enable-shared --enable-languages=c,c++,fortran --enable-threads=posix
> >> --disable-multilib --enable-default-pie --enable-c99 --enable-long-long
> >> --enable-(...)
> >>
> >> As a consequence, the build fails even before it gets to libcfi-zstd:
> >>
> >> ...libcfi-glibc-internal/2.37/recipe-sysroot-native/usr/bin/riscv64-freedomusdkmllibcfi-linux/../../libexec/riscv64-freedomusdkmllibcfi-linux/gcc/riscv64-freedomusdkmllibcfi-linux/13.2.0/ld.bfd:
> >> cannot find -lgcc: No such file or directory
> >> | collect2: error: ld returned 1 exit status
> >> | make[2]: *** [Makefile:1293:
> >
> > OE's notion of multilib is at system level, which means it will need
> > full runtimes for the chosen multilibs, GCC builds the compiler
> > runtime and has its own
> > directory naming structure to place them which may not be inline with
> > what OE toolchain is doing. For hosted targets we do not build a
> > toolchain with all
> > possible multilibs for a given architecture which technically is
> > possible with gcc. our mutlilibs are a small subset of that and that
> > is what toolchain will recognise
> > provided its placed in right locations in sysroot.
> >
>
> Thanks, Khem.
>
> After your reply and Richard's, I checked with a Poky build (arm64 +
> lib32 multilib), and doing the same test (i.e., gcc -v) I see both
> compilers (64-bit and 32-bit) were configured with "--disable-multilib".
> I will get rid of that GCCMULTILIB then.
>
> Please tell me if I got your point correctly: each multilib ABI (lib32,
> lib64, libfoo, libbar) will have it's respective toolchain built, and as
> long as the toolchain's naming matches what OE expects in terms of
> sysroot structure, everything will work transparently.
>
> In case I got you right, how can I tell what exactly OE expects of the
> sysroot locations and directory names? I ask this specially because in
> my arm64 + lib32 test, the target sysroots are:
>
> lib32-zstd/1.5.6/lib32-recipe-sysroot/usr/lib
> zstd/1.5.6/recipe-sysroot/usr/lib (with a symlink inside to lib64)
> zstd/1.5.6/recipe-sysroot/usr/lib64
>
> and to be fair, I was expecting the very opposite. Something like:
> lib32-zstd/1.5.6/lib32-recipe-sysroot/usr/lib32
> zstd/1.5.6/recipe-sysroot/usr/lib

zstd(64-bit ABI) and lib32-zstd(32bit ABI) packages will build in there own
recipe specific sysroots. Difference will be in the the libdir and PN
e.g. see below for 64bit and 32bit respectively
zstd/1.5.6/lib32-recipe-sysroot/usr/lib64
lib32-zstd/1.5.6/lib32-recipe-sysroot/usr/lib

The multilib dir could be lib lib32 lib64 and it is primarily driven from legacy
and how linux systems accomodated 32bit arch on 64bit systems so there
is no rule but convention that was followed by the OS builders and it is not
same across different architectures e.g. ppc/ppc64 mips/mipsn32/mips64
i686/x86_64
arm/aarch64 might have differences e.g. lib may contain 64bit
libraries in that case
lib32 may contain 32bit and lib64 may contain 64bit in that case lib
may have 32bit binaries


>
> In other words, the target architecture libdir would remain the same
> (i.e. usr/lib) and the multilib variant would be usr/lib32).
>
> Thanks once more.
>
> --
> Best regards,
> João Marcos Costa
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#213692): https://lists.openembedded.org/g/openembedded-core/message/213692
> Mute This Topic: https://lists.openembedded.org/mt/111923149/1997914
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>