Re: [OE-core] [v4] [RFC] Merge meta-rust to oe-core - Aug 19 update

["Richard Purdie" <richard.purdie@linuxfoundation.org>]

Hi Esteban,

On Fri, 2021-08-27 at 15:18 +0000, Kuber, Esteban wrote:
> On 2021/8/27, 2:03 AM, "Richard Purdie" <richard.purdie@linuxfoundation.org> wrote:
>     > and:
>     >
>     > 2. a reproducible build failure on CentOS-7:
>     >
>     >     https://autobuilder.yoctoproject.org/typhoon/#/builders/115/builds/597
>     >
>     > where, we see:
>     >     = note: /bin/sh: /lib64/libc.so.6: version `GLIBC_2.33' \
>     >     not found (required by \
>     >     /home/pokybuild/yocto-worker/reproducible-centos/build/\
>     >     build-st/reproducibleB/tmp/work/x86_64-linux/cargo-native/\
>     >     1.54.0-r0/recipe-sysroot-native/usr/lib/libtinfo.so.5)
>     >
>     >
>     >
>     >
>     >
>     >     error: linking with `\
>     >     /home/pokybuild/yocto-worker/reproducible-centos/build/\
>     >     build-st/reproducibleB/tmp/work/x86_64-linux/cargo-native/\
>     >     1.54.0-r0/wrapper/target-rust-ccld` failed: exit status: 1
>    
> 
>     I had a quick look at this. It reproduces if you build cargo-native on a centos7
>     machine with our M2 buildtools tarball in the environment of the build.
> 
>     Adding the uninative relocation hack to the cargo snapshot binary with:
> 
>      do_cargo_setup_snapshot () {
>             ${WORKDIR}/rust-snapshot-components/${CARGO_SNAPSHOT}/install.sh --prefix="${WORKDIR}/${CARGO_SNAPSHOT}" --disable-ldconfig
>     +       # Need to use uninative's loader if enabled/present since the library paths
>     +       # are used internally by rust and result in symbol mismatches if we don't
>     +       if [ ! -z "${UNINATIVE_LOADER}" -a -e "${UNINATIVE_LOADER}" ]; then
>     +               patchelf-uninative ${WORKDIR}/${CARGO_SNAPSHOT}/bin/cargo --set-interpreter ${UNINATIVE_LOADER}
>     +       fi
>      }
> 
>     didn't help.
> 
>     Running the command it mentions failing by hand in the same toolchain enabled
>     shell works. It therefore seems likely that something rust is putting into the
>     environment is breaking things. What that is, I don't know, I'm out of time to
>     debug further.
> 
> I'm looking at what that could be. If I can't give you the actual list (from the
> code), I can give you a patch to print out to stderr *everything* that rustc is
> setting during builds. We currently have a `-v` verbose  flag in cargo that
> gives out the full command with wich rustc in invoked, but as you say, the
> problem is likely the environment variables at play.

With some further debugging as I mentioned in my other reply, it is the
LD_LIBRARY_PATH setting which is confusing things. Probably as a result of:

https://doc.rust-lang.org/cargo/reference/environment-variables.html#dynamic-library-paths

>     It looks to me like it is using the ld from the host instead of the buildtools
>     tarball. I did change tmp/work/x86_64-linux/cargo-native/1.54.0-
>     r0/wrapper/target-rust-ccld to a full path to gcc and messed with PATH to ensure
>     it would find "our" ld first but that didn't help.
> 
> This is making me wonder, could it be that we are defaulting to a linker that is
> not supplied in the buildtools tar.gz? That would explain why it ends up picking
> up the system's even when changing the $PATH.

It is a bit more subtle than that. The ccld script has /bin/sh as the
interpreter which uses the host libc and host libtinfo. The LD_LIBRARY_PATH
injected by cargo causes it to find the libtinfo in the recipe-sysroot-native
which is incompatible with it and things then fail :(.

> 
>     In the error output is some:
> 
>     Usage: which [options] [--] COMMAND [...]
>     Write the full path of COMMAND(s) to standard output.
> 
>     suggesting some which call might not be compatible with centos7?
> 
>     Cheers,
> 
>     Richard
> 
> One clarification I would like to have is, is this the *first* time you are 
> trying to build rustc in this configuration, or is this a *recent* problem 
> introduced in 1.54? Just want to make sure that my understanding that we are 
> dealing with the former is correct.

This is the first time we've tried this as far as I know.

It is a horrible mix of different host tools and libcs from uninative,
buildtools and the host all conflicting when LD_LIBRARY_PATH is set :(.

I'm not sure how to go about resolving this.

Cheers,

Richard