Re: [OE-core] [v4] [RFC] Merge meta-rust to oe-core - Aug 19 update

["Richard Purdie" <richard.purdie@linuxfoundation.org>]

On Fri, 2021-08-27 at 13:04 +0100, Richard Purdie via lists.openembedded.org
wrote:
> On Fri, 2021-08-27 at 10:03 +0100, Richard Purdie via lists.openembedded.org
> wrote:
> > On Fri, 2021-08-27 at 00:05 -0400, Randy MacLeod wrote:
> > > 2. a reproducible build failure on CentOS-7:
> > > 
> > >     https://autobuilder.yoctoproject.org/typhoon/#/builders/115/builds/597
> > > 
> > > where, we see:
> > >     = note: /bin/sh: /lib64/libc.so.6: version `GLIBC_2.33' \
> > >     not found (required by \
> > >     /home/pokybuild/yocto-worker/reproducible-centos/build/\
> > >     build-st/reproducibleB/tmp/work/x86_64-linux/cargo-native/\
> > >     1.54.0-r0/recipe-sysroot-native/usr/lib/libtinfo.so.5)
> > > 
> > > 
> > > 
> > > 
> > > 
> > >     error: linking with `\
> > >     /home/pokybuild/yocto-worker/reproducible-centos/build/\
> > >     build-st/reproducibleB/tmp/work/x86_64-linux/cargo-native/\
> > >     1.54.0-r0/wrapper/target-rust-ccld` failed: exit status: 1
> > > 
> > 
> > I had a quick look at this. It reproduces if you build cargo-native on a centos7
> > machine with our M2 buildtools tarball in the environment of the build.
> > 
> > Adding the uninative relocation hack to the cargo snapshot binary with:
> > 
> >  do_cargo_setup_snapshot () {
> >         ${WORKDIR}/rust-snapshot-components/${CARGO_SNAPSHOT}/install.sh --prefix="${WORKDIR}/${CARGO_SNAPSHOT}" --disable-ldconfig
> > +       # Need to use uninative's loader if enabled/present since the library paths
> > +       # are used internally by rust and result in symbol mismatches if we don't
> > +       if [ ! -z "${UNINATIVE_LOADER}" -a -e "${UNINATIVE_LOADER}" ]; then
> > +               patchelf-uninative ${WORKDIR}/${CARGO_SNAPSHOT}/bin/cargo --set-interpreter ${UNINATIVE_LOADER}
> > +       fi
> >  }
> > 
> > didn't help.
> > 
> > Running the command it mentions failing by hand in the same toolchain enabled
> > shell works. It therefore seems likely that something rust is putting into the
> > environment is breaking things. What that is, I don't know, I'm out of time to
> > debug further.
> > 
> > It looks to me like it is using the ld from the host instead of the buildtools
> > tarball. I did change tmp/work/x86_64-linux/cargo-native/1.54.0-
> > r0/wrapper/target-rust-ccld to a full path to gcc and messed with PATH to ensure
> > it would find "our" ld first but that didn't help.
> > 
> > In the error output is some:
> > 
> > Usage: which [options] [--] COMMAND [...]
> > Write the full path of COMMAND(s) to standard output.
> > 
> > suggesting some which call might not be compatible with centos7?
> 
> temp/run.do_compile:	bbnote "cargo = $(which /home/pokybuild/yocto-worker/reproducible-centos/build/build-st/tmp2/work/x86_64-linux/cargo-native/1.54.0-r0/cargo-1.53.0-x86_64-unknown-linux-gnu/bin/cargo)"
> temp/run.do_compile:	bbnote "rustc = $(which ${RUSTC})"
> 
> so probably not important.
> 

Distilling through all the noise, we reach this test case:

where you:

a) install buildtools-tarball and source the buildtools env
b) build cargo-native

then the failure can be reproduced with something like:

[pokybuild@centos7-ty-4 build-st]$ LD_LIBRARY_PATH=/home/pokybuild/yocto-worker/reproducible-centos/build/build-st/tmp/work/x86_64-linux/cargo-native/1.54.0-r0/recipe-sysroot-native/usr/lib /home/pokybuild/yocto-worker/reproducible-centos/build/build-st/tmp/work/x86_64-linux/cargo-native/1.54.0-r0/wrapper/target-rust-ccld
/bin/sh: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by /home/pokybuild/yocto-worker/reproducible-centos/build/build-st/tmp/work/x86_64-linux/cargo-native/1.54.0-r0/recipe-sysroot-native/usr/lib/libtinfo.so.5)

which is because /bin/sh comes from the host and links against libtinfo,
LD_LIBRARY_PATH is set by cargo to the recipe sysroot, it finds our libtinfo
rather than the host one and it is incompatible with the libc/loader being used
by /bin/sh.

I did find this is documented in: 

https://doc.rust-lang.org/cargo/reference/environment-variables.html#dynamic-library-paths

and I suspect this is because it is adding "the rustc sysroot library path".

This also explains why running the reported ccld command by hand works since the
LD_LIBRARY_PATH is no longer set.

How to fix this? No idea.

Could/should we make rust use a different sysroot location?

Cheers,

Richard