Re: [OE-core] linux-yocto 5.15.48 reproducibility failure

[Alexandre Belloni <alexandre.belloni@bootlin.com>]

On 04/08/2022 09:38:44-0400, Bruce Ashfield wrote:
> On Thu, Aug 4, 2022 at 3:27 AM Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> >
> > On Thu, 2022-08-04 at 07:57 +0100, Richard Purdie via
> > lists.openembedded.org wrote:
> > > On Wed, 2022-08-03 at 22:33 -0400, Bruce Ashfield wrote:
> > > > On Wed, Aug 3, 2022 at 6:46 PM Richard Purdie
> > > > <richard.purdie@linuxfoundation.org> wrote:
> > > > >
> > > > > On Wed, 2022-08-03 at 11:50 -1000, Steve Sakoman wrote:
> > > > > > On Wed, Jul 6, 2022 at 4:05 AM Bruce Ashfield <bruce.ashfield@gmail.com> wrote:
> > > > > > >
> > > > > > > On Wed, Jul 6, 2022 at 9:31 AM Richard Purdie
> > > > > > > <richard.purdie@linuxfoundation.org> wrote:
> > > > > > > >
> > > > > > > > On Wed, 2022-07-06 at 15:29 +0200, Alexandre Belloni wrote:
> > > > > > > > > Hello Bruce,
> > > > > > > > >
> > > > > > > > > I got the following reproducible build failures:
> > > > > > > > >
> > > > > > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/1116/steps/12/logs/stdio
> > > > > > > > >
> > > > > > > > > To me, it looks like the in-kernel config changed, did the compression
> > > > > > > > > alg change?
> > > > > > > > >
> > > > > > > > > Are you able to look at that soon or should I file a bug?
> > > > > > > >
> > > > > > > > FWIW,
> > > > > > > > http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20220705-g8ltxczy/packages/diff-html/
> > > > > > > >
> > > > > > > > and CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT=y changed value...
> > > > > > >
> > > > > > > A quick look at the -stable queue to init/Kconfig does indeed show
> > > > > > > that option arriving .. from my scan of the change, it looks like
> > > > > > > there are different compilers on the hosts, and hence that check is
> > > > > > > detecting a buggy gcc (or not) and changing the value.
> > > > > > >
> > > > > > > So one of the build hosts is failing the test, and another isn't.
> > > > > > >
> > > > > > > I'm not sure how to fix that, unless we revert the change. I'll ponder it more.
> > > > > >
> > > > > > Was this issue ever resolved?  I'm seeing it on kirkstone now too :-(
> > > > >
> > > > > No, it wasn't. I took a quick look and my best guess is that we now
> > > > > have the kernel wanting to do $CC tests in kConfig and we don't set CC
> > > > > properly for all our make config calls. I chucked this in master-next
> > > > > as a quick guess at a possible fix:
> > > > >
> > > >
> > > > That's what we've had to do with the linker in the past, so that is
> > > > generally speaking the right place to force the compiler.
> > > >
> > > > > https://git.yoctoproject.org/poky/commit/?h=master-next&id=944cc04d1ed316a5652f3d058b966d4cfb9a6b73
> > > > >
> > > > > but others might know better where we might be missing the CC settings.
> > > > > I suspect we need to clean up our parameters a bit looking at the
> > > > > different call site differences.
> > > >
> > > > That being said, merge_config uses the compiler like so:
> > > >
> > > > # Use the merged file as the starting point for:
> > > > # alldefconfig: Fills in any missing symbols with Kconfig default
> > > > # allnoconfig: Fills in any missing symbols with # CONFIG_* is not set
> > > > make HOSTCC="$HOSTCC" HOSTLD="$HOSTLD" CC="$CC" LD="$LD" \
> > > >         KCONFIG_ALLCONFIG=$TMP_FILE $OUTPUT_ARG $ALLTARGET
> > > >
> > > > And we call it like this:
> > > >
> > > > CFLAGS="${CFLAGS} ${TOOLCHAIN_OPTIONS}" HOSTCC="${BUILD_CC}
> > > > ${BUILD_CFLAGS} ${BUILD_LDFLAGS}" HOSTCPP="${BUILD_CPP}"
> > > > CC="${KERNEL_CC}" LD="${KERNEL_LD}" ARCH=${ARCH} merge_config.sh -O
> > > > ${B} ${config_flags} ${configs} >
> > > > ${meta_dir}/cfg/merge_config_build.log 2>&1
> > > >
> > > > So for the merging of fragments, we are already covered.
> > > >
> > > > and kernel_do_configure calls: KERNEL_CONFIG_COMMAND as the last thing
> > > > it does, which is:
> > > >
> > > > KERNEL_CONFIG_COMMAND ?= "oe_runmake_call -C ${S} CC="${KERNEL_CC}"
> > > > LD="${KERNEL_LD}" O=${B} olddefconfig || oe_runmake -C ${S} O=${B}
> > > > CC="${KERNEL_CC}" LD="${KERNEL_LD}" oldnoconfig"
> > > >
> > > > Which already does set CC to KERNEL_CC.
> > > >
> > > > So unless I'm missing how KCONFIG_CONFIG_COMMAND is sneaking in, I
> > > > expect the behaviour will still be the same.
> > > >
> > > > I spent the day digging through 5.19 kernel issues, so only had a few
> > > > minutes to look at this tonight .. and it is late, so I very easily
> > > > could have missed something.
> > >
> > > You're not missing anything, it didn't help.
> > >
> > > We're seeing this on ubuntu1804 so I logged in there and experimented.
> > > CC is set correctly. What isn't working well is the "echo". If I change
> > > the command to use /bin/echo instead of plain "echo" (which is probably
> > > a shell builtin), it works. Otherwise running manually I see:
> > >
> > > <stdin>: In function 'foo':
> > > <stdin>:1:29: warning: missing terminating " character
> > > <stdin>:1:29: error: missing terminating " character
> > > <stdin>:2:5: error: expected ':' before '+' token
> > > <stdin>:2:7: warning: missing terminating " character
> > > <stdin>:2:7: error: missing terminating " character
> > > <stdin>:2:1: error: expected declaration or statement at end of input
> > >
> > > It is looking like this is a dash vs. bash issue. Not sure how we fix
> > > it yet but it points to where the issue may be.
> >
> > pokybuild@ubuntu1804-ty-3:~$ /bin/dash
> > $ echo 'int foo(int *x) { asm goto (".long (%l[bar]) - .\n": "+m"(*x) ::: bar); return *x; bar: return 0; }' | x86_64-poky-linux-gcc -x c - -c -o /dev/null; echo $?
> > <stdin>: In function 'foo':
> > <stdin>:1:29: warning: missing terminating " character
> > <stdin>:1:29: error: missing terminating " character
> > <stdin>:2:5: error: expected ':' before '+' token
> > <stdin>:2:7: warning: missing terminating " character
> > <stdin>:2:7: error: missing terminating " character
> > <stdin>:2:1: error: expected declaration or statement at end of input
> > 1
> > $ exit
> > pokybuild@ubuntu1804-ty-3:~$ bash
> > pokybuild@ubuntu1804-ty-3:~$ echo 'int foo(int *x) { asm goto (".long (%l[bar]) - .\n": "+m"(*x) ::: bar); return *x; bar: return 0; }' | x86_64-poky-linux-gcc -x c - -c -o /dev/null; echo $?
> > 0
> >
> > I think the command gets executed by the code in
> > scripts/kconfig/preprocess.c:do_shell() which passes it to popen()
> > which uses /bin/sh. Since this command has characters with interesting
> > quoting requirements, I suspect our easiest solution is to submit a
> > patch upstream which moves the content to a file which we reference in
> > the command instead of using echo. cc-can-link.sh and similar kind of
> > give some prior art for it.
> >
> 
> Interesting. Since you can change the shell for Make, I could have
> sworn we could change the shell in that environment as well.
> 
> Either way, it looks like this is another kernel build bash host side
> requirement. Which is ok, since the kernel  build has never claimed
> multi-shell support.
> 
> We can obviously test against linux-yocto. Once I get a few more 5.19
> builds passing, I can have a go at it (unless you've already tried
> something on that front).
> 

I submitted a patch yesterday, I didn't get any reply yet but we are in
the middle of the merge window:
https://lore.kernel.org/all/20220804190320.262510-1-alexandre.belloni@bootlin.com/

I also sent that to the AB.


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com