[PATCH] kernel-yocto: do_configme: Add $TOOLCHAIN_OPTIONS to CFLAGS

[PATCH] kernel-yocto: do_configme: Add $TOOLCHAIN_OPTIONS to CFLAGS

[Ioan-Adrian Ratiu]

The kernel configure step was broken by the sysroot poisoning in [1].
In linux-yocto.inc this was fixed by appending $TOOLCHAIN_OPTIONS to
KERNEL_CC, TOOLCHAIN_OPTIONS containing the needed --sysroot arg.

However, [1] also breaks the yocto configure fragment generation:
after parsing the fragments, when make is called on the merged file
to fill in any missing symbols with default values. The call chain is:

do_kernel_configme() -> configme -> merge_config.sh
-> make KCONFIG_ALLCONFIG=$TMP_FILE

This issue was uncaught until now because configme redirects all stdout
and stderr to merge_log.txt prior to calling merge_configs.sh and there
is no return value checking, so building continues and the silent error
is ignored.

Adding $TOOLCHAIN_OPTIONS to $CFLAGS before calling configme fixes this.

TODO: In a future patch this kind of error should be made loud and cause
the build to fail.

[1] http://lists.openembedded.org/pipermail/openembedded-core/2014-October/098253.html

Signed-off-by: Ioan-Adrian Ratiu <adrian.ratiu@ni.com>
---
 meta/classes/kernel-yocto.bbclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/classes/kernel-yocto.bbclass b/meta/classes/kernel-yocto.bbclass
index 231e08d..3a67238 100644
--- a/meta/classes/kernel-yocto.bbclass
+++ b/meta/classes/kernel-yocto.bbclass
@@ -248,6 +248,7 @@ do_kernel_configme() {
 
 	cd ${S}
 	PATH=${PATH}:${S}/scripts/util
+	CFLAGS="${CFLAGS} ${TOOLCHAIN_OPTIONS}"
 	configme ${configmeflags} --reconfig --output ${B} ${LINUX_KERNEL_TYPE} ${KMACHINE}
 	if [ $? -ne 0 ]; then
 		bbfatal_log "Could not configure ${KMACHINE}-${LINUX_KERNEL_TYPE}"
-- 
2.1.4

Re: [PATCH] kernel-yocto: do_configme: Add $TOOLCHAIN_OPTIONS to CFLAGS

[Bruce Ashfield]

On Tue, Aug 4, 2015 at 4:41 AM, Ioan-Adrian Ratiu <adrian.ratiu@ni.com> wrote:
> The kernel configure step was broken by the sysroot poisoning in [1].

Can you be more precise here ? Configuration certainly works for me .. and
has been working since last October.

> In linux-yocto.inc this was fixed by appending $TOOLCHAIN_OPTIONS to
> KERNEL_CC, TOOLCHAIN_OPTIONS containing the needed --sysroot arg.
>
> However, [1] also breaks the yocto configure fragment generation:
> after parsing the fragments, when make is called on the merged file
> to fill in any missing symbols with default values. The call chain is:
>
> do_kernel_configme() -> configme -> merge_config.sh
> -> make KCONFIG_ALLCONFIG=$TMP_FILE
>
> This issue was uncaught until now because configme redirects all stdout
> and stderr to merge_log.txt prior to calling merge_configs.sh and there
> is no return value checking, so building continues and the silent error
> is ignored.
>
> Adding $TOOLCHAIN_OPTIONS to $CFLAGS before calling configme fixes this.
>
> TODO: In a future patch this kind of error should be made loud and cause
> the build to fail.
>

That won't be required, I already have a queue of patches that changes
things significantly.

If you can elaborate on the type of breakage you were seeing, I'll add
this to that queue.

Bruce

> [1] http://lists.openembedded.org/pipermail/openembedded-core/2014-October/098253.html
>
> Signed-off-by: Ioan-Adrian Ratiu <adrian.ratiu@ni.com>
> ---
>  meta/classes/kernel-yocto.bbclass | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/meta/classes/kernel-yocto.bbclass b/meta/classes/kernel-yocto.bbclass
> index 231e08d..3a67238 100644
> --- a/meta/classes/kernel-yocto.bbclass
> +++ b/meta/classes/kernel-yocto.bbclass
> @@ -248,6 +248,7 @@ do_kernel_configme() {
>
>         cd ${S}
>         PATH=${PATH}:${S}/scripts/util
> +       CFLAGS="${CFLAGS} ${TOOLCHAIN_OPTIONS}"
>         configme ${configmeflags} --reconfig --output ${B} ${LINUX_KERNEL_TYPE} ${KMACHINE}
>         if [ $? -ne 0 ]; then
>                 bbfatal_log "Could not configure ${KMACHINE}-${LINUX_KERNEL_TYPE}"
> --
> 2.1.4
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"

Re: [PATCH] kernel-yocto: do_configme: Add $TOOLCHAIN_OPTIONS to CFLAGS

[Adrian Ratiu]

[-- Attachment #1: Type: text/plain, Size: 3974 bytes --]

Bruce Ashfield <bruce.ashfield@gmail.com> wrote on 05.08.2015 05:17:14:
 
> On Tue, Aug 4, 2015 at 4:41 AM, Ioan-Adrian Ratiu <adrian.ratiu@ni.com> 
wrote:
> > The kernel configure step was broken by the sysroot poisoning in [1].
> 
> Can you be more precise here ? Configuration certainly works for me .. 
and
> has been working since last October.

This works because it was fixed in linux-yocto.inc *after* the sysroot 
poisoning.
I've mentioned this in the next line :) But this is not what this fix is 
for,
this fix should have been done at the same time with adding 
$TOOLCHAIN_OPTIONS
to $KERNEL_CC in linux-yocto.inc.

This bug was missed because there is NO error reporting in 
yocto-kernel-tools
merge_configs.sh and the build continues despite the error. I've sent a 
patch to
fail loudly in case merge_configs.sh fails in yocto-kernel-tools. You know 
about
it because you responded to that mail :) These issues are related.

> > In linux-yocto.inc this was fixed by appending $TOOLCHAIN_OPTIONS to
> > KERNEL_CC, TOOLCHAIN_OPTIONS containing the needed --sysroot arg.
> >
> > However, [1] also breaks the yocto configure fragment generation:
> > after parsing the fragments, when make is called on the merged file
> > to fill in any missing symbols with default values. The call chain is:
> >
> > do_kernel_configme() -> configme -> merge_config.sh
> > -> make KCONFIG_ALLCONFIG=$TMP_FILE
> >
> > This issue was uncaught until now because configme redirects all 
stdout
> > and stderr to merge_log.txt prior to calling merge_configs.sh and 
there
> > is no return value checking, so building continues and the silent 
error
> > is ignored.
> >
> > Adding $TOOLCHAIN_OPTIONS to $CFLAGS before calling configme fixes 
this.
> >
> > TODO: In a future patch this kind of error should be made loud and 
cause
> > the build to fail.
> >
> 
> That won't be required, I already have a queue of patches that changes
> things significantly.
> 
> If you can elaborate on the type of breakage you were seeing, I'll add
> this to that queue.

The error I'm seeing is caused by merge_configs.sh from yocto-kernel-tools
which calls make --alldefconfig | --allnoconfig with a poisoned sysroot.

The failure is caused by make not getting a --sysroot argument and this 
error
fails silently because of not checking the return code in 
yocto-kernel-tools
(again, I've sent a patch for this also, you responded to it yesterday).

The correct way to fix this IMO is to add the --sysroot ARG to the 
environment
in kernel-yocto.bbacless do_configme *before* calling the 
yocto-kernel-tools,
thus the tools get called with a sane sysroot.

The exact error is this:

Partial contents of merge_log.txt:
"
HOSTCC  scripts/basic/fixdep
/home/ovidiu/000/git/nilrt/build/tmp-glibc/work-shared/x64/kernel-source/scripts/basic/fixdep.c:106:23: 
fatal error: sys/types.h: No such file or directory
 #include <sys/types.h>
                       ^
compilation terminated.
/home/ovidiu/000/git/nilrt/build/tmp-glibc/work-shared/x64/kernel-source/scripts/basic/Makefile:21: 
recipe for target 'scripts/basic/x86_64-nilrt-linux-fixdep' failed
make[2]: *** [scripts/basic/x86_64-nilrt-linux-fixdep] Error 1
/home/ovidiu/000/git/nilrt/build/tmp-glibc/work-shared/x64/kernel-source/Makefile:428: 
recipe for target 'scripts_basic' failed
make[1]: *** [scripts_basic] Error 2
Makefile:133: recipe for target 'sub-make' failed
make: *** [sub-make] Error 2
"




National Instruments Romania S.R.L.
------------------------------------------------------
B-dul 21 Decembrie 1989, nr. 77, A2
Cluj-Napoca 400604, Romania
C.I.F.: RO17961616  |  O.R.C.: J12/3337/2005
Telefon: +40 264 406428  |  Fax: +40 264 406429
E-mail: office.cluj@ni.com 
Web: romania.ni.com 
 
Vanzari si suport tehnic:
Telefon gratuit : 0800 070071
E-mail vanzari: ni.romania@ni.com
E-mail suport tehnic: techsupport@ni.com

[-- Attachment #2: Type: text/html, Size: 5003 bytes --]

Re: [PATCH] kernel-yocto: do_configme: Add $TOOLCHAIN_OPTIONS to CFLAGS

[Bruce Ashfield]

On Wed, Aug 5, 2015 at 4:39 AM, Adrian Ratiu <adrian.ratiu@ni.com> wrote:
> Bruce Ashfield <bruce.ashfield@gmail.com> wrote on 05.08.2015 05:17:14:
>
>> On Tue, Aug 4, 2015 at 4:41 AM, Ioan-Adrian Ratiu <adrian.ratiu@ni.com>
>> wrote:
>> > The kernel configure step was broken by the sysroot poisoning in [1].
>>
>> Can you be more precise here ? Configuration certainly works for me .. and
>> has been working since last October.
>
> This works because it was fixed in linux-yocto.inc *after* the sysroot
> poisoning.
> I've mentioned this in the next line :) But this is not what this fix is
> for,
> this fix should have been done at the same time with adding
> $TOOLCHAIN_OPTIONS
> to $KERNEL_CC in linux-yocto.inc.
>
> This bug was missed because there is NO error reporting in
> yocto-kernel-tools
> merge_configs.sh and the build continues despite the error. I've sent a
> patch to
> fail loudly in case merge_configs.sh fails in yocto-kernel-tools. You know
> about
> it because you responded to that mail :) These issues are related.

right. And as you saw there .. I don't need that patch. So we might as well
let this one sit as well.

>
>> > In linux-yocto.inc this was fixed by appending $TOOLCHAIN_OPTIONS to
>> > KERNEL_CC, TOOLCHAIN_OPTIONS containing the needed --sysroot arg.
>> >
>> > However, [1] also breaks the yocto configure fragment generation:
>> > after parsing the fragments, when make is called on the merged file
>> > to fill in any missing symbols with default values. The call chain is:
>> >
>> > do_kernel_configme() -> configme -> merge_config.sh
>> > -> make KCONFIG_ALLCONFIG=$TMP_FILE
>> >
>> > This issue was uncaught until now because configme redirects all stdout
>> > and stderr to merge_log.txt prior to calling merge_configs.sh and there
>> > is no return value checking, so building continues and the silent error
>> > is ignored.
>> >
>> > Adding $TOOLCHAIN_OPTIONS to $CFLAGS before calling configme fixes this.
>> >
>> > TODO: In a future patch this kind of error should be made loud and cause
>> > the build to fail.
>> >
>>
>> That won't be required, I already have a queue of patches that changes
>> things significantly.
>>
>> If you can elaborate on the type of breakage you were seeing, I'll add
>> this to that queue.
>
> The error I'm seeing is caused by merge_configs.sh from yocto-kernel-tools
> which calls make --alldefconfig | --allnoconfig with a poisoned sysroot.
>
> The failure is caused by make not getting a --sysroot argument and this
> error
> fails silently because of not checking the return code in yocto-kernel-tools
> (again, I've sent a patch for this also, you responded to it yesterday).
>

I'm pretty familiar with these issues .. having fixed many myself. All I was
looking for was the cut and paste of the error message.

The right thing is to ensure that the scripts and tools for the kernel
config are build long before the configuration merging happens, and
that is what I've done with my re-ordering.

Cheers,

Bruce

> The correct way to fix this IMO is to add the --sysroot ARG to the
> environment
> in kernel-yocto.bbacless do_configme *before* calling the
> yocto-kernel-tools,
> thus the tools get called with a sane sysroot.
>
> The exact error is this:
>
> Partial contents of merge_log.txt:
> "
> HOSTCC  scripts/basic/fixdep
> /home/ovidiu/000/git/nilrt/build/tmp-glibc/work-shared/x64/kernel-source/scripts/basic/fixdep.c:106:23:
> fatal error: sys/types.h: No such file or directory
> #include <sys/types.h>
>                       ^
> compilation terminated.
> /home/ovidiu/000/git/nilrt/build/tmp-glibc/work-shared/x64/kernel-source/scripts/basic/Makefile:21:
> recipe for target 'scripts/basic/x86_64-nilrt-linux-fixdep' failed
> make[2]: *** [scripts/basic/x86_64-nilrt-linux-fixdep] Error 1
> /home/ovidiu/000/git/nilrt/build/tmp-glibc/work-shared/x64/kernel-source/Makefile:428:
> recipe for target 'scripts_basic' failed
> make[1]: *** [scripts_basic] Error 2
> Makefile:133: recipe for target 'sub-make' failed
> make: *** [sub-make] Error 2
> "
>
>
>
> National Instruments Romania S.R.L.
> ------------------------------------------------------
> B-dul 21 Decembrie 1989, nr. 77, A2
> Cluj-Napoca 400604, Romania
> C.I.F.: RO17961616 | O.R.C.: J12/3337/2005
> Telefon: +40 264 406428 | Fax: +40 264 406429
> E-mail: office.cluj@ni.com
> Web: romania.ni.com
>
> Vanzari si suport tehnic:
> Telefon gratuit : 0800 070071
> E-mail vanzari: ni.romania@ni.com
> E-mail suport tehnic: techsupport@ni.com



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"

Re: [PATCH] kernel-yocto: do_configme: Add $TOOLCHAIN_OPTIONS to CFLAGS

[Ioan-Adrian Ratiu]

On 05.08.2015 16:50, Bruce Ashfield wrote:
> On Wed, Aug 5, 2015 at 4:39 AM, Adrian Ratiu <adrian.ratiu@ni.com> wrote:
>> Bruce Ashfield <bruce.ashfield@gmail.com> wrote on 05.08.2015 05:17:14:
>>
>>> On Tue, Aug 4, 2015 at 4:41 AM, Ioan-Adrian Ratiu <adrian.ratiu@ni.com>
>>> wrote:
>>>> The kernel configure step was broken by the sysroot poisoning in [1].
>>>
>>> Can you be more precise here ? Configuration certainly works for me .. and
>>> has been working since last October.
>>
>> This works because it was fixed in linux-yocto.inc *after* the sysroot
>> poisoning.
>> I've mentioned this in the next line :) But this is not what this fix is
>> for,
>> this fix should have been done at the same time with adding
>> $TOOLCHAIN_OPTIONS
>> to $KERNEL_CC in linux-yocto.inc.
>>
>> This bug was missed because there is NO error reporting in
>> yocto-kernel-tools
>> merge_configs.sh and the build continues despite the error. I've sent a
>> patch to
>> fail loudly in case merge_configs.sh fails in yocto-kernel-tools. You know
>> about
>> it because you responded to that mail :) These issues are related.
>
> right. And as you saw there .. I don't need that patch. So we might as well
> let this one sit as well.

It's great if these issues were already fixed somehow different than how 
I proposed. Are those local changes pushed to a public branch yet? I 
would really like to test the new code.

P.S. Sorry for the html crap in my previous mail, I hope it's gone now.

Re: [PATCH] kernel-yocto: do_configme: Add $TOOLCHAIN_OPTIONS to CFLAGS

[Bruce Ashfield]

On Thu, Aug 6, 2015 at 3:23 AM, Ioan-Adrian Ratiu <adrian.ratiu@ni.com> wrote:
> On 05.08.2015 16:50, Bruce Ashfield wrote:
>>
>> On Wed, Aug 5, 2015 at 4:39 AM, Adrian Ratiu <adrian.ratiu@ni.com> wrote:
>>>
>>> Bruce Ashfield <bruce.ashfield@gmail.com> wrote on 05.08.2015 05:17:14:
>>>
>>>> On Tue, Aug 4, 2015 at 4:41 AM, Ioan-Adrian Ratiu <adrian.ratiu@ni.com>
>>>> wrote:
>>>>>
>>>>> The kernel configure step was broken by the sysroot poisoning in [1].
>>>>
>>>>
>>>> Can you be more precise here ? Configuration certainly works for me ..
>>>> and
>>>> has been working since last October.
>>>
>>>
>>> This works because it was fixed in linux-yocto.inc *after* the sysroot
>>> poisoning.
>>> I've mentioned this in the next line :) But this is not what this fix is
>>> for,
>>> this fix should have been done at the same time with adding
>>> $TOOLCHAIN_OPTIONS
>>> to $KERNEL_CC in linux-yocto.inc.
>>>
>>> This bug was missed because there is NO error reporting in
>>> yocto-kernel-tools
>>> merge_configs.sh and the build continues despite the error. I've sent a
>>> patch to
>>> fail loudly in case merge_configs.sh fails in yocto-kernel-tools. You
>>> know
>>> about
>>> it because you responded to that mail :) These issues are related.
>>
>>
>> right. And as you saw there .. I don't need that patch. So we might as
>> well
>> let this one sit as well.
>
>
> It's great if these issues were already fixed somehow different than how I
> proposed. Are those local changes pushed to a public branch yet? I would
> really like to test the new code.

The solution is similar in the end, I just came to adding extra error checking
via a different route (missing .cfg fragments not being reported and stopping
the build).

They are unfortunately back on my development machine, and I'm on
vacation at the moment. But I will be back near that equipment next
week!

Cheers,

Bruce

>
> P.S. Sorry for the html crap in my previous mail, I hope it's gone now.



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"

Re: [PATCH] kernel-yocto: do_configme: Add $TOOLCHAIN_OPTIONS to CFLAGS

[Ioan-Adrian Ratiu]

On 06.08.2015 16:19, Bruce Ashfield wrote:
> On Thu, Aug 6, 2015 at 3:23 AM, Ioan-Adrian Ratiu <adrian.ratiu@ni.com> wrote:
>> On 05.08.2015 16:50, Bruce Ashfield wrote:
>>>
>>> On Wed, Aug 5, 2015 at 4:39 AM, Adrian Ratiu <adrian.ratiu@ni.com> wrote:
>>>>
>>>> Bruce Ashfield <bruce.ashfield@gmail.com> wrote on 05.08.2015 05:17:14:
>>>>
>>>>> On Tue, Aug 4, 2015 at 4:41 AM, Ioan-Adrian Ratiu <adrian.ratiu@ni.com>
>>>>> wrote:
>>>>>>
>>>>>> The kernel configure step was broken by the sysroot poisoning in [1].
>>>>>
>>>>>
>>>>> Can you be more precise here ? Configuration certainly works for me ..
>>>>> and
>>>>> has been working since last October.
>>>>
>>>>
>>>> This works because it was fixed in linux-yocto.inc *after* the sysroot
>>>> poisoning.
>>>> I've mentioned this in the next line :) But this is not what this fix is
>>>> for,
>>>> this fix should have been done at the same time with adding
>>>> $TOOLCHAIN_OPTIONS
>>>> to $KERNEL_CC in linux-yocto.inc.
>>>>
>>>> This bug was missed because there is NO error reporting in
>>>> yocto-kernel-tools
>>>> merge_configs.sh and the build continues despite the error. I've sent a
>>>> patch to
>>>> fail loudly in case merge_configs.sh fails in yocto-kernel-tools. You
>>>> know
>>>> about
>>>> it because you responded to that mail :) These issues are related.
>>>
>>>
>>> right. And as you saw there .. I don't need that patch. So we might as
>>> well
>>> let this one sit as well.
>>
>>
>> It's great if these issues were already fixed somehow different than how I
>> proposed. Are those local changes pushed to a public branch yet? I would
>> really like to test the new code.
>
> The solution is similar in the end, I just came to adding extra error checking
> via a different route (missing .cfg fragments not being reported and stopping
> the build).
>
> They are unfortunately back on my development machine, and I'm on
> vacation at the moment. But I will be back near that equipment next
> week!

I still think that we are solving two different issues altogether with 
these patches, but I think it best we wait until you push your changes 
and after that (if need be) I will reapply my patches on top of your 
changes and resend (again, if need be).

Have a nice vacation!