[PATCH] kernel: yocto: Add dependency on do_prepare_recipe_sysroot

[PATCH] kernel: yocto: Add dependency on do_prepare_recipe_sysroot

[Lukasz Majewski]

Without this patch it happens that do_populate_recipe_sysroot is called
just before do_compile (on multi core build machines).
This is way too late as the .config generated in do_kernel_configme()
is already broken.

The problem is that do_kernel_configme() calls native's merge_config.sh
script which may call bison and arm-linux-*-gcc to compile kconfig
infrastructure to create base .config
(scripts/kconfig/conf  --allnoconfig Kconfig.)

It turns out that we only got above binaries after do_prepare_recipe_sysroot()
is called.
If those binaries are not available, the merge_config.sh do not generate
base .config and without any error produces malfunctioned .config.
The build process continues and as a result broken kernel is compiled.

To reproduce:

bitbake -c cleansstate virtual/kernel
bitbake -c kernel_metadata -v virtual/kernel
bitbake -c do_kernel_configme -v virtual/kernel
(one shall see broken .config in ${B}/.config)
bitbake -c do_populate_sysroot -v virtual/kernel
bitbake -c do_compile -v virtual/kernel

(Poky) SHA1: bb91b2ae3ee5cf108aa2f9b78abb14d5aa00831d

Signed-off-by: Lukasz Majewski <lukma@denx.de>
---
 meta/classes/kernel-yocto.bbclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/classes/kernel-yocto.bbclass b/meta/classes/kernel-yocto.bbclass
index 4ac3a39e47..19a3f2bc46 100644
--- a/meta/classes/kernel-yocto.bbclass
+++ b/meta/classes/kernel-yocto.bbclass
@@ -276,6 +276,7 @@ do_kernel_metadata[depends] = "kern-tools-native:do_populate_sysroot"
 do_validate_branches[depends] = "kern-tools-native:do_populate_sysroot"
 
 do_kernel_configme[dirs] += "${S} ${B}"
+do_kernel_configme[depends] += "virtual/kernel:do_prepare_recipe_sysroot"
 do_kernel_configme() {
 	set +e
 
-- 
2.11.0

Re: [PATCH] kernel: yocto: Add dependency on do_prepare_recipe_sysroot

[Bruce Ashfield]

On 08/22/2018 09:40 AM, Lukasz Majewski wrote:
> Without this patch it happens that do_populate_recipe_sysroot is called
> just before do_compile (on multi core build machines).
> This is way too late as the .config generated in do_kernel_configme()
> is already broken.
> 
> The problem is that do_kernel_configme() calls native's merge_config.sh
> script which may call bison and arm-linux-*-gcc to compile kconfig
> infrastructure to create base .config
> (scripts/kconfig/conf  --allnoconfig Kconfig.)
> 
> It turns out that we only got above binaries after do_prepare_recipe_sysroot()
> is called.
> If those binaries are not available, the merge_config.sh do not generate
> base .config and without any error produces malfunctioned .config.
> The build process continues and as a result broken kernel is compiled.
> 
> To reproduce:
> 
> bitbake -c cleansstate virtual/kernel
> bitbake -c kernel_metadata -v virtual/kernel
> bitbake -c do_kernel_configme -v virtual/kernel
> (one shall see broken .config in ${B}/.config)
> bitbake -c do_populate_sysroot -v virtual/kernel
> bitbake -c do_compile -v virtual/kernel
> 

I already have a queue of changes for the 4.18 kernel that has a
similar tweak.

I'll double check the use case above, and if it isn't already
fixed, I'll add this to my queue.

Bruce

> (Poky) SHA1: bb91b2ae3ee5cf108aa2f9b78abb14d5aa00831d
> 
> Signed-off-by: Lukasz Majewski <lukma@denx.de>
> ---
>   meta/classes/kernel-yocto.bbclass | 1 +
>   1 file changed, 1 insertion(+)
> 
> diff --git a/meta/classes/kernel-yocto.bbclass b/meta/classes/kernel-yocto.bbclass
> index 4ac3a39e47..19a3f2bc46 100644
> --- a/meta/classes/kernel-yocto.bbclass
> +++ b/meta/classes/kernel-yocto.bbclass
> @@ -276,6 +276,7 @@ do_kernel_metadata[depends] = "kern-tools-native:do_populate_sysroot"
>   do_validate_branches[depends] = "kern-tools-native:do_populate_sysroot"
>   
>   do_kernel_configme[dirs] += "${S} ${B}"
> +do_kernel_configme[depends] += "virtual/kernel:do_prepare_recipe_sysroot"
>   do_kernel_configme() {
>   	set +e
>   
> 

Re: [PATCH] kernel: yocto: Add dependency on do_prepare_recipe_sysroot

[Lukasz Majewski]

[-- Attachment #1: Type: text/plain, Size: 2916 bytes --]

Hi Bruce,

> On 08/22/2018 09:40 AM, Lukasz Majewski wrote:
> > Without this patch it happens that do_populate_recipe_sysroot is
> > called just before do_compile (on multi core build machines).
> > This is way too late as the .config generated in
> > do_kernel_configme() is already broken.
> > 
> > The problem is that do_kernel_configme() calls native's
> > merge_config.sh script which may call bison and arm-linux-*-gcc to
> > compile kconfig infrastructure to create base .config
> > (scripts/kconfig/conf  --allnoconfig Kconfig.)
> > 
> > It turns out that we only got above binaries after
> > do_prepare_recipe_sysroot() is called.
> > If those binaries are not available, the merge_config.sh do not
> > generate base .config and without any error produces
> > malfunctioned .config. The build process continues and as a result
> > broken kernel is compiled.
> > 
> > To reproduce:
> > 
> > bitbake -c cleansstate virtual/kernel
> > bitbake -c kernel_metadata -v virtual/kernel
> > bitbake -c do_kernel_configme -v virtual/kernel
> > (one shall see broken .config in ${B}/.config)
> > bitbake -c do_populate_sysroot -v virtual/kernel
> > bitbake -c do_compile -v virtual/kernel
> >   
> 
> I already have a queue of changes for the 4.18 kernel that has a
> similar tweak.
> 
> I'll double check the use case above, and if it isn't already
> fixed, I'll add this to my queue.

One more strange observation the merge_config.sh shall use -native
binaries (like bison, gcc, etc).

Unfortunately, in my case it depends on target ones (as we do have
CFLAGS = .... ARCH=arm ... merge_configs.sh).

The bitbake -g -u taskexp virtual/kernel shows however that we do
depend on native bison.

To provide target ones - I do need the do_prepare_recipe_sysroot() to
call before do_configure.

> 
> Bruce
> 
> > (Poky) SHA1: bb91b2ae3ee5cf108aa2f9b78abb14d5aa00831d
> > 
> > Signed-off-by: Lukasz Majewski <lukma@denx.de>
> > ---
> >   meta/classes/kernel-yocto.bbclass | 1 +
> >   1 file changed, 1 insertion(+)
> > 
> > diff --git a/meta/classes/kernel-yocto.bbclass
> > b/meta/classes/kernel-yocto.bbclass index 4ac3a39e47..19a3f2bc46
> > 100644 --- a/meta/classes/kernel-yocto.bbclass
> > +++ b/meta/classes/kernel-yocto.bbclass
> > @@ -276,6 +276,7 @@ do_kernel_metadata[depends] =
> > "kern-tools-native:do_populate_sysroot"
> > do_validate_branches[depends] =
> > "kern-tools-native:do_populate_sysroot" do_kernel_configme[dirs] +=
> > "${S} ${B}" +do_kernel_configme[depends] +=
> > "virtual/kernel:do_prepare_recipe_sysroot" do_kernel_configme() {
> >   	set +e
> >   
> >   
> 




Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 499 bytes --]

Re: [PATCH] kernel: yocto: Add dependency on do_prepare_recipe_sysroot

[Bruce Ashfield]

On 08/22/2018 09:40 AM, Lukasz Majewski wrote:
> Without this patch it happens that do_populate_recipe_sysroot is called
> just before do_compile (on multi core build machines).
> This is way too late as the .config generated in do_kernel_configme()
> is already broken.
> 
> The problem is that do_kernel_configme() calls native's merge_config.sh
> script which may call bison and arm-linux-*-gcc to compile kconfig
> infrastructure to create base .config
> (scripts/kconfig/conf  --allnoconfig Kconfig.)
> 
> It turns out that we only got above binaries after do_prepare_recipe_sysroot()
> is called.
> If those binaries are not available, the merge_config.sh do not generate
> base .config and without any error produces malfunctioned .config.
> The build process continues and as a result broken kernel is compiled.
> 
> To reproduce:
> 
> bitbake -c cleansstate virtual/kernel
> bitbake -c kernel_metadata -v virtual/kernel
> bitbake -c do_kernel_configme -v virtual/kernel
> (one shall see broken .config in ${B}/.config)
> bitbake -c do_populate_sysroot -v virtual/kernel
> bitbake -c do_compile -v virtual/kernel
> 
> (Poky) SHA1: bb91b2ae3ee5cf108aa2f9b78abb14d5aa00831d
> 
> Signed-off-by: Lukasz Majewski <lukma@denx.de>
> ---
>   meta/classes/kernel-yocto.bbclass | 1 +
>   1 file changed, 1 insertion(+)
> 
> diff --git a/meta/classes/kernel-yocto.bbclass b/meta/classes/kernel-yocto.bbclass
> index 4ac3a39e47..19a3f2bc46 100644
> --- a/meta/classes/kernel-yocto.bbclass
> +++ b/meta/classes/kernel-yocto.bbclass
> @@ -276,6 +276,7 @@ do_kernel_metadata[depends] = "kern-tools-native:do_populate_sysroot"
>   do_validate_branches[depends] = "kern-tools-native:do_populate_sysroot"
>   
>   do_kernel_configme[dirs] += "${S} ${B}"
> +do_kernel_configme[depends] += "virtual/kernel:do_prepare_recipe_sysroot"

On a closer look .. what branch is this from ? We already have a
change that triggered breakage on 4.17+ that should fix this.

i.e. commit ff1bdd75d50f0ebac3d599e461685ace29559a82 in oe-core, adds:

----------

  do_kernel_metadata[depends] = "kern-tools-native:do_populate_sysroot"
  do_validate_branches[depends] = "kern-tools-native:do_populate_sysroot"

+do_kernel_configme[depends] += 
"virtual/${TARGET_PREFIX}binutils:do_populate_sysroot"
+do_kernel_configme[depends] += 
"virtual/${TARGET_PREFIX}gcc:do_populate_sysroot"
+do_kernel_configme[depends] += "bc-native:do_populate_sysroot 
bison-native:do_populate_sysroot"
  do_kernel_configme[dirs] += "${S} ${B}"
  do_kernel_configme() {

----------

Which I don't see in the context of your patch.

Bruce

>   do_kernel_configme() {
>   	set +e
>   
> 

Re: [PATCH] kernel: yocto: Add dependency on do_prepare_recipe_sysroot

[Lukasz Majewski]

[-- Attachment #1: Type: text/plain, Size: 3865 bytes --]

Hi Bruce,

> On 08/22/2018 09:40 AM, Lukasz Majewski wrote:
> > Without this patch it happens that do_populate_recipe_sysroot is
> > called just before do_compile (on multi core build machines).
> > This is way too late as the .config generated in
> > do_kernel_configme() is already broken.
> > 
> > The problem is that do_kernel_configme() calls native's
> > merge_config.sh script which may call bison and arm-linux-*-gcc to
> > compile kconfig infrastructure to create base .config
> > (scripts/kconfig/conf  --allnoconfig Kconfig.)
> > 
> > It turns out that we only got above binaries after
> > do_prepare_recipe_sysroot() is called.
> > If those binaries are not available, the merge_config.sh do not
> > generate base .config and without any error produces
> > malfunctioned .config. The build process continues and as a result
> > broken kernel is compiled.
> > 
> > To reproduce:
> > 
> > bitbake -c cleansstate virtual/kernel
> > bitbake -c kernel_metadata -v virtual/kernel
> > bitbake -c do_kernel_configme -v virtual/kernel
> > (one shall see broken .config in ${B}/.config)
> > bitbake -c do_populate_sysroot -v virtual/kernel
> > bitbake -c do_compile -v virtual/kernel
> > 
> > (Poky) SHA1: bb91b2ae3ee5cf108aa2f9b78abb14d5aa00831d
> > 
> > Signed-off-by: Lukasz Majewski <lukma@denx.de>
> > ---
> >   meta/classes/kernel-yocto.bbclass | 1 +
> >   1 file changed, 1 insertion(+)
> > 
> > diff --git a/meta/classes/kernel-yocto.bbclass
> > b/meta/classes/kernel-yocto.bbclass index 4ac3a39e47..19a3f2bc46
> > 100644 --- a/meta/classes/kernel-yocto.bbclass
> > +++ b/meta/classes/kernel-yocto.bbclass
> > @@ -276,6 +276,7 @@ do_kernel_metadata[depends] =
> > "kern-tools-native:do_populate_sysroot"
> > do_validate_branches[depends] =
> > "kern-tools-native:do_populate_sysroot" do_kernel_configme[dirs] +=
> > "${S} ${B}" +do_kernel_configme[depends] +=
> > "virtual/kernel:do_prepare_recipe_sysroot"  
> 
> On a closer look .. what branch is this from ? We already have a
> change that triggered breakage on 4.17+ that should fix this.
> 
> i.e. commit ff1bdd75d50f0ebac3d599e461685ace29559a82 in oe-core, adds:
> 
> ----------
> 
>   do_kernel_metadata[depends] =
> "kern-tools-native:do_populate_sysroot" do_validate_branches[depends]
> = "kern-tools-native:do_populate_sysroot"
> 
> +do_kernel_configme[depends] += 
> "virtual/${TARGET_PREFIX}binutils:do_populate_sysroot"
> +do_kernel_configme[depends] += 
> "virtual/${TARGET_PREFIX}gcc:do_populate_sysroot"
> +do_kernel_configme[depends] += "bc-native:do_populate_sysroot 
> bison-native:do_populate_sysroot"
>   do_kernel_configme[dirs] += "${S} ${B}"
>   do_kernel_configme() {
> 
> ----------
> 
> Which I don't see in the context of your patch.
> 

I do use poky as in the commit message.

Regarding the kernel - I do have my own recipe;
inherit kernel 

require
recipes-kernel/linux/linux-yocto.inc                                                                                                                                                                                                   


And then I do have:
COMPATIBLE_MACHINE_display5 = "display5" 

KBUILD_DEFCONFIG_pn-linux-yocto-custom_display5 =
"imx6q_display5_defconfig"                                                                                                                                                                   

As the board is in mainline I just point to Linus tree.

> Bruce
> 
> >   do_kernel_configme() {
> >   	set +e
> >   
> >   
> 




Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 499 bytes --]

Re: [PATCH] kernel: yocto: Add dependency on do_prepare_recipe_sysroot

[Bruce Ashfield]

On 08/22/2018 10:05 AM, Lukasz Majewski wrote:
> Hi Bruce,
> 
>> On 08/22/2018 09:40 AM, Lukasz Majewski wrote:
>>> Without this patch it happens that do_populate_recipe_sysroot is
>>> called just before do_compile (on multi core build machines).
>>> This is way too late as the .config generated in
>>> do_kernel_configme() is already broken.
>>>
>>> The problem is that do_kernel_configme() calls native's
>>> merge_config.sh script which may call bison and arm-linux-*-gcc to
>>> compile kconfig infrastructure to create base .config
>>> (scripts/kconfig/conf  --allnoconfig Kconfig.)
>>>
>>> It turns out that we only got above binaries after
>>> do_prepare_recipe_sysroot() is called.
>>> If those binaries are not available, the merge_config.sh do not
>>> generate base .config and without any error produces
>>> malfunctioned .config. The build process continues and as a result
>>> broken kernel is compiled.
>>>
>>> To reproduce:
>>>
>>> bitbake -c cleansstate virtual/kernel
>>> bitbake -c kernel_metadata -v virtual/kernel
>>> bitbake -c do_kernel_configme -v virtual/kernel
>>> (one shall see broken .config in ${B}/.config)
>>> bitbake -c do_populate_sysroot -v virtual/kernel
>>> bitbake -c do_compile -v virtual/kernel
>>>
>>> (Poky) SHA1: bb91b2ae3ee5cf108aa2f9b78abb14d5aa00831d
>>>
>>> Signed-off-by: Lukasz Majewski <lukma@denx.de>
>>> ---
>>>    meta/classes/kernel-yocto.bbclass | 1 +
>>>    1 file changed, 1 insertion(+)
>>>
>>> diff --git a/meta/classes/kernel-yocto.bbclass
>>> b/meta/classes/kernel-yocto.bbclass index 4ac3a39e47..19a3f2bc46
>>> 100644 --- a/meta/classes/kernel-yocto.bbclass
>>> +++ b/meta/classes/kernel-yocto.bbclass
>>> @@ -276,6 +276,7 @@ do_kernel_metadata[depends] =
>>> "kern-tools-native:do_populate_sysroot"
>>> do_validate_branches[depends] =
>>> "kern-tools-native:do_populate_sysroot" do_kernel_configme[dirs] +=
>>> "${S} ${B}" +do_kernel_configme[depends] +=
>>> "virtual/kernel:do_prepare_recipe_sysroot"
>>
>> On a closer look .. what branch is this from ? We already have a
>> change that triggered breakage on 4.17+ that should fix this.
>>
>> i.e. commit ff1bdd75d50f0ebac3d599e461685ace29559a82 in oe-core, adds:
>>
>> ----------
>>
>>    do_kernel_metadata[depends] =
>> "kern-tools-native:do_populate_sysroot" do_validate_branches[depends]
>> = "kern-tools-native:do_populate_sysroot"
>>
>> +do_kernel_configme[depends] +=
>> "virtual/${TARGET_PREFIX}binutils:do_populate_sysroot"
>> +do_kernel_configme[depends] +=
>> "virtual/${TARGET_PREFIX}gcc:do_populate_sysroot"
>> +do_kernel_configme[depends] += "bc-native:do_populate_sysroot
>> bison-native:do_populate_sysroot"
>>    do_kernel_configme[dirs] += "${S} ${B}"
>>    do_kernel_configme() {
>>
>> ----------
>>
>> Which I don't see in the context of your patch.
>>
> 
> I do use poky as in the commit message.

Yes, I checked that .. but that isn't a useful part of a patch
to oe-core.

The kernel-yocto bbclass in that context is not the one in master, and
if I run a --contains on that commit ID, I don't get an active
branch.

So the patch as it stands, cannot be applied to master (and I
also checked master-next in case something snuck in that I
missed) of oe-core or poky, so there's no way to work with it.

Bruce

> 
> Regarding the kernel - I do have my own recipe;
> inherit kernel
> 
> require
> recipes-kernel/linux/linux-yocto.inc
> 
> 
> And then I do have:
> COMPATIBLE_MACHINE_display5 = "display5"
> 
> KBUILD_DEFCONFIG_pn-linux-yocto-custom_display5 =
> "imx6q_display5_defconfig"
> 
> As the board is in mainline I just point to Linus tree.
> 
>> Bruce
>>
>>>    do_kernel_configme() {
>>>    	set +e
>>>    
>>>    
>>
> 
> 
> 
> 
> Best regards,
> 
> Lukasz Majewski
> 
> --
> 
> DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
> Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
> 

Re: [PATCH] kernel: yocto: Add dependency on do_prepare_recipe_sysroot

[Lukasz Majewski]

[-- Attachment #1: Type: text/plain, Size: 4880 bytes --]

On Wed, 22 Aug 2018 10:13:33 -0400
Bruce Ashfield <bruce.ashfield@windriver.com> wrote:

> On 08/22/2018 10:05 AM, Lukasz Majewski wrote:
> > Hi Bruce,
> >   
> >> On 08/22/2018 09:40 AM, Lukasz Majewski wrote:  
> >>> Without this patch it happens that do_populate_recipe_sysroot is
> >>> called just before do_compile (on multi core build machines).
> >>> This is way too late as the .config generated in
> >>> do_kernel_configme() is already broken.
> >>>
> >>> The problem is that do_kernel_configme() calls native's
> >>> merge_config.sh script which may call bison and arm-linux-*-gcc to
> >>> compile kconfig infrastructure to create base .config
> >>> (scripts/kconfig/conf  --allnoconfig Kconfig.)
> >>>
> >>> It turns out that we only got above binaries after
> >>> do_prepare_recipe_sysroot() is called.
> >>> If those binaries are not available, the merge_config.sh do not
> >>> generate base .config and without any error produces
> >>> malfunctioned .config. The build process continues and as a result
> >>> broken kernel is compiled.
> >>>
> >>> To reproduce:
> >>>
> >>> bitbake -c cleansstate virtual/kernel
> >>> bitbake -c kernel_metadata -v virtual/kernel
> >>> bitbake -c do_kernel_configme -v virtual/kernel
> >>> (one shall see broken .config in ${B}/.config)
> >>> bitbake -c do_populate_sysroot -v virtual/kernel
> >>> bitbake -c do_compile -v virtual/kernel
> >>>
> >>> (Poky) SHA1: bb91b2ae3ee5cf108aa2f9b78abb14d5aa00831d
> >>>
> >>> Signed-off-by: Lukasz Majewski <lukma@denx.de>
> >>> ---
> >>>    meta/classes/kernel-yocto.bbclass | 1 +
> >>>    1 file changed, 1 insertion(+)
> >>>
> >>> diff --git a/meta/classes/kernel-yocto.bbclass
> >>> b/meta/classes/kernel-yocto.bbclass index 4ac3a39e47..19a3f2bc46
> >>> 100644 --- a/meta/classes/kernel-yocto.bbclass
> >>> +++ b/meta/classes/kernel-yocto.bbclass
> >>> @@ -276,6 +276,7 @@ do_kernel_metadata[depends] =
> >>> "kern-tools-native:do_populate_sysroot"
> >>> do_validate_branches[depends] =
> >>> "kern-tools-native:do_populate_sysroot" do_kernel_configme[dirs]
> >>> += "${S} ${B}" +do_kernel_configme[depends] +=
> >>> "virtual/kernel:do_prepare_recipe_sysroot"  
> >>
> >> On a closer look .. what branch is this from ? We already have a
> >> change that triggered breakage on 4.17+ that should fix this.
> >>
> >> i.e. commit ff1bdd75d50f0ebac3d599e461685ace29559a82 in oe-core,
> >> adds:
> >>
> >> ----------
> >>
> >>    do_kernel_metadata[depends] =
> >> "kern-tools-native:do_populate_sysroot"
> >> do_validate_branches[depends] =
> >> "kern-tools-native:do_populate_sysroot"
> >>
> >> +do_kernel_configme[depends] +=
> >> "virtual/${TARGET_PREFIX}binutils:do_populate_sysroot"
> >> +do_kernel_configme[depends] +=
> >> "virtual/${TARGET_PREFIX}gcc:do_populate_sysroot"
> >> +do_kernel_configme[depends] += "bc-native:do_populate_sysroot
> >> bison-native:do_populate_sysroot"
> >>    do_kernel_configme[dirs] += "${S} ${B}"
> >>    do_kernel_configme() {
> >>
> >> ----------
> >>
> >> Which I don't see in the context of your patch.
> >>  
> > 
> > I do use poky as in the commit message.  
> 
> Yes, I checked that .. but that isn't a useful part of a patch
> to oe-core.
> 
> The kernel-yocto bbclass in that context is not the one in master, and
> if I run a --contains on that commit ID, I don't get an active
> branch.

I might have misunderstood something... apparently.

The changes which I did were to:

poky/meta/classes/kernel-yocto.bbclass

Is this a wrong place to add the fix?

> 
> So the patch as it stands, cannot be applied to master (and I
> also checked master-next in case something snuck in that I
> missed) of oe-core or poky, so there's no way to work with it.
> 
> Bruce
> 
> > 
> > Regarding the kernel - I do have my own recipe;
> > inherit kernel
> > 
> > require
> > recipes-kernel/linux/linux-yocto.inc
> > 
> > 
> > And then I do have:
> > COMPATIBLE_MACHINE_display5 = "display5"
> > 
> > KBUILD_DEFCONFIG_pn-linux-yocto-custom_display5 =
> > "imx6q_display5_defconfig"
> > 
> > As the board is in mainline I just point to Linus tree.
> >   
> >> Bruce
> >>  
> >>>    do_kernel_configme() {
> >>>    	set +e
> >>>    
> >>>      
> >>  
> > 
> > 
> > 
> > 
> > Best regards,
> > 
> > Lukasz Majewski
> > 
> > --
> > 
> > DENX Software Engineering GmbH,      Managing Director: Wolfgang
> > Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell,
> > Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email:
> > wd@denx.de 
> 




Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 499 bytes --]

Re: [PATCH] kernel: yocto: Add dependency on do_prepare_recipe_sysroot

[Bruce Ashfield]

On 08/22/2018 10:20 AM, Lukasz Majewski wrote:
> On Wed, 22 Aug 2018 10:13:33 -0400
> Bruce Ashfield <bruce.ashfield@windriver.com> wrote:
> 
>> On 08/22/2018 10:05 AM, Lukasz Majewski wrote:
>>> Hi Bruce,
>>>    
>>>> On 08/22/2018 09:40 AM, Lukasz Majewski wrote:
>>>>> Without this patch it happens that do_populate_recipe_sysroot is
>>>>> called just before do_compile (on multi core build machines).
>>>>> This is way too late as the .config generated in
>>>>> do_kernel_configme() is already broken.
>>>>>
>>>>> The problem is that do_kernel_configme() calls native's
>>>>> merge_config.sh script which may call bison and arm-linux-*-gcc to
>>>>> compile kconfig infrastructure to create base .config
>>>>> (scripts/kconfig/conf  --allnoconfig Kconfig.)
>>>>>
>>>>> It turns out that we only got above binaries after
>>>>> do_prepare_recipe_sysroot() is called.
>>>>> If those binaries are not available, the merge_config.sh do not
>>>>> generate base .config and without any error produces
>>>>> malfunctioned .config. The build process continues and as a result
>>>>> broken kernel is compiled.
>>>>>
>>>>> To reproduce:
>>>>>
>>>>> bitbake -c cleansstate virtual/kernel
>>>>> bitbake -c kernel_metadata -v virtual/kernel
>>>>> bitbake -c do_kernel_configme -v virtual/kernel
>>>>> (one shall see broken .config in ${B}/.config)
>>>>> bitbake -c do_populate_sysroot -v virtual/kernel
>>>>> bitbake -c do_compile -v virtual/kernel
>>>>>
>>>>> (Poky) SHA1: bb91b2ae3ee5cf108aa2f9b78abb14d5aa00831d
>>>>>
>>>>> Signed-off-by: Lukasz Majewski <lukma@denx.de>
>>>>> ---
>>>>>     meta/classes/kernel-yocto.bbclass | 1 +
>>>>>     1 file changed, 1 insertion(+)
>>>>>
>>>>> diff --git a/meta/classes/kernel-yocto.bbclass
>>>>> b/meta/classes/kernel-yocto.bbclass index 4ac3a39e47..19a3f2bc46
>>>>> 100644 --- a/meta/classes/kernel-yocto.bbclass
>>>>> +++ b/meta/classes/kernel-yocto.bbclass
>>>>> @@ -276,6 +276,7 @@ do_kernel_metadata[depends] =
>>>>> "kern-tools-native:do_populate_sysroot"
>>>>> do_validate_branches[depends] =
>>>>> "kern-tools-native:do_populate_sysroot" do_kernel_configme[dirs]
>>>>> += "${S} ${B}" +do_kernel_configme[depends] +=
>>>>> "virtual/kernel:do_prepare_recipe_sysroot"
>>>>
>>>> On a closer look .. what branch is this from ? We already have a
>>>> change that triggered breakage on 4.17+ that should fix this.
>>>>
>>>> i.e. commit ff1bdd75d50f0ebac3d599e461685ace29559a82 in oe-core,
>>>> adds:
>>>>
>>>> ----------
>>>>
>>>>     do_kernel_metadata[depends] =
>>>> "kern-tools-native:do_populate_sysroot"
>>>> do_validate_branches[depends] =
>>>> "kern-tools-native:do_populate_sysroot"
>>>>
>>>> +do_kernel_configme[depends] +=
>>>> "virtual/${TARGET_PREFIX}binutils:do_populate_sysroot"
>>>> +do_kernel_configme[depends] +=
>>>> "virtual/${TARGET_PREFIX}gcc:do_populate_sysroot"
>>>> +do_kernel_configme[depends] += "bc-native:do_populate_sysroot
>>>> bison-native:do_populate_sysroot"
>>>>     do_kernel_configme[dirs] += "${S} ${B}"
>>>>     do_kernel_configme() {
>>>>
>>>> ----------
>>>>
>>>> Which I don't see in the context of your patch.
>>>>   
>>>
>>> I do use poky as in the commit message.
>>
>> Yes, I checked that .. but that isn't a useful part of a patch
>> to oe-core.
>>
>> The kernel-yocto bbclass in that context is not the one in master, and
>> if I run a --contains on that commit ID, I don't get an active
>> branch.
> 
> I might have misunderstood something... apparently.
> 
> The changes which I did were to:
> 
> poky/meta/classes/kernel-yocto.bbclass
> 
> Is this a wrong place to add the fix?

It is the right place to add the fix, but the tree that you are
using is not up to date with the current poky or oe-core master.

There are other changes in place that address a similar failure.

For us to look at this change, it needs to be rebased on top of
master (or master-next).

Bruce

> 
>>
>> So the patch as it stands, cannot be applied to master (and I
>> also checked master-next in case something snuck in that I
>> missed) of oe-core or poky, so there's no way to work with it.
>>
>> Bruce
>>
>>>
>>> Regarding the kernel - I do have my own recipe;
>>> inherit kernel
>>>
>>> require
>>> recipes-kernel/linux/linux-yocto.inc
>>>
>>>
>>> And then I do have:
>>> COMPATIBLE_MACHINE_display5 = "display5"
>>>
>>> KBUILD_DEFCONFIG_pn-linux-yocto-custom_display5 =
>>> "imx6q_display5_defconfig"
>>>
>>> As the board is in mainline I just point to Linus tree.
>>>    
>>>> Bruce
>>>>   
>>>>>     do_kernel_configme() {
>>>>>     	set +e
>>>>>     
>>>>>       
>>>>   
>>>
>>>
>>>
>>>
>>> Best regards,
>>>
>>> Lukasz Majewski
>>>
>>> --
>>>
>>> DENX Software Engineering GmbH,      Managing Director: Wolfgang
>>> Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell,
>>> Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email:
>>> wd@denx.de
>>
> 
> 
> 
> 
> Best regards,
> 
> Lukasz Majewski
> 
> --
> 
> DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
> Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
> 

Re: [PATCH] kernel: yocto: Add dependency on do_prepare_recipe_sysroot

[Lukasz Majewski]

[-- Attachment #1: Type: text/plain, Size: 6186 bytes --]

On Wed, 22 Aug 2018 10:44:08 -0400
Bruce Ashfield <bruce.ashfield@windriver.com> wrote:

> On 08/22/2018 10:20 AM, Lukasz Majewski wrote:
> > On Wed, 22 Aug 2018 10:13:33 -0400
> > Bruce Ashfield <bruce.ashfield@windriver.com> wrote:
> >   
> >> On 08/22/2018 10:05 AM, Lukasz Majewski wrote:  
> >>> Hi Bruce,
> >>>      
> >>>> On 08/22/2018 09:40 AM, Lukasz Majewski wrote:  
> >>>>> Without this patch it happens that do_populate_recipe_sysroot is
> >>>>> called just before do_compile (on multi core build machines).
> >>>>> This is way too late as the .config generated in
> >>>>> do_kernel_configme() is already broken.
> >>>>>
> >>>>> The problem is that do_kernel_configme() calls native's
> >>>>> merge_config.sh script which may call bison and arm-linux-*-gcc
> >>>>> to compile kconfig infrastructure to create base .config
> >>>>> (scripts/kconfig/conf  --allnoconfig Kconfig.)
> >>>>>
> >>>>> It turns out that we only got above binaries after
> >>>>> do_prepare_recipe_sysroot() is called.
> >>>>> If those binaries are not available, the merge_config.sh do not
> >>>>> generate base .config and without any error produces
> >>>>> malfunctioned .config. The build process continues and as a
> >>>>> result broken kernel is compiled.
> >>>>>
> >>>>> To reproduce:
> >>>>>
> >>>>> bitbake -c cleansstate virtual/kernel
> >>>>> bitbake -c kernel_metadata -v virtual/kernel
> >>>>> bitbake -c do_kernel_configme -v virtual/kernel
> >>>>> (one shall see broken .config in ${B}/.config)
> >>>>> bitbake -c do_populate_sysroot -v virtual/kernel
> >>>>> bitbake -c do_compile -v virtual/kernel
> >>>>>
> >>>>> (Poky) SHA1: bb91b2ae3ee5cf108aa2f9b78abb14d5aa00831d
> >>>>>
> >>>>> Signed-off-by: Lukasz Majewski <lukma@denx.de>
> >>>>> ---
> >>>>>     meta/classes/kernel-yocto.bbclass | 1 +
> >>>>>     1 file changed, 1 insertion(+)
> >>>>>
> >>>>> diff --git a/meta/classes/kernel-yocto.bbclass
> >>>>> b/meta/classes/kernel-yocto.bbclass index 4ac3a39e47..19a3f2bc46
> >>>>> 100644 --- a/meta/classes/kernel-yocto.bbclass
> >>>>> +++ b/meta/classes/kernel-yocto.bbclass
> >>>>> @@ -276,6 +276,7 @@ do_kernel_metadata[depends] =
> >>>>> "kern-tools-native:do_populate_sysroot"
> >>>>> do_validate_branches[depends] =
> >>>>> "kern-tools-native:do_populate_sysroot" do_kernel_configme[dirs]
> >>>>> += "${S} ${B}" +do_kernel_configme[depends] +=
> >>>>> "virtual/kernel:do_prepare_recipe_sysroot"  
> >>>>
> >>>> On a closer look .. what branch is this from ? We already have a
> >>>> change that triggered breakage on 4.17+ that should fix this.
> >>>>
> >>>> i.e. commit ff1bdd75d50f0ebac3d599e461685ace29559a82 in oe-core,
> >>>> adds:
> >>>>
> >>>> ----------
> >>>>
> >>>>     do_kernel_metadata[depends] =
> >>>> "kern-tools-native:do_populate_sysroot"
> >>>> do_validate_branches[depends] =
> >>>> "kern-tools-native:do_populate_sysroot"
> >>>>
> >>>> +do_kernel_configme[depends] +=
> >>>> "virtual/${TARGET_PREFIX}binutils:do_populate_sysroot"
> >>>> +do_kernel_configme[depends] +=
> >>>> "virtual/${TARGET_PREFIX}gcc:do_populate_sysroot"
> >>>> +do_kernel_configme[depends] += "bc-native:do_populate_sysroot
> >>>> bison-native:do_populate_sysroot"
> >>>>     do_kernel_configme[dirs] += "${S} ${B}"
> >>>>     do_kernel_configme() {
> >>>>
> >>>> ----------
> >>>>
> >>>> Which I don't see in the context of your patch.
> >>>>     
> >>>
> >>> I do use poky as in the commit message.  
> >>
> >> Yes, I checked that .. but that isn't a useful part of a patch
> >> to oe-core.
> >>
> >> The kernel-yocto bbclass in that context is not the one in master,
> >> and if I run a --contains on that commit ID, I don't get an active
> >> branch.  
> > 
> > I might have misunderstood something... apparently.
> > 
> > The changes which I did were to:
> > 
> > poky/meta/classes/kernel-yocto.bbclass
> > 
> > Is this a wrong place to add the fix?  
> 
> It is the right place to add the fix, but the tree that you are
> using is not up to date with the current poky or oe-core master.
> 
> There are other changes in place that address a similar failure.
> 
> For us to look at this change, it needs to be rebased on top of
> master (or master-next).

Now I got it. I shall sent the fix against cutting edge -master.

Shall I re-send the fix, of will you first check if patches in the
queue doesn't fix it already?

> 
> Bruce
> 
> >   
> >>
> >> So the patch as it stands, cannot be applied to master (and I
> >> also checked master-next in case something snuck in that I
> >> missed) of oe-core or poky, so there's no way to work with it.
> >>
> >> Bruce
> >>  
> >>>
> >>> Regarding the kernel - I do have my own recipe;
> >>> inherit kernel
> >>>
> >>> require
> >>> recipes-kernel/linux/linux-yocto.inc
> >>>
> >>>
> >>> And then I do have:
> >>> COMPATIBLE_MACHINE_display5 = "display5"
> >>>
> >>> KBUILD_DEFCONFIG_pn-linux-yocto-custom_display5 =
> >>> "imx6q_display5_defconfig"
> >>>
> >>> As the board is in mainline I just point to Linus tree.
> >>>      
> >>>> Bruce
> >>>>     
> >>>>>     do_kernel_configme() {
> >>>>>     	set +e
> >>>>>     
> >>>>>         
> >>>>     
> >>>
> >>>
> >>>
> >>>
> >>> Best regards,
> >>>
> >>> Lukasz Majewski
> >>>
> >>> --
> >>>
> >>> DENX Software Engineering GmbH,      Managing Director: Wolfgang
> >>> Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell,
> >>> Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email:
> >>> wd@denx.de  
> >>  
> > 
> > 
> > 
> > 
> > Best regards,
> > 
> > Lukasz Majewski
> > 
> > --
> > 
> > DENX Software Engineering GmbH,      Managing Director: Wolfgang
> > Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell,
> > Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email:
> > wd@denx.de 
> 




Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 499 bytes --]

Re: [PATCH] kernel: yocto: Add dependency on do_prepare_recipe_sysroot

[Bruce Ashfield]

On 08/22/2018 10:47 AM, Lukasz Majewski wrote:
> On Wed, 22 Aug 2018 10:44:08 -0400
> Bruce Ashfield <bruce.ashfield@windriver.com> wrote:
> 
>> On 08/22/2018 10:20 AM, Lukasz Majewski wrote:
>>> On Wed, 22 Aug 2018 10:13:33 -0400
>>> Bruce Ashfield <bruce.ashfield@windriver.com> wrote:
>>>    
>>>> On 08/22/2018 10:05 AM, Lukasz Majewski wrote:
>>>>> Hi Bruce,
>>>>>       
>>>>>> On 08/22/2018 09:40 AM, Lukasz Majewski wrote:
>>>>>>> Without this patch it happens that do_populate_recipe_sysroot is
>>>>>>> called just before do_compile (on multi core build machines).
>>>>>>> This is way too late as the .config generated in
>>>>>>> do_kernel_configme() is already broken.
>>>>>>>
>>>>>>> The problem is that do_kernel_configme() calls native's
>>>>>>> merge_config.sh script which may call bison and arm-linux-*-gcc
>>>>>>> to compile kconfig infrastructure to create base .config
>>>>>>> (scripts/kconfig/conf  --allnoconfig Kconfig.)
>>>>>>>
>>>>>>> It turns out that we only got above binaries after
>>>>>>> do_prepare_recipe_sysroot() is called.
>>>>>>> If those binaries are not available, the merge_config.sh do not
>>>>>>> generate base .config and without any error produces
>>>>>>> malfunctioned .config. The build process continues and as a
>>>>>>> result broken kernel is compiled.
>>>>>>>
>>>>>>> To reproduce:
>>>>>>>
>>>>>>> bitbake -c cleansstate virtual/kernel
>>>>>>> bitbake -c kernel_metadata -v virtual/kernel
>>>>>>> bitbake -c do_kernel_configme -v virtual/kernel
>>>>>>> (one shall see broken .config in ${B}/.config)
>>>>>>> bitbake -c do_populate_sysroot -v virtual/kernel
>>>>>>> bitbake -c do_compile -v virtual/kernel
>>>>>>>
>>>>>>> (Poky) SHA1: bb91b2ae3ee5cf108aa2f9b78abb14d5aa00831d
>>>>>>>
>>>>>>> Signed-off-by: Lukasz Majewski <lukma@denx.de>
>>>>>>> ---
>>>>>>>      meta/classes/kernel-yocto.bbclass | 1 +
>>>>>>>      1 file changed, 1 insertion(+)
>>>>>>>
>>>>>>> diff --git a/meta/classes/kernel-yocto.bbclass
>>>>>>> b/meta/classes/kernel-yocto.bbclass index 4ac3a39e47..19a3f2bc46
>>>>>>> 100644 --- a/meta/classes/kernel-yocto.bbclass
>>>>>>> +++ b/meta/classes/kernel-yocto.bbclass
>>>>>>> @@ -276,6 +276,7 @@ do_kernel_metadata[depends] =
>>>>>>> "kern-tools-native:do_populate_sysroot"
>>>>>>> do_validate_branches[depends] =
>>>>>>> "kern-tools-native:do_populate_sysroot" do_kernel_configme[dirs]
>>>>>>> += "${S} ${B}" +do_kernel_configme[depends] +=
>>>>>>> "virtual/kernel:do_prepare_recipe_sysroot"
>>>>>>
>>>>>> On a closer look .. what branch is this from ? We already have a
>>>>>> change that triggered breakage on 4.17+ that should fix this.
>>>>>>
>>>>>> i.e. commit ff1bdd75d50f0ebac3d599e461685ace29559a82 in oe-core,
>>>>>> adds:
>>>>>>
>>>>>> ----------
>>>>>>
>>>>>>      do_kernel_metadata[depends] =
>>>>>> "kern-tools-native:do_populate_sysroot"
>>>>>> do_validate_branches[depends] =
>>>>>> "kern-tools-native:do_populate_sysroot"
>>>>>>
>>>>>> +do_kernel_configme[depends] +=
>>>>>> "virtual/${TARGET_PREFIX}binutils:do_populate_sysroot"
>>>>>> +do_kernel_configme[depends] +=
>>>>>> "virtual/${TARGET_PREFIX}gcc:do_populate_sysroot"
>>>>>> +do_kernel_configme[depends] += "bc-native:do_populate_sysroot
>>>>>> bison-native:do_populate_sysroot"
>>>>>>      do_kernel_configme[dirs] += "${S} ${B}"
>>>>>>      do_kernel_configme() {
>>>>>>
>>>>>> ----------
>>>>>>
>>>>>> Which I don't see in the context of your patch.
>>>>>>      
>>>>>
>>>>> I do use poky as in the commit message.
>>>>
>>>> Yes, I checked that .. but that isn't a useful part of a patch
>>>> to oe-core.
>>>>
>>>> The kernel-yocto bbclass in that context is not the one in master,
>>>> and if I run a --contains on that commit ID, I don't get an active
>>>> branch.
>>>
>>> I might have misunderstood something... apparently.
>>>
>>> The changes which I did were to:
>>>
>>> poky/meta/classes/kernel-yocto.bbclass
>>>
>>> Is this a wrong place to add the fix?
>>
>> It is the right place to add the fix, but the tree that you are
>> using is not up to date with the current poky or oe-core master.
>>
>> There are other changes in place that address a similar failure.
>>
>> For us to look at this change, it needs to be rebased on top of
>> master (or master-next).
> 
> Now I got it. I shall sent the fix against cutting edge -master.
> 
> Shall I re-send the fix, of will you first check if patches in the
> queue doesn't fix it already?

I'll run the steps in your reproducer and let you know if it
works here.

Bruce

> 
>>
>> Bruce
>>
>>>    
>>>>
>>>> So the patch as it stands, cannot be applied to master (and I
>>>> also checked master-next in case something snuck in that I
>>>> missed) of oe-core or poky, so there's no way to work with it.
>>>>
>>>> Bruce
>>>>   
>>>>>
>>>>> Regarding the kernel - I do have my own recipe;
>>>>> inherit kernel
>>>>>
>>>>> require
>>>>> recipes-kernel/linux/linux-yocto.inc
>>>>>
>>>>>
>>>>> And then I do have:
>>>>> COMPATIBLE_MACHINE_display5 = "display5"
>>>>>
>>>>> KBUILD_DEFCONFIG_pn-linux-yocto-custom_display5 =
>>>>> "imx6q_display5_defconfig"
>>>>>
>>>>> As the board is in mainline I just point to Linus tree.
>>>>>       
>>>>>> Bruce
>>>>>>      
>>>>>>>      do_kernel_configme() {
>>>>>>>      	set +e
>>>>>>>      
>>>>>>>          
>>>>>>      
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Best regards,
>>>>>
>>>>> Lukasz Majewski
>>>>>
>>>>> --
>>>>>
>>>>> DENX Software Engineering GmbH,      Managing Director: Wolfgang
>>>>> Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell,
>>>>> Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email:
>>>>> wd@denx.de
>>>>   
>>>
>>>
>>>
>>>
>>> Best regards,
>>>
>>> Lukasz Majewski
>>>
>>> --
>>>
>>> DENX Software Engineering GmbH,      Managing Director: Wolfgang
>>> Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell,
>>> Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email:
>>> wd@denx.de
>>
> 
> 
> 
> 
> Best regards,
> 
> Lukasz Majewski
> 
> --
> 
> DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
> Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
> 

Re: [PATCH] kernel: yocto: Add dependency on do_prepare_recipe_sysroot

[Bruce Ashfield]

On 08/22/2018 10:47 AM, Lukasz Majewski wrote:
> On Wed, 22 Aug 2018 10:44:08 -0400
> Bruce Ashfield <bruce.ashfield@windriver.com> wrote:
> 
>> On 08/22/2018 10:20 AM, Lukasz Majewski wrote:
>>> On Wed, 22 Aug 2018 10:13:33 -0400
>>> Bruce Ashfield <bruce.ashfield@windriver.com> wrote:
>>>    
>>>> On 08/22/2018 10:05 AM, Lukasz Majewski wrote:
>>>>> Hi Bruce,
>>>>>       
>>>>>> On 08/22/2018 09:40 AM, Lukasz Majewski wrote:
>>>>>>> Without this patch it happens that do_populate_recipe_sysroot is
>>>>>>> called just before do_compile (on multi core build machines).
>>>>>>> This is way too late as the .config generated in
>>>>>>> do_kernel_configme() is already broken.
>>>>>>>
>>>>>>> The problem is that do_kernel_configme() calls native's
>>>>>>> merge_config.sh script which may call bison and arm-linux-*-gcc
>>>>>>> to compile kconfig infrastructure to create base .config
>>>>>>> (scripts/kconfig/conf  --allnoconfig Kconfig.)
>>>>>>>
>>>>>>> It turns out that we only got above binaries after
>>>>>>> do_prepare_recipe_sysroot() is called.
>>>>>>> If those binaries are not available, the merge_config.sh do not
>>>>>>> generate base .config and without any error produces
>>>>>>> malfunctioned .config. The build process continues and as a
>>>>>>> result broken kernel is compiled.
>>>>>>>
>>>>>>> To reproduce:
>>>>>>>
>>>>>>> bitbake -c cleansstate virtual/kernel
>>>>>>> bitbake -c kernel_metadata -v virtual/kernel
>>>>>>> bitbake -c do_kernel_configme -v virtual/kernel
>>>>>>> (one shall see broken .config in ${B}/.config)
>>>>>>> bitbake -c do_populate_sysroot -v virtual/kernel
>>>>>>> bitbake -c do_compile -v virtual/kernel
>>>>>>>
>>>>>>> (Poky) SHA1: bb91b2ae3ee5cf108aa2f9b78abb14d5aa00831d
>>>>>>>
>>>>>>> Signed-off-by: Lukasz Majewski <lukma@denx.de>
>>>>>>> ---
>>>>>>>      meta/classes/kernel-yocto.bbclass | 1 +
>>>>>>>      1 file changed, 1 insertion(+)
>>>>>>>
>>>>>>> diff --git a/meta/classes/kernel-yocto.bbclass
>>>>>>> b/meta/classes/kernel-yocto.bbclass index 4ac3a39e47..19a3f2bc46
>>>>>>> 100644 --- a/meta/classes/kernel-yocto.bbclass
>>>>>>> +++ b/meta/classes/kernel-yocto.bbclass
>>>>>>> @@ -276,6 +276,7 @@ do_kernel_metadata[depends] =
>>>>>>> "kern-tools-native:do_populate_sysroot"
>>>>>>> do_validate_branches[depends] =
>>>>>>> "kern-tools-native:do_populate_sysroot" do_kernel_configme[dirs]
>>>>>>> += "${S} ${B}" +do_kernel_configme[depends] +=
>>>>>>> "virtual/kernel:do_prepare_recipe_sysroot"
>>>>>>
>>>>>> On a closer look .. what branch is this from ? We already have a
>>>>>> change that triggered breakage on 4.17+ that should fix this.
>>>>>>
>>>>>> i.e. commit ff1bdd75d50f0ebac3d599e461685ace29559a82 in oe-core,
>>>>>> adds:
>>>>>>
>>>>>> ----------
>>>>>>
>>>>>>      do_kernel_metadata[depends] =
>>>>>> "kern-tools-native:do_populate_sysroot"
>>>>>> do_validate_branches[depends] =
>>>>>> "kern-tools-native:do_populate_sysroot"
>>>>>>
>>>>>> +do_kernel_configme[depends] +=
>>>>>> "virtual/${TARGET_PREFIX}binutils:do_populate_sysroot"
>>>>>> +do_kernel_configme[depends] +=
>>>>>> "virtual/${TARGET_PREFIX}gcc:do_populate_sysroot"
>>>>>> +do_kernel_configme[depends] += "bc-native:do_populate_sysroot
>>>>>> bison-native:do_populate_sysroot"
>>>>>>      do_kernel_configme[dirs] += "${S} ${B}"
>>>>>>      do_kernel_configme() {
>>>>>>
>>>>>> ----------
>>>>>>
>>>>>> Which I don't see in the context of your patch.
>>>>>>      
>>>>>
>>>>> I do use poky as in the commit message.
>>>>
>>>> Yes, I checked that .. but that isn't a useful part of a patch
>>>> to oe-core.
>>>>
>>>> The kernel-yocto bbclass in that context is not the one in master,
>>>> and if I run a --contains on that commit ID, I don't get an active
>>>> branch.
>>>
>>> I might have misunderstood something... apparently.
>>>
>>> The changes which I did were to:
>>>
>>> poky/meta/classes/kernel-yocto.bbclass
>>>
>>> Is this a wrong place to add the fix?
>>
>> It is the right place to add the fix, but the tree that you are
>> using is not up to date with the current poky or oe-core master.
>>
>> There are other changes in place that address a similar failure.
>>
>> For us to look at this change, it needs to be rebased on top of
>> master (or master-next).
> 
> Now I got it. I shall sent the fix against cutting edge -master.
> 
> Shall I re-send the fix, of will you first check if patches in the
> queue doesn't fix it already?

I ran the steps yesterday and wasn't able to trigger the issue
here.

so if you try against the current master of poky/oe-core, I'd be
interested to hear if you can make the problem happen (and I can
re-try as required).

Bruce

> 
>>
>> Bruce
>>
>>>    
>>>>
>>>> So the patch as it stands, cannot be applied to master (and I
>>>> also checked master-next in case something snuck in that I
>>>> missed) of oe-core or poky, so there's no way to work with it.
>>>>
>>>> Bruce
>>>>   
>>>>>
>>>>> Regarding the kernel - I do have my own recipe;
>>>>> inherit kernel
>>>>>
>>>>> require
>>>>> recipes-kernel/linux/linux-yocto.inc
>>>>>
>>>>>
>>>>> And then I do have:
>>>>> COMPATIBLE_MACHINE_display5 = "display5"
>>>>>
>>>>> KBUILD_DEFCONFIG_pn-linux-yocto-custom_display5 =
>>>>> "imx6q_display5_defconfig"
>>>>>
>>>>> As the board is in mainline I just point to Linus tree.
>>>>>       
>>>>>> Bruce
>>>>>>      
>>>>>>>      do_kernel_configme() {
>>>>>>>      	set +e
>>>>>>>      
>>>>>>>          
>>>>>>      
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Best regards,
>>>>>
>>>>> Lukasz Majewski
>>>>>
>>>>> --
>>>>>
>>>>> DENX Software Engineering GmbH,      Managing Director: Wolfgang
>>>>> Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell,
>>>>> Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email:
>>>>> wd@denx.de
>>>>   
>>>
>>>
>>>
>>>
>>> Best regards,
>>>
>>> Lukasz Majewski
>>>
>>> --
>>>
>>> DENX Software Engineering GmbH,      Managing Director: Wolfgang
>>> Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell,
>>> Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email:
>>> wd@denx.de
>>
> 
> 
> 
> 
> Best regards,
> 
> Lukasz Majewski
> 
> --
> 
> DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
> Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
> 

Re: [PATCH] kernel: yocto: Add dependency on do_prepare_recipe_sysroot

[Lukasz Majewski]

[-- Attachment #1: Type: text/plain, Size: 7628 bytes --]

Hi Bruce,

> On 08/22/2018 10:47 AM, Lukasz Majewski wrote:
> > On Wed, 22 Aug 2018 10:44:08 -0400
> > Bruce Ashfield <bruce.ashfield@windriver.com> wrote:
> >   
> >> On 08/22/2018 10:20 AM, Lukasz Majewski wrote:  
> >>> On Wed, 22 Aug 2018 10:13:33 -0400
> >>> Bruce Ashfield <bruce.ashfield@windriver.com> wrote:
> >>>      
> >>>> On 08/22/2018 10:05 AM, Lukasz Majewski wrote:  
> >>>>> Hi Bruce,
> >>>>>         
> >>>>>> On 08/22/2018 09:40 AM, Lukasz Majewski wrote:  
> >>>>>>> Without this patch it happens that do_populate_recipe_sysroot
> >>>>>>> is called just before do_compile (on multi core build
> >>>>>>> machines). This is way too late as the .config generated in
> >>>>>>> do_kernel_configme() is already broken.
> >>>>>>>
> >>>>>>> The problem is that do_kernel_configme() calls native's
> >>>>>>> merge_config.sh script which may call bison and
> >>>>>>> arm-linux-*-gcc to compile kconfig infrastructure to create
> >>>>>>> base .config (scripts/kconfig/conf  --allnoconfig Kconfig.)
> >>>>>>>
> >>>>>>> It turns out that we only got above binaries after
> >>>>>>> do_prepare_recipe_sysroot() is called.
> >>>>>>> If those binaries are not available, the merge_config.sh do
> >>>>>>> not generate base .config and without any error produces
> >>>>>>> malfunctioned .config. The build process continues and as a
> >>>>>>> result broken kernel is compiled.
> >>>>>>>
> >>>>>>> To reproduce:
> >>>>>>>
> >>>>>>> bitbake -c cleansstate virtual/kernel
> >>>>>>> bitbake -c kernel_metadata -v virtual/kernel
> >>>>>>> bitbake -c do_kernel_configme -v virtual/kernel
> >>>>>>> (one shall see broken .config in ${B}/.config)
> >>>>>>> bitbake -c do_populate_sysroot -v virtual/kernel
> >>>>>>> bitbake -c do_compile -v virtual/kernel
> >>>>>>>
> >>>>>>> (Poky) SHA1: bb91b2ae3ee5cf108aa2f9b78abb14d5aa00831d
> >>>>>>>
> >>>>>>> Signed-off-by: Lukasz Majewski <lukma@denx.de>
> >>>>>>> ---
> >>>>>>>      meta/classes/kernel-yocto.bbclass | 1 +
> >>>>>>>      1 file changed, 1 insertion(+)
> >>>>>>>
> >>>>>>> diff --git a/meta/classes/kernel-yocto.bbclass
> >>>>>>> b/meta/classes/kernel-yocto.bbclass index
> >>>>>>> 4ac3a39e47..19a3f2bc46 100644 ---
> >>>>>>> a/meta/classes/kernel-yocto.bbclass +++
> >>>>>>> b/meta/classes/kernel-yocto.bbclass @@ -276,6 +276,7 @@
> >>>>>>> do_kernel_metadata[depends] =
> >>>>>>> "kern-tools-native:do_populate_sysroot"
> >>>>>>> do_validate_branches[depends] =
> >>>>>>> "kern-tools-native:do_populate_sysroot"
> >>>>>>> do_kernel_configme[dirs] += "${S} ${B}"
> >>>>>>> +do_kernel_configme[depends] +=
> >>>>>>> "virtual/kernel:do_prepare_recipe_sysroot"  
> >>>>>>
> >>>>>> On a closer look .. what branch is this from ? We already have
> >>>>>> a change that triggered breakage on 4.17+ that should fix this.
> >>>>>>
> >>>>>> i.e. commit ff1bdd75d50f0ebac3d599e461685ace29559a82 in
> >>>>>> oe-core, adds:
> >>>>>>
> >>>>>> ----------
> >>>>>>
> >>>>>>      do_kernel_metadata[depends] =
> >>>>>> "kern-tools-native:do_populate_sysroot"
> >>>>>> do_validate_branches[depends] =
> >>>>>> "kern-tools-native:do_populate_sysroot"
> >>>>>>
> >>>>>> +do_kernel_configme[depends] +=
> >>>>>> "virtual/${TARGET_PREFIX}binutils:do_populate_sysroot"
> >>>>>> +do_kernel_configme[depends] +=
> >>>>>> "virtual/${TARGET_PREFIX}gcc:do_populate_sysroot"
> >>>>>> +do_kernel_configme[depends] += "bc-native:do_populate_sysroot
> >>>>>> bison-native:do_populate_sysroot"
> >>>>>>      do_kernel_configme[dirs] += "${S} ${B}"
> >>>>>>      do_kernel_configme() {
> >>>>>>
> >>>>>> ----------
> >>>>>>
> >>>>>> Which I don't see in the context of your patch.
> >>>>>>        
> >>>>>
> >>>>> I do use poky as in the commit message.  
> >>>>
> >>>> Yes, I checked that .. but that isn't a useful part of a patch
> >>>> to oe-core.
> >>>>
> >>>> The kernel-yocto bbclass in that context is not the one in
> >>>> master, and if I run a --contains on that commit ID, I don't get
> >>>> an active branch.  
> >>>
> >>> I might have misunderstood something... apparently.
> >>>
> >>> The changes which I did were to:
> >>>
> >>> poky/meta/classes/kernel-yocto.bbclass
> >>>
> >>> Is this a wrong place to add the fix?  
> >>
> >> It is the right place to add the fix, but the tree that you are
> >> using is not up to date with the current poky or oe-core master.
> >>
> >> There are other changes in place that address a similar failure.
> >>
> >> For us to look at this change, it needs to be rebased on top of
> >> master (or master-next).  
> > 
> > Now I got it. I shall sent the fix against cutting edge -master.
> > 
> > Shall I re-send the fix, of will you first check if patches in the
> > queue doesn't fix it already?  
> 
> I ran the steps yesterday and wasn't able to trigger the issue
> here.
> 
> so if you try against the current master of poky/oe-core, I'd be
> interested to hear if you can make the problem happen (and I can
> re-try as required).

The upstream patch:

"kernel: specify dependencies for compilation for config tasks"
SHA1: 81e8a52e8e40e47c34f900db5d73e69ffc25f5d0

Fixes the problem, which I've observed.

(And also forces merge_config.sh to use HOST tools, which is a good
thing.)

Tested-by: Lukasz Majewski <lukma@denx.de>

Thanks for providing the patch upstream.

> 
> Bruce
> 
> >   
> >>
> >> Bruce
> >>  
> >>>      
> >>>>
> >>>> So the patch as it stands, cannot be applied to master (and I
> >>>> also checked master-next in case something snuck in that I
> >>>> missed) of oe-core or poky, so there's no way to work with it.
> >>>>
> >>>> Bruce
> >>>>     
> >>>>>
> >>>>> Regarding the kernel - I do have my own recipe;
> >>>>> inherit kernel
> >>>>>
> >>>>> require
> >>>>> recipes-kernel/linux/linux-yocto.inc
> >>>>>
> >>>>>
> >>>>> And then I do have:
> >>>>> COMPATIBLE_MACHINE_display5 = "display5"
> >>>>>
> >>>>> KBUILD_DEFCONFIG_pn-linux-yocto-custom_display5 =
> >>>>> "imx6q_display5_defconfig"
> >>>>>
> >>>>> As the board is in mainline I just point to Linus tree.
> >>>>>         
> >>>>>> Bruce
> >>>>>>        
> >>>>>>>      do_kernel_configme() {
> >>>>>>>      	set +e
> >>>>>>>      
> >>>>>>>            
> >>>>>>        
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> Best regards,
> >>>>>
> >>>>> Lukasz Majewski
> >>>>>
> >>>>> --
> >>>>>
> >>>>> DENX Software Engineering GmbH,      Managing Director: Wolfgang
> >>>>> Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194
> >>>>> Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax:
> >>>>> (+49)-8142-66989-80 Email: wd@denx.de  
> >>>>     
> >>>
> >>>
> >>>
> >>>
> >>> Best regards,
> >>>
> >>> Lukasz Majewski
> >>>
> >>> --
> >>>
> >>> DENX Software Engineering GmbH,      Managing Director: Wolfgang
> >>> Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell,
> >>> Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email:
> >>> wd@denx.de  
> >>  
> > 
> > 
> > 
> > 
> > Best regards,
> > 
> > Lukasz Majewski
> > 
> > --
> > 
> > DENX Software Engineering GmbH,      Managing Director: Wolfgang
> > Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell,
> > Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email:
> > wd@denx.de 
> 




Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 499 bytes --]

✗ patchtest: failure for kernel: yocto: Add dependency on do_prepare_recipe_sysroot

[Patchwork]

== Series Details ==

Series: kernel: yocto: Add dependency on do_prepare_recipe_sysroot
Revision: 1
URL   : https://patchwork.openembedded.org/series/13667/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Issue             Series does not apply on top of target branch [test_series_merge_on_head] 
  Suggested fix    Rebase your series on top of targeted branch
  Targeted branch  master (currently at d2a929d932)



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe