kernel.release being regenerated

kernel.release being regenerated

[Holger Hans Peter Freyther]

Hi,

I have an integration build that at least builds our sysmocom
BTS layers against master of Yocto. Our nightly build started
to be flaky and fail most of the time building the Linux kernel.

We are using a 3.10 LTS kernel for the product and it looks
like the kernel.release will be (re)created in the build directory
when the uImage is generated but also when "make modules" is
called.

This means that shared_workdir in my case should be called at
a later stage, e.g. after do_compilekernelmodules.

Did anyone else see flaky builds?

holger

Re: kernel.release being regenerated

[Richard Purdie]

Hi Holger,

On Tue, 2015-02-10 at 19:43 +0100, Holger Hans Peter Freyther wrote:
> I have an integration build that at least builds our sysmocom
> BTS layers against master of Yocto. Our nightly build started
> to be flaky and fail most of the time building the Linux kernel.
> 
> We are using a 3.10 LTS kernel for the product and it looks
> like the kernel.release will be (re)created in the build directory
> when the uImage is generated but also when "make modules" is
> called.
> 
> This means that shared_workdir in my case should be called at
> a later stage, e.g. after do_compilekernelmodules.
> 
> Did anyone else see flaky builds?

We've not see any on the yocto project autobuilders and I've not heard
reports of this from anyone else. I've cc'd Bruce in case he's aware of
anything and can chime in but the new code is being used in many places,
including with 3.10 kernels and I've not heard reports.

The fact the kernel.release is being recreated suggests something in the
configuration is changing (different environment or commandline
options?) or that there is a problem in your kernel to do with the
Makefiles and the dependencies of the kernel.release file.

You might reach a wider audience with questions like this on the yocto
or openembedded-core mailing lists btw, poky tends to be fairly quiet
these days although many of us are here too :)

Cheers,

Richard

Re: kernel.release being regenerated

[Bruce Ashfield]

On 2015-02-10 5:51 PM, Richard Purdie wrote:
> Hi Holger,
>
> On Tue, 2015-02-10 at 19:43 +0100, Holger Hans Peter Freyther wrote:
>> I have an integration build that at least builds our sysmocom
>> BTS layers against master of Yocto. Our nightly build started
>> to be flaky and fail most of the time building the Linux kernel.
>>
>> We are using a 3.10 LTS kernel for the product and it looks
>> like the kernel.release will be (re)created in the build directory
>> when the uImage is generated but also when "make modules" is
>> called.
>>
>> This means that shared_workdir in my case should be called at
>> a later stage, e.g. after do_compilekernelmodules.
>>
>> Did anyone else see flaky builds?
>
> We've not see any on the yocto project autobuilders and I've not heard
> reports of this from anyone else. I've cc'd Bruce in case he's aware of
> anything and can chime in but the new code is being used in many places,
> including with 3.10 kernels and I've not heard reports.

I haven't seen anything like this either, and I've been doing builds
on a constant loop.

But I'll think on it, and see if anything jumps to mind.

Bruce

>
> The fact the kernel.release is being recreated suggests something in the
> configuration is changing (different environment or commandline
> options?) or that there is a problem in your kernel to do with the
> Makefiles and the dependencies of the kernel.release file.
>
> You might reach a wider audience with questions like this on the yocto
> or openembedded-core mailing lists btw, poky tends to be fairly quiet
> these days although many of us are here too :)
>
> Cheers,
>
> Richard
>
>
>

Re: kernel.release being regenerated

[Holger Hans Peter Freyther]

On Tue, Feb 10, 2015 at 10:51:26PM +0000, Richard Purdie wrote:
> Hi Holger,

Hi Richard,

> The fact the kernel.release is being recreated suggests something in the
> configuration is changing (different environment or commandline
> options?) or that there is a problem in your kernel to do with the
> Makefiles and the dependencies of the kernel.release file.

This issue should be seen by everyone but the race is pretty small.
do_shared_workdir must run shortly after compilemodules has started
and removed the file but didn't create a new one. This is the kind
of race I am more likely to hit than the average population. The
"configuration is changing" part looks like a red herring to me.
From looking at the Makefile I see a FORCE as dependency of the
kernel.release rule. With my 3.10er kernel (and it doesn't look
like we have patched the Makefile) the dependency chain is like
this:

 init -> prepare -> prepare0 .. -> prepare3 -> kernel.release -> FORCE

This means that on any target being executed the kernel.release
will be re-created. I think the only stable way would be to have
do_compile copy the kernel.release to another place and have the
copying task pick this file from another place.


kind regards
	holger


More debugging:

Running make with debug shows that the kernel.release target
will be forced to re-execute and at least for me this leads to
the creation of a new file.

make:

 File '_all' does not exist.
  Considering target file 'all'.
   File 'all' does not exist.
    Considering target file 'vmlinux'.
      Considering target file 'scripts/link-vmlinux.sh'.
       Looking for an implicit rule for 'scripts/link-vmlinux.sh'.
       No implicit rule found for 'scripts/link-vmlinux.sh'.
       Finished prerequisites of target file 'scripts/link-vmlinux.sh'.
      No need to remake target 'scripts/link-vmlinux.sh'.
      Considering target file 'arch/arm/kernel/vmlinux.lds'.
        Considering target file 'init'.
         File 'init' does not exist.
          Considering target file 'prepare'.
           File 'prepare' does not exist.
            Considering target file 'prepare0'.
             File 'prepare0' does not exist.
              Considering target file 'archprepare'.
               File 'archprepare' does not exist.
                Considering target file 'archheaders'.
                 File 'archheaders' does not exist.
                 Finished prerequisites of target file 'archheaders'.
                Must remake target 'archheaders'.
                Successfully remade target file 'archheaders'.
                Considering target file 'archscripts'.
                 File 'archscripts' does not exist.
                 Finished prerequisites of target file 'archscripts'.
                Must remake target 'archscripts'.
                Successfully remade target file 'archscripts'.
                Considering target file 'prepare1'.
                 File 'prepare1' does not exist.
                  Considering target file 'prepare2'.
                   File 'prepare2' does not exist.
                    Considering target file 'prepare3'.
                     File 'prepare3' does not exist.
                      Considering target file 'include/config/kernel.release'.
                        Pruning file 'include/config/auto.conf'.
                        Considering target file 'FORCE'.
                         File 'FORCE' does not exist.
                         Finished prerequisites of target file 'FORCE'.
                        Must remake target 'FORCE'.
                        Successfully remade target file 'FORCE'.
                       Finished prerequisites of target file 'include/config/kernel.release'.

	...



make uImage

Considering target file 'uImage'.
 File 'uImage' does not exist.
  Considering target file 'vmlinux'.
    Considering target file 'scripts/link-vmlinux.sh'.
     Looking for an implicit rule for 'scripts/link-vmlinux.sh'.
     No implicit rule found for 'scripts/link-vmlinux.sh'.
     Finished prerequisites of target file 'scripts/link-vmlinux.sh'.
    No need to remake target 'scripts/link-vmlinux.sh'.
    Considering target file 'arch/arm/kernel/vmlinux.lds'.
      Considering target file 'init'.
       File 'init' does not exist.
        Considering target file 'prepare'.
         File 'prepare' does not exist.
          Considering target file 'prepare0'.
           File 'prepare0' does not exist.
            Considering target file 'archprepare'.
             File 'archprepare' does not exist.
              Considering target file 'archheaders'.
               File 'archheaders' does not exist.
               Finished prerequisites of target file 'archheaders'.
              Must remake target 'archheaders'.
              Successfully remade target file 'archheaders'.
              Considering target file 'archscripts'.
               File 'archscripts' does not exist.
               Finished prerequisites of target file 'archscripts'.
              Must remake target 'archscripts'.
              Successfully remade target file 'archscripts'.
              Considering target file 'prepare1'.
               File 'prepare1' does not exist.
                Considering target file 'prepare2'.
                 File 'prepare2' does not exist.
                  Considering target file 'prepare3'.
                   File 'prepare3' does not exist.
                    Considering target file 'include/config/kernel.release'.
                      Pruning file 'include/config/auto.conf'.
                      Considering target file 'FORCE'.
                       File 'FORCE' does not exist.
                       Finished prerequisites of target file 'FORCE'.
                      Must remake target 'FORCE'.
                      Successfully remade target file 'FORCE'.
                     Finished prerequisites of target file 'include/config/kernel.release'.
                     Prerequisite 'include/config/auto.conf' is older than target 'include/config/kernel.release'.
                     Prerequisite 'FORCE' of target 'include/config/kernel.release' does not exist.
                    Must remake target 'include/config/kernel.release'.



make modules

Considering target file 'modules'.
 File 'modules' does not exist.
  Considering target file 'init'.
   File 'init' does not exist.
    Considering target file 'prepare'.
     File 'prepare' does not exist.
      Considering target file 'prepare0'.
       File 'prepare0' does not exist.
        Considering target file 'archprepare'.
         File 'archprepare' does not exist.
          Considering target file 'archheaders'.
           File 'archheaders' does not exist.
           Finished prerequisites of target file 'archheaders'.
          Must remake target 'archheaders'.
          Successfully remade target file 'archheaders'.
          Considering target file 'archscripts'.
           File 'archscripts' does not exist.
           Finished prerequisites of target file 'archscripts'.
          Must remake target 'archscripts'.
          Successfully remade target file 'archscripts'.
          Considering target file 'prepare1'.
           File 'prepare1' does not exist.
            Considering target file 'prepare2'.
             File 'prepare2' does not exist.
              Considering target file 'prepare3'.
               File 'prepare3' does not exist.
                Considering target file 'include/config/kernel.release'.
                  Pruning file 'include/config/auto.conf'.
                  Considering target file 'FORCE'.
                   File 'FORCE' does not exist.
                   Finished prerequisites of target file 'FORCE'.
                  Must remake target 'FORCE'.
                  Successfully remade target file 'FORCE'.
                 Finished prerequisites of target file 'include/config/kernel.release'.
                 Prerequisite 'include/config/auto.conf' is older than target 'include/config/kernel.release'.
                 Prerequisite 'FORCE' of target 'include/config/kernel.release' does not exist.
                Must remake target 'include/config/kernel.release'.

Re: kernel.release being regenerated

[Richard Purdie]

On Wed, 2015-02-11 at 09:27 +0100, Holger Hans Peter Freyther wrote:
> On Tue, Feb 10, 2015 at 10:51:26PM +0000, Richard Purdie wrote:
> > Hi Holger,
> 
> Hi Richard,
> 
> > The fact the kernel.release is being recreated suggests something in the
> > configuration is changing (different environment or commandline
> > options?) or that there is a problem in your kernel to do with the
> > Makefiles and the dependencies of the kernel.release file.
> 
> This issue should be seen by everyone but the race is pretty small.
> do_shared_workdir must run shortly after compilemodules has started
> and removed the file but didn't create a new one. This is the kind
> of race I am more likely to hit than the average population. The
> "configuration is changing" part looks like a red herring to me.
> From looking at the Makefile I see a FORCE as dependency of the
> kernel.release rule. With my 3.10er kernel (and it doesn't look
> like we have patched the Makefile) the dependency chain is like
> this:
> 
>  init -> prepare -> prepare0 .. -> prepare3 -> kernel.release -> FORCE
> 
> This means that on any target being executed the kernel.release
> will be re-created. I think the only stable way would be to have
> do_compile copy the kernel.release to another place and have the
> copying task pick this file from another place.

This does sound like a flaw in how we're copying the file if it is
regenerated on each execution of make within the kernel build.

What puzzles me is that the use of kernel.release isn't new, the
previous do_install code used to do this and technically would have
raced with do_compilemodules too. So was it just bad luck you started
hitting this now?

(http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/classes/kernel.bbclass?id=92725ad46f4d331bea6a2fa65964158d78a7add8)

As you say, if this really is getting regenerated as you describe, we'll
have to store a safe copy in the tree.

Cheers,

Richard

Re: kernel.release being regenerated

[Bruce Ashfield]

On 15-02-11 04:10 AM, Richard Purdie wrote:
> On Wed, 2015-02-11 at 09:27 +0100, Holger Hans Peter Freyther wrote:
>> On Tue, Feb 10, 2015 at 10:51:26PM +0000, Richard Purdie wrote:
>>> Hi Holger,
>>
>> Hi Richard,
>>
>>> The fact the kernel.release is being recreated suggests something in the
>>> configuration is changing (different environment or commandline
>>> options?) or that there is a problem in your kernel to do with the
>>> Makefiles and the dependencies of the kernel.release file.
>>
>> This issue should be seen by everyone but the race is pretty small.
>> do_shared_workdir must run shortly after compilemodules has started
>> and removed the file but didn't create a new one. This is the kind
>> of race I am more likely to hit than the average population. The
>> "configuration is changing" part looks like a red herring to me.
>>  From looking at the Makefile I see a FORCE as dependency of the
>> kernel.release rule. With my 3.10er kernel (and it doesn't look
>> like we have patched the Makefile) the dependency chain is like
>> this:
>>
>>   init -> prepare -> prepare0 .. -> prepare3 -> kernel.release -> FORCE
>>
>> This means that on any target being executed the kernel.release
>> will be re-created. I think the only stable way would be to have
>> do_compile copy the kernel.release to another place and have the
>> copying task pick this file from another place.
>
> This does sound like a flaw in how we're copying the file if it is
> regenerated on each execution of make within the kernel build.
>
> What puzzles me is that the use of kernel.release isn't new, the
> previous do_install code used to do this and technically would have
> raced with do_compilemodules too. So was it just bad luck you started
> hitting this now?

It looks like bad luck that this is popping up now.

>
> (http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/classes/kernel.bbclass?id=92725ad46f4d331bea6a2fa65964158d78a7add8)
>
> As you say, if this really is getting regenerated as you describe, we'll
> have to store a safe copy in the tree.

What about changing the dependencies of the known tasks ? i.e. making
sure that the kernel modules don't start building until we have the
shared workdir populated ? Otherwise, what does that leave us, staging
a copy at the end of kernel_do_compile and grabbing it from there in the
shared workdir population step ?

Bruce

>
> Cheers,
>
> Richard
>

Re: kernel.release being regenerated

[Holger Hans Peter Freyther]

On Wed, Feb 11, 2015 at 12:32:15PM -0500, Bruce Ashfield wrote:

> It looks like bad luck that this is popping up now.

yes, it looks like bad luck. What is the window for echo
"content" > file and another task reading file not being
able to find. I couple of nano seconds? It is a bit frightening
that my build regulariy break due to this.