Merging problems

Merging problems

[Richard Purdie]

I want to give people a headsup that we're having problems merging
changes at the moment. We've been doing our best but the number of
things building up which are causing issues is overwheling our ability
to fix and stablise the build. It wasn't helped that I took a long
weekend's vacation last weekend. There are changes being made or tested
to the autobuilder which also isn't helping.

The kernel series has several issues:

  * a random failure in do_kernel_configme [i]
  * bzImage not being built for some machines [ii]
  * non linux-yocto kernel failures [iii]
  * the headers upgrade breaks socat (fix pending, thanks Bruce)

There was a change merged to bitbake which broken cups on lsb and
powertop due to unset variable problems. I have two pending fixes for
those.

For the last test build, we have 195 failures to dig through:

http://errors.yoctoproject.org/Errors/Search/Args/?items=150&page=1&query=99e7d5b9c736eb08a3853818a73c51676e7aa7ac

and we've not managed to dive into many of them. I don't have fixes for
the three kernel issues above.

So I'd ask that people be patient and if you have some time to help with
any of these problems, I'd welcome it. Most people are off on vacation
now which is going to leave me with a challenge.

Cheers,

Richard

[i] https://autobuilder.yoctoproject.org/main/builders/nightly-ppc-lsb/builds/129/steps/BuildImages/logs/stdio
[ii] https://autobuilder.yoctoproject.org/main/builders/minnow/builds/129/steps/BuildImages/logs/stdio
[iii] https://autobuilder.yoctoproject.org/main/builders/nightly-fsl-arm/builds/128/steps/BuildImages/logs/stdio
      (5 unknown build failures in this one!)
      https://autobuilder.yoctoproject.org/main/builders/nightly-fsl-ppc/builds/129/steps/BuildImages/logs/stdio
      (only 3 here)

Re: Merging problems

[Richard Purdie]

On Fri, 2014-12-19 at 10:28 +0000, Richard Purdie wrote:
> I want to give people a headsup that we're having problems merging
> changes at the moment. We've been doing our best but the number of
> things building up which are causing issues is overwheling our ability
> to fix and stablise the build. It wasn't helped that I took a long
> weekend's vacation last weekend. There are changes being made or tested
> to the autobuilder which also isn't helping.
> 
> The kernel series has several issues:
> 
>   * a random failure in do_kernel_configme [i]

I think I have a handle on this. If you look at the autobuilder failure
it says:

| [INFO] Configuring target/machine combo: "standard/qemuppc"
| [INFO] collecting configs in patches/meta-series

and what concerns me is "patches/meta-series". I my local builds it
says .meta/meta-series. Poking around kern-tools I see:

"""
# determine the meta directory name. The meta directory is at the top level
# of the repository, and is untracked.
meta_dir_options=`git ls-files -o --directory`
for m in $meta_dir_options; do
    if [ -d "$m" ]; then
	meta_dir=`echo $m | sed 's%/%%'`
    fi
done

if [ -z "$meta_dir" ]; then
    meta_dir=".meta"
fi
"""

which means that if a "patches" directory exists it will use it since
the command looks for untracked directories. I also noticed that some
places define the default as ".meta", some as "meta" and they look a bit
confused but that is probably a separate issue.

The question is then how does a "patches" directory end up in the kernel
source. I was able to create one with the commands:

MACHINE=qemuppc bitbake linux-yocto perf -c clean
MACHINE=qemuppc bitbake linux-yocto -c patch
MACHINE=qemuppc bitbake perf -c unpack
MACHINE=qemuppc bitbake linux-yocto -c kernel_configme

which doesn't fail like the autobuilder but does put the metadata into
the wrong place with the wrong data (into patches). I'm therefore
guessing this is a big horrible race.

Why does perf -c unpack create a patches directory? base.bbclass has:

do_unpack[cleandirs] = "${S}/patches"

The fix is therefore probably to not run the fetch/unpack/patch tasks in
kernelsrc.bbclass.

Cheers,

Richard

Re: Merging problems

[Bruce Ashfield]

On 2014-12-19, 7:41 AM, Richard Purdie wrote:
> On Fri, 2014-12-19 at 10:28 +0000, Richard Purdie wrote:
>> I want to give people a headsup that we're having problems merging
>> changes at the moment. We've been doing our best but the number of
>> things building up which are causing issues is overwheling our ability
>> to fix and stablise the build. It wasn't helped that I took a long
>> weekend's vacation last weekend. There are changes being made or tested
>> to the autobuilder which also isn't helping.
>>
>> The kernel series has several issues:
>>
>>    * a random failure in do_kernel_configme [i]
>
> I think I have a handle on this. If you look at the autobuilder failure
> it says:

do you have a link to the actual failure ?

>
> | [INFO] Configuring target/machine combo: "standard/qemuppc"
> | [INFO] collecting configs in patches/meta-series
>
> and what concerns me is "patches/meta-series". I my local builds it
> says .meta/meta-series. Poking around kern-tools I see:
>
> """
> # determine the meta directory name. The meta directory is at the top level
> # of the repository, and is untracked.
> meta_dir_options=`git ls-files -o --directory`
> for m in $meta_dir_options; do
>      if [ -d "$m" ]; then
> 	meta_dir=`echo $m | sed 's%/%%'`
>      fi
> done
>
> if [ -z "$meta_dir" ]; then
>      meta_dir=".meta"
> fi
> """
>
> which means that if a "patches" directory exists it will use it since
> the command looks for untracked directories. I also noticed that some
> places define the default as ".meta", some as "meta" and they look a bit
> confused but that is probably a separate issue.

They are consistent .. both are supported, we migrated from 'meta' to
.meta some time ago, but there are old trees that still have to build.

>
> The question is then how does a "patches" directory end up in the kernel
> source. I was able to create one with the commands:
>
> MACHINE=qemuppc bitbake linux-yocto perf -c clean
> MACHINE=qemuppc bitbake linux-yocto -c patch
> MACHINE=qemuppc bitbake perf -c unpack
> MACHINE=qemuppc bitbake linux-yocto -c kernel_configme
>
> which doesn't fail like the autobuilder but does put the metadata into
> the wrong place with the wrong data (into patches). I'm therefore
> guessing this is a big horrible race.
>
> Why does perf -c unpack create a patches directory? base.bbclass has:
>
> do_unpack[cleandirs] = "${S}/patches"
>
> The fix is therefore probably to not run the fetch/unpack/patch tasks in
> kernelsrc.bbclass.

Very likely. I'll have a look. I noticed about 6 months ago that a patches
directory was being created .. even when it was never going to be used.
I took steps to remove it before the linux-yocto builds continued, but
apparently it is sneaking back in through other means :)

Bruce

>
> Cheers,
>
> Richard
>
>

Re: Merging problems

[Bruce Ashfield]

On Fri, Dec 19, 2014 at 8:07 AM, Bruce Ashfield
<bruce.ashfield@windriver.com> wrote:
> On 2014-12-19, 7:41 AM, Richard Purdie wrote:
>>
>> On Fri, 2014-12-19 at 10:28 +0000, Richard Purdie wrote:
>>>
>>> I want to give people a headsup that we're having problems merging
>>> changes at the moment. We've been doing our best but the number of
>>> things building up which are causing issues is overwheling our ability
>>> to fix and stablise the build. It wasn't helped that I took a long
>>> weekend's vacation last weekend. There are changes being made or tested
>>> to the autobuilder which also isn't helping.
>>>
>>> The kernel series has several issues:
>>>
>>>    * a random failure in do_kernel_configme [i]
>>
>>
>> I think I have a handle on this. If you look at the autobuilder failure
>> it says:
>
>
> do you have a link to the actual failure ?

Aw crap. I see it in your original email now. I'll have a look.

Bruce

>
>>
>> | [INFO] Configuring target/machine combo: "standard/qemuppc"
>> | [INFO] collecting configs in patches/meta-series
>>
>> and what concerns me is "patches/meta-series". I my local builds it
>> says .meta/meta-series. Poking around kern-tools I see:
>>
>> """
>> # determine the meta directory name. The meta directory is at the top
>> level
>> # of the repository, and is untracked.
>> meta_dir_options=`git ls-files -o --directory`
>> for m in $meta_dir_options; do
>>      if [ -d "$m" ]; then
>>         meta_dir=`echo $m | sed 's%/%%'`
>>      fi
>> done
>>
>> if [ -z "$meta_dir" ]; then
>>      meta_dir=".meta"
>> fi
>> """
>>
>> which means that if a "patches" directory exists it will use it since
>> the command looks for untracked directories. I also noticed that some
>> places define the default as ".meta", some as "meta" and they look a bit
>> confused but that is probably a separate issue.
>
>
> They are consistent .. both are supported, we migrated from 'meta' to
> .meta some time ago, but there are old trees that still have to build.
>
>>
>> The question is then how does a "patches" directory end up in the kernel
>> source. I was able to create one with the commands:
>>
>> MACHINE=qemuppc bitbake linux-yocto perf -c clean
>> MACHINE=qemuppc bitbake linux-yocto -c patch
>> MACHINE=qemuppc bitbake perf -c unpack
>> MACHINE=qemuppc bitbake linux-yocto -c kernel_configme
>>
>> which doesn't fail like the autobuilder but does put the metadata into
>> the wrong place with the wrong data (into patches). I'm therefore
>> guessing this is a big horrible race.
>>
>> Why does perf -c unpack create a patches directory? base.bbclass has:
>>
>> do_unpack[cleandirs] = "${S}/patches"
>>
>> The fix is therefore probably to not run the fetch/unpack/patch tasks in
>> kernelsrc.bbclass.
>
>
> Very likely. I'll have a look. I noticed about 6 months ago that a patches
> directory was being created .. even when it was never going to be used.
> I took steps to remove it before the linux-yocto builds continued, but
> apparently it is sneaking back in through other means :)
>
> Bruce
>
>
>>
>> Cheers,
>>
>> Richard
>>
>>
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"

Re: Merging problems

[Richard Purdie]

On Fri, 2014-12-19 at 08:07 -0500, Bruce Ashfield wrote:
> On 2014-12-19, 7:41 AM, Richard Purdie wrote:
> > On Fri, 2014-12-19 at 10:28 +0000, Richard Purdie wrote:
> >> I want to give people a headsup that we're having problems merging
> >> changes at the moment. We've been doing our best but the number of
> >> things building up which are causing issues is overwheling our ability
> >> to fix and stablise the build. It wasn't helped that I took a long
> >> weekend's vacation last weekend. There are changes being made or tested
> >> to the autobuilder which also isn't helping.
> >>
> >> The kernel series has several issues:
> >>
> >>    * a random failure in do_kernel_configme [i]
> >
> > I think I have a handle on this. If you look at the autobuilder failure
> > it says:
> 
> do you have a link to the actual failure ?

It was in a previous mail:

https://autobuilder.yoctoproject.org/main/builders/nightly-ppc-lsb/builds/129/steps/BuildImages/logs/stdio

> > The fix is therefore probably to not run the fetch/unpack/patch tasks in
> > kernelsrc.bbclass.
> 
> Very likely. I'll have a look. I noticed about 6 months ago that a patches
> directory was being created .. even when it was never going to be used.
> I took steps to remove it before the linux-yocto builds continued, but
> apparently it is sneaking back in through other means :)

kernelsrc.bbclass sets ${S} to point at the directory so other recipes
can create it.

Cheers,

Richard

Re: Merging problems

[Bruce Ashfield]

On Fri, Dec 19, 2014 at 8:15 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Fri, 2014-12-19 at 08:07 -0500, Bruce Ashfield wrote:
>> On 2014-12-19, 7:41 AM, Richard Purdie wrote:
>> > On Fri, 2014-12-19 at 10:28 +0000, Richard Purdie wrote:
>> >> I want to give people a headsup that we're having problems merging
>> >> changes at the moment. We've been doing our best but the number of
>> >> things building up which are causing issues is overwheling our ability
>> >> to fix and stablise the build. It wasn't helped that I took a long
>> >> weekend's vacation last weekend. There are changes being made or tested
>> >> to the autobuilder which also isn't helping.
>> >>
>> >> The kernel series has several issues:
>> >>
>> >>    * a random failure in do_kernel_configme [i]
>> >
>> > I think I have a handle on this. If you look at the autobuilder failure
>> > it says:
>>
>> do you have a link to the actual failure ?
>
> It was in a previous mail:
>
> https://autobuilder.yoctoproject.org/main/builders/nightly-ppc-lsb/builds/129/steps/BuildImages/logs/stdio
>
>> > The fix is therefore probably to not run the fetch/unpack/patch tasks in
>> > kernelsrc.bbclass.
>>
>> Very likely. I'll have a look. I noticed about 6 months ago that a patches
>> directory was being created .. even when it was never going to be used.
>> I took steps to remove it before the linux-yocto builds continued, but
>> apparently it is sneaking back in through other means :)
>
> kernelsrc.bbclass sets ${S} to point at the directory so other recipes
> can create it.

aha yes. I'm familiar with doing battle with 'patches' being created
in the build
directory,  it broke the detection of the directory that is used to hold meta
data .. and cost me about a week in August fixing things up.

With your latest patch, which problems are still outstanding ? I can have a
look at them.

Bruce

>
> Cheers,
>
> Richard
>
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"

Re: Merging problems

[Richard Purdie]

So where are we at?

Thanks to some great help from Ross, we have a number of patches merged
and many of the issues in my last email have been addressed. I'm
continuing to struggle with the kernel series. The last build on the
autobuilder highlighted that:

* there are problems in boot-directdisk.bbclass (have a fix)
* there is a do_rootfs/do_package_qa race (have a fix)
* the report-error.bbclass tasks could crash (have a fix)
* the kernelmodule sanity tests were failing (have a fix)
* qemumips gdb is failing to compile, probably due to new kernel 
  headers (no fix as yet)
* systemd sanity QA tests continue to fail on xorg and systemd-login
  (no fix as yet)
* there are continuing problems with linux-imx from meta-fsl-arm, I 
  thought these were addressed but clearly not :(

Ideally I'd like to take some time off over the holidays but I can't see
that happening until the patch queues are under some kind of control :(.

Cheers,

Richard

Re: Merging problems

[Bruce Ashfield]

On 2014-12-20 6:15 AM, Richard Purdie wrote:
> So where are we at?
>
> Thanks to some great help from Ross, we have a number of patches merged
> and many of the issues in my last email have been addressed. I'm
> continuing to struggle with the kernel series. The last build on the
> autobuilder highlighted that:
>
> * there are problems in boot-directdisk.bbclass (have a fix)
> * there is a do_rootfs/do_package_qa race (have a fix)
> * the report-error.bbclass tasks could crash (have a fix)
> * the kernelmodule sanity tests were failing (have a fix)
> * qemumips gdb is failing to compile, probably due to new kernel
>    headers (no fix as yet)
> * systemd sanity QA tests continue to fail on xorg and systemd-login
>    (no fix as yet)
> * there are continuing problems with linux-imx from meta-fsl-arm, I
>    thought these were addressed but clearly not :(
>
> Ideally I'd like to take some time off over the holidays but I can't see
> that happening until the patch queues are under some kind of control :(.

Let me know if there are one of these that you want me to take. I'm all
for pitching in and getting everyone some down time.

Bruce

>
> Cheers,
>
> Richard
>
>

Re: Merging problems

[Otavio Salvador]

On Sat, Dec 20, 2014 at 9:15 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> So where are we at?
...
> * there are continuing problems with linux-imx from meta-fsl-arm, I
>   thought these were addressed but clearly not :(

I have addressed those:

commit 150b8fe5fb428f9d62b8d5e589ba50bcb64bdb6a (HEAD,
yocto/master-next, pending)
Author: Otavio Salvador <otavio@ossystems.com.br>
Date:   Sat Dec 20 17:11:24 2014 -0200

    fsl-kernel-localversion.bbclass: Fix building using out-of-tree

    Since OE-Core:df9975f4 (kernel: Fix non linux-yocto builds) the way
    the build of kernel is done has change. Now the source code is moved
    placed at STAGING_KERNEL_DIR and the build is done at ${WORKDIR}/build
    which is B.

    This patch adapts the configure task prepending to cope with those changes.

    Change-Id: I9ea4254ffd56e4b7201daf9a90b52756a399c0f6
    Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>


This is under meta-fsl-arm master-next branch so people give it a go
in AB and let me know about any other issue you have.

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: Merging problems

[Richard Purdie]

On Sat, 2014-12-20 at 17:19 -0200, Otavio Salvador wrote:
> On Sat, Dec 20, 2014 at 9:15 AM, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > So where are we at?
> ...
> > * there are continuing problems with linux-imx from meta-fsl-arm, I
> >   thought these were addressed but clearly not :(
> 
> I have addressed those:
> 
> commit 150b8fe5fb428f9d62b8d5e589ba50bcb64bdb6a (HEAD,
> yocto/master-next, pending)
> Author: Otavio Salvador <otavio@ossystems.com.br>
> Date:   Sat Dec 20 17:11:24 2014 -0200
> 
>     fsl-kernel-localversion.bbclass: Fix building using out-of-tree
> 
>     Since OE-Core:df9975f4 (kernel: Fix non linux-yocto builds) the way
>     the build of kernel is done has change. Now the source code is moved
>     placed at STAGING_KERNEL_DIR and the build is done at ${WORKDIR}/build
>     which is B.
> 
>     This patch adapts the configure task prepending to cope with those changes.
> 
>     Change-Id: I9ea4254ffd56e4b7201daf9a90b52756a399c0f6
>     Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>

I had tweaked kernel.bbclass so you shouldn't need to do that, the
system notices if a kernel is configured in ${S} and moves the
configuration to ${B}. I also fixed some other race issues which were
leading to the makefile problems on older autobuilders.

> This is under meta-fsl-arm master-next branch so people give it a go
> in AB and let me know about any other issue you have.

https://autobuilder.yoctoproject.org/main/builders/nightly-fsl-arm-lsb/builds/131/steps/BuildImages/logs/stdio

Is the latest failure (fsl-arm built ok), I think this may be a parallel
make race?

Cheers,

Richard

Re: Merging problems

[Richard Purdie]

On Sat, 2014-12-20 at 10:13 -0500, Bruce Ashfield wrote:
> On 2014-12-20 6:15 AM, Richard Purdie wrote:
> > So where are we at?
> >
> > Thanks to some great help from Ross, we have a number of patches merged
> > and many of the issues in my last email have been addressed. I'm
> > continuing to struggle with the kernel series. The last build on the
> > autobuilder highlighted that:
> >
> > * there are problems in boot-directdisk.bbclass (have a fix)
> > * there is a do_rootfs/do_package_qa race (have a fix)
> > * the report-error.bbclass tasks could crash (have a fix)
> > * the kernelmodule sanity tests were failing (have a fix)
> > * qemumips gdb is failing to compile, probably due to new kernel
> >    headers (no fix as yet)
> > * systemd sanity QA tests continue to fail on xorg and systemd-login
> >    (no fix as yet)
> > * there are continuing problems with linux-imx from meta-fsl-arm, I
> >    thought these were addressed but clearly not :(
> >
> > Ideally I'd like to take some time off over the holidays but I can't see
> > that happening until the patch queues are under some kind of control :(.
> 
> Let me know if there are one of these that you want me to take. I'm all
> for pitching in and getting everyone some down time.

Thanks, I think with luck this is now nearly under control.

The systemd issue remains but I'm going to let someone else tackle that,
its not from any recent patches.

There was another fsl-arm linux-imx failure which I've mentioned to
Otavio. It looks like a parallel make race issue to me, probably
specific to the imx kernel.

I'm still waiting on the autobuilder but things are looking a *lot*
greener.

Cheers,

Richard

Re: Merging problems

[Richard Purdie]

On Sat, 2014-12-20 at 10:13 -0500, Bruce Ashfield wrote:
> On 2014-12-20 6:15 AM, Richard Purdie wrote:
> > So where are we at?
> >
> > Thanks to some great help from Ross, we have a number of patches merged
> > and many of the issues in my last email have been addressed. I'm
> > continuing to struggle with the kernel series. The last build on the
> > autobuilder highlighted that:
> >
> > * there are problems in boot-directdisk.bbclass (have a fix)
> > * there is a do_rootfs/do_package_qa race (have a fix)
> > * the report-error.bbclass tasks could crash (have a fix)
> > * the kernelmodule sanity tests were failing (have a fix)
> > * qemumips gdb is failing to compile, probably due to new kernel
> >    headers (no fix as yet)
> > * systemd sanity QA tests continue to fail on xorg and systemd-login
> >    (no fix as yet)
> > * there are continuing problems with linux-imx from meta-fsl-arm, I
> >    thought these were addressed but clearly not :(
> >
> > Ideally I'd like to take some time off over the holidays but I can't see
> > that happening until the patch queues are under some kind of control :(.
> 
> Let me know if there are one of these that you want me to take. I'm all
> for pitching in and getting everyone some down time.

Looks like I spoke too soon:

https://autobuilder.yoctoproject.org/main/builders/nightly-x86/builds/132/steps/BuildImages_1/logs/stdio

which seems to be a race over the kernel source directory. I'm guessing
the file in question is a temporary build artefact of lttng-modules
which was running at the same time? Any way we can avoid temp files in
the kernel source dir?

Cheers,

Richard

Re: Merging problems

[Bruce Ashfield]

On 2014-12-20 5:40 PM, Richard Purdie wrote:
> On Sat, 2014-12-20 at 10:13 -0500, Bruce Ashfield wrote:
>> On 2014-12-20 6:15 AM, Richard Purdie wrote:
>>> So where are we at?
>>>
>>> Thanks to some great help from Ross, we have a number of patches merged
>>> and many of the issues in my last email have been addressed. I'm
>>> continuing to struggle with the kernel series. The last build on the
>>> autobuilder highlighted that:
>>>
>>> * there are problems in boot-directdisk.bbclass (have a fix)
>>> * there is a do_rootfs/do_package_qa race (have a fix)
>>> * the report-error.bbclass tasks could crash (have a fix)
>>> * the kernelmodule sanity tests were failing (have a fix)
>>> * qemumips gdb is failing to compile, probably due to new kernel
>>>     headers (no fix as yet)
>>> * systemd sanity QA tests continue to fail on xorg and systemd-login
>>>     (no fix as yet)
>>> * there are continuing problems with linux-imx from meta-fsl-arm, I
>>>     thought these were addressed but clearly not :(
>>>
>>> Ideally I'd like to take some time off over the holidays but I can't see
>>> that happening until the patch queues are under some kind of control :(.
>>
>> Let me know if there are one of these that you want me to take. I'm all
>> for pitching in and getting everyone some down time.
>
> Looks like I spoke too soon:
>
> https://autobuilder.yoctoproject.org/main/builders/nightly-x86/builds/132/steps/BuildImages_1/logs/stdio

Amazing how those race conditions pop out on the builders .. I probably
built this 100 times, and never managed to trigger these.

>
> which seems to be a race over the kernel source directory. I'm guessing
> the file in question is a temporary build artefact of lttng-modules
> which was running at the same time? Any way we can avoid temp files in
> the kernel source dir?

Hmm. The modules should already have their output directory set to their
own source (not the kernel), so they really shouldn't be dropping down
any temp files.

I was only using the cpio method to copy the source since that is what
was already being used .. and it is a bit faster. The stat of the
files to the pipe is what is catching us here.

The question is .. do we really care ? A straight copy of the files wouldn't
be so sensitive to this, or we could explicitly exclude then in the
existing cpio pipe. That would buy some time to track down what is
touching down the tmp file in the kernel build process, and I can't see
how the dual use of that staged directory will cause us a problem on
the copied kernel source side (we do clean things up before packaging).

Bruce

>
> Cheers,
>
> Richard
>
>

Re: Merging problems

[Richard Purdie]

On Sat, 2014-12-20 at 21:04 -0500, Bruce Ashfield wrote:
> On 2014-12-20 5:40 PM, Richard Purdie wrote:
> > On Sat, 2014-12-20 at 10:13 -0500, Bruce Ashfield wrote:
> >> On 2014-12-20 6:15 AM, Richard Purdie wrote:
> >>> So where are we at?
> >>>
> >>> Thanks to some great help from Ross, we have a number of patches merged
> >>> and many of the issues in my last email have been addressed. I'm
> >>> continuing to struggle with the kernel series. The last build on the
> >>> autobuilder highlighted that:
> >>>
> >>> * there are problems in boot-directdisk.bbclass (have a fix)
> >>> * there is a do_rootfs/do_package_qa race (have a fix)
> >>> * the report-error.bbclass tasks could crash (have a fix)
> >>> * the kernelmodule sanity tests were failing (have a fix)
> >>> * qemumips gdb is failing to compile, probably due to new kernel
> >>>     headers (no fix as yet)
> >>> * systemd sanity QA tests continue to fail on xorg and systemd-login
> >>>     (no fix as yet)
> >>> * there are continuing problems with linux-imx from meta-fsl-arm, I
> >>>     thought these were addressed but clearly not :(
> >>>
> >>> Ideally I'd like to take some time off over the holidays but I can't see
> >>> that happening until the patch queues are under some kind of control :(.
> >>
> >> Let me know if there are one of these that you want me to take. I'm all
> >> for pitching in and getting everyone some down time.
> >
> > Looks like I spoke too soon:
> >
> > https://autobuilder.yoctoproject.org/main/builders/nightly-x86/builds/132/steps/BuildImages_1/logs/stdio
> 
> Amazing how those race conditions pop out on the builders .. I probably
> built this 100 times, and never managed to trigger these.

We also have one more:

https://autobuilder.yoctoproject.org/main/builders/nightly-ppc/builds/134/steps/Running%20Sanity%20Tests/logs/stdio

On target module building for ppc fails.

> > which seems to be a race over the kernel source directory. I'm guessing
> > the file in question is a temporary build artefact of lttng-modules
> > which was running at the same time? Any way we can avoid temp files in
> > the kernel source dir?
> 
> Hmm. The modules should already have their output directory set to their
> own source (not the kernel), so they really shouldn't be dropping down
> any temp files.
> 
> I was only using the cpio method to copy the source since that is what
> was already being used .. and it is a bit faster. The stat of the
> files to the pipe is what is catching us here.
> 
> The question is .. do we really care ? A straight copy of the files wouldn't
> be so sensitive to this, or we could explicitly exclude then in the
> existing cpio pipe. That would buy some time to track down what is
> touching down the tmp file in the kernel build process, and I can't see
> how the dual use of that staged directory will cause us a problem on
> the copied kernel source side (we do clean things up before packaging).

Well, I think we need to get to the bottom of this. I made some
experiments. The best results were from hacking scripts/Kbuild.include
where these .tmp files get created, I hacked it not to remove them.

When you do that, it becomes clear that do_make_scripts is the task
which is causing the problem.

The world has changed since we last visited the scripts problem, we
could move that task to the main kernel build now and have the modules
and kernel-devsrc depend on it? The function in module.bbclass would
then become a placeholder?

Cheers,

Richard

Re: Merging problems

[Richard Purdie]

On Sun, 2014-12-21 at 11:27 +0000, Richard Purdie wrote:
> On Sat, 2014-12-20 at 21:04 -0500, Bruce Ashfield wrote:
> > On 2014-12-20 5:40 PM, Richard Purdie wrote:
> > > On Sat, 2014-12-20 at 10:13 -0500, Bruce Ashfield wrote:
> > >> On 2014-12-20 6:15 AM, Richard Purdie wrote:
> > >>> So where are we at?
> > >>>
> > >>> Thanks to some great help from Ross, we have a number of patches merged
> > >>> and many of the issues in my last email have been addressed. I'm
> > >>> continuing to struggle with the kernel series. The last build on the
> > >>> autobuilder highlighted that:
> > >>>
> > >>> * there are problems in boot-directdisk.bbclass (have a fix)
> > >>> * there is a do_rootfs/do_package_qa race (have a fix)
> > >>> * the report-error.bbclass tasks could crash (have a fix)
> > >>> * the kernelmodule sanity tests were failing (have a fix)
> > >>> * qemumips gdb is failing to compile, probably due to new kernel
> > >>>     headers (no fix as yet)
> > >>> * systemd sanity QA tests continue to fail on xorg and systemd-login
> > >>>     (no fix as yet)
> > >>> * there are continuing problems with linux-imx from meta-fsl-arm, I
> > >>>     thought these were addressed but clearly not :(
> > >>>
> > >>> Ideally I'd like to take some time off over the holidays but I can't see
> > >>> that happening until the patch queues are under some kind of control :(.
> > >>
> > >> Let me know if there are one of these that you want me to take. I'm all
> > >> for pitching in and getting everyone some down time.
> > >
> > > Looks like I spoke too soon:
> > >
> > > https://autobuilder.yoctoproject.org/main/builders/nightly-x86/builds/132/steps/BuildImages_1/logs/stdio
> > 
> > Amazing how those race conditions pop out on the builders .. I probably
> > built this 100 times, and never managed to trigger these.
> 
> We also have one more:
> 
> https://autobuilder.yoctoproject.org/main/builders/nightly-ppc/builds/134/steps/Running%20Sanity%20Tests/logs/stdio
> 
> On target module building for ppc fails.
> 
> > > which seems to be a race over the kernel source directory. I'm guessing
> > > the file in question is a temporary build artefact of lttng-modules
> > > which was running at the same time? Any way we can avoid temp files in
> > > the kernel source dir?
> > 
> > Hmm. The modules should already have their output directory set to their
> > own source (not the kernel), so they really shouldn't be dropping down
> > any temp files.
> > 
> > I was only using the cpio method to copy the source since that is what
> > was already being used .. and it is a bit faster. The stat of the
> > files to the pipe is what is catching us here.
> > 
> > The question is .. do we really care ? A straight copy of the files wouldn't
> > be so sensitive to this, or we could explicitly exclude then in the
> > existing cpio pipe. That would buy some time to track down what is
> > touching down the tmp file in the kernel build process, and I can't see
> > how the dual use of that staged directory will cause us a problem on
> > the copied kernel source side (we do clean things up before packaging).
> 
> Well, I think we need to get to the bottom of this. I made some
> experiments. The best results were from hacking scripts/Kbuild.include
> where these .tmp files get created, I hacked it not to remove them.
> 
> When you do that, it becomes clear that do_make_scripts is the task
> which is causing the problem.
> 
> The world has changed since we last visited the scripts problem, we
> could move that task to the main kernel build now and have the modules
> and kernel-devsrc depend on it? The function in module.bbclass would
> then become a placeholder?

After a lot of trail and error I've sent out a couple of patches for the
above two issues. These fix the immediate problems however I think there
is more cleanup than needs to be done early next year.

I filed:

https://bugzilla.yoctoproject.org/show_bug.cgi?id=7095

since the overlap between kernel-dev and kernel-devsrc is problematic
and should get resolved. This may help resolve the powerpc problem in a
nicer way if devsrc is *just* the source and dev has some artefacts like
the build config. It may be we have a separate package for the artefacts
too, I'm open to ideas.

I did start to experiment with cleaning up the scripts handling in the
build tree:
http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=760fa19a11165ff02c5df38cc89335fc503afba2

I have mixed feelings about it that, we really do need to try and do
something about it though.

Cheers,

Richard

 

Re: Merging problems

[Richard Purdie]

I thought we were over the hump and I merged the series. Next autobuild
went rather red due to sstate cache usage. Reproducer was:

bitbake kernel-devsrc -c cleansstate
bitbake virtual/kernel
bitbake virtual/kernel -c clean
bitbake kernel-devsrc -c install

I've pushed yet another patch and restarted the build.

Cheers,

Richard

Re: Merging problems

[Bruce Ashfield]

On Sun, Dec 21, 2014 at 7:27 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Sat, 2014-12-20 at 21:04 -0500, Bruce Ashfield wrote:
>> On 2014-12-20 5:40 PM, Richard Purdie wrote:
>> > On Sat, 2014-12-20 at 10:13 -0500, Bruce Ashfield wrote:
>> >> On 2014-12-20 6:15 AM, Richard Purdie wrote:
>> >>> So where are we at?
>> >>>
>> >>> Thanks to some great help from Ross, we have a number of patches merged
>> >>> and many of the issues in my last email have been addressed. I'm
>> >>> continuing to struggle with the kernel series. The last build on the
>> >>> autobuilder highlighted that:
>> >>>
>> >>> * there are problems in boot-directdisk.bbclass (have a fix)
>> >>> * there is a do_rootfs/do_package_qa race (have a fix)
>> >>> * the report-error.bbclass tasks could crash (have a fix)
>> >>> * the kernelmodule sanity tests were failing (have a fix)
>> >>> * qemumips gdb is failing to compile, probably due to new kernel
>> >>>     headers (no fix as yet)
>> >>> * systemd sanity QA tests continue to fail on xorg and systemd-login
>> >>>     (no fix as yet)
>> >>> * there are continuing problems with linux-imx from meta-fsl-arm, I
>> >>>     thought these were addressed but clearly not :(
>> >>>
>> >>> Ideally I'd like to take some time off over the holidays but I can't see
>> >>> that happening until the patch queues are under some kind of control :(.
>> >>
>> >> Let me know if there are one of these that you want me to take. I'm all
>> >> for pitching in and getting everyone some down time.
>> >
>> > Looks like I spoke too soon:
>> >
>> > https://autobuilder.yoctoproject.org/main/builders/nightly-x86/builds/132/steps/BuildImages_1/logs/stdio
>>
>> Amazing how those race conditions pop out on the builders .. I probably
>> built this 100 times, and never managed to trigger these.
>
> We also have one more:
>
> https://autobuilder.yoctoproject.org/main/builders/nightly-ppc/builds/134/steps/Running%20Sanity%20Tests/logs/stdio
>
> On target module building for ppc fails.
>
>> > which seems to be a race over the kernel source directory. I'm guessing
>> > the file in question is a temporary build artefact of lttng-modules
>> > which was running at the same time? Any way we can avoid temp files in
>> > the kernel source dir?
>>
>> Hmm. The modules should already have their output directory set to their
>> own source (not the kernel), so they really shouldn't be dropping down
>> any temp files.
>>
>> I was only using the cpio method to copy the source since that is what
>> was already being used .. and it is a bit faster. The stat of the
>> files to the pipe is what is catching us here.
>>
>> The question is .. do we really care ? A straight copy of the files wouldn't
>> be so sensitive to this, or we could explicitly exclude then in the
>> existing cpio pipe. That would buy some time to track down what is
>> touching down the tmp file in the kernel build process, and I can't see
>> how the dual use of that staged directory will cause us a problem on
>> the copied kernel source side (we do clean things up before packaging).
>
> Well, I think we need to get to the bottom of this. I made some
> experiments. The best results were from hacking scripts/Kbuild.include
> where these .tmp files get created, I hacked it not to remove them.

back online after a 10 jaunt in the car.

I definitely agree that we needed to understand and fix this .. just that
I would be away from my keyboard for about 24 hours, so putting something
in place for the short term was the thought.

>
> When you do that, it becomes clear that do_make_scripts is the task
> which is causing the problem.
>
> The world has changed since we last visited the scripts problem, we
> could move that task to the main kernel build now and have the modules
> and kernel-devsrc depend on it? The function in module.bbclass would
> then become a placeholder?

We can do that, and it makes sense there. Since both have a dependency
on them being created .. moving it to a single place solves our issue and
I don't see a bad side effect of the move.

Bruce

>
> Cheers,
>
> Richard
>
>
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"

Re: Merging problems

[Bruce Ashfield]

On Sun, Dec 21, 2014 at 8:35 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Sun, 2014-12-21 at 11:27 +0000, Richard Purdie wrote:
>> On Sat, 2014-12-20 at 21:04 -0500, Bruce Ashfield wrote:
>> > On 2014-12-20 5:40 PM, Richard Purdie wrote:
>> > > On Sat, 2014-12-20 at 10:13 -0500, Bruce Ashfield wrote:
>> > >> On 2014-12-20 6:15 AM, Richard Purdie wrote:
>> > >>> So where are we at?
>> > >>>
>> > >>> Thanks to some great help from Ross, we have a number of patches merged
>> > >>> and many of the issues in my last email have been addressed. I'm
>> > >>> continuing to struggle with the kernel series. The last build on the
>> > >>> autobuilder highlighted that:
>> > >>>
>> > >>> * there are problems in boot-directdisk.bbclass (have a fix)
>> > >>> * there is a do_rootfs/do_package_qa race (have a fix)
>> > >>> * the report-error.bbclass tasks could crash (have a fix)
>> > >>> * the kernelmodule sanity tests were failing (have a fix)
>> > >>> * qemumips gdb is failing to compile, probably due to new kernel
>> > >>>     headers (no fix as yet)
>> > >>> * systemd sanity QA tests continue to fail on xorg and systemd-login
>> > >>>     (no fix as yet)
>> > >>> * there are continuing problems with linux-imx from meta-fsl-arm, I
>> > >>>     thought these were addressed but clearly not :(
>> > >>>
>> > >>> Ideally I'd like to take some time off over the holidays but I can't see
>> > >>> that happening until the patch queues are under some kind of control :(.
>> > >>
>> > >> Let me know if there are one of these that you want me to take. I'm all
>> > >> for pitching in and getting everyone some down time.
>> > >
>> > > Looks like I spoke too soon:
>> > >
>> > > https://autobuilder.yoctoproject.org/main/builders/nightly-x86/builds/132/steps/BuildImages_1/logs/stdio
>> >
>> > Amazing how those race conditions pop out on the builders .. I probably
>> > built this 100 times, and never managed to trigger these.
>>
>> We also have one more:
>>
>> https://autobuilder.yoctoproject.org/main/builders/nightly-ppc/builds/134/steps/Running%20Sanity%20Tests/logs/stdio
>>
>> On target module building for ppc fails.
>>
>> > > which seems to be a race over the kernel source directory. I'm guessing
>> > > the file in question is a temporary build artefact of lttng-modules
>> > > which was running at the same time? Any way we can avoid temp files in
>> > > the kernel source dir?
>> >
>> > Hmm. The modules should already have their output directory set to their
>> > own source (not the kernel), so they really shouldn't be dropping down
>> > any temp files.
>> >
>> > I was only using the cpio method to copy the source since that is what
>> > was already being used .. and it is a bit faster. The stat of the
>> > files to the pipe is what is catching us here.
>> >
>> > The question is .. do we really care ? A straight copy of the files wouldn't
>> > be so sensitive to this, or we could explicitly exclude then in the
>> > existing cpio pipe. That would buy some time to track down what is
>> > touching down the tmp file in the kernel build process, and I can't see
>> > how the dual use of that staged directory will cause us a problem on
>> > the copied kernel source side (we do clean things up before packaging).
>>
>> Well, I think we need to get to the bottom of this. I made some
>> experiments. The best results were from hacking scripts/Kbuild.include
>> where these .tmp files get created, I hacked it not to remove them.
>>
>> When you do that, it becomes clear that do_make_scripts is the task
>> which is causing the problem.
>>
>> The world has changed since we last visited the scripts problem, we
>> could move that task to the main kernel build now and have the modules
>> and kernel-devsrc depend on it? The function in module.bbclass would
>> then become a placeholder?
>
> After a lot of trail and error I've sent out a couple of patches for the
> above two issues. These fix the immediate problems however I think there
> is more cleanup than needs to be done early next year.


I replied out of order. I should have read this first. :)

>
> I filed:
>
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=7095

I'll grab that.

>
> since the overlap between kernel-dev and kernel-devsrc is problematic
> and should get resolved. This may help resolve the powerpc problem in a
> nicer way if devsrc is *just* the source and dev has some artefacts like
> the build config. It may be we have a separate package for the artefacts
> too, I'm open to ideas.

It's a possibility. I'll throw together some ideas here as well.

Bruce

>
> I did start to experiment with cleaning up the scripts handling in the
> build tree:
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=760fa19a11165ff02c5df38cc89335fc503afba2
>
> I have mixed feelings about it that, we really do need to try and do
> something about it though.
>
> Cheers,
>
> Richard
>
>
>
>
>
>
>
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"

Re: Merging problems

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1882 bytes --]

On Sat, Dec 20, 2014 at 11:15:09AM +0000, Richard Purdie wrote:
> So where are we at?
> 
> Thanks to some great help from Ross, we have a number of patches merged
> and many of the issues in my last email have been addressed. I'm
> continuing to struggle with the kernel series. The last build on the
> autobuilder highlighted that:
> 
> * there are problems in boot-directdisk.bbclass (have a fix)
> * there is a do_rootfs/do_package_qa race (have a fix)
> * the report-error.bbclass tasks could crash (have a fix)
> * the kernelmodule sanity tests were failing (have a fix)
> * qemumips gdb is failing to compile, probably due to new kernel 
>   headers (no fix as yet)
> * systemd sanity QA tests continue to fail on xorg and systemd-login
>   (no fix as yet)
> * there are continuing problems with linux-imx from meta-fsl-arm, I 
>   thought these were addressed but clearly not :(
> 
> Ideally I'd like to take some time off over the holidays but I can't see
> that happening until the patch queues are under some kind of control :(.

One of kernel patches merged last weekend seems to be causing all /usr/src/kernel
files to be installed in sysroot without sstate knowledge (and possibly
missing when kernel is reused from sstate)

sstate-find-cruft.sh from last world build now reports:

ERROR: 52961 issues were found.
ERROR: Following files are installed in sysroot, but not tracked by
sstate:
qemuarm/usr/src/kernel/.git
qemuarm/usr/src/kernel/.git/HEAD
...
qemuarm/usr/src/kernel/.meta
qemuarm/usr/src/kernel/.meta/cfg
qemuarm/usr/src/kernel/.meta/cfg/kernel-cache
qemuarm/usr/src/kernel/.meta/cfg/kernel-cache/00-README\
...
qemuarm/usr/src/kernel/.scmversion
qemuarm/usr/src/kernel/COPYING
qemuarm/usr/src/kernel/CREDITS
qemuarm/usr/src/kernel/Documentation
...

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 188 bytes --]

Re: Merging problems

[Richard Purdie]

On Mon, 2014-12-22 at 17:50 +0100, Martin Jansa wrote:
> On Sat, Dec 20, 2014 at 11:15:09AM +0000, Richard Purdie wrote:
> > So where are we at?
> > 
> > Thanks to some great help from Ross, we have a number of patches merged
> > and many of the issues in my last email have been addressed. I'm
> > continuing to struggle with the kernel series. The last build on the
> > autobuilder highlighted that:
> > 
> > * there are problems in boot-directdisk.bbclass (have a fix)
> > * there is a do_rootfs/do_package_qa race (have a fix)
> > * the report-error.bbclass tasks could crash (have a fix)
> > * the kernelmodule sanity tests were failing (have a fix)
> > * qemumips gdb is failing to compile, probably due to new kernel 
> >   headers (no fix as yet)
> > * systemd sanity QA tests continue to fail on xorg and systemd-login
> >   (no fix as yet)
> > * there are continuing problems with linux-imx from meta-fsl-arm, I 
> >   thought these were addressed but clearly not :(
> > 
> > Ideally I'd like to take some time off over the holidays but I can't see
> > that happening until the patch queues are under some kind of control :(.
> 
> One of kernel patches merged last weekend seems to be causing all /usr/src/kernel
> files to be installed in sysroot without sstate knowledge

That is intentional. The kernel is simply too large and the performance
impact too great to put this under sstate control, we may as well just
run the unpack again. usr/src/kernel should therefore be whitelisted.

>  (and possibly missing when kernel is reused from sstate)

The system should be rerunning the kernel unpack process instead for any
of these cases. If its not, let me know the situation and and we'll
ensure the source does get re-extracted.

Cheers,

Richard