Problem with RDEPENDS in multimachine builds for allarch packages

Problem with RDEPENDS in multimachine builds for allarch packages

[Sergey 'Jin' Bostandzhyan]

Hi,

I recently ran into a problem that I described here:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=8578 and while debugging
this issue an other problem came to light.

I have a multimachine configuration (armv5/imx6) and the same distro for both
of them (poky 2.0 RC based).

I have some allarch packages which have no DEPENDS, however which have an
RDEPENDS_${PN} setting.

I can see that even though there were no changes to the recipe itself and 
also no changes to the recipes listed in RDEPENDS, my recipe still gets 
rebuilt each time.

What we tried:

<bluelightning> so cleansstate the recipe, change machine, cleansstate again,
                then build it, then change machine, build it again and note
                the first task that executes

The logs for these steps, one with RDEPENDS and one without are at the end of
this mail. I can see that with present RDEPENDS "real" tasks get executed
even though no changes occured in the recipe; this is not the case when I 
drop RDEPENDS.

Is this a bug or is this just something that needs to be documented?
What is the "desired" behaviour?

Kind regards,
Sergey


What I get with RDEPENDS is:

***********
NOTE: Running setscene task 347 of 374 (/oe/dss-oe/yocto/dS/meta-digitalstrom-devel/recipes-digitalstrom/dss11-help/dss11-installation-manual-tr-tr_0.9.bb, do_populate_sysroot_setscene)
NOTE: Running setscene task 350 of 374 (/oe/dss-oe/yocto/dS/meta-digitalstrom-devel/recipes-digitalstrom/dss11-help/dss11-installation-manual-tr-tr_0.9.bb, do_packagedata_setscene)
NOTE: recipe dss11-installation-manual-tr-tr-0.9-r2: task do_populate_sysroot_setscene: Started
NOTE: recipe dss11-installation-manual-tr-tr-0.9-r2: task do_packagedata_setscene: Started
NOTE: recipe dss11-installation-manual-tr-tr-0.9-r2: task do_populate_sysroot_setscene: Succeeded
NOTE: recipe dss11-installation-manual-tr-tr-0.9-r2: task do_packagedata_setscene: Succeeded
NOTE: Running setscene task 352 of 374 (/oe/dss-oe/yocto/dS/meta-digitalstrom-devel/recipes-digitalstrom/dss11-help/dss11-installation-manual-tr-tr_0.9.bb, do_package_setscene)
NOTE: recipe dss11-installation-manual-tr-tr-0.9-r2: task do_package_setscene: Started
NOTE: recipe dss11-installation-manual-tr-tr-0.9-r2: task do_package_setscene: Succeeded
NOTE: Running task 1175 of 1181 (ID: 14, /oe/dss-oe/yocto/dS/meta-digitalstrom-devel/recipes-digitalstrom/dss11-help/dss11-installation-manual-tr-tr_0.9.bb, do_package_write_ipk)
NOTE: Running task 1176 of 1181 (ID: 9, /oe/dss-oe/yocto/dS/meta-digitalstrom-devel/recipes-digitalstrom/dss11-help/dss11-installation-manual-tr-tr_0.9.bb, do_package_qa)
NOTE: recipe dss11-installation-manual-tr-tr-0.9-r2: task do_package_write_ipk: Started
NOTE: recipe dss11-installation-manual-tr-tr-0.9-r2: task do_package_qa: Started
NOTE: recipe dss11-installation-manual-tr-tr-0.9-r2: task do_package_qa: Succeeded
NOTE: recipe dss11-installation-manual-tr-tr-0.9-r2: task do_package_write_ipk: Succeeded
NOTE: Running noexec task 1179 of 1181 (ID: 10, /oe/dss-oe/yocto/dS/meta-digitalstrom-devel/recipes-digitalstrom/dss11-help/dss11-installation-manual-tr-tr_0.9.bb, do_build)
NOTE: Running task 1180 of 1181 (ID: 11, /oe/dss-oe/yocto/dS/meta-digitalstrom-devel/recipes-digitalstrom/dss11-help/dss11-installation-manual-tr-tr_0.9.bb, do_rm_work)
NOTE: recipe dss11-installation-manual-tr-tr-0.9-r2: task do_rm_work: Started
NOTE: recipe dss11-installation-manual-tr-tr-0.9-r2: task do_rm_work: Succeeded
NOTE: Running task 1181 of 1181 (ID: 12, /oe/dss-oe/yocto/dS/meta-digitalstrom-devel/recipes-digitalstrom/dss11-help/dss11-installation-manual-tr-tr_0.9.bb, do_rm_work_all)
NOTE: recipe dss11-installation-manual-tr-tr-0.9-r2: task do_rm_work_all: Started
NOTE: recipe dss11-installation-manual-tr-tr-0.9-r2: task do_rm_work_all: Succeeded
***********

We can see that some "real" tasks are getting executed (do_package_write_ipk).

Now, when I remove the RDEPENDS from my recipe the log gets much shorter:

***********
NOTE: Running setscene task 73 of 76 (/oe/dss-oe/yocto/dS/meta-digitalstrom-devel/recipes-digitalstrom/dss11-help/dss11-installation-manual-tr-tr_0.9.bb, do_populate_sysroot_setscene)
NOTE: Running setscene task 74 of 76 (/oe/dss-oe/yocto/dS/meta-digitalstrom-devel/recipes-digitalstrom/dss11-help/dss11-installation-manual-tr-tr_0.9.bb, do_package_write_ipk_setscene)
NOTE: recipe dss11-installation-manual-tr-tr-0.9-r2: task do_populate_sysroot_setscene: Started
NOTE: recipe dss11-installation-manual-tr-tr-0.9-r2: task do_package_write_ipk_setscene: Started
NOTE: recipe dss11-installation-manual-tr-tr-0.9-r2: task do_populate_sysroot_setscene: Succeeded
NOTE: recipe dss11-installation-manual-tr-tr-0.9-r2: task do_package_write_ipk_setscene: Succeeded
NOTE: Running setscene task 75 of 76 (/oe/dss-oe/yocto/dS/meta-digitalstrom-devel/recipes-digitalstrom/dss11-help/dss11-installation-manual-tr-tr_0.9.bb, do_packagedata_setscene)
NOTE: recipe dss11-installation-manual-tr-tr-0.9-r2: task do_packagedata_setscene: Started
NOTE: recipe dss11-installation-manual-tr-tr-0.9-r2: task do_packagedata_setscene: Succeeded
***********

Here no "real" tasks get executed.

Re: Problem with RDEPENDS in multimachine builds for allarch packages

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1605 bytes --]

On Mon, Oct 26, 2015 at 8:13 AM, Sergey 'Jin' Bostandzhyan <jin@mediatomb.cc
> wrote:

> I recently ran into a problem that I described here:
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=8578 and while debugging
> this issue an other problem came to light.
>
> I have a multimachine configuration (armv5/imx6) and the same distro for
> both
> of them (poky 2.0 RC based).
>
> I have some allarch packages which have no DEPENDS, however which have an
> RDEPENDS_${PN} setting.
>
> I can see that even though there were no changes to the recipe itself and
> also no changes to the recipes listed in RDEPENDS, my recipe still gets
> rebuilt each time.
>
> What we tried:
>
> <bluelightning> so cleansstate the recipe, change machine, cleansstate
> again,
>                 then build it, then change machine, build it again and note
>                 the first task that executes
>
> The logs for these steps, one with RDEPENDS and one without are at the end
> of
> this mail. I can see that with present RDEPENDS "real" tasks get executed
> even though no changes occured in the recipe; this is not the case when I
> drop RDEPENDS.
>
> Is this a bug or is this just something that needs to be documented?
> What is the "desired" behaviour?
>

Afaik the most likely cause is your rdepends isn't allarch, so is being
rebuilt for the new machine, which causes yours to be rebuilt since it
depends on it.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 2193 bytes --]

Re: Problem with RDEPENDS in multimachine builds for allarch packages

[Sergey 'Jin' Bostandzhyan]

On Mon, Oct 26, 2015 at 09:13:22AM -0700, Christopher Larson wrote:
> Afaik the most likely cause is your rdepends isn't allarch, so is being rebuilt
> for the new machine, which causes yours to be rebuilt since it depends on it.

So, that means that anything arch specific listed in RDEPENDS will
always cause the package to rebuild even if there were no changes in the
package itself nor in the RDEPENDS packages? Is this really the desired
behaviour?

I'd expect that if nothing has changed, nothing should get rebuild.. I
often saw a huge number of packages being rebuilt and could never
explain why this was happening, now it's becoming clear that probably
some of them had some RDEPENDS set.

If you say it's not a bug - ok, now I at least know what's happening.
Still, imho it would be nice and it would speed up things if only stuff
that has really changed or stuff that depends on other things that have 
really changed gets triggered.

Kind regards,
Sergey

Re: Problem with RDEPENDS in multimachine builds for allarch packages

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1498 bytes --]

On Mon, Oct 26, 2015 at 05:26:50PM +0100, Sergey 'Jin' Bostandzhyan wrote:
> On Mon, Oct 26, 2015 at 09:13:22AM -0700, Christopher Larson wrote:
> > Afaik the most likely cause is your rdepends isn't allarch, so is being rebuilt
> > for the new machine, which causes yours to be rebuilt since it depends on it.
> 
> So, that means that anything arch specific listed in RDEPENDS will
> always cause the package to rebuild even if there were no changes in the
> package itself nor in the RDEPENDS packages? Is this really the desired
> behaviour?
> 
> I'd expect that if nothing has changed, nothing should get rebuild.. I
> often saw a huge number of packages being rebuilt and could never
> explain why this was happening, now it's becoming clear that probably
> some of them had some RDEPENDS set.
> 
> If you say it's not a bug - ok, now I at least know what's happening.
> Still, imho it would be nice and it would speed up things if only stuff
> that has really changed or stuff that depends on other things that have 
> really changed gets triggered.

You can use
openembedded-core/scripts/sstate-diff-machines.sh
to compare the signatures. It's also useful to store the output
directories in case you see some unexpected rebuilds in some future
build.

This script is used to generate reports like this one:
http://lists.openembedded.org/pipermail/openembedded-core/2015-October/111554.html

Regards,
-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 188 bytes --]

Re: Problem with RDEPENDS in multimachine builds for allarch packages

[Mike Looijmans]

On 26-10-15 16:13, Sergey 'Jin' Bostandzhyan wrote:
> Hi,
>
> I recently ran into a problem that I described here:
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=8578 and while debugging
> this issue an other problem came to light.
>
> I have a multimachine configuration (armv5/imx6) and the same distro for both
> of them (poky 2.0 RC based).
>
> I have some allarch packages which have no DEPENDS, however which have an
> RDEPENDS_${PN} setting.
>
> I can see that even though there were no changes to the recipe itself and
> also no changes to the recipes listed in RDEPENDS, my recipe still gets
> rebuilt each time.
>
> What we tried:
>
> <bluelightning> so cleansstate the recipe, change machine, cleansstate again,
>                  then build it, then change machine, build it again and note
>                  the first task that executes
>
> The logs for these steps, one with RDEPENDS and one without are at the end of
> this mail. I can see that with present RDEPENDS "real" tasks get executed
> even though no changes occured in the recipe; this is not the case when I
> drop RDEPENDS.
>
> Is this a bug or is this just something that needs to be documented?

I think it's the same problem as described here:
http://thread.gmane.org/gmane.comp.handhelds.openembedded.core/70070/focus=70072

> What is the "desired" behaviour?

Apparently, this needless rebuilding is "desired".



Kind regards,

Mike Looijmans
System Expert

TOPIC Embedded Products
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: +31 (0) 499 33 69 79
Telefax: +31 (0) 499 33 69 70
E-mail: mike.looijmans@topicproducts.com
Website: www.topicproducts.com

Please consider the environment before printing this e-mail

Visit us at : Aerospace Electrical Systems Expo Europe which will be held from 17.11.2015 till 19.11.2015, Findorffstrasse 101 Bremen, Germany, Hall 5, stand number C65
http://www.aesexpo.eu

Re: Problem with RDEPENDS in multimachine builds for allarch packages

[Richard Purdie]

On Tue, 2015-10-27 at 09:30 +0100, Mike Looijmans wrote:
> On 26-10-15 16:13, Sergey 'Jin' Bostandzhyan wrote:
> > Hi,
> >
> > I recently ran into a problem that I described here:
> > https://bugzilla.yoctoproject.org/show_bug.cgi?id=8578 and while debugging
> > this issue an other problem came to light.
> >
> > I have a multimachine configuration (armv5/imx6) and the same distro for both
> > of them (poky 2.0 RC based).
> >
> > I have some allarch packages which have no DEPENDS, however which have an
> > RDEPENDS_${PN} setting.
> >
> > I can see that even though there were no changes to the recipe itself and
> > also no changes to the recipes listed in RDEPENDS, my recipe still gets
> > rebuilt each time.
> >
> > What we tried:
> >
> > <bluelightning> so cleansstate the recipe, change machine, cleansstate again,
> >                  then build it, then change machine, build it again and note
> >                  the first task that executes
> >
> > The logs for these steps, one with RDEPENDS and one without are at the end of
> > this mail. I can see that with present RDEPENDS "real" tasks get executed
> > even though no changes occured in the recipe; this is not the case when I
> > drop RDEPENDS.
> >
> > Is this a bug or is this just something that needs to be documented?
> 
> I think it's the same problem as described here:
> http://thread.gmane.org/gmane.comp.handhelds.openembedded.core/70070/focus=70072
> 
> > What is the "desired" behaviour?
> 
> Apparently, this needless rebuilding is "desired".

Its more that we can't tell the difference between a case where it is
needless and a case where it is no. The system therefore rebuilds as if
it didn't, there would be cases where it should have rebuilt and did
not.

Cheers,

Richard

Re: Problem with RDEPENDS in multimachine builds for allarch packages

[Mike Looijmans]

On 27-10-15 09:36, Richard Purdie wrote:
> On Tue, 2015-10-27 at 09:30 +0100, Mike Looijmans wrote:
>> On 26-10-15 16:13, Sergey 'Jin' Bostandzhyan wrote:
>>> Hi,
>>>
>>> I recently ran into a problem that I described here:
>>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=8578 and while debugging
>>> this issue an other problem came to light.
>>>
>>> I have a multimachine configuration (armv5/imx6) and the same distro for both
>>> of them (poky 2.0 RC based).
>>>
>>> I have some allarch packages which have no DEPENDS, however which have an
>>> RDEPENDS_${PN} setting.
>>>
>>> I can see that even though there were no changes to the recipe itself and
>>> also no changes to the recipes listed in RDEPENDS, my recipe still gets
>>> rebuilt each time.
>>>
>>> What we tried:
>>>
>>> <bluelightning> so cleansstate the recipe, change machine, cleansstate again,
>>>                   then build it, then change machine, build it again and note
>>>                   the first task that executes
>>>
>>> The logs for these steps, one with RDEPENDS and one without are at the end of
>>> this mail. I can see that with present RDEPENDS "real" tasks get executed
>>> even though no changes occured in the recipe; this is not the case when I
>>> drop RDEPENDS.
>>>
>>> Is this a bug or is this just something that needs to be documented?
>>
>> I think it's the same problem as described here:
>> http://thread.gmane.org/gmane.comp.handhelds.openembedded.core/70070/focus=70072
>>
>>> What is the "desired" behaviour?
>>
>> Apparently, this needless rebuilding is "desired".
>
> Its more that we can't tell the difference between a case where it is
> needless and a case where it is no. The system therefore rebuilds as if
> it didn't, there would be cases where it should have rebuilt and did
> not.

But that would not solve any issue at all.

Whatever package got built last will end up in the feed, under the "all" 
architecture. So if machine X and Y have different versions of some package P, 
and the build for Y ran last, machine X will still install Y's version of P.



Kind regards,

Mike Looijmans
System Expert

TOPIC Embedded Products
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: +31 (0) 499 33 69 79
Telefax: +31 (0) 499 33 69 70
E-mail: mike.looijmans@topicproducts.com
Website: www.topicproducts.com

Please consider the environment before printing this e-mail

Visit us at : Aerospace Electrical Systems Expo Europe which will be held from 17.11.2015 till 19.11.2015, Findorffstrasse 101 Bremen, Germany, Hall 5, stand number C65
http://www.aesexpo.eu