Running BitBake multiple times without rechecking upstream AUTOREV versions

Running BitBake multiple times without rechecking upstream AUTOREV versions

[chris.laplante]

[-- Attachment #1: Type: text/plain, Size: 1280 bytes --]

Hi all,

I'm working on using Jenkins to host our Yocto build. One of the things that would be nice is to be able to do a "bitbake our-user-image", upload the artifacts to our network file storage, and then do a "bitbake our-user-image -c populate_sdk_ext". I'd like to do these separately so that developers are not waiting around for the eSDK to be generated if all they care about is the kernel, for example. My concern is that the second bitbake invocation could end up building different stuff if someone were to check in code in between when the two "bitbake"s are run. This is primarily a concern with recipes that use AUTOREV (as we do for development purposes).

Is there a way to essentially "freeze" the BitBake data store and re-use it across multiple bitbake invocations?

I had a few ideas:

1. Use buildhistory-collect-srcrevs to generate the SRCREV_pn- overrides necessary to lock the SRCREVs of all the packages.
2. Use BB_SERVER_TIMEOUT = "-1" to force BitBake to remain resident - when BitBake is resident is seems to not reparse recipes. As I understand it, the recipe parse stage is when upstreams are checked for new revisions. Not sure if this would work.
3. Somehow use the locked siggens.


Is anything like this possible?

Thanks,
Chris

[-- Attachment #2: Type: text/html, Size: 5261 bytes --]

Re: Running BitBake multiple times without rechecking upstream AUTOREV versions

[Richard Purdie]

On Fri, 2018-05-11 at 18:23 +0000, Chris Laplante via Openembedded-core 
wrote:
> Hi all,
>  
> I’m working on using Jenkins to host our Yocto build. One of the
> things that would be nice is to be able to do a “bitbake our-user-
> image”, upload the artifacts to our network file storage, and then do
> a “bitbake our-user-image -c populate_sdk_ext”. I’d like to do these
> separately so that developers are not waiting around for the eSDK to
> be generated if all they care about is the kernel, for example. My
> concern is that the second bitbake invocation could end up building
> different stuff if someone were to check in code in between when the
> two “bitbake”s are run. This is primarily a concern with recipes that
> use AUTOREV (as we do for development purposes).
>  
> Is there a way to essentially “freeze” the BitBake data store and re-
> use it across multiple bitbake invocations?

Have you tried BB_SRCREV_POLICY = "cache"?

Cheers,

Richard

Re: UNVERIFIED SENDER Re: Running BitBake multiple times without rechecking upstream AUTOREV versions

[chris.laplante]

Hi Richard,

That looks perfect, thanks! On a related note, it seems "bitbake --revisions-changed" is broken in Sumo:

ERROR: Command execution failed: Traceback (most recent call last):
  File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/command.py", line 116, in runAsyncCommand
    commandmethod(self.cmds_async, self, options)
  File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/command.py", line 723, in compareRevisions
    if bb.fetch.fetcher_compare_revisions(command.cooker.data):
TypeError: fetcher_compare_revisions() takes 0 positional arguments but 1 was given



Looks like commit 97617fd6755ffa25c058215ffb060cbb86240b44 dropped the 'd' argument from that method, but command.py was never updated to reflect that.

Shall I submit a patch to change the line to be "if bb.fetch.fetcher_compare_revisions():"  ?

Chris

-----Original Message-----
From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org] 
Sent: Monday, May 14, 2018 4:20 AM
To: LAPLANTE,CHRIS (A-Little Falls,ex1) <chris.laplante@agilent.com>; openembedded-core@lists.openembedded.org
Subject: UNVERIFIED SENDER Re: [OE-core] Running BitBake multiple times without rechecking upstream AUTOREV versions

On Fri, 2018-05-11 at 18:23 +0000, Chris Laplante via Openembedded-core
wrote:
> Hi all,
>  
> I’m working on using Jenkins to host our Yocto build. One of the 
> things that would be nice is to be able to do a “bitbake our-user- 
> image”, upload the artifacts to our network file storage, and then do 
> a “bitbake our-user-image -c populate_sdk_ext”. I’d like to do these 
> separately so that developers are not waiting around for the eSDK to 
> be generated if all they care about is the kernel, for example. My 
> concern is that the second bitbake invocation could end up building 
> different stuff if someone were to check in code in between when the 
> two “bitbake”s are run. This is primarily a concern with recipes that 
> use AUTOREV (as we do for development purposes).
>  
> Is there a way to essentially “freeze” the BitBake data store and re- 
> use it across multiple bitbake invocations?

Have you tried BB_SRCREV_POLICY = "cache"?

Cheers,

Richard

Re: UNVERIFIED SENDER Re: Running BitBake multiple times without rechecking upstream AUTOREV versions

[Richard Purdie]

On Mon, 2018-05-14 at 13:05 +0000, chris.laplante@agilent.com wrote:
> Hi Richard,
> 
> That looks perfect, thanks! On a related note, it seems "bitbake --
> revisions-changed" is broken in Sumo:
> 
> ERROR: Command execution failed: Traceback (most recent call last):
>   File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/command.py",
> line 116, in runAsyncCommand
>     commandmethod(self.cmds_async, self, options)
>   File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/command.py",
> line 723, in compareRevisions
>     if bb.fetch.fetcher_compare_revisions(command.cooker.data):
> TypeError: fetcher_compare_revisions() takes 0 positional arguments
> but 1 was given
> 
> 
> 
> Looks like commit 97617fd6755ffa25c058215ffb060cbb86240b44 dropped
> the 'd' argument from that method, but command.py was never updated
> to reflect that.
> 
> Shall I submit a patch to change the line to be "if
> bb.fetch.fetcher_compare_revisions():"  ?

A patch to the bitbake list would be great thanks!

Cheers,

Richard

Re: Running BitBake multiple times without rechecking upstream AUTOREV versions

[Andrea Galbusera]

[-- Attachment #1: Type: text/plain, Size: 1606 bytes --]

Il 14 mag 2018 10:20 AM, "Richard Purdie" <
richard.purdie@linuxfoundation.org> ha scritto:

On Fri, 2018-05-11 at 18:23 +0000, Chris Laplante via Openembedded-core
wrote:

> Hi all,
>
> I’m working on using Jenkins to host our Yocto build. One of the
> things that would be nice is to be able to do a “bitbake our-user-
> image”, upload the artifacts to our network file storage, and then do
> a “bitbake our-user-image -c populate_sdk_ext”. I’d like to do these
> separately so that developers are not waiting around for the eSDK to
> be generated if all they care about is the kernel, for example. My
> concern is that the second bitbake invocation could end up building
> different stuff if someone were to check in code in between when the
> two “bitbake”s are run. This is primarily a concern with recipes that
> use AUTOREV (as we do for development purposes).
>
> Is there a way to essentially “freeze” the BitBake data store and re-
> use it across multiple bitbake invocations?

Have you tried BB_SRCREV_POLICY = "cache"?


Out of curiosity and probably unrelated to the OP's issue with images/SDK
SRCREVs consistency issue.... Could setting this variable be of any help
even for building usable eSDKs when AUTOREV is around? There's an issue in
bugzilla (don't have id at hand) dealing with eSDK / AUTOREV
incompatibility...


Cheers,

Richard

-- 
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[-- Attachment #2: Type: text/html, Size: 2631 bytes --]

Re: Running BitBake multiple times without rechecking upstream AUTOREV versions

[Andrea Galbusera]

On Tue, May 15, 2018 at 5:43 PM, Andrea Galbusera <gizero@gmail.com> wrote:
>
>
> Il 14 mag 2018 10:20 AM, "Richard Purdie"
> <richard.purdie@linuxfoundation.org> ha scritto:
>
> On Fri, 2018-05-11 at 18:23 +0000, Chris Laplante via Openembedded-core
> wrote:
>
>> Hi all,
>>
>> I’m working on using Jenkins to host our Yocto build. One of the
>> things that would be nice is to be able to do a “bitbake our-user-
>> image”, upload the artifacts to our network file storage, and then do
>> a “bitbake our-user-image -c populate_sdk_ext”. I’d like to do these
>> separately so that developers are not waiting around for the eSDK to
>> be generated if all they care about is the kernel, for example. My
>> concern is that the second bitbake invocation could end up building
>> different stuff if someone were to check in code in between when the
>> two “bitbake”s are run. This is primarily a concern with recipes that
>> use AUTOREV (as we do for development purposes).
>>
>> Is there a way to essentially “freeze” the BitBake data store and re-
>> use it across multiple bitbake invocations?
>
> Have you tried BB_SRCREV_POLICY = "cache"?
>
>
> Out of curiosity and probably unrelated to the OP's issue with images/SDK
> SRCREVs consistency issue.... Could setting this variable be of any help
> even for building usable eSDKs when AUTOREV is around? There's an issue in
> bugzilla (don't have id at hand) dealing with eSDK / AUTOREV
> incompatibility...

Here is the link to the above mentioned bug:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=11350