bitbake vs incremental builds

bitbake vs incremental builds

[Rich Pixley]

I'm wondering if there's any reason that bitbake isn't using the time 
stamps on it's stamp files to determine which recipe+task pairs should 
be executed.

If C depends on B depends on A...

A first build will build A, B, C.
A second build will build nothing since nothing has changed.  (this is 
correct behavior).
However, if I change A and rebuild, only A will be rebuilt, not B or C.

This is a nuisance for developers who are using bitbake as a build 
system.  It's a problem for continuous build systems since they must 
necessarily build everything over from source in order to test whether 
builds work.

I'm thinking that it should be fairly easy to adjust which recipe+task 
pairs are scheduled for execution by looking not only at stamp file 
existence, but also at the comparative difference in stamp file times.  
If the stamp file for A is more recent than the stamp file for B, then B 
needs to be scheduled for execution as does C.

Does anyone know of any reason why this cannot or should not be done?

--rich

Re: bitbake vs incremental builds

[Richard Purdie]

On Wed, 2008-06-18 at 15:40 -0700, Rich Pixley wrote:
> If C depends on B depends on A...
> 
> A first build will build A, B, C.
> A second build will build nothing since nothing has changed.  (this is 
> correct behavior).
> However, if I change A and rebuild, only A will be rebuilt, not B or C.
> 
> This is a nuisance for developers who are using bitbake as a build 
> system.  It's a problem for continuous build systems since they must 
> necessarily build everything over from source in order to test whether 
> builds work.
> 
> I'm thinking that it should be fairly easy to adjust which recipe+task 
> pairs are scheduled for execution by looking not only at stamp file 
> existence, but also at the comparative difference in stamp file times.  
> If the stamp file for A is more recent than the stamp file for B, then B 
> needs to be scheduled for execution as does C.
> 
> Does anyone know of any reason why this cannot or should not be done?

It wasn't done for a long time simply because nobody had taken the time
to implement it. It has however recently been implemented, simply set:

BB_STAMP_POLICY = "full" 
or
BB_STAMP_POLICY = "whitelist" 

Whitelist allows you to exempt certain packages from stamp checking, see
packaged-staging.bbclass for an example.

Cheers,

Richard

Re: bitbake vs incremental builds

[K. Richard Pixley]

Richard Purdie wrote:
> It wasn't done for a long time simply because nobody had taken the time
> to implement it. It has however recently been implemented, simply set:
>
> BB_STAMP_POLICY = "full"
> or
> BB_STAMP_POLICY = "whitelist"
>
> Whitelist allows you to exempt certain packages from stamp checking, see
> packaged-staging.bbclass for an example.
Thanks.

I'm not seeing any references to BB_STAMP_POLICY in bitbake-1.8.10.  Am 
I looking in the wrong place?

--rich

Re: bitbake vs incremental builds

[Cliff Brake]

On Thu, Jun 19, 2008 at 12:57 PM, K. Richard Pixley
<rich.pixley@palm.com> wrote:

> I'm not seeing any references to BB_STAMP_POLICY in bitbake-1.8.10.  Am I
> looking in the wrong place?

Its in the unreleased HEAD of the bitbake-1.8 branch.  You'll need to
get it from SVN.

Cliff

-- 
=======================
Cliff Brake
http://bec-systems.com

Re: bitbake vs incremental builds

[Rich Pixley]

Richard Purdie wrote:
> It wasn't done for a long time simply because nobody had taken the time
> to implement it. It has however recently been implemented, simply set:
>
> BB_STAMP_POLICY = "full"
> or
> BB_STAMP_POLICY = "whitelist"
>
> Whitelist allows you to exempt certain packages from stamp checking, see
> packaged-staging.bbclass for an example.
This isn't working for me simply by taking a top of tree bitbake and 
shoving it into my environment.  However, my OE environment is over a 
year old and is moderately hacked locally.

I tried to write a trivial test case, but I got bogged down.  It's not 
at all clear to me what one would need to do to use bitbake in some 
environment other than OE.  I was hoping it would be simple to just 
write an empty conf/bitbake.conf and a couple of a.bb and b.bb files 
with a depending on b, but 2hrs later I'm still confused.

So this isn't working for me and I don't see an easy way to debug it.  
Any suggestions?

--rich

Re: bitbake vs incremental builds

[Richard Purdie]

On Thu, 2008-06-19 at 18:03 -0700, Rich Pixley wrote:
> Richard Purdie wrote:
> > It wasn't done for a long time simply because nobody had taken the time
> > to implement it. It has however recently been implemented, simply set:
> >
> > BB_STAMP_POLICY = "full"
> > or
> > BB_STAMP_POLICY = "whitelist"
> >
> > Whitelist allows you to exempt certain packages from stamp checking, see
> > packaged-staging.bbclass for an example.
> This isn't working for me simply by taking a top of tree bitbake and 
> shoving it into my environment.  However, my OE environment is over a 
> year old and is moderately hacked locally.

You used bitbake 1.8 svn, not bitbake trunk? It should be near enough a
drop in replacement although there are possibly some tweaks needed, I
can't remember offhand whats changed in the past year.

> I tried to write a trivial test case, but I got bogged down.  It's not 
> at all clear to me what one would need to do to use bitbake in some 
> environment other than OE.  I was hoping it would be simple to just 
> write an empty conf/bitbake.conf and a couple of a.bb and b.bb files 
> with a depending on b, but 2hrs later I'm still confused.
> 
> So this isn't working for me and I don't see an easy way to debug it.  
> Any suggestions?

This is pretty much impossible to help with unless you can share more
information. Using bitbake outside of OE is perfectly possible, you do
just need a bitbake.conf and a base.bbclass. When we're making changes
to bitbake we do mainly care about OE since OE is the major public user.
If there were others we'd take them into consideration too. OE.dev does
work with bitbake 1.8 svn and many people are using that.

If you can show a more specific problem perhaps we can help but at the
moment you're tying our hands :(.

Cheers,

Richard

Re: bitbake vs incremental builds

[Rich Pixley]

Richard Purdie wrote:
> If you can show a more specific problem perhaps we can help but at the
> moment you're tying our hands :(.
And unfortunately, it must stay that way for a while longer.

I'll try to get myself up to speed on using top of tree OE this week.  
That portion of our project was before my time so it'll be a learning curve.

--rich

Re: bitbake vs incremental builds

[Rich Pixley]

Looks like I got something wrong or got confused by the stamps earlier.  
I am seeing top-of-tree org.embedded.stable build more things after 
changes with BB_STAMP_POLICY = "full" than it was previously.

I'm also seeing that behavior in our environment now, although I'm not 
convinced we're doing all that people expect yet.  I'm planning to 
switch us over tomorrow morning.  We'll whether anything hits the fan then.

For this to be accurate, we need to eliminate all cases of "NOSTAMP", 
don't we?  Isn't the only reason to use NOSTAMP to force something to be 
redone every pass?

--rich

rebuild confusion

[Rich Pixley]

Previously, (ie, before I took the top of tree bitbake), I expected 
"bitbake -c rebuild foo" to be akin to "bitbake -c clean foo && bitbake 
foo".

But I'm seeing different results now.  What am I missing?

--rich

Eg, on apmd with:

MACHINE = "qemux86"
DISTRO = "angstrom-2007.1"

I see:

rich@duck.palm1.palmone.com> time bitbake -c rebuild apmd
NOTE: Handling BitBake files: \ (4969/4969) [100 %]
NOTE: Parsing finished. 4749 cached, 0 parsed, 220 skipped, 0 masked.
NOTE: Cache is clean, not saving.
NOTE: build 200806261350: started

OE Build Configuration:
BB_VERSION     = "1.8.11"
OE_REVISION    = "7214a86e8f520136c9e40ffd988ac8d19c92b96c"
TARGET_ARCH    = "i586"
TARGET_OS      = "linux"
MACHINE        = "qemux86"
DISTRO         = "angstrom"
DISTRO_VERSION = "2007.12-r18"
TARGET_FPU     = ""

NOTE: Resolving any missing task queue dependencies
NOTE: Preparing runqueue
NOTE: Executing runqueue
NOTE: Running task 1 of 1 (ID: 0, 
/home/rich/projects/oe/org.openembedded.stable/packages/apmd/apmd_3.2.2.bb, 
do_rebuild)
NOTE: package apmd-3.2.2: started
NOTE: package apmd-3.2.2-r8: task do_rebuild: started
NOTE: package apmd-3.2.2-r8: task do_clean: started
NOTE: removing 
/home/rich/projects/oe/org.openembedded.stable/tmp/work/i586-angstrom-linux/apmd-3.2.2-r8
NOTE: removing 
/home/rich/projects/oe/org.openembedded.stable/tmp/stamps/i586-angstrom-linux/apmd-3.2.2-r8.*
NOTE: package apmd-3.2.2-r8: task do_clean: completed
NOTE: package apmd-3.2.2-r8: task do_build: started
NOTE: package apmd-3.2.2-r8: task do_build: completed
NOTE: package apmd-3.2.2-r8: task do_rebuild: completed
NOTE: package apmd-3.2.2: completed
NOTE: Tasks Summary: Attempted 1 tasks of which 0 didn't need to be 
rerun and 0 failed.
NOTE: build 200806261350: completed

real    0m2.928s
user    0m2.620s
sys     0m0.296s

compared to...

rich@duck.palm1.palmone.com> time bitbake -c clean apmd && time bitbake apmd
NOTE: Handling BitBake files: \ (4969/4969) [100 %]
NOTE: Parsing finished. 4749 cached, 0 parsed, 220 skipped, 0 masked.
NOTE: Cache is clean, not saving.
NOTE: build 200806261351: started

OE Build Configuration:
BB_VERSION     = "1.8.11"
OE_REVISION    = "7214a86e8f520136c9e40ffd988ac8d19c92b96c"
TARGET_ARCH    = "i586"
TARGET_OS      = "linux"
MACHINE        = "qemux86"
DISTRO         = "angstrom"
DISTRO_VERSION = "2007.12-r18"
TARGET_FPU     = ""

NOTE: Resolving any missing task queue dependencies
NOTE: Preparing runqueue
NOTE: Executing runqueue
NOTE: Running task 1 of 1 (ID: 0, 
/home/rich/projects/oe/org.openembedded.stable/packages/apmd/apmd_3.2.2.bb, 
do_clean)
NOTE: package apmd-3.2.2: started
NOTE: package apmd-3.2.2-r8: task do_clean: started
NOTE: removing 
/home/rich/projects/oe/org.openembedded.stable/tmp/work/i586-angstrom-linux/apmd-3.2.2-r8
NOTE: removing 
/home/rich/projects/oe/org.openembedded.stable/tmp/stamps/i586-angstrom-linux/apmd-3.2.2-r8.*
NOTE: package apmd-3.2.2-r8: task do_clean: completed
NOTE: package apmd-3.2.2: completed
NOTE: Tasks Summary: Attempted 1 tasks of which 0 didn't need to be 
rerun and 0 failed.
NOTE: build 200806261351: completed

real    0m2.902s
user    0m2.596s
sys     0m0.296s
NOTE: Handling BitBake files: \ (4969/4969) [100 %]
NOTE: Parsing finished. 4749 cached, 0 parsed, 220 skipped, 0 masked.
NOTE: Cache is clean, not saving.
NOTE: build 200806261351: started

OE Build Configuration:
BB_VERSION     = "1.8.11"
OE_REVISION    = "7214a86e8f520136c9e40ffd988ac8d19c92b96c"
TARGET_ARCH    = "i586"
TARGET_OS      = "linux"
MACHINE        = "qemux86"
DISTRO         = "angstrom"
DISTRO_VERSION = "2007.12-r18"
TARGET_FPU     = ""

NOTE: Resolving any missing task queue dependencies
NOTE: Preparing runqueue
NOTE: Executing runqueue
NOTE: Running task 241 of 288 (ID: 0, 
/home/rich/projects/oe/org.openembedded.stable/packages/apmd/apmd_3.2.2.bb, 
do_fetch)
NOTE: package apmd-3.2.2: started
NOTE: package apmd-3.2.2-r8: task do_fetch: started
NOTE: package apmd-3.2.2-r8: task do_fetch: completed
NOTE: package apmd-3.2.2: completed
NOTE: Running task 242 of 288 (ID: 2, 
/home/rich/projects/oe/org.openembedded.stable/packages/apmd/apmd_3.2.2.bb, 
do_unpack)
NOTE: package apmd-3.2.2: started
NOTE: package apmd-3.2.2-r8: task do_unpack: started
NOTE: Unpacking /home/rich/projects/oe/sources/apmd_3.2.2.orig.tar.gz to 
/home/rich/projects/oe/org.openembedded.stable/tmp/work/i586-angstrom-linux/apmd-3.2.2-r8/
NOTE: Unpacking 
/home/rich/projects/oe/org.openembedded.stable/packages/apmd/apmd-3.2.2/init 
to 
/home/rich/projects/oe/org.openembedded.stable/tmp/work/i586-angstrom-linux/apmd-3.2.2-r8/
NOTE: Unpacking 
/home/rich/projects/oe/org.openembedded.stable/packages/apmd/apmd-3.2.2/default 
to 
/home/rich/projects/oe/org.openembedded.stable/tmp/work/i586-angstrom-linux/apmd-3.2.2-r8/
NOTE: Unpacking 
/home/rich/projects/oe/org.openembedded.stable/packages/apmd/apmd-3.2.2/apmd_proxy 
to 
/home/rich/projects/oe/org.openembedded.stable/tmp/work/i586-angstrom-linux/apmd-3.2.2-r8/
NOTE: Unpacking 
/home/rich/projects/oe/org.openembedded.stable/packages/apmd/apmd-3.2.2/apmd_proxy.conf 
to 
/home/rich/projects/oe/org.openembedded.stable/tmp/work/i586-angstrom-linux/apmd-3.2.2-r8/
NOTE: package apmd-3.2.2-r8: task do_unpack: completed
NOTE: package apmd-3.2.2: completed
NOTE: Running task 256 of 288 (ID: 3, 
/home/rich/projects/oe/org.openembedded.stable/packages/apmd/apmd_3.2.2.bb, 
do_patch)
NOTE: package apmd-3.2.2: started
NOTE: package apmd-3.2.2-r8: task do_patch: started
NOTE: Applying patch 'debian.patch' 
(/home/rich/projects/oe/org.openembedded.stable/packages/apmd/apmd-3.2.2/debian.patch)
NOTE: Applying patch 'workaround.patch' 
(/home/rich/projects/oe/org.openembedded.stable/packages/apmd/apmd-3.2.2/workaround.patch)
NOTE: Applying patch 'zaurus24.patch' 
(/home/rich/projects/oe/org.openembedded.stable/packages/apmd/apmd-3.2.2/zaurus24.patch)
NOTE: Applying patch 'unlinux.patch' 
(/home/rich/projects/oe/org.openembedded.stable/packages/apmd/apmd-3.2.2/unlinux.patch)
NOTE: package apmd-3.2.2-r8: task do_patch: completed
NOTE: package apmd-3.2.2: completed
NOTE: Running task 257 of 288 (ID: 4, 
/home/rich/projects/oe/org.openembedded.stable/packages/apmd/apmd_3.2.2.bb, 
do_configure)
NOTE: package apmd-3.2.2: started
NOTE: package apmd-3.2.2-r8: task do_configure: started
NOTE: package apmd-3.2.2-r8: task do_configure: completed
NOTE: package apmd-3.2.2: completed
NOTE: Running task 262 of 288 (ID: 5, 
/home/rich/projects/oe/org.openembedded.stable/packages/apmd/apmd_3.2.2.bb, 
do_qa_configure)
NOTE: package apmd-3.2.2: started
NOTE: package apmd-3.2.2-r8: task do_qa_configure: started
NOTE: Checking sanity of the config.log file
NOTE: package apmd-3.2.2-r8: task do_qa_configure: completed
NOTE: package apmd-3.2.2: completed
NOTE: Running task 263 of 288 (ID: 6, 
/home/rich/projects/oe/org.openembedded.stable/packages/apmd/apmd_3.2.2.bb, 
do_compile)
NOTE: package apmd-3.2.2: started
NOTE: package apmd-3.2.2-r8: task do_compile: started
NOTE: package apmd-3.2.2-r8: task do_compile: completed
NOTE: package apmd-3.2.2: completed
NOTE: Running task 264 of 288 (ID: 7, 
/home/rich/projects/oe/org.openembedded.stable/packages/apmd/apmd_3.2.2.bb, 
do_install)
NOTE: package apmd-3.2.2: started
NOTE: package apmd-3.2.2-r8: task do_install: started
NOTE: package apmd-3.2.2-r8: task do_install: completed
NOTE: package apmd-3.2.2: completed
NOTE: Running task 271 of 288 (ID: 11, 
/home/rich/projects/oe/org.openembedded.stable/packages/apmd/apmd_3.2.2.bb, 
do_package)
NOTE: package apmd-3.2.2: started
NOTE: package apmd-3.2.2-r8: task do_package: started
NOTE: DO PACKAGE QA
NOTE: Checking Package: libapm
NOTE: Checking Package: libapm-dev
NOTE: Checking Package: apm
NOTE: Checking Package: apmd-dbg
NOTE: Checking Package: apmd
NOTE: Checking Package: apmd-doc
NOTE: Checking Package: apmd-dev
NOTE: Checking Package: apmd-locale
NOTE: DONE with PACKAGE QA
NOTE: package apmd-3.2.2-r8: task do_package: completed
NOTE: package apmd-3.2.2: completed
NOTE: Running task 276 of 288 (ID: 8, 
/home/rich/projects/oe/org.openembedded.stable/packages/apmd/apmd_3.2.2.bb, 
do_populate_staging)
NOTE: package apmd-3.2.2: started
NOTE: package apmd-3.2.2-r8: task do_populate_staging: started
NOTE: package apmd-3.2.2-r8: task do_populate_staging: completed
NOTE: package apmd-3.2.2: completed
NOTE: Running task 285 of 288 (ID: 12, 
/home/rich/projects/oe/org.openembedded.stable/packages/apmd/apmd_3.2.2.bb, 
do_package_write_ipk)
NOTE: package apmd-3.2.2: started
NOTE: package apmd-3.2.2-r8: task do_package_write_ipk: started
Packaged contents of libapm into 
/home/rich/projects/oe/org.openembedded.stable/tmp/deploy/glibc/ipk/i586/libapm1_3.2.2-r8_i586.ipk
Packaged contents of libapm-dev into 
/home/rich/projects/oe/org.openembedded.stable/tmp/deploy/glibc/ipk/i586/libapm-dev_3.2.2-r8_i586.ipk
Packaged contents of apm into 
/home/rich/projects/oe/org.openembedded.stable/tmp/deploy/glibc/ipk/i586/apm_3.2.2-r8_i586.ipk
Packaged contents of apmd-dbg into 
/home/rich/projects/oe/org.openembedded.stable/tmp/deploy/glibc/ipk/i586/apmd-dbg_3.2.2-r8_i586.ipk
Packaged contents of apmd into 
/home/rich/projects/oe/org.openembedded.stable/tmp/deploy/glibc/ipk/i586/apmd_3.2.2-r8_i586.ipk
NOTE: Not creating empty archive for apmd-doc-3.2.2-r8
Packaged contents of apmd-dev into 
/home/rich/projects/oe/org.openembedded.stable/tmp/deploy/glibc/ipk/i586/apmd-dev_3.2.2-r8_i586.ipk
NOTE: Not creating empty archive for apmd-locale-3.2.2-r8
NOTE: package apmd-3.2.2-r8: task do_package_write_ipk: completed
NOTE: package apmd-3.2.2: completed
NOTE: Running task 286 of 288 (ID: 10, 
/home/rich/projects/oe/org.openembedded.stable/packages/apmd/apmd_3.2.2.bb, 
do_package_write)
NOTE: package apmd-3.2.2: started
NOTE: package apmd-3.2.2-r8: task do_package_write: started
NOTE: package apmd-3.2.2-r8: task do_package_write: completed
NOTE: package apmd-3.2.2: completed
NOTE: Running task 287 of 288 (ID: 9, 
/home/rich/projects/oe/org.openembedded.stable/packages/apmd/apmd_3.2.2.bb, 
do_qa_staging)
NOTE: package apmd-3.2.2: started
NOTE: package apmd-3.2.2-r8: task do_qa_staging: started
NOTE: QA checking staging
ERROR: QA Issue: libmudflap.la failed sanity test (installed) in path 
/home/rich/projects/oe/org.openembedded.stable/tmp/staging/i586-angstrom-linux/lib
NOTE: a QA error occured but will not be logged because QA_LOG is not set
ERROR: QA Issue: libstdc++.la failed sanity test (installed) in path 
/home/rich/projects/oe/org.openembedded.stable/tmp/staging/i586-angstrom-linux/lib
NOTE: a QA error occured but will not be logged because QA_LOG is not set
ERROR: QA Issue: libmudflapth.la failed sanity test (installed) in path 
/home/rich/projects/oe/org.openembedded.stable/tmp/staging/i586-angstrom-linux/lib
NOTE: a QA error occured but will not be logged because QA_LOG is not set
ERROR: QA Issue: libsupc++.la failed sanity test (installed) in path 
/home/rich/projects/oe/org.openembedded.stable/tmp/staging/i586-angstrom-linux/lib
NOTE: a QA error occured but will not be logged because QA_LOG is not set
NOTE: package apmd-3.2.2-r8: task do_qa_staging: completed
NOTE: package apmd-3.2.2: completed
NOTE: Running task 288 of 288 (ID: 1, 
/home/rich/projects/oe/org.openembedded.stable/packages/apmd/apmd_3.2.2.bb, 
do_build)
NOTE: package apmd-3.2.2: started
NOTE: package apmd-3.2.2-r8: task do_build: started
NOTE: package apmd-3.2.2-r8: task do_build: completed
NOTE: package apmd-3.2.2: completed
NOTE: Tasks Summary: Attempted 288 tasks of which 275 didn't need to be 
rerun and 0 failed.
NOTE: build 200806261351: completed

real    0m8.932s
user    0m6.476s
sys     0m2.396s