Re: Revisiting bitbake stamp handling

[Richard Purdie <rpurdie@rpsys.net>]

On Tue, 2007-09-18 at 19:14 +0200, Koen Kooi wrote:
> For a number of features in OE (e.g packaged-staging, rm_work) a change to bitbake stamp
> handling would make things a lot easier. As I understand it bitbake currently does for
> 'bitbake foo -c compile':
> 
> if(dependency stamps are missing OR depency stamps have newer mtime) then start from scratch
> 
> For rm_work we want to remove the do_unpack upto do_install stamps, but leave do_fetch and
> do_package_write* and for packaged-staging we want to only create a do_populate_staging
> stamps. So 'bitbake foo -c compile' would only do:
> 
> if(depency stamps have newer mtime) then start from scratch
> 
> this has implications for do_clean and do_rebuild, and possibly other things. Since this
> looks to good to be true, I want to know what else might break and how we could solve that
> when implementing this. This just a gedankenexperiment for now, so bitbake hackers can
> relax now :)

When you type bitbake strace, bitbake splits this into queue of tasks.
This might be for example:

1. strace.do_fetch
2. strace.do_unpack
3. strace.do_patch
4. strace.do_configure
5. strace.do_compile
6. etc.

Say you "bitbake strace -c compile":

* Bitbake looks at step 1. Does it need to build it? If the stamp 
  doesn't exist, yes it does. If the stamp exists, it checks dependency 
  stamps (there are none).

* Bitbake looks at step 2. Does it need to build it? If the stamp 
  doesn't exist, yes it does. If the stamp exists, it checks dependency 
  stamps.

* etc.

The logic you propose above simply breaks this and its fundamental to
the way bitbake currently works.

I guess what you propose is bitbake does this backwards. Assuming you
"bitbake strace -c compile":

* Look at step 5. Does it need to build it? If the stamp exists, and 
  any present dependency stamps are valid and older [repeat for all
present stamps], no, else yes.

* No further processing

This needs two changes:

  a) stamps are checked before we start processing the queue
  b) stamps are checked on a global basis (cross PN)

both of which are fairly major changes.

The breakage:

Say you're using rm_work and want to rebuild strace for the image. You
run "bitbake strace -c compile -f". A subsequent "bitbake someimage"
will not cause strace to install/package/stage or get included in the
image. Broken and one of the things you wanted to fix!

Say you want to trigger a reinstall of all packages like the recent
pkgmaps change. How do you do it? In the case without rm_work you could
touch all a dependent task like compile to trigger the install (ugly),
or remove *every* install task stamp and *every* stamp which depends on
install (near impossible). With rm_work, you're screwed basically.

So I really think this is the wrong tree to bark up...

Regards,

Richard