Revisiting bitbake stamp handling

Revisiting bitbake stamp handling

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

For a number of features in OE (e.g packaged-staging, rm_work) a change to bitbake stamp
handling would make things a lot easier. As I understand it bitbake currently does for
'bitbake foo -c compile':

if(dependency stamps are missing OR depency stamps have newer mtime) then start from scratch

For rm_work we want to remove the do_unpack upto do_install stamps, but leave do_fetch and
do_package_write* and for packaged-staging we want to only create a do_populate_staging
stamps. So 'bitbake foo -c compile' would only do:

if(depency stamps have newer mtime) then start from scratch

this has implications for do_clean and do_rebuild, and possibly other things. Since this
looks to good to be true, I want to know what else might break and how we could solve that
when implementing this. This just a gedankenexperiment for now, so bitbake hackers can
relax now :)

regards,

Koen

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFG8Ad/MkyGM64RGpERArW7AKCY91yVx8se1jUej3FiZyVRTpuZVgCghoca
n9KyhcAJ4a4VhBvxh360wto=
=152E
-----END PGP SIGNATURE-----

Re: Revisiting bitbake stamp handling

[Richard Purdie]

On Tue, 2007-09-18 at 19:14 +0200, Koen Kooi wrote:
> For a number of features in OE (e.g packaged-staging, rm_work) a change to bitbake stamp
> handling would make things a lot easier. As I understand it bitbake currently does for
> 'bitbake foo -c compile':
> 
> if(dependency stamps are missing OR depency stamps have newer mtime) then start from scratch
> 
> For rm_work we want to remove the do_unpack upto do_install stamps, but leave do_fetch and
> do_package_write* and for packaged-staging we want to only create a do_populate_staging
> stamps. So 'bitbake foo -c compile' would only do:
> 
> if(depency stamps have newer mtime) then start from scratch
> 
> this has implications for do_clean and do_rebuild, and possibly other things. Since this
> looks to good to be true, I want to know what else might break and how we could solve that
> when implementing this. This just a gedankenexperiment for now, so bitbake hackers can
> relax now :)

When you type bitbake strace, bitbake splits this into queue of tasks.
This might be for example:

1. strace.do_fetch
2. strace.do_unpack
3. strace.do_patch
4. strace.do_configure
5. strace.do_compile
6. etc.

Say you "bitbake strace -c compile":

* Bitbake looks at step 1. Does it need to build it? If the stamp 
  doesn't exist, yes it does. If the stamp exists, it checks dependency 
  stamps (there are none).

* Bitbake looks at step 2. Does it need to build it? If the stamp 
  doesn't exist, yes it does. If the stamp exists, it checks dependency 
  stamps.

* etc.

The logic you propose above simply breaks this and its fundamental to
the way bitbake currently works.

I guess what you propose is bitbake does this backwards. Assuming you
"bitbake strace -c compile":

* Look at step 5. Does it need to build it? If the stamp exists, and 
  any present dependency stamps are valid and older [repeat for all
present stamps], no, else yes.

* No further processing

This needs two changes:

  a) stamps are checked before we start processing the queue
  b) stamps are checked on a global basis (cross PN)

both of which are fairly major changes.

The breakage:

Say you're using rm_work and want to rebuild strace for the image. You
run "bitbake strace -c compile -f". A subsequent "bitbake someimage"
will not cause strace to install/package/stage or get included in the
image. Broken and one of the things you wanted to fix!

Say you want to trigger a reinstall of all packages like the recent
pkgmaps change. How do you do it? In the case without rm_work you could
touch all a dependent task like compile to trigger the install (ugly),
or remove *every* install task stamp and *every* stamp which depends on
install (near impossible). With rm_work, you're screwed basically.

So I really think this is the wrong tree to bark up...

Regards,

Richard

Re: Revisiting bitbake stamp handling

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Richard Purdie schreef:
> On Tue, 2007-09-18 at 19:14 +0200, Koen Kooi wrote:
>> For a number of features in OE (e.g packaged-staging, rm_work) a change to bitbake stamp
>> handling would make things a lot easier. As I understand it bitbake currently does for
>> 'bitbake foo -c compile':
>>
>> if(dependency stamps are missing OR depency stamps have newer mtime) then start from scratch
>>
>> For rm_work we want to remove the do_unpack upto do_install stamps, but leave do_fetch and
>> do_package_write* and for packaged-staging we want to only create a do_populate_staging
>> stamps. So 'bitbake foo -c compile' would only do:
>>
>> if(depency stamps have newer mtime) then start from scratch
>>
>> this has implications for do_clean and do_rebuild, and possibly other things. Since this
>> looks to good to be true, I want to know what else might break and how we could solve that
>> when implementing this. This just a gedankenexperiment for now, so bitbake hackers can
>> relax now :)
> 
> When you type bitbake strace, bitbake splits this into queue of tasks.
> This might be for example:
> 
> 1. strace.do_fetch
> 2. strace.do_unpack
> 3. strace.do_patch
> 4. strace.do_configure
> 5. strace.do_compile
> 6. etc.
> 
> Say you "bitbake strace -c compile":
> 
> * Bitbake looks at step 1. Does it need to build it? If the stamp 
>   doesn't exist, yes it does. If the stamp exists, it checks dependency 
>   stamps (there are none).
> 
> * Bitbake looks at step 2. Does it need to build it? If the stamp 
>   doesn't exist, yes it does. If the stamp exists, it checks dependency 
>   stamps.
> 
> * etc.
> 
> The logic you propose above simply breaks this and its fundamental to
> the way bitbake currently works.
> 
> I guess what you propose is bitbake does this backwards. Assuming you
> "bitbake strace -c compile":
> 
> * Look at step 5. Does it need to build it? If the stamp exists, and 
>   any present dependency stamps are valid and older [repeat for all
> present stamps], no, else yes.
> 
> * No further processing
> 
> This needs two changes:
> 
>   a) stamps are checked before we start processing the queue
>   b) stamps are checked on a global basis (cross PN)
> 
> both of which are fairly major changes.
> 
> The breakage:
> 
> Say you're using rm_work and want to rebuild strace for the image. You
> run "bitbake strace -c compile -f". A subsequent "bitbake someimage"
> will not cause strace to install/package/stage or get included in the
> image. Broken and one of the things you wanted to fix!

Actually, that would work, since -c compile -f would update the do_compile stamp, so its
mtime is newer than the do_rootfs.


> Say you want to trigger a reinstall of all packages like the recent
> pkgmaps change. How do you do it? In the case without rm_work you could
> touch all a dependent task like compile to trigger the install (ugly),
> or remove *every* install task stamp and *every* stamp which depends on
> install (near impossible). With rm_work, you're screwed basically.

Which highlights exactly why the current situation is broken. In this case the user has to
discover he needs to run install again (which a lot of them don't, looking at IRC and the
ml), so what's stopping us from adding an do_installall task that does exactly that? Or
even requiring a rebuild, since it needs the user to find out *and* the user to take action.

regards,

Koen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFG8JetMkyGM64RGpERAnjNAJ43AqthGee++19Ovs5PLsFEg6howgCeI/xL
eIzIqoYzRvGUbPLcrzWNlKY=
=s3+J
-----END PGP SIGNATURE-----

Re: Revisiting bitbake stamp handling

[Richard Purdie]

On Wed, 2007-09-19 at 05:29 +0200, Koen Kooi wrote:
> > Say you're using rm_work and want to rebuild strace for the image. You
> > run "bitbake strace -c compile -f". A subsequent "bitbake someimage"
> > will not cause strace to install/package/stage or get included in the
> > image. Broken and one of the things you wanted to fix!
> 
> Actually, that would work, since -c compile -f would update the do_compile stamp, so its
> mtime is newer than the do_rootfs.

do_rootfs is stampless. 

> > Say you want to trigger a reinstall of all packages like the recent
> > pkgmaps change. How do you do it? In the case without rm_work you could
> > touch all a dependent task like compile to trigger the install (ugly),
> > or remove *every* install task stamp and *every* stamp which depends on
> > install (near impossible). With rm_work, you're screwed basically.
> 
> Which highlights exactly why the current situation is broken. In this case the user has to
> discover he needs to run install again (which a lot of them don't, looking at IRC and the
> ml), so what's stopping us from adding an do_installall task that does exactly that? Or
> even requiring a rebuild, since it needs the user to find out *and* the user to take action.

I don't see how a communication issue is related to the need to add a
do_installall task?

Cheers,

Richard

Re: Revisiting bitbake stamp handling

[pHilipp Zabel]

On 9/19/07, Richard Purdie <rpurdie@rpsys.net> wrote:
> On Tue, 2007-09-18 at 19:14 +0200, Koen Kooi wrote:
> > For a number of features in OE (e.g packaged-staging, rm_work) a change to bitbake stamp
> > handling would make things a lot easier. As I understand it bitbake currently does for
> > 'bitbake foo -c compile':
> >
> > if(dependency stamps are missing OR depency stamps have newer mtime) then start from scratch
> >
> > For rm_work we want to remove the do_unpack upto do_install stamps, but leave do_fetch and
> > do_package_write* and for packaged-staging we want to only create a do_populate_staging
> > stamps. So 'bitbake foo -c compile' would only do:
> >
> > if(depency stamps have newer mtime) then start from scratch
> >
> > this has implications for do_clean and do_rebuild, and possibly other things. Since this
> > looks to good to be true, I want to know what else might break and how we could solve that
> > when implementing this. This just a gedankenexperiment for now, so bitbake hackers can
> > relax now :)
>
> When you type bitbake strace, bitbake splits this into queue of tasks.
> This might be for example:
>
> 1. strace.do_fetch
> 2. strace.do_unpack
> 3. strace.do_patch
> 4. strace.do_configure
> 5. strace.do_compile
> 6. etc.
>
> Say you "bitbake strace -c compile":
>
> * Bitbake looks at step 1. Does it need to build it? If the stamp
>   doesn't exist, yes it does. If the stamp exists, it checks dependency
>   stamps (there are none).
>
> * Bitbake looks at step 2. Does it need to build it? If the stamp
>   doesn't exist, yes it does. If the stamp exists, it checks dependency
>   stamps.
>
> * etc.
>
> The logic you propose above simply breaks this and its fundamental to
> the way bitbake currently works.
>
> I guess what you propose is bitbake does this backwards. Assuming you
> "bitbake strace -c compile":
>
> * Look at step 5. Does it need to build it? If the stamp exists, and
>   any present dependency stamps are valid and older [repeat for all
> present stamps], no, else yes.
>
> * No further processing
>
> This needs two changes:
>
>   a) stamps are checked before we start processing the queue
>   b) stamps are checked on a global basis (cross PN)
>
> both of which are fairly major changes.

So is it worth it? Possible to do at all? It would enable us to have
missing dependency stamps if that more accurately reflects the state
of work/staging dirs.
I certainly find the idea of walking down from the last step much more
elegant than always starting at step 1. OTOH I have no idea how much
work it would be to implement this and what other problems this might
bring other than the examples you already gave.

> The breakage:
>
> Say you're using rm_work and want to rebuild strace for the image. You
> run "bitbake strace -c compile -f". A subsequent "bitbake someimage"
> will not cause strace to install/package/stage or get included in the
> image. Broken and one of the things you wanted to fix!

For stampless tasks like rootfs we should check the stamps of all its
dependencies
(task-something, depends on strace:do_package_write, which is not
current if a newer do_compile stamp is found). I don't see the problem
here.

> Say you want to trigger a reinstall of all packages like the recent
> pkgmaps change. How do you do it? In the case without rm_work you could
> touch all a dependent task like compile to trigger the install (ugly),
> or remove *every* install task stamp and *every* stamp which depends on
> install (near impossible). With rm_work, you're screwed basically.

That's right, with rm_work you're screwed either way, you have to rebuild.

> So I really think this is the wrong tree to bark up...

But I'm not sure using stamps not only as indicators of work/staging
state, but also as a mechanism to manipulate bitbake behaviour is the
right way either. Wouldn't it be better to have a bitbake option that
just skips task whose dependencies are not met and do
bitbake -cinstall -f world --no-task-dependencies to force a reinstall
of all compiled packages?

regards
Philipp