Re: [PATCH v4 14/18] runqueue.py, build.py: Invalidate setscene tasks based on do_unpack stamps

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Fri, 2012-06-08 at 08:41 -0500, Jason Wessel wrote:
> If you have a fully populated sstate cache and have used it to
> execute a build, it is not possible to invalidate repackage
> an intermediate build after you have forced a compiled
> 
> Example when you have build from a complete sstate cache build:
>   bitbake core-image-sato
>   bitbake -c patch acl
>      *** Make some changes to the C files for experimentation.
>   bitbake -f -c compile acl
>   bitbake acl
> 
> The bitbake will refuse to build the acl package at this
> point and instead keep populating it from the sstate.  Using
> the cleanstate is no longer a good option because it will
> also erase the scratch area.
> 
> Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
> ---
>  lib/bb/build.py    |    8 ++++++++
>  lib/bb/runqueue.py |   14 ++++++++++++++
>  2 files changed, 22 insertions(+)
> 
> diff --git a/lib/bb/build.py b/lib/bb/build.py
> index fb61b00..18c28aa 100644
> --- a/lib/bb/build.py
> +++ b/lib/bb/build.py
> @@ -463,6 +463,14 @@ def del_stamp(task, d, file_name = None):
>      stamp = stamp_internal(task, d, file_name)
>      bb.utils.remove(stamp)
>  
> +def exists_stamp(task, d, file_name = None):
> +    """
> +    Removes a stamp for a given task
> +    (d can be a data dict or dataCache)
> +    """
> +    stamp = stamp_internal(task, d, file_name)
> +    return os.path.exists(stamp)
> +
>  def stampfile(taskname, d, file_name = None):
>      """
>      Return the stamp for a given task
> diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
> index da3fdf9..6c802af 100644
> --- a/lib/bb/runqueue.py
> +++ b/lib/bb/runqueue.py
> @@ -718,6 +718,20 @@ class RunQueueData:
>                      for dep in self.runq_depends[task]:
>                          procdep.append(self.taskData.fn_index[self.runq_fnid[dep]] + "." + self.runq_task[dep])
>                      self.runq_hash[task] = bb.parse.siggen.get_taskhash(self.taskData.fn_index[self.runq_fnid[task]], self.runq_task[task], procdep, self.dataCache)
> +        try:
> +            new_setscene = []
> +            for task in self.runq_setscene:
> +                try:
> +                    fn = self.taskData.fn_index[self.rq.rqdata.runq_fnid[task]]
> +                    if bb.build.exists_stamp("do_unpack", self.dataCache, fn):
> +                        logger.debug(2, 'Removing task %s from queue because do_unpack exists', task)
> +                    else:
> +                        new_setscene.append(task)
> +                except:
> +                    logger.debug(2, 'Failed do_unpack check for %s', task)
> +            self.runq_setscene = new_setscene
> +        except:
> +            logger.debug(2, 'Failed to update runq_setscene')

We've gone to quite a bit of trouble to keep knowledge of the specific
tasks out of bitbake. I've noticed a few of your patches are "blurring"
the separation between bitbake and the metadata.

Here, bitbake shouldn't have knowledge of "unpack" or why its special.
I'd also argue its not in fact special and there are probably other
"force" scenarios which would break with a similar problem even with
this patch. I agree there is a problem here but we need to fix it in a
more generic way. There is already a bug open on this kind of problem
(#2256).

Cheers,

Richard