Re: [PATCH v4 14/18] runqueue.py, build.py: Invalidate setscene tasks based on do_unpack stamps

[Jason Wessel <jason.wessel@windriver.com>]

On 06/11/2012 08:52 AM, Richard Purdie wrote:
> On Fri, 2012-06-08 at 08:41 -0500, Jason Wessel wrote:
>> If you have a fully populated sstate cache and have used it to
>> execute a build, it is not possible to invalidate repackage
>> an intermediate build after you have forced a compiled
>>
>> Example when you have build from a complete sstate cache build:
>>   bitbake core-image-sato
>>   bitbake -c patch acl
>>      *** Make some changes to the C files for experimentation.
>>   bitbake -f -c compile acl
>>   bitbake acl
>>
>> The bitbake will refuse to build the acl package at this
>> point and instead keep populating it from the sstate.  Using
>> the cleanstate is no longer a good option because it will
>> also erase the scratch area.
>>
>> Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
>> ---
>>  lib/bb/build.py    |    8 ++++++++
>>  lib/bb/runqueue.py |   14 ++++++++++++++
>>  2 files changed, 22 insertions(+)
>>
>> diff --git a/lib/bb/build.py b/lib/bb/build.py
>> index fb61b00..18c28aa 100644
>> --- a/lib/bb/build.py
>> +++ b/lib/bb/build.py
>> @@ -463,6 +463,14 @@ def del_stamp(task, d, file_name = None):
>>      stamp = stamp_internal(task, d, file_name)
>>      bb.utils.remove(stamp)
>>  
>> +def exists_stamp(task, d, file_name = None):
>> +    """
>> +    Removes a stamp for a given task
>> +    (d can be a data dict or dataCache)
>> +    """
>> +    stamp = stamp_internal(task, d, file_name)
>> +    return os.path.exists(stamp)
>> +
>>  def stampfile(taskname, d, file_name = None):
>>      """
>>      Return the stamp for a given task
>> diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
>> index da3fdf9..6c802af 100644
>> --- a/lib/bb/runqueue.py
>> +++ b/lib/bb/runqueue.py
>> @@ -718,6 +718,20 @@ class RunQueueData:
>>                      for dep in self.runq_depends[task]:
>>                          procdep.append(self.taskData.fn_index[self.runq_fnid[dep]] + "." + self.runq_task[dep])
>>                      self.runq_hash[task] = bb.parse.siggen.get_taskhash(self.taskData.fn_index[self.runq_fnid[task]], self.runq_task[task], procdep, self.dataCache)
>> +        try:
>> +            new_setscene = []
>> +            for task in self.runq_setscene:
>> +                try:
>> +                    fn = self.taskData.fn_index[self.rq.rqdata.runq_fnid[task]]
>> +                    if bb.build.exists_stamp("do_unpack", self.dataCache, fn):
>> +                        logger.debug(2, 'Removing task %s from queue because do_unpack exists', task)
>> +                    else:
>> +                        new_setscene.append(task)
>> +                except:
>> +                    logger.debug(2, 'Failed do_unpack check for %s', task)
>> +            self.runq_setscene = new_setscene
>> +        except:
>> +            logger.debug(2, 'Failed to update runq_setscene')
> We've gone to quite a bit of trouble to keep knowledge of the specific
> tasks out of bitbake. I've noticed a few of your patches are "blurring"
> the separation between bitbake and the metadata.
>
> Here, bitbake shouldn't have knowledge of "unpack" or why its special.
> I'd also argue its not in fact special and there are probably other
> "force" scenarios which would break with a similar problem even with
> this patch. I agree there is a problem here but we need to fix it in a
> more generic way. There is already a bug open on this kind of problem
> (#2256).

As I mentioned in the 0/18 summary this is certainly the case that there needs to be an API and this should be treated as more of an RFC.

I will break whole series down into a few smaller chunks since there are some important fixes vs the features.


Jason.