[PATCH V4 0/1] bitbake: runqueue: catch hashvalidate error in RunQueueExecuteScenequeue

[PATCH V4 0/1] bitbake: runqueue: catch hashvalidate error in RunQueueExecuteScenequeue

[Hongxu Jia]

Changed in V4:
- Use bb.BBHandledException to instead of custom exception HashValidateFailure

Changed in V3:
- Add exception HashValidateFailure for the specific error

Changed in V2: 
- Strip the commit message within 80 characters a line

//Hongxu


The following changes since commit 52b788c6df9201764130ab744bf67b770b24b896:

  autogen-native: inherit pkgconfig to fix a build failure (2014-08-25 10:26:00 +0100)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib hongxu/bitbake
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/bitbake

Hongxu Jia (1):
  bitbake: runqueue: catch hashvalidate error in
    RunQueueExecuteScenequeue

 bitbake/lib/bb/runqueue.py | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)

-- 
1.9.1

[PATCH 1/1] bitbake: runqueue: catch hashvalidate error in RunQueueExecuteScenequeue

[Hongxu Jia]

We need to catch the failure of self.rq.hashvalidate which invoked
in RunQueueExecuteScenequeue, and exit in a normal build.

So we raise and except bb.BBHandledException for this specific error.

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 bitbake/lib/bb/runqueue.py | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
index e13dc57..59e4188 100644
--- a/bitbake/lib/bb/runqueue.py
+++ b/bitbake/lib/bb/runqueue.py
@@ -1046,7 +1046,14 @@ class RunQueue:
                 self.state = runQueueComplete
             else:
                 self.start_worker()
-                self.rqexe = RunQueueExecuteScenequeue(self)
+                try:
+                    self.rqexe = RunQueueExecuteScenequeue(self)
+                except bb.BBHandledException:
+                    # While hashvalidate failed, we need exit the normal build.
+                    self.state = runQueueComplete
+                except:
+                    logger.error("RunQueueExecuteScenequeue init failed")
+                    raise
 
         if self.state in [runQueueSceneRun, runQueueRunning, runQueueCleanUp]:
             self.dm.check(self)
@@ -1821,7 +1828,11 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
                 sq_task.append(task)
             call = self.rq.hashvalidate + "(sq_fn, sq_task, sq_hash, sq_hashfn, d)"
             locs = { "sq_fn" : sq_fn, "sq_task" : sq_taskname, "sq_hash" : sq_hash, "sq_hashfn" : sq_hashfn, "d" : self.cooker.data }
-            valid = bb.utils.better_eval(call, locs)
+            try:
+                valid = bb.utils.better_eval(call, locs)
+            except Exception as e:
+                logger.error("Hash validation failed in RunQueueExecuteScenequeue %s" % str(e))
+                raise bb.BBHandledException()
 
             valid_new = stamppresent
             for v in valid:
@@ -2003,7 +2014,6 @@ class TaskFailure(Exception):
     def __init__(self, x):
         self.args = x
 
-
 class runQueueExitWait(bb.event.Event):
     """
     Event when waiting for task processes to exit
-- 
1.9.1

Re: [PATCH 1/1] bitbake: runqueue: catch hashvalidate error in RunQueueExecuteScenequeue

[Richard Purdie]

On Wed, 2014-08-27 at 21:57 +0800, Hongxu Jia wrote:
> We need to catch the failure of self.rq.hashvalidate which invoked
> in RunQueueExecuteScenequeue, and exit in a normal build.
> 
> So we raise and except bb.BBHandledException for this specific error.
> 
> Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
> ---
>  bitbake/lib/bb/runqueue.py | 16 +++++++++++++---
>  1 file changed, 13 insertions(+), 3 deletions(-)
> 
> diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
> index e13dc57..59e4188 100644
> --- a/bitbake/lib/bb/runqueue.py
> +++ b/bitbake/lib/bb/runqueue.py
> @@ -1046,7 +1046,14 @@ class RunQueue:
>                  self.state = runQueueComplete
>              else:
>                  self.start_worker()
> -                self.rqexe = RunQueueExecuteScenequeue(self)
> +                try:
> +                    self.rqexe = RunQueueExecuteScenequeue(self)
> +                except bb.BBHandledException:
> +                    # While hashvalidate failed, we need exit the normal build.
> +                    self.state = runQueueComplete
> +                except:
> +                    logger.error("RunQueueExecuteScenequeue init failed")
> +                    raise

Do you still need to do this with the change below? runqueue didn't
complete, it failed so the state doesn't seem right...

Cheers,

Richard

>          if self.state in [runQueueSceneRun, runQueueRunning, runQueueCleanUp]:
>              self.dm.check(self)
> @@ -1821,7 +1828,11 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
>                  sq_task.append(task)
>              call = self.rq.hashvalidate + "(sq_fn, sq_task, sq_hash, sq_hashfn, d)"
>              locs = { "sq_fn" : sq_fn, "sq_task" : sq_taskname, "sq_hash" : sq_hash, "sq_hashfn" : sq_hashfn, "d" : self.cooker.data }
> -            valid = bb.utils.better_eval(call, locs)
> +            try:
> +                valid = bb.utils.better_eval(call, locs)
> +            except Exception as e:
> +                logger.error("Hash validation failed in RunQueueExecuteScenequeue %s" % str(e))
> +                raise bb.BBHandledException()
>  
>              valid_new = stamppresent
>              for v in valid:
> @@ -2003,7 +2014,6 @@ class TaskFailure(Exception):
>      def __init__(self, x):
>          self.args = x
>  
> -
>  class runQueueExitWait(bb.event.Event):
>      """
>      Event when waiting for task processes to exit

Re: [PATCH 1/1] bitbake: runqueue: catch hashvalidate error in RunQueueExecuteScenequeue

[Hongxu Jia]

On 08/27/2014 10:05 PM, Richard Purdie wrote:
> On Wed, 2014-08-27 at 21:57 +0800, Hongxu Jia wrote:
>> We need to catch the failure of self.rq.hashvalidate which invoked
>> in RunQueueExecuteScenequeue, and exit in a normal build.
>>
>> So we raise and except bb.BBHandledException for this specific error.
>>
>> Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
>> ---
>>   bitbake/lib/bb/runqueue.py | 16 +++++++++++++---
>>   1 file changed, 13 insertions(+), 3 deletions(-)
>>
>> diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
>> index e13dc57..59e4188 100644
>> --- a/bitbake/lib/bb/runqueue.py
>> +++ b/bitbake/lib/bb/runqueue.py
>> @@ -1046,7 +1046,14 @@ class RunQueue:
>>                   self.state = runQueueComplete
>>               else:
>>                   self.start_worker()
>> -                self.rqexe = RunQueueExecuteScenequeue(self)
>> +                try:
>> +                    self.rqexe = RunQueueExecuteScenequeue(self)
>> +                except bb.BBHandledException:
>> +                    # While hashvalidate failed, we need exit the normal build.
>> +                    self.state = runQueueComplete
>> +                except:
>> +                    logger.error("RunQueueExecuteScenequeue init failed")
>> +                    raise
> Do you still need to do this with the change below? runqueue didn't
> complete, it failed so the state doesn't seem right...

I want to exit the build peacefully, without too much break messages
if we don't assign the above complete status, we have messages like this:
...
ERROR: Hash validation failed in RunQueueExecuteScenequeue
ERROR: An uncaught exception occured in runqueue, please see the failure 
below:
ERROR: Running idle function
Traceback (most recent call last):
                  try:
     >                retval = function(self, data, False)
                      if retval is False:
                  try:
     >                retval = rq.execute_runqueue()
                  except runqueue.TaskFailure as exc:
   File "/home/jiahongxu/yocto/poky/bitbake/lib/bb/runqueue.py", line 
1103, in RunQueue.execute_runqueue():
              try:
     >            return self._execute_runqueue()
              except bb.runqueue.TaskFailure:
   File "/home/jiahongxu/yocto/poky/bitbake/lib/bb/runqueue.py", line 
1049, in RunQueue._execute_runqueue():
                      self.start_worker()
     >                self.rqexe = RunQueueExecuteScenequeue(self)
                      '''
   File "/home/jiahongxu/yocto/poky/bitbake/lib/bb/runqueue.py", line 
1838, in RunQueueExecuteScenequeue.__init__(rq=<bb.runqueue.RunQueue 
instance at 0x7f509f4bd950>):
                      logger.error("Hash validation failed in 
RunQueueExecuteScenequeue %s" % str(e))
     >                raise bb.BBHandledException()

BBHandledException
...

Do you have ang ideas about how to exit the build peacefully?

//Hongxu


> Cheers,
>
> Richard
>
>>           if self.state in [runQueueSceneRun, runQueueRunning, runQueueCleanUp]:
>>               self.dm.check(self)
>> @@ -1821,7 +1828,11 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
>>                   sq_task.append(task)
>>               call = self.rq.hashvalidate + "(sq_fn, sq_task, sq_hash, sq_hashfn, d)"
>>               locs = { "sq_fn" : sq_fn, "sq_task" : sq_taskname, "sq_hash" : sq_hash, "sq_hashfn" : sq_hashfn, "d" : self.cooker.data }
>> -            valid = bb.utils.better_eval(call, locs)
>> +            try:
>> +                valid = bb.utils.better_eval(call, locs)
>> +            except Exception as e:
>> +                logger.error("Hash validation failed in RunQueueExecuteScenequeue %s" % str(e))
>> +                raise bb.BBHandledException()
>>   
>>               valid_new = stamppresent
>>               for v in valid:
>> @@ -2003,7 +2014,6 @@ class TaskFailure(Exception):
>>       def __init__(self, x):
>>           self.args = x
>>   
>> -
>>   class runQueueExitWait(bb.event.Event):
>>       """
>>       Event when waiting for task processes to exit
>

Re: [PATCH 1/1] bitbake: runqueue: catch hashvalidate error in RunQueueExecuteScenequeue

[Richard Purdie]

On Wed, 2014-08-27 at 22:16 +0800, Hongxu Jia wrote:
> On 08/27/2014 10:05 PM, Richard Purdie wrote:
> > On Wed, 2014-08-27 at 21:57 +0800, Hongxu Jia wrote:
> >> We need to catch the failure of self.rq.hashvalidate which invoked
> >> in RunQueueExecuteScenequeue, and exit in a normal build.
> >>
> >> So we raise and except bb.BBHandledException for this specific error.
> >>
> >> Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
> >> ---
> >>   bitbake/lib/bb/runqueue.py | 16 +++++++++++++---
> >>   1 file changed, 13 insertions(+), 3 deletions(-)
> >>
> >> diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
> >> index e13dc57..59e4188 100644
> >> --- a/bitbake/lib/bb/runqueue.py
> >> +++ b/bitbake/lib/bb/runqueue.py
> >> @@ -1046,7 +1046,14 @@ class RunQueue:
> >>                   self.state = runQueueComplete
> >>               else:
> >>                   self.start_worker()
> >> -                self.rqexe = RunQueueExecuteScenequeue(self)
> >> +                try:
> >> +                    self.rqexe = RunQueueExecuteScenequeue(self)
> >> +                except bb.BBHandledException:
> >> +                    # While hashvalidate failed, we need exit the normal build.
> >> +                    self.state = runQueueComplete
> >> +                except:
> >> +                    logger.error("RunQueueExecuteScenequeue init failed")
> >> +                    raise
> > Do you still need to do this with the change below? runqueue didn't
> > complete, it failed so the state doesn't seem right...
> 
> I want to exit the build peacefully, without too much break messages
> if we don't assign the above complete status, we have messages like this:
> ...
> ERROR: Hash validation failed in RunQueueExecuteScenequeue
> ERROR: An uncaught exception occured in runqueue, please see the failure 
> below:
> ERROR: Running idle function
> Traceback (most recent call last):
>                   try:
>      >                retval = function(self, data, False)
>                       if retval is False:
>                   try:
>      >                retval = rq.execute_runqueue()
>                   except runqueue.TaskFailure as exc:
>    File "/home/jiahongxu/yocto/poky/bitbake/lib/bb/runqueue.py", line 
> 1103, in RunQueue.execute_runqueue():
>               try:
>      >            return self._execute_runqueue()
>               except bb.runqueue.TaskFailure:
>    File "/home/jiahongxu/yocto/poky/bitbake/lib/bb/runqueue.py", line 
> 1049, in RunQueue._execute_runqueue():
>                       self.start_worker()
>      >                self.rqexe = RunQueueExecuteScenequeue(self)
>                       '''
>    File "/home/jiahongxu/yocto/poky/bitbake/lib/bb/runqueue.py", line 
> 1838, in RunQueueExecuteScenequeue.__init__(rq=<bb.runqueue.RunQueue 
> instance at 0x7f509f4bd950>):
>                       logger.error("Hash validation failed in 
> RunQueueExecuteScenequeue %s" % str(e))
>      >                raise bb.BBHandledException()
> 
> BBHandledException
> ...
> 
> Do you have ang ideas about how to exit the build peacefully?

I think bitbake has some deeper issues here we should fix. Firstly, I
think execute_runqueue should really do something like:

        try:
            return self._execute_runqueue()
        except bb.runqueue.TaskFailure:
            raise
        except BaseException as e:
            if not isinstance(e, (SystemExit, BBHandledException)):
                logger.error("An uncaught exception occured in runqueue, please see the failure below:")
            try:
                self.teardown_workers()
            except:
                pass
            self.state = runQueueComplete
            raise

then in buildFileIdle and buildTargetsIdle we should change SystemExit to (SystemExit, BBHandledException) too.

Cheers,

Richard