[PATCH 0/1] Force to teminate the Parser process

[PATCH 0/1] Force to teminate the Parser process

[Kang Kai]

This patch is for Yocto 2142.

Force to exit HOB when hob is parsing recipes, the bitbake doesn't stop.
It hangs on function BitBakeServerConnection::terminate in file
server/process.py:
    else:
        self.procserver.join()
It is waiting for the children process quit.

In stage of parse recipes BBCooker spawns Parser processes as many as
cpu numbers. When quit the Parser processes they make their internal
Queue to call cancel_join_thread() to avoid block but don't work at
this time.

So force to terminate the Parser processes.

The following changes since commit 3bf8069100e54153d13319dc32ca089327b6a139:

  documentation/yocto-project-qs/yocto-project-qs.xml: added quotes (2012-05-01 21:00:36 +0100)

are available in the git repository at:
  git://git.pokylinux.org/poky-contrib kangkai/distro
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kangkai/distro

Kang Kai (1):
  cooker.py: terminate the Parser processes

 bitbake/lib/bb/cooker.py |    9 ++++++---
 1 files changed, 6 insertions(+), 3 deletions(-)

-- 
1.7.5.4

[PATCH 1/1] cooker.py: terminate the Parser processes

[Kang Kai]

[Yocto 2142]

Force to exit HOB when hob is parsing recipes, the bitbake doesn't stop.
It hangs on function BitBakeServerConnection::terminate in file
server/process.py:
    else:
        self.procserver.join()
It is waiting for the children process quit.

In stage of parse recipes BBCooker spawns Parser processes as many as
cpu numbers. When quit the Parser processes they make their internal
Queue to call cancel_join_thread() to avoid block but don't work at
this time.
So force to terminate the Parser processes.

Signed-off-by: Kang Kai <kai.kang@windriver.com>
---
 bitbake/lib/bb/cooker.py |    9 ++++++---
 1 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index dea0aad..4a4dc38 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -1175,7 +1175,7 @@ class BBCooker:
             return
 
         if self.state in (state.shutdown, state.stop):
-            self.parser.shutdown(clean=False)
+            self.parser.shutdown(clean=False, force = True)
             sys.exit(1)
 
         if self.state != state.parsing:
@@ -1608,10 +1608,13 @@ class CookerParser(object):
                 self.parser_quit.put(None)
 
             self.jobs.cancel_join_thread()
-            sys.exit(1)
 
         for process in self.processes:
-            process.join()
+            if force:
+                process.join(.1)
+                process.terminate()
+            else:
+                process.join()
         self.feeder.join()
 
         sync = threading.Thread(target=self.bb_cache.sync)
-- 
1.7.5.4

Re: [PATCH 1/1] cooker.py: terminate the Parser processes

[Chris Larson]

terminate()'ing processes can corrupt any queues those processes have
open, and thereby disrupt any pending log messages in the queues
heading to the ui as a result. If that's not an issue here, so be it,
but be aware of the issues -- we had to remove terminate() from the
shutdown in the past due to this issue.

On Wed, May 23, 2012 at 2:40 AM, Kang Kai <kai.kang@windriver.com> wrote:
> [Yocto 2142]
>
> Force to exit HOB when hob is parsing recipes, the bitbake doesn't stop.
> It hangs on function BitBakeServerConnection::terminate in file
> server/process.py:
>    else:
>        self.procserver.join()
> It is waiting for the children process quit.
>
> In stage of parse recipes BBCooker spawns Parser processes as many as
> cpu numbers. When quit the Parser processes they make their internal
> Queue to call cancel_join_thread() to avoid block but don't work at
> this time.
> So force to terminate the Parser processes.
>
> Signed-off-by: Kang Kai <kai.kang@windriver.com>
> ---
>  bitbake/lib/bb/cooker.py |    9 ++++++---
>  1 files changed, 6 insertions(+), 3 deletions(-)
>
> diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
> index dea0aad..4a4dc38 100644
> --- a/bitbake/lib/bb/cooker.py
> +++ b/bitbake/lib/bb/cooker.py
> @@ -1175,7 +1175,7 @@ class BBCooker:
>             return
>
>         if self.state in (state.shutdown, state.stop):
> -            self.parser.shutdown(clean=False)
> +            self.parser.shutdown(clean=False, force = True)
>             sys.exit(1)
>
>         if self.state != state.parsing:
> @@ -1608,10 +1608,13 @@ class CookerParser(object):
>                 self.parser_quit.put(None)
>
>             self.jobs.cancel_join_thread()
> -            sys.exit(1)
>
>         for process in self.processes:
> -            process.join()
> +            if force:
> +                process.join(.1)
> +                process.terminate()
> +            else:
> +                process.join()
>         self.feeder.join()
>
>         sync = threading.Thread(target=self.bb_cache.sync)
> --
> 1.7.5.4
>
>
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel



-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: [PATCH 1/1] cooker.py: terminate the Parser processes

[Kang Kai]

On 2012年05月23日 23:34, Chris Larson wrote:
> terminate()'ing processes can corrupt any queues those processes have
> open, and thereby disrupt any pending log messages in the queues
> heading to the ui as a result. If that's not an issue here, so be it,
> but be aware of the issues -- we had to remove terminate() from the
> shutdown in the past due to this issue.
Hi Chris,

Call terminate() directly seems not be elegant. But I didn't figure out 
why after the queue belongs to Parse process
calls  cancel_join_thread(), so ..
Before the terminate() the every Parse process, it is waited for 0.1 
second to quit.
If the process will not be blocked, I suppose that 0.1 second is enough 
for them to quit.

What's your opinion?

Thanks,
Kai
>
> On Wed, May 23, 2012 at 2:40 AM, Kang Kai<kai.kang@windriver.com>  wrote:
>> [Yocto 2142]
>>
>> Force to exit HOB when hob is parsing recipes, the bitbake doesn't stop.
>> It hangs on function BitBakeServerConnection::terminate in file
>> server/process.py:
>>     else:
>>         self.procserver.join()
>> It is waiting for the children process quit.
>>
>> In stage of parse recipes BBCooker spawns Parser processes as many as
>> cpu numbers. When quit the Parser processes they make their internal
>> Queue to call cancel_join_thread() to avoid block but don't work at
>> this time.
>> So force to terminate the Parser processes.
>>
>> Signed-off-by: Kang Kai<kai.kang@windriver.com>
>> ---
>>   bitbake/lib/bb/cooker.py |    9 ++++++---
>>   1 files changed, 6 insertions(+), 3 deletions(-)
>>
>> diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
>> index dea0aad..4a4dc38 100644
>> --- a/bitbake/lib/bb/cooker.py
>> +++ b/bitbake/lib/bb/cooker.py
>> @@ -1175,7 +1175,7 @@ class BBCooker:
>>              return
>>
>>          if self.state in (state.shutdown, state.stop):
>> -            self.parser.shutdown(clean=False)
>> +            self.parser.shutdown(clean=False, force = True)
>>              sys.exit(1)
>>
>>          if self.state != state.parsing:
>> @@ -1608,10 +1608,13 @@ class CookerParser(object):
>>                  self.parser_quit.put(None)
>>
>>              self.jobs.cancel_join_thread()
>> -            sys.exit(1)
>>
>>          for process in self.processes:
>> -            process.join()
>> +            if force:
>> +                process.join(.1)
>> +                process.terminate()
>> +            else:
>> +                process.join()
>>          self.feeder.join()
>>
>>          sync = threading.Thread(target=self.bb_cache.sync)
>> --
>> 1.7.5.4
>>
>>
>> _______________________________________________
>> bitbake-devel mailing list
>> bitbake-devel@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel
>
>