RE: [OE-core] [RFC][PATCH] server/process: Avoid hanging if a parser process is terminated

RE: [OE-core] [RFC][PATCH] server/process: Avoid hanging if a parser process is terminated

[Peter Kjellerstedt]

Bah, this was of course intended for the bitbake list.

//Peter

> -----Original Message-----
> From: openembedded-core@lists.openembedded.org <openembedded-
> core@lists.openembedded.org> On Behalf Of Peter Kjellerstedt
> Sent: den 26 mars 2022 06:19
> To: openembedded-core@lists.openembedded.org
> Subject: [OE-core] [RFC][PATCH] server/process: Avoid hanging if a parser
> process is terminated
> 
> If a parser process is terminated while holding a write lock, then it
> will lead to a deadlock (see
> https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Process.terminate).
> 
> Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> ---
> 
> After the discussion on IRC about the hanging parsing processes, I
> just had to understand the code and what happens. This is a solution to
> the problem, though I am not sure it is THE solution. It may also be
> that the rlock in ConnectionReader should be handled the same way.
> Anyway, even if this is not accepted as a solution to the problem, I now
> at least know a lot more about how the parsing processes in bitbake
> work... :)
> 
>  bitbake/lib/bb/server/process.py | 9 +++++++++
>  1 file changed, 9 insertions(+)
> 
> diff --git a/bitbake/lib/bb/server/process.py
> b/bitbake/lib/bb/server/process.py
> index 1636616660..544b00f2cd 100644
> --- a/bitbake/lib/bb/server/process.py
> +++ b/bitbake/lib/bb/server/process.py
> @@ -20,6 +20,7 @@ import os
>  import sys
>  import time
>  import select
> +import signal
>  import socket
>  import subprocess
>  import errno
> @@ -728,6 +729,10 @@ class ConnectionReader(object):
>      def close(self):
>          return self.reader.close()
> 
> +terminated = False
> +def catch_sigterm(signum, frame):
> +    global terminated
> +    terminated = True
> 
>  class ConnectionWriter(object):
> 
> @@ -739,8 +744,12 @@ class ConnectionWriter(object):
> 
>      def send(self, obj):
>          obj = multiprocessing.reduction.ForkingPickler.dumps(obj)
> +        oldsig = signal.signal(signal.SIGTERM, catch_sigterm)
>          with self.wlock:
>              self.writer.send_bytes(obj)
> +        signal.signal(signal.SIGTERM, oldsig)
> +        if terminated:
> +            os.kill(os.getpid(), signal.SIGTERM)
> 
>      def fileno(self):
>          return self.writer.fileno()