[PATCH] server/process: Handle SIGTERM more gracefully

[PATCH] server/process: Handle SIGTERM more gracefully

[Richard Purdie]

Currently if you send a SIGTERM to the bitbake UI process, the system basically
hangs if tasks are executing. This is because the server process doesn't
actually try any kind of shutdown before exiting.

This patch trys executing a stateForceShutdown command first, which is
enough to stop any active tasks before the system exits.

I also noticed that terminate can execute multiple times, once at SIGTERM
from the handler and once from the real exit. Double execution leads to
stack traces and potential hangs (writes to dead pipes), so ensure
the code only can run once.

With these fixes, bitbake much more correctly deals with SIGTERM to the
UI process.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

diff --git a/bitbake/lib/bb/server/process.py b/bitbake/lib/bb/server/process.py
index f022b86..5fca350 100644
--- a/bitbake/lib/bb/server/process.py
+++ b/bitbake/lib/bb/server/process.py
@@ -114,6 +114,10 @@ class ProcessServer(Process, BaseImplServer):
                 if self.quitout.poll():
                     self.quitout.recv()
                     self.quit = True
+                    try:
+                        self.runCommand(["stateForceShutdown"])
+                    except:
+                        pass
 
                 self.idle_commands(.1, [self.command_channel, self.quitout])
             except Exception:
@@ -123,6 +127,7 @@ class ProcessServer(Process, BaseImplServer):
         bb.event.unregister_UIHhandler(self.event_handle.value)
         self.command_channel.close()
         self.cooker.shutdown(True)
+        self.quitout.close()
 
     def idle_commands(self, delay, fds=None):
         nextsleep = delay
@@ -172,12 +177,16 @@ class BitBakeProcessServerConnection(BitBakeBaseServerConnection):
         self.event_queue = event_queue
         self.connection = ServerCommunicator(self.ui_channel, self.procserver.event_handle, self.procserver)
         self.events = self.event_queue
+        self.terminated = False
 
     def sigterm_terminate(self):
         bb.error("UI received SIGTERM")
         self.terminate()
 
     def terminate(self):
+        if self.terminated:
+            return
+        self.terminated = True
         def flushevents():
             while True:
                 try:

Re: [PATCH] server/process: Handle SIGTERM more gracefully

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1753 bytes --]

On Mon, Sep 7, 2015 at 8:55 AM, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> Currently if you send a SIGTERM to the bitbake UI process, the system
> basically
> hangs if tasks are executing. This is because the server process doesn't
> actually try any kind of shutdown before exiting.
>
> This patch trys executing a stateForceShutdown command first, which is
> enough to stop any active tasks before the system exits.
>
> I also noticed that terminate can execute multiple times, once at SIGTERM
> from the handler and once from the real exit. Double execution leads to
> stack traces and potential hangs (writes to dead pipes), so ensure
> the code only can run once.
>
> With these fixes, bitbake much more correctly deals with SIGTERM to the
> UI process.
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>

Just as an FYI for consideration in the longer term, you can raise an
exception from a signal handler, to handle sigterm the same way as sigint
for consistency. Ex.:

https://github.com/kergoth/oe-test-scripts/blob/master/bb_test.py#L12-L17 +
https://github.com/kergoth/oe-test-scripts/blob/master/bb_test.py#L171-L180

Which reminds me, we don't exit with the correct exit code for
sigint/sigterm, so the calling shell doesn't know the cause of the exit.
See also http://www.cons.org/cracauer/sigint.html -- fantastic article
there, highly recommended.

https://docs.python.org/3/library/signal.html#example has an example of
raising an exception from another signal handler, so it seems to be kosher.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 2871 bytes --]