[PATCH] bitbake: enable pdb tracing

[PATCH] bitbake: enable pdb tracing

[joe]

From: Joe MacDonald <joe@deserted.net>

Based on:

   http://blog.devork.be/2009/07/how-to-bring-running-python-program.html

register a signal handler to dynamically start pdb tracing on receiving
SIGUSR1.

Signed-off-by: Joe MacDonald <joe@deserted.net>
---

I've been spending the last week or so looking at some perplexing hangs
(for lack of a more descriptive word) with bitbake.  The behaviour is
difficult to characterize and even harder to reproduce, but over the
course of a day with a dozen machines or so doing various builds maybe a
dozen times we'll get in a scenario where bitbake itself appears to be
completely idle and waiting on something.  Occasionally it's been when
doing a 'bitbake -g' or 'bitbake -e', most recently I saw this on a
'bitbake core-image-minimal'.

Given the difficulty I've had in actually making the hang occur in
controlled circumstances, I can't say for certain, but it appears that
enabling any level of debugging makes the problem disappear.  So I've been
watching for it to happen then attaching gdb and investigating that way.

That's when it occurred to me that it might actually be useful to others
to have pdb available as an option, rather than needing to resort to
attaching gdb/python and debugging that way.

I did a quick search of the mailing list archives and didn't see this
discussed before, but if it has been and dismissed, sorry.

 -J.

 bitbake/bin/bitbake |    9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/bitbake/bin/bitbake b/bitbake/bin/bitbake
index 60c4b96..e40fe28 100755
--- a/bitbake/bin/bitbake
+++ b/bitbake/bin/bitbake
@@ -24,6 +24,7 @@
 
 import os
 import sys, logging
+import signal
 sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)),
                                 'lib'))
 
@@ -57,6 +58,13 @@ try:
 except:
     pass
 
+# start pdb tracing if we receive SIGUSR1.
+def handle_pdb(sig, frame):
+    try:
+        import pdb
+        pdb.Pdb().set_trace(frame)
+    except ImportError:
+        logger.debug(2, 'Unable to import pdb module, interactive debugging of bitbake will not be possible')
 
 def get_ui(config):
     if not config.ui:
@@ -338,6 +346,7 @@ def main():
 
 if __name__ == "__main__":
     try:
+        signal.signal(signal.SIGUSR1, handle_pdb)
         ret = main()
     except Exception:
         ret = 1
-- 
1.7.10.4

Re: [PATCH] bitbake: enable pdb tracing

[Chris Larson]

[-- Attachment #1: Type: text/plain, Size: 809 bytes --]

On Thu, Aug 1, 2013 at 7:19 AM, <joe@deserted.net> wrote:

> From: Joe MacDonald <joe@deserted.net>
>
> Based on:
>
>    http://blog.devork.be/2009/07/how-to-bring-running-python-program.html
>
> register a signal handler to dynamically start pdb tracing on receiving
> SIGUSR1.
>
> Signed-off-by: Joe MacDonald <joe@deserted.net>
>

As long as you keep in mind this will only affect the UI, not the forked
off server, this could be useful. I'd suggest adding something for the
server as well, but it'd need to listen to a socket for the input, lacking
access to the user's tty, or would need to interface with the UI to do it :)
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1374 bytes --]

Re: [PATCH] bitbake: enable pdb tracing

[Joe MacDonald]

[-- Attachment #1: Type: text/plain, Size: 1241 bytes --]

On Fri, Aug 2, 2013 at 5:05 PM, Chris Larson <clarson@kergoth.com> wrote:

> On Thu, Aug 1, 2013 at 7:19 AM, <joe@deserted.net> wrote:
>
>> From: Joe MacDonald <joe@deserted.net>
>>
>> Based on:
>>
>>    http://blog.devork.be/2009/07/how-to-bring-running-python-program.html
>>
>> register a signal handler to dynamically start pdb tracing on receiving
>> SIGUSR1.
>>
>> Signed-off-by: Joe MacDonald <joe@deserted.net>
>>
>
> As long as you keep in mind this will only affect the UI, not the forked
> off server, this could be useful. I'd suggest adding something for the
> server as well, but it'd need to listen to a socket for the input, lacking
> access to the user's tty, or would need to interface with the UI to do it :)
>

That's a really good idea.  It seemed like anything was better than what I
had at the time, but now that you mention it being able to use pdb (or
something like it) to talk to one of the forked processes would be a really
helpful with the current problem I'm looking at.  I'll see what I can do
about extending this (since it doesn't sound like it's a fundamentally bad
idea, just of limited applicability here) to more than just the UI.

Thanks Chris.

-- 
Joe MacDonald
:wq

[-- Attachment #2: Type: text/html, Size: 2106 bytes --]

Re: [PATCH] bitbake: enable pdb tracing

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1612 bytes --]

On Fri, Aug 02, 2013 at 09:07:15PM -0400, Joe MacDonald wrote:
> On Fri, Aug 2, 2013 at 5:05 PM, Chris Larson <clarson@kergoth.com> wrote:
> 
> > On Thu, Aug 1, 2013 at 7:19 AM, <joe@deserted.net> wrote:
> >
> >> From: Joe MacDonald <joe@deserted.net>
> >>
> >> Based on:
> >>
> >>    http://blog.devork.be/2009/07/how-to-bring-running-python-program.html
> >>
> >> register a signal handler to dynamically start pdb tracing on receiving
> >> SIGUSR1.
> >>
> >> Signed-off-by: Joe MacDonald <joe@deserted.net>
> >>
> >
> > As long as you keep in mind this will only affect the UI, not the forked
> > off server, this could be useful. I'd suggest adding something for the
> > server as well, but it'd need to listen to a socket for the input, lacking
> > access to the user's tty, or would need to interface with the UI to do it :)
> >
> 
> That's a really good idea.  It seemed like anything was better than what I
> had at the time, but now that you mention it being able to use pdb (or
> something like it) to talk to one of the forked processes would be a really
> helpful with the current problem I'm looking at.  I'll see what I can do
> about extending this (since it doesn't sound like it's a fundamentally bad
> idea, just of limited applicability here) to more than just the UI.

Hi,

one type of hangs is described here
https://bugzilla.yoctoproject.org/show_bug.cgi?id=4338
I have no experience with pdb, but if it will help debuging this king of
issues I'll happily try it next time I get it hanging.

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]