* Hanging threads with pthread_detach and gdb
@ 2011-08-16 12:58 Philipp Marek
[not found] ` <CAF1ivSZ-CCzFUH4WtxqL2QBoyB3WtF+KMqnCBdjaHhZ7-bbVjA@mail.gmail.com>
0 siblings, 1 reply; 4+ messages in thread
From: Philipp Marek @ 2011-08-16 12:58 UTC (permalink / raw)
To: linux-kernel
[-- Attachment #1: Type: text/plain, Size: 4174 bytes --]
Hello everybody,
I've found a strange behaviour, and I think it's a kernel bug - or, at
least, a bad interaction with GDB.
The attached program creates a few detached pthreads, and quits.
Running the program as-is works without any problem; but when it's started
via gdb there's an occasional hang at the end (1 out of 5 to 10 runs).
CTRL-C doesn't work:
$ gdb -ex r --args ./t
...
Starting program: t
[Thread debugging using libthread_db enabled]
[New Thread 0x7ffff783c700 (LWP 8227)]
thread (nil) START
[New Thread 0x7ffff703b700 (LWP 8228)]
thread 0x1 START
[New Thread 0x7ffff683a700 (LWP 8229)]
thread 0x2 START
[New Thread 0x7ffff6039700 (LWP 8230)]
thread 0x3 START
[New Thread 0x7ffff5838700 (LWP 8231)]
thread 0x4 START
thread 0x4 END
thread 0x2 END
thread (nil) END
the end is near.
[Thread 0x7ffff6039700 (LWP 8230) exited]
[Thread 0x7ffff683a700 (LWP 8229) exited]
[Thread 0x7ffff703b700 (LWP 8228) exited]
[Thread 0x7ffff783c700 (LWP 8227) exited]
^C
"ps fax" shows that the test program would be done:
8210 pts/13 S 0:00 \_ gdb -ex r --args ./t
8226 pts/13 Zl+ 0:00 \_ [t] <defunct>
but GDB still waits for it:
$ strace -p 8210
Process 8120 attached - interrupt to quit
wait4(8226, ^C <unfinished ...>
Process 8120 detached
The kernel stack trace shows need_resched()
$ sudo cat /proc/8226/task/*/stack
[<ffffffff810383fc>] need_resched+0x1a/0x23
[<ffffffff8103840a>] should_resched+0x5/0x24
[<ffffffff81049ea0>] do_exit+0x73e/0x740
[<ffffffff8104a119>] do_group_exit+0x77/0xa1
[<ffffffff8104a155>] sys_exit_group+0x12/0x19
[<ffffffff8133ba92>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
[<ffffffff810383fc>] need_resched+0x1a/0x23
[<ffffffff8103840a>] should_resched+0x5/0x24
[<ffffffff81049ea0>] do_exit+0x73e/0x740
[<ffffffff8104a119>] do_group_exit+0x77/0xa1
[<ffffffff8105676f>] get_signal_to_deliver+0x37c/0x3a3
[<ffffffff810d22bb>] handle_pte_fault+0x295/0x79b
[<ffffffff81008e37>] do_signal+0x6c/0x649
[<ffffffff8133983a>] do_page_fault+0x2d3/0x30e
[<ffffffff810383fc>] need_resched+0x1a/0x23
[<ffffffff8103840a>] should_resched+0x5/0x24
[<ffffffff8103aec8>] mmdrop+0xd/0x1c
[<ffffffff8103b0a2>] finish_task_switch+0x84/0xaf
[<ffffffff810383fc>] need_resched+0x1a/0x23
[<ffffffff81009450>] do_notify_resume+0x25/0x6b
[<ffffffff81336fd2>] paranoid_userspace+0x46/0x50
[<ffffffffffffffff>] 0xffffffffffffffff
(but I've seen traces like this, too:)
[<ffffffff810ed076>] kmem_cache_free+0x2d/0x69
[<ffffffff81055739>] ptrace_stop+0xff/0x19e
[<ffffffff81056550>] get_signal_to_deliver+0x15d/0x3a3
[<ffffffff81008e37>] do_signal+0x6c/0x649
[<ffffffff81035861>] __wake_up_common+0x41/0x78
[<ffffffff810383fc>] need_resched+0x1a/0x23
[<ffffffff8103840a>] should_resched+0x5/0x24
[<ffffffff810121dd>] arch_ptrace+0x7d/0x1bd
[<ffffffff8104fc48>] put_task_struct+0xd/0x1c
[<ffffffff8105084e>] sys_ptrace+0x7d/0x8d
[<ffffffff810fc8b0>] fput+0x1a/0x1a2
[<ffffffff81009450>] do_notify_resume+0x25/0x6b
[<ffffffff810fbd2d>] sys_write+0x5f/0x6b
[<ffffffff81336fd2>] paranoid_userspace+0x46/0x50
[<ffffffffffffffff>] 0xffffffffffffffff
This is with a distribution kernel (sorry), recent userspace:
$ uname -a
Linux 3.0.0-1-amd64 #1 SMP Sun Jul 24 02:24:44 UTC 2011 x86_64 GNU/Linux
$ gdb --version
GNU gdb (GDB) 7.2-debian
$ dpkg-query -l libpth20 gdb
ii libpth20 2.0.7-16 The GNU Portable Threads
ii gdb 7.2-1 The GNU Debugger
I've tried with a vanilla 3.0 ARCH=um (clean 3.0 checkout, git rev
(02f8c6aee8df3cdc935e9bdd4f2d020306035dbe), but get hit by
"Couldn't write debug register: Input/Output error" which seems to be
reported as http://marc.info/?l=user-mode-linux-devel&m=126038615513701 and
http://marc.info/?l=user-mode-linux-devel&m=127181550231140.
Any help would be appreciated!
Please keep me CC'ed; thank you.
Regards,
Phil
[-- Attachment #2: t.c --]
[-- Type: text/x-csrc, Size: 485 bytes --]
#include <stdio.h>
#include <unistd.h>
#include <pthread.h>
int stop;
void *thr(void *x)
{
pthread_detach(pthread_self());
printf("thread %p START\n", x);
while (!stop) ;
printf("thread %p END\n", x);
return NULL;
}
int main(int argc, char *args[])
{
int x, i;
pthread_t pt;
x = sysconf(_SC_NPROCESSORS_ONLN) + 1;
stop = 0;
for(i=0; i<x; i++) {
pthread_create(&pt, NULL, thr, (void*)(long)i);
}
usleep(10e3);
stop = 1;
printf("the end is near.\n");
return 0;
}
[-- Attachment #3: Makefile --]
[-- Type: text/x-makefile, Size: 52 bytes --]
CFLAGS += -g -Wall
LDFLAGS += -lpthread -g
t: t.c
^ permalink raw reply [flat|nested] 4+ messages in thread[parent not found: <CAF1ivSZ-CCzFUH4WtxqL2QBoyB3WtF+KMqnCBdjaHhZ7-bbVjA@mail.gmail.com>]
* Re: Hanging threads with pthread_detach and gdb [not found] ` <CAF1ivSZ-CCzFUH4WtxqL2QBoyB3WtF+KMqnCBdjaHhZ7-bbVjA@mail.gmail.com> @ 2011-08-19 8:54 ` Lin Ming 2011-08-19 10:28 ` Pedro Alves 0 siblings, 1 reply; 4+ messages in thread From: Lin Ming @ 2011-08-19 8:54 UTC (permalink / raw) To: Philipp Marek, Ingo Molnar, Peter Zijlstra; +Cc: linux-kernel [-- Attachment #1: Type: text/plain, Size: 5668 bytes --] > From: Philipp Marek <philipp.marek@linbit.com> > Date: Tue, Aug 16, 2011 at 8:58 PM > Subject: Hanging threads with pthread_detach and gdb > To: linux-kernel@vger.kernel.org > > > Hello everybody, > > > I've found a strange behaviour, and I think it's a kernel bug - or, at > least, a bad interaction with GDB. > > > The attached program creates a few detached pthreads, and quits. > Running the program as-is works without any problem; but when it's started > via gdb there's an occasional hang at the end (1 out of 5 to 10 runs). (Add Ingo and PeterZ) Hi, I can reproduce this problem. After gdb hangs, mlin@wsm:~$ ps -eLf mlin 2277 2220 2277 13 1 00:27 pts/0 00:00:07 gdb test mlin 2431 2277 2431 0 2 00:28 pts/0 00:00:00 [test] <defunct> mlin 2431 2277 2436 0 2 00:28 pts/0 00:00:00 [test] <defunct> I did some investigation and find the cause as below. With the attached debug patch applied, here is the last output lines before gdb hangs. ===== gdb wait for pid=2431 gdb is going to sleep ... gdb children list: pid=2431, exit_state=16, exit_signal=17 gdb ptrace list: pid=2436, exit_state=16, exit_signal=-1 pid=2431, exit_state=16, exit_signal=17 ===== exit_state 16 is EXIT_ZOMBIE state. pid 2431 is the thread group leader. pid 2436 is the thread group member(other members have been removed). gdb is waiting the group leader, but it fails because the group is not empty. Then gdb goes to sleep. At this moment, all threads have gone into EXIT_ZOMBIE state. So no thread can wake up gdb anymore. That's why gdb hangs. I'm not familiar with pthread semantics. Is this a problem need to be fixed? Thanks, Lin Ming > > > CTRL-C doesn't work: > $ gdb -ex r --args ./t > ... > Starting program: t > [Thread debugging using libthread_db enabled] > [New Thread 0x7ffff783c700 (LWP 8227)] > thread (nil) START > [New Thread 0x7ffff703b700 (LWP 8228)] > thread 0x1 START > [New Thread 0x7ffff683a700 (LWP 8229)] > thread 0x2 START > [New Thread 0x7ffff6039700 (LWP 8230)] > thread 0x3 START > [New Thread 0x7ffff5838700 (LWP 8231)] > thread 0x4 START > thread 0x4 END > thread 0x2 END > thread (nil) END > the end is near. > [Thread 0x7ffff6039700 (LWP 8230) exited] > [Thread 0x7ffff683a700 (LWP 8229) exited] > [Thread 0x7ffff703b700 (LWP 8228) exited] > [Thread 0x7ffff783c700 (LWP 8227) exited] > ^C > > > "ps fax" shows that the test program would be done: > 8210 pts/13 S 0:00 \_ gdb -ex r --args ./t > 8226 pts/13 Zl+ 0:00 \_ [t] <defunct> > > > but GDB still waits for it: > $ strace -p 8210 > Process 8120 attached - interrupt to quit > wait4(8226, ^C <unfinished ...> > Process 8120 detached > > The kernel stack trace shows need_resched() > $ sudo cat /proc/8226/task/*/stack > [<ffffffff810383fc>] need_resched+0x1a/0x23 > [<ffffffff8103840a>] should_resched+0x5/0x24 > [<ffffffff81049ea0>] do_exit+0x73e/0x740 > [<ffffffff8104a119>] do_group_exit+0x77/0xa1 > [<ffffffff8104a155>] sys_exit_group+0x12/0x19 > [<ffffffff8133ba92>] system_call_fastpath+0x16/0x1b > [<ffffffffffffffff>] 0xffffffffffffffff > [<ffffffff810383fc>] need_resched+0x1a/0x23 > [<ffffffff8103840a>] should_resched+0x5/0x24 > [<ffffffff81049ea0>] do_exit+0x73e/0x740 > [<ffffffff8104a119>] do_group_exit+0x77/0xa1 > [<ffffffff8105676f>] get_signal_to_deliver+0x37c/0x3a3 > [<ffffffff810d22bb>] handle_pte_fault+0x295/0x79b > [<ffffffff81008e37>] do_signal+0x6c/0x649 > [<ffffffff8133983a>] do_page_fault+0x2d3/0x30e > [<ffffffff810383fc>] need_resched+0x1a/0x23 > [<ffffffff8103840a>] should_resched+0x5/0x24 > [<ffffffff8103aec8>] mmdrop+0xd/0x1c > [<ffffffff8103b0a2>] finish_task_switch+0x84/0xaf > [<ffffffff810383fc>] need_resched+0x1a/0x23 > [<ffffffff81009450>] do_notify_resume+0x25/0x6b > [<ffffffff81336fd2>] paranoid_userspace+0x46/0x50 > [<ffffffffffffffff>] 0xffffffffffffffff > > (but I've seen traces like this, too:) > [<ffffffff810ed076>] kmem_cache_free+0x2d/0x69 > [<ffffffff81055739>] ptrace_stop+0xff/0x19e > [<ffffffff81056550>] get_signal_to_deliver+0x15d/0x3a3 > [<ffffffff81008e37>] do_signal+0x6c/0x649 > [<ffffffff81035861>] __wake_up_common+0x41/0x78 > [<ffffffff810383fc>] need_resched+0x1a/0x23 > [<ffffffff8103840a>] should_resched+0x5/0x24 > [<ffffffff810121dd>] arch_ptrace+0x7d/0x1bd > [<ffffffff8104fc48>] put_task_struct+0xd/0x1c > [<ffffffff8105084e>] sys_ptrace+0x7d/0x8d > [<ffffffff810fc8b0>] fput+0x1a/0x1a2 > [<ffffffff81009450>] do_notify_resume+0x25/0x6b > [<ffffffff810fbd2d>] sys_write+0x5f/0x6b > [<ffffffff81336fd2>] paranoid_userspace+0x46/0x50 > [<ffffffffffffffff>] 0xffffffffffffffff > > > This is with a distribution kernel (sorry), recent userspace: > $ uname -a > Linux 3.0.0-1-amd64 #1 SMP Sun Jul 24 02:24:44 UTC 2011 x86_64 GNU/Linux > $ gdb --version > GNU gdb (GDB) 7.2-debian > $ dpkg-query -l libpth20 gdb > ii libpth20 2.0.7-16 The GNU Portable Threads > ii gdb 7.2-1 The GNU Debugger > > > I've tried with a vanilla 3.0 ARCH=um (clean 3.0 checkout, git rev > (02f8c6aee8df3cdc935e9bdd4f2d020306035dbe), but get hit by > "Couldn't write debug register: Input/Output error" which seems to be > reported as http://marc.info/?l=user-mode-linux-devel&m=126038615513701 and > http://marc.info/?l=user-mode-linux-devel&m=127181550231140. > > > Any help would be appreciated! > Please keep me CC'ed; thank you. > > > Regards, > > Phil [-- Attachment #2: debug.patch --] [-- Type: text/x-patch, Size: 1424 bytes --] diff --git a/kernel/exit.c b/kernel/exit.c index 2913b35..09b81da 100644 --- a/kernel/exit.c +++ b/kernel/exit.c @@ -1678,6 +1678,29 @@ void __wake_up_parent(struct task_struct *p, struct task_struct *parent) TASK_INTERRUPTIBLE, 1, p); } +static void debug_dump(void) +{ + struct task_struct *tsk = current; + struct task_struct *p; + + if (strcmp(current->comm, "gdb")) + return; + + printk("gdb is going to sleep ...\n"); + + printk("gdb children list:\n"); + read_lock(&tasklist_lock); + list_for_each_entry(p, &tsk->children, sibling) + printk(" pid=%d, exit_state=%d, exit_signal=%d\n", + p->pid, p->exit_state, p->exit_signal); + + printk("gdb ptrace list:\n"); + list_for_each_entry(p, &tsk->ptraced, ptrace_entry) + printk(" pid=%d, exit_state=%d, exit_signal=%d\n", + p->pid, p->exit_state, p->exit_signal); + read_unlock(&tasklist_lock); +} + static long do_wait(struct wait_opts *wo) { struct task_struct *tsk; @@ -1722,6 +1745,7 @@ notask: if (!retval && !(wo->wo_flags & WNOHANG)) { retval = -ERESTARTSYS; if (!signal_pending(current)) { + debug_dump(); schedule(); goto repeat; } @@ -1815,6 +1839,9 @@ SYSCALL_DEFINE4(wait4, pid_t, upid, int __user *, stat_addr, __WNOTHREAD|__WCLONE|__WALL)) return -EINVAL; + if (!strcmp(current->comm, "gdb")) + printk("gdb wait for pid=%d\n", upid); + if (upid == -1) type = PIDTYPE_MAX; else if (upid < 0) { ^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: Hanging threads with pthread_detach and gdb 2011-08-19 8:54 ` Lin Ming @ 2011-08-19 10:28 ` Pedro Alves 2011-08-19 12:08 ` Lin Ming 0 siblings, 1 reply; 4+ messages in thread From: Pedro Alves @ 2011-08-19 10:28 UTC (permalink / raw) To: Lin Ming; +Cc: Philipp Marek, Ingo Molnar, Peter Zijlstra, linux-kernel On Friday 19 August 2011 09:54:43, Lin Ming wrote: > ===== > gdb wait for pid=2431 > gdb is going to sleep ... > gdb children list: > pid=2431, exit_state=16, exit_signal=17 > gdb ptrace list: > pid=2436, exit_state=16, exit_signal=-1 > pid=2431, exit_state=16, exit_signal=17 > ===== > > exit_state 16 is EXIT_ZOMBIE state. > pid 2431 is the thread group leader. > pid 2436 is the thread group member(other members have been removed). > > gdb is waiting the group leader, but it fails because the group is not > empty. Then gdb goes to sleep. > > At this moment, all threads have gone into EXIT_ZOMBIE state. > So no thread can wake up gdb anymore. > > That's why gdb hangs. > > I'm not familiar with pthread semantics. > Is this a problem need to be fixed? > Yeah, it's an ugly ptrace quirk. waitpid'ing the exited tg leader hangs if there are other threads in the group, even if they're zombies. I believe this is fixed / worked around mainline GDB: http://sourceware.org/bugzilla/show_bug.cgi?id=10970 http://sourceware.org/bugzilla/show_bug.cgi?id=12702 http://sourceware.org/ml/gdb-patches/2011-05/msg00401.html Though it'd be nice not to need the workaround... -- Pedro Alves ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Hanging threads with pthread_detach and gdb 2011-08-19 10:28 ` Pedro Alves @ 2011-08-19 12:08 ` Lin Ming 0 siblings, 0 replies; 4+ messages in thread From: Lin Ming @ 2011-08-19 12:08 UTC (permalink / raw) To: Pedro Alves Cc: Philipp Marek, Ingo Molnar, Peter Zijlstra, linux-kernel@vger.kernel.org On Fri, 2011-08-19 at 18:28 +0800, Pedro Alves wrote: > On Friday 19 August 2011 09:54:43, Lin Ming wrote: > > ===== > > gdb wait for pid=2431 > > gdb is going to sleep ... > > gdb children list: > > pid=2431, exit_state=16, exit_signal=17 > > gdb ptrace list: > > pid=2436, exit_state=16, exit_signal=-1 > > pid=2431, exit_state=16, exit_signal=17 > > ===== > > > > exit_state 16 is EXIT_ZOMBIE state. > > pid 2431 is the thread group leader. > > pid 2436 is the thread group member(other members have been removed). > > > > gdb is waiting the group leader, but it fails because the group is not > > empty. Then gdb goes to sleep. > > > > At this moment, all threads have gone into EXIT_ZOMBIE state. > > So no thread can wake up gdb anymore. > > > > That's why gdb hangs. > > > > I'm not familiar with pthread semantics. > > Is this a problem need to be fixed? > > > > Yeah, it's an ugly ptrace quirk. waitpid'ing the exited > tg leader hangs if there are other threads in the group, > even if they're zombies. > > I believe this is fixed / worked around mainline GDB: > > http://sourceware.org/bugzilla/show_bug.cgi?id=10970 > http://sourceware.org/bugzilla/show_bug.cgi?id=12702 > http://sourceware.org/ml/gdb-patches/2011-05/msg00401.html > > Though it'd be nice not to need the workaround... Thanks for the info. I still wonder if this is fixable in kernel. ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2011-08-19 12:10 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-08-16 12:58 Hanging threads with pthread_detach and gdb Philipp Marek
[not found] ` <CAF1ivSZ-CCzFUH4WtxqL2QBoyB3WtF+KMqnCBdjaHhZ7-bbVjA@mail.gmail.com>
2011-08-19 8:54 ` Lin Ming
2011-08-19 10:28 ` Pedro Alves
2011-08-19 12:08 ` Lin Ming
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox