recv() hangs until SIGCHLD ?

[Nicolas Cannasse <ncannasse@motion-twin.com>]

Hi,

We've been tracking a bug in our server application for some time now, 
and now that we could isolate it we're stuck without a meaningful 
explanation. Hope somehow would be able to give use some answers.

We run a multithread application which is using pthreads and sockets. A 
thread uses accept() then dispatch the socket to one of the workers 
threads that process it. Sockets are then not used simultaneously by 
several threads.

In some rare cases, one (or several) threads are hanging in recv(). Both 
lsof and ls /proc/<pid>/fd show that the socket used is in ESTABLISHED 
mode but when checking on the host on which it's connected (a mysql DB) 
we can't find the corresponding client socket (as it's been closed 
already on the other side).

We are using the Boehm GC which uses the signals SIGXCPU and SIGPWR to 
pause+restart the threads when running a GC cycle. We are correctly 
handling EINTR in send() and recv() by restarting the call in case they 
get interrupted this way.

However, when attaching GDB to our locked thread it seems that even when 
the GC runs, recv() does not exit (the breakpoint after it is not 
reached). If we send SIGCHLD to the hanging thread with GDB, recv() does 
exit and the thread is correctly unlocked. If we don't, it will hang 
forever.

Additional details : recv() is using MSG_NOSIGNAL and we have enabled 
TCP_NODELAY on the socket by using setsockopt. Some other 
not-multithreaded apps are using the same Databases and this behavior 
does not occur for them.

Any idea how we can stop this from happening or what additional things 
we can check to get more informations on what's occurring ?

Thanks a lot,
Nicolas