Re: Futex hang/lockup problem in 2.6.30+ on AMD64

[Andrew Athan <aathan@memeplex.com>]

Darren Hart wrote:
> Andrew Athan wrote:
>> Andrew Athan wrote:
>>> Américo Wang wrote:
>>>> On Tue, Jan 12, 2010 at 10:55 PM, Peter Zijlstra 
>>>> <peterz@infradead.org> wrote:
>>>>  
>>>>> On Tue, 2010-01-12 at 22:52 +0800, Américo Wang wrote:
>>>>>
>>>>>  
>>>>>>> $ uname -a
>>>>>>> Linux UK22 2.6.30-2-amd64 #1 SMP Fri Sep 25 22:16:56 UTC 2009 
>>>>>>> x86_64
>>>>>>> GNU/Linux
>>>>>>>         
>>>>> Does a recent kernel work?
>>>>>
>>>>>
>>>>>     
>>>>
>>>> Ah, I just wanted to ask the same question, adding the original 
>>>> reporter
>>>> Gong Cheng into Cc...
>>>>
>>>> Gong, could you reproduce it on the latest kernel? And what is your 
>>>> .config?
>>>>
>>>> Thanks!
>>>> -- 
>>>> To unsubscribe from this list: send the line "unsubscribe 
>>>> linux-kernel" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>> Please read the FAQ at  http://www.tux.org/lkml/
>>>>   
>>> Due to remote location of the hardware and I haven't been able to 
>>> test a more recent (or older) kernel.  Remote hands have put a KVM 
>>> on the box as of an hour ago, so I hope to have some information for 
>>> you in a day or two.
>>>
>>> A.
>>>
>>
>>
>> I wanted to report that although I have had no luck (so far) running 
>> anything more recent than 2.6.30, I was able to revert to 2.6.26.  
>> Unfortunately, the application hang still occurs.  I also saw a 
>> similar hang of the application running on a 32 bit Intel box, also 
>> under 2.6.26.  So far, the hang *always* involves threads stuck on 
>> pthread_cond_broadcast()'s  condition variable's internal lock while 
>> other threads are waiting on the outer "public" lock.
>
>
> Are you using real-time scheduling policy or priority inheritance 
> (PTHREAD_PRIO_INHERIT)? It is possible to suffer an unbounded priority 
> inversion on the internal condvar data lock in the current distro 
> implementations of glibc.
>
>
>> These other threads are *not* yet (nor about to) 
>> pthread_cond_wait().  I saw a message from Darren Hart (subject "Re: 
>> Problems with futex") in response to someone who apparently was 
>> having futex problems in 2.6.27, so I'm still operating under the 
>> assumption that this is not an application bug.
>
> Those all turned out to be application issues with one exception which 
> had already been fixed upstream.
>
>
>> Over the next couple of days, I will be running a version of the 
>> application in which I replaced the pthread_cond calls with simpler 
>> locks, in the hopes that it won't hang (because I'm hoping the 
>> underlying implementation in pthreads uses a different set of futex 
>> opcodes).
>>
>> Andrew Athan
>>
>
>

I wanted to report that this application hang is certainly related to 
pthread_cond_* calls.  With them in place, it consistently hangs.  
Without, it consistently does not.  Whether pthread_cond_* is 
misbehaving due to memory corruption or another application bug I 
suppose is an open question.

We have now experienced several lockups where even a kill -9 of the 
application won't get rid of it.  Does this say anything about the 
nature of the hang?

By the way, majordomo stopped sending me emails as of 1/17 so I have not 
seen any updates to this thread sent after this date.  Not sure why this 
happened, as I never asked to be unsubscribed.  I've resubscribed, but 
not sure I will get anything.  Please make sure I am directly cc:ed on 
any responses.

carlinux138:~# uname -a
Linux carlinux138.thinktradellc.com 2.6.26-2-686 #1 SMP Sun Jun 21 
04:57:38 UTC 2009 i686 GNU/Linux

(I have to go look up what the best way to give a system config snapshot 
is, e.g., all major library version etc ... )

Thanks,
Andrew Athan