Re: something changed from 4 to 5

[Ian Kent <raven@themaw.net>]

Joe Pruett wrote:
>> Took me a while to work out what you had done here.
>> To get the backtrace you need only connect to the main thread (in 
>> this case 1741) and do the "thr a a bt".
>> That gives backtrace info for all the sub-threads at once, which is 
>> what we need.
>
> this is a centos 5 box so the other pids from ps are standalone 
> processes (look at the lwp list from the first gdb to see that they 
> aren't the same).
>
> the first gdb should have been the data you want. by connecting to the 
> other pids, it eventually unblocks whatever is stuck and then the 
> world starts working again. oddly, it seems to be a call to access 
> that is hung. in the past i've just used strace and could never tell 
> what was stuck.

And the call to access(2) is seen in the first backtrace as waiting on 
an nfs mount and any other processes calling to access(2) should wait on 
the first in this case, since the path includes the same mount.

When this happens you could also try looking for mount processes to 
match them to those in the backtraces but if any have died that should 
result in the daemon continuing. So it's a bit puzzling.

Ian