Re: pdflush hang on xlog_grant_log_space()

[David Chinner <dgc@sgi.com>]

On Thu, Mar 06, 2008 at 04:31:27PM -0500, Kris Kersey wrote:
> Hello,
> 
> I'm working on a NAS product and we're currently having lock-ups that
> seem to be hanging in XFS code.  We're running a NAS that has 1024 NFSD
> threads accessing three RAID mounts.  All three mounts are running XFS
> file systems.  Lately we've had random lockups on these boxes and I am
> now running a kernel with KDB built-in.
> 
> The lock-up takes the form of all NFSD threads in D state with one out
> of three pdflush threads in D state.  The assumption can be made that
> all NFSD threads are waiting on the one pdflush thread to complete.  So
> two times now when an NAS has gotten in this state I have accessed KDB
> and ran a stack trace on the pdflush thread.  Both times the thread was
> stuck on xlog_grant_log_space+0xdb.

Try bumping XFS_TRANS_PUSH_AIL_RESTARTS to a much larger number and
seeing if the problem goes away....

Alternatively, that restart hack is backed by a "watchdog" timeout
in 2.6.25-rc1, so if that is the cause of the problem perhaps the
latest -rcX kernel will prevent the hang?

BTW, you can get all the traces of D state threads through the sysrq
interface, so you don't need to drop into kdb to get this.....

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group