Re: XFS read hangs in 3.1-rc10

[Christoph Hellwig <hch@infradead.org>]

On Thu, Oct 20, 2011 at 03:42:14PM -0700, Simon Kirby wrote:
> 
> [<ffffffff8126f205>] xfs_reclaim_inode+0x85/0x2b0
> [<ffffffff8126f5b0>] xfs_reclaim_inodes_ag+0x180/0x2f0
> [<ffffffff8126f74e>] xfs_reclaim_inodes_nr+0x2e/0x40
> [<ffffffff8126ccf0>] xfs_fs_free_cached_objects+0x10/0x20
> [<ffffffff81119a70>] prune_super+0x110/0x1b0
> [<ffffffff810e4fa5>] shrink_slab+0x1e5/0x2a0
> [<ffffffff810e5821>] kswapd+0x7c1/0xba0
> [<ffffffff8107ada6>] kthread+0x96/0xb0
> [<ffffffff816c0474>] kernel_thread_helper+0x4/0x10
> [<ffffffffffffffff>] 0xffffffffffffffff

We're stuck in synchronous inode reclaim.

> All of the other processes that get stuck have this stack:
> 
> [<ffffffff81080587>] down+0x47/0x50
> [<ffffffff8125e816>] xfs_buf_lock+0x66/0xd0
> [<ffffffff812603ad>] _xfs_buf_find+0x16d/0x270
> [<ffffffff81260517>] xfs_buf_get+0x67/0x1a0
> [<ffffffff8126067a>] xfs_buf_read+0x2a/0x120
> [<ffffffff812b876f>] xfs_trans_read_buf+0x28f/0x3f0
> [<ffffffff8129e161>] xfs_read_agi+0x71/0x100

They are waiting for the AGI buffer to become unlocked.  The only reason
it is held locked for longer time is when it is under I/O.

> 
> By the way, xfs_reclaim_inode+0x85 (133) disassembles as:

> 
> ...So the next function is wait_for_completion(), which is marked
> __sched and thus doesn't show up in the trace.

So we're waiting for the inode to be flushed, aka I/O again.



What is interesting here is that we're always blocking on the AGI
buffer - which is used during unlinks of inodes, and thus gets hit
fairly heavily for a workload that does a lot of unlinks.

> When the clog happens, "iostat -x -k 1" shows no reads from the XFS
> devices, though writes keep happening. "vmstat 1" matches. I tried
> switching schedulers from CFQ to deadline -- no difference. Queue depth
> is empty on the devices and nothing is actually clogged up at the device
> -- it's not actually plugged at the controller or disk. I did a sysreq-w
> while this was happening. About 10 seconds later, everything unclogs and
> continues. Sysreq-W output below. I poked around at the various XFS
> tracepoints in /sys/kernel/debug/tracing/events/xfs, but I'm not sure
> which tracepoints to use and many of them scroll too fast to see
> anything. Any suggestions?

Given that you are doing a lot of unlinks I wonder if it is related
to the recent ail pushing issues in that area.  While your symptoms
looks completely different we could be blocking on the flush completion
for an inode that gets stuck in the AIL.

Can you run with latest 3.0-stable plus the patches at:

	http://oss.sgi.com/pipermail/xfs/2011-October/053464.html

If this doesn't help I'll probably need to come up with some tracing
patches for you.