Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery

From: Dave Chinner <david@fromorbit.com>
To: Shyam Kaushik <shyam@zadarastorage.com>
Cc: Brian Foster <bfoster@redhat.com>,
	Alex Lyakas <alex@zadarastorage.com>,
	xfs@oss.sgi.com
Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery
Date: Sat, 9 Apr 2016 08:46:48 +1000	[thread overview]
Message-ID: <20160408224648.GD567@dastard> (raw)
In-Reply-To: <232dd85fde11d4ef1625f070eabfd167@mail.gmail.com>

On Fri, Apr 08, 2016 at 04:21:02PM +0530, Shyam Kaushik wrote:
> Hi Dave, Brian, Carlos,
> 
> While trying to reproduce this issue I have been running into different
> issues that are similar. Underlying issue remains the same when backend to
> XFS is failed & we unmount XFS, we run into hung-task timeout (180-secs)
> with stack like
> 
> kernel: [14952.671131]  [<ffffffffc06a5f59>]
> xfs_ail_push_all_sync+0xa9/0xe0 [xfs]
> kernel: [14952.671139]  [<ffffffff810b26b0>] ?
> prepare_to_wait_event+0x110/0x110
> kernel: [14952.671181]  [<ffffffffc0690111>] xfs_unmountfs+0x61/0x1a0
> [xfs]
> 
> while running trace-events, XFS ail push keeps looping around
> 
>    xfsaild/dm-10-21143 [001] ...2 17878.555133: xfs_ilock_nowait: dev
> 253:10 ino 0x0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]

Looks like either a stale inode (which should never reach the AIL)
or it's an inode that's been reclaimed and this is a use after free
situation. Given that we are failing IOs here, I'd suggest it's more
likely to be an IO failure that's caused a writeback problem, not an
interaction with stale inodes.

So, look at xfs_iflush. If an IO fails, it is supposed to unlock the
inode by calling xfs_iflush_abort(), which will also remove it from
the AIL. This can also happen on reclaim of a dirty inode, and if so
we'll still reclaim the inode because reclaim assumes xfs_iflush()
cleans up properly.

Which, apparently, it doesn't:

        /*
         * Get the buffer containing the on-disk inode.
         */
        error = xfs_imap_to_bp(mp, NULL, &ip->i_imap, &dip, &bp, XBF_TRYLOCK, 0);
        if (error || !bp) {
                xfs_ifunlock(ip);
                return error;
        }

This looks like a bug - xfs_iflush hasn't aborted the inode
writeback on failure - it's just unlocked the flush lock. Hence it
has left the inode dirty in the AIL, and then the inode has probably
then been reclaimed, setting the inode number to zero.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs