Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery

[Dave Chinner <david@fromorbit.com>]

On Thu, Mar 24, 2016 at 05:52:44PM +0100, Carlos Maiolino wrote:
> I can now reproduce it, or at least part of the problem.
> 
> Regarding your question Dave, yes, it can be unmounted after I issue xfs_io shutdown
> command. But, if a umount is issued before that, then we can't find the
> mountpoint anymore.
> 
> I'm not sure if I'm correct, but, what it looks like to me, as you already
> mentioned, is that we keep getting IO errors but we never actually shutdown
> the filesystem while doing async metadata writes.

*nod*

> I believe I've found the problem. So, I will try to explain it, so you guys
> can review and let me know if I'm right or not
> 
> I was looking the code, and for me, looks like async retries are designed to
> keep retrying forever, and rely on some other part of the filesystem to actually
> shutdown it.

*nod*

[snip description of metadata IO error behaviour]

Yes, that is exactly how the code is expected to behave - in fact,
that's how it was originally designed to behave.

> Looks like, somebody already noticed it:
> 
>         /*
>         ¦* If the write was asynchronous then no one will be looking for the
>         ¦* error.  Clear the error state and write the buffer out again.
>         ¦*
>         ¦* XXX: This helps against transient write errors, but we need to find
>         ¦* a way to shut the filesystem down if the writes keep failing.
>         ¦*
>         ¦* In practice we'll shut the filesystem down soon as non-transient
>         ¦* errors tend to affect the whole device and a failing log write
>         ¦* will make us give up.  But we really ought to do better here.
>         ¦*/
> 
> 
> So, if I'm write in how we hit this problem, and IIRC, Dave's patchset for
> setting limits to IO errors can be slightly modified to fix this issue too, but,

The patchset I have doesn't need modification to fix this issue - it
has a patch specifically to address this, and it changes the default
behaviour to "fail async writes at unmount":

http://oss.sgi.com/archives/xfs/2015-08/msg00092.html

> the problem is that the user must set it BEFORE he tries to unmount the
> filesystem, otherwise it will get stuck here.

Yes, but that doesn't answer the big question: why don't the
periodic log forces that are failing with EIO cause a filesystem
shutdown? We issue a log force every 30s even during unmount, and a
failed log IO must cause the filesystem to shut down. So why aren't
these causing the filesystem to shutdown as we'd expect when the
device has been pulled?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs