Shutdown preventing umount

Shutdown preventing umount

[Andi Kleen]

Hi,

I had a cable problem on a USB connected XFS file system, triggering 
some IO errors, and the result was that any access to the mount point
resulted in EIO. This prevented unmounting the file system to recover
from the problem. 

I also tried xfs_io with shutdown -f, but it had the same problem
because xfs_io couldn't access anything on the file system.

How is that supposed to work? Having to reboot just to recover
from IO errors doesn't seem to be very available.

I don't think shutdown should prevent unmounting.

From googling I found some old RHEL bugzilla that such a problem
was fixed in some RHEL release. Is that a regression? 

This was on a 5.4.10 kernel.

I got lots of:

 XFS (...): metadata I/O error in "xfs_trans_read_buf_map" at daddr
 0x4a620288 len 8 error 5

Then some

XFS (...): writeback error on sector 7372099184

And finally:

XFS (...): log I/O error -5
 XFS (...): xfs_do_force_shutdown(0x2)
called from line 1250 of file fs/xfs/xfs_log.c. Return address =
00000000f7956130
XFS (...): Log I/O Error Detected.
Shutting down filesystem
XFS (...): Please unmount the filesystem
and rectify the problem(s)

(very funny XFS!)

XFS (...): log I/O error -5

scsi 7:0:0:0: rejecting I/O to dead device

-Andi

Re: Shutdown preventing umount

[Dave Chinner]

On Sat, Mar 14, 2020 at 06:31:10AM -0700, Andi Kleen wrote:
> 
> Hi,
> 
> I had a cable problem on a USB connected XFS file system, triggering 
> some IO errors, and the result was that any access to the mount point
> resulted in EIO. This prevented unmounting the file system to recover
> from the problem. 

Full dmesg output, please.

> I also tried xfs_io with shutdown -f, but it had the same problem
> because xfs_io couldn't access anything on the file system.

Because the IO errors had already shut the filesystem down, as
per the dmesg output you quoted below.

> How is that supposed to work? Having to reboot just to recover
> from IO errors doesn't seem to be very available.
> 
> I don't think shutdown should prevent unmounting.

It doesn't. Something was leaked or not cleaned up properly,
preventing the filesytem from being unmounted. You know, a bug...

> From googling I found some old RHEL bugzilla that such a problem
> was fixed in some RHEL release. Is that a regression? 

RHEL has an upstream first policy, so whatever bug fix you find in
a RHEL kernel is already in the upstream kernels.

> This was on a 5.4.10 kernel.
> 
> I got lots of:
> 
>  XFS (...): metadata I/O error in "xfs_trans_read_buf_map" at daddr
>  0x4a620288 len 8 error 5
> 
> Then some
> 
> XFS (...): writeback error on sector 7372099184
> 
> And finally:
> 
> XFS (...): log I/O error -5
>  XFS (...): xfs_do_force_shutdown(0x2)
> called from line 1250 of file fs/xfs/xfs_log.c. Return address =
> 00000000f7956130
> XFS (...): Log I/O Error Detected.
> Shutting down filesystem
> XFS (...): Please unmount the filesystem
> and rectify the problem(s)
> 
> (very funny XFS!)
> 
> XFS (...): log I/O error -5
> 
> scsi 7:0:0:0: rejecting I/O to dead device

Where is unmount stuck? 'echo w > /proc/sysrq-trigger' output if it
is hung, 'echo l > /proc/sysrq-trigger' if it is spinning, please?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Shutdown preventing umount

[Andi Kleen]

On Mon, Mar 16, 2020 at 01:03:42PM +1100, Dave Chinner wrote:
> On Sat, Mar 14, 2020 at 06:31:10AM -0700, Andi Kleen wrote:
> > 
> > Hi,
> > 
> > I had a cable problem on a USB connected XFS file system, triggering 
> > some IO errors, and the result was that any access to the mount point
> > resulted in EIO. This prevented unmounting the file system to recover
> > from the problem. 
> 
> Full dmesg output, please.

http://www.firstfloor.org/~andi/dmesg-xfs-umount

> > 
> > XFS (...): log I/O error -5
> > 
> > scsi 7:0:0:0: rejecting I/O to dead device
> 
> Where is unmount stuck? 'echo w > /proc/sysrq-trigger' output if it
> is hung, 'echo l > /proc/sysrq-trigger' if it is spinning, please?

It's not stuck. It always errored out with EIO.

-Andi

Re: Shutdown preventing umount

[Dave Chinner]

On Sun, Mar 15, 2020 at 08:37:19PM -0700, Andi Kleen wrote:
> On Mon, Mar 16, 2020 at 01:03:42PM +1100, Dave Chinner wrote:
> > On Sat, Mar 14, 2020 at 06:31:10AM -0700, Andi Kleen wrote:
> > > 
> > > Hi,
> > > 
> > > I had a cable problem on a USB connected XFS file system, triggering 
> > > some IO errors, and the result was that any access to the mount point
> > > resulted in EIO. This prevented unmounting the file system to recover
> > > from the problem. 
> > 
> > Full dmesg output, please.
> 
> http://www.firstfloor.org/~andi/dmesg-xfs-umount

So /dev/sdd went away, XFS shutdown eventually. There's no
indication that an XFS unmount has started, because the first thing
XFS does in it's ->put_super method is emit this to the log:

XFS (dm-4): Unmounting Filesystem

and that is missing. Hence the VFS unmount path has not reached as
far as XFS before it has errorred out. THis is confirmed by....

> > > 
> > > XFS (...): log I/O error -5
> > > 
> > > scsi 7:0:0:0: rejecting I/O to dead device
> > 
> > Where is unmount stuck? 'echo w > /proc/sysrq-trigger' output if it
> > is hung, 'echo l > /proc/sysrq-trigger' if it is spinning, please?
> 
> It's not stuck. It always errored out with EIO.

... the fact that filesystems cannot return errors from unmount
proceedings as generic_shutdown_super() is a void function.  Hence
where this EIO is coming from is not obvious - it isn't from XFS
failing to unmount the filesystem as it's not gettting that far into
the unmount path.

You're going to need to strace umount to find what syscall is
failing, then probably use tracepoints or some one kernel
introspection tool to work out where the EIO is coming from...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com