Re: xfs umount with i/o error hang/memory corruption

From: Dave Chinner <david@fromorbit.com>
To: Bob Mastors <bob.mastors@solidfire.com>
Cc: xfs@oss.sgi.com
Subject: Re: xfs umount with i/o error hang/memory corruption
Date: Sat, 5 Apr 2014 08:57:12 +1100	[thread overview]
Message-ID: <20140404215712.GP16336@dastard> (raw)
In-Reply-To: <CALjwKZCsBbZ27DE4m=-jN7yWPCqn=QxNgtZZpUcdFvfCPvH5tw@mail.gmail.com>

On Fri, Apr 04, 2014 at 03:40:38PM -0600, Bob Mastors wrote:
> On Fri, Apr 4, 2014 at 3:20 PM, Dave Chinner <david@fromorbit.com> wrote:
> 
> > On Fri, Apr 04, 2014 at 12:15:23PM -0600, Bob Mastors wrote:
> > > Greetings,
> > >
> > > I am new to xfs and am running into a problem
> > > and would appreciate any guidance on how to proceed.
> > >
> > > After an i/o error from the block device that xfs is using,
> > > an umount results in a message like:
> > > [  370.636473] XFS (sdx): Log I/O Error Detected.  Shutting down
> > filesystem
> > > [  370.644073] XFS (h           ���h"h          ���H#h          ���bsg):
> > > Please umount the filesystem and rectify the problem(s)
> > > Note the garbage on the previous line which suggests memory corruption.
> > > About half the time I get the garbled log message. About half the time
> > > umount hangs.
> >
> > I got an email about this last night with a different trigger - thin
> > provisioning failing log IO in the unmount path. I know what the
> > problem is, I just don't have a fix for it yet.
> >
> > To confirm it's the same problem, can you post the entirity of the
> > dmesg where the error occurs?
> >
> Here is the console output from putting a BUG() call  in
> xfs_do_force_shutdown
> after the "Please umount ..." message.
> Let me know if want something else.
> 
> I don't know how much help I can be on this, but I can run more tests
> or try out fixes.

I'll let you know when I have a fix for you to try - probably early
next week as it's Saturday morning right now...

> [  129.548644] XFS (sdb): metadata I/O error: block 0x19022 ("xlog_iodone")
> error 19 numblks 64
> [  129.618739] XFS (sdb): xfs_do_force_shutdown(0x2) called from line 1170
> of file fs/xfs/xfs_log.c.  Return address = 0xffffffffa04c8001
> [  129.699768] XFS (sdb): Log I/O Error Detected.  Shutting down filesystem
> [  129.699924] XFS (sdb): Unable to update superblock counters. Freespace
> may not be correct on next mount.
> [  129.699927] XFS (sdb): xfs_log_force: error 5 returned.
> [  129.716031] XFS (P専/): Please umount the filesystem and rectify the

Yup, that is identical to the bug report I got from the dm-thinp
folks. Thanks for the quick followup!

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs