[PATCH] xfs: Fix re-use of EWOULDBLOCK during read on dm-mirror

[PATCH] xfs: Fix re-use of EWOULDBLOCK during read on dm-mirror

[Jeff Mahoney]

When using lvconvert to convert a linear mapping to a dm-raid1 mirror,
we encountered issues where the log would be flooded with messages like:

metadata I/O error: block 0xee7060 ("xfs_trans_read_buf") error 11 numblks 8

The cause is that dm-mirror (and striping, and others) will return
-EWOULDBLOCK for readahead requests while the mirror is rebuilding. XFS's
end_io routine caches the errno and then xfs_buf_iowait bails out early
when it encounters it after issuing the i/o request. The I/O eventually
succeeds and the endio routine resets bp->b_error, but the original read
request has already returned -EWOULDBLOCK to the user and added the log
message above to the kernel log, freaking everyone out.

This patch ignores EWOULDBLOCK when deciding whether to wait for the I/O
to complete and tries again, allowing the read to succeed as expected.

Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
---
 fs/xfs/xfs_buf.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

--- a/fs/xfs/xfs_buf.c
+++ b/fs/xfs/xfs_buf.c
@@ -1252,7 +1252,7 @@ xfs_buf_iowait(
 {
 	trace_xfs_buf_iowait(bp, _RET_IP_);
 
-	if (!bp->b_error)
+	if (!bp->b_error || bp->b_error == EWOULDBLOCK)
 		wait_for_completion(&bp->b_iowait);
 
 	trace_xfs_buf_iowait_done(bp, _RET_IP_);


-- 
Jeff Mahoney
SUSE Labs

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH] xfs: Fix re-use of EWOULDBLOCK during read on dm-mirror

[Dave Chinner]

On Thu, Dec 06, 2012 at 05:39:17PM -0500, Jeff Mahoney wrote:
> When using lvconvert to convert a linear mapping to a dm-raid1 mirror,
> we encountered issues where the log would be flooded with messages like:
> 
> metadata I/O error: block 0xee7060 ("xfs_trans_read_buf") error 11 numblks 8
> 
> The cause is that dm-mirror (and striping, and others) will return
> -EWOULDBLOCK for readahead requests while the mirror is rebuilding.

That's nasty - since when has DM been doing this? I doubt anything
handles a EAGAIN error from the storage layer properly - it's not
an error the filesystem expects from the lower layers at all.

> XFS's
> end_io routine caches the errno and then xfs_buf_iowait bails out early
> when it encounters it after issuing the i/o request.

That doesn't sound right. when XFS issues buffer readahead, it does
not wait for it to complete. i.e. we never get to xfs_buf_iowait()
on readahead buffers.

If something then issues a read on the buffer that failed the
readahead, then we enter xfs_buf_iowait() after reissuing the IO.
If it's aborting because of a stale EWOULDBLOCK as a result of
readahead, then the problem is either:

	- failed readahead should not be leaving an error in
	  b_error; or
	- the read IO did not zero b_error before starting the IO

> The I/O eventually
> succeeds and the endio routine resets bp->b_error,

AFAICT, it's a different IO that succeeds (i.e. the resubmitted one
that is being waited for), not the same one.

> but the original read
> request has already returned -EWOULDBLOCK to the user and added the log
> message above to the kernel log, freaking everyone out.
> 
> This patch ignores EWOULDBLOCK when deciding whether to wait for the I/O
> to complete and tries again, allowing the read to succeed as expected.

Which does not appear to be the correct fix - preventing failed
readahead from leaving a stale error on the buffer seems like the
right thing to do here...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH] xfs: Fix re-use of EWOULDBLOCK during read on dm-mirror

[Dave Chinner]

On Fri, Dec 07, 2012 at 08:53:26PM +1100, Dave Chinner wrote:
> On Thu, Dec 06, 2012 at 05:39:17PM -0500, Jeff Mahoney wrote:
> > When using lvconvert to convert a linear mapping to a dm-raid1 mirror,
> > we encountered issues where the log would be flooded with messages like:
> > 
> > metadata I/O error: block 0xee7060 ("xfs_trans_read_buf") error 11 numblks 8
> > 
> > The cause is that dm-mirror (and striping, and others) will return
> > -EWOULDBLOCK for readahead requests while the mirror is rebuilding.
> 
> That's nasty - since when has DM been doing this? I doubt anything
> handles a EAGAIN error from the storage layer properly - it's not
> an error the filesystem expects from the lower layers at all.

Wow, it's been doing this since at least 2008 - it's taken the
best part of 5 years for someone to trip over this.....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs