* [PATCH] xfs: Fix re-use of EWOULDBLOCK during read on dm-mirror
@ 2012-12-06 22:39 Jeff Mahoney
2012-12-07 9:53 ` Dave Chinner
0 siblings, 1 reply; 3+ messages in thread
From: Jeff Mahoney @ 2012-12-06 22:39 UTC (permalink / raw)
To: xfs
When using lvconvert to convert a linear mapping to a dm-raid1 mirror,
we encountered issues where the log would be flooded with messages like:
metadata I/O error: block 0xee7060 ("xfs_trans_read_buf") error 11 numblks 8
The cause is that dm-mirror (and striping, and others) will return
-EWOULDBLOCK for readahead requests while the mirror is rebuilding. XFS's
end_io routine caches the errno and then xfs_buf_iowait bails out early
when it encounters it after issuing the i/o request. The I/O eventually
succeeds and the endio routine resets bp->b_error, but the original read
request has already returned -EWOULDBLOCK to the user and added the log
message above to the kernel log, freaking everyone out.
This patch ignores EWOULDBLOCK when deciding whether to wait for the I/O
to complete and tries again, allowing the read to succeed as expected.
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
---
fs/xfs/xfs_buf.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--- a/fs/xfs/xfs_buf.c
+++ b/fs/xfs/xfs_buf.c
@@ -1252,7 +1252,7 @@ xfs_buf_iowait(
{
trace_xfs_buf_iowait(bp, _RET_IP_);
- if (!bp->b_error)
+ if (!bp->b_error || bp->b_error == EWOULDBLOCK)
wait_for_completion(&bp->b_iowait);
trace_xfs_buf_iowait_done(bp, _RET_IP_);
--
Jeff Mahoney
SUSE Labs
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 3+ messages in thread* Re: [PATCH] xfs: Fix re-use of EWOULDBLOCK during read on dm-mirror
2012-12-06 22:39 [PATCH] xfs: Fix re-use of EWOULDBLOCK during read on dm-mirror Jeff Mahoney
@ 2012-12-07 9:53 ` Dave Chinner
2012-12-10 1:12 ` Dave Chinner
0 siblings, 1 reply; 3+ messages in thread
From: Dave Chinner @ 2012-12-07 9:53 UTC (permalink / raw)
To: Jeff Mahoney; +Cc: xfs
On Thu, Dec 06, 2012 at 05:39:17PM -0500, Jeff Mahoney wrote:
> When using lvconvert to convert a linear mapping to a dm-raid1 mirror,
> we encountered issues where the log would be flooded with messages like:
>
> metadata I/O error: block 0xee7060 ("xfs_trans_read_buf") error 11 numblks 8
>
> The cause is that dm-mirror (and striping, and others) will return
> -EWOULDBLOCK for readahead requests while the mirror is rebuilding.
That's nasty - since when has DM been doing this? I doubt anything
handles a EAGAIN error from the storage layer properly - it's not
an error the filesystem expects from the lower layers at all.
> XFS's
> end_io routine caches the errno and then xfs_buf_iowait bails out early
> when it encounters it after issuing the i/o request.
That doesn't sound right. when XFS issues buffer readahead, it does
not wait for it to complete. i.e. we never get to xfs_buf_iowait()
on readahead buffers.
If something then issues a read on the buffer that failed the
readahead, then we enter xfs_buf_iowait() after reissuing the IO.
If it's aborting because of a stale EWOULDBLOCK as a result of
readahead, then the problem is either:
- failed readahead should not be leaving an error in
b_error; or
- the read IO did not zero b_error before starting the IO
> The I/O eventually
> succeeds and the endio routine resets bp->b_error,
AFAICT, it's a different IO that succeeds (i.e. the resubmitted one
that is being waited for), not the same one.
> but the original read
> request has already returned -EWOULDBLOCK to the user and added the log
> message above to the kernel log, freaking everyone out.
>
> This patch ignores EWOULDBLOCK when deciding whether to wait for the I/O
> to complete and tries again, allowing the read to succeed as expected.
Which does not appear to be the correct fix - preventing failed
readahead from leaving a stale error on the buffer seems like the
right thing to do here...
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 3+ messages in thread* Re: [PATCH] xfs: Fix re-use of EWOULDBLOCK during read on dm-mirror
2012-12-07 9:53 ` Dave Chinner
@ 2012-12-10 1:12 ` Dave Chinner
0 siblings, 0 replies; 3+ messages in thread
From: Dave Chinner @ 2012-12-10 1:12 UTC (permalink / raw)
To: Jeff Mahoney; +Cc: xfs
On Fri, Dec 07, 2012 at 08:53:26PM +1100, Dave Chinner wrote:
> On Thu, Dec 06, 2012 at 05:39:17PM -0500, Jeff Mahoney wrote:
> > When using lvconvert to convert a linear mapping to a dm-raid1 mirror,
> > we encountered issues where the log would be flooded with messages like:
> >
> > metadata I/O error: block 0xee7060 ("xfs_trans_read_buf") error 11 numblks 8
> >
> > The cause is that dm-mirror (and striping, and others) will return
> > -EWOULDBLOCK for readahead requests while the mirror is rebuilding.
>
> That's nasty - since when has DM been doing this? I doubt anything
> handles a EAGAIN error from the storage layer properly - it's not
> an error the filesystem expects from the lower layers at all.
Wow, it's been doing this since at least 2008 - it's taken the
best part of 5 years for someone to trip over this.....
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2012-12-10 1:09 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-12-06 22:39 [PATCH] xfs: Fix re-use of EWOULDBLOCK during read on dm-mirror Jeff Mahoney
2012-12-07 9:53 ` Dave Chinner
2012-12-10 1:12 ` Dave Chinner
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox