[PATCH] fix barrier fail detection

[PATCH] fix barrier fail detection

[Christoph Hellwig]

Currently we disable barriers as soon as we get a buffer in xlog_iodone
that has the XBF_ORDERED flag cleared.  But this can be the case not only
for buffers where the barrier failed, but also the first buffer of a
split log write in case of a log wraparound.  Due to the disabled
barriers we can easily get directory corruption on unclean shutdowns.
So instead of using this check add a new buffer flag for failed barrier
writes.

This is a regression vs 2.6.26 caused by patch to use the right macro
to check for the ORDERED flag, as we previously got true returned for
every buffer.

Thanks to Toei Rei for reporting the bug.


Signed-off-by: Christoph Hellwig <hch@lst.de>

Index: linux-2.6-xfs/fs/xfs/linux-2.6/xfs_buf.c
===================================================================
--- linux-2.6-xfs.orig/fs/xfs/linux-2.6/xfs_buf.c	2008-10-09 13:36:50.000000000 +0200
+++ linux-2.6-xfs/fs/xfs/linux-2.6/xfs_buf.c	2008-10-09 13:38:38.000000000 +0200
@@ -1007,6 +1007,7 @@ xfs_buf_iodone_work(
 	    (bp->b_flags & (XBF_ORDERED|XBF_ASYNC)) == (XBF_ORDERED|XBF_ASYNC)) {
 		XB_TRACE(bp, "ordered_retry", bp->b_iodone);
 		bp->b_flags &= ~XBF_ORDERED;
+		bp->b_flags |= _XFS_BARRIER_FAILED;
 		xfs_buf_iorequest(bp);
 	} else if (bp->b_iodone)
 		(*(bp->b_iodone))(bp);
Index: linux-2.6-xfs/fs/xfs/linux-2.6/xfs_buf.h
===================================================================
--- linux-2.6-xfs.orig/fs/xfs/linux-2.6/xfs_buf.h	2008-10-09 13:36:50.000000000 +0200
+++ linux-2.6-xfs/fs/xfs/linux-2.6/xfs_buf.h	2008-10-09 13:38:15.000000000 +0200
@@ -85,6 +85,14 @@ typedef enum {
 	 * modifications being lost.
 	 */
 	_XBF_PAGE_LOCKED = (1 << 22),
+
+	/*
+	 * If we try a barrier write, but it fails we have to communicate
+	 * this to the upper layers.  Unfortunately b_error gets overwritten
+	 * when the buffer is re-issued so we have to add another flag to
+	 * keep this information.
+	 */
+	_XFS_BARRIER_FAILED = (1 << 23),
 } xfs_buf_flags_t;
 
 typedef enum {
Index: linux-2.6-xfs/fs/xfs/xfs_log.c
===================================================================
--- linux-2.6-xfs.orig/fs/xfs/xfs_log.c	2008-10-09 13:38:44.000000000 +0200
+++ linux-2.6-xfs/fs/xfs/xfs_log.c	2008-10-09 13:39:32.000000000 +0200
@@ -1037,7 +1037,8 @@ xlog_iodone(xfs_buf_t *bp)
 	 * layer, it means the underlyin device no longer supports
 	 * barrier I/O. Warn loudly and turn off barriers.
 	 */
-	if ((l->l_mp->m_flags & XFS_MOUNT_BARRIER) && !XFS_BUF_ISORDERED(bp)) {
+	if (bp->b_flags & _XFS_BARRIER_FAILED) {
+		bp->b_flags &= ~_XFS_BARRIER_FAILED;
 		l->l_mp->m_flags &= ~XFS_MOUNT_BARRIER;
 		xfs_fs_cmn_err(CE_WARN, l->l_mp,
 				"xlog_iodone: Barriers are no longer supported"

Re: [PATCH] fix barrier fail detection

[Eric Sandeen]

Christoph Hellwig wrote:
> Currently we disable barriers as soon as we get a buffer in xlog_iodone
> that has the XBF_ORDERED flag cleared.  But this can be the case not only
> for buffers where the barrier failed, but also the first buffer of a
> split log write in case of a log wraparound.  Due to the disabled
> barriers we can easily get directory corruption on unclean shutdowns.
> So instead of using this check add a new buffer flag for failed barrier
> writes.
> 
> This is a regression vs 2.6.26 caused by patch to use the right macro
> to check for the ORDERED flag, as we previously got true returned for
> every buffer.
> 
> Thanks to Toei Rei for reporting the bug.

Fix looks good to me (I saw this failure too but at first attributed it
to some hackery I was working on).

Think it can still make 2.6.27?

-Eric

> 
> Signed-off-by: Christoph Hellwig <hch@lst.de>
> 
> Index: linux-2.6-xfs/fs/xfs/linux-2.6/xfs_buf.c
> ===================================================================
> --- linux-2.6-xfs.orig/fs/xfs/linux-2.6/xfs_buf.c	2008-10-09 13:36:50.000000000 +0200
> +++ linux-2.6-xfs/fs/xfs/linux-2.6/xfs_buf.c	2008-10-09 13:38:38.000000000 +0200
> @@ -1007,6 +1007,7 @@ xfs_buf_iodone_work(
>  	    (bp->b_flags & (XBF_ORDERED|XBF_ASYNC)) == (XBF_ORDERED|XBF_ASYNC)) {
>  		XB_TRACE(bp, "ordered_retry", bp->b_iodone);
>  		bp->b_flags &= ~XBF_ORDERED;
> +		bp->b_flags |= _XFS_BARRIER_FAILED;
>  		xfs_buf_iorequest(bp);
>  	} else if (bp->b_iodone)
>  		(*(bp->b_iodone))(bp);
> Index: linux-2.6-xfs/fs/xfs/linux-2.6/xfs_buf.h
> ===================================================================
> --- linux-2.6-xfs.orig/fs/xfs/linux-2.6/xfs_buf.h	2008-10-09 13:36:50.000000000 +0200
> +++ linux-2.6-xfs/fs/xfs/linux-2.6/xfs_buf.h	2008-10-09 13:38:15.000000000 +0200
> @@ -85,6 +85,14 @@ typedef enum {
>  	 * modifications being lost.
>  	 */
>  	_XBF_PAGE_LOCKED = (1 << 22),
> +
> +	/*
> +	 * If we try a barrier write, but it fails we have to communicate
> +	 * this to the upper layers.  Unfortunately b_error gets overwritten
> +	 * when the buffer is re-issued so we have to add another flag to
> +	 * keep this information.
> +	 */
> +	_XFS_BARRIER_FAILED = (1 << 23),
>  } xfs_buf_flags_t;
>  
>  typedef enum {
> Index: linux-2.6-xfs/fs/xfs/xfs_log.c
> ===================================================================
> --- linux-2.6-xfs.orig/fs/xfs/xfs_log.c	2008-10-09 13:38:44.000000000 +0200
> +++ linux-2.6-xfs/fs/xfs/xfs_log.c	2008-10-09 13:39:32.000000000 +0200
> @@ -1037,7 +1037,8 @@ xlog_iodone(xfs_buf_t *bp)
>  	 * layer, it means the underlyin device no longer supports
>  	 * barrier I/O. Warn loudly and turn off barriers.
>  	 */
> -	if ((l->l_mp->m_flags & XFS_MOUNT_BARRIER) && !XFS_BUF_ISORDERED(bp)) {
> +	if (bp->b_flags & _XFS_BARRIER_FAILED) {
> +		bp->b_flags &= ~_XFS_BARRIER_FAILED;
>  		l->l_mp->m_flags &= ~XFS_MOUNT_BARRIER;
>  		xfs_fs_cmn_err(CE_WARN, l->l_mp,
>  				"xlog_iodone: Barriers are no longer supported"
> 
> 

Re: [PATCH] fix barrier fail detection

[Dave Chinner]

On Thu, Oct 09, 2008 at 03:00:42PM +0200, Christoph Hellwig wrote:
> Currently we disable barriers as soon as we get a buffer in xlog_iodone
> that has the XBF_ORDERED flag cleared.  But this can be the case not only
> for buffers where the barrier failed, but also the first buffer of a
> split log write in case of a log wraparound.  Due to the disabled
> barriers we can easily get directory corruption on unclean shutdowns.
> So instead of using this check add a new buffer flag for failed barrier
> writes.
> 
> This is a regression vs 2.6.26 caused by patch to use the right macro
> to check for the ORDERED flag, as we previously got true returned for
> every buffer.
> 
> Thanks to Toei Rei for reporting the bug.

Looks good. Should this be sent straight to Linus to get it into
.27? If you want to do that, you can add my SOB.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [PATCH] fix barrier fail detection

[Christoph Hellwig]

On Fri, Oct 10, 2008 at 09:35:34AM +1100, Dave Chinner wrote:
> Looks good. Should this be sent straight to Linus to get it into
> .27? If you want to do that, you can add my SOB.

I'll hope Lachlan or Tim as maintainers will pick up..

Re: [PATCH] fix barrier fail detection

[Timothy Shimmin]

Christoph Hellwig wrote:
> Currently we disable barriers as soon as we get a buffer in xlog_iodone
> that has the XBF_ORDERED flag cleared.  But this can be the case not only
> for buffers where the barrier failed, but also the first buffer of a
> split log write in case of a log wraparound.  
Oops.


> Due to the disabled
> barriers we can easily get directory corruption on unclean shutdowns.
> So instead of using this check add a new buffer flag for failed barrier
> writes.
> 
Looks good and I'll check it in to ptools.

Thanks,
Tim.

> This is a regression vs 2.6.26 caused by patch to use the right macro
> to check for the ORDERED flag, as we previously got true returned for
> every buffer.
> 
> Thanks to Toei Rei for reporting the bug.
> 
> 
> Signed-off-by: Christoph Hellwig <hch@lst.de>
> 
> Index: linux-2.6-xfs/fs/xfs/linux-2.6/xfs_buf.c
> ===================================================================
> --- linux-2.6-xfs.orig/fs/xfs/linux-2.6/xfs_buf.c	2008-10-09 13:36:50.000000000 +0200
> +++ linux-2.6-xfs/fs/xfs/linux-2.6/xfs_buf.c	2008-10-09 13:38:38.000000000 +0200
> @@ -1007,6 +1007,7 @@ xfs_buf_iodone_work(
>  	    (bp->b_flags & (XBF_ORDERED|XBF_ASYNC)) == (XBF_ORDERED|XBF_ASYNC)) {
>  		XB_TRACE(bp, "ordered_retry", bp->b_iodone);
>  		bp->b_flags &= ~XBF_ORDERED;
> +		bp->b_flags |= _XFS_BARRIER_FAILED;
>  		xfs_buf_iorequest(bp);
>  	} else if (bp->b_iodone)
>  		(*(bp->b_iodone))(bp);
> Index: linux-2.6-xfs/fs/xfs/linux-2.6/xfs_buf.h
> ===================================================================
> --- linux-2.6-xfs.orig/fs/xfs/linux-2.6/xfs_buf.h	2008-10-09 13:36:50.000000000 +0200
> +++ linux-2.6-xfs/fs/xfs/linux-2.6/xfs_buf.h	2008-10-09 13:38:15.000000000 +0200
> @@ -85,6 +85,14 @@ typedef enum {
>  	 * modifications being lost.
>  	 */
>  	_XBF_PAGE_LOCKED = (1 << 22),
> +
> +	/*
> +	 * If we try a barrier write, but it fails we have to communicate
> +	 * this to the upper layers.  Unfortunately b_error gets overwritten
> +	 * when the buffer is re-issued so we have to add another flag to
> +	 * keep this information.
> +	 */
> +	_XFS_BARRIER_FAILED = (1 << 23),
>  } xfs_buf_flags_t;
>  
>  typedef enum {
> Index: linux-2.6-xfs/fs/xfs/xfs_log.c
> ===================================================================
> --- linux-2.6-xfs.orig/fs/xfs/xfs_log.c	2008-10-09 13:38:44.000000000 +0200
> +++ linux-2.6-xfs/fs/xfs/xfs_log.c	2008-10-09 13:39:32.000000000 +0200
> @@ -1037,7 +1037,8 @@ xlog_iodone(xfs_buf_t *bp)
>  	 * layer, it means the underlyin device no longer supports
>  	 * barrier I/O. Warn loudly and turn off barriers.
>  	 */
> -	if ((l->l_mp->m_flags & XFS_MOUNT_BARRIER) && !XFS_BUF_ISORDERED(bp)) {
> +	if (bp->b_flags & _XFS_BARRIER_FAILED) {
> +		bp->b_flags &= ~_XFS_BARRIER_FAILED;
>  		l->l_mp->m_flags &= ~XFS_MOUNT_BARRIER;
>  		xfs_fs_cmn_err(CE_WARN, l->l_mp,
>  				"xlog_iodone: Barriers are no longer supported"
> 

Re: [PATCH] fix barrier fail detection

[Timothy Shimmin]

Christoph Hellwig wrote:
> Currently we disable barriers as soon as we get a buffer in xlog_iodone
> that has the XBF_ORDERED flag cleared.  But this can be the case not only
> for buffers where the barrier failed, but also the first buffer of a
> split log write in case of a log wraparound.  Due to the disabled
> barriers we can easily get directory corruption on unclean shutdowns.
> So instead of using this check add a new buffer flag for failed barrier
> writes.
> 
> This is a regression vs 2.6.26 caused by patch to use the right macro
> to check for the ORDERED flag, as we previously got true returned for
> every buffer.
> 
> Thanks to Toei Rei for reporting the bug.
> 
> 
> Signed-off-by: Christoph Hellwig <hch@lst.de>
> 
> Index: linux-2.6-xfs/fs/xfs/linux-2.6/xfs_buf.c
> ===================================================================
> --- linux-2.6-xfs.orig/fs/xfs/linux-2.6/xfs_buf.c	2008-10-09 13:36:50.000000000 +0200
> +++ linux-2.6-xfs/fs/xfs/linux-2.6/xfs_buf.c	2008-10-09 13:38:38.000000000 +0200
> @@ -1007,6 +1007,7 @@ xfs_buf_iodone_work(
>  	    (bp->b_flags & (XBF_ORDERED|XBF_ASYNC)) == (XBF_ORDERED|XBF_ASYNC)) {
>  		XB_TRACE(bp, "ordered_retry", bp->b_iodone);
>  		bp->b_flags &= ~XBF_ORDERED;
> +		bp->b_flags |= _XFS_BARRIER_FAILED;
>  		xfs_buf_iorequest(bp);
>  	} else if (bp->b_iodone)
>  		(*(bp->b_iodone))(bp);

Actually, probably need to update the comment for this one.

The existing comment being:
> 	/*
> 	 * We can get an EOPNOTSUPP to ordered writes.  Here we clear the
> 	 * ordered flag and reissue them.  Because we can't tell the higher
> 	 * layers directly that they should not issue ordered I/O anymore, they
> 	 * need to check if the ordered flag was cleared during I/O completion.
> 	 */
> 	if ((bp->b_error == EOPNOTSUPP) &&
> 	    (bp->b_flags & (XBF_ORDERED|XBF_ASYNC)) == (XBF_ORDERED|XBF_ASYNC)) {
> 		XB_TRACE(bp, "ordered_retry", bp->b_iodone);
> 		bp->b_flags &= ~XBF_ORDERED;
> 		bp->b_flags |= _XFS_BARRIER_FAILED;


> Index: linux-2.6-xfs/fs/xfs/linux-2.6/xfs_buf.h
> ===================================================================
> --- linux-2.6-xfs.orig/fs/xfs/linux-2.6/xfs_buf.h	2008-10-09 13:36:50.000000000 +0200
> +++ linux-2.6-xfs/fs/xfs/linux-2.6/xfs_buf.h	2008-10-09 13:38:15.000000000 +0200
> @@ -85,6 +85,14 @@ typedef enum {
>  	 * modifications being lost.
>  	 */
>  	_XBF_PAGE_LOCKED = (1 << 22),
> +
> +	/*
> +	 * If we try a barrier write, but it fails we have to communicate
> +	 * this to the upper layers.  Unfortunately b_error gets overwritten
> +	 * when the buffer is re-issued so we have to add another flag to
> +	 * keep this information.
> +	 */
> +	_XFS_BARRIER_FAILED = (1 << 23),
>  } xfs_buf_flags_t;
>  
>  typedef enum {


> Index: linux-2.6-xfs/fs/xfs/xfs_log.c
> ===================================================================
> --- linux-2.6-xfs.orig/fs/xfs/xfs_log.c	2008-10-09 13:38:44.000000000 +0200
> +++ linux-2.6-xfs/fs/xfs/xfs_log.c	2008-10-09 13:39:32.000000000 +0200
> @@ -1037,7 +1037,8 @@ xlog_iodone(xfs_buf_t *bp)
>  	 * layer, it means the underlyin device no longer supports
>  	 * barrier I/O. Warn loudly and turn off barriers.
>  	 */
> -	if ((l->l_mp->m_flags & XFS_MOUNT_BARRIER) && !XFS_BUF_ISORDERED(bp)) {
> +	if (bp->b_flags & _XFS_BARRIER_FAILED) {
> +		bp->b_flags &= ~_XFS_BARRIER_FAILED;
>  		l->l_mp->m_flags &= ~XFS_MOUNT_BARRIER;
>  		xfs_fs_cmn_err(CE_WARN, l->l_mp,
>  				"xlog_iodone: Barriers are no longer supported"
> 
Okay, we probably should update this comment too.

The existing comment being:
> 	/*
> 	 * If the ordered flag has been removed by a lower
> 	 * layer, it means the underlyin device no longer supports
> 	 * barrier I/O. Warn loudly and turn off barriers.
> 	 */


Might as well fix the existing typo "underlyin" in the comment as well :)

Thanks,
--Tim

Re: [PATCH] fix barrier fail detection

[Timothy Shimmin]

Timothy Shimmin wrote:
> Christoph Hellwig wrote:
>> Index: linux-2.6-xfs/fs/xfs/linux-2.6/xfs_buf.c
>> ===================================================================
>> --- linux-2.6-xfs.orig/fs/xfs/linux-2.6/xfs_buf.c	2008-10-09 13:36:50.000000000 +0200
>> +++ linux-2.6-xfs/fs/xfs/linux-2.6/xfs_buf.c	2008-10-09 13:38:38.000000000 +0200
>> @@ -1007,6 +1007,7 @@ xfs_buf_iodone_work(
>>  	    (bp->b_flags & (XBF_ORDERED|XBF_ASYNC)) == (XBF_ORDERED|XBF_ASYNC)) {
>>  		XB_TRACE(bp, "ordered_retry", bp->b_iodone);
>>  		bp->b_flags &= ~XBF_ORDERED;
>> +		bp->b_flags |= _XFS_BARRIER_FAILED;
>>  		xfs_buf_iorequest(bp);
>>  	} else if (bp->b_iodone)
>>  		(*(bp->b_iodone))(bp);
> 
> Actually, probably need to update the comment for this one.
> 

>> Index: linux-2.6-xfs/fs/xfs/xfs_log.c
>> ===================================================================
>> --- linux-2.6-xfs.orig/fs/xfs/xfs_log.c	2008-10-09 13:38:44.000000000 +0200
>> +++ linux-2.6-xfs/fs/xfs/xfs_log.c	2008-10-09 13:39:32.000000000 +0200
>> @@ -1037,7 +1037,8 @@ xlog_iodone(xfs_buf_t *bp)
>>  	 * layer, it means the underlyin device no longer supports
>>  	 * barrier I/O. Warn loudly and turn off barriers.
>>  	 */
>> -	if ((l->l_mp->m_flags & XFS_MOUNT_BARRIER) && !XFS_BUF_ISORDERED(bp)) {
>> +	if (bp->b_flags & _XFS_BARRIER_FAILED) {
>> +		bp->b_flags &= ~_XFS_BARRIER_FAILED;
>>  		l->l_mp->m_flags &= ~XFS_MOUNT_BARRIER;
>>  		xfs_fs_cmn_err(CE_WARN, l->l_mp,
>>  				"xlog_iodone: Barriers are no longer supported"
>>
> Okay, we probably should update this comment too.
> 
> Might as well fix the existing typo "underlyin" in the comment as well :)
> Thanks,
> --Tim
> 

i.e. something like:

Index: 2.6.x-xfs-quilt/fs/xfs/linux-2.6/xfs_buf.c
===================================================================
--- 2.6.x-xfs-quilt.orig/fs/xfs/linux-2.6/xfs_buf.c	2008-09-19 13:47:36.000000000 +1000
+++ 2.6.x-xfs-quilt/fs/xfs/linux-2.6/xfs_buf.c	2008-10-10 15:07:51.316145158 +1100
@@ -1001,12 +1001,13 @@ xfs_buf_iodone_work(
 	 * We can get an EOPNOTSUPP to ordered writes.  Here we clear the
 	 * ordered flag and reissue them.  Because we can't tell the higher
 	 * layers directly that they should not issue ordered I/O anymore, they
-	 * need to check if the ordered flag was cleared during I/O completion.
+	 * need to check if the _XFS_BARRIER_FAILED flag was set during I/O completion.
 	 */
 	if ((bp->b_error == EOPNOTSUPP) &&
 	    (bp->b_flags & (XBF_ORDERED|XBF_ASYNC)) == (XBF_ORDERED|XBF_ASYNC)) {
 		XB_TRACE(bp, "ordered_retry", bp->b_iodone);
 		bp->b_flags &= ~XBF_ORDERED;
+		bp->b_flags |= _XFS_BARRIER_FAILED;
 		xfs_buf_iorequest(bp);
 	} else if (bp->b_iodone)
 		(*(bp->b_iodone))(bp);
Index: 2.6.x-xfs-quilt/fs/xfs/linux-2.6/xfs_buf.h
===================================================================
--- 2.6.x-xfs-quilt.orig/fs/xfs/linux-2.6/xfs_buf.h	2008-09-19 13:47:36.000000000 +1000
+++ 2.6.x-xfs-quilt/fs/xfs/linux-2.6/xfs_buf.h	2008-10-10 11:54:23.269373217 +1100
@@ -85,6 +85,14 @@ typedef enum {
 	 * modifications being lost.
 	 */
 	_XBF_PAGE_LOCKED = (1 << 22),
+
+	/*
+	 * If we try a barrier write, but it fails we have to communicate
+	 * this to the upper layers.  Unfortunately b_error gets overwritten
+	 * when the buffer is re-issued so we have to add another flag to
+	 * keep this information.
+	 */
+	_XFS_BARRIER_FAILED = (1 << 23),
 } xfs_buf_flags_t;
 
 typedef enum {
Index: 2.6.x-xfs-quilt/fs/xfs/xfs_log.c
===================================================================
--- 2.6.x-xfs-quilt.orig/fs/xfs/xfs_log.c	2008-09-22 11:54:19.000000000 +1000
+++ 2.6.x-xfs-quilt/fs/xfs/xfs_log.c	2008-10-10 15:09:56.967725107 +1100
@@ -1033,11 +1033,12 @@ xlog_iodone(xfs_buf_t *bp)
 	l = iclog->ic_log;
 
 	/*
-	 * If the ordered flag has been removed by a lower
-	 * layer, it means the underlyin device no longer supports
+	 * If the _XFS_BARRIER_FAILED flag was set by a lower
+	 * layer, it means the underlying device no longer supports
 	 * barrier I/O. Warn loudly and turn off barriers.
 	 */
-	if ((l->l_mp->m_flags & XFS_MOUNT_BARRIER) && !XFS_BUF_ISORDERED(bp)) {
+	if (bp->b_flags & _XFS_BARRIER_FAILED) {
+		bp->b_flags &= ~_XFS_BARRIER_FAILED;
 		l->l_mp->m_flags &= ~XFS_MOUNT_BARRIER;
 		xfs_fs_cmn_err(CE_WARN, l->l_mp,
 				"xlog_iodone: Barriers are no longer supported"

Re: [PATCH] fix barrier fail detection

[Dave Chinner]

On Fri, Oct 10, 2008 at 03:17:03PM +1100, Timothy Shimmin wrote:
> Timothy Shimmin wrote:
.....
> > Actually, probably need to update the comment for this one.
.....

> i.e. something like:

Looks fine to me.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [PATCH] fix barrier fail detection

[Christoph Hellwig]

On Fri, Oct 10, 2008 at 03:27:32PM +1100, Dave Chinner wrote:
> On Fri, Oct 10, 2008 at 03:17:03PM +1100, Timothy Shimmin wrote:
> > Timothy Shimmin wrote:
> .....
> > > Actually, probably need to update the comment for this one.
> .....
> 
> > i.e. something like:
> 
> Looks fine to me.

Yeah, looks good.