Write barriers on MD RAID1

Write barriers on MD RAID1

[Ken Milmore]

Support for write barriers on MD RAID1 devices appears to be broken on 
2.6.29 and later kernels.  Mounting an ext3 or ext4 filesystem over an 
md raid1 block device gives the kernel message:

   JBD: barrier-based sync failed on md0 - disabling barriers

Is this intentional?  It looks like a regression to me, since write 
barriers (ostensibly) worked for MD RAID1 on earlier kernels (well, all 
of the 2.6.27 series at least...)

Doing a git bisect suggested that this is the culprit:

commit cec0707e40ae25794b5a2de7b7f03c51961f80d9
Author: Jens Axboe <jens.axboe@oracle.com>
Date:   Tue Jan 13 15:28:32 2009 +0100

     block: silently error an unsupported barrier bio

Rolling this back from 2.6.29.4 made the JBD error message go away for 
me.  I tried to do some disk benchmarking to determine if write barriers 
were really still working, but the results were not very conclusive.

Details of test setup:
- 2.6.29.4 kernel (config from slackware-current)
- ext3 filesystem (mounted with barrier=1), also tried ext4
- MD RAID1 block device over two SATA disks
- Intel motherboard AHCI controller (82801)

I got similar behaviour on a virtualised setup using qemu, with the RAID 
over two emulated IDE disks.

Please contact me if I can provide any further information.

-Ken Milmore.

Re: Write barriers on MD RAID1

[NeilBrown]

On Sun, May 24, 2009 9:54 am, Ken Milmore wrote:
>
> Support for write barriers on MD RAID1 devices appears to be broken on
> 2.6.29 and later kernels.  Mounting an ext3 or ext4 filesystem over an
> md raid1 block device gives the kernel message:
>
>    JBD: barrier-based sync failed on md0 - disabling barriers
>
> Is this intentional?  It looks like a regression to me, since write
> barriers (ostensibly) worked for MD RAID1 on earlier kernels (well, all
> of the 2.6.27 series at least...)
>
> Doing a git bisect suggested that this is the culprit:
>
> commit cec0707e40ae25794b5a2de7b7f03c51961f80d9
> Author: Jens Axboe <jens.axboe@oracle.com>
> Date:   Tue Jan 13 15:28:32 2009 +0100
>
>      block: silently error an unsupported barrier bio

Yes, that patch is broken.
the new code it introduces should probably go in the top of
__make_request rather than just before the call to ->make_request_fn.
make_request_fn functions other than __make_request generally
don't use ->next_ordered at all.

Jens?  Would you agree?

NeilBrown


>
> Rolling this back from 2.6.29.4 made the JBD error message go away for
> me.  I tried to do some disk benchmarking to determine if write barriers
> were really still working, but the results were not very conclusive.
>
> Details of test setup:
> - 2.6.29.4 kernel (config from slackware-current)
> - ext3 filesystem (mounted with barrier=1), also tried ext4
> - MD RAID1 block device over two SATA disks
> - Intel motherboard AHCI controller (82801)
>
> I got similar behaviour on a virtualised setup using qemu, with the RAID
> over two emulated IDE disks.
>
> Please contact me if I can provide any further information.
>
> -Ken Milmore.
>

Re: Write barriers on MD RAID1

[Ken Milmore]

I thought I had better bump my previous post as this regression is still 
present in 2.6.29.5.

To recap, commit cec0707e40ae25794b5a2de7b7f03c51961f80d9 has broken 
write barriers on md raid1 block devices in 2.6.29 and later kernels. 
Reversing this commit appears to fix the problem.

Please let me know if I'm harassing the wrong maintainers here!

-Ken.


NeilBrown wrote:
> On Sun, May 24, 2009 9:54 am, Ken Milmore wrote:
>> Support for write barriers on MD RAID1 devices appears to be broken on
>> 2.6.29 and later kernels.  Mounting an ext3 or ext4 filesystem over an
>> md raid1 block device gives the kernel message:
>>
>>    JBD: barrier-based sync failed on md0 - disabling barriers
>>
>> Is this intentional?  It looks like a regression to me, since write
>> barriers (ostensibly) worked for MD RAID1 on earlier kernels (well, all
>> of the 2.6.27 series at least...)
>>
>> Doing a git bisect suggested that this is the culprit:
>>
>> commit cec0707e40ae25794b5a2de7b7f03c51961f80d9
>> Author: Jens Axboe <jens.axboe@oracle.com>
>> Date:   Tue Jan 13 15:28:32 2009 +0100
>>
>>      block: silently error an unsupported barrier bio
> 
> Yes, that patch is broken.
> the new code it introduces should probably go in the top of
> __make_request rather than just before the call to ->make_request_fn.
> make_request_fn functions other than __make_request generally
> don't use ->next_ordered at all.
> 
> Jens?  Would you agree?
> 
> NeilBrown
> 
> 
>> Rolling this back from 2.6.29.4 made the JBD error message go away for
>> me.  I tried to do some disk benchmarking to determine if write barriers
>> were really still working, but the results were not very conclusive.
>>
>> Details of test setup:
>> - 2.6.29.4 kernel (config from slackware-current)
>> - ext3 filesystem (mounted with barrier=1), also tried ext4
>> - MD RAID1 block device over two SATA disks
>> - Intel motherboard AHCI controller (82801)
>>
>> I got similar behaviour on a virtualised setup using qemu, with the RAID
>> over two emulated IDE disks.
>>
>> Please contact me if I can provide any further information.
>>
>> -Ken Milmore.
>>
> 
> 

Re: Write barriers on MD RAID1

[Neil Brown]

On Sunday June 21, ken.milmore@googlemail.com wrote:
> I thought I had better bump my previous post as this regression is still 
> present in 2.6.29.5.

Thanks Ken.

> 
> To recap, commit cec0707e40ae25794b5a2de7b7f03c51961f80d9 has broken 
> write barriers on md raid1 block devices in 2.6.29 and later kernels. 
> Reversing this commit appears to fix the problem.
> 
> Please let me know if I'm harassing the wrong maintainers here!

Jens,
  have you had a chance to look at this?

I think the following patch is appropriate and should go in to
-stable.

Thanks,
NeilBrown


>From addd8b129835a63d6df9a38eae20caaa15de5988 Mon Sep 17 00:00:00 2001
From: NeilBrown <neilb@suse.de>
Date: Wed, 24 Jun 2009 13:39:15 +1000
Subject: [PATCH] Restore barrier support for md and probably other virtual devices.

The next_ordered flag is only meaningful for devices that use __make_request.
So move the test against next_ordered out of generic code and in to
__make_request

Since this test was added, barriers have not worked on md, and (I
think) dm and similar devices that don't use __make_request and so
don't bother to set next_ordered.

Cc: stable@kernel.org
Cc: Ken Milmore <ken.milmore@googlemail.com>
Signed-off-by: NeilBrown <neilb@suse.de>
---
 block/blk-core.c |   10 +++++-----
 1 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/block/blk-core.c b/block/blk-core.c
index b06cf5c..fc221aa 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -1172,6 +1172,11 @@ static int __make_request(struct request_queue *q, struct bio *bio)
 	const int unplug = bio_unplug(bio);
 	int rw_flags;
 
+	if (bio_barrier(bio) && bio_has_data(bio) &&
+	    (q->next_ordered == QUEUE_ORDERED_NONE)) {
+		bio_endio(bio, -EOPNOTSUPP)
+			return 0;
+	}
 	/*
 	 * low level driver can indicate that it wants pages above a
 	 * certain limit bounced to low memory (ie for highmem, or even
@@ -1472,11 +1477,6 @@ static inline void __generic_make_request(struct bio *bio)
 			err = -EOPNOTSUPP;
 			goto end_io;
 		}
-		if (bio_barrier(bio) && bio_has_data(bio) &&
-		    (q->next_ordered == QUEUE_ORDERED_NONE)) {
-			err = -EOPNOTSUPP;
-			goto end_io;
-		}
 
 		ret = q->make_request_fn(q, bio);
 	} while (ret);
-- 
1.6.3.1

Re: [dm-devel] Re: Write barriers on MD RAID1

[Alasdair G Kergon]

On Wed, Jun 24, 2009 at 01:43:35PM +1000, Neil Brown wrote:
> Since this test was added, barriers have not worked on md, and (I
> think) dm and similar devices that don't use __make_request and so
> don't bother to set next_ordered.
 
In dm, we added this:

+       blk_queue_ordered(md->queue, QUEUE_ORDERED_DRAIN, NULL);

Alasdair

Re: [dm-devel] Re: Write barriers on MD RAID1

[NeilBrown]

On Wed, June 24, 2009 5:09 pm, Alasdair G Kergon wrote:
> On Wed, Jun 24, 2009 at 01:43:35PM +1000, Neil Brown wrote:
>> Since this test was added, barriers have not worked on md, and (I
>> think) dm and similar devices that don't use __make_request and so
>> don't bother to set next_ordered.
>
> In dm, we added this:
>
> +       blk_queue_ordered(md->queue, QUEUE_ORDERED_DRAIN, NULL);
>

I obviously didn't grep for the right thing.

This will obviously work, but I can't help thinking it is in the wrong
place.  I don't even have a queue in md/raid1, so making a statement
about how it is ordered just doesn't make sense.

Jens?

NeilBrown

Re: Write barriers on MD RAID1

[Jens Axboe]

On Wed, Jun 24 2009, Neil Brown wrote:
> On Sunday June 21, ken.milmore@googlemail.com wrote:
> > I thought I had better bump my previous post as this regression is still 
> > present in 2.6.29.5.
> 
> Thanks Ken.
> 
> > 
> > To recap, commit cec0707e40ae25794b5a2de7b7f03c51961f80d9 has broken 
> > write barriers on md raid1 block devices in 2.6.29 and later kernels. 
> > Reversing this commit appears to fix the problem.
> > 
> > Please let me know if I'm harassing the wrong maintainers here!
> 
> Jens,
>   have you had a chance to look at this?

Yeah, I think it's the right way to go. I'll queue it up for .31 and we
should put in in -stable as well.

> 
> I think the following patch is appropriate and should go in to
> -stable.
> 
> Thanks,
> NeilBrown
> 
> 
> From addd8b129835a63d6df9a38eae20caaa15de5988 Mon Sep 17 00:00:00 2001
> From: NeilBrown <neilb@suse.de>
> Date: Wed, 24 Jun 2009 13:39:15 +1000
> Subject: [PATCH] Restore barrier support for md and probably other virtual devices.
> 
> The next_ordered flag is only meaningful for devices that use __make_request.
> So move the test against next_ordered out of generic code and in to
> __make_request
> 
> Since this test was added, barriers have not worked on md, and (I
> think) dm and similar devices that don't use __make_request and so
> don't bother to set next_ordered.
> 
> Cc: stable@kernel.org
> Cc: Ken Milmore <ken.milmore@googlemail.com>
> Signed-off-by: NeilBrown <neilb@suse.de>
> ---
>  block/blk-core.c |   10 +++++-----
>  1 files changed, 5 insertions(+), 5 deletions(-)
> 
> diff --git a/block/blk-core.c b/block/blk-core.c
> index b06cf5c..fc221aa 100644
> --- a/block/blk-core.c
> +++ b/block/blk-core.c
> @@ -1172,6 +1172,11 @@ static int __make_request(struct request_queue *q, struct bio *bio)
>  	const int unplug = bio_unplug(bio);
>  	int rw_flags;
>  
> +	if (bio_barrier(bio) && bio_has_data(bio) &&
> +	    (q->next_ordered == QUEUE_ORDERED_NONE)) {
> +		bio_endio(bio, -EOPNOTSUPP)
> +			return 0;
> +	}
>  	/*
>  	 * low level driver can indicate that it wants pages above a
>  	 * certain limit bounced to low memory (ie for highmem, or even
> @@ -1472,11 +1477,6 @@ static inline void __generic_make_request(struct bio *bio)
>  			err = -EOPNOTSUPP;
>  			goto end_io;
>  		}
> -		if (bio_barrier(bio) && bio_has_data(bio) &&
> -		    (q->next_ordered == QUEUE_ORDERED_NONE)) {
> -			err = -EOPNOTSUPP;
> -			goto end_io;
> -		}
>  
>  		ret = q->make_request_fn(q, bio);
>  	} while (ret);
> -- 
> 1.6.3.1
> 

-- 
Jens Axboe

Re: Write barriers on MD RAID1

[Neil Brown]

On Wednesday June 24, jens.axboe@oracle.com wrote:
> On Wed, Jun 24 2009, Neil Brown wrote:
> > On Sunday June 21, ken.milmore@googlemail.com wrote:
> > > I thought I had better bump my previous post as this regression is still 
> > > present in 2.6.29.5.
> > 
> > Thanks Ken.
> > 
> > > 
> > > To recap, commit cec0707e40ae25794b5a2de7b7f03c51961f80d9 has broken 
> > > write barriers on md raid1 block devices in 2.6.29 and later kernels. 
> > > Reversing this commit appears to fix the problem.
> > > 
> > > Please let me know if I'm harassing the wrong maintainers here!
> > 
> > Jens,
> >   have you had a chance to look at this?
> 
> Yeah, I think it's the right way to go. I'll queue it up for .31 and we
> should put in in -stable as well.

Great, thanks.

NeilBrown


> 
> > 
> > I think the following patch is appropriate and should go in to
> > -stable.
> > 
> > Thanks,
> > NeilBrown
> > 
> > 
> > From addd8b129835a63d6df9a38eae20caaa15de5988 Mon Sep 17 00:00:00 2001
> > From: NeilBrown <neilb@suse.de>
> > Date: Wed, 24 Jun 2009 13:39:15 +1000
> > Subject: [PATCH] Restore barrier support for md and probably other virtual devices.
> > 
> > The next_ordered flag is only meaningful for devices that use __make_request.
> > So move the test against next_ordered out of generic code and in to
> > __make_request
> > 
> > Since this test was added, barriers have not worked on md, and (I
> > think) dm and similar devices that don't use __make_request and so
> > don't bother to set next_ordered.
> > 
> > Cc: stable@kernel.org
> > Cc: Ken Milmore <ken.milmore@googlemail.com>
> > Signed-off-by: NeilBrown <neilb@suse.de>
> > ---
> >  block/blk-core.c |   10 +++++-----
> >  1 files changed, 5 insertions(+), 5 deletions(-)
> > 
> > diff --git a/block/blk-core.c b/block/blk-core.c
> > index b06cf5c..fc221aa 100644
> > --- a/block/blk-core.c
> > +++ b/block/blk-core.c
> > @@ -1172,6 +1172,11 @@ static int __make_request(struct request_queue *q, struct bio *bio)
> >  	const int unplug = bio_unplug(bio);
> >  	int rw_flags;
> >  
> > +	if (bio_barrier(bio) && bio_has_data(bio) &&
> > +	    (q->next_ordered == QUEUE_ORDERED_NONE)) {
> > +		bio_endio(bio, -EOPNOTSUPP)
> > +			return 0;
> > +	}
> >  	/*
> >  	 * low level driver can indicate that it wants pages above a
> >  	 * certain limit bounced to low memory (ie for highmem, or even
> > @@ -1472,11 +1477,6 @@ static inline void __generic_make_request(struct bio *bio)
> >  			err = -EOPNOTSUPP;
> >  			goto end_io;
> >  		}
> > -		if (bio_barrier(bio) && bio_has_data(bio) &&
> > -		    (q->next_ordered == QUEUE_ORDERED_NONE)) {
> > -			err = -EOPNOTSUPP;
> > -			goto end_io;
> > -		}
> >  
> >  		ret = q->make_request_fn(q, bio);
> >  	} while (ret);
> > -- 
> > 1.6.3.1
> > 
> 
> -- 
> Jens Axboe