md raid1 passes barriers, but xfs doesn't use them?

md raid1 passes barriers, but xfs doesn't use them?

[Eric Sandeen]

So md raid1 is happy to pass down any barrier writes that it sees, but
this bit in xfs_mountfs_check_barriers() at mount time:

        if (mp->m_ddev_targp->bt_bdev->bd_disk->queue->ordered ==
                                        QUEUE_ORDERED_NONE) {
                xfs_fs_cmn_err(CE_NOTE, mp,
                  "Disabling barriers, not supported by the underlying
device");
                mp->m_flags &= ~XFS_MOUNT_BARRIER;
                return;
        }

winds up with XFS disabling barriers on these devices.  However, if this
is simply commented out, XFS happily tests barriers, finds that they
work, leaves them turned on and all subsequent barrier writes to the
device succeed.

Perhaps what we have here is a failure to communicate?  :)

I'm not sure; *should* XFS be looking for a QUEUE_ORDERED tag?  Should
MD be setting one?  Maybe there should be a QUEUE_ORDERED_PASSTHRU flag?
 Or should XFS just stick with the test write and ignore the flag?  I'm
not sure of the queue->ordered flag details, but it seems that XFS & md
raid1 both try hard to keep barriers in force, and there's a disconnect
here somewhere.

Thanks,
-Eric

Re: md raid1 passes barriers, but xfs doesn't use them?

[Martin Steigerwald]

Hi Eric!

Am Dienstag 24 Juni 2008 schrieb Eric Sandeen:
> So md raid1 is happy to pass down any barrier writes that it sees, but
> this bit in xfs_mountfs_check_barriers() at mount time:

Interesting. Since when?

On my last test also ext3 disabled barries over device mapper / 
reiserfs[1]. well but I used it with LVM, maybe its different with RAID 
1. I thought this to be a generic device mapper issue.

Did you test with ext3 too?

[1] http://bugzilla.kernel.org/show_bug.cgi?id=9554

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: md raid1 passes barriers, but xfs doesn't use them?

[Eric Sandeen]

Martin Steigerwald wrote:
> Hi Eric!
> 
> Am Dienstag 24 Juni 2008 schrieb Eric Sandeen:
>> So md raid1 is happy to pass down any barrier writes that it sees, but
>> this bit in xfs_mountfs_check_barriers() at mount time:
> 
> Interesting. Since when?

Which part?  :)  XFS has done the flag check for a very long time...

> On my last test also ext3 disabled barries over device mapper / 
> reiserfs[1]. well but I used it with LVM, maybe its different with RAID 
> 1. I thought this to be a generic device mapper issue.
> 
> Did you test with ext3 too?

This is xfs-specific; ext3 does not look for the queue ordered flag so
won't have this problem on md raid1.

> [1] http://bugzilla.kernel.org/show_bug.cgi?id=9554

but that's for device-mapper, which has never supported barriers ....
I'm talking about md raid1 here (not dm).

-Eric

> Ciao,

Re: md raid1 passes barriers, but xfs doesn't use them?

[Martin Steigerwald]

Am Dienstag 24 Juni 2008 schrieb Eric Sandeen:
> Martin Steigerwald wrote:
> > Hi Eric!

[...]

> > Did you test with ext3 too?
>
> This is xfs-specific; ext3 does not look for the queue ordered flag so
> won't have this problem on md raid1.
>
> > [1] http://bugzilla.kernel.org/show_bug.cgi?id=9554
>
> but that's for device-mapper, which has never supported barriers ....
> I'm talking about md raid1 here (not dm).

Thanks for the info. My misunderstanding - I always thought and I thought 
I even read it somewhere - that md was  a device mapper application.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: md raid1 passes barriers, but xfs doesn't use them?

[Dave Chinner]

On Mon, Jun 23, 2008 at 09:23:10PM -0500, Eric Sandeen wrote:
> So md raid1 is happy to pass down any barrier writes that it sees, but
> this bit in xfs_mountfs_check_barriers() at mount time:
> 
>         if (mp->m_ddev_targp->bt_bdev->bd_disk->queue->ordered ==
>                                         QUEUE_ORDERED_NONE) {
>                 xfs_fs_cmn_err(CE_NOTE, mp,
>                   "Disabling barriers, not supported by the underlying
> device");
>                 mp->m_flags &= ~XFS_MOUNT_BARRIER;
>                 return;
>         }
> 
> winds up with XFS disabling barriers on these devices.  However, if this
> is simply commented out, XFS happily tests barriers, finds that they
> work, leaves them turned on and all subsequent barrier writes to the
> device succeed.
> 
> Perhaps what we have here is a failure to communicate?  :)

What we have is MD doing something strange and non-standard to
implement barriers on RAID1. All other devices that support barriers
define the barrier implementation as something other than
QUEUE_ORDERED_NONE.

> I'm not sure; *should* XFS be looking for a QUEUE_ORDERED tag?

It was put there for some reason - now lost in the mists of time, I
think. I suspect it was for detecting volume managers that didn't
support barriers properly and weren't returning the correct
errors to barrier I/O....

> Should MD be setting one?

If it supports barriers, then it probably should be.

> Maybe there should be a QUEUE_ORDERED_PASSTHRU flag?
> Or should XFS just stick with the test write and ignore the flag?  I'm
> not sure of the queue->ordered flag details, but it seems that XFS & md
> raid1 both try hard to keep barriers in force, and there's a disconnect
> here somewhere.

Yeah, the problem was that last time this check was removed was
that a bunch of existing hardware had barriers enabled on them when
not necessary (e.g. had NVRAM) and they went 5x slower on MD raid1
devices. Having to change the root drive config on a wide install
base was considered much more of support pain than leaving the
check there. I guess that was more of a distro upgrade issue than
a mainline problem, but that's the history. Hence I think we
should probably do whatever everyone else is doing here....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: md raid1 passes barriers, but xfs doesn't use them?

[Eric Sandeen]

Dave Chinner wrote:
> On Mon, Jun 23, 2008 at 09:23:10PM -0500, Eric Sandeen wrote:


>> Maybe there should be a QUEUE_ORDERED_PASSTHRU flag?
>> Or should XFS just stick with the test write and ignore the flag?  I'm
>> not sure of the queue->ordered flag details, but it seems that XFS & md
>> raid1 both try hard to keep barriers in force, and there's a disconnect
>> here somewhere.
> 
> Yeah, the problem was that last time this check was removed was
> that a bunch of existing hardware had barriers enabled on them when
> not necessary (e.g. had NVRAM) and they went 5x slower on MD raid1
> devices. 

Hm,
http://oss.sgi.com/cgi-bin/cvsweb.cgi/xfs-linux/linux-2.6/xfs_super.c#rev1.402,
 putting back what was removed in
http://oss.sgi.com/cgi-bin/cvsweb.cgi/xfs-linux/linux-2.6/xfs_super.c#rev1.380
 I guess.  But this seems like a very weird argument to me.  Whether a
drive/raid has battery backed raid, is 1 spindle or 100, is connected to
a UPS or whatnot really is orthogonal to what should be set on the queue
flag... This should be an admin decision.  Leaving it this way for this
odd reason leaves smaller users w/ 2 raid1 spindles in the desktop box
actually completely unable to use barriers even if they wanted to;
removing the check at least lets the savvy admin mount with an option to
turn them off.

> Having to change the root drive config on a wide install
> base was considered much more of support pain than leaving the
> check there. I guess that was more of a distro upgrade issue than
> a mainline problem, but that's the history. Hence I think we
> should probably do whatever everyone else is doing here...

I'll submit a patch to remove the check ;)

-Eric

> 
> Cheers,
> 
> Dave.