Re: XFS and nobarrier with SSDs

Re: XFS and nobarrier with SSDs

[Martin Steigerwald]

Hello Georg.

I am adding in some Ccs of kernel devs and mailing lists, as I think this is a 
more generic question. Still of course would be nice to hear something from 
XFS developers as well.

For the broader audience the question is:

Is it safe to use XFS (or any other filesystem) on enterprise SSDs with Power 
Loss Protection (PLP), i.e. some capacitor to provide for enough electricity 
to write out all data in DRAM to flash after a power loss, with a reordering 
I/O scheduler like CFQ?

According to this comment of Jeff Moyer on a report on RedHat´s bugzilla the 
I/O scheduler cannot reorder commit block and log entry, so it would be safe, 
I think (see below).

Am Montag, 14. Dezember 2015, 06:43:48 CET schrieb Georg Schönberger:
> On 2015-12-12 13:26, Martin Steigerwald wrote:
> > Am Samstag, 12. Dezember 2015, 10:24:25 CET schrieb Georg Schönberger:
> >> We are using a lot of SSDs in our Ceph clusters with XFS. Our SSDs have
> >> Power Loss Protection via capacitors, so is it safe in all cases to run
> >> XFS
> >> with nobarrier on them? Or is there indeed a need for a specific I/O
> >> scheduler?
> > 
> > I do think that using nobarrier would be safe with those SSDs as long as
> > there is no other caching happening on the hardware side, for example
> > inside the controller that talks to the SSDs.
[…]
> We are using HBAs and no RAID controller, therefore there is no other
> cache in the I/O stack.
> 
> > I always thought barrier/nobarrier acts independently of the I/O scheduler
> > thing, but I can understand the thought from the bug report you linked to
> > below. As for I/O schedulers, with recent kernels and block multiqueue I
> > see it being set to "none".
> 
> What do you mean by "none" near? Do you think I will be more on the safe
> side with noop scheduler?

I mean that I get this on a 4.3 kernel with blk-mq enabled:

merkaba:/sys/block/sda/queue> grep . rotational scheduler 
rotational:0
scheduler:none

merkaba:/sys/block/sda/queue> echo "cfq" > scheduler 
merkaba:/sys/block/sda/queue> cat scheduler                              
none
merkaba:/sys/block/sda/queue> echo "noop" > scheduler
merkaba:/sys/block/sda/queue> cat scheduler          
none

So with blk-mq I do not get a choice which scheduler to use anyway. Which is 
what I expect. Thats why there has been a discussion about blk-mq on 
rotational devices recently.

> >> I have found a recent discussion on the Ceph mailing list, anyone from
> >> XFS
> >> that can help us?
> >> 
> >> *http://www.spinics.net/lists/ceph-users/msg22053.html
> > 
> > Also see:
> > 
> > http://xfs.org/index.php/
> > XFS_FAQ#Q._Should_barriers_be_enabled_with_storage_which_has_a_persistent_
> > write_cache. 3F
> 
> I've already read that XFS wiki entry before and also found some Intel
> presentations where they suggest to use nobarrier with
> there enterprise SSDs. But a confirmation from any block layer
> specialist would be a good thing!

I think Jens Axboe would be good to ask? But as Jeff Moyer already replied to 
the bug report?

> >> *https://bugzilla.redhat.com/show_bug.cgi?id=1104380
> > 
> > Interesting. Never thought of that one.
> > 
> > So would it be safe to interrupt the flow of data towards the SSD at any
> > point if time with reordering I/O schedulers in place? And how about
> > blk-mq which has mutiple software queus?
> 
> Maybe we should ask the block layer mailing list about that?
> 
> > I like to think that they are still independent of the barrier thing and
> > the> 
> > last bug comment by Eric, where he quoted from Jeff, supports this:
> >> Eric Sandeen 2014-06-24 10:32:06 EDT
> >> 
> >> As Jeff Moyer says:
> >>> The file system will manually order dependent I/O.
> >>> What I mean by that is the file system will send down any I/O for the
> >>> transaction log, wait for that to complete, issue a barrier (which will
> >>> be a noop in the case of a battery-backed write cache), and then send
> >>> down the commit block along with another barrier.  As such, you cannot
> >>> have the I/O scheduler reorder the commit block and the log entry with
> >>> which it is associated.
> 
> If it is truly that way then I do not see any problems using nobarrier
> with the SSDs an power loss protection.
> I have already find some people say that enterprise SSDs with PLP simply
> ignore the sync call. If that's the case
> then using nobarrier would have no performance improvement...

Interesting. I think if they ignore it tough they risk data loss. Cause I 
imagine there might be a slight chance where the data has been sent by the 
controller, but not yet fully stored in SSD DRAM. Or is this operation atomic?

Thanks,
-- 
Martin

Re: XFS and nobarrier with SSDs

[Christoph Hellwig]

On Mon, Dec 14, 2015 at 09:38:56AM +0100, Martin Steigerwald wrote:
> Is it safe to use XFS (or any other filesystem) on enterprise SSDs with Power 
> Loss Protection (PLP), i.e. some capacitor to provide for enough electricity 
> to write out all data in DRAM to flash after a power loss, with a reordering 
> I/O scheduler like CFQ?

If the device does not need cache flushes it should not report requiring
flushes, in which case nobarrier will be a noop.  Or to phrase it
differently:  If nobarrier makes a difference skipping it is not safe.

Re: XFS and nobarrier with SSDs

[Georg Schönberger]

On 2015-12-14 10:58, Christoph Hellwig wrote:
> On Mon, Dec 14, 2015 at 09:38:56AM +0100, Martin Steigerwald wrote:
>> Is it safe to use XFS (or any other filesystem) on enterprise SSDs with Power
>> Loss Protection (PLP), i.e. some capacitor to provide for enough electricity
>> to write out all data in DRAM to flash after a power loss, with a reordering
>> I/O scheduler like CFQ?
> If the device does not need cache flushes it should not report requiring
> flushes, in which case nobarrier will be a noop.
OK - that would also mean that mounting with nobarrier should not make a 
performance difference.

> Or to phrase it
> differently:  If nobarrier makes a difference skipping it is not safe.
I do not fully understand that sentence, what do you mean by "makes a 
difference" and "skipping is not safe"?

-Georg

Re: XFS and nobarrier with SSDs

[Christoph Hellwig]

On Mon, Dec 14, 2015 at 10:18:54AM +0000, Georg Sch?nberger wrote:
> > Or to phrase it
> > differently:  If nobarrier makes a difference skipping it is not safe.
> I do not fully understand that sentence, what do you mean by "makes a 
> difference" and "skipping is not safe"?

The rule of thumb is: if nobarrier makes your workload run faster you
should not be using it, aka: don't use it. 

Re: XFS and nobarrier with SSDs

[Georg Schönberger]

On 2015-12-14 11:27, Christoph Hellwig wrote:
> On Mon, Dec 14, 2015 at 10:18:54AM +0000, Georg Sch?nberger wrote:
>>> Or to phrase it
>>> differently:  If nobarrier makes a difference skipping it is not safe.
>> I do not fully understand that sentence, what do you mean by "makes a
>> difference" and "skipping is not safe"?
> The rule of thumb is: if nobarrier makes your workload run faster you
> should not be using it, aka: don't use it.
OK, thanks for clarification.
Should the XFS FAQ be updated?
*http://xfs.org/index.php/XFS_FAQ#Q._Should_barriers_be_enabled_with_storage_which_has_a_persistent_write_cache.3F

Re: XFS and nobarrier with SSDs

[Christoph Hellwig]

On Mon, Dec 14, 2015 at 10:34:30AM +0000, Georg Sch?nberger wrote:
> OK, thanks for clarification.
> Should the XFS FAQ be updated?
> *http://xfs.org/index.php/XFS_FAQ#Q._Should_barriers_be_enabled_with_storage_which_has_a_persistent_write_cache.3F

Probably.  The text soudns to me like it was written a long time ago
when Linux actually use barriers that also prevent I/O reordering
instead of just issuing the required cache flushes.

Re: XFS and nobarrier with SSDs

[Linda Walsh]

Christoph Hellwig wrote:
> The rule of thumb is: if nobarrier makes your workload run faster you
> should not be using it, aka: don't use it. 
----
	So what is the purpose of the switch if it is to only
be used when it makes no difference?

I.e. My raid controller does write-through if it's internal
battery needs replacing, otherwise, it does write-back.

On top of that my system is on a UPS that is good for a hour or more
of running.  

So, I used to use nobarrier on "work" disks where there were likely
to be alot of "writes".  Those disks are also backed up daily via
xfsdump/restore.  I figured those would benefit most, and at worst
I could restore to previous morning's backup.

Eventually stopped using the option, as for the most part, I couldn't
really measure any reliable difference in performance (which means
I should use it?!?).

Hmmm...

The only times I have experienced disk corruption on a single
disk were either back before I ever tried the option, or when
I had several months to a year where I tried to use software
RAID5 (several-10+ years ago, before it was possible to use
multiple cores for doing some RAID operations).

I doubt I'm going to try it again soon, but being told that
it's only "ok" to use an option when it makes no difference
in performance *sounds* more than a little confusing.