Does ext4 send FUA to flush disk cache

Does ext4 send FUA to flush disk cache

[Mark Busheman]

I plan to use data=journal option with ext4. Would like to know if
ext4 send FUA (Forced Unit Access)
to flush the disk cache?

Cheers
Mark

Re: Does ext4 send FUA to flush disk cache

[Theodore Tso]

On Apr 8, 2011, at 6:02 PM, Mark Busheman wrote:

> I plan to use data=journal option with ext4. Would like to know if
> ext4 send FUA (Forced Unit Access)
> to flush the disk cache?

FUA doesn't cause a cache flush.   Ext4 does send cache flush commands, or barriers, to make sure the data written to disk is flushed all the way down to the disk platters on transaction commits.

-- Ted

Re: Does ext4 send FUA to flush disk cache

[Mark Busheman]

Thanks Ted for replying.

>> FUA doesn't cause a cache flush.
Thanks for correcting. Yes FUA does not flush cache. When the FUA bit
is set in the write
command, the drive make sure that the data goes to platters and not to
be put into its
cache for latter writes.

>> Ext4 does send cache flush commands, or barriers, to make sure the data written to disk
>> is flushed all the way down to the disk platters on transaction commits.
 I have few questions on this and need help:
a. To achieve the above, is any specific options that need to be used
while mounting?
b. Are the above mentioned things done only for data=journal or for
other modes such as
data=ordered and writeback?
c. Are they done for metadata, journal and data writes?
d. Can I know what are the commands sent to the drives to do the cache
flush commands
or barriers? Would like to find if the drive support the commands?
e. Is the above true also for ext3 - I posted a similar question to
ext3 forum and have got no
response so far?

Cheers
Mark

On Sun, Apr 10, 2011 at 5:17 AM, Theodore Tso <tytso@mit.edu> wrote:
>
> On Apr 8, 2011, at 6:02 PM, Mark Busheman wrote:
>
>> I plan to use data=journal option with ext4. Would like to know if
>> ext4 send FUA (Forced Unit Access)
>> to flush the disk cache?
>
> FUA doesn't cause a cache flush.   Ext4 does send cache flush commands, or barriers, to make sure the data written to disk is flushed all the way down to the disk platters on transaction commits.
>
> -- Ted
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Does ext4 send FUA to flush disk cache

[Ted Ts'o]

On Mon, Apr 11, 2011 at 11:23:23AM -0700, Mark Busheman wrote:
> Thanks Ted for replying.
> 
> >> FUA doesn't cause a cache flush.
>
> Thanks for correcting. Yes FUA does not flush cache. When the FUA
> bit is set in the write command, the drive make sure that the data
> goes to platters and not to be put into its cache for latter writes.

For that particular write only; and it forces the write to go directly
to the disk, bypassing any seek-mitigation algorithms the disk might
use.  It's good if you only have a very small number of sectors
(ideally one) which you need to write to the disk.  In the case of
ext4, we want to make sure all of the data and metadata blocks sent to
disk are flushed down to the platters, so FUA isn't as useful to us,
and so we don't use it.

> >> Ext4 does send cache flush commands, or barriers, to make sure the data written to disk
> >> is flushed all the way down to the disk platters on transaction commits.
>  I have few questions on this and need help:
> a. To achieve the above, is any specific options that need to be used
> while mounting?

No, not for ext3.

> b. Are the above mentioned things done only for data=journal or for
> other modes such as
> data=ordered and writeback?

It's done for all journalling modes.

> c. Are they done for metadata, journal and data writes?

Yes; see below.

> d. Can I know what are the commands sent to the drives to do the cache
> flush commands
> or barriers? Would like to find if the drive support the commands?

We use the CACHE FLUSH or CACHE FLUSH EXT command, as appropriate for
IDE/SATA disks.  There is an equivalent command for SCSI, but in
essence it establishes a barrier; all writes sent to the disk before
the CACHE FLUSH/CACHE FLUSH EXT must be written to the disk before the
barrier returns.

When we use journaling, we send a CACHE FLUSH just before the writing
the commit block, and then we send a second CACHE FLUSH after writing
the commit block.  (I suppose we could use a FUA bit for the commit
block, which might be a slight optimization if there is any other
writes happening in parallel that don't need to be affected, but it
doubt it would make a huge difference in most cases.)

> e. Is the above true also for ext3 - I posted a similar question to
> ext3 forum and have got no
> response so far?

ext3 unfortunately defaults to barriers being disabled by default.
Some distributions, such as the enterprise distro's such as RHEL and
SLES, will enable ex3 barriers by default.  You can use the mount
option "barrier" to enable barriers at mount time with ext3, if you
want to be sure barriers are enabled.

With ext4, if you want to disable barriers for some reason (say you
are confident the system has a dead reliable UPS and you need the
performance), you can use the mount option nobarrier.  (It enables
barriers by default.)

						- Ted

Re: Does ext4 send FUA to flush disk cache

[Mark Busheman]

Ted, Thanks for the clear answer and also for answering the question on ext3.

Cheers
Mark

Re: Does ext4 send FUA to flush disk cache

[Christoph Hellwig]

On Sun, Apr 10, 2011 at 08:17:58AM -0400, Theodore Tso wrote:
> 
> On Apr 8, 2011, at 6:02 PM, Mark Busheman wrote:
> 
> > I plan to use data=journal option with ext4. Would like to know if
> > ext4 send FUA (Forced Unit Access)
> > to flush the disk cache?
> 
> FUA doesn't cause a cache flush.   Ext4 does send cache flush commands, or barriers, to make sure the data written to disk is flushed all the way down to the disk platters on transaction commits.

Ext4 actually does send FUA requests, just grep for it.  With the default
libata config they will be turned into a regular write with a
post-flush, but if you enable the fua module option to the libata module
or use plain SCSI devices the FUA bit (if supported) gets sent all the
way down to the device.

The same also applies to btrfs, gfs2, nilfs2, xfs and if you enable the
non-default config ext3 and reiserfs.  Note that all the filesystem also
do regular flushes as pre-flushes during the journal commit or
equivalent and as part of the fsync implementation.

Re: Does ext4 send FUA to flush disk cache

[Ted Ts'o]

On Mon, Apr 11, 2011 at 07:16:06PM -0400, Christoph Hellwig wrote:
> On Sun, Apr 10, 2011 at 08:17:58AM -0400, Theodore Tso wrote:
> > 
> > On Apr 8, 2011, at 6:02 PM, Mark Busheman wrote:
> > 
> > > I plan to use data=journal option with ext4. Would like to know if
> > > ext4 send FUA (Forced Unit Access)
> > > to flush the disk cache?
> > 
> > FUA doesn't cause a cache flush.   Ext4 does send cache flush commands, or barriers, to make sure the data written to disk is flushed all the way down to the disk platters on transaction commits.
> 
> Ext4 actually does send FUA requests, just grep for it.  With the default
> libata config they will be turned into a regular write with a
> post-flush, but if you enable the fua module option to the libata module
> or use plain SCSI devices the FUA bit (if supported) gets sent all the
> way down to the device.

Ah, Jens pushed a patch that I hadn't noticed that appears to
optionally use FUA for the commit block.  Thanks for pointing that
out.

						- Ted

Re: Does ext4 send FUA to flush disk cache

[Christoph Hellwig]

On Mon, Apr 11, 2011 at 10:14:23PM -0400, Ted Ts'o wrote:
> Ah, Jens pushed a patch that I hadn't noticed that appears to
> optionally use FUA for the commit block.  Thanks for pointing that
> out.

The commits actually are from Tejun:

commit 797e7dbbee0a91fa1349192f18ad5c454997d876
Author: Tejun Heo <htejun@gmail.com>
Date:   Fri Jan 6 09:51:03 2006 +0100

    [BLOCK] reimplement handling of barrier request

    Reimplement handling of barrier requests.
	        
     * Flexible handling to deal with various capabilities of
       target devices.
     * Retry support for falling back.
     * Tagged queues which don't support ordered tag can do ordered.
				      
    Signed-off-by: Tejun Heo <htejun@gmail.com>
    Signed-off-by: Jens Axboe <axboe@suse.de>

commit 007365ad60387df30f02f01fdc2b6e6432f6c265
Author: Tejun Heo <htejun@gmail.com>
Date:   Fri Jan 6 09:53:52 2006 +0100

    [BLOCK] scsi: add FUA support to sd

    Add FUA support for barriers to SCSI disk.

    Signed-off-by: Tejun Heo <htejun@gmail.com>
    Signed-off-by: Jens Axboe <axboe@suse.de>