Re: [RFC] relaxed barrier semantics

[Vladislav Bolkhovitin <vst@vlnb.net>]

Ted Ts'o, on 07/30/2010 03:04 AM wrote:
> On Thu, Jul 29, 2010 at 04:30:54PM -0600, Andreas Dilger wrote:
>> Like James wrote, this is basically everything FUA.  It is OK for
>> ordered mode to allow the device to aggregate the normal filesystem
>> and journal IO, but when the commit block is written it should flush
>> all of the previously written data to disk.  This still allows
>> request re-ordering and merging inside the device, but orders the
>> data vs. the commit block.  Having the proposed "flush ranges"
>> interface to the disk would be ideal, since there would be no wasted
>> time flushing data that does not need it (i.e. other partitions).
>
> My understanding is that "everything FUA" can be a performance
> disaster.  That's because it bypasses the track buffer, and things get
> written directly to disk.  So there is no possibility to reorder
> buffers so that they get written in one disk rotation.  Depending on
> the disk, it might even be that if you send N sequential sectors all
> tagged with FUA, it could be slower than sending the N sectors
> followed by a cache flush or SYNCHRONIZE_CACHE command.

It should be, because it gives the drive opportunity to better load 
internal resources and provide data transfer pipelining. Although, of 
course, it's possible to imagine a stupid drive with nearly broken 
caching which would work in write through mode faster.

I used word "drive", not "disk" above, because I think this discussions 
is not only about disks. Storage might be not only disks, but also 
external arrays and even clusters of arrays. They all look to the system 
as single "disks", but are much more advanced and sophisticated in all 
internal capabilities than dumb (S)ATA disks. Now such arrays and 
clusters are getting more and more commonly used. Anybody can make such 
array using just a Linux box with any OSS SCSI target software and use 
them with a variety of interfaces: iSCSI, Fibre Channel, SAS, InfiniBand 
and even familiar parallel SCSI (Funny, 2 Linux boxes connected by Wide 
SCSI :) ).

So, why to only limit discussion to the low end disks? I believe it 
would be more productive if we at first determine the set of 
capabilities which should be used for the best performance and which 
advanced storage devices can provide and then go down to the lower end 
eliminating the use of the advantage features with sacrificing 
performance. Otherwise, ignoring the "hardware offload" which advanced 
devices provide, we would never achieve the best performance they could 
give.

I'd start the analyze of the best performance facilities from the following:

1. Full set of SCSI queuing and task management control facilities. Namely:

  - SIMPLE, ORDERED, ACA and, maybe, HEAD OF QUEUE commands attributes

  - Never draining the queue to wait for completion of one or more 
commands, except some rare recovery error recovery cases.

  - ACA and UA_INTRCK for protecting the queue order in case if one or 
more commands in it finished abnormally.

  - Use of write back caching by default and switch to write through 
only for "blacklisted" drives.

  - FUA for sequences of few write commands, where either 
SYNCHRONIZE_CACHE command is an overkill, or there is internal order 
dependency between the commands, so they must be written to the media 
exactly in the required order.

So, for instance, a naive sequence of meta-data updates with the 
corresponding journal writes would be a chain of commands:

1. 1st journal write command (SIMPLE)

2. 2d  journal write command (SIMPLE)

3. 3d  journal write command (SIMPLE)

4. SYNCHRONIZE_CACHE for blocks written by those 3 commands (ORDERED)

5. Necessary amount of meta-data update commands (all SIMPLE)

6. SYNCHRONIZE_CACHE for blocks written in 5 (ORDERED)

7. Command marking the transaction committed in the journal (ORDERED)

That's all. No queue draining anywhere. Plus, sending commands without 
internal order requirements as SIMPLE would allow the drive to better 
schedule execution of them among internal storage (actual disks).

For an error recovery case consider command (4) abnormally finished 
because of some external event, like Unit Attention. Then the drive 
would establish ACA condition and suspend the commands queue with 
commands from (5) in the head. Then the system would retry this command 
with ACA attribute. Then, when it finished, would clear the ACA 
condition. Then the drive would resume the queue and commands in the 
head ((5)) started being processed.

For a simpler device (a disk without support for ORDERED queuing) the 
same meta-data updates would be:

1. 1st journal write command

2. 2d  journal write command

3. 3d  journal write command

4. The queue draining.

5. SYNCHRONIZE_CACHE

6. The queue draining.

7. Necessary amount of meta-data update commands

8. The queue draining.

9. SYNCHRONIZE_CACHE for blocks written in 5 (ORDERED)

10. The queue draining.

11. Command marking the transaction committed in the journal (ORDERED)

Then we would need to figure out an interface for file systems to let 
them be able to specify the necessary ordering and cache flushing 
requirements in a generic way. The current interface looks almost good, but:

1. In it semantic of "barrier" is quite overloaded, hence confusing and 
hard to implement.

2. It doesn't allow to bind several requests in an ordered chain.

Vlad