Re: Ext4 and the "30 second window of death"

[Ric Wheeler <rwheeler@redhat.com>]

Nick Piggin wrote:
> On Friday 03 April 2009 05:38:34 Matthew Garrett wrote:
>   
>> On Fri, Apr 03, 2009 at 05:34:59AM +1100, Nick Piggin wrote:
>>
>>     
>>> Shouldn't applications have a mode to avoid spinning up the disk if it is
>>> so important?
>>>       
>> They do. It's called "Don't use fsync() unless your data needs to be on 
>> disk". I'm not sure why you'd ever want an application to be in anything 
>> but this mode.
>>
>>     
>
> Well you might decide you are willing to sacrifice timely storage of
> logs, or reducing backups in your editor or something. But obviously
> the kernel can't decide which of those fsyncs is safe to omit (or
> turn into a barrier) while staying within the advertised semantics of
> the app. Application obviously can.
>
>   
One thing that you can do at the application level is to try and batch 
up your fsync() requests -  running one fsync (especially on the most 
recently written file) can take down the earlier files with it.

Clearly, this does require some application level complexity, but you 
get the same strong fsync() semantics that you are used to and can run 
almost at non-fsync speeds if the batch size is large enough. Your 
application should not acknowledge it has safely stored any of the files 
locally until it has done an fsync on that particular file.

This technique would work great for an application like rsync, tar, etc. 
For a mail client, you would see a benefit only when you were pulling 
down batches of messages which clearly is a common case if you are still 
reading this thread :-)

The fs_mark program I wrote plays around with the various ways to do 
this if someone is interested in playing around a bit,

Ric