aio: Question about durability guarantees

aio: Question about durability guarantees

[Ilya Lantukh]

Hi all,

We are considering to use libaio in our project as an optional
alternative to Java NIO. Our goal is to improve performance while
maintaining durability.

We have performed tests on different storage devices. Files were
opened with flags: O_WRONLY | O_DIRECT | O_DSYNC. However, on some SSD
devices O_DSYNC causes io_submit() to block until operation is
finished, which is not acceptable in our case.

The question is: do libaio writes with only WRONLY | O_DIRECT flags
provide any durability guarantees? Can I be sure that after
io_getevents() returns an event indicating successful write the data
is actually flushed to persistent storage?

Thanks!

-- 
Best regards,
Ilya

Re: aio: Question about durability guarantees

[Christoph Hellwig]

On Tue, Jul 12, 2016 at 12:42:28PM +0300, Ilya Lantukh wrote:
> Hi all,
> 
> We are considering to use libaio in our project as an optional
> alternative to Java NIO. Our goal is to improve performance while
> maintaining durability.
> 
> We have performed tests on different storage devices. Files were
> opened with flags: O_WRONLY | O_DIRECT | O_DSYNC. However, on some SSD
> devices O_DSYNC causes io_submit() to block until operation is
> finished, which is not acceptable in our case.

What is your exact setup?  Sounds like you're using a file system
and not a block device node.  Which filesystem shows this behavior?

> 
> The question is: do libaio writes with only WRONLY | O_DIRECT flags
> provide any durability guarantees?

No, they don't, you need O_DSYNC for that.

Re: aio: Question about durability guarantees

[Ilya Lantukh]

On Tue, Jul 12, 2016 at 3:00 PM, Christoph Hellwig <hch@infradead.org> wrote:
> On Tue, Jul 12, 2016 at 12:42:28PM +0300, Ilya Lantukh wrote:
>> Hi all,
>>
>> We are considering to use libaio in our project as an optional
>> alternative to Java NIO. Our goal is to improve performance while
>> maintaining durability.
>>
>> We have performed tests on different storage devices. Files were
>> opened with flags: O_WRONLY | O_DIRECT | O_DSYNC. However, on some SSD
>> devices O_DSYNC causes io_submit() to block until operation is
>> finished, which is not acceptable in our case.
>
> What is your exact setup?  Sounds like you're using a file system
> and not a block device node.  Which filesystem shows this behavior?

The filesystem is ext4. Yes, we access file using a file system (like
fd = open("~/test/file0", ...);) and for most devices it works fine.
Does it mean that we need to open a block device directly to get full
advantage of aio?

>
>>
>> The question is: do libaio writes with only WRONLY | O_DIRECT flags
>> provide any durability guarantees?
>
> No, they don't, you need O_DSYNC for that.

Got it, thanks!

-- 
Best regards,
Ilya

Re: aio: Question about durability guarantees

[Jeff Moyer]

Ilya Lantukh <ilantukh@gridgain.com> writes:

> On Tue, Jul 12, 2016 at 3:00 PM, Christoph Hellwig <hch@infradead.org> wrote:
>> On Tue, Jul 12, 2016 at 12:42:28PM +0300, Ilya Lantukh wrote:
>>> Hi all,
>>>
>>> We are considering to use libaio in our project as an optional
>>> alternative to Java NIO. Our goal is to improve performance while
>>> maintaining durability.
>>>
>>> We have performed tests on different storage devices. Files were
>>> opened with flags: O_WRONLY | O_DIRECT | O_DSYNC. However, on some SSD
>>> devices O_DSYNC causes io_submit() to block until operation is
>>> finished, which is not acceptable in our case.
>>
>> What is your exact setup?  Sounds like you're using a file system
>> and not a block device node.  Which filesystem shows this behavior?
>
> The filesystem is ext4. Yes, we access file using a file system (like
> fd = open("~/test/file0", ...);) and for most devices it works fine.
> Does it mean that we need to open a block device directly to get full
> advantage of aio?

No.  There are always cases where the I/O submission path can block.  I
find it extremely unusual that you've linked blocking to a particular
type of device, though.  Try running the io_submit.stp systemtap script
to see where you're blocking.

>>> The question is: do libaio writes with only WRONLY | O_DIRECT flags
>>> provide any durability guarantees?
>>
>> No, they don't, you need O_DSYNC for that.
>
> Got it, thanks!

Please have a look at this article for more information:

https://lwn.net/Articles/457667/

Cheers,
Jeff