Re: [kvm-devel] [Qemu-devel] [PATCH] QEMU: fsync AIO writes on flush request

[Jamie Lokier <jamie@shareable.org>]

Marcelo Tosatti wrote:
> On Fri, Mar 28, 2008 at 03:07:03PM +0000, Jamie Lokier wrote:
> > Marcelo Tosatti wrote:
> > > Its necessary to guarantee that pending AIO writes have reached stable
> > > storage when the flush request returns.
> > > 
> > > Also change fsync() to fdatasync(), since the modification time is not
> > > critical data.
> > > +    if (aio_fsync(O_DSYNC, &acb->aiocb) < 0) {
> > 
> > >      BDRVRawState *s = bs->opaque;
> > > -    fsync(s->fd);
> > > +    raw_aio_flush(bs);
> > > +    fdatasync(s->fd);
> > > +
> > > +    /* We rely on the fact that no other AIO will be submitted
> > > +     * in parallel, but this should be fixed by per-device
> > > +     * AIO queues when allowing multiple CPU's to process IO
> > > +     * in QEMU.
> > > +     */
> > > +    qemu_aio_flush();
> > 
> > I'm a bit confused by this.  Why do you need aio_fsync(O_DSYNC) _and_
> > synchronous fdatasync() calls?  Aren't they equivalent?
> 
> fdatasync() will write and wait for completion of dirty file data
> present in memory.
> 
> aio_write() only queues data for submission:
> 
>        The "asynchronous" means that this call returns as soon as the  request
>        has  been  enqueued;  the  write may or may not have completed when the
>        call returns. One tests for completion using aio_error(3).
> 

> So fdatasync() is not enough because data written via AIO may not
> have been reflected as "dirty file data" through write() by the time
> raw_flush() is called.

Sure.  But why isn't the aio_fsync(O_DSYNC) enough by itself?

It seems to me you should have something like this:

    /* Flush pending aio_writes until they are dirty data,
       and wait before the aio_fsync. */
    qemu_aio_flush();

    /* Call aio_fsync(O_DSYNC). */
    raw_aio_flush(bs);

    /* Wait for the aio_fsync to complete. */
    qemu_aio_flush();

What am I missing?

-- Jamie