[HELP]: about aio+dio

[HELP]: about aio+dio

[Ma, Jianpeng]

Hi all,
   At present, I met a problem.
Open(block, DIRECT_IO)
Io_submit()--->write operation
Close()
Exit(0)

After exit(), we found the content of block changed. If we only use aio, the bug can't reproduce.

Q1: Is close() wait until all aio to complete? Is exit() to wait until aio to complete?
Q2:Who may change the content? The program don't change.
Q3:My guess: only using aio, after io_submit, the content already copy to kernel from use-space. But for direct, kernel can't copy. 
Later kernel do direct_io but the content changed.


Thanks!
Jianpeng Ma

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org.  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>

RE: [HELP]: about aio+dio

[John Utz]

Hello;

Regretfully, your pseudocode is a bit too vague, would it be possible for you to be a bit more specific, particularly on your open() call?

That might make it easier for folks to figure out what is going wrong.

tnx!

johnu
________________________________________
From: linux-fsdevel-owner@vger.kernel.org [linux-fsdevel-owner@vger.kernel.org] on behalf of Ma, Jianpeng [jianpeng.ma@intel.com]
Sent: Tuesday, August 19, 2014 9:05 PM
To: viro@zeniv.linux.org.uk
Cc: Sage Weil; Mark Kirkwood; linux-aio@kvack.org; linux-fsdevel@vger.kernel.org
Subject: [HELP]:  about aio+dio

Hi all,
   At present, I met a problem.
Open(block, DIRECT_IO)
Io_submit()--->write operation
Close()
Exit(0)

After exit(), we found the content of block changed. If we only use aio, the bug can't reproduce.

Q1: Is close() wait until all aio to complete? Is exit() to wait until aio to complete?
Q2:Who may change the content? The program don't change.
Q3:My guess: only using aio, after io_submit, the content already copy to kernel from use-space. But for direct, kernel can't copy.
Later kernel do direct_io but the content changed.


Thanks!
Jianpeng Ma
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org.  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>

Re: [HELP]: about aio+dio

[Zach Brown]

On Wed, Aug 20, 2014 at 04:05:01AM +0000, Ma, Jianpeng wrote:
> Hi all,
>    At present, I met a problem.
> Open(block, DIRECT_IO)
> Io_submit()--->write operation
> Close()
> Exit(0)
> 
> After exit(), we found the content of block changed. If we only use aio, the bug can't reproduce.
> 
> Q1: Is close() wait until all aio to complete? Is exit() to wait until aio to complete?

exit().

> Q2:Who may change the content? The program don't change.

buf=good
io_submit(buf)
buf=bad
io_getevents()

Could end up with bad on disk.

> Q3:My guess: only using aio, after io_submit, the content already copy to kernel from use-space. But for direct, kernel can't copy. 
> Later kernel do direct_io but the content changed.

I think your guess here is correct.

The program can't rely on the buffer in the iocb being copied during
submission.  It has to assume that the buffer can be referenced at any
point during the lifetime of the asynchronous io.  Any modificatoins to
the buffers after submission but before getting a completion event could
be seen by the async write and could end up on disk. 

If the program assumed that the buffer was always copied during
submission then it would work for buffered io today because it isn't
asynchronous, but it would break for O_DIRECT which often dmas directly
from the pages mapped at the iocb addresses during the async write.  And
it might break in the future if buffered io was ever made asynchronous.

- z

RE: [HELP]: about aio+dio

[Ma, Jianpeng]

> -----Original Message-----
> From: Zach Brown [mailto:zab@zabbo.net]
> Sent: Thursday, August 21, 2014 2:44 AM
> To: Ma, Jianpeng
> Cc: viro@zeniv.linux.org.uk; Sage Weil; Mark Kirkwood; linux-aio@kvack.org;
> linux-fsdevel@vger.kernel.org
> Subject: Re: [HELP]: about aio+dio
> 
> On Wed, Aug 20, 2014 at 04:05:01AM +0000, Ma, Jianpeng wrote:
> > Hi all,
> >    At present, I met a problem.
> > Open(block, DIRECT_IO)
> > Io_submit()--->write operation
> > Close()
> > Exit(0)
> >
> > After exit(), we found the content of block changed. If we only use aio, the
> bug can't reproduce.
> >
> > Q1: Is close() wait until all aio to complete? Is exit() to wait until aio to
> complete?
> 
> exit().
Can you explain in detail? If you can point the code, I think it better.

Jianpeng
> 
> > Q2:Who may change the content? The program don't change.
> 
> buf=good
> io_submit(buf)
> buf=bad
> io_getevents()
> 
> Could end up with bad on disk.
> 
> > Q3:My guess: only using aio, after io_submit, the content already copy to
> kernel from use-space. But for direct, kernel can't copy.
> > Later kernel do direct_io but the content changed.
> 
> I think your guess here is correct.
> 
> The program can't rely on the buffer in the iocb being copied during submission.
> It has to assume that the buffer can be referenced at any point during the
> lifetime of the asynchronous io.  Any modificatoins to the buffers after
> submission but before getting a completion event could be seen by the async
> write and could end up on disk.
> 
> If the program assumed that the buffer was always copied during submission
> then it would work for buffered io today because it isn't asynchronous, but it
> would break for O_DIRECT which often dmas directly from the pages mapped
> at the iocb addresses during the async write.  And it might break in the future
> if buffered io was ever made asynchronous.
> 
> - z

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org.  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>

Re: [HELP]: about aio+dio

[Zach Brown]

On Thu, Aug 21, 2014 at 02:01:38AM +0000, Ma, Jianpeng wrote:
> > -----Original Message-----
> > From: Zach Brown [mailto:zab@zabbo.net]
> > Sent: Thursday, August 21, 2014 2:44 AM
> > To: Ma, Jianpeng
> > Cc: viro@zeniv.linux.org.uk; Sage Weil; Mark Kirkwood; linux-aio@kvack.org;
> > linux-fsdevel@vger.kernel.org
> > Subject: Re: [HELP]: about aio+dio
> > 
> > On Wed, Aug 20, 2014 at 04:05:01AM +0000, Ma, Jianpeng wrote:
> > > Hi all,
> > >    At present, I met a problem.
> > > Open(block, DIRECT_IO)
> > > Io_submit()--->write operation
> > > Close()
> > > Exit(0)
> > >
> > > After exit(), we found the content of block changed. If we only use aio, the
> > bug can't reproduce.
> > >
> > > Q1: Is close() wait until all aio to complete? Is exit() to wait until aio to
> > complete?
> > 
> > exit().
> Can you explain in detail? If you can point the code, I think it better.

Huh, look at that.  It certianly used to.  But it looks like exit_aio()
upstream isn't waiting for iocbs to complete.  So we'll need to get that
fixed :).

- z

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org.  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>

RE: [HELP]: about aio+dio

[Ma, Jianpeng]

Hi Zach,
  Ask other question: Is fsync/fdatasync to wait all aio to complete? AFAK, for fsync, it will send WRITE_FLUSH which make all previous request complete. Is it right?

Thanks!
Jianpeng

> -----Original Message-----
> From: Zach Brown [mailto:zab@zabbo.net]
> Sent: Friday, August 22, 2014 12:59 AM
> To: Ma, Jianpeng
> Cc: viro@zeniv.linux.org.uk; Sage Weil; Mark Kirkwood; linux-aio@kvack.org;
> linux-fsdevel@vger.kernel.org
> Subject: Re: [HELP]: about aio+dio
> 
> On Thu, Aug 21, 2014 at 02:01:38AM +0000, Ma, Jianpeng wrote:
> > > -----Original Message-----
> > > From: Zach Brown [mailto:zab@zabbo.net]
> > > Sent: Thursday, August 21, 2014 2:44 AM
> > > To: Ma, Jianpeng
> > > Cc: viro@zeniv.linux.org.uk; Sage Weil; Mark Kirkwood;
> > > linux-aio@kvack.org; linux-fsdevel@vger.kernel.org
> > > Subject: Re: [HELP]: about aio+dio
> > >
> > > On Wed, Aug 20, 2014 at 04:05:01AM +0000, Ma, Jianpeng wrote:
> > > > Hi all,
> > > >    At present, I met a problem.
> > > > Open(block, DIRECT_IO)
> > > > Io_submit()--->write operation
> > > > Close()
> > > > Exit(0)
> > > >
> > > > After exit(), we found the content of block changed. If we only
> > > > use aio, the
> > > bug can't reproduce.
> > > >
> > > > Q1: Is close() wait until all aio to complete? Is exit() to wait
> > > > until aio to
> > > complete?
> > >
> > > exit().
> > Can you explain in detail? If you can point the code, I think it better.
> 
> Huh, look at that.  It certianly used to.  But it looks like exit_aio() upstream
> isn't waiting for iocbs to complete.  So we'll need to get that fixed :).
> 
> - z

Re: [HELP]: about aio+dio

[Zach Brown]

On Fri, Aug 22, 2014 at 12:40:46AM +0000, Ma, Jianpeng wrote:
> Hi Zach,
>   Ask other question: Is fsync/fdatasync to wait all aio to complete?
>   AFAK, for fsync, it will send WRITE_FLUSH which make all previous
>   request complete. Is it right?

No, sync won't wait for aio operations to complete.  If you want to
flush the cached results of aio operations with sync then you need to
wait to issue the sync until after the aio operations have completed.

- z

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org.  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>

RE: [HELP]: about aio+dio

[Ma, Jianpeng]

For aio+directio:
After io_submit, the content in request-queue of block-device.
In blkdev_fsync(), it will cal blkdev_issue_flush. This operation make all request to disk.

Or am I missing something?

Jianpeng

> -----Original Message-----
> From: Zach Brown [mailto:zab@zabbo.net]
> Sent: Wednesday, August 27, 2014 2:52 AM
> To: Ma, Jianpeng
> Cc: viro@zeniv.linux.org.uk; Sage Weil; Mark Kirkwood; linux-aio@kvack.org;
> linux-fsdevel@vger.kernel.org
> Subject: Re: [HELP]: about aio+dio
> 
> On Fri, Aug 22, 2014 at 12:40:46AM +0000, Ma, Jianpeng wrote:
> > Hi Zach,
> >   Ask other question: Is fsync/fdatasync to wait all aio to complete?
> >   AFAK, for fsync, it will send WRITE_FLUSH which make all previous
> >   request complete. Is it right?
> 
> No, sync won't wait for aio operations to complete.  If you want to flush the
> cached results of aio operations with sync then you need to wait to issue the
> sync until after the aio operations have completed.
> 
> - z

Re: [HELP]: about aio+dio

[Jeff Moyer]

"Ma, Jianpeng" <jianpeng.ma@intel.com> writes:

> For aio+directio:
> After io_submit, the content in request-queue of block-device.

Maybe.  It might be in the per-cpu plug list, or in the elevator's
queues.

> In blkdev_fsync(), it will cal blkdev_issue_flush. This operation make
> all request to disk.

blkdev_issue_flush send a FLUSH to the device.  It does nothing to
re-order existing requests in the queue.  Thus, as Zach said, if you
want to ensure data is on stable media, then you should wait for the I/O
to complete and then issue the fsync.

Cheers,
Jeff

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org.  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>

Re: [HELP]: about aio+dio

[Zach Brown]

On Wed, Aug 27, 2014 at 12:34:44AM +0000, Ma, Jianpeng wrote:
> For aio+directio:
> After io_submit, the content in request-queue of block-device.
> In blkdev_fsync(), it will cal blkdev_issue_flush. This operation make all request to disk.
> 
> Or am I missing something?

You can't rely on implementation details of how far aio operations are
processed before io_submit returns.  It'd be entirely valid to implement
aio by only copying the iocb structures from user space and handing them
off to kernel threads before returning from io_submit().  In that case
the kernel wouldn't even know what file descriptors the aios reference
when fsync is called.

So what you describe might be true today (I doubt it, but I'm not going
to look), but an application that relied on those assumptions runs a
good chance of breaking in the future.

- z

RE: [HELP]: about aio+dio

[Ma, Jianpeng]

Thanks you help!

Jianpeng

> -----Original Message-----
> From: Zach Brown [mailto:zab@zabbo.net]
> Sent: Thursday, August 28, 2014 12:04 AM
> To: Ma, Jianpeng
> Cc: viro@zeniv.linux.org.uk; Sage Weil; Mark Kirkwood; linux-aio@kvack.org;
> linux-fsdevel@vger.kernel.org
> Subject: Re: [HELP]: about aio+dio
> 
> On Wed, Aug 27, 2014 at 12:34:44AM +0000, Ma, Jianpeng wrote:
> > For aio+directio:
> > After io_submit, the content in request-queue of block-device.
> > In blkdev_fsync(), it will cal blkdev_issue_flush. This operation make all
> request to disk.
> >
> > Or am I missing something?
> 
> You can't rely on implementation details of how far aio operations are
> processed before io_submit returns.  It'd be entirely valid to implement aio by
> only copying the iocb structures from user space and handing them off to kernel
> threads before returning from io_submit().  In that case the kernel wouldn't
> even know what file descriptors the aios reference when fsync is called.
> 
> So what you describe might be true today (I doubt it, but I'm not going to look),
> but an application that relied on those assumptions runs a good chance of
> breaking in the future.
> 
> - z

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org.  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>