Re: Question about non asynchronous aio calls.

[Dave Chinner <david@fromorbit.com>]

On Wed, Oct 07, 2015 at 09:13:06PM +0300, Avi Kivity wrote:
> On 07/10/15 18:13, Eric Sandeen wrote:
> >
> >On 10/7/15 10:08 AM, Brian Foster wrote:
> >>On Wed, Oct 07, 2015 at 09:24:15AM -0500, Eric Sandeen wrote:
> >>>
> >>>On 10/7/15 9:18 AM, Gleb Natapov wrote:
> >>>>Hello XFS developers,
> >>>>
> >>>>We are working on scylladb[1] database which is written using seastar[2]
> >>>>- highly asynchronous C++ framework. The code uses aio heavily: no
> >>>>synchronous operation is allowed at all by the framework otherwise
> >>>>performance drops drastically. We noticed that the only mainstream FS
> >>>>in Linux that takes aio seriously is XFS. So let me start by thanking
> >>>>you guys for the great work! But unfortunately we also noticed that
> >>>>sometimes io_submit() is executed synchronously even on XFS.
> >>>>
> >>>>Looking at the code I see two cases when this is happening: unaligned
> >>>>IO and write past EOF. It looks like we hit both. For the first one we
> >>>>make special afford to never issue unaligned IO and we use XFS_IOC_DIOINFO
> >>>>to figure out what alignment should be, but it does not help. Looking at the
> >>>>code though xfs_file_dio_aio_write() checks alignment against m_blockmask which
> >>>>is set to be sbp->sb_blocksize - 1, so aio expects buffer to be aligned to
> >>>>filesystem block size not values that DIOINFO returns. Is it intentional? How
> >>>>should our code know what it should align buffers to?
> >>>         /* "unaligned" here means not aligned to a filesystem block */
> >>>         if ((pos & mp->m_blockmask) || ((pos + count) & mp->m_blockmask))
> >>>                 unaligned_io = 1;
> >>>
> >>>It should be aligned to the filesystem block size.
> >>>
> >>I'm not sure exactly what kinds of races are opened if the above locking
> >>were absent, but I'd guess it's related to the buffer/block state
> >>management, block zeroing and whatnot that is buried in the depths of
> >>the generic dio code.
> >Yep:
> >
> >commit eda77982729b7170bdc9e8855f0682edf322d277
> >Author: Dave Chinner <dchinner@redhat.com>
> >Date:   Tue Jan 11 10:22:40 2011 +1100
> >
> >     xfs: serialise unaligned direct IOs

[...]

> >I fixed something similar in ext4 at the time, FWIW.
> 
> Makes sense.
> 
> Is there a way to relax this for reads?

The above mostly only applies to writes. Reads don't modify data so
racing unaligned reads against other reads won't given unexpected
results and so aren't serialised.

i.e. serialisation will only occur when:
	- unaligned write IO will serialise until sub-block zeroing
	  is complete.
	- write IO extending EOF will serialis until post-EOF
	  zeroing is complete
	- cached pages are found on the inode (i.e. mixing
	  buffered/mmap access with direct IO).
	- truncate/extent manipulation syscall is run

All other DIO will be issued and run concurrently, reads and writes.

Realistically, if you are care about performance (which obviously
you are) then you do not do unaligned IO, and you try hard to
minimise operations that extend the file...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs