Re: Question about non asynchronous aio calls.

[Eric Sandeen <sandeen@sandeen.net>]

On 10/7/15 9:18 AM, Gleb Natapov wrote:
> Hello XFS developers,
> 
> We are working on scylladb[1] database which is written using seastar[2]
> - highly asynchronous C++ framework. The code uses aio heavily: no
> synchronous operation is allowed at all by the framework otherwise
> performance drops drastically. We noticed that the only mainstream FS
> in Linux that takes aio seriously is XFS. So let me start by thanking
> you guys for the great work! But unfortunately we also noticed that
> sometimes io_submit() is executed synchronously even on XFS.
> 
> Looking at the code I see two cases when this is happening: unaligned
> IO and write past EOF. It looks like we hit both. For the first one we
> make special afford to never issue unaligned IO and we use XFS_IOC_DIOINFO
> to figure out what alignment should be, but it does not help. Looking at the
> code though xfs_file_dio_aio_write() checks alignment against m_blockmask which
> is set to be sbp->sb_blocksize - 1, so aio expects buffer to be aligned to
> filesystem block size not values that DIOINFO returns. Is it intentional? How
> should our code know what it should align buffers to?

        /* "unaligned" here means not aligned to a filesystem block */
        if ((pos & mp->m_blockmask) || ((pos + count) & mp->m_blockmask))
                unaligned_io = 1;

It should be aligned to the filesystem block size.

> Second one is harder. We do need to write past the end of a file, actually
> most of our writes are like that, so it would have been great for XFS to
> handle this case asynchronously.

You didn't say what kernel you're on, but these:

9862f62 xfs: allow appending aio writes
7b7a866 direct-io: Implement generic deferred AIO completions

hit kernel v3.15.

However, we had a bug report about this, and Brian has sent a fix
which has not yet been merged, see:

[PATCH 1/2] xfs: always drain dio before extending aio write submission

on this list last week.

With those 3 patches, things should just work for you I think.

-Eric

> Currently we are working to work around
> this by issuing truncate() (or fallocate()) on another thread and doing
> aio on a main thread only after truncate() is complete. It seams to be
> working, but is it guarantied that a thread issuing aio will never sleep
> in this case (may be new file size value needs to hit the disk and it is
> not guarantied that it will happen after truncate() returns, but before
> aio call)?
> 
> [2] http://www.scylladb.com/
> [1] http://www.seastar-project.org/
> 
> Thanks,
> 
> --
> 			Gleb.
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs