[PATCHSET v9] Support for polled and buffered aio (and more)

[Jens Axboe <axboe@kernel.dk>]

A few fixes and cleanups for the core part of this, and then a few
experimental patches at the end:

1) Add support for polling for the SQTHREAD. If enabled, this will
   poll the SQ ring for submissions, and (if polling is enabled), it
   will poll for completions and place them in the CQ ring. This means
   an application will never have to enter the kernel do do IO, it
   simply updates the SQ ring with what it needs. Currently the thread
   goes to sleep after an idle second, to avoid spinning all the time.
   But if IO is continually flowing, it will keep running. Some room
   for improvement here, with some sort of backoff in terms of spinning
   and sleeping, but the concept seems sound. When updating the SQ
   ring with new entries, the application needs to do:

   barrier();
   if (sq_ring->kflags & IORING_SQ_NEED_WAKEUP)
	   io_ring_enter(ctx, to_submit, 0, IORING_SUBMIT);

   in case the thread has gone to sleep. If IORING_SQ_NEED_WAKEUP isn't
   set, then no call needs to be made.

   This works both with polled IO, in which case the SQ thread will poll
   for completions too (as mentioned), and with buffered/O_DIRECT async
   aio. For the latter, completions end up in the CQ ring when the IRQ
   triggers.

2) Utilization of struct io_event ->res2. This is unused in the kernel.
   One idea is to populate this with meta data about the IO operation.
   For buffered aio, we can tell the application if the IO hit media or
   was served out of the page cache. These kinds of metrics may be useful
   for an application.

For the latter, the sample application I wrote:

http://git.kernel.dk/cgit/fio/plain/t/aio-ring.c

will give you by-the-second cache hit rates when run with buffered=1,
and fio with the aio-ring engine will do the same.

As before, find these patches in my aio-poll branch:

http://git.kernel.dk/cgit/linux-block/log/?h=aio-poll

or clone it here:

git://git.kernel.dk/linux-block aio-poll

and the code is based on my for-4.21/aio branch, which holds the aio
patches queued up for 4.21 already.

Changes since v8:

- Get rid of kmalloc/kfree for inline cached buffered aio
- Add IOCTX_FLAG_SQPOLL
- Correct syscall prototype in <linux/syscalls.h> to match actual syscall
- Add "IO was cached" hint to res2
- Fixed an issue with indexing fixed buffer mappings
- Check for fixed buffer location, EFAULT if app changed location
- Unify bio_iov_bvec_add_pages() and bio_iov_iter_get_pages()
- Split out prep patches
- Style fixups


 Documentation/filesystems/vfs.txt      |    3 +
 Documentation/sysctl/fs.txt            |    8 +-
 arch/x86/entry/syscalls/syscall_64.tbl |    2 +
 block/bio.c                            |   56 +-
 fs/aio.c                               | 1938 ++++++++++++++++++++++--
 fs/block_dev.c                         |   23 +-
 fs/file.c                              |   15 +-
 fs/file_table.c                        |    9 +-
 fs/gfs2/file.c                         |    2 +
 fs/iomap.c                             |   52 +-
 fs/xfs/xfs_file.c                      |    1 +
 include/linux/blk_types.h              |    2 +
 include/linux/file.h                   |    2 +
 include/linux/fs.h                     |    6 +-
 include/linux/iomap.h                  |    1 +
 include/linux/syscalls.h               |    4 +
 include/uapi/asm-generic/unistd.h      |    4 +-
 include/uapi/linux/aio_abi.h           |   47 +
 kernel/sys_ni.c                        |    2 +
 19 files changed, 1993 insertions(+), 184 deletions(-)

-- 
Jens Axboe