[PATCHSET 0/2] Add lockless MPSC FIFO queue for task work

[Jens Axboe <axboe@kernel.dk>]

Hi,

Details are in the commits, but this adds a variant of an MPSC FIFO
queued based on Dmitry's intrusive MPSC node-based queue algorithm.
Main motivation is better cache locality between the consumer and
producers, and avoiding the need to reverse the llist before running
it. Numbers in patch 2.

Patch 1 adds the basic queue implementation, patch 2 adopts it for
DEFER_TASKRUN variants of io_uring.

Results are really promising. It clearly scales better with more
task work running or producing, and it avoids the added overhead
of needing to reverse the llist when local task work is run. Runs all
the regression tests, and the benchmarking I've done. I've had a user
harness version of this running on arm64 and x86-64 as well.

Can also be found in a git tree here:

https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux.git/log/?h=io_uring-tw-mpscq

 include/linux/io_uring_types.h |  26 +++++-
 io_uring/io_uring.c            |   2 +-
 io_uring/loop.c                |   2 +-
 io_uring/mpscq.h               | 121 +++++++++++++++++++++++++++
 io_uring/tw.c                  | 145 ++++++++++++++++-----------------
 io_uring/tw.h                  |   4 +-
 io_uring/wait.c                |   8 +-
 io_uring/wait.h                |  20 ++++-
 8 files changed, 239 insertions(+), 89 deletions(-)

-- 
Jens Axboe