[Qemu-devel] [PATCH 0/4] virtiofsd: multithreading preparation part 3

[Stefan Hajnoczi <stefanha@redhat.com>]

This patch series introduces the virtiofsd --thread-pool-size=NUM and sets the
default value to 64.  Each virtqueue has its own thread pool for processing
requests.  Blocking requests no longer pause virtqueue processing and I/O
performance should be greatly improved when the queue depth is greater than 1.

Linux boot and pjdfstest have been tested with these patches and the default
thread pool size of 64.

I have now concluded the thread-safety code audit.  Please let me know if you
have concerns about things I missed.

Performance
-----------
Please try these patches out and share your results.

Scalability
-----------
There are several synchronization primitives that are taken by the virtqueue
processing thread or the thread pool worker.  Unfortunately this is necessary
to protect shared state.  It means that thread pool workers contend on or at
least access thread synchronization primitives.  If anyone has suggestions for
improving this situation, please discuss.

1. vu_dispatch_rwlock - libvhost-user from races between the vhost-user
   protocol thread and the virtqueue processing and thread pool worker threads.

2. vq_lock - protects the virtqueue from races between the virtqueue processing
   thread and thread pool workers.

3. init_rwlock - protects FUSE_INIT/FUSE_DESTROY from races with other
   requests.

4. se->lock - protects se->list and the FUSE_INTERRUPT shared state.

Ideally we could avoid hitting all of these locks on each request.  That would
make the code scale better.

Future work
-----------
This series does not complete the multithreading effort yet.  Two items are
still missing:
1. Multiqueue support
2. CPU affinity for virtqueue processing threads and thread pools

Stefan Hajnoczi (4):
  virtiofsd: process requests in a thread pool
  virtiofsd: prevent FUSE_INIT/FUSE_DESTROY races
  virtiofsd: fix lo_destroy() resource leaks
  virtiofsd: add --thread-pool-size=NUM option

 contrib/virtiofsd/fuse_i.h         |   2 +
 contrib/virtiofsd/fuse_lowlevel.c  |  25 +-
 contrib/virtiofsd/fuse_virtio.c    | 491 ++++++++++++++++-------------
 contrib/virtiofsd/passthrough_ll.c |  43 ++-
 contrib/virtiofsd/seccomp.c        |   1 +
 5 files changed, 318 insertions(+), 244 deletions(-)

-- 
2.21.0