[Qemu-devel] [PATCH 0/2] iohandler: Convert to GSource and use epoll on Linux

[Fam Zheng <famz@redhat.com>]

ppoll is not scalable, it has a complexity of O(n). When we have many virtio
queues, the main loop could be slowed down.

epoll, which is O(1), could solve this problem well.

In order to do this, we need to factor out an interface between main loop and
iohandler. What we have now is not good, so patch 1 changed it to GSource,
which is attached to the main context. The posix implementation, which is
identical, attaches each iohandler fd to the iohandler GSrouce. They are
automatically polled by g_poll, or extracted to ppoll.

Patch 2 adds another GSource which attaches a single epoll fd. The epoll fd set
manages all the iohandler fds. Each time the epoll fd is poked by main loop
polling, we call epoll_wait on it to dispatch the fds that are ready to
read/write.

A pitfall is that certain type(s) of fds can't be added to a epoll set. One
such case is the normal file or here document stdin fd. We have to special case
it (where epoll_ctl returns -1 with errno equals to EPERM), and always notify
these fds in the dispatch function.

To show the difference, let's compare the iodepth=1 read test on virtio-blk
by toggling the virtio-serial module (which adds or removes tens of
ioeventfds).

Before:

case            rw         bs         bw         iops       latency   
----------------------------------------------------------------------
vserial=on      read       64k        1346       21548      45        
vserial=off     read       64k        1956       31305      30        

After:
case            rw         bs         bw         iops       latency   
----------------------------------------------------------------------
vserial=on      read       64k        1727       27647      34        
vserial=off     read       64k        1741       27868      34        

The vserial=on case is better than before because we turned "ppoll with many
fds" into ("ppoll with a few fds" + "epoll with a few fds").

The vserial=off case is worse than before because we turned "ppoll with a few
fds" into ("ppoll with a few fds" + "epoll with a few fds").

To be optimal in both cases, we may dynamically enable/disable epoll depending
on the active ioeventfds. But that's not done yet.

Fam


Fam Zheng (2):
  iohandler: Convert I/O handler to GSource
  iohandler: Add Linux implementation of iohandler GSource

 Makefile.objs            |   2 +
 include/qemu/iohandler.h |  65 +++++++++++++++
 include/qemu/main-loop.h |   2 -
 iohandler-linux.c        | 213 +++++++++++++++++++++++++++++++++++++++++++++++
 iohandler-posix.c        | 150 +++++++++++++++++++++++++++++++++
 iohandler.c              |  90 ++------------------
 main-loop.c              |   6 +-
 7 files changed, 442 insertions(+), 86 deletions(-)
 create mode 100644 include/qemu/iohandler.h
 create mode 100644 iohandler-linux.c
 create mode 100644 iohandler-posix.c

-- 
1.9.3