Re: [Qemu-devel] Linux kernel polling for QEMU

[Stefan Hajnoczi <stefanha@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 6887 bytes --]

On Tue, Nov 29, 2016 at 09:19:22AM +0100, Christian Borntraeger wrote:
> On 11/24/2016 04:12 PM, Stefan Hajnoczi wrote:
> > I looked through the socket SO_BUSY_POLL and blk_mq poll support in
> > recent Linux kernels with an eye towards integrating the ongoing QEMU
> > polling work.  The main missing feature is eventfd polling support which
> > I describe below.
> > 
> > Background
> > ----------
> > We're experimenting with polling in QEMU so I wondered if there are
> > advantages to having the kernel do polling instead of userspace.
> > 
> > One such advantage has been pointed out by Christian Borntraeger and
> > Paolo Bonzini: a userspace thread spins blindly without knowing when it
> > is hogging a CPU that other tasks need.  The kernel knows when other
> > tasks need to run and can skip polling in that case.
> > 
> > Power management might also benefit if the kernel was aware of polling
> > activity on the system.  That way polling can be controlled by the
> > system administrator in a single place.  Perhaps smarter power saving
> > choices can also be made by the kernel.
> > 
> > Another advantage is that the kernel can poll hardware rings (e.g. NIC
> > rx rings) whereas QEMU can only poll its own virtual memory (including
> > guest RAM).  That means the kernel can bypass interrupts for devices
> > that are using kernel drivers.
> > 
> > State of polling in Linux
> > -------------------------
> > SO_BUSY_POLL causes recvmsg(2), select(2), and poll(2) family system
> > calls to spin awaiting new receive packets.  From what I can tell epoll
> > is not supported so that system call will sleep without polling.
> > 
> > blk_mq poll is mainly supported by NVMe.  It is only available with
> > synchronous direct I/O.  select(2), poll(2), epoll, and Linux AIO are
> > therefore not integrated.  It would be nice to extend the code so a
> > process waiting on Linux AIO using io_getevents(2), select(2), poll(2),
> > or epoll will poll.
> > 
> > QEMU and KVM-specific polling
> > -----------------------------
> > There are a few QEMU/KVM-specific items that require polling support:
> > 
> > QEMU's event loop aio_notify() mechanism wakes up the event loop from a
> > blocking poll(2) or epoll call.  It is used when another thread adds or
> > changes an event loop resource (such as scheduling a BH).  There is a
> > userspace memory location (ctx->notified) that is written by
> > aio_notify() as well as an eventfd that can be signalled.
> > 
> > kvm.ko's ioeventfd is signalled upon guest MMIO/PIO accesses.  Virtio
> > devices use ioeventfd as a doorbell after new requests have been placed
> > in a virtqueue, which is a descriptor ring in userspace memory.
> > 
> > Eventfd polling support could look like this:
> > 
> >   struct eventfd_poll_info poll_info = {
> >       .addr = ...memory location...,
> >       .size = sizeof(uint32_t),
> >       .op   = EVENTFD_POLL_OP_NOT_EQUAL, /* check *addr != val */
> >       .val  = ...last value...,
> >   };
> >   ioctl(eventfd, EVENTFD_SET_POLL, &poll_info);
> > 
> > In the kernel, eventfd stashes this information and eventfd_poll()
> > evaluates the operation (e.g. not equal, bitwise and, etc) to detect
> > progress.
> > 
> > Note that this eventfd polling mechanism doesn't actually poll the
> > eventfd counter value.  It's useful for situations where the eventfd is
> > a doorbell/notification that some object in userspace memory has been
> > updated.  So it polls that userspace memory location directly.
> > 
> > This new eventfd feature also provides a poor man's Linux AIO polling
> > support: set the Linux AIO shared ring index as the eventfd polling
> > memory location.  This is not as good as true Linux AIO polling support
> > where the kernel polls the NVMe, virtio_blk, etc ring since we'd still
> > rely on an interrupt to complete I/O requests.
> > 
> > Thoughts?
> 
> Would be an interesting excercise, but we should really try to avoid making
> the iothreads more costly. When I look at some of our measurements, I/O-wise
> we are  slightly behind z/VM, which can be tuned to be in a similar area but
> we use more host CPUs on s390 for the same throughput.
> 
> So I have two concerns and both a related to overhead.
> a: I am able to get a higher bandwidth and lower host cpu utilization
> when running fio for multiple disks when I pin the iothreads to a subset of
> the host CPUs (there is a sweet spot). Is the polling maybe just influencing
> the scheduler to do the same by making the iothread not doing sleep/wakeup
> all the time?

Interesting theory, look at sched_switch tracing data to find out
whether that is true.  Do you get any benefit from combining the sweet
spot pinning with polling?

> b: what about contention with other guests on the host? What
> worries me a bit, is the fact that most performance measurements and
> tunings are done for workloads without that. We (including myself) do our
> microbenchmarks (or fio runs) with just one guest and are happy if we see
> an improvement. But does that reflect real usage? For example have you ever
> measured the aio polling with 10 guests or so?
> My gut feeling (and obviously I have not done proper measurements myself) is
> that we want to stop polling as soon as there is contention.
> 
> As you outlined, we already have something in place in the kernel to stop
> polling
> 
> Interestingly enough, for SO_BUSY_POLL the network code seems to consider
>     !need_resched() && !signal_pending(current)
> for stopping the poll, which allows to consume your time slice. KVM instead
> uses single_task_running() for the halt_poll_thing. This means that KVM 
> yields much more aggressively, which is probably the right thing to do for
> opportunistic spinning.

Another thing I noticed about the busy_poll implementation is that it
will spin if *any* file descriptor supports polling.

In QEMU we decided to implement the opposite: spin only if *all* event
sources support polling.  The reason is that we don't want polling to
introduce any extra latency on the event sources that do not support
polling.

> Another thing to consider: In the kernel we have already other opportunistic
> spinners and we are in the process of making things less aggressive because
> it caused real issues. For example search for the  vcpu_is_preempted​ patch set.
> Which by the way shown another issue, running nested you do not only want to
> consider your own load, but also the load of the hypervisor.

These are good points and it's why I think polling in the kernel can
make smarter decisions than in polling userspace.  There are multiple
components in the system that can do polling, it would be best to have a
single place so that the polling activity doesn't interfere.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]