Re: [Qemu-devel] Linux kernel polling for QEMU

[Avi Kivity <avi@scylladb.com>]

On 12/02/2016 12:12 PM, Stefan Hajnoczi wrote:
> On Thu, Dec 01, 2016 at 09:35:27AM -0500, Paolo Bonzini wrote:
>>>>> Maybe we could do the same for sockets?  When data is available on a
>>>>> socket (or when it becomes writable), write to a user memory location.
>>>>>
>>>>> I, too, have an interest in polling; in my situation most of the polling
>>>>> happens in userspace.
>>>> You are trying to improve on the latency of non-blocking
>>>> ppoll(2)/epoll_wait(2) call?
>>> Yes, but the concern is for throughput, not latency.
>>>
>>> My main loop looks like
>>>
>>>      execute some tasks
>>>      poll from many sources
>>>
>>> Since epoll_wait(..., 0) has non-trivial costs, I have to limit the
>>> polling rate, and even so it shows up in the profile.  If the cost were
>>> lower, I could poll at a higher frequency, resulting in lower worst-case
>>> latencies for high-priority tasks.
>> IMHO, the ideal model wouldn't enter the kernel at all unless you _want_
>> to go to sleep.
> Something like mmapping an epoll file descriptor to get a ring of
> epoll_events.  It has to play nice with blocking epoll_wait(2) and
> ppoll(2) of an eventpoll file descriptor so that the userspace process
> can go to sleep without a lot of work to switch epoll wait vs mmap
> modes.

I think that's overkill; as soon as data is returned, you're going to be 
talking to the kernel in order to get it.  Besides, leave something for 
userspace tcp stack implementations.

> I'm bothered by the fact that the kernel will not be able to poll NIC or
> storage controller rings if userspace is doing the polling :(.


Yes, that's a concern.

We have three conflicting requirements:
  - fast inter-thread IPC (or guest/host IPC) -> shared memory+polling
  - kernel multiplexing of hardware -> kernel interfaces + kernel poll / 
sleep
  - composable interfaces that allow you to use the two methods

You could use something like Flow Director to move interesting flows 
directly to the application, let the application poll those queues (and 
maybe tell the kernel if something interesting happens), while the 
kernel manages the other queues.  It gets ugly very quickly.