Re: [PATCH] vhost: Add polling mode

["Zhang Haoyu" <zhanghy@sangfor.com>]

Maybe tie a knot between "vhost-net scalability tuning: threading for many VMs" and "vhost: Add polling mode" is a good marriage,
because it's more possibility to get work to do with less polling time, so less cpu cycles waste.

Thanks,
Zhang Haoyu

>>>>>> > >>> Hello All,
>>>>>> > >>>
>>>>>> > >>> When vhost is waiting for buffers from the guest driver (e.g., more
>>>>>> > >>> packets
>>>>>> > >>> to send in vhost-net's transmit queue), it normally goes to sleep and
>>>>>> > >>> waits
>>>>>> > >>> for the guest to "kick" it. This kick involves a PIO in the guest, and
>>>>>> > >>> therefore an exit (and possibly userspace involvement in translating
>>>> > > this
>>>>>> > >>> PIO
>>>>>> > >>> exit into a file descriptor event), all of which hurts performance.
>>>>>> > >>>
>>>>>> > >>> If the system is under-utilized (has cpu time to spare), vhost can
>>>>>> > >>> continuously poll the virtqueues for new buffers, and avoid asking
>>>>>> > >>> the guest to kick us.
>>>>>> > >>> This patch adds an optional polling mode to vhost, that can be enabled
>>>>>> > >>> via a kernel module parameter, "poll_start_rate".
>>>>>> > >>>
>>>>>> > >>> When polling is active for a virtqueue, the guest is asked to
>>>>>> > >>> disable notification (kicks), and the worker thread continuously
>>>> > > checks
>>>>>> > >>> for
>>>>>> > >>> new buffers. When it does discover new buffers, it simulates a "kick"
>>>> > > by
>>>>>> > >>> invoking the underlying backend driver (such as vhost-net), which
>>>> > > thinks
>>>>>> > >>> it
>>>>>> > >>> got a real kick from the guest, and acts accordingly. If the
>>>> > > underlying
>>>>>> > >>> driver asks not to be kicked, we disable polling on this virtqueue.
>>>>>> > >>>
>>>>>> > >>> We start polling on a virtqueue when we notice it has
>>>>>> > >>> work to do. Polling on this virtqueue is later disabled after 3
>>>> > > seconds of
>>>>>> > >>> polling turning up no new work, as in this case we are better off
>>>>>> > >>> returning
>>>>>> > >>> to the exit-based notification mechanism. The default timeout of 3
>>>> > > seconds
>>>>>> > >>> can be changed with the "poll_stop_idle" kernel module parameter.
>>>>>> > >>>
>>>>>> > >>> This polling approach makes lot of sense for new HW with
>>>> > > posted-interrupts
>>>>>> > >>> for which we have exitless host-to-guest notifications. But even with
>>>>>> > >>> support
>>>>>> > >>> for posted interrupts, guest-to-host communication still causes exits.
>>>>>> > >>> Polling adds the missing part.
>>>>>> > >>>
>>>>>> > >>> When systems are overloaded, there won?t be enough cpu time for the
>>>>>> > >>> various
>>>>>> > >>> vhost threads to poll their guests' devices. For these scenarios, we
>>>> > > plan
>>>>>> > >>> to add support for vhost threads that can be shared by multiple
>>>> > > devices,
>>>>>> > >>> even of multiple vms.
>>>>>> > >>> Our ultimate goal is to implement the I/O acceleration features
>>>> > > described
>>>>>> > >>> in:
>>>>>> > >>> KVM Forum 2013: Efficient and Scalable Virtio (by Abel Gordon)
>>>>>> > >>> https://www.youtube.com/watch?v=9EyweibHfEs
>>>>>> > >>> and
>>>>>> > >>> https://www.mail-archive.com/kvm@vger.kernel.org/msg98179.html
>>>>>> > >>>
>>>>>> > >>>
>>>>>> > >>> Comments are welcome,
>>>>>> > >>> Thank you,
>>>>>> > >>> Razya
>>>>> > >> Thanks for the work. Do you have perf numbers for this?
>>>>> > >>
>>>> > > Hi Jason,
>>>> > > Thanks for reviewing. I ran some experiments with TCP stream netperf and
>>>> > > filebench (having 2 threads performing random reads) benchmarks on an IBM
>>>> > > System x3650 M4.
>>>> > > All runs loaded the guests in a way that they were (cpu) saturated.
>>>> > > The system had two cores per guest, as to allow for both the vcpu and the
>>>> > > vhost thread to
>>>> > > run concurrently for maximum throughput (but I didn't pin the threads to
>>>> > > specific cores)
>>>> > > I get:
>>>> > >
>>>> > > Netperf, 1 vm:
>>>> > > The polling patch improved throughput by ~33%. Number of exits/sec
>>>> > > decreased 6x.
>>>> > > The same improvement was shown when I tested with 3 vms running netperf.
>>>> > >
>>>> > > filebench, 1 vm:
>>>> > > ops/sec improved by 13% with the polling patch. Number of exits was
>>>> > > reduced by 31%.
>>>> > > The same experiment with 3 vms running filebench showed similar numbers.
>>> >
>>> > Looks good, may worth to add the result in the commit log.
>>>> > >
>>>>> > >> And looks like the patch only poll for virtqueue. In the future, may
>>>>> > >> worth to add callbacks for vhost_net to poll socket. Then it could be
>>>>> > >> used with rx busy polling in host which may speedup the rx also.
>>>> > > Did you mean polling the network device to avoid interrupts?
>>> >
>>> > Yes, recent linux host support rx busy polling which can reduce the
>>> > interrupts. If vhost can utilize this, it can also reduce the latency
>>> > caused by vhost thread wakeups.
>>> >
>>> > And I'm also working on virtio-net busy polling in guest, if vhost can
>>> > poll socket, it can also help in guest rx polling.
>> Nice :)  Note that you may want to check if if the processor support
>> posted interrupts. I guess that if CPU supports posted interrupts then
>> benefits of polling in the front-end (from performance perspective)
>> may not worth the cpu cycles wasted in the guest.
>>
>
>Yes it's worth to check. But I think busy polling in guest may still
>help since it may reduce the overhead of irq and NAPI in guest, also can
>reduce the latency by eliminating wakeups of both vcpu thread in host
>and userspace process in guest.