From mboxrd@z Thu Jan 1 00:00:00 1970 From: "=?utf-8?B?WmhhbmcgSGFveXU=?=" Subject: =?utf-8?B?UmU6IFtQQVRDSF0gdmhvc3Q6IEFkZCBwb2xsaW5nIG1vZGU=?= Date: Tue, 29 Jul 2014 09:30:34 +0800 Message-ID: <201407290930323715763@sangfor.com> References: , <53CF478C.5000403@redhat.com>, , <53CF756C.1050406@redhat.com>, Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Cc: "=?utf-8?B?UmF6eWEgTGFkZWxza3k=?=" , "=?utf-8?B?QWxleCBHbGlrc29u?=" , "=?utf-8?B?RXJhbiBSYWljaHN0ZWlu?=" , "=?utf-8?B?Sm9lbCBOaWRlcg==?=" , "=?utf-8?B?a3Zt?=" , "=?utf-8?B?TWljaGFlbCBTLiBUc2lya2lu?=" , "=?utf-8?B?WW9zc2kgS3VwZXJtYW4x?=" To: "=?utf-8?B?SmFzb24gV2FuZw==?=" , "=?utf-8?B?QWJlbCBHb3Jkb24=?=" Return-path: Received: from smtp.sanfor.com ([58.251.49.30]:54860 "EHLO mail.sangfor.com" rhost-flags-OK-FAIL-OK-OK) by vger.kernel.org with ESMTP id S1751213AbaG2BfU (ORCPT ); Mon, 28 Jul 2014 21:35:20 -0400 Sender: kvm-owner@vger.kernel.org List-ID: Maybe tie a knot between "vhost-net scalability tuning: threading for many VMs" and "vhost: Add polling mode" is a good marriage, because it's more possibility to get work to do with less polling time, so less cpu cycles waste. Thanks, Zhang Haoyu >>>>>> > >>> Hello All, >>>>>> > >>> >>>>>> > >>> When vhost is waiting for buffers from the guest driver (e.g., more >>>>>> > >>> packets >>>>>> > >>> to send in vhost-net's transmit queue), it normally goes to sleep and >>>>>> > >>> waits >>>>>> > >>> for the guest to "kick" it. This kick involves a PIO in the guest, and >>>>>> > >>> therefore an exit (and possibly userspace involvement in translating >>>> > > this >>>>>> > >>> PIO >>>>>> > >>> exit into a file descriptor event), all of which hurts performance. >>>>>> > >>> >>>>>> > >>> If the system is under-utilized (has cpu time to spare), vhost can >>>>>> > >>> continuously poll the virtqueues for new buffers, and avoid asking >>>>>> > >>> the guest to kick us. >>>>>> > >>> This patch adds an optional polling mode to vhost, that can be enabled >>>>>> > >>> via a kernel module parameter, "poll_start_rate". >>>>>> > >>> >>>>>> > >>> When polling is active for a virtqueue, the guest is asked to >>>>>> > >>> disable notification (kicks), and the worker thread continuously >>>> > > checks >>>>>> > >>> for >>>>>> > >>> new buffers. When it does discover new buffers, it simulates a "kick" >>>> > > by >>>>>> > >>> invoking the underlying backend driver (such as vhost-net), which >>>> > > thinks >>>>>> > >>> it >>>>>> > >>> got a real kick from the guest, and acts accordingly. If the >>>> > > underlying >>>>>> > >>> driver asks not to be kicked, we disable polling on this virtqueue. >>>>>> > >>> >>>>>> > >>> We start polling on a virtqueue when we notice it has >>>>>> > >>> work to do. Polling on this virtqueue is later disabled after 3 >>>> > > seconds of >>>>>> > >>> polling turning up no new work, as in this case we are better off >>>>>> > >>> returning >>>>>> > >>> to the exit-based notification mechanism. The default timeout of 3 >>>> > > seconds >>>>>> > >>> can be changed with the "poll_stop_idle" kernel module parameter. >>>>>> > >>> >>>>>> > >>> This polling approach makes lot of sense for new HW with >>>> > > posted-interrupts >>>>>> > >>> for which we have exitless host-to-guest notifications. But even with >>>>>> > >>> support >>>>>> > >>> for posted interrupts, guest-to-host communication still causes exits. >>>>>> > >>> Polling adds the missing part. >>>>>> > >>> >>>>>> > >>> When systems are overloaded, there won?t be enough cpu time for the >>>>>> > >>> various >>>>>> > >>> vhost threads to poll their guests' devices. For these scenarios, we >>>> > > plan >>>>>> > >>> to add support for vhost threads that can be shared by multiple >>>> > > devices, >>>>>> > >>> even of multiple vms. >>>>>> > >>> Our ultimate goal is to implement the I/O acceleration features >>>> > > described >>>>>> > >>> in: >>>>>> > >>> KVM Forum 2013: Efficient and Scalable Virtio (by Abel Gordon) >>>>>> > >>> https://www.youtube.com/watch?v=9EyweibHfEs >>>>>> > >>> and >>>>>> > >>> https://www.mail-archive.com/kvm@vger.kernel.org/msg98179.html >>>>>> > >>> >>>>>> > >>> >>>>>> > >>> Comments are welcome, >>>>>> > >>> Thank you, >>>>>> > >>> Razya >>>>> > >> Thanks for the work. Do you have perf numbers for this? >>>>> > >> >>>> > > Hi Jason, >>>> > > Thanks for reviewing. I ran some experiments with TCP stream netperf and >>>> > > filebench (having 2 threads performing random reads) benchmarks on an IBM >>>> > > System x3650 M4. >>>> > > All runs loaded the guests in a way that they were (cpu) saturated. >>>> > > The system had two cores per guest, as to allow for both the vcpu and the >>>> > > vhost thread to >>>> > > run concurrently for maximum throughput (but I didn't pin the threads to >>>> > > specific cores) >>>> > > I get: >>>> > > >>>> > > Netperf, 1 vm: >>>> > > The polling patch improved throughput by ~33%. Number of exits/sec >>>> > > decreased 6x. >>>> > > The same improvement was shown when I tested with 3 vms running netperf. >>>> > > >>>> > > filebench, 1 vm: >>>> > > ops/sec improved by 13% with the polling patch. Number of exits was >>>> > > reduced by 31%. >>>> > > The same experiment with 3 vms running filebench showed similar numbers. >>> > >>> > Looks good, may worth to add the result in the commit log. >>>> > > >>>>> > >> And looks like the patch only poll for virtqueue. In the future, may >>>>> > >> worth to add callbacks for vhost_net to poll socket. Then it could be >>>>> > >> used with rx busy polling in host which may speedup the rx also. >>>> > > Did you mean polling the network device to avoid interrupts? >>> > >>> > Yes, recent linux host support rx busy polling which can reduce the >>> > interrupts. If vhost can utilize this, it can also reduce the latency >>> > caused by vhost thread wakeups. >>> > >>> > And I'm also working on virtio-net busy polling in guest, if vhost can >>> > poll socket, it can also help in guest rx polling. >> Nice :) Note that you may want to check if if the processor support >> posted interrupts. I guess that if CPU supports posted interrupts then >> benefits of polling in the front-end (from performance perspective) >> may not worth the cpu cycles wasted in the guest. >> > >Yes it's worth to check. But I think busy polling in guest may still >help since it may reduce the overhead of irq and NAPI in guest, also can >reduce the latency by eliminating wakeups of both vcpu thread in host >and userspace process in guest.