From mboxrd@z Thu Jan 1 00:00:00 1970 From: Li Liu Subject: Re: [Qemu-devel] The status about vhost-net on kvm-arm? Date: Fri, 17 Oct 2014 20:26:36 +0800 Message-ID: <54410AFC.1070307@huawei.com> References: <53E97EBD.6070401@huawei.com> <53EC31FF.4060903@huawei.com> <53ED78A8.2050209@huawei.com> <022C7612790E20489F80A6F0D54B849F3B25B51A@EXDAG0-B3.intra.cea.fr> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: "kvmarm@lists.cs.columbia.edu" , "kvm@vger.kernel.org" , qemu-devel To: =?UTF-8?B?R0FVR1VFWSBSw6lteSAyMjg4OTA=?= , "Yingshiuan Pan" Return-path: Received: from szxga01-in.huawei.com ([119.145.14.64]:25323 "EHLO szxga01-in.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751215AbaJQM0y (ORCPT ); Fri, 17 Oct 2014 08:26:54 -0400 In-Reply-To: <022C7612790E20489F80A6F0D54B849F3B25B51A@EXDAG0-B3.intra.cea.fr> Sender: kvm-owner@vger.kernel.org List-ID: On 2014/10/15 22:39, GAUGUEY R=C3=A9my 228890 wrote: > Hello, >=20 > Using this Qemu patchset as well as recent irqfd work, I=E2=80=99ve t= ried to make vhost-net working on Cortex-A15. > Unfortunately, even if I can correctly generate irqs to the guest thr= ough irqfd, it seems to me that some pieces are still missing=E2=80=A6. > Indeed, virtio mmio interrupt status register (@ offset 0x60) is not = updated by vhost thread, and reading it or writing to the peer interrup= t ack register (offset 0x64) from the guest causes an VM exit =E2=80=A6 >=20 Yeah, you are correct. But it's not far away from success if have injec= ted irqs to the guest through irqfd. Do below things to let guest receive packet= s correctly without checking VIRTIO_MMIO_INTERRUPT_STATUS in guest virtio= _mmio.c: static irqreturn_t vm_interrupt(int irq, void *opaque) { ...... /* Read and acknowledge interrupts */ /*status =3D readl(vm_dev->base + VIRTIO_MMIO_INTERRUPT_STATUS); writel(status, vm_dev->base + VIRTIO_MMIO_INTERRUPT_ACK); if (unlikely(status & VIRTIO_MMIO_INT_CONFIG) && vdrv && vdrv->config_changed) { vdrv->config_changed(&vm_dev->vdev); ret =3D IRQ_HANDLED; }*/ //if (likely(status & VIRTIO_MMIO_INT_VRING)) { spin_lock_irqsave(&vm_dev->lock, flags); list_for_each_entry(info, &vm_dev->virtqueues, node) ret |=3D vring_interrupt(irq, info->vq); spin_unlock_irqrestore(&vm_dev->lock, flags); //} return ret; } This is very roughly :), and a lot of coding things need to be done. Li. > After reading older posts, I understand that vhost-net with irqfd sup= port could only work with MSI-X support : >=20 > On 01/20/2011 09:35 AM, Michael S. Tsirkin wrote: > =E2=80=9CWhen MSI is off, each interrupt needs to be bounced through = the io thread when it's set/cleared, so vhost-net causes more context s= witches and > higher CPU utilization than userspace virtio which handles networking= in the same thread. > =E2=80=9C > Indeed, in case of MSI-X support, Virtio spec indicates that the ISR = Status field is unused=E2=80=A6 >=20 > I understand that Vhost does not emulate a complete virtio PCI adapte= r but only manage virtqueue operations. > However I don=E2=80=99t have a clear view of what is performed by Qem= u and what is performed by vhost-thread=E2=80=A6 > Could someone highlight me on this point, and maybe give some clues f= or an implementation of Vhost with irqfd and without MSI support ??? >=20 > Thanks a lot in advance. > Best regards. > R=C3=A9my >=20 >=20 >=20 > De : kvmarm-bounces@lists.cs.columbia.edu [mailto:kvmarm-bounces@list= s.cs.columbia.edu] De la part de Yingshiuan Pan > Envoy=C3=A9 : vendredi 15 ao=C3=BBt 2014 09:25 > =C3=80 : Li Liu > Cc : kvmarm@lists.cs.columbia.edu; kvm@vger.kernel.org; qemu-devel > Objet : Re: [Qemu-devel] The status about vhost-net on kvm-arm? >=20 > Hi, Li, >=20 > It's ok, I did get those mails from mailing list. I guess it was beca= use I did not subscribe some of mailing lists. >=20 > Currently, I think I will not have any plan to renew my patcheset sin= ce I have resigned from my previous company, I do not have Cortex-A15 p= latform to test/verify. >=20 > I'm fine with that, it would be great if you or someone can take it a= nd improve it. > Thanks. >=20 > ---- > Best Regards, > Yingshiuan Pan >=20 > 2014-08-15 11:04 GMT+08:00 Li Liu >: > Hi Ying-Shiuan Pan, >=20 > I don't know why for missing your mail in mailbox. Sorry about that. > The results of vhost-net performance have been attached in another ma= il. >=20 > Do you have a plan to renew your patchset to support irqfd. If not, > we will try to finish it based on yours. >=20 > On 2014/8/14 11:50, Li Liu wrote: >> >> >> On 2014/8/13 19:25, Nikolay Nikolaev wrote: >>> On Wed, Aug 13, 2014 at 12:10 PM, Nikolay Nikolaev >>> > wrote: >>>> On Tue, Aug 12, 2014 at 6:47 PM, Nikolay Nikolaev >>>> > wrote: >>>>> >>>>> Hello, >>>>> >>>>> >>>>> On Tue, Aug 12, 2014 at 5:41 AM, Li Liu > wrote: >>>>>> >>>>>> Hi all, >>>>>> >>>>>> Is anyone there can tell the current status of vhost-net on kvm-= arm? >>>>>> >>>>>> Half a year has passed from Isa Ansharullah asked this question: >>>>>> http://www.spinics.net/lists/kvm-arm/msg08152.html >>>>>> >>>>>> I have found two patches which have provided the kvm-arm support= of >>>>>> eventfd and irqfd: >>>>>> >>>>>> 1) [RFC PATCH 0/4] ARM: KVM: Enable the ioeventfd capability of = KVM on ARM >>>>>> http://lists.gnu.org/archive/html/qemu-devel/2014-01/msg01770.ht= ml >>>>>> >>>>>> 2) [RFC,v3] ARM: KVM: add irqfd and irq routing support >>>>>> https://patches.linaro.org/32261/ >>>>>> >>>>>> And there's a rough patch for qemu to support eventfd from Ying-= Shiuan Pan: >>>>>> >>>>>> [Qemu-devel] [PATCH 0/4] ioeventfd support for virtio-mmio >>>>>> https://lists.gnu.org/archive/html/qemu-devel/2014-02/msg00715.h= tml >>>>>> >>>>>> But there no any comments of this patch. And I can found nothing= about qemu >>>>>> to support irqfd. Do I lost the track? >>>>>> >>>>>> If nobody try to fix it. We have a plan to complete it about vir= tio-mmio >>>>>> supporing irqfd and multiqueue. >>>>>> >>>>>> >>>>> >>>>> we at Virtual Open Systems did some work and tested vhost-net on = ARM >>>>> back in March. >>>>> The setup was based on: >>>>> - host kernel with our ioeventfd patches: >>>>> http://www.spinics.net/lists/kvm-arm/msg08413.html >>>>> >>>>> - qemu with the aforementioned patches from Ying-Shiuan Pan >>>>> https://lists.gnu.org/archive/html/qemu-devel/2014-02/msg00715.ht= ml >>>>> >>>>> The testbed was ARM Chromebook with Exynos 5250, using a 1Gbps US= B3 >>>>> Ethernet adapter connected to a 1Gbps switch. I can't find the ac= tual >>>>> numbers but I remember that with multiple streams the gain was cl= early >>>>> seen. Note that it used the minimum required ioventfd implementat= ion >>>>> and not irqfd. >>>>> >>>>> I guess it is feasible to think that it all can be put together a= nd >>>>> rebased + the recent irqfd work. One can achiev even better >>>>> performance (because of the irqfd). >>>>> >>>> >>>> Managed to replicate the setup with the old versions e used in Mar= ch: >>>> >>>> Single stream from another machine to chromebook with 1Gbps USB3 >>>> Ethernet adapter. >>>> iperf -c
-P 1 -i 1 -p 5001 -f k -t 10 >>>> to HOST: 858316 Kbits/sec >>>> to GUEST: 761563 Kbits/sec >>> to GUEST vhost=3Doff: 508150 Kbits/sec >>>> >>>> 10 parallel streams >>>> iperf -c
-P 10 -i 1 -p 5001 -f k -t 10 >>>> to HOST: 842420 Kbits/sec >>>> to GUEST: 625144 Kbits/sec >>> to GUEST vhost=3Doff: 425276 Kbits/sec >> >> I have tested the same cases on a Hisilicon board (Cortex-A15@1G) >> with Integrated 1Gbps Ethernet adapter. >> >> iperf -c
-P 1 -i 1 -p 5001 -f M -t 10 >> to HOST: 906 Mbits/sec >> to GUEST: 562 Mbits/sec >> to GUEST vhost=3Doff: 340 Mbits/sec >> >> 10 parallel streams, the performance gets <10% plus: >> iperf -c
-P 10 -i 1 -p 5001 -f M -t 10 >> to HOST: 923 Mbits/sec >> to GUEST: 592 Mbits/sec >> to GUEST vhost=3Doff: 364 Mbits/sec >> >> I't easy to see vhost-net brings great performance improvements, >> almost 50%+. >> >> Li. >> >>>> >>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> kvmarm mailing list >>>>>> kvmarm@lists.cs.columbia.edu >>>>>> https://lists.cs.columbia.edu/mailman/listinfo/kvmarm >>>>> >>>>> >>>>> regards, >>>>> Nikolay Nikolaev >>>>> Virtual Open Systems >>> >>> . >>> >> >> >> >> . >> >=20