From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Michael S. Tsirkin" Subject: Re: Performance test result between virtio_pci MSI-X disable and enable Date: Thu, 2 Dec 2010 14:23:21 +0200 Message-ID: <20101202122321.GB2454@redhat.com> References: <20101201140357.GD12912@redhat.com> <201012020913.28475.sheng.yang@intel.com> <20101202094931.GC1154@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: "Yang, Sheng" , lidong chen , Avi Kivity , Gleb Natapov , "aliguori@us.ibm.com" , "rusty@rustcorp.com.au" , "kvm@vger.kernel.org" To: Sheng Yang Return-path: Received: from mx1.redhat.com ([209.132.183.28]:1025 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753358Ab0LBMXr (ORCPT ); Thu, 2 Dec 2010 07:23:47 -0500 Content-Disposition: inline In-Reply-To: Sender: kvm-owner@vger.kernel.org List-ID: On Thu, Dec 02, 2010 at 07:52:00PM +0800, Sheng Yang wrote: > On Thu, Dec 2, 2010 at 5:49 PM, Michael S. Tsirkin w= rote: > > On Thu, Dec 02, 2010 at 09:13:28AM +0800, Yang, Sheng wrote: > >> On Wednesday 01 December 2010 22:03:58 Michael S. Tsirkin wrote: > >> > On Wed, Dec 01, 2010 at 04:41:38PM +0800, lidong chen wrote: > >> > > I used sr-iov, give each vm 2 vf. > >> > > after apply the patch, and i found performence is the same. > >> > > > >> > > the reason is in function msix_mmio_write, mostly addr is not = in mmio > >> > > range. > >> > > > >> > > static int msix_mmio_write(struct kvm_io_device *this, gpa_t a= ddr, int > >> > > len, > >> > > > >> > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0const void *val) > >> > > > >> > > { > >> > > > >> > > =A0 struct kvm_assigned_dev_kernel *adev =3D > >> > > > >> > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 container_of(this, struct = kvm_assigned_dev_kernel, > >> > > > >> > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= msix_mmio_dev); > >> > > > >> > > =A0 int idx, r =3D 0; > >> > > =A0 unsigned long new_val =3D *(unsigned long *)val; > >> > > > >> > > =A0 mutex_lock(&adev->kvm->lock); > >> > > =A0 if (!msix_mmio_in_range(adev, addr, len)) { > >> > > > >> > > =A0 =A0 =A0 =A0 =A0 // return here. > >> > > > >> > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0r =3D -EOPNOTSUPP; > >> > > > >> > > =A0 =A0 =A0 =A0 =A0 goto out; > >> > > > >> > > =A0 } > >> > > > >> > > i printk the value: > >> > > addr =A0 =A0 =A0 =A0 =A0 =A0 start =A0 =A0 =A0 =A0 =A0 end =A0= =A0 =A0 =A0 =A0 len > >> > > F004C00C =A0 F0044000 =A0F0044030 =A0 =A0 4 > >> > > > >> > > 00:06.0 Ethernet controller: Intel Corporation Unknown device = 10ed (rev > >> > > 01) > >> > > > >> > > =A0 Subsystem: Intel Corporation Unknown device 000c > >> > > =A0 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop= - ParErr- > >> > > > >> > > Stepping- SERR- FastB2B- > >> > > > >> > > =A0 Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=3Dfast >T= Abort- > >> > > > >> > > SERR- >> > > > >> > > =A0 Latency: 0 > >> > > =A0 Region 0: Memory at f0040000 (32-bit, non-prefetchable) [s= ize=3D16K] > >> > > =A0 Region 3: Memory at f0044000 (32-bit, non-prefetchable) [s= ize=3D16K] > >> > > =A0 Capabilities: [40] MSI-X: Enable+ Mask- TabSize=3D3 > >> > > > >> > > =A0 =A0 =A0 =A0 =A0 Vector table: BAR=3D3 offset=3D00000000 > >> > > =A0 =A0 =A0 =A0 =A0 PBA: BAR=3D3 offset=3D00002000 > >> > > > >> > > 00:07.0 Ethernet controller: Intel Corporation Unknown device = 10ed (rev > >> > > 01) > >> > > > >> > > =A0 Subsystem: Intel Corporation Unknown device 000c > >> > > =A0 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop= - ParErr- > >> > > > >> > > Stepping- SERR- FastB2B- > >> > > > >> > > =A0 Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=3Dfast >T= Abort- > >> > > > >> > > SERR- >> > > > >> > > =A0 Latency: 0 > >> > > =A0 Region 0: Memory at f0048000 (32-bit, non-prefetchable) [s= ize=3D16K] > >> > > =A0 Region 3: Memory at f004c000 (32-bit, non-prefetchable) [s= ize=3D16K] > >> > > =A0 Capabilities: [40] MSI-X: Enable+ Mask- TabSize=3D3 > >> > > > >> > > =A0 =A0 =A0 =A0 =A0 Vector table: BAR=3D3 offset=3D00000000 > >> > > =A0 =A0 =A0 =A0 =A0 PBA: BAR=3D3 offset=3D00002000 > >> > > > >> > > +static bool msix_mmio_in_range(struct kvm_assigned_dev_kernel= *adev, > >> > > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 gpa_t addr, int = len) > >> > > +{ > >> > > + gpa_t start, end; > >> > > + > >> > > + BUG_ON(adev->msix_mmio_base =3D=3D 0); > >> > > + start =3D adev->msix_mmio_base; > >> > > + end =3D adev->msix_mmio_base + PCI_MSIX_ENTRY_SIZE * > >> > > + =A0 =A0 =A0 =A0 adev->msix_max_entries_nr; > >> > > + if (addr >=3D start && addr + len <=3D end) > >> > > + =A0 =A0 =A0 =A0 return true; > >> > > + > >> > > + return false; > >> > > +} > >> > > >> > Hmm, this check looks wrong to me: there's no guarantee > >> > that guest uses the first N entries in the table. > >> > E.g. it could use a single entry, but only the last one. > >> > >> Please check the PCI spec. > > > > > > This is pretty explicit in the spec: the the last paragraph in the = below: > > > > IMPLEMENTATION NOTE > > Handling MSI-X Vector Shortages > > > > Handling MSI-X Vector Shortages > > For the case where fewer vectors are allocated to a function than d= esired, >=20 > You may not notice the premise here. I noticed it. > Also check for "Table Size" would > help I think. It would help if msix_max_entries_nr was MSIX Table Size N (encoded in PCI config space as N - 1). Is this what it is? Maybe add this in some comment. > --=20 > regards, > Yang, Sheng >=20 > software- > > controlled aliasing as enabled by MSI-X is one approach for handlin= g the situation. For > > example, if a function supports five queues, each with an associate= d MSI-X table entry, but > > only three vectors are allocated, the function could be designed fo= r software still to configure > > all five table entries, assigning one or more vectors to multiple t= able entries. Software could > > assign the three vectors {A,B,C} to the five entries as ABCCC, ABBC= C, ABCBA, or other > > similar combinations. > > > > > > Alternatively, the function could be designed for software to confi= gure it (using a device- > > specific mechanism) to use only three queues and three MSI-X table = entries. Software could > > assign the three vectors {A,B,C} to the five entries as ABC--, A-B-= C, A--CB, or other similar > > combinations. > > > > > > > >> -- > >> regards > >> Yang, Sheng > >> > >> > >> > > 2010/11/30 Yang, Sheng : > >> > > > On Tuesday 30 November 2010 17:10:11 lidong chen wrote: > >> > > >> sr-iov also meet this problem, MSIX mask waste a lot of cpu= resource. > >> > > >> > >> > > >> I test kvm with sriov, which the vf driver could not disabl= e msix. > >> > > >> so the host os waste a lot of cpu. =A0cpu rate of host os i= s 90%. > >> > > >> > >> > > >> then I test xen with sriov, there ara also a lot of vm exit= s caused by > >> > > >> MSIX mask. > >> > > >> but the cpu rate of xen and domain0 is less than kvm. cpu r= ate of xen > >> > > >> and domain0 is 60%. > >> > > >> > >> > > >> without sr-iov, the cpu rate of xen and domain0 is higher t= han kvm. > >> > > >> > >> > > >> so i think the problem is kvm waste more cpu resource to de= al with > >> > > >> MSIX mask. and we can see how xen deal with MSIX mask. > >> > > >> > >> > > >> if this problem sloved, maybe with MSIX enabled, the perfor= mace is > >> > > >> better. > >> > > > > >> > > > Please refer to my posted patches for this issue. > >> > > > > >> > > > http://www.spinics.net/lists/kvm/msg44992.html > >> > > > > >> > > > -- > >> > > > regards > >> > > > Yang, Sheng > >> > > > > >> > > >> 2010/11/23 Avi Kivity : > >> > > >> > On 11/23/2010 09:27 AM, lidong chen wrote: > >> > > >> >> can you tell me something about this problem. > >> > > >> >> thanks. > >> > > >> > > >> > > >> > Which problem? > >> > > >> > > >> > > >> > -- > >> > > >> > I have a truly marvellous patch that fixes the bug which = this > >> > > >> > signature is too narrow to contain. > > -- > > To unsubscribe from this list: send the line "unsubscribe kvm" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at =A0http://vger.kernel.org/majordomo-info.htm= l > >