From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Yang, Sheng" Subject: Re: Performance test result between virtio_pci MSI-X disable and enable Date: Wed, 1 Dec 2010 17:37:50 +0800 Message-ID: <201012011737.50596.sheng.yang@intel.com> References: <201012011702.57185.sheng.yang@intel.com> Mime-Version: 1.0 Content-Type: Text/Plain; charset=utf-8 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: Avi Kivity , Gleb Natapov , "mst@redhat.com" , "aliguori@us.ibm.com" , "rusty@rustcorp.com.au" , "kvm@vger.kernel.org" To: lidong chen Return-path: Received: from mga02.intel.com ([134.134.136.20]:3540 "EHLO mga02.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751118Ab0LAJgH convert rfc822-to-8bit (ORCPT ); Wed, 1 Dec 2010 04:36:07 -0500 In-Reply-To: Sender: kvm-owner@vger.kernel.org List-ID: On Wednesday 01 December 2010 17:29:44 lidong chen wrote: > maybe because i modify the code in assigned_dev_iomem_map(). >=20 > i used RHEL6, and calc_assigned_dev_id is below=EF=BC=9A >=20 > static uint32_t calc_assigned_dev_id(uint8_t bus, uint8_t devfn) > { > return (uint32_t)bus << 8 | (uint32_t)devfn; > } >=20 > and in patch there are there param. > + msix_mmio.id =3D calc_assigned_dev_id(r_dev->h_segnr= , > + r_dev->h_busnr, r_dev->h_devfn); This one should be fine because h_segnr should be 0 here. But I strongly recommend you to use latest KVM and latest QEmu, we won'= t know what=20 would happen during the rebase... (maybe my patch is a little old for t= he latest=20 one, so my kvm base is 365bb670a44b217870c2ee1065f57bb43b57e166, qemu b= ase is=20 420fe74769cc67baec6f3d962dc054e2972ca3ae). Things to be checked: 1. If two devices' MMIO have been registered successfully. 2. If you can see the mask bit accessing in kernel from both devices. -- regards Yang, Sheng >=20 >=20 > #ifdef KVM_CAP_MSIX_MASK > if (cap_mask) { > memset(&msix_mmio, 0, sizeof msix_mmio); > msix_mmio.id =3D calc_assigned_dev_id(r_dev->h_busnr, > r_dev->h_devfn); > msix_mmio.type =3D KVM_MSIX_TYPE_ASSIGNED_DEV; > msix_mmio.base_addr =3D e_phys + offset; > msix_mmio.max_entries_nr =3D r_dev->max_msix_entries_= nr; > msix_mmio.flags =3D KVM_MSIX_MMIO_FLAG_REGISTER; > ret =3D kvm_update_msix_mmio(kvm_context, &msix_mmio)= ; > if (ret) > fprintf(stderr, "fail to register in-kernel > msix_mmio!\n"); } > #endif >=20 > 2010/12/1 Yang, Sheng : > > On Wednesday 01 December 2010 16:54:16 lidong chen wrote: > >> yes, i patch qemu as well. > >>=20 > >> and i found the address of second vf is not in mmio range. the fir= st > >> one is fine. > >=20 > > So looks like something wrong with MMIO register part. Could you ch= eck > > the registeration in assigned_dev_iomem_map() of the 4th patch for = QEmu? > > I suppose something wrong with it. I would try to reproduce it here= =2E > >=20 > > And if you only use one vf, how about the gain? > >=20 > > -- > > regards > > Yang, Sheng > >=20 > >> 2010/12/1 Yang, Sheng : > >> > On Wednesday 01 December 2010 16:41:38 lidong chen wrote: > >> >> I used sr-iov, give each vm 2 vf. > >> >> after apply the patch, and i found performence is the same. > >> >>=20 > >> >> the reason is in function msix_mmio_write, mostly addr is not i= n mmio > >> >> range. > >> >=20 > >> > Did you patch qemu as well? You can see it's impossible for kern= el > >> > part to work alone... > >> >=20 > >> > http://www.mail-archive.com/kvm@vger.kernel.org/msg44368.html > >> >=20 > >> > -- > >> > regards > >> > Yang, Sheng > >> >=20 > >> >> static int msix_mmio_write(struct kvm_io_device *this, gpa_t ad= dr, > >> >> int len, const void *val) > >> >> { > >> >>=20 > >> >> struct kvm_assigned_dev_kernel *adev =3D > >> >> =20 > >> >> container_of(this, struct > >> >> kvm_assigned_dev_kernel, > >> >> =20 > >> >> msix_mmio_dev); > >> >> =20 > >> >> int idx, r =3D 0; > >> >> unsigned long new_val =3D *(unsigned long *)val; > >> >> =20 > >> >> mutex_lock(&adev->kvm->lock); > >> >> if (!msix_mmio_in_range(adev, addr, len)) { > >> >> =20 > >> >> // return here. > >> >> =20 > >> >> r =3D -EOPNOTSUPP; > >> >> =20 > >> >> goto out; > >> >> =20 > >> >> } > >> >>=20 > >> >> i printk the value: > >> >> addr start end len > >> >> F004C00C F0044000 F0044030 4 > >> >>=20 > >> >> 00:06.0 Ethernet controller: Intel Corporation Unknown device 1= 0ed > >> >> (rev 01) Subsystem: Intel Corporation Unknown device 000c > >> >>=20 > >> >> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoo= p- > >> >>=20 > >> >> ParErr- Stepping- SERR- FastB2B- > >> >>=20 > >> >> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=3Dfast >= TAbort- > >> >>=20 > >> >> SERR- >> >>=20 > >> >> Latency: 0 > >> >> Region 0: Memory at f0040000 (32-bit, non-prefetchable) > >> >> [size=3D16K] Region 3: Memory at f0044000 (32-bit, > >> >> non-prefetchable) [size=3D16K] Capabilities: [40] MSI-X: = Enable+ > >> >> Mask- TabSize=3D3 > >> >> =20 > >> >> Vector table: BAR=3D3 offset=3D00000000 > >> >> PBA: BAR=3D3 offset=3D00002000 > >> >>=20 > >> >> 00:07.0 Ethernet controller: Intel Corporation Unknown device 1= 0ed > >> >> (rev 01) Subsystem: Intel Corporation Unknown device 000c > >> >>=20 > >> >> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoo= p- > >> >>=20 > >> >> ParErr- Stepping- SERR- FastB2B- > >> >>=20 > >> >> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=3Dfast >= TAbort- > >> >>=20 > >> >> SERR- >> >>=20 > >> >> Latency: 0 > >> >> Region 0: Memory at f0048000 (32-bit, non-prefetchable) > >> >> [size=3D16K] Region 3: Memory at f004c000 (32-bit, > >> >> non-prefetchable) [size=3D16K] Capabilities: [40] MSI-X: = Enable+ > >> >> Mask- TabSize=3D3 > >> >> =20 > >> >> Vector table: BAR=3D3 offset=3D00000000 > >> >> PBA: BAR=3D3 offset=3D00002000 > >> >>=20 > >> >> +static bool msix_mmio_in_range(struct kvm_assigned_dev_kernel = *adev, > >> >> + gpa_t addr, int len) > >> >> +{ > >> >> + gpa_t start, end; > >> >> + > >> >> + BUG_ON(adev->msix_mmio_base =3D=3D 0); > >> >> + start =3D adev->msix_mmio_base; > >> >> + end =3D adev->msix_mmio_base + PCI_MSIX_ENTRY_SIZE * > >> >> + adev->msix_max_entries_nr; > >> >> + if (addr >=3D start && addr + len <=3D end) > >> >> + return true; > >> >> + > >> >> + return false; > >> >> +} > >> >>=20 > >> >> 2010/11/30 Yang, Sheng : > >> >> > On Tuesday 30 November 2010 17:10:11 lidong chen wrote: > >> >> >> sr-iov also meet this problem, MSIX mask waste a lot of cpu > >> >> >> resource. > >> >> >>=20 > >> >> >> I test kvm with sriov, which the vf driver could not disable= msix. > >> >> >> so the host os waste a lot of cpu. cpu rate of host os is 9= 0%. > >> >> >>=20 > >> >> >> then I test xen with sriov, there ara also a lot of vm exits > >> >> >> caused by MSIX mask. > >> >> >> but the cpu rate of xen and domain0 is less than kvm. cpu ra= te of > >> >> >> xen and domain0 is 60%. > >> >> >>=20 > >> >> >> without sr-iov, the cpu rate of xen and domain0 is higher th= an > >> >> >> kvm. > >> >> >>=20 > >> >> >> so i think the problem is kvm waste more cpu resource to dea= l with > >> >> >> MSIX mask. and we can see how xen deal with MSIX mask. > >> >> >>=20 > >> >> >> if this problem sloved, maybe with MSIX enabled, the perform= ace is > >> >> >> better. > >> >> >=20 > >> >> > Please refer to my posted patches for this issue. > >> >> >=20 > >> >> > http://www.spinics.net/lists/kvm/msg44992.html > >> >> >=20 > >> >> > -- > >> >> > regards > >> >> > Yang, Sheng > >> >> >=20 > >> >> >> 2010/11/23 Avi Kivity : > >> >> >> > On 11/23/2010 09:27 AM, lidong chen wrote: > >> >> >> >> can you tell me something about this problem. > >> >> >> >> thanks. > >> >> >> >=20 > >> >> >> > Which problem? > >> >> >> >=20 > >> >> >> > -- > >> >> >> > I have a truly marvellous patch that fixes the bug which t= his > >> >> >> > signature is too narrow to contain.