Re: Performance test result between virtio_pci MSI-X disable and enable

["Yang, Sheng" <sheng.yang@intel.com>]

On Wednesday 01 December 2010 17:02:57 Yang, Sheng wrote:
> On Wednesday 01 December 2010 16:54:16 lidong chen wrote:
> > yes, i patch qemu as well.
> > 
> > and i found the address of second vf is not in mmio range. the first
> > one is fine.
> 
> So looks like something wrong with MMIO register part. Could you check the
> registeration in assigned_dev_iomem_map() of the 4th patch for QEmu? I
> suppose something wrong with it. I would try to reproduce it here.
> 
> And if you only use one vf, how about the gain?

It's weird... I've tried assign two vfs to the guest, and two devices' MSI-X mask 
bit accessing both being intercepted by kernel as expected...

So for msix_mmio_write/read, you can't see any mask bit accessing from the second 
device? 

Also the benefit of this patch would show only when mask bit operation intensity is 
high. So how about your interrupt intensity?

--
regards
Yang, Sheng

> 
> --
> regards
> Yang, Sheng
> 
> > 2010/12/1 Yang, Sheng <sheng.yang@intel.com>:
> > > On Wednesday 01 December 2010 16:41:38 lidong chen wrote:
> > >> I used sr-iov, give each vm 2 vf.
> > >> after apply the patch, and i found performence is the same.
> > >> 
> > >> the reason is in function msix_mmio_write, mostly addr is not in mmio
> > >> range.
> > > 
> > > Did you patch qemu as well? You can see it's impossible for kernel part
> > > to work alone...
> > > 
> > > http://www.mail-archive.com/kvm@vger.kernel.org/msg44368.html
> > > 
> > > --
> > > regards
> > > Yang, Sheng
> > > 
> > >> static int msix_mmio_write(struct kvm_io_device *this, gpa_t addr, int
> > >> len, const void *val)
> > >> {
> > >> 
> > >>       struct kvm_assigned_dev_kernel *adev =
> > >>       
> > >>                       container_of(this, struct
> > >>                       kvm_assigned_dev_kernel,
> > >>                       
> > >>                                    msix_mmio_dev);
> > >>       
> > >>       int idx, r = 0;
> > >>       unsigned long new_val = *(unsigned long *)val;
> > >>       
> > >>       mutex_lock(&adev->kvm->lock);
> > >>       if (!msix_mmio_in_range(adev, addr, len)) {
> > >>       
> > >>               // return here.
> > >>               
> > >>                  r = -EOPNOTSUPP;
> > >>               
> > >>               goto out;
> > >>       
> > >>       }
> > >> 
> > >> i printk the value:
> > >> addr             start           end           len
> > >> F004C00C   F0044000  F0044030     4
> > >> 
> > >> 00:06.0 Ethernet controller: Intel Corporation Unknown device 10ed
> > >> (rev 01) Subsystem: Intel Corporation Unknown device 000c
> > >> 
> > >>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> > >> 
> > >> ParErr- Stepping- SERR- FastB2B-
> > >> 
> > >>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> > >> 
> > >> <TAbort- <MAbort- >SERR- <PERR-
> > >> 
> > >>       Latency: 0
> > >>       Region 0: Memory at f0040000 (32-bit, non-prefetchable)
> > >>       [size=16K] Region 3: Memory at f0044000 (32-bit,
> > >>       non-prefetchable) [size=16K] Capabilities: [40] MSI-X: Enable+
> > >>       Mask- TabSize=3
> > >>       
> > >>               Vector table: BAR=3 offset=00000000
> > >>               PBA: BAR=3 offset=00002000
> > >> 
> > >> 00:07.0 Ethernet controller: Intel Corporation Unknown device 10ed
> > >> (rev 01) Subsystem: Intel Corporation Unknown device 000c
> > >> 
> > >>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> > >> 
> > >> ParErr- Stepping- SERR- FastB2B-
> > >> 
> > >>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> > >> 
> > >> <TAbort- <MAbort- >SERR- <PERR-
> > >> 
> > >>       Latency: 0
> > >>       Region 0: Memory at f0048000 (32-bit, non-prefetchable)
> > >>       [size=16K] Region 3: Memory at f004c000 (32-bit,
> > >>       non-prefetchable) [size=16K] Capabilities: [40] MSI-X: Enable+
> > >>       Mask- TabSize=3
> > >>       
> > >>               Vector table: BAR=3 offset=00000000
> > >>               PBA: BAR=3 offset=00002000
> > >> 
> > >> +static bool msix_mmio_in_range(struct kvm_assigned_dev_kernel *adev,
> > >> +                           gpa_t addr, int len)
> > >> +{
> > >> +     gpa_t start, end;
> > >> +
> > >> +     BUG_ON(adev->msix_mmio_base == 0);
> > >> +     start = adev->msix_mmio_base;
> > >> +     end = adev->msix_mmio_base + PCI_MSIX_ENTRY_SIZE *
> > >> +             adev->msix_max_entries_nr;
> > >> +     if (addr >= start && addr + len <= end)
> > >> +             return true;
> > >> +
> > >> +     return false;
> > >> +}
> > >> 
> > >> 2010/11/30 Yang, Sheng <sheng.yang@intel.com>:
> > >> > On Tuesday 30 November 2010 17:10:11 lidong chen wrote:
> > >> >> sr-iov also meet this problem, MSIX mask waste a lot of cpu
> > >> >> resource.
> > >> >> 
> > >> >> I test kvm with sriov, which the vf driver could not disable msix.
> > >> >> so the host os waste a lot of cpu.  cpu rate of host os is 90%.
> > >> >> 
> > >> >> then I test xen with sriov, there ara also a lot of vm exits caused
> > >> >> by MSIX mask.
> > >> >> but the cpu rate of xen and domain0 is less than kvm. cpu rate of
> > >> >> xen and domain0 is 60%.
> > >> >> 
> > >> >> without sr-iov, the cpu rate of xen and domain0 is higher than kvm.
> > >> >> 
> > >> >> so i think the problem is kvm waste more cpu resource to deal with
> > >> >> MSIX mask. and we can see how xen deal with MSIX mask.
> > >> >> 
> > >> >> if this problem sloved, maybe with MSIX enabled, the performace is
> > >> >> better.
> > >> > 
> > >> > Please refer to my posted patches for this issue.
> > >> > 
> > >> > http://www.spinics.net/lists/kvm/msg44992.html
> > >> > 
> > >> > --
> > >> > regards
> > >> > Yang, Sheng
> > >> > 
> > >> >> 2010/11/23 Avi Kivity <avi@redhat.com>:
> > >> >> > On 11/23/2010 09:27 AM, lidong chen wrote:
> > >> >> >> can you tell me something about this problem.
> > >> >> >> thanks.
> > >> >> > 
> > >> >> > Which problem?
> > >> >> > 
> > >> >> > --
> > >> >> > I have a truly marvellous patch that fixes the bug which this
> > >> >> > signature is too narrow to contain.