Re: Performance test result between virtio_pci MSI-X disable and enable

["Michael S. Tsirkin" <mst@redhat.com>]

On Thu, Dec 02, 2010 at 07:52:00PM +0800, Sheng Yang wrote:
> On Thu, Dec 2, 2010 at 5:49 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
> > On Thu, Dec 02, 2010 at 09:13:28AM +0800, Yang, Sheng wrote:
> >> On Wednesday 01 December 2010 22:03:58 Michael S. Tsirkin wrote:
> >> > On Wed, Dec 01, 2010 at 04:41:38PM +0800, lidong chen wrote:
> >> > > I used sr-iov, give each vm 2 vf.
> >> > > after apply the patch, and i found performence is the same.
> >> > >
> >> > > the reason is in function msix_mmio_write, mostly addr is not in mmio
> >> > > range.
> >> > >
> >> > > static int msix_mmio_write(struct kvm_io_device *this, gpa_t addr, int
> >> > > len,
> >> > >
> >> > >                      const void *val)
> >> > >
> >> > > {
> >> > >
> >> > >   struct kvm_assigned_dev_kernel *adev =
> >> > >
> >> > >                   container_of(this, struct kvm_assigned_dev_kernel,
> >> > >
> >> > >                                msix_mmio_dev);
> >> > >
> >> > >   int idx, r = 0;
> >> > >   unsigned long new_val = *(unsigned long *)val;
> >> > >
> >> > >   mutex_lock(&adev->kvm->lock);
> >> > >   if (!msix_mmio_in_range(adev, addr, len)) {
> >> > >
> >> > >           // return here.
> >> > >
> >> > >                  r = -EOPNOTSUPP;
> >> > >
> >> > >           goto out;
> >> > >
> >> > >   }
> >> > >
> >> > > i printk the value:
> >> > > addr             start           end           len
> >> > > F004C00C   F0044000  F0044030     4
> >> > >
> >> > > 00:06.0 Ethernet controller: Intel Corporation Unknown device 10ed (rev
> >> > > 01)
> >> > >
> >> > >   Subsystem: Intel Corporation Unknown device 000c
> >> > >   Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> >> > >
> >> > > Stepping- SERR- FastB2B-
> >> > >
> >> > >   Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> >> > >
> >> > > <TAbort- <MAbort- >SERR- <PERR-
> >> > >
> >> > >   Latency: 0
> >> > >   Region 0: Memory at f0040000 (32-bit, non-prefetchable) [size=16K]
> >> > >   Region 3: Memory at f0044000 (32-bit, non-prefetchable) [size=16K]
> >> > >   Capabilities: [40] MSI-X: Enable+ Mask- TabSize=3
> >> > >
> >> > >           Vector table: BAR=3 offset=00000000
> >> > >           PBA: BAR=3 offset=00002000
> >> > >
> >> > > 00:07.0 Ethernet controller: Intel Corporation Unknown device 10ed (rev
> >> > > 01)
> >> > >
> >> > >   Subsystem: Intel Corporation Unknown device 000c
> >> > >   Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> >> > >
> >> > > Stepping- SERR- FastB2B-
> >> > >
> >> > >   Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> >> > >
> >> > > <TAbort- <MAbort- >SERR- <PERR-
> >> > >
> >> > >   Latency: 0
> >> > >   Region 0: Memory at f0048000 (32-bit, non-prefetchable) [size=16K]
> >> > >   Region 3: Memory at f004c000 (32-bit, non-prefetchable) [size=16K]
> >> > >   Capabilities: [40] MSI-X: Enable+ Mask- TabSize=3
> >> > >
> >> > >           Vector table: BAR=3 offset=00000000
> >> > >           PBA: BAR=3 offset=00002000
> >> > >
> >> > > +static bool msix_mmio_in_range(struct kvm_assigned_dev_kernel *adev,
> >> > > +                       gpa_t addr, int len)
> >> > > +{
> >> > > + gpa_t start, end;
> >> > > +
> >> > > + BUG_ON(adev->msix_mmio_base == 0);
> >> > > + start = adev->msix_mmio_base;
> >> > > + end = adev->msix_mmio_base + PCI_MSIX_ENTRY_SIZE *
> >> > > +         adev->msix_max_entries_nr;
> >> > > + if (addr >= start && addr + len <= end)
> >> > > +         return true;
> >> > > +
> >> > > + return false;
> >> > > +}
> >> >
> >> > Hmm, this check looks wrong to me: there's no guarantee
> >> > that guest uses the first N entries in the table.
> >> > E.g. it could use a single entry, but only the last one.
> >>
> >> Please check the PCI spec.
> >
> >
> > This is pretty explicit in the spec: the the last paragraph in the below:
> >
> > IMPLEMENTATION NOTE
> > Handling MSI-X Vector Shortages
> >
> > Handling MSI-X Vector Shortages
> > For the case where fewer vectors are allocated to a function than desired,
> 
> You may not notice the premise here.

I noticed it.

> Also check for "Table Size" would
> help I think.

It would help if msix_max_entries_nr was MSIX Table Size N
(encoded in PCI config space as  N - 1).
Is this what it is? Maybe add this in some comment.

> -- 
> regards,
> Yang, Sheng
> 
> software-
> > controlled aliasing as enabled by MSI-X is one approach for handling the situation. For
> > example, if a function supports five queues, each with an associated MSI-X table entry, but
> > only three vectors are allocated, the function could be designed for software still to configure
> > all five table entries, assigning one or more vectors to multiple table entries. Software could
> > assign the three vectors {A,B,C} to the five entries as ABCCC, ABBCC, ABCBA, or other
> > similar combinations.
> >
> >
> > Alternatively, the function could be designed for software to configure it (using a device-
> > specific mechanism) to use only three queues and three MSI-X table entries. Software could
> > assign the three vectors {A,B,C} to the five entries as ABC--, A-B-C, A--CB, or other similar
> > combinations.
> >
> >
> >
> >> --
> >> regards
> >> Yang, Sheng
> >>
> >>
> >> > > 2010/11/30 Yang, Sheng <sheng.yang@intel.com>:
> >> > > > On Tuesday 30 November 2010 17:10:11 lidong chen wrote:
> >> > > >> sr-iov also meet this problem, MSIX mask waste a lot of cpu resource.
> >> > > >>
> >> > > >> I test kvm with sriov, which the vf driver could not disable msix.
> >> > > >> so the host os waste a lot of cpu.  cpu rate of host os is 90%.
> >> > > >>
> >> > > >> then I test xen with sriov, there ara also a lot of vm exits caused by
> >> > > >> MSIX mask.
> >> > > >> but the cpu rate of xen and domain0 is less than kvm. cpu rate of xen
> >> > > >> and domain0 is 60%.
> >> > > >>
> >> > > >> without sr-iov, the cpu rate of xen and domain0 is higher than kvm.
> >> > > >>
> >> > > >> so i think the problem is kvm waste more cpu resource to deal with
> >> > > >> MSIX mask. and we can see how xen deal with MSIX mask.
> >> > > >>
> >> > > >> if this problem sloved, maybe with MSIX enabled, the performace is
> >> > > >> better.
> >> > > >
> >> > > > Please refer to my posted patches for this issue.
> >> > > >
> >> > > > http://www.spinics.net/lists/kvm/msg44992.html
> >> > > >
> >> > > > --
> >> > > > regards
> >> > > > Yang, Sheng
> >> > > >
> >> > > >> 2010/11/23 Avi Kivity <avi@redhat.com>:
> >> > > >> > On 11/23/2010 09:27 AM, lidong chen wrote:
> >> > > >> >> can you tell me something about this problem.
> >> > > >> >> thanks.
> >> > > >> >
> >> > > >> > Which problem?
> >> > > >> >
> >> > > >> > --
> >> > > >> > I have a truly marvellous patch that fixes the bug which this
> >> > > >> > signature is too narrow to contain.
> > --
> > To unsubscribe from this list: send the line "unsubscribe kvm" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >