Re: [PATCH 13/16] kvm: enable MSI-X capabilty for assigned device

[Sheng Yang <sheng@linux.intel.com>]

On Wednesday 08 April 2009 00:38:10 Alex Williamson wrote:
> On Tue, 2009-04-07 at 14:09 +0800, Sheng Yang wrote:
> > On Saturday 04 April 2009 05:27:43 Alex Williamson wrote:
> > > Do we need some disable logic here?  If I toggle a bnx2 NIC in a guest,
> > > I get the following when it attempts to come back up:
> > >
> > > MSI-X entry number is zero!
> > > assigned_dev_update_msix_mmio: No such device or address
> >
> > It seems that driver didn't fill the MMIO with any correct MSIX
> > information, or the program fail to intercept it after driver set enable
> > bit of MSIX. It's strange... (Have it got something to do with PM and
> > some EXP feature you mentioned?)
>
> My guess was that it filled in the MSIX info, but then can't find a free
> slot to reload the MSIX data when it tries to re-enable MSIX.  I hacked
> the bnx2 driver to not rely on PM and EXP capabilities for this test, it
> seems to work, but it's possible that I broke something.  My host also
> locks up the second time I try to export this device to a guest, maybe a
> problem with my bnx2 hacks, MSIX not getting reset, or prototype
> hardware.  I'll see if I can find another MSIX capable device to export
> to a guest.
>
> > Could you enable DEVICE_ASSSIGNMENT_DEBUG=1 in
> > qemu/hw/device-assignment.c and post the output?
>
> Yup, see below.  The error comes after I 'ifdown eth0; ifup eth0' in the
> guest.  Note bnx2 appears to only turn on MSIX for SMP systems.  Thanks,
>
> Alex

Seems your "ifdown/ifup" script reload the module? Oh god, I found one bug 
after checked the spec:

System software reads this field to determine the MSI-X Table Size *N*, which 
is encoded as *N-1*. For example, a returned value of “00000000011” indicates 
a table size of 4.

But it seems still can't explain the problem...(OK, it may affect the guest in 
a unknown way as well...) I would post a fix for it soon.

> val=0x00000008 len=2 assigned_dev_pci_write_config: (4.0): address=0052
> val=0x00000008 len=2 assigned_dev_pci_read_config: (4.0): address=0006
> val=0x00000010 len=2 assigned_dev_pci_read_config: (4.0): address=0034
> val=0x00000040 len=1 assigned_dev_pci_read_config: (4.0): address=0040
> val=0x00000005 len=1 assigned_dev_pci_read_config: (4.0): address=0041
> val=0x00000050 len=1 assigned_dev_pci_read_config: (4.0): address=0050
> val=0x00000011 len=1 assigned_dev_pci_read_config: (4.0): address=0052
> val=0x00000008 len=2 assigned_dev_pci_read_config: (4.0): address=0054
> val=0x0000c000 len=4 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x0, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table
> mmio offset 0x4, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x8, val 0x4191 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x10, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table
> mmio offset 0x14, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x18, val 0x4199 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x20, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table
> mmio offset 0x24, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x28, val 0x41a1 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x30, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table
> mmio offset 0x34, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x38, val 0x41a9 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x40, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table
> mmio offset 0x44, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x48, val 0x41b1 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x50, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table
> mmio offset 0x54, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x58, val 0x41b9 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x60, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table
> mmio offset 0x64, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x68, val 0x41c1 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x70, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table
> mmio offset 0x74, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x78, val 0x41c9 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x80, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table
> mmio offset 0x84, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x88, val 0x41d1 assigned_dev_pci_read_config: (4.0): address=0004
> val=0x00000046 len=2 assigned_dev_pci_write_config: (4.0): address=0004
> val=0x00000446 len=2 assigned_dev_pci_write_config: NON BAR (4.0):

The writing to MMIO have been intercepted, but code fail to count it? 
Strange...

Could you try this debug?

diff --git a/qemu/hw/device-assignment.c b/qemu/hw/device-assignment.c
index 09e54ae..ba31bed 100644
--- a/qemu/hw/device-assignment.c
+++ b/qemu/hw/device-assignment.c
@@ -45,7 +45,7 @@
 #define IORESOURCE_DMA      0x00000800
 #define IORESOURCE_PREFETCH 0x00001000  /* No side effects */

-/* #define DEVICE_ASSIGNMENT_DEBUG 1 */
+#define DEVICE_ASSIGNMENT_DEBUG 1

 #ifdef DEVICE_ASSIGNMENT_DEBUG
 #define DEBUG(fmt, ...)                                       \
@@ -816,9 +816,13 @@ static int assigned_dev_update_msix_mmio(PCIDevice 
*pci_dev)
     else
         pos = pci_dev->cap.start;

+    fprintf(stderr, "the MSIX capabilty position is 0x%lx\n", pos);
+
     entries_max_nr = pci_dev->config[pos + 2];
     entries_max_nr &= PCI_MSIX_TABSIZE;

+    fprintf(stderr, "the MSIX entries_max_nr is 0x%lx\n", entries_max_nr);
+
     /* Get the usable entry number for allocating */
     for (i = 0; i < entries_max_nr; i++) {
         memcpy(&msg_ctrl, va + i * 16 + 12, 4);

(Further more, you can check the content of msg_ctrl when code try to count 
them). And also please try the fix for MSI-X table size.

Thanks!

-- 
regards
Yang, Sheng