[Qemu-devel] [RFC PATCH 0/4] MSI affinity for assigned devices

[Qemu-devel] [RFC PATCH 0/4] MSI affinity for assigned devices

[Alex Williamson]

This is post-1.3 material, so I'll just post it as an RFC for now.

MSI routing updates aren't currently handled by pci-assign or
vfio-pci (when using KVM acceleration), which means that trying to
set interrupt SMP affinity in the guest has no effect unless MSI is
completely disabled and re-enabled.  This series fixes this for both
device assignment backends using similar schemes.  We store the last
MSIMessage programmed to KVM and do updates to the MSI route when it
changes.  pci-assign takes a little bit of refactoring to make this
happen cleanly.  Thanks,

Alex

---

Alex Williamson (4):
      kvm: Stub kvm_irqchip_update_msi_route
      vfio-pci: Add support for MSI affinity
      pci-assign: Refactor MSI virq array
      pci-assign: Add MSI affinity support


 hw/kvm/pci-assign.c |   80 +++++++++++++++++++++++++++++++++++----------------
 hw/vfio_pci.c       |   31 ++++++++++++++++++--
 kvm-all.c           |    5 +++
 kvm-stub.c          |    5 +++
 4 files changed, 93 insertions(+), 28 deletions(-)

[Qemu-devel] [RFC PATCH 1/4] kvm: Stub kvm_irqchip_update_msi_route

[Alex Williamson]

Its friends are already stubbed out, add this one too so vfio can
call it without wrapping it in #ifdefs.

Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
---
 kvm-all.c  |    5 +++++
 kvm-stub.c |    5 +++++
 2 files changed, 10 insertions(+)

diff --git a/kvm-all.c b/kvm-all.c
index 3bc3347..93e9f0e 100644
--- a/kvm-all.c
+++ b/kvm-all.c
@@ -1179,6 +1179,11 @@ int kvm_irqchip_add_msi_route(KVMState *s, MSIMessage msg)
     return -ENOSYS;
 }
 
+int kvm_irqchip_update_msi_route(KVMState *s, int virq, MSIMessage msg)
+{
+    return -ENOSYS;
+}
+
 static int kvm_irqchip_assign_irqfd(KVMState *s, int fd, int virq, bool assign)
 {
     abort();
diff --git a/kvm-stub.c b/kvm-stub.c
index a3455e2..3a3bf85 100644
--- a/kvm-stub.c
+++ b/kvm-stub.c
@@ -127,6 +127,11 @@ int kvm_irqchip_add_msi_route(KVMState *s, MSIMessage msg)
     return -ENOSYS;
 }
 
+int kvm_irqchip_update_msi_route(KVMState *s, int virq, MSIMessage msg)
+{
+    return -ENOSYS;
+}
+
 void kvm_irqchip_release_virq(KVMState *s, int virq)
 {
 }

[Qemu-devel] [RFC PATCH 2/4] vfio-pci: Add support for MSI affinity

[Alex Williamson]

When MSI is accelerated through KVM the vectors are only programmed
when the guest first enables MSI support.  Subsequent writes to the
vector address or data fields are ignored.  Unfortunately that means
we're ignore updates done to adjust SMP affinity of the vectors.
MSI SMP affinity already works in non-KVM mode because the address
and data fields are read from their backing store on each interrupt.

This patch stores the MSIMessage programmed into KVM so that we can
determine when changes are made and update the routes.  The message
is stored for both MSI and MSI-X for consistency, but we only have
a use for them in MSI mode.

Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
---
 hw/vfio_pci.c |   31 ++++++++++++++++++++++++++++---
 1 file changed, 28 insertions(+), 3 deletions(-)

diff --git a/hw/vfio_pci.c b/hw/vfio_pci.c
index 7c27834..49b9550 100644
--- a/hw/vfio_pci.c
+++ b/hw/vfio_pci.c
@@ -75,6 +75,7 @@ struct VFIODevice;
 typedef struct VFIOMSIVector {
     EventNotifier interrupt; /* eventfd triggered on interrupt */
     struct VFIODevice *vdev; /* back pointer to device */
+    MSIMessage msg; /* cache the MSI message so we know when it changes */
     int virq; /* KVM irqchip route for QEMU bypass */
     bool use;
 } VFIOMSIVector;
@@ -574,6 +575,7 @@ static int vfio_msix_vector_use(PCIDevice *pdev,
 
     vector = &vdev->msi_vectors[nr];
     vector->vdev = vdev;
+    vector->msg = msg;
     vector->use = true;
 
     msix_vector_use(pdev, nr);
@@ -716,7 +718,6 @@ retry:
     vdev->msi_vectors = g_malloc0(vdev->nr_vectors * sizeof(VFIOMSIVector));
 
     for (i = 0; i < vdev->nr_vectors; i++) {
-        MSIMessage msg;
         VFIOMSIVector *vector = &vdev->msi_vectors[i];
 
         vector->vdev = vdev;
@@ -726,13 +727,13 @@ retry:
             error_report("vfio: Error: event_notifier_init failed\n");
         }
 
-        msg = msi_get_message(&vdev->pdev, i);
+        vector->msg = msi_get_message(&vdev->pdev, i);
 
         /*
          * Attempt to enable route through KVM irqchip,
          * default to userspace handling if unavailable.
          */
-        vector->virq = kvm_irqchip_add_msi_route(kvm_state, msg);
+        vector->virq = kvm_irqchip_add_msi_route(kvm_state, vector->msg);
         if (vector->virq < 0 ||
             kvm_irqchip_add_irqfd_notifier(kvm_state, &vector->interrupt,
                                            vector->virq) < 0) {
@@ -1022,6 +1023,30 @@ static void vfio_pci_write_config(PCIDevice *pdev, uint32_t addr,
             vfio_enable_msi(vdev);
         } else if (was_enabled && !is_enabled) {
             vfio_disable_msi(vdev);
+        } else if (was_enabled && is_enabled) {
+            int i;
+
+            for (i = 0; i < vdev->nr_vectors; i++) {
+                VFIOMSIVector *vector = &vdev->msi_vectors[i];
+                MSIMessage msg;
+
+                if (!vector->use || vector->virq < 0) {
+                    continue;
+                }
+
+                msg = msi_get_message(pdev, i);
+
+                if (msg.address != vector->msg.address ||
+                    msg.data != vector->msg.data) {
+
+                    DPRINTF("%s(%04x:%02x:%02x.%x) MSI vector %d changed\n",
+                            __func__, vdev->host.domain, vdev->host.bus,
+                            vdev->host.slot, vdev->host.function, i);
+
+                    kvm_irqchip_update_msi_route(kvm_state, vector->virq, msg);
+                    vector->msg = msg;
+                }
+            }
         }
     }
 

[Qemu-devel] [RFC PATCH 3/4] pci-assign: Refactor MSI virq array

[Alex Williamson]

Convert the msi_virq array into a struct array so we can easily add
a place to track the MSIMessage for each vector.

Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
---
 hw/kvm/pci-assign.c |   51 ++++++++++++++++++++++++++-------------------------
 1 file changed, 26 insertions(+), 25 deletions(-)

diff --git a/hw/kvm/pci-assign.c b/hw/kvm/pci-assign.c
index e80dad0..3e667d1 100644
--- a/hw/kvm/pci-assign.c
+++ b/hw/kvm/pci-assign.c
@@ -130,8 +130,10 @@ typedef struct AssignedDevice {
     } cap;
     uint8_t emulate_config_read[PCI_CONFIG_SPACE_SIZE];
     uint8_t emulate_config_write[PCI_CONFIG_SPACE_SIZE];
-    int msi_virq_nr;
-    int *msi_virq;
+    int msi_nr;
+    struct {
+        int virq;
+    } *msi;
     MSIXTableEntry *msix_table;
     hwaddr msix_table_addr;
     uint16_t msix_max;
@@ -683,19 +685,18 @@ again:
     return 0;
 }
 
-static void free_msi_virqs(AssignedDevice *dev)
+static void free_msi(AssignedDevice *dev)
 {
     int i;
 
-    for (i = 0; i < dev->msi_virq_nr; i++) {
-        if (dev->msi_virq[i] >= 0) {
-            kvm_irqchip_release_virq(kvm_state, dev->msi_virq[i]);
-            dev->msi_virq[i] = -1;
+    for (i = 0; i < dev->msi_nr; i++) {
+        if (dev->msi[i].virq >= 0) {
+            kvm_irqchip_release_virq(kvm_state, dev->msi[i].virq);
         }
     }
-    g_free(dev->msi_virq);
-    dev->msi_virq = NULL;
-    dev->msi_virq_nr = 0;
+    g_free(dev->msi);
+    dev->msi = NULL;
+    dev->msi_nr = 0;
 }
 
 static void free_assigned_device(AssignedDevice *dev)
@@ -750,7 +751,7 @@ static void free_assigned_device(AssignedDevice *dev)
         close(dev->real_device.config_fd);
     }
 
-    free_msi_virqs(dev);
+    free_msi(dev);
 }
 
 static void assign_failed_examine(AssignedDevice *dev)
@@ -989,7 +990,7 @@ static void assigned_dev_update_msi(PCIDevice *pci_dev)
             perror("assigned_dev_update_msi: deassign irq");
         }
 
-        free_msi_virqs(assigned_dev);
+        free_msi(assigned_dev);
 
         assigned_dev->assigned_irq_type = ASSIGNED_IRQ_NONE;
         pci_device_set_intx_routing_notifier(pci_dev, NULL);
@@ -1005,9 +1006,9 @@ static void assigned_dev_update_msi(PCIDevice *pci_dev)
             return;
         }
 
-        assigned_dev->msi_virq = g_malloc(sizeof(*assigned_dev->msi_virq));
-        assigned_dev->msi_virq_nr = 1;
-        assigned_dev->msi_virq[0] = virq;
+        assigned_dev->msi = g_malloc(sizeof(*assigned_dev->msi));
+        assigned_dev->msi_nr = 1;
+        assigned_dev->msi[0].virq = virq;
         if (kvm_device_msi_assign(kvm_state, assigned_dev->dev_id, virq) < 0) {
             perror("assigned_dev_update_msi: kvm_device_msi_assign");
         }
@@ -1055,14 +1056,14 @@ static int assigned_dev_update_msix_mmio(PCIDevice *pci_dev)
         return r;
     }
 
-    free_msi_virqs(adev);
+    free_msi(adev);
 
-    adev->msi_virq_nr = adev->msix_max;
-    adev->msi_virq = g_malloc(adev->msix_max * sizeof(*adev->msi_virq));
+    adev->msi_nr = adev->msix_max;
+    adev->msi = g_malloc(adev->msix_max * sizeof(*adev->msi));
 
     entry = adev->msix_table;
     for (i = 0; i < adev->msix_max; i++, entry++) {
-        adev->msi_virq[i] = -1;
+        adev->msi[i].virq = -1;
 
         if (assigned_dev_msix_masked(entry)) {
             continue;
@@ -1074,13 +1075,13 @@ static int assigned_dev_update_msix_mmio(PCIDevice *pci_dev)
         if (r < 0) {
             return r;
         }
-        adev->msi_virq[i] = r;
+        adev->msi[i].virq = r;
 
         DEBUG("MSI-X vector %d, gsi %d, addr %08x_%08x, data %08x\n", i,
               r, entry->addr_hi, entry->addr_lo, entry->data);
 
         r = kvm_device_msix_set_vector(kvm_state, adev->dev_id, i,
-                                       adev->msi_virq[i]);
+                                       adev->msi[i].virq);
         if (r) {
             error_report("fail to set MSI-X entry! %s", strerror(-r));
             break;
@@ -1108,7 +1109,7 @@ static void assigned_dev_update_msix(PCIDevice *pci_dev)
             perror("assigned_dev_update_msix: deassign irq");
         }
 
-        free_msi_virqs(assigned_dev);
+        free_msi(assigned_dev);
 
         assigned_dev->assigned_irq_type = ASSIGNED_IRQ_NONE;
         pci_device_set_intx_routing_notifier(pci_dev, NULL);
@@ -1120,7 +1121,7 @@ static void assigned_dev_update_msix(PCIDevice *pci_dev)
             return;
         }
 
-        if (assigned_dev->msi_virq_nr > 0) {
+        if (assigned_dev->msi_nr > 0) {
             if (kvm_device_msix_assign(kvm_state, assigned_dev->dev_id) < 0) {
                 perror("assigned_dev_enable_msix: assign irq");
                 return;
@@ -1548,7 +1549,7 @@ static void assigned_dev_msix_mmio_write(void *opaque, hwaddr addr,
         } else if (assigned_dev_msix_masked(&orig) &&
                    !assigned_dev_msix_masked(entry)) {
             /* Vector unmasked */
-            if (i >= adev->msi_virq_nr || adev->msi_virq[i] < 0) {
+            if (i >= adev->msi_nr || adev->msi[i].virq < 0) {
                 /* Previously unassigned vector, start from scratch */
                 assigned_dev_update_msix(pdev);
                 return;
@@ -1562,7 +1563,7 @@ static void assigned_dev_msix_mmio_write(void *opaque, hwaddr addr,
                 msg.data = entry->data;
 
                 ret = kvm_irqchip_update_msi_route(kvm_state,
-                                                   adev->msi_virq[i], msg);
+                                                   adev->msi[i].virq, msg);
                 if (ret) {
                     error_report("Error updating irq routing entry (%d)", ret);
                 }

[Qemu-devel] [RFC PATCH 4/4] pci-assign: Add MSI affinity support

[Alex Williamson]

Track the last MSIMessage programmed so we can determine when it has
changed and update the routing to the guest.  We track MSI-X for
consistency, but we don't do anything with it here.

Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
---
 hw/kvm/pci-assign.c |   29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/hw/kvm/pci-assign.c b/hw/kvm/pci-assign.c
index 3e667d1..500c62c 100644
--- a/hw/kvm/pci-assign.c
+++ b/hw/kvm/pci-assign.c
@@ -133,6 +133,7 @@ typedef struct AssignedDevice {
     int msi_nr;
     struct {
         int virq;
+        MSIMessage msg;
     } *msi;
     MSIXTableEntry *msix_table;
     hwaddr msix_table_addr;
@@ -1009,6 +1010,7 @@ static void assigned_dev_update_msi(PCIDevice *pci_dev)
         assigned_dev->msi = g_malloc(sizeof(*assigned_dev->msi));
         assigned_dev->msi_nr = 1;
         assigned_dev->msi[0].virq = virq;
+        assigned_dev->msi[0].msg = msg;
         if (kvm_device_msi_assign(kvm_state, assigned_dev->dev_id, virq) < 0) {
             perror("assigned_dev_update_msi: kvm_device_msi_assign");
         }
@@ -1021,6 +1023,27 @@ static void assigned_dev_update_msi(PCIDevice *pci_dev)
     }
 }
 
+static void assigned_dev_update_msi_msg(PCIDevice *pci_dev)
+{
+    AssignedDevice *assigned_dev = DO_UPCAST(AssignedDevice, dev, pci_dev);
+    uint8_t ctrl_byte = pci_get_byte(pci_dev->config + pci_dev->msi_cap +
+                                     PCI_MSI_FLAGS);
+    MSIMessage msg;
+
+    if (assigned_dev->assigned_irq_type != ASSIGNED_IRQ_MSI ||
+        !(ctrl_byte & PCI_MSI_FLAGS_ENABLE)) {
+        return;
+    }
+
+    msg = msi_get_message(pci_dev, 0);
+
+    if (msg.address != assigned_dev->msi[0].msg.address ||
+        msg.data != assigned_dev->msi[0].msg.data) {
+        kvm_irqchip_update_msi_route(kvm_state, assigned_dev->msi[0].virq, msg);
+        assigned_dev->msi[0].msg = msg;
+    }
+}
+
 static bool assigned_dev_msix_masked(MSIXTableEntry *entry)
 {
     return (entry->ctrl & cpu_to_le32(0x1)) != 0;
@@ -1076,6 +1099,7 @@ static int assigned_dev_update_msix_mmio(PCIDevice *pci_dev)
             return r;
         }
         adev->msi[i].virq = r;
+        adev->msi[i].msg = msg;
 
         DEBUG("MSI-X vector %d, gsi %d, addr %08x_%08x, data %08x\n", i,
               r, entry->addr_hi, entry->addr_lo, entry->data);
@@ -1183,6 +1207,10 @@ static void assigned_dev_pci_write_config(PCIDevice *pci_dev, uint32_t address,
         if (range_covers_byte(address, len,
                               pci_dev->msi_cap + PCI_MSI_FLAGS)) {
             assigned_dev_update_msi(pci_dev);
+        } else if (ranges_overlap(address, len,
+                                  pci_dev->msi_cap + PCI_MSI_ADDRESS_LO,
+                                  10 - PCI_MSI_ADDRESS_LO)) {
+            assigned_dev_update_msi_msg(pci_dev);
         }
     }
     if (assigned_dev->cap.available & ASSIGNED_DEVICE_CAP_MSIX) {
@@ -1561,6 +1589,7 @@ static void assigned_dev_msix_mmio_write(void *opaque, hwaddr addr,
                 msg.address = entry->addr_lo |
                     ((uint64_t)entry->addr_hi << 32);
                 msg.data = entry->data;
+                adev->msi[i].msg = msg;
 
                 ret = kvm_irqchip_update_msi_route(kvm_state,
                                                    adev->msi[i].virq, msg);

Re: [Qemu-devel] [RFC PATCH 0/4] MSI affinity for assigned devices

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 941 bytes --]

On 2012-11-27 23:00, Alex Williamson wrote:
> This is post-1.3 material, so I'll just post it as an RFC for now.
> 
> MSI routing updates aren't currently handled by pci-assign or
> vfio-pci (when using KVM acceleration), which means that trying to
> set interrupt SMP affinity in the guest has no effect unless MSI is
> completely disabled and re-enabled.  This series fixes this for both
> device assignment backends using similar schemes.  We store the last
> MSIMessage programmed to KVM and do updates to the MSI route when it
> changes.  pci-assign takes a little bit of refactoring to make this
> happen cleanly.  Thanks,

This should rather be done by implementing vector notifiers for MSI as
well. That way the device model no longer has to track reasons for
vector changes in an open-coded fashion, just like we already do for MSI-X.

Was on my todo list for a long time, but I never reached this item.

Jan



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 261 bytes --]

Re: [Qemu-devel] [RFC PATCH 0/4] MSI affinity for assigned devices

[Alex Williamson]

On Wed, 2012-11-28 at 00:08 +0100, Jan Kiszka wrote:
> On 2012-11-27 23:00, Alex Williamson wrote:
> > This is post-1.3 material, so I'll just post it as an RFC for now.
> > 
> > MSI routing updates aren't currently handled by pci-assign or
> > vfio-pci (when using KVM acceleration), which means that trying to
> > set interrupt SMP affinity in the guest has no effect unless MSI is
> > completely disabled and re-enabled.  This series fixes this for both
> > device assignment backends using similar schemes.  We store the last
> > MSIMessage programmed to KVM and do updates to the MSI route when it
> > changes.  pci-assign takes a little bit of refactoring to make this
> > happen cleanly.  Thanks,
> 
> This should rather be done by implementing vector notifiers for MSI as
> well. That way the device model no longer has to track reasons for
> vector changes in an open-coded fashion, just like we already do for MSI-X.
> 
> Was on my todo list for a long time, but I never reached this item.

MSI masking is optional and not many devices seem to support it.  What I
see with a linux guest is that it just overwrites the address/data while
MSI is enabled.  What were you thinking for notifiers? mask, unmask,
update?  I'm not sure I'm interested enough in this to add MSI vector
notifiers.  Thanks,

Alex

Re: [Qemu-devel] [RFC PATCH 0/4] MSI affinity for assigned devices

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1826 bytes --]

On 2012-11-28 00:21, Alex Williamson wrote:
> On Wed, 2012-11-28 at 00:08 +0100, Jan Kiszka wrote:
>> On 2012-11-27 23:00, Alex Williamson wrote:
>>> This is post-1.3 material, so I'll just post it as an RFC for now.
>>>
>>> MSI routing updates aren't currently handled by pci-assign or
>>> vfio-pci (when using KVM acceleration), which means that trying to
>>> set interrupt SMP affinity in the guest has no effect unless MSI is
>>> completely disabled and re-enabled.  This series fixes this for both
>>> device assignment backends using similar schemes.  We store the last
>>> MSIMessage programmed to KVM and do updates to the MSI route when it
>>> changes.  pci-assign takes a little bit of refactoring to make this
>>> happen cleanly.  Thanks,
>>
>> This should rather be done by implementing vector notifiers for MSI as
>> well. That way the device model no longer has to track reasons for
>> vector changes in an open-coded fashion, just like we already do for MSI-X.
>>
>> Was on my todo list for a long time, but I never reached this item.
> 
> MSI masking is optional and not many devices seem to support it.  What I
> see with a linux guest is that it just overwrites the address/data while
> MSI is enabled.  What were you thinking for notifiers? mask, unmask,
> update?  I'm not sure I'm interested enough in this to add MSI vector
> notifiers.  Thanks,

We didn't merge "mask" notifiers but a more generic concept. Our
notifiers a supposed to fire when the configuration changes effectively.
So, for MSI, a "use" event has to trigger also on "change", i.e. when
the configuration is overwritten while the vector is already active.
Alternatively, you could also add such an "update" notifier, using the
MSIVectorUseNotifier signature if it makes the user side simpler.

Jan



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 261 bytes --]

[Qemu-devel] PCI device pass through support

[Krishna J]

Hi Alex,
I am trying to pass through a PCI device to the guest to compare the MSI interrupt latency with normal device pass through and pass through using VFIO framework. I used the following script

for dev in $(ls /sys/bus/pci/devices/0000:06:00.0/iommu_group/devices); do
    vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
    device=$(cat /sys/bus/pci/devices/$dev/device)
    if [ -e /sys/bus/pci/devices/$dev/driver ]; then
        echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
    fi
    echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
done

and added "-device vfio-pci,host,host=0000:06:00.0" to my qemu command line. 

I am getting Error - qemu-system-x86_64: -device vfio-pci,host,host=0000:06:00.0: Property 'vfio-pci.host' doesn't take value 'on'-- when i use the qemu for your git repo and and also the latest v1.3.0-rc1 qemu. I am using Kernel 3.6.7-rt18. Am i missing step ?


Krishna 		 	   		  

Re: [Qemu-devel] PCI device pass through support

[Alex Williamson]

On Wed, 2012-11-28 at 16:32 +0000, Krishna J wrote:
> Hi Alex,
> I am trying to pass through a PCI device to the guest to compare the
> MSI interrupt latency with normal device pass through and pass through
> using VFIO framework. I used the following script
> 
> for dev in $(ls /sys/bus/pci/devices/0000:06:00.0/iommu_group/devices); do
>     vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
>     device=$(cat /sys/bus/pci/devices/$dev/device)
>     if [ -e /sys/bus/pci/devices/$dev/driver ]; then
>         echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
>     fi
>     echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
> done
> 
> and added "-device vfio-pci,host,host=0000:06:00.0" to my qemu command line. 
                             ^^^^^

> I am getting Error - qemu-system-x86_64: -device
> vfio-pci,host,host=0000:06:00.0: Property 'vfio-pci.host' doesn't take
> value 'on'-- when i use the qemu for your git repo and and also the
> latest v1.3.0-rc1 qemu. I am using Kernel 3.6.7-rt18. Am i missing
> step ?

Hi Krishna,

There's an extra "host," in the option above, is that the problem?
Please let me know if I've mis-documented it anywhere.  I hope you can
share your results when you get it working.  Thanks,

Alex

Re: [Qemu-devel] [RFC PATCH 0/4] MSI affinity for assigned devices

[Krishna J]

[-- Attachment #1: Type: text/plain, Size: 2733 bytes --]

Hi Alex,
> MSI routing updates aren't currently handled by pci-assign or
> vfio-pci (when using KVM acceleration), which means that trying to
> set interrupt SMP affinity in the guest has no effect unless MSI is
> completely disabled and re-enabled.  This series fixes this for both
> device assignment backends using similar schemes.  We store the last
> MSIMessage programmed to KVM and do updates to the MSI route when it
> changes.  pci-assign takes a little bit of refactoring to make this
> happen cleanly.  Thanks,

I am using the MSI affinity for assigned devices patch 1 to 4. I have setup the guest such that VCPU0 is pinned to PCPU1, VCPU1 is pinned to PCPU2, VCPU2 is pinned to PCPU3 and VCPU3 is pinned to PCPU4. I do this by taskset after the guest boots. I then start generating interrupts affined to VCPU3. I see all the interrupts directly delivered to VCPU 3. Now i do the same test but interrupt affined to VCPU 2. Although the interrupts are delivered to VCPU2  there are lot of "Rescheduling interrupts" in VCPU 3. I have checked the smp_affinity and it is updated to VCPU 2. 
Wanted to know your feedback on this usecase and what might be the impact. 
           CPU0       CPU1	 CPU2       CPU3  0:        211          0          0          0   IO-APIC-edge      timer  4:	  60940          0          0          0   IO-APIC-edge      serial  8:         65          0          0          0   IO-APIC-edge      rtc0  9:          0          0          0          0   IO-APIC-fasteoi   acpi 40:          0          0          0          0   PCI-MSI-edge      virtio1-config 41:	   1910          0          0          0   PCI-MSI-edge      virtio1-requests 42:          0          0          0          0   PCI-MSI-edge      virtio0-config 43:        127          0          0          0   PCI-MSI-edge      virtio0-input 44:          1          0          0          0   PCI-MSI-edge      virtio0-output 45:          1          0       3377      11194   PCI-MSI-edge      FPGA_DEVNMI:          0          0          0          0   Non-maskable interruptsLOC:     225880     231572     223670     223612   Local timer interruptsSPU:          0          0          0          0   Spurious interruptsPMI:          0          0          0          0   Performance monitoring interruptsIWI:          0          0          0          0   IRQ work interruptsRTR:          0          0          0          0   APIC ICR read retriesRES:         14         20         21       3398   Rescheduling interrupts-------> Many RES Interrtups!!!!!!CAL:          0         14         14         16   Function call interruptsTLB:          0          0          0          0   TLB shootdowns

thanks,Krishna  		 	   		  

[-- Attachment #2: Type: text/html, Size: 5849 bytes --]

Re: [Qemu-devel] [RFC PATCH 0/4] MSI affinity for assigned devices

[Alex Williamson]

On Mon, 2013-01-07 at 20:14 +0000, Krishna J wrote:
> Hi Alex,
> > MSI routing updates aren't currently handled by pci-assign or
> > vfio-pci (when using KVM acceleration), which means that trying to
> > set interrupt SMP affinity in the guest has no effect unless MSI is
> > completely disabled and re-enabled.  This series fixes this for both
> > device assignment backends using similar schemes.  We store the last
> > MSIMessage programmed to KVM and do updates to the MSI route when it
> > changes.  pci-assign takes a little bit of refactoring to make this
> > happen cleanly.  Thanks,
> 
> I am using the MSI affinity for assigned devices patch 1 to 4. I have
> setup the guest such that VCPU0 is pinned to PCPU1, VCPU1 is pinned to
> PCPU2, VCPU2 is pinned to PCPU3 and VCPU3 is pinned to PCPU4. I do
> this by taskset after the guest boots. I then start generating
> interrupts affined to VCPU3. I see all the interrupts directly
> delivered to VCPU 3. Now i do the same test but interrupt affined to
> VCPU 2. Although the interrupts are delivered to VCPU2  there are lot
> of "Rescheduling interrupts" in VCPU 3. I have checked the
> smp_affinity and it is updated to VCPU 2. 
> Wanted to know your feedback on this usecase and what might be the
> impact. 
>            CPU0       CPU1	 CPU2       CPU3
>   0:        211          0          0          0   IO-APIC-edge      timer
>   4:	   60940          0          0          0   IO-APIC-edge      serial
>   8:         65          0          0          0   IO-APIC-edge      rtc0
>   9:          0          0          0          0   IO-APIC-fasteoi   acpi
>  40:          0          0          0          0   PCI-MSI-edge      virtio1-config
>  41:	    1910          0          0          0   PCI-MSI-edge      virtio1-requests
>  42:          0          0          0          0   PCI-MSI-edge      virtio0-config
>  43:        127          0          0          0   PCI-MSI-edge      virtio0-input
>  44:          1          0          0          0   PCI-MSI-edge      virtio0-output
>  45:          1          0       3377      11194   PCI-MSI-edge      FPGA_DEV
> NMI:          0          0          0          0   Non-maskable interrupts
> LOC:     225880     231572     223670     223612   Local timer interrupts
> SPU:          0          0          0          0   Spurious interrupts
> PMI:          0          0          0          0   Performance monitoring interrupts
> IWI:          0          0          0          0   IRQ work interrupts
> RTR:          0          0          0          0   APIC ICR read retries
> RES:         14         20         21       3398   Rescheduling interrupts-------> Many RES Interrtups!!!!!!
> CAL:          0         14         14         16   Function call interrupts
> TLB:          0          0          0          0   TLB shootdowns

I don't know, but I'll fix the line wrap for anyone else that wants to
have a look.  The count looks roughly similar to the number of
interrupts to VCPU2.  Is your application somehow tied to VCPU3?
Thanks,

Alex

Re: [Qemu-devel] [RFC PATCH 0/4] MSI affinity for assigned devices

[Krishna J]

[-- Attachment #1: Type: text/plain, Size: 488 bytes --]

> >  45:          1          0       3377      11194   PCI-MSI-edge      FPGA_DEV> > RES:         14         20         21       3398   Rescheduling interrupts-------> Many RES Interrtups!!!!!!
> 
> I don't know, but I'll fix the line wrap for anyone else that wants to
> have a look.  The count looks roughly similar to the number of
> interrupts to VCPU2.  Is your application somehow tied to VCPU3?

Yes, my mistake sorry. The application was still tied to VCPU 3. 
 		 	   		  

[-- Attachment #2: Type: text/html, Size: 936 bytes --]