[Qemu-devel] [PATCH 0/5] Support more virtio queues

[Qemu-devel] [PATCH 0/5] Support more virtio queues

[Jason Wang]

We current limit the max virtio queues to 64. This is not sufficient
to support some multiqueue deivces (e.g recent Linux support up to 256
tap queues). So this series try to let virtio to support more queues.

No much works needs to be done except:

- Patch 1 renames VIRTIO_PCI_QUEUE_MAX to a more generic name:
  VIRTIO_QUEUE_MAX
- Patch 2 simply increase the the limitation to 513 (256 queue pairs)
- Patch 3 fixes a bug in virtio_del_queue()
- Patch 4 tries to remove the hard coded MSI-X bar size (4096) and allow up to
  2048 MSI-X entries.
- Patch 5 add a boolean property to let the virtio-net can calculate
  the MSI-X bar size based on the number of MSI-X vectors and keep
  migration compability with legacy version whose bar size is 4096.

With this patch, we can support up to 256 queues.Since x86 can only
allow about 240 interrupt vectors for MSI-X, current Linux driver can
only have about 80 queue pairs has their private MSI-X interrupt
vectors. With sharing IRQ with queue pairs (RFC posted in
https://lkml.org/lkml/2014/12/25/169), Linux driver can have up
to about 186 queue pairs has their private MSI-X interrupt vectors.

Thanks

Jason Wang (5):
  virtio: rename VIRTIO_PCI_QUEUE_MAX to VIRTIO_QUEUE_MAX
  virtio: increase VIRITO_QUEUE_MAX to 513
  virtio-net: fix the upper bound when trying to delete queues
  pci: remove hard-coded bar size in msix_init_exclusive_bar()
  virtio-pci: introduce auto_msix_bar_size property

 hw/block/nvme.c              |  2 +-
 hw/char/virtio-serial-bus.c  |  2 +-
 hw/i386/pc_piix.c            |  4 ++++
 hw/i386/pc_q35.c             |  4 ++++
 hw/misc/ivshmem.c            |  2 +-
 hw/net/virtio-net.c          |  2 +-
 hw/pci/msix.c                | 18 +++++++-----------
 hw/ppc/spapr.c               |  5 +++++
 hw/s390x/s390-virtio-bus.c   |  4 ++--
 hw/s390x/s390-virtio-ccw.c   |  2 +-
 hw/s390x/virtio-ccw.c        | 10 +++++-----
 hw/scsi/virtio-scsi.c        |  4 ++--
 hw/virtio/virtio-mmio.c      |  4 ++--
 hw/virtio/virtio-pci.c       | 27 ++++++++++++++++++++-------
 hw/virtio/virtio-pci.h       |  3 +++
 hw/virtio/virtio.c           | 26 +++++++++++++-------------
 include/hw/compat.h          |  8 ++++++++
 include/hw/pci/msix.h        |  2 +-
 include/hw/s390x/s390_flic.h |  2 +-
 include/hw/virtio/virtio.h   |  2 +-
 20 files changed, 83 insertions(+), 50 deletions(-)

-- 
1.9.1

[Qemu-devel] [PATCH 1/5] virtio: rename VIRTIO_PCI_QUEUE_MAX to VIRTIO_QUEUE_MAX

[Jason Wang]

VIRTIO_PCI_QUEUE_MAX was not specific to pci, so rename it to VIRTIO_QUEUE_MAX.

Cc: Amit Shah <amit.shah@redhat.com>
Cc: Anthony Liguori <aliguori@amazon.com>
Cc: Michael S. Tsirkin <mst@redhat.com>
Cc: Alexander Graf <agraf@suse.de>
Cc: Richard Henderson <rth@twiddle.net>
Cc: Cornelia Huck <cornelia.huck@de.ibm.com>
Cc: Christian Borntraeger <borntraeger@de.ibm.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 hw/char/virtio-serial-bus.c  |  2 +-
 hw/s390x/s390-virtio-bus.c   |  4 ++--
 hw/s390x/s390-virtio-ccw.c   |  2 +-
 hw/s390x/virtio-ccw.c        | 10 +++++-----
 hw/scsi/virtio-scsi.c        |  4 ++--
 hw/virtio/virtio-mmio.c      |  4 ++--
 hw/virtio/virtio-pci.c       | 10 +++++-----
 hw/virtio/virtio.c           | 26 +++++++++++++-------------
 include/hw/s390x/s390_flic.h |  2 +-
 include/hw/virtio/virtio.h   |  2 +-
 10 files changed, 33 insertions(+), 33 deletions(-)

diff --git a/hw/char/virtio-serial-bus.c b/hw/char/virtio-serial-bus.c
index 37a6f44..186d094 100644
--- a/hw/char/virtio-serial-bus.c
+++ b/hw/char/virtio-serial-bus.c
@@ -942,7 +942,7 @@ static void virtio_serial_device_realize(DeviceState *dev, Error **errp)
     }
 
     /* Each port takes 2 queues, and one pair is for the control queue */
-    max_supported_ports = VIRTIO_PCI_QUEUE_MAX / 2 - 1;
+    max_supported_ports = VIRTIO_QUEUE_MAX / 2 - 1;
 
     if (vser->serial.max_virtserial_ports > max_supported_ports) {
         error_setg(errp, "maximum ports supported: %u", max_supported_ports);
diff --git a/hw/s390x/s390-virtio-bus.c b/hw/s390x/s390-virtio-bus.c
index 39dc201..b7f64f0 100644
--- a/hw/s390x/s390-virtio-bus.c
+++ b/hw/s390x/s390-virtio-bus.c
@@ -331,7 +331,7 @@ static ram_addr_t s390_virtio_device_num_vq(VirtIOS390Device *dev)
     VirtIODevice *vdev = dev->vdev;
     int num_vq;
 
-    for (num_vq = 0; num_vq < VIRTIO_PCI_QUEUE_MAX; num_vq++) {
+    for (num_vq = 0; num_vq < VIRTIO_QUEUE_MAX; num_vq++) {
         if (!virtio_queue_get_num(vdev, num_vq)) {
             break;
         }
@@ -438,7 +438,7 @@ VirtIOS390Device *s390_virtio_bus_find_vring(VirtIOS390Bus *bus,
     QTAILQ_FOREACH(kid, &bus->bus.children, sibling) {
         VirtIOS390Device *dev = (VirtIOS390Device *)kid->child;
 
-        for(i = 0; i < VIRTIO_PCI_QUEUE_MAX; i++) {
+        for(i = 0; i < VIRTIO_QUEUE_MAX; i++) {
             if (!virtio_queue_get_addr(dev->vdev, i))
                 break;
             if (virtio_queue_get_addr(dev->vdev, i) == mem) {
diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c
index 71bafe0..ca3535a 100644
--- a/hw/s390x/s390-virtio-ccw.c
+++ b/hw/s390x/s390-virtio-ccw.c
@@ -55,7 +55,7 @@ static int virtio_ccw_hcall_notify(const uint64_t *args)
     if (!sch || !css_subch_visible(sch)) {
         return -EINVAL;
     }
-    if (queue >= VIRTIO_PCI_QUEUE_MAX) {
+    if (queue >= VIRTIO_QUEUE_MAX) {
         return -EINVAL;
     }
     virtio_queue_notify(virtio_ccw_get_vdev(sch), queue);
diff --git a/hw/s390x/virtio-ccw.c b/hw/s390x/virtio-ccw.c
index ea236c9..837df16 100644
--- a/hw/s390x/virtio-ccw.c
+++ b/hw/s390x/virtio-ccw.c
@@ -170,7 +170,7 @@ static void virtio_ccw_start_ioeventfd(VirtioCcwDevice *dev)
         return;
     }
     vdev = virtio_bus_get_device(&dev->bus);
-    for (n = 0; n < VIRTIO_PCI_QUEUE_MAX; n++) {
+    for (n = 0; n < VIRTIO_QUEUE_MAX; n++) {
         if (!virtio_queue_get_num(vdev, n)) {
             continue;
         }
@@ -205,7 +205,7 @@ static void virtio_ccw_stop_ioeventfd(VirtioCcwDevice *dev)
         return;
     }
     vdev = virtio_bus_get_device(&dev->bus);
-    for (n = 0; n < VIRTIO_PCI_QUEUE_MAX; n++) {
+    for (n = 0; n < VIRTIO_QUEUE_MAX; n++) {
         if (!virtio_queue_get_num(vdev, n)) {
             continue;
         }
@@ -266,7 +266,7 @@ static int virtio_ccw_set_vqs(SubchDev *sch, uint64_t addr, uint32_t align,
 {
     VirtIODevice *vdev = virtio_ccw_get_vdev(sch);
 
-    if (index > VIRTIO_PCI_QUEUE_MAX) {
+    if (index > VIRTIO_QUEUE_MAX) {
         return -EINVAL;
     }
 
@@ -291,7 +291,7 @@ static int virtio_ccw_set_vqs(SubchDev *sch, uint64_t addr, uint32_t align,
         virtio_queue_set_vector(vdev, index, index);
     }
     /* tell notify handler in case of config change */
-    vdev->config_vector = VIRTIO_PCI_QUEUE_MAX;
+    vdev->config_vector = VIRTIO_QUEUE_MAX;
     return 0;
 }
 
@@ -1026,7 +1026,7 @@ static void virtio_ccw_notify(DeviceState *d, uint16_t vector)
         return;
     }
 
-    if (vector < VIRTIO_PCI_QUEUE_MAX) {
+    if (vector < VIRTIO_QUEUE_MAX) {
         if (!dev->indicators) {
             return;
         }
diff --git a/hw/scsi/virtio-scsi.c b/hw/scsi/virtio-scsi.c
index 9e2c718..51fdecb 100644
--- a/hw/scsi/virtio-scsi.c
+++ b/hw/scsi/virtio-scsi.c
@@ -822,10 +822,10 @@ void virtio_scsi_common_realize(DeviceState *dev, Error **errp,
                 sizeof(VirtIOSCSIConfig));
 
     if (s->conf.num_queues == 0 ||
-            s->conf.num_queues > VIRTIO_PCI_QUEUE_MAX - 2) {
+            s->conf.num_queues > VIRTIO_QUEUE_MAX - 2) {
         error_setg(errp, "Invalid number of queues (= %" PRIu32 "), "
                          "must be a positive integer less than %d.",
-                   s->conf.num_queues, VIRTIO_PCI_QUEUE_MAX - 2);
+                   s->conf.num_queues, VIRTIO_QUEUE_MAX - 2);
         virtio_cleanup(vdev);
         return;
     }
diff --git a/hw/virtio/virtio-mmio.c b/hw/virtio/virtio-mmio.c
index 2450c13..9ccd0ef 100644
--- a/hw/virtio/virtio-mmio.c
+++ b/hw/virtio/virtio-mmio.c
@@ -237,7 +237,7 @@ static void virtio_mmio_write(void *opaque, hwaddr offset, uint64_t value,
                 proxy->guest_page_shift);
         break;
     case VIRTIO_MMIO_QUEUESEL:
-        if (value < VIRTIO_PCI_QUEUE_MAX) {
+        if (value < VIRTIO_QUEUE_MAX) {
             vdev->queue_sel = value;
         }
         break;
@@ -257,7 +257,7 @@ static void virtio_mmio_write(void *opaque, hwaddr offset, uint64_t value,
         }
         break;
     case VIRTIO_MMIO_QUEUENOTIFY:
-        if (value < VIRTIO_PCI_QUEUE_MAX) {
+        if (value < VIRTIO_QUEUE_MAX) {
             virtio_queue_notify(vdev, value);
         }
         break;
diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index dde1d73..abaefaa 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -215,7 +215,7 @@ static void virtio_pci_start_ioeventfd(VirtIOPCIProxy *proxy)
         return;
     }
 
-    for (n = 0; n < VIRTIO_PCI_QUEUE_MAX; n++) {
+    for (n = 0; n < VIRTIO_QUEUE_MAX; n++) {
         if (!virtio_queue_get_num(vdev, n)) {
             continue;
         }
@@ -251,7 +251,7 @@ static void virtio_pci_stop_ioeventfd(VirtIOPCIProxy *proxy)
         return;
     }
 
-    for (n = 0; n < VIRTIO_PCI_QUEUE_MAX; n++) {
+    for (n = 0; n < VIRTIO_QUEUE_MAX; n++) {
         if (!virtio_queue_get_num(vdev, n)) {
             continue;
         }
@@ -287,11 +287,11 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val)
             virtio_queue_set_addr(vdev, vdev->queue_sel, pa);
         break;
     case VIRTIO_PCI_QUEUE_SEL:
-        if (val < VIRTIO_PCI_QUEUE_MAX)
+        if (val < VIRTIO_QUEUE_MAX)
             vdev->queue_sel = val;
         break;
     case VIRTIO_PCI_QUEUE_NOTIFY:
-        if (val < VIRTIO_PCI_QUEUE_MAX) {
+        if (val < VIRTIO_QUEUE_MAX) {
             virtio_queue_notify(vdev, val);
         }
         break;
@@ -792,7 +792,7 @@ static int virtio_pci_set_guest_notifiers(DeviceState *d, int nvqs, bool assign)
     bool with_irqfd = msix_enabled(&proxy->pci_dev) &&
         kvm_msi_via_irqfd_enabled();
 
-    nvqs = MIN(nvqs, VIRTIO_PCI_QUEUE_MAX);
+    nvqs = MIN(nvqs, VIRTIO_QUEUE_MAX);
 
     /* When deassigning, pass a consistent nvqs value
      * to avoid leaking notifiers.
diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
index 013979a..1136e78 100644
--- a/hw/virtio/virtio.c
+++ b/hw/virtio/virtio.c
@@ -599,7 +599,7 @@ void virtio_reset(void *opaque)
     vdev->config_vector = VIRTIO_NO_VECTOR;
     virtio_notify_vector(vdev, vdev->config_vector);
 
-    for(i = 0; i < VIRTIO_PCI_QUEUE_MAX; i++) {
+    for(i = 0; i < VIRTIO_QUEUE_MAX; i++) {
         vdev->vq[i].vring.desc = 0;
         vdev->vq[i].vring.avail = 0;
         vdev->vq[i].vring.used = 0;
@@ -738,7 +738,7 @@ int virtio_queue_get_num(VirtIODevice *vdev, int n)
 int virtio_queue_get_id(VirtQueue *vq)
 {
     VirtIODevice *vdev = vq->vdev;
-    assert(vq >= &vdev->vq[0] && vq < &vdev->vq[VIRTIO_PCI_QUEUE_MAX]);
+    assert(vq >= &vdev->vq[0] && vq < &vdev->vq[VIRTIO_QUEUE_MAX]);
     return vq - &vdev->vq[0];
 }
 
@@ -773,13 +773,13 @@ void virtio_queue_notify(VirtIODevice *vdev, int n)
 
 uint16_t virtio_queue_vector(VirtIODevice *vdev, int n)
 {
-    return n < VIRTIO_PCI_QUEUE_MAX ? vdev->vq[n].vector :
+    return n < VIRTIO_QUEUE_MAX ? vdev->vq[n].vector :
         VIRTIO_NO_VECTOR;
 }
 
 void virtio_queue_set_vector(VirtIODevice *vdev, int n, uint16_t vector)
 {
-    if (n < VIRTIO_PCI_QUEUE_MAX)
+    if (n < VIRTIO_QUEUE_MAX)
         vdev->vq[n].vector = vector;
 }
 
@@ -788,12 +788,12 @@ VirtQueue *virtio_add_queue(VirtIODevice *vdev, int queue_size,
 {
     int i;
 
-    for (i = 0; i < VIRTIO_PCI_QUEUE_MAX; i++) {
+    for (i = 0; i < VIRTIO_QUEUE_MAX; i++) {
         if (vdev->vq[i].vring.num == 0)
             break;
     }
 
-    if (i == VIRTIO_PCI_QUEUE_MAX || queue_size > VIRTQUEUE_MAX_SIZE)
+    if (i == VIRTIO_QUEUE_MAX || queue_size > VIRTQUEUE_MAX_SIZE)
         abort();
 
     vdev->vq[i].vring.num = queue_size;
@@ -805,7 +805,7 @@ VirtQueue *virtio_add_queue(VirtIODevice *vdev, int queue_size,
 
 void virtio_del_queue(VirtIODevice *vdev, int n)
 {
-    if (n < 0 || n >= VIRTIO_PCI_QUEUE_MAX) {
+    if (n < 0 || n >= VIRTIO_QUEUE_MAX) {
         abort();
     }
 
@@ -928,14 +928,14 @@ void virtio_save(VirtIODevice *vdev, QEMUFile *f)
     qemu_put_be32(f, vdev->config_len);
     qemu_put_buffer(f, vdev->config, vdev->config_len);
 
-    for (i = 0; i < VIRTIO_PCI_QUEUE_MAX; i++) {
+    for (i = 0; i < VIRTIO_QUEUE_MAX; i++) {
         if (vdev->vq[i].vring.num == 0)
             break;
     }
 
     qemu_put_be32(f, i);
 
-    for (i = 0; i < VIRTIO_PCI_QUEUE_MAX; i++) {
+    for (i = 0; i < VIRTIO_QUEUE_MAX; i++) {
         if (vdev->vq[i].vring.num == 0)
             break;
 
@@ -1000,7 +1000,7 @@ int virtio_load(VirtIODevice *vdev, QEMUFile *f, int version_id)
     qemu_get_8s(f, &vdev->status);
     qemu_get_8s(f, &vdev->isr);
     qemu_get_be16s(f, &vdev->queue_sel);
-    if (vdev->queue_sel >= VIRTIO_PCI_QUEUE_MAX) {
+    if (vdev->queue_sel >= VIRTIO_QUEUE_MAX) {
         return -1;
     }
     qemu_get_be32s(f, &features);
@@ -1027,7 +1027,7 @@ int virtio_load(VirtIODevice *vdev, QEMUFile *f, int version_id)
 
     num = qemu_get_be32(f);
 
-    if (num > VIRTIO_PCI_QUEUE_MAX) {
+    if (num > VIRTIO_QUEUE_MAX) {
         error_report("Invalid number of PCI queues: 0x%x", num);
         return -1;
     }
@@ -1143,9 +1143,9 @@ void virtio_init(VirtIODevice *vdev, const char *name,
     vdev->isr = 0;
     vdev->queue_sel = 0;
     vdev->config_vector = VIRTIO_NO_VECTOR;
-    vdev->vq = g_malloc0(sizeof(VirtQueue) * VIRTIO_PCI_QUEUE_MAX);
+    vdev->vq = g_malloc0(sizeof(VirtQueue) * VIRTIO_QUEUE_MAX);
     vdev->vm_running = runstate_is_running();
-    for (i = 0; i < VIRTIO_PCI_QUEUE_MAX; i++) {
+    for (i = 0; i < VIRTIO_QUEUE_MAX; i++) {
         vdev->vq[i].vector = VIRTIO_NO_VECTOR;
         vdev->vq[i].vdev = vdev;
         vdev->vq[i].queue_index = i;
diff --git a/include/hw/s390x/s390_flic.h b/include/hw/s390x/s390_flic.h
index 489d73b..17b8457 100644
--- a/include/hw/s390x/s390_flic.h
+++ b/include/hw/s390x/s390_flic.h
@@ -20,7 +20,7 @@
 typedef struct AdapterRoutes {
     AdapterInfo adapter;
     int num_routes;
-    int gsi[VIRTIO_PCI_QUEUE_MAX];
+    int gsi[VIRTIO_QUEUE_MAX];
 } AdapterRoutes;
 
 #define TYPE_S390_FLIC_COMMON "s390-flic"
diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
index f24997d..220c09d 100644
--- a/include/hw/virtio/virtio.h
+++ b/include/hw/virtio/virtio.h
@@ -92,7 +92,7 @@ typedef struct VirtQueueElement
     struct iovec out_sg[VIRTQUEUE_MAX_SIZE];
 } VirtQueueElement;
 
-#define VIRTIO_PCI_QUEUE_MAX 64
+#define VIRTIO_QUEUE_MAX 64
 
 #define VIRTIO_NO_VECTOR 0xffff
 
-- 
1.9.1

[Qemu-devel] [PATCH 2/5] virtio: increase VIRITO_QUEUE_MAX to 513

[Jason Wang]

Recent linux kernel supports up to 256 tap queues. Increase the limit
to 513 (256 * 2 + 1(ctrl vq)).

Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 include/hw/virtio/virtio.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
index 220c09d..21bb30f 100644
--- a/include/hw/virtio/virtio.h
+++ b/include/hw/virtio/virtio.h
@@ -92,7 +92,7 @@ typedef struct VirtQueueElement
     struct iovec out_sg[VIRTQUEUE_MAX_SIZE];
 } VirtQueueElement;
 
-#define VIRTIO_QUEUE_MAX 64
+#define VIRTIO_QUEUE_MAX 513
 
 #define VIRTIO_NO_VECTOR 0xffff
 
-- 
1.9.1

[Qemu-devel] [PATCH 3/5] virtio-net: fix the upper bound when trying to delete queues

[Jason Wang]

Virtqueue were indexed from zero, so don't delete virtqueue whose
index is n->max_queues * 2 + 1.

Cc: Anthony Liguori <aliguori@amazon.com>
Cc: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 hw/net/virtio-net.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index 45da34a..184b2f2 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -1306,7 +1306,7 @@ static void virtio_net_set_multiqueue(VirtIONet *n, int multiqueue)
 
     n->multiqueue = multiqueue;
 
-    for (i = 2; i <= n->max_queues * 2 + 1; i++) {
+    for (i = 2; i < n->max_queues * 2 + 1; i++) {
         virtio_del_queue(vdev, i);
     }
 
-- 
1.9.1

[Qemu-devel] [PATCH 4/5] pci: remove hard-coded bar size in msix_init_exclusive_bar()

[Jason Wang]

Let msix_init_exclusive_bar() can accept bar_size parameter other than
a hard-coded limit 4096. Then caller can specify a bar_size depends on
msix entries and can use up to 2048 msix entries as PCI spec
allows. To keep migration compatibility, 4096 is used for all callers
and pba were start from half of bar size.

Cc: Keith Busch <keith.busch@intel.com>
Cc: Kevin Wolf <kwolf@redhat.com>
Cc: Stefan Hajnoczi <stefanha@redhat.com>
Cc: Michael S. Tsirkin <mst@redhat.com>
Cc: Anthony Liguori <aliguori@amazon.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 hw/block/nvme.c        |  2 +-
 hw/misc/ivshmem.c      |  2 +-
 hw/pci/msix.c          | 18 +++++++-----------
 hw/virtio/virtio-pci.c |  2 +-
 include/hw/pci/msix.h  |  2 +-
 5 files changed, 11 insertions(+), 15 deletions(-)

diff --git a/hw/block/nvme.c b/hw/block/nvme.c
index ce079ae..0fa1396 100644
--- a/hw/block/nvme.c
+++ b/hw/block/nvme.c
@@ -786,7 +786,7 @@ static int nvme_init(PCIDevice *pci_dev)
     pci_register_bar(&n->parent_obj, 0,
         PCI_BASE_ADDRESS_SPACE_MEMORY | PCI_BASE_ADDRESS_MEM_TYPE_64,
         &n->iomem);
-    msix_init_exclusive_bar(&n->parent_obj, n->num_queues, 4);
+    msix_init_exclusive_bar(&n->parent_obj, n->num_queues, 4, 4096);
 
     id->vid = cpu_to_le16(pci_get_word(pci_conf + PCI_VENDOR_ID));
     id->ssvid = cpu_to_le16(pci_get_word(pci_conf + PCI_SUBSYSTEM_VENDOR_ID));
diff --git a/hw/misc/ivshmem.c b/hw/misc/ivshmem.c
index 5d272c8..3e2a2d4 100644
--- a/hw/misc/ivshmem.c
+++ b/hw/misc/ivshmem.c
@@ -631,7 +631,7 @@ static uint64_t ivshmem_get_size(IVShmemState * s) {
 
 static void ivshmem_setup_msi(IVShmemState * s)
 {
-    if (msix_init_exclusive_bar(PCI_DEVICE(s), s->vectors, 1)) {
+    if (msix_init_exclusive_bar(PCI_DEVICE(s), s->vectors, 1, 4096)) {
         IVSHMEM_DPRINTF("msix initialization failed\n");
         exit(1);
     }
diff --git a/hw/pci/msix.c b/hw/pci/msix.c
index 24de260..9a1894f 100644
--- a/hw/pci/msix.c
+++ b/hw/pci/msix.c
@@ -291,33 +291,29 @@ int msix_init(struct PCIDevice *dev, unsigned short nentries,
 }
 
 int msix_init_exclusive_bar(PCIDevice *dev, unsigned short nentries,
-                            uint8_t bar_nr)
+                            uint8_t bar_nr, uint32_t bar_size)
 {
     int ret;
     char *name;
+    uint32_t bar_pba_offset = bar_size / 2;
 
     /*
      * Migration compatibility dictates that this remains a 4k
      * BAR with the vector table in the lower half and PBA in
      * the upper half.  Do not use these elsewhere!
      */
-#define MSIX_EXCLUSIVE_BAR_SIZE 4096
-#define MSIX_EXCLUSIVE_BAR_TABLE_OFFSET 0
-#define MSIX_EXCLUSIVE_BAR_PBA_OFFSET (MSIX_EXCLUSIVE_BAR_SIZE / 2)
-#define MSIX_EXCLUSIVE_CAP_OFFSET 0
-
-    if (nentries * PCI_MSIX_ENTRY_SIZE > MSIX_EXCLUSIVE_BAR_PBA_OFFSET) {
+    if (nentries * PCI_MSIX_ENTRY_SIZE > bar_pba_offset) {
         return -EINVAL;
     }
 
     name = g_strdup_printf("%s-msix", dev->name);
-    memory_region_init(&dev->msix_exclusive_bar, OBJECT(dev), name, MSIX_EXCLUSIVE_BAR_SIZE);
+    memory_region_init(&dev->msix_exclusive_bar, OBJECT(dev), name, bar_size);
     g_free(name);
 
     ret = msix_init(dev, nentries, &dev->msix_exclusive_bar, bar_nr,
-                    MSIX_EXCLUSIVE_BAR_TABLE_OFFSET, &dev->msix_exclusive_bar,
-                    bar_nr, MSIX_EXCLUSIVE_BAR_PBA_OFFSET,
-                    MSIX_EXCLUSIVE_CAP_OFFSET);
+                    0, &dev->msix_exclusive_bar,
+                    bar_nr, bar_pba_offset,
+                    0);
     if (ret) {
         return ret;
     }
diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index abaefaa..616d3be 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -973,7 +973,7 @@ static void virtio_pci_device_plugged(DeviceState *d)
     config[PCI_INTERRUPT_PIN] = 1;
 
     if (proxy->nvectors &&
-        msix_init_exclusive_bar(&proxy->pci_dev, proxy->nvectors, 1)) {
+        msix_init_exclusive_bar(&proxy->pci_dev, proxy->nvectors, 1, 4096)) {
         error_report("unable to init msix vectors to %" PRIu32,
                      proxy->nvectors);
         proxy->nvectors = 0;
diff --git a/include/hw/pci/msix.h b/include/hw/pci/msix.h
index 954d82b..43edebc 100644
--- a/include/hw/pci/msix.h
+++ b/include/hw/pci/msix.h
@@ -11,7 +11,7 @@ int msix_init(PCIDevice *dev, unsigned short nentries,
               unsigned table_offset, MemoryRegion *pba_bar,
               uint8_t pba_bar_nr, unsigned pba_offset, uint8_t cap_pos);
 int msix_init_exclusive_bar(PCIDevice *dev, unsigned short nentries,
-                            uint8_t bar_nr);
+                            uint8_t bar_nr, uint32_t bar_size);
 
 void msix_write_config(PCIDevice *dev, uint32_t address, uint32_t val, int len);
 
-- 
1.9.1

[Qemu-devel] [PATCH 5/5] virtio-pci: introduce auto_msix_bar_size property

[Jason Wang]

This patch introduces boolean auto_msix_bar_size property for virito-pci
devices. Enable this will let the device calculate the msix bar size
based on the number of MSI-X entries instead of previous 4096
hard-coded limit.

This is a must to let virtio-net can up to 256 queues and each queue
were associated with a specific MSI-X entry.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Richard Henderson <rth@twiddle.net>
Cc: Anthony Liguori <aliguori@amazon.com>
Cc: Michael S. Tsirkin <mst@redhat.com>
Cc: Alexander Graf <agraf@suse.de>
Cc: qemu-ppc@nongnu.org
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 hw/i386/pc_piix.c      |  4 ++++
 hw/i386/pc_q35.c       |  4 ++++
 hw/ppc/spapr.c         |  5 +++++
 hw/virtio/virtio-pci.c | 17 +++++++++++++++--
 hw/virtio/virtio-pci.h |  3 +++
 include/hw/compat.h    |  8 ++++++++
 6 files changed, 39 insertions(+), 2 deletions(-)

diff --git a/hw/i386/pc_piix.c b/hw/i386/pc_piix.c
index 38b42b0..2181435 100644
--- a/hw/i386/pc_piix.c
+++ b/hw/i386/pc_piix.c
@@ -531,6 +531,10 @@ static QEMUMachine pc_i440fx_machine_v2_2 = {
     PC_I440FX_2_2_MACHINE_OPTIONS,
     .name = "pc-i440fx-2.2",
     .init = pc_init_pci_2_2,
+    .compat_props = (GlobalProperty[]) {
+        HW_COMPAT_2_2,
+        { /* end of list */ }
+    },
 };
 
 #define PC_I440FX_2_1_MACHINE_OPTIONS                           \
diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
index 63027ee..dc65c51 100644
--- a/hw/i386/pc_q35.c
+++ b/hw/i386/pc_q35.c
@@ -428,6 +428,10 @@ static QEMUMachine pc_q35_machine_v2_2 = {
     PC_Q35_2_2_MACHINE_OPTIONS,
     .name = "pc-q35-2.2",
     .init = pc_q35_init_2_2,
+    .compat_props = (GlobalProperty[]) {
+        HW_COMPAT_2_2,
+        { /* end of list */ }
+    },
 };
 
 #define PC_Q35_2_1_MACHINE_OPTIONS                      \
diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
index b560459..b50d600 100644
--- a/hw/ppc/spapr.c
+++ b/hw/ppc/spapr.c
@@ -1777,11 +1777,16 @@ static const TypeInfo spapr_machine_2_1_info = {
 static void spapr_machine_2_2_class_init(ObjectClass *oc, void *data)
 {
     MachineClass *mc = MACHINE_CLASS(oc);
+    static GlobalProperty compat_props[] = {
+        HW_COMPAT_2_2,
+        { /* end of list */ }
+    };
 
     mc->name = "pseries-2.2";
     mc->desc = "pSeries Logical Partition (PAPR compliant) v2.2";
     mc->alias = "pseries";
     mc->is_default = 1;
+    mc->compat_props = compat_props;
 }
 
 static const TypeInfo spapr_machine_2_2_info = {
diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index 616d3be..5cffa18 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -961,7 +961,7 @@ static void virtio_pci_device_plugged(DeviceState *d)
     VirtIOPCIProxy *proxy = VIRTIO_PCI(d);
     VirtioBusState *bus = &proxy->bus;
     uint8_t *config;
-    uint32_t size;
+    uint32_t size, bar_size;
 
     config = proxy->pci_dev.config;
     if (proxy->class_code) {
@@ -972,8 +972,19 @@ static void virtio_pci_device_plugged(DeviceState *d)
     pci_set_word(config + PCI_SUBSYSTEM_ID, virtio_bus_get_vdev_id(bus));
     config[PCI_INTERRUPT_PIN] = 1;
 
+    if (proxy->flags & VIRTIO_PCI_FLAG_AUTO_MSIX_SIZE) {
+        bar_size = proxy->nvectors * PCI_MSIX_ENTRY_SIZE * 2;
+        if (bar_size & (bar_size - 1)) {
+            bar_size = 1 << qemu_fls(bar_size);
+        }
+    } else {
+        /* For migration compatibility */
+        bar_size = 4096;
+    }
+
     if (proxy->nvectors &&
-        msix_init_exclusive_bar(&proxy->pci_dev, proxy->nvectors, 1, 4096)) {
+        msix_init_exclusive_bar(&proxy->pci_dev, proxy->nvectors, 1,
+                                bar_size)) {
         error_report("unable to init msix vectors to %" PRIu32,
                      proxy->nvectors);
         proxy->nvectors = 0;
@@ -1418,6 +1429,8 @@ static const TypeInfo virtio_serial_pci_info = {
 static Property virtio_net_properties[] = {
     DEFINE_PROP_BIT("ioeventfd", VirtIOPCIProxy, flags,
                     VIRTIO_PCI_FLAG_USE_IOEVENTFD_BIT, false),
+    DEFINE_PROP_BIT("auto_msix_bar_size", VirtIOPCIProxy, flags,
+                    VIRTIO_PCI_FLAG_AUTO_MSIX_SIZE_BIT, true),
     DEFINE_PROP_UINT32("vectors", VirtIOPCIProxy, nvectors, 3),
     DEFINE_VIRTIO_NET_FEATURES(VirtIOPCIProxy, host_features),
     DEFINE_PROP_END_OF_LIST(),
diff --git a/hw/virtio/virtio-pci.h b/hw/virtio/virtio-pci.h
index 8873b6d..96dea82 100644
--- a/hw/virtio/virtio-pci.h
+++ b/hw/virtio/virtio-pci.h
@@ -62,6 +62,9 @@ typedef struct VirtioBusClass VirtioPCIBusClass;
  * vcpu thread using ioeventfd for some devices. */
 #define VIRTIO_PCI_FLAG_USE_IOEVENTFD_BIT 1
 #define VIRTIO_PCI_FLAG_USE_IOEVENTFD   (1 << VIRTIO_PCI_FLAG_USE_IOEVENTFD_BIT)
+#define VIRTIO_PCI_FLAG_AUTO_MSIX_SIZE_BIT 2
+#define VIRTIO_PCI_FLAG_AUTO_MSIX_SIZE \
+    (1 << VIRTIO_PCI_FLAG_AUTO_MSIX_SIZE_BIT)
 
 typedef struct {
     MSIMessage msg;
diff --git a/include/hw/compat.h b/include/hw/compat.h
index 313682a..3e5c5ae 100644
--- a/include/hw/compat.h
+++ b/include/hw/compat.h
@@ -1,7 +1,15 @@
 #ifndef HW_COMPAT_H
 #define HW_COMPAT_H
 
+#define HW_COMPAT_2_2 \
+        {\
+            .driver   = "virtio-net-pci",\
+            .property = "auto_msix_bar_size",\
+            .value    = "off",\
+        }
+
 #define HW_COMPAT_2_1 \
+        HW_COMPAT_2_2, \
         {\
             .driver   = "intel-hda",\
             .property = "old_msi_addr",\
-- 
1.9.1

Re: [Qemu-devel] [PATCH 1/5] virtio: rename VIRTIO_PCI_QUEUE_MAX to VIRTIO_QUEUE_MAX

[Cornelia Huck]

On Fri,  6 Feb 2015 15:54:09 +0800
Jason Wang <jasowang@redhat.com> wrote:

> VIRTIO_PCI_QUEUE_MAX was not specific to pci, so rename it to VIRTIO_QUEUE_MAX.
> 
> Cc: Amit Shah <amit.shah@redhat.com>
> Cc: Anthony Liguori <aliguori@amazon.com>
> Cc: Michael S. Tsirkin <mst@redhat.com>
> Cc: Alexander Graf <agraf@suse.de>
> Cc: Richard Henderson <rth@twiddle.net>
> Cc: Cornelia Huck <cornelia.huck@de.ibm.com>
> Cc: Christian Borntraeger <borntraeger@de.ibm.com>
> Cc: Paolo Bonzini <pbonzini@redhat.com>
> Signed-off-by: Jason Wang <jasowang@redhat.com>
> ---
>  hw/char/virtio-serial-bus.c  |  2 +-
>  hw/s390x/s390-virtio-bus.c   |  4 ++--
>  hw/s390x/s390-virtio-ccw.c   |  2 +-
>  hw/s390x/virtio-ccw.c        | 10 +++++-----
>  hw/scsi/virtio-scsi.c        |  4 ++--
>  hw/virtio/virtio-mmio.c      |  4 ++--
>  hw/virtio/virtio-pci.c       | 10 +++++-----
>  hw/virtio/virtio.c           | 26 +++++++++++++-------------
>  include/hw/s390x/s390_flic.h |  2 +-
>  include/hw/virtio/virtio.h   |  2 +-
>  10 files changed, 33 insertions(+), 33 deletions(-)

While I certainly agree with the patch on its own, I read through your
complete patch series as well and I have some additional comments.

Your patch series bumps up the maximum number of virtqueues for all
transports, but I think we need to consider carefully what this means
for each transport.

For virtio-ccw, for example, we'll run into trouble when the guest uses
classic queue indicators: These are limited to 64 per device (as I
assumed 64 virtqueues per device initially and they fit quite nicely
into a 64 bit integer). If the guest uses adapter interrupts with
two-stage indicators, we should probably be fine. The guest will
hopefully care about that, but we need to describe failure modes for
this in the spec.

For s390-virtio, I frankly have no idea what large numbers of queues
will do :) Maybe we just run out of space for devices much earlier.

You seem to need some changes for this to work for virtio-pci as well,
don't you?

Re: [Qemu-devel] [PATCH 2/5] virtio: increase VIRITO_QUEUE_MAX to 513

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 345 bytes --]

On Fri, Feb 06, 2015 at 03:54:10PM +0800, Jason Wang wrote:

Subject has a typo s/VIRITIO/VIRTIO/

> Recent linux kernel supports up to 256 tap queues. Increase the limit
> to 513 (256 * 2 + 1(ctrl vq)).

For other reviewers:
sizeof(VirtQueue) on x86_64 is 88 bytes.  We waste memory but it's only
on the order of 40 KB so I think this is okay.

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PATCH 0/5] Support more virtio queues

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1551 bytes --]

On Fri, Feb 06, 2015 at 03:54:08PM +0800, Jason Wang wrote:
> We current limit the max virtio queues to 64. This is not sufficient
> to support some multiqueue deivces (e.g recent Linux support up to 256
> tap queues). So this series try to let virtio to support more queues.
> 
> No much works needs to be done except:
> 
> - Patch 1 renames VIRTIO_PCI_QUEUE_MAX to a more generic name:
>   VIRTIO_QUEUE_MAX
> - Patch 2 simply increase the the limitation to 513 (256 queue pairs)
> - Patch 3 fixes a bug in virtio_del_queue()
> - Patch 4 tries to remove the hard coded MSI-X bar size (4096) and allow up to
>   2048 MSI-X entries.
> - Patch 5 add a boolean property to let the virtio-net can calculate
>   the MSI-X bar size based on the number of MSI-X vectors and keep
>   migration compability with legacy version whose bar size is 4096.
> 
> With this patch, we can support up to 256 queues.Since x86 can only
> allow about 240 interrupt vectors for MSI-X, current Linux driver can
> only have about 80 queue pairs has their private MSI-X interrupt
> vectors. With sharing IRQ with queue pairs (RFC posted in
> https://lkml.org/lkml/2014/12/25/169), Linux driver can have up
> to about 186 queue pairs has their private MSI-X interrupt vectors.

Worth mentioning that the change to VIRTIO_QUEUE_MAX is safe from a live
migration perspective:

Old QEMUs will not accept migration from new QEMUs launched with >64
queues:

  Invalid number of PCI queues: 0x41

Other than that, live migration works fine.

Stefan

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PATCH 2/5] virtio: increase VIRITO_QUEUE_MAX to 513

[Michael S. Tsirkin]

On Fri, Feb 06, 2015 at 03:54:10PM +0800, Jason Wang wrote:
> Recent linux kernel supports up to 256 tap queues. Increase the limit
> to 513 (256 * 2 + 1(ctrl vq)).
> 
> Signed-off-by: Jason Wang <jasowang@redhat.com>

We have a bunch of loops over all possible VQ numbers in virtio pci.
Doing this for 1000 VQs when most of them are inactive seems extreme - I
suspect we need a list of active VQs now.

> ---
>  include/hw/virtio/virtio.h | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
> index 220c09d..21bb30f 100644
> --- a/include/hw/virtio/virtio.h
> +++ b/include/hw/virtio/virtio.h
> @@ -92,7 +92,7 @@ typedef struct VirtQueueElement
>      struct iovec out_sg[VIRTQUEUE_MAX_SIZE];
>  } VirtQueueElement;
>  
> -#define VIRTIO_QUEUE_MAX 64
> +#define VIRTIO_QUEUE_MAX 513
>  
>  #define VIRTIO_NO_VECTOR 0xffff
>  
> -- 
> 1.9.1

Re: [Qemu-devel] [PATCH 1/5] virtio: rename VIRTIO_PCI_QUEUE_MAX to VIRTIO_QUEUE_MAX

[Jason Wang]

On Fri, Feb 6, 2015 at 7:54 PM, Cornelia Huck 
<cornelia.huck@de.ibm.com> wrote:
> On Fri,  6 Feb 2015 15:54:09 +0800
> Jason Wang <jasowang@redhat.com> wrote:
> 
>>  VIRTIO_PCI_QUEUE_MAX was not specific to pci, so rename it to 
>> VIRTIO_QUEUE_MAX.
>>  
>>  Cc: Amit Shah <amit.shah@redhat.com>
>>  Cc: Anthony Liguori <aliguori@amazon.com>
>>  Cc: Michael S. Tsirkin <mst@redhat.com>
>>  Cc: Alexander Graf <agraf@suse.de>
>>  Cc: Richard Henderson <rth@twiddle.net>
>>  Cc: Cornelia Huck <cornelia.huck@de.ibm.com>
>>  Cc: Christian Borntraeger <borntraeger@de.ibm.com>
>>  Cc: Paolo Bonzini <pbonzini@redhat.com>
>>  Signed-off-by: Jason Wang <jasowang@redhat.com>
>>  ---
>>   hw/char/virtio-serial-bus.c  |  2 +-
>>   hw/s390x/s390-virtio-bus.c   |  4 ++--
>>   hw/s390x/s390-virtio-ccw.c   |  2 +-
>>   hw/s390x/virtio-ccw.c        | 10 +++++-----
>>   hw/scsi/virtio-scsi.c        |  4 ++--
>>   hw/virtio/virtio-mmio.c      |  4 ++--
>>   hw/virtio/virtio-pci.c       | 10 +++++-----
>>   hw/virtio/virtio.c           | 26 +++++++++++++-------------
>>   include/hw/s390x/s390_flic.h |  2 +-
>>   include/hw/virtio/virtio.h   |  2 +-
>>   10 files changed, 33 insertions(+), 33 deletions(-)
> 
> While I certainly agree with the patch on its own, I read through your
> complete patch series as well and I have some additional comments.
> 
> Your patch series bumps up the maximum number of virtqueues for all
> transports, but I think we need to consider carefully what this means
> for each transport.
> 
> For virtio-ccw, for example, we'll run into trouble when the guest 
> uses
> classic queue indicators: These are limited to 64 per device (as I
> assumed 64 virtqueues per device initially and they fit quite nicely
> into a 64 bit integer). If the guest uses adapter interrupts with
> two-stage indicators, we should probably be fine. The guest will
> hopefully care about that, but we need to describe failure modes for
> this in the spec.
> 
> For s390-virtio, I frankly have no idea what large numbers of queues
> will do :) Maybe we just run out of space for devices much earlier.

I see the issue, thanks. So looks like we need a transport specific 
limitation for this.
> 
> 
> You seem to need some changes for this to work for virtio-pci as well,
> don't you?

PCI allows at most 2048 MSI-X entires, and also allow one entry to be 
shared by several virtqueues which looks fine for even thousands of 
virtqueues. If there's a architecture limitation (e.g x86 only allow 
240 interrupt vectors for MSI-X), the rest work was in the driver. If 
not, probably no much work needs to be done for PCI.

Thanks

Re: [Qemu-devel] [PATCH 2/5] virtio: increase VIRITO_QUEUE_MAX to 513

[Jason Wang]

On Sun, Feb 8, 2015 at 6:51 PM, Michael S. Tsirkin <mst@redhat.com> 
wrote:
> On Fri, Feb 06, 2015 at 03:54:10PM +0800, Jason Wang wrote:
>>  Recent linux kernel supports up to 256 tap queues. Increase the 
>> limit
>>  to 513 (256 * 2 + 1(ctrl vq)).
>>  
>>  Signed-off-by: Jason Wang <jasowang@redhat.com>
> 
> We have a bunch of loops over all possible VQ numbers in virtio pci.
> Doing this for 1000 VQs when most of them are inactive seems extreme 
> - I
> suspect we need a list of active VQs now.

We can do it on top of this series. But not sure it was really needed 
since none of those loops happen on data path. So we were probably ok.
> 
> 
>>  ---
>>   include/hw/virtio/virtio.h | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>  
>>  diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
>>  index 220c09d..21bb30f 100644
>>  --- a/include/hw/virtio/virtio.h
>>  +++ b/include/hw/virtio/virtio.h
>>  @@ -92,7 +92,7 @@ typedef struct VirtQueueElement
>>       struct iovec out_sg[VIRTQUEUE_MAX_SIZE];
>>   } VirtQueueElement;
>>   
>>  -#define VIRTIO_QUEUE_MAX 64
>>  +#define VIRTIO_QUEUE_MAX 513
>>   
>>   #define VIRTIO_NO_VECTOR 0xffff
>>   
>>  -- 
>>  1.9.1
>