[Qemu-devel] [PATCH v3 0/3] ivshmem: add a new PIO BAR4(Doorbell) besides MMIO BAR0 to reduce notification time

[Qemu-devel] [PATCH v3 0/3] ivshmem: add a new PIO BAR4(Doorbell) besides MMIO BAR0 to reduce notification time

[zanghongyong]

From: Hongyong Zang <zanghongyong@huawei.com>

This patch series, adds a PIO BAR4 for guest notifying qemu. And the new
notification way of PIO BAR4 reduces 30% time in comparison with the original
MMIO BAR0 way.
Meantime, this patch adds a memory API named kvm_set_ioeventfd_pio_long which
is about ioeventfd for PIO long. Also, this patch introduces a new feature
named IVSHMEM_PIO_NOTIFY to make PIO BAR4 disappeared for compatible with
machine type pc-1.0 or blow.

Notes: the patch series are based on the former patch "[PATCH] ivshmem: fix
guest unable to start with ioeventfd".

In v3, some changes come from Avi and Cam's comments:
  -split the v2 patch to an infrastructure patch(memory API) and an ivshmem patch
  -change BAR3 to BAR4 so that the shared memory region could be made a 64-bit
    BAR(take BAR2 and BAR3)
  -make BAR4 disappeared for compatible with machine type pc-1.0 or blow
  -update the ivshmem spec


Hongyong Zang (3):
  memory: add a memory API about ioeventfd for PIO long
  ivshmem: add a new PIO BAR4(Doorbell) to reduce notification time
  ivshmem: update the spec

 docs/specs/ivshmem_device_spec.txt |    8 ++++++--
 hw/ivshmem.c                       |   34 ++++++++++++++++++++++++++++++++--
 hw/pc_piix.c                       |   28 ++++++++++++++++++++++++++++
 kvm-all.c                          |   23 +++++++++++++++++++++++
 kvm-stub.c                         |    5 +++++
 kvm.h                              |    1 +
 memory.c                           |   20 ++++++++++++++++----
 7 files changed, 111 insertions(+), 8 deletions(-)

[Qemu-devel] [PATCH v3 1/3] memory: add a memory API about ioeventfd for PIO long

[zanghongyong]

From: Hongyong Zang <zanghongyong@huawei.com>

The new memory API, named kvm_set_ioeventfd_pio_long, is about ioeventfd for PIO long.

Signed-off-by: Hongyong Zang <zanghongyong@huawei.com>
---
 kvm-all.c  |   23 +++++++++++++++++++++++
 kvm-stub.c |    5 +++++
 kvm.h      |    1 +
 memory.c   |   20 ++++++++++++++++----
 4 files changed, 45 insertions(+), 4 deletions(-)

diff --git a/kvm-all.c b/kvm-all.c
index 4c466d6..4614c5d 100644
--- a/kvm-all.c
+++ b/kvm-all.c
@@ -1365,6 +1365,29 @@ int kvm_set_ioeventfd_mmio_long(int fd, uint32_t addr, uint32_t val, bool assign
     return 0;
 }
 
+int kvm_set_ioeventfd_pio_long(int fd, uint32_t addr, uint32_t val, bool assign)
+{
+    struct kvm_ioeventfd kick = {
+        .datamatch = val,
+        .addr = addr,
+        .len = 4,
+        .flags = KVM_IOEVENTFD_FLAG_DATAMATCH | KVM_IOEVENTFD_FLAG_PIO,
+        .fd = fd,
+    };
+    int r;
+    if (!kvm_enabled()) {
+        return -ENOSYS;
+    }
+    if (!assign) {
+        kick.flags |= KVM_IOEVENTFD_FLAG_DEASSIGN;
+    }
+    r = kvm_vm_ioctl(kvm_state, KVM_IOEVENTFD, &kick);
+    if (r < 0) {
+        return r;
+    }
+    return 0;
+}
+
 int kvm_set_ioeventfd_pio_word(int fd, uint16_t addr, uint16_t val, bool assign)
 {
     struct kvm_ioeventfd kick = {
diff --git a/kvm-stub.c b/kvm-stub.c
index 06064b9..64cdd7c 100644
--- a/kvm-stub.c
+++ b/kvm-stub.c
@@ -115,6 +115,11 @@ int kvm_set_ioeventfd_pio_word(int fd, uint16_t addr, uint16_t val, bool assign)
     return -ENOSYS;
 }
 
+int kvm_set_ioeventfd_pio_long(int fd, uint32_t adr, uint32_t val, bool assign)
+{
+    return -ENOSYS;
+}
+
 int kvm_set_ioeventfd_mmio_long(int fd, uint32_t adr, uint32_t val, bool assign)
 {
     return -ENOSYS;
diff --git a/kvm.h b/kvm.h
index 243b063..64b1737 100644
--- a/kvm.h
+++ b/kvm.h
@@ -195,5 +195,6 @@ int kvm_physical_memory_addr_from_ram(KVMState *s, ram_addr_t ram_addr,
 #endif
 int kvm_set_ioeventfd_mmio_long(int fd, uint32_t adr, uint32_t val, bool assign);
 
+int kvm_set_ioeventfd_pio_long(int fd, uint32_t adr, uint32_t val, bool assign);
 int kvm_set_ioeventfd_pio_word(int fd, uint16_t adr, uint16_t val, bool assign);
 #endif
diff --git a/memory.c b/memory.c
index adfdf14..544c955 100644
--- a/memory.c
+++ b/memory.c
@@ -480,10 +480,16 @@ static void as_io_ioeventfd_add(AddressSpace *as, MemoryRegionIoeventfd *fd)
 {
     int r;
 
-    assert(fd->match_data && int128_get64(fd->addr.size) == 2);
-
-    r = kvm_set_ioeventfd_pio_word(fd->fd, int128_get64(fd->addr.start),
+    assert(fd->match_data && (int128_get64(fd->addr.size) == 2 ||
+                              int128_get64(fd->addr.size) == 4));
+    if(int128_get64(fd->addr.size) == 2) {
+        r = kvm_set_ioeventfd_pio_word(fd->fd, int128_get64(fd->addr.start),
+                                   fd->data, true);
+    }
+    else {
+        r = kvm_set_ioeventfd_pio_long(fd->fd, int128_get64(fd->addr.start),
                                    fd->data, true);
+    }
     if (r < 0) {
         abort();
     }
@@ -493,8 +499,14 @@ static void as_io_ioeventfd_del(AddressSpace *as, MemoryRegionIoeventfd *fd)
 {
     int r;
 
-    r = kvm_set_ioeventfd_pio_word(fd->fd, int128_get64(fd->addr.start),
+    if(int128_get64(fd->addr.size) == 2) {
+        r = kvm_set_ioeventfd_pio_word(fd->fd, int128_get64(fd->addr.start),
                                    fd->data, false);
+    }
+    else {
+        r = kvm_set_ioeventfd_pio_long(fd->fd, int128_get64(fd->addr.start),
+                                   fd->data, false);
+    }
     if (r < 0) {
         abort();
     }
-- 
1.7.1

[Qemu-devel] [PATCH v3 2/3] ivshmem: add a new PIO BAR4(Doorbell) to reduce notification time

[zanghongyong]

From: Hongyong Zang <zanghongyong@huawei.com>

This patch adds a PIO BAR4 for guest notifying qemu to reduce notification time.
And the new notification way of PIO BAR4 reduces 30% time in comparison with the
original MMIO BAR0 way.

Also, this patch introduces a new feature named IVSHMEM_PIO_NOTIFY to make PIO
BAR4 disappeared for compatible with machine type pc-1.0 or blow.

Signed-off-by: Hongyong Zang <zanghongyong@huawei.com>
---
 hw/ivshmem.c |   34 ++++++++++++++++++++++++++++++++--
 hw/pc_piix.c |   28 ++++++++++++++++++++++++++++
 2 files changed, 60 insertions(+), 2 deletions(-)

diff --git a/hw/ivshmem.c b/hw/ivshmem.c
index 80b5db0..6845ade 100644
--- a/hw/ivshmem.c
+++ b/hw/ivshmem.c
@@ -26,11 +26,13 @@
 
 #define IVSHMEM_IOEVENTFD   0
 #define IVSHMEM_MSI     1
+#define IVSHMEM_PIO_NOTIFY 2
 
 #define IVSHMEM_PEER    0
 #define IVSHMEM_MASTER  1
 
 #define IVSHMEM_REG_BAR_SIZE 0x100
+#define IVSHIO_REG_BAR_SIZE 0x10
 
 //#define DEBUG_IVSHMEM
 #ifdef DEBUG_IVSHMEM
@@ -59,6 +61,7 @@ typedef struct IVShmemState {
     CharDriverState **eventfd_chr;
     CharDriverState *server_chr;
     MemoryRegion ivshmem_mmio;
+    MemoryRegion ivshmem_pio;
 
     /* We might need to register the BAR before we actually have the memory.
      * So prepare a container MemoryRegion for the BAR immediately and
@@ -237,7 +240,7 @@ static uint64_t ivshmem_io_read(void *opaque, target_phys_addr_t addr,
     return ret;
 }
 
-static const MemoryRegionOps ivshmem_mmio_ops = {
+static const MemoryRegionOps ivshmem_io_ops = {
     .read = ivshmem_io_read,
     .write = ivshmem_io_write,
     .endianness = DEVICE_NATIVE_ENDIAN,
@@ -356,6 +359,14 @@ static void close_guest_eventfds(IVShmemState *s, int posn)
                                      true,
                                      (posn << 16) | i,
                                      s->peers[posn].eventfds[i]);
+            if(ivshmem_has_feature(s, IVSHMEM_PIO_NOTIFY)) {
+                memory_region_del_eventfd(&s->ivshmem_pio,
+                                         DOORBELL,
+                                         4,
+                                         true,
+                                         (posn << 16) | i,
+                                         s->peers[posn].eventfds[i]);
+            }
         }
         close(s->peers[posn].eventfds[i]);
     }
@@ -490,6 +501,14 @@ static void ivshmem_read(void *opaque, const uint8_t * buf, int flags)
                                   true,
                                   (incoming_posn << 16) | guest_max_eventfd,
                                   incoming_fd);
+        if(ivshmem_has_feature(s, IVSHMEM_PIO_NOTIFY)) {
+            memory_region_add_eventfd(&s->ivshmem_pio,
+                                      DOORBELL,
+                                      4,
+                                      true,
+                                      (incoming_posn << 16) | guest_max_eventfd,
+                                      incoming_fd);
+        }
     }
 
     return;
@@ -652,13 +671,20 @@ static int pci_ivshmem_init(PCIDevice *dev)
 
     s->shm_fd = 0;
 
-    memory_region_init_io(&s->ivshmem_mmio, &ivshmem_mmio_ops, s,
+    memory_region_init_io(&s->ivshmem_mmio, &ivshmem_io_ops, s,
                           "ivshmem-mmio", IVSHMEM_REG_BAR_SIZE);
 
     /* region for registers*/
     pci_register_bar(&s->dev, 0, PCI_BASE_ADDRESS_SPACE_MEMORY,
                      &s->ivshmem_mmio);
 
+    if(ivshmem_has_feature(s, IVSHMEM_PIO_NOTIFY)) {
+        memory_region_init_io(&s->ivshmem_pio, &ivshmem_io_ops, s,
+                          "ivshmem-pio", IVSHIO_REG_BAR_SIZE);
+        pci_register_bar(&s->dev, 4, PCI_BASE_ADDRESS_SPACE_IO,
+                     &s->ivshmem_pio);
+    }
+
     memory_region_init(&s->bar, "ivshmem-bar2-container", s->ivshmem_size);
 
     if ((s->server_chr != NULL) &&
@@ -738,6 +764,9 @@ static int pci_ivshmem_uninit(PCIDevice *dev)
         error_free(s->migration_blocker);
     }
 
+    if(ivshmem_has_feature(s, IVSHMEM_PIO_NOTIFY)) {
+        memory_region_destroy(&s->ivshmem_pio);
+    }
     memory_region_destroy(&s->ivshmem_mmio);
     memory_region_del_subregion(&s->bar, &s->ivshmem);
     memory_region_destroy(&s->ivshmem);
@@ -762,6 +791,7 @@ static PCIDeviceInfo ivshmem_info = {
         DEFINE_PROP_UINT32("vectors", IVShmemState, vectors, 1),
         DEFINE_PROP_BIT("ioeventfd", IVShmemState, features, IVSHMEM_IOEVENTFD, false),
         DEFINE_PROP_BIT("msi", IVShmemState, features, IVSHMEM_MSI, true),
+        DEFINE_PROP_BIT("pio_notify", IVShmemState, features, IVSHMEM_PIO_NOTIFY, true),
         DEFINE_PROP_STRING("shm", IVShmemState, shmobj),
         DEFINE_PROP_STRING("role", IVShmemState, role),
         DEFINE_PROP_END_OF_LIST(),
diff --git a/hw/pc_piix.c b/hw/pc_piix.c
index 970f43c..fe64874 100644
--- a/hw/pc_piix.c
+++ b/hw/pc_piix.c
@@ -304,6 +304,14 @@ static QEMUMachine pc_machine_v1_0 = {
     .init = pc_init_pci,
     .max_cpus = 255,
     .is_default = 1,
+    .compat_props = (GlobalProperty[]) {
+        {
+            .driver   = "ivshmem",
+            .property = "pio_notify",
+            .value    = "off",
+        },
+        { /* end of list */ }
+    },
 };
 
 static QEMUMachine pc_machine_v0_14 = {
@@ -320,6 +328,10 @@ static QEMUMachine pc_machine_v0_14 = {
             .driver   = "qxl-vga",
             .property = "revision",
             .value    = stringify(2),
+        },{
+            .driver   = "ivshmem",
+            .property = "pio_notify",
+            .value    = "off",
         },
         { /* end of list */ }
     },
@@ -363,6 +375,10 @@ static QEMUMachine pc_machine_v0_13 = {
             .driver   = "AC97",
             .property = "use_broken_id",
             .value    = stringify(1),
+        },{
+            .driver   = "ivshmem",
+            .property = "pio_notify",
+            .value    = "off",
         },
         { /* end of list */ }
     },
@@ -410,6 +426,10 @@ static QEMUMachine pc_machine_v0_12 = {
             .driver   = "AC97",
             .property = "use_broken_id",
             .value    = stringify(1),
+        },{
+            .driver   = "ivshmem",
+            .property = "pio_notify",
+            .value    = "off",
         },
         { /* end of list */ }
     }
@@ -465,6 +485,10 @@ static QEMUMachine pc_machine_v0_11 = {
             .driver   = "AC97",
             .property = "use_broken_id",
             .value    = stringify(1),
+        },{
+            .driver   = "ivshmem",
+            .property = "pio_notify",
+            .value    = "off",
         },
         { /* end of list */ }
     }
@@ -532,6 +556,10 @@ static QEMUMachine pc_machine_v0_10 = {
             .driver   = "AC97",
             .property = "use_broken_id",
             .value    = stringify(1),
+        },{
+            .driver   = "ivshmem",
+            .property = "pio_notify",
+            .value    = "off",
         },
         { /* end of list */ }
     },
-- 
1.7.1

[Qemu-devel] [PATCH v3 3/3] ivshmem: update the spec

[zanghongyong]

From: Hongyong Zang <zanghongyong@huawei.com>


Signed-off-by: Hongyong Zang <zanghongyong@huawei.com>
---
 docs/specs/ivshmem_device_spec.txt |    8 ++++++--
 1 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/docs/specs/ivshmem_device_spec.txt b/docs/specs/ivshmem_device_spec.txt
index 23dd2ba..d36c737 100644
--- a/docs/specs/ivshmem_device_spec.txt
+++ b/docs/specs/ivshmem_device_spec.txt
@@ -13,10 +13,14 @@ The Inter-VM PCI device
 
 *BARs*
 
-The device supports three BARs.  BAR0 is a 1 Kbyte MMIO region to support
+The device supports four BARs.  BAR0 is a 1 Kbyte MMIO region to support
 registers.  BAR1 is used for MSI-X when it is enabled in the device.  BAR2 is
 used to map the shared memory object from the host.  The size of BAR2 is
 specified when the guest is started and must be a power of 2 in size.
+BAR4 is a 16 bytes PIO region to support registers. BAR4 plays the same role
+as BAR0, while it reduces notifying time 30% in comparison with BAR0. For
+compatibility, BAR4 will be not visible on guests created with -M pc-1.0 or
+below.
 
 *Registers*
 
@@ -89,7 +93,7 @@ Usage in the Guest
 ------------------
 
 The shared memory device is intended to be used with the provided UIO driver.
-Very little configuration is needed.  The guest should map BAR0 to access the
+Very little configuration is needed.  The guest should map BAR0 or BAR4 to access the
 registers (an array of 32-bit ints allows simple writing) and map BAR2 to
 access the shared memory region itself.  The size of the shared memory region
 is specified when the guest (or shared memory server) is started.  A guest may
-- 
1.7.1