[Qemu-devel] [PATCH V8 0/4] hw/pvrdma: PVRDMA device implementation

[Qemu-devel] [PATCH V8 0/4] hw/pvrdma: PVRDMA device implementation

[Marcel Apfelbaum]

V7 -> V8:
 - Addressed Michael S. Tsirkin comments:
   - fail to init the pvrdma device if target page size is different
     from the host size, or if the guest RAM is not backed by memory
     and shared.
   - Update documentation to include a note on huge memory regions
     registration and remove not needed info.
 - Removed "pci/shpc: Move function to generic header file" since it
   appears in latest maintainer pull request
 - Rebased on master

V6 -> V7:
 - Addressed Philippe Mathieu-Daudé comments
   - modified pow2roundup32 signature
   - added his RB tag (thanks)
 - Addressed Corenlia Huck comments:
   - Compiled the pvrdma for all archs and not only x86/arm (thanks)
   - Fixed typo in documentation
 - Rebased on latest master

V5 -> V6:
 - Found a ppc machine and solved ppc compilation issues
 - Tried to fix the s390x issue (still looking of a machine)

V4 -> V5:
 - Fixed (at least tried to) compilation issues

V3 -> V4:
 - Fixed documentation (added more impl details)
 - Fixed compilation errors discovered by patchew.
 - Addressed Michael S. Tsirkin comments:
   - Removed unnecessary typedefs and replace them with
     macros in VMware header files, together with explanations.
   - Moved more code from vmw specific to rdma generic code.
   - Added page size limitations to the documentation.

V2 -> V3:
 - Addressed Michael S. Tsirkin and Philippe Mathieu-Daudé comments:
   - Moved the device to hw/rdma
 - Addressed Michael S. Tsirkin comments:
   - Split the code into generic (hw/rdma) and VMWare
     specific (hw/rdma/vmw)
   - Added more details to documentation - VMware guest-host protocol.
   - Remove mad processing
   - limited the memory the Guest can pin.
 - Addressed Philippe Mathieu-Daudé comment:
   - s/roundup_pow_of_two/pow2roundup32 and move it to qemu/host-utils.h 
 - Added Shamit Rabinovici's review to documentation
 - Rebased to latest master 

RFC -> V2:
 - Full implementation of the pvrdma device
 - Backend is an ibdevice interface, no need for the KDBR module


General description
===================
PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device.
It works with its Linux Kernel driver AS IS, no need for any special guest
modifications.

While it complies with the VMware device, it can also communicate with bare
metal RDMA-enabled machines and does not require an RDMA HCA in the host, it
can work with Soft-RoCE (rxe).

It does not require the whole guest RAM to be pinned allowing memory
over-commit and, even if not implemented yet, migration support will be
possible with some HW assistance.


 Design
 ======
 - Follows the behavior of VMware's pvrdma device, however is not tightly
   coupled with it and most of the code can be reused if we decide to
   continue to a Virtio based RDMA device.

 - It exposes 3 BARs:
    BAR 0 - MSIX, utilize 3 vectors for command ring, async events and
            completions
    BAR 1 - Configuration of registers
    BAR 2 - UAR, used to pass HW commands from driver.

 - The device performs internal management of the RDMA
   resources (PDs, CQs, QPs, ...), meaning the objects
   are not directly coupled to a physical RDMA device resources.

The pvrdma backend is an ibdevice interface that can be exposed
either by a Soft-RoCE(rxe) device on machines with no RDMA device,
or an HCA SRIOV function(VF/PF).
Note that ibdevice interfaces can't be shared between pvrdma devices,
each one requiring a separate instance (rxe or SRIOV VF).


Tests and performance
=====================
Tested with SoftRoCE backend (rxe)/Mellanox ConnectX3,
and Mellanox ConnectX4 HCAs with:
  - VMs in the same host
  - VMs in different hosts 
  - VMs to bare metal.

The best performance achieved with ConnectX HCAs and buffer size
bigger than 1MB which was the line rate ~ 50Gb/s.
The conclusion is that using the PVRDMA device there are no
actual performance penalties compared to bare metal for big enough
buffers (which is quite common when using RDMA), while allowing
memory overcommit.

Marcel Apfelbaum (3):
  mem: add share parameter to memory-backend-ram
  docs: add pvrdma device documentation.
  MAINTAINERS: add entry for hw/rdma

Yuval Shaia (1):
  pvrdma: initial implementation

 MAINTAINERS                   |   8 +
 Makefile.objs                 |   2 +
 backends/hostmem-file.c       |  25 +-
 backends/hostmem-ram.c        |   4 +-
 backends/hostmem.c            |  21 ++
 configure                     |   9 +-
 docs/pvrdma.txt               | 255 +++++++++++++
 exec.c                        |  26 +-
 hw/Makefile.objs              |   1 +
 hw/rdma/Makefile.objs         |   6 +
 hw/rdma/rdma_backend.c        | 815 ++++++++++++++++++++++++++++++++++++++++++
 hw/rdma/rdma_backend.h        |  92 +++++
 hw/rdma/rdma_backend_defs.h   |  62 ++++
 hw/rdma/rdma_rm.c             | 619 ++++++++++++++++++++++++++++++++
 hw/rdma/rdma_rm.h             |  69 ++++
 hw/rdma/rdma_rm_defs.h        | 106 ++++++
 hw/rdma/rdma_utils.c          |  52 +++
 hw/rdma/rdma_utils.h          |  43 +++
 hw/rdma/trace-events          |   5 +
 hw/rdma/vmw/pvrdma.h          | 122 +++++++
 hw/rdma/vmw/pvrdma_cmd.c      | 679 +++++++++++++++++++++++++++++++++++
 hw/rdma/vmw/pvrdma_dev_api.h  | 602 +++++++++++++++++++++++++++++++
 hw/rdma/vmw/pvrdma_dev_ring.c | 139 +++++++
 hw/rdma/vmw/pvrdma_dev_ring.h |  42 +++
 hw/rdma/vmw/pvrdma_ib_verbs.h | 433 ++++++++++++++++++++++
 hw/rdma/vmw/pvrdma_main.c     | 671 ++++++++++++++++++++++++++++++++++
 hw/rdma/vmw/pvrdma_qp_ops.c   | 212 +++++++++++
 hw/rdma/vmw/pvrdma_qp_ops.h   |  27 ++
 hw/rdma/vmw/pvrdma_ring.h     | 134 +++++++
 hw/rdma/vmw/trace-events      |   5 +
 hw/rdma/vmw/vmw_pvrdma-abi.h  | 311 ++++++++++++++++
 include/exec/memory.h         |  23 ++
 include/exec/ram_addr.h       |   3 +-
 include/hw/pci/pci_ids.h      |   3 +
 include/qemu/osdep.h          |   2 +-
 include/sysemu/hostmem.h      |   2 +-
 include/sysemu/kvm.h          |   2 +-
 memory.c                      |  16 +-
 target/s390x/kvm.c            |   4 +-
 util/oslib-posix.c            |   4 +-
 util/oslib-win32.c            |   2 +-
 41 files changed, 5605 insertions(+), 53 deletions(-)
 create mode 100644 docs/pvrdma.txt
 create mode 100644 hw/rdma/Makefile.objs
 create mode 100644 hw/rdma/rdma_backend.c
 create mode 100644 hw/rdma/rdma_backend.h
 create mode 100644 hw/rdma/rdma_backend_defs.h
 create mode 100644 hw/rdma/rdma_rm.c
 create mode 100644 hw/rdma/rdma_rm.h
 create mode 100644 hw/rdma/rdma_rm_defs.h
 create mode 100644 hw/rdma/rdma_utils.c
 create mode 100644 hw/rdma/rdma_utils.h
 create mode 100644 hw/rdma/trace-events
 create mode 100644 hw/rdma/vmw/pvrdma.h
 create mode 100644 hw/rdma/vmw/pvrdma_cmd.c
 create mode 100644 hw/rdma/vmw/pvrdma_dev_api.h
 create mode 100644 hw/rdma/vmw/pvrdma_dev_ring.c
 create mode 100644 hw/rdma/vmw/pvrdma_dev_ring.h
 create mode 100644 hw/rdma/vmw/pvrdma_ib_verbs.h
 create mode 100644 hw/rdma/vmw/pvrdma_main.c
 create mode 100644 hw/rdma/vmw/pvrdma_qp_ops.c
 create mode 100644 hw/rdma/vmw/pvrdma_qp_ops.h
 create mode 100644 hw/rdma/vmw/pvrdma_ring.h
 create mode 100644 hw/rdma/vmw/trace-events
 create mode 100644 hw/rdma/vmw/vmw_pvrdma-abi.h

-- 
2.13.5

[Qemu-devel] [PATCH V8 1/4] mem: add share parameter to memory-backend-ram

[Marcel Apfelbaum]

Currently only file backed memory backend can
be created with a "share" flag in order to allow
sharing guest RAM with other processes in the host.

Add the "share" flag also to RAM Memory Backend
in order to allow remapping parts of the guest RAM
to different host virtual addresses. This is needed
by the RDMA devices in order to remap non-contiguous
QEMU virtual addresses to a contiguous virtual address range.

Moved the "share" flag to the Host Memory base class,
modified phys_mem_alloc to include the new parameter
and a new interface memory_region_init_ram_shared_nomigrate.

There are no functional changes if the new flag is not used.

Signed-off-by: Marcel Apfelbaum <marcel@redhat.com>
---
 backends/hostmem-file.c  | 25 +------------------------
 backends/hostmem-ram.c   |  4 ++--
 backends/hostmem.c       | 21 +++++++++++++++++++++
 exec.c                   | 26 +++++++++++++++-----------
 include/exec/memory.h    | 23 +++++++++++++++++++++++
 include/exec/ram_addr.h  |  3 ++-
 include/qemu/osdep.h     |  2 +-
 include/sysemu/hostmem.h |  2 +-
 include/sysemu/kvm.h     |  2 +-
 memory.c                 | 16 +++++++++++++---
 target/s390x/kvm.c       |  4 ++--
 util/oslib-posix.c       |  4 ++--
 util/oslib-win32.c       |  2 +-
 13 files changed, 85 insertions(+), 49 deletions(-)

diff --git a/backends/hostmem-file.c b/backends/hostmem-file.c
index e44c319915..bc95022a68 100644
--- a/backends/hostmem-file.c
+++ b/backends/hostmem-file.c
@@ -31,7 +31,6 @@ typedef struct HostMemoryBackendFile HostMemoryBackendFile;
 struct HostMemoryBackendFile {
     HostMemoryBackend parent_obj;
 
-    bool share;
     bool discard_data;
     char *mem_path;
 };
@@ -58,7 +57,7 @@ file_backend_memory_alloc(HostMemoryBackend *backend, Error **errp)
         path = object_get_canonical_path(OBJECT(backend));
         memory_region_init_ram_from_file(&backend->mr, OBJECT(backend),
                                  path,
-                                 backend->size, fb->share,
+                                 backend->size, backend->share,
                                  fb->mem_path, errp);
         g_free(path);
     }
@@ -85,25 +84,6 @@ static void set_mem_path(Object *o, const char *str, Error **errp)
     fb->mem_path = g_strdup(str);
 }
 
-static bool file_memory_backend_get_share(Object *o, Error **errp)
-{
-    HostMemoryBackendFile *fb = MEMORY_BACKEND_FILE(o);
-
-    return fb->share;
-}
-
-static void file_memory_backend_set_share(Object *o, bool value, Error **errp)
-{
-    HostMemoryBackend *backend = MEMORY_BACKEND(o);
-    HostMemoryBackendFile *fb = MEMORY_BACKEND_FILE(o);
-
-    if (host_memory_backend_mr_inited(backend)) {
-        error_setg(errp, "cannot change property value");
-        return;
-    }
-    fb->share = value;
-}
-
 static bool file_memory_backend_get_discard_data(Object *o, Error **errp)
 {
     return MEMORY_BACKEND_FILE(o)->discard_data;
@@ -136,9 +116,6 @@ file_backend_class_init(ObjectClass *oc, void *data)
     bc->alloc = file_backend_memory_alloc;
     oc->unparent = file_backend_unparent;
 
-    object_class_property_add_bool(oc, "share",
-        file_memory_backend_get_share, file_memory_backend_set_share,
-        &error_abort);
     object_class_property_add_bool(oc, "discard-data",
         file_memory_backend_get_discard_data, file_memory_backend_set_discard_data,
         &error_abort);
diff --git a/backends/hostmem-ram.c b/backends/hostmem-ram.c
index 38977be73e..7ddd08d370 100644
--- a/backends/hostmem-ram.c
+++ b/backends/hostmem-ram.c
@@ -28,8 +28,8 @@ ram_backend_memory_alloc(HostMemoryBackend *backend, Error **errp)
     }
 
     path = object_get_canonical_path_component(OBJECT(backend));
-    memory_region_init_ram_nomigrate(&backend->mr, OBJECT(backend), path,
-                           backend->size, errp);
+    memory_region_init_ram_shared_nomigrate(&backend->mr, OBJECT(backend), path,
+                           backend->size, backend->share, errp);
     g_free(path);
 }
 
diff --git a/backends/hostmem.c b/backends/hostmem.c
index ee2c2d5bfd..1daf13bd2e 100644
--- a/backends/hostmem.c
+++ b/backends/hostmem.c
@@ -369,6 +369,24 @@ static void set_id(Object *o, const char *str, Error **errp)
     backend->id = g_strdup(str);
 }
 
+static bool host_memory_backend_get_share(Object *o, Error **errp)
+{
+    HostMemoryBackend *backend = MEMORY_BACKEND(o);
+
+    return backend->share;
+}
+
+static void host_memory_backend_set_share(Object *o, bool value, Error **errp)
+{
+    HostMemoryBackend *backend = MEMORY_BACKEND(o);
+
+    if (host_memory_backend_mr_inited(backend)) {
+        error_setg(errp, "cannot change property value");
+        return;
+    }
+    backend->share = value;
+}
+
 static void
 host_memory_backend_class_init(ObjectClass *oc, void *data)
 {
@@ -399,6 +417,9 @@ host_memory_backend_class_init(ObjectClass *oc, void *data)
         host_memory_backend_get_policy,
         host_memory_backend_set_policy, &error_abort);
     object_class_property_add_str(oc, "id", get_id, set_id, &error_abort);
+    object_class_property_add_bool(oc, "share",
+        host_memory_backend_get_share, host_memory_backend_set_share,
+        &error_abort);
 }
 
 static void host_memory_backend_finalize(Object *o)
diff --git a/exec.c b/exec.c
index d28fc0cd3d..7f543fba80 100644
--- a/exec.c
+++ b/exec.c
@@ -1285,7 +1285,7 @@ static int subpage_register (subpage_t *mmio, uint32_t start, uint32_t end,
                              uint16_t section);
 static subpage_t *subpage_init(FlatView *fv, hwaddr base);
 
-static void *(*phys_mem_alloc)(size_t size, uint64_t *align) =
+static void *(*phys_mem_alloc)(size_t size, uint64_t *align, bool shared) =
                                qemu_anon_ram_alloc;
 
 /*
@@ -1293,7 +1293,7 @@ static void *(*phys_mem_alloc)(size_t size, uint64_t *align) =
  * Accelerators with unusual needs may need this.  Hopefully, we can
  * get rid of it eventually.
  */
-void phys_mem_set_alloc(void *(*alloc)(size_t, uint64_t *align))
+void phys_mem_set_alloc(void *(*alloc)(size_t, uint64_t *align, bool shared))
 {
     phys_mem_alloc = alloc;
 }
@@ -1915,7 +1915,7 @@ static void dirty_memory_extend(ram_addr_t old_ram_size,
     }
 }
 
-static void ram_block_add(RAMBlock *new_block, Error **errp)
+static void ram_block_add(RAMBlock *new_block, Error **errp, bool shared)
 {
     RAMBlock *block;
     RAMBlock *last_block = NULL;
@@ -1938,7 +1938,7 @@ static void ram_block_add(RAMBlock *new_block, Error **errp)
             }
         } else {
             new_block->host = phys_mem_alloc(new_block->max_length,
-                                             &new_block->mr->align);
+                                             &new_block->mr->align, shared);
             if (!new_block->host) {
                 error_setg_errno(errp, errno,
                                  "cannot set up guest memory '%s'",
@@ -2043,7 +2043,7 @@ RAMBlock *qemu_ram_alloc_from_fd(ram_addr_t size, MemoryRegion *mr,
         return NULL;
     }
 
-    ram_block_add(new_block, &local_err);
+    ram_block_add(new_block, &local_err, share);
     if (local_err) {
         g_free(new_block);
         error_propagate(errp, local_err);
@@ -2085,7 +2085,7 @@ RAMBlock *qemu_ram_alloc_internal(ram_addr_t size, ram_addr_t max_size,
                                   void (*resized)(const char*,
                                                   uint64_t length,
                                                   void *host),
-                                  void *host, bool resizeable,
+                                  void *host, bool resizeable, bool share,
                                   MemoryRegion *mr, Error **errp)
 {
     RAMBlock *new_block;
@@ -2108,7 +2108,7 @@ RAMBlock *qemu_ram_alloc_internal(ram_addr_t size, ram_addr_t max_size,
     if (resizeable) {
         new_block->flags |= RAM_RESIZEABLE;
     }
-    ram_block_add(new_block, &local_err);
+    ram_block_add(new_block, &local_err, share);
     if (local_err) {
         g_free(new_block);
         error_propagate(errp, local_err);
@@ -2120,12 +2120,15 @@ RAMBlock *qemu_ram_alloc_internal(ram_addr_t size, ram_addr_t max_size,
 RAMBlock *qemu_ram_alloc_from_ptr(ram_addr_t size, void *host,
                                    MemoryRegion *mr, Error **errp)
 {
-    return qemu_ram_alloc_internal(size, size, NULL, host, false, mr, errp);
+    return qemu_ram_alloc_internal(size, size, NULL, host, false,
+                                   false, mr, errp);
 }
 
-RAMBlock *qemu_ram_alloc(ram_addr_t size, MemoryRegion *mr, Error **errp)
+RAMBlock *qemu_ram_alloc(ram_addr_t size, bool share,
+                         MemoryRegion *mr, Error **errp)
 {
-    return qemu_ram_alloc_internal(size, size, NULL, NULL, false, mr, errp);
+    return qemu_ram_alloc_internal(size, size, NULL, NULL, false,
+                                   share, mr, errp);
 }
 
 RAMBlock *qemu_ram_alloc_resizeable(ram_addr_t size, ram_addr_t maxsz,
@@ -2134,7 +2137,8 @@ RAMBlock *qemu_ram_alloc_resizeable(ram_addr_t size, ram_addr_t maxsz,
                                                      void *host),
                                      MemoryRegion *mr, Error **errp)
 {
-    return qemu_ram_alloc_internal(size, maxsz, resized, NULL, true, mr, errp);
+    return qemu_ram_alloc_internal(size, maxsz, resized, NULL, true,
+                                   false, mr, errp);
 }
 
 static void reclaim_ramblock(RAMBlock *block)
diff --git a/include/exec/memory.h b/include/exec/memory.h
index a4cabdf44c..dd28eaba68 100644
--- a/include/exec/memory.h
+++ b/include/exec/memory.h
@@ -428,6 +428,29 @@ void memory_region_init_ram_nomigrate(MemoryRegion *mr,
                                       Error **errp);
 
 /**
+ * memory_region_init_ram_shared_nomigrate:  Initialize RAM memory region.
+ *                                           Accesses into the region will
+ *                                           modify memory directly.
+ *
+ * @mr: the #MemoryRegion to be initialized.
+ * @owner: the object that tracks the region's reference count
+ * @name: Region name, becomes part of RAMBlock name used in migration stream
+ *        must be unique within any device
+ * @size: size of the region.
+ * @share: allow remapping RAM to different addresses
+ * @errp: pointer to Error*, to store an error if it happens.
+ *
+ * Note that this function is similar to memory_region_init_ram_nomigrate.
+ * The only difference is part of the RAM region can be remapped.
+ */
+void memory_region_init_ram_shared_nomigrate(MemoryRegion *mr,
+                                             struct Object *owner,
+                                             const char *name,
+                                             uint64_t size,
+                                             bool share,
+                                             Error **errp);
+
+/**
  * memory_region_init_resizeable_ram:  Initialize memory region with resizeable
  *                                     RAM.  Accesses into the region will
  *                                     modify memory directly.  Only an initial
diff --git a/include/exec/ram_addr.h b/include/exec/ram_addr.h
index 7633ef6342..cf2446a176 100644
--- a/include/exec/ram_addr.h
+++ b/include/exec/ram_addr.h
@@ -80,7 +80,8 @@ RAMBlock *qemu_ram_alloc_from_fd(ram_addr_t size, MemoryRegion *mr,
                                  Error **errp);
 RAMBlock *qemu_ram_alloc_from_ptr(ram_addr_t size, void *host,
                                   MemoryRegion *mr, Error **errp);
-RAMBlock *qemu_ram_alloc(ram_addr_t size, MemoryRegion *mr, Error **errp);
+RAMBlock *qemu_ram_alloc(ram_addr_t size, bool share, MemoryRegion *mr,
+                         Error **errp);
 RAMBlock *qemu_ram_alloc_resizeable(ram_addr_t size, ram_addr_t max_size,
                                     void (*resized)(const char*,
                                                     uint64_t length,
diff --git a/include/qemu/osdep.h b/include/qemu/osdep.h
index adb3758275..41658060a7 100644
--- a/include/qemu/osdep.h
+++ b/include/qemu/osdep.h
@@ -255,7 +255,7 @@ extern int daemon(int, int);
 int qemu_daemon(int nochdir, int noclose);
 void *qemu_try_memalign(size_t alignment, size_t size);
 void *qemu_memalign(size_t alignment, size_t size);
-void *qemu_anon_ram_alloc(size_t size, uint64_t *align);
+void *qemu_anon_ram_alloc(size_t size, uint64_t *align, bool shared);
 void qemu_vfree(void *ptr);
 void qemu_anon_ram_free(void *ptr, size_t size);
 
diff --git a/include/sysemu/hostmem.h b/include/sysemu/hostmem.h
index ed6a437f4d..4d8f859f03 100644
--- a/include/sysemu/hostmem.h
+++ b/include/sysemu/hostmem.h
@@ -55,7 +55,7 @@ struct HostMemoryBackend {
     char *id;
     uint64_t size;
     bool merge, dump;
-    bool prealloc, force_prealloc, is_mapped;
+    bool prealloc, force_prealloc, is_mapped, share;
     DECLARE_BITMAP(host_nodes, MAX_NODES + 1);
     HostMemPolicy policy;
 
diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
index bbf12a1723..85002ac49a 100644
--- a/include/sysemu/kvm.h
+++ b/include/sysemu/kvm.h
@@ -248,7 +248,7 @@ int kvm_on_sigbus(int code, void *addr);
 
 /* interface with exec.c */
 
-void phys_mem_set_alloc(void *(*alloc)(size_t, uint64_t *align));
+void phys_mem_set_alloc(void *(*alloc)(size_t, uint64_t *align, bool shared));
 
 /* internal API */
 
diff --git a/memory.c b/memory.c
index 4b41fb837b..cb4fe1a55a 100644
--- a/memory.c
+++ b/memory.c
@@ -1538,11 +1538,21 @@ void memory_region_init_ram_nomigrate(MemoryRegion *mr,
                                       uint64_t size,
                                       Error **errp)
 {
+    memory_region_init_ram_shared_nomigrate(mr, owner, name, size, false, errp);
+}
+
+void memory_region_init_ram_shared_nomigrate(MemoryRegion *mr,
+                                             Object *owner,
+                                             const char *name,
+                                             uint64_t size,
+                                             bool share,
+                                             Error **errp)
+{
     memory_region_init(mr, owner, name, size);
     mr->ram = true;
     mr->terminates = true;
     mr->destructor = memory_region_destructor_ram;
-    mr->ram_block = qemu_ram_alloc(size, mr, errp);
+    mr->ram_block = qemu_ram_alloc(size, share, mr, errp);
     mr->dirty_log_mask = tcg_enabled() ? (1 << DIRTY_MEMORY_CODE) : 0;
 }
 
@@ -1651,7 +1661,7 @@ void memory_region_init_rom_nomigrate(MemoryRegion *mr,
     mr->readonly = true;
     mr->terminates = true;
     mr->destructor = memory_region_destructor_ram;
-    mr->ram_block = qemu_ram_alloc(size, mr, errp);
+    mr->ram_block = qemu_ram_alloc(size, false, mr, errp);
     mr->dirty_log_mask = tcg_enabled() ? (1 << DIRTY_MEMORY_CODE) : 0;
 }
 
@@ -1670,7 +1680,7 @@ void memory_region_init_rom_device_nomigrate(MemoryRegion *mr,
     mr->terminates = true;
     mr->rom_device = true;
     mr->destructor = memory_region_destructor_ram;
-    mr->ram_block = qemu_ram_alloc(size, mr, errp);
+    mr->ram_block = qemu_ram_alloc(size, false,  mr, errp);
 }
 
 void memory_region_init_iommu(void *_iommu_mr,
diff --git a/target/s390x/kvm.c b/target/s390x/kvm.c
index 6a18a413b4..73629ebb25 100644
--- a/target/s390x/kvm.c
+++ b/target/s390x/kvm.c
@@ -144,7 +144,7 @@ static int cap_gs;
 
 static int active_cmma;
 
-static void *legacy_s390_alloc(size_t size, uint64_t *align);
+static void *legacy_s390_alloc(size_t size, uint64_t *align, bool shared);
 
 static int kvm_s390_query_mem_limit(uint64_t *memory_limit)
 {
@@ -743,7 +743,7 @@ int kvm_s390_mem_op(S390CPU *cpu, vaddr addr, uint8_t ar, void *hostbuf,
  * to grow. We also have to use MAP parameters that avoid
  * read-only mapping of guest pages.
  */
-static void *legacy_s390_alloc(size_t size, uint64_t *align)
+static void *legacy_s390_alloc(size_t size, uint64_t *align, bool shared)
 {
     void *mem;
 
diff --git a/util/oslib-posix.c b/util/oslib-posix.c
index 77369c92ce..0cf3548778 100644
--- a/util/oslib-posix.c
+++ b/util/oslib-posix.c
@@ -127,10 +127,10 @@ void *qemu_memalign(size_t alignment, size_t size)
 }
 
 /* alloc shared memory pages */
-void *qemu_anon_ram_alloc(size_t size, uint64_t *alignment)
+void *qemu_anon_ram_alloc(size_t size, uint64_t *alignment, bool shared)
 {
     size_t align = QEMU_VMALLOC_ALIGN;
-    void *ptr = qemu_ram_mmap(-1, size, align, false);
+    void *ptr = qemu_ram_mmap(-1, size, align, shared);
 
     if (ptr == MAP_FAILED) {
         return NULL;
diff --git a/util/oslib-win32.c b/util/oslib-win32.c
index 69a6286d50..bb5ad28bd3 100644
--- a/util/oslib-win32.c
+++ b/util/oslib-win32.c
@@ -67,7 +67,7 @@ void *qemu_memalign(size_t alignment, size_t size)
     return qemu_oom_check(qemu_try_memalign(alignment, size));
 }
 
-void *qemu_anon_ram_alloc(size_t size, uint64_t *align)
+void *qemu_anon_ram_alloc(size_t size, uint64_t *align, bool shared)
 {
     void *ptr;
 
-- 
2.13.5

Re: [Qemu-devel] [PATCH V8 1/4] mem: add share parameter to memory-backend-ram

[Eduardo Habkost]

On Wed, Jan 17, 2018 at 11:54:18AM +0200, Marcel Apfelbaum wrote:
> Currently only file backed memory backend can
> be created with a "share" flag in order to allow
> sharing guest RAM with other processes in the host.
> 
> Add the "share" flag also to RAM Memory Backend
> in order to allow remapping parts of the guest RAM
> to different host virtual addresses. This is needed
> by the RDMA devices in order to remap non-contiguous
> QEMU virtual addresses to a contiguous virtual address range.
> 

Why do we need to make this configurable?  Would anything break
if MAP_SHARED was always used if possible?


> Moved the "share" flag to the Host Memory base class,
> modified phys_mem_alloc to include the new parameter
> and a new interface memory_region_init_ram_shared_nomigrate.
> 
> There are no functional changes if the new flag is not used.
> 
> Signed-off-by: Marcel Apfelbaum <marcel@redhat.com>
[...]

-- 
Eduardo

Re: [Qemu-devel] [PATCH V8 1/4] mem: add share parameter to memory-backend-ram

[Michael S. Tsirkin]

On Wed, Jan 31, 2018 at 06:40:59PM -0200, Eduardo Habkost wrote:
> On Wed, Jan 17, 2018 at 11:54:18AM +0200, Marcel Apfelbaum wrote:
> > Currently only file backed memory backend can
> > be created with a "share" flag in order to allow
> > sharing guest RAM with other processes in the host.
> > 
> > Add the "share" flag also to RAM Memory Backend
> > in order to allow remapping parts of the guest RAM
> > to different host virtual addresses. This is needed
> > by the RDMA devices in order to remap non-contiguous
> > QEMU virtual addresses to a contiguous virtual address range.
> > 
> 
> Why do we need to make this configurable?  Would anything break
> if MAP_SHARED was always used if possible?

See Documentation/vm/numa_memory_policy.txt for a list
of complications.

Maybe we should more of an effort to detect and report these
issues.

> 
> > Moved the "share" flag to the Host Memory base class,
> > modified phys_mem_alloc to include the new parameter
> > and a new interface memory_region_init_ram_shared_nomigrate.
> > 
> > There are no functional changes if the new flag is not used.
> > 
> > Signed-off-by: Marcel Apfelbaum <marcel@redhat.com>
> [...]
> 
> -- 
> Eduardo

Re: [Qemu-devel] [PATCH V8 1/4] mem: add share parameter to memory-backend-ram

[Eduardo Habkost]

On Wed, Jan 31, 2018 at 11:10:07PM +0200, Michael S. Tsirkin wrote:
> On Wed, Jan 31, 2018 at 06:40:59PM -0200, Eduardo Habkost wrote:
> > On Wed, Jan 17, 2018 at 11:54:18AM +0200, Marcel Apfelbaum wrote:
> > > Currently only file backed memory backend can
> > > be created with a "share" flag in order to allow
> > > sharing guest RAM with other processes in the host.
> > > 
> > > Add the "share" flag also to RAM Memory Backend
> > > in order to allow remapping parts of the guest RAM
> > > to different host virtual addresses. This is needed
> > > by the RDMA devices in order to remap non-contiguous
> > > QEMU virtual addresses to a contiguous virtual address range.
> > > 
> > 
> > Why do we need to make this configurable?  Would anything break
> > if MAP_SHARED was always used if possible?
> 
> See Documentation/vm/numa_memory_policy.txt for a list
> of complications.

Ew.

> 
> Maybe we should more of an effort to detect and report these
> issues.

Probably.  Having other features breaking silently when using
pvrdma doesn't sound good.  We must at least document those
problems in the documentation for memory-backend-ram.

BTW, what's the root cause for requiring HVAs in the buffer?  Can
this be fixed?

-- 
Eduardo

Re: [Qemu-devel] [PATCH V8 1/4] mem: add share parameter to memory-backend-ram

[Michael S. Tsirkin]

On Wed, Jan 31, 2018 at 09:34:22PM -0200, Eduardo Habkost wrote:
> On Wed, Jan 31, 2018 at 11:10:07PM +0200, Michael S. Tsirkin wrote:
> > On Wed, Jan 31, 2018 at 06:40:59PM -0200, Eduardo Habkost wrote:
> > > On Wed, Jan 17, 2018 at 11:54:18AM +0200, Marcel Apfelbaum wrote:
> > > > Currently only file backed memory backend can
> > > > be created with a "share" flag in order to allow
> > > > sharing guest RAM with other processes in the host.
> > > > 
> > > > Add the "share" flag also to RAM Memory Backend
> > > > in order to allow remapping parts of the guest RAM
> > > > to different host virtual addresses. This is needed
> > > > by the RDMA devices in order to remap non-contiguous
> > > > QEMU virtual addresses to a contiguous virtual address range.
> > > > 
> > > 
> > > Why do we need to make this configurable?  Would anything break
> > > if MAP_SHARED was always used if possible?
> > 
> > See Documentation/vm/numa_memory_policy.txt for a list
> > of complications.
> 
> Ew.
> 
> > 
> > Maybe we should more of an effort to detect and report these
> > issues.
> 
> Probably.  Having other features breaking silently when using
> pvrdma doesn't sound good.  We must at least document those
> problems in the documentation for memory-backend-ram.
> 
> BTW, what's the root cause for requiring HVAs in the buffer?

It's a side effect of the kernel/userspace API which always wants
a single HVA/len pair to map memory for the application.


>  Can
> this be fixed?

I think yes.  It'd need to be a kernel patch for the RDMA subsystem
mapping an s/g list with actual memory. The HVA/len pair would then just
be used to refer to the region, without creating the two mappings.

Something like splitting the register mr into

mr = create mr (va/len) - allocate a handle and record the va/len

addmemory(mr, offset, hva, len) - pin memory

register mr - pass it to HW

As a nice side effect we won't burn so much virtual address space.

This will fix rdma with hugetlbfs as well which is currently broken.


> -- 
> Eduardo

Re: [Qemu-devel] [PATCH V8 1/4] mem: add share parameter to memory-backend-ram

[Marcel Apfelbaum]

On 01/02/2018 4:22, Michael S. Tsirkin wrote:
> On Wed, Jan 31, 2018 at 09:34:22PM -0200, Eduardo Habkost wrote:
>> On Wed, Jan 31, 2018 at 11:10:07PM +0200, Michael S. Tsirkin wrote:
>>> On Wed, Jan 31, 2018 at 06:40:59PM -0200, Eduardo Habkost wrote:
>>>> On Wed, Jan 17, 2018 at 11:54:18AM +0200, Marcel Apfelbaum wrote:
>>>>> Currently only file backed memory backend can
>>>>> be created with a "share" flag in order to allow
>>>>> sharing guest RAM with other processes in the host.
>>>>>
>>>>> Add the "share" flag also to RAM Memory Backend
>>>>> in order to allow remapping parts of the guest RAM
>>>>> to different host virtual addresses. This is needed
>>>>> by the RDMA devices in order to remap non-contiguous
>>>>> QEMU virtual addresses to a contiguous virtual address range.
>>>>>
>>>>
>>>> Why do we need to make this configurable?  Would anything break
>>>> if MAP_SHARED was always used if possible?
>>>
>>> See Documentation/vm/numa_memory_policy.txt for a list
>>> of complications.
>>
>> Ew.
>>
>>>
>>> Maybe we should more of an effort to detect and report these
>>> issues.
>>
>> Probably.  Having other features breaking silently when using
>> pvrdma doesn't sound good.  We must at least document those
>> problems in the documentation for memory-backend-ram.
>>
>> BTW, what's the root cause for requiring HVAs in the buffer?
> 
> It's a side effect of the kernel/userspace API which always wants
> a single HVA/len pair to map memory for the application.
> 
> 

Hi Eduardo and Michael,

>>  Can
>> this be fixed?
> 
> I think yes.  It'd need to be a kernel patch for the RDMA subsystem
> mapping an s/g list with actual memory. The HVA/len pair would then just
> be used to refer to the region, without creating the two mappings.
> 
> Something like splitting the register mr into
> 
> mr = create mr (va/len) - allocate a handle and record the va/len
> 
> addmemory(mr, offset, hva, len) - pin memory
> 
> register mr - pass it to HW
> 
> As a nice side effect we won't burn so much virtual address space.
>

We would still need a contiguous virtual address space range (for post-send)
which we don't have since guest contiguous virtual address space
will always end up as non-contiguous host virtual address space.

I am not sure the RDMA HW can handle a large VA with holes.

An alternative would be 0-based MR, QEMU intercepts the post-send
operations and can substract the guest VA base address.
However I didn't see the implementation in kernel for 0 based MRs
and also the RDMA maintainer said it would work for local keys
and not for remote keys.

> This will fix rdma with hugetlbfs as well which is currently broken.
> 
> 

There is already a discussion on the linux-rdma list:
    https://www.spinics.net/lists/linux-rdma/msg60079.html
But it will take some (actually a lot of) time, we are currently talking about
a possible API. And it does not solve the re-mapping...

Thanks,
Marcel

>> -- 
>> Eduardo

Re: [Qemu-devel] [PATCH V8 1/4] mem: add share parameter to memory-backend-ram

[Eduardo Habkost]

On Thu, Feb 01, 2018 at 07:36:50AM +0200, Marcel Apfelbaum wrote:
> On 01/02/2018 4:22, Michael S. Tsirkin wrote:
> > On Wed, Jan 31, 2018 at 09:34:22PM -0200, Eduardo Habkost wrote:
[...]
> >> BTW, what's the root cause for requiring HVAs in the buffer?
> > 
> > It's a side effect of the kernel/userspace API which always wants
> > a single HVA/len pair to map memory for the application.
> > 
> > 
> 
> Hi Eduardo and Michael,
> 
> >>  Can
> >> this be fixed?
> > 
> > I think yes.  It'd need to be a kernel patch for the RDMA subsystem
> > mapping an s/g list with actual memory. The HVA/len pair would then just
> > be used to refer to the region, without creating the two mappings.
> > 
> > Something like splitting the register mr into
> > 
> > mr = create mr (va/len) - allocate a handle and record the va/len
> > 
> > addmemory(mr, offset, hva, len) - pin memory
> > 
> > register mr - pass it to HW
> > 
> > As a nice side effect we won't burn so much virtual address space.
> >
> 
> We would still need a contiguous virtual address space range (for post-send)
> which we don't have since guest contiguous virtual address space
> will always end up as non-contiguous host virtual address space.
> 
> I am not sure the RDMA HW can handle a large VA with holes.

I'm confused.  Why would the hardware see and care about virtual
addresses?  How exactly does the hardware translates VAs to
PAs?  What if the process page tables change?

> 
> An alternative would be 0-based MR, QEMU intercepts the post-send
> operations and can substract the guest VA base address.
> However I didn't see the implementation in kernel for 0 based MRs
> and also the RDMA maintainer said it would work for local keys
> and not for remote keys.

This is also unexpected: are GVAs visible to the virtual RDMA
hardware?  Where does the QEMU pvrdma code translates GVAs to
GPAs?

> 
> > This will fix rdma with hugetlbfs as well which is currently broken.
> > 
> > 
> 
> There is already a discussion on the linux-rdma list:
>     https://www.spinics.net/lists/linux-rdma/msg60079.html
> But it will take some (actually a lot of) time, we are currently talking about
> a possible API. And it does not solve the re-mapping...
> 
> Thanks,
> Marcel
> 
> >> -- 
> >> Eduardo
> 

-- 
Eduardo

Re: [Qemu-devel] [PATCH V8 1/4] mem: add share parameter to memory-backend-ram

[Marcel Apfelbaum]

On 01/02/2018 14:10, Eduardo Habkost wrote:
> On Thu, Feb 01, 2018 at 07:36:50AM +0200, Marcel Apfelbaum wrote:
>> On 01/02/2018 4:22, Michael S. Tsirkin wrote:
>>> On Wed, Jan 31, 2018 at 09:34:22PM -0200, Eduardo Habkost wrote:
> [...]
>>>> BTW, what's the root cause for requiring HVAs in the buffer?
>>>
>>> It's a side effect of the kernel/userspace API which always wants
>>> a single HVA/len pair to map memory for the application.
>>>
>>>
>>
>> Hi Eduardo and Michael,
>>
>>>>  Can
>>>> this be fixed?
>>>
>>> I think yes.  It'd need to be a kernel patch for the RDMA subsystem
>>> mapping an s/g list with actual memory. The HVA/len pair would then just
>>> be used to refer to the region, without creating the two mappings.
>>>
>>> Something like splitting the register mr into
>>>
>>> mr = create mr (va/len) - allocate a handle and record the va/len
>>>
>>> addmemory(mr, offset, hva, len) - pin memory
>>>
>>> register mr - pass it to HW
>>>
>>> As a nice side effect we won't burn so much virtual address space.
>>>
>>
>> We would still need a contiguous virtual address space range (for post-send)
>> which we don't have since guest contiguous virtual address space
>> will always end up as non-contiguous host virtual address space.
>>
>> I am not sure the RDMA HW can handle a large VA with holes.
> 
> I'm confused.  Why would the hardware see and care about virtual
> addresses? 

The post-send operations bypasses the kernel, and the process
puts in the work request GVA addresses.

> How exactly does the hardware translates VAs to
> PAs? 

The HW maintains a page-directory like structure different form MMU
VA -> phys pages

> What if the process page tables change?
> 

Since the page tables the HW uses are their own, we just need the phys
page to be pinned.

>>
>> An alternative would be 0-based MR, QEMU intercepts the post-send
>> operations and can substract the guest VA base address.
>> However I didn't see the implementation in kernel for 0 based MRs
>> and also the RDMA maintainer said it would work for local keys
>> and not for remote keys.
> 
> This is also unexpected: are GVAs visible to the virtual RDMA
> hardware? 

Yes, explained above

> Where does the QEMU pvrdma code translates GVAs to
> GPAs?
> 

During reg_mr (memory registration commands)
Then it registers the same addresses to the real HW.
(as Host virtual addresses)

Thanks,
Marcel

>>
>>> This will fix rdma with hugetlbfs as well which is currently broken.
>>>
>>>
>>
>> There is already a discussion on the linux-rdma list:
>>     https://www.spinics.net/lists/linux-rdma/msg60079.html
>> But it will take some (actually a lot of) time, we are currently talking about
>> a possible API. And it does not solve the re-mapping...
>>
>> Thanks,
>> Marcel
>>
>>>> -- 
>>>> Eduardo
>>
> 

Re: [Qemu-devel] [PATCH V8 1/4] mem: add share parameter to memory-backend-ram

[Eduardo Habkost]

On Thu, Feb 01, 2018 at 02:29:25PM +0200, Marcel Apfelbaum wrote:
> On 01/02/2018 14:10, Eduardo Habkost wrote:
> > On Thu, Feb 01, 2018 at 07:36:50AM +0200, Marcel Apfelbaum wrote:
> >> On 01/02/2018 4:22, Michael S. Tsirkin wrote:
> >>> On Wed, Jan 31, 2018 at 09:34:22PM -0200, Eduardo Habkost wrote:
> > [...]
> >>>> BTW, what's the root cause for requiring HVAs in the buffer?
> >>>
> >>> It's a side effect of the kernel/userspace API which always wants
> >>> a single HVA/len pair to map memory for the application.
> >>>
> >>>
> >>
> >> Hi Eduardo and Michael,
> >>
> >>>>  Can
> >>>> this be fixed?
> >>>
> >>> I think yes.  It'd need to be a kernel patch for the RDMA subsystem
> >>> mapping an s/g list with actual memory. The HVA/len pair would then just
> >>> be used to refer to the region, without creating the two mappings.
> >>>
> >>> Something like splitting the register mr into
> >>>
> >>> mr = create mr (va/len) - allocate a handle and record the va/len
> >>>
> >>> addmemory(mr, offset, hva, len) - pin memory
> >>>
> >>> register mr - pass it to HW
> >>>
> >>> As a nice side effect we won't burn so much virtual address space.
> >>>
> >>
> >> We would still need a contiguous virtual address space range (for post-send)
> >> which we don't have since guest contiguous virtual address space
> >> will always end up as non-contiguous host virtual address space.
> >>
> >> I am not sure the RDMA HW can handle a large VA with holes.
> > 
> > I'm confused.  Why would the hardware see and care about virtual
> > addresses? 
> 
> The post-send operations bypasses the kernel, and the process
> puts in the work request GVA addresses.
> 
> > How exactly does the hardware translates VAs to
> > PAs? 
> 
> The HW maintains a page-directory like structure different form MMU
> VA -> phys pages
> 
> > What if the process page tables change?
> > 
> 
> Since the page tables the HW uses are their own, we just need the phys
> page to be pinned.

So there's no hardware-imposed requirement that the hardware VAs
(mapped by the HW page directory) match the VAs in QEMU
address-space, right?  If the RDMA API is updated to remove this
requirement, couldn't you just use the untranslated guest VAs
directly?

-- 
Eduardo

Re: [Qemu-devel] [PATCH V8 1/4] mem: add share parameter to memory-backend-ram

[Marcel Apfelbaum]

On 01/02/2018 15:53, Eduardo Habkost wrote:
> On Thu, Feb 01, 2018 at 02:29:25PM +0200, Marcel Apfelbaum wrote:
>> On 01/02/2018 14:10, Eduardo Habkost wrote:
>>> On Thu, Feb 01, 2018 at 07:36:50AM +0200, Marcel Apfelbaum wrote:
>>>> On 01/02/2018 4:22, Michael S. Tsirkin wrote:
>>>>> On Wed, Jan 31, 2018 at 09:34:22PM -0200, Eduardo Habkost wrote:
>>> [...]
>>>>>> BTW, what's the root cause for requiring HVAs in the buffer?
>>>>>
>>>>> It's a side effect of the kernel/userspace API which always wants
>>>>> a single HVA/len pair to map memory for the application.
>>>>>
>>>>>
>>>>
>>>> Hi Eduardo and Michael,
>>>>
>>>>>>  Can
>>>>>> this be fixed?
>>>>>
>>>>> I think yes.  It'd need to be a kernel patch for the RDMA subsystem
>>>>> mapping an s/g list with actual memory. The HVA/len pair would then just
>>>>> be used to refer to the region, without creating the two mappings.
>>>>>
>>>>> Something like splitting the register mr into
>>>>>
>>>>> mr = create mr (va/len) - allocate a handle and record the va/len
>>>>>
>>>>> addmemory(mr, offset, hva, len) - pin memory
>>>>>
>>>>> register mr - pass it to HW
>>>>>
>>>>> As a nice side effect we won't burn so much virtual address space.
>>>>>
>>>>
>>>> We would still need a contiguous virtual address space range (for post-send)
>>>> which we don't have since guest contiguous virtual address space
>>>> will always end up as non-contiguous host virtual address space.
>>>>
>>>> I am not sure the RDMA HW can handle a large VA with holes.
>>>
>>> I'm confused.  Why would the hardware see and care about virtual
>>> addresses? 
>>
>> The post-send operations bypasses the kernel, and the process
>> puts in the work request GVA addresses.
>>
>>> How exactly does the hardware translates VAs to
>>> PAs? 
>>
>> The HW maintains a page-directory like structure different form MMU
>> VA -> phys pages
>>
>>> What if the process page tables change?
>>>
>>
>> Since the page tables the HW uses are their own, we just need the phys
>> page to be pinned.
> 
> So there's no hardware-imposed requirement that the hardware VAs
> (mapped by the HW page directory) match the VAs in QEMU
> address-space, right? 

Actually there is. Today it works exactly as you described.

> If the RDMA API is updated to remove this
> requirement, couldn't you just use the untranslated guest VAs
> directly?
> 

When the RDMA API will be decided we will still need to pass
some kind of VA to the HW, but if we succeed to remove the
phys page registration from the VA, yes, it will be possible.


Thanks,
Marcel

Re: [Qemu-devel] [PATCH V8 1/4] mem: add share parameter to memory-backend-ram

[Eduardo Habkost]

On Thu, Feb 01, 2018 at 08:03:53PM +0200, Marcel Apfelbaum wrote:
> On 01/02/2018 15:53, Eduardo Habkost wrote:
> > On Thu, Feb 01, 2018 at 02:29:25PM +0200, Marcel Apfelbaum wrote:
> >> On 01/02/2018 14:10, Eduardo Habkost wrote:
> >>> On Thu, Feb 01, 2018 at 07:36:50AM +0200, Marcel Apfelbaum wrote:
> >>>> On 01/02/2018 4:22, Michael S. Tsirkin wrote:
> >>>>> On Wed, Jan 31, 2018 at 09:34:22PM -0200, Eduardo Habkost wrote:
> >>> [...]
> >>>>>> BTW, what's the root cause for requiring HVAs in the buffer?
> >>>>>
> >>>>> It's a side effect of the kernel/userspace API which always wants
> >>>>> a single HVA/len pair to map memory for the application.
> >>>>>
> >>>>>
> >>>>
> >>>> Hi Eduardo and Michael,
> >>>>
> >>>>>>  Can
> >>>>>> this be fixed?
> >>>>>
> >>>>> I think yes.  It'd need to be a kernel patch for the RDMA subsystem
> >>>>> mapping an s/g list with actual memory. The HVA/len pair would then just
> >>>>> be used to refer to the region, without creating the two mappings.
> >>>>>
> >>>>> Something like splitting the register mr into
> >>>>>
> >>>>> mr = create mr (va/len) - allocate a handle and record the va/len
> >>>>>
> >>>>> addmemory(mr, offset, hva, len) - pin memory
> >>>>>
> >>>>> register mr - pass it to HW
> >>>>>
> >>>>> As a nice side effect we won't burn so much virtual address space.
> >>>>>
> >>>>
> >>>> We would still need a contiguous virtual address space range (for post-send)
> >>>> which we don't have since guest contiguous virtual address space
> >>>> will always end up as non-contiguous host virtual address space.
> >>>>
> >>>> I am not sure the RDMA HW can handle a large VA with holes.
> >>>
> >>> I'm confused.  Why would the hardware see and care about virtual
> >>> addresses? 
> >>
> >> The post-send operations bypasses the kernel, and the process
> >> puts in the work request GVA addresses.
> >>
> >>> How exactly does the hardware translates VAs to
> >>> PAs? 
> >>
> >> The HW maintains a page-directory like structure different form MMU
> >> VA -> phys pages
> >>
> >>> What if the process page tables change?
> >>>
> >>
> >> Since the page tables the HW uses are their own, we just need the phys
> >> page to be pinned.
> > 
> > So there's no hardware-imposed requirement that the hardware VAs
> > (mapped by the HW page directory) match the VAs in QEMU
> > address-space, right? 
> 
> Actually there is. Today it works exactly as you described.

Are you sure there's such hardware-imposed requirement?

Why would the hardware require VAs to match the ones in the
userspace address-space, if it doesn't use the CPU MMU at all?

-- 
Eduardo

Re: [Qemu-devel] [PATCH V8 1/4] mem: add share parameter to memory-backend-ram

[Marcel Apfelbaum]

On 01/02/2018 20:21, Eduardo Habkost wrote:
> On Thu, Feb 01, 2018 at 08:03:53PM +0200, Marcel Apfelbaum wrote:
>> On 01/02/2018 15:53, Eduardo Habkost wrote:
>>> On Thu, Feb 01, 2018 at 02:29:25PM +0200, Marcel Apfelbaum wrote:
>>>> On 01/02/2018 14:10, Eduardo Habkost wrote:
>>>>> On Thu, Feb 01, 2018 at 07:36:50AM +0200, Marcel Apfelbaum wrote:
>>>>>> On 01/02/2018 4:22, Michael S. Tsirkin wrote:
>>>>>>> On Wed, Jan 31, 2018 at 09:34:22PM -0200, Eduardo Habkost wrote:
>>>>> [...]
>>>>>>>> BTW, what's the root cause for requiring HVAs in the buffer?
>>>>>>>
>>>>>>> It's a side effect of the kernel/userspace API which always wants
>>>>>>> a single HVA/len pair to map memory for the application.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> Hi Eduardo and Michael,
>>>>>>
>>>>>>>>  Can
>>>>>>>> this be fixed?
>>>>>>>
>>>>>>> I think yes.  It'd need to be a kernel patch for the RDMA subsystem
>>>>>>> mapping an s/g list with actual memory. The HVA/len pair would then just
>>>>>>> be used to refer to the region, without creating the two mappings.
>>>>>>>
>>>>>>> Something like splitting the register mr into
>>>>>>>
>>>>>>> mr = create mr (va/len) - allocate a handle and record the va/len
>>>>>>>
>>>>>>> addmemory(mr, offset, hva, len) - pin memory
>>>>>>>
>>>>>>> register mr - pass it to HW
>>>>>>>
>>>>>>> As a nice side effect we won't burn so much virtual address space.
>>>>>>>
>>>>>>
>>>>>> We would still need a contiguous virtual address space range (for post-send)
>>>>>> which we don't have since guest contiguous virtual address space
>>>>>> will always end up as non-contiguous host virtual address space.
>>>>>>
>>>>>> I am not sure the RDMA HW can handle a large VA with holes.
>>>>>
>>>>> I'm confused.  Why would the hardware see and care about virtual
>>>>> addresses? 
>>>>
>>>> The post-send operations bypasses the kernel, and the process
>>>> puts in the work request GVA addresses.
>>>>
>>>>> How exactly does the hardware translates VAs to
>>>>> PAs? 
>>>>
>>>> The HW maintains a page-directory like structure different form MMU
>>>> VA -> phys pages
>>>>
>>>>> What if the process page tables change?
>>>>>
>>>>
>>>> Since the page tables the HW uses are their own, we just need the phys
>>>> page to be pinned.
>>>
>>> So there's no hardware-imposed requirement that the hardware VAs
>>> (mapped by the HW page directory) match the VAs in QEMU
>>> address-space, right? 
>>
>> Actually there is. Today it works exactly as you described.
> 
> Are you sure there's such hardware-imposed requirement?
> 

Yes.

> Why would the hardware require VAs to match the ones in the
> userspace address-space, if it doesn't use the CPU MMU at all?
> 

It works like that:

1. We register a buffer from the process address space
   giving its base address and length.
   This call goes to kernel which in turn pins the phys pages
   and registers them with the device *together* with the base
   address (virtual address!)
2. The device builds its own page tables to be able to translate
   the virtual addresses to actual phys pages.
3. The process executes post-send requests directly to hw by-passing
   the kernel giving process virtual addresses in work requests.
4. The device uses its own page tables to translate the virtual
   addresses to phys pages and sending them.

Theoretically is possible to send any contiguous IOVA instead of
process's one but is not how is working today.

Makes sense?

Thanks,
Marcel

Re: [Qemu-devel] [PATCH V8 1/4] mem: add share parameter to memory-backend-ram

[Eduardo Habkost]

On Thu, Feb 01, 2018 at 08:31:09PM +0200, Marcel Apfelbaum wrote:
> On 01/02/2018 20:21, Eduardo Habkost wrote:
> > On Thu, Feb 01, 2018 at 08:03:53PM +0200, Marcel Apfelbaum wrote:
> >> On 01/02/2018 15:53, Eduardo Habkost wrote:
> >>> On Thu, Feb 01, 2018 at 02:29:25PM +0200, Marcel Apfelbaum wrote:
> >>>> On 01/02/2018 14:10, Eduardo Habkost wrote:
> >>>>> On Thu, Feb 01, 2018 at 07:36:50AM +0200, Marcel Apfelbaum wrote:
> >>>>>> On 01/02/2018 4:22, Michael S. Tsirkin wrote:
> >>>>>>> On Wed, Jan 31, 2018 at 09:34:22PM -0200, Eduardo Habkost wrote:
> >>>>> [...]
> >>>>>>>> BTW, what's the root cause for requiring HVAs in the buffer?
> >>>>>>>
> >>>>>>> It's a side effect of the kernel/userspace API which always wants
> >>>>>>> a single HVA/len pair to map memory for the application.
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>> Hi Eduardo and Michael,
> >>>>>>
> >>>>>>>>  Can
> >>>>>>>> this be fixed?
> >>>>>>>
> >>>>>>> I think yes.  It'd need to be a kernel patch for the RDMA subsystem
> >>>>>>> mapping an s/g list with actual memory. The HVA/len pair would then just
> >>>>>>> be used to refer to the region, without creating the two mappings.
> >>>>>>>
> >>>>>>> Something like splitting the register mr into
> >>>>>>>
> >>>>>>> mr = create mr (va/len) - allocate a handle and record the va/len
> >>>>>>>
> >>>>>>> addmemory(mr, offset, hva, len) - pin memory
> >>>>>>>
> >>>>>>> register mr - pass it to HW
> >>>>>>>
> >>>>>>> As a nice side effect we won't burn so much virtual address space.
> >>>>>>>
> >>>>>>
> >>>>>> We would still need a contiguous virtual address space range (for post-send)
> >>>>>> which we don't have since guest contiguous virtual address space
> >>>>>> will always end up as non-contiguous host virtual address space.
> >>>>>>
> >>>>>> I am not sure the RDMA HW can handle a large VA with holes.
> >>>>>
> >>>>> I'm confused.  Why would the hardware see and care about virtual
> >>>>> addresses? 
> >>>>
> >>>> The post-send operations bypasses the kernel, and the process
> >>>> puts in the work request GVA addresses.
> >>>>
> >>>>> How exactly does the hardware translates VAs to
> >>>>> PAs? 
> >>>>
> >>>> The HW maintains a page-directory like structure different form MMU
> >>>> VA -> phys pages
> >>>>
> >>>>> What if the process page tables change?
> >>>>>
> >>>>
> >>>> Since the page tables the HW uses are their own, we just need the phys
> >>>> page to be pinned.
> >>>
> >>> So there's no hardware-imposed requirement that the hardware VAs
> >>> (mapped by the HW page directory) match the VAs in QEMU
> >>> address-space, right? 
> >>
> >> Actually there is. Today it works exactly as you described.
> > 
> > Are you sure there's such hardware-imposed requirement?
> > 
> 
> Yes.
> 
> > Why would the hardware require VAs to match the ones in the
> > userspace address-space, if it doesn't use the CPU MMU at all?
> > 
> 
> It works like that:
> 
> 1. We register a buffer from the process address space
>    giving its base address and length.
>    This call goes to kernel which in turn pins the phys pages
>    and registers them with the device *together* with the base
>    address (virtual address!)
> 2. The device builds its own page tables to be able to translate
>    the virtual addresses to actual phys pages.

How would the device be able to do that?  It would require the
device to look at the process page tables, wouldn't it?  Isn't
the HW IOVA->PA translation table built by the OS?


> 3. The process executes post-send requests directly to hw by-passing
>    the kernel giving process virtual addresses in work requests.
> 4. The device uses its own page tables to translate the virtual
>    addresses to phys pages and sending them.
> 
> Theoretically is possible to send any contiguous IOVA instead of
> process's one but is not how is working today.
> 
> Makes sense?

-- 
Eduardo

Re: [Qemu-devel] [PATCH V8 1/4] mem: add share parameter to memory-backend-ram

[Marcel Apfelbaum]

On 01/02/2018 20:51, Eduardo Habkost wrote:
> On Thu, Feb 01, 2018 at 08:31:09PM +0200, Marcel Apfelbaum wrote:
>> On 01/02/2018 20:21, Eduardo Habkost wrote:
>>> On Thu, Feb 01, 2018 at 08:03:53PM +0200, Marcel Apfelbaum wrote:
>>>> On 01/02/2018 15:53, Eduardo Habkost wrote:
>>>>> On Thu, Feb 01, 2018 at 02:29:25PM +0200, Marcel Apfelbaum wrote:
>>>>>> On 01/02/2018 14:10, Eduardo Habkost wrote:
>>>>>>> On Thu, Feb 01, 2018 at 07:36:50AM +0200, Marcel Apfelbaum wrote:
>>>>>>>> On 01/02/2018 4:22, Michael S. Tsirkin wrote:
>>>>>>>>> On Wed, Jan 31, 2018 at 09:34:22PM -0200, Eduardo Habkost wrote:
>>>>>>> [...]
>>>>>>>>>> BTW, what's the root cause for requiring HVAs in the buffer?
>>>>>>>>>
>>>>>>>>> It's a side effect of the kernel/userspace API which always wants
>>>>>>>>> a single HVA/len pair to map memory for the application.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> Hi Eduardo and Michael,
>>>>>>>>
>>>>>>>>>>  Can
>>>>>>>>>> this be fixed?
>>>>>>>>>
>>>>>>>>> I think yes.  It'd need to be a kernel patch for the RDMA subsystem
>>>>>>>>> mapping an s/g list with actual memory. The HVA/len pair would then just
>>>>>>>>> be used to refer to the region, without creating the two mappings.
>>>>>>>>>
>>>>>>>>> Something like splitting the register mr into
>>>>>>>>>
>>>>>>>>> mr = create mr (va/len) - allocate a handle and record the va/len
>>>>>>>>>
>>>>>>>>> addmemory(mr, offset, hva, len) - pin memory
>>>>>>>>>
>>>>>>>>> register mr - pass it to HW
>>>>>>>>>
>>>>>>>>> As a nice side effect we won't burn so much virtual address space.
>>>>>>>>>
>>>>>>>>
>>>>>>>> We would still need a contiguous virtual address space range (for post-send)
>>>>>>>> which we don't have since guest contiguous virtual address space
>>>>>>>> will always end up as non-contiguous host virtual address space.
>>>>>>>>
>>>>>>>> I am not sure the RDMA HW can handle a large VA with holes.
>>>>>>>
>>>>>>> I'm confused.  Why would the hardware see and care about virtual
>>>>>>> addresses? 
>>>>>>
>>>>>> The post-send operations bypasses the kernel, and the process
>>>>>> puts in the work request GVA addresses.
>>>>>>
>>>>>>> How exactly does the hardware translates VAs to
>>>>>>> PAs? 
>>>>>>
>>>>>> The HW maintains a page-directory like structure different form MMU
>>>>>> VA -> phys pages
>>>>>>
>>>>>>> What if the process page tables change?
>>>>>>>
>>>>>>
>>>>>> Since the page tables the HW uses are their own, we just need the phys
>>>>>> page to be pinned.
>>>>>
>>>>> So there's no hardware-imposed requirement that the hardware VAs
>>>>> (mapped by the HW page directory) match the VAs in QEMU
>>>>> address-space, right? 
>>>>
>>>> Actually there is. Today it works exactly as you described.
>>>
>>> Are you sure there's such hardware-imposed requirement?
>>>
>>
>> Yes.
>>
>>> Why would the hardware require VAs to match the ones in the
>>> userspace address-space, if it doesn't use the CPU MMU at all?
>>>
>>
>> It works like that:
>>
>> 1. We register a buffer from the process address space
>>    giving its base address and length.
>>    This call goes to kernel which in turn pins the phys pages
>>    and registers them with the device *together* with the base
>>    address (virtual address!)
>> 2. The device builds its own page tables to be able to translate
>>    the virtual addresses to actual phys pages.
> 
> How would the device be able to do that?  It would require the
> device to look at the process page tables, wouldn't it?  Isn't
> the HW IOVA->PA translation table built by the OS?
> 

As stated above, these are tables private for the device.
(They even have a hw vendor specific layout I think,
 since the device holds some cache)

The device looks at its own private page tables, and not
to the OS ones.

> 
>> 3. The process executes post-send requests directly to hw by-passing
>>    the kernel giving process virtual addresses in work requests.
>> 4. The device uses its own page tables to translate the virtual
>>    addresses to phys pages and sending them.
>>
>> Theoretically is possible to send any contiguous IOVA instead of
>> process's one but is not how is working today.
>>
>> Makes sense?
> 

Re: [Qemu-devel] [PATCH V8 1/4] mem: add share parameter to memory-backend-ram

[Eduardo Habkost]

On Thu, Feb 01, 2018 at 08:58:32PM +0200, Marcel Apfelbaum wrote:
> On 01/02/2018 20:51, Eduardo Habkost wrote:
> > On Thu, Feb 01, 2018 at 08:31:09PM +0200, Marcel Apfelbaum wrote:
> >> On 01/02/2018 20:21, Eduardo Habkost wrote:
> >>> On Thu, Feb 01, 2018 at 08:03:53PM +0200, Marcel Apfelbaum wrote:
> >>>> On 01/02/2018 15:53, Eduardo Habkost wrote:
> >>>>> On Thu, Feb 01, 2018 at 02:29:25PM +0200, Marcel Apfelbaum wrote:
> >>>>>> On 01/02/2018 14:10, Eduardo Habkost wrote:
> >>>>>>> On Thu, Feb 01, 2018 at 07:36:50AM +0200, Marcel Apfelbaum wrote:
> >>>>>>>> On 01/02/2018 4:22, Michael S. Tsirkin wrote:
> >>>>>>>>> On Wed, Jan 31, 2018 at 09:34:22PM -0200, Eduardo Habkost wrote:
> >>>>>>> [...]
> >>>>>>>>>> BTW, what's the root cause for requiring HVAs in the buffer?
> >>>>>>>>>
> >>>>>>>>> It's a side effect of the kernel/userspace API which always wants
> >>>>>>>>> a single HVA/len pair to map memory for the application.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> Hi Eduardo and Michael,
> >>>>>>>>
> >>>>>>>>>>  Can
> >>>>>>>>>> this be fixed?
> >>>>>>>>>
> >>>>>>>>> I think yes.  It'd need to be a kernel patch for the RDMA subsystem
> >>>>>>>>> mapping an s/g list with actual memory. The HVA/len pair would then just
> >>>>>>>>> be used to refer to the region, without creating the two mappings.
> >>>>>>>>>
> >>>>>>>>> Something like splitting the register mr into
> >>>>>>>>>
> >>>>>>>>> mr = create mr (va/len) - allocate a handle and record the va/len
> >>>>>>>>>
> >>>>>>>>> addmemory(mr, offset, hva, len) - pin memory
> >>>>>>>>>
> >>>>>>>>> register mr - pass it to HW
> >>>>>>>>>
> >>>>>>>>> As a nice side effect we won't burn so much virtual address space.
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> We would still need a contiguous virtual address space range (for post-send)
> >>>>>>>> which we don't have since guest contiguous virtual address space
> >>>>>>>> will always end up as non-contiguous host virtual address space.
> >>>>>>>>
> >>>>>>>> I am not sure the RDMA HW can handle a large VA with holes.
> >>>>>>>
> >>>>>>> I'm confused.  Why would the hardware see and care about virtual
> >>>>>>> addresses? 
> >>>>>>
> >>>>>> The post-send operations bypasses the kernel, and the process
> >>>>>> puts in the work request GVA addresses.
> >>>>>>
> >>>>>>> How exactly does the hardware translates VAs to
> >>>>>>> PAs? 
> >>>>>>
> >>>>>> The HW maintains a page-directory like structure different form MMU
> >>>>>> VA -> phys pages
> >>>>>>
> >>>>>>> What if the process page tables change?
> >>>>>>>
> >>>>>>
> >>>>>> Since the page tables the HW uses are their own, we just need the phys
> >>>>>> page to be pinned.
> >>>>>
> >>>>> So there's no hardware-imposed requirement that the hardware VAs
> >>>>> (mapped by the HW page directory) match the VAs in QEMU
> >>>>> address-space, right? 
> >>>>
> >>>> Actually there is. Today it works exactly as you described.
> >>>
> >>> Are you sure there's such hardware-imposed requirement?
> >>>
> >>
> >> Yes.
> >>
> >>> Why would the hardware require VAs to match the ones in the
> >>> userspace address-space, if it doesn't use the CPU MMU at all?
> >>>
> >>
> >> It works like that:
> >>
> >> 1. We register a buffer from the process address space
> >>    giving its base address and length.
> >>    This call goes to kernel which in turn pins the phys pages
> >>    and registers them with the device *together* with the base
> >>    address (virtual address!)
> >> 2. The device builds its own page tables to be able to translate
> >>    the virtual addresses to actual phys pages.
> > 
> > How would the device be able to do that?  It would require the
> > device to look at the process page tables, wouldn't it?  Isn't
> > the HW IOVA->PA translation table built by the OS?
> > 
> 
> As stated above, these are tables private for the device.
> (They even have a hw vendor specific layout I think,
>  since the device holds some cache)
> 
> The device looks at its own private page tables, and not
> to the OS ones.

I'm still confused by your statement that the device builds its
own [IOVA->PA] page table.  How would the device do that if it
doesn't have access to the CPU MMU state?  Isn't the IOVA->PA
translation table built by the OS?

> 
> > 
> >> 3. The process executes post-send requests directly to hw by-passing
> >>    the kernel giving process virtual addresses in work requests.
> >> 4. The device uses its own page tables to translate the virtual
> >>    addresses to phys pages and sending them.
> >>
> >> Theoretically is possible to send any contiguous IOVA instead of
> >> process's one but is not how is working today.
> >>
> >> Makes sense?
> > 
> 

-- 
Eduardo

Re: [Qemu-devel] [PATCH V8 1/4] mem: add share parameter to memory-backend-ram

[Marcel Apfelbaum]

On 01/02/2018 21:21, Eduardo Habkost wrote:
> On Thu, Feb 01, 2018 at 08:58:32PM +0200, Marcel Apfelbaum wrote:
>> On 01/02/2018 20:51, Eduardo Habkost wrote:
>>> On Thu, Feb 01, 2018 at 08:31:09PM +0200, Marcel Apfelbaum wrote:
>>>> On 01/02/2018 20:21, Eduardo Habkost wrote:
>>>>> On Thu, Feb 01, 2018 at 08:03:53PM +0200, Marcel Apfelbaum wrote:
>>>>>> On 01/02/2018 15:53, Eduardo Habkost wrote:
>>>>>>> On Thu, Feb 01, 2018 at 02:29:25PM +0200, Marcel Apfelbaum wrote:
>>>>>>>> On 01/02/2018 14:10, Eduardo Habkost wrote:
>>>>>>>>> On Thu, Feb 01, 2018 at 07:36:50AM +0200, Marcel Apfelbaum wrote:
>>>>>>>>>> On 01/02/2018 4:22, Michael S. Tsirkin wrote:
>>>>>>>>>>> On Wed, Jan 31, 2018 at 09:34:22PM -0200, Eduardo Habkost wrote:
>>>>>>>>> [...]
>>>>>>>>>>>> BTW, what's the root cause for requiring HVAs in the buffer?
>>>>>>>>>>>
>>>>>>>>>>> It's a side effect of the kernel/userspace API which always wants
>>>>>>>>>>> a single HVA/len pair to map memory for the application.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi Eduardo and Michael,
>>>>>>>>>>
>>>>>>>>>>>>  Can
>>>>>>>>>>>> this be fixed?
>>>>>>>>>>>
>>>>>>>>>>> I think yes.  It'd need to be a kernel patch for the RDMA subsystem
>>>>>>>>>>> mapping an s/g list with actual memory. The HVA/len pair would then just
>>>>>>>>>>> be used to refer to the region, without creating the two mappings.
>>>>>>>>>>>
>>>>>>>>>>> Something like splitting the register mr into
>>>>>>>>>>>
>>>>>>>>>>> mr = create mr (va/len) - allocate a handle and record the va/len
>>>>>>>>>>>
>>>>>>>>>>> addmemory(mr, offset, hva, len) - pin memory
>>>>>>>>>>>
>>>>>>>>>>> register mr - pass it to HW
>>>>>>>>>>>
>>>>>>>>>>> As a nice side effect we won't burn so much virtual address space.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> We would still need a contiguous virtual address space range (for post-send)
>>>>>>>>>> which we don't have since guest contiguous virtual address space
>>>>>>>>>> will always end up as non-contiguous host virtual address space.
>>>>>>>>>>
>>>>>>>>>> I am not sure the RDMA HW can handle a large VA with holes.
>>>>>>>>>
>>>>>>>>> I'm confused.  Why would the hardware see and care about virtual
>>>>>>>>> addresses? 
>>>>>>>>
>>>>>>>> The post-send operations bypasses the kernel, and the process
>>>>>>>> puts in the work request GVA addresses.
>>>>>>>>
>>>>>>>>> How exactly does the hardware translates VAs to
>>>>>>>>> PAs? 
>>>>>>>>
>>>>>>>> The HW maintains a page-directory like structure different form MMU
>>>>>>>> VA -> phys pages
>>>>>>>>
>>>>>>>>> What if the process page tables change?
>>>>>>>>>
>>>>>>>>
>>>>>>>> Since the page tables the HW uses are their own, we just need the phys
>>>>>>>> page to be pinned.
>>>>>>>
>>>>>>> So there's no hardware-imposed requirement that the hardware VAs
>>>>>>> (mapped by the HW page directory) match the VAs in QEMU
>>>>>>> address-space, right? 
>>>>>>
>>>>>> Actually there is. Today it works exactly as you described.
>>>>>
>>>>> Are you sure there's such hardware-imposed requirement?
>>>>>
>>>>
>>>> Yes.
>>>>
>>>>> Why would the hardware require VAs to match the ones in the
>>>>> userspace address-space, if it doesn't use the CPU MMU at all?
>>>>>
>>>>
>>>> It works like that:
>>>>
>>>> 1. We register a buffer from the process address space
>>>>    giving its base address and length.
>>>>    This call goes to kernel which in turn pins the phys pages
>>>>    and registers them with the device *together* with the base
>>>>    address (virtual address!)
>>>> 2. The device builds its own page tables to be able to translate
>>>>    the virtual addresses to actual phys pages.
>>>
>>> How would the device be able to do that?  It would require the
>>> device to look at the process page tables, wouldn't it?  Isn't
>>> the HW IOVA->PA translation table built by the OS?
>>>
>>
>> As stated above, these are tables private for the device.
>> (They even have a hw vendor specific layout I think,
>>  since the device holds some cache)
>>
>> The device looks at its own private page tables, and not
>> to the OS ones.
> 
> I'm still confused by your statement that the device builds its
> own [IOVA->PA] page table.  How would the device do that if it
> doesn't have access to the CPU MMU state?  Isn't the IOVA->PA
> translation table built by the OS?
> 

Sorry about the confusion. The device gets a base virtual address,
the memory region length and a list of phys pages.
This is enough information to create its own kind of tables
which will tell, for example, if the IOVA starts at address
0x1000, that address 0x1001 is at page 0 and address 0x2000 is at page 1.

Be aware this base virtual address can be from any address space, not only
from the process address, the process address space is only the current software
implementation.

Thanks,
Marcel

>>
>>>
>>>> 3. The process executes post-send requests directly to hw by-passing
>>>>    the kernel giving process virtual addresses in work requests.
>>>> 4. The device uses its own page tables to translate the virtual
>>>>    addresses to phys pages and sending them.
>>>>
>>>> Theoretically is possible to send any contiguous IOVA instead of
>>>> process's one but is not how is working today.
>>>>
>>>> Makes sense?
>>>
>>
> 

Re: [Qemu-devel] [PATCH V8 1/4] mem: add share parameter to memory-backend-ram

[Paolo Bonzini]

On 01/02/2018 14:21, Eduardo Habkost wrote:
>> The device looks at its own private page tables, and not
>> to the OS ones.
> I'm still confused by your statement that the device builds its
> own [IOVA->PA] page table.  How would the device do that if it
> doesn't have access to the CPU MMU state?  Isn't the IOVA->PA
> translation table built by the OS?

The driver builds a page table for the device, either when it pins the
pages or by using MMU notifiers.

Paolo

Re: [Qemu-devel] [PATCH V8 1/4] mem: add share parameter to memory-backend-ram

[Michael S. Tsirkin]

On Thu, Feb 01, 2018 at 08:31:09PM +0200, Marcel Apfelbaum wrote:
> Theoretically is possible to send any contiguous IOVA instead of
> process's one but is not how is working today.

It works this way today in hardware but it's not hardware that limits
it to work this way - it's a software limitation.

-- 
MST

Re: [Qemu-devel] [PATCH V8 1/4] mem: add share parameter to memory-backend-ram

[Michael S. Tsirkin]

On Thu, Feb 01, 2018 at 02:29:25PM +0200, Marcel Apfelbaum wrote:
> On 01/02/2018 14:10, Eduardo Habkost wrote:
> > On Thu, Feb 01, 2018 at 07:36:50AM +0200, Marcel Apfelbaum wrote:
> >> On 01/02/2018 4:22, Michael S. Tsirkin wrote:
> >>> On Wed, Jan 31, 2018 at 09:34:22PM -0200, Eduardo Habkost wrote:
> > [...]
> >>>> BTW, what's the root cause for requiring HVAs in the buffer?
> >>>
> >>> It's a side effect of the kernel/userspace API which always wants
> >>> a single HVA/len pair to map memory for the application.
> >>>
> >>>
> >>
> >> Hi Eduardo and Michael,
> >>
> >>>>  Can
> >>>> this be fixed?
> >>>
> >>> I think yes.  It'd need to be a kernel patch for the RDMA subsystem
> >>> mapping an s/g list with actual memory. The HVA/len pair would then just
> >>> be used to refer to the region, without creating the two mappings.
> >>>
> >>> Something like splitting the register mr into
> >>>
> >>> mr = create mr (va/len) - allocate a handle and record the va/len
> >>>
> >>> addmemory(mr, offset, hva, len) - pin memory
> >>>
> >>> register mr - pass it to HW
> >>>
> >>> As a nice side effect we won't burn so much virtual address space.
> >>>
> >>
> >> We would still need a contiguous virtual address space range (for post-send)
> >> which we don't have since guest contiguous virtual address space
> >> will always end up as non-contiguous host virtual address space.
> >>
> >> I am not sure the RDMA HW can handle a large VA with holes.
> > 
> > I'm confused.  Why would the hardware see and care about virtual
> > addresses? 
> 
> The post-send operations bypasses the kernel, and the process
> puts in the work request GVA addresses.

To be more precise, it's the guest supplied IOVA that is sent to the card.

> > How exactly does the hardware translates VAs to
> > PAs? 
> 
> The HW maintains a page-directory like structure different form MMU
> VA -> phys pages
> 
> > What if the process page tables change?
> > 
> 
> Since the page tables the HW uses are their own, we just need the phys
> page to be pinned.
> 
> >>
> >> An alternative would be 0-based MR, QEMU intercepts the post-send
> >> operations and can substract the guest VA base address.
> >> However I didn't see the implementation in kernel for 0 based MRs
> >> and also the RDMA maintainer said it would work for local keys
> >> and not for remote keys.
> > 
> > This is also unexpected: are GVAs visible to the virtual RDMA
> > hardware? 
> 
> Yes, explained above
> 
> > Where does the QEMU pvrdma code translates GVAs to
> > GPAs?
> > 
> 
> During reg_mr (memory registration commands)
> Then it registers the same addresses to the real HW.
> (as Host virtual addresses)
> 
> Thanks,
> Marcel


The full fix would be to allow QEMU to map a list of
pages to a guest supplied IOVA.

> >>
> >>> This will fix rdma with hugetlbfs as well which is currently broken.
> >>>
> >>>
> >>
> >> There is already a discussion on the linux-rdma list:
> >>     https://www.spinics.net/lists/linux-rdma/msg60079.html
> >> But it will take some (actually a lot of) time, we are currently talking about
> >> a possible API. And it does not solve the re-mapping...
> >>
> >> Thanks,
> >> Marcel
> >>
> >>>> -- 
> >>>> Eduardo
> >>
> > 

Re: [Qemu-devel] [PATCH V8 1/4] mem: add share parameter to memory-backend-ram

[Eduardo Habkost]

On Thu, Feb 01, 2018 at 04:24:30PM +0200, Michael S. Tsirkin wrote:
[...]
> The full fix would be to allow QEMU to map a list of
> pages to a guest supplied IOVA.

Thanks, that's what I expected.

While this is not possible, the only requests I have for this
patch is that we clearly document:
* What's the only purpose of share=on on a host-memory-backend
  object (due to pvrdma limitations).
* The potential undesirable side-effects of setting share=on.
* On the commit message and other comments, clearly distinguish
  HVAs in the QEMU address-space from IOVAs, to avoid confusion.

-- 
Eduardo

Re: [Qemu-devel] [PATCH V8 1/4] mem: add share parameter to memory-backend-ram

[Michael S. Tsirkin]

On Thu, Feb 01, 2018 at 02:31:32PM -0200, Eduardo Habkost wrote:
> On Thu, Feb 01, 2018 at 04:24:30PM +0200, Michael S. Tsirkin wrote:
> [...]
> > The full fix would be to allow QEMU to map a list of
> > pages to a guest supplied IOVA.
> 
> Thanks, that's what I expected.
> 
> While this is not possible, the only requests I have for this
> patch is that we clearly document:
> * What's the only purpose of share=on on a host-memory-backend
>   object (due to pvrdma limitations).
> * The potential undesirable side-effects of setting share=on.
> * On the commit message and other comments, clearly distinguish
>   HVAs in the QEMU address-space from IOVAs, to avoid confusion.

Looking forward, when we do support it, how will management find out
it no longer needs to pass the share parameter?

Further, if the side effects of the share parameter go away,
how will it know these no longer hold?

> -- 
> Eduardo

Re: [Qemu-devel] [PATCH V8 1/4] mem: add share parameter to memory-backend-ram

[Eduardo Habkost]

On Thu, Feb 01, 2018 at 06:48:54PM +0200, Michael S. Tsirkin wrote:
> On Thu, Feb 01, 2018 at 02:31:32PM -0200, Eduardo Habkost wrote:
> > On Thu, Feb 01, 2018 at 04:24:30PM +0200, Michael S. Tsirkin wrote:
> > [...]
> > > The full fix would be to allow QEMU to map a list of
> > > pages to a guest supplied IOVA.
> > 
> > Thanks, that's what I expected.
> > 
> > While this is not possible, the only requests I have for this
> > patch is that we clearly document:
> > * What's the only purpose of share=on on a host-memory-backend
> >   object (due to pvrdma limitations).
> > * The potential undesirable side-effects of setting share=on.
> > * On the commit message and other comments, clearly distinguish
> >   HVAs in the QEMU address-space from IOVAs, to avoid confusion.
> 
> Looking forward, when we do support it, how will management find out
> it no longer needs to pass the share parameter?
> 
> Further, if the side effects of the share parameter go away,
> how will it know these no longer hold?

A query-host-capabilities or similar QMP command seems necessary
for that.  It would be useful for other stuff like MAP_SYNC.

-- 
Eduardo

Re: [Qemu-devel] [PATCH V8 1/4] mem: add share parameter to memory-backend-ram

[Michael S. Tsirkin]

On Thu, Feb 01, 2018 at 02:57:39PM -0200, Eduardo Habkost wrote:
> On Thu, Feb 01, 2018 at 06:48:54PM +0200, Michael S. Tsirkin wrote:
> > On Thu, Feb 01, 2018 at 02:31:32PM -0200, Eduardo Habkost wrote:
> > > On Thu, Feb 01, 2018 at 04:24:30PM +0200, Michael S. Tsirkin wrote:
> > > [...]
> > > > The full fix would be to allow QEMU to map a list of
> > > > pages to a guest supplied IOVA.
> > > 
> > > Thanks, that's what I expected.
> > > 
> > > While this is not possible, the only requests I have for this
> > > patch is that we clearly document:
> > > * What's the only purpose of share=on on a host-memory-backend
> > >   object (due to pvrdma limitations).
> > > * The potential undesirable side-effects of setting share=on.
> > > * On the commit message and other comments, clearly distinguish
> > >   HVAs in the QEMU address-space from IOVAs, to avoid confusion.
> > 
> > Looking forward, when we do support it, how will management find out
> > it no longer needs to pass the share parameter?
> > 
> > Further, if the side effects of the share parameter go away,
> > how will it know these no longer hold?
> 
> A query-host-capabilities or similar QMP command seems necessary
> for that.

Is anyone working on that?

> It would be useful for other stuff like MAP_SYNC.


> -- 
> Eduardo

Re: [Qemu-devel] [PATCH V8 1/4] mem: add share parameter to memory-backend-ram

[Eduardo Habkost]

On Thu, Feb 01, 2018 at 06:59:07PM +0200, Michael S. Tsirkin wrote:
> On Thu, Feb 01, 2018 at 02:57:39PM -0200, Eduardo Habkost wrote:
> > On Thu, Feb 01, 2018 at 06:48:54PM +0200, Michael S. Tsirkin wrote:
> > > On Thu, Feb 01, 2018 at 02:31:32PM -0200, Eduardo Habkost wrote:
> > > > On Thu, Feb 01, 2018 at 04:24:30PM +0200, Michael S. Tsirkin wrote:
> > > > [...]
> > > > > The full fix would be to allow QEMU to map a list of
> > > > > pages to a guest supplied IOVA.
> > > > 
> > > > Thanks, that's what I expected.
> > > > 
> > > > While this is not possible, the only requests I have for this
> > > > patch is that we clearly document:
> > > > * What's the only purpose of share=on on a host-memory-backend
> > > >   object (due to pvrdma limitations).
> > > > * The potential undesirable side-effects of setting share=on.
> > > > * On the commit message and other comments, clearly distinguish
> > > >   HVAs in the QEMU address-space from IOVAs, to avoid confusion.
> > > 
> > > Looking forward, when we do support it, how will management find out
> > > it no longer needs to pass the share parameter?
> > > 
> > > Further, if the side effects of the share parameter go away,
> > > how will it know these no longer hold?
> > 
> > A query-host-capabilities or similar QMP command seems necessary
> > for that.
> 
> Is anyone working on that?

Not yet.

-- 
Eduardo

Re: [Qemu-devel] [PATCH V8 1/4] mem: add share parameter to memory-backend-ram

[Michael S. Tsirkin]

On Thu, Feb 01, 2018 at 03:01:36PM -0200, Eduardo Habkost wrote:
> On Thu, Feb 01, 2018 at 06:59:07PM +0200, Michael S. Tsirkin wrote:
> > On Thu, Feb 01, 2018 at 02:57:39PM -0200, Eduardo Habkost wrote:
> > > On Thu, Feb 01, 2018 at 06:48:54PM +0200, Michael S. Tsirkin wrote:
> > > > On Thu, Feb 01, 2018 at 02:31:32PM -0200, Eduardo Habkost wrote:
> > > > > On Thu, Feb 01, 2018 at 04:24:30PM +0200, Michael S. Tsirkin wrote:
> > > > > [...]
> > > > > > The full fix would be to allow QEMU to map a list of
> > > > > > pages to a guest supplied IOVA.
> > > > > 
> > > > > Thanks, that's what I expected.
> > > > > 
> > > > > While this is not possible, the only requests I have for this
> > > > > patch is that we clearly document:
> > > > > * What's the only purpose of share=on on a host-memory-backend
> > > > >   object (due to pvrdma limitations).
> > > > > * The potential undesirable side-effects of setting share=on.
> > > > > * On the commit message and other comments, clearly distinguish
> > > > >   HVAs in the QEMU address-space from IOVAs, to avoid confusion.
> > > > 
> > > > Looking forward, when we do support it, how will management find out
> > > > it no longer needs to pass the share parameter?
> > > > 
> > > > Further, if the side effects of the share parameter go away,
> > > > how will it know these no longer hold?
> > > 
> > > A query-host-capabilities or similar QMP command seems necessary
> > > for that.
> > 
> > Is anyone working on that?
> 
> Not yet.
> 
> -- 
> Eduardo

Do these patches need to wait until we do have that command?

I'm thinking it's better to have "share=on required with rdma"
and "hugetlbfs not supported with rdma"
than the reverse, this way new hosts do not need to carry
thus stuff around forever.

Also, how does management know which devices are affected?

-- 
MST

Re: [Qemu-devel] [PATCH V8 1/4] mem: add share parameter to memory-backend-ram

[Eduardo Habkost]

On Thu, Feb 01, 2018 at 07:12:45PM +0200, Michael S. Tsirkin wrote:
> On Thu, Feb 01, 2018 at 03:01:36PM -0200, Eduardo Habkost wrote:
> > On Thu, Feb 01, 2018 at 06:59:07PM +0200, Michael S. Tsirkin wrote:
> > > On Thu, Feb 01, 2018 at 02:57:39PM -0200, Eduardo Habkost wrote:
> > > > On Thu, Feb 01, 2018 at 06:48:54PM +0200, Michael S. Tsirkin wrote:
> > > > > On Thu, Feb 01, 2018 at 02:31:32PM -0200, Eduardo Habkost wrote:
> > > > > > On Thu, Feb 01, 2018 at 04:24:30PM +0200, Michael S. Tsirkin wrote:
> > > > > > [...]
> > > > > > > The full fix would be to allow QEMU to map a list of
> > > > > > > pages to a guest supplied IOVA.
> > > > > > 
> > > > > > Thanks, that's what I expected.
> > > > > > 
> > > > > > While this is not possible, the only requests I have for this
> > > > > > patch is that we clearly document:
> > > > > > * What's the only purpose of share=on on a host-memory-backend
> > > > > >   object (due to pvrdma limitations).
> > > > > > * The potential undesirable side-effects of setting share=on.
> > > > > > * On the commit message and other comments, clearly distinguish
> > > > > >   HVAs in the QEMU address-space from IOVAs, to avoid confusion.
> > > > > 
> > > > > Looking forward, when we do support it, how will management find out
> > > > > it no longer needs to pass the share parameter?
> > > > > 
> > > > > Further, if the side effects of the share parameter go away,
> > > > > how will it know these no longer hold?
> > > > 
> > > > A query-host-capabilities or similar QMP command seems necessary
> > > > for that.
> > > 
> > > Is anyone working on that?
> > 
> > Not yet.
> > 
> > -- 
> > Eduardo
> 
> Do these patches need to wait until we do have that command?

I don't think so.  The command will be needed only when
support for pvrdma without share=on gets implemented.

Right now, all we need is clear documentation.

> 
> I'm thinking it's better to have "share=on required with rdma"
> and "hugetlbfs not supported with rdma"
> than the reverse, this way new hosts do not need to carry
> thus stuff around forever.

What do you mean by "the reverse"?

IIUC, the requirements/limitations are:

* share=on required for pvrdma.  Already documented and enforced
  by pvrdma code in this series.
* hugetlbfs not supported with rdma. Is this detected/reported by
  QEMU?  Is it documented?
* side-effects of share=on.  This is not detected nor documented,
  and probably already applies to other memory backends.
  * Nice to have: document when share=on is useful (answer:
    because of pvrdma), when adding share=on support to
    host-memory-backend.

> 
> Also, how does management know which devices are affected?

Right now?  By reading documentation.

-- 
Eduardo

Re: [Qemu-devel] [PATCH V8 1/4] mem: add share parameter to memory-backend-ram

[Marcel Apfelbaum]

On 01/02/2018 19:36, Eduardo Habkost wrote:
> On Thu, Feb 01, 2018 at 07:12:45PM +0200, Michael S. Tsirkin wrote:
>> On Thu, Feb 01, 2018 at 03:01:36PM -0200, Eduardo Habkost wrote:
>>> On Thu, Feb 01, 2018 at 06:59:07PM +0200, Michael S. Tsirkin wrote:
>>>> On Thu, Feb 01, 2018 at 02:57:39PM -0200, Eduardo Habkost wrote:
>>>>> On Thu, Feb 01, 2018 at 06:48:54PM +0200, Michael S. Tsirkin wrote:
>>>>>> On Thu, Feb 01, 2018 at 02:31:32PM -0200, Eduardo Habkost wrote:
>>>>>>> On Thu, Feb 01, 2018 at 04:24:30PM +0200, Michael S. Tsirkin wrote:
>>>>>>> [...]
>>>>>>>> The full fix would be to allow QEMU to map a list of
>>>>>>>> pages to a guest supplied IOVA.
>>>>>>>
>>>>>>> Thanks, that's what I expected.
>>>>>>>
>>>>>>> While this is not possible, the only requests I have for this
>>>>>>> patch is that we clearly document:
>>>>>>> * What's the only purpose of share=on on a host-memory-backend
>>>>>>>   object (due to pvrdma limitations).
>>>>>>> * The potential undesirable side-effects of setting share=on.
>>>>>>> * On the commit message and other comments, clearly distinguish
>>>>>>>   HVAs in the QEMU address-space from IOVAs, to avoid confusion.
>>>>>>
>>>>>> Looking forward, when we do support it, how will management find out
>>>>>> it no longer needs to pass the share parameter?
>>>>>>
>>>>>> Further, if the side effects of the share parameter go away,
>>>>>> how will it know these no longer hold?
>>>>>
>>>>> A query-host-capabilities or similar QMP command seems necessary
>>>>> for that.
>>>>
>>>> Is anyone working on that?
>>>
>>> Not yet.
>>>
>>> -- 
>>> Eduardo
>>
>> Do these patches need to wait until we do have that command?
> 
> I don't think so.  The command will be needed only when
> support for pvrdma without share=on gets implemented.
> 
> Right now, all we need is clear documentation.
> 
>>
>> I'm thinking it's better to have "share=on required with rdma"
>> and "hugetlbfs not supported with rdma"
>> than the reverse, this way new hosts do not need to carry
>> thus stuff around forever.
> 
> What do you mean by "the reverse"?
> 
> IIUC, the requirements/limitations are:
> 
> * share=on required for pvrdma.  Already documented and enforced
>   by pvrdma code in this series.

Right.

> * hugetlbfs not supported with rdma. Is this detected/reported by
>   QEMU?  Is it documented?

Yes, enforced by the pvrdma device initialization and documented in the
corresponding pvrdma doc.

> * side-effects of share=on.  This is not detected nor documented,
>   and probably already applies to other memory backends.
>   * Nice to have: document when share=on is useful (answer:
>     because of pvrdma), when adding share=on support to
>     host-memory-backend.
> 

The documentation is part of the pvrdma doc.
What are the side-effects of share=on? I missed that.
(share=on is new for the memory backed RAM, the file
backed RAM already had the share parameter)

One can just grep for "share=on" in the docs directory
and can easily see the only current usage. But maybe will
be more, maybe we don't want to limit it for now.

I am planning to re-spin today/tomorrow before sending
a pull-request, can you please point me on what documentation
to add and what side-effects I should document?

Thanks,
Marcel

>>
>> Also, how does management know which devices are affected?
> 
> Right now?  By reading documentation.
> 

Re: [Qemu-devel] [PATCH V8 1/4] mem: add share parameter to memory-backend-ram

[Eduardo Habkost]

On Thu, Feb 01, 2018 at 07:58:10PM +0200, Marcel Apfelbaum wrote:
> On 01/02/2018 19:36, Eduardo Habkost wrote:
> > On Thu, Feb 01, 2018 at 07:12:45PM +0200, Michael S. Tsirkin wrote:
> >> On Thu, Feb 01, 2018 at 03:01:36PM -0200, Eduardo Habkost wrote:
> >>> On Thu, Feb 01, 2018 at 06:59:07PM +0200, Michael S. Tsirkin wrote:
> >>>> On Thu, Feb 01, 2018 at 02:57:39PM -0200, Eduardo Habkost wrote:
> >>>>> On Thu, Feb 01, 2018 at 06:48:54PM +0200, Michael S. Tsirkin wrote:
> >>>>>> On Thu, Feb 01, 2018 at 02:31:32PM -0200, Eduardo Habkost wrote:
> >>>>>>> On Thu, Feb 01, 2018 at 04:24:30PM +0200, Michael S. Tsirkin wrote:
> >>>>>>> [...]
> >>>>>>>> The full fix would be to allow QEMU to map a list of
> >>>>>>>> pages to a guest supplied IOVA.
> >>>>>>>
> >>>>>>> Thanks, that's what I expected.
> >>>>>>>
> >>>>>>> While this is not possible, the only requests I have for this
> >>>>>>> patch is that we clearly document:
> >>>>>>> * What's the only purpose of share=on on a host-memory-backend
> >>>>>>>   object (due to pvrdma limitations).
> >>>>>>> * The potential undesirable side-effects of setting share=on.
> >>>>>>> * On the commit message and other comments, clearly distinguish
> >>>>>>>   HVAs in the QEMU address-space from IOVAs, to avoid confusion.
> >>>>>>
> >>>>>> Looking forward, when we do support it, how will management find out
> >>>>>> it no longer needs to pass the share parameter?
> >>>>>>
> >>>>>> Further, if the side effects of the share parameter go away,
> >>>>>> how will it know these no longer hold?
> >>>>>
> >>>>> A query-host-capabilities or similar QMP command seems necessary
> >>>>> for that.
> >>>>
> >>>> Is anyone working on that?
> >>>
> >>> Not yet.
> >>>
> >>> -- 
> >>> Eduardo
> >>
> >> Do these patches need to wait until we do have that command?
> > 
> > I don't think so.  The command will be needed only when
> > support for pvrdma without share=on gets implemented.
> > 
> > Right now, all we need is clear documentation.
> > 
> >>
> >> I'm thinking it's better to have "share=on required with rdma"
> >> and "hugetlbfs not supported with rdma"
> >> than the reverse, this way new hosts do not need to carry
> >> thus stuff around forever.
> > 
> > What do you mean by "the reverse"?
> > 
> > IIUC, the requirements/limitations are:
> > 
> > * share=on required for pvrdma.  Already documented and enforced
> >   by pvrdma code in this series.
> 
> Right.
> 
> > * hugetlbfs not supported with rdma. Is this detected/reported by
> >   QEMU?  Is it documented?
> 
> Yes, enforced by the pvrdma device initialization and documented in the
> corresponding pvrdma doc.
> 
> > * side-effects of share=on.  This is not detected nor documented,
> >   and probably already applies to other memory backends.
> >   * Nice to have: document when share=on is useful (answer:
> >     because of pvrdma), when adding share=on support to
> >     host-memory-backend.
> > 
> 
> The documentation is part of the pvrdma doc.
> What are the side-effects of share=on? I missed that.
> (share=on is new for the memory backed RAM, the file
> backed RAM already had the share parameter)
> 
> One can just grep for "share=on" in the docs directory
> and can easily see the only current usage. But maybe will
> be more, maybe we don't want to limit it for now.
> 
> I am planning to re-spin today/tomorrow before sending
> a pull-request, can you please point me on what documentation
> to add and what side-effects I should document?
> 

The full list of side-effects is not clear to me.  For some of
them, see Documentation/vm/numa_memory_policy.txt on the kernel
tree.

The documentation for memory backend options is at
qemu-options.hx.  Maybe something like this, extending the
existing paragraph:

  The @option{share} boolean option determines whether the memory
  region is marked as private to QEMU, or shared (mapped using
  the MAP_SHARED flag).  The latter allows a co-operating
  external process to access the QEMU memory region.

  @option{share} is also required for pvrdma devices due to
  limitations in the RDMA API provided by Linux.

  Setting share=on might affect the ability to configure NUMA
  bindings for the memory backend under some circumstances, see
  Documentation/vm/numa_memory_policy.txt on the Linux kernel
  source tree for additional details.

I hate to point users to low-level documentation on the kernel
tree, but it's better than nothing.

We also need to list "share" as a valid option at the
"@item -object memory-backend-ram,[...]" line.

-- 
Eduardo

Re: [Qemu-devel] [PATCH V8 1/4] mem: add share parameter to memory-backend-ram

[Marcel Apfelbaum]

On 01/02/2018 20:18, Eduardo Habkost wrote:
> On Thu, Feb 01, 2018 at 07:58:10PM +0200, Marcel Apfelbaum wrote:
>> On 01/02/2018 19:36, Eduardo Habkost wrote:
>>> On Thu, Feb 01, 2018 at 07:12:45PM +0200, Michael S. Tsirkin wrote:
>>>> On Thu, Feb 01, 2018 at 03:01:36PM -0200, Eduardo Habkost wrote:
>>>>> On Thu, Feb 01, 2018 at 06:59:07PM +0200, Michael S. Tsirkin wrote:
>>>>>> On Thu, Feb 01, 2018 at 02:57:39PM -0200, Eduardo Habkost wrote:
>>>>>>> On Thu, Feb 01, 2018 at 06:48:54PM +0200, Michael S. Tsirkin wrote:
>>>>>>>> On Thu, Feb 01, 2018 at 02:31:32PM -0200, Eduardo Habkost wrote:
>>>>>>>>> On Thu, Feb 01, 2018 at 04:24:30PM +0200, Michael S. Tsirkin wrote:
>>>>>>>>> [...]
>>>>>>>>>> The full fix would be to allow QEMU to map a list of
>>>>>>>>>> pages to a guest supplied IOVA.
>>>>>>>>>
>>>>>>>>> Thanks, that's what I expected.
>>>>>>>>>
>>>>>>>>> While this is not possible, the only requests I have for this
>>>>>>>>> patch is that we clearly document:
>>>>>>>>> * What's the only purpose of share=on on a host-memory-backend
>>>>>>>>>   object (due to pvrdma limitations).
>>>>>>>>> * The potential undesirable side-effects of setting share=on.
>>>>>>>>> * On the commit message and other comments, clearly distinguish
>>>>>>>>>   HVAs in the QEMU address-space from IOVAs, to avoid confusion.
>>>>>>>>
>>>>>>>> Looking forward, when we do support it, how will management find out
>>>>>>>> it no longer needs to pass the share parameter?
>>>>>>>>
>>>>>>>> Further, if the side effects of the share parameter go away,
>>>>>>>> how will it know these no longer hold?
>>>>>>>
>>>>>>> A query-host-capabilities or similar QMP command seems necessary
>>>>>>> for that.
>>>>>>
>>>>>> Is anyone working on that?
>>>>>
>>>>> Not yet.
>>>>>
>>>>> -- 
>>>>> Eduardo
>>>>
>>>> Do these patches need to wait until we do have that command?
>>>
>>> I don't think so.  The command will be needed only when
>>> support for pvrdma without share=on gets implemented.
>>>
>>> Right now, all we need is clear documentation.
>>>
>>>>
>>>> I'm thinking it's better to have "share=on required with rdma"
>>>> and "hugetlbfs not supported with rdma"
>>>> than the reverse, this way new hosts do not need to carry
>>>> thus stuff around forever.
>>>
>>> What do you mean by "the reverse"?
>>>
>>> IIUC, the requirements/limitations are:
>>>
>>> * share=on required for pvrdma.  Already documented and enforced
>>>   by pvrdma code in this series.
>>
>> Right.
>>
>>> * hugetlbfs not supported with rdma. Is this detected/reported by
>>>   QEMU?  Is it documented?
>>
>> Yes, enforced by the pvrdma device initialization and documented in the
>> corresponding pvrdma doc.
>>
>>> * side-effects of share=on.  This is not detected nor documented,
>>>   and probably already applies to other memory backends.
>>>   * Nice to have: document when share=on is useful (answer:
>>>     because of pvrdma), when adding share=on support to
>>>     host-memory-backend.
>>>
>>
>> The documentation is part of the pvrdma doc.
>> What are the side-effects of share=on? I missed that.
>> (share=on is new for the memory backed RAM, the file
>> backed RAM already had the share parameter)
>>
>> One can just grep for "share=on" in the docs directory
>> and can easily see the only current usage. But maybe will
>> be more, maybe we don't want to limit it for now.
>>
>> I am planning to re-spin today/tomorrow before sending
>> a pull-request, can you please point me on what documentation
>> to add and what side-effects I should document?
>>
> 
> The full list of side-effects is not clear to me.  For some of
> them, see Documentation/vm/numa_memory_policy.txt on the kernel
> tree.
> 
> The documentation for memory backend options is at
> qemu-options.hx.  Maybe something like this, extending the
> existing paragraph:
> 
>   The @option{share} boolean option determines whether the memory
>   region is marked as private to QEMU, or shared (mapped using
>   the MAP_SHARED flag).  The latter allows a co-operating
>   external process to access the QEMU memory region.
> 
>   @option{share} is also required for pvrdma devices due to
>   limitations in the RDMA API provided by Linux.
> 
>   Setting share=on might affect the ability to configure NUMA
>   bindings for the memory backend under some circumstances, see
>   Documentation/vm/numa_memory_policy.txt on the Linux kernel
>   source tree for additional details.
> 
> I hate to point users to low-level documentation on the kernel
> tree, but it's better than nothing.
> 
> We also need to list "share" as a valid option at the
> "@item -object memory-backend-ram,[...]" line.
> 

Thanks for the help Eduardo, I'll be sure to update the docs as
advised.
Marcel

Re: [Qemu-devel] [PATCH V8 1/4] mem: add share parameter to memory-backend-ram

[Michael S. Tsirkin]

On Thu, Feb 01, 2018 at 03:36:55PM -0200, Eduardo Habkost wrote:
> On Thu, Feb 01, 2018 at 07:12:45PM +0200, Michael S. Tsirkin wrote:
> > On Thu, Feb 01, 2018 at 03:01:36PM -0200, Eduardo Habkost wrote:
> > > On Thu, Feb 01, 2018 at 06:59:07PM +0200, Michael S. Tsirkin wrote:
> > > > On Thu, Feb 01, 2018 at 02:57:39PM -0200, Eduardo Habkost wrote:
> > > > > On Thu, Feb 01, 2018 at 06:48:54PM +0200, Michael S. Tsirkin wrote:
> > > > > > On Thu, Feb 01, 2018 at 02:31:32PM -0200, Eduardo Habkost wrote:
> > > > > > > On Thu, Feb 01, 2018 at 04:24:30PM +0200, Michael S. Tsirkin wrote:
> > > > > > > [...]
> > > > > > > > The full fix would be to allow QEMU to map a list of
> > > > > > > > pages to a guest supplied IOVA.
> > > > > > > 
> > > > > > > Thanks, that's what I expected.
> > > > > > > 
> > > > > > > While this is not possible, the only requests I have for this
> > > > > > > patch is that we clearly document:
> > > > > > > * What's the only purpose of share=on on a host-memory-backend
> > > > > > >   object (due to pvrdma limitations).
> > > > > > > * The potential undesirable side-effects of setting share=on.
> > > > > > > * On the commit message and other comments, clearly distinguish
> > > > > > >   HVAs in the QEMU address-space from IOVAs, to avoid confusion.
> > > > > > 
> > > > > > Looking forward, when we do support it, how will management find out
> > > > > > it no longer needs to pass the share parameter?
> > > > > > 
> > > > > > Further, if the side effects of the share parameter go away,
> > > > > > how will it know these no longer hold?
> > > > > 
> > > > > A query-host-capabilities or similar QMP command seems necessary
> > > > > for that.
> > > > 
> > > > Is anyone working on that?
> > > 
> > > Not yet.
> > > 
> > > -- 
> > > Eduardo
> > 
> > Do these patches need to wait until we do have that command?
> 
> I don't think so.  The command will be needed only when
> support for pvrdma without share=on gets implemented.
> 
> Right now, all we need is clear documentation.
> 
> > 
> > I'm thinking it's better to have "share=on required with rdma"
> > and "hugetlbfs not supported with rdma"
> > than the reverse, this way new hosts do not need to carry
> > thus stuff around forever.
> 
> What do you mean by "the reverse"?
> 
> IIUC, the requirements/limitations are:
> 
> * share=on required for pvrdma.  Already documented and enforced
>   by pvrdma code in this series.
> * hugetlbfs not supported with rdma. Is this detected/reported by
>   QEMU?  Is it documented?

Probably should be.

> * side-effects of share=on.  This is not detected nor documented,
>   and probably already applies to other memory backends.
>   * Nice to have: document when share=on is useful (answer:
>     because of pvrdma), when adding share=on support to
>     host-memory-backend.
> 
> > 
> > Also, how does management know which devices are affected?
> 
> Right now?  By reading documentation.


> -- 
> Eduardo

Re: [Qemu-devel] [PATCH V8 1/4] mem: add share parameter to memory-backend-ram

[Marcel Apfelbaum]

On 01/02/2018 16:24, Michael S. Tsirkin wrote:
> On Thu, Feb 01, 2018 at 02:29:25PM +0200, Marcel Apfelbaum wrote:
>> On 01/02/2018 14:10, Eduardo Habkost wrote:
>>> On Thu, Feb 01, 2018 at 07:36:50AM +0200, Marcel Apfelbaum wrote:
>>>> On 01/02/2018 4:22, Michael S. Tsirkin wrote:
>>>>> On Wed, Jan 31, 2018 at 09:34:22PM -0200, Eduardo Habkost wrote:
>>> [...]
>>>>>> BTW, what's the root cause for requiring HVAs in the buffer?
>>>>>
>>>>> It's a side effect of the kernel/userspace API which always wants
>>>>> a single HVA/len pair to map memory for the application.
>>>>>
>>>>>
>>>>
>>>> Hi Eduardo and Michael,
>>>>
>>>>>>  Can
>>>>>> this be fixed?
>>>>>
>>>>> I think yes.  It'd need to be a kernel patch for the RDMA subsystem
>>>>> mapping an s/g list with actual memory. The HVA/len pair would then just
>>>>> be used to refer to the region, without creating the two mappings.
>>>>>
>>>>> Something like splitting the register mr into
>>>>>
>>>>> mr = create mr (va/len) - allocate a handle and record the va/len
>>>>>
>>>>> addmemory(mr, offset, hva, len) - pin memory
>>>>>
>>>>> register mr - pass it to HW
>>>>>
>>>>> As a nice side effect we won't burn so much virtual address space.
>>>>>
>>>>
>>>> We would still need a contiguous virtual address space range (for post-send)
>>>> which we don't have since guest contiguous virtual address space
>>>> will always end up as non-contiguous host virtual address space.
>>>>
>>>> I am not sure the RDMA HW can handle a large VA with holes.
>>>
>>> I'm confused.  Why would the hardware see and care about virtual
>>> addresses? 
>>
>> The post-send operations bypasses the kernel, and the process
>> puts in the work request GVA addresses.
> 
> To be more precise, it's the guest supplied IOVA that is sent to the card.
> 
>>> How exactly does the hardware translates VAs to
>>> PAs? 
>>
>> The HW maintains a page-directory like structure different form MMU
>> VA -> phys pages
>>
>>> What if the process page tables change?
>>>
>>
>> Since the page tables the HW uses are their own, we just need the phys
>> page to be pinned.
>>
>>>>
>>>> An alternative would be 0-based MR, QEMU intercepts the post-send
>>>> operations and can substract the guest VA base address.
>>>> However I didn't see the implementation in kernel for 0 based MRs
>>>> and also the RDMA maintainer said it would work for local keys
>>>> and not for remote keys.
>>>
>>> This is also unexpected: are GVAs visible to the virtual RDMA
>>> hardware? 
>>
>> Yes, explained above
>>
>>> Where does the QEMU pvrdma code translates GVAs to
>>> GPAs?
>>>
>>
>> During reg_mr (memory registration commands)
>> Then it registers the same addresses to the real HW.
>> (as Host virtual addresses)
>>
>> Thanks,
>> Marcel
> 
> 
> The full fix would be to allow QEMU to map a list of
> pages to a guest supplied IOVA.
> 

Agreed, we are trying to influence the RDMA discussion
on the new API in this direction.

Thanks,
Marcel

>>>>
>>>>> This will fix rdma with hugetlbfs as well which is currently broken.
>>>>>
>>>>>
>>>>
>>>> There is already a discussion on the linux-rdma list:
>>>>     https://www.spinics.net/lists/linux-rdma/msg60079.html
>>>> But it will take some (actually a lot of) time, we are currently talking about
>>>> a possible API. And it does not solve the re-mapping...
>>>>
>>>> Thanks,
>>>> Marcel
>>>>
>>>>>> -- 
>>>>>> Eduardo
>>>>
>>>

Re: [Qemu-devel] [PATCH V8 1/4] mem: add share parameter to memory-backend-ram

[Michael S. Tsirkin]

On Thu, Feb 01, 2018 at 07:36:50AM +0200, Marcel Apfelbaum wrote:
> On 01/02/2018 4:22, Michael S. Tsirkin wrote:
> > On Wed, Jan 31, 2018 at 09:34:22PM -0200, Eduardo Habkost wrote:
> >> On Wed, Jan 31, 2018 at 11:10:07PM +0200, Michael S. Tsirkin wrote:
> >>> On Wed, Jan 31, 2018 at 06:40:59PM -0200, Eduardo Habkost wrote:
> >>>> On Wed, Jan 17, 2018 at 11:54:18AM +0200, Marcel Apfelbaum wrote:
> >>>>> Currently only file backed memory backend can
> >>>>> be created with a "share" flag in order to allow
> >>>>> sharing guest RAM with other processes in the host.
> >>>>>
> >>>>> Add the "share" flag also to RAM Memory Backend
> >>>>> in order to allow remapping parts of the guest RAM
> >>>>> to different host virtual addresses. This is needed
> >>>>> by the RDMA devices in order to remap non-contiguous
> >>>>> QEMU virtual addresses to a contiguous virtual address range.
> >>>>>
> >>>>
> >>>> Why do we need to make this configurable?  Would anything break
> >>>> if MAP_SHARED was always used if possible?
> >>>
> >>> See Documentation/vm/numa_memory_policy.txt for a list
> >>> of complications.
> >>
> >> Ew.
> >>
> >>>
> >>> Maybe we should more of an effort to detect and report these
> >>> issues.
> >>
> >> Probably.  Having other features breaking silently when using
> >> pvrdma doesn't sound good.  We must at least document those
> >> problems in the documentation for memory-backend-ram.
> >>
> >> BTW, what's the root cause for requiring HVAs in the buffer?
> > 
> > It's a side effect of the kernel/userspace API which always wants
> > a single HVA/len pair to map memory for the application.
> > 
> > 
> 
> Hi Eduardo and Michael,
> 
> >>  Can
> >> this be fixed?
> > 
> > I think yes.  It'd need to be a kernel patch for the RDMA subsystem
> > mapping an s/g list with actual memory. The HVA/len pair would then just
> > be used to refer to the region, without creating the two mappings.
> > 
> > Something like splitting the register mr into
> > 
> > mr = create mr (va/len) - allocate a handle and record the va/len
> > 
> > addmemory(mr, offset, hva, len) - pin memory
> > 
> > register mr - pass it to HW
> > 
> > As a nice side effect we won't burn so much virtual address space.
> >
> 
> We would still need a contiguous virtual address space range (for post-send)
> which we don't have since guest contiguous virtual address space
> will always end up as non-contiguous host virtual address space.

It just needs to be contiguous in the HCA virtual address space.
Software never accesses through this pointer.
In other words - basically expose register physical mr to userspace.


> 
> I am not sure the RDMA HW can handle a large VA with holes.
> 
> An alternative would be 0-based MR, QEMU intercepts the post-send
> operations and can substract the guest VA base address.
> However I didn't see the implementation in kernel for 0 based MRs
> and also the RDMA maintainer said it would work for local keys
> and not for remote keys.
> 
> > This will fix rdma with hugetlbfs as well which is currently broken.
> > 
> > 
> 
> There is already a discussion on the linux-rdma list:
>     https://www.spinics.net/lists/linux-rdma/msg60079.html
> But it will take some (actually a lot of) time, we are currently talking about
> a possible API.

You probably need to pass the s/g piece by piece since it might exceed
any reasonable array size.

> And it does not solve the re-mapping...
> 
> Thanks,
> Marcel

Haven't read through that discussion. But at least what I posted solves
it since you do not need it contiguous in HVA any longer.

> >> -- 
> >> Eduardo

Re: [Qemu-devel] [PATCH V8 1/4] mem: add share parameter to memory-backend-ram

[Marcel Apfelbaum]

On 01/02/2018 14:57, Michael S. Tsirkin wrote:
> On Thu, Feb 01, 2018 at 07:36:50AM +0200, Marcel Apfelbaum wrote:
>> On 01/02/2018 4:22, Michael S. Tsirkin wrote:
>>> On Wed, Jan 31, 2018 at 09:34:22PM -0200, Eduardo Habkost wrote:
>>>> On Wed, Jan 31, 2018 at 11:10:07PM +0200, Michael S. Tsirkin wrote:
>>>>> On Wed, Jan 31, 2018 at 06:40:59PM -0200, Eduardo Habkost wrote:
>>>>>> On Wed, Jan 17, 2018 at 11:54:18AM +0200, Marcel Apfelbaum wrote:
>>>>>>> Currently only file backed memory backend can
>>>>>>> be created with a "share" flag in order to allow
>>>>>>> sharing guest RAM with other processes in the host.
>>>>>>>
>>>>>>> Add the "share" flag also to RAM Memory Backend
>>>>>>> in order to allow remapping parts of the guest RAM
>>>>>>> to different host virtual addresses. This is needed
>>>>>>> by the RDMA devices in order to remap non-contiguous
>>>>>>> QEMU virtual addresses to a contiguous virtual address range.
>>>>>>>
>>>>>>
>>>>>> Why do we need to make this configurable?  Would anything break
>>>>>> if MAP_SHARED was always used if possible?
>>>>>
>>>>> See Documentation/vm/numa_memory_policy.txt for a list
>>>>> of complications.
>>>>
>>>> Ew.
>>>>
>>>>>
>>>>> Maybe we should more of an effort to detect and report these
>>>>> issues.
>>>>
>>>> Probably.  Having other features breaking silently when using
>>>> pvrdma doesn't sound good.  We must at least document those
>>>> problems in the documentation for memory-backend-ram.
>>>>
>>>> BTW, what's the root cause for requiring HVAs in the buffer?
>>>
>>> It's a side effect of the kernel/userspace API which always wants
>>> a single HVA/len pair to map memory for the application.
>>>
>>>
>>
>> Hi Eduardo and Michael,
>>
>>>>  Can
>>>> this be fixed?
>>>
>>> I think yes.  It'd need to be a kernel patch for the RDMA subsystem
>>> mapping an s/g list with actual memory. The HVA/len pair would then just
>>> be used to refer to the region, without creating the two mappings.
>>>
>>> Something like splitting the register mr into
>>>
>>> mr = create mr (va/len) - allocate a handle and record the va/len
>>>
>>> addmemory(mr, offset, hva, len) - pin memory
>>>
>>> register mr - pass it to HW
>>>
>>> As a nice side effect we won't burn so much virtual address space.
>>>
>>
>> We would still need a contiguous virtual address space range (for post-send)
>> which we don't have since guest contiguous virtual address space
>> will always end up as non-contiguous host virtual address space.
> 
> It just needs to be contiguous in the HCA virtual address space.
> Software never accesses through this pointer.
> In other words - basically expose register physical mr to userspace.
> 
> 
>>
>> I am not sure the RDMA HW can handle a large VA with holes.
>>
>> An alternative would be 0-based MR, QEMU intercepts the post-send
>> operations and can substract the guest VA base address.
>> However I didn't see the implementation in kernel for 0 based MRs
>> and also the RDMA maintainer said it would work for local keys
>> and not for remote keys.
>>
>>> This will fix rdma with hugetlbfs as well which is currently broken.
>>>
>>>
>>
>> There is already a discussion on the linux-rdma list:
>>     https://www.spinics.net/lists/linux-rdma/msg60079.html
>> But it will take some (actually a lot of) time, we are currently talking about
>> a possible API.
> 
> You probably need to pass the s/g piece by piece since it might exceed
> any reasonable array size.

Right. They say the new API is ioctl based but so this is not a limitation.
We proposed also a bitmap representation of a large range,
but what we really need is what you mentioned:
  to pass the Guest VA directly to reg_mr.

Thanks,
Marcel

> 
>> And it does not solve the re-mapping...
>>
>> Thanks,
>> Marcel
> 
> Haven't read through that discussion. But at least what I posted solves
> it since you do not need it contiguous in HVA any longer.
> 
>>>> -- 
>>>> Eduardo

[Qemu-devel] [PATCH V8 2/4] docs: add pvrdma device documentation.

[Marcel Apfelbaum]

Signed-off-by: Marcel Apfelbaum <marcel@redhat.com>
Signed-off-by: Yuval Shaia <yuval.shaia@oracle.com>
Reviewed-by: Shamir Rabinovitch <shamir.rabinovitch@oracle.com>
---
 docs/pvrdma.txt | 255 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 255 insertions(+)
 create mode 100644 docs/pvrdma.txt

diff --git a/docs/pvrdma.txt b/docs/pvrdma.txt
new file mode 100644
index 0000000000..5599318159
--- /dev/null
+++ b/docs/pvrdma.txt
@@ -0,0 +1,255 @@
+Paravirtualized RDMA Device (PVRDMA)
+====================================
+
+
+1. Description
+===============
+PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device.
+It works with its Linux Kernel driver AS IS, no need for any special guest
+modifications.
+
+While it complies with the VMware device, it can also communicate with bare
+metal RDMA-enabled machines and does not require an RDMA HCA in the host, it
+can work with Soft-RoCE (rxe).
+
+It does not require the whole guest RAM to be pinned allowing memory
+over-commit and, even if not implemented yet, migration support will be
+possible with some HW assistance.
+
+A project presentation accompany this document:
+- http://events.linuxfoundation.org/sites/events/files/slides/lpc-2017-pvrdma-marcel-apfelbaum-yuval-shaia.pdf
+
+
+
+2. Setup
+========
+
+
+2.1 Guest setup
+===============
+Fedora 27+ kernels work out of the box, older distributions
+require updating the kernel to 4.14 to include the pvrdma driver.
+
+However the libpvrdma library needed by User Level Software is still
+not available as part of the distributions, so the rdma-core library
+needs to be compiled and optionally installed.
+
+Please follow the instructions at:
+  https://github.com/linux-rdma/rdma-core.git
+
+
+2.2 Host Setup
+==============
+The pvrdma backend is an ibdevice interface that can be exposed
+either by a Soft-RoCE(rxe) device on machines with no RDMA device,
+or an HCA SRIOV function(VF/PF).
+Note that ibdevice interfaces can't be shared between pvrdma devices,
+each one requiring a separate instance (rxe or SRIOV VF).
+
+
+2.2.1 Soft-RoCE backend(rxe)
+===========================
+A stable version of rxe is required, Fedora 27+ or a Linux
+Kernel 4.14+ is preferred.
+
+The rdma_rxe module is part of the Linux Kernel but not loaded by default.
+Install the User Level library (librxe) following the instructions from:
+https://github.com/SoftRoCE/rxe-dev/wiki/rxe-dev:-Home
+
+Associate an ETH interface with rxe by running:
+   rxe_cfg add eth0
+An rxe0 ibdevice interface will be created and can be used as pvrdma backend.
+
+
+2.2.2 RDMA device Virtual Function backend
+==========================================
+Nothing special is required, the pvrdma device can work not only with
+Ethernet Links, but also Infinibands Links.
+All is needed is an ibdevice with an active port, for Mellanox cards
+will be something like mlx5_6 which can be the backend.
+
+
+2.2.3 QEMU setup
+================
+Configure QEMU with --enable-rdma flag, installing
+the required RDMA libraries.
+
+
+
+3. Usage
+========
+Currently the device is working only with memory backed RAM
+and it must be mark as "shared":
+   -m 1G \
+   -object memory-backend-ram,id=mb1,size=1G,share \
+   -numa node,memdev=mb1 \
+
+The pvrdma device is composed of two functions:
+ - Function 0 is a vmxnet Ethernet Device which is redundant in Guest
+   but is required to pass the ibdevice GID using its MAC.
+   Examples:
+     For an rxe backend using eth0 interface it will use its mac:
+       -device vmxnet3,addr=<slot>.0,multifunction=on,mac=<eth0 MAC>
+     For an SRIOV VF, we take the Ethernet Interface exposed by it:
+       -device vmxnet3,multifunction=on,mac=<RoCE eth MAC>
+ - Function 1 is the actual device:
+       -device pvrdma,addr=<slot>.1,backend-dev=<ibdevice>,backend-gid-idx=<gid>,backend-port=<port>
+   where the ibdevice can be rxe or RDMA VF (e.g. mlx5_4)
+ Note: Pay special attention that the GID at backend-gid-idx matches vmxnet's MAC.
+ The rules of conversion are part of the RoCE spec, but since manual conversion
+ is not required, spotting problems is not hard:
+    Example: GID: fe80:0000:0000:0000:7efe:90ff:fecb:743a
+             MAC: 7c:fe:90:cb:74:3a
+    Note the difference between the first byte of the MAC and the GID.
+
+
+
+4. Implementation details
+=========================
+
+
+4.1 Overview
+============
+The device acts like a proxy between the Guest Driver and the host
+ibdevice interface.
+On configuration path:
+ - For every hardware resource request (PD/QP/CQ/...) the pvrdma will request
+   a resource from the backend interface, maintaining a 1-1 mapping
+   between the guest and host.
+On data path:
+ - Every post_send/receive received from the guest will be converted into
+   a post_send/receive for the backend. The buffers data will not be touched
+   or copied resulting in near bare-metal performance for large enough buffers.
+ - Completions from the backend interface will result in completions for
+   the pvrdma device.
+
+
+4.2 PCI BARs
+============
+PCI Bars:
+	BAR 0 - MSI-X
+        MSI-X vectors:
+		(0) Command - used when execution of a command is completed.
+		(1) Async - not in use.
+		(2) Completion - used when a completion event is placed in
+		  device's CQ ring.
+	BAR 1 - Registers
+        --------------------------------------------------------
+        | VERSION |  DSR | CTL | REQ | ERR |  ICR | IMR  | MAC |
+        --------------------------------------------------------
+		DSR - Address of driver/device shared memory used
+              for the command channel, used for passing:
+			    - General info such as driver version
+			    - Address of 'command' and 'response'
+			    - Address of async ring
+			    - Address of device's CQ ring
+			    - Device capabilities
+		CTL - Device control operations (activate, reset etc)
+		IMG - Set interrupt mask
+		REQ - Command execution register
+		ERR - Operation status
+
+	BAR 2 - UAR
+        ---------------------------------------------------------
+        | QP_NUM  | SEND/RECV Flag ||  CQ_NUM |   ARM/POLL Flag |
+        ---------------------------------------------------------
+		- Offset 0 used for QP operations (send and recv)
+		- Offset 4 used for CQ operations (arm and poll)
+
+
+4.3 Major flows
+===============
+
+4.3.1 Create CQ
+===============
+    - Guest driver
+        - Allocates pages for CQ ring
+        - Creates page directory (pdir) to hold CQ ring's pages
+        - Initializes CQ ring
+        - Initializes 'Create CQ' command object (cqe, pdir etc)
+        - Copies the command to 'command' address
+        - Writes 0 into REQ register
+    - Device
+        - Reads the request object from the 'command' address
+        - Allocates CQ object and initialize CQ ring based on pdir
+        - Creates the backend CQ
+        - Writes operation status to ERR register
+        - Posts command-interrupt to guest
+    - Guest driver
+        - Reads the HW response code from ERR register
+
+4.3.2 Create QP
+===============
+    - Guest driver
+        - Allocates pages for send and receive rings
+        - Creates page directory(pdir) to hold the ring's pages
+        - Initializes 'Create QP' command object (max_send_wr,
+          send_cq_handle, recv_cq_handle, pdir etc)
+        - Copies the object to 'command' address
+        - Write 0 into REQ register
+    - Device
+        - Reads the request object from 'command' address
+        - Allocates the QP object and initialize
+            - Send and recv rings based on pdir
+            - Send and recv ring state
+        - Creates the backend QP
+        - Writes the operation status to ERR register
+        - Posts command-interrupt to guest
+    - Guest driver
+        - Reads the HW response code from ERR register
+
+4.3.3 Post receive
+==================
+    - Guest driver
+        - Initializes a wqe and place it on recv ring
+        - Write to qpn|qp_recv_bit (31) to QP offset in UAR
+    - Device
+        - Extracts qpn from UAR
+        - Walks through the ring and does the following for each wqe
+            - Prepares the backend CQE context to be used when
+              receiving completion from backend (wr_id, op_code, emu_cq_num)
+            - For each sge prepares backend sge
+            - Calls backend's post_recv
+
+4.3.4 Process backend events
+============================
+    - Done by a dedicated thread used to process backend events;
+      at initialization is attached to the device and creates
+      the communication channel.
+    - Thread main loop:
+        - Polls for completions
+        - Extracts QEMU _cq_num, wr_id and op_code from context
+        - Writes CQE to CQ ring
+        - Writes CQ number to device CQ
+        - Sends completion-interrupt to guest
+        - Deallocates context
+        - Acks the event to backend
+
+
+
+5. Limitations
+==============
+- The device obviously is limited by the Guest Linux Driver features implementation
+  of the VMware device API.
+- Memory registration mechanism requires mremap for every page in the buffer in order
+  to map it to a contiguous virtual address range. Since this is not the data path
+  it should not matter much. If the default max mr size is increased, be aware that
+  memory registration can take up to 0.5 seconds for 1GB of memory.
+- The device requires target page size to be the same as the host page size,
+  otherwise it will fail to init.
+- QEMU cannot map guest RAM from a file descriptor if a pvrdma device is attached,
+  so it can't work with huge pages. The limitation will be addressed in the future,
+  however QEMU allocates Guest RAM with MADV_HUGEPAGE so if there are enough huge
+  pages available, QEMU will use them. QEMU will fail to init if the requirements
+  are not met.
+
+
+
+6. Performance
+==============
+By design the pvrdma device exits on each post-send/receive, so for small buffers
+the performance is affected; however for medium buffers it will became close to
+bare metal and from 1MB buffers and  up it reaches bare metal performance.
+(tested with 2 VMs, the pvrdma devices connected to 2 VFs of the same device)
+
+All the above assumes no memory registration is done on data path.
-- 
2.13.5

[Qemu-devel] [PATCH V8 3/4] pvrdma: initial implementation

[Marcel Apfelbaum]

From: Yuval Shaia <yuval.shaia@oracle.com>

PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device.
It works with its Linux Kernel driver AS IS, no need for any special guest
modifications.

While it complies with the VMware device, it can also communicate with bare
metal RDMA-enabled machines and does not require an RDMA HCA in the host, it
can work with Soft-RoCE (rxe).

It does not require the whole guest RAM to be pinned allowing memory
over-commit and, even if not implemented yet, migration support will be
possible with some HW assistance.

Signed-off-by: Yuval Shaia <yuval.shaia@oracle.com>
Signed-off-by: Marcel Apfelbaum <marcel@redhat.com>
---
 Makefile.objs                 |   2 +
 configure                     |   9 +-
 hw/Makefile.objs              |   1 +
 hw/rdma/Makefile.objs         |   6 +
 hw/rdma/rdma_backend.c        | 815 ++++++++++++++++++++++++++++++++++++++++++
 hw/rdma/rdma_backend.h        |  92 +++++
 hw/rdma/rdma_backend_defs.h   |  62 ++++
 hw/rdma/rdma_rm.c             | 619 ++++++++++++++++++++++++++++++++
 hw/rdma/rdma_rm.h             |  69 ++++
 hw/rdma/rdma_rm_defs.h        | 106 ++++++
 hw/rdma/rdma_utils.c          |  52 +++
 hw/rdma/rdma_utils.h          |  43 +++
 hw/rdma/trace-events          |   5 +
 hw/rdma/vmw/pvrdma.h          | 122 +++++++
 hw/rdma/vmw/pvrdma_cmd.c      | 679 +++++++++++++++++++++++++++++++++++
 hw/rdma/vmw/pvrdma_dev_api.h  | 602 +++++++++++++++++++++++++++++++
 hw/rdma/vmw/pvrdma_dev_ring.c | 139 +++++++
 hw/rdma/vmw/pvrdma_dev_ring.h |  42 +++
 hw/rdma/vmw/pvrdma_ib_verbs.h | 433 ++++++++++++++++++++++
 hw/rdma/vmw/pvrdma_main.c     | 671 ++++++++++++++++++++++++++++++++++
 hw/rdma/vmw/pvrdma_qp_ops.c   | 212 +++++++++++
 hw/rdma/vmw/pvrdma_qp_ops.h   |  27 ++
 hw/rdma/vmw/pvrdma_ring.h     | 134 +++++++
 hw/rdma/vmw/trace-events      |   5 +
 hw/rdma/vmw/vmw_pvrdma-abi.h  | 311 ++++++++++++++++
 include/hw/pci/pci_ids.h      |   3 +
 26 files changed, 5257 insertions(+), 4 deletions(-)
 create mode 100644 hw/rdma/Makefile.objs
 create mode 100644 hw/rdma/rdma_backend.c
 create mode 100644 hw/rdma/rdma_backend.h
 create mode 100644 hw/rdma/rdma_backend_defs.h
 create mode 100644 hw/rdma/rdma_rm.c
 create mode 100644 hw/rdma/rdma_rm.h
 create mode 100644 hw/rdma/rdma_rm_defs.h
 create mode 100644 hw/rdma/rdma_utils.c
 create mode 100644 hw/rdma/rdma_utils.h
 create mode 100644 hw/rdma/trace-events
 create mode 100644 hw/rdma/vmw/pvrdma.h
 create mode 100644 hw/rdma/vmw/pvrdma_cmd.c
 create mode 100644 hw/rdma/vmw/pvrdma_dev_api.h
 create mode 100644 hw/rdma/vmw/pvrdma_dev_ring.c
 create mode 100644 hw/rdma/vmw/pvrdma_dev_ring.h
 create mode 100644 hw/rdma/vmw/pvrdma_ib_verbs.h
 create mode 100644 hw/rdma/vmw/pvrdma_main.c
 create mode 100644 hw/rdma/vmw/pvrdma_qp_ops.c
 create mode 100644 hw/rdma/vmw/pvrdma_qp_ops.h
 create mode 100644 hw/rdma/vmw/pvrdma_ring.h
 create mode 100644 hw/rdma/vmw/trace-events
 create mode 100644 hw/rdma/vmw/vmw_pvrdma-abi.h

diff --git a/Makefile.objs b/Makefile.objs
index c8b1bba593..85dde3c4a1 100644
--- a/Makefile.objs
+++ b/Makefile.objs
@@ -129,6 +129,8 @@ trace-events-subdirs += hw/block/dataplane
 trace-events-subdirs += hw/char
 trace-events-subdirs += hw/intc
 trace-events-subdirs += hw/net
+trace-events-subdirs += hw/rdma
+trace-events-subdirs += hw/rdma/vmw
 trace-events-subdirs += hw/virtio
 trace-events-subdirs += hw/audio
 trace-events-subdirs += hw/misc
diff --git a/configure b/configure
index 6d8c996c62..eb0ba343bb 100755
--- a/configure
+++ b/configure
@@ -1548,7 +1548,7 @@ disabled with --disable-FEATURE, default is enabled if available:
   kvm             KVM acceleration support
   hax             HAX acceleration support
   hvf             Hypervisor.framework acceleration support
-  rdma            RDMA-based migration support
+  rdma            Enable RDMA-based migration support and PVRDMA
   vde             support for vde network
   netmap          support for netmap network
   linux-aio       Linux AIO support
@@ -2875,15 +2875,16 @@ if test "$rdma" != "no" ; then
 #include <rdma/rdma_cma.h>
 int main(void) { return 0; }
 EOF
-  rdma_libs="-lrdmacm -libverbs"
+  rdma_libs="-lrdmacm -libverbs -libumad"
   if compile_prog "" "$rdma_libs" ; then
     rdma="yes"
+    libs_softmmu="$libs_softmmu $rdma_libs"
   else
     if test "$rdma" = "yes" ; then
         error_exit \
-            " OpenFabrics librdmacm/libibverbs not present." \
+            " OpenFabrics librdmacm/libibverbs/libibumad not present." \
             " Your options:" \
-            "  (1) Fast: Install infiniband packages from your distro." \
+            "  (1) Fast: Install infiniband packages (devel) from your distro." \
             "  (2) Cleanest: Install libraries from www.openfabrics.org" \
             "  (3) Also: Install softiwarp if you don't have RDMA hardware"
     fi
diff --git a/hw/Makefile.objs b/hw/Makefile.objs
index cf4cb2010b..6a0ffe0afd 100644
--- a/hw/Makefile.objs
+++ b/hw/Makefile.objs
@@ -18,6 +18,7 @@ devices-dirs-$(CONFIG_IPMI) += ipmi/
 devices-dirs-$(CONFIG_SOFTMMU) += isa/
 devices-dirs-$(CONFIG_SOFTMMU) += misc/
 devices-dirs-$(CONFIG_SOFTMMU) += net/
+devices-dirs-$(CONFIG_SOFTMMU) += rdma/
 devices-dirs-$(CONFIG_SOFTMMU) += nvram/
 devices-dirs-$(CONFIG_SOFTMMU) += pci/
 devices-dirs-$(CONFIG_PCI) += pci-bridge/ pci-host/
diff --git a/hw/rdma/Makefile.objs b/hw/rdma/Makefile.objs
new file mode 100644
index 0000000000..6c6272cfcc
--- /dev/null
+++ b/hw/rdma/Makefile.objs
@@ -0,0 +1,6 @@
+ifeq ($(CONFIG_RDMA),y)
+obj-$(CONFIG_PCI) += rdma_utils.o rdma_backend.o rdma_rm.o
+obj-$(CONFIG_PCI) += vmw/pvrdma_dev_ring.o vmw/pvrdma_cmd.o \
+                     vmw/pvrdma_qp_ops.o vmw/pvrdma_main.o
+endif
+
diff --git a/hw/rdma/rdma_backend.c b/hw/rdma/rdma_backend.c
new file mode 100644
index 0000000000..b26a1a01a1
--- /dev/null
+++ b/hw/rdma/rdma_backend.c
@@ -0,0 +1,815 @@
+/*
+ * QEMU paravirtual RDMA - Generic RDMA backend
+ *
+ * Copyright (C) 2018 Oracle
+ * Copyright (C) 2018 Red Hat Inc
+ *
+ * Authors:
+ *     Yuval Shaia <yuval.shaia@oracle.com>
+ *     Marcel Apfelbaum <marcel@redhat.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include <qemu/osdep.h>
+#include <qemu/error-report.h>
+#include <qapi/error.h>
+
+#include <infiniband/verbs.h>
+
+#include "trace.h"
+#include "rdma_utils.h"
+#include "rdma_rm.h"
+#include "rdma_backend.h"
+
+/* Vendor Errors */
+#define VENDOR_ERR_FAIL_BACKEND     0x201
+#define VENDOR_ERR_TOO_MANY_SGES    0x202
+#define VENDOR_ERR_NOMEM            0x203
+#define VENDOR_ERR_QP0              0x204
+#define VENDOR_ERR_NO_SGE           0x205
+#define VENDOR_ERR_MAD_SEND         0x206
+#define VENDOR_ERR_INVLKEY          0x207
+#define VENDOR_ERR_MR_SMALL         0x208
+
+typedef struct BackendCtx {
+    void *up_ctx;
+    uint64_t req_id;
+    bool is_tx_req;
+} BackendCtx;
+
+static void (*comp_handler)(int status, unsigned int vendor_err, void *ctx);
+
+static void dummy_comp_handler(int status, unsigned int vendor_err, void *ctx)
+{
+    pr_err("No completion handler is registered\n");
+}
+
+static void poll_cq(RdmaDeviceResources *rdma_dev_res, struct ibv_cq *ibcq,
+                    bool one_poll)
+{
+    int i, ne;
+    BackendCtx *bctx;
+    struct ibv_wc wc[2];
+
+    pr_dbg("Entering poll_cq loop on cq %p\n", ibcq);
+    do {
+        ne = ibv_poll_cq(ibcq, 2, wc);
+        if (ne == 0 && one_poll) {
+            pr_dbg("CQ is empty\n");
+            return;
+        }
+    } while (ne < 0);
+
+    pr_dbg("Got %d completion(s) from cq %p\n", ne, ibcq);
+
+    for (i = 0; i < ne; i++) {
+        pr_dbg("wr_id=0x%lx\n", wc[i].wr_id);
+        pr_dbg("status=%d\n", wc[i].status);
+
+        bctx = rdma_rm_get_cqe_ctx(rdma_dev_res, wc[i].wr_id);
+        if (unlikely(!bctx)) {
+            pr_dbg("Error: Failed to find ctx for req %ld\n", wc[i].wr_id);
+            continue;
+        }
+        pr_dbg("Processing %s CQE\n", bctx->is_tx_req ? "send" : "recv");
+
+        comp_handler(wc[i].status, wc[i].vendor_err, bctx->up_ctx);
+
+        rdma_rm_dealloc_cqe_ctx(rdma_dev_res, wc[i].wr_id);
+        free(bctx);
+    }
+}
+
+static void *comp_handler_thread(void *arg)
+{
+    RdmaBackendDev *backend_dev = (RdmaBackendDev *)arg;
+    int rc;
+    struct ibv_cq *ev_cq;
+    void *ev_ctx;
+
+    pr_dbg("Starting\n");
+
+    while (backend_dev->comp_thread.run) {
+        pr_dbg("Waiting for completion on channel %p\n", backend_dev->channel);
+        rc = ibv_get_cq_event(backend_dev->channel, &ev_cq, &ev_ctx);
+        pr_dbg("ibv_get_cq_event=%d\n", rc);
+        if (unlikely(rc)) {
+            pr_dbg("---> ibv_get_cq_event (%d)\n", rc);
+            continue;
+        }
+
+        if (unlikely(ibv_req_notify_cq(ev_cq, 0))) {
+            pr_dbg("---> ibv_req_notify_cq\n");
+        }
+
+        poll_cq(backend_dev->rdma_dev_res, ev_cq, false);
+
+        ibv_ack_cq_events(ev_cq, 1);
+    }
+
+    pr_dbg("Going down\n");
+    /* TODO: Post cqe for all remaining buffs that were posted */
+
+    return NULL;
+}
+
+void rdma_backend_register_comp_handler(void (*handler)(int status,
+                                        unsigned int vendor_err, void *ctx))
+{
+    comp_handler = handler;
+}
+
+void rdma_backend_unregister_comp_handler(void)
+{
+    rdma_backend_register_comp_handler(dummy_comp_handler);
+}
+
+int rdma_backend_query_port(RdmaBackendDev *backend_dev,
+                            struct ibv_port_attr *port_attr)
+{
+    int rc;
+
+    memset(port_attr, 0, sizeof(*port_attr));
+
+    rc = ibv_query_port(backend_dev->context, backend_dev->port_num, port_attr);
+    if (rc) {
+        pr_dbg("Error %d from ibv_query_port\n", rc);
+        return -EIO;
+    }
+
+    return 0;
+}
+
+void rdma_backend_poll_cq(RdmaDeviceResources *rdma_dev_res, RdmaBackendCQ *cq)
+{
+    poll_cq(rdma_dev_res, cq->ibcq, true);
+}
+
+static GHashTable *ah_hash;
+
+static struct ibv_ah *create_ah(RdmaBackendDev *backend_dev, struct ibv_pd *pd,
+                                uint8_t sgid_idx, union ibv_gid *dgid)
+{
+    GBytes *ah_key = g_bytes_new(dgid, sizeof(*dgid));
+    struct ibv_ah *ah = g_hash_table_lookup(ah_hash, ah_key);
+
+    if (ah) {
+        trace_create_ah_cache_hit(be64_to_cpu(dgid->global.subnet_prefix),
+                                  be64_to_cpu(dgid->global.interface_id));
+        g_bytes_unref(ah_key);
+    } else {
+        struct ibv_ah_attr ah_attr = {
+            .is_global     = 1,
+            .port_num      = backend_dev->port_num,
+            .grh.hop_limit = 1,
+        };
+
+        ah_attr.grh.dgid = *dgid;
+        ah_attr.grh.sgid_index = sgid_idx;
+
+        ah = ibv_create_ah(pd, &ah_attr);
+        if (ah) {
+            g_hash_table_insert(ah_hash, ah_key, ah);
+        } else {
+            pr_dbg("ibv_create_ah failed for gid <%lx %lx>\n",
+                    be64_to_cpu(dgid->global.subnet_prefix),
+                    be64_to_cpu(dgid->global.interface_id));
+        }
+
+        trace_create_ah_cache_miss(be64_to_cpu(dgid->global.subnet_prefix),
+                                   be64_to_cpu(dgid->global.interface_id));
+    }
+
+    return ah;
+}
+
+static void destroy_ah_hash_key(gpointer data)
+{
+    g_bytes_unref(data);
+}
+
+static void destroy_ah_hast_data(gpointer data)
+{
+    struct ibv_ah *ah = data;
+    ibv_destroy_ah(ah);
+}
+
+static void ah_cache_init(void)
+{
+    ah_hash = g_hash_table_new_full(g_bytes_hash, g_bytes_equal,
+                                    destroy_ah_hash_key, destroy_ah_hast_data);
+}
+
+static int build_host_sge_array(RdmaDeviceResources *rdma_dev_res,
+                                struct ibv_sge *dsge, struct ibv_sge *ssge,
+                                uint8_t num_sge)
+{
+    RdmaRmMR *mr;
+    int ssge_idx;
+    int ret = 0;
+
+    pr_dbg("num_sge=%d\n", num_sge);
+
+    for (ssge_idx = 0; ssge_idx < num_sge; ssge_idx++) {
+        mr = rdma_rm_get_mr(rdma_dev_res, ssge[ssge_idx].lkey);
+        if (unlikely(!mr)) {
+            ret = VENDOR_ERR_INVLKEY | ssge[ssge_idx].lkey;
+            pr_dbg("Invalid lkey 0x%x\n", ssge[ssge_idx].lkey);
+            goto out;
+        }
+
+        dsge->addr = mr->user_mr.host_virt + ssge[ssge_idx].addr -
+                     mr->user_mr.guest_start;
+        dsge->length = ssge[ssge_idx].length;
+        dsge->lkey = rdma_backend_mr_lkey(&mr->backend_mr);
+
+        pr_dbg("ssge->addr=0x%lx\n", (uint64_t)ssge[ssge_idx].addr);
+        pr_dbg("dsge->addr=0x%lx\n", dsge->addr);
+        pr_dbg("dsge->lenght=%d\n", dsge->length);
+        pr_dbg("dsge->lkey=0x%x\n", dsge->lkey);
+
+        dsge++;
+    }
+
+out:
+    return ret;
+}
+
+void rdma_backend_post_send(RdmaBackendDev *backend_dev,
+                            RdmaDeviceResources *rdma_dev_res,
+                            RdmaBackendQP *qp, uint8_t qp_type,
+                            struct ibv_sge *sge, uint32_t num_sge,
+                            union ibv_gid *dgid, uint32_t dqpn,
+                            uint32_t dqkey, void *ctx)
+{
+    BackendCtx *bctx;
+    struct ibv_sge new_sge[MAX_SGE];
+    uint32_t bctx_id;
+    int rc;
+    struct ibv_send_wr wr = {0}, *bad_wr;
+
+    if (!qp->ibqp && qp_type == 0) {
+        pr_dbg("QP0 is not supported\n");
+        comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_QP0, ctx);
+        return;
+    }
+
+    pr_dbg("num_sge=%d\n", num_sge);
+
+    if (!qp->ibqp && qp_type == 1) {
+        pr_dbg("QP1\n");
+        comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_MAD_SEND, ctx);
+    }
+
+    if (!num_sge) {
+        pr_dbg("num_sge=0\n");
+        comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_NO_SGE, ctx);
+        return;
+    }
+
+    bctx = malloc(sizeof(*bctx));
+    if (unlikely(!bctx)) {
+        pr_dbg("Failed to allocate request ctx\n");
+        comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_NOMEM, ctx);
+        return;
+    }
+    memset(bctx, 0, sizeof(*bctx));
+
+    bctx->up_ctx = ctx;
+    bctx->is_tx_req = 1;
+
+    rc = rdma_rm_alloc_cqe_ctx(rdma_dev_res, &bctx_id, bctx);
+    if (unlikely(rc)) {
+        pr_dbg("Failed to allocate cqe_ctx\n");
+        comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_NOMEM, ctx);
+        goto out_free_bctx;
+    }
+
+    rc = build_host_sge_array(rdma_dev_res, new_sge, sge, num_sge);
+    if (rc) {
+        pr_dbg("Error: Failed to build host SGE array\n");
+        comp_handler(IBV_WC_GENERAL_ERR, rc, ctx);
+        goto out_dealloc_cqe_ctx;
+    }
+
+    if (qp_type == IBV_QPT_UD) {
+        wr.wr.ud.ah = create_ah(backend_dev, qp->ibpd,
+                                backend_dev->backend_gid_idx, dgid);
+        wr.wr.ud.remote_qpn = dqpn;
+        wr.wr.ud.remote_qkey = dqkey;
+    }
+
+    wr.num_sge = num_sge;
+    wr.opcode = IBV_WR_SEND;
+    wr.send_flags = IBV_SEND_SIGNALED;
+    wr.sg_list = &new_sge[0];
+    wr.wr_id = bctx_id;
+
+    rc = ibv_post_send(qp->ibqp, &wr, &bad_wr);
+    pr_dbg("ibv_post_send=%d\n", rc);
+    if (rc) {
+        pr_dbg("Fail (%d, %d) to post send WQE to qpn %d\n", rc, errno,
+                qp->ibqp->qp_num);
+        comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_FAIL_BACKEND, ctx);
+        goto out_dealloc_cqe_ctx;
+    }
+
+    return;
+
+out_dealloc_cqe_ctx:
+    rdma_rm_dealloc_cqe_ctx(rdma_dev_res, bctx_id);
+
+out_free_bctx:
+    free(bctx);
+}
+
+void rdma_backend_post_recv(RdmaBackendDev *backend_dev,
+                            RdmaDeviceResources *rdma_dev_res,
+                            RdmaBackendQP *qp, uint8_t qp_type,
+                            struct ibv_sge *sge, uint32_t num_sge, void *ctx)
+{
+    BackendCtx *bctx;
+    struct ibv_sge new_sge[MAX_SGE];
+    uint32_t bctx_id;
+    int rc;
+    struct ibv_recv_wr wr = {0}, *bad_wr;
+
+    if (!qp->ibqp && qp_type == 0) {
+        pr_dbg("QP0 is not supported\n");
+        comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_QP0, ctx);
+        return;
+    }
+
+    pr_dbg("num_sge=%d\n", num_sge);
+
+    if (!qp->ibqp && qp_type == 1) {
+        pr_dbg("QP1\n");
+        comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_MAD_SEND, ctx);
+        return;
+    }
+
+    if (!num_sge) {
+        pr_dbg("num_sge=0\n");
+        comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_NO_SGE, ctx);
+        return;
+    }
+
+    bctx = malloc(sizeof(*bctx));
+    if (unlikely(!bctx)) {
+        pr_dbg("Failed to allocate request ctx\n");
+        comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_NOMEM, ctx);
+        return;
+    }
+    memset(bctx, 0, sizeof(*bctx));
+
+    bctx->up_ctx = ctx;
+    bctx->is_tx_req = 0;
+
+    rc = rdma_rm_alloc_cqe_ctx(rdma_dev_res, &bctx_id, bctx);
+    if (unlikely(rc)) {
+        pr_dbg("Failed to allocate cqe_ctx\n");
+        comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_NOMEM, ctx);
+        goto out_free_bctx;
+    }
+
+    rc = build_host_sge_array(rdma_dev_res, new_sge, sge, num_sge);
+    if (rc) {
+        pr_dbg("Error: Failed to build host SGE array\n");
+        comp_handler(IBV_WC_GENERAL_ERR, rc, ctx);
+        goto out_dealloc_cqe_ctx;
+    }
+
+    wr.num_sge = num_sge;
+    wr.sg_list = &new_sge[0];
+    wr.wr_id = bctx_id;
+    rc = ibv_post_recv(qp->ibqp, &wr, &bad_wr);
+    pr_dbg("ibv_post_recv=%d\n", rc);
+    if (rc) {
+        pr_dbg("Fail (%d, %d) to post recv WQE to qpn %d\n", rc, errno,
+                qp->ibqp->qp_num);
+        comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_FAIL_BACKEND, ctx);
+        goto out_dealloc_cqe_ctx;
+    }
+
+    return;
+
+out_dealloc_cqe_ctx:
+    rdma_rm_dealloc_cqe_ctx(rdma_dev_res, bctx_id);
+
+out_free_bctx:
+    free(bctx);
+}
+
+int rdma_backend_create_pd(RdmaBackendDev *backend_dev, RdmaBackendPD *pd)
+{
+    pd->ibpd = ibv_alloc_pd(backend_dev->context);
+
+    return pd->ibpd ? 0 : -EIO;
+}
+
+void rdma_backend_destroy_pd(RdmaBackendPD *pd)
+{
+    if (pd->ibpd) {
+        ibv_dealloc_pd(pd->ibpd);
+    }
+}
+
+int rdma_backend_create_mr(RdmaBackendMR *mr, RdmaBackendPD *pd, uint64_t addr,
+                           size_t length, int access)
+{
+    pr_dbg("addr=0x%lx\n", addr);
+    pr_dbg("len=%ld\n", length);
+    mr->ibpd = pd->ibpd;
+    mr->ibmr = ibv_reg_mr(mr->ibpd, (void *)addr, length, access);
+
+    if (mr->ibmr) {
+        pr_dbg("lkey=0x%x\n", mr->ibmr->lkey);
+        pr_dbg("rkey=0x%x\n", mr->ibmr->rkey);
+    }
+
+    return mr->ibmr ? 0 : -EIO;
+}
+
+void rdma_backend_destroy_mr(RdmaBackendMR *mr)
+{
+    if (mr->ibmr) {
+        ibv_dereg_mr(mr->ibmr);
+    }
+}
+
+int rdma_backend_create_cq(RdmaBackendDev *backend_dev, RdmaBackendCQ *cq,
+                           int cqe)
+{
+    pr_dbg("cqe=%d\n", cqe);
+
+    pr_dbg("dev->channel=%p\n", backend_dev->channel);
+    cq->ibcq = ibv_create_cq(backend_dev->context, cqe + 1, NULL,
+                             backend_dev->channel, 0);
+
+    if (cq->ibcq) {
+        if (ibv_req_notify_cq(cq->ibcq, 0)) {
+            pr_dbg("---> ibv_req_notify_cq\n");
+        }
+    }
+
+    cq->backend_dev = backend_dev;
+
+    return cq->ibcq ? 0 : -EIO;
+}
+
+void rdma_backend_destroy_cq(RdmaBackendCQ *cq)
+{
+    if (cq->ibcq) {
+        ibv_req_notify_cq(cq->ibcq, 0);
+
+        /* Cleanup the queue before destruction */
+        poll_cq(cq->backend_dev->rdma_dev_res, cq->ibcq, false);
+
+        ibv_destroy_cq(cq->ibcq);
+    }
+}
+
+int rdma_backend_create_qp(RdmaBackendQP *qp, uint8_t qp_type,
+                           RdmaBackendPD *pd, RdmaBackendCQ *scq,
+                           RdmaBackendCQ *rcq, uint32_t max_send_wr,
+                           uint32_t max_recv_wr, uint32_t max_send_sge,
+                           uint32_t max_recv_sge)
+{
+    struct ibv_qp_init_attr attr = {0};
+
+    qp->ibqp = 0;
+    pr_dbg("qp_type=%d\n", qp_type);
+
+    if (qp_type == 0) {
+        pr_dbg("QP0 is not supported\n");
+        return -EPERM;
+    }
+
+    if (qp_type == 1) {
+        pr_dbg("QP1\n");
+        return 0;
+    }
+
+    attr.qp_type = qp_type;
+    attr.send_cq = scq->ibcq;
+    attr.recv_cq = rcq->ibcq;
+    attr.cap.max_send_wr = max_send_wr;
+    attr.cap.max_recv_wr = max_recv_wr;
+    attr.cap.max_send_sge = max_send_sge;
+    attr.cap.max_recv_sge = max_recv_sge;
+
+    pr_dbg("max_send_wr=%d\n", max_send_wr);
+    pr_dbg("max_recv_wr=%d\n", max_recv_wr);
+    pr_dbg("max_send_sge=%d\n", max_send_sge);
+    pr_dbg("max_recv_sge=%d\n", max_recv_sge);
+
+    qp->ibpd = pd->ibpd;
+    qp->ibqp = ibv_create_qp(qp->ibpd, &attr);
+
+    if (likely(!qp->ibqp)) {
+        pr_dbg("Error from ibv_create_qp\n");
+        return -EIO;
+    }
+
+    /* TODO: Query QP to get max_inline_data and save it to be used in send */
+
+    pr_dbg("qpn=0x%x\n", qp->ibqp->qp_num);
+
+    return 0;
+}
+
+int rdma_backend_qp_state_init(RdmaBackendDev *backend_dev, RdmaBackendQP *qp,
+                               uint8_t qp_type, uint32_t qkey)
+{
+    struct ibv_qp_attr attr = {0};
+    int rc, attr_mask;
+
+    pr_dbg("qpn=0x%x\n", qp->ibqp->qp_num);
+    pr_dbg("sport_num=%d\n", backend_dev->port_num);
+
+    attr_mask = IBV_QP_STATE | IBV_QP_PKEY_INDEX | IBV_QP_PORT;
+    attr.qp_state        = IBV_QPS_INIT;
+    attr.pkey_index      = 0;
+    attr.port_num        = backend_dev->port_num;
+    if (qp_type == IBV_QPT_RC) {
+        attr_mask |= IBV_QP_ACCESS_FLAGS;
+    }
+    if (qp_type == IBV_QPT_UD) {
+        attr.qkey = qkey;
+        if (qkey) {
+            attr_mask |= IBV_QP_QKEY;
+        }
+    }
+
+    rc = ibv_modify_qp(qp->ibqp, &attr, attr_mask);
+    if (rc) {
+        pr_dbg("Error %d from ibv_modify_qp\n", rc);
+        return -EIO;
+    }
+
+    return 0;
+}
+
+int rdma_backend_qp_state_rtr(RdmaBackendDev *backend_dev, RdmaBackendQP *qp,
+                              uint8_t qp_type, union ibv_gid *dgid,
+                              uint32_t dqpn, uint32_t rq_psn, uint32_t qkey)
+{
+    struct ibv_qp_attr attr = {0};
+    union ibv_gid ibv_gid = {
+        .global.interface_id = dgid->global.interface_id,
+        .global.subnet_prefix = dgid->global.subnet_prefix
+    };
+    int rc, attr_mask = 0;
+
+    attr.qp_state = IBV_QPS_RTR;
+    attr_mask = IBV_QP_STATE;
+
+    if (qp_type == IBV_QPT_RC) {
+        pr_dbg("dgid=0x%lx,%lx\n", be64_to_cpu(ibv_gid.global.subnet_prefix),
+               be64_to_cpu(ibv_gid.global.interface_id));
+        pr_dbg("dqpn=0x%x\n", dqpn);
+        pr_dbg("sgid_idx=%d\n", backend_dev->backend_gid_idx);
+        pr_dbg("sport_num=%d\n", backend_dev->port_num);
+        pr_dbg("rq_psn=0x%x\n", rq_psn);
+
+        attr.path_mtu               = IBV_MTU_1024;
+        attr.dest_qp_num            = dqpn;
+        attr.max_dest_rd_atomic     = 1;
+        attr.min_rnr_timer          = 12;
+        attr.ah_attr.port_num       = backend_dev->port_num;
+        attr.ah_attr.is_global      = 1;
+        attr.ah_attr.grh.hop_limit  = 1;
+        attr.ah_attr.grh.dgid       = ibv_gid;
+        attr.ah_attr.grh.sgid_index = backend_dev->backend_gid_idx;
+        attr.rq_psn                 = rq_psn;
+
+        attr_mask |= IBV_QP_AV | IBV_QP_PATH_MTU | IBV_QP_DEST_QPN |
+                     IBV_QP_RQ_PSN | IBV_QP_MAX_DEST_RD_ATOMIC |
+                     IBV_QP_MIN_RNR_TIMER;
+    }
+
+    if (qp_type == IBV_QPT_UD) {
+        pr_dbg("qkey=0x%x\n", qkey);
+        attr.qkey = qkey;
+        if (qkey) {
+            attr_mask |= IBV_QP_QKEY;
+        }
+    }
+
+    rc = ibv_modify_qp(qp->ibqp, &attr, attr_mask);
+    if (rc) {
+        pr_dbg("Error %d from ibv_modify_qp\n", rc);
+        return -EIO;
+    }
+
+    return 0;
+}
+
+int rdma_backend_qp_state_rts(RdmaBackendQP *qp, uint8_t qp_type,
+                              uint32_t sq_psn, uint32_t qkey)
+{
+    struct ibv_qp_attr attr = {0};
+    int rc, attr_mask = 0;
+
+    pr_dbg("qpn=0x%x\n", qp->ibqp->qp_num);
+    pr_dbg("sq_psn=0x%x\n", sq_psn);
+
+    attr_mask |= IBV_QP_SQ_PSN;
+    attr.sq_psn = sq_psn;
+
+    attr.qp_state = IBV_QPS_RTS;
+    attr_mask = IBV_QP_STATE | IBV_QP_SQ_PSN;
+
+    if (qp_type == IBV_QPT_RC) {
+        attr.timeout       = 14;
+        attr.retry_cnt     = 7;
+        attr.rnr_retry     = 7;
+        attr.max_rd_atomic = 1;
+
+        attr_mask |= IBV_QP_TIMEOUT | IBV_QP_RETRY_CNT | IBV_QP_RNR_RETRY |
+                     IBV_QP_MAX_QP_RD_ATOMIC;
+    }
+
+    if (qp_type == IBV_QPT_UD) {
+        attr.qkey = qkey;
+        if (qkey) {
+            attr_mask |= IBV_QP_QKEY;
+        }
+    }
+
+    rc = ibv_modify_qp(qp->ibqp, &attr, attr_mask);
+    if (rc) {
+        pr_dbg("Error %d from ibv_modify_qp\n", rc);
+        return -EIO;
+    }
+
+    return 0;
+}
+
+void rdma_backend_destroy_qp(RdmaBackendQP *qp)
+{
+    if (qp->ibqp) {
+        ibv_destroy_qp(qp->ibqp);
+    }
+}
+
+#define CHK_ATTR(req, dev, member, fmt) ({ \
+    pr_dbg("%s="fmt","fmt"\n", #member, dev.member, req->member); \
+    if (req->member > dev.member) { \
+        warn_report("Setting of %s to 0x%lx higher than host device capability 0x%lx", \
+                    #member, (uint64_t)req->member, (uint64_t)dev.member); \
+        req->member = dev.member; \
+    } \
+    pr_dbg("%s="fmt"\n", #member, req->member); })
+
+static int init_device_caps(RdmaBackendDev *backend_dev,
+                            struct ibv_device_attr *dev_attr)
+{
+    memset(&backend_dev->dev_attr, 0, sizeof(backend_dev->dev_attr));
+
+    if (ibv_query_device(backend_dev->context, &backend_dev->dev_attr)) {
+        return -EIO;
+    }
+
+    CHK_ATTR(dev_attr, backend_dev->dev_attr, max_mr_size, "%ld");
+    CHK_ATTR(dev_attr, backend_dev->dev_attr, max_qp, "%d");
+    CHK_ATTR(dev_attr, backend_dev->dev_attr, max_sge, "%d");
+    CHK_ATTR(dev_attr, backend_dev->dev_attr, max_qp_wr, "%d");
+    CHK_ATTR(dev_attr, backend_dev->dev_attr, max_cq, "%d");
+    CHK_ATTR(dev_attr, backend_dev->dev_attr, max_cqe, "%d");
+    CHK_ATTR(dev_attr, backend_dev->dev_attr, max_mr, "%d");
+    CHK_ATTR(dev_attr, backend_dev->dev_attr, max_pd, "%d");
+    CHK_ATTR(dev_attr, backend_dev->dev_attr, max_qp_rd_atom, "%d");
+    CHK_ATTR(dev_attr, backend_dev->dev_attr, max_qp_init_rd_atom, "%d");
+    CHK_ATTR(dev_attr, backend_dev->dev_attr, max_ah, "%d");
+
+    return 0;
+}
+
+int rdma_backend_init(RdmaBackendDev *backend_dev,
+                      RdmaDeviceResources *rdma_dev_res,
+                      const char *backend_device_name, uint8_t port_num,
+                      uint8_t backend_gid_idx, struct ibv_device_attr *dev_attr,
+                      Error **errp)
+{
+    int i;
+    int ret = 0;
+    int num_ibv_devices;
+    char thread_name[80] = {0};
+    struct ibv_device **dev_list;
+    struct ibv_port_attr port_attr;
+
+    backend_dev->backend_gid_idx = backend_gid_idx;
+    backend_dev->port_num = port_num;
+    backend_dev->rdma_dev_res = rdma_dev_res;
+
+    rdma_backend_register_comp_handler(dummy_comp_handler);
+
+    dev_list = ibv_get_device_list(&num_ibv_devices);
+    if (!dev_list) {
+        error_setg(errp, "Failed to get IB devices list");
+        ret = -EIO;
+        goto out;
+    }
+    if (num_ibv_devices == 0) {
+        error_setg(errp, "No IB devices were found");
+        ret = -ENXIO;
+        goto out;
+    }
+
+    if (backend_device_name) {
+        for (i = 0; dev_list[i]; ++i) {
+            if (!strcmp(ibv_get_device_name(dev_list[i]),
+                        backend_device_name)) {
+                break;
+            }
+        }
+
+        backend_dev->ib_dev = dev_list[i];
+        if (!backend_dev->ib_dev) {
+            error_setg(errp, "Failed to find IB device %s",
+                       backend_device_name);
+            ret = -EIO;
+            goto out;
+        }
+    } else {
+        backend_dev->ib_dev = *dev_list;
+    }
+    ibv_free_device_list(dev_list);
+
+    pr_dbg("Using backend device %s, port %d, gid_idx %d\n",
+           ibv_get_device_name(backend_dev->ib_dev),
+           backend_dev->port_num, backend_dev->backend_gid_idx);
+
+    backend_dev->context = ibv_open_device(backend_dev->ib_dev);
+    if (!backend_dev->context) {
+        error_setg(errp, "Failed to open IB device");
+        ret = -EIO;
+        goto out;
+    }
+
+    backend_dev->channel = ibv_create_comp_channel(backend_dev->context);
+    if (!backend_dev->channel) {
+        error_setg(errp, "Failed to create IB communication channel");
+        ret = -EIO;
+        goto out_close_device;
+    }
+    pr_dbg("dev->backend_dev.channel=%p\n", backend_dev->channel);
+
+    ret = ibv_query_gid(backend_dev->context, backend_dev->port_num,
+                         backend_dev->backend_gid_idx, &backend_dev->gid);
+    if (ret) {
+        error_setg(errp, "Failed to query gid %d",
+                   backend_dev->backend_gid_idx);
+        ret = -EIO;
+        goto out_destroy_comm_channel;
+    }
+    pr_dbg("subnet_prefix=0x%lx\n",
+           be64_to_cpu(backend_dev->gid.global.subnet_prefix));
+    pr_dbg("interface_id=0x%lx\n",
+           be64_to_cpu(backend_dev->gid.global.interface_id));
+
+    ret = ibv_query_port(backend_dev->context, backend_dev->port_num,
+                         &port_attr);
+    if (ret) {
+        error_setg(errp, "Error %d from ibv_query_port", ret);
+        ret = -EIO;
+        goto out_destroy_comm_channel;
+    }
+
+    ret = init_device_caps(backend_dev, dev_attr);
+    if (ret) {
+        error_setg(errp, "Failed to initialize device capabilities");
+        ret = -EIO;
+        goto out_destroy_comm_channel;
+    }
+
+    sprintf(thread_name, "pvrdma_comp_%s",
+            ibv_get_device_name(backend_dev->ib_dev));
+    backend_dev->comp_thread.run = true;
+    qemu_thread_create(&backend_dev->comp_thread.thread, thread_name,
+                       comp_handler_thread, backend_dev, QEMU_THREAD_DETACHED);
+
+    ah_cache_init();
+
+    goto out;
+
+out_destroy_comm_channel:
+    ibv_destroy_comp_channel(backend_dev->channel);
+
+out_close_device:
+    ibv_close_device(backend_dev->context);
+
+out:
+    return ret;
+}
+
+void rdma_backend_fini(RdmaBackendDev *backend_dev)
+{
+    g_hash_table_destroy(ah_hash);
+    ibv_destroy_comp_channel(backend_dev->channel);
+    ibv_close_device(backend_dev->context);
+}
diff --git a/hw/rdma/rdma_backend.h b/hw/rdma/rdma_backend.h
new file mode 100644
index 0000000000..0a594a42ca
--- /dev/null
+++ b/hw/rdma/rdma_backend.h
@@ -0,0 +1,92 @@
+/*
+ *  RDMA device: Definitions of Backend Device functions
+ *
+ * Copyright (C) 2018 Oracle
+ * Copyright (C) 2018 Red Hat Inc
+ *
+ * Authors:
+ *     Yuval Shaia <yuval.shaia@oracle.com>
+ *     Marcel Apfelbaum <marcel@redhat.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef RDMA_BACKEND_H
+#define RDMA_BACKEND_H
+
+#include <qapi/error.h>
+#include "rdma_rm_defs.h"
+#include "rdma_backend_defs.h"
+
+static inline union ibv_gid *rdma_backend_gid(RdmaBackendDev *dev)
+{
+    return &dev->gid;
+}
+
+static inline uint32_t rdma_backend_qpn(RdmaBackendQP *qp)
+{
+    return qp->ibqp ? qp->ibqp->qp_num : 0;
+}
+
+static inline uint32_t rdma_backend_mr_lkey(RdmaBackendMR *mr)
+{
+    return mr->ibmr ? mr->ibmr->lkey : 0;
+}
+
+static inline uint32_t rdma_backend_mr_rkey(RdmaBackendMR *mr)
+{
+    return mr->ibmr ? mr->ibmr->rkey : 0;
+}
+
+int rdma_backend_init(RdmaBackendDev *backend_dev,
+                      RdmaDeviceResources *rdma_dev_res,
+                      const char *backend_device_name, uint8_t port_num,
+                      uint8_t backend_gid_idx, struct ibv_device_attr *dev_attr,
+                      Error **errp);
+void rdma_backend_fini(RdmaBackendDev *backend_dev);
+void rdma_backend_register_comp_handler(void (*handler)(int status,
+                                        unsigned int vendor_err, void *ctx));
+void rdma_backend_unregister_comp_handler(void);
+
+int rdma_backend_query_port(RdmaBackendDev *backend_dev,
+                            struct ibv_port_attr *port_attr);
+int rdma_backend_create_pd(RdmaBackendDev *backend_dev, RdmaBackendPD *pd);
+void rdma_backend_destroy_pd(RdmaBackendPD *pd);
+
+int rdma_backend_create_mr(RdmaBackendMR *mr, RdmaBackendPD *pd, uint64_t addr,
+                           size_t length, int access);
+void rdma_backend_destroy_mr(RdmaBackendMR *mr);
+
+int rdma_backend_create_cq(RdmaBackendDev *backend_dev, RdmaBackendCQ *cq,
+                           int cqe);
+void rdma_backend_destroy_cq(RdmaBackendCQ *cq);
+void rdma_backend_poll_cq(RdmaDeviceResources *rdma_dev_res, RdmaBackendCQ *cq);
+
+int rdma_backend_create_qp(RdmaBackendQP *qp, uint8_t qp_type,
+                           RdmaBackendPD *pd, RdmaBackendCQ *scq,
+                           RdmaBackendCQ *rcq, uint32_t max_send_wr,
+                           uint32_t max_recv_wr, uint32_t max_send_sge,
+                           uint32_t max_recv_sge);
+int rdma_backend_qp_state_init(RdmaBackendDev *backend_dev, RdmaBackendQP *qp,
+                               uint8_t qp_type, uint32_t qkey);
+int rdma_backend_qp_state_rtr(RdmaBackendDev *backend_dev, RdmaBackendQP *qp,
+                              uint8_t qp_type, union ibv_gid *dgid,
+                              uint32_t dqpn, uint32_t rq_psn, uint32_t qkey);
+int rdma_backend_qp_state_rts(RdmaBackendQP *qp, uint8_t qp_type,
+                              uint32_t sq_psn, uint32_t qkey);
+void rdma_backend_destroy_qp(RdmaBackendQP *qp);
+
+void rdma_backend_post_send(RdmaBackendDev *backend_dev,
+                            RdmaDeviceResources *rdma_dev_res,
+                            RdmaBackendQP *qp, uint8_t qp_type,
+                            struct ibv_sge *sge, uint32_t num_sge,
+                            union ibv_gid *dgid, uint32_t dqpn, uint32_t dqkey,
+                            void *ctx);
+void rdma_backend_post_recv(RdmaBackendDev *backend_dev,
+                            RdmaDeviceResources *rdma_dev_res,
+                            RdmaBackendQP *qp, uint8_t qp_type,
+                            struct ibv_sge *sge, uint32_t num_sge, void *ctx);
+
+#endif
diff --git a/hw/rdma/rdma_backend_defs.h b/hw/rdma/rdma_backend_defs.h
new file mode 100644
index 0000000000..516e6e0b19
--- /dev/null
+++ b/hw/rdma/rdma_backend_defs.h
@@ -0,0 +1,62 @@
+/*
+ *  RDMA device: Definitions of Backend Device structures
+ *
+ * Copyright (C) 2018 Oracle
+ * Copyright (C) 2018 Red Hat Inc
+ *
+ * Authors:
+ *     Yuval Shaia <yuval.shaia@oracle.com>
+ *     Marcel Apfelbaum <marcel@redhat.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef RDMA_BACKEND_DEFS_H
+#define RDMA_BACKEND_DEFS_H
+
+#include <infiniband/verbs.h>
+#include <qemu/thread.h>
+
+typedef struct RdmaDeviceResources RdmaDeviceResources;
+
+typedef struct RdmaBackendThread {
+    QemuThread thread;
+    QemuMutex mutex;
+    bool run;
+} RdmaBackendThread;
+
+typedef struct RdmaBackendDev {
+    PCIDevice *dev;
+    RdmaBackendThread comp_thread;
+    struct ibv_device *ib_dev;
+    uint8_t port_num;
+    struct ibv_context *context;
+    struct ibv_comp_channel *channel;
+    union ibv_gid gid;
+    struct ibv_device_attr dev_attr;
+    uint8_t backend_gid_idx;
+    RdmaDeviceResources *rdma_dev_res;
+} RdmaBackendDev;
+
+typedef struct RdmaBackendPD {
+    struct ibv_pd *ibpd;
+} RdmaBackendPD;
+
+typedef struct RdmaBackendMR {
+    struct ibv_pd *ibpd;
+    struct ibv_mr *ibmr;
+} RdmaBackendMR;
+
+typedef struct RdmaBackendCQ {
+    RdmaBackendDev *backend_dev;
+    struct ibv_cq *ibcq;
+} RdmaBackendCQ;
+
+typedef struct RdmaBackendQP {
+    struct ibv_pd *ibpd;
+    struct ibv_qp *ibqp;
+} RdmaBackendQP;
+
+#endif
diff --git a/hw/rdma/rdma_rm.c b/hw/rdma/rdma_rm.c
new file mode 100644
index 0000000000..888718698c
--- /dev/null
+++ b/hw/rdma/rdma_rm.c
@@ -0,0 +1,619 @@
+/*
+ * QEMU paravirtual RDMA - Resource Manager Implementation
+ *
+ * Copyright (C) 2018 Oracle
+ * Copyright (C) 2018 Red Hat Inc
+ *
+ * Authors:
+ *     Yuval Shaia <yuval.shaia@oracle.com>
+ *     Marcel Apfelbaum <marcel@redhat.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include <qemu/osdep.h>
+#include <qapi/error.h>
+#include <cpu.h>
+
+#include "rdma_utils.h"
+#include "rdma_backend.h"
+#include "rdma_rm.h"
+
+#define MAX_RM_TBL_NAME 16
+
+/* Page directory and page tables */
+#define PG_DIR_SZ { TARGET_PAGE_SIZE / sizeof(__u64) }
+#define PG_TBL_SZ { TARGET_PAGE_SIZE / sizeof(__u64) }
+
+static inline int res_tbl_init(const char *name, RdmaRmResTbl *tbl,
+                               uint32_t tbl_sz, uint32_t res_sz)
+{
+    tbl->tbl = malloc(tbl_sz * res_sz);
+    if (!tbl->tbl) {
+        return -ENOMEM;
+    }
+
+    strncpy(tbl->name, name, MAX_RM_TBL_NAME);
+    tbl->name[MAX_RM_TBL_NAME - 1] = 0;
+
+    tbl->bitmap = bitmap_new(tbl_sz);
+    tbl->tbl_sz = tbl_sz;
+    tbl->res_sz = res_sz;
+    qemu_mutex_init(&tbl->lock);
+
+    return 0;
+}
+
+static inline void res_tbl_free(RdmaRmResTbl *tbl)
+{
+    qemu_mutex_destroy(&tbl->lock);
+    free(tbl->tbl);
+    bitmap_zero_extend(tbl->bitmap, tbl->tbl_sz, 0);
+}
+
+static inline void *res_tbl_get(RdmaRmResTbl *tbl, uint32_t handle)
+{
+    pr_dbg("%s, handle=%d\n", tbl->name, handle);
+
+    if ((handle < tbl->tbl_sz) && (test_bit(handle, tbl->bitmap))) {
+        return tbl->tbl + handle * tbl->res_sz;
+    } else {
+        pr_dbg("Invalid handle %d\n", handle);
+        return NULL;
+    }
+}
+
+static inline void *res_tbl_alloc(RdmaRmResTbl *tbl, uint32_t *handle)
+{
+    qemu_mutex_lock(&tbl->lock);
+
+    *handle = find_first_zero_bit(tbl->bitmap, tbl->tbl_sz);
+    if (*handle > tbl->tbl_sz) {
+        pr_dbg("Failed to alloc, bitmap is full\n");
+        qemu_mutex_unlock(&tbl->lock);
+        return NULL;
+    }
+
+    set_bit(*handle, tbl->bitmap);
+
+    qemu_mutex_unlock(&tbl->lock);
+
+    memset(tbl->tbl + *handle * tbl->res_sz, 0, tbl->res_sz);
+
+    pr_dbg("%s, handle=%d\n", tbl->name, *handle);
+
+    return tbl->tbl + *handle * tbl->res_sz;
+}
+
+static inline void res_tbl_dealloc(RdmaRmResTbl *tbl, uint32_t handle)
+{
+    pr_dbg("%s, handle=%d\n", tbl->name, handle);
+
+    qemu_mutex_lock(&tbl->lock);
+
+    if (handle < tbl->tbl_sz) {
+        clear_bit(handle, tbl->bitmap);
+    }
+
+    qemu_mutex_unlock(&tbl->lock);
+}
+
+int rdma_rm_alloc_pd(RdmaDeviceResources *dev_res, RdmaBackendDev *backend_dev,
+                     uint32_t *pd_handle, uint32_t ctx_handle)
+{
+    RdmaRmPD *pd;
+    int ret = -ENOMEM;
+
+    pd = res_tbl_alloc(&dev_res->pd_tbl, pd_handle);
+    if (!pd) {
+        goto out;
+    }
+
+    ret = rdma_backend_create_pd(backend_dev, &pd->backend_pd);
+    if (ret) {
+        ret = -EIO;
+        goto out_tbl_dealloc;
+    }
+
+    pd->ctx_handle = ctx_handle;
+
+    return 0;
+
+out_tbl_dealloc:
+    res_tbl_dealloc(&dev_res->pd_tbl, *pd_handle);
+
+out:
+    return ret;
+}
+
+RdmaRmPD *rdma_rm_get_pd(RdmaDeviceResources *dev_res, uint32_t pd_handle)
+{
+    return res_tbl_get(&dev_res->pd_tbl, pd_handle);
+}
+
+void rdma_rm_dealloc_pd(RdmaDeviceResources *dev_res, uint32_t pd_handle)
+{
+    RdmaRmPD *pd = rdma_rm_get_pd(dev_res, pd_handle);
+
+    if (pd) {
+        rdma_backend_destroy_pd(&pd->backend_pd);
+        res_tbl_dealloc(&dev_res->pd_tbl, pd_handle);
+    }
+}
+
+int rdma_rm_alloc_mr(RdmaDeviceResources *dev_res, uint32_t pd_handle,
+                     uint64_t guest_start, size_t guest_length, void *host_virt,
+                     int access_flags, uint32_t *mr_handle, uint32_t *lkey,
+                     uint32_t *rkey)
+{
+    RdmaRmMR *mr;
+    int ret = 0;
+    RdmaRmPD *pd;
+    uint64_t addr;
+    size_t length;
+
+    pd = rdma_rm_get_pd(dev_res, pd_handle);
+    if (!pd) {
+        pr_dbg("Invalid PD\n");
+        ret = -EINVAL;
+        goto out;
+    }
+
+    mr = res_tbl_alloc(&dev_res->mr_tbl, mr_handle);
+    if (!mr) {
+        pr_dbg("Failed to allocate obj in table\n");
+        ret = -ENOMEM;
+        goto out;
+    }
+
+    if (!host_virt) {
+        /* TODO: This is my guess but not so sure that this needs to be
+         * done */
+        length = mr->length = TARGET_PAGE_SIZE;
+        addr = mr->addr = (uint64_t)malloc(length);
+    } else {
+        mr->addr = 0;
+
+        mr->user_mr.host_virt = (uint64_t) host_virt;
+        pr_dbg("host_virt=0x%lx\n", mr->user_mr.host_virt);
+        mr->user_mr.length = guest_length;
+        pr_dbg("length=0x%lx\n", guest_length);
+        mr->user_mr.guest_start = guest_start;
+        pr_dbg("guest_start=0x%lx\n", mr->user_mr.guest_start);
+
+        length = mr->user_mr.length;
+        addr = mr->user_mr.host_virt;
+    }
+
+    ret = rdma_backend_create_mr(&mr->backend_mr, &pd->backend_pd, addr, length,
+                                 access_flags);
+    if (ret) {
+        pr_dbg("Fail in rdma_backend_create_mr, err=%d\n", ret);
+        ret = -EIO;
+        goto out_dealloc_mr;
+    }
+
+    if (!host_virt) {
+        *lkey = mr->lkey = rdma_backend_mr_lkey(&mr->backend_mr);
+        *rkey = mr->rkey = rdma_backend_mr_rkey(&mr->backend_mr);
+    } else {
+        /* We keep mr_handle in lkey so send and recv get get mr ptr */
+        *lkey = *mr_handle;
+        *rkey = -1;
+    }
+
+    mr->pd_handle = pd_handle;
+
+    return 0;
+
+out_dealloc_mr:
+    res_tbl_dealloc(&dev_res->mr_tbl, *mr_handle);
+
+out:
+    return ret;
+}
+
+RdmaRmMR *rdma_rm_get_mr(RdmaDeviceResources *dev_res, uint32_t mr_handle)
+{
+    return res_tbl_get(&dev_res->mr_tbl, mr_handle);
+}
+
+void rdma_rm_dealloc_mr(RdmaDeviceResources *dev_res, uint32_t mr_handle)
+{
+    RdmaRmMR *mr = rdma_rm_get_mr(dev_res, mr_handle);
+
+    if (mr) {
+        rdma_backend_destroy_mr(&mr->backend_mr);
+        munmap((void *)mr->user_mr.host_virt, mr->user_mr.length);
+        free((void *)mr->addr);
+        res_tbl_dealloc(&dev_res->mr_tbl, mr_handle);
+    }
+}
+
+int rdma_rm_alloc_uc(RdmaDeviceResources *dev_res, uint32_t pfn,
+                     uint32_t *uc_handle)
+{
+    RdmaRmUC *uc;
+
+    /* TODO: Need to make sure pfn is between bar start address and
+     * bsd+RDMA_BAR2_UAR_SIZE
+    if (pfn > RDMA_BAR2_UAR_SIZE) {
+        pr_err("pfn out of range (%d > %d)\n", pfn, RDMA_BAR2_UAR_SIZE);
+        return -ENOMEM;
+    }
+    */
+
+    uc = res_tbl_alloc(&dev_res->uc_tbl, uc_handle);
+    if (!uc) {
+        return -ENOMEM;
+    }
+
+    return 0;
+}
+
+RdmaRmUC *rdma_rm_get_uc(RdmaDeviceResources *dev_res, uint32_t uc_handle)
+{
+    return res_tbl_get(&dev_res->uc_tbl, uc_handle);
+}
+
+void rdma_rm_dealloc_uc(RdmaDeviceResources *dev_res, uint32_t uc_handle)
+{
+    RdmaRmUC *uc = rdma_rm_get_uc(dev_res, uc_handle);
+
+    if (uc) {
+        res_tbl_dealloc(&dev_res->uc_tbl, uc_handle);
+    }
+}
+
+RdmaRmCQ *rdma_rm_get_cq(RdmaDeviceResources *dev_res, uint32_t cq_handle)
+{
+    return res_tbl_get(&dev_res->cq_tbl, cq_handle);
+}
+
+int rdma_rm_alloc_cq(RdmaDeviceResources *dev_res, RdmaBackendDev *backend_dev,
+                     uint32_t cqe, uint32_t *cq_handle, void *opaque)
+{
+    int rc = -ENOMEM;
+    RdmaRmCQ *cq;
+
+    cq = res_tbl_alloc(&dev_res->cq_tbl, cq_handle);
+    if (!cq) {
+        return -ENOMEM;
+    }
+
+    cq->opaque = opaque;
+    cq->notify = false;
+
+    rc = rdma_backend_create_cq(backend_dev, &cq->backend_cq, cqe);
+    if (rc) {
+        rc = -EIO;
+        goto out_dealloc_cq;
+    }
+
+    goto out;
+
+out_dealloc_cq:
+    rdma_rm_dealloc_cq(dev_res, *cq_handle);
+
+out:
+    return rc;
+}
+
+void rdma_rm_req_notify_cq(RdmaDeviceResources *dev_res, uint32_t cq_handle,
+                           bool notify)
+{
+    RdmaRmCQ *cq;
+
+    pr_dbg("cq_handle=%d, notify=0x%x\n", cq_handle, notify);
+
+    cq = rdma_rm_get_cq(dev_res, cq_handle);
+    if (!cq) {
+        return;
+    }
+
+    cq->notify = notify;
+    pr_dbg("notify=%d\n", cq->notify);
+}
+
+void rdma_rm_dealloc_cq(RdmaDeviceResources *dev_res, uint32_t cq_handle)
+{
+    RdmaRmCQ *cq;
+
+    cq = rdma_rm_get_cq(dev_res, cq_handle);
+    if (!cq) {
+        return;
+    }
+
+    rdma_backend_destroy_cq(&cq->backend_cq);
+
+    res_tbl_dealloc(&dev_res->cq_tbl, cq_handle);
+}
+
+RdmaRmQP *rdma_rm_get_qp(RdmaDeviceResources *dev_res, uint32_t qpn)
+{
+    GBytes *key = g_bytes_new(&qpn, sizeof(qpn));
+
+    RdmaRmQP *qp = g_hash_table_lookup(dev_res->qp_hash, key);
+
+    g_bytes_unref(key);
+
+    return qp;
+}
+
+int rdma_rm_alloc_qp(RdmaDeviceResources *dev_res, uint32_t pd_handle,
+                     uint8_t qp_type, uint32_t max_send_wr,
+                     uint32_t max_send_sge, uint32_t send_cq_handle,
+                     uint32_t max_recv_wr, uint32_t max_recv_sge,
+                     uint32_t recv_cq_handle, void *opaque, uint32_t *qpn)
+{
+    int rc = 0;
+    RdmaRmQP *qp;
+    RdmaRmCQ *scq, *rcq;
+    RdmaRmPD *pd;
+    uint32_t rm_qpn;
+
+    pr_dbg("qp_type=%d\n", qp_type);
+
+    pd = rdma_rm_get_pd(dev_res, pd_handle);
+    if (!pd) {
+        pr_err("Invalid pd handle (%d)\n", pd_handle);
+        return -EINVAL;
+    }
+
+    scq = rdma_rm_get_cq(dev_res, send_cq_handle);
+    rcq = rdma_rm_get_cq(dev_res, recv_cq_handle);
+
+    if (!scq || !rcq) {
+        pr_err("Invalid send_cqn or recv_cqn (%d, %d)\n",
+               send_cq_handle, recv_cq_handle);
+        return -EINVAL;
+    }
+
+    qp = res_tbl_alloc(&dev_res->qp_tbl, &rm_qpn);
+    if (!qp) {
+        return -ENOMEM;
+    }
+    pr_dbg("rm_qpn=%d\n", rm_qpn);
+
+    qp->qpn = rm_qpn;
+    qp->qp_state = IBV_QPS_ERR;
+    qp->qp_type = qp_type;
+    qp->send_cq_handle = send_cq_handle;
+    qp->recv_cq_handle = recv_cq_handle;
+    qp->opaque = opaque;
+
+    rc = rdma_backend_create_qp(&qp->backend_qp, qp_type, &pd->backend_pd,
+                                &scq->backend_cq, &rcq->backend_cq, max_send_wr,
+                                max_recv_wr, max_send_sge, max_recv_sge);
+    if (rc) {
+        rc = -EIO;
+        goto out_dealloc_qp;
+    }
+
+    *qpn = rdma_backend_qpn(&qp->backend_qp);
+    pr_dbg("rm_qpn=%d, backend_qpn=0x%x\n", rm_qpn, *qpn);
+    g_hash_table_insert(dev_res->qp_hash, g_bytes_new(qpn, sizeof(*qpn)), qp);
+
+    return 0;
+
+out_dealloc_qp:
+    res_tbl_dealloc(&dev_res->qp_tbl, qp->qpn);
+
+    return rc;
+}
+
+int rdma_rm_modify_qp(RdmaDeviceResources *dev_res, RdmaBackendDev *backend_dev,
+                      uint32_t qp_handle, uint32_t attr_mask,
+                      union ibv_gid *dgid, uint32_t dqpn,
+                      enum ibv_qp_state qp_state, uint32_t qkey,
+                      uint32_t rq_psn, uint32_t sq_psn)
+{
+    RdmaRmQP *qp;
+    int ret;
+
+    pr_dbg("qpn=%d\n", qp_handle);
+
+    qp = rdma_rm_get_qp(dev_res, qp_handle);
+    if (!qp) {
+        return -EINVAL;
+    }
+
+    pr_dbg("qp_type=%d\n", qp->qp_type);
+    pr_dbg("attr_mask=0x%x\n", attr_mask);
+
+    if (qp->qp_type == 0) {
+        pr_dbg("QP0 is not supported\n");
+        return -EPERM;
+    }
+
+    if (qp->qp_type == 1) {
+        pr_dbg("QP1\n");
+        return 0;
+    }
+
+    if (attr_mask & IBV_QP_STATE) {
+        qp->qp_state = qp_state;
+        pr_dbg("qp_state=%d\n", qp->qp_state);
+
+        if (qp->qp_state == IBV_QPS_INIT) {
+            ret = rdma_backend_qp_state_init(backend_dev, &qp->backend_qp,
+                                             qp->qp_type, qkey);
+            if (ret) {
+                return -EIO;
+            }
+        }
+
+        if (qp->qp_state == IBV_QPS_RTR) {
+            if (!(attr_mask & IBV_QP_DEST_QPN) && qp->qp_type == IBV_QPT_RC) {
+                pr_dbg("IBV_QPS_RTR but not IBV_QP_DEST_QPN\n");
+                return -EIO;
+            }
+            if (!(attr_mask & IBV_QP_AV) && qp->qp_type == IBV_QPT_RC) {
+                pr_dbg("IBV_QPS_RTR but not IBV_QP_AV\n");
+                return -EIO;
+            }
+
+            ret = rdma_backend_qp_state_rtr(backend_dev, &qp->backend_qp,
+                                            qp->qp_type, dgid, dqpn, rq_psn,
+                                            qkey);
+            if (ret) {
+                return -EIO;
+            }
+        }
+
+        if (qp->qp_state == IBV_QPS_RTS) {
+            ret = rdma_backend_qp_state_rts(&qp->backend_qp, qp->qp_type,
+                                            sq_psn, qkey);
+            if (ret) {
+                return -EIO;
+            }
+        }
+    }
+
+    return 0;
+}
+
+void rdma_rm_dealloc_qp(RdmaDeviceResources *dev_res, uint32_t qp_handle)
+{
+    RdmaRmQP *qp;
+    GBytes *key;
+
+    key = g_bytes_new(&qp_handle, sizeof(qp_handle));
+    qp = g_hash_table_lookup(dev_res->qp_hash, key);
+    if (!qp) {
+        return;
+    }
+    g_hash_table_remove(dev_res->qp_hash, key);
+    g_bytes_unref(key);
+
+    rdma_backend_destroy_qp(&qp->backend_qp);
+
+    res_tbl_dealloc(&dev_res->qp_tbl, qp->qpn);
+}
+
+void *rdma_rm_get_cqe_ctx(RdmaDeviceResources *dev_res, uint32_t cqe_ctx_id)
+{
+    void **cqe_ctx;
+
+    cqe_ctx = res_tbl_get(&dev_res->cqe_ctx_tbl, cqe_ctx_id);
+    if (!cqe_ctx) {
+        return NULL;
+    }
+
+    pr_dbg("ctx=%p\n", *cqe_ctx);
+
+    return *cqe_ctx;
+}
+
+int rdma_rm_alloc_cqe_ctx(RdmaDeviceResources *dev_res, uint32_t *cqe_ctx_id,
+                          void *ctx)
+{
+    void **cqe_ctx;
+
+    cqe_ctx = res_tbl_alloc(&dev_res->cqe_ctx_tbl, cqe_ctx_id);
+    if (!cqe_ctx) {
+        return -ENOMEM;
+    }
+
+    pr_dbg("ctx=%p\n", ctx);
+    *cqe_ctx = ctx;
+
+    return 0;
+}
+
+void rdma_rm_dealloc_cqe_ctx(RdmaDeviceResources *dev_res, uint32_t cqe_ctx_id)
+{
+    res_tbl_dealloc(&dev_res->cqe_ctx_tbl, cqe_ctx_id);
+}
+
+static void destroy_qp_hash_key(gpointer data)
+{
+    g_bytes_unref(data);
+}
+
+int rdma_rm_init(RdmaDeviceResources *dev_res, struct ibv_device_attr *dev_attr,
+                 Error **errp)
+{
+    int ret = 0;
+
+    ret = res_tbl_init("PD", &dev_res->pd_tbl, dev_attr->max_pd,
+                       sizeof(RdmaRmPD));
+    if (ret) {
+        goto out;
+    }
+
+    ret = res_tbl_init("CQ", &dev_res->cq_tbl, dev_attr->max_cq,
+                       sizeof(RdmaRmCQ));
+    if (ret) {
+        goto cln_pds;
+    }
+
+    ret = res_tbl_init("MR", &dev_res->mr_tbl, dev_attr->max_mr,
+                       sizeof(RdmaRmMR));
+    if (ret) {
+        goto cln_cqs;
+    }
+
+    ret = res_tbl_init("QP", &dev_res->qp_tbl, dev_attr->max_qp,
+                       sizeof(RdmaRmQP));
+    if (ret) {
+        goto cln_mrs;
+    }
+
+    ret = res_tbl_init("CQE_CTX", &dev_res->cqe_ctx_tbl, dev_attr->max_qp *
+                       dev_attr->max_qp_wr, sizeof(void *));
+    if (ret) {
+        goto cln_qps;
+    }
+
+    ret = res_tbl_init("UC", &dev_res->uc_tbl, MAX_UCS, sizeof(RdmaRmUC));
+    if (ret) {
+        goto cln_cqe_ctxs;
+    }
+
+    dev_res->qp_hash = g_hash_table_new_full(g_bytes_hash, g_bytes_equal,
+                                             destroy_qp_hash_key, NULL);
+    if (!dev_res->qp_hash) {
+        goto cln_ucs;
+    }
+
+    goto out;
+
+cln_ucs:
+    res_tbl_free(&dev_res->uc_tbl);
+
+cln_cqe_ctxs:
+    res_tbl_free(&dev_res->cqe_ctx_tbl);
+
+cln_qps:
+    res_tbl_free(&dev_res->qp_tbl);
+
+cln_mrs:
+    res_tbl_free(&dev_res->mr_tbl);
+
+cln_cqs:
+    res_tbl_free(&dev_res->cq_tbl);
+
+cln_pds:
+    res_tbl_free(&dev_res->pd_tbl);
+
+out:
+    if (ret) {
+        error_setg(errp, "Failed to initialize RM");
+    }
+
+    return ret;
+}
+
+void rdma_rm_fini(RdmaDeviceResources *dev_res)
+{
+    res_tbl_free(&dev_res->uc_tbl);
+    res_tbl_free(&dev_res->cqe_ctx_tbl);
+    res_tbl_free(&dev_res->qp_tbl);
+    res_tbl_free(&dev_res->cq_tbl);
+    res_tbl_free(&dev_res->mr_tbl);
+    res_tbl_free(&dev_res->pd_tbl);
+    g_hash_table_destroy(dev_res->qp_hash);
+}
diff --git a/hw/rdma/rdma_rm.h b/hw/rdma/rdma_rm.h
new file mode 100644
index 0000000000..82a1026629
--- /dev/null
+++ b/hw/rdma/rdma_rm.h
@@ -0,0 +1,69 @@
+/*
+ * RDMA device: Definitions of Resource Manager functions
+ *
+ * Copyright (C) 2018 Oracle
+ * Copyright (C) 2018 Red Hat Inc
+ *
+ * Authors:
+ *     Yuval Shaia <yuval.shaia@oracle.com>
+ *     Marcel Apfelbaum <marcel@redhat.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef RDMA_RM_H
+#define RDMA_RM_H
+
+#include <qapi/error.h>
+#include "rdma_backend_defs.h"
+#include "rdma_rm_defs.h"
+
+int rdma_rm_init(RdmaDeviceResources *dev_res, struct ibv_device_attr *dev_attr,
+                 Error **errp);
+void rdma_rm_fini(RdmaDeviceResources *dev_res);
+
+int rdma_rm_alloc_pd(RdmaDeviceResources *dev_res, RdmaBackendDev *backend_dev,
+                     uint32_t *pd_handle, uint32_t ctx_handle);
+RdmaRmPD *rdma_rm_get_pd(RdmaDeviceResources *dev_res, uint32_t pd_handle);
+void rdma_rm_dealloc_pd(RdmaDeviceResources *dev_res, uint32_t pd_handle);
+
+int rdma_rm_alloc_mr(RdmaDeviceResources *dev_res, uint32_t pd_handle,
+                     uint64_t guest_start, size_t guest_length, void *host_virt,
+                     int access_flags, uint32_t *mr_handle, uint32_t *lkey,
+                     uint32_t *rkey);
+RdmaRmMR *rdma_rm_get_mr(RdmaDeviceResources *dev_res, uint32_t mr_handle);
+void rdma_rm_dealloc_mr(RdmaDeviceResources *dev_res, uint32_t mr_handle);
+
+int rdma_rm_alloc_uc(RdmaDeviceResources *dev_res, uint32_t pfn,
+                     uint32_t *uc_handle);
+RdmaRmUC *rdma_rm_get_uc(RdmaDeviceResources *dev_res, uint32_t uc_handle);
+void rdma_rm_dealloc_uc(RdmaDeviceResources *dev_res, uint32_t uc_handle);
+
+int rdma_rm_alloc_cq(RdmaDeviceResources *dev_res, RdmaBackendDev *backend_dev,
+                     uint32_t cqe, uint32_t *cq_handle, void *opaque);
+RdmaRmCQ *rdma_rm_get_cq(RdmaDeviceResources *dev_res, uint32_t cq_handle);
+void rdma_rm_req_notify_cq(RdmaDeviceResources *dev_res, uint32_t cq_handle,
+                           bool notify);
+void rdma_rm_dealloc_cq(RdmaDeviceResources *dev_res, uint32_t cq_handle);
+
+int rdma_rm_alloc_qp(RdmaDeviceResources *dev_res, uint32_t pd_handle,
+                     uint8_t qp_type, uint32_t max_send_wr,
+                     uint32_t max_send_sge, uint32_t send_cq_handle,
+                     uint32_t max_recv_wr, uint32_t max_recv_sge,
+                     uint32_t recv_cq_handle, void *opaque, uint32_t *qpn);
+RdmaRmQP *rdma_rm_get_qp(RdmaDeviceResources *dev_res, uint32_t qpn);
+int rdma_rm_modify_qp(RdmaDeviceResources *dev_res, RdmaBackendDev *backend_dev,
+                      uint32_t qp_handle, uint32_t attr_mask,
+                      union ibv_gid *dgid, uint32_t dqpn,
+                      enum ibv_qp_state qp_state, uint32_t qkey,
+                      uint32_t rq_psn, uint32_t sq_psn);
+void rdma_rm_dealloc_qp(RdmaDeviceResources *dev_res, uint32_t qp_handle);
+
+int rdma_rm_alloc_cqe_ctx(RdmaDeviceResources *dev_res, uint32_t *cqe_ctx_id,
+                          void *ctx);
+void *rdma_rm_get_cqe_ctx(RdmaDeviceResources *dev_res, uint32_t cqe_ctx_id);
+void rdma_rm_dealloc_cqe_ctx(RdmaDeviceResources *dev_res, uint32_t cqe_ctx_id);
+
+#endif
diff --git a/hw/rdma/rdma_rm_defs.h b/hw/rdma/rdma_rm_defs.h
new file mode 100644
index 0000000000..e327b7177a
--- /dev/null
+++ b/hw/rdma/rdma_rm_defs.h
@@ -0,0 +1,106 @@
+/*
+ * RDMA device: Definitions of Resource Manager structures
+ *
+ * Copyright (C) 2018 Oracle
+ * Copyright (C) 2018 Red Hat Inc
+ *
+ * Authors:
+ *     Yuval Shaia <yuval.shaia@oracle.com>
+ *     Marcel Apfelbaum <marcel@redhat.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef RDMA_RM_DEFS_H
+#define RDMA_RM_DEFS_H
+
+#include "rdma_backend_defs.h"
+
+#define MAX_PORTS             1
+#define MAX_PORT_GIDS         1
+#define MAX_PORT_PKEYS        1
+#define MAX_PKEYS             1
+#define MAX_GIDS              2048
+#define MAX_UCS               512
+#define MAX_MR_SIZE           (1UL << 27)
+#define MAX_QP                1024
+#define MAX_SGE               4
+#define MAX_CQ                2048
+#define MAX_MR                1024
+#define MAX_PD                1024
+#define MAX_QP_RD_ATOM        16
+#define MAX_QP_INIT_RD_ATOM   16
+#define MAX_AH                64
+
+#define MAX_RMRESTBL_NAME_SZ 16
+typedef struct RdmaRmResTbl {
+    char name[MAX_RMRESTBL_NAME_SZ];
+    unsigned long *bitmap;
+    size_t tbl_sz;
+    size_t res_sz;
+    void *tbl;
+    QemuMutex lock;
+} RdmaRmResTbl;
+
+typedef struct RdmaRmPD {
+    uint32_t ctx_handle;
+    RdmaBackendPD backend_pd;
+} RdmaRmPD;
+
+typedef struct RdmaRmCQ {
+    void *opaque;
+    bool notify;
+    RdmaBackendCQ backend_cq;
+} RdmaRmCQ;
+
+typedef struct RdmaRmUserMR {
+    uint64_t host_virt;
+    uint64_t guest_start;
+    size_t length;
+} RdmaRmUserMR;
+
+/* MR (DMA region) */
+typedef struct RdmaRmMR {
+    uint32_t pd_handle;
+    uint32_t lkey;
+    uint32_t rkey;
+    RdmaBackendMR backend_mr;
+    RdmaRmUserMR user_mr;
+    uint64_t addr;
+    size_t length;
+} RdmaRmMR;
+
+typedef struct RdmaRmUC {
+    uint64_t uc_handle;
+} RdmaRmUC;
+
+typedef struct RdmaRmQP {
+    uint32_t qp_type;
+    enum ibv_qp_state qp_state;
+    uint32_t qpn;
+    void *opaque;
+    uint32_t send_cq_handle;
+    uint32_t recv_cq_handle;
+    RdmaBackendQP backend_qp;
+} RdmaRmQP;
+
+typedef struct RdmaRmPort {
+    enum ibv_port_state state;
+    union ibv_gid gid_tbl[MAX_PORT_GIDS];
+    int *pkey_tbl; /* TODO: Not yet supported */
+} RdmaRmPort;
+
+typedef struct RdmaDeviceResources {
+    RdmaRmPort ports[MAX_PORTS];
+    RdmaRmResTbl pd_tbl;
+    RdmaRmResTbl mr_tbl;
+    RdmaRmResTbl uc_tbl;
+    RdmaRmResTbl qp_tbl;
+    RdmaRmResTbl cq_tbl;
+    RdmaRmResTbl cqe_ctx_tbl;
+    GHashTable *qp_hash; /* Keeps mapping between real and emulated */
+} RdmaDeviceResources;
+
+#endif
diff --git a/hw/rdma/rdma_utils.c b/hw/rdma/rdma_utils.c
new file mode 100644
index 0000000000..fb3e2a76c0
--- /dev/null
+++ b/hw/rdma/rdma_utils.c
@@ -0,0 +1,52 @@
+
+/*
+ * QEMU paravirtual RDMA - Generic RDMA backend
+ *
+ * Copyright (C) 2018 Oracle
+ * Copyright (C) 2018 Red Hat Inc
+ *
+ * Authors:
+ *     Yuval Shaia <yuval.shaia@oracle.com>
+ *     Marcel Apfelbaum <marcel@redhat.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "rdma_utils.h"
+
+void *rdma_pci_dma_map(PCIDevice *dev, dma_addr_t addr, dma_addr_t plen)
+{
+    void *p;
+    hwaddr len = plen;
+
+    if (!addr) {
+        pr_dbg("addr is NULL\n");
+        return NULL;
+    }
+
+    p = pci_dma_map(dev, addr, &len, DMA_DIRECTION_TO_DEVICE);
+    if (!p) {
+        pr_dbg("Fail in pci_dma_map, addr=0x%llx, len=%ld\n",
+               (long long unsigned int)addr, len);
+        return NULL;
+    }
+
+    if (len != plen) {
+        rdma_pci_dma_unmap(dev, p, len);
+        return NULL;
+    }
+
+    pr_dbg("0x%llx -> %p (len=%ld)\n", (long long unsigned int)addr, p, len);
+
+    return p;
+}
+
+void rdma_pci_dma_unmap(PCIDevice *dev, void *buffer, dma_addr_t len)
+{
+    pr_dbg("%p\n", buffer);
+    if (buffer) {
+        pci_dma_unmap(dev, buffer, len, DMA_DIRECTION_TO_DEVICE, 0);
+    }
+}
diff --git a/hw/rdma/rdma_utils.h b/hw/rdma/rdma_utils.h
new file mode 100644
index 0000000000..b54e96b0ca
--- /dev/null
+++ b/hw/rdma/rdma_utils.h
@@ -0,0 +1,43 @@
+/*
+ * RDMA device: Debug utilities
+ *
+ * Copyright (C) 2018 Oracle
+ * Copyright (C) 2018 Red Hat Inc
+ *
+ *
+ * Authors:
+ *     Yuval Shaia <yuval.shaia@oracle.com>
+ *     Marcel Apfelbaum <marcel@redhat.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef RDMA_UTILS_H
+#define RDMA_UTILS_H
+
+#include <qemu/osdep.h>
+#include <include/hw/pci/pci.h>
+#include <include/sysemu/dma.h>
+
+#define pr_info(fmt, ...) \
+    fprintf(stdout, "%s: %-20s (%3d): " fmt, "pvrdma",  __func__, __LINE__,\
+           ## __VA_ARGS__)
+
+#define pr_err(fmt, ...) \
+    fprintf(stderr, "%s: Error at %-20s (%3d): " fmt, "pvrdma", __func__, \
+        __LINE__, ## __VA_ARGS__)
+
+#ifdef PVRDMA_DEBUG
+#define pr_dbg(fmt, ...) \
+    fprintf(stdout, "%s: %-20s (%3d): " fmt, "pvrdma", __func__, __LINE__,\
+           ## __VA_ARGS__)
+#else
+#define pr_dbg(fmt, ...)
+#endif
+
+void *rdma_pci_dma_map(PCIDevice *dev, dma_addr_t addr, dma_addr_t plen);
+void rdma_pci_dma_unmap(PCIDevice *dev, void *buffer, dma_addr_t len);
+
+#endif
diff --git a/hw/rdma/trace-events b/hw/rdma/trace-events
new file mode 100644
index 0000000000..421986adbe
--- /dev/null
+++ b/hw/rdma/trace-events
@@ -0,0 +1,5 @@
+# See docs/tracing.txt for syntax documentation.
+
+#hw/rdma/rdma_backend.c
+create_ah_cache_hit(uint64_t subnet, uint64_t net_id) "subnet = 0x%lx net_id = 0x%lx"
+create_ah_cache_miss(uint64_t subnet, uint64_t net_id) "subnet = 0x%lx net_id = 0x%lx"
diff --git a/hw/rdma/vmw/pvrdma.h b/hw/rdma/vmw/pvrdma.h
new file mode 100644
index 0000000000..2a8a4beb78
--- /dev/null
+++ b/hw/rdma/vmw/pvrdma.h
@@ -0,0 +1,122 @@
+/*
+ * QEMU VMWARE paravirtual RDMA device definitions
+ *
+ * Copyright (C) 2018 Oracle
+ * Copyright (C) 2018 Red Hat Inc
+ *
+ * Authors:
+ *     Yuval Shaia <yuval.shaia@oracle.com>
+ *     Marcel Apfelbaum <marcel@redhat.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef PVRDMA_PVRDMA_H
+#define PVRDMA_PVRDMA_H
+
+#include <hw/pci/pci.h>
+#include <hw/pci/msix.h>
+
+#include "../rdma_backend_defs.h"
+#include "../rdma_rm_defs.h"
+
+#include "pvrdma_dev_api.h"
+#include "pvrdma_ring.h"
+#include "pvrdma_dev_ring.h"
+
+/* BARs */
+#define RDMA_MSIX_BAR_IDX    0
+#define RDMA_REG_BAR_IDX     1
+#define RDMA_UAR_BAR_IDX     2
+#define RDMA_BAR0_MSIX_SIZE  (16 * 1024)
+#define RDMA_BAR1_REGS_SIZE  256
+#define RDMA_BAR2_UAR_SIZE   (0x1000 * MAX_UCS) /* each uc gets page */
+
+/* MSIX */
+#define RDMA_MAX_INTRS       3
+#define RDMA_MSIX_TABLE      0x0000
+#define RDMA_MSIX_PBA        0x2000
+
+/* Interrupts Vectors */
+#define INTR_VEC_CMD_RING            0
+#define INTR_VEC_CMD_ASYNC_EVENTS    1
+#define INTR_VEC_CMD_COMPLETION_Q    2
+
+/* HW attributes */
+#define PVRDMA_HW_NAME       "pvrdma"
+#define PVRDMA_HW_VERSION    17
+#define PVRDMA_FW_VERSION    14
+
+typedef struct DSRInfo {
+    dma_addr_t dma;
+    struct pvrdma_device_shared_region *dsr;
+
+    union pvrdma_cmd_req *req;
+    union pvrdma_cmd_resp *rsp;
+
+    struct pvrdma_ring *async_ring_state;
+    PvrdmaRing async;
+
+    struct pvrdma_ring *cq_ring_state;
+    PvrdmaRing cq;
+} DSRInfo;
+
+typedef struct PVRDMADev {
+    PCIDevice parent_obj;
+    MemoryRegion msix;
+    MemoryRegion regs;
+    __u32 regs_data[RDMA_BAR1_REGS_SIZE];
+    MemoryRegion uar;
+    __u32 uar_data[RDMA_BAR2_UAR_SIZE];
+    DSRInfo dsr_info;
+    int interrupt_mask;
+    struct ibv_device_attr dev_attr;
+    u64 node_guid;
+    char *backend_device_name;
+    u8 backend_gid_idx;
+    u8 backend_port_num;
+    RdmaBackendDev backend_dev;
+    RdmaDeviceResources rdma_dev_res;
+} PVRDMADev;
+#define PVRDMA_DEV(dev) OBJECT_CHECK(PVRDMADev, (dev), PVRDMA_HW_NAME)
+
+static inline int get_reg_val(PVRDMADev *dev, hwaddr addr, __u32 *val)
+{
+    int idx = addr >> 2;
+
+    if (idx > RDMA_BAR1_REGS_SIZE) {
+        return -EINVAL;
+    }
+
+    *val = dev->regs_data[idx];
+
+    return 0;
+}
+
+static inline int set_reg_val(PVRDMADev *dev, hwaddr addr, __u32 val)
+{
+    int idx = addr >> 2;
+
+    if (idx > RDMA_BAR1_REGS_SIZE) {
+        return -EINVAL;
+    }
+
+    dev->regs_data[idx] = val;
+
+    return 0;
+}
+
+static inline void post_interrupt(PVRDMADev *dev, unsigned vector)
+{
+    PCIDevice *pci_dev = PCI_DEVICE(dev);
+
+    if (likely(!dev->interrupt_mask)) {
+        msix_notify(pci_dev, vector);
+    }
+}
+
+int execute_command(PVRDMADev *dev);
+
+#endif
diff --git a/hw/rdma/vmw/pvrdma_cmd.c b/hw/rdma/vmw/pvrdma_cmd.c
new file mode 100644
index 0000000000..0cf92a8010
--- /dev/null
+++ b/hw/rdma/vmw/pvrdma_cmd.c
@@ -0,0 +1,679 @@
+#include <qemu/osdep.h>
+#include <qemu/error-report.h>
+#include <cpu.h>
+#include "hw/hw.h"
+#include "hw/pci/pci.h"
+#include "hw/pci/pci_ids.h"
+
+#include "../rdma_backend.h"
+#include "../rdma_rm.h"
+#include "../rdma_utils.h"
+
+#include "pvrdma.h"
+#include "vmw_pvrdma-abi.h"
+
+static void *pvrdma_map_to_pdir(PCIDevice *pdev, uint64_t pdir_dma,
+                                uint32_t nchunks, size_t length)
+{
+    uint64_t *dir = NULL, *tbl = NULL;
+    int tbl_idx, dir_idx, addr_idx;
+    void *host_virt = NULL, *curr_page;
+
+    if (!nchunks) {
+        pr_dbg("nchunks=0\n");
+        goto out;
+    }
+
+    dir = rdma_pci_dma_map(pdev, pdir_dma, TARGET_PAGE_SIZE);
+    if (!dir) {
+        error_report("PVRDMA: Failed to map to page directory");
+        goto out;
+    }
+
+    tbl = rdma_pci_dma_map(pdev, dir[0], TARGET_PAGE_SIZE);
+    if (!tbl) {
+        error_report("PVRDMA: Failed to map to page table 0");
+        goto out_unmap_dir;
+    }
+
+    curr_page = rdma_pci_dma_map(pdev, (dma_addr_t)tbl[0], TARGET_PAGE_SIZE);
+    if (!curr_page) {
+        error_report("PVRDMA: Failed to map the first page");
+        goto out_unmap_tbl;
+    }
+
+    host_virt = mremap(curr_page, 0, length, MREMAP_MAYMOVE);
+    if (host_virt == MAP_FAILED) {
+        host_virt = NULL;
+        error_report("PVRDMA: Failed to remap memory for host_virt");
+        goto out_unmap_tbl;
+    }
+
+    rdma_pci_dma_unmap(pdev, curr_page, TARGET_PAGE_SIZE);
+
+    pr_dbg("host_virt=%p\n", host_virt);
+
+    dir_idx = 0;
+    tbl_idx = 1;
+    addr_idx = 1;
+    while (addr_idx < nchunks) {
+        if ((tbl_idx == (TARGET_PAGE_SIZE / sizeof(uint64_t)))) {
+            tbl_idx = 0;
+            dir_idx++;
+            pr_dbg("Mapping to table %d\n", dir_idx);
+            rdma_pci_dma_unmap(pdev, tbl, TARGET_PAGE_SIZE);
+            tbl = rdma_pci_dma_map(pdev, dir[dir_idx], TARGET_PAGE_SIZE);
+            if (!tbl) {
+                error_report("PVRDMA: Failed to map to page table %d", dir_idx);
+                goto out_unmap_host_virt;
+            }
+        }
+
+        pr_dbg("guest_dma[%d]=0x%lx\n", addr_idx, tbl[tbl_idx]);
+
+        curr_page = rdma_pci_dma_map(pdev, (dma_addr_t)tbl[tbl_idx],
+                                     TARGET_PAGE_SIZE);
+        if (!curr_page) {
+            error_report("PVRDMA: Failed to map to page %d, dir %d", tbl_idx,
+                         dir_idx);
+            goto out_unmap_host_virt;
+        }
+
+        mremap(curr_page, 0, TARGET_PAGE_SIZE, MREMAP_MAYMOVE | MREMAP_FIXED,
+               host_virt + TARGET_PAGE_SIZE * addr_idx);
+
+        rdma_pci_dma_unmap(pdev, curr_page, TARGET_PAGE_SIZE);
+
+        addr_idx++;
+
+        tbl_idx++;
+    }
+
+    goto out_unmap_tbl;
+
+out_unmap_host_virt:
+    munmap(host_virt, length);
+    host_virt = NULL;
+
+out_unmap_tbl:
+    rdma_pci_dma_unmap(pdev, tbl, TARGET_PAGE_SIZE);
+
+out_unmap_dir:
+    rdma_pci_dma_unmap(pdev, dir, TARGET_PAGE_SIZE);
+
+out:
+    return host_virt;
+}
+
+static int query_port(PVRDMADev *dev, union pvrdma_cmd_req *req,
+                      union pvrdma_cmd_resp *rsp)
+{
+    struct pvrdma_cmd_query_port *cmd = &req->query_port;
+    struct pvrdma_cmd_query_port_resp *resp = &rsp->query_port_resp;
+    struct pvrdma_port_attr attrs = {0};
+
+    pr_dbg("port=%d\n", cmd->port_num);
+
+    if (rdma_backend_query_port(&dev->backend_dev,
+                                (struct ibv_port_attr *)&attrs)) {
+        return -ENOMEM;
+    }
+
+    memset(resp, 0, sizeof(*resp));
+    resp->hdr.response = cmd->hdr.response;
+    resp->hdr.ack = PVRDMA_CMD_QUERY_PORT_RESP;
+    resp->hdr.err = 0;
+
+    resp->attrs.state = attrs.state;
+    resp->attrs.max_mtu = attrs.max_mtu;
+    resp->attrs.active_mtu = attrs.active_mtu;
+    resp->attrs.phys_state = attrs.phys_state;
+    resp->attrs.gid_tbl_len = MIN(MAX_PORT_GIDS, attrs.gid_tbl_len);
+    resp->attrs.port_cap_flags = 0;
+    resp->attrs.max_msg_sz = 1024;
+    resp->attrs.bad_pkey_cntr = 0;
+    resp->attrs.qkey_viol_cntr = 0;
+    resp->attrs.pkey_tbl_len = MIN(MAX_PORT_PKEYS, attrs.pkey_tbl_len);
+    resp->attrs.lid = 0;
+    resp->attrs.sm_lid = 0;
+    resp->attrs.lmc = 0;
+    resp->attrs.max_vl_num = 0;
+    resp->attrs.sm_sl = 0;
+    resp->attrs.subnet_timeout = 0;
+    resp->attrs.init_type_reply = 0;
+    resp->attrs.active_width = 1;
+    resp->attrs.active_speed = 1;
+
+    return 0;
+}
+
+static int query_pkey(PVRDMADev *dev, union pvrdma_cmd_req *req,
+                      union pvrdma_cmd_resp *rsp)
+{
+    struct pvrdma_cmd_query_pkey *cmd = &req->query_pkey;
+    struct pvrdma_cmd_query_pkey_resp *resp = &rsp->query_pkey_resp;
+
+    pr_dbg("port=%d\n", cmd->port_num);
+    pr_dbg("index=%d\n", cmd->index);
+
+    memset(resp, 0, sizeof(*resp));
+    resp->hdr.response = cmd->hdr.response;
+    resp->hdr.ack = PVRDMA_CMD_QUERY_PKEY_RESP;
+    resp->hdr.err = 0;
+
+    resp->pkey = 0x7FFF;
+    pr_dbg("pkey=0x%x\n", resp->pkey);
+
+    return 0;
+}
+
+static int create_pd(PVRDMADev *dev, union pvrdma_cmd_req *req,
+                     union pvrdma_cmd_resp *rsp)
+{
+    struct pvrdma_cmd_create_pd *cmd = &req->create_pd;
+    struct pvrdma_cmd_create_pd_resp *resp = &rsp->create_pd_resp;
+
+    pr_dbg("context=0x%x\n", cmd->ctx_handle ? cmd->ctx_handle : 0);
+
+    memset(resp, 0, sizeof(*resp));
+    resp->hdr.response = cmd->hdr.response;
+    resp->hdr.ack = PVRDMA_CMD_CREATE_PD_RESP;
+    resp->hdr.err = rdma_rm_alloc_pd(&dev->rdma_dev_res, &dev->backend_dev,
+                                     &resp->pd_handle, cmd->ctx_handle);
+
+    pr_dbg("ret=%d\n", resp->hdr.err);
+    return resp->hdr.err;
+}
+
+static int destroy_pd(PVRDMADev *dev, union pvrdma_cmd_req *req,
+                      union pvrdma_cmd_resp *rsp)
+{
+    struct pvrdma_cmd_destroy_pd *cmd = &req->destroy_pd;
+
+    pr_dbg("pd_handle=%d\n", cmd->pd_handle);
+
+    rdma_rm_dealloc_pd(&dev->rdma_dev_res, cmd->pd_handle);
+
+    return 0;
+}
+
+static int create_mr(PVRDMADev *dev, union pvrdma_cmd_req *req,
+                     union pvrdma_cmd_resp *rsp)
+{
+    struct pvrdma_cmd_create_mr *cmd = &req->create_mr;
+    struct pvrdma_cmd_create_mr_resp *resp = &rsp->create_mr_resp;
+    PCIDevice *pci_dev = PCI_DEVICE(dev);
+    void *host_virt = 0;
+
+    memset(resp, 0, sizeof(*resp));
+    resp->hdr.response = cmd->hdr.response;
+    resp->hdr.ack = PVRDMA_CMD_CREATE_MR_RESP;
+
+    pr_dbg("pd_handle=%d\n", cmd->pd_handle);
+    pr_dbg("access_flags=0x%x\n", cmd->access_flags);
+    pr_dbg("flags=0x%x\n", cmd->flags);
+
+    if (!(cmd->flags & PVRDMA_MR_FLAG_DMA)) {
+        host_virt = pvrdma_map_to_pdir(pci_dev, cmd->pdir_dma, cmd->nchunks,
+                                       cmd->length);
+        if (!host_virt) {
+            pr_dbg("Failed to map to pdir\n");
+            resp->hdr.err = -EINVAL;
+            goto out;
+        }
+    }
+
+    resp->hdr.err = rdma_rm_alloc_mr(&dev->rdma_dev_res, cmd->pd_handle,
+                                     cmd->start, cmd->length, host_virt,
+                                     cmd->access_flags, &resp->mr_handle,
+                                     &resp->lkey, &resp->rkey);
+    if (!resp->hdr.err) {
+        munmap(host_virt, cmd->length);
+    }
+
+out:
+    pr_dbg("ret=%d\n", resp->hdr.err);
+    return resp->hdr.err;
+}
+
+static int destroy_mr(PVRDMADev *dev, union pvrdma_cmd_req *req,
+                      union pvrdma_cmd_resp *rsp)
+{
+    struct pvrdma_cmd_destroy_mr *cmd = &req->destroy_mr;
+
+    pr_dbg("mr_handle=%d\n", cmd->mr_handle);
+
+    rdma_rm_dealloc_mr(&dev->rdma_dev_res, cmd->mr_handle);
+
+    return 0;
+}
+
+static int create_cq_ring(PCIDevice *pci_dev , PvrdmaRing **ring, u64 pdir_dma,
+                          u32 nchunks, u32 cqe)
+{
+    u64 *dir, *tbl = 0;
+    PvrdmaRing *r;
+    int rc = -EINVAL;
+    char ring_name[MAX_RING_NAME_SZ];
+
+    pr_dbg("pdir_dma=0x%llx\n", (long long unsigned int)pdir_dma);
+    dir = rdma_pci_dma_map(pci_dev, pdir_dma, TARGET_PAGE_SIZE);
+    if (!dir) {
+        pr_dbg("Failed to map to CQ page directory\n");
+        goto out;
+    }
+
+    tbl = rdma_pci_dma_map(pci_dev, dir[0], TARGET_PAGE_SIZE);
+    if (!tbl) {
+        pr_dbg("Failed to map to CQ page table\n");
+        goto out;
+    }
+
+    r = malloc(sizeof(*r));
+    if (!r) {
+        pr_dbg("Fail allocate memory for CQ ring\n");
+        rc = -ENOMEM;
+        goto out;
+    }
+    *ring = r;
+
+    r->ring_state = (struct pvrdma_ring *)
+        rdma_pci_dma_map(pci_dev, tbl[0], TARGET_PAGE_SIZE);
+
+    if (!r->ring_state) {
+        pr_dbg("Failed to map to CQ ring state\n");
+        goto out_free_ring;
+    }
+
+    sprintf(ring_name, "cq_ring_%lx", pdir_dma);
+    rc = pvrdma_ring_init(r, ring_name, pci_dev, &r->ring_state[1],
+                          cqe, sizeof(struct pvrdma_cqe),
+                          /* first page is ring state */
+                          (dma_addr_t *)&tbl[1], nchunks - 1);
+    if (rc) {
+        goto out_unmap_ring_state;
+    }
+
+    goto out;
+
+out_unmap_ring_state:
+    /* ring_state was in slot 1, not 0 so need to jump back */
+    rdma_pci_dma_unmap(pci_dev, --r->ring_state, TARGET_PAGE_SIZE);
+
+out_free_ring:
+    free(r);
+    r = NULL;
+
+out:
+    rdma_pci_dma_unmap(pci_dev, tbl, TARGET_PAGE_SIZE);
+    rdma_pci_dma_unmap(pci_dev, dir, TARGET_PAGE_SIZE);
+
+    return rc;
+}
+
+static int create_cq(PVRDMADev *dev, union pvrdma_cmd_req *req,
+                     union pvrdma_cmd_resp *rsp)
+{
+    struct pvrdma_cmd_create_cq *cmd = &req->create_cq;
+    struct pvrdma_cmd_create_cq_resp *resp = &rsp->create_cq_resp;
+    PvrdmaRing *ring = NULL;
+
+    memset(resp, 0, sizeof(*resp));
+    resp->hdr.response = cmd->hdr.response;
+    resp->hdr.ack = PVRDMA_CMD_CREATE_CQ_RESP;
+
+    resp->cqe = cmd->cqe;
+
+    resp->hdr.err = create_cq_ring(PCI_DEVICE(dev), &ring, cmd->pdir_dma,
+                                   cmd->nchunks, cmd->cqe);
+    if (resp->hdr.err) {
+        goto out;
+    }
+
+    pr_dbg("ring=%p\n", ring);
+
+    resp->hdr.err = rdma_rm_alloc_cq(&dev->rdma_dev_res, &dev->backend_dev,
+                                     cmd->cqe, &resp->cq_handle, ring);
+    resp->cqe = cmd->cqe;
+
+out:
+    pr_dbg("ret=%d\n", resp->hdr.err);
+    return resp->hdr.err;
+}
+
+static int destroy_cq(PVRDMADev *dev, union pvrdma_cmd_req *req,
+                      union pvrdma_cmd_resp *rsp)
+{
+    struct pvrdma_cmd_destroy_cq *cmd = &req->destroy_cq;
+    RdmaRmCQ *cq;
+    PvrdmaRing *ring;
+
+    pr_dbg("cq_handle=%d\n", cmd->cq_handle);
+
+    cq = rdma_rm_get_cq(&dev->rdma_dev_res, cmd->cq_handle);
+    if (!cq) {
+        pr_dbg("Invalid CQ handle\n");
+        return -EINVAL;
+    }
+
+    ring = (PvrdmaRing *)cq->opaque;
+    pvrdma_ring_free(ring);
+    /* ring_state was in slot 1, not 0 so need to jump back */
+    rdma_pci_dma_unmap(PCI_DEVICE(dev), --ring->ring_state, TARGET_PAGE_SIZE);
+    free(ring);
+
+    rdma_rm_dealloc_cq(&dev->rdma_dev_res, cmd->cq_handle);
+
+    return 0;
+}
+
+static int create_qp_rings(PCIDevice *pci_dev, u64 pdir_dma, PvrdmaRing **rings,
+                           u32 scqe, u32 smax_sge, u32 spages, u32 rcqe,
+                           u32 rmax_sge, u32 rpages)
+{
+    u64 *dir, *tbl = 0;
+    PvrdmaRing *sr, *rr;
+    int rc = -EINVAL;;
+    char ring_name[MAX_RING_NAME_SZ];
+    uint32_t wqe_sz;
+
+    pr_dbg("pdir_dma=0x%llx\n", (long long unsigned int)pdir_dma);
+    dir = rdma_pci_dma_map(pci_dev, pdir_dma, TARGET_PAGE_SIZE);
+    if (!dir) {
+        pr_dbg("Failed to map to CQ page directory\n");
+        goto out;
+    }
+
+    tbl = rdma_pci_dma_map(pci_dev, dir[0], TARGET_PAGE_SIZE);
+    if (!tbl) {
+        pr_dbg("Failed to map to CQ page table\n");
+        goto out;
+    }
+
+    sr = malloc(2 * sizeof(*rr));
+    if (!sr) {
+        pr_dbg("Fail allocate memory for QP send and recv rings\n");
+        rc = -ENOMEM;
+        goto out;
+    }
+    rr = &sr[1];
+    pr_dbg("sring=%p\n", sr);
+    pr_dbg("rring=%p\n", rr);
+
+    *rings = sr;
+
+    pr_dbg("scqe=%d\n", scqe);
+    pr_dbg("smax_sge=%d\n", smax_sge);
+    pr_dbg("spages=%d\n", spages);
+    pr_dbg("rcqe=%d\n", rcqe);
+    pr_dbg("rmax_sge=%d\n", rmax_sge);
+    pr_dbg("rpages=%d\n", rpages);
+
+    /* Create send ring */
+    sr->ring_state = (struct pvrdma_ring *)
+        rdma_pci_dma_map(pci_dev, tbl[0], TARGET_PAGE_SIZE);
+    if (!sr->ring_state) {
+        pr_dbg("Failed to map to CQ ring state\n");
+        goto out_free_sr_mem;
+    }
+
+    wqe_sz = pow2roundup32(sizeof(struct pvrdma_sq_wqe_hdr) +
+                           sizeof(struct pvrdma_sge) * smax_sge - 1);
+
+    sprintf(ring_name, "qp_sring_%lx", pdir_dma);
+    rc = pvrdma_ring_init(sr, ring_name, pci_dev, sr->ring_state,
+                          scqe, wqe_sz, (dma_addr_t *)&tbl[1], spages);
+    if (rc) {
+        goto out_unmap_ring_state;
+    }
+
+    /* Create recv ring */
+    rr->ring_state = &sr->ring_state[1];
+    wqe_sz = pow2roundup32(sizeof(struct pvrdma_rq_wqe_hdr) +
+                           sizeof(struct pvrdma_sge) * rmax_sge - 1);
+    sprintf(ring_name, "qp_rring_%lx", pdir_dma);
+    rc = pvrdma_ring_init(rr, ring_name, pci_dev, rr->ring_state,
+                          rcqe, wqe_sz, (dma_addr_t *)&tbl[1 + spages], rpages);
+    if (rc) {
+        goto out_free_sr;
+    }
+
+    goto out;
+
+out_free_sr:
+    pvrdma_ring_free(sr);
+
+out_unmap_ring_state:
+    rdma_pci_dma_unmap(pci_dev, sr->ring_state, TARGET_PAGE_SIZE);
+
+out_free_sr_mem:
+    free(sr);
+    sr = NULL;
+
+out:
+    rdma_pci_dma_unmap(pci_dev, tbl, TARGET_PAGE_SIZE);
+    rdma_pci_dma_unmap(pci_dev, dir, TARGET_PAGE_SIZE);
+
+    return rc;
+}
+
+static int create_qp(PVRDMADev *dev, union pvrdma_cmd_req *req,
+                     union pvrdma_cmd_resp *rsp)
+{
+    struct pvrdma_cmd_create_qp *cmd = &req->create_qp;
+    struct pvrdma_cmd_create_qp_resp *resp = &rsp->create_qp_resp;
+    PvrdmaRing *rings = NULL;
+
+    memset(resp, 0, sizeof(*resp));
+    resp->hdr.response = cmd->hdr.response;
+    resp->hdr.ack = PVRDMA_CMD_CREATE_QP_RESP;
+
+    pr_dbg("total_chunks=%d\n", cmd->total_chunks);
+    pr_dbg("send_chunks=%d\n", cmd->send_chunks);
+
+    resp->hdr.err = create_qp_rings(PCI_DEVICE(dev), cmd->pdir_dma, &rings,
+                                    cmd->max_send_wr, cmd->max_send_sge,
+                                    cmd->send_chunks, cmd->max_recv_wr,
+                                    cmd->max_recv_sge, cmd->total_chunks -
+                                    cmd->send_chunks - 1);
+    if (resp->hdr.err) {
+        goto out;
+    }
+
+    pr_dbg("rings=%p\n", rings);
+
+    resp->hdr.err = rdma_rm_alloc_qp(&dev->rdma_dev_res, cmd->pd_handle,
+                                     cmd->qp_type, cmd->max_send_wr,
+                                     cmd->max_send_sge, cmd->send_cq_handle,
+                                     cmd->max_recv_wr, cmd->max_recv_sge,
+                                     cmd->recv_cq_handle, rings, &resp->qpn);
+
+    resp->max_send_wr = cmd->max_send_wr;
+    resp->max_recv_wr = cmd->max_recv_wr;
+    resp->max_send_sge = cmd->max_send_sge;
+    resp->max_recv_sge = cmd->max_recv_sge;
+    resp->max_inline_data = cmd->max_inline_data;
+
+out:
+    pr_dbg("ret=%d\n", resp->hdr.err);
+    return resp->hdr.err;
+}
+
+static int modify_qp(PVRDMADev *dev, union pvrdma_cmd_req *req,
+                     union pvrdma_cmd_resp *rsp)
+{
+    struct pvrdma_cmd_modify_qp *cmd = &req->modify_qp;
+
+    pr_dbg("qp_handle=%d\n", cmd->qp_handle);
+
+    memset(rsp, 0, sizeof(*rsp));
+    rsp->hdr.response = cmd->hdr.response;
+    rsp->hdr.ack = PVRDMA_CMD_MODIFY_QP_RESP;
+
+    rsp->hdr.err = rdma_rm_modify_qp(&dev->rdma_dev_res, &dev->backend_dev,
+                                 cmd->qp_handle, cmd->attr_mask,
+                                 (union ibv_gid *)&cmd->attrs.ah_attr.grh.dgid,
+                                 cmd->attrs.dest_qp_num, cmd->attrs.qp_state,
+                                 cmd->attrs.qkey, cmd->attrs.rq_psn,
+                                 cmd->attrs.sq_psn);
+
+    pr_dbg("ret=%d\n", rsp->hdr.err);
+    return rsp->hdr.err;
+}
+
+static int destroy_qp(PVRDMADev *dev, union pvrdma_cmd_req *req,
+                      union pvrdma_cmd_resp *rsp)
+{
+    struct pvrdma_cmd_destroy_qp *cmd = &req->destroy_qp;
+    RdmaRmQP *qp;
+    PvrdmaRing *ring;
+
+    qp = rdma_rm_get_qp(&dev->rdma_dev_res, cmd->qp_handle);
+    if (!qp) {
+        pr_dbg("Invalid QP handle\n");
+        return -EINVAL;
+    }
+
+    rdma_rm_dealloc_qp(&dev->rdma_dev_res, cmd->qp_handle);
+
+    ring = (PvrdmaRing *)qp->opaque;
+    pr_dbg("sring=%p\n", &ring[0]);
+    pvrdma_ring_free(&ring[0]);
+    pr_dbg("rring=%p\n", &ring[1]);
+    pvrdma_ring_free(&ring[1]);
+
+    rdma_pci_dma_unmap(PCI_DEVICE(dev), ring->ring_state, TARGET_PAGE_SIZE);
+    free(ring);
+
+    return 0;
+}
+
+static int create_bind(PVRDMADev *dev, union pvrdma_cmd_req *req,
+                       union pvrdma_cmd_resp *rsp)
+{
+    struct pvrdma_cmd_create_bind *cmd = &req->create_bind;
+#ifdef PVRDMA_DEBUG
+    __be64 *subnet = (__be64 *)&cmd->new_gid[0];
+    __be64 *if_id = (__be64 *)&cmd->new_gid[8];
+#endif
+
+    pr_dbg("index=%d\n", cmd->index);
+
+    if (cmd->index > MAX_PORT_GIDS) {
+        return -EINVAL;
+    }
+
+    pr_dbg("gid[%d]=0x%llx,0x%llx\n", cmd->index,
+           (long long unsigned int)be64_to_cpu(*subnet),
+           (long long unsigned int)be64_to_cpu(*if_id));
+
+    /* Driver forces to one port only */
+    memcpy(dev->rdma_dev_res.ports[0].gid_tbl[cmd->index].raw, &cmd->new_gid,
+           sizeof(cmd->new_gid));
+
+    /* TODO: Since drivers stores node_guid at load_dsr phase then this
+     * assignment is not relevant, i need to figure out a way how to
+     * retrieve MAC of our netdev */
+    dev->node_guid = dev->rdma_dev_res.ports[0].gid_tbl[0].global.interface_id;
+    pr_dbg("dev->node_guid=0x%llx\n",
+           (long long unsigned int)be64_to_cpu(dev->node_guid));
+
+    return 0;
+}
+
+static int destroy_bind(PVRDMADev *dev, union pvrdma_cmd_req *req,
+                        union pvrdma_cmd_resp *rsp)
+{
+    struct pvrdma_cmd_destroy_bind *cmd = &req->destroy_bind;
+
+    pr_dbg("clear index %d\n", cmd->index);
+
+    memset(dev->rdma_dev_res.ports[0].gid_tbl[cmd->index].raw, 0,
+           sizeof(dev->rdma_dev_res.ports[0].gid_tbl[cmd->index].raw));
+
+    return 0;
+}
+
+static int create_uc(PVRDMADev *dev, union pvrdma_cmd_req *req,
+                     union pvrdma_cmd_resp *rsp)
+{
+    struct pvrdma_cmd_create_uc *cmd = &req->create_uc;
+    struct pvrdma_cmd_create_uc_resp *resp = &rsp->create_uc_resp;
+
+    pr_dbg("pfn=%d\n", cmd->pfn);
+
+    memset(resp, 0, sizeof(*resp));
+    resp->hdr.response = cmd->hdr.response;
+    resp->hdr.ack = PVRDMA_CMD_CREATE_UC_RESP;
+    resp->hdr.err = rdma_rm_alloc_uc(&dev->rdma_dev_res, cmd->pfn,
+                                     &resp->ctx_handle);
+
+    pr_dbg("ret=%d\n", resp->hdr.err);
+
+    return 0;
+}
+
+static int destroy_uc(PVRDMADev *dev, union pvrdma_cmd_req *req,
+                      union pvrdma_cmd_resp *rsp)
+{
+    struct pvrdma_cmd_destroy_uc *cmd = &req->destroy_uc;
+
+    pr_dbg("ctx_handle=%d\n", cmd->ctx_handle);
+
+    rdma_rm_dealloc_uc(&dev->rdma_dev_res, cmd->ctx_handle);
+
+    return 0;
+}
+struct cmd_handler {
+    __u32 cmd;
+    int (*exec)(PVRDMADev *dev, union pvrdma_cmd_req *req,
+            union pvrdma_cmd_resp *rsp);
+};
+
+static struct cmd_handler cmd_handlers[] = {
+    {PVRDMA_CMD_QUERY_PORT, query_port},
+    {PVRDMA_CMD_QUERY_PKEY, query_pkey},
+    {PVRDMA_CMD_CREATE_PD, create_pd},
+    {PVRDMA_CMD_DESTROY_PD, destroy_pd},
+    {PVRDMA_CMD_CREATE_MR, create_mr},
+    {PVRDMA_CMD_DESTROY_MR, destroy_mr},
+    {PVRDMA_CMD_CREATE_CQ, create_cq},
+    {PVRDMA_CMD_RESIZE_CQ, NULL},
+    {PVRDMA_CMD_DESTROY_CQ, destroy_cq},
+    {PVRDMA_CMD_CREATE_QP, create_qp},
+    {PVRDMA_CMD_MODIFY_QP, modify_qp},
+    {PVRDMA_CMD_QUERY_QP, NULL},
+    {PVRDMA_CMD_DESTROY_QP, destroy_qp},
+    {PVRDMA_CMD_CREATE_UC, create_uc},
+    {PVRDMA_CMD_DESTROY_UC, destroy_uc},
+    {PVRDMA_CMD_CREATE_BIND, create_bind},
+    {PVRDMA_CMD_DESTROY_BIND, destroy_bind},
+};
+
+int execute_command(PVRDMADev *dev)
+{
+    int err = 0xFFFF;
+    DSRInfo *dsr_info;
+
+    dsr_info = &dev->dsr_info;
+
+    pr_dbg("cmd=%d\n", dsr_info->req->hdr.cmd);
+    if (dsr_info->req->hdr.cmd >= sizeof(cmd_handlers) /
+                      sizeof(struct cmd_handler)) {
+        pr_dbg("Unsupported command\n");
+        goto out;
+    }
+
+    if (!cmd_handlers[dsr_info->req->hdr.cmd].exec) {
+        pr_dbg("Unsupported command (not implemented yet)\n");
+        goto out;
+    }
+
+    err = cmd_handlers[dsr_info->req->hdr.cmd].exec(dev, dsr_info->req,
+                            dsr_info->rsp);
+out:
+    set_reg_val(dev, PVRDMA_REG_ERR, err);
+    post_interrupt(dev, INTR_VEC_CMD_RING);
+
+    return (err == 0) ? 0 : -EINVAL;
+}
diff --git a/hw/rdma/vmw/pvrdma_dev_api.h b/hw/rdma/vmw/pvrdma_dev_api.h
new file mode 100644
index 0000000000..bf1986a976
--- /dev/null
+++ b/hw/rdma/vmw/pvrdma_dev_api.h
@@ -0,0 +1,602 @@
+/*
+ * QEMU VMWARE paravirtual RDMA device definitions
+ *
+ * Copyright (C) 2018 Oracle
+ * Copyright (C) 2018 Red Hat Inc
+ *
+ * Authors:
+ *     Yuval Shaia <yuval.shaia@oracle.com>
+ *     Marcel Apfelbaum <marcel@redhat.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef PVRDMA_DEV_API_H
+#define PVRDMA_DEV_API_H
+
+/*
+ * Following is an interface definition for PVRDMA device as provided by
+ * VMWARE.
+ * See original copyright from Linux kernel v4.14.5 header file
+ * drivers/infiniband/hw/vmw_pvrdma/pvrdma_dev_api.h
+ */
+
+/*
+ * Copyright (c) 2012-2016 VMware, Inc.  All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of EITHER the GNU General Public License
+ * version 2 as published by the Free Software Foundation or the BSD
+ * 2-Clause License. This program is distributed in the hope that it
+ * will be useful, but WITHOUT ANY WARRANTY; WITHOUT EVEN THE IMPLIED
+ * WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
+ * See the GNU General Public License version 2 for more details at
+ * http://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program available in the file COPYING in the main
+ * directory of this source tree.
+ *
+ * The BSD 2-Clause License
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+ * COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
+ * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+ * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+ * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
+ * OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <linux/types.h>
+
+#include "pvrdma_ib_verbs.h"
+
+#define PVRDMA_VERSION            17
+#define PVRDMA_BOARD_ID            1
+#define PVRDMA_REV_ID            1
+
+/*
+ * Masks and accessors for page directory, which is a two-level lookup:
+ * page directory -> page table -> page. Only one directory for now, but we
+ * could expand that easily. 9 bits for tables, 9 bits for pages, gives one
+ * gigabyte for memory regions and so forth.
+ */
+
+#define PVRDMA_PDIR_SHIFT        18
+#define PVRDMA_PTABLE_SHIFT        9
+#define PVRDMA_PAGE_DIR_DIR(x)        (((x) >> PVRDMA_PDIR_SHIFT) & 0x1)
+#define PVRDMA_PAGE_DIR_TABLE(x)    (((x) >> PVRDMA_PTABLE_SHIFT) & 0x1ff)
+#define PVRDMA_PAGE_DIR_PAGE(x)        ((x) & 0x1ff)
+#define PVRDMA_PAGE_DIR_MAX_PAGES    (1 * 512 * 512)
+#define PVRDMA_MAX_FAST_REG_PAGES    128
+
+/*
+ * Max MSI-X vectors.
+ */
+
+#define PVRDMA_MAX_INTERRUPTS    3
+
+/* Register offsets within PCI resource on BAR1. */
+#define PVRDMA_REG_VERSION    0x00    /* R: Version of device. */
+#define PVRDMA_REG_DSRLOW    0x04    /* W: Device shared region low PA. */
+#define PVRDMA_REG_DSRHIGH    0x08    /* W: Device shared region high PA. */
+#define PVRDMA_REG_CTL        0x0c    /* W: PVRDMA_DEVICE_CTL */
+#define PVRDMA_REG_REQUEST    0x10    /* W: Indicate device request. */
+#define PVRDMA_REG_ERR        0x14    /* R: Device error. */
+#define PVRDMA_REG_ICR        0x18    /* R: Interrupt cause. */
+#define PVRDMA_REG_IMR        0x1c    /* R/W: Interrupt mask. */
+#define PVRDMA_REG_MACL        0x20    /* R/W: MAC address low. */
+#define PVRDMA_REG_MACH        0x24    /* R/W: MAC address high. */
+
+/* Object flags. */
+#define PVRDMA_CQ_FLAG_ARMED_SOL    BIT(0)    /* Armed for solicited-only. */
+#define PVRDMA_CQ_FLAG_ARMED        BIT(1)    /* Armed. */
+#define PVRDMA_MR_FLAG_DMA        BIT(0)    /* DMA region. */
+#define PVRDMA_MR_FLAG_FRMR        BIT(1)    /* Fast reg memory region. */
+
+/*
+ * Atomic operation capability (masked versions are extended atomic
+ * operations.
+ */
+
+#define PVRDMA_ATOMIC_OP_COMP_SWAP    BIT(0)    /* Compare and swap. */
+#define PVRDMA_ATOMIC_OP_FETCH_ADD    BIT(1)    /* Fetch and add. */
+#define PVRDMA_ATOMIC_OP_MASK_COMP_SWAP    BIT(2) /* Masked compare and swap. */
+#define PVRDMA_ATOMIC_OP_MASK_FETCH_ADD    BIT(3) /* Masked fetch and add. */
+
+/*
+ * Base Memory Management Extension flags to support Fast Reg Memory Regions
+ * and Fast Reg Work Requests. Each flag represents a verb operation and we
+ * must support all of them to qualify for the BMME device cap.
+ */
+
+#define PVRDMA_BMME_FLAG_LOCAL_INV    BIT(0)    /* Local Invalidate. */
+#define PVRDMA_BMME_FLAG_REMOTE_INV    BIT(1)    /* Remote Invalidate. */
+#define PVRDMA_BMME_FLAG_FAST_REG_WR    BIT(2)    /* Fast Reg Work Request. */
+
+/*
+ * GID types. The interpretation of the gid_types bit field in the device
+ * capabilities will depend on the device mode. For now, the device only
+ * supports RoCE as mode, so only the different GID types for RoCE are
+ * defined.
+ */
+
+#define PVRDMA_GID_TYPE_FLAG_ROCE_V1    BIT(0)
+#define PVRDMA_GID_TYPE_FLAG_ROCE_V2    BIT(1)
+
+enum pvrdma_pci_resource {
+    PVRDMA_PCI_RESOURCE_MSIX,    /* BAR0: MSI-X, MMIO. */
+    PVRDMA_PCI_RESOURCE_REG,    /* BAR1: Registers, MMIO. */
+    PVRDMA_PCI_RESOURCE_UAR,    /* BAR2: UAR pages, MMIO, 64-bit. */
+    PVRDMA_PCI_RESOURCE_LAST,    /* Last. */
+};
+
+enum pvrdma_device_ctl {
+    PVRDMA_DEVICE_CTL_ACTIVATE,    /* Activate device. */
+    PVRDMA_DEVICE_CTL_UNQUIESCE,    /* Unquiesce device. */
+    PVRDMA_DEVICE_CTL_RESET,    /* Reset device. */
+};
+
+enum pvrdma_intr_vector {
+    PVRDMA_INTR_VECTOR_RESPONSE,    /* Command response. */
+    PVRDMA_INTR_VECTOR_ASYNC,    /* Async events. */
+    PVRDMA_INTR_VECTOR_CQ,        /* CQ notification. */
+    /* Additional CQ notification vectors. */
+};
+
+enum pvrdma_intr_cause {
+    PVRDMA_INTR_CAUSE_RESPONSE    = (1 << PVRDMA_INTR_VECTOR_RESPONSE),
+    PVRDMA_INTR_CAUSE_ASYNC        = (1 << PVRDMA_INTR_VECTOR_ASYNC),
+    PVRDMA_INTR_CAUSE_CQ        = (1 << PVRDMA_INTR_VECTOR_CQ),
+};
+
+enum pvrdma_gos_bits {
+    PVRDMA_GOS_BITS_UNK,        /* Unknown. */
+    PVRDMA_GOS_BITS_32,        /* 32-bit. */
+    PVRDMA_GOS_BITS_64,        /* 64-bit. */
+};
+
+enum pvrdma_gos_type {
+    PVRDMA_GOS_TYPE_UNK,        /* Unknown. */
+    PVRDMA_GOS_TYPE_LINUX,        /* Linux. */
+};
+
+enum pvrdma_device_mode {
+    PVRDMA_DEVICE_MODE_ROCE,    /* RoCE. */
+    PVRDMA_DEVICE_MODE_IWARP,    /* iWarp. */
+    PVRDMA_DEVICE_MODE_IB,        /* InfiniBand. */
+};
+
+struct pvrdma_gos_info {
+    u32 gos_bits:2;            /* W: PVRDMA_GOS_BITS_ */
+    u32 gos_type:4;            /* W: PVRDMA_GOS_TYPE_ */
+    u32 gos_ver:16;            /* W: Guest OS version. */
+    u32 gos_misc:10;        /* W: Other. */
+    u32 pad;            /* Pad to 8-byte alignment. */
+};
+
+struct pvrdma_device_caps {
+    u64 fw_ver;                /* R: Query device. */
+    __be64 node_guid;
+    __be64 sys_image_guid;
+    u64 max_mr_size;
+    u64 page_size_cap;
+    u64 atomic_arg_sizes;            /* EX verbs. */
+    u32 ex_comp_mask;            /* EX verbs. */
+    u32 device_cap_flags2;            /* EX verbs. */
+    u32 max_fa_bit_boundary;        /* EX verbs. */
+    u32 log_max_atomic_inline_arg;        /* EX verbs. */
+    u32 vendor_id;
+    u32 vendor_part_id;
+    u32 hw_ver;
+    u32 max_qp;
+    u32 max_qp_wr;
+    u32 device_cap_flags;
+    u32 max_sge;
+    u32 max_sge_rd;
+    u32 max_cq;
+    u32 max_cqe;
+    u32 max_mr;
+    u32 max_pd;
+    u32 max_qp_rd_atom;
+    u32 max_ee_rd_atom;
+    u32 max_res_rd_atom;
+    u32 max_qp_init_rd_atom;
+    u32 max_ee_init_rd_atom;
+    u32 max_ee;
+    u32 max_rdd;
+    u32 max_mw;
+    u32 max_raw_ipv6_qp;
+    u32 max_raw_ethy_qp;
+    u32 max_mcast_grp;
+    u32 max_mcast_qp_attach;
+    u32 max_total_mcast_qp_attach;
+    u32 max_ah;
+    u32 max_fmr;
+    u32 max_map_per_fmr;
+    u32 max_srq;
+    u32 max_srq_wr;
+    u32 max_srq_sge;
+    u32 max_uar;
+    u32 gid_tbl_len;
+    u16 max_pkeys;
+    u8  local_ca_ack_delay;
+    u8  phys_port_cnt;
+    u8  mode;                /* PVRDMA_DEVICE_MODE_ */
+    u8  atomic_ops;                /* PVRDMA_ATOMIC_OP_* bits */
+    u8  bmme_flags;                /* FRWR Mem Mgmt Extensions */
+    u8  gid_types;                /* PVRDMA_GID_TYPE_FLAG_ */
+    u8  reserved[4];
+};
+
+struct pvrdma_ring_page_info {
+    u32 num_pages;                /* Num pages incl. header. */
+    u32 reserved;                /* Reserved. */
+    u64 pdir_dma;                /* Page directory PA. */
+};
+
+#pragma pack(push, 1)
+
+struct pvrdma_device_shared_region {
+    u32 driver_version;            /* W: Driver version. */
+    u32 pad;                /* Pad to 8-byte align. */
+    struct pvrdma_gos_info gos_info;    /* W: Guest OS information. */
+    u64 cmd_slot_dma;            /* W: Command slot address. */
+    u64 resp_slot_dma;            /* W: Response slot address. */
+    struct pvrdma_ring_page_info async_ring_pages;
+                        /* W: Async ring page info. */
+    struct pvrdma_ring_page_info cq_ring_pages;
+                        /* W: CQ ring page info. */
+    u32 uar_pfn;                /* W: UAR pageframe. */
+    u32 pad2;                /* Pad to 8-byte align. */
+    struct pvrdma_device_caps caps;        /* R: Device capabilities. */
+};
+
+#pragma pack(pop)
+
+/* Event types. Currently a 1:1 mapping with enum ib_event. */
+enum pvrdma_eqe_type {
+    PVRDMA_EVENT_CQ_ERR,
+    PVRDMA_EVENT_QP_FATAL,
+    PVRDMA_EVENT_QP_REQ_ERR,
+    PVRDMA_EVENT_QP_ACCESS_ERR,
+    PVRDMA_EVENT_COMM_EST,
+    PVRDMA_EVENT_SQ_DRAINED,
+    PVRDMA_EVENT_PATH_MIG,
+    PVRDMA_EVENT_PATH_MIG_ERR,
+    PVRDMA_EVENT_DEVICE_FATAL,
+    PVRDMA_EVENT_PORT_ACTIVE,
+    PVRDMA_EVENT_PORT_ERR,
+    PVRDMA_EVENT_LID_CHANGE,
+    PVRDMA_EVENT_PKEY_CHANGE,
+    PVRDMA_EVENT_SM_CHANGE,
+    PVRDMA_EVENT_SRQ_ERR,
+    PVRDMA_EVENT_SRQ_LIMIT_REACHED,
+    PVRDMA_EVENT_QP_LAST_WQE_REACHED,
+    PVRDMA_EVENT_CLIENT_REREGISTER,
+    PVRDMA_EVENT_GID_CHANGE,
+};
+
+/* Event queue element. */
+struct pvrdma_eqe {
+    u32 type;    /* Event type. */
+    u32 info;    /* Handle, other. */
+};
+
+/* CQ notification queue element. */
+struct pvrdma_cqne {
+    u32 info;    /* Handle */
+};
+
+enum {
+    PVRDMA_CMD_FIRST,
+    PVRDMA_CMD_QUERY_PORT = PVRDMA_CMD_FIRST,
+    PVRDMA_CMD_QUERY_PKEY,
+    PVRDMA_CMD_CREATE_PD,
+    PVRDMA_CMD_DESTROY_PD,
+    PVRDMA_CMD_CREATE_MR,
+    PVRDMA_CMD_DESTROY_MR,
+    PVRDMA_CMD_CREATE_CQ,
+    PVRDMA_CMD_RESIZE_CQ,
+    PVRDMA_CMD_DESTROY_CQ,
+    PVRDMA_CMD_CREATE_QP,
+    PVRDMA_CMD_MODIFY_QP,
+    PVRDMA_CMD_QUERY_QP,
+    PVRDMA_CMD_DESTROY_QP,
+    PVRDMA_CMD_CREATE_UC,
+    PVRDMA_CMD_DESTROY_UC,
+    PVRDMA_CMD_CREATE_BIND,
+    PVRDMA_CMD_DESTROY_BIND,
+    PVRDMA_CMD_MAX,
+};
+
+enum {
+    PVRDMA_CMD_FIRST_RESP = (1 << 31),
+    PVRDMA_CMD_QUERY_PORT_RESP = PVRDMA_CMD_FIRST_RESP,
+    PVRDMA_CMD_QUERY_PKEY_RESP,
+    PVRDMA_CMD_CREATE_PD_RESP,
+    PVRDMA_CMD_DESTROY_PD_RESP_NOOP,
+    PVRDMA_CMD_CREATE_MR_RESP,
+    PVRDMA_CMD_DESTROY_MR_RESP_NOOP,
+    PVRDMA_CMD_CREATE_CQ_RESP,
+    PVRDMA_CMD_RESIZE_CQ_RESP,
+    PVRDMA_CMD_DESTROY_CQ_RESP_NOOP,
+    PVRDMA_CMD_CREATE_QP_RESP,
+    PVRDMA_CMD_MODIFY_QP_RESP,
+    PVRDMA_CMD_QUERY_QP_RESP,
+    PVRDMA_CMD_DESTROY_QP_RESP,
+    PVRDMA_CMD_CREATE_UC_RESP,
+    PVRDMA_CMD_DESTROY_UC_RESP_NOOP,
+    PVRDMA_CMD_CREATE_BIND_RESP_NOOP,
+    PVRDMA_CMD_DESTROY_BIND_RESP_NOOP,
+    PVRDMA_CMD_MAX_RESP,
+};
+
+struct pvrdma_cmd_hdr {
+    u64 response;        /* Key for response lookup. */
+    u32 cmd;        /* PVRDMA_CMD_ */
+    u32 reserved;        /* Reserved. */
+};
+
+struct pvrdma_cmd_resp_hdr {
+    u64 response;        /* From cmd hdr. */
+    u32 ack;        /* PVRDMA_CMD_XXX_RESP */
+    u8 err;            /* Error. */
+    u8 reserved[3];        /* Reserved. */
+};
+
+struct pvrdma_cmd_query_port {
+    struct pvrdma_cmd_hdr hdr;
+    u8 port_num;
+    u8 reserved[7];
+};
+
+struct pvrdma_cmd_query_port_resp {
+    struct pvrdma_cmd_resp_hdr hdr;
+    struct pvrdma_port_attr attrs;
+};
+
+struct pvrdma_cmd_query_pkey {
+    struct pvrdma_cmd_hdr hdr;
+    u8 port_num;
+    u8 index;
+    u8 reserved[6];
+};
+
+struct pvrdma_cmd_query_pkey_resp {
+    struct pvrdma_cmd_resp_hdr hdr;
+    u16 pkey;
+    u8 reserved[6];
+};
+
+struct pvrdma_cmd_create_uc {
+    struct pvrdma_cmd_hdr hdr;
+    u32 pfn; /* UAR page frame number */
+    u8 reserved[4];
+};
+
+struct pvrdma_cmd_create_uc_resp {
+    struct pvrdma_cmd_resp_hdr hdr;
+    u32 ctx_handle;
+    u8 reserved[4];
+};
+
+struct pvrdma_cmd_destroy_uc {
+    struct pvrdma_cmd_hdr hdr;
+    u32 ctx_handle;
+    u8 reserved[4];
+};
+
+struct pvrdma_cmd_create_pd {
+    struct pvrdma_cmd_hdr hdr;
+    u32 ctx_handle;
+    u8 reserved[4];
+};
+
+struct pvrdma_cmd_create_pd_resp {
+    struct pvrdma_cmd_resp_hdr hdr;
+    u32 pd_handle;
+    u8 reserved[4];
+};
+
+struct pvrdma_cmd_destroy_pd {
+    struct pvrdma_cmd_hdr hdr;
+    u32 pd_handle;
+    u8 reserved[4];
+};
+
+struct pvrdma_cmd_create_mr {
+    struct pvrdma_cmd_hdr hdr;
+    u64 start;
+    u64 length;
+    u64 pdir_dma;
+    u32 pd_handle;
+    u32 access_flags;
+    u32 flags;
+    u32 nchunks;
+};
+
+struct pvrdma_cmd_create_mr_resp {
+    struct pvrdma_cmd_resp_hdr hdr;
+    u32 mr_handle;
+    u32 lkey;
+    u32 rkey;
+    u8 reserved[4];
+};
+
+struct pvrdma_cmd_destroy_mr {
+    struct pvrdma_cmd_hdr hdr;
+    u32 mr_handle;
+    u8 reserved[4];
+};
+
+struct pvrdma_cmd_create_cq {
+    struct pvrdma_cmd_hdr hdr;
+    u64 pdir_dma;
+    u32 ctx_handle;
+    u32 cqe;
+    u32 nchunks;
+    u8 reserved[4];
+};
+
+struct pvrdma_cmd_create_cq_resp {
+    struct pvrdma_cmd_resp_hdr hdr;
+    u32 cq_handle;
+    u32 cqe;
+};
+
+struct pvrdma_cmd_resize_cq {
+    struct pvrdma_cmd_hdr hdr;
+    u32 cq_handle;
+    u32 cqe;
+};
+
+struct pvrdma_cmd_resize_cq_resp {
+    struct pvrdma_cmd_resp_hdr hdr;
+    u32 cqe;
+    u8 reserved[4];
+};
+
+struct pvrdma_cmd_destroy_cq {
+    struct pvrdma_cmd_hdr hdr;
+    u32 cq_handle;
+    u8 reserved[4];
+};
+
+struct pvrdma_cmd_create_qp {
+    struct pvrdma_cmd_hdr hdr;
+    u64 pdir_dma;
+    u32 pd_handle;
+    u32 send_cq_handle;
+    u32 recv_cq_handle;
+    u32 srq_handle;
+    u32 max_send_wr;
+    u32 max_recv_wr;
+    u32 max_send_sge;
+    u32 max_recv_sge;
+    u32 max_inline_data;
+    u32 lkey;
+    u32 access_flags;
+    u16 total_chunks;
+    u16 send_chunks;
+    u16 max_atomic_arg;
+    u8 sq_sig_all;
+    u8 qp_type;
+    u8 is_srq;
+    u8 reserved[3];
+};
+
+struct pvrdma_cmd_create_qp_resp {
+    struct pvrdma_cmd_resp_hdr hdr;
+    u32 qpn;
+    u32 max_send_wr;
+    u32 max_recv_wr;
+    u32 max_send_sge;
+    u32 max_recv_sge;
+    u32 max_inline_data;
+};
+
+struct pvrdma_cmd_modify_qp {
+    struct pvrdma_cmd_hdr hdr;
+    u32 qp_handle;
+    u32 attr_mask;
+    struct pvrdma_qp_attr attrs;
+};
+
+struct pvrdma_cmd_query_qp {
+    struct pvrdma_cmd_hdr hdr;
+    u32 qp_handle;
+    u32 attr_mask;
+};
+
+struct pvrdma_cmd_query_qp_resp {
+    struct pvrdma_cmd_resp_hdr hdr;
+    struct pvrdma_qp_attr attrs;
+};
+
+struct pvrdma_cmd_destroy_qp {
+    struct pvrdma_cmd_hdr hdr;
+    u32 qp_handle;
+    u8 reserved[4];
+};
+
+struct pvrdma_cmd_destroy_qp_resp {
+    struct pvrdma_cmd_resp_hdr hdr;
+    u32 events_reported;
+    u8 reserved[4];
+};
+
+struct pvrdma_cmd_create_bind {
+    struct pvrdma_cmd_hdr hdr;
+    u32 mtu;
+    u32 vlan;
+    u32 index;
+    u8 new_gid[16];
+    u8 gid_type;
+    u8 reserved[3];
+};
+
+struct pvrdma_cmd_destroy_bind {
+    struct pvrdma_cmd_hdr hdr;
+    u32 index;
+    u8 dest_gid[16];
+    u8 reserved[4];
+};
+
+union pvrdma_cmd_req {
+    struct pvrdma_cmd_hdr hdr;
+    struct pvrdma_cmd_query_port query_port;
+    struct pvrdma_cmd_query_pkey query_pkey;
+    struct pvrdma_cmd_create_uc create_uc;
+    struct pvrdma_cmd_destroy_uc destroy_uc;
+    struct pvrdma_cmd_create_pd create_pd;
+    struct pvrdma_cmd_destroy_pd destroy_pd;
+    struct pvrdma_cmd_create_mr create_mr;
+    struct pvrdma_cmd_destroy_mr destroy_mr;
+    struct pvrdma_cmd_create_cq create_cq;
+    struct pvrdma_cmd_resize_cq resize_cq;
+    struct pvrdma_cmd_destroy_cq destroy_cq;
+    struct pvrdma_cmd_create_qp create_qp;
+    struct pvrdma_cmd_modify_qp modify_qp;
+    struct pvrdma_cmd_query_qp query_qp;
+    struct pvrdma_cmd_destroy_qp destroy_qp;
+    struct pvrdma_cmd_create_bind create_bind;
+    struct pvrdma_cmd_destroy_bind destroy_bind;
+};
+
+union pvrdma_cmd_resp {
+    struct pvrdma_cmd_resp_hdr hdr;
+    struct pvrdma_cmd_query_port_resp query_port_resp;
+    struct pvrdma_cmd_query_pkey_resp query_pkey_resp;
+    struct pvrdma_cmd_create_uc_resp create_uc_resp;
+    struct pvrdma_cmd_create_pd_resp create_pd_resp;
+    struct pvrdma_cmd_create_mr_resp create_mr_resp;
+    struct pvrdma_cmd_create_cq_resp create_cq_resp;
+    struct pvrdma_cmd_resize_cq_resp resize_cq_resp;
+    struct pvrdma_cmd_create_qp_resp create_qp_resp;
+    struct pvrdma_cmd_query_qp_resp query_qp_resp;
+    struct pvrdma_cmd_destroy_qp_resp destroy_qp_resp;
+};
+
+#endif /* PVRDMA_DEV_API_H */
diff --git a/hw/rdma/vmw/pvrdma_dev_ring.c b/hw/rdma/vmw/pvrdma_dev_ring.c
new file mode 100644
index 0000000000..e348db028d
--- /dev/null
+++ b/hw/rdma/vmw/pvrdma_dev_ring.c
@@ -0,0 +1,139 @@
+#include <qemu/osdep.h>
+#include <hw/pci/pci.h>
+#include <cpu.h>
+
+#include "../rdma_utils.h"
+#include "pvrdma_ring.h"
+#include "pvrdma_dev_ring.h"
+
+int pvrdma_ring_init(PvrdmaRing *ring, const char *name, PCIDevice *dev,
+                     struct pvrdma_ring *ring_state, uint32_t max_elems,
+                     size_t elem_sz, dma_addr_t *tbl, dma_addr_t npages)
+{
+    int i;
+    int rc = 0;
+
+    strncpy(ring->name, name, MAX_RING_NAME_SZ);
+    ring->name[MAX_RING_NAME_SZ - 1] = 0;
+    pr_dbg("Initializing %s ring\n", ring->name);
+    ring->dev = dev;
+    ring->ring_state = ring_state;
+    ring->max_elems = max_elems;
+    ring->elem_sz = elem_sz;
+    pr_dbg("ring->elem_sz=%ld\n", ring->elem_sz);
+    pr_dbg("npages=%ld\n", npages);
+    /* TODO: Give a moment to think if we want to redo driver settings
+    atomic_set(&ring->ring_state->prod_tail, 0);
+    atomic_set(&ring->ring_state->cons_head, 0);
+    */
+    ring->npages = npages;
+    ring->pages = malloc(npages * sizeof(void *));
+    for (i = 0; i < npages; i++) {
+        if (!tbl[i]) {
+            pr_err("npages=%ld but tbl[%d] is NULL\n", (long)npages, i);
+            continue;
+        }
+
+        ring->pages[i] = rdma_pci_dma_map(dev, tbl[i], TARGET_PAGE_SIZE);
+        if (!ring->pages[i]) {
+            rc = -ENOMEM;
+            pr_dbg("Failed to map to page %d\n", i);
+            goto out_free;
+        }
+        memset(ring->pages[i], 0, TARGET_PAGE_SIZE);
+    }
+
+    goto out;
+
+out_free:
+    while (i--) {
+        rdma_pci_dma_unmap(dev, ring->pages[i], TARGET_PAGE_SIZE);
+    }
+    free(ring->pages);
+
+out:
+    return rc;
+}
+
+void *pvrdma_ring_next_elem_read(PvrdmaRing *ring)
+{
+    unsigned int idx = 0, offset;
+
+    /*
+    pr_dbg("%s: t=%d, h=%d\n", ring->name, ring->ring_state->prod_tail,
+           ring->ring_state->cons_head);
+    */
+
+    if (!pvrdma_idx_ring_has_data(ring->ring_state, ring->max_elems, &idx)) {
+        pr_dbg("No more data in ring\n");
+        return NULL;
+    }
+
+    offset = idx * ring->elem_sz;
+    /*
+    pr_dbg("idx=%d\n", idx);
+    pr_dbg("offset=%d\n", offset);
+    */
+    return ring->pages[offset / TARGET_PAGE_SIZE] + (offset % TARGET_PAGE_SIZE);
+}
+
+void pvrdma_ring_read_inc(PvrdmaRing *ring)
+{
+    pvrdma_idx_ring_inc(&ring->ring_state->cons_head, ring->max_elems);
+    /*
+    pr_dbg("%s: t=%d, h=%d, m=%ld\n", ring->name,
+           ring->ring_state->prod_tail, ring->ring_state->cons_head,
+           ring->max_elems);
+    */
+}
+
+void *pvrdma_ring_next_elem_write(PvrdmaRing *ring)
+{
+    unsigned int idx, offset, tail;
+
+    /*
+    pr_dbg("%s: t=%d, h=%d\n", ring->name, ring->ring_state->prod_tail,
+           ring->ring_state->cons_head);
+    */
+
+    if (!pvrdma_idx_ring_has_space(ring->ring_state, ring->max_elems, &tail)) {
+        pr_dbg("CQ is full\n");
+        return NULL;
+    }
+
+    idx = pvrdma_idx(&ring->ring_state->prod_tail, ring->max_elems);
+    /* TODO: tail == idx */
+
+    offset = idx * ring->elem_sz;
+    return ring->pages[offset / TARGET_PAGE_SIZE] + (offset % TARGET_PAGE_SIZE);
+}
+
+void pvrdma_ring_write_inc(PvrdmaRing *ring)
+{
+    pvrdma_idx_ring_inc(&ring->ring_state->prod_tail, ring->max_elems);
+    /*
+    pr_dbg("%s: t=%d, h=%d, m=%ld\n", ring->name,
+           ring->ring_state->prod_tail, ring->ring_state->cons_head,
+           ring->max_elems);
+    */
+}
+
+void pvrdma_ring_free(PvrdmaRing *ring)
+{
+    if (!ring) {
+        return;
+    }
+
+    if (!ring->pages) {
+        return;
+    }
+
+    pr_dbg("ring->npages=%d\n", ring->npages);
+    while (ring->npages--) {
+        rdma_pci_dma_unmap(ring->dev, ring->pages[ring->npages],
+                           TARGET_PAGE_SIZE);
+    }
+
+    free(ring->pages);
+    ring->pages = NULL;
+}
diff --git a/hw/rdma/vmw/pvrdma_dev_ring.h b/hw/rdma/vmw/pvrdma_dev_ring.h
new file mode 100644
index 0000000000..26188b3543
--- /dev/null
+++ b/hw/rdma/vmw/pvrdma_dev_ring.h
@@ -0,0 +1,42 @@
+/*
+ * QEMU VMWARE paravirtual RDMA ring utilities
+ *
+ * Copyright (C) 2018 Oracle
+ * Copyright (C) 2018 Red Hat Inc
+ *
+ * Authors:
+ *     Yuval Shaia <yuval.shaia@oracle.com>
+ *     Marcel Apfelbaum <marcel@redhat.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef PVRDMA_DEV_RING_H
+#define PVRDMA_DEV_RING_H
+
+#include <qemu/typedefs.h>
+
+#define MAX_RING_NAME_SZ 32
+
+typedef struct PvrdmaRing {
+    char name[MAX_RING_NAME_SZ];
+    PCIDevice *dev;
+    uint32_t max_elems;
+    size_t elem_sz;
+    struct pvrdma_ring *ring_state; /* used only for unmap */
+    int npages;
+    void **pages;
+} PvrdmaRing;
+
+int pvrdma_ring_init(PvrdmaRing *ring, const char *name, PCIDevice *dev,
+                     struct pvrdma_ring *ring_state, uint32_t max_elems,
+                     size_t elem_sz, dma_addr_t *tbl, dma_addr_t npages);
+void *pvrdma_ring_next_elem_read(PvrdmaRing *ring);
+void pvrdma_ring_read_inc(PvrdmaRing *ring);
+void *pvrdma_ring_next_elem_write(PvrdmaRing *ring);
+void pvrdma_ring_write_inc(PvrdmaRing *ring);
+void pvrdma_ring_free(PvrdmaRing *ring);
+
+#endif
diff --git a/hw/rdma/vmw/pvrdma_ib_verbs.h b/hw/rdma/vmw/pvrdma_ib_verbs.h
new file mode 100644
index 0000000000..cf1430024b
--- /dev/null
+++ b/hw/rdma/vmw/pvrdma_ib_verbs.h
@@ -0,0 +1,433 @@
+/*
+ * QEMU VMWARE paravirtual RDMA device definitions
+ *
+ * Copyright (C) 2018 Oracle
+ * Copyright (C) 2018 Red Hat Inc
+ *
+ * Authors:
+ *     Yuval Shaia <yuval.shaia@oracle.com>
+ *     Marcel Apfelbaum <marcel@redhat.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef PVRDMA_IB_VERBS_H
+#define PVRDMA_IB_VERBS_H
+
+/*
+ * VMWARE headers we got from Linux kernel do not fully comply QEMU coding
+ * standards in sense of types and defines used.
+ * Since we didn't want to change VMWARE code, following set of typedefs
+ * and defines needed to compile these headers with QEMU introduced.
+ */
+
+#define u8     uint8_t
+#define u16    unsigned short
+#define u32    uint32_t
+#define u64    uint64_t
+
+/*
+ * Following is an interface definition for PVRDMA device as provided by
+ * VMWARE.
+ * See original copyright from Linux kernel v4.14.5 header file
+ * drivers/infiniband/hw/vmw_pvrdma/pvrdma_verbs.h
+ */
+
+/*
+ * [PLEASE NOTE:  VMWARE, INC. ELECTS TO USE AND DISTRIBUTE THIS COMPONENT
+ * UNDER THE TERMS OF THE OpenIB.org BSD license.  THE ORIGINAL LICENSE TERMS
+ * ARE REPRODUCED BELOW ONLY AS A REFERENCE.]
+ *
+ * Copyright (c) 2004 Mellanox Technologies Ltd.  All rights reserved.
+ * Copyright (c) 2004 Infinicon Corporation.  All rights reserved.
+ * Copyright (c) 2004 Intel Corporation.  All rights reserved.
+ * Copyright (c) 2004 Topspin Corporation.  All rights reserved.
+ * Copyright (c) 2004 Voltaire Corporation.  All rights reserved.
+ * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
+ * Copyright (c) 2005, 2006, 2007 Cisco Systems.  All rights reserved.
+ * Copyright (c) 2015-2016 VMware, Inc.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+
+#include <linux/types.h>
+
+union pvrdma_gid {
+    u8    raw[16];
+    struct {
+        __be64    subnet_prefix;
+        __be64    interface_id;
+    } global;
+};
+
+enum pvrdma_link_layer {
+    PVRDMA_LINK_LAYER_UNSPECIFIED,
+    PVRDMA_LINK_LAYER_INFINIBAND,
+    PVRDMA_LINK_LAYER_ETHERNET,
+};
+
+enum pvrdma_mtu {
+    PVRDMA_MTU_256  = 1,
+    PVRDMA_MTU_512  = 2,
+    PVRDMA_MTU_1024 = 3,
+    PVRDMA_MTU_2048 = 4,
+    PVRDMA_MTU_4096 = 5,
+};
+
+static inline int pvrdma_mtu_enum_to_int(enum pvrdma_mtu mtu)
+{
+    switch (mtu) {
+    case PVRDMA_MTU_256:    return  256;
+    case PVRDMA_MTU_512:    return  512;
+    case PVRDMA_MTU_1024:    return 1024;
+    case PVRDMA_MTU_2048:    return 2048;
+    case PVRDMA_MTU_4096:    return 4096;
+    default:        return   -1;
+    }
+}
+
+static inline enum pvrdma_mtu pvrdma_mtu_int_to_enum(int mtu)
+{
+    switch (mtu) {
+    case 256:    return PVRDMA_MTU_256;
+    case 512:    return PVRDMA_MTU_512;
+    case 1024:    return PVRDMA_MTU_1024;
+    case 2048:    return PVRDMA_MTU_2048;
+    case 4096:
+    default:    return PVRDMA_MTU_4096;
+    }
+}
+
+enum pvrdma_port_state {
+    PVRDMA_PORT_NOP            = 0,
+    PVRDMA_PORT_DOWN        = 1,
+    PVRDMA_PORT_INIT        = 2,
+    PVRDMA_PORT_ARMED        = 3,
+    PVRDMA_PORT_ACTIVE        = 4,
+    PVRDMA_PORT_ACTIVE_DEFER    = 5,
+};
+
+enum pvrdma_port_cap_flags {
+    PVRDMA_PORT_SM                = 1 <<  1,
+    PVRDMA_PORT_NOTICE_SUP            = 1 <<  2,
+    PVRDMA_PORT_TRAP_SUP            = 1 <<  3,
+    PVRDMA_PORT_OPT_IPD_SUP            = 1 <<  4,
+    PVRDMA_PORT_AUTO_MIGR_SUP        = 1 <<  5,
+    PVRDMA_PORT_SL_MAP_SUP            = 1 <<  6,
+    PVRDMA_PORT_MKEY_NVRAM            = 1 <<  7,
+    PVRDMA_PORT_PKEY_NVRAM            = 1 <<  8,
+    PVRDMA_PORT_LED_INFO_SUP        = 1 <<  9,
+    PVRDMA_PORT_SM_DISABLED            = 1 << 10,
+    PVRDMA_PORT_SYS_IMAGE_GUID_SUP        = 1 << 11,
+    PVRDMA_PORT_PKEY_SW_EXT_PORT_TRAP_SUP    = 1 << 12,
+    PVRDMA_PORT_EXTENDED_SPEEDS_SUP        = 1 << 14,
+    PVRDMA_PORT_CM_SUP            = 1 << 16,
+    PVRDMA_PORT_SNMP_TUNNEL_SUP        = 1 << 17,
+    PVRDMA_PORT_REINIT_SUP            = 1 << 18,
+    PVRDMA_PORT_DEVICE_MGMT_SUP        = 1 << 19,
+    PVRDMA_PORT_VENDOR_CLASS_SUP        = 1 << 20,
+    PVRDMA_PORT_DR_NOTICE_SUP        = 1 << 21,
+    PVRDMA_PORT_CAP_MASK_NOTICE_SUP        = 1 << 22,
+    PVRDMA_PORT_BOOT_MGMT_SUP        = 1 << 23,
+    PVRDMA_PORT_LINK_LATENCY_SUP        = 1 << 24,
+    PVRDMA_PORT_CLIENT_REG_SUP        = 1 << 25,
+    PVRDMA_PORT_IP_BASED_GIDS        = 1 << 26,
+    PVRDMA_PORT_CAP_FLAGS_MAX        = PVRDMA_PORT_IP_BASED_GIDS,
+};
+
+enum pvrdma_port_width {
+    PVRDMA_WIDTH_1X        = 1,
+    PVRDMA_WIDTH_4X        = 2,
+    PVRDMA_WIDTH_8X        = 4,
+    PVRDMA_WIDTH_12X    = 8,
+};
+
+static inline int pvrdma_width_enum_to_int(enum pvrdma_port_width width)
+{
+    switch (width) {
+    case PVRDMA_WIDTH_1X:    return  1;
+    case PVRDMA_WIDTH_4X:    return  4;
+    case PVRDMA_WIDTH_8X:    return  8;
+    case PVRDMA_WIDTH_12X:    return 12;
+    default:        return -1;
+    }
+}
+
+enum pvrdma_port_speed {
+    PVRDMA_SPEED_SDR    = 1,
+    PVRDMA_SPEED_DDR    = 2,
+    PVRDMA_SPEED_QDR    = 4,
+    PVRDMA_SPEED_FDR10    = 8,
+    PVRDMA_SPEED_FDR    = 16,
+    PVRDMA_SPEED_EDR    = 32,
+};
+
+struct pvrdma_port_attr {
+    enum pvrdma_port_state    state;
+    enum pvrdma_mtu        max_mtu;
+    enum pvrdma_mtu        active_mtu;
+    u32            gid_tbl_len;
+    u32            port_cap_flags;
+    u32            max_msg_sz;
+    u32            bad_pkey_cntr;
+    u32            qkey_viol_cntr;
+    u16            pkey_tbl_len;
+    u16            lid;
+    u16            sm_lid;
+    u8            lmc;
+    u8            max_vl_num;
+    u8            sm_sl;
+    u8            subnet_timeout;
+    u8            init_type_reply;
+    u8            active_width;
+    u8            active_speed;
+    u8            phys_state;
+    u8            reserved[2];
+};
+
+struct pvrdma_global_route {
+    union pvrdma_gid    dgid;
+    u32            flow_label;
+    u8            sgid_index;
+    u8            hop_limit;
+    u8            traffic_class;
+    u8            reserved;
+};
+
+struct pvrdma_grh {
+    __be32            version_tclass_flow;
+    __be16            paylen;
+    u8            next_hdr;
+    u8            hop_limit;
+    union pvrdma_gid    sgid;
+    union pvrdma_gid    dgid;
+};
+
+enum pvrdma_ah_flags {
+    PVRDMA_AH_GRH = 1,
+};
+
+enum pvrdma_rate {
+    PVRDMA_RATE_PORT_CURRENT    = 0,
+    PVRDMA_RATE_2_5_GBPS        = 2,
+    PVRDMA_RATE_5_GBPS        = 5,
+    PVRDMA_RATE_10_GBPS        = 3,
+    PVRDMA_RATE_20_GBPS        = 6,
+    PVRDMA_RATE_30_GBPS        = 4,
+    PVRDMA_RATE_40_GBPS        = 7,
+    PVRDMA_RATE_60_GBPS        = 8,
+    PVRDMA_RATE_80_GBPS        = 9,
+    PVRDMA_RATE_120_GBPS        = 10,
+    PVRDMA_RATE_14_GBPS        = 11,
+    PVRDMA_RATE_56_GBPS        = 12,
+    PVRDMA_RATE_112_GBPS        = 13,
+    PVRDMA_RATE_168_GBPS        = 14,
+    PVRDMA_RATE_25_GBPS        = 15,
+    PVRDMA_RATE_100_GBPS        = 16,
+    PVRDMA_RATE_200_GBPS        = 17,
+    PVRDMA_RATE_300_GBPS        = 18,
+};
+
+struct pvrdma_ah_attr {
+    struct pvrdma_global_route    grh;
+    u16                dlid;
+    u16                vlan_id;
+    u8                sl;
+    u8                src_path_bits;
+    u8                static_rate;
+    u8                ah_flags;
+    u8                port_num;
+    u8                dmac[6];
+    u8                reserved;
+};
+
+enum pvrdma_cq_notify_flags {
+    PVRDMA_CQ_SOLICITED        = 1 << 0,
+    PVRDMA_CQ_NEXT_COMP        = 1 << 1,
+    PVRDMA_CQ_SOLICITED_MASK    = PVRDMA_CQ_SOLICITED |
+                      PVRDMA_CQ_NEXT_COMP,
+    PVRDMA_CQ_REPORT_MISSED_EVENTS    = 1 << 2,
+};
+
+struct pvrdma_qp_cap {
+    u32    max_send_wr;
+    u32    max_recv_wr;
+    u32    max_send_sge;
+    u32    max_recv_sge;
+    u32    max_inline_data;
+    u32    reserved;
+};
+
+enum pvrdma_sig_type {
+    PVRDMA_SIGNAL_ALL_WR,
+    PVRDMA_SIGNAL_REQ_WR,
+};
+
+enum pvrdma_qp_type {
+    PVRDMA_QPT_SMI,
+    PVRDMA_QPT_GSI,
+    PVRDMA_QPT_RC,
+    PVRDMA_QPT_UC,
+    PVRDMA_QPT_UD,
+    PVRDMA_QPT_RAW_IPV6,
+    PVRDMA_QPT_RAW_ETHERTYPE,
+    PVRDMA_QPT_RAW_PACKET = 8,
+    PVRDMA_QPT_XRC_INI = 9,
+    PVRDMA_QPT_XRC_TGT,
+    PVRDMA_QPT_MAX,
+};
+
+enum pvrdma_qp_create_flags {
+    PVRDMA_QP_CREATE_IPOPVRDMA_UD_LSO        = 1 << 0,
+    PVRDMA_QP_CREATE_BLOCK_MULTICAST_LOOPBACK    = 1 << 1,
+};
+
+enum pvrdma_qp_attr_mask {
+    PVRDMA_QP_STATE            = 1 << 0,
+    PVRDMA_QP_CUR_STATE        = 1 << 1,
+    PVRDMA_QP_EN_SQD_ASYNC_NOTIFY    = 1 << 2,
+    PVRDMA_QP_ACCESS_FLAGS        = 1 << 3,
+    PVRDMA_QP_PKEY_INDEX        = 1 << 4,
+    PVRDMA_QP_PORT            = 1 << 5,
+    PVRDMA_QP_QKEY            = 1 << 6,
+    PVRDMA_QP_AV            = 1 << 7,
+    PVRDMA_QP_PATH_MTU        = 1 << 8,
+    PVRDMA_QP_TIMEOUT        = 1 << 9,
+    PVRDMA_QP_RETRY_CNT        = 1 << 10,
+    PVRDMA_QP_RNR_RETRY        = 1 << 11,
+    PVRDMA_QP_RQ_PSN        = 1 << 12,
+    PVRDMA_QP_MAX_QP_RD_ATOMIC    = 1 << 13,
+    PVRDMA_QP_ALT_PATH        = 1 << 14,
+    PVRDMA_QP_MIN_RNR_TIMER        = 1 << 15,
+    PVRDMA_QP_SQ_PSN        = 1 << 16,
+    PVRDMA_QP_MAX_DEST_RD_ATOMIC    = 1 << 17,
+    PVRDMA_QP_PATH_MIG_STATE    = 1 << 18,
+    PVRDMA_QP_CAP            = 1 << 19,
+    PVRDMA_QP_DEST_QPN        = 1 << 20,
+    PVRDMA_QP_ATTR_MASK_MAX        = PVRDMA_QP_DEST_QPN,
+};
+
+enum pvrdma_qp_state {
+    PVRDMA_QPS_RESET,
+    PVRDMA_QPS_INIT,
+    PVRDMA_QPS_RTR,
+    PVRDMA_QPS_RTS,
+    PVRDMA_QPS_SQD,
+    PVRDMA_QPS_SQE,
+    PVRDMA_QPS_ERR,
+};
+
+enum pvrdma_mig_state {
+    PVRDMA_MIG_MIGRATED,
+    PVRDMA_MIG_REARM,
+    PVRDMA_MIG_ARMED,
+};
+
+enum pvrdma_mw_type {
+    PVRDMA_MW_TYPE_1 = 1,
+    PVRDMA_MW_TYPE_2 = 2,
+};
+
+struct pvrdma_qp_attr {
+    enum pvrdma_qp_state    qp_state;
+    enum pvrdma_qp_state    cur_qp_state;
+    enum pvrdma_mtu        path_mtu;
+    enum pvrdma_mig_state    path_mig_state;
+    u32            qkey;
+    u32            rq_psn;
+    u32            sq_psn;
+    u32            dest_qp_num;
+    u32            qp_access_flags;
+    u16            pkey_index;
+    u16            alt_pkey_index;
+    u8            en_sqd_async_notify;
+    u8            sq_draining;
+    u8            max_rd_atomic;
+    u8            max_dest_rd_atomic;
+    u8            min_rnr_timer;
+    u8            port_num;
+    u8            timeout;
+    u8            retry_cnt;
+    u8            rnr_retry;
+    u8            alt_port_num;
+    u8            alt_timeout;
+    u8            reserved[5];
+    struct pvrdma_qp_cap    cap;
+    struct pvrdma_ah_attr    ah_attr;
+    struct pvrdma_ah_attr    alt_ah_attr;
+};
+
+enum pvrdma_send_flags {
+    PVRDMA_SEND_FENCE    = 1 << 0,
+    PVRDMA_SEND_SIGNALED    = 1 << 1,
+    PVRDMA_SEND_SOLICITED    = 1 << 2,
+    PVRDMA_SEND_INLINE    = 1 << 3,
+    PVRDMA_SEND_IP_CSUM    = 1 << 4,
+    PVRDMA_SEND_FLAGS_MAX    = PVRDMA_SEND_IP_CSUM,
+};
+
+enum pvrdma_access_flags {
+    PVRDMA_ACCESS_LOCAL_WRITE    = 1 << 0,
+    PVRDMA_ACCESS_REMOTE_WRITE    = 1 << 1,
+    PVRDMA_ACCESS_REMOTE_READ    = 1 << 2,
+    PVRDMA_ACCESS_REMOTE_ATOMIC    = 1 << 3,
+    PVRDMA_ACCESS_MW_BIND        = 1 << 4,
+    PVRDMA_ZERO_BASED        = 1 << 5,
+    PVRDMA_ACCESS_ON_DEMAND        = 1 << 6,
+    PVRDMA_ACCESS_FLAGS_MAX        = PVRDMA_ACCESS_ON_DEMAND,
+};
+
+enum ib_wc_status {
+    IB_WC_SUCCESS,
+    IB_WC_LOC_LEN_ERR,
+    IB_WC_LOC_QP_OP_ERR,
+    IB_WC_LOC_EEC_OP_ERR,
+    IB_WC_LOC_PROT_ERR,
+    IB_WC_WR_FLUSH_ERR,
+    IB_WC_MW_BIND_ERR,
+    IB_WC_BAD_RESP_ERR,
+    IB_WC_LOC_ACCESS_ERR,
+    IB_WC_REM_INV_REQ_ERR,
+    IB_WC_REM_ACCESS_ERR,
+    IB_WC_REM_OP_ERR,
+    IB_WC_RETRY_EXC_ERR,
+    IB_WC_RNR_RETRY_EXC_ERR,
+    IB_WC_LOC_RDD_VIOL_ERR,
+    IB_WC_REM_INV_RD_REQ_ERR,
+    IB_WC_REM_ABORT_ERR,
+    IB_WC_INV_EECN_ERR,
+    IB_WC_INV_EEC_STATE_ERR,
+    IB_WC_FATAL_ERR,
+    IB_WC_RESP_TIMEOUT_ERR,
+    IB_WC_GENERAL_ERR
+};
+
+#endif /* PVRDMA_IB_VERBS_H */
diff --git a/hw/rdma/vmw/pvrdma_main.c b/hw/rdma/vmw/pvrdma_main.c
new file mode 100644
index 0000000000..d667131d3a
--- /dev/null
+++ b/hw/rdma/vmw/pvrdma_main.c
@@ -0,0 +1,671 @@
+#include <qemu/osdep.h>
+#include <qapi/error.h>
+#include <hw/hw.h>
+#include <hw/pci/pci.h>
+#include <hw/pci/pci_ids.h>
+#include <hw/pci/msi.h>
+#include <hw/pci/msix.h>
+#include <hw/qdev-core.h>
+#include <hw/qdev-properties.h>
+#include <cpu.h>
+#include "trace.h"
+
+#include "../rdma_rm.h"
+#include "../rdma_backend.h"
+#include "../rdma_utils.h"
+
+#include "pvrdma.h"
+#include "vmw_pvrdma-abi.h"
+#include "pvrdma_dev_api.h"
+#include "pvrdma_qp_ops.h"
+
+static Property pvrdma_dev_properties[] = {
+    DEFINE_PROP_STRING("backend-dev", PVRDMADev, backend_device_name),
+    DEFINE_PROP_UINT8("backend-port", PVRDMADev, backend_port_num, 1),
+    DEFINE_PROP_UINT8("backend-gid-idx", PVRDMADev, backend_gid_idx, 0),
+    DEFINE_PROP_UINT64("dev-caps-max-mr-size", PVRDMADev, dev_attr.max_mr_size,
+                       MAX_MR_SIZE),
+    DEFINE_PROP_INT32("dev-caps-max-qp", PVRDMADev, dev_attr.max_qp, MAX_QP),
+    DEFINE_PROP_INT32("dev-caps-max-sge", PVRDMADev, dev_attr.max_sge, MAX_SGE),
+    DEFINE_PROP_INT32("dev-caps-max-cq", PVRDMADev, dev_attr.max_cq, MAX_CQ),
+    DEFINE_PROP_INT32("dev-caps-max-mr", PVRDMADev, dev_attr.max_mr, MAX_MR),
+    DEFINE_PROP_INT32("dev-caps-max-pd", PVRDMADev, dev_attr.max_pd, MAX_PD),
+    DEFINE_PROP_INT32("dev-caps-qp-rd-atom", PVRDMADev, dev_attr.max_qp_rd_atom,
+                      MAX_QP_RD_ATOM),
+    DEFINE_PROP_INT32("dev-caps-max-qp-init-rd-atom", PVRDMADev,
+                      dev_attr.max_qp_init_rd_atom, MAX_QP_INIT_RD_ATOM),
+    DEFINE_PROP_INT32("dev-caps-max-ah", PVRDMADev, dev_attr.max_ah, MAX_AH),
+    DEFINE_PROP_END_OF_LIST(),
+};
+
+static void free_dev_ring(PCIDevice *pci_dev, PvrdmaRing *ring,
+                          void *ring_state)
+{
+    pvrdma_ring_free(ring);
+    rdma_pci_dma_unmap(pci_dev, ring_state, TARGET_PAGE_SIZE);
+}
+
+static int init_dev_ring(PvrdmaRing *ring, struct pvrdma_ring **ring_state,
+                         const char *name, PCIDevice *pci_dev,
+                         dma_addr_t dir_addr, u32 num_pages)
+{
+    __u64 *dir, *tbl;
+    int rc = 0;
+
+    pr_dbg("Initializing device ring %s\n", name);
+    pr_dbg("pdir_dma=0x%llx\n", (long long unsigned int)dir_addr);
+    pr_dbg("num_pages=%d\n", num_pages);
+    dir = rdma_pci_dma_map(pci_dev, dir_addr, TARGET_PAGE_SIZE);
+    if (!dir) {
+        pr_err("Failed to map to page directory\n");
+        rc = -ENOMEM;
+        goto out;
+    }
+    tbl = rdma_pci_dma_map(pci_dev, dir[0], TARGET_PAGE_SIZE);
+    if (!tbl) {
+        pr_err("Failed to map to page table\n");
+        rc = -ENOMEM;
+        goto out_free_dir;
+    }
+
+    *ring_state = rdma_pci_dma_map(pci_dev, tbl[0], TARGET_PAGE_SIZE);
+    if (!*ring_state) {
+        pr_err("Failed to map to ring state\n");
+        rc = -ENOMEM;
+        goto out_free_tbl;
+    }
+    /* RX ring is the second */
+    (struct pvrdma_ring *)(*ring_state)++;
+    rc = pvrdma_ring_init(ring, name, pci_dev,
+                          (struct pvrdma_ring *)*ring_state,
+                          (num_pages - 1) * TARGET_PAGE_SIZE /
+                          sizeof(struct pvrdma_cqne),
+                          sizeof(struct pvrdma_cqne),
+                          (dma_addr_t *)&tbl[1], (dma_addr_t)num_pages - 1);
+    if (rc) {
+        pr_err("Failed to initialize ring\n");
+        rc = -ENOMEM;
+        goto out_free_ring_state;
+    }
+
+    goto out_free_tbl;
+
+out_free_ring_state:
+    rdma_pci_dma_unmap(pci_dev, *ring_state, TARGET_PAGE_SIZE);
+
+out_free_tbl:
+    rdma_pci_dma_unmap(pci_dev, tbl, TARGET_PAGE_SIZE);
+
+out_free_dir:
+    rdma_pci_dma_unmap(pci_dev, dir, TARGET_PAGE_SIZE);
+
+out:
+    return rc;
+}
+
+static void free_dsr(PVRDMADev *dev)
+{
+    PCIDevice *pci_dev = PCI_DEVICE(dev);
+
+    if (!dev->dsr_info.dsr) {
+        return;
+    }
+
+    free_dev_ring(pci_dev, &dev->dsr_info.async,
+                  dev->dsr_info.async_ring_state);
+
+    free_dev_ring(pci_dev, &dev->dsr_info.cq, dev->dsr_info.cq_ring_state);
+
+    rdma_pci_dma_unmap(pci_dev, dev->dsr_info.req,
+                         sizeof(union pvrdma_cmd_req));
+
+    rdma_pci_dma_unmap(pci_dev, dev->dsr_info.rsp,
+                         sizeof(union pvrdma_cmd_resp));
+
+    rdma_pci_dma_unmap(pci_dev, dev->dsr_info.dsr,
+                         sizeof(struct pvrdma_device_shared_region));
+
+    dev->dsr_info.dsr = NULL;
+}
+
+static int load_dsr(PVRDMADev *dev)
+{
+    int rc = 0;
+    PCIDevice *pci_dev = PCI_DEVICE(dev);
+    DSRInfo *dsr_info;
+    struct pvrdma_device_shared_region *dsr;
+
+    free_dsr(dev);
+
+    /* Map to DSR */
+    pr_dbg("dsr_dma=0x%llx\n", (long long unsigned int)dev->dsr_info.dma);
+    dev->dsr_info.dsr = rdma_pci_dma_map(pci_dev, dev->dsr_info.dma,
+                              sizeof(struct pvrdma_device_shared_region));
+    if (!dev->dsr_info.dsr) {
+        pr_err("Failed to map to DSR\n");
+        rc = -ENOMEM;
+        goto out;
+    }
+
+    /* Shortcuts */
+    dsr_info = &dev->dsr_info;
+    dsr = dsr_info->dsr;
+
+    /* Map to command slot */
+    pr_dbg("cmd_dma=0x%llx\n", (long long unsigned int)dsr->cmd_slot_dma);
+    dsr_info->req = rdma_pci_dma_map(pci_dev, dsr->cmd_slot_dma,
+                                     sizeof(union pvrdma_cmd_req));
+    if (!dsr_info->req) {
+        pr_err("Failed to map to command slot address\n");
+        rc = -ENOMEM;
+        goto out_free_dsr;
+    }
+
+    /* Map to response slot */
+    pr_dbg("rsp_dma=0x%llx\n", (long long unsigned int)dsr->resp_slot_dma);
+    dsr_info->rsp = rdma_pci_dma_map(pci_dev, dsr->resp_slot_dma,
+                                     sizeof(union pvrdma_cmd_resp));
+    if (!dsr_info->rsp) {
+        pr_err("Failed to map to response slot address\n");
+        rc = -ENOMEM;
+        goto out_free_req;
+    }
+
+    /* Map to CQ notification ring */
+    rc = init_dev_ring(&dsr_info->cq, &dsr_info->cq_ring_state, "dev_cq",
+                       pci_dev, dsr->cq_ring_pages.pdir_dma,
+                       dsr->cq_ring_pages.num_pages);
+    if (rc) {
+        pr_err("Failed to map to initialize CQ ring\n");
+        rc = -ENOMEM;
+        goto out_free_rsp;
+    }
+
+    /* Map to event notification ring */
+    rc = init_dev_ring(&dsr_info->async, &dsr_info->async_ring_state,
+                       "dev_async", pci_dev, dsr->async_ring_pages.pdir_dma,
+                       dsr->async_ring_pages.num_pages);
+    if (rc) {
+        pr_err("Failed to map to initialize event ring\n");
+        rc = -ENOMEM;
+        goto out_free_rsp;
+    }
+
+    goto out;
+
+out_free_rsp:
+    rdma_pci_dma_unmap(pci_dev, dsr_info->rsp, sizeof(union pvrdma_cmd_resp));
+
+out_free_req:
+    rdma_pci_dma_unmap(pci_dev, dsr_info->req, sizeof(union pvrdma_cmd_req));
+
+out_free_dsr:
+    rdma_pci_dma_unmap(pci_dev, dsr_info->dsr,
+                       sizeof(struct pvrdma_device_shared_region));
+    dsr_info->dsr = NULL;
+
+out:
+    return rc;
+}
+
+static void init_dsr_dev_caps(PVRDMADev *dev)
+{
+    struct pvrdma_device_shared_region *dsr;
+
+    if (dev->dsr_info.dsr == NULL) {
+        pr_err("Can't initialized DSR\n");
+        return;
+    }
+
+    dsr = dev->dsr_info.dsr;
+
+    dsr->caps.fw_ver = PVRDMA_FW_VERSION;
+    pr_dbg("fw_ver=0x%lx\n", dsr->caps.fw_ver);
+
+    dsr->caps.mode = PVRDMA_DEVICE_MODE_ROCE;
+    pr_dbg("mode=%d\n", dsr->caps.mode);
+
+    dsr->caps.gid_types |= PVRDMA_GID_TYPE_FLAG_ROCE_V1;
+    pr_dbg("gid_types=0x%x\n", dsr->caps.gid_types);
+
+    dsr->caps.max_uar = RDMA_BAR2_UAR_SIZE;
+    pr_dbg("max_uar=%d\n", dsr->caps.max_uar);
+
+    dsr->caps.max_mr_size = dev->dev_attr.max_mr_size;
+    dsr->caps.max_qp = dev->dev_attr.max_qp;
+    dsr->caps.max_qp_wr = dev->dev_attr.max_qp_wr;
+    dsr->caps.max_sge = dev->dev_attr.max_sge;
+    dsr->caps.max_cq = dev->dev_attr.max_cq;
+    dsr->caps.max_cqe = dev->dev_attr.max_cqe;
+    dsr->caps.max_mr = dev->dev_attr.max_mr;
+    dsr->caps.max_pd = dev->dev_attr.max_pd;
+    dsr->caps.max_ah = dev->dev_attr.max_ah;
+
+    dsr->caps.gid_tbl_len = MAX_GIDS;
+    pr_dbg("gid_tbl_len=%d\n", dsr->caps.gid_tbl_len);
+
+    dsr->caps.sys_image_guid = 0;
+    pr_dbg("sys_image_guid=%llx\n", dsr->caps.sys_image_guid);
+
+    dsr->caps.node_guid = cpu_to_be64(dev->node_guid);
+    pr_dbg("node_guid=%llx\n",
+           (long long unsigned int)be64_to_cpu(dsr->caps.node_guid));
+
+    dsr->caps.phys_port_cnt = MAX_PORTS;
+    pr_dbg("phys_port_cnt=%d\n", dsr->caps.phys_port_cnt);
+
+    dsr->caps.max_pkeys = MAX_PKEYS;
+    pr_dbg("max_pkeys=%d\n", dsr->caps.max_pkeys);
+
+    pr_dbg("Initialized\n");
+}
+
+static void free_ports(PVRDMADev *dev)
+{
+    int i;
+
+    for (i = 0; i < MAX_PORTS; i++) {
+        free(dev->rdma_dev_res.ports[i].gid_tbl);
+    }
+}
+
+static int init_ports(PVRDMADev *dev, Error **errp)
+{
+    int i, ret = 0;
+
+    memset(dev->rdma_dev_res.ports, 0, sizeof(dev->rdma_dev_res.ports));
+
+    for (i = 0; i < MAX_PORTS; i++) {
+        dev->rdma_dev_res.ports[i].state = PVRDMA_PORT_DOWN;
+
+        dev->rdma_dev_res.ports[i].pkey_tbl =
+            malloc(sizeof(*dev->rdma_dev_res.ports[i].pkey_tbl) *
+                   MAX_PORT_PKEYS);
+        if (dev->rdma_dev_res.ports[i].gid_tbl == NULL) {
+            goto err_free_ports;
+        }
+
+        memset(dev->rdma_dev_res.ports[i].gid_tbl, 0,
+               sizeof(dev->rdma_dev_res.ports[i].gid_tbl));
+    }
+
+    return 0;
+
+err_free_ports:
+    free_ports(dev);
+
+    error_setg(errp, "Failed to initialize device's ports");
+
+    return ret;
+}
+
+static void activate_device(PVRDMADev *dev)
+{
+    set_reg_val(dev, PVRDMA_REG_ERR, 0);
+    pr_dbg("Device activated\n");
+}
+
+static int unquiesce_device(PVRDMADev *dev)
+{
+    pr_dbg("Device unquiesced\n");
+    return 0;
+}
+
+static int reset_device(PVRDMADev *dev)
+{
+    pr_dbg("Device reset complete\n");
+    return 0;
+}
+
+static uint64_t regs_read(void *opaque, hwaddr addr, unsigned size)
+{
+    PVRDMADev *dev = opaque;
+    __u32 val;
+
+    /* pr_dbg("addr=0x%lx, size=%d\n", addr, size); */
+
+    if (get_reg_val(dev, addr, &val)) {
+        pr_dbg("Error trying to read REG value from address 0x%x\n",
+               (__u32)addr);
+        return -EINVAL;
+    }
+
+    trace_pvrdma_regs_read(addr, val);
+
+    return val;
+}
+
+static void regs_write(void *opaque, hwaddr addr, uint64_t val, unsigned size)
+{
+    PVRDMADev *dev = opaque;
+
+    /* pr_dbg("addr=0x%lx, val=0x%x, size=%d\n", addr, (uint32_t)val, size); */
+
+    if (set_reg_val(dev, addr, val)) {
+        pr_err("Error trying to set REG value, addr=0x%lx, val=0x%lx\n",
+               (uint64_t)addr, val);
+        return;
+    }
+
+    trace_pvrdma_regs_write(addr, val);
+
+    switch (addr) {
+    case PVRDMA_REG_DSRLOW:
+        dev->dsr_info.dma = val;
+        break;
+    case PVRDMA_REG_DSRHIGH:
+        dev->dsr_info.dma |= val << 32;
+        load_dsr(dev);
+        init_dsr_dev_caps(dev);
+        break;
+    case PVRDMA_REG_CTL:
+        switch (val) {
+        case PVRDMA_DEVICE_CTL_ACTIVATE:
+            activate_device(dev);
+            break;
+        case PVRDMA_DEVICE_CTL_UNQUIESCE:
+            unquiesce_device(dev);
+            break;
+        case PVRDMA_DEVICE_CTL_RESET:
+            reset_device(dev);
+            break;
+        }
+    break;
+    case PVRDMA_REG_IMR:
+        pr_dbg("Interrupt mask=0x%lx\n", val);
+        dev->interrupt_mask = val;
+        break;
+    case PVRDMA_REG_REQUEST:
+        if (val == 0) {
+            execute_command(dev);
+        }
+    break;
+    default:
+        break;
+    }
+}
+
+static const MemoryRegionOps regs_ops = {
+    .read = regs_read,
+    .write = regs_write,
+    .endianness = DEVICE_LITTLE_ENDIAN,
+    .impl = {
+        .min_access_size = sizeof(uint32_t),
+        .max_access_size = sizeof(uint32_t),
+    },
+};
+
+static void uar_write(void *opaque, hwaddr addr, uint64_t val, unsigned size)
+{
+    PVRDMADev *dev = opaque;
+
+    /* pr_dbg("addr=0x%lx, val=0x%x, size=%d\n", addr, (uint32_t)val, size); */
+
+    switch (addr & 0xFFF) { /* Mask with 0xFFF as each UC gets page */
+    case PVRDMA_UAR_QP_OFFSET:
+        pr_dbg("UAR QP command, addr=0x%x, val=0x%lx\n", (__u32)addr, val);
+        if (val & PVRDMA_UAR_QP_SEND) {
+            pvrdma_qp_send(dev, val & PVRDMA_UAR_HANDLE_MASK);
+        }
+        if (val & PVRDMA_UAR_QP_RECV) {
+            pvrdma_qp_recv(dev, val & PVRDMA_UAR_HANDLE_MASK);
+        }
+        break;
+    case PVRDMA_UAR_CQ_OFFSET:
+        /* pr_dbg("UAR CQ cmd, addr=0x%x, val=0x%lx\n", (__u32)addr, val); */
+        if (val & PVRDMA_UAR_CQ_ARM) {
+            rdma_rm_req_notify_cq(&dev->rdma_dev_res,
+                                  val & PVRDMA_UAR_HANDLE_MASK,
+                                  !!(val & PVRDMA_UAR_CQ_ARM_SOL));
+        }
+        if (val & PVRDMA_UAR_CQ_ARM_SOL) {
+            pr_dbg("UAR_CQ_ARM_SOL (%ld)\n", val & PVRDMA_UAR_HANDLE_MASK);
+        }
+        if (val & PVRDMA_UAR_CQ_POLL) {
+            pr_dbg("UAR_CQ_POLL (%ld)\n", val & PVRDMA_UAR_HANDLE_MASK);
+            pvrdma_cq_poll(&dev->rdma_dev_res, val & PVRDMA_UAR_HANDLE_MASK);
+        }
+        break;
+    default:
+        pr_err("Unsupported command, addr=0x%lx, val=0x%lx\n",
+               (uint64_t)addr, val);
+        break;
+    }
+}
+
+static const MemoryRegionOps uar_ops = {
+    .write = uar_write,
+    .endianness = DEVICE_LITTLE_ENDIAN,
+    .impl = {
+        .min_access_size = sizeof(uint32_t),
+        .max_access_size = sizeof(uint32_t),
+    },
+};
+
+static void init_pci_config(PCIDevice *pdev)
+{
+    pdev->config[PCI_INTERRUPT_PIN] = 1;
+}
+
+static void init_bars(PCIDevice *pdev)
+{
+    PVRDMADev *dev = PVRDMA_DEV(pdev);
+
+    /* BAR 0 - MSI-X */
+    memory_region_init(&dev->msix, OBJECT(dev), "pvrdma-msix",
+                       RDMA_BAR0_MSIX_SIZE);
+    pci_register_bar(pdev, RDMA_MSIX_BAR_IDX, PCI_BASE_ADDRESS_SPACE_MEMORY,
+                     &dev->msix);
+
+    /* BAR 1 - Registers */
+    memset(&dev->regs_data, 0, sizeof(dev->regs_data));
+    memory_region_init_io(&dev->regs, OBJECT(dev), &regs_ops, dev,
+                          "pvrdma-regs", RDMA_BAR1_REGS_SIZE);
+    pci_register_bar(pdev, RDMA_REG_BAR_IDX, PCI_BASE_ADDRESS_SPACE_MEMORY,
+                     &dev->regs);
+
+    /* BAR 2 - UAR */
+    memset(&dev->uar_data, 0, sizeof(dev->uar_data));
+    memory_region_init_io(&dev->uar, OBJECT(dev), &uar_ops, dev, "rdma-uar",
+                          RDMA_BAR2_UAR_SIZE);
+    pci_register_bar(pdev, RDMA_UAR_BAR_IDX, PCI_BASE_ADDRESS_SPACE_MEMORY,
+                     &dev->uar);
+}
+
+static void init_regs(PCIDevice *pdev)
+{
+    PVRDMADev *dev = PVRDMA_DEV(pdev);
+
+    set_reg_val(dev, PVRDMA_REG_VERSION, PVRDMA_HW_VERSION);
+    set_reg_val(dev, PVRDMA_REG_ERR, 0xFFFF);
+}
+
+static void uninit_msix(PCIDevice *pdev, int used_vectors)
+{
+    PVRDMADev *dev = PVRDMA_DEV(pdev);
+    int i;
+
+    for (i = 0; i < used_vectors; i++) {
+        msix_vector_unuse(pdev, i);
+    }
+
+    msix_uninit(pdev, &dev->msix, &dev->msix);
+}
+
+static int init_msix(PCIDevice *pdev, Error **errp)
+{
+    PVRDMADev *dev = PVRDMA_DEV(pdev);
+    int i;
+    int rc;
+
+    rc = msix_init(pdev, RDMA_MAX_INTRS, &dev->msix, RDMA_MSIX_BAR_IDX,
+                   RDMA_MSIX_TABLE, &dev->msix, RDMA_MSIX_BAR_IDX,
+                   RDMA_MSIX_PBA, 0, NULL);
+
+    if (rc < 0) {
+        error_setg(errp, "Failed to initialize MSI-X");
+        return rc;
+    }
+
+    for (i = 0; i < RDMA_MAX_INTRS; i++) {
+        rc = msix_vector_use(PCI_DEVICE(dev), i);
+        if (rc < 0) {
+            error_setg(errp, "Fail mark MSI-X vercor %d", i);
+            uninit_msix(pdev, i);
+            return rc;
+        }
+    }
+
+    return 0;
+}
+
+static void init_dev_caps(PVRDMADev *dev)
+{
+    size_t pg_tbl_bytes = TARGET_PAGE_SIZE * (TARGET_PAGE_SIZE / sizeof(u64));
+    size_t wr_sz = MAX(sizeof(struct pvrdma_sq_wqe_hdr),
+                       sizeof(struct pvrdma_rq_wqe_hdr));
+
+    dev->dev_attr.max_qp_wr = pg_tbl_bytes /
+                              (wr_sz + sizeof(struct pvrdma_sge) * MAX_SGE) -
+                              TARGET_PAGE_SIZE; /* First page is ring state */
+    pr_dbg("max_qp_wr=%d\n", dev->dev_attr.max_qp_wr);
+
+    dev->dev_attr.max_cqe = pg_tbl_bytes / sizeof(struct pvrdma_cqe) -
+                            TARGET_PAGE_SIZE; /* First page is ring state */
+    pr_dbg("max_cqe=%d\n", dev->dev_attr.max_cqe);
+}
+
+static int pvrdma_check_ram_shared(Object *obj, void *opaque)
+{
+    bool *shared = opaque;
+
+    if (object_dynamic_cast(obj, "memory-backend-ram")) {
+        *shared = object_property_get_bool(obj, "share", NULL);
+    }
+
+    return 0;
+}
+
+static void pvrdma_realize(PCIDevice *pdev, Error **errp)
+{
+    int rc;
+    PVRDMADev *dev = PVRDMA_DEV(pdev);
+    Object *memdev_root;
+    bool ram_shared = false;
+
+    pr_dbg("Initializing device %s %x.%x\n", pdev->name,
+           PCI_SLOT(pdev->devfn), PCI_FUNC(pdev->devfn));
+
+    if (TARGET_PAGE_SIZE != getpagesize()) {
+        error_setg(errp, "Target page size must be the same as host page size");
+        return;
+    }
+
+    memdev_root = object_resolve_path("/objects", NULL);
+    if (memdev_root) {
+        object_child_foreach(memdev_root, pvrdma_check_ram_shared, &ram_shared);
+    }
+    if (!ram_shared) {
+        error_setg(errp, "Only shared memory backed ram is supported");
+        return;
+    }
+
+    dev->dsr_info.dsr = NULL;
+
+    init_pci_config(pdev);
+
+    init_bars(pdev);
+
+    init_regs(pdev);
+
+    init_dev_caps(dev);
+
+    rc = init_msix(pdev, errp);
+    if (rc) {
+        goto out;
+    }
+
+    rc = rdma_backend_init(&dev->backend_dev, &dev->rdma_dev_res,
+                           dev->backend_device_name, dev->backend_port_num,
+                           dev->backend_gid_idx, &dev->dev_attr, errp);
+    if (rc) {
+        goto out;
+    }
+
+    rc = rdma_rm_init(&dev->rdma_dev_res, &dev->dev_attr, errp);
+    if (rc) {
+        goto out;
+    }
+
+    rc = init_ports(dev, errp);
+    if (rc) {
+        goto out;
+    }
+
+    rc = pvrdma_qp_ops_init();
+    if (rc) {
+        goto out;
+    }
+
+out:
+    if (rc) {
+        error_append_hint(errp, "Device fail to load\n");
+    }
+}
+
+static void pvrdma_exit(PCIDevice *pdev)
+{
+    PVRDMADev *dev = PVRDMA_DEV(pdev);
+
+    pr_dbg("Closing device %s %x.%x\n", pdev->name, PCI_SLOT(pdev->devfn),
+           PCI_FUNC(pdev->devfn));
+
+    pvrdma_qp_ops_fini();
+
+    free_ports(dev);
+
+    rdma_rm_fini(&dev->rdma_dev_res);
+
+    rdma_backend_fini(&dev->backend_dev);
+
+    free_dsr(dev);
+
+    if (msix_enabled(pdev)) {
+        uninit_msix(pdev, RDMA_MAX_INTRS);
+    }
+}
+
+static void pvrdma_class_init(ObjectClass *klass, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(klass);
+    PCIDeviceClass *k = PCI_DEVICE_CLASS(klass);
+
+    k->realize = pvrdma_realize;
+    k->exit = pvrdma_exit;
+    k->vendor_id = PCI_VENDOR_ID_VMWARE;
+    k->device_id = PCI_DEVICE_ID_VMWARE_PVRDMA;
+    k->revision = 0x00;
+    k->class_id = PCI_CLASS_NETWORK_OTHER;
+
+    dc->desc = "RDMA Device";
+    dc->props = pvrdma_dev_properties;
+    set_bit(DEVICE_CATEGORY_NETWORK, dc->categories);
+}
+
+static const TypeInfo pvrdma_info = {
+    .name = PVRDMA_HW_NAME,
+    .parent = TYPE_PCI_DEVICE,
+    .instance_size = sizeof(PVRDMADev),
+    .class_init = pvrdma_class_init,
+    .interfaces = (InterfaceInfo[]) {
+        { INTERFACE_CONVENTIONAL_PCI_DEVICE },
+        { }
+    }
+};
+
+static void register_types(void)
+{
+    type_register_static(&pvrdma_info);
+}
+
+type_init(register_types)
diff --git a/hw/rdma/vmw/pvrdma_qp_ops.c b/hw/rdma/vmw/pvrdma_qp_ops.c
new file mode 100644
index 0000000000..5b95833569
--- /dev/null
+++ b/hw/rdma/vmw/pvrdma_qp_ops.c
@@ -0,0 +1,212 @@
+#include <qemu/osdep.h>
+
+#include "../rdma_utils.h"
+#include "../rdma_rm.h"
+#include "../rdma_backend.h"
+
+#include "pvrdma.h"
+#include "vmw_pvrdma-abi.h"
+#include "pvrdma_qp_ops.h"
+
+typedef struct CompHandlerCtx {
+    PVRDMADev *dev;
+    u32 cq_handle;
+    struct pvrdma_cqe cqe;
+} CompHandlerCtx;
+
+/* Send Queue WQE */
+typedef struct PvrdmaSqWqe {
+    struct pvrdma_sq_wqe_hdr hdr;
+    struct pvrdma_sge sge[0];
+} PvrdmaSqWqe;
+
+/* Recv Queue WQE */
+typedef struct PvrdmaRqWqe {
+    struct pvrdma_rq_wqe_hdr hdr;
+    struct pvrdma_sge sge[0];
+} PvrdmaRqWqe;
+
+/*
+ * 1. Put CQE on send CQ ring
+ * 2. Put CQ number on dsr completion ring
+ * 3. Interrupt host
+ */
+static int pvrdma_post_cqe(PVRDMADev *dev, u32 cq_handle,
+                           struct pvrdma_cqe *cqe)
+{
+    struct pvrdma_cqe *cqe1;
+    struct pvrdma_cqne *cqne;
+    PvrdmaRing *ring;
+    RdmaRmCQ *cq = rdma_rm_get_cq(&dev->rdma_dev_res, cq_handle);
+
+    if (unlikely(!cq)) {
+        pr_dbg("Invalid cqn %d\n", cq_handle);
+        return -EINVAL;
+    }
+
+    ring = (PvrdmaRing *)cq->opaque;
+    pr_dbg("ring=%p\n", ring);
+
+    /* Step #1: Put CQE on CQ ring */
+    pr_dbg("Writing CQE\n");
+    cqe1 = pvrdma_ring_next_elem_write(ring);
+    if (unlikely(!cqe1)) {
+        return -EINVAL;
+    }
+
+    cqe1->wr_id = cqe->wr_id;
+    cqe1->qp = cqe->qp;
+    cqe1->opcode = cqe->opcode;
+    cqe1->status = cqe->status;
+    cqe1->vendor_err = cqe->vendor_err;
+
+    pvrdma_ring_write_inc(ring);
+
+    /* Step #2: Put CQ number on dsr completion ring */
+    pr_dbg("Writing CQNE\n");
+    cqne = pvrdma_ring_next_elem_write(&dev->dsr_info.cq);
+    if (unlikely(!cqne)) {
+        return -EINVAL;
+    }
+
+    cqne->info = cq_handle;
+    pvrdma_ring_write_inc(&dev->dsr_info.cq);
+
+    pr_dbg("cq->notify=%d\n", cq->notify);
+    if (cq->notify) {
+        cq->notify = false;
+        post_interrupt(dev, INTR_VEC_CMD_COMPLETION_Q);
+    }
+
+    return 0;
+}
+
+static void pvrdma_qp_ops_comp_handler(int status, unsigned int vendor_err,
+                                       void *ctx)
+{
+    CompHandlerCtx *comp_ctx = (CompHandlerCtx *)ctx;
+
+    pr_dbg("cq_handle=%d\n", comp_ctx->cq_handle);
+    pr_dbg("wr_id=%lld\n", comp_ctx->cqe.wr_id);
+    pr_dbg("status=%d\n", status);
+    pr_dbg("vendor_err=0x%x\n", vendor_err);
+    comp_ctx->cqe.status = status;
+    comp_ctx->cqe.vendor_err = vendor_err;
+    pvrdma_post_cqe(comp_ctx->dev, comp_ctx->cq_handle, &comp_ctx->cqe);
+    free(ctx);
+}
+
+void pvrdma_qp_ops_fini(void)
+{
+    rdma_backend_unregister_comp_handler();
+}
+
+int pvrdma_qp_ops_init(void)
+{
+    rdma_backend_register_comp_handler(pvrdma_qp_ops_comp_handler);
+
+    return 0;
+}
+
+int pvrdma_qp_send(PVRDMADev *dev, __u32 qp_handle)
+{
+    RdmaRmQP *qp;
+    PvrdmaSqWqe *wqe;
+    PvrdmaRing *ring;
+
+    qp = rdma_rm_get_qp(&dev->rdma_dev_res, qp_handle);
+    if (unlikely(!qp)) {
+        return -EINVAL;
+    }
+
+    ring = (PvrdmaRing *)qp->opaque;
+    pr_dbg("sring=%p\n", ring);
+
+    if (qp->qp_state < IBV_QPS_RTS) {
+        pr_dbg("Invalid QP state for send (%d < %d) for qp %d\n", qp->qp_state,
+               IBV_QPS_RTS, qp_handle);
+        return -EINVAL;
+    }
+
+    wqe = (struct PvrdmaSqWqe *)pvrdma_ring_next_elem_read(ring);
+    while (wqe) {
+        CompHandlerCtx *comp_ctx;
+
+        pr_dbg("wr_id=%lld\n", wqe->hdr.wr_id);
+
+        /* Prepare CQE */
+        comp_ctx = malloc(sizeof(CompHandlerCtx));
+        comp_ctx->dev = dev;
+        comp_ctx->cq_handle = qp->send_cq_handle;
+        comp_ctx->cqe.wr_id = wqe->hdr.wr_id;
+        comp_ctx->cqe.qp = qp_handle;
+        comp_ctx->cqe.opcode = wqe->hdr.opcode;
+
+        rdma_backend_post_send(&dev->backend_dev, &dev->rdma_dev_res,
+                               &qp->backend_qp, qp->qp_type,
+                               (struct ibv_sge *)&wqe->sge[0], wqe->hdr.num_sge,
+                               (union ibv_gid *)wqe->hdr.wr.ud.av.dgid,
+                               wqe->hdr.wr.ud.remote_qpn,
+                               wqe->hdr.wr.ud.remote_qkey, comp_ctx);
+
+        pvrdma_ring_read_inc(ring);
+
+        wqe = pvrdma_ring_next_elem_read(ring);
+    }
+
+    return 0;
+}
+
+int pvrdma_qp_recv(PVRDMADev *dev, __u32 qp_handle)
+{
+    RdmaRmQP *qp;
+    PvrdmaRqWqe *wqe;
+    PvrdmaRing *ring;
+
+    pr_dbg("qp_handle=%d\n", qp_handle);
+
+    qp = rdma_rm_get_qp(&dev->rdma_dev_res, qp_handle);
+    if (unlikely(!qp)) {
+        return -EINVAL;
+    }
+
+    ring = &((PvrdmaRing *)qp->opaque)[1];
+    pr_dbg("rring=%p\n", ring);
+
+    wqe = (struct PvrdmaRqWqe *)pvrdma_ring_next_elem_read(ring);
+    while (wqe) {
+        CompHandlerCtx *comp_ctx;
+
+        pr_dbg("wr_id=%lld\n", wqe->hdr.wr_id);
+
+        /* Prepare CQE */
+        comp_ctx = malloc(sizeof(CompHandlerCtx));
+        comp_ctx->dev = dev;
+        comp_ctx->cq_handle = qp->recv_cq_handle;
+        comp_ctx->cqe.qp = qp_handle;
+        comp_ctx->cqe.wr_id = wqe->hdr.wr_id;
+
+        rdma_backend_post_recv(&dev->backend_dev, &dev->rdma_dev_res,
+                               &qp->backend_qp, qp->qp_type,
+                               (struct ibv_sge *)&wqe->sge[0], wqe->hdr.num_sge,
+                               comp_ctx);
+
+        pvrdma_ring_read_inc(ring);
+
+        wqe = pvrdma_ring_next_elem_read(ring);
+    }
+
+    return 0;
+}
+
+void pvrdma_cq_poll(RdmaDeviceResources *dev_res, __u32 cq_handle)
+{
+    RdmaRmCQ *cq;
+
+    cq = rdma_rm_get_cq(dev_res, cq_handle);
+    if (!cq) {
+        pr_dbg("Invalid CQ# %d\n", cq_handle);
+    }
+
+    rdma_backend_poll_cq(dev_res, &cq->backend_cq);
+}
diff --git a/hw/rdma/vmw/pvrdma_qp_ops.h b/hw/rdma/vmw/pvrdma_qp_ops.h
new file mode 100644
index 0000000000..61ec036835
--- /dev/null
+++ b/hw/rdma/vmw/pvrdma_qp_ops.h
@@ -0,0 +1,27 @@
+/*
+ * QEMU VMWARE paravirtual RDMA QP Operations
+ *
+ * Copyright (C) 2018 Oracle
+ * Copyright (C) 2018 Red Hat Inc
+ *
+ * Authors:
+ *     Yuval Shaia <yuval.shaia@oracle.com>
+ *     Marcel Apfelbaum <marcel@redhat.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef PVRDMA_QP_H
+#define PVRDMA_QP_H
+
+#include "pvrdma.h"
+
+int pvrdma_qp_ops_init(void);
+void pvrdma_qp_ops_fini(void);
+int pvrdma_qp_send(PVRDMADev *dev, __u32 qp_handle);
+int pvrdma_qp_recv(PVRDMADev *dev, __u32 qp_handle);
+void pvrdma_cq_poll(RdmaDeviceResources *dev_res, __u32 cq_handle);
+
+#endif
diff --git a/hw/rdma/vmw/pvrdma_ring.h b/hw/rdma/vmw/pvrdma_ring.h
new file mode 100644
index 0000000000..c616cc586c
--- /dev/null
+++ b/hw/rdma/vmw/pvrdma_ring.h
@@ -0,0 +1,134 @@
+/*
+ * Copyright (c) 2012-2016 VMware, Inc.  All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of EITHER the GNU General Public License
+ * version 2 as published by the Free Software Foundation or the BSD
+ * 2-Clause License. This program is distributed in the hope that it
+ * will be useful, but WITHOUT ANY WARRANTY; WITHOUT EVEN THE IMPLIED
+ * WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
+ * See the GNU General Public License version 2 for more details at
+ * http://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program available in the file COPYING in the main
+ * directory of this source tree.
+ *
+ * The BSD 2-Clause License
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+ * COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
+ * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+ * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+ * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
+ * OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef PVRDMA_RING_H
+#define PVRDMA_RING_H
+
+#include <qemu/atomic.h>
+#include <linux/types.h>
+
+#define PVRDMA_INVALID_IDX -1 /* Invalid index. */
+
+struct pvrdma_ring {
+    int prod_tail;  /* Producer tail. */
+    int cons_head;  /* Consumer head. */
+};
+
+struct pvrdma_ring_state {
+    struct pvrdma_ring tx; /* Tx ring. */
+    struct pvrdma_ring rx; /* Rx ring. */
+};
+
+static inline int pvrdma_idx_valid(__u32 idx, __u32 max_elems)
+{
+    /* Generates fewer instructions than a less-than. */
+    return (idx & ~((max_elems << 1) - 1)) == 0;
+}
+
+static inline __s32 pvrdma_idx(int *var, __u32 max_elems)
+{
+    const unsigned int idx = atomic_read(var);
+
+    if (pvrdma_idx_valid(idx, max_elems)) {
+        return idx & (max_elems - 1);
+    }
+    return PVRDMA_INVALID_IDX;
+}
+
+static inline void pvrdma_idx_ring_inc(int *var, __u32 max_elems)
+{
+    __u32 idx = atomic_read(var) + 1; /* Increment. */
+
+    idx &= (max_elems << 1) - 1; /* Modulo size, flip gen. */
+    atomic_set(var, idx);
+}
+
+static inline __s32 pvrdma_idx_ring_has_space(const struct pvrdma_ring *r,
+                                              __u32 max_elems, __u32 *out_tail)
+{
+    const __u32 tail = atomic_read(&r->prod_tail);
+    const __u32 head = atomic_read(&r->cons_head);
+
+    if (pvrdma_idx_valid(tail, max_elems) &&
+        pvrdma_idx_valid(head, max_elems)) {
+        *out_tail = tail & (max_elems - 1);
+        return tail != (head ^ max_elems);
+    }
+    return PVRDMA_INVALID_IDX;
+}
+
+static inline __s32 pvrdma_idx_ring_has_data(const struct pvrdma_ring *r,
+                                             __u32 max_elems, __u32 *out_head)
+{
+    const __u32 tail = atomic_read(&r->prod_tail);
+    const __u32 head = atomic_read(&r->cons_head);
+
+    if (pvrdma_idx_valid(tail, max_elems) &&
+        pvrdma_idx_valid(head, max_elems)) {
+        *out_head = head & (max_elems - 1);
+        return tail != head;
+    }
+    return PVRDMA_INVALID_IDX;
+}
+
+static inline bool pvrdma_idx_ring_is_valid_idx(const struct pvrdma_ring *r,
+                                                __u32 max_elems, __u32 *idx)
+{
+    const __u32 tail = atomic_read(&r->prod_tail);
+    const __u32 head = atomic_read(&r->cons_head);
+
+    if (pvrdma_idx_valid(tail, max_elems) &&
+        pvrdma_idx_valid(head, max_elems) &&
+        pvrdma_idx_valid(*idx, max_elems)) {
+        if (tail > head && (*idx < tail && *idx >= head)) {
+            return true;
+        } else if (head > tail && (*idx >= head || *idx < tail)) {
+            return true;
+        }
+    }
+    return false;
+}
+
+#endif /* PVRDMA_RING_H */
diff --git a/hw/rdma/vmw/trace-events b/hw/rdma/vmw/trace-events
new file mode 100644
index 0000000000..dd4dde311e
--- /dev/null
+++ b/hw/rdma/vmw/trace-events
@@ -0,0 +1,5 @@
+# See docs/tracing.txt for syntax documentation.
+
+# hw/rdma/vmw/pvrdma_main.c
+pvrdma_regs_read(uint64_t addr, uint64_t val) "regs[0x%lx] = 0x%lx"
+pvrdma_regs_write(uint64_t addr, uint64_t val) "regs[0x%lx] = 0x%lx"
diff --git a/hw/rdma/vmw/vmw_pvrdma-abi.h b/hw/rdma/vmw/vmw_pvrdma-abi.h
new file mode 100644
index 0000000000..8cfb9d7745
--- /dev/null
+++ b/hw/rdma/vmw/vmw_pvrdma-abi.h
@@ -0,0 +1,311 @@
+/*
+ * QEMU VMWARE paravirtual RDMA device definitions
+ *
+ * Copyright (C) 2018 Oracle
+ * Copyright (C) 2018 Red Hat Inc
+ *
+ * Authors:
+ *     Yuval Shaia <yuval.shaia@oracle.com>
+ *     Marcel Apfelbaum <marcel@redhat.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef VMW_PVRDMA_ABI_H
+#define VMW_PVRDMA_ABI_H
+
+/*
+ * Following is an interface definition for PVRDMA device as provided by
+ * VMWARE.
+ * See original copyright from Linux kernel v4.14.5 header file
+ * include/uapi/rdma/vmw_pvrdma-abi.h
+ */
+
+/*
+ * Copyright (c) 2012-2016 VMware, Inc.  All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of EITHER the GNU General Public License
+ * version 2 as published by the Free Software Foundation or the BSD
+ * 2-Clause License. This program is distributed in the hope that it
+ * will be useful, but WITHOUT ANY WARRANTY; WITHOUT EVEN THE IMPLIED
+ * WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
+ * See the GNU General Public License version 2 for more details at
+ * http://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program available in the file COPYING in the main
+ * directory of this source tree.
+ *
+ * The BSD 2-Clause License
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+ * COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
+ * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+ * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+ * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
+ * OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <linux/types.h>
+
+#define PVRDMA_UVERBS_ABI_VERSION    3        /* ABI Version. */
+#define PVRDMA_UAR_HANDLE_MASK        0x00FFFFFF    /* Bottom 24 bits. */
+#define PVRDMA_UAR_QP_OFFSET        0        /* QP doorbell. */
+#define PVRDMA_UAR_QP_SEND        BIT(30)        /* Send bit. */
+#define PVRDMA_UAR_QP_RECV        BIT(31)        /* Recv bit. */
+#define PVRDMA_UAR_CQ_OFFSET        4        /* CQ doorbell. */
+#define PVRDMA_UAR_CQ_ARM_SOL        BIT(29)        /* Arm solicited bit. */
+#define PVRDMA_UAR_CQ_ARM        BIT(30)        /* Arm bit. */
+#define PVRDMA_UAR_CQ_POLL        BIT(31)        /* Poll bit. */
+
+enum pvrdma_wr_opcode {
+    PVRDMA_WR_RDMA_WRITE,
+    PVRDMA_WR_RDMA_WRITE_WITH_IMM,
+    PVRDMA_WR_SEND,
+    PVRDMA_WR_SEND_WITH_IMM,
+    PVRDMA_WR_RDMA_READ,
+    PVRDMA_WR_ATOMIC_CMP_AND_SWP,
+    PVRDMA_WR_ATOMIC_FETCH_AND_ADD,
+    PVRDMA_WR_LSO,
+    PVRDMA_WR_SEND_WITH_INV,
+    PVRDMA_WR_RDMA_READ_WITH_INV,
+    PVRDMA_WR_LOCAL_INV,
+    PVRDMA_WR_FAST_REG_MR,
+    PVRDMA_WR_MASKED_ATOMIC_CMP_AND_SWP,
+    PVRDMA_WR_MASKED_ATOMIC_FETCH_AND_ADD,
+    PVRDMA_WR_BIND_MW,
+    PVRDMA_WR_REG_SIG_MR,
+};
+
+enum pvrdma_wc_status {
+    PVRDMA_WC_SUCCESS,
+    PVRDMA_WC_LOC_LEN_ERR,
+    PVRDMA_WC_LOC_QP_OP_ERR,
+    PVRDMA_WC_LOC_EEC_OP_ERR,
+    PVRDMA_WC_LOC_PROT_ERR,
+    PVRDMA_WC_WR_FLUSH_ERR,
+    PVRDMA_WC_MW_BIND_ERR,
+    PVRDMA_WC_BAD_RESP_ERR,
+    PVRDMA_WC_LOC_ACCESS_ERR,
+    PVRDMA_WC_REM_INV_REQ_ERR,
+    PVRDMA_WC_REM_ACCESS_ERR,
+    PVRDMA_WC_REM_OP_ERR,
+    PVRDMA_WC_RETRY_EXC_ERR,
+    PVRDMA_WC_RNR_RETRY_EXC_ERR,
+    PVRDMA_WC_LOC_RDD_VIOL_ERR,
+    PVRDMA_WC_REM_INV_RD_REQ_ERR,
+    PVRDMA_WC_REM_ABORT_ERR,
+    PVRDMA_WC_INV_EECN_ERR,
+    PVRDMA_WC_INV_EEC_STATE_ERR,
+    PVRDMA_WC_FATAL_ERR,
+    PVRDMA_WC_RESP_TIMEOUT_ERR,
+    PVRDMA_WC_GENERAL_ERR,
+};
+
+enum pvrdma_wc_opcode {
+    PVRDMA_WC_SEND,
+    PVRDMA_WC_RDMA_WRITE,
+    PVRDMA_WC_RDMA_READ,
+    PVRDMA_WC_COMP_SWAP,
+    PVRDMA_WC_FETCH_ADD,
+    PVRDMA_WC_BIND_MW,
+    PVRDMA_WC_LSO,
+    PVRDMA_WC_LOCAL_INV,
+    PVRDMA_WC_FAST_REG_MR,
+    PVRDMA_WC_MASKED_COMP_SWAP,
+    PVRDMA_WC_MASKED_FETCH_ADD,
+    PVRDMA_WC_RECV = 1 << 7,
+    PVRDMA_WC_RECV_RDMA_WITH_IMM,
+};
+
+enum pvrdma_wc_flags {
+    PVRDMA_WC_GRH            = 1 << 0,
+    PVRDMA_WC_WITH_IMM        = 1 << 1,
+    PVRDMA_WC_WITH_INVALIDATE    = 1 << 2,
+    PVRDMA_WC_IP_CSUM_OK        = 1 << 3,
+    PVRDMA_WC_WITH_SMAC        = 1 << 4,
+    PVRDMA_WC_WITH_VLAN        = 1 << 5,
+    PVRDMA_WC_FLAGS_MAX        = PVRDMA_WC_WITH_VLAN,
+};
+
+struct pvrdma_alloc_ucontext_resp {
+    __u32 qp_tab_size;
+    __u32 reserved;
+};
+
+struct pvrdma_alloc_pd_resp {
+    __u32 pdn;
+    __u32 reserved;
+};
+
+struct pvrdma_create_cq {
+    __u64 buf_addr;
+    __u32 buf_size;
+    __u32 reserved;
+};
+
+struct pvrdma_create_cq_resp {
+    __u32 cqn;
+    __u32 reserved;
+};
+
+struct pvrdma_resize_cq {
+    __u64 buf_addr;
+    __u32 buf_size;
+    __u32 reserved;
+};
+
+struct pvrdma_create_srq {
+    __u64 buf_addr;
+};
+
+struct pvrdma_create_srq_resp {
+    __u32 srqn;
+    __u32 reserved;
+};
+
+struct pvrdma_create_qp {
+    __u64 rbuf_addr;
+    __u64 sbuf_addr;
+    __u32 rbuf_size;
+    __u32 sbuf_size;
+    __u64 qp_addr;
+};
+
+/* PVRDMA masked atomic compare and swap */
+struct pvrdma_ex_cmp_swap {
+    __u64 swap_val;
+    __u64 compare_val;
+    __u64 swap_mask;
+    __u64 compare_mask;
+};
+
+/* PVRDMA masked atomic fetch and add */
+struct pvrdma_ex_fetch_add {
+    __u64 add_val;
+    __u64 field_boundary;
+};
+
+/* PVRDMA address vector. */
+struct pvrdma_av {
+    __u32 port_pd;
+    __u32 sl_tclass_flowlabel;
+    __u8 dgid[16];
+    __u8 src_path_bits;
+    __u8 gid_index;
+    __u8 stat_rate;
+    __u8 hop_limit;
+    __u8 dmac[6];
+    __u8 reserved[6];
+};
+
+/* PVRDMA scatter/gather entry */
+struct pvrdma_sge {
+    __u64   addr;
+    __u32   length;
+    __u32   lkey;
+};
+
+/* PVRDMA receive queue work request */
+struct pvrdma_rq_wqe_hdr {
+    __u64 wr_id;        /* wr id */
+    __u32 num_sge;        /* size of s/g array */
+    __u32 total_len;    /* reserved */
+};
+/* Use pvrdma_sge (ib_sge) for receive queue s/g array elements. */
+
+/* PVRDMA send queue work request */
+struct pvrdma_sq_wqe_hdr {
+    __u64 wr_id;        /* wr id */
+    __u32 num_sge;        /* size of s/g array */
+    __u32 total_len;    /* reserved */
+    __u32 opcode;        /* operation type */
+    __u32 send_flags;    /* wr flags */
+    union {
+        __be32 imm_data;
+        __u32 invalidate_rkey;
+    } ex;
+    __u32 reserved;
+    union {
+        struct {
+            __u64 remote_addr;
+            __u32 rkey;
+            __u8 reserved[4];
+        } rdma;
+        struct {
+            __u64 remote_addr;
+            __u64 compare_add;
+            __u64 swap;
+            __u32 rkey;
+            __u32 reserved;
+        } atomic;
+        struct {
+            __u64 remote_addr;
+            __u32 log_arg_sz;
+            __u32 rkey;
+            union {
+                struct pvrdma_ex_cmp_swap  cmp_swap;
+                struct pvrdma_ex_fetch_add fetch_add;
+            } wr_data;
+        } masked_atomics;
+        struct {
+            __u64 iova_start;
+            __u64 pl_pdir_dma;
+            __u32 page_shift;
+            __u32 page_list_len;
+            __u32 length;
+            __u32 access_flags;
+            __u32 rkey;
+        } fast_reg;
+        struct {
+            __u32 remote_qpn;
+            __u32 remote_qkey;
+            struct pvrdma_av av;
+        } ud;
+    } wr;
+};
+/* Use pvrdma_sge (ib_sge) for send queue s/g array elements. */
+
+/* Completion queue element. */
+struct pvrdma_cqe {
+    __u64 wr_id;
+    __u64 qp;
+    __u32 opcode;
+    __u32 status;
+    __u32 byte_len;
+    __be32 imm_data;
+    __u32 src_qp;
+    __u32 wc_flags;
+    __u32 vendor_err;
+    __u16 pkey_index;
+    __u16 slid;
+    __u8 sl;
+    __u8 dlid_path_bits;
+    __u8 port_num;
+    __u8 smac[6];
+    __u8 reserved2[7]; /* Pad to next power of 2 (64). */
+};
+
+#endif /* VMW_PVRDMA_ABI_H */
diff --git a/include/hw/pci/pci_ids.h b/include/hw/pci/pci_ids.h
index 35df1874a9..1dbf53627c 100644
--- a/include/hw/pci/pci_ids.h
+++ b/include/hw/pci/pci_ids.h
@@ -266,4 +266,7 @@
 #define PCI_VENDOR_ID_TEWS               0x1498
 #define PCI_DEVICE_ID_TEWS_TPCI200       0x30C8
 
+#define PCI_VENDOR_ID_VMWARE             0x15ad
+#define PCI_DEVICE_ID_VMWARE_PVRDMA      0x0820
+
 #endif
-- 
2.13.5

Re: [Qemu-devel] [PATCH V8 3/4] pvrdma: initial implementation

[Michael S. Tsirkin]

> diff --git a/hw/rdma/vmw/vmw_pvrdma-abi.h b/hw/rdma/vmw/vmw_pvrdma-abi.h
> new file mode 100644
> index 0000000000..8cfb9d7745
> --- /dev/null
> +++ b/hw/rdma/vmw/vmw_pvrdma-abi.h

Why do you copy this here? Why not import it
from include/uapi/rdma/vmw_pvrdma-abi.h in Linux into
standard headers?

-- 
MST

Re: [Qemu-devel] [PATCH V8 3/4] pvrdma: initial implementation

[Marcel Apfelbaum]

Hi Michael,

On 01/02/2018 21:10, Michael S. Tsirkin wrote:
>> diff --git a/hw/rdma/vmw/vmw_pvrdma-abi.h b/hw/rdma/vmw/vmw_pvrdma-abi.h
>> new file mode 100644
>> index 0000000000..8cfb9d7745
>> --- /dev/null
>> +++ b/hw/rdma/vmw/vmw_pvrdma-abi.h
> 
> Why do you copy this here? Why not import it
> from include/uapi/rdma/vmw_pvrdma-abi.h in Linux into
> standard headers?
> 

Do you mean adding it to QEMU's include/standard-headers/ ?
I am sorry for the possibly dumb question, but is there
a special way to do it or we simply copy it?


Regarding why I am not sure is worth it (Cornelia asked the same question
some time ago):
We are only using the header because we are too lazy to copy the structs
that are passed between the guest driver and QEMU.
This ABI is against the pvrdma device and we don't want/need to update
it every time it gets updated in Linux kernel.
If such ABI will be changed, the command channel version will be updated
but we will support only the current version.

The bottom line: we don't want to depend on the Linux kernel version,
we want to import it only once. When we will want to upgrade, we want
us to be the ones to decide and not re-act.

(We saw the same pattern in other implementations)

Thanks,
Marcel

[Qemu-devel] [PATCH V8 4/4] MAINTAINERS: add entry for hw/rdma

[Marcel Apfelbaum]

Signed-off-by: Marcel Apfelbaum <marcel@redhat.com>
Signed-off-by: Yuval Shaia <yuval.shaia@oracle.com>
---
 MAINTAINERS | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 4770f105d4..fc4f54eebb 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1984,6 +1984,14 @@ F: block/replication.c
 F: tests/test-replication.c
 F: docs/block-replication.txt
 
+PVRDMA
+M: Yuval Shaia <yuval.shaia@oracle.com>
+M: Marcel Apfelbaum <marcel@redhat.com>
+S: Maintained
+F: hw/rdma/*
+F: hw/rdma/vmw/*
+F: docs/pvrdma.txt
+
 Build and test automation
 -------------------------
 Build and test automation
-- 
2.13.5

Re: [Qemu-devel] [PATCH V8 0/4] hw/pvrdma: PVRDMA device implementation

[no-reply]

Hi,

This series failed build test on ppc host. Please find the details below.

Subject: [Qemu-devel] [PATCH V8 0/4] hw/pvrdma: PVRDMA device implementation
Type: series
Message-id: 20180117095421.124787-1-marcel@redhat.com

=== TEST SCRIPT BEGIN ===
#!/bin/bash
# Testing script will be invoked under the git checkout with
# HEAD pointing to a commit that has the patches applied on top of "base"
# branch
set -e
echo "=== ENV ==="
env
echo "=== PACKAGES ==="
rpm -qa
echo "=== TEST BEGIN ==="
INSTALL=$PWD/install
BUILD=$PWD/build
mkdir -p $BUILD $INSTALL
SRC=$PWD
cd $BUILD
$SRC/configure --prefix=$INSTALL
make -j100
# XXX: we need reliable clean up
# make check -j100 V=1
make install
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
 - [tag update]      patchew/1516034665-27606-1-git-send-email-walling@linux.vnet.ibm.com -> patchew/1516034665-27606-1-git-send-email-walling@linux.vnet.ibm.com
 - [tag update]      patchew/20180117090700.25811-1-stefanha@redhat.com -> patchew/20180117090700.25811-1-stefanha@redhat.com
 * [new tag]         patchew/20180117095421.124787-1-marcel@redhat.com -> patchew/20180117095421.124787-1-marcel@redhat.com
Submodule 'capstone' (git://git.qemu.org/capstone.git) registered for path 'capstone'
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc'
Submodule 'roms/QemuMacDrivers' (git://git.qemu.org/QemuMacDrivers.git) registered for path 'roms/QemuMacDrivers'
Submodule 'roms/SLOF' (git://git.qemu-project.org/SLOF.git) registered for path 'roms/SLOF'
Submodule 'roms/ipxe' (git://git.qemu-project.org/ipxe.git) registered for path 'roms/ipxe'
Submodule 'roms/openbios' (git://git.qemu-project.org/openbios.git) registered for path 'roms/openbios'
Submodule 'roms/openhackware' (git://git.qemu-project.org/openhackware.git) registered for path 'roms/openhackware'
Submodule 'roms/qemu-palcode' (git://github.com/rth7680/qemu-palcode.git) registered for path 'roms/qemu-palcode'
Submodule 'roms/seabios' (git://git.qemu-project.org/seabios.git/) registered for path 'roms/seabios'
Submodule 'roms/sgabios' (git://git.qemu-project.org/sgabios.git) registered for path 'roms/sgabios'
Submodule 'roms/skiboot' (git://git.qemu.org/skiboot.git) registered for path 'roms/skiboot'
Submodule 'roms/u-boot' (git://git.qemu-project.org/u-boot.git) registered for path 'roms/u-boot'
Submodule 'roms/vgabios' (git://git.qemu-project.org/vgabios.git/) registered for path 'roms/vgabios'
Submodule 'ui/keycodemapdb' (git://git.qemu.org/keycodemapdb.git) registered for path 'ui/keycodemapdb'
Cloning into 'capstone'...
Submodule path 'capstone': checked out '22ead3e0bfdb87516656453336160e0a37b066bf'
Cloning into 'dtc'...
Submodule path 'dtc': checked out 'e54388015af1fb4bf04d0bca99caba1074d9cc42'
Cloning into 'roms/QemuMacDrivers'...
Submodule path 'roms/QemuMacDrivers': checked out 'd4e7d7ac663fcb55f1b93575445fcbca372f17a7'
Cloning into 'roms/SLOF'...
Submodule path 'roms/SLOF': checked out 'fa981320a1e0968d6fc1b8de319723ff8212b337'
Cloning into 'roms/ipxe'...
Submodule path 'roms/ipxe': checked out '0600d3ae94f93efd10fc6b3c7420a9557a3a1670'
Cloning into 'roms/openbios'...
Submodule path 'roms/openbios': checked out '83818bdb4460170621d789429ad5a75e8c73efd1'
Cloning into 'roms/openhackware'...
Submodule path 'roms/openhackware': checked out 'c559da7c8eec5e45ef1f67978827af6f0b9546f5'
Cloning into 'roms/qemu-palcode'...
Submodule path 'roms/qemu-palcode': checked out 'f3c7e44c70254975df2a00af39701eafbac4d471'
Cloning into 'roms/seabios'...
Submodule path 'roms/seabios': checked out '63451fca13c75870e1703eb3e20584d91179aebc'
Cloning into 'roms/sgabios'...
Submodule path 'roms/sgabios': checked out 'cbaee52287e5f32373181cff50a00b6c4ac9015a'
Cloning into 'roms/skiboot'...
Submodule path 'roms/skiboot': checked out '762d0082f18e4fb921a2d44a1051b02d8b0f6381'
Cloning into 'roms/u-boot'...
Submodule path 'roms/u-boot': checked out 'd85ca029f257b53a96da6c2fb421e78a003a9943'
Cloning into 'roms/vgabios'...
Submodule path 'roms/vgabios': checked out '19ea12c230ded95928ecaef0db47a82231c2e485'
Cloning into 'ui/keycodemapdb'...
Submodule path 'ui/keycodemapdb': checked out '10739aa26051a5d49d88132604539d3ed085e72e'
Switched to a new branch 'test'
05d1c2d MAINTAINERS: add entry for hw/rdma
1cf6dca pvrdma: initial implementation
ced1cca docs: add pvrdma device documentation.
2c82094 mem: add share parameter to memory-backend-ram

=== OUTPUT BEGIN ===
=== ENV ===
XDG_SESSION_ID=141128
SHELL=/bin/sh
USER=patchew
PATCHEW=/home/patchew/patchew/patchew-cli -s http://patchew.org --nodebug
PATH=/usr/bin:/bin
PWD=/var/tmp/patchew-tester-tmp-0kkotmms/src
LANG=en_US.UTF-8
HOME=/home/patchew
SHLVL=2
LOGNAME=patchew
XDG_RUNTIME_DIR=/run/user/1000
_=/usr/bin/env
=== PACKAGES ===
plymouth-core-libs-0.8.9-0.28.20140113.el7.centos.ppc64le
vim-common-7.4.160-2.el7.ppc64le
perl-Test-Simple-0.98-243.el7.noarch
hplip-common-3.15.9-3.el7.ppc64le
valgrind-3.12.0-8.el7.ppc64le
gamin-0.1.10-16.el7.ppc64le
libpeas-loader-python-1.20.0-1.el7.ppc64le
telepathy-filesystem-0.0.2-6.el7.noarch
colord-libs-1.3.4-1.el7.ppc64le
kbd-legacy-1.15.5-13.el7.noarch
perl-CPAN-Meta-YAML-0.008-14.el7.noarch
libvirt-daemon-driver-nwfilter-3.2.0-14.el7.ppc64le
ntsysv-1.7.4-1.el7.ppc64le
kernel-bootwrapper-3.10.0-693.el7.ppc64le
telepathy-farstream-0.6.0-5.el7.ppc64le
kdenetwork-common-4.10.5-8.el7_0.noarch
elfutils-devel-0.168-8.el7.ppc64le
pm-utils-1.4.1-27.el7.ppc64le
perl-Error-0.17020-2.el7.noarch
usbmuxd-1.1.0-1.el7.ppc64le
bzip2-devel-1.0.6-13.el7.ppc64le
blktrace-1.0.5-8.el7.ppc64le
gnome-keyring-pam-3.20.0-3.el7.ppc64le
tzdata-java-2017b-1.el7.noarch
perl-devel-5.16.3-292.el7.ppc64le
gnome-getting-started-docs-3.22.0-1.el7.noarch
perl-Log-Message-Simple-0.10-2.el7.noarch
totem-pl-parser-3.10.7-1.el7.ppc64le
lohit-oriya-fonts-2.5.4.1-3.el7.noarch
python-coverage-3.6-0.5.b3.el7.ppc64le
java-1.7.0-openjdk-1.7.0.141-2.6.10.5.el7.ppc64le
mailcap-2.1.41-2.el7.noarch
perl-CPANPLUS-0.91.38-4.el7.noarch
fprintd-pam-0.5.0-4.0.el7_0.ppc64le
less-458-9.el7.ppc64le
gupnp-igd-0.2.4-1.el7.ppc64le
thai-scalable-waree-fonts-0.5.0-7.el7.noarch
python-di-0.3-2.el7.noarch
yelp-libs-3.22.0-1.el7.ppc64le
vte-profile-0.46.2-1.el7.ppc64le
gpm-libs-1.20.7-5.el7.ppc64le
gnome-clocks-3.22.1-1.el7.ppc64le
p11-kit-trust-0.23.5-3.el7.ppc64le
gssproxy-0.7.0-4.el7.ppc64le
gnu-free-mono-fonts-20120503-8.el7.noarch
python-dateutil-1.5-7.el7.noarch
gucharmap-libs-3.18.2-1.el7.ppc64le
glibc-common-2.17-196.el7.ppc64le
libreport-plugin-mantisbt-2.1.11-38.el7.centos.ppc64le
motif-devel-2.3.4-8.1.el7_3.ppc64le
celt051-0.5.1.3-8.el7.ppc64le
radvd-1.9.2-9.el7.ppc64le
lohit-tamil-fonts-2.5.3-2.el7.noarch
python-ipaddress-1.0.16-2.el7.noarch
anaconda-widgets-21.48.22.121-1.el7.centos.ppc64le
zlib-1.2.7-17.el7.ppc64le
system-config-printer-1.4.1-19.el7.ppc64le
mozjs24-24.2.0-7.el7.ppc64le
device-mapper-multipath-libs-0.4.9-111.el7.ppc64le
wqy-microhei-fonts-0.2.0-0.12.beta.el7.noarch
python-schedutils-0.4-6.el7.ppc64le
gnome-bluetooth-3.20.1-1.el7.ppc64le
nss-util-3.28.4-3.el7.ppc64le
dotconf-1.3-8.el7.ppc64le
ibus-rawcode-1.3.2-3.el7.ppc64le
abattis-cantarell-fonts-0.0.25-1.el7.noarch
sssd-common-1.15.2-50.el7.ppc64le
sil-padauk-fonts-2.8-5.el7.noarch
bind-utils-9.9.4-50.el7.ppc64le
sox-14.4.1-6.el7.ppc64le
libSM-1.2.2-2.el7.ppc64le
libtiff-devel-4.0.3-27.el7_3.ppc64le
plymouth-system-theme-0.8.9-0.28.20140113.el7.centos.ppc64le
python-libs-2.7.5-58.el7.ppc64le
sssd-1.15.2-50.el7.ppc64le
rfkill-0.4-9.el7.ppc64le
cyrus-sasl-md5-2.1.26-21.el7.ppc64le
libXtst-devel-1.2.3-1.el7.ppc64le
avahi-libs-0.6.31-17.el7.ppc64le
ruby-2.0.0.648-30.el7.ppc64le
seahorse-3.20.0-1.el7.ppc64le
python-six-1.9.0-2.el7.noarch
gpgme-1.3.2-5.el7.ppc64le
iwl7260-firmware-22.0.7.0-56.el7.noarch
libsss_certmap-1.15.2-50.el7.ppc64le
xorg-x11-drv-wacom-0.34.2-2.el7.ppc64le
libXau-1.0.8-2.1.el7.ppc64le
shadow-utils-4.1.5.1-24.el7.ppc64le
evolution-ews-3.22.6-6.el7.ppc64le
libsecret-0.18.5-2.el7.ppc64le
perl-Module-Signature-0.73-2.el7.noarch
rootfiles-8.1-11.el7.noarch
trace-cmd-2.6.0-8.el7.ppc64le
hamcrest-1.3-6.el7.noarch
gawk-4.0.2-4.el7_3.1.ppc64le
usermode-1.111-5.el7.ppc64le
gnome-terminal-nautilus-3.22.1-2.el7.ppc64le
gvfs-client-1.30.4-3.el7.ppc64le
yum-utils-1.1.31-42.el7.noarch
iwl3945-firmware-15.32.2.9-56.el7.noarch
perl-Archive-Zip-1.30-11.el7.noarch
spice-glib-0.33-6.el7.ppc64le
augeas-libs-1.4.0-2.el7.ppc64le
openlmi-providers-0.5.0-4.el7.ppc64le
gnome-color-manager-3.22.2-1.el7.ppc64le
imsettings-libs-1.6.3-9.el7.ppc64le
nss-softokn-devel-3.28.3-6.el7.ppc64le
python34-3.4.5-4.el7.ppc64le
perl-DBI-1.627-4.el7.ppc64le
plymouth-plugin-label-0.8.9-0.28.20140113.el7.centos.ppc64le
binutils-2.25.1-31.base.el7.ppc64le
libsss_nss_idmap-1.15.2-50.el7.ppc64le
gvfs-smb-1.30.4-3.el7.ppc64le
freetype-devel-2.4.11-15.el7.ppc64le
libXi-1.7.9-1.el7.ppc64le
libitm-4.8.5-16.el7_4.1.ppc64le
perl-Text-Diff-1.41-5.el7.noarch
gcr-devel-3.20.0-1.el7.ppc64le
numactl-libs-2.0.9-6.el7_2.ppc64le
hardlink-1.0-19.el7.ppc64le
gnome-disk-utility-3.22.1-1.el7.ppc64le
mariadb-libs-5.5.56-2.el7.ppc64le
libnotify-0.7.7-1.el7.ppc64le
perl-TimeDate-2.30-2.el7.noarch
soprano-devel-2.9.2-3.el7.ppc64le
pixman-0.34.0-1.el7.ppc64le
kmod-20-15.el7.ppc64le
qt3-PostgreSQL-3.3.8b-51.el7.ppc64le
python2-pyasn1-0.1.9-7.el7.noarch
libXt-1.1.5-3.el7.ppc64le
perl-Font-AFM-1.20-13.el7.noarch
ibus-1.5.3-13.el7.ppc64le
findutils-4.5.11-5.el7.ppc64le
ibus-libs-1.5.3-13.el7.ppc64le
iprutils-2.4.14.1-1.el7.ppc64le
libpwquality-1.2.3-4.el7.ppc64le
libXrender-devel-0.9.10-1.el7.ppc64le
perl-IO-stringy-2.110-22.el7.noarch
kdelibs-4.14.8-6.el7_3.ppc64le
flac-libs-1.3.0-5.el7_1.ppc64le
device-mapper-event-libs-1.02.140-8.el7.ppc64le
gnutls-devel-3.3.26-9.el7.ppc64le
libXau-devel-1.0.8-2.1.el7.ppc64le
gstreamer1-plugins-base-1.10.4-1.el7.ppc64le
perl-HTML-Tree-5.03-2.el7.noarch
kdenetwork-kopete-4.10.5-8.el7_0.ppc64le
libepoxy-1.3.1-1.el7.ppc64le
mesa-libGLES-17.0.1-6.20170307.el7.ppc64le
qt-postgresql-4.8.5-13.el7.ppc64le
fontconfig-devel-2.10.95-11.el7.ppc64le
java-1.8.0-openjdk-headless-1.8.0.131-11.b12.el7.ppc64le
libXfont-1.5.2-1.el7.ppc64le
libkexiv2-4.10.5-3.el7.ppc64le
openjpeg-libs-1.5.1-17.el7.ppc64le
iscsi-initiator-utils-6.2.0.874-4.el7.ppc64le
NetworkManager-adsl-1.8.0-9.el7.ppc64le
libgtop2-2.34.2-1.el7.ppc64le
libXdamage-devel-1.1.4-4.1.el7.ppc64le
ipset-libs-6.29-1.el7.ppc64le
kde-runtime-drkonqi-4.10.5-8.el7.ppc64le
e2fsprogs-libs-1.42.9-10.el7.ppc64le
dhclient-4.2.5-58.el7.centos.ppc64le
usbutils-007-5.el7.ppc64le
python-ethtool-0.8-5.el7.ppc64le
gstreamer1-plugins-bad-free-1.10.4-2.el7.ppc64le
fftw-libs-double-3.3.3-8.el7.ppc64le
kdenetwork-krdc-4.10.5-8.el7_0.ppc64le
fuse-libs-2.9.2-8.el7.ppc64le
pciutils-3.5.1-2.el7.ppc64le
at-3.1.13-22.el7.ppc64le
python-IPy-0.75-6.el7.noarch
libXp-1.0.2-2.1.el7.ppc64le
vim-minimal-7.4.160-2.el7.ppc64le
kdesdk-kmtrace-4.10.5-6.el7.ppc64le
libraw1394-2.1.0-2.el7.ppc64le
libdrm-devel-2.4.74-1.el7.ppc64le
irqbalance-1.0.7-10.el7.ppc64le
fipscheck-lib-1.4.1-6.el7.ppc64le
gvfs-1.30.4-3.el7.ppc64le
libiscsi-1.9.0-7.el7.ppc64le
motif-2.3.4-8.1.el7_3.ppc64le
keyutils-1.5.8-3.el7.ppc64le
NetworkManager-ppp-1.8.0-9.el7.ppc64le
systemtap-3.1-3.el7.ppc64le
boost-serialization-1.53.0-27.el7.ppc64le
grilo-0.3.3-1.el7.ppc64le
rpm-4.11.3-25.el7.ppc64le
kdegraphics-libs-4.10.5-3.el7.noarch
libfontenc-1.1.3-3.el7.ppc64le
perl-Git-1.8.3.1-11.el7.noarch
rubygem-abrt-0.3.0-1.el7.noarch
tcl-8.5.13-8.el7.ppc64le
gtksourceview3-3.22.2-1.el7.ppc64le
cmake-2.8.12.2-2.el7.ppc64le
pulseaudio-utils-10.0-3.el7.ppc64le
libusal-1.1.11-23.el7.ppc64le
grub2-ppc64le-2.02-0.64.el7.centos.ppc64le
libreport-plugin-mailx-2.1.11-38.el7.centos.ppc64le
libvisual-0.4.0-16.el7.ppc64le
metacity-2.34.13-7.el7.ppc64le
redland-virtuoso-1.0.16-6.el7.ppc64le
nautilus-3.22.3-3.el7.ppc64le
pciutils-libs-3.5.1-2.el7.ppc64le
soprano-2.9.2-3.el7.ppc64le
mariadb-devel-5.5.56-2.el7.ppc64le
libxkbcommon-x11-0.7.1-1.el7.ppc64le
farstream02-0.2.3-3.el7.ppc64le
redhat-rpm-config-9.1.0-76.el7.centos.noarch
skkdic-20130104-6.T1435.el7.noarch
perl-HTTP-Tiny-0.033-3.el7.noarch
lvm2-libs-2.02.171-8.el7.ppc64le
perl-XML-Grove-0.46alpha-52.el7.noarch
boost-devel-1.53.0-27.el7.ppc64le
pycairo-1.8.10-8.el7.ppc64le
popt-devel-1.13-16.el7.ppc64le
gnome-settings-daemon-3.22.2-5.el7.ppc64le
perl-Socket-2.010-4.el7.ppc64le
numad-0.5-17.20150602git.el7.ppc64le
e2fsprogs-devel-1.42.9-10.el7.ppc64le
libsecret-devel-0.18.5-2.el7.ppc64le
libXv-devel-1.0.11-1.el7.ppc64le
libchewing-0.3.4-6.el7.ppc64le
gnome-shell-extension-places-menu-3.22.2-10.el7.noarch
perl-Time-HiRes-1.9725-3.el7.ppc64le
openchange-2.3-2.el7.ppc64le
audit-libs-devel-2.7.6-3.el7.ppc64le
python-dmidecode-3.12.2-1.el7.ppc64le
libmediaart-1.9.1-1.el7.ppc64le
elfutils-default-yama-scope-0.168-8.el7.noarch
quota-4.01-14.el7.ppc64le
perl-threads-1.87-4.el7.ppc64le
realmd-0.16.1-9.el7.ppc64le
nautilus-sendto-3.8.4-1.el7.ppc64le
gstreamer-0.10.36-7.el7.ppc64le
cairo-gobject-devel-1.14.8-2.el7.ppc64le
abrt-libs-2.1.11-48.el7.centos.ppc64le
libvirt-daemon-driver-storage-iscsi-3.2.0-14.el7.ppc64le
perl-Pod-Parser-1.61-2.el7.noarch
python-devel-2.7.5-58.el7.ppc64le
mpfr-devel-3.1.1-4.el7.ppc64le
kernel-headers-3.10.0-693.el7.ppc64le
powerpc-utils-python-1.2.1-9.el7.noarch
linux-firmware-20170606-56.gitc990aae.el7.noarch
libqmi-1.16.0-1.el7.ppc64le
libvirt-libs-3.2.0-14.el7.ppc64le
perl-Digest-1.17-245.el7.noarch
libgcab1-0.7-3.el7.ppc64le
flex-2.5.37-3.el7.ppc64le
tzdata-2017b-1.el7.noarch
phonon-4.6.0-10.el7.ppc64le
anaconda-tui-21.48.22.121-1.el7.centos.ppc64le
libmbim-utils-1.14.0-2.el7.ppc64le
gnutls-utils-3.3.26-9.el7.ppc64le
perl-Parse-CPAN-Meta-1.4404-5.el7.noarch
flite-1.3-22.el7.ppc64le
nfs4-acl-tools-0.3.3-15.el7.ppc64le
poppler-data-0.4.6-3.el7.noarch
gvfs-fuse-1.30.4-3.el7.ppc64le
gnome-software-3.22.7-1.el7.ppc64le
perl-ExtUtils-ParseXS-3.18-3.el7.noarch
libvirt-python-3.2.0-3.el7.ppc64le
perl-Module-Load-Conditional-0.54-3.el7.noarch
python-netifaces-0.10.4-3.el7.ppc64le
swig-2.0.10-5.el7.ppc64le
ipa-client-common-4.5.0-20.el7.centos.noarch
cheese-libs-3.22.1-1.el7.ppc64le
gnome-tweak-tool-3.22.0-1.el7.noarch
perl-ExtUtils-CBuilder-0.28.2.6-292.el7.noarch
libsoup-devel-2.56.0-3.el7.ppc64le
perl-IO-Zlib-1.10-292.el7.noarch
fros-1.0-2.el7.noarch
lohit-devanagari-fonts-2.5.3-4.el7.noarch
grub2-ppc64le-modules-2.02-0.64.el7.centos.noarch
libgdata-0.17.8-1.el7.ppc64le
evince-nautilus-3.22.1-5.el7.ppc64le
perl-ExtUtils-Embed-1.30-292.el7.noarch
dleyna-connector-dbus-0.2.0-2.el7.ppc64le
libiec61883-1.2.0-10.el7.ppc64le
python-lxml-3.2.1-4.el7.ppc64le
liberation-serif-fonts-1.07.2-15.el7.noarch
tigervnc-license-1.8.0-1.el7.noarch
gnome-packagekit-3.22.1-2.el7.ppc64le
hpijs-3.15.9-3.el7.ppc64le
libmodman-2.0.1-8.el7.ppc64le
ntp-4.2.6p5-25.el7.centos.2.ppc64le
gmp-devel-6.0.0-15.el7.ppc64le
pyxattr-0.5.1-5.el7.ppc64le
sil-abyssinica-fonts-1.200-6.el7.noarch
ncurses-libs-5.9-13.20130511.el7.ppc64le
gnome-dictionary-libs-3.20.0-1.el7.ppc64le
kdesdk-devel-4.10.5-6.el7.ppc64le
libreport-rhel-anaconda-bugzilla-2.1.11-38.el7.centos.ppc64le
libvirt-daemon-config-network-3.2.0-14.el7.ppc64le
boost-iostreams-1.53.0-27.el7.ppc64le
python-ply-3.4-11.el7.noarch
ucs-miscfixed-fonts-0.3-11.el7.noarch
info-5.1-4.el7.ppc64le
libXxf86misc-devel-1.0.3-7.1.el7.ppc64le
ibus-qt-1.3.2-4.el7.ppc64le
gnome-video-effects-0.4.3-1.el7.noarch
bridge-utils-1.5-9.el7.ppc64le
make-3.82-23.el7.ppc64le
pywbem-0.7.0-25.20130827svn625.el7.noarch
pnm2ppa-1.04-28.el7.ppc64le
chkconfig-1.7.4-1.el7.ppc64le
at-spi2-atk-devel-2.22.0-2.el7.ppc64le
freeglut-devel-2.8.1-3.el7.ppc64le
jbigkit-libs-2.0-11.el7.ppc64le
sssd-ipa-1.15.2-50.el7.ppc64le
openssl-libs-1.0.2k-8.el7.ppc64le
ldns-1.6.16-10.el7.ppc64le
rdate-1.4-25.el7.ppc64le
libdb-5.3.21-20.el7.ppc64le
evince-libs-3.22.1-5.el7.ppc64le
empathy-3.12.12-4.el7.ppc64le
rubygem-json-1.7.7-30.el7.ppc64le
dmraid-1.0.0.rc16-28.el7.ppc64le
libblkid-2.23.2-43.el7.ppc64le
logrotate-3.8.6-14.el7.ppc64le
iwl105-firmware-18.168.6.1-56.el7.noarch
grep-2.20-3.el7.ppc64le
xorg-x11-drv-synaptics-1.9.0-1.el7.ppc64le
iowatcher-1.0-6.el7.ppc64le
rubygem-net-http-persistent-2.8-5.el7.noarch
setroubleshoot-plugins-3.0.65-1.el7.noarch
atk-2.22.0-3.el7.ppc64le
libcacard-2.5.2-2.el7.ppc64le
iwl6050-firmware-41.28.5.1-56.el7.noarch
lcms2-2.6-3.el7.ppc64le
tigervnc-server-minimal-1.8.0-1.el7.ppc64le
gvfs-goa-1.30.4-3.el7.ppc64le
authconfig-6.2.8-30.el7.ppc64le
yum-plugin-fastestmirror-1.1.31-42.el7.noarch
dbus-python-1.1.1-9.el7.ppc64le
perl-Archive-Tar-1.92-2.el7.noarch
iwl5000-firmware-8.83.5.1_1-56.el7.noarch
libacl-2.2.51-12.el7.ppc64le
farstream-0.1.2-8.el7.ppc64le
ppc64-utils-0.14-16.el7.ppc64le
servicelog-1.1.14-3.el7.ppc64le
python2-ipaclient-4.5.0-20.el7.centos.noarch
libpeas-1.20.0-1.el7.ppc64le
perl-TermReadKey-2.30-20.el7.ppc64le
hdparm-9.43-5.el7.ppc64le
libicu-50.1.2-15.el7.ppc64le
polkit-qt-0.103.0-10.el7_0.ppc64le
gnome-weather-3.20.2-1.el7.noarch
libmspack-0.5-0.5.alpha.el7.ppc64le
libkkc-data-0.3.1-9.el7.ppc64le
hicolor-icon-theme-0.12-7.el7.noarch
perl-Newt-1.08-36.el7.ppc64le
libstdc++-devel-4.8.5-16.el7_4.1.ppc64le
libexif-0.6.21-6.el7.ppc64le
gtk3-devel-3.22.10-4.el7.ppc64le
gvfs-mtp-1.30.4-3.el7.ppc64le
ncompress-4.2.4.4-3.el7.ppc64le
libXcomposite-0.4.4-4.1.el7.ppc64le
python-decorator-3.4.0-3.el7.noarch
perl-Business-ISBN-Data-20120719.001-2.el7.noarch
gcc-gfortran-4.8.5-16.el7_4.1.ppc64le
cpio-2.11-24.el7.ppc64le
mesa-libGLU-9.0.0-4.el7.ppc64le
baobab-3.22.1-1.el7.ppc64le
device-mapper-libs-1.02.140-8.el7.ppc64le
libXtst-1.2.3-1.el7.ppc64le
ModemManager-glib-1.6.0-2.el7.ppc64le
perl-HTML-Parser-3.71-4.el7.ppc64le
libical-1.0.1-1.el7.ppc64le
xorg-x11-xinit-1.3.4-1.el7.ppc64le
gstreamer1-plugins-base-devel-1.10.4-1.el7.ppc64le
libdrm-2.4.74-1.el7.ppc64le
libXfixes-devel-5.0.3-1.el7.ppc64le
python-gssapi-1.2.0-3.el7.ppc64le
perl-Text-Unidecode-0.04-20.el7.noarch
hunspell-1.3.2-15.el7.ppc64le
kde-settings-19-23.5.el7.centos.noarch
perl-App-cpanminus-1.6922-2.el7.noarch
parted-3.1-28.el7.ppc64le
mesa-libGL-17.0.1-6.20170307.el7.ppc64le
elfutils-libelf-devel-0.168-8.el7.ppc64le
perl-Net-LibIDN-0.12-15.el7.ppc64le
apr-1.4.8-3.el7.ppc64le
kdepimlibs-4.10.5-4.el7.ppc64le
virt-top-1.0.8-23.el7.ppc64le
samba-client-libs-4.6.2-8.el7.ppc64le
gstreamer-plugins-base-0.10.36-10.el7.ppc64le
json-glib-devel-1.2.6-1.el7.ppc64le
perl-autodie-2.16-2.el7.noarch
tar-1.26-32.el7.ppc64le
ksysguard-libs-4.11.19-8.el7.ppc64le
rdma-core-devel-13-7.el7.ppc64le
accountsservice-0.6.45-2.el7.ppc64le
libxklavier-5.4-7.el7.ppc64le
libxml2-devel-2.9.1-6.el7_2.3.ppc64le
ghostscript-fonts-5.50-32.el7.noarch
libassuan-2.1.0-3.el7.ppc64le
libkipi-devel-4.10.5-3.el7.ppc64le
python-smbc-1.0.13-7.el7.ppc64le
initscripts-9.49.39-1.el7.ppc64le
qt3-3.3.8b-51.el7.ppc64le
yum-metadata-parser-1.1.4-10.el7.ppc64le
device-mapper-persistent-data-0.7.0-0.1.rc6.el7.ppc64le
adwaita-icon-theme-3.22.0-1.el7.noarch
kdepim-4.10.5-6.el7.ppc64le
postfix-2.10.1-6.el7.ppc64le
abrt-addon-pstoreoops-2.1.11-48.el7.centos.ppc64le
freerdp-libs-1.0.2-10.el7.ppc64le
langtable-python-0.0.31-3.el7.noarch
tcp_wrappers-7.6-77.el7.ppc64le
lm_sensors-libs-3.4.0-4.20160601gitf9185e5.el7.ppc64le
kde-style-oxygen-4.11.19-8.el7.ppc64le
powertop-2.3-12.el7.ppc64le
wpa_supplicant-2.6-5.el7.ppc64le
gtk3-3.22.10-4.el7.ppc64le
boost-python-1.53.0-27.el7.ppc64le
keyutils-libs-devel-1.5.8-3.el7.ppc64le
libdvdread-5.0.3-3.el7.ppc64le
im-chooser-common-1.6.4-4.el7.ppc64le
aic94xx-firmware-30-6.el7.noarch
media-player-info-17-4.el7.noarch
compat-gnome-desktop314-3.14.2-1.el7.ppc64le
harfbuzz-1.3.2-1.el7.ppc64le
libgcrypt-devel-1.5.3-14.el7.ppc64le
groff-base-1.22.2-8.el7.ppc64le
sane-backends-1.0.24-9.el7.ppc64le
setuptool-1.19.11-8.el7.ppc64le
ebtables-2.0.10-15.el7.ppc64le
libchamplain-0.12.15-1.el7.ppc64le
boost-math-1.53.0-27.el7.ppc64le
libuser-0.60-7.el7_1.ppc64le
boost-date-time-1.53.0-27.el7.ppc64le
espeak-1.47.11-4.el7.ppc64le
tbb-devel-4.1-9.20130314.el7.ppc64le
grub2-tools-minimal-2.02-0.64.el7.centos.ppc64le
gjs-1.46.0-1.el7.ppc64le
libsss_autofs-1.15.2-50.el7.ppc64le
deltarpm-3.6-3.el7.ppc64le
libnl-1.1.4-3.el7.ppc64le
libgpod-0.8.2-12.el7.ppc64le
postgresql-devel-9.2.21-1.el7.ppc64le
libibcm-13-7.el7.ppc64le
abrt-gui-libs-2.1.11-48.el7.centos.ppc64le
libxkbcommon-0.7.1-1.el7.ppc64le
passwd-0.79-4.el7.ppc64le
lsvpd-1.7.8-1.el7.ppc64le
fprintd-0.5.0-4.0.el7_0.ppc64le
hunspell-en-0.20121024-6.el7.noarch
qca-ossl-2.0.0-0.19.beta3.el7.ppc64le
libdmapsharing-2.9.37-1.el7.ppc64le
ortp-0.20.0-10.el7.ppc64le
python-pycurl-7.19.0-19.el7.ppc64le
perl-Pod-Escapes-1.04-292.el7.noarch
pcp-3.11.8-7.el7.ppc64le
libblkid-devel-2.23.2-43.el7.ppc64le
dracut-network-033-502.el7.ppc64le
pyatspi-2.20.3-1.el7.noarch
systemtap-sdt-devel-3.1-3.el7.ppc64le
check-0.9.9-5.el7.ppc64le
perl-threads-shared-1.43-6.el7.ppc64le
gnome-shell-extension-common-3.22.2-10.el7.noarch
gnome-icon-theme-symbolic-3.12.0-2.el7.noarch
abrt-cli-2.1.11-48.el7.centos.ppc64le
festival-speechtools-libs-1.2.96-28.el7.ppc64le
python-slip-dbus-0.4.0-2.el7.noarch
mesa-private-llvm-3.9.1-3.el7.ppc64le
perl-Time-Local-1.2300-2.el7.noarch
yelp-3.22.0-1.el7.ppc64le
fuse-devel-2.9.2-8.el7.ppc64le
dnsmasq-2.76-2.el7.ppc64le
festvox-slt-arctic-hts-0.20061229-28.el7.noarch
libtasn1-devel-4.10-1.el7.ppc64le
libgudev1-219-42.el7.ppc64le
perl-version-0.99.07-2.el7.ppc64le
libvirt-daemon-driver-qemu-3.2.0-14.el7.ppc64le
ps_mem-3.1-7.el7.noarch
rtkit-0.11-10.el7.ppc64le
abrt-gui-2.1.11-48.el7.centos.ppc64le
nettle-devel-2.7.1-8.el7.ppc64le
perl-ExtUtils-Manifest-1.61-244.el7.noarch
libreswan-3.20-3.el7.ppc64le
python-pyudev-0.15-9.el7.noarch
appstream-data-7-20170301.el7.noarch
powerpc-utils-1.3.3-4.el7.ppc64le
setup-2.8.71-7.el7.noarch
enscript-1.6.6-6.el7.ppc64le
libgexiv2-0.10.4-2.el7.ppc64le
perl-Digest-SHA-5.85-4.el7.ppc64le
upower-0.99.4-2.el7.ppc64le
dhcp-libs-4.2.5-58.el7.centos.ppc64le
kbd-1.15.5-13.el7.ppc64le
phonon-backend-gstreamer-4.6.3-3.el7.ppc64le
dejavu-fonts-common-2.33-6.el7.noarch
libaio-devel-0.3.109-13.el7.ppc64le
grubby-8.28-23.el7.ppc64le
perl-CPAN-Meta-2.120921-5.el7.noarch
libmusicbrainz5-5.0.1-9.el7.ppc64le
liberation-mono-fonts-1.07.2-15.el7.noarch
fcoe-utils-1.0.32-1.el7.ppc64le
gvfs-afc-1.30.4-3.el7.ppc64le
m17n-db-1.6.4-3.el7.noarch
time-1.7-45.el7.ppc64le
python-configobj-4.7.2-7.el7.noarch
perl-Log-Message-0.08-3.el7.noarch
glib-networking-2.50.0-1.el7.ppc64le
gnome-classic-session-3.22.2-10.el7.noarch
libglade2-2.6.4-11.el7.ppc64le
langtable-data-0.0.31-3.el7.noarch
dejavu-serif-fonts-2.33-6.el7.noarch
python-requests-2.6.0-1.el7_1.noarch
perl-HTML-Tagset-3.20-15.el7.noarch
gssdp-1.0.1-1.el7.ppc64le
perl-CPANPLUS-Dist-Build-0.70-3.el7.noarch
brasero-nautilus-3.12.1-2.el7.ppc64le
evolution-data-server-3.22.7-6.el7.ppc64le
khmeros-fonts-common-5.0-17.el7.noarch
dejavu-sans-fonts-2.33-6.el7.noarch
python-kmod-0.9-4.el7.ppc64le
lzop-1.03-10.el7.ppc64le
telepathy-salut-0.8.1-6.el7.ppc64le
tbb-4.1-9.20130314.el7.ppc64le
kdegraphics-devel-4.10.5-3.el7.noarch
libcryptui-3.12.2-1.el7.ppc64le
ncurses-base-5.9-13.20130511.el7.noarch
lohit-nepali-fonts-2.5.3-2.el7.noarch
python-configshell-1.1.fb23-3.el7.noarch
acl-2.2.51-12.el7.ppc64le
python-rtslib-2.1.fb63-2.el7.noarch
libreport-plugin-rhtsupport-2.1.11-38.el7.centos.ppc64le
imsettings-qt-1.6.3-9.el7.ppc64le
webkitgtk3-2.4.11-2.el7.ppc64le
libsepol-2.5-6.el7.ppc64le
smc-meera-fonts-6.0-7.el7.noarch
python-mako-0.8.1-2.el7.noarch
pinentry-0.8.1-17.el7.ppc64le
alsa-tools-firmware-1.1.0-1.el7.ppc64le
libgdither-0.6-8.el7.ppc64le
ibus-libpinyin-1.6.91-4.el7.ppc64le
libXp-devel-1.0.2-2.1.el7.ppc64le
nspr-4.13.1-1.0.el7_3.ppc64le
cscope-15.8-10.el7.ppc64le
m2crypto-0.21.1-17.el7.ppc64le
libatomic-4.8.5-16.el7.ppc64le
opencc-0.4.3-3.el7.ppc64le
sbc-1.0-5.el7.ppc64le
SDL-devel-1.2.15-14.el7.ppc64le
vorbis-tools-1.4.0-12.el7.ppc64le
bzip2-libs-1.0.6-13.el7.ppc64le
google-crosextra-carlito-fonts-1.103-0.2.20130920.el7.noarch
nmap-ncat-6.40-7.el7.ppc64le
krb5-libs-1.15.1-8.el7.ppc64le
sssd-krb5-1.15.2-50.el7.ppc64le
cups-filters-libs-1.0.35-22.el7.ppc64le
virt-manager-1.4.1-7.el7.noarch
evince-3.22.1-5.el7.ppc64le
readline-6.2-10.el7.ppc64le
ctags-5.8-13.el7.ppc64le
sound-theme-freedesktop-0.8-3.el7.noarch
ruby-libs-2.0.0.648-30.el7.ppc64le
pth-2.0.7-23.el7.ppc64le
rubygems-2.0.14.1-30.el7.noarch
gnome-dictionary-3.20.0-1.el7.ppc64le
xorg-x11-drv-evdev-2.10.5-2.1.el7.ppc64le
audit-libs-2.7.6-3.el7.ppc64le
iwl135-firmware-18.168.6.1-56.el7.noarch
python-nss-0.16.0-3.el7.ppc64le
json-glib-1.2.6-1.el7.ppc64le
flatpak-libs-0.8.7-1.el7.ppc64le
libutempter-1.1.6-4.el7.ppc64le
ekiga-4.0.1-7.el7.ppc64le
easymock2-2.5.2-12.el7.noarch
keyutils-libs-1.5.8-3.el7.ppc64le
iwl1000-firmware-39.31.5.1-56.el7.noarch
teamd-1.25-5.el7.ppc64le
telepathy-glib-0.24.0-1.el7.ppc64le
PackageKit-yum-1.1.5-1.el7.centos.ppc64le
virt-what-1.13-10.el7.ppc64le
ppc64-diag-2.7.3-3.el7.ppc64le
libpurple-2.10.11-5.el7.ppc64le
libffi-3.0.13-18.el7.ppc64le
iwl2000-firmware-18.168.6.1-56.el7.noarch
perl-YAML-0.84-5.el7.noarch
libxml2-python-2.9.1-6.el7_2.3.ppc64le
lsscsi-0.27-6.el7.ppc64le
systemtap-client-3.1-3.el7.ppc64le
virt-viewer-5.0-7.el7.ppc64le
dbusmenu-qt-0.9.2-7.el7.ppc64le
libtar-1.2.11-29.el7.ppc64le
ccache-3.3.4-1.el7.ppc64le
perl-DBD-SQLite-1.39-3.el7.ppc64le
gnome-icon-theme-3.12.0-1.el7.noarch
gdk-pixbuf2-2.36.5-1.el7.ppc64le
libpath_utils-0.2.1-27.el7.ppc64le
gvfs-archive-1.30.4-3.el7.ppc64le
gnome-online-accounts-devel-3.22.5-1.el7.ppc64le
yajl-2.0.4-4.el7.ppc64le
gcc-4.8.5-16.el7_4.1.ppc64le
perl-Pod-Coverage-0.23-3.el7.noarch
libselinux-python-2.5-11.el7.ppc64le
libX11-devel-1.6.5-1.el7.ppc64le
qrencode-libs-3.4.1-3.el7.ppc64le
gnome-system-log-3.9.90-3.el7.ppc64le
mesa-libGLU-devel-9.0.0-4.el7.ppc64le
boost-system-1.53.0-27.el7.ppc64le
perl-HTTP-Message-6.06-6.el7.noarch
cracklib-2.9.0-11.el7.ppc64le
libXcursor-1.1.14-8.el7.ppc64le
dbus-1.6.12-17.el7.ppc64le
libnotify-devel-0.7.7-1.el7.ppc64le
ibus-gtk3-1.5.3-13.el7.ppc64le
libv4l-0.9.5-4.el7.ppc64le
perl-Time-Piece-1.20.1-292.el7.ppc64le
cracklib-dicts-2.9.0-11.el7.ppc64le
startup-notification-0.12-8.el7.ppc64le
dconf-0.26.0-2.el7.ppc64le
net-snmp-devel-5.7.2-28.el7.ppc64le
kate-part-4.10.5-4.el7.ppc64le
orc-0.4.26-1.el7.ppc64le
kernel-devel-3.10.0-693.el7.ppc64le
avahi-gobject-0.6.31-17.el7.ppc64le
cairo-gobject-1.14.8-2.el7.ppc64le
httpd-2.4.6-67.el7.centos.ppc64le
subversion-1.7.14-10.el7.ppc64le
kdepimlibs-akonadi-4.10.5-4.el7.ppc64le
gdbm-1.10-8.el7.ppc64le
perl-File-CheckTree-4.42-3.el7.noarch
atk-devel-2.22.0-3.el7.ppc64le
java-1.8.0-openjdk-devel-1.8.0.131-11.b12.el7.ppc64le
abrt-dbus-2.1.11-48.el7.centos.ppc64le
qt-mysql-4.8.5-13.el7.ppc64le
libkdcraw-4.10.5-4.el7.ppc64le
libaio-0.3.109-13.el7.ppc64le
urw-fonts-2.4-16.el7.noarch
libgee06-0.6.8-3.el7.ppc64le
libXrandr-devel-1.5.1-2.el7.ppc64le
cronie-anacron-1.4.11-17.el7.ppc64le
mlocate-0.26-6.el7.ppc64le
kdesdk-okteta-devel-4.10.5-6.el7.ppc64le
iso-codes-3.46-2.el7.noarch
e2fsprogs-1.42.9-10.el7.ppc64le
at-spi2-atk-2.22.0-2.el7.ppc64le
libstoragemgmt-python-clibs-1.4.0-3.el7.ppc64le
PackageKit-command-not-found-1.1.5-1.el7.centos.ppc64le
kdenetwork-kopete-devel-4.10.5-8.el7_0.ppc64le
libmnl-1.0.3-7.el7.ppc64le
tcp_wrappers-devel-7.6-77.el7.ppc64le
python-dns-1.12.0-4.20150617git465785f.el7.noarch
libXinerama-devel-1.1.3-2.1.el7.ppc64le
libibverbs-13-7.el7.ppc64le
net-tools-2.0-0.22.20131004git.el7.ppc64le
kde-workspace-libs-4.11.19-8.el7.ppc64le
libwebp-0.3.0-7.el7.ppc64le
libattr-devel-2.4.46-12.el7.ppc64le
libkadm5-1.15.1-8.el7.ppc64le
gcr-3.20.0-1.el7.ppc64le
colord-1.3.4-1.el7.ppc64le
rsyslog-8.24.0-12.el7.ppc64le
im-chooser-1.6.4-4.el7.ppc64le
boost-filesystem-1.53.0-27.el7.ppc64le
libgpg-error-devel-1.12-3.el7.ppc64le
harfbuzz-icu-1.3.2-1.el7.ppc64le
libpeas-gtk-1.20.0-1.el7.ppc64le
abrt-addon-python-2.1.11-48.el7.centos.ppc64le
selinux-policy-targeted-3.13.1-166.el7.noarch
libksane-4.10.5-4.el7.ppc64le
m4-1.4.16-10.el7.ppc64le
xmlrpc-c-client-1.32.5-1905.svn2451.el7.ppc64le
sysvinit-tools-2.88-14.dsf.el7.ppc64le
libnma-1.8.0-3.el7.ppc64le
os-prober-1.58-9.el7.ppc64le
libproxy-mozjs-0.4.11-10.el7.ppc64le
speech-dispatcher-0.7.1-15.el7.ppc64le
boost-signals-1.53.0-27.el7.ppc64le
python-ldap-2.4.15-2.el7.ppc64le
libvpx-1.3.0-5.el7_0.ppc64le
nm-connection-editor-1.8.0-3.el7.ppc64le
NetworkManager-team-1.8.0-9.el7.ppc64le
perf-3.10.0-693.el7.ppc64le
libgsf-1.14.26-7.el7.ppc64le
libpfm-4.7.0-4.el7.ppc64le
postgresql-9.2.21-1.el7.ppc64le
ethtool-4.8-1.el7.ppc64le
xorg-x11-server-utils-7.7-20.el7.ppc64le
attica-0.4.2-1.el7.ppc64le
xfsdump-3.1.4-1.el7.ppc64le
firewalld-filesystem-0.4.4.4-6.el7.noarch
libXfont2-2.0.1-2.el7.ppc64le
net-snmp-agent-libs-5.7.2-28.el7.ppc64le
tcl-devel-8.5.13-8.el7.ppc64le
libgxps-0.2.5-1.el7.ppc64le
cyrus-sasl-devel-2.1.26-21.el7.ppc64le
hmaccalc-0.9.13-4.el7.ppc64le
libwacom-data-0.24-1.el7.noarch
perl-Pod-Usage-1.63-3.el7.noarch
python-yubico-1.2.3-1.el7.noarch
libXxf86vm-devel-1.1.4-1.el7.ppc64le
abrt-tui-2.1.11-48.el7.centos.ppc64le
pinfo-0.6.10-9.el7.ppc64le
gnome-shell-extension-user-theme-3.22.2-10.el7.noarch
perl-File-Path-2.09-2.el7.noarch
xorg-x11-fonts-Type1-7.5-9.el7.noarch
python-firewall-0.4.4.4-6.el7.noarch
libXres-1.0.7-2.1.el7.ppc64le
libcgroup-tools-0.41-13.el7.ppc64le
libnl-devel-1.1.4-3.el7.ppc64le
gnome-user-docs-3.22.0-1.el7.noarch
perl-Pod-Simple-3.28-4.el7.noarch
systemd-libs-219-42.el7.ppc64le
ncurses-devel-5.9-13.20130511.el7.ppc64le
mesa-libEGL-devel-17.0.1-6.20170307.el7.ppc64le
audit-2.7.6-3.el7.ppc64le
iotop-0.6-2.el7.noarch
libvirt-daemon-driver-storage-logical-3.2.0-14.el7.ppc64le
perl-Module-CoreList-2.76.02-292.el7.noarch
libmbim-1.14.0-2.el7.ppc64le
xdg-desktop-portal-0.5-2.el7.ppc64le
perl-Module-Load-0.24-3.el7.noarch
caribou-gtk3-module-0.4.21-1.el7.ppc64le
sqlite-devel-3.7.17-8.el7.ppc64le
centos-indexhtml-7-9.el7.centos.noarch
elfutils-0.168-8.el7.ppc64le
centos-release-7-4.1708.el7.centos.ppc64le
trousers-0.3.14-2.el7.ppc64le
perl-Thread-Queue-3.02-2.el7.noarch
python-meh-gui-0.25.2-1.el7.noarch
gom-0.3.2-1.el7.ppc64le
lldpad-1.0.1-3.git036e314.el7.ppc64le
libgusb-0.2.9-1.el7.ppc64le
liberation-fonts-common-1.07.2-15.el7.noarch
libimobiledevice-1.2.0-1.el7.ppc64le
perl-Module-Pluggable-4.8-3.el7.noarch
ghostscript-cups-9.07-28.el7.ppc64le
osinfo-db-tools-1.1.0-1.el7.ppc64le
kbd-misc-1.15.5-13.el7.noarch
dhcp-common-4.2.5-58.el7.centos.ppc64le
control-center-filesystem-3.22.2-5.el7.ppc64le
libvirt-glib-1.0.0-1.el7.ppc64le
perl-CPAN-Meta-Requirements-2.122-7.el7.noarch
PyQt4-4.10.1-13.el7.ppc64le
btrfs-progs-4.9.1-1.el7.ppc64le
anaconda-gui-21.48.22.121-1.el7.centos.ppc64le
libatasmart-0.19-6.el7.ppc64le
shared-desktop-ontologies-0.11.0-2.el7.noarch
libvirt-daemon-config-nwfilter-3.2.0-14.el7.ppc64le
autoconf-2.69-11.el7.noarch
gnome-terminal-3.22.1-2.el7.ppc64le
python-cups-1.9.63-6.el7.ppc64le
intltool-0.50.2-7.el7.noarch
glibc-headers-2.17-196.el7.ppc64le
kdesdk-common-4.10.5-6.el7.noarch
libvirt-daemon-driver-secret-3.2.0-14.el7.ppc64le
perl-Locale-Maketext-Simple-0.21-292.el7.noarch
gnome-keyring-3.20.0-3.el7.ppc64le
python-sss-murmur-1.15.2-50.el7.ppc64le
vim-enhanced-7.4.160-2.el7.ppc64le
perl-ExtUtils-MakeMaker-6.68-3.el7.noarch
emacs-filesystem-24.3-19.el7_3.noarch
libvncserver-0.9.9-9.el7_0.1.ppc64le
perl-Object-Accessor-0.42-292.el7.noarch
gnome-desktop3-3.22.2-2.el7.ppc64le
python-backports-1.0-8.el7.ppc64le
evolution-help-3.22.6-10.el7.noarch
systemtap-devel-3.1-3.el7.ppc64le
langtable-0.0.31-3.el7.noarch
geocode-glib-3.20.1-1.el7.ppc64le
perl-Compress-Raw-Bzip2-2.061-3.el7.ppc64le
pygtk2-libglade-2.24.0-9.el7.ppc64le
python-urllib3-1.10.2-3.el7.noarch
orca-3.6.3-4.el7.ppc64le
perl-File-Fetch-0.42-2.el7.noarch
latencytop-common-0.5-13.el7.ppc64le
geoclue2-libs-2.4.5-1.el7.ppc64le
perl-Module-Loaded-0.08-292.el7.noarch
webkitgtk4-2.14.7-2.el7.ppc64le
python-paste-1.7.5.1-9.20111221hg1498.el7.noarch
totem-nautilus-3.22.1-1.el7.ppc64le
libtool-2.4.2-22.el7_3.ppc64le
smc-fonts-common-6.0-7.el7.noarch
libnice-0.1.3-4.el7.ppc64le
libdvdnav-5.0.3-1.el7.ppc64le
folks-0.11.3-1.el7.ppc64le
python-ipaddr-2.1.11-1.el7.noarch
xorg-x11-utils-7.5-22.el7.ppc64le
oxygen-icon-theme-4.10.5-2.el7.noarch
libkkc-common-0.3.1-9.el7.noarch
libgovirt-0.3.3-5.el7.ppc64le
boost-timer-1.53.0-27.el7.ppc64le
gnome-packagekit-common-3.22.1-2.el7.ppc64le
javapackages-tools-3.4.1-11.el7.noarch
sane-backends-devel-1.0.24-9.el7.ppc64le
konkretcmpi-0.9.1-5.el7.ppc64le
perl-srpm-macros-1-8.el7.noarch
chrony-3.1-2.el7.centos.ppc64le
fuse-2.9.2-8.el7.ppc64le
evolution-3.22.6-10.el7.ppc64le
python-urwid-1.1.1-3.el7.ppc64le
shotwell-0.24.5-1.el7.ppc64le
libreport-web-2.1.11-38.el7.centos.ppc64le
glibc-2.17-196.el7.ppc64le
usb_modeswitch-data-20160612-2.el7.noarch
patch-2.7.1-8.el7.ppc64le
file-roller-3.22.3-1.el7.ppc64le
python-netaddr-0.7.5-7.el7.noarch
ibus-table-chinese-1.4.6-3.el7.noarch
libreport-plugin-reportuploader-2.1.11-38.el7.centos.ppc64le
pcre-8.32-17.el7.ppc64le
libvirt-daemon-driver-network-3.2.0-14.el7.ppc64le
cyrus-sasl-plain-2.1.26-21.el7.ppc64le
glade-libs-3.20.0-1.el7.ppc64le
python-markupsafe-0.11-10.el7.ppc64le
kdenetwork-devel-4.10.5-8.el7_0.noarch
libreport-plugin-ureport-2.1.11-38.el7.centos.ppc64le
dbus-libs-1.6.12-17.el7.ppc64le
alsa-firmware-1.0.28-2.el7.noarch
mozjs17-17.0.0-19.el7.ppc64le
avahi-ui-gtk3-0.6.31-17.el7.ppc64le
python-cffi-1.6.0-5.el7.ppc64le
xdg-user-dirs-gtk-0.10-4.el7.ppc64le
gavl-1.4.0-4.el7.ppc64le
libjpeg-turbo-1.2.90-5.el7.ppc64le
device-mapper-multipath-0.4.9-111.el7.ppc64le
libcdio-0.92-1.el7.ppc64le
pulseaudio-module-bluetooth-10.0-3.el7.ppc64le
pytalloc-2.1.9-1.el7.ppc64le
ibus-sayura-1.3.2-3.el7.ppc64le
checkpolicy-2.5-4.el7.ppc64le
libICE-1.0.9-9.el7.ppc64le
libvirt-daemon-driver-interface-3.2.0-14.el7.ppc64le
libunistring-0.9.3-9.el7.ppc64le
libXScrnSaver-devel-1.2.2-6.1.el7.ppc64le
openlmi-python-base-0.5.0-4.el7.noarch
PyQt4-devel-4.10.1-13.el7.ppc64le
libndp-1.2-7.el7.ppc64le
libxml2-2.9.1-6.el7_2.3.ppc64le
sssd-krb5-common-1.15.2-50.el7.ppc64le
ncurses-5.9-13.20130511.el7.ppc64le
icedax-1.1.11-23.el7.ppc64le
libmsn-4.2.1-7.el7.ppc64le
evolution-data-server-devel-3.22.7-6.el7.ppc64le
poppler-0.26.5-16.el7.ppc64le
sed-4.2.2-5.el7.ppc64le
sssd-ldap-1.15.2-50.el7.ppc64le
fontconfig-2.10.95-11.el7.ppc64le
pinentry-qt-0.8.1-17.el7.ppc64le
cyrus-sasl-scram-2.1.26-21.el7.ppc64le
paps-0.6.8-28.el7.1.ppc64le
libyaml-0.1.4-11.el7_0.ppc64le
libgpg-error-1.12-3.el7.ppc64le
sgpio-1.2.0.10-13.el7.ppc64le
alsa-lib-1.1.3-3.el7.ppc64le
gutenprint-5.2.9-18.el7.ppc64le
openslp-2.0.0-6.el7.ppc64le
ruby-irb-2.0.0.648-30.el7.noarch
libgcrypt-1.5.3-14.el7.ppc64le
python-blivet-0.61.15.65-1.el7.noarch
gzip-1.5-9.el7.ppc64le
xorg-x11-drv-void-1.4.1-2.el7.ppc64le
nss-pem-1.0.3-4.el7.ppc64le
rubygem-rdoc-4.0.0-30.el7.noarch
libcap-ng-0.7.5-4.el7.ppc64le
rpm-build-libs-4.11.3-25.el7.ppc64le
shared-mime-info-1.8-3.el7.ppc64le
xorg-x11-drv-v4l-0.2.0-47.el7.ppc64le
nss-tools-3.28.4-8.el7.ppc64le
libsemanage-2.5-8.el7.ppc64le
libxcb-1.12-1.el7.ppc64le
flatpak-0.8.7-1.el7.ppc64le
gstreamer1-1.10.4-2.el7.ppc64le
xorg-x11-drv-nouveau-1.0.13-3.el7.ppc64le
sgml-common-0.6.3-39.el7.noarch
util-linux-2.23.2-43.el7.ppc64le
libtdb-1.3.12-2.el7.ppc64le
rpm-devel-4.11.3-25.el7.ppc64le
gobject-introspection-1.50.0-1.el7.ppc64le
qdox-1.12.1-10.el7.noarch
libteam-1.25-5.el7.ppc64le
openssh-clients-7.4p1-11.el7.ppc64le
libattr-2.4.46-12.el7.ppc64le
python-meh-0.25.2-1.el7.noarch
avahi-glib-0.6.31-17.el7.ppc64le
rhino-1.7R5-1.el7.noarch
perl-Pod-Checker-1.60-2.el7.noarch
rarian-0.8.1-11.el7.ppc64le
gmp-6.0.0-15.el7.ppc64le
createrepo-0.9.9-28.el7.noarch
python-gobject-base-3.22.0-1.el7.ppc64le
telepathy-haze-0.8.0-1.el7.ppc64le
perl-Version-Requirements-0.101022-244.el7.noarch
tog-pegasus-2.14.1-5.el7.ppc64le
lua-5.1.4-15.el7.ppc64le
libburn-1.2.8-4.el7.ppc64le
openssl-1.0.2k-8.el7.ppc64le
dleyna-server-0.5.0-1.el7.ppc64le
perl-IO-HTML-1.00-2.el7.noarch
libsemanage-python-2.5-8.el7.ppc64le
libidn-1.28-4.el7.ppc64le
nss-devel-3.28.4-8.el7.ppc64le
net-snmp-libs-5.7.2-28.el7.ppc64le
paps-libs-0.6.8-28.el7.1.ppc64le
perl-DBIx-Simple-1.35-7.el7.noarch
lzo-minilzo-2.06-8.el7.ppc64le
libref_array-0.1.5-27.el7.ppc64le
libX11-1.6.5-1.el7.ppc64le
xdg-utils-1.1.0-0.17.20120809git.el7.noarch
harfbuzz-devel-1.3.2-1.el7.ppc64le
perl-CGI-3.63-4.el7.noarch
libini_config-1.3.0-27.el7.ppc64le
xmlrpc-c-1.32.5-1905.svn2451.el7.ppc64le
libXfixes-5.0.3-1.el7.ppc64le
glibmm24-2.50.0-1.el7.ppc64le
webkitgtk4-devel-2.14.7-2.el7.ppc64le
perl-Devel-Symdump-2.10-2.el7.noarch
libpipeline-1.2.3-3.el7.ppc64le
mpfr-3.1.1-4.el7.ppc64le
libXrandr-1.5.1-2.el7.ppc64le
cyrus-sasl-gssapi-2.1.26-21.el7.ppc64le
gtk2-devel-2.24.31-1.el7.ppc64le
perl-URI-1.60-9.el7.noarch
kpartx-0.4.9-111.el7.ppc64le
file-libs-5.11-33.el7.ppc64le
libXext-devel-1.3.3-3.el7.ppc64le
libSM-devel-1.2.2-2.el7.ppc64le
qt-devel-4.8.5-13.el7.ppc64le
perl-HTTP-Date-6.02-8.el7.noarch
dracut-033-502.el7.ppc64le
libtool-ltdl-2.4.2-22.el7_3.ppc64le
libcanberra-0.30-5.el7.ppc64le
python-enum34-1.0.4-1.el7.noarch
libxkbfile-devel-1.0.9-3.el7.ppc64le
perl-HTTP-Cookies-6.01-5.el7.noarch
polkit-0.112-12.el7_3.ppc64le
libtheora-1.1.1-8.el7.ppc64le
libXpm-3.5.12-1.el7.ppc64le
libevent-2.0.21-4.el7.ppc64le
ibus-gtk2-1.5.3-13.el7.ppc64le
kdelibs-common-4.14.8-6.el7_3.ppc64le
systemd-sysv-219-42.el7.ppc64le
diffutils-3.3-4.el7.ppc64le
libXv-1.0.11-1.el7.ppc64le
pam-1.1.8-18.el7.ppc64le
imsettings-gsettings-1.6.3-9.el7.ppc64le
perl-YAML-Tiny-1.51-6.el7.noarch
GConf2-3.2.6-8.el7.ppc64le
libtasn1-4.10-1.el7.ppc64le
libxkbfile-1.0.9-3.el7.ppc64le
gettext-libs-0.19.8.1-2.el7.ppc64le
kdelibs-ktexteditor-4.14.8-6.el7_3.ppc64le
perl-Env-1.04-2.el7.noarch
libpciaccess-0.13.4-3.el7_3.ppc64le
nss-softokn-3.28.3-6.el7.ppc64le
pango-1.40.4-1.el7.ppc64le
telepathy-logger-0.8.0-5.el7.ppc64le
nepomuk-core-4.10.5-5.el7.ppc64le
perl-Net-HTTP-6.06-2.el7.noarch
samba-common-4.6.2-8.el7.noarch
libsigc++20-2.10.0-1.el7.ppc64le
cogl-1.22.2-1.el7.ppc64le
pcre-devel-8.32-17.el7.ppc64le
kdenetwork-kopete-libs-4.10.5-8.el7_0.ppc64le
icoutils-0.31.3-1.el7_3.ppc64le
pyparted-3.9-13.el7.ppc64le
apr-util-1.5.2-6.el7.ppc64le
giflib-4.1.6-9.el7.ppc64le
expat-devel-2.1.0-10.el7_3.ppc64le
kdesdk-okteta-4.10.5-6.el7.ppc64le
papi-5.2.0-23.el7.ppc64le
abrt-python-2.1.11-48.el7.centos.ppc64le
lzo-2.06-8.el7.ppc64le
java-1.8.0-openjdk-1.8.0.131-11.b12.el7.ppc64le
libffi-devel-3.0.13-18.el7.ppc64le
kwin-libs-4.11.19-8.el7.ppc64le
xorg-x11-font-utils-7.5-20.el7.ppc64le
iscsi-initiator-utils-iscsiuio-6.2.0.874-4.el7.ppc64le
file-5.11-33.el7.ppc64le
libXft-devel-2.3.2-2.el7.ppc64le
libipa_hbac-1.15.2-50.el7.ppc64le
kwin-gles-libs-4.11.19-8.el7.ppc64le
libsamplerate-0.1.8-6.el7.ppc64le
cronie-1.4.11-17.el7.ppc64le
xml-common-0.6.3-39.el7.noarch
ghostscript-9.07-28.el7.ppc64le
libpinyin-data-0.9.93-4.el7.ppc64le
kde-runtime-libs-4.10.5-8.el7.ppc64le
ipset-6.29-1.el7.ppc64le
plymouth-0.8.9-0.28.20140113.el7.centos.ppc64le
taglib-1.8-7.20130218git.el7.ppc64le
at-spi2-core-2.22.0-1.el7.ppc64le
xfsprogs-4.5.0-12.el7.ppc64le
kdepim-runtime-4.10.5-3.el7.ppc64le
libusbmuxd-1.0.10-5.el7.ppc64le
libstoragemgmt-python-1.4.0-3.el7.noarch
libseccomp-2.3.1-3.el7.ppc64le
gstreamer1-plugins-good-1.10.4-2.el7.ppc64le
pyusb-1.0.0-0.11.b1.el7.noarch
nepomuk-core-devel-4.10.5-5.el7.ppc64le
libofa-0.9.3-24.el7.ppc64le
device-mapper-event-1.02.140-8.el7.ppc64le
librtas-2.0.1-1.el7.ppc64le
libXcomposite-devel-0.4.4-4.1.el7.ppc64le
audit-libs-python-2.7.6-3.el7.ppc64le
okular-libs-4.10.5-4.el7.ppc64le
gdisk-0.8.6-5.el7.ppc64le
libibumad-13-7.el7.ppc64le
libsndfile-1.0.25-10.el7.ppc64le
libXxf86misc-1.0.3-7.1.el7.ppc64le
pyparsing-1.5.6-9.el7.noarch
kdesdk-kmtrace-libs-4.10.5-6.el7.ppc64le
attr-2.4.46-12.el7.ppc64le
rpcbind-0.2.0-42.el7.ppc64le
slang-2.2.4-11.el7.ppc64le
gtk2-2.24.31-1.el7.ppc64le
libssh2-1.4.3-10.el7_2.1.ppc64le
kdesdk-kompare-4.10.5-6.el7.ppc64le
openssl-devel-1.0.2k-8.el7.ppc64le
bluez-5.44-2.el7.ppc64le
boost-thread-1.53.0-27.el7.ppc64le
clutter-gtk-1.8.2-1.el7.ppc64le
soundtouch-1.4.0-9.el7.ppc64le
ibus-table-1.5.0-5.el7.noarch
setools-libs-3.3.8-1.1.el7.ppc64le
ppp-2.4.5-33.el7.ppc64le
libvpd-2.2.5-1.el7.ppc64le
clutter-gst3-3.0.22-1.el7.ppc64le
boost-test-1.53.0-27.el7.ppc64le
libgphoto2-2.5.2-5.el7.ppc64le
libcurl-7.29.0-42.el7.ppc64le
libmtp-1.1.6-5.el7.ppc64le
unzip-6.0-16.el7.ppc64le
vte291-0.46.2-1.el7.ppc64le
boost-random-1.53.0-27.el7.ppc64le
hplip-libs-3.15.9-3.el7.ppc64le
openldap-2.4.44-5.el7.ppc64le
rsync-3.0.9-18.el7.ppc64le
psmisc-22.20-15.el7.ppc64le
compat-cheese314-3.14.2-1.el7.ppc64le
dosfstools-3.0.20-9.el7.ppc64le
sane-backends-drivers-cameras-1.0.24-9.el7.ppc64le
kde-filesystem-4-47.el7.ppc64le
cryptsetup-1.7.4-3.el7.ppc64le
boost-program-options-1.53.0-27.el7.ppc64le
libgnomekbd-3.22.0.1-1.el7.ppc64le
libsrtp-1.4.4-10.20101004cvs.el7.ppc64le
speech-dispatcher-python-0.7.1-15.el7.ppc64le
raptor2-2.0.9-3.el7.ppc64le
grub2-tools-2.02-0.64.el7.centos.ppc64le
libiodbc-3.52.7-7.el7.ppc64le
gtk-vnc2-0.7.0-2.el7.ppc64le
libdv-1.0.0-17.el7.ppc64le
libXxf86dga-1.1.4-2.1.el7.ppc64le
python-deltarpm-3.6-3.el7.ppc64le
ibacm-13-7.el7.ppc64le
opus-1.0.2-6.el7.ppc64le
system-config-printer-libs-1.4.1-19.el7.noarch
libthai-0.1.14-9.el7.ppc64le
tracker-1.10.5-4.el7.ppc64le
shared-desktop-ontologies-devel-0.11.0-2.el7.noarch
qt-4.8.5-13.el7.ppc64le
pcre2-10.23-2.el7.ppc64le
gtkspell3-3.0.3-4.el7.ppc64le
libevdev-1.5.6-1.el7.ppc64le
totem-3.22.1-1.el7.ppc64le
virtuoso-opensource-6.1.6-6.el7.ppc64le
strigi-libs-0.7.7-12.20120626.el7.ppc64le
boost-wave-1.53.0-27.el7.ppc64le
libXmu-devel-1.1.2-2.el7.ppc64le
iproute-3.10.0-87.el7.ppc64le
firewalld-0.4.4.4-6.el7.noarch
color-filesystem-1-13.el7.noarch
automoc-1.0-0.20.rc3.el7.ppc64le
perl-Pod-Perldoc-3.20-4.el7.noarch
poppler-utils-0.26.5-16.el7.ppc64le
boost-1.53.0-27.el7.ppc64le
pcp-libs-3.11.8-7.el7.ppc64le
pykickstart-1.99.66.12-1.el7.noarch
openldap-devel-2.4.44-5.el7.ppc64le
perl-Encode-2.51-7.el7.ppc64le
python-gobject-3.22.0-1.el7.ppc64le
krb5-workstation-1.15.1-8.el7.ppc64le
libwacom-0.24-1.el7.ppc64le
isomd5sum-1.0.10-5.el7.ppc64le
abrt-addon-vmcore-2.1.11-48.el7.centos.ppc64le
perl-constant-1.27-2.el7.noarch
compat-libcogl12-1.14.0-3.el7.ppc64le
python-libipa_hbac-1.15.2-50.el7.ppc64le
gdm-3.22.3-11.el7.ppc64le
gstreamer1-devel-1.10.4-2.el7.ppc64le
abrt-retrace-client-2.1.11-48.el7.centos.ppc64le
perl-Exporter-5.68-3.el7.noarch
libXpm-devel-3.5.12-1.el7.ppc64le
python2-pyasn1-modules-0.1.9-7.el7.noarch
gnome-shell-extension-alternate-tab-3.22.2-10.el7.noarch
ttmkfdir-3.0.9-42.el7.ppc64le
samba-libs-4.6.2-8.el7.ppc64le
perl-File-Temp-0.23.01-3.el7.noarch
brltty-4.5-15.el7.ppc64le
sos-3.4-6.el7.centos.noarch
gnome-shell-extension-window-list-3.22.2-10.el7.noarch
clucene-core-2.3.3.4-11.el7.ppc64le
osinfo-db-20170423-2.el7.noarch
perl-macros-5.16.3-292.el7.ppc64le
python-brlapi-0.6.0-15.el7.ppc64le
libselinux-devel-2.5-11.el7.ppc64le
quota-nls-4.01-14.el7.noarch
elfutils-libs-0.168-8.el7.ppc64le
oddjob-mkhomedir-0.31.5-4.el7.ppc64le
perl-5.16.3-292.el7.ppc64le
opal-3.10.10-4.el7.ppc64le
gstreamer-tools-0.10.36-7.el7.ppc64le
libvirt-daemon-driver-storage-scsi-3.2.0-14.el7.ppc64le
satyr-0.13-14.el7.ppc64le
polkit-docs-0.112-12.el7_3.noarch
perl-Compress-Raw-Zlib-2.061-4.el7.ppc64le
compat-libcogl-pango12-1.14.0-3.el7.ppc64le
alsa-lib-devel-1.1.3-3.el7.ppc64le
libvirt-daemon-driver-storage-mpath-3.2.0-14.el7.ppc64le
NetworkManager-libnm-1.8.0-9.el7.ppc64le
tcsh-6.18.01-15.el7.ppc64le
perl-XML-Dumper-0.81-17.el7.noarch
libpfm-devel-4.7.0-4.el7.ppc64le
unixODBC-devel-2.3.1-11.el7.ppc64le
rcs-5.9.0-5.el7.ppc64le
ltrace-0.7.91-14.el7.ppc64le
ed-1.9-4.el7.ppc64le
wqy-zenhei-fonts-0.9.46-11.el7.noarch
lohit-bengali-fonts-2.5.3-4.el7.noarch
paratype-pt-sans-fonts-20101909-3.el7.noarch
paktype-naskh-basic-fonts-4.1-3.el7.noarch
lklug-fonts-0.6-10.20090803cvs.el7.noarch
lohit-kannada-fonts-2.5.3-3.el7.noarch
cjkuni-uming-fonts-0.2.20080216.1-53.el7.noarch
vlgothic-fonts-20130607-2.el7.noarch
lohit-telugu-fonts-2.5.3-3.el7.noarch
gnu-free-serif-fonts-20120503-8.el7.noarch
jomolhari-fonts-0.003-17.el7.noarch
scl-utils-20130529-17.el7_1.ppc64le
diffstat-1.57-4.el7.ppc64le
xorg-x11-drivers-7.7-6.el7.ppc64le
setserial-2.17-33.el7.ppc64le
vinagre-3.22.0-8.el7.ppc64le
man-pages-overrides-7.4.3-1.el7.ppc64le
gedit-3.22.0-3.el7.ppc64le
iwl5150-firmware-8.24.2.2-56.el7.noarch
gnome-contacts-3.22.1-1.el7.ppc64le
words-3.0-22.el7.noarch
setroubleshoot-3.2.28-3.el7.ppc64le
iwl7265-firmware-22.0.7.0-56.el7.noarch
gnome-system-monitor-3.22.2-2.el7.ppc64le
man-pages-3.53-5.el7.noarch
librsvg2-devel-2.40.16-1.el7.ppc64le
gpg-pubkey-f4a80eb5-53a7ff4b
system-config-printer-udev-1.4.1-19.el7.ppc64le
gnome-calculator-3.22.3-1.el7.ppc64le
libgcc-4.8.5-16.el7_4.1.ppc64le
gvfs-afp-1.30.4-3.el7.ppc64le
cpp-4.8.5-16.el7_4.1.ppc64le
latencytop-0.5-13.el7.ppc64le
gcc-c++-4.8.5-16.el7_4.1.ppc64le
gtk3-immodule-xim-3.22.10-4.el7.ppc64le
mousetweaks-3.12.0-1.el7.ppc64le
qt3-MySQL-3.3.8b-51.el7.ppc64le
xvattr-1.3-27.el7.ppc64le
yum-langpacks-0.4.2-7.el7.noarch
rpm-build-4.11.3-25.el7.ppc64le
virt-install-1.4.1-7.el7.noarch
samba-client-4.6.2-8.el7.ppc64le
qt-odbc-4.8.5-13.el7.ppc64le
NetworkManager-tui-1.8.0-9.el7.ppc64le
avahi-0.6.31-17.el7.ppc64le
httpd-manual-2.4.6-67.el7.centos.noarch
PackageKit-gstreamer-plugin-1.1.5-1.el7.centos.ppc64le
tuned-2.8.0-5.el7.noarch
qemu-guest-agent-2.8.0-2.el7.ppc64le
smartmontools-6.2-8.el7.ppc64le
openssh-server-7.4p1-11.el7.ppc64le
dracut-config-rescue-033-502.el7.ppc64le
openlmi-providers-devel-0.5.0-4.el7.ppc64le
oprofile-0.9.9-22.el7.ppc64le
perl-homedir-1.008010-4.el7.noarch
libgudev1-devel-219-42.el7.ppc64le
sudo-1.8.19p2-10.el7.ppc64le
libacl-devel-2.2.51-12.el7.ppc64le
perl-XML-Twig-3.44-2.el7.noarch
crash-trace-command-2.0-12.el7.ppc64le
crash-gcore-command-1.3.1-0.el7.ppc64le
libgnome-keyring-devel-3.12.0-1.el7.ppc64le
binutils-devel-2.25.1-31.base.el7.ppc64le
libcap-ng-devel-0.7.5-4.el7.ppc64le
bash-completion-2.1-6.el7.noarch
dstat-0.7.2-12.el7.noarch
wget-1.14-15.el7.ppc64le
gpg-pubkey-352c64e5-52ae6884
certmonger-0.78.4-3.el7.ppc64le
libatomic-static-4.8.5-16.el7.ppc64le
libicu-devel-50.1.2-15.el7.ppc64le
caribou-0.4.21-1.el7.ppc64le
grub2-common-2.02-0.64.el7.centos.noarch
plymouth-graphics-libs-0.8.9-0.28.20140113.el7.centos.ppc64le
kernel-3.10.0-693.el7.ppc64le
perl-Perl-OSType-1.003-3.el7.noarch
libvirt-daemon-3.2.0-14.el7.ppc64le
ledmon-0.80-2.el7.ppc64le
gupnp-av-0.12.10-1.el7.ppc64le
cups-1.6.3-29.el7.ppc64le
mozilla-filesystem-1.9-11.el7.ppc64le
libqmi-utils-1.16.0-1.el7.ppc64le
anaconda-core-21.48.22.121-1.el7.centos.ppc64le
perl-JSON-PP-2.27202-2.el7.noarch
libvirt-client-3.2.0-14.el7.ppc64le
numactl-devel-2.0.9-6.el7_2.ppc64le
cups-client-1.6.3-29.el7.ppc64le
mutter-3.22.3-11.el7.ppc64le
ipa-common-4.5.0-20.el7.centos.noarch
glibc-devel-2.17-196.el7.ppc64le
firefox-52.2.0-2.el7.centos.ppc64le
perl-Params-Check-0.38-2.el7.noarch
virt-manager-common-1.4.1-7.el7.noarch
indent-2.2.11-13.el7.ppc64le
python-linux-procfs-0.4.9-3.el7.noarch
gnome-session-3.22.3-4.el7.ppc64le
adwaita-cursor-theme-3.22.0-1.el7.noarch
perl-Archive-Extract-0.68-3.el7.noarch
gnome-initial-setup-3.22.1-4.el7.ppc64le
perl-IO-Compress-2.061-2.el7.noarch
geoclue2-2.4.5-1.el7.ppc64le
khmeros-base-fonts-5.0-17.el7.noarch
python-tempita-0.5.1-6.el7.noarch
gnome-online-accounts-3.22.5-1.el7.ppc64le
nhn-nanum-fonts-common-3.020-9.el7.noarch
gobject-introspection-devel-1.50.0-1.el7.ppc64le
rhythmbox-3.4.1-1.el7.ppc64le
libavc1394-0.5.3-14.el7.ppc64le
telepathy-gabble-0.18.1-4.el7.ppc64le
stix-fonts-1.1.0-5.el7.noarch
python-javapackages-3.4.1-11.el7.noarch
gnome-packagekit-installer-3.22.1-2.el7.ppc64le
mesa-filesystem-17.0.1-6.20170307.el7.ppc64le
konkretcmpi-python-0.9.1-5.el7.ppc64le
libsane-hpaio-3.15.9-3.el7.ppc64le
copy-jdk-configs-2.2-3.el7.noarch
usb_modeswitch-2.4.0-5.el7.ppc64le
nhn-nanum-gothic-fonts-3.020-9.el7.noarch
pytz-2016.10-2.el7.noarch
librsvg2-tools-2.40.16-1.el7.ppc64le
bash-4.2.46-28.el7.ppc64le
libreport-plugin-bugzilla-2.1.11-38.el7.centos.ppc64le
kde-workspace-devel-4.11.19-8.el7.ppc64le
libdb-devel-5.3.21-20.el7.ppc64le
fxload-2002_04_11-16.el7.ppc64le
google-crosextra-caladea-fonts-1.002-0.4.20130214.el7.noarch
python-pycparser-2.14-1.el7.noarch
libtimezonemap-0.4.4-1.el7.ppc64le
libcom_err-1.42.9-10.el7.ppc64le
frei0r-plugins-1.3-13.el7.ppc64le
ibus-m17n-1.3.4-13.el7.ppc64le
libcdio-paranoia-10.2+0.90-11.el7.ppc64le
netcf-libs-0.2.8-4.el7.ppc64le
lohit-punjabi-fonts-2.5.3-2.el7.noarch
cmpi-bindings-pywbem-0.9.5-6.el7.ppc64le
at-spi2-core-devel-2.22.0-1.el7.ppc64le
xz-libs-5.2.2-1.el7.ppc64le
libasyncns-0.8-7.el7.ppc64le
libcanberra-devel-0.30-5.el7.ppc64le
coreutils-8.22-18.el7.ppc64le
sssd-ad-1.15.2-50.el7.ppc64le
doxygen-1.8.5-3.el7.ppc64le
httpd-tools-2.4.6-67.el7.centos.ppc64le
libspectre-0.2.8-1.el7.ppc64le
cyrus-sasl-lib-2.1.26-21.el7.ppc64le
rubygem-bigdecimal-1.2.0-30.el7.ppc64le
icedtea-web-1.6.2-4.el7.ppc64le
libarchive-3.1.2-10.el7_2.ppc64le
python-pyblock-0.53-6.el7.ppc64le
byacc-1.9.20130304-3.el7.ppc64le
wodim-1.1.11-23.el7.ppc64le
xorg-x11-drv-qxl-0.1.5-3.el7.ppc64le
elfutils-libelf-0.168-8.el7.ppc64le
rubygem-thor-0.19.1-1.el7.noarch
file-roller-nautilus-3.22.3-1.el7.ppc64le
pkgconfig-0.27.1-4.el7.ppc64le
setroubleshoot-server-3.2.28-3.el7.ppc64le
iwl2030-firmware-18.168.6.1-56.el7.noarch
mailx-12.5-16.el7.ppc64le
xorg-x11-drv-fbdev-0.4.3-25.el7.ppc64le
libtevent-0.9.31-1.el7.ppc64le
policycoreutils-2.5-17.1.el7.ppc64le
java-1.7.0-openjdk-devel-1.7.0.141-2.6.10.5.el7.ppc64le
gsettings-desktop-schemas-3.22.0-1.el7.ppc64le
yum-3.4.3-154.el7.centos.noarch
iwl6000g2a-firmware-17.168.5.3-56.el7.noarch
perl-B-Lint-1.17-3.el7.noarch
gstreamer-plugins-bad-free-0.10.23-23.el7.ppc64le
libvorbis-1.3.3-8.el7.ppc64le
rarian-compat-0.8.1-11.el7.ppc64le
abrt-desktop-2.1.11-48.el7.centos.ppc64le
desktop-file-utils-0.23-1.el7.ppc64le
libiptcdata-1.0.4-11.el7.ppc64le
gpg-pubkey-f533f4fa-56585169
perl-DB_File-1.830-6.el7.ppc64le
compat-poppler022-qt-0.22.5-4.el7.ppc64le
libldb-1.1.29-1.el7.ppc64le
http-parser-2.7.1-1.el7.ppc64le
NetworkManager-libreswan-gnome-1.2.4-2.el7.ppc64le
centos-logos-70.0.6-3.el7.centos.noarch
libX11-common-1.6.5-1.el7.noarch
libstdc++-4.8.5-16.el7_4.1.ppc64le
perl-FCGI-0.74-8.el7.ppc64le
pango-devel-1.40.4-1.el7.ppc64le
libbasicobjects-0.1.1-27.el7.ppc64le
gtk2-immodule-xim-2.24.31-1.el7.ppc64le
libgnome-keyring-3.12.0-1.el7.ppc64le
libXrender-0.9.10-1.el7.ppc64le
libitm-devel-4.8.5-16.el7_4.1.ppc64le
perl-Business-ISBN-2.06-2.el7.noarch
freeglut-2.8.1-3.el7.ppc64le
device-mapper-1.02.140-8.el7.ppc64le
xdg-desktop-portal-gtk-0.5-1.el7.ppc64le
libudisks2-2.1.2-6.el7.ppc64le
pulseaudio-libs-10.0-3.el7.ppc64le
perl-HTTP-Daemon-6.01-5.el7.noarch
xorg-x11-xauth-1.0.9-1.el7.ppc64le
nettle-2.7.1-8.el7.ppc64le
polkit-pkla-compat-0.1-4.el7.ppc64le
startup-notification-devel-0.12-8.el7.ppc64le
genisoimage-1.1.11-23.el7.ppc64le
dbus-x11-1.6.12-17.el7.ppc64le
perl-Text-Soundex-3.04-4.el7.ppc64le
xdg-user-dirs-0.15-4.el7.ppc64le
jansson-2.10-1.el7.ppc64le
NetworkManager-glib-1.8.0-9.el7.ppc64le
rpm-sign-4.11.3-25.el7.ppc64le
gettext-0.19.8.1-2.el7.ppc64le
cairo-1.14.8-2.el7.ppc64le
perl-IO-Socket-SSL-1.94-6.el7.noarch
kdepimlibs-kxmlrpcclient-4.10.5-4.el7.ppc64le
libplist-1.12-3.el7.ppc64le
libwbclient-4.6.2-8.el7.ppc64le
cgdcbxd-1.0.2-7.el7.ppc64le
glib2-devel-2.50.3-3.el7.ppc64le
gdk-pixbuf2-devel-2.36.5-1.el7.ppc64le
theora-tools-1.1.1-8.el7.ppc64le
libkipi-4.10.5-3.el7.ppc64le
libmng-1.0.10-14.el7.ppc64le
abrt-addon-kerneloops-2.1.11-48.el7.centos.ppc64le
grub2-2.02-0.64.el7.centos.ppc64le
xz-devel-5.2.2-1.el7.ppc64le
xorg-x11-xkb-utils-7.7-12.el7.ppc64le
libverto-tevent-0.2.5-4.el7.ppc64le
libkdcraw-devel-4.10.5-4.el7.ppc64le
bzip2-1.0.6-13.el7.ppc64le
iputils-20160308-10.el7.ppc64le
cifs-utils-6.2-10.el7.ppc64le
libpinyin-0.9.93-4.el7.ppc64le
libao-1.1.0-8.el7.ppc64le
gdbm-devel-1.10-8.el7.ppc64le
kdepim-libs-4.10.5-6.el7.ppc64le
libxshmfence-1.2-1.el7.ppc64le
libstoragemgmt-1.4.0-3.el7.ppc64le
psacct-6.6.1-13.el7.ppc64le
pyliblzma-0.5.3-11.el7.ppc64le
libXcursor-devel-1.1.14-8.el7.ppc64le
hesiod-3.2.1-3.el7.ppc64le
okular-devel-4.10.5-4.el7.ppc64le
gsm-1.0.13-11.el7.ppc64le
telepathy-mission-control-5.16.3-3.el7.ppc64le
rng-tools-5-11.el7.ppc64le
python-chardet-2.2.1-1.el7_1.noarch
libcanberra-gtk3-0.30-5.el7.ppc64le
krb5-devel-1.15.1-8.el7.ppc64le
kdesdk-kompare-devel-4.10.5-6.el7.ppc64le
unixODBC-2.3.1-11.el7.ppc64le
dbus-devel-1.6.12-17.el7.ppc64le
kpatch-0.4.0-1.el7.noarch
graphite2-1.3.6-1.el7_2.ppc64le
nautilus-extensions-3.22.3-3.el7.ppc64le
libdb-utils-5.3.21-20.el7.ppc64le
sane-backends-libs-1.0.24-9.el7.ppc64le
zip-3.0-11.el7.ppc64le
mdadm-4.0-5.el7.ppc64le
memstomp-0.1.4-11.el7.ppc64le
libconfig-1.4.9-5.el7.ppc64le
clutter-gst2-2.0.18-1.el7.ppc64le
postgresql-libs-9.2.21-1.el7.ppc64le
gsound-1.0.2-2.el7.ppc64le
ilmbase-1.0.3-7.el7.ppc64le
udisks2-2.1.2-6.el7.ppc64le
perl-core-5.16.3-292.el7.ppc64le
pcsc-lite-libs-1.8.8-6.el7.ppc64le
gvnc-0.7.0-2.el7.ppc64le
qemu-img-1.5.3-141.el7.ppc64le
libappstream-glib-0.6.10-1.el7.ppc64le
sg3_utils-libs-1.37-12.el7.ppc64le
librdmacm-13-7.el7.ppc64le
adcli-0.8.1-3.el7.ppc64le
libnfnetlink-1.0.1-4.el7.ppc64le
colord-gtk-0.1.25-4.el7.ppc64le
libuser-python-0.60-7.el7_1.ppc64le
libfprint-0.5.0-4.el7.ppc64le
OpenEXR-libs-1.7.1-7.el7.ppc64le
attica-devel-0.4.2-1.el7.ppc64le
papi-devel-5.2.0-23.el7.ppc64le
m17n-lib-1.6.4-14.el7.ppc64le
qimageblitz-0.0.6-7.el7.ppc64le
python-urlgrabber-3.10-8.el7.noarch
pcp-selinux-3.11.8-7.el7.ppc64le
perl-Text-ParseWords-3.29-4.el7.noarch
apr-util-devel-1.5.2-6.el7.ppc64le
readline-devel-6.2-10.el7.ppc64le
python-kitchen-1.1.1-5.el7.noarch
gnome-abrt-0.3.4-8.el7.ppc64le
check-devel-0.9.9-5.el7.ppc64le
pulseaudio-gdm-hooks-10.0-3.el7.ppc64le
perl-Scalar-List-Utils-1.27-248.el7.ppc64le
abrt-addon-ccpp-2.1.11-48.el7.centos.ppc64le
gnome-icon-theme-extras-3.12.0-1.el7.noarch
python-slip-0.4.0-2.el7.noarch
brlapi-0.6.0-15.el7.ppc64le
qpdf-libs-5.0.1-3.el7.ppc64le
yelp-xsl-3.20.1-1.el7.noarch
perl-Storable-2.45-3.el7.ppc64le
libosinfo-1.0.0-1.el7.ppc64le
libcap-devel-2.22-9.el7.ppc64le
libepoxy-devel-1.3.1-1.el7.ppc64le
festival-1.96-28.el7.ppc64le
libusbx-1.0.20-1.el7.ppc64le
libvirt-daemon-driver-storage-disk-3.2.0-14.el7.ppc64le
perl-Test-Harness-3.28-3.el7.noarch
polkit-devel-0.112-12.el7_3.ppc64le
perl-Crypt-SSLeay-0.64-5.el7.ppc64le
libverto-devel-0.2.5-4.el7.ppc64le
caribou-gtk2-module-0.4.21-1.el7.ppc64le
vim-filesystem-7.4.160-2.el7.ppc64le
procps-ng-3.3.10-16.el7.ppc64le
NetworkManager-libreswan-1.2.4-2.el7.ppc64le
perl-Module-Metadata-1.000018-2.el7.noarch
pixman-devel-0.34.0-1.el7.ppc64le
patchutils-0.3.3-4.el7.ppc64le
filesystem-3.2-21.el7.ppc64le
cups-filesystem-1.6.3-29.el7.noarch
gettext-devel-0.19.8.1-2.el7.ppc64le
usbredir-0.7.1-2.el7.ppc64le
neon-0.30.0-3.el7.ppc64le
perl-LWP-MediaTypes-6.02-2.el7.noarch
python-qrcode-core-5.0.1-1.el7.noarch
hyphen-en-2.8.6-5.el7.noarch
gnu-free-fonts-common-20120503-8.el7.noarch
gtkmm30-3.22.0-1.el7.ppc64le
initial-setup-gui-0.3.9.40-1.el7.centos.ppc64le
libhugetlbfs-2.16-12.el7.ppc64le
subversion-libs-1.7.14-10.el7.ppc64le
perl-Encode-Locale-1.03-5.el7.noarch
python-inotify-0.9.4-4.el7.noarch
nano-2.3.1-10.el7.ppc64le
mobile-broadband-provider-info-1.20170310-1.el7.noarch
adwaita-gtk2-theme-3.22.2-1.el7.ppc64le
ipa-client-4.5.0-20.el7.centos.ppc64le
perl-IPC-Cmd-0.80-4.el7.noarch
libsoup-2.56.0-3.el7.ppc64le
perl-Term-UI-0.36-2.el7.noarch
python-setuptools-0.9.8-7.el7.noarch
dejavu-sans-mono-fonts-2.33-6.el7.noarch
bind-license-9.9.4-50.el7.noarch
webkitgtk4-jsc-2.14.7-2.el7.ppc64le
firewall-config-0.4.4.4-6.el7.noarch
perl-CPAN-1.9800-292.el7.noarch
gupnp-1.0.1-1.el7.ppc64le
boost-graph-1.53.0-27.el7.ppc64le
python-perf-3.10.0-693.el7.ppc64le
overpass-fonts-2.1-1.el7.noarch
thai-scalable-fonts-common-0.5.0-7.el7.noarch
webkitgtk4-jsc-devel-2.14.7-2.el7.ppc64le
pulseaudio-module-x11-10.0-3.el7.ppc64le
marisa-0.2.4-4.el7.ppc64le
gnutls-c++-3.3.26-9.el7.ppc64le
ca-certificates-2017.2.14-71.el7.noarch
python-idna-2.4-1.el7.noarch
strace-4.12-4.el7.ppc64le
nss-softokn-freebl-3.28.3-6.el7.ppc64le
vino-3.22.0-3.el7.ppc64le
libXaw-devel-1.0.13-4.el7.ppc64le
libreport-centos-2.1.11-38.el7.centos.ppc64le
alsa-utils-1.1.3-2.el7.ppc64le
libnl3-cli-3.2.28-4.el7.ppc64le
python-iniparse-0.4-9.el7.noarch
traceroute-2.0.22-2.el7.ppc64le
libselinux-2.5-11.el7.ppc64le
keybinder3-0.3.0-1.el7.ppc64le
kdepim-devel-4.10.5-6.el7.ppc64le
pakchois-0.4-10.el7.ppc64le
cryptsetup-python-1.7.4-3.el7.ppc64le
libjpeg-turbo-devel-1.2.90-5.el7.ppc64le
python-jwcrypto-0.2.1-1.el7.noarch
lohit-malayalam-fonts-2.5.3-2.el7.noarch
libpng-1.5.13-7.el7_2.ppc64le
freerdp-plugins-1.0.2-10.el7.ppc64le
ibus-chewing-1.4.4-14.el7.ppc64le
libfastjson-0.99.4-2.el7.ppc64le
libsss_sudo-1.15.2-50.el7.ppc64le
redhat-menus-12.0.2-8.el7.noarch
bind-libs-9.9.4-50.el7.ppc64le
gnu-free-sans-fonts-20120503-8.el7.noarch
libuuid-2.23.2-43.el7.ppc64le
festival-freebsoft-utils-0.10-7.el7.noarch
unique3-devel-3.0.2-8.el7.ppc64le
compat-poppler022-0.22.5-4.el7.ppc64le
sssd-proxy-1.15.2-50.el7.ppc64le
python-2.7.5-58.el7.ppc64le
libwvstreams-4.6.1-11.el7.ppc64le
lrzsz-0.12.20-36.el7.ppc64le
sqlite-3.7.17-8.el7.ppc64le
xorg-x11-server-common-1.19.3-11.el7.ppc64le
sushi-3.21.91-1.el7.ppc64le
rubygem-psych-2.0.0-30.el7.ppc64le
gnupg2-2.0.22-4.el7.ppc64le
libmount-2.23.2-43.el7.ppc64le
nss-3.28.4-8.el7.ppc64le
iwl3160-firmware-22.0.7.0-56.el7.noarch
libnl3-3.2.28-4.el7.ppc64le
xorg-x11-drv-ati-7.7.1-3.20160928git3fc839ff.el7.ppc64le
evolution-mapi-3.22.6-1.el7.ppc64le
libservicelog-1.1.17-2.el7.ppc64le
perl-PAR-Dist-0.49-2.el7.noarch
dbus-glib-0.100-7.el7.ppc64le
docbook-style-xsl-1.78.1-3.el7.noarch
iwl100-firmware-39.31.5.1-56.el7.noarch
libxslt-1.1.28-5.el7.ppc64le
junit-4.11-8.el7.noarch
gnome-session-xsession-3.22.3-4.el7.ppc64le
selinux-policy-3.13.1-166.el7.noarch
PackageKit-1.1.5-1.el7.centos.ppc64le
zlib-devel-1.2.7-17.el7.ppc64le
perl-libxml-perl-0.08-19.el7.noarch
iwl4965-firmware-228.61.2.24-56.el7.noarch
p11-kit-0.23.5-3.el7.ppc64le
spice-gtk3-0.33-6.el7.ppc64le
pygobject3-devel-3.22.0-1.el7.ppc64le
systemtap-runtime-3.1-3.el7.ppc64le
nss-softokn-freebl-devel-3.28.3-6.el7.ppc64le
libgee-0.18.1-1.el7.ppc64le
perl-PlRPC-0.2020-14.el7.noarch
python34-libs-3.4.5-4.el7.ppc64le
json-c-0.11-4.el7_0.ppc64le
plymouth-plugin-two-step-0.8.9-0.28.20140113.el7.centos.ppc64le
gnome-font-viewer-3.22.0-1.el7.ppc64le
sssd-client-1.15.2-50.el7.ppc64le
libXext-1.3.3-3.el7.ppc64le
nspr-devel-4.13.1-1.0.el7_3.ppc64le
perl-Algorithm-Diff-1.1902-17.el7.noarch
libgomp-4.8.5-16.el7_4.1.ppc64le
tcp_wrappers-libs-7.6-77.el7.ppc64le
libgdata-devel-0.17.8-1.el7.ppc64le
gnome-screenshot-3.22.0-1.el7.ppc64le
mtdev-1.1.5-5.el7.ppc64le
mesa-libEGL-17.0.1-6.20170307.el7.ppc64le
libpng-devel-1.5.13-7.el7_2.ppc64le
perl-Digest-SHA1-2.13-9.el7.ppc64le
libdhash-0.4.3-27.el7.ppc64le
phonon-devel-4.6.0-10.el7.ppc64le
qt3-ODBC-3.3.8b-51.el7.ppc64le
systemd-219-42.el7.ppc64le
libXinerama-1.1.3-2.1.el7.ppc64le
gdb-7.6.1-100.el7.ppc64le
perl-File-Listing-6.04-7.el7.noarch
jasper-libs-1.900.1-31.el7.ppc64le
ibus-setup-1.5.3-13.el7.noarch
spice-vdagent-0.14.0-14.el7.ppc64le
PackageKit-glib-1.1.5-1.el7.centos.ppc64le
libXmu-1.1.2-2.el7.ppc64le
atkmm-2.24.2-1.el7.ppc64le
perl-Sys-Syslog-0.33-3.el7.ppc64le
libXdmcp-1.1.2-6.el7.ppc64le
kdelibs-devel-4.14.8-6.el7_3.ppc64le
targetcli-2.1.fb46-1.el7.noarch
libcgroup-0.41-13.el7.ppc64le
qt-x11-4.8.5-13.el7.ppc64le
libxcb-devel-1.12-1.el7.ppc64le
perl-HTML-Format-2.10-7.el7.noarch
libsss_idmap-1.15.2-50.el7.ppc64le
kactivities-4.10.5-3.el7.ppc64le
httpd-devel-2.4.6-67.el7.centos.ppc64le
abrt-2.1.11-48.el7.centos.ppc64le
java-1.7.0-openjdk-headless-1.7.0.141-2.6.10.5.el7.ppc64le
apr-devel-1.4.8-3.el7.ppc64le
cdparanoia-10.2-17.el7.ppc64le
libpcap-1.5.3-9.el7.ppc64le
libkworkspace-4.11.19-8.el7.ppc64le
dbus-glib-devel-0.100-7.el7.ppc64le
crontabs-1.11-6.20121102git.el7.noarch
libXi-devel-1.7.9-1.el7.ppc64le
gnome-menus-3.13.3-3.el7.ppc64le
libieee1284-devel-0.2.11-15.el7.ppc64le
kmod-libs-20-15.el7.ppc64le
kde-runtime-4.10.5-8.el7.ppc64le
mod_ssl-2.4.6-67.el7.centos.ppc64le
cyrus-sasl-2.1.26-21.el7.ppc64le
libXScrnSaver-1.2.2-6.1.el7.ppc64le
python-augeas-0.5.0-2.el7.noarch
LibRaw-0.14.8-5.el7.20120830git98d925.ppc64le
hyphen-2.8.6-5.el7.ppc64le
kdenetwork-krdc-libs-4.10.5-8.el7_0.ppc64le
opal-prd-5.5.0-1.el7.ppc64le
rdma-core-13-7.el7.ppc64le
pulseaudio-10.0-3.el7.ppc64le
python-sssdconfig-1.15.2-50.el7.noarch
libisofs-1.2.8-4.el7.ppc64le
libverto-0.2.5-4.el7.ppc64le
kdesdk-kmtrace-devel-4.10.5-6.el7.ppc64le
systemd-devel-219-42.el7.ppc64le
mesa-dri-drivers-17.0.1-6.20170307.el7.ppc64le
clutter-1.26.0-1.el7.ppc64le
fipscheck-1.4.1-6.el7.ppc64le
dwz-0.11-3.el7.ppc64le
boost-regex-1.53.0-27.el7.ppc64le
libXaw-1.0.13-4.el7.ppc64le
systemd-python-219-42.el7.ppc64le
zenity-3.22.0-1.el7.ppc64le
boost-atomic-1.53.0-27.el7.ppc64le
rpm-libs-4.11.3-25.el7.ppc64le
GeoIP-1.5.0-11.el7.ppc64le
libksane-devel-4.10.5-4.el7.ppc64le
rubygem-bundler-1.7.8-3.el7.noarch
git-1.8.3.1-11.el7.ppc64le
brasero-libs-3.12.1-2.el7.ppc64le
c-ares-1.10.0-3.el7.ppc64le
libnfsidmap-0.25-17.el7.ppc64le
cdparanoia-libs-10.2-17.el7.ppc64le
tk-8.5.13-6.el7.ppc64le
libhugetlbfs-devel-2.16-12.el7.ppc64le
NetworkManager-wifi-1.8.0-9.el7.ppc64le
libcanberra-gtk2-0.30-5.el7.ppc64le
hostname-3.13-3.el7.ppc64le
redland-1.0.16-6.el7.ppc64le
libdaemon-0.14-7.el7.ppc64le
brasero-3.12.1-2.el7.ppc64le
cups-devel-1.6.3-29.el7.ppc64le
qca2-2.0.3-7.el7.ppc64le
pangomm-2.40.1-1.el7.ppc64le
libnetfilter_conntrack-1.0.6-1.el7_3.ppc64le
sip-devel-4.14.6-4.el7.ppc64le
perl-parent-0.225-244.el7.noarch
libkkc-0.3.1-9.el7.ppc64le
crypto-utils-2.4.1-42.el7.ppc64le
lvm2-2.02.171-8.el7.ppc64le
poppler-glib-0.26.5-16.el7.ppc64le
crash-7.1.9-2.el7.ppc64le
libbluray-0.2.3-5.el7.ppc64le
perl-Filter-1.49-3.el7.ppc64le
control-center-3.22.2-5.el7.ppc64le
c-ares-devel-1.10.0-3.el7.ppc64le
sysstat-10.1.5-12.el7.ppc64le
mesa-libGL-devel-17.0.1-6.20170307.el7.ppc64le
python-pwquality-1.2.3-4.el7.ppc64le
liblouis-python-2.5.2-10.el7.noarch
perl-PathTools-3.40-5.el7.ppc64le
gnome-shell-extension-apps-menu-3.22.2-10.el7.noarch
hunspell-devel-1.3.2-15.el7.ppc64le
policycoreutils-python-2.5-17.1.el7.ppc64le
libwnck3-3.20.1-1.el7.ppc64le
gsettings-desktop-schemas-devel-3.22.0-1.el7.ppc64le
lsof-4.87-4.el7.ppc64le
perl-Getopt-Long-2.40-2.el7.noarch
nfs-utils-1.3.0-0.48.el7.ppc64le
mtr-0.85-7.el7.ppc64le
autofs-5.0.7-69.el7.ppc64le
cairo-devel-1.14.8-2.el7.ppc64le
xorg-x11-xbitmaps-1.1.1-6.el7.noarch
libreport-2.1.11-38.el7.centos.ppc64le
perl-XML-Parser-2.41-10.el7.ppc64le
libvirt-daemon-driver-storage-3.2.0-14.el7.ppc64le
python2-caribou-0.4.21-1.el7.noarch
fontpackages-filesystem-1.44-8.el7.noarch
perl-Test-Pod-1.48-3.el7.noarch
libuuid-devel-2.23.2-43.el7.ppc64le
perl-Package-Constants-0.02-292.el7.noarch
gnutls-3.3.26-9.el7.ppc64le
libreport-cli-2.1.11-38.el7.centos.ppc64le
gettext-common-devel-0.19.8.1-2.el7.noarch
cups-filters-1.0.35-22.el7.ppc64le
xkeyboard-config-2.20-1.el7.noarch
bison-3.0.4-1.el7.ppc64le
compat-libcolord1-1.0.4-1.el7.ppc64le
perl-Digest-MD5-2.52-3.el7.ppc64le
gnutls-dane-3.3.26-9.el7.ppc64le
libusbx-devel-1.0.20-1.el7.ppc64le
initial-setup-0.3.9.40-1.el7.centos.ppc64le
libchamplain-gtk-0.12.15-1.el7.ppc64le
libreport-filesystem-2.1.11-38.el7.centos.ppc64le
m17n-contrib-1.1.14-3.el7.noarch
newt-python-0.52.15-4.el7.ppc64le
perl-Locale-Maketext-1.23-3.el7.noarch
libvirt-daemon-driver-nodedev-3.2.0-14.el7.ppc64le
perl-ExtUtils-Install-1.58-292.el7.noarch
libvirt-3.2.0-14.el7.ppc64le
gnome-themes-standard-3.22.2-1.el7.ppc64le
gl-manpages-1.1-7.20130122.el7.noarch
lohit-gujarati-fonts-2.5.3-2.el7.noarch
python-backports-ssl_match_hostname-3.4.0.2-4.el7.noarch
perl-local-lib-1.008010-4.el7.noarch
rest-0.8.0-1.el7.ppc64le
perl-Module-Build-0.40.05-2.el7.noarch
ibus-kkc-1.5.18-7.el7.ppc64le
webkitgtk4-plugin-process-gtk2-2.14.7-2.el7.ppc64le
basesystem-10.0-7.el7.centos.noarch
madan-fonts-2.000-11.el7.noarch
python-beaker-1.5.4-10.el7.noarch
boost-locale-1.53.0-27.el7.ppc64le
dleyna-core-0.5.0-1.el7.ppc64le
liberation-sans-fonts-1.07.2-15.el7.noarch
tk-devel-8.5.13-6.el7.ppc64le
gnome-packagekit-updater-3.22.1-2.el7.ppc64le
cim-schema-2.33.0-6.el7.noarch
lohit-assamese-fonts-2.5.3-2.el7.noarch
tagsoup-1.2.1-8.el7.noarch
libshout-2.2.2-11.el7.ppc64le
ntpdate-4.2.6p5-25.el7.centos.2.ppc64le
libproxy-0.4.11-10.el7.ppc64le
gvfs-gphoto2-1.30.4-3.el7.ppc64le
gspell-1.2.3-1.el7.ppc64le
sil-nuosu-fonts-2.1.1-5.el7.noarch
python-ntplib-0.3.2-1.el7.noarch
bc-1.06.95-13.el7.ppc64le
libvirt-daemon-driver-lxc-3.2.0-14.el7.ppc64le
libreport-anaconda-2.1.11-38.el7.centos.ppc64le
kdepimlibs-devel-4.10.5-4.el7.ppc64le
unique3-3.0.2-8.el7.ppc64le
freetype-2.4.11-15.el7.ppc64le
lohit-marathi-fonts-2.5.3-2.el7.noarch
python2-cryptography-1.7.2-1.el7.ppc64le
libss-1.42.9-10.el7.ppc64le
kernel-tools-libs-3.10.0-693.el7.ppc64le
libsysfs-2.1.0-16.el7.ppc64le
ibus-hangul-1.4.2-10.el7.ppc64le
freerdp-1.0.2-10.el7.ppc64le
popt-1.13-16.el7.ppc64le
open-sans-fonts-1.10-1.el7.noarch
bind-libs-lite-9.9.4-50.el7.ppc64le
lksctp-tools-1.0.17-2.el7.ppc64le
sssd-common-pac-1.15.2-50.el7.ppc64le
libtiff-4.0.3-27.el7_3.ppc64le
gnome-desktop3-devel-3.22.2-2.el7.ppc64le
cdrdao-1.2.3-20.el7.ppc64le
expat-2.1.0-10.el7_3.ppc64le
latrace-0.5.11-6.1.el7.ppc64le
perl-Net-SSLeay-1.55-6.el7.ppc64le
cups-libs-1.6.3-29.el7.ppc64le
dmraid-events-1.0.0.rc16-28.el7.ppc64le
rubygem-io-console-0.4.2-30.el7.ppc64le
gutenprint-cups-5.2.9-18.el7.ppc64le
xorg-x11-server-Xorg-1.19.3-11.el7.ppc64le
libtalloc-2.1.9-1.el7.ppc64le
iwl6000g2b-firmware-17.168.5.2-56.el7.noarch
nss-sysinit-3.28.4-8.el7.ppc64le
glib2-2.50.3-3.el7.ppc64le
rpm-python-4.11.3-25.el7.ppc64le
ustr-1.0.4-16.el7.ppc64le
gucharmap-3.18.2-1.el7.ppc64le
xorg-x11-drv-dummy-0.3.7-1.el7.ppc64le
libogg-1.3.0-7.el7.ppc64le
iwl6000-firmware-9.221.4.1-56.el7.noarch
docbook-dtds-1.0-60.el7.noarch
xorg-x11-proto-devel-7.7-20.el7.noarch
pygpgme-0.3-9.el7.ppc64le
openssh-7.4p1-11.el7.ppc64le
cheese-3.22.1-1.el7.ppc64le
jline-1.0-8.el7.noarch
libcap-2.22-9.el7.ppc64le
ivtv-firmware-20080701-26.el7.noarch
perl-Pod-LaTeX-0.61-2.el7.noarch
enchant-1.6.0-8.el7.ppc64le
python2-ipalib-4.5.0-20.el7.centos.noarch
tog-pegasus-libs-2.14.1-5.el7.ppc64le
firstboot-19.12-1.el7.ppc64le
gupnp-dlna-0.10.5-1.el7.ppc64le
which-2.20-7.el7.ppc64le
epel-release-7-9.noarch
perl-Net-Daemon-0.48-5.el7.noarch
libcroco-0.6.11-1.el7.ppc64le
liboauth-devel-0.9.7-4.el7.ppc64le
libhangul-0.1.0-8.el7.ppc64le
eog-3.20.5-2.el7.ppc64le
plymouth-theme-charge-0.8.9-0.28.20140113.el7.centos.ppc64le
libcollection-0.6.2-27.el7.ppc64le
libgfortran-4.8.5-16.el7_4.1.ppc64le
perl-Locale-Codes-3.26-2.el7.noarch
pygobject2-2.28.6-11.el7.ppc64le
libXdamage-1.1.4-4.1.el7.ppc64le
libestr-0.1.9-2.el7.ppc64le
PackageKit-gtk3-module-1.1.5-1.el7.centos.ppc64le
libgweather-devel-3.20.4-1.el7.ppc64le
xz-5.2.2-1.el7.ppc64le
perl-WWW-RobotRules-6.02-5.el7.noarch
libICE-devel-1.0.9-9.el7.ppc64le
libXft-2.3.2-2.el7.ppc64le
cryptsetup-libs-1.7.4-3.el7.ppc64le
alsa-plugins-pulseaudio-1.1.1-1.el7.ppc64le
glx-utils-8.2.0-3.el7.ppc64le
speex-1.2-0.19.rc1.el7.ppc64le
perl-HTTP-Negotiate-6.01-5.el7.noarch
libtirpc-0.2.4-0.10.el7.ppc64le
pulseaudio-libs-glib2-10.0-3.el7.ppc64le
mesa-libgbm-17.0.1-6.20170307.el7.ppc64le
pulseaudio-libs-devel-10.0-3.el7.ppc64le
imsettings-1.6.3-9.el7.ppc64le
hunspell-en-US-0.20121024-6.el7.noarch
perl-IO-Socket-IP-0.21-4.el7.noarch
nss-util-devel-3.28.4-3.el7.ppc64le
libXxf86vm-1.1.4-1.el7.ppc64le
hwdata-0.252-8.6.el7.ppc64le
kernel-tools-3.10.0-693.el7.ppc64le
nepomuk-core-libs-4.10.5-5.el7.ppc64le
exiv2-libs-0.23-6.el7.ppc64le
perl-libwww-perl-6.05-2.el7.noarch
p11-kit-devel-0.23.5-3.el7.ppc64le
librsvg2-2.40.16-1.el7.ppc64le
libsmbclient-4.6.2-8.el7.ppc64le
abrt-console-notification-2.1.11-48.el7.centos.ppc64le
kdesdk-okteta-libs-4.10.5-6.el7.ppc64le
boost-chrono-1.53.0-27.el7.ppc64le
iw-4.3-1.el7.ppc64le
libcom_err-devel-1.42.9-10.el7.ppc64le
akonadi-1.9.2-4.el7.ppc64le
accountsservice-libs-0.6.45-2.el7.ppc64le
wvdial-1.61-9.el7.ppc64le
libkexiv2-devel-4.10.5-3.el7.ppc64le
libmpc-1.0.1-3.el7.ppc64le
lm_sensors-devel-3.4.0-4.20160601gitf9185e5.el7.ppc64le
meanwhile-1.1.0-12.el7.ppc64le
libXt-devel-1.1.5-3.el7.ppc64le
plymouth-scripts-0.8.9-0.28.20140113.el7.centos.ppc64le
mod_fcgid-2.3.9-4.el7.ppc64le
kdepim-runtime-libs-4.10.5-3.el7.ppc64le
mesa-libglapi-17.0.1-6.20170307.el7.ppc64le
hunspell-en-GB-0.20121024-6.el7.noarch
sip-4.14.6-4.el7.ppc64le
cairomm-1.12.0-1.el7.ppc64le
abrt-addon-xorg-2.1.11-48.el7.centos.ppc64le
ModemManager-1.6.0-2.el7.ppc64le
kdenetwork-krdc-devel-4.10.5-8.el7_0.ppc64le
libieee1284-0.2.11-15.el7.ppc64le
highlight-3.13-3.el7.ppc64le
pyOpenSSL-0.13.1-3.el7.ppc64le
gtk-update-icon-cache-3.22.10-4.el7.ppc64le
NetworkManager-1.8.0-9.el7.ppc64le
crda-3.13_2016.02.08-1.el7.ppc64le
kdesdk-kompare-libs-4.10.5-6.el7.ppc64le
newt-0.52.15-4.el7.ppc64le
xcb-util-0.4.0-2.el7.ppc64le
automake-1.13.4-3.el7.noarch
libgweather-3.20.4-1.el7.ppc64le
lockdev-1.0.4-0.13.20111007git.el7.ppc64le
man-db-2.6.3-9.el7.ppc64le
gd-2.0.35-26.el7.ppc64le
exempi-2.2.0-8.el7.ppc64le
curl-7.29.0-42.el7.ppc64le
snappy-1.1.0-3.el7.ppc64le
libreport-gtk-2.1.11-38.el7.centos.ppc64le
unbound-libs-1.4.20-34.el7.ppc64le
tcpdump-4.9.0-5.el7.ppc64le
sane-backends-drivers-scanners-1.0.24-9.el7.ppc64le
libedit-3.0-12.20121213cvs.el7.ppc64le
liboauth-0.9.7-4.el7.ppc64le
libmpcdec-1.2.6-12.el7.ppc64le
libnm-gtk-1.8.0-3.el7.ppc64le
grub2-tools-extra-2.02-0.64.el7.centos.ppc64le
libdmx-1.1.3-3.el7.ppc64le
wavpack-4.60.1-9.el7.ppc64le
rasqal-0.9.30-4.el7.ppc64le
autogen-libopts-5.18-5.el7.ppc64le
gnome-bluetooth-libs-3.20.1-1.el7.ppc64le
qt-settings-19-23.5.el7.centos.noarch
libxslt-devel-1.1.28-5.el7.ppc64le
grilo-plugins-0.3.4-1.el7.ppc64le
SDL-1.2.15-14.el7.ppc64le
sip-macros-4.14.6-4.el7.ppc64le
iptables-1.4.21-18.0.1.el7.centos.ppc64le
gstreamer-plugins-good-0.10.31-13.el7.ppc64le
qjson-0.8.1-4.el7.ppc64le
perl-Test-Pod-Coverage-1.08-21.el7.noarch
pcp-conf-3.11.8-7.el7.ppc64le
perl-podlators-2.5.1-3.el7.noarch
libcurl-devel-7.29.0-42.el7.ppc64le
graphite2-devel-1.3.6-1.el7_2.ppc64le
pygtk2-2.24.0-9.el7.ppc64le
kexec-tools-2.0.14-17.el7.ppc64le
iptables-devel-1.4.21-18.0.1.el7.centos.ppc64le
gnome-shell-3.22.3-17.el7.ppc64le
perl-Carp-1.26-244.el7.noarch
liblouis-2.5.2-10.el7.ppc64le
dvd+rw-tools-7.1-15.el7.ppc64le
ptlib-2.10.10-6.el7.ppc64le
samba-common-libs-4.6.2-8.el7.ppc64le
gvfs-devel-1.30.4-3.el7.ppc64le
gnome-shell-extension-launch-new-instance-3.22.2-10.el7.noarch
perl-libs-5.16.3-292.el7.ppc64le
libselinux-utils-2.5-11.el7.ppc64le
libsepol-devel-2.5-6.el7.ppc64le
festival-lib-1.96-28.el7.ppc64le
oddjob-0.31.5-4.el7.ppc64le
latencytop-tui-0.5-13.el7.ppc64le
libvirt-daemon-driver-storage-core-3.2.0-14.el7.ppc64le
perl-Data-Dumper-2.145-3.el7.ppc64le
libreport-python-2.1.11-38.el7.centos.ppc64le
libical-devel-1.0.1-1.el7.ppc64le
libmx-1.4.7-10.el7.ppc64le
cups-pk-helper-0.2.6-2.el7.ppc64le
=== TEST BEGIN ===
Install prefix    /var/tmp/patchew-tester-tmp-0kkotmms/src/install
BIOS directory    /var/tmp/patchew-tester-tmp-0kkotmms/src/install/share/qemu
firmware path     /var/tmp/patchew-tester-tmp-0kkotmms/src/install/share/qemu-firmware
binary directory  /var/tmp/patchew-tester-tmp-0kkotmms/src/install/bin
library directory /var/tmp/patchew-tester-tmp-0kkotmms/src/install/lib
module directory  /var/tmp/patchew-tester-tmp-0kkotmms/src/install/lib/qemu
libexec directory /var/tmp/patchew-tester-tmp-0kkotmms/src/install/libexec
include directory /var/tmp/patchew-tester-tmp-0kkotmms/src/install/include
config directory  /var/tmp/patchew-tester-tmp-0kkotmms/src/install/etc
local state directory   /var/tmp/patchew-tester-tmp-0kkotmms/src/install/var
Manual directory  /var/tmp/patchew-tester-tmp-0kkotmms/src/install/share/man
ELF interp prefix /usr/gnemul/qemu-%M
Source path       /var/tmp/patchew-tester-tmp-0kkotmms/src
GIT binary        git
GIT submodules    ui/keycodemapdb dtc capstone
C compiler        cc
Host C compiler   cc
C++ compiler      c++
Objective-C compiler cc
ARFLAGS           rv
CFLAGS            -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -g 
QEMU_CFLAGS       -I/usr/include/pixman-1   -I$(SRC_PATH)/dtc/libfdt -Werror -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include   -DNCURSES_WIDECHAR   -m64 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv  -Wendif-labels -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -Wno-missing-braces -I/usr/include/p11-kit-1       -I/usr/include/libpng15   -I$(SRC_PATH)/capstone/include
LDFLAGS           -Wl,--warn-common -m64 -g 
make              make
install           install
python            python -B
smbd              /usr/sbin/smbd
module support    no
host CPU          ppc64
host big endian   no
target list       aarch64-softmmu alpha-softmmu arm-softmmu cris-softmmu i386-softmmu lm32-softmmu m68k-softmmu microblazeel-softmmu microblaze-softmmu mips64el-softmmu mips64-softmmu mipsel-softmmu mips-softmmu moxie-softmmu nios2-softmmu or1k-softmmu ppc64-softmmu ppcemb-softmmu ppc-softmmu s390x-softmmu sh4eb-softmmu sh4-softmmu sparc64-softmmu sparc-softmmu tricore-softmmu unicore32-softmmu x86_64-softmmu xtensaeb-softmmu xtensa-softmmu aarch64_be-linux-user aarch64-linux-user alpha-linux-user armeb-linux-user arm-linux-user cris-linux-user hppa-linux-user i386-linux-user m68k-linux-user microblazeel-linux-user microblaze-linux-user mips64el-linux-user mips64-linux-user mipsel-linux-user mips-linux-user mipsn32el-linux-user mipsn32-linux-user nios2-linux-user or1k-linux-user ppc64abi32-linux-user ppc64le-linux-user ppc64-linux-user ppc-linux-user s390x-linux-user sh4eb-linux-user sh4-linux-user sparc32plus-linux-user sparc64-linux-user sparc-linux-user tilegx-linux-user x86_64-linux-user
gprof enabled     no
sparse enabled    no
strip binaries    yes
profiler          no
static build      no
SDL support       yes (1.2.15)
GTK support       yes (3.22.10)
GTK GL support    no
VTE support       no 
TLS priority      NORMAL
GNUTLS support    yes
GNUTLS rnd        yes
libgcrypt         no
libgcrypt kdf     no
nettle            yes (2.7.1)
nettle kdf        yes
libtasn1          yes
curses support    yes
virgl support     no
curl support      yes
mingw32 support   no
Audio drivers     oss
Block whitelist (rw) 
Block whitelist (ro) 
VirtFS support    yes
Multipath support no
VNC support       yes
VNC SASL support  yes
VNC JPEG support  yes
VNC PNG support   yes
xen support       no
brlapi support    no
bluez  support    no
Documentation     no
PIE               no
vde support       no
netmap support    no
Linux AIO support yes
ATTR/XATTR support yes
Install blobs     yes
KVM support       yes
HAX support       no
HVF support       no
TCG support       yes
TCG debug enabled no
TCG interpreter   no
malloc trim support yes
RDMA support      yes
fdt support       yes
preadv support    yes
fdatasync         yes
madvise           yes
posix_madvise     yes
libcap-ng support yes
vhost-net support yes
vhost-scsi support yes
vhost-vsock support yes
vhost-user support yes
Trace backends    log
spice support     no 
rbd support       no
xfsctl support    no
smartcard support no
libusb            yes
usb net redir     no
OpenGL support    no
OpenGL dmabufs    no
libiscsi support  no
libnfs support    no
build guest agent yes
QGA VSS support   no
QGA w32 disk info no
QGA MSI support   no
seccomp support   no
coroutine backend ucontext
coroutine pool    yes
debug stack usage no
crypto afalg      no
GlusterFS support no
gcov              gcov
gcov enabled      no
TPM support       yes
libssh2 support   no
TPM passthrough   no
TPM emulator      yes
QOM debugging     yes
Live block migration yes
lzo support       no
snappy support    no
bzip2 support     yes
NUMA host support yes
tcmalloc support  no
jemalloc support  no
avx2 optimization no
replication support yes
VxHS block device no
capstone          git
  GEN     aarch64-softmmu/config-devices.mak.tmp
  GEN     alpha-softmmu/config-devices.mak.tmp
  GEN     arm-softmmu/config-devices.mak.tmp
  GEN     cris-softmmu/config-devices.mak.tmp
  GEN     i386-softmmu/config-devices.mak.tmp
  GEN     lm32-softmmu/config-devices.mak.tmp
  GEN     m68k-softmmu/config-devices.mak.tmp
  GEN     microblazeel-softmmu/config-devices.mak.tmp
  GEN     mips64-softmmu/config-devices.mak.tmp
  GEN     mips64el-softmmu/config-devices.mak.tmp
  GEN     microblaze-softmmu/config-devices.mak.tmp
  GEN     alpha-softmmu/config-devices.mak
  GEN     mipsel-softmmu/config-devices.mak.tmp
  GEN     mips-softmmu/config-devices.mak.tmp
  GEN     moxie-softmmu/config-devices.mak.tmp
  GEN     nios2-softmmu/config-devices.mak.tmp
  GEN     lm32-softmmu/config-devices.mak
  GEN     arm-softmmu/config-devices.mak
  GEN     or1k-softmmu/config-devices.mak.tmp
  GEN     ppc64-softmmu/config-devices.mak.tmp
  GEN     cris-softmmu/config-devices.mak
  GEN     m68k-softmmu/config-devices.mak
  GEN     ppcemb-softmmu/config-devices.mak.tmp
  GEN     ppc-softmmu/config-devices.mak.tmp
  GEN     microblaze-softmmu/config-devices.mak
  GEN     sparc64-softmmu/config-devices.mak.tmp
  GEN     sparc-softmmu/config-devices.mak.tmp
  GEN     tricore-softmmu/config-devices.mak.tmp
  GEN     sh4eb-softmmu/config-devices.mak.tmp
  GEN     s390x-softmmu/config-devices.mak.tmp
  GEN     unicore32-softmmu/config-devices.mak.tmp
  GEN     sh4-softmmu/config-devices.mak.tmp
  GEN     cris-linux-user/config-devices.mak.tmp
  GEN     hppa-linux-user/config-devices.mak.tmp
  GEN     xtensa-softmmu/config-devices.mak.tmp
  GEN     m68k-linux-user/config-devices.mak.tmp
  GEN     i386-linux-user/config-devices.mak.tmp
  GEN     x86_64-softmmu/config-devices.mak.tmp
  GEN     aarch64_be-linux-user/config-devices.mak.tmp
  GEN     xtensaeb-softmmu/config-devices.mak.tmp
  GEN     mips64el-linux-user/config-devices.mak.tmp
  GEN     microblazeel-linux-user/config-devices.mak.tmp
  GEN     alpha-linux-user/config-devices.mak.tmp
  GEN     armeb-linux-user/config-devices.mak.tmp
  GEN     arm-linux-user/config-devices.mak.tmp
  GEN     aarch64-linux-user/config-devices.mak.tmp
  GEN     microblaze-linux-user/config-devices.mak.tmp
  GEN     i386-softmmu/config-devices.mak
  GEN     mipsel-linux-user/config-devices.mak.tmp
  GEN     mips64-linux-user/config-devices.mak.tmp
  GEN     s390x-linux-user/config-devices.mak.tmp
  GEN     nios2-softmmu/config-devices.mak
  GEN     or1k-linux-user/config-devices.mak.tmp
  GEN     mips-linux-user/config-devices.mak.tmp
  GEN     mipsn32el-linux-user/config-devices.mak.tmp
  GEN     sparc32plus-linux-user/config-devices.mak.tmp
  GEN     ppc64le-linux-user/config-devices.mak.tmp
  GEN     ppc64-linux-user/config-devices.mak.tmp
  GEN     sh4eb-linux-user/config-devices.mak.tmp
  GEN     tilegx-linux-user/config-devices.mak.tmp
  GEN     nios2-linux-user/config-devices.mak.tmp
  GEN     sh4-linux-user/config-devices.mak.tmp
  GEN     sparc64-linux-user/config-devices.mak.tmp
  GEN     ppc64abi32-linux-user/config-devices.mak.tmp
  GEN     sparc-linux-user/config-devices.mak.tmp
  GEN     aarch64-softmmu/config-devices.mak
  GEN     mipsn32-linux-user/config-devices.mak.tmp
  GEN     microblazeel-softmmu/config-devices.mak
  GEN     ppc-linux-user/config-devices.mak.tmp
  GEN     config-host.h
  GEN     qemu-options.def
  GIT     ui/keycodemapdb dtc capstone
  GEN     m68k-linux-user/config-devices.mak
  GEN     qapi-types.h
  GEN     qmp-commands.h
  GEN     microblazeel-linux-user/config-devices.mak
  GEN     mipsel-softmmu/config-devices.mak
  GEN     sparc32plus-linux-user/config-devices.mak
  GEN     qapi-visit.h
  GEN     qapi-event.h
  GEN     unicore32-softmmu/config-devices.mak
  GEN     qmp-marshal.c
  GEN     moxie-softmmu/config-devices.mak
  GEN     qapi-types.c
  GEN     qapi-visit.c
  GEN     x86_64-linux-user/config-devices.mak.tmp
  GEN     ppc64-linux-user/config-devices.mak
  GEN     qapi-event.c
  GEN     or1k-softmmu/config-devices.mak
  GEN     sparc64-softmmu/config-devices.mak
  GEN     xtensa-softmmu/config-devices.mak
  GEN     alpha-linux-user/config-devices.mak
  GEN     sh4-softmmu/config-devices.mak
  GEN     cris-linux-user/config-devices.mak
  GEN     mipsn32el-linux-user/config-devices.mak
  GEN     mips64-softmmu/config-devices.mak
  GEN     armeb-linux-user/config-devices.mak
  GEN     sh4eb-linux-user/config-devices.mak
  GEN     tricore-softmmu/config-devices.mak
  GEN     qmp-introspect.c
  GEN     qmp-introspect.h
  GEN     mips64el-linux-user/config-devices.mak
  GEN     trace/generated-tcg-tracers.h
  GEN     arm-linux-user/config-devices.mak
  GEN     s390x-linux-user/config-devices.mak
  GEN     trace/generated-helpers-wrappers.h
  GEN     xtensaeb-softmmu/config-devices.mak
  GEN     aarch64_be-linux-user/config-devices.mak
  GEN     ppc-softmmu/config-devices.mak
  GEN     i386-linux-user/config-devices.mak
  GEN     aarch64-linux-user/config-devices.mak
  GEN     mips-linux-user/config-devices.mak
  GEN     nios2-linux-user/config-devices.mak
  GEN     s390x-softmmu/config-devices.mak
  GEN     ppc-linux-user/config-devices.mak
  GEN     x86_64-linux-user/config-devices.mak
  GEN     trace/generated-helpers.h
  GEN     trace/generated-helpers.c
  GEN     or1k-linux-user/config-devices.mak
  GEN     ppc64le-linux-user/config-devices.mak
  GEN     sparc64-linux-user/config-devices.mak
  GEN     ppc64-softmmu/config-devices.mak
  GEN     ppcemb-softmmu/config-devices.mak
  GEN     mips64el-softmmu/config-devices.mak
  GEN     sh4eb-softmmu/config-devices.mak
  GEN     sparc-softmmu/config-devices.mak
  GEN     mipsel-linux-user/config-devices.mak
  GEN     module_block.h
  GEN     hppa-linux-user/config-devices.mak
  GEN     sh4-linux-user/config-devices.mak
  GEN     sparc-linux-user/config-devices.mak
  GEN     mips-softmmu/config-devices.mak
  GEN     tilegx-linux-user/config-devices.mak
  GEN     microblaze-linux-user/config-devices.mak
  GEN     mips64-linux-user/config-devices.mak
  GEN     mipsn32-linux-user/config-devices.mak
  GEN     x86_64-softmmu/config-devices.mak
  GEN     ppc64abi32-linux-user/config-devices.mak
  GEN     tests/test-qapi-types.h
  GEN     tests/test-qapi-visit.h
  GEN     tests/test-qmp-commands.h
  GEN     tests/test-qapi-event.h
  GEN     tests/test-qmp-introspect.h
  GEN     trace-root.h
  GEN     util/trace.h
  GEN     crypto/trace.h
  GEN     io/trace.h
  GEN     migration/trace.h
  GEN     block/trace.h
  GEN     chardev/trace.h
  GEN     hw/block/trace.h
  GEN     hw/block/dataplane/trace.h
  GEN     hw/char/trace.h
  GEN     hw/intc/trace.h
  GEN     hw/net/trace.h
  GEN     hw/rdma/trace.h
  GEN     hw/rdma/vmw/trace.h
  GEN     hw/virtio/trace.h
  GEN     hw/audio/trace.h
  GEN     hw/misc/trace.h
  GEN     hw/usb/trace.h
  GEN     hw/scsi/trace.h
  GEN     hw/nvram/trace.h
  GEN     hw/display/trace.h
  GEN     hw/input/trace.h
  GEN     hw/timer/trace.h
  GEN     hw/dma/trace.h
  GEN     hw/sparc/trace.h
  GEN     hw/sparc64/trace.h
  GEN     hw/sd/trace.h
  GEN     hw/isa/trace.h
  GEN     hw/mem/trace.h
  GEN     hw/i386/trace.h
  GEN     hw/i386/xen/trace.h
  GEN     hw/9pfs/trace.h
  GEN     hw/ppc/trace.h
  GEN     hw/pci/trace.h
  GEN     hw/s390x/trace.h
  GEN     hw/vfio/trace.h
  GEN     hw/acpi/trace.h
  GEN     hw/arm/trace.h
  GEN     hw/alpha/trace.h
  GEN     hw/xen/trace.h
  GEN     hw/ide/trace.h
  GEN     ui/trace.h
  GEN     audio/trace.h
  GEN     net/trace.h
  GEN     target/arm/trace.h
  GEN     target/i386/trace.h
  GEN     target/mips/trace.h
  GEN     target/sparc/trace.h
  GEN     target/s390x/trace.h
  GEN     target/ppc/trace.h
  GEN     qom/trace.h
  GEN     linux-user/trace.h
  GEN     qapi/trace.h
  GEN     accel/tcg/trace.h
  GEN     accel/kvm/trace.h
  GEN     nbd/trace.h
  GEN     scsi/trace.h
  GEN     trace-root.c
  GEN     util/trace.c
  GEN     crypto/trace.c
  GEN     io/trace.c
  GEN     migration/trace.c
  GEN     block/trace.c
  GEN     chardev/trace.c
  GEN     hw/block/trace.c
  GEN     hw/block/dataplane/trace.c
  GEN     hw/char/trace.c
  GEN     hw/intc/trace.c
  GEN     hw/net/trace.c
  GEN     hw/rdma/trace.c
  GEN     hw/rdma/vmw/trace.c
  GEN     hw/virtio/trace.c
  GEN     hw/audio/trace.c
  GEN     hw/misc/trace.c
  GEN     hw/usb/trace.c
  GEN     hw/scsi/trace.c
  GEN     hw/nvram/trace.c
  GEN     hw/display/trace.c
  GEN     hw/input/trace.c
  GEN     hw/timer/trace.c
  GEN     hw/dma/trace.c
  GEN     hw/sparc/trace.c
  GEN     hw/sparc64/trace.c
  GEN     hw/sd/trace.c
  GEN     hw/isa/trace.c
  GEN     hw/mem/trace.c
  GEN     hw/i386/trace.c
  GEN     hw/i386/xen/trace.c
  GEN     hw/9pfs/trace.c
  GEN     hw/ppc/trace.c
  GEN     hw/pci/trace.c
  GEN     hw/s390x/trace.c
  GEN     hw/vfio/trace.c
  GEN     hw/acpi/trace.c
  GEN     hw/arm/trace.c
  GEN     hw/alpha/trace.c
  GEN     hw/xen/trace.c
  GEN     hw/ide/trace.c
  GEN     ui/trace.c
  GEN     audio/trace.c
  GEN     net/trace.c
  GEN     target/arm/trace.c
  GEN     target/i386/trace.c
  GEN     target/mips/trace.c
  GEN     target/sparc/trace.c
  GEN     target/s390x/trace.c
  GEN     target/ppc/trace.c
  GEN     qom/trace.c
  GEN     linux-user/trace.c
  GEN     qapi/trace.c
  GEN     accel/tcg/trace.c
  GEN     accel/kvm/trace.c
  GEN     nbd/trace.c
  GEN     scsi/trace.c
  GEN     config-all-devices.mak
  GEN     ui/input-keymap-linux-to-qcode.c
  GEN     ui/input-keymap-qcode-to-qnum.c
  GEN     ui/input-keymap-qnum-to-qcode.c
  GEN     ui/input-keymap-qcode-to-linux.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/dumptrees.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/trees.S
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/testutils.c
  CC      cs.o
  CC      utils.o
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/value-labels.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/asm_tree_dump.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/check_path.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/truncated_property.c
  CC      SStream.o
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/overlay_bad_fixup.c
  CC      MCInstrDesc.o
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/overlay.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/subnode_iterate.c
  CC      MCRegisterInfo.o
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/property_iterate.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/integer-expressions.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/utilfdt_test.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/path_offset_aliases.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/add_subnode_with_nops.c
  CC      arch/ARM/ARMInstPrinter.o
  CC      arch/ARM/ARMMapping.o
  CC      arch/ARM/ARMDisassembler.o
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/dtbs_equal_unordered.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/dtb_reverse.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/dtbs_equal_ordered.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/extra-terminating-null.c
  CC      arch/ARM/ARMModule.o
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/incbin.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/appendprop2.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/references.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/path-references.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/string_escapes.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/boot-cpuid.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/phandle_format.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/propname_escapes.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/appendprop1.c
  CC      arch/AArch64/AArch64BaseInfo.o
  CC      arch/AArch64/AArch64Disassembler.o
  CC      arch/AArch64/AArch64InstPrinter.o
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/del_node.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/del_property.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/open_pack.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/set_name.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/setprop.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/mangle-layout.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/sw_tree1.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/nopulate.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/move_and_save.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/rw_tree1.c
  CC      arch/AArch64/AArch64Module.o
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/nop_node.c
  CC      arch/AArch64/AArch64Mapping.o
  CC      arch/Mips/MipsDisassembler.o
  CC      arch/Mips/MipsInstPrinter.o
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/nop_property.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/setprop_inplace.c
  CC      arch/PowerPC/PPCDisassembler.o
  CC      arch/Mips/MipsModule.o
  CC      arch/PowerPC/PPCInstPrinter.o
  CC      arch/PowerPC/PPCMapping.o
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/stringlist.c
  CC      arch/Mips/MipsMapping.o
  CC      arch/PowerPC/PPCModule.o
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/addr_size_cells.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/notfound.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/sized_cells.c
  CC      arch/Sparc/SparcInstPrinter.o
  CC      arch/Sparc/SparcDisassembler.o
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/char_literal.c
  CC      arch/Sparc/SparcModule.o
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/node_offset_by_compatible.c
  CC      arch/SystemZ/SystemZDisassembler.o
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/node_check_compatible.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/node_offset_by_prop_value.c
  CC      arch/Sparc/SparcMapping.o
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/node_offset_by_phandle.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/parent_offset.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/supernode_atdepth_offset.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/get_path.c
  CC      arch/SystemZ/SystemZInstPrinter.o
  CC      arch/SystemZ/SystemZMapping.o
  CC      arch/SystemZ/SystemZModule.o
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/get_alias.c
  CC      arch/SystemZ/SystemZMCTargetDesc.o
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/get_phandle.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/getprop.c
  CC      arch/X86/X86DisassemblerDecoder.o
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/get_name.c
  CC      arch/X86/X86Disassembler.o
  CC      arch/X86/X86IntelInstPrinter.o
  CC      arch/X86/X86ATTInstPrinter.o
  CC      arch/XCore/XCoreDisassembler.o
  CC      arch/X86/X86Mapping.o
  CC      arch/X86/X86Module.o
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/path_offset.c
  CC      arch/XCore/XCoreInstPrinter.o
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/get_mem_rsv.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/root_node.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/libfdt/fdt_overlay.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/find_property.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/libfdt/fdt_addresses.c
  CC      arch/XCore/XCoreMapping.o
  CC      arch/XCore/XCoreModule.o
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/tests/subnode_offset.c
  CC      MCInst.o
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/libfdt/fdt_empty_tree.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/libfdt/fdt_strerror.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/libfdt/fdt_rw.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/libfdt/fdt_sw.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/libfdt/fdt_wip.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/libfdt/fdt_ro.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/libfdt/fdt.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/util.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/fdtoverlay.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/fdtput.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/fdtdump.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/fdtget.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/srcpos.c
	 LEX convert-dtsv0-lexer.lex.c
	 BISON dtc-parser.tab.c
	 LEX dtc-lexer.lex.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/treesource.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/livetree.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/fstree.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/flattree.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/dtc.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/data.c
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/checks.c
	 DEP convert-dtsv0-lexer.lex.c
	 DEP dtc-parser.tab.c
	 DEP dtc-lexer.lex.c
	CHK version_gen.h
	UPD version_gen.h
	 DEP /var/tmp/patchew-tester-tmp-0kkotmms/src/dtc/util.c
	 CC libfdt/fdt.o
	 CC libfdt/fdt_ro.o
	 CC libfdt/fdt_wip.o
	 CC libfdt/fdt_sw.o
	 CC libfdt/fdt_rw.o
	 CC libfdt/fdt_strerror.o
	 CC libfdt/fdt_addresses.o
	 CC libfdt/fdt_empty_tree.o
	 CC libfdt/fdt_overlay.o
	 AR libfdt/libfdt.a
ar: creating libfdt/libfdt.a
a - libfdt/fdt.o
a - libfdt/fdt_ro.o
a - libfdt/fdt_wip.o
a - libfdt/fdt_sw.o
a - libfdt/fdt_rw.o
a - libfdt/fdt_strerror.o
a - libfdt/fdt_empty_tree.o
a - libfdt/fdt_addresses.o
a - libfdt/fdt_overlay.o
  AR      libcapstone.a
ar: creating /var/tmp/patchew-tester-tmp-0kkotmms/src/build/capstone/libcapstone.a
  CC      tests/qemu-iotests/socket_scm_helper.o
  GEN     qga/qapi-generated/qga-qapi-types.h
  GEN     qga/qapi-generated/qga-qapi-visit.h
  GEN     qga/qapi-generated/qga-qmp-commands.h
  GEN     qga/qapi-generated/qga-qapi-visit.c
  GEN     qga/qapi-generated/qga-qapi-types.c
  GEN     qga/qapi-generated/qga-qmp-marshal.c
  CC      qapi-types.o
  CC      qapi-visit.o
  CC      qapi-event.o
  CC      qmp-introspect.o
  CC      qapi/qapi-visit-core.o
  CC      qapi/qapi-dealloc-visitor.o
  CC      qapi/qobject-input-visitor.o
  CC      qapi/qobject-output-visitor.o
  CC      qapi/qmp-registry.o
  CC      qapi/qmp-dispatch.o
  CC      qapi/string-input-visitor.o
  CC      qapi/string-output-visitor.o
  CC      qapi/opts-visitor.o
  CC      qapi/qapi-util.o
  CC      qobject/qnull.o
  CC      qapi/qmp-event.o
  CC      qapi/qapi-clone-visitor.o
  CC      qobject/qnum.o
  CC      qobject/qlist.o
  CC      qobject/qstring.o
  CC      qobject/qbool.o
  CC      qobject/qlit.o
  CC      qobject/qobject.o
  CC      qobject/qdict.o
  CC      qobject/qjson.o
  CC      qobject/json-lexer.o
  CC      qobject/json-streamer.o
  CC      qobject/json-parser.o
  CC      trace/control.o
  CC      trace/qmp.o
  CC      util/cutils.o
  CC      util/osdep.o
  CC      util/unicode.o
  CC      util/qemu-timer-common.o
  CC      util/lockcnt.o
  CC      util/aiocb.o
  CC      util/bufferiszero.o
  CC      util/async.o
  CC      util/thread-pool.o
  CC      util/qemu-timer.o
  CC      util/main-loop.o
  CC      util/iohandler.o
  CC      util/aio-posix.o
  CC      util/oslib-posix.o
  CC      util/event_notifier-posix.o
  CC      util/compatfd.o
  CC      util/qemu-openpty.o
  CC      util/mmap-alloc.o
  CC      util/memfd.o
  CC      util/qemu-thread-posix.o
  CC      util/path.o
  CC      util/envlist.o
  CC      util/module.o
  CC      util/host-utils.o
  CC      util/acl.o
  CC      util/bitops.o
  CC      util/error.o
  CC      util/fifo8.o
  CC      util/hbitmap.o
  CC      util/qemu-error.o
  CC      util/bitmap.o
  CC      util/cacheinfo.o
  CC      util/iov.o
  CC      util/id.o
  CC      util/qemu-config.o
  CC      util/qemu-sockets.o
  CC      util/uri.o
  CC      util/notify.o
  CC      util/qemu-option.o
  CC      util/qemu-progress.o
  CC      util/hexdump.o
  CC      util/keyval.o
  CC      util/crc32c.o
  CC      util/uuid.o
  CC      util/getauxval.o
  CC      util/throttle.o
  CC      util/readline.o
  CC      util/rcu.o
  CC      util/qemu-coroutine.o
  CC      util/qemu-coroutine-lock.o
  CC      util/qemu-coroutine-io.o
  CC      util/qemu-coroutine-sleep.o
  CC      util/coroutine-ucontext.o
  CC      util/buffer.o
  CC      util/timed-average.o
  CC      util/base64.o
  CC      util/pagesize.o
  CC      util/qdist.o
  CC      util/log.o
  CC      util/qht.o
  CC      util/range.o
  CC      util/stats64.o
  CC      util/systemd.o
  CC      trace-root.o
  CC      util/trace.o
  CC      io/trace.o
  CC      crypto/trace.o
  CC      migration/trace.o
  CC      block/trace.o
  CC      hw/block/trace.o
  CC      chardev/trace.o
  CC      hw/block/dataplane/trace.o
  CC      hw/char/trace.o
  CC      hw/intc/trace.o
  CC      hw/net/trace.o
  CC      hw/rdma/trace.o
  CC      hw/rdma/vmw/trace.o
  CC      hw/virtio/trace.o
  CC      hw/audio/trace.o
  CC      hw/misc/trace.o
  CC      hw/usb/trace.o
  CC      hw/scsi/trace.o
  CC      hw/nvram/trace.o
  CC      hw/display/trace.o
  CC      hw/input/trace.o
  CC      hw/timer/trace.o
  CC      hw/dma/trace.o
  CC      hw/sparc/trace.o
  CC      hw/sparc64/trace.o
  CC      hw/sd/trace.o
  CC      hw/isa/trace.o
  CC      hw/mem/trace.o
  CC      hw/i386/trace.o
  CC      hw/i386/xen/trace.o
  CC      hw/9pfs/trace.o
  CC      hw/ppc/trace.o
  CC      hw/s390x/trace.o
  CC      hw/pci/trace.o
  CC      hw/vfio/trace.o
  CC      hw/acpi/trace.o
  CC      hw/arm/trace.o
  CC      hw/alpha/trace.o
  CC      hw/xen/trace.o
  CC      hw/ide/trace.o
  CC      ui/trace.o
  CC      audio/trace.o
  CC      net/trace.o
  CC      target/arm/trace.o
  CC      target/i386/trace.o
  CC      target/mips/trace.o
  CC      target/sparc/trace.o
  CC      target/s390x/trace.o
  CC      target/ppc/trace.o
  CC      qom/trace.o
  CC      accel/tcg/trace.o
  CC      linux-user/trace.o
  CC      qapi/trace.o
  CC      accel/kvm/trace.o
  CC      nbd/trace.o
  CC      scsi/trace.o
  CC      stubs/arch-query-cpu-def.o
  CC      crypto/pbkdf-stub.o
  CC      stubs/arch-query-cpu-model-expansion.o
  CC      stubs/arch-query-cpu-model-comparison.o
  CC      stubs/arch-query-cpu-model-baseline.o
  CC      stubs/bdrv-next-monitor-owned.o
  CC      stubs/blk-commit-all.o
  CC      stubs/blockdev-close-all-bdrv-states.o
  CC      stubs/clock-warp.o
  CC      stubs/cpu-get-clock.o
  CC      stubs/cpu-get-icount.o
  CC      stubs/dump.o
  CC      stubs/error-printf.o
  CC      stubs/fdset.o
  CC      stubs/gdbstub.o
  CC      stubs/get-vm-name.o
  CC      stubs/iothread.o
  CC      stubs/iothread-lock.o
  CC      stubs/is-daemonized.o
  CC      stubs/linux-aio.o
  CC      stubs/machine-init-done.o
  CC      stubs/migr-blocker.o
  CC      stubs/change-state-handler.o
  CC      stubs/monitor.o
  CC      stubs/notify-event.o
  CC      stubs/qtest.o
  CC      stubs/replay.o
  CC      stubs/runstate-check.o
  CC      stubs/set-fd-handler.o
  CC      stubs/slirp.o
  CC      stubs/sysbus.o
  CC      stubs/tpm.o
  CC      stubs/trace-control.o
  CC      stubs/uuid.o
  CC      stubs/vmstate.o
  CC      stubs/vm-stop.o
  CC      stubs/qmp_pc_dimm.o
  CC      stubs/target-monitor-defs.o
  CC      stubs/target-get-monitor-def.o
  CC      stubs/pc_madt_cpu_entry.o
  CC      stubs/vmgenid.o
  CC      stubs/xen-common.o
  CC      stubs/xen-hvm.o
  CC      contrib/ivshmem-client/ivshmem-client.o
  CC      contrib/ivshmem-client/main.o
  CC      contrib/ivshmem-server/ivshmem-server.o
  CC      stubs/pci-host-piix.o
  CC      contrib/ivshmem-server/main.o
  CC      qemu-nbd.o
  CC      block.o
  CC      blockjob.o
  CC      qemu-io-cmds.o
  CC      replication.o
  CC      block/raw-format.o
  CC      block/qcow.o
  CC      block/vdi.o
  CC      block/vmdk.o
  CC      block/cloop.o
  CC      block/bochs.o
  CC      block/vpc.o
  CC      block/vvfat.o
  CC      block/dmg.o
  CC      block/qcow2.o
  CC      block/qcow2-refcount.o
  CC      block/qcow2-snapshot.o
  CC      block/qcow2-cluster.o
  CC      block/qcow2-cache.o
  CC      block/qcow2-bitmap.o
  CC      block/qed.o
  CC      block/qed-l2-cache.o
  CC      block/qed-table.o
  CC      block/qed-cluster.o
  CC      block/qed-check.o
  CC      block/vhdx.o
  CC      block/vhdx-endian.o
  CC      block/vhdx-log.o
  CC      block/quorum.o
  CC      block/parallels.o
  CC      block/blkdebug.o
  CC      block/blkverify.o
  CC      block/blkreplay.o
  CC      block/block-backend.o
  CC      block/snapshot.o
  CC      block/qapi.o
  CC      block/file-posix.o
  CC      block/linux-aio.o
  CC      block/mirror.o
  CC      block/commit.o
  CC      block/null.o
  CC      block/io.o
  CC      block/throttle-groups.o
  CC      block/nbd.o
  CC      block/nbd-client.o
  CC      block/sheepdog.o
  CC      block/accounting.o
  CC      block/write-threshold.o
  CC      block/backup.o
  CC      block/dirty-bitmap.o
  CC      block/replication.o
  CC      block/throttle.o
  CC      block/crypto.o
  CC      nbd/server.o
  CC      nbd/client.o
  CC      nbd/common.o
  CC      scsi/utils.o
  CC      scsi/pr-manager.o
  CC      scsi/pr-manager-helper.o
  CC      block/curl.o
  CC      block/dmg-bz2.o
  CC      crypto/init.o
  CC      crypto/hash.o
  CC      crypto/hash-nettle.o
  CC      crypto/hmac.o
  CC      crypto/hmac-nettle.o
  CC      crypto/aes.o
  CC      crypto/desrfb.o
  CC      crypto/tlscreds.o
  CC      crypto/tlscredsanon.o
  CC      crypto/cipher.o
  CC      crypto/tlscredsx509.o
  CC      crypto/tlssession.o
  CC      crypto/secret.o
  CC      crypto/random-gnutls.o
  CC      crypto/pbkdf.o
  CC      crypto/pbkdf-nettle.o
  CC      crypto/ivgen.o
  CC      crypto/ivgen-essiv.o
  CC      crypto/ivgen-plain.o
  CC      crypto/ivgen-plain64.o
  CC      crypto/afsplit.o
  CC      crypto/xts.o
  CC      crypto/block.o
  CC      crypto/block-qcow.o
  CC      io/channel.o
  CC      crypto/block-luks.o
  CC      io/channel-buffer.o
  CC      io/channel-command.o
  CC      io/channel-file.o
  CC      io/channel-socket.o
  CC      io/channel-tls.o
  CC      io/channel-watch.o
  CC      io/channel-util.o
  CC      io/channel-websock.o
  CC      io/dns-resolver.o
  CC      io/net-listener.o
  CC      io/task.o
  CC      qom/object.o
  CC      qom/container.o
  CC      qom/qom-qobject.o
  CC      qom/object_interfaces.o
  GEN     qemu-img-cmds.h
  CC      qemu-io.o
  CC      fsdev/virtfs-proxy-helper.o
  CC      fsdev/9p-marshal.o
  CC      fsdev/9p-iov-marshal.o
  CC      scsi/qemu-pr-helper.o
  CC      qemu-bridge-helper.o
  CC      blockdev.o
  CC      blockdev-nbd.o
  CC      bootdevice.o
  CC      iothread.o
  CC      qdev-monitor.o
  CC      device-hotplug.o
  CC      os-posix.o
  CC      bt-host.o
  CC      bt-vhci.o
  CC      dma-helpers.o
  CC      vl.o
  CC      tpm.o
  CC      device_tree.o
  CC      qmp-marshal.o
  CC      qmp.o
  CC      hmp.o
  CC      cpus-common.o
  CC      audio/audio.o
  CC      audio/noaudio.o
  CC      audio/mixeng.o
  CC      audio/wavaudio.o
  CC      audio/sdlaudio.o
  CC      audio/ossaudio.o
  CC      audio/wavcapture.o
  CC      backends/rng.o
  CC      backends/rng-egd.o
  CC      backends/rng-random.o
  CC      backends/tpm.o
  CC      backends/hostmem.o
  CC      backends/hostmem-ram.o
  CC      backends/hostmem-file.o
  CC      backends/cryptodev.o
  CC      backends/cryptodev-builtin.o
  CC      block/stream.o
  CC      chardev/msmouse.o
  CC      chardev/wctablet.o
  CC      chardev/testdev.o
  CC      disas/arm.o
  CC      disas/alpha.o
  CXX     disas/arm-a64.o
  CC      disas/cris.o
  CC      disas/hppa.o
  CC      disas/i386.o
  CC      disas/m68k.o
  CC      disas/mips.o
  CC      disas/microblaze.o
  CC      disas/nios2.o
  CC      disas/ppc.o
  CC      disas/moxie.o
  CC      disas/s390.o
  CC      disas/sh4.o
  CC      disas/sparc.o
  CC      disas/lm32.o
  CC      disas/xtensa.o
  CXX     disas/libvixl/vixl/utils.o
  CXX     disas/libvixl/vixl/compiler-intrinsics.o
  CXX     disas/libvixl/vixl/a64/instructions-a64.o
  CXX     disas/libvixl/vixl/a64/decoder-a64.o
  CXX     disas/libvixl/vixl/a64/disasm-a64.o
  CC      fsdev/qemu-fsdev.o
  CC      fsdev/qemu-fsdev-opts.o
  CC      fsdev/qemu-fsdev-throttle.o
  CC      fsdev/qemu-fsdev-dummy.o
  CC      hw/9pfs/9p.o
  CC      hw/9pfs/9p-util.o
  CC      hw/9pfs/9p-local.o
  CC      hw/9pfs/9p-xattr.o
  CC      hw/9pfs/9p-posix-acl.o
  CC      hw/9pfs/9p-xattr-user.o
  CC      hw/9pfs/cofs.o
  CC      hw/9pfs/coth.o
  CC      hw/9pfs/codir.o
  CC      hw/9pfs/cofile.o
  CC      hw/9pfs/coxattr.o
  CC      hw/9pfs/9p-synth.o
  CC      hw/9pfs/9p-handle.o
  CC      hw/9pfs/9p-proxy.o
  CC      hw/acpi/core.o
  CC      hw/acpi/piix4.o
  CC      hw/acpi/pcihp.o
  CC      hw/acpi/ich9.o
  CC      hw/acpi/tco.o
  CC      hw/acpi/cpu_hotplug.o
  CC      hw/acpi/memory_hotplug.o
  CC      hw/acpi/cpu.o
  CC      hw/acpi/nvdimm.o
  CC      hw/acpi/vmgenid.o
  CC      hw/acpi/acpi_interface.o
  CC      hw/acpi/bios-linker-loader.o
  CC      hw/acpi/aml-build.o
  CC      hw/acpi/ipmi.o
  CC      hw/acpi/acpi-stub.o
  CC      hw/acpi/ipmi-stub.o
  CC      hw/audio/sb16.o
  CC      hw/audio/es1370.o
  CC      hw/audio/fmopl.o
  CC      hw/audio/ac97.o
  CC      hw/audio/adlib.o
  CC      hw/audio/gus.o
  CC      hw/audio/gusemu_hal.o
  CC      hw/audio/gusemu_mixer.o
  CC      hw/audio/cs4231a.o
  CC      hw/audio/intel-hda.o
  CC      hw/audio/hda-codec.o
  CC      hw/audio/pcspk.o
  CC      hw/audio/wm8750.o
  CC      hw/audio/pl041.o
  CC      hw/audio/lm4549.o
  CC      hw/audio/cs4231.o
  CC      hw/audio/marvell_88w8618.o
  CC      hw/audio/milkymist-ac97.o
  CC      hw/audio/soundhw.o
  CC      hw/block/block.o
  CC      hw/block/cdrom.o
  CC      hw/block/hd-geometry.o
  CC      hw/block/fdc.o
  CC      hw/block/m25p80.o
  CC      hw/block/nand.o
  CC      hw/block/pflash_cfi01.o
  CC      hw/block/ecc.o
  CC      hw/block/pflash_cfi02.o
  CC      hw/block/onenand.o
  CC      hw/block/nvme.o
  CC      hw/bt/core.o
  CC      hw/bt/l2cap.o
  CC      hw/bt/sdp.o
  CC      hw/bt/hci.o
  CC      hw/bt/hid.o
  CC      hw/bt/hci-csr.o
  CC      hw/char/ipoctal232.o
  CC      hw/char/escc.o
  CC      hw/char/parallel.o
  CC      hw/char/pl011.o
  CC      hw/char/serial.o
  CC      hw/char/serial-isa.o
  CC      hw/char/serial-pci.o
  CC      hw/char/virtio-console.o
  CC      hw/char/xilinx_uartlite.o
  CC      hw/char/cadence_uart.o
  CC      hw/char/cmsdk-apb-uart.o
  CC      hw/char/etraxfs_ser.o
  CC      hw/char/debugcon.o
  CC      hw/char/grlib_apbuart.o
  CC      hw/char/imx_serial.o
  CC      hw/char/lm32_juart.o
  CC      hw/char/lm32_uart.o
  CC      hw/char/milkymist-uart.o
  CC      hw/char/sclpconsole.o
  CC      hw/char/sclpconsole-lm.o
  CC      hw/core/qdev.o
  CC      hw/core/qdev-properties.o
  CC      hw/core/bus.o
  CC      hw/core/reset.o
  CC      hw/core/fw-path-provider.o
  CC      hw/core/irq.o
  CC      hw/core/hotplug.o
  CC      hw/core/nmi.o
  CC      hw/core/empty_slot.o
  CC      hw/core/stream.o
  CC      hw/core/ptimer.o
  CC      hw/core/sysbus.o
  CC      hw/core/loader-fit.o
  CC      hw/core/loader.o
  CC      hw/core/machine.o
  CC      hw/core/qdev-properties-system.o
  CC      hw/core/register.o
  CC      hw/core/or-irq.o
  CC      hw/core/platform-bus.o
  CC      hw/cpu/core.o
  CC      hw/display/ads7846.o
  CC      hw/display/cirrus_vga.o
  CC      hw/display/g364fb.o
  CC      hw/display/jazz_led.o
  CC      hw/display/pl110.o
  CC      hw/display/ssd0303.o
  CC      hw/display/ssd0323.o
  CC      hw/display/vga-pci.o
  CC      hw/display/vga-isa.o
  CC      hw/display/vga-isa-mm.o
  CC      hw/display/vmware_vga.o
  CC      hw/display/blizzard.o
  CC      hw/display/exynos4210_fimd.o
  CC      hw/display/milkymist-vgafb.o
  CC      hw/display/framebuffer.o
  CC      hw/display/tc6393xb.o
  CC      hw/dma/pl080.o
  CC      hw/dma/puv3_dma.o
  CC      hw/dma/i82374.o
  CC      hw/dma/rc4030.o
  CC      hw/dma/pl330.o
  CC      hw/dma/i8257.o
  CC      hw/dma/xilinx_axidma.o
  CC      hw/dma/xlnx-zynq-devcfg.o
  CC      hw/dma/sparc32_dma.o
  CC      hw/dma/etraxfs_dma.o
  CC      hw/gpio/max7310.o
  CC      hw/gpio/pl061.o
  CC      hw/gpio/puv3_gpio.o
  CC      hw/gpio/zaurus.o
  CC      hw/gpio/mpc8xxx.o
  CC      hw/gpio/gpio_key.o
  CC      hw/i2c/core.o
  CC      hw/i2c/smbus.o
  CC      hw/i2c/smbus_eeprom.o
  CC      hw/i2c/i2c-ddc.o
  CC      hw/i2c/smbus_ich9.o
  CC      hw/i2c/pm_smbus.o
  CC      hw/i2c/versatile_i2c.o
  CC      hw/i2c/bitbang_i2c.o
  CC      hw/i2c/exynos4210_i2c.o
  CC      hw/i2c/imx_i2c.o
  CC      hw/i2c/aspeed_i2c.o
  CC      hw/ide/core.o
  CC      hw/ide/atapi.o
  CC      hw/ide/qdev.o
  CC      hw/ide/pci.o
  CC      hw/ide/isa.o
  CC      hw/ide/piix.o
  CC      hw/ide/cmd646.o
  CC      hw/ide/macio.o
  CC      hw/ide/mmio.o
  CC      hw/ide/via.o
  CC      hw/ide/microdrive.o
  CC      hw/ide/ahci.o
  CC      hw/ide/ich.o
  CC      hw/ide/ahci-allwinner.o
  CC      hw/ide/sii3112.o
  CC      hw/input/adb.o
  CC      hw/input/adb-mouse.o
  CC      hw/input/adb-kbd.o
  CC      hw/input/hid.o
  CC      hw/input/lm832x.o
  CC      hw/input/pckbd.o
  CC      hw/input/pl050.o
  CC      hw/input/ps2.o
  CC      hw/input/stellaris_input.o
  CC      hw/input/tsc2005.o
  CC      hw/input/virtio-input.o
  CC      hw/input/virtio-input-hid.o
  CC      hw/input/virtio-input-host.o
  CC      hw/intc/heathrow_pic.o
  CC      hw/intc/i8259_common.o
  CC      hw/intc/i8259.o
  CC      hw/intc/pl190.o
  CC      hw/intc/puv3_intc.o
  CC      hw/intc/xilinx_intc.o
  CC      hw/intc/etraxfs_pic.o
  CC      hw/intc/imx_avic.o
  CC      hw/intc/lm32_pic.o
  CC      hw/intc/realview_gic.o
  CC      hw/intc/slavio_intctl.o
  CC      hw/intc/ioapic_common.o
  CC      hw/intc/arm_gic_common.o
  CC      hw/intc/arm_gicv2m.o
  CC      hw/intc/arm_gic.o
  CC      hw/intc/arm_gicv3_common.o
  CC      hw/intc/arm_gicv3.o
  CC      hw/intc/arm_gicv3_dist.o
  CC      hw/intc/arm_gicv3_redist.o
  CC      hw/intc/arm_gicv3_its_common.o
  CC      hw/intc/openpic.o
  CC      hw/intc/intc.o
  CC      hw/ipack/ipack.o
  CC      hw/ipack/tpci200.o
  CC      hw/ipmi/ipmi.o
  CC      hw/ipmi/ipmi_bmc_sim.o
  CC      hw/ipmi/ipmi_bmc_extern.o
  CC      hw/ipmi/isa_ipmi_kcs.o
  CC      hw/ipmi/isa_ipmi_bt.o
  CC      hw/isa/isa-bus.o
  CC      hw/isa/apm.o
  CC      hw/isa/i82378.o
  CC      hw/isa/pc87312.o
  CC      hw/isa/piix4.o
  CC      hw/isa/vt82c686.o
  CC      hw/mem/pc-dimm.o
  CC      hw/mem/nvdimm.o
  CC      hw/misc/applesmc.o
  CC      hw/misc/max111x.o
  CC      hw/misc/tmp105.o
  CC      hw/misc/tmp421.o
  CC      hw/misc/debugexit.o
  CC      hw/misc/sga.o
  CC      hw/misc/pc-testdev.o
  CC      hw/misc/pci-testdev.o
  CC      hw/misc/edu.o
  CC      hw/misc/unimp.o
  CC      hw/misc/vmcoreinfo.o
  CC      hw/misc/arm_l2x0.o
  CC      hw/misc/arm_integrator_debug.o
  CC      hw/misc/a9scu.o
  CC      hw/misc/arm11scu.o
  CC      hw/misc/puv3_pm.o
  CC      hw/misc/macio/macio.o
  CC      hw/misc/macio/cuda.o
  CC      hw/misc/macio/mac_dbdma.o
  CC      hw/net/dp8393x.o
  CC      hw/net/ne2000.o
  CC      hw/net/eepro100.o
  CC      hw/net/pcnet-pci.o
  CC      hw/net/pcnet.o
  CC      hw/net/e1000.o
  CC      hw/net/e1000x_common.o
  CC      hw/net/net_tx_pkt.o
  CC      hw/net/net_rx_pkt.o
  CC      hw/net/e1000e.o
  CC      hw/net/e1000e_core.o
  CC      hw/net/rtl8139.o
  CC      hw/net/vmxnet3.o
  CC      hw/net/smc91c111.o
  CC      hw/net/ne2000-isa.o
  CC      hw/net/lan9118.o
  CC      hw/net/opencores_eth.o
  CC      hw/net/xgmac.o
  CC      hw/net/xilinx_axienet.o
  CC      hw/net/mipsnet.o
  CC      hw/net/imx_fec.o
  CC      hw/net/allwinner_emac.o
  CC      hw/net/cadence_gem.o
  CC      hw/net/stellaris_enet.o
  CC      hw/net/lance.o
  CC      hw/net/sunhme.o
  CC      hw/net/ftgmac100.o
  CC      hw/net/sungem.o
  CC      hw/net/rocker/rocker.o
  CC      hw/net/rocker/rocker_fp.o
  CC      hw/net/rocker/rocker_desc.o
  CC      hw/net/rocker/rocker_world.o
  CC      hw/net/rocker/rocker_of_dpa.o
  CC      hw/nvram/ds1225y.o
  CC      hw/nvram/eeprom93xx.o
  CC      hw/nvram/eeprom_at24c.o
  CC      hw/nvram/fw_cfg.o
  CC      hw/nvram/chrp_nvram.o
  CC      hw/pci-bridge/pci_bridge_dev.o
  CC      hw/nvram/mac_nvram.o
  CC      hw/pci-bridge/pcie_root_port.o
  CC      hw/pci-bridge/pcie_pci_bridge.o
  CC      hw/pci-bridge/gen_pcie_root_port.o
  CC      hw/pci-bridge/pci_expander_bridge.o
  CC      hw/pci-bridge/xio3130_upstream.o
  CC      hw/pci-bridge/xio3130_downstream.o
  CC      hw/pci-bridge/ioh3420.o
  CC      hw/pci-bridge/i82801b11.o
  CC      hw/pci-bridge/dec.o
  CC      hw/pci-host/pam.o
  CC      hw/pci-host/prep.o
  CC      hw/pci-host/grackle.o
  CC      hw/pci-host/uninorth.o
  CC      hw/pci-host/ppce500.o
  CC      hw/pci-host/versatile.o
  CC      hw/pci-host/apb.o
  CC      hw/pci-host/bonito.o
  CC      hw/pci-host/piix.o
  CC      hw/pci-host/q35.o
  CC      hw/pci-host/gpex.o
  CC      hw/pci-host/xilinx-pcie.o
  CC      hw/pci/pci.o
  CC      hw/pci/pci_bridge.o
  CC      hw/pci/msix.o
  CC      hw/pci/msi.o
  CC      hw/pci/shpc.o
  CC      hw/pci/slotid_cap.o
  CC      hw/pci/pci_host.o
  CC      hw/pci/pcie_host.o
  CC      hw/pci/pcie.o
  CC      hw/pci/pcie_aer.o
  CC      hw/pci/pcie_port.o
  CC      hw/pci/pci-stub.o
  CC      hw/pcmcia/pcmcia.o
  CC      hw/scsi/scsi-disk.o
  CC      hw/scsi/scsi-generic.o
  CC      hw/scsi/scsi-bus.o
  CC      hw/scsi/lsi53c895a.o
  CC      hw/scsi/mptsas.o
  CC      hw/scsi/mptconfig.o
  CC      hw/scsi/mptendian.o
  CC      hw/scsi/megasas.o
  CC      hw/scsi/vmw_pvscsi.o
  CC      hw/scsi/esp.o
  CC      hw/scsi/esp-pci.o
  CC      hw/sd/pl181.o
  CC      hw/sd/ssi-sd.o
  CC      hw/sd/sd.o
  CC      hw/sd/core.o
  CC      hw/sd/sdhci.o
  CC      hw/smbios/smbios.o
  CC      hw/smbios/smbios_type_38.o
  CC      hw/smbios/smbios-stub.o
  CC      hw/smbios/smbios_type_38-stub.o
  CC      hw/ssi/pl022.o
  CC      hw/ssi/ssi.o
  CC      hw/ssi/xilinx_spi.o
  CC      hw/ssi/xilinx_spips.o
  CC      hw/ssi/aspeed_smc.o
  CC      hw/ssi/stm32f2xx_spi.o
  CC      hw/ssi/mss-spi.o
  CC      hw/timer/arm_timer.o
  CC      hw/timer/arm_mptimer.o
  CC      hw/timer/armv7m_systick.o
  CC      hw/timer/a9gtimer.o
  CC      hw/timer/cadence_ttc.o
  CC      hw/timer/ds1338.o
  CC      hw/timer/hpet.o
  CC      hw/timer/i8254_common.o
  CC      hw/timer/i8254.o
  CC      hw/timer/m48t59.o
  CC      hw/timer/m48t59-isa.o
  CC      hw/timer/pl031.o
  CC      hw/timer/puv3_ost.o
  CC      hw/timer/twl92230.o
  CC      hw/timer/xilinx_timer.o
  CC      hw/timer/slavio_timer.o
  CC      hw/timer/etraxfs_timer.o
  CC      hw/timer/grlib_gptimer.o
  CC      hw/timer/imx_epit.o
  CC      hw/timer/imx_gpt.o
  CC      hw/timer/lm32_timer.o
  CC      hw/timer/milkymist-sysctl.o
  CC      hw/timer/stm32f2xx_timer.o
  CC      hw/timer/aspeed_timer.o
  CC      hw/timer/sun4v-rtc.o
  CC      hw/timer/cmsdk-apb-timer.o
  CC      hw/timer/mss-timer.o
  CC      hw/tpm/tpm_util.o
  CC      hw/tpm/tpm_tis.o
  CC      hw/tpm/tpm_emulator.o
  CC      hw/usb/core.o
  CC      hw/usb/combined-packet.o
  CC      hw/usb/bus.o
  CC      hw/usb/libhw.o
  CC      hw/usb/desc.o
  CC      hw/usb/desc-msos.o
  CC      hw/usb/hcd-uhci.o
  CC      hw/usb/hcd-ohci.o
  CC      hw/usb/hcd-ehci.o
  CC      hw/usb/hcd-ehci-pci.o
  CC      hw/usb/hcd-ehci-sysbus.o
  CC      hw/usb/hcd-xhci.o
  CC      hw/usb/hcd-xhci-nec.o
  CC      hw/usb/hcd-musb.o
  CC      hw/usb/dev-hub.o
  CC      hw/usb/dev-hid.o
  CC      hw/usb/dev-wacom.o
  CC      hw/usb/dev-storage.o
  CC      hw/usb/dev-uas.o
  CC      hw/usb/dev-audio.o
  CC      hw/usb/dev-network.o
  CC      hw/usb/dev-serial.o
  CC      hw/usb/dev-bluetooth.o
  CC      hw/usb/dev-smartcard-reader.o
  CC      hw/usb/dev-mtp.o
  CC      hw/usb/host-libusb.o
  CC      hw/usb/host-legacy.o
  CC      hw/virtio/virtio-rng.o
  CC      hw/usb/host-stub.o
  CC      hw/virtio/virtio-pci.o
  CC      hw/virtio/virtio-bus.o
  CC      hw/virtio/virtio-mmio.o
  CC      hw/virtio/vhost-stub.o
  CC      hw/watchdog/watchdog.o
  CC      hw/watchdog/wdt_i6300esb.o
  CC      hw/watchdog/wdt_ib700.o
  CC      hw/watchdog/wdt_diag288.o
  CC      hw/watchdog/wdt_aspeed.o
  CC      migration/migration.o
  CC      migration/socket.o
  CC      migration/fd.o
  CC      migration/exec.o
  CC      migration/tls.o
  CC      migration/channel.o
  CC      migration/savevm.o
  CC      migration/colo-comm.o
  CC      migration/colo.o
  CC      migration/colo-failover.o
  CC      migration/vmstate-types.o
  CC      migration/vmstate.o
  CC      migration/page_cache.o
  CC      migration/qemu-file.o
  CC      migration/qemu-file-channel.o
  CC      migration/global_state.o
  CC      migration/xbzrle.o
  CC      migration/postcopy-ram.o
  CC      migration/qjson.o
  CC      migration/rdma.o
  CC      migration/block.o
  CC      net/net.o
  CC      net/queue.o
  CC      net/checksum.o
  CC      net/util.o
  CC      net/hub.o
  CC      net/socket.o
  CC      net/dump.o
  CC      net/eth.o
  CC      net/l2tpv3.o
  CC      net/vhost-user.o
  CC      net/slirp.o
  CC      net/filter.o
  CC      net/filter-buffer.o
  CC      net/filter-mirror.o
  CC      net/colo-compare.o
  CC      net/colo.o
  CC      net/filter-rewriter.o
  CC      net/filter-replay.o
  CC      net/tap.o
  CC      net/tap-linux.o
  CC      qom/cpu.o
  CC      replay/replay.o
  CC      replay/replay-internal.o
  CC      replay/replay-events.o
  CC      replay/replay-time.o
  CC      replay/replay-input.o
  CC      replay/replay-char.o
  CC      replay/replay-snapshot.o
  CC      replay/replay-net.o
  CC      replay/replay-audio.o
  CC      slirp/cksum.o
  CC      slirp/if.o
  CC      slirp/ip_icmp.o
  CC      slirp/ip6_icmp.o
  CC      slirp/ip6_input.o
  CC      slirp/ip6_output.o
  CC      slirp/ip_input.o
  CC      slirp/ip_output.o
  CC      slirp/dnssearch.o
  CC      slirp/dhcpv6.o
  CC      slirp/slirp.o
  CC      slirp/mbuf.o
  CC      slirp/misc.o
  CC      slirp/sbuf.o
  CC      slirp/socket.o
  CC      slirp/tcp_input.o
  CC      slirp/tcp_output.o
  CC      slirp/tcp_subr.o
  CC      slirp/tcp_timer.o
  CC      slirp/udp.o
  CC      slirp/udp6.o
  CC      slirp/bootp.o
  CC      slirp/tftp.o
  CC      slirp/arp_table.o
  CC      slirp/ndp_table.o
  CC      slirp/ncsi.o
  CC      ui/keymaps.o
  CC      ui/console.o
  CC      ui/cursor.o
  CC      ui/qemu-pixman.o
  CC      ui/input.o
  CC      ui/input-keymap.o
  CC      ui/input-legacy.o
  CC      ui/input-linux.o
  CC      ui/sdl.o
  CC      ui/sdl_zoom.o
  CC      ui/x_keymap.o
  CC      ui/curses.o
  CC      ui/vnc.o
  CC      ui/vnc-enc-zlib.o
  CC      ui/vnc-enc-hextile.o
  CC      ui/vnc-enc-tight.o
  CC      ui/vnc-palette.o
  CC      ui/vnc-enc-zrle.o
  CC      ui/vnc-auth-vencrypt.o
  CC      ui/vnc-auth-sasl.o
  CC      ui/vnc-ws.o
  CC      ui/vnc-jobs.o
  CC      ui/gtk.o
  CC      chardev/char.o
  CC      chardev/char-fd.o
  CC      chardev/char-fe.o
  CC      chardev/char-file.o
  CC      chardev/char-io.o
  CC      chardev/char-mux.o
  CC      chardev/char-null.o
  CC      chardev/char-parallel.o
  CC      chardev/char-pipe.o
  CC      chardev/char-ringbuf.o
  CC      chardev/char-pty.o
  CC      chardev/char-serial.o
  CC      chardev/char-socket.o
  CC      chardev/char-stdio.o
  CC      chardev/char-udp.o
  CCAS    spapr-rtas/spapr-rtas.o
  LINK    tests/qemu-iotests/socket_scm_helper
  CC      qga/commands.o
  Building spapr-rtas/spapr-rtas.img
  CC      qga/guest-agent-command-state.o
  CC      qga/main.o
  Building spapr-rtas/spapr-rtas.bin
  CC      qga/channel-posix.o
  CC      qga/commands-posix.o
  CC      qga/qapi-generated/qga-qapi-types.o
  CC      qga/qapi-generated/qga-qapi-visit.o
  CC      qga/qapi-generated/qga-qmp-marshal.o
  CC      qemu-img.o
  AR      libqemuutil.a
  LINK    qemu-ga
  LINK    ivshmem-client
  LINK    ivshmem-server
  LINK    qemu-nbd
  LINK    qemu-img
  LINK    qemu-io
  LINK    fsdev/virtfs-proxy-helper
  LINK    scsi/qemu-pr-helper
  LINK    qemu-bridge-helper
  GEN     i386-linux-user/config-target.h
  CC      i386-linux-user/exec.o
  CC      i386-linux-user/tcg/tcg.o
  CC      i386-linux-user/tcg/optimize.o
  CC      i386-linux-user/tcg/tcg-op.o
  CC      i386-linux-user/fpu/softfloat.o
  CC      i386-linux-user/disas.o
  CC      i386-linux-user/gdbstub.o
  GEN     i386-linux-user/gdbstub-xml.c
  CC      i386-linux-user/tcg/tcg-common.o
  GEN     cris-linux-user/config-target.h
  CC      i386-linux-user/accel/stubs/hax-stub.o
  CC      i386-linux-user/accel/stubs/hvf-stub.o
  CC      i386-linux-user/accel/stubs/kvm-stub.o
  CC      i386-linux-user/thunk.o
  CC      i386-linux-user/accel/tcg/tcg-runtime.o
  CC      i386-linux-user/accel/tcg/cpu-exec.o
  CC      i386-linux-user/accel/tcg/cpu-exec-common.o
  CC      i386-linux-user/accel/tcg/translate-all.o
  CC      i386-linux-user/accel/tcg/translator.o
  GEN     arm-linux-user/config-target.h
  GEN     armeb-linux-user/config-target.h
  GEN     ppc64-linux-user/config-target.h
  CC      i386-linux-user/accel/tcg/user-exec.o
  GEN     microblazeel-linux-user/config-target.h
  CC      i386-linux-user/accel/tcg/user-exec-stub.o
  CC      i386-linux-user/linux-user/strace.o
  CC      i386-linux-user/linux-user/mmap.o
  GEN     m68k-linux-user/config-target.h
  CC      i386-linux-user/linux-user/signal.o
  GEN     mipsel-linux-user/config-target.h
  GEN     aarch64_be-linux-user/config-target.h
  GEN     nios2-linux-user/config-target.h
  CC      i386-linux-user/linux-user/main.o
  CC      cris-linux-user/tcg/tcg-op.o
  CC      cris-linux-user/exec.o
  CC      cris-linux-user/accel/stubs/hvf-stub.o
  CC      i386-linux-user/linux-user/elfload.o
  CC      i386-linux-user/linux-user/syscall.o
  CC      cris-linux-user/tcg/tcg.o
  CC      cris-linux-user/tcg/optimize.o
  CC      cris-linux-user/tcg/tcg-common.o
  CC      cris-linux-user/disas.o
  CC      cris-linux-user/gdbstub.o
  CC      cris-linux-user/accel/stubs/hax-stub.o
  CC      cris-linux-user/fpu/softfloat.o
  GEN     mips64el-linux-user/config-target.h
  GEN     microblaze-linux-user/config-target.h
  CC      i386-linux-user/linux-user/linuxload.o
  CC      cris-linux-user/accel/stubs/kvm-stub.o
  CC      cris-linux-user/thunk.o
  CC      cris-linux-user/accel/tcg/cpu-exec.o
  CC      microblazeel-linux-user/exec.o
  CC      cris-linux-user/accel/tcg/tcg-runtime.o
  CC      microblazeel-linux-user/tcg/tcg.o
  GEN     hppa-linux-user/config-target.h
  GEN     ppc64le-linux-user/config-target.h
  CC      ppc64-linux-user/exec.o
  CC      arm-linux-user/exec.o
  GEN     aarch64-linux-user/config-target.h
  CC      armeb-linux-user/exec.o
  GEN     alpha-linux-user/config-target.h
  GEN     or1k-linux-user/config-target.h
  CC      aarch64_be-linux-user/exec.o
  GEN     mipsn32el-linux-user/config-target.h
  CC      m68k-linux-user/exec.o
  GEN     mipsn32-linux-user/config-target.h
  GEN     sh4-linux-user/config-target.h
  GEN     mips-linux-user/config-target.h
  CC      nios2-linux-user/exec.o
  CC      aarch64_be-linux-user/tcg/tcg.o
  GEN     sparc64-linux-user/config-target.h
  GEN     ppc64abi32-linux-user/config-target.h
  CC      ppc64le-linux-user/exec.o
  CC      s390x-linux-user/gen-features
  CC      mips64el-linux-user/exec.o
  CC      mipsel-linux-user/exec.o
  GEN     sparc32plus-linux-user/config-target.h
  CC      hppa-linux-user/exec.o
  GEN     sh4eb-linux-user/config-target.h
  CC      aarch64-linux-user/exec.o
  GEN     ppc-linux-user/config-target.h
  CC      microblaze-linux-user/exec.o
  GEN     sparc-linux-user/config-target.h
  CC      or1k-linux-user/exec.o
  GEN     tilegx-linux-user/config-target.h
  CC      alpha-linux-user/exec.o
  CC      sparc64-linux-user/exec.o
  CC      sparc32plus-linux-user/exec.o
  CC      mips-linux-user/exec.o
  GEN     mips64-linux-user/config-target.h
  CC      ppc64abi32-linux-user/exec.o
  CC      mipsn32-linux-user/exec.o
  CC      mipsn32el-linux-user/exec.o
  CC      sh4-linux-user/exec.o
  CC      sparc-linux-user/exec.o
  CC      sh4eb-linux-user/exec.o
  CC      tilegx-linux-user/exec.o
  CC      mips64-linux-user/exec.o
  CC      ppc-linux-user/exec.o
  GEN     x86_64-linux-user/config-target.h
  CC      x86_64-linux-user/exec.o
  CC      mips64-linux-user/tcg/tcg.o
  GEN     s390x-linux-user/config-target.h
  GEN     moxie-softmmu/hmp-commands.h
  CC      x86_64-linux-user/tcg/tcg.o
  CC      mips64-linux-user/tcg/tcg-op.o
  GEN     s390x-linux-user/gen-features.h
  GEN     moxie-softmmu/hmp-commands-info.h
  CC      x86_64-linux-user/tcg/tcg-op.o
  CC      ppc64abi32-linux-user/tcg/tcg.o
  CC      or1k-linux-user/tcg/tcg.o
  CC      sh4-linux-user/tcg/tcg.o
  CC      s390x-linux-user/exec.o
  CC      ppc64abi32-linux-user/tcg/tcg-op.o
  CC      ppc64le-linux-user/tcg/tcg.o
  CC      m68k-linux-user/tcg/tcg.o
  GEN     moxie-softmmu/config-target.h
  CC      moxie-softmmu/exec.o
  CC      sh4-linux-user/tcg/tcg-op.o
  GEN     m68k-softmmu/hmp-commands.h
  GEN     cris-softmmu/hmp-commands.h
  GEN     nios2-softmmu/hmp-commands.h
  GEN     tricore-softmmu/hmp-commands.h
  CC      sparc-linux-user/tcg/tcg.o
  CC      mips64el-linux-user/tcg/tcg.o
  GEN     xtensaeb-softmmu/hmp-commands.h
  GEN     microblazeel-softmmu/hmp-commands.h
  GEN     xtensa-softmmu/hmp-commands.h
  CC      ppc64le-linux-user/tcg/tcg-op.o
  CC      armeb-linux-user/tcg/tcg.o
  GEN     lm32-softmmu/hmp-commands.h
  CC      mips-linux-user/tcg/tcg.o
  CC      ppc64abi32-linux-user/tcg/optimize.o
  GEN     or1k-softmmu/hmp-commands.h
  GEN     m68k-softmmu/hmp-commands-info.h
  CC      mipsn32el-linux-user/tcg/tcg.o
  GEN     cris-softmmu/hmp-commands-info.h
  CC      arm-linux-user/tcg/tcg.o
  CC      s390x-linux-user/tcg/tcg.o
  GEN     nios2-softmmu/hmp-commands-info.h
  CC      nios2-linux-user/tcg/tcg.o
  CC      microblaze-linux-user/tcg/tcg.o
  CC      alpha-linux-user/tcg/tcg.o
  GEN     xtensaeb-softmmu/hmp-commands-info.h
  CC      tilegx-linux-user/tcg/tcg.o
  CC      ppc64-linux-user/tcg/tcg.o
  CC      moxie-softmmu/tcg/tcg.o
  GEN     microblaze-softmmu/hmp-commands.h
  CC      nios2-linux-user/tcg/tcg-op.o
  GEN     tricore-softmmu/hmp-commands-info.h
  CC      aarch64-linux-user/tcg/tcg.o
  CC      i386-linux-user/linux-user/uaccess.o
  CC      ppc64abi32-linux-user/tcg/tcg-common.o
  GEN     microblazeel-softmmu/hmp-commands-info.h
  CC      m68k-linux-user/tcg/tcg-op.o
  GEN     mipsel-softmmu/hmp-commands.h
  CC      sparc64-linux-user/tcg/tcg.o
  GEN     microblazeel-softmmu/config-target.h
  GEN     m68k-softmmu/config-target.h
  CC      mipsn32-linux-user/tcg/tcg.o
  CC      cris-linux-user/accel/tcg/cpu-exec-common.o
  CC      m68k-linux-user/tcg/optimize.o
  GEN     cris-softmmu/config-target.h
  GEN     xtensa-softmmu/hmp-commands-info.h
  GEN     nios2-softmmu/config-target.h
  GEN     lm32-softmmu/hmp-commands-info.h
  GEN     sparc-softmmu/hmp-commands.h
  GEN     s390x-softmmu/hmp-commands.h
  CC      hppa-linux-user/tcg/tcg.o
  GEN     sparc-softmmu/hmp-commands-info.h
  CC      sparc32plus-linux-user/tcg/tcg.o
  CC      sh4-linux-user/tcg/optimize.o
  CC      cris-linux-user/accel/tcg/translate-all.o
  CC      m68k-softmmu/exec.o
  GEN     xtensaeb-softmmu/config-target.h
  CC      cris-softmmu/exec.o
  GEN     mips64-softmmu/hmp-commands.h
  CC      cris-softmmu/tcg/tcg.o
  GEN     unicore32-softmmu/hmp-commands.h
  GEN     or1k-softmmu/hmp-commands-info.h
  CC      cris-linux-user/accel/tcg/translator.o
  CC      nios2-softmmu/exec.o
  GEN     lm32-softmmu/config-target.h
  GEN     tricore-softmmu/config-target.h
  GEN     xtensa-softmmu/config-target.h
  CC      mipsel-linux-user/tcg/tcg.o
  CC      sh4eb-linux-user/tcg/tcg.o
  GEN     microblaze-softmmu/hmp-commands-info.h
  GEN     sh4-softmmu/hmp-commands.h
  GEN     mips64el-softmmu/hmp-commands.h
  GEN     or1k-softmmu/config-target.h
  CC      xtensaeb-softmmu/exec.o
  CC      microblazeel-softmmu/exec.o
  GEN     sh4eb-softmmu/hmp-commands.h
  CC      tricore-softmmu/exec.o
  CC      xtensa-softmmu/exec.o
  GEN     sparc-softmmu/config-target.h
  CC      nios2-linux-user/tcg/optimize.o
  GEN     ppcemb-softmmu/hmp-commands.h
  CC      mips64el-linux-user/tcg/tcg-op.o
  CC      hppa-linux-user/tcg/tcg-op.o
  GEN     mips64-softmmu/hmp-commands-info.h
  CC      lm32-softmmu/exec.o
  CC      lm32-softmmu/tcg/tcg.o
  CC      ppc-linux-user/tcg/tcg.o
  CC      microblaze-linux-user/tcg/tcg-op.o
  CC      sparc-softmmu/exec.o
  CC      sparc64-linux-user/tcg/tcg-op.o
  GEN     mipsel-softmmu/hmp-commands-info.h
  CC      or1k-softmmu/exec.o
  CC      m68k-softmmu/tcg/tcg.o
  GEN     s390x-softmmu/hmp-commands-info.h
  GEN     sparc64-softmmu/hmp-commands.h
  GEN     microblaze-softmmu/config-target.h
  CC      microblazeel-softmmu/tcg/tcg.o
  GEN     sh4eb-softmmu/hmp-commands-info.h
  GEN     arm-softmmu/hmp-commands.h
  GEN     mipsel-softmmu/config-target.h
  GEN     ppcemb-softmmu/hmp-commands-info.h
  GEN     mips64-softmmu/config-target.h
  CC      microblazeel-softmmu/tcg/tcg-op.o
  CC      microblaze-softmmu/exec.o
  CC      m68k-linux-user/tcg/tcg-common.o
  GEN     ppc64-softmmu/hmp-commands.h
  GEN     unicore32-softmmu/hmp-commands-info.h
  CC      s390x-softmmu/gen-features
  GEN     mips64el-softmmu/hmp-commands-info.h
  CC      mips64-softmmu/exec.o
  GEN     sh4-softmmu/hmp-commands-info.h
  CC      mipsel-softmmu/exec.o
  GEN     ppc-softmmu/hmp-commands.h
  GEN     alpha-softmmu/hmp-commands.h
  CC      nios2-linux-user/tcg/tcg-common.o
  GEN     ppcemb-softmmu/config-target.h
  GEN     unicore32-softmmu/config-target.h
  GEN     sh4eb-softmmu/config-target.h
  GEN     x86_64-softmmu/hmp-commands.h
  GEN     mips-softmmu/hmp-commands.h
  CC      sh4eb-softmmu/exec.o
  GEN     aarch64-softmmu/hmp-commands.h
  CC      ppcemb-softmmu/exec.o
  GEN     sh4-softmmu/config-target.h
  GEN     arm-softmmu/hmp-commands-info.h
  CC      unicore32-softmmu/exec.o
  GEN     mips64el-softmmu/config-target.h
  GEN     sparc64-softmmu/hmp-commands-info.h
  CC      sh4-softmmu/exec.o
  GEN     ppc64-softmmu/hmp-commands-info.h
  CC      mips64el-softmmu/exec.o
  GEN     s390x-softmmu/config-target.h
  GEN     ppc-softmmu/hmp-commands-info.h
  GEN     s390x-softmmu/gen-features.h
  CC      unicore32-softmmu/tcg/tcg.o
  GEN     sparc64-softmmu/config-target.h
  CC      sparc64-softmmu/exec.o
  CC      s390x-softmmu/exec.o
  GEN     arm-softmmu/config-target.h
  GEN     ppc64-softmmu/config-target.h
  GEN     x86_64-softmmu/hmp-commands-info.h
  CC      microblaze-softmmu/tcg/tcg.o
  CC      ppc64-softmmu/exec.o
  CC      arm-softmmu/exec.o
  GEN     i386-softmmu/hmp-commands.h
  GEN     alpha-softmmu/hmp-commands-info.h
  GEN     mips-softmmu/hmp-commands-info.h
  GEN     ppc-softmmu/config-target.h
  CC      arm-softmmu/tcg/tcg.o
  CC      ppc-softmmu/exec.o
  GEN     aarch64-softmmu/hmp-commands-info.h
  GEN     x86_64-softmmu/config-target.h
  CC      mips-linux-user/tcg/tcg-op.o
  CC      ppc64le-linux-user/tcg/optimize.o
  CC      x86_64-softmmu/exec.o
  GEN     mips-softmmu/config-target.h
  GEN     alpha-softmmu/config-target.h
  GEN     aarch64-softmmu/config-target.h
  CC      alpha-softmmu/exec.o
  CC      mipsel-linux-user/tcg/tcg-op.o
  CC      aarch64-softmmu/exec.o
  GEN     i386-softmmu/hmp-commands-info.h
  CC      mips-softmmu/exec.o
  CC      mips64el-softmmu/tcg/tcg.o
  GEN     i386-softmmu/config-target.h
  CC      i386-softmmu/exec.o
  CC      ppc64-linux-user/tcg/tcg-op.o
  CC      aarch64_be-linux-user/tcg/tcg-op.o
  CC      mips64el-softmmu/tcg/tcg-op.o
  CC      microblaze-softmmu/tcg/tcg-op.o
  CC      mips-softmmu/tcg/tcg.o
  CC      or1k-softmmu/tcg/tcg.o
  CC      sh4eb-linux-user/tcg/tcg-op.o
  CC      cris-softmmu/tcg/tcg-op.o
  CC      i386-linux-user/linux-user/uname.o
  CC      xtensa-softmmu/tcg/tcg.o
  CC      hppa-linux-user/tcg/optimize.o
  CC      ppc64-linux-user/tcg/optimize.o
  CC      mips64-linux-user/tcg/optimize.o
  CC      or1k-linux-user/tcg/tcg-op.o
  CC      sparc32plus-linux-user/tcg/tcg-op.o
  CC      microblazeel-linux-user/tcg/tcg-op.o
  CC      microblazeel-linux-user/tcg/optimize.o
  CC      nios2-softmmu/tcg/tcg.o
  CC      x86_64-linux-user/tcg/optimize.o
  CC      x86_64-linux-user/tcg/tcg-common.o
  CC      sh4-linux-user/tcg/tcg-common.o
  CC      mipsel-softmmu/tcg/tcg.o
  CCAS    i386-linux-user/linux-user/safe-syscall.o
  CC      ppc64-softmmu/tcg/tcg.o
  CC      mipsel-linux-user/tcg/optimize.o
  CC      armeb-linux-user/tcg/tcg-op.o
  CC      sparc32plus-linux-user/tcg/optimize.o
  CC      ppc-linux-user/tcg/tcg-op.o
  CC      mipsel-linux-user/tcg/tcg-common.o
  CC      sh4-linux-user/fpu/softfloat.o
  CC      hppa-linux-user/tcg/tcg-common.o
  CC      armeb-linux-user/tcg/optimize.o
  CC      ppc64abi32-linux-user/fpu/softfloat.o
  CC      unicore32-softmmu/tcg/tcg-op.o
  CC      microblaze-linux-user/tcg/optimize.o
  CC      nios2-softmmu/tcg/tcg-op.o
  CC      m68k-linux-user/fpu/softfloat.o
  CC      nios2-softmmu/tcg/optimize.o
  CC      m68k-linux-user/disas.o
  CC      arm-softmmu/tcg/tcg-op.o
  CC      nios2-linux-user/fpu/softfloat.o
  CC      mipsel-softmmu/tcg/tcg-op.o
  CC      aarch64-linux-user/tcg/tcg-op.o
  CC      moxie-softmmu/tcg/tcg-op.o
  CC      sparc-linux-user/tcg/tcg-op.o
  CC      tilegx-linux-user/tcg/tcg-op.o
  CC      microblazeel-linux-user/tcg/tcg-common.o
  CC      aarch64_be-linux-user/tcg/optimize.o
  CC      alpha-softmmu/tcg/tcg.o
  GEN     m68k-linux-user/gdbstub-xml.c
  CC      s390x-linux-user/tcg/tcg-op.o
  CC      mipsn32-linux-user/tcg/tcg-op.o
  CC      aarch64-softmmu/tcg/tcg.o
  CC      mipsel-softmmu/tcg/optimize.o
  CC      x86_64-linux-user/fpu/softfloat.o
  CC      x86_64-softmmu/tcg/tcg.o
  CC      tricore-softmmu/tcg/tcg.o
  CC      mipsel-softmmu/tcg/tcg-common.o
  CC      x86_64-softmmu/tcg/tcg-op.o
  CC      sparc32plus-linux-user/tcg/tcg-common.o
  CC      ppc-softmmu/tcg/tcg.o
  CC      s390x-linux-user/tcg/optimize.o
  CC      s390x-softmmu/tcg/tcg.o
  CC      sparc64-linux-user/tcg/optimize.o
  CC      mips64-linux-user/tcg/tcg-common.o
  CC      hppa-linux-user/fpu/softfloat.o
  CC      alpha-linux-user/tcg/tcg-op.o
  CC      arm-linux-user/tcg/tcg-op.o
  CC      mips64el-linux-user/tcg/optimize.o
  CC      aarch64-linux-user/tcg/optimize.o
  CC      s390x-softmmu/tcg/tcg-op.o
  CC      moxie-softmmu/tcg/optimize.o
  CC      microblazeel-linux-user/fpu/softfloat.o
  CC      cris-linux-user/accel/tcg/user-exec.o
  CC      ppc64le-linux-user/tcg/tcg-common.o
  CC      microblazeel-softmmu/tcg/optimize.o
  CC      microblazeel-linux-user/disas.o
  CC      sh4eb-linux-user/tcg/optimize.o
  CC      mipsel-softmmu/fpu/softfloat.o
  CC      mips64-linux-user/fpu/softfloat.o
  CC      m68k-softmmu/tcg/tcg-op.o
  CC      xtensa-softmmu/tcg/tcg-op.o
  CC      sparc-softmmu/tcg/tcg.o
  CC      mips-linux-user/tcg/optimize.o
  CC      mipsel-softmmu/disas.o
  CC      mips64el-softmmu/tcg/optimize.o
  CC      armeb-linux-user/tcg/tcg-common.o
  CC      mipsn32el-linux-user/tcg/tcg-op.o
  CC      ppc64le-linux-user/fpu/softfloat.o
  CC      microblaze-softmmu/tcg/optimize.o
  CC      nios2-softmmu/tcg/tcg-common.o
  CC      s390x-linux-user/tcg/tcg-common.o
  CC      xtensa-softmmu/tcg/optimize.o
  CC      or1k-softmmu/tcg/tcg-op.o
  CC      ppc64-linux-user/tcg/tcg-common.o
  CC      microblaze-linux-user/tcg/tcg-common.o
  CC      arm-linux-user/tcg/optimize.o
  CC      tilegx-linux-user/tcg/optimize.o
  CC      mips64el-softmmu/tcg/tcg-common.o
  CC      mips64-linux-user/disas.o
  CC      sparc64-softmmu/tcg/tcg.o
  CC      sh4-softmmu/tcg/tcg.o
  CC      sh4-linux-user/disas.o
  CC      m68k-linux-user/gdbstub.o
  CC      i386-linux-user/linux-user/vm86.o
  CC      m68k-softmmu/tcg/optimize.o
  CC      mipsel-linux-user/fpu/softfloat.o
  CC      m68k-softmmu/tcg/tcg-common.o
  CC      sparc-softmmu/tcg/tcg-op.o
  CC      tricore-softmmu/tcg/tcg-op.o
  CC      microblazeel-linux-user/gdbstub.o
  CC      s390x-linux-user/fpu/softfloat.o
  CC      mips-linux-user/tcg/tcg-common.o
  CC      microblaze-linux-user/fpu/softfloat.o
  CC      ppc64-linux-user/fpu/softfloat.o
  CC      sh4eb-softmmu/tcg/tcg.o
  CC      ppc-softmmu/tcg/tcg-op.o
  CC      cris-linux-user/accel/tcg/user-exec-stub.o
  CC      nios2-softmmu/fpu/softfloat.o
  CC      xtensaeb-softmmu/tcg/tcg.o
  CC      mips-linux-user/fpu/softfloat.o
  CC      mips-linux-user/disas.o
  CC      cris-softmmu/tcg/optimize.o
  CC      alpha-linux-user/tcg/optimize.o
  CC      mipsel-softmmu/arch_init.o
  CC      or1k-linux-user/tcg/optimize.o
  CC      xtensa-softmmu/tcg/tcg-common.o
  CC      lm32-softmmu/tcg/tcg-op.o
  CC      mips-softmmu/tcg/tcg-op.o
  CC      i386-softmmu/tcg/tcg.o
  CC      i386-softmmu/tcg/tcg-op.o
  CC      cris-linux-user/linux-user/main.o
  CC      mips64-softmmu/tcg/tcg.o
  CC      alpha-linux-user/tcg/tcg-common.o
  CC      unicore32-softmmu/tcg/optimize.o
  CC      mips64el-linux-user/tcg/tcg-common.o
  CC      ppc-linux-user/tcg/optimize.o
  CC      mipsel-linux-user/disas.o
  CC      sparc64-linux-user/tcg/tcg-common.o
  CC      sh4eb-softmmu/tcg/tcg-op.o
  CC      xtensa-softmmu/fpu/softfloat.o
  CC      alpha-linux-user/fpu/softfloat.o
  CC      nios2-linux-user/disas.o
  CC      lm32-softmmu/tcg/optimize.o
  CC      sh4eb-softmmu/tcg/optimize.o
  CC      ppc-linux-user/tcg/tcg-common.o
  CC      ppc64-linux-user/disas.o
  CC      alpha-softmmu/tcg/tcg-op.o
  CC      sh4-linux-user/gdbstub.o
  CC      aarch64-linux-user/tcg/tcg-common.o
  CC      ppcemb-softmmu/tcg/tcg.o
  CC      microblazeel-linux-user/thunk.o
  CC      aarch64_be-linux-user/tcg/tcg-common.o
  CC      microblazeel-softmmu/tcg/tcg-common.o
  CC      aarch64-softmmu/tcg/tcg-op.o
  CC      sparc64-linux-user/fpu/softfloat.o
  CC      mips-softmmu/tcg/optimize.o
  CC      sh4eb-linux-user/tcg/tcg-common.o
  CC      mips64el-softmmu/fpu/softfloat.o
  CC      tricore-softmmu/tcg/optimize.o
  CC      microblaze-linux-user/disas.o
  CC      cris-linux-user/linux-user/syscall.o
  CC      mips64-softmmu/tcg/tcg-op.o
  CC      sh4-linux-user/thunk.o
  CC      mips64el-softmmu/disas.o
  CC      microblazeel-softmmu/fpu/softfloat.o
  CC      aarch64_be-linux-user/fpu/softfloat.o
  CC      microblaze-softmmu/tcg/tcg-common.o
  CC      microblaze-softmmu/fpu/softfloat.o
  CC      sh4eb-linux-user/fpu/softfloat.o
  CC      lm32-softmmu/tcg/tcg-common.o
  CC      ppcemb-softmmu/tcg/tcg-op.o
  CC      nios2-softmmu/disas.o
  CC      alpha-linux-user/disas.o
  CC      mips64el-linux-user/fpu/softfloat.o
  CC      microblazeel-linux-user/accel/stubs/hax-stub.o
  CC      ppc64-softmmu/tcg/tcg-op.o
  CC      sh4eb-softmmu/tcg/tcg-common.o
  CC      xtensa-softmmu/disas.o
  CC      mips64-linux-user/gdbstub.o
  CC      sparc64-linux-user/disas.o
  CC      sparc64-softmmu/tcg/tcg-op.o
  CC      x86_64-linux-user/disas.o
  CC      sh4eb-softmmu/fpu/softfloat.o
  CC      cris-softmmu/tcg/tcg-common.o
  CC      sparc-linux-user/tcg/optimize.o
  CC      mipsn32-linux-user/tcg/optimize.o
  CC      or1k-linux-user/tcg/tcg-common.o
  CC      sparc32plus-linux-user/fpu/softfloat.o
  CC      sparc32plus-linux-user/disas.o
  CC      microblaze-linux-user/gdbstub.o
  CC      ppc-softmmu/tcg/optimize.o
  CC      microblazeel-softmmu/disas.o
  CC      cris-softmmu/fpu/softfloat.o
  CC      xtensa-softmmu/arch_init.o
  CC      or1k-linux-user/fpu/softfloat.o
  CC      unicore32-softmmu/tcg/tcg-common.o
  CC      microblazeel-softmmu/arch_init.o
  CC      aarch64-softmmu/tcg/optimize.o
  CC      s390x-linux-user/disas.o
  CC      mipsn32-linux-user/tcg/tcg-common.o
  CC      mipsn32-linux-user/fpu/softfloat.o
  CC      sparc-linux-user/tcg/tcg-common.o
  CC      arm-softmmu/tcg/optimize.o
  CC      armeb-linux-user/fpu/softfloat.o
  CC      sh4-softmmu/tcg/tcg-op.o
  CC      unicore32-softmmu/fpu/softfloat.o
  CC      sh4eb-linux-user/disas.o
  CC      xtensaeb-softmmu/tcg/tcg-op.o
  CC      alpha-softmmu/tcg/optimize.o
  CC      cris-linux-user/linux-user/strace.o
  CC      cris-softmmu/disas.o
  CC      mipsn32el-linux-user/tcg/optimize.o
  CC      mips64el-linux-user/disas.o
  CC      ppcemb-softmmu/tcg/optimize.o
  CC      m68k-linux-user/thunk.o
  CC      nios2-softmmu/arch_init.o
  CC      moxie-softmmu/tcg/tcg-common.o
  CC      mipsel-softmmu/cpus.o
  CC      m68k-softmmu/fpu/softfloat.o
  CC      aarch64-linux-user/fpu/softfloat.o
  CC      ppc-linux-user/fpu/softfloat.o
  CC      microblaze-linux-user/thunk.o
  CC      moxie-softmmu/fpu/softfloat.o
  CC      unicore32-softmmu/disas.o
  CC      or1k-linux-user/disas.o
  CC      microblazeel-linux-user/accel/stubs/hvf-stub.o
  CC      mipsel-softmmu/monitor.o
  CC      tilegx-linux-user/tcg/tcg-common.o
  CC      or1k-softmmu/tcg/optimize.o
  CC      mips64el-softmmu/arch_init.o
  CC      or1k-softmmu/tcg/tcg-common.o
  CC      hppa-linux-user/disas.o
  CC      ppc64le-linux-user/disas.o
  GEN     ppc64le-linux-user/gdbstub-xml.c
  CC      sparc64-softmmu/tcg/optimize.o
  CC      ppc64le-linux-user/gdbstub.o
  CC      or1k-softmmu/fpu/softfloat.o
  CC      m68k-linux-user/accel/stubs/hax-stub.o
  CC      s390x-softmmu/tcg/optimize.o
  CC      microblazeel-softmmu/cpus.o
  CC      lm32-softmmu/fpu/softfloat.o
  CC      tilegx-linux-user/fpu/softfloat.o
  CC      i386-linux-user/target/i386/helper.o
  CC      mips64-softmmu/tcg/optimize.o
  CC      arm-softmmu/tcg/tcg-common.o
  CC      mipsel-linux-user/gdbstub.o
  CC      m68k-linux-user/accel/stubs/hvf-stub.o
  CC      ppc-softmmu/tcg/tcg-common.o
  CC      mipsel-linux-user/thunk.o
  CC      sparc-linux-user/fpu/softfloat.o
  CC      x86_64-softmmu/tcg/optimize.o
  GEN     ppc64-linux-user/gdbstub-xml.c
  CC      xtensa-softmmu/cpus.o
  CC      alpha-softmmu/tcg/tcg-common.o
  CC      mipsn32el-linux-user/tcg/tcg-common.o
  CC      mips64-softmmu/tcg/tcg-common.o
  CC      mips64el-linux-user/gdbstub.o
  CC      unicore32-softmmu/arch_init.o
  CC      nios2-linux-user/gdbstub.o
  CC      ppcemb-softmmu/tcg/tcg-common.o
  CC      s390x-softmmu/tcg/tcg-common.o
  CC      ppc-linux-user/disas.o
  CC      i386-linux-user/target/i386/cpu.o
  CC      nios2-linux-user/thunk.o
  CC      s390x-softmmu/fpu/softfloat.o
  CC      tricore-softmmu/tcg/tcg-common.o
  CC      mipsel-linux-user/accel/stubs/hax-stub.o
  CC      nios2-linux-user/accel/stubs/hax-stub.o
  CC      mips64el-softmmu/cpus.o
  CC      aarch64-softmmu/tcg/tcg-common.o
  CC      ppc64-softmmu/tcg/optimize.o
  CC      arm-linux-user/tcg/tcg-common.o
  CC      ppc-softmmu/fpu/softfloat.o
  CC      sh4-linux-user/accel/stubs/hax-stub.o
  CC      mips64-linux-user/thunk.o
  CC      sh4-linux-user/accel/stubs/hvf-stub.o
  CC      sparc-softmmu/tcg/optimize.o
  CC      sh4-linux-user/accel/stubs/kvm-stub.o
  CC      sh4eb-softmmu/disas.o
  CC      mipsel-softmmu/gdbstub.o
  CC      mips64el-softmmu/monitor.o
  CC      sparc32plus-linux-user/gdbstub.o
  CC      arm-softmmu/fpu/softfloat.o
  CC      armeb-linux-user/disas.o
  CC      nios2-softmmu/cpus.o
  CC      mips-softmmu/tcg/tcg-common.o
  CC      tricore-softmmu/fpu/softfloat.o
  CC      i386-linux-user/target/i386/gdbstub.o
  CC      cris-linux-user/linux-user/mmap.o
  CC      mipsn32-linux-user/disas.o
  CC      aarch64-linux-user/disas.o
  CC      mips-linux-user/gdbstub.o
  CC      mipsn32-linux-user/gdbstub.o
  CC      mipsn32-linux-user/thunk.o
  CC      x86_64-softmmu/tcg/tcg-common.o
  CC      i386-linux-user/target/i386/xsave_helper.o
  CC      aarch64-softmmu/fpu/softfloat.o
  CC      mipsn32el-linux-user/fpu/softfloat.o
  CC      ppc64-softmmu/tcg/tcg-common.o
  CC      arm-linux-user/fpu/softfloat.o
  GEN     armeb-linux-user/gdbstub-xml.c
  CC      i386-linux-user/target/i386/translate.o
  CC      ppc64abi32-linux-user/disas.o
  CC      aarch64_be-linux-user/disas.o
  GEN     x86_64-linux-user/gdbstub-xml.c
  CC      i386-linux-user/target/i386/bpt_helper.o
  CC      lm32-softmmu/disas.o
  CC      sparc64-linux-user/gdbstub.o
  CC      sh4eb-linux-user/gdbstub.o
  CC      i386-softmmu/tcg/optimize.o
  CC      sparc32plus-linux-user/thunk.o
  CC      x86_64-linux-user/gdbstub.o
  CC      or1k-softmmu/disas.o
  CC      ppcemb-softmmu/fpu/softfloat.o
  CC      alpha-softmmu/fpu/softfloat.o
  CC      mips-softmmu/fpu/softfloat.o
  CC      x86_64-softmmu/fpu/softfloat.o
  CC      alpha-softmmu/disas.o
  CC      alpha-softmmu/arch_init.o
  CC      ppc64-linux-user/gdbstub.o
  CC      ppcemb-softmmu/disas.o
  CC      mips64el-softmmu/gdbstub.o
  GEN     aarch64_be-linux-user/gdbstub-xml.c
  CC      x86_64-linux-user/thunk.o
  CC      m68k-linux-user/accel/stubs/kvm-stub.o
  CC      hppa-linux-user/gdbstub.o
  CC      alpha-linux-user/gdbstub.o
  CC      sh4-linux-user/accel/tcg/tcg-runtime.o
  CC      s390x-softmmu/disas.o
  CC      xtensa-softmmu/monitor.o
  CC      sh4-softmmu/tcg/optimize.o
  CC      xtensa-softmmu/gdbstub.o
  CC      mipsn32el-linux-user/disas.o
  CC      xtensa-softmmu/balloon.o
  CC      microblaze-linux-user/accel/stubs/hax-stub.o
  CC      ppc64le-linux-user/thunk.o
  CC      nios2-linux-user/accel/stubs/hvf-stub.o
  GEN     ppc64abi32-linux-user/gdbstub-xml.c
  CC      ppc64le-linux-user/accel/stubs/hax-stub.o
  CC      sparc64-softmmu/tcg/tcg-common.o
  CC      lm32-softmmu/arch_init.o
  CC      mips64-linux-user/accel/stubs/hax-stub.o
  CC      microblazeel-softmmu/monitor.o
  CC      sh4eb-softmmu/arch_init.o
  CC      m68k-linux-user/accel/tcg/tcg-runtime.o
  CC      m68k-softmmu/disas.o
  CC      ppc64abi32-linux-user/gdbstub.o
  CC      sparc32plus-linux-user/accel/stubs/hax-stub.o
  CC      mipsel-linux-user/accel/stubs/hvf-stub.o
  GEN     m68k-softmmu/gdbstub-xml.c
  CC      aarch64-softmmu/disas.o
  CC      nios2-linux-user/accel/stubs/kvm-stub.o
  CC      sh4eb-linux-user/thunk.o
  CC      mips64-softmmu/fpu/softfloat.o
  CC      mips64-softmmu/disas.o
  CC      sh4-linux-user/accel/tcg/cpu-exec.o
  CC      mipsn32-linux-user/accel/stubs/hax-stub.o
  CC      sparc64-linux-user/thunk.o
  CC      mips64-linux-user/accel/stubs/hvf-stub.o
  CC      microblaze-linux-user/accel/stubs/hvf-stub.o
  CC      microblazeel-softmmu/gdbstub.o
  CC      mipsel-softmmu/balloon.o
  CC      ppc-softmmu/disas.o
  CC      tricore-softmmu/disas.o
  CC      m68k-linux-user/accel/tcg/cpu-exec.o
  CC      moxie-softmmu/disas.o
  CC      microblaze-softmmu/disas.o
  CC      ppc64le-linux-user/accel/stubs/hvf-stub.o
  CC      xtensa-softmmu/ioport.o
  CC      mips64el-linux-user/thunk.o
  GEN     aarch64-linux-user/gdbstub-xml.c
  CC      mips64el-linux-user/accel/stubs/hax-stub.o
  GEN     ppc-linux-user/gdbstub-xml.c
  CC      nios2-linux-user/accel/tcg/tcg-runtime.o
  CC      nios2-linux-user/accel/tcg/cpu-exec.o
  CC      mips64el-softmmu/balloon.o
  CC      x86_64-linux-user/accel/stubs/hax-stub.o
  CC      mips64-linux-user/accel/stubs/kvm-stub.o
  CC      sparc64-softmmu/fpu/softfloat.o
  CC      mips64el-softmmu/ioport.o
  CC      mips-linux-user/thunk.o
  GEN     ppcemb-softmmu/gdbstub-xml.c
  CC      sparc-softmmu/tcg/tcg-common.o
  CC      ppc64le-linux-user/accel/stubs/kvm-stub.o
  CC      mips64-linux-user/accel/tcg/tcg-runtime.o
  CC      lm32-softmmu/cpus.o
  CC      sh4eb-softmmu/cpus.o
  CC      microblazeel-linux-user/accel/stubs/kvm-stub.o
  CC      ppc-linux-user/gdbstub.o
  CC      arm-linux-user/disas.o
  CC      ppc64le-linux-user/accel/tcg/tcg-runtime.o
  CC      mips-linux-user/accel/stubs/hax-stub.o
  CC      ppc64abi32-linux-user/thunk.o
  CC      sh4-linux-user/accel/tcg/cpu-exec-common.o
  CC      aarch64-linux-user/gdbstub.o
  CC      ppc64-softmmu/fpu/softfloat.o
  CC      tricore-softmmu/arch_init.o
  CC      aarch64-linux-user/thunk.o
  CC      sparc32plus-linux-user/accel/stubs/hvf-stub.o
  CC      or1k-softmmu/arch_init.o
  CC      x86_64-linux-user/accel/stubs/hvf-stub.o
  CC      sh4eb-softmmu/monitor.o
  CC      ppc64le-linux-user/accel/tcg/cpu-exec.o
  CC      nios2-linux-user/accel/tcg/cpu-exec-common.o
  CC      alpha-softmmu/cpus.o
  CC      cris-linux-user/linux-user/signal.o
  CC      ppc64-softmmu/disas.o
  CC      nios2-linux-user/accel/tcg/translate-all.o
  CC      sparc-softmmu/fpu/softfloat.o
  CC      i386-linux-user/target/i386/cc_helper.o
  CC      m68k-softmmu/arch_init.o
  CC      mips64-softmmu/arch_init.o
  CC      x86_64-linux-user/accel/stubs/kvm-stub.o
  CC      sparc-softmmu/disas.o
  CC      i386-linux-user/target/i386/excp_helper.o
  CC      mipsn32-linux-user/accel/stubs/hvf-stub.o
  CC      sh4eb-linux-user/accel/stubs/hax-stub.o
  CC      sh4-linux-user/accel/tcg/translate-all.o
  CC      microblazeel-linux-user/accel/tcg/tcg-runtime.o
  GEN     s390x-linux-user/gdbstub-xml.c
  CC      sparc-linux-user/disas.o
  CC      cris-softmmu/arch_init.o
  CC      mipsel-softmmu/ioport.o
  CC      m68k-linux-user/accel/tcg/cpu-exec-common.o
  CC      i386-linux-user/target/i386/fpu_helper.o
  CC      unicore32-softmmu/cpus.o
  CC      mips-softmmu/disas.o
  CC      sh4-linux-user/accel/tcg/translator.o
  CC      ppc64-linux-user/thunk.o
  CC      sh4-linux-user/accel/tcg/user-exec.o
  CC      mips64-linux-user/accel/tcg/cpu-exec.o
  CC      tricore-softmmu/cpus.o
  CC      microblaze-linux-user/accel/stubs/kvm-stub.o
  CC      microblazeel-linux-user/accel/tcg/cpu-exec.o
  CC      alpha-softmmu/monitor.o
  CC      i386-softmmu/tcg/tcg-common.o
  CC      mipsn32-linux-user/accel/stubs/kvm-stub.o
  GEN     ppc64-softmmu/gdbstub-xml.c
  CC      sparc-softmmu/arch_init.o
  CC      or1k-softmmu/cpus.o
  CC      x86_64-linux-user/accel/tcg/tcg-runtime.o
  CC      mips-linux-user/accel/stubs/hvf-stub.o
  CC      s390x-linux-user/gdbstub.o
  CC      or1k-softmmu/monitor.o
  CC      sh4-softmmu/tcg/tcg-common.o
  CC      hppa-linux-user/thunk.o
  CC      aarch64_be-linux-user/gdbstub.o
  CC      aarch64_be-linux-user/thunk.o
  CC      sh4-linux-user/accel/tcg/user-exec-stub.o
  CC      sh4-linux-user/linux-user/main.o
  CC      xtensaeb-softmmu/tcg/optimize.o
  CC      xtensa-softmmu/numa.o
  CC      alpha-linux-user/thunk.o
  CC      mipsel-linux-user/accel/stubs/kvm-stub.o
  CC      tricore-softmmu/monitor.o
  CC      mips-linux-user/accel/stubs/kvm-stub.o
  CC      mips64el-linux-user/accel/stubs/hvf-stub.o
  CC      or1k-softmmu/gdbstub.o
  CC      m68k-linux-user/accel/tcg/translate-all.o
  GEN     s390x-softmmu/gdbstub-xml.c
  CC      mips64-softmmu/cpus.o
  CC      sparc64-linux-user/accel/stubs/hax-stub.o
  CC      arm-softmmu/disas.o
  CC      i386-softmmu/fpu/softfloat.o
  CC      microblaze-linux-user/accel/tcg/tcg-runtime.o
  CC      mips64el-linux-user/accel/stubs/kvm-stub.o
  CC      tricore-softmmu/gdbstub.o
  CC      sh4-softmmu/fpu/softfloat.o
  GEN     aarch64-softmmu/gdbstub-xml.c
  CC      ppc64le-linux-user/accel/tcg/cpu-exec-common.o
  CC      mips-linux-user/accel/tcg/tcg-runtime.o
  CC      mipsel-linux-user/accel/tcg/tcg-runtime.o
  CC      microblaze-linux-user/accel/tcg/cpu-exec.o
  CC      mips64-linux-user/accel/tcg/cpu-exec-common.o
  GEN     arm-softmmu/gdbstub-xml.c
  CC      sh4-linux-user/linux-user/syscall.o
  CC      ppc64-linux-user/accel/stubs/hax-stub.o
  CC      x86_64-linux-user/accel/tcg/cpu-exec.o
  GEN     ppc-softmmu/gdbstub-xml.c
  CC      s390x-linux-user/thunk.o
  CC      ppc64abi32-linux-user/accel/stubs/hax-stub.o
  CC      ppc64abi32-linux-user/accel/stubs/hvf-stub.o
  CC      sh4-softmmu/disas.o
  CC      sh4eb-linux-user/accel/stubs/hvf-stub.o
  CC      mipsel-softmmu/numa.o
  CC      mipsel-softmmu/qtest.o
  CC      nios2-linux-user/accel/tcg/translator.o
  CC      tricore-softmmu/balloon.o
  CC      tricore-softmmu/ioport.o
  CC      tricore-softmmu/numa.o
  CC      sparc32plus-linux-user/accel/stubs/kvm-stub.o
  CC      ppc-softmmu/arch_init.o
  CC      x86_64-linux-user/accel/tcg/cpu-exec-common.o
  CC      x86_64-softmmu/disas.o
  CC      hppa-linux-user/accel/stubs/hax-stub.o
  CC      ppc64-softmmu/arch_init.o
  CC      ppc64-linux-user/accel/stubs/hvf-stub.o
  CC      ppc64le-linux-user/accel/tcg/translate-all.o
  CC      sparc64-linux-user/accel/stubs/hvf-stub.o
  CC      microblazeel-linux-user/accel/tcg/cpu-exec-common.o
  GEN     arm-linux-user/gdbstub-xml.c
  CC      tricore-softmmu/qtest.o
  CC      sparc64-linux-user/accel/stubs/kvm-stub.o
  CC      s390x-linux-user/accel/stubs/hax-stub.o
  CC      microblazeel-softmmu/balloon.o
  CC      mipsel-softmmu/memory.o
  CC      ppc64le-linux-user/accel/tcg/translator.o
  CC      mips64-linux-user/accel/tcg/translate-all.o
  CC      ppc-softmmu/cpus.o
  CC      cris-linux-user/linux-user/elfload.o
  CC      nios2-softmmu/monitor.o
  CC      alpha-linux-user/accel/stubs/hax-stub.o
  CC      moxie-softmmu/arch_init.o
  CC      ppc64-linux-user/accel/stubs/kvm-stub.o
  CC      ppc64-softmmu/cpus.o
  CC      hppa-linux-user/accel/stubs/hvf-stub.o
  CC      ppc64abi32-linux-user/accel/stubs/kvm-stub.o
  CC      mips-linux-user/accel/tcg/cpu-exec.o
  CC      hppa-linux-user/accel/stubs/kvm-stub.o
  CC      armeb-linux-user/gdbstub.o
  CC      sparc32plus-linux-user/accel/tcg/tcg-runtime.o
  CC      microblaze-linux-user/accel/tcg/cpu-exec-common.o
  CC      moxie-softmmu/cpus.o
  CC      ppc64le-linux-user/accel/tcg/user-exec.o
  CC      mipsel-linux-user/accel/tcg/cpu-exec.o
  CC      mipsn32el-linux-user/gdbstub.o
  CC      sh4-linux-user/linux-user/strace.o
  CC      microblazeel-linux-user/accel/tcg/translate-all.o
  CC      tilegx-linux-user/disas.o
  CC      nios2-linux-user/accel/tcg/user-exec.o
  CC      x86_64-linux-user/accel/tcg/translate-all.o
  CC      cris-softmmu/cpus.o
  CC      hppa-linux-user/accel/tcg/tcg-runtime.o
  CC      hppa-linux-user/accel/tcg/cpu-exec.o
  CC      m68k-linux-user/accel/tcg/translator.o
  CC      sparc64-linux-user/accel/tcg/tcg-runtime.o
  CC      tricore-softmmu/memory.o
  CC      sparc64-softmmu/disas.o
  CC      alpha-linux-user/accel/stubs/hvf-stub.o
  CC      hppa-linux-user/accel/tcg/cpu-exec-common.o
  CC      sh4eb-softmmu/gdbstub.o
  CC      sparc64-linux-user/accel/tcg/cpu-exec.o
  CC      ppc64-linux-user/accel/tcg/tcg-runtime.o
  CC      sh4eb-softmmu/balloon.o
  CC      alpha-linux-user/accel/stubs/kvm-stub.o
  CC      microblaze-linux-user/accel/tcg/translate-all.o
  CC      mips64el-softmmu/numa.o
  CC      mipsel-softmmu/memory_mapping.o
  CC      tricore-softmmu/memory_mapping.o
  CC      ppc64-linux-user/accel/tcg/cpu-exec.o
  CC      i386-linux-user/target/i386/int_helper.o
  CC      microblaze-softmmu/arch_init.o
  CC      microblazeel-linux-user/accel/tcg/translator.o
  CC      mips64el-linux-user/accel/tcg/tcg-runtime.o
  CC      microblazeel-linux-user/accel/tcg/user-exec.o
  CC      mipsel-softmmu/dump.o
  CC      unicore32-softmmu/monitor.o
  CC      ppc64abi32-linux-user/accel/tcg/tcg-runtime.o
  CC      i386-softmmu/disas.o
  CC      sh4eb-softmmu/ioport.o
  CC      s390x-linux-user/accel/stubs/hvf-stub.o
  CC      ppc-linux-user/thunk.o
  CC      sparc32plus-linux-user/accel/tcg/cpu-exec.o
  CC      sparc-softmmu/cpus.o
  CC      ppc64-softmmu/monitor.o
  CC      or1k-linux-user/gdbstub.o
  CC      or1k-linux-user/thunk.o
  CC      arm-linux-user/gdbstub.o
  CC      aarch64-softmmu/arch_init.o
  CC      microblaze-softmmu/cpus.o
  CC      mips-linux-user/accel/tcg/cpu-exec-common.o
  CC      mips64-linux-user/accel/tcg/translator.o
  CC      m68k-linux-user/accel/tcg/user-exec.o
  CC      s390x-linux-user/accel/stubs/kvm-stub.o
  CC      mips-softmmu/arch_init.o
  CC      xtensaeb-softmmu/tcg/tcg-common.o
  CC      microblazeel-linux-user/accel/tcg/user-exec-stub.o
  CC      alpha-linux-user/accel/tcg/tcg-runtime.o
  CC      or1k-softmmu/balloon.o
  CC      i386-linux-user/target/i386/mem_helper.o
  CC      hppa-linux-user/accel/tcg/translate-all.o
  CC      sh4eb-linux-user/accel/stubs/kvm-stub.o
  CC      cris-softmmu/monitor.o
  CC      sparc64-linux-user/accel/tcg/cpu-exec-common.o
  CC      aarch64_be-linux-user/accel/stubs/hax-stub.o
  CC      sparc64-linux-user/accel/tcg/translate-all.o
  CC      mipsn32el-linux-user/thunk.o
  CC      mipsel-linux-user/accel/tcg/cpu-exec-common.o
  CC      ppc64abi32-linux-user/accel/tcg/cpu-exec.o
  CC      microblaze-linux-user/accel/tcg/translator.o
  CC      mips64el-softmmu/qtest.o
  CC      mips64el-linux-user/accel/tcg/cpu-exec.o
  CC      lm32-softmmu/monitor.o
  CC      arm-linux-user/thunk.o
  CC      sh4eb-linux-user/accel/tcg/tcg-runtime.o
  CC      ppc64-linux-user/accel/tcg/cpu-exec-common.o
  CC      m68k-softmmu/cpus.o
  CC      nios2-linux-user/accel/tcg/user-exec-stub.o
  CC      sparc-linux-user/gdbstub.o
  CC      or1k-linux-user/accel/stubs/hax-stub.o
  CC      ppc-linux-user/accel/stubs/hax-stub.o
  CC      xtensaeb-softmmu/fpu/softfloat.o
  CC      mips-linux-user/accel/tcg/translate-all.o
  CC      sh4eb-linux-user/accel/tcg/cpu-exec.o
  CC      ppc64-softmmu/gdbstub.o
  CC      cris-softmmu/gdbstub.o
  CC      alpha-linux-user/accel/tcg/cpu-exec.o
  CC      xtensa-softmmu/qtest.o
  CC      lm32-softmmu/gdbstub.o
  CC      sh4-softmmu/arch_init.o
  CC      sh4-linux-user/linux-user/mmap.o
  CC      mipsn32-linux-user/accel/tcg/tcg-runtime.o
  CC      sparc32plus-linux-user/accel/tcg/cpu-exec-common.o
  CC      x86_64-linux-user/accel/tcg/translator.o
  CC      sh4-softmmu/cpus.o
  CC      sparc64-softmmu/arch_init.o
  CC      mipsel-linux-user/accel/tcg/translate-all.o
  CC      microblazeel-softmmu/ioport.o
  CC      s390x-linux-user/accel/tcg/tcg-runtime.o
  CC      ppc64-linux-user/accel/tcg/translate-all.o
  CC      microblazeel-softmmu/numa.o
  CC      sparc64-softmmu/cpus.o
  CC      ppc64le-linux-user/accel/tcg/user-exec-stub.o
  CC      aarch64_be-linux-user/accel/stubs/hvf-stub.o
  CC      arm-softmmu/arch_init.o
  CC      arm-linux-user/accel/stubs/hax-stub.o
  CC      nios2-linux-user/linux-user/main.o
  CC      tilegx-linux-user/gdbstub.o
  CC      mipsn32-linux-user/accel/tcg/cpu-exec.o
  CC      aarch64-linux-user/accel/stubs/hax-stub.o
  CC      aarch64_be-linux-user/accel/stubs/kvm-stub.o
  CC      or1k-softmmu/ioport.o
  CC      mips64-softmmu/monitor.o
  CC      s390x-softmmu/arch_init.o
  CC      sh4-softmmu/monitor.o
  CC      xtensaeb-softmmu/disas.o
  CC      sparc32plus-linux-user/accel/tcg/translate-all.o
  CC      microblaze-softmmu/monitor.o
  CC      microblaze-linux-user/accel/tcg/user-exec.o
  CC      i386-linux-user/target/i386/misc_helper.o
  CC      microblaze-linux-user/accel/tcg/user-exec-stub.o
  CC      ppc64le-linux-user/linux-user/main.o
  CC      ppc64abi32-linux-user/accel/tcg/cpu-exec-common.o
  CC      ppcemb-softmmu/arch_init.o
  CC      ppc64-linux-user/accel/tcg/translator.o
  CC      hppa-linux-user/accel/tcg/translator.o
  CC      sparc-softmmu/monitor.o
  CC      mips64el-linux-user/accel/tcg/cpu-exec-common.o
  CC      mips64el-linux-user/accel/tcg/translate-all.o
  CC      xtensaeb-softmmu/arch_init.o
  CC      microblazeel-linux-user/linux-user/main.o
  CC      or1k-softmmu/numa.o
  CC      s390x-linux-user/accel/tcg/cpu-exec.o
  CC      aarch64-linux-user/accel/stubs/hvf-stub.o
  CC      or1k-softmmu/qtest.o
  CC      ppc-linux-user/accel/stubs/hvf-stub.o
  CC      ppc-linux-user/accel/stubs/kvm-stub.o
  CC      microblaze-linux-user/linux-user/main.o
  CC      sh4eb-linux-user/accel/tcg/cpu-exec-common.o
  CC      or1k-softmmu/memory.o
  CC      moxie-softmmu/monitor.o
  CC      ppc64abi32-linux-user/accel/tcg/translate-all.o
  CC      microblaze-linux-user/linux-user/syscall.o
  CC      armeb-linux-user/thunk.o
  CC      sparc-linux-user/thunk.o
  CC      m68k-softmmu/monitor.o
  CC      alpha-linux-user/accel/tcg/cpu-exec-common.o
  CC      aarch64_be-linux-user/accel/tcg/tcg-runtime.o
  CC      microblazeel-linux-user/linux-user/syscall.o
  CC      mips64el-linux-user/accel/tcg/translator.o
  CC      mips-linux-user/accel/tcg/translator.o
  CC      nios2-softmmu/gdbstub.o
  CC      mips64el-softmmu/memory.o
  CC      hppa-linux-user/accel/tcg/user-exec.o
  CC      mips64-linux-user/accel/tcg/user-exec.o
  CC      xtensa-softmmu/memory.o
  CC      aarch64-linux-user/accel/stubs/kvm-stub.o
  CC      tilegx-linux-user/thunk.o
  CC      m68k-linux-user/accel/tcg/user-exec-stub.o
  CC      microblazeel-softmmu/qtest.o
  CC      aarch64-linux-user/accel/tcg/tcg-runtime.o
  CC      sh4eb-softmmu/numa.o
  CC      sh4eb-linux-user/accel/tcg/translate-all.o
  CC      or1k-linux-user/accel/stubs/hvf-stub.o
  CC      sparc64-linux-user/accel/tcg/translator.o
  CC      mipsn32-linux-user/accel/tcg/cpu-exec-common.o
  CC      aarch64-linux-user/accel/tcg/cpu-exec.o
  CC      ppc-linux-user/accel/tcg/tcg-runtime.o
  CC      alpha-linux-user/accel/tcg/translate-all.o
  CC      ppc-linux-user/accel/tcg/cpu-exec.o
  CC      x86_64-linux-user/accel/tcg/user-exec.o
  CC      tricore-softmmu/dump.o
  GEN     i386-softmmu/gdbstub-xml.c
  CC      sh4eb-linux-user/accel/tcg/translator.o
  CC      sh4eb-linux-user/accel/tcg/user-exec.o
  CC      mips64-softmmu/gdbstub.o
  CC      s390x-linux-user/accel/tcg/cpu-exec-common.o
  CC      xtensa-softmmu/memory_mapping.o
  CC      sparc64-linux-user/accel/tcg/user-exec.o
  CC      nios2-linux-user/linux-user/syscall.o
  CC      m68k-linux-user/linux-user/main.o
  CC      mips64el-linux-user/accel/tcg/user-exec.o
  CC      or1k-linux-user/accel/stubs/kvm-stub.o
  CC      microblazeel-softmmu/memory.o
  CC      sh4eb-softmmu/qtest.o
  CC      mipsn32-linux-user/accel/tcg/translate-all.o
  CC      mipsel-linux-user/accel/tcg/translator.o
  CC      ppc-linux-user/accel/tcg/cpu-exec-common.o
  CC      lm32-softmmu/balloon.o
  CC      sh4eb-softmmu/memory.o
  CC      ppc-softmmu/monitor.o
  CC      ppc-linux-user/accel/tcg/translate-all.o
  CC      microblaze-linux-user/linux-user/strace.o
  CC      i386-linux-user/target/i386/mpx_helper.o
  CC      alpha-linux-user/accel/tcg/translator.o
  CC      aarch64_be-linux-user/accel/tcg/cpu-exec.o
  CC      armeb-linux-user/accel/stubs/hax-stub.o
  CC      unicore32-softmmu/gdbstub.o
  CC      s390x-linux-user/accel/tcg/translate-all.o
  CC      mips-softmmu/cpus.o
  CC      or1k-linux-user/accel/tcg/tcg-runtime.o
  CC      microblaze-linux-user/linux-user/mmap.o
  CC      s390x-softmmu/cpus.o
  CC      ppc64abi32-linux-user/accel/tcg/translator.o
  CC      ppc-linux-user/accel/tcg/translator.o
  CC      arm-linux-user/accel/stubs/hvf-stub.o
  CC      cris-linux-user/linux-user/linuxload.o
  CC      ppc64-linux-user/accel/tcg/user-exec.o
  CC      unicore32-softmmu/balloon.o
  CC      sparc-linux-user/accel/stubs/hax-stub.o
  CC      aarch64_be-linux-user/accel/tcg/cpu-exec-common.o
  CC      ppc64le-linux-user/linux-user/syscall.o
  CC      ppcemb-softmmu/cpus.o
  CC      hppa-linux-user/accel/tcg/user-exec-stub.o
  CC      mips64-linux-user/accel/tcg/user-exec-stub.o
  CC      arm-softmmu/cpus.o
  CC      arm-softmmu/monitor.o
  CC      unicore32-softmmu/ioport.o
  CC      lm32-softmmu/ioport.o
  CC      sparc64-linux-user/accel/tcg/user-exec-stub.o
  CC      mipsn32-linux-user/accel/tcg/translator.o
  CC      aarch64-softmmu/cpus.o
  CC      tilegx-linux-user/accel/stubs/hax-stub.o
  CC      sparc32plus-linux-user/accel/tcg/translator.o
  CC      sparc32plus-linux-user/accel/tcg/user-exec.o
  CC      mips-linux-user/accel/tcg/user-exec.o
  CC      armeb-linux-user/accel/stubs/hvf-stub.o
  CC      lm32-softmmu/numa.o
  CC      or1k-softmmu/memory_mapping.o
  CC      ppc64-softmmu/balloon.o
  CC      mips-softmmu/monitor.o
  CC      microblazeel-softmmu/memory_mapping.o
  CC      microblazeel-linux-user/linux-user/strace.o
  CC      mips64-softmmu/balloon.o
  GEN     x86_64-softmmu/gdbstub-xml.c
  CC      or1k-linux-user/accel/tcg/cpu-exec.o
  CC      sparc-linux-user/accel/stubs/hvf-stub.o
  CC      aarch64-linux-user/accel/tcg/cpu-exec-common.o
  CC      sh4-linux-user/linux-user/signal.o
  CC      mips64-linux-user/linux-user/main.o
  CC      arm-linux-user/accel/stubs/kvm-stub.o
  CC      m68k-linux-user/linux-user/syscall.o
  CC      alpha-linux-user/accel/tcg/user-exec.o
  CC      s390x-linux-user/accel/tcg/translator.o
  CC      sparc-linux-user/accel/stubs/kvm-stub.o
  CC      cris-softmmu/balloon.o
  CC      alpha-linux-user/accel/tcg/user-exec-stub.o
  CC      mipsel-linux-user/accel/tcg/user-exec.o
  CC      or1k-linux-user/accel/tcg/cpu-exec-common.o
  CC      sparc64-softmmu/monitor.o
  CC      sparc32plus-linux-user/accel/tcg/user-exec-stub.o
  CC      tilegx-linux-user/accel/stubs/hvf-stub.o
  CC      nios2-softmmu/balloon.o
  CC      ppc-linux-user/accel/tcg/user-exec.o
  CC      arm-softmmu/gdbstub.o
  CC      armeb-linux-user/accel/stubs/kvm-stub.o
  CC      armeb-linux-user/accel/tcg/tcg-runtime.o
  CC      s390x-linux-user/accel/tcg/user-exec.o
  CC      mips64el-linux-user/accel/tcg/user-exec-stub.o
  CC      mips-linux-user/accel/tcg/user-exec-stub.o
  CC      ppc64le-linux-user/linux-user/strace.o
  CC      s390x-softmmu/monitor.o
  CC      mipsn32el-linux-user/accel/stubs/hax-stub.o
  CC      arm-linux-user/accel/tcg/tcg-runtime.o
  CC      hppa-linux-user/linux-user/main.o
  CC      mipsn32-linux-user/accel/tcg/user-exec.o
  CC      mipsn32-linux-user/accel/tcg/user-exec-stub.o
  CC      ppc-softmmu/gdbstub.o
  CC      mips64el-linux-user/linux-user/main.o
  CC      aarch64-linux-user/accel/tcg/translate-all.o
  CC      mips64el-linux-user/linux-user/syscall.o
  CC      sh4-softmmu/gdbstub.o
  CC      sparc-linux-user/accel/tcg/tcg-runtime.o
  CC      sh4-softmmu/balloon.o
  CC      tilegx-linux-user/accel/stubs/kvm-stub.o
  CC      mipsel-softmmu/migration/ram.o
  CC      mips64el-linux-user/linux-user/strace.o
  CC      ppc64abi32-linux-user/accel/tcg/user-exec.o
  CC      cris-linux-user/linux-user/uaccess.o
  CC      or1k-softmmu/dump.o
  CC      ppc64-softmmu/ioport.o
  CC      x86_64-linux-user/accel/tcg/user-exec-stub.o
  CC      arm-linux-user/accel/tcg/cpu-exec.o
  CC      mips64-softmmu/ioport.o
  CC      sparc64-linux-user/linux-user/main.o
  CC      alpha-softmmu/gdbstub.o
  CC      aarch64_be-linux-user/accel/tcg/translate-all.o
  CC      mips64-softmmu/numa.o
  CC      mipsn32-linux-user/linux-user/main.o
  CC      mips64el-linux-user/linux-user/mmap.o
  CC      ppc-softmmu/balloon.o
  CC      ppc-linux-user/accel/tcg/user-exec-stub.o
  CC      or1k-linux-user/accel/tcg/translate-all.o
  CC      ppc64-softmmu/numa.o
  CC      mips-softmmu/gdbstub.o
  CC      m68k-linux-user/linux-user/strace.o
  CC      aarch64-softmmu/monitor.o
  CC      tricore-softmmu/migration/ram.o
  CC      sh4eb-softmmu/memory_mapping.o
  CC      sh4eb-linux-user/accel/tcg/user-exec-stub.o
  CC      mipsn32el-linux-user/accel/stubs/hvf-stub.o
  CC      ppc-softmmu/ioport.o
  CC      nios2-softmmu/ioport.o
  CC      or1k-softmmu/migration/ram.o
  CC      tilegx-linux-user/accel/tcg/tcg-runtime.o
  CC      sparc-softmmu/gdbstub.o
  CC      x86_64-softmmu/arch_init.o
  CC      tricore-softmmu/accel/accel.o
  CC      mipsel-softmmu/accel/accel.o
  CC      mips64el-softmmu/memory_mapping.o
  CC      mipsn32el-linux-user/accel/stubs/kvm-stub.o
  CC      alpha-softmmu/balloon.o
  CC      x86_64-linux-user/linux-user/main.o
  CC      sh4eb-linux-user/linux-user/main.o
  CC      i386-softmmu/arch_init.o
  CC      sparc-linux-user/accel/tcg/cpu-exec.o
  CC      mipsel-linux-user/accel/tcg/user-exec-stub.o
  CC      cris-linux-user/linux-user/uname.o
  CC      ppcemb-softmmu/monitor.o
  CC      s390x-linux-user/accel/tcg/user-exec-stub.o
  CC      sh4-softmmu/ioport.o
  CC      alpha-linux-user/linux-user/main.o
  CC      armeb-linux-user/accel/tcg/cpu-exec.o
  CC      sh4eb-linux-user/linux-user/syscall.o
  CC      sparc64-softmmu/gdbstub.o
  CC      mips64-linux-user/linux-user/syscall.o
  CC      mipsn32el-linux-user/accel/tcg/tcg-runtime.o
  CC      arm-linux-user/accel/tcg/cpu-exec-common.o
  CCAS    cris-linux-user/linux-user/safe-syscall.o
  CC      ppcemb-softmmu/gdbstub.o
  CC      tricore-softmmu/accel/stubs/hax-stub.o
  CC      cris-linux-user/target/cris/translate.o
  CC      sparc32plus-linux-user/linux-user/main.o
  CC      mips-linux-user/linux-user/main.o
  CC      sparc64-softmmu/balloon.o
  CC      sh4eb-linux-user/linux-user/strace.o
  CC      tilegx-linux-user/accel/tcg/cpu-exec.o
  CC      x86_64-softmmu/cpus.o
  CC      cris-softmmu/ioport.o
  CC      ppcemb-softmmu/balloon.o
  CC      mips64el-softmmu/dump.o
  CC      mips-linux-user/linux-user/syscall.o
  CC      sh4-linux-user/linux-user/elfload.o
  CC      ppc64-linux-user/accel/tcg/user-exec-stub.o
  CC      sparc64-softmmu/ioport.o
  CC      mipsel-linux-user/linux-user/main.o
  CC      aarch64_be-linux-user/accel/tcg/translator.o
  CC      unicore32-softmmu/numa.o
  CC      or1k-linux-user/accel/tcg/translator.o
  CC      ppc-softmmu/numa.o
  CC      mips64el-softmmu/migration/ram.o
  CC      m68k-linux-user/linux-user/mmap.o
  CC      hppa-linux-user/linux-user/syscall.o
  CC      mipsel-linux-user/linux-user/syscall.o
  CC      alpha-linux-user/linux-user/syscall.o
  CC      aarch64_be-linux-user/accel/tcg/user-exec.o
  CC      alpha-linux-user/linux-user/strace.o
  CC      sh4-linux-user/linux-user/linuxload.o
  CC      sparc64-linux-user/linux-user/syscall.o
  CC      nios2-softmmu/numa.o
  CC      mips-linux-user/linux-user/strace.o
  CC      arm-linux-user/accel/tcg/translate-all.o
  CC      i386-linux-user/target/i386/seg_helper.o
  CC      microblaze-linux-user/linux-user/signal.o
  CC      s390x-softmmu/gdbstub.o
  CC      cris-softmmu/numa.o
  CC      lm32-softmmu/qtest.o
  CC      sparc-linux-user/accel/tcg/cpu-exec-common.o
  CC      sparc-linux-user/accel/tcg/translate-all.o
  CC      xtensa-softmmu/dump.o
  CC      ppc-linux-user/linux-user/main.o
  CC      ppc64-linux-user/linux-user/main.o
  CC      s390x-linux-user/linux-user/main.o
  CC      sh4eb-softmmu/dump.o
  CC      mipsn32el-linux-user/accel/tcg/cpu-exec.o
  CC      aarch64-linux-user/accel/tcg/translator.o
  CC      mipsn32el-linux-user/accel/tcg/cpu-exec-common.o
  CC      mipsn32-linux-user/linux-user/syscall.o
  CC      sparc64-linux-user/linux-user/strace.o
  CC      sparc32plus-linux-user/linux-user/syscall.o
  CC      sh4-softmmu/numa.o
  CC      moxie-softmmu/gdbstub.o
  CC      microblaze-softmmu/gdbstub.o
  CC      hppa-linux-user/linux-user/strace.o
  CC      or1k-linux-user/accel/tcg/user-exec.o
  CC      ppc64-softmmu/qtest.o
  CC      ppc-softmmu/qtest.o
  CC      arm-linux-user/accel/tcg/translator.o
  CC      nios2-linux-user/linux-user/strace.o
  CC      mips-linux-user/linux-user/mmap.o
  CC      mips-linux-user/linux-user/signal.o
  CC      tilegx-linux-user/accel/tcg/cpu-exec-common.o
  CC      alpha-softmmu/ioport.o
  CC      x86_64-softmmu/monitor.o
  CC      x86_64-linux-user/linux-user/syscall.o
  CC      mips64-softmmu/qtest.o
  CC      unicore32-softmmu/qtest.o
  CC      tricore-softmmu/accel/stubs/hvf-stub.o
  CC      mips64-linux-user/linux-user/strace.o
  CC      ppc64abi32-linux-user/accel/tcg/user-exec-stub.o
  CC      nios2-softmmu/qtest.o
  CC      mipsel-softmmu/accel/stubs/hax-stub.o
  CC      nios2-softmmu/memory.o
  CC      mips64el-softmmu/accel/accel.o
  CC      nios2-softmmu/memory_mapping.o
  CC      armeb-linux-user/accel/tcg/cpu-exec-common.o
  CC      ppc64abi32-linux-user/linux-user/main.o
  CC      mips-softmmu/balloon.o
  CC      x86_64-linux-user/linux-user/strace.o
  CC      xtensaeb-softmmu/cpus.o
  CC      i386-softmmu/cpus.o
  CC      hppa-linux-user/linux-user/mmap.o
  CC      cris-softmmu/qtest.o
  CC      m68k-softmmu/gdbstub.o
  CC      microblazeel-softmmu/dump.o
  CC      tilegx-linux-user/accel/tcg/translate-all.o
  CC      tilegx-linux-user/accel/tcg/translator.o
  CC      mipsel-softmmu/accel/stubs/hvf-stub.o
  CC      sh4eb-softmmu/migration/ram.o
  CC      x86_64-linux-user/linux-user/mmap.o
  CC      mips64-linux-user/linux-user/mmap.o
  CC      mipsn32el-linux-user/accel/tcg/translate-all.o
  CC      mipsel-softmmu/accel/stubs/kvm-stub.o
  CC      moxie-softmmu/balloon.o
  CC      moxie-softmmu/ioport.o
  CC      tricore-softmmu/accel/stubs/kvm-stub.o
  CC      cris-softmmu/memory.o
  CC      armeb-linux-user/accel/tcg/translate-all.o
  CC      ppc64-softmmu/memory.o
  CC      s390x-linux-user/linux-user/syscall.o
  CC      alpha-softmmu/numa.o
  CC      sparc-softmmu/balloon.o
  CC      moxie-softmmu/numa.o
  CC      aarch64-linux-user/accel/tcg/user-exec.o
  CC      hppa-linux-user/linux-user/signal.o
  CC      microblaze-linux-user/linux-user/elfload.o
  CC      sparc-linux-user/accel/tcg/translator.o
  CC      sparc64-linux-user/linux-user/mmap.o
  CC      lm32-softmmu/memory.o
  CC      mips-softmmu/ioport.o
  CC      cris-softmmu/memory_mapping.o
  CC      mipsn32el-linux-user/accel/tcg/translator.o
  CC      tilegx-linux-user/accel/tcg/user-exec.o
  CC      mipsel-softmmu/accel/tcg/tcg-all.o
  CC      alpha-linux-user/linux-user/mmap.o
  CC      hppa-linux-user/linux-user/elfload.o
  CC      xtensaeb-softmmu/monitor.o
  CC      mipsn32el-linux-user/accel/tcg/user-exec.o
  CC      ppc-linux-user/linux-user/syscall.o
  CC      arm-linux-user/accel/tcg/user-exec.o
  CC      sh4-softmmu/qtest.o
  CC      x86_64-linux-user/linux-user/signal.o
  CC      arm-linux-user/accel/tcg/user-exec-stub.o
  CC      ppc-softmmu/memory.o
  CC      sparc64-softmmu/numa.o
  CC      aarch64_be-linux-user/accel/tcg/user-exec-stub.o
  CC      aarch64-linux-user/accel/tcg/user-exec-stub.o
  CC      mipsn32el-linux-user/accel/tcg/user-exec-stub.o
  CC      mips64-linux-user/linux-user/signal.o
  CC      m68k-linux-user/linux-user/signal.o
  CC      ppc64-linux-user/linux-user/syscall.o
  CC      alpha-softmmu/qtest.o
  CC      mipsel-linux-user/linux-user/strace.o
  CC      moxie-softmmu/qtest.o
  CC      tricore-softmmu/accel/tcg/tcg-all.o
  CC      or1k-softmmu/accel/accel.o
  CC      unicore32-softmmu/memory.o
  CC      hppa-linux-user/linux-user/linuxload.o
  CC      or1k-linux-user/accel/tcg/user-exec-stub.o
  CC      mips64-softmmu/memory.o
  CC      sparc-softmmu/ioport.o
  CC      s390x-softmmu/balloon.o
  CC      ppc64abi32-linux-user/linux-user/syscall.o
  CC      m68k-softmmu/balloon.o
  CC      mipsel-linux-user/linux-user/mmap.o
  CC      sparc-linux-user/accel/tcg/user-exec.o
  CC      lm32-softmmu/memory_mapping.o
  CC      mipsel-softmmu/accel/tcg/cputlb.o
  CC      cris-linux-user/target/cris/op_helper.o
  CC      alpha-softmmu/memory.o
  CC      xtensa-softmmu/migration/ram.o
  CC      sh4-softmmu/memory.o
  CC      s390x-linux-user/linux-user/strace.o
  CC      nios2-softmmu/dump.o
  CC      aarch64_be-linux-user/linux-user/main.o
  CC      microblaze-softmmu/balloon.o
  CC      m68k-linux-user/linux-user/elfload.o
  CC      s390x-softmmu/ioport.o
  CC      arm-softmmu/balloon.o
  CC      mipsel-softmmu/accel/tcg/tcg-runtime.o
  CC      nios2-linux-user/linux-user/mmap.o
  CC      sh4-linux-user/linux-user/uaccess.o
  CC      unicore32-softmmu/memory_mapping.o
  CC      tricore-softmmu/accel/tcg/cputlb.o
  CC      or1k-linux-user/linux-user/main.o
  CC      alpha-softmmu/memory_mapping.o
  CC      microblazeel-softmmu/migration/ram.o
  CC      arm-linux-user/linux-user/main.o
  CC      ppc64le-linux-user/linux-user/mmap.o
  CC      s390x-softmmu/numa.o
  CC      or1k-softmmu/accel/stubs/hax-stub.o
  CC      mips-softmmu/numa.o
  CC      aarch64-softmmu/gdbstub.o
  CC      xtensaeb-softmmu/gdbstub.o
  CC      mips64el-softmmu/accel/stubs/hax-stub.o
  CC      mips64el-softmmu/accel/stubs/hvf-stub.o
  CC      m68k-softmmu/ioport.o
  CC      cris-linux-user/target/cris/helper.o
  CC      moxie-softmmu/memory.o
  CC      mips64-softmmu/memory_mapping.o
  CC      hppa-linux-user/linux-user/uaccess.o
  CC      i386-softmmu/monitor.o
  CC      m68k-softmmu/numa.o
  CC      nios2-softmmu/migration/ram.o
  CC      m68k-linux-user/linux-user/linuxload.o
  CC      arm-softmmu/ioport.o
  CC      microblaze-softmmu/ioport.o
  CC      aarch64-softmmu/balloon.o
  CC      arm-softmmu/numa.o
  CC      armeb-linux-user/accel/tcg/translator.o
  CC      sparc64-softmmu/qtest.o
  CC      aarch64-softmmu/ioport.o
  CC      aarch64-softmmu/numa.o
  CC      sparc-softmmu/numa.o
  CC      arm-linux-user/linux-user/syscall.o
  CC      ppcemb-softmmu/ioport.o
  CC      or1k-softmmu/accel/stubs/hvf-stub.o
  CC      mips64el-softmmu/accel/stubs/kvm-stub.o
  CC      microblazeel-softmmu/accel/accel.o
  CC      armeb-linux-user/accel/tcg/user-exec.o
  CC      microblazeel-softmmu/accel/stubs/hax-stub.o
  CC      cris-linux-user/target/cris/cpu.o
  CC      mipsn32el-linux-user/linux-user/main.o
  CC      ppc64abi32-linux-user/linux-user/strace.o
  CC      sparc32plus-linux-user/linux-user/strace.o
  CC      x86_64-linux-user/linux-user/elfload.o
  CC      tricore-softmmu/accel/tcg/tcg-runtime.o
  CC      i386-linux-user/target/i386/smm_helper.o
  CC      sh4eb-linux-user/linux-user/mmap.o
  CC      mipsel-linux-user/linux-user/signal.o
  CC      moxie-softmmu/memory_mapping.o
  CC      mipsel-softmmu/accel/tcg/cpu-exec.o
  CC      aarch64-softmmu/qtest.o
  CC      i386-softmmu/gdbstub.o
  CC      tilegx-linux-user/accel/tcg/user-exec-stub.o
  CC      aarch64_be-linux-user/linux-user/syscall.o
  CC      mipsn32-linux-user/linux-user/strace.o
  CC      m68k-linux-user/linux-user/uaccess.o
  CC      m68k-linux-user/linux-user/uname.o
  CC      aarch64-linux-user/linux-user/main.o
  CC      nios2-softmmu/accel/accel.o
  CC      s390x-softmmu/qtest.o
  CC      sh4eb-linux-user/linux-user/signal.o
  CC      xtensa-softmmu/accel/accel.o
  CC      nios2-softmmu/accel/stubs/hax-stub.o
  CC      xtensaeb-softmmu/balloon.o
  CC      microblazeel-softmmu/accel/stubs/hvf-stub.o
  CC      microblazeel-softmmu/accel/stubs/kvm-stub.o
  CC      mips64-linux-user/linux-user/elfload.o
  CC      sparc64-softmmu/memory.o
  CC      or1k-linux-user/linux-user/syscall.o
  CC      or1k-softmmu/accel/stubs/kvm-stub.o
  CC      microblazeel-softmmu/accel/tcg/tcg-all.o
  CC      sh4eb-linux-user/linux-user/elfload.o
  CC      mips-softmmu/qtest.o
  CC      tilegx-linux-user/linux-user/main.o
  CC      x86_64-softmmu/gdbstub.o
  CC      mipsn32-linux-user/linux-user/mmap.o
  CC      lm32-softmmu/dump.o
  CC      mips64el-softmmu/accel/tcg/tcg-all.o
  CC      sh4eb-softmmu/accel/accel.o
  CCAS    m68k-linux-user/linux-user/safe-syscall.o
  CC      mipsel-softmmu/accel/tcg/cpu-exec-common.o
  CC      sparc32plus-linux-user/linux-user/mmap.o
  CC      tilegx-linux-user/linux-user/syscall.o
  CC      microblaze-softmmu/numa.o
  CC      sparc-linux-user/accel/tcg/user-exec-stub.o
  CC      mipsn32el-linux-user/linux-user/syscall.o
  CC      m68k-softmmu/qtest.o
  CC      m68k-softmmu/memory.o
  CC      armeb-linux-user/accel/tcg/user-exec-stub.o
  CC      ppc64le-linux-user/linux-user/signal.o
  CC      nios2-softmmu/accel/stubs/hvf-stub.o
  CC      sparc-softmmu/qtest.o
  CC      ppcemb-softmmu/numa.o
  CC      aarch64-softmmu/memory.o
  CC      xtensa-softmmu/accel/stubs/hax-stub.o
  CC      tilegx-linux-user/linux-user/strace.o
  CC      ppc64abi32-linux-user/linux-user/mmap.o
  CC      hppa-linux-user/linux-user/uname.o
  CC      aarch64_be-linux-user/linux-user/strace.o
  CC      arm-softmmu/qtest.o
  CC      m68k-softmmu/memory_mapping.o
  CC      aarch64-softmmu/memory_mapping.o
  CC      or1k-softmmu/accel/tcg/tcg-all.o
  CC      cris-linux-user/target/cris/gdbstub.o
  CC      nios2-linux-user/linux-user/signal.o
  CC      sh4eb-softmmu/accel/stubs/hax-stub.o
  CC      mips-softmmu/memory.o
  CC      nios2-linux-user/linux-user/elfload.o
  CC      mips64el-softmmu/accel/tcg/cputlb.o
  CC      mips64-linux-user/linux-user/linuxload.o
  CC      m68k-softmmu/dump.o
  CC      sparc-linux-user/linux-user/main.o
  CC      or1k-linux-user/linux-user/strace.o
  GEN     trace/generated-helpers.c
  CC      xtensa-softmmu/accel/stubs/hvf-stub.o
  CC      sh4-linux-user/linux-user/uname.o
  CC      aarch64_be-linux-user/linux-user/mmap.o
  CC      moxie-softmmu/dump.o
  CC      or1k-softmmu/accel/tcg/cputlb.o
  CC      arm-softmmu/memory.o
  CC      mips64-linux-user/linux-user/uaccess.o
  CC      i386-softmmu/balloon.o
  CC      i386-softmmu/ioport.o
  CC      sh4eb-softmmu/accel/stubs/hvf-stub.o
  CC      s390x-softmmu/memory.o
  CC      arm-softmmu/memory_mapping.o
  CC      mips64el-softmmu/accel/tcg/tcg-runtime.o
  CC      aarch64-linux-user/linux-user/syscall.o
  CC      lm32-softmmu/migration/ram.o
  CC      microblaze-softmmu/qtest.o
  CC      mipsn32el-linux-user/linux-user/strace.o
  CC      mipsel-softmmu/accel/tcg/translate-all.o
  CC      xtensa-softmmu/accel/stubs/kvm-stub.o
  CC      or1k-softmmu/accel/tcg/tcg-runtime.o
  CC      sparc32plus-linux-user/linux-user/signal.o
  CC      xtensa-softmmu/accel/tcg/tcg-all.o
  CC      armeb-linux-user/linux-user/main.o
  CC      sh4eb-softmmu/accel/stubs/kvm-stub.o
  CC      sparc-softmmu/memory.o
  CC      ppcemb-softmmu/qtest.o
  CC      sparc64-softmmu/memory_mapping.o
  CC      mipsel-softmmu/accel/tcg/translator.o
  CC      mipsn32-linux-user/linux-user/signal.o
  CC      mipsn32-linux-user/linux-user/elfload.o
  CC      s390x-softmmu/memory_mapping.o
  CC      arm-linux-user/linux-user/strace.o
  CC      aarch64_be-linux-user/linux-user/signal.o
  CC      tilegx-linux-user/linux-user/mmap.o
  CC      sh4eb-linux-user/linux-user/linuxload.o
  CC      nios2-softmmu/accel/stubs/kvm-stub.o
  CC      aarch64_be-linux-user/linux-user/elfload.o
  CC      ppc64-softmmu/memory_mapping.o
  CC      microblazeel-softmmu/accel/tcg/cputlb.o
  CC      xtensaeb-softmmu/ioport.o
  CC      cris-linux-user/trace/control-target.o
  CC      sparc-softmmu/memory_mapping.o
  CC      lm32-softmmu/accel/accel.o
  CC      ppcemb-softmmu/memory.o
  CC      or1k-softmmu/accel/tcg/cpu-exec.o
  CC      or1k-softmmu/accel/tcg/cpu-exec-common.o
  CC      unicore32-softmmu/dump.o
  CC      sparc-linux-user/linux-user/syscall.o
  CC      xtensa-softmmu/accel/tcg/cputlb.o
  CC      m68k-softmmu/migration/ram.o
  CC      sparc64-linux-user/linux-user/signal.o
  CC      ppc64-linux-user/linux-user/strace.o
  CC      ppcemb-softmmu/memory_mapping.o
  CC      sh4eb-softmmu/accel/tcg/tcg-all.o
  CC      nios2-softmmu/accel/tcg/tcg-all.o
  CC      alpha-softmmu/dump.o
  CC      sh4eb-linux-user/linux-user/uaccess.o
  CC      cris-linux-user/trace/generated-helpers.o
  CC      cris-softmmu/dump.o
  CC      ppc64le-linux-user/linux-user/elfload.o
  CC      aarch64-softmmu/dump.o
  CC      aarch64-linux-user/linux-user/strace.o
  CCAS    sh4-linux-user/linux-user/safe-syscall.o
  CC      xtensaeb-softmmu/numa.o
  CC      microblazeel-linux-user/linux-user/mmap.o
  CC      alpha-softmmu/migration/ram.o
  CC      microblazeel-softmmu/accel/tcg/tcg-runtime.o
  CC      sparc-softmmu/dump.o
  CC      mips64el-linux-user/linux-user/signal.o
  CC      ppc64-linux-user/linux-user/mmap.o
  CC      ppc64-softmmu/dump.o
  CC      ppc64le-linux-user/linux-user/linuxload.o
  CC      microblaze-softmmu/memory.o
  CC      microblaze-softmmu/memory_mapping.o
  CC      mips64-linux-user/linux-user/uname.o
  CC      arm-softmmu/dump.o
  CC      x86_64-softmmu/balloon.o
  CC      ppc-softmmu/memory_mapping.o
  CC      nios2-softmmu/accel/tcg/cputlb.o
  CC      sparc32plus-linux-user/linux-user/elfload.o
  CC      sh4eb-softmmu/accel/tcg/cputlb.o
  CC      xtensaeb-softmmu/qtest.o
  CC      ppc-softmmu/dump.o
  CC      ppcemb-softmmu/dump.o
  CC      s390x-softmmu/dump.o
  CCAS    hppa-linux-user/linux-user/safe-syscall.o
  CC      microblaze-linux-user/linux-user/linuxload.o
  CC      mips64el-linux-user/linux-user/elfload.o
  CC      ppc64le-linux-user/linux-user/uaccess.o
  CC      alpha-softmmu/accel/accel.o
  CC      mips-softmmu/memory_mapping.o
  CC      ppc64-linux-user/linux-user/signal.o
  LINK    cris-linux-user/qemu-cris
  CC      armeb-linux-user/linux-user/syscall.o
  CC      microblaze-linux-user/linux-user/uaccess.o
  CC      sh4-softmmu/memory_mapping.o
  CC      m68k-linux-user/linux-user/flatload.o
  CC      nios2-linux-user/linux-user/linuxload.o
  CC      mipsn32-linux-user/linux-user/linuxload.o
  CC      x86_64-softmmu/ioport.o
  CC      ppc64abi32-linux-user/linux-user/signal.o
  CC      cris-softmmu/migration/ram.o
  CC      aarch64-softmmu/migration/ram.o
  CC      aarch64_be-linux-user/linux-user/linuxload.o
  CC      aarch64-softmmu/accel/accel.o
  CC      sparc64-softmmu/dump.o
  CC      tilegx-linux-user/linux-user/signal.o
  CC      aarch64_be-linux-user/linux-user/uaccess.o
  CC      moxie-softmmu/migration/ram.o
  CC      tilegx-linux-user/linux-user/elfload.o
  CC      aarch64-softmmu/accel/stubs/hax-stub.o
  CC      aarch64-linux-user/linux-user/mmap.o
  CC      aarch64-linux-user/linux-user/signal.o
  CC      alpha-linux-user/linux-user/signal.o
  CC      mips64-softmmu/dump.o
  CC      x86_64-softmmu/numa.o
  CC      cris-softmmu/accel/accel.o
  CC      xtensaeb-softmmu/memory.o
  CC      i386-softmmu/numa.o
  CC      ppc64-linux-user/linux-user/elfload.o
  CC      sh4-softmmu/dump.o
  CC      m68k-softmmu/accel/accel.o
  CC      alpha-linux-user/linux-user/elfload.o
  CC      ppcemb-softmmu/migration/ram.o
  CC      sh4eb-softmmu/accel/tcg/tcg-runtime.o
  CC      x86_64-softmmu/qtest.o
  CC      nios2-linux-user/linux-user/uaccess.o
  CC      ppc64le-linux-user/linux-user/uname.o
  CC      mipsn32el-linux-user/linux-user/mmap.o
  CC      nios2-softmmu/accel/tcg/tcg-runtime.o
  CC      xtensa-softmmu/accel/tcg/tcg-runtime.o
  CC      sparc64-linux-user/linux-user/elfload.o
  CC      ppcemb-softmmu/accel/accel.o
  CC      sh4-softmmu/migration/ram.o
  CC      nios2-softmmu/accel/tcg/cpu-exec.o
  CC      mips64el-linux-user/linux-user/linuxload.o
  CC      nios2-softmmu/accel/tcg/cpu-exec-common.o
  CC      nios2-softmmu/accel/tcg/translate-all.o
  CC      ppc64abi32-linux-user/linux-user/elfload.o
  CC      mipsn32-linux-user/linux-user/uaccess.o
  CC      unicore32-softmmu/migration/ram.o
  CC      m68k-linux-user/linux-user/m68k-sim.o
  CC      s390x-linux-user/linux-user/mmap.o
  CC      moxie-softmmu/accel/accel.o
  CC      hppa-linux-user/target/hppa/translate.o
  CC      tricore-softmmu/accel/tcg/cpu-exec.o
  CC      hppa-linux-user/target/hppa/helper.o
  CCAS    mips64-linux-user/linux-user/safe-syscall.o
  CC      microblazeel-softmmu/accel/tcg/cpu-exec.o
  CC      mips-softmmu/dump.o
  CC      hppa-linux-user/target/hppa/cpu.o
  CC      or1k-linux-user/linux-user/mmap.o
  CC      mipsel-softmmu/hw/9pfs/virtio-9p-device.o
  CC      sh4eb-linux-user/linux-user/uname.o
  CC      hppa-linux-user/target/hppa/op_helper.o
  CC      ppcemb-softmmu/accel/kvm/kvm-all.o
  CC      lm32-softmmu/accel/stubs/hax-stub.o
  CC      microblaze-softmmu/dump.o
  CC      ppc-softmmu/migration/ram.o
  CC      ppc64abi32-linux-user/linux-user/linuxload.o
  CC      arm-softmmu/migration/ram.o
  CC      sh4-linux-user/linux-user/flatload.o
  CC      or1k-softmmu/accel/tcg/translate-all.o
  CC      alpha-linux-user/linux-user/linuxload.o
  CC      ppc64-softmmu/migration/ram.o
  CC      alpha-linux-user/linux-user/uaccess.o
  CC      mipsel-linux-user/linux-user/elfload.o
  CC      s390x-linux-user/linux-user/signal.o
  CC      mipsn32-linux-user/linux-user/uname.o
  CC      ppc64abi32-linux-user/linux-user/uaccess.o
  CC      sh4-linux-user/target/sh4/translate.o
  CC      mipsn32el-linux-user/linux-user/signal.o
  CC      microblaze-linux-user/linux-user/uname.o
  CC      m68k-softmmu/accel/stubs/hax-stub.o
  CC      i386-softmmu/qtest.o
  CC      sparc64-linux-user/linux-user/linuxload.o
  CC      x86_64-linux-user/linux-user/linuxload.o
  CC      or1k-softmmu/accel/tcg/translator.o
  CC      or1k-softmmu/hw/core/generic-loader.o
  CC      ppc-linux-user/linux-user/strace.o
  CC      s390x-linux-user/linux-user/elfload.o
  CC      mipsel-linux-user/linux-user/linuxload.o
  CC      s390x-softmmu/migration/ram.o
  CC      i386-softmmu/memory.o
  CC      mipsn32el-linux-user/linux-user/elfload.o
  CC      mipsn32el-linux-user/linux-user/linuxload.o
  CC      x86_64-linux-user/linux-user/uaccess.o
  CC      lm32-softmmu/accel/stubs/hvf-stub.o
  CC      sparc64-softmmu/migration/ram.o
  CC      sh4eb-softmmu/accel/tcg/cpu-exec.o
  CC      s390x-softmmu/accel/accel.o
  CC      mipsel-softmmu/hw/block/virtio-blk.o
  CC      m68k-softmmu/accel/stubs/hvf-stub.o
  CC      nios2-linux-user/linux-user/uname.o
  CC      tilegx-linux-user/linux-user/linuxload.o
  CC      ppc64-softmmu/accel/accel.o
  CC      mipsel-linux-user/linux-user/uaccess.o
  CC      mips64-linux-user/target/mips/translate.o
  CC      tricore-softmmu/accel/tcg/cpu-exec-common.o
  CC      mips64-softmmu/migration/ram.o
  CC      tilegx-linux-user/linux-user/uaccess.o
  CC      x86_64-softmmu/memory.o
  CC      sh4-linux-user/target/sh4/op_helper.o
  CC      tilegx-linux-user/linux-user/uname.o
  CCAS    mipsn32-linux-user/linux-user/safe-syscall.o
  CC      xtensaeb-softmmu/memory_mapping.o
  CC      lm32-softmmu/accel/stubs/kvm-stub.o
  CC      microblaze-softmmu/migration/ram.o
  CC      alpha-softmmu/accel/stubs/hax-stub.o
  CC      m68k-linux-user/target/m68k/m68k-semi.o
  CC      moxie-softmmu/accel/stubs/hax-stub.o
  CC      xtensaeb-softmmu/dump.o
  CC      cris-softmmu/accel/stubs/hax-stub.o
  CC      arm-linux-user/linux-user/mmap.o
  CC      m68k-softmmu/accel/stubs/kvm-stub.o
  CC      or1k-linux-user/linux-user/signal.o
  CC      moxie-softmmu/accel/stubs/hvf-stub.o
  CC      lm32-softmmu/accel/tcg/tcg-all.o
  CCAS    sh4eb-linux-user/linux-user/safe-syscall.o
  CC      s390x-softmmu/accel/stubs/hax-stub.o
  CC      aarch64-linux-user/linux-user/elfload.o
  CC      m68k-linux-user/target/m68k/translate.o
  CC      x86_64-softmmu/memory_mapping.o
  CC      s390x-softmmu/accel/stubs/hvf-stub.o
  CC      mipsel-linux-user/linux-user/uname.o
  CC      tricore-softmmu/accel/tcg/translate-all.o
  CCAS    microblaze-linux-user/linux-user/safe-syscall.o
  CC      x86_64-softmmu/dump.o
  CC      sh4-softmmu/accel/accel.o
  CC      s390x-linux-user/linux-user/linuxload.o
  CC      aarch64-linux-user/linux-user/linuxload.o
  CC      mips-linux-user/linux-user/elfload.o
  CC      sparc-softmmu/migration/ram.o
  CC      alpha-softmmu/accel/stubs/hvf-stub.o
  CC      mips-linux-user/linux-user/linuxload.o
  CC      ppc64-linux-user/linux-user/linuxload.o
  CC      m68k-softmmu/accel/tcg/tcg-all.o
  CCAS    mipsel-linux-user/linux-user/safe-syscall.o
  CC      sh4eb-linux-user/linux-user/flatload.o
  CC      moxie-softmmu/accel/stubs/kvm-stub.o
  CC      ppc-linux-user/linux-user/mmap.o
  CC      armeb-linux-user/linux-user/strace.o
  CC      sparc64-softmmu/accel/accel.o
  CC      microblaze-linux-user/linux-user/flatload.o
  CC      ppcemb-softmmu/accel/stubs/hax-stub.o
  CC      lm32-softmmu/accel/tcg/cputlb.o
  CC      mipsn32el-linux-user/linux-user/uaccess.o
  CC      alpha-softmmu/accel/stubs/kvm-stub.o
  CC      mips64el-linux-user/linux-user/uaccess.o
  CC      sparc-softmmu/accel/accel.o
  CC      mips64-linux-user/target/mips/dsp_helper.o
  CC      xtensaeb-softmmu/migration/ram.o
  CC      mips64el-softmmu/accel/tcg/cpu-exec.o
  CC      mips-softmmu/migration/ram.o
  CC      mips64-softmmu/accel/accel.o
  CC      tricore-softmmu/accel/tcg/translator.o
  CC      x86_64-softmmu/migration/ram.o
  CC      or1k-softmmu/hw/core/null-machine.o
  CC      m68k-softmmu/accel/tcg/cputlb.o
  CC      sh4eb-linux-user/target/sh4/translate.o
  CC      moxie-softmmu/accel/tcg/tcg-all.o
  CCAS    nios2-linux-user/linux-user/safe-syscall.o
  CC      unicore32-softmmu/accel/accel.o
  CC      s390x-linux-user/linux-user/uaccess.o
  CC      sh4-softmmu/accel/stubs/hax-stub.o
  CC      arm-linux-user/linux-user/signal.o
  CC      cris-softmmu/accel/stubs/hvf-stub.o
  CC      nios2-linux-user/target/nios2/translate.o
  CC      sparc32plus-linux-user/linux-user/linuxload.o
  CC      sh4-linux-user/target/sh4/helper.o
  CC      ppc-linux-user/linux-user/signal.o
  CC      sh4eb-linux-user/target/sh4/op_helper.o
  CC      x86_64-softmmu/accel/accel.o
  CC      aarch64-softmmu/accel/stubs/hvf-stub.o
  CCAS    tilegx-linux-user/linux-user/safe-syscall.o
  CC      sparc32plus-linux-user/linux-user/uaccess.o
  CC      sparc32plus-linux-user/linux-user/uname.o
  CC      alpha-softmmu/accel/tcg/tcg-all.o
  CC      aarch64_be-linux-user/linux-user/uname.o
  CC      ppcemb-softmmu/accel/stubs/hvf-stub.o
  CC      tricore-softmmu/hw/core/generic-loader.o
  CC      x86_64-linux-user/linux-user/uname.o
  CC      sh4eb-softmmu/accel/tcg/cpu-exec-common.o
  CCAS    aarch64_be-linux-user/linux-user/safe-syscall.o
  CC      sh4-linux-user/target/sh4/cpu.o
  CC      sparc64-linux-user/linux-user/uaccess.o
  CC      or1k-softmmu/hw/intc/ompic.o
  CC      aarch64-softmmu/accel/stubs/kvm-stub.o
  CC      sh4-softmmu/accel/stubs/hvf-stub.o
  CC      mipsel-linux-user/target/mips/translate.o
  CC      mips64-linux-user/target/mips/op_helper.o
  CC      tricore-softmmu/hw/core/null-machine.o
  CC      tricore-softmmu/hw/misc/mmio_interface.o
  CC      sparc-linux-user/linux-user/strace.o
  CC      sparc-linux-user/linux-user/mmap.o
  CC      sh4-linux-user/target/sh4/gdbstub.o
  CC      mipsel-softmmu/hw/block/dataplane/virtio-blk.o
  CC      mipsn32el-linux-user/linux-user/uname.o
  CC      moxie-softmmu/accel/tcg/cputlb.o
  CC      arm-softmmu/accel/accel.o
  CC      x86_64-softmmu/accel/stubs/hax-stub.o
  CC      cris-softmmu/accel/stubs/kvm-stub.o
  CCAS    sparc32plus-linux-user/linux-user/safe-syscall.o
  CC      m68k-linux-user/target/m68k/op_helper.o
  CC      mips-softmmu/accel/accel.o
  CC      mipsel-softmmu/hw/char/virtio-serial-bus.o
  CC      cris-softmmu/accel/tcg/tcg-all.o
  CC      unicore32-softmmu/accel/stubs/hax-stub.o
  CC      alpha-softmmu/accel/tcg/cputlb.o
  CC      s390x-linux-user/linux-user/uname.o
  CC      mipsel-softmmu/hw/core/generic-loader.o
  CC      mips64el-softmmu/accel/tcg/cpu-exec-common.o
  CC      mipsel-softmmu/hw/core/null-machine.o
  CC      xtensa-softmmu/accel/tcg/cpu-exec.o
  CC      mips64el-softmmu/accel/tcg/translate-all.o
  CC      tilegx-linux-user/target/tilegx/cpu.o
  CC      microblazeel-softmmu/accel/tcg/cpu-exec-common.o
  CC      ppc64abi32-linux-user/linux-user/uname.o
  CC      aarch64-softmmu/accel/tcg/tcg-all.o
  CC      ppc64-linux-user/linux-user/uaccess.o
  CC      ppc-softmmu/accel/accel.o
  CC      sh4-softmmu/accel/stubs/kvm-stub.o
  CC      or1k-softmmu/hw/misc/mmio_interface.o
  CC      tricore-softmmu/hw/net/vhost_net.o
  CC      or1k-linux-user/linux-user/elfload.o
  CC      sparc32plus-linux-user/target/sparc/translate.o
  CC      ppc64-softmmu/accel/kvm/kvm-all.o
  CC      mips-linux-user/linux-user/uaccess.o
  GEN     trace/generated-helpers.c
  CC      x86_64-softmmu/accel/stubs/hvf-stub.o
  CC      lm32-softmmu/accel/tcg/tcg-runtime.o
  CC      tricore-softmmu/hw/net/rocker/qmp-norocker.o
  CC      alpha-softmmu/accel/tcg/tcg-runtime.o
  CC      unicore32-softmmu/accel/stubs/hvf-stub.o
  CC      tilegx-linux-user/target/tilegx/translate.o
  CC      i386-softmmu/memory_mapping.o
  CC      mips64el-linux-user/linux-user/uname.o
  CC      cris-softmmu/accel/tcg/cputlb.o
  CC      nios2-softmmu/accel/tcg/translator.o
  CC      aarch64-softmmu/accel/tcg/cputlb.o
  CC      ppcemb-softmmu/accel/tcg/tcg-all.o
  CC      microblaze-softmmu/accel/accel.o
  CC      sparc64-softmmu/accel/stubs/hax-stub.o
  CC      nios2-softmmu/hw/core/generic-loader.o
  CC      ppcemb-softmmu/accel/tcg/cputlb.o
  CC      unicore32-softmmu/accel/stubs/kvm-stub.o
  CC      microblaze-linux-user/target/microblaze/translate.o
  CC      microblazeel-softmmu/accel/tcg/translate-all.o
  CCAS    ppc64le-linux-user/linux-user/safe-syscall.o
  CC      mips-softmmu/accel/stubs/hax-stub.o
  CC      nios2-linux-user/target/nios2/op_helper.o
  CC      alpha-linux-user/linux-user/uname.o
  CC      m68k-softmmu/accel/tcg/tcg-runtime.o
  CC      ppc64le-linux-user/target/ppc/cpu-models.o
  CC      arm-softmmu/accel/stubs/hax-stub.o
  CC      or1k-softmmu/hw/net/vhost_net.o
  CC      sparc-linux-user/linux-user/signal.o
  CC      ppc64le-linux-user/target/ppc/cpu.o
  CC      i386-softmmu/dump.o
  CC      microblaze-linux-user/target/microblaze/op_helper.o
  CCAS    ppc64abi32-linux-user/linux-user/safe-syscall.o
  CC      ppc64abi32-linux-user/target/ppc/cpu-models.o
  CC      sh4-softmmu/accel/tcg/tcg-all.o
  CC      xtensaeb-softmmu/accel/accel.o
  CC      mips64el-softmmu/accel/tcg/translator.o
  CC      unicore32-softmmu/accel/tcg/tcg-all.o
  CC      mips64-linux-user/target/mips/lmi_helper.o
  CC      tricore-softmmu/hw/vfio/common.o
  CCAS    x86_64-linux-user/linux-user/safe-syscall.o
  CC      xtensa-softmmu/accel/tcg/cpu-exec-common.o
  CC      xtensa-softmmu/accel/tcg/translate-all.o
  CC      armeb-linux-user/linux-user/mmap.o
  CC      x86_64-softmmu/accel/stubs/kvm-stub.o
  CC      sh4-softmmu/accel/tcg/cputlb.o
  CCAS    alpha-linux-user/linux-user/safe-syscall.o
  CC      ppc-softmmu/accel/kvm/kvm-all.o
  CC      ppc-linux-user/linux-user/elfload.o
  CC      arm-softmmu/accel/stubs/hvf-stub.o
  CC      arm-softmmu/accel/stubs/kvm-stub.o
  CC      ppc64le-linux-user/target/ppc/translate.o
  CC      x86_64-softmmu/accel/tcg/tcg-all.o
  CC      microblaze-softmmu/accel/stubs/hax-stub.o
  CC      mips-linux-user/linux-user/uname.o
  CC      sparc-softmmu/accel/stubs/hax-stub.o
  CC      microblaze-softmmu/accel/stubs/hvf-stub.o
  CC      ppc64-softmmu/accel/stubs/hax-stub.o
  CCAS    mips64el-linux-user/linux-user/safe-syscall.o
  CC      or1k-linux-user/linux-user/linuxload.o
  CC      s390x-softmmu/accel/stubs/kvm-stub.o
  CC      sparc64-softmmu/accel/stubs/hvf-stub.o
  CC      mips64el-linux-user/target/mips/translate.o
  CC      sparc64-softmmu/accel/stubs/kvm-stub.o
  CC      nios2-linux-user/target/nios2/helper.o
  CCAS    mipsn32el-linux-user/linux-user/safe-syscall.o
  CC      mips64-softmmu/accel/stubs/hax-stub.o
  CC      ppc64-softmmu/accel/stubs/hvf-stub.o
  CC      ppc64-softmmu/accel/tcg/tcg-all.o
  CC      or1k-softmmu/hw/net/rocker/qmp-norocker.o
  CC      sparc-linux-user/linux-user/elfload.o
  CC      unicore32-softmmu/accel/tcg/cputlb.o
  CC      arm-softmmu/accel/tcg/tcg-all.o
  CC      nios2-softmmu/hw/core/null-machine.o
  CC      mips64-softmmu/accel/stubs/hvf-stub.o
  CCAS    s390x-linux-user/linux-user/safe-syscall.o
  CC      sparc64-softmmu/accel/tcg/tcg-all.o
  CC      microblazeel-linux-user/linux-user/signal.o
  CC      microblaze-linux-user/target/microblaze/helper.o
  CC      aarch64_be-linux-user/linux-user/flatload.o
  CC      sparc-softmmu/accel/stubs/hvf-stub.o
  CC      sh4eb-softmmu/accel/tcg/translate-all.o
  CC      mipsel-softmmu/hw/display/vga.o
  CC      ppc-softmmu/accel/stubs/hax-stub.o
  CC      microblaze-softmmu/accel/stubs/kvm-stub.o
  CC      ppcemb-softmmu/accel/tcg/tcg-runtime.o
  CC      aarch64_be-linux-user/target/arm/arm-semi.o
  CC      ppc-linux-user/linux-user/linuxload.o
  CC      ppc-linux-user/linux-user/uaccess.o
  CC      m68k-linux-user/target/m68k/helper.o
  CC      tilegx-linux-user/target/tilegx/helper.o
  CC      or1k-linux-user/linux-user/uaccess.o
  CC      nios2-linux-user/target/nios2/cpu.o
  CC      or1k-linux-user/linux-user/uname.o
  CC      s390x-softmmu/accel/tcg/tcg-all.o
  CC      armeb-linux-user/linux-user/signal.o
  CC      or1k-softmmu/hw/vfio/common.o
  CC      alpha-linux-user/target/alpha/translate.o
  CC      mips64-softmmu/accel/stubs/kvm-stub.o
  CC      aarch64-linux-user/linux-user/uaccess.o
  CC      nios2-linux-user/target/nios2/mmu.o
  CC      x86_64-linux-user/target/i386/helper.o
  CC      m68k-softmmu/accel/tcg/cpu-exec.o
  CC      sh4eb-softmmu/accel/tcg/translator.o
  CC      armeb-linux-user/linux-user/elfload.o
  CC      arm-softmmu/accel/tcg/cputlb.o
  CC      sparc64-softmmu/accel/tcg/cputlb.o
  CC      tricore-softmmu/hw/vfio/platform.o
  CC      mipsn32-linux-user/target/mips/translate.o
  CC      armeb-linux-user/linux-user/linuxload.o
  CC      nios2-softmmu/hw/intc/nios2_iic.o
  CC      i386-linux-user/target/i386/svm_helper.o
  CC      mips-softmmu/accel/stubs/hvf-stub.o
  CC      microblazeel-softmmu/accel/tcg/translator.o
  CC      moxie-softmmu/accel/tcg/tcg-runtime.o
  CC      mipsel-linux-user/target/mips/dsp_helper.o
  GEN     trace/generated-helpers.c
  CC      s390x-softmmu/accel/tcg/cputlb.o
  CC      sparc-softmmu/accel/stubs/kvm-stub.o
  CC      lm32-softmmu/accel/tcg/cpu-exec.o
  CC      mipsn32el-linux-user/target/mips/translate.o
  CC      microblaze-softmmu/accel/tcg/tcg-all.o
  CC      sparc64-softmmu/accel/tcg/tcg-runtime.o
  CC      lm32-softmmu/accel/tcg/cpu-exec-common.o
  CC      alpha-linux-user/target/alpha/helper.o
  CC      ppc-linux-user/linux-user/uname.o
  CC      s390x-linux-user/target/s390x/cpu.o
  CC      s390x-softmmu/accel/tcg/tcg-runtime.o
  CC      sparc-linux-user/linux-user/linuxload.o
  CC      xtensaeb-softmmu/accel/stubs/hax-stub.o
  CC      alpha-softmmu/accel/tcg/cpu-exec.o
  CC      ppc64-linux-user/linux-user/uname.o
  CC      lm32-softmmu/accel/tcg/translate-all.o
  CC      mips64-softmmu/accel/tcg/tcg-all.o
  CC      nios2-linux-user/trace/control-target.o
  CC      mips-softmmu/accel/stubs/kvm-stub.o
  CC      xtensa-softmmu/accel/tcg/translator.o
  CC      mips64-softmmu/accel/tcg/cputlb.o
  CC      sparc32plus-linux-user/target/sparc/helper.o
  CC      sh4eb-linux-user/target/sh4/helper.o
  CC      nios2-softmmu/hw/misc/mmio_interface.o
  CC      mips64el-softmmu/hw/9pfs/virtio-9p-device.o
  CC      or1k-softmmu/hw/vfio/platform.o
  CC      m68k-softmmu/accel/tcg/cpu-exec-common.o
  CC      lm32-softmmu/accel/tcg/translator.o
  CC      microblaze-softmmu/accel/tcg/cputlb.o
  CC      microblaze-softmmu/accel/tcg/tcg-runtime.o
  CC      sparc-softmmu/accel/tcg/tcg-all.o
  CC      xtensaeb-softmmu/accel/stubs/hvf-stub.o
  CC      sh4eb-softmmu/hw/9pfs/virtio-9p-device.o
  CC      microblazeel-softmmu/hw/core/generic-loader.o
  CC      mips-softmmu/accel/tcg/tcg-all.o
  CC      mipsel-linux-user/target/mips/op_helper.o
  CC      moxie-softmmu/accel/tcg/cpu-exec.o
  CC      sparc64-softmmu/accel/tcg/cpu-exec.o
  CC      cris-softmmu/accel/tcg/tcg-runtime.o
  CC      ppc64le-linux-user/target/ppc/kvm-stub.o
  CC      microblazeel-linux-user/linux-user/elfload.o
  CC      ppcemb-softmmu/accel/tcg/cpu-exec.o
  CC      x86_64-linux-user/target/i386/cpu.o
  CC      ppc64abi32-linux-user/target/ppc/cpu.o
  CC      microblaze-softmmu/accel/tcg/cpu-exec.o
  CC      m68k-softmmu/accel/tcg/translate-all.o
  CC      arm-linux-user/linux-user/elfload.o
  CC      nios2-linux-user/trace/generated-helpers.o
  CCAS    ppc-linux-user/linux-user/safe-syscall.o
  CC      ppc64-softmmu/accel/tcg/cputlb.o
  CC      aarch64_be-linux-user/target/arm/kvm-stub.o
  CC      tricore-softmmu/hw/vfio/spapr.o
  CC      ppc64le-linux-user/target/ppc/dfp_helper.o
  CC      ppcemb-softmmu/accel/tcg/cpu-exec-common.o
  CC      xtensaeb-softmmu/accel/stubs/kvm-stub.o
  CC      i386-softmmu/migration/ram.o
  CC      nios2-softmmu/hw/net/vhost_net.o
  CC      mips64-softmmu/accel/tcg/tcg-runtime.o
  CC      microblaze-softmmu/accel/tcg/cpu-exec-common.o
  CC      x86_64-softmmu/accel/tcg/cputlb.o
  CC      sh4-softmmu/accel/tcg/tcg-runtime.o
  CC      ppcemb-softmmu/accel/tcg/translate-all.o
  CC      sparc-softmmu/accel/tcg/cputlb.o
  CC      ppc64le-linux-user/target/ppc/excp_helper.o
  CC      mipsel-linux-user/target/mips/lmi_helper.o
  CC      alpha-softmmu/accel/tcg/cpu-exec-common.o
  CC      s390x-softmmu/accel/tcg/cpu-exec.o
  CC      mips-softmmu/accel/tcg/cputlb.o
  CC      microblazeel-softmmu/hw/core/null-machine.o
  CC      aarch64_be-linux-user/target/arm/translate.o
  CC      xtensa-softmmu/hw/core/generic-loader.o
  CC      ppc-softmmu/accel/stubs/hvf-stub.o
  CC      mips64el-softmmu/hw/block/virtio-blk.o
  CC      mipsel-softmmu/hw/display/virtio-gpu.o
  CC      sh4-linux-user/trace/control-target.o
  CC      m68k-softmmu/accel/tcg/translator.o
  CC      i386-linux-user/target/i386/kvm-stub.o
  CC      x86_64-softmmu/accel/tcg/tcg-runtime.o
  CC      aarch64_be-linux-user/target/arm/op_helper.o
  CC      sh4eb-softmmu/hw/block/tc58128.o
  CC      mipsn32-linux-user/target/mips/dsp_helper.o
  CC      alpha-softmmu/accel/tcg/translate-all.o
  CC      cris-softmmu/accel/tcg/cpu-exec.o
  CC      nios2-softmmu/hw/net/rocker/qmp-norocker.o
  CC      nios2-softmmu/hw/timer/altera_timer.o
  CC      mips64-linux-user/target/mips/helper.o
  CC      aarch64_be-linux-user/target/arm/helper.o
  CC      mipsel-linux-user/target/mips/helper.o
  CC      xtensaeb-softmmu/accel/tcg/tcg-all.o
  CC      s390x-softmmu/accel/tcg/cpu-exec-common.o
  CC      or1k-softmmu/hw/vfio/spapr.o
  CC      unicore32-softmmu/accel/tcg/tcg-runtime.o
  CC      sparc-softmmu/accel/tcg/tcg-runtime.o
  CC      sh4-softmmu/accel/tcg/cpu-exec.o
  CC      hppa-linux-user/target/hppa/gdbstub.o
  CC      ppc64le-linux-user/target/ppc/fpu_helper.o
  CC      microblazeel-softmmu/hw/misc/mmio_interface.o
  LINK    nios2-linux-user/qemu-nios2
  CC      microblazeel-softmmu/hw/net/xilinx_ethlite.o
  CC      ppc-softmmu/accel/tcg/tcg-all.o
  CCAS    mips-linux-user/linux-user/safe-syscall.o
  CC      ppc-softmmu/accel/tcg/cputlb.o
  CC      mips-linux-user/target/mips/translate.o
  CC      xtensa-softmmu/hw/core/null-machine.o
  GEN     trace/generated-helpers.c
  CC      tricore-softmmu/hw/tricore/tricore_testboard.o
  CC      i386-linux-user/trace/control-target.o
  CC      ppc-softmmu/accel/tcg/tcg-runtime.o
  CC      x86_64-linux-user/target/i386/gdbstub.o
  CC      sh4eb-softmmu/hw/block/virtio-blk.o
  CC      lm32-softmmu/hw/core/generic-loader.o
  CC      xtensaeb-softmmu/accel/tcg/cputlb.o
  CC      sparc64-linux-user/linux-user/uname.o
  CC      nios2-softmmu/hw/vfio/common.o
  GEN     trace/generated-helpers.c
  CC      nios2-softmmu/hw/vfio/platform.o
  CC      m68k-softmmu/hw/char/mcf_uart.o
  CC      sh4-linux-user/trace/generated-helpers.o
  CC      m68k-linux-user/target/m68k/cpu.o
  CC      m68k-linux-user/target/m68k/fpu_helper.o
  CC      hppa-linux-user/trace/control-target.o
  CC      cris-softmmu/accel/tcg/cpu-exec-common.o
  CC      ppcemb-softmmu/accel/tcg/translator.o
  CC      microblaze-linux-user/target/microblaze/cpu.o
  CC      microblazeel-softmmu/hw/net/vhost_net.o
  CC      xtensa-softmmu/hw/misc/mmio_interface.o
  CC      lm32-softmmu/hw/core/null-machine.o
  CC      ppcemb-softmmu/hw/9pfs/virtio-9p-device.o
  CC      mipsn32-linux-user/target/mips/op_helper.o
  CC      mipsn32-linux-user/target/mips/lmi_helper.o
  CC      m68k-linux-user/target/m68k/gdbstub.o
  CC      sh4eb-linux-user/target/sh4/cpu.o
  CC      x86_64-softmmu/accel/tcg/cpu-exec.o
  CC      or1k-softmmu/hw/openrisc/pic_cpu.o
  CC      x86_64-softmmu/accel/tcg/cpu-exec-common.o
  CC      tricore-softmmu/target/tricore/translate.o
  CC      x86_64-softmmu/accel/tcg/translate-all.o
  CC      x86_64-softmmu/accel/tcg/translator.o
  CC      sh4eb-linux-user/target/sh4/gdbstub.o
  LINK    sh4-linux-user/qemu-sh4
  CC      m68k-softmmu/hw/core/generic-loader.o
  CC      i386-linux-user/gdbstub-xml.o
  CCAS    or1k-linux-user/linux-user/safe-syscall.o
  CC      xtensa-softmmu/hw/net/vhost_net.o
  CC      m68k-softmmu/hw/core/null-machine.o
  CC      mipsel-linux-user/target/mips/cpu.o
  CC      sparc-softmmu/accel/tcg/cpu-exec.o
  CC      sparc-linux-user/linux-user/uaccess.o
  CCAS    sparc64-linux-user/linux-user/safe-syscall.o
  CC      ppc64le-linux-user/target/ppc/int_helper.o
  CC      hppa-linux-user/trace/generated-helpers.o
  CC      sparc64-linux-user/target/sparc/translate.o
  CC      cris-softmmu/accel/tcg/translate-all.o
  CC      armeb-linux-user/linux-user/uaccess.o
  CC      m68k-softmmu/hw/misc/mmio_interface.o
  CC      microblazeel-linux-user/linux-user/linuxload.o
  CC      microblazeel-softmmu/hw/net/rocker/qmp-norocker.o
  CC      sparc-linux-user/linux-user/uname.o
  CC      alpha-softmmu/accel/tcg/translator.o
  CC      aarch64-linux-user/linux-user/uname.o
  CC      sh4-softmmu/accel/tcg/cpu-exec-common.o
  CCAS    aarch64-linux-user/linux-user/safe-syscall.o
  CC      mips64el-softmmu/hw/block/dataplane/virtio-blk.o
  CC      xtensaeb-softmmu/accel/tcg/tcg-runtime.o
  CC      aarch64-softmmu/accel/tcg/tcg-runtime.o
  CC      i386-linux-user/trace/generated-helpers.o
  CC      alpha-softmmu/hw/9pfs/virtio-9p-device.o
  CC      arm-softmmu/accel/tcg/tcg-runtime.o
  CC      or1k-softmmu/hw/openrisc/cputimer.o
  CCAS    sparc-linux-user/linux-user/safe-syscall.o
  CC      alpha-linux-user/target/alpha/cpu.o
  CC      s390x-linux-user/target/s390x/cpu_models.o
  CC      microblazeel-linux-user/linux-user/uaccess.o
  CC      sparc-softmmu/accel/tcg/cpu-exec-common.o
  CC      arm-softmmu/accel/tcg/cpu-exec.o
  CC      armeb-linux-user/linux-user/uname.o
  CC      alpha-softmmu/hw/block/virtio-blk.o
  CC      mipsn32-linux-user/target/mips/helper.o
  CC      xtensa-softmmu/hw/net/rocker/qmp-norocker.o
  CC      xtensa-softmmu/hw/vfio/common.o
  CC      microblazeel-softmmu/hw/vfio/common.o
  CC      lm32-softmmu/hw/input/milkymist-softusb.o
  CC      i386-softmmu/accel/accel.o
  CC      or1k-linux-user/target/openrisc/cpu.o
  CC      m68k-softmmu/hw/net/mcf_fec.o
  CC      mipsn32-linux-user/target/mips/cpu.o
  CCAS    ppc64-linux-user/linux-user/safe-syscall.o
  CC      sparc-softmmu/accel/tcg/translate-all.o
  GEN     trace/generated-helpers.c
  LINK    hppa-linux-user/qemu-hppa
  CC      arm-linux-user/linux-user/linuxload.o
  CC      ppc64le-linux-user/target/ppc/timebase_helper.o
  CC      tricore-softmmu/target/tricore/helper.o
  CC      microblazeel-linux-user/linux-user/uname.o
  CC      sh4eb-linux-user/trace/control-target.o
  CC      ppc64abi32-linux-user/target/ppc/translate.o
  CC      mipsel-softmmu/hw/display/virtio-gpu-3d.o
  CC      ppc-linux-user/target/ppc/cpu-models.o
  CC      x86_64-linux-user/target/i386/xsave_helper.o
  CC      ppc-linux-user/target/ppc/cpu.o
  CC      sparc64-linux-user/target/sparc/helper.o
  CC      moxie-softmmu/accel/tcg/cpu-exec-common.o
  CC      tricore-softmmu/target/tricore/cpu.o
  CC      tilegx-linux-user/target/tilegx/simd_helper.o
  CC      or1k-softmmu/hw/openrisc/openrisc_sim.o
  CC      lm32-softmmu/hw/misc/milkymist-hpdmc.o
  CC      mipsn32-linux-user/target/mips/gdbstub.o
  CC      mips64el-softmmu/hw/char/virtio-serial-bus.o
  CC      mips-softmmu/accel/tcg/tcg-runtime.o
  CC      mips64el-linux-user/target/mips/dsp_helper.o
  CC      sh4-softmmu/accel/tcg/translate-all.o
  CC      ppc-linux-user/target/ppc/translate.o
  LINK    i386-linux-user/qemu-i386
  CC      mipsn32-linux-user/target/mips/msa_helper.o
  CC      mipsel-softmmu/hw/display/virtio-gpu-pci.o
  CC      ppc64le-linux-user/target/ppc/misc_helper.o
  CC      i386-softmmu/accel/stubs/hax-stub.o
  CC      ppc64le-linux-user/target/ppc/mem_helper.o
  CC      arm-softmmu/accel/tcg/cpu-exec-common.o
  CC      sparc64-softmmu/accel/tcg/cpu-exec-common.o
  CC      sh4eb-softmmu/hw/block/dataplane/virtio-blk.o
  CC      x86_64-softmmu/hw/9pfs/virtio-9p-device.o
  CCAS    microblazeel-linux-user/linux-user/safe-syscall.o
  CC      unicore32-softmmu/accel/tcg/cpu-exec.o
  CC      sparc-softmmu/accel/tcg/translator.o
  CC      microblazeel-linux-user/linux-user/flatload.o
  CC      microblazeel-linux-user/target/microblaze/translate.o
  CC      tricore-softmmu/target/tricore/op_helper.o
  CC      ppcemb-softmmu/hw/block/virtio-blk.o
  CC      microblazeel-softmmu/hw/vfio/platform.o
  CC      cris-softmmu/accel/tcg/translator.o
  CC      ppc64-softmmu/accel/tcg/tcg-runtime.o
  CC      mips64el-softmmu/hw/core/generic-loader.o
  CC      sh4eb-linux-user/trace/generated-helpers.o
  CC      aarch64-linux-user/linux-user/flatload.o
  CCAS    armeb-linux-user/linux-user/safe-syscall.o
  CC      mips64-softmmu/accel/tcg/cpu-exec.o
  CC      microblazeel-softmmu/hw/vfio/spapr.o
  CC      moxie-softmmu/accel/tcg/translate-all.o
  CC      microblazeel-linux-user/target/microblaze/op_helper.o
  CC      moxie-softmmu/accel/tcg/translator.o
  CC      lm32-softmmu/hw/misc/milkymist-pfpu.o
  CC      mips64-linux-user/target/mips/cpu.o
  CC      sh4-softmmu/accel/tcg/translator.o
  GEN     trace/generated-helpers.c
  CC      x86_64-linux-user/target/i386/translate.o
  CC      i386-softmmu/accel/stubs/hvf-stub.o
  CC      unicore32-softmmu/accel/tcg/cpu-exec-common.o
  CC      or1k-softmmu/target/openrisc/machine.o
  CC      ppcemb-softmmu/hw/block/dataplane/virtio-blk.o
  CC      sparc-linux-user/target/sparc/translate.o
  CC      m68k-softmmu/hw/net/vhost_net.o
  CC      s390x-linux-user/target/s390x/cpu_features.o
  CC      lm32-softmmu/hw/misc/mmio_interface.o
  CC      lm32-softmmu/hw/net/milkymist-minimac2.o
  CC      ppcemb-softmmu/hw/char/virtio-serial-bus.o
  CC      mipsel-softmmu/hw/intc/mips_gic.o
  CC      or1k-linux-user/target/openrisc/exception.o
  CC      nios2-softmmu/hw/vfio/spapr.o
  CC      microblaze-linux-user/target/microblaze/gdbstub.o
  CC      ppc-linux-user/target/ppc/kvm-stub.o
  CC      sparc-softmmu/hw/core/generic-loader.o
  GEN     trace/generated-helpers.c
  CC      ppc-softmmu/accel/tcg/cpu-exec.o
  CC      tilegx-linux-user/trace/control-target.o
  CC      i386-softmmu/accel/stubs/kvm-stub.o
  CC      m68k-softmmu/hw/net/rocker/qmp-norocker.o
  CC      microblaze-softmmu/accel/tcg/translate-all.o
  CC      alpha-softmmu/hw/block/dataplane/virtio-blk.o
  CC      mipsn32el-linux-user/target/mips/dsp_helper.o
  CC      sh4eb-softmmu/hw/char/sh_serial.o
  CC      i386-softmmu/accel/tcg/tcg-all.o
  CC      xtensaeb-softmmu/accel/tcg/cpu-exec.o
  CC      sparc64-softmmu/accel/tcg/translate-all.o
  CC      sparc-linux-user/target/sparc/helper.o
  CC      m68k-linux-user/trace/control-target.o
  CC      nios2-softmmu/hw/nios2/boot.o
  LINK    sh4eb-linux-user/qemu-sh4eb
  CC      cris-softmmu/hw/core/generic-loader.o
  CC      sparc32plus-linux-user/target/sparc/cpu.o
  CC      xtensa-softmmu/hw/vfio/platform.o
  CC      s390x-linux-user/target/s390x/gdbstub.o
  CC      s390x-linux-user/target/s390x/interrupt.o
  CC      sh4eb-softmmu/hw/char/virtio-serial-bus.o
  CC      aarch64-linux-user/target/arm/arm-semi.o
  CC      or1k-softmmu/target/openrisc/cpu.o
  CC      aarch64-softmmu/accel/tcg/cpu-exec.o
  GEN     trace/generated-helpers.c
  CC      aarch64-linux-user/target/arm/kvm-stub.o
  CC      xtensa-softmmu/hw/vfio/spapr.o
  CC      m68k-softmmu/hw/vfio/common.o
  CC      xtensaeb-softmmu/accel/tcg/cpu-exec-common.o
  CC      mips64el-linux-user/target/mips/op_helper.o
  CC      microblaze-linux-user/trace/control-target.o
  CC      cris-softmmu/hw/core/null-machine.o
  CC      m68k-softmmu/hw/vfio/platform.o
  CC      aarch64-softmmu/accel/tcg/cpu-exec-common.o
  CC      mips64el-softmmu/hw/core/null-machine.o
  CC      mips64-linux-user/target/mips/gdbstub.o
  CC      aarch64-linux-user/target/arm/translate.o
  CC      microblazeel-softmmu/hw/microblaze/petalogix_s3adsp1800_mmu.o
  CC      alpha-linux-user/target/alpha/int_helper.o
  CC      or1k-linux-user/target/openrisc/interrupt.o
  CC      or1k-linux-user/target/openrisc/mmu.o
  CC      s390x-linux-user/target/s390x/helper.o
  CC      moxie-softmmu/hw/core/generic-loader.o
  CC      i386-softmmu/accel/tcg/cputlb.o
  CC      lm32-softmmu/hw/net/vhost_net.o
  CC      unicore32-softmmu/accel/tcg/translate-all.o
  CC      i386-softmmu/accel/tcg/tcg-runtime.o
  CC      alpha-softmmu/hw/char/virtio-serial-bus.o
  CC      ppc64-linux-user/target/ppc/cpu-models.o
  CC      nios2-softmmu/hw/nios2/cpu_pic.o
  CC      or1k-softmmu/target/openrisc/exception.o
  CC      ppc64-softmmu/accel/tcg/cpu-exec.o
  CC      mips64-softmmu/accel/tcg/cpu-exec-common.o
  CC      x86_64-softmmu/hw/block/virtio-blk.o
  CC      sh4-softmmu/hw/9pfs/virtio-9p-device.o
  CC      alpha-linux-user/target/alpha/fpu_helper.o
  CC      sparc32plus-linux-user/target/sparc/fop_helper.o
  CC      aarch64-softmmu/accel/tcg/translate-all.o
  CC      aarch64-linux-user/target/arm/op_helper.o
  CC      tilegx-linux-user/trace/generated-helpers.o
  CC      s390x-softmmu/accel/tcg/translate-all.o
  CC      alpha-linux-user/target/alpha/vax_helper.o
  CC      microblaze-linux-user/trace/generated-helpers.o
  CC      m68k-softmmu/hw/vfio/spapr.o
  CC      nios2-softmmu/hw/nios2/10m50_devboard.o
  CC      x86_64-linux-user/target/i386/bpt_helper.o
  CC      mips64el-softmmu/hw/display/vga.o
  CC      mipsn32-linux-user/target/mips/mips-semi.o
  CC      unicore32-softmmu/accel/tcg/translator.o
  CC      alpha-linux-user/target/alpha/sys_helper.o
  CC      cris-softmmu/hw/misc/mmio_interface.o
  CC      sh4eb-softmmu/hw/core/generic-loader.o
  CC      or1k-linux-user/target/openrisc/translate.o
  CC      mipsel-softmmu/hw/misc/ivshmem.o
  CC      mips64-softmmu/accel/tcg/translate-all.o
  CC      m68k-linux-user/gdbstub-xml.o
  CC      mipsel-softmmu/hw/misc/mips_cmgcr.o
  CC      mipsel-linux-user/target/mips/gdbstub.o
  CC      s390x-linux-user/target/s390x/translate.o
  CC      moxie-softmmu/hw/core/null-machine.o
  CC      moxie-softmmu/hw/display/vga.o
  CC      lm32-softmmu/hw/net/rocker/qmp-norocker.o
  CC      mips64-linux-user/target/mips/msa_helper.o
  CC      or1k-softmmu/target/openrisc/interrupt.o
  CC      microblazeel-softmmu/hw/microblaze/petalogix_ml605_mmu.o
  CC      ppc-softmmu/accel/tcg/cpu-exec-common.o
  CC      sparc32plus-linux-user/target/sparc/cc_helper.o
  CC      x86_64-linux-user/target/i386/cc_helper.o
  CC      xtensa-softmmu/hw/xtensa/pic_cpu.o
  CC      cris-softmmu/hw/net/etraxfs_eth.o
  CC      mips64-linux-user/target/mips/mips-semi.o
  CC      lm32-softmmu/hw/sd/milkymist-memcard.o
  CC      ppc-softmmu/accel/tcg/translate-all.o
  CC      ppcemb-softmmu/hw/core/generic-loader.o
  CC      nios2-softmmu/target/nios2/translate.o
  CC      m68k-softmmu/hw/m68k/an5206.o
  CC      cris-softmmu/hw/net/vhost_net.o
  CC      nios2-softmmu/target/nios2/op_helper.o
  CC      alpha-linux-user/target/alpha/mem_helper.o
  CC      lm32-softmmu/hw/vfio/common.o
  CC      mips-linux-user/target/mips/dsp_helper.o
  CC      sparc64-softmmu/accel/tcg/translator.o
  CC      mips-linux-user/target/mips/op_helper.o
  CC      mips-linux-user/target/mips/lmi_helper.o
  CC      mips-linux-user/target/mips/helper.o
  LINK    tilegx-linux-user/qemu-tilegx
  CC      alpha-linux-user/target/alpha/gdbstub.o
  CC      nios2-softmmu/target/nios2/helper.o
  CC      i386-softmmu/accel/tcg/cpu-exec.o
  CC      sparc32plus-linux-user/target/sparc/win_helper.o
  CC      m68k-linux-user/trace/generated-helpers.o
  CC      ppc-softmmu/accel/tcg/translator.o
  CC      sparc-softmmu/hw/core/null-machine.o
  CC      mips-softmmu/accel/tcg/cpu-exec.o
  CC      sh4-softmmu/hw/block/tc58128.o
  LINK    microblaze-linux-user/qemu-microblaze
  CC      microblazeel-softmmu/hw/microblaze/boot.o
  CC      cris-softmmu/hw/net/rocker/qmp-norocker.o
  CC      moxie-softmmu/hw/misc/mmio_interface.o
  CC      or1k-softmmu/target/openrisc/mmu.o
  CC      microblazeel-softmmu/target/microblaze/translate.o
  CC      mips-softmmu/accel/tcg/cpu-exec-common.o
  CC      mipsel-softmmu/hw/misc/mips_cpc.o
  CC      mips-softmmu/accel/tcg/translate-all.o
  GEN     trace/generated-helpers.c
  CC      ppcemb-softmmu/hw/core/null-machine.o
  CC      ppc64le-linux-user/target/ppc/user_only_helper.o
  CC      arm-softmmu/accel/tcg/translate-all.o
  GEN     trace/generated-helpers.c
  CC      i386-softmmu/accel/tcg/cpu-exec-common.o
  CC      ppc64-linux-user/target/ppc/cpu.o
  GEN     trace/generated-helpers.c
  CC      alpha-softmmu/hw/core/generic-loader.o
  CC      microblaze-softmmu/accel/tcg/translator.o
  CC      arm-linux-user/linux-user/uaccess.o
  CC      xtensa-softmmu/hw/xtensa/sim.o
  CC      aarch64-softmmu/accel/tcg/translator.o
  CC      moxie-softmmu/hw/net/vhost_net.o
  CC      cris-softmmu/hw/vfio/common.o
  CC      armeb-linux-user/linux-user/flatload.o
  CC      unicore32-softmmu/hw/core/generic-loader.o
  CC      mips-linux-user/target/mips/cpu.o
  CC      ppcemb-softmmu/hw/display/sm501.o
  CC      s390x-linux-user/target/s390x/cc_helper.o
  CC      mips-linux-user/target/mips/gdbstub.o
  CC      alpha-linux-user/trace/control-target.o
  CC      or1k-linux-user/target/openrisc/exception_helper.o
  CC      sparc-softmmu/hw/display/tcx.o
  CC      x86_64-softmmu/hw/block/dataplane/virtio-blk.o
  CC      sh4-softmmu/hw/block/virtio-blk.o
  CC      aarch64-linux-user/target/arm/helper.o
  CC      mips64el-linux-user/target/mips/lmi_helper.o
  CC      or1k-softmmu/target/openrisc/translate.o
  CC      mips64-softmmu/accel/tcg/translator.o
  CC      ppc64abi32-linux-user/target/ppc/kvm-stub.o
  CC      nios2-softmmu/target/nios2/cpu.o
  CC      xtensaeb-softmmu/accel/tcg/translate-all.o
  CC      or1k-linux-user/target/openrisc/fpu_helper.o
  CC      ppc64le-linux-user/target/ppc/gdbstub.o
  CC      ppc64-softmmu/accel/tcg/cpu-exec-common.o
  CC      mipsel-softmmu/hw/misc/mips_itu.o
  CC      sh4eb-softmmu/hw/core/null-machine.o
  CC      sh4eb-softmmu/hw/display/sm501.o
  CC      xtensa-softmmu/hw/xtensa/xtfpga.o
  CC      m68k-softmmu/hw/m68k/mcf5208.o
  LINK    m68k-linux-user/qemu-m68k
  CC      ppc-softmmu/hw/9pfs/virtio-9p-device.o
  CC      armeb-linux-user/linux-user/arm/nwfpe/fpa11.o
  CC      or1k-softmmu/target/openrisc/exception_helper.o
  CC      aarch64-softmmu/hw/9pfs/virtio-9p-device.o
  CC      sh4-softmmu/hw/block/dataplane/virtio-blk.o
  CC      or1k-softmmu/target/openrisc/fpu_helper.o
  CC      mips64el-linux-user/target/mips/helper.o
  CC      s390x-softmmu/accel/tcg/translator.o
  CC      arm-linux-user/linux-user/uname.o
  CC      or1k-softmmu/target/openrisc/interrupt_helper.o
  CC      xtensa-softmmu/target/xtensa/xtensa-semi.o
  CC      x86_64-softmmu/hw/char/virtio-serial-bus.o
  CC      ppc64le-linux-user/target/ppc/../../libdecnumber/decContext.o
  CC      sparc-softmmu/hw/display/cg3.o
  CC      sh4eb-softmmu/hw/display/vga.o
  CC      sparc64-softmmu/hw/9pfs/virtio-9p-device.o
  CC      xtensaeb-softmmu/accel/tcg/translator.o
  CC      i386-softmmu/accel/tcg/translate-all.o
  CC      microblazeel-linux-user/target/microblaze/helper.o
  CC      alpha-softmmu/hw/core/null-machine.o
  CC      x86_64-linux-user/target/i386/excp_helper.o
  CC      unicore32-softmmu/hw/core/null-machine.o
  CC      mipsel-softmmu/hw/misc/mmio_interface.o
  CC      sparc64-linux-user/target/sparc/cpu.o
  CC      aarch64-linux-user/target/arm/cpu.o
  CC      mips64el-linux-user/target/mips/cpu.o
  CC      ppc64-linux-user/target/ppc/translate.o
  CC      ppc64-softmmu/accel/tcg/translate-all.o
  CC      sparc64-softmmu/hw/block/virtio-blk.o
  CC      mips-softmmu/accel/tcg/translator.o
  CC      ppc64-linux-user/target/ppc/kvm-stub.o
  CC      aarch64-linux-user/target/arm/neon_helper.o
  CC      microblaze-softmmu/hw/core/generic-loader.o
  CC      alpha-linux-user/trace/generated-helpers.o
  CC      aarch64-softmmu/hw/adc/stm32f2xx_adc.o
  CC      m68k-softmmu/hw/m68k/mcf5206.o
  CC      tricore-softmmu/target/tricore/fpu_helper.o
  CC      microblazeel-linux-user/target/microblaze/cpu.o
  CC      x86_64-softmmu/hw/core/generic-loader.o
  CCAS    arm-linux-user/linux-user/safe-syscall.o
  CC      microblaze-softmmu/hw/core/null-machine.o
  CC      mipsn32el-linux-user/target/mips/op_helper.o
  CC      nios2-softmmu/target/nios2/mmu.o
  CC      arm-linux-user/linux-user/flatload.o
  CC      sparc64-linux-user/target/sparc/fop_helper.o
  CC      lm32-softmmu/hw/vfio/platform.o
  CC      s390x-linux-user/target/s390x/excp_helper.o
  CC      aarch64-softmmu/hw/block/virtio-blk.o
  CC      aarch64-linux-user/target/arm/iwmmxt_helper.o
  CC      ppc-softmmu/hw/block/virtio-blk.o
  CC      microblaze-softmmu/hw/misc/mmio_interface.o
  CC      unicore32-softmmu/hw/misc/mmio_interface.o
  CC      m68k-softmmu/hw/m68k/mcf_intc.o
  CC      mipsel-softmmu/hw/net/virtio-net.o
  CC      s390x-softmmu/hw/9pfs/virtio-9p-device.o
  CC      alpha-softmmu/hw/display/vga.o
  CC      armeb-linux-user/linux-user/arm/nwfpe/fpa11_cpdo.o
  GEN     trace/generated-helpers.c
  CC      sh4eb-softmmu/hw/display/virtio-gpu.o
  CC      sh4eb-softmmu/hw/display/virtio-gpu-3d.o
  CC      sparc64-softmmu/hw/block/dataplane/virtio-blk.o
  CC      x86_64-linux-user/target/i386/fpu_helper.o
  CC      s390x-linux-user/target/s390x/fpu_helper.o
  CC      ppc64le-linux-user/target/ppc/../../libdecnumber/decNumber.o
  CC      moxie-softmmu/hw/net/rocker/qmp-norocker.o
  CC      mips64el-linux-user/target/mips/gdbstub.o
  CC      sparc-softmmu/hw/intc/grlib_irqmp.o
  CC      sh4-softmmu/hw/char/sh_serial.o
  CC      arm-softmmu/accel/tcg/translator.o
  CC      s390x-softmmu/hw/block/virtio-blk.o
  CC      xtensa-softmmu/target/xtensa/core-dc232b.o
  CC      ppc64le-linux-user/target/ppc/../../libdecnumber/dpd/decimal32.o
  CC      sh4-softmmu/hw/char/virtio-serial-bus.o
  CC      unicore32-softmmu/hw/net/vhost_net.o
  CC      xtensaeb-softmmu/hw/core/generic-loader.o
  CC      sh4-softmmu/hw/core/generic-loader.o
  CC      ppcemb-softmmu/hw/display/vga.o
  CC      ppc64le-linux-user/target/ppc/../../libdecnumber/dpd/decimal64.o
  CC      mipsn32-linux-user/trace/control-target.o
  CC      ppc-softmmu/hw/block/dataplane/virtio-blk.o
  CC      mipsel-linux-user/target/mips/msa_helper.o
  CC      sparc-linux-user/target/sparc/cpu.o
  CC      sparc64-linux-user/target/sparc/cc_helper.o
  CC      mips64-softmmu/hw/9pfs/virtio-9p-device.o
  LINK    alpha-linux-user/qemu-alpha
  CC      mips64-softmmu/hw/block/virtio-blk.o
  CC      microblaze-softmmu/hw/net/xilinx_ethlite.o
  CC      mips64-softmmu/hw/block/dataplane/virtio-blk.o
  CC      sparc32plus-linux-user/target/sparc/mmu_helper.o
  CC      xtensa-softmmu/target/xtensa/core-dc233c.o
  CC      x86_64-linux-user/target/i386/int_helper.o
  CC      nios2-softmmu/target/nios2/monitor.o
  CC      i386-softmmu/accel/tcg/translator.o
  CC      i386-softmmu/hw/9pfs/virtio-9p-device.o
  CC      m68k-softmmu/target/m68k/m68k-semi.o
  CC      s390x-linux-user/target/s390x/int_helper.o
  CC      xtensa-softmmu/target/xtensa/core-fsf.o
  CC      arm-linux-user/linux-user/arm/nwfpe/fpa11.o
  CC      mipsel-linux-user/target/mips/mips-semi.o
  CC      microblazeel-linux-user/target/microblaze/gdbstub.o
  CC      microblaze-softmmu/hw/net/vhost_net.o
  CC      mips64el-softmmu/hw/display/virtio-gpu.o
  CC      microblazeel-softmmu/target/microblaze/op_helper.o
  CC      microblaze-softmmu/hw/net/rocker/qmp-norocker.o
  CC      moxie-softmmu/hw/timer/mc146818rtc.o
  GEN     trace/generated-helpers.c
  CC      moxie-softmmu/hw/vfio/common.o
  CC      aarch64-linux-user/target/arm/gdbstub.o
  CC      armeb-linux-user/linux-user/arm/nwfpe/fpa11_cpdt.o
  CC      unicore32-softmmu/hw/net/rocker/qmp-norocker.o
  CC      xtensaeb-softmmu/hw/core/null-machine.o
  CC      armeb-linux-user/linux-user/arm/nwfpe/fpa11_cprt.o
  CC      x86_64-linux-user/target/i386/mem_helper.o
  CC      aarch64-linux-user/target/arm/cpu64.o
  CC      x86_64-softmmu/hw/core/null-machine.o
  CC      cris-softmmu/hw/vfio/platform.o
  CC      mips64el-softmmu/hw/display/virtio-gpu-3d.o
  CC      alpha-softmmu/hw/display/virtio-gpu.o
  CC      mips-softmmu/hw/9pfs/virtio-9p-device.o
  CC      unicore32-softmmu/hw/vfio/common.o
  CC      alpha-softmmu/hw/display/virtio-gpu-3d.o
  CC      sparc-softmmu/hw/misc/eccmemctl.o
  CC      ppc64le-linux-user/target/ppc/../../libdecnumber/dpd/decimal128.o
  CC      x86_64-linux-user/target/i386/misc_helper.o
  CC      ppc64-linux-user/target/ppc/dfp_helper.o
  GEN     trace/generated-helpers.c
  CC      lm32-softmmu/hw/vfio/spapr.o
  CC      sparc32plus-linux-user/target/sparc/ldst_helper.o
  CC      ppc64-linux-user/target/ppc/excp_helper.o
  CC      xtensaeb-softmmu/hw/misc/mmio_interface.o
  GEN     trace/generated-helpers.c
  CC      microblazeel-linux-user/trace/control-target.o
  CC      mipsel-linux-user/trace/control-target.o
  CC      nios2-softmmu/trace/control-target.o
  CC      mips-softmmu/hw/block/virtio-blk.o
  CC      ppc64-softmmu/accel/tcg/translator.o
  CC      aarch64_be-linux-user/target/arm/cpu.o
  CC      aarch64_be-linux-user/target/arm/neon_helper.o
  CC      arm-linux-user/linux-user/arm/nwfpe/fpa11_cpdo.o
  CC      m68k-softmmu/target/m68k/translate.o
  CC      microblaze-softmmu/hw/vfio/common.o
  CC      x86_64-linux-user/target/i386/mpx_helper.o
  CC      xtensaeb-softmmu/hw/net/vhost_net.o
  CC      sparc64-softmmu/hw/char/virtio-serial-bus.o
  CC      or1k-linux-user/target/openrisc/interrupt_helper.o
  CC      sh4eb-softmmu/hw/display/virtio-gpu-pci.o
  CC      ppc64-softmmu/hw/9pfs/virtio-9p-device.o
  CC      sparc64-softmmu/hw/core/generic-loader.o
  CC      sparc64-softmmu/hw/core/null-machine.o
  CC      aarch64-softmmu/hw/block/dataplane/virtio-blk.o
  CC      mipsn32el-linux-user/target/mips/lmi_helper.o
  CC      sparc-linux-user/target/sparc/fop_helper.o
  CC      mips64-linux-user/trace/control-target.o
  CC      lm32-softmmu/hw/lm32/lm32_boards.o
  CC      x86_64-softmmu/hw/display/vga.o
  CC      m68k-softmmu/target/m68k/op_helper.o
  CC      arm-softmmu/hw/9pfs/virtio-9p-device.o
  CC      sparc64-softmmu/hw/display/vga.o
  CC      mips64-softmmu/hw/char/virtio-serial-bus.o
  CC      ppc-softmmu/hw/char/virtio-serial-bus.o
  CC      mipsel-softmmu/hw/net/vhost_net.o
  CC      microblazeel-linux-user/trace/generated-helpers.o
  CC      sh4-softmmu/hw/core/null-machine.o
  CC      sparc-softmmu/hw/misc/slavio_misc.o
  CC      sh4-softmmu/hw/display/sm501.o
  CC      arm-linux-user/linux-user/arm/nwfpe/fpa11_cpdt.o
  CC      mips-linux-user/target/mips/msa_helper.o
  CC      armeb-linux-user/linux-user/arm/nwfpe/fpopcode.o
  CC      lm32-softmmu/hw/lm32/milkymist.o
  CC      microblazeel-softmmu/target/microblaze/helper.o
  CC      mips-softmmu/hw/block/dataplane/virtio-blk.o
  CC      mips64el-linux-user/target/mips/msa_helper.o
  CC      nios2-softmmu/trace/generated-helpers.o
  GEN     trace/generated-helpers.c
  CC      x86_64-softmmu/hw/display/virtio-gpu.o
  CC      cris-softmmu/hw/vfio/spapr.o
  CC      sparc-linux-user/target/sparc/cc_helper.o
  CC      xtensaeb-softmmu/hw/net/rocker/qmp-norocker.o
  CC      unicore32-softmmu/hw/vfio/platform.o
  CC      sparc32plus-linux-user/target/sparc/int64_helper.o
  CC      ppc64le-linux-user/trace/control-target.o
  CC      arm-softmmu/hw/adc/stm32f2xx_adc.o
  CC      or1k-linux-user/target/openrisc/mmu_helper.o
  CC      mips64-softmmu/hw/core/generic-loader.o
  CC      mipsel-softmmu/hw/rdma/rdma_utils.o
  CC      sparc-softmmu/hw/misc/mmio_interface.o
  CC      aarch64-linux-user/target/arm/translate-a64.o
  CC      sh4-softmmu/hw/display/vga.o
  CC      ppc-linux-user/target/ppc/dfp_helper.o
  CC      mipsel-softmmu/hw/rdma/rdma_backend.o
  CC      xtensaeb-softmmu/hw/vfio/common.o
  CC      m68k-softmmu/target/m68k/helper.o
  CC      armeb-linux-user/linux-user/arm/nwfpe/single_cpdo.o
  CC      xtensa-softmmu/target/xtensa/monitor.o
  CC      x86_64-softmmu/hw/display/virtio-gpu-3d.o
  CC      arm-softmmu/hw/block/virtio-blk.o
  CC      s390x-softmmu/hw/block/dataplane/virtio-blk.o
  CC      sh4eb-softmmu/hw/intc/sh_intc.o
  CC      sh4eb-softmmu/hw/misc/ivshmem.o
  CC      xtensa-softmmu/target/xtensa/xtensa-isa.o
  CC      aarch64-softmmu/hw/char/exynos4210_uart.o
  CC      sh4-softmmu/hw/display/virtio-gpu.o
  CC      moxie-softmmu/hw/vfio/platform.o
  CC      s390x-softmmu/hw/char/virtio-serial-bus.o
  CC      mips64el-linux-user/target/mips/mips-semi.o
  CC      armeb-linux-user/linux-user/arm/nwfpe/double_cpdo.o
  LINK    microblazeel-linux-user/qemu-microblazeel
  CC      sparc-softmmu/hw/net/vhost_net.o
  CC      xtensaeb-softmmu/hw/vfio/platform.o
  CC      m68k-softmmu/target/m68k/cpu.o
  CC      lm32-softmmu/target/lm32/translate.o
  CC      cris-softmmu/hw/cris/boot.o
  CC      or1k-softmmu/target/openrisc/mmu_helper.o
  CC      or1k-linux-user/target/openrisc/sys_helper.o
  CC      sparc64-linux-user/target/sparc/win_helper.o
  CC      sparc64-linux-user/target/sparc/mmu_helper.o
  CC      sparc32plus-linux-user/target/sparc/vis_helper.o
  CC      microblazeel-softmmu/target/microblaze/cpu.o
  CC      s390x-linux-user/target/s390x/mem_helper.o
  CC      aarch64-softmmu/hw/char/omap_uart.o
  GEN     trace/generated-helpers.c
  CC      microblazeel-softmmu/target/microblaze/gdbstub.o
  CC      sparc-linux-user/target/sparc/win_helper.o
  CC      alpha-softmmu/hw/display/virtio-gpu-pci.o
  CC      arm-linux-user/linux-user/arm/nwfpe/fpa11_cprt.o
  CC      x86_64-linux-user/target/i386/seg_helper.o
  CC      mipsn32-linux-user/trace/generated-helpers.o
  CC      lm32-softmmu/target/lm32/op_helper.o
  CC      i386-softmmu/hw/block/virtio-blk.o
  CC      or1k-softmmu/target/openrisc/sys_helper.o
  CC      arm-linux-user/linux-user/arm/nwfpe/fpopcode.o
  CC      ppc64-softmmu/hw/block/virtio-blk.o
  CC      mipsel-softmmu/hw/rdma/rdma_rm.o
  CC      mips-linux-user/target/mips/mips-semi.o
  LINK    nios2-softmmu/qemu-system-nios2
  CC      mips-softmmu/hw/char/virtio-serial-bus.o
  GEN     trace/generated-helpers.c
  CC      cris-softmmu/hw/cris/axis_dev88.o
  CC      armeb-linux-user/linux-user/arm/nwfpe/extended_cpdo.o
  CC      or1k-linux-user/target/openrisc/gdbstub.o
  CC      armeb-linux-user/target/arm/arm-semi.o
  CC      unicore32-softmmu/hw/vfio/spapr.o
  CC      ppc-softmmu/hw/core/generic-loader.o
  CC      unicore32-softmmu/hw/unicore32/puv3.o
  CC      sparc-softmmu/hw/net/rocker/qmp-norocker.o
  CC      sh4-softmmu/hw/display/virtio-gpu-3d.o
  CC      ppc64le-linux-user/gdbstub-xml.o
  CC      mips64el-softmmu/hw/display/virtio-gpu-pci.o
  CC      arm-linux-user/linux-user/arm/nwfpe/single_cpdo.o
  CC      arm-linux-user/linux-user/arm/nwfpe/double_cpdo.o
  CC      lm32-softmmu/target/lm32/helper.o
  CC      cris-softmmu/target/cris/translate.o
  CC      sparc64-softmmu/hw/display/virtio-gpu.o
  CC      m68k-softmmu/target/m68k/fpu_helper.o
  CC      arm-softmmu/hw/block/dataplane/virtio-blk.o
  CC      aarch64-linux-user/target/arm/helper-a64.o
  CC      aarch64-linux-user/target/arm/gdbstub64.o
  CC      aarch64-softmmu/hw/char/digic-uart.o
  CC      s390x-linux-user/target/s390x/misc_helper.o
  CC      sparc-linux-user/target/sparc/mmu_helper.o
  CC      arm-softmmu/hw/char/exynos4210_uart.o
  CC      s390x-softmmu/hw/char/terminal3270.o
  CC      sparc-linux-user/target/sparc/ldst_helper.o
  CC      cris-softmmu/target/cris/op_helper.o
  GEN     trace/generated-helpers.c
  CC      cris-softmmu/target/cris/helper.o
  CC      or1k-softmmu/target/openrisc/gdbstub.o
  CC      ppcemb-softmmu/hw/display/virtio-gpu.o
  CC      moxie-softmmu/hw/vfio/spapr.o
  CC      mips-softmmu/hw/core/generic-loader.o
  CC      microblaze-softmmu/hw/vfio/platform.o
  CC      s390x-linux-user/target/s390x/crypto_helper.o
  CC      or1k-linux-user/trace/control-target.o
  CC      cris-softmmu/target/cris/cpu.o
  CC      sparc-softmmu/hw/vfio/common.o
  CC      alpha-softmmu/hw/misc/ivshmem.o
  CC      ppc-softmmu/hw/core/null-machine.o
  CC      arm-linux-user/linux-user/arm/nwfpe/extended_cpdo.o
  CC      aarch64-linux-user/target/arm/crypto_helper.o
  CC      mips64-softmmu/hw/core/null-machine.o
  CC      sparc32plus-linux-user/target/sparc/gdbstub.o
  CC      x86_64-linux-user/target/i386/smm_helper.o
  GEN     trace/generated-helpers.c
  CC      cris-softmmu/target/cris/gdbstub.o
  CC      aarch64-softmmu/hw/char/stm32f2xx_usart.o
  CC      arm-linux-user/target/arm/arm-semi.o
  CC      unicore32-softmmu/target/unicore32/translate.o
  CC      unicore32-softmmu/target/unicore32/op_helper.o
  CC      mips64el-softmmu/hw/intc/mips_gic.o
  CC      mips64-linux-user/trace/generated-helpers.o
  CC      aarch64-softmmu/hw/char/bcm2835_aux.o
  CC      s390x-linux-user/target/s390x/kvm-stub.o
  CC      tricore-softmmu/trace/control-target.o
  CC      ppc-softmmu/hw/display/sm501.o
  CC      microblaze-softmmu/hw/vfio/spapr.o
  CC      armeb-linux-user/target/arm/kvm-stub.o
  CC      sh4eb-softmmu/hw/misc/mmio_interface.o
  CC      ppcemb-softmmu/hw/display/virtio-gpu-3d.o
  GEN     trace/generated-helpers.c
  CC      x86_64-softmmu/hw/display/virtio-gpu-pci.o
  CC      ppcemb-softmmu/hw/display/virtio-gpu-pci.o
  CC      xtensaeb-softmmu/hw/vfio/spapr.o
  CC      sparc-linux-user/target/sparc/int32_helper.o
  GEN     trace/generated-helpers.c
  CC      ppc-softmmu/hw/display/vga.o
  CC      mipsel-softmmu/hw/rdma/vmw/pvrdma_dev_ring.o
  CC      alpha-softmmu/hw/misc/mmio_interface.o
  CC      x86_64-softmmu/hw/display/virtio-vga.o
  CC      m68k-softmmu/target/m68k/gdbstub.o
  CC      s390x-softmmu/hw/core/generic-loader.o
  CC      or1k-softmmu/trace/control-target.o
  CC      moxie-softmmu/hw/moxie/moxiesim.o
  CC      arm-softmmu/hw/char/omap_uart.o
  CC      microblaze-softmmu/hw/microblaze/petalogix_s3adsp1800_mmu.o
  CC      mipsel-softmmu/hw/rdma/vmw/pvrdma_cmd.o
  CC      lm32-softmmu/target/lm32/cpu.o
  CC      mips-linux-user/trace/control-target.o
  CC      mips64-softmmu/hw/display/vga.o
  CC      arm-linux-user/target/arm/kvm-stub.o
  CC      xtensa-softmmu/target/xtensa/translate.o
  CC      sparc-linux-user/target/sparc/gdbstub.o
  GEN     trace/generated-helpers.c
  CC      moxie-softmmu/target/moxie/translate.o
  CC      sparc64-linux-user/target/sparc/ldst_helper.o
  CC      x86_64-softmmu/hw/intc/apic.o
  CC      ppc-linux-user/target/ppc/excp_helper.o
  CC      ppc-linux-user/target/ppc/fpu_helper.o
  CC      mipsel-softmmu/hw/rdma/vmw/pvrdma_qp_ops.o
  CC      ppc64-linux-user/target/ppc/fpu_helper.o
  CC      alpha-softmmu/hw/net/virtio-net.o
  CC      aarch64-softmmu/hw/char/virtio-serial-bus.o
  CC      armeb-linux-user/target/arm/translate.o
  CC      microblazeel-softmmu/target/microblaze/mmu.o
  CC      xtensa-softmmu/target/xtensa/op_helper.o
  CC      mips-softmmu/hw/core/null-machine.o
  CC      or1k-linux-user/trace/generated-helpers.o
  CC      arm-linux-user/target/arm/translate.o
  CC      ppc-linux-user/target/ppc/int_helper.o
  CC      mips-softmmu/hw/display/vga.o
  CC      sparc32plus-linux-user/trace/control-target.o
  CC      ppc64le-linux-user/trace/generated-helpers.o
  CC      sh4eb-softmmu/hw/net/virtio-net.o
  CC      ppc64-softmmu/hw/block/dataplane/virtio-blk.o
  CC      m68k-softmmu/target/m68k/monitor.o
  CC      moxie-softmmu/target/moxie/helper.o
  GEN     trace/generated-helpers.c
  CC      aarch64-linux-user/trace/control-target.o
  CC      i386-softmmu/hw/block/dataplane/virtio-blk.o
  CC      unicore32-softmmu/target/unicore32/helper.o
  CC      s390x-softmmu/hw/core/null-machine.o
  CC      sparc64-softmmu/hw/display/virtio-gpu-3d.o
  CC      sparc-softmmu/hw/vfio/platform.o
  CC      xtensa-softmmu/target/xtensa/helper.o
  CC      ppc64abi32-linux-user/target/ppc/dfp_helper.o
  CC      ppc-softmmu/hw/display/virtio-gpu.o
  CC      sparc-softmmu/hw/vfio/spapr.o
  CC      sparc-linux-user/trace/control-target.o
  CC      microblaze-softmmu/hw/microblaze/petalogix_ml605_mmu.o
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c: In function ‘create_qp_rings’:
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: implicit declaration of function ‘pow2roundup32’ [-Werror=implicit-function-declaration]
     wqe_sz = pow2roundup32(sizeof(struct pvrdma_sq_wqe_hdr) +
     ^
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: nested extern declaration of ‘pow2roundup32’ [-Werror=nested-externs]
  CC      alpha-softmmu/hw/net/vhost_net.o
cc1: all warnings being treated as errors
  CC      mips64el-softmmu/hw/misc/ivshmem.o
  CC      mips64el-linux-user/trace/control-target.o
make[1]: *** [hw/rdma/vmw/pvrdma_cmd.o] Error 1
make[1]: *** Waiting for unfinished jobs....
  CC      x86_64-linux-user/target/i386/svm_helper.o
  CC      sh4-softmmu/hw/display/virtio-gpu-pci.o
  CC      arm-softmmu/hw/char/digic-uart.o
  CC      sparc64-linux-user/target/sparc/int64_helper.o
  CC      arm-linux-user/target/arm/op_helper.o
  CC      sparc-softmmu/hw/sparc/sun4m_iommu.o
  CC      i386-softmmu/hw/char/virtio-serial-bus.o
  CC      sparc-softmmu/hw/sparc/sun4m.o
  CC      lm32-softmmu/target/lm32/gdbstub.o
  CC      sparc-softmmu/hw/sparc/leon3.o
  CC      xtensaeb-softmmu/hw/xtensa/pic_cpu.o
make: *** [subdir-mipsel-softmmu] Error 2
make: *** Waiting for unfinished jobs....
  CC      sparc-softmmu/target/sparc/machine.o
  CC      xtensa-softmmu/target/xtensa/cpu.o
  CC      or1k-softmmu/trace/generated-helpers.o
  CC      xtensa-softmmu/target/xtensa/gdbstub.o
  CC      ppc-linux-user/target/ppc/timebase_helper.o
  CC      mips64el-softmmu/hw/misc/mips_cmgcr.o
  LINK    or1k-linux-user/qemu-or1k
  CC      mips64-softmmu/hw/display/virtio-gpu.o
  CC      ppc-linux-user/target/ppc/misc_helper.o
  CC      ppc64-linux-user/target/ppc/int_helper.o
  CC      aarch64-linux-user/gdbstub-xml.o
  CC      s390x-softmmu/hw/display/virtio-gpu.o
  CC      sparc-linux-user/trace/generated-helpers.o
  CC      mips64-softmmu/hw/display/virtio-gpu-3d.o
  CC      armeb-linux-user/target/arm/op_helper.o
  CC      ppc64abi32-linux-user/target/ppc/excp_helper.o
  CC      sparc32plus-linux-user/trace/generated-helpers.o
  GEN     trace/generated-helpers.c
  GEN     trace/generated-helpers.c
  CC      xtensa-softmmu/trace/control-target.o
  CC      arm-softmmu/hw/char/stm32f2xx_usart.o
  CC      mipsn32el-linux-user/target/mips/helper.o
  CC      arm-linux-user/target/arm/helper.o
  CC      ppc-softmmu/hw/display/virtio-gpu-3d.o
  CC      mips-linux-user/trace/generated-helpers.o
  CC      mips-softmmu/hw/display/virtio-gpu.o
  CC      sparc64-softmmu/hw/display/virtio-gpu-pci.o
  CC      ppc64-softmmu/hw/char/spapr_vty.o
  CC      ppc64-softmmu/hw/char/virtio-serial-bus.o
  CC      microblazeel-softmmu/trace/control-target.o
  CC      mipsn32el-linux-user/target/mips/cpu.o
  CC      microblaze-softmmu/hw/microblaze/boot.o
  CC      x86_64-linux-user/target/i386/kvm-stub.o
  CC      xtensaeb-softmmu/hw/xtensa/sim.o
  GEN     trace/generated-helpers.c
  CC      lm32-softmmu/target/lm32/lm32-semi.o
  CC      ppc-softmmu/hw/display/virtio-gpu-pci.o
  CC      x86_64-linux-user/trace/control-target.o
  CC      sparc64-linux-user/target/sparc/vis_helper.o
  CC      microblaze-softmmu/target/microblaze/translate.o
  CC      ppc-softmmu/hw/i2c/ppc4xx_i2c.o
  CC      x86_64-linux-user/gdbstub-xml.o
  CC      s390x-softmmu/hw/display/virtio-gpu-3d.o
  CC      sparc64-linux-user/target/sparc/gdbstub.o
  GEN     trace/generated-helpers.c
  CC      ppcemb-softmmu/hw/i2c/ppc4xx_i2c.o
  CC      mips-softmmu/hw/display/virtio-gpu-3d.o
  CC      tricore-softmmu/trace/generated-helpers.o
  CC      sh4-softmmu/hw/intc/sh_intc.o
  CC      aarch64-softmmu/hw/core/generic-loader.o
  CC      s390x-linux-user/trace/control-target.o
  LINK    or1k-softmmu/qemu-system-or1k
  CC      arm-softmmu/hw/char/bcm2835_aux.o
  LINK    sparc-linux-user/qemu-sparc
  CC      ppc-softmmu/hw/intc/openpic_kvm.o
  CC      sparc-softmmu/target/sparc/monitor.o
  CC      alpha-softmmu/hw/rdma/rdma_utils.o
  GEN     trace/generated-helpers.c
  CC      x86_64-softmmu/hw/intc/apic_common.o
  CC      ppcemb-softmmu/hw/misc/ivshmem.o
  CC      aarch64_be-linux-user/target/arm/iwmmxt_helper.o
  CC      sparc64-softmmu/hw/misc/ivshmem.o
  CC      microblazeel-softmmu/trace/generated-helpers.o
  CC      mips64el-softmmu/hw/misc/mips_cpc.o
  LINK    sparc32plus-linux-user/qemu-sparc32plus
  CC      arm-softmmu/hw/char/virtio-serial-bus.o
  CC      armeb-linux-user/target/arm/helper.o
  CC      ppc64-linux-user/target/ppc/timebase_helper.o
  CC      xtensaeb-softmmu/hw/xtensa/xtfpga.o
  CC      aarch64-softmmu/hw/core/null-machine.o
  CC      ppc64-linux-user/target/ppc/misc_helper.o
  CC      mips64el-linux-user/trace/generated-helpers.o
  CC      s390x-softmmu/hw/display/virtio-gpu-pci.o
  CC      mips64el-softmmu/hw/misc/mips_itu.o
  CC      x86_64-softmmu/hw/intc/ioapic.o
  CC      armeb-linux-user/target/arm/cpu.o
  CC      ppc64-linux-user/target/ppc/mem_helper.o
  CC      lm32-softmmu/target/lm32/machine.o
  CC      mipsel-linux-user/trace/generated-helpers.o
  CC      unicore32-softmmu/target/unicore32/cpu.o
  CC      mips64el-softmmu/hw/misc/mmio_interface.o
  CC      arm-linux-user/target/arm/cpu.o
  CC      ppc-softmmu/hw/misc/ivshmem.o
  CC      aarch64-softmmu/hw/cpu/arm11mpcore.o
  CC      ppc-softmmu/hw/misc/mmio_interface.o
  CC      mips64-softmmu/hw/display/virtio-gpu-pci.o
  CC      mipsn32el-linux-user/target/mips/gdbstub.o
  CC      cris-softmmu/target/cris/mmu.o
  CC      mips64-softmmu/hw/intc/mips_gic.o
  CC      sh4eb-softmmu/hw/net/vhost_net.o
  CC      alpha-softmmu/hw/rdma/rdma_backend.o
  CC      microblaze-softmmu/target/microblaze/op_helper.o
  CC      moxie-softmmu/target/moxie/machine.o
  CC      ppc64-softmmu/hw/core/generic-loader.o
  CC      mipsn32el-linux-user/target/mips/msa_helper.o
  CC      i386-softmmu/hw/core/generic-loader.o
  CC      armeb-linux-user/target/arm/neon_helper.o
  CC      mips-softmmu/hw/display/virtio-gpu-pci.o
  CC      sparc-softmmu/target/sparc/translate.o
  CC      sparc-softmmu/target/sparc/helper.o
  CC      arm-softmmu/hw/core/generic-loader.o
  GEN     trace/generated-helpers.c
  CC      ppc64abi32-linux-user/target/ppc/fpu_helper.o
  CC      microblaze-softmmu/target/microblaze/helper.o
  LINK    microblazeel-softmmu/qemu-system-microblazeel
  CC      lm32-softmmu/trace/control-target.o
  CC      sparc64-linux-user/trace/control-target.o
  CC      ppc64abi32-linux-user/target/ppc/int_helper.o
  CC      aarch64_be-linux-user/target/arm/gdbstub.o
  CC      armeb-linux-user/target/arm/iwmmxt_helper.o
  CC      aarch64_be-linux-user/target/arm/cpu64.o
  CC      s390x-softmmu/hw/intc/s390_flic.o
  CC      ppc64-softmmu/hw/core/null-machine.o
  CC      i386-softmmu/hw/core/null-machine.o
  CC      ppc64abi32-linux-user/target/ppc/timebase_helper.o
  CC      ppc64abi32-linux-user/target/ppc/misc_helper.o
  CC      aarch64-softmmu/hw/cpu/realview_mpcore.o
  CC      sh4-softmmu/hw/misc/ivshmem.o
  CC      ppc64-linux-user/target/ppc/user_only_helper.o
  CC      aarch64-linux-user/trace/generated-helpers.o
  CC      microblaze-softmmu/target/microblaze/cpu.o
  CC      moxie-softmmu/target/moxie/cpu.o
  CC      aarch64_be-linux-user/target/arm/translate-a64.o
  CC      s390x-softmmu/hw/misc/mmio_interface.o
  CC      mips64el-softmmu/hw/net/virtio-net.o
  CC      sparc64-softmmu/hw/misc/mmio_interface.o
  CC      i386-softmmu/hw/display/vga.o
  CC      aarch64_be-linux-user/target/arm/helper-a64.o
  CC      sh4eb-softmmu/hw/rdma/rdma_utils.o
  CC      ppc64abi32-linux-user/target/ppc/mem_helper.o
  CC      xtensaeb-softmmu/target/xtensa/xtensa-semi.o
  CC      sh4-softmmu/hw/misc/mmio_interface.o
  CC      x86_64-softmmu/hw/isa/lpc_ich9.o
  CC      x86_64-softmmu/hw/misc/ivshmem.o
  CC      armeb-linux-user/target/arm/gdbstub.o
  CC      ppc-softmmu/hw/net/xilinx_ethlite.o
  CC      ppc64-softmmu/hw/display/sm501.o
  CC      ppcemb-softmmu/hw/misc/mmio_interface.o
  CC      ppcemb-softmmu/hw/net/xilinx_ethlite.o
  CC      ppc-softmmu/hw/net/virtio-net.o
  CC      armeb-linux-user/target/arm/crypto_helper.o
  GEN     trace/generated-helpers.c
  CC      aarch64_be-linux-user/target/arm/gdbstub64.o
  CC      aarch64_be-linux-user/target/arm/crypto_helper.o
  CC      cris-softmmu/target/cris/machine.o
  CC      aarch64-softmmu/hw/cpu/a9mpcore.o
  CC      x86_64-softmmu/hw/misc/pvpanic.o
  CC      xtensaeb-softmmu/target/xtensa/core-dc232b.o
  CC      arm-softmmu/hw/core/null-machine.o
  CC      mips-softmmu/hw/intc/mips_gic.o
  CC      x86_64-softmmu/hw/misc/mmio_interface.o
  CC      sparc64-linux-user/trace/generated-helpers.o
  CC      arm-linux-user/target/arm/neon_helper.o
  CC      s390x-softmmu/hw/net/virtio-net.o
  CC      ppc64abi32-linux-user/target/ppc/user_only_helper.o
  CC      alpha-softmmu/hw/rdma/rdma_rm.o
  CC      ppc-linux-user/target/ppc/mem_helper.o
  CC      m68k-softmmu/trace/control-target.o
  CC      aarch64-softmmu/hw/cpu/a15mpcore.o
  CC      sparc64-softmmu/hw/net/virtio-net.o
  CC      lm32-softmmu/trace/generated-helpers.o
  CC      microblaze-softmmu/target/microblaze/gdbstub.o
  CC      moxie-softmmu/target/moxie/mmu.o
  CC      mips64-softmmu/hw/misc/ivshmem.o
  CC      mips64-softmmu/hw/misc/mips_cmgcr.o
  CC      sh4eb-softmmu/hw/rdma/rdma_backend.o
  CC      ppc-softmmu/hw/net/vhost_net.o
  CC      armeb-linux-user/trace/control-target.o
  CC      arm-softmmu/hw/cpu/arm11mpcore.o
  CC      microblaze-softmmu/target/microblaze/mmu.o
  CC      sh4eb-softmmu/hw/rdma/rdma_rm.o
  CC      sh4eb-softmmu/hw/rdma/vmw/pvrdma_dev_ring.o
  CC      arm-linux-user/target/arm/iwmmxt_helper.o
  CC      ppcemb-softmmu/hw/net/virtio-net.o
  CC      ppcemb-softmmu/hw/net/vhost_net.o
  CC      s390x-linux-user/gdbstub-xml.o
  CC      aarch64-softmmu/hw/display/omap_dss.o
  CC      armeb-linux-user/gdbstub-xml.o
  CC      ppc-softmmu/hw/net/fsl_etsec/etsec.o
  CC      aarch64-softmmu/hw/display/omap_lcdc.o
  CC      s390x-softmmu/hw/net/vhost_net.o
  CC      i386-softmmu/hw/display/virtio-gpu.o
  CC      i386-softmmu/hw/display/virtio-gpu-3d.o
  CC      aarch64-softmmu/hw/display/pxa2xx_lcd.o
  CC      xtensaeb-softmmu/target/xtensa/core-dc233c.o
  GEN     trace/generated-helpers.c
  CC      ppc64-softmmu/hw/display/vga.o
  CC      s390x-linux-user/trace/generated-helpers.o
  CC      ppc-softmmu/hw/net/fsl_etsec/registers.o
  CC      sh4-softmmu/hw/net/virtio-net.o
  CC      sh4-softmmu/hw/net/vhost_net.o
  CC      ppc64abi32-linux-user/target/ppc/gdbstub.o
  CC      sh4-softmmu/hw/rdma/rdma_utils.o
  GEN     trace/generated-helpers.c
  CC      alpha-softmmu/hw/rdma/vmw/pvrdma_dev_ring.o
  CC      arm-softmmu/hw/cpu/realview_mpcore.o
  CC      moxie-softmmu/trace/control-target.o
  CC      arm-softmmu/hw/cpu/a9mpcore.o
  LINK    lm32-softmmu/qemu-system-lm32
  CC      xtensaeb-softmmu/target/xtensa/core-fsf.o
  CC      mipsn32el-linux-user/target/mips/mips-semi.o
  CC      alpha-softmmu/hw/rdma/vmw/pvrdma_cmd.o
  LINK    sparc64-linux-user/qemu-sparc64
  CC      unicore32-softmmu/target/unicore32/ucf64_helper.o
  CC      ppc-softmmu/hw/net/fsl_etsec/rings.o
  CC      aarch64_be-linux-user/trace/control-target.o
  CC      ppcemb-softmmu/hw/rdma/rdma_utils.o
  CC      s390x-softmmu/hw/net/rocker/qmp-norocker.o
  CC      sh4eb-softmmu/hw/rdma/vmw/pvrdma_cmd.o
  GEN     trace/generated-helpers.c
  CC      armeb-linux-user/trace/generated-helpers.o
  GEN     trace/generated-helpers.c
  CC      cris-softmmu/trace/control-target.o
  CC      mips-softmmu/hw/misc/ivshmem.o
  CC      s390x-softmmu/hw/rdma/rdma_utils.o
  CC      arm-linux-user/target/arm/gdbstub.o
  CC      arm-softmmu/hw/cpu/a15mpcore.o
  CC      sh4eb-softmmu/hw/rdma/vmw/pvrdma_qp_ops.o
  GEN     trace/generated-helpers.c
  CC      microblaze-softmmu/trace/control-target.o
  CC      x86_64-softmmu/hw/net/virtio-net.o
  CC      aarch64_be-linux-user/gdbstub-xml.o
  CC      ppc64abi32-linux-user/target/ppc/../../libdecnumber/decContext.o
  CC      sh4-softmmu/hw/rdma/rdma_backend.o
  CC      x86_64-softmmu/hw/net/vhost_net.o
  CC      sparc64-softmmu/hw/net/vhost_net.o
  CC      mipsn32el-linux-user/trace/control-target.o
  CC      ppc-linux-user/target/ppc/user_only_helper.o
  CC      sh4eb-softmmu/hw/rdma/vmw/pvrdma_main.o
  CC      mips64el-softmmu/hw/net/vhost_net.o
  CC      ppc64abi32-linux-user/target/ppc/../../libdecnumber/decNumber.o
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c: In function ‘create_qp_rings’:
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: implicit declaration of function ‘pow2roundup32’ [-Werror=implicit-function-declaration]
     wqe_sz = pow2roundup32(sizeof(struct pvrdma_sq_wqe_hdr) +
     ^
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: nested extern declaration of ‘pow2roundup32’ [-Werror=nested-externs]
cc1: all warnings being treated as errors
make[1]: *** [hw/rdma/vmw/pvrdma_cmd.o] Error 1
make[1]: *** Waiting for unfinished jobs....
  CC      i386-softmmu/hw/display/virtio-gpu-pci.o
  CC      unicore32-softmmu/target/unicore32/softmmu.o
  CC      moxie-softmmu/trace/generated-helpers.o
  CC      aarch64-softmmu/hw/display/bcm2835_fb.o
  CC      sh4eb-softmmu/hw/scsi/virtio-scsi.o
  CC      sh4eb-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      ppc-softmmu/hw/net/fsl_etsec/miim.o
  CC      ppc-softmmu/hw/rdma/rdma_utils.o
  CC      x86_64-softmmu/hw/rdma/rdma_utils.o
  CC      mips64-softmmu/hw/misc/mips_cpc.o
  CC      ppc-softmmu/hw/rdma/rdma_backend.o
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c: In function ‘create_qp_rings’:
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: implicit declaration of function ‘pow2roundup32’ [-Werror=implicit-function-declaration]
     wqe_sz = pow2roundup32(sizeof(struct pvrdma_sq_wqe_hdr) +
     ^
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: nested extern declaration of ‘pow2roundup32’ [-Werror=nested-externs]
cc1: all warnings being treated as errors
make[1]: *** [hw/rdma/vmw/pvrdma_cmd.o] Error 1
make[1]: *** Waiting for unfinished jobs....
  CC      arm-linux-user/target/arm/crypto_helper.o
  CC      arm-softmmu/hw/display/omap_dss.o
  CC      arm-softmmu/hw/display/omap_lcdc.o
  CC      ppc64-softmmu/hw/display/virtio-gpu.o
  CC      sparc-softmmu/target/sparc/cpu.o
  CC      mips-softmmu/hw/misc/mips_cmgcr.o
  CC      ppc-linux-user/target/ppc/gdbstub.o
make: *** [subdir-alpha-softmmu] Error 2
  CC      sparc64-softmmu/hw/rdma/rdma_utils.o
  CC      mips-softmmu/hw/misc/mips_cpc.o
  CC      cris-softmmu/trace/generated-helpers.o
  CC      sparc64-softmmu/hw/rdma/rdma_backend.o
  CC      ppc64-linux-user/target/ppc/gdbstub.o
  CC      mips64-softmmu/hw/misc/mips_itu.o
  CC      microblaze-softmmu/trace/generated-helpers.o
  CC      ppc64-linux-user/target/ppc/../../libdecnumber/decContext.o
  CC      sparc-softmmu/target/sparc/fop_helper.o
  CC      ppc64-softmmu/hw/display/virtio-gpu-3d.o
  GEN     trace/generated-helpers.c
  CC      sparc-softmmu/target/sparc/cc_helper.o
  CC      s390x-softmmu/hw/rdma/rdma_backend.o
  CC      sparc-softmmu/target/sparc/win_helper.o
  CC      aarch64-softmmu/hw/display/vga.o
  CC      ppc64abi32-linux-user/target/ppc/../../libdecnumber/dpd/decimal32.o
  CC      ppc-softmmu/hw/rdma/rdma_rm.o
  CC      aarch64-softmmu/hw/display/virtio-gpu.o
  CC      mips-softmmu/hw/misc/mips_itu.o
  CC      ppc64abi32-linux-user/target/ppc/../../libdecnumber/dpd/decimal64.o
  GEN     trace/generated-helpers.c
  CC      aarch64_be-linux-user/trace/generated-helpers.o
  CC      mips64el-softmmu/hw/rdma/rdma_utils.o
  CC      ppc64abi32-linux-user/target/ppc/../../libdecnumber/dpd/decimal128.o
  CC      mips64-softmmu/hw/misc/mmio_interface.o
  CC      ppc64-linux-user/target/ppc/../../libdecnumber/decNumber.o
  CC      xtensaeb-softmmu/target/xtensa/monitor.o
  CC      ppcemb-softmmu/hw/rdma/rdma_backend.o
  CC      ppc-softmmu/hw/rdma/vmw/pvrdma_dev_ring.o
  CC      m68k-softmmu/gdbstub-xml.o
  CC      ppc-linux-user/target/ppc/../../libdecnumber/decContext.o
  CC      sh4-softmmu/hw/rdma/rdma_rm.o
  LINK    moxie-softmmu/qemu-system-moxie
  CC      ppc-softmmu/hw/rdma/vmw/pvrdma_cmd.o
  CC      i386-softmmu/hw/display/virtio-vga.o
  CC      arm-linux-user/trace/control-target.o
  CC      mips-softmmu/hw/misc/mmio_interface.o
  GEN     trace/generated-helpers.c
  CC      ppc-softmmu/hw/rdma/vmw/pvrdma_qp_ops.o
  CC      sparc-softmmu/target/sparc/mmu_helper.o
  CC      unicore32-softmmu/trace/control-target.o
  CC      i386-softmmu/hw/intc/apic.o
  CC      ppc64abi32-linux-user/trace/control-target.o
  CC      ppc-softmmu/hw/rdma/vmw/pvrdma_main.o
  CC      mips64el-softmmu/hw/rdma/rdma_backend.o
  CC      ppc-softmmu/hw/scsi/virtio-scsi.o
  CC      ppc-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      sh4-softmmu/hw/rdma/vmw/pvrdma_dev_ring.o
  CC      arm-softmmu/hw/display/pxa2xx_lcd.o
  CC      ppc64-linux-user/target/ppc/../../libdecnumber/dpd/decimal32.o
  CC      ppc-softmmu/hw/scsi/vhost-scsi-common.o
  CC      arm-softmmu/hw/display/bcm2835_fb.o
  LINK    microblaze-softmmu/qemu-system-microblaze
  CC      i386-softmmu/hw/intc/apic_common.o
  CC      ppc64-softmmu/hw/display/virtio-gpu-pci.o
  CC      mips-softmmu/hw/net/virtio-net.o
make: *** [subdir-sh4eb-softmmu] Error 2
  CC      mips-softmmu/hw/net/vhost_net.o
  CC      ppc64-softmmu/hw/display/virtio-vga.o
  CC      mips64-softmmu/hw/net/virtio-net.o
  CC      sparc-softmmu/target/sparc/ldst_helper.o
  CC      mips-softmmu/hw/rdma/rdma_utils.o
  CC      ppc-linux-user/target/ppc/../../libdecnumber/decNumber.o
  CC      s390x-softmmu/hw/rdma/rdma_rm.o
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c: In function ‘create_qp_rings’:
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: implicit declaration of function ‘pow2roundup32’ [-Werror=implicit-function-declaration]
     wqe_sz = pow2roundup32(sizeof(struct pvrdma_sq_wqe_hdr) +
     ^
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: nested extern declaration of ‘pow2roundup32’ [-Werror=nested-externs]
cc1: all warnings being treated as errors
make[1]: *** [hw/rdma/vmw/pvrdma_cmd.o] Error 1
make[1]: *** Waiting for unfinished jobs....
  CC      x86_64-softmmu/hw/rdma/rdma_backend.o
  CC      x86_64-softmmu/hw/rdma/rdma_rm.o
  CC      mips64el-softmmu/hw/rdma/rdma_rm.o
  CC      mips64-softmmu/hw/net/vhost_net.o
  CC      i386-softmmu/hw/intc/ioapic.o
  CC      ppc64-softmmu/hw/i2c/ppc4xx_i2c.o
  CC      arm-linux-user/gdbstub-xml.o
  CC      sparc64-softmmu/hw/rdma/rdma_rm.o
  CC      sparc64-softmmu/hw/rdma/vmw/pvrdma_dev_ring.o
  CC      i386-softmmu/hw/isa/lpc_ich9.o
  CC      arm-softmmu/hw/display/vga.o
  CC      ppc64-softmmu/hw/intc/openpic_kvm.o
  CC      i386-softmmu/hw/misc/ivshmem.o
  CC      unicore32-softmmu/trace/generated-helpers.o
  CC      mips64el-softmmu/hw/rdma/vmw/pvrdma_dev_ring.o
  CC      ppc64-softmmu/hw/intc/xics.o
  CC      ppc64-linux-user/target/ppc/../../libdecnumber/dpd/decimal64.o
  CC      ppc64-linux-user/target/ppc/../../libdecnumber/dpd/decimal128.o
  CC      sparc-softmmu/target/sparc/int32_helper.o
  CC      i386-softmmu/hw/misc/pvpanic.o
  CC      ppcemb-softmmu/hw/rdma/rdma_rm.o
  CC      sparc64-softmmu/hw/rdma/vmw/pvrdma_cmd.o
  CC      arm-softmmu/hw/display/virtio-gpu.o
  CC      i386-softmmu/hw/misc/mmio_interface.o
  CC      x86_64-softmmu/hw/rdma/vmw/pvrdma_dev_ring.o
  CC      sh4-softmmu/hw/rdma/vmw/pvrdma_cmd.o
  CC      ppc-linux-user/target/ppc/../../libdecnumber/dpd/decimal32.o
  CC      i386-softmmu/hw/net/virtio-net.o
  CC      mips-softmmu/hw/rdma/rdma_backend.o
  CC      mips64-softmmu/hw/rdma/rdma_utils.o
  CC      xtensa-softmmu/trace/generated-helpers.o
  CC      sparc64-softmmu/hw/rdma/vmw/pvrdma_qp_ops.o
  CC      ppc64abi32-linux-user/gdbstub-xml.o
  CC      ppcemb-softmmu/hw/rdma/vmw/pvrdma_dev_ring.o
  CC      mips64-softmmu/hw/rdma/rdma_backend.o
  CC      mipsn32el-linux-user/trace/generated-helpers.o
  CC      sparc64-softmmu/hw/rdma/vmw/pvrdma_main.o
  CC      xtensaeb-softmmu/target/xtensa/xtensa-isa.o
  CC      ppc64-softmmu/hw/intc/xics_spapr.o
  CC      ppc64-softmmu/hw/intc/xics_kvm.o
  CC      mips-softmmu/hw/rdma/rdma_rm.o
  CC      ppc-linux-user/target/ppc/../../libdecnumber/dpd/decimal64.o
  CC      ppc64-softmmu/hw/intc/xics_pnv.o
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c: In function ‘create_qp_rings’:
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: implicit declaration of function ‘pow2roundup32’ [-Werror=implicit-function-declaration]
     wqe_sz = pow2roundup32(sizeof(struct pvrdma_sq_wqe_hdr) +
     ^
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: nested extern declaration of ‘pow2roundup32’ [-Werror=nested-externs]
  CC      mips64el-softmmu/hw/rdma/vmw/pvrdma_cmd.o
  CC      s390x-softmmu/hw/rdma/vmw/pvrdma_dev_ring.o
cc1: all warnings being treated as errors
make[1]: *** [hw/rdma/vmw/pvrdma_cmd.o] Error 1
make[1]: *** Waiting for unfinished jobs....
  CC      s390x-softmmu/hw/rdma/vmw/pvrdma_cmd.o
  CC      xtensaeb-softmmu/target/xtensa/translate.o
  LINK    unicore32-softmmu/qemu-system-unicore32
  CC      mips64-softmmu/hw/rdma/rdma_rm.o
  CC      mips64el-softmmu/hw/rdma/vmw/pvrdma_qp_ops.o
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c: In function ‘create_qp_rings’:
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: implicit declaration of function ‘pow2roundup32’ [-Werror=implicit-function-declaration]
     wqe_sz = pow2roundup32(sizeof(struct pvrdma_sq_wqe_hdr) +
     ^
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: nested extern declaration of ‘pow2roundup32’ [-Werror=nested-externs]
  CC      x86_64-softmmu/hw/rdma/vmw/pvrdma_cmd.o
cc1: all warnings being treated as errors
  CC      mips64-softmmu/hw/rdma/vmw/pvrdma_dev_ring.o
make[1]: *** [hw/rdma/vmw/pvrdma_cmd.o] Error 1
  CC      s390x-softmmu/hw/rdma/vmw/pvrdma_qp_ops.o
make: *** [subdir-sh4-softmmu] Error 2
  CC      m68k-softmmu/trace/generated-helpers.o
  GEN     trace/generated-helpers.c
  CC      arm-linux-user/trace/generated-helpers.o
  CC      arm-softmmu/hw/display/virtio-gpu-3d.o
  CC      mips-softmmu/hw/rdma/vmw/pvrdma_dev_ring.o
  CC      ppc64-linux-user/trace/control-target.o
  CC      xtensaeb-softmmu/target/xtensa/op_helper.o
  CC      mips64-softmmu/hw/rdma/vmw/pvrdma_cmd.o
  CC      xtensaeb-softmmu/target/xtensa/helper.o
  CC      x86_64-linux-user/trace/generated-helpers.o
  CC      xtensaeb-softmmu/target/xtensa/cpu.o
  CC      xtensaeb-softmmu/target/xtensa/gdbstub.o
  CC      mips64el-softmmu/hw/rdma/vmw/pvrdma_main.o
  GEN     trace/generated-helpers.c
  CC      ppc64abi32-linux-user/trace/generated-helpers.o
  CC      mips-softmmu/hw/rdma/vmw/pvrdma_cmd.o
  CC      sparc-softmmu/target/sparc/gdbstub.o
  LINK    xtensa-softmmu/qemu-system-xtensa
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c: In function ‘create_qp_rings’:
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: implicit declaration of function ‘pow2roundup32’ [-Werror=implicit-function-declaration]
     wqe_sz = pow2roundup32(sizeof(struct pvrdma_sq_wqe_hdr) +
     ^
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: nested extern declaration of ‘pow2roundup32’ [-Werror=nested-externs]
cc1: all warnings being treated as errors
make[1]: *** [hw/rdma/vmw/pvrdma_cmd.o] Error 1
make[1]: *** Waiting for unfinished jobs....
  GEN     trace/generated-helpers.c
make: *** [subdir-ppc-softmmu] Error 2
  CC      ppc-linux-user/target/ppc/../../libdecnumber/dpd/decimal128.o
  LINK    cris-softmmu/qemu-system-cris
  CC      ppc64-softmmu/hw/misc/ivshmem.o
  CC      x86_64-softmmu/hw/rdma/vmw/pvrdma_qp_ops.o
  CC      xtensaeb-softmmu/trace/control-target.o
  CC      mips-softmmu/hw/rdma/vmw/pvrdma_qp_ops.o
  CC      aarch64-softmmu/hw/display/virtio-gpu-3d.o
  CC      mips-softmmu/hw/rdma/vmw/pvrdma_main.o
  CC      mips-softmmu/hw/scsi/virtio-scsi.o
  CC      ppc64-softmmu/hw/misc/mmio_interface.o
  CC      mips-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      ppcemb-softmmu/hw/rdma/vmw/pvrdma_cmd.o
  CC      aarch64-softmmu/hw/display/virtio-gpu-pci.o
  GEN     trace/generated-helpers.c
  CC      ppc-linux-user/trace/control-target.o
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c: In function ‘create_qp_rings’:
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: implicit declaration of function ‘pow2roundup32’ [-Werror=implicit-function-declaration]
     wqe_sz = pow2roundup32(sizeof(struct pvrdma_sq_wqe_hdr) +
     ^
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: nested extern declaration of ‘pow2roundup32’ [-Werror=nested-externs]
  CC      arm-softmmu/hw/display/virtio-gpu-pci.o
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c: In function ‘create_qp_rings’:
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: implicit declaration of function ‘pow2roundup32’ [-Werror=implicit-function-declaration]
     wqe_sz = pow2roundup32(sizeof(struct pvrdma_sq_wqe_hdr) +
     ^
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: nested extern declaration of ‘pow2roundup32’ [-Werror=nested-externs]
cc1: all warnings being treated as errors
  CC      aarch64-softmmu/hw/display/dpcd.o
make[1]: *** [hw/rdma/vmw/pvrdma_cmd.o] Error 1
make[1]: *** Waiting for unfinished jobs....
  CC      mips64-softmmu/hw/rdma/vmw/pvrdma_qp_ops.o
cc1: all warnings being treated as errors
make[1]: *** [hw/rdma/vmw/pvrdma_cmd.o] Error 1
make[1]: *** Waiting for unfinished jobs....
  CC      aarch64-softmmu/hw/display/xlnx_dp.o
  CC      ppc64-softmmu/hw/net/spapr_llan.o
  CC      arm-softmmu/hw/dma/omap_dma.o
  CC      ppc-linux-user/gdbstub-xml.o
make: *** [subdir-sparc64-softmmu] Error 2
  CC      mips-softmmu/hw/scsi/vhost-scsi.o
  CC      mips-softmmu/hw/scsi/vhost-scsi-common.o
  CC      aarch64-softmmu/hw/dma/xlnx_dpdma.o
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c: In function ‘create_qp_rings’:
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: implicit declaration of function ‘pow2roundup32’ [-Werror=implicit-function-declaration]
     wqe_sz = pow2roundup32(sizeof(struct pvrdma_sq_wqe_hdr) +
     ^
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: nested extern declaration of ‘pow2roundup32’ [-Werror=nested-externs]
  CC      ppc64-softmmu/hw/net/xilinx_ethlite.o
  CC      arm-softmmu/hw/dma/soc_dma.o
  CC      sparc-softmmu/trace/control-target.o
cc1: all warnings being treated as errors
make[1]: *** [hw/rdma/vmw/pvrdma_cmd.o] Error 1
make[1]: *** Waiting for unfinished jobs....
  CC      i386-softmmu/hw/net/vhost_net.o
  CC      ppc-linux-user/trace/generated-helpers.o
  CC      ppc64-softmmu/hw/net/virtio-net.o
  CC      i386-softmmu/hw/rdma/rdma_utils.o
  CC      arm-softmmu/hw/dma/pxa2xx_dma.o
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c: In function ‘create_qp_rings’:
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: implicit declaration of function ‘pow2roundup32’ [-Werror=implicit-function-declaration]
     wqe_sz = pow2roundup32(sizeof(struct pvrdma_sq_wqe_hdr) +
     ^
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: nested extern declaration of ‘pow2roundup32’ [-Werror=nested-externs]
  CC      ppc64-linux-user/gdbstub-xml.o
  CC      mips-softmmu/hw/scsi/vhost-user-scsi.o
cc1: all warnings being treated as errors
make: *** [subdir-s390x-softmmu] Error 2
make[1]: *** [hw/rdma/vmw/pvrdma_cmd.o] Error 1
make[1]: *** Waiting for unfinished jobs....
  CC      i386-softmmu/hw/rdma/rdma_backend.o
  CC      mips-softmmu/hw/timer/mips_gictimer.o
  CC      ppcemb-softmmu/hw/rdma/vmw/pvrdma_qp_ops.o
  CC      xtensaeb-softmmu/trace/generated-helpers.o
  CC      ppc64-softmmu/hw/net/vhost_net.o
  CC      aarch64-softmmu/hw/dma/omap_dma.o
  CC      i386-softmmu/hw/rdma/rdma_rm.o
  CC      sparc-softmmu/trace/generated-helpers.o
  CC      arm-softmmu/hw/dma/bcm2835_dma.o
  CC      aarch64-softmmu/hw/dma/soc_dma.o
  CC      ppc64-linux-user/trace/generated-helpers.o
  CC      ppcemb-softmmu/hw/rdma/vmw/pvrdma_main.o
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c: In function ‘create_qp_rings’:
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: implicit declaration of function ‘pow2roundup32’ [-Werror=implicit-function-declaration]
     wqe_sz = pow2roundup32(sizeof(struct pvrdma_sq_wqe_hdr) +
     ^
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: nested extern declaration of ‘pow2roundup32’ [-Werror=nested-externs]
  CC      ppcemb-softmmu/hw/scsi/virtio-scsi.o
cc1: all warnings being treated as errors
make[1]: *** [hw/rdma/vmw/pvrdma_cmd.o] Error 1
make[1]: *** Waiting for unfinished jobs....
  CC      ppc64-softmmu/hw/net/fsl_etsec/etsec.o
  CC      ppc64-softmmu/hw/net/fsl_etsec/registers.o
  CC      arm-softmmu/hw/gpio/omap_gpio.o
make: *** [subdir-x86_64-softmmu] Error 2
  CC      i386-softmmu/hw/rdma/vmw/pvrdma_dev_ring.o
  CC      i386-softmmu/hw/rdma/vmw/pvrdma_cmd.o
  CC      ppc64-softmmu/hw/net/fsl_etsec/rings.o
  CC      aarch64-softmmu/hw/dma/pxa2xx_dma.o
  CC      ppc64-softmmu/hw/net/fsl_etsec/miim.o
  CC      ppc64-softmmu/hw/nvram/spapr_nvram.o
  CC      arm-softmmu/hw/gpio/imx_gpio.o
make: *** [subdir-mips64el-softmmu] Error 2
  CC      i386-softmmu/hw/rdma/vmw/pvrdma_qp_ops.o
  CC      ppc64-softmmu/hw/rdma/rdma_utils.o
  CC      arm-softmmu/hw/gpio/bcm2835_gpio.o
  CC      ppc64-softmmu/hw/rdma/rdma_backend.o
  CC      ppc64-softmmu/hw/rdma/rdma_rm.o
  CC      aarch64-softmmu/hw/dma/bcm2835_dma.o
  CC      aarch64-softmmu/hw/gpio/omap_gpio.o
  CC      i386-softmmu/hw/rdma/vmw/pvrdma_main.o
  CC      ppc64-softmmu/hw/rdma/vmw/pvrdma_dev_ring.o
  CC      arm-softmmu/hw/i2c/omap_i2c.o
  CC      arm-softmmu/hw/input/pxa2xx_keypad.o
  CC      aarch64-softmmu/hw/gpio/imx_gpio.o
  CC      i386-softmmu/hw/scsi/virtio-scsi.o
  CC      i386-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      i386-softmmu/hw/scsi/vhost-scsi-common.o
  CC      arm-softmmu/hw/input/tsc210x.o
  CC      aarch64-softmmu/hw/gpio/bcm2835_gpio.o
  CC      ppc64-softmmu/hw/rdma/vmw/pvrdma_cmd.o
  CC      ppc64-softmmu/hw/rdma/vmw/pvrdma_qp_ops.o
  CC      aarch64-softmmu/hw/i2c/omap_i2c.o
  CC      i386-softmmu/hw/scsi/vhost-scsi.o
  CC      i386-softmmu/hw/scsi/vhost-user-scsi.o
  CC      aarch64-softmmu/hw/input/pxa2xx_keypad.o
  CC      arm-softmmu/hw/intc/armv7m_nvic.o
  CC      ppc64-softmmu/hw/rdma/vmw/pvrdma_main.o
  CC      ppc64-softmmu/hw/scsi/spapr_vscsi.o
make: *** [subdir-mips64-softmmu] Error 2
  CC      ppc64-softmmu/hw/scsi/virtio-scsi.o
  CC      i386-softmmu/hw/timer/mc146818rtc.o
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c: In function ‘create_qp_rings’:
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: implicit declaration of function ‘pow2roundup32’ [-Werror=implicit-function-declaration]
     wqe_sz = pow2roundup32(sizeof(struct pvrdma_sq_wqe_hdr) +
     ^
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: nested extern declaration of ‘pow2roundup32’ [-Werror=nested-externs]
  CC      ppc64-softmmu/hw/scsi/virtio-scsi-dataplane.o
cc1: all warnings being treated as errors
  CC      ppc64-softmmu/hw/scsi/vhost-scsi-common.o
make[1]: *** [hw/rdma/vmw/pvrdma_cmd.o] Error 1
make[1]: *** Waiting for unfinished jobs....
  CC      aarch64-softmmu/hw/input/tsc210x.o
  CC      ppc64-softmmu/hw/scsi/vhost-scsi.o
  CC      arm-softmmu/hw/intc/exynos4210_gic.o
  CC      aarch64-softmmu/hw/intc/armv7m_nvic.o
  CC      aarch64-softmmu/hw/intc/exynos4210_gic.o
  CC      aarch64-softmmu/hw/intc/exynos4210_combiner.o
  CC      arm-softmmu/hw/intc/exynos4210_combiner.o
  CC      ppc64-softmmu/hw/scsi/vhost-user-scsi.o
  CC      ppc64-softmmu/hw/timer/mc146818rtc.o
  CC      ppc64-softmmu/hw/vfio/common.o
  CC      arm-softmmu/hw/intc/omap_intc.o
  LINK    sparc-softmmu/qemu-system-sparc
  CC      ppc64-softmmu/hw/vfio/pci.o
  CC      aarch64-softmmu/hw/intc/omap_intc.o
  CC      aarch64-softmmu/hw/intc/bcm2835_ic.o
  CC      arm-softmmu/hw/intc/bcm2835_ic.o
  CC      ppc64-softmmu/hw/vfio/pci-quirks.o
  CC      aarch64-softmmu/hw/intc/bcm2836_control.o
  CC      ppc64-softmmu/hw/vfio/platform.o
  CC      aarch64-softmmu/hw/intc/allwinner-a10-pic.o
  CC      arm-softmmu/hw/intc/bcm2836_control.o
  CC      ppc64-softmmu/hw/vfio/spapr.o
  CC      arm-softmmu/hw/intc/allwinner-a10-pic.o
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c: In function ‘create_qp_rings’:
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: implicit declaration of function ‘pow2roundup32’ [-Werror=implicit-function-declaration]
     wqe_sz = pow2roundup32(sizeof(struct pvrdma_sq_wqe_hdr) +
     ^
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: nested extern declaration of ‘pow2roundup32’ [-Werror=nested-externs]
  CC      aarch64-softmmu/hw/intc/aspeed_vic.o
cc1: all warnings being treated as errors
make[1]: *** [hw/rdma/vmw/pvrdma_cmd.o] Error 1
make[1]: *** Waiting for unfinished jobs....
  CC      aarch64-softmmu/hw/intc/arm_gicv3_cpuif.o
  CC      arm-softmmu/hw/intc/aspeed_vic.o
  CC      arm-softmmu/hw/intc/arm_gicv3_cpuif.o
  CC      aarch64-softmmu/hw/misc/ivshmem.o
  CC      arm-softmmu/hw/misc/ivshmem.o
  CC      arm-softmmu/hw/misc/arm_sysctl.o
  CC      arm-softmmu/hw/misc/cbus.o
  CC      arm-softmmu/hw/misc/exynos4210_pmu.o
  CC      arm-softmmu/hw/misc/exynos4210_clk.o
  CC      aarch64-softmmu/hw/misc/arm_sysctl.o
  CC      aarch64-softmmu/hw/misc/cbus.o
  CC      aarch64-softmmu/hw/misc/exynos4210_pmu.o
  CC      arm-softmmu/hw/misc/exynos4210_rng.o
  CC      arm-softmmu/hw/misc/imx_ccm.o
  CC      aarch64-softmmu/hw/misc/exynos4210_clk.o
  CC      arm-softmmu/hw/misc/imx31_ccm.o
  CC      arm-softmmu/hw/misc/imx25_ccm.o
  CC      arm-softmmu/hw/misc/imx6_ccm.o
  LINK    s390x-linux-user/qemu-s390x
make: *** [subdir-mips-softmmu] Error 2
  CC      arm-softmmu/hw/misc/imx6_src.o
  CC      arm-softmmu/hw/misc/mst_fpga.o
  CC      aarch64-softmmu/hw/misc/exynos4210_rng.o
  CC      arm-softmmu/hw/misc/omap_clk.o
  CC      aarch64-softmmu/hw/misc/imx_ccm.o
  CC      aarch64-softmmu/hw/misc/imx31_ccm.o
  CC      arm-softmmu/hw/misc/omap_gpmc.o
  CC      arm-softmmu/hw/misc/omap_l4.o
  CC      aarch64-softmmu/hw/misc/imx25_ccm.o
  CC      aarch64-softmmu/hw/misc/imx6_ccm.o
  CC      aarch64-softmmu/hw/misc/imx6_src.o
  CC      aarch64-softmmu/hw/misc/mst_fpga.o
  CC      arm-softmmu/hw/misc/omap_sdrc.o
  CC      aarch64-softmmu/hw/misc/omap_clk.o
  CC      arm-softmmu/hw/misc/omap_tap.o
  CC      arm-softmmu/hw/misc/bcm2835_property.o
  CC      arm-softmmu/hw/misc/bcm2835_mbox.o
  CC      aarch64-softmmu/hw/misc/omap_gpmc.o
  CC      arm-softmmu/hw/misc/bcm2835_rng.o
  CC      aarch64-softmmu/hw/misc/omap_l4.o
make: *** [subdir-ppcemb-softmmu] Error 2
  CC      arm-softmmu/hw/misc/zynq_slcr.o
  CC      aarch64-softmmu/hw/misc/omap_sdrc.o
  CC      arm-softmmu/hw/misc/zynq-xadc.o
  CC      arm-softmmu/hw/misc/mps2-scc.o
  CC      arm-softmmu/hw/misc/stm32f2xx_syscfg.o
  CC      aarch64-softmmu/hw/misc/omap_tap.o
  CC      aarch64-softmmu/hw/misc/bcm2835_mbox.o
  CC      arm-softmmu/hw/misc/aspeed_scu.o
  CC      arm-softmmu/hw/misc/aspeed_sdmc.o
  CC      aarch64-softmmu/hw/misc/bcm2835_property.o
  CC      aarch64-softmmu/hw/misc/bcm2835_rng.o
  CC      arm-softmmu/hw/misc/mmio_interface.o
  CC      arm-softmmu/hw/misc/msf2-sysreg.o
  CC      aarch64-softmmu/hw/misc/zynq_slcr.o
  CC      arm-softmmu/hw/net/virtio-net.o
  CC      arm-softmmu/hw/net/vhost_net.o
  CC      arm-softmmu/hw/pcmcia/pxa2xx.o
  CC      aarch64-softmmu/hw/misc/zynq-xadc.o
  CC      arm-softmmu/hw/rdma/rdma_utils.o
  LINK    m68k-softmmu/qemu-system-m68k
  CC      arm-softmmu/hw/rdma/rdma_backend.o
  CC      arm-softmmu/hw/rdma/rdma_rm.o
  CC      aarch64-softmmu/hw/misc/stm32f2xx_syscfg.o
  CC      aarch64-softmmu/hw/misc/mps2-scc.o
  CC      arm-softmmu/hw/rdma/vmw/pvrdma_dev_ring.o
  CC      aarch64-softmmu/hw/misc/auxbus.o
  CC      aarch64-softmmu/hw/misc/aspeed_scu.o
  CC      aarch64-softmmu/hw/misc/aspeed_sdmc.o
  CC      arm-softmmu/hw/rdma/vmw/pvrdma_cmd.o
  CC      arm-softmmu/hw/rdma/vmw/pvrdma_qp_ops.o
  CC      arm-softmmu/hw/rdma/vmw/pvrdma_main.o
  CC      arm-softmmu/hw/scsi/virtio-scsi.o
  CC      aarch64-softmmu/hw/misc/mmio_interface.o
  CC      aarch64-softmmu/hw/misc/msf2-sysreg.o
  CC      aarch64-softmmu/hw/net/virtio-net.o
  CC      arm-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      aarch64-softmmu/hw/net/vhost_net.o
  CC      arm-softmmu/hw/scsi/vhost-scsi-common.o
  CC      aarch64-softmmu/hw/pcmcia/pxa2xx.o
  CC      aarch64-softmmu/hw/rdma/rdma_utils.o
  CC      arm-softmmu/hw/scsi/vhost-scsi.o
  CC      aarch64-softmmu/hw/rdma/rdma_backend.o
  CC      arm-softmmu/hw/scsi/vhost-user-scsi.o
  CC      aarch64-softmmu/hw/rdma/rdma_rm.o
  CC      aarch64-softmmu/hw/rdma/vmw/pvrdma_dev_ring.o
  CC      arm-softmmu/hw/sd/omap_mmc.o
  CC      aarch64-softmmu/hw/rdma/vmw/pvrdma_cmd.o
  CC      aarch64-softmmu/hw/rdma/vmw/pvrdma_qp_ops.o
  CC      arm-softmmu/hw/sd/pxa2xx_mmci.o
  CC      aarch64-softmmu/hw/rdma/vmw/pvrdma_main.o
make: *** [subdir-i386-softmmu] Error 2
  CC      aarch64-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      aarch64-softmmu/hw/scsi/virtio-scsi.o
  CC      arm-softmmu/hw/sd/bcm2835_sdhost.o
  CC      aarch64-softmmu/hw/scsi/vhost-scsi-common.o
  CC      aarch64-softmmu/hw/scsi/vhost-scsi.o
  CC      aarch64-softmmu/hw/scsi/vhost-user-scsi.o
  CC      arm-softmmu/hw/ssi/omap_spi.o
  CC      aarch64-softmmu/hw/sd/omap_mmc.o
  CC      aarch64-softmmu/hw/sd/pxa2xx_mmci.o
  CC      arm-softmmu/hw/ssi/imx_spi.o
  CC      arm-softmmu/hw/timer/exynos4210_mct.o
  CC      arm-softmmu/hw/timer/exynos4210_pwm.o
  CC      arm-softmmu/hw/timer/exynos4210_rtc.o
  CC      arm-softmmu/hw/timer/omap_gptimer.o
  CC      aarch64-softmmu/hw/sd/bcm2835_sdhost.o
  CC      arm-softmmu/hw/timer/omap_synctimer.o
  CC      arm-softmmu/hw/timer/pxa2xx_timer.o
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c: In function ‘create_qp_rings’:
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: implicit declaration of function ‘pow2roundup32’ [-Werror=implicit-function-declaration]
     wqe_sz = pow2roundup32(sizeof(struct pvrdma_sq_wqe_hdr) +
     ^
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: nested extern declaration of ‘pow2roundup32’ [-Werror=nested-externs]
  CC      arm-softmmu/hw/timer/digic-timer.o
cc1: all warnings being treated as errors
make[1]: *** [hw/rdma/vmw/pvrdma_cmd.o] Error 1
make[1]: *** Waiting for unfinished jobs....
  CC      aarch64-softmmu/hw/ssi/omap_spi.o
  CC      aarch64-softmmu/hw/ssi/imx_spi.o
  CC      aarch64-softmmu/hw/timer/exynos4210_mct.o
  CC      aarch64-softmmu/hw/timer/exynos4210_pwm.o
  CC      aarch64-softmmu/hw/timer/exynos4210_rtc.o
  CC      aarch64-softmmu/hw/timer/omap_gptimer.o
  CC      aarch64-softmmu/hw/timer/omap_synctimer.o
  CC      aarch64-softmmu/hw/timer/pxa2xx_timer.o
  CC      aarch64-softmmu/hw/timer/digic-timer.o
  CC      aarch64-softmmu/hw/timer/allwinner-a10-pit.o
  CC      aarch64-softmmu/hw/usb/tusb6010.o
  CC      aarch64-softmmu/hw/vfio/common.o
  CC      aarch64-softmmu/hw/vfio/pci.o
  CC      aarch64-softmmu/hw/vfio/pci-quirks.o
  CC      aarch64-softmmu/hw/vfio/platform.o
  CC      aarch64-softmmu/hw/vfio/calxeda-xgmac.o
  CC      aarch64-softmmu/hw/vfio/amd-xgbe.o
  CC      aarch64-softmmu/hw/vfio/spapr.o
  CC      aarch64-softmmu/hw/virtio/virtio.o
  CC      aarch64-softmmu/hw/virtio/virtio-balloon.o
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c: In function ‘create_qp_rings’:
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: implicit declaration of function ‘pow2roundup32’ [-Werror=implicit-function-declaration]
     wqe_sz = pow2roundup32(sizeof(struct pvrdma_sq_wqe_hdr) +
     ^
/var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: nested extern declaration of ‘pow2roundup32’ [-Werror=nested-externs]
  CC      aarch64-softmmu/hw/virtio/vhost.o
  CC      aarch64-softmmu/hw/virtio/vhost-backend.o
  CC      aarch64-softmmu/hw/virtio/vhost-user.o
cc1: all warnings being treated as errors
make[1]: *** [hw/rdma/vmw/pvrdma_cmd.o] Error 1
make[1]: *** Waiting for unfinished jobs....
  LINK    tricore-softmmu/qemu-system-tricore
make: *** [subdir-arm-softmmu] Error 2
make: *** [subdir-ppc64-softmmu] Error 2
  LINK    xtensaeb-softmmu/qemu-system-xtensaeb
make: *** [subdir-aarch64-softmmu] Error 2
  LINK    mipsel-linux-user/qemu-mipsel
  LINK    aarch64_be-linux-user/qemu-aarch64_be
  LINK    x86_64-linux-user/qemu-x86_64
  LINK    aarch64-linux-user/qemu-aarch64
  LINK    mips-linux-user/qemu-mips
  LINK    mips64-linux-user/qemu-mips64
  LINK    mipsn32-linux-user/qemu-mipsn32
  LINK    mipsn32el-linux-user/qemu-mipsn32el
  LINK    mips64el-linux-user/qemu-mips64el
  LINK    armeb-linux-user/qemu-armeb
  LINK    arm-linux-user/qemu-arm
  LINK    ppc64le-linux-user/qemu-ppc64le
  LINK    ppc-linux-user/qemu-ppc
  LINK    ppc64abi32-linux-user/qemu-ppc64abi32
  LINK    ppc64-linux-user/qemu-ppc64
=== OUTPUT END ===

Test command exited with code: 2


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@freelists.org

Re: [Qemu-devel] [PATCH V8 0/4] hw/pvrdma: PVRDMA device implementation

[Yuval Shaia]

On Wed, Jan 17, 2018 at 02:50:53AM -0800, no-reply@patchew.org wrote:
> Hi,
> 
> This series failed build test on ppc host. Please find the details below.
> 
> Subject: [Qemu-devel] [PATCH V8 0/4] hw/pvrdma: PVRDMA device implementation
> Type: series
> Message-id: 20180117095421.124787-1-marcel@redhat.com
>
>   CC      ppc64abi32-linux-user/target/ppc/../../libdecnumber/decNumber.o
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c: In function ‘create_qp_rings’:
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: implicit declaration of function ‘pow2roundup32’ [-Werror=implicit-function-declaration]
>      wqe_sz = pow2roundup32(sizeof(struct pvrdma_sq_wqe_hdr) +
>      ^
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: nested extern declaration of ‘pow2roundup32’ [-Werror=nested-externs]
> cc1: all warnings being treated as errors
> make[1]: *** [hw/rdma/vmw/pvrdma_cmd.o] Error 1
> make[1]: *** Waiting for unfinished jobs....
>   CC      i386-softmmu/hw/display/virtio-gpu-pci.o
>   CC      unicore32-softmmu/target/unicore32/softmmu.o
>   CC      moxie-softmmu/trace/generated-helpers.o
>   CC      aarch64-softmmu/hw/display/bcm2835_fb.o
>   CC      sh4eb-softmmu/hw/scsi/virtio-scsi.o
>   CC      sh4eb-softmmu/hw/scsi/virtio-scsi-dataplane.o
>   CC      ppc-softmmu/hw/net/fsl_etsec/miim.o
>   CC      ppc-softmmu/hw/rdma/rdma_utils.o
>   CC      x86_64-softmmu/hw/rdma/rdma_utils.o
>   CC      mips64-softmmu/hw/misc/mips_cpc.o
>   CC      ppc-softmmu/hw/rdma/rdma_backend.o
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c: In function ‘create_qp_rings’:
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: implicit declaration of function ‘pow2roundup32’ [-Werror=implicit-function-declaration]
>      wqe_sz = pow2roundup32(sizeof(struct pvrdma_sq_wqe_hdr) +
>      ^
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: nested extern declaration of ‘pow2roundup32’ [-Werror=nested-externs]

For v8 we omit the patch "pci/shpc: Move function to generic header file"
as it is already part of a pull-request made by Michael S. Tsirkin at Jan 16.

And this is the reason for the above failure.

Please ignore these build errors for now.

> cc1: all warnings being treated as errors
> make[1]: *** [hw/rdma/vmw/pvrdma_cmd.o] Error 1
> make[1]: *** Waiting for unfinished jobs....
>   CC      arm-linux-user/target/arm/crypto_helper.o
>   CC      arm-softmmu/hw/display/omap_dss.o
>   CC      arm-softmmu/hw/display/omap_lcdc.o
>   CC      ppc64-softmmu/hw/display/virtio-gpu.o
>   CC      sparc-softmmu/target/sparc/cpu.o
>   CC      mips-softmmu/hw/misc/mips_cmgcr.o
>   CC      ppc-linux-user/target/ppc/gdbstub.o
> make: *** [subdir-alpha-softmmu] Error 2
>   CC      sparc64-softmmu/hw/rdma/rdma_utils.o
>   CC      mips-softmmu/hw/misc/mips_cpc.o
>   CC      cris-softmmu/trace/generated-helpers.o
>   CC      sparc64-softmmu/hw/rdma/rdma_backend.o
>   CC      ppc64-linux-user/target/ppc/gdbstub.o
>   CC      mips64-softmmu/hw/misc/mips_itu.o
>   CC      microblaze-softmmu/trace/generated-helpers.o
>   CC      ppc64-linux-user/target/ppc/../../libdecnumber/decContext.o
>   CC      sparc-softmmu/target/sparc/fop_helper.o
>   CC      ppc64-softmmu/hw/display/virtio-gpu-3d.o
>   GEN     trace/generated-helpers.c
>   CC      sparc-softmmu/target/sparc/cc_helper.o
>   CC      s390x-softmmu/hw/rdma/rdma_backend.o
>   CC      sparc-softmmu/target/sparc/win_helper.o
>   CC      aarch64-softmmu/hw/display/vga.o
>   CC      ppc64abi32-linux-user/target/ppc/../../libdecnumber/dpd/decimal32.o
>   CC      ppc-softmmu/hw/rdma/rdma_rm.o
>   CC      aarch64-softmmu/hw/display/virtio-gpu.o
>   CC      mips-softmmu/hw/misc/mips_itu.o
>   CC      ppc64abi32-linux-user/target/ppc/../../libdecnumber/dpd/decimal64.o
>   GEN     trace/generated-helpers.c
>   CC      aarch64_be-linux-user/trace/generated-helpers.o
>   CC      mips64el-softmmu/hw/rdma/rdma_utils.o
>   CC      ppc64abi32-linux-user/target/ppc/../../libdecnumber/dpd/decimal128.o
>   CC      mips64-softmmu/hw/misc/mmio_interface.o
>   CC      ppc64-linux-user/target/ppc/../../libdecnumber/decNumber.o
>   CC      xtensaeb-softmmu/target/xtensa/monitor.o
>   CC      ppcemb-softmmu/hw/rdma/rdma_backend.o
>   CC      ppc-softmmu/hw/rdma/vmw/pvrdma_dev_ring.o
>   CC      m68k-softmmu/gdbstub-xml.o
>   CC      ppc-linux-user/target/ppc/../../libdecnumber/decContext.o
>   CC      sh4-softmmu/hw/rdma/rdma_rm.o
>   LINK    moxie-softmmu/qemu-system-moxie
>   CC      ppc-softmmu/hw/rdma/vmw/pvrdma_cmd.o
>   CC      i386-softmmu/hw/display/virtio-vga.o
>   CC      arm-linux-user/trace/control-target.o
>   CC      mips-softmmu/hw/misc/mmio_interface.o
>   GEN     trace/generated-helpers.c
>   CC      ppc-softmmu/hw/rdma/vmw/pvrdma_qp_ops.o
>   CC      sparc-softmmu/target/sparc/mmu_helper.o
>   CC      unicore32-softmmu/trace/control-target.o
>   CC      i386-softmmu/hw/intc/apic.o
>   CC      ppc64abi32-linux-user/trace/control-target.o
>   CC      ppc-softmmu/hw/rdma/vmw/pvrdma_main.o
>   CC      mips64el-softmmu/hw/rdma/rdma_backend.o
>   CC      ppc-softmmu/hw/scsi/virtio-scsi.o
>   CC      ppc-softmmu/hw/scsi/virtio-scsi-dataplane.o
>   CC      sh4-softmmu/hw/rdma/vmw/pvrdma_dev_ring.o
>   CC      arm-softmmu/hw/display/pxa2xx_lcd.o
>   CC      ppc64-linux-user/target/ppc/../../libdecnumber/dpd/decimal32.o
>   CC      ppc-softmmu/hw/scsi/vhost-scsi-common.o
>   CC      arm-softmmu/hw/display/bcm2835_fb.o
>   LINK    microblaze-softmmu/qemu-system-microblaze
>   CC      i386-softmmu/hw/intc/apic_common.o
>   CC      ppc64-softmmu/hw/display/virtio-gpu-pci.o
>   CC      mips-softmmu/hw/net/virtio-net.o
> make: *** [subdir-sh4eb-softmmu] Error 2
>   CC      mips-softmmu/hw/net/vhost_net.o
>   CC      ppc64-softmmu/hw/display/virtio-vga.o
>   CC      mips64-softmmu/hw/net/virtio-net.o
>   CC      sparc-softmmu/target/sparc/ldst_helper.o
>   CC      mips-softmmu/hw/rdma/rdma_utils.o
>   CC      ppc-linux-user/target/ppc/../../libdecnumber/decNumber.o
>   CC      s390x-softmmu/hw/rdma/rdma_rm.o
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c: In function ‘create_qp_rings’:
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: implicit declaration of function ‘pow2roundup32’ [-Werror=implicit-function-declaration]
>      wqe_sz = pow2roundup32(sizeof(struct pvrdma_sq_wqe_hdr) +
>      ^
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: nested extern declaration of ‘pow2roundup32’ [-Werror=nested-externs]
> cc1: all warnings being treated as errors
> make[1]: *** [hw/rdma/vmw/pvrdma_cmd.o] Error 1
> make[1]: *** Waiting for unfinished jobs....
>   CC      x86_64-softmmu/hw/rdma/rdma_backend.o
>   CC      x86_64-softmmu/hw/rdma/rdma_rm.o
>   CC      mips64el-softmmu/hw/rdma/rdma_rm.o
>   CC      mips64-softmmu/hw/net/vhost_net.o
>   CC      i386-softmmu/hw/intc/ioapic.o
>   CC      ppc64-softmmu/hw/i2c/ppc4xx_i2c.o
>   CC      arm-linux-user/gdbstub-xml.o
>   CC      sparc64-softmmu/hw/rdma/rdma_rm.o
>   CC      sparc64-softmmu/hw/rdma/vmw/pvrdma_dev_ring.o
>   CC      i386-softmmu/hw/isa/lpc_ich9.o
>   CC      arm-softmmu/hw/display/vga.o
>   CC      ppc64-softmmu/hw/intc/openpic_kvm.o
>   CC      i386-softmmu/hw/misc/ivshmem.o
>   CC      unicore32-softmmu/trace/generated-helpers.o
>   CC      mips64el-softmmu/hw/rdma/vmw/pvrdma_dev_ring.o
>   CC      ppc64-softmmu/hw/intc/xics.o
>   CC      ppc64-linux-user/target/ppc/../../libdecnumber/dpd/decimal64.o
>   CC      ppc64-linux-user/target/ppc/../../libdecnumber/dpd/decimal128.o
>   CC      sparc-softmmu/target/sparc/int32_helper.o
>   CC      i386-softmmu/hw/misc/pvpanic.o
>   CC      ppcemb-softmmu/hw/rdma/rdma_rm.o
>   CC      sparc64-softmmu/hw/rdma/vmw/pvrdma_cmd.o
>   CC      arm-softmmu/hw/display/virtio-gpu.o
>   CC      i386-softmmu/hw/misc/mmio_interface.o
>   CC      x86_64-softmmu/hw/rdma/vmw/pvrdma_dev_ring.o
>   CC      sh4-softmmu/hw/rdma/vmw/pvrdma_cmd.o
>   CC      ppc-linux-user/target/ppc/../../libdecnumber/dpd/decimal32.o
>   CC      i386-softmmu/hw/net/virtio-net.o
>   CC      mips-softmmu/hw/rdma/rdma_backend.o
>   CC      mips64-softmmu/hw/rdma/rdma_utils.o
>   CC      xtensa-softmmu/trace/generated-helpers.o
>   CC      sparc64-softmmu/hw/rdma/vmw/pvrdma_qp_ops.o
>   CC      ppc64abi32-linux-user/gdbstub-xml.o
>   CC      ppcemb-softmmu/hw/rdma/vmw/pvrdma_dev_ring.o
>   CC      mips64-softmmu/hw/rdma/rdma_backend.o
>   CC      mipsn32el-linux-user/trace/generated-helpers.o
>   CC      sparc64-softmmu/hw/rdma/vmw/pvrdma_main.o
>   CC      xtensaeb-softmmu/target/xtensa/xtensa-isa.o
>   CC      ppc64-softmmu/hw/intc/xics_spapr.o
>   CC      ppc64-softmmu/hw/intc/xics_kvm.o
>   CC      mips-softmmu/hw/rdma/rdma_rm.o
>   CC      ppc-linux-user/target/ppc/../../libdecnumber/dpd/decimal64.o
>   CC      ppc64-softmmu/hw/intc/xics_pnv.o
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c: In function ‘create_qp_rings’:
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: implicit declaration of function ‘pow2roundup32’ [-Werror=implicit-function-declaration]
>      wqe_sz = pow2roundup32(sizeof(struct pvrdma_sq_wqe_hdr) +
>      ^
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: nested extern declaration of ‘pow2roundup32’ [-Werror=nested-externs]
>   CC      mips64el-softmmu/hw/rdma/vmw/pvrdma_cmd.o
>   CC      s390x-softmmu/hw/rdma/vmw/pvrdma_dev_ring.o
> cc1: all warnings being treated as errors
> make[1]: *** [hw/rdma/vmw/pvrdma_cmd.o] Error 1
> make[1]: *** Waiting for unfinished jobs....
>   CC      s390x-softmmu/hw/rdma/vmw/pvrdma_cmd.o
>   CC      xtensaeb-softmmu/target/xtensa/translate.o
>   LINK    unicore32-softmmu/qemu-system-unicore32
>   CC      mips64-softmmu/hw/rdma/rdma_rm.o
>   CC      mips64el-softmmu/hw/rdma/vmw/pvrdma_qp_ops.o
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c: In function ‘create_qp_rings’:
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: implicit declaration of function ‘pow2roundup32’ [-Werror=implicit-function-declaration]
>      wqe_sz = pow2roundup32(sizeof(struct pvrdma_sq_wqe_hdr) +
>      ^
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: nested extern declaration of ‘pow2roundup32’ [-Werror=nested-externs]
>   CC      x86_64-softmmu/hw/rdma/vmw/pvrdma_cmd.o
> cc1: all warnings being treated as errors
>   CC      mips64-softmmu/hw/rdma/vmw/pvrdma_dev_ring.o
> make[1]: *** [hw/rdma/vmw/pvrdma_cmd.o] Error 1
>   CC      s390x-softmmu/hw/rdma/vmw/pvrdma_qp_ops.o
> make: *** [subdir-sh4-softmmu] Error 2
>   CC      m68k-softmmu/trace/generated-helpers.o
>   GEN     trace/generated-helpers.c
>   CC      arm-linux-user/trace/generated-helpers.o
>   CC      arm-softmmu/hw/display/virtio-gpu-3d.o
>   CC      mips-softmmu/hw/rdma/vmw/pvrdma_dev_ring.o
>   CC      ppc64-linux-user/trace/control-target.o
>   CC      xtensaeb-softmmu/target/xtensa/op_helper.o
>   CC      mips64-softmmu/hw/rdma/vmw/pvrdma_cmd.o
>   CC      xtensaeb-softmmu/target/xtensa/helper.o
>   CC      x86_64-linux-user/trace/generated-helpers.o
>   CC      xtensaeb-softmmu/target/xtensa/cpu.o
>   CC      xtensaeb-softmmu/target/xtensa/gdbstub.o
>   CC      mips64el-softmmu/hw/rdma/vmw/pvrdma_main.o
>   GEN     trace/generated-helpers.c
>   CC      ppc64abi32-linux-user/trace/generated-helpers.o
>   CC      mips-softmmu/hw/rdma/vmw/pvrdma_cmd.o
>   CC      sparc-softmmu/target/sparc/gdbstub.o
>   LINK    xtensa-softmmu/qemu-system-xtensa
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c: In function ‘create_qp_rings’:
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: implicit declaration of function ‘pow2roundup32’ [-Werror=implicit-function-declaration]
>      wqe_sz = pow2roundup32(sizeof(struct pvrdma_sq_wqe_hdr) +
>      ^
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: nested extern declaration of ‘pow2roundup32’ [-Werror=nested-externs]
> cc1: all warnings being treated as errors
> make[1]: *** [hw/rdma/vmw/pvrdma_cmd.o] Error 1
> make[1]: *** Waiting for unfinished jobs....
>   GEN     trace/generated-helpers.c
> make: *** [subdir-ppc-softmmu] Error 2
>   CC      ppc-linux-user/target/ppc/../../libdecnumber/dpd/decimal128.o
>   LINK    cris-softmmu/qemu-system-cris
>   CC      ppc64-softmmu/hw/misc/ivshmem.o
>   CC      x86_64-softmmu/hw/rdma/vmw/pvrdma_qp_ops.o
>   CC      xtensaeb-softmmu/trace/control-target.o
>   CC      mips-softmmu/hw/rdma/vmw/pvrdma_qp_ops.o
>   CC      aarch64-softmmu/hw/display/virtio-gpu-3d.o
>   CC      mips-softmmu/hw/rdma/vmw/pvrdma_main.o
>   CC      mips-softmmu/hw/scsi/virtio-scsi.o
>   CC      ppc64-softmmu/hw/misc/mmio_interface.o
>   CC      mips-softmmu/hw/scsi/virtio-scsi-dataplane.o
>   CC      ppcemb-softmmu/hw/rdma/vmw/pvrdma_cmd.o
>   CC      aarch64-softmmu/hw/display/virtio-gpu-pci.o
>   GEN     trace/generated-helpers.c
>   CC      ppc-linux-user/trace/control-target.o
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c: In function ‘create_qp_rings’:
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: implicit declaration of function ‘pow2roundup32’ [-Werror=implicit-function-declaration]
>      wqe_sz = pow2roundup32(sizeof(struct pvrdma_sq_wqe_hdr) +
>      ^
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: nested extern declaration of ‘pow2roundup32’ [-Werror=nested-externs]
>   CC      arm-softmmu/hw/display/virtio-gpu-pci.o
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c: In function ‘create_qp_rings’:
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: implicit declaration of function ‘pow2roundup32’ [-Werror=implicit-function-declaration]
>      wqe_sz = pow2roundup32(sizeof(struct pvrdma_sq_wqe_hdr) +
>      ^
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: nested extern declaration of ‘pow2roundup32’ [-Werror=nested-externs]
> cc1: all warnings being treated as errors
>   CC      aarch64-softmmu/hw/display/dpcd.o
> make[1]: *** [hw/rdma/vmw/pvrdma_cmd.o] Error 1
> make[1]: *** Waiting for unfinished jobs....
>   CC      mips64-softmmu/hw/rdma/vmw/pvrdma_qp_ops.o
> cc1: all warnings being treated as errors
> make[1]: *** [hw/rdma/vmw/pvrdma_cmd.o] Error 1
> make[1]: *** Waiting for unfinished jobs....
>   CC      aarch64-softmmu/hw/display/xlnx_dp.o
>   CC      ppc64-softmmu/hw/net/spapr_llan.o
>   CC      arm-softmmu/hw/dma/omap_dma.o
>   CC      ppc-linux-user/gdbstub-xml.o
> make: *** [subdir-sparc64-softmmu] Error 2
>   CC      mips-softmmu/hw/scsi/vhost-scsi.o
>   CC      mips-softmmu/hw/scsi/vhost-scsi-common.o
>   CC      aarch64-softmmu/hw/dma/xlnx_dpdma.o
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c: In function ‘create_qp_rings’:
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: implicit declaration of function ‘pow2roundup32’ [-Werror=implicit-function-declaration]
>      wqe_sz = pow2roundup32(sizeof(struct pvrdma_sq_wqe_hdr) +
>      ^
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: nested extern declaration of ‘pow2roundup32’ [-Werror=nested-externs]
>   CC      ppc64-softmmu/hw/net/xilinx_ethlite.o
>   CC      arm-softmmu/hw/dma/soc_dma.o
>   CC      sparc-softmmu/trace/control-target.o
> cc1: all warnings being treated as errors
> make[1]: *** [hw/rdma/vmw/pvrdma_cmd.o] Error 1
> make[1]: *** Waiting for unfinished jobs....
>   CC      i386-softmmu/hw/net/vhost_net.o
>   CC      ppc-linux-user/trace/generated-helpers.o
>   CC      ppc64-softmmu/hw/net/virtio-net.o
>   CC      i386-softmmu/hw/rdma/rdma_utils.o
>   CC      arm-softmmu/hw/dma/pxa2xx_dma.o
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c: In function ‘create_qp_rings’:
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: implicit declaration of function ‘pow2roundup32’ [-Werror=implicit-function-declaration]
>      wqe_sz = pow2roundup32(sizeof(struct pvrdma_sq_wqe_hdr) +
>      ^
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: nested extern declaration of ‘pow2roundup32’ [-Werror=nested-externs]
>   CC      ppc64-linux-user/gdbstub-xml.o
>   CC      mips-softmmu/hw/scsi/vhost-user-scsi.o
> cc1: all warnings being treated as errors
> make: *** [subdir-s390x-softmmu] Error 2
> make[1]: *** [hw/rdma/vmw/pvrdma_cmd.o] Error 1
> make[1]: *** Waiting for unfinished jobs....
>   CC      i386-softmmu/hw/rdma/rdma_backend.o
>   CC      mips-softmmu/hw/timer/mips_gictimer.o
>   CC      ppcemb-softmmu/hw/rdma/vmw/pvrdma_qp_ops.o
>   CC      xtensaeb-softmmu/trace/generated-helpers.o
>   CC      ppc64-softmmu/hw/net/vhost_net.o
>   CC      aarch64-softmmu/hw/dma/omap_dma.o
>   CC      i386-softmmu/hw/rdma/rdma_rm.o
>   CC      sparc-softmmu/trace/generated-helpers.o
>   CC      arm-softmmu/hw/dma/bcm2835_dma.o
>   CC      aarch64-softmmu/hw/dma/soc_dma.o
>   CC      ppc64-linux-user/trace/generated-helpers.o
>   CC      ppcemb-softmmu/hw/rdma/vmw/pvrdma_main.o
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c: In function ‘create_qp_rings’:
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: implicit declaration of function ‘pow2roundup32’ [-Werror=implicit-function-declaration]
>      wqe_sz = pow2roundup32(sizeof(struct pvrdma_sq_wqe_hdr) +
>      ^
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: nested extern declaration of ‘pow2roundup32’ [-Werror=nested-externs]
>   CC      ppcemb-softmmu/hw/scsi/virtio-scsi.o
> cc1: all warnings being treated as errors
> make[1]: *** [hw/rdma/vmw/pvrdma_cmd.o] Error 1
> make[1]: *** Waiting for unfinished jobs....
>   CC      ppc64-softmmu/hw/net/fsl_etsec/etsec.o
>   CC      ppc64-softmmu/hw/net/fsl_etsec/registers.o
>   CC      arm-softmmu/hw/gpio/omap_gpio.o
> make: *** [subdir-x86_64-softmmu] Error 2
>   CC      i386-softmmu/hw/rdma/vmw/pvrdma_dev_ring.o
>   CC      i386-softmmu/hw/rdma/vmw/pvrdma_cmd.o
>   CC      ppc64-softmmu/hw/net/fsl_etsec/rings.o
>   CC      aarch64-softmmu/hw/dma/pxa2xx_dma.o
>   CC      ppc64-softmmu/hw/net/fsl_etsec/miim.o
>   CC      ppc64-softmmu/hw/nvram/spapr_nvram.o
>   CC      arm-softmmu/hw/gpio/imx_gpio.o
> make: *** [subdir-mips64el-softmmu] Error 2
>   CC      i386-softmmu/hw/rdma/vmw/pvrdma_qp_ops.o
>   CC      ppc64-softmmu/hw/rdma/rdma_utils.o
>   CC      arm-softmmu/hw/gpio/bcm2835_gpio.o
>   CC      ppc64-softmmu/hw/rdma/rdma_backend.o
>   CC      ppc64-softmmu/hw/rdma/rdma_rm.o
>   CC      aarch64-softmmu/hw/dma/bcm2835_dma.o
>   CC      aarch64-softmmu/hw/gpio/omap_gpio.o
>   CC      i386-softmmu/hw/rdma/vmw/pvrdma_main.o
>   CC      ppc64-softmmu/hw/rdma/vmw/pvrdma_dev_ring.o
>   CC      arm-softmmu/hw/i2c/omap_i2c.o
>   CC      arm-softmmu/hw/input/pxa2xx_keypad.o
>   CC      aarch64-softmmu/hw/gpio/imx_gpio.o
>   CC      i386-softmmu/hw/scsi/virtio-scsi.o
>   CC      i386-softmmu/hw/scsi/virtio-scsi-dataplane.o
>   CC      i386-softmmu/hw/scsi/vhost-scsi-common.o
>   CC      arm-softmmu/hw/input/tsc210x.o
>   CC      aarch64-softmmu/hw/gpio/bcm2835_gpio.o
>   CC      ppc64-softmmu/hw/rdma/vmw/pvrdma_cmd.o
>   CC      ppc64-softmmu/hw/rdma/vmw/pvrdma_qp_ops.o
>   CC      aarch64-softmmu/hw/i2c/omap_i2c.o
>   CC      i386-softmmu/hw/scsi/vhost-scsi.o
>   CC      i386-softmmu/hw/scsi/vhost-user-scsi.o
>   CC      aarch64-softmmu/hw/input/pxa2xx_keypad.o
>   CC      arm-softmmu/hw/intc/armv7m_nvic.o
>   CC      ppc64-softmmu/hw/rdma/vmw/pvrdma_main.o
>   CC      ppc64-softmmu/hw/scsi/spapr_vscsi.o
> make: *** [subdir-mips64-softmmu] Error 2
>   CC      ppc64-softmmu/hw/scsi/virtio-scsi.o
>   CC      i386-softmmu/hw/timer/mc146818rtc.o
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c: In function ‘create_qp_rings’:
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: implicit declaration of function ‘pow2roundup32’ [-Werror=implicit-function-declaration]
>      wqe_sz = pow2roundup32(sizeof(struct pvrdma_sq_wqe_hdr) +
>      ^
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: nested extern declaration of ‘pow2roundup32’ [-Werror=nested-externs]
>   CC      ppc64-softmmu/hw/scsi/virtio-scsi-dataplane.o
> cc1: all warnings being treated as errors
>   CC      ppc64-softmmu/hw/scsi/vhost-scsi-common.o
> make[1]: *** [hw/rdma/vmw/pvrdma_cmd.o] Error 1
> make[1]: *** Waiting for unfinished jobs....
>   CC      aarch64-softmmu/hw/input/tsc210x.o
>   CC      ppc64-softmmu/hw/scsi/vhost-scsi.o
>   CC      arm-softmmu/hw/intc/exynos4210_gic.o
>   CC      aarch64-softmmu/hw/intc/armv7m_nvic.o
>   CC      aarch64-softmmu/hw/intc/exynos4210_gic.o
>   CC      aarch64-softmmu/hw/intc/exynos4210_combiner.o
>   CC      arm-softmmu/hw/intc/exynos4210_combiner.o
>   CC      ppc64-softmmu/hw/scsi/vhost-user-scsi.o
>   CC      ppc64-softmmu/hw/timer/mc146818rtc.o
>   CC      ppc64-softmmu/hw/vfio/common.o
>   CC      arm-softmmu/hw/intc/omap_intc.o
>   LINK    sparc-softmmu/qemu-system-sparc
>   CC      ppc64-softmmu/hw/vfio/pci.o
>   CC      aarch64-softmmu/hw/intc/omap_intc.o
>   CC      aarch64-softmmu/hw/intc/bcm2835_ic.o
>   CC      arm-softmmu/hw/intc/bcm2835_ic.o
>   CC      ppc64-softmmu/hw/vfio/pci-quirks.o
>   CC      aarch64-softmmu/hw/intc/bcm2836_control.o
>   CC      ppc64-softmmu/hw/vfio/platform.o
>   CC      aarch64-softmmu/hw/intc/allwinner-a10-pic.o
>   CC      arm-softmmu/hw/intc/bcm2836_control.o
>   CC      ppc64-softmmu/hw/vfio/spapr.o
>   CC      arm-softmmu/hw/intc/allwinner-a10-pic.o
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c: In function ‘create_qp_rings’:
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: implicit declaration of function ‘pow2roundup32’ [-Werror=implicit-function-declaration]
>      wqe_sz = pow2roundup32(sizeof(struct pvrdma_sq_wqe_hdr) +
>      ^
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: nested extern declaration of ‘pow2roundup32’ [-Werror=nested-externs]
>   CC      aarch64-softmmu/hw/intc/aspeed_vic.o
> cc1: all warnings being treated as errors
> make[1]: *** [hw/rdma/vmw/pvrdma_cmd.o] Error 1
> make[1]: *** Waiting for unfinished jobs....
>   CC      aarch64-softmmu/hw/intc/arm_gicv3_cpuif.o
>   CC      arm-softmmu/hw/intc/aspeed_vic.o
>   CC      arm-softmmu/hw/intc/arm_gicv3_cpuif.o
>   CC      aarch64-softmmu/hw/misc/ivshmem.o
>   CC      arm-softmmu/hw/misc/ivshmem.o
>   CC      arm-softmmu/hw/misc/arm_sysctl.o
>   CC      arm-softmmu/hw/misc/cbus.o
>   CC      arm-softmmu/hw/misc/exynos4210_pmu.o
>   CC      arm-softmmu/hw/misc/exynos4210_clk.o
>   CC      aarch64-softmmu/hw/misc/arm_sysctl.o
>   CC      aarch64-softmmu/hw/misc/cbus.o
>   CC      aarch64-softmmu/hw/misc/exynos4210_pmu.o
>   CC      arm-softmmu/hw/misc/exynos4210_rng.o
>   CC      arm-softmmu/hw/misc/imx_ccm.o
>   CC      aarch64-softmmu/hw/misc/exynos4210_clk.o
>   CC      arm-softmmu/hw/misc/imx31_ccm.o
>   CC      arm-softmmu/hw/misc/imx25_ccm.o
>   CC      arm-softmmu/hw/misc/imx6_ccm.o
>   LINK    s390x-linux-user/qemu-s390x
> make: *** [subdir-mips-softmmu] Error 2
>   CC      arm-softmmu/hw/misc/imx6_src.o
>   CC      arm-softmmu/hw/misc/mst_fpga.o
>   CC      aarch64-softmmu/hw/misc/exynos4210_rng.o
>   CC      arm-softmmu/hw/misc/omap_clk.o
>   CC      aarch64-softmmu/hw/misc/imx_ccm.o
>   CC      aarch64-softmmu/hw/misc/imx31_ccm.o
>   CC      arm-softmmu/hw/misc/omap_gpmc.o
>   CC      arm-softmmu/hw/misc/omap_l4.o
>   CC      aarch64-softmmu/hw/misc/imx25_ccm.o
>   CC      aarch64-softmmu/hw/misc/imx6_ccm.o
>   CC      aarch64-softmmu/hw/misc/imx6_src.o
>   CC      aarch64-softmmu/hw/misc/mst_fpga.o
>   CC      arm-softmmu/hw/misc/omap_sdrc.o
>   CC      aarch64-softmmu/hw/misc/omap_clk.o
>   CC      arm-softmmu/hw/misc/omap_tap.o
>   CC      arm-softmmu/hw/misc/bcm2835_property.o
>   CC      arm-softmmu/hw/misc/bcm2835_mbox.o
>   CC      aarch64-softmmu/hw/misc/omap_gpmc.o
>   CC      arm-softmmu/hw/misc/bcm2835_rng.o
>   CC      aarch64-softmmu/hw/misc/omap_l4.o
> make: *** [subdir-ppcemb-softmmu] Error 2
>   CC      arm-softmmu/hw/misc/zynq_slcr.o
>   CC      aarch64-softmmu/hw/misc/omap_sdrc.o
>   CC      arm-softmmu/hw/misc/zynq-xadc.o
>   CC      arm-softmmu/hw/misc/mps2-scc.o
>   CC      arm-softmmu/hw/misc/stm32f2xx_syscfg.o
>   CC      aarch64-softmmu/hw/misc/omap_tap.o
>   CC      aarch64-softmmu/hw/misc/bcm2835_mbox.o
>   CC      arm-softmmu/hw/misc/aspeed_scu.o
>   CC      arm-softmmu/hw/misc/aspeed_sdmc.o
>   CC      aarch64-softmmu/hw/misc/bcm2835_property.o
>   CC      aarch64-softmmu/hw/misc/bcm2835_rng.o
>   CC      arm-softmmu/hw/misc/mmio_interface.o
>   CC      arm-softmmu/hw/misc/msf2-sysreg.o
>   CC      aarch64-softmmu/hw/misc/zynq_slcr.o
>   CC      arm-softmmu/hw/net/virtio-net.o
>   CC      arm-softmmu/hw/net/vhost_net.o
>   CC      arm-softmmu/hw/pcmcia/pxa2xx.o
>   CC      aarch64-softmmu/hw/misc/zynq-xadc.o
>   CC      arm-softmmu/hw/rdma/rdma_utils.o
>   LINK    m68k-softmmu/qemu-system-m68k
>   CC      arm-softmmu/hw/rdma/rdma_backend.o
>   CC      arm-softmmu/hw/rdma/rdma_rm.o
>   CC      aarch64-softmmu/hw/misc/stm32f2xx_syscfg.o
>   CC      aarch64-softmmu/hw/misc/mps2-scc.o
>   CC      arm-softmmu/hw/rdma/vmw/pvrdma_dev_ring.o
>   CC      aarch64-softmmu/hw/misc/auxbus.o
>   CC      aarch64-softmmu/hw/misc/aspeed_scu.o
>   CC      aarch64-softmmu/hw/misc/aspeed_sdmc.o
>   CC      arm-softmmu/hw/rdma/vmw/pvrdma_cmd.o
>   CC      arm-softmmu/hw/rdma/vmw/pvrdma_qp_ops.o
>   CC      arm-softmmu/hw/rdma/vmw/pvrdma_main.o
>   CC      arm-softmmu/hw/scsi/virtio-scsi.o
>   CC      aarch64-softmmu/hw/misc/mmio_interface.o
>   CC      aarch64-softmmu/hw/misc/msf2-sysreg.o
>   CC      aarch64-softmmu/hw/net/virtio-net.o
>   CC      arm-softmmu/hw/scsi/virtio-scsi-dataplane.o
>   CC      aarch64-softmmu/hw/net/vhost_net.o
>   CC      arm-softmmu/hw/scsi/vhost-scsi-common.o
>   CC      aarch64-softmmu/hw/pcmcia/pxa2xx.o
>   CC      aarch64-softmmu/hw/rdma/rdma_utils.o
>   CC      arm-softmmu/hw/scsi/vhost-scsi.o
>   CC      aarch64-softmmu/hw/rdma/rdma_backend.o
>   CC      arm-softmmu/hw/scsi/vhost-user-scsi.o
>   CC      aarch64-softmmu/hw/rdma/rdma_rm.o
>   CC      aarch64-softmmu/hw/rdma/vmw/pvrdma_dev_ring.o
>   CC      arm-softmmu/hw/sd/omap_mmc.o
>   CC      aarch64-softmmu/hw/rdma/vmw/pvrdma_cmd.o
>   CC      aarch64-softmmu/hw/rdma/vmw/pvrdma_qp_ops.o
>   CC      arm-softmmu/hw/sd/pxa2xx_mmci.o
>   CC      aarch64-softmmu/hw/rdma/vmw/pvrdma_main.o
> make: *** [subdir-i386-softmmu] Error 2
>   CC      aarch64-softmmu/hw/scsi/virtio-scsi-dataplane.o
>   CC      aarch64-softmmu/hw/scsi/virtio-scsi.o
>   CC      arm-softmmu/hw/sd/bcm2835_sdhost.o
>   CC      aarch64-softmmu/hw/scsi/vhost-scsi-common.o
>   CC      aarch64-softmmu/hw/scsi/vhost-scsi.o
>   CC      aarch64-softmmu/hw/scsi/vhost-user-scsi.o
>   CC      arm-softmmu/hw/ssi/omap_spi.o
>   CC      aarch64-softmmu/hw/sd/omap_mmc.o
>   CC      aarch64-softmmu/hw/sd/pxa2xx_mmci.o
>   CC      arm-softmmu/hw/ssi/imx_spi.o
>   CC      arm-softmmu/hw/timer/exynos4210_mct.o
>   CC      arm-softmmu/hw/timer/exynos4210_pwm.o
>   CC      arm-softmmu/hw/timer/exynos4210_rtc.o
>   CC      arm-softmmu/hw/timer/omap_gptimer.o
>   CC      aarch64-softmmu/hw/sd/bcm2835_sdhost.o
>   CC      arm-softmmu/hw/timer/omap_synctimer.o
>   CC      arm-softmmu/hw/timer/pxa2xx_timer.o
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c: In function ‘create_qp_rings’:
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: implicit declaration of function ‘pow2roundup32’ [-Werror=implicit-function-declaration]
>      wqe_sz = pow2roundup32(sizeof(struct pvrdma_sq_wqe_hdr) +
>      ^
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: nested extern declaration of ‘pow2roundup32’ [-Werror=nested-externs]
>   CC      arm-softmmu/hw/timer/digic-timer.o
> cc1: all warnings being treated as errors
> make[1]: *** [hw/rdma/vmw/pvrdma_cmd.o] Error 1
> make[1]: *** Waiting for unfinished jobs....
>   CC      aarch64-softmmu/hw/ssi/omap_spi.o
>   CC      aarch64-softmmu/hw/ssi/imx_spi.o
>   CC      aarch64-softmmu/hw/timer/exynos4210_mct.o
>   CC      aarch64-softmmu/hw/timer/exynos4210_pwm.o
>   CC      aarch64-softmmu/hw/timer/exynos4210_rtc.o
>   CC      aarch64-softmmu/hw/timer/omap_gptimer.o
>   CC      aarch64-softmmu/hw/timer/omap_synctimer.o
>   CC      aarch64-softmmu/hw/timer/pxa2xx_timer.o
>   CC      aarch64-softmmu/hw/timer/digic-timer.o
>   CC      aarch64-softmmu/hw/timer/allwinner-a10-pit.o
>   CC      aarch64-softmmu/hw/usb/tusb6010.o
>   CC      aarch64-softmmu/hw/vfio/common.o
>   CC      aarch64-softmmu/hw/vfio/pci.o
>   CC      aarch64-softmmu/hw/vfio/pci-quirks.o
>   CC      aarch64-softmmu/hw/vfio/platform.o
>   CC      aarch64-softmmu/hw/vfio/calxeda-xgmac.o
>   CC      aarch64-softmmu/hw/vfio/amd-xgbe.o
>   CC      aarch64-softmmu/hw/vfio/spapr.o
>   CC      aarch64-softmmu/hw/virtio/virtio.o
>   CC      aarch64-softmmu/hw/virtio/virtio-balloon.o
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c: In function ‘create_qp_rings’:
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: implicit declaration of function ‘pow2roundup32’ [-Werror=implicit-function-declaration]
>      wqe_sz = pow2roundup32(sizeof(struct pvrdma_sq_wqe_hdr) +
>      ^
> /var/tmp/patchew-tester-tmp-0kkotmms/src/hw/rdma/vmw/pvrdma_cmd.c:420:5: error: nested extern declaration of ‘pow2roundup32’ [-Werror=nested-externs]
>   CC      aarch64-softmmu/hw/virtio/vhost.o
>   CC      aarch64-softmmu/hw/virtio/vhost-backend.o
>   CC      aarch64-softmmu/hw/virtio/vhost-user.o
> cc1: all warnings being treated as errors
> make[1]: *** [hw/rdma/vmw/pvrdma_cmd.o] Error 1
> make[1]: *** Waiting for unfinished jobs....
>   LINK    tricore-softmmu/qemu-system-tricore
> make: *** [subdir-arm-softmmu] Error 2
> make: *** [subdir-ppc64-softmmu] Error 2
>   LINK    xtensaeb-softmmu/qemu-system-xtensaeb
> make: *** [subdir-aarch64-softmmu] Error 2
>   LINK    mipsel-linux-user/qemu-mipsel
>   LINK    aarch64_be-linux-user/qemu-aarch64_be
>   LINK    x86_64-linux-user/qemu-x86_64
>   LINK    aarch64-linux-user/qemu-aarch64
>   LINK    mips-linux-user/qemu-mips
>   LINK    mips64-linux-user/qemu-mips64
>   LINK    mipsn32-linux-user/qemu-mipsn32
>   LINK    mipsn32el-linux-user/qemu-mipsn32el
>   LINK    mips64el-linux-user/qemu-mips64el
>   LINK    armeb-linux-user/qemu-armeb
>   LINK    arm-linux-user/qemu-arm
>   LINK    ppc64le-linux-user/qemu-ppc64le
>   LINK    ppc-linux-user/qemu-ppc
>   LINK    ppc64abi32-linux-user/qemu-ppc64abi32
>   LINK    ppc64-linux-user/qemu-ppc64
> === OUTPUT END ===
> 
> Test command exited with code: 2
> 
> 
> ---
> Email generated automatically by Patchew [http://patchew.org/].
> Please send your feedback to patchew-devel@freelists.org