[PATCH v1 00/15] Add Paravirtual RDMA Driver

Adit Ranadive
2016-07-06T06:14:33Z

Hi Doug, others,

This patch series adds a driver for a paravirtual RDMA device. The device
is developed for VMware's Virtual Machines and allows existing RDMA
applications to continue to use existing Verbs API when deployed in VMs on
ESX. We recently did a presentation in the OFA Workshop regarding this
device. See - https://goo.gl/pHOXJ8 for our workshop presentation. Note,
that this patch series should be applied all together. I split out the
commits so that it may be easier to review.

Description and RDMA Support
-------------------------------
The virtual device is exposed as a dual function PCIe device. One part is
a virtual network device (VMXNet3) which provides networking properties
like MAC, IP addresses to the RDMA part of the device. The networking
properties are used to register GIDs required by RDMA applications to
communicate.

These patches add support and the all required infrastructure for letting
applications use such a device. We support the mandatory Verbs API as well
as the base memory management extensions (Local Inv, Send with Inv and Fast
Register Work Requests). We currently support both Reliable Connected and
Unreliable Datagram QPs but do not support Shared Receive Queues (SRQs).
Also, we support the following types of Work Requests:
o Send/Receive (with or without Immediate Data)
o RDMA Write (with or without Immediate Data)
o RDMA Read
o Local Invalidate
o Send with Invalidate
o Fast Register Work Requests

This version only adds support for version 1 of RoCE. We will add RoCEv2
support in a future patch. We do support registration of both MAC-based and
IP-based GIDs.

I have also created a git tree for our user-level driver here:
http://git.openfabrics.org/?p=~aditr/libpvrdma.git;a=summary

Testing
--------
We have tested this internally for various types of Guest OS - Red Hat,
Centos, Ubuntu 12.04/14.04/16.04, Oracle Enterprise Linux, SLES 12
using backported versions of this driver. The tests included several runs
of the performance tests (included with OFED), Intel MPI PingPong benchmark
on OpenMPI, krping for FRWRs. Mellanox has been kind enough to test the
backported version of the driver internally on their hardware using a
VMware provided ESX build. I have also applied and tested this with Doug's
master, k.o/for-4.7, k.o/for-4.8 branches.

---

Adit Ranadive (15):
IB/pvrdma: Add paravirtual rdma device
IB/pvrdma: Add device command support
IB/pvrdma: Add support for Completion Queues
IB/pvrdma: Add the paravirtual RDMA device specification
IB/pvrdma: Add UAR support
IB/pvrdma: Add virtual device RDMA structures
IB/pvrdma: Add the main driver module for PVRDMA
IB/pvrdma: Add helper functions
IB/pvrdma: Add support for memory regions
IB/pvrdma: Add Queue Pair support
IB/pvrdma: Add user-level shared functions
IB/pvrdma: Add functions for Verbs support
IB/pvrdma: Add Kconfig and Makefile
IB: Add PVRDMA driver
MAINTAINERS: Update for PVRDMA driver

MAINTAINERS | 7 +
drivers/infiniband/Kconfig | 1 +
drivers/infiniband/hw/Makefile | 1 +
drivers/infiniband/hw/pvrdma/Kconfig | 8 +
drivers/infiniband/hw/pvrdma/Makefile | 3 +
drivers/infiniband/hw/pvrdma/pvrdma.h | 459 +++++++++
drivers/infiniband/hw/pvrdma/pvrdma_cmd.c | 104 +++
drivers/infiniband/hw/pvrdma/pvrdma_cq.c | 436 +++++++++
drivers/infiniband/hw/pvrdma/pvrdma_defs.h | 300 ++++++
drivers/infiniband/hw/pvrdma/pvrdma_dev_api.h | 342 +++++++
drivers/infiniband/hw/pvrdma/pvrdma_doorbell.c | 128 +++
drivers/infiniband/hw/pvrdma/pvrdma_ib_verbs.h | 450 +++++++++
drivers/infiniband/hw/pvrdma/pvrdma_main.c | 1188 ++++++++++++++++++++++++
drivers/infiniband/hw/pvrdma/pvrdma_misc.c | 309 ++++++
drivers/infiniband/hw/pvrdma/pvrdma_mr.c | 333 +++++++
drivers/infiniband/hw/pvrdma/pvrdma_qp.c | 976 +++++++++++++++++++
drivers/infiniband/hw/pvrdma/pvrdma_uapi.h | 247 +++++
drivers/infiniband/hw/pvrdma/pvrdma_user.h | 99 ++
drivers/infiniband/hw/pvrdma/pvrdma_verbs.c | 593 ++++++++++++
drivers/infiniband/hw/pvrdma/pvrdma_verbs.h | 108 +++
20 files changed, 6092 insertions(+)
create mode 100644 drivers/infiniband/hw/pvrdma/Kconfig
create mode 100644 drivers/infiniband/hw/pvrdma/Makefile
create mode 100644 drivers/infiniband/hw/pvrdma/pvrdma.h
create mode 100644 drivers/infiniband/hw/pvrdma/pvrdma_cmd.c
create mode 100644 drivers/infiniband/hw/pvrdma/pvrdma_cq.c
create mode 100644 drivers/infiniband/hw/pvrdma/pvrdma_defs.h
create mode 100644 drivers/infiniband/hw/pvrdma/pvrdma_dev_api.h
create mode 100644 drivers/infiniband/hw/pvrdma/pvrdma_doorbell.c
create mode 100644 drivers/infiniband/hw/pvrdma/pvrdma_ib_verbs.h
create mode 100644 drivers/infiniband/hw/pvrdma/pvrdma_main.c
create mode 100644 drivers/infiniband/hw/pvrdma/pvrdma_misc.c
create mode 100644 drivers/infiniband/hw/pvrdma/pvrdma_mr.c
create mode 100644 drivers/infiniband/hw/pvrdma/pvrdma_qp.c
create mode 100644 drivers/infiniband/hw/pvrdma/pvrdma_uapi.h
create mode 100644 drivers/infiniband/hw/pvrdma/pvrdma_user.h
create mode 100644 drivers/infiniband/hw/pvrdma/pvrdma_verbs.c
create mode 100644 drivers/infiniband/hw/pvrdma/pvrdma_verbs.h