Re: [PATCH v8] verbs: Introduce a new reg_mr API for virtual address space

[Jason Gunthorpe <jgg@mellanox.com>]

On Mon, Jul 15, 2019 at 05:13:28PM +0300, Yuval Shaia wrote:
> The virtual address that is registered is used as a base for any address
> passed later in post_recv and post_send operations.
> 
> On some virtualized environment this is not correct.
> 
> A guest cannot register its memory so hypervisor maps the guest physical
> address to a host virtual address and register it with the HW. Later on,
> at datapath phase, the guest fills the SGEs with addresses from its
> address space.
> Since HW cannot access guest virtual address space an extra translation
> is needed to map those addresses to be based on the host virtual address
> that was registered with the HW.
> This datapath interference affects performances.
> 
> To avoid this, a logical separation between the address that is
> registered and the address that is used as a offset at datapath phase is
> needed.
> This separation is already implemented in the lower layer part
> (ibv_cmd_reg_mr) but blocked at the API level.
> 
> Fix it by introducing a new API function which accepts an address from
> guest virtual address space as well, to be used as offset for later
> datapath operations.
> 
> Signed-off-by: Yuval Shaia <yuval.shaia@oracle.com>
> Reviewed-by: Leon Romanovsky <leonro@mellanox.com>
> v0 -> v1:
> 	* Change reg_mr callback signature instead of adding new callback
> 	* Add the new API to libibverbs/libibverbs.map.in
> v1 -> v2:
> 	* Do not modify reg_mr signature for version 1.0
> 	* Add note to man page
> v2 -> v3:
> 	* Rename function to reg_mr_iova (and arg-name to iova)
> 	* Some checkpatch issues not related to this fix but detected now
> 		* s/__FUNCTION__/__func
> 		* WARNING: function definition argument 'void *' should
> 		  also have an identifier name
> v3 -> v4:
> 	* Fix commit message as suggested by Adit Ranadiv
> 	* Add support for efa
> v4 -> v5:
> 	* Update PABI
> 	* Update debian files
> v5 -> v6:
> 	* Move the new API to section in libibverbs/libibverbs.map.in
> 	  (IBVERBS_1.7) as pointed out by Mark Haywood
> v6 -> v7:
> 	* 
> v7 -> v8:
> 	* Update also redhat and suse specfiles so now all CI checks in
> 	  github passed.
> 	* Leon, i have your r-b from v5, appriciate if you can take a look
> 	  again now, with all the latest changes
>  CMakeLists.txt                    |  4 ++--
>  buildlib/cbuild                   |  6 ++++++
>  debian/control                    |  2 +-
>  debian/libibverbs1.symbols        |  4 +++-
>  libibverbs/CMakeLists.txt         |  2 +-
>  libibverbs/driver.h               |  2 +-
>  libibverbs/dummy_ops.c            |  2 +-
>  libibverbs/libibverbs.map.in      |  5 +++++
>  libibverbs/man/ibv_reg_mr.3       | 15 +++++++++++++--
>  libibverbs/verbs.c                | 23 ++++++++++++++++++++++-
>  libibverbs/verbs.h                |  7 +++++++
>  providers/bnxt_re/verbs.c         |  6 +++---
>  providers/bnxt_re/verbs.h         |  2 +-
>  providers/cxgb3/iwch.h            |  4 ++--
>  providers/cxgb3/verbs.c           | 15 +++++----------
>  providers/cxgb4/libcxgb4.h        |  4 ++--
>  providers/cxgb4/verbs.c           | 15 +++++----------
>  providers/efa/verbs.c             |  4 ++--
>  providers/efa/verbs.h             |  2 +-
>  providers/hfi1verbs/hfiverbs.h    |  4 ++--
>  providers/hfi1verbs/verbs.c       |  8 ++++----
>  providers/hns/hns_roce_u.h        |  2 +-
>  providers/hns/hns_roce_u_verbs.c  |  6 +++---
>  providers/i40iw/i40iw_umain.h     |  3 ++-
>  providers/i40iw/i40iw_uverbs.c    |  8 ++++----
>  providers/ipathverbs/ipathverbs.h |  4 ++--
>  providers/ipathverbs/verbs.c      |  8 ++++----
>  providers/mlx4/mlx4.h             |  4 ++--
>  providers/mlx4/verbs.c            |  7 +++----
>  providers/mlx5/mlx5.h             |  4 ++--
>  providers/mlx5/verbs.c            |  7 +++----
>  providers/mthca/ah.c              |  3 ++-
>  providers/mthca/mthca.h           |  4 ++--
>  providers/mthca/verbs.c           |  6 +++---
>  providers/nes/nes_umain.h         |  3 ++-
>  providers/nes/nes_uverbs.c        |  9 ++++-----
>  providers/ocrdma/ocrdma_main.h    |  4 ++--
>  providers/ocrdma/ocrdma_verbs.c   | 10 ++++------
>  providers/qedr/qelr_main.h        |  4 ++--
>  providers/qedr/qelr_verbs.c       | 11 ++++-------
>  providers/qedr/qelr_verbs.h       |  4 ++--
>  providers/rxe/rxe.c               |  6 +++---
>  providers/siw/siw.c               |  4 ++--
>  providers/vmw_pvrdma/pvrdma.h     |  4 ++--
>  providers/vmw_pvrdma/verbs.c      |  7 +++----
>  redhat/rdma-core.spec             |  2 +-
>  suse/rdma-core.spec               |  2 +-
>  47 files changed, 154 insertions(+), 118 deletions(-)
> 
> diff --git a/CMakeLists.txt b/CMakeLists.txt
> index b2613284..67112ae3 100644
> +++ b/CMakeLists.txt
> @@ -68,11 +68,11 @@ endif()
>  set(PACKAGE_NAME "RDMA")
>  
>  # See Documentation/versioning.md
> -set(PACKAGE_VERSION "25.0")
> +set(PACKAGE_VERSION "26.0")
>  # When this is changed the values in these files need changing too:
>  #   debian/control
>  #   debian/libibverbs1.symbols
> -set(IBVERBS_PABI_VERSION "25")
> +set(IBVERBS_PABI_VERSION "26")
>  set(IBVERBS_PROVIDER_SUFFIX "-rdmav${IBVERBS_PABI_VERSION}.so")

'25' is still the current release-in progress.

Jason