[PATCH rdma-next v1 0/5] Introduce dynamic UAR allocation mode

[PATCH rdma-next v1 0/5] Introduce dynamic UAR allocation mode

[Leon Romanovsky]

From: Leon Romanovsky <leonro@mellanox.com>

Changelog:
v1: * Added patch that moved mlx5_bfreg_info from global header to the mlx5_ib.h
    * No other changes.
v0: * https://lore.kernel.org/linux-rdma/20200318124329.52111-1-leon@kernel.org

----------------------------------------------------------------------------------

From Yishai,

This series exposes API to enable a dynamic allocation and management of a
UAR which now becomes to be a regular uobject.

Moving to that mode enables allocating a UAR only upon demand and drop the
redundant static allocation of UARs upon context creation.

In addition, it allows master and secondary processes that own the same command
FD to allocate and manage UARs according to their needs, this can’t be achieved
today.

As part of this option, QP & CQ creation flows were adapted to support this
dynamic UAR mode once asked by user space.

Once this mode is asked by mlx5 user space driver on a given context, it will
be mutual exclusive, means both the static and legacy dynamic modes for using
UARs will be blocked.

The legacy modes are supported for backward compatible reasons, looking
forward we expect this new mode to be the default.

Thanks

Leon Romanovsky (1):
  IB/mlx5: Limit the scope of struct mlx5_bfreg_info to mlx5_ib

Yishai Hadas (4):
  IB/mlx5: Expose UAR object and its alloc/destroy commands
  IB/mlx5: Extend CQ creation to get uar page index from user space
  IB/mlx5: Extend QP creation to get uar page index from user space
  IB/mlx5: Move to fully dynamic UAR mode once user space supports it

 drivers/infiniband/hw/mlx5/cq.c           |  21 ++-
 drivers/infiniband/hw/mlx5/main.c         | 185 ++++++++++++++++++++--
 drivers/infiniband/hw/mlx5/mlx5_ib.h      |  20 +++
 drivers/infiniband/hw/mlx5/qp.c           |  33 ++--
 include/linux/mlx5/driver.h               |  17 --
 include/rdma/uverbs_ioctl.h               |   2 +-
 include/uapi/rdma/mlx5-abi.h              |   6 +
 include/uapi/rdma/mlx5_user_ioctl_cmds.h  |  18 +++
 include/uapi/rdma/mlx5_user_ioctl_verbs.h |   5 +
 9 files changed, 263 insertions(+), 44 deletions(-)

--
2.24.1

[PATCH mlx5-next v1 4/5] IB/mlx5: Limit the scope of struct mlx5_bfreg_info to mlx5_ib

[Leon Romanovsky]

From: Leon Romanovsky <leonro@mellanox.com>

struct mlx5_bfreg_info is used by mlx5_ib only but is exposed to both
RDMA and netdev parts of mlx5 driver. Move that struct to mlx5_ib
namespace, clean vertical space alignment and convert lib_uar_4k
from bool to bitfield.

Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
---
 drivers/infiniband/hw/mlx5/mlx5_ib.h | 17 +++++++++++++++++
 include/linux/mlx5/driver.h          | 17 -----------------
 2 files changed, 17 insertions(+), 17 deletions(-)

diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h
index 370b03db366b..e26ba6c390ad 100644
--- a/drivers/infiniband/hw/mlx5/mlx5_ib.h
+++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h
@@ -128,6 +128,23 @@ enum mlx5_ib_mmap_type {
 	MLX5_IB_MMAP_TYPE_UAR_NC = 4,
 };
 
+struct mlx5_bfreg_info {
+	u32 *sys_pages;
+	int num_low_latency_bfregs;
+	unsigned int *count;
+
+	/*
+	 * protect bfreg allocation data structs
+	 */
+	struct mutex lock;
+	u32 ver;
+	u8 lib_uar_4k : 1;
+	u32 num_sys_pages;
+	u32 num_static_sys_pages;
+	u32 total_num_bfregs;
+	u32 num_dyn_bfregs;
+};
+
 struct mlx5_ib_ucontext {
 	struct ib_ucontext	ibucontext;
 	struct list_head	db_page_list;
diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h
index 3f10a9633012..6ef3e4368550 100644
--- a/include/linux/mlx5/driver.h
+++ b/include/linux/mlx5/driver.h
@@ -213,23 +213,6 @@ enum mlx5_port_status {
 	MLX5_PORT_DOWN      = 2,
 };
 
-struct mlx5_bfreg_info {
-	u32		       *sys_pages;
-	int			num_low_latency_bfregs;
-	unsigned int	       *count;
-
-	/*
-	 * protect bfreg allocation data structs
-	 */
-	struct mutex		lock;
-	u32			ver;
-	bool			lib_uar_4k;
-	u32			num_sys_pages;
-	u32			num_static_sys_pages;
-	u32			total_num_bfregs;
-	u32			num_dyn_bfregs;
-};
-
 struct mlx5_cmd_first {
 	__be32		data[4];
 };
-- 
2.24.1

Re: [PATCH rdma-next v1 0/5] Introduce dynamic UAR allocation mode

[Jason Gunthorpe]

On Tue, Mar 24, 2020 at 08:01:38AM +0200, Leon Romanovsky wrote:
> From: Leon Romanovsky <leonro@mellanox.com>
> 
> Changelog:
> v1: * Added patch that moved mlx5_bfreg_info from global header to the mlx5_ib.h
>     * No other changes.
> v0: * https://lore.kernel.org/linux-rdma/20200318124329.52111-1-leon@kernel.org
> 
> ----------------------------------------------------------------------------------
> 
> >From Yishai,
> 
> This series exposes API to enable a dynamic allocation and management of a
> UAR which now becomes to be a regular uobject.
> 
> Moving to that mode enables allocating a UAR only upon demand and drop the
> redundant static allocation of UARs upon context creation.
> 
> In addition, it allows master and secondary processes that own the same command
> FD to allocate and manage UARs according to their needs, this can’t be achieved
> today.
> 
> As part of this option, QP & CQ creation flows were adapted to support this
> dynamic UAR mode once asked by user space.
> 
> Once this mode is asked by mlx5 user space driver on a given context, it will
> be mutual exclusive, means both the static and legacy dynamic modes for using
> UARs will be blocked.
> 
> The legacy modes are supported for backward compatible reasons, looking
> forward we expect this new mode to be the default.
> 
> Thanks
> 
> Leon Romanovsky (1):
>   IB/mlx5: Limit the scope of struct mlx5_bfreg_info to mlx5_ib
> 
> Yishai Hadas (4):
>   IB/mlx5: Expose UAR object and its alloc/destroy commands
>   IB/mlx5: Extend CQ creation to get uar page index from user space
>   IB/mlx5: Extend QP creation to get uar page index from user space
>   IB/mlx5: Move to fully dynamic UAR mode once user space supports it

Applied to for-next, thanks

Jason