* [PATCH rdma-next v2 0/4] Rely on firmware to get special mkeys
@ 2023-01-17 13:14 Leon Romanovsky
2023-01-17 13:14 ` [PATCH mlx5-next v2 1/4] net/mlx5: Expose bits for querying " Leon Romanovsky
` (4 more replies)
0 siblings, 5 replies; 6+ messages in thread
From: Leon Romanovsky @ 2023-01-17 13:14 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Eric Dumazet, Jakub Kicinski, linux-kernel, linux-rdma,
Michael Guralnik, netdev, Or Har-Toov, Paolo Abeni,
Saeed Mahameed
Changelog:
v2:
* Took a liberty and rewrote net/mlx5 patches
* change logic around terminate_scatter_list_mkey
* Added capability checks to be before executing command
v1: https://lore.kernel.org/all/cover.1672917578.git.leonro@nvidia.com
* Use already stored mkeys.terminate_scatter_list_mkey.
v0: https://lore.kernel.org/all/cover.1672819469.git.leonro@nvidia.com
-----------------------------------------------------------------------
This series from Or extends mlx5 driver to rely on firmware to get
special mkey values.
Thanks
Or Har-Toov (4):
net/mlx5: Expose bits for querying special mkeys
net/mlx5: Change define name for 0x100 lkey value
net/mlx5e: Use query_special_contexts for mkeys
RDMA/mlx5: Use query_special_contexts for mkeys
drivers/infiniband/hw/mlx5/cmd.c | 45 +++++++++++--------
drivers/infiniband/hw/mlx5/cmd.h | 3 +-
drivers/infiniband/hw/mlx5/main.c | 10 ++---
drivers/infiniband/hw/mlx5/mlx5_ib.h | 9 +++-
drivers/infiniband/hw/mlx5/odp.c | 27 ++++-------
drivers/infiniband/hw/mlx5/srq.c | 2 +-
drivers/infiniband/hw/mlx5/wr.c | 2 +-
.../net/ethernet/mellanox/mlx5/core/en_main.c | 22 ++++++++-
include/linux/mlx5/mlx5_ifc.h | 10 ++++-
include/linux/mlx5/qp.h | 2 +-
10 files changed, 81 insertions(+), 51 deletions(-)
--
2.39.0
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH mlx5-next v2 1/4] net/mlx5: Expose bits for querying special mkeys
2023-01-17 13:14 [PATCH rdma-next v2 0/4] Rely on firmware to get special mkeys Leon Romanovsky
@ 2023-01-17 13:14 ` Leon Romanovsky
2023-01-17 13:14 ` [PATCH mlx5-next v2 2/4] net/mlx5: Change define name for 0x100 lkey value Leon Romanovsky
` (3 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Leon Romanovsky @ 2023-01-17 13:14 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Or Har-Toov, Eric Dumazet, Jakub Kicinski, linux-rdma,
Michael Guralnik, netdev, Paolo Abeni, Saeed Mahameed
From: Or Har-Toov <ohartoov@nvidia.com>
Add needed HW bits to query the values of all special mkeys.
Signed-off-by: Or Har-Toov <ohartoov@nvidia.com>
Reviewed-by: Michael Guralnik <michaelgur@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
---
include/linux/mlx5/mlx5_ifc.h | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h
index 2d17b6a6d82d..8bbf15433bb2 100644
--- a/include/linux/mlx5/mlx5_ifc.h
+++ b/include/linux/mlx5/mlx5_ifc.h
@@ -1480,7 +1480,9 @@ struct mlx5_ifc_cmd_hca_cap_bits {
u8 relaxed_ordering_write[0x1];
u8 relaxed_ordering_read[0x1];
u8 log_max_mkey[0x6];
- u8 reserved_at_f0[0x8];
+ u8 reserved_at_f0[0x6];
+ u8 terminate_scatter_list_mkey[0x1];
+ u8 repeated_mkey[0x1];
u8 dump_fill_mkey[0x1];
u8 reserved_at_f9[0x2];
u8 fast_teardown[0x1];
@@ -5202,7 +5204,11 @@ struct mlx5_ifc_query_special_contexts_out_bits {
u8 null_mkey[0x20];
- u8 reserved_at_a0[0x60];
+ u8 terminate_scatter_list_mkey[0x20];
+
+ u8 repeated_mkey[0x20];
+
+ u8 reserved_at_a0[0x20];
};
struct mlx5_ifc_query_special_contexts_in_bits {
--
2.39.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH mlx5-next v2 2/4] net/mlx5: Change define name for 0x100 lkey value
2023-01-17 13:14 [PATCH rdma-next v2 0/4] Rely on firmware to get special mkeys Leon Romanovsky
2023-01-17 13:14 ` [PATCH mlx5-next v2 1/4] net/mlx5: Expose bits for querying " Leon Romanovsky
@ 2023-01-17 13:14 ` Leon Romanovsky
2023-01-17 13:14 ` [PATCH mlx5-next v2 3/4] net/mlx5e: Use query_special_contexts for mkeys Leon Romanovsky
` (2 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Leon Romanovsky @ 2023-01-17 13:14 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Or Har-Toov, Eric Dumazet, Jakub Kicinski, linux-rdma,
Michael Guralnik, netdev, Paolo Abeni, Saeed Mahameed
From: Or Har-Toov <ohartoov@nvidia.com>
Change define of 0x100 lkey value from MLX5_INVALID_LKEY to be
MLX5_TERMINATE_SCATTER_LIST_LKEY as 0x100 is the value of
terminate_scatter_list_mkey.
Signed-off-by: Or Har-Toov <ohartoov@nvidia.com>
Reviewed-by: Michael Guralnik <michaelgur@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
---
drivers/infiniband/hw/mlx5/odp.c | 10 +++++-----
drivers/infiniband/hw/mlx5/srq.c | 2 +-
drivers/infiniband/hw/mlx5/wr.c | 2 +-
drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 3 ++-
include/linux/mlx5/qp.h | 2 +-
5 files changed, 10 insertions(+), 9 deletions(-)
diff --git a/drivers/infiniband/hw/mlx5/odp.c b/drivers/infiniband/hw/mlx5/odp.c
index e6e021af6aa9..b4ebeadce67c 100644
--- a/drivers/infiniband/hw/mlx5/odp.c
+++ b/drivers/infiniband/hw/mlx5/odp.c
@@ -986,7 +986,7 @@ static int pagefault_data_segments(struct mlx5_ib_dev *dev,
{
int ret = 0, npages = 0;
u64 io_virt;
- u32 key;
+ __be32 key;
u32 byte_count;
size_t bcnt;
int inline_segment;
@@ -1000,7 +1000,7 @@ static int pagefault_data_segments(struct mlx5_ib_dev *dev,
struct mlx5_wqe_data_seg *dseg = wqe;
io_virt = be64_to_cpu(dseg->addr);
- key = be32_to_cpu(dseg->lkey);
+ key = dseg->lkey;
byte_count = be32_to_cpu(dseg->byte_count);
inline_segment = !!(byte_count & MLX5_INLINE_SEG);
bcnt = byte_count & ~MLX5_INLINE_SEG;
@@ -1014,8 +1014,8 @@ static int pagefault_data_segments(struct mlx5_ib_dev *dev,
}
/* receive WQE end of sg list. */
- if (receive_queue && bcnt == 0 && key == MLX5_INVALID_LKEY &&
- io_virt == 0)
+ if (receive_queue && bcnt == 0 &&
+ key == MLX5_TERMINATE_SCATTER_LIST_LKEY && io_virt == 0)
break;
if (!inline_segment && total_wqe_bytes) {
@@ -1034,7 +1034,7 @@ static int pagefault_data_segments(struct mlx5_ib_dev *dev,
continue;
}
- ret = pagefault_single_data_segment(dev, NULL, key,
+ ret = pagefault_single_data_segment(dev, NULL, be32_to_cpu(key),
io_virt, bcnt,
&pfault->bytes_committed,
bytes_mapped);
diff --git a/drivers/infiniband/hw/mlx5/srq.c b/drivers/infiniband/hw/mlx5/srq.c
index 757756c50cc6..bcceb14a07f9 100644
--- a/drivers/infiniband/hw/mlx5/srq.c
+++ b/drivers/infiniband/hw/mlx5/srq.c
@@ -447,7 +447,7 @@ int mlx5_ib_post_srq_recv(struct ib_srq *ibsrq, const struct ib_recv_wr *wr,
if (i < srq->msrq.max_avail_gather) {
scat[i].byte_count = 0;
- scat[i].lkey = cpu_to_be32(MLX5_INVALID_LKEY);
+ scat[i].lkey = MLX5_TERMINATE_SCATTER_LIST_LKEY;
scat[i].addr = 0;
}
}
diff --git a/drivers/infiniband/hw/mlx5/wr.c b/drivers/infiniband/hw/mlx5/wr.c
index 855f3f4fefad..bc44551493e2 100644
--- a/drivers/infiniband/hw/mlx5/wr.c
+++ b/drivers/infiniband/hw/mlx5/wr.c
@@ -1252,7 +1252,7 @@ int mlx5_ib_post_recv(struct ib_qp *ibqp, const struct ib_recv_wr *wr,
if (i < qp->rq.max_gs) {
scat[i].byte_count = 0;
- scat[i].lkey = cpu_to_be32(MLX5_INVALID_LKEY);
+ scat[i].lkey = MLX5_TERMINATE_SCATTER_LIST_LKEY;
scat[i].addr = 0;
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index cff5f2e29e1e..db4e66c38395 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -825,7 +825,8 @@ static int mlx5e_alloc_rq(struct mlx5e_params *params,
/* check if num_frags is not a pow of two */
if (rq->wqe.info.num_frags < (1 << rq->wqe.info.log_num_frags)) {
wqe->data[f].byte_count = 0;
- wqe->data[f].lkey = cpu_to_be32(MLX5_INVALID_LKEY);
+ wqe->data[f].lkey =
+ MLX5_TERMINATE_SCATTER_LIST_LKEY;
wqe->data[f].addr = 0;
}
}
diff --git a/include/linux/mlx5/qp.h b/include/linux/mlx5/qp.h
index 4657d5c54abe..df55fbb65717 100644
--- a/include/linux/mlx5/qp.h
+++ b/include/linux/mlx5/qp.h
@@ -36,7 +36,7 @@
#include <linux/mlx5/device.h>
#include <linux/mlx5/driver.h>
-#define MLX5_INVALID_LKEY 0x100
+#define MLX5_TERMINATE_SCATTER_LIST_LKEY cpu_to_be32(0x100)
/* UMR (3 WQE_BB's) + SIG (3 WQE_BB's) + PSV (mem) + PSV (wire) */
#define MLX5_SIG_WQE_SIZE (MLX5_SEND_WQE_BB * 8)
#define MLX5_DIF_SIZE 8
--
2.39.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH mlx5-next v2 3/4] net/mlx5e: Use query_special_contexts for mkeys
2023-01-17 13:14 [PATCH rdma-next v2 0/4] Rely on firmware to get special mkeys Leon Romanovsky
2023-01-17 13:14 ` [PATCH mlx5-next v2 1/4] net/mlx5: Expose bits for querying " Leon Romanovsky
2023-01-17 13:14 ` [PATCH mlx5-next v2 2/4] net/mlx5: Change define name for 0x100 lkey value Leon Romanovsky
@ 2023-01-17 13:14 ` Leon Romanovsky
2023-01-17 13:14 ` [PATCH rdma-next v2 4/4] RDMA/mlx5: " Leon Romanovsky
2023-02-17 20:30 ` [PATCH rdma-next v2 0/4] Rely on firmware to get special mkeys Jason Gunthorpe
4 siblings, 0 replies; 6+ messages in thread
From: Leon Romanovsky @ 2023-01-17 13:14 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Or Har-Toov, Eric Dumazet, Jakub Kicinski, linux-rdma,
Michael Guralnik, netdev, Paolo Abeni, Saeed Mahameed
From: Or Har-Toov <ohartoov@nvidia.com>
Use query_sepcial_contexts in order to get the correct value of
terminate_scatter_list_mkey, as FW will change it for certain
configurations.
Signed-off-by: Or Har-Toov <ohartoov@nvidia.com>
Reviewed-by: Michael Guralnik <michaelgur@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
---
.../net/ethernet/mellanox/mlx5/core/en_main.c | 23 +++++++++++++++++--
1 file changed, 21 insertions(+), 2 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index db4e66c38395..de7ab0911b30 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -661,6 +661,26 @@ static void mlx5e_rq_free_shampo(struct mlx5e_rq *rq)
mlx5e_rq_shampo_hd_free(rq);
}
+static __be32 mlx5e_get_terminate_scatter_list_mkey(struct mlx5_core_dev *dev)
+{
+ u32 out[MLX5_ST_SZ_DW(query_special_contexts_out)] = {};
+ u32 in[MLX5_ST_SZ_DW(query_special_contexts_in)] = {};
+ int res;
+
+ if (!MLX5_CAP_GEN(dev, terminate_scatter_list_mkey))
+ return MLX5_TERMINATE_SCATTER_LIST_LKEY;
+
+ MLX5_SET(query_special_contexts_in, in, opcode,
+ MLX5_CMD_OP_QUERY_SPECIAL_CONTEXTS);
+ res = mlx5_cmd_exec_inout(dev, query_special_contexts, in, out);
+ if (res)
+ return MLX5_TERMINATE_SCATTER_LIST_LKEY;
+
+ res = MLX5_GET(query_special_contexts_out, out,
+ terminate_scatter_list_mkey);
+ return cpu_to_be32(res);
+}
+
static int mlx5e_alloc_rq(struct mlx5e_params *params,
struct mlx5e_xsk_param *xsk,
struct mlx5e_rq_param *rqp,
@@ -825,8 +845,7 @@ static int mlx5e_alloc_rq(struct mlx5e_params *params,
/* check if num_frags is not a pow of two */
if (rq->wqe.info.num_frags < (1 << rq->wqe.info.log_num_frags)) {
wqe->data[f].byte_count = 0;
- wqe->data[f].lkey =
- MLX5_TERMINATE_SCATTER_LIST_LKEY;
+ wqe->data[f].lkey = mlx5e_get_terminate_scatter_list_mkey(mdev);
wqe->data[f].addr = 0;
}
}
--
2.39.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH rdma-next v2 4/4] RDMA/mlx5: Use query_special_contexts for mkeys
2023-01-17 13:14 [PATCH rdma-next v2 0/4] Rely on firmware to get special mkeys Leon Romanovsky
` (2 preceding siblings ...)
2023-01-17 13:14 ` [PATCH mlx5-next v2 3/4] net/mlx5e: Use query_special_contexts for mkeys Leon Romanovsky
@ 2023-01-17 13:14 ` Leon Romanovsky
2023-02-17 20:30 ` [PATCH rdma-next v2 0/4] Rely on firmware to get special mkeys Jason Gunthorpe
4 siblings, 0 replies; 6+ messages in thread
From: Leon Romanovsky @ 2023-01-17 13:14 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Or Har-Toov, David S. Miller, Eric Dumazet, Jakub Kicinski,
linux-rdma, Michael Guralnik, netdev, Paolo Abeni, Saeed Mahameed
From: Or Har-Toov <ohartoov@nvidia.com>
Use query_sepcial_contexts to get the correct value of mkeys such as
null_mkey, terminate_scatter_list_mkey and dump_fill_mkey, as FW will
change them in certain configurations.
Signed-off-by: Or Har-Toov <ohartoov@nvidia.com>
Reviewed-by: Michael Guralnik <michaelgur@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
---
drivers/infiniband/hw/mlx5/cmd.c | 45 ++++++++++++++++------------
drivers/infiniband/hw/mlx5/cmd.h | 3 +-
drivers/infiniband/hw/mlx5/main.c | 10 +++----
drivers/infiniband/hw/mlx5/mlx5_ib.h | 9 +++++-
drivers/infiniband/hw/mlx5/odp.c | 21 ++++---------
drivers/infiniband/hw/mlx5/srq.c | 2 +-
drivers/infiniband/hw/mlx5/wr.c | 2 +-
7 files changed, 48 insertions(+), 44 deletions(-)
diff --git a/drivers/infiniband/hw/mlx5/cmd.c b/drivers/infiniband/hw/mlx5/cmd.c
index ff3742b0460a..1d0c8d5e745b 100644
--- a/drivers/infiniband/hw/mlx5/cmd.c
+++ b/drivers/infiniband/hw/mlx5/cmd.c
@@ -5,34 +5,41 @@
#include "cmd.h"
-int mlx5_cmd_dump_fill_mkey(struct mlx5_core_dev *dev, u32 *mkey)
+int mlx5r_cmd_query_special_mkeys(struct mlx5_ib_dev *dev)
{
u32 out[MLX5_ST_SZ_DW(query_special_contexts_out)] = {};
u32 in[MLX5_ST_SZ_DW(query_special_contexts_in)] = {};
+ bool is_terminate, is_dump, is_null;
int err;
- MLX5_SET(query_special_contexts_in, in, opcode,
- MLX5_CMD_OP_QUERY_SPECIAL_CONTEXTS);
- err = mlx5_cmd_exec_inout(dev, query_special_contexts, in, out);
- if (!err)
- *mkey = MLX5_GET(query_special_contexts_out, out,
- dump_fill_mkey);
- return err;
-}
+ is_terminate = MLX5_CAP_GEN(dev->mdev, terminate_scatter_list_mkey);
+ is_dump = MLX5_CAP_GEN(dev->mdev, dump_fill_mkey);
+ is_null = MLX5_CAP_GEN(dev->mdev, null_mkey);
-int mlx5_cmd_null_mkey(struct mlx5_core_dev *dev, u32 *null_mkey)
-{
- u32 out[MLX5_ST_SZ_DW(query_special_contexts_out)] = {};
- u32 in[MLX5_ST_SZ_DW(query_special_contexts_in)] = {};
- int err;
+ dev->mkeys.terminate_scatter_list_mkey = MLX5_TERMINATE_SCATTER_LIST_LKEY;
+ if (!is_terminate && !is_dump && !is_null)
+ return 0;
MLX5_SET(query_special_contexts_in, in, opcode,
MLX5_CMD_OP_QUERY_SPECIAL_CONTEXTS);
- err = mlx5_cmd_exec_inout(dev, query_special_contexts, in, out);
- if (!err)
- *null_mkey = MLX5_GET(query_special_contexts_out, out,
- null_mkey);
- return err;
+ err = mlx5_cmd_exec_inout(dev->mdev, query_special_contexts, in, out);
+ if (err)
+ return err;
+
+ if (is_dump)
+ dev->mkeys.dump_fill_mkey = MLX5_GET(query_special_contexts_out,
+ out, dump_fill_mkey);
+
+ if (is_null)
+ dev->mkeys.null_mkey = cpu_to_be32(
+ MLX5_GET(query_special_contexts_out, out, null_mkey));
+
+ if (is_terminate)
+ dev->mkeys.terminate_scatter_list_mkey =
+ cpu_to_be32(MLX5_GET(query_special_contexts_out, out,
+ terminate_scatter_list_mkey));
+
+ return 0;
}
int mlx5_cmd_query_cong_params(struct mlx5_core_dev *dev, int cong_point,
diff --git a/drivers/infiniband/hw/mlx5/cmd.h b/drivers/infiniband/hw/mlx5/cmd.h
index ee46638db5de..93a971a40d11 100644
--- a/drivers/infiniband/hw/mlx5/cmd.h
+++ b/drivers/infiniband/hw/mlx5/cmd.h
@@ -37,8 +37,7 @@
#include <linux/kernel.h>
#include <linux/mlx5/driver.h>
-int mlx5_cmd_dump_fill_mkey(struct mlx5_core_dev *dev, u32 *mkey);
-int mlx5_cmd_null_mkey(struct mlx5_core_dev *dev, u32 *null_mkey);
+int mlx5r_cmd_query_special_mkeys(struct mlx5_ib_dev *dev);
int mlx5_cmd_query_cong_params(struct mlx5_core_dev *dev, int cong_point,
void *out);
int mlx5_cmd_dealloc_pd(struct mlx5_core_dev *dev, u32 pdn, u16 uid);
diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index 8588f2fc0cba..fb0d97bd4074 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -1756,13 +1756,9 @@ static int set_ucontext_resp(struct ib_ucontext *uctx,
struct mlx5_ib_dev *dev = to_mdev(ibdev);
struct mlx5_ib_ucontext *context = to_mucontext(uctx);
struct mlx5_bfreg_info *bfregi = &context->bfregi;
- int err;
if (MLX5_CAP_GEN(dev->mdev, dump_fill_mkey)) {
- err = mlx5_cmd_dump_fill_mkey(dev->mdev,
- &resp->dump_fill_mkey);
- if (err)
- return err;
+ resp->dump_fill_mkey = dev->mkeys.dump_fill_mkey;
resp->comp_mask |=
MLX5_IB_ALLOC_UCONTEXT_RESP_MASK_DUMP_FILL_MKEY;
}
@@ -3634,6 +3630,10 @@ static int mlx5_ib_stage_init_init(struct mlx5_ib_dev *dev)
dev->port[i].roce.last_port_state = IB_PORT_DOWN;
}
+ err = mlx5r_cmd_query_special_mkeys(dev);
+ if (err)
+ return err;
+
err = mlx5_ib_init_multiport_master(dev);
if (err)
return err;
diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h
index 7e1fb71bf7d4..ddb36c757074 100644
--- a/drivers/infiniband/hw/mlx5/mlx5_ib.h
+++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h
@@ -1051,6 +1051,13 @@ struct mlx5_port_caps {
u8 ext_port_cap;
};
+
+struct mlx5_special_mkeys {
+ u32 dump_fill_mkey;
+ __be32 null_mkey;
+ __be32 terminate_scatter_list_mkey;
+};
+
struct mlx5_ib_dev {
struct ib_device ib_dev;
struct mlx5_core_dev *mdev;
@@ -1081,7 +1088,6 @@ struct mlx5_ib_dev {
struct xarray odp_mkeys;
- u32 null_mkey;
struct mlx5_ib_flow_db *flow_db;
/* protect resources needed as part of reset flow */
spinlock_t reset_flow_resource_lock;
@@ -1110,6 +1116,7 @@ struct mlx5_ib_dev {
struct mlx5_port_caps port_caps[MLX5_MAX_PORTS];
u16 pkey_table_len;
u8 lag_ports;
+ struct mlx5_special_mkeys mkeys;
};
static inline struct mlx5_ib_cq *to_mibcq(struct mlx5_core_cq *mcq)
diff --git a/drivers/infiniband/hw/mlx5/odp.c b/drivers/infiniband/hw/mlx5/odp.c
index b4ebeadce67c..4998eaeadcbb 100644
--- a/drivers/infiniband/hw/mlx5/odp.c
+++ b/drivers/infiniband/hw/mlx5/odp.c
@@ -104,7 +104,7 @@ static void populate_klm(struct mlx5_klm *pklm, size_t idx, size_t nentries,
if (flags & MLX5_IB_UPD_XLT_ZAP) {
for (; pklm != end; pklm++, idx++) {
pklm->bcount = cpu_to_be32(MLX5_IMR_MTT_SIZE);
- pklm->key = cpu_to_be32(mr_to_mdev(imr)->null_mkey);
+ pklm->key = mr_to_mdev(imr)->mkeys.null_mkey;
pklm->va = 0;
}
return;
@@ -137,7 +137,7 @@ static void populate_klm(struct mlx5_klm *pklm, size_t idx, size_t nentries,
pklm->key = cpu_to_be32(mtt->ibmr.lkey);
pklm->va = cpu_to_be64(idx * MLX5_IMR_MTT_SIZE);
} else {
- pklm->key = cpu_to_be32(mr_to_mdev(imr)->null_mkey);
+ pklm->key = mr_to_mdev(imr)->mkeys.null_mkey;
pklm->va = 0;
}
}
@@ -1015,7 +1015,8 @@ static int pagefault_data_segments(struct mlx5_ib_dev *dev,
/* receive WQE end of sg list. */
if (receive_queue && bcnt == 0 &&
- key == MLX5_TERMINATE_SCATTER_LIST_LKEY && io_virt == 0)
+ key == dev->mkeys.terminate_scatter_list_mkey &&
+ io_virt == 0)
break;
if (!inline_segment && total_wqe_bytes) {
@@ -1615,25 +1616,15 @@ static const struct ib_device_ops mlx5_ib_dev_odp_ops = {
int mlx5_ib_odp_init_one(struct mlx5_ib_dev *dev)
{
- int ret = 0;
-
internal_fill_odp_caps(dev);
if (!(dev->odp_caps.general_caps & IB_ODP_SUPPORT))
- return ret;
+ return 0;
ib_set_device_ops(&dev->ib_dev, &mlx5_ib_dev_odp_ops);
- if (dev->odp_caps.general_caps & IB_ODP_SUPPORT_IMPLICIT) {
- ret = mlx5_cmd_null_mkey(dev->mdev, &dev->null_mkey);
- if (ret) {
- mlx5_ib_err(dev, "Error getting null_mkey %d\n", ret);
- return ret;
- }
- }
-
mutex_init(&dev->odp_eq_mutex);
- return ret;
+ return 0;
}
void mlx5_ib_odp_cleanup_one(struct mlx5_ib_dev *dev)
diff --git a/drivers/infiniband/hw/mlx5/srq.c b/drivers/infiniband/hw/mlx5/srq.c
index bcceb14a07f9..32c6643d0f7a 100644
--- a/drivers/infiniband/hw/mlx5/srq.c
+++ b/drivers/infiniband/hw/mlx5/srq.c
@@ -447,7 +447,7 @@ int mlx5_ib_post_srq_recv(struct ib_srq *ibsrq, const struct ib_recv_wr *wr,
if (i < srq->msrq.max_avail_gather) {
scat[i].byte_count = 0;
- scat[i].lkey = MLX5_TERMINATE_SCATTER_LIST_LKEY;
+ scat[i].lkey = dev->mkeys.terminate_scatter_list_mkey;
scat[i].addr = 0;
}
}
diff --git a/drivers/infiniband/hw/mlx5/wr.c b/drivers/infiniband/hw/mlx5/wr.c
index bc44551493e2..df1d1b0a3ef7 100644
--- a/drivers/infiniband/hw/mlx5/wr.c
+++ b/drivers/infiniband/hw/mlx5/wr.c
@@ -1252,7 +1252,7 @@ int mlx5_ib_post_recv(struct ib_qp *ibqp, const struct ib_recv_wr *wr,
if (i < qp->rq.max_gs) {
scat[i].byte_count = 0;
- scat[i].lkey = MLX5_TERMINATE_SCATTER_LIST_LKEY;
+ scat[i].lkey = dev->mkeys.terminate_scatter_list_mkey;
scat[i].addr = 0;
}
--
2.39.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH rdma-next v2 0/4] Rely on firmware to get special mkeys
2023-01-17 13:14 [PATCH rdma-next v2 0/4] Rely on firmware to get special mkeys Leon Romanovsky
` (3 preceding siblings ...)
2023-01-17 13:14 ` [PATCH rdma-next v2 4/4] RDMA/mlx5: " Leon Romanovsky
@ 2023-02-17 20:30 ` Jason Gunthorpe
4 siblings, 0 replies; 6+ messages in thread
From: Jason Gunthorpe @ 2023-02-17 20:30 UTC (permalink / raw)
To: Leon Romanovsky
Cc: Eric Dumazet, Jakub Kicinski, linux-kernel, linux-rdma,
Michael Guralnik, netdev, Or Har-Toov, Paolo Abeni,
Saeed Mahameed
On Tue, Jan 17, 2023 at 03:14:48PM +0200, Leon Romanovsky wrote:
> Changelog:
> v2:
> * Took a liberty and rewrote net/mlx5 patches
> * change logic around terminate_scatter_list_mkey
> * Added capability checks to be before executing command
> v1: https://lore.kernel.org/all/cover.1672917578.git.leonro@nvidia.com
> * Use already stored mkeys.terminate_scatter_list_mkey.
> v0: https://lore.kernel.org/all/cover.1672819469.git.leonro@nvidia.com
>
> -----------------------------------------------------------------------
> This series from Or extends mlx5 driver to rely on firmware to get
> special mkey values.
>
> Thanks
>
> Or Har-Toov (4):
> net/mlx5: Expose bits for querying special mkeys
> net/mlx5: Change define name for 0x100 lkey value
> net/mlx5e: Use query_special_contexts for mkeys
> RDMA/mlx5: Use query_special_contexts for mkeys
I pulled this into rdma, Saeed/Leon if you need these commits in the
shared branch then you can fast-forward the shared branch to:
commit 1b1e4868836a4b5b375be75fd4c9583d29500517
Author: Or Har-Toov <ohartoov@nvidia.com>
Date: Tue Jan 17 15:14:51 2023 +0200
net/mlx5e: Use query_special_contexts for mkeys
Thanks,
Jason
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2023-02-17 20:30 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-01-17 13:14 [PATCH rdma-next v2 0/4] Rely on firmware to get special mkeys Leon Romanovsky
2023-01-17 13:14 ` [PATCH mlx5-next v2 1/4] net/mlx5: Expose bits for querying " Leon Romanovsky
2023-01-17 13:14 ` [PATCH mlx5-next v2 2/4] net/mlx5: Change define name for 0x100 lkey value Leon Romanovsky
2023-01-17 13:14 ` [PATCH mlx5-next v2 3/4] net/mlx5e: Use query_special_contexts for mkeys Leon Romanovsky
2023-01-17 13:14 ` [PATCH rdma-next v2 4/4] RDMA/mlx5: " Leon Romanovsky
2023-02-17 20:30 ` [PATCH rdma-next v2 0/4] Rely on firmware to get special mkeys Jason Gunthorpe
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.