* [PATCH 1/2] net/mlx5: Define interface bits for fencing UMR wqe @ 2017-05-25 20:13 Max Gurtovoy 2017-05-25 20:13 ` [PATCH 2/2] IB/mlx5: set UMR wqe fence according to HCA cap Max Gurtovoy 2017-05-28 8:43 ` [PATCH 1/2] net/mlx5: Define interface bits for fencing UMR wqe Christoph Hellwig 0 siblings, 2 replies; 5+ messages in thread From: Max Gurtovoy @ 2017-05-25 20:13 UTC (permalink / raw) HW can implement UMR wqe re-transmission in various ways. Thus, add HCA cap to distinguish the needed fence for UMR to make sure that the wqe wouldn't fail on mkey checks. Signed-off-by: Max Gurtovoy <maxg at mellanox.com> Acked-by: Leon Romanovsky <leon at kernel.org> --- include/linux/mlx5/mlx5_ifc.h | 10 +++++++++- 1 files changed, 9 insertions(+), 1 deletions(-) diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h index 32de072..edafedb 100644 --- a/include/linux/mlx5/mlx5_ifc.h +++ b/include/linux/mlx5/mlx5_ifc.h @@ -766,6 +766,12 @@ enum { MLX5_CAP_PORT_TYPE_ETH = 0x1, }; +enum { + MLX5_CAP_UMR_FENCE_STRONG = 0x0, + MLX5_CAP_UMR_FENCE_SMALL = 0x1, + MLX5_CAP_UMR_FENCE_NONE = 0x2, +}; + struct mlx5_ifc_cmd_hca_cap_bits { u8 reserved_at_0[0x80]; @@ -875,7 +881,9 @@ struct mlx5_ifc_cmd_hca_cap_bits { u8 reserved_at_202[0x1]; u8 ipoib_enhanced_offloads[0x1]; u8 ipoib_basic_offloads[0x1]; - u8 reserved_at_205[0xa]; + u8 reserved_at_205[0x5]; + u8 umr_fence[0x2]; + u8 reserved_at_20c[0x3]; u8 drain_sigerr[0x1]; u8 cmdif_checksum[0x2]; u8 sigerr_cqe[0x1]; -- 1.7.1 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH 2/2] IB/mlx5: set UMR wqe fence according to HCA cap 2017-05-25 20:13 [PATCH 1/2] net/mlx5: Define interface bits for fencing UMR wqe Max Gurtovoy @ 2017-05-25 20:13 ` Max Gurtovoy 2017-05-26 6:50 ` Christoph Hellwig 2017-05-28 8:43 ` [PATCH 1/2] net/mlx5: Define interface bits for fencing UMR wqe Christoph Hellwig 1 sibling, 1 reply; 5+ messages in thread From: Max Gurtovoy @ 2017-05-25 20:13 UTC (permalink / raw) Cache the needed umr_fence and set the wqe ctrl segmennt accordingly. Signed-off-by: Max Gurtovoy <maxg at mellanox.com> Acked-by: Leon Romanovsky <leon at kernel.org> --- drivers/infiniband/hw/mlx5/main.c | 14 ++++++++++++++ drivers/infiniband/hw/mlx5/mlx5_ib.h | 1 + drivers/infiniband/hw/mlx5/qp.c | 15 +++++++-------- 3 files changed, 22 insertions(+), 8 deletions(-) diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c index d45772d..83d1f9b 100644 --- a/drivers/infiniband/hw/mlx5/main.c +++ b/drivers/infiniband/hw/mlx5/main.c @@ -2979,6 +2979,18 @@ static int create_umr_res(struct mlx5_ib_dev *dev) return ret; } +static u8 mlx5_get_umr_fence(u8 umr_fence_cap) +{ + switch (umr_fence_cap) { + case MLX5_CAP_UMR_FENCE_STRONG: + return MLX5_FENCE_MODE_STRONG_ORDERING; + case MLX5_CAP_UMR_FENCE_SMALL: + return MLX5_FENCE_MODE_INITIATOR_SMALL; + default: + return MLX5_FENCE_MODE_NONE; + } +} + static int create_dev_resources(struct mlx5_ib_resources *devr) { struct ib_srq_init_attr attr; @@ -3693,6 +3705,8 @@ static void mlx5_ib_free_rdma_netdev(struct net_device *netdev) mlx5_ib_internal_fill_odp_caps(dev); + dev->umr_fence = mlx5_get_umr_fence(MLX5_CAP_GEN(mdev, umr_fence)); + if (MLX5_CAP_GEN(mdev, imaicl)) { dev->ib_dev.alloc_mw = mlx5_ib_alloc_mw; dev->ib_dev.dealloc_mw = mlx5_ib_dealloc_mw; diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h index 38c877b..0e08a58 100644 --- a/drivers/infiniband/hw/mlx5/mlx5_ib.h +++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h @@ -654,6 +654,7 @@ struct mlx5_ib_dev { struct mlx5_ib_port *port; struct mlx5_sq_bfreg bfreg; struct mlx5_sq_bfreg fp_bfreg; + u8 umr_fence; }; static inline struct mlx5_ib_cq *to_mibcq(struct mlx5_core_cq *mcq) diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c index 93959e1..876a429 100644 --- a/drivers/infiniband/hw/mlx5/qp.c +++ b/drivers/infiniband/hw/mlx5/qp.c @@ -3738,11 +3738,10 @@ static void dump_wqe(struct mlx5_ib_qp *qp, int idx, int size_16) } } -static u8 get_fence(u8 fence, struct ib_send_wr *wr) +static u8 get_fence(u8 fence, struct ib_send_wr *wr, struct mlx5_ib_dev *dev) { - if (unlikely(wr->opcode == IB_WR_LOCAL_INV && - wr->send_flags & IB_SEND_FENCE)) - return MLX5_FENCE_MODE_STRONG_ORDERING; + if (wr->opcode == IB_WR_LOCAL_INV || wr->opcode == IB_WR_REG_MR) + return dev->umr_fence; if (unlikely(fence)) { if (wr->send_flags & IB_SEND_FENCE) @@ -3928,7 +3927,7 @@ int mlx5_ib_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, } finish_wqe(qp, ctrl, size, idx, wr->wr_id, - nreq, get_fence(fence, wr), + nreq, get_fence(fence, wr, dev), next_fence, MLX5_OPCODE_UMR); /* * SET_PSV WQEs are not signaled and solicited @@ -3955,7 +3954,7 @@ int mlx5_ib_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, } finish_wqe(qp, ctrl, size, idx, wr->wr_id, - nreq, get_fence(fence, wr), + nreq, get_fence(fence, wr, dev), next_fence, MLX5_OPCODE_SET_PSV); err = begin_wqe(qp, &seg, &ctrl, wr, &idx, &size, nreq); @@ -3977,7 +3976,7 @@ int mlx5_ib_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, } finish_wqe(qp, ctrl, size, idx, wr->wr_id, - nreq, get_fence(fence, wr), + nreq, get_fence(fence, wr, dev), next_fence, MLX5_OPCODE_SET_PSV); num_sge = 0; goto skip_psv; @@ -4090,7 +4089,7 @@ int mlx5_ib_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, } finish_wqe(qp, ctrl, size, idx, wr->wr_id, nreq, - get_fence(fence, wr), next_fence, + get_fence(fence, wr, dev), next_fence, mlx5_ib_opcode[wr->opcode]); skip_psv: if (0) -- 1.7.1 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH 2/2] IB/mlx5: set UMR wqe fence according to HCA cap 2017-05-25 20:13 ` [PATCH 2/2] IB/mlx5: set UMR wqe fence according to HCA cap Max Gurtovoy @ 2017-05-26 6:50 ` Christoph Hellwig 2017-05-26 19:04 ` Max Gurtovoy 0 siblings, 1 reply; 5+ messages in thread From: Christoph Hellwig @ 2017-05-26 6:50 UTC (permalink / raw) On Thu, May 25, 2017@11:13:39PM +0300, Max Gurtovoy wrote: > Cache the needed umr_fence and set the wqe ctrl segmennt > accordingly. > > Signed-off-by: Max Gurtovoy <maxg at mellanox.com> > Acked-by: Leon Romanovsky <leon at kernel.org> > --- > drivers/infiniband/hw/mlx5/main.c | 14 ++++++++++++++ > drivers/infiniband/hw/mlx5/mlx5_ib.h | 1 + > drivers/infiniband/hw/mlx5/qp.c | 15 +++++++-------- > 3 files changed, 22 insertions(+), 8 deletions(-) > > diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c > index d45772d..83d1f9b 100644 > --- a/drivers/infiniband/hw/mlx5/main.c > +++ b/drivers/infiniband/hw/mlx5/main.c > @@ -2979,6 +2979,18 @@ static int create_umr_res(struct mlx5_ib_dev *dev) > return ret; > } > > +static u8 mlx5_get_umr_fence(u8 umr_fence_cap) > +{ > + switch (umr_fence_cap) { > + case MLX5_CAP_UMR_FENCE_STRONG: > + return MLX5_FENCE_MODE_STRONG_ORDERING; > + case MLX5_CAP_UMR_FENCE_SMALL: > + return MLX5_FENCE_MODE_INITIATOR_SMALL; > + default: > + return MLX5_FENCE_MODE_NONE; Of course I don't really understand your firmware interface, but shouldn't strong ordering be the default, and only certain fw cap bits would relax it? ^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH 2/2] IB/mlx5: set UMR wqe fence according to HCA cap 2017-05-26 6:50 ` Christoph Hellwig @ 2017-05-26 19:04 ` Max Gurtovoy 0 siblings, 0 replies; 5+ messages in thread From: Max Gurtovoy @ 2017-05-26 19:04 UTC (permalink / raw) On 5/26/2017 9:50 AM, Christoph Hellwig wrote: > On Thu, May 25, 2017@11:13:39PM +0300, Max Gurtovoy wrote: >> Cache the needed umr_fence and set the wqe ctrl segmennt >> accordingly. >> >> Signed-off-by: Max Gurtovoy <maxg at mellanox.com> >> Acked-by: Leon Romanovsky <leon at kernel.org> >> --- >> drivers/infiniband/hw/mlx5/main.c | 14 ++++++++++++++ >> drivers/infiniband/hw/mlx5/mlx5_ib.h | 1 + >> drivers/infiniband/hw/mlx5/qp.c | 15 +++++++-------- >> 3 files changed, 22 insertions(+), 8 deletions(-) >> >> diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c >> index d45772d..83d1f9b 100644 >> --- a/drivers/infiniband/hw/mlx5/main.c >> +++ b/drivers/infiniband/hw/mlx5/main.c >> @@ -2979,6 +2979,18 @@ static int create_umr_res(struct mlx5_ib_dev *dev) >> return ret; >> } >> >> +static u8 mlx5_get_umr_fence(u8 umr_fence_cap) >> +{ >> + switch (umr_fence_cap) { >> + case MLX5_CAP_UMR_FENCE_STRONG: >> + return MLX5_FENCE_MODE_STRONG_ORDERING; >> + case MLX5_CAP_UMR_FENCE_SMALL: >> + return MLX5_FENCE_MODE_INITIATOR_SMALL; >> + default: >> + return MLX5_FENCE_MODE_NONE; > > Of course I don't really understand your firmware interface, but > shouldn't strong ordering be the default, and only certain fw > cap bits would relax it? > Actually it's a default value (MLX5_CAP_UMR_FENCE_STRONG = 0x0), but I can set it default in this function too in V2. ^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH 1/2] net/mlx5: Define interface bits for fencing UMR wqe 2017-05-25 20:13 [PATCH 1/2] net/mlx5: Define interface bits for fencing UMR wqe Max Gurtovoy 2017-05-25 20:13 ` [PATCH 2/2] IB/mlx5: set UMR wqe fence according to HCA cap Max Gurtovoy @ 2017-05-28 8:43 ` Christoph Hellwig 1 sibling, 0 replies; 5+ messages in thread From: Christoph Hellwig @ 2017-05-28 8:43 UTC (permalink / raw) Looks fine, Reviewed-by: Christoph Hellwig <hch at lst.de> ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2017-05-28 8:43 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2017-05-25 20:13 [PATCH 1/2] net/mlx5: Define interface bits for fencing UMR wqe Max Gurtovoy 2017-05-25 20:13 ` [PATCH 2/2] IB/mlx5: set UMR wqe fence according to HCA cap Max Gurtovoy 2017-05-26 6:50 ` Christoph Hellwig 2017-05-26 19:04 ` Max Gurtovoy 2017-05-28 8:43 ` [PATCH 1/2] net/mlx5: Define interface bits for fencing UMR wqe Christoph Hellwig
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox