[PATCH mlx5-next 0/4] mlx5 core CQE API and misc updates

[PATCH mlx5-next 0/4] mlx5 core CQE API and misc updates

[Saeed Mahameed]

Hi

This patchset is for mlx5-next shared branch, and will be applied there
once the review is done.

Patches 1,2,3 are trivial improvements to CQE API
1. return CQE pointer instead of void pointer in get_cqe function
2. helper function for retrieving the CQE opcode, used in rdma and netdev

Patch 4, Move flow counters data structures definition to c file.

Thanks,
Saeed.

---

Daniel Jurgens (1):
  net/mlx5: When fetching CQEs return CQE instead of void pointer

Saeed Mahameed (1):
  net/mlx5: Move flow counters data structures from flow steering header

Tariq Toukan (2):
  net/mlx5: Use helper to get CQE opcode
  IB/mlx5: Use helper to get CQE opcode

 drivers/infiniband/hw/mlx5/cq.c               |  8 +++----
 .../net/ethernet/mellanox/mlx5/core/en_rx.c   | 10 ++++----
 .../net/ethernet/mellanox/mlx5/core/en_tx.c   |  2 +-
 .../ethernet/mellanox/mlx5/core/fpga/conn.c   |  2 +-
 .../net/ethernet/mellanox/mlx5/core/fs_core.h | 23 -------------------
 .../ethernet/mellanox/mlx5/core/fs_counters.c | 23 +++++++++++++++++++
 drivers/net/ethernet/mellanox/mlx5/core/wq.h  |  2 +-
 include/linux/mlx5/device.h                   |  5 ++++
 8 files changed, 40 insertions(+), 35 deletions(-)

-- 
2.19.2

[PATCH mlx5-next 1/4] net/mlx5: When fetching CQEs return CQE instead of void pointer

[Saeed Mahameed]

From: Daniel Jurgens <danielj@mellanox.com>

The function is only used to retrieve CQEs, use the proper type as the
return value.

Signed-off-by: Daniel Jurgens <danielj@mellanox.com>
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
---
 drivers/net/ethernet/mellanox/mlx5/core/wq.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/wq.h b/drivers/net/ethernet/mellanox/mlx5/core/wq.h
index b1293d153a58..9bc2184a46bc 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/wq.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/wq.h
@@ -177,7 +177,7 @@ static inline u32 mlx5_cqwq_get_ci(struct mlx5_cqwq *wq)
 	return mlx5_cqwq_ctr2ix(wq, wq->cc);
 }
 
-static inline void *mlx5_cqwq_get_wqe(struct mlx5_cqwq *wq, u32 ix)
+static inline struct mlx5_cqe64 *mlx5_cqwq_get_wqe(struct mlx5_cqwq *wq, u32 ix)
 {
 	return mlx5_frag_buf_get_wqe(&wq->fbc, ix);
 }
-- 
2.19.2

[PATCH mlx5-next 2/4] net/mlx5: Use helper to get CQE opcode

[Saeed Mahameed]

From: Tariq Toukan <tariqt@mellanox.com>

Introduce and use a helper that extracts the opcode
from a CQE (completion queue entry) structure.

Signed-off-by: Tariq Toukan <tariqt@mellanox.com>
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
---
 drivers/net/ethernet/mellanox/mlx5/core/en_rx.c     | 10 +++++-----
 drivers/net/ethernet/mellanox/mlx5/core/en_tx.c     |  2 +-
 drivers/net/ethernet/mellanox/mlx5/core/fpga/conn.c |  2 +-
 include/linux/mlx5/device.h                         |  5 +++++
 4 files changed, 12 insertions(+), 7 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
index 79638dcbae78..31956ddd394e 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
@@ -554,9 +554,9 @@ static inline void mlx5e_poll_ico_single_cqe(struct mlx5e_cq *cq,
 
 	mlx5_cqwq_pop(&cq->wq);
 
-	if (unlikely((cqe->op_own >> 4) != MLX5_CQE_REQ)) {
+	if (unlikely(get_cqe_opcode(cqe) != MLX5_CQE_REQ)) {
 		netdev_WARN_ONCE(cq->channel->netdev,
-				 "Bad OP in ICOSQ CQE: 0x%x\n", cqe->op_own);
+				 "Bad OP in ICOSQ CQE: 0x%x\n", get_cqe_opcode(cqe));
 		return;
 	}
 
@@ -898,7 +898,7 @@ mlx5e_skb_from_cqe_linear(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe,
 	prefetchw(va); /* xdp_frame data area */
 	prefetch(data);
 
-	if (unlikely((cqe->op_own >> 4) != MLX5_CQE_RESP_SEND)) {
+	if (unlikely(get_cqe_opcode(cqe) != MLX5_CQE_RESP_SEND)) {
 		rq->stats->wqe_err++;
 		return NULL;
 	}
@@ -930,7 +930,7 @@ mlx5e_skb_from_cqe_nonlinear(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe,
 	u16 byte_cnt     = cqe_bcnt - headlen;
 	struct sk_buff *skb;
 
-	if (unlikely((cqe->op_own >> 4) != MLX5_CQE_RESP_SEND)) {
+	if (unlikely(get_cqe_opcode(cqe) != MLX5_CQE_RESP_SEND)) {
 		rq->stats->wqe_err++;
 		return NULL;
 	}
@@ -1148,7 +1148,7 @@ void mlx5e_handle_rx_cqe_mpwrq(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe)
 
 	wi->consumed_strides += cstrides;
 
-	if (unlikely((cqe->op_own >> 4) != MLX5_CQE_RESP_SEND)) {
+	if (unlikely(get_cqe_opcode(cqe) != MLX5_CQE_RESP_SEND)) {
 		rq->stats->wqe_err++;
 		goto mpwrq_cqe_out;
 	}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
index 6dacaeba2fbf..46b5a6914d71 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
@@ -507,7 +507,7 @@ bool mlx5e_poll_tx_cq(struct mlx5e_cq *cq, int napi_budget)
 
 		wqe_counter = be16_to_cpu(cqe->wqe_counter);
 
-		if (unlikely(cqe->op_own >> 4 == MLX5_CQE_REQ_ERR)) {
+		if (unlikely(get_cqe_opcode(cqe) == MLX5_CQE_REQ_ERR)) {
 			if (!test_and_set_bit(MLX5E_SQ_STATE_RECOVERING,
 					      &sq->state)) {
 				mlx5e_dump_error_cqe(sq,
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fpga/conn.c b/drivers/net/ethernet/mellanox/mlx5/core/fpga/conn.c
index 8ca1d1949d93..873541ef4c1b 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/fpga/conn.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/fpga/conn.c
@@ -334,7 +334,7 @@ static void mlx5_fpga_conn_handle_cqe(struct mlx5_fpga_conn *conn,
 {
 	u8 opcode, status = 0;
 
-	opcode = cqe->op_own >> 4;
+	opcode = get_cqe_opcode(cqe);
 
 	switch (opcode) {
 	case MLX5_CQE_REQ_ERR:
diff --git a/include/linux/mlx5/device.h b/include/linux/mlx5/device.h
index f7c8bebfe472..c66867c8fc2f 100644
--- a/include/linux/mlx5/device.h
+++ b/include/linux/mlx5/device.h
@@ -781,6 +781,11 @@ static inline u8 mlx5_get_cqe_format(struct mlx5_cqe64 *cqe)
 	return (cqe->op_own >> 2) & 0x3;
 }
 
+static inline u8 get_cqe_opcode(struct mlx5_cqe64 *cqe)
+{
+	return cqe->op_own >> 4;
+}
+
 static inline u8 get_cqe_lro_tcppsh(struct mlx5_cqe64 *cqe)
 {
 	return (cqe->lro_tcppsh_abort_dupack >> 6) & 1;
-- 
2.19.2

Re: [PATCH mlx5-next 2/4] net/mlx5: Use helper to get CQE opcode

[Cong Wang]

On Tue, Dec 4, 2018 at 6:06 PM Saeed Mahameed <saeedm@mellanox.com> wrote:
>
> +static inline u8 get_cqe_opcode(struct mlx5_cqe64 *cqe)

Make it const please.

Re: [PATCH mlx5-next 2/4] net/mlx5: Use helper to get CQE opcode

[Saeed Mahameed]

On Tue, 2018-12-04 at 20:55 -0800, Cong Wang wrote:
> On Tue, Dec 4, 2018 at 6:06 PM Saeed Mahameed <saeedm@mellanox.com>
> wrote:
> > +static inline u8 get_cqe_opcode(struct mlx5_cqe64 *cqe)
> 
> Make it const please.

Please be specific

I can do:
static inline u8 get_cqe_opcode(struct mlx5_cqe64 const *cqe)
if that what you mean.

But i will leave it for a later time to do it for all cqe fields getter
functions in this file, i will keep the code uniform for now.

[PATCH mlx5-next 3/4] IB/mlx5: Use helper to get CQE opcode

[Saeed Mahameed]

From: Tariq Toukan <tariqt@mellanox.com>

Use the new helper that extracts the opcode
from a CQE (completion queue entry) structure.

Signed-off-by: Tariq Toukan <tariqt@mellanox.com>
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
---
 drivers/infiniband/hw/mlx5/cq.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/drivers/infiniband/hw/mlx5/cq.c b/drivers/infiniband/hw/mlx5/cq.c
index 0b99f7d0630d..26ab9041f94a 100644
--- a/drivers/infiniband/hw/mlx5/cq.c
+++ b/drivers/infiniband/hw/mlx5/cq.c
@@ -82,7 +82,7 @@ static void *get_sw_cqe(struct mlx5_ib_cq *cq, int n)
 
 	cqe64 = (cq->mcq.cqe_sz == 64) ? cqe : cqe + 64;
 
-	if (likely((cqe64->op_own) >> 4 != MLX5_CQE_INVALID) &&
+	if (likely(get_cqe_opcode(cqe64) != MLX5_CQE_INVALID) &&
 	    !((cqe64->op_own & MLX5_CQE_OWNER_MASK) ^ !!(n & (cq->ibcq.cqe + 1)))) {
 		return cqe;
 	} else {
@@ -197,7 +197,7 @@ static void handle_responder(struct ib_wc *wc, struct mlx5_cqe64 *cqe,
 	}
 	wc->byte_len = be32_to_cpu(cqe->byte_cnt);
 
-	switch (cqe->op_own >> 4) {
+	switch (get_cqe_opcode(cqe)) {
 	case MLX5_CQE_RESP_WR_IMM:
 		wc->opcode	= IB_WC_RECV_RDMA_WITH_IMM;
 		wc->wc_flags	= IB_WC_WITH_IMM;
@@ -537,7 +537,7 @@ static int mlx5_poll_one(struct mlx5_ib_cq *cq,
 	 */
 	rmb();
 
-	opcode = cqe64->op_own >> 4;
+	opcode = get_cqe_opcode(cqe64);
 	if (unlikely(opcode == MLX5_CQE_RESIZE_CQ)) {
 		if (likely(cq->resize_buf)) {
 			free_cq_buf(dev, &cq->buf);
@@ -1295,7 +1295,7 @@ static int copy_resize_cqes(struct mlx5_ib_cq *cq)
 		return -EINVAL;
 	}
 
-	while ((scqe64->op_own >> 4) != MLX5_CQE_RESIZE_CQ) {
+	while (get_cqe_opcode(scqe64) != MLX5_CQE_RESIZE_CQ) {
 		dcqe = mlx5_frag_buf_get_wqe(&cq->resize_buf->fbc,
 					     (i + 1) & cq->resize_buf->nent);
 		dcqe64 = dsize == 64 ? dcqe : dcqe + 64;
-- 
2.19.2

Re: [PATCH mlx5-next 3/4] IB/mlx5: Use helper to get CQE opcode

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 537 bytes --]

On Tue, Dec 04, 2018 at 06:03:02PM -0800, Saeed Mahameed wrote:
> From: Tariq Toukan <tariqt@mellanox.com>
>
> Use the new helper that extracts the opcode
> from a CQE (completion queue entry) structure.
>
> Signed-off-by: Tariq Toukan <tariqt@mellanox.com>
> Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
> ---
>  drivers/infiniband/hw/mlx5/cq.c | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)
>

I recommend you to squash this patch into the previous one.

Thanks,
Acked-by: Leon Romanovsky <leonro@mellanox.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

Re: [PATCH mlx5-next 3/4] IB/mlx5: Use helper to get CQE opcode

[Saeed Mahameed]

On Wed, 2018-12-05 at 08:09 +0000, Leon Romanovsky wrote:
> On Tue, Dec 04, 2018 at 06:03:02PM -0800, Saeed Mahameed wrote:
> > From: Tariq Toukan <tariqt@mellanox.com>
> > 
> > Use the new helper that extracts the opcode
> > from a CQE (completion queue entry) structure.
> > 
> > Signed-off-by: Tariq Toukan <tariqt@mellanox.com>
> > Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
> > ---
> >  drivers/infiniband/hw/mlx5/cq.c | 8 ++++----
> >  1 file changed, 4 insertions(+), 4 deletions(-)
> > 
> 
> I recommend you to squash this patch into the previous one.
> 
> Thanks,
> Acked-by: Leon Romanovsky <leonro@mellanox.com>

Thanks Leon, I would like to keep netdev and rdma patches separated as
much as possible 

[PATCH mlx5-next 4/4] net/mlx5: Move flow counters data structures from flow steering header

[Saeed Mahameed]

After the following flow counters API refactoring:
("net/mlx5: Use flow counter IDs and not the wrapping cache object")
flow counters private data structures mlx5_fc_cache and mlx5_fc are
redundantly exposed in fs_core.h, they have nothing to do with flow
steering core and they are private to fs_counter.c, this patch moves them
to where they belong and reduces their exposure in the driver.

Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
---
 .../net/ethernet/mellanox/mlx5/core/fs_core.h | 23 -------------------
 .../ethernet/mellanox/mlx5/core/fs_counters.c | 23 +++++++++++++++++++
 2 files changed, 23 insertions(+), 23 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.h b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.h
index b51ad217da32..2dc86347af58 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.h
@@ -145,29 +145,6 @@ struct mlx5_flow_table {
 	struct rhltable			fgs_hash;
 };
 
-struct mlx5_fc_cache {
-	u64 packets;
-	u64 bytes;
-	u64 lastuse;
-};
-
-struct mlx5_fc {
-	struct list_head list;
-	struct llist_node addlist;
-	struct llist_node dellist;
-
-	/* last{packets,bytes} members are used when calculating the delta since
-	 * last reading
-	 */
-	u64 lastpackets;
-	u64 lastbytes;
-
-	u32 id;
-	bool aging;
-
-	struct mlx5_fc_cache cache ____cacheline_aligned_in_smp;
-};
-
 struct mlx5_ft_underlay_qp {
 	struct list_head list;
 	u32 qpn;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fs_counters.c b/drivers/net/ethernet/mellanox/mlx5/core/fs_counters.c
index 32accd6b041b..c6c28f56aa29 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/fs_counters.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/fs_counters.c
@@ -41,6 +41,29 @@
 /* Max number of counters to query in bulk read is 32K */
 #define MLX5_SW_MAX_COUNTERS_BULK BIT(15)
 
+struct mlx5_fc_cache {
+	u64 packets;
+	u64 bytes;
+	u64 lastuse;
+};
+
+struct mlx5_fc {
+	struct list_head list;
+	struct llist_node addlist;
+	struct llist_node dellist;
+
+	/* last{packets,bytes} members are used when calculating the delta since
+	 * last reading
+	 */
+	u64 lastpackets;
+	u64 lastbytes;
+
+	u32 id;
+	bool aging;
+
+	struct mlx5_fc_cache cache ____cacheline_aligned_in_smp;
+};
+
 /* locking scheme:
  *
  * It is the responsibility of the user to prevent concurrent calls or bad
-- 
2.19.2

Re: [PATCH mlx5-next 4/4] net/mlx5: Move flow counters data structures from flow steering header

[Cong Wang]

On Tue, Dec 4, 2018 at 6:05 PM Saeed Mahameed <saeedm@mellanox.com> wrote:
>
> After the following flow counters API refactoring:
> ("net/mlx5: Use flow counter IDs and not the wrapping cache object")
> flow counters private data structures mlx5_fc_cache and mlx5_fc are
> redundantly exposed in fs_core.h, they have nothing to do with flow
> steering core and they are private to fs_counter.c, this patch moves them
> to where they belong and reduces their exposure in the driver.
>
> Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
> ---
>  .../net/ethernet/mellanox/mlx5/core/fs_core.h | 23 -------------------
>  .../ethernet/mellanox/mlx5/core/fs_counters.c | 23 +++++++++++++++++++

The #include <linux/llist.h> can be moved together too.

Re: [PATCH mlx5-next 4/4] net/mlx5: Move flow counters data structures from flow steering header

[Saeed Mahameed]

On Tue, 2018-12-04 at 21:04 -0800, Cong Wang wrote:
> On Tue, Dec 4, 2018 at 6:05 PM Saeed Mahameed <saeedm@mellanox.com>
> wrote:
> > After the following flow counters API refactoring:
> > ("net/mlx5: Use flow counter IDs and not the wrapping cache
> > object")
> > flow counters private data structures mlx5_fc_cache and mlx5_fc are
> > redundantly exposed in fs_core.h, they have nothing to do with flow
> > steering core and they are private to fs_counter.c, this patch
> > moves them
> > to where they belong and reduces their exposure in the driver.
> > 
> > Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
> > ---
> >  .../net/ethernet/mellanox/mlx5/core/fs_core.h | 23 -------------
> > ------
> >  .../ethernet/mellanox/mlx5/core/fs_counters.c | 23
> > +++++++++++++++++++
> 
> The #include <linux/llist.h> can be moved together too.

sure will do that in V2
thanks Cong!