From: Leon Romanovsky <leon@kernel.org>
To: Selvin Xavier <selvin.xavier@broadcom.com>
Cc: jgg@ziepe.ca, linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com
Subject: Re: [PATCH for-next 1/2] RDMA/bnxt_re: Allow MSN table capability check
Date: Wed, 8 May 2024 11:22:52 +0300 [thread overview]
Message-ID: <20240508082252.GC78961@unreal> (raw)
In-Reply-To: <CA+sbYW0O=h4hdkCqtT3SievAyReV7uKMsbLsAg-s7-YHOHvyBw@mail.gmail.com>
On Mon, May 06, 2024 at 07:30:12PM +0530, Selvin Xavier wrote:
> On Sun, May 5, 2024 at 7:16 PM Leon Romanovsky <leon@kernel.org> wrote:
> >
> > On Fri, May 03, 2024 at 09:10:18PM -0700, Selvin Xavier wrote:
> > > FW reports the HW capability to use PSN table or MSN table and
> > > driver/library need to select it based on this capability.
> > > Use the new capability instead of the older capability check for HW
> > > retransmission while handling the MSN/PSN table.
> > > Also, Updated the FW interface structures to handle the new fields.
> > >
> > > Signed-off-by: Selvin Xavier <selvin.xavier@broadcom.com>
> > > ---
> > > drivers/infiniband/hw/bnxt_re/qplib_fp.c | 12 ++++++------
> > > drivers/infiniband/hw/bnxt_re/qplib_fp.h | 2 +-
> > > drivers/infiniband/hw/bnxt_re/qplib_res.h | 6 ++++++
> > > drivers/infiniband/hw/bnxt_re/qplib_sp.c | 1 +
> > > drivers/infiniband/hw/bnxt_re/qplib_sp.h | 1 +
> > > drivers/infiniband/hw/bnxt_re/roce_hsi.h | 30 +++++++++++++++++++++++++++++-
> > > 6 files changed, 44 insertions(+), 8 deletions(-)
> >
> > I have same comment as I gave in rdma-core PR.
> > It seems like your change will cause to old devices to behave as they
> > don't have PSN table anymore.
> Hi Leon,
>
> I replied to your comment in the rdma-core PR. For older devices, the
> PSN table is created. For new devices, MSN table will be created.
> This patch is changing the capability check from BNXT_RE_HW_RETX to
> _is_host_msn_table. Both capabilities are reported by FW.
> Driver is deciding which table to be created and it's better to decide
> it using _is_host_msn_table capability. For older adapters, this
> capability will be 0 which means PSN table.
Thanks.
>
> Thanks
> >
> > Thanks
> >
> > >
> > > diff --git a/drivers/infiniband/hw/bnxt_re/qplib_fp.c b/drivers/infiniband/hw/bnxt_re/qplib_fp.c
> > > index 439d0c7..3c961a8 100644
> > > --- a/drivers/infiniband/hw/bnxt_re/qplib_fp.c
> > > +++ b/drivers/infiniband/hw/bnxt_re/qplib_fp.c
> > > @@ -984,7 +984,7 @@ int bnxt_qplib_create_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp)
> > > u16 nsge;
> > >
> > > if (res->dattr)
> > > - qp->dev_cap_flags = res->dattr->dev_cap_flags;
> > > + qp->is_host_msn_tbl = _is_host_msn_table(res->dattr->dev_cap_flags2);
> > >
> > > sq->dbinfo.flags = 0;
> > > bnxt_qplib_rcfw_cmd_prep((struct cmdq_base *)&req,
> > > @@ -1002,7 +1002,7 @@ int bnxt_qplib_create_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp)
> > > sizeof(struct sq_psn_search_ext) :
> > > sizeof(struct sq_psn_search);
> > >
> > > - if (BNXT_RE_HW_RETX(qp->dev_cap_flags)) {
> > > + if (qp->is_host_msn_tbl) {
> > > psn_sz = sizeof(struct sq_msn_search);
> > > qp->msn = 0;
> > > }
> > > @@ -1015,7 +1015,7 @@ int bnxt_qplib_create_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp)
> > > hwq_attr.aux_stride = psn_sz;
> > > hwq_attr.aux_depth = bnxt_qplib_set_sq_size(sq, qp->wqe_mode);
> > > /* Update msn tbl size */
> > > - if (BNXT_RE_HW_RETX(qp->dev_cap_flags) && psn_sz) {
> > > + if (qp->is_host_msn_tbl && psn_sz) {
> > > hwq_attr.aux_depth = roundup_pow_of_two(bnxt_qplib_set_sq_size(sq, qp->wqe_mode));
> > > qp->msn_tbl_sz = hwq_attr.aux_depth;
> > > qp->msn = 0;
> > > @@ -1636,7 +1636,7 @@ static void bnxt_qplib_fill_psn_search(struct bnxt_qplib_qp *qp,
> > > if (!swq->psn_search)
> > > return;
> > > /* Handle MSN differently on cap flags */
> > > - if (BNXT_RE_HW_RETX(qp->dev_cap_flags)) {
> > > + if (qp->is_host_msn_tbl) {
> > > bnxt_qplib_fill_msn_search(qp, wqe, swq);
> > > return;
> > > }
> > > @@ -1818,7 +1818,7 @@ int bnxt_qplib_post_send(struct bnxt_qplib_qp *qp,
> > > }
> > >
> > > swq = bnxt_qplib_get_swqe(sq, &wqe_idx);
> > > - bnxt_qplib_pull_psn_buff(qp, sq, swq, BNXT_RE_HW_RETX(qp->dev_cap_flags));
> > > + bnxt_qplib_pull_psn_buff(qp, sq, swq, qp->is_host_msn_tbl);
> > >
> > > idx = 0;
> > > swq->slot_idx = hwq->prod;
> > > @@ -2008,7 +2008,7 @@ int bnxt_qplib_post_send(struct bnxt_qplib_qp *qp,
> > > rc = -EINVAL;
> > > goto done;
> > > }
> > > - if (!BNXT_RE_HW_RETX(qp->dev_cap_flags) || msn_update) {
> > > + if (!qp->is_host_msn_tbl || msn_update) {
> > > swq->next_psn = sq->psn & BTH_PSN_MASK;
> > > bnxt_qplib_fill_psn_search(qp, wqe, swq);
> > > }
> > > diff --git a/drivers/infiniband/hw/bnxt_re/qplib_fp.h b/drivers/infiniband/hw/bnxt_re/qplib_fp.h
> > > index 7fd4506..5b8d097 100644
> > > --- a/drivers/infiniband/hw/bnxt_re/qplib_fp.h
> > > +++ b/drivers/infiniband/hw/bnxt_re/qplib_fp.h
> > > @@ -340,7 +340,7 @@ struct bnxt_qplib_qp {
> > > struct list_head rq_flush;
> > > u32 msn;
> > > u32 msn_tbl_sz;
> > > - u16 dev_cap_flags;
> > > + u16 is_host_msn_tbl;
> > > };
> > >
> > > #define BNXT_QPLIB_MAX_CQE_ENTRY_SIZE sizeof(struct cq_base)
> > > diff --git a/drivers/infiniband/hw/bnxt_re/qplib_res.h b/drivers/infiniband/hw/bnxt_re/qplib_res.h
> > > index 61628f7..a0f78cd 100644
> > > --- a/drivers/infiniband/hw/bnxt_re/qplib_res.h
> > > +++ b/drivers/infiniband/hw/bnxt_re/qplib_res.h
> > > @@ -554,6 +554,12 @@ static inline bool _is_hw_retx_supported(u16 dev_cap_flags)
> > >
> > > #define BNXT_RE_HW_RETX(a) _is_hw_retx_supported((a))
> > >
> > > +static inline bool _is_host_msn_table(u16 dev_cap_ext_flags2)
> > > +{
> > > + return (dev_cap_ext_flags2 & CREQ_QUERY_FUNC_RESP_SB_REQ_RETRANSMISSION_SUPPORT_MASK) ==
> > > + CREQ_QUERY_FUNC_RESP_SB_REQ_RETRANSMISSION_SUPPORT_HOST_MSN_TABLE;
> > > +}
> > > +
> > > static inline u8 bnxt_qplib_dbr_pacing_en(struct bnxt_qplib_chip_ctx *cctx)
> > > {
> > > return cctx->modes.dbr_pacing;
> > > diff --git a/drivers/infiniband/hw/bnxt_re/qplib_sp.c b/drivers/infiniband/hw/bnxt_re/qplib_sp.c
> > > index 8beeedd..9328db9 100644
> > > --- a/drivers/infiniband/hw/bnxt_re/qplib_sp.c
> > > +++ b/drivers/infiniband/hw/bnxt_re/qplib_sp.c
> > > @@ -156,6 +156,7 @@ int bnxt_qplib_get_dev_attr(struct bnxt_qplib_rcfw *rcfw,
> > > (0x01 << RCFW_DBR_BASE_PAGE_SHIFT);
> > > attr->max_sgid = BNXT_QPLIB_NUM_GIDS_SUPPORTED;
> > > attr->dev_cap_flags = le16_to_cpu(sb->dev_cap_flags);
> > > + attr->dev_cap_flags2 = le16_to_cpu(sb->dev_cap_ext_flags_2);
> > >
> > > bnxt_qplib_query_version(rcfw, attr->fw_ver);
> > >
> > > diff --git a/drivers/infiniband/hw/bnxt_re/qplib_sp.h b/drivers/infiniband/hw/bnxt_re/qplib_sp.h
> > > index d33c78b..16a67d7 100644
> > > --- a/drivers/infiniband/hw/bnxt_re/qplib_sp.h
> > > +++ b/drivers/infiniband/hw/bnxt_re/qplib_sp.h
> > > @@ -72,6 +72,7 @@ struct bnxt_qplib_dev_attr {
> > > u8 tqm_alloc_reqs[MAX_TQM_ALLOC_REQ];
> > > bool is_atomic;
> > > u16 dev_cap_flags;
> > > + u16 dev_cap_flags2;
> > > u32 max_dpi;
> > > };
> > >
> > > diff --git a/drivers/infiniband/hw/bnxt_re/roce_hsi.h b/drivers/infiniband/hw/bnxt_re/roce_hsi.h
> > > index 605c946..0425309 100644
> > > --- a/drivers/infiniband/hw/bnxt_re/roce_hsi.h
> > > +++ b/drivers/infiniband/hw/bnxt_re/roce_hsi.h
> > > @@ -2157,8 +2157,36 @@ struct creq_query_func_resp_sb {
> > > __le32 tqm_alloc_reqs[12];
> > > __le32 max_dpi;
> > > u8 max_sge_var_wqe;
> > > - u8 reserved_8;
> > > + u8 dev_cap_ext_flags;
> > > + #define CREQ_QUERY_FUNC_RESP_SB_ATOMIC_OPS_NOT_SUPPORTED 0x1UL
> > > + #define CREQ_QUERY_FUNC_RESP_SB_DRV_VERSION_RGTR_SUPPORTED 0x2UL
> > > + #define CREQ_QUERY_FUNC_RESP_SB_CREATE_QP_BATCH_SUPPORTED 0x4UL
> > > + #define CREQ_QUERY_FUNC_RESP_SB_DESTROY_QP_BATCH_SUPPORTED 0x8UL
> > > + #define CREQ_QUERY_FUNC_RESP_SB_ROCE_STATS_EXT_CTX_SUPPORTED 0x10UL
> > > + #define CREQ_QUERY_FUNC_RESP_SB_CREATE_SRQ_SGE_SUPPORTED 0x20UL
> > > + #define CREQ_QUERY_FUNC_RESP_SB_FIXED_SIZE_WQE_DISABLED 0x40UL
> > > + #define CREQ_QUERY_FUNC_RESP_SB_DCN_SUPPORTED 0x80UL
> > > __le16 max_inline_data_var_wqe;
> > > + __le32 start_qid;
> > > + u8 max_msn_table_size;
> > > + u8 reserved8_1;
> > > + __le16 dev_cap_ext_flags_2;
> > > + #define CREQ_QUERY_FUNC_RESP_SB_OPTIMIZE_MODIFY_QP_SUPPORTED 0x1UL
> > > + #define CREQ_QUERY_FUNC_RESP_SB_CHANGE_UDP_SRC_PORT_WQE_SUPPORTED 0x2UL
> > > + #define CREQ_QUERY_FUNC_RESP_SB_CQ_COALESCING_SUPPORTED 0x4UL
> > > + #define CREQ_QUERY_FUNC_RESP_SB_MEMORY_REGION_RO_SUPPORTED 0x8UL
> > > + #define CREQ_QUERY_FUNC_RESP_SB_REQ_RETRANSMISSION_SUPPORT_MASK 0x30UL
> > > + #define CREQ_QUERY_FUNC_RESP_SB_REQ_RETRANSMISSION_SUPPORT_SFT 4
> > > + #define CREQ_QUERY_FUNC_RESP_SB_REQ_RETRANSMISSION_SUPPORT_HOST_PSN_TABLE (0x0UL << 4)
> > > + #define CREQ_QUERY_FUNC_RESP_SB_REQ_RETRANSMISSION_SUPPORT_HOST_MSN_TABLE (0x1UL << 4)
> > > + #define CREQ_QUERY_FUNC_RESP_SB_REQ_RETRANSMISSION_SUPPORT_IQM_MSN_TABLE (0x2UL << 4)
> > > + #define CREQ_QUERY_FUNC_RESP_SB_REQ_RETRANSMISSION_SUPPORT_LAST \
> > > + CREQ_QUERY_FUNC_RESP_SB_REQ_RETRANSMISSION_SUPPORT_IQM_MSN_TABLE
> > > + __le16 max_xp_qp_size;
> > > + __le16 create_qp_batch_size;
> > > + __le16 destroy_qp_batch_size;
> > > + __le16 reserved16;
> > > + __le64 reserved64;
> > > };
> > >
> > > /* cmdq_set_func_resources (size:448b/56B) */
> > > --
> > > 2.5.5
> > >
> >
> >
> >
next prev parent reply other threads:[~2024-05-08 8:22 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-05-04 4:10 [PATCH for-next 0/2] RDMA/bnxt_re: MSN table capability check for latest adapters Selvin Xavier
2024-05-04 4:10 ` [PATCH for-next 1/2] RDMA/bnxt_re: Allow MSN table capability check Selvin Xavier
2024-05-05 13:46 ` Leon Romanovsky
2024-05-06 14:00 ` Selvin Xavier
2024-05-08 8:22 ` Leon Romanovsky [this message]
2024-05-04 4:10 ` [PATCH for-next 2/2] RDMA/bnxt_re: Expose the MSN table capability for user library Selvin Xavier
2024-05-06 17:47 ` Jason Gunthorpe
2024-05-07 4:27 ` Selvin Xavier
2024-05-07 16:34 ` Jason Gunthorpe
2024-05-07 19:36 ` Selvin Xavier
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240508082252.GC78961@unreal \
--to=leon@kernel.org \
--cc=andrew.gospodarek@broadcom.com \
--cc=jgg@ziepe.ca \
--cc=linux-rdma@vger.kernel.org \
--cc=selvin.xavier@broadcom.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.