* [PATCH rdma-next 1/9] RDMA/bnxt_re: Add support for collecting the Queue dumps
2025-02-20 18:34 [PATCH rdma-next 0/9] RDMA/bnxt_re: Driver Debug Enhancements Selvin Xavier
@ 2025-02-20 18:34 ` Selvin Xavier
2025-02-20 18:34 ` [PATCH rdma-next 2/9] RDMA/bnxt_re: Cache the QP information Selvin Xavier
` (8 subsequent siblings)
9 siblings, 0 replies; 15+ messages in thread
From: Selvin Xavier @ 2025-02-20 18:34 UTC (permalink / raw)
To: leon, jgg
Cc: linux-rdma, andrew.gospodarek, kalesh-anakkur.purayil, netdev,
davem, edumazet, kuba, abeni, horms, michael.chan,
Saravanan Vajravel, Selvin Xavier
From: Saravanan Vajravel <saravanan.vajravel@broadcom.com>
As part of enhancing the debug data collection, allocate
few data structures to hold the resources after the queues
are destroyed.
Initialize the data structures to capture the data. By default,
driver will cache the info of the QPs that are in error state.
The dump levels can be changed from debugfs hook in a later
patch. Driver caches the info of the last 1024 entries only.
Signed-off-by: Saravanan Vajravel <saravanan.vajravel@broadcom.com>
Signed-off-by: Selvin Xavier <selvin.xavier@broadcom.com>
---
drivers/infiniband/hw/bnxt_re/bnxt_re.h | 64 +++++++++++++++++++++++++++++++++
drivers/infiniband/hw/bnxt_re/main.c | 18 ++++++++++
2 files changed, 82 insertions(+)
diff --git a/drivers/infiniband/hw/bnxt_re/bnxt_re.h b/drivers/infiniband/hw/bnxt_re/bnxt_re.h
index b33b04e..5818db1 100644
--- a/drivers/infiniband/hw/bnxt_re/bnxt_re.h
+++ b/drivers/infiniband/hw/bnxt_re/bnxt_re.h
@@ -173,6 +173,67 @@ static inline bool bnxt_re_chip_gen_p7(u16 chip_num)
chip_num == CHIP_NUM_57608);
}
+#define BNXT_RE_MAX_QDUMP_ENTRIES 1024
+
+struct qdump_qpinfo {
+ u32 id;
+ u32 dest_qpid;
+ u64 qp_handle;
+ u32 mtu;
+ u8 type;
+ u8 wqe_mode;
+ u8 state;
+ u8 is_user;
+ u64 scq_handle;
+ u64 rcq_handle;
+ u32 scq_id;
+ u32 rcq_id;
+};
+
+struct qdump_mrinfo {
+ int type;
+ u32 lkey;
+ u32 rkey;
+ u64 total_size;
+ u64 mr_handle;
+};
+
+struct qdump_element {
+ struct bnxt_qplib_pbl pbl[PBL_LVL_MAX];
+ enum bnxt_qplib_pbl_lvl level;
+ struct bnxt_qplib_hwq *hwq;
+ struct bnxt_re_dev *rdev;
+ struct ib_umem *umem;
+ bool is_user_qp;
+ char des[32];
+ char *buf;
+ size_t len;
+ u16 stride;
+ u32 prod;
+ u32 cons;
+};
+
+struct qdump_array {
+ struct qdump_qpinfo qpinfo;
+ struct qdump_mrinfo mrinfo;
+ bool valid;
+ bool is_mr;
+};
+
+struct bnxt_re_qdump_head {
+ struct qdump_array *qdump;
+ u32 max_elements;
+ struct mutex lock; /* lock qdump array elements */
+ u32 index;
+};
+
+enum {
+ BNXT_RE_SNAPDUMP_NONE = 0,
+ BNXT_RE_SNAPDUMP_ERR,
+ /* Add new entry before this */
+ BNXT_RE_SNAPDUMP_ALL
+};
+
struct bnxt_re_dev {
struct ib_device ibdev;
struct list_head list;
@@ -232,6 +293,9 @@ struct bnxt_re_dev {
unsigned long event_bitmap;
struct bnxt_qplib_cc_param cc_param;
struct workqueue_struct *dcb_wq;
+ /* Head to track all QP dump */
+ struct bnxt_re_qdump_head qdump_head;
+ u8 snapdump_dbg_lvl;
struct dentry *cc_config;
struct bnxt_re_dbg_cc_config_params *cc_config_params;
};
diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c
index e9e4da4..87fdf69 100644
--- a/drivers/infiniband/hw/bnxt_re/main.c
+++ b/drivers/infiniband/hw/bnxt_re/main.c
@@ -2008,6 +2008,11 @@ static void bnxt_re_free_nqr_mem(struct bnxt_re_dev *rdev)
rdev->nqr = NULL;
}
+static void bnxt_re_clean_qdump(struct bnxt_re_dev *rdev)
+{
+ vfree(rdev->qdump_head.qdump);
+}
+
static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev, u8 op_type)
{
u8 type;
@@ -2018,6 +2023,7 @@ static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev, u8 op_type)
bnxt_re_net_unregister_async_event(rdev);
bnxt_re_uninit_dcb_wq(rdev);
+ bnxt_re_clean_qdump(rdev);
if (test_and_clear_bit(BNXT_RE_FLAG_QOS_WORK_REG, &rdev->flags))
cancel_delayed_work_sync(&rdev->worker);
@@ -2063,6 +2069,16 @@ static void bnxt_re_worker(struct work_struct *work)
schedule_delayed_work(&rdev->worker, msecs_to_jiffies(30000));
}
+static void bnxt_re_init_qdump(struct bnxt_re_dev *rdev)
+{
+ rdev->qdump_head.max_elements = BNXT_RE_MAX_QDUMP_ENTRIES;
+ rdev->qdump_head.index = 0;
+ rdev->snapdump_dbg_lvl = BNXT_RE_SNAPDUMP_ERR;
+ mutex_init(&rdev->qdump_head.lock);
+ rdev->qdump_head.qdump = vzalloc(rdev->qdump_head.max_elements *
+ sizeof(struct qdump_array));
+}
+
static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type)
{
struct bnxt_re_ring_attr rattr = {};
@@ -2235,6 +2251,8 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type)
hash_init(rdev->srq_hash);
bnxt_re_debugfs_add_pdev(rdev);
+ if (bnxt_qplib_is_chip_gen_p5_p7(rdev->chip_ctx))
+ bnxt_re_init_qdump(rdev);
bnxt_re_init_dcb_wq(rdev);
bnxt_re_net_register_async_event(rdev);
--
2.5.5
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH rdma-next 2/9] RDMA/bnxt_re: Cache the QP information
2025-02-20 18:34 [PATCH rdma-next 0/9] RDMA/bnxt_re: Driver Debug Enhancements Selvin Xavier
2025-02-20 18:34 ` [PATCH rdma-next 1/9] RDMA/bnxt_re: Add support for collecting the Queue dumps Selvin Xavier
@ 2025-02-20 18:34 ` Selvin Xavier
2025-02-20 18:34 ` [PATCH rdma-next 3/9] RDMA/bnxt_re : Initialize the HW context dump collection Selvin Xavier
` (7 subsequent siblings)
9 siblings, 0 replies; 15+ messages in thread
From: Selvin Xavier @ 2025-02-20 18:34 UTC (permalink / raw)
To: leon, jgg
Cc: linux-rdma, andrew.gospodarek, kalesh-anakkur.purayil, netdev,
davem, edumazet, kuba, abeni, horms, michael.chan,
Saravanan Vajravel, Selvin Xavier
From: Saravanan Vajravel <saravanan.vajravel@broadcom.com>
Add routines to cache the information about the QP before
destroying. This can be retrieved and used for further
debugging.
The default behavior is to capture the QPs that are moved
to error. The driver sets the flags to capture the dump
from the error asynchronous event.
Signed-off-by: Saravanan Vajravel <saravanan.vajravel@broadcom.com>
Signed-off-by: Selvin Xavier <selvin.xavier@broadcom.com>
---
drivers/infiniband/hw/bnxt_re/ib_verbs.c | 67 ++++++++++++++++++++++++++++++++
drivers/infiniband/hw/bnxt_re/ib_verbs.h | 42 ++++++++++++++++++++
drivers/infiniband/hw/bnxt_re/main.c | 8 +++-
drivers/infiniband/hw/bnxt_re/qplib_fp.h | 7 ++++
4 files changed, 123 insertions(+), 1 deletion(-)
diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c
index 2de101d..060143e 100644
--- a/drivers/infiniband/hw/bnxt_re/ib_verbs.c
+++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c
@@ -41,6 +41,7 @@
#include <linux/pci.h>
#include <linux/netdevice.h>
#include <linux/if_ether.h>
+#include <linux/vmalloc.h>
#include <net/addrconf.h>
#include <rdma/ib_verbs.h>
@@ -954,6 +955,71 @@ static int bnxt_re_destroy_gsi_sqp(struct bnxt_re_qp *qp)
return rc;
}
+static struct qdump_array *bnxt_re_get_next_qpdump(struct bnxt_re_dev *rdev)
+{
+ struct qdump_array *qdump;
+ u32 index;
+
+ index = rdev->qdump_head.index;
+ qdump = &rdev->qdump_head.qdump[index];
+ memset(qdump, 0, sizeof(*qdump));
+
+ index++;
+ index %= rdev->qdump_head.max_elements;
+ rdev->qdump_head.index = index;
+
+ return qdump;
+}
+
+/*
+ * bnxt_re_capture_qpdump - Capture snapshot of various queues of a QP.
+ * @qp - Pointer to QP for which data has to be collected
+ *
+ * This function will capture info about SQ/RQ/SCQ/RCQ of a QP which
+ * can be used to debug any issue
+ *
+ */
+void bnxt_re_capture_qpdump(struct bnxt_re_qp *qp)
+{
+ struct bnxt_qplib_qp *qpl = &qp->qplib_qp;
+ struct bnxt_re_dev *rdev = qp->rdev;
+ struct qdump_qpinfo *qpinfo;
+ struct qdump_array *qdump;
+ bool capture_snapdump;
+
+ if (rdev->snapdump_dbg_lvl == BNXT_RE_SNAPDUMP_NONE)
+ return;
+
+ capture_snapdump = test_bit(QP_FLAGS_CAPTURE_SNAPDUMP, &qpl->flags);
+ if (rdev->snapdump_dbg_lvl == BNXT_RE_SNAPDUMP_ERR &&
+ !capture_snapdump)
+ return;
+
+ if (qp->is_snapdump_captured || !rdev->qdump_head.qdump)
+ return;
+
+ mutex_lock(&rdev->qdump_head.lock);
+ qdump = bnxt_re_get_next_qpdump(rdev);
+
+ qpinfo = &qdump->qpinfo;
+ qpinfo->id = qpl->id;
+ qpinfo->dest_qpid = qpl->dest_qpn;
+ qpinfo->is_user = qpl->is_user;
+ qpinfo->mtu = qpl->mtu;
+ qpinfo->state = qpl->state;
+ qpinfo->type = qpl->type;
+ qpinfo->wqe_mode = qpl->wqe_mode;
+ qpinfo->qp_handle = qpl->qp_handle;
+ qpinfo->scq_handle = qp->scq->qplib_cq.cq_handle;
+ qpinfo->rcq_handle = qp->rcq->qplib_cq.cq_handle;
+ qpinfo->scq_id = qp->scq->qplib_cq.id;
+ qpinfo->rcq_id = qp->rcq->qplib_cq.id;
+
+ qdump->valid = true;
+ qp->is_snapdump_captured = true;
+ mutex_unlock(&rdev->qdump_head.lock);
+}
+
/* Queue Pairs */
int bnxt_re_destroy_qp(struct ib_qp *ib_qp, struct ib_udata *udata)
{
@@ -965,6 +1031,7 @@ int bnxt_re_destroy_qp(struct ib_qp *ib_qp, struct ib_udata *udata)
unsigned int flags;
int rc;
+ bnxt_re_capture_qpdump(qp);
bnxt_re_debug_rem_qpinfo(rdev, qp);
bnxt_qplib_flush_cqn_wq(&qp->qplib_qp);
diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.h b/drivers/infiniband/hw/bnxt_re/ib_verbs.h
index fbb16a4..8d82066 100644
--- a/drivers/infiniband/hw/bnxt_re/ib_verbs.h
+++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.h
@@ -96,6 +96,7 @@ struct bnxt_re_qp {
struct bnxt_re_cq *scq;
struct bnxt_re_cq *rcq;
struct dentry *dentry;
+ bool is_snapdump_captured;
};
struct bnxt_re_cq {
@@ -113,6 +114,7 @@ struct bnxt_re_cq {
int resize_cqe;
void *uctx_cq_page;
struct hlist_node hash_entry;
+ bool is_snapdump_captured;
};
struct bnxt_re_mr {
@@ -267,6 +269,46 @@ int bnxt_re_alloc_ucontext(struct ib_ucontext *ctx, struct ib_udata *udata);
void bnxt_re_dealloc_ucontext(struct ib_ucontext *context);
int bnxt_re_mmap(struct ib_ucontext *context, struct vm_area_struct *vma);
void bnxt_re_mmap_free(struct rdma_user_mmap_entry *rdma_entry);
+void bnxt_re_capture_qpdump(struct bnxt_re_qp *qp);
+
+static inline const char *__to_qp_type_str(u8 type)
+{
+ switch (type) {
+ case CMDQ_CREATE_QP1_TYPE_GSI:
+ case CMDQ_CREATE_QP_TYPE_GSI:
+ return "GSI";
+ case CMDQ_CREATE_QP_TYPE_RC:
+ return "RC";
+ case CMDQ_CREATE_QP_TYPE_UD:
+ return "UD";
+ case CMDQ_CREATE_QP_TYPE_RAW_ETHERTYPE:
+ return "RAW_ETH";
+ default:
+ return "NotSupp";
+ }
+}
+
+static inline const char *__to_qp_state_str(u8 state)
+{
+ switch (state) {
+ case CMDQ_MODIFY_QP_NEW_STATE_RESET:
+ return "RESET";
+ case CMDQ_MODIFY_QP_NEW_STATE_INIT:
+ return "INIT";
+ case CMDQ_MODIFY_QP_NEW_STATE_RTR:
+ return "RTR";
+ case CMDQ_MODIFY_QP_NEW_STATE_RTS:
+ return "RTS";
+ case CMDQ_MODIFY_QP_NEW_STATE_SQD:
+ return "SQD";
+ case CMDQ_MODIFY_QP_NEW_STATE_SQE:
+ return "SQE";
+ case CMDQ_MODIFY_QP_NEW_STATE_ERR:
+ return "ERR";
+ default:
+ return "NotSupp";
+ }
+}
static inline u32 __to_ib_port_num(u16 port_id)
{
diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c
index 87fdf69..67d2bf0 100644
--- a/drivers/infiniband/hw/bnxt_re/main.c
+++ b/drivers/infiniband/hw/bnxt_re/main.c
@@ -1420,9 +1420,15 @@ static int bnxt_re_handle_qp_async_event(struct creq_qp_event *qp_event,
struct ib_event event = {};
unsigned int flags;
- if (qp->qplib_qp.srq)
+ if (qp->qplib_qp.srq) {
srq = container_of(qp->qplib_qp.srq, struct bnxt_re_srq,
qplib_srq);
+ set_bit(SRQ_FLAGS_CAPTURE_SNAPDUMP, &srq->qplib_srq.flags);
+ }
+
+ set_bit(QP_FLAGS_CAPTURE_SNAPDUMP, &qp->qplib_qp.flags);
+ set_bit(CQ_FLAGS_CAPTURE_SNAPDUMP, &qp->scq->qplib_cq.flags);
+ set_bit(CQ_FLAGS_CAPTURE_SNAPDUMP, &qp->rcq->qplib_cq.flags);
if (qp->qplib_qp.state == CMDQ_MODIFY_QP_NEW_STATE_ERR &&
rdma_is_kernel_res(&qp->ib_qp.res)) {
diff --git a/drivers/infiniband/hw/bnxt_re/qplib_fp.h b/drivers/infiniband/hw/bnxt_re/qplib_fp.h
index 0d9487c..d1acb01 100644
--- a/drivers/infiniband/hw/bnxt_re/qplib_fp.h
+++ b/drivers/infiniband/hw/bnxt_re/qplib_fp.h
@@ -106,6 +106,8 @@ struct bnxt_qplib_srq {
u16 eventq_hw_ring_id;
spinlock_t lock; /* protect SRQE link list */
u8 toggle;
+ unsigned long flags;
+#define SRQ_FLAGS_CAPTURE_SNAPDUMP 1
};
struct bnxt_qplib_sge {
@@ -279,6 +281,8 @@ struct bnxt_qplib_qp {
u8 wqe_mode;
u8 state;
u8 cur_qp_state;
+ u16 ctx_size_sb;
+ u8 is_user;
u64 modify_flags;
u32 max_inline_data;
u32 mtu;
@@ -344,6 +348,8 @@ struct bnxt_qplib_qp {
u32 msn_tbl_sz;
bool is_host_msn_tbl;
u8 tos_dscp;
+ unsigned long flags;
+#define QP_FLAGS_CAPTURE_SNAPDUMP 1
};
#define BNXT_QPLIB_MAX_CQE_ENTRY_SIZE sizeof(struct cq_base)
@@ -448,6 +454,7 @@ struct bnxt_qplib_cq {
#define CQ_RESIZE_WAIT_TIME_MS 500
unsigned long flags;
#define CQ_FLAGS_RESIZE_IN_PROG 1
+#define CQ_FLAGS_CAPTURE_SNAPDUMP 2
wait_queue_head_t waitq;
struct list_head sqf_head, rqf_head;
atomic_t arm_state;
--
2.5.5
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH rdma-next 3/9] RDMA/bnxt_re : Initialize the HW context dump collection
2025-02-20 18:34 [PATCH rdma-next 0/9] RDMA/bnxt_re: Driver Debug Enhancements Selvin Xavier
2025-02-20 18:34 ` [PATCH rdma-next 1/9] RDMA/bnxt_re: Add support for collecting the Queue dumps Selvin Xavier
2025-02-20 18:34 ` [PATCH rdma-next 2/9] RDMA/bnxt_re: Cache the QP information Selvin Xavier
@ 2025-02-20 18:34 ` Selvin Xavier
2025-02-20 18:34 ` [PATCH rdma-next 4/9] RDMA/bnxt_re: Get the resource contexts from the HW Selvin Xavier
` (6 subsequent siblings)
9 siblings, 0 replies; 15+ messages in thread
From: Selvin Xavier @ 2025-02-20 18:34 UTC (permalink / raw)
To: leon, jgg
Cc: linux-rdma, andrew.gospodarek, kalesh-anakkur.purayil, netdev,
davem, edumazet, kuba, abeni, horms, michael.chan, Kashyap Desai,
Selvin Xavier
From: Kashyap Desai <kashyap.desai@broadcom.com>
HW context of the destroyed resources are returned during
the destroy operation of the resource. Initialize the
data structures to collect the HW contexts. Check if the
FW supports the context dump and initialize the FW to
send the dump to host.
Signed-off-by: Kashyap Desai <kashyap.desai@broadcom.com>
Signed-off-by: Selvin Xavier <selvin.xavier@broadcom.com>
---
drivers/infiniband/hw/bnxt_re/main.c | 32 ++++++++++++++++++++++++++++++
drivers/infiniband/hw/bnxt_re/qplib_rcfw.c | 2 ++
drivers/infiniband/hw/bnxt_re/qplib_rcfw.h | 12 +++++++++++
drivers/infiniband/hw/bnxt_re/qplib_res.h | 14 +++++++++++++
drivers/infiniband/hw/bnxt_re/roce_hsi.h | 2 ++
5 files changed, 62 insertions(+)
diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c
index 67d2bf0..6b5a169 100644
--- a/drivers/infiniband/hw/bnxt_re/main.c
+++ b/drivers/infiniband/hw/bnxt_re/main.c
@@ -2016,6 +2016,12 @@ static void bnxt_re_free_nqr_mem(struct bnxt_re_dev *rdev)
static void bnxt_re_clean_qdump(struct bnxt_re_dev *rdev)
{
+ struct bnxt_qplib_rcfw *rcfw = &rdev->rcfw;
+
+ vfree(rcfw->qp_ctxm_data);
+ vfree(rcfw->cq_ctxm_data);
+ vfree(rcfw->srq_ctxm_data);
+ vfree(rcfw->mrw_ctxm_data);
vfree(rdev->qdump_head.qdump);
}
@@ -2075,14 +2081,40 @@ static void bnxt_re_worker(struct work_struct *work)
schedule_delayed_work(&rdev->worker, msecs_to_jiffies(30000));
}
+static void bnxt_re_init_ctxm_size(struct bnxt_re_dev *rdev)
+{
+ struct bnxt_qplib_rcfw *rcfw = &rdev->rcfw;
+
+ if (bnxt_qplib_is_chip_gen_p7(rdev->chip_ctx)) {
+ rcfw->qp_ctxm_size = BNXT_RE_CONTEXT_TYPE_QPC_SIZE_P7;
+ rcfw->cq_ctxm_size = BNXT_RE_CONTEXT_TYPE_CQ_SIZE_P7;
+ rcfw->srq_ctxm_size = BNXT_RE_CONTEXT_TYPE_SRQ_SIZE_P7;
+ rcfw->mrw_ctxm_size = BNXT_RE_CONTEXT_TYPE_MRW_SIZE_P7;
+
+ } else if (bnxt_qplib_is_chip_gen_p5(rdev->chip_ctx)) {
+ rcfw->qp_ctxm_size = BNXT_RE_CONTEXT_TYPE_QPC_SIZE_P5;
+ rcfw->cq_ctxm_size = BNXT_RE_CONTEXT_TYPE_CQ_SIZE_P5;
+ rcfw->srq_ctxm_size = BNXT_RE_CONTEXT_TYPE_SRQ_SIZE_P5;
+ rcfw->mrw_ctxm_size = BNXT_RE_CONTEXT_TYPE_MRW_SIZE_P5;
+ }
+}
+
static void bnxt_re_init_qdump(struct bnxt_re_dev *rdev)
{
+ struct bnxt_qplib_rcfw *rcfw = &rdev->rcfw;
+
rdev->qdump_head.max_elements = BNXT_RE_MAX_QDUMP_ENTRIES;
rdev->qdump_head.index = 0;
rdev->snapdump_dbg_lvl = BNXT_RE_SNAPDUMP_ERR;
mutex_init(&rdev->qdump_head.lock);
rdev->qdump_head.qdump = vzalloc(rdev->qdump_head.max_elements *
sizeof(struct qdump_array));
+ /* Setup Context cache information */
+ bnxt_re_init_ctxm_size(rdev);
+ rcfw->qp_ctxm_data = vzalloc(BNXT_RE_MAX_QDUMP_ENTRIES * rcfw->qp_ctxm_size);
+ rcfw->cq_ctxm_data = vzalloc(BNXT_RE_MAX_QDUMP_ENTRIES * rcfw->cq_ctxm_size);
+ rcfw->srq_ctxm_data = vzalloc(BNXT_RE_MAX_QDUMP_ENTRIES * rcfw->srq_ctxm_size);
+ rcfw->mrw_ctxm_data = vzalloc(BNXT_RE_MAX_QDUMP_ENTRIES * rcfw->mrw_ctxm_size);
}
static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type)
diff --git a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c
index 17e62f2..435b01f 100644
--- a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c
+++ b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c
@@ -903,6 +903,8 @@ int bnxt_qplib_init_rcfw(struct bnxt_qplib_rcfw *rcfw,
flags |= CMDQ_INITIALIZE_FW_FLAGS_OPTIMIZE_MODIFY_QP_SUPPORTED;
if (rcfw->res->en_dev->flags & BNXT_EN_FLAG_ROCE_VF_RES_MGMT)
flags |= CMDQ_INITIALIZE_FW_FLAGS_L2_VF_RESOURCE_MGMT;
+ if (is_roce_context_destroy_sb_enabled(rcfw->res->dattr->dev_cap_flags2))
+ flags |= CMDQ_INITIALIZE_FW_FLAGS_DESTROY_CONTEXT_SB_SUPPORTED;
req.flags |= cpu_to_le16(flags);
req.stat_ctx_id = cpu_to_le32(ctx->stats.fw_id);
bnxt_qplib_fill_cmdqmsg(&msg, &req, &resp, NULL, sizeof(req), sizeof(resp), 0);
diff --git a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h
index 88814cb..155e24f 100644
--- a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h
+++ b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h
@@ -228,6 +228,18 @@ struct bnxt_qplib_rcfw {
struct bnxt_qplib_qp_node *qp_tbl;
/* To synchronize the qp-handle hash table */
spinlock_t tbl_lock;
+ u32 qp_ctxm_data_index;
+ u32 cq_ctxm_data_index;
+ u32 mrw_ctxm_data_index;
+ u32 srq_ctxm_data_index;
+ u16 qp_ctxm_size;
+ u16 cq_ctxm_size;
+ u16 mrw_ctxm_size;
+ u16 srq_ctxm_size;
+ void *qp_ctxm_data;
+ void *cq_ctxm_data;
+ void *mrw_ctxm_data;
+ void *srq_ctxm_data;
u64 oos_prev;
u32 init_oos_stats;
u32 cmdq_depth;
diff --git a/drivers/infiniband/hw/bnxt_re/qplib_res.h b/drivers/infiniband/hw/bnxt_re/qplib_res.h
index be5d907..93200ea 100644
--- a/drivers/infiniband/hw/bnxt_re/qplib_res.h
+++ b/drivers/infiniband/hw/bnxt_re/qplib_res.h
@@ -592,4 +592,18 @@ static inline bool _is_cq_coalescing_supported(u16 dev_cap_ext_flags2)
return dev_cap_ext_flags2 & CREQ_QUERY_FUNC_RESP_SB_CQ_COALESCING_SUPPORTED;
}
+static inline bool is_roce_context_destroy_sb_enabled(u16 dev_cap_ext_flags2)
+{
+ return !!(dev_cap_ext_flags2 & CREQ_QUERY_FUNC_RESP_SB_DESTROY_CONTEXT_SB_SUPPORTED);
+}
+
+#define BNXT_RE_CONTEXT_TYPE_QPC_SIZE_P5 1088
+#define BNXT_RE_CONTEXT_TYPE_CQ_SIZE_P5 128
+#define BNXT_RE_CONTEXT_TYPE_MRW_SIZE_P5 128
+#define BNXT_RE_CONTEXT_TYPE_SRQ_SIZE_P5 192
+
+#define BNXT_RE_CONTEXT_TYPE_QPC_SIZE_P7 1088
+#define BNXT_RE_CONTEXT_TYPE_CQ_SIZE_P7 192
+#define BNXT_RE_CONTEXT_TYPE_MRW_SIZE_P7 192
+#define BNXT_RE_CONTEXT_TYPE_SRQ_SIZE_P7 192
#endif /* __BNXT_QPLIB_RES_H__ */
diff --git a/drivers/infiniband/hw/bnxt_re/roce_hsi.h b/drivers/infiniband/hw/bnxt_re/roce_hsi.h
index 0ee60fd..041e9ac 100644
--- a/drivers/infiniband/hw/bnxt_re/roce_hsi.h
+++ b/drivers/infiniband/hw/bnxt_re/roce_hsi.h
@@ -218,6 +218,7 @@ struct cmdq_initialize_fw {
#define CMDQ_INITIALIZE_FW_FLAGS_HW_REQUESTER_RETX_SUPPORTED 0x2UL
#define CMDQ_INITIALIZE_FW_FLAGS_OPTIMIZE_MODIFY_QP_SUPPORTED 0x8UL
#define CMDQ_INITIALIZE_FW_FLAGS_L2_VF_RESOURCE_MGMT 0x10UL
+ #define CMDQ_INITIALIZE_FW_FLAGS_DESTROY_CONTEXT_SB_SUPPORTED 0x20UL
__le16 cookie;
u8 resp_size;
u8 reserved8;
@@ -2215,6 +2216,7 @@ struct creq_query_func_resp_sb {
#define CREQ_QUERY_FUNC_RESP_SB_REQ_RETRANSMISSION_SUPPORT_IQM_MSN_TABLE (0x2UL << 4)
#define CREQ_QUERY_FUNC_RESP_SB_REQ_RETRANSMISSION_SUPPORT_LAST \
CREQ_QUERY_FUNC_RESP_SB_REQ_RETRANSMISSION_SUPPORT_IQM_MSN_TABLE
+ #define CREQ_QUERY_FUNC_RESP_SB_DESTROY_CONTEXT_SB_SUPPORTED 0x100UL
#define CREQ_QUERY_FUNC_RESP_SB_MIN_RNR_RTR_RTS_OPT_SUPPORTED 0x1000UL
__le16 max_xp_qp_size;
__le16 create_qp_batch_size;
--
2.5.5
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH rdma-next 4/9] RDMA/bnxt_re: Get the resource contexts from the HW
2025-02-20 18:34 [PATCH rdma-next 0/9] RDMA/bnxt_re: Driver Debug Enhancements Selvin Xavier
` (2 preceding siblings ...)
2025-02-20 18:34 ` [PATCH rdma-next 3/9] RDMA/bnxt_re : Initialize the HW context dump collection Selvin Xavier
@ 2025-02-20 18:34 ` Selvin Xavier
2025-02-20 18:34 ` [PATCH rdma-next 5/9] bnxt_en: Introduce ULP coredump callbacks Selvin Xavier
` (5 subsequent siblings)
9 siblings, 0 replies; 15+ messages in thread
From: Selvin Xavier @ 2025-02-20 18:34 UTC (permalink / raw)
To: leon, jgg
Cc: linux-rdma, andrew.gospodarek, kalesh-anakkur.purayil, netdev,
davem, edumazet, kuba, abeni, horms, michael.chan, Kashyap Desai,
Selvin Xavier
From: Kashyap Desai <kashyap.desai@broadcom.com>
While destroying the resources like QP/CQ/SRQ/MRs, FW sends
the contexts to the host. Driver caches these information
to be dumped later for debugging. Driver can hold the information
of the last 1024 entries of each resources.
Signed-off-by: Kashyap Desai <kashyap.desai@broadcom.com>
Signed-off-by: Selvin Xavier <selvin.xavier@broadcom.com>
---
drivers/infiniband/hw/bnxt_re/bnxt_re.h | 9 +++
drivers/infiniband/hw/bnxt_re/ib_verbs.c | 120 +++++++++++++++++++++++++++----
drivers/infiniband/hw/bnxt_re/qplib_fp.c | 47 +++++++++++-
drivers/infiniband/hw/bnxt_re/qplib_fp.h | 9 ++-
drivers/infiniband/hw/bnxt_re/qplib_sp.c | 15 +++-
drivers/infiniband/hw/bnxt_re/qplib_sp.h | 3 +-
6 files changed, 181 insertions(+), 22 deletions(-)
diff --git a/drivers/infiniband/hw/bnxt_re/bnxt_re.h b/drivers/infiniband/hw/bnxt_re/bnxt_re.h
index 5818db1..45601635 100644
--- a/drivers/infiniband/hw/bnxt_re/bnxt_re.h
+++ b/drivers/infiniband/hw/bnxt_re/bnxt_re.h
@@ -175,6 +175,15 @@ static inline bool bnxt_re_chip_gen_p7(u16 chip_num)
#define BNXT_RE_MAX_QDUMP_ENTRIES 1024
+enum {
+ BNXT_RE_RES_TYPE_CQ = 0,
+ BNXT_RE_RES_TYPE_UCTX,
+ BNXT_RE_RES_TYPE_QP,
+ BNXT_RE_RES_TYPE_SRQ,
+ BNXT_RE_RES_TYPE_MR,
+ BNXT_RE_RES_TYPE_MAX
+};
+
struct qdump_qpinfo {
u32 id;
u32 dest_qpid;
diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c
index 060143e..a1ee6ca 100644
--- a/drivers/infiniband/hw/bnxt_re/ib_verbs.c
+++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c
@@ -537,7 +537,7 @@ static void bnxt_re_destroy_fence_mr(struct bnxt_re_pd *pd)
bnxt_qplib_dereg_mrw(&rdev->qplib_res, &mr->qplib_mr,
true);
if (mr->ib_mr.lkey)
- bnxt_qplib_free_mrw(&rdev->qplib_res, &mr->qplib_mr);
+ bnxt_qplib_free_mrw(&rdev->qplib_res, &mr->qplib_mr, 0, NULL);
kfree(mr);
fence->mr = NULL;
}
@@ -886,6 +886,55 @@ int bnxt_re_query_ah(struct ib_ah *ib_ah, struct rdma_ah_attr *ah_attr)
return 0;
}
+static inline void bnxt_re_save_resource_context(struct bnxt_re_dev *rdev,
+ void *ctx_sb_data,
+ u8 res_type, bool do_snapdump)
+{
+ void *drv_ctx_data;
+ u32 *ctx_index;
+ u16 ctx_size;
+
+ if (!ctx_sb_data)
+ return;
+
+ switch (res_type) {
+ case BNXT_RE_RES_TYPE_QP:
+ drv_ctx_data = rdev->rcfw.qp_ctxm_data;
+ ctx_index = &rdev->rcfw.qp_ctxm_data_index;
+ ctx_size = rdev->rcfw.qp_ctxm_size;
+ break;
+ case BNXT_RE_RES_TYPE_CQ:
+ drv_ctx_data = rdev->rcfw.cq_ctxm_data;
+ ctx_index = &rdev->rcfw.cq_ctxm_data_index;
+ ctx_size = rdev->rcfw.cq_ctxm_size;
+ break;
+ case BNXT_RE_RES_TYPE_MR:
+ drv_ctx_data = rdev->rcfw.mrw_ctxm_data;
+ ctx_index = &rdev->rcfw.mrw_ctxm_data_index;
+ ctx_size = rdev->rcfw.mrw_ctxm_size;
+ break;
+ case BNXT_RE_RES_TYPE_SRQ:
+ drv_ctx_data = rdev->rcfw.srq_ctxm_data;
+ ctx_index = &rdev->rcfw.srq_ctxm_data_index;
+ ctx_size = rdev->rcfw.srq_ctxm_size;
+ break;
+ default:
+ return;
+ }
+
+ if (rdev->snapdump_dbg_lvl == BNXT_RE_SNAPDUMP_ALL ||
+ (rdev->snapdump_dbg_lvl == BNXT_RE_SNAPDUMP_ERR && do_snapdump)) {
+ memcpy(drv_ctx_data + (*ctx_index * ctx_size),
+ ctx_sb_data, ctx_size);
+ *ctx_index = *ctx_index + 1;
+ *ctx_index = *ctx_index % BNXT_RE_MAX_QDUMP_ENTRIES;
+ dev_dbg(rdev_to_dev(rdev),
+ "%s : res_type %d ctx_index %d 0x%lx\n", __func__,
+ res_type, *ctx_index,
+ (unsigned long)drv_ctx_data + (*ctx_index * ctx_size));
+ }
+}
+
unsigned long bnxt_re_lock_cqs(struct bnxt_re_qp *qp)
__acquires(&qp->scq->cq_lock) __acquires(&qp->rcq->cq_lock)
{
@@ -930,7 +979,7 @@ static int bnxt_re_destroy_gsi_sqp(struct bnxt_re_qp *qp)
bnxt_qplib_clean_qp(&qp->qplib_qp);
ibdev_dbg(&rdev->ibdev, "Destroy the shadow QP\n");
- rc = bnxt_qplib_destroy_qp(&rdev->qplib_res, &gsi_sqp->qplib_qp);
+ rc = bnxt_qplib_destroy_qp(&rdev->qplib_res, &gsi_sqp->qplib_qp, 0, NULL);
if (rc) {
ibdev_err(&rdev->ibdev, "Destroy Shadow QP failed");
goto fail;
@@ -1029,17 +1078,28 @@ int bnxt_re_destroy_qp(struct ib_qp *ib_qp, struct ib_udata *udata)
struct bnxt_qplib_nq *scq_nq = NULL;
struct bnxt_qplib_nq *rcq_nq = NULL;
unsigned int flags;
+ void *ctx_sb_data = NULL;
+ bool do_snapdump;
+ u16 ctx_size;
int rc;
bnxt_re_capture_qpdump(qp);
bnxt_re_debug_rem_qpinfo(rdev, qp);
bnxt_qplib_flush_cqn_wq(&qp->qplib_qp);
+ ctx_size = qplib_qp->ctx_size_sb;
+ if (ctx_size)
+ ctx_sb_data = vzalloc(ctx_size);
+ do_snapdump = test_bit(QP_FLAGS_CAPTURE_SNAPDUMP, &qplib_qp->flags);
- rc = bnxt_qplib_destroy_qp(&rdev->qplib_res, &qp->qplib_qp);
+ rc = bnxt_qplib_destroy_qp(&rdev->qplib_res, &qp->qplib_qp, ctx_size, ctx_sb_data);
if (rc)
ibdev_err(&rdev->ibdev, "Failed to destroy HW QP");
+ else
+ bnxt_re_save_resource_context(rdev, ctx_sb_data,
+ BNXT_RE_RES_TYPE_QP, do_snapdump);
+ vfree(ctx_sb_data);
if (rdma_is_kernel_res(&qp->ib_qp.res)) {
flags = bnxt_re_lock_cqs(qp);
bnxt_qplib_clean_qp(&qp->qplib_qp);
@@ -1599,7 +1659,7 @@ static int bnxt_re_create_shadow_gsi(struct bnxt_re_qp *qp,
&qp->qplib_qp);
if (!sah) {
bnxt_qplib_destroy_qp(&rdev->qplib_res,
- &sqp->qplib_qp);
+ &sqp->qplib_qp, 0, NULL);
rc = -ENODEV;
ibdev_err(&rdev->ibdev,
"Failed to create AH entry for ShadowQP");
@@ -1747,7 +1807,7 @@ int bnxt_re_create_qp(struct ib_qp *ib_qp, struct ib_qp_init_attr *qp_init_attr,
return 0;
qp_destroy:
- bnxt_qplib_destroy_qp(&rdev->qplib_res, &qp->qplib_qp);
+ bnxt_qplib_destroy_qp(&rdev->qplib_res, &qp->qplib_qp, 0, NULL);
free_umem:
ib_umem_release(qp->rumem);
ib_umem_release(qp->sumem);
@@ -1841,6 +1901,9 @@ int bnxt_re_destroy_srq(struct ib_srq *ib_srq, struct ib_udata *udata)
struct bnxt_re_dev *rdev = srq->rdev;
struct bnxt_qplib_srq *qplib_srq = &srq->qplib_srq;
struct bnxt_qplib_nq *nq = NULL;
+ void *ctx_sb_data = NULL;
+ bool do_snapdump;
+ u16 ctx_size;
if (qplib_srq->cq)
nq = qplib_srq->cq->nq;
@@ -1848,7 +1911,18 @@ int bnxt_re_destroy_srq(struct ib_srq *ib_srq, struct ib_udata *udata)
free_page((unsigned long)srq->uctx_srq_page);
hash_del(&srq->hash_entry);
}
- bnxt_qplib_destroy_srq(&rdev->qplib_res, qplib_srq);
+ ctx_size = rdev->rcfw.srq_ctxm_size;
+ if (ctx_size)
+ ctx_sb_data = vzalloc(ctx_size);
+
+ bnxt_qplib_destroy_srq(&rdev->qplib_res, qplib_srq, ctx_size, ctx_sb_data);
+
+ do_snapdump = test_bit(SRQ_FLAGS_CAPTURE_SNAPDUMP, &qplib_srq->flags);
+ bnxt_re_save_resource_context(rdev, ctx_sb_data,
+ BNXT_RE_RES_TYPE_SRQ, do_snapdump);
+
+ vfree(ctx_sb_data);
+
ib_umem_release(srq->umem);
atomic_dec(&rdev->stats.res.srq_count);
if (nq)
@@ -1968,7 +2042,7 @@ int bnxt_re_create_srq(struct ib_srq *ib_srq,
if (rc) {
ibdev_err(&rdev->ibdev, "SRQ copy to udata failed!");
bnxt_qplib_destroy_srq(&rdev->qplib_res,
- &srq->qplib_srq);
+ &srq->qplib_srq, 0, NULL);
goto fail;
}
}
@@ -3127,6 +3201,9 @@ int bnxt_re_destroy_cq(struct ib_cq *ib_cq, struct ib_udata *udata)
struct bnxt_qplib_nq *nq;
struct bnxt_re_dev *rdev;
struct bnxt_re_cq *cq;
+ void *ctx_sb_data = NULL;
+ bool do_snapdump;
+ u16 ctx_size;
cq = container_of(ib_cq, struct bnxt_re_cq, ib_cq);
rdev = cq->rdev;
@@ -3137,7 +3214,15 @@ int bnxt_re_destroy_cq(struct ib_cq *ib_cq, struct ib_udata *udata)
free_page((unsigned long)cq->uctx_cq_page);
hash_del(&cq->hash_entry);
}
- bnxt_qplib_destroy_cq(&rdev->qplib_res, &cq->qplib_cq);
+ ctx_size = rdev->rcfw.cq_ctxm_size;
+ if (ctx_size)
+ ctx_sb_data = vzalloc(ctx_size);
+ bnxt_qplib_destroy_cq(&rdev->qplib_res, &cq->qplib_cq, ctx_size, ctx_sb_data);
+ do_snapdump = test_bit(CQ_FLAGS_CAPTURE_SNAPDUMP, &cq->qplib_cq.flags);
+ bnxt_re_save_resource_context(rdev, ctx_sb_data,
+ BNXT_RE_RES_TYPE_CQ, do_snapdump);
+
+ vfree(ctx_sb_data);
bnxt_re_put_nq(rdev, nq);
ib_umem_release(cq->umem);
@@ -3247,7 +3332,7 @@ int bnxt_re_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
rc = ib_copy_to_udata(udata, &resp, min(sizeof(resp), udata->outlen));
if (rc) {
ibdev_err(&rdev->ibdev, "Failed to copy CQ udata");
- bnxt_qplib_destroy_cq(&rdev->qplib_res, &cq->qplib_cq);
+ bnxt_qplib_destroy_cq(&rdev->qplib_res, &cq->qplib_cq, 0, NULL);
goto free_mem;
}
}
@@ -4070,7 +4155,7 @@ struct ib_mr *bnxt_re_get_dma_mr(struct ib_pd *ib_pd, int mr_access_flags)
return &mr->ib_mr;
fail_mr:
- bnxt_qplib_free_mrw(&rdev->qplib_res, &mr->qplib_mr);
+ bnxt_qplib_free_mrw(&rdev->qplib_res, &mr->qplib_mr, 0, NULL);
fail:
kfree(mr);
return ERR_PTR(rc);
@@ -4080,13 +4165,20 @@ int bnxt_re_dereg_mr(struct ib_mr *ib_mr, struct ib_udata *udata)
{
struct bnxt_re_mr *mr = container_of(ib_mr, struct bnxt_re_mr, ib_mr);
struct bnxt_re_dev *rdev = mr->rdev;
+ void *ctx_sb_data = NULL;
+ u16 ctx_size;
int rc;
- rc = bnxt_qplib_free_mrw(&rdev->qplib_res, &mr->qplib_mr);
+ ctx_size = rdev->rcfw.mrw_ctxm_size;
+ if (ctx_size)
+ ctx_sb_data = vzalloc(ctx_size);
+ rc = bnxt_qplib_free_mrw(&rdev->qplib_res, &mr->qplib_mr, ctx_size, ctx_sb_data);
if (rc) {
ibdev_err(&rdev->ibdev, "Dereg MR failed: %#x\n", rc);
return rc;
}
+ bnxt_re_save_resource_context(rdev, ctx_sb_data, BNXT_RE_RES_TYPE_MR, 0);
+ vfree(ctx_sb_data);
if (mr->pages) {
rc = bnxt_qplib_free_fast_reg_page_list(&rdev->qplib_res,
@@ -4175,7 +4267,7 @@ struct ib_mr *bnxt_re_alloc_mr(struct ib_pd *ib_pd, enum ib_mr_type type,
fail_mr:
kfree(mr->pages);
fail:
- bnxt_qplib_free_mrw(&rdev->qplib_res, &mr->qplib_mr);
+ bnxt_qplib_free_mrw(&rdev->qplib_res, &mr->qplib_mr, 0, NULL);
bail:
kfree(mr);
return ERR_PTR(rc);
@@ -4222,7 +4314,7 @@ int bnxt_re_dealloc_mw(struct ib_mw *ib_mw)
struct bnxt_re_dev *rdev = mw->rdev;
int rc;
- rc = bnxt_qplib_free_mrw(&rdev->qplib_res, &mw->qplib_mw);
+ rc = bnxt_qplib_free_mrw(&rdev->qplib_res, &mw->qplib_mw, 0, NULL);
if (rc) {
ibdev_err(&rdev->ibdev, "Free MW failed: %#x\n", rc);
return rc;
@@ -4301,7 +4393,7 @@ static struct ib_mr *__bnxt_re_user_reg_mr(struct ib_pd *ib_pd, u64 length, u64
return &mr->ib_mr;
free_mrw:
- bnxt_qplib_free_mrw(&rdev->qplib_res, &mr->qplib_mr);
+ bnxt_qplib_free_mrw(&rdev->qplib_res, &mr->qplib_mr, 0, NULL);
free_mr:
kfree(mr);
return ERR_PTR(rc);
diff --git a/drivers/infiniband/hw/bnxt_re/qplib_fp.c b/drivers/infiniband/hw/bnxt_re/qplib_fp.c
index 5336f74..fa221a7 100644
--- a/drivers/infiniband/hw/bnxt_re/qplib_fp.c
+++ b/drivers/infiniband/hw/bnxt_re/qplib_fp.c
@@ -616,14 +616,21 @@ int bnxt_qplib_alloc_nq(struct bnxt_qplib_res *res, struct bnxt_qplib_nq *nq)
/* SRQ */
void bnxt_qplib_destroy_srq(struct bnxt_qplib_res *res,
- struct bnxt_qplib_srq *srq)
+ struct bnxt_qplib_srq *srq, u16 sb_resp_size, void *sb_resp_va)
{
struct bnxt_qplib_rcfw *rcfw = res->rcfw;
struct creq_destroy_srq_resp resp = {};
+ struct bnxt_qplib_rcfw_sbuf sbuf = {};
struct bnxt_qplib_cmdqmsg msg = {};
struct cmdq_destroy_srq req = {};
int rc;
+ if (sb_resp_size && sb_resp_va) {
+ sbuf.size = sb_resp_size;
+ sbuf.sb = dma_alloc_coherent(&rcfw->pdev->dev, sbuf.size,
+ &sbuf.dma_addr, GFP_KERNEL);
+ }
+
bnxt_qplib_rcfw_cmd_prep((struct cmdq_base *)&req,
CMDQ_BASE_OPCODE_DESTROY_SRQ,
sizeof(req));
@@ -632,7 +639,13 @@ void bnxt_qplib_destroy_srq(struct bnxt_qplib_res *res,
req.srq_cid = cpu_to_le32(srq->id);
bnxt_qplib_fill_cmdqmsg(&msg, &req, &resp, NULL, sizeof(req), sizeof(resp), 0);
+ req.resp_addr = cpu_to_le64(sbuf.dma_addr);
+ req.resp_size = sb_resp_size / BNXT_QPLIB_CMDQE_UNITS;
rc = bnxt_qplib_rcfw_send_message(rcfw, &msg);
+ if (sbuf.sb) {
+ memcpy(sb_resp_va, sbuf.sb, sb_resp_size);
+ dma_free_coherent(&rcfw->pdev->dev, sbuf.size, sbuf.sb, sbuf.dma_addr);
+ }
kfree(srq->swq);
if (rc)
return;
@@ -1593,15 +1606,22 @@ static void __clean_cq(struct bnxt_qplib_cq *cq, u64 qp)
}
int bnxt_qplib_destroy_qp(struct bnxt_qplib_res *res,
- struct bnxt_qplib_qp *qp)
+ struct bnxt_qplib_qp *qp,
+ u16 sb_resp_size, void *sb_resp_va)
{
struct bnxt_qplib_rcfw *rcfw = res->rcfw;
+ struct bnxt_qplib_rcfw_sbuf sbuf = {};
struct creq_destroy_qp_resp resp = {};
struct bnxt_qplib_cmdqmsg msg = {};
struct cmdq_destroy_qp req = {};
u32 tbl_indx;
int rc;
+ if (sb_resp_size && sb_resp_va) {
+ sbuf.size = sb_resp_size;
+ sbuf.sb = dma_alloc_coherent(&rcfw->pdev->dev, sbuf.size,
+ &sbuf.dma_addr, GFP_KERNEL);
+ }
spin_lock_bh(&rcfw->tbl_lock);
tbl_indx = map_qp_id_to_tbl_indx(qp->id, rcfw);
rcfw->qp_tbl[tbl_indx].qp_id = BNXT_QPLIB_QP_ID_INVALID;
@@ -1613,6 +1633,8 @@ int bnxt_qplib_destroy_qp(struct bnxt_qplib_res *res,
sizeof(req));
req.qp_cid = cpu_to_le32(qp->id);
+ req.resp_addr = cpu_to_le64(sbuf.dma_addr);
+ req.resp_size = sb_resp_size / BNXT_QPLIB_CMDQE_UNITS;
bnxt_qplib_fill_cmdqmsg(&msg, &req, &resp, NULL, sizeof(req),
sizeof(resp), 0);
rc = bnxt_qplib_rcfw_send_message(rcfw, &msg);
@@ -1623,6 +1645,10 @@ int bnxt_qplib_destroy_qp(struct bnxt_qplib_res *res,
spin_unlock_bh(&rcfw->tbl_lock);
return rc;
}
+ if (sbuf.sb) {
+ memcpy(sb_resp_va, sbuf.sb, sb_resp_size);
+ dma_free_coherent(&rcfw->pdev->dev, sbuf.size, sbuf.sb, sbuf.dma_addr);
+ }
return 0;
}
@@ -2355,25 +2381,40 @@ int bnxt_qplib_resize_cq(struct bnxt_qplib_res *res, struct bnxt_qplib_cq *cq,
return rc;
}
-int bnxt_qplib_destroy_cq(struct bnxt_qplib_res *res, struct bnxt_qplib_cq *cq)
+int bnxt_qplib_destroy_cq(struct bnxt_qplib_res *res, struct bnxt_qplib_cq *cq,
+ u16 sb_resp_size, void *sb_resp_va)
{
struct bnxt_qplib_rcfw *rcfw = res->rcfw;
+ struct bnxt_qplib_rcfw_sbuf sbuf = {};
struct creq_destroy_cq_resp resp = {};
struct bnxt_qplib_cmdqmsg msg = {};
struct cmdq_destroy_cq req = {};
u16 total_cnq_events;
int rc;
+ if (sb_resp_size && sb_resp_va) {
+ sbuf.size = sb_resp_size;
+ sbuf.sb = dma_alloc_coherent(&rcfw->pdev->dev, sbuf.size,
+ &sbuf.dma_addr, GFP_KERNEL);
+ }
+
bnxt_qplib_rcfw_cmd_prep((struct cmdq_base *)&req,
CMDQ_BASE_OPCODE_DESTROY_CQ,
sizeof(req));
+ req.resp_addr = cpu_to_le64(sbuf.dma_addr);
+ req.resp_size = sb_resp_size / BNXT_QPLIB_CMDQE_UNITS;
+
req.cq_cid = cpu_to_le32(cq->id);
bnxt_qplib_fill_cmdqmsg(&msg, &req, &resp, NULL, sizeof(req),
sizeof(resp), 0);
rc = bnxt_qplib_rcfw_send_message(rcfw, &msg);
if (rc)
return rc;
+ if (sbuf.sb) {
+ memcpy(sb_resp_va, sbuf.sb, sb_resp_size);
+ dma_free_coherent(&rcfw->pdev->dev, sbuf.size, sbuf.sb, sbuf.dma_addr);
+ }
total_cnq_events = le16_to_cpu(resp.total_cnq_events);
__wait_for_all_nqes(cq, total_cnq_events);
bnxt_qplib_free_hwq(res, &cq->hwq);
diff --git a/drivers/infiniband/hw/bnxt_re/qplib_fp.h b/drivers/infiniband/hw/bnxt_re/qplib_fp.h
index d1acb01..1710b59 100644
--- a/drivers/infiniband/hw/bnxt_re/qplib_fp.h
+++ b/drivers/infiniband/hw/bnxt_re/qplib_fp.h
@@ -555,14 +555,16 @@ int bnxt_qplib_modify_srq(struct bnxt_qplib_res *res,
int bnxt_qplib_query_srq(struct bnxt_qplib_res *res,
struct bnxt_qplib_srq *srq);
void bnxt_qplib_destroy_srq(struct bnxt_qplib_res *res,
- struct bnxt_qplib_srq *srq);
+ struct bnxt_qplib_srq *srq,
+ u16 sb_resp_size, void *sb_resp_va);
int bnxt_qplib_post_srq_recv(struct bnxt_qplib_srq *srq,
struct bnxt_qplib_swqe *wqe);
int bnxt_qplib_create_qp1(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp);
int bnxt_qplib_create_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp);
int bnxt_qplib_modify_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp);
int bnxt_qplib_query_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp);
-int bnxt_qplib_destroy_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp);
+int bnxt_qplib_destroy_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp,
+ u16 ctx_size, void *ctx_sb_data);
void bnxt_qplib_clean_qp(struct bnxt_qplib_qp *qp);
void bnxt_qplib_free_qp_res(struct bnxt_qplib_res *res,
struct bnxt_qplib_qp *qp);
@@ -584,7 +586,8 @@ int bnxt_qplib_resize_cq(struct bnxt_qplib_res *res, struct bnxt_qplib_cq *cq,
int new_cqes);
void bnxt_qplib_resize_cq_complete(struct bnxt_qplib_res *res,
struct bnxt_qplib_cq *cq);
-int bnxt_qplib_destroy_cq(struct bnxt_qplib_res *res, struct bnxt_qplib_cq *cq);
+int bnxt_qplib_destroy_cq(struct bnxt_qplib_res *res, struct bnxt_qplib_cq *cq,
+ u16 sb_resp_size, void *sb_resp_v);
int bnxt_qplib_poll_cq(struct bnxt_qplib_cq *cq, struct bnxt_qplib_cqe *cqe,
int num, struct bnxt_qplib_qp **qp);
bool bnxt_qplib_is_cq_empty(struct bnxt_qplib_cq *cq);
diff --git a/drivers/infiniband/hw/bnxt_re/qplib_sp.c b/drivers/infiniband/hw/bnxt_re/qplib_sp.c
index 4ccd440..d2c519d 100644
--- a/drivers/infiniband/hw/bnxt_re/qplib_sp.c
+++ b/drivers/infiniband/hw/bnxt_re/qplib_sp.c
@@ -504,10 +504,12 @@ int bnxt_qplib_destroy_ah(struct bnxt_qplib_res *res, struct bnxt_qplib_ah *ah,
}
/* MRW */
-int bnxt_qplib_free_mrw(struct bnxt_qplib_res *res, struct bnxt_qplib_mrw *mrw)
+int bnxt_qplib_free_mrw(struct bnxt_qplib_res *res, struct bnxt_qplib_mrw *mrw,
+ u16 sb_resp_size, void *sb_resp_va)
{
struct creq_deallocate_key_resp resp = {};
struct bnxt_qplib_rcfw *rcfw = res->rcfw;
+ struct bnxt_qplib_rcfw_sbuf sbuf = {};
struct cmdq_deallocate_key req = {};
struct bnxt_qplib_cmdqmsg msg = {};
int rc;
@@ -517,6 +519,11 @@ int bnxt_qplib_free_mrw(struct bnxt_qplib_res *res, struct bnxt_qplib_mrw *mrw)
return 0;
}
+ if (sb_resp_size && sb_resp_va) {
+ sbuf.size = sb_resp_size;
+ sbuf.sb = dma_alloc_coherent(&rcfw->pdev->dev, sbuf.size,
+ &sbuf.dma_addr, GFP_KERNEL);
+ }
bnxt_qplib_rcfw_cmd_prep((struct cmdq_base *)&req,
CMDQ_BASE_OPCODE_DEALLOCATE_KEY,
sizeof(req));
@@ -530,12 +537,18 @@ int bnxt_qplib_free_mrw(struct bnxt_qplib_res *res, struct bnxt_qplib_mrw *mrw)
else
req.key = cpu_to_le32(mrw->lkey);
+ req.resp_addr = cpu_to_le64(sbuf.dma_addr);
+ req.resp_size = sb_resp_size / BNXT_QPLIB_CMDQE_UNITS;
bnxt_qplib_fill_cmdqmsg(&msg, &req, &resp, NULL, sizeof(req),
sizeof(resp), 0);
rc = bnxt_qplib_rcfw_send_message(rcfw, &msg);
if (rc)
return rc;
+ if (sbuf.sb) {
+ memcpy(sb_resp_va, sbuf.sb, sb_resp_size);
+ dma_free_coherent(&rcfw->pdev->dev, sbuf.size, sbuf.sb, sbuf.dma_addr);
+ }
/* Free the qplib's MRW memory */
if (mrw->hwq.max_elements)
bnxt_qplib_free_hwq(res, &mrw->hwq);
diff --git a/drivers/infiniband/hw/bnxt_re/qplib_sp.h b/drivers/infiniband/hw/bnxt_re/qplib_sp.h
index e626b05..728b1f1 100644
--- a/drivers/infiniband/hw/bnxt_re/qplib_sp.h
+++ b/drivers/infiniband/hw/bnxt_re/qplib_sp.h
@@ -340,7 +340,8 @@ int bnxt_qplib_dereg_mrw(struct bnxt_qplib_res *res, struct bnxt_qplib_mrw *mrw,
bool block);
int bnxt_qplib_reg_mr(struct bnxt_qplib_res *res, struct bnxt_qplib_mrw *mr,
struct ib_umem *umem, int num_pbls, u32 buf_pg_size);
-int bnxt_qplib_free_mrw(struct bnxt_qplib_res *res, struct bnxt_qplib_mrw *mr);
+int bnxt_qplib_free_mrw(struct bnxt_qplib_res *res, struct bnxt_qplib_mrw *mr,
+ u16 ctx_size, void *ctx_sb_data);
int bnxt_qplib_alloc_fast_reg_mr(struct bnxt_qplib_res *res,
struct bnxt_qplib_mrw *mr, int max);
int bnxt_qplib_alloc_fast_reg_page_list(struct bnxt_qplib_res *res,
--
2.5.5
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH rdma-next 5/9] bnxt_en: Introduce ULP coredump callbacks
2025-02-20 18:34 [PATCH rdma-next 0/9] RDMA/bnxt_re: Driver Debug Enhancements Selvin Xavier
` (3 preceding siblings ...)
2025-02-20 18:34 ` [PATCH rdma-next 4/9] RDMA/bnxt_re: Get the resource contexts from the HW Selvin Xavier
@ 2025-02-20 18:34 ` Selvin Xavier
2025-02-22 0:40 ` Jakub Kicinski
2025-02-20 18:34 ` [PATCH rdma-next 6/9] RDMA/bnxt_re: Support the dump infrastructure Selvin Xavier
` (4 subsequent siblings)
9 siblings, 1 reply; 15+ messages in thread
From: Selvin Xavier @ 2025-02-20 18:34 UTC (permalink / raw)
To: leon, jgg
Cc: linux-rdma, andrew.gospodarek, kalesh-anakkur.purayil, netdev,
davem, edumazet, kuba, abeni, horms, michael.chan, Selvin Xavier
From: Michael Chan <michael.chan@broadcom.com>
Add .ulp_get_dump_info() and .ulp_get_dump_data() callbacks to
struct bnxt_ulp_ops. When ethtool -w is invoked to get the coredump,
these 2 callbacks to the bnxt_re auxbus driver will be called if
they are populated by the bnxt_re driver. The first callback gets
the number of coredump segments and the size of each coredump
segment. The 2nd callback copies the coredump data for each segment.
Signed-off-by: Michael Chan <michael.chan@broadcom.com>
Reviewed-by: Andy Gospodarek <andrew.gospodarek@broadcom.com>
Reviewed-by: Selvin Xavier <selvin.xavier@broadcom.com>
Signed-off-by: Selvin Xavier <selvin.xavier@broadcom.com>
---
drivers/net/ethernet/broadcom/bnxt/bnxt_coredump.c | 12 ++++-
drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c | 57 ++++++++++++++++++++++
drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h | 22 +++++++++
3 files changed, 89 insertions(+), 2 deletions(-)
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_coredump.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_coredump.c
index 7236d8e..2106d0d 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt_coredump.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_coredump.c
@@ -14,6 +14,7 @@
#include "bnxt.h"
#include "bnxt_hwrm.h"
#include "bnxt_coredump.h"
+#include "bnxt_ulp.h"
static const u16 bnxt_bstore_to_seg_id[] = {
[BNXT_CTX_QP] = BNXT_CTX_MEM_SEG_QP,
@@ -414,13 +415,20 @@ static int __bnxt_get_coredump(struct bnxt *bp, u16 dump_type, void *buf,
}
if (dump_type == BNXT_DUMP_DRIVER) {
- u32 drv_len, segs = 0;
+ u32 drv_len, drv_segs, segs = 0;
+ void *drv_buf = NULL;
drv_len = bnxt_get_ctx_coredump(bp, buf, offset, &segs);
+ drv_segs = segs;
+ segs = 0;
+ if (buf)
+ drv_buf = buf + offset + drv_len;
+ drv_len += bnxt_get_ulp_dump(bp, dump_type, drv_buf, &segs);
+ drv_segs += segs;
*dump_len += drv_len;
offset += drv_len;
if (buf)
- coredump.total_segs += segs;
+ coredump.total_segs += drv_segs;
goto err;
}
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c
index e4a7f37..6f640b2 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c
@@ -25,6 +25,7 @@
#include "bnxt.h"
#include "bnxt_hwrm.h"
#include "bnxt_ulp.h"
+#include "bnxt_coredump.h"
static DEFINE_IDA(bnxt_aux_dev_ids);
@@ -393,6 +394,62 @@ void bnxt_register_async_events(struct bnxt_en_dev *edev,
}
EXPORT_SYMBOL(bnxt_register_async_events);
+static void bnxt_ulp_fill_dump_hdr(struct bnxt *bp, void *buf, u32 seg_id,
+ u32 seg_len)
+{
+ struct bnxt_coredump_segment_hdr seg_hdr;
+
+ bnxt_fill_coredump_seg_hdr(bp, &seg_hdr, NULL, seg_len, 0, 0, 0,
+ BNXT_DRV_COMP_ID, seg_id);
+ memcpy(buf, &seg_hdr, sizeof(seg_hdr));
+}
+
+u32 bnxt_get_ulp_dump(struct bnxt *bp, u32 dump_flag, void *buf, u32 *segs)
+{
+ struct bnxt_en_dev *edev = bp->edev;
+ struct bnxt_ulp_dump *dump;
+ struct bnxt_ulp_ops *ops;
+ struct bnxt_ulp *ulp;
+ u32 i, dump_len = 0;
+
+ *segs = 0;
+ if (!edev || !bnxt_ulp_registered(edev))
+ return 0;
+
+ ulp = edev->ulp_tbl;
+ ops = rtnl_dereference(ulp->ulp_ops);
+ if (!ops || !ops->ulp_get_dump_info || !ops->ulp_get_dump_data)
+ return 0;
+
+ dump = &ulp->ulp_dump;
+ if (!buf) {
+ memset(dump, 0, sizeof(*dump));
+ ops->ulp_get_dump_info(ulp->handle, dump_flag, dump);
+ if (dump->segs > BNXT_ULP_MAX_DUMP_SEGS)
+ return 0;
+ for (i = 0; i < dump->segs; i++) {
+ dump_len += dump->seg_tbl[i].seg_len;
+ dump_len += BNXT_SEG_HDR_LEN;
+ }
+ } else {
+ for (i = 0; i < dump->segs; i++) {
+ struct bnxt_ulp_dump_tbl *tbl = &dump->seg_tbl[i];
+ u32 seg_len = tbl->seg_len;
+ u32 seg_id = tbl->seg_id;
+
+ bnxt_ulp_fill_dump_hdr(bp, buf, seg_id, seg_len);
+ buf += BNXT_SEG_HDR_LEN;
+ dump_len += BNXT_SEG_HDR_LEN;
+ ops->ulp_get_dump_data(ulp->handle, seg_id, buf,
+ seg_len);
+ buf += seg_len;
+ dump_len += seg_len;
+ }
+ }
+ *segs = dump->segs;
+ return dump_len;
+}
+
void bnxt_rdma_aux_device_uninit(struct bnxt *bp)
{
struct bnxt_aux_priv *aux_priv;
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h
index 7fa3b8d..9298589 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h
@@ -29,11 +29,31 @@ struct bnxt_msix_entry {
u32 db_offset;
};
+#define BNXT_ULP_MAX_DUMP_SEGS 8
+
+/**
+ * struct bnxt_ulp_dump - bnxt ULP aux device coredump info
+ * @segs: number of coredump segments with info in the seg_tbl
+ * @seg_tbl: coredump segment table
+ * @seg_tbl.seg_id: coredump segment ID
+ * @seg_tbl.seg_len: coredump segment len
+ */
+struct bnxt_ulp_dump {
+ u32 segs;
+ struct bnxt_ulp_dump_tbl {
+ u32 seg_id;
+ u32 seg_len;
+ } seg_tbl[BNXT_ULP_MAX_DUMP_SEGS];
+};
+
struct bnxt_ulp_ops {
/* async_notifier() cannot sleep (in BH context) */
void (*ulp_async_notifier)(void *, struct hwrm_async_event_cmpl *);
void (*ulp_irq_stop)(void *, bool);
void (*ulp_irq_restart)(void *, struct bnxt_msix_entry *);
+ void (*ulp_get_dump_info)(void *handle, u32 dump_flags,
+ struct bnxt_ulp_dump *dump);
+ void (*ulp_get_dump_data)(void *handle, u32 seg_id, void *buf, u32 len);
};
struct bnxt_fw_msg {
@@ -51,6 +71,7 @@ struct bnxt_ulp {
u16 max_async_event_id;
u16 msix_requested;
atomic_t ref_count;
+ struct bnxt_ulp_dump ulp_dump;
};
struct bnxt_en_dev {
@@ -119,6 +140,7 @@ void bnxt_ulp_start(struct bnxt *bp, int err);
void bnxt_ulp_sriov_cfg(struct bnxt *bp, int num_vfs);
void bnxt_ulp_irq_stop(struct bnxt *bp);
void bnxt_ulp_irq_restart(struct bnxt *bp, int err);
+u32 bnxt_get_ulp_dump(struct bnxt *bp, u32 dump_flag, void *buf, u32 *segs);
void bnxt_ulp_async_events(struct bnxt *bp, struct hwrm_async_event_cmpl *cmpl);
void bnxt_rdma_aux_device_uninit(struct bnxt *bp);
void bnxt_rdma_aux_device_del(struct bnxt *bp);
--
2.5.5
^ permalink raw reply related [flat|nested] 15+ messages in thread* Re: [PATCH rdma-next 5/9] bnxt_en: Introduce ULP coredump callbacks
2025-02-20 18:34 ` [PATCH rdma-next 5/9] bnxt_en: Introduce ULP coredump callbacks Selvin Xavier
@ 2025-02-22 0:40 ` Jakub Kicinski
0 siblings, 0 replies; 15+ messages in thread
From: Jakub Kicinski @ 2025-02-22 0:40 UTC (permalink / raw)
To: Selvin Xavier
Cc: leon, jgg, linux-rdma, andrew.gospodarek, kalesh-anakkur.purayil,
netdev, davem, edumazet, abeni, horms, michael.chan
On Thu, 20 Feb 2025 10:34:52 -0800 Selvin Xavier wrote:
> From: Michael Chan <michael.chan@broadcom.com>
>
> Add .ulp_get_dump_info() and .ulp_get_dump_data() callbacks to
> struct bnxt_ulp_ops. When ethtool -w is invoked to get the coredump,
> these 2 callbacks to the bnxt_re auxbus driver will be called if
> they are populated by the bnxt_re driver. The first callback gets
> the number of coredump segments and the size of each coredump
> segment. The 2nd callback copies the coredump data for each segment.
Acked-by: Jakub Kicinski <kuba@kernel.org>
--
pw-bot: nap
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH rdma-next 6/9] RDMA/bnxt_re: Support the dump infrastructure
2025-02-20 18:34 [PATCH rdma-next 0/9] RDMA/bnxt_re: Driver Debug Enhancements Selvin Xavier
` (4 preceding siblings ...)
2025-02-20 18:34 ` [PATCH rdma-next 5/9] bnxt_en: Introduce ULP coredump callbacks Selvin Xavier
@ 2025-02-20 18:34 ` Selvin Xavier
2025-02-20 18:34 ` [PATCH rdma-next 7/9] RDMA/bnxt_re: Dump the debug information in snapdump Selvin Xavier
` (3 subsequent siblings)
9 siblings, 0 replies; 15+ messages in thread
From: Selvin Xavier @ 2025-02-20 18:34 UTC (permalink / raw)
To: leon, jgg
Cc: linux-rdma, andrew.gospodarek, kalesh-anakkur.purayil, netdev,
davem, edumazet, kuba, abeni, horms, michael.chan, Selvin Xavier
From: Michael Chan <michael.chan@broadcom.com>
Adds the stubs for the debug dump in bnxt_re
driver.
Each segment of the dump data holds a particalar resource
type. get_dump_info returns the number of segments and the size
of each segments. Driver currently supports segments for
QP/CQ/SRQ/MR and for the generic host logs.
While taking the dump, get_dump_data will be invoked for each
of the segments and bnxt_re driver returns the data associated
with each segment.
Signed-off-by: Michael Chan <michael.chan@broadcom.com>
Signed-off-by: Selvin Xavier <selvin.xavier@broadcom.com>
---
drivers/infiniband/hw/bnxt_re/main.c | 85 +++++++++++++++++++++++++++++++++++-
1 file changed, 84 insertions(+), 1 deletion(-)
diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c
index 6b5a169..76dd0fa 100644
--- a/drivers/infiniband/hw/bnxt_re/main.c
+++ b/drivers/infiniband/hw/bnxt_re/main.c
@@ -497,10 +497,93 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent)
}
}
+#define BNXT_SEGMENT_ROCE 255
+#define BNXT_SEGMENT_QP_CTX 256
+#define BNXT_SEGMENT_SRQ_CTX 257
+#define BNXT_SEGMENT_CQ_CTX 258
+#define BNXT_SEGMENT_MR_CTX 270
+
+static void bnxt_re_dump_ctx(struct bnxt_re_dev *rdev, u32 seg_id, void *buf,
+ u32 buf_len)
+{
+}
+
+/* bnxt_re_snapdump - Collect RoCE debug data for coredump.
+ * @rdev - rdma device instance
+ * @buf - Pointer to dump buffer
+ * @buf_len - Buffer length
+ *
+ * This function will dump RoCE debug data to the coredump.
+ *
+ * Returns: Nothing
+ *
+ */
+static void bnxt_re_snapdump(struct bnxt_re_dev *rdev, void *buf, u32 buf_len)
+{
+}
+
+#define BNXT_RE_TRACE_DUMP_SIZE 0x2000000
+
+/* bnxt_re_get_dump_info - ULP callback from L2 driver to collect dump info
+ * @handle - en_dev information. L2 and RoCE device information
+ * @dump_flags - ethtool dump flags
+ * @dump - ulp structure containing all coredump segment info
+ *
+ * This function is the callback from the L2 driver to provide the list of
+ * dump segments for the ethtool coredump.
+ *
+ * Returns: Nothing
+ *
+ */
+static void bnxt_re_get_dump_info(void *handle, u32 dump_flags,
+ struct bnxt_ulp_dump *dump)
+{
+ struct bnxt_re_en_dev_info *en_info = auxiliary_get_drvdata(handle);
+ struct bnxt_ulp_dump_tbl *tbl = dump->seg_tbl;
+ struct bnxt_re_dev *rdev = en_info->rdev;
+
+ dump->segs = 5;
+ tbl[0].seg_id = BNXT_SEGMENT_QP_CTX;
+ tbl[0].seg_len = rdev->rcfw.qp_ctxm_size * BNXT_RE_MAX_QDUMP_ENTRIES;
+ tbl[1].seg_id = BNXT_SEGMENT_CQ_CTX;
+ tbl[1].seg_len = rdev->rcfw.cq_ctxm_size * BNXT_RE_MAX_QDUMP_ENTRIES;
+ tbl[2].seg_id = BNXT_SEGMENT_MR_CTX;
+ tbl[2].seg_len = rdev->rcfw.mrw_ctxm_size * BNXT_RE_MAX_QDUMP_ENTRIES;
+ tbl[3].seg_id = BNXT_SEGMENT_SRQ_CTX;
+ tbl[3].seg_len = rdev->rcfw.srq_ctxm_size * BNXT_RE_MAX_QDUMP_ENTRIES;
+ tbl[4].seg_id = BNXT_SEGMENT_ROCE;
+ tbl[4].seg_len = BNXT_RE_TRACE_DUMP_SIZE;
+}
+
+/* bnxt_re_get_dump_data - ULP callback from L2 driver to collect dump data
+ * @handle - en_dev information. L2 and RoCE device information
+ * @seg_id - segment ID of the dump
+ * @buf - dump buffer pointer
+ * @len - length of the buffer
+ *
+ * This function is the callback from the L2 driver to fill the buffer with
+ * coredump data for each segment.
+ *
+ * Returns: Nothing
+ *
+ */
+static void bnxt_re_get_dump_data(void *handle, u32 seg_id, void *buf, u32 len)
+{
+ struct bnxt_re_en_dev_info *en_info = auxiliary_get_drvdata(handle);
+ struct bnxt_re_dev *rdev = en_info->rdev;
+
+ if (seg_id == BNXT_SEGMENT_ROCE)
+ return bnxt_re_snapdump(rdev, buf, len);
+
+ bnxt_re_dump_ctx(rdev, seg_id, buf, len);
+}
+
static struct bnxt_ulp_ops bnxt_re_ulp_ops = {
.ulp_async_notifier = bnxt_re_async_notifier,
.ulp_irq_stop = bnxt_re_stop_irq,
- .ulp_irq_restart = bnxt_re_start_irq
+ .ulp_irq_restart = bnxt_re_start_irq,
+ .ulp_get_dump_info = bnxt_re_get_dump_info,
+ .ulp_get_dump_data = bnxt_re_get_dump_data,
};
/* RoCE -> Net driver */
--
2.5.5
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH rdma-next 7/9] RDMA/bnxt_re: Dump the debug information in snapdump
2025-02-20 18:34 [PATCH rdma-next 0/9] RDMA/bnxt_re: Driver Debug Enhancements Selvin Xavier
` (5 preceding siblings ...)
2025-02-20 18:34 ` [PATCH rdma-next 6/9] RDMA/bnxt_re: Support the dump infrastructure Selvin Xavier
@ 2025-02-20 18:34 ` Selvin Xavier
2025-02-20 18:34 ` [PATCH rdma-next 8/9] RDMA/bnxt_re: Dump the HW context information Selvin Xavier
` (2 subsequent siblings)
9 siblings, 0 replies; 15+ messages in thread
From: Selvin Xavier @ 2025-02-20 18:34 UTC (permalink / raw)
To: leon, jgg
Cc: linux-rdma, andrew.gospodarek, kalesh-anakkur.purayil, netdev,
davem, edumazet, kuba, abeni, horms, michael.chan,
Saravanan Vajravel, Selvin Xavier
From: Saravanan Vajravel <saravanan.vajravel@broadcom.com>
Dump the cached debug information when the L2 driver
invokes the get_dump_data hook.
Signed-off-by: Saravanan Vajravel <saravanan.vajravel@broadcom.com>
Signed-off-by: Selvin Xavier <selvin.xavier@broadcom.com>
---
drivers/infiniband/hw/bnxt_re/main.c | 109 +++++++++++++++++++++++++++++++++++
1 file changed, 109 insertions(+)
diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c
index 76dd0fa..afde0ef 100644
--- a/drivers/infiniband/hw/bnxt_re/main.c
+++ b/drivers/infiniband/hw/bnxt_re/main.c
@@ -508,6 +508,106 @@ static void bnxt_re_dump_ctx(struct bnxt_re_dev *rdev, u32 seg_id, void *buf,
{
}
+/*
+ * bnxt_re_snapdump_qp - Log QP dump to the coredump buffer
+ * @rdev - Pointer to RoCE device instance
+ * @buf - Pointer to dump buffer
+ * @buf_len - Buffer length
+ *
+ * This function will invoke ULP logger to capture snapshot of
+ * SQ/RQ/SCQ/RCQ of a QP in ASCII format.
+ *
+ * Returns: Buffer length
+ */
+static u32 bnxt_re_snapdump_qp(struct bnxt_re_dev *rdev, void *buf, u32 buf_len)
+{
+ u32 index, len = 0, count = 0;
+ struct qdump_qpinfo *qpinfo;
+ struct qdump_array *qdump;
+
+ if (!rdev->qdump_head.qdump)
+ return 0;
+
+ mutex_lock(&rdev->qdump_head.lock);
+ index = rdev->qdump_head.index;
+ while (count < rdev->qdump_head.max_elements) {
+ count++;
+ index = (!index) ? (rdev->qdump_head.max_elements - 1) : (index - 1);
+ qdump = &rdev->qdump_head.qdump[index];
+ if (!qdump->valid || qdump->is_mr)
+ continue;
+
+ qpinfo = &qdump->qpinfo;
+ len += snprintf(buf + len, buf_len - len,
+ "qp_handle 0x%.8x 0x%.8x (xid %d) dest_qp %d type %s state %s ",
+ (u32)(qpinfo->qp_handle >> 32),
+ (u32)(qpinfo->qp_handle & 0xFFFFFFFF),
+ qpinfo->id, qpinfo->dest_qpid,
+ __to_qp_type_str(qpinfo->type),
+ __to_qp_state_str(qpinfo->state));
+ len += snprintf(buf + len, buf_len - len,
+ "is_usr %d scq_handle 0x%llx rcq_handle 0x%llx",
+ qpinfo->is_user, qpinfo->scq_handle, qpinfo->rcq_handle);
+ len += snprintf(buf + len, buf_len - len,
+ "scq_id %d rcq_id %d\n",
+ qpinfo->scq_id, qpinfo->rcq_id);
+ if (len >= buf_len)
+ goto dump_full;
+
+ if (len >= buf_len)
+ goto dump_full;
+ }
+ mutex_unlock(&rdev->qdump_head.lock);
+ return len;
+
+dump_full:
+ mutex_unlock(&rdev->qdump_head.lock);
+ return buf_len;
+}
+
+/*
+ * bnxt_re_snapdump_mr - Log PBL of MR to the coredump buffer
+ * @rdev - Pointer to RoCE device instance
+ * @buf - Pointer to dump buffer
+ * @buf_len - Buffer length
+ *
+ * This function will invoke ULP logger to capture PBL list of
+ * Memory Region in ASCII format.
+ *
+ * Returns: Buffer length
+ */
+static u32 bnxt_re_snapdump_mr(struct bnxt_re_dev *rdev, void *buf, u32 buf_len)
+{
+ u32 index, count = 0, len = 0;
+ struct qdump_array *qdump;
+
+ if (!rdev->qdump_head.qdump)
+ return 0;
+
+ mutex_lock(&rdev->qdump_head.lock);
+ index = rdev->qdump_head.index;
+ while (count < rdev->qdump_head.max_elements) {
+ count++;
+ index = (!index) ? (rdev->qdump_head.max_elements - 1) : (index - 1);
+ qdump = &rdev->qdump_head.qdump[index];
+ if (!qdump->valid || !qdump->is_mr)
+ continue;
+
+ len += snprintf(buf + len, buf_len - len,
+ "**MemoryRegion: type %d lkey 0x%x rkey 0x%x tot_sz %llu **\n",
+ qdump->mrinfo.type, qdump->mrinfo.lkey,
+ qdump->mrinfo.rkey, qdump->mrinfo.total_size);
+ if (len >= buf_len)
+ goto dump_full;
+ }
+ mutex_unlock(&rdev->qdump_head.lock);
+ return len;
+
+dump_full:
+ mutex_unlock(&rdev->qdump_head.lock);
+ return buf_len;
+}
+
/* bnxt_re_snapdump - Collect RoCE debug data for coredump.
* @rdev - rdma device instance
* @buf - Pointer to dump buffer
@@ -520,6 +620,15 @@ static void bnxt_re_dump_ctx(struct bnxt_re_dev *rdev, u32 seg_id, void *buf,
*/
static void bnxt_re_snapdump(struct bnxt_re_dev *rdev, void *buf, u32 buf_len)
{
+ u32 len = 0;
+
+ len += bnxt_re_snapdump_qp(rdev, buf + len, buf_len - len);
+ if (len >= buf_len)
+ return;
+
+ len += bnxt_re_snapdump_mr(rdev, buf + len, buf_len - len);
+ if (len >= buf_len)
+ return;
}
#define BNXT_RE_TRACE_DUMP_SIZE 0x2000000
--
2.5.5
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH rdma-next 8/9] RDMA/bnxt_re: Dump the HW context information
2025-02-20 18:34 [PATCH rdma-next 0/9] RDMA/bnxt_re: Driver Debug Enhancements Selvin Xavier
` (6 preceding siblings ...)
2025-02-20 18:34 ` [PATCH rdma-next 7/9] RDMA/bnxt_re: Dump the debug information in snapdump Selvin Xavier
@ 2025-02-20 18:34 ` Selvin Xavier
2025-02-20 18:34 ` [PATCH rdma-next 9/9] RDMA/bnxt_re: Add support for changing the snap dump level Selvin Xavier
2025-02-23 13:34 ` [PATCH rdma-next 0/9] RDMA/bnxt_re: Driver Debug Enhancements Leon Romanovsky
9 siblings, 0 replies; 15+ messages in thread
From: Selvin Xavier @ 2025-02-20 18:34 UTC (permalink / raw)
To: leon, jgg
Cc: linux-rdma, andrew.gospodarek, kalesh-anakkur.purayil, netdev,
davem, edumazet, kuba, abeni, horms, michael.chan, Kashyap Desai,
Selvin Xavier
From: Kashyap Desai <kashyap.desai@broadcom.com>
Dump the cached HW context information for QP/CQ/SRQ/MRs
when the L2 driver queries using get_dump_data
Signed-off-by: Kashyap Desai <kashyap.desai@broadcom.com>
Signed-off-by: Selvin Xavier <selvin.xavier@broadcom.com>
---
drivers/infiniband/hw/bnxt_re/ib_verbs.c | 10 ++++-----
drivers/infiniband/hw/bnxt_re/main.c | 38 ++++++++++++++++++++++++++++++++
2 files changed, 42 insertions(+), 6 deletions(-)
diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c
index a1ee6ca..dca435e 100644
--- a/drivers/infiniband/hw/bnxt_re/ib_verbs.c
+++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c
@@ -1079,7 +1079,7 @@ int bnxt_re_destroy_qp(struct ib_qp *ib_qp, struct ib_udata *udata)
struct bnxt_qplib_nq *rcq_nq = NULL;
unsigned int flags;
void *ctx_sb_data = NULL;
- bool do_snapdump;
+ bool do_snapdump = false;
u16 ctx_size;
int rc;
@@ -1087,17 +1087,15 @@ int bnxt_re_destroy_qp(struct ib_qp *ib_qp, struct ib_udata *udata)
bnxt_re_debug_rem_qpinfo(rdev, qp);
bnxt_qplib_flush_cqn_wq(&qp->qplib_qp);
- ctx_size = qplib_qp->ctx_size_sb;
+ ctx_size = rdev->rcfw.qp_ctxm_size;
if (ctx_size)
ctx_sb_data = vzalloc(ctx_size);
- do_snapdump = test_bit(QP_FLAGS_CAPTURE_SNAPDUMP, &qplib_qp->flags);
rc = bnxt_qplib_destroy_qp(&rdev->qplib_res, &qp->qplib_qp, ctx_size, ctx_sb_data);
if (rc)
ibdev_err(&rdev->ibdev, "Failed to destroy HW QP");
- else
- bnxt_re_save_resource_context(rdev, ctx_sb_data,
- BNXT_RE_RES_TYPE_QP, do_snapdump);
+ do_snapdump = test_bit(QP_FLAGS_CAPTURE_SNAPDUMP, &qplib_qp->flags);
+ bnxt_re_save_resource_context(rdev, ctx_sb_data, BNXT_RE_RES_TYPE_QP, do_snapdump);
vfree(ctx_sb_data);
if (rdma_is_kernel_res(&qp->ib_qp.res)) {
diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c
index afde0ef..b2bf0d0 100644
--- a/drivers/infiniband/hw/bnxt_re/main.c
+++ b/drivers/infiniband/hw/bnxt_re/main.c
@@ -506,6 +506,44 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent)
static void bnxt_re_dump_ctx(struct bnxt_re_dev *rdev, u32 seg_id, void *buf,
u32 buf_len)
{
+ int ctx_index, i;
+ void *ctx_data;
+ u16 ctx_size;
+
+ switch (seg_id) {
+ case BNXT_SEGMENT_QP_CTX:
+ ctx_data = rdev->rcfw.qp_ctxm_data;
+ ctx_size = rdev->rcfw.qp_ctxm_size;
+ ctx_index = rdev->rcfw.qp_ctxm_data_index;
+ break;
+ case BNXT_SEGMENT_CQ_CTX:
+ ctx_data = rdev->rcfw.cq_ctxm_data;
+ ctx_size = rdev->rcfw.cq_ctxm_size;
+ ctx_index = rdev->rcfw.cq_ctxm_data_index;
+ break;
+ case BNXT_SEGMENT_MR_CTX:
+ ctx_data = rdev->rcfw.mrw_ctxm_data;
+ ctx_size = rdev->rcfw.mrw_ctxm_size;
+ ctx_index = rdev->rcfw.mrw_ctxm_data_index;
+ break;
+ case BNXT_SEGMENT_SRQ_CTX:
+ ctx_data = rdev->rcfw.srq_ctxm_data;
+ ctx_size = rdev->rcfw.srq_ctxm_size;
+ ctx_index = rdev->rcfw.srq_ctxm_data_index;
+ break;
+ default:
+ return;
+ }
+
+ if (!ctx_data || (ctx_size * BNXT_RE_MAX_QDUMP_ENTRIES) > buf_len)
+ return;
+
+ for (i = ctx_index; i < BNXT_RE_MAX_QDUMP_ENTRIES + ctx_index; i++) {
+ memcpy(buf, ctx_data + ((i % BNXT_RE_MAX_QDUMP_ENTRIES) * ctx_size),
+ ctx_size);
+ buf += ctx_size;
+ }
+
}
/*
--
2.5.5
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH rdma-next 9/9] RDMA/bnxt_re: Add support for changing the snap dump level
2025-02-20 18:34 [PATCH rdma-next 0/9] RDMA/bnxt_re: Driver Debug Enhancements Selvin Xavier
` (7 preceding siblings ...)
2025-02-20 18:34 ` [PATCH rdma-next 8/9] RDMA/bnxt_re: Dump the HW context information Selvin Xavier
@ 2025-02-20 18:34 ` Selvin Xavier
2025-02-23 13:34 ` [PATCH rdma-next 0/9] RDMA/bnxt_re: Driver Debug Enhancements Leon Romanovsky
9 siblings, 0 replies; 15+ messages in thread
From: Selvin Xavier @ 2025-02-20 18:34 UTC (permalink / raw)
To: leon, jgg
Cc: linux-rdma, andrew.gospodarek, kalesh-anakkur.purayil, netdev,
davem, edumazet, kuba, abeni, horms, michael.chan, Selvin Xavier
Add support for changing the default debug dump level.
Reviewed-by: Saravanan Vajravel <saravanan.vajravel@broadcom.com>
Signed-off-by: Selvin Xavier <selvin.xavier@broadcom.com>
---
drivers/infiniband/hw/bnxt_re/bnxt_re.h | 2 ++
drivers/infiniband/hw/bnxt_re/debugfs.c | 49 +++++++++++++++++++++++++++++++++
2 files changed, 51 insertions(+)
diff --git a/drivers/infiniband/hw/bnxt_re/bnxt_re.h b/drivers/infiniband/hw/bnxt_re/bnxt_re.h
index 45601635..45d62f6 100644
--- a/drivers/infiniband/hw/bnxt_re/bnxt_re.h
+++ b/drivers/infiniband/hw/bnxt_re/bnxt_re.h
@@ -304,6 +304,8 @@ struct bnxt_re_dev {
struct workqueue_struct *dcb_wq;
/* Head to track all QP dump */
struct bnxt_re_qdump_head qdump_head;
+ struct dentry *tunables;
+ struct dentry *snapdump_dbg;
u8 snapdump_dbg_lvl;
struct dentry *cc_config;
struct bnxt_re_dbg_cc_config_params *cc_config_params;
diff --git a/drivers/infiniband/hw/bnxt_re/debugfs.c b/drivers/infiniband/hw/bnxt_re/debugfs.c
index af91d16..e3fba53 100644
--- a/drivers/infiniband/hw/bnxt_re/debugfs.c
+++ b/drivers/infiniband/hw/bnxt_re/debugfs.c
@@ -306,6 +306,52 @@ static const struct file_operations bnxt_re_cc_config_ops = {
.write = bnxt_re_cc_config_set,
};
+static ssize_t bnxt_re_snapdump_dbg_lvl_set(struct file *filp, const char __user *buffer,
+ size_t count, loff_t *ppos)
+{
+ struct bnxt_re_dev *rdev = filp->private_data;
+ char buf[16];
+ u32 val;
+
+ if (count >= sizeof(buf))
+ return -EINVAL;
+
+ if (copy_from_user(buf, buffer, count))
+ return -EFAULT;
+
+ buf[count] = '\0';
+ if (kstrtou32(buf, 0, &val))
+ return -EINVAL;
+
+ if (val > BNXT_RE_SNAPDUMP_ALL)
+ return -EINVAL;
+
+ rdev->snapdump_dbg_lvl = val;
+
+ return count;
+}
+
+static ssize_t bnxt_re_snapdump_dbg_lvl_get(struct file *filp, char __user *buffer,
+ size_t usr_buf_len, loff_t *ppos)
+{
+ struct bnxt_re_dev *rdev = filp->private_data;
+ char buf[16];
+ int rc;
+
+ rc = snprintf(buf, sizeof(buf), "%d\n", rdev->snapdump_dbg_lvl);
+ if (rc < 0)
+ return rc;
+
+ return simple_read_from_buffer(buffer, usr_buf_len, ppos, (u8 *)(buf), rc);
+}
+
+static const struct file_operations bnxt_re_snapdump_dbg_lvl = {
+ .owner = THIS_MODULE,
+ .open = simple_open,
+ .read = bnxt_re_snapdump_dbg_lvl_get,
+ .write = bnxt_re_snapdump_dbg_lvl_set,
+};
+
void bnxt_re_debugfs_add_pdev(struct bnxt_re_dev *rdev)
{
struct pci_dev *pdev = rdev->en_dev->pdev;
@@ -329,6 +375,9 @@ void bnxt_re_debugfs_add_pdev(struct bnxt_re_dev *rdev)
rdev->cc_config, tmp_params,
&bnxt_re_cc_config_ops);
}
+ rdev->tunables = debugfs_create_dir("tunables", rdev->dbg_root);
+ rdev->snapdump_dbg = debugfs_create_file("snapdump_dbg_lvl", 0400, rdev->tunables, rdev,
+ &bnxt_re_snapdump_dbg_lvl);
}
void bnxt_re_debugfs_rem_pdev(struct bnxt_re_dev *rdev)
--
2.5.5
^ permalink raw reply related [flat|nested] 15+ messages in thread* Re: [PATCH rdma-next 0/9] RDMA/bnxt_re: Driver Debug Enhancements
2025-02-20 18:34 [PATCH rdma-next 0/9] RDMA/bnxt_re: Driver Debug Enhancements Selvin Xavier
` (8 preceding siblings ...)
2025-02-20 18:34 ` [PATCH rdma-next 9/9] RDMA/bnxt_re: Add support for changing the snap dump level Selvin Xavier
@ 2025-02-23 13:34 ` Leon Romanovsky
2025-02-24 9:00 ` Selvin Xavier
9 siblings, 1 reply; 15+ messages in thread
From: Leon Romanovsky @ 2025-02-23 13:34 UTC (permalink / raw)
To: Selvin Xavier
Cc: jgg, linux-rdma, andrew.gospodarek, kalesh-anakkur.purayil,
netdev, davem, edumazet, kuba, abeni, horms, michael.chan
On Thu, Feb 20, 2025 at 10:34:47AM -0800, Selvin Xavier wrote:
> For debugging issues in the field, we need to track some of
> the resources destroyed in the past. This is primarily required
> for tracking certain QPs that encountered errors, leading to
> application exits. A framework has been implemented to
> save this information and retrieve it during coredump collection.
>
> The Broadcom bnxt L2 driver supports collecting driver dumps
> using the ethtool -w option. This feature now also supports
> collecting coredump information from the bnxt_re auxiliary driver.
> Two new callbacks have been implemented to exchange dump
> information supported by the auxbus bnxt_re driver.
>
> The bnxt_re driver caches certain hardware information before
> resources are destroyed in the HW.
Unfortunately, no. The idea that you will cache kernel objects and they
live beyond their HW counterpart doesn't fit RDMA object model.
I'm aware that you are not keeping objects itself, but their shadow
copy. So if you want, your FW can store these failed objects and you
will retrieve them through existing netdev side (ethtool -w ...).
Thanks
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: [PATCH rdma-next 0/9] RDMA/bnxt_re: Driver Debug Enhancements
2025-02-23 13:34 ` [PATCH rdma-next 0/9] RDMA/bnxt_re: Driver Debug Enhancements Leon Romanovsky
@ 2025-02-24 9:00 ` Selvin Xavier
2025-04-01 13:41 ` Jason Gunthorpe
0 siblings, 1 reply; 15+ messages in thread
From: Selvin Xavier @ 2025-02-24 9:00 UTC (permalink / raw)
To: Leon Romanovsky
Cc: jgg, linux-rdma, andrew.gospodarek, kalesh-anakkur.purayil,
netdev, davem, edumazet, kuba, abeni, horms, michael.chan
[-- Attachment #1: Type: text/plain, Size: 2123 bytes --]
On Sun, Feb 23, 2025 at 7:05 PM Leon Romanovsky <leon@kernel.org> wrote:
>
> On Thu, Feb 20, 2025 at 10:34:47AM -0800, Selvin Xavier wrote:
> > For debugging issues in the field, we need to track some of
> > the resources destroyed in the past. This is primarily required
> > for tracking certain QPs that encountered errors, leading to
> > application exits. A framework has been implemented to
> > save this information and retrieve it during coredump collection.
> >
> > The Broadcom bnxt L2 driver supports collecting driver dumps
> > using the ethtool -w option. This feature now also supports
> > collecting coredump information from the bnxt_re auxiliary driver.
> > Two new callbacks have been implemented to exchange dump
> > information supported by the auxbus bnxt_re driver.
> >
> > The bnxt_re driver caches certain hardware information before
> > resources are destroyed in the HW.
>
> Unfortunately, no. The idea that you will cache kernel objects and they
> live beyond their HW counterpart doesn't fit RDMA object model.
Since the scale of the resources are in thousands usually, we can not dump
the debug information to the system logs. So we are not having much context of
the failure and this is the reason for having this new mechanism.
>
> I'm aware that you are not keeping objects itself, but their shadow
> copy. So if you want, your FW can store these failed objects and you
> will retrieve them through existing netdev side (ethtool -w ...).
FW doesn't have enough memory to backup this info. It needs to
be backed up in the host memory and FW has to write it to host memory
when an error happens. This is possible in some newer FW versions.
But itt is not just the HW context that we are caching here. We need to backup
some host side driver/lib info also to correlate with the HW context.
We have been debugging issues like this using our Out of box driver
and we find it useful to get the context
of failure. Some of the internal tools can decode this information and
we want to
have the same behavior between inbox and Out of Box driver.
>
> Thanks
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4224 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH rdma-next 0/9] RDMA/bnxt_re: Driver Debug Enhancements
2025-02-24 9:00 ` Selvin Xavier
@ 2025-04-01 13:41 ` Jason Gunthorpe
2025-04-01 20:48 ` Andy Gospodarek
0 siblings, 1 reply; 15+ messages in thread
From: Jason Gunthorpe @ 2025-04-01 13:41 UTC (permalink / raw)
To: Selvin Xavier
Cc: Leon Romanovsky, linux-rdma, andrew.gospodarek,
kalesh-anakkur.purayil, netdev, davem, edumazet, kuba, abeni,
horms, michael.chan
On Mon, Feb 24, 2025 at 02:30:04PM +0530, Selvin Xavier wrote:
> > I'm aware that you are not keeping objects itself, but their shadow
> > copy. So if you want, your FW can store these failed objects and you
> > will retrieve them through existing netdev side (ethtool -w ...).
> FW doesn't have enough memory to backup this info. It needs to
> be backed up in the host memory and FW has to write it to host memory
> when an error happens. This is possible in some newer FW versions.
> But itt is not just the HW context that we are caching here. We need to backup
> some host side driver/lib info also to correlate with the HW context.
> We have been debugging issues like this using our Out of box driver
> and we find it useful to get the context
> of failure. Some of the internal tools can decode this information and
> we want to
> have the same behavior between inbox and Out of Box driver.
Can you run some kind of daemon in userspace to collect this
information in real time, maybe using fwctl or something instead of
having the driver capture it?
Jason
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH rdma-next 0/9] RDMA/bnxt_re: Driver Debug Enhancements
2025-04-01 13:41 ` Jason Gunthorpe
@ 2025-04-01 20:48 ` Andy Gospodarek
0 siblings, 0 replies; 15+ messages in thread
From: Andy Gospodarek @ 2025-04-01 20:48 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Selvin Xavier, Leon Romanovsky, linux-rdma,
kalesh-anakkur.purayil, netdev, davem, edumazet, kuba, abeni,
horms, michael.chan
On Tue, Apr 1, 2025 at 9:41 AM Jason Gunthorpe <jgg@ziepe.ca> wrote:
>
> On Mon, Feb 24, 2025 at 02:30:04PM +0530, Selvin Xavier wrote:
> > > I'm aware that you are not keeping objects itself, but their shadow
> > > copy. So if you want, your FW can store these failed objects and you
> > > will retrieve them through existing netdev side (ethtool -w ...).
>
> > FW doesn't have enough memory to backup this info. It needs to
> > be backed up in the host memory and FW has to write it to host memory
> > when an error happens. This is possible in some newer FW versions.
> > But itt is not just the HW context that we are caching here. We need to backup
> > some host side driver/lib info also to correlate with the HW context.
> > We have been debugging issues like this using our Out of box driver
> > and we find it useful to get the context
> > of failure. Some of the internal tools can decode this information and
> > we want to
> > have the same behavior between inbox and Out of Box driver.
>
> Can you run some kind of daemon in userspace to collect this
> information in real time, maybe using fwctl or something instead of
> having the driver capture it?
>
Looking at a real-time log is exactly what we are doing. We have
support for infrastructure to do that already so just adding this log
would not be too difficult.
^ permalink raw reply [flat|nested] 15+ messages in thread