* [PATCH for-4.1 01/11] RDMA/ocrdma: Fix EQ destroy failure during driver unload
[not found] ` <1431762709-20740-1-git-send-email-selvin.xavier-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
@ 2015-05-16 7:51 ` Selvin Xavier
2015-05-16 7:51 ` [PATCH for-4.1 02/11] RDMA/ocrdma: Report EQ full fatal error Selvin Xavier
` (9 subsequent siblings)
10 siblings, 0 replies; 21+ messages in thread
From: Selvin Xavier @ 2015-05-16 7:51 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA; +Cc: dledford-H+wXaHxf7aLQT0dZR+AlfA
Changing the destroy sequence of mailbox queue and event queues.
FW expects mailbox queue to be destroyed before desroying the EQs.
Signed-off-by: Selvin Xavier <selvin.xavier-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
Signed-off-by: Devesh Sharma <devesh.sharma-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
---
drivers/infiniband/hw/ocrdma/ocrdma_hw.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_hw.c b/drivers/infiniband/hw/ocrdma/ocrdma_hw.c
index 0c9e959..3a5ea5af 100644
--- a/drivers/infiniband/hw/ocrdma/ocrdma_hw.c
+++ b/drivers/infiniband/hw/ocrdma/ocrdma_hw.c
@@ -3147,9 +3147,9 @@ void ocrdma_cleanup_hw(struct ocrdma_dev *dev)
ocrdma_free_pd_pool(dev);
ocrdma_mbx_delete_ah_tbl(dev);
- /* cleanup the eqs */
- ocrdma_destroy_eqs(dev);
-
/* cleanup the control path */
ocrdma_destroy_mq(dev);
+
+ /* cleanup the eqs */
+ ocrdma_destroy_eqs(dev);
}
--
2.2.0
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 21+ messages in thread* [PATCH for-4.1 02/11] RDMA/ocrdma: Report EQ full fatal error
[not found] ` <1431762709-20740-1-git-send-email-selvin.xavier-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
2015-05-16 7:51 ` [PATCH for-4.1 01/11] RDMA/ocrdma: Fix EQ destroy failure during driver unload Selvin Xavier
@ 2015-05-16 7:51 ` Selvin Xavier
2015-05-16 7:51 ` [PATCH for-4.1 03/11] RDMA/ocrdma: Fix QP state transition in destroy_qp Selvin Xavier
` (8 subsequent siblings)
10 siblings, 0 replies; 21+ messages in thread
From: Selvin Xavier @ 2015-05-16 7:51 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA; +Cc: dledford-H+wXaHxf7aLQT0dZR+AlfA
From: Devesh Sharma <devesh.sharma-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
Detect when Event Queue (EQ) becomes full and print a warning message.
Signed-off-by: Devesh Sharma <devesh.sharma-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
Signed-off-by: Selvin Xavier <selvin.xavier-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
---
drivers/infiniband/hw/ocrdma/ocrdma_hw.c | 6 ++++++
drivers/infiniband/hw/ocrdma/ocrdma_sli.h | 7 +++++++
2 files changed, 13 insertions(+)
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_hw.c b/drivers/infiniband/hw/ocrdma/ocrdma_hw.c
index 3a5ea5af..65759ac 100644
--- a/drivers/infiniband/hw/ocrdma/ocrdma_hw.c
+++ b/drivers/infiniband/hw/ocrdma/ocrdma_hw.c
@@ -933,12 +933,18 @@ static irqreturn_t ocrdma_irq_handler(int irq, void *handle)
struct ocrdma_eqe eqe;
struct ocrdma_eqe *ptr;
u16 cq_id;
+ u8 mcode;
int budget = eq->cq_cnt;
do {
ptr = ocrdma_get_eqe(eq);
eqe = *ptr;
ocrdma_le32_to_cpu(&eqe, sizeof(eqe));
+ mcode = (eqe.id_valid & OCRDMA_EQE_MAJOR_CODE_MASK)
+ >> OCRDMA_EQE_MAJOR_CODE_SHIFT;
+ if (mcode == OCRDMA_MAJOR_CODE_SENTINAL)
+ pr_err("EQ full on eqid = 0x%x, eqe = 0x%x\n",
+ eq->q.id, eqe.id_valid);
if ((eqe.id_valid & OCRDMA_EQE_VALID_MASK) == 0)
break;
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_sli.h b/drivers/infiniband/hw/ocrdma/ocrdma_sli.h
index 243c87c..baf9b8a 100644
--- a/drivers/infiniband/hw/ocrdma/ocrdma_sli.h
+++ b/drivers/infiniband/hw/ocrdma/ocrdma_sli.h
@@ -1624,12 +1624,19 @@ struct ocrdma_delete_ah_tbl_rsp {
enum {
OCRDMA_EQE_VALID_SHIFT = 0,
OCRDMA_EQE_VALID_MASK = BIT(0),
+ OCRDMA_EQE_MAJOR_CODE_MASK = 0x0E,
+ OCRDMA_EQE_MAJOR_CODE_SHIFT = 0x01,
OCRDMA_EQE_FOR_CQE_MASK = 0xFFFE,
OCRDMA_EQE_RESOURCE_ID_SHIFT = 16,
OCRDMA_EQE_RESOURCE_ID_MASK = 0xFFFF <<
OCRDMA_EQE_RESOURCE_ID_SHIFT,
};
+enum major_code {
+ OCRDMA_MAJOR_CODE_COMPLETION = 0x00,
+ OCRDMA_MAJOR_CODE_SENTINAL = 0x01
+};
+
struct ocrdma_eqe {
u32 id_valid;
};
--
2.2.0
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 21+ messages in thread* [PATCH for-4.1 03/11] RDMA/ocrdma: Fix QP state transition in destroy_qp
[not found] ` <1431762709-20740-1-git-send-email-selvin.xavier-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
2015-05-16 7:51 ` [PATCH for-4.1 01/11] RDMA/ocrdma: Fix EQ destroy failure during driver unload Selvin Xavier
2015-05-16 7:51 ` [PATCH for-4.1 02/11] RDMA/ocrdma: Report EQ full fatal error Selvin Xavier
@ 2015-05-16 7:51 ` Selvin Xavier
2015-05-16 7:51 ` [PATCH for-4.1 04/11] RDMA/ocrdma: Use VID 0 if PFC is enabled and vlan is not configured Selvin Xavier
` (7 subsequent siblings)
10 siblings, 0 replies; 21+ messages in thread
From: Selvin Xavier @ 2015-05-16 7:51 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA; +Cc: dledford-H+wXaHxf7aLQT0dZR+AlfA
From: Devesh Sharma <devesh.sharma-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
Don't move QP to error state, if QP is in reset state during QP
destroy operation.
Signed-off-by: Devesh Sharma <devesh.sharma-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
Signed-off-by: Selvin Xavier <selvin.xavier-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
---
drivers/infiniband/hw/ocrdma/ocrdma_verbs.c | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c b/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c
index 8771755..06e8ab7 100644
--- a/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c
+++ b/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c
@@ -1721,18 +1721,20 @@ int ocrdma_destroy_qp(struct ib_qp *ibqp)
struct ocrdma_qp *qp;
struct ocrdma_dev *dev;
struct ib_qp_attr attrs;
- int attr_mask = IB_QP_STATE;
+ int attr_mask;
unsigned long flags;
qp = get_ocrdma_qp(ibqp);
dev = get_ocrdma_dev(ibqp->device);
- attrs.qp_state = IB_QPS_ERR;
pd = qp->pd;
/* change the QP state to ERROR */
- _ocrdma_modify_qp(ibqp, &attrs, attr_mask);
-
+ if (qp->state != OCRDMA_QPS_RST) {
+ attrs.qp_state = IB_QPS_ERR;
+ attr_mask = IB_QP_STATE;
+ _ocrdma_modify_qp(ibqp, &attrs, attr_mask);
+ }
/* ensure that CQEs for newly created QP (whose id may be same with
* one which just getting destroyed are same), dont get
* discarded until the old CQEs are discarded.
--
2.2.0
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 21+ messages in thread* [PATCH for-4.1 04/11] RDMA/ocrdma: Use VID 0 if PFC is enabled and vlan is not configured
[not found] ` <1431762709-20740-1-git-send-email-selvin.xavier-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
` (2 preceding siblings ...)
2015-05-16 7:51 ` [PATCH for-4.1 03/11] RDMA/ocrdma: Fix QP state transition in destroy_qp Selvin Xavier
@ 2015-05-16 7:51 ` Selvin Xavier
2015-05-16 7:51 ` [PATCH for-4.1 05/11] RDMA/ocrdma: Fix the request length for RDMA_QUERY_QP mailbox command to FW Selvin Xavier
` (6 subsequent siblings)
10 siblings, 0 replies; 21+ messages in thread
From: Selvin Xavier @ 2015-05-16 7:51 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA; +Cc: dledford-H+wXaHxf7aLQT0dZR+AlfA
From: Devesh Sharma <devesh.sharma-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
If the adapter ports are in PFC mode and VLAN is not configured,
use vlan tag 0 for RoCE traffic. Also, log an advisory message
in system logs.
Signed-off-by: Padmanabh Ratnakar <padmanabh.ratnakar-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
Signed-off-by: Devesh Sharma <devesh.sharma-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
Signed-off-by: Selvin Xavier <selvin.xavier-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
---
drivers/infiniband/hw/ocrdma/ocrdma_ah.c | 8 +++++++-
drivers/infiniband/hw/ocrdma/ocrdma_hw.c | 12 +++++++++++-
2 files changed, 18 insertions(+), 2 deletions(-)
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_ah.c b/drivers/infiniband/hw/ocrdma/ocrdma_ah.c
index d812904..2a5993b 100644
--- a/drivers/infiniband/hw/ocrdma/ocrdma_ah.c
+++ b/drivers/infiniband/hw/ocrdma/ocrdma_ah.c
@@ -56,7 +56,13 @@ static inline int set_av_attr(struct ocrdma_dev *dev, struct ocrdma_ah *ah,
vlan_tag = attr->vlan_id;
if (!vlan_tag || (vlan_tag > 0xFFF))
vlan_tag = dev->pvid;
- if (vlan_tag && (vlan_tag < 0x1000)) {
+ if (vlan_tag || dev->pfc_state) {
+ if (!vlan_tag) {
+ pr_err("ocrdma%d:Using VLAN with PFC is recommended\n",
+ dev->id);
+ pr_err("ocrdma%d:Using VLAN 0 for this connection\n",
+ dev->id);
+ }
eth.eth_type = cpu_to_be16(0x8100);
eth.roce_eth_type = cpu_to_be16(OCRDMA_ROCE_ETH_TYPE);
vlan_tag |= (dev->sl & 0x07) << OCRDMA_VID_PCP_SHIFT;
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_hw.c b/drivers/infiniband/hw/ocrdma/ocrdma_hw.c
index 65759ac..da688d7 100644
--- a/drivers/infiniband/hw/ocrdma/ocrdma_hw.c
+++ b/drivers/infiniband/hw/ocrdma/ocrdma_hw.c
@@ -2434,7 +2434,7 @@ static int ocrdma_set_av_params(struct ocrdma_qp *qp,
int status;
struct ib_ah_attr *ah_attr = &attrs->ah_attr;
union ib_gid sgid, zgid;
- u32 vlan_id;
+ u32 vlan_id = 0xFFFF;
u8 mac_addr[6];
struct ocrdma_dev *dev = get_ocrdma_dev(qp->ibqp.device);
@@ -2474,12 +2474,22 @@ static int ocrdma_set_av_params(struct ocrdma_qp *qp,
cmd->params.vlan_dmac_b4_to_b5 = mac_addr[4] | (mac_addr[5] << 8);
if (attr_mask & IB_QP_VID) {
vlan_id = attrs->vlan_id;
+ } else if (dev->pfc_state) {
+ vlan_id = 0;
+ pr_err("ocrdma%d:Using VLAN with PFC is recommended\n",
+ dev->id);
+ pr_err("ocrdma%d:Using VLAN 0 for this connection\n",
+ dev->id);
+ }
+
+ if (vlan_id < 0x1000) {
cmd->params.vlan_dmac_b4_to_b5 |=
vlan_id << OCRDMA_QP_PARAMS_VLAN_SHIFT;
cmd->flags |= OCRDMA_QP_PARA_VLAN_EN_VALID;
cmd->params.rnt_rc_sl_fl |=
(dev->sl & 0x07) << OCRDMA_QP_PARAMS_SL_SHIFT;
}
+
return 0;
}
--
2.2.0
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 21+ messages in thread* [PATCH for-4.1 05/11] RDMA/ocrdma: Fix the request length for RDMA_QUERY_QP mailbox command to FW.
[not found] ` <1431762709-20740-1-git-send-email-selvin.xavier-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
` (3 preceding siblings ...)
2015-05-16 7:51 ` [PATCH for-4.1 04/11] RDMA/ocrdma: Use VID 0 if PFC is enabled and vlan is not configured Selvin Xavier
@ 2015-05-16 7:51 ` Selvin Xavier
2015-05-16 7:51 ` [PATCH for-4.1 06/11] RDMA/ocrdma: Prevent allocation of DPP PDs if FW doesnt support it Selvin Xavier
` (5 subsequent siblings)
10 siblings, 0 replies; 21+ messages in thread
From: Selvin Xavier @ 2015-05-16 7:51 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA; +Cc: dledford-H+wXaHxf7aLQT0dZR+AlfA
From: Mitesh Ahuja <mitesh.ahuja-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
Fix ocrdma_query_qp to pass correct mailbox request length to FW.
Signed-off-by: Mitesh Ahuja <mitesh.ahuja-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
Signed-off-by: Selvin Xavier <selvin.xavier-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
---
drivers/infiniband/hw/ocrdma/ocrdma_hw.c | 2 +-
drivers/infiniband/hw/ocrdma/ocrdma_sli.h | 2 ++
2 files changed, 3 insertions(+), 1 deletion(-)
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_hw.c b/drivers/infiniband/hw/ocrdma/ocrdma_hw.c
index da688d7..5636eb6 100644
--- a/drivers/infiniband/hw/ocrdma/ocrdma_hw.c
+++ b/drivers/infiniband/hw/ocrdma/ocrdma_hw.c
@@ -2412,7 +2412,7 @@ int ocrdma_mbx_query_qp(struct ocrdma_dev *dev, struct ocrdma_qp *qp,
struct ocrdma_query_qp *cmd;
struct ocrdma_query_qp_rsp *rsp;
- cmd = ocrdma_init_emb_mqe(OCRDMA_CMD_QUERY_QP, sizeof(*cmd));
+ cmd = ocrdma_init_emb_mqe(OCRDMA_CMD_QUERY_QP, sizeof(*rsp));
if (!cmd)
return status;
cmd->qp_id = qp->id;
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_sli.h b/drivers/infiniband/hw/ocrdma/ocrdma_sli.h
index baf9b8a..02ad0ae 100644
--- a/drivers/infiniband/hw/ocrdma/ocrdma_sli.h
+++ b/drivers/infiniband/hw/ocrdma/ocrdma_sli.h
@@ -1176,6 +1176,8 @@ struct ocrdma_query_qp_rsp {
struct ocrdma_mqe_hdr hdr;
struct ocrdma_mbx_rsp rsp;
struct ocrdma_qp_params params;
+ u32 dpp_credits_cqid;
+ u32 rbq_id;
};
enum {
--
2.2.0
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 21+ messages in thread* [PATCH for-4.1 06/11] RDMA/ocrdma: Prevent allocation of DPP PDs if FW doesnt support it
[not found] ` <1431762709-20740-1-git-send-email-selvin.xavier-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
` (4 preceding siblings ...)
2015-05-16 7:51 ` [PATCH for-4.1 05/11] RDMA/ocrdma: Fix the request length for RDMA_QUERY_QP mailbox command to FW Selvin Xavier
@ 2015-05-16 7:51 ` Selvin Xavier
[not found] ` <1431762709-20740-7-git-send-email-selvin.xavier-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
2015-05-16 7:51 ` [PATCH for-4.1 07/11] RDMA/ocrdma: Fix dmac resolution for link local address Selvin Xavier
` (4 subsequent siblings)
10 siblings, 1 reply; 21+ messages in thread
From: Selvin Xavier @ 2015-05-16 7:51 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA; +Cc: dledford-H+wXaHxf7aLQT0dZR+AlfA
From: Mitesh Ahuja <mitesh.ahuja-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
If DPP PDs are not supported by the FW, allocate only normal PDs.
Signed-off-by: Mitesh Ahuja <mitesh.ahuja-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
Signed-off-by: Devesh Sharma <devesh.sharma-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
Signed-off-by: Selvin Xavier <selvin.xavier-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
---
drivers/infiniband/hw/ocrdma/ocrdma_hw.c | 48 ++++++++++++++---------------
drivers/infiniband/hw/ocrdma/ocrdma_verbs.c | 2 +-
2 files changed, 25 insertions(+), 25 deletions(-)
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_hw.c b/drivers/infiniband/hw/ocrdma/ocrdma_hw.c
index 5636eb6..4b77f34 100644
--- a/drivers/infiniband/hw/ocrdma/ocrdma_hw.c
+++ b/drivers/infiniband/hw/ocrdma/ocrdma_hw.c
@@ -1440,27 +1440,30 @@ static int ocrdma_mbx_alloc_pd_range(struct ocrdma_dev *dev)
struct ocrdma_alloc_pd_range_rsp *rsp;
/* Pre allocate the DPP PDs */
- cmd = ocrdma_init_emb_mqe(OCRDMA_CMD_ALLOC_PD_RANGE, sizeof(*cmd));
- if (!cmd)
- return -ENOMEM;
- cmd->pd_count = dev->attr.max_dpp_pds;
- cmd->enable_dpp_rsvd |= OCRDMA_ALLOC_PD_ENABLE_DPP;
- status = ocrdma_mbx_cmd(dev, (struct ocrdma_mqe *)cmd);
- if (status)
- goto mbx_err;
- rsp = (struct ocrdma_alloc_pd_range_rsp *)cmd;
-
- if ((rsp->dpp_page_pdid & OCRDMA_ALLOC_PD_RSP_DPP) && rsp->pd_count) {
- dev->pd_mgr->dpp_page_index = rsp->dpp_page_pdid >>
- OCRDMA_ALLOC_PD_RSP_DPP_PAGE_SHIFT;
- dev->pd_mgr->pd_dpp_start = rsp->dpp_page_pdid &
- OCRDMA_ALLOC_PD_RNG_RSP_START_PDID_MASK;
- dev->pd_mgr->max_dpp_pd = rsp->pd_count;
- pd_bitmap_size = BITS_TO_LONGS(rsp->pd_count) * sizeof(long);
- dev->pd_mgr->pd_dpp_bitmap = kzalloc(pd_bitmap_size,
- GFP_KERNEL);
+ if (dev->attr.max_dpp_pds) {
+ cmd = ocrdma_init_emb_mqe(OCRDMA_CMD_ALLOC_PD_RANGE,
+ sizeof(*cmd));
+ if (!cmd)
+ return -ENOMEM;
+ cmd->pd_count = dev->attr.max_dpp_pds;
+ cmd->enable_dpp_rsvd |= OCRDMA_ALLOC_PD_ENABLE_DPP;
+ status = ocrdma_mbx_cmd(dev, (struct ocrdma_mqe *)cmd);
+ rsp = (struct ocrdma_alloc_pd_range_rsp *)cmd;
+
+ if (!status && (rsp->dpp_page_pdid & OCRDMA_ALLOC_PD_RSP_DPP) &&
+ rsp->pd_count) {
+ dev->pd_mgr->dpp_page_index = rsp->dpp_page_pdid >>
+ OCRDMA_ALLOC_PD_RSP_DPP_PAGE_SHIFT;
+ dev->pd_mgr->pd_dpp_start = rsp->dpp_page_pdid &
+ OCRDMA_ALLOC_PD_RNG_RSP_START_PDID_MASK;
+ dev->pd_mgr->max_dpp_pd = rsp->pd_count;
+ pd_bitmap_size =
+ BITS_TO_LONGS(rsp->pd_count) * sizeof(long);
+ dev->pd_mgr->pd_dpp_bitmap = kzalloc(pd_bitmap_size,
+ GFP_KERNEL);
+ }
+ kfree(cmd);
}
- kfree(cmd);
cmd = ocrdma_init_emb_mqe(OCRDMA_CMD_ALLOC_PD_RANGE, sizeof(*cmd));
if (!cmd)
@@ -1468,10 +1471,8 @@ static int ocrdma_mbx_alloc_pd_range(struct ocrdma_dev *dev)
cmd->pd_count = dev->attr.max_pd - dev->attr.max_dpp_pds;
status = ocrdma_mbx_cmd(dev, (struct ocrdma_mqe *)cmd);
- if (status)
- goto mbx_err;
rsp = (struct ocrdma_alloc_pd_range_rsp *)cmd;
- if (rsp->pd_count) {
+ if (!status && rsp->pd_count) {
dev->pd_mgr->pd_norm_start = rsp->dpp_page_pdid &
OCRDMA_ALLOC_PD_RNG_RSP_START_PDID_MASK;
dev->pd_mgr->max_normal_pd = rsp->pd_count;
@@ -1486,7 +1487,6 @@ static int ocrdma_mbx_alloc_pd_range(struct ocrdma_dev *dev)
} else {
return -ENOMEM;
}
-mbx_err:
kfree(cmd);
return status;
}
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c b/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c
index 06e8ab7..9dcb660 100644
--- a/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c
+++ b/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c
@@ -365,7 +365,7 @@ static struct ocrdma_pd *_ocrdma_alloc_pd(struct ocrdma_dev *dev,
if (!pd)
return ERR_PTR(-ENOMEM);
- if (udata && uctx) {
+ if (udata && uctx && dev->attr.max_dpp_pds) {
pd->dpp_enabled =
ocrdma_get_asic_type(dev) == OCRDMA_ASIC_GEN_SKH_R;
pd->num_dpp_qp =
--
2.2.0
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 21+ messages in thread* [PATCH for-4.1 07/11] RDMA/ocrdma: Fix dmac resolution for link local address
[not found] ` <1431762709-20740-1-git-send-email-selvin.xavier-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
` (5 preceding siblings ...)
2015-05-16 7:51 ` [PATCH for-4.1 06/11] RDMA/ocrdma: Prevent allocation of DPP PDs if FW doesnt support it Selvin Xavier
@ 2015-05-16 7:51 ` Selvin Xavier
2015-05-16 7:51 ` [PATCH for-4.1 08/11] RDMA/ocrdma: Fail connection for MTU lesser than 512 Selvin Xavier
` (3 subsequent siblings)
10 siblings, 0 replies; 21+ messages in thread
From: Selvin Xavier @ 2015-05-16 7:51 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA; +Cc: dledford-H+wXaHxf7aLQT0dZR+AlfA
From: Mitesh Ahuja <mitesh.ahuja-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
rdma_addr_find_dmac_by_grh fails to resolve dmac for link local address.
Use rdma_get_ll_mac to resolve the link local address.
Signed-off-by: Padmanabh Ratnakar <padmanabh.ratnakar-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
Signed-off-by: Mitesh Ahuja <mitesh.ahuja-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
Signed-off-by: Devesh Sharma <devesh.sharma-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
Signed-off-by: Selvin Xavier <selvin.xavier-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
---
drivers/infiniband/hw/ocrdma/ocrdma.h | 2 ++
drivers/infiniband/hw/ocrdma/ocrdma_ah.c | 4 +++-
2 files changed, 5 insertions(+), 1 deletion(-)
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma.h b/drivers/infiniband/hw/ocrdma/ocrdma.h
index c9780d9..ee9e335 100644
--- a/drivers/infiniband/hw/ocrdma/ocrdma.h
+++ b/drivers/infiniband/hw/ocrdma/ocrdma.h
@@ -515,6 +515,8 @@ static inline int ocrdma_resolve_dmac(struct ocrdma_dev *dev,
memcpy(&in6, ah_attr->grh.dgid.raw, sizeof(in6));
if (rdma_is_multicast_addr(&in6))
rdma_get_mcast_mac(&in6, mac_addr);
+ else if (rdma_link_local_addr(&in6))
+ rdma_get_ll_mac(&in6, mac_addr);
else
memcpy(mac_addr, ah_attr->dmac, ETH_ALEN);
return 0;
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_ah.c b/drivers/infiniband/hw/ocrdma/ocrdma_ah.c
index 2a5993b..f5a5ea836 100644
--- a/drivers/infiniband/hw/ocrdma/ocrdma_ah.c
+++ b/drivers/infiniband/hw/ocrdma/ocrdma_ah.c
@@ -127,7 +127,9 @@ struct ib_ah *ocrdma_create_ah(struct ib_pd *ibpd, struct ib_ah_attr *attr)
goto av_conf_err;
}
- if (pd->uctx) {
+ if ((pd->uctx) &&
+ (!rdma_is_multicast_addr((struct in6_addr *)attr->grh.dgid.raw)) &&
+ (!rdma_link_local_addr((struct in6_addr *)attr->grh.dgid.raw))) {
status = rdma_addr_find_dmac_by_grh(&sgid, &attr->grh.dgid,
attr->dmac, &attr->vlan_id);
if (status) {
--
2.2.0
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 21+ messages in thread* [PATCH for-4.1 08/11] RDMA/ocrdma: Fail connection for MTU lesser than 512
[not found] ` <1431762709-20740-1-git-send-email-selvin.xavier-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
` (6 preceding siblings ...)
2015-05-16 7:51 ` [PATCH for-4.1 07/11] RDMA/ocrdma: Fix dmac resolution for link local address Selvin Xavier
@ 2015-05-16 7:51 ` Selvin Xavier
2015-05-16 7:51 ` [PATCH for-4.1 09/11] RDMA/ocrdma: Add CQ usage statistics counters Selvin Xavier
` (2 subsequent siblings)
10 siblings, 0 replies; 21+ messages in thread
From: Selvin Xavier @ 2015-05-16 7:51 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA; +Cc: dledford-H+wXaHxf7aLQT0dZR+AlfA
From: Naga Irrinki <naga.irrinki-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
HW currently restricts the IB MTU range between 512 and 4096.
Fail connection for MTUs lesser than 512.
Signed-off-by: Naga Irrinki <naga.irrinki-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
Signed-off-by: Selvin Xavier <selvin.xavier-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
---
drivers/infiniband/hw/ocrdma/ocrdma_hw.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_hw.c b/drivers/infiniband/hw/ocrdma/ocrdma_hw.c
index 4b77f34..319675c 100644
--- a/drivers/infiniband/hw/ocrdma/ocrdma_hw.c
+++ b/drivers/infiniband/hw/ocrdma/ocrdma_hw.c
@@ -2535,8 +2535,10 @@ static int ocrdma_set_qp_params(struct ocrdma_qp *qp,
cmd->flags |= OCRDMA_QP_PARA_DST_QPN_VALID;
}
if (attr_mask & IB_QP_PATH_MTU) {
- if (attrs->path_mtu < IB_MTU_256 ||
+ if (attrs->path_mtu < IB_MTU_512 ||
attrs->path_mtu > IB_MTU_4096) {
+ pr_err("ocrdma%d: IB MTU %d is not supported\n",
+ dev->id, ib_mtu_enum_to_int(attrs->path_mtu));
status = -EINVAL;
goto pmtu_err;
}
--
2.2.0
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 21+ messages in thread* [PATCH for-4.1 09/11] RDMA/ocrdma: Add CQ usage statistics counters
[not found] ` <1431762709-20740-1-git-send-email-selvin.xavier-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
` (7 preceding siblings ...)
2015-05-16 7:51 ` [PATCH for-4.1 08/11] RDMA/ocrdma: Fail connection for MTU lesser than 512 Selvin Xavier
@ 2015-05-16 7:51 ` Selvin Xavier
[not found] ` <1431762709-20740-10-git-send-email-selvin.xavier-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
2015-05-16 7:51 ` [PATCH for-4.1 10/11] RDMA/ocrdma: Prevent returning failures in the resource destroy path Selvin Xavier
2015-05-16 7:51 ` [PATCH for-4.1 11/11] RDMA/ocrdma: Update ocrdma version number Selvin Xavier
10 siblings, 1 reply; 21+ messages in thread
From: Selvin Xavier @ 2015-05-16 7:51 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA; +Cc: dledford-H+wXaHxf7aLQT0dZR+AlfA
From: Meghana Cheripady <meghana.cheripady-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
Add counters for tracking the CQ usage by applications.
Signed-off-by: Meghana Cheripady <meghana.cheripady-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
Signed-off-by: Selvin Xavier <selvin.xavier-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
---
drivers/infiniband/hw/ocrdma/ocrdma.h | 2 ++
drivers/infiniband/hw/ocrdma/ocrdma_hw.c | 8 ++++++++
drivers/infiniband/hw/ocrdma/ocrdma_stats.c | 15 +++++++++++++++
drivers/infiniband/hw/ocrdma/ocrdma_verbs.c | 2 ++
4 files changed, 27 insertions(+)
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma.h b/drivers/infiniband/hw/ocrdma/ocrdma.h
index ee9e335..8782e1f 100644
--- a/drivers/infiniband/hw/ocrdma/ocrdma.h
+++ b/drivers/infiniband/hw/ocrdma/ocrdma.h
@@ -290,6 +290,8 @@ struct ocrdma_dev {
struct dentry *dir;
atomic_t async_err_stats[OCRDMA_MAX_ASYNC_ERRORS];
atomic_t cqe_err_stats[OCRDMA_MAX_CQE_ERR];
+ atomic_t act_cq_stats;
+ atomic_t th_cq_stats;
struct ocrdma_pd_resource_mgr *pd_mgr;
};
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_hw.c b/drivers/infiniband/hw/ocrdma/ocrdma_hw.c
index 319675c..596d8c9 100644
--- a/drivers/infiniband/hw/ocrdma/ocrdma_hw.c
+++ b/drivers/infiniband/hw/ocrdma/ocrdma_hw.c
@@ -1720,6 +1720,7 @@ int ocrdma_mbx_create_cq(struct ocrdma_dev *dev, struct ocrdma_cq *cq,
struct ocrdma_create_cq *cmd;
struct ocrdma_create_cq_rsp *rsp;
u32 hw_pages, cqe_size, page_size, cqe_count;
+ int act_cq_cnt, th_cq_cnt;
if (entries > dev->attr.max_cqe) {
pr_err("%s(%d) max_cqe=0x%x, requester_cqe=0x%x\n",
@@ -1808,6 +1809,13 @@ int ocrdma_mbx_create_cq(struct ocrdma_dev *dev, struct ocrdma_cq *cq,
rsp = (struct ocrdma_create_cq_rsp *)cmd;
cq->id = (u16) (rsp->rsp.cq_id & OCRDMA_CREATE_CQ_RSP_CQ_ID_MASK);
+
+ atomic_inc(&dev->act_cq_stats);
+ act_cq_cnt = atomic_read(&dev->act_cq_stats);
+ th_cq_cnt = atomic_read(&dev->th_cq_stats);
+ if (th_cq_cnt < act_cq_cnt)
+ atomic_set(&dev->th_cq_stats, act_cq_cnt);
+
kfree(cmd);
return 0;
mbx_err:
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_stats.c b/drivers/infiniband/hw/ocrdma/ocrdma_stats.c
index 48d7ef5..47fef14 100644
--- a/drivers/infiniband/hw/ocrdma/ocrdma_stats.c
+++ b/drivers/infiniband/hw/ocrdma/ocrdma_stats.c
@@ -146,6 +146,8 @@ static char *ocrdma_resource_stats(struct ocrdma_dev *dev)
(u64)rsrc_stats->phy_mr);
pcur += ocrdma_add_stat(stats, pcur, "active_mw",
(u64)rsrc_stats->mw);
+ pcur += ocrdma_add_stat(stats, pcur, "active_cq_usage_stats",
+ (u64)atomic_read(&dev->act_cq_stats));
/* Print the threshold stats */
rsrc_stats = &rdma_stats->th_rsrc_stats;
@@ -200,6 +202,8 @@ static char *ocrdma_resource_stats(struct ocrdma_dev *dev)
(u64)rsrc_stats->phy_mr);
pcur += ocrdma_add_stat(stats, pcur, "threshold_mw",
(u64)rsrc_stats->mw);
+ pcur += ocrdma_add_stat(stats, pcur, "threshold_cq_usage_stats",
+ (u64)atomic_read(&dev->th_cq_stats));
return stats;
}
@@ -745,6 +749,16 @@ static const struct file_operations ocrdma_dbg_ops = {
.write = ocrdma_dbgfs_ops_write,
};
+static void ocrdma_init_cq_stats(struct ocrdma_dev *dev)
+{
+ int i, cq_cnt = 0;
+
+ for (i = 0; i < dev->eq_cnt; i++)
+ cq_cnt += dev->eq_tbl[i].cq_cnt;
+ atomic_set(&dev->act_cq_stats, cq_cnt);
+ atomic_set(&dev->th_cq_stats, cq_cnt);
+}
+
void ocrdma_add_port_stats(struct ocrdma_dev *dev)
{
if (!ocrdma_dbgfs_dir)
@@ -828,6 +842,7 @@ void ocrdma_add_port_stats(struct ocrdma_dev *dev)
goto err;
mutex_init(&dev->stats_lock);
+ ocrdma_init_cq_stats(dev);
return;
err:
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c b/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c
index 9dcb660..48ec56f 100644
--- a/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c
+++ b/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c
@@ -1119,6 +1119,8 @@ int ocrdma_destroy_cq(struct ib_cq *ibcq)
dev->nic_info.db_page_size);
}
+ atomic_dec(&dev->act_cq_stats);
+
kfree(cq);
return 0;
}
--
2.2.0
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 21+ messages in thread* [PATCH for-4.1 10/11] RDMA/ocrdma: Prevent returning failures in the resource destroy path
[not found] ` <1431762709-20740-1-git-send-email-selvin.xavier-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
` (8 preceding siblings ...)
2015-05-16 7:51 ` [PATCH for-4.1 09/11] RDMA/ocrdma: Add CQ usage statistics counters Selvin Xavier
@ 2015-05-16 7:51 ` Selvin Xavier
[not found] ` <1431762709-20740-11-git-send-email-selvin.xavier-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
2015-05-16 7:51 ` [PATCH for-4.1 11/11] RDMA/ocrdma: Update ocrdma version number Selvin Xavier
10 siblings, 1 reply; 21+ messages in thread
From: Selvin Xavier @ 2015-05-16 7:51 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA; +Cc: dledford-H+wXaHxf7aLQT0dZR+AlfA
Avoid returning FW failures in the resource destroy verbs.
Signed-off-by: Padmanabh Ratnakar <padmanabh.ratnakar-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
Signed-off-by: Devesh Sharma <devesh.sharma-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
Signed-off-by: Selvin Xavier <selvin.xavier-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
---
drivers/infiniband/hw/ocrdma/ocrdma_verbs.c | 17 +++++++----------
1 file changed, 7 insertions(+), 10 deletions(-)
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c b/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c
index 48ec56f..b1f9d7d 100644
--- a/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c
+++ b/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c
@@ -531,13 +531,12 @@ map_err:
int ocrdma_dealloc_ucontext(struct ib_ucontext *ibctx)
{
- int status = 0;
struct ocrdma_mm *mm, *tmp;
struct ocrdma_ucontext *uctx = get_ocrdma_ucontext(ibctx);
struct ocrdma_dev *dev = get_ocrdma_dev(ibctx->device);
struct pci_dev *pdev = dev->nic_info.pdev;
- status = ocrdma_dealloc_ucontext_pd(uctx);
+ (void)ocrdma_dealloc_ucontext_pd(uctx);
ocrdma_del_mmap(uctx, uctx->ah_tbl.pa, uctx->ah_tbl.len);
dma_free_coherent(&pdev->dev, uctx->ah_tbl.len, uctx->ah_tbl.va,
@@ -548,7 +547,7 @@ int ocrdma_dealloc_ucontext(struct ib_ucontext *ibctx)
kfree(mm);
}
kfree(uctx);
- return status;
+ return 0;
}
int ocrdma_mmap(struct ib_ucontext *context, struct vm_area_struct *vma)
@@ -690,7 +689,6 @@ int ocrdma_dealloc_pd(struct ib_pd *ibpd)
struct ocrdma_pd *pd = get_ocrdma_pd(ibpd);
struct ocrdma_dev *dev = get_ocrdma_dev(ibpd->device);
struct ocrdma_ucontext *uctx = NULL;
- int status = 0;
u64 usr_db;
uctx = pd->uctx;
@@ -704,11 +702,11 @@ int ocrdma_dealloc_pd(struct ib_pd *ibpd)
if (is_ucontext_pd(uctx, pd)) {
ocrdma_release_ucontext_pd(uctx);
- return status;
+ return 0;
}
}
- status = _ocrdma_dealloc_pd(dev, pd);
- return status;
+ (void)_ocrdma_dealloc_pd(dev, pd);
+ return 0;
}
static int ocrdma_alloc_lkey(struct ocrdma_dev *dev, struct ocrdma_mr *mr,
@@ -1905,13 +1903,12 @@ int ocrdma_query_srq(struct ib_srq *ibsrq, struct ib_srq_attr *srq_attr)
int ocrdma_destroy_srq(struct ib_srq *ibsrq)
{
- int status;
struct ocrdma_srq *srq;
struct ocrdma_dev *dev = get_ocrdma_dev(ibsrq->device);
srq = get_ocrdma_srq(ibsrq);
- status = ocrdma_mbx_destroy_srq(dev, srq);
+ (void)ocrdma_mbx_destroy_srq(dev, srq);
if (srq->pd->uctx)
ocrdma_del_mmap(srq->pd->uctx, (u64) srq->rq.pa,
@@ -1920,7 +1917,7 @@ int ocrdma_destroy_srq(struct ib_srq *ibsrq)
kfree(srq->idx_bit_fields);
kfree(srq->rqe_wr_id_tbl);
kfree(srq);
- return status;
+ return 0;
}
/* unprivileged verbs and their support functions. */
--
2.2.0
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 21+ messages in thread* [PATCH for-4.1 11/11] RDMA/ocrdma: Update ocrdma version number
[not found] ` <1431762709-20740-1-git-send-email-selvin.xavier-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
` (9 preceding siblings ...)
2015-05-16 7:51 ` [PATCH for-4.1 10/11] RDMA/ocrdma: Prevent returning failures in the resource destroy path Selvin Xavier
@ 2015-05-16 7:51 ` Selvin Xavier
10 siblings, 0 replies; 21+ messages in thread
From: Selvin Xavier @ 2015-05-16 7:51 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA; +Cc: dledford-H+wXaHxf7aLQT0dZR+AlfA
Updating the driver version to 10.6.0.0
Signed-off-by: Selvin Xavier <selvin.xavier-1wcpHE2jlwO1Z/+hSey0Gg@public.gmane.org>
---
drivers/infiniband/hw/ocrdma/ocrdma.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma.h b/drivers/infiniband/hw/ocrdma/ocrdma.h
index 8782e1f..5ea13f2 100644
--- a/drivers/infiniband/hw/ocrdma/ocrdma.h
+++ b/drivers/infiniband/hw/ocrdma/ocrdma.h
@@ -40,7 +40,7 @@
#include <be_roce.h>
#include "ocrdma_sli.h"
-#define OCRDMA_ROCE_DRV_VERSION "10.4.205.0u"
+#define OCRDMA_ROCE_DRV_VERSION "10.6.0.0"
#define OCRDMA_ROCE_DRV_DESC "Emulex OneConnect RoCE Driver"
#define OCRDMA_NODE_DESC "Emulex OneConnect RoCE HCA"
--
2.2.0
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 21+ messages in thread